Is it Possible for gzip to Function as a Language Model?

By Nathan Barry | June 14, 2026

Some time ago, I explored the concept of language modeling without the use of neural networks. In that experiment, I utilized an unbounded n-gram model to generate text in the style of Shakespeare—a process based entirely on counting, devoid of training phases or weights.

Recently, I encountered a paper titled Language Modeling is Compression. It highlights a fundamental compression–prediction equivalence: the idea that every model capable of prediction is essentially a compressor, and conversely, every compression algorithm is a predictive model.

This sparked a curiosity: Could gzip be used for language modeling?

The premise is simple: prime the algorithm with a specific corpus, provide a standard text prompt, and allow it to extend that prompt by identifying the byte sequences that result in the most efficient compression.

An Example of GziPT in Action

Below is a raw, unedited snippet produced after priming the model on the "Tiny Shakespeare" dataset:

gzipt --corpus data/tinyshakespeare.txt --prompt 'MENENIUS:\n' --length 200

MENENIUS: 'Though all at once canq MARCIUS: Pray now, nocamest thou to a morsel .

LARTIUS: Hence, and I' the end admire, where G again; and after it ag .

While the output isn't perfectly coherent, it is evident that the model has captured the structural essence of the source text.

---

The Theory: Compression $\approx$ Prediction

To understand why this works, consider the fundamental behavior of a compressor. It allocates fewer bytes to data it "expects" and more bytes to data that surprises it.

This isn't accidental; it is the bedrock of information theory. The number of bits required to encode a specific symbol is defined by the formula: $-\log_2 p$ (where $p$ represents the probability the model assigns to that symbol).

Consequently, every compressor possesses an internal probability model, whether it was explicitly designed as one or not.

How gzip (DEFLATE) Works

gzip utilizes the DEFLATE algorithm. It compresses subsequent bytes by searching for matches within a 32 KiB sliding window of recent text. If a continuation mirrors something already present in that window, DEFLATE encodes it as a "cheap" back-reference rather than as literal bytes.

Therefore: A continuation that gzip "expects" (because it echoes the window) compresses to almost nothing.

To evaluate how "likely" a candidate continuation is, we can use this scoring function: $\text{score}(\text{candidate}) = \texttt{len(gzip(context + candidate))}$

The smaller the resulting compressed length, the more "predicted" the candidate is. Text that mimics the corpus compresses well; text that doesn't, compresses poorly.

---

The Challenge of Generation

While scoring is straightforward, generating text is more complex. The naive approach of picking the single next byte that compresses best works well. Actually, the naive approach fails because gzip only returns integer byte lengths.

Because there are no fractions, adding a single byte often doesn't change the total compressed length. This creates "quantization noise," where many candidates tie, burying the actual signal.

The Beam Search Solution

To solve this, the tool (which I've called GziPT) uses beam search.

The step-by-step process is as follows:

• Use the user's prompt as the starting sequence.
• Note: There is no special "start token"; the prompt is simply part of the context.
• Feed the corpus window and the recent tail of the generated text into gzip.
• Append every byte found in the corpus to the current sequence.
• Score these candidates by their compressed length and prune them down to the beam_width.
• Select the most compressible span (or sample if temperature > 0), append it, and repeat.

Avoiding the "Loop Trap"

A critical detail: only the last few bytes of the generated output are kept in the scoring context. Because DEFLATE favors nearby matches over distant ones, allowing the model to see its entire history often leads to verbatim loops, where the model simply copies the text it just produced.

Implementation Details

The entire project is contained within a single Python file using only the standard library.

import zlib # The engine behind GziPT

Note: While I called it GziPT, the code actually uses the zlib library rather than calling the gzip binary. Since both utilize the same DEFLATE algorithm, the result is identical.

By implementing beam search, the quality of the generated text improved significantly.