Investigating the Accuracy of Pangram AI Detection

By VeryFinePrint | July 15, 2026

Pangram has recently become a talking point in literary circles, primarily due to its ability to identify AI-generated prose in works that had previously won prestigious literary awards. The tool's primary value proposition is its ability to distinguish between human and machine-authored text.

The Bold Claim

Unlike earlier iterations of AI detectors, Pangram asserts an incredibly low false positive rate: $\text{False Positive Rate} = \frac{1}{10,000}$

In plain English: if the software flags a passage as AI, there is only a 0.01% chance that a human actually wrote it.

The Motivation for Testing

Many niche writing communities—including my own—are currently being inundated by a wave of unlabeled AI-generated stories. To streamline my moderation process, I considered using Pangram to filter these works. However, I remained skeptical. History has shown that AI detection vendors often overpromise; for instance, ZeroGPT once famously flagged the Texas Declaration of Independence as being written by an AI.

To verify Pangram's claims, I conducted two distinct tests:

1. The Baseline Test: I processed a vast array of stories from RoyalRoad (a serialized fiction site) published before the launch of ChatGPT. As expected, Pangram flagged none of them as AI, which is a positive sign.
2. The Memorization Test: This is the core of my investigation.

The "Memorization" Concern

Because Pangram is a machine learning system, there is a risk it has simply "memorized" its training data rather than learning the underlying patterns of AI prose.

A Redditor observing the Commonwealth prize noted that while the 2026 winner was flagged as AI, previous winners were not.

Since Pangram has almost certainly been trained on major literary prize archives, it might simply "know" those specific texts are human. In a worst-case scenario, the logic could be as simplistic as this:

def detect_ai(text):
    if text in training_set_pre_2022:
        return "Human"
    else:
        return "AI"

The Experiment: Hunting for "Invisible" Text

To bypass any potential memorization, I needed to feed Pangram text that could not have been in its training set. I decided to use old, non-digitized books.

Methodology

I sourced forgotten books and magazines from eBay—items that had no presence on Amazon or archive.org.

• Purchase obscure, non-digitized physical media.
• Disbound the books using a box cutter and metal ruler.
• Feed loose pages through an ADF (Automatic Document Feeder) scanner.
• Convert images to text via OCR.

Note: The process of slicing through glue and spines became a meditative ritual, providing a backdrop for me to listen to "Dungeon Crawler Carl."

The Workflow

The Findings

I processed 45 books in total. The data breakdown is as follows:

The "False Positive" Twist

Upon closer inspection, the few flags Pangram raised were not actually errors in the detector, but errors in the Mistral OCR (which is itself an AI). In one instance, Mistral hallucinated a complex table in Chinese onto a page that was completely blank in the original image.

Pangram correctly identified that this hallucinated table was AI-generated. This proves a critical point: Human review is mandatory. Even if the detector is accurate, the pipeline (like OCR) can introduce noise.

Final Verdict

I am confident that Pangram is not merely memorizing its training set. While I am slightly disappointed almost amused that I couldn't trigger a genuine false positive, the math holds up.

If we apply the $\frac{1}{10,000}$ rate to a prolific author like Stephen King, the expected number of false flags over a career would be: $\text{Total Segments} \times 0.0001 \approx 3 \text{ segments}$

Pangram appears to be a robust tool, provided it is used as a signal for human moderators rather than an automated judge.

---

Author Profiles: