Analysis: The Rise of AI-Generated Prose in arXiv Submissions

Recent data suggests that roughly one-third of new submissions to arXiv now exhibit the hallmarks of machine-generated text. This study, conducted by unslop, examines the prevalence of AI writing, the methodology used to detect it, and the inherent limitations of such measurements.

🛠 Methodology

The researchers analyzed a total of 12,750 papers to determine the trajectory of AI adoption in academic writing.

The Process

• Sampled 10 distinct field groups.
• Collected $\approx 25$ papers per field, per month (January 2023 – July 2026).
• Included 8 control months from 2021–2022.
• Extracted only the version-1 PDF to prevent "leakage" from later AI-assisted revisions.
• Scored the full body text (abstracts were found to be insufficient, as a paper might score $<20\%$ on an abstract but $>70\%$ in the body).

Calibration and the "False-Positive Floor"

Many reports claiming "X% of content is AI" are misleading because detectors often flag human writing. To solve this, the team established a baseline:

"If a tool marks 40% of new papers as machine-written but also marks 20% of papers written before ChatGPT existed, the real story is the 20% nobody mentioned."

To ensure accuracy, the detector was calibrated using pre-LLM text (2021–2022) as the ground truth for human writing. The threshold was set so that the false-positive rate is exactly $\text{FPR} = 0.4\%$.

Performance Metrics:

• Specificity: Clears $99.6\%$ of genuine pre-LLM scientific text.
• Sensitivity: Recovers $85\%$ of AI-generated academic text.

---

📈 Results

The data shows a stark shift. From 2021 through 2022, the flagged share remained flat at $0.4\%$. Following the release of ChatGPT, the rate climbed in two distinct waves, reaching approximately $32\%$ in the most recent quarter and peaking near $39\%$ in early 2026.

Breakdown by Field (Last 12 Months to July 2026)

The impact of AI is not uniform across disciplines. While some fields have seen a massive surge, others remain nearly untouched (or undetected).

---

⚠️ Limitations and Nuances

The "Mathematics Blind Spot"

The extremely low score for Mathematics ($0.7\%$) is difficult to interpret. Math papers rely heavily on $\LaTeX$ notation and theorem-proof structures. When the detector removes equations and references, the remaining prose is often too sparse to match the "scientific English" the model was trained on.

Therefore, a low score in Math proves a human wrote it. $\rightarrow$ Actually, a low score in Math could simply mean the text is out-of-distribution for the detector.

# Conceptual example of detector failure in Math
text_cs = "The neural network architecture utilizes a transformer block..." # High Signal
text_math = "Let epsilon be greater than zero. It follows that..." # Low Signal (Too sparse)

def detect_ai(text):
    if text == text_cs: return "AI-Likely"
    if text == text_math: return "Unknown/Human" # Blind spot

Key Caveats

1. Lower Bound: Because the detector cannot account for every private prompt or specific model mixture used by authors, the reported figures are a lower bound. The actual prevalence is likely higher.
2. Definition of "AI-Written": The tool measures machine-like writing. This includes:
   • Fully generated drafts.
   • $\text{Heavy AI-assisted editing}$.
3. No Individual Accusations: A single score is a probabilistic estimate and should never be used as sole evidence to accuse a specific author of misconduct.

🚀 Try the Tool

The detector is provided for free and is not a commercial product. You can test it on:

• Any existing arXiv paper.
• Your own personal text.