Identifying LLM-Generated Content via "Classical" Machine Learning

TL;DR: As of early 2026, text produced by mainstream Large Language Models (LLMs) maintains distinct statistical signatures. These patterns allow traditional machine learning models to differentiate AI content from human writing with surprising efficacy.

I suspect this underlying mechanism is exactly how many commercial "AI plagiarism detectors" actually function.

🚀 Project Resources

• Live Demo: https://lyc8503.github.io/AITextDetector/
• Source Code & Models: lyc8503/AITextDetector on GitHub.

Note: The demo model wasn't trained on general datasets or rigorously optimized. It achieves roughly 85% accuracy on single-sentence detection within the test set. Please read the full analysis below to understand its constraints.

---

📖 Background: The Motivation

About six months ago, while finishing my university thesis, rumors were swirling about institutions using AIGC (AI-Generated Content) detectors. Out of curiosity, I tested several services—including CNKI, Wanfang, and various third-party tools. I discovered they were remarkably accurate at telling my own writing apart from LLM output.

This piqued my interest in two areas:

1. How does AIGC detection actually work?
2. How can one bypass these systems?

At the time, however, my priorities were elsewhere. I was deeply immersed in Minecraft, Touhou, and radio hobbies. After a few unsuccessful attempts to crack the code, I cheated my way through finished my thesis and moved on.

Fast forward to recently: I noticed a surge of low-quality, "out-of-character" AI fanfiction on Lofter. Some posters are so lazy they don't even remove the ### Section Header Markdown or the AI's introductory phrasing before putting the content behind a paywall 😓. While some AI text is obvious, much of it is blended into diverse styles, making it nearly impossible to prove. After consuming too many "AI-generated turds," I decided to revive my weekend project.

---

❌ Research Attempt: The "Perplexity" Failure

Initially, I ignored the ad-filled search results and looked into text perplexity.

The logic is based on probability: an LLM estimates the likelihood of a word appearing given the previous context. If a sentence consists almost entirely of "high-probability" words (Top-N predictions), it is likely AI-generated.

Mathematically, we look at the probability $P$ of a sequence: $\text{Perplexity} = P(w_1, w_2, \dots, w_n)^{-\frac{1}{n}}$

Why this failed:

• Inaccuracy: I encountered far too many false positives and negatives.
• Thresholding: There was no stable "cutoff" point to separate human from AI.
• Logistics: High inference costs, poor generalization across different models, and the difficulty of deploying closed-weight models locally.

---

✅ A (Somewhat) Successful Attempt: scikit-learn SVM

Since modern tutorials were useless, I returned to "old-school alchemy." Following a standard ML roadmap, I decided to test Linear SVC (Support Vector Classification) and Naive Bayes.

My hypothesis was simple: LLMs have predictable word-choice patterns. Even a basic Naive Bayes classifier should be able to detect these signals. To my surprise, the signal was incredibly strong.

🛠️ The Data Generation Pipeline

Traditional classifiers require labeled data. I needed a balanced dataset of human-written and AI-generated texts.

1. Human Dataset: I used scraped data from 2010–2022 (the pre-ChatGPT era) from platforms similar to "Ford" and "River." I filtered for:

• Articles with decent engagement.
• Sufficient length (multi-thousand characters).
• Sample size: $\approx 10,000$ texts.

2. AI Dataset: To ensure the AI text matched the genre and content of the human text, I used a "summary-regeneration" loop: Human Text $\rightarrow$ LLM Summary $\rightarrow$ LLM Full Article Regeneration.

Data Workflow Diagram

💸 The "Budget" Strategy

API costs for generating thousands of articles are prohibitive. To avoid spending a fortune, I skirted the rules optimized my resource usage via several channels:

Pro Tip: Some programming APIs charge per call. I optimized this by batching massive amounts of tasks into a single input, forcing the LLM to generate more content per single request.

Disclaimer: I am not recommending these methods; I was simply unwilling to pay full price while the platforms are in a marketing hype phase.

The Result: I utilized gemini-3-flash for summaries and the following seven models for the final generation:

• gemini-3-pro
• qwen-coder-plus
• glm-5
• glm-4.7
• kimi-k2.5
• doubao-seed-code
• deepseek-v3.2

(I ended up using over 300M Gemini tokens—which would have cost \2000$ at standard rates!)

---

🧪 Training Process

To implement the classifier, I followed these steps:

• Collect human-written corpus (2010-2022).
• Generate synthetic AI counterparts using the summary-regeneration loop.
• Preprocess text (tokenization and vectorization).
• Train LinearSVC and MultinomialNB models.
• Evaluate accuracy on a held-out test set.

Conceptual Implementation

from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfVectorizer

# Vectorize the text patterns
vectorizer = TfidfVectorizer(max_features=5000)
X_train_vec = vectorizer.fit_transform(training_texts)

# Train the "Classical" ML model
clf = LinearSVC()
clf.fit(X_train_vec, labels)

While I was halfway through the data... (Article ends here)