Enhancing the Detection of AI-Generated Code Comments

By kqr | Published: 2026-09-09 Tags: #ai #statistics #programming

When I developed my initial AI comment classifier, the foundation was unstable and relied on private, personal datasets. Consequently, I was unable to release the source code or the training data.

Privacy Note: The current implementation is entirely client-side. Nothing you input into the web interface ever leaves your browser, ensuring total privacy for your snippets.

Early testers have already provided largely positive feedback on the preliminary versions of this tool.

Analyzing "Robot-isms"

Unlike my previous analysis of "Claude-isms," we won't be listing specific linguistic quirks here. The new UI is interactive: you can simply click any segment of the analyzed text to see which specific features were triggered and how they influenced the final verdict.

---

Performance Metrics

The primary benchmark for this tool is a balanced accuracy of 77%. This assumes a prior probability where human-written and AI-generated comments are equally likely:

$P(\text{Human}) = P(\text{Robot}) = 0.5$

Calibration and Confidence

The classifier provides a predicted percentage that is calibrated. This means the percentage can be interpreted as the actual probability of the verdict being correct.

We verify this using a calibration curve. Because the data points align closely with the reference diagonal, the probabilities are accurate. To maintain this across varying comment lengths, a fitted temperature parameter is used to adjust confidence levels as the volume of data increases.

The Confusion Matrix (Synthetic Data)

To understand where the model fails, we look at the confusion matrix.

• Human Input: Correctly identified 73% of the time.
• Robot Input: Correctly identified 80% of the time.
• Aggregate Error: The mistake rate hovers around 25%.

However, the calibrated percentage is more useful for individual cases. For example:

• High Confidence ($\ge 80\%$): The risk of a false positive plummets to $\approx 5\%$.
• Low Confidence ($\approx 50\%$): The verdict is essentially a coin flip.

Academic Summary

For those interested in standard statistical markers, here are the results based on the "ignorance assumption":

• Accuracy: 77%
• Precision: 75%
• Recall: 80%
• Sensitivity: 80%
• Specificity: 73%
• F1 Score: 77%

---

Real-World Generalization

I tested the model against a smaller, non-synthetic dataset of actual human and AI comments to see how it performs "out of sample."

Resulting Metrics:

• Accuracy: 88% | Precision: 89% | Recall: 86% | F1: 87%

Conclusion: The classifier actually performs better on real-world data than on the synthetic training set. Note that this tool is specialized for code comments and is not intended for general text detection.

---

Data Collection Process

The goal was to build a robust dataset. If planned perfectly, the cost would be roughly \30$. While more data can increase power, the returns diminish quickly because the classifier's strength scales logarithmically:

$\text{Power} \propto \log(\text{Investment})$

My actual path was less linear:

1. First attempt: Data was junk. 💸
2. Second attempt: Still junk. 💸💸
3. Third attempt: Success! 💸💸💸

The Workflow

To prevent subject matter leakage (where the AI learns to recognize the file rather than the style), I ensured the number of tokens remained balanced across all classes.

Lessons Learned (Mistakes Made)

• File Mismatch: I accidentally used different source files for different LLMs.
• Sparse Data: I generated AI comments for files that had very few human comments, causing leakage across the human-robot divide.
• Docstring Neglect: I forgot to remove docstrings when blinding the LLMs. This led the AI to mimic the existing repository style, making them look more "human."
• Syntax Overlook: I failed to account for all comment syntaxes in certain languages, leaving "contaminant" comments behind and missing potential data.

Observation: Most human comments are utilitarian and brief, often consisting of simple labels like main task structure or chIcon.

# Example of a typical human comment:
# Alias and including
def handle_request():
    pass