The Resume Lottery: Testing HackerRank's Open-Source ATS

By Dan Kinsky | June 28, 2026

Recently, a tool from HackerRank has been gaining massive traction across Reddit and LinkedIn: the hiring-agent open-source ATS. You can find the repository here: https://github.com/interviewstreet/hiring-agent.

I decided to put it to the test. After cleaning up some debug prints from my initial setup, I performed a stress test on my own resume.

🛠️ The Experimental Setup

To ensure a clean test, I followed these steps:

• Remove all troubleshooting debug prints.
• Set DEVELOPMENT_MODE = false.
• Wrap the execution in a loop to run 100 consecutive iterations.

---

⚙️ How the "Hiring Agent" Works

The system follows a specific pipeline to transform a PDF into a numerical score:

The Scoring Rubric

The total score is calculated based on the following weights:

The system defaults to the gemma3:4b model with a temperature of 0.1. This low temperature is intended to make the model deterministic consistent.

---

📉 The Findings: A Study in Inconsistency

When I analyzed the results, the variance was startling. While some categories were stable, others were essentially random.

1. The "Binary" Success

For Technical Skills, I scored $8/10$ in 98% of the runs. This is because skill matching is a simple checklist—something even a child could do. There is no "judgment" involved.

2. The "Judgment" Failure

The Projects category was a disaster. In one run, the LLM might claim my work "demonstrates real-world deployment," while in the next, it decides the same project "lacks architectural complexity."

Even dropping the temperature to 0 did not solve this. This isn't a tuning issue; it is a fundamental design flaw. A GitHub issue from October 2025 highlighted this, showing a single user receiving scores of 27, 34, 32, 34, 34, 30 in six straight runs at $T=0$.

3. Model Comparison

I tried switching to Gemini, which produced a tighter cluster of scores (between 48 and 64). However, the math still doesn't favor the candidate:

$\text{If Cutoff} = 60 \implies P(\text{Failure}) \approx 28\%$

Even with a "better" model, you could be rejected simply because the LLM had a "bad day."

---

🚩 The "Experience" Paradox

I tested an old resume containing only one internship. The prompt used by the tool is revealing:

Production (0-25 points)

• Analyze the 'work' and 'volunteer' sections for real-world, internship, or production experience
• SPECIAL CONSIDERATION: Give extra points for founder roles, co-founder positions, or early-stage engineer roles (first 10-20 employees) at startups

The result? A Principal Engineer with a decade of experience in distributed systems gets $25/25$. A student with two lines of text also gets a consistent score, but it's meaningless.

The biggest issue is the weighting. With 65% of the score tied to Open Source and Projects, the tool values a hobbyist with a few GitHub repos over a veteran engineer who built industry-standard infrastructure (like S3) but doesn't post their corporate work on GitHub.

---

⚠️ Final Verdict

LLMs are fantastic for parsing (turning a PDF into structured data). They are terrible at judging (deciding if a candidate is worth 18 points vs 24 points).

If you are in a position to influence your company's hiring process, be extremely cautious with AI screening. A tool that cannot consistently grade the same resume is not "filtering for quality"—it is simply filtering. Using this is equivalent to tossing a coin and telling the loser they had "bad luck."

---

Post-Script Corrections:

• Template Oddity: A reader noted that resume_evaluation_criteria.jinja mentions "Software Intern" on line 1, despite the tool claiming to be position-agnostic.
• Timeline: While marketed as "recently" open-sourced, commit history suggests it has been public since October 2025 and only recently went viral.