An engineer testing HackerRank’s open-source hiring agent reported large swings in the scores assigned to an unchanged resume, raising questions about using generative models as gatekeepers in recruitment. The account, published by Dan Unparsed, describes repeated local runs of the software and distinguishes between categories that behaved consistently and those requiring more subjective judgment.
The hiring agent parses a resume, obtains information from a candidate’s GitHub profile and asks a language model to assess several dimensions. In the author’s tests, the default Gemma 3 4B model was run 100 times at a temperature setting of 0.1. With an illustrative hiring cutoff of 85, the same resume fell below the line in 65% of runs, the author said.
Not every component varied equally. Technical-skills scores were almost fixed, landing at eight out of 10 in 98 runs. The author interpreted that stability as a consequence of checklist-like evaluation: a named skill is relatively straightforward to identify. Project scores, by contrast, changed substantially as the model alternated between different qualitative readings of the same work. Lowering temperature to zero did not eliminate the variation, according to the report.
A comparison using Gemini produced a narrower but still consequential range of 48 to 64. At a hypothetical cutoff of 60, 28% of identical applications would have failed. A later trial with Claude Opus 4.8 reportedly tightened the overall range only slightly, from 48–64 to 49–63, while the project category continued to fluctuate. These are tests by one writer on one resume, not a broad audit of production hiring systems, and the results should be read within that limit.
The critique also examined the evaluator’s weighting. Open-source work and projects together accounted for 65% of the score in the tested setup, potentially disadvantaging experienced engineers whose professional work is proprietary or absent from GitHub. The author also found that some rubric areas lacked clear anchors for distinguishing junior and senior experience.
The underlying repository appears to have been public since October 2025, despite social posts describing it as newly open-sourced. The central concern is therefore not the release date but whether a probabilistic judgment can safely determine access to a human review. The experiment supports a narrower conclusion: automated parsing and factual extraction may be useful, but unreviewed numerical rankings can introduce chance where employers expect consistency. The account does not say HackerRank deployed this exact scoring configuration to make employment decisions. It evaluates the public agent as configured by the writer, so production policies, model choices, human review and candidate safeguards may differ. Those details would be necessary before estimating real-world rejection rates.
