- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Agentic benchmarks can be exploited by capable agents, making the detection and prevention of shortcuts and leakage important for the agent auditing community. The paper introduces a taxonomy of benchmark shortcut exposure with five categories and proposes HackDetect, an evidence-grounded audit pipeline that identifies benchmark-side shortcuts, determines whether and how agents exploit them, and measures the resulting score changes after the shortcuts are repaired or removed. The authors apply HackDetect to 11 public benchmarks to quantify the extent of shortcut exploitation.
Critical strengths
- The paper tackles an important task, where detecting shortcut exploitation is important in agent audit.
- Overall, I find the paper clearly organized and easy to follow.
Critical weaknesses
-
The claimed contribution does not seem sound. The manuscript offers limited formalization beyond introducing terms such as “Exposure,” “Engagement,” and “Mislead.” I would characterize the contribution as an observational study and a taxonomy rather than a formalization, which would require rigorous definitions and clearly specified conditions of applicability.
-
The novelty of the audit pipeline is unclear. Identifying benchmark-side shortcuts, determining whether and how agents exploit them, and measuring score changes after repairing or removing them is consistent with standard auditing practices. It is unclear how HackDetect differs from existing open-source agent auditing pipelines. A comparison with prior open-sourced libraries and workflows would help clarify its methodological contribution.
-
The reliability of the trajectory assessments needs further validation. Applying the taxonomy in Figure 3 appears to require fine-grained judgments. Are all assessments performed only by GPT-5.5? How closely do these judgments agree with human-annotated ground truth (other than answer source leakage), and how sensitive are the findings to the choice of judge model?
Questions for authors
See above