- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Current benchmarks measuring how able LLM agents are at fixing are vulnerable to reward hacking through incorrectly saying a patch is "correct" even when it breaks the system it's meant to be patching. This paper proposes a new benchmark which fixes that issue through verifying that the patch works and the system still works after the patch is applied.
Critical strengths
- The examples of reward hacking like the one shown with Qwen3.5-4B on page 13 show why this benchmark is useful, as models can appear to be patching vulnerabilities on the surface when, in reality, they are just breaking the system.
Critical weaknesses
- The authors' work may not be as novel as they claim. A claim in the paper's abstract is that no existing benchmark measures vulnerability patching while also measuring that the server continues to work. However, one of the cited papers, Zhang et al. 2025a, does measure the ability of LLMs to patch vulnerabilities and then runs test suites on those patched repos to ensure functionality is intact, which is a very similar idea.
Questions for authors
For 'discovery-required,' what guidance is the agent given? And, how do you ensure they have identified the vulnerability you were targeting vs. another one? Are you assuming there is exactly one vulnerability in the project?
Thank you for the review and for the note on the reward-hacking example.
Novelty. BountyBench is our closest neighbour, and we agree it checks runtime health: it runs code and runtime invariants (server and database checks) before the exploit check. We credit this partially in Table 3 (Appendix). The distinction is the target and the fix. BountyBench edits source and re-instantiates the runtimes from the patched code. SysRepair-Bench grades the running system after operator-level changes (configuration, service, permission, end-of-life packages, vulnerabilities with no upstream patch), which a source patch cannot express. We will reword the abstract and introduction to "no existing benchmark measures operator-level remediation of a live system under an availability constraint", and name BountyBench explicitly as the nearest work.
Discovery-required. The agent receives only the running system and the availability requirement, with no disclosure text (§5).
Vulnerability. Each scenario is built around a single target vulnerability, and its container carries no other planted flaw. We grade only with that scenario's PoC, which replays the target vulnerability, and the security component passes only when it fails. We do not check whether the agent finds or fixes anything else. A scenario may bundle a few related settings that make up one finding, such as SSH X11 forwarding with no MaxAuthTries. We will state this grading unit explicitly in §5.