- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces CYBERBINGYM, a binary-only benchmark for analyzing and exploiting 1-day vulnerabilities. The authors evaluate their benchmark on three AI agents and show that, given a vulnerable and patched binary, AI agents can recover up to 88% of the vulnerabilities and successfully exploit them 22% of the time.
Critical strengths
The paper presents a useful benchmark for binary vulnerability analysis. CYBERBINGYM provides 100 tasks across 53 projects, with task selection accounting for the relative frequency of vulnerability families. By withholding source code, vulnerability descriptions, and reference PoCs, it evaluates capabilities that source-based benchmarks do not directly capture.
Practical relevance to the patch-to-exploitation window:The paper highlights how agents may accelerate vulnerability analysis once a patch becomes available. Its binary-only reproduction and exploitation results raise an important concern: attackers may require less time to turn published patches into actionable attacks, increasing the urgency of timely patch deployment.
Critical weaknesses
Insufficient evidence that the recovery metric measures the intended vulnerability: If the metric only checks exit behavior instead of vulnerability identity, an input could receive credit for triggering another bug or assertion that occurs only in the vulnerable version. This uncertainty undermines confidence in the headline the 88% recovery rate and the conclusions drawn from it.
Questions for authors
-
Section 3.1 says accepting a PoC requires an "expected sanitizer-detected failure,” whereas Appendix A.9.2 says the oracle checks exit behavior rather than vulnerability identity. Which criteria determine the reported recovery scores, and how do they distinguish the assigned vulnerability from another bug or assertion?
-
Have you independently checked accepted PoCs to confirm they trigger the assigned vulnerability? If so, how was this verified, and what recovery rate remains after excluding unrelated failures?
2 comments
Dear Reviewer Pe9V, thank you again for your thoughtful feedback and suggestions. We would like to kindly follow up on our rebuttal and ask whether our responses have adequately addressed your concerns. We would greatly appreciate any further feedback or clarification you may have, and we would be happy to address any remaining questions.
We thank the reviewer for recognizing the benchmark's value and for the precise questions, which pinpoint a wording inconsistency we are glad to fix.
W1 / Q1 / Q2 (combined): "Section 3.1 says a PoC needs an 'expected sanitizer-detected failure', but Appendix A.9.2 says the oracle checks exit behavior rather than vulnerability identity. Which governs, how do you distinguish the assigned vulnerability from another bug/assertion, and what recovery rate remains after excluding unrelated failures?"
We understand W1, Q1, and Q2 as the same concern and answer them together; please let us know if we have misread any of them.
Which description governs. A.9.2 is the accurate one: the oracle only checks exit behavior. A PoC counts iff the same input makes the vulnerable build fail (crash or non-zero exit) and the patched build exit normally, a differential crash check. It does not compare the crash against the reference vulnerability's type or location. Every recovery number in the paper uses exactly this check, so the results already match A.9.2; Section 3.1's phrase "expected sanitizer-detected failure" overstates it, and we will rewrite Section 3.1 to agree.
Why we leave vulnerability identity out of the oracle. This is deliberate, and it follows from the 1-day threat model. A released version is not a minimal single-CVE fix: one update usually bundles fixes for several distinct vulnerabilities, and the gap between the vulnerable and patched builds covers the whole release (median ~1,000 changed lines; sample 368 alone spans 86 C files). An attacker diffing the old and new releases benefits from triggering any defect the update closed, and usually can't even tell which of the co-shipped fixes a given crash belongs to, not just the one on record. So any input the patch removes is a real, actionable defect the release closed, which is the right notion of success here; demanding an exact-CVE match would undercount genuine 1-day capability. This differential check is established practice: CyberGym (Wang et al., ICLR 2026) validates PoCs by differential execution on vulnerable and patched builds, and SEC-bench (Lee et al., NeurIPS 2025) verifies its reproduced PoCs the same way (a sanitizer crash before the patch, none after). Our recovery oracle applies the same criterion.
We still checked identity directly, as validation rather than re-scoring. We checked on every accepted PoC, comparing the crash it actually produces on the vulnerable build against the task's OSS-Fuzz/ARVO ground truth. Most reproduce the specific recorded vulnerability; a minority trigger a real memory-safety defect that is not the intended bug but is genuinely present in the vulnerable build and removed by the patch. We keep those as successes, for the 1-day reason above.
Recovery rates stay the same (88/100 Codex, 85/100 Claude, 64/100 Kimi), and under the stricter identity rule based on sanitizer report, 66, 59, and 50 of those respectively are the specifically-assigned vulnerability. The checker ships as appendix validation, showing the differential successes are overwhelmingly genuine memory-safety defects rather than spurious crashes; it does not change the scored metric.
References