- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AnonVul, a benchmark for evaluating AI agents on real-world business-logic vulnerability discovery. The benchmark contains 132 vulnerabilities across nine programming languages and 36 CWE categories. The authors develop a semi-automated pipeline to construct executable environments, remove software identity cues, and build hidden runtime oracles. Each task uses a reference proof of concept (PoC) and a patched version to validate the target vulnerability and its oracle.
The paper evaluates six agent systems under black-box and grey-box settings. The best system achieves a discovery rate of 23.11%, while the overall discovery rate is 14.58%. On a selected set of 20 vulnerabilities, restoring software identity increases the discovery rate from 21.25% to 40.83%. The paper also studies the effect of larger exploration budgets, oracle reliability, and common agent failure modes.
Critical strengths
- A valuable and realistic benchmark for business-logic vulnerability discovery.
The paper addresses an important limitation of existing cybersecurity benchmarks. Business-logic vulnerabilities depend on application-specific permissions, states, and workflows, and are difficult to evaluate using simple tests.
AnonVul provides 132 executable vulnerabilities across different applications and programming languages. Its black-box and grey-box settings support useful comparisons between agent systems. The released artifacts also support reproducibility. This benchmark is a meaningful contribution because it provides a practical evaluation setting for a challenging security problem.
- A carefully designed validation pipeline with application-specific runtime oracles.
The paper uses runtime evidence to evaluate security violations rather than relying on agent-reported success. The oracles check relevant application states, responses, or side effects. The authors also validate the target behavior using reference PoCs and matched patched versions.
This design is important because a successful HTTP response or an agent's own claim does not necessarily establish a business-logic vulnerability. The validation pipeline makes the benchmark scores more meaningful and supports the credibility of the evaluation.
Critical weaknesses
- The identity experiment does not establish the difference between discovery and memorization.
A central motivation of the paper is to distinguish genuine vulnerability discovery from memorization of known vulnerabilities. However, the experimental design does not fully support this distinction.
The anonymization process changes multiple aspects of the application, including product names, source identifiers, routes, and runtime information (Section 3 and Figure 4). These changes can affect how easily an agent understands and explores the application. Even when the reference PoC still works, the search difficulty may change. Moreover, recognizing a software product is not equivalent to memorizing its vulnerabilities. An agent may use legitimate knowledge of an application's API, features, or architecture to discover a vulnerability without recalling the vulnerability itself.
- The experimental design provides limited evidence about result stability and the effect of exploration budgets.
The main evaluation uses only one attempt per system, vulnerability, and access setting. Agent exploration is stochastic, so a single attempt may not reliably represent the success probability of a system.
The identity experiment also uses only 20 vulnerabilities selected to cover different success and failure cases. The paper does not report uncertainty estimates or systematic repeated trials for the main results. Appendix E.2 shows that some outcomes change across attempts, although its selected reruns do not provide a representative estimate of variance. This makes it difficult to assess the stability of the leaderboard and the generality of the identity effect. The scaling experiment introduces additional systems while also increasing the budget. Furthermore, five of the eight successful scaling attempts finish within the original time and cost limits (Section 5, RQ2). Thus, these successes do not necessarily result from the larger budget. Provider refusals further limit the comparison.
Questions for authors
-
How do the authors separate vulnerability memorization from legitimate software knowledge and changes in interface difficulty?
-
The oracle validation relies on reference PoCs, patched counterparts, and trajectory inspection. How do the authors establish that the oracles detect other valid exploitation strategies that differ from the reference PoCs?
Response to Reviewer xMp9
We thank the reviewer for recognizing AnonVul as a valuable and realistic benchmark and for highlighting its carefully designed validation pipeline.
W1/Q1 (Memorization vs. legitimate knowledge, and change in interface difficulty). Thank you for this question. We separate memorization from legitimate software knowledge by auditing trajectories, and we find no evidence that anonymization changes interface difficulty. We address each part below.
(1) As defined in RQ3, we distinguish vulnerability-specific knowledge from application-specific knowledge. In our trajectory audit, we count a cell as vulnerability memorization only if the trajectory names the correct CVE or GHSA identifier. This is a lower-bound estimate of memorization, as models may use memorization without explicitly mentioning the identifiers in the trajectories.
(2) Identifiers are renamed consistently across all agent-visible surfaces, to neutral names that keep the function of the original (e.g.,
/users/{userId}/tokens/{name}→.../gen-token/{name}in Section 2), which yields an equivalent program (α-renaming; Pierce, 2002, Ch. 5). Beyond the PoC replay, we checked search difficulty empirically: in 100 randomly sampled Anonymized trajectories, 87 never referred to the renaming and the other 13 noticed a generic-looking name but continued normally, so no step failed or was redirected because of a renamed identifier. Moreover, under Greybox the agent can read every renamed identifier in the source code, yet the identity effect persists (+21.4 pp, 95% CI [12.2, 31.9], over three runs).We will make these definitions explicit in RQ3 and add the trajectory audit to Appendix B.
Reference: B. C. Pierce. Types and Programming Languages. MIT Press, 2002.
Q2 (Oracle coverage of alternative strategies). We clarify that each oracle is derived from the specification and checks the security effect itself (a state change, protected content in a response, or a side effect), not the request sequence that produces it, so any strategy that causes the violation is observed at the same point (Section 3). In the Graylog example (Section 2), the oracle triggers when the authenticated caller differs from the token's owner and the application returns a real token, regardless of the attack route that is taken. This is also consistent with our answer to RQ4: we inspected the execution traces for target security effects that the oracle might have missed and found no false negatives.
W2 (Result stability and budget scaling). Thank you for your suggestions, and we address the concern with new experiments.
(1) To assess stability, we reran the identity experiment with all six systems, both conditions, and both access settings, giving three runs per cell. The pooled Anonymized rate (the Main setting) is 21.3%, 22.5%, and 17.5% across runs. Overall, 75.8% of the 480 cells give the same outcome in all three runs, and the identity effect is +19.6, +17.1, and +22.9 pp across runs, with a bootstrap 95% CI of [12.5, 27.8] that accounts for both shared vulnerabilities and runs. Details are in our responses to Reviewer hE4w (W2) and Reviewer bHh7 (Q2).
(2) To assess the generality of the identity effect, we extended the Visible experiment to an additional 20 randomly selected vulnerabilities. Both RQ3 findings hold in the same direction. Pooling both cohorts gives +13.5 pp. Details are in our response to Reviewer hE4w (W1/Q1).
(3) To separate the effect of budget from that of system, we reran all eight Scaling systems on the same ten tasks at the original budget: 3/160 and 3/160 succeed, vs. 8/160 under the larger budget. We will narrow the conclusion in RQ2. Details are in our response to Reviewer bHh7 (Q3).