acceptodds
Under review as a conference paper at ICLR 2027

Discovery or Memorization? A Faithful Benchmark for Real-World Business-Logic Vulnerability Discovery

10 views

Abstract

The rapid advancement of cybersecurity agents in vulnerability discovery has increased the need for benchmarks that evaluate their capabilities accurately. However, synthetic benchmarks lack realism, real-life benchmarks are prone to contamination, and many existing benchmarks have already been saturated by current models. We therefore set out to develop a benchmark that is realistic, reproducible, resistant to contamination, and challenging. Our key insight is that real-life business-logic vulnerabilities are subtle and exercise complex application interactions, making them naturally demanding. The difficulty lies in specifying evaluation criteria precise enough to faithfully measure model capabilities while keeping the tasks free of data-leakage. We introduce ANONVUL, a cross-language benchmark of 132 application-level business-logic vulnerabilities, built by a semi-automated pipeline that anonymizes each task, injects oracles from expert-written specifications, and differentially verifies proof-of-concept exploits. The benchmark spans nine languages, with two evaluation tracks on black-box and grey-box settings, where each task runs in a sandbox with specialized oracles embedded in key exploitation steps. We evaluate six state-of-the-art vulnerability-discovery agents on the ANONVUL benchmark, measuring discovery success, cost-effectiveness, anonymization sensitivity, oracle validity, and failure modes. Our results show that AnonVul remains challenging, with the best agent succeeding in only 23.11% of attempts. On a matched cohort of 20 vulnerabilities, restoring software identity increases discovery from 21.25% to 40.83%, highlighting the importance of controlling recognizable software knowledge when evaluating vulnerability discovery.

Primary area:
datasets and benchmarks
Keywords:
benchmark, cybersecurity agents, vulnerability discovery, business-logic vulnerabilities, semantics-preserving anonymization, contamination, agent evaluation, semantic oracles

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xMp9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AnonVul, a benchmark for evaluating AI agents on real-world business-logic vulnerability discovery. The benchmark contains 132 vulnerabilities across nine programming languages and 36 CWE categories. The authors develop a semi-automated pipeline to construct executable environments, remove software identity cues, and build hidden runtime oracles. Each task uses a reference proof of concept (PoC) and a patched version to validate the target vulnerability and its oracle.

The paper evaluates six agent systems under black-box and grey-box settings. The best system achieves a discovery rate of 23.11%, while the overall discovery rate is 14.58%. On a selected set of 20 vulnerabilities, restoring software identity increases the discovery rate from 21.25% to 40.83%. The paper also studies the effect of larger exploration budgets, oracle reliability, and common agent failure modes.

Critical strengths
  1. A valuable and realistic benchmark for business-logic vulnerability discovery.

The paper addresses an important limitation of existing cybersecurity benchmarks. Business-logic vulnerabilities depend on application-specific permissions, states, and workflows, and are difficult to evaluate using simple tests.

AnonVul provides 132 executable vulnerabilities across different applications and programming languages. Its black-box and grey-box settings support useful comparisons between agent systems. The released artifacts also support reproducibility. This benchmark is a meaningful contribution because it provides a practical evaluation setting for a challenging security problem.

  1. A carefully designed validation pipeline with application-specific runtime oracles.

The paper uses runtime evidence to evaluate security violations rather than relying on agent-reported success. The oracles check relevant application states, responses, or side effects. The authors also validate the target behavior using reference PoCs and matched patched versions.

This design is important because a successful HTTP response or an agent's own claim does not necessarily establish a business-logic vulnerability. The validation pipeline makes the benchmark scores more meaningful and supports the credibility of the evaluation.

Critical weaknesses
  1. The identity experiment does not establish the difference between discovery and memorization.

A central motivation of the paper is to distinguish genuine vulnerability discovery from memorization of known vulnerabilities. However, the experimental design does not fully support this distinction.

The anonymization process changes multiple aspects of the application, including product names, source identifiers, routes, and runtime information (Section 3 and Figure 4). These changes can affect how easily an agent understands and explores the application. Even when the reference PoC still works, the search difficulty may change. Moreover, recognizing a software product is not equivalent to memorizing its vulnerabilities. An agent may use legitimate knowledge of an application's API, features, or architecture to discover a vulnerability without recalling the vulnerability itself.

  1. The experimental design provides limited evidence about result stability and the effect of exploration budgets.

The main evaluation uses only one attempt per system, vulnerability, and access setting. Agent exploration is stochastic, so a single attempt may not reliably represent the success probability of a system.

The identity experiment also uses only 20 vulnerabilities selected to cover different success and failure cases. The paper does not report uncertainty estimates or systematic repeated trials for the main results. Appendix E.2 shows that some outcomes change across attempts, although its selected reruns do not provide a representative estimate of variance. This makes it difficult to assess the stability of the leaderboard and the generality of the identity effect. The scaling experiment introduces additional systems while also increasing the budget. Furthermore, five of the eight successful scaling attempts finish within the original time and cost limits (Section 5, RQ2). Thus, these successes do not necessarily result from the larger budget. Provider refusals further limit the comparison.

Questions for authors
  1. How do the authors separate vulnerability memorization from legitimate software knowledge and changes in interface difficulty?

  2. The oracle validation relies on reference PoCs, patched counterparts, and trajectory inspection. How do the authors establish that the oracles detect other valid exploitation strategies that differ from the reference PoCs?

1 comment
AuthorsOpenReview ↗

Response to Reviewer xMp9

We thank the reviewer for recognizing AnonVul as a valuable and realistic benchmark and for highlighting its carefully designed validation pipeline.

W1/Q1 (Memorization vs. legitimate knowledge, and change in interface difficulty). Thank you for this question. We separate memorization from legitimate software knowledge by auditing trajectories, and we find no evidence that anonymization changes interface difficulty. We address each part below.

(1) As defined in RQ3, we distinguish vulnerability-specific knowledge from application-specific knowledge. In our trajectory audit, we count a cell as vulnerability memorization only if the trajectory names the correct CVE or GHSA identifier. This is a lower-bound estimate of memorization, as models may use memorization without explicitly mentioning the identifiers in the trajectories.

(2) Identifiers are renamed consistently across all agent-visible surfaces, to neutral names that keep the function of the original (e.g., /users/{userId}/tokens/{name} → .../gen-token/{name} in Section 2), which yields an equivalent program (α-renaming; Pierce, 2002, Ch. 5). Beyond the PoC replay, we checked search difficulty empirically: in 100 randomly sampled Anonymized trajectories, 87 never referred to the renaming and the other 13 noticed a generic-looking name but continued normally, so no step failed or was redirected because of a renamed identifier. Moreover, under Greybox the agent can read every renamed identifier in the source code, yet the identity effect persists (+21.4 pp, 95% CI [12.2, 31.9], over three runs).

We will make these definitions explicit in RQ3 and add the trajectory audit to Appendix B.

Reference: B. C. Pierce. Types and Programming Languages. MIT Press, 2002.

Q2 (Oracle coverage of alternative strategies). We clarify that each oracle is derived from the specification and checks the security effect itself (a state change, protected content in a response, or a side effect), not the request sequence that produces it, so any strategy that causes the violation is observed at the same point (Section 3). In the Graylog example (Section 2), the oracle triggers when the authenticated caller differs from the token's owner and the application returns a real token, regardless of the attack route that is taken. This is also consistent with our answer to RQ4: we inspected the execution traces for target security effects that the oracle might have missed and found no false negatives.

W2 (Result stability and budget scaling). Thank you for your suggestions, and we address the concern with new experiments.

(1) To assess stability, we reran the identity experiment with all six systems, both conditions, and both access settings, giving three runs per cell. The pooled Anonymized rate (the Main setting) is 21.3%, 22.5%, and 17.5% across runs. Overall, 75.8% of the 480 cells give the same outcome in all three runs, and the identity effect is +19.6, +17.1, and +22.9 pp across runs, with a bootstrap 95% CI of [12.5, 27.8] that accounts for both shared vulnerabilities and runs. Details are in our responses to Reviewer hE4w (W2) and Reviewer bHh7 (Q2).

(2) To assess the generality of the identity effect, we extended the Visible experiment to an additional 20 randomly selected vulnerabilities. Both RQ3 findings hold in the same direction. Pooling both cohorts gives +13.5 pp. Details are in our response to Reviewer hE4w (W1/Q1).

(3) To separate the effect of budget from that of system, we reran all eight Scaling systems on the same ten tasks at the original budget: 3/160 and 3/160 succeed, vs. 8/160 under the larger budget. We will narrow the conclusion in RQ2. Details are in our response to Reviewer bHh7 (Q3).

Reviewer_bHh7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AnonVul, a benchmark of 132 real-world business-logic vulnerabilities spanning nine programming languages. Its construction pipeline combines vulnerability specifications, executable environments, software-identity anonymization, and hidden runtime oracles that verify application-specific security violations. Reference exploits are validated against vulnerable and patched counterparts. The benchmark provides matched blackbox and greybox evaluation settings.

The authors evaluate six agent systems under fixed time and monetary budgets. The strongest system achieves a 23.11% success rate, and the combined successes cover 63 of the 132 vulnerabilities. On a selected 20-vulnerability cohort, restoring software identity increases pooled success from 21.25% to 40.83%. Additional experiments investigate larger budgets, oracle reliability, and agent failure modes.

Critical strengths
  1. A useful benchmark for an important evaluation gap. Business-logic vulnerabilities require reasoning about application-specific relationships among principals, permissions, objects, and workflow states. AnonVul turns 132 such vulnerabilities into executable tasks with matched blackbox and greybox access. This is a meaningful contribution beyond code-level vulnerability classification or exploit generation from a supplied vulnerability description. The benchmark remains challenging for the evaluated systems and could support useful research on agents that combine semantic reasoning with runtime exploration.

  2. Careful verification of security effects. The benchmark checks the relevant unauthorized effect rather than relying on HTTP status codes, agent success declarations, or an LLM judge alone. The construction process includes reference-PoC replay, vulnerable-versus-patched validation, and legitimate and other negative controls where applicable. These choices are particularly consequential for business-logic vulnerabilities, where an apparently successful operation is not necessarily a security violation. This verification methodology is the strongest technical aspect of the paper.

Critical weaknesses
  1. A useful benchmark for an important evaluation gap. Business-logic vulnerabilities require reasoning about application-specific relationships among principals, permissions, objects, and workflow states. AnonVul turns 132 such vulnerabilities into executable tasks with matched blackbox and greybox access. This is a meaningful contribution beyond code-level vulnerability classification or exploit generation from a supplied vulnerability description. The benchmark remains challenging for the evaluated systems and could support useful research on agents that combine semantic reasoning with runtime exploration.

  2. Careful verification of security effects. The benchmark checks the relevant unauthorized effect rather than relying on HTTP status codes, agent success declarations, or an LLM judge alone. The construction process includes reference-PoC replay, vulnerable-versus-patched validation, and legitimate and other negative controls where applicable. These choices are particularly consequential for business-logic vulnerabilities, where an apparently successful operation is not necessarily a security violation. This verification methodology is the strongest technical aspect of the paper.

Questions for authors
  1. Clarification of the identity intervention: Precisely which artifacts differ between Anonymized and Visible? In particular, are routes, source identifiers, documentation, and tool interfaces otherwise identical? Among successful Visible runs, how often is there evidence of recalling the specific target vulnerability, as opposed to general product knowledge or unrelated historical vulnerabilities?

  2. Validation of the identity effect: Can you provide uncertainty estimates that account for shared vulnerabilities across systems and access settings, together with repeated paired attempts on a representative subset of the existing cohort? This would help establish how stable the reported identity effect is across tasks and stochastic runs.

  3. Interpretation of budget scaling: For the systems shared between Main and Scaling, is there a matched repeat at the original budget on the same ten tasks? If unavailable, would you narrow the conclusion to the observed low success under the larger budget, rather than infer a general marginal benefit from additional exploration?

3 comments
AuthorsOpenReview ↗

Response to Reviewer bHh7

We thank the reviewer for recognizing AnonVul as a useful benchmark for an important evaluation gap and for highlighting our verification of security effects.

Q1 (Identity intervention). (1) Artifacts differ only when they carry identity cues. Routes and source identifiers follow the same rule: those that reveal the product are renamed under Anonymized, using neutral names that keep the function of the original (Section 3), and restored under Visible. Neither arm provides product documentation, and tool interfaces are identical. We will state this scope explicitly in RQ3.

(2) We assign each of the 98 successful Visible cells to one category: 25 (25.5%) recall the target vulnerability, 13 (13.3%) use only general product knowledge without recalling which component is vulnerable (e.g., recalling Copyparty’s search API, Appendix D.3), and 3 (3.1%) recall only unrelated historical vulnerabilities, which incidentally point the agent to a component involved in the target (e.g., a route that the target vulnerability also uses). The remaining 57 (58.2%) do not explicitly rely on prior knowledge to reach the target. We will report this breakdown in RQ3.

Q2 (Validation of the identity effect). Thank you for this suggestion. We ran repeated paired attempts and computed uncertainty estimates that account for both shared vulnerabilities and run-to-run variation.

(1) We reran all six systems on the full 20-vulnerability cohort under both identity conditions and both access settings, giving three runs per cell (r0 is the paper's run). Each entry shows successes out of 40 (20 vulnerabilities × 2 access settings), Anonymized → Visible, with the paired difference. The pooled identity effect stays positive and stable across runs:

SystemFrameworkr0r1 (NEW)r2 (NEW)
Opus 4.8Claude Code9 → 24 (+15)12 → 24 (+12)12 → 24 (+12)
Sonnet 4.6Claude Code7 → 21 (+14)9 → 21 (+12)8 → 23 (+15)
GPT-5.4Codex16 → 23 (+7)18 → 20 (+2)13 → 18 (+5)
GLM 5.2mini-swe-agent11 → 14 (+3)8 → 15 (+7)3 → 18 (+15)
Kimi K2.6mini-swe-agent3 → 11 (+8)5 → 12 (+7)3 → 11 (+8)
MiniMax M2.5mini-swe-agent5 → 5 (0)2 → 3 (+1)3 → 3 (0)
Pooled (/240)–21.3% → 40.8% (+19.6 pp)22.5% → 39.6% (+17.1 pp)17.5% → 40.4% (+22.9 pp)

(2) Regarding uncertainty estimates, we use a two-level bootstrap (20,000 resamples): we first resample vulnerabilities, keeping each vulnerability's cells across systems and access settings together, and then resample runs within each cell. The resulting intervals therefore reflect variation across tasks and across stochastic runs (rates are pooled over the three runs):

SettingAnonymizedVisibleIdentity effect95% CI
Blackbox15.0%33.3%+18.3 pp[9.4, 27.8]
Greybox25.8%47.2%+21.4 pp[12.2, 31.9]
All20.4%40.3%+19.9 pp[12.5, 27.8]

All intervals exclude zero. At the vulnerability level, Visible is higher on 18 of 20 vulnerabilities and lower on none. We will add these results to Appendix D.

Q3 (Budget scaling). Thank you for raising this. We extended the Scaling experiment to address your concern: we reran all eight Scaling systems on the same ten tasks at the original budget, giving a matched baseline for all systems. Overall, the larger budget yields a small gain and success remains low in every condition. Following your suggestion, we will narrow the conclusion in RQ2 to the observed low success under the larger budget.

SystemFrameworkSettingMain (r0)r1 (NEW)r2 (NEW)Scaling (60 min / $50)
GPT-6 AstraCodexBlackboxn/ablockedblockedblocked
GPT-6 AstraCodexGreyboxn/ablockedblockedblocked
GPT-5.6 SolCodexBlackboxn/a111
GPT-5.6 SolCodexGreyboxn/a213
Opus 4.8Claude CodeBlackbox0003
Opus 4.8Claude CodeGreybox0000
GPT-5.4CodexBlackbox0001
GPT-5.4CodexGreybox0000
Sonnet 4.6Claude CodeBlackbox0000
Sonnet 4.6Claude CodeGreybox0010
GLM 5.2mini-swe-agentBlackbox0000
GLM 5.2mini-swe-agentGreybox0000
Kimi K2.6mini-swe-agentBlackbox0000
Kimi K2.6mini-swe-agentGreybox0000
MiniMax M2.5mini-swe-agentBlackbox0000
MiniMax M2.5mini-swe-agentGreybox0000
Passed (/160)––0/120*3/1603/1608/160

*Main includes only the six original systems.

AuthorsOpenReview ↗

Revised paper uploaded

Dear Reviewer bHh7,

Thank you again for your detailed review. We have uploaded a revised paper (changes in blue; deletions in red strikethrough) reflecting our responses:

  • Identity intervention (Q1): RQ3 now states the scope of the Visible condition and reports the breakdown of the 98 successful Visible cells.

  • Validation of the identity effect (Q2): We added three repeated runs and two-level bootstrap intervals, all of which exclude zero (Section 7, Table 4; Appendix D.3).

  • Budget scaling (Q3): RQ2 now includes the matched baseline at the original budget and a narrowed conclusion (Section 5; Appendix D, Table 5).

We hope the revised version addresses your concerns. Thank you.

Best regards,

The Authors

Thank you for the detailed responses and revisions. I believe my three questions have been adequately addressed:

Q1 (Identity Intervention): The clarification of the experimental differences and the breakdown of prior knowledge usage address my concerns.

Q2 (Identity Effect): The repeated experiments and bootstrap confidence intervals provide sufficient evidence for the stability of the reported effect.

Q3 (Budget Scaling): The additional matched baseline experiments and the more cautious interpretation adequately address my concerns.

Overall, I am satisfied with the authors' responses and appreciate the additional experiments and revisions.

Reviewer_hE4wOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark to evaluate agents' vulnerability discovery capabilities. The benchmark contains 132 application-level business-logic vulnerabilities. The results show that the benchmark is challenging for agents, with the best agent succeeding in only 23.11% of attempts.

Critical strengths
  • The benchmark addresses an important question: whether agents can genuinely discover business-logic vulnerabilities or merely rely on memorization.

  • The benchmark construction appears well-designed, and the experimental evaluation is relatively comprehensive, yielding several interesting findings.

Critical weaknesses
  • The benchmark is relatively small, comprising only 20 vulnerabilities, which limits the breadth and generalizability of the insights it can provide.

  • The agents are evaluated with only a single attempt per task, without repeated trials or statistical analysis to account for potential variability in performance.

  • The paper does not sufficiently justify its novelty compared to existing benchmarks, such as BaxBench. A clearer discussion of its unique contributions and distinctions from prior work would strengthen the paper.

Questions for authors
  1. How representative are the 20 selected vulnerabilities? Would the conclusions remain valid if the benchmark were expanded to include a larger and more diverse set of vulnerabilities?

  2. How robust are the findings across repeated trials? Could the authors conduct multiple runs per task and provide statistical analyses to assess the variability and reliability of the results?

  3. How does this benchmark differ from existing benchmarks for business-logic vulnerabilities, such as BaxBench? Could the authors clarify its novelty and unique contributions compared to prior work?

2 comments
AuthorsOpenReview ↗

Response to Reviewer hE4w

We thank the reviewer for recognizing that AnonVul addresses an important limitation of existing cybersecurity benchmarks and that its construction is rigorous.

W1/Q1 (Representativeness of the identity cohort). Thank you for the question. We ran further experiments on an additional 20 randomly selected vulnerabilities and find that our conclusion holds when we increase the subset from 20 to 40 vulnerabilities, strengthening generalizability. Below, we elaborate on the extended experimental results.

SystemFrameworkOriginal Anon.Original VisibleRandom Anon. (NEW)Random Visible (NEW)
Opus 4.8Claude Code9241315
Sonnet 4.6Claude Code721611
GPT-5.4Codex1623812
GLM 5.2mini-swe-agent111469
Kimi K2.6mini-swe-agent31135
MiniMax M2.5mini-swe-agent5513
Total (/240)–51 (21.25%)98 (40.83%)37 (15.42%)55 (22.92%)

Both RQ3 findings derived from the original cohort hold in the random cohort: (1) Restoring identity increases discovery. (2) Identity cuts both ways: 21 cells move from fail to pass and 3 from pass to fail. At the vulnerability level, Visible is higher on 9, lower on 1, and equal on 10, all of which are unsolved under both conditions. These 10 create a floor effect that lowers the gain: on the other 10 vulnerabilities, discovery rises from 30.8% to 45.8%. The original cohort was chosen for a balanced spread of Main successes and failures to avoid this effect (RQ3).

Pooling both cohorts (40 vulnerabilities, 480 paired cells), discovery rises from 18.3% to 31.9%. In the revision, we will report both cohorts and the pooled estimate.

Q2 (Anonymization and discovery difficulty). Standing on the principle that consistent renaming preserves program semantics (α-renaming; Pierce, 2002, Ch. 5), we argue that anonymization does not add difficulty beyond removing the identity cues. We support this from two aspects.

(1) By construction. Behavior is unchanged: consistently renaming identifiers to fresh names across all agent-visible surfaces yields an equivalent program. Meaning is also unchanged: replacements are neutral names that keep the function of the original (e.g., /users/{userId}/tokens/{name} → .../gen-token/{name} in Section 2), and roles, permissions, and workflows are kept identical to the original. We verify this per task: the reference PoC still succeeds on every anonymized instance and is blocked on its patched counterpart.

(2) Empirically. We observe no added difficulty. We manually reviewed 100 randomly sampled Anonymized trajectories for any step that failed or went astray because of a renamed identifier. In 87, the agent never referred to the renaming; in the other 13, it noticed a generic-looking name but continued normally. No step failed or was redirected because of anonymization.

Moreover, under Greybox the agent can read every renamed identifier in the source code and use it against the running application, so any obstacle created by renaming should disappear. Yet the identity effect persists in Greybox (+21.4 pp, 95% CI [12.2, 31.9], over three runs). We will add the formal renaming conditions and the empirical evidence to Appendix B.

Reference: B. C. Pierce. Types and Programming Languages. MIT Press, 2002.

W2 (Single attempt per task). We agree that single attempts introduce variance (Section 7), and we address it by repeating the identity experiment under both conditions. All reported findings remain consistent, and the pooled variance is moderate. Specifically, we reran all six systems on the original 20-vulnerability cohort under the same conditions as the paper, giving three runs per cell (r0 is the paper's run).

SystemFrameworkAnon. r0Anon. r1 (NEW)Anon. r2 (NEW)Visible r0Visible r1 (NEW)Visible r2 (NEW)
Opus 4.8Claude Code91212242424
Sonnet 4.6Claude Code798212123
GPT-5.4Codex161813232018
GLM 5.2mini-swe-agent1183141518
Kimi K2.6mini-swe-agent353111211
MiniMax M2.5mini-swe-agent523533
Pooled (/240)–21.3%22.5%17.5%40.8%39.6%40.4%

Per system and condition, the standard deviation ranges from 0 to 4.0 successes out of 40, and 75.8% of the 480 cells give the same outcome in all three runs. We will add the repeated-run results to Appendix D and revise the limitation in Section 7 accordingly.

AuthorsOpenReview ↗

Revised paper uploaded

Dear Reviewer hE4w,

Thank you again for your thoughtful review. We have uploaded a revised paper (changes in blue; deletions in red strikethrough) that incorporates our responses to your comments:

  • Representativeness (W1/Q1): RQ3 now includes 20 additional randomly selected vulnerabilities, and both findings hold. Pooled over 40 vulnerabilities, discovery rises from 18.3% to 31.9% (Section 5; Appendix D.3).

  • Anonymization and difficulty (Q2): Appendix B now describes the consistent renaming and our audit of 100 Anonymized trajectories.

  • Single attempt (W2): The identity experiment was repeated three times. 75.8% of cells give the same outcome, and the effect remains stable (95% CI [12.5, 27.8]). We have revised the limitation in Section 7 and added Table 4.

We hope these changes address your concerns. If you find that your concerns have been resolved, we would be grateful if you would consider updating your assessment. Thank you.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.