acceptodds
Under review as a conference paper at ICLR 2027

CyberCraft: Scaling Agentic Synthesis of Repository-Level Vulnerability Environments for Security Agent Training

Abstract

Training agents to discover and repair software vulnerabilities requires executable environments in which both a vulnerability and its remediation can be verified. However, real vulnerabilities are scarce. Synthesizing vulnerabilities is a scalable alternative, but existing methods either operate on isolated functions without an executable context or inject repository-level vulnerabilities with limited diversity. We introduce CyberCraft, an agentic framework that synthesizes diverse, verifiable repository-level vulnerability environments at scale. An injector agent, guided by an atlas of vulnerability mechanisms mined from real-world CVEs, introduces vulnerabilities into real repositories. The atlas encourages the agent to explore beyond common patterns while grounding injections in real-world mechanisms, improving diversity. The resulting environments support verified teacher demonstrations and student reinforcement learning with execution-based rewards, while the injector co-evolves with the student by targeting its failures in new environments. We construct 7,979 verified environments across 713 C/C++ projects, hold out part of them as CyberCraft-Bench, and evaluate transfer to SEC-bench, PatchEval, and CyberGym-E2E. Training on these environments substantially improves Qwen3.5-9B on the three benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2zKdOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
3
Presentation
2(−1)
Soundness
3
Read review
Paper summary

This paper uses an agentic pipeline to synthesize vulnerability tasks that do not rely on existing CVE instances, builds 7,979 verifiable environments from 713 projects, and shows the value of these data for the security domain through SFT and RL, while comparative experiments show the value of co-evolution relative to static datasets.

Critical strengths

Data construction is practically meaningful. It does not rely on existing CVE instances and yields 7,979 verifiable environments from 713 projects, each of which must pass the differential criterion, and it has practical potential for alleviating the diversity limits of tasks built from existing CVE examples.

Critical weaknesses

1.The comparison remains insufficient. Beyond comparisons with several strong models and a static ablation, the paper does not compare with students trained on other datasets of the same positioning or on real CVE environments, so the advantage of this data over other data is not established. 2.The empirical support for environment diversity is limited. The current evidence relies on the labels of the atlas itself, these labels are not manually verified, diversity is not examined from other angles, and teacher filtering may reduce diversity in difficulty.

Questions for authors

1.Under equal conditions, could the authors add comparisons with students trained on real CVE environments and on data from existing injectors? 2.Given the practical cost of tasks, could the authors report the injection acceptance rate and the token cost per environment?

8 comments
AuthorsOpenReview ↗

Part 1 of our response to Reviewer 2zKd

Dear Reviewer 2zKd,

We deeply appreciate your constructive critiques, which highlight important nuances that we should clarify in the revision. Below, we try our best to address your specific questions. To address each of them in sufficient detail, we split our response into two parts. This is Part 1 (W1 & Q1); Part 2 (W2 and Q2) follows in the next comment.

(We apologize for the late reply. The additional experiments below took longer than we expected to complete.)


W1 & Q1: additional comparisons with real CVE environments and existing injectors

Following it, we ran a new experiment that compares our environments with real CVE environments under equal conditions (part 1). With the same number of environments, a student trained on ours gains nearly as much on real vulnerabilities as one trained on real CVE environments. Few real environments remain usable for training, while our pipeline can produce many more.

1. Students trained on real CVE environments (new experiment)

We used ARVO (Mei et al., 2026), a dataset of real vulnerabilities found by OSS-Fuzz in C/C++ projects of the same kind as ours. Each instance provides a buildable vulnerable version, the fixed version, and an input that crashes the vulnerable one, but no agent trajectories. We processed it in the same way as our own data. Since ARVO has no description of the vulnerability, DeepSeek-V4-Flash writes one from the fix and the crash report, and our teacher, also DeepSeek-V4-Flash, then collects the trajectories. After removing the projects of SEC-bench and CyberGym-E2E (PatchEval has no C/C++ projects) and re-verifying each instance, 900 of the 5,001 instances remain, and the teacher produces three verified solutions for 731 of them.

We then trained Qwen3.5-9B with SFT on these 731 ARVO environments and, separately, on 731 environments sampled at random from the static set of the paper.

Table R4. Students trained under equal conditions (success rates, %).

Training dataEnvsCyberCraft-BenchSEC-benchPatchEvalCyberGym-E2EReal gain
None (Qwen3.5-9B)–14.812.015.211.4–
ARVO (real)73115.918.516.517.1+4.5
CyberCraft subset73117.617.518.714.2+3.9
CyberCraft static (paper)4,00021.125.527.418.5+10.9
CyberCraft static + ARVO4,73121.126.527.820.0+11.9

Although we removed the projects these benchmarks evaluate, ARVO is still much closer than our data to two of them. CyberGym-E2E is built from OSS-Fuzz vulnerabilities, partly taken from ARVO itself, and SEC-bench also consists of sanitizer-verified memory-safety bugs in C/C++ projects. Even so, with 731 environments each, our student gains nearly as much on real vulnerabilities as the ARVO student (+3.9 vs. +4.5). Real data, however, cannot grow in the same way. After decontamination ARVO yields only 731 training environments, while the static set alone holds 4,000 and gains 10.9 points. The two sources also add up, and adding ARVO to the static set raises the gain to 11.9 points.

2. Students trained on data from existing injectors (existing ablation in the paper)

Existing injectors cannot be used directly to train agents like ours. Function-level injectors such as VulGen, VGX, VulScribeR, and AVIATOR produce vulnerable functions that cannot be built or run. Repository-level work has released only small evaluation sets (22 and 212 samples, see related works).

The closest repository-level injectors (Migliaccio et al.; Wei et al.) simply ask an LLM to insert a bug or a named weakness. Their injections lack diversity, because the LLM falls back on a few familiar edits, such as removing a bounds check before a memory write. Our manuscript already tests this approach under equal conditions in the w/o-atlas ablation of Section 4.6, which removes only the atlas and keeps everything else the same. Its 4,000 environments cover far fewer weaknesses than ours (35 vs. 102 CWEs per 450 environments, see W2). With the same teacher, Qwen3.6-27B, its student is weaker on all four security benchmarks, and the average gain drops from 5.5 to 1.7 points (Table 2 of the paper).


The environments, benchmark, atlas, and reusable synthesis pipeline will be released to support community training and evaluation of security agents and the generation of new verified environments.

Thank you again for your thoughtful review. We hope our clarifications and new experiments results fully address your questions, and we would be deeply grateful if they could positively influence your final assessment of our work.

With sincere appreciation,

The Authors

AuthorsOpenReview ↗

Part 2 of our response to Reviewer 2zKd

Dear Reviewer 2zKd,

This is Part 2 of our response, covering W2 and Q2.


W2 — part 1/2: The empirical support for environment diversity

Our 7,979 environments cover 514 CWEs and 1,523 mechanisms. These labels come from the atlas card each environment was injected from, so we also labeled the environments again with a model that does not use the atlas.

Diversity check from other dimensions

We had DeepSeek v4.1 flash label the CWE of every environment from only its diff and its sanitizer or oracle report. DeepSeek v4.1 took no part in building the environments and labels every set in the same way, so the comparison no longer depends on the atlas. This check is new. It compares the static set of the paper with the w/o-atlas set of Section 4.6, which was built by the same injector on the same repositories without the atlas, and uses the real ARVO environments from W1 as a reference. Since the sets differ in size, we count CWEs in 450 random environments per set, averaged over 100 draws. The static set covers 102 CWEs, against 35 for the w/o-atlas set and 29 for ARVO. For ARVO this is more than the 11 CWEs of Section 3.2, because ARVO's own labels follow only the crash type. The five most common CWEs account for 31.6% of the static set, against 52.4% and 64.9%. Under labels that do not come from the atlas, the static set is therefore far more diverse than the w/o-atlas set and also more diverse than real vulnerabilities.


W2 — part 2/2: teacher filtering and difficulty

We agree that filtering removes some hard environments, but only a few. The teacher gets up to six attempts on each training environment, and an environment is replaced if fewer than three of them pass. For this rebuttal we counted the outcomes in the teacher logs of the paper's two training runs. 91% of the attempted environments were kept, including 81.5% of the hard ones, so hard environments drop only from 31.9% to 28.6% of the set.

Few are lost because the teacher is given the description of the vulnerability. Without it, the same teacher repairs only 34.1% of the hard CyberCraft-Bench environments (Table 1).

The kept environments remain hard for the student. In RL, 32% of the completed groups needed the reproduction or repair view, which an environment reaches only after the student fails it with the description. The evaluation sets are not filtered by the teacher, since CyberCraft-Bench and the feedback set never pass through the teacher.


Q2: acceptance rate and cost per environment

Across the injection runs behind the 7,979 environments, 29.1% of attempts pass Equation 1. An attempt is one injector session on one repository, limited to 3 hours (Appendix B.1). Most failed attempts find no witness input within this budget, and the rest fail to build, break regression tests, or also trigger on the clean snapshot.

Some accepted environments are later dropped as duplicates or replaced after teacher filtering, so each of the 7,979 environments takes about 3.9 attempts. Counting all of them, injecting one environment takes about 11M input tokens (92% cached) and 140k output tokens. CPU time for builds, tests, and witness search limits throughput more than tokens do.

The atlas lowers the acceptance rate from 38.6% in the w/o-atlas run of Section 4.6 to 30.4% in the static run, because without it the injector falls back on familiar edits that are easy to verify. These extra attempts pay for the diversity shown in W2 and the larger gains in Table 2 of the paper.

We will add the breakdown of attempts and costs to Appendix C and state the acceptance rate and cost per environment in Section 3.4.


The environments, benchmark, atlas, and reusable synthesis pipeline will be released to support community training and evaluation of security agents and the generation of new verified environments.

Thank you again for your thoughtful review. We hope our clarifications and new experiments results fully address your questions, and we would be deeply grateful if they could positively influence your final assessment of our work.

With sincere appreciation,

The Authors

AuthorsOpenReview ↗

Thank You and Follow-up

Dear Reviewer 2zKd,

Thank you again for your thoughtful comments and for taking the time to consider our rebuttal. We would sincerely appreciate any further questions or suggestions you may have, and would be happy to provide additional clarification if helpful.

If you feel that our responses have addressed your questions, we would be deeply grateful if you would consider reassessing our submission in light of these clarifications.

Thank you again for your time and consideration!

With sincere appreciation,

The Authors

Reviewer_MW72OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to synthesize vulnerabilities with repository-level environments for security agent training. To this end, it proposes CyberCraft, an agentic framework that produces diverse, and verifiable repository-level vulnerability environments at scale. More specifically, the injector agent introduces synthetic vulnerabilities to real-world code repositories based on an vulnerability atlas mined from real-world CVEs. Moreover, the injector agent can co-evolve with the model under training by targeting the its failures in new environments. The paper has constructed 7,979 verified environments across 713 real-world C or C++ projects. Part of the verified environments was also retained as a new synthetic benchmark named CyberCraft-Bench. The evaluation shows that training on the synthetic environment can improve Qwen3.5-9B on other benchmarks including SEC-bench, PatchEval, and CyberGym-E2E.

Critical strengths

Training agents to discover and repair software vulnerabilities is becoming increasingly important and critical. However, real-world vulnerability data can be scarce, preventing large-scale training of security agents. It is great to see this paper targeting such an important and timely problem.

The paper has synthesized a large number of verified environments for training security agents. Moreover, the experimental study covers both supervised fine-tuning and reinforcement learning of the Qwen3.5-9B model under different setups. Besides the new CyberCraft-Bench, the trained Qwen3.5-9B model has also been evaluated on a number of existing benchmarks.

The experimental results are also promising. The synthetic dataset helped substantially boost the performance Qwen3.5-9B on all the studied benchmarks. In addition, the paper also included ablation studies to isolate various design choices, including the co-evolution and vulnerability atlas.

Critical weaknesses

Overall, I feel the novelty is a bit limited. For example, there has been a large number studies on synthesizing artificial bug/vulnerability data and further post-training models based on such data. The current related work section only covers data synthesis and training for security agents, but there has also been quite similar work in the general coding agent domain.

It is great to see the paper includes multiple existing benchmarks including SEC-bench, PatchEval, CyberGym-E2E, and SWE-bench Verified. Meanwhile, the studied benchmarks are getting saturated given the recent model releases, and there have been a number of new and more challenging benchmarks. It would be great if the paper can further include those more recent benchmarks.

According to the paper, the data contamination issue can be mitigated by the synthetic dataset. While this may be true for some models, this may no longer be true for more recent models. Recent models have strong semantic understanding and reasoning capabilities, and may be able to find relevant vulnerabilities even if they have different syntactic representations.

Questions for authors
  • What is the key novelty of this work?

  • How do you ensure the validity of contract-oracle vulnerabilities?

3 comments
AuthorsOpenReview ↗

Part 1 of our response to Reviewer MW72

Dear Reviewer MW72,

We sincerely thank you for your detailed and constructive review. To address each of your questions in sufficient detail, we split our response into two parts. This is Part 1; Part 2 follows in the next comment.

(We apologize for the late reply. The additional experiments below took longer than we expected to complete. )


W1 & Q1: novelty and related work

Most prior vulnerability synthesis works at the function or file level, as discussed in our related work.

CVE-Factory (Luo et al., 2026, ICML) rebuilds repository-level environments from CVE reports, so its data grows only as fast as CVEs are disclosed.

Prior works on repository-level synthesis does create new vulnerable code, but it is small, has little diversity, and depends on manual work or a single analyzer. ZeroDayBench (Lau et al., 2026, ICLR) ports 22 disclosed CVEs by hand, and GraftyVul (Ram et al., 2026, arXiv) grafts the findings of one commercial static analyzer into five applications, giving 212 samples. Both are built for evaluation. CyberCraft is fully automated and produces 7,979 executable environments across 713 C/C++ projects. Its atlas, mined from over 26,000 CVE fixes, makess them diverse across 514 CWEs.

We agree that the related work should also cover general coding agents, and we will add them. However, data synthesis for general coding differs substantially from vulnerability synthesis, and its method are hard to transfer. SWE-smith (Yang et al., 2025, NeurIPS), BugPilot (Sonwane et al., 2026, ICLR), and SSR (Wei et al., 2026, ICML) inject bugs into real repositories and keep a bug if it makes existing tests fail.

A realistic vulnerability, by contrast, makes no test fail and shows up only on a crafted input. The agent also gets no failing test or issue, so it must localize the flaw and build a triggering input before checking a patch.

CyberCraft contributes a scalable synthesis framework and reusable resources for security agent research. We will release 7,979 verified environments, CyberCraft-Bench, the vulnerability atlas, and the synthesis pipeline, enabling the community to train and evaluate agents and generate additional environments. The key novelty of our framework lies in three ideas:

  1. A mechanism atlas that broadens diversity. Without guidance, LLM injectors fall back on a few familiar edits. We mine real CVE fixes into 1,751 mechanisms under 531 CWEs, each giving the invariant that correct code keeps and the edits that break it. With the same teacher, removing the atlas cuts the student's average gain from 5.5 to 1.7 points.
  2. Verification by triggering. An injection is accepted only if the project still builds and passes its tests, and an injector-found input causes a violation only in the injected version. Beyond sanitizers, we add contract oracles, runtime checks of the broken invariant that yield 16% more environments from the same attempts.
  3. Training that exploits the hidden ground truth of each environment. SFT keeps only teacher trajectories that pass the hidden checks, so even a weak 27B teacher helps. When all rollouts on an environment fail, RL shows it again with part of its hidden answer, roughly doubling what RL adds over SFT. Co-evolution targets the CWEs the student fails on, and its student beats the static one on all four security benchmarks with 700 fewer environments.

Together, these contributions provide a reusable approach to expanding the supply of executable vulnerability environments and demonstrate how the resulting environments can improve security agents through both supervised fine-tuning and reinforcement learning.


W2: newer and harder benchmarks

We chose SEC-bench, PatchEval, and CyberGym-E2E because most frontier models report on them, and CyberGym-E2E is recent (ICML 2026). On their leaderboards, top agents score 75% on SEC-bench patching, 84% on PatchEval-Verified, and 87% on CyberGym-E2E patch-only.

As the reviewer suggested, we did a new experiment to evaluate on ExploitBench (May 2026), a newer and harder benchmark, where even Claude Mythos 5 scores only 78% (Anthropic's system card). We follow its protocol (three seeds per bug) and run all models in Codex.

ModelScore (of 16)Reached (of 41)Triggered (of 41)
DeepSeek-V4-Flash (teacher)2.7 (16.9%)4018
Qwen3.5-9B1.0 (6.3%)273
+ SFT1.3 (8.1%)315
+ SFT + RL1.5 (9.4%)336

So our training also helps on a newer, harder benchmark, mainly in reaching and triggering bugs. We will add this evaluation to the paper.


We sincerely thank you for your constructive feedback.and hope that these clarifications and the additional experiment results address your questions. If you feel that our response and proposed revisions resolve the issues raised, we would be deeply grateful if you might consider re-evaluating our submission.

With sincere appreciation,

The Authors

AuthorsOpenReview ↗

Part 2 of our response to Reviewer MW72

Dear Reviewer MW72,

This is Part 2 of our response, covering W3 and Q2.


W3: contamination and semantic recognition

We propose the synthesis pipeline mainly to scale training data and broaden its diversity, since real vulnerabilities are scarce and concentrate on a few CWEs. Lower contamination is a secondary benefit of CyberCraft-Bench, and our transfer results do not rely on it. Each student is compared with its own base model, so pretraining exposure affects both equally, and our training data excludes the benchmark projects and the fixes of their vulnerabilities (Appendix E).

We agree that a strong model may recognize a mechanism it learned from real CVEs. We see this knowledge as part of the skill being measured, and it does not make our tasks easy. GPT-5.5 finds the vulnerable function in 94.2% of CyberCraft-Bench environments but repairs 76.5%, and only 55.6% of the hard ones (Table 1).

CyberCraft-Bench mitigates contamination at the instance level. An injected synthetic vulnerability has no public report or fix, so a model cannot recall where it is or how to trigger and patch it. Contamination at this level is a real problem for recent models. For example, OpenAI's GPT-6 Cyber system card (November 2026) says that the historical vulnerabilities in one saturated benchmark raise "a risk of contamination that may inflate the reported results." PatchBench (September 2026) also finds that 25% of agent patches on SEC-bench closely match the historical fix, and almost none do after the vulnerabilities are transplanted and mutated.

In the revision, we will state this purpose and change "reduces the risk of data contamination" in Sections 1 and 3.4 to "avoids instance-level contamination".


Q2: validity of contract-oracle vulnerabilities

Contract oracles let us verify vulnerabilities that sanitizers miss, such as an overflow that stays inside one object (Section 3.3). We ensure their validity in three ways.

  1. The oracle checks a real safety bound. It sits right before one operation, such as a copy, an index, or an allocation, and checks the bound that this operation needs, for example that a copy writes no more than its destination holds. The bound is either the invariant of an atlas mechanism, which real CVE fixes enforce, or one the injector writes itself. Mechanisms about the order of events, such as use-after-free, have no such bound, so they get no oracle and must be confirmed by a sanitizer (Appendix A.2).
  2. The violation comes from the injection. The oracle is compiled into both the clean and the injected build, which differ only in the edit. The clean build must replay every seed and the witness without a violation, which rejects an oracle that is misplaced or that the original code already violates (Appendix B.3). After acceptance, we also fuzzed the clean build of each of the 1,100 oracle-only environments with its oracle enabled and found no violation (Section 4.6).
  3. The harness gives the verdict. It rebuilds both snapshots from scratch, replays the witness three times on each, and evaluates Equation 1 itself. The injector cannot report a result on its own and only submits the record of a passing run (Appendix B.1).

Appendix C.6 gives an example. In the CoAP parser of Wakaama, the injection deletes the check that keeps the token within its 8-byte array, and the oracle checks exactly this bound. The witness carries a 15-byte token, so the copy writes 7 bytes past the array into the next fields of the same object. AddressSanitizer and UBSan report nothing because the write stays inside the object, and a sanitizer-only pipeline would discard this environment. Yet it is the same kind of overflow that real CVEs fixed. Five of the ten CVE fixes behind this mechanism repair an overflow from an array field into the fields after it (Appendix A.4).

In conlcusion, The three checks hold an oracle-confirmed environment to the same differential test as a sanitizer-confirmed one.


We sincerely thank you for your constructive feedback.and hope that these clarifications and the additional experiment results address your questions. If you feel that our response and proposed revisions resolve the issues raised, we would be deeply grateful if you might consider re-evaluating our submission.

With sincere appreciation,

The Authors

AuthorsOpenReview ↗

Thank You and Follow-up

Dear Reviewer MW72,

Thank you again for your thoughtful comments and for taking the time to consider our rebuttal. We would sincerely appreciate any further questions or suggestions you may have, and would be happy to provide additional clarification if helpful.

If you feel that our responses have addressed your questions, we would be deeply grateful if you would consider reassessing our submission in light of these clarifications.

Thank you again for your time and consideration!

With sincere appreciation,

The Authors

Reviewer_StorOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CyberCraft, an agentic framework for automatically synthesizing repository-level vulnerability environments to train security code agents. CyberCraft employs a vulnerability injector guided by a structured atlas to generate vulnerability environments, a teacher agent to generate verified solution trajectories, and a student agent trained through supervised fine-tuning and reinforcement learning. The framework constructs a dataset of 7,979 vulnerability environments covering 514 CWEs. Experiments on CyberCraft-Bench and real-world vulnerability benchmarks demonstrate that CyberCraft improves the student's vulnerability localization, reproduction, and repair capabilities.

Critical strengths

Automated synthesis of diverse vulnerability environments. This paper proposes an agentic framework that leverages a structured vulnerability mechanism atlas and differential verification to generate repository-level vulnerability environments. This approach reduces reliance on publicly disclosed vulnerabilities and enables the construction of a large-scale, verifiable training dataset for security tasks.

Critical weaknesses

none

Questions for authors

Q1: This paper introduces an atlas containing 1,751 vulnerability mechanisms and reports that the synthesized environments cover 514 CWEs. However, CWE-level coverage does not necessarily reflect mechanism-level diversity, as multiple distinct mechanisms may belong to the same CWE. Although the ablation study demonstrates that the atlas improves CWE diversity, it remains unclear how many of the 1,751 mechanisms are actually represented in the generated environments and whether certain long-tail mechanisms are underrepresented or absent from the generated environments. Furthermore, this paper demonstrates improvements on real-world vulnerability benchmarks, but it does not explicitly evaluate the student’s generalization to vulnerability mechanisms absent from its training data. Repository-level separation and cross-language transfer do not necessarily establish mechanism-level generalization. Could the authors report mechanism-level coverage statistics and clarify whether the student can generalize to previously unseen vulnerability mechanisms?

Q2: Although this paper defines the student’s tasks as vulnerability localization, reproduction, and repair, it remains unclear when and under what conditions the trained student agent is expected to be invoked in a real-world software development workflow. For example, is the agent intended to operate automatically upon a new code commit or pull request, after a security analysis tool reports a potential vulnerability, or only when a vulnerability has already been disclosed? These scenarios differ substantially in the information available to the agent and the assumptions underlying its tasks. The authors should clarify the intended invocation scenario and explain how the evaluation settings correspond to the envisioned practical workflow.

Q3: This paper describes a workflow involving three components: the injector agent, the teacher agent, and the student agent. Although their basic responsibilities and information flow are presented, the rationale behind this separation of roles is not sufficiently discussed. In particular, since the injector already possesses information about the injected vulnerability, including its mechanism, location, and triggering inputs, it is unclear why a separate teacher agent is necessary to generate training demonstrations rather than deriving supervision directly from the injection process. Furthermore, the teacher receives the vulnerability description, whereas the student is ultimately evaluated without such information. The implications of this difference in information availability deserve further clarification. The authors should more explicitly explain the design rationale for these components, their respective information dependencies, and how this architecture contributes to the effectiveness of student training.

3 comments
AuthorsOpenReview ↗

Reply part 1 to Reviewer Stor

Dear Reviewer Stor,

We sincerely thank you for your careful reading and thoughtful questions. Your insightful comments have helped us significantly strengthen the manuscript. We have carefully considered your valuable insights and try our best to address your questions below. To address each of them in sufficient detail, we split our response into two parts. This is Part 1 (Q1 and Q2); Part 2 (Q3) follows in the next comment.

(We apologize for the late reply. We wanted to respond to all reviewers together, and some additional experiments for the other reviews took longer than we expected.)


Q1 — mechanism-level coverage and generalization to unseen mechanisms

We agree that one CWE can contain several distinct mechanisms, so we answer at the mechanism level.

1. How many of the 1,751 mechanisms are represented in the sythetic envrionments?

1,523 (87.0%), across 514 of the 531 CWE nodes. No small group of them dominates. The five most common account for 3.8% of the environments, and it takes 430 mechanisms to cover half of them.

This diversity of the envrionments comes from the atlas. In the ablation of Section 4.6, the same injector without the atlas produced 4,000 environments from only 151 mechanisms, and 11 mechanisms account for half of them. The 4,000 static environments made with the atlas use 1,012.

2. Are long-tail mechanisms underrepresented or absent?

No. Mechanisms backed by a single real fix make up 74% of the atlas, and most of them are used and supply over half of all sythetic environments:

Real fixes per mechanismMechanismsUsedShare of environments
11,29684.7%57.2%
2–443793.1%35.3%
≥518100%7.5%

Most of the 228 unused mechanisms are specific to other languages (e.g., prototype pollution) or need behavior a fuzzing entry point cannot drive (e.g., thread races).

3. Can the student generalize to mechanisms it has not seen?

Yes, although the gain is smaller than on seen mechanisms. We agree that a repository split and cross-language transfer do not show this, so we measured it directly. We split the 379 CyberCraft-Bench environments by whether the student's training environments contain (A) the same mechanism, (B) only other mechanisms of the same or a related CWE, or (C) neither. "Related" means an ancestor or descendant in CWE-1000, excluding the ten Pillars.

Table R1. CyberCraft-Bench score by group.

A: mechanism seenB: mechanism unseenC: CWE unseen
Environments2757331
Base (Qwen3.5-9B)15.513.710.8
Student (SFT + RL)32.525.619.4
Gain+17.0+11.9+8.6

On unseen mechanisms the student still gains 11.9 points, about 70% of its gain on seen ones. Even when the CWE is new, it gains 8.6 points.

PatchEval shows the same on real CVEs. Its vulnerabilities are in Go, JavaScript, and Python, and 70 of its 230 instances have a CWE absent from our training data. On these the student improves from 14.3 to 32.9, a gain of 18.6 points, against 23.1 on the other 160.

The gain is still largest on seen mechanisms, so wider coverage helps, and the atlas provides it (e.g., CWE-806).


Q2 — application scenarios

We designed the agent mainly for routine audits, e.g., periodic or before a release, where no vulnerability has been reported and the agent has to find, reproduce, and fix one on its own. CyberCraft-Bench evaluates this setting. The agent can also start from a report, such as a fuzzer crash or a disclosed vulnerability, and do the remaining steps. SEC-bench, CyberGym-E2E, and PatchEval evaluate this case. Table R2 places each scenario in your question.

Table R2. Application scenarios and their evaluation. Results are base → trained student from Table 2, with SEC-bench as the mean of its two tasks.

ScenarioAgent receives, besides the repositoryEvaluation: base → student
Routine or pre-release audit (main use)only that it contains a vulnerabilityCyberCraft-Bench: 14.8 → 30.1
A crash is reported, e.g., by a fuzzerthe crash report, and for patching also the crashing inputSEC-bench: 12.0 → 35.5; CyberGym-E2E: 11.4 → 24.8
A vulnerability is disclosedits descriptionPatchEval: 15.2 → 37.0

Training covers the three scenarios. In RL, each environment comes with no hint, with its description, with a crash report to reproduce, or with the crashing input to repair.


The environments, benchmark, atlas, and reusable synthesis pipeline will be released to support community training and evaluation of security agents and the generation of new verified environments.

Thank you again for your time and expertise. We hope these clarifications address your questions. We would be incredibly grateful if you might consider these updates and improve your perspective on our work.

With sincere appreciation,

The Authors

AuthorsOpenReview ↗

Part 2 of our response to Reviewer Stor

Dear Reviewer Stor,

This is Part 2 of our response, covering Q3.


Q3 — why the role are separated, and what the teacher's description means for the student

We separate the roles by what each one knows (Table R3). The injector knows the answer. The student has to learn to find it on its own. The teacher solves each task from the student's position, with the description as its only extra input.

Table R3. Information per role.

RoleReceivesProduces
Injector (V4-Flash with injection tools)clean repository and atlas, plus a profile of the student's failures in co-evolutionpublic task and a hidden answer key (diff, triggering inputs, mechanism, description)
Teacher (V4-Flash in Codex)public task and descriptionverified demonstrations for SFT
Student (Qwen3.5-9B in Codex)teacher transcripts in SFT, then the public task with a hint if needed in RL. At test time, the public task or the benchmark's inputfailures that update its profile

1. Why a separate teacher, instead of supervision from the injection?

We do use what the injector knows, but as an answer key. The verifier checks each attempt against it to select demonstrations, give RL rewards, and score evaluations. It cannot serve as a demonstration, because the student has to learn to find the bug from the repository alone.

2. What does the teacher's description mean for the student?

In SFT the student sees the same description as the teacher, so the difference you point out is between training and evaluation. We made this choice on purpose. The description helps the student learn, and CyberCraft-Bench leaves it out to measure what the student can do alone.

The teacher needs the description for hard environments. In one blind attempt, V4-Flash repairs 58.3% of CyberCraft-Bench but only 34.1% of hard environments (Table 1). Since we keep an environment only if three of at most six attempts pass, a blind teacher would lose many hard ones.

In SFT, the description stays in each transcript, so every action the student imitates has a visible reason. Most of what it learns, such as building, running inputs, and patching, does not depend on the description. Even with no hint at test time, SFT alone raises the CyberCraft-Bench score from 14.8 to 25.6.

In RL, the student practices without the description whenever it can. An environment starts with no hint if the teacher solved it easily, and with the description otherwise. It gets more information, up to a crash report and then the crashing input, only when a batch of the student's attempts all fail. This raises the CyberCraft-Bench score to 30.1, against 26.9 when every rollout only gets the description (Table 2).

3. What does each role contribute?

The paper ablates each design choice. Removing the atlas from the injector lowers the student's mean gain over the base model on the four security benchmarks from 5.5 to 1.7 points, with the same Qwen3.6-27B teacher. Replacing the V4-Flash teacher with the weaker Qwen3.6-27B lowers this gain from 9.8 to 5.5. After SFT, having the injector target the student's failures gives 1.4 to 1.7 times the static student's gain on each benchmark, with 700 fewer environments.


The environments, benchmark, atlas, and reusable synthesis pipeline will be released to support community training and evaluation of security agents and the generation of new verified environments.

Thank you again for your time and expertise. We hope these clarifications address your questions. We would be incredibly grateful if you might consider these updates and improve your perspective on our work.

With sincere appreciation,

The Authors

AuthorsOpenReview ↗

Thank You and Follow-up

Dear Reviewer Stor,

Thank you again for your thoughtful comments and for taking the time to consider our rebuttal. We would sincerely appreciate any further questions or suggestions you may have, and would be happy to provide additional clarification if helpful.

If you feel that our responses have addressed your questions, we would be deeply grateful if you would consider reassessing our submission in light of these clarifications.

Thank you again for your time and consideration!

With sincere appreciation,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.