- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 3
- Presentation
- 2(−1)
- Soundness
- 3
Read review
Paper summary
This paper uses an agentic pipeline to synthesize vulnerability tasks that do not rely on existing CVE instances, builds 7,979 verifiable environments from 713 projects, and shows the value of these data for the security domain through SFT and RL, while comparative experiments show the value of co-evolution relative to static datasets.
Critical strengths
Data construction is practically meaningful. It does not rely on existing CVE instances and yields 7,979 verifiable environments from 713 projects, each of which must pass the differential criterion, and it has practical potential for alleviating the diversity limits of tasks built from existing CVE examples.
Critical weaknesses
1.The comparison remains insufficient. Beyond comparisons with several strong models and a static ablation, the paper does not compare with students trained on other datasets of the same positioning or on real CVE environments, so the advantage of this data over other data is not established. 2.The empirical support for environment diversity is limited. The current evidence relies on the labels of the atlas itself, these labels are not manually verified, diversity is not examined from other angles, and teacher filtering may reduce diversity in difficulty.
Questions for authors
1.Under equal conditions, could the authors add comparisons with students trained on real CVE environments and on data from existing injectors? 2.Given the practical cost of tasks, could the authors report the injection acceptance rate and the token cost per environment?
8 comments
Part 2 of our response to Reviewer 2zKd
Dear Reviewer 2zKd,
This is Part 2 of our response, covering W2 and Q2.
W2 — part 1/2: The empirical support for environment diversity
Our 7,979 environments cover 514 CWEs and 1,523 mechanisms. These labels come from the atlas card each environment was injected from, so we also labeled the environments again with a model that does not use the atlas.
Diversity check from other dimensions
We had DeepSeek v4.1 flash label the CWE of every environment from only its diff and its sanitizer or oracle report. DeepSeek v4.1 took no part in building the environments and labels every set in the same way, so the comparison no longer depends on the atlas. This check is new. It compares the static set of the paper with the w/o-atlas set of Section 4.6, which was built by the same injector on the same repositories without the atlas, and uses the real ARVO environments from W1 as a reference. Since the sets differ in size, we count CWEs in 450 random environments per set, averaged over 100 draws. The static set covers 102 CWEs, against 35 for the w/o-atlas set and 29 for ARVO. For ARVO this is more than the 11 CWEs of Section 3.2, because ARVO's own labels follow only the crash type. The five most common CWEs account for 31.6% of the static set, against 52.4% and 64.9%. Under labels that do not come from the atlas, the static set is therefore far more diverse than the w/o-atlas set and also more diverse than real vulnerabilities.
W2 — part 2/2: teacher filtering and difficulty
We agree that filtering removes some hard environments, but only a few. The teacher gets up to six attempts on each training environment, and an environment is replaced if fewer than three of them pass. For this rebuttal we counted the outcomes in the teacher logs of the paper's two training runs. 91% of the attempted environments were kept, including 81.5% of the hard ones, so hard environments drop only from 31.9% to 28.6% of the set.
Few are lost because the teacher is given the description of the vulnerability. Without it, the same teacher repairs only 34.1% of the hard CyberCraft-Bench environments (Table 1).
The kept environments remain hard for the student. In RL, 32% of the completed groups needed the reproduction or repair view, which an environment reaches only after the student fails it with the description. The evaluation sets are not filtered by the teacher, since CyberCraft-Bench and the feedback set never pass through the teacher.
Q2: acceptance rate and cost per environment
Across the injection runs behind the 7,979 environments, 29.1% of attempts pass Equation 1. An attempt is one injector session on one repository, limited to 3 hours (Appendix B.1). Most failed attempts find no witness input within this budget, and the rest fail to build, break regression tests, or also trigger on the clean snapshot.
Some accepted environments are later dropped as duplicates or replaced after teacher filtering, so each of the 7,979 environments takes about 3.9 attempts. Counting all of them, injecting one environment takes about 11M input tokens (92% cached) and 140k output tokens. CPU time for builds, tests, and witness search limits throughput more than tokens do.
The atlas lowers the acceptance rate from 38.6% in the w/o-atlas run of Section 4.6 to 30.4% in the static run, because without it the injector falls back on familiar edits that are easy to verify. These extra attempts pay for the diversity shown in W2 and the larger gains in Table 2 of the paper.
We will add the breakdown of attempts and costs to Appendix C and state the acceptance rate and cost per environment in Section 3.4.
The environments, benchmark, atlas, and reusable synthesis pipeline will be released to support community training and evaluation of security agents and the generation of new verified environments.
Thank you again for your thoughtful review. We hope our clarifications and new experiments results fully address your questions, and we would be deeply grateful if they could positively influence your final assessment of our work.
With sincere appreciation,
The Authors
Thank You and Follow-up
Dear Reviewer 2zKd,
Thank you again for your thoughtful comments and for taking the time to consider our rebuttal. We would sincerely appreciate any further questions or suggestions you may have, and would be happy to provide additional clarification if helpful.
If you feel that our responses have addressed your questions, we would be deeply grateful if you would consider reassessing our submission in light of these clarifications.
Thank you again for your time and consideration!
With sincere appreciation,
The Authors
Part 1 of our response to Reviewer 2zKd
Dear Reviewer 2zKd,
We deeply appreciate your constructive critiques, which highlight important nuances that we should clarify in the revision. Below, we try our best to address your specific questions. To address each of them in sufficient detail, we split our response into two parts. This is Part 1 (W1 & Q1); Part 2 (W2 and Q2) follows in the next comment.
(We apologize for the late reply. The additional experiments below took longer than we expected to complete.)
W1 & Q1: additional comparisons with real CVE environments and existing injectors
Following it, we ran a new experiment that compares our environments with real CVE environments under equal conditions (part 1). With the same number of environments, a student trained on ours gains nearly as much on real vulnerabilities as one trained on real CVE environments. Few real environments remain usable for training, while our pipeline can produce many more.
1. Students trained on real CVE environments (new experiment)
We used ARVO (Mei et al., 2026), a dataset of real vulnerabilities found by OSS-Fuzz in C/C++ projects of the same kind as ours. Each instance provides a buildable vulnerable version, the fixed version, and an input that crashes the vulnerable one, but no agent trajectories. We processed it in the same way as our own data. Since ARVO has no description of the vulnerability, DeepSeek-V4-Flash writes one from the fix and the crash report, and our teacher, also DeepSeek-V4-Flash, then collects the trajectories. After removing the projects of SEC-bench and CyberGym-E2E (PatchEval has no C/C++ projects) and re-verifying each instance, 900 of the 5,001 instances remain, and the teacher produces three verified solutions for 731 of them.
We then trained Qwen3.5-9B with SFT on these 731 ARVO environments and, separately, on 731 environments sampled at random from the static set of the paper.
Table R4. Students trained under equal conditions (success rates, %).
Although we removed the projects these benchmarks evaluate, ARVO is still much closer than our data to two of them. CyberGym-E2E is built from OSS-Fuzz vulnerabilities, partly taken from ARVO itself, and SEC-bench also consists of sanitizer-verified memory-safety bugs in C/C++ projects. Even so, with 731 environments each, our student gains nearly as much on real vulnerabilities as the ARVO student (+3.9 vs. +4.5). Real data, however, cannot grow in the same way. After decontamination ARVO yields only 731 training environments, while the static set alone holds 4,000 and gains 10.9 points. The two sources also add up, and adding ARVO to the static set raises the gain to 11.9 points.
2. Students trained on data from existing injectors (existing ablation in the paper)
Existing injectors cannot be used directly to train agents like ours. Function-level injectors such as VulGen, VGX, VulScribeR, and AVIATOR produce vulnerable functions that cannot be built or run. Repository-level work has released only small evaluation sets (22 and 212 samples, see related works).
The closest repository-level injectors (Migliaccio et al.; Wei et al.) simply ask an LLM to insert a bug or a named weakness. Their injections lack diversity, because the LLM falls back on a few familiar edits, such as removing a bounds check before a memory write. Our manuscript already tests this approach under equal conditions in the w/o-atlas ablation of Section 4.6, which removes only the atlas and keeps everything else the same. Its 4,000 environments cover far fewer weaknesses than ours (35 vs. 102 CWEs per 450 environments, see W2). With the same teacher, Qwen3.6-27B, its student is weaker on all four security benchmarks, and the average gain drops from 5.5 to 1.7 points (Table 2 of the paper).
The environments, benchmark, atlas, and reusable synthesis pipeline will be released to support community training and evaluation of security agents and the generation of new verified environments.
Thank you again for your thoughtful review. We hope our clarifications and new experiments results fully address your questions, and we would be deeply grateful if they could positively influence your final assessment of our work.
With sincere appreciation,
The Authors