- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AuraForge, a pipeline for constructing executable secure-coding tasks from historical vulnerability fixes, and AuraGym, a dataset of 679 tasks from 344 repositories across Python, JavaScript, and TypeScript. AuraForge generates security tests from attacker-oriented invariants and attack variants. It provides language adapters and removes channels through which coding agents could retrieve or manipulate reference implementations. The synthesized tests achieve similar vulnerability-detection recall to human tests while substantially reducing false positives. SFT of Qwen3.5-4B on successful AuraGym trajectories improves functional and security performance on SusVibes-style evaluations.
Critical strengths
- The paper addresses an important bottleneck: scalable security supervision for repository-level coding agents.
- AuraForge provides a framework for security supervision: its attack-effect assertions avoid overfitting to historical patches, its evaluation includes detailed comparisons and qualitative case studies, AuraGym offers broad repository, CWE, and language coverage, and its environment safeguards address realistic reward-hacking channels.
- The SFT experiment demonstrates the practical value of the generated dataset: fine-tuning Qwen3.5-4B on AuraGym trajectories improves average functional and security pass rates by 19.7 and 6.2 percentage points, respectively, outperforming training based on human-written security tests.
Critical weaknesses
- Evaluation is limited to the closely related SusVibes task family, and more evaluation over secure-coding benchmarks should be included.
- Only one student model size, Qwen3.5-4B, is fine-tuned, leaving it unclear whether the observed gains generalize to other model families or scale consistently with model capacity.
- No data-scaling. The paper lacks a data-scaling analysis: it does not vary the number of training tasks or trajectories, making it unclear whether performance continues to improve with additional synthesized data or how data quantity contributes to the reported gains.
Questions for authors
-
What is the exact size and composition of the SFT dataset? Please clarify the apparent discrepancy between the trajectory counts reported in the main text and Table 8, including how many tasks, trajectories, and tokens were ultimately used.
-
Do the improvements remain when AuraGym and are matched for the number of tasks, trajectories, and training tokens? This would help separate the effect of test quality from that of greater data quantity and diversity.
-
How do the benefits scale with model and dataset size? Results across multiple student-model sizes and different fractions of AuraGym would clarify whether the observed gains generalize beyond Qwen3.5-4B.
-
Does training on AuraGym improve performance on security benchmarks with substantially different task formats, such as backend generation, code completion, or vulnerability repair? The current evaluation is limited to SusVibes-style tasks.
-
Is every generated attack variant independently executed and validated against both the vulnerable and fixed implementations? If acceptance occurs only at the suite level, how does the pipeline ensure meaningful coverage of all identified attack variants?
4 comments
Q5: Is every generated attack variant independently executed and validated against both the vulnerable and fixed implementations?
A: Validation happens at the suite level, by design. A suite is accepted only if both runs complete without test errors, at least one test fails on the vulnerable implementation, and the suite passes on the fixed implementation (Section 3.2.2). We do not require every test to fail on the historical vulnerable code, because that code may exhibit only one of the possible attack variants. Other variants target vulnerable implementations that the original commit does not contain; keeping only the tests that fail on the historical code would discard them and reduce coverage of alternative vulnerable implementations. Instead, we let each identified attack variant be covered by at least one test.
Thank the authors for their response. The data-scaling experiment using 25%, 50%, 75%, and 100% of the data seems to suggest that performance has already plateaued, despite the dataset containing only 679 tasks. This raises concerns that the synthetic dataset may lack sufficient diversity, thereby limiting further improvements as more data is added. Given this, I maintain my original score.
Thanks for the reviewer's response, and we are glad to have further discussion!
First, even if we only fine-tune on a small set of 594 AuraGym trajectories, it can already improve the Qwen 3.5 4B model significantly. And the improvement from 50% to 100% training data (297->594) also shows the promise of scaling up.
At the same time, we would like to highlight the main contribution is the synthesis pipeline AuraForge to scale up security training instance collection for security. We show its effectiveness by collecting AuraGym, which has
- High diversity and coverage: 679 executable tasks, spanning repositories (344 real-world), languages (Python, JavaScript, and TypeScript), and vulnerability categories (177 CWEs)
- High quality: AuraForge's synthesized test cases have 83% fewer false positives and are more effective in agentic training.
All these show that AuraForge can benefit the community by providing reliable, diverse, expendable training signals and encouraging the building of more powerful and secure coding agents by sampling more trajectories and training on larger models.
We are grateful for Reviewer fEAK's thoughtful review and valuable suggestions. It is encouraging that the reviewer sees AuraForge as addressing an important bottleneck in scalable security supervision, and recognizes the practical value of our SFT results. The suggested experiments on model families and data scaling strengthened the paper, and we present them below.
W1/Q4: Evaluation beyond SusVibes-style tasks.
Similar to training on SWE-bench series[1], AuraGym targets on scaling up training for repository-level feature implementation. Beyond generic functionality, we care more about the security-sensitive feature implementation. Here, security depends on how new code interacts with an existing codebase, and our evaluation covers this setting across three languages(Python, Javascript, Typescript). We agree that formats such as backend generation and vulnerability repair are valuable, and we are exploring more task formats and vulnerability as our future work.
[1] SWE-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents.
[2] SWE-Gym: Training software engineering agents and verifiers with swe-gym.
To explore the effectiveness of our synthesis test suits in training different models, we do similar experiments on FrogNano-4B (https://huggingface.co/microsoft/FrogNano-4B-2609)
As shown above, with a stronger base model, the observations are similar. With AuraGym, the agent achieves a higher SecPass on both languages. AuraGym can improve SecPass on SusVibes but degrade a little on FuncPass of SusVibes.
W3/Q3: Data scaling.
A: We fine-tuned Qwen3.5-4B on random 25%, 50%, and 75% subsets of the 594 AuraGym trajectories.
Performance generally improves with more data: from 25% to 100%, FuncPass rises by 5.9 and 8.5 points and SecPass by 1.1 and 3.7 points on the two benchmarks. TS/JS SecPass fluctuates between 50% and 75%, within run-to-run variation. The trend suggests that more synthesized tasks would bring further gains.
Q1: Exact size and composition of the SFT dataset.
A4: We use only DeepSeek-V4.1-Flash trajectories for fine-tuning, because Muse Spark 1.3 does not expose its reasoning. Of 2,218 valid DeepSeek trajectories, filtering with each suite leaves 309 trajectories for AuraGym_h and 594 for AuraGym. The 3,753 Muse Spark trajectories are used only to compare human-written and synthesized tests (Section 5.1 and Appendix B). The appendix text now states this.
Q2: Matching tasks, trajectories, and training tokens.
A5: We highlight 2 advantages of our synthesis test suites in our paper: (i) high quality: 83% fewer false positive (Section 5.1); (ii) easier to scale up (eg. multiple languages and repositories).
When tasks are matched, synthesized suites still train a more secure model. On the 368 tasks shared by AuraGym and AuraGym_h, we selected trajectories with either the human suites (309) or the synthesized suites (336) and fine-tuned Qwen3.5-4B on each.
On the same tasks, synthesized suites improve SecPass on both benchmarks, consistent with their 83% fewer false positives (Section 5.1). Full AuraGym adds a further gain from scale, which the pipeline makes possible because it does not depend on human-written security tests.
We agree that controlling the number of training trajectories and tokens helps isolate their effect on training. However, different verifiers naturally retain different numbers of trajectories because their verification accuracy differs [1,2,3]. We therefore treat the trajectory count as an outcome of test quality rather than a confound.
[1] HARDTESTGEN: A High-Quality RL Verifier Generation Pipeline for LLM Algorithmic Coding, ICLR 2026
[2] ACECODER: Acing Coder RL via Automated Test-Case Synthesis, ACL 2025
[3] Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning, NeurIPS 2025