acceptodds
Under review as a conference paper at ICLR 2027

AuraForge: Scaling Security Supervision for Training Coding Agents

Abstract

Coding agents are now proficient enough to generate complex software applications from a single prompt. As their capabilities have grown, human oversight has increasingly shifted from line-by-line code review toward hands-off evaluation of outcomes. However, recent studies have shown that such a transition exposes a critical risk: functional correctness alone does not guarantee a secure implementation. Despite growing attention to code security, training safer coding agents remains challenging because reliable security supervision is difficult to obtain at scale from real-world repositories. We introduce AuraForge to synthesize and validate executable security tests for training secure coding agents. Our approach combines attack-oriented test synthesis, language-extensible task construction, and safeguards against reward hacking. Using AuraForge, we construct AuraGym, a multi-language and multi-CWE executable training gym: 679 executable feature-implementation tasks from 344 real-world repositories across Python, JavaScript, and TypeScript, covering 177 CWE categories. On the subset with human-written security tests, AuraForge produces about 3 times as many test cases on average and reduces the false-positive rate by 83.23%, allowing alternative secure implementations to receive correct supervision. Training Qwen3.5-4B with synthesized security tests gains larger improvements than human-written security tests (average 19.7 FuncPass and 6.2 SecPass vs. 14.9 FuncPass and 4.4 SecPass) on three languages. These results demonstrate that AuraForge provides more diverse and reliable security supervision to train secure coding agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Cybersecurity, Coding Agent, Security Test Synthesis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fEAKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AuraForge, a pipeline for constructing executable secure-coding tasks from historical vulnerability fixes, and AuraGym, a dataset of 679 tasks from 344 repositories across Python, JavaScript, and TypeScript. AuraForge generates security tests from attacker-oriented invariants and attack variants. It provides language adapters and removes channels through which coding agents could retrieve or manipulate reference implementations. The synthesized tests achieve similar vulnerability-detection recall to human tests while substantially reducing false positives. SFT of Qwen3.5-4B on successful AuraGym trajectories improves functional and security performance on SusVibes-style evaluations.

Critical strengths
  • The paper addresses an important bottleneck: scalable security supervision for repository-level coding agents.
  • AuraForge provides a framework for security supervision: its attack-effect assertions avoid overfitting to historical patches, its evaluation includes detailed comparisons and qualitative case studies, AuraGym offers broad repository, CWE, and language coverage, and its environment safeguards address realistic reward-hacking channels.
  • The SFT experiment demonstrates the practical value of the generated dataset: fine-tuning Qwen3.5-4B on AuraGym trajectories improves average functional and security pass rates by 19.7 and 6.2 percentage points, respectively, outperforming training based on human-written security tests.
Critical weaknesses
  • Evaluation is limited to the closely related SusVibes task family, and more evaluation over secure-coding benchmarks should be included.
  • Only one student model size, Qwen3.5-4B, is fine-tuned, leaving it unclear whether the observed gains generalize to other model families or scale consistently with model capacity.
  • No data-scaling. The paper lacks a data-scaling analysis: it does not vary the number of training tasks or trajectories, making it unclear whether performance continues to improve with additional synthesized data or how data quantity contributes to the reported gains.
Questions for authors
  • What is the exact size and composition of the SFT dataset? Please clarify the apparent discrepancy between the trajectory counts reported in the main text and Table 8, including how many tasks, trajectories, and tokens were ultimately used.

  • Do the improvements remain when AuraGym and AuraGymh\text{AuraGym}_h are matched for the number of tasks, trajectories, and training tokens? This would help separate the effect of test quality from that of greater data quantity and diversity.

  • How do the benefits scale with model and dataset size? Results across multiple student-model sizes and different fractions of AuraGym would clarify whether the observed gains generalize beyond Qwen3.5-4B.

  • Does training on AuraGym improve performance on security benchmarks with substantially different task formats, such as backend generation, code completion, or vulnerability repair? The current evaluation is limited to SusVibes-style tasks.

  • Is every generated attack variant independently executed and validated against both the vulnerable and fixed implementations? If acceptance occurs only at the suite level, how does the pipeline ensure meaningful coverage of all identified attack variants?

4 comments

We are grateful for Reviewer fEAK's thoughtful review and valuable suggestions. It is encouraging that the reviewer sees AuraForge as addressing an important bottleneck in scalable security supervision, and recognizes the practical value of our SFT results. The suggested experiments on model families and data scaling strengthened the paper, and we present them below.

W1/Q4: Evaluation beyond SusVibes-style tasks.

Similar to training on SWE-bench series[1], AuraGym targets on scaling up training for repository-level feature implementation. Beyond generic functionality, we care more about the security-sensitive feature implementation. Here, security depends on how new code interacts with an existing codebase, and our evaluation covers this setting across three languages(Python, Javascript, Typescript). We agree that formats such as backend generation and vulnerability repair are valuable, and we are exploring more task formats and vulnerability as our future work.

[1] SWE-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents.

[2] SWE-Gym: Training software engineering agents and verifiers with swe-gym.


To explore the effectiveness of our synthesis test suits in training different models, we do similar experiments on FrogNano-4B (https://huggingface.co/microsoft/FrogNano-4B-2609)

ModelSusVibes Func / SecSusVibes_tsjs Func / Sec
Qwen 3.5 4B12.90 / 2.1012.19 / 0.00
FrogNano-4B27.96 / 3.2335.37 / 6.10
+ AuraGym_h25.27 / 8.0635.37 / 6.10
+ AuraGym30.65 / 8.0639.02 / 9.76

As shown above, with a stronger base model, the observations are similar. With AuraGym, the agent achieves a higher SecPass on both languages. AuraGym can improve SecPass on SusVibes but degrade a little on FuncPass of SusVibes.


W3/Q3: Data scaling.

A: We fine-tuned Qwen3.5-4B on random 25%, 50%, and 75% subsets of the 594 AuraGym trajectories.

Fraction of trajectoriesSusVibes Func / SecSusVibes_tsjs Func / Sec
25%22.04 / 4.8428.05 / 4.88
50%22.04 / 5.3832.93 / 8.54
75%27.96 / 5.9132.93 / 6.10
100% (AuraGym)27.96 / 5.9136.59 / 8.54

Performance generally improves with more data: from 25% to 100%, FuncPass rises by 5.9 and 8.5 points and SecPass by 1.1 and 3.7 points on the two benchmarks. TS/JS SecPass fluctuates between 50% and 75%, within run-to-run variation. The trend suggests that more synthesized tasks would bring further gains.


Q1: Exact size and composition of the SFT dataset.

A4: We use only DeepSeek-V4.1-Flash trajectories for fine-tuning, because Muse Spark 1.3 does not expose its reasoning. Of 2,218 valid DeepSeek trajectories, filtering with each suite leaves 309 trajectories for AuraGym_h and 594 for AuraGym. The 3,753 Muse Spark trajectories are used only to compare human-written and synthesized tests (Section 5.1 and Appendix B). The appendix text now states this.


Q2: Matching tasks, trajectories, and training tokens.

A5: We highlight 2 advantages of our synthesis test suites in our paper: (i) high quality: 83% fewer false positive (Section 5.1); (ii) easier to scale up (eg. multiple languages and repositories).

When tasks are matched, synthesized suites still train a more secure model. On the 368 tasks shared by AuraGym and AuraGym_h, we selected trajectories with either the human suites (309) or the synthesized suites (336) and fine-tuned Qwen3.5-4B on each.

Training dataSusVibes Func / SecSusVibes_tsjs Func / Sec
Base (Qwen3.5-4B)12.90 / 2.1012.19 / 0.00
Human suites (AuraGym_h)22.04 / 4.8432.93 / 6.10
Synthesized suites, same 368 tasks23.12 / 5.3832.93 / 7.32
Full AuraGym27.96 / 5.9136.59 / 8.54

On the same tasks, synthesized suites improve SecPass on both benchmarks, consistent with their 83% fewer false positives (Section 5.1). Full AuraGym adds a further gain from scale, which the pipeline makes possible because it does not depend on human-written security tests.

We agree that controlling the number of training trajectories and tokens helps isolate their effect on training. However, different verifiers naturally retain different numbers of trajectories because their verification accuracy differs [1,2,3]. We therefore treat the trajectory count as an outcome of test quality rather than a confound.

[1] HARDTESTGEN: A High-Quality RL Verifier Generation Pipeline for LLM Algorithmic Coding, ICLR 2026

[2] ACECODER: Acing Coder RL via Automated Test-Case Synthesis, ACL 2025

[3] Co-Evolving LLM Coder and Unit Tester via Reinforcement Learning, NeurIPS 2025

Q5: Is every generated attack variant independently executed and validated against both the vulnerable and fixed implementations?

A: Validation happens at the suite level, by design. A suite is accepted only if both runs complete without test errors, at least one test fails on the vulnerable implementation, and the suite passes on the fixed implementation (Section 3.2.2). We do not require every test to fail on the historical vulnerable code, because that code may exhibit only one of the possible attack variants. Other variants target vulnerable implementations that the original commit does not contain; keeping only the tests that fail on the historical code would discard them and reduce coverage of alternative vulnerable implementations. Instead, we let each identified attack variant be covered by at least one test.

Thank the authors for their response. The data-scaling experiment using 25%, 50%, 75%, and 100% of the data seems to suggest that performance has already plateaued, despite the dataset containing only 679 tasks. This raises concerns that the synthetic dataset may lack sufficient diversity, thereby limiting further improvements as more data is added. Given this, I maintain my original score.

Thanks for the reviewer's response, and we are glad to have further discussion!

First, even if we only fine-tune on a small set of 594 AuraGym trajectories, it can already improve the Qwen 3.5 4B model significantly. And the improvement from 50% to 100% training data (297->594) also shows the promise of scaling up.

At the same time, we would like to highlight the main contribution is the synthesis pipeline AuraForge to scale up security training instance collection for security. We show its effectiveness by collecting AuraGym, which has

  • High diversity and coverage: 679 executable tasks, spanning repositories (344 real-world), languages (Python, JavaScript, and TypeScript), and vulnerability categories (177 CWEs)
  • High quality: AuraForge's synthesized test cases have 83% fewer false positives and are more effective in agentic training.

All these show that AuraForge can benefit the community by providing reliable, diverse, expendable training signals and encouraging the building of more powerful and secure coding agents by sampling more trajectories and training on larger models.

Reviewer_GQD5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AuraForge is a pipeline that turns real vulnerability-fixing commits into executable training tasks with synthesized security tests for training secure coding agents. It has three components: The Forge masks vulnerable implementations and generates feature requests without security hints across multiple languages; The Aura synthesizes attack-oriented security tests that assert observable attack effects rather than fix mechanisms; and The Seal removes reward hacking channels including git history, installed packages, and network access. The resulting AuraGym contains 679 tasks from 344 repositories across Python, JavaScript, and TypeScript covering 177 CWEs. Synthesized tests match human recall at 98.6% while reducing false positives by 83%. Training Qwen3.5-4B on AuraGym improves functional and security pass rates more than training on human-written tests.

Critical strengths
  1. The core insight is important and well-executed.

Human-written security tests naturally assert fix mechanisms because the maintainer who wrote the fix also wrote the test. AuraForge avoids this by synthesizing tests from the attacker’s perspective without access to the fix. The false positive reduction from 16.7% to 2.8% is large and well-explained: the error categorization in Table 9 shows that 64% of human false positives are fix-bound tests that demand a specific output, error message, or helper function defined only in the fix. The case studies make this concrete: a markdownify vulnerability can be fixed by clamping heading levels to six or by treating out-of-range headings as plain text, but the human test asserts the clamping behavior and fails the second approach, while the synthesized test just checks that a large heading tag does not produce a large output.

  1. The reward hacking analysis is empirically grounded.

Rather than simply asserting that reward hacking is a problem, the paper measures retrieval rates across 26 agent-model combinations on SusVibes, finding it occurs in 16% of runs and up to 40% for one model. The identification of specific channels including the scaffold’s own Python environment as a recovery path is a practically important finding.

Critical weaknesses
  1. The ground truth for security test quality is partially circular.

The paper’s central claim is that synthesized tests are more reliable than human-written tests. This rests on reported recall and false positive rates measured against ground truth labels. However 93% of those labels, covering 2,271 of 2,450 solutions, are established by agreement between the two test suites rather than by independent adjudication. Only the 179 disagreement cases receive attack-based adjudication with manual verification. If both suites share systematic blind spots, for example consistently accepting the same class of vulnerable implementation or rejecting the same class of secure one, those errors are invisible to the comparison. The reported 98.6% recall and 2.8% false positive rate therefore partially depend on an assumption that shared verdicts are correct. An independent audit of a random sample of agreement cases, not just disagreement cases, would strengthen the central test-quality results considerably.

  1. The training experiment does not isolate test quality from data quantity.

The comparison between AuraGym and AuraGymh training conflates several differences simultaneously: AuraGym has 679 tasks versus 431, covers 344 repositories versus 199, and provides approximately three times as many security tests per task. The improvement over AuraGymh training could therefore reflect more training data and broader repository coverage rather than better test quality. A controlled comparison using the same 431 tasks with only the security test suites changed would be needed to isolate the effect of test quality from data quantity.

Questions for authors
  1. 93% of security labels are established by agreement between the two test suites rather than by independent adjudication. Have you audited a random sample of agreement cases to verify that shared verdicts are correct? Shared systematic errors would affect the reported recall and false positive rates without appearing in the disagreement analysis.

  2. The comparison between AuraGym and AuraGymh training changes dataset size, repository coverage, and test quality simultaneously. Have you evaluated whether the improvement holds when AuraGym is subsampled to match AuraGymh in task count and security test density?

1 comment

Thank you for the encouraging feedback and the insightful suggestions. We are delighted that the reviewer finds our core insight important, namely that human-written security tests tend to assert the fix mechanism, and finds our reward-hacking analysis empirically grounded. Both suggested experiments were valuable, and we report them below.

W1/Q1: Auditing agreement cases.

A1: We sample 200 solutions from the 2,271 solutions where the human-written and synthesized suites agree with each other (Section 5.1), 50 for each combination of language (Python, TypeScript/JavaScript) and shared verdict (both accept, both reject). We use the same blind attack-based judge and verification method used for the disagreements on these samples.

Shared verdictValid samplesShared verdict correct
Both accept (secure)10096 (96.0%)
Both reject (vulnerable)99*96 (97.0%)
Total199192 (96.5%)

*1 sample is excluded due to flakiness.

Shared blind spots also cannot create the gap between suites. On an agreement case, both suites give the same verdict, so a wrong label counts against both equally. The false-positive gap (16.7% vs. 2.8%) comes entirely from the 179 independently adjudicated disagreement cases.


W2/Q2: Isolating test quality from data quantity.

A: We highlight 2 advantages of our synthesis test suites in our paper: (i) high quality: 83% fewer false positive (Section 5.1); (ii) easier to scale up (eg. multiple languages and repositories).

We ran the suggested controlled comparison. On the 368 tasks shared by AuraGym and AuraGym_h, we selected trajectories with either the human suites (309 trajectories) or the synthesized suites (336), and fine-tuned Qwen3.5-4B on each. The task set and repository coverage are identical; only the security supervision differs.

Training dataSusVibes Func / SecSusVibes_tsjs Func / Sec
Base (Qwen3.5-4B)12.90 / 2.1012.19 / 0.00
Human suites (AuraGym_h)22.04 / 4.8432.93 / 6.10
Synthesized suites, same 368 tasks23.12 / 5.3832.93 / 7.32
Full AuraGym27.96 / 5.9136.59 / 8.54

On the same tasks, synthesized suites improve SecPass on both benchmarks, consistent with their 83% fewer false positives (Section 5.1). Full AuraGym adds a further gain from scale, which the pipeline makes possible because it does not depend on human-written security tests.

On test density: we view it as part of what AuraForge contributes rather than a confound, since generating several attack variants per vulnerability is a design goal (Table 7). Higher density does not make filtering stricter: the synthesized suites accepted more trajectories than the human suites, because they wrongly reject fewer secure solutions.

Reviewer_hcp6OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a framework, namely AuraForge, for generating executable security tests to train coding agents. It combines attack-oriented test synthesis, multilingual task construction, and safeguards against implementation leakage. The paper also introduce AuraGym, a dataset containing 679 tasks from 344 repositories across three programming languages. Experiments show that synthesized tests achieve comparable vulnerability detection recall with fewer false positives than human-written tests. Training on AuraGym also improves functional and security performance.

Critical strengths
  • The paper addresses an important problem in training secure coding agents.
  • AuraGym covers a diverse set of repositories, languages, and vulnerability categories.
  • The evaluation includes detailed test-quality analysis and downstream training experiments.
Critical weaknesses

Limited independent validation of security labels. Section 5.1 treats the matching verdicts of the human-written and synthesized test suites as ground truth for 2,271 of 2,450 solutions. Only disagreements receive separate attack-based adjudication. Appendix F.2 further reports that 33 of the 190 LLM judge rulings were manually checked, with two overturned. This procedure cannot identify errors shared by both test suites. For example, a vulnerable implementation accepted by both suites would be incorrectly labelled secure. The authors acknowledge this limitation in Appendix F.7. Consequently, the reported recall and false-positive rates depend on labels that have not been independently verified for most solutions.
Insufficient control of training-data differences. Section 5.2 reports better fine-tuning results with synthesized security supervision. However, the paper does not clearly report the final training-set sizes and task distributions for the two variants. It is therefore difficult to separate the effects of test quality from differences in training-data availability and composition. The dataset descriptions also require clarification. Section 4 describes AuraGymh as a subset, but Table 7 reports more JavaScript and TypeScript instances in AuraGymh than in AuraGym. Appendix F.1 also reports 434 tasks carrying both test suites, compared with 431 instances in Table 7. These differences make the training comparison difficult to interpret.
Security gains are not clearly separated from functional gains. SecPass measures the percentage of solutions passing both functional and security tests. An increase in FuncPass can therefore raise SecPass without improving security among functionally correct solutions. Table 3 illustrates this distinction. On the Python SusVibes benchmark, SecPass/FuncPass is approximately 22.0% with human-written supervision and 21.1% with synthesized supervision. Although the synthesized variant achieves higher absolute SecPass, its conditional security rate is slightly lower. The TypeScript/JavaScript results show the opposite pattern. Thus, the experiments demonstrate improvements in overall performance but do not establish consistent gains in security conditional on functional correctness.
Unclear separation between Python training and evaluation data. Appendix A.6 describes a repository-disjoint split for the TypeScript/JavaScript evaluation set. However, the paper does not provide an equivalent description for the Python evaluation on the original SusVibes benchmark. Both AuraGym and SusVibes are derived from real-world vulnerability-fixing commits, making possible overlap relevant to the evaluation. The reported information does not establish whether the Python training and evaluation sets are disjoint at the repository or vulnerability level. This does not demonstrate that leakage occurred, but it leaves an important uncertainty about whether the Python results reflect generalization to unseen tasks.
Limited attribution of test-quality improvements. AuraForge combines attack-oriented test synthesis, vulnerability information, historical implementation feedback, and execution-based validation. Section 5.1 shows that the complete pipeline produces tests with substantially fewer false positives than human-written suites. The case studies also provide useful evidence of implementation-independent assertions. However, the evaluation does not isolate the contributions of these components. The synthesized suites also differ in size from the human-written suites. As a result, the experiments establish the effectiveness of the complete synthesis pipeline, but not the individual contribution of the proposed attack-oriented strategy.

Questions for authors
  • How reliable are the reported recall and false-positive rates when matching test verdicts are treated as ground truth?
  • Were the two fine-tuned models trained on comparable numbers of tasks and trajectories?
  • Does synthesized supervision improve security success conditional on functional correctness?
  • Are the Python training repositories disjoint from the SusVibes evaluation repositories?
  • What is the individual contribution of attack-oriented synthesis to test quality?
2 comments

We sincerely thank Reviewer hcp6 for the detailed and careful review. We are glad the reviewer finds the problem of training secure coding agents important, and appreciates the diversity of AuraGym and the depth of our test-quality and training analyses. The questions on labels, data control, and attribution were very helpful, and each led to a new experiment below.

W1/Q1: Ground-truth labels: 93% come from suite agreement

A: An independent audit confirms that 96.5% of shared verdicts are correct. We sample 200 solutions from the 2,271 solutions where the human-written and synthesized suites agree with each other (Section 5.1), 50 for each combination of language (Python, TypeScript/JavaScript) and shared verdict (both accept, both reject). We use the same blind attack-based judge and verification method used for the disagreements on these samples.

Shared verdictValid samplesShared verdict correct
Both accept (secure)10096 (96.0%)
Both reject (vulnerable)99*96 (97.0%)
Total199192 (96.5%)

*1 sample is excluded due to flakiness.

However, shared errors also cannot distinguish the gap between the two suites. On an agreement case, both suites give the same verdict, so a wrong label counts against both equally. The difference in false positives (16.7% vs. 2.8%) therefore comes entirely from the 179 disagreement cases, all of which were adjudicated independently.


W2/Q2: Comparable training data for the two fine-tuned models.

A: When tasks are matched, synthesized suites still train a more secure model. We restricted training to the 368 tasks shared by AuraGym and AuraGym_h and selected trajectories with either the human suites (309 trajectories) or the synthesized suites (336), then fine-tuned Qwen3.5-4B on each.

Training dataSusVibes Func / SecSusVibes_tsjs Func / Sec
Base (Qwen3.5-4B)12.90 / 2.1012.19 / 0.00
Human suites (AuraGym_h)22.04 / 4.8432.93 / 6.10
Synthesized suites, same 368 tasks23.12 / 5.3832.93 / 7.32
Full AuraGym27.96 / 5.9136.59 / 8.54

On the same tasks, synthesized suites improve SecPass on both benchmarks, consistent with their 83% fewer false positives (Section 5.1). Full AuraGym adds a further gain from scale, which the pipeline makes possible because it does not depend on human-written security tests.

We also clarified the dataset descriptions in the manuscript:

  • Final SFT sets. We use only DeepSeek-V4.1-Flash trajectories for fine-tuning, because Muse Spark 1.3 does not expose its reasoning. Filtering with each suite gives 309 trajectories for AuraGym_h and 594 for AuraGym. We have detailed this in Appendix E.
  • "Subset" wording. AuraGym_h is a comparable dataset with human-written security test cases, instead of a strict subset of AuraGym. We have revised Section 4.
  • Dataset Size: Table 7 reports the final 431 tasks, after removing 3 tasks (2 Python, 1 TypeScript) from the 434 in Appendix F.1 because of unrecoverable environment leakage. Besides, 4 AuraGym tasks and 6 AuraGym_h tasks are labeled with both JavaScript and TypeScript, as now noted in the caption of Table 7.

W3/Q3: Security gains conditional on functional correctness.

A: Thank the reviewer for this valuable suggestion. We added SecPass/FuncPass as below:

ModelSusVibes (Python)SusVibes_tsjs (TS/JS)
Base2.10 / 12.90 = 16.3%0.00 / 12.19 = 0.0%
+ AuraGym_h4.84 / 22.04 = 22.0%6.10 / 32.93 = 18.5%
+ AuraGym, same 368 tasks5.38 / 23.12 = 23.3%7.32 / 32.93 = 22.2%
+ AuraGym (full)5.91 / 27.96 = 21.1%8.54 / 36.59 = 23.3%

When tasks are matched, synthesized supervision raises the conditional security rate on both languages. With full AuraGym, the Python rate stays comparable while FuncPass rises from 22.04 to 27.96: the additional functionally correct solutions are secure at about the same rate, so the number of secure solutions grows by 22%.


W4/Q4: Separation of Python training and evaluation data.

A4: The Python training and evaluation sets share no vulnerabilities: there is no overlap in CVE IDs or fix commits between AuraGym Python and SusVibes. 38 repositories appear in both, but with different vulnerabilities and fix commits, so no evaluation task is seen during training.


W5/Q5: Contribution of attack-oriented synthesis.

A: Attack-oriented synthesis is what removes the false positives. We ran an ablation on 30 tasks (13 Python, 17 TS/JS), regenerating their security tests with one component removed at a time, and graded all 212 functionally correct solutions (70 secure, 142 vulnerable; labels as in Section 5.1).

  • w/o attack-oriented synthesis: removes the security invariant, attack variants, risk narrative, and the instruction to test the attack rather than the fix. The agent still receives the CVE ID and the historical fix, and can run its tests against the fix.
  • w/o historical fix feedback: still provides the fix as text, but removes the validation step that tells the agent whether its suite distinguishes the vulnerable code from the fixed code.
Test suiteFalse positive rateRecallValid suites
Human-written34.3%99.3%–
AuraForge (full)1.4%99.3%30/30
w/o attack-oriented synthesis28.6%96.5%30/30
w/o historical fix feedback5.7%95.1%28/30

A suite is valid if it fails on the reference vulnerable code and passes on the reference fix.

The two components do different jobs:

  • Attack-oriented synthesis is what removes the false positives. Without it, the false positive rate goes from 1.4% back to 28.6%, close to the human-written suites, even though every suite still passes validation. The tests are written from the fix and reject other correct fixes. The 20 errors are spread over 10 of the 30 tasks.
  • Historical fix feedback is what makes the tests work. Without it, 2 of the 30 suites are unusable (one passes on the vulnerable code, one does not run), and recall drops from 99.3% to 95.1%. The false positive rate stays low because the attack-oriented instruction is still there.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.