acceptodds
Under review as a conference paper at ICLR 2027

SABER: From Vulnerability Reports to Executable Repair Tasks at Scale

Abstract

Training reliable vulnerability-repair agents benefits from executable tasks that verify both security and functionality. However, existing resources remain limited in scale or executability. We introduce (calable gent-ased xecutable epair), a pipeline that turns public vulnerability evidence into repository-level repair tasks, each with a reproducible environment, an upstream fix, and behavioral tests. In particular, first links NVD and GHSA records into deduplicated targets, whose complementary fields fill gaps left by either source alone. A tool-using agent then interprets each target's irregular evidence and synthesizes a candidate repair task. It can adapt to heterogeneous project contexts without project-specific rules or manual reconstruction. Quality control further involves deterministic processing, layered validation, and feedback-driven refinement. Applied to 60,000 targets, promotes 22,089 tasks across 10,163 upstream repositories and more than 60 languages. Sampled ablations demonstrate the benefits of combined evidence, complementary validation gates, and two-layer trajectory filtering. Fine-tuning Qwen3.5-4B and Qwen3.5-35B-A3B on filtered trajectories improves vulnerability-repair performance across multiple programming languages and evaluation criteria.

Primary area:
datasets and benchmarks
Keywords:
vulnerability repair, agent-based data synthesis, executable tasks, training trajectories

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LRsAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes SABER, an agent driven pipeline that converts public vulnerability evidence from NVD and GHSA into repository level repair tasks, each with a reproducible Docker environment, an upstream fix, and both proof of concept and hidden behavioral tests. It applies deterministic postprocessing, four validation gates, and feedback driven repair, producing 22089 tasks across 10163 repositories and more than 60 languages. Fine tuning two Qwen models on filtered trajectories improves vulnerability repair across several benchmarks and languages with up to 11.63 points improvement.

Critical strengths
  1. The combination of scale and executability addresses a genuine gap. Prior resources were either large but not executable, or executable but limited to C/C++ and small in scale. The authors lay out this tradeoff clearly in Table 1, and SABER checks every box at once, multi-language coverage, repository-level executability, upstream patches, vulnerability verification, and regression testing, a combination that is rare in existing work.
  2. The validation design is methodologically sound. It separates structural, provenance, semantic, and execution checks into four independent gates, and three of them use independently configured agents or a separate verifier rather than letting the generating agent grade its own output. This reduces the risk of self-scoring and makes the quality control more trustworthy.
  3. There is a dedicated audit for cheating and evasion. RQ5 reports that runtime isolation neutralizes 99.2% of observed evasion attempts and gives a breakdown in Table 12, addressing the often-overlooked question of whether the training trajectories are contaminated.
  4. The two representative tasks (Checkmk XSS and Chromium CORS bypass) in Appendix C are illustrative and concrete. They show the full task layout, the before-and-after PoC behavior, and the hidden-test matrix, which makes the construction readable and lets the reader judge task quality directly.
Critical weaknesses
  1. I noticed that PHP accounts for a relatively large portion of the dataset (40.33%), but the paper does not provide much discussion of this distribution. It would be helpful if the authors could clarify whether this is mainly due to the characteristics of the GHSA ecosystem, differences in project containerization, or limitations of the crawling process. I also wonder whether this imbalance could introduce a bias toward PHP-related repair patterns when models are trained on the resulting dataset.
  2. One concern I have is the pipeline's reliance on three proprietary models. While I understand that stronger models may be necessary to ensure the quality of generated tasks, it would be interesting to see whether the pipeline remains effective under a more limited budget, perhaps using smaller or open-weight models. From my point of view, this would help readers better understand how much of the improvement comes from the proposed methodology rather than the capabilities of the underlying models. It would also provide useful insights into the reproducibility and practical cost of the pipeline.
  3. I wonder whether the experimental subsets used in RQ3 to RQ5 are sufficiently representative of the full dataset. These experiments are conducted on approximately 750 to 1,000 samples, while the complete dataset contains around 60,000 entries. Although I understand the computational cost of conducting large-scale ablations, I think reporting confidence intervals or statistical significance tests for the promotion rates and contributions of individual gates would make the conclusions more convincing, particularly when discussing their implications at the full dataset scale.
  4. I appreciate that the paper considers teacher and student models from different families. However, the two models evaluated in the downstream experiments both belong to the Qwen 3.5 family, which somewhat limits the evidence for generalization across model families. Given that the paper also discusses SKT and its emphasis on cross-model generalization, I think including at least one additional model from a different family would strengthen this claim. Even a smaller-scale experiment on a non-Qwen model could provide useful evidence that the observed improvements are not specific to Qwen.
  5. I would also appreciate some additional analysis of the targets that were discarded during dataset construction. The paper starts with approximately 60,000 selected targets, but it is not entirely clear how many ultimately failed to become valid tasks or what the main reasons for these failures were. In particular, I wonder whether unsuccessful cases are concentrated in certain programming languages or vulnerability categories. A breakdown of the filtering outcomes would help readers better understand the coverage and potential selection biases of the final dataset.
  6. I like the inclusion of human validation, as it is an important component of evaluating the quality of automatically constructed tasks. However, some details of the annotation protocol remain unclear, including the number and background of the annotators, whether their assessments were independent, and whether they were blinded to the pipeline's decisions. Given the scale of the dataset, I think the current evaluation of 50 tasks would be more convincing with a clearer description of the validation procedure and, if feasible, additional samples or measures of inter-annotator agreement.
  7. The paper mentions that task construction relies on Codex with gpt-5.6-sol at maximum reasoning effort, but I would appreciate more details about how the agent makes its decisions throughout the pipeline. In particular, it would be helpful to clarify the operational definitions of "faithful reproduction" and "insufficient evidence," and whether these decisions are supported by explicit criteria or independent checks rather than relying primarily on the model's own judgment. I also wonder whether the authors could briefly explain the choice of Codex and gpt-5.6-sol over alternatives such as Claude Code. A comparison is not necessarily required, but some discussion of the selection rationale and potential limitations would make the methodology more transparent and easier to assess.
Questions for authors
  1. As mentioned in W2 and W7, how much of the improvement comes from the method itself versus the three proprietary models? Does the pipeline still produce usable tasks with smaller or open-weight models, and why were Codex and gpt-5.6-sol chosen over alternatives such as Claude Code?
  2. Both evaluated models are from the Qwen3.5 family. Could you add maybe one model from other families base to show the gains are not Qwen specific? (See W4)
  3. I wonder how are “faithful reproduction” and “insufficient evidence” operationally defined, and are these decisions backed by explicit criteria or independent checks rather than the construction model’s own judgment? (See W7)
  4. As mentioned in W5, how many of the 60000 targets failed to become valid tasks, and do these failures concentrate in particular languages or vulnerability types?
  5. Seems to be like a typo, the abstract says 22,089 (Page 1) tasks from 60,000 targets, but the text calls 60,000 a random subset of 197,084. Could you confirm the 22,089 (Page 5 Section 3.2) come from the 60,000, and reword the abstract to avoid implying all 197,084 were used?
7 comments

Q1: PHP Distribution

A1: To examine the language distribution, we traced major language groups among the 337,600 targets before dockerization:

LanguageTargets n (%)Dockerization n (%)Crawling (% of URLs)
PHP57,050 (16.90%)47,046 (82.46%)57.96%
C/C++46,365 (13.73%)25,325 (54.62%)55.19%
JavaScript7,791 (2.31%)6,876 (88.26%)69.73%
Java10,608 (3.14%)8,980 (84.65%)62.58%
Python7,637 (2.26%)6,529 (85.49%)62.61%

PHP's high final share appears to reflect both its larger initial target pool and the lower dockerization pass rate of C/C++, rather than a consistent advantage in all pipeline stages. We added this analysis to Appendix C.4.

Despite the high representation of PHP, SABER training also improves non-PHP benchmarks: SEC-bench (C/C++, 6.00% → 20.33%) and Vul4J (Java, 34.11% → 45.74%), as reported in Table 2. The observed benefits are therefore not confined to PHP.

Q2: Construction Cost

A2: Using the stagewise task counts in Figure 3 and recorded processing times, we estimate a total construction effort of approximately 120,000 agent-hours.

Average processing time by stage (in minutes) is shown below:

StageOperationAverage Time (min)
Initial generation (Round 0)Generation with insufficient evidence6.3311
Initial generation (Round 0)Generation entering verification110.8925
Initial generation (Round 0)Semantic verification35.8753
Initial generation (Round 0)Provenance verification22.9315
Revision 1Revision31.9988
Revision 1Semantic verification38.8461
Revision 1Provenance verification23.2910
Revision 2Revision32.6990
Revision 2Semantic verification40.3540
Revision 2Provenance verification23.2022
Revision 3Revision30.4739
Revision 3Semantic verification41.4918
Revision 3Provenance verification23.1856

Using an average throughput of 4,000 tokens per agent-minute, we estimate monetary cost using the following API rates:

  • Uncached input: $4.00 per million tokens
  • Cached input: $0.40 per million tokens
  • Output: $20.00 per million tokens

Assuming a 1:10:1 ratio of uncached input, cached input, and output tokens, the estimated total cost is $67,200. We added the calculation to Appendix B.1 and released the complete dataset for reuse.

We also evaluated DeepSeek-V4.1-Flash as a lower-cost construction model on the same targets used in Figure 6. Under the same three-hour budget, 60.2% were classified as insufficient evidence, 17.2% were promoted, 2.4% exhausted the repair budget, and 20.2% timed out.

Among promoted tasks, the counts requiring 0–5 revision rounds were 66, 65, 25, 11, 3, and 2 for DeepSeek-V4.1-Flash, versus 201, 105, 30, 21, 4, and 0 for GPT-5.6-Sol.

Under the fixed time budget, the lower-cost model promoted fewer tasks and generally needed more revisions. This highlights a trade-off between model cost and construction throughput. We added the comparison to Appendix B.2.

Q3: Ablation Studies and Contributions of Individual Stages

A3: Figure 3 provides full-production promotion counts: 12,970 tasks (21.6%) passed after initial construction, increasing to 22,089 (36.8%) after three repair rounds.

To complement the 1,000-task ablation, we also summarized each validation gate across all completed production records. This broader analysis provides evidence at the full dataset-construction scale:

Validation gateFailed attemptsUnique rejectionFixed at next repair
Structural7,14827.9%59.3%
Provenance2,92431.4%57.1%
Semantic19,74530.7%48.3%
Execution19,75237.7%29.7%

These full-production gate statistics are included in Appendix C.6.

Q4: Generalization to Other Model Families

A4: To examine cross-family generalization, we fine-tuned Gemma-4-E4B-it on the same data using the same training configuration. We then compared performance before and after training:

BenchmarkBefore TrainingAfter Training
PatchEval10.00% / 13.91%11.74% / 15.22%
SEC-bench Patch5.33% / 6.67% / 7.00%7.33% / 12.00% / 13.00%
Vul4J15.50%19.38%
SABER PHP Val.13.00% / 14.00%17.00% / 19.00%

Gemma improves on all reported benchmarks, suggesting that SABER's training benefits are not limited to Qwen models. We added the results to Section 4.1.

Q5: Failure Distribution

A5: Figure 2 shows that the dockerization assessment excluded 140,516 of 337,600 targets, including VM-dependent, hardware-dependent, and underspecified cases. This left 197,084 candidates.

Of the 60,000 sampled targets (Figure 3), 22,089 (36.8%) became valid tasks, 33,388 (55.6%) lacked sufficient evidence, and 4,523 (7.5%) exhausted the three-round repair budget. The valid tasks came from initial construction (12,970) and repair rounds 1–3 (6,247, 2,038, and 834). Among 26,614 candidates entering initial validation, 4,309 failed structural checks, 1,509 provenance checks, 9,547 semantic checks, and 7,706 execution checks.

The following table breaks down dockerization outcomes and promoted tasks by primary language. Promotion rates are calculated over all targets that passed the dockerization assessment in each group.

LanguageDockerization passedPromoted
PHP47,046/57,050 (82.46%)8,908/47,046 (18.93%)
C/C++25,325/46,365 (54.62%)4,135/25,325 (16.33%)
JavaScript6,876/7,791 (88.26%)2,070/6,876 (30.10%)
Java8,980/10,608 (84.65%)1,866/8,980 (20.78%)
Python6,529/7,637 (85.49%)1,731/6,529 (26.51%)

Among the languages shown, C/C++ has both the lowest dockerization pass rate and the lowest promotion rate, contributing to differences in language coverage. We added the breakdown to Appendix C.4.

By vulnerability category, the highest failure rates are for Improper Restriction of Operations within the Bounds of a Memory Buffer (CWE-119), SQL Injection (CWE-89), and Use After Free (CWE-416): 92.2% (1,259/1,365), 90.3% (3,439/3,808), and 89.7% (498/555), respectively.

By absolute count, Cross-Site Scripting (CWE-79) and SQL Injection (CWE-89) account for the most failures: 5,027 and 3,439 cases, respectively.

These patterns may partly reflect missing public source code or patch records, particularly for closed-source projects. The prevalence of reported web vulnerabilities may also explain their high absolute failure counts. We added this discussion to Section 4.2.

Q6: Human Review and Data Quality

A6: The original 50-task evaluation was conducted independently by two security and programming experts who were not involved in the SABER pipeline.

Human review focuses on provenance and fidelity: repositories, vulnerable revisions, PoCs, and reference fixes must be supported by publicly available NVD/GHSA evidence.

Hidden tests are intended to filter training trajectories, not to certify exhaustive patch correctness. SABER is a training dataset rather than an evaluation benchmark.

We also expanded the evaluation to 500 additional tasks assessed by 10 domain experts. Each expert reviewed 100 tasks, and every task received at least two independent assessments. The results are:

DimensionPassCaveatFail
Repository and vulnerable revision50000
Upstream fix458420
PoC-based test fidelity485150
Regression and hidden tests49640

We used a conservative aggregation rule: a task was labeled Fail if either expert assigned Fail. Caveat if at least one assigned Caveat and neither assigned Fail. Pass only if both assigned Pass.

The 42 Upstream fix Caveats involved patches that included changes beyond the specific vulnerability fix or originated from downstream distribution security patches. These cases are consistent with our expectations.

The 15 PoC-based test fidelity Caveats involved records that described an exploit mechanism without supplying a PoC script. In these cases, the agent reconstructed a PoC from the published mechanism, which we consider acceptable.

The four Regression and hidden tests Caveats concerned overly restrictive conditions that might reject other valid fixes.

No task received a Fail in any dimension. The Caveats identify specific limitations, but the results provide additional evidence of task reliability. We added the evaluation to Appendix C.3.

Q7: Clarification of Definitions and Model Selection

A7: Faithful reproduction means reconstructing a vulnerability from public evidence without fabricating source code, PoCs, or other artifacts. This requirement is specified in the full prompts and assessed through the pipeline's validation checks.

Insufficient evidence refers to a target whose public records do not support a reliable reproduction. We exclude such targets rather than allowing agents to fill gaps with unsupported artifacts.

We selected GPT-5.6-Sol for its strong performance on vulnerability-related tasks at the time of construction. We used Codex as the official harness for GPT-series models. This choice prioritizes compatibility with the selected model.

Q8: Clarification Regarding the Reported Numbers

A8: We thank the reviewer for pointing this out and clarify the numbers as follows. The three numbers refer to different pipeline stages. As described in the Method section, evidence integration and preprocessing precede agent-based task synthesis and validation.

After evidence integration and preprocessing, 197,084 candidate targets remained.

We randomly selected 60,000 of these candidates for task construction under the available time and budget.

Of those 60,000 sampled targets, 22,089 were successfully constructed and validated as tasks.

We have also open-sourced the SABER pipeline, SABER dataset, training trajectories, and fine-tuned model weights. These materials are intended to facilitate verification of the reported results and to benefit the broader community.

Reviewer_Nm1iOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(−1)
Read review
Paper summary

SABER introduces an automated pipeline that converts public vulnerability reports into executable repository-level repair tasks. It integrates evidence from NVD and GHSA with sandboxed environments, layered validation, and feedback-driven refinement.

Applied to 60,000 targets, the pipeline produces 22,089 validated tasks across 10,163 repositories and over 60 programming languages. The authors evaluate the construction pipeline and trajectory-filtering strategies through ablation studies. They further demonstrate that fine-tuning two Qwen models on filtered DeepSeek-generated trajectories improves vulnerability-repair performance across multiple programming languages and evaluation criteria.

Critical strengths
  • S1: Substantial scale and practical value. The collection significantly expands executable, real-world vulnerability-repair resources while incorporating both security and functionality checks. This addresses a meaningful bottleneck in training repair agents.

  • S2: Reusable automated construction pipeline. Automating environment reconstruction, task generation, validation, and refinement across heterogeneous projects is a consequential systems contribution. The demonstrated production scale supports its potential for continued expansion.

Critical weaknesses
  • W1: The training experiments do not isolate the dataset’s comparative value. Improvements over the original Qwen models combine the effects of SABER tasks, DeepSeek supervision, and additional training. The filtering ablations usefully control trajectory counts and approximately match token budgets, but do not compare against equally sized trajectories generated by the same teacher on alternative datasets. They therefore demonstrate training utility without establishing SABER’s advantage over other data sources.

  • W2: The cost of further scaling is insufficiently characterized. The pipeline uses multiple high-budget agent calls, repeated validation, and repair attempts, but does not report comprehensive construction costs. This limits assessment of how economically the approach can scale beyond the current release.

Questions for authors
  • Q1: Can you provide a comparison using the same DeepSeek teacher, Qwen student, and training budget with trajectories from an existing vulnerability-repair dataset or general SWE tasks?

  • Q2: What are the total inference and execution costs, average construction time, and amortized cost per accepted task, including failed attempts and refinement rounds?

3 comments

Q1: Training Benefits

A1: We conducted a controlled comparison by training Qwen3.5-4B on general software engineering trajectories from nvidia/Open-SWE-Traces v1.1. We used the same DeepSeek-V4-Flash teacher and training configuration, with comparable trajectory counts and token budgets.

General software engineering training increased SWE-bench performance from 37.67% to 41.67% but produced mixed results on the security benchmarks:

  • PatchEval: 15.65% / 21.74% → 14.35% / 17.39%
  • SEC-bench Patch: 6.00% / 14.00% / 16.67% → 8.67% / 15.00% / 15.00%
  • Vul4J: 34.11% → 28.68%
  • SABER PHP Val.: 13.00% / 17.00% → 14.00% / 15.00%

General software engineering trajectories yield no consistent security-patching gains under this setting. By contrast, SABER training improves performance on the security benchmarks while matching general software engineering training on SWE-bench (41.67% vs. 41.67%). This suggests that SABER supports security-specific learning without sacrificing general software engineering performance.

We also compared SABER with ARVO, a security-related dataset, under the same training setting:

BenchmarkBaseSABERARVO
PatchEval15.65% / 21.74%23.48% / 28.26%18.26% / 23.04%
SEC-bench Patch6.00% / 14.00% / 16.67%20.33% / 36.00% / 37.33%20.67% / 32.00% / 33.33%
Vul4J34.11%45.74%32.56%
SABER PHP Val.13.00% / 17.00%25.00% / 29.00%15.00% / 20.00%

SABER outperforms ARVO on most reported metrics, with a small exception on the first SEC-bench Patch metric. We added both comparisons to Section 4.1.

Q2: Construction Cost

A2: Using the stagewise task counts in Figure 3 and recorded processing times, we estimate a total construction effort of approximately 120,000 agent-hours. The cost breakdown is included in Appendix B.1.

Average processing time by stage (in minutes) is shown below:

StageOperationAverage Time (min)
Initial generation (Round 0)Generation with insufficient evidence6.3311
Initial generation (Round 0)Generation entering verification110.8925
Initial generation (Round 0)Semantic verification35.8753
Initial generation (Round 0)Provenance verification22.9315
Revision 1Revision31.9988
Revision 1Semantic verification38.8461
Revision 1Provenance verification23.2910
Revision 2Revision32.6990
Revision 2Semantic verification40.3540
Revision 2Provenance verification23.2022
Revision 3Revision30.4739
Revision 3Semantic verification41.4918
Revision 3Provenance verification23.1856

Using an average throughput of 4,000 tokens per agent-minute, we estimate monetary cost using the following API rates:

  • Uncached input: $4.00 per million tokens
  • Cached input: $0.40 per million tokens
  • Output: $20.00 per million tokens

Assuming a 1:10:1 ratio of uncached input, cached input, and output tokens, the estimated total cost is $67,200.

The complete dataset is publicly available for reuse.

Thanks to the authors for the detailed responses throughout the discussion. The Section 4.1 comparisons address W1, and the cost breakdown in Appendix B.1 addresses W2. I also appreciate the open release, which made it possible to inspect the tasks more closely.

My main remaining concern stems from the executable audit in Reviewer o5mX's thread, which came up after my initial review. Both that audit and the authors' own random sample show that the verifier accepts roughly a quarter of deliberately broken fixes (several of which remain exploitable) and rejects a similar fraction of valid alternative fixes. I understand that verifier completeness isn't claimed and isn't strictly required for a training set. My concern is narrower: the paper calls every task "behaviorally verified" and RQ5 relies on the graders to separate good fixes from bad ones, yet this non-trivial error rate is never measured or reported. I'm mainly noting this as a soundness and overclaiming issue to address in a revision, not something I expect to be fixed before the discussion closes.

Overall, I still see this as a useful contribution given the scale, multilingual coverage, open release, and solid RQ1 gains, so I am keeping my rating at 3. In light of the verifier issue above, though I am lowering my soundness score to 2. Thanks again for the thoughtful engagement.

We sincerely thank the reviewer for the thoughtful engagement throughout the discussion and for recognizing the value of our contributions.

We appreciate your concerns regarding potentially overstated claims. In the latest revision, we have incorporated the experimental findings and analyses agreed upon during our discussion with Reviewer o5mX into Appendix C.11. We have also revised potentially misleading or overstated wording, replacing "behaviorally verified" with more precise terminology to avoid ambiguity.

We are grateful for your constructive feedback and support, which have helped improve the quality and clarity of our paper.

Reviewer_o5mXOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SABER, a pipeline that turns public vulnerability records (NVD and GHSA) into repository-level vulnerability-repair tasks. A GPT-5.6-sol agent finds the repository, the vulnerable version, and the upstream fix, builds a Docker environment, and writes a PoC test and hidden tests. Each task must pass four checks (structural, provenance, semantic, and execution), with up to three automatic repair rounds. The authors report 22,089 tasks from 10,163 repositories in over 60 languages. They fine-tune Qwen3.5-4B and Qwen3.5-35B-A3B on filtered trajectories and show gains on PatchEval, SEC-bench, and Vul4J.

Critical strengths

The paper addresses an important problem. Large, executable vulnerability-repair datasets are rare, and the authors cover many languages and repositories.

The execution check is a good idea. A task is kept only if its tests fail on the vulnerable code and pass after the real upstream fix is applied.

The authors include many analyses, such as evidence ablations, repair rounds, gate contributions, filtering ablations, and a check for overlap with the evaluation benchmarks.

Critical weaknesses

No artifact to review: The main contribution is a dataset, but the authors did not share any tasks, code, or full prompts. Appendix A only gives a summary of the prompts. Appendix C mentions “published” task packages, but I could not find any link. So I cannot check any of the quality claims.

Doubtful PoC quality: The PoCs and hidden tests are all written by an LLM. Recent studies such as PoCGym and CyberGym [1,2] shows that LLM-written PoCs are often fake or do not reach the vulnerable code. In this paper, the PoCs are checked mainly by other agents from the same model family. The two examples in Appendix C use small custom test programs, not the real attack path. The authors did not report code coverage or test the tests with wrong or partial patches.

Small human check: The authors checked only 50 of 22,089 tasks (about 0.2%), and all 50 passed the PoC check. This seems too clean. The paper does not say who did the check or if more than one person rated each task.

Weak baseline for training: The authors only compare against the base model. Table 2 shows that training on unfiltered trajectories already gives most of the gain (for 4B, SEC-bench Strict goes from 6.00 to 20.00, versus 20.33 with full filtering). So the gains may come from learning from the teacher model, not from SABER itself. A baseline using general software tasks is missing.

Small test sets and one seed: The benchmarks are small (129 to 300 tasks), and the results use one seed with no error bars.

Missing discussion: The paper does not discuss the risk of releasing working PoCs for over 22,000 real CVEs. It also does not report the cost of building the dataset.


[1] Gezgin, Derin, et al. PoC-Gym: Towards More Reliable LLM-Assisted Proof-of-Concept Exploit Generation. In LLM4SE Workshop, co-located with STAF 2026, 2026. arXiv:2602.04165.

[2] Zhun Wang,, et al. CyberGym: Evaluating AI Agents’ Real-World Cybersecurity Capabilities at Scale. In International Conference on Learning Representations (ICLR), 2026.

Questions for authors

Can the authors please share the full set of task files (instructions, PoC scripts, verify wrappers, hidden tests, and patches), without docker images or source checkouts, along with the pipeline code and full prompts? If that is not possible, please share at least 200 tasks chosen at random with a fixed seed, and explain how they were selected ?

Can the authors also share the filtered training trajectories and the training scripts? If possible, please also release the fine-tuned checkpoints (at least the 4B model) so the benchmark results can be checked ?

How often do the PoCs actually run the patched lines? Do the tests still pass if a wrong or partial patch is applied?

How do models trained on SABER compare with models trained on the same amount of general software-task data from the same teacher?

2 comments

Q1: Open-Source Contributions.

A1: Thank you for this valuable suggestion. We have accordingly open-sourced the SABER pipeline, SABER dataset, training trajectories, and fine-tuned model weights. These materials are intended to facilitate verification of the reported results and to benefit the broader community.

Q2: Data quality.

A2: As detailed in Sections 3.2–3.3 and Appendix Table 5, SABER grounds its PoCs in publicly available NVD/GHSA evidence rather than accepting unsupported agent-generated exploits. Agent-fabricated PoCs are rejected during review. Hidden tests serve a different purpose: they are LLM-generated checks used to filter training trajectories and identify functional regressions (Section 3.3; Appendix Table 4). We clarified this distinction in Section 3.3.

Figure 7 shows that PoC tests reject 30.3% of fixes. Of the tasks that pass these tests, 897 (20.8%) fail at least one hidden behavior check. The two checks therefore capture different failure modes, and Table 2 shows the benefit of combining them during training.

Q3: Human check.

A3: The manual review provides an additional check on task provenance and fidelity. Repositories, vulnerable revisions, PoCs, and reference fixes are grounded in public NVD/GHSA records. Hidden tests primarily support training-data filtering rather than exhaustive test coverage. The original 50-task review was conducted independently by two security and programming experts who were not involved in the pipeline. We expanded the evaluation to 500 additional tasks reviewed by 10 domain experts. Each expert assessed 100 tasks, and every task received at least two independent assessments. The results are:

DimensionPassCaveatFail
Repository and vulnerable revision50000
Upstream fix458420
PoC-based test fidelity485150
Regression and hidden tests49640

We adopted a conservative aggregation rule: a task was labeled Fail if either expert assigned Fail. Caveat if at least one assigned Caveat and neither assigned Fail. Pass only if both assigned Pass.

The 42 Upstream fix Caveats involved patches that included changes beyond the specific vulnerability fix or originated from downstream distribution security patches. These cases are consistent with our expectations.

The 15 PoC-based test fidelity Caveats involved records that described an exploit mechanism without supplying a PoC script. In these cases, the agent reconstructed a PoC according to the published mechanism, which we consider acceptable.

The four Regression and hidden tests Caveats concerned overly restrictive conditions that might reject other valid fixes.

No task received a Fail in any dimension. The Caveats identify specific limitations, but the results provide additional evidence of task reliability. We added the evaluation to Appendix C.3.

Q4: Training Benefits

A4: We thank the reviewer for this insightful observation. We agree that filtering yields only a modest additional gain on SEC-bench Strict (20.00 vs. 20.33). However, both filtered and unfiltered trajectories come from SABER tasks. This ablation therefore does not isolate the effect of the task source from that of teacher supervision.

To isolate the effect of task source, we compared SABER against general software engineering trajectories from nvidia/Open-SWE-Traces v1.1. Both sets of trajectories used the same teacher (DeepSeek-V4-Flash) and training configuration, with comparable trajectory counts and token budgets.

General software engineering training increased SWE-bench performance from 37.67% to 41.67% but produced mixed results on the security benchmarks:

  • PatchEval: 15.65% / 21.74% → 14.35% / 17.39%
  • SEC-bench Patch: 6.00% / 14.00% / 16.67% → 8.67% / 15.00% / 15.00%
  • Vul4J: 34.11% → 28.68%
  • SABER PHP Val.: 13.00% / 17.00% → 14.00% / 15.00%

Under this controlled setting, general software engineering trajectories do not consistently improve security patching. This supports the value of SABER-specific task data for the observed security gains. In contrast, SABER matches the general software engineering baseline on SWE-bench and performs better across the reported security benchmarks. We added this comparison to Section 4.1.

Q5: Benchmark scale and Training stability.

A5: We use established vulnerability-repair benchmarks also adopted in prior work (e.g., CyberForge). Their scale reflects the cost of preparing executable environments and verified PoCs for individual vulnerabilities.

To assess sensitivity to training randomness, we repeated training and evaluation with two additional seeds (43 and 44) under the same configuration. The results are:

BenchmarkSeed 42Seed 43Seed 44Mean ± SD
PatchEval23.48% / 28.26%24.78% / 30.00%25.65% / 29.57%24.64 ± 1.09% / 29.28 ± 0.91%
SEC-bench Patch20.33% / 36.00% / 37.33%20.00% / 35.00% / 36.00%22.33% / 39.33% / 41.33%20.89 ± 1.26% / 36.78 ± 2.27% / 38.22 ± 2.77%
Vul4J45.74%41.86%43.41%43.67 ± 1.95%
SABER PHP Val.25.00% / 29.00%29.00% / 33.00%25.00% / 26.00%26.33 ± 2.31% / 29.33 ± 3.51%

Performance is broadly consistent across the three seeds, with the variation summarized by the reported mean and standard deviation. We added these results to Appendix C.8.

Q6: Open-Source risks and Construction cost.

A6: As described in Sections 3.3–3.4, each SABER task must have a documented reference fix. Thus, our release focuses on disclosed vulnerabilities for which a patch is available. We have also added an Ethics Statement addressing the release of executable PoCs.

Using the stagewise task counts in Figure 3 and recorded processing times, we estimate a total construction effort of approximately 120,000 agent-hours.

Average processing time by stage (in minutes) is shown below:

StageOperationAverage Time (min)
Initial generation (Round 0)Generation with insufficient evidence6.3311
Initial generation (Round 0)Generation entering verification110.8925
Initial generation (Round 0)Semantic verification35.8753
Initial generation (Round 0)Provenance verification22.9315
Revision 1Revision31.9988
Revision 1Semantic verification38.8461
Revision 1Provenance verification23.2910
Revision 2Revision32.6990
Revision 2Semantic verification40.3540
Revision 2Provenance verification23.2022
Revision 3Revision30.4739
Revision 3Semantic verification41.4918
Revision 3Provenance verification23.1856

Using an average throughput of 4,000 tokens per agent-minute, we estimate monetary cost using the following API rates:

  • Uncached input: $4.00 per million tokens
  • Cached input: $0.40 per million tokens
  • Output: $20.00 per million tokens

Assuming a 1:10:1 ratio of uncached input, cached input, and output tokens, the estimated cost is $67,200. We added this analysis to Appendix B.1.

We have also open-sourced the SABER pipeline, SABER dataset, training trajectories, and fine-tuned model weights. These materials are intended to facilitate verification of the reported results and to benefit the broader community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.