- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work proposes SABER, an agent driven pipeline that converts public vulnerability evidence from NVD and GHSA into repository level repair tasks, each with a reproducible Docker environment, an upstream fix, and both proof of concept and hidden behavioral tests. It applies deterministic postprocessing, four validation gates, and feedback driven repair, producing 22089 tasks across 10163 repositories and more than 60 languages. Fine tuning two Qwen models on filtered trajectories improves vulnerability repair across several benchmarks and languages with up to 11.63 points improvement.
Critical strengths
- The combination of scale and executability addresses a genuine gap. Prior resources were either large but not executable, or executable but limited to C/C++ and small in scale. The authors lay out this tradeoff clearly in Table 1, and SABER checks every box at once, multi-language coverage, repository-level executability, upstream patches, vulnerability verification, and regression testing, a combination that is rare in existing work.
- The validation design is methodologically sound. It separates structural, provenance, semantic, and execution checks into four independent gates, and three of them use independently configured agents or a separate verifier rather than letting the generating agent grade its own output. This reduces the risk of self-scoring and makes the quality control more trustworthy.
- There is a dedicated audit for cheating and evasion. RQ5 reports that runtime isolation neutralizes 99.2% of observed evasion attempts and gives a breakdown in Table 12, addressing the often-overlooked question of whether the training trajectories are contaminated.
- The two representative tasks (Checkmk XSS and Chromium CORS bypass) in Appendix C are illustrative and concrete. They show the full task layout, the before-and-after PoC behavior, and the hidden-test matrix, which makes the construction readable and lets the reader judge task quality directly.
Critical weaknesses
- I noticed that PHP accounts for a relatively large portion of the dataset (40.33%), but the paper does not provide much discussion of this distribution. It would be helpful if the authors could clarify whether this is mainly due to the characteristics of the GHSA ecosystem, differences in project containerization, or limitations of the crawling process. I also wonder whether this imbalance could introduce a bias toward PHP-related repair patterns when models are trained on the resulting dataset.
- One concern I have is the pipeline's reliance on three proprietary models. While I understand that stronger models may be necessary to ensure the quality of generated tasks, it would be interesting to see whether the pipeline remains effective under a more limited budget, perhaps using smaller or open-weight models. From my point of view, this would help readers better understand how much of the improvement comes from the proposed methodology rather than the capabilities of the underlying models. It would also provide useful insights into the reproducibility and practical cost of the pipeline.
- I wonder whether the experimental subsets used in RQ3 to RQ5 are sufficiently representative of the full dataset. These experiments are conducted on approximately 750 to 1,000 samples, while the complete dataset contains around 60,000 entries. Although I understand the computational cost of conducting large-scale ablations, I think reporting confidence intervals or statistical significance tests for the promotion rates and contributions of individual gates would make the conclusions more convincing, particularly when discussing their implications at the full dataset scale.
- I appreciate that the paper considers teacher and student models from different families. However, the two models evaluated in the downstream experiments both belong to the Qwen 3.5 family, which somewhat limits the evidence for generalization across model families. Given that the paper also discusses SKT and its emphasis on cross-model generalization, I think including at least one additional model from a different family would strengthen this claim. Even a smaller-scale experiment on a non-Qwen model could provide useful evidence that the observed improvements are not specific to Qwen.
- I would also appreciate some additional analysis of the targets that were discarded during dataset construction. The paper starts with approximately 60,000 selected targets, but it is not entirely clear how many ultimately failed to become valid tasks or what the main reasons for these failures were. In particular, I wonder whether unsuccessful cases are concentrated in certain programming languages or vulnerability categories. A breakdown of the filtering outcomes would help readers better understand the coverage and potential selection biases of the final dataset.
- I like the inclusion of human validation, as it is an important component of evaluating the quality of automatically constructed tasks. However, some details of the annotation protocol remain unclear, including the number and background of the annotators, whether their assessments were independent, and whether they were blinded to the pipeline's decisions. Given the scale of the dataset, I think the current evaluation of 50 tasks would be more convincing with a clearer description of the validation procedure and, if feasible, additional samples or measures of inter-annotator agreement.
- The paper mentions that task construction relies on Codex with gpt-5.6-sol at maximum reasoning effort, but I would appreciate more details about how the agent makes its decisions throughout the pipeline. In particular, it would be helpful to clarify the operational definitions of "faithful reproduction" and "insufficient evidence," and whether these decisions are supported by explicit criteria or independent checks rather than relying primarily on the model's own judgment. I also wonder whether the authors could briefly explain the choice of Codex and gpt-5.6-sol over alternatives such as Claude Code. A comparison is not necessarily required, but some discussion of the selection rationale and potential limitations would make the methodology more transparent and easier to assess.
Questions for authors
- As mentioned in W2 and W7, how much of the improvement comes from the method itself versus the three proprietary models? Does the pipeline still produce usable tasks with smaller or open-weight models, and why were Codex and gpt-5.6-sol chosen over alternatives such as Claude Code?
- Both evaluated models are from the Qwen3.5 family. Could you add maybe one model from other families base to show the gains are not Qwen specific? (See W4)
- I wonder how are “faithful reproduction” and “insufficient evidence” operationally defined, and are these decisions backed by explicit criteria or independent checks rather than the construction model’s own judgment? (See W7)
- As mentioned in W5, how many of the 60000 targets failed to become valid tasks, and do these failures concentrate in particular languages or vulnerability types?
- Seems to be like a typo, the abstract says 22,089 (Page 1) tasks from 60,000 targets, but the text calls 60,000 a random subset of 197,084. Could you confirm the 22,089 (Page 5 Section 3.2) come from the 60,000, and reword the abstract to avoid implying all 197,084 were used?
7 comments
Q5: Failure Distribution
A5: Figure 2 shows that the dockerization assessment excluded 140,516 of 337,600 targets, including VM-dependent, hardware-dependent, and underspecified cases. This left 197,084 candidates.
Of the 60,000 sampled targets (Figure 3), 22,089 (36.8%) became valid tasks, 33,388 (55.6%) lacked sufficient evidence, and 4,523 (7.5%) exhausted the three-round repair budget. The valid tasks came from initial construction (12,970) and repair rounds 1–3 (6,247, 2,038, and 834). Among 26,614 candidates entering initial validation, 4,309 failed structural checks, 1,509 provenance checks, 9,547 semantic checks, and 7,706 execution checks.
The following table breaks down dockerization outcomes and promoted tasks by primary language. Promotion rates are calculated over all targets that passed the dockerization assessment in each group.
| Language | Dockerization passed | Promoted |
|---|---|---|
| PHP | 47,046/57,050 (82.46%) | 8,908/47,046 (18.93%) |
| C/C++ | 25,325/46,365 (54.62%) | 4,135/25,325 (16.33%) |
| JavaScript | 6,876/7,791 (88.26%) | 2,070/6,876 (30.10%) |
| Java | 8,980/10,608 (84.65%) | 1,866/8,980 (20.78%) |
| Python | 6,529/7,637 (85.49%) | 1,731/6,529 (26.51%) |
Among the languages shown, C/C++ has both the lowest dockerization pass rate and the lowest promotion rate, contributing to differences in language coverage. We added the breakdown to Appendix C.4.
By vulnerability category, the highest failure rates are for Improper Restriction of Operations within the Bounds of a Memory Buffer (CWE-119), SQL Injection (CWE-89), and Use After Free (CWE-416): 92.2% (1,259/1,365), 90.3% (3,439/3,808), and 89.7% (498/555), respectively.
By absolute count, Cross-Site Scripting (CWE-79) and SQL Injection (CWE-89) account for the most failures: 5,027 and 3,439 cases, respectively.
These patterns may partly reflect missing public source code or patch records, particularly for closed-source projects. The prevalence of reported web vulnerabilities may also explain their high absolute failure counts. We added this discussion to Section 4.2.
Q6: Human Review and Data Quality
A6: The original 50-task evaluation was conducted independently by two security and programming experts who were not involved in the SABER pipeline.
Human review focuses on provenance and fidelity: repositories, vulnerable revisions, PoCs, and reference fixes must be supported by publicly available NVD/GHSA evidence.
Hidden tests are intended to filter training trajectories, not to certify exhaustive patch correctness. SABER is a training dataset rather than an evaluation benchmark.
We also expanded the evaluation to 500 additional tasks assessed by 10 domain experts. Each expert reviewed 100 tasks, and every task received at least two independent assessments. The results are:
| Dimension | Pass | Caveat | Fail |
|---|---|---|---|
| Repository and vulnerable revision | 500 | 0 | 0 |
| Upstream fix | 458 | 42 | 0 |
| PoC-based test fidelity | 485 | 15 | 0 |
| Regression and hidden tests | 496 | 4 | 0 |
We used a conservative aggregation rule: a task was labeled Fail if either expert assigned Fail. Caveat if at least one assigned Caveat and neither assigned Fail. Pass only if both assigned Pass.
The 42 Upstream fix Caveats involved patches that included changes beyond the specific vulnerability fix or originated from downstream distribution security patches. These cases are consistent with our expectations.
The 15 PoC-based test fidelity Caveats involved records that described an exploit mechanism without supplying a PoC script. In these cases, the agent reconstructed a PoC from the published mechanism, which we consider acceptable.
The four Regression and hidden tests Caveats concerned overly restrictive conditions that might reject other valid fixes.
No task received a Fail in any dimension. The Caveats identify specific limitations, but the results provide additional evidence of task reliability. We added the evaluation to Appendix C.3.
Q7: Clarification of Definitions and Model Selection
A7: Faithful reproduction means reconstructing a vulnerability from public evidence without fabricating source code, PoCs, or other artifacts. This requirement is specified in the full prompts and assessed through the pipeline's validation checks.
Insufficient evidence refers to a target whose public records do not support a reliable reproduction. We exclude such targets rather than allowing agents to fill gaps with unsupported artifacts.
We selected GPT-5.6-Sol for its strong performance on vulnerability-related tasks at the time of construction. We used Codex as the official harness for GPT-series models. This choice prioritizes compatibility with the selected model.
Q8: Clarification Regarding the Reported Numbers
A8: We thank the reviewer for pointing this out and clarify the numbers as follows. The three numbers refer to different pipeline stages. As described in the Method section, evidence integration and preprocessing precede agent-based task synthesis and validation.
After evidence integration and preprocessing, 197,084 candidate targets remained.
We randomly selected 60,000 of these candidates for task construction under the available time and budget.
Of those 60,000 sampled targets, 22,089 were successfully constructed and validated as tasks.
We have also open-sourced the SABER pipeline, SABER dataset, training trajectories, and fine-tuned model weights. These materials are intended to facilitate verification of the reported results and to benefit the broader community.
Q1: PHP Distribution
A1: To examine the language distribution, we traced major language groups among the 337,600 targets before dockerization:
PHP's high final share appears to reflect both its larger initial target pool and the lower dockerization pass rate of C/C++, rather than a consistent advantage in all pipeline stages. We added this analysis to Appendix C.4.
Despite the high representation of PHP, SABER training also improves non-PHP benchmarks: SEC-bench (C/C++, 6.00% → 20.33%) and Vul4J (Java, 34.11% → 45.74%), as reported in Table 2. The observed benefits are therefore not confined to PHP.
Q2: Construction Cost
A2: Using the stagewise task counts in Figure 3 and recorded processing times, we estimate a total construction effort of approximately 120,000 agent-hours.
Average processing time by stage (in minutes) is shown below:
Using an average throughput of 4,000 tokens per agent-minute, we estimate monetary cost using the following API rates:
Assuming a 1:10:1 ratio of uncached input, cached input, and output tokens, the estimated total cost is $67,200. We added the calculation to Appendix B.1 and released the complete dataset for reuse.
We also evaluated DeepSeek-V4.1-Flash as a lower-cost construction model on the same targets used in Figure 6. Under the same three-hour budget, 60.2% were classified as insufficient evidence, 17.2% were promoted, 2.4% exhausted the repair budget, and 20.2% timed out.
Among promoted tasks, the counts requiring 0–5 revision rounds were 66, 65, 25, 11, 3, and 2 for DeepSeek-V4.1-Flash, versus 201, 105, 30, 21, 4, and 0 for GPT-5.6-Sol.
Under the fixed time budget, the lower-cost model promoted fewer tasks and generally needed more revisions. This highlights a trade-off between model cost and construction throughput. We added the comparison to Appendix B.2.
Q3: Ablation Studies and Contributions of Individual Stages
A3: Figure 3 provides full-production promotion counts: 12,970 tasks (21.6%) passed after initial construction, increasing to 22,089 (36.8%) after three repair rounds.
To complement the 1,000-task ablation, we also summarized each validation gate across all completed production records. This broader analysis provides evidence at the full dataset-construction scale:
These full-production gate statistics are included in Appendix C.6.
Q4: Generalization to Other Model Families
A4: To examine cross-family generalization, we fine-tuned Gemma-4-E4B-it on the same data using the same training configuration. We then compared performance before and after training:
Gemma improves on all reported benchmarks, suggesting that SABER's training benefits are not limited to Qwen models. We added the results to Section 4.1.