- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EviRepair, an evidence-driven multi-agent framework for repository-level automated vulnerability repair (AVR). The framework divides the repair process into two stages: Vulnerability Diagnosis and Patch Synthesis. The former collects evidence (e.g., reachability, reproducibility, security boundaries) via specialized agents and forms a structured Vulnerability Model through consistency review. The latter generates and iteratively refines patches based on this model using a validation matrix and patch review. Evaluated on PatchEval and SEC-bench with three backbone LLMs against Codex and OpenCode, EviRepair achieves the highest overall resolved rates across all configurations, averaging 79.13% and 93.11%, respectively. Stage-level ablations further support the practical value of the two-stage multi-agent design.
Critical strengths
- Practical insights from pairing smaller models with structured workflows: RQ5 shows that EviRepair with Qwen3.8-27B outperforms Codex/OpenCode paired with the larger GLM-5.2 on both benchmarks. This demonstrates that framework and workflow design can substantially impact repair performance, providing positive evidence for using smaller LLMs in complex AVR tasks. Although total cost reduction is not proven, this finding offers practical value for model selection and system design.
- Clear motivation addressing evidence reliability in AVR: Automated repair requires not only patch generation but also ensuring the reliability of vulnerability diagnosis, reproducers, and validation rules. Incorrect or incomplete intermediate evidence can mislead subsequent modifications. By explicitly integrating evidence review into the workflow, this paper makes such risks inspectable and correctable, which is of clear practical significance.
Critical weaknesses
- Uniqueness of methodological innovation is not fully established: EviRepair integrates role specialization, structured diagnostic records, and generation-review-feedback loops into a two-stage process. While reviewing vulnerability evidence has sound engineering motivation, the paper lacks a clear explanation of its key technical increment compared to existing multi-agent analysis and constraint-driven repair methods in related work. The experiments only compare against general coding agents, and the ablation study does not isolate the evidence review mechanism. Thus, the results mainly support the overall pipeline's utility rather than its unique methodological contribution.
- Lack of direct measurement for evidence quality and review effectiveness: The paper relies primarily on the final hidden test pass rate, which does not directly indicate whether intermediate evidence is more accurate. There is no systematic reporting on how many flawed reproducers, unreliable validation rules, or missed security boundaries the reviewer caught and corrected, nor are false rejections and missed detections quantified. A representative case study helps illustrate the workflow but is insufficient to verify the review mechanism's reliability across the dataset. Given that evidence reliability is the core claim, this measurement gap limits the assessment of the mechanism's effectiveness.
Questions for authors
- What is the key technical increment of EviRepair compared to existing multi-agent vulnerability analysis and constraint-driven repair methods? Please clarify which review steps are retained in each variant of Table 2. Are there comparisons that keep role specialization, context isolation, and resource budgets constant but solely remove the evidence consistency review or patch review, to isolate the contribution of the review mechanisms from other pipeline designs?
- Have you conducted independent sampling verifications on the intermediate evidence and review outcomes? For instance, how many flawed reproducers, unreliable validation rules, or missed boundaries were caught and corrected by the reviewer, and how many false rejections or missed detections occurred? What is the correlation between these review corrections and the final hidden test pass rates?
4 comments
Thank you for the detailed response. The additional experiments have addressed some of my earlier concerns.
I still have some reservations about adding separate Reviewer Agents for diagnosis and synthesis. It seems more like an engineering improvement to the workflow than a substantial methodological contribution. Therefore, I prefer to maintain my original score.
We appreciate the reviewer’s reservations regarding our two specialized reviewers. We would like to further elaborate our core design rationale:
-
Prior work mainly focuses on generating patches, with limited attention to reliable patch construction. We introduce evidence review to ensure the validity and consistency of intermediate vulnerability reasoning.
-
Existing systems tend to couple vulnerability analysis and patch synthesis. However, these two stages have fundamentally different objectives. Vulnerability repair naturally follows a two-step logic: we must first establish the root cause of the vulnerability, and only then work out how to resolve it. Our work decouples them into separate phases.
Dear Reviewer oi6h,
We sincerely appreciate the time and care you have invested in reviewing our work. Your comments have been very helpful, and we want to ensure that we have fully and accurately addressed each of your concerns. If any part of our current understanding is incomplete or incorrect, please kindly let us know so we may revise our response accordingly.
If you feel that additional clarification, experiments, or analyses would further strengthen the submission, we would be grateful for your suggestions. Receiving such guidance early would allow us to prepare a more comprehensive and meaningful update during the rebuttal period.
Thank you again for your thoughtful feedback and for contributing to the improvement of our manuscript.
Best regards,
Authors of Submission 6863
We thank the reviewer for recognizing the practical importance of reliable vulnerability evidence and the value of EviRepair’s two-stage design. We address the reviewer’s concerns point by point below, and all changes described here have been incorporated into the revised version of the paper.
Q1: Clarify EviRepair’s technical contributions and the scope of its ablation studies.
A: Technical increment. EviRepair makes two key contributions: (1) Evidence Consistency: Existing agent-based AVR frameworks often lack explicit evidence review, allowing incomplete or inconsistent vulnerability findings to propagate into patch generation. EviRepair addresses this issue by introducing dedicated review mechanisms in both Vulnerability Diagnosis and Patch Synthesis, ensuring that vulnerability understanding is supported by consistent evidence and generated patches are evaluated against established security and compatibility criteria. (2) Agent specialization: Repository-level repair involves complex tasks that can overwhelm individual agents and blur their responsibilities. EviRepair assigns specialized agents distinct roles across two stages: Vulnerability Diagnosis determines what the vulnerability is, while Patch Synthesis determines how to fix it. This division of labor reduces context burden and keeps each agent focused on its assigned tasks.
Specialized method baselines. We added comparisons with the specialized vulnerability-repair methods ContraFixer and AgenticRepair. The results are shown in Table R1. AgenticRepair supports C and C++ only, so it cannot be evaluated on the Go, Python, and JavaScript portions of PatchEval; we mark those entries as “–”.
Table R1: Comparison with Specialized Vulnerability Repair Baselines
Ablation results. To evaluate the above contributions of agent specialization and evidence consistency, we conducted two groups of ablation experiments, as shown in Table R2. The results show that both agent specialization and the evidence-review components contribute to EviRepair’s performance: the single-agent variants and reviewer-removal variants achieve lower overall resolved rates than the full pipeline.
Table R2: Ablation Study of Multi-Agent Specialization and Evidence Review
Q2: Quantify evidence-review outcomes and final verification.
A: We performed an independent manual audit on 100 randomly sampled PatchEval cases processed by EviRepair.
In vulnerability diagnosis, the Evidence Consistency Reviewer correctly caught 24 cases with flawed evidence without false rejections, and fixed all of them after revision, while missing 7 cases of insufficient evidence. In patch synthesis, the Patch Reviewer intercepted 6 problematic cases with no false rejections and missed another 11 cases. 88% of revised patches passed official benchmark verification.
These results show the evidence consistency review yields higher-quality Vulnerability Models for patch synthesis. Better evidence also preserves initial patch quality.