acceptodds
Under review as a conference paper at ICLR 2027

EviRepair: Evidence-Driven Agents for Reliable Vulnerability Repair

Abstract

Recent advances in large language models (LLMs) have enabled agent-based approaches for repository-level automated vulnerability repair (AVR), showing promising repair capabilities. However, an important issue has received limited attention: **how can we ensure that the evidence generated during the repair process is reliable enough to support accurate vulnerability understanding and patch generation**? In this paper, we propose EviRepair, an evidence-driven multi-agent framework for repository-level AVR. EviRepair introduces evidence review into the repair process by enabling agents to collaboratively collect vulnerability evidence, verify evidence consistency, and construct a reviewed vulnerability model for patch synthesis and validation. Extensive experiments on PatchEval and SecBench benchmarks with multiple agent frameworks and LLMs demonstrate that EviRepair consistently outperforms existing agent-based baselines, achieving average improvements of 16.37% and 40.33% in resolved rates on PatchEval and SecBench, respectively. Codes are available at https://anonymous.4open.science/r/EviRepair-BBB2/.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multi-agent systems, vulnerability repair, program repair, software security, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_oi6hOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EviRepair, an evidence-driven multi-agent framework for repository-level automated vulnerability repair (AVR). The framework divides the repair process into two stages: Vulnerability Diagnosis and Patch Synthesis. The former collects evidence (e.g., reachability, reproducibility, security boundaries) via specialized agents and forms a structured Vulnerability Model through consistency review. The latter generates and iteratively refines patches based on this model using a validation matrix and patch review. Evaluated on PatchEval and SEC-bench with three backbone LLMs against Codex and OpenCode, EviRepair achieves the highest overall resolved rates across all configurations, averaging 79.13% and 93.11%, respectively. Stage-level ablations further support the practical value of the two-stage multi-agent design.

Critical strengths
  1. Practical insights from pairing smaller models with structured workflows: RQ5 shows that EviRepair with Qwen3.8-27B outperforms Codex/OpenCode paired with the larger GLM-5.2 on both benchmarks. This demonstrates that framework and workflow design can substantially impact repair performance, providing positive evidence for using smaller LLMs in complex AVR tasks. Although total cost reduction is not proven, this finding offers practical value for model selection and system design.
  2. Clear motivation addressing evidence reliability in AVR: Automated repair requires not only patch generation but also ensuring the reliability of vulnerability diagnosis, reproducers, and validation rules. Incorrect or incomplete intermediate evidence can mislead subsequent modifications. By explicitly integrating evidence review into the workflow, this paper makes such risks inspectable and correctable, which is of clear practical significance.
Critical weaknesses
  1. Uniqueness of methodological innovation is not fully established: EviRepair integrates role specialization, structured diagnostic records, and generation-review-feedback loops into a two-stage process. While reviewing vulnerability evidence has sound engineering motivation, the paper lacks a clear explanation of its key technical increment compared to existing multi-agent analysis and constraint-driven repair methods in related work. The experiments only compare against general coding agents, and the ablation study does not isolate the evidence review mechanism. Thus, the results mainly support the overall pipeline's utility rather than its unique methodological contribution.
  2. Lack of direct measurement for evidence quality and review effectiveness: The paper relies primarily on the final hidden test pass rate, which does not directly indicate whether intermediate evidence is more accurate. There is no systematic reporting on how many flawed reproducers, unreliable validation rules, or missed security boundaries the reviewer caught and corrected, nor are false rejections and missed detections quantified. A representative case study helps illustrate the workflow but is insufficient to verify the review mechanism's reliability across the dataset. Given that evidence reliability is the core claim, this measurement gap limits the assessment of the mechanism's effectiveness.
Questions for authors
  1. What is the key technical increment of EviRepair compared to existing multi-agent vulnerability analysis and constraint-driven repair methods? Please clarify which review steps are retained in each variant of Table 2. Are there comparisons that keep role specialization, context isolation, and resource budgets constant but solely remove the evidence consistency review or patch review, to isolate the contribution of the review mechanisms from other pipeline designs?
  2. Have you conducted independent sampling verifications on the intermediate evidence and review outcomes? For instance, how many flawed reproducers, unreliable validation rules, or missed boundaries were caught and corrected by the reviewer, and how many false rejections or missed detections occurred? What is the correlation between these review corrections and the final hidden test pass rates?
4 comments

We thank the reviewer for recognizing the practical importance of reliable vulnerability evidence and the value of EviRepair’s two-stage design. We address the reviewer’s concerns point by point below, and all changes described here have been incorporated into the revised version of the paper.

Q1: Clarify EviRepair’s technical contributions and the scope of its ablation studies.

(Weakness1) The paper needs to better explain EviRepair’s technical contributions and distinguish the value of its evidence-review mechanism from that of the overall pipeline.

(Question1) Clarify how EviRepair advances existing vulnerability-repair approaches, which components are removed in each Table 2 variant, and how the review mechanisms’ contributions are isolated.

A: Technical increment. EviRepair makes two key contributions: (1) Evidence Consistency: Existing agent-based AVR frameworks often lack explicit evidence review, allowing incomplete or inconsistent vulnerability findings to propagate into patch generation. EviRepair addresses this issue by introducing dedicated review mechanisms in both Vulnerability Diagnosis and Patch Synthesis, ensuring that vulnerability understanding is supported by consistent evidence and generated patches are evaluated against established security and compatibility criteria. (2) Agent specialization: Repository-level repair involves complex tasks that can overwhelm individual agents and blur their responsibilities. EviRepair assigns specialized agents distinct roles across two stages: Vulnerability Diagnosis determines what the vulnerability is, while Patch Synthesis determines how to fix it. This division of labor reduces context burden and keeps each agent focused on its assigned tasks.

Specialized method baselines. We added comparisons with the specialized vulnerability-repair methods ContraFixer and AgenticRepair. The results are shown in Table R1. AgenticRepair supports C and C++ only, so it cannot be evaluated on the Go, Python, and JavaScript portions of PatchEval; we mark those entries as “–”.

Table R1: Comparison with Specialized Vulnerability Repair Baselines

MethodPatchEval GoPatchEval PythonPatchEval JavaScriptPatchEval OverallSEC-Bench CVESEC-Bench OSS-FuzzSEC-Bench Overall
ContraFix62.65%68.57%66.23%65.65%76.50%78.00%78.00%
AgenticRepair––––81.00%89.00%83.70%
EviRepair69.88%77.14%72.73%73.04%93.00%96.00%94.00%

Ablation results. To evaluate the above contributions of agent specialization and evidence consistency, we conducted two groups of ablation experiments, as shown in Table R2. The results show that both agent specialization and the evidence-review components contribute to EviRepair’s performance: the single-agent variants and reviewer-removal variants achieve lower overall resolved rates than the full pipeline.

Table R2: Ablation Study of Multi-Agent Specialization and Evidence Review

ConfigurationOverall (230)Go (83)Python (70)JavaScript (77)
Multi-Agent Specialization
Single-Agent Diagnosis69.13%67.47%67.14%72.73%
Single-Agent Synthesis67.83%69.88%65.71%67.53%
Single-Agent E2E60.87%49.40%64.29%70.13%
Evidence Consistency
w/o Evidence Consistency Reviewer66.09%66.27%62.86%68.83%
w/o Patch Reviewer69.13%68.67%65.71%72.73%
w/o both reviewers65.65%65.06%68.57%63.64%
Full EviRepair pipeline73.04%69.88%77.14%72.73%

Q2: Quantify evidence-review outcomes and final verification.

(Weakness2) The paper lacks measurements of how often the reviewers catch, correct, or miss evidence and patch issues, including false rejections.

(Question2) Report an independent audit of intermediate evidence and review outcomes, including detected and missed issues, false rejections, and final verification results.

A: We performed an independent manual audit on 100 randomly sampled PatchEval cases processed by EviRepair.

In vulnerability diagnosis, the Evidence Consistency Reviewer correctly caught 24 cases with flawed evidence without false rejections, and fixed all of them after revision, while missing 7 cases of insufficient evidence. In patch synthesis, the Patch Reviewer intercepted 6 problematic cases with no false rejections and missed another 11 cases. 88% of revised patches passed official benchmark verification.

These results show the evidence consistency review yields higher-quality Vulnerability Models for patch synthesis. Better evidence also preserves initial patch quality.

Thank you for the detailed response. The additional experiments have addressed some of my earlier concerns.

I still have some reservations about adding separate Reviewer Agents for diagnosis and synthesis. It seems more like an engineering improvement to the workflow than a substantial methodological contribution. Therefore, I prefer to maintain my original score.

We appreciate the reviewer’s reservations regarding our two specialized reviewers. We would like to further elaborate our core design rationale:

  1. Prior work mainly focuses on generating patches, with limited attention to reliable patch construction. We introduce evidence review to ensure the validity and consistency of intermediate vulnerability reasoning.

  2. Existing systems tend to couple vulnerability analysis and patch synthesis. However, these two stages have fundamentally different objectives. Vulnerability repair naturally follows a two-step logic: we must first establish the root cause of the vulnerability, and only then work out how to resolve it. Our work decouples them into separate phases.

Dear Reviewer oi6h,

We sincerely appreciate the time and care you have invested in reviewing our work. Your comments have been very helpful, and we want to ensure that we have fully and accurately addressed each of your concerns. If any part of our current understanding is incomplete or incorrect, please kindly let us know so we may revise our response accordingly.

If you feel that additional clarification, experiments, or analyses would further strengthen the submission, we would be grateful for your suggestions. Receiving such guidance early would allow us to prepare a more comprehensive and meaningful update during the rebuttal period.

Thank you again for your thoughtful feedback and for contributing to the improvement of our manuscript.

Best regards,

Authors of Submission 6863

Reviewer_dMR6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper propses EVIREPAIR, an evidence-driven multi-agent framework for repository-level vulnerability repair. EVIREPAIR define evidence as Root Cause, Attack Path, Runtime Oracle, Security Boundary and Preserved Behaviors, and introduces evidence review into the repair process by enabling agents to collaboratively collect vulnerability evidence, verify evidence consistency, for patch synthesis and validation. Experiments on PatchEval and SecBench benchmarks with different underlying LLMs showed the proposed agent pipeline could outperfrom general purpose agent pipeline such as codex and opencode.

Critical strengths
  1. Overall, the paper is well-written and easy to follow, although some details could be clarified.

  2. The problem studied is important and significant.

  3. The evaluation results are promising, although I have some concerns about the experimental setup.

Critical weaknesses
  1. The definition of “evidence” is unclear. Is the evidence generated through LLM reasoning with a specific prompt, or gathered using external analysis tools?

  2. The evaluation baselines are not specifically designed for vulnerability or bug repair.

  3. The token consumption of each method is not reported, making it difficult to compare their efficiency and practical costs.

Questions for authors
  1. Could you clarify how “evidence” is obtained? Is it generated through LLM reasoning using a specific prompt, or collected through external analysis tools? Please describe the process and provide an example.

  2. Why did you choose baselines that are not specifically designed for vulnerability or bug repair? Could you include a comparison with methods tailored to these tasks?

  3. Could you report the token consumption of each method? This would help readers compare their efficiency and practical costs.

  4. What is the ablation of each item in the evidence?

  5. What is the key technique contribution of this work?

9 comments

We thank the reviewer for recognizing the importance of reliable repository-level vulnerability repair and evidence validation, and for the constructive comments that helped us clarify EviRepair’s design and evaluation. All changes described below have been incorporated into the revised version of the paper.

Q1: Clarify how evidence is collected and provide an example.

(Weakness1) The definition of “evidence” is unclear. Is the evidence generated through LLM reasoning with a specific prompt, or gathered using external analysis tools?

(Question1) Could you clarify how “evidence” is obtained? Is it generated through LLM reasoning using a specific prompt, or collected through external analysis tools? Please describe the process and provide an example.

A: In EviRepair, evidence is generated through the combination of LLMs and execution tools. The LLM decides what evidence needs to be collected, tools conduct repository-level execution to get raw results, and the LLM integrates these outputs into a full evidence package. Taking a path-traversal vulnerability as an example: the LLM identified that we needed to check unsafe path concatenation and test the ../ payload. Tools then analyzed the source code and ran the corresponding requests. The LLM combined the code inspection and runtime test results, confirming the service could read files outside its designated directory, thereby establishing the existence of this path-traversal vulnerability.

Q2: Compare EviRepair with methods designed for vulnerability repair.

(Weakness2) The evaluation baselines are not specifically designed for vulnerability or bug repair.

(Question2) Why did you choose baselines that are not specifically designed for vulnerability or bug repair? Could you include a comparison with methods tailored to these tasks?

A: We have added comparisons with ContraFix and AgenticRepair, methods specifically designed for vulnerability repair. AgenticRepair supports C and C++ only, so it cannot be evaluated on the Go, Python, and JavaScript portions of PatchEval; we mark those entries as “–”. The results are:

MethodPatchEval GoPatchEval PythonPatchEval JavaScriptPatchEval OverallSEC-Bench CVESEC-Bench OSS-FuzzSEC-Bench Overall
ContraFix62.65%68.57%66.23%65.65%76.50%78.00%78.00%
AgenticRepair––––81.00%89.00%83.70%
EviRepair69.88%77.14%72.73%73.04%93.00%96.00%94.00%

EviRepair outperforms these specialized methods on the reported benchmarks while supporting multiple programming languages.

Q3: Report token consumption.

(Weakness3) The token consumption of each method is not reported, making it difficult to compare their efficiency and practical costs.

(Question3) Could you report the token consumption of each method? This would help readers compare their efficiency and practical costs.

A: We have added average token consumption per task for each method. EviRepair’s evidence collection and review can require more tokens than general-purpose agent frameworks, reflecting a trade-off between resource consumption and repair performance.

MethodBackbone modelPatchEval tokens (M/task)SecBench tokens (M/task)
CodexDeepSeek-v4.1-flash2.265.66
CodexGLM-5.23.588.94
CodexQwen-3.8-27B1.84.1
EviRepairDeepSeek-v4.1-flash5.094.85
EviRepairGLM-5.26.25.95
EviRepairQwen-3.8-27B6.7511.33

Q4: Ablate the evidence components.

(Question4) What is the ablation of each item in the evidence?

A: We added PatchEval ablations with GLM-5.2, separately removing the Evidence Consistency Reviewer, the Patch Reviewer, and both reviewers:

ConfigurationOverall (230)Go (83)Python (70)JavaScript (77)
Full EviRepair pipeline73.04%69.88%77.14%72.73%
w/o Evidence Consistency Reviewer66.09%66.27%62.86%68.83%
w/o Patch Reviewer69.13%68.67%65.71%72.73%
w/o both reviewers65.65%65.06%68.57%63.64%

Removing either reviewer reduces the overall resolved rate, and removing both leads to the largest overall drop. These results demonstrate the effectiveness of the evidence review stages.

Q5: Explain the key technical contribution.

(Question5) What is the key technique contribution of this work?

A: EviRepair makes two key contributions: (1) Evidence Consistency: Existing agent-based AVR frameworks often lack explicit evidence review, allowing incomplete or inconsistent vulnerability findings to propagate into patch generation. EviRepair addresses this issue by introducing dedicated review mechanisms in both Vulnerability Diagnosis and Patch Synthesis, ensuring that vulnerability understanding is supported by consistent evidence and generated patches are evaluated against established security and compatibility criteria. (2) Agent specialization: Repository-level repair involves complex tasks that can overwhelm individual agents and blur their responsibilities. EviRepair assigns specialized agents distinct roles across two stages: Vulnerability Diagnosis determines what the vulnerability is, while Patch Synthesis determines how to fix it. This division of labor reduces context burden and keeps each agent focused on its assigned tasks.

I thank the authors for their response and the additional experiments, which addressed my concern about the definition of evidence. Two concerns remain.

  1. Baseline coverage. The authors compare their method with a vulnerability repair agent, but it remains unclear what prevents them from evaluating AgenticRepair on Go, Python, and JavaScript. Could the authors clarify this limitation and explain why they excluded general-purpose bug repair agents such as SWE-agent and mini-SWE-agent? These comparisons would help assess how much the proposed method improves on existing repair agents.
  2. Token efficiency. The proposed method appears to consume substantially more tokens than the baseline, in some cases roughly twice as many. This makes it difficult to determine how much of the reported improvement comes from the method itself and how much comes from the larger token budget. Given these remaining concerns, I am maintaining my original negative score.

Comments on the paper

7 comments

We thank the reviewer for recognizing EviRepair’s practical value, including its evidence-based design and performance across model backbones. We respond to the concerns point by point below. All changes described here have been incorporated into the revised version of the paper.

Q1: Comparison with specialized vulnerability-repair methods.

(Weakness1) The baseline comparison relies mainly on general-purpose coding agents, leaving the results against specialized vulnerability-repair methods unclear.

A: We added comparisons with ContraFix and AgenticRepair. AgenticRepair supports C and C++ only, so it cannot be evaluated on the Go, Python, and JavaScript portions of PatchEval; those entries are marked “–”. The results are shown below.

MethodPatchEval GoPatchEval PythonPatchEval JavaScriptPatchEval OverallSEC-Bench CVESEC-Bench OSS-FuzzSEC-Bench Overall
ContraFix62.65%68.57%66.23%65.65%76.50%78.00%78.00%
AgenticRepair––––81.00%89.00%83.70%
EviRepair69.88%77.14%72.73%73.04%93.00%96.00%94.00%

EviRepair outperforms these methods on the reported benchmarks and supports multiple programming languages.

Q2: Report computational costs.

(Weakness2) The paper does not report token usage or other cost measures, making it difficult to assess EviRepair’s practical cost.

(Question3) What are the average token counts, model calls, wall-clock time, and cost per task for EviRepair, OpenCode, and Codex?

A: Due to the limited time available for this revision, we prioritized reporting the cost comparison between Codex and EviRepair; cost measurements for OpenCode are unavailable. We report per-task wall-clock time and token usage for both methods, but not model-call counts or monetary cost.

MethodBackbone modelPatchEval time (min/task)PatchEval tokens (M/task)SecBench time (min/task)SecBench tokens (M/task)
CodexDeepSeek-v4.1-flash24.72.26435.66
CodexGLM-5.223.63.58508.94
CodexQwen-3.8-27B68.71.893.44.1
EviRepairDeepSeek-v4.1-flash66.75.0941.74.85
EviRepairGLM-5.242.36.288.85.95
EviRepairQwen-3.8-27B93.76.75125.111.33

Overall, evidence collection and review increase EviRepair’s cost but improve repair performance, reflecting a trade-off. However, runtime and token use can diverge: on SecBench with GLM-5.2, EviRepair’s longer runtime but lower token use likely reflects time spent on tool execution rather than LLM generation.

Q3: Clarify the identical Single-Agent E2E result and the reported performance calculations.

(Weakness3) The identical Single-Agent E2E and OpenCode + GLM-5.2 results, as well as the reported average improvements and unresolved-task reductions, need clarification.

(Question1) Is Single-Agent E2E in Table 2 effectively the same system as OpenCode + GLM-5.2 in Table 1? If so, state this explicitly.

(Question2) How were the reported 16.37% and 40.33% improvements calculated?

A: Identical Single-Agent E2E result. Single-Agent E2E uses the same system as OpenCode + GLM-5.2, which is why their results are identical. We have clarified this relationship in the revised paper.

Corrected average improvements. The reported improvements were incorrectly calculated: we summed the average gains over Codex and OpenCode without dividing by two. On PatchEval, the average improvements over Codex and OpenCode are:

(89.13−78.26)+(73.04−67.39)+(75.22−64.35)3=9.13\frac{(89.13-78.26)+(73.04-67.39)+(75.22-64.35)}{3}=9.13\\%

(89.13−87.83)+(73.04−60.87)+(75.22−66.96)3=7.24\frac{(89.13-87.83)+(73.04-60.87)+(75.22-66.96)}{3}=7.24\\%

We mistakenly added these values to obtain 16.37%. The corrected average improvement is:

\frac{9.13\\%+7.24\\%}{2}=8.19\\%

Using the same corrected calculation, the average improvement on SEC-Bench is 20.17%, rather than 40.33%. We sincerely apologize for this calculation error and for the confusion it caused. We have corrected the reported values in the revised paper and clarified how each metric is calculated.

Relative reductions in unresolved tasks. The reported 26% and 59% are relative reductions in unresolved tasks compared with OpenCode, the stronger average baseline. On PatchEval, OpenCode and EviRepair have average resolved rates of 71.89% and 79.13%, respectively:

(1−0.7189)−(1−0.7913)1−0.7189×100\frac{(1-0.7189)-(1-0.7913)}{1-0.7189}\times100\\%=25.76\\%\approx26\\%

Applying the same calculation to SEC-Bench, using the average resolved rates of 83.33% for OpenCode and 93.11% for EviRepair, gives 58.67%, which rounds to 59%.

Q4: Clarify the ablations.

(Weakness4) The original ablations do not directly measure evidence-review outcomes or the effect of the review components.

Evidence-Review Ablation. We added dedicated ablation experiments for the evidence-related modules in both the vulnerability diagnosis and patch synthesis stages, with the results shown in the table below.

ConfigurationOverall (230)Go (83)Python (70)JavaScript (77)
Full EviRepair pipeline73.04%69.88%77.14%72.73%
w/o Evidence Consistency Reviewer66.09%66.27%62.86%68.83%
w/o Patch Reviewer69.13%68.67%65.71%72.73%
w/o both reviewers65.65%65.06%68.57%63.64%

Removing either reviewer reduces the overall resolved rate, and removing both leads to the largest overall drop, showing that both evidence-review components contribute to performance.

Manual Audit. We also conducted a manual audit of 100 randomly sampled PatchEval cases. The results are summarized below.

Review StageRejection FrequencyRejection AccuracyPost-revision Success Rate
Vulnerability Diagnosis24%100%100%
Patch Synthesis6%100%88%

The diagnosis-stage results show that the Evidence Consistency Reviewer accurately intercepts problematic evidence and substantially improves its quality after revision. With more reliable Vulnerability Models as input, only a small proportion of initial patches contain quality issues. The problematic patches identified by the Patch Reviewer are also substantially improved after revision, with 88% passing the official validation.

Q5: Report run-to-run variation.

(Weakness5) The paper does not clearly report the number of runs, making it difficult to assess whether small performance differences are reliable.

(Question6) How many independent runs were used, and can you report variance for close comparisons?

A: Considering the computational cost, we used one run per configuration in the main experiments. To further examine the numerical differences between closely performing methods, we conducted three independent runs on PatchEval using OpenCode + DeepSeek-v4.1-Flash and EviRepair + DeepSeek-v4.1-Flash. OpenCode achieved an average resolved rate of 85.07% with a standard deviation of 2.40%, while EviRepair achieved 89.71% with a standard deviation of 1.00%. These repeated results suggest that EviRepair maintains a consistent advantage in this close comparison.

Q6: Discuss the reliability of benchmark-test outcomes.

(Weakness6) Passing benchmark tests may not guarantee that a patch correctly fixes the underlying vulnerability.

We manually examined 100 randomly sampled PatchEval cases and found only 2 false rejections by the official tests, corresponding to a 2% misjudgment rate. The two cases resulted from different mismatches between the tested behavior and the target vulnerability. These results suggest that the benchmark metric is generally reliable.

Q7: Discuss possible benchmark contamination.

(Weakness7) Historical vulnerabilities or fixes may have appeared in model training data, so benchmark contamination cannot be ruled out.

We acknowledge that the evaluated models may have encountered some historical vulnerabilities or fixes during training, so benchmark contamination cannot be completely ruled out. Our main comparisons use the same underlying model and evaluation setting for all methods. Therefore, although contamination may affect the absolute resolved rates, the relative performance differences remain a controlled and fair comparison.

Q8: Report time-limit outcomes.

(Question4) What fraction of EviRepair tasks hit the diagnosis or synthesis time limit?

So far, none of the samples in our experiments have exceeded the time limit.

Q9: Clarify whether all methods had equal tool access.

(Question5) Did OpenCode and Codex receive the same reproduction and runtime tools as EviRepair?

OpenCode and Codex were provided with the same vulnerability reproducers and runtime tools as EviRepair. All methods were evaluated in the same isolated environment with identical benchmark instances. Therefore, the performance differences are attributable to the repair workflows rather than differences in tool access.

I appreciate the authors' detailed rebuttal and the additional experiments. The response addresses several of my main concerns and improves the empirical evidence supporting EviRepair. In particular, the inclusion of ContraFix and AgenticRepair provides a more relevant comparison with specialized vulnerability-repair approaches. The new ablation results also provide stronger evidence for the contribution of the review components: removing the Evidence Consistency Reviewer reduces the overall resolved rate from 73.04% to 66.09%, while removing the Patch Reviewer reduces it to 69.13%. The authors have also clarified that Single-Agent E2E is identical to OpenCode + GLM-5.2 and corrected the previously reported average improvements from 16.37% and 40.33% to 8.19% and 20.17%, respectively. These corrections resolve my concerns about the numerical inconsistencies.

However, some concerns remain only partially addressed. First, although the specialized baseline comparisons are useful, the authors should clarify whether ContraFix and AgenticRepair were evaluated with comparable backbone models, computational budgets, and experimental configurations. Second, the efficiency analysis remains incomplete. For example, on PatchEval with DeepSeek-v4.1-Flash, EviRepair consumes 5.09M tokens and 66.7 minutes per task, compared with 2.26M tokens and 24.7 minutes for Codex. This is a substantial overhead that should be considered when interpreting the performance improvements. The absence of OpenCode cost measurements and model-call counts also limits the comparison. Third, the additional three-run experiment is helpful, but it covers only one backbone and benchmark. The reported results (89.71% ± 1.00% versus 85.07% ± 2.40%) suggest an advantage, but broader conclusions about statistical reliability remain limited.

The manual audit is another useful addition, although its methodology requires clarification. The reported 100% rejection accuracy and post-revision success rates of 100% and 88% need clearer definitions, denominators, and validation criteria, particularly given that only 6% of the 100 sampled cases were reportedly rejected during patch synthesis. More importantly, identifying two false rejections by benchmark tests does not establish how often patches that pass those tests still contain the original vulnerability or introduce incorrect behavior. The benchmark-contamination concern also remains unresolved, although I acknowledge that a full contamination analysis may be difficult. Overall, the rebuttal substantially strengthens the paper and addresses several important weaknesses. I am more positive about the work than in my original review, but would still like the authors to clarify these remaining methodological issues and appropriately qualify their claims about efficiency and repair reliability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.