acceptodds
Under review as a conference paper at ICLR 2027

DAVBench: Benchmarking Agentic Verification of Real-World Data Work

Abstract

Data work addresses user-specified data needs, spanning data engineering (DE), which builds and transforms datasets and data pipelines, and data analysis (DA), which derives metrics and analytical results. Reliable verification is critical for preventing faulty transformations and flawed analytical conclusions before candidate data work is delivered. Yet existing benchmarks largely evaluate agents' ability to complete data tasks, while overlooking their ability to verify whether candidate data work satisfies user requirements and is free of defects. We introduce **DAVBench** (Data Agentic Verification Benchmark), comprising **216 repository-level verification tasks of data work** with reproduced real-world defects across data engineering pipelines, analytical workflows, and delivered artifacts. Tasks involve connecting evidence across data, code, and domain documentation, tracing defects through multi-stage workflows, and distinguishing faulty from correct candidates. We evaluate verification along two dimensions: ***(1) verification quality***, which measures the accuracy and completeness of verification reports using location matching and evidence-grounded semantic judging; and ***(2) repair utility***, which measures how effectively these reports help a downstream model correct faulty candidate work under behavioral tests. Our empirical results show that current agents still struggle to verify real-world data work reliably. Even the flagship model GPT-6 Astra achieves a finding F1 of only **49.82%** and complete defect coverage on just **39.61%** of faulty tasks. Further analysis reveals that verification quality is a key bottleneck for correcting data work. When given reference verification reports, a downstream repair model successfully repairs **69.48%** of tasks, compared with **24.03%** without reports and only **37.66%** with reports from the best-performing verifier. Overall, DAVBench provides a testbed for improving the verification of real-world data work.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, verification, data engineering, data analysis, benchmarks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Tp8kOpenReview ↗
Rating
3
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

DAVBench tests whether agents can find defects in completed data work. It contains 216 tasks from 102 repositories, including 154 faulty and 62 correct cases, with 389 reference defects. Eight models are evaluated on verification and a repair experiment tests whether their reports help another agent fix the work.

Critical strengths
  1. Imho, verification is a useful target in its own right. The executable repository context and correct cases make the benchmark more informative than simply asking whether an agent can spot a seeded bug. Including correct work also exposes agents that raise alarms too freely.

  2. The construction and validation effort is substantial. Human refinement expands the reference defects from 239 to 389 while the independent matching check covers 245 report-reference pairs. The repair experiment adds practical value: the best model report raises repair success from 24.03% without a report to 37.66% while reference reports reach 69.48%.

Critical weaknesses
  1. Precision depends on the reference defects being sufficiently complete. In the paper, an unmatched finding is treated as a false positive but a verifier might discover a valid defect outside the reference set. The reported pairwise validation checks matching quality and so it does not fully answer this completeness question. Since the human refinement already uncovered many extra defects, an audit of unmatched model findings seems particularly important.

  2. The repair results establish that reports can help but not that verification is the best use of an equal total budget. The report condition includes an earlier verification pass whereas the no-report condition lacks that extra work. I would keep the claim to report usefulness unless a control gives the repair agent the same additional compute. Also, the repair experiment covers faulty cases, hence it does not measure damage from unnecessary repairs to correct work.

Questions for authors
  1. Clarification: were unmatched model findings independently checked for valid defects missing from the frozen reference set? If so, how often did that happen and did correcting those cases change the model ranking?

  2. Clarification or additional evidence: is there an equal-total-budget repair control? Without one, can the practical conclusion explicitly distinguish the benefit of diagnostic information from the benefit of spending more compute?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Tp8k (W1 & Q1)

Thank you for your thoughtful review, and for recognizing (1) data work verification as an important research problem, (2) the executable, repository-level setting with correct candidates that goes beyond spotting bugs in code, (3) and the rigorous construction and practical value of the repair experiment.

Below, we address your concerns and questions.

W1: Precision depends on the reference defects being sufficiently complete. In the paper, an unmatched finding is treated as a false positive but a verifier might discover a valid defect outside the reference set. The reported pairwise validation checks matching quality and so it does not fully answer this completeness question. Since the human refinement already uncovered many extra defects, an audit of unmatched model findings seems particularly important.

Q1: Clarification: were unmatched model findings independently checked for valid defects missing from the frozen reference set? If so, how often did that happen and did correcting those cases change the model ranking?

Thank you for this thoughtful question about unmatched model findings! Yes, we checked a sample of unmatched findings: few of them identify missing defects, and correcting these cases leaves all model rankings unchanged (details below). The more important point is that DAVBench does not claim that reference defects are exhaustive, which is also difficult to guarantee in real-world settings. Instead, its goal is to reliably evaluate verification ability, capture this real-world research problem, distinguish verifiers, expose where they fail, and thus provide insights for improvement. These goals are supported by our benchmark design principles:

  1. Defective data workflows reconstructed from real-world projects. Defective workflows and deliverables are based on real-world defects rather than defects fabricated by LLMs, so the tasks capture real-world data verification challenges. Reference defects are grounded in real project feedback and fixes, and only substantial defects that cause errors in data workflows are retained; trivial issues, such as purely textual or documentation-only changes, are excluded (Table 19).
  2. Independent review during construction. Before the reference set is frozen, two additional annotators independently complete each task and review it against the task-quality checklist, and supported new defects are added to the reference findings (Section 2.3, Appendix E.4). This step expands the 239 source-grounded defects to 389.
  3. Complementary repair-based evaluation. Beyond finding-based scoring, we evaluate whether a report helps a repair agent fix the candidate, using behavioral tests grounded in project tests and documented fixes to check whether defects are fixed and correct behavior is preserved. This evaluation does not depend on matching findings to reference defects, and it is consistent with the finding-based evaluation: Finding F1 is strongly correlated with repair success for both repair agents (r = 0.92 and 0.96, Section 3.4).

Independent review of unmatched model findings. However, we also conducted an independent review of 70 sampled unmatched findings across 23 faulty tasks, selected to cover both DE and DA and different sources and models, as you suggested. A finding counts as valid only if it violates the stated user requirement, falls within the scope of the candidate, is supported by project evidence, and has a material effect on correctness, results, or required behavior. Here is the result:

Audit outcomeCount (%)Details
Valid findings4 (5.7%)Correspond to 3 distinct defects missing from the reference set
Invalid findings42 (60.0%)More than half concern documentation, comments, or formatting
Insufficient evidence24 (34.3%)Do not meet the evidence standard and are not counted as valid

Most of the reviewed unmatched findings do not identify missing defects: many concern trivial issues such as documentation, comments, or formatting, or lack sufficient evidence.

After adding the three defects to the reference set and rescoring all eight verifiers, the rankings by precision, Finding F1, and All-found Rate remain unchanged, and the three metrics change by at most 0.32, 0.26, and 0.65 percentage points, respectively.

In the revised manuscript, we add this audit and the rescoring results to Appendix B.3 (Table 12; changes marked in blue).

AuthorsOpenReview ↗

Response to Reviewer Tp8k (W2 & Q2)

W2: The repair results establish that reports can help but not that verification is the best use of an equal total budget. The report condition includes an earlier verification pass whereas the no-report condition lacks that extra work. I would keep the claim to report usefulness unless a control gives the repair agent the same additional compute. Also, the repair experiment covers faulty cases, hence it does not measure damage from unnecessary repairs to correct work.

Q2: Clarification or additional evidence: is there an equal-total-budget repair control? Without one, can the practical conclusion explicitly distinguish the benefit of diagnostic information from the benefit of spending more compute?

Thank you for this question about compute trade-off between verifier and repairer! Actually, we do not claim verification with repair is the best use of an equal total budget, and our experiments and conclusions also do not rely on this control. The main purpose of repair-utility experiment is to evaluate the utility of verification reports from the perspective of downstream repair and to compare models.

The no-report condition just serves as a baseline to measure the repair actor’s unaided performance. To ensure fair comparisons among verifiers, we use the same per-task verification time budget and hold the downstream repair actor and its budget fixed across report conditions.

However, allocating compute between verification and repair is also a very interesting research direction! We will explore this as a future work. Thank you for your insightful question.


The question about the unnecessary repair on correct candidates. We further sampled 19 correct candidates on which the GPT-5.6-Sol verifier reported findings and evaluated repairs by GPT-5.6-Sol with and without those reports (Appendix C.4, Table 18):

ConditionCandidates passing all testsPass rate
Original candidates, unchanged19/19100%
Repair without a report16/1984.21%
Repair with the findings report14/1973.68%

These results illustrate the risk of unnecessary repairs to correct work.

Actually, in our faulty candidate tasks, our P2P tests could also check whether repair damages previously correct behavior (Table 16). Table 17 shows an association between unmatched findings and P2P failures, where the P2P failures get worse with more unmatched findings:

Repair actor0 unmatched findings≥1 unmatched finding≥5 unmatched findings
GPT-5.6-Sol3.76%9.08%27.03%
Qwen 3.8-27B2.34%7.65%22.97%

We have added these analyses to Appendix C.4 of the revised manuscript (Table 18; changes marked in blue). Thank you!

Thanks for the extra checks. The audit and unchanged rankings largely address my concern about missing defects. Also, the new experiment helps clarify the risk of unnecessary repairs. I’m also happy with limiting the repair claim to report usefulness. Please make clear in the paper that total compute is not matched and some audit findings remain inconclusive. Most of my concerns are now addressed so I’ll increase my soundness score and informativeness to 3.

AuthorsOpenReview ↗

Response to Reviewer Tp8k

We sincerely thank you for your time and for raising the Soundness and Informativeness scores. Your thoughtful comments have greatly helped us improve the paper. As you suggested, we will make your mentioned points in the final paper version. If any concerns remain, we would be very glad to address them. Thanks again!

Reviewer_q77pOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a benchmark with 216 candidates to evaluate agentic verification. The benchmark primarily focuses on data engineering (DE) and data analysis (DA) workflows and includes both faulty and correct candidates. The authors evaluate the performance of several frontier models on both verification and repair tasks.

Critical strengths
  • The motivation of the paper is clear to me. The authors primarily focus on data engineering and analysis workflows, which are very important domains.

  • The benchmark construction seems reasonable to me. The authors provide executable environments and perform rigorous evaluation, which should be appreciated. They also include both faulty and correct candidates to allow a more robust evaluation.

  • The results are promising for both verification and repair. The authors evaluate several models, which strengthens the experimental results.

Critical weaknesses
  • I am a bit confused by the term "verification". In program analysis, verification usually has a different meaning. In this work, I do not consider the task to be verification in the traditional sense. The authors primarily perform some form of defect localization and classification with natural language explanations, which seems closer to code review. I would recommend finding a better term that suits this work or explaining more clearly how it differs from existing code review tasks.

  • The novelty of this work is relatively low. The agents and other components seem basic. If I look at the novelty, the main contribution is essentially a benchmark for the specific DE/DA domain. Apart from the domain, I do not see a major difference from existing benchmarks such as SWE-bench and other code review benchmarks. From a novelty perspective, I do not feel very excited about this work.

  • Although the authors provide an analysis of missed defects in Section 3.3, I would appreciate a more detailed qualitative analysis with concrete examples showing why agents fail even after accessing the relevant evidence. Such examples could help identify specific limitations and inspire future research.

  • Since the benchmark reconstructs historical defects from publicly available GitHub repositories and pull requests, data contamination and model memorization are important concerns. Although the authors discuss some mitigation measures, it is unclear whether these measures are sufficient to address models' possible prior exposure to the original bugs and fixes. An empirical analysis of contamination risk would strengthen the benchmark.

Questions for authors
  • Could the authors please highlight the novelty of this work beyond introducing a domain-specific benchmark for DE/DA workflows?

  • What is the authors' opinion on model memorization and data contamination in this benchmark? How effective are the proposed mitigation measures in addressing these concerns?

7 comments
AuthorsOpenReview ↗

Response to Reviewer q77p (W1, W2 & Q1, Part 1/2)

Thank you for your thoughtful review! Thanks for recognizing the critical strengths: (1) the clear motivation of data work verification this research problem, (2) the importance of the focused domains of data engineering and analytics, (3) reasonable benchmark construction and rigorous evaluation, and (4) promising verification and repair results.

Below, we clarify your concerns and questions.

W1: I am a bit confused by the term "verification". In program analysis, verification usually has a different meaning. In this work, I do not consider the task to be verification in the traditional sense. The authors primarily perform some form of defect localization and classification with natural language explanations, which seems closer to code review. I would recommend finding a better term that suits this work or explaining more clearly how it differs from existing code review tasks.

W2: The novelty of this work is relatively low. The agents and other components seem basic. If I look at the novelty, the main contribution is essentially a benchmark for the specific DE/DA domain. Apart from the domain, I do not see a major difference from existing benchmarks such as SWE-bench and other code review benchmarks. From a novelty perspective, I do not feel very excited about this work.

Q1: Could the authors please highlight the novelty of this work beyond introducing a domain-specific benchmark for DE/DA workflows?

Thank you for raising questions about the terminology of "verification", the differences from SWE-bench and code review, the novelty, and the agent framework. In short, data work verification tasks ask agents to decide whether completed data work (not just a code change as in code review) is correct and to diagnose its defects, and without being told whether or where a defect exists.

Terminology. Following common usage in LLM research [1–4], we use verification to mean judging whether a candidate solution or artifact is correct with respect to the task requirement, rather than formally proving program correctness. DAVBench applies this sense to data work: the verifier decides whether the candidate satisfies the user requirement and reports the defects it finds.

Differences from SWE-bench and code review. The table below summarizes the key differences:

BenchmarkWhat is assessedEvaluation
SWE-bench [5]No candidate is assessed: the issue is given, and the agent writes a patchTests for the issue and regressions
Code review [6, 7]A proposed code changeReview quality
DAVBenchCompleted data work: workflows and their outputs, such as tables, charts, and reports, which may be correctVerdict accuracy, finding quality, and repair utility

One of the main difference is what is verified. For example, In 32.4% of tasks the candidate includes delivered results or datasets, and in 14.4% it includes charts or reports (more candidates types are in Figure 3(b)), so the verifier must judge whether outputs are correct rather than whether a code change looks right. Faulty workflows often run without failure and yield plausible but incorrect results (i.e., silent error) [8]; the verifier therefore has to check outputs against data semantics and documentation and trace errors to their causes in the workflow. 62 of the 216 candidates are correct, so it must also avoid false alarms. More comparisons summarized in Table 1.

SWE-bench and code review benchmarks also build on real PRs. Our use of PRs may cause some confusion. PRs are just a source of evidence for constructing our tasks. Verifiers are not given the PRs, but the reconstructed workflow or the outputs/artifacts.

Novelty. Overall, as discussed above, the novelty lies first in the research problem itself, not only in the DE/DA domain. Related work has not yet addressed this problem well, especially for data domain. These motivate our contributions, restated briefly below:

  • Problem. To our knowledge, DAVBench is the first repository-level benchmark for verifying real-world data work across DE and DA.
  • Benchmark. Each task reconstructed from real data work, faulty or correct, in an executable workspace that retains the evidence needed to judge it, so verifiers must connect data, code, documentation, and outputs rather than read a single code change.
  • Evaluation. Beyond the finding quality measured by code review benchmarks [6, 7], we jointly evaluate verdicts, defect coverage, false alarms, and the utility of reports for downstream repair under behavioral tests.
  • Findings. Experiments with state-of-the-art models, including GPT-6 Astra, highlight the difficulty of verifying data work: complete diagnosis becomes harder as defects accumulate, verifiers often surface the relevant evidence without turning it into a diagnosis, and incomplete diagnosis is a key bottleneck for repair.

(References are listed in Part 2.)

AuthorsOpenReview ↗

Response to Reviewer q77p (W1, W2 & Q1, Part 2/2)

Agent framework. We do not claim a new agent architecture. We use the bash-only mini-SWE-agent, which is commonly used in the research community [9, 10, 11]. This holds the scaffold fixed across models and reduces confounding from harness differences. However, we are also exploring specialized verification agents as an important future research direction, as you mentioned.

Thank you again for these questions, which helped us clarify our contributions. In the revised manuscript (changes marked in blue), we clarify the meaning of verification and these differences in the Related Work section and add SWE-bench to Table 1.

References

[1] Karl Cobbe et al. Training Verifiers to Solve Math Word Problems. arXiv:2110.14168, 2021.

[2] Lunjun Zhang et al. Generative Verifiers: Reward Modeling as Next-Token Prediction. ICLR, 2025.

[3] Jacky Kwok et al. LLM-as-a-Verifier: A General-Purpose Verification Framework. arXiv:2607.05391, 2026.

[4] Zhisong Qiu et al. Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis. KDD, 2026.

[5] Carlos E. Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR, 2024.

[6] Zhengran Zeng et al. SWR-Bench: Assessing LLM Performance in Real-World Code Review Comment Generation. arXiv:2509.01494, 2025; revised 2026.

[7] Lei Zhang et al. AACR-Bench: Evaluating Automatic Code Review with Holistic Repository-Level Context. arXiv:2601.19494, 2026.

[8] Anthropic. How Anthropic Enables Self-Service Data Analytics with Claude. June 3, 2026.

[9] John Yang et al. ProgramBench: Can Language Models Rebuild Programs From Scratch? arXiv:2605.03546, 2026.

[10] Android Bench Technical Report. First release, February 25, 2026.

[11] Kilian Lieret et al. SWE-sweep: Can Agents Autonomously Find and Fix Bugs? October 1, 2026.

AuthorsOpenReview ↗

Response to Reviewer q77p (W3)

W3: Although the authors provide an analysis of missed defects in Section 3.3, I would appreciate a more detailed qualitative analysis with concrete examples showing why agents fail even after accessing the relevant evidence. Such examples could help identify specific limitations and inspire future research.

Thank you for this suggestion about qualitative analysis. In short, agents fail because they do not turn the evidence they have into a check: they read what the correct behavior should be, but do not compare the candidate against it. Section 3.3 shows that half of the reviewed misses (206 of 411) occur after related materials were surfaced. We also traced these misses through the saved trajectories, as you suggested, and added two case studies to Appendix D with raw trajectories (Figures 10–11). They reveal three specific limitations:

  1. The intended behavior is read but not turned into a check. In a clinical study, a code comment says a medication count covers "the year prior to index date", and three lines below, the code filters on days(90). The verifiers that missed this defect read both lines in the same tool output and never mentioned the window. Every verifier that reported it cited the one-year definition.
  2. Checks run only on the supplied data. In a dbt accounting package, a cash-flow model leaves the first period's opening balance and net change NULL. Verifiers printed these NULLs, but every first-period balance in the supplied data is zero, so no total changes. The only verifier that reported the defect, GPT-6 Astra, compared the NULLs with the ledger's zero opening balance and built a two-period input on which the lost amount shows.
  3. Noticed anomalies are dropped. In the same task, GPT-5.6-Sol counted the six NULL rows and then turned to sign conventions; no later step returns to them.

These patterns are common. In 36 of 40 randomly sampled misses, the verifier had received both the faulty code or output and evidence of the intended behavior, and in 26 it noticed the anomaly and moved on. The defects are within reach: 60.7% of these misses were diagnosed by another verifier.

The three limitations point to concrete directions for future verifiers: deriving explicit expectations from the requirement and documentation before judging the code, generating inputs that target a suspected anomaly, and keeping each anomaly open until it is explained or reported.

The revised Section 3.3 summarizes these findings, and Appendix D gives the full analysis with raw trajectories (changes marked in blue). Thanks again!

Reviewer_Z8ArOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces DAVBench, a repository-level benchmark for evaluating agents' ability to verify real-world data engineering and analysis work. It contains 216 tasks with reconstructed real-world defects, covering both faulty and correct candidates. The authors evaluate eight LLM agents using verification accuracy, defect identification, and downstream repair utility. Results show that current agents struggle to identify all defects, while high-quality verification reports can substantially improve repair success.

Critical strengths
  1. The paper addresses an important but underexplored problem: evaluating agents' ability to verify existing data work rather than simply complete tasks. This provides a meaningful new perspective on agent evaluation.

  2. DAVBench covers real-world data engineering and analysis tasks across 102 repositories, including both faulty and correct candidates. This provides a realistic and diverse setting for evaluating verification capabilities.

Critical weaknesses
  1. The reference defects are initially derived from PR reviews, which typically identify specific issues rather than all defects in a candidate. Although independent annotation expands the defect set from 239 to 389, the paper does not establish whether the final annotations are exhaustive. Missing defects could affect the reliability of Finding F1 and All-found Rate.

  2. The main evaluation uses only one run per model-task pair, with repeated experiments limited to a 60-task subset. Although the authors report relatively stable results, broader repeated evaluations would provide stronger evidence for the reliability of model comparisons.

  3. The hybrid evaluation requires reported findings to match predefined defect locations before semantic judging. This may penalize valid diagnoses that correctly identify the underlying issue but point to locations outside the annotated ranges.

Questions for authors
  1. Given that the reference defects are initially collected from PR reviews, how do the authors ensure that the final annotations cover all relevant defects in each candidate? Could additional independent reviews reveal unannotated defects that affect Finding F1 and All-found Rate?

  2. How does the evaluation handle semantically correct defect diagnoses that point to locations outside the predefined reference ranges? Could the authors quantify how often the location-matching constraint incorrectly rejects valid findings?

  3. Since the main experiments use a single run per model-task pair, could the authors provide additional repeated evaluations or confidence intervals to further establish the robustness of the reported model comparisons?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Z8Ar (W1 & Q1)

Thank you for your thoughtful review, and for especially recognizing the (1) contribution and importance of proposing a new research problem of data work verification, (2) as well as the realism and diversity of DAVBench.

Below, we address each of your concerns and questions.

W1: The reference defects are initially derived from PR reviews, which typically identify specific issues rather than all defects in a candidate. Although independent annotation expands the defect set from 239 to 389, the paper does not establish whether the final annotations are exhaustive. Missing defects could affect the reliability of Finding F1 and All-found Rate.

Q1: Given that the reference defects are initially collected from PR reviews, how do the authors ensure that the final annotations cover all relevant defects in each candidate? Could additional independent reviews reveal unannotated defects that affect Finding F1 and All-found Rate?

Thank you for raising this point about exhaustive annotations! Actually, DAVBench does not claim exhaustive annotations, which is also difficult to guarantee in real-world settings. The more important is the goal to reliably evaluate verification ability, capture this real-world research problem, distinguish verifiers, expose where they fail, and thus provide insights for improvement. These goals are supported by our benchmark design principles:

  1. Defective data workflows reconstructed from real-world projects. Defective workflows and deliverables are based on real-world defects rather than defects fabricated by LLMs, so the tasks capture real-world data verification challenges. Reference defects are grounded in real project feedback and fixes, and only substantial defects that cause errors in data workflows are retained; trivial issues, such as purely textual or documentation-only changes, are excluded (Table 19).
  2. Independent review during construction. Before the reference set is frozen, two additional annotators independently complete each task and review it against the task-quality checklist, and supported new defects are added to the reference findings (Section 2.3, Appendix E.4). This step expands the 239 source-grounded defects to 389.
  3. Complementary repair-based evaluation. Beyond finding-based scoring, we evaluate whether a report helps a repair agent fix the candidate, using behavioral tests grounded in project tests and documented fixes to check whether defects are fixed and correct behavior is preserved. This evaluation does not depend on matching findings to reference defects, and it is consistent with the finding-based evaluation: Finding F1 is strongly correlated with repair success for both repair agents (r = 0.92 and 0.96, Section 3.4).

Independent review of unmatched findings. However, we also try best to conduct an independent review of 70 sampled unmatched findings across 23 faulty tasks to check whether such a review reveals unannotated defects, as you suggested. They cover both DE and DA and different sources and models. A finding counts as valid only if it violates the stated user requirement, falls within the scope of the candidate, is supported by project evidence, and has a material effect on correctness, results, or required behavior. Here is the result:

Audit outcomeCount (%)Details
Valid findings4 (5.7%)Correspond to 3 distinct defects missing from the reference set
Invalid findings42 (60.0%)More than half concern documentation, comments, or formatting
Insufficient evidence24 (34.3%)Do not meet the evidence standard and are not counted as valid

We can see that most of the reviewed unmatched findings do not identify missing defects: many concern trivial issues such as documentation, comments, or formatting, or lack sufficient evidence.

Meanwhile, after adding the three defects to the reference set and rescoring all eight verifiers, Finding F1 and All-found Rate change by at most 0.26 and 0.65 percentage points, and the rankings by precision, Finding F1, and All-found Rate remain unchanged.

In the revised manuscript, we add this audit and the rescoring results to Appendix B.3 (Table 12; changes marked in blue). Thanks again for raising this question!

AuthorsOpenReview ↗

Response to Reviewer Z8Ar (W2 & Q3)

W2: The main evaluation uses only one run per model-task pair, with repeated experiments limited to a 60-task subset. Although the authors report relatively stable results, broader repeated evaluations would provide stronger evidence for the reliability of model comparisons.

Q3: Since the main experiments use a single run per model-task pair, could the authors provide additional repeated evaluations or confidence intervals to further establish the robustness of the reported model comparisons?

Thank you for raising this question and the suggestions on the robustness of results. We have (1) extended the repeated evaluation from the 60-task subset to all 154 faulty tasks for three models, and (2) computed 95% confidence intervals for the original single-run results of all eight models.

  1. Expanded repeated evaluation. We evaluate each of the three models over three runs on the same fixed set of 154 faulty tasks. The table reports mean ± sample SD, with SD calculated using the (n-1) denominator.
ModelFinding F1: Mean ± SDAll-found Rate: Mean ± SD
GPT-5.6-Sol40.33 ± 1.9627.71 ± 1.35
DeepSeek V4 Flash29.34 ± 1.8619.48 ± 1.72
Qwen 3.8-27B20.93 ± 0.6514.50 ± 0.99

For both metrics, the ordering of these three models remains unchanged across all three runs, and the sample SDs are below 2 percentage points.

  1. Confidence intervals for all eight models. For the original single-run scores on the 154 faulty tasks, we use 10,000 paired, source-group bootstrap resamples, stratified by the two DE collections and DA. The 95% confidence intervals are the 2.5th and 97.5th percentiles of the bootstrap scores. The table below shows four verifiers; Table 10 of the revised manuscript lists all eight.
ModelFinding F1 [95% CI]All-found Rate [95% CI]
GPT-6 Astra49.82 [44.62, 54.85]39.61 [32.15, 47.29]
GPT-5.6-Sol41.86 [36.84, 46.95]24.68 [18.19, 31.33]
Gemini 3.8 Flash36.08 [30.50, 41.89]24.03 [17.66, 30.90]
DeepSeek V4 Flash29.22 [24.10, 34.70]20.13 [14.00, 26.61]

In the revised manuscript, we add these confidence intervals, the bootstrap procedure, and the expanded repeated-evaluation results to Appendix A (Tables 8 and 10; changes marked in blue). Thank you again for this suggestion.

AuthorsOpenReview ↗

Response to Reviewer Z8Ar (W3 & Q2)

W3: The hybrid evaluation requires reported findings to match predefined defect locations before semantic judging. This may penalize valid diagnoses that correctly identify the underlying issue but point to locations outside the annotated ranges.

Q2: How does the evaluation handle semantically correct defect diagnoses that point to locations outside the predefined reference ranges? Could the authors quantify how often the location-matching constraint incorrectly rejects valid findings?

Thank you for raising this question about robustness of location matching. We clarify the design and the quantative results of human validation already reported in Section 3.5 and Appendix B.

  1. Design. (1) Reference locations are annotated as one or more admissible ranges, rather than exact-line targets. These ranges can cover the faulty expression, the surrounding transformation, or other locations directly exposing the same root cause (Appendix B.1). A reported line can match any admissible range;(2) We additionally evaluate report utility through behavioral repair tests, independently of finding-reference matching, to reflect the correctness of the finding indirectly.
  2. Existing human validation in Section 3.5. Three annotators independently reviewed all 245 finding-reference pairs formed within 42 sampled reports from 35 tasks, without location-based prefiltering. Among the 46 pairs unanimously confirmed as valid in both diagnosis and location, the location filter retains 44/46 (95.65%), with two rejected (4.35%). These results quantify the filtering loss in the reviewed sample.
  3. Purpose of location matching. The location stage is designed to encourage actionable localization (the location of root cause) and focus semantic judging on relevant candidate pairs. For example, in this audit, only 50 of the 245 pairs proceeded to semantic judging. This reduces the number of semantic comparisons for LLM judges while retaining a separate check on diagnostic correctness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.