- Rating
- 3
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
DAVBench tests whether agents can find defects in completed data work. It contains 216 tasks from 102 repositories, including 154 faulty and 62 correct cases, with 389 reference defects. Eight models are evaluated on verification and a repair experiment tests whether their reports help another agent fix the work.
Critical strengths
-
Imho, verification is a useful target in its own right. The executable repository context and correct cases make the benchmark more informative than simply asking whether an agent can spot a seeded bug. Including correct work also exposes agents that raise alarms too freely.
-
The construction and validation effort is substantial. Human refinement expands the reference defects from 239 to 389 while the independent matching check covers 245 report-reference pairs. The repair experiment adds practical value: the best model report raises repair success from 24.03% without a report to 37.66% while reference reports reach 69.48%.
Critical weaknesses
-
Precision depends on the reference defects being sufficiently complete. In the paper, an unmatched finding is treated as a false positive but a verifier might discover a valid defect outside the reference set. The reported pairwise validation checks matching quality and so it does not fully answer this completeness question. Since the human refinement already uncovered many extra defects, an audit of unmatched model findings seems particularly important.
-
The repair results establish that reports can help but not that verification is the best use of an equal total budget. The report condition includes an earlier verification pass whereas the no-report condition lacks that extra work. I would keep the claim to report usefulness unless a control gives the repair agent the same additional compute. Also, the repair experiment covers faulty cases, hence it does not measure damage from unnecessary repairs to correct work.
Questions for authors
-
Clarification: were unmatched model findings independently checked for valid defects missing from the frozen reference set? If so, how often did that happen and did correcting those cases change the model ranking?
-
Clarification or additional evidence: is there an equal-total-budget repair control? Without one, can the practical conclusion explicitly distinguish the benefit of diagnostic information from the benefit of spending more compute?
4 comments
Response to Reviewer Tp8k (W2 & Q2)
W2: The repair results establish that reports can help but not that verification is the best use of an equal total budget. The report condition includes an earlier verification pass whereas the no-report condition lacks that extra work. I would keep the claim to report usefulness unless a control gives the repair agent the same additional compute. Also, the repair experiment covers faulty cases, hence it does not measure damage from unnecessary repairs to correct work.
Q2: Clarification or additional evidence: is there an equal-total-budget repair control? Without one, can the practical conclusion explicitly distinguish the benefit of diagnostic information from the benefit of spending more compute?
Thank you for this question about compute trade-off between verifier and repairer! Actually, we do not claim verification with repair is the best use of an equal total budget, and our experiments and conclusions also do not rely on this control. The main purpose of repair-utility experiment is to evaluate the utility of verification reports from the perspective of downstream repair and to compare models.
The no-report condition just serves as a baseline to measure the repair actor’s unaided performance. To ensure fair comparisons among verifiers, we use the same per-task verification time budget and hold the downstream repair actor and its budget fixed across report conditions.
However, allocating compute between verification and repair is also a very interesting research direction! We will explore this as a future work. Thank you for your insightful question.
The question about the unnecessary repair on correct candidates. We further sampled 19 correct candidates on which the GPT-5.6-Sol verifier reported findings and evaluated repairs by GPT-5.6-Sol with and without those reports (Appendix C.4, Table 18):
| Condition | Candidates passing all tests | Pass rate |
|---|---|---|
| Original candidates, unchanged | 19/19 | 100% |
| Repair without a report | 16/19 | 84.21% |
| Repair with the findings report | 14/19 | 73.68% |
These results illustrate the risk of unnecessary repairs to correct work.
Actually, in our faulty candidate tasks, our P2P tests could also check whether repair damages previously correct behavior (Table 16). Table 17 shows an association between unmatched findings and P2P failures, where the P2P failures get worse with more unmatched findings:
| Repair actor | 0 unmatched findings | ≥1 unmatched finding | ≥5 unmatched findings |
|---|---|---|---|
| GPT-5.6-Sol | 3.76% | 9.08% | 27.03% |
| Qwen 3.8-27B | 2.34% | 7.65% | 22.97% |
We have added these analyses to Appendix C.4 of the revised manuscript (Table 18; changes marked in blue). Thank you!
Thanks for the extra checks. The audit and unchanged rankings largely address my concern about missing defects. Also, the new experiment helps clarify the risk of unnecessary repairs. I’m also happy with limiting the repair claim to report usefulness. Please make clear in the paper that total compute is not matched and some audit findings remain inconclusive. Most of my concerns are now addressed so I’ll increase my soundness score and informativeness to 3.
Response to Reviewer Tp8k
We sincerely thank you for your time and for raising the Soundness and Informativeness scores. Your thoughtful comments have greatly helped us improve the paper. As you suggested, we will make your mentioned points in the final paper version. If any concerns remain, we would be very glad to address them. Thanks again!
Response to Reviewer Tp8k (W1 & Q1)
Thank you for your thoughtful review, and for recognizing (1) data work verification as an important research problem, (2) the executable, repository-level setting with correct candidates that goes beyond spotting bugs in code, (3) and the rigorous construction and practical value of the repair experiment.
Below, we address your concerns and questions.
Thank you for this thoughtful question about unmatched model findings! Yes, we checked a sample of unmatched findings: few of them identify missing defects, and correcting these cases leaves all model rankings unchanged (details below). The more important point is that DAVBench does not claim that reference defects are exhaustive, which is also difficult to guarantee in real-world settings. Instead, its goal is to reliably evaluate verification ability, capture this real-world research problem, distinguish verifiers, expose where they fail, and thus provide insights for improvement. These goals are supported by our benchmark design principles:
Independent review of unmatched model findings. However, we also conducted an independent review of 70 sampled unmatched findings across 23 faulty tasks, selected to cover both DE and DA and different sources and models, as you suggested. A finding counts as valid only if it violates the stated user requirement, falls within the scope of the candidate, is supported by project evidence, and has a material effect on correctness, results, or required behavior. Here is the result:
Most of the reviewed unmatched findings do not identify missing defects: many concern trivial issues such as documentation, comments, or formatting, or lack sufficient evidence.
After adding the three defects to the reference set and rescoring all eight verifiers, the rankings by precision, Finding F1, and All-found Rate remain unchanged, and the three metrics change by at most 0.32, 0.26, and 0.65 percentage points, respectively.
In the revised manuscript, we add this audit and the rescoring results to Appendix B.3 (Table 12; changes marked in blue).