- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to determine whether a coding agent’s task-completion claim can be trusted at runtime. Through controlled experiments, real repositories, cross-model replication, and a preregistered study, the authors show that reviewer independence alone is insufficient to detect plausible-but-incomplete fixes, while execution evidence that can discriminate between correct and incorrect behavior substantially improves detection. The authors further find that stronger review instructions improve detection but also increase false alarms, and that partial evidence can sometimes be less reliable than complete evidence. Based on these findings, they propose a control-plane architecture that treats agent completion claims as proposals and only updates system state after evidence-gated verification.
Critical strengths
-
The research problem is practically important.
The paper addresses a fundamental reliability issue in coding agents: how to determine whether an agent’s claim that a task is complete should actually be trusted. The finding that reviewer independence alone is insufficient is particularly relevant to multi-agent coding systems. -
The experimental design is systematic and carefully controlled.
The authors progressively examine controlled tasks, cross-model settings, real repositories, preregistered replication, and matched-visibility experiments. In particular, the visibility-matched experiment helps distinguish the effect of evidence availability from that of the verification architecture.
Critical weaknesses
-
The task selection and failure mode may introduce some bias.
Most experiments are constructed around or selected for the narrow-fix/broad-test failure mode, while the random-sample study is relatively small. Therefore, it remains unclear how frequently this failure mode occurs in realistic coding-agent workloads. -
Execution evidence does not fully solve the verification-coverage problem.
The proposed approach relies on having a discriminating check, but ensuring that the configured checks cover all relevant task requirements remains an open problem. In this sense, the paper may shift the verification problem from the worker to a higher-level supervisor rather than fully resolving it.
Questions for authors
-
How can the system obtain discriminating execution evidence in realistic settings where held-out tests are unavailable?
Since the effectiveness of the proposed verification approach depends strongly on the availability of useful evidence, it would be helpful to discuss how such checks could be generated or discovered autonomously without access to benchmark hidden tests. -
How well do the findings generalize beyond the narrow-fix/broad-test failure mode?
For example, would execution-grounded verification also reliably detect incorrect algorithmic assumptions, subtle behavioral requirements, or cases where the configured tests pass but the resulting behavior is still incorrect?