acceptodds
Under review as a conference paper at ICLR 2027

Evidence, Not Independence: Reviewer Identity Alone Does Not Verify a Coding Agent's Claims

Abstract

When a coding agent reports a task complete, something else must decide whether to believe it. We test whether reviewer independence, by itself, is sufficient to catch a false completion claim. In a controlled setting where a coding agent repeatedly produces a fix that passes a ticket's stated example but misses an unstated requirement, we vary who reviews (the same model; the same model reframed as reviewing a colleague; a genuinely different model), what they are told to look for, and what evidence they are shown. Reviewer identity alone never catches the omission under a neutral instruction; a stronger instruction narrows the gap for one reviewer pairing but not the other, and even then leaves a case undetected. Showing the reviewer the actual execution output resolves it in every case tested, including when the reviewer is the same agent that wrote the fix, and a visibility-matched check is consistent with the advantage coming from evidence visibility rather than reviewer architecture. A preregistered, fully-crossed replication confirms the pattern at larger scale and surfaces a cost the earlier experiments could not show: the instruction that most improves detection without evidence also raises false alarms on genuinely correct work, an effect evidence only partly offsets. A further probe finds partial evidence is not simply better than none: at an intermediate level of detail, reliability can fall below both a bare pass/fail signal and full output. An unselected random sample of real-repository tasks confirms the underlying failure is not an artifact of how examples were chosen for study. We report this as a mechanism paper, not a general solution: it establishes when execution evidence resolves the failure, leaving how an autonomous system obtains a sufficiently discriminating check without benchmark-held-out tests as open work. We translate the mechanism into a runtime control plane in which completion claims remain proposals until evidence-gated checks admit them into system state.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, coding agents, meta-agents, agent verification, execution-grounded verification, multi-agent systems, self-verification, AI agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vLB6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to determine whether a coding agent’s task-completion claim can be trusted at runtime. Through controlled experiments, real repositories, cross-model replication, and a preregistered study, the authors show that reviewer independence alone is insufficient to detect plausible-but-incomplete fixes, while execution evidence that can discriminate between correct and incorrect behavior substantially improves detection. The authors further find that stronger review instructions improve detection but also increase false alarms, and that partial evidence can sometimes be less reliable than complete evidence. Based on these findings, they propose a control-plane architecture that treats agent completion claims as proposals and only updates system state after evidence-gated verification.

Critical strengths
  1. The research problem is practically important.
    The paper addresses a fundamental reliability issue in coding agents: how to determine whether an agent’s claim that a task is complete should actually be trusted. The finding that reviewer independence alone is insufficient is particularly relevant to multi-agent coding systems.

  2. The experimental design is systematic and carefully controlled.
    The authors progressively examine controlled tasks, cross-model settings, real repositories, preregistered replication, and matched-visibility experiments. In particular, the visibility-matched experiment helps distinguish the effect of evidence availability from that of the verification architecture.

Critical weaknesses
  1. The task selection and failure mode may introduce some bias.
    Most experiments are constructed around or selected for the narrow-fix/broad-test failure mode, while the random-sample study is relatively small. Therefore, it remains unclear how frequently this failure mode occurs in realistic coding-agent workloads.

  2. Execution evidence does not fully solve the verification-coverage problem.
    The proposed approach relies on having a discriminating check, but ensuring that the configured checks cover all relevant task requirements remains an open problem. In this sense, the paper may shift the verification problem from the worker to a higher-level supervisor rather than fully resolving it.

Questions for authors
  1. How can the system obtain discriminating execution evidence in realistic settings where held-out tests are unavailable?
    Since the effectiveness of the proposed verification approach depends strongly on the availability of useful evidence, it would be helpful to discuss how such checks could be generated or discovered autonomously without access to benchmark hidden tests.

  2. How well do the findings generalize beyond the narrow-fix/broad-test failure mode?
    For example, would execution-grounded verification also reliably detect incorrect algorithmic assumptions, subtle behavioral requirements, or cases where the configured tests pass but the resulting behavior is still incorrect?

Reviewer_ETpTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript investigates whether independent LLM reviewers can reliably verify completion claims made by coding agents. The authors compare several verification settings, including self-review, persona-reframed review, cross-model review, and execution-grounded verification. The central finding is that reviewer independence alone does not reliably identify incomplete code modifications, whereas providing execution evidence substantially improves error detection.

The experiments involve synthetic programming tasks, selected SWE-bench instances, and a preregistered replication. The authors further propose an evidence-gated runtime control plane for managing coding-agent completion claims.

Critical strengths

The manuscript addresses a relevant problem in agentic software engineering. As coding agents become increasingly autonomous, distinguishing between claimed and verified task completion is important for reliable deployment.

The authors make an effort to separate reviewer identity, instruction, and evidence availability. The preregistered experiment and additional real-repository evaluations are appreciated. The analysis of false alarms is also useful, as it illustrates that stronger verification instructions do not necessarily produce more reliable overall judgments.

The manuscript is relatively transparent about several experimental limitations, including selection bias, confounding factors, and the dependence on discriminating execution evidence. The release of experimental traces and implementation details is another positive aspect.

Critical weaknesses

My primary concern is that the central finding is somewhat expected, and the experiments do not convincingly establish a sufficiently novel scientific contribution. The strongest results demonstrate that reviewers can identify an incomplete implementation when they are explicitly provided with execution results showing that the implementation fails the relevant tests. In the preregistered experiment, even a binary pass/fail signal is sufficient to identify all tested incomplete implementations under the evidence-provided conditions. While this supports the importance of external verification, it is unclear how substantially the finding advances existing knowledge about execution-based testing, tool-assisted verification, and coding-agent supervision.

More importantly, the experiments assume access to discriminating evidence without addressing how such evidence can be obtained in practice. In several SWE-bench experiments, the decisive information comes from held-out tests that are unavailable to the coding agent during normal execution. Providing these results to the reviewer naturally reveals failures that would otherwise remain undiscovered. However, in a realistic deployment, the system must identify the missing requirements and construct appropriate verification procedures without access to benchmark-held-out tests. The manuscript acknowledges this limitation, but it substantially restricts the practical significance of the proposed verification mechanism.

The experimental controls for reviewer independence are not fully convincing. The initial cross-model comparisons involve different sets of code modifications, making reviewer identity and task difficulty difficult to disentangle. The preregistered replication improves this aspect, but it still does not fully isolate the effect of writer–reviewer independence. Furthermore, nine of the 120 preregistered substantive calls are excluded because of model response failures, all concentrated in one experimental condition. These limitations weaken the interpretation of the observed differences between reviewer configurations.

Questions for authors

Q1. Could the authors clarify what new verification capability is established beyond the expected observation that a reviewer can identify an incorrect implementation when explicitly shown a failing test result? In particular, how does the proposed approach improve upon a simple deterministic verification gate that rejects completion claims whenever the available tests fail?


Q2. Since several experiments rely on held-out tests that are unavailable during normal agent execution, how would the proposed framework identify incomplete implementations when no discriminating test is initially available? Is there evidence that the framework can improve verification reliability without access to benchmark-provided failure signals?


Q3. How well do the conclusions extend beyond the specific narrow-fix/broad-test failure pattern? Could the authors clarify whether the observed advantages persist across more representative coding tasks, including cases where the underlying implementation error is not directly exposed by an existing test suite?

Reviewer_JHg4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

this paper studies whether an independent reviewer is sufficient to catch incorrect completion claims made by coding agents. The authors compare self verification, reviewing from a different persona, cross-model review, and execution-grounded verification.

They find that independent reviewer agents often fail to identify incomplete fixes when they only have access to the diff, whereas providing execution evidence substantially improves detection. They also find that stronger review instructions increase false positives, and that different levels of execution evidence can have non-monotonic effects on reviewer performance. The authors use these findings to motivate a runtime control-plane architecture where agent completion claims require verification before being accepted.

Critical strengths
  1. As coding agents become more autonomous, understanding when their completion claims can be trusted is a relevant question.
  2. appreciate that the authors acknowledge several limitations
Critical weaknesses
  1. The central finding is essentially that a model reviewing code without executing tests is less reliable than a model that is explicitly shown a failing test. this is not surprising to me.
  2. The experimental setup makes the primary finding somewhat trivial. In a real deployment, the difficult part is often figuring out what to test in the first place, particularly when the specification is incomplete or the failure is not anticipated. The authors acknowledge this, but I think it undermines the significance of the central empirical result.
  3. I'm not convinced the experiments actually isolate the effect of reviewer independence.For example, in Section 6 the two cross-model review directions are evaluated on different sets of diffs. As the authors acknowledge, this means the performance difference could be due to task difficulty rather than reviewer identity.
  4. The experiments are small and heavily selected for a specific failure mode.

Overall, I think the paper studies a relevant problem, but I'm not convinced the results are sufficiently novel or technically substantial for ICLR. The main finding feels fairly expected, and the setup provides reviewers with information that essentially reveals the answer. The more interesting questions, such as how agents obtain reliable verification evidence in the first place, remain largely unexplored.

Questions for authors
  1. what is the threat model you are trying to solve for?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.