- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies coding agents that receive diagnostic feedback from the same small test suite (the audit) that later decides acceptance, and asks how much of the resulting false acceptance comes from fitting that specific audit. It introduces placebo audits: each frozen output is scored, in closed form, on a fresh audit drawn from the same conditional law as the real one (given that the first attempt was rejected). The gap between real and placebo acceptance, audit-specific acceptance (Spec), equals the audit-specific part of false acceptance, is zero for policies whose output is independent of the audit given the first attempt (providing a built-in calibration check), is bounded by the information the feedback carries about the audit, and yields an exact decomposition of excess false acceptance. As a remedy, diagnostics computed on an independently drawn feedback set provably add no audit-specific acceptance after one round.
Experiments cover 540 EvalPlus tasks with four open models (8B–32B), audit sizes 1–8, up to 8 retries, and eleven runs (ten preregistered), plus 444 LiveCodeBench problems. The calibration check holds (i.i.d. resampling and binary feedback: Spec ≈ 0 at one round). One round of diagnostic feedback yields about +2.9 points of Spec for Qwen3-8B and positive estimates for every model, roughly half of the excess false acceptance over resampling; selection over i.i.d. retries reaches a similar level by four retries. Independent feedback retains 74% of the useful-acceptance gain with Spec ≈ 0. The authors also report a failed preregistered replication of their own exploratory headline.
Critical strengths
1. A clean, falsifiable estimand with supporting theory. Placebo audits turn "did the agent fit the tests that accepted it?" into a per-output quantity computed in closed form, with a built-in calibration check (Spec = 0 for policies independent of the audit), an information-theoretic bound, and an exact decomposition of excess false acceptance. The calibration check passes in every run, and a 200-draw physical placebo reproduces the closed form, which makes the measured effects credible.
2. Exemplary empirical rigor and transparency. Ten of eleven runs are preregistered with fixed estimators, across four models, two benchmarks and multiple audit sizes, and the authors report every preregistered outcome, including a failed replication of their own exploratory headline (+19.5 points). This level of reporting is rare and sets a useful standard for evaluation research on agent feedback loops.
Critical weaknesses
1. The measured effect and the remedy have limited practical impact. One round of diagnostics adds about 3 points of audit-specific acceptance, and by four retries plain i.i.d. selection matches it, so the high-bandwidth diagnostic channel is not distinctly harmful at realistic retry budgets. The independent-feedback remedy only removes the one-round component (at K=8, INDEP still has Spec +5.7), and at a fixed test budget reallocating tests changes the false-acceptance share by at most about one point (Table 16, post hoc). Audit size appears to matter more than feedback type (though the paper notes this is not formally established), which is largely expected.
2. Narrow scope relative to the motivating setting. The experiments use 8B–32B open models with stateless feedback on function-level benchmarks. The motivating cases (CI-gated merges, frontier agents with memory and multi-turn interaction, repository-level tests) are where test special-casing is most concerning, but none are tested. Moreover, computing Spec requires a randomly split test pool with per-test results on hidden tests, so the method is a research measurement tool rather than something a deployer could apply to a real CI suite.
Questions for authors
Questions that could change my assessment:
- Deployment applicability: Can Spec be estimated without a held-out test pool (e.g., by cross-fitting splits of the visible suite), or is it inherently limited to benchmark settings with many tests per task?
- Stronger agents: Do you expect, or have you observed, larger audit-specific acceptance with frontier models, stateful multi-turn feedback, or repository-level tasks (e.g., SWE-bench-style CI)?
- Truncation: 30% of Qwen3-8B activation records (8k budget) and 56% on LiveCodeBench are first attempts that hit the thinking budget. The 16k run (E13) suggests estimates are similar with less truncation, but could you report Spec directly on non-truncated records, especially for LiveCodeBench where truncation dominates the activation set?
Clarifications:
- Appendix G has only a heading; is Table 17 its intended content?
- The placebo assumes audits are uniform random subsets of a fixed pool. How sensitive are the conclusions to non-uniform audit construction, as in real CI suites?
Suggestions (not affecting my score): a summary table mapping run IDs (V2, E2–E13) to model, status and purpose would greatly improve readability, and the abstract could carry fewer numbers.