acceptodds
Under review as a conference paper at ICLR 2027

Placebo Audits Quantify Audit-Specific Acceptance under Adaptive Reuse of Verifier Tests

Abstract

Coding agents are often given diagnostic feedback (failing inputs, expected and actual outputs) by the same small test suite that later decides whether their code is accepted. Holdout-reuse theory warns that this lets an agent fit the audit instead of the task. We make this measurable and show how to avoid it. A placebo audit re-evaluates each frozen output on a fresh audit drawn from the same conditional law as the real one. The gap between the two acceptance rates, audit-specific acceptance, equals the audit-specific part of false acceptance, is zero for any policy whose output is independent of the audit given the first attempt, is bounded by the information the feedback carries about the audit, and has a closed-form unbiased estimator. The same argument gives a remedy: diagnostics computed on a feedback set drawn independently of the acceptance audit provably add no audit-specific acceptance after one round. On 540 EvalPlus tasks, with four models from three families (8B to 32B), four audit sizes, up to 8 feedback rounds and eleven full runs, ten of them preregistered, the placebo meets its calibration prediction (i.i.d. resampling and binary feedback: +0.2 and +0.1 points after one round). One round of diagnostic feedback yields +2.9 [+1.8, +4.1] points of audit-specific acceptance for Qwen3-8B (four runs) and a positive estimate for every other model, including +2.6 [+0.8, +4.5] in a preregistered test at 32B; the point estimates assign roughly half of diagnostics' excess false acceptance to it. After eight retries, i.i.d. selection over nine candidates reaches a similar level. Independent feedback audits kept a fraction 0.74 [0.54, 0.95] of the diagnostic gain in useful acceptance, with audit-specific acceptance +0.1 [-1.0, +1.3]. On 444 LiveCodeBench problems, one-round audit-specific acceptance is +2.6 [+1.0, +4.3], and the preregistered precision contrast is inconclusive. We also report a preregistered replication failure of our own exploratory headline.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
alignment, agents, AI safety, overfitting, verifier, evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zUYyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies coding agents that receive diagnostic feedback from the same small test suite (the audit) that later decides acceptance, and asks how much of the resulting false acceptance comes from fitting that specific audit. It introduces placebo audits: each frozen output is scored, in closed form, on a fresh audit drawn from the same conditional law as the real one (given that the first attempt was rejected). The gap between real and placebo acceptance, audit-specific acceptance (Spec), equals the audit-specific part of false acceptance, is zero for policies whose output is independent of the audit given the first attempt (providing a built-in calibration check), is bounded by the information the feedback carries about the audit, and yields an exact decomposition of excess false acceptance. As a remedy, diagnostics computed on an independently drawn feedback set provably add no audit-specific acceptance after one round.

Experiments cover 540 EvalPlus tasks with four open models (8B–32B), audit sizes 1–8, up to 8 retries, and eleven runs (ten preregistered), plus 444 LiveCodeBench problems. The calibration check holds (i.i.d. resampling and binary feedback: Spec ≈ 0 at one round). One round of diagnostic feedback yields about +2.9 points of Spec for Qwen3-8B and positive estimates for every model, roughly half of the excess false acceptance over resampling; selection over i.i.d. retries reaches a similar level by four retries. Independent feedback retains 74% of the useful-acceptance gain with Spec ≈ 0. The authors also report a failed preregistered replication of their own exploratory headline.

Critical strengths

1. A clean, falsifiable estimand with supporting theory. Placebo audits turn "did the agent fit the tests that accepted it?" into a per-output quantity computed in closed form, with a built-in calibration check (Spec = 0 for policies independent of the audit), an information-theoretic bound, and an exact decomposition of excess false acceptance. The calibration check passes in every run, and a 200-draw physical placebo reproduces the closed form, which makes the measured effects credible.

2. Exemplary empirical rigor and transparency. Ten of eleven runs are preregistered with fixed estimators, across four models, two benchmarks and multiple audit sizes, and the authors report every preregistered outcome, including a failed replication of their own exploratory headline (+19.5 points). This level of reporting is rare and sets a useful standard for evaluation research on agent feedback loops.

Critical weaknesses

1. The measured effect and the remedy have limited practical impact. One round of diagnostics adds about 3 points of audit-specific acceptance, and by four retries plain i.i.d. selection matches it, so the high-bandwidth diagnostic channel is not distinctly harmful at realistic retry budgets. The independent-feedback remedy only removes the one-round component (at K=8, INDEP still has Spec +5.7), and at a fixed test budget reallocating tests changes the false-acceptance share by at most about one point (Table 16, post hoc). Audit size appears to matter more than feedback type (though the paper notes this is not formally established), which is largely expected.

2. Narrow scope relative to the motivating setting. The experiments use 8B–32B open models with stateless feedback on function-level benchmarks. The motivating cases (CI-gated merges, frontier agents with memory and multi-turn interaction, repository-level tests) are where test special-casing is most concerning, but none are tested. Moreover, computing Spec requires a randomly split test pool with per-test results on hidden tests, so the method is a research measurement tool rather than something a deployer could apply to a real CI suite.

Questions for authors

Questions that could change my assessment:

  1. Deployment applicability: Can Spec be estimated without a held-out test pool (e.g., by cross-fitting splits of the visible suite), or is it inherently limited to benchmark settings with many tests per task?
  2. Stronger agents: Do you expect, or have you observed, larger audit-specific acceptance with frontier models, stateful multi-turn feedback, or repository-level tasks (e.g., SWE-bench-style CI)?
  3. Truncation: 30% of Qwen3-8B activation records (8k budget) and 56% on LiveCodeBench are first attempts that hit the thinking budget. The 16k run (E13) suggests estimates are similar with less truncation, but could you report Spec directly on non-truncated records, especially for LiveCodeBench where truncation dominates the activation set?

Clarifications:

  1. Appendix G has only a heading; is Table 17 its intended content?
  2. The placebo assumes audits are uniform random subsets of a fixed pool. How sensitive are the conclusions to non-uniform audit construction, as in real CI suites?

Suggestions (not affecting my score): a summary table mapping run IDs (V2, E2–E13) to model, status and purpose would greatly improve readability, and the abstract could carry fewer numbers.

Reviewer_27smOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

When coding agents receive feedback from the same tests that later decide whether their code is accepted, programs can be falsely accepted. This paper proposes a metric, Spec (audit-specific acceptance), to measure this. It further shows that giving the coding agent feedback on an independently drawn feedback set yields near-zero Spec after one round, while increasing the useful-acceptance rate over i.i.d. resampling.

Critical strengths

Meaningful theory and the proposal of a remedy.

Critical weaknesses

1. Writing prioritizes compression over readability

The paper is so densely written that it is very difficult to follow. For example, the third contribution bullet is mostly a list of experimental settings rather than sentences. The text introduces “IID draws the retries…” without explicitly stating that IID is a policy (the policies are described only briefly in Figure 1.) “We preregistered E9” appears in Section 4, before the experimental design of E9 is described. Thereafter, the paper refers to experimental settings and tests by labels such as E9 and M1 without a consolidated description of them. Many captions also contain sentences like “Full numbers: Table 5,” leaving interpretation to the reader. As a result, the paper as a whole does not read as continuous English prose.

2. Unclear contributions, disorganized structure, and imbalanced experiments

The structure of the paper is not well organized, and neither are the contributions. Introducing Spec, developing its theory, and proposing a remedy are all good; however, the subsequent experiments appear to cover only Spec. The remedy is one of the two major contributions, yet its experiments are weak, and most of the paper is devoted to Spec. Overall, I think the paper does not clearly state its overall claim and does not support it with experiments.

Questions for authors

What is the main message of the paper? The abstract and contributions present the remedy as a contribution on par with placebo audits, yet its evaluation is limited. If the remedy is a primary contribution, why is its experimental support so limited compared with that for Spec? If it is secondary, should the framing be adjusted accordingly?

Reviewer_1di1OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies coding agents that repeatedly revise a program using feedback from a fixed set of tests. For each task, an audit is sampled uniformly from a larger test pool. Starting from a program that fails the audit, the agent receives up to KK retries. The paper introduces 4 feedback policies:

  • IID generates a fresh candidate without feedback
  • BIN receives the previous program and “rejected”
  • DIAG additionally receives the failing inputs, expected outputs, and actual outputs.
  • INDEP uses two independently sampled test sets: one provides diagnostic feedback for debugging, while the other decides acceptance without revealing its inputs or expected answers. The first candidate passing every audit test is returned; otherwise, the last candidate is returned.

The paper evaluates each final program on the entire test pool. Besides measuring acceptance and full-pool correctness, it introduces a comparison called the “placebo audit.” I remain unclear about the additional insight this construction provides beyond conventional held-out evaluation.

For example, suppose the initial program fails all ten tests. Three tests are sampled as the original audit, and the agent revises the program using their feedback. Suppose the final program passes six tests, including the three original audit tests.

Keeping this final program fixed, the authors calculate the fraction of all possible three-test subsets that it would pass. There are (10 choose 3) = 120 subsets, of which (6 choose 3) = 20 contain only passing tests. Its average acceptance across these subsets is therefore 1/61/6. This calculation involves no further program generation or revision.

The original audit accepted the program, giving an acceptance indicator of 1. Subtracting the independent-audit baseline gives 1−1/6=5/61-1/6=5/6 for this output. Across experiments, different original audits and generation seeds can produce different final programs. The authors perform this calculation for each output and average the differences to estimate audit-specific acceptance.

Finally, the experiments show that DIAG gives the largest gain in useful acceptance but also a few points of audit-specific acceptance, while INDEP keeps most of that gain with no audit-specific acceptance.

Critical strengths

The INDEP strategy is interesting, and working on feedback strategies is an important area of research.

Critical weaknesses

I had an extremely hard time parsing what the placebo audit was, actually I think in my summary it might not be so complicated - but the way it is presented in the paper is unclear.

The contribution of the paper is unsurprisingly that feedback helps and the theoretical contribution is over stated, so overall I don't think these results are at A* conference level

The experiments are completely automated, yet run on 2024-2025 models

Questions for authors

Please update the experiments with SOTA 2026 open weight models

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.