acceptodds
Withdrawn from ICLR 2027

The Verifier-Bit Ledger: A Leakage Ceiling and an Executable Audit for Verifier-in-the-Loop Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Verifier feedback drives much of the recent progress in iterative reasoning, yet the channel that carries it can also carry the answer: on hidden-test pass rate, a model that learns to repair looks identical to one fed the hidden tests, and contamination audits, which cover only static training data, cannot say with a guarantee how much of a gain the feedback could have supplied. We introduce the verifier-bit ledger. Its budget meters what each round of feedback reveals about the hidden answer; its ceiling bounds the improvement any policy can buy with that budget over a ghost baseline whose verifier is keyed to decoy answers, and is tight to first order as the budget vanishes and exactly tight for declared feedback alphabets. Inverting the ceiling yields an executable audit that certifies an unlogged information path, at a chosen confidence level, whenever improvement outruns the budget; we characterize exactly when it can fire, and it is deployable today for low-rate or structurally capped feedback. Two conservation theorems show that compliant training cannot smuggle answers and that amortized gains are prepaid. On a synthetic family where every quantity is exactly computable, the bounds close with matching constants and strong injected side channels are caught at the predicted confidence. On real code repair, free-text feedback is priced rather than tested, and a planted leak lifts repair success by five points over the ghost; on three controlled real-task protocols with declared or capped feedback, planted leaks trigger the certificate and honest runs never do. A residual audit conditioned on legally disclosed feedback extends certificates to full first-failure feedback on randomized suites, catching a one-test exploit the ceiling cannot. Verification information can be budgeted, audited, and priced like computation.

Primary area:
learning theory
Keywords:
verifier-in-the-loop reasoning, information leakage, leakage auditing, mutual information, adaptive evaluation, code repair

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1aWQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how much improvement an iterative reasoning system can obtain from feedback about a hidden specification. It defines an information budget for the feedback transcript and compares performance with a “ghost” interaction driven by an independently sampled decoy specification. Standard information inequalities yield an improvement ceiling, which the authors turn into statistical tests for information paths outside the accounted feedback. The paper also proposes a tight ceiling for bounded transcript alphabets, structural caps for verbose feedback, and training-time conservation results. Validation includes an exactly solvable synthetic family and controlled code-repair protocols. Unrestricted text feedback is evaluated descriptively through pricing, while certificates are reported for constrained protocols and a separate residual audit.

Critical strengths
  1. I liked how the paper connects an explicit ghost baseline, an information budget, and a statistical audit. Theorem 1 bounds improvement over the decoy-driven interaction, while Proposition 2 gives an exactly tight ceiling for declared feedback alphabets. Together, these results provide a precise way to assess whether accounted feedback can explain observed gains.

  2. I think the residual audit is a useful contribution because it accounts for information that legitimate feedback has already disclosed. Appendix G, Propositions 6–8, develops the corresponding posterior and e-value construction. The comparison in Table 6 between legal overfitting and exploiting an undisclosed test provides a meaningful controlled demonstration of why this conditioning matters.

Critical weaknesses
  1. I think the practical evidence remains limited to controlled settings. The declared-budget experiments use deliberately constrained protocols, and the successful residual-audit exploit in Appendix G is scripted around model-generated programs. The model itself did not meaningfully exploit the supplied leak. These results support the proposed mechanisms, but provide limited evidence about detecting leakage arising naturally in deployed reasoning systems. I view this as a limitation of the demonstrated practical scope rather than a failure of the formal guarantees.

  2. I had a concern about the broader generalization in Appendix D, lines 1073–1075. This passage appears to extend the uniform-prior success bound by replacing (m\ln 2) with general conditional entropy. High entropy alone can coexist with a dominant answer that is easy to guess without feedback, so this substitution seems to require additional assumptions. I would appreciate clarification.

Questions for authors
  1. Could you clarify the assumptions behind the entropy-based extension in Appendix D, lines 1073–1075? In particular, is an additional uniformity assumption intended, or is a different bound needed for nonuniform specifications? This would help establish the scope of the claimed generalization.

  2. In Theorem 4, should τnew\tau_{\mathrm{new}} explicitly include both the feedback transcript and the final output, as used in Appendix C’s proof? I think this interpretation addresses the policy-randomness concern, but the main text previously defines τ\tau as feedback alone. Making the definition consistent would clarify exactly which information quantity the conservation guarantee bounds.

Reviewer_cQsGOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a way to audit whether verifier feedback leaks the answer when an LLM revises its output based on that feedback, for example, failing test cases or a pass/fail verdict. The authors measure how much the feedback reveals about the hidden tests with mutual information and call this the budget. They also define a ghost baseline, where the same loop is run with the verifier keyed to a decoy answer drawn from a prior, so that the generic value of receiving feedback is kept, but the value that depends on the true answer is removed. They show that the gain over this baseline cannot exceed a ceiling set by the budget, and turn this into a test. If the observed gain is above the ceiling, the test concludes, at a chosen confidence level, that information about the answer reached the model through a path that is not in the log. The method is evaluated on synthetic data, where every quantity can be computed exactly, and on code repair with HumanEval+ and MBPP+. When feedback is restricted to a few possible values, such as pass/fail, planted leaks are detected, and honest runs never trigger the test.

Critical strengths

The problem is important for evaluation integrity. Feedback given during evaluation can carry the answer, and existing contamination checks, which look only at training data, do not cover this. The ghost baseline is a sensible design. It keeps the effect of receiving feedback at all while removing the part that depends on the true answer. I also appreciate that the authors are open about the limits of their method, including that the theory builds on standard inequalities, that free-text feedback cannot be tested, and that the detected leaks were planted.

Critical weaknesses
  1. The method does not address the problem raised in the introduction. The introduction says the issue is that a model that learned to repair cannot be told apart from one that was fed the answer. But the method does not separate feedback into legitimate help and leakage, and all logged feedback is treated as legitimate. What it detects is only the presence of an unlogged path. If the answer is contained in the logged feedback itself, the method cannot tell.
  2. The assumption of an unlogged path does not fit LLMs well. Everything an LLM receives is its context, which can be logged in full, so there is little room for an unlogged channel. If the auditor can already separate logged feedback from other paths, it seems much simpler to inspect and block those paths directly. The paper does not show when the statistical audit is more useful than that.
  3. The audit has almost no power in practice. With free-text feedback, the budget is estimated to be very large, and the ceiling becomes meaningless. When the budget is estimated with a predictor, the finite-sample correction is so large that the test cannot trigger at realistic sample sizes. Even with one-bit feedback, the gain has to exceed about 0.5 before anything is detected, so only leaks that hand over nearly the whole answer can be caught. The leaks detected in the experiments are planted ones that would also be found by looking at the model's context.
  4. The theoretical novelty is limited. The main results are direct applications of the chain rule for mutual information and Pinsker's inequality, and the training-time results follow almost directly from the data processing inequality. The ghost baseline also needs a prior over hidden tests, which the authors admit is not defined for a fixed benchmark.
Questions for authors
  1. Suppose the logged feedback contains part of the answer, for example, the expected output of a failing test, and the model hard-codes that case. Can the method detect this as leakage? If not, how does the method relate to the problem stated in the introduction?
  2. Given that the auditor controls the evaluation harness, in what situation is the statistical audit more useful than directly inspecting the model's context and the system?
  3. Does the method have any power against leaks that have been reported in practice, such as agents reading test files from the file system?
Reviewer_4fNbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I am not able to fully comprehend what the paper is trying to accomplish. It seems like the paper is looking at the case where you have an agent that takes actions, and receives feedback from a verifier. They want to measure how much of the improvement the agent has throughout this iterative process of receiving feedback and revising actions is driven by the verifier leaking information about some hidden-unit tests that the verifier has access to and how much of it is purely by the agent's correct reasoning and critical thinking to revise its action.

Critical strengths

None

Critical weaknesses

Incomprehensible writing. No clear connection or motivation for all the theoretical results no practicality of the theoretical results is demonstrated or linked to the experiments. the experiment plots are hard to understand and its not clear what they're communicating or what metrics they are measuring and there is no clear metrics that is consistent with the rest of the paper.

Questions for authors

This paper needs a full rewrite. The problem is not clear, the solution is not clear, and the theory and empirical results are not consistent.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.