acceptodds
Under review as a conference paper at ICLR 2027

When Honesty is Not Enough in AI Debate

Abstract

Scalable oversight aims to verify the behavior of agents whose capabilities exceed those of their overseers. AI debate has been proposed as an oversight solution in which competing agents help a resource-limited verifier assess claims that it cannot reliably evaluate unaided. Much of its promise rests on incentivizing honest arguments that lead to correct verdicts. Yet a correct verdict need not uniquely determine the arguments used to support it. Agents may retain discretion over which correct claims to present, how to frame them, and in what order to disclose them. This residual freedom can allow agents to shape what the verifier learns beyond the task-relevant conclusion, pursuing latent objectives without compromising verdict correctness. To study this phenomenon, we introduce the framework strategic interactive oversight (SIO), which treats oversight jointly as a verification mechanism and a strategic communication channel. Within this framework, we formalize the notion of task-admissible latent optimization, which entails the pursuit of latent objectives while maintaining a prescribed task performance. As proof-of-concept, we instantiate SIO in the establish protocol debate with cross-examination and quantify a tradeoff between task success and information disclosure about a hidden variable. The trade-off identifies a strategic window in which substantial disclosure remains compatible with task admissibility. Towards mitigation, we reduce admissible bias by expanding the cross-examiner’s role to mitigate persistent disclosure over finite interaction horizons. Our results highlight the need to evaluate oversight not only by the correctness of its verdicts, but also by the information conveyed through its transcripts.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Scalable Oversight, AI Safety

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RLSQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a model to study when in AI debates, the freedom of communication in the debate may allow AI agents to achieve task-irrelevant goals. The model treats debate as both a verification mechanism and a space for strategic communication. It explores the trade-off between task success and disclosure of task-irrelevant information.

Critical strengths

I like the formal modeling of the problem. I thought the stylized formulation is interesting, and the paper identifies a limitation of the correctness goal of AI debate.

Critical weaknesses
  • My biggest reservation about this paper is that I don’t think the setting studied lives up to its AI oversight motivation. It’s true that an AI agent may reveal information about things that aren’t directly task-related. However, this alone is not an oversight concern. Only when the AI agent can cause harm by doing this does it become an oversight problem. In all human activities, humans reveal information that are not task-related. This is simply because any task at hand has correlated variables. The paper doesn’t establish that its disclosure objective represents an undesirable outcome. I find the formal construction supports a narrower claim.

  • I couldn’t verify/confirm/follow some of the results. There seem to be missing assumptions or some inconsistency. See questions below.

Questions for authors
  • Page 7 attacking strategy clipping. But it seems that the proof of theorem 4 assume no clipping.
  • It seems that some of the results (e.g. Lemma 1, theorem 1) requires some stability conditions (K-Lipschitz). But this was not introduced (or maybe I missed where it was introduced).
2 comments
AuthorsOpenReview ↗

Addressing concern of harm

Thank you for recognizing the interest of the formal model and for identifying where the motivation and assumptions need clarification. We will submit a revised manuscript as part of this revision, with an expanded threat model that connects disclosure to concrete harmful outcomes and a replacement mitigation analysis.

  • Disclosure, harm, and the oversight motivation. We agree that disclosure of task-irrelevant information is not inherently harmful. Ordinary communication can reveal additional information, and a correctly specified Bayesian observer may benefit from learning it. The oversight concern arises when the agent uses this freedom to pursue an undesirable downstream objective. The revision will make that additional step explicit through concrete persuasion and steganography instances.

In the persuasion extension, the prover seeks to steer the verifier toward an incorrect conclusion about a hidden state while preserving performance on the monitored task. We will specify both the verifier’s belief-update rule and the prover’s signalling strategy. In particular, the verifier may interpret evidence using an assumed truthful signalling model that differs from the model actually used by the prover. The verifier then updates incorrectly even though its update is Bayesian under its assumed model.

The extension is motivated by "Black-Box Auditing of Epistemic Reliability in Multi-Agent Debate Distillation". That work studies adversarial arguments supporting the correct monitored response while transcript-based verifier adaptation degrades reliability on related hidden tasks.

In the steganography extension, the recipient is the verifier, and the private signal represents a message or coordination instruction. We will specify the recipient’s observation and decoding rule and the concealment criterion. This makes clear which additional behaviour constitutes the oversight failure.

Accordingly, the revised paper will treat mutual information as a measure of the communication capability supporting these attacks, rather than equating every positive disclosure value with harm.

Q1. (Clipping and the mitigation proof)

You are correct that the reporting strategy includes clipping and the theorem statement and proof therefore need to be reconciled.

We will replace the current signed-drift mitigation with a quadratic discrepancy test. Its theorem statements and proofs will explicitly account for clipping.

Q2. (The missing stability definition.)

The stability property used in our analysis is already stated in Fact 1, Appendix B.3. It specifies that an auxiliary execution whose transition probabilities differ from the reference probabilities by at most (1/d) has acceptance probability within (K/d) of the reference value (\alpha^\star). This property is used specifically in Lemma 4 to control the change in terminal acceptance probability caused by perturbing the simulated computation.

We therefore view this concern as a presentation issue concerning the location and visibility of an existing assumption. A statement in the main body on why Lipschitz is required will be included.

Thank you for your response! This is helpful. But I'm not completely convinced that the setting is truly about oversight. If side information revealed about a task can serve some other harmful objective, then some other task should also be monitored (or the task should be expanded). The problem seems to arise when we have to select which objective to monitor directly and can't monitor all of them.

Reviewer_hdvzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper argues that guarantees about verdict correctness in AI debate do not control what else a debate transcript conveys. It introduces strategic interactive oversight (SIO), which separates an agent's task objective Φ from a latent objective Ψ. It defines task-admissible latent optimisation (TALO) as maximizing Ψ subject to Φ ≥ γ.

The latent objective studied is hidden-hypothesis disclosure, Ψ = I(Q; Π_V(h) | X). In the doubly-efficient debate protocol of Brown-Cohen et al. (2024), the prover reports p̄ₜ + ΔQ instead of its estimate p̄ₜ. The paper shows three things about this strategy:

  • For Δ < 1/(16d), the acceptance probability stays within about 1/100 of the honest value (Thm 1).
  • For general Δ, task success is bounded above by a one-round Gaussian survival probability (Thm 2).
  • Disclosure is Ω(hₓ · min(1, Δ²T)) (Thm 3).

As mitigation, the paper adds a sliding-window "drift" challenge. Under it, constant-sign perturbations with Δ ≥ 1/(d√W) lose with high probability, while honest reporting still wins with probability ≥ 3/5 (Thm 4).

Critical strengths
  • The framework is clean and the question is interesting.
Critical weaknesses
  • The formal threat seems to not match the motivating threat. The abstract and introduction motivate the work with framing and selection among true claims to steer a human's interpretation. The instantiation differs in two ways.

    • (a) Mutual information does not measure manipulation. Suppose the verifier is Bayesian and Q is a true fact. Then a larger I(Q; Π_V) means the verifier holds more accurate beliefs about Q, which is not obviously harmful. It is more similar to a covert-channel result rather than belief distortion.

    • (b) The adversarial prover is not honest (more minor). It knowingly reports p̄ₜ + ΔQ, misreporting by Δ while staying inside the protocol's statistical tolerance. The result therefore shows that tolerance-level misreporting can carry a signal, not that honesty is insufficient. A faithful instantiation would let the prover choose among exactly correct moves, such as which witness to present, which subclaim to recurse on, or in what order, similar to Appendix A.

  • The main result is quite narrow. For example, the secret is one bit and the perturbation is quite standard.

Questions for authors
  1. Who is the intended receiver of the disclosed information? Under what threat model is I(Q; Π_V) > 0 a harm when the verifier is Bayesian and Q is true?
  2. Can SIO be instantiated where the freedom is among exactly correct moves rather than sub-threshold misreports, to match the title and motivation?
  3. Do you have any results around the belief-distortion objective in footnote 1?
2 comments
AuthorsOpenReview ↗

Updated threat model

Thank you for recognising the value of the framework and for identifying the gap between the motivating threats and the formal disclosure objective. We will submit a revised manuscript as part of this revision.

Q1. (relationship between disclosure and harm) In the current theorem, the receiver is the verifier through its observed transcript. This observation may also become available to downstream systems through transcript retention, reuse, or training. We agree with your central point: if QQ is a true fact and the verifier knows the actual signalling process, positive mutual information is not itself evidence of manipulation. It measures information acquisition. Our intended interpretation is that disclosure identifies a communication capability that can support a latent objective. Whether that capability causes harm depends on the signal, the recipient, its interpretation of the signal, and the sender’s objective.

The revised manuscript will make this distinction explicit and provide two concrete extensions:

  • Steganographic communication: QQ is a private message or coordination instruction that one agent encodes through task-admissible choices. The verifier attempts to decode it. We will specify the receiver’s observation, decoding objective, and the criterion under which this communication is concealed from oversight.
  • Persuasion: QQ represents a hidden state, and the prover seeks to induce a particular belief or action concerning it. We will explicitly specify the verifier’s assumed likelihood model.

For the persuasion instance, the threat is not simply that the verifier learns a true fact. It is that the prover supplies a signal that the verifier interprets using an assumed truthful signalling rule, while the prover strategically uses a different rule. The verifier can therefore update incorrectly even though its update is Bayesian under its assumed model. This model is supported by "Black-Box Auditing of Epistemic Reliability in Multi-Agent Debate Distillation". That work studies an adversarial debater that continues to defend the correct monitored response while shaping arguments so that transcript-based verifier adaptation degrades reliability on related hidden tasks.

Q2. (what is a correct move) We would push back on the suggestion that within-tolerance reporting places the instantiation outside the paper’s motivating concern. In the stochastic debate protocol, the cross-examiner and verifier assess reports through statistical consistency checks. Even the prescribed honest empirical estimate generally differs from the true conditional probability. The operational requirement is therefore sufficiently accurate reporting, rather than exact equality.

This freedom also reflects the motivating natural-language setting. A task can admit multiple acceptable formulations, choices of evidence, emphases, or orderings. An agent can select strategically among these while satisfying the criteria used by the opposing debater. The probability-reporting construction provides a tractable model of such residual discretion. We regard "honest" arguments as those that are considered correct by the opposing debater.

Q3. (result for the belief-distortion objective) The current submission does not establish a theorem for the objective in footnote 1. As mentioned above, the revised manuscript will include a concrete persuasion extension that provides models of the signalling strategy and how verifier updates posterior beliefs. Details will be available soon in updated manuscript, but we can show that under the model, posterior beliefs move away from the truth.

Finally, we agree that the binary perturbation is a deliberately restricted instance. We will position it as a quantitative demonstration of the phenomenon within stochastic debate, and use the additional instances to clarify the broader scope of SIO without attributing that broader scope to the existing theorem alone.

Thanks for the response! I would like to keep my positive score.

Reviewer_Ko59OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Strategic Interactive Oversight (SIO), a framework that models oversight protocols not only as mechanisms for eliciting correct task outcomes but also as strategic communication channels that can transmit latent information. Within SIO, the authors formalize task-admissible latent optimization (TALO), where an agent optimizes a latent objective while maintaining acceptable task performance, and instantiate this in doubly-efficient debate with cross-examination. They analyze a concrete latent objective—hidden-hypothesis disclosure measured via conditional mutual information—and prove a trade-off: small, undetectable biases can preserve debate success while leaking information at a rate that scales with the number of rounds; they also propose a mitigation via a sliding-window drift check that reduces the space of admissible persistent disclosure strategies.

Critical strengths
  1. A good and underexplored question. The scalable-oversight literature has concentrated on whether a bounded verifier reaches the right verdict. Asking what else the transcript carries, and treating the protocol as a communication channel, is a useful lens with direct implications for logging and reuse of debate transcripts.

  2. The disclosure measure is the right one. Conditional mutual information on the verifier's projection ΠV(h)\Pi_V(h) is decoder-agnostic and, by data processing, bounds what any downstream interpretation of the transcript could extract. This is reusable beyond the debate instantiation.

  3. Theorem 3 is correct and non-trivial. The argument (Pinsker per round, chain rule over the stopped transcript, truncation when the posterior leaves (β,1−β)(\beta, 1−\beta), elimination of the stopping probability) is sound and gives the right qualitative scaling: Θ(Δ2)\Theta(\Delta²) bits per round, linear accumulation, saturation at the prior entropy. Combined with Theorem 1 it exhibits a genuine strategic window inside the Brown-Cohen completeness slack. This is the paper's real contribution.

  4. Working inside an established formal protocol lets the paper say precisely what cross-examination does and does not catch, rather than arguing informally about LLM debates.

Critical weaknesses

The paper's central result (Theorems 1 and 3 together) is sound, but the submission has one correctness problem and several soundness and presentation problems that keep it below the bar as written. Specifics are deferred to the questions below.

  • The mitigation claim is stronger than what Theorem 4 proves. The drift check in Section 5 detects one class of perturbation, and the analysis bounds only the task objective of one strategy in that class. It establishes nothing about disclosure under the modified protocol, and the paper's own lemmas indicate that simple variants of the attack are unaffected. The abstract and Section 1.2 should be revised accordingly.

  • The attack is a within-tolerance misreport, and its framing and positioning overstate the novelty. The perturbation sits inside the slack that the protocol must allow, so Theorem 1 follows closely from the prior completeness analysis, and "honesty" is not the right description of the strategy. The connection to the covert-channel and steganography literature is not discussed.

Questions for authors
  1. Under Figure 3, what is the disclosure of the alternating strategy δt=(−1)tΔQ\delta_t = (-1)^t \Delta Q with Δ=1/(16d)\Delta = 1/(16d)? If it equals the Theorem 3 bound and is never challenged, how will the abstract's mitigation claim be revised? Is there any upper bound on I(Q;ΠV(h)∣X)I(Q; \Pi_V(\mathbf{h}) \mid X) over all task-admissible strategies under either protocol?
  2. (Lemma 3) Please provide a corrected Lemma 3 with a single consistent threshold per quantity, the correct exponents (the threshold 1/(16d)1/(16d) gives exp⁡(−R/(768d2))\exp(-R/(768d^2)) under Lemma 2, not exp⁡(−R/(192d2))\exp(-R/(192d^2))), the correct verifier-sample term, and the resulting constants in Theorems 1 and 2. Which base is the logarithm in R≥192d2log⁡(100T)R \ge 192 d^2 \log(100T)?
  3. (Theorem 2) Theorem 2 is proved from Lemma 7 at t=1t = 1 only. Is a multi-round bound intended? If so, please supply it; if not, please present Theorem 2 as a single-round bound and revise Figure 1(b). Also, Lemma 6 assumes no clipping, which the text justifies only for Δ<1/d\Delta < 1/d; what is the hypothesis of Theorem 2 for larger Δ\Delta?
  4. (Theorem 3) Please add the condition 2RΔ∉Z2R\Delta \notin \mathbb{Z} (used in Lemma 9) to the theorem statement or remove the dependence on Lemma 9, and either justify or delete the clause "there exists R∈O(d2log⁡T)R \in O(d^2 \log T)." Please also confirm that the bound holds for any sign pattern of the per-round shift, since Lemma 8 uses only ∣p+−p−∣≥Δ|p_+ - p_-| \ge \Delta.
  5. (Section 4.5) Section 4.5 states that disclosure grows "proportionally to hxΔTh_x \Delta T." Theorem 3 gives Ω(hx⋅min⁡{1,Δ2T})\Omega(h_x \cdot \min\{1, \Delta^2 T\}). Which is intended?
  6. (Theorem 4) Please state the drift threshold in one normalisation across Theorem 4 (τ2=1/(2dW)\tau_2 = 1/(2d\sqrt{W}) as an average), Lemma 13 (W/(2d)\sqrt{W}/(2d) as a sum), Figure 3 (W/(4d)\sqrt{W}/(4d) for the verifier) and Section 5.1, and define the cross-examiner's challenge statistic Dt,ND_{t,N} with its own estimate q^s\hat q_s rather than the verifier's p^sO\hat p^{\mathcal{O}}_s. Please also fix the inequality direction in equation (18).
2 comments
AuthorsOpenReview ↗

Updates to address mitigation and threat model

Thank you for the detailed review and for recognizing the contribution of the disclosure analysis. Your comments identify several places where the scope of our claims and the presentation of our assumptions need improvement. We will submit a revised manuscript as part of this revision, incorporating the changes described below.

Q1. (Alternating perturbations) We agree that the alternating-sign example exposes a limitation of the signed-drift statistic, where signed deviations can cancel across a window. Such strategies can therefore retain the disclosure scaling while avoiding systematic detection by the current drift check. We will replace the signed-drift mitigation with a test based on accumulated squared discrepancies. The updated section will include:

  • An honest-completeness theorem
  • A new worst-case bound (that retains the same disclosure reduction) within an explicitly specified prover class, rather than analysing only one persistent perturbation strategy.

Q2. (Lemma 3, concentration bounds, and constants.)

You are correct that the exponents do not follow from the stated Lemma 2. We will replace that preliminary inequality with Hoeffding’s bound.

Pr⁡(∣Y‾−p∣≥u)≤2e−2Ru2.\Pr(|\overline Y-p|\ge u)\le 2e^{-2Ru^2}.

The stated failure probability can be retained with a corrected proof. We will propagate these terms consistently through Theorems 1 and 2 and state the resulting constants explicitly.

Q3. Theorem 2 and the first-round survival bound. The use of the first round is intentional. Let KtK_t denote survival through round tt. Since

KT⊆K1,Pr⁡(KT)≤Pr⁡(K1),K_T\subseteq K_1, \qquad \Pr(K_T)\le\Pr(K_1),

a bound on first-round survival provides an upper bound on survival of the entire debate. Together with the audit-error term, this bounds the task objective. We will clarify this reasoning in the theorem and its discussion. We also agree that the clipping assumptions need to be explicit. The revision will state the perturbation range in which the Gaussian comparison applies and provide the necessary treatment of clipping outside that range.

Q4. (Theorem assumptions) We will include the nonintegrality condition 2RΔ∉Z2R\Delta\notin\mathbb Z used by Lemma 9, replace “sufficient entropy” with a quantitative prior condition, and state the applicable perturbation range.

Q5. (Disclosure scaling) The intended scaling is Δ2T\Delta^2T. The expression in Section 4.5 is a typographical error and will be corrected.

Update to threat model with persuasion, and steganography.

We will expand the threat model and related-work discussion to explain why disclosure is relevant to these downstream risks. Mutual information measures the availability of a communication channel and it is a prerequisite for transmitting a private message through the transcript and a useful proxy for that communication capability. Successful persuasion or steganographic coordination additionally depends on how a recipient uses the signal.

The revised manuscript will provide concrete extensions in both directions:

  • Steganography and collusion: QQ represents a private message or coordination instruction held by one debater. The verifier observes the permitted communication and attempts to decode QQ, while the sender maintains task-admissible performance. We will specify the recipient’s observation, decoding objective, and concealment criterion, and connect this model to existing work on covert channels and steganographic collusion.
  • Persuasion and debate distillation: we will distinguish the hidden task’s true state from the debater’s desired target answer. The debater selects task-admissible arguments to influence the verifier’s subsequent beliefs or predictions (either directly or through training on debate transcripts). The model will explicitly connect transcript disclosure to a receiver update rule and a downstream persuasion objective.

The latter extension is motivated by "Black-Box Auditing of Epistemic Reliability in Multi-Agent Debate Distillation". That work studies adversarial arguments supporting the correct monitored response while inducing degradation on related hidden tasks after verifier adaptation. It demonstrates why successful monitored performance does not establish reliability on other tasks affected by the same training transcripts.

Finally, our use of “honesty” is motivated by its role in debate-based oversight. The concrete strategy deliberately perturbs reports within the protocol’s statistical tolerance and this is consistent with a strategy in natural language debate where it provides honest and debate wining arguments but perturbs those arguments in directions on hidden tasks.

Thank the authors for the detailed response. The rebuttal has addressed all my technical concerns, assuming the derivation with the proposed inequalities proceeds smoothly, and I appreciate the authors' acknowledgement of the signed-drift statistic. The expanded discussion and the connection to covert channels and steganographic collusion are welcome and address my second weakness. These address the points I weigh most heavily, so I can raise the score to 3 (weak acceptance). The holdback is mainly because several desired items are proposed but not presented, and it's unclear how smoothly the proofs of the new theorems will proceed.

On the wording choice for "honesty", it was a subtle conceptual difference between what the introduction motivated and what the model in Section 4.3 tries to capture. From the introduction, a debater can choose among all true and winning arguments, so everything it claims is correct. This is "truly honest but strategic" with honesty in the ordinary sense: it never says anything false. On the contrary, the (attacker) prover in Section 4.3 samples the machine RR times, observes the unbiased estimate pˉt\bar{p}_t, but reports pˉt+ΔQ\bar{p}_t+\Delta Q instead. Namely, the prover has an unbiased estimate in hand (i.e., the true and winning argument) and reports a different number. The only reason it was not caught is that the protocol's check has a tolerance of 1/(2d)1/(2d). This is "dishonest by a margin that the checker cannot see". It is a different threat model than what the introduction prescribes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript

We thank the reviewers for their careful feedback. Their comments helped us clarify the oversight motivation, connect disclosure to concrete harmful outcomes, and strengthen the mitigation analysis. We believe these revisions have significantly improved the manuscript.

  1. Clarifying the oversight problem. The introduction and discussion now emphasise the asymmetry of verification. A smaller verifier must assess a more capable agent without reproducing its computation. Existing theoretical guarantees primarily establish task correctness. We ask how to extend these guarantees to safety properties of the interaction (over task-admissible strategies). Adding another monitor does not itself establish such a guarantee. Its effectiveness against strategic behaviour and its resource requirements must also be justified.

  2. Connecting disclosure to concrete harm. Section 3.4 explicitly distinguishes communication capability from harm and specifies additional assumptions for persuasion and steganography. Section 4.5 gives a persuasion construction in which reversing the signal preserves task performance and disclosure but increases hidden-task error for a receiver using the assumed signalling model. Appendix A.2 gives a steganographic example where choosing among exactly valid witnesses communicates a prohibited instruction to another agent while preserving the monitor’s benign transcript distribution. These examples show how acceptable task behaviour can support an undesirable downstream objective.

  3. Providing a broader mitigation guarantee. Section 5 replaces the signed-drift check with window cross-examination using a quadratic statistic. We prove honest completeness and a worst-case disclosure upper bound over a specified regular reporting class, including alternating biases and biases adapted to earlier sampled bits. Under the stated assumptions, the bound is

ΨA(x,σ)≤min⁡{hx,OTd2W+1−γ+δ)}.\Psi_A(x,\sigma)\le \min\{h_x,O\frac{T}{d^2\sqrt W}+1-\gamma+\delta)\}.

Thus, the accumulating disclosure term improves by a factor of W\sqrt W. The revision makes the scope explicit, including conditional independence, concentration and restrictions on persistent private state. It also explains the residual contribution from challenged runs and sampling error, identifying why expected task success alone does not ensure low total disclosure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.