- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes a model to study when in AI debates, the freedom of communication in the debate may allow AI agents to achieve task-irrelevant goals. The model treats debate as both a verification mechanism and a space for strategic communication. It explores the trade-off between task success and disclosure of task-irrelevant information.
Critical strengths
I like the formal modeling of the problem. I thought the stylized formulation is interesting, and the paper identifies a limitation of the correctness goal of AI debate.
Critical weaknesses
-
My biggest reservation about this paper is that I don’t think the setting studied lives up to its AI oversight motivation. It’s true that an AI agent may reveal information about things that aren’t directly task-related. However, this alone is not an oversight concern. Only when the AI agent can cause harm by doing this does it become an oversight problem. In all human activities, humans reveal information that are not task-related. This is simply because any task at hand has correlated variables. The paper doesn’t establish that its disclosure objective represents an undesirable outcome. I find the formal construction supports a narrower claim.
-
I couldn’t verify/confirm/follow some of the results. There seem to be missing assumptions or some inconsistency. See questions below.
Questions for authors
- Page 7 attacking strategy clipping. But it seems that the proof of theorem 4 assume no clipping.
- It seems that some of the results (e.g. Lemma 1, theorem 1) requires some stability conditions (K-Lipschitz). But this was not introduced (or maybe I missed where it was introduced).
2 comments
Thank you for your response! This is helpful. But I'm not completely convinced that the setting is truly about oversight. If side information revealed about a task can serve some other harmful objective, then some other task should also be monitored (or the task should be expanded). The problem seems to arise when we have to select which objective to monitor directly and can't monitor all of them.
Addressing concern of harm
Thank you for recognizing the interest of the formal model and for identifying where the motivation and assumptions need clarification. We will submit a revised manuscript as part of this revision, with an expanded threat model that connects disclosure to concrete harmful outcomes and a replacement mitigation analysis.
In the persuasion extension, the prover seeks to steer the verifier toward an incorrect conclusion about a hidden state while preserving performance on the monitored task. We will specify both the verifier’s belief-update rule and the prover’s signalling strategy. In particular, the verifier may interpret evidence using an assumed truthful signalling model that differs from the model actually used by the prover. The verifier then updates incorrectly even though its update is Bayesian under its assumed model.
The extension is motivated by "Black-Box Auditing of Epistemic Reliability in Multi-Agent Debate Distillation". That work studies adversarial arguments supporting the correct monitored response while transcript-based verifier adaptation degrades reliability on related hidden tasks.
In the steganography extension, the recipient is the verifier, and the private signal represents a message or coordination instruction. We will specify the recipient’s observation and decoding rule and the concealment criterion. This makes clear which additional behaviour constitutes the oversight failure.
Accordingly, the revised paper will treat mutual information as a measure of the communication capability supporting these attacks, rather than equating every positive disclosure value with harm.
Q1. (Clipping and the mitigation proof)
You are correct that the reporting strategy includes clipping and the theorem statement and proof therefore need to be reconciled.
We will replace the current signed-drift mitigation with a quadratic discrepancy test. Its theorem statements and proofs will explicitly account for clipping.
Q2. (The missing stability definition.)
The stability property used in our analysis is already stated in Fact 1, Appendix B.3. It specifies that an auxiliary execution whose transition probabilities differ from the reference probabilities by at most (1/d) has acceptance probability within (K/d) of the reference value (\alpha^\star). This property is used specifically in Lemma 4 to control the change in terminal acceptance probability caused by perturbing the simulated computation.
We therefore view this concern as a presentation issue concerning the location and visibility of an existing assumption. A statement in the main body on why Lipschitz is required will be included.