- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies “spot-check debate,” a debate protocol where the judge can check only a few values that the debaters choose to state. Debate assumes that the honest debater can point to a false claim that the lying debater makes, and the judge can catch the false claim by checking it; this paper considers a setting where the values that the lying debater computes differ from the honest debater’s only after a step where a lie is inserted. These values where the debaters differ are called “discriminating values.” Then the lying debater can state only values where it and the honest debater agree, and if this is the case, the judge can only catch the lying debater if the honest debater states a discriminating value that the judge checks.
The authors build a testbed of debate protocols for long arithmetic programs and check for each debate whether the honest debater held a discriminating value, whether the honest debater stated a discriminating value, and whether the judge checked a discriminating value. They find that most of the lost accuracy is due to the honest debater not stating a discriminating value or not asking to have such a value checked.
Critical strengths
-- The point that, in spot-check debate, a judge that does not check a discriminating value cannot catch a lying debater that states only true values is clear and interesting. The paper carries out careful experiments to investigate whether the honest debater states a discriminating value that the judge checks, i.e., whether spot-check debate works.
Critical weaknesses
-- The setting is somewhat narrow. The authors run experiments using one synthetic task family (arithmetic programs), and the fixes were run with one main model. Also, the lying debater is constructed by the authors, so the paper shows this strategy breaks the protocol, but models may not use this strategy.
Questions for authors
None
3 comments
Thank you for the detailed response and additional experiments. While these strengthen the paper, I still have some reservations about the generality of the results beyond the settings studied. I remain positive about the paper but will keep my score.
Thank you
Thank you for considering our response and the new experiments, and for remaining positive about the paper. We agree that how far our measured rates and fixes carry beyond the settings studied remains to be tested; the revised §5 states which parts of the argument we expect to transfer and what must be checked again in each new setting.
Response to Reviewer 1w5c
Thank you for the review. On your three points (details in the general response):
One task family. We agree that, apart from the exploratory check below, the experiments use one synthetic task family. The attack itself is not specific to arithmetic: it needs only that the false derivation share its checkable values with the true one up to the false step, as when one step of a proof, an execution trace or a retrieval chain is altered; in quote-verified debate, a dishonest debater that quotes only real passages is in this position, since no check of its quotes can fail. As supporting evidence, Table 2 includes an exploratory check on twelve classic loop algorithms, written as Python programs, where the gap recurs (0.86 against an ordinary liar vs. 0.67 against the attacker; post hoc, p = .02, and p = .09 and .17 after dropping two and three of the algorithms). The revised §5 states which parts of the argument transfer to other domains and how to test them there.
Fixes run with one main model. Your concern was warranted, and testing it showed that what carries over is the diagnosis rather than the script. On a thinned witness, the submission had repeated only the late-claim prompt with Qwen2.5-72B, which raised claiming but not accuracy (§3.5). For the revision, we repeated the main model's best fix, in a registered run, with Qwen2.5-72B in every role. It brought claiming to its ceiling but raised coverage only from 0.18 to 0.24 and accuracy from 0.61 to 0.69 (one-sided p = .12 and .14): at this model's depth only one or two of its three requested values lie in the region, so the check, not the claim, was the binding step.
Requesting only the latest held value, which lies in the region whenever any held value does, raised coverage to 0.48 and accuracy to 0.80 (one-sided p < 10⁻⁶ and p = .002), level with the ordinary liar (0.82). This variant was designed after the first result and registered before its run, and it is the last variant we ran on this model; both results are in Table 2, §3.5 and the updated Limitations, and the revised §5 states that a request rule must fit the task's structure.
The constructed attacker. We agree that current models do not adopt this strategy unprompted; the paper treats the attacker as an instrument, not as a prediction of model behavior. A safety argument must hold for every strategy a protocol permits, and our findings concern the honest side and the judge, which the attacker serves to measure. Instruction alone moves models toward the strategy without reaching it: handed only the shared values, a cover story lowers the main model's false-claim rate from 43% to 17%, and a strict rule lowers GPT-5.5's from 86% to 13% (App. Table 3, App. F.18); the filter removes the rest. A judge that follows confirmations, which is optimal against ordinary liars, would also reward a debater in training for claiming only shared values.
We hope this addresses your three points; if it does, we would be grateful if you would take the revision into account in your assessment. We are happy to answer further questions.