- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies heterogeneous debate, where one strong model debates two weaker ones. The strong models are Claude-Opus, GPT-5.2, and Gemini-3. The experiments use 4,554 BBH instances pooled over four configurations. On the 3,475 instances the strong agent gets right at round 0, debate turns 11.6% into wrong answers. Nearly all of these harmful flips happen when the agents disagree at round 0. The harmful rate is 26.4% under disagreement and 0.2% under consensus. In 93.3% of harmful flips, the final wrong answer matches one that a weak peer gave at round 0. The paper proposes PROTECT, which keeps the strong agent's round-0 answer when the agents disagree. It also tests two variants. PROTECT and its variants are evaluated by offline replay of the logged debate transcripts.
Critical strengths
-
The strong-plus-weak setting is realistic, and looking at item-level outcomes is the right way to study it. Table 2 splits outcomes into harmful, helpful, and neutral. It shows nicely how a near-zero net change can hide large movements in both directions.
-
Table 3, on where harmful flips end up, is informative. In most harmful flips, the final wrong answer matches a weak peer's round-0 answer.
-
The per-task-family breakdown in Fig. 5 shows a lot of heterogeneity. The authors are also upfront about the null merged net effect.
Critical weaknesses
-
No cost comparison with the strong model alone. WEIGHTED gives the strong agent "a double vote on disagreement" (§6). The paper says it "beats both blind debate and the strong-solo baseline" and calls this "a usable gain". This comparison is made on accuracy alone. The strong-solo baseline is a single round-0 answer from the strong agent (App. A). WEIGHTED is triggered by round-0 disagreement. So in deployment it needs round-0 answers from all three agents, and possibly the debate rounds as well. The paper reports that PROTECT skips about half of the debate calls. For WEIGHTED, it reports no calls, tokens, or prices. The number of debate rounds is not given either. So it is unclear whether a gain of +1.3 to +5.7 points (Table 5) is worth the extra calls.
-
The weak peers' own accuracy is not reported. The paper's story is that a capable model is "talked out of correct answers" by weaker peers (§4). The peers are described as "smaller" and "cheaper" models (§3, App. A). The task filter is described only in terms of the strong agent (§3). The peers' round-0 answers are logged (App. A), yet their accuracy on these tasks is never reported. So the paper does not show how much weaker the peers actually are on these tasks. This matters for reading the results. The two Claude configs use the same strong model with different weak peers. Their net effects in Table 2 are −9.0% and +2.0%. §4 refers to the config with the higher harmful rate as Claude-Opus's "more adversarial weak-peer config". Without the peers' accuracy, this difference is hard to interpret.
-
Only one panel size and one vote weight are tested. Every debate in the paper uses one strong agent and two weak agents. The paper motivates this by realistic deployments (§3) and by not relying on a single weak model (App. A). §7 leaves "larger debates" to future work, but no smaller panel is tested either. A one-strong-one-weak panel is the natural, cheaper point of comparison. It matters because 51.0% of harmful flips end on an answer that both weak peers gave at round 0 (Table 3). With one peer, this two-against-one case cannot occur. The weight in WEIGHTED is fixed at a double vote without explanation, and no other weight is tried. With two weak peers of one vote each, a double vote ties whenever both peers agree against the strong agent. The paper does not say how such ties are broken, or which round's answers enter the vote.
Questions for authors
-
(W1) Could you add the cost of each policy to Table 5, in model calls and tokens, or in price? Please also give the number of debate rounds. How does WEIGHTED compare with the strong model alone at a similar cost? One option is a majority vote over several samples from the strong model.
-
(W2) Could you report each weak peer's round-0 accuracy for every config, overall and per task family? This should be computable from the existing logs. It would also help explain the gap between the two Claude configs.
-
(W3) Could you add a one-strong-one-weak configuration for at least one strong model? For WEIGHTED, why a double vote, and how are ties broken? Suppose each weak peer's answer counts once and the strong agent's answer counts w times. If the answer with the largest total wins, the decision changes only at w = 1 and w = 2. Besides the reported w = 2, it would help to report one value between 1 and 2 and one above 2. Together these cover every case in which the strong agent counts more than a peer.