acceptodds
Under review as a conference paper at ICLR 2027

AI Safety via Debate is Compromised by Cognitive Biases

13 views

Abstract

Reinforcement learning from human feedback (RLHF) has played a central role in making large language models responsive to human instructions. However, human evaluators often favor flattering or persuasive responses over truthful ones, creating incentives for models to appeal to evaluators at the expense of accuracy. AI safety via debate has been proposed as a way to improve the supervision of language models: in this paradigm, two agents argue opposing positions and challenge each other’s claims, potentially exposing falsehoods to the adjudicator. A central premise of AI safety via debate is that truthful arguments are easier to defend than false ones under adversarial scrutiny. In this work, we investigate whether this advantage persists when debaters use rhetorical strategies that exploit biases in human judgment. Inspired by competitive debate, we construct 68 LLM-generated dialogues about detective mysteries with known culprits, spanning four interventions: anchoring, fallacious reasoning, pro-jargon, and verbosity. We apply each intervention to either the side advocating for the true culprit or the side advocating for an innocent suspect, allowing us to distinguish influence on adjudication from correctness. In a study with 369 participants, we find that, pooled across bias types, these interventions significantly shift judgments toward the manipulated side. These findings expose a vulnerability in debate-based supervision: human adjudication is sensitive to manipulative rhetorical strategies.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Debate, Cognitive Biases, Human Adjudication, Scalable Oversight

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_A7esOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows how humans can be affected by contaminated information with four biases (anchoring, fallacy, pro-jargon, verbosity). Through LLM debate trajectories, the authors make several scenarios and inject some biases into the process of debate. This makes the humans affected by the injected biases and favor the rhetorical one.

Critical strengths
  1. The evaluation is clear.
    • The approach to testing hypothesis looks structured.
    • Human subject research requires IRB and they report about all information clearly.
  2. The content is easy to follow and interesting.
    • I like the topic of how the AI-based system will influence to human society and this paper deals with it.
    • This paper is easy to follow for non-expert reader in this area.
Critical weaknesses
  1. I like this kind of paper, which study about ai-system with human being, however this paper lacks novelty fitted to ICLR.
    • As introduced in abstract and Introduction section, human like rhetorical phrase in someway, so it is obvious that human will be affected by the 4 biases (anchoring, fallacy, pro-jargon, verbosity).
    • And there are works stating similar claim (Language Models Learn to Mislead Humans via RLHF, Wen et al., 2025 ICLR; Deceptive Explanations by Large Language Models Lead People to Change their Beliefs About Misinformation More Often than Honest Explanations, Danry, 2025 CHI).
  2. Results and Analysis Section is narrow.
    • It looks clear that humans are influced by rhetorical phrases seeing Table 1, but no more results in main page without Table 1 makes the results less interesting.
    • I want to know whether the human evaluators are experts about the detective mysteries or not. If they are the experts on them, their judgement could be different by the expertise regardless of four interventions.
Questions for authors
  1. Are the human evaluators experts on the detective scenario?
  2. How does adjudication change if LLM judge on same debate trajectories?
5 comments
AuthorsOpenReview ↗

Responses to Weaknesses (Reviewer A7es)

Thank you for your thoughtful review.

We agree that human susceptibility to rhetorical influence is well established in general, in the field of cognitive science, human decision-making, and beyond. Our contribution’s novelty comes from studying implications for human adjudication in debate settings for training AI systems. We address the two points separately below.

Weakness 1.1: As introduced in abstract and Introduction section, human like rhetorical phrase in someway, so it is obvious that human will be affected by the 4 biases (anchoring, fallacy, pro-jargon, verbosity).

1. Human susceptibility in AI debate is not obvious and requires further empirical investigation.

Debate has been proposed as a way to improve human supervision for training AI systems. The idea is that competing agents present arguments and counterarguments to help a judge distinguish correct from incorrect answers, and the judge relies on the facts to adjudicate. This motivation makes it important to test whether human adjudication remains sensitive to bias-targeted interventions within such a structure. Prior evidence that people can be influenced by rhetoric motivates this hypothesis, but does not establish whether, or to what extent, these interventions shift judgments when people evaluate competing evidence-based arguments and rebuttals in the debate setting.

Our study directly examines this question under a controlled debate protocol. We find that, pooled across the four intervention types, judgments shift toward the manipulated side. We provide empirical evidence of susceptibility in the debate protocol studied, which is, to our knowledge, a novel contribution.

Weakness 1.2: And there are works stating similar claim (Language Models Learn to Mislead Humans via RLHF, Wen et al., 2025 ICLR; Deceptive Explanations by Large Language Models Lead People to Change their Beliefs About Misinformation More Often than Honest Explanations, Danry, 2025 CHI).

2. How our contribution differs from Wen et al. and Danry et al.

We appreciate these relevant references and will incorporate them into the related work with an explicit discussion of the distinctions. Most importantly, neither paper empirically tests human adjudication of competing arguments and rebuttals in an AI debate setting. Their findings establish important concerns about misleading AI outputs more generally, but they do not directly examine whether bias-targeted interventions influence human judgments in debate.

The treatment of bias also differs. Wen et al. examine misleading behavior emerging through RLHF and identify strategies such as subtle causal fallacies, but do not experimentally vary the four bias-targeted interventions we study. Danry et al. compare honest and deceptive AI feedback and examine how the logical validity of explanations relates to their credibility. Our study instead operationalizes anchoring, fallacy oversight, pro-jargon, and verbosity through explicit intervention rubrics and compares manipulated debates against their unmanipulated baselines.

We apply these interventions to both the truthful and untruthful sides of the debate. This allows us to test whether the manipulations shift judgments toward the targeted side, independently of whether that side is correct. Together, our controlled interventions and corpus of 68 debates provide evidence about the susceptibility of human adjudication in AI debate to cognitive bias manipulations, a question that was not directly examined by these prior studies.

Weakness 2: It looks clear that humans are influenced by rhetorical phrases seeing Table 1, but no more results in main page without Table 1 makes the results less interesting. I want to know whether the human evaluators are experts about detective mysteries or not. If they are the experts on them, their judgement could be different by the expertise regardless of four interventions.

We measured participants’ self-reported familiarity with detective stories and mystery puzzles. Among the 369 participants in our analysis sample, 19.2% reported low familiarity (1–2 on a seven-point scale), 57.5% moderate familiarity (3–5), and 23.3% high familiarity (6–7).

The original submission examined the relationship between self-reported expertise and adjudication in Appendix C.5. In particular, participants reporting high detective familiarity achieved 42% accuracy (n = 86), compared with 58% among those reporting low familiarity (n = 71). Thus, we did not find evidence that greater familiarity with detective stories improves adjudication accuracy, although this comparison is exploratory.

We agree that this finding deserves greater visibility. We will highlight it in Section 6.2 and add aggregate familiarity statistics to Appendix C.5, while keeping more detailed participant-level background responses private.

AuthorsOpenReview ↗

Responses to Questions (Reviewer A7es)

Q1: Are the human evaluators experts on the detective scenario?

Please see table below and the commentary regarding expertise above.

Self-reported measureMean (SD)Low, n (%)Moderate, n (%)High, n (%)
Detective-story and mystery-puzzle familiarity4.13 (1.63)71 (19.2)212 (57.5)86 (23.3)
Debate-judging experience3.33 (1.70)122 (33.1)205 (55.6)42 (11.4)
Debate-participation experience3.31 (1.75)141 (38.2)183 (49.6)45 (12.2)

*Low, moderate, and high denote ratings of 1–2, 3–5, and 6–7, respectively.

Q2: How does adjudication change if LLM judge on same debate trajectories?

The original manuscript reported the adjudication prompt and decisions of six LLM judges on the same debate trajectories in Appendix A.4.2 (Tables 30–31). We will add Appendix C.7 to compare these decisions with human judgments.

Weighting dialogues equally, baseline accuracy is similar for humans and LLMs (53.9% and 54.2%). On manipulated dialogues, the average shift toward the manipulated side is +9.0 percentage points for humans and +25.3 for LLMs. When the manipulation supports the culprit, LLMs achieve higher accuracy than humans (78.1% versus 61.4%), but when it supports the innocent rival, LLMs achieve lower accuracy (27.6% versus 43.4%). Overall accuracy on manipulated dialogues is similar (52.9% for LLMs versus 52.4% for humans).

The LLM judges were used to select the generated dialogues, rather than for an independent evaluation. Dialogues were regenerated unless at least one of six LLM judges shifted toward the manipulated side and none shifted away. The larger shift among LLMs is therefore expected from our selection procedure and should not be interpreted as evidence that LLMs are more susceptible than humans. Our human study tests whether manipulations selected using LLM judgments also influence human adjudication; our results show that they do.

Thank you for the detailed response. The clarification on related work is helpful, but some concerns remain. Compared with prior work on human susceptibility to persuasive responses and on judge biases in debate and oversight [1, 2, 3], the contribution still feels incremental to me. In addition, the opponent’s turns are reused from the baseline and never respond to the manipulated rhetoric, so the claim that the biases survive adversarial debate is not fully supported. The expertise analysis also reports only accuracy, not whether expertise changes the intervention effect. Finally, the main results still rely mostly on Table 1, where no bias-specific effect is significant. For these reasons, I will keep my current score.

[1] Sharma, M., et al. Towards Understanding Sycophancy in Language Models. ICLR 2024. [2] Rahman, S., et al. AI Debate Aids Assessment of Controversial Claims. NeurIPS 2025. [3] Recchia, G., et al. Confirmation Bias: A Challenge for Scalable Oversight. AAAI 2026.

Reviewer_DCRfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tests a central premise of AI safety via debate: that truthful positions are easier to defend than false ones. The authors ask whether this advantage persists when one debater uses rhetorical strategies that exploit known cognitive biases. Using four detective stories, they generate 68 six-turn LLM debates and apply one of four interventions to either the side accusing the true culprit or the side accusing an innocent suspect. Results shows that interventions shift judgments toward the manipulated side by 8.6 percentage points, while no individual bias type reaches significance on its own.

Critical strengths
  • The paper raises an interesting question: whether human judgement in debate is robust to rhetorical manipulation is central to the viability of debate as a scalable oversight method,
  • The experimental design is clean and the study is transparent and reproducible. Although the corpus is small (68 dialogues from four stories), the generation pipeline (evidence-indexed stories, content and style plans, controlled interventions) is a reusable framework that future work could scale up.
Critical weaknesses
  1. The honest side has no opportunity to respond to the manipulation.

A key argument for debate is that each side can expose flaws in the other’s reasoning, including rhetorical tricks. In this setup, however, manipulated turns are generated after the baseline debate is complete, and the opposing side’s turns are reused unchanged.

  1. Baseline accuracy is close to chance, which makes it unclear what the experiment is measuring.

Without any manipulation, judges are correct only about 51% of the time. This suggests that, in this setting, judges are often unable to identify the truthful side from the debate alone. Under these conditions, the observed shift may say less about how specific manipulations change judgments, and more about how any surface-level difference between the two sides can shift a decision.

Questions for authors

The most direct way to test the paper’s claim would be to let the honest side respond to the manipulation. Have the authors considered a condition in which the opposing debater’s rebuttal is regenerated with access to the manipulated text?

4 comments
AuthorsOpenReview ↗

Weakness 1: Responding to the Manipulation (Reviewer DCRf)

Thank you for your thoughtful feedback. We find some overlap in your review with that of 9W4f, and we copy some relevant parts of our response here and address new concerns.

Weakness 1: The honest side has no opportunity to respond to the manipulation. A key argument for debate is that each side can expose flaws in the other’s reasoning, including rhetorical tricks. In this setup, however, manipulated turns are generated after the baseline debate is complete, and the opposing side’s turns are reused unchanged.

On responding to the manipulation: Our paper is motivated by Irving et al.'s (2018) paper on "AI Safety via Debate", which proposes using debate for training AI, arguing that "it is harder to lie than to refute a lie" (Sec. 2). Their MNIST experiment illustrates that even when debaters cannot falsify evidence, they can strategically select which evidence to present to influence a judge (Sec. 3.1). Our study similarly investigates whether human judges are influenced by misleading presentation, rather than the evidential merits of the debaters' arguments.

We agree that our experiment allows debaters to respond to claims and evidence, but does not test the full interactive setting in which the opposing debater could also challenge a rhetorical manipulation directly. Recognizing the importance of this concern raised by both you and reviewer 9W4f, we will narrow the claimed scope throughout the paper. Reviewer 9W4f additionally suggested narrowing the title and abstract, and we will take this advice and update the title to "Bias Manipulations Can Influence Human Adjudication in AI Debates".

Within this revised scope, though, we defend the design decisions of the paper. Namely, the dialogues retain an exchange of claims and rebuttals, which is central to the debate structure. Rebuttals are grounded in content plans: each response answers the opponent's accusation and the evidence it cites, and explains why that case is insufficient (Tables 15 and 26). Reviewer 9W4f also highlighted the instruction constraints over how the rebuttals are generated, but it does not remove their engagement with the opponent's claims and evidence, which are core features of debate. Our experiment preserves substantive evidential exchanges while isolating the effects of rhetorical manipulations on human adjudication.

Holding the unmanipulated side fixed serves our research question: whether interventions designed to elicit cognitive biases change human adjudication in AI debate. Each manipulated dialogue is designed to differ from its baseline only in the manipulated side's regenerated turns, following the same content plan with a bias-specific style plan. The 8.6 pp shift toward the manipulated side (p = .002) therefore estimates the effect of this intervention under a controlled evidential exchange.

With interactive generation, the opposing side's responses could also change in content, emphasis, length, and quality, beyond whether they respond directly to the manipulation. That would address the complementary question of the intervention's effect when the opponent can adapt. In our explorations of this idea earlier in the paper design, we found it much harder to maintain an adversarial debate structure when agents had greater flexibility to respond to one another. The agents often became overly agreeable, accepting the opposing side's arguments rather than challenging them (i.e., sycophantic). We will add a discussion of this limitation and the challenges of interactive debate generation to the paper and future work section.

In revising the title and framing, we will make both this contribution and the role of the fixed-response design explicit.

AuthorsOpenReview ↗

Weakness 2: Baseline Accuracy and Study Measures (Reviewer DCRf)

Both you and reviewer 9W4f raised concerns about the approximately 51% baseline accuracy. We reproduce our clarification here, while addressing your further concern about surface-level differences separately below.

Weakness 2: Baseline accuracy is close to chance, which makes it unclear what the experiment is measuring. Without any manipulation, judges are correct only about 51% of the time. This suggests that, in this setting, judges are often unable to identify the truthful side from the debate alone. Under these conditions, the observed shift may say less about how specific manipulations change judgments, and more about how any surface-level difference between the two sides can shift a decision.

On the ~51% baseline accuracy: The approximately 51% baseline accuracy is a participant-weighted average across stories. Baseline accuracy varies across the four stories (72%, 66%, 51%, and 29%; Appendix C.6), with different numbers of retained participants contributing to each estimate. The aggregate therefore does not indicate uniformly near-chance performance.

Our primary analysis (Section 5.5) asks whether bias-targeting interventions shift which side a judge selects. It models side selection as a function of where the manipulation is placed (Side A, Side B, or neither), with manipulations applied to both truthful and untruthful sides across dialogues. We are interested in the direction of the shift relative to the manipulated side. Pooled across 4 stories and 64 manipulated dialogues, judgments shift toward that side by 8.6 pp (95% CI [+2.9, +14.2], p = .002), and all four bias-specific estimates point in the same direction (+7.1 to +11.1 pp).

We also find evidence of variation in susceptibility: judges who correctly identified the biased side selected it less often than those who did not (47.7% vs. 63.2%, p = .014; Appendix C.2). This association suggests that recognition of the manipulation may be relevant to judges' susceptibility.

On whether any surface-level difference could produce the shift: We would like to emphasize how our experimental design addresses this concern. Each manipulation is applied to both sides, with Side A manipulated in half of the dialogues and Side B in the other half, and both are compared to a baseline-vs-baseline comparison. Across stories, manipulations are also balanced between the culprit and innocent rival. This allows us to examine whether judgments shift toward the manipulated side, rather than toward a particular speaker or the truthful side.

The manipulated dialogues differ from the shared baseline only in the bias-targeted intervention, with the same underlying content plans and the opposing debater's responses held fixed. Pooled across 4 stories and 64 manipulated dialogues, judgments shift toward the manipulated side by 8.6 pp (95% CI [+2.9, +14.2], p = .002). All four bias-specific estimates point in the same direction (+7.1 to +11.1 pp).

We agree that other surface-level changes introduced during regeneration could contribute to the effect. However, these interventions were specifically designed to elicit cognitive biases, and the consistent direction of the results provides evidence that bias-targeted rhetorical changes can influence human adjudication. Thank you again for engaging with our work.

AuthorsOpenReview ↗

Responses to Question (Reviewer DCRf)

Q1: The most direct way to test the paper’s claim would be to let the honest side respond to the manipulation. Have the authors considered a condition in which the opposing debater’s rebuttal is regenerated with access to the manipulated text?

We agree that this would be an interesting and important experiment. Our current design holds the opposing debater's responses fixed to isolate the effect of the manipulation on human adjudication, while preserving rebuttals that respond to the opponent's substantive claims and evidence. Regenerating the opposing rebuttal with access to the manipulated text would instead test whether an adaptive opponent can identify and counteract the manipulation. In particular, the opponent could explicitly call out the rhetorical device, which is a different mechanism from challenging the factual claims and evidence presented in the debate. Our experiment focuses on the latter mechanism, motivated by Irving et al.'s argument that debate can help judges identify the truth by exposing flaws in an opponent's claims and evidence.

Early in this project, we explored less constrained debate generation, but found that debaters frequently became overly agreeable rather than challenging one another. This made it difficult to maintain the adversarial structure of the debate. We will discuss these challenges and explicitly identify evaluating manipulations against adaptive opponents as an important direction for future work.

Thank you for the response. However, I am not convinced by the argument that holding the opponent fixed simply studies a “different mechanism.” The question the field cares about is not whether rhetorical manipulation can influence human judgment in isolation (which is well established), but whether such bias can be corrected through debate. As a result, the paper shows that bias exists in a setting where it cannot be challenged, but not whether debate fails to correct it. This makes the claim considerably narrower than the title and discussion suggest.

I find the observation that unconstrained debaters “frequently became overly agreeable” interesting, and I think it deserves further investigation. This seems to be largely an engineering issue. It could be addressed through prompt design, stronger role constraints on the opponent, etc. Overall, I think the current results establish a narrow finding that does not address whether debate can correct bias-targeting manipulation. I therefore maintain my rating of weak reject.

Reviewer_9W4fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether AI safety via debate still works when one debater uses rhetoric aimed at human cognitive biases. The authors build 68 LLM-generated debates from 4 detective stories with known culprits. Two LLM debaters each accuse a different suspect over 6 turns. Four manipulations (anchoring, fallacy oversight, pro-jargon, verbosity) are applied either to the side accusing the real culprit or to the side accusing the innocent rival, so influence and correctness can be told apart. A biased dialogue is kept only if at least one of six LLM judges moves toward the biased side and none moves away. In a preregistered Prolific study (369 participants, one debate each), the pooled manipulations move choices toward the manipulated side by 8.6 pp (p = .002). No single bias type is significant on its own (Table 1). The paper also looks at bias detection, position effects, and the free-text answers.

Critical strengths
  1. The design is clean and well documented. Putting each manipulation on the culprit side in some dialogues and on the rival side in others is the right way to separate "favors the manipulated side" from "favors the truth". The study is preregistered, deviations are listed (App. B.1), and the dialogues, code and participant data are released. Null results (Table 1, H2b) are reported as they are. Human studies of debate judges are expensive and there are not many of them, so a careful one is worth having even if the claim has to shrink.

  2. Some side results are useful for anyone planning to use human judges in debate. Judges in manipulated debates report bias only a bit more often than judges in baseline debates (55.2% vs 48.6% false alarms, Table 40), confidence does not track accuracy (App. C.4), and self-reported expertise does not protect against the manipulation (App. C.5). These back up the general worry about human oversight, whatever the size of the main effect.

Critical weaknesses

The protocol switches off the part of debate that the paper says it is testing, so the title claim does not follow. The whole idea of debate is that the honest side can point out the flaw. Here the unmanipulated side's turns are copied unchanged from the baseline dialogue (Sec. 4.1, lines 226-229), and rebuttals are generated with the instruction "Do not use generated opponent wording" (Table 26). So the unmanipulated debater never sees or answers the straw man, the jargon or the extra length. It is closer to two fixed essays where one of them gets extra rhetoric. On top of that, judges are near chance even without manipulation (about 51% baseline accuracy from Table 43), there are only 4 stories, and the corpus is pre-filtered to move LLM judges. What I think the study shows is "added rhetoric shifts non-expert judges of short LLM debates where the other side cannot respond". That is a fair finding, but "AI Safety via Debate is Compromised" is a much bigger claim.

Questions for authors
  1. If the unmanipulated debater can see and respond to the manipulated wording, does the effect hold? Without this I would ask you to narrow the title and abstract.
  2. Could you report bias-on-A vs baseline and bias-on-B vs baseline separately? In Table 36 the share choosing Side A is 70.8% (baseline), 66.9% (bias on A) and 49.7% (bias on B), so the shift seems to come only from biasing the second speaker.
  3. Does H1 hold with a mixed-effects model with random intercepts for dialogue and story, or with a bootstrap over dialogues? The treatment varies by dialogue, but the test treats the 369 judges as independent. If H1 survives this, and the side-specific effects (Q2) go into the main text with narrower claims, I would raise my rating to 3.
  4. What is the H1 result without verbosity, given that it also adds a new reason? Is there a regenerated but unbiased control for the manipulated side?
3 comments
AuthorsOpenReview ↗

Weakness 1: Responding to the Manipulation (Reviewer 9W4f)

Thank you for closely reading our paper and providing constructive feedback.

Weakness 1: The protocol switches off the part of debate that the paper says it is testing, so the title claim does not follow. The whole idea of debate is that the honest side can point out the flaw. Here the unmanipulated side's turns are copied unchanged from the baseline dialogue (Sec. 4.1, lines 226-229), and rebuttals are generated with the instruction "Do not use generated opponent wording" (Table 26). So the unmanipulated debater never sees or answers the straw man, the jargon or the extra length. It is closer to two fixed essays where one of them gets extra rhetoric.

Our paper is motivated by Irving et al.'s (2018) paper on "AI Safety via Debate", which proposes using debate for training AI, arguing that "it is harder to lie than to refute a lie" (Sec. 2). Their MNIST experiment illustrates that even when debaters cannot falsify evidence, they can strategically select which evidence to present to influence a judge (Sec. 3.1). Our study similarly investigates whether human judges are influenced by misleading presentation, rather than the evidential merits of the debaters' arguments.

We agree that our experiment allows debaters to respond to claims and evidence, but does not test the full interactive setting in which the opposing debater could also challenge a rhetorical manipulation directly. We appreciate your concerns about overclaiming, so we will narrow the claimed scope throughout the paper and update the title to "Bias Manipulations Can Influence Human Adjudication in AI Debates".

Within this revised scope, though, we defend the design decisions of the paper. Namely, the dialogues retain an exchange of claims and rebuttals, which is central to the debate structure. Rebuttals are grounded in content plans: each response answers the opponent's accusation and the evidence it cites, and explains why that case is insufficient (Tables 15 and 26). The instruction "Do not use generated opponent wording" constrains how the rebuttals are generated, but it does not remove their engagement with the opponent's claims and evidence, which are core features of debate. Our experiment preserves substantive evidential exchanges while isolating the effects of rhetorical manipulations on human adjudication.

Holding the unmanipulated side fixed serves our research question: whether interventions designed to elicit cognitive biases change human adjudication in AI debate. Each manipulated dialogue is designed to differ from its baseline only in the manipulated side's regenerated turns, following the same content plan with a bias-specific style plan. The 8.6 pp shift toward the manipulated side (p = .002) therefore estimates the effect of this intervention under a controlled evidential exchange.

With interactive generation, the opposing side's responses could also change in content, emphasis, length, and quality, beyond whether they respond directly to the manipulation. That would address the complementary question of the intervention's effect when the opponent can adapt. In our explorations of this idea earlier in the paper design, we found it much harder to maintain an adversarial debate structure when agents had greater flexibility to respond to one another. The agents often became overly agreeable, accepting the opposing side's arguments rather than challenging them (i.e., sycophantic). We will add a discussion of this limitation and the challenges of interactive debate generation to the paper and future work section.

In revising the title and framing, we will make both this contribution and the role of the fixed-response design explicit.

AuthorsOpenReview ↗

Weakness 2: Study Scope (Reviewer 9W4f)

Weakness 2: On top of that, judges are near chance even without manipulation (about 51% baseline accuracy from Table 43), there are only 4 stories, and the corpus is pre-filtered to move LLM judges. What I think the study shows is "added rhetoric shifts non-expert judges of short LLM debates where the other side cannot respond". That is a fair finding, but "AI Safety via Debate is Compromised" is a much bigger claim.

On the ~51% baseline accuracy: The approximately 51% baseline accuracy is a participant-weighted average across stories. Baseline accuracy varies across the four stories (72%, 66%, 51%, and 29%; Appendix C.6), with different numbers of retained participants contributing to each estimate. The aggregate therefore does not indicate uniformly near-chance performance.

Our primary analysis (Section 5.5) asks whether bias-targeting interventions shift which side a judge selects. It models side selection as a function of where the manipulation is placed (Side A, Side B, or neither), with manipulations applied to both truthful and untruthful sides across dialogues. We are interested in the direction of the shift relative to the manipulated side. Pooled across 4 stories and 64 manipulated dialogues, judgments shift toward that side by 8.6 pp (95% CI [+2.9, +14.2], p = .002), and all four bias-specific estimates point in the same direction (+7.1 to +11.1 pp).

We also find evidence of variation in susceptibility: judges who correctly identified the biased side selected it less often than those who did not (47.7% vs. 63.2%, p = .014; Appendix C.2). This association suggests that recognition of the manipulation may be relevant to judges' susceptibility.

On the number of stories: The choice of 4 stories was a practical decision based on the resources available for our human study. Our design includes 17 dialogues per story (1 baseline and 16 manipulated), with a planned 100 participants per story. We chose 4 stories to balance variation across stories with sufficient judgments per condition, within our planned N = 400 budget.

On the LLM-as-judge pre-selection: We used LLM judges to check whether our generated bias manipulations could influence adjudication in theory before running an expensive human study. Specifically, a biased dialogue passes if at least one of six LLM judges shifts toward the manipulated side and none shifts away; otherwise, it is regenerated (but it is not dropped, Section 4.2). Generating effective bias manipulations with LLMs is difficult, and this step helped us check that the manipulations were at least somewhat effective. The filter acts on generated manipulations, not on which stories or sides enter the study.

LLM-judge biases do not necessarily transfer to humans [26], so whether these interventions also affect human adjudication remains an empirical question. We recognize that this procedure selects for manipulations that affect LLM judges, but our goal is to establish whether human judges can be influenced by these manipulations, not how frequently this occurs. As stated in Section 7.1, our results do not determine how often such strategies would arise outside laboratory conditions. We will clarify this distinction in the revision.

AuthorsOpenReview ↗

Responses to Questions (Reviewer 9W4f)

Q1: If the unmanipulated debater can see and respond to the manipulated wording, does the effect hold? Without this I would ask you to narrow the title and abstract.

As explained above, the debater cannot see or respond to the manipulated wording; it can only respond to the factual claims made by the other agent. We propose to change the title to “Bias Manipulations Can Influence Human Adjudication in AI Debates” and are open to further suggestions. We will also update the abstract to narrow the claims.

Q2: Could you report bias-on-A vs baseline and bias-on-B vs baseline separately? In Table 36 the share choosing Side A is 70.8% (baseline), 66.9% (bias on A) and 49.7% (bias on B), so the shift seems to come only from biasing the second speaker.

We will report the side-specific results in Section 6.1 and an additional Figure 4. Bias on Side A changed its support from 70.8% to 66.9% (−3.9 pp, 95% CI [−16.8, +8.8]), whereas bias on Side B increased its support from 29.2% to 50.3% (+21.2 pp, 95% CI [+7.9, +34.3]). Thus, the aggregate shift toward the manipulated side (+8.6 pp, p = .002) is driven primarily by interventions on Side B.

This asymmetry may partly reflect the stronger baseline preference for Side A, which leaves more room for interventions on Side B to increase its support. However, our design does not establish the explanation for this asymmetry, and we cannot rule out other explanations, including speaker-order effects. We will make explicit in the main text that the aggregate result does not imply comparable effects across the two placements.

Q3: Does H1 hold with a mixed-effects model with random intercepts for dialogue and story, or with a bootstrap over dialogues? The treatment varies by dialogue, but the test treats the 369 judges as independent.

Yes. We will report the full updates in Section 6.1 and Appendix C.1, including all three clustering-aware analyses in an additional Table 38. All three analyses replicate the result: GEE β = 0.366 (p = .003), GLMM β = 0.387 (p = .002), and a dialogue-level cluster bootstrap gives 95% CI [+2.8, +14.3] pp. We hope these additional analyses address your statistical concerns and warrant reconsideration of your assessment.

Q4: What is the H1 result without verbosity, given that it also adds a new reason? Is there a regenerated but unbiased control for the manipulated side?

Regarding the regeneration control: We did not include a regenerated-but-unbiased control for the manipulated side, so we cannot completely separate the effects of bias-targeted rewriting from those of regeneration more generally. However, the generated arguments follow the same content plans as the originals, constraining changes to their substantive claims and evidence. We will acknowledge this limitation in the paper.

Regarding verbosity: Excluding verbosity, judges selected the manipulated side in 57.8% of adjudications (130/225). We repeated the clustering-aware H1 analysis on these observations together with all 72 baseline observations (total N = 297; 52 dialogues). The estimated shift toward the manipulated side was +7.8 pp (dialogue-level cluster bootstrap 95% CI [+1.4, +13.9]), and the dialogue-clustered GEE supported a positive effect (β = 0.338, OR = 1.40, 95% CI [1.07, 1.83], two-sided p = .013). Thus, the aggregate finding persists after excluding verbosity. The remaining interventions—anchoring, jargon, and fallacy—rewrite the original argument rather than add a new reason, addressing the additional-reason concern specific to verbosity. This does not, however, resolve the broader regeneration confound. We will report these results in the revised paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Final Remarks for Reviewers and the Area Chair

We thank the reviewers for their thoughtful engagement and are pleased that they recognized the following strengths of our work:

  • We provide a reusable framework for testing whether rhetoric sways human debate judges. This question is central to debate as a scalable oversight method because the outcome depends on how a judge weighs competing arguments. Our generation pipeline combines evidence-indexed stories, explicit content and style plans, and controlled interventions, giving researchers a reproducible way to study this question across a larger set of debates. (DCRf)

  • Our design separates support for the manipulation from support for the correct answer. We apply each intervention to the side accusing the true culprit in some dialogues and to the side accusing an innocent rival in others. This lets us measure whether support shifts toward the side using a tactic, whether that shift favors a correct or an incorrect verdict. (9W4f)

  • Our preregistered human study contributes reproducible evidence in an area where human evaluations remain scarce. We organize the study around explicit hypotheses and document how we tested them, making the evaluation accessible to readers outside the field (A7es). We disclose deviations from the preregistration, report null findings alongside positive results, and release the dialogues, code, and participant data so others can independently examine and reproduce the analysis (9W4f).

  • Our findings on bias detection, confidence, and expertise reveal limitations of human adjudication. Judges reported bias only slightly more often in manipulated debates than in baseline debates, and their confidence did not reliably track whether they identified the true culprit. We also found no clear evidence that self-reported expertise protected judges against manipulation. These findings help researchers assess how much weight to give a judge’s confidence, reported expertise, or perception of bias when interpreting a verdict. (9W4f)

We appreciate the reviewers’ feedback and have revised the manuscript to address their concerns and questions:

  • We clarified what each debater responds to and narrowed our claims (9W4f W1/Q1, DCRf W1/Q1, A7es follow-up). We revised the title, abstract, and Section 1 to limit our claim to human adjudication under the debate protocol we tested. In Section 4.1, we clarified that each debater challenges the opposing accusation and evidence. Giving judges the same counterargument across conditions lets us attribute the vote shift to the bias-targeted rewrite. Regenerating the opponent’s reply could also change which evidence it emphasizes or how effectively it argues. In Section 7, we added a discussion of whether an opponent that sees the manipulated wording can recognize and counter the tactic itself.

  • We added speaker-specific estimates and tested the robustness of the pooled result (9W4f Q2–Q4, A7es W2 and follow-up). We added Figure 3 and expanded Section 6.1 to show that the second speaker gained 21.2 percentage points, bringing the vote to an even split, while the first speaker saw no significant gain. In Appendix C.1, we added analyses showing that the pooled result (+8.6 percentage points, p = .002) holds when we account for clustering and when we exclude verbosity. In our reply to 9W4f Q4, we clarified that without a separately regenerated, unbiased control, the estimate includes incidental wording changes from the rewrite.

  • We clarified what near-chance baseline accuracy means for our analysis and added a human–LLM comparison (9W4f W2, DCRf W2, A7es Q2). We clarified the rationale for using four stories in Section 5.1 and how we measure shifts in side selection in Section 5.2. A gain for the manipulated side can improve accuracy when it accuses the culprit and reduce accuracy when it accuses the innocent rival. We also clarified the role of LLM screening in Section 4.2 and added Appendix C.7 to compare human judgments with decisions from the six LLM judges on the same debates.

  • We expanded the discussion of prior work and the analysis of human judgments (A7es W1.1/W1.2, W2/Q1, and follow-up). We incorporated the reviewer’s suggested references into Section 2 and revised Sections 1–2 to explain what the debate setting adds. Prior work shows that persuasive AI outputs can mislead people. Debate gives judges an opportunity to hear an opponent challenge the evidence behind a claim. Our study asks whether changing how one side presents its case still shifts the verdict when judges hear that challenge. In Section 6.2 and Appendix C.5, we expanded the expertise analysis to examine whether self-reported expertise predicts selection of the manipulated side as well as accuracy. We also brought findings from participants’ written explanations into Section 6.2, with the full analysis in Appendix C.6.

Please find more details directly under the reviewers’ comments.

For the final consideration of the Area Chair and reviewers: We hope you will consider the value of this work as a resource for the scalable oversight community. How specific rhetorical tactics influence human judges in AI debate remains understudied. Our large-scale, preregistered human study builds on research on persuasion in more general settings by testing these tactics in debates where judges hear both an argument and a rebuttal. We developed a clean, reproducible generation pipeline that lets researchers vary a rhetorical tactic while preserving the opposing case. The resulting dataset pairs judges’ verdicts with their confidence ratings, self-reported experience, bias-detection responses, and written explanations. By releasing the dialogues, participant data, and generation code, we make it possible for others to examine differences across judges and interventions and pursue questions beyond our original hypotheses. Together, these materials provide a shared empirical starting point for testing which debate protocols help human judges identify better-supported arguments.

Thank you very much for your time and thoughtful feedback on our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.