acceptodds
Under review as a conference paper at ICLR 2027

Characterizing Rhetorical Misalignment in Decision-Making with Language Models

Abstract

Human decision-making is often shaped by a range of well-documented cognitive biases. As large language models (LLMs) become increasingly integrated into high-stakes human-AI decision-making, it is important to understand whether their assistance can amplify these biases and lead to harmful consequences. In this paper, we study rhetorical misalignment: a failure mode where a language model presents information in rhetorically inappropriate forms, thereby inducing suboptimal human decisions in a given decision context. We conduct a human-subject experiment using decision problems curated from the United States Medical Licensing Examination (USMLE). We observe an average harmful flip rate of 2.81% across models, where participants revise initially correct answers to incorrect ones after viewing responses generated by language models. Participants' rationales suggest possible mechanisms related to the language used by the models, consistent with cognitive biases including anchoring, authority bias, and loss aversion. Motivated by the observations, we develop a decision-theoretic framework that characterizes rhetorical misalignment through the expected utility gap between rational and behavioral use of the information provided by language models. To empirically measure rhetorical misalignment, we instantiate the framework using decision-makers simulated by language models. Overall, our findings reveal a safety concern previously underexplored in high-stakes domains: language models can be factually aligned yet still induce harm through its rhetorical presentation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI-Assisted Decision-Making, Reward Misspecification, Human Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_cmNUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies rhetorical misalignment: loss in decision quality attributed to the linguistic presentation of AI-provided information. A motivating study collects 939 responses from 162 participants with self-reported medical training or experience on USMLE-derived questions. It reports improved aggregate accuracy after AI assistance, alongside harmful correct-to-incorrect revisions and qualitative accounts consistent with several cognitive biases. The paper then defines a utility gap between ideal rational and behavioral decision policies and implements two DeepSeek-V3.1-based simulators prompted to follow Bayesian reasoning or exhibit cognitive biases. Experiments compare their accuracy and disagreement across model-generated explanations, including explanations constructed from a shared claim pool, naturalistic generations, and post-training variants.

Critical strengths
  1. A consequential evaluation target grounded in human observations. Studying the quality of downstream decisions, including harmful revisions despite overall improvement, is more informative than reporting model accuracy alone. The human study provides concrete motivating evidence: 939 responses from 162 participants, with accuracy increasing from 45.7% to 57.5% (p. 3). Reporting helpful and harmful changes separately is useful because a net improvement can coexist with important failure cases.

  2. An explicit attempt to hold information fixed and distinguish theory from simulation. The shared-claim-pool experiment addresses an important confound, and the paper explicitly acknowledges that a prompted rational simulator is only an approximation (pp. 7–9). The multiple model and training-variant comparisons could become a useful screening methodology if semantic preservation, statistical reliability, and the simulator's interpretation are established.

Critical weaknesses
  1. The evidence does not identify the central presentation-only causal claim. In the human experiment, participants answer before and after reading AI analyses; the reported design does not include a randomized content-matched alternative framing or a no-AI reconsideration condition. The 19 harmful revisions following a correct model answer are important observations, but answer correctness does not establish that every supporting claim is factual or that rhetoric, rather than ambiguity, content, or ordinary reconsideration, caused the revision. Rationales are collected after the correct answer is revealed, further limiting causal interpretations of the coded mechanisms. The controlled simulation also does not fully resolve this issue. Table 8’s claim-pool construction examples change “pathognomonic” to “characteristic” and add “body image distortion” to an explanation originally mentioning denial. This preprocessing changes diagnostic strength or explanatory content. These are source-to-pool examples, so they do not by themselves show that every final model-versus-neutral contrast differs in content; however, they also do not validate the assumed semantic equivalence. Actual paired outputs and a preservation audit are needed. Tables 2–3 do not explicitly report both the model and neutral representations and their paired difference, despite Section 4.2 claiming that this contrast isolates a causal effect. Consequently, the conclusion that human decisions were worsened through language use alone is stronger than the evidence presented.

  2. The proposed measure is not sufficiently validated, and central numerical results are inconsistent. The ideal policy conditions on the full information pair (X0, X1), although an arbitrary utterance map need not preserve that information. Unless sufficient information is recoverable from the message, its gap from a message-limited listener can include information loss. Even with lossless messages, a gap under one representation can reflect generic policy or reasoning differences rather than presentation-specific harm. Appendix E explicitly instructs the behavioral simulator to exhibit six biases; an LLM's subsequent labeling of its reasoning mainly tests prompt compliance. Appendix D does not establish known-posterior Bayesian accuracy, calibration, action-level framing invariance, or correspondence with human framing effects. Table 6 reports high targeted-bias classification rates even for the rational simulator (0.69–0.86 under the biased framings), without a clear operational interpretation or uncertainty.

The numerical problems independently undermine the measurement. For common paired trials, the accuracy difference must equal P(rational correct, behavioral wrong) minus P(rational wrong, behavioral correct). In Table 2, DeepSeek-V3.1 has accuracy difference 89.2% − 89.8% = −0.6 percentage points, whereas the directional columns imply 1.9% − 1.1% = +0.8 points. Other discrepancies include Gemini (−0.6 versus −0.3 points) and Tülu-SFT (+5.8 versus +5.0 points); these exceed rounding tolerance. Moreover, the human-study totals and conditional counts need reconciliation: using the stated 939 equally weighted trials, 45.7% initial accuracy requires 429 initially correct trials; 19 harmful changes at a 5.6% conditional rate implies 337–342 initially-human-correct/model-correct trials, and 32 helpful changes at a 17.4% conditional rate implies 184 initially-human-wrong/model-wrong trials. Thus the model-correct count must be 663–668 (70.61%–71.14%), whereas the reported 65.0% implies 610 trials. These ranges allow rounding to one decimal place and assume that the statistics share a sample and weighting. No uncertainty intervals or formal tests support the claimed differences or the statement that larger models do not yield significant controlled gaps. These issues prevent reliable interpretation of the proposed metric and model comparisons.

Questions for authors

The following are clarification or evidence requests that could materially change the assessment, rather than suggestions for unrelated future work.

  1. Numerical reconciliation: Please provide the per-model paired correctness contingency counts underlying Tables 2–4, identify every denominator or exclusion, and reconcile Table 2's accuracy differences with its directional correctness columns. For the human study, please provide the initial-human-correctness × model-correctness × final-human-correctness counts and reconcile the conditional percentages/counts with 939 trials, 45.7% initial accuracy, and 65.0% model accuracy. Were different subsets used?

  2. Identification: Where are the paired results for the model representation and its neutral counterpart specified in Section 4.2? Please provide that contrast and evidence that the two preserve all decision-relevant claims, uncertainty, and recommendation content, including how Table 8's semantic changes are handled. Is there a human content-matched framing comparison? If not, would the authors restrict the human-study conclusion to descriptive evidence and avoid claiming identified human harm from language alone?

  3. Simulator validity: What evidence distinguishes the reported gap from baseline differences induced by the rational/behavioral system prompts? Please clarify Table 6's neutral-condition metric and provide action-level framing effects and an objectively computable Bayesian benchmark, with uncertainty. Can the measured effects be reproduced with another simulator or related to human effects on matched items? Please also clarify the information-sufficiency assumption connecting the utterance map to conditioning on (X0, X1).

  4. Statistical support: What are the exact sample sizes, repeated-generation settings, and confidence intervals for each comparison? Please use paired analyses for simulator outcomes, account for repeated participants and questions in the human study, and explain the inferential basis and multiplicity treatment for model-size and training-stage comparisons. A nonsignificant result should not be interpreted as evidence of negligible effect without an equivalence analysis or sufficiently informative interval.

Reviewer_FNB2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces the rhetorical misalignment, where an LLM's presentation of information pushes people toward worse decisions. After reading model analyses, 162 participants with self-reported medical training revised their USMLE answers at an average rate of 27.58% across models, with harmful revisions averaging 2.81% of trials.

Critical strengths
  1. The human study speaks to a practical safety question for AI in health care. Participants with self-reported medical training answered USMLE questions before and after reading model analyses (939 annotations from 162 annotators), and harmful changes are reported separately from helpful ones, together with written reasons for each change. Such data reveal real human reactions to model explanations, something that model accuracy by itself does not measure.

  2. The paper makes a clear and relevant point: being factually right is not enough if the way information is presented steers people in the wrong direction. The decision-theoretic framing separates the value of the information from the effect of its wording, which is a helpful lens for evaluating decision-support systems.

Critical weaknesses
  1. It would be better to show more concretely that the wording, rather than the content, drives the harmful changes in the human study. The design lacks a condition presenting the same information in neutral wording, and among answers that were initially correct, harmful revisions occur at 10.3% when the model answer is wrong compared with 5.6% when it is right, so some harmful changes may come from wrong model answers rather than from the wording.

  2. It would be better to rely less on prompting for the main measurement. Both decision-makers are DeepSeek-V3.1 under two different prompts, and the behavioral prompt explicitly instructs the model to display anchoring, availability, confirmation bias, conservatism, overconfidence, and loss aversion, so the measured gap may largely mirror what the prompts ask for.

  3. It would be better to connect the theory and the experiments more closely. The framework uses a binary state, a binary action with a safe default, and payoffs r+ and r−, while the experiments use multiple-choice USMLE questions and report accuracy and disagreement rates without stating how these map to the framework's payoffs.

  4. It would be better to align the conclusion with what was measured. While the conclusion says that LLMs can induce suboptimal human decisions through language use alone, the information is not held fixed in the human study, and the controlled evidence is drawn from simulated decision-makers instead of people.

Questions for authors
  1. Tables 2 to 4 report results for model-generated text. Could the authors report the same measures for the neutral representation described in Section 4.2, which seems necessary to isolate the effect of wording?

  2. Which setting does Table 4 use? For Llama-3.1-8B-Instruct, the gap is +6.9% in Table 4 but +6.3% in both Table 2 and Table 3.

  3. Could the authors report the number of coded rationales, the measure of agreement between annotators, and the count of coded cases behind the 58.9% authority-bias figure?

Reviewer_HaHeOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether the way in which LLM explanations are worded can lead to humans making poor decisions. In a study involving USMLE questions, participants changed a correct answer to an incorrect one in considerable trials after reading the explanations. The authors formalise this phenomenon as the difference in expected payoff between a Bayesian decision-maker and a biased one. They simulated both scenarios using an LLM.

Critical strengths

This paper reports correct-to-incorrect changes , showing that such changes occur even for models. Comparing eight models under the same protocol is also useful.

They then compare the wording of different models with the information held fixed. This is a worthwhile attempt to separate content from presentation.

This paper collects participants’ written reasons for each change and their highlights of the passages that influenced them. This provides qualitative evidence of pathways such as deference to authority.

Critical weaknesses

This paper does not cite prior studies that randomized the wording of AI outputs and measured human decisions. Examples include Kim et al. (2024), who varied LLM uncertainty expressions on medical questions, and Danry et al. (2025), who varied whether explanations were shown. As a result, the paper’s novelty is unclear.

The human study has no condition that manipulates wording. It therefore cannot tell whether correct-to-incorrect changes stem from model errors or from wording. The paper also does not compare its results with a randomized trial that measured harm to physicians from recommendations containing planted errors (Qazi et al., 2025).

In this paper’s formulation, the Bayesian decision-maker conditions directly on the information, so wording never enters the benchmark, and no result beyond the definitions is derived. The measurement is an accuracy gap between two prompts given to the same LLM. The authors do not check whether this gap agrees with the human-study results.

Questions for authors

What do the authors show that Kim et al. (2024) and Steyvers et al. (2025) did not?

https://arxiv.org/abs/2405.00623 What large language models know and what people think they know

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.