acceptodds
Under review as a conference paper at ICLR 2027

Framing the Narrative: Ideological Mimicry in Large Language Models

Abstract

Large language models (LLMs) are increasingly used to answer questions about politically contentious issues, yet evaluations typically treat a model’s stance as a relatively stable property. Real users, however, communicate political signals through their terminology, assumptions, and personal context. We investigate whether such signals produce ideological mimicry: systematic shifts in the political stance expressed by an LLM toward the position conveyed by the interaction. If LLMs adapt their responses to these signals, they risk creating personalised political information environments in which users with opposing views receive systematically different accounts of the same issue, potentially reinforcing existing divisions. We build the Poli-SHIFT dataset and evaluation framework and assess seven open-weight LLMs across ten contentious political topics in the United States, United Kingdom, and Australia, systematically manipulating contested terminology, politically valenced premises, and user information, and eliciting responses in both multiple-choice and open-text formats. Across models, we find robust evidence that prompt framing shapes the political stance of LLM outputs. Changing terminology alone reverses which side of an issue a model supports in 16.9% of matched comparisons. Stated political ideology also systematically shifts responses toward the user’s position. These findings show that political stance is not a fixed property of LLMs; the views expressed are conditional on the interaction with the user. As LLMs become increasingly personalised sources of information, such interaction-dependent adaptation could contribute to political information environments that reinforce users' existing perspectives.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, LLM evaluation, political bias, prompt sensitivity, political framing, sycophancy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RbVCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper studies how LLMs change their expressed political views in response to question wording, political assumptions, and user information. It introduces POLI-SHIFT, a dataset and evaluation framework covering ten political topics in the United States, United Kingdom, and Australia. The experiments evaluate seven open-weight models using multiple-choice and short written answers, with six models included in the biography analysis.

The evaluation includes five repeated generations for the wording and premise experiments, statistical tests with checks for dependent observations, and LLM scoring validated against 200 human-rated responses. The authors report that answers tend to move toward the political position suggested by the prompt or the user’s stated ideology. Other demographic details have smaller and less consistent effects.

Critical strengths
  1. The paper provides a broad evaluation across seven models, ten political topics, and three countries. It examines question wording, political premises, and user biographies. The wording and premise experiments also include both multiple-choice and short written answers. This coverage helps show how political responses vary across models, topics, and ways of asking questions, making the findings useful for evaluating political bias.

  2. The biography experiment uses 2,700 profiles that systematically vary ideology, race, gender, age, and education while keeping the survey questions fixed. This design helps distinguish the influence of stated political views from other personal details. The finding that ideology has a stronger and more consistent influence than most demographic cues provides useful insight into how personalization can shape political answers.

Critical weaknesses
  1. The reported results contain unexplained differences in sample counts, values, and definitions. Table 18 implies about 15,900 complete three-judge ratings, but Table 17 uses only 12,454, 9,339, and 8,915 responses. Figure 18 shows Gemma 3’s open-text effects at approximately 0.47 and 1.46, while Table 5 reports 0.44 and 1.51. Figure 6 shows Qwen and Llama 4 reversal rates of approximately 0.9% and 40.0%, while the text gives 1.7% and 39.5%. Judge correlations range from 0.78–0.88 in the text but 0.77–0.85 in Table 17. Figure 18 also measures contradictions using repeated-run pairs, while Table 16 uses prompts. These differences need reconciliation because they make the evaluation and reported results difficult to verify.

  2. The framing experiments sometimes change the question’s meaning or task alongside the political cue. Table 2 replaces “Russia’s unprovoked invasion” with “Russia’s special military operation,” changing the assumptions conveyed. In Table 3, multiple-choice questions ask models to assess a statement, while written-answer questions ask “How…” and assume the statement is true. These changes weaken the claim that terminology alone or response format explains the measured differences. Controls that preserve the meaning and task are needed to support that interpretation.

Questions for authors
  1. How do Table 17’s judge-pair counts of 12,454, 9,339, and 8,915 fit with Table 18’s nearly complete three-judge coverage? Does a different response set also explain the judge-correlation ranges of 0.78–0.88 in the text and 0.77–0.85 in Table 17?

  2. Does Figure 18 use a different sample or averaging method from Tables 5 and 16? Please explain Gemma 3’s differing open-text effects and clarify whether contradiction rates count prompts or repeated-run pairs.

  3. Why do the text and Figure 6 give different reversal rates for Qwen’s open-text answers and Llama 4’s multiple-choice answers? Were these calculated from individual runs or responses averaged across five runs?

  4. What checks show that the terminology variants preserve the question’s meaning? How do you separate response-format effects from the difference between assessing a statement and answering a “How…” question that assumes it is true?

  5. Does the biography-related ideology effect remain when comparing the same questions answered across ideology conditions, given Llama 4’s refusal rates of approximately 10% for conservative profiles and 2% for liberal profiles?

11 comments

Thank you for identifying these inconsistencies and for the comments on treatment validity, refusal-related selection, and release safeguards. In response, we reran the judging pipeline on the frozen data, regenerated the affected figures and tables from a common analysis set, explicitly separated item- and run-level reversal statistics, and added matched-response and worst-case analyses of biography non-response.

1. Reporting inconsistencies.

We agree that the discrepancies in counts and definitions made the results difficult to verify. Table 17 reflected an earlier partial-overlap judging output, whereas Table 18 reflected substantially more complete coverage. We reran the judging pipeline on the frozen analysis data and regenerated both tables from the same response set. The revised figure and table now use the same frozen response set and aggregation. Contradiction rate is defined at the prompt level: a repeated prompt is counted as contradictory if its generations fall on opposite sides of the neutral midpoint. It is therefore not a count of all pairwise combinations of repeated runs.

For the seven open-weight models, the judging pipeline now has essentially complete coverage: 99.7% of open-text responses are scored by all three judges at the run level, with at least 99.6% full-jury coverage at the item level. The three pairwise comparisons contain 16,067, 16,049, and 16,068 responses (out of a total of 16,100). On this common response set, Pearson correlations are 0.75, 0.70, and 0.79 and same-side agreement is 73.0%, 70.2%, and 78.1%. These replace the previously reported statistics based on smaller pairwise overlaps, and the corresponding tables and main-text values have been regenerated from the same response set.

Second, the other cited discrepancies have been regenerated from the same frozen data. Gemma 3's open-text terminology and premise effects are 0.414 and 1.555, respectively (reported as 0.41 and 1.56), and the corresponding figure and table now use the same values. Its open-text terminology and premise contradiction rates are 1.7% and 7.0%, respectively.

Third, our check identified inconsistencies in how item-level and run-level reversal quantities were distinguished, as well as several outdated model-specific values. The headline reversal rate for the seven open-weight models is 16.89% at the item level and 12.74% at the run level. Qwen's open-text reversal rate is 4.35% at the item level versus 2.26% at the run level, while Llama 4's MCQ rate is 40.0% versus 38.6%. We now state the aggregation level explicitly throughout and regenerated the affected figures and tables from the same analysis.

We have also corrected the Qwen reference and added exact model identifiers and inference settings in the appendix.

Corrected values in the revised draft.

QuantityRevised value
Judge pair N16,067 / 16,049 / 16,068
Judge Pearson r.75 / .70 / .79
Judge same-side agreement73.0 / 70.2 / 78.1%
Gemma 3 OT terminology effect0.414
Gemma 3 OT premise effect1.555
Gemma 3 OT contradiction rate, term / premise1.7 / 7.0%
Reversal rate (open-weight models), item / run level16.89 / 12.74%
Qwen OT reversal, item / run4.35 / 2.26%
Llama 4 MCQ reversal, item / run40.0 / 38.6%

2. Meaning and task differences across framing conditions.

We agree that the premise condition does not test semantically equivalent statements. The pro- and anti-premise prompts intentionally introduce different politically valenced assumptions about the same broad issue. The intended test is whether opposing assumptions produce systematic directional differences in model responses, not whether a minimal wording change alters an otherwise identical proposition. We have clarified this distinction and narrowed the interpretation accordingly: premise effects may reflect engagement with or endorsement of the supplied assumption as well as differences in the political position expressed.

Importantly, however, the open-text outcome itself is held fixed across the paired conditions. The judges do not evaluate each response for agreement with the particular premise that generated it; instead, responses from both premise conditions are mapped onto the same five-point pro-anti political stance scale for that issue. Thus, although the prompt changes the assumption supplied to the model, the evaluative standard used to compare the resulting responses does not change. This does not make the premise treatment a clean estimate of stance change independent of premise engagement, but it means the observed pro-anti difference is not mechanically produced by changing the outcome definition.

The MCQ and open-text conditions intentionally operationalise premise framing differently. In MCQ, the politically valenced proposition is stated explicitly and the model is asked to agree or disagree. In open text, the premise is presupposed in a leading “How…” question and the model generates a response conditional on that assumption. We therefore do not interpret differences in effect magnitude as a clean causal effect of response format alone. The relevant result is that opposing premises produce the same directional difference under both elicitation structures.

The terminology condition does hold the substantive question fixed, while politically contested labels used to describe the same underlying actor, event, or issue are substituted. We agree that these labels are not necessarily strictly semantically equivalent: some carry different characterisations or presuppositions, which form part of the political signal being tested. We have clarified this distinction and no longer describe the substituted expressions themselves as semantically equivalent.

The premise and terminology treatments are designed to represent a plausible feature of user-LLM interaction, where users may convey political perspective through both their choice of terminology and the assumptions embedded in their questions.

3. Differential refusals and the biography effect.

We repeated the ideology analysis restricted to cells answered under all three ideology conditions and added worst-case sensitivity analyses, where missing values were filled with the extreme case. In the case of a refusal on the first run, we also reran each prompt up to five additional times (although this led to modest decreases in refusal).

For Llama 4 after the retry procedure, the liberal-conservative gap went from 1.121 points, when using the full dataset, to 1.081 points in the restricted, fully matched set: the directly matched liberal-conservative estimate is 1.084 (95% CI [0.91, 1.25]). Under worst-case assignment of the remaining missing responses, the lower bound remains positive at 0.77. For DeepSeek, Gemma 3, Mistral, and Qwen, 98-100% of cells are retained and matching changes the estimated ideology gap negligibly; Command A changes from 0.828 to 0.809.

4. Ethics and release safeguards.

All biographies are synthetic and Poli-SHIFT is designed for diagnostic auditing rather than political persuasion. We have expanded the Ethics Statement to state that the release documentation will explicitly identify targeted political persuasion and user-specific political-message optimisation as out-of-scope uses, and that the released materials are intended for auditing and reproducibility.

Thank you for the detailed response. The corrected reporting and additional refusal analyses address several concerns. In particular, the matched-response and worst-case analyses strengthen the evidence for the biography-related ideology effect. Two concerns remain.

  1. The framing experiments still combine political cues with changes in meaning, assumptions, or task. I appreciate the narrower interpretation, but using a common stance scale does not separate political accommodation from ordinary responses to different premises. Similarly, the different tasks used for multiple-choice and written answers limit what their comparison establishes. These limitations weaken the evidence supporting the paper’s interpretation of ideological mimicry.

  2. The revised judging results leave uncertainty about measurement reliability. The updated correlations of 0.70–0.79 and same-side agreement of 70.2–78.1% make sensitivity to judge choice important. The response does not establish whether the main effect sizes and reversal findings remain consistent across individual judges, or how the corrected ratings relate to the existing human validation.

The dataset and findings remain useful, and I acknowledge the improvements described in the response. The remaining concerns affect the interpretation and reliability of the central results.

Reviewer_honbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

POLI-SHIFT evaluates how political wording and user information change LLM responses across ten topics and three English-speaking countries. The study tests seven models using contested terminology and premise framing in both multiple-choice and open-text formats, and six models using synthetic biographies with otherwise unchanged survey questions. The authors find directional effects in all 28 primary framing comparisons, with 27 remaining significant under topic-clustered inference, and report a 16.9% reversal rate for matched terminology pairs. Explicit user ideology has a much larger effect than the other biography attributes. Open-text scores come from an LLM judging panel, checked against human annotations on 200 responses.

Critical strengths
  1. The biography experiment gives a reasonably direct test of user conditioning because the survey question stays fixed while the stated user profile changes. The liberal–conservative gaps of roughly 0.82–1.48 scale points across six models are substantial. This is the strongest evidence for the paper's argument that a model's measured political stance depends on the interaction.

  2. Testing both multiple-choice and open-text responses is useful. The different patterns for terminology and premise framing show that elicitation format matters, and the effects are not confined to forced-choice questionnaires. The authors also acknowledge that the prediction of uniformly larger open-text effects was not supported.

  3. The analysis includes several worthwhile robustness checks. Appendix I accounts for repeated questions and profiles through clustering, while the other appendices report repeated-generation variation, refusals, and judge agreement. The human validation of open-text scores is also helpful, even though it does not settle every measurement question below.

Critical weaknesses
  1. The premise condition does not cleanly measure a change in political stance. In Appendix B, “Immigrants contribute more in taxes than they receive in public benefits” and “Immigration depresses wages and takes jobs” are different empirical claims, not opposing versions of the same policy question. A model could answer them differently without changing its political commitments. The open-text versions further ask how each claim occurs, encouraging acceptance of the premise. The terminology example also changes “unprovoked invasion” to “special military operation,” which carries a different characterization of the event. Please show how semantic equivalence was assessed and distinguish factual-premise endorsement from movement on a common policy position.

  2. The judge-coverage numbers do not appear to describe the same data. Table 18 reports 96–100% complete three-judge coverage, but Table 17 gives pairwise overlaps of 12,454, 9,339, and 8,915 responses. These differences are too large to follow from the reported near-complete coverage without some additional filtering or aggregation. Please reconcile the counts and the judge-correlation range, which also differs between the main text and Table 17. Exact model identifiers are needed as well: the evaluated “Qwen 3.6 27B” is linked to a Qwen3.6-35B-A3B reference.

  3. Differential refusals complicate the biography comparisons. Appendix O reports that Llama 4 refuses about 10% of conservative profiles versus 2% of liberal profiles. Excluding those responses means the stance comparison is conditional on a willingness to answer that itself depends on ideology. Please report results on matched question/profile combinations with valid answers in both conditions, along with a sensitivity analysis for the excluded responses. The refusal analysis is informative, but reporting it separately does not resolve this selection issue.

  4. The 16.9% reversal rate needs validation at the level of matched pairs. Human–jury agreement on political side is 80.5% (Appendix P), and marginal agreement does not tell us how reliably the scorer identifies a reversal between two responses. Please specify how the five generations and any incomplete judge panels enter the reversal calculation, and check a sample of the claimed open-text reversals with human raters. This would help distinguish actual stance changes from instability near the neutral boundary.

Questions for authors

See weaknesses 1–4.

8 comments
AuthorsOpenReview ↗

Terminology and premise framing

Thank you for these careful comments. In response to the review, we reran the full judging pipeline on the frozen data, added matched-response, neutral-recoding, and worst-case analyses of biography non-response, audited and recomputed the reversal statistics at both item and run level, and clarified the interpretation of the framing treatments in the text.

1. Terminology and premise framing.

We agree that the premise condition does not test semantically equivalent statements. The pro- and anti-premise prompts intentionally introduce different politically valenced assumptions about the same broad issue. The intended test is whether opposing assumptions produce systematic directional differences in model responses, not whether a minimal wording change alters an otherwise identical proposition. We have clarified this distinction and narrowed the interpretation accordingly: premise effects may reflect engagement with or endorsement of the supplied assumption as well as differences in the political position expressed.

Importantly, however, the open-text outcome itself is held fixed across the paired conditions. The judges do not evaluate each response for agreement with the particular premise that generated it; instead, responses from both premise conditions are mapped onto the same five-point pro-anti political stance scale for that issue. Thus, although the prompt changes the assumption supplied to the model, the evaluative standard used to compare the resulting responses does not change. This does not make the premise treatment a clean estimate of stance change independent of premise engagement, but it means the observed pro-anti difference is not mechanically produced by changing the outcome definition.

The MCQ and open-text conditions intentionally operationalise premise framing differently. In MCQ, the politically valenced proposition is stated explicitly and the model is asked to agree or disagree. In open text, the premise is presupposed in a leading “How…” question and the model generates a response conditional on that assumption. We therefore do not interpret differences in effect magnitude as a clean causal effect of response format alone. The relevant result is that opposing premises produce the same directional difference under both elicitation structures.

The terminology condition does hold the substantive question fixed, while politically contested labels used to describe the same underlying actor, event, or issue are substituted. We agree that these labels are not necessarily strictly semantically equivalent: some carry different characterisations or presuppositions, which form part of the political signal being tested. We have clarified this distinction and no longer describe the substituted expressions themselves as semantically equivalent.

The premise and terminology treatments are designed to represent a plausible feature of user-LLM interaction, where users may convey political perspective through both their choice of terminology and the assumptions embedded in their questions.

AuthorsOpenReview ↗

Judge coverage and model identifiers

2. Judge coverage and model identifiers.

Thank you for identifying the inconsistency between the previous judge-coverage and agreement statistics. Table 17 reported an earlier partial-overlap judging output, whereas Table 18 reflected substantially more complete coverage. We reran the judging pipeline on the frozen analysis data and regenerated both tables from the same response set.

The three judges are Gemma 3 27B, Command A, and Llama 4 Scout. Each independently assigns a five-point stance score while blinded to the treatment condition and the identity of the model that generated the response; the final open-text stance score is the median of the available judge ratings. Within the seven-model primary analysis, 99.7% of open-text responses are now scored by all three judges at the run level, and the lowest item-level full-jury coverage is 99.6%.

This results in substantially greater overlap across the pairwise comparisons (see table below). The three judge pairs now contain 16,067, 16,049, and 16,068 responses out of 16,100, rather than the previous partial overlaps of approximately 9,000-12,000 responses. On this common response set, pairwise Pearson correlations are 0.75, 0.70, and 0.79, and same-side agreement is 73.0%, 70.2%, and 78.1%. These values supersede the previously reported correlations and side-agreement figures, and the corresponding tables and main-text reliability statement have been regenerated from the same data. We have also added results for three closed-source models; for these, judge agreement is lower (r=0.44-0.62), which we now report explicitly and use to qualify interpretation of their open-text estimates.

We also corrected the Qwen reference and now report exact identifiers for all models evaluated:

  • Open-weight models: command-a-03-2025, deepseek/deepseek-v4-flash, tiiuae/Falcon3-10B-Instruct, google/gemma-3-27b-it, meta-llama/llama-4-scout, mistralai/mistral-large, and qwen/qwen3.6-27b for the seven primary open-weight models; google/gemma-2-27b-it and meta-llama/llama-3.1-8b-instruct for the two supplementary previous-generation models. Qwen was evaluated with reasoning disabled.
  • Closed-source models: anthropic/claude-haiku-5.5, openai/gpt-6-luna, and google/gemini-3.1-flash-lite.

Pairwise agreement among the three open-text judges for the seven primary open-weight models.

Judge pairNPearson rExact agreementWithin 1 pointSame-side agreement
Gemma 3 27B / Command A16,067.7558.2%89.5%73.0%
Gemma 3 27B / Llama 4 Scout16,049.7055.1%85.4%70.2%
Command A / Llama 4 Scout16,068.7969.5%90.1%78.1%
AuthorsOpenReview ↗

Differential refusals in the biography condition

3. Differential refusals in the biography condition.

We agree that reporting refusals separately does not resolve possible selection bias. We therefore repeated the ideology analysis restricted to cells answered under all three ideology conditions and added worst-case sensitivity analyses that impute the remaining missing responses adversarially. In the case of a refusal on the first run, we also reran each prompt up to five additional times (although this led to modest decreases in refusal).

For Llama 4 after the retry procedure, the liberal-conservative gap went from 1.121 points when using the full dataset, to 1.081 points in the restricted, fully matched set; the directly matched liberal-conservative estimate is 1.084 (95% CI [0.91, 1.25]). Under worst-case assignment of the remaining missing responses, the lower bound remains positive at 0.77. For DeepSeek, Gemma 3, Mistral, and Qwen, 98-100% of cells are retained and matching changes the estimated ideology gap negligibly; Command A changes from 0.828 to 0.809.

The same analysis is reported for the closed-source models retained in the biography condition. Gemini is excluded from biography analyses because only 59% of its biography prompts produce a usable rating after the retry procedure. GPT-6 Luna is unchanged at 0.679. Claude Haiku 5.5 changes from 0.615 to 0.580 under triple matching (directly matched 0.579, 95% CI [0.49, 0.67]), but its worst-case range (-0.045 to 1.146) crosses zero, which we now state explicitly.

As an additional sensitivity check, we recoded every remaining non-answer to the neutral midpoint; this changes the liberal-conservative gap by at most 0.06 points among the models retained in the biography analysis. We have included the table below to show the sensitivity of the liberal-conservative biography effect to different treatments of missing data. The estimates are highly stable under neutral recoding and matched-response analyses. Under maximally adversarial assignment of the remaining missing responses, all models aside from Claude Haiku 5.5 maintain positive effects; Claude Haiku 5.5 is the only one with a worst-case range that crosses zero.

Sensitivity of the liberal-conservative biography effect to alternative treatments of non-response. All entries are estimated liberal-conservative stance gaps in points on the five-point stance scale. Complete-case uses all scorable responses after the retry procedure. Non-answers coded neutral assigns every remaining non-answer to the neutral midpoint before estimating the gap. Triple-matched retains only question-by-profile cells with scorable responses under liberal, neutral, and conservative ideology conditions. Directly matched requires scorable responses under the liberal and conservative conditions only; 95% confidence intervals use standard errors clustered by survey question. Worst-case gap range gives the minimum and maximum gap obtainable when the remaining missing responses are assigned adversarially.

ModelComplete-case stance gapGap with non-answers coded neutralTriple-matched gapDirectly matched gap [95% CI]Worst-case gap range
Command A0.830.830.810.81 [0.67, 0.95][0.80, 0.85]
DeepSeek V4 Flash1.151.141.151.14 [0.99, 1.30][1.13, 1.15]
Gemma 3 27B1.471.471.471.47 [1.31, 1.63][1.47, 1.47]
Llama 4 Scout1.121.101.081.08 [0.91, 1.25][0.77, 1.43]
Mistral Large0.980.980.980.98 [0.84, 1.11][0.98, 0.98]
Qwen 3.6 27B1.481.461.481.48 [1.29, 1.67][1.43, 1.50]
Claude Haiku 5.50.610.550.580.58 [0.49, 0.67][-0.04, 1.15]
GPT-6 Luna0.680.680.680.68 [0.55, 0.81][0.68, 0.68]
Reviewer_2NTTOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates “ideological mimicry”, where an LLM shifts its political stance toward the side signaled in the prompt. Specifically, the authors use national election survey questions from the US, UK, and Australia as the baseline, and test three kinds of political signals: contested terminology, premise framing, and user biographies. Then they collect multiple-choice responses from 7 open-weight LLMs (6 for user biographies), plus open-text responses for terminology and premise, which are scored by LLM judges and checked by humans. The authors find that (1) changing only the terminology flips the side that LLMs support in 16.9% of comparisons; (2) in the user biographies, stated ideology has the largest effect.

Critical strengths
  1. The contested terminology design is commendable, which only changes the politically associated words and keeps the question the same.

  2. The authors conduct extensive experiments with 7 LLMs, three countries, and both multiple-choice and open-text formats, which make the results solid.

Critical weaknesses
  1. The authors claim that they unify three strands of prior work. However, some very important references are missing. For example, Santurkar et al. [1] measure LLM opinions with public opinion polls. Also, there are some other LLM sycophancy papers. Wei et al. [2] give an example where the model takes liberal views once the user says they are liberal, which is almost the same as the biography finding here. Steering LLMs toward demographic groups or communities is also studied in [1, 3]. Not sure how the authors differentiate their contributions from these papers. The authors say the biography treatment tests “sycophantic alignment”, but sycophancy work is only briefly mentioned. This paper, especially the biography part, looks like a special case of LLM sycophancy. How do the authors establish their contributions?

  2. The findings are meaningful but not surprising, especially the results on user biographies. I would like to see some deeper analysis on why different LLMs behave differently. For example, Qwen has the smallest terminology effects in Table 5, but the largest gap between liberal and conservative users (1.48). Can the authors find any insights on how the different training recipes of these LLMs lead to such differences? In the introduction, the authors mention that the biography effect may be consistent with sycophancy from post-training such as RLHF. Then how do these LLMs differ in RLHF? I know that not all LLMs have everything public, but at least the authors can find some insights.

[1] Santurkar, Shibani, et al. "Whose opinions do language models reflect?." International Conference on Machine Learning. PMLR, 2023.

[2] Wei, Jerry, et al. "Simple synthetic data reduces sycophancy in large language models." arXiv preprint arXiv:2308.03958 (2023).

[3] Chen, Kai, et al. "STEER-BENCH: A Benchmark for Evaluating the Steerability of Large Language Models." arXiv preprint arXiv:2505.20645 (2025).

Questions for authors
  1. A lot of LLMs have safety guardrails and would refuse to answer some political questions. From my understanding, the refusals are excluded after a retry, and in Appendix O, Llama 4 Scout refuses 10% of the conservative personas but only 2% of the liberal ones. Would excluding them bias the results on user ideology? What if the refusals are treated as neutral answers?

  2. The authors only evaluate open-weight LLMs, for reproducibility and transparency. Are the guardrails and refusals also the reason that no commercial API-based LLMs are studied? The paper is motivated by “personalised political information environments”, so I’m curious to see if one or two commercial LLMs show the same pattern, even on a small subset of the data.

6 comments
AuthorsOpenReview ↗

Relation to prior work, model variation and addition of new models.

Thank you for these helpful comments. We have revised the paper substantially in response, including evaluating three additional closed-source models under the full design, adding matched-response, neutral-recoding, and worst-case sensitivity analyses for biography non-response, and revising the positioning relative to sycophancy and steerability.

1. Relation to prior work.

We agree that Santurkar et al. (2023), Wei et al. (2023), and STEER-BENCH (Chen et al., 2025) are important related work and should have been discussed more directly. We have added these papers, together with additional work on sycophancy, political persona conditioning, and steerability, and revised the Related Work and Introduction accordingly.

In particular, we now make clear that the biography result is closely related to political sycophancy: our claim is not that we are the first to show that models adapt to a user's stated political preferences. We instead position Poli-SHIFT as a common framework for testing directional political adaptation across three distinct interaction signals: contested terminology, politically valenced premises, and user context. This allows us to test whether the same directional pattern extends beyond explicit user-preference conditioning to lexical and contextual political signals and, for the framing treatments, across both MCQ and open-text elicitation.

2. Why models differ.

Explaining the differences across models is indeed an important direction. In an attempt to address this, we examined publicly available training documentation. However, models differ simultaneously in architecture, pre-training data, scale, instruction tuning, preference optimisation, and other post-training procedures, while several relevant details are not publicly disclosed. Moreover, the differences we observed do not follow a consistent ordering across treatments. For example, Qwen has one of the smallest terminology effects while exhibiting the largest liberal-conservative biography gap (1.48 points), essentially the same as Gemma 3 (1.47). We therefore do not think there is enough publicly available information or clear trends in the findings to support attributing these differences to variations in training approaches. We have added this limitation to the Discussion.

3. New evaluation on closed-source models.

We initially focused only on open-weight models to facilitate reproducibility and transparency of results. We agree, however, that closed-source assistants are particularly relevant to the paper's motivation around deployed and personalised information environments. We therefore evaluated Claude Haiku 5.5, GPT-6 Luna, and Gemini 3.1 Flash-Lite under the full experimental design. We have added the full results to the main paper, and present them as a visually distinct group in the figures.

The directional results we found for the open-weight models extend to these closed-source LLMs (see table below): all 12 framing comparisons are in the predicted direction and significant under paired tests, and 11 of 12 remain significant under topic-clustered inference. The exception is Gemini’s MCQ premise effect (+0.34, p=0.164). Compared to the open-weight models, these three additional models display a lower rate of stance reversal in the matched terminology items (4.2% versus 16.9% for the seven open-weight models). This average masks substantial heterogeneity: GPT-6 Luna reaches 20.9% reversal in MCQ terminology pairs, whereas Haiku and Gemini are at or below 0.9%.

User ideology shifts both Haiku and GPT-6 Luna in the same direction as the open-weight models; Gemini is excluded from biography analyses because of insufficient usable coverage (persistent non-parseable or discursive responses rather than the requested numerical rating). We also report that inter-judge agreement is lower on the closed-source models' open-text responses (r=0.44-0.62 versus 0.70-0.79 for the seven open-weight models).

Overview of closed-source model results: framing effects per treatment. Values are pro-side minus anti-side stance differences.

ModelMCQ termOT termMCQ premiseOT premise
Claude Haiku 5.5+0.35+0.32+0.71+1.18
GPT-6 Luna+1.01+0.36+0.92+0.70
Gemini 3.1 Flash-Lite+0.16+0.25+0.34+1.79
AuthorsOpenReview ↗

Bias from refusals

4. Bias from refusals.

As the reviewer points out, differential non-response could bias the ideology comparison (although we note that most of the models we reported had negligible refusal rates). We address this with a retry procedure and three complementary sensitivity analyses: (i) recoding persistent non-answers as neutral, as the reviewer suggests; (ii) restricting the analysis to matched question-by-profile cells answered across ideology conditions; and (iii) adversarial worst-case bounds for the remaining missing responses.

In the case of a refusal on the first run, we rerun each prompt up to five additional times. This led to a modest decrease in refusal rates; for instance, Llama 4 Scout went from 7.8% to 6.1%, and Claude Haiku 5.5 went from 17.9% to 11.2%. We retain the first-pass refusal analysis because this behaviour is itself informative, such as higher refusal for conservative-coded biographies, but we now supplement it with matched-response, neutral-recoding, and worst-case sensitivity analyses.

We then restrict the analysis to cells answered under all three ideology conditions to see whether higher refusals for some ideology attributes were biasing the results. For Llama 4, the liberal-conservative gap went from 1.121 points when using the full dataset, to 1.081 points in the restricted, fully matched set; the directly matched liberal-conservative estimate is 1.084 (95% CI [0.91, 1.25]). Estimates are similarly stable for the other open-weight models: Command A changes from 0.828 to 0.809, DeepSeek from 1.146 to 1.145, and Gemma 3, Mistral, and Qwen are essentially unchanged.

We apply the same checks to the closed-source models. GPT-6 Luna's gap is unchanged at 0.679. Claude Haiku 5.5 changes from 0.615 to 0.580 under matching across all ideology conditions. Gemini is excluded from biography analyses because only 59% of its biography prompts yield a usable rating after the retry procedure. These analyses and exclusions are now reported in the appendix.

Following the reviewer’s suggestion, we ran a further sensitivity check recoding every non-answer (refusals, hedges and unparseable responses) to the scale midpoint for all personas (neutral). This changes the liberal-conservative gap by at most 0.06 for any model reported in the biography analyses (e.g. 0.615 to 0.551 for Claude Haiku 5.5 and 1.121 to 1.098 for Llama 4 Scout), and the ideology effect remains clearly positive in every case.

We have included the table below to show the sensitivity of the liberal-conservative biography effect to different treatments of missing data. The estimates are highly stable under neutral recoding and matched-response analyses. Under maximally adversarial assignment of the remaining missing responses, all models aside from Claude Haiku 5.5 maintain positive effects; Claude Haiku 5.5 is the only one with a worst-case range that crosses zero.

Sensitivity of the liberal-conservative biography effect to alternative treatments of non-response. All entries are estimated liberal-conservative stance gaps in points on the five-point stance scale. Complete-case uses all scorable responses after the retry procedure. Non-answers coded neutral assigns every remaining non-answer to the neutral midpoint before estimating the gap. Triple-matched retains only question-by-profile cells with scorable responses under liberal, neutral, and conservative ideology conditions. Directly matched requires scorable responses under the liberal and conservative conditions only; 95% confidence intervals use standard errors clustered by survey question. Worst-case gap range gives the minimum and maximum gap obtainable when the remaining missing responses are assigned adversarially.

ModelComplete-case stance gapGap with non-answers coded neutralTriple-matched gapDirectly matched gap [95% CI]Worst-case gap range
Command A0.830.830.810.81 [0.67, 0.95][0.80, 0.85]
DeepSeek V4 Flash1.151.141.151.14 [0.99, 1.30][1.13, 1.15]
Gemma 3 27B1.471.471.471.47 [1.31, 1.63][1.47, 1.47]
Llama 4 Scout1.121.101.081.08 [0.91, 1.25][0.77, 1.43]
Mistral Large0.980.980.980.98 [0.84, 1.11][0.98, 0.98]
Qwen 3.6 27B1.481.461.481.48 [1.29, 1.67][1.43, 1.50]
Claude Haiku 5.50.610.550.580.58 [0.49, 0.67][-0.04, 1.15]
GPT-6 Luna0.680.680.680.68 [0.55, 0.81][0.68, 0.68]

Thank you again for highlighting these issues, as they have helped us to significantly strengthen the paper.

Thank the authors for the detailed response and the new experiments, which addressed most of my concerns. I still have some remaining concerns.

  1. I understand that the training details of these LLMs are not all public. However, there are still ways to test whether the biography effect comes from post-training. For example, some open models release checkpoints from each training stage, e.g., OLMo 2 releases its SFT, DPO, and final checkpoints. Since all of them follow instructions, the authors could run the same biography prompts on each checkpoint and see at which stage the effect appears. Base models could also be tested by turning the prompt into a continuation, e.g., a biography followed by a survey question and "Answer:", and then comparing the probabilities of the answer options, as Santurkar et al. do. Also, my question about the larger user ideology effect in Gemma 3 than in Gemma 2 is not answered.

  2. Gemini is excluded from the biography analyses because only 59% of its prompts give a usable rating, and Claude Haiku has a refusal rate of 17.9% at the first attempt. This seems to confirm that guardrails matter a lot for commercial LLMs. I think this is worth discussing in the main text, instead of only reporting it in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.