- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies how LLMs change their expressed political views in response to question wording, political assumptions, and user information. It introduces POLI-SHIFT, a dataset and evaluation framework covering ten political topics in the United States, United Kingdom, and Australia. The experiments evaluate seven open-weight models using multiple-choice and short written answers, with six models included in the biography analysis.
The evaluation includes five repeated generations for the wording and premise experiments, statistical tests with checks for dependent observations, and LLM scoring validated against 200 human-rated responses. The authors report that answers tend to move toward the political position suggested by the prompt or the user’s stated ideology. Other demographic details have smaller and less consistent effects.
Critical strengths
-
The paper provides a broad evaluation across seven models, ten political topics, and three countries. It examines question wording, political premises, and user biographies. The wording and premise experiments also include both multiple-choice and short written answers. This coverage helps show how political responses vary across models, topics, and ways of asking questions, making the findings useful for evaluating political bias.
-
The biography experiment uses 2,700 profiles that systematically vary ideology, race, gender, age, and education while keeping the survey questions fixed. This design helps distinguish the influence of stated political views from other personal details. The finding that ideology has a stronger and more consistent influence than most demographic cues provides useful insight into how personalization can shape political answers.
Critical weaknesses
-
The reported results contain unexplained differences in sample counts, values, and definitions. Table 18 implies about 15,900 complete three-judge ratings, but Table 17 uses only 12,454, 9,339, and 8,915 responses. Figure 18 shows Gemma 3’s open-text effects at approximately 0.47 and 1.46, while Table 5 reports 0.44 and 1.51. Figure 6 shows Qwen and Llama 4 reversal rates of approximately 0.9% and 40.0%, while the text gives 1.7% and 39.5%. Judge correlations range from 0.78–0.88 in the text but 0.77–0.85 in Table 17. Figure 18 also measures contradictions using repeated-run pairs, while Table 16 uses prompts. These differences need reconciliation because they make the evaluation and reported results difficult to verify.
-
The framing experiments sometimes change the question’s meaning or task alongside the political cue. Table 2 replaces “Russia’s unprovoked invasion” with “Russia’s special military operation,” changing the assumptions conveyed. In Table 3, multiple-choice questions ask models to assess a statement, while written-answer questions ask “How…” and assume the statement is true. These changes weaken the claim that terminology alone or response format explains the measured differences. Controls that preserve the meaning and task are needed to support that interpretation.
Questions for authors
-
How do Table 17’s judge-pair counts of 12,454, 9,339, and 8,915 fit with Table 18’s nearly complete three-judge coverage? Does a different response set also explain the judge-correlation ranges of 0.78–0.88 in the text and 0.77–0.85 in Table 17?
-
Does Figure 18 use a different sample or averaging method from Tables 5 and 16? Please explain Gemma 3’s differing open-text effects and clarify whether contradiction rates count prompts or repeated-run pairs.
-
Why do the text and Figure 6 give different reversal rates for Qwen’s open-text answers and Llama 4’s multiple-choice answers? Were these calculated from individual runs or responses averaged across five runs?
-
What checks show that the terminology variants preserve the question’s meaning? How do you separate response-format effects from the difference between assessing a statement and answering a “How…” question that assumes it is true?
-
Does the biography-related ideology effect remain when comparing the same questions answered across ideology conditions, given Llama 4’s refusal rates of approximately 10% for conservative profiles and 2% for liberal profiles?
11 comments
2. Meaning and task differences across framing conditions.
We agree that the premise condition does not test semantically equivalent statements. The pro- and anti-premise prompts intentionally introduce different politically valenced assumptions about the same broad issue. The intended test is whether opposing assumptions produce systematic directional differences in model responses, not whether a minimal wording change alters an otherwise identical proposition. We have clarified this distinction and narrowed the interpretation accordingly: premise effects may reflect engagement with or endorsement of the supplied assumption as well as differences in the political position expressed.
Importantly, however, the open-text outcome itself is held fixed across the paired conditions. The judges do not evaluate each response for agreement with the particular premise that generated it; instead, responses from both premise conditions are mapped onto the same five-point pro-anti political stance scale for that issue. Thus, although the prompt changes the assumption supplied to the model, the evaluative standard used to compare the resulting responses does not change. This does not make the premise treatment a clean estimate of stance change independent of premise engagement, but it means the observed pro-anti difference is not mechanically produced by changing the outcome definition.
The MCQ and open-text conditions intentionally operationalise premise framing differently. In MCQ, the politically valenced proposition is stated explicitly and the model is asked to agree or disagree. In open text, the premise is presupposed in a leading “How…” question and the model generates a response conditional on that assumption. We therefore do not interpret differences in effect magnitude as a clean causal effect of response format alone. The relevant result is that opposing premises produce the same directional difference under both elicitation structures.
The terminology condition does hold the substantive question fixed, while politically contested labels used to describe the same underlying actor, event, or issue are substituted. We agree that these labels are not necessarily strictly semantically equivalent: some carry different characterisations or presuppositions, which form part of the political signal being tested. We have clarified this distinction and no longer describe the substituted expressions themselves as semantically equivalent.
The premise and terminology treatments are designed to represent a plausible feature of user-LLM interaction, where users may convey political perspective through both their choice of terminology and the assumptions embedded in their questions.
3. Differential refusals and the biography effect.
We repeated the ideology analysis restricted to cells answered under all three ideology conditions and added worst-case sensitivity analyses, where missing values were filled with the extreme case. In the case of a refusal on the first run, we also reran each prompt up to five additional times (although this led to modest decreases in refusal).
For Llama 4 after the retry procedure, the liberal-conservative gap went from 1.121 points, when using the full dataset, to 1.081 points in the restricted, fully matched set: the directly matched liberal-conservative estimate is 1.084 (95% CI [0.91, 1.25]). Under worst-case assignment of the remaining missing responses, the lower bound remains positive at 0.77. For DeepSeek, Gemma 3, Mistral, and Qwen, 98-100% of cells are retained and matching changes the estimated ideology gap negligibly; Command A changes from 0.828 to 0.809.
4. Ethics and release safeguards.
All biographies are synthetic and Poli-SHIFT is designed for diagnostic auditing rather than political persuasion. We have expanded the Ethics Statement to state that the release documentation will explicitly identify targeted political persuasion and user-specific political-message optimisation as out-of-scope uses, and that the released materials are intended for auditing and reproducibility.
Thank you for the detailed response. The corrected reporting and additional refusal analyses address several concerns. In particular, the matched-response and worst-case analyses strengthen the evidence for the biography-related ideology effect. Two concerns remain.
-
The framing experiments still combine political cues with changes in meaning, assumptions, or task. I appreciate the narrower interpretation, but using a common stance scale does not separate political accommodation from ordinary responses to different premises. Similarly, the different tasks used for multiple-choice and written answers limit what their comparison establishes. These limitations weaken the evidence supporting the paper’s interpretation of ideological mimicry.
-
The revised judging results leave uncertainty about measurement reliability. The updated correlations of 0.70–0.79 and same-side agreement of 70.2–78.1% make sensitivity to judge choice important. The response does not establish whether the main effect sizes and reversal findings remain consistent across individual judges, or how the corrected ratings relate to the existing human validation.
The dataset and findings remain useful, and I acknowledge the improvements described in the response. The remaining concerns affect the interpretation and reliability of the central results.
Thank you for identifying these inconsistencies and for the comments on treatment validity, refusal-related selection, and release safeguards. In response, we reran the judging pipeline on the frozen data, regenerated the affected figures and tables from a common analysis set, explicitly separated item- and run-level reversal statistics, and added matched-response and worst-case analyses of biography non-response.
1. Reporting inconsistencies.
We agree that the discrepancies in counts and definitions made the results difficult to verify. Table 17 reflected an earlier partial-overlap judging output, whereas Table 18 reflected substantially more complete coverage. We reran the judging pipeline on the frozen analysis data and regenerated both tables from the same response set. The revised figure and table now use the same frozen response set and aggregation. Contradiction rate is defined at the prompt level: a repeated prompt is counted as contradictory if its generations fall on opposite sides of the neutral midpoint. It is therefore not a count of all pairwise combinations of repeated runs.
For the seven open-weight models, the judging pipeline now has essentially complete coverage: 99.7% of open-text responses are scored by all three judges at the run level, with at least 99.6% full-jury coverage at the item level. The three pairwise comparisons contain 16,067, 16,049, and 16,068 responses (out of a total of 16,100). On this common response set, Pearson correlations are 0.75, 0.70, and 0.79 and same-side agreement is 73.0%, 70.2%, and 78.1%. These replace the previously reported statistics based on smaller pairwise overlaps, and the corresponding tables and main-text values have been regenerated from the same response set.
Second, the other cited discrepancies have been regenerated from the same frozen data. Gemma 3's open-text terminology and premise effects are 0.414 and 1.555, respectively (reported as 0.41 and 1.56), and the corresponding figure and table now use the same values. Its open-text terminology and premise contradiction rates are 1.7% and 7.0%, respectively.
Third, our check identified inconsistencies in how item-level and run-level reversal quantities were distinguished, as well as several outdated model-specific values. The headline reversal rate for the seven open-weight models is 16.89% at the item level and 12.74% at the run level. Qwen's open-text reversal rate is 4.35% at the item level versus 2.26% at the run level, while Llama 4's MCQ rate is 40.0% versus 38.6%. We now state the aggregation level explicitly throughout and regenerated the affected figures and tables from the same analysis.
We have also corrected the Qwen reference and added exact model identifiers and inference settings in the appendix.
Corrected values in the revised draft.