acceptodds
Under review as a conference paper at ICLR 2027

Political Tendencies in Language Models: Measurement and Mitigation

Abstract

We investigate political tendencies in language models through their choices among four collective decision rules and evaluate whether adjusting responses toward a measured background tendency reduces political skew, checking accuracy against known preference targets, sensitivity to explicitly irrelevant country labels, and preservation of stated preferences separately. Comparing GPT-5.4 nano and DeepSeek V4 Flash, we find that both favor simple majority when no personal information is supplied, but their responses change with political profiles and reasoning settings. Under the common full-profile protocol, high-effort responses select centralized leadership for 77.2–79.6% of Chinese profiles and simple majority for 79.1–81.6% of U.S. profiles. Masking designated identity and institutional spans sharply reduces the Chinese centralized-leadership share, and removing one further role sentence partly restores it. On held-out synthetic tasks, the most flexible adjustment changes average Brier error by -0.0047 (GPT) and -0.0066 (DeepSeek) relative to raw responses, but neither model passes the prespecified joint criterion and country-label differences increase. Explicit preferences are preserved because development selects zero adjustment for that evidence. A derived exact-recovery condition explains why favorable average-error estimates can coexist with greater identity sensitivity. Neither masking, reasoning changes, nor response-probability adjustment is shown to mitigate political skew reliably.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, political tendencies, bias mitigation, political preference, reasoning effort

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SsYpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This work tried to investigate the "political tendencies" of llms through their choice on one of the four collective decision rules. The proposed debiasing method estimates a background response tendency by removing preference evidence, and then adjusts the evidence-conditioned responses by down-weighting choices favored by this baseline. The experiments show that this adjustment can slightly improve average predictive accuracy, but may increase sensitivity to irrelevant country labels and does not reliably mitigate political skew.

Critical strengths

The analysis shows that reference-based debiasing can improve average accuracy while simultaneously increasing identity sensitivity.

Critical weaknesses
  1. The paper operationalizes “political tendencies” through choices among only four collective decision rules, which captures a very narrow dimension of political behavior and seems insufficient to support the paper’s broader framing about political tendencies in LLMs.
  2. The real-profile experiments lack ground-truth preferences, so the large differences between U.S. and Chinese profiles only show that model outputs are sensitive to background information, but cannot establish whether these responses are biased, inaccurate, or simply reasonable predictions about the represented individuals.
  3. The debiasing method assumes that the response distribution obtained after removing preference evidence represents an undesirable background tendency that should be discounted, but this baseline may also contain legitimate predictive information, making the conceptual and probabilistic justification of the ratio-based correction unclear.
Questions for authors
  1. How do the authors justify using these four collective decision rules as a sufficiently broad measure of “political tendencies,” and would the conclusions hold across other political dimensions?
  2. Without ground-truth preferences for the real political figures, how should the observed U.S.–China response differences be interpreted as evidence about political bias rather than profile-conditioned prediction?
  3. What exactly is the intended interpretation of the evidence-removed baseline, why should it be treated as something to remove rather than as potentially valid prior information, and under what assumptions is the proposed ratio-based adjustment consistent with Bayesian principles?
Reviewer_GRu5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the political biases of GPT and DeepSeek by looking at how they choose among four collective decision rules. The authors test how these choices shift under different political profiles, masking techniques, and reasoning settings. They also evaluate a reference-based probability adjustment method using synthetic tasks with known preference targets. The results show that it can simultaneously make the models more sensitive to country labels. Overall, the paper address an interesting problem and offers a sobering look at the limits of reference-based adjustment.

Critical strengths
  1. The paper asks whether an LLM’s output actually reflects the user persona it is meant to represent, or if it is just falling back on priors induced by the model and prompt structure.
  2. The synthetic tasks are well-crafted, featuring clear preference targets, matched country-label pairs, and context-dependent explicit preferences.
  3. The paper examines a common assumption: that measuring a baseline without preference evidence accurately captures the bias that remains after evidence is introduced. Both the theory and experiments nicely show how post-hoc adjustments can accidentally introduce identity biases even as overall accuracy improves.
Critical weaknesses
  1. Narrow view of political tendencies: The entire evaluation relies on preferences over four collective decision rules. It leaves out broader, more typical political dimensions like economic policy, social issues, or civil liberties.
  2. Novelty: The ratio-based adjustment looks very similar to existing reference-calibration methods (e.g., Contextual Calibration), and it is already well-known in the literature that a poorly chosen reference can inject new biases.
  3. Coverage: Testing only two target models (and nano and flash variants at that) makes it hard to tell if these patterns generalize. Adding another model family or testing a larger, more capable frontier model would make the empirical claims much more convincing.
  4. It is not entirely clear how the reference-based adjustment experiments connects back to the political tendencies measured in the first half of the paper. The authors need to better distinguish between mitigating base-model behavior, fixing a deployed system's output distribution, and simply improving preference-estimation accuracy.
Questions for authors

Please see the weakness above.

Reviewer_nLv7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper measures political tendencies in LMs through choices across different decision rules (unanimity, simple majority, two-thirds super majority and centralised leadership). They test two models where responses change based on political profiles and reasoning settings. They also investigate possible methods to address political skew. However, using the three different techniques, they find that improvements in average accuracy can coexist with increased country-label sensitivity. They finally share standards for mitigation methods.

Critical strengths
  1. Interesting and very relevant topic. The paper explores political tendencies in LMs and shares methods to understand what drives these changes. This is an important research direction and very relevant given the widespread LLM usage across different contexts.
  2. I like the mitigation recommendations provided in the paper; also the paper shared unsuccessful results transparently. These findings provide useful caution against assuming that some standard methods might not work well.
  3. The distinction between accuracy and sensitivity to irrelevant identity cues is also important. That lower Brier error cannot mean insensitivity and smaller gaps can’t always mean preference inference.
Critical weaknesses
  1. Profile construction and initiation are difficult to understand. Although the paper shares the biography sources and conditions, the appendix contains only a schematic diagram. The main paper would benefit from complete worked examples of profiles and conditions, that shows the bio, English rendering, spans removed due to masking and so on.

  2. Also, using these demographics, how do you make sure stereotypes (that may not be related to the task) are not carried over to these profiles and in turn, the reasoning? I understand this task checks for political skews however feeding them through the personas may amplify or reduce an existing behavior. Is there any control for this ?

  3. The synthetic tasks provide controlled ground truth, but their connection to real political preferences remains insufficiently validated. The authors acknowledge this but it remains a major limitation. Possible evaluation with verified human preferences over same decision rules would strengthen these contributions.

  4. The scope of the findings are very limited, there are only two tested models and two countries. Although the authors mention it, it is important that validation across a few more models is important to justify the claims that they provide.

Questions for authors

See above weaknesses.

Additional Q: Please clarify how the interrupted requests were distributed across H and if conclusions remain unchanged when recovered responses are excluded. Can you also share if response availability or the recovery procedure systematically affect the reported comparisons?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.