- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This work tried to investigate the "political tendencies" of llms through their choice on one of the four collective decision rules. The proposed debiasing method estimates a background response tendency by removing preference evidence, and then adjusts the evidence-conditioned responses by down-weighting choices favored by this baseline. The experiments show that this adjustment can slightly improve average predictive accuracy, but may increase sensitivity to irrelevant country labels and does not reliably mitigate political skew.
Critical strengths
The analysis shows that reference-based debiasing can improve average accuracy while simultaneously increasing identity sensitivity.
Critical weaknesses
- The paper operationalizes “political tendencies” through choices among only four collective decision rules, which captures a very narrow dimension of political behavior and seems insufficient to support the paper’s broader framing about political tendencies in LLMs.
- The real-profile experiments lack ground-truth preferences, so the large differences between U.S. and Chinese profiles only show that model outputs are sensitive to background information, but cannot establish whether these responses are biased, inaccurate, or simply reasonable predictions about the represented individuals.
- The debiasing method assumes that the response distribution obtained after removing preference evidence represents an undesirable background tendency that should be discounted, but this baseline may also contain legitimate predictive information, making the conceptual and probabilistic justification of the ratio-based correction unclear.
Questions for authors
- How do the authors justify using these four collective decision rules as a sufficiently broad measure of “political tendencies,” and would the conclusions hold across other political dimensions?
- Without ground-truth preferences for the real political figures, how should the observed U.S.–China response differences be interpreted as evidence about political bias rather than profile-conditioned prediction?
- What exactly is the intended interpretation of the evidence-removed baseline, why should it be treated as something to remove rather than as potentially valid prior information, and under what assumptions is the proposed ratio-based adjustment consistent with Bayesian principles?