- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors examine if candidate likelihood can reliably predict a model's political stance in free generation, while also showing how language switching and lexical artifacts can game evaluation metrics. The core problem is important and the failure cases are quite revealing. That said, the paper's unique contribution over prior work is a bit blurry, and the empirical evidence feels too thin to support some of its stronger conclusions.
Critical strengths
- Important, practical focus on auditing multilingual political bias.
- The matched comparisons and constant baselines effectively tease apart actual prediction from surface-level agreement.
- Compelling qualitative examples that clearly show how metric improvements can hide weird model behaviors.
Critical weaknesses
- Novelty: We already know about the probability-generation mismatch and the limits of forced-choice political evals from recent work like [1,2]. It’s not entirely clear what new insights the proposed paired comparisons and within-model analysis offer, other than showing these same problems exist in this specific setup.
- The data doesn't fully back up the punchy claim that scores "rank models, not answers," especially given the small within-model sample sizes and wide AUC intervals (even if the authors note this as a limitation). I also would have liked to see more ablation on alternative candidate wordings and structures to prove robustness.
- The abstract includes too many unexplained numbers (e.g., 0.89 baseline, 0.91 AUC) before the reader even knows what the baseline or prediction targets are. Dropping these numbers from the abstract and adding a clear figure to explain the evaluation setup early on would help readability a lot.
[1] Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Models (Lyu et al., KnowLLM 2024)
[2] Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models (Röttger et al., ACL 2024)
Questions for authors
Please see the weakness above.