- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates how figurative language affects LLM judgments when the underlying context remains unchanged. The authors evaluate nine models across four types of figurative expressions (metaphor, hyperbole, sarcasm, and simile), primarily focusing on moral judgments. Their results suggest that figurative framing generally leads to harsher evaluations, with smaller open-weight models showing greater sensitivity and higher decision-flip rates. The authors further extend their analysis to other domains, including scientific peer review and resume evaluation, and compare LLM behavior with human judgments. Finally, they investigate potential explanations for this effect, suggesting that figurative expressions shift model attention away from the underlying context toward the wording of the question.
Critical strengths
1. Clear presentation and well-motivated research question.
The paper is well written, easy to follow, and addresses an interesting question concerning the sensitivity of LLM judgments to figurative language. Although the underlying research question is relatively simple, the authors motivate it effectively and provide a clear experimental methodology. I think it is an interesting contribution for the community.
2. Systematic evaluation across different models and settings.
The evaluation covers several model families and figurative-language categories, together with a comparison against human judgments. The additional analysis across different evaluation domains also provides an interesting starting point for investigating the generality of the observed phenomenon.
Critical weaknesses
1. Limited evidence of the practical significance of the observed effect.
My main concern is that the magnitude and practical implications of the reported effect do not appear as substantial as the paper's framing suggests.
In the primary moral-judgment experiments, many perturbations correspond to changes in the strength of a judgment rather than actual changes in its polarity. As shown in Figure 3, decision flips are relatively infrequent for the larger and proprietary models, with more pronounced effects concentrated in some smaller open-weight models. While the directional bias is interesting, I am not entirely convinced that its magnitude is sufficient to support the broader claims about the reliability of LLM-based evaluations.
Furthermore, Appendix F provides an important additional perspective: increasing reasoning effort within the same model generally reduces sensitivity to figurative framing. Although the experiments with standalone reasoning models reveal a more nuanced picture, these findings suggest that the observed effect depends substantially on the model configuration and reasoning capabilities.
The potentially most consequential results seem to be those concerning scientific peer review and resume evaluation. However, these experiments receive comparatively limited discussion in the main text. In particular, Table 4 reports substantial directional perturbations in these domains, but Appendix E (Table 8) shows relatively modest differences in average evaluation scores. Moreover, the scientific peer-review experiment relies only on paper titles and abstracts rather than full manuscripts, which limits how representative the setup is of actual peer review. More generally, it remains unclear whether the observed score changes would translate into meaningful differences in real-world decisions, such as accepting or rejecting a paper or selecting a candidate.
I believe a stronger analysis of these practical consequences is necessary to establish the significance of the proposed problem.
2. Model selection and insufficient investigation of recent open-weight and reasoning models.
I find the selection of evaluated models insufficiently justified, particularly considering the substantial differences between the open-weight and proprietary models.
The authors evaluate relatively recent closed-source models (GPT-5.6 Sol, Claude Opus 5, and Gemini 3.7 Flash), while the main open-weight evaluation relies on older families, including Llama 3.1, Qwen3, and Gemma 3. This creates an imbalance that makes it difficult to interpret the apparent gap between open-weight and proprietary models.
I would have expected the evaluation to include more recent open-weight families, such as Qwen3.5 and Gemma 4, and a more systematic comparison with reasoning-oriented models.
In this regard, the results in Appendix F are particularly relevant. The authors already consider additional models, including GPT-OSS-20B and DeepSeek-R1 distillations, and report that increasing reasoning effort within a model can reduce the effect, while standalone reasoning models can remain highly sensitive. These findings substantially qualify the conclusions drawn from the main experiments.
I strongly recommend moving this analysis into the main paper and extending it to a more representative set of contemporary open-weight models. If computational resources are a limitation, hosted inference services could be considered to broaden the experimental coverage.
Overall, I believe these limitations weaken the generality of the conclusions and leave important questions about the magnitude and practical relevance of the identified phenomenon unresolved.
Questions for authors
1. Practical implications of the observed perturbations.
Could the authors provide a more concrete assessment of the downstream impact of figurative framing in scientific peer review and resume evaluation? For instance, how often do the observed rating changes lead to different acceptance/rejection or candidate-selection decisions under reasonable decision thresholds? This would help establish whether the measured bias has meaningful practical consequences.
2. Model selection and reasoning capabilities.
What motivated the choice of open-weight models, particularly given the availability of newer model families? Could the authors extend the comparison to more recent open-weight models and systematically examine the effect of reasoning effort? Given the findings in Appendix F, this seems particularly important for assessing whether the phenomenon persists in stronger reasoning configurations.
Suggestion for future work.
It would be interesting to investigate whether specialized LLM judges, such as Prometheus 2 or M-Prometheus, exhibit greater robustness to figurative framing than general-purpose models. This could help clarify whether evaluation-specific training mitigates the observed sensitivity.
4 comments
Rebuttal Part 2
W2 and Q1. Practical Implication.
We respectfully disagree that the practical significance is limited. Our primary setting, moral judgment of everyday interpersonal scenarios, is itself a direct practical application. In these situations, users naturally express themselves figuratively, especially when emotionally invested, and the model’s verdict can shape how they view themselves, the other person, and their next decision. A systematic bias that makes the model harsher simply because the user phrased their question figuratively, while the described situation is identical, directly affects the advice people receive.
We also note that the significance of the effect lies in its direction rather than only its magnitude. Random fluctuations in judgment would average out across users, but a consistent shift toward harshness does not. Even when flips are infrequent for an individual model, LLMs serve a very large number of users, so a small systematic bias still affects many real decisions. Moreover, as we show in Section 4, humans exposed to the same figurative framing adjust the strength of their judgment but rarely change their verdict, and when they do, the change is not biased toward harshness. LLMs, including the strongest proprietary models, carry this harshness into their final decisions, which is precisely the property that matters when their verdicts are acted upon.
We also emphasize that large score changes should not be expected in this setting. The figurative rewrites preserve the meaning of the question and leave the scenario untouched, so an ideal evaluator’s judgment should not move at all. In practice, many decisions are made near a threshold, such as a borderline paper or a candidate on the edge of a shortlist, where even a small, systematic shift is enough to change the outcome.
Finally, the domains we study are representative rather than exhaustive. The same vulnerability applies wherever an LLM judges a situation described in a person’s own words, and people rarely describe their situations in neutral, literal language. In medical triage, a patient may write “my head is splitting” rather than “I have a severe headache.” In legal or workplace settings, an employee may say “my manager treats me like a punching bag” when describing a harassment complaint, and the model’s assessment of the claim’s credibility should not depend on that phrasing. In customer service, a complaint describing a product as “a complete disaster” should not be prioritized or dismissed differently from a literal description of the same defect. In each case, the evidence is the same, and a judgment that shifts with the phrasing alone undermines the reliability of the decision.
Qualitative Assessment Changes. We also conducted additional experiments in the two most consequential domains apart from moral scenarios: scientific peer review and resume review. We repeated the same evaluations with literally and figuratively framed questions, but this time we let the model generate a full written assessment instead of a score alone. In real-world settings, this written assessment is what matters most. A human reviewer, editor, or hiring manager reads the model’s evaluation, forms an impression from it, and then makes the decision. We already report the change in scores in Table 11. This experiment examines whether figurative framing also changes the substance of the assessment that a human would act on.
We provided each pair of assessments (literal and figurative) to a judge model (GPT-6.1-Sol) and asked it to select the more favorable one. The judge favored the assessment produced under the literal question in 46.6% of cases, the figurative one in 29%, and rated 23.3% as ties.
| Lit | Tie | Fig | |
|---|---|---|---|
| GPT-5.6-Sol | 52.0 | 16.0 | 32.0 |
| Gemini-3.7-Flash | 49.0 | 25.0 | 26.0 |
| Claude-Opus-5 | 42.0 | 29.0 | 29.0 |
In other words, when the question is framed figuratively, the model’s written evaluation of the same candidate or paper is more negative, even though the underlying evidence is identical. A human reading the assessment would likely reach a less favorable decision about the same candidate or paper. This shows that the effect is not confined to small score differences but carries over into the qualitative evaluations that drive real-world decisions. We have added this experiment to the paper (Table 9).
Please engage with the authors during the discussion period.
Dear reviewer,
The authors have provided a response to your review. Please ensure that you have read it and taken it into consideration when making your final recommendation. Even if you do not change your score, please post a response with your rationale.
Thank you to the authors for their response and additional experiments.
I appreciate the updated evaluation with more recent open-weight models. This was an important check to ensure that the findings still hold for current model families, and the new results address my concern in this regard. I also appreciate that the analysis of reasoning models has been moved to the main paper.
However, my main concern regarding practical significance remains. I agree with the authors that even small systematic biases can matter, particularly near decision thresholds. The additional qualitative experiment is interesting and provides further evidence of the directional bias, but it still does not establish how often these changes translate into different downstream decisions. The argument that more negative assessments would lead to different outcomes remains plausible, but largely untested.
To clarify, I did not question the existence of the observed phenomenon in my initial review. My concern was primarily whether its magnitude and consequences justify the broader claims made in the paper.
I remain unconvinced about the practical significance of the effect. I therefore maintain my initial rating of 2 (weak rejection).
Rebuttal Part 1
Thank you for your thoughtful feedback. We have added the reasoning model experiments to the main body (page 7).
W1 and Q2. Model Selection.
Thank you for pointing this out. Following your suggestion, we have updated the manuscript to replace the older open-weight models with their recent counterparts.
Qwen3-4b → Qwen3.5-4b,
Qwen3-30b → Qwen3.5-35b,
Gemma-3-12b → Gemma-4-12b,
Gemma-3-27b → Gemma-4-31b
Our primary findings remain unchanged. The newer models continue to show the same harshness bias under figurative framing. The effect is slightly weaker for the Gemma-4 models and stronger for the Qwen3.5 models, but in all cases the shift remains systematically skewed toward harsher judgments. This suggests that the bias is not an artifact of older model generations, and that it persists in the most recent open-weight families.