acceptodds
Under review as a conference paper at ICLR 2027

LLMs Judge You Harshly When You Speak Figuratively

Abstract

LLMs are increasingly used to judge people and their work, from moral advice to scientific paper review. We show that these judgments become harsher when the question is phrased figuratively. Given a moral scenario, we rewrite only the question as metaphor, hyperbole, simile and sarcasm, verified by human annotators to preserve the same meaning. We then evaluate nine models from six families, including both open-source and proprietary state-of-the-art models. All nine models offer a harsher judgment score, with GPT-5.6 Sol judging the user more harshly 17.5% of the time, comparable to human performance. However, humans exhibit a significantly lower flip rate (i.e., changing their judgment polarity from positive to negative or vice versa), while LLMs, especially open-source models, flip their decision by up to 11.1% of the time. This effect extends to mathematical reasoning, truthfulness QA, scientific paper review and resume review. We further conduct analysis to show that figurative questions draw the model's attention away from the scenario, providing cues for the judgment shifts.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Moral Judgment, Figurative Language

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_L33COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how figurative language affects LLM judgments when the underlying context remains unchanged. The authors evaluate nine models across four types of figurative expressions (metaphor, hyperbole, sarcasm, and simile), primarily focusing on moral judgments. Their results suggest that figurative framing generally leads to harsher evaluations, with smaller open-weight models showing greater sensitivity and higher decision-flip rates. The authors further extend their analysis to other domains, including scientific peer review and resume evaluation, and compare LLM behavior with human judgments. Finally, they investigate potential explanations for this effect, suggesting that figurative expressions shift model attention away from the underlying context toward the wording of the question.

Critical strengths

1. Clear presentation and well-motivated research question.

The paper is well written, easy to follow, and addresses an interesting question concerning the sensitivity of LLM judgments to figurative language. Although the underlying research question is relatively simple, the authors motivate it effectively and provide a clear experimental methodology. I think it is an interesting contribution for the community.

2. Systematic evaluation across different models and settings.

The evaluation covers several model families and figurative-language categories, together with a comparison against human judgments. The additional analysis across different evaluation domains also provides an interesting starting point for investigating the generality of the observed phenomenon.

Critical weaknesses

1. Limited evidence of the practical significance of the observed effect.

My main concern is that the magnitude and practical implications of the reported effect do not appear as substantial as the paper's framing suggests.

In the primary moral-judgment experiments, many perturbations correspond to changes in the strength of a judgment rather than actual changes in its polarity. As shown in Figure 3, decision flips are relatively infrequent for the larger and proprietary models, with more pronounced effects concentrated in some smaller open-weight models. While the directional bias is interesting, I am not entirely convinced that its magnitude is sufficient to support the broader claims about the reliability of LLM-based evaluations.

Furthermore, Appendix F provides an important additional perspective: increasing reasoning effort within the same model generally reduces sensitivity to figurative framing. Although the experiments with standalone reasoning models reveal a more nuanced picture, these findings suggest that the observed effect depends substantially on the model configuration and reasoning capabilities.

The potentially most consequential results seem to be those concerning scientific peer review and resume evaluation. However, these experiments receive comparatively limited discussion in the main text. In particular, Table 4 reports substantial directional perturbations in these domains, but Appendix E (Table 8) shows relatively modest differences in average evaluation scores. Moreover, the scientific peer-review experiment relies only on paper titles and abstracts rather than full manuscripts, which limits how representative the setup is of actual peer review. More generally, it remains unclear whether the observed score changes would translate into meaningful differences in real-world decisions, such as accepting or rejecting a paper or selecting a candidate.

I believe a stronger analysis of these practical consequences is necessary to establish the significance of the proposed problem.

2. Model selection and insufficient investigation of recent open-weight and reasoning models.

I find the selection of evaluated models insufficiently justified, particularly considering the substantial differences between the open-weight and proprietary models.

The authors evaluate relatively recent closed-source models (GPT-5.6 Sol, Claude Opus 5, and Gemini 3.7 Flash), while the main open-weight evaluation relies on older families, including Llama 3.1, Qwen3, and Gemma 3. This creates an imbalance that makes it difficult to interpret the apparent gap between open-weight and proprietary models.

I would have expected the evaluation to include more recent open-weight families, such as Qwen3.5 and Gemma 4, and a more systematic comparison with reasoning-oriented models.

In this regard, the results in Appendix F are particularly relevant. The authors already consider additional models, including GPT-OSS-20B and DeepSeek-R1 distillations, and report that increasing reasoning effort within a model can reduce the effect, while standalone reasoning models can remain highly sensitive. These findings substantially qualify the conclusions drawn from the main experiments.

I strongly recommend moving this analysis into the main paper and extending it to a more representative set of contemporary open-weight models. If computational resources are a limitation, hosted inference services could be considered to broaden the experimental coverage.

Overall, I believe these limitations weaken the generality of the conclusions and leave important questions about the magnitude and practical relevance of the identified phenomenon unresolved.

Questions for authors

1. Practical implications of the observed perturbations.

Could the authors provide a more concrete assessment of the downstream impact of figurative framing in scientific peer review and resume evaluation? For instance, how often do the observed rating changes lead to different acceptance/rejection or candidate-selection decisions under reasonable decision thresholds? This would help establish whether the measured bias has meaningful practical consequences.

2. Model selection and reasoning capabilities.

What motivated the choice of open-weight models, particularly given the availability of newer model families? Could the authors extend the comparison to more recent open-weight models and systematically examine the effect of reasoning effort? Given the findings in Appendix F, this seems particularly important for assessing whether the phenomenon persists in stronger reasoning configurations.

Suggestion for future work.

It would be interesting to investigate whether specialized LLM judges, such as Prometheus 2 or M-Prometheus, exhibit greater robustness to figurative framing than general-purpose models. This could help clarify whether evaluation-specific training mitigates the observed sensitivity.

4 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Thank you for your thoughtful feedback. We have added the reasoning model experiments to the main body (page 7).

W1 and Q2. Model Selection.

Thank you for pointing this out. Following your suggestion, we have updated the manuscript to replace the older open-weight models with their recent counterparts.

Qwen3-4b → Qwen3.5-4b,
Qwen3-30b → Qwen3.5-35b,
Gemma-3-12b → Gemma-4-12b,
Gemma-3-27b → Gemma-4-31b

Our primary findings remain unchanged. The newer models continue to show the same harshness bias under figurative framing. The effect is slightly weaker for the Gemma-4 models and stronger for the Qwen3.5 models, but in all cases the shift remains systematically skewed toward harsher judgments. This suggests that the bias is not an artifact of older model generations, and that it persists in the most recent open-weight families.

AuthorsOpenReview ↗

Rebuttal Part 2

W2 and Q1. Practical Implication.

We respectfully disagree that the practical significance is limited. Our primary setting, moral judgment of everyday interpersonal scenarios, is itself a direct practical application. In these situations, users naturally express themselves figuratively, especially when emotionally invested, and the model’s verdict can shape how they view themselves, the other person, and their next decision. A systematic bias that makes the model harsher simply because the user phrased their question figuratively, while the described situation is identical, directly affects the advice people receive.

We also note that the significance of the effect lies in its direction rather than only its magnitude. Random fluctuations in judgment would average out across users, but a consistent shift toward harshness does not. Even when flips are infrequent for an individual model, LLMs serve a very large number of users, so a small systematic bias still affects many real decisions. Moreover, as we show in Section 4, humans exposed to the same figurative framing adjust the strength of their judgment but rarely change their verdict, and when they do, the change is not biased toward harshness. LLMs, including the strongest proprietary models, carry this harshness into their final decisions, which is precisely the property that matters when their verdicts are acted upon.

We also emphasize that large score changes should not be expected in this setting. The figurative rewrites preserve the meaning of the question and leave the scenario untouched, so an ideal evaluator’s judgment should not move at all. In practice, many decisions are made near a threshold, such as a borderline paper or a candidate on the edge of a shortlist, where even a small, systematic shift is enough to change the outcome.

Finally, the domains we study are representative rather than exhaustive. The same vulnerability applies wherever an LLM judges a situation described in a person’s own words, and people rarely describe their situations in neutral, literal language. In medical triage, a patient may write “my head is splitting” rather than “I have a severe headache.” In legal or workplace settings, an employee may say “my manager treats me like a punching bag” when describing a harassment complaint, and the model’s assessment of the claim’s credibility should not depend on that phrasing. In customer service, a complaint describing a product as “a complete disaster” should not be prioritized or dismissed differently from a literal description of the same defect. In each case, the evidence is the same, and a judgment that shifts with the phrasing alone undermines the reliability of the decision.

Qualitative Assessment Changes. We also conducted additional experiments in the two most consequential domains apart from moral scenarios: scientific peer review and resume review. We repeated the same evaluations with literally and figuratively framed questions, but this time we let the model generate a full written assessment instead of a score alone. In real-world settings, this written assessment is what matters most. A human reviewer, editor, or hiring manager reads the model’s evaluation, forms an impression from it, and then makes the decision. We already report the change in scores in Table 11. This experiment examines whether figurative framing also changes the substance of the assessment that a human would act on.

We provided each pair of assessments (literal and figurative) to a judge model (GPT-6.1-Sol) and asked it to select the more favorable one. The judge favored the assessment produced under the literal question in 46.6% of cases, the figurative one in 29%, and rated 23.3% as ties.

LitTieFig
GPT-5.6-Sol52.016.032.0
Gemini-3.7-Flash49.025.026.0
Claude-Opus-542.029.029.0

In other words, when the question is framed figuratively, the model’s written evaluation of the same candidate or paper is more negative, even though the underlying evidence is identical. A human reading the assessment would likely reach a less favorable decision about the same candidate or paper. This shows that the effect is not confined to small score differences but carries over into the qualitative evaluations that drive real-world decisions. We have added this experiment to the paper (Table 9).

Area_Chair_gNEAOpenReview ↗

Please engage with the authors during the discussion period.

Dear reviewer,

The authors have provided a response to your review. Please ensure that you have read it and taken it into consideration when making your final recommendation. Even if you do not change your score, please post a response with your rationale.

Thank you to the authors for their response and additional experiments.

I appreciate the updated evaluation with more recent open-weight models. This was an important check to ensure that the findings still hold for current model families, and the new results address my concern in this regard. I also appreciate that the analysis of reasoning models has been moved to the main paper.

However, my main concern regarding practical significance remains. I agree with the authors that even small systematic biases can matter, particularly near decision thresholds. The additional qualitative experiment is interesting and provides further evidence of the directional bias, but it still does not establish how often these changes translate into different downstream decisions. The argument that more negative assessments would lead to different outcomes remains plausible, but largely untested.

To clarify, I did not question the existence of the observed phenomenon in my initial review. My concern was primarily whether its magnitude and consequences justify the broader claims made in the paper.

I remain unconvinced about the practical significance of the effect. I therefore maintain my initial rating of 2 (weak rejection).

Reviewer_1FVCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate how adding figurative language to a user’s question changes the model’s evaluation. They examine rephrasings using metaphors, hyperbole, comparisons, and sarcasm across 9 models. Their conclusion is that such rephrasings result in a harsh evaluation. The study also includes human evaluations, assessments of reformulated questions with explicit emotional overtones, and evaluations of similar changes in other tasks. However, the effects vary by category: for example, reformulating a question using sarcasm more often leads to leniency for open models, whereas closed models maintain harsh evaluations.

Critical strengths

There are useful controls: literal rephrasing, human feedback, and rephrasing with emotional tone. Additionally, to rule out the possibility of a correlation between the polarity of the question and the model’s rating, the authors swap “in the right/in the wrong,” but keep the main part of the question in figurative language— it's useful, but it’s better to state that the effect of polarization is weakened rather than completely eliminated. There is a useful qualitative study on why changes in ratings occur. The authors test several model families and additional tasks, thereby making their findings more substantive.

Critical weaknesses

The figurative language is not sufficiently separated from the connotation it imparts to the question. The authors test for the preservation of meaning across paraphrases, but this does not measure the connotation imparted to the question. Appendix H adds additional measurements regarding the evaluation of questions with explicit connotations, but overall, this does not address the insufficient separation of figurative language from connotation.

There is also a limitation regarding human feedback. The study involved five participants, who were initially interviewed about stories using standard questions; then, two months later, they were interviewed about the same stories but with rephrased questions containing figurative language. There are two problems: the small number of participants limits the generalizability of the results, and there is a risk of participants recalling their previous answers—perhaps a control condition involving the repeated presentation of unchanged questions would have helped assess this risk.

Questions for authors
  1. Was the change in the tone of the questions assessed, beyond just the absence of a significant change in emotion? For example, the wording might have become more justifying or accusatory.
  2. Are there results from a re-evaluation of the unchanged questions among the participants? Is there an analysis of consistency across participants? This would help assess how reliable the comparison of changes in ratings is between models and humans.
3 comments

Thank you for your thoughtful review.

On separating figurative language from connotation.

Thank you for raising this point. We addressed it directly in the Discussion (Section 5, “Figurative Language and Pragmatic Enrichment”) and in Appendix G. Figurative language naturally adds emphasis, emotional intensity, speaker attitude, or evaluative force (Lakoff & Johnson, 1980). These pragmatic effects are constitutive properties of figurative communication, not nuisance variables to be controlled away (Popa-Wyatt, 2020; Ortony, 1975). A hyperbole without exaggeration, or a metaphor without the connotations of its source domain, is no longer figurative. Requiring literal and figurative variants to be matched in connotation would therefore remove the very phenomenon we study.

Moreover, as Appendix G shows, reproducing the same pragmatic force with literal language requires introducing new factual claims (e.g., “in a condescending and insulting way”), whereas a figurative expression conveys that force while preserving the underlying event. Our research question is therefore not whether figurative language carries connotation, which it inherently does, but whether models can separate how a user phrases the question from the evidence in the scenario that should ground the judgment. Humans adjust the strength of their judgment but rarely change their verdict while LLMs are more impacted.

That being said, we examined whether the judgment shifts are driven by the degree of pragmatic intensity a rewrite carries. We provided each literal and figurative question to GPT-5-mini and asked it to rate how pragmatically intensified each one is on a scale of 1 to 10. We then measured the sentence-level correlation between this intensity and the change in the model’s judgment (figurative minus literal Likert score), both for all perturbations and for flipped decisions.

Perturbation (all judgment shifts):

MetaphorHyperboleSarcasmSimile
Spearman ρ-0.020.11-0.040.05
Pearson r0.020.09-0.070.04

Signed shift (flipped decisions):

MetaphorHyperboleSarcasmSimile
Spearman ρ-0.01-0.03-0.08-0.13
Pearson r0.02-0.020.10-0.14

Across all figurative categories, the correlation between pragmatic intensity and the judgment shift is negligible. Rewrites that sound more intense do not produce larger or harsher shifts, and the decision flips in particular are unrelated to intensity. This indicates that the harshness bias is not simply a response to the stronger connotation of a rewrite. Rather, it arises from the figurative framing itself. We have added this analysis to Appendix G.

Appendix C reports decision-level inter-annotator agreement (88% literal, 91% figurative).

Area_Chair_gNEAOpenReview ↗

Please engage with the authors during the discussion period.

Dear reviewer,

The authors have provided a response to your review. Please ensure that you have read it and taken it into consideration when making your final recommendation. Even if you do not change your score, please post a response with your rationale.

Thank you for the thoughtful response and for adding the pragmatic-intensity analysis. I appreciate the clarification that connotation is constitutive to figurative language, and the new correlation results are a useful addition. They make the effect less likely to be explained by intensity alone.

I still have some reservations. The intensity ratings come from GPT-5-mini and remain a proxy rather than a direct separation of figurative form from connotation. I also did not see a direct check of whether the rewrites become more justifying or accusatory, beyond general emotional intensity. For the human study, the within-subject design and agreement figures help, but the five-annotator sample and the lack of a repeated unchanged-question control still leave recall and consistency only partially assessed.

Overall, I found the response helpful and I am closer to being convinced, but I do not think these points are fully resolved yet. I therefore keep my rating at 2 (weak rejection), while encouraging the authors to add a repeated unchanged-question control and a more direct tone analysis in the revision.

Reviewer_1ik7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether LLM judgments of people and their work are affected when a question is phrased figuratively (metaphor, hyperbole, sarcasm, simile) while the underlying context remains fixed. Using AITA-style moral scenarios from Social-Chem-101 and ETHICS, the authors generate figurative rewrites of only the judgment question (via GPT-5-mini, validated by humans), and measure perturbation and flip rates on a 7-point Likert scale across nine models from six families. They find a systematic bias toward harsher judgments under figurative framing; open-source models additionally flip their decision polarity up to 11.1% of the time, while closed-source models and humans mostly modulate judgment strength. Controls include a literal-rewrite baseline (ruling out pure paraphrase effects), polarity reversal (ruling out sycophancy), temperature and reasoning-effort ablations, and Wilcoxon significance tests. The effect is shown to extend to math reasoning, scientific reasoning, truthfulness QA, peer review, and resume review. Qualitative examples and an attention-based analysis (lookback ratio) suggest models over-attend to the figurative question and under-attend to the context.

Critical strengths
  1. Well-motivated and naturalistic research question.

  2. Careful experimental design with a fixed context and a varying question.

  3. Nine models across six families, five additional domains, temperature ablations, reasoning ablations, and statistical significance testing.

Critical weaknesses
  1. The focus is narrow and lacks insight from a learning perspective.

  2. Meaning preservation is questionable for hyperbole specifically. "aita for refusing to lend money to my brother after he's asked me a thousand times?" arguably asserts a history of repeated requests more strongly than the original "once again" does.

  3. The paper frames the harshness shift as a robustness failure, but its own human data show a nearly identical harsh perturbation rate to GPT-5.6 Sol (17.6% vs. 17.5%).

  4. The human evaluation is small, with only five annotators.

Questions for authors
  1. What fraction of generated rewrites was discarded/replaced by the 0.75 similarity filter, and do results change if the threshold is raised (e.g., 0.85)?

  2. How were the "gold labels" in Table 7 constructed, and on what subset?

  3. Can you report a per-instance correlation between the lookback-ratio shift and the Likert-score shift?

  4. For the polarity-reversal experiment, why are only flip rates reported (Table 3) and not perturbation rates?

3 comments
AuthorsOpenReview ↗

Rebuttal Part 1

Thank you for your review. We appreciate your feedback and suggestions.

W1: On scope and relevance to learning.

We respectfully disagree that the scope is narrow. Psycholinguistic research has long shown that the wording of a question shapes human judgment, from Loftus & Palmer’s (1974) eyewitness study to work showing that metaphors covertly influence human reasoning (Thibodeau & Boroditsky, 2011; 2013). Prior work on LLM robustness has mainly studied artificial perturbations and format changes. To our knowledge, ours is the first work to test whether naturally occurring figurative language shifts LLM judgments while the underlying evidence stays fixed. Our main findings are:

  • A consistent harshness bias. All nine models across six families judge the user more harshly when the question is phrased figuratively, even though the scenario is unchanged and human annotators confirm the meaning is preserved.
  • LLMs differ from humans. Humans adjust the strength of their judgment but rarely flip their decision or show any directional bias towards harsh or lenient judgments. LLMs, especially open-source models, flip their decision up to 11.1% of the time with a consistent bias towards harshness.
  • Not sycophancy. The bias holds when the question is reframed with either positive or negative polarity (Table 3). Beyond moral judgment. The same harshness appears in mathematical reasoning (GSM8K), scientific reasoning (GPQA), truthfulness (TruthfulQA), scientific peer review, and resume review.
  • Reasoning does not solve it. More reasoning effort within a model family improves robustness only slightly. Distilled reasoning models (R1-Distill-Qwen/Llama) are far more sensitive, with perturbation rates above 60% . Since LLM judgments are increasingly used as evaluators and automatic judges, a bias driven by how an input is phrased rather than what it says has broad practical consequences.

W2: On meaning preservation in hyperbole.

We thank the reviewer for this careful example. We want to emphasize three points.

First, regarding the given example, the history of repeated requests is already in the scenario context, which the model sees in both conditions (“came to me once again asking for money... almost never pays me back unless I remind him multiple times”). The phrase “a thousand times” indicates this event has happened before in an exaggerated tone, which is exactly what defines hyperbole.

Second, our human validation explicitly asked annotators to check that the rewrite “does not introduce or remove any information.” 98.5% of hyperbole rewrites passed (κ = 0.94 across categories).

Third, and most importantly, our conclusions do not depend on hyperbole itself. All figurative categories studied show the same trends, indicating the results are not tied to one specific form of rewrite.

W3: On the comparison to humans.

We thank the reviewer for raising this point. We believe it comes from a misunderstanding of two different metrics, and we would like to clarify the difference between them.

The perturbation rate counts any change in the Likert score, for example from “1: Strongly Disagree” to “2: Moderately Disagree.” The verdict stays the same, only its strength changes. The flip rate counts changes that cross the decision boundary, for example from “3: Slightly Disagree” to “5: Slightly Agree.” Here the verdict itself changes. We report both metrics because together they show how model and human behavior differ.

Our findings are that humans have a high perturbation rate towards harshness, but low and balanced flip rate. This indicates that humans change the strength of their decision but rarely alter their final verdict. Because the human flip rates in Figure 3 are equal, human moral judgments show no bias toward harshness in their final verdicts.

GPT-5.6 Sol shows a perturbation rate similar to that of humans, also biased toward harshness. However, its flip rate is higher and unlike humans, it is also biased toward harshness. While Gemini and Claude have a lower flip rate compared to GPT, they are still biased toward harshness unlike humans, who show no such bias.

This distinction is critical. A shift in judgment strength is a natural human response to expressive language, and our findings are consistent with prior work (O’Hara et al., 2010). However, this shift does not lead humans to change their final moral judgment. In contrast, LLMs show a bias toward harshness both in judgment strength (perturbation rate) and in their final judgment (flip rate).

Finally, we note that being as sensitive as humans would not by itself make models robust. The framing effects that inspired our work (Loftus & Palmer, 1974) are documented as biases in human judgment. When LLMs are used to judge moral advice, research papers, or job candidates, the verdict should depend on the evidence, which is identical in both conditions, and not on how the question is phrased.

AuthorsOpenReview ↗

Rebuttal Part 2

W4: On the size of the human study.

Our design is within-subject where each annotator rates both the literal and figurative variants of the same scenario, two months apart. Recalling the same annotators after this interval ensures that individual moral and cultural priors cancel out within each pair. The unit of analysis is therefore the 400 paired judgments, not the five annotators. Decision-level inter-annotator agreement is high (88% literal, 91% figurative, mentioned in Appendix C).

Also, the human study serves as a reference point for interpreting model behavior. The core claims rest on the model experiments (2,000 scenarios × 5 rewrites × 9 models, plus 5 additional domains).

Q1. Similarity Threshold.

Thank you for raising this. At the 0.75 threshold, 15.65% of generated rewrites were discarded and regenerated (reported in Table 1). We chose this value based on prior work on figurative rewriting [1] and on qualitative inspection during pilot runs.

As suggested by the reviewer, we re-ran the filter at 0.85. The share of rewrites discarded rises sharply, and unevenly across figurative types:

MetaphorHyperboleSarcasmSimile
Discarded @ 0.75 (%)21.5115.2713.5112.32
Discarded @ 0.85 (%)61.5841.3138.9149.14

Inspecting the additional discards shows that most are valid figurative rewrites according to human annotation reported in Table 1. The drop in similarity comes from how sentence embeddings handle figurative devices, which by design change the wording of a sentence while keeping its meaning. We present two examples below where the rewrites preserve the literal meaning, but it falls below 0.85:

Literal: AITA for finally ending things with my boyfriend?
Figurative (metaphor): AITA for closing the final chapter with my boyfriend?

Literal: AITA for refusing to lend money to my brother again?
Figurative (simile): AITA for refusing to lend money to my brother because he keeps coming back to me like a customer returning to the same ATM?

Metaphor describes a concept in terms of an entirely different domain. In the example below, a relationship is described as a book, and ending it becomes “closing the final chapter.” The meaning is unchanged, but the key words now come from the domain of books rather than relationships, so the surface overlap with the literal sentence drops.

Similarly, a simile has to introduce an explicit comparison (“like a…”). It adds lexical material that is absent from the literal sentence, which pushes cosine similarity down even when the core meaning is intact.

A stricter threshold therefore does not yield more faithful rewrites and systematically filters out good rewrites due to the linguistic nature of figurative language (out-of-domain mapping in metaphors, explicit comparison in simile)

Q2: How were the gold labels in Table 7 constructed, and on what subset?

Thank you for pointing this out, we omitted this detail. The gold labels in Table 7 are not constructed by us. They come from the original Commonsense subset of the ETHICS dataset, which assigns each AITA post a binary label derived from the Reddit community’s verdict on whether the poster acted wrongly (YTA) or not (NTA). We have added this clarification to Appendix C in the revised version.

Q3: Correlation Between Lookback Ratio and Likert Shift

As requested, we computed the sentence-level correlation between the change in lookback ratio and the change in Likert score (figurative − literal). The correlation is negligible (Spearman ρ = −0.02, Pearson r = −0.01). Thus, while figurative framing consistently shifts attention from the context toward the question at the aggregate level, the magnitude of this shift does not predict the magnitude of the judgment change for individual instances. This is consistent with prior findings that attention weights are an imperfect proxy for the factors driving a model’s output (Jain & Wallace, 2019). We have revised Section 5 to present the attention analysis as a systematic correlate of figurative framing rather than a direct explanation of the judgment shift.

Q4: For the polarity-reversal experiment, why are only flip rates reported (Table 3) and not perturbation rates?

Thank you for the question. We left the perturbation rates out of the main text for space, as they closely follow the trends in Figure 3. This is also visible in the flip rates themselves. The polarity-reversed flip rates in Table 3 closely track the original ones, therefore we just put the harsh rates in the main body as we did not want to repeat largely redundant numbers.

Area_Chair_gNEAOpenReview ↗

Please engage with the authors during the discussion period.

Dear reviewer,

The authors have provided a response to your review. Please ensure that you have read it and taken it into consideration when making your final recommendation. Even if you do not change your score, please post a response with your rationale.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your feedback and suggestions.

We have done our best to answer your questions and address your concerns, and we have revised our draft based on your feedback. The main changes are:

  • Updated models. We have replaced the Qwen3 and Gemma 3 families with the Qwen3.5 and Gemma 4 families. Overall results remain unchanged.
  • The reasoning model experiments have been moved into the main body.
  • Updated attention analysis. We now include the correlation between the lookback ratio and the Likert score shift.
  • Clarification of how the gold labels in Appendix C were constructed.
  • Additional experiment showing the practical implications of the harsher judgments caused by figurative language. Tested on resume review and scientific peer review, the model’s written assessment, not just its score, becomes harsher. This is important because people often rely on these written assessments to form an impression.
  • Additional experiment to separate the effect of figurative language from its connotations. We find no correlation between pragmatic intensity and the change in judgment score, suggesting that differences in intensity do not adequately explain the observed effect.

Ultimately, we believe the importance of this work extends far beyond changes in evaluation scores. Imagine a patient receiving inadequate pain treatment because they describe their symptoms as "my head is killing me" (hyperbole) rather than "I have a severe headache," an employee’s harassment complaint being dismissed because they say "my boss is the worst," (hyperbole) or a qualified job applicant losing an opportunity simply because the hiring manager asks whether the candidate "meets the bar" (metaphor). We believe that this paper is an important step towards addressing this bias.

We understand that you have a busy schedule. We would kindly ask you to review the revisions and let us know whether they address the questions and concerns you raised.

Regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.