acceptodds
Under review as a conference paper at ICLR 2027

When Framing Changes the Score: A Comparative Study of LLM Judge Representations

Abstract

Large language models used as judges can change their ratings in response to an answer's framing as well as its content. We study how these sources of score change relate internally and what their relationship implies for controlling ratings. Behavioral measurements across seven judges and nine benchmarks show widespread but heterogeneous framing responses. A controlled comparison in Llama-3.1-8B-Instruct and Qwen3-4B-Instruct-2507 matches reviewer-consensus framing to known final-answer errors on the original rating and score decrease. Both sources have concentrated activation changes, and their mean directions align closely at the final blocks, with cosines of and . Interventions clarify how these directions influence scoring. A framing direction's components can differ in the magnitude and sign of their effects, and a component with little effect at its original magnitude can control ratings after independent scaling. Directions learned from unframed ratings also control scores, while ranking preservation varies across configurations. In Llama, moving each framed rating toward its unframed reference can narrow the score gap between clean and corrupted answers. Together, the source comparison and interventions connect framing responses to broader scoring behavior: distinct input changes approach similar late-layer mean directions, and control along these directions extends beyond the targeted rating.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, framing sensitivity, score representations, activation steering, model evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WrfTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how contextual framing (framing cues) affects ratings of LLM-as-a-judge, and how changes in internal representations caused by framing are related to changes in representations caused by errors in true answers. The paper first conducts a large-scale behavioral study covering seven review models, nine benchmarks and seven categories of framing. Results show that adverse framing generally reduces ratings, but that the magnitude and even direction of its effect vary significantly across prompt types, task domains, models, and review prompt words. Controlled characterization experiments focused on Llama-3.1-8B-Instruct and Qwen3-4B-Instruct-2507. The authors compare the hidden state shift caused by unfavorable "reviewer consensus" or "review status" metadata to the shift caused by modifying the final conclusion of the answer. Framing samples and content error samples are accurately matched based on baseline, original rating, and rating drop. For unfavorable consensus framing, the average displacement directions produced by the two input changes gradually become highly aligned in later layers, and the cosine similarities on the final block reach 0.880 and 0.901 respectively. The authors further decompose the framing direction based on the low-rank reference space learned from the framing-free scoring and apply intervention on the hidden state of the last input token of the model. Experiments have shown that the geometric size of a component does not reliably predict the size or sign of its scoring effect. Directions obtained by framing can change the scoring, but general clean sample scoring directions usually have similar effects. On Llama, inverting the direction of consensus framing brings the framing score close to the no-framing score, but at the same time significantly changes the answer ordering and closes the score gap between correct answers and artificially corrupted answers.

Critical strengths
  1. The empirical design is comprehensive and transparent. The paper combines a large-scale behavioral study covering seven review models with controlled analysis on two white-box models, with independent training, dev, and test sets, exact matching of rating changes, neutral metadata controls, alternative wording templates, output format checks, confidence intervals clustered by question, and detailed reporting of sample support and output validity. These designs lend greater credibility to the paper's behavioral observations about framing sensitivity and reveal significant heterogeneity that a single summary metric may mask.
  2. Rather than just reporting the change in scores caused by the intervention, the paper systematically assesses the collateral effects of the intervention. The authors also measured the degree of answer order preservation, output validity, the impact on the score of clean answers, and the impact on the discrimination between clean answers and artificially corrupted answers. This is important. The paper finds that seemingly successful recovery of framing scores can simultaneously lead to severe ordering disruption and reduced error discrimination. This has practical significance for evaluating activation steering-based review model debiasing or control methods.
Critical weaknesses
  1. Deep alignment cannot yet prove "framework-specific shared representations". Both groups first limited the score drop samples, and then accurately matched the baseline, original score and drop amplitude; the representation was taken from the last input token before the score was generated, and the similarity of the average displacement of the two groups was compared. Under this structure, the average displacement of both groups is likely to contain a common "generating lower score" component - the frame direction itself is highly aligned with the contrast direction of high and low scores for unframed samples, and the unframed scoring direction can also produce similar intervention effects. Therefore, the cosine similarity of 0.880/0.901 in the last layer is not enough to prove that the two share more specific internal calculations. Although the paper admits that it is impossible to identify a unique causal path, it still gives it a greater explanatory weight and requires more meaningful null hypothesis comparisons, such as comparing the alignment of other irrelevant perturbations after matching the same score transfer, or recalculating the alignment after eliminating the score prediction subspace.
  2. Functional experiments did not demonstrate the specificity or practical control effect of frame orientation. The universal high and low score directions without a frame can often reproduce the scoring effect of the frame direction; the complete frame direction on Qwen does not reach the preset ordering preservation threshold in the test set; the backward recovery experiment does not have any non-random directions on the development set that simultaneously satisfy the output validity and ordering constraints. Although the strongest conformity recovery significantly improved the target score, it reduced the rank correlation to 0.203 and narrowed the difference between correct and error scores from 3.565 to 1.944. Therefore, the intervention can only prove that the relevant direction affects the score, not that it selectively encodes the frame; the evidence supports the cautious conclusion of "non-specific score control and its risks" rather than "the identified direction explains the frame response".
Questions for authors
  1. Regarding the conditionalized null hypothesis: If you select two classes of input perturbations that are unrelated to each other, and apply exactly the same filtering and matching procedures to them—i.e., matching baseline, original score, and score drop magnitude—how much cosine similarity will you get between the mean displacement directions of the two classes? Furthermore, if score-related subspaces that are richer than the current scalar/high–low reference space of rank two are removed, is there still significant alignment between Bandwagon framing and answer errors? These comparisons would materially affect my interpretation of the 0.880/0.901 cosine similarities: are they evidence specific to framing, or are they a general result of conditioning on the same score transfer?
  2. Regarding same-question matching and misconstruction: If we only analyze the 150 Llama matches and the 26 Qwen matches that have the same parent question, will the late-level alignment results still be similar? Conversely, what if we exclude all matches from the same parent question? Could the authors further test for content errors that keep the reasoning process and the final answer consistent with each other, rather than just replacing the final answer field? These experiments help determine whether the results are affected by shared raw inputs or whether they arise primarily from apparent "inference-conclusion inconsistencies" generated by the current incorrect construction.
  3. Regarding the specificity of the intervention: Are there any hold-out test settings where framing orientation is better than clean high–low, scalar-ridge, neutral metadata, or match content error orientation in selectively removing the effects of framing while maintaining clean-answer-wrong-answer discrimination, under the conditions of matching mean rating changes while matching ordering preservation and output validity? If not, I think the paper should more explicitly state the conclusion as a "general, non-specific control of scoring" rather than a control of framing representations.
Reviewer_sD2SOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how framing the answer in an LLM-as-Judge setup changes the score. Typically, LLM quality ratings are elicited by providing the prompt and response to an LLM, embedded in a prompt that asks for a rating (in the context of this paper, an integer score between 1 and 10). This paper adds statements (cues) to the prompt that correspond to known biases, such as authority bias and bandwagon bias, to test whether the LLM ratings are affected by such indicators.

The paper finds that cues have noticeable effect, but the effect strength differs between cue types. The analysis in the main paper focuses more on negative framing than on positive framing of biases, but it seems that negative effects are stronger. Finally, the paper analyses whether judges can be steered with vector addition and find mixed effects. Generally, it seems to be possible to steer the score, but the paper doesn't state a clear conclusion of these experiments.

Critical strengths

I think the finding that some modifications of the prompt do not affect the score is genuinely interesting, because understanding this could inform developing more robust judge and reward models, i.e., models that are more robust to adversarially written judges, and it could help mitigate reward hacking.

Critical weaknesses

To me, the following weaknesses stand out:

(W1) The paper is very difficult to understand, which makes it difficult to recommend the paper for acceptance. Many details are unclear, not introduced, or deferred to the appendix. It is possible to understand the general idea and experimental setup, but many evaluations or experimental setups are opaque, and the setup differs considerably between experiments. However, a paper should be self-contained and have a reasonable educational value, so readers can easily understand the results as well as their motivation and significance.

(W2) The significance of the results is unclear to me. I do not find it surprising that prompting can influence the scores, and this may even be desired (for example, to bias judges to be stricter or more lenient depending on the task). Arguably, the biases evaluated in this paper could indeed represent undesired behavior, but even assuming this, it is unclear how or why they would be added to a scoring prompt.

(W3) Similarly, the motivation and consequences of the presented findings remain unclear. The paper does not explain why we or the general community should care about these particular findings. It only empirically states that prompt framing influences scores and internal representations, and that steering effects exist in LLM judges.

Questions for authors

The only question that could seriously convince me to reconsider the score is if the authors can give a convincing explanation why their findings are practically relevant and what improvements of LLM judges they enable or motivate.

Even then, as the paper is currently hard to read, I would recommend to thoroughly revise it first and develop a clearer storyline.

Reviewer_QDvKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how contextual framing and substantive answer errors affect LLM judges’ scores and internal representations. It combines a broad behavioral audit with controlled experiments in Llama-3.1-8B-Instruct and Qwen3-4B-Instruct-2507. Its central finding is that reviewer-consensus framing and final-answer errors produce closely aligned late-layer mean displacements when matched on baseline rating and score decrease. Component interventions further examine how geometric energy relates to score control, while recovery experiments measure changes in rankings and clean–corrupted answer separation.

Critical strengths
  1. Controlled comparison: Matching framing and content-error examples on benchmark, baseline score, and score decrease strengthens the comparison between different sources of score changes.

  2. Useful intervention design: Parallel/residual decompositions and comparisons at original and normalized scales distinguish geometric magnitude from functional influence.

  3. Multiple evaluation criteria: Assessing paired recovery, ranking fidelity, formatting compliance, and clean–corrupted separation reveals effects that average score changes alone would miss.

  4. Substantial supporting analysis: Neutral metadata controls, a second white-box model, and extensive sensitivity results improve the empirical foundation.

Critical weaknesses
  1. The content-error control supports a narrower interpretation than general error evaluation. Altering only the final answer can introduce a reasoning–conclusion inconsistency. The observed alignment could therefore reflect sensitivity to this corruption pattern rather than to coherent substantive errors generally. Furthermore, the strongest alignment evidence concerns reviewer-consensus framing; Refinement has only five matched test examples in Llama and one in Qwen, preventing reliable generalization across cue families.

  2. Reproducibility and human-validation details remain incomplete. The supplied manuscript does not fully specify generation and rewriting prompts, rewriting-model identity, quantitative edit budgets, and sampling procedures. Human equivalence testing provides aggregate evidence, but the polarity composition and polarity-stratified results are not reported. The acknowledged aggregate limitation leaves uncertainty about preservation of quality within individual conditions.

  3. Recovery results require clearer baseline context and broader replication. Bandwagon template-A test rank correlation is already 0.376 before steering, compared with 0.203 at the reported endpoint. Without adjacent baseline rows, readers may overattribute poor ranking to the intervention. Failure of the tested nonrandom configurations establishes a limitation of the evaluated search, rather than structural impossibility of recovery. The recovery and clean–corrupted gap analyses also remain limited to Llama.

  4. Scoring-policy sensitivity complicates interpretation. Llama’s first-line policy was adopted after inspecting unsteered outputs but before steering search; subsequent common-reference and response-curve analyses were retrospective. Qwen’s protocol was specified before outcome acquisition. Additionally, the different strict-format penalty conventions in Appendices H and J make comparisons difficult. Numerical score changes, formatting transitions, and baseline penalties should be presented separately.

  5. Robustness to intervention selection remains uncertain. Forward effects are modest, and some generic controls perform comparably, although several paired contrasts are statistically distinguishable. Results depend on discrete ratings, fitted directions, and a coarse strength grid. Bootstrap intervals condition on the training fit and development selection, leaving uncertainty about stability across these choices.

Questions for authors
  1. Does the late-layer alignment persist when content errors are coherent throughout the answer, rather than introduced only in the final field?

  2. Can you provide the missing construction details and human-equivalence results stratified by cue polarity?

  3. Can Table 4 include zero-strength ranking baselines, and can the recovery and clean–corrupted gap analyses be replicated in Qwen?

  4. Can you jointly report valid-pair numerical effects, formatting transitions, and zero-strength penalty baselines? Can collateral effects also be reported as signed changes alongside absolute changes?

  5. How stable are the intervention comparisons under denser strength grids and repeated training fits? Can per-item transition counts clarify effects obscured by integer quantization?

  6. How does the behavioral setup relate to CALM (Justice or Prejudice?) and Inside the Unfair Judge, and which datasets, constructions, or findings overlap?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.