acceptodds
Under review as a conference paper at ICLR 2027

Earlier Scores Count More: Causal Anchoring in Multi-Attribute LLM Evaluation

Abstract

LLM-as-a-judge protocols often score multiple criteria in a single response and aggregate them using an unweighted mean. Criterion order is known to shift scores, but correlation between earlier and later scores does not distinguish anchoring from shared dependence on item quality. We test for self-anchoring—the influence of a judge’s own earlier score tokens on its later scores—by intervening directly on generated score tokens. Holding the item, rubric, criterion order, and other preceding scores fixed, we set an earlier score above or below its baseline and measure the expected later score. Across five open-weight judges evaluated on teledermatology images and text, the mean controlled direct effect is positive for every judge and dataset. In both text settings, knocking out score-to-score attention removes the positive effect, while blocking intermediate positions does not. This dependence is concentrated in late decoder layers. Replacing earlier scores with a fixed reference value removes 97–98% of variance in aggregate scoring across criterion permutations. Measuring each position's total effect on the aggregate, with later scores generated sequentially, shows that the first output position carries roughly 5× its nominal weight in the smaller judges. Scoring one criterion per call improves agreement with dermatologists for most image criteria, but not for the aggregates, and it lowers correlation with human helpfulness on text. Equal nominal weights therefore do not imply equal criterion influence, and removing self-anchoring may not necessarily improve the validity of aggregate scoring.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, LLM evaluation, causal intervention, interpretability, evaluation bias

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Xt1xOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies self-anchoring in multi-attribute LLM evaluation. The research question is interesting, and the authors provide several analyses of the observed behavior. However, I have major concerns about the overall quality of the work. The experiments rely on very small datasets and a limited selection of models. Several key methodological choices are also not sufficiently justified, which raises questions about the reliability and generalizability of the findings. Overall, I do not find the current experimental evidence strong enough to support the paper's main claims, and I lean toward rejection.

Critical strengths
  1. Clear motivation. The paper raises an interesting question about how earlier scores may influence later scores in multi-attribute LLM evaluation. The problem is easy to understand and relevant to LLM-as-a-Judge.

  2. Multiple perspectives in the analysis. The authors examine self-anchoring through score interventions, attention knockout, and effective weight analysis. These experiments provide different perspectives on the observed behavior.

Critical weaknesses
  1. Writing suggestion. The Introduction should briefly highlight the main findings or takeaways of the paper in a short paragraph, instead of presenting them as a list of contributions.

  2. The experimental datasets are too small. SCIN contains 10,408 images, but the authors only use 20 images for the order experiment and 96 for the intervention experiments. Text Setting 1 is based on just 10 author-written requests, and the quality of the constructed responses is not sufficiently validated. For Text Setting 2, the main intervention and mechanistic experiments use only 40 responses from HelpSteer, but the dataset contains 37,000 responses. I don't think it can draw any convincing, meaningful, or generalizable conclusions from such limited data.

  3. Arbitrary baseline selection. Why are all preceding scores fixed to 5 when constructing the baseline? The authors do not examine whether choosing a different reference value would affect the results.

  4. Sensitivity to the intervention design. The authors use baseline ±3 as the main intervention on a 1–10 scoring scale. Although Appendix A also tests ±1 and ±2, the estimated effects vary considerably with the perturbation magnitude and sample selection, particularly for the image judge. Why was ±3 chosen as the main setting, and how reliable are the reported effect sizes given their sensitivity to these choices?

  5. Limited validation under natural scoring conditions. The main direct-effect experiments rely on manually constructed contexts through teacher forcing. The authors do not directly compare these results with those obtained from naturally generated score sequences. How can we be sure that the measured effects reflect self-anchoring in real LLM evaluation, rather than the specific experimental setup?

  6. Limited and questionable model selection. The number of models evaluated is insufficient, especially for the text experiments. Most of the main results rely on only two small judges (1.5B and 2B). Why is the VLM Qwen2-VL-2B used for text evaluation without any image input? The two text settings also use different models and datasets. It is hard to convince me with such a limited and unusual model selection.

Questions for authors

Please refer to the weaknesses.

3 comments

Thank you for the detailed review. We address each point below, and the General Response summarizes the new experiments. Effects are the change in a later score's expected value per one-point change in an earlier score. The paper gives 95% intervals.

W1. Introduction

The introduction now ends with a paragraph of main findings.

W2. Datasets

ExperimentSubmissionRevision
Order20 photographs96 (7 judges); 716 (2)
Image intervention96 photographs716 (2 judges)
Text intervention30 and 40 responses500 HelpSteer2 (9 judges); UltraFeedback (4)
Knockout30 and 40 responses100–200 HelpSteer2 (5 of 6 judges)

We use 716 photographs, one per SCIN case, because only these were rated by three dermatologists, as the gradability label requires. The larger samples reproduce the original estimates within 0.002 on photographs, and the effect is positive in all ten judges on HelpSteer2. Blind graders found a flaw in all 20 flawed Setting 1 responses and none in the 10 strong ones (Appendix C).

W3. Reference value of 5

We chose 5 because it sits near the middle of the 1–10 scale, keeping baselines away from the ends where ±3 does not fit. Any fixed value can act as an anchor, so we constructed the baselines at 3, 7, and from the judge’s own generated scores. Relative to 5, the effect changes by at most 0.017:

Earlier scores set to357judge's own
Gemma 4 12B, images0.1480.1470.1470.147
Ministral 3 14B, images0.1220.1250.1290.133
Qwen3.5-122B-A10B, images0.1730.1580.1700.169
Gemma 4 12B, 100 HelpSteer20.0920.0910.0920.108
Ministral 3 14B, 100 HelpSteer20.1180.1250.1310.138

W4. Step size

We chose ±3 as a large change that stays on the 1–10 scale for most image baselines. Each step below uses only baselines where ±3 fits both ways (Appendix A):

Judge, data±1±2±3
Qwen2-VL-2B, Setting 10.2350.2280.209
Qwen2.5-1.5B, Setting 20.2070.2030.197
Qwen2-VL-2B, images0.0290.0620.059
Gemma 4 12B, images0.1430.1420.139
Ministral 3 14B, images0.1040.1280.123
Qwen3.5-122B-A10B, images0.1740.1550.155
Gemma 4 12B, 100 HelpSteer20.0990.1070.090
Ministral 3 14B, 100 HelpSteer20.1350.1350.122

In text, ±3 is slightly smaller than ±1, so it is conservative. In larger judges the step changes the effect by at most 0.024. The sensitivity is largest in the 2B image judge, whose estimate depends on the baseline (mostly 6 to 8): at ±1 it is 0.157 overall but 0.029 where ±3 fits.

W5. Teacher forcing and naturally generated score sequences

We had judges sample their scores 25 times per item without forcing. We regressed later on earlier scores within each item, so the item is held fixed.

Judge, itemsSampledForced
Gemma 4 12B, 96 photographs0.1460.152
Ministral 3 14B, 96 photographs0.1210.125
Qwen3.8-27B, 96 photographs0.1870.167
Gemma 4 12B, 300 HelpSteer20.1370.097
Ministral 3 14B, 300 HelpSteer20.1260.108

Sampled estimates are within 0.006 of the forced ones on Gemma and Ministral photographs and exceed them in the other three rows, so forcing does not inflate the effect.

W6. Model selection

Agreed, all ten judges score the same HelpSteer2 responses, and the text results no longer mix judges and datasets. Qwen2-VL-2B was chosen because the study arose from an on-device teledermatology quality tool. We also ran it on text to compare modalities.

Judge500 HelpSteer296 photographs
Qwen2.5-1.5B0.193text only
Qwen2-VL-2B0.2110.091
Gemma 4 12B0.1010.147
Ministral 3 14B0.1260.125
Qwen3.8-27B0.1430.165
Muse Glimmer-30B0.1560.096
Llama 3.3 70B0.134text only
Qwen2.5-VL-72B0.1580.140
Qwen3.5-122B-A10B0.1440.158
GLM-4.6V0.155 (first 100)0.149

Novelty

To our knowledge, this is the first causal mechanistic analysis of scores the judge itself has emitted. We hold the item fixed, change only earlier scores, and trace the effect with mediation analysis and attention knockout. It changes three practical conclusions:

  1. Fixing the earlier scores removes 98–99% of order sensitivity in small judges and 52–78% in larger ones.
  2. Blocking attention to earlier scores removes 98% of order variance in a small judge but at most 60% in larger ones, where part of the order sensitivity bypasses the score tokens.
  3. Scoring criteria in separate calls removes the dependence but rarely improves the aggregate, so it does not by itself fix aggregate validity.

Because order alone flips 31–57% of decisions in six larger judges at the median cutoff, we recommend reporting it with effective weights and flip rates.

We welcome further questions.

I sincerely thank the authors for their detailed response and the substantial effort they have put into improving the manuscript. I appreciate that most of my concerns have been addressed, and the additional experiments have significantly strengthened the paper.

However, I remain concerned about the extent of the revisions during the rebuttal period. The number of evaluated models has doubled, and the experimental coverage has expanded substantially across multiple datasets. These are not minor clarifications, but major additions to the original submission. While I appreciate the authors' efforts, I believe the quality of the original submission should still carry significant weight in our assessment.

I also want to raise a concern about the considerable disagreement in reviewer ratings. In particular, I find it difficult to understand how the original submission could justify a clear-acceptance rating (4). Even the positive review acknowledges the limited experimental coverage, which I consider a fundamental weakness rather than a minor limitation, especially for a paper focused on empirical evaluation. I would appreciate hearing from the reviewer who recommended clear acceptance about which specific contributions of the original submission justify such a strong recommendation, independently of the extensive experiments added during rebuttal.

Overall, I appreciate the authors' serious effort and constructive response. The revised manuscript is clearly stronger, but given my concerns about the original submission and the scale of the rebuttal-stage revisions, I will maintain my original rating.

Finally, I want to emphasize that my decision does not diminish my appreciation for the authors' hard work. I genuinely appreciate their willingness to address the reviewers' concerns and the effort they have invested in improving the paper. I encourage the authors to continue pursuing this research direction, and I wish them the best with their future work.

We thank the reviewer for their continued engagement and kind words.

To clarify, every new experiment reuses the submission's intervention, estimand and knockout procedure, applied to more judges and items as requested by reviewers. The additions test whether the original results generalize, and they reproduce:

ResultOriginal submissionExpanded
Effect, Gemma 4 12B0.1470.147 [0.142,0.153][0.142, 0.153] (96 photographs)0.1480.148 [0.146,0.150][0.146, 0.150] (716)
Effect, Ministral 3 14B0.1250.125 [0.121,0.128][0.121, 0.128] (96 photographs)0.1230.123 [0.122,0.124][0.122, 0.124] (716)
Order variance removed by fixing earlier scores, Qwen2-VL-2B / Qwen2.5-1.5B97.5%97.5\% / 98.2%98.2\% (Settings 1, 2)98.3%98.3\% / 98.4%98.4\% (100 HelpSteer2)

Each estimate is read from the judge's score-token probabilities with intervals clustered by item, so the original intervals were already narrow. The 716-photograph also estimates fall inside them.

The original submission also found the effect positive in all six judge-and-dataset settings tested, across five judges from three model families and two modalities. The one conclusion that did change was that larger judges carry the effect mainly through intermediate tokens, while small judges read the earlier score directly, so the revision accordingly limits the original mechanistic claim to small judges.

We hope the revised manuscript can be weighed together with the original in the final assessment.

Reviewer_LWacOpenReview ↗
Rating
3(−1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2(−1)
Read review
Paper summary

This paper studies how earlier scores influence later scores within a single multi-attribute LLM evaluation. It intervenes on earlier score values, uses attention knockout to investigate the mechanism, and measures how these dependencies change each criterion’s effective influence on the aggregate score. Experiments across text and image settings demonstrate positive average anchoring effects and disproportionate influence of early positions. Comparisons with human labels further show that removing these dependencies has mixed effects on predictive validity.

Critical strengths
  1. The paper provides convincing causal evidence that earlier score values influence later scores within a multi-attribute judgment. Controlled interventions, attention knockout, and replication across models and modalities give a coherent account of the phenomenon and its mechanism.

  2. The effective-weight analysis reveals how nominally equal criteria acquire unequal influence on the aggregate score, with order reversal showing that the excess influence follows output position. The human-label experiments add practical insight by showing that removing score dependencies can either improve or worsen agreement, depending on the evaluation setting.

Critical weaknesses

The main limitation is experimental coverage. The detailed text intervention studies use 30 and 40 responses, and the human-agreement comparison for the larger judges uses 60 images, leaving some uncertainty about effect magnitudes and differences between scoring strategies. Larger models replicate the anchoring effect, but the detailed attention analysis and strongest weight amplification are concentrated in small judges.

Questions for authors
  1. As an exploratory extension, have you considered retaining the joint scoring format while blocking all subsequent positions from attending to earlier score-value tokens? If explored, how does this compare with ordinary joint scoring and isolated scoring in terms of order sensitivity and agreement with human judgments?

  2. Beyond the distinction between criterion-level feedback and aggregate evaluation discussed in Section 5, do you have observations about which dataset or task properties favor joint versus isolated scoring? For example, might the choice depend on correlations between criteria or the nature of the human evaluation target?

2 comments

We thank the reviewer for the positive assessment and thoughtful questions. We have addressed each and updated the paper to reflect the changes, as discussed in our General Response to All Reviewers.

Experimental Coverage

Text datasets. We added HelpSteer2 and UltraFeedback (General Response, Datasets). Across ten judges, a one-point increase in an earlier score raises the expected later score by 0.101–0.211 points.

Human agreement. The 60 photographs are replaced by 480 for seven image judges and all 716 dermatologist-rated photographs for two. On the 716, isolated scoring changes aggregate AUROC by only +0.009 and +0.010, reproducing the earlier small differences. On the 480, isolation reliably improves the artifacts score in six of seven judges, but aggregate AUROC in only one (Section 4.5).

Attention analysis. We extended attention knockout (Geva et al., 2023) to four larger judges (Section 4.3).

Share of the effect that remains when the earlier score is hidden (100% means the mask has no effect):

Judge (responses)Hidden from later scoreHidden from intermediate tokens
Qwen2.5-1.5B (100)−2%110%
Qwen2-VL-2B (100)−8%106%
Gemma 4 12B (175)81%17%
Ministral 3 14B (200)91%10%
Muse Glimmer-30B (100)85%39%
Qwen2.5-VL-72B (40)75%43%

Small judges lose the whole effect when the later score cannot attend to the earlier score. Larger judges keep most of it then, but lose 57–90% when the earlier score is hidden from the intermediate tokens. The knockout could not be applied to the linear-attention layers of Qwen3.8-27B and Qwen3.5-122B-A10B (Appendix B).

Weight amplification. We agree, and the abstract now limits the 5× to the 1.5B and 2B judges. In four judges from 12B to 122B, the first criterion counts 1.7–2.6× its intended weight. Moving a criterion in the output while keeping its place in the rubric text shows that the excess follows output position, not rubric position (Section 4.4).

Q1. Attention blocking versus joint and isolated scoring

We explored joint scoring with every later position blocked from attending to earlier score-value tokens (Sections 4.4 and 4.5).

Human agreement. Blocking leaves agreement essentially unchanged in Ministral 3 14B:

MeasureJointAttention blocked
Correlation with HelpSteer helpfulness0.4160.431
AUROC against dermatologist gradability0.8160.814

Isolated scoring is similar (aggregate AUROC +0.016 on 480 photographs, not significant).

Order sensitivity. Share of the variance removed across seven criterion orders, relative to ordinary joint scoring, on 100 HelpSteer2 responses:

JudgeAttention blockedEarlier scores fixed
Qwen2.5-1.5B98%99%
Gemma 4 12B60%78%
Ministral 3 14B0.5%73%
Qwen2.5-VL-72B45%52%

Blocking removes less than fixing the earlier scores in every judge, most clearly in Ministral (0.5% against 73%). Fixing the scores leaves attention intact, whereas the mask also changes other token interactions. An equal-token masking control removes 24% of the direct effect in Ministral.

Isolated scoring has no earlier scores in its output, so this route is absent by construction, and we did not score it across orders. Rubric text position may still exert some influence, however Section 4.4 demonstrates a very small effect: being first in the rubric changes a criterion’s weight by −0.3 to 0.1, against 1.3 to 2.4 for being first in the output.

Q2. Which task properties favor joint versus isolated scoring?

Thank you for raising this important question. We agree this topic would benefit from further exploration and hope to investigate more systematically in future work. Our current results suggest that both criterion correlations and the human evaluation target may influence which is favored.

Correlations between criteria. Isolation helps most where joint scoring lowers a criterion's correlation with the preceding criteria (ρ=−0.70\rho = -0.70 across criteria after the first, three image judges). The artifacts criterion benefits most in all three judges.

Human evaluation target. When the target is an individual attribute, dependence on earlier scores can distort it, as the repeated gains for artifacts show. When the target is overall quality, earlier scores carry useful information: correlation with HelpSteer helpfulness is 0.254 joint against 0.113 isolated in Qwen2.5-1.5B, and 0.393 against 0.398 in Gemma 4 12B. Isolation can therefore improve individual criterion scores without improving, and sometimes while lowering, the validity of their aggregate.

This qualifies the advice to score attributes separately (Stureborg et al., 2024). Isolation appears to suit criterion-level feedback, while joint scoring does as well or better for an overall quality target. We welcome your further comments and suggestions.

Thank you for the thoughtful response and additional experiments, which address my concerns about experimental coverage.

My original clear-acceptance recommendation was based on the paper’s coherent and thorough investigation of an important evaluation issue. The controlled interventions, effective-weight analysis, and comparison with human judgments together provide really useful insights into how scoring order affects multi-attribute evaluation.

On reflection, I gave too little weight in my original assessment to the limited number of models and examples used in the mechanistic experiments. The additional experiments strengthen the paper, and I continue to find its central insight valuable. Overall, I am revising my rating to Weak acceptance.

Reviewer_z8MROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Currently, LLM judges are often asked to return evaluation scores for several criteria in a single output (usually in JSON format). This paper studies whether an LLM judge’s earlier score tokens causally shift its later scores in multi-criterion evaluation. The authors fix the item, overwrite one earlier score, and then measure the change in a later score’s expected value. Across five open judges (1.5B–27B) evaluated on skin photos and text, earlier scores consistently pull later scores in the same direction. The authors also use attention knockout to locate the effect in direct attention to score tokens in late layers. As a result, the first output position carries up to about 5× its nominal weight in small judges and about 1.8× in a 12B judge. Finally, scoring each criterion in isolation improves agreement with human labels for most individual criteria, but not for the aggregate score.

Critical strengths

I enjoyed the problem statement and its motivation. LLM judges are commonly asked to return scores for all criteria in a single output to save tokens, instead of evaluating each rubric criterion separately. This paper studies how later scores in such an output depend on earlier ones. The causal design for estimating this shift is clean and makes sense to me. I also enjoyed the attention knockout analysis, which identifies how the shift arises inside the model. Finally, I think this paper studies an important problem in LLM evaluation.

Critical weaknesses
  1. Most of the key results rely on small judges. The primary image judge (Qwen2-VL-2B) and both text judges have only 1.5B–2B parameters. Smaller models may align poorly with ground truth. Since the conclusions depend on how these judges behave, their limited quality also limits the strength of this study.
  2. Moreover, the judges are asked to return JSON directly. In practice, LLM evaluation often asks the judge to give a justification or to reason before scoring. The paper does not test whether anchoring persists under these more common setups.
  3. The data size is limited. Text Setting 1 contains only 30 responses written by the authors, and Text Setting 2 contains only 40 responses sampled from HelpSteer. Many public LLM preference datasets with multi-dimensional ratings (e.g., HelpSteer2, UltraFeedback) could be used to scale up the experiments.
Questions for authors
  1. In the Text Setting 1, why did we write 30 responses instead of using an existing multi-attribute dataset (e.g., HelpSteer2 or UltraFeedback)? How were the planted flaws validated, and do the flaw labels play any role in the analysis?
4 comments

We thank the reviewer for their close reading, constructive comments, and questions. We have addressed each and updated the paper to reflect the changes, as discussed in our General Response to All Reviewers.

W1. Most of the key results rely on small judges. ... Smaller models may align poorly with ground truth.

We now test ten judges, up to 122B (General Response, Models). We began with small judges because the study arose from development of an automated teledermatology quality assessment tool designed to run locally on patients' mobile devices.

The larger judges agree much better with dermatologists, and continue to display self-anchoring. Effects are in points per point, defined as the change in the later score's expected value per one-point change in the earlier score.

2B judgeJudges of 12B or more
AUROC against dermatologists' gradability labels0.60–0.740.78–0.85
Effect on 96 photographs (points per point)0.0910.096–0.165

On HelpSteer2 the eight text judges of 12B or more give 0.101–0.158, against 0.193 and 0.211 for the 1.5B and 2B judges. Better agreement with ground truth does not remove the effect, so it is not an artifact of weak judges (Sections 4.1 and 4.2). It also changes decisions in six larger judges, where criterion order alone flips 31–57% of accept-or-retake decisions (Sections 4.1 and 4.2).

W2. ... LLM evaluation often asks the judge to give a justification or to reason before scoring. The paper does not test whether anchoring persists under these more common setups.

We added both free-form reasoning before the scores, as in G-Eval [1], and a one-sentence justification before each score. The text is generated once and held fixed while we perturb the earlier score. The table gives the direct effect of the score and excludes any effect carried by rewritten text (Section 4.2).

Judge (96 photographs)Justification: without →\rightarrow withReasoning: without →\rightarrow with
Gemma 4 12B0.147→0.0740.147 \rightarrow 0.0740.147→0.0890.147 \rightarrow 0.089
Ministral 3 14B0.133→0.0750.133 \rightarrow 0.0750.125→0.0680.125 \rightarrow 0.068
Qwen3.5-122B-A10B0.169→0.1260.169 \rightarrow 0.1260.158→0.1940.158 \rightarrow 0.194
Qwen2-VL-2B0.117→0.2570.117 \rightarrow 0.2570.093→0.1530.093 \rightarrow 0.153

The effect of self-anchoring remains positive in all settings. Justification lowers it in the three larger judges, while reasoning lowers it in two judges and raises it in two. Overall, we conclude that neither is a reliable fix.

[1] Liu, Yang, et al. "G-Eval: NLG evaluation using GPT-4 with better human alignment." EMNLP (2023).

W3. The data size is limited. ... Many public LLM preference datasets with multi-dimensional ratings (e.g., HelpSteer2, UltraFeedback) could be used to scale up the experiments.

We added both suggested datasets (General Response, Datasets), and increased the sample size of experiments throughout. The effect remains similar across all sources:

Text dataJudgesEffect (points per point)
Original Settings 1 and 2 (30 and 40 responses)20.206 and 0.202
HelpSteer2 [2] (500 responses; 100 for GLM-4.6V)100.101–0.211
UltraFeedback [3]40.113–0.160

[2] Wang, Zhilin, et al. "HelpSteer2: Open-source dataset for training top-performing reward models." NeurIPS Datasets and Benchmarks (2024).

[3] Cui, Ganqu, et al. "UltraFeedback: Boosting language models with scaled AI feedback." ICML (2024).

Q1. In the Text Setting 1, why did we write 30 responses instead of using an existing multi-attribute dataset ...? How were the planted flaws validated, and do the flaw labels play any role in the analysis?

We wrote them to decorrelate the criteria. Because a weak response is usually weak on several criteria at once in these public multi-attribute datasets, the per-criterion ratings are correlated. Setting 1 pairs each of 10 requests with one strong response and two responses that each contain a deliberate flaw in one named criterion (e.g., an answer that names the correct capital but omits the reason). One criterion’s true quality is then low while the other five remain high.

Graders who had not written the responses and did not see the labels found a flaw in all 20 flawed responses, identified the targeted criterion in 19, and found none in the 10 strong ones (Appendix C). The labels are not used in any analysis. The same judge on HelpSteer2 gives a similar effect (0.211 against 0.206).

We are happy to clarify anything that remains unclear.

Dear Reviewer z8MR,

Thank you again for your review. As the discussion period is drawing to a close, we wanted to check whether our response and the revised manuscript address your concerns. In brief:

W1 (small judges). The revision tests ten judges up to 122B. The judges of 12B or more agree better with dermatologists (AUROC 0.78–0.85) and still show the effect, at 0.096–0.165 points per point on photographs and 0.101–0.158 on HelpSteer2.

W2 (justification or reasoning before scoring). We tested both in four judges. The effect remains positive in every case (Section 4.2).

W3 (data size). All ten judges now score HelpSteer2, and four also score UltraFeedback (0.113–0.160). The original two text judges give 0.211 and 0.193 on 500 HelpSteer2 responses, against 0.206 and 0.202 in the original settings.

If any point remains unclear, or there is a further analysis you would find useful, we would be glad to respond in the time remaining. If the revision resolves your concerns, we would be grateful if you would consider reflecting that in your final assessment.

Thank you for the rebuttal, and for adding larger judges and new datasets to the evaluation. I appreciate the effort that went into the revision. However, my concerns about the dataset setup remain. The samples are selected from the original datasets, and the justification provided does not yet convince me that the experimental results generalize beyond these selected subsets. For these reasons, I do not think the paper is ready for acceptance at this conference in its current form, though I encourage the authors to keep developing this direction.

To clarify the dataset setup: because our 95% intervals are bootstrapped over items, they already measure how much each estimate would move under a different draw from the same dataset. Where we scored the full labelled population, the subset matched it. The 96 photographs gave effects of 0.147 and 0.125, and all 716 dermatologist-rated SCIN cases gave 0.148 and 0.123 (Table 6). The effect was also not carried by a few items - it was positive for 96–100% of individual HelpSteer2 responses in every judge, and at every human helpfulness level.

Items were not selected or excluded based on judge outputs, and each sample was fixed by a seeded hash of the item IDs.

  • Photographs: the 716 SCIN cases rated by three dermatologists (the only cases with a gradability label) were sorted by a seeded hash of the case ID.
  • HelpSteer: the human agreement set was the first 200 validation responses in seeded hash order, and Setting 2 is the first 40 of these.
  • HelpSteer2 and UltraFeedback: For each dataset, we sampled 500 responses (one per prompt) stratified on a label hidden from the judge: 100 per rounded human helpfulness level (0–4) in HelpSteer2, and 100 per rounded mean GPT-4 rating (1–5) in UltraFeedback. Both samples were fixed prior to any scoring.

Stratification does not inflate the effect. We stratified so that low-quality items were represented. Reweighted to each dataset's natural quality distribution, the effect changes by −0.012 to +0.011 and stays positive for every judge:

JudgeHelpSteer2: stratifiedHelpSteer2: reweighted [95% CI]UltraFeedback: stratifiedUltraFeedback: reweighted [95% CI]
Qwen2.5-1.5B0.1930.193 [0.191, 0.195]––
Qwen2-VL-2B0.2110.210 [0.208, 0.211]––
Gemma 4 12B0.1010.094 [0.087, 0.101]0.1130.101 [0.093, 0.110]
Ministral 3 14B0.1260.131 [0.129, 0.133]0.1210.127 [0.125, 0.130]
Qwen3.8-27B0.1430.145 [0.142, 0.149]––
Muse Glimmer-30B0.1560.167 [0.163, 0.171]0.1600.161 [0.153, 0.169]
Llama 3.3 70B0.1340.144 [0.139, 0.149]0.1490.149 [0.136, 0.162]
Qwen2.5-VL-72B0.1580.164 [0.162, 0.167]––
Qwen3.5-122B-A10B0.1440.147 [0.143, 0.151]––
GLM-4.6V (first 100)0.1550.158 [0.152, 0.163]––

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their constructive feedback. We appreciate that they found the question relevant to LLM evaluation and valued the mechanistic analyses and causal design. We provide an updated submission with changes shown in blue.

The shared concern among reviewers was the scale and generalizability of the evidence. We address it with larger samples in varied datasets, replication in larger judges, and validation under natural score generation, while leaving our core methods unchanged. The intervention, estimand and knockout procedure are those of the submission, and the original estimates reproduce on the larger samples. The one new finding is that larger judges often carry the effect of self-anchoring through intermediate tokens.

Models

We now provide results from ten open-weight judges across six model families, eight of them 12B or larger (Section 3.2). The effect of self-anchoring remains positive in all ten and does not decline from 12B to 122B (Figure 3b).

Design

For each methodological consideration raised by reviewers, we evaluated alternatives that included varied reference values, unconstrained score sampling, reasoning or justification prior to scoring, smaller perturbation magnitudes, and two additional rubric formats. The effect of self-anchoring remains positive across all designs (Table 1). Self-anchoring also appears when judges generate all their scores freely. A one-point rise in an earlier score still raises later scores by 0.12 to 0.19 points, within 0.006 of or above the forced estimates (Table 16).

Ablations

The judge's attention to the earlier score is now masked in four larger judges in addition to the two small ones. Small judges read the earlier score directly, and larger judges through the tokens in between. In the larger judges, hiding the earlier score from the tokens in between removes 57 to 90% of the effect, and hiding it from the later score only 9 to 25% (Table 19). As a control, masking the same number of other tokens removes only 9 to 26%. On 100 HelpSteer2 responses in seven orders, holding the earlier scores fixed at each response's mean score removes 99% of the variation that criterion order causes in the total score in the two small judges, compared to 52-78% in four larger judges (Table 22). Self-anchoring therefore accounts for nearly all of the small judges' sensitivity to criterion order and for part of the larger judges' (Section 4.4).

Datasets

All ten judges now score HelpSteer2. Four judges also score UltraFeedback, and two score all 716 SCIN photographs rated by three dermatologists. The original samples provide nearly the same estimates as these larger ones. A one-point change in an earlier score moves a later score by 0.147 [0.142, 0.153] and 0.125 [0.121, 0.128] points on the original 96 photographs, against 0.148 [0.146, 0.150] and 0.123 [0.122, 0.124] on all 716 (Table 6). Tables 6, 13, 16, 19 and 22 give bootstrap 95% intervals over items.

Importance

Changing only the criterion order flips the accept-or-retake decision at each judge's median cutoff for 46% and 43% of the 716 photographs with Gemma 4 12B and Ministral 3 14B, whose AUROCs against the dermatologists' gradability labels are 0.83–0.84 and 0.78–0.80 (Section 4.1, Table 11). All ten judges tested exhibit self-anchoring, where their earlier scores pull later scores in the same direction. The effective weight of the first criterion counts about twice its intended weight, and up to about five times in the 1.5B and 2B judges (Section 4.4).

We address each reviewer's remaining points in the individual responses and welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.