- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies self-anchoring in multi-attribute LLM evaluation. The research question is interesting, and the authors provide several analyses of the observed behavior. However, I have major concerns about the overall quality of the work. The experiments rely on very small datasets and a limited selection of models. Several key methodological choices are also not sufficiently justified, which raises questions about the reliability and generalizability of the findings. Overall, I do not find the current experimental evidence strong enough to support the paper's main claims, and I lean toward rejection.
Critical strengths
-
Clear motivation. The paper raises an interesting question about how earlier scores may influence later scores in multi-attribute LLM evaluation. The problem is easy to understand and relevant to LLM-as-a-Judge.
-
Multiple perspectives in the analysis. The authors examine self-anchoring through score interventions, attention knockout, and effective weight analysis. These experiments provide different perspectives on the observed behavior.
Critical weaknesses
-
Writing suggestion. The Introduction should briefly highlight the main findings or takeaways of the paper in a short paragraph, instead of presenting them as a list of contributions.
-
The experimental datasets are too small. SCIN contains 10,408 images, but the authors only use 20 images for the order experiment and 96 for the intervention experiments. Text Setting 1 is based on just 10 author-written requests, and the quality of the constructed responses is not sufficiently validated. For Text Setting 2, the main intervention and mechanistic experiments use only 40 responses from HelpSteer, but the dataset contains 37,000 responses. I don't think it can draw any convincing, meaningful, or generalizable conclusions from such limited data.
-
Arbitrary baseline selection. Why are all preceding scores fixed to 5 when constructing the baseline? The authors do not examine whether choosing a different reference value would affect the results.
-
Sensitivity to the intervention design. The authors use baseline ±3 as the main intervention on a 1–10 scoring scale. Although Appendix A also tests ±1 and ±2, the estimated effects vary considerably with the perturbation magnitude and sample selection, particularly for the image judge. Why was ±3 chosen as the main setting, and how reliable are the reported effect sizes given their sensitivity to these choices?
-
Limited validation under natural scoring conditions. The main direct-effect experiments rely on manually constructed contexts through teacher forcing. The authors do not directly compare these results with those obtained from naturally generated score sequences. How can we be sure that the measured effects reflect self-anchoring in real LLM evaluation, rather than the specific experimental setup?
-
Limited and questionable model selection. The number of models evaluated is insufficient, especially for the text experiments. Most of the main results rely on only two small judges (1.5B and 2B). Why is the VLM Qwen2-VL-2B used for text evaluation without any image input? The two text settings also use different models and datasets. It is hard to convince me with such a limited and unusual model selection.
Questions for authors
Please refer to the weaknesses.
3 comments
I sincerely thank the authors for their detailed response and the substantial effort they have put into improving the manuscript. I appreciate that most of my concerns have been addressed, and the additional experiments have significantly strengthened the paper.
However, I remain concerned about the extent of the revisions during the rebuttal period. The number of evaluated models has doubled, and the experimental coverage has expanded substantially across multiple datasets. These are not minor clarifications, but major additions to the original submission. While I appreciate the authors' efforts, I believe the quality of the original submission should still carry significant weight in our assessment.
I also want to raise a concern about the considerable disagreement in reviewer ratings. In particular, I find it difficult to understand how the original submission could justify a clear-acceptance rating (4). Even the positive review acknowledges the limited experimental coverage, which I consider a fundamental weakness rather than a minor limitation, especially for a paper focused on empirical evaluation. I would appreciate hearing from the reviewer who recommended clear acceptance about which specific contributions of the original submission justify such a strong recommendation, independently of the extensive experiments added during rebuttal.
Overall, I appreciate the authors' serious effort and constructive response. The revised manuscript is clearly stronger, but given my concerns about the original submission and the scale of the rebuttal-stage revisions, I will maintain my original rating.
Finally, I want to emphasize that my decision does not diminish my appreciation for the authors' hard work. I genuinely appreciate their willingness to address the reviewers' concerns and the effort they have invested in improving the paper. I encourage the authors to continue pursuing this research direction, and I wish them the best with their future work.
We thank the reviewer for their continued engagement and kind words.
To clarify, every new experiment reuses the submission's intervention, estimand and knockout procedure, applied to more judges and items as requested by reviewers. The additions test whether the original results generalize, and they reproduce:
| Result | Original submission | Expanded |
|---|---|---|
| Effect, Gemma 4 12B | (96 photographs) | (716) |
| Effect, Ministral 3 14B | (96 photographs) | (716) |
| Order variance removed by fixing earlier scores, Qwen2-VL-2B / Qwen2.5-1.5B | / (Settings 1, 2) | / (100 HelpSteer2) |
Each estimate is read from the judge's score-token probabilities with intervals clustered by item, so the original intervals were already narrow. The 716-photograph also estimates fall inside them.
The original submission also found the effect positive in all six judge-and-dataset settings tested, across five judges from three model families and two modalities. The one conclusion that did change was that larger judges carry the effect mainly through intermediate tokens, while small judges read the earlier score directly, so the revision accordingly limits the original mechanistic claim to small judges.
We hope the revised manuscript can be weighed together with the original in the final assessment.
Thank you for the detailed review. We address each point below, and the General Response summarizes the new experiments. Effects are the change in a later score's expected value per one-point change in an earlier score. The paper gives 95% intervals.
W1. Introduction
The introduction now ends with a paragraph of main findings.
W2. Datasets
We use 716 photographs, one per SCIN case, because only these were rated by three dermatologists, as the gradability label requires. The larger samples reproduce the original estimates within 0.002 on photographs, and the effect is positive in all ten judges on HelpSteer2. Blind graders found a flaw in all 20 flawed Setting 1 responses and none in the 10 strong ones (Appendix C).
W3. Reference value of 5
We chose 5 because it sits near the middle of the 1–10 scale, keeping baselines away from the ends where ±3 does not fit. Any fixed value can act as an anchor, so we constructed the baselines at 3, 7, and from the judge’s own generated scores. Relative to 5, the effect changes by at most 0.017:
W4. Step size
We chose ±3 as a large change that stays on the 1–10 scale for most image baselines. Each step below uses only baselines where ±3 fits both ways (Appendix A):
In text, ±3 is slightly smaller than ±1, so it is conservative. In larger judges the step changes the effect by at most 0.024. The sensitivity is largest in the 2B image judge, whose estimate depends on the baseline (mostly 6 to 8): at ±1 it is 0.157 overall but 0.029 where ±3 fits.
W5. Teacher forcing and naturally generated score sequences
We had judges sample their scores 25 times per item without forcing. We regressed later on earlier scores within each item, so the item is held fixed.
Sampled estimates are within 0.006 of the forced ones on Gemma and Ministral photographs and exceed them in the other three rows, so forcing does not inflate the effect.
W6. Model selection
Agreed, all ten judges score the same HelpSteer2 responses, and the text results no longer mix judges and datasets. Qwen2-VL-2B was chosen because the study arose from an on-device teledermatology quality tool. We also ran it on text to compare modalities.
Novelty
To our knowledge, this is the first causal mechanistic analysis of scores the judge itself has emitted. We hold the item fixed, change only earlier scores, and trace the effect with mediation analysis and attention knockout. It changes three practical conclusions:
Because order alone flips 31–57% of decisions in six larger judges at the median cutoff, we recommend reporting it with effective weights and flip rates.
We welcome further questions.