- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper investigates the mechanism behind how LLM judges make their decisions in a binary classification framework where a (question, answer) pair is classified as correct or incorrect. It introduces “static” and “dynamic” heads, which are attention heads whose average vote favors the same verdict regardless of the true label and those whose average vote flips with the label, respectively. The authors find that static heads make up the majority of attention heads in LLM judges and also account for most of the heads' total contribution. Beyond correlational analyses, they also performed interventions on the top-k strongest heads in each acceptance and rejection class to establish a causal relationship between head outputs and the model’s decision. An ablation study shows that removing important static heads changes the true positive and true negative rates in predictable ways. Rescaling the strongest acceptance and rejection heads’ outputs using a probe for the decision can improve judge classification accuracy across 4 judges and 4 answer models. Overall, this paper offers an interesting hypothesis about LLM judges’ mechanism, but its evidence for this hypothesis is currently limited by a lack of important baselines in the interventional experiments.
Critical strengths
- The paper introduces an interesting hypothesis that LLM judges make predictions based on “votes” from static heads that mostly output one label, either “correct” or “incorrect”.
- The decomposition of the verdict margin into head contributions is exact, and the theory is correct, though generic.
- The ablations in section 4 provide initial evidence for the heads’ causal effect on the output.
Critical weaknesses
- Figure 3a: the largest change is from error level 0 to 1. After level 1 the strengths of the heads seem relatively constant irrespective of the error level, which undermines the figure’s main claim.
- Figure 3b: baselines are missing, such as probes on random or non-static heads. The lack of baselines make it difficult to determine the significance of the accuracy.
- Figure 5 undercuts the claim that static heads carry the correctness signal. The head with the clearest shift (L27H8) is a dynamic head, and the static acceptance head (L33H16) shifts least.
- Section 4.2: both methods feed in the prediction of a probe trained on labels from the same benchmark. The prompting baselines does not use labels, so the comparison is unfair. For a better baseline, consider using f(x) as the classifier directly, or add it to m(x). Currently, we cannot rule out that the probe alone matches Voting Calibration’s accuracy.
- Table 2: the baselines and introduced methods are evaluated on slightly different sets, which may unfairly reward the paper’s methods.
Questions for authors
- How accurate is the probe f(x) on its own, used directly as a classifier or added to the margin as m(x) + cf(x)? Does Voting Calibration beat both?
- What is Pr(|m(x) − S_static(x)| >= |S_static(x)|) for each model, with MLP and other non-head terms included? How often do sign(m) and sign(S_static) agree?
3 comments
[W5]. "The baselines and proposed methods are evaluated on slightly different datasets."
We thank the reviewer for pointing this out. In the original paper, we evaluated the Base Model and the two prompting baselines on the full benchmark to provide a comprehensive assessment of their performance. We acknowledge that this resulted in different evaluation splits and agree that a shared test set provides a fairer comparison. Accordingly, we have revised the Table 2 and its description in Sec 4.2, evaluating all methods on the same 956 examples. Although some numerical values have changed slightly, the conclusions stated in our paper remain unchanged. The results preserve our main conclusions:
- Voting Calibration achieves the highest ACC across all four models, improving over Base by +14.3, +8.2, +3.3, and +6.1 percentage points, respectively. It also achieves the highest balanced accuracy on all four models.
- VC consistently outperforms Dynamic Steering, despite both using the same correctness signal, supporting the effectiveness of calibrating existing static votes over directly steering head representations.
- VC improves both TPR and TNR on Qwen3-8B and Gemma4-12B. On Ministral3-3B and Qwen3.5-9B, it introduces class-wise trade-offs, while still achieving the highest overall and balanced accuracy.
These results confirm that our main findings hold under a common evaluation split.
[Q2]. "What is the probability of the residual contribution matching or exceeding the static score, and how often do their predictions agree?"
We thank the reviewer for this suggestion. We evaluated both quantities on all 1,084 valid -MATH examples. Let and , which includes all remaining contributions. The results are:
Model Agreement Correct Incorrect Qwen3-8B 93.91% 95.38% 97.07% 93.43% Qwen3.5-9B 48.94% 51.15% 71.08% 28.09% Ministral3-3B 81.07% 18.93% 19.10% 18.73% Gemma4-12B-IT 80.98% 79.13% 85.89% 71.31% Importantly, Theorem 2.2 establishes a sufficient condition for prediction agreement, not an equivalence: even when , the two scores can still have the same sign. The results reveal substantial architectural variation in this agreement. We emphasize that our claim concerns the predominance and causal influence of static heads within attention-head contributions, not their ability to reproduce the full-model margin independently. The remaining components can still substantially shape the final judgment.
I'd like to thank the authors for their responses. They have sufficiently addressed my concerns, so I am raising my score.
We thank the reviewer for their comments. We address their concerns below:
[W1]. "The strengths after level 1 may differ with error levels"
[W2]. "Figure 3b lacks random-head and non-static-head baselines."
[W3]. "Figure 5 appears to weaken the claim that static heads carry correctness information."
[W4 & Q1]. "Direct probe prediction and additive margin correction are needed."