acceptodds
Under review as a conference paper at ICLR 2027

Heads Take Sides: Imbalanced Static Voting in Unfair LLM Judges

Abstract

LLM judges can encode correctness-related information internally, yet how this information is converted into a final verdict remains unclear. By decomposing the verdict-logit margin into attention-head contributions, we identify a predominant head-level mechanism of **static voting** in LLM judges: opposing head groups consistently favor acceptance or rejection, while input-dependent voting strengths determine their competition. Although their voting directions remain fixed, their voting strengths vary with solution quality—acceptance heads weaken and rejection heads strengthen as errors accumulate—enabling input-dependent judgments through competition. This static voting structure recurs across model architectures and verdict formats, while targeted ablations confirm that the two camps causally push judgments in opposite directions. This account suggests that reliable correction should adapt the balance between the two camps rather than suppress either one. We therefore introduce **Voting Calibration**, which uses an internal correctness signal to modulate their voting strengths while preserving their specialized directions. Relying on a small reference set, Voting Calibration improves overall accuracy and true-negative rate by averages of 8.3% and 14.6%, respectively, across four models on -MATH benchmark.

Primary area:
interpretability and explainable AI
Keywords:
Transformers, Interpretability, LLM-as-a-Judge

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_d1SdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper investigates the mechanism behind how LLM judges make their decisions in a binary classification framework where a (question, answer) pair is classified as correct or incorrect. It introduces “static” and “dynamic” heads, which are attention heads whose average vote favors the same verdict regardless of the true label and those whose average vote flips with the label, respectively. The authors find that static heads make up the majority of attention heads in LLM judges and also account for most of the heads' total contribution. Beyond correlational analyses, they also performed interventions on the top-k strongest heads in each acceptance and rejection class to establish a causal relationship between head outputs and the model’s decision. An ablation study shows that removing important static heads changes the true positive and true negative rates in predictable ways. Rescaling the strongest acceptance and rejection heads’ outputs using a probe for the decision can improve judge classification accuracy across 4 judges and 4 answer models. Overall, this paper offers an interesting hypothesis about LLM judges’ mechanism, but its evidence for this hypothesis is currently limited by a lack of important baselines in the interventional experiments.

Critical strengths
  • The paper introduces an interesting hypothesis that LLM judges make predictions based on “votes” from static heads that mostly output one label, either “correct” or “incorrect”.
  • The decomposition of the verdict margin into head contributions is exact, and the theory is correct, though generic.
  • The ablations in section 4 provide initial evidence for the heads’ causal effect on the output.
Critical weaknesses
  • Figure 3a: the largest change is from error level 0 to 1. After level 1 the strengths of the heads seem relatively constant irrespective of the error level, which undermines the figure’s main claim.
  • Figure 3b: baselines are missing, such as probes on random or non-static heads. The lack of baselines make it difficult to determine the significance of the accuracy.
  • Figure 5 undercuts the claim that static heads carry the correctness signal. The head with the clearest shift (L27H8) is a dynamic head, and the static acceptance head (L33H16) shifts least.
  • Section 4.2: both methods feed in the prediction of a probe trained on labels from the same benchmark. The prompting baselines does not use labels, so the comparison is unfair. For a better baseline, consider using f(x) as the classifier directly, or add it to m(x). Currently, we cannot rule out that the probe alone matches Voting Calibration’s accuracy.
  • Table 2: the baselines and introduced methods are evaluated on slightly different sets, which may unfairly reward the paper’s methods.
Questions for authors
  • How accurate is the probe f(x) on its own, used directly as a classifier or added to the margin as m(x) + cf(x)? Does Voting Calibration beat both?
  • What is Pr(|m(x) − S_static(x)| >= |S_static(x)|) for each model, with MLP and other non-head terms included? How often do sign(m) and sign(S_static) agree?
3 comments

We thank the reviewer for their comments. We address their concerns below:

[W1]. "The strengths after level 1 may differ with error levels"

We thank the reviewer for pointing out that the visual presentation may overemphasize the transition from error level 0 to 1. We re-examined the underlying intensity tensor used for Fig. 3a, which has shape (8,6)(8,6), corresponding to 2×ranknum=82\times\mathrm{rank_num}=8 heads and error levels 0--5. Averaging the signed intensities over all eight heads gives [0.0401,−0.1851,−0.2008,−0.2404,−0.2417,−0.2625][0.0401,-0.1851,-0.2008,-0.2404,-0.2417,-0.2625]. Thus, the 0$\to$1 change is indeed the largest (−0.2252-0.2252), but the signal continues to change from level 1 to 5 (−0.0775-0.0775), accounting for 25.6% of the total level-0-to-5 change. All eight heads also show a negative net change from level 1 to 5. In terms of magnitude, the mean absolute intensity increases from 0.4597 to 0.4998 (+8.7%) over levels 1--5.

Overall, the most significant change occurs between 0 and 1, followed by further, albeit subtler and continuous, changes. This is consistent with expectations: the transition from incorrect to correct (0–1) should entail a substantial, qualitative shift, whereas changes in intensity between similar levels of error are relatively smaller but persistent.

[W2]. "Figure 3b lacks random-head and non-static-head baselines."

We thank the reviewer for this helpful suggestion. We added a random-head baseline by randomly sampling 32 attention heads and training probes on Math-Shepherd using the same setup as Fig. 3b. The results are:

ClassMean (%)Variance (pp²)Max (%)Min (%)
Correct59.928.7365.0651.86
Incorrect61.023.7765.0456.25

In comparison, probes on the selected static heads achieve 72–78% accuracy on correct solutions and 86–90% on incorrect solutions (Fig. 3b). Notably, even the best-performing random head falls below the lowest accuracy of the selected static heads in both classes. This contrast indicates that the correctness information captured by our probes is substantially more pronounced in the identified static heads than in randomly selected heads, rather than being a generic property of arbitrary attention heads.

[W3]. "Figure 5 appears to weaken the claim that static heads carry correctness information."

We thank the reviewer for this observation. We agree that L33H16 exhibits a smaller displacement, as noted in Section 3.4. However, our claim is that static verdict preferences can coexist with correctness information, not that every static head encodes correctness more strongly than dynamic heads. Even though the changes in L27H8 are more distinct, this does not invalidate the accuracy-related information contained in the static header. Moreover, the static rejection head L33H16 also exhibits a clear error-dependent shift. The direct evidence comes from Fig. 3: probes on L33H16 achieve 75% accuracy on correct solutions and 89% on incorrect solutions on Math-Shepherd. Thus, L27H8’s clearer trajectory and L33H16’s weaker visible movement do not contradict our claim. In a nutshell, Fig. 5 illustrates that the geometric manifolds of representative static and dynamic heads carry correctness signal, rather than uniform signal strength across static heads or their superiority over dynamic heads.

[W4 & Q1]. "Direct probe prediction and additive margin correction are needed."

We thank the reviewer for pointing this out. We agree that label-matched controls are needed. We therefore added both suggested baselines: direct classification with sign⁡(f(x))\operatorname{sign}(f(x)) and additive margin correction, using the same fitted f(x)f(x).

For m(x)+cf(x)m(x)+cf(x), we selected cc solely on the 128-example reference set by maximizing balanced accuracy over 801 grid points in [−20M,20M][-20M,20M], where MM is the reference-set median of ∣m(x)∣|m(x)|; ties favored smaller ∣c∣|c|. We then fixed cc and evaluated all methods on the same 956 held-out examples. No test labels were used to select cc.

Modelff-only ACC (%)m+cfm+cf ACC (%)VC ACC (%)
Ministral3-3B53.3553.0370.0
Qwen3-8B61.9263.7074.1
Qwen3.5-9B68.6268.7274.3
Gemma4-12B69.5671.3481.2

Voting Calibration achieves higher accuracy than both label-matched baselines on every model. These results address the possibility that direct probe prediction or the tested additive correction already matches its accuracy.

[W5]. "The baselines and proposed methods are evaluated on slightly different datasets."

We thank the reviewer for pointing this out. In the original paper, we evaluated the Base Model and the two prompting baselines on the full benchmark to provide a comprehensive assessment of their performance. We acknowledge that this resulted in different evaluation splits and agree that a shared test set provides a fairer comparison. Accordingly, we have revised the Table 2 and its description in Sec 4.2, evaluating all methods on the same 956 examples. Although some numerical values ​​have changed slightly, the conclusions stated in our paper remain unchanged. The results preserve our main conclusions:

  1. Voting Calibration achieves the highest ACC across all four models, improving over Base by +14.3, +8.2, +3.3, and +6.1 percentage points, respectively. It also achieves the highest balanced accuracy on all four models.
  2. VC consistently outperforms Dynamic Steering, despite both using the same correctness signal, supporting the effectiveness of calibrating existing static votes over directly steering head representations.
  3. VC improves both TPR and TNR on Qwen3-8B and Gemma4-12B. On Ministral3-3B and Qwen3.5-9B, it introduces class-wise trade-offs, while still achieving the highest overall and balanced accuracy.

These results confirm that our main findings hold under a common evaluation split.

[Q2]. "What is the probability of the residual contribution matching or exceeding the static score, and how often do their predictions agree?"

We thank the reviewer for this suggestion. We evaluated both quantities on all 1,084 valid μ\mu-MATH examples. Let S=Sstatic(x)S=S_{\mathrm{static}}(x) and e(x)=m(x)−S(x)e(x)=m(x)-S(x), which includes all remaining contributions. The results are:

ModelPr⁡(∥e∥≥∥S∥)\Pr(\|e\|\geq\|S\|)AgreementCorrectIncorrect
Qwen3-8B93.91%95.38%97.07%93.43%
Qwen3.5-9B48.94%51.15%71.08%28.09%
Ministral3-3B81.07%18.93%19.10%18.73%
Gemma4-12B-IT80.98%79.13%85.89%71.31%

Importantly, Theorem 2.2 establishes a sufficient condition for prediction agreement, not an equivalence: even when ∣e(x)∣≥∣S(x)∣|e(x)|\geq|S(x)|, the two scores can still have the same sign. The results reveal substantial architectural variation in this agreement. We emphasize that our claim concerns the predominance and causal influence of static heads within attention-head contributions, not their ability to reproduce the full-model margin independently. The remaining components can still substantially shape the final judgment.

I'd like to thank the authors for their responses. They have sufficiently addressed my concerns, so I am raising my score.

Reviewer_GVquOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how LLM judges assess the correctness of mathematical solutions, finding that most attention heads exhibit a consistent average bias toward either acceptance or rejection across both correct and incorrect inputs. Despite these static biases, their contribution strengths retain correctness-related information, and ablation experiments support their opposing causal effects on judgments. Based on these findings, the authors propose Voting Calibration, which uses a small labeled dataset to extract correctness signals from internal representations and dynamically adjusts the contributions of acceptance- and rejection-biased heads. The method reports an average accuracy improvement of approximately 8.3 percentage points across four models, although differences between baseline and intervention test sets, together with the absence of a direct correctness-probe baseline, leave the magnitude of the gains and the added value of the intervention insufficiently established.

Critical strengths
  1. An informative distinction between head bias and correctness signals. The finding that attention heads can exhibit a consistent directional bias while their contribution strengths still encode correctness-related information is conceptually useful. It suggests that static acceptance or rejection tendencies do not necessarily imply an absence of input-sensitive information.

  2. Causal evidence from ablation experiments. Ablating attention heads provides stronger evidence than observational analysis alone. The reported opposing effects of acceptance- and rejection-biased heads support their causal involvement in the model’s judgments.

  3. Promising results across multiple models. The reported average improvement of approximately 8.3 percentage points across four models suggests that the approach merits further investigation. Its use of a small labeled calibration set also indicates potential applicability when annotation resources are limited.

Critical weaknesses
  1. Differences in test sets weaken the main empirical comparison. If the baseline and intervention are evaluated on different test sets, the reported gains cannot be cleanly attributed to Voting Calibration. A paired evaluation on identical examples is needed to establish the magnitude of the improvement.

  2. Calibration robustness and practical requirements need clearer characterization. The use of a small labeled dataset is promising, but its effectiveness may depend on sample selection, label balance, and similarity to the evaluation distribution. Moreover, the need to access and modify internal attention-head contributions limits applicability to models that expose these internals, excluding typical black-box judging APIs.

Questions for authors

None

1 comment

We thank the reviewer for their comments. We address their concerns below:

[W1]. "A paired evaluation on identical examples is needed."

We thank the reviewer for pointing this out. In the original paper, we evaluated the Base Model and the two prompting baselines on the full benchmark to provide a comprehensive assessment of their performance. We acknowledge that this resulted in different evaluation splits and agree that a shared test set provides a fairer comparison. Accordingly, we have revised the Table 2 and its description in Sec 4.2, evaluating all methods on the same 956 examples. Although some numerical values ​​have changed slightly, the conclusions stated in our paper remain unchanged. The results preserve our main conclusions:

  1. Voting Calibration achieves the highest ACC across all four models, improving over Base by +14.3, +8.2, +3.3, and +6.1 percentage points, respectively. It also achieves the highest balanced accuracy on all four models.
  2. VC consistently outperforms Dynamic Steering, despite both using the same correctness signal, supporting the effectiveness of calibrating existing static votes over directly steering head representations.
  3. VC improves both TPR and TNR on Qwen3-8B and Gemma4-12B. On Ministral3-3B and Qwen3.5-9B, it introduces class-wise trade-offs, while still achieving the highest overall and balanced accuracy.

These results confirm that our main findings hold under a common evaluation split.

[W2]. "The robustness and practical requirements of calibration need clarification, and access to internal attention heads limits applicability to black-box judge APIs."

We thank the reviewer for raising these important practical considerations. Regarding reference-set selection, we conducted an additional experiment on Qwen3-8B using 10 random seeds, each with a different 128-example reference set. The standard deviations across runs were only 0.2 percentage points for probe accuracy, 0.3 points for Dynamic Steering accuracy, and 0.1 points for Voting Calibration accuracy, indicating low sensitivity to reference-set sampling in this setting.

Our current experiments, however, use balanced reference sets sampled from the target benchmark and do not establish robustness to arbitrary class imbalance or distribution shifts. These remain important directions for future investigation. In practice, representative labeled examples from the target domain would be needed for recalibration when the data distribution changes.

Regarding deployment, we clarify that Voting Calibration is designed as a white-box intervention, not a prompt-level technique intended for black-box APIs. Access to internal attention-head outputs is essential to its mechanism, as the method explicitly reweights existing head contributions rather than modifying only the input prompt or final output. Therefore, compatibility with standard black-box APIs is outside its intended scope.

Reviewer_ecJbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper reveals that LLM judges primarily rely on static acceptance and rejection heads rather than dynamically switching their voting directions. These heads maintain fixed voting preferences, while their voting strengths adapt to the correctness of the evaluated response. Causal ablation experiments show that acceptance and rejection heads exert opposing influences on the final judgment. Based on this finding, the authors propose Voting Calibration to improve judging accuracy by rebalancing these two groups of heads .

Critical strengths
  1. This paper presents an interesting and intuitive mechanistic finding.

  2. A clear decomposition of head contributions to verdict logits is provided.

  3. The main findings are consistent across four different models.

  4. The ablation experiments provide convincing causal evidence.

Critical weaknesses
  1. The definition of “static” may make same-sign heads naturally dominant, so it is unclear whether static voting is a judge-specific computational mechanism or a generic property of logit decomposition.

  2. Voting Calibration relies on a supervised correctness signal trained from labeled target-domain examples. Thus, it is not clear how much improvement comes from the proposed voting mechanism versus the correctness probe itself.

  3. Baselines and interventions are not reported on exactly the same evaluation subset.

  4. Experiments are restricted almost entirely to mathematical correctness judging, making the generalization of this work unclear.

Questions for authors

Please see the weakness part.

1 comment

We thank the reviewer for their comments. We address their concerns below:

[W1]. "The definition of 'static' may inherently favor same-sign heads."

We thank the reviewer for raising this important concern. Our definition simply partitions the (μh+,μh−)(\mu_h^+, \mu_h^-) plane into four quadrants, assigning two opposite-sign quadrants to static heads and the other two same-sign quadrants to dynamic heads. This partition is symmetric, with no prior preference for either category. Therefore, the observed predominance of static heads is an empirical finding rather than an artifact of the definition.

Importantly, we do not claim that static voting is exclusive to LLM judges or universal across all logit decomposition tasks. Our work focuses on correctness judgment, where we provide systematic evidence across architectures, behavioral analyses, and causal interventions. If similar structures exist in other prediction tasks, this would suggest broader relevance rather than undermine our findings.

[W2.] "Voting Calibration's gains may come from the supervised correctness probe rather than the proposed voting mechanism."

We thank the reviewer for pointing this out. We added controlled comparisons using the identical correctness signal f(x)f(x), fitted on the same 128 reference examples, and evaluated all methods on the same 956 held-out examples. The table reports accuracy (%); DS denotes Dynamic Steering and VC denotes Voting Calibration.

Modelff-onlyDSVC
Ministral3-3B53.3558.670.0
Qwen3-8B61.9270.974.1
Qwen3.5-9B68.6270.574.3
Gemma4-12B69.5675.581.2

Direct probe prediction does not match VC's accuracy on any model. Moreover, VC consistently outperforms DS, although both interventions use the same correctness signal. These results show that access to the probe alone does not explain the observed performance; how this signal is applied is equally important. By making effective use of this signal—as demonstrated by the DS and VC methods—one can achieve significant gains. This is precisely where the innovation of our approach lies.

[W3]. "The baselines and proposed methods are evaluated on slightly different datasets."

We thank the reviewer for highlighting this distinction. In the original paper, we evaluated the Base Model and the two prompting baselines on the full benchmark to provide a comprehensive assessment of their performance. We acknowledge that this resulted in different evaluation splits and agree that a shared test set provides a fairer comparison. Accordingly, we have revised the Table 2 and its description in Sec 4.2, evaluating all methods on the same 956 examples. Although some numerical values ​​have changed slightly, the conclusions stated in our paper remain unchanged. The results preserve our main conclusions:

  1. Voting Calibration achieves the highest ACC across all four models, improving over Base by +14.3, +8.2, +3.3, and +6.1 percentage points, respectively. It also achieves the highest balanced accuracy on all four models.
  2. VC consistently outperforms Dynamic Steering, despite both using the same correctness signal, supporting the effectiveness of calibrating existing static votes over directly steering head representations.
  3. VC improves both TPR and TNR on Qwen3-8B and Gemma4-12B. On Ministral3-3B and Qwen3.5-9B, it introduces class-wise trade-offs, while still achieving the highest overall and balanced accuracy.

These results confirm that our main findings hold under a common evaluation split.

[W4]. "Generalization beyond mathematical correctness."

We thank the reviewer for pointing this out. To examine the generality of our findings, we conducted an additional experiment on TruthfulQA, a factual correctness benchmark. Using all 790 question pairs from its updated binary version, we evaluated the correctness of both the Best Answer and the Best Incorrect Answer with the same judging prompt and head-level attribution procedure used in Fig.2. Across 4 different models, the resulting heatmaps exhibit the same symmetric pattern: head contributions form opposing sign configurations for correct and incorrect answers. This reproduces the central structural phenomenon in a non-mathematical factual evaluation task and extends the empirical scope of our analysis beyond mathematical correctness. The results are shown in Appendix D.1 Fig 10-11.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.