- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents VoirDire, a new way to aggregate LLM-as-a-judge scores by using counterfactual calibration. To do this, the authors use paired interventions to identify nuisance-sensitive variation in scoring, which in turn allows one to select aggregation weights to reduce the effects of nuisance level variations. They evaluate across 12 datasets and compare to existing LLM-as-a-judge baselines. The results show a positive impact: VoirDire reduces sensitivity to the nuisance changes in scoring, and improves overall accuracy.
While the method of aggregation is well-motivated and interesting, there are issues with numerical reporting (no error bars, uncertainty, or otherwise statistical difference testing) that make the results harder to trust. The choice of which attributes qualify as nuisance attributes also seems a core issue that is not really addressed in the paper, although I do recognize that the aggregation is the more important contribution of the work.
Adding uncertainty and justification for nuisance attribute choice (even if the argument is that it’s not relevant to the contributions) would slightly increase my score. In general, however, I’m not sure about the overall novelty of this work as using paired interventions for LLM-as-a-judge for calibration is well established; I would also have liked to see comparisons to other recent baselines besides CARE and ISP.
Critical strengths
- The attenuation by strongest sensitive direction is an interesting and well-motivated approach.
- The method is compared against several known baselines and datasets.
- The ablation on calibration pairs is a solid result and shows that authors testing the robustness of their method under different parameters.
Critical weaknesses
- How do we properly decide which attributes are nuisances as opposed to non-nuisances? It's hard to know if know if using LLMs to decide is giving proper objective coverage here; I'd like to see that discussed in the paper.
- It’s unclear if using LLMs to generate the nuisance attributes themselves is a robust choice, would like to see a justification here. It’s hard to know if the LLMs are also biased in picking these attributes. I would also like to see ablations for this, and how different LLMs choose the attributes.
- Table 1 and 2 do not have uncertainty estimates. Figure 3 has no error bars. These should be added. You cannot correctly make claims such as "VoirDire improves on existing aggregation methods” otherwise. Statistical difference testing would be preferable on top of that.
- I think more recent baselines deserve comparison other than CARE and ISP alone.
Questions for authors
Recommendations:
- Figure 4 is somewhat hard to interpret and probably deserves more explanation.
- There are some missing citations on LLM as a judge, e.g. Dubois et al., ICML 2026; Bavaresco et al., ACL 2025.
Nitpicks:
- Switching between LLM-as-a-judge and LLM-judge
- The third paragraph of the intro talks about CARE without introducing it (I have to check the reference to see)
3 comments
(W3) On uncertainty estimates and statistical reporting. We have updated Tables 1 and 2 to report mean ± standard deviation across five cross-validation folds and added uncertainty estimates to Figure 3.
(W4) On comparisons with recent baselines. We have expanded our evaluation to include two recent baselines, SkillAggregation [3] and class-dependent Ising aggregation [4], on binary/preference tasks. These methods operate on discrete judge outputs and are not directly applicable to continuous-score tasks. As shown below, VoirDire outperforms both methods on five of six datasets and retains the best mean performance on three of six datasets across all baselines.
Table. Binary/preference accuracy (↑): mean ± SD across five evaluation folds.
| Dataset | SkillAggregation [3] | Ising [4] | VoirDire |
|---|---|---|---|
| Chatbot Arena | .524 ± .008 | .534 ± .025 | .645 ± .014 |
| RewardBench 2 | .571 ± .026 | .532 ± .008 | .667 ± .033 |
| LIAR2 | .684 ± .010 | .678 ± .010 | .680 ± .012 |
| HelpSteer3 | .480 ± .034 | .509 ± .032 | .607 ± .019 |
| CivilComments | .722 ± .009 | .723 ± .010 | .745 ± .011 |
| MT-Bench | .594 ± .018 | .513 ± .018 | .608 ± .042 |
(Q1) On Figure 4. We thank the reviewer for this suggestion. We have expanded the caption and discussion of Figure 4 in Section 5.3. Panels (a–b) show that judge sensitivity and target alignment are negatively correlated on 11 of 12 datasets. Panels (c–d) suggest that VoirDire tends to assign greater positive weight to less sensitive judges.
(Q2) On additional references. We thank the reviewer for suggesting Dubois et al. [5] and Bavaresco et al. [6], which we have incorporated into Section 2. Dubois et al. estimate systematic judge biases from observed evaluations, while Bavaresco et al. demonstrate that LLM-judge reliability varies across tasks and evaluation criteria. Both are highly relevant to our work.
(Q3) On terminology. We thank the reviewer for pointing this out. We have revised the terminology for consistency throughout the manuscript.
(Q4) On introducing CARE. We have expanded CARE to Confounder-Aware Aggregation at its first mention in the introduction.
References [1] Chen et al. (2024). Humans or LLMs as the Judge? A Study on Judgement Bias. EMNLP 2024.
[2] Wang et al. (2024). Large Language Models are not Fair Evaluators. ACL 2024.
[3] Sun et al. (2025). SkillAggregation: Reference-free LLM-dependent Aggregation. ACL 2025.
[4] Balasubramanian et al. (2026). Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models. arXiv.
[5] Dubois et al. (2026). Quantifying Biases in LLM-as-a-Judge Evaluations. ICML 2026.
[6] Bavaresco et al. (2025). LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks. ACL 2025.
Dear Reviewer gHDt,
Thank you again for your time and effort in reviewing our paper. We believe our rebuttal addresses your main concerns through additional experiments and manuscript revisions:
- Nuisance definition: We clarify that nuisance attributes are features that may influence judge scores without causally affecting the evaluation target.
- LLM proposer robustness: We clarify that LLMs are used as an optional way to automate candidate intervention proposals, rather than as authoritative sources of nuisance identification. A new ablation with six different LLM proposers shows consistent gains over the strongest baselines (Appendix E.6).
- Recent baselines: We extend our evaluation to include SkillAggregation and class-dependent Ising aggregation, with VoirDire outperforming both on five of six binary/preference datasets (updated Tables 1–2).
- Uncertainty estimates, presentation, and references: We report mean ± standard deviation in Tables 1–2 and add uncertainty estimates to Figure 3. We also expand the discussion of Figure 4 and incorporate the suggested references (Sections 2 and 5.3).
As the author–reviewer discussion period comes to a close, we would be happy to address any remaining concerns or questions.
We appreciate your thoughtful feedback and engagement!
We thank the reviewer for their thoughtful comments and constructive suggestions. We appreciate their recognition of the spectral attenuation mechanism, broad empirical evaluation, and detailed analyses. Below, we address the reviewer's concerns.
On novelty. Prior work has used controlled interventions to study biases in LLM-based evaluation [1, 2]. VoirDire integrates task-specific intervention proposal and selection with a calibration method that uses paired responses across multiple judges to identify shared nuisance-sensitive directions. These directions inform both spectral attenuation and intervention-aware aggregation. Our theoretical analysis further establishes why observational judge agreement alone cannot generally identify these directions.
(W1) On defining nuisance attributes. Nuisance attributes are features that may influence judge scores without causally affecting the intended evaluation target. Whether an attribute is a nuisance depends on the evaluation criterion. For example, verbosity may be irrelevant to factual correctness but relevant when evaluating conciseness. In practice, LLM assistants propose candidate intervention families based on task-specific evaluation criteria and dataset examples, followed by preservation checks and validation-based selection (Appendix B.2). We have added this clarification to Appendix B.5.
(W2) On using LLMs to propose nuisance attributes. We use LLM assistants to propose candidate intervention families, rather than as an authoritative source of nuisance identification. Candidates are generated from unlabeled examples, undergo preservation checks, and are subsequently selected using validation performance (Appendices B.2 and B.5). Importantly, VoirDire does not require LLM-generated interventions; domain experts can also specify deterministic transformations or rewriting instructions.
To examine sensitivity to proposer choice, we additionally compare six LLM proposers on Yelp, RewardBench 2, and Chatbot Arena, where VoirDire achieves particularly large gains. Each proposer generates 12 candidate intervention families using the same unlabeled examples and instructions. We fix the rewrite model, judge panels, evaluation splits, and downstream selection procedure across proposers (Appendix E.6).
All six proposers retain gains over the strongest existing aggregation baselines, reducing MAE by 0.347–0.413 on Yelp and improving accuracy by 5.09–6.29 percentage points on RewardBench 2 and 3.79–4.51 points on Chatbot Arena. These results suggest that VoirDire's gains are not specific to a particular LLM proposer, despite variation in the proposed interventions.
Table. Effect of intervention proposer choice: mean ± SD across five evaluation folds.
† Original main-experiment results are shown for reference; the six new proposers use a common, fixed candidate-pool construction protocol.
‡ Strongest existing baseline for each dataset: CARE-SVD (Yelp), AVG (RewardBench 2), and CARE-SVD (Chatbot Arena).