acceptodds
Under review as a conference paper at ICLR 2027

VoirDire: Counterfactual Calibration for Reliable LLM-as-a-Judge Aggregation

Abstract

LLM-as-a-judge systems often aggregate multiple judges to improve evaluation reliability. However, judge agreement can reflect shared biases toward verbosity, formatting, or fluency rather than the intended evaluation target. Addressing these shared biases is challenging because observational judge outputs alone cannot distinguish agreement driven by target quality from agreement driven by nuisance attributes. We introduce VoirDire, an intervention-calibrated aggregation method for LLM-judge panels. VoirDire applies controlled interventions that vary specified surface attributes while aiming to preserve the correct evaluation; it then uses paired changes in judge scores to identify nuisance-sensitive directions. It attenuates these directions and selects aggregation weights that retain variation across items while penalizing sensitivity to the interventions, making the final consensus less dependent on superficial features. We show theoretically that observational judge outputs alone cannot identify which shared direction is target-relevant, while paired interventions provide the contrast needed to identify nuisance-sensitive variation. We evaluate VoirDire on 12 datasets spanning continuous scoring, binary classification, and pairwise preference. Compared with the existing LLM-judge aggregation baselines, VoirDire reduces MAE by up to 26.8% on continuous-score tasks and improves accuracy by up to 5.3% on binary and preference tasks. These findings suggest that counterfactual calibration offers a more reliable and controllable basis for LLM-based evaluation than judge agreement alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, judge aggregation, counterfactual calibration, shared biases, reliable evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gHDtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents VoirDire, a new way to aggregate LLM-as-a-judge scores by using counterfactual calibration. To do this, the authors use paired interventions to identify nuisance-sensitive variation in scoring, which in turn allows one to select aggregation weights to reduce the effects of nuisance level variations. They evaluate across 12 datasets and compare to existing LLM-as-a-judge baselines. The results show a positive impact: VoirDire reduces sensitivity to the nuisance changes in scoring, and improves overall accuracy.

While the method of aggregation is well-motivated and interesting, there are issues with numerical reporting (no error bars, uncertainty, or otherwise statistical difference testing) that make the results harder to trust. The choice of which attributes qualify as nuisance attributes also seems a core issue that is not really addressed in the paper, although I do recognize that the aggregation is the more important contribution of the work.

Adding uncertainty and justification for nuisance attribute choice (even if the argument is that it’s not relevant to the contributions) would slightly increase my score. In general, however, I’m not sure about the overall novelty of this work as using paired interventions for LLM-as-a-judge for calibration is well established; I would also have liked to see comparisons to other recent baselines besides CARE and ISP.

Critical strengths
  • The attenuation by strongest sensitive direction is an interesting and well-motivated approach.
  • The method is compared against several known baselines and datasets.
  • The ablation on calibration pairs mm is a solid result and shows that authors testing the robustness of their method under different parameters.
Critical weaknesses
  • How do we properly decide which attributes are nuisances as opposed to non-nuisances? It's hard to know if know if using LLMs to decide is giving proper objective coverage here; I'd like to see that discussed in the paper.
  • It’s unclear if using LLMs to generate the nuisance attributes themselves is a robust choice, would like to see a justification here. It’s hard to know if the LLMs are also biased in picking these attributes. I would also like to see ablations for this, and how different LLMs choose the attributes.
  • Table 1 and 2 do not have uncertainty estimates. Figure 3 has no error bars. These should be added. You cannot correctly make claims such as "VoirDire improves on existing aggregation methods” otherwise. Statistical difference testing would be preferable on top of that.
  • I think more recent baselines deserve comparison other than CARE and ISP alone.
Questions for authors

Recommendations:

  • Figure 4 is somewhat hard to interpret and probably deserves more explanation.
  • There are some missing citations on LLM as a judge, e.g. Dubois et al., ICML 2026; Bavaresco et al., ACL 2025.

Nitpicks:

  • Switching between LLM-as-a-judge and LLM-judge
  • The third paragraph of the intro talks about CARE without introducing it (I have to check the reference to see)
3 comments

We thank the reviewer for their thoughtful comments and constructive suggestions. We appreciate their recognition of the spectral attenuation mechanism, broad empirical evaluation, and detailed analyses. Below, we address the reviewer's concerns.

On novelty. Prior work has used controlled interventions to study biases in LLM-based evaluation [1, 2]. VoirDire integrates task-specific intervention proposal and selection with a calibration method that uses paired responses across multiple judges to identify shared nuisance-sensitive directions. These directions inform both spectral attenuation and intervention-aware aggregation. Our theoretical analysis further establishes why observational judge agreement alone cannot generally identify these directions.

(W1) On defining nuisance attributes. Nuisance attributes are features that may influence judge scores without causally affecting the intended evaluation target. Whether an attribute is a nuisance depends on the evaluation criterion. For example, verbosity may be irrelevant to factual correctness but relevant when evaluating conciseness. In practice, LLM assistants propose candidate intervention families based on task-specific evaluation criteria and dataset examples, followed by preservation checks and validation-based selection (Appendix B.2). We have added this clarification to Appendix B.5.

(W2) On using LLMs to propose nuisance attributes. We use LLM assistants to propose candidate intervention families, rather than as an authoritative source of nuisance identification. Candidates are generated from unlabeled examples, undergo preservation checks, and are subsequently selected using validation performance (Appendices B.2 and B.5). Importantly, VoirDire does not require LLM-generated interventions; domain experts can also specify deterministic transformations or rewriting instructions.

To examine sensitivity to proposer choice, we additionally compare six LLM proposers on Yelp, RewardBench 2, and Chatbot Arena, where VoirDire achieves particularly large gains. Each proposer generates 12 candidate intervention families using the same unlabeled examples and instructions. We fix the rewrite model, judge panels, evaluation splits, and downstream selection procedure across proposers (Appendix E.6).

All six proposers retain gains over the strongest existing aggregation baselines, reducing MAE by 0.347–0.413 on Yelp and improving accuracy by 5.09–6.29 percentage points on RewardBench 2 and 3.79–4.51 points on Chatbot Arena. These results suggest that VoirDire's gains are not specific to a particular LLM proposer, despite variation in the proposed interventions.

Table. Effect of intervention proposer choice: mean ± SD across five evaluation folds.

ProposerYelp MAE ↓RewardBench 2 Acc. ↑Chatbot Arena Acc. ↑
Strongest baseline‡.921 ± .019.596 ± .021.583 ± .009
Original set†.492 ± .015.667 ± .033.645 ± .014
GPT-6.1 Sol.508 ± .016.649 ± .051.626 ± .027
GPT-6.1 Terra.554 ± .037.659 ± .054.625 ± .023
Claude Opus 5.5.549 ± .030.648 ± .047.628 ± .030
Claude Sonnet 5.5.552 ± .024.651 ± .058.625 ± .027
Gemini 4 Carbon.533 ± .020.647 ± .049.624 ± .033
Gemini 4 Argon.574 ± .066.657 ± .051.621 ± .035

† Original main-experiment results are shown for reference; the six new proposers use a common, fixed candidate-pool construction protocol.

‡ Strongest existing baseline for each dataset: CARE-SVD (Yelp), AVG (RewardBench 2), and CARE-SVD (Chatbot Arena).

(W3) On uncertainty estimates and statistical reporting. We have updated Tables 1 and 2 to report mean ± standard deviation across five cross-validation folds and added uncertainty estimates to Figure 3.

(W4) On comparisons with recent baselines. We have expanded our evaluation to include two recent baselines, SkillAggregation [3] and class-dependent Ising aggregation [4], on binary/preference tasks. These methods operate on discrete judge outputs and are not directly applicable to continuous-score tasks. As shown below, VoirDire outperforms both methods on five of six datasets and retains the best mean performance on three of six datasets across all baselines.

Table. Binary/preference accuracy (↑): mean ± SD across five evaluation folds.

DatasetSkillAggregation [3]Ising [4]VoirDire
Chatbot Arena.524 ± .008.534 ± .025.645 ± .014
RewardBench 2.571 ± .026.532 ± .008.667 ± .033
LIAR2.684 ± .010.678 ± .010.680 ± .012
HelpSteer3.480 ± .034.509 ± .032.607 ± .019
CivilComments.722 ± .009.723 ± .010.745 ± .011
MT-Bench.594 ± .018.513 ± .018.608 ± .042

(Q1) On Figure 4. We thank the reviewer for this suggestion. We have expanded the caption and discussion of Figure 4 in Section 5.3. Panels (a–b) show that judge sensitivity and target alignment are negatively correlated on 11 of 12 datasets. Panels (c–d) suggest that VoirDire tends to assign greater positive weight to less sensitive judges.

(Q2) On additional references. We thank the reviewer for suggesting Dubois et al. [5] and Bavaresco et al. [6], which we have incorporated into Section 2. Dubois et al. estimate systematic judge biases from observed evaluations, while Bavaresco et al. demonstrate that LLM-judge reliability varies across tasks and evaluation criteria. Both are highly relevant to our work.

(Q3) On terminology. We thank the reviewer for pointing this out. We have revised the terminology for consistency throughout the manuscript.

(Q4) On introducing CARE. We have expanded CARE to Confounder-Aware Aggregation at its first mention in the introduction.

References [1] Chen et al. (2024). Humans or LLMs as the Judge? A Study on Judgement Bias. EMNLP 2024.

[2] Wang et al. (2024). Large Language Models are not Fair Evaluators. ACL 2024.

[3] Sun et al. (2025). SkillAggregation: Reference-free LLM-dependent Aggregation. ACL 2025.

[4] Balasubramanian et al. (2026). Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models. arXiv.

[5] Dubois et al. (2026). Quantifying Biases in LLM-as-a-Judge Evaluations. ICML 2026.

[6] Bavaresco et al. (2025). LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks. ACL 2025.

Dear Reviewer gHDt,

Thank you again for your time and effort in reviewing our paper. We believe our rebuttal addresses your main concerns through additional experiments and manuscript revisions:

  • Nuisance definition: We clarify that nuisance attributes are features that may influence judge scores without causally affecting the evaluation target.
  • LLM proposer robustness: We clarify that LLMs are used as an optional way to automate candidate intervention proposals, rather than as authoritative sources of nuisance identification. A new ablation with six different LLM proposers shows consistent gains over the strongest baselines (Appendix E.6).
  • Recent baselines: We extend our evaluation to include SkillAggregation and class-dependent Ising aggregation, with VoirDire outperforming both on five of six binary/preference datasets (updated Tables 1–2).
  • Uncertainty estimates, presentation, and references: We report mean ± standard deviation in Tables 1–2 and add uncertainty estimates to Figure 3. We also expand the discussion of Figure 4 and incorporate the suggested references (Sections 2 and 5.3).

As the author–reviewer discussion period comes to a close, we would be happy to address any remaining concerns or questions.

We appreciate your thoughtful feedback and engagement!

Reviewer_SHZXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VoirDire uses controlled input interventions to estimate nuisance-sensitive directions in a panel of judge scores. It attenuates these directions and learns aggregation weights that balance across-item variation against intervention sensitivity. The paper provides an observational non-identifiability example and reports improvements on eleven of twelve datasets against the listed aggregation baselines.

Critical strengths

S1. The intervention-based identification source is compelling. Sections 3 and 4 clearly connect the ambiguity of observational agreement to the use of paired score changes.

S2. The evaluation covers diverse tasks and includes useful component, sensitivity, calibration-budget, and related-nuisance transfer analyses. The CivilComments exception and configurations with zero attenuation also help characterize the method.

Critical weaknesses

W1. The separation between model selection and final testing requires clarification. Section 5 delegates protocol details to Appendix B.2, which describes five random row-level splits, open-ended candidate-pool refinement using validation feedback, and one configuration selected by mean validation performance across the five splits. Unless there is an additional outer holdout, a row tested in one split may inform configuration selection through another split. This is a potential contamination route, not a verified implementation error, but it requires an explicit explanation and an independently held-out or nested evaluation.

W2. The target-invariance assumption underlying Section 4 is not adequately validated for the actual interventions. Surface preservation checks do not establish that changes in verbosity, fluency, or framing preserve the intended evaluation. Section 5.5 shows reduced score movement, but contraction can also reduce sensitivity. Independent human judgments of target preservation and positive controls with genuine quality changes are needed.

W3. The contribution of intervention structure is not isolated from validation supervision. Sections 3 and 5 use reference-based selection of families and hyperparameters. Tables 1 and 2 lack simple supervised aggregation or calibration controls with the same validation budget, as well as simple intervention averaging where applicable. These controls are important for attributing the gains.

Questions for authors

Question about identification versus preservation of target information.

CARE addresses confounded judge outputs through structural assumptions, CalibraEval calibrates prediction distributions to mitigate selection bias, and AtC introduces human-derived ranking constraints. Against these different sources of information, could you clarify precisely what VoirDire identifies beyond intervention sensitivity?

In particular, if the target and nuisance subspaces in Equation (4) overlap, accurately estimating the nuisance matrix does not by itself establish that attenuation in Equation (2) preserves target information. What assumptions or diagnostics control this potential loss, beyond the matrix-estimation guarantee in Proposition 1? Would an independent human-comparison audit be appropriate for checking whether attenuation preserves quality ordering, separately from reducing sensitivity to the interventions? This would help clarify when intervention-based calibration provides sufficient information and when additional human anchoring remains necessary.

[1] CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

[2] CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judges

[3] AtC: Aggregate-then-Calibrate for Human-centered Assessment

6 comments

We thank the reviewer for recognizing the value of our intervention-based approach and empirical analyses. Below, we address the reviewer's concerns.

(W1) On evaluation protocol and potential contamination. First, we clarify that candidate intervention pools were constructed without validation labels or performance feedback. LLM assistants proposed candidate families based on dataset examples, while validation performance was used only for subsequent family selection and hyperparameter tuning. We have clarified this distinction in Appendix B.2.

Second, to eliminate potential label reuse across evaluation splits, we have updated the main experiments to use five-fold cross-validation, selecting intervention families and hyperparameters independently within each training fold, without accessing held-out evaluation labels. Under this revised protocol, VoirDire achieves the best mean performance on 8 of 12 datasets and ranks among the top two on 11 of 12. We retain the original shared-configuration protocol only for controlled ablations, where fixed configurations help isolate component effects.

We have clarified both protocols in Section 5.1 and Appendix B.2. The updated results are reproduced below.

Table 1. Continuous-score aggregation: mean ± SD of MAE across five evaluation folds (↓).

DatasetMVAVGISPPCACARE-SVDVoirDire
Yelp1.014 ± .0251.087 ± .0171.026 ± .0251.052 ± .018.921 ± .019.492 ± .015
SummEval.816 ± .053.751 ± .049.831 ± .050.735 ± .032.785 ± .069.707 ± .052
Summarize from Feedback1.434 ± .0271.453 ± .0481.586 ± .0341.764 ± .0561.399 ± .0471.350 ± .044
HelpSteer2.996 ± .047.983 ± .0441.011 ± .048.993 ± .044.996 ± .041.965 ± .055
ASSET31.153 ± 2.49533.663 ± 3.46031.815 ± 3.05728.860 ± 2.19229.094 ± 4.11529.663 ± 5.075
Review-5K2.581 ± .0632.483 ± .0702.585 ± .0632.101 ± .0711.610 ± .0681.550 ± .275

Table 2. Binary/preference aggregation: mean ± SD of accuracy across five evaluation folds (↑).

DatasetMVAVGISPPCACARE-SVDCARE-TensorSkillAgg.IsingVoirDire
Chatbot Arena.518 ± .016.552 ± .007.571 ± .017.544 ± .010.583 ± .009.566 ± .024.524 ± .008.534 ± .025.645 ± .014
RewardBench 2.578 ± .027.596 ± .021.570 ± .032.594 ± .008.571 ± .010.553 ± .023.571 ± .026.532 ± .008.667 ± .033
LIAR2.678 ± .008.665 ± .025.678 ± .008.662 ± .019.673 ± .014.660 ± .026.684 ± .010.678 ± .010.680 ± .012
HelpSteer3.490 ± .031.516 ± .025.554 ± .019.518 ± .029.595 ± .023.579 ± .025.480 ± .034.509 ± .032.607 ± .019
CivilComments.702 ± .015.715 ± .018.716 ± .011.713 ± .014.747 ± .010.739 ± .010.722 ± .009.723 ± .010.745 ± .011
MT-Bench (human).601 ± .025.613 ± .025.601 ± .025.600 ± .015.594 ± .053.542 ± .019.594 ± .018.513 ± .018.608 ± .042

The updated results support our main empirical findings under the revised evaluation protocol, with up to 46.6% MAE reduction on Yelp and 7.1 percentage-point accuracy gain on RewardBench 2.

(W2) On target preservation and score contraction. Our task-specific interventions use explicit rewrite instructions and programmatic preservation checks to minimize unintended changes (Appendix B.5), although these safeguards cannot guarantee target invariance.

To further assess intervention quality in the controlled-bias experiments of Section 5.5, we manually reviewed the original–modified answers from Chen et al. [1] for content preservation. In the authority condition, the rewrites introduce citation and source-attribution cues without removing or reordering any original words across all 142 pairs (ignoring punctuation and capitalization). Among the beauty rewrites, 93 of 142 preserve the complete original word sequence, although some others contain rephrasing or expansions. These findings provide additional evidence of content preservation for the interventions used in the controlled-bias evaluation.

Normalized nuisance sensitivity. To address the concern that reduced score movement might merely reflect contraction, we additionally evaluate normalized nuisance sensitivity:

{\operatorname{SD}_i(s_i)},$$ where $s_i$ and $s_i'$ are aggregate scores for original and rewritten inputs. This metric is invariant to positive affine rescaling, so uniform score shrinkage alone cannot explain reductions in normalized sensitivity. We follow the original protocol with five folds, ten seeds, and 32 calibration pairs per fold. **Table. Normalized nuisance sensitivity (↓): mean ± SD across ten seeds.** | Benchmark | MV | AVG | ISP | PCA | CARE-SVD | VoirDire | |---|---|---|---|---|---|---| | Beauty | 1.029 ± .020 | 1.096 ± .076 | 1.101 ± .035 | 1.109 ± .193 | 1.113 ± .067 | **.891 ± .127** | | Authority | .873 ± .010 | .706 ± .059 | .894 ± .027 | .869 ± .286 | .529 ± .041 | **.451 ± .049** | VoirDire achieves the lowest normalized sensitivity on both benchmarks, improving over the strongest baseline by **13.4% on beauty** and **14.7% on authority**. These improvements cannot be explained solely by uniform score contraction (Appendix E.5). **Quality-discrimination control.** We further test whether VoirDire preserves target-relevant information while reducing nuisance sensitivity. We use **200 held-out preference pairs from the original RewardBench**, where the benchmark's preference labels provide the target quality signal. We prepend a fixed authority cue to the preferred answer, the dispreferred answer, or both, leaving the original answer content unchanged. **VoirDire maintains a minimum preference accuracy of 67.9% across the three perturbed conditions, compared with 51.0% for the strongest baseline (CARE-SVD)**. On the same pairs, it also achieves **lower normalized authority sensitivity (0.467 versus 0.518 for CARE-SVD)**. These results provide additional evidence that VoirDire reduces nuisance sensitivity while retaining quality-relevant distinctions (Appendix E.7).

(W3) On validation supervision and supervised aggregation baselines. Following CARE [2], our main experiments reserve 10% (continuous) or 15% (binary/preference) of each evaluation fold's training partition for validation. We additionally compare against supervised linear models that directly fit human labels, whereas VoirDire uses them for intervention selection and hyperparameter tuning. We find that supervised aggregation is highly competitive under these relatively substantial validation budgets, outperforming VoirDire on most datasets. Additionally we evaluate reduced budgets of 1% and 5%, reselecting intervention families and hyperparameters within each budget. VoirDire outperforms the supervised baselines on 10 of 12 datasets at 1%, compared with 5 of 12 at 5% and 3 of 12 under the original budgets. These results suggest that VoirDire remains effective with limited validation supervision.

Table. Performance across validation-label budgets: mean over five evaluation folds. Supervised denotes OLS regression for continuous tasks (MAE ↓) and logistic regression for binary/preference tasks (accuracy ↑). Original denotes the 10%/15% budgets.

DatasetVoirDire 1%VoirDire 5%VoirDire OriginalSupervised 1%Supervised 5%Supervised Original
Continuous: MAE ↓
Yelp0.5010.4910.4920.5100.4800.477
SummEval0.7760.7040.7071.2050.7400.725
Summarize1.3841.3541.3501.4131.3471.342
HelpSteer21.0000.9700.9653.2831.0530.956
ASSET30.65033.70429.66338.35632.98052.237
Review-5K1.6301.5711.5501.1230.9600.957
Binary/preference: Acc. ↑
Chatbot Arena0.6130.6340.6450.5340.6300.650
RewardBench 20.6230.6600.6670.6130.6740.676
LIAR20.6730.6760.6800.6310.6980.690
HelpSteer30.5310.6180.6070.5290.5660.602
CivilComments0.6990.7330.7450.7030.7510.762
MT-Bench0.5800.6100.6080.5610.6040.629
Reviewer_59nJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VoirDire, an intervention-calibrated method for aggregating LLM judges whose agreement may reflect shared sensitivity to superficial attributes rather than the intended evaluation target. The method estimates a second-moment matrix from paired judge scores on original and modified inputs, uses it to attenuate intervention-sensitive directions, and obtains aggregation weights through a generalized eigenvalue objective. The theoretical analysis demonstrates observational non-identifiability in a latent-factor model and bounds calibration errors under assumptions about target preservation, intervention quality, and sampling. Experiments across 12 datasets report improvements over the strongest uncalibrated aggregation baseline on 11 datasets, accompanied by analyses of calibration budgets, judge-panel size, targeted nuisance sensitivity, and transfer across related interventions.

Critical strengths
  1. The use of paired interventions addresses a meaningful limitation of judge aggregation. Multiple judges can agree because they share presentation biases, so stronger agreement alone does not establish more reliable evaluation. Using original–modified score differences provides an additional, interpretable source of information about judge sensitivity.
  2. The empirical evaluation covers diverse tasks and reports reasonably consistent gains. Tables 1–2 include six continuous-score datasets and six binary/preference datasets, with improvements over the strongest uncalibrated baseline on all six continuous tasks and five of the six binary/preference tasks. The larger gains, such as the Yelp MAE reduction from approximately 0.694 to 0.508 and the RewardBench 2 accuracy increase from 0.605 to 0.658, make the approach worth investigating further.
  3. The paper includes useful analyses of where the gains come from. Section 5.2 separates score attenuation from intervention-aware aggregation, while Appendices E.1–E.4 examine hyperparameter sensitivity, removal of individual intervention families, judge-panel size, and calibration applied to alternative aggregators.
Critical weaknesses
  1. The described selection protocol raises a serious concern about evaluation-label reuse across splits. Appendix B.2 states that both intervention families and a single configuration per dataset are selected using mean validation performance across five random row-level splits. Under this procedure, an example can inform selection through its validation label in one split and appear in another split’s evaluation set. Unless an additional untouched outer test partition exists, the reported evaluation is therefore not cleanly separated from selection. The open-ended construction of intervention candidates using validation feedback further increases this concern. A fixed outer holdout, with all intervention design and selection confined to the remaining data, is needed to substantiate the main performance claims.

  2. Preservation of the evaluation target is insufficiently validated. Appendix B.5 checks numbers, negation counts, word overlap, and length, but these properties do not establish that the correct score or preference remains unchanged. The accepted Yelp concision example illustrates the issue: “I love the outside patio and bar area” becomes “outside patio/bar area are nice,” potentially weakening sentiment relevant to the rating target. Formatting and wording changes can also legitimately affect coherence, fluency, or helpfulness. Moreover, Appendix B.8 explicitly uses formatting rewrites without preservation filtering. A human check is needed.

  3. The targeted-bias and transfer evaluations do not distinguish useful invariance from score contraction. Sections 5.5–5.6 measure absolute changes in aggregate scores while fixing maximum attenuation, α=1\alpha=1. Shrinking all scores reduces this metric, and a constant evaluator would achieve zero shift while providing no useful discrimination. Normalizing the aggregation vector does not necessarily preserve the scale of the final attenuated predictor. Consequently, Tables 3–4 establish lower raw sensitivity but do not establish that quality information is retained. We need variance-normalized sensitivity, a scale-matched shrinkage baseline, and human-alignment or genuine-quality-change controls on the same benchmark.

  4. The small-calibration-budget results omit the preceding intervention-search budget. Appendix B.7 states that family selection uses pools of 48 pairs per family, whereas the budget ablation holds those previously selected families fixed when evaluating as few as 4 or 8 pairs. Table 6 lists 12–53 candidate families per dataset, substantially more than the 2–6 ultimately selected. Thus, these curves characterize estimation after intervention discovery rather than the total budget required to obtain the reported system. Although Appendix B.2 acknowledges candidate-generation and scoring costs, the paper should quantify the complete search cost and distinguish it from the final fitting cost.

Questions for authors

See Weakness Above.

3 comments

We thank the reviewer for their thoughtful comments and constructive suggestions. We appreciate their recognition of the intervention-based approach and the breadth of our empirical evaluation. Below, we address the reviewer's concerns.

(W1) On validation-set usage and evaluation protocol. We have revised the main experiments to use five-fold cross-validation, with intervention families and hyperparameters selected independently within each fold, without using held-out evaluation labels. Under this revised protocol, VoirDire achieves the best mean performance on 8 of 12 datasets and ranks among the top two on 11 of 12, including comparisons with the newly added SkillAggregation and class-dependent Ising baselines for binary/preference tasks.

We retain the original shared-configuration protocol for controlled ablations, where fixed intervention families and hyperparameters allow us to isolate component effects. We have clarified the distinction between these protocols in Section 5.1 and Appendix B.2. The updated results are reproduced below.

Table 1. Continuous-score aggregation: mean ± SD of MAE across five evaluation folds (↓).

DatasetMVAVGISPPCACARE-SVDVoirDire
Yelp1.014 ± .0251.087 ± .0171.026 ± .0251.052 ± .018.921 ± .019.492 ± .015
SummEval.816 ± .053.751 ± .049.831 ± .050.735 ± .032.785 ± .069.707 ± .052
Summarize from Feedback1.434 ± .0271.453 ± .0481.586 ± .0341.764 ± .0561.399 ± .0471.350 ± .044
HelpSteer2.996 ± .047.983 ± .0441.011 ± .048.993 ± .044.996 ± .041.965 ± .055
ASSET31.153 ± 2.49533.663 ± 3.46031.815 ± 3.05728.860 ± 2.19229.094 ± 4.11529.663 ± 5.075
Review-5K2.581 ± .0632.483 ± .0702.585 ± .0632.101 ± .0711.610 ± .0681.550 ± .275

Table 2. Binary/preference aggregation: mean ± SD of accuracy across five evaluation folds (↑).

DatasetMVAVGISPPCACARE-SVDCARE-TensorSkillAgg.IsingVoirDire
Chatbot Arena.518 ± .016.552 ± .007.571 ± .017.544 ± .010.583 ± .009.566 ± .024.524 ± .008.534 ± .025.645 ± .014
RewardBench 2.578 ± .027.596 ± .021.570 ± .032.594 ± .008.571 ± .010.553 ± .023.571 ± .026.532 ± .008.667 ± .033
LIAR2.678 ± .008.665 ± .025.678 ± .008.662 ± .019.673 ± .014.660 ± .026.684 ± .010.678 ± .010.680 ± .012
HelpSteer3.490 ± .031.516 ± .025.554 ± .019.518 ± .029.595 ± .023.579 ± .025.480 ± .034.509 ± .032.607 ± .019
CivilComments.702 ± .015.715 ± .018.716 ± .011.713 ± .014.747 ± .010.739 ± .010.722 ± .009.723 ± .010.745 ± .011
MT-Bench (human).601 ± .025.613 ± .025.601 ± .025.600 ± .015.594 ± .053.542 ± .019.594 ± .018.513 ± .018.608 ± .042

The updated results provide further evidence of VoirDire's effectiveness under the revised evaluation protocol, with up to a 46.6% MAE reduction on Yelp and a 7.1-percentage-point accuracy gain on RewardBench 2.

(W2) On target preservation and intervention quality. We agree that our preservation checks cannot guarantee target invariance, and independent human validation would provide stronger evidence. Nevertheless, VoirDire improves aggregation across diverse tasks despite potentially imperfect interventions. Our theory shows that recovery can tolerate unintended changes that are small relative to the nuisance signal (Proposition 2, Appendix C.4), including with finite calibration samples (Theorem 2, Appendix C.5). Synthetic experiments (Appendix D.1) further illustrate how recovery degrades as intervention errors increase. We have made these assumptions and limitations more explicit in Appendix B.5 and Section 6.

We also manually inspected the Chen et al. [1] benchmark rewrites to assess preservation of the original answer content. The authority interventions add references and attribution cues while retaining every original word in order in all 142 cases (ignoring punctuation and capitalization). For beauty interventions, 93 of 142 rewrites similarly retain the original word sequence, while some others rephrase or expand the answers. These observations provide additional evidence of content preservation alongside our normalized-sensitivity results.

(W3) On nuisance robustness versus score contraction. To distinguish nuisance reduction from score contraction, we additionally evaluate normalized nuisance sensitivity:

{\operatorname{SD}_i(s_i)},$$ where $s_i$ and $s_i'$ are the aggregate scores of original and rewritten inputs, respectively. This metric is invariant to positive affine rescaling, so a scale-matched shrinkage baseline yields identical normalized sensitivity. We follow the original experimental protocol with five folds, ten seeds, and 32 calibration pairs per fold. **Table 3. Normalized nuisance sensitivity (↓): mean ± SD across ten seeds.** | Benchmark | MV | AVG | ISP | PCA | CARE-SVD | VoirDire | |---|---|---|---|---|---|---| | Beauty | 1.029 ± .020 | 1.096 ± .076 | 1.101 ± .035 | 1.109 ± .193 | 1.113 ± .067 | **.891 ± .127** | | Authority | .873 ± .010 | .706 ± .059 | .894 ± .027 | .869 ± .286 | .529 ± .041 | **.451 ± .049** | VoirDire achieves the lowest normalized sensitivity on both benchmarks, improving over the strongest baseline by **13.4% on beauty** and **14.7% on authority**. Thus, the gains cannot be explained solely by uniform score contraction. To further examine whether VoirDire preserves quality-relevant information while reducing nuisance sensitivity, we use 200 held-out preference pairs from the original RewardBench, where the benchmark's preference labels provide the target quality signal. We introduce a fixed authority cue to the preferred answer, the dispreferred answer, or both, leaving the original answer text unchanged. We evaluate **normalized authority sensitivity and preference accuracy on the same pairs** to assess whether reduced nuisance sensitivity preserves quality discrimination. Across the three perturbed conditions, **VoirDire maintains a minimum preference accuracy of 67.9%, compared with 51.0% for the strongest baseline (CARE-SVD)**. It also achieves **the lowest normalized authority sensitivity (0.467 versus 0.518 for CARE-SVD)**. These results demonstrate that VoirDire reduces nuisance sensitivity while retaining discrimination between preferred and dispreferred answers (Appendix E.7). **(W4) On intervention-search cost.** The small-budget ablation measures calibration efficiency **after intervention-family selection**, rather than the full discovery budget. As clarified in Appendix B.2, our experiments evaluate 12–53 candidate families per dataset, selecting 2–6 for final calibration (Table 6). We collected 64 pairs per candidate family for selection and budget analyses, while final calibration uses 32 pairs per selected family. For $C$ candidates, $k$ selected families, and $p$ judges, these correspond to $64Cp$ candidate-scoring and $32kp$ final-fitting judge scores, respectively. **These costs overlap**, since final calibration reuses candidate-pool scores. Importantly, **intervention discovery is an offline cost amortized across subsequent evaluations**. Once the calibration matrix and aggregation weights are fixed, evaluating new items requires only their original judge outputs, without additional rewrites or modified-input judge calls. We have clarified this cost breakdown further in Appendix B.2. **References** [1] Chen et al. (2024). *Humans or LLMs as the Judge? A Study on Judgement Bias.* EMNLP 2024.

Dear Reviewer 59nJ, Thank you again for your time and effort in reviewing our paper. We believe our rebuttal addresses your main concerns through several new experiments and analyses:

  • Evaluation protocol: We adopt five-fold cross-validation with independent intervention selection and hyperparameter tuning within each fold. VoirDire achieves the best performance on 8 of 12 datasets and ranks among the top two on 11 of 12 (updated Tables 1–2), showing consistent empirical gains as before.
  • Target preservation: We manually inspect rewrites used in the controlled-bias experiments (Section 5.5), finding that the original answer content is largely preserved.
  • Nuisance robustness versus score contraction: We introduce scale-invariant sensitivity measurements, showing consistent improvements over the strongest baselines. A new quality-discrimination control further demonstrates that VoirDire preserves preference accuracy under nuisance interventions (Appendices E.5 and E.7).
  • Intervention-search cost: We clarify the full candidate-selection and calibration costs, including their overlap and amortization across subsequent evaluations (Appendix B.2).

As the author–reviewer discussion period comes to a close, we would be happy to address any remaining concerns or questions. We appreciate your thoughtful feedback and engagement!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common Response

We thank all reviewers for their thoughtful comments and constructive suggestions. Before addressing individual reviews, we briefly highlight the strengths recognized by the reviewers:

  • A compelling intervention-based approach to shared judge biases. Paired interventions provide information about nuisance sensitivities beyond observational agreement (59nJ, SHZX), with a well-motivated spectral attenuation mechanism (gHDt).
  • Broad empirical evaluation across diverse tasks. The paper evaluates VoirDire against multiple aggregation baselines on 12 datasets spanning continuous scoring, binary classification, and pairwise preference (59nJ, SHZX, gHDt).
  • Extensive analyses of the method's behavior. Component ablations, hyperparameter sensitivity, calibration-budget experiments, and judge-panel analyses help characterize how VoirDire works and where its gains arise (59nJ, SHZX, gHDt).

Below, we address three common points and summarize additional analyses conducted during the rebuttal. We have updated the manuscript accordingly, with revisions highlighted in blue.

On evaluation protocol and validation-set usage (Reviewers gHDt, SHZX, 59nJ)

Our original protocol selected one configuration per dataset using mean validation performance across five random splits, allowing controlled ablations without confounding changes in intervention families or hyperparameters.

To address concerns about validation-set reuse, we have updated the main experiments to use five-fold cross-validation, selecting intervention families and hyperparameters independently within each fold without using evaluation-fold labels. Tables 1 and 2 now report mean ± standard deviation across folds. We also expanded the binary/preference comparisons to include SkillAggregation (Sun et al., 2025) and class-dependent Ising (Balasubramanian et al., 2026), addressing requests for stronger baselines. Both methods are designed for discrete-label aggregation and are not directly applicable to continuous-score tasks. We show the updated results below.

Continuous-score aggregation: mean ± SD of MAE across five evaluation folds (↓).

DatasetMVAVGISPPCACARE-SVDVoirDire
Yelp1.014 ± .0251.087 ± .0171.026 ± .0251.052 ± .018.921 ± .019.492 ± .015
SummEval.816 ± .053.751 ± .049.831 ± .050.735 ± .032.785 ± .069.707 ± .052
Summarize from Feedback1.434 ± .0271.453 ± .0481.586 ± .0341.764 ± .0561.399 ± .0471.350 ± .044
HelpSteer2.996 ± .047.983 ± .0441.011 ± .048.993 ± .044.996 ± .041.965 ± .055
ASSET31.153 ± 2.49533.663 ± 3.46031.815 ± 3.05728.860 ± 2.19229.094 ± 4.11529.663 ± 5.075
Review-5K2.581 ± .0632.483 ± .0702.585 ± .0632.101 ± .0711.610 ± .0681.550 ± .275

Binary/preference aggregation: mean ± SD of accuracy across five evaluation folds (↑).

DatasetMVAVGISPPCACARE-SVDCARE-TensorSkillAgg.IsingVoirDire
Chatbot Arena.518 ± .016.552 ± .007.571 ± .017.544 ± .010.583 ± .009.566 ± .024.524 ± .008.534 ± .025.645 ± .014
RewardBench 2.578 ± .027.596 ± .021.570 ± .032.594 ± .008.571 ± .010.553 ± .023.571 ± .026.532 ± .008.667 ± .033
LIAR2.678 ± .008.665 ± .025.678 ± .008.662 ± .019.673 ± .014.660 ± .026.684 ± .010.678 ± .010.680 ± .012
HelpSteer3.490 ± .031.516 ± .025.554 ± .019.518 ± .029.595 ± .023.579 ± .025.480 ± .034.509 ± .032.607 ± .019
CivilComments.702 ± .015.715 ± .018.716 ± .011.713 ± .014.747 ± .010.739 ± .010.722 ± .009.723 ± .010.745 ± .011
MT-Bench (human).601 ± .025.613 ± .025.601 ± .025.600 ± .015.594 ± .053.542 ± .019.594 ± .018.513 ± .018.608 ± .042

The updated results show that VoirDire outperforms the strongest of the existing aggregation baselines on 8 of 12 datasets, with up to 46.6% MAE reduction on Yelp and 7.1 percentage-point accuracy gain on RewardBench 2 (6.2 pp on Chatbot Arena). These results support our main empirical findings under the revised evaluation protocol.

On nuisance choice, target preservation, and identification (Reviewers gHDt, SHZX, 59nJ)

Nuisance attributes are features that may correlate with target quality and influence judge scores without causally affecting the target itself. We use LLM assistants as a practical means of proposing candidate interventions, rather than as an authoritative source of nuisance identification. Domain experts can also propose interventions directly. Our new ablation shows that six different LLM proposers retain gains over the strongest aggregation baselines across datasets (Appendix E.6), suggesting that the results are not specific to a particular proposer. Although preservation checks cannot guarantee target invariance, VoirDire improves aggregation even with potentially imperfect interventions.

Our theoretical analysis also clarifies the underlying conditions. Under target invariance, paired interventions identify the nuisance-sensitive directions revealed by the interventions (Appendix C.3), while our bounds characterize recovery under imperfect interventions and finite samples (Appendices C.4–C.5). We also analyze target-information preservation and potential loss during attenuation (Appendix C.6). These results characterize when intervention-based calibration can remain effective despite imperfect interventions. We have clarified the assumptions and limitations in Appendix B.5 and Section 6.

On nuisance sensitivity beyond score contraction (Reviewers 59nJ, SHZX)

We thank the reviewers for pointing out that reduced score shifts may simply result from uniformly shrinking aggregate scores. To address this concern, we studied a scale-invariant sensitivity metric, normalizing the mean absolute score shift by the standard deviation of original aggregate scores. This metric is unchanged by uniform rescaling, allowing us to distinguish reduced nuisance sensitivity from a mere change in score scale.

Normalized nuisance sensitivity (↓): mean ± SD across ten seeds.

BenchmarkMVAVGISPPCACARE-SVDVoirDire
Beauty1.029 ± .0201.096 ± .0761.101 ± .0351.109 ± .1931.113 ± .067.891 ± .127
Authority.873 ± .010.706 ± .059.894 ± .027.869 ± .286.529 ± .041.451 ± .049

VoirDire achieves the lowest normalized sensitivity on both benchmarks, improving over the strongest baseline by 13.4% on beauty and 14.7% on authority. These results demonstrate that VoirDire's reduced sensitivity to nuisance changes cannot be explained solely by uniform score contraction. We have added this analysis to Appendix E.5.

We further examine whether VoirDire preserves target-relevant information while reducing nuisance sensitivity. We use 200 held-out preference pairs from the original RewardBench, where the benchmark's preferred and dispreferred answers provide the target quality signal. We prepend a fixed authority cue to the preferred answer, the dispreferred answer, or both, leaving the original answer content unchanged. We then evaluate both preference accuracy and normalized authority sensitivity on the same pairs.

Across the three authority-perturbed conditions, VoirDire maintains a minimum preference accuracy of 67.9%, compared with 51.0% for the strongest baseline (CARE-SVD). It also achieves lower normalized authority sensitivity (0.467 versus 0.518 for CARE-SVD). These results provide evidence that VoirDire reduces nuisance sensitivity while retaining quality-relevant distinctions (Appendix E.7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.