acceptodds
Under review as a conference paper at ICLR 2027

Judge2Vec: Selecting Compact LLM Juries from Unlabeled Judgments

Abstract

LLMs are increasingly used as judges because human evaluation does not scale, yet relying on a single judge can be unreliable and querying many judges is costly. We introduce Judge2Vec, an unsupervised framework that selects a compact jury from a large, heterogeneous pool using only the judges' multi-criterion ratings on unlabeled pilot examples. Rather than ranking judges by a single reliability score, Judge2Vec represents each judge with a criterion-specific competence embedding estimated through leave-one-family-out agreement with a family-balanced cross-family consensus, then composes a jury whose strengths cover different criteria while avoiding redundant judges. Remarkably, selecting fewer judges can improve both quality and efficiency: across eight text benchmarks spanning qualitative and quantitative evaluation, Judge2Vec achieves the highest performance among all compared methods, while four selected judges outperform full-pool aggregation and reduce average inference cost by 83.9%. Our approach also extends to multimodal and agentic evaluation. Beyond a fixed jury, the learned competence representations support budget-adaptive deployment and transfer across tasks, enabling jury construction for unseen tasks without any target-task examples or judge evaluations. These results show that LLM evaluation need not choose between a single unreliable judge and an expensive large jury: a small jury, intelligently composed from unlabeled judgments, can be both more reliable and more efficient.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-as-a-judge, LLM juries, criterion-aware competence embeddings, unsupervised jury selection, multi-criterion evaluation, cost-aware deployment, zero-shot cross-task transfer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5vJ8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an unsupervised, criterion-aware method for selecting LLM judges, achieving better evaluation quality at lower inference cost.

Critical strengths

Paper is well organized and good writing. Easy to understand the main claim, and the core method.

Critical weaknesses
  1. Fig.1 should be larger. Also, it contains too much information, making it difficult to follow. The current four-panel layout does not clearly convey the sequence of steps. I suggest simplifying the content and redesigning the figure as a more intuitive, sequential pipeline.
  2. Here I have a question: is the comparison with current methods appropriate? Actually these baselines are Jury aggregation, but the posted method is Jury Selection. I think a more direct comparison is required.
  3. I don’t like to challenge ‘novelty’, however, Jury selection is already a well defined problem, and current method is more like a combination of existing techniques, e.g cross-family agreement, competence embeddings and greedy selection. Thus, authors should explain more on why this combinations, to make the core contribution stands out. The current version independently explain why the current method is better than a naive method, but doesn't explain the whole method as integrity. Authors should turn the intuitive design into more insightful instructions.
Questions for authors

Can authors compare with some more direct baselines (selection rather than aggregation)?

4 comments
AuthorsOpenReview ↗

On Q1/W2: Direct jury-selection baselines

We thank the reviewer for the constructive and detailed review.

Q1/W2: Direct selection baselines. We agree that jury-selection methods are the most direct comparison, and we respectfully note that the submission already includes them: Table 2 in the submission has a block of five panel-selection baselines, and Judge2Vec outperforms all of them on all eight datasets (first block below, unchanged). We now also add representative jury-selection baselines built on the Jury Selection Problem (JSP; Cao et al., 2012; Zheng et al., 2015), the canonical formulation of jury selection in crowdsourcing: given each judge's reliability, JSP selects the jury whose vote is most likely to be correct, which for a fixed jury size is the set of the most reliable judges. Since reliability is unknown without ground-truth labels, we run JSP with each of five representative label-free reliability estimators: Dawid–Skene (Dawid and Skene, 1979), MACE (Hovy et al., 2013), GLAD (Whitehill et al., 2009), a spectral estimator (Parisi et al., 2014), and consensus agreement (each judge's agreement with the consensus of the other judges). We also add Best-per-family, which selects the most competent judge from each of the six model families according to our label-free competence estimates. Besides Random-3 and Random-5, already in the submission, we also add Random-4. We do not compare with LLM Jury-on-Demand (Li et al., 2025), which the other reviewers raised, because it is supervised: it trains a reliability predictor for each judge on human expert scores and then keeps the top-ranked judges for each instance, whereas Judge2Vec and all baselines above use no ground-truth labels. The results are below. The best result in each column is in bold and the second best is underlined; Avg. rank is the mean rank over the eight datasets among all 14 methods (lower is better).

HANNASummEvalHelpSteer2FEDMATHExpertQAHAGRIDAgentCompanyAvg. rank
Judge2Vec.352†.380†.576‾\underline{.576}.561†.860.201†.226.336‾\underline{.336}1.3
In the submission (Table 2): jury selection
Largest-per-family.286.270.561.540.836.163.199.3275.8‾\underline{5.8}
Largest-3.251.223.552.511.491.129.199.31212.5
Largest-5.260.238.561.515.710.149.209.30910.3
Random-3.297.270.553.536.775.150.189.3268.1
Random-5.303‾\underline{.303}.274.560.539.793.156.199.3286.0
New: JSP with label-free reliability
JSP + Dawid–Skene.295.239.528.546.796.169.202.3227.6
JSP + MACE.255.319‾\underline{.319}.581.530.841.126.152.3377.4
JSP + GLAD.259.240.575.537.858‾\underline{.858}.113.093.3229.1
JSP + spectral.270.237.545.548‾\underline{.548}.846.148.218.3148.3
JSP + consensus agreement.299.244.548.538.841.182‾\underline{.182}.214.3176.7
JSP, average of the five.276.256.555.540.836.148.176.3238.3
New: other jury selection methods
Best-per-family.284.268.554.532.813.153.223‾\underline{.223}.3227.8
Random-4.294.281.566.542.818.162.180.3235.9

Judge2Vec is significantly better than each JSP variant on at least six of the eight datasets: on all eight against Dawid–Skene, on seven against spectral and consensus agreement, and on six against MACE and GLAD (paired bootstrap, p<0.05p < 0.05; † marks a significant gain over the strongest baseline on that dataset). It ranks first on six datasets and second on the other two, for an average rank of 1.3 versus 5.8 or worse for all 13 baselines. Even where it ranks second, it is never significantly worse: on HelpSteer2 and AgentCompany, its difference from JSP + MACE is not significant (p=0.36p = 0.36 and 0.800.80). No single reliability estimator is consistently strong: JSP + MACE, the only baseline that ranks above Judge2Vec on any dataset (two of the eight), is the second-weakest of all 13 baselines on HANNA, ExpertQA, and HAGRID, whereas Judge2Vec is first or second on every dataset.

AuthorsOpenReview ↗

On W3: Why this combination

We thank the reviewer for the constructive and detailed review.

W3: Why this combination. We agree that the method should be explained as one design. Jury selection is indeed well defined, but its well-established solution, JSP, rests on three assumptions that LLM juries violate, and each component of Judge2Vec replaces exactly one of them:

  • JSP needs each judge's reliability, i.e., ground-truth labels. Judge2Vec estimates reliability without labels, from agreement with the other model families. Leaving out the judge's own family is essential: otherwise a bias shared within the family is mistaken for accuracy.
  • JSP summarizes each judge by one overall reliability score. An LLM judge can be reliable on one criterion and weak on another, so Judge2Vec keeps a competence vector over the task's criteria.
  • JSP assumes that judges make mistakes independently, so the most reliable judges also form the best jury. LLM judges from the same family share biases and tend to make the same mistakes, so two strong judges from one family add less to a jury than a strong judge plus one with a complementary competence profile. Judge2Vec therefore builds the jury as a whole: it adds one judge at a time, favoring the judge that most improves the jury's weakest criterion and penalizing judges whose profile duplicates one already selected.

Classical jury selection therefore cannot be applied to LLM juries as is; Judge2Vec is a new, label-free formulation built around exactly these gaps.

The components work jointly. We verify this with an ablation study that changes one component at a time and keeps everything else fixed (same pilot, jury size, and aggregation): (i) we start from JSP, which gives each judge a single reliability score (here our label-free agreement, computed over all items and criteria at once) and selects the top-ranked judges; (ii) we replace this single score with per-criterion competence, but still select the top-ranked judges, now ranked by their average competence over the criteria; (iii) we replace ranking with complementary selection, which gives the full Judge2Vec.

HANNASummEvalHelpSteer2FEDMATHExpertQAHAGRIDAgentCompany
(i) Single reliability score + ranking (= JSP).297.249‾\underline{.249}.573‾\underline{.573}.560‾\underline{.560}.847‾\underline{.847}.149.214‾\underline{.214}.308
(ii) Per-criterion competence + ranking.304‾\underline{.304}.245.537.536.845.182‾\underline{.182}.214‾\underline{.214}.314‾\underline{.314}
(iii) Per-criterion competence + complementary selection (= Judge2Vec).352.380.576.561.860.201.226.336

Per-criterion competence alone does not help consistently (better on three datasets, worse on four), because ranking judges by their average competence discards which criteria each judge is good at. Adding complementary selection, which uses exactly this information, improves all eight datasets. The gain therefore comes from the combination, not from either component alone. This gives three insightful design rules: estimate reliability against other families, keep it per criterion, and compose the jury for coverage instead of ranking judges one by one.

AuthorsOpenReview ↗

On W1, and references

W1: Figure 1. We agree, and we are redesigning Figure 1 as the reviewer suggests: one left-to-right pipeline of four numbered steps joined by arrows (rate an unlabeled pilot, embed each judge's per-criterion competence, compose a complementary jury, and deploy only the jury), with larger text throughout. We will upload the redesigned figure to the supplementary material.

We thank the reviewer again for the specific and actionable suggestions: the request for direct jury-selection baselines led us to add JSP with five label-free reliability estimators, Best-per-family, and Random-4; the question of why the components are combined led to the analysis and ablation showing that they work jointly; and the feedback on Figure 1 will make the method much easier to follow. We hope our responses address the reviewer's concerns, and we would be grateful if the reviewer would consider raising the score.

References

  • Caleb Chen Cao, Jieying She, Yongxin Tong, and Lei Chen. 2012. Whom to ask? jury selection for decision making tasks on micro-blog services. Proc. VLDB Endow. 5, 11 (July 2012), 1495–1506. https://doi.org/10.14778/2350229.2350264
  • Zheng, Yudian, et al. "On optimality of jury selection in crowdsourcing." Proceedings of the 18th International Conference on Extending Database Technology, EDBT 2015. OpenProceedings. org., 2015.
  • Dawid, Alexander Philip, and Allan M. Skene. "Maximum likelihood estimation of observer error‐rates using the EM algorithm." Journal of the Royal Statistical Society: Series C (Applied Statistics) 28.1 (1979): 20-28.
  • Hovy, Dirk, et al. "Learning whom to trust with MACE." Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: Human language technologies. 2013.
  • Whitehill, Jacob, et al. "Whose vote should count more: Optimal integration of labels from labelers of unknown expertise." Advances in neural information processing systems 22 (2009).
  • Parisi, Fabio, et al. "Ranking and combining multiple predictors without labeled data." Proceedings of the National Academy of Sciences 111.4 (2014): 1253-1258.
  • Li, Xiaochuan, et al. "Who judges the judge? llm jury-on-demand: Building trustworthy llm evaluation systems." arXiv preprint arXiv:2512.01786 (2025).
AuthorsOpenReview ↗

Follow-up on Our Responses

Dear Reviewer 5vJ8,

Thank you again for your time and constructive comments. As the discussion period will end soon, we would like to check whether our responses have addressed your concerns. We would be happy to clarify anything further. If our responses have resolved your concerns, we would be grateful if you would consider raising your score.

Best regards,

The Authors

Reviewer_iAGaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Judge2Vec, a framework for selecting compact LLM juries without human annotations. From multi-criterion ratings on unlabeled pilot examples, it estimates criterion-specific competence through agreement with a family-balanced consensus that excludes each judge's own family. It then selects judges with complementary strengths and limited redundancy. At deployment, only selected judges are queried, and ratings are aggregated by criterion. The paper also explores budget adaptation and cross-task transfer, with experiments assessing quality and efficiency across text, multimodal, and agent evaluation.

Critical strengths
  1. The problem is practically relevant. Selecting compact juries without human annotations addresses reliability and deployment cost concerns in LLM-based evaluation.
  2. The framework accounts for multidimensional differences in judge competence. Constructing complementary juries from criterion-specific competence profiles offers a more nuanced approach than selecting models based on a single overall score.
  3. The empirical evaluation covers a broad range of settings. The experiments span text, multimodal, and agent evaluation, while also exploring budget adaptation and cross-task transfer, demonstrating the framework's applicability across different scenarios.
Critical weaknesses
  1. Methodological novelty is limited. Prior work has studied criterion-specific judge reliability estimation (BT-σ-asp) and reliability-based judge selection (Jury-on-Demand). Judge2Vec combines cross-family consensus with complementary selection without human labels, offering some novelty. However, the approach primarily integrates existing ideas, and the methodological advance remains limited.
  2. Insufficient family-bias controls obscure the independent benefit of complementary selection. Largest-per-family selects the largest model from each family, coupling family diversity with model size and obscuring the respective benefits of family balancing and complementary selection. Useful controls include family-balanced full-pool aggregation and repeated random four-judge panels from distinct families, the latter also matching jury size. Otherwise, gains may mainly reflect family balancing or redundancy reduction. This gap also affects the multimodal, cost, and transfer experiments.
  3. Multimodal gains over a simple baseline are modest. In Figure 2a, Judge2Vec achieves accuracy close to Largest-3, which selects the three largest models, with only a slightly lower total parameter count. Although it lies on the observed Pareto frontier, these modest differences in quality and parameter requirements provide limited evidence for the additional value of the more complex selection mechanism.
  4. Missing targeted ablations leave the added value of multidimensional complementary selection unclear. The paper does not compare two direct variants: selecting the four judges with the highest mean competence across criteria, and retaining weakest-criterion coverage while removing the redundancy penalty. With competence estimation, jury size, and aggregation held fixed, these controls would clarify whether complementary selection outperforms overall-competence ranking and whether the redundancy penalty adds value.
Questions for authors
  1. Do Random-3/5, Largest-3/5, and Largest-per-family use Judge2Vec's trim-one mean, discarding one highest and one lowest rating before averaging? If not, please compare under a common aggregation rule to separate gains from judge selection and aggregation.
  2. Please address the Critical Weaknesses above.
6 comments
AuthorsOpenReview ↗

On Q1, W4, and W2: Where the gains come from

We thank the reviewer for the careful and constructive review, and for recognizing the practical relevance of the problem, the value of criterion-specific competence profiles, and the breadth of our evaluation. We ran every experiment the reviewer suggested, on the same splits as in the submission; all results below are new and will be added to the paper.

Q1: Common aggregation rule. They do not: in Table 2 of the submission, these panel baselines average the ratings of their judges, as PoLL does. We re-ran each of them with Judge2Vec's trim-one mean (new, Table Q1 below). Under this common rule, Judge2Vec remains the best method on every dataset, so the gains come from which judges are selected, not from the aggregation rule.

Table Q1. Panel baselines under their original rule (Table 2 in the submission) and under Judge2Vec's trim-one mean (new); best in bold, second best underlined. On MATH, every method uses majority vote, so the two rules coincide.

HANNASummEvalHelpSteer2FEDMATHExpertQAHAGRIDAgentCompany
Judge2Vec.352.380.576.561.860.201.226.336
Largest-per-family, average (Table 2).286.270.561.540.836‾\underline{.836}.163.199.327
Largest-per-family, trim-one (new).293.282‾\underline{.282}.567‾\underline{.567}.545‾\underline{.545}.836‾\underline{.836}.181‾\underline{.181}.177.321
Largest-3, average (Table 2).251.223.552.511.491.129.199.312
Largest-3, trim-one (new).256.230.552.505.491.134.157.259
Largest-5, average (Table 2).260.238.561.515.710.149.209‾\underline{.209}.309
Largest-5, trim-one (new).263.237.565.522.710.168.206.305
Random-3, average (Table 2).297.270.553.536.775.150.189.326
Random-3, trim-one (new).293.276.561.534.775.157.164.320
Random-5, average (Table 2).303‾\underline{.303}.274.560.539.793.156.199.328‾\underline{.328}
Random-5, trim-one (new).296.282‾\underline{.282}.567‾\underline{.567}.544.793.165.190.325

W4 and W2: Where the gains come from. Table W below reports the four suggested experiments (two ablations and two family controls), including results on MathVista (multimodal).

Table W. Suggested ablations (W4) and family controls (W2). Ablations and random juries (300 draws per split) match Judge2Vec's jury size and aggregation; the family-balanced pool uses all judges, with equal weight per family. † Significant gain over both ablations (paired bootstrap, p < 0.05).

HANNASummEvalHelpSteer2FEDMATHExpertQAHAGRIDAgentCompanyMathVista
Judge2Vec.352†.380†.576†.561†.860†.201†.226.336†.863†
Suggested ablations (same competence estimation, jury size, aggregation)
Highest mean competence across criteria.304.245.537.536.845.182.214.314.831
No redundancy penalty.304.245.537.535.837.182.214.314.831
Family controls
Family-balanced full pool (all judges).297.275.562.531.793.167.212.326.857
Random juries from distinct families (same size).289.284.569.541.815.159.179.318.847
  • Judge2Vec's complementary selection outperforms overall-competence ranking, and the redundancy penalty adds value (W4). Both ablations are below Judge2Vec on every dataset, significantly on seven of the eight text datasets and on MathVista (†). Neither has the redundancy penalty, and that is what they miss: strong judges may share the same strengths and weaknesses, and the penalty, defined on criterion-level profiles, steers each new member away from profiles the jury already has, so members complement rather than repeat each other.
  • Neither family balancing nor redundancy reduction alone explains the gains (W2). Family-balanced full-pool aggregation is below Judge2Vec on every dataset, significantly on seven of the eight text datasets. Random juries from distinct families, which separate family diversity from model size and are the simplest form of redundancy reduction, are also below Judge2Vec on every dataset: they avoid judges from the same family but not judges with the same strengths, which is what Judge2Vec's criterion-level selection avoids.

Tables Q1 and W thus rule out each alternative explanation: the gains come from Judge2Vec's complementary selection itself.

AuthorsOpenReview ↗

On W2 (continued): multimodal, cost, and transfer experiments

W2 (continued): multimodal, cost, and transfer experiments. Judge2Vec's advantages in these experiments still hold after adding the suggested family controls. Multimodal: both controls are below Judge2Vec on MathVista (Table W above). Cost: Judge2Vec is also cheaper than both; the family-balanced pool queries all 19 judges (28.97 US dollars per 1,000 items, Table 3 in the submission), and random four-judge juries from distinct families cost 6.42 on average, versus 4.67 for Judge2Vec. Transfer: both controls need no pilot, as Judge2Vec's zero-shot transfer does, and zero-shot Judge2Vec (Table 4 in the submission) beats both on all target tasks, including random distinct-family juries of its own size.

AuthorsOpenReview ↗

On W1: Novelty Compared with BT-σ-asp and Jury-on-Demand

W1: Novelty. We thank the reviewer for acknowledging the novelty of combining cross-family agreement with complementary selection without human labels. We respectfully clarify that the two works mentioned by the reviewer address different problems, and neither does what Judge2Vec does: select, without any labels, a compact jury that is the only one queried at deployment.

  • BT-σ-asp (Qian et al., 2026) is an aggregation model for comparative assessment: every judge compares pairs of responses to the same input, and judge reliabilities per aspect weight these comparisons. All judges are queried on every item, and no jury is selected. It also has two limitations that Judge2Vec avoids. (i) It needs several responses to each input, so it cannot run on datasets where each question has a single response; Judge2Vec works in a more generalizable manner and runs on all nine datasets. (ii) Its soft version takes preference probabilities from token logits, which, in the authors' words, "restricts the jury to open-weight models": closed-source API judges expose no logits. With black-box judges, only the hard version can run. Judge2Vec, in contrast, needs only the ratings that judges return through standard API calls, so it applies directly to any black-box judge, including the strongest closed-source models. Where BT-σ is reported in our Table 2 of the submission, Judge2Vec is far above it: .352 vs. .144 on HANNA and .380 vs. .242 on SummEval.
  • Jury-on-Demand (Li et al., 2025) is supervised: it trains reliability predictors on human expert scores for each evaluation criterion, and, for each instance, keeps the K judges with the highest predicted reliability, ranking each judge individually. Judge2Vec needs no such annotations: a small unlabeled pilot suffices, and with zero-shot transfer it builds a jury for an unseen task without any target-task examples, even unlabeled ones (Section 4.2, Table 4 of the submission), so it applies where expert labels are costly or unavailable. In addition, Judge2Vec selects judges as a team rather than one by one: since judges that are each reliable can share the same weak criteria, it chooses judges whose strengths cover each other's weaknesses.

Judge2Vec's contributions are:

  • Fully unsupervised. Judge2Vec estimates each judge's competence on each criterion from its rank agreement with other model families on a small unlabeled pilot. It uses no human or ground-truth labels for jury selection, and it needs only the judges' ratings, so any black-box API judge can be used.
  • Composition, not ranking. Judge2Vec selects judges that cover each other's weak criteria and penalizes redundant competence profiles. This goes beyond combining existing ideas: keeping the judges with the highest reliability, as Jury-on-Demand does but with our label-free competence, is the first ablation in Table W above. Both this ablation and Judge2Vec without the redundancy penalty are below Judge2Vec on every dataset, significantly on seven of the eight text datasets and on MathVista.
  • Compact juries that adapt and transfer. Only the selected jury is queried, which cuts cost by 83.9% relative to all 19 judges (Table 3 of the submission), while Judge2Vec is the best method on all eight text datasets (Table 2 of the submission). The same competence embeddings adapt the jury to different budgets without additional judge evaluations (on HelpSteer2, about 15-fold lower cost for a drop of only 0.003 in Kendall τ; Section 4.1, Figure 2b of the submission). They also transfer zero-shot to unseen tasks without any target-task examples, even unlabeled ones: zero-shot Judge2Vec beats PoLL on every target task and retains 98.5% of the in-domain performance on average (Section 4.2, Table 4 of the submission).
Reviewer_HsQcOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses a practical problems: when using multiple LLM judges, judge jury, how to come up with a way to select a compact jury, using few LLM-as-judges.

The paper proposed a simple and unsupervised method. Given a label-free validation set, all judges are used to score on multiple dimensions. Then the model-family based consistency/relevance scores are calculated for each evaluation dimension. Last, the paper proposed a greedy search based method to select jury members such that dimensions with less consistency are covered more while dimensions with high consistency are covered less.

Critical strengths
  • To compare with full Judge jury, the proposed method can reduce 80% of the cost, and even have small performance improvement. Given the validation set is label-free, this is very positive.

  • Good judges and tasks coverage.

  • The paper also shows that the proposed jury selection method can be applied to multimodal LLM-as-judge and also budge-aware scenario.

Critical weaknesses
  • The proposed method is a bit too heuristic, for example, the dimensions need to be predefined and same between validation and target set. The dimension is also more coarse-grained than actual example to evaluate.

  • A critical assumption in the unsupervised validation set is that the paper assumes consistency leads to high accuracy (majority vote), but they are not equivalent and sometimes not highly correlated.

  • The overall novelty is a bit limited. Jury selection is not completely new, e.g. LLM Jury-on-Demand and else. Further to this, having more jury member can always give marginal improvement, but linear cost. Therefore jury selection method reduces cost while maintaining performance is somewhat expected.

Questions for authors

N/a

6 comments
AuthorsOpenReview ↗

On W3 (a): Novelty over existing jury selection

We thank the reviewer for the constructive and detailed review.

W3: Novelty, and whether the result is expected. The reviewer makes three points: (a) jury selection already exists, (b) adding jury members always gives a small improvement, and (c) a method that cuts cost while keeping performance is therefore expected. We answer each in turn.

(a) Jury selection already exists. Existing methods differ from Judge2Vec in two ways. First, they need labels or known judge quality: LLM Jury-on-Demand (Li et al., 2025) trains models on human expert scores to predict which judges to trust for each example, and classical jury selection (JSP; Cao et al., 2012; Zheng et al., 2015) assumes that the quality of each judge is known. Second, they rank judges and keep the top ones. Judge2Vec needs no ground-truth labels: it measures how good each judge is on each criterion from unlabeled examples, and it builds a jury whose members cover each other's weak criteria, instead of keeping the top-ranked judges. Table R1 below adds representative jury selection baselines: JSP with five ways of estimating judge quality without labels (Dawid–Skene, MACE, GLAD, a spectral method, and agreement with the other judges), Best-per-family, and Random-4, alongside the selection baselines already included in the submission. The last column counts the datasets on which each method beats all 19 judges, both when their ratings are averaged (PoLL, as in the submission) and when they are combined with the trim-one mean used by Judge2Vec, which drops the highest and the lowest rating and averages the rest.

Table R1. Jury selection comparison (best in bold, second best underlined; † significant gain over the strongest baseline, paired bootstrap, p < 0.05; Avg. rank: average rank over the eight datasets, lower is better).

HANNASummEvalHelpSteer2FEDMATHExpertQAHAGRIDAgentCompanyAvg. rankBeats all 19
Judge2Vec.352†.380†.576‾\underline{.576}.561†.860.201†.226.336‾\underline{.336}1.38/8
All 19 judges, average (PoLL).310‾\underline{.310}.278.568.540.804.171.213.3305.3–
All 19 judges, trim-one mean.309.283.570.542.804.176.211.3314.6‾\underline{4.6}–
Selection baselines in the submission
Largest-per-family.286.270.561.540.836.163.199.3277.41/8
Largest-3.251.223.552.511.491.129.199.31214.50/8
Largest-5.260.238.561.515.710.149.209.30912.30/8
Random-3.297.270.553.536.775.150.189.32610.10/8
Random-5.303.274.560.539.793.156.199.3288.00/8
JSP, judge quality estimated without labels (new)
JSP + Dawid–Skene.295.239.528.546.796.169.202.3229.41/8
JSP + MACE.255.319‾\underline{.319}.581.530.841.126.152.3378.44/8
JSP + GLAD.259.240.575.537.858‾\underline{.858}.113.093.32210.62/8
JSP + spectral.270.237.545.548‾\underline{.548}.846.148.218.3149.63/8
JSP + consensus agreement.299.244.548.538.841.182‾\underline{.182}.214.3177.93/8
JSP, average of the five.276.256.555.540.836.148.176.3239.91/8
Other selection methods (new)
Best-per-family.284.268.554.532.813.153.223‾\underline{.223}.3229.32/8
Random-4.294.281.566.542.818.162.180.3237.41/8

Judge2Vec is best on six datasets and second on the other two; and on those two where Judge2Vec is ranked the second, its difference from JSP + MACE is not statistically significant (p = 0.36 and 0.80). It is also the only method that beats all 19 judges on every dataset. Its average rank over the eight datasets is 1.3 among the 16 rows of Table R1 (lower is better), far ahead of all 19 judges with the trim-one mean (4.6) and of every other selection method (7.4 or worse).

AuthorsOpenReview ↗

On W3 (b) and (c): Adding jury members, and whether the result is expected

(b) Does adding jury members always help? No. First, Judge2Vec picks its jury from the same 19 judges, and adding the remaining judges back to the Judge2Vec jury lowers accuracy on all eight datasets (Table R1). Second, this is not specific to Judge2Vec. In an oracle experiment that uses the test labels (new), we rank the 19 judges by their accuracy on the test set and form a jury from the m most accurate judges, for m = 1 to 19. Table R2 reports the size of the most accurate jury and how much accuracy drops when all remaining judges are added.

Table R2. Oracle that ranks the judges with the test labels.

HANNASummEvalHelpSteer2FEDMATHExpertQAHAGRIDAgentCompany
Most accurate jury size26841444
Accuracy lost by adding the remaining judges.107.104.014.049.067.035.026.040

On every dataset, the most accurate jury has at most eight judges, and in every split, adding the remaining judges lowers accuracy. Extra judges mostly add weak or repeated votes: many judges are weak on some criteria, and judges from the same family tend to make similar mistakes.

(c) Is the result expected? If adding jury members always helped, a smaller jury could at best match the full jury at a lower cost. Instead, Judge2Vec is more accurate than all 19 judges on all eight datasets, and it costs 83.9% less. It also beats every method that combines all 19 judges in Table 2 of the submission (PoLL, RoPoLL, CARE, BT-σ, Dawid–Skene, MACE, and GLAD), on every dataset where they apply. Using fewer judges alone is not enough: every other selection method in Table R1 falls below both versions of the full jury on at least half of the datasets. The gain comes from the new way Judge2Vec builds the jury (Table R3). JSP (row i) gives each judge one overall score and keeps the top-ranked judges. Judge2Vec changes both steps: it scores each judge separately on each criterion, and it uses these scores to build a jury whose members cover each other's weak criteria. Both changes are needed. Keeping the judges with the highest average score across criteria (row ii) is not consistently better than JSP, because averaging turns the per-criterion scores back into one number. With the same jury size and the same per-criterion scores, building a complementary jury (row iii, the full Judge2Vec) is more accurate than rows (i) and (ii) on all eight datasets, and it is what lets a small jury beat all 19 judges on every dataset (Table R1). Thus, beating all 19 judges with a small jury is not an expected result of using fewer judges; it rests on the key insight of Judge2Vec that judges are strong on different criteria, so a strong jury is one whose members complement each other, not one made of the judges that are best on average.

Table R3. Where the gain comes from: changing one part at a time (same pilot, jury size, and aggregation).

HANNASummEvalHelpSteer2FEDMATHExpertQAHAGRIDAgentCompany
(i) One overall score per judge + keep the top judges (= JSP).297.249‾\underline{.249}.573‾\underline{.573}.560‾\underline{.560}.847‾\underline{.847}.149.214‾\underline{.214}.308
(ii) One score per criterion + keep the top judges.304‾\underline{.304}.245.537.536.845.182‾\underline{.182}.214‾\underline{.214}.314‾\underline{.314}
(iii) One score per criterion + build a complementary jury (= Judge2Vec).352.380.576.561.860.201.226.336
AuthorsOpenReview ↗

On W1 (a): Predefined criteria

W1: Predefined criteria and granularity.

We thank the reviewer and answer the two points in turn.

(a) “The dimensions need to be predefined and same between validation and target set.” We respectfully clarify that Judge2Vec does not require the criteria to be the same, and the submission devotes a full section to exactly this case. In Section 4.2 (Zero-Shot Cross-Task Jury Transfer), Judge2Vec builds a jury for a new task whose criteria differ from those of the previous tasks, without any examples from the new task, not even unlabeled ones. It maps the judge competence measured on the previous tasks onto a shared set of criteria, keeps the structure these tasks have in common via SVD decomposition, and projects this shared structure onto the new task's own criteria; the same selection algorithm then builds the jury. Section 5.4 evaluates this by holding out each task in turn. As Table 4 of the submission (reproduced below) shows, the zero-shot jury is more accurate than PoLL with all 19 judges on every target task, is even more accurate than the jury built from the target task's own pilot on HANNA, and keeps 98.5% of that jury's accuracy on average (Kendall τ .460 vs. .467). In the standard setting, the criteria are simply the task's own rubric (e.g., coherence, consistency, fluency, and relevance in SummEval), which any LLM-as-judge evaluation must give to its judges anyway, and the pilot (the reviewer's validation set) is a small unlabeled sample of the same task. Judge2Vec therefore needs nothing beyond what the evaluation task already defines. The submission thus covers both cases: without any pilot, zero-shot transfer already beats PoLL on every target task, and when a small unlabeled pilot is available, Judge2Vec uses it to raise the accuracy further.

Table 4 of the submission (unchanged). Zero-shot cross-task transfer (Kendall τ). The in-domain reference denotes Judge2Vec constructed from the target task's own pilot.

MethodHANNASummEvalHelpSteer2FED
PoLL (all 19).310±.009.278±.003.568±.000.540±.009
Zero-shot transfer (no target examples, even unlabeled ones)
Judge2Vec.358±.015.356±.014‾\underline{.356±.014}.571±.006‾\underline{.571±.006}.556±.017‾\underline{.556±.017}
In-domain reference: uses target pilot examples
Target pilot.352±.020‾\underline{.352±.020}.380±.015.576±.003.561±.022

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.