- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes an unsupervised, criterion-aware method for selecting LLM judges, achieving better evaluation quality at lower inference cost.
Critical strengths
Paper is well organized and good writing. Easy to understand the main claim, and the core method.
Critical weaknesses
- Fig.1 should be larger. Also, it contains too much information, making it difficult to follow. The current four-panel layout does not clearly convey the sequence of steps. I suggest simplifying the content and redesigning the figure as a more intuitive, sequential pipeline.
- Here I have a question: is the comparison with current methods appropriate? Actually these baselines are Jury aggregation, but the posted method is Jury Selection. I think a more direct comparison is required.
- I don’t like to challenge ‘novelty’, however, Jury selection is already a well defined problem, and current method is more like a combination of existing techniques, e.g cross-family agreement, competence embeddings and greedy selection. Thus, authors should explain more on why this combinations, to make the core contribution stands out. The current version independently explain why the current method is better than a naive method, but doesn't explain the whole method as integrity. Authors should turn the intuitive design into more insightful instructions.
Questions for authors
Can authors compare with some more direct baselines (selection rather than aggregation)?
4 comments
On W3: Why this combination
We thank the reviewer for the constructive and detailed review.
W3: Why this combination. We agree that the method should be explained as one design. Jury selection is indeed well defined, but its well-established solution, JSP, rests on three assumptions that LLM juries violate, and each component of Judge2Vec replaces exactly one of them:
- JSP needs each judge's reliability, i.e., ground-truth labels. Judge2Vec estimates reliability without labels, from agreement with the other model families. Leaving out the judge's own family is essential: otherwise a bias shared within the family is mistaken for accuracy.
- JSP summarizes each judge by one overall reliability score. An LLM judge can be reliable on one criterion and weak on another, so Judge2Vec keeps a competence vector over the task's criteria.
- JSP assumes that judges make mistakes independently, so the most reliable judges also form the best jury. LLM judges from the same family share biases and tend to make the same mistakes, so two strong judges from one family add less to a jury than a strong judge plus one with a complementary competence profile. Judge2Vec therefore builds the jury as a whole: it adds one judge at a time, favoring the judge that most improves the jury's weakest criterion and penalizing judges whose profile duplicates one already selected.
Classical jury selection therefore cannot be applied to LLM juries as is; Judge2Vec is a new, label-free formulation built around exactly these gaps.
The components work jointly. We verify this with an ablation study that changes one component at a time and keeps everything else fixed (same pilot, jury size, and aggregation): (i) we start from JSP, which gives each judge a single reliability score (here our label-free agreement, computed over all items and criteria at once) and selects the top-ranked judges; (ii) we replace this single score with per-criterion competence, but still select the top-ranked judges, now ranked by their average competence over the criteria; (iii) we replace ranking with complementary selection, which gives the full Judge2Vec.
| HANNA | SummEval | HelpSteer2 | FED | MATH | ExpertQA | HAGRID | AgentCompany | |
|---|---|---|---|---|---|---|---|---|
| (i) Single reliability score + ranking (= JSP) | .297 | .149 | .308 | |||||
| (ii) Per-criterion competence + ranking | .245 | .537 | .536 | .845 | ||||
| (iii) Per-criterion competence + complementary selection (= Judge2Vec) | .352 | .380 | .576 | .561 | .860 | .201 | .226 | .336 |
Per-criterion competence alone does not help consistently (better on three datasets, worse on four), because ranking judges by their average competence discards which criteria each judge is good at. Adding complementary selection, which uses exactly this information, improves all eight datasets. The gain therefore comes from the combination, not from either component alone. This gives three insightful design rules: estimate reliability against other families, keep it per criterion, and compose the jury for coverage instead of ranking judges one by one.
On W1, and references
W1: Figure 1. We agree, and we are redesigning Figure 1 as the reviewer suggests: one left-to-right pipeline of four numbered steps joined by arrows (rate an unlabeled pilot, embed each judge's per-criterion competence, compose a complementary jury, and deploy only the jury), with larger text throughout. We will upload the redesigned figure to the supplementary material.
We thank the reviewer again for the specific and actionable suggestions: the request for direct jury-selection baselines led us to add JSP with five label-free reliability estimators, Best-per-family, and Random-4; the question of why the components are combined led to the analysis and ablation showing that they work jointly; and the feedback on Figure 1 will make the method much easier to follow. We hope our responses address the reviewer's concerns, and we would be grateful if the reviewer would consider raising the score.
References
- Caleb Chen Cao, Jieying She, Yongxin Tong, and Lei Chen. 2012. Whom to ask? jury selection for decision making tasks on micro-blog services. Proc. VLDB Endow. 5, 11 (July 2012), 1495–1506. https://doi.org/10.14778/2350229.2350264
- Zheng, Yudian, et al. "On optimality of jury selection in crowdsourcing." Proceedings of the 18th International Conference on Extending Database Technology, EDBT 2015. OpenProceedings. org., 2015.
- Dawid, Alexander Philip, and Allan M. Skene. "Maximum likelihood estimation of observer error‐rates using the EM algorithm." Journal of the Royal Statistical Society: Series C (Applied Statistics) 28.1 (1979): 20-28.
- Hovy, Dirk, et al. "Learning whom to trust with MACE." Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: Human language technologies. 2013.
- Whitehill, Jacob, et al. "Whose vote should count more: Optimal integration of labels from labelers of unknown expertise." Advances in neural information processing systems 22 (2009).
- Parisi, Fabio, et al. "Ranking and combining multiple predictors without labeled data." Proceedings of the National Academy of Sciences 111.4 (2014): 1253-1258.
- Li, Xiaochuan, et al. "Who judges the judge? llm jury-on-demand: Building trustworthy llm evaluation systems." arXiv preprint arXiv:2512.01786 (2025).
Follow-up on Our Responses
Dear Reviewer 5vJ8,
Thank you again for your time and constructive comments. As the discussion period will end soon, we would like to check whether our responses have addressed your concerns. We would be happy to clarify anything further. If our responses have resolved your concerns, we would be grateful if you would consider raising your score.
Best regards,
The Authors
On Q1/W2: Direct jury-selection baselines
We thank the reviewer for the constructive and detailed review.
Q1/W2: Direct selection baselines. We agree that jury-selection methods are the most direct comparison, and we respectfully note that the submission already includes them: Table 2 in the submission has a block of five panel-selection baselines, and Judge2Vec outperforms all of them on all eight datasets (first block below, unchanged). We now also add representative jury-selection baselines built on the Jury Selection Problem (JSP; Cao et al., 2012; Zheng et al., 2015), the canonical formulation of jury selection in crowdsourcing: given each judge's reliability, JSP selects the jury whose vote is most likely to be correct, which for a fixed jury size is the set of the most reliable judges. Since reliability is unknown without ground-truth labels, we run JSP with each of five representative label-free reliability estimators: Dawid–Skene (Dawid and Skene, 1979), MACE (Hovy et al., 2013), GLAD (Whitehill et al., 2009), a spectral estimator (Parisi et al., 2014), and consensus agreement (each judge's agreement with the consensus of the other judges). We also add Best-per-family, which selects the most competent judge from each of the six model families according to our label-free competence estimates. Besides Random-3 and Random-5, already in the submission, we also add Random-4. We do not compare with LLM Jury-on-Demand (Li et al., 2025), which the other reviewers raised, because it is supervised: it trains a reliability predictor for each judge on human expert scores and then keeps the top-ranked judges for each instance, whereas Judge2Vec and all baselines above use no ground-truth labels. The results are below. The best result in each column is in bold and the second best is underlined; Avg. rank is the mean rank over the eight datasets among all 14 methods (lower is better).
Judge2Vec is significantly better than each JSP variant on at least six of the eight datasets: on all eight against Dawid–Skene, on seven against spectral and consensus agreement, and on six against MACE and GLAD (paired bootstrap, p<0.05; † marks a significant gain over the strongest baseline on that dataset). It ranks first on six datasets and second on the other two, for an average rank of 1.3 versus 5.8 or worse for all 13 baselines. Even where it ranks second, it is never significantly worse: on HelpSteer2 and AgentCompany, its difference from JSP + MACE is not significant (p=0.36 and 0.80). No single reliability estimator is consistently strong: JSP + MACE, the only baseline that ranks above Judge2Vec on any dataset (two of the eight), is the second-weakest of all 13 baselines on HANNA, ExpertQA, and HAGRID, whereas Judge2Vec is first or second on every dataset.