acceptodds
Under review as a conference paper at ICLR 2027

From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks

Abstract

LLM-as-a-Judge is increasingly used to evaluate policy responses on open-ended tasks that lack ground-truth answers. Existing work often directly converts the resulting judgments into reward signals for policy training, paying limited attention to intrinsic judgment quality and largely restricting the use of Judges to training-time supervision. We systematically investigate judgment quality and downstream utility by examining both how judgments are elicited and how they are used. For judgment elicitation, we vary the Judge protocol along three dimensions: verdict granularity, critique usage, and evaluation batching. For judgment usage, beyond policy training, we extend Judge to test-time inference through Best-of-N selection, Judge-guided revision, and beam search. We find that, (i) Surprisingly, judgment quality and downstream utility do not always align. (ii) Judge protocol design substantially affects both intrinsic judgment quality and downstream utility. (iii) Judge guidance effectively converts test-time compute into performance gains, with benefits varying across inference strategies. Our results call for a multifaceted evaluation of LLM Judges on open-ended tasks, encompassing intrinsic judgment quality, and downstream utility.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Open-ended task, Large Language Model, LLM-as-a-Judge, Rubric, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KgBiOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the relationship between the intrinsic judgment quality of LLM-as-a-Judge systems and their downstream utility. It examines how verdict granularity, critique usage, and evaluation batching affect judgment quality and RL training performance. The paper further explores judge-guided test-time inference strategies, including Best-of-N, judge-guided revision, and beam search.

The authors report that intrinsic judgment quality does not always align with downstream utility, that judge protocols influence training outcomes, and that judge guidance can improve performance by leveraging additional inference-time compute.

Critical strengths

The research question is practically relevant, and the experimental coverage is reasonably broad.

Jointly examining intrinsic judgment quality and downstream utility is a worthwhile research direction. The paper evaluates multiple judges, policies, and open-ended tasks, while considering both training-time and test-time applications. These experiments provide some potentially useful empirical observations for practitioners.

Critical weaknesses

1. The central scientific question remains insufficiently answered, limiting the paper's contribution.

The paper aims to investigate the relationship between judgment quality and downstream utility, yet most experiments primarily compare the performance of different judge protocols. There is little direct analysis of how these two quantities relate, or under what conditions and why they diverge.

The authors also use reductions in Reward Tie Rate to help explain training improvements. However, tie rate only measures whether rewards are distinguishable; it does not establish whether their rankings are correct. A low tie rate could even result from random or otherwise uninformative scores.

Furthermore, Best-of-N, revision, and beam search are well-established techniques. The corresponding test-time experiments lack compute-matched comparisons and have a relatively weak connection to the paper's central motivation. Overall, the paper does not yet provide sufficiently coherent and generalizable new insights.

2. Key conclusions are weakened by incomparable metrics and insufficient experimental controls.

In Section 5.2, the comparison between T/F and Rating relies on different reference targets and accuracy definitions: binary classification accuracy for T/F versus distance-based soft accuracy for Rating. The distance-based formulation of the stability metrics may also inherently favor Rating. Consequently, these results cannot be directly interpreted as evidence that one protocol is superior.

In Section 6.1, the authors attribute the advantage of Verdict+Critique to weaker judges being anchored by incorrect critiques, but this mechanism is not directly validated. Importantly, judge capability and task difficulty may interact substantially. Easy tasks may be straightforward even for relatively weak judges, regardless of output order, whereas difficult tasks may require extensive reasoning.

Furthermore, for reasoning-capable models, all three protocols may involve internal reasoning before the final output. Thus, the visible order of Verdict and Critique does not necessarily reflect the actual reasoning order. Without systematic evaluations across judge strengths, task difficulties, and thinking configurations, the generalizability of the Critique Ordering conclusions remains questionable.

Finally, several mean differences in Table 2 are substantially smaller than the corresponding reported standard deviations, while adequate statistical significance testing and effect-size analysis are lacking.

Questions for authors
  1. Connection to the motivation: The paper emphasizes a mismatch between Judgment Quality and Downstream Utility, yet most experiments compare verdict formats, critique protocols, and batching. How do these experiments directly address the central research question?

  2. Fair comparison of T/F and Rating: T/F and Rating use different accuracy definitions and reference labels, and their stability metrics are not directly comparable. Why is this comparison valid? Can the conclusions be re-evaluated using comparable metrics?

  3. Validity of Reward Tie Rate: How does a lower tie rate imply a more useful reward signal? If all rewards are distinct but their ranking is completely wrong, the tie rate is still low. Can the authors directly relate tie rate to reward ranking quality or downstream training gains?

  4. Effect of Thinking Mode: Was Thinking enabled for the judges in Section 6.1? All three protocols may involve internal reasoning regardless of the visible output order. Please provide Thinking On/Off comparisons to verify whether the Critique Ordering conclusions still hold.

  5. Judge capability and task difficulty: Easy tasks may be trivial even for weaker judges, while difficult tasks may require more reasoning. Can the authors evaluate stronger judges and stratify tasks by difficulty to examine how these factors interact with Critique Ordering?

  6. Statistical reliability: Some differences in Table 2 are smaller than the reported standard deviations. How many random seeds were used, and are these differences statistically reliable? Does the claimed 12/16 advantage of Verdict+Critique remain robust?

  7. Contribution of the test-time experiments: Best-of-N, revision, and beam search are established methods that use additional compute. Can the authors provide compute-matched baselines and clarify what is new in these experiments, as well as how they relate to the paper's central motivation?

Reviewer_u6bcOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the influence of different LLM-as-a-Judge protocols for judgement quality and downstream utility, and further explores the applications of judge under test-time inference. The paper conducts experiments across multiple judge, policy models and tasks.

Critical strengths
  1. The research topic is meaningful. Investigating the relationship between a judge’s intrinsic quality and its downstream utility is a practically meaningful research question.

  2. This paper systematically compares verdict granularity, critique usage, and evaluation batching, and additionally analyzes accuracy, consistency, stability, and reward resolution.

  3. The studies both RL training and test-time inference, which covers different models and tasks.

Critical weaknesses
  1. The novelty is limited. One of the main claims is judgement quality and downstream utility may not be consistent. However, this is not a new finding, which is studied in RewardBench 2[1]. Additionally, RUBRIC-ARROW[2] shows the effectiveness of rubric-based reward models under BoN setting, and also studies reward ties and reward resolution. JETTS[3] also explores judge-guided reranking, revision, and beam search.

  2. The core claim lacks sufficient experiment evidence. The paper claims that judgment quality and downstream utility do not always align. However, Fig. 2 shows that rating does not always outperform T/F across different quality metrics, and we do not have a clear overall trend. Furthermore, Table 1 shows that Rating achieves higher average downstream performance in only 10 out of the 16 settings.

  3. The experiment results are not convincing. First, the paper chooses GPT-OSS-20B as training-time judge on partial RL experiments and it also uses the same family LLM GPT-OSS-120B to judge policy models. GPT-OSS-120B is related to both judgment reference design and critique–verdict consistency evaluation. It may result in evaluator bias. Furthermore, although the paper studies open-ended tasks, it can include instruction following benchmarks such as IFEval and IFBench, which provide programmatic verifiers for independently measuring downstream performance and reducing potential confounding factors introduced by LLM-based evaluation.

[1] Malik S, Pyatkin V, Land S, et al. Rewardbench 2: Advancing reward model evaluation[C]//International Conference on Learning Representations. 2026, 2026: 144839-144866.

[2] Jiang H, Dong Z, Liu T, et al. RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domains[J]. arXiv preprint arXiv:2605.29156, 2026.

[3] Zhou Y, Xu A, Wang P, et al. Evaluating judges as evaluators: The jetts benchmark of llm-as-judges as test-time scaling evaluators[J]. arXiv preprint arXiv:2504.15253, 2025.

Questions for authors

refer to weakness

Reviewer_ACu1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the relationship between intrinsic judgment quality and downstream utility for rubric-based LLM judges on open-ended medical and scientific tasks. It varies verdict granularity, critique/verdict ordering, and criterion batching, examining their effects on judgment accuracy, consistency, stability, reward resolution, and policy performance after GRPO training. It also evaluates judges as inference-time assistants through Best-of-N selection, judge-guided revision, and beam search. The results suggest that intrinsic judgment metrics and downstream performance do not always improve together, while the benefits of particular protocols and inference strategies depend on the judge, policy, and task.

Critical strengths
  1. The paper addresses a practically relevant evaluation question. Distinguishing intrinsic judgment quality from the usefulness of judgments for optimization is valuable, since these quantities capture different requirements of an LLM judge. The separation between judgment elicitation and judgment usage provides a clear framework for organizing the investigation (Sections 3–4).
  2. The training experiments cover a useful range of configurations. The main study includes two policy models, two training judges, two domains, and four evaluation datasets. HealthBench and ResearchQA are excluded from training, and downstream evaluation uses a stronger grader distinct from the training judges. These choices support examining how protocol effects vary across models and evaluation settings (Section 4.1; Appendix A.1; Tables 1–3).
  3. The reward-resolution diagnostics provide concrete insight into the training signals. Measuring response-level ties and entirely uniform response groups helps contextualize the downstream results. For example, with Qwen3-1.7B and the Qwen2.5-3B judge on science, batching increases the uniform-group rate from 2.6% to 33.8%, revealing a substantial reduction in groups that provide nonzero relative advantages (Appendix B.1; Table 6).
Critical weaknesses
  1. The granularity comparison appears to change critique order as well. Section 5 states that other protocol components are held fixed, but the default T/F template generates explanation before criteria_met, whereas the Rating template lists criteria_score before explanation (Appendices A.2.1–A.2.2). Since Section 6 identifies output order as consequential, this discrepancy raises a confound in attributing the observed differences to verdict granularity. The actual generation order needs clarification, followed by a comparison with matched ordering if the published templates reflect the implementation.
  2. The intrinsic metrics are not directly comparable across verdict formats. T/F uses majority-vote binary reference labels, while Rating uses the mean of numeric reference scores; Rating “accuracy” is then 1−MAE1-\mathrm{MAE}, whereas binary accuracy measures exact agreement. Repeatability and decision-rule robustness similarly give partial credit to numeric differences but zero credit to binary disagreements (Appendix B.2, Equations 5–13). These are reasonable diagnostics within each format, but their raw percentages do not establish a common ordering of judgment quality. This weakens the central interpretation of quality–utility divergence; evaluation against a shared reference target and comparable ranking or error metrics would be more convincing.
  3. The validity of the quality measurements remains dependent on model-based proxies. The three-model reference ensemble is a useful design choice, but Gpt-oss-120B also audits critique consistency and supplies the downstream evaluation scores (Sections 4.1–4.2; Appendix F.1). The paper does not report expert validation, reference-model disagreement, or sensitivity to an alternative final grader. Consequently, it remains unclear how robust the conclusions are to the choice of evaluators, particularly when interpreting model agreement as judgment correctness in medical and scientific tasks.
Questions for authors
  1. What was the actual output order in the Rating experiments? If it differed from the T/F baseline, do the main granularity findings persist when both protocols use the same critique/verdict order?
  2. Does the reported quality–utility mismatch persist when all protocols are evaluated on identical response–criterion pairs against a shared reference target and common metrics? How sensitive are these findings to the reference models or final grader?
  3. How many independent runs underlie Tables 1–3, how were checkpoints selected, and how was the static judgment dataset constructed? Which key protocol differences remain supported by paired uncertainty estimates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.