- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the relationship between the intrinsic judgment quality of LLM-as-a-Judge systems and their downstream utility. It examines how verdict granularity, critique usage, and evaluation batching affect judgment quality and RL training performance. The paper further explores judge-guided test-time inference strategies, including Best-of-N, judge-guided revision, and beam search.
The authors report that intrinsic judgment quality does not always align with downstream utility, that judge protocols influence training outcomes, and that judge guidance can improve performance by leveraging additional inference-time compute.
Critical strengths
The research question is practically relevant, and the experimental coverage is reasonably broad.
Jointly examining intrinsic judgment quality and downstream utility is a worthwhile research direction. The paper evaluates multiple judges, policies, and open-ended tasks, while considering both training-time and test-time applications. These experiments provide some potentially useful empirical observations for practitioners.
Critical weaknesses
1. The central scientific question remains insufficiently answered, limiting the paper's contribution.
The paper aims to investigate the relationship between judgment quality and downstream utility, yet most experiments primarily compare the performance of different judge protocols. There is little direct analysis of how these two quantities relate, or under what conditions and why they diverge.
The authors also use reductions in Reward Tie Rate to help explain training improvements. However, tie rate only measures whether rewards are distinguishable; it does not establish whether their rankings are correct. A low tie rate could even result from random or otherwise uninformative scores.
Furthermore, Best-of-N, revision, and beam search are well-established techniques. The corresponding test-time experiments lack compute-matched comparisons and have a relatively weak connection to the paper's central motivation. Overall, the paper does not yet provide sufficiently coherent and generalizable new insights.
2. Key conclusions are weakened by incomparable metrics and insufficient experimental controls.
In Section 5.2, the comparison between T/F and Rating relies on different reference targets and accuracy definitions: binary classification accuracy for T/F versus distance-based soft accuracy for Rating. The distance-based formulation of the stability metrics may also inherently favor Rating. Consequently, these results cannot be directly interpreted as evidence that one protocol is superior.
In Section 6.1, the authors attribute the advantage of Verdict+Critique to weaker judges being anchored by incorrect critiques, but this mechanism is not directly validated. Importantly, judge capability and task difficulty may interact substantially. Easy tasks may be straightforward even for relatively weak judges, regardless of output order, whereas difficult tasks may require extensive reasoning.
Furthermore, for reasoning-capable models, all three protocols may involve internal reasoning before the final output. Thus, the visible order of Verdict and Critique does not necessarily reflect the actual reasoning order. Without systematic evaluations across judge strengths, task difficulties, and thinking configurations, the generalizability of the Critique Ordering conclusions remains questionable.
Finally, several mean differences in Table 2 are substantially smaller than the corresponding reported standard deviations, while adequate statistical significance testing and effect-size analysis are lacking.
Questions for authors
-
Connection to the motivation: The paper emphasizes a mismatch between Judgment Quality and Downstream Utility, yet most experiments compare verdict formats, critique protocols, and batching. How do these experiments directly address the central research question?
-
Fair comparison of T/F and Rating: T/F and Rating use different accuracy definitions and reference labels, and their stability metrics are not directly comparable. Why is this comparison valid? Can the conclusions be re-evaluated using comparable metrics?
-
Validity of Reward Tie Rate: How does a lower tie rate imply a more useful reward signal? If all rewards are distinct but their ranking is completely wrong, the tie rate is still low. Can the authors directly relate tie rate to reward ranking quality or downstream training gains?
-
Effect of Thinking Mode: Was Thinking enabled for the judges in Section 6.1? All three protocols may involve internal reasoning regardless of the visible output order. Please provide Thinking On/Off comparisons to verify whether the Critique Ordering conclusions still hold.
-
Judge capability and task difficulty: Easy tasks may be trivial even for weaker judges, while difficult tasks may require more reasoning. Can the authors evaluate stronger judges and stratify tasks by difficulty to examine how these factors interact with Critique Ordering?
-
Statistical reliability: Some differences in Table 2 are smaller than the reported standard deviations. How many random seeds were used, and are these differences statistically reliable? Does the claimed 12/16 advantage of Verdict+Critique remain robust?
-
Contribution of the test-time experiments: Best-of-N, revision, and beam search are established methods that use additional compute. Can the authors provide compute-matched baselines and clarify what is new in these experiments, as well as how they relate to the paper's central motivation?