- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates reinforcement learning with semi-verifiable rubric-based rewards for post-training LLMs. Specifically, the authors explore whether inexpensive imperfect LLMs used as training-time verifiers can provide sufficiently strong training signals compared to expensive frontier models. To this end, they benchmark twelve training-time verifiers across three domains and evaluate their grading agreement against three golden verifier frontier LLMs.
The paper presents two main findings. First, while verifier agreement with a golden judge is informative, it does not consistently predict or rank downstream post-training policy performance. Second, inexpensive verifiers can indeed provide highly effective training signals, yielding post-training scores close to those of the best-performing far more expensive verifiers. The authors retrospectively propose two low-cost choices that reduce grading costs by an estimated ninety-eight percent to ninety-nine percent relative to the golden grading protocols, with average post-training score gaps of only one to three points from the best evaluated verifiers.
Critical strengths
-
The paper is well written, organized, and easy to follow.
-
The paper presents an exceptionally comprehensive evaluation of training time verifiers across three professional domains and multiple trainee model sizes.
-
The evaluation design is robust because it separates the training time verifiers from the evaluation verifiers to avoid evaluation bias. Furthermore, the authors reduce single judge bias by averaging performance across three distinct reference frontier models from different vendors.
-
The paper provides practical and actionable conclusions that demonstrate how utilizing inexpensive training time verifiers can significantly reduce post training reinforcement learning costs while maintaining strong performance.
Critical weaknesses
The study does not include a comparison where a frontier model is used directly as the training time verifier. This is the most crucial benchmark because without training a policy using the most powerful models as the verifier, it is impossible to establish the actual performance ceiling or to definitively prove how much performance is sacrificed by choosing a cheaper verifier. Although the authors acknowledge this as a limitation, the lack of this baseline comparison weakens the core claim regarding the adequacy of cheaper verifiers.
While the paper presents a comprehensive evaluation with practical and useful cost analysis, the methodological novelty is somewhat limited. The paper primarily conducts an empirical study using standard training frameworks and existing benchmarks without introducing new algorithms, architectures, or theoretical frameworks. Because the finding that smaller models can act as decent judges is already widely suspected in the community (and even if not), the intellectual contribution may feel incremental and slightly below the bar expected for a top tier conference.
Questions for authors
N/A
2 comments
I thank the authors for their clarifications. The finding presented in the paper is interesting and worth exploring, yet it is not worth acceptance on its own. I recommend further investing and expanding the work to provide additional insights or methods. Therefore, I maintain my score.
This is correct for the golden judges, and Section 7 lists it as a limitation; the reason is grading cost (Table 2: $133.20 against $0.45 per 1,000 graded responses). The closest comparisons in the pool are Gemini 3 flash preview, which has the highest capability score (GDPval) of the twelve training verifiers, and Grok 4.20, the most expensive (Table 7). With Qwen3 8B on HealthBench they give 0.32/0.44 and 0.32/0.46 (hard/all), against 0.34/0.45 for Gemma 4 26B (Tables 9 and 10). The paper does not claim equivalence with a golden-trained policy (Section 4).
The paper is empirical, but its finding is not that small models are decent judges. It is that judge quality as usually measured does not tell you which judge to train with: verifiers with the same F1 of 0.76 give 0.34 and 0.08 (Table 9), and a verifier with higher agreement can train a worse policy (GPT oss 20B against Grok 4.20: 0.37 against 0.46 on HealthBench-all, Section 3.1). This comes from actual post-training runs with twelve verifiers, scored by three reference judges, with grading cost measured alongside.