acceptodds
Under review as a conference paper at ICLR 2027

A Cheap Verifier Is Good Enough: LLM Post-training Is Robust To Erroneous Rewards

Abstract

When post-training large language models on tasks with semi-verifiable rewards, there are many factors (training steps, base model size, training order, data quality, verifier accuracy, etc.) that practitioners must contend with to maximize model performance. Yet, it remains unclear how well verifier agreement predicts post-training performance on such tasks. In this paper, we explore this question with over 11k H100 GPU-hours, across HealthBench and PRBench tasks in medical, legal, and finance domains. Across the tested domains, Qwen3 trainees (1.7B–8B on HealthBench; 8B on PRBench), evaluation splits, and frontier LLM reference judges (which we call golden verifiers), higher verifier agreement does not consistently identify the best training verifier. Expensive verifiers need not outperform inexpensive ones, and open-weight Gemma verifiers produce strong training outcomes. We compare two low-cost choices retrospectively — a cost-reducing choice and a balanced choice — with estimated grading cost reductions of – relative to the golden grading protocols and average post-training score gaps of – points from the best evaluated training verifier. These averages include larger losses in individual settings; they do not establish that verifier choices are interchangeable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-training, rubrics, rl with rubrics rewards, rlrr, Qwen, Healthbench, PRBench

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3mNHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates reinforcement learning with semi-verifiable rubric-based rewards for post-training LLMs. Specifically, the authors explore whether inexpensive imperfect LLMs used as training-time verifiers can provide sufficiently strong training signals compared to expensive frontier models. To this end, they benchmark twelve training-time verifiers across three domains and evaluate their grading agreement against three golden verifier frontier LLMs.

The paper presents two main findings. First, while verifier agreement with a golden judge is informative, it does not consistently predict or rank downstream post-training policy performance. Second, inexpensive verifiers can indeed provide highly effective training signals, yielding post-training scores close to those of the best-performing far more expensive verifiers. The authors retrospectively propose two low-cost choices that reduce grading costs by an estimated ninety-eight percent to ninety-nine percent relative to the golden grading protocols, with average post-training score gaps of only one to three points from the best evaluated verifiers.

Critical strengths
  • The paper is well written, organized, and easy to follow.

  • The paper presents an exceptionally comprehensive evaluation of training time verifiers across three professional domains and multiple trainee model sizes.

  • The evaluation design is robust because it separates the training time verifiers from the evaluation verifiers to avoid evaluation bias. Furthermore, the authors reduce single judge bias by averaging performance across three distinct reference frontier models from different vendors.

  • The paper provides practical and actionable conclusions that demonstrate how utilizing inexpensive training time verifiers can significantly reduce post training reinforcement learning costs while maintaining strong performance.

Critical weaknesses

The study does not include a comparison where a frontier model is used directly as the training time verifier. This is the most crucial benchmark because without training a policy using the most powerful models as the verifier, it is impossible to establish the actual performance ceiling or to definitively prove how much performance is sacrificed by choosing a cheaper verifier. Although the authors acknowledge this as a limitation, the lack of this baseline comparison weakens the core claim regarding the adequacy of cheaper verifiers.

While the paper presents a comprehensive evaluation with practical and useful cost analysis, the methodological novelty is somewhat limited. The paper primarily conducts an empirical study using standard training frameworks and existing benchmarks without introducing new algorithms, architectures, or theoretical frameworks. Because the finding that smaller models can act as decent judges is already widely suspected in the community (and even if not), the intellectual contribution may feel incremental and slightly below the bar expected for a top tier conference.

Questions for authors

N/A

2 comments

The study does not include a comparison where a frontier model is used directly as the training time verifier. This is the most crucial benchmark ...

This is correct for the golden judges, and Section 7 lists it as a limitation; the reason is grading cost (Table 2: $133.20 against $0.45 per 1,000 graded responses). The closest comparisons in the pool are Gemini 3 flash preview, which has the highest capability score (GDPval) of the twelve training verifiers, and Grok 4.20, the most expensive (Table 7). With Qwen3 8B on HealthBench they give 0.32/0.44 and 0.32/0.46 (hard/all), against 0.34/0.45 for Gemma 4 26B (Tables 9 and 10). The paper does not claim equivalence with a golden-trained policy (Section 4).

Methodological novelty is somewhat limited; the finding that smaller models can act as decent judges is already widely suspected.

The paper is empirical, but its finding is not that small models are decent judges. It is that judge quality as usually measured does not tell you which judge to train with: verifiers with the same F1 of 0.76 give 0.34 and 0.08 (Table 9), and a verifier with higher agreement can train a worse policy (GPT oss 20B against Grok 4.20: 0.37 against 0.46 on HealthBench-all, Section 3.1). This comes from actual post-training runs with twelve verifiers, scored by three reference judges, with grading cost measured alongside.

I thank the authors for their clarifications. The finding presented in the paper is interesting and worth exploring, yet it is not worth acceptance on its own. I recommend further investing and expanding the work to provide additional insights or methods. Therefore, I maintain my score.

Reviewer_mNqbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, Qwen3 models are trained with 12 different LLM graders on medical, finance, and legal benchmarks, then evaluates them on those benchmarks with three stronger reference judges. It finds that graders that agree more with reference judges do not always result in better models being trained from those graders, and that cheap graders like Gemma 4 26B score within 1–3 points of the best grader and cost much less.

Critical strengths
  • The paper compares 12 graders through actual training on multiple tasks with multiple trainee sizes and reference judges. Using real graders with different costs and error patterns reflects real-world training conditions and budget constraints more closely than synthetic-noise experiments, making the findings almost immediately transferrable/actionable

  • Gemma 4 26B achieves about 95% of the available improvement at much lower grading cost (Table 3), which is a useful for those training with a limited budget. The authors also include failed training runs and break down the costs (Table 2, App. M), making the trade-offs easier to assess.

Critical weaknesses
  • The main claim of the paper seems bolder than what the results actually support. Qwen3 8B has the same F1 as Gemma 4 26B but using it as a post-training verifier results in all three trainee sizes attaining accuracies below their pre-trained states (Table 9). On HealthBench-all, the cheapest grader improves the model’s score by less than half as much as the best grader (Table 3).

  • The paper measures how often graders agree on checklist items, but this does not show whether their disagreements change what the model learns. For example, a cheap grader might miss a few details but still give the best answer the highest score, so training would still favor that answer. Compare whether cheap and reference graders give higher scores to the same answers, using existing training samples. This would help distinguish disagreements that matter for training from those that do not.

Questions for authors

W1: What exactly do you quantify to be "good enough"?

W2: See comment

1 comment

The main claim seems bolder than the results support. Qwen3 8B has the same F1 as Gemma 4 26B but using it as a verifier results in all three trainee sizes below their pre-trained states (Table 9). On HealthBench-all, the cheapest grader improves the score by less than half as much as the best grader (Table 3).

Both observations are correct, and both are stated in the main text (Sections 3.1 and 4). The claim is scoped accordingly: the abstract says the averages "include larger losses in individual settings", and Section 6 limits the cost claim to "selected inexpensive verifiers". The Qwen3 8B case is our first finding: at the same F1 of 0.76, Gemma 4 26B trains the 8B trainee to 0.34 on HealthBench-hard and Qwen3 8B to 0.08 (Table 9). We accept that the title reads more broadly than this and propose to change it to "A Cheap Verifier Can Be Good Enough: Rubric-Based Post-Training with Imperfect Verifiers".

Agreement on checklist items does not show whether disagreements change what the model learns. Compare whether cheap and reference graders give higher scores to the same answers, using existing training samples.

We agree, and Section 3.2 makes the same point. The paper reports one score-level measure, the Spearman correlation between verifier and reference scores (Tables 9–14), and it does not rank training outcomes either: on HealthBench-all, GPT oss 20B has the highest Spearman (0.87) but trains the 8B trainee to 0.37, against 0.45 for Gemma 4 26B (Table 10). The within-prompt comparison is not possible from existing data, because the reference judges did not grade groups of training rollouts; we will state this as a limitation.

W1: What exactly do you quantify to be "good enough"?

The phrase appears only in the title; the quantities are defined in Section 4: the score gap to the best verifier with a completed run and a cost estimate, and the uplift (the share of the improvement over the untrained baseline that is retained). We will define "good enough" there as a mean score gap of at most 2 points. Gemma 4 26B meets this (+0.010, 95% uplift, 98.8% lower grading cost); the cheapest-above-0.80-accuracy rule (+0.027, 82% uplift) does not, and we will say so (Tables 3 and 19).

W2: See comment

Please see our response to the second weakness above.

Reviewer_DsLoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackled an important question of how to choose a verifier to scale the post training. The authors empirically showed that using a cheaper verifier, which does not always have strong agreement with the golden verifier, can also retain the same level of performance as high cost api models. The experiments were covered on Healthbench and PRbench, and a wide range of verifer models were used to do post training on Qwen family’s models. Also large computational resources were used to support the authors’ claim.

Critical strengths
  • Empirically showed that using models that do not have high agreement with the golden verifier can achieve good performance in the post training process

  • A wide range of models and benchmarks were covered, and computational resources were used to demonstrate generality.

Critical weaknesses
  • As mentioned in the limitations, the experiments only used Qwen3 models as trainees, and the post training algorithm was fixed to GRPO, so it is hard to see whether this phenomenon is reproducible across different seeds and transferable to other models.

  • The phenomenon is interesting, but the authors only chose models retrospectively and did not provide any principles or guidelines for choosing the weak verifer in the practical pipeline, or explain the more fundamental mechanism behind the phenomenon, such as why a good policy can be learned using a cheap verifier with low agreement.

Questions for authors
  • As stated in the weaknesses, it is really interesting to know that sometimes a weaker verifier can achieve good performance at a low cost, but if I need to choose a cheap verifier restropectively, why shouldn’t I use the strong or golden verifier in the first place?

  • The empirifcal finding is meaningful, but as Figure 2 shows, Gemma / Gemini seem to achieve the overall better performance than the other models. Can I say that I can use Gemma / Gemini in the first place, if not, how can we know which model to use as a verifer for which dataset?

  • How robust are the findings across different model families, given that verifier effectiveness may depend on the trainee's architecture and training dynamics?

  • Though it might be expensive, could you show a comparison with a gold verifier?

  • Could you tell me how you calculated the score based on which dataset was used in Figure 3 ?

  • For Figure 2 and Figure 3, I know Gemma and Gemini are represented by blue boxes from the setup, but can you distinguish between different models in the figures, even if they are in the same family?

  • What is the point of using 1-F as horizontal axis on Figure 3? I would be more comfortable if it were be the same as in Figure 2.

  • When you list references, could you include the publication venues? Even arXiv citations would be helpful to identify the papers.

2 comments

The experiments only used Qwen3 models as trainees, and the post training algorithm was fixed to GRPO, so it is hard to see whether this phenomenon is reproducible across different seeds and transferable to other models.

This is a limitation and Section 7 states it: one trainee family (Qwen3 1.7B, 4B, 8B), one GRPO setup, and no controlled seed replications. The one configuration we launched twice (Qwen3 8B with GPT oss 20B on HealthBench, with different verifier-serving backends) differs by 1.4 points on the hard split and 2.5 on the all split, so we do not rank verifiers whose scores differ by this little. The conclusions rest on patterns that recur across three trainee sizes, three domains and three reference judges, and on large effects, such as two verifiers with the same F1 of 0.76 giving 0.34 and 0.08 on HealthBench-hard (Table 9).

The authors only chose models retrospectively and did not provide any principles or guidelines for choosing the weak verifier in the practical pipeline, or explain the more fundamental mechanism behind the phenomenon.

Section 4 and Appendix N give a selection rule that needs no training run: take the cheapest candidate with accuracy ≥ 0.80 against the reference judge on a 20–40 task screening slice. Table 19 compares it with 19 other rules (+0.027 score gap, 82% uplift, 99.7% lower grading cost); as the paper says, the rules are assessed on the same runs and not validated on new candidate pools. On the mechanism, Section 3.2 gives a hypothesis (GRPO standardizes rewards within each rollout group, which criterion-level F1 does not capture); we have not tested it.

If I need to choose a cheap verifier retrospectively, why shouldn't I use the strong or golden verifier in the first place?

Cost. Grading 1,000 responses under the golden protocol costs $133.20 with Claude Opus 4.6 and $99.70 with GPT-5.5, against $0.45 with Gemma 4 26B (Table 2), and one 100-step run grades about 96k rollouts (Appendix M). The choice need not be retrospective: the cost-reducing rule uses only screening agreement and price, both known before training.

Gemma / Gemini seem to achieve the overall better performance. Can I say that I can use Gemma / Gemini in the first place, if not, how can we know which model to use as a verifier for which dataset?

On HealthBench, yes: Gemma 4 26B and Gemma 4 31B give the best Qwen3 8B score on the hard split (0.34), and on the all split Gemma 4 26B (0.45) is second only to Grok 4.20 (0.46), with Gemini 3 flash preview at 0.44 (Tables 9 and 10). For a new dataset, screening candidates against a reference judge on a small slice flags very weak verifiers but not every failure (the Qwen3 8B verifier, Table 9); we do not claim the ordering transfers to other trainees or domains.

How robust are the findings across different model families, given that verifier effectiveness may depend on the trainee's architecture and training dynamics?

We have no results beyond Qwen3 (Section 7). Within Qwen3 the dependence on the trainee is visible: with LFM2 24B as verifier, the 8B trainee improves on HealthBench-hard (0.21 against a 0.16 baseline) while the 1.7B trainee falls below its baseline (0.03 against 0.08), whereas Gemma 4 26B improves all three sizes (Table 9).

Could you show a comparison with a gold verifier?

We did not train with a golden verifier, because of grading cost (see the answer above). The closest comparisons are Gemini 3 flash preview, which has the highest capability score (GDPval) of the twelve training verifiers, and Grok 4.20, the most expensive (Table 7). With Qwen3 8B on HealthBench they give 0.32/0.44 and 0.32/0.46 (hard/all), against 0.34/0.45 for Gemma 4 26B (Tables 9 and 10).

How did you calculate the score in Figure 3, and based on which dataset?

Figure 3 uses HealthBench, the one dataset on which we vary trainee size (Section 2.3); its points are the 8B, 4B and 1.7B columns of Tables 9 (hard) and 10 (all). The score is the rubric score of Eq. (2) at rollout step 99 on held-out tasks, averaged over the three golden verifiers. We will name the dataset in the caption.

Figures 2 and 3: can you distinguish between different models within the same family?

Not in the current figures: marker shape and colour encode only the provider family, and LFM2 24B is the only labelled point. The per-verifier values are in Tables 9–14.

Why 1-F as horizontal axis in Figure 3? Prefer the same as in Figure 2.

It is the same F1 as in Figure 2, plotted as 1 − F1 so that the axis reads as disagreement with the reference (caption and Section 3.1). We agree that one convention is better and will use F1 in the figures.

References: include publication venues.

Agreed. 23 of the 40 references currently show only a title and a date, because the bibliography style drops the arXiv identifier for preprints. We will add arXiv identifiers, and venues where the work has been published.

Thank you for the clarification. Though it is a really interesting problem, I still have remaining concerns about the scope and the practical meaning. I will maintain my score.

Comments on the paper

1 comment

We thank the reviewers. We could not run additional training during the rebuttal period, so the responses use results already in the paper. Two points recur. First, we did not train with a golden verifier, because of grading cost, and the paper does not claim equivalence with a golden-trained policy; the claim is relative to the strongest verifiers we trained with. Second, the claim concerns selected inexpensive verifiers, not all cheap ones; the revision will define "good enough" explicitly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.