- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
One key challenge in model self-evolution is using LLMs to rate and select useful data, while LLMs still struggle to produce fine-grained, multi-level scores to distinguish between good and bad instances. This paper proposes decomposing difficult evaluations into a sequence of binary decisions regarding quality degrees, further using their normalized verdict token (i.e., yes and No) probabilities to distinguish quality levels. Experiments on multiple reasoning benchmarks show the effectiveness of the method in preference learning.
Critical strengths
- The method can be combined with both off-policy and on-policy optimization.
- Two model families and multiple model sizes are used in the experiments.
- Results support the effectiveness of fine-grained quality judgment for optimization.
Critical weaknesses
- The combination with on-policy optimization (GRPO) is only evaluated on a 1.7B model.
- The recursive criteria rely heavily on human-constructed, task-specific definitions. The degree of manual engineering required to design these hierarchical criteria remains unclear when applying the method to new domains or less structured, reasoning-intensive tasks.
- The computational overhead during verification is not reported or compared against other methods and baselines; please report wall-clock time and token/compute costs.
Questions for authors
Will more rounds of iterative optimization help?
10 comments
Response to W1 (2/2)
Response to W1 (2/2)
3. Why RSV is particularly effective with GRPO
DPO uses only the highest- and lowest-scoring candidates. GRPO, by contrast, converts the scores of all 16 rollouts into group-relative advantages. This has two consequences:
- If a verifier assigns nearly identical scores to most rollouts in a group, the advantages vanish and the group contributes little gradient.
- If the within-group ordering is unreliable, the gradient points in the wrong direction.
To measure both effects, we computed two statistics over all GRPO training groups:
- the fraction of degenerate groups, defined as groups whose reward standard deviation is below 0.05;
- the within-group Spearman correlation between the reward and oracle quality, measured as the fraction of correctly assigned characters on K&K.
Table 3: Reward diagnostics on K&K (L=3), MLV → RSV.
| Model | Degenerate groups (%) ↓ | Reward–oracle correlation ↑ |
|---|---|---|
| Qwen3-1.7B | 21.3 → 3.2 | 0.23 → 0.36 |
| Gemma-3-4B-IT | 24.8 → 4.1 | 0.19 → 0.31 |
| Qwen2.5-7B-Instruct | 18.6 → 2.9 | 0.21 → 0.33 |
MLV's one-shot judgment tends to collapse many rollouts onto the same level. As a result, roughly one in five groups provides almost no learning signal. RSV assigns graded scores that separate rollouts within a group, so nearly every group contributes a useful gradient. Its rewards are also substantially better aligned with true quality.
Consistent with this, finer granularity helps GRPO for longer than it helps DPO. For Qwen3-1.7B on K&K, GRPO accuracy rises from 59.1 to 61.4, 62.6 and 62.8 at L=3, 5, 7 and 9.
4. Training stability
A self-generated reward can be exploited during on-policy training, so we monitored for reward hacking. Throughout training, the RSV reward and oracle accuracy on held-out training prompts rise together. Across checkpoints, their Pearson correlation is 0.94 for Qwen3-1.7B and 0.91 for Gemma-3-4B-IT. Average response length stays within ±8% of the base model, which indicates no drift toward length-based reward exploitation. This is consistent with RSV's weak dependence on length reported in Table 2 of the paper.
Summary. The benefit of RSV under GRPO is not specific to the 1.7B model. It holds for all three backbones in the paper and for larger models up to 8B, including Llama-3.1-8B-Instruct from an additional model family. The diagnostics explain why: RSV supplies graded, quality-aligned rewards for nearly every rollout group, whereas one-shot multi-level verification leaves many groups without a learning signal.
Response to W2 (Manual engineering of the criteria) (1/2)
Response to W2 (Manual engineering of the criteria) (1/2)
We thank the reviewer for this comment. RSV does not require designing new, task-specific definitions. It can be built directly on the same level definitions used by rubric-based or multi-level evaluation. The only requirement is that these definitions are posed as nested criteria ordered from strict to relaxed. Below we explain this conversion, give an example for a less structured task, and report results in new domains.
1. RSV can reuse existing level definitions
Any rubric that describes ordered quality levels can be turned into RSV criteria with one fixed rule. Level ("the response is at level ") becomes criterion ("the response is at level or better"). Because each criterion includes every level above it, the resulting criteria are nested by construction. No new definitions need to be written.
This is also how our comparisons are set up. MLV uses exactly the same level definitions as RSV (Table 8). The two methods therefore receive the same task-specific information and differ only in how it is posed to the model: all levels in one judgment for MLV, one threshold at a time for RSV. RSV's advantage does not come from additional engineering effort.
2. Example: a less structured task
Consider open-ended instruction following, where responses have no fixed structure and no single correct answer. A standard 5-point helpfulness rubric, such as the one used by our Prometheus-style baseline, describes the levels as follows:
- 5: fully addresses all parts of the request, accurately and clearly.
- 4: addresses the request with only minor omissions or inaccuracies.
- 3: addresses the main request but misses some parts or contains notable inaccuracies.
- 2: is only partially relevant and misses most of the request.
- 1: is irrelevant or unhelpful.
Applying the conversion rule gives RSV's criteria directly:
- : Does the response fully address all parts of the request, accurately and clearly?
- : Does it address the request with at most minor omissions or inaccuracies?
- : Does it address at least the main request?
- : Is it at least partially relevant to the request?
The same simple design applies to other loosely structured settings. For an agent trajectory, the criteria can be: the goal is achieved; the goal is nearly achieved, with only the final action missing or wrong; at least half of the necessary subgoals are completed; at least one subgoal is completed. No domain expertise or prompt tuning is needed beyond describing the quality levels, which rubric-based methods require anyway.
(continued in the next comment)
Response to W1 (GRPO evaluated only on a 1.7B model) (1/2)
Response to W1 (GRPO evaluated only on a 1.7B model) (1/2)
We thank the reviewer for this comment. We have extended the GRPO experiments in two directions:
All runs use the GRPO configuration in Table 9 (50 questions per policy update, 16 rollouts per question, one PPO epoch, learning rate 1×10−6) and L=3, as in Table 3 of the paper.
1. Results on the backbones used in the paper
Table 1: GRPO accuracy (%) with different reward signals.
RSV provides the best GRPO reward in all six settings. It outperforms the strongest baseline (IPO) by 1.9–5.3 points on K&K and 1.3–2.2 points on MATH-Hard. MLV stays close to, or even below, the base model on every backbone. This matches what we observed for Qwen3-1.7B in the paper.
2. Results on larger models
Table 2: GRPO accuracy (%) on larger models (L=3).
The advantage of RSV under on-policy optimization therefore holds across model sizes from 1.7B to 8B and across three model families.
(continued in the next comment)