acceptodds
Under review as a conference paper at ICLR 2027

Recursive Self-Verification: Progressive Binary Verification for Language Model Self-Evolution

Abstract

Self-evolving language models learn from their own generations by using the model to identify responses that yield useful training data. However, directly assigning fine-grained, multi-level scores to responses is often unreliable, particularly for smaller models that struggle with subtle quality distinctions, resulting in unreliable preference pairs. To address this limitation, we investigate whether fine-grained self-verification can be achieved by decomposing complex evaluations into a sequence of simpler binary decisions. Specifically, we introduce Recursive Self-Verification (RSV), a concise yet effective framework that evaluates responses through sequential binary verification steps ordered from strict to relaxed quality criteria. At each step, RSV uses normalized token probabilities for Yes and No to recursively allocate probability mass across quality levels, deriving the final evaluation score as the expected value of this distribution. These refined scores are then used to rank responses and construct high-quality preference pairs for model optimization. Experiments on the Knights-and-Knaves, TruthQuest, and MATH-Hard benchmarks demonstrate that RSV-guided preference learning consistently outperforms baseline methods in reasoning tasks across various model scales. Notably, on K&K with Qwen3-1.7B, the best RSV setting achieves an average accuracy of 63.1%, outperforming the strongest non-RSV baseline by 8.3 percentage points. These gains also extend to GRPO, confirming RSV's effectiveness under on-policy optimization for language model self-evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Recursive Self-Verification, Self-Evolving Language Models, Fine-Grained Self-Verification, Preference Learning, Binary Decision Decomposition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RNkHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

One key challenge in model self-evolution is using LLMs to rate and select useful data, while LLMs still struggle to produce fine-grained, multi-level scores to distinguish between good and bad instances. This paper proposes decomposing difficult evaluations into a sequence of binary decisions regarding quality degrees, further using their normalized verdict token (i.e., yes and No) probabilities to distinguish quality levels. Experiments on multiple reasoning benchmarks show the effectiveness of the method in preference learning.

Critical strengths
  • The method can be combined with both off-policy and on-policy optimization.
  • Two model families and multiple model sizes are used in the experiments.
  • Results support the effectiveness of fine-grained quality judgment for optimization.
Critical weaknesses
  • The combination with on-policy optimization (GRPO) is only evaluated on a 1.7B model.
  • The recursive criteria rely heavily on human-constructed, task-specific definitions. The degree of manual engineering required to design these hierarchical criteria remains unclear when applying the method to new domains or less structured, reasoning-intensive tasks.
  • The computational overhead during verification is not reported or compared against other methods and baselines; please report wall-clock time and token/compute costs.
Questions for authors

Will more rounds of iterative optimization help?

10 comments
AuthorsOpenReview ↗

Response to W1 (GRPO evaluated only on a 1.7B model) (1/2)

Response to W1 (GRPO evaluated only on a 1.7B model) (1/2)

We thank the reviewer for this comment. We have extended the GRPO experiments in two directions:

  • to the other two backbones used in the paper, Gemma-3-4B-IT and Qwen2.5-7B-Instruct;
  • to larger models, namely Qwen3-4B and Qwen3-8B (both in non-thinking mode), and Llama-3.1-8B-Instruct from a different developer.

All runs use the GRPO configuration in Table 9 (50 questions per policy update, 16 rollouts per question, one PPO epoch, learning rate 1×10−61\times10^{-6}) and L=3, as in Table 3 of the paper.

1. Results on the backbones used in the paper

Table 1: GRPO accuracy (%) with different reward signals.

MethodQwen3-1.7B K&KQwen3-1.7B MATH-HardGemma-3-4B K&KGemma-3-4B MATH-HardQwen2.5-7B K&KQwen2.5-7B MATH-Hard
Base35.039.131.650.519.139.9
SimpleSE46.043.034.952.619.842.5
IPO53.843.836.253.420.643.6
GenRM42.441.233.852.018.941.5
MLV (L=3)35.740.527.951.118.441.0
RSV (L=3)59.145.539.654.722.545.8

RSV provides the best GRPO reward in all six settings. It outperforms the strongest baseline (IPO) by 1.9–5.3 points on K&K and 1.3–2.2 points on MATH-Hard. MLV stays close to, or even below, the base model on every backbone. This matches what we observed for Qwen3-1.7B in the paper.

2. Results on larger models

Table 2: GRPO accuracy (%) on larger models (L=3).

ModelMethodK&K Avg.MATH-Hard
Qwen3-4BBase54.852.4
IPO63.154.9
MLV55.653.0
RSV68.757.0
Qwen3-8BBase61.456.8
IPO68.258.9
MLV62.057.3
RSV73.560.6
Llama-3.1-8B-InstructBase17.317.9
IPO18.819.9
MLV17.018.3
RSV21.022.1

The advantage of RSV under on-policy optimization therefore holds across model sizes from 1.7B to 8B and across three model families.

(continued in the next comment)

AuthorsOpenReview ↗

Response to W1 (2/2)

Response to W1 (2/2)

3. Why RSV is particularly effective with GRPO

DPO uses only the highest- and lowest-scoring candidates. GRPO, by contrast, converts the scores of all 16 rollouts into group-relative advantages. This has two consequences:

  • If a verifier assigns nearly identical scores to most rollouts in a group, the advantages vanish and the group contributes little gradient.
  • If the within-group ordering is unreliable, the gradient points in the wrong direction.

To measure both effects, we computed two statistics over all GRPO training groups:

  • the fraction of degenerate groups, defined as groups whose reward standard deviation is below 0.05;
  • the within-group Spearman correlation between the reward and oracle quality, measured as the fraction of correctly assigned characters on K&K.

Table 3: Reward diagnostics on K&K (L=3), MLV → RSV.

ModelDegenerate groups (%) ↓Reward–oracle correlation ↑
Qwen3-1.7B21.3 → 3.20.23 → 0.36
Gemma-3-4B-IT24.8 → 4.10.19 → 0.31
Qwen2.5-7B-Instruct18.6 → 2.90.21 → 0.33

MLV's one-shot judgment tends to collapse many rollouts onto the same level. As a result, roughly one in five groups provides almost no learning signal. RSV assigns graded scores that separate rollouts within a group, so nearly every group contributes a useful gradient. Its rewards are also substantially better aligned with true quality.

Consistent with this, finer granularity helps GRPO for longer than it helps DPO. For Qwen3-1.7B on K&K, GRPO accuracy rises from 59.1 to 61.4, 62.6 and 62.8 at L=3, 5, 7 and 9.

4. Training stability

A self-generated reward can be exploited during on-policy training, so we monitored for reward hacking. Throughout training, the RSV reward and oracle accuracy on held-out training prompts rise together. Across checkpoints, their Pearson correlation is 0.94 for Qwen3-1.7B and 0.91 for Gemma-3-4B-IT. Average response length stays within ±8% of the base model, which indicates no drift toward length-based reward exploitation. This is consistent with RSV's weak dependence on length reported in Table 2 of the paper.

Summary. The benefit of RSV under GRPO is not specific to the 1.7B model. It holds for all three backbones in the paper and for larger models up to 8B, including Llama-3.1-8B-Instruct from an additional model family. The diagnostics explain why: RSV supplies graded, quality-aligned rewards for nearly every rollout group, whereas one-shot multi-level verification leaves many groups without a learning signal.

AuthorsOpenReview ↗

Response to W2 (Manual engineering of the criteria) (1/2)

Response to W2 (Manual engineering of the criteria) (1/2)

We thank the reviewer for this comment. RSV does not require designing new, task-specific definitions. It can be built directly on the same level definitions used by rubric-based or multi-level evaluation. The only requirement is that these definitions are posed as nested criteria ordered from strict to relaxed. Below we explain this conversion, give an example for a less structured task, and report results in new domains.

1. RSV can reuse existing level definitions

Any rubric that describes ordered quality levels can be turned into RSV criteria with one fixed rule. Level kk ("the response is at level kk") becomes criterion CkC_k ("the response is at level kk or better"). Because each criterion includes every level above it, the resulting criteria are nested by construction. No new definitions need to be written.

This is also how our comparisons are set up. MLV uses exactly the same level definitions as RSV (Table 8). The two methods therefore receive the same task-specific information and differ only in how it is posed to the model: all levels in one judgment for MLV, one threshold at a time for RSV. RSV's advantage does not come from additional engineering effort.

2. Example: a less structured task

Consider open-ended instruction following, where responses have no fixed structure and no single correct answer. A standard 5-point helpfulness rubric, such as the one used by our Prometheus-style baseline, describes the levels as follows:

  • 5: fully addresses all parts of the request, accurately and clearly.
  • 4: addresses the request with only minor omissions or inaccuracies.
  • 3: addresses the main request but misses some parts or contains notable inaccuracies.
  • 2: is only partially relevant and misses most of the request.
  • 1: is irrelevant or unhelpful.

Applying the conversion rule gives RSV's criteria directly:

  • C1C_1: Does the response fully address all parts of the request, accurately and clearly?
  • C2C_2: Does it address the request with at most minor omissions or inaccuracies?
  • C3C_3: Does it address at least the main request?
  • C4C_4: Is it at least partially relevant to the request?

The same simple design applies to other loosely structured settings. For an agent trajectory, the criteria can be: the goal is achieved; the goal is nearly achieved, with only the final action missing or wrong; at least half of the necessary subgoals are completed; at least one subgoal is completed. No domain expertise or prompt tuning is needed beyond describing the quality levels, which rubric-based methods require anyway.

(continued in the next comment)

Reviewer_7FQpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Recursive Self-Verification (RSV), a framework for constructing preference supervision in self-improving LLMs. RSV decomposes response evaluation into a sequence of progressively relaxed binary verification prompts. The normalized token confidences from these separate binary queries are composed via a recursive residual probability-mass allocation rule to form a categorical quality distribution, then expected value is used as a quality score for selecting preference pairs DPO. Experiments are conducted across different models and benchmarks, and the authors show improvements over other self-rewarding baselines.

Critical strengths
  • The method is straightforward, which is a strength since it is easy to implement.
  • The paper has extensive experiments with some nice ablations/analyses.
Critical weaknesses
  • It is unclear why recursive verification is superior to multi-level verification, as this claim is not well justified by the analysis and/or examples. Because the multi-level verification approach is commonly used in rubric-based or checklist-based evaluation, the paper should compare against these baselines or provide a stronger justification.

  • Moreover, questions such as determining if a response is "partially correct" are highly ambiguous and may be even harder to evaluate than well-defined criteria in a rubric-based framework. This could become even more challenging in open-ended tasks.

  • The method largely relies on the strong assumption that the yes/no token probabilities are well calibrated. However, it is unclear whether this assumption holds across different verification levels or across diverse tasks. It would be interesting to see different distributions across levels/tasks.

Questions for authors
  • In different tasks, the performance seems generally improve when increasing LL. Have you tested what maximum value of LL achieves peak performance?
  • Since Qwen-2.5 has been shown to have some post-training problems, such as improving performance even when optimized against spurious rewards. Have you experimented with larger Qwen models, such as Qwen3-4B?
20 comments
AuthorsOpenReview ↗

Response to W1 (1/3)

Response to W1 (1/3)

We thank the reviewer for this comment. We address it in two parts. First, we explain with analysis and a concrete example why recursive verification is superior to multi-level verification. Second, we compare RSV directly against rubric-based and checklist-based evaluation.

1. Why recursive verification is superior to multi-level verification

The clearest way to see the difference is to compare how the two methods behave as the quality scale becomes finer. They start from exactly the same point and differ only in how they refine it.

Same starting point. At L=2, RSV and multi-level verification (MLV) coincide. Both make a single binary judgment on one boundary, fully correct or not, and with the same prompt they produce identical scores. This judgment is relatively easy but coarse: "not fully correct" lumps a nearly perfect response together with a completely wrong one. Both methods add levels to split this coarse side, but in fundamentally different ways.

RSV refines by adding a threshold. Going to L=3, RSV keeps the first question unchanged and adds an independent one: "Is the response at least mostly correct?" Because the criteria are nested, a fully correct response also counts as "at least mostly correct". The new question therefore only asks whether a response lies above or below one line, so it is no harder in form than the first. Residual allocation applies the new answer only to the mass that the first criterion left on its "No" side. Formally, by Eq. (10), appending a more relaxed criterion leaves all existing level masses PL1,…,PLKP_{L_1},\dots,P_{L_K} exactly unchanged and only splits the residual mass RK=∏j≤K(1−qj)R_K=\prod_{j\le K}(1-q_j). Granularity grows, but no question becomes harder and nothing already established is disturbed.

MLV refines by making the judgment harder. MLV instead changes the question itself. At L=3, the model must place the response into "fully correct", "mostly correct" or "other" within one prediction, and this makes both boundaries harder:

  • The old boundary gets harder. "Fully correct" is now contrasted with its immediate neighbor, "mostly correct", rather than with a coarse remainder.
  • The new level is a band, not a threshold. To assign "mostly correct", the model must separate it both from "fully correct" above and from lower quality below, which means two boundaries at once.

In general, an L-level judgment is correct only if all of its L−1L-1 boundary decisions are correct simultaneously:

1[h^≠r]=max⁡k1[c^k≠ck],∣h^−r∣=∑k1[c^k≠ck].\mathbf{1}[\hat h\ne r]=\max_k \mathbf{1}[\hat c_k\ne c_k],\qquad |\hat h-r|=\sum_k\mathbf{1}[\hat c_k\ne c_k].

Every refinement of MLV adds another boundary that one prediction must resolve jointly. Under a perfect verifier the two methods give the same answer (Prop. 1), so the difference lies entirely in how reliably an imperfect model answers these two kinds of questions.

(continued in the next comment)

AuthorsOpenReview ↗

Response to W1 (2/3)

Response to W1 (2/3)

A concrete example. The table below shows an illustrative pair of candidates for the same K&K puzzle (Qwen3-1.7B, L=4, level values {1, 0.3, 0.1, 0}). Response A is fully correct. Response B contains one invalid deduction and assigns two of three characters correctly.

ResponseMLV level probs (L1L_1–L4L_4)MLV scoreRSV q1,q2,q3q_1, q_2, q_3RSV distributionRSV score
A: fully correct0.47 / 0.38 / 0.11 / 0.040.5950.81 / 0.88 / 0.930.81 / 0.17 / 0.02 / 0.000.862
B: one invalid deduction0.44 / 0.41 / 0.12 / 0.030.5750.24 / 0.73 / 0.860.24 / 0.55 / 0.18 / 0.030.424

MLV hesitates between "fully" and "mostly correct" for both responses, because it must decide which band each one falls into, and the two end up nearly tied (gap 0.020). RSV's first question only asks whether the response is fully correct, and the model answers it clearly for both (0.81 vs. 0.24). B's partial quality is then captured by the second threshold question. The gap becomes 0.438. Table 6 shows on Gemma-3-4B-IT that near-ties are systematic under MLV: 65–72% of top-4 candidate pairs are near-ties, versus about 24% under RSV.

Evidence across refinement levels. The table tracks both methods as L grows on K&K with Qwen3-1.7B (entries are MLV / RSV). Q1Q_1 AUROC measures how well the "fully correct" belief identifies exactly solved puzzles. New-boundary AUROC evaluates the boundary added at that L.

LQ1Q_1 AUROCNew-boundary AUROCAccuracy
20.734 / 0.734—55.1 / 55.1
30.701 / 0.7340.668 / 0.70933.8 / 60.0
40.706 / 0.7340.623 / 0.67139.4 / 61.9
50.689 / 0.7340.596 / 0.64249.9 / 63.1

Three observations follow:

  • RSV's top-boundary judgment is unchanged at every L by construction. MLV's degrades as soon as a neighboring level is introduced.
  • RSV's threshold question is consistently more discriminative than MLV's band judgment at each newly added boundary.
  • RSV improves at every step. MLV's accuracy partly recovers at larger L, mainly because extra low-quality levels help it pick clearly wrong candidates as rejected responses, but its top-boundary judgment does not recover, and at every L≥3 it stays below the binary judgment it started from.

The gap is not explained by compute: averaging three MLV judgments barely helps (39.4 → 40.3, Table 8). It also holds under on-policy GRPO (59.1 vs. 35.7, Table 3).

2. Comparison with rubric-based and checklist-based evaluation

We agree these are important baselines. They relate to RSV as follows.

  • Rubric-based evaluation (e.g., Prometheus-style 1–5 rubrics, or the additive rubric of Self-Rewarding) is multi-level verification. The rubric describes the levels in detail, but the model still places the response on the whole scale in one judgment. The analysis above therefore applies directly.
  • Checklist-based evaluation (e.g., CheckEval, TICK) does decompose evaluation into yes/no questions, but along content: different aspects of the response. The items are not nested, have no known logical relation and are typically averaged. A failure on a decisive item, such as a wrong final assignment, is diluted by passes on minor ones. RSV decomposes along the quality scale of one holistic judgment, so its questions are nested thresholds with a principled composition.

(continued in the next comment)

AuthorsOpenReview ↗

Response to W1 (3/3)

Response to W1 (3/3)

We implemented five representative baselines with the same backbone, candidates and DPO pipeline as RSV:

  1. Additive rubric (Self-Rewarding; Yuan et al., 2024): five cumulative criteria, with a short justification and a total score per generation, averaged over three sampled judgments as in the original method.
  2. Rubric scoring (Prometheus-style; Kim et al., 2024a): a 1–5 scale with a description for each score, probability-weighted.
  3. Rubric + rationale (LLM-as-a-judge; Zheng et al., 2023): written feedback followed by a 1–5 score.
  4. Fixed checklist (CheckEval-style; Lee et al., 2024): four expert-written aspect questions, scored by mean Yes-probability. It uses the same four calls as RSV.
  5. Generated checklist (TICK-style; Cook et al., 2024): the model writes an instance-specific checklist, then answers each item.

RSV uses L=5 to match the five-point rubrics. The open-ended column trains on UltraFeedback prompts and reports the AlpacaEval 2.0 length-controlled win rate (the same setting as in our response to Reviewer 1, W4), where checklist methods are most commonly applied. Values are accuracy (%), or LC win rate (%) for AlpacaEval.

ModelMethodCalls/cand.K&KTruthQuestMATH-HardAlpacaEval 2.0
Qwen3-1.7BAdditive rubric348.549.543.021.5
Rubric scoring146.244.142.120.8
Rubric + rationale150.347.643.422.0
Fixed checklist451.946.743.522.4
Generated checklist1 + 6.252.647.944.023.3
RSV (L=5)463.157.445.324.7
Gemma-3-4B-ITAdditive rubric338.232.053.632.8
Rubric scoring135.431.052.832.1
Rubric + rationale137.632.953.933.4
Fixed checklist435.332.253.733.6
Generated checklist1 + 6.236.032.854.034.5
RSV (L=5)442.237.355.236.2

Four observations follow from this table.

  • RSV is best in all 16 settings. This holds across two model families, three reasoning benchmarks and open-ended generation.
  • Rubrics behave like multi-level verification. Detailed level descriptions or a written rationale help somewhat, but they do not remove the joint decision over all levels. On K&K, the best rubric variant still trails RSV by 12.8 points (Qwen) and 4.0 points (Gemma).
  • Checklists are the strongest baselines, but still fall short. Their yes/no questions help, yet even with the same number of calls (fixed checklist) or more (generated checklist), they trail RSV by 10.5 points on K&K for Qwen3-1.7B. This holds even on open-ended generation, the setting checklists were designed for (23.3 vs. 24.7 and 34.5 vs. 36.2). The difference lies in the structure of the questions: nested thresholds versus independent aspects.
  • RSV is also cheaper than generative evaluators. It reads a single token per criterion, whereas the additive rubric, rubric-with-rationale and generated-checklist methods decode about 450, 340 and 410 tokens per candidate, respectively.

The two forms of decomposition are also complementary. For tasks with several genuinely distinct quality aspects, RSV's nested thresholds can be applied within each checklist item or rubric dimension.

Summary. RSV is superior to multi-level verification because it refines quality by adding single-threshold questions whose answers only refine what stricter criteria left undecided. Multi-level verification instead turns one judgment into a joint decision over more boundaries. The analysis, the concrete example and the refinement experiments support this, and the advantage holds against widely used rubric-based and checklist-based evaluators across reasoning and open-ended tasks.

Reviewer_4zeEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Recursive Self-Verification (RSV), a sequential binary verification method that aggregates Yes/No token probabilities over progressively relaxed quality criteria to rank responses and construct preference pairs for DPO. Experiments on three reasoning benchmarks show improvements over several baselines.

Critical strengths
  • Addresses the practical challenge of unreliable self-evaluation in language model self-evolution.
  • Includes comparisons with multi-level verification baselines and evaluations across several benchmarks.
  • The method is simple to implement and does not require an external verifier.
Critical weaknesses
  • Limited methodological novelty. RSV primarily combines sequential binary judgments with a probability aggregation scheme. The paper does not sufficiently establish why this specific formulation constitutes a substantive advance over existing multi-level scoring or decomposed verification approaches.
  • Insufficient evidence for the proposed mechanism. Although RSV outperforms the MLV baseline, the experiments do not adequately isolate whether the gains come from recursive probability aggregation, criterion design, or repeated binary evaluations.
  • Weak justification for the probabilistic formulation. Normalizing Yes/No token probabilities does not ensure calibrated probabilities of criterion satisfaction. Moreover, independently evaluated nested criteria may yield inconsistent judgments, undermining the semantic interpretation of the resulting quality distribution.
  • Limited scope of evaluation. The experiments primarily focus on reasoning benchmarks, making it difficult to assess whether the approach generalizes to broader language generation or complex agent tasks.
Questions for authors
  • What fundamental advantage does RSV offer over existing multi-level verification methods beyond its particular probability aggregation scheme?
  • Can the authors demonstrate through controlled ablations that recursive aggregation, rather than criterion design or repeated verification, is responsible for the observed improvements?
  • How do the authors validate that the induced quality distribution meaningfully reflects response quality rather than merely serving as a heuristic ranking score?
11 comments
AuthorsOpenReview ↗

Response to W1 & Q1: Why RSV improves over multi-level verification (1/2)

Response to W1 & Q1: Why RSV improves over multi-level verification (1/2)

We thank the reviewer for pushing us to articulate this more clearly. The clearest way to see RSV's advantage is to compare how the two methods behave as the quality scale becomes finer. They start from exactly the same point and differ only in how they refine it.

Same starting point. At L=2, RSV and MLV coincide. Both make a single binary judgment on one boundary, fully correct or not, and with the same prompt they produce identical scores. This judgment is relatively easy for the model, but it is coarse: "not fully correct" lumps a nearly perfect response together with a completely wrong one. Both methods add levels to split this coarse side, but they do so in fundamentally different ways.

How RSV refines: by adding a threshold. Going to L=3, RSV keeps the first question unchanged and adds an independent one: "Is the response at least mostly correct?" Because the criteria are nested, a fully correct response also counts as "at least mostly correct." The new question therefore only asks whether a response lies above or below one line. It is a single-boundary question, no harder in form than the first.

Residual allocation then applies the new answer only to the probability mass that the first criterion left on its "No" side. Formally, it follows from Eq. (10) that appending a more relaxed criterion CK+1C_{K+1} leaves PL1,…,PLKP_{L_1},\dots,P_{L_K}, and hence every existing boundary belief Q1,…,QKQ_1,\dots,Q_K, exactly unchanged. The new criterion only splits the residual mass RK=∏j≤K(1−qj)R_K=\prod_{j\le K}(1-q_j) into qK+1RKq_{K+1}R_K and (1−qK+1)RK(1-q_{K+1})R_K. Each refinement thus adds information about lower-quality responses without disturbing what has already been established, and no question becomes harder as L grows.

How MLV refines: by making the judgment harder. MLV instead changes the question itself. At L=3, the model must place the response into "fully correct", "mostly correct" or "other" within one prediction. This makes both the old boundary and the new one harder to judge:

  • The old boundary gets harder. "Fully correct" is no longer contrasted with a coarse remainder but with its immediate neighbor, "mostly correct", which is a much finer distinction.
  • The new level is a band, not a threshold. To assign "mostly correct", the model must separate it from "fully correct" above and from lower quality below, which means two boundaries at once. RSV's new question groups "fully" and "mostly correct" together and asks about only one line.

More generally, an L-level judgment is correct only if all L−1L-1 of its boundary decisions are correct simultaneously, and its ordinal error equals the number of boundaries it gets wrong:

1[h^≠r]=max⁡k1[c^k≠ck],∣h^−r∣=∑k1[c^k≠ck].\mathbf{1}[\hat h\ne r]=\max_k \mathbf{1}[\hat c_k\ne c_k],\qquad |\hat h-r|=\sum_k\mathbf{1}[\hat c_k\ne c_k].

Every refinement of MLV adds another boundary that the same single prediction must resolve jointly. Finer granularity is paid for with less reliable judgments, including on boundaries the model could already judge.

This is the only place where the two methods differ. Their scores take the same form, S=SLK+1+∑mΔmQmS=S_{L_{K+1}}+\sum_m\Delta_mQ_m, and they share the same target and expressiveness (Props. 1 and 3). What differs is how the boundary beliefs QmQ_m are produced as the scale is refined.

(continued in the next comment)

AuthorsOpenReview ↗

Response to W1 & Q1 (2/2)

Response to W1 & Q1 (2/2)

Evidence. Table R1 tracks both methods as L grows on K&K with Qwen3-1.7B (entries are MLV / RSV).

  • Q1Q_1 AUROC measures how well the "fully correct" belief identifies exactly solved puzzles.
  • New-boundary AUROC evaluates the boundary added at that L. Labels are derived from per-character assignment correctness.
  • Accuracy is the downstream result.

Table R1: Refinement trajectory on K&K (Qwen3-1.7B; entries are MLV / RSV).

LQ1Q_1 AUROC (MLV / RSV)New-boundary AUROC (MLV / RSV)Accuracy (MLV / RSV)
20.734 / 0.734—55.1 / 55.1
30.701 / 0.7340.668 / 0.70933.8 / 60.0
40.706 / 0.7340.623 / 0.67139.4 / 61.9
50.689 / 0.7340.596 / 0.64249.9 / 63.1

The results match the analysis:

  • Top boundary. RSV's judgment is unchanged at every L by construction, since its first question never changes. MLV's judgment drops as soon as a neighboring level is introduced and does not recover.
  • New boundary. RSV's single-threshold question is consistently more discriminative than MLV's band judgment.
  • Downstream accuracy. RSV improves at every step. MLV's accuracy partly recovers from L=3 to L=5, mainly because the additional low-quality levels help it pick clearly wrong candidates as rejected responses. Its top-boundary judgment does not recover, however, and at every L≥3 it stays below the binary judgment it started from.

The same pattern holds for Gemma-3-4B-IT. MLV's Q1Q_1 AUROC falls from 0.716 at L=2 to 0.682 and 0.674 at L=3 and L=4, while RSV's stays at 0.716. Accuracy moves from 37.0 at L=2 to 24.6 and 26.7 for MLV, versus 39.2 and 41.1 for RSV.

Other results are consistent with this picture:

  • Not extra compute. Repeating the multi-level question at RSV's call budget barely helps (39.4 → 40.3, Table 8).
  • MLV collapses onto one label. On Gemma-3-4B-IT, pmax⁡>0.9p_{\max}>0.9 for 56–61% of MLV judgments versus about 18% for RSV, leaving 65–72% of top-4 candidate pairs as near-ties (Table 6).
  • On-policy RL. The advantage carries over to GRPO (59.1 vs. 35.7, Table 3).

Summary. RSV refines a coarse quality judgment by adding independent single-threshold questions, and residual allocation ensures that each new answer only refines what stricter criteria left undecided. Granularity therefore grows without making any judgment harder. Multi-level verification refines by turning one judgment into a joint decision over more boundaries, which weakens both the new distinctions and the old ones. This structural property, realized jointly by nested single-boundary questions and residual allocation rather than by extra compute, is the substantive advance.

AuthorsOpenReview ↗

Response to W2 & Q2 (1/2)

Response to W2 & Q2 (1/2)

Thank you for this question. We understand the concern as follows: RSV's improvement over MLV might come not from recursive aggregation, but simply from verifying more times or from better-designed criteria. We first clarify how these factors relate within RSV, and then separate them with controlled ablations.

What is and is not a confound. For L>2, asking one question per boundary with nested criteria is not an extra ingredient added to RSV. It is RSV's verification paradigm and, as discussed in W1/Q1, the source of its main advantage: each question involves a single threshold instead of a joint multi-level decision. Multiple verifications follow directly from this paradigm. Splitting MLV's one-shot decision into simpler single-boundary judgments requires one judgment per boundary.

What could genuinely confound the comparison is (i) extra verifier calls as such, regardless of what they ask, and (ii) particular prompt wording. Within the paradigm, the remaining question is whether nesting and residual allocation each matter when everything else is held fixed. Our ablations address all of these points.

Ablation design. We start from L=2, where all variants coincide by construction: a single "fully correct?" judgment in one call, so nesting and aggregation play no role. We then increase L and change one factor at a time. Every variant except single-call MLV uses the same number of calls as RSV (1/2/3 at L=2/3/4).

Table R2: Controlled ablation starting from L=2 (K&K accuracy %, Qwen3-1.7B).

VariantFactor changedL=2L=3L=4
RSV (nested thresholds + residual allocation)—55.160.061.9
MLV, single calljoint judgment55.133.839.4
MLV, same prompt, sampled verdicts (T=0.6)+ calls55.134.439.9
MLV, paraphrased prompts (Table 8)+ calls, wording55.133.940.3
L=2 question repeated+ calls, no new boundary55.155.555.8
RSV with non-nested (band) criteriacriterion structure55.157.458.6
RSV with naive sum instead of residual allocationaggregation55.157.057.3

(continued in the next comment)

Comments on the paper

2 comments
Area_Chair_RvSmOpenReview ↗

Reminder: Author-reviewer discussion period ending soon

Dear Reviewers,

Thank you again for your time and effort in reviewing this paper. This is a friendly reminder that the author-reviewer discussion period is coming to a close.

If you haven't already, please check out the authors' rebuttal and engage in the discussion by leaving comments. Even a brief acknowledgment confirming whether your questions were addressed would be very helpful for our discussion.

Dear Area Chair,

Thank you for overseeing the review of our submission. We have carefully considered the reviewers' concerns and devoted substantial effort to addressing them through additional theoretical analyses and experiments. We would like to briefly summarize our responses and the current discussion status.

Reviewer 4zeE. We provided theoretical explanations and controlled ablations clarifying RSV's methodological contribution (Tables R1–R2), validated the induced quality distribution using oracle labels and independent evaluators (Tables R6–R7), and extended experiments to code generation, open-ended instruction following, and agent tasks (Table R5).

Reviewer 7FQp. We added rubric- and checklist-based comparisons, analyses across verification levels, and experiments with larger models. We sincerely appreciate the reviewer's further feedback on W2 and W3. In response, we conducted additional experiments on generic and automatically generated criteria, including MedQA, and provided further theoretical and empirical analyses of calibration, ranking correctness, and downstream performance.

Reviewer RNkH. We extended GRPO experiments to larger and more diverse models, investigated criterion transferability, reported comprehensive computational cost comparisons, and evaluated multiple rounds of iterative self-evolution.

Regarding the discussion progress, we have followed up with Reviewers 4zeE and RNkH but have not yet received further feedback. Reviewer 7FQp kindly responded and clarified the remaining W2 and W3 concerns; we subsequently provided additional experiments and explanations addressing both points, but have not yet received another response.

We fully appreciate the reviewers' time and effort. As the discussion period comes to an end, we would be sincerely grateful if these additional findings could receive careful consideration in the reviewer–AC discussion and final assessment, particularly where they help resolve concerns raised in the initial reviews.

If possible, we would also greatly appreciate your guidance on any important concerns that may still require clarification.

Thank you again for your valuable time, effort, and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.