- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces RISE (Recursive Improvement via Self-Extrapolating Policy Distillation), a post-training framework that alternates reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) from a synthetic "future" teacher constructed out of the model's own training trajectory (Section 3, Figure 1). At iteration , RISE first runs GRPO on rollouts sampled from to obtain , and then extrapolates beyond relative to a trailing anchor with a decaying factor (Section 3.1–3.3, Eqs. 2–4). The paper instantiates this extrapolation in both weight space (, Eq. 3, Algorithm 2) and output-logit space ( projected onto the top- tokens plus a single aggregated tail bucket, Eqs. 4 and 6, Algorithm 1), and distills back into on the reused pre-RLVR rollouts using token-level Jensen–Shannon divergence (JSD).
RISE is evaluated across mathematical reasoning (Qwen3-1.7B, Qwen3-8B, and Qwen3-1.7B-Base on DAPOMath; OLMo3-7B-Instruct-SFT on OpenR1-Math-46K, Table 1 and Appendix Table E.1), multi-domain math + STEM (Qwen3-4B-Base, Table 2, Figure 3), code generation (Qwen3-8B-Base, Figure 4, Appendix Figure E.2), and multi-turn agentic tasks (Qwen2.5-3B-Instruct on ALFWorld and WebShop, Table 3), alongside component ablations in Section 4.2–4.3 and Appendix E.
Critical strengths
1. Simple, external-teacher-free construction of a dense token-level distillation target.
RISE addresses a practical limitation of on-policy distillation: strong external teachers often suffer from student-prefix mismatch or vocabulary differences, while privileged-context self-distillation requires ground-truth reference solutions and only applies to prompts with at least one successful rollout. Extrapolating along the model's own training trajectory in either parameter space (Eq. 3) or top- logit space (Eqs. 4 and 6) requires no external model or privileged context, and can be dropped directly into existing VeRL/GRPO pipelines.
2. Consistent mathematical reasoning gains on post-trained models backed by component ablations.
On Qwen3-1.7B, Qwen3-8B (with 3-seed verification in Appendix Table E.8), and OLMo3-7B-Instruct-SFT (Table 1), RISE consistently improves in-domain and OOD math averages over GRPO and the evaluated privileged-context self-distillation baselines (GRPO+SDPO, SDAR, RLSD), raising OLMo3-7B Math Avg from 47.6 to 56.4 (+8.8) and Qwen3-1.7B from 45.4 to 50.2 (+4.8). The ablations removing RLVR (w/o RLVR) and removing OPD (w/o OPD) in Figure 5 and Table E.3 clearly show that neither random walk extrapolation nor raw parameter extrapolation without distillation suffices on its own.
Critical weaknesses
1. The theoretical justification does not match the implemented algorithm or explain why distillation works when w/o OPD fails.
The theoretical narrative in Section 3.2 and Appendix B relies on (i) bounding student suboptimality via reverse KL (Theorem 2, Eqs. B.4–B.7) and claiming that transfers this bound to JSD minimization, and (ii) analyzing 1D extrapolation from the initial checkpoint under a Lipschitz reward assumption (Proposition 3 and Corollary 4, Eqs. B.8–B.13). As detailed under Soundness, the JSD-to-KL inequality is in the wrong direction to bound from a small ; the actual Qwen implementation uses an EMA anchor that mixes historical OPD and RLVR steps rather than isolating the current RLVR update; the OPD step reuses off-policy pre-RLVR rollouts; and the w/o OPD ablation (Table E.3) shows that the extrapolated teacher does not actually achieve a higher task return than GRPO, contradicting the teacher-gap reduction story in Theorem 2.
2. Incomplete compute/optimization controls, confounded baseline comparisons, and an inconsistent model–task matrix.
- Unequal wall-clock compute and missing momentum/extrapolation controls: Section 4.3 (p. 9) reports that each RISE step takes (
1.7B) to (8B) the wall-clock time ofGRPO. WhileGRPO-2x(Appendix Table E.7) takes a second inner-loop gradient pass on the same rollouts, the paper never compares againstGRPOtrained for equal wall-clock time / GPU-hours (i.e., more rollout steps), nor against weight-space momentum / Lookahead / Nesterov extrapolation or tuned GRPO learning rates. In code generation (Figure 4, p. 7), where RISE reachesGRPO's step-90 accuracy at roughly step 50 with "comparable final accuracy," a per-step wall-clock cost (50 * 1.6 = 80GRPO-equivalent time units) largely eliminates the wall-clock speedup. - Confounded teacher-construction comparison: In Table 1, the privileged-teacher baselines (
SDPO,SDAR,RLSD) only apply distillation on the of prompts that produce at least one correct rollout (Appendix D.3, p. 24) and use different loss formulations (auxiliary KL, gating, or advantage reweighting), whereas RISE runs a separate JSD OPD stage on of prompts. There is no controlled ablation keeping the exact RISE two-stage JSD loop and replacing only with the non-extrapolated post-RLVR policy (), an EMA teacher, or a privileged teacher. - Inconsistent model variants across tasks and weak gains on
Qwen3-1.7B-Base: Table 1 uses post-trainedQwen3-1.7BandQwen3-8B(skippingQwen3-4B), while relegating pretrainedQwen3-1.7B-Baseto the unreferencedAppendix Table E.1, where RISE's Math Avg gain overGRPO(28.0) shrinks to+1.2(29.2logit) and+0.1(28.1weight). Meanwhile, multi-domain training (Table 2) tests onlyQwen3-4B-Base; code generation (Figure 4) switches toQwen3-8B-Baseand omits all OPSD baselines; agentic tasks (Table 3) switch toQwen2.5-3B-Instruct(whereRISE (logit)trailsGRPOonWS-S,78.8vs.79.8) and omit all OPSD andGIGPObaselines; and key hyperparameters (, decay schedule, and for Qwen vs. for OLMo, which swingsQwen3-1.7BMath Avg by up to6.3points in Table E.6) appear to be selected directly on the test benchmarks.
Questions for authors
-
Assessment-critical (Equal wall-clock / GPU-hour comparison and controlled teacher swap inside the RISE loop):
- Given the per-step wall-clock overhead of RISE (Section 4.3), how does RISE compare against
GRPOtrained for equal wall-clock time / total GPU-hours (i.e., more rollout steps) and against standard weight-space Lookahead / Nesterov momentum extrapolation? - In Table 1, what happens if you keep the exact RISE two-stage JSD training loop on of prompts, and only replace the extrapolated teacher with (a) the unextrapolated post-RLVR checkpoint (), (b) the trailing EMA checkpoint , or (c) a privileged-context teacher?
- Given the per-step wall-clock overhead of RISE (Section 4.3), how does RISE compare against
-
Assessment-critical (Reconciling the theory with JSD, EMA anchors, top- tail aggregation, and
w/o OPD):- Can you provide a corrected suboptimality bound in terms of (e.g., via ) rather than relying on the reversed inequality , and account for reusing pre-RLVR rollouts ?
- Under the default EMA anchor, what fraction of the displacement aligns with the current-step RLVR update versus accumulated past OPD updates? Also, why does distilling improve accuracy by
3–5points when directly evaluating the extrapolated weights (w/o OPD, Table E.3) yields almost zero gain (+0.3on8B,+0.2on1.7B,-0.3on1.7B-Base)?
-
Assessment-critical (Base vs. post-trained checkpoint behavior, hyperparameter selection protocol, and unbiased
pass@k):- Why does
RISE (weight)improve overGRPOby+4.5points on post-trainedQwen3-1.7B(Table 1) but only+0.1points (28.1vs.28.0) on pretrainedQwen3-1.7B-Base(Appendix Table E.1)? WasQwen3thinking mode disabled during RLVR rollout training as well as evaluation (Appendix D.2)? - Were , the decay schedule, and ( for Qwen vs. for OLMo) tuned on a held-out validation split or selected on the reported test benchmarks? Finally, can you report the exact final numerical table for code generation (including
LiveCodeBench v6) and recompute Table E.2 using the standard unbiasedpass@kestimator?
- Why does
3 comments
Compute controls and teacher comparison (W2, Q1)
(a) Compute and optimization controls. We now report accuracy against measured training wall-clock time (App. E.13). RISE (logit) takes 1.53× and 1.65× GRPO's time per iteration on Qwen3-8B and OLMo3-7B. Even so, it reaches the Math Avg that GRPO attains at the end of training earlier: after 11.3 vs. 14.4 hours on Qwen3-8B and 17.6 vs. 24.2 hours on OLMo3-7B. At the time GRPO finishes:
- RISE (logit) is on par with it on Qwen3-8B.
- RISE (logit) leads by 3.8 points on OLMo3-7B, where GRPO gains only 0.8 points over the last quarter of its training time against 2.1 for RISE.
Weight-space RISE is costlier per iteration, because its teacher is a separate set of parameters that must be materialized and run forward during distillation. Our implementation of it is not optimized for speed.
On code, the per-iteration cost is nearly identical: executing the generated programs for the reward dominates each iteration (995 vs. 1,009 seconds median). RISE (logit) reaches 80.9 HumanEval+ after 12.1 hours, a level GRPO first reaches after 23.6 hours. The iteration-count speedup therefore also holds in wall-clock time.
Regarding optimizer controls, the w/o-OPD variant (Table E.4) extrapolates the parameters along the recent RLVR direction, the operation underlying Lookahead- and Nesterov-style extrapolation. It ends at GRPO's level (45.6 vs. 45.4 on Qwen3-1.7B), whereas RISE reaches 50.2. Momentum is already present through AdamW's first moment, and all methods share the same learning rate (App. D.1). RISE leaves the optimizer unchanged, so optimizer-level variants are complementary to it.
(b) Teacher-construction controls. The privileged-teacher baselines differ from RISE in two respects: how the teacher's signal enters the update, and on which prompts a teacher exists.
- The first is a design choice, which the controls below hold fixed: every teacher, including the privileged one, enters through the same JSD distillation phase as RISE's.
- The second is inherent to privileged teachers. A privileged teacher requires a correct sibling solution, so it does not exist on prompts where every rollout fails, whereas RISE's teacher is defined on every prompt. This coverage gap is moderate: the privileged teacher is active on of prompts for Qwen3-1.7B and for Qwen3-8B (App. D.3).
Following the reviewer's design, we keep the RISE loop identical and change only the teacher (App. E.12, Table E.10, Qwen3-1.7B). Against RISE's 50.2 Math Avg, the three teachers the reviewer proposes give:
- (a) Post-RLVR policy (): the teacher equals the student after its RLVR step, so the distillation signal vanishes and the variant reduces to GRPO (45.4).
- (b) Trailing EMA anchor: 42.9, and a teacher interpolated toward it () reaches 44.5.
- (c) Privileged-context teacher: it conditions the EMA anchor on a correct sibling solution as in SDPO. It reaches 41.4, below GRPO, in line with the limited gains of the privileged-teacher baselines in Table 1.
Model–task matrix, hyperparameters, and evaluation protocol (W2, Q3)
Model–task matrix. The experiments are designed for breadth: each task family tests RISE under a different reward structure, model family, or starting point. Together, they cover two model families (Qwen, OLMo), sizes from 1.7B to 8B, and base, SFT, and post-trained starting points. Running every model on every task family exceeds our compute budget, so each task family other than math is tested on one model. The OPSD baselines are compared on all five math and multi-domain settings (Tables 1, 2, E.1), while code and agentic tasks compare RISE against GRPO, the RL objective it builds on. GiGPO is likewise an RL objective, so RISE could be built on it in the same way.
Base vs. post-trained models. On Qwen3-1.7B-Base, the gain is smaller (+1.2 Math Avg for RISE (logit), +0.1 for RISE (weight); Table E.1). This smaller margin is not specific to RISE: on the base model, every method that adds a teacher signal to GRPO stays within 1.2 points of it, whereas on the post-trained model the same methods range from 2.2 below to 4.8 above:
| Model | GRPO | Teacher-based methods (Math Avg) |
|---|---|---|
| Qwen3-1.7B-Base | 28.0 | GRPO+SDPO 28.4, SDAR 26.9, RLSD 27.6, RISE 29.2 / 28.1 |
| Qwen3-1.7B (post-trained) | 45.4 | 43.2–50.2 |
RISE (logit) remains the best method on the base model.
Agentic tasks. On WebShop, RISE (logit) trails GRPO by 1.0 point in score while improving accuracy by 4.7 points. RISE (weight) improves all three agentic metrics (Table 3).
Hyperparameters. During development, we selected hyperparameters by MATH-500 accuracy. The other five in-domain math benchmarks and the out-of-distribution benchmarks were not used for selection, and we did not hold out a separate validation split (App. D.1). The settings are:
- and ( for code) are shared across all settings.
- is set per model family and shared between the logit- and weight-space variants.
RISE's gains are not concentrated on the selection benchmark. In every configuration of Table 1, the gain over GRPO averaged over the five other math benchmarks exceeds the gain on MATH-500:
| Model | Gain on MATH-500 (logit / weight) | Avg. gain on the other five (logit / weight) |
|---|---|---|
| Qwen3-8B | +1.0 / +0.6 | +2.8 / +3.1 |
| Qwen3-1.7B | −0.4 / +1.7 | +5.9 / +4.3 |
| OLMo3-7B | +5.4 / +5.3 | +9.5 / +6.3 |
Moreover, the gains do not hinge on the chosen values:
- Every tested and every decay schedule gives 61.8–63.0 Math Avg on Qwen3-8B, against 60.0 for GRPO (Table 4).
- RISE exceeds GRPO for every anchor rate tested, including , i.e., the previous checkpoint without averaging (Table E.7). The 6.3-point spread the reviewer notes (OLMo3-7B, 56.4 vs. 50.1) is thus between two settings that both exceed GRPO's 47.6.
Thinking mode. Thinking is disabled in the chat template for all Qwen3 models, for RLVR rollouts as well as for evaluation (App. D.2).
Final code results. Table E.2 reports the final avg@4 accuracy, including LiveCodeBench v6:
| Method | HumanEval+ | MBPP+ | LiveCodeBench v6 |
|---|---|---|---|
| GRPO | 80.5 | 63.5 | 54.8 |
| RISE (logit) | 81.6 | 65.7 | 55.6 |
| RISE (weight) | 81.3 | 64.2 | 53.3 |
pass@. Table E.3 now uses the standard estimator of Chen et al. (2021), . With , this is the fraction of problems solved by at least one of the 16 samples. Under this estimator, RISE improves pass@16 over GRPO in six of the eight cells and matches it in the other two (RISE (logit) on Qwen3-8B). The gains reach on AIME'25 for both RISE (logit) on Qwen3-1.7B and RISE (weight) on Qwen3-8B.
Response to Reviewer aPSv
We thank the reviewer for the detailed review and precise questions.
Theory and the implemented algorithm (W1, Q2)
We address the four points in turn.
(i) JSD. The main text states the analysis for reverse KL for convenience; the bound also holds for JSD directly (App. B.1, Eq. B.9). Applying Pinsker's inequality to each half of the JSD gives JSD(P,Q)≥21TV(P,Q)2. Substituting TV≤2JSD into the proof of Theorem 2 then yields J(π∗)−J(πθ)≤[J(π∗)−J(πT)]+T2LˉJSD. Empirically, JSD also outperforms reverse KL as the distillation loss in the RISE loop (50.2 vs. 48.7 Math Avg on Qwen3-1.7B; App. E.14).
(ii) EMA anchor. An exponential moving average of past checkpoints is a common slowly moving reference in self-distillation; the SDPO teacher, for instance, is an EMA of the student (Hübotter et al., 2026). The choice of anchor does not affect the main analysis:
On the composition of the displacement (Q2), unrolling the anchor update of Algorithm 1 gives the following.
The current RLVR update enters with weight 1, and past iterations enter with geometrically decaying weights that sum to (1−η)/η:
The anchor ablation (Table E.7) measures this trade-off.
(iii) Rollout reuse. Reusing the rollout batch is the same mild off-policyness as in standard GRPO and PPO, where each batch is reused across several mini-batch updates. App. B.1 now accounts for it in two steps:
The bound therefore holds with the loss measured as RISE computes it, plus an additive term TTV(Pπθ,Pπθn) (Eq. B.12). Empirically, regenerating rollouts after the RLVR phase removes the reuse altogether and leaves results unchanged: 62.5 vs. 62.5 Math Avg on Qwen3-8B (Table E.6).
(iv) w/o OPD. The w/o-OPD variant adopts the extrapolated weights directly, i.e., it takes a longer parameter step along the same sequence-level RLVR update. Its parity with GRPO is therefore expected. The extrapolated model itself does achieve a higher return: extrapolating saved GRPO checkpoints within β≤1.2 improves AIME'24 by 2–4 points (Fig. 6).
RISE uses this teacher differently. By Remark 1, distillation applies a dense per-token update at every visited position. The new controls show that what matters is distilling an extrapolated teacher (Table E.10):