acceptodds
Under review as a conference paper at ICLR 2027

RISE: Recursive Improvement via Self-Extrapolating Policy Distillation

Abstract

On-policy distillation (OPD) provides dense, per-token supervision for language model post-training, but its effectiveness is bottlenecked by teacher quality: external teachers suffer from distribution mismatch, while self-distillation with privileged conditioning is limited by in-context learning capacity. We propose RISE (Recursive Improvement via Self-Extrapolating Policy Distillation), which constructs a synthetic teacher directly from the model's own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor—in parameter space or output logit space—RISE converts a sparse outcome-induced parameter update into a dense token-level target, without any external model or privileged conditioning. RISE combines RLVR and OPD in a complementary loop: outcome rewards ground the extrapolation toward correct reasoning, while the extrapolated teacher refines token-level decisions. Moreover, since the teacher is refreshed every iteration as the student improves, distillation becomes a recursive improvement mechanism rather than a one-shot compression step. Experiments spanning mathematical reasoning, multi-domain STEM, code generation, and multi-turn agentic tasks show that RISE outperforms RLVR-only training and on-policy self-distillation across all settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post training, on policy distillation, reinforcement learning, self improvement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aPSvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RISE (Recursive Improvement via Self-Extrapolating Policy Distillation), a post-training framework that alternates reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD) from a synthetic "future" teacher constructed out of the model's own training trajectory (Section 3, Figure 1). At iteration nn, RISE first runs GRPO on rollouts sampled from πθn\pi _ {\theta _ n} to obtain θn+1′\theta' _ {n+1}, and then extrapolates beyond θn+1′\theta' _ {n+1} relative to a trailing anchor θanchor\theta _ {\mathrm{anchor}} with a decaying factor βn=1+(β0−1)(1−n/N)>1\beta _ n = 1 + (\beta _ 0 - 1)(1 - n/N) > 1 (Section 3.1–3.3, Eqs. 2–4). The paper instantiates this extrapolation in both weight space (θfuture=θanchor+βn(θn+1′−θanchor)\theta _ {\mathrm{future}} = \theta _ {\mathrm{anchor}} + \beta _ n(\theta' _ {n+1} - \theta _ {\mathrm{anchor}}), Eq. 3, Algorithm 2) and output-logit space (πfuture(⋅∣st)∝πθanchor(⋅∣st)1−βnπθn+1′(⋅∣st)βn\pi _ {\mathrm{future}}(\cdot \mid s _ t) \propto \pi _ {\theta _ {\mathrm{anchor}}}(\cdot \mid s _ t)^{1-\beta _ n}\pi _ {\theta' _ {n+1}}(\cdot \mid s _ t)^{\beta _ n} projected onto the top-KK tokens plus a single aggregated tail bucket, Eqs. 4 and 6, Algorithm 1), and distills sg[πfuture]\mathrm{sg}[\pi _ {\mathrm{future}}] back into πθn+1′\pi _ {\theta' _ {n+1}} on the reused pre-RLVR rollouts using token-level Jensen–Shannon divergence (JSD).

RISE is evaluated across mathematical reasoning (Qwen3-1.7B, Qwen3-8B, and Qwen3-1.7B-Base on DAPOMath; OLMo3-7B-Instruct-SFT on OpenR1-Math-46K, Table 1 and Appendix Table E.1), multi-domain math + STEM (Qwen3-4B-Base, Table 2, Figure 3), code generation (Qwen3-8B-Base, Figure 4, Appendix Figure E.2), and multi-turn agentic tasks (Qwen2.5-3B-Instruct on ALFWorld and WebShop, Table 3), alongside component ablations in Section 4.2–4.3 and Appendix E.

Critical strengths

1. Simple, external-teacher-free construction of a dense token-level distillation target.
RISE addresses a practical limitation of on-policy distillation: strong external teachers often suffer from student-prefix mismatch or vocabulary differences, while privileged-context self-distillation requires ground-truth reference solutions and only applies to prompts with at least one successful rollout. Extrapolating along the model's own training trajectory in either parameter space (Eq. 3) or top-KK logit space (Eqs. 4 and 6) requires no external model or privileged context, and can be dropped directly into existing VeRL/GRPO pipelines.

2. Consistent mathematical reasoning gains on post-trained models backed by component ablations.
On Qwen3-1.7B, Qwen3-8B (with 3-seed verification in Appendix Table E.8), and OLMo3-7B-Instruct-SFT (Table 1), RISE consistently improves in-domain and OOD math averages over GRPO and the evaluated privileged-context self-distillation baselines (GRPO+SDPO, SDAR, RLSD), raising OLMo3-7B Math Avg from 47.6 to 56.4 (+8.8) and Qwen3-1.7B from 45.4 to 50.2 (+4.8). The ablations removing RLVR (w/o RLVR) and removing OPD (w/o OPD) in Figure 5 and Table E.3 clearly show that neither random walk extrapolation nor raw parameter extrapolation without distillation suffices on its own.

Critical weaknesses

1. The theoretical justification does not match the implemented algorithm or explain why distillation works when w/o OPD fails.
The theoretical narrative in Section 3.2 and Appendix B relies on (i) bounding student suboptimality via reverse KL (Theorem 2, Eqs. B.4–B.7) and claiming that JSD(P∥Q)≤12DKL(P∥Q)\mathrm{JSD}(P \parallel Q) \le \frac{1}{2}D _ {\mathrm{KL}}(P \parallel Q) transfers this bound to JSD minimization, and (ii) analyzing 1D extrapolation from the initial checkpoint θ0\theta _ 0 under a Lipschitz reward assumption (Proposition 3 and Corollary 4, Eqs. B.8–B.13). As detailed under Soundness, the JSD-to-KL inequality is in the wrong direction to bound DKLD _ {\mathrm{KL}} from a small JSD\mathrm{JSD}; the actual Qwen implementation uses an η=0.1\eta = 0.1 EMA anchor that mixes historical OPD and RLVR steps rather than isolating the current RLVR update; the OPD step reuses off-policy pre-RLVR rollouts; and the w/o OPD ablation (Table E.3) shows that the extrapolated teacher θfuture\theta _ {\mathrm{future}} does not actually achieve a higher task return J(πfuture)J(\pi _ {\mathrm{future}}) than GRPO, contradicting the teacher-gap reduction story in Theorem 2.

2. Incomplete compute/optimization controls, confounded baseline comparisons, and an inconsistent model–task matrix.

  • Unequal wall-clock compute and missing momentum/extrapolation controls: Section 4.3 (p. 9) reports that each RISE step takes 1.3×1.3\times (1.7B) to 1.6×1.6\times (8B) the wall-clock time of GRPO. While GRPO-2x (Appendix Table E.7) takes a second inner-loop gradient pass on the same rollouts, the paper never compares against GRPO trained for equal wall-clock time / GPU-hours (i.e., 1.3–1.6×1.3\text{--}1.6\times more rollout steps), nor against weight-space momentum / Lookahead / Nesterov extrapolation or tuned GRPO learning rates. In code generation (Figure 4, p. 7), where RISE reaches GRPO's step-90 accuracy at roughly step 50 with "comparable final accuracy," a 1.6×1.6\times per-step wall-clock cost (50 * 1.6 = 80 GRPO-equivalent time units) largely eliminates the wall-clock speedup.
  • Confounded teacher-construction comparison: In Table 1, the privileged-teacher baselines (SDPO, SDAR, RLSD) only apply distillation on the ∼70%–85%\sim 70\%\text{--}85\% of prompts that produce at least one correct rollout (Appendix D.3, p. 24) and use different loss formulations (auxiliary KL, gating, or advantage reweighting), whereas RISE runs a separate JSD OPD stage on 100%100\% of prompts. There is no controlled ablation keeping the exact RISE two-stage JSD loop and replacing only πfuture\pi _ {\mathrm{future}} with the non-extrapolated post-RLVR policy πθn+1′\pi _ {\theta' _ {n+1}} (β=1\beta = 1), an EMA teacher, or a privileged teacher.
  • Inconsistent model variants across tasks and weak gains on Qwen3-1.7B-Base: Table 1 uses post-trained Qwen3-1.7B and Qwen3-8B (skipping Qwen3-4B), while relegating pretrained Qwen3-1.7B-Base to the unreferenced Appendix Table E.1, where RISE's Math Avg gain over GRPO (28.0) shrinks to +1.2 (29.2 logit) and +0.1 (28.1 weight). Meanwhile, multi-domain training (Table 2) tests only Qwen3-4B-Base; code generation (Figure 4) switches to Qwen3-8B-Base and omits all OPSD baselines; agentic tasks (Table 3) switch to Qwen2.5-3B-Instruct (where RISE (logit) trails GRPO on WS-S, 78.8 vs. 79.8) and omit all OPSD and GIGPO baselines; and key hyperparameters (β0\beta _ 0, decay schedule, and η=0.1\eta = 0.1 for Qwen vs. η=1.0\eta = 1.0 for OLMo, which swings Qwen3-1.7B Math Avg by up to 6.3 points in Table E.6) appear to be selected directly on the test benchmarks.
Questions for authors
  1. Assessment-critical (Equal wall-clock / GPU-hour comparison and controlled teacher swap inside the RISE loop):

    • Given the 1.3–1.6×1.3\text{--}1.6\times per-step wall-clock overhead of RISE (Section 4.3), how does RISE compare against GRPO trained for equal wall-clock time / total GPU-hours (i.e., 1.3–1.6×1.3\text{--}1.6\times more rollout steps) and against standard weight-space Lookahead / Nesterov momentum extrapolation?
    • In Table 1, what happens if you keep the exact RISE two-stage JSD training loop on 100%100\% of prompts, and only replace the extrapolated teacher πfuture\pi _ {\mathrm{future}} with (a) the unextrapolated post-RLVR checkpoint πθn+1′\pi _ {\theta' _ {n+1}} (β=1\beta = 1), (b) the trailing EMA checkpoint πθanchor\pi _ {\theta _ {\mathrm{anchor}}}, or (c) a privileged-context teacher?
  2. Assessment-critical (Reconciling the theory with JSD, η=0.1\eta = 0.1 EMA anchors, top-KK tail aggregation, and w/o OPD):

    • Can you provide a corrected suboptimality bound in terms of JSD(πθ∥πfuture)\mathrm{JSD}(\pi _ \theta \parallel \pi _ {\mathrm{future}}) (e.g., via TV≤2 JSD\mathrm{TV} \le \sqrt{2\,\mathrm{JSD}}) rather than relying on the reversed inequality JSD≤12DKL\mathrm{JSD} \le \frac{1}{2}D _ {\mathrm{KL}}, and account for reusing pre-RLVR rollouts y∼πθny \sim \pi _ {\theta _ n}?
    • Under the default η=0.1\eta = 0.1 EMA anchor, what fraction of the displacement θn+1′−θanchor\theta' _ {n+1} - \theta _ {\mathrm{anchor}} aligns with the current-step RLVR update versus accumulated past OPD updates? Also, why does distilling πfuture\pi _ {\mathrm{future}} improve accuracy by 3–5 points when directly evaluating the extrapolated weights (w/o OPD, Table E.3) yields almost zero gain (+0.3 on 8B, +0.2 on 1.7B, -0.3 on 1.7B-Base)?
  3. Assessment-critical (Base vs. post-trained checkpoint behavior, hyperparameter selection protocol, and unbiased pass@k):

    • Why does RISE (weight) improve over GRPO by +4.5 points on post-trained Qwen3-1.7B (Table 1) but only +0.1 points (28.1 vs. 28.0) on pretrained Qwen3-1.7B-Base (Appendix Table E.1)? Was Qwen3 thinking mode disabled during RLVR rollout training as well as evaluation (Appendix D.2)?
    • Were β0\beta _ 0, the decay schedule, and η\eta (η=0.1\eta = 0.1 for Qwen vs. η=1.0\eta = 1.0 for OLMo) tuned on a held-out validation split or selected on the reported test benchmarks? Finally, can you report the exact final numerical table for code generation (including LiveCodeBench v6) and recompute Table E.2 using the standard unbiased pass@k estimator?
3 comments
AuthorsOpenReview ↗

Response to Reviewer aPSv

We thank the reviewer for the detailed review and precise questions.

Theory and the implemented algorithm (W1, Q2)

We address the four points in turn.

(i) JSD. The main text states the analysis for reverse KL for convenience; the bound also holds for JSD directly (App. B.1, Eq. B.9). Applying Pinsker's inequality to each half of the JSD gives JSD(P,Q)≥12TV(P,Q)2\mathrm{JSD}(P,Q)\ge\frac12\mathrm{TV}(P,Q)^2. Substituting TV≤2 JSD\mathrm{TV}\le\sqrt{2\,\mathrm{JSD}} into the proof of Theorem 2 then yields J(π∗)−J(πθ)≤[J(π∗)−J(πT)]+T2 LˉJSD.J(\pi^*)-J(\pi_\theta)\le[J(\pi^*)-J(\pi_T)]+T\sqrt{2\,\bar{\mathcal L}_{\mathrm{JSD}}}. Empirically, JSD also outperforms reverse KL as the distillation loss in the RISE loop (50.2 vs. 48.7 Math Avg on Qwen3-1.7B; App. E.14).

(ii) EMA anchor. An exponential moving average of past checkpoints is a common slowly moving reference in self-distillation; the SDPO teacher, for instance, is an EMA of the student (Hübotter et al., 2026). The choice of anchor does not affect the main analysis:

  • Theorem 2 holds for any teacher.
  • The anchor enters only the idealized linear model of Proposition 3, which extends directly to EMA anchors, since an average of iterates on the trajectory also lies on it (App. B.2). Setting the anchor to the initial checkpoint recovers Proposition 3.

On the composition of the displacement (Q2), unrolling the anchor update of Algorithm 1 gives the following.

  1. Let un+1=θn+1′−θnu_{n+1}=\theta'_{n+1}-\theta_n be the current RLVR update.
  2. Let Δj=θj−θj−1\Delta_j=\theta_j-\theta_{j-1} be the full update of a past iteration, i.e., its RLVR step followed by its OPD step.
  3. Then θn+1′−θanchor  =  un+1+∑k≥1(1−η)k Δn+1−k.\theta'_{n+1}-\theta_{\text{anchor}} \;=\; u_{n+1}+\sum_{k\ge 1}(1-\eta)^k\,\Delta_{n+1-k}.

The current RLVR update enters with weight 1, and past iterations enter with geometrically decaying weights that sum to (1−η)/η(1-\eta)/\eta:

  • With η=1\eta=1, the displacement is exactly the current RLVR update.
  • With η=0.1\eta=0.1, it is a smoothed direction over roughly the last 10 iterations, with the current RLVR step as the most recent term.

The anchor ablation (Table E.7) measures this trade-off.

(iii) Rollout reuse. Reusing the rollout batch is the same mild off-policyness as in standard GRPO and PPO, where each batch is reused across several mini-batch updates. App. B.1 now accounts for it in two steps:

  • The distillation error on the student's own contexts is at most the error on the reused rollouts plus the total-variation distance between the two context distributions.
  • By data processing, that distance is at most the distance between the full-response distributions.

The bound therefore holds with the loss measured as RISE computes it, plus an additive term T TV(Pπθ,Pπθn)T\,\mathrm{TV}(P_{\pi_\theta},P_{\pi_{\theta_n}}) (Eq. B.12). Empirically, regenerating rollouts after the RLVR phase removes the reuse altogether and leaves results unchanged: 62.5 vs. 62.5 Math Avg on Qwen3-8B (Table E.6).

(iv) w/o OPD. The w/o-OPD variant adopts the extrapolated weights directly, i.e., it takes a longer parameter step along the same sequence-level RLVR update. Its parity with GRPO is therefore expected. The extrapolated model itself does achieve a higher return: extrapolating saved GRPO checkpoints within β≤1.2\beta\le1.2 improves AIME'24 by 2–4 points (Fig. 6).

RISE uses this teacher differently. By Remark 1, distillation applies a dense per-token update at every visited position. The new controls show that what matters is distilling an extrapolated teacher (Table E.10):

  • With the same distillation phase, teachers that point back toward the anchor fall below GRPO: 44.5 and 42.9, vs. 45.4.
  • RISE's extrapolated teacher reaches 50.2.

Compute controls and teacher comparison (W2, Q1)

(a) Compute and optimization controls. We now report accuracy against measured training wall-clock time (App. E.13). RISE (logit) takes 1.53× and 1.65× GRPO's time per iteration on Qwen3-8B and OLMo3-7B. Even so, it reaches the Math Avg that GRPO attains at the end of training earlier: after 11.3 vs. 14.4 hours on Qwen3-8B and 17.6 vs. 24.2 hours on OLMo3-7B. At the time GRPO finishes:

  • RISE (logit) is on par with it on Qwen3-8B.
  • RISE (logit) leads by 3.8 points on OLMo3-7B, where GRPO gains only 0.8 points over the last quarter of its training time against 2.1 for RISE.

Weight-space RISE is costlier per iteration, because its teacher is a separate set of parameters that must be materialized and run forward during distillation. Our implementation of it is not optimized for speed.

On code, the per-iteration cost is nearly identical: executing the generated programs for the reward dominates each iteration (995 vs. 1,009 seconds median). RISE (logit) reaches 80.9 HumanEval+ after 12.1 hours, a level GRPO first reaches after 23.6 hours. The iteration-count speedup therefore also holds in wall-clock time.

Regarding optimizer controls, the w/o-OPD variant (Table E.4) extrapolates the parameters along the recent RLVR direction, the operation underlying Lookahead- and Nesterov-style extrapolation. It ends at GRPO's level (45.6 vs. 45.4 on Qwen3-1.7B), whereas RISE reaches 50.2. Momentum is already present through AdamW's first moment, and all methods share the same learning rate (App. D.1). RISE leaves the optimizer unchanged, so optimizer-level variants are complementary to it.

(b) Teacher-construction controls. The privileged-teacher baselines differ from RISE in two respects: how the teacher's signal enters the update, and on which prompts a teacher exists.

  • The first is a design choice, which the controls below hold fixed: every teacher, including the privileged one, enters through the same JSD distillation phase as RISE's.
  • The second is inherent to privileged teachers. A privileged teacher requires a correct sibling solution, so it does not exist on prompts where every rollout fails, whereas RISE's teacher is defined on every prompt. This coverage gap is moderate: the privileged teacher is active on ∼70%{\sim}70\% of prompts for Qwen3-1.7B and ∼85%{\sim}85\% for Qwen3-8B (App. D.3).

Following the reviewer's design, we keep the RISE loop identical and change only the teacher (App. E.12, Table E.10, Qwen3-1.7B). Against RISE's 50.2 Math Avg, the three teachers the reviewer proposes give:

  • (a) Post-RLVR policy (β=1\beta=1): the teacher equals the student after its RLVR step, so the distillation signal vanishes and the variant reduces to GRPO (45.4).
  • (b) Trailing EMA anchor: 42.9, and a teacher interpolated toward it (β0=0.8\beta_0=0.8) reaches 44.5.
  • (c) Privileged-context teacher: it conditions the EMA anchor on a correct sibling solution as in SDPO. It reaches 41.4, below GRPO, in line with the limited gains of the privileged-teacher baselines in Table 1.

Model–task matrix, hyperparameters, and evaluation protocol (W2, Q3)

Model–task matrix. The experiments are designed for breadth: each task family tests RISE under a different reward structure, model family, or starting point. Together, they cover two model families (Qwen, OLMo), sizes from 1.7B to 8B, and base, SFT, and post-trained starting points. Running every model on every task family exceeds our compute budget, so each task family other than math is tested on one model. The OPSD baselines are compared on all five math and multi-domain settings (Tables 1, 2, E.1), while code and agentic tasks compare RISE against GRPO, the RL objective it builds on. GiGPO is likewise an RL objective, so RISE could be built on it in the same way.

Base vs. post-trained models. On Qwen3-1.7B-Base, the gain is smaller (+1.2 Math Avg for RISE (logit), +0.1 for RISE (weight); Table E.1). This smaller margin is not specific to RISE: on the base model, every method that adds a teacher signal to GRPO stays within 1.2 points of it, whereas on the post-trained model the same methods range from 2.2 below to 4.8 above:

ModelGRPOTeacher-based methods (Math Avg)
Qwen3-1.7B-Base28.0GRPO+SDPO 28.4, SDAR 26.9, RLSD 27.6, RISE 29.2 / 28.1
Qwen3-1.7B (post-trained)45.443.2–50.2

RISE (logit) remains the best method on the base model.

Agentic tasks. On WebShop, RISE (logit) trails GRPO by 1.0 point in score while improving accuracy by 4.7 points. RISE (weight) improves all three agentic metrics (Table 3).

Hyperparameters. During development, we selected hyperparameters by MATH-500 accuracy. The other five in-domain math benchmarks and the out-of-distribution benchmarks were not used for selection, and we did not hold out a separate validation split (App. D.1). The settings are:

  • β0=1.2\beta_0=1.2 and K=100K=100 (K=20K=20 for code) are shared across all settings.
  • η\eta is set per model family and shared between the logit- and weight-space variants.

RISE's gains are not concentrated on the selection benchmark. In every configuration of Table 1, the gain over GRPO averaged over the five other math benchmarks exceeds the gain on MATH-500:

ModelGain on MATH-500 (logit / weight)Avg. gain on the other five (logit / weight)
Qwen3-8B+1.0 / +0.6+2.8 / +3.1
Qwen3-1.7B−0.4 / +1.7+5.9 / +4.3
OLMo3-7B+5.4 / +5.3+9.5 / +6.3

Moreover, the gains do not hinge on the chosen values:

  • Every tested β0∈[1.2,1.5]\beta_0\in[1.2,1.5] and every decay schedule gives 61.8–63.0 Math Avg on Qwen3-8B, against 60.0 for GRPO (Table 4).
  • RISE exceeds GRPO for every anchor rate tested, including η=1\eta=1, i.e., the previous checkpoint without averaging (Table E.7). The 6.3-point spread the reviewer notes (OLMo3-7B, 56.4 vs. 50.1) is thus between two settings that both exceed GRPO's 47.6.

Thinking mode. Thinking is disabled in the chat template for all Qwen3 models, for RLVR rollouts as well as for evaluation (App. D.2).

Final code results. Table E.2 reports the final avg@4 accuracy, including LiveCodeBench v6:

MethodHumanEval+MBPP+LiveCodeBench v6
GRPO80.563.554.8
RISE (logit)81.665.755.6
RISE (weight)81.364.253.3

pass@kk. Table E.3 now uses the standard estimator of Chen et al. (2021), pass@k=Ex[1−(n−ck)/(nk)]\text{pass@}k=\mathbb{E}_x\big[1-\binom{n-c}{k}/\binom{n}{k}\big]. With n=k=16n=k=16, this is the fraction of problems solved by at least one of the 16 samples. Under this estimator, RISE improves pass@16 over GRPO in six of the eight cells and matches it in the other two (RISE (logit) on Qwen3-8B). The gains reach +10.0+10.0 on AIME'25 for both RISE (logit) on Qwen3-1.7B and RISE (weight) on Qwen3-8B.

Reviewer_crWEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RISE, a framework that alternates reinforcement learning with verifiable rewards and self-distillation from an extrapolated teacher. The teacher is constructed from the current policy and a historical anchor, using either parameter-space or output-distribution extrapolation. The central idea is to turn recent training progress into a continuously refreshed source of token-level supervision.

Critical strengths
  1. RISE addresses the challenge of obtaining dense training signals without a separately trained teacher or privileged contextual information. Constructing a teacher from the policy’s recent RL progress provides a natural connection between outcome-based reinforcement learning and token-level self-distillation.

  2. The method integrates teacher extrapolation and self-distillation into an iterative RL training loop, with implementations in both parameter and output-distribution spaces. Reusing sampled trajectories and avoiding an external teacher make the approach attractive for existing RLVR pipelines.

  3. The evaluation covers multiple model families and several task categories, including mathematics, STEM, code generation, and agent tasks. The reported improvements across these settings provide encouraging evidence that the approach is useful beyond a single model or benchmark.

Critical weaknesses
  1. Extrapolation, on-policy distillation, and self-distillation have substantial precedent. RISE’s main contribution therefore lies in its dynamically refreshed teacher and integration with ongoing RL. This is meaningful, but more direct comparisons with fixed extrapolated teachers and existing trajectory-extrapolation methods would better establish the advantage of this particular design.

  2. The ablations establish the usefulness of key components, but do not fully disentangle the benefits of extrapolation, continuous teacher refresh, and the distillation objective itself. A controlled comparison of fixed and dynamically refreshed teachers would be particularly informative.

  3. The method assumes that recent RL updates provide a useful direction to amplify. The experiments support this assumption in the evaluated settings, but its validity under noisy updates or near convergence is less clear. A more systematic analysis of when extrapolation helps or becomes unreliable would strengthen the understanding of the method’s applicability.

Questions for authors

Please refer to weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer crWE

We thank the reviewer for the positive assessment of the method's practicality and for the focused suggestions.

Fixed teachers and trajectory extrapolation (W1)

We added both comparisons.

Fixed extrapolated teacher. We keep the RISE loop identical and replace the refreshed teacher with a fixed ExOPD-style one, extrapolated from the final checkpoint of a completed GRPO run relative to the base model (App. E.12). It reaches 47.0 Math Avg on Qwen3-1.7B, above GRPO (45.4) but well below RISE (50.2). It does so even though it has seen the end of the training trajectory and requires that GRPO run before distillation begins.

Trajectory extrapolation without distillation. The paper evaluates its two uses:

  • Applied post hoc to saved GRPO checkpoints, it gains 2–4 points on AIME'24 within β≤1.2\beta\le1.2 (Fig. 6), compared with RISE's 6.3-point gain over GRPO on the same benchmark (Table 1).
  • Applied during training in place of distillation (w/o OPD), it leaves Math Avg essentially at GRPO's level: 45.6 vs. 45.4 on Qwen3-1.7B (Table E.4).

Disentangling extrapolation, refresh, and distillation (W2)

The controls in App. E.12 (Table E.10, Qwen3-1.7B) change one component at a time inside the RISE loop.

  • Extrapolation. With β=1\beta=1, the teacher equals the post-RLVR policy and the distillation signal vanishes (§1), so every gain over GRPO comes from the extrapolation. Teachers that point back along the trajectory instead of beyond it fall below GRPO's 45.4: 44.5 when interpolating toward the anchor (β0=0.8\beta_0=0.8) and 42.9 when distilling the anchor itself. Extrapolating beyond the current policy therefore adds 5.7 points over interpolation, with the largest differences on AIME'24/'25 (36.3/33.3 for RISE vs. 27.3/26.2).
  • Refresh. A fixed extrapolated teacher built from a completed GRPO run reaches 47.0, against 50.2 for RISE's refreshed teacher.
  • Distillation. Adopting the extrapolated weights directly instead of distilling them stays at GRPO's level (Table E.4). Removing RLVR, so that self-distillation alone drives the extrapolation, collapses training (Fig. 5a). A privileged-context teacher in the same loop reaches 41.4. As for the distillation objective, replacing JSD with reverse KL lowers Math Avg from 50.2 to 48.7 (Table E.11).

When extrapolation helps or becomes unreliable (W3)

We collect the relevant findings in a new App. E.9, organized by the three design choices that keep extrapolation reliable.

  • Grounding in outcome reward. When the displacement comes from self-distillation alone, repeated amplification collapses training within 60 steps (Fig. 5a).
  • Decaying β\beta near convergence. On saved GRPO checkpoints, β=2.0\beta=2.0 adds 7.3 points at step 50 but costs 15 points by step 100, while β≤1.2\beta\le1.2 never degrades accuracy (Fig. 6). Accordingly, RISE decays β\beta to 1, and every decay schedule outperforms a fixed β\beta (Table 4).
  • Averaging the direction. On Qwen3-1.7B, lowering the anchor rate from η=1\eta=1 to η=0.1\eta=0.1 raises Math Avg monotonically in both variants: from 46.2 to 50.2 (logit) and from 46.4 to 49.2 (weight). RISE improves over GRPO for every anchor rate tested (Table E.7).

A systematic study of noisy or corrupted rewards is beyond our scope, since robustness to reward noise is an open problem for RLVR itself; we note it in the Limitations. One existing experiment bears on it: the multi-domain run (Table 2) grades open-ended STEM questions, about 13% of training prompts, with a 1.5B generative verifier rather than a rule-based checker, and RISE still improves both math and STEM over GRPO.

The rebuttal partially addresses my concerns, particularly regarding extrapolation near convergence. However, the fixed-teacher comparison also changes the training trajectory and anchor, so it does not isolate the benefit of continuous teacher refresh. The generative-verifier experiment likewise does not directly test whether extrapolation amplifies noisy updates.

Consequently, I will maintain my original score.

Reviewer_nCT4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, a two-stage framework RISE (Recursive Improvement via Self-Extrapolating Policy Distillation) is proposed, which constructs a synthetic teacher directly from the model’s own RLVR training trajectory. The extrapolated policy is distilled into the post-RLVR model. It is a interesting idea. The paper reports improvements over GRPO and privileged-conditioning self-distillation baselines across mathematical reasoning, mixed math/STEM, code generation, and agentic tasks.

Critical strengths
  1. The central idea is simple, and potentially useful.
  2. The paper is unusually thorough experimentally for a methods paper. The theory analysis is given to support RISE.
  3. The discussion of extrapolation decay and the empirical evidence that aggressive extrapolation becomes less safe late in training are useful practical observations.
Critical weaknesses
  1. The novelty is limited with a simple self-extrapolating policy.
  2. The main theory result is a generic teacher-gap plus distillation-error decomposition for an arbitrary teacher. The subsequent guarantee assumes that the trajectory is exactly linear in a representation space, that the optimum lies on the same ray, and that return is Lipschitz in that representation. These assumptions are especially difficult to justify in weight space because neural-network parameterizations are highly non-identifiable, and in logit space because the representation covers a very large collection of context-dependent distributions.
  3. The logit-space top-KK approximation is insufficiently validated.
  4. Most of ODP methods are not compared directly.
  5. The strongest multi-seed analysis is limited to two Qwen math configurations. The multi-domain, code, and agentic results appear to be single runs, without confidence intervals or seed variation.
Questions for authors
  1. Can you implement a direct ExOPD-style baseline under the same RLVR trajectory, checkpoints, data, and compute budget?
  2. What is the OPD/JSD loss coefficient relative to the RLVR update, and which optimizer details are used beyond the reported learning rate?
  3. Can you derive a correct performance bound for the actual JSD objective, or experimentally compare reverse KL and JSD under matched settings?
  4. How does RISE perform under several values of KK, and how often does the approximate logit-space teacher differ materially from the exact extrapolated distribution?
  5. Does the method still help when RLVR updates are noisier, rewards are partially corrupted, or the reward is less reliably verifiable?
3 comments
AuthorsOpenReview ↗

Response to Reviewer nCT4

We thank the reviewer for the careful reading, especially of the theory.

Novelty (W1)

As the reviewer notes, the central idea is simple, and we regard this as a strength: RISE drops into a GRPO pipeline with no external model, privileged context, or extra sampling. Its novelty lies in where the teacher comes from. Existing OPD methods take the teacher as given, and each source has a limitation:

  • External teachers are queried on prefixes they never generated.
  • Privileged-context self-distillation (SDPO, SDAR, RLSD) needs a correct rollout and presumes in-context learning ability the model may lack.
  • ExOPD, whose optimal policy is the same geometric mixture as our logit-space teacher, extrapolates from a fixed teacher. As a result, the teacher-gap term in Theorem 2 stays constant throughout training.

RISE instead builds the teacher from the model's own RLVR trajectory and rebuilds it every iteration, so the teacher advances with the student. Rank-1 trajectory extrapolation (Cai et al., 2025; Wei et al., 2026) also extrapolates RLVR weights, but as a one-time edit of a trained checkpoint. RISE, in contrast, turns the extrapolation into dense token-level supervision at every training step.

The new controls (Table E.10, Qwen3-1.7B) show that the gain comes from this teacher rather than from adding a distillation phase. Inside the identical loop, a fixed ExOPD-style teacher reaches 47.0 Math Avg, against 45.4 for GRPO and 50.2 for RISE. Teachers that point back along the trajectory fall below GRPO: 44.5 when interpolating toward the anchor (β0=0.8\beta_0=0.8) and 42.9 when distilling the anchor itself.

Theoretical assumptions (W2)

We agree that exact linearity is hard to justify in either weight or logit space, which is why we use it only as an idealized model behind Proposition 3 and Corollary 4. Theorem 2 holds for any teacher and makes none of these assumptions; we now state this scope at the start of App. B.2. The linear model serves to explain why a safe extrapolation range exists and why it narrows as training converges, which motivates the decaying β\beta. As Remark 2 notes, the resulting bounds characterize the mechanism rather than give exact rates.

We therefore test the model's predictions empirically:

  • Three directions capture ∼87%{\sim}87\% of the RLVR weight trajectory (App. B.3).
  • Extrapolating saved GRPO checkpoints gains 2–4 points within β≤1.2\beta \le 1.2, while the tolerable range shrinks over training (Fig. 6).

Top-KK approximation (W3, Q4)

We validate the approximation in three ways (App. B.4).

(i) Where the error can arise. On the retained tokens, the approximate and exact teachers assign the same probabilities up to a common rescaling, so any discrepancy is confined to the tail bucket. Throughout training, the K=100K=100 retained tokens hold at least 99.4% (Qwen3-1.7B) and 99.8% (OLMo3-7B) of the mass of both the post-RLVR policy and the extrapolated teacher. The new Lemma 1 bounds the exact teacher's tail mass by ε ρˉ β−1\varepsilon\,\bar\rho^{\,\beta-1}, where:

  • ε\varepsilon is the post-RLVR tail mass, and
  • ρˉ\bar\rho is the average factor by which RLVR raised tail tokens relative to the anchor.

With β≤1.2\beta\le1.2, the exact tail therefore stays below 2.4% even if RLVR raised tail tokens 1000-fold on average.

(ii) Direct measurement. We computed the exact full-vocabulary teacher alongside the top-KK teacher at every response token (Table B.1). To make the test conservative, we used checkpoint pairs whose displacement exceeds a single RISE iteration: GRPO checkpoints 50 iterations apart and, as a stress test, the base model paired with a final checkpoint. At β=1.2\beta=1.2 and K=100K=100, the results are:

  • the mean total-variation distance is at most 1.1×10−41.1\times10^{-4},
  • it exceeds 10−210^{-2} at no more than 0.02% of positions, and
  • the two teachers pick the same top-1 token at over 99.9% of positions.

(iii) Other values of KK. Retraining RISE (logit) on Qwen3-1.7B with K=20K=20 gives 49.5 Math Avg, close to 50.2 at K=100K=100 and well above GRPO's 45.4 (Table B.2). We also show in App. B.4 how computing the loss on the top-KK simplex enters the suboptimality bound: it adds at most TT times the average tail mass.

OPD baselines and ExOPD (W4, Q1)

Most OPD methods distill from an external, stronger teacher, which is the dependency RISE is designed to remove. The main comparison therefore uses methods with the same constraint as RISE: three self-distillation methods with a privileged-context teacher (Table 1).

ExOPD-style baseline (Q1). We implemented it inside the identical RISE loop on Qwen3-1.7B (App. E.12):

  • The teacher is built ExOPD-style by extrapolating the final checkpoint of a completed GRPO run away from the base model.
  • It is held fixed throughout training.
  • Everything else matches RISE: data, RLVR phase, JSD distillation phase, top-KK, and number of iterations.

This teacher has seen the end of the training trajectory, yet it reaches 47.0 Math Avg, well below RISE's 50.2 (GRPO: 45.4). Moreover, it requires a completed GRPO run before distillation begins.

External-teacher OPD and ExOPD. For reference, we distill an external Qwen3-8B teacher with standard OPD and with ExOPD, without an RLVR phase to further adapt the teacher in the target domain. These reach 32.1 and 31.7 Math Avg (Table E.10). They depend on a separately trained model 4.7× larger than the student, so they are not directly comparable to the controlled rows. Their limited benefit is consistent with prior findings on off-the-shelf external teachers (Li et al., 2026c).

Seeds and confidence intervals (W5)

Running multiple seeds for every configuration exceeds our compute budget, since each run takes one to two days on a full 8-GPU node. We therefore concentrated seeds on the two main configurations. There, three seeds give a Math Avg standard deviation of at most 0.6 points, well below RISE's gains (Table E.9):

  • Qwen3-8B: 62.4 ± 0.2 for RISE (weight) vs. 60.1 ± 0.2 for GRPO.
  • Qwen3-1.7B: 49.6 ± 0.4 for RISE (logit) vs. 45.1 ± 0.4 for GRPO.

The single-run settings point the same way. Across the five math and multi-domain configurations, each a separate pair of training runs, both RISE variants improve over GRPO in every one: on Math Avg for Qwen3-8B, Qwen3-1.7B, OLMo3-7B, and Qwen3-1.7B-Base (Tables 1, E.1), and on both Math Avg and STEM Avg for the multi-domain Qwen3-4B-Base run (Table 2). Under the null hypothesis that a variant wins each configuration with probability 0.5, independently across configurations, a one-sided sign test gives p=1/32≈0.031p = 1/32 \approx 0.031 for each variant, significant at the 5% level. At the benchmark level, RISE (logit) outperforms GRPO in 30 of the 34 benchmark–configuration cells and RISE (weight) in 32; we report these counts descriptively, since benchmarks within a configuration share a training run. The gains are also not confined to small competition sets: on OLMo3-7B, RISE (logit) improves the two largest benchmarks, MATH500 and OlympiadBench, by 5.4 and 8.4 points. The code and agentic results remain single runs; on the agentic tasks, the margins are large, with RISE (weight) improving ALFWorld by 9.4 points and WebShop accuracy by 10.9 points over GRPO (Table 3).

Loss coefficient and optimizer (Q2)

The RLVR and OPD phases are optimized as separate steps on the same rollouts, rather than as one weighted objective (App. D.1). The optimizer and loss settings are:

  • Optimizer: a single AdamW instance with β1=0.9\beta_1=0.9, β2=0.999\beta_2=0.999, weight decay 0.01, and gradient-norm clipping at 1.0. Its moment estimates are shared across both phases, at one constant learning rate of 10−610^{-6} with no warmup.
  • RLVR phase: the GRPO clipped surrogate with clip range [0.8,1.28][0.8, 1.28], token-mean aggregation, and no KL or entropy term.
  • OPD phase: the JSD loss of Eq. 7, averaged over response tokens and multiplied by a per-token importance ratio to the rollout policy, clipped at 2.

JSD objective: bound and experiment (Q3)

We now derive the bound for the JSD objective directly (§3.2, App. B.1). The main text states Theorem 2 for reverse KL for convenience.

  1. With M=12(P+Q)M=\frac12(P+Q), Pinsker's inequality applied to each half of the JSD gives JSD(P,Q)≥12TV(P,Q)2\mathrm{JSD}(P,Q)\ge\frac12\mathrm{TV}(P,Q)^2, i.e., TV≤2 JSD\mathrm{TV}\le\sqrt{2\,\mathrm{JSD}}.
  2. Using this in place of Pinsker's inequality, with the same Jensen and Cauchy–Schwarz steps, gives J(π∗)−J(πθ)  ≤  [J(π∗)−J(πT)]+T2 LˉJSD(Eq. B.9).J(\pi^*)-J(\pi_\theta)\;\le\;\big[J(\pi^*)-J(\pi_T)\big]+T\sqrt{2\,\bar{\mathcal L}_{\mathrm{JSD}}}\qquad\text{(Eq. B.9).}
  3. Only the distillation-error term changes. Near agreement, JSD≈14KL\mathrm{JSD}\approx\frac14\mathrm{KL}, so this term coincides to leading order with Theorem 2's TLˉOPD/2T\sqrt{\bar{\mathcal L}_{\mathrm{OPD}}/2}.

We also compare the two losses experimentally (App. E.14, Table E.11). Keeping the RISE (logit) loop on Qwen3-1.7B and replacing JSD with reverse KL lowers Math Avg from 50.2 to 48.7, and JSD is ahead on five of the six benchmarks.

Noisy or less verifiable rewards (Q5)

A systematic study of noisy or corrupted rewards is beyond the scope of this work, since robustness to reward noise is an open problem for RLVR itself. RISE inherits whatever reward its RLVR phase uses; we state this in the Limitations. Two design choices limit how far a poor direction can propagate (App. E.9):

  • the extrapolation scale decays to β=1\beta=1, so RISE reverts to GRPO by the end of training;
  • the EMA anchor averages the direction over recent iterations.

The multi-domain experiment (Table 2) also trains with a partly model-verified reward: math and multiple-choice STEM questions are scored by rule-based checkers, while open-ended STEM questions (about 13% of training prompts) are graded by a 1.5B generative verifier (TIGER-Lab/general-verifier). RISE still improves both math and STEM over GRPO in this setting (Math Avg 44.8 vs. 40.2 and STEM Avg 47.5 vs. 45.5 for RISE (weight)).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their careful and constructive reviews. The revised PDF marks all changes in blue. The main additions are:

  • Controlled teacher comparisons (App. E.12, Table E.10). Inside the identical RISE loop, we replace only the teacher with a fixed ExOPD-style teacher, the trailing EMA anchor, a teacher interpolated toward the anchor, or a privileged-context teacher. For reference, we also report OPD and ExOPD with an external Qwen3-8B teacher.
  • Theory for the implemented objective (App. B.1, B.2). This covers the suboptimality bound for the JSD loss, the cost of reusing pre-RLVR rollouts, and general EMA anchors.
  • Top-KK fidelity (App. B.4). We add a bound on the tail mass, a direct measurement of the approximate vs. exact teacher (Table B.1), and a K=20K=20 run (Table B.2).
  • Accuracy vs. training wall-clock time for math and code (App. E.13).
  • Reverse KL vs. JSD (App. E.14, Table E.11), an experimental comparison of the two distillation losses inside the RISE loop.
  • Evaluation and protocol details: the final code results with LiveCodeBench v6 (Table E.2), pass@16 recomputed with the standard estimator (Table E.3), the hyperparameter-selection protocol (App. D.1), and thinking mode (App. D.2).

Since new tables were inserted, the appendix tables after E.1 are renumbered. In the revision, the pass@kk table is E.3, the w/o-OPD ablation E.4, the resample ablation E.6, the anchor ablation E.7, the compute-matched comparison E.8, and the multi-seed table E.9. Proposition 3 and Corollary 4 are now Eqs. B.13–B.18. We use the new numbers throughout.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.