acceptodds
Under review as a conference paper at ICLR 2027

The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation

Abstract

On-policy distillation (OPD) improves student models by aligning the token distributions induced by the student and teacher logits on student-generated trajectories, and has demonstrated substantial empirical gains. Existing generalized OPD methods enable the student to surpass the teacher through reward extrapolation in the output space. However, the language-model head constitutes an information bottleneck: it discards information encoded in the teacher's intermediate hidden states and propagates sampling noise from token-level log-probability ratios into the amplified learning signal, making output-space extrapolation unstable.We observe that reinforcement learning induces a distributional shift from the base model to the RL-trained model, and that the direction of this shift can be measured throughout the network. Motivated by this observation, we propose **RIDE** (**R**L-**I**nduced **D**irection **E**xtrapolation), which extrapolates RL-induced changes directly in the model's internal representation space.At every layer and token position, RIDE computes an RL-induced representation residual and trains the student through layerwise regression toward targets displaced along this residual. Conditioned on a sampled trajectory, we show that regression toward these targets is equivalent to maximizing a linear directional reward defined by the residual, subject to a quadratic penalty centered at the teacher representation. This interpretation makes explicit how the objective encourages the student to move along the direction of the changes induced by reinforcement learning while penalizing excessive deviation from the teacher.Experiments demonstrate that RIDE substantially outperforms OPRD and matches or even surpasses the teacher model.

Primary area:
reinforcement learning
Keywords:
on-policy distillation, large language model post-training, knowledge distillation, hidden-state alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nLBuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes RIDE, an on-policy representation distillation method that treats the change induced by RL as a direction to extrapolate. On student-generated prefixes, the method computes the layerwise hidden-state difference between an RL-trained teacher and its pre-RL checkpoint, then trains the student toward targets displaced beyond the teacher along this residual. The student starts from the same base checkpoint, and setting the extrapolation coefficient to one recovers ordinary representation matching. Across four base/teacher pairs on mathematical reasoning benchmarks, RIDE improves over OPRD and achieves mean scores close to or slightly above the teacher. The paper also investigates head attenuation, conditional sampling variance, and alternative residual directions to explain the observed gains.

Critical strengths
  1. The method is simple and clearly defined. RIDE makes a concrete change to representation distillation by extrapolating the teacher-to-base residual. Recovering OPRD at λ=1 provides a useful reference point, and the reward-and-quadratic-penalty interpretation helps explain the objective.

  2. The empirical improvements over OPRD are encouraging. RIDE improves the reported macro-average score on all four model pairs, with gains ranging from 0.97 to 4.06 points. The use of three independent training seeds and multiple model families strengthens the evidence beyond a single-model result.

  3. The paper includes useful diagnostic experiments and acknowledges important limitations. The coefficient sweep and random, reversed, mismatched-origin, and trajectory-mismatched direction controls are relevant to the proposed explanation. I also appreciate the explicit acknowledgment that most margins over the teacher are small relative to training-seed variation, and that the experiments are limited to mathematical reasoning and a shared-initialization setting.

Critical weaknesses
  1. The evidence for reliably surpassing the teacher and for direction-specific gains remains limited. On three model pairs, the margin over the teacher is only 0.32–0.48 points, smaller than the reported training-seed standard deviation. The teacher’s evaluation sampling uncertainty is also not reported. In Table 3, RIDE exceeds the strongest direction control by 1.26 points, but the controls’ seed counts and uncertainty are unspecified. The mean improvements are worth reporting, but stronger conclusions require uncertainty estimates for the relevant differences.

  2. The ExOPD comparison does not establish performance under equally thorough tuning. The main table fixes λ=1.25, which is selected as the best value for RIDE. ExOPD’s extrapolating coefficients are comparatively coarse, while the additional values 1.15 and 1.35 are tested only for RIDE. This is particularly relevant because Appendix F places ExOPD’s conditional-variance minimum near λ=1.14. The paper also uses the reported benchmarks for validation curves and coefficient selection without clearly identifying an independent selection set. These choices complicate the interpretation of both the method comparison and the small gains over the teacher.

  3. The stability analysis does not fully separate representation-space supervision from the choice of gradient estimator. RIDE uses deterministic layerwise regression, whereas ExOPD uses a sampled-token update. A full-vocabulary output-space objective would also be deterministic at a fixed prefix, so zero conditional next-token variance is not unique to representation-space training. Moreover, the fitted polynomial in Appendix F gives an advantage variance of approximately 0.00864 at λ=1.25, below 0.0096 at λ=1 on those fixed prefixes, even though ExOPD already underperforms at λ=1.25. The analysis therefore does not by itself explain the main performance gap, and advantage variance is not equivalent to parameter-gradient variance.

  4. The claimed relationship between the actual RIDE target and output-space extrapolation needs clarification. Equation (8) is a linear identity at the head input, but the implemented loss supervises pre-normalization residual states. Final normalization generally does not commute with affine extrapolation. Appendix E discusses this distinction, but it remains unclear whether the reported near-machine-precision agreement uses the normalized actual RIDE target or an affine combination of already-normalized states. The reported head drift also produces a non-negligible change in logits, despite its small relative weight norm. These details matter for the claim that the methods differ only in the space where extrapolation is performed.

Questions for authors
  1. How were λ and the optimization settings selected? Were AIME24, AIME25, and AIMO used both for selection and final reporting? Could ExOPD be evaluated with modest extrapolation coefficients such as 1.05, 1.10, and 1.15 under the same selection budget?

  2. How much of RIDE’s advantage remains against a lower-variance output-space extrapolation objective? A full-vocabulary or sufficiently supported approximation would help separate the effect of the estimator from the effect of the supervision space. How should the variance result at λ=1.25 be reconciled with ExOPD’s degradation at that coefficient?

  3. What exactly is compared in the Equation (8) diagnostic? Does the reported error use the actual pre-normalization RIDE target after final normalization, or the affine combination of normalized teacher and base states? Please also clarify how teacher-head drift affects the correspondence to the native ExOPD target.

  4. Could you provide results on additional mathematical reasoning benchmarks? The evaluation is limited to AIME24, AIME25, and AIMO, and I am concerned about potential exposure to these benchmarks during pre-training or post-training. Results on additional, preferably more recent or independently held-out benchmarks would help establish whether the improvements generalize beyond these three datasets.

7 comments
AuthorsOpenReview ↗

Response to Reviewer nLBu (1/5): W1

Thank you for reading the experiments and derivations so closely, and for recognising the simplicity of the method and the consistency of its gains across four model families. Your questions prompted four new experiments: direction controls with full uncertainty, a finer ExOPD sweep, a deterministic output-space baseline, and two held-out benchmarks. Each supports the paper's main result. All intervals below are unpaired Welch 95% CIs over training seeds.

W1. How reliable are the gains over OPRD, the comparison with the teacher, and the direction controls?

RIDE improves over OPRD on all four pairs, and the aggregate improvement is statistically reliable. Original benchmarks, macro Avg@16 (%), mean ± SD over three seeds.

PairOPRDRIDERIDE − OPRD [95% CI]
R1-Distill-1.5B54.50 ± 1.0456.38 ± 0.97+1.88 [−0.40, 4.16]
Qwen3-4B62.01 ± 0.9666.07 ± 0.88+4.06 [1.97, 6.15]
Llama-3.2-3B10.94 ± 0.9013.33 ± 1.13+2.39 [0.03, 4.75]
Phi-4-mini17.33 ± 1.0618.30 ± 1.05+0.97 [−1.42, 3.36]
Mean of the four pairs——+2.33 [1.46, 3.19]

The improvement is positive on every pair, and the four-pair mean has an interval well clear of zero. The same holds out of sample (5/5).

RIDE closes the gap to the teacher that teacher matching leaves. The teacher is a single checkpoint, so we bound its decoding standard error from the binomial variance of its Avg@16 scores on the fixed problems.

PairBaseTeacherOPRDRIDERIDE − TeacherTeacher decoding SE (upper bound)Base-to-teacher gain recovered: OPRD / RIDE
R1-Distill-1.5B39.0055.3054.5056.38+1.081.1295% / 107%
Qwen3-4B62.8265.5962.0166.07+0.481.11−29% / 117%
Llama-3.2-3B6.9913.0110.9413.33+0.320.6666% / 105%
Phi-4-mini15.1317.9617.3318.30+0.340.7578% / 112%

RIDE's margin over the teacher is within the teacher's decoding SE bound on every pair, so we describe RIDE as matching its teacher and do not claim a significant margin above it. What the table establishes is the gap that matching leaves and extrapolation closes: OPRD stays below its teacher on all four pairs, and RIDE reaches the teacher on all four.

Direction controls. Every entry of Table 3 is a mean over three training runs. All displaced targets keep the norm of the original residual at each layer and position, with λ = 1.25 and loss weight 0.64. We added teacher matching at that same loss weight. R1-Distill-1.5B, macro Avg@16 (%).

TargetMean ± SDRIDE − target [95% CI]
RIDE (matched RL residual)56.38 ± 0.97—
Mismatched trajectory55.12 ± 1.00+1.26 [−0.97, 3.49]
Random direction55.04 ± 0.79+1.34 [−0.70, 3.38]
Teacher matching (weight 0.64)54.85 ± 0.91+1.53 [−0.60, 3.66]
Mismatched base54.75 ± 0.77+1.63 [−0.39, 3.65]
OPRD (λ = 1, weight 1)54.50 ± 1.04+1.88 [−0.40, 4.16]
Reversed residual53.90 ± 1.02+2.48 [0.23, 4.73]
  • The matched RL direction has the highest mean, and it is the only direction that lifts the student clearly above teacher matching. The three unrelated directions of the same norm land at the teacher-matching level, so displacing the target is not what helps.
  • The sign matters: the reversed direction is 2.48 points below RIDE, and that interval excludes zero. The other gaps also favour the matched direction in mean; with three runs their intervals include zero, as you anticipated for the 1.26-point gap.
  • Loss rescaling is ruled out: teacher matching at weight 0.64 is level with OPRD.
  • Two of RIDE's three runs are above all 18 runs of the six alternative targets.

The revision reports this table in place of Table 3 and states the conclusion as the ordering it shows: RL direction > unrelated directions ≈ teacher matching > reversed.

AuthorsOpenReview ↗

Response to Reviewer nLBu (2/5): W2, Q1

Continuation of our response to Reviewer nLBu.

W2 / Q1. Was ExOPD tuned as thoroughly as RIDE, and how were λ and the optimization settings selected?

Optimization settings. All methods share one configuration: AdamW with peak learning rate 10−510^{-5}, 3% warmup, cosine decay and 500 steps (Table 5). Nothing was tuned per method. The factor λ−2\lambda^{-2} in the RIDE loss only matches the initial loss scale (Remark G.1).

ExOPD at the coefficients you proposed. We ran ExOPD at λ = 1.05, 1.10 and 1.15, close to the variance minimum of Appendix F. R1-Distill-1.5B, Avg@16 (%), one run per row.

ExOPD λAIME24AIME25AIMOMacro
1.00 (= OPD top-1; submission)47.133.578.152.9
1.05 (new)46.4633.7578.6952.97
1.10 (new)48.5432.7176.2052.48
1.15 (new)44.3833.5478.3152.08
1.25 (submission)44.229.675.849.9

To give ExOPD every advantage, the three new rows use the best checkpoint on these benchmarks; the submission rows use the final checkpoint.

Two conclusions follow.

  • RIDE's advantage holds at ExOPD's best coefficient. ExOPD peaks at 52.97, which is 3.4 points below RIDE (56.38 ± 0.97) and also below OPRD (54.50). The ranking in Table 1 is therefore not an artifact of fixing λ = 1.25. The revised Table 1 reports ExOPD at its best coefficient, and Table 2 includes the finer sweep.
  • No coefficient makes extrapolation help in output space. No λ > 1 improves on λ = 1 by more than 0.1 points, and λ ≥ 1.10 is lower. Extrapolation helps in representation space and not in output space, which is the paper's central point. The deterministic baseline in (3/5) confirms this with sampling noise removed.

How λ was selected. We chose RIDE's λ = 1.25 once, on the R1-Distill-1.5B pair, using the benchmarks reported in Table 2 (AIME24, AIME25, AIMO). It was never re-tuned. Most of the evidence therefore lies outside the selection: the three remaining pairs, where the untuned coefficient improves over OPRD by +4.06, +2.39 and +0.97 points, and the two newer benchmarks in (5/5), which were used only for evaluation. RIDE is also insensitive to the coefficient: every value in [1.15, 1.35] improves over OPRD in the sweep of Table 2. The revision states this selection protocol explicitly in Section 5.1.

AuthorsOpenReview ↗

Response to Reviewer nLBu (3/5): W3, Q2

Continuation of our response to Reviewer nLBu.

W3 / Q2. How much of RIDE's advantage remains against a deterministic output-space objective, and how does the variance result at λ = 1.25 fit?

We ran the experiment you proposed, and it separates the two factors cleanly.

Setup. We keep the output target fixed and compute reverse KL exactly over the whole vocabulary: KL(p∥p⋆)\mathrm{KL}(p \parallel p^{\star}) at every position, toward the same p⋆∝qλb1−λp^{\star} \propto q^{\lambda} b^{1-\lambda} as ExOPD. At a fixed prefix this update, like the RIDE update, is independent of the next-token draw. R1-Distill-1.5B, macro Avg@16 (%), three runs per row. Sampled-token means are from Table 1; the full-vocabulary runs use seeds 42, 43 and 44.

ObjectiveλMean ± SDRIDE − row [95% CI where available]
Sampled KL (OPD top-1)1.0052.90+3.48
Sampled KL (ExOPD)1.2549.87+6.51
Full-vocabulary KL1.0053.89 ± 0.90+2.49 [0.37, 4.62]
Full-vocabulary KL1.2553.77 ± 0.76+2.61 [0.59, 4.63]
OPRD1.0054.50 ± 1.04+1.88 [−0.40, 4.16]
RIDE1.2556.38 ± 0.97—
  • Estimator noise is what breaks ExOPD. Removing sampling lifts λ = 1.25 from 49.87 to 53.77, a gain of 3.90 points. This confirms the paper's claim that sampled-token noise destabilises output-space extrapolation.
  • The supervision space accounts for the rest. Against the deterministic objective, RIDE remains 2.61 points above output-space extrapolation ([0.59, 4.63]) and 2.49 points above output-space matching ([0.37, 4.62]). Both intervals exclude zero.
  • Extrapolation helps only in representation space. Within the full-vocabulary objective, moving from λ = 1 to 1.25 changes the mean by −0.11 points ([−2.03, 1.80]); in representation space the same move gives +1.88. At λ = 1 the two spaces perform similarly (53.89 and 54.50). The difference emerges when the target moves beyond the teacher, which is the thesis of the paper.

So the answer to your first question is that RIDE's advantage over output-space extrapolation remains, with an interval that excludes zero, once the estimator is made deterministic.

On the variance at λ = 1.25. You are right. In Eq. (9) the negative covariance term brings the advantage variance down to 0.00864 at λ = 1.25, from 0.0096 at λ = 1, on the prefixes of Appendix F; it rises at larger coefficients. That fit is a local measurement: it uses fixed prefixes from a trained student at step 500, where the student is already close to the teacher. At initialization the student equals the base, so the advantage reduces to −λρ-\lambda\rho and its variance scales as λ2\lambda^2, a factor of 1.56 at λ = 1.25. The step-500 fit therefore does not rank coefficients by the noise seen during training, and, as you note, the parameter-gradient variance also depends on the score function.

The decisive test is the one above: for the output baselines, the difference between estimators at λ = 1.25 (3.90 points) is far larger than the change across coefficients under exact KL (0.11 points). In the revision, Proposition 4.1 is presented as a statement about estimator noise, the Appendix F fit is described as a local diagnostic, and the causal claim rests on the full-vocabulary experiment, which is added to Table 1.

Reviewer_EaCjOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how a student trained via On-Policy Distillation (OPD) can not only imitate an RL-trained teacher, but potentially surpass it. The authors propose RIDE (RL-Induced Direction Extrapolation). Its core hypothesis is that when a teacher is obtained by training a base model with RL, the difference in hidden representations between the teacher and the base model provides a meaningful direction for improvement.

Critical strengths

Key Strengths Practical Motivation & Simple Design: Extracting further capabilities from existing RL-trained models is a highly practical problem. The proposed method is simple and naturally extends the existing OPRD framework. The key modification in RIDE is shifting the representation-level supervision target, requiring neither training an additional reward model nor continuing RL on the teacher. Thorough Mechanistic & Empirical Analysis: The paper connects method design with empirical phenomena through spectral analysis of the LM head, conditional variance analysis, tracking response length and format scores, and sensitivity analysis across different λ values.

Critical weaknesses
  • Unconfounded Mechanistic Factors: The authors attribute RIDE's advantage over ExOPD to two main factors:

    1. Hidden representations preserve information that would otherwise be attenuated by the LM head.
    2. Representation regression avoids the high variance of sampled-token log-ratios. However, the current experiments do not sufficiently disentangle the individual contributions of these two factors.
  • Statistical Significance & Limited Ablation Scope: Each setting uses only three training seeds. Except for R1-Distill-1.5B, the margin by which RIDE surpasses the teacher on the other three model pairs is smaller than one standard deviation of a single run. Furthermore, all ablation and analytical experiments are conducted solely on R1-Distill-1.5B.

  • Missing System Efficiency & Compute-Matched Comparisons: While per-rollout cost analysis is provided, the paper does not report total training wall-clock time, GPU hours, peak memory usage, or performance comparisons under an equal-compute budget.

Questions for authors

Please see weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer EaCj (1/3): W1

Thank you for the encouraging review, and for highlighting the practical motivation, the simplicity of the design, and the mechanistic analysis. Each of your three points prompted a new experiment: a deterministic output-space baseline that separates the two mechanisms (W1), aggregate statistics and two held-out benchmarks across all four model pairs (W2), and wall-clock and memory measurements (W3). All three strengthen the paper, and we report them below. Intervals are unpaired Welch 95% CIs over training seeds.

W1. Can the contributions of the two factors, representation-space supervision and estimator variance, be separated?

Yes. We added the missing condition between ExOPD and RIDE: a full-vocabulary reverse-KL objective that computes KL(p∥p⋆)\mathrm{KL}(p \parallel p^{\star}) exactly over the vocabulary, toward the same extrapolated target p⋆∝qλb1−λp^{\star} \propto q^{\lambda} b^{1-\lambda} as ExOPD. It stays in output space but removes sampling: at a fixed prefix this update, like the RIDE update, is independent of the next-token draw. Comparing it with ExOPD therefore isolates the estimator, and comparing it with RIDE isolates the supervision space. R1-Distill-1.5B, macro Avg@16 (%), three runs per row; the sampled-token mean is from Table 1 and the full-vocabulary runs use seeds 42, 43 and 44.

Method (λ = 1.25)SpaceEstimatorMacro Avg@16Gain over previous row
ExOPDOutputSampled token49.87—
Full-vocabulary reverse KLOutputDeterministic53.77 ± 0.76+3.90 (estimator variance)
RIDERepresentationDeterministic56.38 ± 0.97+2.61 [0.59, 4.63] (supervision space)

The 6.51-point gap between ExOPD and RIDE splits into two parts:

  • Estimator variance: +3.90 points (60%). Removing sampling alone lifts output-space extrapolation from 49.87 to 53.77. This confirms the paper's second factor.
  • Supervision space: +2.61 points (40%). With both objectives deterministic, RIDE is still 2.61 points above output-space extrapolation, and the interval [0.59, 4.63] excludes zero. This confirms the paper's first factor.

Both mechanisms proposed in the paper are therefore real, and each accounts for a substantial share of RIDE's advantage.

The same decomposition at λ = 1 (plain teacher matching) shows why both factors matter specifically for going beyond the teacher:

λ = 1 (matching)λ = 1.25 (extrapolation)
Sampled-token, output space52.9049.87
Deterministic, output space53.89 ± 0.9053.77 ± 0.76
Deterministic, representation space54.50 ± 1.0456.38 ± 0.97
Estimator contribution+0.99+3.90
Supervision-space contribution+0.61+2.61

Both contributions are about four times larger under extrapolation than under matching. Extrapolation also helps only in representation space: it changes the deterministic output-space objective by −0.11 points and the representation objective by +1.88. This is the central claim of the paper, that the RL-induced change is a direction worth extrapolating in hidden-state space, and it now rests on a controlled comparison.

The head diagnostic gives a complementary view of the supervision-space contribution. The 512 directions that the head attenuates most carry 79.8% of the RL residual energy. At the head input they receive 73.0% of the representation-loss gradient energy but only 48.6% of the output-KL gradient energy (Figure 5(a), Appendix E). An output-space loss, even a deterministic one, thus supervises most of the residual weakly, whereas RIDE supervises it directly, and at every layer.

The revision adds this baseline to Table 1 and the decomposition to Section 5.4.

AuthorsOpenReview ↗

Response to Reviewer EaCj (2/3): W2

Continuation of our response to Reviewer EaCj.

W2. Are the gains statistically reliable with three seeds, and do the findings extend beyond R1-Distill-1.5B?

The improvement over OPRD is reliable in aggregate. Original benchmarks, macro Avg@16 (%), mean ± SD over three seeds:

PairOPRDRIDERIDE − OPRD95% CI
R1-Distill-1.5B54.50 ± 1.0456.38 ± 0.97+1.88[−0.40, 4.16]
Qwen3-4B62.01 ± 0.9666.07 ± 0.88+4.06[1.97, 6.15]
Llama-3.2-3B10.94 ± 0.9013.33 ± 1.13+2.39[0.03, 4.75]
Phi-4-mini17.33 ± 1.0618.30 ± 1.05+0.97[−1.42, 3.36]
Average of the four pairs——+2.33[1.46, 3.19]

RIDE improves on every pair, and the four-pair average of +2.33 points has an interval well clear of zero.

RIDE recovers the full teacher performance; teacher matching does not.

PairBaseTeacherOPRDRIDERIDE − TeacherTeacher decoding SE (upper bound)Base-to-teacher gain recovered: OPRD / RIDE
R1-Distill-1.5B39.0055.3054.5056.38+1.081.1295% / 107%
Qwen3-4B62.8265.5962.0166.07+0.481.11−29% / 117%
Llama-3.2-3B6.9913.0110.9413.33+0.320.6666% / 105%
Phi-4-mini15.1317.9617.3318.30+0.340.7578% / 112%

The teacher is a single checkpoint, so the table gives an upper bound on its decoding standard error, from the binomial variance of its Avg@16 scores. RIDE's margin over the teacher is within that bound on every pair, so we describe RIDE as matching its teacher and do not claim a significant margin above it. What the table establishes is the gap that matching leaves and extrapolation closes: OPRD stays below its teacher on all four pairs, and on Qwen3-4B it ends below the untrained student, while RIDE reaches the teacher on all four.

The findings extend beyond R1-Distill-1.5B in three ways.

  1. The coefficient was never re-tuned. λ = 1.25 was chosen once on R1-Distill-1.5B and transferred unchanged. The gains of +4.06, +2.39 and +0.97 points on the other three families are therefore obtained without any tuning on those families.

  2. New held-out benchmarks on all four pairs. We evaluated on AIME 2026 (30 problems) and HMMT February 2026 (33 problems), which postdate the base models and the training corpus, with 16 draws per problem and one checkpoint per method. Coefficients and checkpoints were fixed beforehand, with no selection on either test. Entries are Teacher / OPRD / RIDE, Avg@16 (%).

PairAIME 2026HMMT Feb 2026Mean of the two
R1-Distill-1.5B40.21 / 37.50 / 39.5824.62 / 23.67 / 25.5732.41 / 30.59 / 32.58
Qwen3-4B51.04 / 51.67 / 53.1334.66 / 33.33 / 34.2842.85 / 42.50 / 43.70
Llama-3.2-3B0.42 / 0.63 / 1.461.70 / 1.89 / 1.891.06 / 1.26 / 1.68
Phi-4-mini6.46 / 5.63 / 6.672.46 / 1.89 / 2.274.46 / 3.76 / 4.47

RIDE leads OPRD in seven of the eight pair-by-benchmark comparisons and ties in the eighth. On the mean of the two benchmarks it is above OPRD on all four pairs and at or above the teacher on all four; benchmark by benchmark the teacher comparison is close and varies by test, as expected when RIDE matches its teacher. The pattern of the main results carries over to problems used only for evaluation.

  1. The failure of output-space training reproduces on another family. On Phi-4-mini, OPD and ExOPD fall to 0.21% and 0.42% on AIME 2026, against 6.46% for the teacher, while RIDE reaches 6.67%.

Taken together, RIDE's mean is at or above the teacher's and above OPRD's on all four pairs, both on the original benchmarks and on the mean of the two new ones.

The controlled experiments (direction controls, the ExOPD coefficient sweep, and the deterministic baseline of W1) are run on R1-Distill-1.5B because its teacher-to-base gap is the largest of the four (16.3 points), which makes effects easiest to resolve; the direction controls and the deterministic baseline use three seeds per condition. The revision adds the tables above and the new benchmarks to Section 5.2.

AuthorsOpenReview ↗

Response to Reviewer EaCj (3/3): W3

Continuation of our response to Reviewer EaCj.

W3. What are RIDE's wall-clock, GPU-hour and memory costs, and how does it compare under equal compute?

What RIDE adds. One no-grad forward pass of the frozen base model per rollout, computed on the teacher worker. The student-side computation, the number of updates and the rollouts are identical to OPRD, and the worker sends one representation tensor in either method (Table 6). ExOPD needs the same extra pass, because its target uses the base model's log-probabilities.

Measured cost. We timed OPRD, RIDE and ExOPD in the same setup: the same model pair (R1-Distill-1.5B), a single A800, 4 prompts with 2 responses each per step, up to 1,024 tokens, averaged over steps 6 to 30. OPRD and RIDE use the same data order and seed and process exactly the same tokens.

OPRDRIDEExOPD
Time per training step27.63 s29.76 s57.21 s
Relative to RIDE0.93×1.00×1.92×
GPU-hours for 500 steps at this workload (from the per-step time)3.844.137.95
  • RIDE trains about twice as fast as ExOPD. A RIDE step takes 29.76 s against 57.21 s for ExOPD, so RIDE needs 52% of ExOPD's training time. ExOPD students also generate longer responses than RIDE students (Table 2), which widens the gap further in full-length training.
  • RIDE costs 7.7% more per step than OPRD. The increase is concentrated in the teacher-worker phase (9.89 s versus 11.35 s), which in RIDE runs the frozen base, builds the extrapolated target and logs diagnostics, so 7.7% is an upper bound on the cost of the extra forward pass.

Memory. The additional memory is the frozen base model on the teacher worker, plus one more set of layerwise hidden states while the target is formed. The student's footprint is unchanged. In these runs the peaks reported by the PyTorch allocator are 72.9 GiB for OPRD and 80.1 GiB for RIDE; they aggregate peaks that need not coincide in time, so they are upper bounds on simultaneous usage.

Equal compute.

  • Against ExOPD, RIDE is better and cheaper at the same time. It scores 3.4 points above ExOPD at ExOPD's best coefficient while using about half the compute. In the time ExOPD needs for 260 steps, RIDE completes all 500.
  • Against OPRD, both methods perform the same 500 student updates on the same number of rollouts and tokens. The extra 7.7% is inference by a frozen model and produces no additional updates, so RIDE's gain (+2.33 points on average over four pairs) does not come from more optimization. Matching the budgets exactly would shorten RIDE's run by only 36 of its 500 steps.

Inference cost. RIDE students produce shorter responses than OPRD students (about 5.5k versus about 7k tokens, Table 2). This reduces rollout cost during training and lowers the cost of every query at deployment.

In short, RIDE trains about twice as fast as the output-space extrapolation baseline, costs about 8% more per step than plain teacher matching, keeps the student's memory footprint unchanged, and yields a student that is cheaper to run. The revision adds these measurements to Section 4.2.

Thank you again for the constructive review. We hope the new experiments address your three points.

Reviewer_NcGwOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes RIDE for on-policy distillation from an RL-trained teacher. It computes layerwise hidden-state differences between the teacher and its pre-RL checkpoint on student-generated prefixes. It extrapolates the targets beyond the teacher and trains the student through representation regression. Across four model pairs, RIDE outperforms OPRD and output-space baselines. Its mean scores slightly exceed the teachers.

Critical strengths
  1. RIDE provides a simple extension of OPRD [1]. It replaces teacher matching with context-dependent teacher–base residual extrapolation. The construction is easy to implement and directly compare with OPRD.

  2. RIDE improves over OPRD across four model families. Fixed-magnitude direction controls provide additional evidence for the proposed residual.

References

[1] OPRD: On-Policy Representation Distillation. 2026.

Critical weaknesses
  1. The novelty and baseline comparisons need stronger support. Representation matching and reward extrapolation already appear in OPRD [1] and ExOPD [2]. Difference-based directions also have precedent in activation engineering [3]. RIDE’s distinctive contribution is their specific combination. However, the paper omits the directly applicable ExPO baseline [4] based on results from prior work. This is insufficient, especially since ExOPD often degrades in the current experiments. AlignDistil [5] also studies adaptive logit extrapolation and deserves discussion. These gaps limit the evidence for RIDE’s incremental contribution.

  2. The mechanism explanation and beyond-teacher gains are not sufficiently established. Head attenuation alone does not show that extrapolating low-gain directions improves performance. A deterministic output-space baseline and subspace ablations would help test this explanation. Mean gains over teachers are only 0.32–1.08 points. Three margins are smaller than one training-seed standard deviation. Evaluation covers 143 math problems under one RL recipe. The coefficient sweep also uses the reported benchmarks as validation, without a clearly separate selection set. These issues weaken confidence in the central claims.

References

[1] OPRD: On-Policy Representation Distillation. 2026.

[2] Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation. 2026.

[3] Steering Language Models With Activation Engineering. 2023.

[4] Model Extrapolation Expedites Alignment. ACL 2025.

[5] AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation. ACL 2025.

Questions for authors
  1. What is novel about the contribution stated in Section 1 beyond combining OPRD’s representation matching [1] with ExOPD’s extrapolation [2]? Please distinguish it from existing difference-based directions [3] and discuss AlignDistil [5].

  2. Section 5.1 omits ExPO [4] based on its ranking in prior work. Could you include it in Table 1 under the current setting? For Figure 4 and Table 2, please clarify whether each method’s coefficient was selected independently on a separate validation set.

  3. How do Proposition 4.1 and Figure 5(c) establish training instability? For λ > 1, the teacher is not ExOPD’s target, so nonzero variance at the teacher does not itself establish a failure. Please clarify this connection and compare with deterministic output-space extrapolation.

  4. Figures 1 and 5 mainly analyze the final-layer residual. Do low-gain directions drive the gains? Please provide matched-norm subspace ablations. Appendix G mentions per-layer similarity diagnostics; could you show student–teacher and student–base similarities before and after training across layers? This would help assess the all-layer supervision in Eq. 7.

  5. Could you report mean ± SD for all methods in Table 1 and the direction controls in Table 3? Table 4 provides SD only for OPRD and RIDE’s aggregate scores. Paired confidence intervals would help assess the small gains over teachers.

  6. Sections 5.3–5.5 mainly use the R1-Distill-1.5B pair. Do the coefficient sensitivity and mechanistic findings hold across the other pairs in Table 1? Please provide corresponding analyses, if available, and clarify the limits of generalization to larger models and other RL recipes.

9 comments
AuthorsOpenReview ↗

Response to Reviewer NcGw (1/5): W1, Q1

Thank you for the detailed review and for the concrete list of analyses. We ran the experiments you asked for: ExPO on all four pairs, a deterministic output-space extrapolation baseline, a finer ExOPD sweep, direction controls with uncertainty, and two held-out benchmarks. They consistently support the paper's claims. We go through your questions in order; all intervals are unpaired Welch 95% CIs over training seeds.

Q1 / W1. What is new beyond combining OPRD's representation matching with ExOPD's extrapolation?

The contribution is a finding that neither prior method contains: extrapolating beyond an RL-trained teacher works in representation space and does not work in output space. A controlled comparison on R1-Distill-1.5B (three seeds per cell, macro Avg@16) shows this directly.

Where the target is extrapolatedλ = 1λ = 1.25Effect of extrapolation
Output space, sampled tokens (OPD → ExOPD)52.9049.87−3.03
Output space, deterministic (full-vocabulary KL, new)53.8953.77−0.11
Representation space (OPRD → RIDE)54.5056.38+1.88

Effects are computed from unrounded means.

Across the four model families, representation-space extrapolation gains +2.33 points on average over OPRD (95% CI [1.46, 3.19]), while ExOPD does not improve on OPD at any coefficient we tried (2/5). OPRD does not extrapolate at all, and ExOPD extrapolates a token log-ratio at the output. RIDE extrapolates a different object: we run each student prefix through the checkpoints before and after RL, and the difference in their hidden states gives a direction at every layer and position (Eqs. 5–7), which the student learns by deterministic regression. Proposition 4.2 shows that this target maximizes a reward for moving along the RL direction under a quadratic penalty on the distance from the teacher, and λ = 1 recovers OPRD. The result is a one-line change to the target that recovers the full teacher performance where matching does not (5/5) and trains about twice as fast as ExOPD (29.76 s versus 57.21 s per step in the same setup).

Relation to activation engineering [3]. ActAdd builds a steering vector from contrastive prompts and adds it to the activations of a chosen layer at inference time, so the vector is fixed and prompt-independent and the model must be intervened on at every forward pass. RIDE differs on each point: the residual is recomputed for every prefix and position, it is applied at all layers, and it is used as a training target, so the improvement is distilled into ordinary weights and no intervention is needed at deployment. Our controls test whether this context dependence matters. A residual taken from a different trajectory (55.12) or a random direction of the same norm (55.04) leaves the student at the teacher-matching level (54.50 to 54.85), whereas the context-matched RL residual reaches 56.38 (4/5).

Relation to AlignDistil [5]. AlignDistil addresses preference alignment. It distills preferences by adapting token logits built from a DPO and a reverse-DPO checkpoint, with an extrapolation coefficient chosen per token. It is therefore a logit-space extrapolation method, in the same family as ExOPD, applied to a different problem. Our results speak to this family as a whole: for RL-trained reasoning teachers, logit-space extrapolation yields no gain even with a deterministic full-vocabulary objective, whereas hidden-state extrapolation does. Token-adaptive coefficients are complementary to RIDE and could be combined with it.

Relation to ExPO [4]. ExPO extrapolates in weight space. We now include it on all four pairs (2/5): it matches the teacher on the original benchmarks but falls below it on held-out ones, where RIDE does not.

The revision adds [3], [4] and [5] to Section 2 with this discussion, and states the contribution in Section 1 as the finding above.

AuthorsOpenReview ↗

Response to Reviewer NcGw (2/5): Q2

Continuation of our response to Reviewer NcGw.

Q2. Can ExPO be included under the current setting, and how were the coefficients selected?

ExPO on all four pairs. We evaluated θ=θT+α(θT−θB)\theta = \theta_T + \alpha(\theta_T - \theta_B) with α = 0.25, the weight-space counterpart of λ = 1.25. ExPO and RIDE therefore extrapolate by the same factor, and neither is tuned per pair.

Original benchmarks (macro Avg@16 over AIME24, AIME25, AIMO):

PairTeacherOPRDExPORIDE
R1-Distill-1.5B55.3054.5055.5556.38
Qwen3-4B65.5962.0165.7866.07
Llama-3.2-3B13.0110.9412.7813.33
Phi-4-mini17.9617.3318.3918.30
Average37.9736.2038.1338.52

Held-out benchmarks (average of AIME 2026 and HMMT Feb 2026, described in 5/5):

PairTeacherOPRDExPORIDE
R1-Distill-1.5B32.4130.5929.1132.58
Qwen3-4B42.8542.5042.3743.70
Llama-3.2-3B1.061.260.971.68
Phi-4-mini4.463.763.994.47
Average20.2019.5319.1120.61
  • On the original benchmarks ExPO stays at the teacher's level, within 0.5 points on every pair. RIDE is above ExPO on three of the four pairs and level with it on the fourth.
  • On the held-out benchmarks ExPO falls below the teacher on all four pairs (by 3.3 points on R1-Distill-1.5B) and below RIDE on all four, as well as in each of the eight individual pair-by-benchmark cells. RIDE stays at or above the teacher on all four.

Weight-space extrapolation therefore does not carry over to unseen problems, while representation-space extrapolation does. This is consistent with ExPO taking a single global, context-independent step in weight space, whereas RIDE's target is recomputed for every prefix. ExPO is added to Table 1.

How the coefficients were selected. Table 1 compares the extrapolation methods at a common coefficient. In Figure 4 and Table 2 each coefficient is one run on the original benchmarks, and RIDE's λ = 1.25 was chosen from that sweep, once, on R1-Distill-1.5B. It was never re-tuned. Most of the evidence therefore lies outside the selection: the other three model families use the untuned coefficient, ExPO uses the matched α without tuning, and nothing was selected on the two held-out benchmarks.

For ExOPD we extended the sweep to the modest coefficients where it might be expected to do best (R1-Distill-1.5B):

ExOPD λAIME24AIME25AIMOMacro
1.00 (= OPD)47.133.578.152.9
1.05 (new)46.4633.7578.6952.97
1.10 (new)48.5432.7176.2052.48
1.15 (new)44.3833.5478.3152.08
1.2544.229.675.849.9

To give ExOPD every advantage, the new rows are reported at their best checkpoint. Even so, no coefficient improves on OPD by more than 0.1 points, and ExOPD's best score (52.97) is 3.4 points below RIDE. The comparison in Table 1 is therefore not an artifact of the coefficient. The revised Table 1 reports ExOPD at its best coefficient, and Section 5.1 states the selection protocol.

AuthorsOpenReview ↗

Response to Reviewer NcGw (3/5): Q3

Continuation of our response to Reviewer NcGw.

Q3. How do Proposition 4.1 and Figure 5(c) relate to training instability, and how does RIDE compare with deterministic output-space extrapolation?

What the proposition establishes. You are right that for λ > 1 the fixed point of ExOPD is the extrapolated distribution p∗p^* rather than the teacher, and we have rephrased Section 4 accordingly. The proposition describes how the sampled-token estimator behaves on the path the student actually travels, from the base toward the teacher:

  • At initialization the student equals the base, the advantage reduces to −λρ-\lambda\rho, and its variance is λ2\lambda^2 times that of OPD (1.56× at λ = 1.25).
  • At the teacher, OPD's advantage variance vanishes, while ExOPD's remains at (λ−1)2 Var[ρ](\lambda-1)^2\,\mathrm{Var}[\rho].

At both ends of this path, extrapolation makes the sampled-token estimator noisier than OPD. Whether that noise is what degrades ExOPD is an empirical question, and the experiment you propose answers it.

Deterministic output-space extrapolation. We trained a full-vocabulary reverse-KL objective, KL(p∥p⋆)\mathrm{KL}(p \parallel p^{\star}), computed exactly over the vocabulary at every position, toward the same target p⋆∝qλb1−λp^{\star} \propto q^{\lambda} b^{1-\lambda} as ExOPD. It differs from ExOPD only in not sampling. R1-Distill-1.5B, three seeds per method, macro Avg@16.

Method (λ = 1.25)SpaceEstimatorMacro Avg@16Gain over previous row
ExOPDOutputSampled token49.87—
Full-vocabulary reverse KLOutputDeterministic53.77 ± 0.76+3.90
RIDERepresentationDeterministic56.38 ± 0.97+2.61 [0.59, 4.63]
  • The instability is real and comes from the estimator. Removing sampling alone lifts output-space extrapolation from 49.87 to 53.77. This is the effect Proposition 4.1 points to, now established by a controlled experiment.
  • RIDE's advantage survives the removal of that noise. Against the deterministic objective, RIDE is still 2.61 points higher, and the interval [0.59, 4.63] excludes zero.
  • Deterministic output-space extrapolation does not help. The full-vocabulary objective scores 53.89 at λ = 1 and 53.77 at λ = 1.25. Extrapolation gains nothing in output space even without noise, and gains +1.88 in representation space.

The 6.51-point gap between ExOPD and RIDE thus splits into +3.90 from the estimator and +2.61 from the supervision space. Both mechanisms proposed in the paper are confirmed, and each is quantified.

In the revision, Section 4 presents Proposition 4.1 as a statement about estimator noise along the training path, with the fixed point identified as p∗p^*, and Section 5.4 rests the instability claim on the experiment above. The full-vocabulary baseline is added to Table 1.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the paper's contributions, the revision, and the discussion

Dear Area Chair,

Thank you for handling our submission. This note summarises the paper's contributions, what the revised PDF adds, and where the discussion stands. Changes in the PDF are marked in blue.

Contributions

  1. A new signal for distillation: the RL-induced representation residual. When an RL-trained teacher is distilled into a student that starts from the teacher's pre-RL checkpoint, both checkpoints are available. We run them on the student's own prefixes and take the layerwise difference of their hidden states. This residual measures what RL changed on the contexts the student visits, at every layer. Earlier methods treat the teacher as an endpoint, or extrapolate in output or weight space.

  2. A one-line method with a clear interpretation. RIDE places the hidden-state target beyond the teacher along this residual, with a single coefficient. At λ = 1 it is exactly OPRD, so any difference between the two is due to the residual term alone. It maximizes a reward for moving along the RL direction under a quadratic penalty around the teacher (Proposition 4.2).

  3. A finding: extrapolating beyond the teacher works in representation space and not in output space. On R1-Distill-1.5B, moving from λ = 1 to λ = 1.25 changes macro Avg@16 by −3.03 points with sampled-token output-space training (OPD → ExOPD), by −0.11 with an exact full-vocabulary output-space objective, and by +1.88 in representation space (OPRD → RIDE).

  4. An explanation, with its two factors separated. The LM head attenuates the RL residual unevenly: the 512 head directions with the smallest gain hold 79.8% of the residual's hidden-state energy but 30.0% of its logit energy. Sampled-token extrapolation also amplifies estimator noise (Proposition 4.1). A deterministic baseline splits the 6.51-point gap between ExOPD and RIDE into 3.90 points from the estimator and 2.61 points from the supervision space (95% CI [0.59, 4.63]).

  5. Consistent gains on four model families. With a coefficient chosen once on one pair and never re-tuned, RIDE improves over OPRD on all four families, by +2.33 points on average (95% CI [1.46, 3.19]). It recovers 105–117% of the base-to-teacher gain, against −29% to 95% for OPRD, and it is the only distillation method whose mean reaches its teacher on every family.

Our claim is exact: RIDE matches its teacher and closes the gap that teacher matching leaves. We do not claim a significant margin above the teacher.

What the revision added, by concern

  • Separating the two mechanisms (all three reviewers): a full-vocabulary reverse-KL baseline at λ = 1 and 1.25, three seeds each (Table 4, Appendix H).
  • Fair tuning of ExOPD (nLBu, NcGw): ExOPD at λ = 1.05, 1.10 and 1.15. Its best score, even at its best checkpoint, is 52.97, against 56.38 for RIDE (Appendix B).
  • Weight-space baseline (NcGw): ExPO on all four pairs. It matches the teacher on the original benchmarks and falls below it on all four pairs on the held-out ones, where RIDE stays at or above it (Tables 2 and 3).
  • Statistical reliability (all three): mean ± SD and 95% intervals for every direction control and for RIDE versus OPRD on every pair (Appendices C and D).
  • Held-out evidence (nLBu, NcGw): AIME 2026 and HMMT February 2026 on all four pairs, with nothing selected on them. RIDE leads OPRD in seven of eight comparisons and ties in the eighth. We also report results without AIMO, which overlaps with the training corpus; RIDE still leads on every pair (Appendix I).
  • Per-layer analysis (NcGw): the student's update is aligned with the RL residual at every layer from 4 to 28, with a cosine of at least 0.92 (Appendix K).
  • Cost (EaCj): a RIDE step takes 7.7% longer than an OPRD step and half as long as an ExOPD step (Appendix J).
  • Proposition 4.1 and Eq. (8) (nLBu, NcGw): we accepted both points and rewrote the passages (Section 4.3, Appendix E).
  • Related work (NcGw): AlignDistil, activation steering and ExPO are now discussed (Section 2).

Status of the discussion

  • Reviewer nLBu has confirmed that our response fully addresses the concerns raised, which covered ExOPD tuning, estimator versus supervision space, the reliability of the gains, and additional benchmarks.
  • Reviewer NcGw replied on 19 November and maintained the rating. The reply cites a mismatch between our responses and the PDF, which had not yet been updated; difficulty locating the Q3 evidence and the per-layer analysis; and a view that the improvements are small. Our follow-up gives the section for every item, the Q3 evidence in one table, the per-layer results, and the magnitudes: RIDE is 2.3 to 9.1 points above every distillation baseline. We believe the revised paper answers each point of that reply.
  • Reviewer EaCj: we have answered every point and have not yet received feedback.

We ask that the paper be assessed on the revised PDF. Thank you for your time.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.