- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes RIDE, an on-policy representation distillation method that treats the change induced by RL as a direction to extrapolate. On student-generated prefixes, the method computes the layerwise hidden-state difference between an RL-trained teacher and its pre-RL checkpoint, then trains the student toward targets displaced beyond the teacher along this residual. The student starts from the same base checkpoint, and setting the extrapolation coefficient to one recovers ordinary representation matching. Across four base/teacher pairs on mathematical reasoning benchmarks, RIDE improves over OPRD and achieves mean scores close to or slightly above the teacher. The paper also investigates head attenuation, conditional sampling variance, and alternative residual directions to explain the observed gains.
Critical strengths
-
The method is simple and clearly defined. RIDE makes a concrete change to representation distillation by extrapolating the teacher-to-base residual. Recovering OPRD at λ=1 provides a useful reference point, and the reward-and-quadratic-penalty interpretation helps explain the objective.
-
The empirical improvements over OPRD are encouraging. RIDE improves the reported macro-average score on all four model pairs, with gains ranging from 0.97 to 4.06 points. The use of three independent training seeds and multiple model families strengthens the evidence beyond a single-model result.
-
The paper includes useful diagnostic experiments and acknowledges important limitations. The coefficient sweep and random, reversed, mismatched-origin, and trajectory-mismatched direction controls are relevant to the proposed explanation. I also appreciate the explicit acknowledgment that most margins over the teacher are small relative to training-seed variation, and that the experiments are limited to mathematical reasoning and a shared-initialization setting.
Critical weaknesses
-
The evidence for reliably surpassing the teacher and for direction-specific gains remains limited. On three model pairs, the margin over the teacher is only 0.32–0.48 points, smaller than the reported training-seed standard deviation. The teacher’s evaluation sampling uncertainty is also not reported. In Table 3, RIDE exceeds the strongest direction control by 1.26 points, but the controls’ seed counts and uncertainty are unspecified. The mean improvements are worth reporting, but stronger conclusions require uncertainty estimates for the relevant differences.
-
The ExOPD comparison does not establish performance under equally thorough tuning. The main table fixes λ=1.25, which is selected as the best value for RIDE. ExOPD’s extrapolating coefficients are comparatively coarse, while the additional values 1.15 and 1.35 are tested only for RIDE. This is particularly relevant because Appendix F places ExOPD’s conditional-variance minimum near λ=1.14. The paper also uses the reported benchmarks for validation curves and coefficient selection without clearly identifying an independent selection set. These choices complicate the interpretation of both the method comparison and the small gains over the teacher.
-
The stability analysis does not fully separate representation-space supervision from the choice of gradient estimator. RIDE uses deterministic layerwise regression, whereas ExOPD uses a sampled-token update. A full-vocabulary output-space objective would also be deterministic at a fixed prefix, so zero conditional next-token variance is not unique to representation-space training. Moreover, the fitted polynomial in Appendix F gives an advantage variance of approximately 0.00864 at λ=1.25, below 0.0096 at λ=1 on those fixed prefixes, even though ExOPD already underperforms at λ=1.25. The analysis therefore does not by itself explain the main performance gap, and advantage variance is not equivalent to parameter-gradient variance.
-
The claimed relationship between the actual RIDE target and output-space extrapolation needs clarification. Equation (8) is a linear identity at the head input, but the implemented loss supervises pre-normalization residual states. Final normalization generally does not commute with affine extrapolation. Appendix E discusses this distinction, but it remains unclear whether the reported near-machine-precision agreement uses the normalized actual RIDE target or an affine combination of already-normalized states. The reported head drift also produces a non-negligible change in logits, despite its small relative weight norm. These details matter for the claim that the methods differ only in the space where extrapolation is performed.
Questions for authors
-
How were λ and the optimization settings selected? Were AIME24, AIME25, and AIMO used both for selection and final reporting? Could ExOPD be evaluated with modest extrapolation coefficients such as 1.05, 1.10, and 1.15 under the same selection budget?
-
How much of RIDE’s advantage remains against a lower-variance output-space extrapolation objective? A full-vocabulary or sufficiently supported approximation would help separate the effect of the estimator from the effect of the supervision space. How should the variance result at λ=1.25 be reconciled with ExOPD’s degradation at that coefficient?
-
What exactly is compared in the Equation (8) diagnostic? Does the reported error use the actual pre-normalization RIDE target after final normalization, or the affine combination of normalized teacher and base states? Please also clarify how teacher-head drift affects the correspondence to the native ExOPD target.
-
Could you provide results on additional mathematical reasoning benchmarks? The evaluation is limited to AIME24, AIME25, and AIMO, and I am concerned about potential exposure to these benchmarks during pre-training or post-training. Results on additional, preferably more recent or independently held-out benchmarks would help establish whether the improvements generalize beyond these three datasets.
7 comments
Response to Reviewer nLBu (2/5): W2, Q1
Continuation of our response to Reviewer nLBu.
W2 / Q1. Was ExOPD tuned as thoroughly as RIDE, and how were λ and the optimization settings selected?
Optimization settings. All methods share one configuration: AdamW with peak learning rate , 3% warmup, cosine decay and 500 steps (Table 5). Nothing was tuned per method. The factor in the RIDE loss only matches the initial loss scale (Remark G.1).
ExOPD at the coefficients you proposed. We ran ExOPD at λ = 1.05, 1.10 and 1.15, close to the variance minimum of Appendix F. R1-Distill-1.5B, Avg@16 (%), one run per row.
| ExOPD λ | AIME24 | AIME25 | AIMO | Macro |
|---|---|---|---|---|
| 1.00 (= OPD top-1; submission) | 47.1 | 33.5 | 78.1 | 52.9 |
| 1.05 (new) | 46.46 | 33.75 | 78.69 | 52.97 |
| 1.10 (new) | 48.54 | 32.71 | 76.20 | 52.48 |
| 1.15 (new) | 44.38 | 33.54 | 78.31 | 52.08 |
| 1.25 (submission) | 44.2 | 29.6 | 75.8 | 49.9 |
To give ExOPD every advantage, the three new rows use the best checkpoint on these benchmarks; the submission rows use the final checkpoint.
Two conclusions follow.
- RIDE's advantage holds at ExOPD's best coefficient. ExOPD peaks at 52.97, which is 3.4 points below RIDE (56.38 ± 0.97) and also below OPRD (54.50). The ranking in Table 1 is therefore not an artifact of fixing λ = 1.25. The revised Table 1 reports ExOPD at its best coefficient, and Table 2 includes the finer sweep.
- No coefficient makes extrapolation help in output space. No λ > 1 improves on λ = 1 by more than 0.1 points, and λ ≥ 1.10 is lower. Extrapolation helps in representation space and not in output space, which is the paper's central point. The deterministic baseline in (3/5) confirms this with sampling noise removed.
How λ was selected. We chose RIDE's λ = 1.25 once, on the R1-Distill-1.5B pair, using the benchmarks reported in Table 2 (AIME24, AIME25, AIMO). It was never re-tuned. Most of the evidence therefore lies outside the selection: the three remaining pairs, where the untuned coefficient improves over OPRD by +4.06, +2.39 and +0.97 points, and the two newer benchmarks in (5/5), which were used only for evaluation. RIDE is also insensitive to the coefficient: every value in [1.15, 1.35] improves over OPRD in the sweep of Table 2. The revision states this selection protocol explicitly in Section 5.1.
Response to Reviewer nLBu (3/5): W3, Q2
Continuation of our response to Reviewer nLBu.
W3 / Q2. How much of RIDE's advantage remains against a deterministic output-space objective, and how does the variance result at λ = 1.25 fit?
We ran the experiment you proposed, and it separates the two factors cleanly.
Setup. We keep the output target fixed and compute reverse KL exactly over the whole vocabulary: at every position, toward the same as ExOPD. At a fixed prefix this update, like the RIDE update, is independent of the next-token draw. R1-Distill-1.5B, macro Avg@16 (%), three runs per row. Sampled-token means are from Table 1; the full-vocabulary runs use seeds 42, 43 and 44.
| Objective | λ | Mean ± SD | RIDE − row [95% CI where available] |
|---|---|---|---|
| Sampled KL (OPD top-1) | 1.00 | 52.90 | +3.48 |
| Sampled KL (ExOPD) | 1.25 | 49.87 | +6.51 |
| Full-vocabulary KL | 1.00 | 53.89 ± 0.90 | +2.49 [0.37, 4.62] |
| Full-vocabulary KL | 1.25 | 53.77 ± 0.76 | +2.61 [0.59, 4.63] |
| OPRD | 1.00 | 54.50 ± 1.04 | +1.88 [−0.40, 4.16] |
| RIDE | 1.25 | 56.38 ± 0.97 | — |
- Estimator noise is what breaks ExOPD. Removing sampling lifts λ = 1.25 from 49.87 to 53.77, a gain of 3.90 points. This confirms the paper's claim that sampled-token noise destabilises output-space extrapolation.
- The supervision space accounts for the rest. Against the deterministic objective, RIDE remains 2.61 points above output-space extrapolation ([0.59, 4.63]) and 2.49 points above output-space matching ([0.37, 4.62]). Both intervals exclude zero.
- Extrapolation helps only in representation space. Within the full-vocabulary objective, moving from λ = 1 to 1.25 changes the mean by −0.11 points ([−2.03, 1.80]); in representation space the same move gives +1.88. At λ = 1 the two spaces perform similarly (53.89 and 54.50). The difference emerges when the target moves beyond the teacher, which is the thesis of the paper.
So the answer to your first question is that RIDE's advantage over output-space extrapolation remains, with an interval that excludes zero, once the estimator is made deterministic.
On the variance at λ = 1.25. You are right. In Eq. (9) the negative covariance term brings the advantage variance down to 0.00864 at λ = 1.25, from 0.0096 at λ = 1, on the prefixes of Appendix F; it rises at larger coefficients. That fit is a local measurement: it uses fixed prefixes from a trained student at step 500, where the student is already close to the teacher. At initialization the student equals the base, so the advantage reduces to and its variance scales as , a factor of 1.56 at λ = 1.25. The step-500 fit therefore does not rank coefficients by the noise seen during training, and, as you note, the parameter-gradient variance also depends on the score function.
The decisive test is the one above: for the output baselines, the difference between estimators at λ = 1.25 (3.90 points) is far larger than the change across coefficients under exact KL (0.11 points). In the revision, Proposition 4.1 is presented as a statement about estimator noise, the Appendix F fit is described as a local diagnostic, and the causal claim rests on the full-vocabulary experiment, which is added to Table 1.
Response to Reviewer nLBu (1/5): W1
Thank you for reading the experiments and derivations so closely, and for recognising the simplicity of the method and the consistency of its gains across four model families. Your questions prompted four new experiments: direction controls with full uncertainty, a finer ExOPD sweep, a deterministic output-space baseline, and two held-out benchmarks. Each supports the paper's main result. All intervals below are unpaired Welch 95% CIs over training seeds.
RIDE improves over OPRD on all four pairs, and the aggregate improvement is statistically reliable. Original benchmarks, macro Avg@16 (%), mean ± SD over three seeds.
The improvement is positive on every pair, and the four-pair mean has an interval well clear of zero. The same holds out of sample (5/5).
RIDE closes the gap to the teacher that teacher matching leaves. The teacher is a single checkpoint, so we bound its decoding standard error from the binomial variance of its Avg@16 scores on the fixed problems.
RIDE's margin over the teacher is within the teacher's decoding SE bound on every pair, so we describe RIDE as matching its teacher and do not claim a significant margin above it. What the table establishes is the gap that matching leaves and extrapolation closes: OPRD stays below its teacher on all four pairs, and RIDE reaches the teacher on all four.
Direction controls. Every entry of Table 3 is a mean over three training runs. All displaced targets keep the norm of the original residual at each layer and position, with λ = 1.25 and loss weight 0.64. We added teacher matching at that same loss weight. R1-Distill-1.5B, macro Avg@16 (%).
The revision reports this table in place of Table 3 and states the conclusion as the ordering it shows: RL direction > unrelated directions ≈ teacher matching > reversed.