- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies what on-policy distillation (OPD) transfers to the student, using sparse crosscoders trained jointly on the student before and after OPD and the teacher. To read each student’s feature usage within the shared crosscoder, it proposes the swap readout. Across three settings, it reports that OPD creates no new features and passes on none of the teacher’s, but slightly reweights shared features, especially at decision tokens such as “Wait” and “So.” It also finds that an SFT warm-up before OPD works in a similar way and supports this with a feature-level intervention.
Critical strengths
The paper asks an interesting question about what OPD actually transfers to the student, and its main message, that OPD mainly teaches the student how to use features it already has, is intuitive and compelling. The swap readout is also a useful tool. It is simple, it separates each student’s feature usage within a jointly trained crosscoder, and it can also be applied to checkpoints not used to train the crosscoder.
Critical weaknesses
The paper’s central message is that “OPD behaves more like a reweighting of existing features than an acquisition of new ones: what the student learns from the teacher is how to use the features they already share.” I have two concerns about the evidence for this message.
W1. The evidence that OPD does not acquire new features may only reflect the small size of the update. Model-specific features appear when activations differ enough, and MAS also shifts when an existing feature is simply used more strongly. So the number of specific features naturally grows with model distance, as in Fig. 3(a–c) and in Shi et al. (2026), where the large SFT update yields many and the small RL update yields few. Since OPD makes small updates, finding no new features may just reflect a small change. This is a problem because the paper’s main claim, that OPD reweights rather than acquires features, cannot be separated from this simpler explanation. A comparison with SFT or RL at a matched distance is needed.
W2. The evidence that OPD reweights existing features is also open to other interpretations and its connection to OPD is unclear. The changes in Section 3.2 appear as changes in how existing features are used, but a small new feature absorbed into an existing one would look exactly the same, so these results do not by themselves show reweighting rather than acquisition. Even if they do reflect reweighting, it is unclear what this has to do with OPD specifically. The defining aspect of OPD, training on the student’s own rollouts, is not varied in the analysis. Meanwhile, the only off-policy comparison, the SFT warm-up in Section 4.2, moves the features in a similar direction, and the results also resemble those of Shi et al. (2026) for RL. The findings may therefore describe post-training on a teacher in general rather than OPD. A comparison with off-policy distillation or SFT at a matched distance would be needed to support the claim about OPD.
Questions for authors
-
Is the absence of new features due to OPD itself or to the small size of the update? A positive control with a large update would not answer this. Would the crosscoder detect new features from a training known to add them, if that training changed the student as little as OPD does?
-
Is any finding specific to OPD? The SFT warm-up in Section 4 already shows similar reweighting. Which result, if any, would differ for off-policy distillation that changes the student by the same amount? If none, the conclusions describe post-training on a teacher in general rather than OPD.
5 comments
W2/Q2: absorbed new features and the rollout author
We thank the reviewer for the question. In the current paper, the reweighting is measured directly by the swap readout and changes accuracy when imposed on a student (Table 3); it is not inferred from the absence of new features. We address two questions: whether it could be an absorbed new feature, and which results depend on who writes the rollouts. The new results are in Appendices F and G of the revision, marked in blue.
1) Absorbed new components. In the equal-magnitude synthetic setting of our response to W1, the new latent is identically zero in the base, its activation change equals that of OPD (5.0%), and its direction has cosine 0, 0.9, or 0.99 with an existing direction, with three seeds each.
| Cosine with an existing direction | Direction and activity recovered | MAS > 0.5 | Zero base firings |
|---|---|---|---|
| 0 | 3/3 | 3/3 | 3/3 |
| 0.9 | 3/3 | 3/3 | 2/3 |
| 0.99 | 0/3 | 0/3 | 0/3 |
Hence a new component escapes the criteria only when its cosine with an existing feature exceeds 0.9; at cosine 0.99, i.e., within about 8°, adding it changes the representation almost exactly as amplifying that feature does. OPD shows neither signature left by less aligned components (MAS > 0.5, zero base firings). We further test whether the features OPD lowers keep their original contexts. The 50 features with the largest decrease are selected on one held-out shard and evaluated on the other three: all 50 decrease again (by 21.8% on average), their token-conditional activations before and after OPD correlate with median 0.91, and their 100 strongest positions overlap by a median of 93%. In addition, the 65K dictionary yields no OPD-dominated or OPD-specific feature.
2) Varying only the author of the rollouts. W2 notes that the defining aspect of OPD, training on the student's own rollouts, is not varied, and the warm-up also differs from OPD in loss, LoRA, and data. We therefore train two students from the same base with the same teacher (Qwen3-4B-Base-GRPO), distillation loss (a KL on the teacher's top-16 tokens), full-parameter optimizer, prompts, prompt order, and number of trained tokens per prompt; the rollouts are written either by the current student (on-policy) or by the teacher (off-policy). We note that the shared loss isolates the rollout source rather than reproducing the paper's objective. Each group runs 64 steps with two seeds and is read with the paper's dictionary and swap readout on 7,562 pre-specified features. Checkpoints are selected on a calibration set to match the activation change.
| Seed 42 | Seed 43 | |
|---|---|---|
| Activation change, on / off | 8.26% / 8.36% | 7.77% / 7.70% |
| Features within ±20% of base rate, on / off | 99.2% / 99.3% | 99.1% / 99.6% |
| Cosine between on and off reweighting vectors | 0.68 | 0.65 |
| Usage of feature 32128 relative to base, on / off | 0.08× / 0.34× | 0.08× / 0.30× |
The on−off difference vectors of the two seeds are positively correlated (cosine 0.23; 0.22 at equal steps and 0.40 near the scale of OPD), and the largest differences fall on the same features in both seeds. It is perhaps worth noting that near the activation change of OPD (5.4%–5.9%), the cosine is again 0.66–0.68.
3) Implications for W2 and Q2. As for absorption, part 1 shows that a new component is flagged unless it is nearly parallel to an existing feature, that the lowered features keep their contexts, and that a larger dictionary resolves no new OPD feature. With Table 3, the reweighting is therefore a measured and functional change, rather than a relabeling of undetected features. As for Q2, the result that differs at the same activation change is the direction of reweighting. Both rollout sources reweight shared features only slightly (about 99% of frequent features within ±20%), but they reweight different features, and part of this difference reproduces across seeds on the same features. This agrees with Section 4.2, where more than half of the warm-up's change lies outside OPD's direction, and adding it to a directly distilled student still raises accuracy (Table 3). Finally, the findings do not describe post-training on a teacher in general: SFT on solutions written by a stronger model already yields an SFT-specific feature at an activation change of 7.2% (response to W1), and off-policy distillation from the same teacher reweights in a different direction. This agrees with the central conclusion of the current paper: OPD reweights the features that the student already shares with the teacher, and training on the student's own rollouts affects which features are reweighted.
Dear Reviewer Ay8n,
We would like to express our sincere gratitude once again for the time and effort you have devoted to reviewing our work. As the discussion period draws to a close, we would be grateful if you could let us know whether our responses have adequately addressed your concerns. Should you have any further questions or comments, please do not hesitate to share them. We look forward to continuing our communication with you.
Best, Authors
W1/Q1: detection at the activation change of OPD
We thank the reviewer for the question. We test, at the activation change of OPD, whether (i) a new feature is detected and (ii) an amplified existing feature is not counted as new.
1) Equal-magnitude synthetic control. Following W1, the change is measured by the difference in activations, ‖H − H_base‖_F/‖H_base‖_F at the layer we analyze, which is 5.0% for OPD. A 128-dimensional representation is generated from 256 sparse background latents, and three post-training representations are constructed: one adds a new latent (the positive control), which is identically zero in the base, orthogonal to the background subspace, and active at 10% of positions; one amplifies an existing latent; and one is unchanged (the negative controls). The first two are calibrated to the activation change of OPD (measured 4.96%–5.03%). The crosscoder is configured as in the current paper (BatchTopK, 32K, k = 50), trained on 20.48M positions, and evaluated on 131,072 independent positions, with three seeds per condition. We note that newness is fixed by construction, and recovery is checked against the ground truth rather than through MAS or NRN.
The new latent is recovered in all three seeds, with decoder cosine above 0.9987 and activity correlation above 0.99998, and it never fires on the base. The amplified latent already fires on the base, and no feature in these runs reaches NRN > 0.8. It is perhaps worth noting that, at a shorter budget of 5,000 steps, one amplification run reaches NRN = 0.801; the swap readout shows that this feature already fires before amplification (714 → 1,396 firings), so it is not counted as new either. Consequently, condition (i) holds, since the new latent is detected and attributed to the post-training model, and condition (ii) holds, since amplification raises MAS to at most 0.49 and NRN to at most 0.66.
2) Real SFT control. The synthetic control involves no training, whereas Q1 asks about a training known to add features. We therefore fine-tune the same student with the SFT recipe of Shi et al. (Appendix F) and train a joint crosscoder of the base, OPD, and SFT models as in the current paper. At step 6, whose activation change is 7.2%, the crosscoder detects one SFT-specific feature (MAS = 0.801, NRN = 0.868), and the detection persists with a 65K dictionary; OPD meets neither criterion. The swap readout of this dictionary gives the usage of this feature:
That is, at step 4, below the activation change of OPD, SFT already uses this feature 3.8 times as often as the base, whereas OPD uses it one-seventh as often.
3) Implications for W1 and Q1. Parts 1 and 2 answer Q1 affirmatively, in two steps. First, a new feature whose activation change equals that of OPD is detected in all seeds. Second, in a training known to add features, the feature that the crosscoder identifies as SFT-specific is already used 3.8 times as often below the activation change of OPD. Hence a change as small as that of OPD does not place a new feature outside the detection range. As for W1, the count of model-specific features tracks distance rather than acquisition only if (i) or (ii) fails. Both hold at this scale, and real models show the same separation between attribution and usage: a feature whose usage rises 70-fold has MAS 0.46, whereas a feature with MAS 0.82 is used less after SFT. Therefore, at the activation change of OPD, the count of model-specific features responds to acquisition rather than to distance, and the absence of new features under OPD cannot be explained by the small update alone. This agrees with the central conclusion of the current paper, which also rests on the reweighting that the swap readout measures (Section 3.2) and on its effect on accuracy (Section 4.2): OPD reweights the features that the student already shares with the teacher, rather than acquiring new ones. All results are in Appendix F of the revision, marked in blue.