acceptodds
Under review as a conference paper at ICLR 2027

Understanding On-Policy Distillation: A Mechanistic Interpretability Perspective via Sparse Crosscoders

Abstract

On-policy distillation (OPD) has emerged as a widely adopted post-training technique for LLM reasoning. It is commonly believed to transfer knowledge from a stronger teacher to the student, yet what OPD actually distills into the student's internal representations remains unclear. In this study, we examine this question through the lens of sparse crosscoders, which learn one feature dictionary shared by the student before and after OPD and the teacher. Standard crosscoder analyses, however, identify features specific to one model but cannot tell how a model's use of its features changes, since the crosscoder encodes all models into a single set of feature activations. We therefore propose the swap readout, which reads the feature activations of each student checkpoint on its own and thus measures how training changes the student's use of each feature, even for checkpoints the crosscoder has never seen. Specifically, across three OPD settings, we observe that OPD neither creates features nor passes on the teacher's own, and leaves the firing rates of over of the student's frequently used features within . These results suggest that OPD primarily reweights the features the student already shares with the teacher rather than acquiring new ones. We further examine the SFT warm-up on the teacher's rollouts that commonly precedes OPD and makes it more effective. Rather than adding features, as one might expect, the warm-up reweights the shared ones, partly along OPD's direction, making part of OPD's change in advance, and partly in directions that OPD does not take and that persist through OPD. Imposing this reweighting on the features of a directly distilled student, without changing its weights, brings its accuracy close to that of the warmed-up student, whereas the same change on shuffled features does not. Together, these findings suggest that OPD behaves more like a reweighting of existing features than an acquisition of new ones: the student learns from the teacher how to use the features they already share.

Primary area:
interpretability and explainable AI
Keywords:
On-Policy Distillation, Sparse Crosscoders

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Ay8nOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper studies what on-policy distillation (OPD) transfers to the student, using sparse crosscoders trained jointly on the student before and after OPD and the teacher. To read each student’s feature usage within the shared crosscoder, it proposes the swap readout. Across three settings, it reports that OPD creates no new features and passes on none of the teacher’s, but slightly reweights shared features, especially at decision tokens such as “Wait” and “So.” It also finds that an SFT warm-up before OPD works in a similar way and supports this with a feature-level intervention.

Critical strengths

The paper asks an interesting question about what OPD actually transfers to the student, and its main message, that OPD mainly teaches the student how to use features it already has, is intuitive and compelling. The swap readout is also a useful tool. It is simple, it separates each student’s feature usage within a jointly trained crosscoder, and it can also be applied to checkpoints not used to train the crosscoder.

Critical weaknesses

The paper’s central message is that “OPD behaves more like a reweighting of existing features than an acquisition of new ones: what the student learns from the teacher is how to use the features they already share.” I have two concerns about the evidence for this message.

W1. The evidence that OPD does not acquire new features may only reflect the small size of the update. Model-specific features appear when activations differ enough, and MAS also shifts when an existing feature is simply used more strongly. So the number of specific features naturally grows with model distance, as in Fig. 3(a–c) and in Shi et al. (2026), where the large SFT update yields many and the small RL update yields few. Since OPD makes small updates, finding no new features may just reflect a small change. This is a problem because the paper’s main claim, that OPD reweights rather than acquires features, cannot be separated from this simpler explanation. A comparison with SFT or RL at a matched distance is needed.

W2. The evidence that OPD reweights existing features is also open to other interpretations and its connection to OPD is unclear. The changes in Section 3.2 appear as changes in how existing features are used, but a small new feature absorbed into an existing one would look exactly the same, so these results do not by themselves show reweighting rather than acquisition. Even if they do reflect reweighting, it is unclear what this has to do with OPD specifically. The defining aspect of OPD, training on the student’s own rollouts, is not varied in the analysis. Meanwhile, the only off-policy comparison, the SFT warm-up in Section 4.2, moves the features in a similar direction, and the results also resemble those of Shi et al. (2026) for RL. The findings may therefore describe post-training on a teacher in general rather than OPD. A comparison with off-policy distillation or SFT at a matched distance would be needed to support the claim about OPD.

Questions for authors
  • Is the absence of new features due to OPD itself or to the small size of the update? A positive control with a large update would not answer this. Would the crosscoder detect new features from a training known to add them, if that training changed the student as little as OPD does?

  • Is any finding specific to OPD? The SFT warm-up in Section 4 already shows similar reweighting. Which result, if any, would differ for off-policy distillation that changes the student by the same amount? If none, the conclusions describe post-training on a teacher in general rather than OPD.

5 comments
AuthorsOpenReview ↗

W1/Q1: detection at the activation change of OPD

We thank the reviewer for the question. We test, at the activation change of OPD, whether (i) a new feature is detected and (ii) an amplified existing feature is not counted as new.

1) Equal-magnitude synthetic control. Following W1, the change is measured by the difference in activations, ‖H − H_base‖_F/‖H_base‖_F at the layer we analyze, which is 5.0% for OPD. A 128-dimensional representation is generated from 256 sparse background latents, and three post-training representations are constructed: one adds a new latent (the positive control), which is identically zero in the base, orthogonal to the background subspace, and active at 10% of positions; one amplifies an existing latent; and one is unchanged (the negative controls). The first two are calibrated to the activation change of OPD (measured 4.96%–5.03%). The crosscoder is configured as in the current paper (BatchTopK, 32K, k = 50), trained on 20.48M positions, and evaluated on 131,072 independent positions, with three seeds per condition. We note that newness is fixed by construction, and recovery is checked against the ground truth rather than through MAS or NRN.

Post-training representationMASNRNSwap firings, base → postFeatures with NRN > 0.8 and zero base firings
New latent added0.92–0.980.95–0.990 → ~13,0001 (the new latent)
Existing latent amplified0.48–0.490.65–0.66~13,100 → ~13,1000
Unchanged–––0

The new latent is recovered in all three seeds, with decoder cosine above 0.9987 and activity correlation above 0.99998, and it never fires on the base. The amplified latent already fires on the base, and no feature in these runs reaches NRN > 0.8. It is perhaps worth noting that, at a shorter budget of 5,000 steps, one amplification run reaches NRN = 0.801; the swap readout shows that this feature already fires before amplification (714 → 1,396 firings), so it is not counted as new either. Consequently, condition (i) holds, since the new latent is detected and attributed to the post-training model, and condition (ii) holds, since amplification raises MAS to at most 0.49 and NRN to at most 0.66.

2) Real SFT control. The synthetic control involves no training, whereas Q1 asks about a training known to add features. We therefore fine-tune the same student with the SFT recipe of Shi et al. (Appendix F) and train a joint crosscoder of the base, OPD, and SFT models as in the current paper. At step 6, whose activation change is 7.2%, the crosscoder detects one SFT-specific feature (MAS = 0.801, NRN = 0.868), and the detection persists with a 65K dictionary; OPD meets neither criterion. The swap readout of this dictionary gives the usage of this feature:

ModelActivation changeUsage of this feature, relative to base
SFT, step 44.1%3.8×
SFT, step 56.4%9.0×
SFT, step 67.2%20.8×
OPD5.0%0.14× (246 → 35 firings)

That is, at step 4, below the activation change of OPD, SFT already uses this feature 3.8 times as often as the base, whereas OPD uses it one-seventh as often.

3) Implications for W1 and Q1. Parts 1 and 2 answer Q1 affirmatively, in two steps. First, a new feature whose activation change equals that of OPD is detected in all seeds. Second, in a training known to add features, the feature that the crosscoder identifies as SFT-specific is already used 3.8 times as often below the activation change of OPD. Hence a change as small as that of OPD does not place a new feature outside the detection range. As for W1, the count of model-specific features tracks distance rather than acquisition only if (i) or (ii) fails. Both hold at this scale, and real models show the same separation between attribution and usage: a feature whose usage rises 70-fold has MAS 0.46, whereas a feature with MAS 0.82 is used less after SFT. Therefore, at the activation change of OPD, the count of model-specific features responds to acquisition rather than to distance, and the absence of new features under OPD cannot be explained by the small update alone. This agrees with the central conclusion of the current paper, which also rests on the reweighting that the swap readout measures (Section 3.2) and on its effect on accuracy (Section 4.2): OPD reweights the features that the student already shares with the teacher, rather than acquiring new ones. All results are in Appendix F of the revision, marked in blue.

AuthorsOpenReview ↗

W2/Q2: absorbed new features and the rollout author

We thank the reviewer for the question. In the current paper, the reweighting is measured directly by the swap readout and changes accuracy when imposed on a student (Table 3); it is not inferred from the absence of new features. We address two questions: whether it could be an absorbed new feature, and which results depend on who writes the rollouts. The new results are in Appendices F and G of the revision, marked in blue.

1) Absorbed new components. In the equal-magnitude synthetic setting of our response to W1, the new latent is identically zero in the base, its activation change equals that of OPD (5.0%), and its direction has cosine 0, 0.9, or 0.99 with an existing direction, with three seeds each.

Cosine with an existing directionDirection and activity recoveredMAS > 0.5Zero base firings
03/33/33/3
0.93/33/32/3
0.990/30/30/3

Hence a new component escapes the criteria only when its cosine with an existing feature exceeds 0.9; at cosine 0.99, i.e., within about 8°, adding it changes the representation almost exactly as amplifying that feature does. OPD shows neither signature left by less aligned components (MAS > 0.5, zero base firings). We further test whether the features OPD lowers keep their original contexts. The 50 features with the largest decrease are selected on one held-out shard and evaluated on the other three: all 50 decrease again (by 21.8% on average), their token-conditional activations before and after OPD correlate with median 0.91, and their 100 strongest positions overlap by a median of 93%. In addition, the 65K dictionary yields no OPD-dominated or OPD-specific feature.

2) Varying only the author of the rollouts. W2 notes that the defining aspect of OPD, training on the student's own rollouts, is not varied, and the warm-up also differs from OPD in loss, LoRA, and data. We therefore train two students from the same base with the same teacher (Qwen3-4B-Base-GRPO), distillation loss (a KL on the teacher's top-16 tokens), full-parameter optimizer, prompts, prompt order, and number of trained tokens per prompt; the rollouts are written either by the current student (on-policy) or by the teacher (off-policy). We note that the shared loss isolates the rollout source rather than reproducing the paper's objective. Each group runs 64 steps with two seeds and is read with the paper's dictionary and swap readout on 7,562 pre-specified features. Checkpoints are selected on a calibration set to match the activation change.

Seed 42Seed 43
Activation change, on / off8.26% / 8.36%7.77% / 7.70%
Features within ±20% of base rate, on / off99.2% / 99.3%99.1% / 99.6%
Cosine between on and off reweighting vectors0.680.65
Usage of feature 32128 relative to base, on / off0.08× / 0.34×0.08× / 0.30×

The on−off difference vectors of the two seeds are positively correlated (cosine 0.23; 0.22 at equal steps and 0.40 near the scale of OPD), and the largest differences fall on the same features in both seeds. It is perhaps worth noting that near the activation change of OPD (5.4%–5.9%), the cosine is again 0.66–0.68.

3) Implications for W2 and Q2. As for absorption, part 1 shows that a new component is flagged unless it is nearly parallel to an existing feature, that the lowered features keep their contexts, and that a larger dictionary resolves no new OPD feature. With Table 3, the reweighting is therefore a measured and functional change, rather than a relabeling of undetected features. As for Q2, the result that differs at the same activation change is the direction of reweighting. Both rollout sources reweight shared features only slightly (about 99% of frequent features within ±20%), but they reweight different features, and part of this difference reproduces across seeds on the same features. This agrees with Section 4.2, where more than half of the warm-up's change lies outside OPD's direction, and adding it to a directly distilled student still raises accuracy (Table 3). Finally, the findings do not describe post-training on a teacher in general: SFT on solutions written by a stronger model already yields an SFT-specific feature at an activation change of 7.2% (response to W1), and off-policy distillation from the same teacher reweights in a different direction. This agrees with the central conclusion of the current paper: OPD reweights the features that the student already shares with the teacher, and training on the student's own rollouts affects which features are reweighted.

Dear Reviewer Ay8n,

We would like to express our sincere gratitude once again for the time and effort you have devoted to reviewing our work. As the discussion period draws to a close, we would be grateful if you could let us know whether our responses have adequately addressed your concerns. Should you have any further questions or comments, please do not hesitate to share them. We look forward to continuing our communication with you.

Best, Authors

Reviewer_iFMZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how on-policy distillation changes internal representations using sparse crosscoders. Its swap readout places a student's activations in both student slots, cancelling the poorly constrained difference between the two encoders. Across three teacher settings, the detected feature inventories remain stable while feature usage changes. A further study of SFT warm-up finds partially aligned changes and shows that transferring the decoded warm-up change can recover much of its downstream benefit.

Critical strengths
  1. The swap readout addresses a concrete measurement problem. The algebra in Appendix B explains its invariance to the student-encoder difference, and the reconstruction results include checkpoints unseen during crosscoder training. The random-direction, noise and disjoint-token controls also strengthen the warm-up alignment analysis.

  2. The intervention goes beyond correlating feature counts with accuracy. Adding the decoded warm-up change improves the directly distilled student, while removing it hurts the warmed-up student; both reported bootstrap intervals exclude zero. The shuffled-feature controls and detailed feature examples make this a useful study of representation changes during distillation.

Critical weaknesses

The conclusion that OPD creates no new features extends beyond the measured representation. Appendix C.4 uses one middle-layer site per model and a fixed 32,768-feature dictionary with k=50. The reconstruction checks support measurements in this representation, but stability here cannot rule out changes elsewhere or features the dictionary does not resolve. The conclusion should be restricted to the detected features at the measured sites. This limits the generality of the claim, rather than invalidating the observed stability.

Questions for authors

Can the no-new-features statements in Section 3.1 and the conclusion be explicitly limited to the measured layers and dictionary? If results for another layer or dictionary size are already available, do they support the same observation?

3 comments
AuthorsOpenReview ↗

W1/Q1: dictionary size and another layer

We thank the reviewer for the question. We address the two possibilities raised in W1, and then the two parts of Q1.

1) W1: features that the dictionary does not resolve. The crosscoders of the current paper read the residual stream of a middle block with 32,768 features, trained on equal parts of OpenThoughts-114k and RedPajama, as in Shi et al. (2026), where SFT is found to add features. Whether such a dictionary resolves new features at this site is tested directly by positive controls, in which new features are known to be present (Appendix F). The crosscoder finds the features that SFT adds to the student, both after 6 and after 242 steps (1 and 10 features dominated by the SFT student, against 0 for OPD). In synthetic activations, it recovers an inserted new feature whose activation change equals that of OPD, and it misses the feature only when its direction nearly coincides with that of an existing feature (cosine 0.99). A new feature of OPD that the dictionary merges with others for lack of capacity would have more room to separate in a larger dictionary. We therefore train, for the student of Section 4, three-model crosscoders of the base, OPD, and SFT students with 32,768 and 65,536 features at blocks 14 and 21 of 28, all with the same data, sparsity (k = 50), number of updates, and held-out positions.

BlockDictionaryOPD: MAS > 0.5 / NRN > 0.8SFT: MAS > 0.5 / NRN > 0.8Joint FVE (base / OPD / SFT)
1432,7680 / 01 / 10.815 / 0.821 / 0.813
1465,5360 / 01 / 10.823 / 0.829 / 0.820
2132,7680 / 05 / 00.815 / 0.821 / 0.813
2165,5360 / 03 / 00.823 / 0.829 / 0.821

At both blocks, doubling the dictionary raises the FVE of every model by about 0.008, i.e., it resolves structure that the smaller dictionary does not; yet the OPD student gains no dominated or specific feature, whereas the SFT student keeps its specific feature at block 14 and still dominates features at block 21.

2) W1: changes at other sites. The residual stream at the output of block 14 accumulates the outputs of all earlier blocks, so a feature that OPD writes in an earlier block and that persists to this block is part of the measured representation; a feature that arises only in later blocks is not. We therefore repeat the analysis at block 21. There, the SFT student dominates five features in the dictionary of 32,768 and three in that of 65,536 (MAS > 0.5), whereas the OPD student dominates none in either. None of these reaches NRN > 0.8 (the largest is 0.74 with 32,768 features), so at this block MAS separates the two students, whereas NRN flags neither. Moreover, block 14 carries the change that matters for accuracy: adding the warm-up's feature-level change at this block to a directly distilled student raises its avg@8 from 18.6% to 21.3%, close to the 21.9% of the warmed-up student, and removing it from the warmed-up student lowers its avg@8 to 19.2%, close to direct OPD (Table 3).

3) Q1. As for the first part, Section 3.1 now states that OPD "adds no feature that the crosscoder identifies", and a new Limitations paragraph in the conclusion states the layers, dictionary sizes, and token distribution that the measurement covers. As for the second part, both the dictionaries with 65,536 features and the crosscoders at block 21 support the same observation (table above).

4) Implications for W1 and Q1. The same observation holds across the settings of the revision: in the three crosscoders of Section 3, with three teachers, and in the five new crosscoders of Appendix F, with a second student family, two blocks, and two dictionary sizes, the OPD student dominates no feature. Consequently, the stability of the feature inventory does not stem from a dictionary that cannot resolve new features or from the choice of block: at block 14, the dictionary resolves new features when they are present and a dictionary twice as large finds none for OPD; at block 21, the SFT student dominates features in both dictionaries whereas the OPD student dominates none. This agrees with the central conclusion of the current paper: OPD reweights the features that the student already shares with the teacher, rather than acquiring new ones. All new results are in Appendix F of the revision, marked in blue.

The response and the revisions address most of the issues I raised. I have no further questions at this stage and will keep my original recommendation.

We appreciate your thoughtful review and the time you took to assess our revisions. Your feedback helped us improve the manuscript. Have a very nice day!

Reviewer_CpgUOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how on-policy distillation changes the internal representations of student language models using sparse crosscoders. The authors propose a swap readout method to measure changes in feature activation between student checkpoints, addressing an encoder identifiability issue associated with the highly correlated representations of pre- and post-OPD models. Applying this method across three OPD settings, they find little evidence of newly acquired student-specific features or transferred teacher-specific features, with most existing features exhibiting relatively small changes in aggregate firing rate. They further observe that some of the most changed features are associated with reasoning decision tokens, where teacher–student disagreement is also higher. The paper also investigates the effects of an SFT warm-up before OPD, finding that it changes the use of existing features in ways that partly align with direct OPD. Feature-level interventions show that transferring the activation changes associated with the warm-up to a directly distilled student can recover a substantial portion of the warm-up's performance benefit.

Critical strengths

The proposed swap readout is a useful contribution that addresses a well-motivated limitation of standard crosscoder analyses when comparing closely related model checkpoints. The authors provide an interesting empirical characterization of how representations change during OPD across multiple teacher settings. The feature-level intervention experiments also provide useful causal evidence connecting the identified representation changes to performance.

Critical weaknesses

My primary concern is the lack of validation of the crosscoder pipeline's sensitivity to newly acquired features, given that the absence of feature acquisition is central to the paper's conclusions. The absence of detected new features is only as informative as the method's ability to detect them when they are present. Without a positive control or characterization of detection sensitivity, I find the categorical claim that OPD does not acquire new features stronger than the evidence currently supports.

Questions for authors

Could the authors validate the feature-acquisition detection pipeline in a setting previously shown to induce model-specific features, such as the SFT setting discussed in Shi et al. (2026) or similar? Additionally, a controlled synthetic experiment characterizing detection sensitivity for newly introduced features of varying strength, activation frequency, and similarity to existing features would meaningfully strengthen my confidence in the paper's central conclusion. And more generally, how does the sensitivity of the crosscoder analysis to features of varying strength and prominence affect the extent to which the absence of detected new features can be interpreted as evidence that no new features were acquired during OPD?

2 comments
AuthorsOpenReview ↗

Response to Reviewer CpgU: validation of detection sensitivity

We thank the reviewer. The statement in Section 3.1 concerns the features resolved by the crosscoder at the measured site, and we will make this explicit in Section 3.1, the abstract, and the conclusion. To examine whether the observation depends on the site or on the dictionary size, we repeat the analysis at a second layer and with a doubled dictionary.

Setting. We use the OPD student of Section 4 (Qwen3-1.7B-Base distilled from Qwen3-4B-Base-GRPO), together with the SFT model described in our response to Reviewer CpgU, which serves as a positive control. Joint crosscoders of the base, OPD, and SFT models are trained at block 14 (the site of the current paper) and at block 21 of 28 (counting from 0), each with 32,768 and 65,536 features, with k = 50 and the same corpus, training budget, and 3.02M held-out positions.

Block / dictionaryOPD: MAS > 0.5 / NRN > 0.8SFT: MAS > 0.5 / NRN > 0.8FVE (base / OPD / SFT)
14 / 32K0 / 01 / 10.815 / 0.821 / 0.813
14 / 65K0 / 01 / 10.823 / 0.829 / 0.820
21 / 32K0 / 05 / 00.815 / 0.821 / 0.813
21 / 65K0 / 03 / 00.823 / 0.829 / 0.821

Results. In all four combinations, OPD has no feature that it dominates (MAS > 0.5) and none that is specific to it relative to the base (NRN > 0.8). The SFT control yields SFT-dominated features in all four combinations, which shows that each crosscoder registers model-dominated structure where it exists; at block 21, these features reach MAS up to 0.62 and NRN up to 0.74. Doubling the dictionary raises FVE by about 0.008 at both blocks and yields no OPD-dominated feature. We note that the features a dictionary does not resolve are characterized by the sensitivity analysis in our response to Reviewer CpgU, which identifies the strength, frequency, and overlap at which an added feature is resolved.

Conclusion. Consequently, within the range tested, the observation does not depend on the site or on the dictionary size: at two layers and two dictionary sizes, OPD dominates no feature, whereas SFT does. Nevertheless, two layers and two sizes do not cover the whole network. We therefore state the result as the stability of the features resolved at the measured sites and dictionary sizes, and add these results to the appendix. The measured findings, namely the stability of feature usage, the concentration of the largest changes at decision tokens, and the causal effect of the decoded warm-up change, are unaffected by this qualification, and they support the central conclusion of the current paper: OPD primarily reweights the features that the student already shares with the teacher, rather than acquiring new ones.

Thank you for the clarifications.

Comments on the paper

1 comment

Dear Area Chair, Senior Area Chair, and Program Chairs,

We thank all reviewers for their time and feedback. All three reviews find that the swap readout addresses a limitation of standard crosscoder analyses (CpgU, iFMZ, Ay8n), and two find that the feature-level intervention provides causal evidence (CpgU, iFMZ). The questions raised concern the evidence for the central conclusion of the current paper, namely that OPD reweights the features the student already shares with the teacher rather than acquiring new ones. This conclusion rests on two measurements: the crosscoder finds no feature specific to the OPD student, and the swap readout measures which shared features OPD reweights, a reweighting that changes accuracy when imposed on a student (Table 3). The reviews ask three questions about these measurements. The revision answers each with new experiments, collected in two new appendices and marked in blue.

1) Would the analysis find a new feature in a change as small as that of OPD? (CpgU; Ay8n, W1/Q1) We measure the change by the relative change of the activations, ‖H − H_base‖_F/‖H_base‖_F, which is 5.0% for OPD.

  • Positive control, real training. We apply the SFT recipe of Shi et al. (2026) to the student of Section 4. After 6 steps (7.2%), the crosscoder finds a feature specific to the SFT student (MAS 0.80, NRN 0.87), also with 65,536 features; after 242 steps, it finds 10 features dominated by the SFT student. The OPD student has none in every crosscoder. At step 4 (4.1%, below OPD), SFT already uses this feature 3.8 times as often as the base student, whereas OPD uses it 0.14 times as often.
  • Positive control, synthetic. A new latent inserted at the activation change of OPD is recovered in all seeds. Among directions with cosine 0, 0.9, and 0.99 to an existing latent, it is missed only at 0.99; weaker or rarer latents can go undetected, as the Limitations paragraph states.
  • Negative control. An existing latent amplified to the same activation change is not counted as new (MAS ≤ 0.49, NRN ≤ 0.66).

2) Does the result depend on the layer and the dictionary? (iFMZ) With 65,536 features at block 14, the OPD student has no dominated or specific feature, whereas the SFT student keeps one. At block 21 with 32,768 features, the SFT student dominates five features (none reaches NRN > 0.8) and the OPD student none. A crosscoder with 65,536 features at block 21 is being trained and will be reported. A Limitations paragraph states the layers, dictionary sizes, and token distribution that the measurement covers.

3) Could the reweighting be an absorbed new feature, and which result is specific to OPD? (Ay8n, W2/Q2)

  • Absorption. An inserted component is flagged at cosine 0 and 0.9 with an existing feature, and merges with it only at 0.99. The 50 features that OPD lowers most keep firing on the contexts they fired on before OPD (median correlation 0.91 of token-conditional activations; median overlap 93% of the 100 strongest positions).
  • Rollout source. With the teacher, the loss, and the prompts fixed, we vary only who writes the rollouts. At matched activation change, both students reweight shared features only slightly (about 99% of frequent features within ±20%), but they reweight different features (cosine 0.65–0.68), and part of the difference reproduces across seeds on the same features; e.g., feature 32128 falls to 0.08× on-policy and to 0.30–0.34× off-policy.

Revisions (in blue).

  • Section 2: NRN is attributed to Baek & Tegmark (2025).
  • Section 3.1: a paragraph on the sensitivity of the analysis, pointing to Appendix F, and the statement that OPD adds no feature that the crosscoder identifies.
  • Section 4.1: the contrast with SFT on solutions of a stronger model, reproduced on our student. Section 4.2: a pointer to Appendix G.
  • Conclusion: a Limitations paragraph.
  • Appendix C.4: the new crosscoders. Appendix F (new): sensitivity analysis. Appendix G (new): varying only who writes the rollouts.

Together, these results support the central conclusion of the current paper: OPD reweights the features the student already shares with the teacher, rather than acquiring new ones, and training on the student's own rollouts affects which features are reweighted. Detailed responses follow under each review.

Sincerely, Authors of Submission 53615

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.