acceptodds
Under review as a conference paper at ICLR 2027

Activation-Conditioned Self-Distillation

Abstract

On-policy self-distillation uses a model as its own teacher to provide dense supervision for reasoning, often through reference-solution conditioning. Providing privileged information does not by itself ensure effective token-level supervision throughout long responses. We introduce Activation-Conditioned Self-Distillation (ACSD), which extracts a steering vector by contrasting activations of self-generated trajectories that reach verified correct answers within a generation budget with those of all remaining trajectories. A frozen copy of the base model applies this vector at each prediction position, and the student learns from its next-token distributions on student-generated prefixes. Outcome verification is used for direction construction and calibration; distillation requires neither problem-specific reference text nor teacher parameter updates. The distilled student is used alone at inference. On each of five models, ACSD achieves the highest mean accuracy over four mathematical benchmarks among the evaluated methods. On DeepSeek-R1-0528-Qwen3-8B, mean mathematical accuracy reaches 71.9% and LiveCodeBench v6 pass@12 reaches 70.9%, compared with 69.0% and 66.3% for the reference-conditioned OPSD baseline. Contrasts among correct trajectories also support distillation, and extracted directions can be reused across mathematical training datasets. On fixed student trajectories, ACSD maintains more stable late-position logit-update magnitudes than OPSD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On Policy Self Distillation, Large Language Model, Activation Steering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MMqBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ACSD (Activation-Conditioned Self-Distillation) as an alternative to context-conditioned self-distillation, whose idea is to collect correct/short responses, to calculate mean differences in activations for them v/s incorrect answers, and then use these differences to construct a self-teacher by using them as steering vectors. Results show that on two R1-distilled Qwens and Olmo-3-7B-Think, there are small but consistent gains with ACSD vs other methods on AIME/HMMT/LiveCodeBenchv6. Ablation studies show that the method mainly upweights tokens like “wait,” “however,” etc. and also that steering with random vectors loses the gains.

Critical strengths

The positive angle to my assessment is stood up by the following:

  • The one critical strength here is that the gains are small but consistently higher than other methods like SFT and GRPO.
  • A less salient but still noteworthy strength is that averaging out the steering vector over several tokens makes the method less likely to break the prose as OPSD often does.
Critical weaknesses

Here are the main weaknesses that inform my assessment.

  • The fact that most of the upweighted words are turning words like however, wait, and so on; I fear that all this method (and others, to be fair) is doing is to upweight these words. See [1] for a recent analysis of this problem. In this case I have doubts whether we are just finding the right words to steer the model than actually have it learn.
  • I have personally run experiments with most of these models on these same benchmarks, and in the past I have seen a majority of the gains in my experiments come from problems simply go from truncated responses to non-truncated responses. Given that ACSD explicitly favors shorter responses I fear this may also be happening here.
  • The gains of ACSD are larger than other methods but none of the methods actually gains a lot of accuracy - the base model is often less than 3-4 points away from the best model on all benchmarks. While this is not an issue with this specific method, I believe it is a big factor in whether the class of *PO or *SD methods are promising for the next generation of training algorithms.
  • Having to calculate and use steering vectors brings practical problems: (a) we may want to do post-training on a general set of prompts where I have a continuous reward and not a “correct” and “incorrect” set (b) if I do general post-training on a prompt mixture, I now have to catalog each prompt into its corresponding set for steering (c) some model training endpoints like Tinker may not expose enough internal activations for this method to be usable.

[1] “Base Models Can Reason By Taking a Cue From Training Data,” Wang et. al., arXiv 2026.

Questions for authors

I have the following questions for the authors:

  • How often do the responses get truncated by sequence length in your experiments with the base model as well as after different methods? On problems with the 32 (say) largest avg@k gains, what were the truncation fractions before/after?
  • Have you attempted GRPO where only the “correct and short” answers are given a reward of 1 instead of “correct” answers? If we call this GRPO (S), how does GRPO (S) compare with GRPO and ACSD in your experiments?
  • Same question as the second, with OPSD (S) as the variant of OPSD where we only distill from correct and short responses as conditioning.
Reviewer_ggUzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Activation-Conditioned Self-Distillation (ACSD). It groups self-generated trajectories by answer correctness and generation length, extracts a steering direction from their activation differences, and constructs a frozen teacher that supervises the student on student-generated prefixes. Experiments cover five models on mathematical reasoning, with an extension to code generation.

Critical strengths
  1. The method is simple and uses a reusable steering direction. Distillation requires no problem-specific reference text, and the resulting student requires no activation intervention at inference.
  2. The experiments cover multiple models and two task types, with additional analyses of direction perturbations and cross-dataset transfer.
  3. The authors report the degradation caused by direct inference-time steering, providing an interesting observation about the relationship between teacher intervention and student learning.
Critical weaknesses
  1. The justification for pooling across positions is insufficient. Pooling activations across reasoning stages and trajectory lengths may conflate correctness with positional and stylistic differences. The existing experiments do not sufficiently disentangle these factors.
  2. The continued effectiveness of the fixed direction is unverified. As the student learns, its prefix distribution changes. The paper does not establish whether the initially calibrated direction continues to provide useful supervision throughout training.
  3. The teacher-student performance discrepancy is insufficiently explained. Direct steering reduces accuracy to 25.6%, whereas the distilled student reaches 71.9%. This is not inherently contradictory, but the paper should clarify why the student acquires beneficial changes without inheriting the teacher’s degraded behavior.
Questions for authors
  1. Does the direction remain effective with position-matched extraction, stage-specific pooling, or exclusion of final-answer spans?
  2. Does teacher supervision remain useful on prefixes generated by successive student checkpoints?
  3. What roles do pointwise loss clipping and the intervention’s starting position play in explaining teacher degradation and student improvement?
Reviewer_EYS5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ACSD constructs a frozen self-teacher using an activation direction contrasting verified, within-budget correct trajectories with other self-generated trajectories. The intervention layer and strength are selected through continuation recovery on a calibration pool. The student then learns from the steered teacher’s distributions on its own generated prefixes using clipped forward KL. Experiments cover five models on mathematical reasoning and one model on code generation, with direction controls and transfer analyses. Outcome verification is required during preparation; inference uses only the distilled student.

Critical strengths
  1. A reusable activation direction supplies context-dependent supervision without problem-specific reference text or teacher parameter updates during distillation. This offers a practical alternative to reference-conditioned self-distillation.

  2. The evaluation spans multiple models and includes random, shuffled, and reversed directions, alternative trajectory contrasts, and direct inference-time steering. These experiments provide meaningful evidence that the extracted direction matters.

Critical weaknesses
  1. The reliability of the reported advantage remains insufficiently established.

Each mathematical benchmark contains 30 problems, and ACSD’s mean margins over the strongest reported comparator range from approximately 0.1 to 1.6 percentage points. No uncertainty for paired differences or variability across training seeds is reported. Appendix A.1 also references OPSD’s checkpoint-selection protocol without clearly stating whether each row represents one checkpoint or separately selected benchmark maxima. These omissions directly affect confidence in the comparative results; the aggregate scores alone do not establish statistical significance.

  1. Method-specific baseline validation is insufficiently documented.

Appendix A.1 assigns SFT, GRPO, OPSD, and ACSD the same learning rate and update count despite their different training objectives and data usage, without reporting method-specific validation or sensitivity. SFT degradation alone does not establish an incorrect baseline, but the current evidence does not show whether the comparison is robust to reasonable baseline tuning.

Questions for authors
  1. Protocol clarification: Does each Table 1 row correspond to a single checkpoint, and what data determine its selection? Please report paired confidence intervals over problems, preserving benchmark structure, for independently selected checkpoints, together with variability across independent training seeds.

  2. Baseline clarification: How were learning rates and training durations validated for each baseline, particularly SFT? Does reasonable method-specific tuning preserve the reported ordering?

  3. Additional evaluation: Using the existing 12 samples per mathematical problem, please report pass@4/8/12 with the standard unbiased estimator. This would help assess whether higher average correctness preserves coverage across problems under repeated sampling.

  4. Mechanism clarification: Appendix A.3 uses 442 trajectories for the length-matched correctness control versus 2048 for ACSD. Does the ablation ordering persist when extraction sample counts are matched? Output lengths and truncation rates would also help distinguish changes in solution accuracy from changes in completion within the generation budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.