- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes ACSD (Activation-Conditioned Self-Distillation) as an alternative to context-conditioned self-distillation, whose idea is to collect correct/short responses, to calculate mean differences in activations for them v/s incorrect answers, and then use these differences to construct a self-teacher by using them as steering vectors. Results show that on two R1-distilled Qwens and Olmo-3-7B-Think, there are small but consistent gains with ACSD vs other methods on AIME/HMMT/LiveCodeBenchv6. Ablation studies show that the method mainly upweights tokens like “wait,” “however,” etc. and also that steering with random vectors loses the gains.
Critical strengths
The positive angle to my assessment is stood up by the following:
- The one critical strength here is that the gains are small but consistently higher than other methods like SFT and GRPO.
- A less salient but still noteworthy strength is that averaging out the steering vector over several tokens makes the method less likely to break the prose as OPSD often does.
Critical weaknesses
Here are the main weaknesses that inform my assessment.
- The fact that most of the upweighted words are turning words like however, wait, and so on; I fear that all this method (and others, to be fair) is doing is to upweight these words. See [1] for a recent analysis of this problem. In this case I have doubts whether we are just finding the right words to steer the model than actually have it learn.
- I have personally run experiments with most of these models on these same benchmarks, and in the past I have seen a majority of the gains in my experiments come from problems simply go from truncated responses to non-truncated responses. Given that ACSD explicitly favors shorter responses I fear this may also be happening here.
- The gains of ACSD are larger than other methods but none of the methods actually gains a lot of accuracy - the base model is often less than 3-4 points away from the best model on all benchmarks. While this is not an issue with this specific method, I believe it is a big factor in whether the class of *PO or *SD methods are promising for the next generation of training algorithms.
- Having to calculate and use steering vectors brings practical problems: (a) we may want to do post-training on a general set of prompts where I have a continuous reward and not a “correct” and “incorrect” set (b) if I do general post-training on a prompt mixture, I now have to catalog each prompt into its corresponding set for steering (c) some model training endpoints like Tinker may not expose enough internal activations for this method to be usable.
[1] “Base Models Can Reason By Taking a Cue From Training Data,” Wang et. al., arXiv 2026.
Questions for authors
I have the following questions for the authors:
- How often do the responses get truncated by sequence length in your experiments with the base model as well as after different methods? On problems with the 32 (say) largest avg@k gains, what were the truncation fractions before/after?
- Have you attempted GRPO where only the “correct and short” answers are given a reward of 1 instead of “correct” answers? If we call this GRPO (S), how does GRPO (S) compare with GRPO and ACSD in your experiments?
- Same question as the second, with OPSD (S) as the variant of OPSD where we only distill from correct and short responses as conditioning.