- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates how the order of sequential on-policy distillation affects benchmark performance and model weights. The authors find that resetting the optimizer reduces the benchmark order gap while largely preserving the norm of the parameter difference between the two training orders. They further analyze this difference, termed the order imprint, through several geometric and functional experiments.
Critical strengths
-
I appreciate the attempt to look beyond benchmark scores when studying sequential post-training. The question of how different capability operators interact is relevant, especially with the growing interest in multi-teacher distillation and capability integration.
-
The authors put considerable effort into experimental controls, including multiple seeds, null comparisons, and transparent reporting of unsuccessful experiments.
Critical weaknesses
-
I am not fully convinced of the significance of the main finding. Benchmark performance and weight-space differences are inherently different quantities, so I would not expect them to correspond directly in the first place. Related work such as Neural Thickets [1] has already shown that task-dependent behaviors can vary considerably within a model's weight-space neighborhood. What remains unclear to me is what we actually learn from the order imprint beyond knowing that different training trajectories produce different weights. Does it tell us anything about capability interference or future adaptation? I would have liked to see a clearer motivation for why this imprint matters.
-
Some conclusions seem stronger than what the experiments support. In Sec. 4, the authors claim that the imprint is largely preserved after optimizer reset. However, while its norm barely changes, the cosine similarity is only around 0.55, suggesting a considerable directional change. The reduction in benchmark order effect is also relatively weak statistically (around 1.5 standard errors for the paired reduction). I am also not entirely convinced by the interpretation of the unexplained imprint in Secs. 5–6, since the explained proportion depends heavily on the chosen reference directions.
-
The connection to practical capability integration is limited, and the presentation could be clearer. Recent work such as MOPD [2] investigates how different teacher capabilities can be integrated into one student. I understand that this paper focuses on analysis rather than improving training, but I find it difficult to see what the measured imprint tells us about capability composition in practice. The paper was also difficult to follow despite careful reading. In particular, the motivation for moving from benchmark effects to parameter-space analysis is not clearly established, and several interpretations are introduced before their significance becomes clear.
[1] Gan et al. "Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights", ICML 2026.
[2] Ma et al. "MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training".
Questions for authors
Q1. What does the order imprint tell us beyond the fact that different training orders lead to different weights? Why is this distinction important in practice?
Q2. Have you considered training both order endpoints on a third task to see whether their different imprints affect future learning or capability retention?
Q3. If the imprint is largely preserved after optimizer reset, how should we interpret the relatively low cosine similarity (~0.55)? Could the unexplained component simply reflect differences between the two training trajectories?