- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work study on On-policy self-distillation domain which is turn the sparse signal of reinforcement learning with verifiable rewards into token-level supervision. The motivation is to detect the value of privileged changes during the distillation period. The authors utilize representation shifts and find that shifts are more concentrated, which means downstream gains despite revealing less information. To this end, this work proposes Privileged Representation Isolation for On-Policy Self-Distillation (PreSO). PreSO combines the source and target stage to limit the detail entering the shift from the representation-level distillation target. The experimental results under five benchmarks show that PreSO is the best baseline across the Qwen variants and sustains its gains through training.
Critical strengths
-
The motivation of this paper is clear and insightful. Finding that the student should internalize part from teacher is a reliable task.
-
The presentation is clear and easy to follow. All figures and tables are nice and comprehensive. The pipeline in Figure 3 clearly lets me understand how PreSO works.
-
The results of PreSO are comprehensive and well support the claim in this paper.
Critical weaknesses
-
The fact that a certain direction possesses significant energy across different samples does not imply that it offers stronger generalization capability for inference. The insight of this paper should have more clarification.
-
Lack of a comparison with other representation-distillation baselines.
-
Can the authors provide more detail on the optimization in Eqs. 7- 8? The reason for constructing a complete projected target rather than directly using MSE should be clearer.
Questions for authors
-
I wonder about the cross-dataset transferability of PreSO. Although the authors evaluate three Qwen3 models of different scales, two training datasets, and five mathematical benchmarks, they are still in the same domain competition-level mathematical reasoning. Can the authors propose the results when transferred to other tasks, for example, Logic reasoning and program synthesis?
-
In Table 2, when removing cluster routing, the performance on Qwen3-1.7B becomes better. Can the authors propose more experiments or clarifications?
-
The frozen subspace maintains a relatively stable shift-energy retention during training in the appendix. But can the authors provide a comparison with periodically updated subspace deployment?
6 comments
Response to W3 and Q1
W3: Clarifying target construction and optimization in Eqs. 7–8.
To clarify, PreSO uses normalized MSE: Eq. 7 defines the target, while Eq. 8 applies the matching loss. The design concerns which privileged corrections to transfer. After warm-up, let denote the normalized unprivileged student representation, the privileged shift, and the cluster-specific projector. We construct
h^\star = \operatorname{sg}\left(\frac{u}{\lVert u\rVert_2}\right).$$ The student anchor is needed because a projected shift is a correction, not a complete representation. Before normalization, $$(I-P_c)u = (I-P_c)h_s.$$ Thus, the target adds a correction only along the selected directions while retaining the student’s representation in the orthogonal complement before normalization. Directly matching the full privileged teacher would instead supervise all teacher-induced changes. We normalize the target and compute MSE over the **full hidden vector**. Stop-gradient detaches the entire target, so gradients flow only through the student representation being optimized. **Q1: Applicability beyond mathematical reasoning.** Following the reviewer’s suggestion to evaluate tasks beyond mathematics, we additionally train Qwen3-1.7B and Qwen3-4B on **RaR-Medicine for 400 optimization steps** and evaluate on three open-ended medical QA benchmarks. We directly use **unmet rubric criteria as privileged information**, extending the evaluation beyond solution-derived mathematical hints. | Model | Method | RaR-Medicine | HealthBench | LLMEval-Med | | :--------- | :-------- | -----------: | ----------: | ----------: | | Qwen3-1.7B | Base | 39.4 | 51.1 | 25.5 | | | OPSD | 35.5 | 45.8 | 23.5 | | | GRPO | 37.3 | 54.3 | 25.6 | | | **PreSO** | **39.7** | **55.8** | **26.5** | | Qwen3-4B | Base | 49.7 | 59.5 | 38.1 | | | OPSD | 53.7 | 51.4 | 33.1 | | | GRPO | 57.8 | 62.7 | 39.1 | | | **PreSO** | **62.0** | **63.9** | **39.3** | PreSO outperforms both OPSD and GRPO on all three benchmarks at both model scales, exceeding GRPO by **0.2–4.2 points**. These additional results support its applicability beyond competition-level mathematics to open-ended medical QA. We appreciate the reviewer’s suggestion, which prompted this broader evaluation.Response to Q2 and Q3
Q2: Effect of cluster routing on Qwen3-1.7B.
Thank you for pointing out this benchmark-specific result. Removing routing improves AIME24 by 0.2 points at the best checkpoint and 0.5 points at the last checkpoint. On AIME25, however, it reduces accuracy by 1.4 and 3.1 points, respectively. On AMC23, best-checkpoint accuracy decreases by 1.7 points, while final accuracy remains unchanged.
To clarify the overall effect, we summarize the averages across the three benchmarks in Table 2:
| Model | Configuration | Best-checkpoint Avg. | Last-checkpoint Avg. |
|---|---|---|---|
| Qwen3-1.7B | Without routing | 28.7 | 28.1 |
| With routing (PreSO) | 29.7 | 29.0 | |
| Qwen3-8B | Without routing | 44.2 | 42.9 |
| With routing (PreSO) | 47.8 | 47.7 |
Routing improves aggregate performance at both scales, although its benefit varies across benchmarks. Its purpose is to estimate correction subspaces among related problems, rather than pool heterogeneous shifts into a single subspace. These results support that design overall, with a larger observed benefit on Qwen3-8B. We will clarify the mixed effects on Qwen3-1.7B alongside the aggregate improvements.
Q3: Frozen versus periodically updated subspaces.
Following the reviewer’s suggestion, we add a direct comparison on Qwen3-1.7B between freezing the subspaces after warm-up and re-estimating them every 100 optimization steps:
| Subspace strategy | AIME24 | AIME25 | HMMT25 | Training time |
|---|---|---|---|---|
| Frozen after warm-up | 18.9 | 15.6 | 9.7 | 6h 18m |
| Updated every 100 steps | 20.0 | 16.1 | 10.3 | 6h 43m |
Periodic updating improves accuracy by 1.1, 0.5, and 0.6 points, respectively, with an additional 25 minutes of training time (approximately 6.6%). This comparison shows that refreshing the subspaces can provide further gains within PreSO.
The energy-retention analysis in Appendix D.3 indicates stable coverage of evolving privileged shifts by the frozen subspaces, but does not establish their optimality throughout training. The new results complement that diagnostic: freezing is an effective, simpler default, while periodic updating offers additional accuracy at a modest increase in training time. We will incorporate this comparison and clarify that freezing is a practical choice rather than a requirement of the method.
Response to W1 and W2
Thank you for the constructive comments! We hope the following responses can address your concerns.
W1: Clarifying the relationship between shift energy and generalization.
We thank the reviewer for highlighting this distinction. High energy across samples does not, by itself, imply better generalization, and Appendix B.1 clarifies how we interpret the leading subspace. We use it as an empirical proxy for cross-problem structure, rather than a guaranteed separation of transferable and instance-specific information.
Our support for this choice comes from controlled evidence beyond energy concentration:
Together, these results support the selected subspace as an effective empirical choice for constructing privileged corrections. Its usefulness is evaluated through downstream performance without privileged information, rather than inferred from energy concentration alone. We will make this distinction more prominent in the main text.
W2: Comparisons with representation-distillation baselines.
We thank the reviewer for suggesting these comparisons. We add the classical representation-distillation methods FitNets and RKD, together with the recent on-policy method OPRD. We also implement OPRSD, our adaptation of OPRD’s matching loss to privileged self-distillation, where the model conditioned on privileged information serves as its own teacher. This complements OPRD’s separate-teacher setting with a representation-level self-distillation baseline.
PreSO achieves higher scores than all compared representation-distillation baselines on the three benchmarks. Relative to OPRSD, the improvements are 2.5, 3.4, and 3.0 points on AIME24, AIME25, and HMMT25, respectively. These results provide additional support for selecting privileged corrections beyond using representation-level matching alone. We appreciate this suggestion and will incorporate the comparisons and baseline definitions into the revised manuscript.
References
[1] Romero, A., et al. FitNets: Hints for Thin Deep Nets. ICLR, 2015.
[2] Park, W., Kim, D., Lu, Y., and Cho, M. Relational Knowledge Distillation. CVPR, 2019.
[3] Yang, S., et al. OPRD: On-Policy Representation Distillation. arXiv:2606.06021, 2026.