acceptodds
Under review as a conference paper at ICLR 2027

PreSO: Privileged Representation Isolation for On-Policy Self-Distillation

Abstract

On-policy self-distillation (OPSD) uses a self-teacher conditioned on privileged information to turn the sparse outcome signal of reinforcement learning with verifiable rewards (RLVR) into dense token-level supervision. Yet the teacher's advantage may depend on information unavailable to the student at inference, raising a central question: which privileged changes are worth distilling? We use cross-problem recurrence as an empirical cue for potentially reusable guidance. Comparing the representation shifts induced by full solutions and compact hints, we find that hint-induced shifts are more concentrated and directionally consistent across problems, and yield more sustained downstream gains despite revealing less information. Motivated by this association, we propose Privileged Representation Isolation for On-Policy Self-Distillation (PreSO). At the source, PreSO uses compact hints to limit the solution-specific detail entering the privileged shift. At the target, it estimates a leading low-rank subspace from shifts pooled across related problems, projects the privilege-induced correction onto this subspace, and adds it to the student's unprivileged representation to form the representation-level distillation target. The selected subspace serves as an operational proxy for recurring structure, rather than a guaranteed separation of transferable and instance-specific information. On five competition-level mathematics benchmarks, PreSO consistently outperforms strong baselines across Qwen3-1.7B, 4B, and 8B, improving average accuracy over the base model by 6.8, 9.0, and 10.0 points. Beyond peak accuracy, PreSO attains the highest final-checkpoint average at every model scale, with improvements that persist through the end of training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, On-Policy Self-Distillation, Representation Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9KgTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work study on On-policy self-distillation domain which is turn the sparse signal of reinforcement learning with verifiable rewards into token-level supervision. The motivation is to detect the value of privileged changes during the distillation period. The authors utilize representation shifts and find that shifts are more concentrated, which means downstream gains despite revealing less information. To this end, this work proposes Privileged Representation Isolation for On-Policy Self-Distillation (PreSO). PreSO combines the source and target stage to limit the detail entering the shift from the representation-level distillation target. The experimental results under five benchmarks show that PreSO is the best baseline across the Qwen variants and sustains its gains through training.

Critical strengths
  1. The motivation of this paper is clear and insightful. Finding that the student should internalize part from teacher is a reliable task.

  2. The presentation is clear and easy to follow. All figures and tables are nice and comprehensive. The pipeline in Figure 3 clearly lets me understand how PreSO works.

  3. The results of PreSO are comprehensive and well support the claim in this paper.

Critical weaknesses
  1. The fact that a certain direction possesses significant energy across different samples does not imply that it offers stronger generalization capability for inference. The insight of this paper should have more clarification.

  2. Lack of a comparison with other representation-distillation baselines.

  3. Can the authors provide more detail on the optimization in Eqs. 7- 8? The reason for constructing a complete projected target rather than directly using MSE should be clearer.

Questions for authors
  1. I wonder about the cross-dataset transferability of PreSO. Although the authors evaluate three Qwen3 models of different scales, two training datasets, and five mathematical benchmarks, they are still in the same domain competition-level mathematical reasoning. Can the authors propose the results when transferred to other tasks, for example, Logic reasoning and program synthesis?

  2. In Table 2, when removing cluster routing, the performance on Qwen3-1.7B becomes better. Can the authors propose more experiments or clarifications?

  3. The frozen subspace maintains a relatively stable shift-energy retention during training in the appendix. But can the authors provide a comparison with periodically updated subspace deployment?

6 comments
AuthorsOpenReview ↗

Response to W1 and W2

Thank you for the constructive comments! We hope the following responses can address your concerns.

W1: Clarifying the relationship between shift energy and generalization.

We thank the reviewer for highlighting this distinction. High energy across samples does not, by itself, imply better generalization, and Appendix B.1 clarifies how we interpret the leading subspace. We use it as an empirical proxy for cross-problem structure, rather than a guaranteed separation of transferable and instance-specific information.

Our support for this choice comes from controlled evidence beyond energy concentration:

  • Cross-problem structure (Appendix B.2, Figures 7–8): Correct hints exhibit greater subspace overlap than length- and shift-norm-matched wrong or cross-problem hints, alongside more sustained downstream gains. The overlap metric measures directional agreement without weighting by singular values after subspace estimation.
  • Direction versus magnitude (Appendix B.3, Figure 9): Top-r remains ahead when Random-r and Tail-r corrections are rescaled to match its pre-normalization correction norm, supporting a role for the selected directions beyond correction magnitude alone.
  • Geometry is not sufficient (Appendix D.2, Figure 14): Concept-only guidance also induces low-rank, directionally consistent shifts but underperforms hints, showing that these geometric properties alone do not establish useful supervision.

Together, these results support the selected subspace as an effective empirical choice for constructing privileged corrections. Its usefulness is evaluated through downstream performance without privileged information, rather than inferred from energy concentration alone. We will make this distinction more prominent in the main text.

W2: Comparisons with representation-distillation baselines.

We thank the reviewer for suggesting these comparisons. We add the classical representation-distillation methods FitNets and RKD, together with the recent on-policy method OPRD. We also implement OPRSD, our adaptation of OPRD’s matching loss to privileged self-distillation, where the model conditioned on privileged information serves as its own teacher. This complements OPRD’s separate-teacher setting with a representation-level self-distillation baseline.

MethodAIME24AIME25HMMT25
Qwen3-1.7B12.89.45.6
FitNets14.28.95.8
RKD15.810.86.1
OPRD-Bridge (8B teacher → 1.7B student)16.711.26.1
OPRSD (privileged self-teacher)16.412.26.7
PreSO18.915.69.7

PreSO achieves higher scores than all compared representation-distillation baselines on the three benchmarks. Relative to OPRSD, the improvements are 2.5, 3.4, and 3.0 points on AIME24, AIME25, and HMMT25, respectively. These results provide additional support for selecting privileged corrections beyond using representation-level matching alone. We appreciate this suggestion and will incorporate the comparisons and baseline definitions into the revised manuscript.

References

[1] Romero, A., et al. FitNets: Hints for Thin Deep Nets. ICLR, 2015.

[2] Park, W., Kim, D., Lu, Y., and Cho, M. Relational Knowledge Distillation. CVPR, 2019.

[3] Yang, S., et al. OPRD: On-Policy Representation Distillation. arXiv:2606.06021, 2026.

AuthorsOpenReview ↗

Response to W3 and Q1

W3: Clarifying target construction and optimization in Eqs. 7–8.

To clarify, PreSO uses normalized MSE: Eq. 7 defines the target, while Eq. 8 applies the matching loss. The design concerns which privileged corrections to transfer. After warm-up, let hsh_s denote the normalized unprivileged student representation, Δ\Delta the privileged shift, and PcP_c the cluster-specific projector. We construct

h^\star = \operatorname{sg}\left(\frac{u}{\lVert u\rVert_2}\right).$$ The student anchor is needed because a projected shift is a correction, not a complete representation. Before normalization, $$(I-P_c)u = (I-P_c)h_s.$$ Thus, the target adds a correction only along the selected directions while retaining the student’s representation in the orthogonal complement before normalization. Directly matching the full privileged teacher would instead supervise all teacher-induced changes. We normalize the target and compute MSE over the **full hidden vector**. Stop-gradient detaches the entire target, so gradients flow only through the student representation being optimized. **Q1: Applicability beyond mathematical reasoning.** Following the reviewer’s suggestion to evaluate tasks beyond mathematics, we additionally train Qwen3-1.7B and Qwen3-4B on **RaR-Medicine for 400 optimization steps** and evaluate on three open-ended medical QA benchmarks. We directly use **unmet rubric criteria as privileged information**, extending the evaluation beyond solution-derived mathematical hints. | Model | Method | RaR-Medicine | HealthBench | LLMEval-Med | | :--------- | :-------- | -----------: | ----------: | ----------: | | Qwen3-1.7B | Base | 39.4 | 51.1 | 25.5 | | | OPSD | 35.5 | 45.8 | 23.5 | | | GRPO | 37.3 | 54.3 | 25.6 | | | **PreSO** | **39.7** | **55.8** | **26.5** | | Qwen3-4B | Base | 49.7 | 59.5 | 38.1 | | | OPSD | 53.7 | 51.4 | 33.1 | | | GRPO | 57.8 | 62.7 | 39.1 | | | **PreSO** | **62.0** | **63.9** | **39.3** | PreSO outperforms both OPSD and GRPO on all three benchmarks at both model scales, exceeding GRPO by **0.2–4.2 points**. These additional results support its applicability beyond competition-level mathematics to open-ended medical QA. We appreciate the reviewer’s suggestion, which prompted this broader evaluation.
AuthorsOpenReview ↗

Response to Q2 and Q3

Q2: Effect of cluster routing on Qwen3-1.7B.

Thank you for pointing out this benchmark-specific result. Removing routing improves AIME24 by 0.2 points at the best checkpoint and 0.5 points at the last checkpoint. On AIME25, however, it reduces accuracy by 1.4 and 3.1 points, respectively. On AMC23, best-checkpoint accuracy decreases by 1.7 points, while final accuracy remains unchanged.

To clarify the overall effect, we summarize the averages across the three benchmarks in Table 2:

ModelConfigurationBest-checkpoint Avg.Last-checkpoint Avg.
Qwen3-1.7BWithout routing28.728.1
With routing (PreSO)29.729.0
Qwen3-8BWithout routing44.242.9
With routing (PreSO)47.847.7

Routing improves aggregate performance at both scales, although its benefit varies across benchmarks. Its purpose is to estimate correction subspaces among related problems, rather than pool heterogeneous shifts into a single subspace. These results support that design overall, with a larger observed benefit on Qwen3-8B. We will clarify the mixed effects on Qwen3-1.7B alongside the aggregate improvements.

Q3: Frozen versus periodically updated subspaces.

Following the reviewer’s suggestion, we add a direct comparison on Qwen3-1.7B between freezing the subspaces after warm-up and re-estimating them every 100 optimization steps:

Subspace strategyAIME24AIME25HMMT25Training time
Frozen after warm-up18.915.69.76h 18m
Updated every 100 steps20.016.110.36h 43m

Periodic updating improves accuracy by 1.1, 0.5, and 0.6 points, respectively, with an additional 25 minutes of training time (approximately 6.6%). This comparison shows that refreshing the subspaces can provide further gains within PreSO.

The energy-retention analysis in Appendix D.3 indicates stable coverage of evolving privileged shifts by the frozen subspaces, but does not establish their optimality throughout training. The new results complement that diagnostic: freezing is an effective, simpler default, while periodic updating offers additional accuracy at a modest increase in training time. We will incorporate this comparison and clarify that freezing is a practical choice rather than a requirement of the method.

Reviewer_Su9GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PreSO constructs representation-level self-distillation targets by filtering privileged information at two stages: compact hints reduce solution-specific detail, and cluster-specific low-rank projectors select recurring hint-induced hidden-state corrections. After warm-up, the projected correction is added to the unprivileged student representation and matched with a normalized MSE loss. Experiments on three Qwen3 scales and five math benchmarks show substantial gains and stronger final-checkpoint stability. Extensive appendix controls examine projection direction, correction norm, hint quality, seeds, alternative training data and training-time cost.

Critical strengths

A concrete, well-tested target-design contribution. Projecting only the privileged correction preserves the student’s complementary representation. Persistent empirical gains. Improvements hold across three model scales, both peak and final checkpoints, a second training dataset and fixed-time comparisons

Critical weaknesses
  1. Generalization of the method is insufficiently established. Appendix C.6 tests thinking-mode inference only after non-thinking training. Without a convincing rationale for this restriction or evidence from thinking-mode training or another training domain, it is unclear how broadly the proposed isolation mechanism applies. This materially limits the significance of a method motivated as extracting reusable privileged guidance.

  2. Several consequential design choices remain under-explained and may depend on the setting. Supervising only the last 512 response tokens is supported by the 1.7B positional ablation, but the mechanism, window-size choice and portability across reasoning lengths are unresolved. Similarly, the K sweep in Figure 6 does not provide a principled way to choose K for heterogeneous tasks, while W = 150 fixes calibration in optimizer steps rather than an explicit adequacy criterion. The issue is not an absence of all ablations; it is insufficient evidence that the selected recipe and its interpretation transfer beyond the setting where the choices were validated.

Questions for authors
  1. Appendix C.8 supports the last 512 positions on 1.7B, but why is 512 appropriate? How sensitive are gains to window size and response length, and does the positional advantage persist at larger scales? Can the evidence distinguish reusable reasoning guidance from effects concentrated near final-answer formation?
  2. How should W and K be selected for a new domain without extensive downstream tuning? The K sweep is useful, but what indicates that each cluster has enough calibration data and that the frozen subspaces remain appropriate? Does final prompt-position routing outperform alternative pooling strategies under otherwise matched settings?
6 comments
AuthorsOpenReview ↗

Response to W1

Thank you for the constructive comments! We hope the following responses can address your concerns.

W1: Generalization across training modes and domains.

The reviewer raises an important question about whether PreSO remains effective beyond the training conditions studied in the main experiments. Our main experiments use non-thinking student rollouts, a configuration also adopted in prior self-distillation work [1–3]. To examine broader applicability, we add experiments with thinking-mode training and open-ended medical QA.

1. Training with thinking mode enabled.

We conduct additional experiments with thinking mode enabled during both training and evaluation. Given the higher cost of thinking-mode rollouts and the limited rebuttal period, we train Qwen3-1.7B for 100 optimization steps.

MethodAIME24AIME25HMMT25
Qwen3-1.7B (thinking)47.033.321.9
+ GRPO49.734.422.8
+ OPSD49.435.022.8
+ RLSD49.234.722.5
+ RLCSD50.835.623.6
+ PreSO52.236.925.8

Within this 100-step training budget, PreSO achieves the highest accuracy on all three benchmarks, exceeding the strongest baseline, RLCSD, by 1.4, 1.3, and 2.2 points, respectively. These results support its effectiveness when thinking mode is enabled during training itself, complementing the previously reported thinking-mode inference evaluation.

2. Training in a non-mathematical domain.

We additionally train Qwen3-1.7B and Qwen3-4B on RaR-Medicine for 400 optimization steps and evaluate on three open-ended medical QA benchmarks. Privileged information consists directly of unmet rubric criteria, extending the evaluation beyond solution-derived mathematical hints.

ModelMethodRaR-MedicineHealthBenchLLMEval-Med
Qwen3-1.7BBase39.451.125.5
OPSD35.545.823.5
GRPO37.354.325.6
PreSO39.755.826.5
Qwen3-4BBase49.759.538.1
OPSD53.751.433.1
GRPO57.862.739.1
PreSO62.063.939.3

PreSO outperforms both OPSD and GRPO on all three benchmarks at both model scales, exceeding GRPO by 0.2–4.2 points. These additional results support its applicability beyond competition-level mathematics to open-ended medical QA. We appreciate the reviewer’s suggestion, which prompted this broader evaluation.

References

[1] Zhao, S., et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. ICML, 2026.

[2] Hübotter, J., et al. Reinforcement Learning via Self-Distillation. ICML, 2026.

[3] Lu, Z., et al. Self-Distilled Agentic Reinforcement Learning. arXiv:2605.15155, 2026.

AuthorsOpenReview ↗

Response to W2 and Q1–Q2 (Part 1/2)

W2 and Q1–Q2: Origins, sensitivity, and portability of the design choices.

To clarify how these settings were chosen, we follow OPRD’s [4] suffix-supervision strategy and use a fixed 512-token window. W = 150 and last-prompt-token routing were also fixed as initial defaults, while K = 4 and rank = 128 were selected based on validation-set performance. Below, we examine their sensitivity and practical selection. The ablations below use Qwen3-1.7B.

1. Supervision window and reasoning length.

To assess sensitivity to the supervision budget, we compare three suffix lengths:

Supervision windowAIME24AIME25HMMT25
Last 256 tokens18.615.68.1
Last 512 tokens18.915.69.7
Last 1024 tokens19.114.210.3

All windows substantially improve over the base model, with average scores ranging from 14.10 to 14.73. These results support 512 as a practical supervision budget rather than a uniquely required value.

To examine performance across inference budgets at a larger model scale, we evaluate Qwen3-8B using different generation limits, reporting mean accuracy (%) over AIME24 and AIME25:

Token budgetBest baselinePreSO
2K20.1420.83
4K25.5626.11
8K25.5629.72
16K26.7830.28

PreSO outperforms the strongest baseline at every budget shown, indicating that its gains persist across different generation limits on Qwen3-8B.

2. Warm-up budget W.

To evaluate the initially fixed warm-up duration, we compare three budgets before projected distillation begins:

Warm-up stepsAIME24AIME25HMMT25
5017.814.28.6
10018.115.39.7
15018.915.69.7

All budgets yield substantial gains. Increasing W from 100 to 150 improves average accuracy by 0.37 points, showing that effectiveness does not require precisely 150 steps. In a new domain, the practical consideration is calibration coverage per cluster relative to the remaining projected-distillation budget, rather than a universal step count.

Frozen versus periodically updated subspaces. Freezing is a practical choice that simplifies training and avoids repeated subspace estimation, rather than a requirement of PreSO. We also compare it with updating the subspaces every 100 optimization steps on Qwen3-1.7B:

Subspace strategyAIME24AIME25HMMT25Training time
Frozen after warm-up18.915.69.76h 18m
Updated every 100 steps20.016.110.36h 43m

Periodic updating provides further gains with 6.6% additional training time. Together with the stable held-out energy retention reported in Appendix D.3, these results support freezing as an effective, simpler default while leaving periodic updating available when additional accuracy is desired.

AuthorsOpenReview ↗

Response to W2 and Q1–Q2 (Part 2/2)

3. Cluster count K.

We compare a single pooled subspace with increasingly fine partitions to assess sensitivity to clustering granularity:

Number of clustersAIME24AIME25HMMT25
119.114.28.1
216.713.18.6
418.915.69.7
817.811.49.4

All tested values improve over the base model, indicating that PreSO’s benefits do not depend on a single clustering configuration. K = 4 achieves the highest average performance. The selection trade-off is between capturing task heterogeneity and maintaining sufficient calibration support per cluster, making coarse partitions a reasonable starting point when calibration data are limited.

4. Routing representation.

To directly address the pooling question, we compare last-prompt-token routing with mean pooling:

Routing representationAIME24AIME25HMMT25
Last prompt token18.915.69.7
Mean pooling17.514.79.4

Last-prompt-token routing performs better on all three benchmarks, supporting our default choice. Mean pooling nevertheless retains substantial gains, indicating that effectiveness is not specific to a single routing representation.

Portability beyond mathematics. For a new domain, a practical starting point is to retain the supervision-window, warm-up, and routing defaults, and adjust K and rank using validation performance if needed, considering task heterogeneity and available calibration data. Our new medical experiments reuse the default supervision window, W, and K, with unmet rubric criteria as privileged information. PreSO outperforms GRPO and OPSD across three medical benchmarks at both 1.7B and 4B scales, exceeding GRPO by 0.2–4.2 points. These results support the existing configuration as an effective starting point beyond mathematics, without requiring a new search over all parameters.

References

[4] Yang, S., et al. OPRD: On-Policy Representation Distillation. arXiv:2606.06021, 2026.

Reviewer_P3QXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PreSO, a representation-level self-distillation method for identifying reusable components of privileged guidance in on-policy self-distillation. Specifically, it routes problems into content-based clusters, estimates a cluster-specific low-rank subspace from privileged representation shifts, and projects each shift onto the corresponding subspace before constructing the distillation target. The method is evaluated on Qwen3 models across five mathematical reasoning benchmarks, with comparisons against GRPO, OPSD, RLSD, and RLCSD.

Critical strengths

S1: The paper is well motivated, taking the meaningful perspective that not all changes induced by privileged information are useful or transferable. The wrong-hint comparison in Figure 8 further supports this motivation.

S2: PreSO consistently outperforms the baselines across different model scales and mathematical reasoning benchmarks, and the paper provides a comprehensive set of control experiments and ablations.

S3: The paper is well written, with a clear presentation of the methodology and experimental results.

Critical weaknesses

W1: The experiments focus predominantly on mathematical reasoning, and the paper does not convincingly establish that PreSO generalizes to other domains. The OOD results in Table 9 are mostly marginal for Qwen3-8B, with negative results on several tasks for Qwen3-1.7B.

W2: PreSO requires an additional pipeline to construct privileged hints from high-quality solutions, incurring extra data-generation costs and reliance on external model-based annotation. Its effectiveness in other domains may further depend on the availability of high-quality, domain-specific solutions and reliable hint-generation pipelines, which may limit its practical utility.

W3: The limited OOD gains suggest that applying PreSO to new domains may require domain-specific training. Moreover, its representation-level training requires more wall-clock time than all compared baselines (Table 11), which could make scaling it across multiple domains computationally expensive.

W4: The core idea behind PreSO is reminiscent of activation/representation steering: it extracts representation shifts between privileged and unprivileged conditions and applies their low-rank projections to the student representation. The authors could add a training-free experiment that directly injects these extracted shifts into the student model at inference time. If direct steering is effective, it would support the usefulness of the extracted directions themselves; if PreSO further outperforms direct steering, it would demonstrate the additional value of distilling these directions into the model parameters.

Questions for authors

Please refer to the above comments.

5 comments
AuthorsOpenReview ↗

Response to W1 and W2

Thank you for the constructive comments! We hope the following responses can address your concerns.

W1: Generalization beyond mathematical reasoning.

Our original OOD experiments were designed to assess whether math-only training with PreSO compromises the model’s existing capabilities in unrelated domains, rather than to demonstrate improvements in those domains. To directly evaluate PreSO’s effectiveness when trained outside mathematics, we additionally train Qwen3-1.7B and Qwen3-4B on RaR-Medicine for 400 optimization steps and evaluate on three open-ended medical QA benchmarks. We use Qwen3-32B as the evaluation judge, consistent with the original OOD experiments.

ModelMethodRaR-MedicineHealthBenchLLMEval-Med
Qwen3-1.7BBase39.451.125.5
OPSD35.545.823.5
GRPO37.354.325.6
PreSO39.755.826.5
Qwen3-4BBase49.759.538.1
OPSD53.751.433.1
GRPO57.862.739.1
PreSO62.063.939.3

PreSO outperforms both OPSD and GRPO on all three benchmarks at both model scales, exceeding GRPO by 0.2–4.2 points. These additional results support its applicability beyond competition-level mathematics to open-ended medical QA. We appreciate the reviewer’s suggestion, which prompted this broader evaluation.

W2: Hint-generation costs and applicability to new domains.

Our mathematical implementation does require a solution-to-hint preparation step, which incurs additional data-generation cost. However, the required preparation depends on the domain and the guidance already available in the dataset. For example, in our RaR-Medicine experiments, we directly use unmet rubric criteria as privileged information, without a separate solution-to-hint pipeline. As reported in W1, this formulation outperforms OPSD and GRPO across all three medical benchmarks at both model scales.

When hint generation is needed, it does not necessarily require an external model. We compare our default GPT-5.5 hints with hints generated by the student itself, either online during training or offline beforehand:

MethodHint sourceAIME24AIME25HMMT25Generation time (s/hint)
Qwen3-1.7B—12.89.45.6—
+ PreSOGPT-5.518.915.69.7≈0.68
+ PreSOStudent (online)21.715.010.3≈1.8
+ PreSOStudent (offline)19.715.610.3≈1.8

Offline student-generated hints match or exceed GPT-5.5 hints on all three benchmarks, providing an alternative to external annotation. Their generation remains a one-time preprocessing cost, but the hints can be reused across training runs. Thus, PreSO requires suitable privileged guidance, while the need for additional hint generation or an external annotator depends on the domain and available data.

AuthorsOpenReview ↗

Response to W3 and W4

W3: Computational efficiency and extension to new domains.

As discussed in W1, our additional medical experiments support PreSO’s effectiveness when trained beyond mathematics. Regarding computational cost, Table 11 reports the time required to complete the full training run. To evaluate performance under matched time constraints, our appendix additionally compares all methods under identical wall-clock training budgets. The table below summarizes average accuracy across the five mathematical benchmarks:

MethodQwen3-1.7B: 4hQwen3-1.7B: 8hQwen3-8B: 4hQwen3-8B: 8h
GRPO22.323.737.537.5
OPSD23.124.035.526.3
RLSD24.024.138.738.5
RLCSD22.523.637.937.8
PreSO26.828.640.645.5

PreSO with a 4-hour budget already outperforms every baseline with an 8-hour budget at both model scales. At 8 hours, its margins over the strongest baseline reach 4.5 and 7.0 points on Qwen3-1.7B and Qwen3-8B, respectively. Thus, its longer full-run training time does not translate into poorer accuracy under a fixed training-time budget.

Regarding extension to other domains, our new medical QA experiments in W1 demonstrate effectiveness beyond mathematics. As discussed in W2, these experiments directly use unmet rubric criteria as privileged information, avoiding a separate hint-generation pipeline.

W4: Training-free activation steering versus distillation.

This is a valuable suggestion for examining the usefulness of the extracted directions independently of distillation, and we add a training-free activation-steering experiment using the initial Qwen3-1.7B model. Both vectors and subspaces are estimated with this frozen model on 512 calibration problems disjoint from the development and test sets. We inject either the cluster-averaged hint-induced shift or its rank-128 projection at each next-token prediction, preserving the hidden-state norm without updating parameters or accessing test-time hints or solutions.

MethodAIME24AIME25HMMT25Avg.
Base model12.89.45.69.27
Full-shift steering14.210.36.110.20
Projected-shift steering15.612.57.511.87
PreSO18.915.69.714.73

Both steering variants improve over the base model, supporting the usefulness of the extracted shifts independently of distillation. Projected steering performs better on all three benchmarks, while PreSO further exceeds it by 2.87 points on average. These results support the additional benefit of learning through parameter updates beyond directly injecting the corrections at inference time. We will incorporate this comparison into the revised manuscript.

Thank you for your valuable suggestions. We have incorporated the corresponding revisions and additional results into the main text and appendices of the revised manuscript.

Comments on the paper

1 comment

We sincerely thank the reviewers for their constructive feedback. We appreciate their recognition of the motivation for selectively learning privileged guidance (R1, R3), the representation-level target design (R2), and the consistent empirical gains and extensive evaluation (R1, R2, R3).

We have responded to each reviewer individually. The main additions and clarifications are summarized below:

  • [R1, R2, R3] Added medical-domain experiments, where PreSO outperforms OPSD and GRPO on three benchmarks at both model scales. We also reported preliminary thinking-mode training results to R2 during the discussion.
  • [R1] Added offline student-generated hints as an alternative to external annotation, and highlighted fixed-time comparisons supporting PreSO’s computational efficiency.
  • [R1] Added training-free activation-steering experiments, supporting the usefulness of the extracted directions and the additional benefit of distillation.
  • [R2] Clarified how the parameters were selected and added sensitivity analyses covering supervision windows, warm-up budgets, cluster counts, and routing representations. Medical experiments further demonstrate reuse of the default configuration.
  • [R2, R3] Added periodically updated subspaces, showing further accuracy gains with 6.6% additional training time and clarifying freezing as a practical default.
  • [R3] Added comparisons with FitNets, RKD, OPRD-Bridge, and OPRSD; PreSO outperforms all four on the evaluated benchmarks.
  • [R3] Clarified the empirical interpretation of the low-rank subspace, target construction and gradient flow, and the aggregate benefit of cluster routing.

Following the discussion, R1 and R3 maintained their positive scores, with R3 explicitly confirming that all concerns were addressed. We have not yet received a follow-up from R2. We appreciate the AC’s consideration of the responses and additional evidence.

For clarity, we refer to reviewers P3QX, Su9G, and 9KgT as R1, R2, and R3, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.