acceptodds
Withdrawn from ICLR 2027

The Gravity of Privilege: Tilting Learning Measures in On-Policy Self-Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-Policy Self-Distillation (OPSD) is an emerging paradigm for LLM post-training, in which a model learns from a privileged view of itself through dense supervision on student-generated trajectories. However, existing OPSD methods incorporate privileged information through the conditional teacher distribution, while leaving the empirical measure over student-generated trajectories unchanged. This induces a structural asymmetry: privilege determines what the student learns at each visited trajectory, but not the measure under which different trajectories contribute to learning, even though privileged conditioning can vary substantially across sampled trajectories. To address this limitation, we propose Privileged Measure Self-Distillation (PMSD), which leverages privilege not only to construct distillation targets, but also to adaptively reweight the learning measure across training problems. Specifically, PMSD contrasts the same frozen teacher's support for each trajectory with and without privileged information, aggregating log-likelihood ratios into a policy-dependent signal for problem-level allocation. We formulate this adaptation as a relative-entropy trust-region optimization that assigns greater learning mass to problems receiving stronger support from privilege, while bounding the KL divergence from the original empirical measure. This mechanism intrinsically favors problems with larger estimated privileged likelihood contrasts while preventing extreme polarization in the learning measure. The resulting Gibbs-form solution is governed by a scalar dual variable, with the KL constraint directly bounding the induced shift in population gradient aggregation for the exact solution under bounded per-problem gradients. To obtain a more reliable estimate of the privilege-induced effect under the current policy, PMSD additionally uses multiple on-policy rollouts per problem. Extensive experiments on mathematical reasoning across Qwen3-1.7B/4B/8B show that PMSD consistently surpasses selected baselines by 0.84–2.50 percentage points in Avg@12 while maintaining superior token efficiency. Our code is available on https://anonymous.4open.science/r/pmsd-paper-code-B260.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, On-Policy Self-Distillation, Empirical Bayes

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qvqXOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a re-weighting of training sample in standard OPSD. For each example xx, the log probability ratio defines the score Ck(x)C_k(x), which is used in an optimization problem to give the desired re-weighted distribution. Experiments examined the re-weighting mechanism and verified improvement over standard OPSD.

Critical strengths

The topic of re-weighting the examples is interesting and clear. The re-weighting is designed as a solution to an optimization problem that maximizes the expected score. Experiments and ablation studies are conducted to verify the effectiveness of the proposed re-weighted optimization problem.

Critical weaknesses

The paper lacks theoretical justifications of the proposed method. I don't understand why the score Ck(x)C_k(x) is defined in that way. This score represents the change in the confidence of the student's generated token after the teacher reads the solution. This does not capture the correctness or the value of the student's generated token, and the meaning of your score makes me really confused. The issue is: a very bad token that increases the probability from 1e-10 to 1e-9 still has a large score; a token that decreases the probability from 1e-1 to 1e-10 has very low score, but it doesn't mean the example is not important. The solution to the relative-entropy trust-region optimization is not guaranteed to improve the training. I don't understand why the new distribution is better than the original distribution, and why it gives a better direction of the OPSD problem. The paper does not solve the relative-entropy trust-region optimization directly, and instead uses an online algorithm that keeps updating the distribution during training. However, this online update algorithm is not guaranteed in the paper to approach the desired distribution. At least some guarantee should be given for your algorithm.

Questions for authors

I already wrote everything in the Critical Weaknesses section. Please refer to that.

Reviewer_vJnAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Privileged Measure Self-Distillation, which extends on-policy self-distillation through adaptive problem-level weighting. It compares a frozen teacher’s likelihoods of student-generated trajectories with and without reference solutions, aggregates these contrasts across multiple rollouts, and applies KL-constrained reweighting while preserving the distillation target. Experiments on mathematical reasoning benchmarks show consistent improvements over OPSD across three Qwen3 model scales.

Critical strengths
  1. The method is clearly formulated with proofs, with a KL constraint that controls changes in problem-level learning weights.
  2. PMSD consistently improves over OPSD across three Qwen3 scales, while the matched multi-rollout baseline helps isolate the benefits of reweighting.
Critical weaknesses

The experiment set-ups have several problems:

  1. The claimed token-efficiency advantage lacks a clear definition and direct comparisons under matched token budgets. The only exisiting experiments only show that the output answers are shorter. Also, the overhead of computing resources and time haven't been taken into consideration.
  2. Several closely related adaptive supervision baselines mentioned in related works are missing.
  3. Evaluation is also restricted to the Qwen3 family and mathematical reasoning, leaving generalization across model families and tasks unclear.

Novelty Issues: FiRe-OPD [1] already explores trajectory filtering and token reweighting, RLCSD [2] uses contrastive teacher signals, and REPS [3] establishes KL-constrained exponential updates. PMSD’s main distinction is using privileged contrasts for problem-level weighting while retaining the distillation target. This appears to be an incremental extension whose distinct benefits require stronger justification.

[1] Li, Yuying, et al. "Filter, then reweight: Rethinking optimization granularity in on-policy distillation." arXiv preprint arXiv:2606.02684 (2026).

[2] Pan, Leyi, et al. "RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation." arXiv preprint arXiv:2606.11709 (2026).

[3] Peters, Jan, Katharina Mulling, and Yasemin Altun. "Relative entropy policy search." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 24. No. 1. 2010.

Questions for authors

See weakness

  1. How is token efficiency measured, and does the advantage hold under matched budgets, including computational overhead?
  2. How does PMSD compare with the closely related adaptive supervision baselines?
  3. Do the gains generalize to other model families and tasks?
Reviewer_vQWSOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PMSD, a variant of OPSD. PMSD leverages privilege not only to construct distillation targets, but also to adaptively reweight the learning measure across training problems. PMSD compares a frozen teacher’s support for student trajectories with and without privileged information, then applies KL-constrained problem-level reweighting. Experiments on mathematical reasoning across Qwen3-1.7B/4B/8B report improvements of 0.84–2.50 percentage points over OPSD.

Critical strengths
  1. The motivation is clear. By allocating training weights according to "support degree", the PMSD focuses on learning more valuable trajectories.

  2. The matched multi-rollout baseline and component ablations help isolate the contributions of reweighting, signed privilege evidence, and the KL constraint.

  3. Comprehensive experiments support the claims of PMSD.

Critical weaknesses
  1. Increased privileged-teacher support does not necessarily indicate correctness or greater learning value; the score may also capture stylistic agreement rather than useful reasoning information.
Questions for authors

No questions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.