acceptodds
Under review as a conference paper at ICLR 2027

Structured Category-wise Selective On-Policy Distillation

Abstract

On-policy distillation (OPD) aligns teacher and student predictions on student-generated trajectories, but conventional token-level objectives entangle semantic-category choices with preferences among alternative realizations within the same category. We propose Structured Category-wise Selective On-Policy Distillation (SCS-OPD), which separates category-mass alignment from intra-category supervision. Category-level alignment drives the main rollout update; an optimizer-matched finite-step selective gating mechanism then compares category-plus-residual and category-only updates, retaining intra-category residuals only when the gate indicates a lower category-level loss. Across OPD experiments covering multiple methods, the Qwen and DeepSeek model families, and multiple random seeds, SCS-OPD achieves the best performance on most mathematical benchmarks. In the settings where it leads, SCS-OPD improves over methods including Reverse-KL OPD, EOPD, and TSD by 0.3–3.4 percentage points, while maintaining stable performance on out-of-domain benchmarks. Mechanistic analysis shows that, after semantic aggregation, intra-category supervision is sparse, highly concentrated, and heterogeneous. Selective gating over fine-grained realizations therefore provides substantial optimization benefits for OPD.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
On-Policy Distillation, Knowledge Distillation, Semantic Chunking, Selective Supervision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_v6uWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper separates category-level alignment from within-category imitation in on-policy language-model distillation. SCS-OPD uses category alignment for rollout training and selects auxiliary residuals by comparing category-only and category-plus-residual AdamW trials from identical parameters and optimizer states. It leads AIME24 and MATH500 across three Qwen3 seeds, with exceptions on AIME25 and AMC23, and leads three of four mathematical benchmarks in a second configuration evaluated at seed 17. A single-seed ablation supports residual selection, while local probes characterize residual concentration and interference.

Critical strengths

Principled selective supervision. Sections 4.3–4.4 introduce matched AdamW trials that assess the incremental benefit of residual supervision over category-only updates while accounting for optimizer state.

Encouraging downstream evidence. SCS-OPD leads three of four benchmarks in Table 1, and Section 5.2 reports consistent AIME24 and MATH500 advantages across three Qwen3 seeds. The controlled ablation in Table 2 supports selective residual use at the evaluated setting.

Critical weaknesses

The selector’s benefit is insufficiently isolated. Table 2 evaluates one seed and residual weight, without uncertainty estimates, a tuned uniform-residual baseline, or simpler-selector comparisons.

Local selection is not validated at the aggregate-update level. Section 4.3 selects residuals through individual-pack trials, whereas Section 4.4 commits a joint update. The experiments do not establish whether accepted residuals remain beneficial when combined.

The semantic interpretation lacks direct evidence. Section 4.1 uses lexical rules to approximate semantic equivalence. The perturbations in Section 5.3 preserve category masses by construction, but do not establish contextual interchangeability or show that semantic grouping outperforms simpler grouping rules.

Questions for authors

Clarification: Comparison fairness. Which baselines receive equivalent auxiliary supervision, and how are total training budgets matched? What runtime and peak-memory overhead does SCS-OPD introduce?

Additional evidence: Gate reliability. Could you report acceptance rates over training, the frequency of all-rejected iterations, and decision stability across repeated trials or a small positive gate margin? These would clarify whether the zero-margin selector operates reliably.

Clarification: Result consistency. What run, checkpoint, or evaluation differences explain the β=1.0\beta=1.0 discrepancies in Appendix B.3? Was β\beta selected using validation data separate from the reported benchmarks?

Clarification: Transfer across configurations. What is the exact 7B teacher checkpoint, and were the grouping rules and selector hyperparameters transferred unchanged? This determines whether the second configuration supports transferability or required configuration-specific tuning.

Reviewer_bVtAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is about on-policy distillation of language models, where a student learns to match a teacher's next-token predictions on text the student generates itself. The starting observation is that several candidate tokens often express the same choice, such as case variants, the same number, or synonyms. Standard token-level matching then mixes two signals: which kind of token to produce, and which of the equivalent forms the teacher happens to prefer.

SCS-OPD does three things:

  • It groups the teacher's candidate tokens into such categories.
  • It trains the student mainly to match the teacher at the category level.
  • It adds the finer within-category signal only when a trial update shows that including it gives a better category-level match.

Experiments distill Qwen3 and DeepSeek-family students on math benchmarks and report gains of 0.3–3.4 points over existing on-policy distillation methods.

Critical strengths
  • The paper tackles an important problem.
  • The formulation is clean, and the paper is honest about what the stop-gradient does and about its limitations.
  • MATH500 is consistently higher: SCS-OPD scores 92.4–93.0, and no baseline goes above 91.7 at any seed.
Critical weaknesses
  • In general, the gain seems somewhat marginal. For instance, according to Table 1, the benefit of the proposed method seems marginal compared to the baselines, even losing to Forward-KL by a large margin for AMC23. Similarly, the (Table 2) the proposed method seems very close to F+All-R.
  • Moreover, based on my understanding, the proposed method of SCS-OPD gets more training than the methods it is compared against. As such, it is not immediately clear if the gains are from the extra training? Can we design additional experiments to budget the compute cost as well?
Questions for authors

Would be great if you could address the compute issue mentioned above.

Reviewer_BoxLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies on-policy distillation (OPD) of language models and asks how teacher supervision should be structured among semantically related candidate tokens at a student-generated prefix. The authors decompose normalized candidate-level reverse KL on the teacher's top-K support into a category-level mass-alignment term F and an intra-category conditional discrepancy term, where local semantic categories are constructed by grouping decoded tokens via surface-form identity, complete numerals, case folding, and WordNet synsets (Appendix A.1). The proposed method, SCS-OPD, trains the main rollout update with F only, and injects the intra-category residual R (with stop-gradient on category weights) in a separate auxiliary update only for packs where an optimizer-matched finite-step trial — comparing an F+βR update against an F-only update from the same parameter snapshot and persistent AdamW state, with full rollback — yields a strictly lower post-update category-level loss. Experiments cover Qwen3-4B → Qwen3-1.7B on AIME24/25, AMC23, and MATH500 (seeds 17/21/42 for four of the six main methods; seed 21 for the main table), a second teacher–student configuration (DeepSeek-R1-Distill-Qwen-1.5B with a 7B teacher, one seed), a controlled seed-17 ablation (F-only vs. F+All-R vs. SCS-OPD), a five-value β sensitivity sweep, and mechanistic probes. SCS-OPD reports the best score on most mathematical benchmarks, with consistent positive margins over Reverse-KL and EOPD in the settings where it leads and markedly larger margins over TSD and GRPO, while MMLU/HumanEval diagnostics show no degradation. The mechanistic analysis shows that intra-category supervision is sparse and highly concentrated in a small subset of eligible positions, that category-preserving reallocation changes token-level reverse KL and its gradients substantially while F remains essentially unchanged, and that residual-reducing updates heterogeneously help or hurt F, motivating selective retention.

Critical strengths
  1. A well-motivated decomposition paired with a selection mechanism tied to the actual optimizer. The chain-rule decomposition makes explicit how token-level reverse KL couples category allocation with conditional fitting difficulty, and the optimizer-matched finite-step trial measures a residual's incremental effect after the actual AdamW step rather than via gradient-level proxies. The controlled seed-17 ablation directly tests the two uniform alternatives (drop all residuals vs. retain all residuals), and the probes (Fig. 2) show that category-preserving reallocation leaves F essentially unchanged while substantially altering token-level reverse KL and its gradients.
  2. Consistent primary results with transparent reporting. SCS-OPD is the highest reported method on AIME24 and MATH500 at all three Qwen3 seeds, wins 10 of 12 matched comparisons against each of Reverse-KL and EOPD, and leads three of four benchmarks in the second configuration. The paper documents its own exceptions and declines to claim statistical significance — transparency that increases confidence in the results that are claimed.
Critical weaknesses
  1. Small effect sizes without statistical assessment, and single-seed ablations. The closest margins — +0.4/+0.3 points on AIME24 at seed 21 vs. Reverse-KL/EOPD — are within the ≈0.42-point shift of a single flipped response on the 30-problem AIME benchmark at Avg@8, and the paper calls its 10/12 win counts "descriptive… not a statistical-significance test". Evidence that selection (rather than simply adding residuals) helps rests on the seed-17 ablation and β sweep, each at a single seed; the second configuration is single-seed as well. Cross-seed variability is non-trivial (AIME25 at seed 42: SCS-OPD 39.2 vs. EOPD 45.0), so the ordering of the three strongest methods is not firmly established.
  2. Two core aspects are acknowledged but never quantified: grouping quality and selector cost. (a) Grouping quality. The categories defining F and R are built by lexical rules (surface-form identity, numerals, case folding, WordNet synsets; Appendix A.1), which the authors themselves call "a reproducible lexical approximation, not a proof of contextual semantic equivalence". Apart from an audit of 500 sampled merges (Appendix D.1), no experiment measures how grouping errors affect the method, even though R is computed entirely with respect to this partition. (b) Selector cost. The gate runs two extra trial updates plus two post-update evaluations per pack (Section 4.3), yet the paper reports no wall-clock or throughput comparison — only "trial updates add training cost" (Conclusion) — so the accuracy-per-compute trade-off cannot be assessed.
Questions for authors
  1. The β = 1.0 row of the sweep (Table 7, Appendix B.3) reports AMC23/MATH500 = 92.0/92.9 for the seed-17, β = 1.0 configuration, whereas Tables 2 and 5 report 91.0/92.7 for what appears to be the same configuration. Appendix B.3 attributes this to different "evaluation blocks" without describing what differs. Which evaluation settings differ between these blocks, and which value should be considered canonical for the released configuration?
  2. What is the measured training-time overhead of the trial-based selector relative to F-only and Reverse-KL OPD at the same scale? Per pack, the gate requires two trial forward/backward/optimizer steps plus two post-update F evaluations (Section 4.3) in addition to the main update. Reporting the actual elapsed training time (wall-clock hours) or throughput (tokens/second) under matched hardware would allow an accuracy-per-compute comparison.
  3. The closest reported margins — in particular the +0.4 (vs. Reverse-KL) and +0.3 (vs. EOPD) AIME24 margins at seed 21 — are smaller than the ≈0.42-point shift caused by a single AIME response flip at Avg@8, and the paper itself notes the win counts are "descriptive… not a statistical-significance test". Can you provide a significance or robustness assessment indicating whether these margins exceed the evaluation noise floor?
  4. (Clarification) Please define and cite the "SEM" baseline that appears in Tables 5 and 8 — Appendix F describes it only as "an additional seed-17 reference"; is it a method, and under what setting was it run?
  5. How much of the reported gain is earned by the risky grouping rules rather than the safe ones? Two of the four rules (Appendix A.1) are lossless — they merge tokens that print identically or as equal numerals; the other two — case folding ("China" vs. "china") and context-free WordNet-synset links — can merge tokens with different meanings in context. All reported results use the full grouping. Can you provide evidence separating the contribution of synonym merging from that of the selection mechanism?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.