acceptodds
Under review as a conference paper at ICLR 2027

When Selective On-Policy Dstillation Rules Conflict: From Supervision Collapse to Minimal Repair

Abstract

On-policy distillation (OPD) uses teacher feedback to train a student on responses generated by its current policy. Existing methods select which tokens contribute to the training loss, using criteria such as uncertainty or teacher–student disagreement. However, these methods can select different positions, leaving too little shared feedback when every retained token must satisfy two rules. We discover that strict combination can eliminate all teacher feedback during token selection, yielding a zero joint gradient even when each rule alone produces a non-zero gradient. In this paper, we introduce SupportGuard to restore feedback through a weighted loss on tokens selected by only one rule. Separate training runs vary the mean loss weight over all response tokens. The lowest tested level meeting the parameter update criterion defines the repair target. SupportGuard compares the current mean loss weight with this target. Combinations that meet the target remain unchanged. Otherwise, it retains the full loss contributions of jointly selected tokens and applies a common weight to losses on tokens selected by only one rule. This weight equals the gap to the target divided by the increase in mean weight when these tokens receive full weight. We evaluate the mechanism and repair across Qwen-MATH, Granite-MATH, and Qwen-Code. Across the tested unsafe settings, SupportGuard restores parameter updates using, on average, only 1.02% of the additional training weight available from strict combination to the union of tokens selected by either rule. It leaves the safe control unchanged and improves accuracy on the fixed MATH evaluation by 2.11 percentage points over strict combination.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Dstillation, Selective On-Policy Dstillation, Token Selection, Selector Composition

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BHbROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates supervision collapse in multi-rule selective on-policy distillation (OPD). The authors observe that taking the intersection of token selections from multiple criteria can result in very sparse or empty supervision, even when individual selectors retain useful training signals. To address this, they propose SupportGuard, which assigns full weight to jointly selected tokens and supplements the remaining union-selected tokens when the average supervision weight falls below a predefined threshold.

The paper evaluates SupportGuard on mathematical reasoning and code generation benchmarks. The reported results show improvements over vanilla OPD and several selective distillation baselines. Additional ablations and gradient analyses investigate the effects of supervision recovery, shared-token prioritization, and the amount of supplementary supervision.

Critical strengths
  1. Identification and characterization of a meaningful optimization failure mode.

The paper highlights a practical limitation of combining token selection rules through strict intersection. The analysis shows that 133 out of 280 responses with two nonempty selectors have empty intersections, substantially exceeding the approximately 3.1 cases expected under a count-matched random-selection reference. The accompanying same-state gradient measurements and optimizer-update diagnostics provide evidence that missing overlap can eliminate direct distillation feedback. This is a useful observation for designing multi-criterion selective OPD methods.

  1. Simple and computationally efficient repair mechanism with supporting ablations.

SupportGuard introduces a straightforward weighting strategy that preserves jointly selected supervision and recovers additional signals only when needed. The ablations indicate that both supervision recovery and prioritizing jointly selected positions contribute to the reported performance. The method also introduces negligible overhead for constructing token weights and requires no additional teacher forward passes. Its simplicity and compatibility with existing OPD training pipelines are practical strengths.

Critical weaknesses
  1. The experiments do not sufficiently disentangle multi-rule conflict from excessive sparsity of an individual selector.

A central motivation is that distinct selection rules can conflict and eliminate otherwise useful supervision. However, Appendix D.1 reports that among 603 zero-feedback responses, 470 arise because selector A is already empty, whereas only 133 correspond to nonempty but disjoint selectors. Furthermore, Appendix E.2 shows a highly imbalanced selection pattern: jointly selected positions account for only 0.074% of tokens, while teacher-confidence-only positions account for approximately 98.2%.

These statistics raise the question of whether the observed failure mode and improvements are primarily driven by an unusually sparse individual selection rule rather than a general difficulty in combining multiple meaningful selection criteria. Although the count-matched randomization supports systematic disagreement when both selectors are active, it does not establish that the proposed repair is broadly advantageous under more balanced selector configurations. Comparisons across selection thresholds and against simpler supervision-budget safeguards would strengthen the central claim.

  1. Inconsistent experimental descriptions and insufficient statistical evidence weaken confidence in the empirical conclusions.

There are important inconsistencies in the reported experimental configurations. Section 4 describes Gemma 3 4B/14B experiments, Appendix C.1 instead specifies Gemma 3 4B/12B, while Table 2 reports results on Granite and the appendix describes Granite as historical experiments. It is therefore unclear which model family and experimental protocol support the claimed cross-family generalization.

Additionally, Table 3 reports a SupportGuard accuracy of 54.00%, whereas the best MATH-500 result in Table 1 is 47.55%, without clearly explaining the difference in evaluation settings. Although the main results are reportedly averaged over three training seeds, variance estimates are not provided, despite improvements over the strongest baselines being relatively small (0.43–1.31 percentage points on Qwen). Finally, the primary benchmark tables do not provide comprehensive strict-intersection and full-union comparisons under the same evaluation protocol. These issues make it difficult to determine the statistical reliability and precise source of the reported improvements.

Questions for authors
  1. Experimental consistency and statistical reliability. Could the authors clarify the actual model families and checkpoints used in the cross-family experiments, reconcile the differences between Section 4, Table 2, and Appendix C.1, and explain how the 54.00% result in Table 3 relates to Table 1? Could they also provide per-seed results or standard deviations for the main benchmark comparisons? Resolving these issues would substantially affect confidence in the empirical findings.

  2. Generality and causal attribution of the improvements. How does SupportGuard perform when the two selectors have more balanced selection rates, under different selection thresholds, or with a simple minimum-supervision fallback? In particular, can the authors compare against baselines that recover a similar amount of supervision or execute a comparable number of optimizer updates? Such results would help establish whether the gains arise specifically from the proposed weighting strategy rather than mainly from restoring training updates suppressed by strict selection.

Reviewer_HR6SOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies supervision loss when multiple token-selection rules are combined by hard intersection in on-policy distillation. It proposes SupportGuard, which keeps jointly selected tokens at full weight and, when the mean loss weight is below a target, adds a common weight to the remaining union positions. Experiments cover mathematics and code generation with Qwen2.5 models and report a second-family mathematics evaluation. The paper reports gains over individual selection components and other controls, together with ablations and gradient diagnostics investigating the allocation of supervision.

Critical strengths

S1: The paper empirically diagnoses a concrete interaction between selectors.

Section 2 reports 133 empty intersections among 280 responses on which both rules select at least one token. Preserving selection counts but randomizing positions gives only about 3.1 expected empty intersections. This is more informative than simply observing that intersections can be empty: it suggests systematic disagreement between the tested selectors.

S2: The controls go beyond adding more supervision.

Table 3 compares against redistributing the same total weight uniformly over the union, and Section 5.3 exchanges weights between shared and singly selected positions to inspect gradient changes. These controls support the relevance of weight allocation. The paper also acknowledges that sparse random supplementation performs close to SupportGuard, rather than claiming every part of its allocation is indispensable.

Critical weaknesses

W1: The practical importance and additional methodological contribution are not sufficiently established.

Supervision collapse follows from combining hard masks by intersection, but the paper does not establish why this should be the preferred way to combine complementary signals. TIP already combines entropy and disagreement through a Soft-OR score, while FiRe-OPD combines teacher confidence and student uncertainty through soft weighting. SupportGuard has a specific difference: it preserves full weight on the intersection and restores only enough union weight to meet a target. This is a straightforward and potentially useful rule, but its practical advantage over these alternatives is not sufficiently isolated. Tables 1–2 do include TIP and FiRe-OPD comparisons, so this is not a claim that they are absent; however, TIP is explicitly a “TIP-style allocation control,” and several other baselines are components of the original methods. The main text does not establish a faithful Soft-OR comparison or clarify whether the advantage comes from intersection priority, different input signals, or different supervision scales. Table 3 supports repair over strict intersection and the tested union controls, but does not establish that an intersection-first repair is necessary or preferable to combining the signals differently from the outset.

W2: The evaluation has limited scope and important reporting gaps.

Section 4 uses pretrained Qwen2.5 models and a 2,048-token training response cap, without an identified dedicated thinking-model evaluation or response-length and truncation statistics. Since the target uses the full response length as its denominator, the cap is relevant to the repair behavior; without these statistics, it is unclear whether the result carries over to longer reasoning trajectories. This limits the demonstrated scope rather than proving the cap invalid. Mathematics evaluation reports Avg@8 but not pass@8, so the results do not establish whether higher average correctness also preserves the probability of solving a problem across multiple attempts. In addition, the Gemma/Granite inconsistency and the unidentified 54.00% ablation setting prevent a reliable assessment of the cross-family and component evidence. These issues need clarification before the experimental conclusions can be fully assessed.

Questions for authors

Q1: Why is preserving the hard intersection a preferable starting point to directly fusing the selection signals? What exactly is implemented as “TIP-style allocation control,” and how are the signals, retention/weight budgets, and loss normalization made comparable? Please clarify whether existing matched results against faithful Soft-OR and FiRe-OPD support the claimed advantage. This would affect my assessment of the additional methodological contribution.

Q2: Which exact student and teacher checkpoints produced Table 2: Gemma or Granite? What dataset, model pair, selector pair, threshold, metric, and seed scope produced the 54.00% result in Table 3 and Section 5.3, and how does this setting relate to Tables 1–2? These details are necessary to assess the main supporting evidence.

Q3: Can the existing eight mathematics generations per problem be used to report pass@8 alongside Avg@8? What fraction of training responses hit the 2,048-token cap, and what do the available length statistics show? Please also clarify whether any existing evaluation uses a dedicated thinking model; if not, the conclusions should remain limited to the tested models. These answers would help distinguish gains in average correctness from multi-attempt coverage and assess dependence on the response-length setting, without requiring a broad new benchmark suite.

Reviewer_AtJNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to combine multiple filtering rules in selective on-policy distillation (OPD). It shows that directly taking the intersection of tokens selected by different rules can leave too little supervision or even eliminate it entirely. To address this issue, the paper proposes SupportGuard, which retains full weights for tokens in the intersection and, when the amount of supervision falls below a predefined threshold, assigns smaller weights to additional tokens selected by individual rules to supplement the supervision signal. The authors evaluate SupportGuard on mathematical reasoning and code generation, reporting accuracy improvements over standard OPD and other single-rule baselines. However, the paper contains inconsistent descriptions of key experimental details, including the model configurations used in Table 2, making it difficult to determine the actual experimental setup.

Critical strengths

The core idea of the proposed method is intuitive: when combining two token selection rules, it prioritizes retaining full supervision for tokens in their intersection—those for which the student is uncertain but the teacher is relatively confident in this paper. If too few such tokens are available, it supplements the supervision signal with other tokens selected by only one of the two rules. This design preserves the emphasis on jointly selected positions while alleviating insufficient supervision caused by overly strict filtering. Moreover, the supplementary supervision can be obtained directly from existing filtering results, without generating additional responses or making extra calls to the teacher model, making the method straightforward to implement.

Critical weaknesses
  1. There are clear inconsistencies between the experimental setup and the reported results. The experimental setup states that, in addition to Qwen, Gemma 3 4B and 14B models are used for mathematical reasoning experiments (lines 266–269). However, the corresponding cross-family results are labeled and described as Granite experiments (Table 2 and lines 334–353). Meanwhile, the appendix explicitly states that “Historical Granite experiments are retained as mechanism checks and are not pooled with the current public-benchmark results” (Appendix C.1, lines 802–808). This paragraph also specifies a 12B Gemma teacher, contradicting the 14B model stated in line 269 of the main text. These inconsistencies make it difficult to determine the student model, teacher model, and other experimental settings actually used in Table 2, as well as to distinguish the current experimental results from the mechanism checks.

  2. The key hyperparameter τ\tau lacks a detailed selection procedure and sensitivity analysis. SupportGuard uses τ\tau to determine whether the current supervision is sufficient and, accordingly, how much supplementary supervision is needed, making τ\tau a central hyperparameter of the method. However, despite brief descriptions (lines 310 and 449), the paper does not clearly specify the values of τ\tau used in each main experiment, how these values were selected, or how different values affect the results. The existing descriptions are insufficient to assess how strongly SupportGuard’s performance gains depend on the choice of τ\tau.

Questions for authors
  1. Could the authors include experiments with more recent model versions? The Qwen 2.5 series is relatively dated. Could the authors validate SupportGuard on newer Qwen models, such as the Qwen 3 series?
  2. Could the authors include experiments with different model types? The paper states that both the teacher and student are base models (lines 802–803). This setup is relatively uncommon in practical OPD applications. Experiments with instruction-tuned models would help establish whether SupportGuard’s benefits extend to more commonly used settings.
  3. Could the authors provide qualitative case studies of the tokens selected and weighted by SupportGuard? It would be helpful to examine the characteristics of tokens in the intersection, as well as those added to supplement supervision when the intersection is too sparse, and assess whether their assigned weights are appropriate. For example, are these tokens primarily associated with key reasoning steps or the use of relevant knowledge?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.