acceptodds
Under review as a conference paper at ICLR 2027

When and Where Should Teachers Stop Teaching? Local Discriminability Collapse in Strong-to-Weak On-Policy Distillation

Abstract

In strong-to-weak on-policy distillation (OPD), where a high-capacity teacher supervises a substantially smaller student on student-generated rollouts with dense token-level feedback, conventional practice assumes that supervising every generated token is uniformly beneficial whenever teacher guidance is accessible. In this paper, we uncover a prevalent and trajectory-specific failure mode that overturns this premise: local discriminability collapse. Specifically, while the global teacher-student advantage (the mean log-probability gap on sampled tokens) persists late into a response, the teacher's local discriminative contrast across the student's reachable alternative actions sharply decays, forcing standard full-trajectory OPD to optimize over low-signal, high-variance suffix noise. To resolve this mismatch, we introduce Local-Discriminability-Aware On-Policy Distillation (LDA-OPD), a framework that selectively restricts dense teacher supervision to trajectory regions where guidance remains locally discriminative. Concretely, LDA-OPD applies three steps to each trajectory: it evaluates the teacher's log-probability margin between the top-two competitors within the student's high-probability candidate set, aggregates these margins over sentence-level semantic segments, and identifies the downward transition using a lightweight change-point test based on the profiled Bayesian Information Criterion (BIC) that is sweep-free over cutoff lengths. Past this boundary, we release dense supervision by zeroing the token-level loss on the suffix, and preserve per-sample loss mass by rescaling retained-prefix advantages so that the total loss magnitude remains unchanged. This adaptively truncates uninformative suffixes in a single training pass without requiring hyperparameter sweeps over cutoff lengths, while leaving student inference-time generation unconstrained. Extensive evaluations across multiple student scales (1.7B, 4B, 8B), model families (Qwen3, Gemma3), and diverse task types (multi-step mathematical reasoning, programmatic code generation, and out-of-domain scientific reasoning and instruction following) show that LDA-OPD consistently outperforms standard full-trajectory OPD and matches or surpasses multi-run fixed-prefix oracles in a single training run. Furthermore, matched-framework signal ablations and multi-seed evaluations provide strong empirical and counterfactual evidence that these gains are driven by local action contrast rather than generic entropy or position heuristics, establishing that effective strong-to-weak distillation requires evaluating not merely the presence of teacher guidance, but its local discriminative sharpness.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Strong-to-Weak On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hSy8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies that in strong-to-weak on-policy distillation (OPD), the teacher's contrast among the student's likely next tokens decays along a rollout, even though the mean advantage stays non-zero. This is an interesting observation. The proposed LDA-OPD measures the teacher's top-1/2 margin M_t within the student's top-K tokens and averages it per sentence. A profiled RSS-BIC test finds one downward change point. The loss after it is zeroed, and the prefix is rescaled to keep the loss mass. With a Qwen3-30B-A3B teacher, Qwen3-1.7B rises from 36.8% (OPD) to 40.1% math AVG, against 39.4% for the best fixed prefix. Results for 4B/8B students, Gemma3, code, GPQA and IFEval show similar trends.

Critical strengths
  • The proposed approach is simple, practical yet effective. A drop-in loss mask that removes the need for a cutoff sweep at no extra cost.
  • The proposed method has consistent empirical gains, Improving over full OPD and fixed-prefix baselines across three scales, two model families and four task types.
  • The paper is easy to follow. The motivation and method are clearly presented. It's appreciated that the code is released.
Critical weaknesses
  • The empirical evidence seems not to fully support the theory. App. A.2 links useful updates to the teacher–student contrast, but M_t depends only on the teacher. Var_{C_t}(A_t) is never tested, and the KL signal closest to the theory does worse (37.8%, Table 3). Fig. 1a (late mean advantage ≈ −0.3) and Case 2 (A.4) suggest the released suffix still carries corrective signal.

  • Gains sit within hindsight choices and noise. The lead is 0.7 pts over the best fixed prefix (1.7B) and 0.5 over FastOPD at 8B (Table 4). K=4 is the test-set "peak", the same hindsight choice the paper faults in fixed-prefix oracles. Seed runs compare only with FastOPD-2048. MATH500 uses 1 sample per problem, and the GPQA gain (42.4 vs 41.2) is called "significant" without a rigorous statistical test.

  • The ablations seem confounded. Rescaling alone adds 0.8 pts, more than the 0.6 lead over FastOPD without it, and it is unclear whether the baselines rescale. All Table 3 alternatives (single runs) fall below plain fixed prefixes (38.6–39.4%). Entropy rises as contrast falls, so a downward-change detector may misplace its release.

Questions for authors
  • Can the authors compare Var_{C_t}(A_t) and student-weighted reverse KL as release signals

  • Can the authors measure gradient SNR on released vs retained tokens to test A.2 empirically

  • Can the authors apply the same rescaling to the fixed-prefix baselines

4 comments

W1

Regarding "Var_{C_t}(A_t)" and "student-weighted reverse KL": We added experiments on the 1.7B student using Var_{C_t}(A_t) and student-weighted reverse KL as release signals, obtaining Math-AVG scores of 39.1% and 38.3%, below the margin signal's 40.1%.

While the advantage variance theoretically provides an upper bound on the magnitude of the centered update, achieving this bound is not guaranteed in practice due to factors such as training stability and the size of the candidate set. Furthermore, we acknowledge that, given the student's candidate set, the margin relies solely on teacher probabilities and is not strictly equivalent to the teacher–student advantage variance. Instead, we treat it as an empirically effective proxy.

We will clarify this distinction in the revised manuscript (And we will incorporate this and all other updates into the revised PDF once all reviewer feedback has been collected). Thank you very much for this insightful suggestion. Regarding "released suffix still carries corrective signal": The additional suffix-only result is 34.7% (please see our response to Reviewer AwDc's first weakness), above the base student's 24.9%. This also indicates that suffix supervision can retain learning value, although training with suffix-only supervision is less effective.

W2

We added three Math-AVG experiments of FastOPD-4096 on the 1.7B student, obtaining 39.4%, 38.6%, and 38.3%, below the 40.07% reported for LDA-OPD in Table 8. On the OOD benchmark GPQA, the three FastOPD-2048 experiments yield 41.2%, 41.4%, and 40.4%, while LDA-OPD yields 42.4%, 41.9%, and 42.9%, with means of 41.0% and 42.4%, respectively. We will replace “significant” with “a higher observed mean accuracy.” Thank you also for the feedback on our “sweep-free” claim. We will qualify the corresponding wording.

W3

Regarding "ablations seem confounded": We additionally evaluated FastOPD-2048 with reweighting on the 4B student, using the same per-sample reweighting formula and a loss denominator based on all valid response tokens, as in LDA-OPD. It achieves 52.7%, while LDA-OPD achieves 53.9%, a gain of 1.2 percentage points. Without reweighting, FastOPD-2048 and LDA-OPD achieve 52.5% and 53.1%, respectively. Thus, LDA-OPD consistently retains its performance advantage under both settings.

Regarding "below plain fixed prefixes": We posit that an adaptive selection mechanism does not inherently guarantee superiority over fixed prefixes; rather, its effectiveness depends critically on the choice of the underlying guiding signal.

Regarding "a downward-change detector": Our implementation already accounts for the direction of the entropy signal. We first transform the sentence-level teacher entropy within the candidate set as Hi′=max⁡jHj−HiH'_i=\max_j H_j-H_i, then apply log1p and detect a downward change. An increase in the original entropy thus corresponds to a decrease in the transformed signal.

Q1

Thank you for the suggestion. Please see our response to the first weakness.

Q2

Thank you for this insightful suggestion. In our exploratory measurements of directional gradient SNR, we found that the reference direction was sensitive to the sampled trajectories and their token counts. We also lacked an independently justified basis for selecting a direction that represents useful optimization progress. We therefore evaluated conditional gradient SNR using the action-centered gradient component defined in Appendix A.2. On the initial Qwen3-1.7B checkpoint, we sampled 1,024 states from each region across 256 student-generated responses and enumerated the student’s top-4 candidate actions at each state. At each state, we centered the candidate-action advantages under the locally renormalized student distribution. We defined the signal as the squared norm of the expected centered gradient and the noise as the expected squared deviation of the advantage-weighted action gradients from that mean. The pooled conditional SNR is the sum of these signals divided by the sum of the corresponding noise terms. This full-vector measurement requires no reference projection direction and uses equal numbers of sampled states from both regions.

The mean teacher margin was 12.295 for the retained prefix and 8.072 for the released suffix, while the mean action-wise advantage variance was 1.432 and 0.865, respectively. The released suffix also had a 79.1% lower mean squared norm of the centered expected gradient than the retained prefix. The pooled conditional gradient SNR point estimate was also higher for retained states: 0.548 versus 0.283. These measurements provide empirical evidence of a weaker average local action-dependent signal in the released region, consistent with the motivation in Appendix A.2.

As an additional experiment, we applied direct supervision only to valid response tokens after the cutoff. This suffix-only variant achieves an in-domain Math-AVG of 34.7%, compared with 24.9% for the base student, 36.8% for full OPD, and 40.1% for LDA-OPD. Thus, suffix supervision retains learning value, although relying exclusively on it yields smaller gains in this setting. We will refine our claims to avoid broadly characterizing the released suffix as uninformative or harmful. Concurrent work has also reported related observations (e.g., arXiv:2605.27028).

Q3

Thank you for the suggestion. Please see our response to the third weakness.

Dear Reviewer hSy8,

Since the discussion phase is ending soon, we would like to kindly ask you to take a look at our rebuttal. We have carefully addressed all the concerns raised in your review and provided point-by-point responses in the rebuttal. If our response has addressed your concerns, we would be grateful if you could consider updating your assessment during the discussion phase.

Thank you very much for your time and thoughtful feedback. We would be happy to provide any further clarifications.

Best regards, The Authors

Dear Reviewer,

Thank you so much for your thoughtful initial reviews!

With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.

Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!

Warm regards

Reviewer_gkqMOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors argue that teacher–student disagreement alone is insufficient to determine whether a token provides useful learning information, and identify a phenomenon called local discriminability collapse. They introduce LDA-OPD, which measures the teacher's preference margin between the top two candidates within the student's high-probability next-token set. These measurements are aggregated over sentence-level segments, and a BIC-based change-point procedure identifies a trajectory-specific boundary after which token-level distillation losses are removed. The retained prefix is reweighted, while inference-time generation remains unchanged.

Experiments across multiple Qwen3 student sizes, Gemma3 models, mathematical reasoning, code generation, and out-of-domain benchmarks demonstrate improvements over full OPD. Several controlled ablations support the advantage of adaptive release over fixed-prefix and alternative signal-based methods.

Critical strengths
  1. A simple and practical adaptive supervision mechanism.

LDA-OPD addresses a relevant limitation of fixed-prefix distillation: different trajectories may stop benefiting from dense supervision at different points.

The proposed method provides a concrete mechanism for choosing these boundaries without separately training models for several cutoff lengths. The combination of local candidate margins, sentence-level aggregation, and change-point detection offers a reasonably simple procedure that can be incorporated into existing OPD training workflows.

I consider the use of a trajectory-specific signal preferable in principle to assuming that a single cutoff is appropriate across all responses.

  1. Convincing empirical evidence across multiple settings.

The paper demonstrates improvements over full OPD for 1.7B, 4B, and 8B Qwen3 students, with additional positive results on Gemma3. Importantly, the evaluation extends beyond the mathematical reasoning benchmarks used for the primary development experiments. Results on GPQA, IFEval, and programmatic code generation suggest that the method does not merely improve performance on a single narrow benchmark family. The ablations also strengthen the contribution.

Critical weaknesses
  1. The methodological novelty is incremental relative to recent selective-OPD and prefix-distillation methods.

Prior work has already established that teacher supervision is not uniformly useful across token positions and that restricting OPD to reasoning prefixes can be effective. The distinguishing feature of LDA-OPD is therefore not selective supervision itself, but the use of a particular local margin and change-point mechanism to determine trajectory-specific prefix boundaries.

I view this as a meaningful engineering and empirical contribution, although the conceptual advance is narrower than the introduction occasionally suggests. In particular, the assertion that standard OPD relies on the premise that every token is uniformly beneficial is somewhat overstated given the related work.

  1. The scope and practical advantage over existing approaches need clearer delineation.

The method depends on white-box access to teacher logits and computes release decisions after complete rollouts. This limits applicability when the teacher is available only through a black-box API and prevents the training-time savings available to methods that terminate rollouts early.

The relative gains over fixed-prefix baselines also vary across configurations. The improvement becomes relatively small for the 8B student, and the Gemma3 evaluation covers only two mathematical reasoning benchmarks.

Furthermore, the mathematical reasoning experiments remain the most extensive part of the evaluation. The supplementary code experiments use a separate code-training source, so they demonstrate applicability to another domain rather than zero-shot transfer from the math-trained policy.

Questions for authors
  1. What is the primary conceptual distinction between LDA-OPD and recent token-teachability or contrast-aware OPD approaches? Is the contribution best understood as detecting a new failure mode, or as a practical trajectory-level selection strategy?

  2. Can the authors compare LDA-OPD with a stronger length-adaptive baseline that uses student uncertainty, response length, or another inexpensive trajectory statistic to choose per-sample cutoffs? This would help isolate the value of the proposed teacher-margin signal beyond its comparison with one fixed 25% prefix.

  3. Under what teacher–student capacity ratios does local discriminability collapse become sufficiently pronounced for LDA-OPD to provide meaningful improvements? Does the advantage diminish as the capacity gap narrows?

3 comments

W1

Regarding "selective-OPD and prefix-distillation methods" and "The distinguishing feature of LDA-OPD": Thank you for the suggestion. We will expand and clarify our discussion of recent related work. Our perspective focuses on how local discriminability changes along a trajectory. Compared with methods that select supervision at the token level, our contribution is a trajectory-level supervision allocation rule: it detects a sustained downward transition in local teacher contrast and selects a contiguous supervised prefix. This complements pointwise token selection and weighting. We will revise the introduction to acknowledge that selective supervision is already established and make the temporal boundary-selection contribution explicit. We also discussed the relationship with token-teachability or contrast-aware OPD approaches in Question 1.

Regarding "standard OPD relies on the premise that every token is uniformly beneficial": Thank you for the suggestion. We agree with this concern and will qualify the corresponding claims. (And we will incorporate this and all other updates into the revised PDF once all reviewer feedback has been collected).

W2

Regarding "the scope": Thank you for the suggestion. We will clarify the applicability conditions: the current method requires access to teacher logits and determines the supervision boundary after a complete rollout.

Regarding "relative gains over fixed-prefix baselines also vary across configurations" and "code experiments": Thank you for pointing these out. We will refine our discussion to clarify both the varying relative gains over the fixed-prefix baselines and the details of the code experiments with greater clarity in the final version.

Q1

(1) TA-OPD (arXiv:2605.26844), a recent token-teachability method, also considers the teacher's probability allocation within the student's local candidate set at each token position. It distinguishes learnable disagreement from disagreement caused by incompatible support and selects token positions with high teachability. We focus on changes in local discriminability across successive positions in a trajectory and use change-point detection to determine the supervision boundary of a contiguous prefix. We therefore characterize our contribution as a diagnosis of changes in local discriminability and a practical trajectory-level supervision selection strategy motivated by that diagnosis. The two approaches can be complementary: trajectory-level boundary selection can be combined with fine-grained token weighting, and the suffix may still contain useful local feedback. (2) CaOPD (arXiv:2604.16830), which we also cite, adjusts confidence targets using empirical correctness rates across multiple student rollouts and focuses on calibrating confidence to capability. Our release rule selects segments within a single trajectory and does not rely on outcome comparisons across multiple rollouts or an additional outcome reward.

Q2

Regarding other "length-adaptive baseline" and "the value of the proposed teacher-margin signal": Teacher Entropy on Candidate Set in Table 3 uses the teacher's entropy within the student's candidate set to select a cutoff and achieves a Math-AVG of 37.2%. We additionally evaluated Student Entropy on Candidate Set, which selects a cutoff for each sample using only student uncertainty. It achieves 37.5%, compared with 40.1% for the margin signal. These results support the effectiveness of the teacher-margin signal relative to the tested entropy-based alternatives in this setting.

Q3

Thank you for the question. For the 1.7B, 4B, and 8B students evaluated in the paper, LDA-OPD improves math AVG over full OPD by 3.3, 2.3, and 2.1 percentage points, respectively, with the gain decreasing as student size increases. However, the gains over FastOPD-2048 are 0.8, 1.4, and 0.5 percentage points and are not monotonic. The current results therefore do not establish a general threshold for the teacher–student capacity ratio. Our results and those of ExOPD (arXiv:2602.12125) also suggest that smaller students do not necessarily obtain larger overall gains from OPD, possibly because of limits on their learning capacity. We hypothesize that teacher–student distributional mismatch also affects the benefit, but this factor is coupled with student capacity. We plan to investigate this interplay more thoroughly in future work.

Dear Reviewer gkqM,

Since the discussion phase is ending soon, we would like to kindly ask you to take a look at our rebuttal. We have carefully addressed all the concerns raised in your review and provided point-by-point responses in the rebuttal. If our response has addressed your concerns, we would be grateful if you could consider updating your assessment during the discussion phase.

Thank you very much for your time and thoughtful feedback. We would be happy to provide any further clarifications.

Best regards, The Authors

Dear Reviewer,

Thank you so much for your thoughtful initial reviews!

With only two days left before the discussion window closes, I wanted to send a friendly reminder to check out the authors' rebuttal. They have put a lot of effort into answering the specific questions you raised.

Whenever you have a spare moment today, please drop a brief reply in the thread to let the authors know if their responses have addressed your concerns, or if any questions remain. Your timely feedback makes a huge difference in ensuring a fair and constructive review process!

Warm regards

Reviewer_AwDcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies strong-to-weak on-policy distillation (OPD), where a teacher supervises a smaller student on student-generated rollouts. The authors claim that teacher guidance can lose local discriminative contrast later in a response even when the mean sampled-token teacher-student advantage remains nonzero. Their proposed method, Local-Discriminability-Aware OPD (LDA-OPD) computes the teacher's top-1/top-2 log-probability margin within the student's top-K candidate set, aggregates it over sentence segments, and selects a downward change point using a criterion. When a change is accepted, suffix loss is masked and retained-prefix advantages are rescaled by the ratio of total valid tokens to retained valid tokens. This preserves total token-mask weight, although it does not generally preserve the realized loss magnitude. The method leaves inference-time generation unconstrained. Experiments cover mathematical reasoning, scientific QA, code generation, instruction following, several student sizes, and a second model family. The reported results show improvements over full OPD and the evaluated fixed-prefix baselines.

Critical strengths
  1. The paper points out an interesting research question about where teacher supervision remains informative along a student rollout. Its specific approach combines teacher margins on student-selected candidates with a trajectory-dependent release boundary. Also, Figure 1 provides an interesting analysis to diagnose the local discriminability collapse and the state-local candidate margin.

  2. The core margin, change-point, and masking operations are clearly described and the selection operates on the training loss rather than constraining inference-time generation, which enhance the practical usage.

Critical weaknesses
  1. The diagnostics in the main paper show declining teacher contrast, but this does not show that the corresponding supervision has become uninformative or harmful. The criterion detects a relative change in teacher margin without directly assessing the learning value of the discarded feedback. Although the candidate set is student-dependent, the margin measures teacher preferences without directly incorporating student probabilities within that set. The signal ablations support the effectiveness of margin-based release among the tested rules. However, I think it does not directly show that feedback in the released regions produces less useful updates. The causal interpretation of local discriminability collapse therefore remains insufficiently supported.

  2. The results support an improvement over the default fixed-prefix baseline, but the repeated evaluations do not exhibit the reliability of the gain over the strongest static setting. Comparisons with closely related selective OPD methods are also missing. The random-release control suggests that trajectory-dependent allocation matters, but this manuscript does not sufficiently explain how response length, actual supervised-token counts, and rescaling weights are controlled.

  3. This study describes rescaling as preserving loss magnitude, although the rule normalizes retained-token weights using token counts. This does not generally preserve the realized loss or gradient scale. The truncation-reweighting ablation study is useful, but consistent loss normalization and matched weighting across baselines are still needed to attribute the gains to boundary selection. The method also removes supervision from the entire suffix, potentially excluding useful later teacher feedback. The experimental results do not verify whether retaining such supervision would preserve or improve the reported gains.

Questions for authors
  1. Could the authors test whether feedback from the released suffix yields less useful updates than feedback from the retained prefix? Linking the margin to measured learning gains would strengthen the case for the proposed mechanism.

  2. I would like to see the comparison repeated against the strongest fixed-prefix setting and against the closely related selective OPD baselines discussed in related work. For the random-release control, I also wonder whether it could account for response length, with the supervised-token budget and the rescaling weights matched to LDA-OPD.

  3. Could the authors clarify what the rescaling rule actually preserves, and compare the methods under a consistent loss normalization and matched reweighting? I am also curious whether keeping supervision on the final-answer segment would preserve, or even improve, the gains from the current release rule.

8 comments

W1

Regarding "become uninformative or harmful": Thank you for the constructive suggestion. We agree with your assessment and will nuance our phrasing in the revision to describe the feedback in these regions more precisely as relatively less informative. (And we will incorporate this and all other updates into the revised PDF once all reviewer feedback has been collected).

Regarding "causal interpretation of local discriminability collapse": We added an experiment that applies direct supervision only to valid response tokens after the cutoff. The in-domain math AVG is 34.7%, above the base student's 24.9% but below full OPD's 36.8% and LDA-OPD's 40.1%. This indicates that suffix supervision still has learning value, although a training strategy that relies solely on suffix supervision yields smaller gains in this setting. We will refine our claims to avoid broadly characterizing the released suffix as uninformative or harmful. Furthermore, we note that concurrent work has observed similar phenomena (e.g., arXiv:2605.27028). We also conducted an analysis experiment based on the conditional gradient SNR. On the initial Qwen3-1.7B checkpoint, using equal numbers of sampled states from the retained and released regions, the pooled conditional gradient SNR is 0.548 versus 0.283, and the released region has a 79.1% lower mean squared norm of the centered expected gradient. These measurements support weaker average local action-dependent signal in the released region. Please refer to our response to Reviewer hSy8 regarding question 2.

W2

Regarding "the gain over the strongest static setting": Thank you for pointing this out. For the 1.7B student, Table 8 reports 40.07±0.12% for LDA-OPD versus 38.93±0.34% for FastOPD-2048. This comparison does not cover the strongest fixed prefix of 4096 tokens. The three FastOPD-4096 experiments average 38.77%(39.4%, 38.6%, and 38.3%), below the 40.07% reported for LDA-OPD in Table 8.

Regarding "Comparisons with closely related selective OPD methods": We also compared against Uni-OPD and SAF-OPD on the 1.7B student. Their Math-AVG scores are 38.2% and 39.1%, respectively, both below LDA-OPD's 40.1%.

Regarding more "random-release control": Thank you for suggesting these clarifications and ablations. In our random-release setup, we first determine the truncation points following the exact procedure of LDA-OPD, and then randomly permute these truncation points across different trajectories within the same batch. If an assigned truncation point exceeds the maximum length of its new trajectory, it is clipped to that trajectory's length, and the truncation points of the remaining trajectories are randomly extended to compensate for the number of training tokens. Subsequent rescaling weights are then computed based on the final truncation point and the length of the new trajectory. This design ensures that the total number of supervised tokens remains strictly consistent with that of LDA-OPD. Furthermore, on the 4B student model, the paper reports 53.1% for LDA-OPD without reweighting and 53.9% with reweighting. We additionally evaluated Random-release without and with reweighting on the 4B student, obtaining 51.1% and 50.7%, respectively. These results suggest that the effect of reweighting may depend on the quality of the updates induced by the supervision selection strategy. We hypothesize that because random-release induces relatively unstable optimization directions, applying reweighting leads to a slight degradation in performance.

W3

Regarding "describes rescaling as preserving loss magnitude": Thank you for noting this. The phrase “preserving loss magnitude” is inaccurate. The rule preserves the total token-mask weight, rather than the realized loss or expected loss. We will correct the wording accordingly.

Regarding "consistent loss normalization and matched weighting across baselines": We compared the methods without reweighting.

Qwen3-4B Math-AVGWithout reweightingWith reweighting
FastOPD-204852.5%52.7%
Random release51.1%50.7%
LDA-OPD53.1%53.9%

LDA-OPD exceeds FastOPD-2048 by 0.6 percentage points without reweighting and by 1.2 points with the same per-sample reweighting formula. Its advantage over random release is also present under both settings. These comparisons show that the gains are not entirely attributable to reweighting.

Regarding "useful later teacher feedback": Please see our response to the first weakness.

Q1

Regarding "the released suffix yields less useful updates": Thank you for the suggestion. We added an experiment with suffix-only supervision. Please see our response to the first weakness. We also measured local action-centered gradient signal and conditional SNR, in addition to evaluating suffix-only supervision. Please refer to our response to Reviewer hSy8 regarding question 2.

Q2

Regarding "the comparison repeated against the strongest fixed-prefix setting", "related selective OPD baselines" and "the random-release control": Please see our response to the second weakness.

Q3

Regarding "what the rescaling rule actually preserves" and "a consistent loss normalization and matched reweighting": The rescaling preserves the total token-mask weight, rather than the realized loss or gradient norm. We additionally evaluated FastOPD-2048 on the 4B student using the same per-sample reweighting formula and a loss denominator based on all valid response tokens, as in LDA-OPD. It achieves 52.7%, compared with 53.9% for LDA-OPD. Please also see our response to Weakness 3 for the comparisons without reweighting. We hypothesize that when the supervision signal is of sufficiently high quality, the rescaling mechanism essentially serves to amplify the benefits of this high-quality supervision.

Regarding "keeping supervision on the final-answer segment": We also evaluated supervision on both the retained prefix and the final-answer segment on the 4B student. This variant achieves 51.8%, below LDA-OPD's 53.9%, indicating that additionally supervising the final-answer segment (after "<\/think>") does not improve performance in this setting.

Dear Reviewer AwDc,

Since the discussion phase is ending soon, we would like to kindly ask you to take a look at our rebuttal. We have carefully addressed all the concerns raised in your review and provided point-by-point responses in the rebuttal. If our response has addressed your concerns, we would be grateful if you could consider updating your assessment during the discussion phase.

Thank you very much for your time and thoughtful feedback. We would be happy to provide any further clarifications.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.