- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
On-policy distillation matches the teacher on prefixes generated by the student. On failing trajectories, the teacher’s continuation accuracy declines as generation proceeds, whereas on successful trajectories it stays stable (Figure 1a). Using as a proxy for the gradient budget, the positional budget shares of vanilla OPD differ by at most 1.8 points across these outcomes (Figure 1c). The authors therefore cast matching as a latent lifetime and replace the truncated objective by its expectation, which produces a survival weight. Cox-OPD instantiates that weight as a proportional hazard with baseline and one-sided deviation . The main experiments use and , and the zero-hazard case recovers ordinary OPD. Math and code are distilled into Qwen3 students at 0.6B, 1.7B, and 4B, with reproduced OPD and IW-OPD as the comparisons.
Critical strengths
-
Figure 1 shows that teacher recoverability separates by the final outcome, whereas the positional budget of vanilla OPD does not. Figure 2a shows that, within a position bin, prefixes with lower are more often unrecoverable, and Figure 2b shows that shuffling removes the weight gap between trajectory groups. This is the most informative measurement in the paper.
-
The weight uses log probabilities that OPD already caches, and zero hazard reduces to vanilla OPD. Proposition 2 bounds the suffix sensitivity of this link, and Figure 3b shows that the bound is nearly attained. The method is therefore a small and easily checked modification of OPD.
Critical weaknesses
-
The lifetime is motivated by the teacher’s continuation accuracy . The hazard does not take as an input. Equation 8 constructs only from the sampled token, and only through the one-sided gap . The paper states that means the teacher assigns at least as much probability to that token as the student does, even if the teacher cannot continue from the prefix. Read in the other direction, means only that the student assigns the sampled token more probability than the teacher does. It does not mean that the continuation has failed. A prefix on which matching has already failed can therefore receive the same local deviation as a prefix from which the teacher can still complete the solution. does not record the event that the lifetime was introduced to describe.
-
The baseline satisfies . The training weight does not evaluate that endpoint. Equation 11 sums the increments over , and Proposition 2 sets the baseline envelope to . The last-token envelope is therefore , which is positive and depends only on and . Appendix A.4 places the remaining mass on the restricted lifetime through . In Figure 1a, the recoverability ratio on successful trajectories remains near 1.07 at the end, so those trajectories are still continuable. That ratio does not change , , or . The endpoint is a property of the chosen curve, not a death time estimated from those continuations.
-
The paper proves that every positive non-increasing weight with has a hazard . Positional decay satisfies this condition, and so does Cox-OPD. Proposition 1 and Corollary 1 hold for either weight: each is an identity within a model that has already been chosen. The survival representation is therefore shared by the whole class. What is specific to Cox-OPD is the pair in points 1 and 2, not the existence of a hazard. Figure 4 changes the baseline, the deviation, or . Each alternative is another weight rule in this class, and none of those rules takes a continuation outcome as input. The comparison does not reject the present pair while retaining another non-increasing weight.
-
The relative risk is with and , so a larger deviation can only raise the hazard. Proposition 2 therefore gives . At the final token this is . No value of raises the last-token weight above that upper bound. When every , the bound is attained and , which is still a function of and . Figure 1a finds that successful trajectories remain recoverable at the final token. The formula can assign those trajectories positive probability . The success of the continuation does not raise the bound. This is a property of the formula, not of any particular training run.
-
The paper states that Definition 1 specifies a latent lifetime that is never observed, and that the weights are computed from cached scores and realized lengths without outcome labels. Figure 2 then compares these weights with a recoverability label computed separately. That label is not an input to Equation 11, so the comparison cannot change , , or . The figure can establish a correlation. It does not supply a measurement equation in which observes . The weight in the training advantage, Equation 6, remains the specified function of and .
Questions for authors
-
The paper states that the lifetime is never observed and that the weights do not use outcome labels. Which quantity is then an observation of , as opposed to the specified function of and ?
-
Which observation rejects the present while still accepting another positive non-increasing weight with ? The hazard representation in the paper covers both.
-
For fixed and , Proposition 2 gives . Figure 1a reports a recoverability ratio near 1.07 at the end of successful trajectories. Which quantity in the model increases this upper bound on those trajectories?
-
Figure 2 compares with a recoverability label that Equation 11 does not take as input. Which equation treats as a measurement of , so that this comparison could update the hazard rather than only plot it?
-
If the latent lifetime is fully specified before any continuation is observed, which later continuation outcome can count as evidence against that specification?
-
The students are Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B, and the teachers are Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507. The reported evaluations are math, on AIME24/25/26 and HMMT25, and code, on HumanEval+, MBPP+, and LiveCodeBench in the multi-teacher setting. Can the same comparison be run on a model family other than Qwen3, and on a domain other than math and code?
3 comments
Response to Reviewer 7YBR (2/2)
W3 and Q2. What is specific to Cox-OPD. We agree that the hazard representation, Proposition 1, and Corollary 1 hold for every positive non-increasing weight with , and that what is specific to Cox-OPD is its hazard, built from and through the Cox link. This generality comes from the purpose of the survival view, to deepen the understanding of OPD and give a general form of credit assignment (G1 of the General Response). Within this form, choosing a positive non-increasing credit rule is equivalent to choosing a hazard, so every such rule implicitly specifies a lifetime for teacher matching. Uniform OPD treats teacher matching as surviving the full rollout, positional decay uses a hazard that depends only on position, and the direct product limit uses the deviation itself as the hazard.
For the experiments, we choose the Cox hazard as one natural instance of this form. Its covariate links the hazard to teacher acceptance, since speculative-decoding verification by the teacher would reject the sampled token with probability (Section 4). The classical proportional-hazards form combines this covariate with the baseline , and with this baseline Proposition 2 bounds the cumulative sensitivity of subsequent log-weights to any single deviation by . In Figures 4a and 4b, Cox-OPD gives the largest AIME25 gain among the tested variants (G3 of the General Response).
For Q2, the understanding of OPD and the general form above do not depend on the particular hazard, so neither requires an observation that rejects the present while accepting another rule. Every positive non-increasing weight with is a valid credit rule under the survival view, and the Cox hazard is the instance we use. Its and are hyperparameters set by grid search (W5 and Q4 below), not fitted to observed failure times. In practice, this instance improves distillation over vanilla OPD. It raises average performance in all six single-teacher pairs (G3 of the General Response), at all three student scales in both domains of the multi-teacher setting (Section 5), and in the three further settings reported in G4 of the General Response.
W5 and Q4. Recoverability as a measurement of . As discussed in G3 of the General Response, teacher recoverability serves as a diagnostic rather than as a measurement of . Section 3 introduces it "as an empirical diagnostic", and Figure 2 tests "whether these weights track independently measured teacher recoverability" (Section 4). Figure 2 therefore checks a fixed rule without updating , , or . As W5 notes, the figure establishes a correlation rather than a measurement, and that is its intended role. The hyperparameters and are instead selected by grid search on downstream evaluations. Using recoverability during training would be costly at scale (G3 of the General Response).
Q1. Observation of . As discussed under W1 and W5, neither nor is an observation of , and our method does not require one. The paper specifies the latent lifetime through the hazard (G2 of the General Response), so the weight is the specified function that Q1 describes. More precisely, it depends on the position , the realized length , and the earlier deviations , while the current affects only later weights (W1). Q5 below discusses how the resulting credit is evaluated.
Q5. Evidence against the specification. As discussed in G2 of the General Response, the specification is a credit design within the survival view rather than a hypothesis about when continuations fail. It is therefore evaluated by the quality of the resulting credit, mainly by downstream accuracy relative to other credit rules (G3 of the General Response, and W3 and Q2 above), and not by individual continuation outcomes. These outcomes instead supply recoverability, the diagnostic that motivated the survival view (Figure 1) and that the resulting weights track beyond position (Figure 2).
Q6. Other model families and domains. Yes. We ran the same comparison between Cox-OPD and vanilla OPD in three further settings, and Cox-OPD improves the overall score in all three (G4 of the General Response). All three add a domain beyond math and code, and two use a model family other than Qwen3. τ²-bench tests agentic tool use with a Qwen3-4B student, Multi-IF tests multilingual multi-turn instruction following with an R1-Distill-Qwen-1.5B student, and Open-MOPD tests multi-teacher distillation across math, code, and instruction following with a SmolLM3-3B student.
We hope these answers clarify the role of the survival view and the evidence behind the design. We would be glad to discuss any remaining point.
Thank you for the clarification and the additional experiments; however, since the paper itself motivates the matching lifetime with recoverability while the training weights in Eqs. 8–11 depend only on and , with remaining a post-hoc diagnostic, I do not regard the core concern as resolved and maintain my original rating.
Response to Reviewer 7YBR (2/2)
We thank the reviewer for a careful and detailed review. Most of the concerns regards our survival view as a model of an observed failure event, with a lifetime to be measured and a hazard to be fitted. G1 and G2 of the General Response clarify the role of the survival view. The view mainly provides a deeper understanding of OPD and a general form of credit assignment, and its lifetime is latent and specified through the hazard rather than fitted to observed failure times. Below, we address each weakness and question.
W1. δt and the failure event. In Cox-OPD, δt is only the per-step input to the hazard, and the quantity meant to track the event is the accumulated survival weight wt. Section 4 makes this point for a zero δt. "A zero value of δt only indicates that the teacher assigns at least as much probability to the sampled token as the student. The teacher may still struggle to produce a successful continuation from the resulting prefix." Likewise, a positive δt only raises the hazard and does not mark a failure. For a fixed realized length L, wt depends only on the hazards before position t (Definition 1 and Remark 1). In the reviewer's example, the shared current deviation therefore does not enter the weight at that position, and the two prefixes can still receive different weights when their earlier deviations differ. Figure 2c checks whether these differences track recoverability. Within groups defined by normalized position and the magnitude of the current log-ratio ∣ℓt∣, recoverability rises from 56% at low wt to 75% at high wt, while it stays flat when tokens are ranked by the position-only weight Bt. Since wt differs from Bt only through the earlier deviations, this contrast is consistent with those deviations carrying information about recoverability.
W2, W4, and Q3. What the endpoint and the bound describe. The failure in the survival view concerns the teacher's supervision rather than the trajectory. In OPD, the teacher scores every token of the student's own rollout, and the divergence between the student's and the teacher's prefix distributions does not decrease with depth (Section 3). This motivates prefix length as the clock of a latent matching lifetime. Section 3 states, "We model matching failure as a latent event, using teacher-continuation recoverability as an empirical diagnostic." We think the "death time" in W2 corresponds to our latent failure time of this matching. Past this time, the teacher's supervision is no longer trusted at full strength. Proposition 1 averages over this latent time, so every token of the realized rollout keeps a positive weight without locating a failure point. Because this failure is hard to identify on an individual trajectory (G2 of the General Response), the paper specifies its distribution through the hazard rather than estimating it from continuations. The endpoint and the bound in W2 and W4 are therefore intended properties of this specification.
Within this specification, the baseline sets the prior level of trust at token t to Bt=S0(t−1). The endpoint S0(L)=0 does not mark a failure at L, because Pr(T=L∣y)=wL in Appendix A.4 is the probability that matching is still alive at the last token. Figure 1a shows that the recoverability ratio RS/RT separates failing from successful rollouts, 0.49 against 1.07 near the end. Section 1 uses this separation to motivate a trajectory-dependent lifetime of teacher matching. In Cox-OPD, each rollout's credit is discounted from this shared prior by its own earlier deviations, and Figures 2a and 2c check the resulting separation against recoverability. A positive δt lowers the credit of later tokens, while a zero δt adds only the baseline hazard, since agreement on one token does not show that the matching remains relevant. For β>0, the weight therefore reaches the prior Bt only when all earlier deviations vanish, and the bound in Q3 follows from this one-sided design. The bound also applies to successful, recoverable prefixes, and we tested whether raising it helps.
A signed variant within the general form replaces δt with st=δt−(1−π0/πT)+, so agreement can raise the weight above Bt. In the Figure 4a setting (30B teacher, 4B student, mean@32), it falls below Cox-OPD on both Math benchmark. This is likely because agreement on one token does not show the teacher can still continue from the prefix and therefore does not guarantee a weight above Bt. Later agreement can also offset the extra hazard from earlier deviations, so the weight can reach Bt despite them. We will add the signed variant to Figure 4a in the revision.