acceptodds
Under review as a conference paper at ICLR 2027

Matching While Alive: A Survival View of On-Policy Trajectory Distillation

Abstract

On-policy distillation (OPD) brings teacher supervision to student-generated prefixes, but its continued availability does not ensure that it remains equally relevant along each trajectory. Our analysis reveals differences in teacher recoverability across trajectories, while vanilla OPD treats teacher matching as an uninterrupted process along every rollout. We therefore formulate on-policy trajectory distillation as a survival problem with a latent failure time for teacher matching. Marginalizing over the latent alive state converts the truncated matching objective into a survival-weighted one. This connects trajectory-level credit assignment to hazard modeling, with vanilla OPD recovered as the zero-hazard case. This framework admits a simple realization (Cox-OPD) based on a Cox proportional hazards model that combines a baseline hazard over prefix depth with observable student–teacher deviation. The accumulated hazard defines a trajectory-specific survival profile, allowing prefix history to shape how matching credit evolves along the rollout. Empirically, these weights reflect variation in independently measured teacher recoverability, and experiments on mathematical reasoning and code generation show gains across student scales in both single- and multi-teacher settings. Across all six single-teacher pairs, this survival-based extension improves average performance over vanilla OPD, with mathematical reasoning scores for 4B students increasing by 2.51 points on average across four benchmarks and both teacher settings. Ablations further highlight how the survival formulation structures the trade-off between retained credit and trajectory differentiation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy distillation, trajectory distillation, survival analysis, latent failure time, hazard modeling, credit assignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7YBROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

On-policy distillation matches the teacher on prefixes generated by the student. On failing trajectories, the teacher’s continuation accuracy declines as generation proceeds, whereas on successful trajectories it stays stable (Figure 1a). Using ∣ℓt∣|\ell_t| as a proxy for the gradient budget, the positional budget shares of vanilla OPD differ by at most 1.8 points across these outcomes (Figure 1c). The authors therefore cast matching as a latent lifetime and replace the truncated objective by its expectation, which produces a survival weight. Cox-OPD instantiates that weight as a proportional hazard with baseline S0(t)=1−(t/L)pS_0(t)=1-(t/L)^p and one-sided deviation δt\delta_t. The main experiments use β=3\beta=3 and p=2p=2, and the zero-hazard case recovers ordinary OPD. Math and code are distilled into Qwen3 students at 0.6B, 1.7B, and 4B, with reproduced OPD and IW-OPD as the comparisons.

Critical strengths
  1. Figure 1 shows that teacher recoverability separates by the final outcome, whereas the positional budget of vanilla OPD does not. Figure 2a shows that, within a position bin, prefixes with lower wtw_t are more often unrecoverable, and Figure 2b shows that shuffling δ\delta removes the weight gap between trajectory groups. This is the most informative measurement in the paper.

  2. The weight uses log probabilities that OPD already caches, and zero hazard reduces to vanilla OPD. Proposition 2 bounds the suffix sensitivity of this link, and Figure 3b shows that the bound is nearly attained. The method is therefore a small and easily checked modification of OPD.

Critical weaknesses
  1. The lifetime is motivated by the teacher’s continuation accuracy RS(u)R_S(u). The hazard does not take RSR_S as an input. Equation 8 constructs δt\delta_t only from the sampled token, and only through the one-sided gap (1−πT/π0)+(1-\pi_T/\pi_0)^+. The paper states that δt=0\delta_t=0 means the teacher assigns at least as much probability to that token as the student does, even if the teacher cannot continue from the prefix. Read in the other direction, δt>0\delta_t>0 means only that the student assigns the sampled token more probability than the teacher does. It does not mean that the continuation has failed. A prefix on which matching has already failed can therefore receive the same local deviation as a prefix from which the teacher can still complete the solution. δt\delta_t does not record the event that the lifetime was introduced to describe.

  2. The baseline S0(t)=1−(t/L)pS_0(t)=1-(t/L)^p satisfies S0(L)=0S_0(L)=0. The training weight does not evaluate that endpoint. Equation 11 sums the increments over k<tk<t, and Proposition 2 sets the baseline envelope to Bt=S0(t−1)B_t=S_0(t-1). The last-token envelope is therefore BL=S0(L−1)B_L=S_0(L-1), which is positive and depends only on LL and pp. Appendix A.4 places the remaining mass on the restricted lifetime through Pr⁡(T=L∣y)=wL\Pr(T=L\mid y)=w_L. In Figure 1a, the recoverability ratio on successful trajectories remains near 1.07 at the end, so those trajectories are still continuable. That ratio does not change LL, pp, or S0S_0. The endpoint is a property of the chosen curve, not a death time estimated from those continuations.

  3. The paper proves that every positive non-increasing weight with w1=1w_1=1 has a hazard hk=1−wk+1/wkh_k=1-w_{k+1}/w_k. Positional decay satisfies this condition, and so does Cox-OPD. Proposition 1 and Corollary 1 hold for either weight: each is an identity within a model that has already been chosen. The survival representation is therefore shared by the whole class. What is specific to Cox-OPD is the pair (S0,δ)(S_0,\delta) in points 1 and 2, not the existence of a hazard. Figure 4 changes the baseline, the deviation, or (β,p)(\beta,p). Each alternative is another weight rule in this class, and none of those rules takes a continuation outcome as input. The comparison does not reject the present pair while retaining another non-increasing weight.

  4. The relative risk is eβδte^{\beta\delta_t} with β≥0\beta\ge 0 and δt≥0\delta_t\ge 0, so a larger deviation can only raise the hazard. Proposition 2 therefore gives wt≤Bt=S0(t−1)w_t\le B_t=S_0(t-1). At the final token this is wL≤S0(L−1)w_L\le S_0(L-1). No value of δ\delta raises the last-token weight above that upper bound. When every δk=0\delta_k=0, the bound is attained and wL=S0(L−1)w_L=S_0(L-1), which is still a function of LL and pp. Figure 1a finds that successful trajectories remain recoverable at the final token. The formula can assign those trajectories positive probability wLw_L. The success of the continuation does not raise the bound. This is a property of the formula, not of any particular training run.

  5. The paper states that Definition 1 specifies a latent lifetime that is never observed, and that the weights are computed from cached scores and realized lengths without outcome labels. Figure 2 then compares these weights with a recoverability label computed separately. That label is not an input to Equation 11, so the comparison cannot change β\beta, pp, or S0S_0. The figure can establish a correlation. It does not supply a measurement equation in which RSR_S observes τ\tau. The weight in the training advantage, Equation 6, remains the specified function of t/Lt/L and δt\delta_t.

Questions for authors
  1. The paper states that the lifetime is never observed and that the weights do not use outcome labels. Which quantity is then an observation of τ\tau, as opposed to the specified function of t/Lt/L and δt\delta_t?

  2. Which observation rejects the present (S0,β)(S_0,\beta) while still accepting another positive non-increasing weight with w1=1w_1=1? The hazard representation in the paper covers both.

  3. For fixed LL and pp, Proposition 2 gives wL≤S0(L−1)w_L\le S_0(L-1). Figure 1a reports a recoverability ratio near 1.07 at the end of successful trajectories. Which quantity in the model increases this upper bound on those trajectories?

  4. Figure 2 compares wtw_t with a recoverability label that Equation 11 does not take as input. Which equation treats RSR_S as a measurement of τ\tau, so that this comparison could update the hazard rather than only plot it?

  5. If the latent lifetime is fully specified before any continuation is observed, which later continuation outcome can count as evidence against that specification?

  6. The students are Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B, and the teachers are Qwen3-4B-Instruct-2507 and Qwen3-30B-A3B-Instruct-2507. The reported evaluations are math, on AIME24/25/26 and HMMT25, and code, on HumanEval+, MBPP+, and LiveCodeBench in the multi-teacher setting. Can the same comparison be run on a model family other than Qwen3, and on a domain other than math and code?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7YBR (2/2)

We thank the reviewer for a careful and detailed review. Most of the concerns regards our survival view as a model of an observed failure event, with a lifetime to be measured and a hazard to be fitted. G1 and G2 of the General Response clarify the role of the survival view. The view mainly provides a deeper understanding of OPD and a general form of credit assignment, and its lifetime is latent and specified through the hazard rather than fitted to observed failure times. Below, we address each weakness and question.

W1. δt\delta_t and the failure event. In Cox-OPD, δt\delta_t is only the per-step input to the hazard, and the quantity meant to track the event is the accumulated survival weight wtw_t. Section 4 makes this point for a zero δt\delta_t. "A zero value of δt\delta_t only indicates that the teacher assigns at least as much probability to the sampled token as the student. The teacher may still struggle to produce a successful continuation from the resulting prefix." Likewise, a positive δt\delta_t only raises the hazard and does not mark a failure. For a fixed realized length LL, wtw_t depends only on the hazards before position tt (Definition 1 and Remark 1). In the reviewer's example, the shared current deviation therefore does not enter the weight at that position, and the two prefixes can still receive different weights when their earlier deviations differ. Figure 2c checks whether these differences track recoverability. Within groups defined by normalized position and the magnitude of the current log-ratio ∣ℓt∣|\ell_t|, recoverability rises from 56% at low wtw_t to 75% at high wtw_t, while it stays flat when tokens are ranked by the position-only weight BtB_t. Since wtw_t differs from BtB_t only through the earlier deviations, this contrast is consistent with those deviations carrying information about recoverability.

W2, W4, and Q3. What the endpoint and the bound describe. The failure in the survival view concerns the teacher's supervision rather than the trajectory. In OPD, the teacher scores every token of the student's own rollout, and the divergence between the student's and the teacher's prefix distributions does not decrease with depth (Section 3). This motivates prefix length as the clock of a latent matching lifetime. Section 3 states, "We model matching failure as a latent event, using teacher-continuation recoverability as an empirical diagnostic." We think the "death time" in W2 corresponds to our latent failure time of this matching. Past this time, the teacher's supervision is no longer trusted at full strength. Proposition 1 averages over this latent time, so every token of the realized rollout keeps a positive weight without locating a failure point. Because this failure is hard to identify on an individual trajectory (G2 of the General Response), the paper specifies its distribution through the hazard rather than estimating it from continuations. The endpoint and the bound in W2 and W4 are therefore intended properties of this specification.

Within this specification, the baseline sets the prior level of trust at token tt to Bt=S0(t−1)B_t=S_0(t-1). The endpoint S0(L)=0S_0(L)=0 does not mark a failure at LL, because Pr⁡(T=L∣y)=wL\Pr(T=L\mid y)=w_L in Appendix A.4 is the probability that matching is still alive at the last token. Figure 1a shows that the recoverability ratio RS/RTR^S/R^T separates failing from successful rollouts, 0.49 against 1.07 near the end. Section 1 uses this separation to motivate a trajectory-dependent lifetime of teacher matching. In Cox-OPD, each rollout's credit is discounted from this shared prior by its own earlier deviations, and Figures 2a and 2c check the resulting separation against recoverability. A positive δt\delta_t lowers the credit of later tokens, while a zero δt\delta_t adds only the baseline hazard, since agreement on one token does not show that the matching remains relevant. For β>0\beta>0, the weight therefore reaches the prior BtB_t only when all earlier deviations vanish, and the bound in Q3 follows from this one-sided design. The bound also applies to successful, recoverable prefixes, and we tested whether raising it helps.

A signed variant within the general form replaces δt\delta_t with st=δt−(1−π0/πT)+s_t=\delta_t-(1-\pi_0/\pi_T)^+, so agreement can raise the weight above BtB_t. In the Figure 4a setting (30B teacher, 4B student, mean@32), it falls below Cox-OPD on both Math benchmark. This is likely because agreement on one token does not show the teacher can still continue from the prefix and therefore does not guarantee a weight above BtB_t. Later agreement can also offset the extra hazard from earlier deviations, so the weight can reach BtB_t despite them. We will add the signed variant to Figure 4a in the revision.

MethodAIME25AIME26
OPD46.1554.06
Cox-OPD50.7354.79
Cox-OPD (signed score)47.2353.21
AuthorsOpenReview ↗

Response to Reviewer 7YBR (2/2)

W3 and Q2. What is specific to Cox-OPD. We agree that the hazard representation, Proposition 1, and Corollary 1 hold for every positive non-increasing weight with w1=1w_1=1, and that what is specific to Cox-OPD is its hazard, built from S0S_0 and δ\delta through the Cox link. This generality comes from the purpose of the survival view, to deepen the understanding of OPD and give a general form of credit assignment (G1 of the General Response). Within this form, choosing a positive non-increasing credit rule is equivalent to choosing a hazard, so every such rule implicitly specifies a lifetime for teacher matching. Uniform OPD treats teacher matching as surviving the full rollout, positional decay uses a hazard that depends only on position, and the direct product limit ∏k<t(1−δk)\prod_{k<t}(1-\delta_k) uses the deviation itself as the hazard.

For the experiments, we choose the Cox hazard as one natural instance of this form. Its covariate δt\delta_t links the hazard to teacher acceptance, since speculative-decoding verification by the teacher would reject the sampled token with probability δt\delta_t (Section 4). The classical proportional-hazards form combines this covariate with the baseline S0S_0, and with this baseline Proposition 2 bounds the cumulative sensitivity of subsequent log-weights to any single deviation by βeβ\beta e^\beta. In Figures 4a and 4b, Cox-OPD gives the largest AIME25 gain among the tested variants (G3 of the General Response).

For Q2, the understanding of OPD and the general form above do not depend on the particular hazard, so neither requires an observation that rejects the present (S0,β)(S_0,\beta) while accepting another rule. Every positive non-increasing weight with w1=1w_1=1 is a valid credit rule under the survival view, and the Cox hazard is the instance we use. Its β\beta and pp are hyperparameters set by grid search (W5 and Q4 below), not fitted to observed failure times. In practice, this instance improves distillation over vanilla OPD. It raises average performance in all six single-teacher pairs (G3 of the General Response), at all three student scales in both domains of the multi-teacher setting (Section 5), and in the three further settings reported in G4 of the General Response.

W5 and Q4. Recoverability as a measurement of τ\tau. As discussed in G3 of the General Response, teacher recoverability RSR^S serves as a diagnostic rather than as a measurement of τ\tau. Section 3 introduces it "as an empirical diagnostic", and Figure 2 tests "whether these weights track independently measured teacher recoverability" (Section 4). Figure 2 therefore checks a fixed rule without updating β\beta, pp, or S0S_0. As W5 notes, the figure establishes a correlation rather than a measurement, and that is its intended role. The hyperparameters β\beta and pp are instead selected by grid search on downstream evaluations. Using recoverability during training would be costly at scale (G3 of the General Response).

Q1. Observation of τ\tau. As discussed under W1 and W5, neither δt\delta_t nor RSR^S is an observation of τ\tau, and our method does not require one. The paper specifies the latent lifetime through the hazard (G2 of the General Response), so the weight is the specified function that Q1 describes. More precisely, it depends on the position tt, the realized length LL, and the earlier deviations δ<t\delta_{<t}, while the current δt\delta_t affects only later weights (W1). Q5 below discusses how the resulting credit is evaluated.

Q5. Evidence against the specification. As discussed in G2 of the General Response, the specification is a credit design within the survival view rather than a hypothesis about when continuations fail. It is therefore evaluated by the quality of the resulting credit, mainly by downstream accuracy relative to other credit rules (G3 of the General Response, and W3 and Q2 above), and not by individual continuation outcomes. These outcomes instead supply recoverability, the diagnostic that motivated the survival view (Figure 1) and that the resulting weights track beyond position (Figure 2).

Q6. Other model families and domains. Yes. We ran the same comparison between Cox-OPD and vanilla OPD in three further settings, and Cox-OPD improves the overall score in all three (G4 of the General Response). All three add a domain beyond math and code, and two use a model family other than Qwen3. τ²-bench tests agentic tool use with a Qwen3-4B student, Multi-IF tests multilingual multi-turn instruction following with an R1-Distill-Qwen-1.5B student, and Open-MOPD tests multi-teacher distillation across math, code, and instruction following with a SmolLM3-3B student.

We hope these answers clarify the role of the survival view and the evidence behind the design. We would be glad to discuss any remaining point.

Thank you for the clarification and the additional experiments; however, since the paper itself motivates the matching lifetime with recoverability RSR_S while the training weights in Eqs. 8–11 depend only on t/Lt/L and δt\delta_t, with RSR_S remaining a post-hoc diagnostic, I do not regard the core concern as resolved and maintain my original rating.

Reviewer_pFgYOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a modification to on-policy distillation. The proposed method, Cox-OPD, reduces the weight of later trainingsignals when earlier student tokens have departed from the teacher’s distribution. The weights use asurvival-model form, with a chosen baseline hazard and a student–teacher disagreement measure. They are inexpensive to compute and improve the reported average performance across several settings. However, the interpretation of the weights and their connection to the training objective need further clarification.

Critical strengths
  1. The method is simple and improves the reported average performance across several settings.
  2. The reported improvements across all six single-teacher configurations are encouraging.
  3. The recoverability analysis provides useful evidence that the weights contain information beyond token position.
Critical weaknesses
  1. The meaning of the latent lifetime seems unclear. The lifetime is specified through the weighting rule and is never observed. A survival representation does not establish that the weights measure the probability that supervision remains useful.
  2. the survival function in formula 4 is different from the commonly used definition. it should be the probability of greater than t instead of greater and equal to t.
  3. The comparisons change both the allocation and total amount of supervision. They do not clearly separate the contribution of disagreement history from position decay or changes in update size.
  4. The parameter-selection procedure needs clarification.
Questions for authors
  1. Do larger weights predict more beneficial student updates? If the weights are intended as failure probabilities, what event could be used to assess their calibration?
  2. Can the authors clarify the optimization target and compare the current weights with weights based on a horizon fixed before generation?
  3. How variable are the gains across training seeds?
  4. Were the evaluation benchmarks used to choose the default parameters?
4 comments
AuthorsOpenReview ↗

Response to Reviewer pFgY (1/2) W1 to W4

We thank the reviewer for the careful reading and detailed questions. Below, we address each weakness and question.

W1. What the lifetime means. As discussed in G2 of the General Response, the lifetime models whether the teacher's supervision can still be trusted at full strength along the rollout, so what fails is the supervision rather than the trajectory. The paper does not use the survival representation as evidence that the weights track the usefulness of supervision. As Section 3 states, whether the lifetime reflects teacher relevance depends on the hazard design (G2 of the General Response). The paper therefore evaluates the resulting credit empirically (G3 of the General Response), by whether it tracks teacher recoverability, an independent diagnostic of teacher relevance (Figure 2), and whether it improves distillation (Section 5).

W2. The convention in Equation 4. Equation 4 uses a standard left-continuous convention for discrete failure times, wt=Pr⁡(τ≥t∣y)=∏k<t(1−hk)w_t=\Pr(\tau\ge t\mid y)=\prod_{k<t}(1-h_k) (e.g., Rodríguez, Generalized Linear Models lecture notes, Section 7.5.1). It aligns each weight with its prefix, so w1=1w_1=1 and, for a fixed realized length, "wtw_t depends only on the hazards accumulated before position tt" (Definition 1). In the reviewer's form S(t∣y)=Pr⁡(τ>t∣y)S(t\mid y)=\Pr(\tau>t\mid y), the same weight is wt=S(t−1∣y)w_t=S(t-1\mid y), so the two forms differ only by this one-step shift, and Proposition 1 and the method are unchanged.

W3. Total credit, position, and history. At about the same normalized credit horizon, the Cox rule gains much more than position-only decay. In Figure 4c, β=4\beta=4 at p=2p=2 lies at about the same horizon as linear decay (β=0\beta=0, p=1p=1), and its AIME25 gain over vanilla OPD is 4.28 points against 1.20 (Figure 4b). Within groups defined by normalized position and the gradient proxy ∣ℓt∣|\ell_t|, recoverability still rises from 56% at low wtw_t to 75% at high wtw_t, while it stays flat under the position-only weight BtB_t (Figure 2c). At the fixed shape p=1p=1, raising β\beta from 0 adds the history-dependent hazard and raises the AIME25 gain from 1.20 points to as much as 4.12 (Figure 4b), while the normalized credit horizon falls below 0.50 (Figure 4c). Because wt≤Btw_t\le B_t (Proposition 2), the hazard can only shrink each token's update relative to the position-only weight, so this gain does not come from larger updates.

W4. Hyperparameter selection. The hyperparameters (β,p)(\beta,p) were selected by grid search on the evaluation benchmarks. Figure 4b reports the eight settings tested on AIME25 and AIME26.

AuthorsOpenReview ↗

Response to Reviewer pFgY (2/2) Q1 to Q4

Q1. Larger weights and calibration. The paper tests larger weights against teacher recoverability and downstream accuracy rather than the benefit of each update. At the same normalized position and gradient proxy ∣ℓt∣|\ell_t|, prefixes with larger wtw_t are more often recoverable (Figure 2c), and the resulting credit improves distillation (Section 5). The observable event is the success of a teacher continuation from a prefix. The weights are survival probabilities under the specified model rather than forecasts of this event, so Figure 2 checks whether they order prefixes by recoverability rather than whether they are calibrated to it (G3 of the General Response).

Q2. Optimization target and a fixed horizon. The target is the truncated objective in Equation 5, and Proposition 1 converts it exactly into the survival-weighted objective (Section 3). Training uses the standard clipped policy-gradient surrogate in Equation 3, as vanilla OPD and RL fine-tuning do, and only rescales each advantage −ℓt-\ell_t by the stop-gradient weight wtw_t (Equation 6). Appendix B.4 shows that this surrogate marginalizes the stochastic truncation for each completed rollout. Because the weights are computed after each rollout completes, the depth prior S0(t)=1−(t/L)pS_0(t)=1-(t/L)^p can use the realized length LL and is therefore trajectory-adaptive. Rollout lengths differ widely across prompts, so any fixed horizon would be too short for some rollouts and too long for others. For the requested comparison, we replace LL in S0S_0 with a horizon HH of 5,000 or 10,000 tokens fixed before generation and keep β=3\beta=3 and p=2p=2. The table reports the mean@32 accuracy of the 4B student under the 30B teacher, with gains over vanilla OPD in parentheses.

Horizon in S0S_0AIME25AIME26
Fixed HH = 5,00048.25 (+2.10+2.10)52.78 (−1.28-1.28)
Fixed HH = 10,00048.89 (+2.74+2.74)53.21 (−0.85-0.85)
Realized LL (Cox-OPD)50.73 (+4.58+4.58)54.79 (+0.73+0.73)

Q3. Repeated evaluation. We re-evaluate the vanilla OPD and Cox-OPD checkpoints of the 4B and 1.7B students under the 30B teacher with three evaluation seeds (G4 of the General Response). In the submission, the average gain over OPD is positive in all six separately trained single-teacher pairs.

Q4. Benchmarks and the default hyperparameters. Yes, the default comes from the grid search on the evaluation benchmarks described under W4, and we did not carefully tune (β,p)(\beta,p). All eight settings in Figure 4b improve over vanilla OPD on both AIME25 and AIME26, and several of them give gains comparable to the default.

We hope these answers address the reviewer's questions, and we would be glad to discuss any of them further.

Thank the authors for their responses. The explaination of survival view and latent lifetime is helpful. My remaining concern is still the value of the survival view. The procedure is named Cox-OPD, so I expected to see a deeper understanding of OPD initially. But it seems to have no connection with Cox models. The survival interpretation provides a general form of credit assignment, but not clear whether it leads to a deeper understanding of OPD. In addition, a minor and picky comment: although a lecture note used the survival function definition as in this paper, it is not the stanard or commonly used way to define it. As the authors mentioned, it doesn't change the formulation or estimations, just a notation. Then I would still recommend the authors to use more standard definitions in the future, to avoid confusions and simplify the understanding.

We thank the reviewer for the follow-up, and we are glad that the explanation of the survival view and the latent lifetime was helpful.

What the survival view adds to the understanding of OPD. Its claim is that teacher supervision has a lifetime along each rollout, and the student should learn from a teacher only while that matching is alive. The lifetime is easiest to see in speculative decoding, another trajectory-matching problem. There a draft prefix passes target verification with the product of its per-token acceptance probabilities. DSpark [1] names this the prefix survival probability and schedules verification by it, and D-PACE [2] derives its position weights from the same products. For all their added machinery, both designs are built on this probability, so trajectory matching already uses it in practice. Matching stops at the first rejection, so the lifetime is observed, and simply adding our Cox survival weights to DFlash raises its mean acceptance length on five benchmarks (Figure 6).

OPD is the same matching with no one to stop it. The teacher would reject the sampled token with probability δt\delta_t (Section 4), so the student's prefix survives with probability ∏k<t(1−δk)\prod_{k<t}(1-\delta_k). Vanilla OPD ignores this and distills every position at full strength, the zero-hazard case. Figure 1 shows the cost. The recoverability ratio near the end of failing rollouts is 0.49, against 1.07 on successful ones, while both groups receive nearly the same share of credit.

The same idea helps where position-based reweighting gives no reason to expect it. With several domain teachers, the supervision most likely to interfere across domains is discounted without any explicit handling of gradient conflict. In Figure 5, weighting only the code rollouts raises the math score (AIME24) at nearly every checkpoint, and weighting only the math rollouts generally raises the code score (MBPP+). In agentic tool use, an early action changes the state of every later step, so a long rollout can run far past the point where the teacher's guidance applies, and Cox-OPD improves on vanilla OPD on τ²-bench (G4 of the General Response). The view thus adds more than a general form of credit assignment. It says when a teacher should stop being followed.

The connection with Cox models. Cox-OPD is named after its hazard, the proportional-hazards form in Equation 10, a baseline over prefix depth times the relative risk eβδte^{\beta\delta_t}, with δt\delta_t as a time-varying covariate. LLM rollouts are long, so teacher supervision can easily fail along the way, and this form suits that setting. It separates a depth prior from trajectory-specific risk, so two hyperparameters with distinct roles suffice. Its log-linear link bounds the influence of any single token on all later weights by βeβ\beta e^\beta at any rollout length (Proposition 2), whereas under the product-limit weight ∏k<t(1−δk)\prod_{k<t}(1-\delta_k) one near-certain rejection erases the credit of the whole suffix. In Figure 4a, the product limit lowers AIME25 accuracy by 2.40 points, while Cox-OPD raises it by 4.58.

We will adopt the standard definition S(t∣y)=Pr⁡(τ>t∣y)S(t\mid y)=\Pr(\tau>t\mid y) in the revision, with wt=S(t−1∣y)w_t=S(t-1\mid y), and we thank the reviewer for the suggestion. In the future, we will explore more settings of LLM trajectory matching and develop theoretical guarantees that match how these models are deployed in practice. We hope the survival view encourages more statistical study of LLMs in the agent era.

[1] X. Cheng et al. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv:2607.05147, 2026.

[2] T. Wu et al. D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Drafting. arXiv:2605.18810, 2026.

Reviewer_ekWcOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies on-policy distillation for autoregressive models and observes that teacher matching may not remain equally valid along a student-generated rollout: recoverability can decline on failing trajectories even when a substantial suffix remains. The authors therefore reformulate on-policy distillation as a latent survival problem, where per-step hazards define a teacher-matching lifetime and each token’s distillation loss is weighted by its modeled survival probability, with zero hazard recovering vanilla on-policy distillation. They instantiate this framework as Cox-OPD, which combines a depth-dependent baseline hazard with a local student–teacher deviation score, and provide truncation, variance-reduction, and sensitivity analyses. Experiments on math and code generation show consistent average gains over on-policy distillation.

Critical strengths
  1. The paper introduces a well-motivated view of on-policy trajectory distillation. Modeling teacher matching as having a latent lifetime, rather than treating all visited prefixes as equally valid, is conceptually clean and directly motivated by the recoverability diagnostic in Figure 1.

  2. The survival formulation is theoretically coherent. Proposition 1 exactly converts the stochastically truncated KL objective into a survival-weighted objective, and the Rao–Blackwellization argument justifies deterministic survival weights as a variance-reduced implementation. Zero hazard recovering vanilla on-policy distillation makes the framework backward-compatible and easy to interpret.

  3. The empirical evaluation is reasonably broad for an on-policy distillation paper. It covers multiple student scales, single- and multi-teacher settings, math and code benchmarks, hazard-design ablations, parameter ablations, credit-assignment analyses, training dynamics, and speculative decoding.

Critical weaknesses
  1. The latent lifetime is not directly identified or calibrated. Recoverability is used as an empirical diagnostic, but the training-time survival weights come from a Cox hazard parameterization and are not supervised by recoverability or any auxiliary signal. The connection between the modeled lifetime and the actual point at which teacher matching becomes unhelpful remains largely qualitative. Ablations on some key parameters are not provided.

  2. The theoretical results characterize properties of the survival weights rather than downstream distillation performance. Proposition 2 bounds the sensitivity of log-weights to local deviation, but it does not quantitatively link this bound to recoverability, optimization error, or generalization. Thus, the theoretical analysis supports the design but does not fully explain the empirical gains.

  3. The empirical scope is limited in model families and task types. Main experiments are concentrated on Qwen3 models and on math/code domains; other model families, dialogue, multilingual tasks, and tool-use settings are not evaluated. Multi-teacher results show some fluctuation across benchmarks, and variance across seeds is not reported, which makes the statistical strength of the improvements unclear.

Questions for authors

Please refer to the weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer eKWC

We thank the reviewer for the constructive review and for finding the survival view well motivated and theoretically coherent. Below, we address each weakness.

W1. Identification and calibration. As discussed in G2 of the General Response, the paper specifies the latent lifetime through the hazard rather than identifying or calibrating it. Teacher recoverability serves as an empirical diagnostic and does not supervise the weights, since using it during training would be costly at scale (G3 of the General Response).

The design does not aim to locate the actual point at which teacher matching becomes unhelpful. What it claims, that the weights track recoverability beyond position, is tested quantitatively in Figure 2c. Within groups of normalized position and the gradient proxy ∣ℓt∣|\ell_t|, recoverability rises from 56% at low wtw_t to 75% at high wtw_t and stays flat under the position-only weight BtB_t. The weights are survival probabilities under the specified hazard rather than estimates of that point, so Figure 2 checks their ordering rather than their calibration (G3 of the General Response).

W1. Ablations. The method has two hyperparameters, β\beta and pp, and Figure 4b varies both. It sweeps β\beta from 0 to 3 at p=1p=1 and pp from 1 to 3 at β=3\beta=3, and adds β=4\beta=4 at p=2p=2 and p=3p=3. All eight settings improve over vanilla OPD on AIME25 and AIME26. Figure 4a replaces the hazard with four alternative designs, and Section 5.3 applies the Cox parameterization to math only, code only, both domains, or neither.

W2. What the theory explains. The theory serves the understanding of OPD and the design of credit (G1 of the General Response), while the link to recoverability and the downstream gains are evaluated empirically (G3 of the General Response). Proposition 1 and the hazard representation are exact identities, while Proposition 2 characterizes the Cox choice, bounding the cumulative sensitivity of subsequent log-weights to any single deviation by βeβ\beta e^\beta. The raw product-limit rule lacks the same uniform guarantee, and Section 5.3 reports that its median sensitivity exceeds this bound. The same rule lowers AIME25 accuracy by 2.40 points, while Cox-OPD raises it by 4.58 (Figure 4a), consistent with the design motivation. A quantitative link from credit assignment to optimization error or generalization remains open for on-policy distillation in general. We see this work as a first step toward a statistical treatment of trajectory matching with the tools of survival analysis.

W3. Scope. We add three settings that cover other model families, dialogue, multilingual tasks, and tool use, and Cox-OPD improves the overall score over vanilla OPD in all three (G4 of the General Response). τ²-bench tests tool use in multi-turn customer-service dialogues with a Qwen3-4B student, Multi-IF tests multilingual multi-turn instruction following with an R1-Distill-Qwen-1.5B student, and Open-MOPD tests multi-teacher distillation across math, code, and instruction following with a SmolLM3-3B student.

W3. Multi-teacher fluctuation. In the multi-teacher setting, Cox-OPD improves both domain averages at all three student scales and is lower in only 1 of the 21 benchmark entries in Table 3, by 0.21 points. Figure 5 applies the Cox parameterization to math only, code only, or both domains. Each variant generally stays above vanilla OPD across the training checkpoints of the 4B student, and both domains together perform best on AIME24 and MBPP+ (all seven benchmarks in Appendix C.1).

W3. Seeds. We re-evaluate the vanilla OPD and Cox-OPD checkpoints of the 4B and 1.7B students under the 30B teacher with three evaluation seeds (G4 of the General Response).

We hope these responses address the reviewer's concerns, and we are happy to elaborate on any of them.

Thank the authors for the detailed rebuttal. As I am not an expert in this field, I will refer to the other reviewers’ comments when determining my final score.

Thank you for your time and consideration. We appreciate your engagement with our work and would be happy to clarify any further questions that arise during the discussion.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and are encouraged by the strengths they note.

  1. Reviewer eKWC finds the survival view well motivated and theoretically coherent, and Reviewers pFgY and 7YBR consider Cox-OPD a simple and easily checked modification of OPD.
  2. Reviewer eKWC notes that the recoverability analysis directly motivates the latent lifetime, and Reviewers pFgY and 7YBR find that it shows information in the weights beyond token position. Reviewer 7YBR regards it as the most informative measurement in the paper.
  3. Reviewer pFgY finds the gains across all six single-teacher configurations encouraging, and Reviewer eKWC considers the evaluation reasonably broad.

The reviews also raise three common questions about what role the survival view plays, what its latent lifetime means, and how the credit it yields is evaluated. G1 to G3 answer them in turn, and G4 follows in a separate comment with additional experiments in broader settings.

G1. The role of the survival view (7YBR W3, eKWC W2). The survival view mainly provides a deeper understanding of OPD and a general form of credit assignment, with consistent gains from Cox-OPD (G3). Vanilla OPD assigns unit weight to every token, so it is the zero-hazard case of this view and "implicitly treats teacher matching as surviving the full rollout" (Section 2). Proposition 1 turns any lifetime model into survival-weighted credit. Conversely, every positive non-increasing credit sequence with w1=1w_1=1 has a hazard representation, so choosing a credit assignment rule is equivalent to choosing a hazard. Both results are exact identities for this whole class of credit assignment rules, and neither requires observing the lifetime. Uniform OPD, positional decay, the product-limit weight, and Cox-OPD are therefore instances of one general form and can be compared directly.

G2. What the latent lifetime means (7YBR Q1 and Q5, pFgY W1, eKWC W1). The lifetime models whether the teacher's supervision can still be trusted at full strength along the rollout, so what fails is the supervision rather than the trajectory. Because the growing divergence between prefix distributions "does not identify a failure event along an individual trajectory" (Section 3), the paper treats the lifetime as latent and specifies it through the hazard. The lifetime gives the weights their meaning, since Proposition 1 shows that weighting each token by its survival probability wtw_t is equivalent in expectation to distilling only while matching is alive. The specification is thus a credit design rather than a hypothesis about when continuations fail, and the method does not require a measurement of τ\tau. Cox-OPD is one natural instance of such a design, and Proposition 2 bounds the cumulative sensitivity of its subsequent log-weights to any single deviation by βeβ\beta e^\beta. However, as Section 3 notes, "Whether this lifetime reflects teacher relevance depends on the hazard design." For the Cox hazard, this is evaluated empirically (G3).

G3. How the credit is evaluated (7YBR W5, Q2, and Q4, pFgY Q1, eKWC W1 and W2). Teacher recoverability serves as a diagnostic rather than as a measurement of τ\tau, and the credit is evaluated mainly by downstream accuracy against other credit assignment rules. Measured separately from training, recoverability motivates the survival view in Figure 1, and Figure 2 checks whether the fixed weights track it. Within groups defined by normalized position and the gradient proxy ∣ℓt∣|\ell_t|, recoverability rises from 56% at low wtw_t to 75% at high wtw_t, while it stays flat under the position-only weight BtB_t (Figure 2c). The weights are survival probabilities rather than predictions of recoverability, so Figure 2 checks whether they order prefixes by recoverability, not whether they match its level. Using recoverability during training would require teacher continuations and outcome checks at sampled prefixes of each rollout, a cost that would make large-scale training difficult. Recoverability-guided training is left to future work, while our framework needs neither outcome labels nor additional teacher queries (Appendix B.3). In the downstream comparison of Figures 4a and 4b, Cox-OPD gives the largest AIME25 gain among the tested variants, +4.58+4.58 points against +1.20+1.20 for linear positional decay and −2.40-2.40 for the direct product limit. It also raises average performance in all six single-teacher pairs, by 1.43 points on average (Section 5).

AuthorsOpenReview ↗

General Response (additional experiments)

G4. Additional experiments on more model families and domains, and repeated evaluation (7YBR Q6, pFgY Q3, eKWC W3). The main OPD experiments use Qwen3 models on math and code. We compare Cox-OPD with vanilla OPD in three further settings that add agentic tool use and instruction following as domains, and R1-Distill-Qwen and SmolLM3 as model families. Cox-OPD improves the overall score over vanilla OPD in all three settings. Recovery, the share of the student-teacher gap in the overall score that a method closes, rises from 88.5% to 93.7% on τ²-bench, from 99.5% to 99.8% on Multi-IF, and from 90.1% to 99.4% on Open-MOPD. We also evaluate two single-teacher configurations of the main experiments with three different seeds.

Agentic tool use on τ²-bench (Qwen3). The tasks are multi-turn customer-service dialogues that require tool calls, with gpt-4o-mini simulating the user. Scores are pass@1 over 100 tasks with four trajectories each. Cox-OPD scores higher than vanilla OPD on Airline, Retail, and Telecom.

MethodAirlineRetailTelecomAvg.Recovery (%)
Teacher (Qwen3-4B-Thinking-2507)47.548.134.443.3100.0
Student (Qwen3-4B)15.017.510.614.40.0
Vanilla OPD45.045.629.440.088.5
Cox-OPD46.246.831.541.593.7

Multilingual multi-turn instruction following on Multi-IF (R1-Distill-Qwen). Score averages four pass rates over eight languages and three turns, Turn 3 is the score on the last turn, and Gain is over the student. Vanilla OPD already closes almost the whole gap, and Cox-OPD comes within 0.02 of the teacher in both Score and Turn 3.

MethodScoreTurn 3GainRecovery
Teacher (UltraData-IF-1.5B)41.2129.31—100%
Student (R1-Distill-Qwen-1.5B)30.9722.31—0%
Vanilla OPD41.1628.97+10.1999.5%
Cox-OPD41.1929.30+10.2299.8%

Multi-teacher distillation across math, code, and instruction following (SmolLM3). In the Open-MOPD setting, the SmolLM3-3B-MixSFT student learns from three RL-trained teachers, one per domain, and each teacher is evaluated on its own domain. Math is avg@16 on AIME24 and AIME25, code is avg@4 on LiveCodeBench v5 and v6, instruction following is a single run on IFEval and IFBench, and Total averages the six benchmarks. Cox-OPD scores higher than vanilla OPD on five of the six benchmarks and in Total.

MethodMathCodeIFTotalRecovery
AIME24AIME25LCBv5LCBv6IFEvalIFBench
Teacher (RL-Math / Code / IF)22.9227.2923.2024.5775.7922.6732.74100%
Student (MixSFT)15.6320.2615.9919.2066.9116.0025.670%
Vanilla OPD23.3327.5022.7523.2971.7223.6732.0490.1%
Cox-OPD23.5528.1023.4524.3173.2423.5432.7099.4%

Variance across evaluation seeds on the main math and code benchmarks (Qwen3). We evaluate the vanilla OPD and Cox-OPD checkpoints of the Qwen3-4B and Qwen3-1.7B students distilled from Qwen3-30B-A3B-Instruct-2507 with three different seeds.

StudentMethodMathCode
AIME24AIME25AIME26HMMT25HE+MBPP+
Qwen3-4BVanilla OPD55.91 ± 1.2046.72 ± 1.1253.71 ± 0.8532.51 ± 0.8079.78 ± 1.9074.27 ± 0.85
Cox-OPD57.45 ± 0.9350.41 ± 1.2354.22 ± 0.9434.74 ± 0.4380.85 ± 1.7874.88 ± 0.90
Qwen3-1.7BVanilla OPD35.47 ± 2.0528.79 ± 1.3225.72 ± 1.1214.48 ± 0.7369.81 ± 1.9358.69 ± 0.84
Cox-OPD36.59 ± 1.4029.26 ± 1.1327.65 ± 0.9716.62 ± 0.6069.94 ± 1.8959.90 ± 0.67

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.