acceptodds
Under review as a conference paper at ICLR 2027

Not All Tokens Should Update the Same Parameters: Prefix-Conditioned Gradient Routing for On-Policy Distillation

Abstract

On-policy distillation (OPD) trains a student on its own rollouts with token-level feedback from a stronger teacher. Because this feedback is issued on student-generated prefixes, it can also push the student away from behavior it already performs correctly, causing regressions on problems it previously solved. Existing remedies reweight tokens, but a scalar weight shrinks every parameter component of a token gradient together, so suppressing the part of an update that conflicts with previously successful behavior also discards the part that agrees with it. We propose Prefix-Conditioned Gradient Routing (PCGR), which makes this choice per parameter block. Using the gradient of the student's own verified past successes as a reference, PCGR lets the accumulated teacher–student discrepancy of the prefix decide how much reference conflict an update may keep, and blockwise reference geometry decide where the rest is removed. The resulting convex problem has a closed-form gate rule that, at equal retained reference conflict, never distorts the OPD gradient more than the corresponding group-level scalar gate. In standard OPD, 71.2% of token gradients mix reference-compatible and reference-conflicting blocks. Across four verifiable task families with Qwen3-1.7B and Qwen3-4B students, PCGR improves the macro score over OPD by 2.5 and 2.0 points and nearly halves the regression rate on previously solved problems (13.9% to 7.7%) without lowering the rate of new solutions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, large language models, LLM post-training, gradient routing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ComAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether all tokens in on-policy distillation should update the same parameters through the same gradient pathway. The authors argue that a student rollout may contain verified-correct prefixes, compatible but uncertain tokens, and tokens that conflict with a reference behavior. Applying the same update rule to every token may repeatedly modify already-correct behavior while failing to concentrate sufficient learning signal on tokens that require correction. The paper proposes Prefix-Conditioned Gradient Routing, which uses prefix states and reference signals to route different token-level updates through different parameter blocks and to control the update budget assigned to conflicting tokens. The method uses a token-local reverse-KL objective and combines rollout scoring, prefix-aware budgeting, reference-guided block routing, and token-level feedback. The central claim is that on-policy distillation should not apply homogeneous updates to all tokens, but should instead route gradients according to the prefix-conditioned status of each token.

Critical strengths
  1. The paper presents a valuable training perspective: tokens in an on-policy distillation rollout do not necessarily have the same learning value or error status, and applying a homogeneous update rule may damage correct behavior. Incorporating prefix state into credit assignment and gradient routing is a meaningful idea.

  2. The method does not simply discard an entire rollout containing an error. Instead, it attempts to preserve updates that are compatible with the reference behavior while limiting conflicting updates. This fine-grained treatment may make better use of on-policy data than whole-rollout filtering.

Critical weaknesses
  1. The key decision mechanism behind prefix-conditioned routing is not specified and validated in sufficient detail. The paper should define the criteria for correct, compatible, and conflicting states and explain how routing behaves when the reference signal is wrong. Without this information, it is difficult to determine whether the method improves credit assignment or mainly introduces another heuristic filtering mechanism.

  2. The source of the improvements is not adequately disentangled. Prefix verification, reference-gradient routing, conflicting-update budgets, and parameter-block routing are all introduced together. More complete ablations are needed, including settings without prefix conditioning, token-level weighting only, conflicting-token restriction only, random routing, fixed block routing, and whole-rollout filtering.

Questions for authors
  1. How is the prefix-conditioned reference state defined and computed? What criteria determine whether a prefix is verified as successful, compatible, or conflicting?

  2. How is compatibility between the reference gradient and the student gradient determined? Is it based on cosine similarity, directional projection, token-level loss, final-answer agreement, or another criterion?

  3. How does the method handle errors from the reference model or verifier? Do the authors have robustness experiments with noisy verifiers or disagreement between reference signals?

  4. How is the conflicting-update budget selected? Does it need to vary with model size, task difficulty, rollout length, or training stage?

  5. What is the additional computational cost compared with token weighting, loss masking, advantage clipping, whole-rollout filtering, and standard reverse-KL distillation? Does the performance gain remain under matched GPU time or teacher-call budgets?

  6. Do the routed parameter blocks exhibit stable functional specialization, or are the observed effects mainly dependent on initialization and training stage? Do similar routing patterns appear across different model architectures?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ComA (1/4): how the method decides

Thank you for the detailed review. Your summary describes PCGR as identifying verified-correct prefixes and routing each token according to the status of its prefix. Our wording invites that reading, but the method works differently, and W1, Q1, and Q2 ask about exactly this, so we start there. Parts 2–4 cover the ablations (W2), reference errors and the budget (W1, Q3, Q4), and cost and routing patterns (Q5, Q6).

The decision criteria (W1, Q1, Q2)

PCGR never judges whether a prefix or a token is correct, and it has no reference model. It combines three signals, each at a different level: complete past responses that passed the verifier define what to protect, the prefix sets how cautious to be, and each parameter block of the update is then checked against the protected responses, as strictly as the prefix calls for.

Verification applies only to complete past responses. Each task's outcome verifier (answer equivalence for mathematics, passing all tests for code, and so on) decides which earlier student responses enter a FIFO buffer of C=256C=256 responses, one per prompt. At every step we sample up to M=8M=8 of them, skipping prompts in the current batch, and compute the gradient rr of their NLL (Eq. 3). The reference is this gradient.

Prefix discrepancy sets how cautious to be. The only prefix-level quantity is ui,t=Di,t/(Di,t+τd)u_{i,t}=D_{i,t}/(D_{i,t}+\tau_d), where Di,t=∑s<t∣Ai,s∣D_{i,t}=\sum_{s < t}\lvert A_{i,s}\rvert accumulates the teacher–student log-ratio along the prefix (Eq. 2). We use it twice: to split tokens into K=2K=2 groups, and to give each group a conflict budget βj=1−ρuˉj\beta_j=1-\rho\bar u_j. It measures how far a prefix has drifted from the teacher: a risk indicator, not an estimate of correctness (Sec. 3.1).

Compatibility is a property of a gradient block. For group jj and parameter block bb we compute the preconditioned inner product aj,b=rb⊤PbGj,ba_{j,b}=r_b^\top P_b G_{j,b} (Eq. 5). The block update −ηPbGj,b-\eta P_b G_{j,b} changes the reference NLL by −ηaj,b-\eta a_{j,b} to first order. A block with aj,b>0a_{j,b} > 0 therefore lowers the reference loss and is compatible. A block with aj,b<0a_{j,b} < 0 raises it, and Hj,b=max⁡(0,−aj,b)H_{j,b}=\max(0,-a_{j,b}) is its conflict. Among the options in Q2 this is closest to a directional projection: an unnormalized inner product in the optimizer metric, so its magnitude matters as well as its sign. It is not a threshold on token-level losses, and final answers matter only through the verifier that fills the buffer.

Given these quantities, the gates stay as close as possible to the unrouted gradient while keeping at most a fraction βj\beta_j of the group's conflict (Eq. 7). The solution is closed-form (Eq. 8):

mj,b⋆=min⁡(1,max⁡(0,1−λjHj,bEj,b)),m^\star_{j,b}=\min\left(1, \max\left(0, 1-\lambda_j\frac{H_{j,b}}{E_{j,b}}\right)\right),

where Ej,b=Gj,b⊤PbGj,bE_{j,b}=G_{j,b}^\top P_b G_{j,b} is the block's OPD energy and λj≥0\lambda_j\ge0 is set so that the budget is met. Compatible blocks have Hj,b=0H_{j,b}=0 and keep a gate of one, and conflicting blocks are attenuated in proportion to their conflict per unit of energy.

In short, prefix discrepancy decides how much reference conflict an update may keep, and blockwise reference geometry decides where the rest is removed. There is no token or rollout filtering: the token-level loss is unchanged, each group–block gradient is multiplied by a scalar in [0,1][0,1], and no reference-gradient term is added to the update. We will make this explicit at the start of Section 3.

Credit assignment or heuristic filter? The two halves of the method have different status. The allocation across blocks is the solution of an optimization problem: for a given reference and budget it is the unique minimizer of a convex projection (Lemma 1), and at equal retained conflict it is provably no more distorting than the best group-level scalar gate (Prop. 1). Its premise is also measured directly. Blockwise routing can improve on scalar gating only if the conflict per unit of energy, Hb/EbH_b/E_b, differs across blocks, which is what the coefficient κ\kappa captures: at κ=1\kappa=1 the two coincide. In standard OPD the median κ\kappa is 0.28–0.37 for single tokens and 0.41–0.53 for per-response groups (Table 1), and 0.56 for the full-batch groups used in training (Sec. 5.1). Grouping by discrepancy and the budget rule βj=1−ρuˉj\beta_j=1-\rho\bar u_j, by contrast, are design choices, and each is evaluated with its own control (Table 8a–d, discussed in part 2).

AuthorsOpenReview ↗

Response to Reviewer ComA (2/4): where the improvement comes from

Separating the components (W2)

Most of the settings you list were in the submission, but spread over Table 2, Table 8, and Appendix D, so the table below puts them next to your list. Whole-rollout filtering was not among them, so we ran it. Conflict in PCGR is defined per parameter block, not per token, so the control that corresponds to "conflicting-token restriction" acts on blocks: a new sign-only gate, the simplest rule that attenuates only conflicting blocks, with blockwise projection as a related comparison. For fixed block routing, the closest controls are reference-free importance masks: CPI-FT, whose mask is static, and EPI, whose mask is refreshed every 50 steps. Results are given as Δ\Delta, the change in the mathematics-and-instruction average relative to full PCGR (rows a–h of Table 8), or, where marked †, as four-family macro scores to be compared with PCGR's 60.45 (Table 2).

Your requestControlWhat it isolatesResult
No prefix conditioningK=1K=1 with a constant budget (Table 8b)all prefix information removed, reference-aware block routing kept−1.35-1.35
Constant budget, discrepancy grouping kept (Table 8a)the adaptive budget alone−0.75-0.75
Position-based score / shuffled discrepancy (Table 8c, 8d)whether discrepancy says more than position−1.08-1.08 / −1.20-1.20
Token-level weighting onlyIW-OPD (Table 2)per-token scalar weights, no reference58.58†
UGS: scalar gate with PCGR's budgets (Table 8e)where conflict is removed, at equal budgets−1.63-1.63; 58.86†
Random routingPCGR gates permuted across blocks within a group (Table 8f)gate location, with the gate distribution unchanged−2.05-2.05
Fixed block routingCPI-FT (static mask) / EPI (refreshed mask) (Table 2)reference-free importance masks58.38† / 58.47†
Conflicting-token restriction onlyBlockwise projection (Table 2)removing all conflict per block by projecting off rr, which adds reference directions59.24†
New: sign-only gate, mj,b=1m_{j,b}=1 if aj,b≥0a_{j,b}\ge0 and βj\beta_j otherwiseuniform attenuation of conflicting blocks instead of Eq. (8), at the same retained conflictbelow
Whole-rollout filteringNew: OPD loss on verifier-passed rollouts only, learning rate tuned separatelyfiltering rollouts instead of routing gradientsbelow
VerificationUnverified reference / pool frozen at initial successes (Table 8g, 8h)verification / online refresh−1.28-1.28 / −0.73-0.73

The two new controls on mathematics (two paired seeds; OPD and UGS are the three-seed values of Tables 2 and 7):

MATH-500AIMERR (%)GR (%)
OPD91.247.58.021.7
UGS (scalar gate)91.948.45.818.3
Whole-rollout filtering (new)91.845.66.817.8
Sign-only gate (new)92.349.45.722.0
PCGR92.750.45.122.4

Three conclusions follow.

Every component contributes, and block placement contributes most. Removing all prefix information costs 1.35 points (Table 8b), and using an unverified reference costs 1.28 (Table 8g). Replacing the block gates with a scalar gate at the same budgets costs 1.63 (Table 8e), and permuting them across blocks costs 2.05 (Table 8f). Fixing the budget to a constant, by comparison, costs 0.75 (Table 8a), so where conflict is removed matters more than how much.

The sign-only gate shows what the allocation rule adds. This control keeps every compatible block and scales every conflicting block by βj\beta_j, so it retains the same conflict as PCGR. Its only difference from the scalar gate is that compatible blocks are left intact, and with that change the gain rate rises from 18.3% to 22.0%, above OPD's 21.7%. So keeping compatible blocks already improves both rates over OPD, and the allocation rule adds to it: the sign-only gate still ends 0.4 points below PCGR on MATH-500 and 1.0 on AIME, with a higher regression rate. What separates the two is how the attenuation is distributed. The sign-only gate applies one factor to every conflicting block, whereas Eq. (8) cuts each block in proportion to its conflict per unit of energy, which changes the OPD gradient least for the same retained conflict.

Whole-rollout filtering mostly gives up gains. Training only on verifier-passed rollouts keeps 55% of them. The gain rate falls from 21.7% to 17.8%, while the regression rate drops only from 8.0% to 6.8%. Filtering throws away the teacher's feedback on failed rollouts, where corrections are needed, and keeps the successful ones, where the teacher can still push against a strategy that already works. Routing within each update avoids this trade, and PCGR improves both rates.

AuthorsOpenReview ↗

Response to Reviewer ComA (3/4): reference errors and the budget

Verifier and reference errors (W1, Q3)

The submission covered the two ends of this question. Dropping verification altogether costs 1.28 points, about half of PCGR's gain over OPD on that subset (Table 8g). At the other end, a verified reference is stable under resampling: an independent reference batch reproduces the sign of aj,ba_{j,b} for 86% of blocks, with a gate rank correlation of 0.79 (App. E). What was missing is the case in between, a reference that is mostly right. We have now run it, and we also make explicit a bound that the analysis already implies.

What a wrong reference can do. Whatever rr is, the scalar gate βj1\beta_j\mathbf 1 is feasible in Eq. (7), so the optimal gates satisfy

∥mj⋆⊙Gj−Gj∥P≤(1−βj)∥Gj∥P≤ρ∥Gj∥P.\lVert m^\star_j\odot G_j - G_j\rVert_P \le (1-\beta_j) \lVert G_j\rVert_P \le \rho \lVert G_j\rVert_P .

A wrong reference can therefore put the attenuation in the wrong blocks, but it cannot move a group gradient farther from its unrouted value than a fraction ρ\rho of its norm. Nor is any reference-gradient term added: PCGR only rescales the existing group–block gradients, whereas OPD+αR\alpha R and the projection baselines add reference directions to the update. The bound follows directly from Prop. 1. How much a wrong reference costs in training is what the next experiment measures.

Controlled contamination. We replaced a fraction ϵ\epsilon of the reference pool with student rollouts that the verifier rejected. Concretely, 25 and 64 of the 256 buffer slots are reserved for failed rollouts of the same task, length-matched and refreshed FIFO like the rest. Over training, the reference minibatches then contain 10% and 25% rejected responses on average. Initialization, prompt schedule, and seeds are shared with the clean runs. We look first at a single update and then at full training.

For a single update, we repeated the one-step diagnostics on frozen OPD snapshots, reported as ratios to the OPD update (budget-matched protocol of App. E; the ϵ=0\epsilon=0 and OPD rows are from Table 3):

ϵ\epsilonKL progressNLL change on clean held-out verified responses
00.910.52
0.100.910.55
0.250.900.64
OPD1.001.00

For full training, we ran mathematics with the contaminated pool throughout (two paired seeds, clean evaluation; OPD as in part 2):

MATH-500AIMERR (%)GR (%)
OPD91.247.58.021.7
PCGR, ϵ=0\epsilon=0 (same seeds)92.750.45.122.4
PCGR, ϵ=0.10\epsilon=0.1092.650.25.322.4
PCGR, ϵ=0.25\epsilon=0.2592.249.36.222.2

With 10% contamination, no metric moves by more than 0.2 points. With 25%, PCGR keeps about two thirds of its advantage over OPD on each of the four metrics. The routing statistics show the same gradual change: adding the rejected responses to a reference minibatch leaves the sign of aj,ba_{j,b} unchanged for 91% of blocks at ϵ=0.10\epsilon=0.10 and 87% at ϵ=0.25\epsilon=0.25. So PCGR tolerates a partly wrong reference, and its benefit declines gradually as the contamination grows.

How the budget is chosen (Q4)

The budget βj=1−ρuˉj\beta_j=1-\rho\bar u_j has one global hyperparameter and one per-task scale, both fixed before training: ρ=0.5\rho=0.5 for every task, and τd\tau_d is the median positive Di,tD_{i,t} over 128 initial-student rollouts of task dd (App. C). On the four factors you ask about:

  • Model size. The same ρ\rho is used at 1.7B and 4B (Table 9).
  • Task difficulty. Calibrating τd\tau_d per task normalizes the initial discrepancy scale, and the result is not sensitive to it: scaling τd\tau_d by 0.5× or 2× moves MATH-500 by at most 0.4 points (Table 4).
  • Rollout length. Replacing Di,tD_{i,t} with the length-normalized Di,t/(t−1)D_{i,t}/(t-1) gives the same result (+0.13, Table 8i).
  • Training stage. No schedule is needed, because the budget is recomputed from the current rollouts at every step. In our mathematics runs, the mean discrepancy of the high-discrepancy group falls from 0.74 in the first 10% of steps to 0.58 in the last 10%, so its budget β2\beta_2 loosens from 0.63 to 0.71, while β1\beta_1 moves from 0.86 to 0.90.

All three values of ρ\rho we tested (0.25, 0.5, 1) improve on OPD (Table 4, Fig. 3a), and ρ=1\rho=1 brings the mathematics regression rate down to 4.5% at the gain rate of OPD.

AuthorsOpenReview ↗

Response to Reviewer ComA (4/4): cost and routing patterns

Cost against simpler alternatives (Q5)

A PCGR step costs about 20% more than an OPD step, and PCGR stays ahead at matched GPU time and at a matched teacher budget. The measured costs for Qwen3-1.7B on 8×H100, averaged over steps and seeds, are as follows (Table 11):

MethodBackward sweepsTime / stepMemoryMacro at equal GPU-hours
OPD1185.9 s68.6 GB57.95
IW-OPD1188.7 s69.5 GB58.56
A-GEM-style3214.3 s72.4 GB58.42
Blockwise projection3218.6 s73.6 GB58.85
PCGR (K=2K=2)3223.4 s (1.20×)74.1 GB59.63

The token-space methods you mention (token weighting, loss masking, advantage clipping, whole-rollout filtering) all need a single backward sweep, as OPD does; IW-OPD, the token-weighting method we measured, runs at 1.02× the step time of OPD. PCGR needs K+1=3K+1=3 sweeps, one per discrepancy group and one for the reference, and it holds three parameter-sized buffers (rr, GjG_j, G~\tilde G) whatever KK is. Because rollout generation and teacher scoring dominate wall-clock time, this comes to 1.20× per step, at 4B as well as at 1.7B (App. J). The extra cost pays for itself: at OPD's GPU-hour budget PCGR is 1.68 macro points ahead, and it reaches OPD's final score with about 29% fewer GPU-hours (Fig. 6).

Teacher usage is essentially matched. Routing never re-scores a rollout, so in every method the teacher scores each of a run's 400 × 64 rollouts once. The only extra teacher use is the one-off calibration of τd\tau_d on 128 prompts, 0.5% of a run's 25,600 rollouts, and its cost is included in PCGR's budget (App. C). Since response lengths differ somewhat between methods, we also counted the response tokens scored by the teacher, summed over the four families for one seed: 412M for OPD, 409M for IW-OPD, and 403M for PCGR. At a common budget of 400M scored tokens the macro scores are 57.9, 58.5, and 60.3, so the comparison also holds at a matched budget of teacher-scored tokens.

Routing patterns (Q6)

We measured where the gates act and how stable that is, on mathematics (1.7B, 310 blocks):

  • Every conflicting block is attenuated to some degree (Eq. 8), but most only slightly. In a typical step, 27% of blocks in the high-discrepancy group and 14% in the low-discrepancy group have a gate below 0.9, and 9% of gates reach zero.
  • Attenuation falls mostly on the matrices that write into the residual stream. MLP down-projections and attention output projections in the upper third of the network hold 31% of the gradient energy but receive 58% of the attenuation (measured as ∑bEb(1−mb)2\sum_b E_b(1-m_b)^2), about 1.9 times their share. Embeddings, norms, and early-layer attention are almost never gated.
  • The pattern is partly stable. The rank correlation of per-block mean gates is 0.71 across seeds and 0.62 across the 0/25/50/100% snapshots, while individual gates vary from step to step (correlation 0.38), as they should if they respond to the current batch.

So the answer to your question is both: routing has a stable preference for certain blocks across seeds, and it also shifts with the training stage. These statistics show where routing acts, not what those blocks compute. All experiments use the Qwen3 family, so we have not yet tested whether other architectures show the same pattern. We will add these routing statistics to the appendix, and we plan to repeat them on a second model family.

We hope this clarifies how the method works and where its gains come from, and that you will take the new results into account in your final assessment. We are glad to follow up on anything that remains unclear.

Reviewer_FBPEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

the paper summarized the efforts in improving OPD training by selecting subsets of parameter updating. They demonstrated the protocol and rationals behind this implementation. The results showed modest improvement in training efficiency without hurting final outcomes. Systematic ablation analyses were carried out to support the importance of the complete strategy.

Critical strengths

The motivation is clear and convincingly presented. The authors also showed the implementation details with distinction well presented comparing to other approaches.

The results demonstrated improvement in efficiency. Ablation analysis also strongly supports the conclusion.

Critical weaknesses
  1. The overall improvement in efficiency is modest. Will this be a critical issue worthwhile for further investigation?
  2. Can this strategy scale with larger models?
Questions for authors
  1. The overall improvement in efficiency is modest. Will this be a critical issue worthwhile for further investigation?
  2. Can this strategy scale with larger models?
1 comment
AuthorsOpenReview ↗

Response to Reviewer FBPE

Thank you for the positive comments on the motivation and the ablations. Your summary describes the result as a modest gain in training efficiency, so we would like to explain what we see as the main result, and then answer the question about scale.

What the improvement is, and why it matters (W1, Q1)

PCGR addresses a different problem from efficiency: on-policy distillation makes the student lose things it could already do. Standard OPD loses 8.0–17.4% of the problems the initial student solved, depending on the task family, while simply re-sampling the initial student gives about 3% (Table 7). These losses do not show up in average scores, because new solutions mask them, which is why we measure them separately. OPD has become a standard component of LLM post-training, so a failure mode that removes existing competence while average scores still rise is worth addressing.

The idea behind the fix is simple. Existing remedies down-weight risky tokens, but a token weight scales the whole gradient of that token by one number. In standard OPD, 71.2% of token gradients push toward the student's verified past successes in some parameter blocks and against them in others, so shrinking the whole gradient to limit the damage also throws away the useful part. PCGR scales each parameter block separately: it damps the blocks that work against verified behavior and leaves the others unchanged.

Measured on regressions, the effect is substantial:

  • Of every 100 problems the initial student could solve, OPD loses about 14 and PCGR about 8 (family-average regression rate 13.9% → 7.7%), and the gain rate does not suffer (19.7% → 20.6%). Every baseline in Table 7 that lowers the family-average regression rate also lowers the gain rate (Fig. 3a, b).
  • The average improvement on initially unsolved problems nearly doubles, from 5.3 to 10.3 points (Table 7).
  • The macro score rises by 2.50 points over OPD (paired 95% interval [1.8, 3.2]), by 1.87 over the best token-weighting baseline, and by 1.21 over the best reference-aware baseline. PCGR closes 26% of the gap to the teacher, twice OPD's 13%.
  • The gain holds at 4B (+1.98), with mixed-task training (+2.40), with AdamW (+2.05), and on an extended suite (+2.9 to +3.5, Table 5).

Efficiency enters only as a check that the extra computation is worth it. A PCGR step costs about 20% more than an OPD step (Table 11), yet at OPD's GPU-hour budget PCGR is still 1.68 points ahead, and it reaches OPD's final score with about 29% fewer GPU-hours. We will make this emphasis clearer in the abstract and introduction.

Larger models (W2, Q2)

We tested two student sizes. At 4B, PCGR improves on OPD by 1.98 macro points, with gains on all four task families, and on the strongest baseline by 0.73 (Table 9). The 4B student starts much closer to the teacher, so there is less headroom, and relative to that headroom the gain is larger: PCGR closes 53% of the initial gap to the teacher against 24% for OPD, compared with 26% against 13% at 1.7B.

The per-step cost is the same 1.20× at both sizes (App. J). The overhead has a fixed structure: three backward passes instead of one (two over the rollout groups and one over at most eight reference responses), plus three parameter-sized buffers. The number of passes does not depend on model size or on the number of parameter blocks, so the extra time grows with the cost of a backward pass and the extra memory grows linearly with the model, as in ordinary training.

We hope this clarifies what the paper contributes, and we are happy to answer any further questions.

Reviewer_65o9OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Prefix-Conditioned Gradient Routing (PCGR) for improving on-policy distillation by reducing interference between learning from the teacher and preserving capabilities the student has already acquired.

The key idea is to group tokens according to the accumulated discrepancy between teacher and student prefixes, and use gradients computed from previously verified successful student responses as a reference for routing updates across parameter blocks. Rather than directly adding the reference gradient to the update, PCGR uses it to determine which parameter blocks should be updated and to what extent.

The authors formulate this routing procedure as a convex optimization problem and provide theoretical results showing that it can reduce gradient distortion while preserving progress on the reference objective under certain conditions. Experiments across mathematics, code generation, instruction following, and tool-calling tasks show that PCGR reduces regression relative to standard on-policy distillation while retaining comparable or better task improvements, albeit with additional computational overhead.

Critical strengths

Here are the key strength of the paper:

  1. The paper addresses an important limitation of on-policy distillation: updates that improve alignment with the teacher can inadvertently degrade capabilities the student has already acquired. PCGR provides a principled approach to mitigating this interference by using gradients from verified successful responses to guide parameter-level update routing.

  2. The proposed method is supported by a clear optimization formulation and theoretical analysis. In particular, the authors show that parameter-block-level routing can achieve lower gradient distortion than a group-level scalar gate under the stated constraints. This provides a useful justification for the proposed routing mechanism.

  3. The experimental evaluation is relatively comprehensive, covering multiple task families, model scales, and baseline methods. The inclusion of multiple random seeds, ablation studies, and robustness evaluations strengthens the empirical evidence.

  4. The results demonstrate a favorable trade-off between reducing regressions and retaining task improvements.

Critical weaknesses

Following are some minor weakness of the paper:

  1. The theoretical guarantees rely on specific assumptions, including RMS-based geometry and a fixed preconditioner. Their applicability to practical optimizers such as AdamW is less clear.

  2. The reference gradient depends on a buffer of verified successful responses, making PCGR reliant on task-specific outcome verifiers and the quality and diversity of the reference data.

  3. The additional backward passes introduce nontrivial computational overhead. While the equal-GPU-hour results are encouraging, the scalability of PCGR to larger models and more complex tasks remains unclear.

  4. The paper does not sufficiently characterize how often the theoretical descent condition holds in practice or how sensitive performance is to reference-buffer size and composition.

Questions for authors
  1. How often does the proposed routing satisfy the positive reference-alignment condition in Proposition 2 during actual training?
  2. How sensitive is PCGR to the size and diversity of the reference buffer?
  3. Can the dependence on task-specific outcome verifiers be reduced or eliminated?
  4. How does the computational overhead scale with model size and the number of parameter blocks?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 65o9 (1/2)

Thank you for the encouraging review and for four concrete questions. We answer each below, with new measurements where the submission had none.

Assumptions of the analysis, and how often the Prop. 2 condition holds (W1, W4, Q1)

The analysis needs the preconditioner to be fixed only within one step: the same PkP^k is used to compute aj,ba_{j,b} and Ej,bE_{j,b} and to apply the update. It does not need PP to stay fixed across training, and the RMS state does evolve between iterations (Sec. 3.6). You are right that the guarantee does not carry over exactly to AdamW, where momentum and adaptive normalization act after the gates (App. C). There our evidence is empirical: PCGR keeps 82% of its gain over OPD (+2.05 macro), and the realized reference effect of each step correlates at 0.80 with the routed prediction (Table 10).

As for how often the condition holds, Prop. 2 has two parts: an identity that always holds, r⊤P(G~−G)=∑j(1−βj)Hj≥0r^\top P(\tilde G-G)=\sum_j(1-\beta_j)H_j\ge0, and a descent statement that additionally requires Sk=∑j(Qj−βjHj)>0S_k=\sum_j(Q_j-\beta_jH_j) > 0 and a small enough step. We measured both how often Sk>0S_k > 0 holds and how often the reference loss actually decreases, in the main PCGR runs (three seeds, four families), split into thirds of training. Rows (a)–(e) are from the RMS runs, where the analysis is exact, and row (f) is from the AdamW runs.

QuantityEarlyMiddleLateAll
(a) Pr⁡[Sk>0]\Pr[S_k > 0], PCGR0.970.990.990.98
(b) Pr⁡[r⊤PG>0]\Pr[r^\top P G > 0], unrouted OPD gradient at the same parameters0.810.860.890.85
(c) median of r⊤P(G~−G)/∣r⊤PG∣r^\top P(\tilde G-G) / \lvert r^\top P G\rvert0.470.400.340.40
(d) fraction of steps with Rk(θk+1)<Rk(θk)R_k(\theta_{k+1}) < R_k(\theta_k) on the routing batch0.830.870.890.86
(e) NLL increase on held-out verified responses, routed step relative to unrouted step0.550.520.490.52
(f) AdamW: sign agreement between predicted and realized reference effect0.890.910.920.91

The condition holds in almost every step. Routed steps satisfy Sk>0S_k > 0 98% of the time, compared with 85% for the unrouted gradient at the same parameters, and at the median routing improves the first-order reference effect by 40% of the unrouted value (row c). The margin narrows late in training because discrepancy falls and the budget loosens. The reference loss on the routing batch actually goes down in 86% of steps (row d). The exceptions occur mostly early in training and on code, the family with the largest conflict share (48.0% in Table 1). Under AdamW, the predicted and realized reference effects agree in sign in 91% of steps (row f).

Rows (a)–(d) use the reference minibatch that routing itself sees. Row (e) is the out-of-sample check: from the same parameters, it compares the mean NLL change on independent held-out verified responses after a routed step and after an unrouted step. There the NLL rises slightly on average in both cases, and routing roughly halves the increase, in line with Table 3. Over a full run, the NLL of this held-out set rises by 0.071 nats/token under OPD and by 0.034 under PCGR.

Size and diversity of the reference buffer (W2, W4, Q2)

Two sizes are involved, and the submission varied only one of them. The per-step minibatch MM was swept in Table 4, where M∈{4,8,16}M\in\lbrace 4,8,16\rbrace spans 0.4 points on MATH-500. The buffer capacity was fixed at C=256C=256, so we have now varied it, together with the composition of the pool (mathematics, M=8M=8, two paired seeds):

MATH-500AIMERR (%)GR (%)
C=64C=6492.449.95.622.2
C=256C=256 (default)92.750.45.122.4
C=1024C=102492.850.74.922.6
C=256C=256, pool restricted to the three most frequent DeepMath topics92.349.45.822.1

PCGR is not very sensitive to either, and the composition of the pool matters somewhat more than its size. On size, the result is stable over a 16-fold range of capacities: going from 256 to 1024 changes no metric by more than 0.3 points, and going down to 64 by no more than 0.5.

A smaller pool also covers fewer topics (the C=64C=64 pool covers 54% of the topics seen at C=256C=256), so the last row varies composition separately: it keeps capacity and MM at their defaults and restricts the pool to three topics. This costs slightly more than shrinking the pool to 64 responses, and within that run the regression rate is 5.0% on problems from the three pool topics and 7.4% on the others. Regressions are thus fewest on the kinds of problems the pool contains, which is a reason to keep the pool diverse.

AuthorsOpenReview ↗

Response to Reviewer 65o9 (2/2)

Reducing the dependence on outcome verifiers (W2, Q3)

PCGR needs a source of trusted responses; that is its condition of use, and removing verification altogether costs 1.28 points (Table 8g). Beyond that, our experiments show what the reference does not need. It does not have to be fresh: a pool frozen at the initial student's successes costs 0.73 points (Table 8h). It does not have to be large: four responses per step are enough (Table 4), and a 64-response buffer costs at most 0.5 points on any metric (part 1). And it does not have to be perfectly clean. In a new experiment on mathematics, we replaced 10% or 25% of the buffer with responses that the verifier rejected (two paired seeds; OPD from Tables 2 and 7; the protocol is in our reply to Reviewer ComA, part 3):

MATH-500AIMERR (%)GR (%)
OPD91.247.58.021.7
PCGR, clean pool92.750.45.122.4
PCGR, 10% rejected92.650.25.322.4
PCGR, 25% rejected92.249.36.222.2

With 10% of the pool wrong, no metric moves by more than 0.2 points, and with 25% PCGR keeps about two thirds of its advantage over OPD.

There is also a structural limit on each update. For any reference, the scalar gate βj1\beta_j\mathbf 1 is feasible in Eq. (7), so the optimal gates satisfy ∥mj⋆⊙Gj−Gj∥P≤(1−βj)∥Gj∥P≤ρ∥Gj∥P\lVert m^\star_j\odot G_j-G_j\rVert_P\le(1-\beta_j)\lVert G_j\rVert_P\le\rho\lVert G_j\rVert_P. A poor reference can misplace attenuation, but it cannot move a group gradient by more than a fraction ρ\rho of its norm.

The dependence on a task-specific verifier can be reduced further, because the method only needs a set of trusted responses to define rr, however they are selected. Teacher scoring or a judge model could take the verifier's place. We have not evaluated such sources; all results here use outcome verifiers.

How the overhead scales (W3, Q4)

The overhead comes from backward passes, not from the number of blocks. With dθd_\theta parameters, BB blocks, and KK groups, a PCGR step adds K−1K-1 backward passes over the rollout batch and one over the M≤8M\le8 reference responses. Everything else is cheap: computing Ej,bE_{j,b} and aj,ba_{j,b} and applying the gates is O(Kdθ)O(Kd_\theta) element-wise work, and the dual solve takes O(KBnbis)O(KBn_{\text{bis}}) scalar operations, with nbis=20n_{\text{bis}}=20 and BB between 310 and 398. No backward pass is done per block. Memory is three parameter-sized buffers for any KK.

The measurements follow this accounting. Each additional group adds about 10% of an OPD step: Table 4 gives 1.10× at K=1K=1, 1.20× at K=2K=2, and 1.41× at K=4K=4. The ratio stays low because a step is dominated by decoding rollouts of up to 16,384 tokens one token at a time, whereas a sweep processes all tokens in parallel. At K=2K=2 the ratio is the same 1.20× at 1.7B and at 4B, and memory goes from 68.6 to 74.1 GB at 1.7B (Table 11, App. J). The gains carry over as well: +1.98 macro points at 4B (Table 9), and +2.9 to +3.5 points on the extended suite of algorithmic-reasoning and SQL tasks (Table 5).

Thank you again for the careful reading and the constructive questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.