- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether all tokens in on-policy distillation should update the same parameters through the same gradient pathway. The authors argue that a student rollout may contain verified-correct prefixes, compatible but uncertain tokens, and tokens that conflict with a reference behavior. Applying the same update rule to every token may repeatedly modify already-correct behavior while failing to concentrate sufficient learning signal on tokens that require correction. The paper proposes Prefix-Conditioned Gradient Routing, which uses prefix states and reference signals to route different token-level updates through different parameter blocks and to control the update budget assigned to conflicting tokens. The method uses a token-local reverse-KL objective and combines rollout scoring, prefix-aware budgeting, reference-guided block routing, and token-level feedback. The central claim is that on-policy distillation should not apply homogeneous updates to all tokens, but should instead route gradients according to the prefix-conditioned status of each token.
Critical strengths
-
The paper presents a valuable training perspective: tokens in an on-policy distillation rollout do not necessarily have the same learning value or error status, and applying a homogeneous update rule may damage correct behavior. Incorporating prefix state into credit assignment and gradient routing is a meaningful idea.
-
The method does not simply discard an entire rollout containing an error. Instead, it attempts to preserve updates that are compatible with the reference behavior while limiting conflicting updates. This fine-grained treatment may make better use of on-policy data than whole-rollout filtering.
Critical weaknesses
-
The key decision mechanism behind prefix-conditioned routing is not specified and validated in sufficient detail. The paper should define the criteria for correct, compatible, and conflicting states and explain how routing behaves when the reference signal is wrong. Without this information, it is difficult to determine whether the method improves credit assignment or mainly introduces another heuristic filtering mechanism.
-
The source of the improvements is not adequately disentangled. Prefix verification, reference-gradient routing, conflicting-update budgets, and parameter-block routing are all introduced together. More complete ablations are needed, including settings without prefix conditioning, token-level weighting only, conflicting-token restriction only, random routing, fixed block routing, and whole-rollout filtering.
Questions for authors
-
How is the prefix-conditioned reference state defined and computed? What criteria determine whether a prefix is verified as successful, compatible, or conflicting?
-
How is compatibility between the reference gradient and the student gradient determined? Is it based on cosine similarity, directional projection, token-level loss, final-answer agreement, or another criterion?
-
How does the method handle errors from the reference model or verifier? Do the authors have robustness experiments with noisy verifiers or disagreement between reference signals?
-
How is the conflicting-update budget selected? Does it need to vary with model size, task difficulty, rollout length, or training stage?
-
What is the additional computational cost compared with token weighting, loss masking, advantage clipping, whole-rollout filtering, and standard reverse-KL distillation? Does the performance gain remain under matched GPU time or teacher-call budgets?
-
Do the routed parameter blocks exhibit stable functional specialization, or are the observed effects mainly dependent on initialization and training stage? Do similar routing patterns appear across different model architectures?
4 comments
Response to Reviewer ComA (2/4): where the improvement comes from
Separating the components (W2)
Most of the settings you list were in the submission, but spread over Table 2, Table 8, and Appendix D, so the table below puts them next to your list. Whole-rollout filtering was not among them, so we ran it. Conflict in PCGR is defined per parameter block, not per token, so the control that corresponds to "conflicting-token restriction" acts on blocks: a new sign-only gate, the simplest rule that attenuates only conflicting blocks, with blockwise projection as a related comparison. For fixed block routing, the closest controls are reference-free importance masks: CPI-FT, whose mask is static, and EPI, whose mask is refreshed every 50 steps. Results are given as , the change in the mathematics-and-instruction average relative to full PCGR (rows a–h of Table 8), or, where marked †, as four-family macro scores to be compared with PCGR's 60.45 (Table 2).
| Your request | Control | What it isolates | Result |
|---|---|---|---|
| No prefix conditioning | with a constant budget (Table 8b) | all prefix information removed, reference-aware block routing kept | |
| Constant budget, discrepancy grouping kept (Table 8a) | the adaptive budget alone | ||
| Position-based score / shuffled discrepancy (Table 8c, 8d) | whether discrepancy says more than position | / | |
| Token-level weighting only | IW-OPD (Table 2) | per-token scalar weights, no reference | 58.58† |
| UGS: scalar gate with PCGR's budgets (Table 8e) | where conflict is removed, at equal budgets | ; 58.86† | |
| Random routing | PCGR gates permuted across blocks within a group (Table 8f) | gate location, with the gate distribution unchanged | |
| Fixed block routing | CPI-FT (static mask) / EPI (refreshed mask) (Table 2) | reference-free importance masks | 58.38† / 58.47† |
| Conflicting-token restriction only | Blockwise projection (Table 2) | removing all conflict per block by projecting off , which adds reference directions | 59.24† |
| New: sign-only gate, if and otherwise | uniform attenuation of conflicting blocks instead of Eq. (8), at the same retained conflict | below | |
| Whole-rollout filtering | New: OPD loss on verifier-passed rollouts only, learning rate tuned separately | filtering rollouts instead of routing gradients | below |
| Verification | Unverified reference / pool frozen at initial successes (Table 8g, 8h) | verification / online refresh | / |
The two new controls on mathematics (two paired seeds; OPD and UGS are the three-seed values of Tables 2 and 7):
| MATH-500 | AIME | RR (%) | GR (%) | |
|---|---|---|---|---|
| OPD | 91.2 | 47.5 | 8.0 | 21.7 |
| UGS (scalar gate) | 91.9 | 48.4 | 5.8 | 18.3 |
| Whole-rollout filtering (new) | 91.8 | 45.6 | 6.8 | 17.8 |
| Sign-only gate (new) | 92.3 | 49.4 | 5.7 | 22.0 |
| PCGR | 92.7 | 50.4 | 5.1 | 22.4 |
Three conclusions follow.
Every component contributes, and block placement contributes most. Removing all prefix information costs 1.35 points (Table 8b), and using an unverified reference costs 1.28 (Table 8g). Replacing the block gates with a scalar gate at the same budgets costs 1.63 (Table 8e), and permuting them across blocks costs 2.05 (Table 8f). Fixing the budget to a constant, by comparison, costs 0.75 (Table 8a), so where conflict is removed matters more than how much.
The sign-only gate shows what the allocation rule adds. This control keeps every compatible block and scales every conflicting block by , so it retains the same conflict as PCGR. Its only difference from the scalar gate is that compatible blocks are left intact, and with that change the gain rate rises from 18.3% to 22.0%, above OPD's 21.7%. So keeping compatible blocks already improves both rates over OPD, and the allocation rule adds to it: the sign-only gate still ends 0.4 points below PCGR on MATH-500 and 1.0 on AIME, with a higher regression rate. What separates the two is how the attenuation is distributed. The sign-only gate applies one factor to every conflicting block, whereas Eq. (8) cuts each block in proportion to its conflict per unit of energy, which changes the OPD gradient least for the same retained conflict.
Whole-rollout filtering mostly gives up gains. Training only on verifier-passed rollouts keeps 55% of them. The gain rate falls from 21.7% to 17.8%, while the regression rate drops only from 8.0% to 6.8%. Filtering throws away the teacher's feedback on failed rollouts, where corrections are needed, and keeps the successful ones, where the teacher can still push against a strategy that already works. Routing within each update avoids this trade, and PCGR improves both rates.
Response to Reviewer ComA (3/4): reference errors and the budget
Verifier and reference errors (W1, Q3)
The submission covered the two ends of this question. Dropping verification altogether costs 1.28 points, about half of PCGR's gain over OPD on that subset (Table 8g). At the other end, a verified reference is stable under resampling: an independent reference batch reproduces the sign of for 86% of blocks, with a gate rank correlation of 0.79 (App. E). What was missing is the case in between, a reference that is mostly right. We have now run it, and we also make explicit a bound that the analysis already implies.
What a wrong reference can do. Whatever is, the scalar gate is feasible in Eq. (7), so the optimal gates satisfy
A wrong reference can therefore put the attenuation in the wrong blocks, but it cannot move a group gradient farther from its unrouted value than a fraction of its norm. Nor is any reference-gradient term added: PCGR only rescales the existing group–block gradients, whereas OPD+ and the projection baselines add reference directions to the update. The bound follows directly from Prop. 1. How much a wrong reference costs in training is what the next experiment measures.
Controlled contamination. We replaced a fraction of the reference pool with student rollouts that the verifier rejected. Concretely, 25 and 64 of the 256 buffer slots are reserved for failed rollouts of the same task, length-matched and refreshed FIFO like the rest. Over training, the reference minibatches then contain 10% and 25% rejected responses on average. Initialization, prompt schedule, and seeds are shared with the clean runs. We look first at a single update and then at full training.
For a single update, we repeated the one-step diagnostics on frozen OPD snapshots, reported as ratios to the OPD update (budget-matched protocol of App. E; the and OPD rows are from Table 3):
| KL progress | NLL change on clean held-out verified responses | |
|---|---|---|
| 0 | 0.91 | 0.52 |
| 0.10 | 0.91 | 0.55 |
| 0.25 | 0.90 | 0.64 |
| OPD | 1.00 | 1.00 |
For full training, we ran mathematics with the contaminated pool throughout (two paired seeds, clean evaluation; OPD as in part 2):
| MATH-500 | AIME | RR (%) | GR (%) | |
|---|---|---|---|---|
| OPD | 91.2 | 47.5 | 8.0 | 21.7 |
| PCGR, (same seeds) | 92.7 | 50.4 | 5.1 | 22.4 |
| PCGR, | 92.6 | 50.2 | 5.3 | 22.4 |
| PCGR, | 92.2 | 49.3 | 6.2 | 22.2 |
With 10% contamination, no metric moves by more than 0.2 points. With 25%, PCGR keeps about two thirds of its advantage over OPD on each of the four metrics. The routing statistics show the same gradual change: adding the rejected responses to a reference minibatch leaves the sign of unchanged for 91% of blocks at and 87% at . So PCGR tolerates a partly wrong reference, and its benefit declines gradually as the contamination grows.
How the budget is chosen (Q4)
The budget has one global hyperparameter and one per-task scale, both fixed before training: for every task, and is the median positive over 128 initial-student rollouts of task (App. C). On the four factors you ask about:
- Model size. The same is used at 1.7B and 4B (Table 9).
- Task difficulty. Calibrating per task normalizes the initial discrepancy scale, and the result is not sensitive to it: scaling by 0.5× or 2× moves MATH-500 by at most 0.4 points (Table 4).
- Rollout length. Replacing with the length-normalized gives the same result (+0.13, Table 8i).
- Training stage. No schedule is needed, because the budget is recomputed from the current rollouts at every step. In our mathematics runs, the mean discrepancy of the high-discrepancy group falls from 0.74 in the first 10% of steps to 0.58 in the last 10%, so its budget loosens from 0.63 to 0.71, while moves from 0.86 to 0.90.
All three values of we tested (0.25, 0.5, 1) improve on OPD (Table 4, Fig. 3a), and brings the mathematics regression rate down to 4.5% at the gain rate of OPD.
Response to Reviewer ComA (4/4): cost and routing patterns
Cost against simpler alternatives (Q5)
A PCGR step costs about 20% more than an OPD step, and PCGR stays ahead at matched GPU time and at a matched teacher budget. The measured costs for Qwen3-1.7B on 8×H100, averaged over steps and seeds, are as follows (Table 11):
| Method | Backward sweeps | Time / step | Memory | Macro at equal GPU-hours |
|---|---|---|---|---|
| OPD | 1 | 185.9 s | 68.6 GB | 57.95 |
| IW-OPD | 1 | 188.7 s | 69.5 GB | 58.56 |
| A-GEM-style | 3 | 214.3 s | 72.4 GB | 58.42 |
| Blockwise projection | 3 | 218.6 s | 73.6 GB | 58.85 |
| PCGR () | 3 | 223.4 s (1.20×) | 74.1 GB | 59.63 |
The token-space methods you mention (token weighting, loss masking, advantage clipping, whole-rollout filtering) all need a single backward sweep, as OPD does; IW-OPD, the token-weighting method we measured, runs at 1.02× the step time of OPD. PCGR needs sweeps, one per discrepancy group and one for the reference, and it holds three parameter-sized buffers (, , ) whatever is. Because rollout generation and teacher scoring dominate wall-clock time, this comes to 1.20× per step, at 4B as well as at 1.7B (App. J). The extra cost pays for itself: at OPD's GPU-hour budget PCGR is 1.68 macro points ahead, and it reaches OPD's final score with about 29% fewer GPU-hours (Fig. 6).
Teacher usage is essentially matched. Routing never re-scores a rollout, so in every method the teacher scores each of a run's 400 × 64 rollouts once. The only extra teacher use is the one-off calibration of on 128 prompts, 0.5% of a run's 25,600 rollouts, and its cost is included in PCGR's budget (App. C). Since response lengths differ somewhat between methods, we also counted the response tokens scored by the teacher, summed over the four families for one seed: 412M for OPD, 409M for IW-OPD, and 403M for PCGR. At a common budget of 400M scored tokens the macro scores are 57.9, 58.5, and 60.3, so the comparison also holds at a matched budget of teacher-scored tokens.
Routing patterns (Q6)
We measured where the gates act and how stable that is, on mathematics (1.7B, 310 blocks):
- Every conflicting block is attenuated to some degree (Eq. 8), but most only slightly. In a typical step, 27% of blocks in the high-discrepancy group and 14% in the low-discrepancy group have a gate below 0.9, and 9% of gates reach zero.
- Attenuation falls mostly on the matrices that write into the residual stream. MLP down-projections and attention output projections in the upper third of the network hold 31% of the gradient energy but receive 58% of the attenuation (measured as ), about 1.9 times their share. Embeddings, norms, and early-layer attention are almost never gated.
- The pattern is partly stable. The rank correlation of per-block mean gates is 0.71 across seeds and 0.62 across the 0/25/50/100% snapshots, while individual gates vary from step to step (correlation 0.38), as they should if they respond to the current batch.
So the answer to your question is both: routing has a stable preference for certain blocks across seeds, and it also shifts with the training stage. These statistics show where routing acts, not what those blocks compute. All experiments use the Qwen3 family, so we have not yet tested whether other architectures show the same pattern. We will add these routing statistics to the appendix, and we plan to repeat them on a second model family.
We hope this clarifies how the method works and where its gains come from, and that you will take the new results into account in your final assessment. We are glad to follow up on anything that remains unclear.
Response to Reviewer ComA (1/4): how the method decides
Thank you for the detailed review. Your summary describes PCGR as identifying verified-correct prefixes and routing each token according to the status of its prefix. Our wording invites that reading, but the method works differently, and W1, Q1, and Q2 ask about exactly this, so we start there. Parts 2–4 cover the ablations (W2), reference errors and the budget (W1, Q3, Q4), and cost and routing patterns (Q5, Q6).
The decision criteria (W1, Q1, Q2)
PCGR never judges whether a prefix or a token is correct, and it has no reference model. It combines three signals, each at a different level: complete past responses that passed the verifier define what to protect, the prefix sets how cautious to be, and each parameter block of the update is then checked against the protected responses, as strictly as the prefix calls for.
Verification applies only to complete past responses. Each task's outcome verifier (answer equivalence for mathematics, passing all tests for code, and so on) decides which earlier student responses enter a FIFO buffer of C=256 responses, one per prompt. At every step we sample up to M=8 of them, skipping prompts in the current batch, and compute the gradient r of their NLL (Eq. 3). The reference is this gradient.
Prefix discrepancy sets how cautious to be. The only prefix-level quantity is ui,t=Di,t/(Di,t+τd), where Di,t=∑s<t∣Ai,s∣ accumulates the teacher–student log-ratio along the prefix (Eq. 2). We use it twice: to split tokens into K=2 groups, and to give each group a conflict budget βj=1−ρuˉj. It measures how far a prefix has drifted from the teacher: a risk indicator, not an estimate of correctness (Sec. 3.1).
Compatibility is a property of a gradient block. For group j and parameter block b we compute the preconditioned inner product aj,b=rb⊤PbGj,b (Eq. 5). The block update −ηPbGj,b changes the reference NLL by −ηaj,b to first order. A block with aj,b>0 therefore lowers the reference loss and is compatible. A block with aj,b<0 raises it, and Hj,b=max(0,−aj,b) is its conflict. Among the options in Q2 this is closest to a directional projection: an unnormalized inner product in the optimizer metric, so its magnitude matters as well as its sign. It is not a threshold on token-level losses, and final answers matter only through the verifier that fills the buffer.
Given these quantities, the gates stay as close as possible to the unrouted gradient while keeping at most a fraction βj of the group's conflict (Eq. 7). The solution is closed-form (Eq. 8):
mj,b⋆=min(1,max(0,1−λjEj,bHj,b)),
where Ej,b=Gj,b⊤PbGj,b is the block's OPD energy and λj≥0 is set so that the budget is met. Compatible blocks have Hj,b=0 and keep a gate of one, and conflicting blocks are attenuated in proportion to their conflict per unit of energy.
In short, prefix discrepancy decides how much reference conflict an update may keep, and blockwise reference geometry decides where the rest is removed. There is no token or rollout filtering: the token-level loss is unchanged, each group–block gradient is multiplied by a scalar in [0,1], and no reference-gradient term is added to the update. We will make this explicit at the start of Section 3.
Credit assignment or heuristic filter? The two halves of the method have different status. The allocation across blocks is the solution of an optimization problem: for a given reference and budget it is the unique minimizer of a convex projection (Lemma 1), and at equal retained conflict it is provably no more distorting than the best group-level scalar gate (Prop. 1). Its premise is also measured directly. Blockwise routing can improve on scalar gating only if the conflict per unit of energy, Hb/Eb, differs across blocks, which is what the coefficient κ captures: at κ=1 the two coincide. In standard OPD the median κ is 0.28–0.37 for single tokens and 0.41–0.53 for per-response groups (Table 1), and 0.56 for the full-batch groups used in training (Sec. 5.1). Grouping by discrepancy and the budget rule βj=1−ρuˉj, by contrast, are design choices, and each is evaluated with its own control (Table 8a–d, discussed in part 2).