- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies token supervision selection in on-policy distillation. The authors point out that teacher correction signals at different positions may repeat or cancel one another, so selecting tokens solely by their individual importance, or simply removing redundant signals, may change the average update direction of the entire response.
To address this issue, the authors propose CoRD. The method first constructs a correction vector for each token using output-head gradients, then uses pivoted Cholesky factorization to iteratively select tokens whose corrections are not adequately represented by the directions already selected. It subsequently computes reconstruction weights, which may be negative, so that the weighted sum of the selected vectors approximates the mean correction under dense supervision, and applies these weights to actual LoRA training.
The authors provide reconstruction error bounds in the proxy space and further measure how closely the actual LoRA gradients and AdamW updates match those under dense supervision. Across three student models and six mathematical reasoning benchmarks, CoRD achieves better average performance under matched GPU-hour budgets. It is important to distinguish that the 25% budget limits the positions contributing weighted losses; the teacher and student still score all candidate positions, so the actual runtime savings are relatively modest.
Critical strengths
-
The method combines direction selection with contribution reconstruction and has a clear motivation.
The paper does not simply treat repeated correction signals as information that can be discarded; it considers their contribution to the average update. By selecting complementary directions and reconstructing the full mean with signed weights, CoRD addresses both directional redundancy and imbalanced contributions. Sharing a single factorization between selection and reconstruction also makes the design relatively compact. Component experiments further show that both appropriate selection and reconstruction contribute improvements, with their combination performing best.
-
The evaluation extends beyond the proxy space, and the experiments are comprehensive.
The authors report not only proxy reconstruction errors but also per-trajectory and batch LoRA gradients, as well as parameter updates starting from identical AdamW states. CoRD closely matches the dense reference in these diagnostics. The paper also uses three training seeds and compares performance under fixed supervision budgets and matched compute budgets across multiple student models and mathematical reasoning benchmarks, while transparently reporting that actual runtime savings are limited. This evidence makes the method’s effectiveness reasonably convincing and is a main reason for my support for acceptance.
Critical weaknesses
-
Why directional coverage benefits actual updates more than directly fitting the proxy mean remains insufficiently explained.
Selection and reconstruction optimize different quantities. Appendix D.1.4 explicitly states that selecting the largest diagonal residual neither maximizes the reduction in total residual energy nor maximizes the reduction in mean error at each step. The authors also provide an example in which large directions cancel, leaving the selected direction unhelpful for mean reconstruction. Therefore, the selection rule alone does not establish that it is preferable to direct mean fitting under a limited budget.
The results in Table 11 are insightful: signed mean pursuit achieves lower proxy-mean error but higher actual LoRA batch-gradient error. This supports the conclusion that proxy-mean error alone is insufficient, but it does not yet explain why covering these directions is better. Possible explanations include position-dependent mappings to LoRA parameters, which prevent cancellation in proxy space from carrying over to parameter space, or differences in reconstruction coefficients, which affect how proxy mismatch is amplified. Evidence distinguishing these explanations is currently limited.
Similarly, high batch-gradient and AdamW-update cosine similarities are important evidence, but their sources could be further disentangled. Low batch-gradient error may partly result from cancellation of errors across trajectories, while optimizer-step similarity after 1,000 updates may also be influenced by shared historical moments. The authors already discuss theoretical limitations at these levels, but do not sufficiently establish which factors dominate in practice or when these close correspondences may break down.
-
The final gains cannot be fully explained by more faithful reconstruction of the dense update.
The authors appropriately distinguish the budget needed for reconstruction at a fixed state from the budget that yields the best training performance. However, in Table 10, increasing the retention fraction from 0.25 to 1 reduces mean@16 from 66.9 to 63.4, with the full budget exactly recovering Dense-RKL. The fixed-state, one-step tests in Table 13 also show that Dense can reduce the full KL loss more than CoRD.
Thus, better reconstruction can explain why CoRD outperforms weaker sparse-selection methods, but it does not fully explain why CoRD ultimately outperforms Dense. An important open question is what CoRD discards that makes this deviation beneficial. The projection may regularize update directions, the update scale may change, or differences in early updates may alter subsequent on-policy trajectories. The current results do not sufficiently distinguish these possibilities.
Questions for authors
-
Why is it worthwhile to retain large directions that nearly cancel in the proxy mean?
Could you further analyze the difference between signed mean pursuit and CoRD in Table 11? Is it mainly caused by token-specific Jacobians preventing proxy-space cancellation from carrying over to LoRA parameter space, or by differences in the magnitude and stability of reconstruction coefficients?
For example, the two methods could be compared on the same trajectories in terms of weight norms, actual gradient errors, and deviations in the mapping from proxy corrections to LoRA gradients. This would help explain why allocating a limited budget to directional coverage is more effective than directly pursuing lower proxy-mean error.
-
What contributes to the high similarity of actual updates, and how can proxy failure be detected in advance?
Could you report how errors change before and after aggregation across trajectories, and test whether the high batch-gradient similarity is sensitive to batch size and trajectory composition? This would help distinguish the accuracy of the proxy itself from the effects of cross-trajectory error cancellation.
For AdamW, could you compare diagnostics with retained versus reset historical moments at the same model checkpoint to assess how much the existing optimizer state contributes to update similarity? In addition, which indicators—coverage residuals, weight norms, factorization condition numbers, or a small sample of actual gradients—best predict when the correspondence between proxy corrections and actual updates begins to fail?
-
Which beneficial deviation from Dense primarily explains CoRD’s gains?
Could you compare CoRD and Dense after matching update norms, and conduct comparisons both on shared trajectories and with independently regenerated on-policy trajectories, to separate the effects of update direction, update scale, and changes in subsequent data distributions?
If the benefit is mainly a scale effect, the gap should shrink after matching update norms. If it mainly comes from projection-based direction selection, the advantage should persist on the same trajectories with comparable update scales. If it mainly arises from changes in subsequent visitation distributions, the advantage should be more evident during training that continually regenerates trajectories. Which explanation do the authors consider most consistent with the existing observations?
3 comments
Response to Reviewer yq6w (2/3): Q1, Q2
Continuation of our response to Reviewer yq6w.
Q1. Is the coverage advantage due to token-specific mappings or coefficient amplification?
On the same trajectories, CoRD has smaller coefficient norms and lower error when transferring proxy-derived reconstructions to actual gradients. Define , , and
This measures the transfer error of proxy-derived token reconstructions in actual-gradient space. It captures the combined effects of uncovered proxy components and mapping inconsistency.
| Method, common 1k state | mean / P95 | ||||
|---|---|---|---|---|---|
| Signed mean pursuit | .069 | .367 | 2.60 / 5.20 | .395 | .405 |
| CoRD | .144 | .217 | 1.55 / 2.38 | .270 | .350 |
Holding each selected set fixed and solving LS subject to a common coefficient constraint gives:
| Constraint | Pursuit / | CoRD / |
|---|---|---|
| Unconstrained | .069 / .405 | .144 / .350 |
| .124 / .385 | .153 / .347 | |
| .226 / .438 | .204 / .380 |
At a common cap of 2, the actual-gradient error gap narrows from .055 to .038. A cap of 1 increases both methods' errors. This supports a role for coefficient sensitivity, with a remaining advantage associated with CoRD's selected span. The algebra also explains the relevance of token-specific mappings: proxy cancellation depends on , whereas actual-gradient cancellation depends on . The smaller transfer residual is consistent with the combined influence of coverage and these mappings.
Q2. Batch aggregation, historical moments, and advance detection of proxy failure.
The controls you suggested show contributions from both aggregation and optimizer history. The same 1k trajectory pool is regrouped using the original length weights :
| Method | Batch size | Random | Similar length | Similar geometry |
|---|---|---|---|---|
| CoRD | 1 | .350 | .350 | .350 |
| CoRD | 4 | .284 | .304 | .328 |
| CoRD | 16 | .196 | .235 | .285 |
| CoRD | 32 | .178 | .218 | .273 |
| Signed mean pursuit | 1 | .405 | .405 | .405 |
| Signed mean pursuit | 16 | .313 | .360 | .395 |
| Signed mean pursuit | 32 | .290 | .340 | .382 |
For representative normalized batches, let , , , and , with :
| CoRD, | Cancellation | Error | |||
|---|---|---|---|---|---|
| Random | .3500 | .1372 | .700 | .392 | .196 |
| Similar geometry | .3500 | .2337 | .820 | .668 | .285 |
Random grouping reduces the error numerator from .2337 to .1372 even though the gradient denominator also decreases. This directly demonstrates cancellation of errors across trajectories.
At the original fixed-order batch size 16, cloning the checkpoint and resetting both moments and the step counter gives:
| Method | AdamW history | Step cosine | Relative step error |
|---|---|---|---|
| CoRD | Retained | .985 | .175 |
| CoRD | Reset | .976 | .225 |
| Signed mean pursuit | Retained | .958 | .292 |
| Signed mean pursuit | Reset | .932 | .370 |
The actual batch-gradient errors are .199/.316 for CoRD/pursuit under both history conditions. These fixed-order batches differ from the random groups above. Resetting the history increases step error for both methods, and CoRD retains the closer match to Dense in both conditions.
For advance detection, failure means . The split uses 128 development and 128 held-out prompt identities. The held-out set contains 1,152 seed/checkpoint records and 804 failures. Predictors use the diagnostic features below, with full-gradient errors reserved for constructing labels.
| Indicator | Held-out AUROC | Cost relative to full post-scoring diagnostic |
|---|---|---|
| Coverage residual | .67 | .2% |
| Coefficient norm | .64 | .1% |
| Factor condition number | .62 | .5% |
| Eight actual-token-gradient probes | .80 | 9.0% |
| Three inexpensive signals combined | .73 | .8% |
| Combined signals + probes | .84 | 9.8% |
Combining the inexpensive signals gives AUROC .73. Adding eight actual-token-gradient probes raises it to .84 at 9.8% of the full post-scoring diagnostic cost. This provides a practical way to rank responses by their risk of proxy mismatch.
Response to Reviewer yq6w (3/3): Q3
Continuation of our response to Reviewer yq6w.
Q3. Which deviation from Dense best explains the gain: direction, scale, or visitation?
The factorial comparison you suggested supports contributions from update direction and trajectory feedback. For each seed, the shared condition replays Dense-reference responses and rescores them at each model's own parameters. Norm matching scales the final parameter increment to the reference increment's norm at that update. Each model retains its own optimizer moments.
| CoRD trajectory / scale condition | Mean norm ratio | M16 | P8 | Common held-out KL |
|---|---|---|---|---|
| Shared reference / native | .94 | 65.5 | 81.5 | .121 |
| Shared reference / matched | 1.00 | 65.0 | 81.0 | .122 |
| Own on-policy / native | .93 | 66.9±0.9 | 82.9±0.8 | .124 |
| Own on-policy / matched | 1.00 | 66.2 | 82.2 | .125 |
| Dense reference | 1.00 | 63.4±0.9 | 79.4±1.0 | .118 |
Matching norms narrows the gain by 0.5 points on shared trajectories and 0.7 on own trajectories. CoRD still gains 1.6 points with both trajectories and norms controlled, providing direct evidence for a direction/weighting effect. With norms matched, its own trajectories yield a 2.8-point gain over Dense, compared with 1.6 under shared replay. This supports an additional role for on-policy feedback. Because replay also changes the match between the model and its training distribution, this comparison measures the combined trajectory effect. The most consistent interpretation is that direction, scale, and trajectory feedback contribute jointly.
Response to Reviewer yq6w (1/3): W1, W2
Thank you for your time, careful reading, and valuable suggestions on the mechanisms behind CoRD. We address your questions below through the coefficient, optimizer, and trajectory controls.
M16/P8 denote six-task macro mean@16/pass@8 (%). Unless specified otherwise, experiments use Qwen3-8B → Qwen3-4B with rank-32 attention/MLP adapters, 1,000 updates, three seeds, and ρ=0.25 for sparse methods. Reported ± values are seed standard deviations. Comparisons without uncertainty estimates describe differences in means. Errors are relative errors. Fixed-state diagnostics share Dense-reference checkpoints and trajectories, with 1k as the default checkpoint. Training scores come from each method's own endpoint. Numbered tables and figures refer to the submission.
Response. As you point out, coverage and mean fitting serve different purposes. In Appendix D.1.4, each greedy pivot maximizes individual residual energy, and the subsequent reconstruction fits the mean within the selected span. The following controls examine coefficient sensitivity, aggregation, and optimizer history at the common 1k checkpoint:
Coefficient constraints reduce the trajectory-error gap. Aggregation and optimizer history further improve agreement with Dense. CoRD's advantage persists under the coefficient constraint, correlated grouping, and moment reset. The results identify several contributors to actual-update fidelity and show that the advantage survives each control. Q1 and Q2 detail the mapping and failure-detection results.
Response. Your question motivates a direct test of which changes to Dense training improve reasoning. The shared-trajectory and norm-matching controls separate the effects of update direction, scale, and trajectory feedback:
The same Dense reference serves its equivalent control conditions. CoRD retains a gain when both trajectories and norms are matched, supporting a direction/weighting effect. The larger gain with its own on-policy trajectories is consistent with beneficial trajectory feedback. On common held-out prefixes, native on-policy CoRD has KL .124 versus Dense .118 nats/token and achieves higher reasoning scores. Together with Table 13, this shows why final reasoning quality should be assessed alongside dense-loss fidelity.