acceptodds
Under review as a conference paper at ICLR 2027

CoRD: On-Policy Distillation via Correction Reconstruction

Abstract

On-policy distillation helps language models improve reasoning through teacher feedback on their own responses, but selecting fewer supervised tokens can distort the collective learning signal. We aim to preserve this signal under a limited token supervision budget by accounting for how individual corrections reinforce or cancel one another. We introduce Correction Reconstruction Distillation (CoRD), which preserves the contributions of repeated correction directions to the average update. CoRD selects complementary token corrections in an output-head gradient representation and assigns signed weights to reconstruct the dense mean within their span, with theoretical bounds on approximation error in this representation. Across three student models and six mathematical reasoning benchmarks, including MATH-500 and AIME 2024/2025, CoRD improves macro mean@16 by 1.2–1.3 percentage points and pass@8 by 1.0–2.3 points over the strongest compared baselines under matched GPU-hour caps, using a 25% supervised-token budget. Coupling complementary selection with mean reconstruction allows CoRD to exploit redundancy while accounting for omitted tokens' aggregate contributions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, large language models, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yq6wOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies token supervision selection in on-policy distillation. The authors point out that teacher correction signals at different positions may repeat or cancel one another, so selecting tokens solely by their individual importance, or simply removing redundant signals, may change the average update direction of the entire response.

To address this issue, the authors propose CoRD. The method first constructs a correction vector for each token using output-head gradients, then uses pivoted Cholesky factorization to iteratively select tokens whose corrections are not adequately represented by the directions already selected. It subsequently computes reconstruction weights, which may be negative, so that the weighted sum of the selected vectors approximates the mean correction under dense supervision, and applies these weights to actual LoRA training.

The authors provide reconstruction error bounds in the proxy space and further measure how closely the actual LoRA gradients and AdamW updates match those under dense supervision. Across three student models and six mathematical reasoning benchmarks, CoRD achieves better average performance under matched GPU-hour budgets. It is important to distinguish that the 25% budget limits the positions contributing weighted losses; the teacher and student still score all candidate positions, so the actual runtime savings are relatively modest.

Critical strengths
  1. The method combines direction selection with contribution reconstruction and has a clear motivation.

    The paper does not simply treat repeated correction signals as information that can be discarded; it considers their contribution to the average update. By selecting complementary directions and reconstructing the full mean with signed weights, CoRD addresses both directional redundancy and imbalanced contributions. Sharing a single factorization between selection and reconstruction also makes the design relatively compact. Component experiments further show that both appropriate selection and reconstruction contribute improvements, with their combination performing best.

  2. The evaluation extends beyond the proxy space, and the experiments are comprehensive.

    The authors report not only proxy reconstruction errors but also per-trajectory and batch LoRA gradients, as well as parameter updates starting from identical AdamW states. CoRD closely matches the dense reference in these diagnostics. The paper also uses three training seeds and compares performance under fixed supervision budgets and matched compute budgets across multiple student models and mathematical reasoning benchmarks, while transparently reporting that actual runtime savings are limited. This evidence makes the method’s effectiveness reasonably convincing and is a main reason for my support for acceptance.

Critical weaknesses
  1. Why directional coverage benefits actual updates more than directly fitting the proxy mean remains insufficiently explained.

    Selection and reconstruction optimize different quantities. Appendix D.1.4 explicitly states that selecting the largest diagonal residual neither maximizes the reduction in total residual energy nor maximizes the reduction in mean error at each step. The authors also provide an example in which large directions cancel, leaving the selected direction unhelpful for mean reconstruction. Therefore, the selection rule alone does not establish that it is preferable to direct mean fitting under a limited budget.

    The results in Table 11 are insightful: signed mean pursuit achieves lower proxy-mean error but higher actual LoRA batch-gradient error. This supports the conclusion that proxy-mean error alone is insufficient, but it does not yet explain why covering these directions is better. Possible explanations include position-dependent mappings to LoRA parameters, which prevent cancellation in proxy space from carrying over to parameter space, or differences in reconstruction coefficients, which affect how proxy mismatch is amplified. Evidence distinguishing these explanations is currently limited.

    Similarly, high batch-gradient and AdamW-update cosine similarities are important evidence, but their sources could be further disentangled. Low batch-gradient error may partly result from cancellation of errors across trajectories, while optimizer-step similarity after 1,000 updates may also be influenced by shared historical moments. The authors already discuss theoretical limitations at these levels, but do not sufficiently establish which factors dominate in practice or when these close correspondences may break down.

  2. The final gains cannot be fully explained by more faithful reconstruction of the dense update.

    The authors appropriately distinguish the budget needed for reconstruction at a fixed state from the budget that yields the best training performance. However, in Table 10, increasing the retention fraction from 0.25 to 1 reduces mean@16 from 66.9 to 63.4, with the full budget exactly recovering Dense-RKL. The fixed-state, one-step tests in Table 13 also show that Dense can reduce the full KL loss more than CoRD.

    Thus, better reconstruction can explain why CoRD outperforms weaker sparse-selection methods, but it does not fully explain why CoRD ultimately outperforms Dense. An important open question is what CoRD discards that makes this deviation beneficial. The projection may regularize update directions, the update scale may change, or differences in early updates may alter subsequent on-policy trajectories. The current results do not sufficiently distinguish these possibilities.

Questions for authors
  1. Why is it worthwhile to retain large directions that nearly cancel in the proxy mean?

    Could you further analyze the difference between signed mean pursuit and CoRD in Table 11? Is it mainly caused by token-specific Jacobians preventing proxy-space cancellation from carrying over to LoRA parameter space, or by differences in the magnitude and stability of reconstruction coefficients?

    For example, the two methods could be compared on the same trajectories in terms of weight norms, actual gradient errors, and deviations in the mapping from proxy corrections to LoRA gradients. This would help explain why allocating a limited budget to directional coverage is more effective than directly pursuing lower proxy-mean error.

  2. What contributes to the high similarity of actual updates, and how can proxy failure be detected in advance?

    Could you report how errors change before and after aggregation across trajectories, and test whether the high batch-gradient similarity is sensitive to batch size and trajectory composition? This would help distinguish the accuracy of the proxy itself from the effects of cross-trajectory error cancellation.

    For AdamW, could you compare diagnostics with retained versus reset historical moments at the same model checkpoint to assess how much the existing optimizer state contributes to update similarity? In addition, which indicators—coverage residuals, weight norms, factorization condition numbers, or a small sample of actual gradients—best predict when the correspondence between proxy corrections and actual updates begins to fail?

  3. Which beneficial deviation from Dense primarily explains CoRD’s gains?

    Could you compare CoRD and Dense after matching update norms, and conduct comparisons both on shared trajectories and with independently regenerated on-policy trajectories, to separate the effects of update direction, update scale, and changes in subsequent data distributions?

    If the benefit is mainly a scale effect, the gap should shrink after matching update norms. If it mainly comes from projection-based direction selection, the advantage should persist on the same trajectories with comparable update scales. If it mainly arises from changes in subsequent visitation distributions, the advantage should be more evident during training that continually regenerates trajectories. Which explanation do the authors consider most consistent with the existing observations?

3 comments
AuthorsOpenReview ↗

Response to Reviewer yq6w (1/3): W1, W2

Thank you for your time, careful reading, and valuable suggestions on the mechanisms behind CoRD. We address your questions below through the coefficient, optimizer, and trajectory controls.

M16/P8 denote six-task macro mean@16/pass@8 (%). Unless specified otherwise, experiments use Qwen3-8B → Qwen3-4B with rank-32 attention/MLP adapters, 1,000 updates, three seeds, and ρ=0.25\rho=0.25 for sparse methods. Reported ±\pm values are seed standard deviations. Comparisons without uncertainty estimates describe differences in means. Errors are relative errors. Fixed-state diagnostics share Dense-reference checkpoints and trajectories, with 1k as the default checkpoint. Training scores come from each method's own endpoint. Numbered tables and figures refer to the submission.

W1. Why coverage helps actual updates, including the roles of coefficients, aggregation, and optimizer history.

Response.{\color{green}\mathbf{Response.}} As you point out, coverage and mean fitting serve different purposes. In Appendix D.1.4, each greedy pivot maximizes individual residual energy, and the subsequent reconstruction fits the mean within the selected span. The following controls examine coefficient sensitivity, aggregation, and optimizer history at the common 1k checkpoint:

DiagnosticSigned mean pursuitCoRD
Proxy-mean error.069.144
Mean coefficient ℓ1\ell_1 norm2.601.55
Trajectory-gradient error.405.350
Trajectory error with ∥w∥1≤2\lVert w\rVert_1\le2.385.347
Random-batch error, B=16B=16.313.196
Geometry-grouped batch error, B=16B=16.395.285
AdamW step error, retained moments.292.175
AdamW step error, reset moments.370.225

Coefficient constraints reduce the trajectory-error gap. Aggregation and optimizer history further improve agreement with Dense. CoRD's advantage persists under the coefficient constraint, correlated grouping, and moment reset. The results identify several contributors to actual-update fidelity and show that the advantage survives each control. Q1 and Q2 detail the mapping and failure-detection results.


W2. Dense-update fidelity does not fully explain outperforming Dense.

Response.{\color{green}\mathbf{Response.}} Your question motivates a direct test of which changes to Dense training improve reasoning. The shared-trajectory and norm-matching controls separate the effects of update direction, scale, and trajectory feedback:

Training conditionDense M16CoRD M16DifferenceHeld-out KL, Dense / CoRD
Shared reference trajectories, native norms63.465.5+2.1.118 / .121
Shared reference trajectories, matched norms63.465.0+1.6.118 / .122
Each method's on-policy trajectories, native norms63.466.9+3.5.118 / .124
Each method's on-policy trajectories, matched norms63.466.2+2.8.118 / .125

The same Dense reference serves its equivalent control conditions. CoRD retains a gain when both trajectories and norms are matched, supporting a direction/weighting effect. The larger gain with its own on-policy trajectories is consistent with beneficial trajectory feedback. On common held-out prefixes, native on-policy CoRD has KL .124 versus Dense .118 nats/token and achieves higher reasoning scores. Together with Table 13, this shows why final reasoning quality should be assessed alongside dense-loss fidelity.

AuthorsOpenReview ↗

Response to Reviewer yq6w (2/3): Q1, Q2

Continuation of our response to Reviewer yq6w.

Q1. Is the coverage advantage due to token-specific mappings or coefficient amplification?

Response.{\color{green}\mathbf{Response.}} On the same trajectories, CoRD has smaller coefficient norms and lower error when transferring proxy-derived reconstructions to actual gradients. Define at=KBB†KBta_t=K_{BB}^{\dagger}K_{Bt}, GB=[gb]b∈BG_B=[g_b]_{b\in B}, and

Rmap=∑t∥gt−GBat∥2∑t∥gt∥2.R_{\rm map}=\frac{\sum_t\lVert g_t-G_Ba_t\rVert_2}{\sum_t\lVert g_t\rVert_2}.

This measures the transfer error of proxy-derived token reconstructions in actual-gradient space. It captures the combined effects of uncovered proxy components and mapping inconsistency.

Method, common 1k stateeμe_\muηcov\eta_{\rm cov}∥w∥1\lVert w\rVert_1 mean / P95RmapR_{\rm map}egtraje_g^{\rm traj}
Signed mean pursuit.069.3672.60 / 5.20.395.405
CoRD.144.2171.55 / 2.38.270.350

Holding each selected set fixed and solving LS subject to a common coefficient constraint gives:

ConstraintPursuit eμe_\mu / egtraje_g^{\rm traj}CoRD eμe_\mu / egtraje_g^{\rm traj}
Unconstrained.069 / .405.144 / .350
∥w∥1≤2\lVert w\rVert_1\le2.124 / .385.153 / .347
∥w∥1≤1\lVert w\rVert_1\le1.226 / .438.204 / .380

At a common cap of 2, the actual-gradient error gap narrows from .055 to .038. A cap of 1 increases both methods' errors. This supports a role for coefficient sensitivity, with a remaining advantage associated with CoRD's selected span. The algebra also explains the relevance of token-specific mappings: proxy cancellation depends on rtht⊤r_th_t^\top, whereas actual-gradient cancellation depends on Jt⊤rtJ_t^\top r_t. The smaller transfer residual is consistent with the combined influence of coverage and these mappings.


Q2. Batch aggregation, historical moments, and advance detection of proxy failure.

Response.{\color{green}\mathbf{Response.}} The controls you suggested show contributions from both aggregation and optimizer history. The same 1k trajectory pool is regrouped using the original length weights αa=Ta/∑aTa\alpha_a=T_a/\sum_aT_a:

MethodBatch sizeRandom egbatche_g^{\rm batch}Similar lengthSimilar geometry
CoRD1.350.350.350
CoRD4.284.304.328
CoRD16.196.235.285
CoRD32.178.218.273
Signed mean pursuit1.405.405.405
Signed mean pursuit16.313.360.395
Signed mean pursuit32.290.340.382

For representative normalized batches, let δa=g^a−ga\delta_a=\widehat g_a-g_a, A=∑aαa∥δa∥A=\sum_a\alpha_a\lVert\delta_a\rVert, N=∥∑aαaδa∥N=\lVert\sum_a\alpha_a\delta_a\rVert, and G=∥∑aαaga∥G=\lVert\sum_a\alpha_ag_a\rVert, with ∑aαa∥ga∥=1\sum_a\alpha_a\lVert g_a\rVert=1:

CoRD, B=16B=16AANNGGCancellation N/AN/AError N/GN/G
Random.3500.1372.700.392.196
Similar geometry.3500.2337.820.668.285

Random grouping reduces the error numerator from .2337 to .1372 even though the gradient denominator also decreases. This directly demonstrates cancellation of errors across trajectories.

At the original fixed-order batch size 16, cloning the checkpoint and resetting both moments and the step counter gives:

MethodAdamW historyStep cosineRelative step error
CoRDRetained.985.175
CoRDReset.976.225
Signed mean pursuitRetained.958.292
Signed mean pursuitReset.932.370

The actual batch-gradient errors are .199/.316 for CoRD/pursuit under both history conditions. These fixed-order batches differ from the random groups above. Resetting the history increases step error for both methods, and CoRD retains the closer match to Dense in both conditions.

For advance detection, failure means egtraj>.30e_g^{\rm traj}>.30. The split uses 128 development and 128 held-out prompt identities. The held-out set contains 1,152 seed/checkpoint records and 804 failures. Predictors use the diagnostic features below, with full-gradient errors reserved for constructing labels.

IndicatorHeld-out AUROCCost relative to full post-scoring diagnostic
Coverage residual.67.2%
Coefficient ℓ1\ell_1 norm.64.1%
Factor condition number.62.5%
Eight actual-token-gradient probes.809.0%
Three inexpensive signals combined.73.8%
Combined signals + probes.849.8%

Combining the inexpensive signals gives AUROC .73. Adding eight actual-token-gradient probes raises it to .84 at 9.8% of the full post-scoring diagnostic cost. This provides a practical way to rank responses by their risk of proxy mismatch.

AuthorsOpenReview ↗

Response to Reviewer yq6w (3/3): Q3

Continuation of our response to Reviewer yq6w.

Q3. Which deviation from Dense best explains the gain: direction, scale, or visitation?

Response.{\color{green}\mathbf{Response.}} The factorial comparison you suggested supports contributions from update direction and trajectory feedback. For each seed, the shared condition replays Dense-reference responses and rescores them at each model's own parameters. Norm matching scales the final parameter increment to the reference increment's norm at that update. Each model retains its own optimizer moments.

CoRD trajectory / scale conditionMean norm ratioM16P8Common held-out KL
Shared reference / native.9465.581.5.121
Shared reference / matched1.0065.081.0.122
Own on-policy / native.9366.9±0.982.9±0.8.124
Own on-policy / matched1.0066.282.2.125
Dense reference1.0063.4±0.979.4±1.0.118

Matching norms narrows the gain by 0.5 points on shared trajectories and 0.7 on own trajectories. CoRD still gains 1.6 points with both trajectories and norms controlled, providing direct evidence for a direction/weighting effect. With norms matched, its own trajectories yield a 2.8-point gain over Dense, compared with 1.6 under shared replay. This supports an additional role for on-policy feedback. Because replay also changes the match between the model and its training distribution, this comparison measures the combined trajectory effect. The most consistent interpretation is that direction, scale, and trajectory feedback contribute jointly.

Reviewer_nEHCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoRD, or Correction Reconstruction Distillation, for on-policy distillation from teacher feedback on student-generated responses. The central idea is to construct a correction kernel from the discrepancy between the teacher and student distributions, select representative token-level correction directions in an implicit output-head proxy space using pivoted Cholesky factorization, and transfer the resulting directions to LoRA updates. The paper also provides a theoretical analysis of spectral compression and evaluates CoRD against several on-policy distillation and token-selection baselines under matched compute budgets. The experiments consider multiple reasoning benchmarks and report task performance, token-selection accuracy, training cost, and comparisons across different compression strategies. Overall, the paper addresses the redundancy and computational cost of using dense teacher feedback in on-policy distillation by attempting to identify a compact set of informative correction directions.

Critical strengths
  1. The paper proposes a clearly structured correction-reconstruction framework. Representing the teacher-student discrepancy as correction signals and selecting representative directions through an implicit kernel provides a unified perspective on compressing feedback for on-policy distillation.

  2. The paper combines theoretical analysis with empirical validation. The spectral properties of pivoted Cholesky decomposition provide a mathematical motivation for the selection procedure, while the matched-budget comparisons involving task performance and training cost make the evaluation more relevant to practical deployment.

Critical weaknesses
  1. There is a substantial gap between the theoretical space and the actual optimization space. The paper provides strong properties for kernel projection in the output-head proxy space, but the final model is trained through LoRA parameter updates. The effect of proxy reconstruction error on the actual gradient update and final model quality is not sufficiently analyzed. The current evidence suggests that the method is empirically useful, but it does not yet establish that kernel reconstruction is the main source of the observed distillation gains.

  2. The source of the method’s improvement is not sufficiently isolated. CoRD combines correction construction, kernel selection, pivoted Cholesky decomposition, compression ratio, and LoRA transfer, and each component may affect the final results. More complete comparisons are needed against top-k selection, random selection, loss-based selection, KL-based selection, gradient-based selection, and direct distillation without reconstruction. Without these controls, it is difficult to determine whether the gains come from structured selection or from additional computation, filtering, or training configuration.

Questions for authors
  1. How are corrections reconstructed in the output-head proxy space converted into LoRA updates? Can the authors report the relationship between proxy reconstruction error and the corresponding LoRA gradient or parameter update?

  2. In the matched-compute experiments, are teacher rollouts, kernel construction, pivoted Cholesky selection, LoRA training, and final evaluation all included in the reported cost? Are the GPU-hour measurements based on identical hardware and batch scheduling across methods?

  3. How do performance and token-selection accuracy change as the selection budget varies? Is there a stable scaling trend or a clear compression-ratio threshold?

  4. Do the advantages of CoRD over top-k, random, loss-based, KL-based, and gradient-based selection remain consistent across model sizes, task types, and LoRA ranks?

  5. Can the theoretical spectral results be connected to a verifiable upper bound on final training loss or model performance, or do they only characterize reconstruction quality in the proxy space?

4 comments
AuthorsOpenReview ↗

Response to Reviewer nEHC (1/3): W1, W2

Thank you for your time and constructive feedback, especially your questions about gradient transfer and experimental controls. Below, we clarify the implementation and present the results relevant to each concern.

M16/P8 denote six-task macro mean@16/pass@8 (%). Unless specified otherwise, experiments use Qwen3-8B → Qwen3-4B with rank-32 attention/MLP adapters, 1,000 updates, three seeds, and ρ=0.25\rho=0.25 for sparse methods. Reported ±\pm values are seed standard deviations. Comparisons without uncertainty estimates describe differences in means. Errors are relative errors. Fixed-state diagnostics share Dense-reference checkpoints and trajectories, with 1k as the default checkpoint. Training scores come from each method's own endpoint. Numbered tables and figures refer to the submission.

W1. Gap between proxy theory, actual optimization, and the source of reasoning gains.

Response.{\color{green}\mathbf{Response.}} The analysis distinguishes proxy reconstruction, trainable-gradient fidelity, and final reasoning quality. CoRD uses the proxy to choose loss weights, then computes the actual update through ordinary backpropagation. Tables 11 and 13 measure the resulting gradients and parameter updates. The following same-trajectory comparisons examine coefficient sensitivity:

Method / coefficient constraintProxy errorLoRA trajectory error
Signed mean pursuit / unconstrained.069.405
CoRD / unconstrained.144.350
Signed mean pursuit / ∥w∥1≤2\lVert w\rVert_1\le2.124.385
CoRD / ∥w∥1≤2\lVert w\rVert_1\le2.153.347

CoRD achieves lower actual-gradient error under both coefficient settings. To examine the effect on reasoning while controlling update scale and subsequent sampling, both methods consume the same Dense-reference trajectory stream with matched parameter-increment norms:

Shared trajectories, matched normsM16P8
Dense63.479.4
CoRD65.081.0

The 1.6-point gain under these controls supports a benefit from CoRD's update direction and weighting. Together with the gradient diagnostics, it links correction selection to actual optimization and reasoning performance. Q5 gives the assumptions under which the theory connects proxy and trainable-gradient errors.


W2. Missing controls for selection, reconstruction, and gradient-based alternatives.

Response.{\color{green}\mathbf{Response.}} Table 2 includes uniform, entropy, RKL-top-mm, correction-norm, and proxy-token adaptations of CRAIG [1] and GRAD-MATCH [2]. Table 9 separates selection and reconstruction. Since the token loss is full-vocabulary reverse KL, loss-top-mm and KL-top-mm give the same ranking. The additional comparison applies the same signed LS to each selector:

SelectionWeightsM16P8
DenseAll-position uniform63.479.4
UniformUniform61.178.0
CoRDUniform64.680.9
UniformLS63.579.9
KL-top-mmLS64.681.0
Correction-norm-top-mmLS65.081.4
Signed mean pursuitLS64.981.2
Eigen-QRLS66.782.7
CoRDLS66.982.9

Actual LoRA-gradient-norm selection is also evaluated on 96 common trajectories, balanced across short/medium/long responses:

Selector + LSLoRA trajectory errorRelative selection/gradient-construction cost
Correction-norm-top-mm.3861.1×
Actual-LoRA-gradient-norm-top-mm.36918.0×
CoRD.3541.0×

On this fixed-state diagnostic, CoRD gives the lowest actual-gradient error at a fraction of the actual-gradient selector's cost. The training comparisons and gradient diagnostics jointly support the value of reconstruction and complementary direction selection.

[1] Coresets for Data-efficient Training of Machine Learning Models.

[2] GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training.

AuthorsOpenReview ↗

Response to Reviewer nEHC (2/3): Q1, Q2, Q3

Continuation of our response to Reviewer nEHC.

Q1. How are output-head corrections converted into LoRA updates?

Response.{\color{green}\mathbf{Response.}} Reconstruction coefficients become weights on the original token losses. Ordinary backpropagation then computes the LoRA update through the student graph. For rt=∇ztℓtr_t=\nabla_{z_t}\ell_t, Jt=∂zt/∂θJ_t=\partial z_t/\partial\theta, and selected indices BB,

\qquad \widehat g_\theta=\sum_{j\in B}w_jJ_j^\top r_j.$$ The retained student scoring graph supplies these Jacobians. The output head remains frozen in LoRA training. Your requested measurements use common Dense-reference states: | Method | State | Proxy error | LoRA trajectory error | LoRA batch error | |---|---|---:|---:|---:| | Signed mean pursuit | Init. | .083 | .377 | .296 | | CoRD | Init. | .158 | .373 | .181 | | Signed mean pursuit | 1k | .069 | .405 | .316 | | CoRD | 1k | .144 | .350 | .199 | Cloned 1k AdamW updates, with the original fixed-order batches, give: | Method | Retained-history cosine / error | Reset-history cosine / error | |---|---:|---:| | Signed mean pursuit | .958 / .292 | .932 / .370 | | CoRD | **.985 / .175** | **.976 / .225** | CoRD retains its advantage after both moments and the step counter are reset. This confirms favorable actual-update fidelity in both optimizer-history conditions. --- > #### **Q2. Cost accounting, hardware, and scheduling.** ${\color{green}\mathbf{Response.}}$ Training GPU-hours include response generation, teacher/student scoring, kernel construction, selection, reconstruction, backward, and optimizer/synchronization. CoRD generates responses with the student, and the teacher scores every valid position. Teacher-generated rollouts are included in the training cost of baselines that require them. Validation/test evaluation is reported separately. | Profile stage, seconds/update | Dense | CoRD | |---|---:|---:| | Student response generation | 71.8 | 72.0 | | Teacher scoring | 31.7 | 31.8 | | Student scoring | 28.4 | 28.5 | | Kernel + selection + reconstruction | .2 | 6.3 | | Backward | 44.5 | 32.3 | | Optimizer + synchronization | 5.1 | 4.9 | | Total | 181.7 | 175.8 | | Complete-run accounting, per seed | Dense | CoRD | |---|---:|---:| | Training GPUh | 412.6 | 397.3 | | Validation/test GPUh | 48.0 | 48.0 | | Combined GPUh | 460.6 | 445.3 | | Teacher-scored positions, millions | 11.45 | 11.30 | Both methods use eight A100-80GB GPUs and global batch 16. The nonoverlapping stage profile covers 100 updates after 20 warm-up updates. Full-run costs also include warm-up and saving overhead. The combined total covers each run's training and validation/test evaluation. Development searches, shared reference preparation, and mechanism diagnostics are separate experimental costs. Under this accounting, CoRD improves both M16 and P8 by 3.5 points with a lower training-plus-evaluation cost. --- > #### **Q3. Budget scaling and token-selection accuracy.** ${\color{green}\mathbf{Response.}}$ “Token-selection accuracy” in Table 2 refers to downstream reasoning accuracy obtained with each selector. Reconstruction quality is measured separately on the Table 11 diagnostic cache: | $\rho$ | $e_\mu$, Init. | $e_\mu$, 500 | $e_\mu$, 1k | Qwen3-4B M16 | |---|---:|---:|---:|---:| | .10 | .285 | .274 | .263 | 65.5 | | .25 | .158 | .151 | .144 | **66.9** | | .50 | .061 | .056 | .052 | 66.2 | | 1 | 0 | 0 | 0 | 63.4 | The fixed-state reconstruction error decreases as the span expands. At $\rho=1$, uniform weights on all positions recover Dense exactly. The additional M16 curves also favor intermediate budgets: | Configuration | $\rho=.10$ | $\rho=.25$ | $\rho=.50$ | $\rho=1$ | |---|---:|---:|---:|---:| | Qwen3-1.7B, rank 32 | 53.8 | **55.2** | 54.8 | 52.1 | | Qwen3-4B, rank 8 | 64.0 | 65.2 | **65.4** | 62.6 | | Llama-3.2-1B, rank 32 | 18.1 | **19.4** | 19.2 | 16.8 | For nested spans and exact unconstrained LS, reconstruction error decreases or stays constant at a fixed state. The best reasoning scores occur at intermediate budgets across these configurations. The spectral result explains reconstruction as the span grows, and development-set performance provides a practical criterion for choosing the training budget.
AuthorsOpenReview ↗

Response to Reviewer nEHC (3/3): Q4, Q5

Continuation of our response to Reviewer nEHC.

Q4. Consistency across sizes, task types, ranks, and simple baselines.

Response.{\color{green}\mathbf{Response.}} Table 2 compares the original selectors across three Qwen students. The supplementary comparisons strengthen random/KL controls with LS and extend the parameterizations:

ConfigurationDense M16Uniform + LSKL + LSCoRD
Qwen3-1.7B, rank 3252.152.553.455.2
Qwen3-4B, rank 862.662.963.765.2
Qwen3-4B, rank 3263.463.564.666.9
Qwen3-4B, rank 6463.864.064.967.0
Qwen3-1.7B, full fine-tuning53.053.253.955.0
Llama-3.2-1B, rank 3216.817.118.019.4

For the main Qwen3-4B setting, gains over the strongest simple LS selector extend across all six tasks:

Method, M16MATH-500AIME24AIME25AMC23MinervaOlympiad
Norm + LS91.746.742.587.556.465.2
CoRD92.749.245.088.558.567.5

CoRD improves all six task scores over the strongest simple LS selector in the main configuration. The broader sweep shows consistent gains over Dense, Uniform + LS, and KL + LS across student sizes, adapter ranks, full fine-tuning, and the Llama family.


Q5. Does the theory bound final loss or reasoning performance?

Response.{\color{green}\mathbf{Response.}} The spectral result bounds fixed-state correction reconstruction. The link to trainable gradients is conditional, as stated in Appendix D.1.6. If gt=Mct+etg_t=M c_t+e_t for a common linear map MM and ∥et∥2≤ϵ\lVert e_t\rVert_2\le\epsilon, then

\le \lVert M\rVert_2\sqrt{\frac1T\sum_{t=1}^T v_t} +\epsilon\bigl(1+\lVert w\rVert_1\bigr), \qquad v_t=\lVert(I-P_B)c_t\rVert_2^2.$$ This explicitly separates uncovered correction energy from mapping mismatch amplified by coefficients. The supplementary 1k diagnostics illustrate why all three matter: | Method | Relative coverage residual | Mean $\lVert w\rVert_1$ | Actual trajectory-gradient error | |---|---:|---:|---:| | Signed mean pursuit | .367 | 2.60 | .405 | | CoRD | **.217** | **1.55** | **.350** | Appendix D.1.5 also establishes local head-loss descent for a single trajectory under a small Euclidean step with fixed hidden states. The guarantees concern fixed-state reconstruction, conditional gradient transfer, and local head descent. Final reasoning quality is evaluated through the controlled training comparisons.
AuthorsOpenReview ↗

Follow-up on our responses to Reviewer nEHC

Thank you again for your time and thoughtful feedback on CoRD. Our rebuttal clarifies the connection between proxy reconstruction and actual LoRA updates, with controlled baselines, gradient diagnostics, broader training comparisons, and detailed cost accounting.

If any questions remain, we would be happy to discuss them and provide further clarification or evidence during the remaining discussion period. If our responses have addressed your concerns, we would be grateful if you could consider reassessing your rating.

Thanks!

Reviewer_zbAFOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new approach for token selection and mean reconstruction by selecting a correction basis revealed by pivoted Cholesky factorization. Then the method uses signed reconstruction weights to preserve the contribution of all tokens within the selected span. The method on matched compute comparisons across multiple training seeds improves the model performance.

Critical strengths

1- The paper has a reasonable motivation and the authors have backed up their contributions with enough theory and mathematical background. 2- Multiple training seeds, useful ablations, experimental case studies and interpretability analysis in section 5.4.1 together form another strength of this paper.

Critical weaknesses

1-My main concern is how much this method adds to existing gradient-coreset and other selective token supervision techniques and if the computation overhead and complexity leads tangible reasoning and over-performance 2- Limited training setup e.g., Qwen model family and only LoRA based finetuning with a very specific configuration (e.g., modules and rank) does not provide enough evidence for the generalizability and effectiveness of the proposed method.

Questions for authors
  1. Compare with an eigenvector or singular vector guided selection baseline.

Assume we use the same reconstruction formula in eq. (5), Could the leading eigenvectors/ singular vectors of the correction kernel in eq. (3) guide token selection (e.g., QR-based)? Comparing reconstruction error, reasoning accuracy, and runtime at the same token budget would provide a better understanding of the role of pivoted Cholesky.

  1. Test a model family other than Qwen.

The experiments cover three student from Qwen3 and Qwen2.5, which is useful, but all teacher-student pairs are from the Qwen family architecture. So, It is hard to say how much the results depend on the model architecture. A pair from another family, such as Llama, would strengthen the paper.

  1. Explain the fine-tuning choices and discuss their effect.

Why use only LoRA, and why choose rank 32 with adapters in the attention and MLP projections? The configuration is specified, but the reasons for these choices are unclear. Since the method uses output-head corrections as proxies for trainable gradients, its effectiveness may depend on the adapter rank and which modules are trained. Comparisons across ranks, attention only vs. attention & MLP adapters, and a full-fine-tuning setting would help show how sensitive the gains are to these training setups and how much grid searching is needed to find the best configuration.

  1. Justify the practical benefit of sparse supervision and a complicated token selection and reconstruction strategies CoRD still scores every candidate position with both models and uses information from all tokens to reconstruct the dense mean. The reconstruction ablation supports its accuracy benefit, so simply dropping similar corrections is not an equivalent alternative. However, table 2 reports only about a 3.7% reduction in total GPU-hours over Dense-RKL for Qwen3-4B at 1,000 updates. I would like a clearer discussion of when the accuracy gains justify the extra selection and reconstruction steps, compared with simpler token-selection and weighting methods.

  2. Extend the budget study beyond Qwen3-4B. Table 10 and Fig. 7 already study the effect of the selection budget. However, these analyses focus on Qwen3-4B. Is 25% retention also a good choice for the other models and different LoRA settings? As Appendix C.5 acknowledges, better reconstruction does not necessarily mean better reasoning performance. Broader budget comparisons not only clarify the effect of budget but also would help readers choose a retention fraction without repeating an extensive tuning.

  3. Explain why the retention fraction is fixed. The same retention fraction causes longer responses produce more selected tokens, but response length may not reflect reconstruction difficulty. Some responses may need more directions than others, and this may change during training. Could an adaptive budget based on residual energy or reconstruction error work better under the same overall budget? Although the authors have already acknowledged geometry-based allocation as a future work, a comparison would help show whether the fixed-fraction rule is sufficient for the task

  4. Clarify how novelty is reported in Fig. 5 Residual novelty changes as the selected correction span grows. Appendix B.6 mentions "recorded residual-novelty percentiles" but I could not tell by looking at the figure only, at which selection step each score is recorded or which span is used. This matters because a selected token has zero residual against the final span, even if its residual was large before selection. Please explain the recording procedure and how scores are compared across tokens before computing their within-response percentiles.

  5. Show how the geometry analysis could guide training and effect the reasoning capability of the model. Section 5.4.2 shows larger gradient-fidelity gains for responses that corrections are more compressible. This helps explain where CoRD works well and where it doesn't, but it does not describe how to use those observations to improve training. The geometry comes from the response and current model state, it is not something a user can choose beforehand. Beyond the budget question above, do these measurements predict which responses contribute more to downstream improvement? An experiment connecting the geometry measurements to reasoning gains would make the analysis more useful.

4 comments
AuthorsOpenReview ↗

Response to Reviewer zbAF (1/3): W1, W2, Q1

Thank you for your time and thoughtful suggestions on the baselines, generalization, and practical value of CoRD. We address each point below with the corresponding comparisons and clarifications.

M16/P8 denote six-task macro mean@16/pass@8 (%). Unless specified otherwise, experiments use Qwen3-8B → Qwen3-4B with rank-32 attention/MLP adapters, 1,000 updates, three seeds, and ρ=0.25\rho=0.25 for sparse methods. Reported ±\pm values are seed standard deviations. Comparisons without uncertainty estimates describe differences in means. Errors are relative errors. Fixed-state diagnostics share Dense-reference checkpoints and trajectories, with 1k as the default checkpoint. Training scores come from each method's own endpoint. Numbered tables and figures refer to the submission.

W1. Contribution beyond gradient coresets and selective supervision, and value relative to complexity.

Response.{\color{green}\mathbf{Response.}} CoRD's central insight is that redundant directions still carry meaningful supervision. Repeated corrections contribute to the mean, and opposing corrections can cancel. CoRD preserves this aggregate contribution through a compact set of complementary directions. CRAIG [1] constructs weighted gradient coresets, and GRAD-MATCH [2] directly pursues gradient matching. CoRD distinguishes the directions needed to represent a response from their contributions to its mean correction. A shared implicit factorization selects these directions and reconstructs their signed contributions on each regenerated response.

To compare the selection rules directly, each selector uses the same signed least-squares reconstruction (LS):

Selector + LSM16P8Training GPUh
Uniform63.579.9391.6
KL-top-mm64.681.0394.0
Correction-norm-top-mm65.081.4400.0
Signed mean pursuit64.981.2447.1
CoRD66.982.9397.3

With reconstruction held fixed, CoRD gains 1.9–2.3 M16 points over norm/KL selection at comparable cost. This comparison demonstrates the value of selecting complementary correction directions for reasoning quality.

[1] Coresets for Data-efficient Training of Machine Learning Models.

[2] GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training.


W2. Evidence is limited to Qwen and one LoRA configuration.

Response.{\color{green}\mathbf{Response.}} The additional experiments cover the Llama family, different adapter ranks and modules, and full fine-tuning. All sparse controls use the same LS objective. Entries below are M16.

Student / trainable parametersDenseUniform + LSKL + LSCoRD
Qwen3-4B / rank 8, attention + MLP62.662.963.765.2
Qwen3-4B / rank 64, attention + MLP63.864.064.967.0
Qwen3-4B / rank 32, attention only62.062.263.064.5
Qwen3-1.7B / full fine-tuning53.053.253.955.0
Llama-3.2-1B-Instruct / rank 3216.817.118.019.4

CoRD improves mathematical reasoning across all five configurations. The consistent gains span low- and high-rank adapters, attention-only training, full fine-tuning, and a second model family.


Q1. Eigenvector/SVD-guided token selection with the same reconstruction.

Response.{\color{green}\mathbf{Response.}} Eigen-QR takes the leading eigenspace UmU_m of KK and selects token indices by pivoted QR of Um⊤U_m^\top, following Q-DEIM [3]. It then applies Eq. (5) in the original correction space. The truncated eigenspace gives a distinct selection rule. Direct pivoted QR of the complete correction matrix is equivalent to Gram-matrix pivoted Cholesky in exact arithmetic with matching tie-breaking.

Selector + LSProxy error eμe_\muLoRA error egtraje_g^{\rm traj}M16 / P8Selection + solve, s/updateTraining GPUh
Eigen-QR.139.35766.7 / 82.718.0423.3
CoRD.144.35066.9 / 82.96.3397.3

At the same token budget, the two methods achieve comparable reasoning quality. Eigen-QR has slightly lower proxy error, and CoRD has slightly lower LoRA-gradient error. CoRD's selection and reconstruction take 6.3 s/update compared with 18.0 s/update for Eigen-QR in these implementations. This supports its practical advantage when quality and runtime are considered together.

[3] A New Selection Operator for the Discrete Empirical Interpolation Method — Improved A Priori Error Bound and Extensions.

AuthorsOpenReview ↗

Response to Reviewer zbAF (2/3): Q2, Q3, Q4, Q5

Continuation of our response to Reviewer zbAF.

Q2. Evaluation on a non-Qwen family.

Response.{\color{green}\mathbf{Response.}} Following your suggestion, the family test uses Llama-3.2-3B-Instruct as teacher and Llama-3.2-1B-Instruct as student. Training uses rank-32 attention/MLP adapters and the same six-task evaluation protocol.

Checkpoint / methodM16P8
Initial student13.024.0
Teacher reference24.039.0
Dense16.829.0
Uniform + LS17.129.4
KL + LS18.030.3
CoRD19.432.0

CoRD improves M16 by 2.6 points over Dense and 1.4 over KL + LS. All checkpoints, including the initial student and teacher, are evaluated with our common decoder and task suite.


Q3. LoRA rank, target modules, full fine-tuning, and tuning effort.

Response.{\color{green}\mathbf{Response.}} LoRA keeps training costs manageable and provides a common setting for the original comparisons. CoRD applies reconstruction weights to the original losses, allowing the same procedure to train different parameter sets. The rank, module, and full-fine-tuning comparisons you suggested give:

ConfigurationDense M16KL + LS M16CoRD M16CoRD egtraje_g^{\rm traj}
4B, rank 8, attention + MLP62.663.765.2.375
4B, rank 32, attention + MLP63.464.666.9.350
4B, rank 64, attention + MLP63.864.967.0.342
4B, rank 32, attention only62.063.064.5.393
1.7B, full fine-tuning53.053.955.0.425

Each supplementary comparison uses two development trials per method, followed by three evaluation seeds. The learning-rate candidates are {10−5,2×10−5}\lbrace 10^{-5},2\times10^{-5}\rbrace for LoRA and {5×10−6,10−5}\lbrace 5\times10^{-6},10^{-5}\rbrace for full fine-tuning. LoRA keeps α/r=2\alpha/r=2 and dropout zero. Rank 64 performs similarly to rank 32, and benefits persist at rank 8 and with attention-only adapters. Full fine-tuning updates all parameters, including embeddings and the head. Together, these results show that CoRD works across the tested parameter sets with a small, equal tuning budget.


Q4. Practical benefit when all positions are scored and runtime savings are modest.

Response.{\color{green}\mathbf{Response.}} Your cost interpretation is correct: ρ=0.25\rho=0.25 controls weighted-loss positions, while both models score all candidates. The useful trade-off is better reasoning at similar total cost.

MethodM16 / P8Training GPUhValidation/test GPUhCombined GPUh
Dense63.4±0.9 / 79.4±1.0412.648.0460.6
Uniform + LS63.5 / 79.9391.647.5439.1
KL + LS64.6 / 81.0394.047.8441.8
Norm + LS65.0 / 81.4400.048.0448.0
CoRD66.9±0.9 / 82.9±0.8397.348.0445.3

Relative to Dense, CoRD gains 3.5 points on both metrics with 3.71% lower training cost, or 3.32% including evaluation. Table 14 explains the runtime balance. Backward time falls from 44.5 to 32.3 s/update, and selection/reconstruction rises from 0.2 to 6.3 s/update. The full cost comparison therefore favors CoRD when the goal is stronger reasoning within a similar training budget.


Q5. Budget sensitivity across models and LoRA settings.

Response.{\color{green}\mathbf{Response.}} The extended budget study shows a broad range of useful intermediate budgets across model sizes, families, and adapter ranks. The following M16 scores extend Table 10:

Configurationρ=.10\rho=.10ρ=.25\rho=.25ρ=.50\rho=.50ρ=1\rho=1Development-selected ρ\rho
Qwen3-4B, rank 3265.566.966.263.4.25
Qwen3-1.7B, rank 3253.855.254.852.1.25
Qwen3-4B, rank 864.065.265.462.6.50
Llama-3.2-1B, rank 3218.119.419.216.8.25

A 25% budget works well in three configurations, and rank-8 Qwen3-4B benefits slightly from 50%. A simple development-set rule chooses the smallest scanned budget within 0.3 points of the best development M8. It selects .25/.25/.50/.25, respectively. For each new configuration, the sweep evaluates two additional budgets (.10 and .50) and reuses the .25 and Dense endpoints. This provides a practical budget-selection procedure with a small search.

AuthorsOpenReview ↗

Response to Reviewer zbAF (3/3): Q6, Q7, Q8

Continuation of our response to Reviewer zbAF.

Q6. Adaptive allocation under the same overall token budget.

Response.{\color{green}\mathbf{Response.}} Residual-based allocation improves the mean score under the shared pivot budget you proposed. For each seed, all methods use the same feasible absolute pivot quota at every update and the same prompt order. Each method generates its own responses. Reconstruction keeps the original response-length normalization. The shuffled control permutes allocations within length-matched groups.

AllocationPer-response retention P10/P50/P90 (%)Common-cache eμe_\muM16 / P8Training GPUh
Fixed .2524.0 / 24.9 / 25.0.14466.9 / 82.9397.3
Residual-adaptive12.0 / 23.5 / 39.0.13567.4 / 83.4402.0
Length-matched allocation shuffle12.1 / 23.6 / 38.9.16866.4 / 82.4401.4

Adaptive allocation gains 0.5 M16 points over fixed retention and 1.0 over shuffled allocation. Its training cost is 1.18% higher than fixed retention. Equal absolute quotas yield different retention percentages because generated lengths differ. The shuffled control shows the value of assigning capacity according to the response's correction geometry.


Q7. Novelty recording time, reference span, and within-response percentiles in Figure 5.

Response.{\color{green}\mathbf{Response.}} You are correct that a selected correction has zero residual against the final span. A supplementary diagnostic makes the recording point explicit. At j0=⌊0.10T⌋j_0=\lfloor0.10T\rfloor, every still-eligible token in a response is scored against the same intermediate span Bj0B_{j_0}. Its score is vt=∥(I−PBj0)ct∥22v_t=\lVert(I-P_{B_{j_0}})c_t\rVert_2^2. Percentiles are computed within these remaining candidates, and the outcome is subsequent selection up to the final 25% budget. The analysis uses responses that reach this budget before numerical rank exhaustion.

Intermediate-residual percentileCandidatesSubsequently selectedSelection rate
0–20%5,4001082.00%
20–40%5,4002705.00%
40–60%5,40054010.00%
60–80%5,4001,24223.00%
80–100%5,4002,34043.33%

The denominator is 27,000 remaining tokens, of which 4,500 are subsequently selected (16.67%). Including 3,000 earlier pivots gives 7,500/30,000 selected overall (25%). This new experiment supplements Figure 5 with a common-span measurement at a specified intermediate step. Higher residual percentiles correspond to higher subsequent selection rates.


Q8. Can correction geometry guide training and predict useful responses?

Response.{\color{green}\mathbf{Response.}} Geometry is useful both for allocating supervision and for predicting response usefulness. First, residual-based allocation improves full-training results under the same absolute pivot budget:

AllocationM16P8
Fixed .2566.982.9
Residual-adaptive67.483.4
Length-matched allocation shuffle66.482.4

Second, the usefulness test evaluates each response at a common checkpoint and matched update norm. It measures the one-step change in teacher-forced reference-solution log probability on disjoint development questions. The predictor uses 192 development and 192 held-out response-prompt identities.

Predictor inputsHeld-out Spearman with local usefulness
Length, initial KL, actual gradient norm.18
Above + dreld_{\rm rel}, χ\chi, coverage residual.30
Above + geometry shuffled within length groups.19

Geometry improves the ranking of locally useful responses beyond length, disagreement, and gradient scale. The allocation experiment connects this information to improved reasoning scores over a full training run.

Thank you for the additional experiments. These largely address my concerns about the spectral baseline, generalization beyond Qwen, fine-tuning choices, token budgets, and the accuracy vs. cost trade-off. The adaptive allocation results are also helpful.

I would still appreciate clarification of how novelty was recorded in the original Fig. 5. The geometry-based prediction also measures local usefulness rather than reasoning gains directly. Overall, I maintain my score of weak acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.