acceptodds
Under review as a conference paper at ICLR 2027

Learn from Your Better Self: On-Policy Recovery for Sparse Language Models

Abstract

Sparse attention reduces inference cost but can induce *sparse reasoning drift*, where early token deviations compound through autoregressive feedback, degrading reasoning and prolonging generation. We introduce sparse on‑policy distillation (**SparseOPD**), which uses the original checkpoint's dense execution as a privileged teacher to supervise next‑token distributions on trajectories generated by the evolving sparse student. Across Qwen3‑4B, Qwen3‑14B, and Ministral‑3‑14B, SparseOPD improves mathematical reasoning and code generation over cross‑entropy recovery and fixed‑trajectory distillation. On AIME 2026, it nearly restores dense accuracy on Qwen3‑4B and exceeds it on Qwen3‑14B. It also curbs excessive generation and retains a request‑level speedup over dense inference on Qwen3‑4B HMMT. Our analyses localize on‑policy gains to later sparse states and reveal that recovery signals concentrate at low‑probability token positions. This motivates **LowP‑40**, which supervises only the lowest‑probability of tokens. On Qwen3‑4B, LowP‑40 maintains recovery quality comparable to full‑token supervision while accelerating end‑to‑end training by .

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Sparse Attention, On‑Policy Distillation, Dense‑to‑Sparse Conversion, Sparse Reasoning Drift

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XX3xOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SparseOPD recovers sparse language-model execution by distilling from a frozen dense execution of the same checkpoint. The sparse student generates its own training trajectories as it changes, and compact teacher/student token supports with a residual tail define the distributional targets. LowP further reduces supervision to positions with low unwarped sampled-token probabilities recorded by the rollout student. The paper first diagnoses prefill and decode degradation, then tests recovery across three backbones, including Ministral. For example, OPD-RKL reaches 63.33 on AIME against 63.75 for dense Qwen3-4B. The study also evaluates long-context behavior, compares with GRPO and OPRD, and reports recovery cost and amortization, giving a useful picture of both the learning effect and its computational cost.

Critical strengths

S1. The prefill/decode diagnosis is useful because it separates two sources of sparse degradation. Same-checkpoint comparisons and fixed dense-state/sparse-state replay show where recovery helps. A forward-KL comparison with the target held fixed further supports supervising on sparse-student states.

S2. On Qwen3-4B, OPD-RKL reaches 63.33 on AIME, close to dense execution's 63.75. Recovery is also tested on two other backbones, including Ministral, and the paper reports full recovery cost and amortization, which helps assess whether the recovered quality is useful in practice.

Critical weaknesses

W1. I cannot tell how much continual rollout refresh adds to the reported recovery. The same-target FKL result supports supervision on sparse-student states, and the appendix documents the collection schedule. What is missing is a trained control whose trajectories all come from the initial sparse student. That control would still receive sparse-state supervision, so it would distinguish the value of continually updating the trajectories from the value of using sparse trajectories at all.

W2. LowP-40 reduces recovery time from 22.70 to 19.04 hours, with accuracy differences of −0.41, +0.38 and −0.57 points. Supervising fewer positions already reduces the amount of work. The logit-gradient proxies explain why low-probability positions might be useful. There is no trained random-40% control to tell whether this ranking preserves quality better than an equally small unranked subset.

Questions for authors

Q1. Does refresh improve recovery over trajectories collected entirely from the initial sparse student? Please use one backbone with the same prompt schedule, distributional targets and optimizer-update budget. For both runs, report total recovery GPU-hours, including teacher scoring and rollout collection, to show whether any gain is worth the added cost.

Q2. How does LowP-40 compare with a trained random-40% baseline at the same objective and supervised fraction? Quality, recovery time and available variation across training runs on one backbone would be enough to judge the value of the ranking.

3 comments

Thank you for identifying the controls that isolate continual refresh and LowP ranking. We completed both on Qwen3-4B with the same prompt schedule, FKL targets, sparse backend, and 200-step update budget.

W1 & Q1 — Value and cost of continual refresh

Response: Continual refresh improves recovery beyond supervision on initial sparse-policy states. In the new fixed-policy control, all trajectories are generated by the initial sparse student; the learner still receives dense-teacher FKL supervision on those trajectories. The refreshed run updates its collection policy every 20 steps.

AIME/HMMT report Mean@8 and LCB reports Pass@1. Time is total recovery wall-clock on eight RTX 6000D GPUs, including rollout collection, teacher scoring, and training.

Collection policyAIMEHMMTLCBTime (h)
Fixed initial sparse61.2535.6135.4317.58
Refreshed sparse62.0835.9840.5722.70

Refresh improves AIME/HMMT by 0.83/0.38 pp and LCB by 5.14 pp, at 5.12 additional recovery hours. Refreshing supplies supervision on states induced by the updated policy and delivers a clear gain on code generation.

The refresh-frequency ablation provides further evidence (for details, please see our response to Reviewer AMET, Q2). With the same 200 updates, refreshing sparse-student trajectories every H=5/20/50 steps yields AIME scores of 64.58/62.08/62.50 and HMMT scores of 37.12/35.98/35.61. The most frequent refresh achieves the highest score on both tasks. Together, these results support maintaining on-policy state coverage as the sparse student evolves.


W2 & Q2 — LowP ranking versus random selection

Response: LowP improves performance recovery at the same supervised fraction. Random-40 uniformly samples 40% of positions within each trajectory; LowP-40 selects the 40% with the lowest unwarped sampled-token probabilities. Both retain the OPD-FKL objective and refresh schedule. The added Random-40 control uses one training seed. Metrics and timing follow the table above.

SelectionAIMEHMMTLCBTime (h)
Random-4060.4234.4740.0021.97
LowP-4061.6736.3640.0019.04

LowP gains 1.25 pp on AIME and 1.89 pp on HMMT while matching LCB accuracy. At an equal supervision fraction, this comparison directly measures the value of ranking. The result complements the paper's gradient analysis: the lowest-probability 40% of positions capture 99.3% of the FKL logit-gradient energy. LowP therefore concentrates supervision on informative positions while retaining the computational benefit of selective supervision.


We appreciate your review once again and hope our responses adequately address your concerns.

If any further questions remain, please do not hesitate to raise them.

Thank you for adding both requested controls. The fixed-initial-policy comparison addresses my concern about continual refresh.

LowP-40 improves the reported math scores over Random-40 and matches its code accuracy. This supports the ranking, although one Random-40 training seed leaves the reliability of these gains uncertain.

The added evidence strengthens my assessment. I keep my rating at 3.

Thank you for your follow-up and for recognizing the value of the added controls. We will include both the fixed-initial-policy comparison and the LowP-40 versus Random-40 comparison in the forthcoming revised manuscript. We appreciate your constructive feedback, which has helped strengthen our experimental evaluation.

Reviewer_AMETOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies post-conversion recovery training for language models whose attention layers are switched from dense execution to block-sparse execution (Block-Top-k and Block-Top-p, Section 2.1 and Appendix C.1 Eq. 15). In Section 2.2 (Table 1, Figures 1–2), the authors separate sparse prefill (SD) from sparse decode (DS) on Qwen3-4B with identical weights and uncover sparse reasoning drift: while sparse prefill mainly hurts long-context retrieval (RULER drops from 83.01 to 60.26), sparse decode severely degrades multi-step reasoning (HMMT Feb. 2026 Mean@4 drops from 56.94% to 40.28%) and causes length-cap failures (Hit-cap jumps from 8.33% to 51.39%) as small token discrepancies at high-entropy states compound autoregressively.

To counter this drift, the paper proposes SparseOPD (Section 3, Figure 3, Eqs. 2–5), which freezes the original dense model pdensep _ {\mathrm{dense}} as a privileged teacher—sharing the exact same vocabulary and initialization—and trains the sparse student psparseθp _ {\mathrm{sparse}}^\theta on trajectories rolled out by the sparse student itself (refreshed every 20 optimizer steps across 10 rounds, Section 4.1 and Appendix Table 15) using a top-64 union-plus-tail approximation of forward or reverse KL (OPD-FKL / OPD-RKL, Eqs. 2–3 and Eq. 8). They also introduce a token-selective variant, LowP-40 (Section 4.4, Table 3, Appendix C.4 Eqs. 16–17), which supervises only the 40% lowest-probability student tokens per sequence (capturing 99.3% of the FKL logit-gradient energy, Figure 6a). Across Qwen3-4B, Qwen3-14B, and Ministral-3-14B evaluated on AIME 2026, HMMT Feb. 2026, LiveCodeBench v6, RULER, and LongBench (Table 2, Appendix Table 12), SparseOPD substantially outperforms one-hot CE recovery and generally improves over fixed dense-state distillation (FKL), while preserving sparse decode speedups (up to 3.47×3.47\times per-token at 32K and 1.73×1.73\times end-to-end on HMMT, Figure 4).

Critical strengths

1. Sharp phase-level and gradient-level diagnosis of why sparse reasoning models fail under standard recovery.
Table 1 (DD, SD, DS, SS phase attribution) and the trajectory/gradient analyses in Sections 2.2–2.3 make a compelling case that long-chain reasoning degradation under sparse attention is fundamentally an autoregressive decode-state distribution-shift problem (with first divergences concentrating 93.3%–100% in top-20% high-entropy states, Appendix Table 4) compounded by one-hot CE gradient conflict (Appendix B.3). This diagnosis directly justifies both key axes of the method: dense distribution targets (instead of one-hot CE) and on-policy sparse student rollouts (instead of fixed dense prefixes).

2. Clean, self-contained recovery framework validated across three backbones, two sparse rules, and thorough component ablations.
Using the frozen dense checkpoint as a privileged teacher requires no external larger model or reward verifier, works across Block-Top-k and Block-Top-p on Qwen3-4B, Qwen3-14B, and Ministral-3-14B (Table 2), and is supported by clean 2×22\times 2 factor ablations (CE, FKL, OPD-CE, OPD-FKL, OPD-RKL in Appendix Table 10) plus comparisons against GRPO and hidden-state OPRD (Appendix Table 11, where OPD-FKL cuts LiveCodeBench Hit-cap from 46.29% under OPRD down to 5.71%). The LowP-40 variant (Table 3, Figure 6) is a nice practical bonus, cutting 60% of supervised token positions and speeding up training by 1.19×1.19\times (22.70 h to 19.04 h) with essentially no loss in accuracy.

Critical weaknesses

1. Lack of training-seed variance or sampling confidence intervals on 30–33 problem benchmarks where OPD-FKL vs. FKL margins are small.
While the gains over Sparse Ref. and one-hot CE in Table 2 are massive and unmistakable, the incremental gain of on-policy OPD-FKL over fixed dense-state FKL is sometimes modest: e.g., on Qwen3-4B (Block-Top-k), HMMT Mean@8 is 35.98% vs. 35.23% (+0.75 pp), LiveCodeBench Pass@1 is 40.57% vs. 41.14% (-0.57 pp), and in Appendix Table 12 (LongBench Avg.), FKL (43.91), OPD-FKL (44.34), and OPD-RKL (43.79) are within half a point. Because AIME 2026 (30 problems) and HMMT Feb. 2026 (33 problems) have small problem counts, single-run point estimates without multi-seed standard deviations or problem-level paired bootstrap CIs make it hard to separate statistically significant OPD vs. FKL gains from sampling noise.

2. Missing direct comparison against sparse-attention-specific alignment/recovery baselines.
In Section 1 and Appendix A.2 (p. 13), the paper discusses specialized sparse-attention recovery methods such as SeerAttention (Gao et al., 2024), DuoAttention (Xiao et al., 2025), DeepSeek Sparse Attention indexer alignment (Liu et al., 2025), and SSA (Shen et al., 2025), which align sparse/dense attention weights, gates, or layer outputs. While Appendix B.4.1 (Table 11, Appendix C.6 Eq. 21) includes OPRD (generic layerwise hidden-state MSE on the last 2,000 tokens), it does not compare SparseOPD against a sparse-attention-specific alignment objective (e.g., attention-output MSE or sparse block-selection/gate distillation, or combining attention alignment with FKL) under the same 200-step budget and sparse backend.

Questions for authors
  1. Clarification affecting assessment (Statistical uncertainty for OPD vs. FKL): Can you provide problem-level paired bootstrap 95% confidence intervals for the FKL vs. OPD-FKL and OPD-RKL comparisons in Table 2 across all three models (using the existing 30×830 \times 8 AIME and 33×833 \times 8 HMMT generations), and ideally report 3-seed training mean ±\pm std for the Qwen3-4B setting?

  2. Clarification affecting assessment (Sensitivity to Top-64 union-tail KL and 20-step refresh interval): On Qwen3-4B, how sensitive is OPD-FKL to (a) the top-KK union-plus-tail vocabulary truncation size (e.g., K=16,64,128K = 16, 64, 128 or exact full-vocab KL during online training), and (b) the trajectory refresh period R=5,20,50R = 5, 20, 50 steps (given the 20-step within-round staleness in Appendix Table 15)?

  3. Clarification affecting assessment (Comparison or complementarity with attention-level alignment): In Appendix Table 11, you compare against generic last-2,000-token hidden-state OPRD (Eq. 21). How does SparseOPD compare against (or combine with) a sparse-attention-specific recovery baseline from Appendix A.2—such as layerwise dense-to-sparse attention-output alignment (SSA-style) under the same 200-step budget?

1 comment

Thank you for the concrete suggestions. We now report multi-seed results, paired confidence intervals, sensitivity ablations, and an SSA-style comparison.

W1 & Q1 — Statistical reliability

Response: Three Qwen3-4B training seeds give the following Mean@8 (%) results; ± denotes sample standard deviation across training seeds.

MethodAIMEHMMT
FKL59.31 ± 0.9633.08 ± 2.09
OPD-FKL63.33 ± 1.1037.37 ± 1.53

The mean improvements are +4.03 pp on AIME (95% CI [1.39, 6.94]) and +4.29 pp on HMMT ([1.77, 7.20]). Both intervals are above zero.

For the original Table 2 runs, we also computed 20,000 problem-level paired bootstrap resamples, using the same sampled problems for both methods and retaining all eight generations per problem. Below are OPD − FKL differences and 95% CIs, in percentage points.

Model / taskOPD-FKL − FKL [95% CI]OPD-RKL − FKL [95% CI]
Qwen3-4B / AIME+1.67 [0.42, 3.33]+2.92 [0.83, 5.83]
Qwen3-4B / HMMT+0.76 [−1.14, 3.03]+1.52 [0.38, 3.03]
Qwen3-14B / AIME+1.67 [−1.75, 7.50]+3.33 [0.83, 6.67]
Qwen3-14B / HMMT+2.65 [−0.27, 7.95]+5.68 [1.52, 10.61]
Ministral / AIME+4.58 [0.83, 9.17]+7.50 [1.67, 14.17]
Ministral / HMMT+10.23 [3.03, 18.56]+4.92 [1.14, 9.47]

All six RKL comparisons have positive intervals. The added three-seed results also show positive average OPD-FKL gains over FKL on Qwen3-4B, complementing the original single-run estimates.


Q2 — Vocabulary support and refresh interval

Response: A compact vocabulary support already enables strong recovery. All ablations below use Qwen3-4B, OPD-FKL, and 200 optimizer steps. K is the top-K size for each distribution before union and a residual-tail bucket; H is the refresh interval in steps. Scores are Mean@8 (%).

KHAIMEHMMT
322061.6737.50
642062.0835.98
1282063.7538.26
Full vocab2062.5036.36
64564.5837.12
645062.5035.61

Expanding K gives limited additional gains. Thus a compact support captures an effective recovery signal. This is consistent with the findings of Li et al. [1].

More frequent refresh gives the highest scores on both math tasks: H=5 reaches 64.58/37.12, versus 62.08/35.98 at H=20 and 62.50/35.61 at H=50. This supports the importance of keeping supervision aligned with the evolving sparse policy.


W2 & Q3 — Direct comparison with attention-output alignment

Response: We trained SSA-style recovery on Qwen3-4B using the same sparse backend and 200-step budget, learning rate 1e-5, and SSA's default alignment weight α=10. It combines dense/sparse dual-stream CE with layerwise bidirectional attention-output SmoothL1 over valid response positions. AIME/HMMT use Mean@8; LCB uses Pass@1 (%).

MethodAIMEHMMTLCB
SSA57.9231.0636.00
OPD-FKL62.0835.9840.57
OPD-RKL63.3336.7443.43

Both OPD variants outperform SSA-style recovery on all three quality metrics. SSA-style recovery still yields high hit-cap rates on AIME/HMMT/LCB (23.75%/38.26%/38.86%), substantially higher across all three benchmarks than both OPD-FKL (12.92%/25.38%/5.71%) and OPD-RKL (10.42%/22.35%/9.14%). SSA aligns intermediate attention outputs on fixed sequences; SparseOPD supervises next-token distributions on current sparse-student states. This comparison highlights the value of on-policy supervision for recovering both task performance and termination behavior.


Reference

[1] Li et al. (2026). Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe.


We appreciate your review once again and hope our responses adequately address your concerns.

If any further questions remain, please do not hesitate to raise them.

Reviewer_sf4iOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper considers the problem of improving models using sparse attention. To do so they propose using the dense attention model as a teacher to the sparse attention counterpart. To do so they consider using a KL divergence between both of the generated distributions. The experimental results demonstrate the effectiveness of the approach.

Critical strengths
  • The paper proposes a very simple approach to improve the performance of sparse attention models
  • The paper is easy to read and understand
  • Empirical results show that for Qwen family of models the performance of the distilled sparse attention model can be at a similar level to its dense counterpart. While this is not the case for Ministral 3-14B, suggesting that not all models are suited for sparse attention.
Critical weaknesses
  • I will admit that I am less familiar with the existing works, so unless this is a first of its kind it is unlikely that there is much novelty. The idea of using the KL divergence for a teacher model, seems like the most straight forward approach. If this is indeed the case, then it would also be helpful to have some comparisons to other divergences. This would help understand if there is anything fundamental to this approach.
Questions for authors
  • Why do we think that different models keep more of the performance when using sparse attention?
  • Can the authors confirm that there is no existing work on distillation of dense attention models into sparse ones?
  • When should one use forward/backward KL? Or would the guideline be to try both, and then decide?
3 comments

Thank you for recognizing the clarity, simplicity, and effectiveness of SparseOPD.

W1 & Q2 — The fundamental difference from existing sparse recovery

Response: SparseOPD makes sparse-attention recovery on-policy: the evolving sparse student generates its own training prefixes, and dense execution of the same pretrained model supplies next-token distributional supervision on those prefixes. This directly targets sparse reasoning drift: a token changed by sparse decoding alters the subsequent reasoning states that require recovery.

The distinction is both where and what we supervise. SeerAttention transfers dense attention information to a gate; DuoAttention learns head assignments through representation matching; SSA aligns dense/sparse attention outputs. Their supervision operates on fixed training sequences. SparseOPD instead brings dense feedback to the states visited by the current sparse policy and directly trains its next-token behavior.

The paper supports this design with phase attribution (Table 1) and a same-target comparison: holding FKL fixed, OPD-FKL reduces teacher-student KL by 14.0% in the final quarter of held-out sparse trajectories, while remaining closely matched to FKL on dense trajectories (Section 5.1). Our added 200-step SSA-style comparison further favors OPD-FKL on AIME/HMMT/LiveCodeBench: 62.08/35.98/40.57 versus 57.92/31.06/36.00. The full comparison and training setup are reported in our response to Reviewer AMET (W2 & Q3).


Q1 — Why does recovery differ across models?

Response: Recovery reflects both the initial sparsification damage and the task. A shared sparse selection rule can discard information of different importance across models and tasks; autoregressive feedback amplifies these perturbations. On AIME, sparsification costs Ministral-3-14B 20.83 points, versus 7.92 for Qwen3-14B, creating different recovery gaps.

Ministral also recovers strongly: OPD-FKL raises HMMT from 37.12% to 50.38%, approaching dense execution at 53.03%; OPD-RKL reaches 57.14% on LiveCodeBench, above dense execution at 52.57%. Its larger remaining gap is on AIME. These results show model- and task-dependent sparse robustness, with substantial recovery on both model families.


W1 & Q3 — Other divergences and practical guidance

Response: We trained OPD-JSD on Qwen3-4B for 200 steps, with the same Top-64 union-tail support and 20-step refresh. We use Jensen-Shannon divergence (JSD), JSD(pT,pS)=12KL(pT∥m)+12KL(pS∥m)\mathrm{JSD}(p_T,p_S)=\frac{1}{2}\mathrm{KL}(p_T\|m)+\frac{1}{2}\mathrm{KL}(p_S\|m), where m=(pT+pS)/2m=(p_T+p_S)/2 and pTp_T and pSp_S are the teacher and student next-token distributions. AIME/HMMT report Mean@8 and LiveCodeBench (LCB) reports Pass@1; all scores are percentages.

ObjectiveAIMEHMMTLCB
OPD-FKL62.0835.9840.57
OPD-RKL63.3336.7443.43
OPD-JSD63.3339.7740.57

SparseOPD’s central contribution is dense distributional supervision on states visited by the evolving sparse student. FKL, RKL, and JSD provide different ways to instantiate this supervision. OPD-JSD retains strong recovery and achieves the highest HMMT score among the three objectives, showing that the framework’s effectiveness extends beyond a particular divergence.

We recommend trying both FKL and RKL and choosing empirically for the target model and task; the same principle applies to divergence selection more broadly. Compare candidates on a development set under the same recovery budget, selecting by quality and termination behavior. In our paper, RKL outperforms FKL on all three tasks for both Qwen backbones. On Ministral, FKL leads on HMMT, while RKL leads on AIME and LCB.


We appreciate your review once again and hope our responses adequately address your concerns.

If any further questions remain, please do not hesitate to raise them.

Dear authors,

Thanks for claryfing my questions, the understanding of the existing work does help to clarify the contributions done in the work, and the extra experiments and guidelines are helpful. I keep my inclination towards acceptance of the work

Thank you for your thoughtful follow-up and positive assessment. We are glad that the clarifications, additional experiments, and practical guidelines addressed your questions and helped clarify our contributions. We appreciate your constructive feedback and the time you devoted to reviewing our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.