acceptodds
Under review as a conference paper at ICLR 2027

ReCal: Calibrating Structured Pruning for On-Policy Distillation Recovery

Abstract

Structured pruning reduces the deployment cost of reasoning language models, but the resulting capability degradation can hinder subsequent on-policy distillation (OPD) recovery. Because OPD relies on student-generated trajectories, pruning damage that persists after offline distillation can limit its effectiveness. We propose RECAL, Recovery-Aware Calibration, a simple plug-and-play approach that improves OPD recovery by adjusting calibration before pruning. RECAL uses forward KL between an unpruned teacher and a pruned probe to identify teacher-supported predictions disrupted by pruning, then reweights calibration statistics to guide existing pruning criteria toward preserving these predictions. Across multiple models and pruning methods, RECAL consistently improves mathematical reasoning after OPD, achieving gains of up to 16.7 percentage points on AIME, alongside improvements in most code-generation comparisons. Further analysis shows that RECAL reduces residual damage at heavily affected tokens and establishes performance advantages that persist through recovery. These results demonstrate the value of recovery-aware calibration for improving on-policy distillation recovery of pruned reasoning models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, distillation, on-policy ditillation, pruning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1HHGOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

ReCal proposes a method to recover the model ability after token pruning (structured FFN pruner). Using a single pass probe and measuring difference with the teacher, they propose to give a weight on higher difference tokens.

Critical strengths
  • This paper shows a clear limitation from existing method: pruning makes the inference LLM cheap but the performance drop is severe, whereas SFT / OPD for recovery only gains portion of the original performance. This paper suggests selecting tokens based on what could be recovered.
  • They find out that pruning damage is concentrated to small number of tokens, and after recovery, these tokens becomes the major reason of performance mismatch.
  • Experimental results are strong, showing that ReCal helps performance recovery after structure pruning.
Critical weaknesses
  • The baseline calibrates with a prompt, while ReCal calibrates with weighted teacher responses. There's no "teacher responses + uniform weights" baseline, so we can't tell whether the gain comes from the damage weighting or just from switching to response-token calibration.
  • They didn't compare against closer recovery-aware and reasoning-aware pruners (NIRVANA, DarwinLM, RESP, and ShortOPD as another recovery method), and they didn't compare against stronger reconstruction-based structured pruners (SlimGPT, OSSCAR).
Questions for authors

How sensitive is ReCal to a probe?

Reviewer_2H1UOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReCal uses teacher-probe disagreement to reweight calibration tokens before structured pruning, followed by unchanged SFT and on-policy distillation. It offers a practical way to improve recovered reasoning, but the experiments do not isolate damage-aware weighting or establish a specifically OPD-related recovery advantage.

Critical strengths

S1. The problem is practically relevant, connecting structure selection with the capabilities retained after recovery rather than evaluating pruning only before training.

S2. The method is simple and compatible with several channel-selection criteria. Matched widths and shared recovery settings make the paired comparisons informative.

S3. The fixed-prefix analysis tracks the same damaged tokens across stages, and the experiments report substantial mathematics gains alongside exceptions and task trade-offs.

Critical weaknesses

W1. The central intervention is confounded. Baselines use prompts, whereas ReCal uses teacher responses and damage weights; uniform weighting on the same responses is essential to establish what the proposed weighting adds.

W2. The evidence does not distinguish improved recoverability from better initial preservation. Most advantages already exist after SFT and do not systematically increase during OPD, so the OPD-specific motivation remains only partly tested.

W3. Token disagreement is not yet validated as a recovery-relevant importance signal. Fixed teacher-prefix KL reductions do not establish that the selected tokens improve the student states actually encountered during OPD; alternative weighting controls are needed.

W4. Practical robustness is incompletely quantified. Repeated recovery runs, uncertainty on downstream gains, and the extra calibration cost are not reported; fixed update counts also permit different rollout-token budgets.

Questions for authors

See Weaknesses

2 comments

Thank you to the authors for their patient and detailed response. The substantial additional experiments have addressed my concerns, and the results have already been incorporated into the revised paper. I am therefore raising my score to a positive rating. Good luck!

Dear Reviewer 2H1U,

Thank you so much for your positive response and for your kind wishes. We greatly appreciate your time and constructive feedback, which have helped us improve the paper significantly. We are delighted that the revisions addressed your concerns.

Best regards,

Authors of Paper 42888

Reviewer_fHrVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

ReCal presents a calibration-based pruning method for language models that takes into account damaged tokens to reweight calibration statistics, guiding which channels to prune or retain from the language model. The technique is plug-and-play, and composes with multiple pruning baselines such as Minitron, Wanda, and FLAP. ReCal is evaluated in multiple settings. ReCal composed with Wanda is compared to multiple baselines and is shown to improve student performance across multiple benchmarks, before recovery and after SFT and OPD. ReCal is demonstrated to be plug-and-play by showing improvements over the baseline when composed with multiple pruning methods and math and coding tasks. ReCal is also shown to preserve and improve compression / accuracy ratios during distillation across math benchmarks. The authors discuss limitations and future work, such as the regression on instruction following tasks, and the extension beyond FFN channels.

I also want to thank the authors for producing a well-written and clear paper, with an introduction that provides motivation and background, and a results section with a clear progression of research questions. I enjoyed reading this submission, and it was the clearest paper I have reviewed this round, by far. Thank you.

Critical strengths
  • The paper is well written and easy to follow. The results section is rich, with the research questions clearly stated and addressed.
  • The paper demonstrates that the technique is broadly effective in multiple settings (different pruning baselines and benchmarks) whilst acknowledging the limitations around instruction-following.
Critical weaknesses
  • Only one model family is used, so it raises questions about how broadly applicable the technique is. The comparison with multiple pruning approaches addresses some of the concerns.
  • The regression on instruction following is notable. I would not block acceptance of the work based on strictly improving on every benchmark, however some insight or indication of why the gap occurs or how to address it, would make the research more insightful and useful.
Questions for authors

Without reselecting any tokens, this group accounts for 64.6% of within-response damage after SFT and 69.3% after OPD

Can I think of these Q5 tokens as being harder to fix? In that they take a larger share of error after recovery?

Can you elaborate more on the regression seen in instruction following tasks, or speculate why this happens? Is this because instruction following is cross domain and the pruning loses more general knowledge rather than reasoning that could be identified using a small set of calibration prompts?

Other approaches to damage-aware compression are discussed in the related work. However I am not left with a clear understanding about their practical trade-offs, or whether they are complementary. A direct comparison would be the most helpful, if applicable, but otherwise some discussion about relative strengths would help situate this research.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.