- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
ReCal proposes a method to recover the model ability after token pruning (structured FFN pruner). Using a single pass probe and measuring difference with the teacher, they propose to give a weight on higher difference tokens.
Critical strengths
- This paper shows a clear limitation from existing method: pruning makes the inference LLM cheap but the performance drop is severe, whereas SFT / OPD for recovery only gains portion of the original performance. This paper suggests selecting tokens based on what could be recovered.
- They find out that pruning damage is concentrated to small number of tokens, and after recovery, these tokens becomes the major reason of performance mismatch.
- Experimental results are strong, showing that ReCal helps performance recovery after structure pruning.
Critical weaknesses
- The baseline calibrates with a prompt, while ReCal calibrates with weighted teacher responses. There's no "teacher responses + uniform weights" baseline, so we can't tell whether the gain comes from the damage weighting or just from switching to response-token calibration.
- They didn't compare against closer recovery-aware and reasoning-aware pruners (NIRVANA, DarwinLM, RESP, and ShortOPD as another recovery method), and they didn't compare against stronger reconstruction-based structured pruners (SlimGPT, OSSCAR).
Questions for authors
How sensitive is ReCal to a probe?
Thank you to the authors for their patient and detailed response. The substantial additional experiments have addressed my concerns, and the results have already been incorporated into the revised paper. I am therefore raising my score to a positive rating. Good luck!