acceptodds
Withdrawn from ICLR 2027

What Does Recovery Recover? An Empirical Study of Prediction Retention after Structured Pruning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

To reduce the environmental impact of artificial intelligence, researchers aim to reduce the computation required for model inference while maintaining task performance. A common approach uses structured pruning to remove network components and recovery training to regain performance in the compressed model. Recovery is commonly evaluated through overall accuracy. However, improvements on some examples can compensate for unresolved pruning damage on others, concealing persistent failures that affect model reliability. To identify these hidden failures, we investigate how damage repair, corrections of original errors, and losses of correct predictions contribute to recovery gains, and whether repaired predictions remain correct during training. Our empirical study connects overall performance, group outcomes, and individual prediction histories, with matched continuation of the original unpruned model as a training reference. The analysis shows that small overall accuracy differences can coexist with larger group differences as corrections and losses offset each other. Prediction histories reveal the contributions of damage repair and other prediction changes, while accounting for continuation losses can change the interpretation of where additional losses concentrate. Repaired predictions can also become incorrect again. These findings characterize recovery quality through the composition of gains, the distribution of losses, and the persistence of repairs. Based on this analysis, we define a continuation-referenced retention diagnostic that complements overall and group accuracy and negative flips relative to the original model. Together, these measures provide criteria for evaluating recovery training and comparing compressed models according to computational savings, predictive performance, and prediction retention.

Primary area:
optimization
Keywords:
structured pruning, recovery training, model compression, prediction retention, negative flips, group performance, matched continuation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_x29zOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates what recovery training actually recovers after structured pruning. The authors point out that evaluating recovery training solely by final overall accuracy may obscure important instance-level changes, because the repair of pruning damage, the correction of errors made by the original model, and the loss of originally correct predictions may offset one another. To address this issue, the paper starts from the same original model and constructs both a post-pruning recovery-training path and a matched continued-training path without pruning. It then analyzes overall accuracy, performance across predefined groups, and prediction changes for individual examples on Waterbirds, CelebA, MetaShift, and CIFAR-100.

Critical strengths
  1. The paper correctly identifies the limitations of considering only final accuracy. Two models with the same accuracy may make correct predictions on entirely different examples, resulting in different reliability and group-level risks.
  2. In addition to comparing the original, pruned, and recovered models, the authors include an unpruned model that undergoes continued training with the same data, number of training steps, and learning-rate schedule. This design helps distinguish changes specific to pruning recovery from those caused by continued training in general.
  3. The main experiments include three network architectures and three datasets with predefined groups, supplemented by experiments on CIFAR-100 using different pruning strategies and pruning ratios. The authors also report results from multiple independently trained original models and repeated training runs.
Critical weaknesses
  1. The methodological novelty is relatively limited. The core analysis is based mainly on existing concepts. Therefore, the contribution is closer to a systematic empirical study than to a new pruning or recovery method.
  2. The proposed RkeepR_{\mathrm{keep}} has not been directly validated. The paper proposes using RkeepR_{\mathrm{keep}} to select recovery checkpoints or pruned models, but it does not actually conduct an ablation or comparative evaluation. At present, the results establish only that the metric is intuitively meaningful, not that it leads to better model-selection outcomes.
  3. The scope of pruning in the main experiments is relatively narrow. The main experiments focus on approximately 10% FLOP reduction and primarily use the authors' group-sensitive channel-pruning method. Although the CIFAR-100 experiments add 10% and 20% pruning levels and random pruning, this is still insufficient to establish whether the conclusions generalize to a wider range of settings.
Questions for authors
  1. Why are the original models trained with GroupDRO, while both recovery and continued training use cross-entropy? Could the authors add a control experiment that uses GroupDRO throughout, to distinguish the effects of the change in training objective from those of pruning recovery?
  2. Does RkeepR_{\mathrm{keep}} actually improve checkpoint selection? I suggest directly comparing selection based on overall accuracy, worst-group accuracy, negative-flip rate, and RkeepR_{\mathrm{keep}}, and evaluating the resulting models on an independent test set.
  3. Do the main conclusions still hold under more aggressive pruning? When the FLOP reduction reaches 30%, 50%, or even higher levels, does the offsetting relationship between error corrections and negative flips change substantially?
  4. How sensitive are the conclusions to the pruning and recovery strategies? If unstructured pruning, learning-rate rewinding, weight rewinding, knowledge distillation, or other recovery methods are used, do the phenomena observed in the paper still hold?
Reviewer_qAYQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the effect of structured pruning in the overall accuracy of neural networks, as well as the effect on groups and individual samples used for validation. The authors conclude that recovering a same accuracy may come with distortions across groups, which can be attributed to individual samples.

Critical strengths

None.

Critical weaknesses

W1) Novelty: the paper discusses changes in prediction across groups and changes in predictions across individual samples due to network pruning. However, there is nothing new here: changes in predictions across groups have already been studied by Tran et al. (2022), who showed something akin to the Mathew Effect from economics ("the rich getting richer") in terms of what groups end up with increased or decreased predictive performance compared to before; and changes in predictions across individual samples have been discussed since Hooker et al. (2019) as Pruning Identified Exemplars (PIEs).

W2) Terminology (and clarity): the paper uses non-standard terminology, which makes it difficult to follow up. For example, "recovery training" is actually fine-tuning; and it has always been fine-tuning within the network pruning literature, regardless of a slightly different use for this term in foundation models more recently. Moreover, if we consider how often the predictions for a given class are correct, the correct term for that is recall. Finally, I still do not understand what the authors mean by "continuation model": is that the model if not pruned and trained for another epoch? If so, does that mean that we are pruning a model that has not converged yet? Those and other word choices, as well as longer terms where a single word would have sufficed, result in a paper that is much longer than it should have been.

W3) Limited evaluation: the entire study is based on imposing the equivalent of a single pruning rate of 10% across all models and datasets. As noted by Liebwen et al. (2021), the amount that you can prune from a model depends on the task for which it is trained. Likewise, and basically a folklore in network pruning at this point, you can certainly prune more from a larger model. Moreover, this narrow focus on a specific pruning rate misses the fact that the distortions that the authors are trying to evaluate may cut both ways: if a model is not pruned too aggressively, fine-tuning may actually correct for differences across groups. This is a finding from Good et al. (2022), not cited by the authors, where differences in recall are evaluated across different pruning rates and compared to variations in accuracy.

Questions for authors

Q1-A) Regarding W1, can you explain in simple terms what is the novelty in comparison to the cited works?

Q1-B) The variation of performance across groups is identified in Tran et al. (2022) as having to do with the sizes of the groups themselves and addressed by increasing the importance of recovery for individual samples in smaller groups. Have you considered group sizes in this study?

Q2) Regarding W2, what is this continuation model? Can you explain in simple terms what you expect by evaluating models O, P, C, D?

Q3) Regarding W3, why fix such a modest amount of pruning? Why not evaluate a broader spectrum of pruning rates?

Reviewer_Pn4nOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper empirically studies prediction changes during recovery training after structured pruning. It compares recovered models with matched continuation of unpruned models, examining overall accuracy, group-level outcomes, and individual prediction histories. The analysis shows that accuracy gains can combine repairs of pruning damage, corrections of original errors, and losses of previously correct predictions, and that repaired predictions may become incorrect again. It also proposes a continuation-referenced retention diagnostic to complement accuracy and negative-flip rate.

Critical strengths

None

Critical weaknesses
  • Limited novelty and empirical insight. The paper combines established concepts—negative flips, group-level performance disparities, and forgetting events—but does not clearly establish a substantive new insight beyond their joint analysis in the pruning–recovery setting. The proposed retention diagnostic is a conditional reformulation of negative-flip rate, and its added value is not demonstrated.

  • Insufficient justification and validation of the experimental setup. The rationale for the chosen model–dataset combinations, group-sensitive pruning, and modest 10% FLOP reduction in the main experiments is insufficiently developed. Moreover, original models use GroupDRO, while recovery and continuation use cross-entropy. Although this transition is shared across both paths, the comparison remains specific to that transition and its interaction with pruning. The complementary CIFAR-100 experiments provide some broader evidence, but do not establish the robustness of the main group-level findings under alternative training objectives and pruning strategies.

  • Weak connection between the proposed diagnostic and recovery. The retention diagnostic does not use the immediately pruned model and therefore cannot distinguish predictions repaired during recovery from those that survived pruning. Its role as a recovery-specific diagnostic, beyond measuring prediction retention after a general model update, requires stronger justification.

  • Sensitivity to evaluation conditions without sufficient explanation. The pattern of simultaneous extra corrections and extra negative flips occurs in eight of nine primary settings, but only five under validation-based checkpoint selection. The paper documents this variation without sufficiently explaining when or why the pattern holds, limiting the broader conclusions that can be drawn.

  • Unverified practical utility. The paper proposes retention-based model selection but leaves its evaluation to future work. Consequently, it does not establish whether the diagnostic leads to better selection decisions than overall accuracy, worst-group accuracy, or existing negative-flip criteria.

Questions for authors
  1. What substantive empirical insight does this study establish beyond existing work on negative flips, compression effects across groups, and forgetting? Specifically, what does the proposed retention diagnostic reveal that existing measures do not?

  2. What motivates the particular combinations of models, datasets, and pruning settings used in the main experiments? How robust are the main findings to these choices and to the checkpoint selection criterion?

  3. Given that the retention diagnostic does not distinguish repaired predictions from those that survived pruning, why is it particularly informative for recovery? Is there evidence that using it for checkpoint selection improves outcomes compared with selection based on accuracy or NFR?

Please refer to the weaknesses above for further details and additional concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.