acceptodds
Withdrawn from ICLR 2027

STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pruning is a process designed to reduce the number of weights in a large neural network. This can substantially speed up inference but might cause a considerable reduction in the model's accuracy, and thus it is usually followed by a healing process that regains some of the lost accuracy. In this paper, we propose a new healing method, *STARFISH*, that can recover (most of) the accuracy of any pruned network efficiently. The main idea of STARFISH is to optimize the pruned network to align with the original network's internal state representations using a tiny calibration set of unlabeled examples. For the common case of removing of the weights, STARFISH healing improves the recovered accuracy by up to over the state-of-the-art methods on ViT-based networks. Its advantage is even more pronounced under aggressive pruning. For example, after eliminating of the weights in a DeiT-B network for ImageNet, STARFISH uses only of the number of training images as a calibration set and recovers of the original dense accuracy, whereas competing recovery techniques reach only of the dense model accuracy. The paper includes numerous ablations demonstrating the advantages of STARFISH over other common healing and distillation methods. Open-source code will be made available upon acceptance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
neural network pruning, model compression, post-training pruning, post-pruning recovery, accuracy recovery, representation alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sCBROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents STARFISH, a method for recovering the accuracy of pruned networks using a small set of unlabeled calibration images. The authors establish that existing recovery methods can require substantial retraining or rely on local approximations that become less effective under aggressive pruning. STARFISH caches the dense model’s intermediate representations and updates the surviving weights of the pruned model to match their directions through a cosine-similarity loss applied after every block. The pruning mask remains fixed throughout recovery. The paper also derives a bound connecting final-representation and classification-head errors to output KL divergence. The authors conduct experiments on ImageNet, covering vision transformers and MobileNet under structured, unstructured, and semi-structured pruning. The reported results show substantial accuracy recovery, particularly at high sparsity.

Critical strengths
  • The method is simple and addresses a practical problem: recovering pruned models with limited data and computation. Its use of unlabeled calibration images makes it useful when the original training data are unavailable.

  • The evaluation covers several architectures and pruning settings, with substantial recovery at high sparsity. The ablations on alignment objectives, calibration data, and initialization help explain which choices contribute to the results.

Critical weaknesses
  • The contribution beyond existing feature-based distillation and pruning recovery is not sufficiently established. Intermediate matching, cosine alignment, and pretrained initialization are already established techniques. Comparing pretrained and randomly initialized weights demonstrates the importance of initialization, but does not establish a distinction between STARFISH and distillation.

  • The main comparisons do not fully isolate the benefit of the recovery method. CORP results are taken from the original paper, while STARFISH uses a separately implemented mask-selection procedure. Without matched masks, calibration data, and recovery budgets, it is difficult to determine how much of the improvement comes from STARFISH itself.

  • The theoretical analysis does not directly explain the proposed objective. The bound concerns final-representation and classification-head errors, whereas STARFISH matches the directions of representations across all blocks. Cosine alignment alone does not control representation magnitude, so the connection between the optimized loss and the stated bound remains incomplete.

Questions for authors
  • Q1. What is the main contribution beyond applying existing feature distillation to pruning recovery? Please clarify the additional benefit of cosine alignment and all-block supervision under matched settings. The pretrained-versus-random initialization comparison does not by itself distinguish recovery from distillation, since distillation can also reuse pretrained weights.

  • Q2. The CORP results come from its original paper, while STARFISH uses masks generated by your implementation of CORP’s pruning procedure. Can you evaluate both recovery methods on the same pruned model using the same calibration data? Please also report total recovery time, including feature extraction and caching, so we can assess accuracy gains alongside the required computation.

  • Q3. The KL bound depends on the difference between the final representations, while the cosine loss only aligns their directions. How does minimizing this loss reduce the representation error used in the bound? Please clarify whether the bound explains STARFISH’s specific objective or provides general motivation for representation matching.

Reviewer_b1QgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

STARFISH recovers accuracy after pruning by fixing the pruning mask and optimizing surviving weights to align intermediate block outputs with cached dense-model representations. It uses an average cosine-alignment objective and a small unlabeled calibration set. Experiments evaluate ImageNet classification across DeiT/ViT architectures and MobileNetV1, covering structured, unstructured, and 2:4 sparsity. The reported results show substantial recovery under severe pruning and improvements over the evaluated baselines. Supporting studies examine recovery objectives, alignment depth, initialization, calibration data, and runtime. A theoretical bound relates representation and classification-head errors to predictive KL divergence.

Critical strengths
  1. Substantial recovery under severe pruning. At 0.85 structured sparsity within the QK+MLP scope, DeiT3-H improves from 5.88% to 78.24% top-1 accuracy, retaining 92.08% of dense accuracy. Recovering useful performance from such a degraded starting point is a consequential empirical result.

  2. Simple recovery procedure supported by informative ablations. STARFISH uses unlabeled calibration inputs, preserves the pruning mask, and adds no inference modules. Comparisons involving output losses, MSE alignment, individual-block supervision, and alternative calibration conditions support the usefulness of the approach across the evaluated settings.

Critical weaknesses
  1. The comparative accuracy–efficiency advantage is not sufficiently established under controlled conditions. CORP [1] accuracies are imported while its mask-selection procedure is reimplemented, so an identical-mask comparison is not demonstrated; CORP calibration budgets are also omitted from the comparison tables. CORP results are missing at 0.8/0.85 sparsity in Table 12. For MobileNetV1, Table 10 reports different dense baselines for STARFISH and SNOWS [2], and p. 19 explicitly describes matched ImageNet-mini evaluation only for Table 2, leaving the MobileNet comparison’s evaluation protocol unclear. Finally, Table 9 provides STARFISH runtimes without matched baseline timings or feature-cache memory accounting. These limitations make it difficult to isolate recovery quality and assess comparative efficiency.

  2. The incremental benefit over feature-based recovery needs stronger isolation. The reinitialization experiment establishes the importance of pretrained weights under the tested budget, but does not distinguish STARFISH from distillation initialized with pruned pretrained weights. Table 4 provides a useful MSE baseline, with cosine alignment improving accuracy by 0.19–0.69 points at sparsities 0.5–0.7 and 2.72 points at 0.8. However, comparable tuning for this baseline is not explicitly documented. Although results are averaged over three seeds, across-seed standard deviations or confidence intervals for accuracy are not reported. It discloses tuning for Table 1 and Figure 9, the selection criterion and general default weight decay remain unspecified. A clearer distinction from related few-sample feature recovery, including FSKD [3], would strengthen the novelty assessment.

Questions for authors
  1. Were STARFISH and SNOWS [2] evaluated on identical images and checkpoints for MobileNetV1? Please reconcile Table 10 with the evaluation description on p. 19. Can CORP [1] and STARFISH be compared using identical masks and reported calibration budgets, including the highest structured sparsities?

2.What dataset and criterion selected the best hyperparameters, and what is the default weight decay? Were MSE and single-block alternatives tuned comparably? Please clarify whether configuration differences explain 73.66% in Table 1 versus 73.28% in the appendix, and report accuracy variability across seeds.

3.Which stages are included in Table 9’s runtimes, and what are the cache precision, storage location, and peak memory requirements? How do the methods compare in accuracy versus total recovery time on the same hardware?

  1. Since changing calibration size at fixed epochs also changes optimizer steps, how does the recovery curve change under a fixed-update budget? The repeated-image experiment is informative, but a controlled comparison would better separate sample diversity from optimization effort.

References:

[1] Boxiang Zhang and Baijian Yang. CORP: Closed-Form One-Shot Representation-Preserving Structured Pruning for Vision Transformers. arXiv preprint arXiv:2602.05243, 2026.

[2] Ryan Lucas and Rahul Mazumder. Preserving Deep Representations in One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework (SNOWS). arXiv preprint arXiv:2411.18376, 2024.

[3] Tianhong Li, Jianguo Li, Zhuang Liu, and Changshui Zhang. Few Sample Knowledge Distillation for Efficient Network Compression (FSKD). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14639–14647, 2020.

Reviewer_3EYUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tries to diminish the performance drop between models before and after pruning. A representation alignment based method is proposed to do so. Experiments show the proposed STARFISH outperforms baseline methods like CORP and SNOW.

Critical strengths
  1. This draft has a strong motivation that AI model compression is getting more important nowadays.
  2. Using representation alignment between the complete model and its pruned counterpart is a wise angle to investigate.
  3. Overall, the draft is easy to follow and understand.
Critical weaknesses
  1. More baseline methods are needed. I understand that this paper directly follow these two baselines SNOW and CORP, but pruning is a general topic, more methods following other technical families are necessary to compare with, which helps to strength the statement of this paper.
  2. Paper format should be improved. For example, Tab2 is almost placed by the end of the draft, while its corresponding content is in experiment section. The font size if figures should be polished such as fig3 and fig5, which makes these figures hard to read.
  3. That will be great if more architecture or backbone can be added under the proposed method like language model or diffusion model, which will further support the proposed method as a general pruning strategy.
Questions for authors

I am voting for a borderline for this draft, please check the above section for details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.