- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents STARFISH, a method for recovering the accuracy of pruned networks using a small set of unlabeled calibration images. The authors establish that existing recovery methods can require substantial retraining or rely on local approximations that become less effective under aggressive pruning. STARFISH caches the dense model’s intermediate representations and updates the surviving weights of the pruned model to match their directions through a cosine-similarity loss applied after every block. The pruning mask remains fixed throughout recovery. The paper also derives a bound connecting final-representation and classification-head errors to output KL divergence. The authors conduct experiments on ImageNet, covering vision transformers and MobileNet under structured, unstructured, and semi-structured pruning. The reported results show substantial accuracy recovery, particularly at high sparsity.
Critical strengths
-
The method is simple and addresses a practical problem: recovering pruned models with limited data and computation. Its use of unlabeled calibration images makes it useful when the original training data are unavailable.
-
The evaluation covers several architectures and pruning settings, with substantial recovery at high sparsity. The ablations on alignment objectives, calibration data, and initialization help explain which choices contribute to the results.
Critical weaknesses
-
The contribution beyond existing feature-based distillation and pruning recovery is not sufficiently established. Intermediate matching, cosine alignment, and pretrained initialization are already established techniques. Comparing pretrained and randomly initialized weights demonstrates the importance of initialization, but does not establish a distinction between STARFISH and distillation.
-
The main comparisons do not fully isolate the benefit of the recovery method. CORP results are taken from the original paper, while STARFISH uses a separately implemented mask-selection procedure. Without matched masks, calibration data, and recovery budgets, it is difficult to determine how much of the improvement comes from STARFISH itself.
-
The theoretical analysis does not directly explain the proposed objective. The bound concerns final-representation and classification-head errors, whereas STARFISH matches the directions of representations across all blocks. Cosine alignment alone does not control representation magnitude, so the connection between the optimized loss and the stated bound remains incomplete.
Questions for authors
-
Q1. What is the main contribution beyond applying existing feature distillation to pruning recovery? Please clarify the additional benefit of cosine alignment and all-block supervision under matched settings. The pretrained-versus-random initialization comparison does not by itself distinguish recovery from distillation, since distillation can also reuse pretrained weights.
-
Q2. The CORP results come from its original paper, while STARFISH uses masks generated by your implementation of CORP’s pruning procedure. Can you evaluate both recovery methods on the same pruned model using the same calibration data? Please also report total recovery time, including feature extraction and caching, so we can assess accuracy gains alongside the required computation.
-
Q3. The KL bound depends on the difference between the final representations, while the cosine loss only aligns their directions. How does minimizing this loss reduce the representation error used in the bound? Please clarify whether the bound explains STARFISH’s specific objective or provides general motivation for representation matching.