- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studied the structured compression for vision models without access to the original training data or fine-tuning. The authors takes Information Preservation Priority (IPP) to select retained channels using the maximum of their incoming and outgoing squared weight norms. Optimal Information Migration (OIM) constructs descriptors from structurally coupled parameters, solves an optimal transport problem between the original channels and the retained subset, and adds weighted downstream contributions from removed channels to retained ones. A subsequent DeepInversion-based procedure, DI-BN, recalibrates BatchNorm statistics using 256 synthetic images.
Critical strengths
- The research question is interesting.
- The experiments cover several relevant architectures and datasets.
- The downstream update is concrete and interpretable. Equation (6) clearly identifies which weights are changed and which are preserved. Using dependency groups to coordinate structural removal is appropriate, and Figure 1 helps explain the procedure.
Critical weaknesses
- The contribution relative to the closest prior work is not sufficiently established. There are some highly relative works like Model Folding[2], IFM[1]. The comparison to these methods is missing.
- Similarly, the DI-BN procedure described in Section 3.3 closely follows the published Fold-DIR procedure in Model Folding paper[2]: synthesize inputs using the dense model’s BN statistics, then update the compressed model’s BN statistics without weight fine-tuning. The submission cites Model Folding, but does not explain what, if anything, is technically new in DI-BN.
- The absence of Direct + DI-BN leaves a central experimental question unanswered. Direct versus OIM provides useful evidence for migration when the dense model’s BN statistics are retained. OIM versus OIM + DI-BN then measures the benefit of recalibration for the migrated model. However, this does not show how much OIM helps when direct pruning receives the same recalibration.
- One thing I’m confusing is that the transport objective is not connected closely enough to the functional preservation claimed for the update.
For example, consider a local compression step with fixed hidden activations , followed by a linear layer with outgoing weights . Equation (6) gives the local preactivation discrepancy
\left( h_p(x)-\sum_j M_{pj}h_{q_j}(x) \right).$$ Thus, functional preservation depends on how well the retained activations reconstruct each deleted activation, weighted by its downstream contribution. The OT objective instead uses distances between normalized, concatenated parameter descriptors. The submission provides neither a bound relating these quantities nor a quantitative evaluation of this reconstruction error. 5.The paper does not adequately specify how a global compression budget is converted into per-layer or per-dependency-group retained widths. It also leaves unclear the group-processing order, whether descriptors and scores are recomputed after earlier updates, and the precise parameter slices included in each descriptor. [1] Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory https://arxiv.org/abs/2310.06756 [2] Forget the Data and Fine-Tuning! Just Fold the Network to Compress https://arxiv.org/abs/2502.10216Questions for authors
1.What is the precise methodological advance over Intra-Fusion’s OT-based aggregation and Fold-DIR’s statistics repair? Can the benefit of downstream-only writeback be demonstrated under matched compression and calibration settings?
2.How does Direct + DI-BN compare with OIM + DI-BN when both use exactly the same synthetic images and recalibration protocol? This is the most important missing control for interpreting the final pipeline.