- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes TROPICS, a post-training, data-free, one-shot, structured compression method for ReLU based simple networks. Each neuron or filter is mapped to a “zonotope generator which scales its input weights and bias by the norm of its output weights. Generators are standardized, L2-normalized and clustered with K-means. Each cluster is merged by averaging input weights and summing output weights, followed by two scale factors that restore variance (inputs) and L2 energy (outputs) which were lost in the merging process. An optional second phase magnitude-prunes the weakest merged units, and batch-norm statistics are merged with the law of total variance. The paper provides a theoretical error bound, based on the Hausdorff-distance between the original and compressed layer zonotopes, and reports that this bound correlates with measured accuracy drop (using both Pearson and Spearman metrics). Experiments cover relatively small models, such as VGG16-BN on CIFAR-10/100 and ResNet18 on ImageNet, with comparisons to TropNNC, CUP, Model Folding and three classic filter-pruning methods. The results show ~85% parameter reduction on VGG16-BN/CIFAR-10 with under 1% drop and no fine-tuning, and wins over other methods on ResNet18/ImageNet.
Critical strengths
S1. A Practical and desireable settings which is: Data-free, fine-tuning-free, with one-pass method which is simple to implement. Anonymous code is provided.
S2. The reporting is responsible and mostly careful. Many tables and graphs report mean ± std/error-bars over five K-means seeds, give parameters and FLOPs together, and include both scaled and unscaled variants. The paper also honestly states where it loses (aggressive FLOPs reduction on ImageNet against Model Folding).
S3. The appendix presents a variety of measurements and results to backup the claims of the paper
S4. The scale factors matter, and the ablation shows it. On VGG16-BN/CIFAR-10, the extreme setting (92.5% reduction) retains 71.3% accuracy with scaling versus 40.7% without (Table 2). Similar gaps appear for clustering at 87–88% reduction and on CIFAR-100. This is a clear, reproducible effect, and the explanation (signal decay through deep non-residual stacks) is plausible.
S5. A principled motivation is attempted. A weight-only error bound, derived in the appendix, is more than most data-free heuristics offer.
Critical weaknesses
W1. Limited scope and practical evidence. Architectures are small or old: custom MLP/CNN, VGG16-BN on CIFAR, and ResNet18 on ImageNet only. At most about 35% parameter reduction is shown on ImageNet (8% drop), which is not a strong practical result. VGG specifically is known as a compressible model. The method does not compress residual connections and mostly targets early layers on ResNet, so it does not handle the architectures in common use. No modern CNNs (MobileNet/EfficientNet), transformers or non-ReLU activations are tested, though the conclusion suggests extension. No wall-clock latency or memory measurements are given for the “structured” speedup claims, and FLOPs are reported analytically only. Hardware speedup for irregular per-layer ratios is not guaranteed. There is no comparison with a short fine-tuning budget, which would show how much headroom the data-free setting leaves, and whether TROPICS’ compressed models are a better starting point than baselines’.
W2. The bound is presented as a “reliable estimator” of accuracy drop, but the evidence does not support that. The Hausdorff distance is between layer zonotopes in the weight space. Not on the network output. No result relates this distance to the change in activations or logits. There is no propagation through depth, batch norm, pooling or residual connections, so how per-layer bounds combine into a network-level number is unclear. It should be stated precisely how the bound in Tables 14–17 is aggregated across layers. The correlation claim is cherry-picked and weak. The abstract and contributions quote “peaking at 0.97 and 0.98”. Table 14 shows values from 0.84 to 0.97 for Pearson depending on the setting. Each cell uses only n = 15 hand-chosen configurations from a single network, and the configurations span a narrow range, with many drops below 0.4% (Table 15). A few large points can dominate Pearson’s r. Rank correlation over many near-zero drops is weak evidence. Please report all cells, confidence intervals, and a pooled correlation across configurations. The paper shows the bound increases along with the measured accuracy drop within each network, but that does not make it a validated predictor.
W3. Novelty relative to TropNNC, Misiakos et al., and non-tropical neuron-merging work. TropNNC and Misiakos et al. already use zonotope approximation and Hausdorff distance for compression. The incremental pieces for this work appear to be K-means with output-norm scaling, the two scale factors, a magnitude-pruning phase, and BN merging. The related-work section is two short paragraphs and does not clearly state what is new versus TropNNC, which the paper describes only as having “an iterative optimization phase.” Closely related data-free merging and compensation methods are not discussed. The most obvious is Neuron Merging (Kim et al., 2020), which merges neurons data-free with compensation, and the neuron-merging/model-fusion line more broadly (please verify the exact references). Data-free pruning baselines such as DFPC, or plain L1/L2 filter-norm pruning and FPGM’s data-free variant on the same networks, are also missing as controls.
W4. Hyperparameters are hand-tuned on the test set, and the headline numbers are the best of many. Section 5.1 states that per-layer ratios p and r were set manually (“a manual selection of p and r was set regarding our scope”), with different strategies for parameter versus FLOPs reduction. All results are evaluated on the test sets, with no validation split. Tables 2–3 contain 20+ configurations per model. The method is advertised as “non-uniform” and “single-pass”, but the ratios come, as far as I understand, from trial and error, which is a search cost the comparison does not count. Baselines such as TropNNC and Model Folding appear to use uniform or automatically set ratios, so the comparison is not exactly apples tp apples. Please describe how layer ratios were chosen, give a rule or automatic procedure, and report results using a validation split.
W5. inconsistent reporting A. Abstract: "on VGG16-BN (CIFAR-10): our scaled Mixed strategy achieves an 84.7% parameter reduction with only a 0.44% accuracy drop" Section 5.2: "On VGG16-BN, the extensive compression finds our Mixed strategy achieves only a 0.58% drop at a massive 84.7% parameter reduction" The later is reported also in Table 2.
B. The abstract says scale factors preserve “up to 29% more accuracy” at 92.5% reduction, while Section 5.2 and Table 2 reports “over 30%" and 30.6%, respectively..
C. The abstract says TROPICS “surpasses ... Model Folding” on ResNet18, but Section 5.3 says it outperforms only at mild FLOPs reduction (below about 23%) and loses at aggressive reduction
D. Table 1’s comparison with classic pruning methods uses different baseline networks. The “TROPICS (baseline)” accuracies (94.27, 93.91, 94.02) are higher than the original papers’ baselines (93.25, 93.58, 93.66), so these are not matched models. The text asserts “we can reasonably assume” the competitors do not reach zero drop without retraining. This should be measured or removed. Only the FPGM row is a like-for-like data-free comparison, and there it needs the same network.
Questions for authors
Q1. How are per-layer ratios chosen, and can this be automated? How do results change if they are selected on a validation split? Q2. How is the network-level bound computed from per-layer Hausdorff distances, and is there any theoretical link from layer-wise Hausdorff distance to output error?
Q3. What does the tropical view contribute to the algorithm, beyond the bound, compared with merging neurons by cosine similarity plus norm-restoring scaling? Please include an ablation without standardization and with plain filter clustering. Q4. What exactly is new relative to TropNNC? Can you compare directly with Neuron Merging and data-free norm pruning at matched ratios? Q5. Can you evaluate on a modern CNN (e.g. MobileNetV2 or ResNet50) and measure real latency? Q6. Please fix or explain the discrepancy between the 0.44% (abstract) and 0.58% (Table 2) figures. Q7. Why does the bound increase with scale factors in Table 15 even though accuracy improves? How should this be interpreted?
4 comments
ResNet-50 on ImageNet, with measured latency (Q5). Ratios are set by the same automatic Brent's-method rule, there is no fine-tuning, and results are mean ± std over 3 seeds. Speedup is measured wall-clock GPU time for a full inference pass over the validation set, compared with the uncompressed model:
| FLOPs red. | 5.1% | 11.1% | 16.9% | 20.9% | 24.5% | 30.5% |
|---|---|---|---|---|---|---|
| Param. red. | 2.9% | 6.9% | 10.5% | 14.3% | 16.9% | 20.6% |
| TROPICS, no scale | 75.01 ± 0.63 | 67.75 ± 0.49 | 46.43 ± 0.89 | 19.26 ± 7.41 | 3.15 ± 0.96 | 0.66 ± 0.20 |
| TROPICS, scaled | 75.79 ± 0.56 | 67.68 ± 0.39 | 46.29 ± 3.05 | 28.17 ± 13.65 | 4.20 ± 2.23 | 1.62 ± 0.66 |
| Measured GPU speedup (scaled) | 1.02× | 1.05× | 1.07× | 1.08× | 1.10× | 1.14× |
(Original model: 79.72% top-1, 8.27 GFLOPs, 60.3 s for the GPU inference pass.)
Thank you for the detailed response and the additional experiments. The ablation, the Neuron Merging and norm-pruning comparisons, and the ViT and ResNet-50 runs address several of my concerns. Below I note what is resolved and what remains. Many of the new results appear only in the reply, so I would like to see them integrated into a revised PDF.
Resolved to some extent Q3: The generator/standardization ablation is informative. So the standardization carries much of the gain. Please mention it clearly in the paper, since standardization has no tropical-geometric justification (seeds and STD will be also welcome). Q5: ResNet-50 and ViT results are welcome, as is the acknowledgment of W5 and Q6.
Other concerns: Q5. evaluate on a modern CNN: reported times cover a full validation pass which is typically dominated by data loading rather than GPU computation. Please report model-only GPU latency (batch size, hardware, warmup) so that data loading does not dominate.
Q4. Neuron Merging comparison. In your new table, Neuron Merging matches TROPICS at some points and is even better at other points. TROPICS is clearly ahead only at ≥87.6% and on CIFAR-100. Please revise the abstract and the “substantially outperforms” claim. Because the specific ratios in the table were tuned for TROPICS, please also report each baseline at its own best ratios. Is this row correct or is the data noisy? CIFAR-100 (75.0%) 28.52 12.15 54.18 it seems non-monotone with the other rows. Also, I could not find a direct statement of what is new relative to TropNNC beyond the numerical comparison. A short paragraph in related work on TropNNC, Misiakos et al. and Neuron Merging would help.
Q1. you note that "The 75/25 split fo cluestering/pruning is an empirical choice, fixed across all experiments." How do you choose it. The headline numbers therefore depend on manual search, please say it also in the abstract.
Q2. I appreciate the move to a “per-layer worst-case certificate”, but my questions were not answered. (a) How is the network-level bound in Tables 14–17 aggregated across layers? (b) Please report the pooled correlation across all configurations, with confidence intervals, and compare against a trivial predictor (e.g. fraction of parameters removed, or total removed generator norm). (c) Since scaling can increase the bound while improving accuracy, please state in the abstract and contributions that the bound does not capture the effect of scaling. The revised paper should remove the “data-free performance estimator” claim, not only soften it.
Assessment. The empirical evidence is stronger than at submission. However, the theoretical contribution is now much narrower, the headline results depend on manual configuration, and the ResNet-50 and ImageNet results show limited practical reach. I will potentially raise my score by 1, conditional on a revised PDF that includes these changes.
We sincerely appreciate your detailed review and insightful suggestions, which have helped strengthen our paper. We have revised the manuscript to the best of our ability to address all your comments within the constraints of the rebuttal period.
Thank you once again for your time.
We thank the reviewer for the detailed review.
W4 / Q1 — How per-layer ratios are chosen; automation; test-set tuning
We accept that a manual search over configurations is not a fixed rule, and we have replaced it with one: Given a single global FLOP or parameter target τ, each compressible layer keeps a fraction r_l = clip(1 − α·w_l, 0.05, 1) of its filters. Here w is a fixed, hand-set intensity vector per stage (front-loaded for FLOPs, back-loaded for parameters). Brent's method solves for the unique α at which the exact linear cost model reaches (1 − τ) of the original cost. The 75/25 split fo cluestering/pruning is an empirical choice, fixed across all experiments. Automatic rule, VGG16 on CIFAR-10 vs FPGM, oiginal acc 93.33%
Automatic rule, ResNet-18 on ImageNet (single shared checkpoint at 69.32%, full 50k validation set, 3 seeds), compared with TropNNC and CUP at matched FLOPs:
W3 / Q3 / Q4 — tropical geometry contribution; Neuron Merging, norm pruning, FPGM
**Ablation requested in Q3.
(Top-1 %, VGG16-BN, scale correction on, identical per-layer budgets. Originals: 92.09% on CIFAR-10 and 73.21% on CIFAR-100.) . Neuron Merging (Kim et al., 2020) and data-free norm pruning, at identical per-layer ratios:
(Top-1 %, VGG16-BN, identical per-layer ratios for all three methods.)
W2 / Q2 / Q7 — What the bound is and is not
Bound as accuracy predictors is a promising direction. It seems to have high corellations in some experiments. However, in this work, we present it as a per-layer worst-case certificate on how much each layer's zonotope changes, and we soften the estimator claims. The scale factor restores the norm (energy) of the merged generators. This repairs the typical activation magnitude seen by the next layer, but it can move the zonotope further from the original in the worst-case direction, which is what Hausdorff distance measures.
W1 / Q5 — A modern architecture, non-ReLU activations, latency
We evaluated ViT-B/16 on ImageNet, (GELU, LayerNorm and multi-head attention). The per-layer ratios are set automatically.
How each component is handled: MLP hidden neurons and whole attention heads are compressed. LayerNorm, residual streams and the QKᵀ softmax interaction are kept intact.
Real speedup. Whole neurons and heads are removed, so the pruned model is a smaller dense network and needs no sparse kernels. The evaluation pass becomes up to 1.32× faster end-to-end (44.8% parameter reduction).
W5 / Q6 — Inconsistencies
We apologise for these; all will be fixed.
We thank the reviewer once more. These changes make the paper stronger.