- Rating
- 4(+2)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HOPE, a mathematical framework for progressively deconstructing representations in trained neural networks through functional capacity reduction. HOPE models neurons in a Hilbert space and uses functional distortion to unify pruning, neuron merging, and block eviction. For networks with BatchNorm, it constructs a surrogate distribution from stored BN statistics, enabling data-free evaluation. The paper further presents proof-of-concept applications in model compression and cross-domain fine-tuning through DEFT.
Critical strengths
- The functional-space perspective provides an interesting way to address scaling and parameterization ambiguities of magnitude-based criteria.
- The unified treatment of pruning, merging, and block eviction is conceptually appealing.
- Using BN statistics for data-free functional evaluation is an interesting idea.
- The paper contains substantial theoretical development, and the proof-of-concept results are promising.
Critical weaknesses
My main concern is that the empirical evaluation does not sufficiently validate the paper’s central motivation of deconstructing learned representations.
-
The experiments mainly demonstrate downstream utility rather than representation deconstruction. The compression results suggest that HOPE provides a useful importance criterion, while DEFT shows a promising stability-plasticity trade-off. However, these results do not directly establish what representational structure is uncovered by progressive encoding, whether such structure is stable across models/tasks, or whether it provides insights beyond a useful functional importance ranking.
-
The core/slack interpretation needs stronger controlled validation. DEFT evaluates the complete pipeline, including the HOPE partition, freezing, structural masking, and merging, and therefore does not isolate the quality of the HOPE-derived partition itself. I would like to see comparisons against random, magnitude/BN-based, and reversed-HOPE (freezing neurons that HOPE considers less important while allowing high-importance neurons to adapt) partitions under the same DEFT pipeline. If the proposed core/slack interpretation is meaningful, this reversed partition should perform substantially worse.
-
The “universal” terminology appears stronger than the current evidence. The paper uses terms such as “universal feature space” and “universal core of invariants,” while the current experiments do not establish universality across diverse tasks or domains. I suggest using more conservative terminology such as compression-resistant, source-critical, or high-capacity representations unless stronger cross-task evidence is provided.
Questions for authors
- Is the representation structure identified by HOPE stable across random seeds, checkpoints, or related tasks?
- How well do HOPE’s analytical importance/similarity estimates correlate with estimates computed from real activations?
6 comments
Part 1
Part 1/2: Learned structure, stability, and surrogate accuracy (W1, Q1, Q2)
1. Structure beyond a scalar importance ranking (W1). A scalar per-neuron score only orders single-unit deletions; it cannot separate redundant pairs from dead units or weigh filter pruning against dropping a whole residual block. With denoting the remaining parameter fraction, HOPE on ResNet-50 exhibits three regimes without per-layer thresholds. First, for , it prunes weak filters mainly in the widest final residual stage (2048 channels, 80.2% kept at ), while 97.7% and 99.4% survive in the first two stages. Second, HOPE merges correlated active filters rather than deleting one outright (180 merges in 3,610 steps in our submitted run). Subtracting each unit's analytical mean before measuring alignment (paired with BN running-mean updates) cuts merges from 180 to 67, showing that over 60% of raw post-ReLU merges stem from shared positive bias rather than feature covariance. Third, below , HOPE evicts residual blocks whose branches add little beyond their skip paths, dropping 1, 2, and 3 blocks in the 2048-channel stage by and (16 over the full run). On zero-shot ImageNet compression, block eviction lifts Top-1 accuracy over neuron-only pruning by 4.3 points at (45.28% vs. 41.02%) and 6.9 points at (31.11% vs. 24.24%, or 31.29% with merging).
A similar pattern holds in the 20 CIFAR-100 8-layer CNNs (Appendix I.2): at a 70.6% global core budget (), DEFT freezes %, %, and % of filters in the 4th, 5th, and 6th convolutional layers, compared with % in the 512-unit dense layer, leaving nearly half of the dense bottleneck plastic for target adaptation.
2. Stability across checkpoints, seeds, and related tasks (Q1). We tracked stability during CIFAR-100 training over 3 seeds trained for 100 epochs (% test accuracy). Against epoch 100, checkpoints at epochs 60, 70, 80, and 90 yield within-layer Spearman and on initial pruning scores, and frozen-core Jaccard overlap and (90.0% to 99.8% agreement) on the full progressive partitions.
To test stability across 3 random initializations (2,000 test images across all 7 hidden layers), we use Linear CKA since neuron order permutes across runs. HOPE's frozen core (70.6% of neurons) preserves 99.8% of full-layer cross-seed CKA: mean CKA is versus for the full layer, ahead of a size-matched random core () in every layer. By contrast, the slack subset (29.4% of neurons) has mean CKA of only ( and in the 4th and 5th convolutional layers) and contains nearly all dead post-ReLU units (% active in core vs. % in slack). Across all pairs of 20-class CIFAR-100 tasks, the 7-layer core-fraction profiles correlate at and .
3. Analytical estimates vs. real activations (Q2). To check how closely the Gaussian surrogate tracks real data, we measured empirical activations on 2,000 test images per model across the 20 CIFAR-100 CNNs (7 hidden layers 20 models = 140 layers) and on 1,024 ImageNet validation images across all 32 bottleneck layers of ResNet-50 (7,552 neurons, 1.72M pairs). The table compares our closed-form expressions for post-ReLU RMS , capacity , and inner products (Appendix G.3.2) against empirical values via mean SD of within-layer Pearson and Spearman across layers.
| Within-Layer Quantity | CNN Pearson | CNN Spearman | ResNet-50 Pearson | ResNet-50 Spearman |
|---|---|---|---|---|
| Post-ReLU RMS | ||||
| Neuron capacity | ||||
| Hilbert inner product |
Pooling all 7,552 ResNet-50 neurons across layers gives () for post-ReLU RMS and () for . On 336 single-step compression actions on ResNet-50, analytical distortion also matches the empirical change at the next layer's pre-BN input with within-layer and .
Part 2
Part 2/2: Controlled DEFT ablation and terminology (W2, W3)
4. Isolating the HOPE partition in DEFT (W2). To isolate the HOPE partition from weight merging, structural masking, and budget differences, we hold the entire DEFT pipeline fixed on all 20 CIFAR-100 SVHN tasks. Every partition rule starts from the same unmerged source weights , freezes 70.6% of neurons as core and trains 29.4% as slack (, binary ), applies the same structural masking rule , and trains for 30 epochs with SGD at learning rate .
We also clarify a discrepancy between Section 9.2 and Appendix J / Algorithm 3: in Table 1, fine-tuning started from unmerged (as in Section 9.2) but marked one child of each merged pair as slack (), so severed that child's weights to the core without folding them into a parent. Keeping merged pairs frozen in the core (), using mean-centered capacities, and selecting from the Appendix I.5 grid (rather than from Table 1's 1,000-sample pre-sweep) raises CIFAR-100 source retention from 52.14% to 61.18%, which remains exact across fine-tuning epochs by Theorem J.2. We will align Appendix J and Algorithm 3 with Section 9.2 accordingly. The table below reports mean SD across the 20 tasks (where unmasked Head-Only FT at fixed learning rate has % source accuracy on these 20 checkpoints, vs. % in Table 1's earlier run), along with paired -test -values on H-Score relative to HOPE.
| Partition Rule | SVHN Target (%) | CIFAR-100 Source (%) | H-Score (%) | Paired -test (Wins) |
|---|---|---|---|---|
| Head-Only FT (100% frozen, no mask ) | N/A | |||
| HOPE | N/A | |||
| LAMP (Lee et al., 2021) | (20/20) | |||
| L1-Joint (effective weights) | (20/20) | |||
| BN-Scale | (20/20) | |||
| Random (layer-matched) | (20/20) | |||
| Random (global) | (20/20) | |||
| Reversed-HOPE (layer-matched) | (20/20) | |||
| Reversed-HOPE (global) | (20/20) |
Inverting the HOPE ranking in Reversed-HOPE so that low-cost neurons are frozen and high-cost neurons are adapted reduces CIFAR-100 source accuracy to exact 20-class chance (%) on every task, both globally and with layer-matched budgets. Layer-matched random selection drops source retention by 51.3 points, from 61.18% to 9.85%. Under the exact same masking rule and trainable budget, HOPE retains 3.5 points more source accuracy than LAMP, 6.3 points more than L1-Joint, and 9.1 points more than BN-Scale; it wins all 20 tasks in every comparison (two-sided Wilcoxon ).
5. Replacing "universal" terminology (W3). We agree that phrases such as "universal feature space" and "universal core of invariants" overstate what our experiments establish. In the revision, we will replace them throughout Sections 1, 2, and 9.2 and Appendices I through K with "high-capacity core" or simply "core neurons".
Thanks
Thank you for your careful and constructive feedback. We respond to your review in 2 parts.