acceptodds
Under review as a conference paper at ICLR 2027

Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networks

Abstract

Deep neural networks encode complex representations, but deconstructing this internal knowledge remains a challenge. Given the link between learning and compression, network compression offers a promising lens to analyze this knowledge. However, standard compression heuristics often suffer from scale symmetries and architectural biases. To resolve these, we introduce Hilbert Operator for Progressive Encoding (HOPE), a mathematical framework to gradually deconstruct the representations in trained network weights. HOPE shifts network compression from the discrete domain into a Hilbert space of continuous functions. By modeling individual neurons as rank-1 Hilbert-Schmidt operators, HOPE unifies pruning and neuron merging as low-rank subspace projection. Extending this formulation, HOPE introduces macro block eviction to encompass multi-layer structures like entire residual pathways under the same unified metric. This unified approach enables unbiased architectural decisions across layers with different types and sizes. HOPE is a data-free and hyperparameter-free framework. We present proof-of-concept experiments in model compression and fine-tuning to highlight the practical potential of our theory.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Deconstruction, Progressive Compression, Model Compression, Transfer Learning

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.50 over 2 reviews, above 99% of the others.
Reviewer_ziccOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HOPE, a mathematical framework for progressively deconstructing representations in trained neural networks through functional capacity reduction. HOPE models neurons in a Hilbert space and uses functional distortion to unify pruning, neuron merging, and block eviction. For networks with BatchNorm, it constructs a surrogate distribution from stored BN statistics, enabling data-free evaluation. The paper further presents proof-of-concept applications in model compression and cross-domain fine-tuning through DEFT.

Critical strengths
  1. The functional-space perspective provides an interesting way to address scaling and parameterization ambiguities of magnitude-based criteria.
  2. The unified treatment of pruning, merging, and block eviction is conceptually appealing.
  3. Using BN statistics for data-free functional evaluation is an interesting idea.
  4. The paper contains substantial theoretical development, and the proof-of-concept results are promising.
Critical weaknesses

My main concern is that the empirical evaluation does not sufficiently validate the paper’s central motivation of deconstructing learned representations.

  1. The experiments mainly demonstrate downstream utility rather than representation deconstruction. The compression results suggest that HOPE provides a useful importance criterion, while DEFT shows a promising stability-plasticity trade-off. However, these results do not directly establish what representational structure is uncovered by progressive encoding, whether such structure is stable across models/tasks, or whether it provides insights beyond a useful functional importance ranking.

  2. The core/slack interpretation needs stronger controlled validation. DEFT evaluates the complete pipeline, including the HOPE partition, freezing, structural masking, and merging, and therefore does not isolate the quality of the HOPE-derived partition itself. I would like to see comparisons against random, magnitude/BN-based, and reversed-HOPE (freezing neurons that HOPE considers less important while allowing high-importance neurons to adapt) partitions under the same DEFT pipeline. If the proposed core/slack interpretation is meaningful, this reversed partition should perform substantially worse.

  3. The “universal” terminology appears stronger than the current evidence. The paper uses terms such as “universal feature space” and “universal core of invariants,” while the current experiments do not establish universality across diverse tasks or domains. I suggest using more conservative terminology such as compression-resistant, source-critical, or high-capacity representations unless stronger cross-task evidence is provided.

Questions for authors
  1. Is the representation structure identified by HOPE stable across random seeds, checkpoints, or related tasks?
  2. How well do HOPE’s analytical importance/similarity estimates correlate with estimates computed from real activations?
6 comments

Thank you for your careful and constructive feedback. We respond to your review in 2 parts.

Part 1/2: Learned structure, stability, and surrogate accuracy (W1, Q1, Q2)

1. Structure beyond a scalar importance ranking (W1). A scalar per-neuron score only orders single-unit deletions; it cannot separate redundant pairs from dead units or weigh filter pruning against dropping a whole residual block. With d=P/P0d = P / P_0 denoting the remaining parameter fraction, HOPE on ResNet-50 exhibits three regimes without per-layer thresholds. First, for d∈[0.85,1.0]d \in [0.85, 1.0], it prunes weak filters mainly in the widest final residual stage (2048 channels, 80.2% kept at d=0.85d=0.85), while 97.7% and 99.4% survive in the first two stages. Second, HOPE merges correlated active filters rather than deleting one outright (180 merges in 3,610 steps in our submitted run). Subtracting each unit's analytical mean E[ReLU(yi)]\mathbb{E}[\text{ReLU}(y_i)] before measuring alignment (paired with BN running-mean updates) cuts merges from 180 to 67, showing that over 60% of raw post-ReLU merges stem from shared positive bias rather than feature covariance. Third, below d≈0.84d \approx 0.84, HOPE evicts residual blocks whose branches F(X)F(X) add little beyond their skip paths, dropping 1, 2, and 3 blocks in the 2048-channel stage by d=0.70,0.60,d = 0.70, 0.60, and 0.500.50 (16 over the full run). On zero-shot ImageNet compression, block eviction lifts Top-1 accuracy over neuron-only pruning by 4.3 points at d=0.50d=0.50 (45.28% vs. 41.02%) and 6.9 points at d=0.40d=0.40 (31.11% vs. 24.24%, or 31.29% with merging).

A similar pattern holds in the 20 CIFAR-100 8-layer CNNs (Appendix I.2): at a 70.6% global core budget (P=30P=30), DEFT freezes 73.4±2.173.4 \pm 2.1%, 77.1±3.377.1 \pm 3.3%, and 85.5±1.985.5 \pm 1.9% of filters in the 4th, 5th, and 6th convolutional layers, compared with 51.2±3.551.2 \pm 3.5% in the 512-unit dense layer, leaving nearly half of the dense bottleneck plastic for target adaptation.

2. Stability across checkpoints, seeds, and related tasks (Q1). We tracked stability during CIFAR-100 training over 3 seeds trained for 100 epochs (61.6±1.061.6 \pm 1.0% test accuracy). Against epoch 100, checkpoints at epochs 60, 70, 80, and 90 yield within-layer Spearman ρ=0.890,0.978,0.999,\rho = 0.890, 0.978, 0.999, and 0.99990.9999 on initial pruning scores, and frozen-core Jaccard overlap 0.868,0.929,0.980,0.868, 0.929, 0.980, and 0.9970.997 (90.0% to 99.8% agreement) on the full progressive partitions.

To test stability across 3 random initializations (2,000 test images across all 7 hidden layers), we use Linear CKA since neuron order permutes across runs. HOPE's frozen core (70.6% of neurons) preserves 99.8% of full-layer cross-seed CKA: mean CKA is 0.7030.703 versus 0.7050.705 for the full layer, ahead of a size-matched random core (0.6350.635) in every layer. By contrast, the slack subset (29.4% of neurons) has mean CKA of only 0.2740.274 (0.0360.036 and 0.0750.075 in the 4th and 5th convolutional layers) and contains nearly all dead post-ReLU units (99.8±0.299.8 \pm 0.2% active in core vs. 56.6±3.656.6 \pm 3.6% in slack). Across all (202)=190\binom{20}{2} = 190 pairs of 20-class CIFAR-100 tasks, the 7-layer core-fraction profiles correlate at r=0.874±0.099r = 0.874 \pm 0.099 and ρ=0.838±0.135\rho = 0.838 \pm 0.135.

3. Analytical estimates vs. real activations (Q2). To check how closely the Gaussian surrogate tracks real data, we measured empirical activations on 2,000 test images per model across the 20 CIFAR-100 CNNs (7 hidden layers ×\times 20 models = 140 layers) and on 1,024 ImageNet validation images across all 32 bottleneck layers of ResNet-50 (7,552 neurons, 1.72M pairs). The table compares our closed-form expressions for post-ReLU RMS K(i,i)\sqrt{K(i,i)}, capacity ∥fi∥H\Vert f_i\Vert _ {\mathcal{H}}, and inner products ⟨fi,fj⟩H\langle f_i, f_j \rangle _ {\mathcal{H}} (Appendix G.3.2) against empirical values via mean ±\pm SD of within-layer Pearson rr and Spearman ρ\rho across layers.

Within-Layer QuantityCNN Pearson rrCNN Spearman ρ\rhoResNet-50 Pearson rrResNet-50 Spearman ρ\rho
Post-ReLU RMS0.965±0.0250.965 \pm 0.0250.964±0.0300.964 \pm 0.0300.801±0.1440.801 \pm 0.1440.668±0.1820.668 \pm 0.182
Neuron capacity0.978±0.0220.978 \pm 0.0220.973±0.0280.973 \pm 0.0280.912±0.0760.912 \pm 0.0760.805±0.1390.805 \pm 0.139
Hilbert inner product0.924±0.0630.924 \pm 0.0630.980±0.0130.980 \pm 0.0130.938±0.0860.938 \pm 0.0860.983±0.0090.983 \pm 0.009

Pooling all 7,552 ResNet-50 neurons across layers gives r=0.941r = 0.941 (ρ=0.829\rho = 0.829) for post-ReLU RMS and r=0.973r = 0.973 (ρ=0.861\rho = 0.861) for ∥fi∥H\Vert f_i\Vert _ {\mathcal{H}}. On 336 single-step compression actions on ResNet-50, analytical distortion DkD_k also matches the empirical L2L_2 change at the next layer's pre-BN input with within-layer r=0.973±0.015r = 0.973 \pm 0.015 and ρ=0.877±0.138\rho = 0.877 \pm 0.138.

Part 2/2: Controlled DEFT ablation and terminology (W2, W3)

4. Isolating the HOPE partition in DEFT (W2). To isolate the HOPE partition from weight merging, structural masking, and budget differences, we hold the entire DEFT pipeline fixed on all 20 CIFAR-100 →\to SVHN tasks. Every partition rule starts from the same unmerged source weights Wsource\mathbf{W} _ {\text{source}}, freezes 70.6% of neurons as core and trains 29.4% as slack (P=30P=30, binary Ei∈{0,1}E_i \in \lbrace 0, 1 \rbrace), applies the same structural masking rule W0=M(E)⊙Wsource\mathbf{W} _ 0 = \mathbf{M}(\mathbf{E}) \odot \mathbf{W} _ {\text{source}}, and trains for 30 epochs with SGD at learning rate 0.0050.005.

We also clarify a discrepancy between Section 9.2 and Appendix J / Algorithm 3: in Table 1, fine-tuning started from unmerged Wsource\mathbf{W} _ {\text{source}} (as in Section 9.2) but marked one child of each merged pair as slack (Ej=1E_j = 1), so M\mathbf{M} severed that child's weights to the core without folding them into a parent. Keeping merged pairs frozen in the core (Ei=Ej=0E_i = E_j = 0), using mean-centered capacities, and selecting P=30P=30 from the Appendix I.5 grid (rather than P=40P=40 from Table 1's 1,000-sample pre-sweep) raises CIFAR-100 source retention from 52.14% to 61.18%, which remains exact across fine-tuning epochs by Theorem J.2. We will align Appendix J and Algorithm 3 with Section 9.2 accordingly. The table below reports mean ±\pm SD across the 20 tasks (where unmasked Head-Only FT at fixed learning rate 0.0050.005 has 62.74±3.9362.74 \pm 3.93% source accuracy on these 20 checkpoints, vs. 63.13±4.6263.13 \pm 4.62% in Table 1's earlier run), along with paired tt-test pp-values on H-Score relative to HOPE.

Partition RuleSVHN Target (%)CIFAR-100 Source (%)H-Score (%)Paired tt-test pp (Wins)
Head-Only FT (100% frozen, no mask M\mathbf{M})36.01±3.1436.01 \pm 3.1462.74±3.9362.74 \pm 3.9345.63±2.5345.63 \pm 2.53N/A
HOPE89.77±0.4189.77 \pm 0.4161.18±4.1661.18 \pm 4.1672.69±2.86\mathbf{72.69 \pm 2.86}N/A
LAMP (Lee et al., 2021)89.77±0.5189.77 \pm 0.5157.69±5.4157.69 \pm 5.4170.10±4.0070.10 \pm 4.007.78×10−77.78 \times 10^{-7} (20/20)
L1-Joint (effective weights)89.61±0.4889.61 \pm 0.4854.91±6.4454.91 \pm 6.4467.88±4.8867.88 \pm 4.889.30×10−89.30 \times 10^{-8} (20/20)
BN-Scale89.79±0.5789.79 \pm 0.5752.08±7.5852.08 \pm 7.5865.61±6.0065.61 \pm 6.007.66×10−87.66 \times 10^{-8} (20/20)
Random (layer-matched)92.22±0.4292.22 \pm 0.429.85±3.919.85 \pm 3.9117.58±6.1017.58 \pm 6.101.30×10−181.30 \times 10^{-18} (20/20)
Random (global)92.26±0.2692.26 \pm 0.2610.15±3.9310.15 \pm 3.9318.06±6.1418.06 \pm 6.141.61×10−201.61 \times 10^{-20} (20/20)
Reversed-HOPE (layer-matched)92.68±0.3592.68 \pm 0.355.00±0.005.00 \pm 0.009.49±0.009.49 \pm 0.003.05×10−273.05 \times 10^{-27} (20/20)
Reversed-HOPE (global)92.71±0.2992.71 \pm 0.295.00±0.005.00 \pm 0.009.49±0.009.49 \pm 0.003.05×10−273.05 \times 10^{-27} (20/20)

Inverting the HOPE ranking in Reversed-HOPE so that low-cost neurons are frozen and high-cost neurons are adapted reduces CIFAR-100 source accuracy to exact 20-class chance (5.00±0.005.00 \pm 0.00%) on every task, both globally and with layer-matched budgets. Layer-matched random selection drops source retention by 51.3 points, from 61.18% to 9.85%. Under the exact same masking rule and trainable budget, HOPE retains 3.5 points more source accuracy than LAMP, 6.3 points more than L1-Joint, and 9.1 points more than BN-Scale; it wins all 20 tasks in every comparison (two-sided Wilcoxon p=1.91×10−6p = 1.91 \times 10^{-6}).

5. Replacing "universal" terminology (W3). We agree that phrases such as "universal feature space" and "universal core of invariants" overstate what our experiments establish. In the revision, we will replace them throughout Sections 1, 2, and 9.2 and Appendices I through K with "high-capacity core" or simply "core neurons".

Reviewer_67wMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a mathematical framework, Hilbert Operator for Progressive Encoding (HOPE), that models a range of weight perturbations in a unified way as low rank subspace projections. These perturbations cover neuron pruning, neuron merging but also larger structural modification such as block eviction. Using this unified framework, the authors derive distortion costs for each of these perturbations that allows balancing compression and distortion under a given budget in a principled way. This approach is data free by leveraging a surrogate calculated from the BN statistics. Furthermore, the paper also introduces Dispersed Elastic Fine-Tuning (DEFT), using HOPE, for transfer learning, by separating what may amount to the core of a model and what is its periphery.

Critical strengths
  1. This paper provides an interesting approach, considering a neuron as a rank-1 operator and leveraging this to provide a unified comparable costs pruning, merging, and block eviction, allowing for a principled trade off between them when compressing under a given budget. The closed-form BN-derived kernels make this practical.

  2. The data free setting is an important and valuable for practical cases where calibration data is not available.

Critical weaknesses
  1. In Section 6, for the cost derivation of merging, the cost seems to evaluate a different operation than merging. Indeed, the equation at line 352 seems to aim that each parent reproduce closely each of neuron i and j. However, a merging operation should aim to get close to the output, and thus I would have believed the correct cost to be ∥fi+fj−fp∥\lVert f_i + f_j - f_p \rVert instead of ∥fi−fp∥2+∥fj−fp∥2\sqrt{\lVert f_i - f_p \rVert^2 + \lVert f_j - f_p \rVert^2}. This may have impact on the balancing as it misevaluates the distortion of the perturbation operated. Thus, it impacts the core contribution, but I believe it is repairable.

  2. The empirical results are thin and the baselines old. For compression, a single figure on one model, that evaluates accuracy with respect to neuron density, with no information about FLOPS or parameters count. For DEFT, it would have been useful to have ablations on the selection criteria by replacing HOPE criteria with other selection criteria, like magnitude based.

Questions for authors
  1. Can you provide experiments on how well correlated your analytical estimation of cost are with the realized activation errors and output distortion? Could you also report the share of each perturbaton chosen by the optimizer (pruning, merging and block eviction), and maybe an ablation by resricting only or two kind of perturbations to assess the relevancy of the unifying framework?

  2. The paper would benefit from stronger baselines to evaluate better what the authors' approach contributes to. Could you provide stronger baselines? For example LAMP (Lee et al. 2021) cited by the paper that also address global scale symmetry, data free structured methods, approaches using synthetic images derivation as a reference for a similar data free approach that leverage BN statistics and data-driven approaches as an upper bound.

8 comments

Thank you for your careful and constructive feedback. We respond to your review in 4 parts covering W1, W2, Q1, and Q2.

Part 1/4: Merge objective in Section 6, Eq. (5) (W1)

Let N⊂H\mathcal{N} \subset \mathcal{H} be the set of realizable single-neuron functions in Appendix E.1, Eq. (38). We compare D(fp)=∥fi−fp∥H2+∥fj−fp∥H2D(f_p) = \sqrt{\Vert f_i - f_p\Vert _ {\mathcal{H}}^2 + \Vert f_j - f_p\Vert _ {\mathcal{H}}^2} in Eq. (5) against Dsum(fp)=∥(fi+fj)−fp∥HD _ {\text{sum}}(f_p) = \Vert (f_i + f_j) - f_p\Vert _ {\mathcal{H}}. Writing fp=sψ∈Nf_p = s\psi \in \mathcal{N} with ∥ψ∥H=1\Vert \psi\Vert _ {\mathcal{H}} = 1 and s>0s > 0, both D2(sψ)D^2(s\psi) and Dsum2(sψ)D _ {\text{sum}}^2(s\psi) depend on ψ\psi only through −2s⟨ψ,fi+fj⟩H-2s\langle \psi, f_i + f_j\rangle _ {\mathcal{H}}, so both select the same unit parent direction ψ∗∈arg⁡max⁡ψ∈N,∥ψ∥H=1⟨ψ,fi+fj⟩H\psi^* \in \arg\max _ {\psi \in \mathcal{N}, \Vert \psi\Vert _ {\mathcal{H}}=1}\langle \psi, f_i + f_j\rangle _ {\mathcal{H}} (Theorem C.7).

They differ in the scale ss and in how merging competes with pruning inside J=N⋅DErem+∥fp∥H\mathcal{J} = \frac{N \cdot D}{E _ {\text{rem}} + \Vert f_p\Vert _ {\mathcal{H}}}, where Erem=Ea−∥fi∥H−∥fj∥HE _ {\text{rem}} = E_a - \Vert f_i\Vert _ {\mathcal{H}} - \Vert f_j\Vert _ {\mathcal{H}}. Within any layer, pruning one neuron and merging two free the same parameter count ΔPinit\Delta P^{\text{init}} (Appendix B.2.1), so the encoder compares their J\mathcal{J} values directly. Because fi+fjf_i + f_j generally has rank 2 and lies outside N\mathcal{N}, replacing Eq. (5) with DsumD _ {\text{sum}} makes merging degenerate.

Specifically, DsumD _ {\text{sum}} makes single-neuron pruning strictly dominated. Setting fp=fi∈Nf_p = f_i \in \mathcal{N} drops fjf_j and keeps fif_i, giving Dsum(fi)=∥fj∥HD _ {\text{sum}}(f_i) = \Vert f_j\Vert _ {\mathcal{H}} and Erem+∥fi∥H=Ea−∥fj∥HE _ {\text{rem}} + \Vert f_i\Vert _ {\mathcal{H}} = E_a - \Vert f_j\Vert _ {\mathcal{H}}, so

Jmergesum(i,j)≤N∥fj∥HEa−∥fj∥H=Jprune(j).\mathcal{J} _ {\text{merge}}^{\text{sum}}(i, j) \le \frac{N \Vert f_j\Vert _ {\mathcal{H}}}{E_a - \Vert f_j\Vert _ {\mathcal{H}}} = \mathcal{J} _ {\text{prune}}(j).

For any nonzero fi,fjf_i, f_j with ⟨fi,fj⟩H≥0\langle f_i, f_j\rangle _ {\mathcal{H}} \ge 0, differentiating Jmergesum\mathcal{J} _ {\text{merge}}^{\text{sum}} along the ray fp(s)=sfi/∥fi∥H∈Nf_p(s) = s f_i / \Vert f_i\Vert _ {\mathcal{H}} \in \mathcal{N} at s=∥fi∥Hs = \Vert f_i\Vert _ {\mathcal{H}} gives a strictly negative slope (since DsumD _ {\text{sum}} is non-increasing at s=∥fi∥Hs = \Vert f_i\Vert _ {\mathcal{H}} while Erem+sE _ {\text{rem}} + s increases). Thus Jmergesum(i,j)<Jprune(j)\mathcal{J} _ {\text{merge}}^{\text{sum}}(i, j) < \mathcal{J} _ {\text{prune}}(j), and single-neuron pruning is never chosen.

To see this concretely in 2D, take N0=R≥0e1∪R≥0e2⊂R2\mathcal{N} _ 0 = \mathbb{R} _ {\ge 0}\mathbf{e} _ 1 \cup \mathbb{R} _ {\ge 0}\mathbf{e} _ 2 \subset \mathbb{R}^2 with orthogonal units fi=Me1f_i = M\mathbf{e} _ 1 and fj=me2f_j = m\mathbf{e} _ 2, M≥m>0M \ge m > 0. Any fp∈N0f_p \in \mathcal{N} _ 0 lies on one axis and must drop one unit. Minimizing Jmergesum\mathcal{J} _ {\text{merge}}^{\text{sum}} along fp=se1f_p = s\mathbf{e} _ 1 yields

fpsum=(M+m2Erem+M)e1,Jmergesum(i,j)=Nm(Erem+M)2+m2<Jprune(j).f_p^{\text{sum}} = \left(M + \frac{m^2}{E _ {\text{rem}} + M}\right)\mathbf{e} _ 1, \qquad \mathcal{J} _ {\text{merge}}^{\text{sum}}(i, j) = \frac{N m}{\sqrt{(E _ {\text{rem}} + M)^2 + m^2}} < \mathcal{J} _ {\text{prune}}(j).

Under DsumD _ {\text{sum}}, the encoder still deletes fjf_j, but labels the deletion as a merge with orthogonal fif_i by inflating fif_i past MM to enlarge Erem+sE _ {\text{rem}} + s, which increases the sum error DsumD _ {\text{sum}} from mm to m2+(∥fpsum∥−M)2>m\sqrt{m^2 + (\Vert f_p^{\text{sum}}\Vert - M)^2} > m.

Eq. (5) avoids this because ∥fi−fp∥H2+∥fj−fp∥H2=2∥fp−12(fi+fj)∥H2+12∥fi−fj∥H2\Vert f_i - f_p\Vert _ {\mathcal{H}}^2 + \Vert f_j - f_p\Vert _ {\mathcal{H}}^2 = 2\Vert f_p - \frac{1}{2}(f_i + f_j)\Vert _ {\mathcal{H}}^2 + \frac{1}{2}\Vert f_i - f_j\Vert _ {\mathcal{H}}^2 retains the pairwise discrepancy 12∥fi−fj∥H2\frac{1}{2}\Vert f_i - f_j\Vert _ {\mathcal{H}}^2. On the orthogonal pair above, Jmerge(i,j)=NM2+2m22Erem2+2MErem+M2+m2>Jprune(j)\mathcal{J} _ {\text{merge}}(i, j) = N\sqrt{\frac{M^2 + 2m^2}{2E _ {\text{rem}}^2 + 2ME _ {\text{rem}} + M^2 + m^2}} > \mathcal{J} _ {\text{prune}}(j) for all M≥m>0M \ge m > 0 and Erem≥0E _ {\text{rem}} \ge 0, so HOPE prunes isolated units and merges only aligned pairs. We will add this derivation, the 2D example, and a note on selection versus parent scaling to Section 6.

Re-examining Eq. (5) also led us to a practical refinement for ReLU networks. Under our Gaussian model in Sections 4 and 5, μi+=E[ReLU(yi)]>0\mu_i^+ = \mathbb{E}[\text{ReLU}(y_i)] > 0 has a closed form, so pruning fif_i or replacing (fi,fj)(f_i, f_j) by fpf_p shifts the next layer's mean input by a known analytical amount. Evaluating capacities and Eq. (5) on mean-centered outputs (subtracting μi+μj+\mu_i^+\mu_j^+ from the biased-ReLU kernel K(i,j)K(i,j) of Appendix G.3.2) and folding each step's mean shift into the next layer's BN running mean keeps HOPE completely data-free. In Parts 3 and 4, this mean-centered HOPE improves zero-shot ResNet-50 accuracy by 5.8 to 13.4 points for remaining parameter fractions d=P/P0≤0.70d = P/P_0 \le 0.70.

Part 2/4: Parameter counts, GFLOPs, 8-layer CNN, and DEFT ablation (W2)

1. Parameter counts, GFLOPs, and a second architecture. In the revised Section 9.1, we replace neuron density on the horizontal axis with the remaining parameter fraction d=P/P0d = P / P_0, alongside parameter count PP in millions and GFLOPs, all evaluated without fine-tuning. Full parameter counts and GFLOPs for ResNet-50 (P0=25.61MP_0 = 25.61\text{M}, 7.72 GFLOPs) appear in Parts 3 and 4.

To test a second architecture with a different topology, we ran zero-shot structured compression on the 20 VGG-style 8-layer CNNs from Section 9.2 and Appendix I.2. Each network is trained independently on a 20-class CIFAR-100 task and has six convolutional layers, a 512-unit dense layer, and no skip connections, with P0=8.84MP_0 = 8.84\text{M} parameters, 0.763 GFLOPs, and 62.74±3.9362.74 \pm 3.93% dense Top-1 accuracy. The table below reports mean Top-1 test accuracy ±\pm SD across the 20 models (with GFLOPs in brackets) using mean-centered HOPE; block eviction does not apply since the CNN has no skip connections.

Methodd=0.70d=0.70 (6.19M)d=0.60d=0.60 (5.30M)d=0.50d=0.50 (4.42M)d=0.40d=0.40 (3.54M)
HOPE (Prune + Merge)62.42±4.1462.42 \pm 4.14 [0.528]61.57±4.2661.57 \pm 4.26 [0.466]60.17±4.4760.17 \pm 4.47 [0.402]56.78±5.1056.78 \pm 5.10 [0.336]
HOPE (Prune-Only)62.52±4.08\mathbf{62.52 \pm 4.08} [0.514]61.72±4.13\mathbf{61.72 \pm 4.13} [0.437]60.24±4.33\mathbf{60.24 \pm 4.33} [0.368]57.11±4.99\mathbf{57.11 \pm 4.99} [0.298]
SynFlow61.88±4.4561.88 \pm 4.45 [0.506]59.56±5.7159.56 \pm 5.71 [0.429]55.11±7.2455.11 \pm 7.24 [0.361]44.20±9.8144.20 \pm 9.81 [0.296]
LAMP (Lee et al., 2021)60.63±4.3060.63 \pm 4.30 [0.474]57.36±5.3057.36 \pm 5.30 [0.397]50.83±7.4250.83 \pm 7.42 [0.328]38.14±10.1438.14 \pm 10.14 [0.264]
L1-Joint60.10±4.9960.10 \pm 4.99 [0.555]54.11±7.0854.11 \pm 7.08 [0.491]42.11±8.9242.11 \pm 8.92 [0.428]26.33±9.3626.33 \pm 9.36 [0.363]
BN-Scale60.14±5.1960.14 \pm 5.19 [0.563]51.93±7.5651.93 \pm 7.56 [0.503]37.75±8.9037.75 \pm 8.90 [0.442]20.48±9.6620.48 \pm 9.66 [0.377]

At d=0.40d=0.40 (60% of parameters removed without fine-tuning), HOPE (Prune-Only) retains 57.1% Top-1 accuracy with 61% fewer FLOPs, and HOPE (Prune + Merge) retains 56.8% with 56% fewer FLOPs, outperforming SynFlow by nearly 13 points, LAMP by 19 points, and L1-Joint and BN-Scale by over 30 points.

2. DEFT neuron-selection ablation. To isolate the neuron-selection rule in DEFT across all 20 CIFAR-100 →\to SVHN tasks, we fix every other component of the pipeline: unmerged initial weights Wsource\mathbf{W} _ {\text{source}}, the 70.6% frozen core and 29.4% plastic slack split (P=30P=30, binary Ei∈{0,1}E_i \in \lbrace 0, 1 \rbrace), the structural masking rule W0=M(E)⊙Wsource\mathbf{W} _ 0 = \mathbf{M}(\mathbf{E}) \odot \mathbf{W} _ {\text{source}}, and 30 epochs of SGD at learning rate 0.0050.005. Only the rule used to select core versus slack neurons varies. The last column gives paired tt-test pp-values on H-Score relative to HOPE and HOPE's win count across the 20 tasks.

Selection CriterionSVHN Target (%)CIFAR-100 Source (%)H-Score (%)Paired tt-test pp (Wins)
HOPE (DEFT)89.77±0.4189.77 \pm 0.4161.18±4.1661.18 \pm 4.1672.69±2.86\mathbf{72.69 \pm 2.86}N/A
LAMP89.77±0.5189.77 \pm 0.5157.69±5.4157.69 \pm 5.4170.10±4.0070.10 \pm 4.007.78×10−77.78 \times 10^{-7} (20/20)
L1-Joint (effective weights)89.61±0.4889.61 \pm 0.4854.91±6.4454.91 \pm 6.4467.88±4.8867.88 \pm 4.889.30×10−89.30 \times 10^{-8} (20/20)
BN-Scale89.79±0.5789.79 \pm 0.5752.08±7.5852.08 \pm 7.5865.61±6.0065.61 \pm 6.007.66×10−87.66 \times 10^{-8} (20/20)
Random (layer-matched)92.22±0.4292.22 \pm 0.429.85±3.919.85 \pm 3.9117.58±6.1017.58 \pm 6.101.30×10−181.30 \times 10^{-18} (20/20)
Reversed-HOPE (layer-matched)92.68±0.3592.68 \pm 0.355.00±0.005.00 \pm 0.009.49±0.009.49 \pm 0.003.05×10−273.05 \times 10^{-27} (20/20)

CIFAR-100 retention here is 61.18% versus 52.14% in Table 1 for two reasons: Table 1's 1,000-sample pre-sweep selected P=40P=40 instead of P=30P=30 from the Appendix I.5 grid, and it marked one child of each merged pair as slack even though fine-tuning started from unmerged Wsource\mathbf{W} _ {\text{source}} (so M\mathbf{M} severed that child's weights to the core without folding them into a parent). Using P=30P=30 with mean-centered capacities and keeping merged pairs frozen in the core (Ei=Ej=0E_i = E_j = 0) raises source retention to 61.18%, which is invariant across fine-tuning epochs by Theorem J.2. At around 89.8% SVHN target accuracy, HOPE retains 3.5 to 9.1 points more CIFAR-100 accuracy than LAMP, L1-Joint, and BN-Scale, and wins all 20 tasks against every baseline (two-sided Wilcoxon p=1.91×10−6p = 1.91 \times 10^{-6}).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.