acceptodds
Under review as a conference paper at ICLR 2027

Do ResNets Route? Sparse Interaction Experts in Residual Networks

Abstract

Residual networks execute every block for every input, yet their functional contributions need not be input independent. We formulate a trained ResNet as a set function over binary residual-branch masks and apply M\"obius inversion to decompose its output exactly into individual residual corrections and higher-order interactions. For smooth residual stacks, we show that each fixed -way interaction scales as under residual scaling. Exhaustive analysis of ImageNet-pretrained ResNet-18 and ResNet-34 reveals that interaction mass peaks at orders five and ten, respectively, rather than at low orders. Reducing the residual scale shifts both spectra toward lower orders, but also changes model predictions. The interaction coefficients are concentrated in magnitude but not hard sparse, and prediction-preserving sparsity weakens with depth. Crucially, the dominant interactions vary across inputs and predicted classes around a shared global core, while their overall complexity changes little with sample difficulty. These results show that dense ResNets implement an implicit form of soft routing: every block is executed, but different inputs rely on different residual interaction experts. Routing can therefore emerge at the level of functional contribution without an explicit router or sparse execution.

Primary area:
interpretability and explainable AI
Keywords:
residual networks, mixture of experts, implicit routing, Möbius inversion, higher-order interactions, interpretability, path ensembles, conditional computation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wVQJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper treats a pretrained ResNet as a set function over binary residual-branch masks and uses Möbius inversion to decompose its logits exactly into individual branch effects and higher-order interactions. It proves an O(λk)O(\lambda^k) bound for a fixed kk-way interaction in an idealized smooth residual stack as the residual scale decreases. Exhaustive mask evaluations on ImageNet-pretrained ResNet-18 and ResNet-34 show that squared interaction-coefficient mass peaks at orders 5 and 10, respectively. The authors also study coefficient concentration, reconstruction from selected interactions, and how the selected sets vary across images and classes. They interpret the latter as implicit soft routing despite dense execution.

Critical strengths
  1. The residual-mask formulation gives a precise way to separate masked-network responses from additive interaction coefficients. The exhaustive evaluation makes the decomposition and its numerical reconstruction check convincing.

  2. The paper distinguishes several quantities that are easily conflated: coefficient mass, logit reconstruction, prediction agreement, and input dependence. Reporting both ResNet-18 and ResNet-34 reveals an important depth-dependent limitation rather than hiding it.

Critical weaknesses
  1. The evidence for a sparse functional representation is substantially weaker than the main narrative suggests. Appendix H, Tables 8 and 10 show that retaining 50% of ResNet-34’s interactions captures 97.90% of vector squared-coefficient mass, yet gives relative logit error 13.87 and only 0.60% agreement with the full model’s predictions. Reaching 10% logit error requires 99.495% of the interactions on average. Signed cancellation therefore makes coefficient-mass concentration a poor proxy for a small set of experts carrying the prediction. This directly limits the proposed routing interpretation.

  2. The evidence that inputs select meaningfully different routes is modest for the label-free vector readout. At roughly 13% retention, the selected sets have mean Jaccard overlap 0.871 and 0.892 with a global set in ResNet-18 and ResNet-34, while the same-class versus different-class gaps are only 0.0179 and 0.0137 (Appendix I, Tables 11–12). The stronger scalar results select a potentially different predicted-class logit for each image, so they do not independently establish class-dependent routing over a common readout. The results support small, structured variation around a shared core; the paper should calibrate its central claim to that effect size.

Questions for authors
  1. What quantitative criterion makes the observed representation “sparse routing” when ResNet-34 needs almost its entire interaction dictionary for faithful logit reconstruction? Would a cancellation-aware selection method yield substantially better reconstruction at a small retained fraction?

  2. How much class-dependent variation remains after accounting for the shared global interaction set and, for the scalar analysis, the choice of predicted-class coordinate? An evaluation on a common readout across images would materially clarify the routing claim.

10 comments
AuthorsOpenReview ↗

Rebuttal for [W1&Q1] by Authors 【Part 1】

We thank the reviewer for recognizing the precise residual-mask formulation, the exhaustive evaluation, and our distinction between coefficient mass, reconstruction error, prediction agreement, and input dependence. Below, we address your questions about cancellation-aware selection and class structure under a common readout. We have added experiments for both and revised the claims to distinguish prediction-level sparsity from sparse execution.

[W1/Q1] Prediction-level sparsity under cancellation-aware selection

Response. We tested the reviewer's proposed cancellation-aware selection. The results show that small sets of original interaction terms can preserve most predictions, even when magnitude ranking fails. Three points clarify the result and its scope:

  1. Just 128 original terms preserve 86.45% of ResNet-34 predictions. On 2,000 images spanning all 1,000 ImageNet classes, these terms constitute 0.1953% of the dictionary. At this same budget, prediction agreement is 86.45% for forward selection versus 0.30% for magnitude ranking (Table 1). Selected coefficients retain their original values and unit weights.

  2. Joint selection accounts for cancellation that magnitude ranking ignores. Forward selection chooses each added term by its reduction in the combined reconstruction error, accounting for the terms already retained. It changes which original terms are selected, without refitting or rescaling their coefficients.

  3. Our sparsity claim concerns prediction preservation at a reported logit error. We report the retained fraction together with prediction agreement and relative logit error: 0.1953%, 86.45%, and 0.2977, respectively. This does not meet the separate 10%-logit-error criterion. The selection uses the full output and decomposition, so it is an offline test of the representation, not an online router.


1. Retaining 0.1953% of the dictionary preserves 86.45% of predictions

Experimental setup. We use frozen ImageNet-pretrained ResNet-34 at λ=1\lambda=1 on a fixed cohort of 2,000 ImageNet-1K validation images, two per class. A seeded hash of sample IDs fixes this cohort before analysis, independently of logits and outcomes. All images use the complete 1,000-dimensional logit vector as the common readout. For each input, magnitude ranking and greedy forward selection choose K=32K=32 or K=128K=128 terms from the same 65,535 nonempty interactions. Forward selection adds the term that most reduces the current reconstruction error, accounting for the terms already selected.

Both methods include the shortcut-only baseline and retain every selected original vector coefficient at weight one. We do not refit or rescale coefficients, and true labels do not enter support selection. Relative logit error is ∥h^x−hx(1[L])∥2max⁡(∥hx(1[L])∥2,10−12)\frac{\lVert\widehat{h}_x-h_x(\mathbf{1}_{[L]})\rVert_2}{\max(\lVert h_x(\mathbf{1}_{[L]})\rVert_2,10^{-12})}; prediction agreement compares the reconstructed and full-model argmax. Forward selection uses the full-model target and the complete decomposition, so it is an offline diagnostic, not an online router.

Table 1: Magnitude-based and cancellation-aware reconstruction in ResNet-34. Both methods use the same 2,000 images, common vector readout, original coefficients, and support cardinality. Agreement is with the full-model prediction, not ground-truth accuracy.

Retained terms KKDictionary fractionMagnitude relative errorForward relative errorMagnitude prediction agreementForward prediction agreement
320.0488%24.97480.46290.20%74.40%
1280.1953%53.50320.29770.30%86.45%

Results and analysis. Table 1 shows that forward selection preserves 74.40% of predictions at K=32K=32 (class-block bootstrap 95% CI [72.30%, 76.35%]), with relative error 0.4629 [0.4612, 0.4646]. At K=128K=128, agreement reaches 86.45% [85.00%, 88.10%], while relative error falls to 0.2977 [0.2964, 0.2992]. Ground-truth accuracy is 73.40% for the full model, 64.00% at K=32K=32, and 70.10% at K=128K=128.

AuthorsOpenReview ↗

Rebuttal for [W1&Q1] by Authors 【Part 2】

2. Cancellation explains why individual coefficient size is a poor guide to reconstruction

Analysis. Reconstruction error depends on the sum of omitted terms, not just their individual sizes. Writing δS=ΔShx\delta_S=\Delta_Sh_x and O\mathcal O for the omitted interactions, we have, under any fixed ordering of the interaction indices,

∥∑S∈OδS∥22=∑S∈O∥δS∥22+2∑S<T, S,T∈O⟨δS,δT⟩.\left\lVert\sum_{S\in\mathcal{O}}\delta_S\right\rVert_2^2=\sum_{S\in\mathcal{O}}\lVert\delta_S\rVert_2^2+2\sum_{S\lt T,\,S,T\in\mathcal{O}}\langle\delta_S,\delta_T\rangle.

Magnitude ranking uses individual norms but ignores the cross terms. It can retain a large term while omitting the terms needed to cancel it, worsening reconstruction as the budget increases. Forward selection accounts for the terms already retained through the joint error. This explains why Table 1 gives very different outcomes at the same support size, and why magnitude error need not decrease monotonically.


3. Prediction agreement and 10%-error logit reconstruction are different criteria

We quantify prediction-level sparsity by the retained dictionary fraction at a stated prediction agreement, while reporting logit error alongside it. The new result is 128 of 65,535 terms, 86.45% agreement and 0.2977 relative error. The original 99.495% retention figure concerns reaching 10% relative logit error under magnitude ranking. These are different selection procedures and fidelity criteria; we do not present the new result as achieving that 10% threshold. Nor does an arbitrary interaction support specify an executable branch subset.

Conclusion. The proposed cancellation-aware test finds small supports that preserve most predictions using unchanged original coefficients. It establishes prediction-level sparsity on this cohort while keeping logit fidelity and sparse execution separate.


Manuscript changes:

  • Main text: Abstract (p. 1); Introduction, Section 1 (pp. 1–2); cancellation-aware reconstruction, Section 4 and manuscript Table 2 (pp. 7–8; table on p. 7).
  • Definitions and explanation: Section 2.1 (pp. 2–3); cancellation identity and prediction criterion, Appendix K.2 (pp. 48–49).
  • Experimental details: Appendix L.2 (pp. 52–53); reconstruction results, Appendix L.3.2 (pp. 53–56), with the primary 2,000-image comparison in manuscript Table 24 and the separate 200-image diagnostics in Tables 25–26 (all p. 55).
AuthorsOpenReview ↗

Rebuttal for [W2&Q2] by Authors 【Part 1】

[W2/Q2] Class structure under common readouts and beyond the shared core

Response. Class-dependent variation remains under a common readout and after excluding the shared core. We added three tests to distinguish this variation from the two factors raised by the reviewer:

  1. Removing the shared core leaves measurable class structure in both models. With the same full-logit readout for every input, the remaining interaction sets have same-class minus different-class Jaccard gaps of 0.032887 for ResNet-18 and 0.036207 for ResNet-34. Both have Bonferroni-adjusted p=0.03960p=0.03960 under the four-test primary family (Table 2). Thus, global reuse does not explain all class-dependent variation.

  2. The class signal persists without choosing an input's predicted-class coordinate. Every image uses the same four-coordinate vector to rank interactions. After core exclusion, the gaps remain positive: 0.034914 for ResNet-18 and 0.023951 for ResNet-34 (Table 3). These are exploratory checks, reported separately from the primary tests.

  3. The same sparse supports that preserve predictions also show class structure. The saved ResNet-34 supports with K=128K=128 preserve 86.45% of predictions and have a Jaccard class gap of 0.002532, with Holm-adjusted p=0.001998p=0.001998 (Table 4). Labels do not enter support selection. This post-hoc test connects prediction preservation and class structure to the same supports; it is separate from the core-exclusion experiments.


1. Removing the shared core leaves positive class gaps under a common full-logit readout

Experimental setup. We fit the global interaction ranking on 5,000 calibration images and evaluate on a separate fixed cohort of 640 images from 128 classes, five per class. The ranking uses mean normalized squared vector-coefficient energy and does not use labels. At approximately 13% total retention, we fix a core occupying 75% of that budget: (K,∣G∣,K−∣G∣)=(33,24,9)(K,|G|,K-|G|)=(33,24,9) for ResNet-18 and (8,520,6,390,2,130)(8{,}520,6{,}390,2{,}130) for ResNet-34. For each evaluation image, we exclude these global indices before selecting the remaining tail.

Every image uses the same full-logit readout. We measure the mean same-class minus different-class Jaccard similarity of the tail indices. Class controls use 10,000 sampled pairs of each kind, 200 class-block bootstrap repeats, and 100 label shuffles with supports fixed. The prespecified primary family contains four tests: outside-core vector Jaccard and cosine gaps for each model, with a four-test Bonferroni correction.

Table 2: Class structure outside the calibration-fixed global core. Results use the same 640 evaluation images and the complete vector readout. Intervals are marginal class-block bootstrap 95% intervals; the last column gives the one-sided permutation pp after correction over the four primary tests.

ModelOutside-core vector Jaccard gapClass-block 95% CIPrimary-family adjusted pp
ResNet-180.032887[0.019679, 0.045522]0.03960
ResNet-340.036207[0.029767, 0.042473]0.03960

Results and analysis. Table 2 shows that same-class inputs share more of their remaining interaction indices than different-class inputs in both models. The gaps remain positive after core exclusion and pass the stated four-test correction, while every input uses the identical full-logit readout. The shared global core therefore does not account for all class structure. This detects structured variation around shared computation, not disjoint class-specific routes.

Reviewer_pGTsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper models a trained ResNet as a set function over binary residual-branch masks and applies Möbius inversion to exactly decompose its logits into a shortcut-only baseline, individual residual corrections, and higher-order interactions. It establishes that each k-way interaction scales as O(λᵏ) under suitable smoothness assumptions and exhaustively analyzes ImageNet-pretrained ResNet-18 (256 masks) and ResNet-34 (65,536 masks). The authors find that interaction mass peaks at orders 5 and 10, respectively, and that reducing residual scale shifts the interaction spectrum toward lower orders but substantially changes model predictions. Interaction coefficients exhibit magnitude concentration, although accurate prediction reconstruction requires a relatively large subset. Dominant interaction sets vary across inputs and classes around a largely shared global core, while interaction complexity is weakly associated with input difficulty. The authors interpret these findings as evidence of implicit soft routing in densely executed ResNets.

Critical strengths
  • Exact and well-defined analytical framework: The application of Möbius inversion provides a unique additive decomposition of residual-mask responses into individual corrections and higher-order interactions. The authors verify the reconstruction numerically (Tables 1 and 3), establishing a sound basis for the subsequent analysis.

  • Exhaustive experimental analysis: The authors evaluate all possible residual-branch masks for ResNet-18 and ResNet-34, including 65,536 configurations for the latter.

  • Meaningful experimental controls: The evaluation includes random-K, order-matched, and low-order interaction selection, along with label-shuffling and random-set similarity controls. These comparisons strengthen the analysis of interaction concentration and class-dependent variation.

  • Careful interpretation and acknowledgment of limitations: The authors explicitly distinguish coefficient mass from reconstruction fidelity, recognize that interaction coefficients are non-orthogonal, and clarify that the selected expert sets are post-hoc oracle constructions rather than executable sparse pathways. They also acknowledge the readout dependence of the analysis and avoid making causal claims from the observed relationship between interaction complexity and input difficulty.

Critical weaknesses
  • Limited practical implications and novelty: The identified interaction supports are not used to predict block importance, guide input-dependent computation, or improve efficiency. Consequently, the routing interpretation remains descriptive.

  • Narrow experimental scope and scalability: The analysis is restricted to two ImageNet-pretrained BasicBlock ResNets. It does not consider bottleneck architectures, alternative training procedures, or other residual networks. Additionally, exhaustive enumeration scales exponentially with depth, making the framework impractical for substantially deeper models.

  • Limited applicability of the theoretical result: Proposition 2.3 assumes smooth residual functions in an idealized architecture, whereas the evaluated ResNets contain ReLU activations. Furthermore, the result concerns the small-residual regime, while the networks operate at λ = 1. The empirical scaling experiment substantially changes predictions, limiting the proposition's relevance to the models' normal behavior.

  • Limited evidence for implicit routing: The class-dependent differences in vector interaction supports are small (Jaccard gaps of 0.01–0.04), and individual supports overlap heavily with a global set (0.87–0.89). The stronger scalar results are partly influenced by selecting the predicted-class logit as the readout. Moreover, ResNet-34 requires nearly all interaction terms for accurate reconstruction. These findings support input-dependent contributions but provide weaker evidence for sparse, specialized routing comparable to an MoE.

Questions for authors
  • Would a ResNet trained with stochastic depth, or a bottleneck ResNet-50, show stronger class-dependent routing?

  • Can the per-input interaction sets be used for anything practical, such as per-input block skipping?

See Critical weaknesses as well.

14 comments
AuthorsOpenReview ↗

Rebuttal for [W1/Q2] by Authors 【Part 1】

We thank the reviewer for recognizing the exact formulation, exhaustive evaluation, experimental controls, and distinction between coefficient concentration and reconstruction fidelity. Below, we address the practical use of the analysis, its generalization, the scope of the theory, and the evidence for functional routing.


[W1/Q2] Can interaction analysis guide branch selection and improve inference efficiency?

Response. Yes. Interaction analysis guides branch selection for fixed compact networks, and we measure their actual forward acceleration. Our additional experiments show the following:

  1. Joint deletion coefficients help select better subnetworks without retraining. We estimate joint effects from singleton and pair deletion queries and use them to predict unqueried branch combinations. These coefficients aggregate the residual interactions affected by deletion. For ResNet-50, including joint effects raises accuracy from 56.05% with matched additive selection to 62.30% at the same three-branch deletion budget. The selected network runs 1.261× faster than the full model at Batch-32 (Table 2). The ResNet-34 test separately verifies improved candidate ranking (Table 1).

  2. The selected masks yield executable networks whose accuracy can be recovered. We physically omit the removed computation, then apply distillation. For the interaction-selected three-deletion ResNet-50 mask, accuracy rises from 62.440% to 73.575% on the same fresh 10,000-image test, versus 75.890% for the original model, at 1.262× Batch-32 speedup (Table 3). Distillation supplies recovery; point 1 tests the selection benefit before training. Table 3 also reports smaller accuracy losses for ResNet-34 and ResNet-50 at lower deletion budgets.

  3. Input-dependent selection can further improve prediction quality at a fixed branch count. At 12 active ResNet-34 branches, an offline per-input error oracle raises accuracy from 48.16% with a fixed subset to 58.32%, and prediction agreement from 53.72% to 66.62% (Table 4). This identifies room for a future input-dependent selector. Our deployed networks currently use fixed offline masks, not an online router.

1. Joint effects improve ResNet-34 subnetwork ranking from 0.772 to 0.897

Experimental setup. For frozen ResNet-34, we use 640 images from 128 classes, five per class. On each image, the additive logit model and second-order interaction model receive the same 56 responses: the full network, ten single-branch deletions and 45 pair deletions. They predict the full-network-to-subnetwork KL divergence for 120 three-branch deletion combinations that were not queried.

The second-order model includes the additional effect observed when two branches are removed together, beyond the sum of their separate deletion effects.

We measure Spearman correlation across candidates within each image, then average over images. Selection KL regret is the actual KL of the proxy-selected candidate minus the minimum actual KL among those candidates. Each image supplies its own queried responses, so this tests offline search quality.

Table 1: Predicting unqueried three-branch deletions in frozen ResNet-34. Results average over 640 inputs. The matched additive logit fit and second-order model use identical queried logits; the singleton baseline ignores pair responses. Higher correlation and lower regret indicate better candidate selection.

Prediction methodMean Spearman correlationMean selection KL regret
Sum of single-deletion effects0.69410.2987
Additive logit fit, matched queries0.77240.1646
Additive KL fit, matched queries0.74750.2143
Second-order deletion interactions0.89660.0892

Results and analysis. Table 1 shows that including pairwise effects raises correlation from 0.77235 to 0.89663 and lowers selection KL regret from 0.16457 to 0.08924, a 45.8% reduction, using the same queries. Joint deletion effects therefore improve candidate ranking and selection beyond what the additive models extract from those same responses.

AuthorsOpenReview ↗

Rebuttal for [W1/Q2] by Authors 【Part 2】

2. Without retraining, interaction-guided ResNet-50 gains 6.25 accuracy points over additive selection

Experimental setup. We use ImageNet-pretrained ResNet-50. A 512-image calibration set from 128 classes generates candidates. The additive and second-order models receive the same 79 responses per image: the full network and all eligible singleton/pair deletions. Each proposes eight three-deletion candidates; singleton scoring and a random-search shortlist are additional controls. A separate 1,000-image calibration-validation set, one image per class, selects the final mask by accuracy, with teacher KL breaking ties. The mask is fixed before testing on 2,000 independent images, two per ImageNet class.

There is no fine-tuning, distillation or BN recalibration. We physically omit closed identity-shortcut blocks and preserve projection shortcuts. At one and two deletions, all compared methods select the same mask from already queried responses.

Timing uses one isolated A40, native FP32, GPU-resident inputs, 20 warmups and 100 synchronized repetitions. It includes the complete forward pass, including the stem and classifier, and excludes preprocessing, input transfer and offline search.

Table 2: Fixed ResNet-50 branch selection without distillation or fine-tuning. Accuracy and prediction agreement use the same 2,000 held-out images. Agreement is with the original full-model prediction; accuracy uses ground truth. Timing cells report median forward milliseconds per batch / speedup on the same A40, using native FP32. “Shared mask” means the methods select the identical mask at that budget.

SelectionRemoved branchesAccuracyAccuracy loss (pp)Prediction agreementBatch-1: ms / speedupBatch-32: ms / speedup
Original full model0/1675.45%0.00100.00%5.262 / 1.000×35.512 / 1.000×
Shared mask1/1671.50%3.9584.55%5.002 / 1.052×33.855 / 1.049×
Shared mask2/1669.20%6.2577.75%4.746 / 1.109×29.789 / 1.192×
Single-branch scoring3/1656.45%19.0064.20%4.523 / 1.164×30.302 / 1.172×
Additive logit fit, matched queries3/1656.05%19.4062.90%4.495 / 1.171×29.355 / 1.210×
Random-search shortlist3/1661.35%14.1068.55%4.501 / 1.169×28.131 / 1.262×
Second-order deletion interactions3/1662.30%13.1570.60%4.488 / 1.172×28.152 / 1.261×

Results and analysis. At three deletions, interaction-guided selection improves accuracy over the matched additive method by 6.25 points [4.45, 8.00], raises prediction agreement from 62.90% to 70.60%, and achieves 1.261× Batch-32 acceleration (Table 2). Its gain over singleton scoring is 5.85 points [3.85, 7.95]; the gain over random search is 0.95 points [−0.70, 2.65]. Intervals use class-block bootstrap. The smaller deletion budgets retain more accuracy, but use shared masks and therefore do not distinguish the selection methods.

This frozen comparison tests what interaction information adds to branch selection before any recovery training.

We also test frozen ResNet-34. Removing one branch gives 69.70% accuracy and 85.15% prediction agreement, compared with 72.65% full-model accuracy, at 1.077× Batch-32 acceleration on the same 2,000-image protocol. On the separate fresh 10,000-image cohort below, this mask scores 70.47% versus 72.84% for the original model. Its frozen-weight speed was measured in the earlier study.

AuthorsOpenReview ↗

Rebuttal for [W1/Q2] by Authors 【Part 3】

3. After distillation, ResNet-50 retains 74.455% accuracy at 1.191× forward speedup

Experimental setup. We fix the selected masks and train each compact network and a matched full-model control for 40 epochs on 20,000 official ImageNet training images. The 1,000-image calibration-validation set selects the recovery recipe and checkpoint epoch. All methods within an architecture receive the same recipe and two seeds, starting from the pretrained weights. Recovery uses cross-entropy and teacher KL at temperature 2; ResNet-34 additionally uses feature matching and frozen BN running statistics.

Final quality uses a fresh 10,000-image test set covering all 1,000 classes and excluding previous evaluation images. The original model and the same compact masks before recovery are evaluated on those identical inputs. Timing follows the A40 protocol above and uses the predetermined seed-zero checkpoint. Deployment runs only the compact network, without a teacher, interaction queries or a selector.

Table 3: Accuracy, prediction agreement and forward acceleration after distillation. Quality uses the fresh 10,000-image test set. Recovered results average two separate runs, not an ensemble; agreement is with the original full-model prediction. Speedups use the predetermined seed-zero recovered checkpoint relative to the original full model, rather than independent timings of the Frozen column. Accuracy, relative error and speed results for all 19 configurations appear in manuscript Tables 40–41; the prediction-agreement column is an additional metric from those same runs, reported here.

ModelRemoved branchesOriginal full accuracyFrozen compact accuracyRecovered compact accuracyRecovered prediction agreementBatch-1 speedupBatch-32 speedup
ResNet-341/1672.840%70.470%71.890%89.270%1.047×1.077×
ResNet-342/1672.840%65.740%70.885%86.305%1.103×1.168×
ResNet-502/1675.890%68.380%74.455%90.135%1.108×1.191×
ResNet-503/1675.890%62.440%73.575%87.960%1.169×1.262×

Results and analysis. ResNet-34 with one removed branch is 0.95 accuracy points below the original model at 1.077× acceleration. ResNet-50 with two removed branches is 1.435 points below it at 1.191×. For the interaction-selected three-deletion ResNet-50 mask, distillation raises accuracy from 62.440% to 73.575% on the same 10,000 inputs, while retaining 1.262× forward speedup (Table 3). The recovered full-model controls reach 72.660% for ResNet-34 and 75.405% for ResNet-50.

At one and two deletions, masks are shared across methods. These rows demonstrate recovery and acceleration, rather than an interaction-specific selection advantage. Tables 1–2 test that advantage without recovery.

Reviewer_hdbxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors investigate the routing characteristics of ResNets, leveraging the inverse-Mobius/Mobius Transform to consistently define both the total output vx(S)v_x(S) and output contribution over proper subsets ΔSvx\Delta_S v_x attributable to a set SS of residual blocks (for outputs such as the logits of the winning class prediction, c.f. Def. 2.1 and Lemma 2.2).

The study focuses on the Mobius residual coefficients ΔSvx\Delta_S v_x, investigating their properties wrt additivity (Table 1), order (Figure 1 and 2), and sparsity (eq. 4, Figures 3-5).

The paper is technically sound and the Mobius decomposition is an interesting analysis tool, but the paper falls short of establishing its significance or contributions. Mobius residual coefficients are studied extensively, but as emphasized throughout the paper, analyze high-order interactions post-hoc to execution of the network, and have no implications for sparse execution. Section titles like "A RESIDUAL NETWORK ROUTES LIKE AN MOE, WITHOUT A ROUTER" are not appropriate.

The analysis, furthermore, is not adequately motivated or connected to potential future research directions. Meanwhile, analysis around the total contribution of subsets (i.e. vx(S)v_x(S)) which could directly inform future research on efficient conditional computation is completely skipped.

In summary, while the Mobius formation is interesting and likely ultimately of utility in analyzing networks with residual connections, the current analysis in the paper is incomplete, and falls short of establishing its significance and contributions, and so my current recommendation is that the paper be rejected.

Critical strengths

Strengths:

  • An interesting formulation for analyzing the emergence of routing in multipath networks, which is an interesting and important research area.
  • The paper is well written and technically sound (with caveats, see limitations section).
Critical weaknesses
  • As described in the summary, the paper falls short of establishing the significance of their analysis of the residual coefficients of subsets (i.e. their contributions over proper subsets), and completely skips the analysis of total contributions of subsets, which in contrast would be highly relevant to future research on efficient conditional computation.
Questions for authors
  • Authors, please do attempt to elaborate on how your analysis is valuable and informs future research directions. Thanks!
10 comments
AuthorsOpenReview ↗

Rebuttal for W1 by Authors 【Part 1】

We thank the reviewer for recognizing the clarity and technical soundness of our paper and the potential of the Möbius formulation. Your questions prompted us to examine more closely what the interaction coefficients reveal and how they can guide computation. Below, we address your concerns about their significance, total subset responses, and the connection to conditional computation.


[W1] The significance of analyzing residual interaction coefficients is not established

Response. We conducted additional experiments to assess the value of analyzing residual interaction coefficients. These experiments support the following three conclusions:

  1. Joint interaction effects guide better branch selection. Our decomposition shows that removing a branch removes every residual interaction containing it. We estimate joint deletion coefficients from singleton and pair queries; these coefficients aggregate the original residual interactions and predict the outputs of unqueried branch combinations. Using those predictions to select three branches for removal raises ResNet-50 accuracy from 56.05% with matched additive selection to 62.30%, without retraining, while the compact network runs 1.261× faster than the full model at Batch-32 (Table 2).

  2. The selected branch combinations yield executable networks whose accuracy can be recovered. We physically remove the selected blocks, then apply distillation after selection. For the interaction-selected three-deletion ResNet-50 mask, accuracy increases from 62.440% to 73.575%, versus 75.890% for the original model, at 1.262× Batch-32 speedup on the fresh 10,000-image test (Table 1). This validates deployment of the selected mask; distillation supplies the recovery, while the frozen comparison in point 1 isolates the selection benefit. Table 1 also reports the smaller-loss ResNet-34/50 operating points.

  3. Accounting for cancellation between coefficients reveals small prediction-preserving supports. We select original coefficient vectors according to how their joint sum reconstructs the full logits, instead of ranking their magnitudes independently. At the same K=128K=128 budget, this preserves 86.45% of ResNet-34 predictions versus 0.30% for magnitude ranking, using 0.1953% of the dictionary, with relative logit error 0.2977 (Table 5). The coefficients remain unchanged; this is an offline test of the functional representation, separate from executable branch skipping.


1. After distillation, ResNet-50 retains 74.455% accuracy at 1.191× forward speedup

Experimental setup. We fix branch masks using calibration data, then recover the compact ResNet-34/50 models and matched full-model controls for 40 epochs on 20,000 official ImageNet training images. A separate 1,000-image calibration-validation set selects one recovery recipe per architecture and the checkpoint epoch. Every selection method within an architecture receives that same recipe, starts from the original pretrained weights, and uses seeds 0 and 1.

Final quality uses a fresh 10,000-image test set covering all 1,000 classes and excluding previously used evaluation images. The original full model and each compact mask before and after recovery are evaluated on those identical inputs. Recovered accuracy averages the two runs, not an ensemble. The Frozen column below is the same mask before recovery on this new test set.

We physically omit deleted identity-shortcut blocks while preserving projection shortcuts. Both deployment studies measure the complete native FP32 forward on one isolated A40, including the stem and classifier, with GPU-resident inputs, 20 warmups and 100 synchronized repetitions. Preprocessing, input transfer and offline search/training are excluded equally. Recovered-model speed uses the predetermined seed-zero checkpoint; inference requires no teacher, interaction queries or online selector.

Table 1: Compact networks retain near-original accuracy with measured forward acceleration after distillation. Quality uses 10,000 fresh test images. Accuracy loss is relative to the original full model on the same images. Speedups measure the recovered compact model against the original full model; they are not independent timings of the Frozen column. Manuscript Tables 40–41 report accuracy, relative error and speed for all 19 configurations and both seed accuracies.

ModelRemoved branchesOriginal full accuracyFrozen compact accuracyRecovered compact accuracyAccuracy loss (pp)Batch-1 speedupBatch-32 speedup
ResNet-341/1672.840%70.470%71.890%0.9501.047×1.077×
ResNet-342/1672.840%65.740%70.885%1.9551.103×1.168×
ResNet-502/1675.890%68.380%74.455%1.4351.108×1.191×
ResNet-503/1675.890%62.440%73.575%2.3151.169×1.262×
AuthorsOpenReview ↗

Rebuttal for W1 by Authors 【Part 2】

Results and analysis. ResNet-34 with one deleted branch reaches 71.890% versus 72.840% original accuracy, a 0.950-point loss at 1.077× Batch-32 acceleration. ResNet-50 with two deleted branches reaches 74.455% versus 75.890%, a 1.435-point loss at 1.191× acceleration. Removing three ResNet-50 branches gives 73.575% accuracy at 1.262× acceleration, a 2.315-point loss (Table 1). These are measured accuracy–speed tradeoffs in networks that actually skip the removed computation.

The matched recovered full-model controls reach 72.660% for ResNet-34 and 75.405% for ResNet-50. One- and two-deletion masks are shared across the selection methods, so their recovery gains are not specific to interaction-based selection. The frozen comparisons below separately test what interaction information adds before any recovery training.


2. Without retraining, interaction-guided ResNet-50 gains 6.25 accuracy points over additive selection

Experimental setup. ResNet-50 is also frozen at λ=1\lambda=1, with no fine-tuning, distillation or BN recalibration. We use 512 calibration images from 128 classes to generate candidate masks. The additive and second-order models both receive the same 79 responses per calibration image, covering the full model and all eligible singleton/pair deletions. Each proposes eight three-deletion candidates; single-branch scoring and a random-search shortlist are additional controls.

A separate 1,000-image calibration-validation set, one image per class, selects the final mask by ground-truth accuracy, with teacher KL breaking ties. Each selected mask is fixed for all 2,000 independent test images, two per ImageNet class. We physically remove the closed identity-shortcut blocks and measure native FP32 forward time on one A40. Projection shortcuts are preserved. All masks are chosen using calibration data before the test set is evaluated. We also include one- and two-deletion settings, whose responses are already queried; these use the same selected masks across methods.

Table 2: Fixed ResNet-50 branch selection without distillation or fine-tuning. All results use the same 2,000 held-out images and the original pretrained weights and BN statistics. Accuracy is measured against ground truth; agreement is with the full model's prediction. Forward times are median milliseconds per batch on the same A40 using native FP32. Each timing cell gives latency / speedup relative to the full model. Timing excludes offline search, preprocessing and input transfer. “Shared mask” denotes the identical mask selected across the methods evaluated at one or two deletions.

SelectionRemoved branchesAccuracyAccuracy loss (pp)Prediction agreementBatch-1: ms / speedupBatch-32: ms / speedup
Original full model0/1675.45%0.00100.00%5.262 / 1.000×35.512 / 1.000×
Shared mask1/1671.50%3.9584.55%5.002 / 1.052×33.855 / 1.049×
Shared mask2/1669.20%6.2577.75%4.746 / 1.109×29.789 / 1.192×
Single-branch scoring3/1656.45%19.0064.20%4.523 / 1.164×30.302 / 1.172×
Additive logit fit, matched queries3/1656.05%19.4062.90%4.495 / 1.171×29.355 / 1.210×
Random-search shortlist3/1661.35%14.1068.55%4.501 / 1.169×28.131 / 1.262×
Second-order deletion interactions3/1662.30%13.1570.60%4.488 / 1.172×28.152 / 1.261×

Results and analysis. At the same three-deletion budget, interaction-guided selection achieves 62.30% accuracy and 70.60% prediction agreement at 1.261× Batch-32 acceleration over the full model (Table 2). Compared with the matched additive method, it improves accuracy by 6.25 percentage points, with a class-block 95% interval of [4.45, 8.00], and raises prediction agreement from 62.90% to 70.60%. Its Batch-32 latency is also lower, 28.152 versus 29.355 ms. The accuracy gain over single-branch scoring is 5.85 points [3.85, 7.95]; the gain over random search is 0.95 points [−0.70, 2.65]. The removed blocks are omitted during inference. Accuracy losses relative to the full model are reported alongside the speedups so that the prediction–speed tradeoff is explicit.

Removing one branch retains 71.50% accuracy and 84.55% prediction agreement. Removing two retains 69.20% accuracy and 77.75% agreement, while reducing Batch-32 forward latency from 35.512 to 29.789 ms (1.192× acceleration). These low-deletion rows show the tradeoff for shared masks; they do not distinguish the selection methods.

AuthorsOpenReview ↗

Rebuttal for W1 by Authors 【Part 3】

On the same 2,000-image protocol, frozen ResNet-34 with one branch removed reaches 69.70% accuracy versus 72.65% for the full model, with 1.077× Batch-32 acceleration, without fine-tuning, distillation or BN recalibration. Its fixed mask also reaches 70.47% versus 72.84% on the new 10,000-image cohort. That larger-cohort result rechecks accuracy; the frozen-weight speed was measured in the earlier study.


3. Interactions improve ResNet-34 subnetwork ranking: Spearman 0.772 to 0.897

Experimental setup. The ImageNet-pretrained model is frozen at λ=1\lambda=1, with no fine-tuning, distillation or BN recalibration. For ResNet-34, we evaluate 640 images from 128 classes, five images per class. For each image, the matched additive logit model and second-order deletion model observe exactly the same 56 responses: the full network, ten single-branch deletions and 45 pair deletions. They predict the responses of 120 three-branch deletion combinations that were not queried. We measure mean per-image Spearman correlation between predicted and actual teacher KL across candidates. Selection KL regret is the actual KL of the proxy-selected candidate minus the lowest actual KL among those candidates. This is an offline test: each evaluation image supplies its own queried responses.

Table 3: Predicting unqueried three-branch deletions in frozen ResNet-34. Results average over 640 inputs. The matched additive logit model and the second-order model use identical queried logits; the singleton baseline ignores pair responses. Higher correlation and lower regret indicate better search predictions.

Prediction methodMean Spearman correlationMean selection KL regret
Sum of single-deletion effects0.69410.2987
Additive logit fit, matched queries0.77240.1646
Additive KL fit, matched queries0.74750.2143
Second-order deletion interactions0.89660.0892

Results and analysis. With the same queried responses, adding pairwise deletion effects raises mean Spearman correlation from 0.77235 to 0.89663 and reduces mean selection KL regret from 0.16457 to 0.08924, a 45.8% reduction (Table 3). The interaction model therefore ranks the unqueried candidates more accurately and chooses candidates with lower actual KL.


4. Pretrained ResNet-18 captures 93.65% of mass in Top-64, versus 52.81% for random models

Experimental setup. Models are frozen at λ=1\lambda=1. Residual masks preserve the shortcuts, and every input uses the full 1,000-dimensional logit vector as the readout. We compare an ImageNet-pretrained ResNet-18 with three randomly initialized models of the same architecture. Each model is evaluated on the same 10,000 class-balanced ImageNet-1K validation images, split into 5,000 calibration and 5,000 evaluation images. We enumerate all 256 masks per image. On the evaluation split, we calculate the squared coefficient mass retained by each input’s 64 largest vector interactions, then average over inputs.

Table 4: Coefficient concentration in pretrained and randomly initialized ResNet-18 models. Captured mass is averaged over 5,000 evaluation images; the random-model result averages three initializations.

ResNet-18 checkpointSquared coefficient mass retained by Top-64
Random initialization, mean over three seeds52.81%
ImageNet-pretrained93.65%

Results and analysis. The Top-64 mass is 40.84 percentage points higher in the pretrained model than in the random-model mean (Table 4). Thus, the concentration observed in the trained checkpoint is substantially stronger than in the same architecture with random weights.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response 【Part 1】

We sincerely thank all three reviewers for their careful reading and constructive feedback. We appreciate their recognition of the precise formulation, numerical verification, and clear presentation. The revised paper strengthens two contributions: uncovering implicit functional routing through input- and class-dependent residual interactions, and using joint interaction effects to select executable subnetworks for faster inference. Below, we summarize the main concerns, new evidence, and corresponding revisions. Detailed protocols and additional comparisons are provided in our individual responses.

All locations refer to the revised manuscript. Substantive changes are marked in burgundy.


1. Evidence for implicit functional routing (Reviewers hdbx, pGTs, and wVQJ)

Concern. Reviewers pGTs and wVQJ asked whether class structure survives common readouts and exclusion of shared interactions. Reviewer hdbx questioned the MoE-style interpretation of post-hoc analysis in a densely executed network.

Response. We find measurable class-dependent variation around shared computation, and now distinguish this functional organization explicitly from sparse execution. The new controls address the two proposed explanations directly:

  • Exclude the shared core under a common full-logit readout. Global indices are fitted on 5,000 calibration images and excluded before selecting the remaining interactions on 640 held-out images from 128 classes. The outside-core Jaccard class gaps are 0.032887 for ResNet-18 and 0.036207 for ResNet-34. Both have four-test Bonferroni-adjusted p = 0.03960; ResNet-34's 95% interval is [0.029767, 0.042473].
  • Hold output coordinates fixed. On the same split, class gaps remain positive using one common four-coordinate vector. No coordinate is selected from an image's predicted class. These are exploratory controls, separate from the primary test family.

Our claim is class-structured, input-dependent functional contributions around shared computation. We renamed Section 4 from “A Residual Network Routes Like an MoE, Without a Router” to “Sparse, Class-Structured Functional Specialization.” Analytical masks define the decomposition; the original checkpoint remains densely executed. Static branch removal is a separate downstream application.

Manuscript changes: Abstract and Introduction (pp. 1–2); Section 4 title and analysis (pp. 7–8); Section 2.1 and Figure 1 (pp. 2–4; figure on p. 3); Appendix K.1 and K.3 (pp. 47–49); Appendix L.3.3, Tables 27–29 (pp. 56–57).

AuthorsOpenReview ↗

General Response 【Part 2】

2. Practical value and inference acceleration (Reviewers hdbx and pGTs)

Concern. What does the interaction analysis contribute beyond post-hoc description? Can it inform actual branch removal, and how does it relate to total masked-network responses?

Response. Joint interaction effects now guide branch selection, and the selected compact networks skip computation with measured forward acceleration. Removing a branch removes every interaction containing it. We use this connection to estimate joint deletion effects from singleton and pair queries and predict unqueried branch combinations. These deletion coefficients aggregate the original residual interactions; the practical predictor does not require computing the full interaction dictionary.

Experimental setup. Frozen ResNet-50 predictors receive identical 79-response query sets. We generate candidates on 512 calibration images, select masks on a separate 1,000-image validation set, and test on 2,000 independent images, without fine-tuning, distillation or BN recalibration. Recovery uses 20,000 official training images, 40 epochs, matched recipes and two seeds, with a fresh 10,000-image test.

Table 1: Executable compact networks before and after distillation. Ground-truth accuracy uses 2,000 images for frozen models and a separate fresh 10,000-image test for recovered models. Each original/compact pair uses identical inputs. Recovered quality averages two runs; timing uses predetermined seed zero. Speed is complete GPU-resident FP32 Batch-32 forward acceleration over the full model on one A40, excluding offline search/training, preprocessing and transfer. “Shared” denotes identical masks across selection methods.

ModelWeights / selectionRemoved branchesOriginal accuracyCompact accuracyForward speedup
ResNet-50Frozen / additive3/1675.45%56.05%1.210×
ResNet-50Frozen / interaction3/1675.45%62.30%1.261×
ResNet-34Recovered / shared1/1672.840%71.890%1.077×
ResNet-34Recovered / shared2/1672.840%70.885%1.168×
ResNet-50Recovered / shared2/1675.890%74.455%1.191×
ResNet-50Recovered / interaction3/1675.890%73.575%1.262×

Results and analysis. At the same three-deletion budget, the frozen interaction-guided model gains 6.25 accuracy points over additive selection, with class-block 95% interval [4.45, 8.00]. After recovery, ResNet-34 with one deletion loses 0.950 points at 1.077× speedup; ResNet-50 with two deletions loses 1.435 points at 1.191×. The frozen comparison isolates the selection benefit. Distillation supplies recovery, and the shared low-deletion masks do not establish an interaction-specific selection advantage. These are fixed compact networks, with no teacher or interaction queries during deployment.

Additional tests connect these results to the interaction analysis:

  • Unqueried subnetwork search. On 640 ResNet-34 images from 128 classes, identical 56-query sets predict 120 unqueried three-deletion combinations. Joint effects improve Spearman correlation from 0.77235 to 0.89663 and reduce selection KL regret by 45.8% in this offline per-input test.
  • Total subset responses. We enumerate all masks on 10,000 images per model, split into 5,000 calibration and 5,000 evaluation images. A fixed 15-branch ResNet-34 retains 70.74% accuracy versus 73.18% for full execution. At 12 branches, a per-input error oracle raises accuracy from 48.16% to 58.32% and agreement from 53.72% to 66.62% over fixed selection. This oracle uses full-model and candidate outputs; it is an offline reference, not a deployable or accuracy-optimal selector.

Manuscript changes: Section 2.1, Eq. 1 (p. 2); Section 6 and manuscript Table 4 (p. 10); Appendix L.3.1 and Table 23 (pp. 53–54); Appendix M.2–M.3, Tables 38–43 (pp. 63–69). Table 1 above consolidates results reported across the manuscript's frozen and recovery tables.

AuthorsOpenReview ↗

General Response 【Part 3】

3. Prediction-level sparsity and cancellation (Reviewers pGTs and wVQJ)

Concern. Does coefficient concentration justify a sparse representation when magnitude ranking requires nearly the full ResNet-34 dictionary for faithful logit reconstruction? Would cancellation-aware selection perform better?

Response. Accounting for cancellation reveals small prediction-preserving supports that magnitude ranking misses. A fixed 2,000-image reconstruction cohort covers all 1,000 ImageNet classes; this is a separate experiment from Table 1's branch-selection test. Both methods use the same full-logit readout, shortcut baseline and original unit-weight coefficients, without refitting. Forward selection adds the term that most reduces joint reconstruction error.

Table 2: ResNet-34 predictions reconstructed from unchanged original coefficients. Agreement is with the full-model prediction, not ground-truth accuracy. Both methods use the same inputs and support budget.

Retained termsDictionary retainedMagnitude agreementForward agreementForward relative logit error
320.0488%0.20%74.40%0.4629
1280.1953%0.30%86.45%0.2977

Table 2 shows that 128 of 65,535 terms preserve 86.45% of predictions, with class-block 95% interval [85.00%, 88.10%]. The relative error is 0.2977: this does not meet the separate 10%-logit-error criterion behind the original 99.495% magnitude-retention figure. We report the retained fraction, agreement and error together. Selection uses the full target and decomposition, so it is an offline representation test rather than an executed-branch budget.

A separate post-hoc analysis of these same saved supports gives a 128-term Jaccard class gap of 0.002532 [0.002075, 0.002983], with two-budget Holm-adjusted p = 0.001998. Labels do not select the supports. Thus, prediction preservation and class structure occur in the same interaction sets under a common readout. These modest gaps do not imply disjoint class-specific routes or establish that core-excluded variation causes the reconstruction benefit.

Manuscript changes: Section 2.1 (pp. 2–3); Section 4 and manuscript Table 2 (pp. 7–8); Appendix K.2 (pp. 48–49); Appendix L.3.2 and Table 24 (pp. 53–56; table on p. 55); same-support class analysis, Appendix L.3.4 and Table 30 (pp. 57–58).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.