acceptodds
Under review as a conference paper at ICLR 2027

Routing the Lottery: Adaptive Subnetworks for Heterogeneous Data

Abstract

The Lottery Ticket Hypothesis (LTH) posits that large neural networks contain sparse subnetworks, or winning tickets, that can be trained in isolation to match the performance of their dense counterparts. The mask defining such a ticket is extracted from data but static at inference: a single subnetwork serves the entire data distribution, and input-conditional computation arises through activations rather than parameters. We ask whether the structure of the data distribution is also reflected at the parameter level: what family of tickets emerges when ticket extraction is conditioned on partitions of the data, and how is that family organized? To study this, we propose Routing the Lottery (RTL), an adaptive pruning framework that reveals such data-conditional subnetworks, which we term adaptive tickets. Rather than focusing on compression, we use pruning as a probe to study how computation is organized within a network under controlled data partitions. We analyze these subnetworks through mask overlap and structural decomposition calibrated against chance-level null models, uncovering systematic variation in parameter sharing across data-dependent pathways. Our results show that classical universal winning tickets can be interpreted as structured superpositions of adaptive pathways within a bounded sparsity regime. At extreme sparsity levels, this relationship largely breaks down as universal pruning converges to structurally degenerate solutions. Overall, RTL reframes pruning as an analytical tool for studying data-dependent neural pathways and provides a new lens for understanding how modular computational structure emerges in sparse deep networks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Neural network pruning, Lottery ticket hypothesis, Iterative magnitude pruning, Conditional computation, Data-dependent pruning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_qoK6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose Routing the Lottery (RTL), which extending the lottery ticket from single global winning ticket to multiple adaptive tickets resulted by different data partition. The authors analysis the parameter pathways through mask overlap, specialization, semantic alignment, and the connection with universal ticket. The main contribution is empirical/interpretability, net the pruning algorithm.

Critical strengths

The question is interesting, especially the partition-noise control.

The paper is well writing.

Critical weaknesses

The main weakness is whether the specialization revealed by RTL is genuinely discovered in the underlying network, or is instead induced by the RTL training procedure itself. The interpretation of the high-sparsity regime as a functional “structural collapse” may be stronger than the evidence supports, since the appendix notes that part of the apparent mask re-convergence can be explained by surviving weights concentrating in a few layers.

The superposition interpretation is interesting, but its strongest evidence comes mainly from the INR experiments. It is therefore unclear how broadly this conclusion generalizes across architectures and tasks.How can the authors distinguish between pre-existing specialization in the dense network and specialization induced by partition-conditioned pruning? How broadly does the proposed “structured superposition” interpretation generalize beyond the INR setting, where most of the detailed structural analysis is performed?

Questions for authors

How can the authors distinguish between pre-existing specialization in the dense network and specialization induced by partition-conditioned pruning?

Can the authors clarify to what extent the observed “structural collapse” reflects genuine functional reorganization, rather than a consequence of surviving weights concentrating in a small number of layers at extreme sparsity?

How broadly does the proposed “structured superposition” interpretation generalize beyond the INR setting, where most of the detailed structural analysis is performed?

5 comments

Thank you for the review. Your three questions go to the core of the interpretation. Each of them led to new analysis and to changes in the text.

Q1. Pre-existing specialization versus specialization induced by partition-conditioned pruning

Partition-conditioned pruning returns K different masks for any partition, so the existence of distinct masks proves nothing by itself. The question is whether their organization reflects the data and the network, or only the procedure. Five results show that it reflects the data and the network.

  1. The dense network relies on the same positions. If the tickets were only a product of partition-conditioned pruning, they would mean nothing to a network that was trained densely and has never seen a mask or a partition. We tested this by applying the RTL masks, without any retraining, to the dense network trained from the same initialization, and we extended the experiment of Section 5.4 from one image to 10 images and 3 seeds. The masks retain 2.24 dB more than copies of themselves shuffled within each layer (26 of 29 cases, p = 2e-7, and 2.9 dB on 10 of 10 images in the original setting of Table 3), 2.13 dB more than random masks with the same per-layer counts, and 1.44 dB more than the complement. The weights that RTL selects are therefore the ones the dense network uses, position by position. Within one image, removing the specialized weights of one object from the dense network degrades that object most for three of the four objects (Appendix B.9). As a control, the effect is absent for a dense network trained from another seed, which is expected because a ticket is defined relative to its initialization.
  2. The selection is learned, not inherited from the initialization. Magnitude pruning keeps the largest weights, so one could suspect that a ticket is simply the set of weights that were large at initialization, fixed before any data is seen. We tested this by building, for every RTL ticket, a mask from the initialization alone: it keeps the largest initial weights, with the same number of weights per layer as the ticket. Two comparisons rule the suspicion out. First, in the dense network the RTL tickets retain 1.33 dB more than these initialization-based masks (25 of 29 cases, p = 7e-6), so RTL selects better positions than the initial magnitudes do. Second, the initialization-based masks of different partitions are nearly identical to each other (Jaccard 0.94 to 0.99), whereas the actual tickets of different partitions overlap much less (0.57 to 0.87 across the five settings at 75% sparsity). The initialization is the same for all partitions, so this difference between tickets can only come from training on different data.
  3. Same procedure, different partition. With the algorithm, K and the capacity per mask fixed, randomizing the partition removes the gain in proportion to the noise level (Table 2 in Section 5.2). The procedure alone does not produce it.
  4. Structure that a random procedure cannot produce. Ticket overlap exceeds random masks with the same per-layer counts by large margins in five settings (see Q3). On CIFAR-10 it follows class relatedness under three independent references (WordNet, CLIP image embeddings and confusion rates), with correlations of 0.42 to 0.52 in deep layers.
  5. Gradient geometry. During joint retraining on INRs (10 images, 6 sparsity levels), we took the weights that two tickets share and compared the gradients that the two partitions produce on them. This comparison has a direct meaning: to first order, the inner product of the two gradients says how much an update computed on one partition also reduces the loss on the other. We ran the measurement twice, once with semantic partitions and once with random partitions of the same sizes, on the same images, with the same initialization and the same procedure. Semantic partitions agree far less than random ones. At the start of retraining the cosine similarity is 0.54 for semantic against 0.97 for random partitions at 23% sparsity, and 0.10 against 0.60 at 76%. The share of shared weights that receive gradients of opposite sign is 27% against 9% at 23% sparsity, and 46% against 30% at 76%. Semantic partitions show the lower cosine and the higher share of opposite signs in 60 of 60 image and sparsity combinations. The two runs differ only in the content of the partitions, since in both the masks are produced by the same procedure from the respective partition. The lower agreement therefore traces back to the semantic content of the data and not to the procedure. An update that serves one semantic partition transfers less to the others, which is consistent with the tickets keeping part of their weights separate.

The revised Section 5.4 also states the scope of the result explicitly. As in the original LTH, a ticket is a property of an initialization together with the data. RTL shows how that pair organizes computation across partitions, and that the dense network trained from the same initialization uses the same positions.

Q2. Is the "structural collapse" a functional reorganization or a consequence of weights concentrating in few layers?

You quoted our own statement that much of the re-convergence is explained by layer concentration. We re-analysed this point and corrected the statement.

Our null places, in every layer, exactly as many random weights as the real mask has there, so it already contains everything that concentration in few layers can explain. Measured against this null, the tickets merge beyond chance: on INRs the excess of ticket overlap is +0.36 at 91% sparsity, falls to +0.24 at 99.1%, and rises again to +0.30 at 99.5% as the tickets begin to merge. A layer audit (surviving weights per layer, for every ticket and every pruning step) shows that no layer is empty in any ticket below 99.8% sparsity. The differentiation of the tickets, the point of their largest separation and the onset of merging therefore all take place while every layer is still populated. The statement in the submission was based on a single value at 99.8% sparsity, which lies beyond that point, and on an average over layers. At 99.8% some layers are already empty, and under layer averaging an empty layer counts as full agreement for real and random masks alike, which hides the difference. We now compute the overlap over all weights jointly, which avoids this effect.

The same holds for wider MLPs. We repeated the sweep with hidden width 512 and 1024 (805K and 3.18M prunable weights). At width 512 a single ticket of a single image loses a layer at the last pruning step, and at width 1024 no layer is ever empty.

What the collapse is. The revised text does not claim a functional reorganization. We removed the sentence in Section 5.3 that described the collapse as a point at which subnetworks can no longer maintain distinct functional roles. What the data show is that beyond a threshold that depends on the setting, task performance fails and the ticket structure changes. The form of this change differs between settings.

  • On INRs the tickets merge. This happens in a regime of few surviving weights per ticket, and Section 5.3 now says so and reports the weight counts.
  • On GhostNet the tickets also merge as accuracy falls to chance (pairwise Jaccard rises from 0.87 to 0.99).
  • On speech we observe a different form of collapse: the tickets keep diverging (pairwise Jaccard falls from 0.89 to 0.34 at 98.7% sparsity) and the shared core erodes. Two factors likely contribute. With only K = 3 partitions the tickets have room to separate instead of being forced onto the same weights. In addition, the three acoustic environments may be more distinct from each other than the classes of an image dataset or the objects of one image. The erosion of the shared core reduces performance sharply but not to chance: at 98.7% sparsity the tickets still give about 4 dB of SI-SNR improvement.
  • ResNet-18 shows no collapse up to 95% sparsity, where each ticket still holds about 560K weights.

Section 5.3 now defines collapse by the drop in performance alone and no longer by the merging of masks, describes the ticket structure at collapse separately for each setting, and reports the excess over the null instead of raw similarity.

Q3. Generality of the superposition interpretation beyond INRs

We ran the analysis of Section 5.6 on four further settings, including ResNet-18 with 11.2M prunable weights. Each value is the excess over chance: the observed metric minus the same metric on random masks with the same number of masks, mask sizes and per-layer counts (3 seeds on GhostNet and speech, 1 on ResNet-18). For the Jaccard column the observed and chance values are given in brackets. The table shows two sparsity levels, and the revised paper adds the full curves over the whole sparsity range.

SettingSparsityOverlap between adaptive tickets (pairwise Jaccard)Share of the universal ticket found in all adaptive tickets (core ratio)Share of adaptive weights inside the universal ticket (precision)
CIFAR-10, GhostNet (K = 10)49%+0.51 (0.91 vs 0.40)+0.79+0.34
75%+0.53 (0.87 vs 0.34)+0.76+0.38
CIFAR-100, GhostNet (K = 8)49%+0.44 (0.84 vs 0.40)+0.67+0.27
75%+0.51 (0.83 vs 0.32)+0.69+0.36
CIFAR-100, ResNet-18 (K = 8)50%+0.51 (0.86 vs 0.35)+0.69+0.25
75%+0.64 (0.82 vs 0.18)+0.65+0.44
Speech, U-Net (K = 3)55%+0.41 (0.72 vs 0.31)+0.60+0.31
77%+0.41 (0.58 vs 0.17)+0.51+0.36

In every setting and at every sparsity tested, the adaptive tickets share a core and align with the universal ticket far beyond chance, with each seed at least 20 null standard deviations away. The interpretation of the universal ticket as a superposition of adaptive tickets therefore holds on five settings, four architectures and three modalities, and not only on INRs. On INRs themselves it also holds at larger scale: with hidden width 512 and 1024, the excess of ticket overlap over the null peaks at 67 to 79% sparsity (+0.57 and +0.58, against +0.42 at width 128), with 264K and 1.04M weights per ticket.

Two points differ between settings, and the revised abstract and Section 5.6 say so. First, coverage of the universal ticket is saturated near 1 outside INRs, so there the evidence is the shared core, the overlap and the alignment with the universal ticket. Second, the form of the collapse at extreme sparsity depends on the setting, as described under Q2.

Reviewer_S6DpOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The authors introduce RTL (Routing The Lottery) a method which subdivides the dataset in several clusters (the method is independent of the clustering approach), and aims to find a sparse Lottery Ticket structure (adaptive ticket) associated with each cluster. These tickets are then trained via an interleaved approach, and used in inference with argmax over the different structures. While the main purpose of this method is defined as an analytical tool for interpreting which structures in a network correspond to which classes, the authors show that this approach also results in better performance compared to 'universal' lottery tickets.

Critical strengths
  • Thoroughness and clarity. The paper is well structured and easy to follow. The algorithms are clear and well illustrated, it includes both experiments showing the performance of the method w.r.t. the original IMP approach as well as an in-depth analysis of the behaviour of the method in different scenarios (e.g. noise , few-shot transferability)

  • Novelty: The paper opens up a fresh perspective by using pruning for model interpretation and probing.

Critical weaknesses
  • Practical applicability: Simple settings. The most complex settings that was considered is ResNet-18 on CIFAR-100, which is a relatively simple baseline. Can you verify the applicability of this method on more complex settings?

  • The analysis on performance is a bit limited. I would have liked a more thorough analysis on why exactly RTL outperforms the 'universal' lottery tickets. Could this be linked to the training methodology as well? In my opinion the comparison between universal and RTL tickets is also a bit flawed, as universal tickets have the same weights for all classes, while RTL tickets can benefit from this specialization to have more effective capacity for only a single class. I feel the paper could be a bit more convincing in that aspect.

Questions for authors
  • What are the typical values of T in this paper? You claim to have roughly twice the training cost of regular IMP, which fully trains the network each iteration. This implies that your subnetworks are not fully trained during each iteration, and in fact that if K increases then T decreases. Is this a conscious deviation of the original approach? Can you motivate why, and / or give some intuition on the influence of T?
  • Did you in your experiments ever encounter destructive inference between different subnetworks?
  • What is the typical pruning rate of the full model? Are there any connections that are not used at all, and as such removed completely from the network?
  • The zero-shot mask transfer shows that its structures transfer well to other random initializations. However, the strength of IMP lies in the combination of initial weights and connections as shown in Frankle et al. 2021, compared to other pruning-at-initialization methods. Would you say that this is merely because your approach finds good layerwise sparsity rates? Have you considered using layerwise mask randomization to validate the structural hypothesis?

References

Frankle et al. 2021 "Pruning Neural Networks at Initialization: Why are We Missing the Mark?", ICLR 2021

5 comments

Thank you for the constructive review and the concrete questions.

W1. Complexity of the settings

We agree that the settings are compact, and we want to be clear about why. The paper is an analysis of a phenomenon and does not target deployment. It builds directly on the Lottery Ticket Hypothesis, whose original study made similar choices (MLPs, small convolutional networks and ResNets). Both IMP and RTL are computationally expensive, because every sparsity level requires training runs, and our analyses repeat complete sparsity sweeps over seeds, partitions and controls. We hope that this restriction will be eased in future work, by us or by other groups, and we believe that the analysis is substantial enough to be worth presenting to the community at this stage.

Within these limits we extended the scale. A clarification first: the CIFAR experiments in the submission use a truncated GhostNet (about 126K prunable weights, Appendix C), not ResNet-18. We have now run ResNet-18 on CIFAR-100 (K = 8, 11.2M prunable weights, one seed):

Balanced accuracy25%50%75%
RTL0.8190.8050.810
IMP0.7580.7460.744
IMP 2× (over budget)0.7630.7520.759
Union mask, single network0.7520.7420.739

RTL leads by about 0.06 over IMP and by about 0.05 over IMP with twice the training. At 95% sparsity RTL still reaches 0.787 against 0.743 for IMP, with no collapse. The structural analysis reproduces as well. At 50% sparsity the overlap between adaptive tickets is 0.86 against 0.35 for random masks with the same per-layer counts, and the share of the universal ticket that is found in all adaptive tickets exceeds chance by 0.69. On GhostNet, three seeds confirm Table 1 (CIFAR-10: 0.780 / 0.774 / 0.762, CIFAR-100: 0.764 / 0.750 / 0.758, standard deviation at most 0.006). The results now cover five settings, four architectures and three modalities.

W2. Why RTL outperforms universal tickets

You raise two possible explanations: the training procedure, and the extra effective capacity of a ticket that serves only one partition. We tested both with single-network controls. Each cell gives the values at 25 / 50 / 75% mask sparsity:

SettingRTLIMPIMP 2× (over budget)Union mask, single network
INR, MLP, PSNR in dB18.95 / 17.26 / 14.7916.85 / 15.38 / 13.0418.08 / 16.18 / 13.4817.66 / 16.74 / 14.84
CIFAR-10, GhostNet0.780 / 0.774 / 0.7620.741 / 0.746 / 0.7360.766 / 0.766 / 0.7520.726 / 0.719 / 0.700
CIFAR-100, GhostNet0.764 / 0.750 / 0.7580.673 / 0.671 / 0.7000.669 / 0.669 / 0.7060.685 / 0.684 / 0.694
CIFAR-100, ResNet-180.819 / 0.805 / 0.8100.758 / 0.746 / 0.7440.763 / 0.752 / 0.7590.752 / 0.742 / 0.739
Speech, U-Net, SI-SNRi in dB7.25 / 7.20 / 6.986.89 / 7.00 / 6.987.79 / 7.81 / 7.636.76 / 6.82 / 6.84

INR values are means over 3 seeds and 10 images, from a dedicated rerun with new seeds in which the final masks of all five variants are retrained from the same initializations under one protocol, so that the columns are directly comparable. Speech and GhostNet values are means over 3 seeds, with standard deviations of at most 0.09 dB on speech and 0.006 on CIFAR, and they replace the single-seed values of Table 1. ResNet-18 is one seed. CIFAR rows report balanced accuracy.

Training procedure. In both methods the evaluated network is obtained in the same way: the final mask is applied, the weights are rewound to initialization, and the network is retrained. IMP at the same budget is therefore the matched baseline, and RTL is ahead of it in every setting: by 1.75 to 2.1 dB on INRs (in 30 of 30 seed and image pairs at every sparsity), by 0.026 to 0.039 on CIFAR-10, by 0.058 to 0.091 on CIFAR-100, by about 0.06 on ResNet-18, and by 0.36 and 0.20 dB on speech at 25% and 50%. On speech at 75% the two methods are on par. Extra training does not close the gap. On INRs IMP with twice the training is still 0.9 to 1.3 dB behind. Even with K times the training, where K is the number of clusters, it is ahead only at 25% and behind at 50% and 75%. On CIFAR the IMP baseline already receives more training epochs than RTL's evaluated network, as noted in Appendix C, and doubling it again still leaves RTL ahead: by 0.008 to 0.014 on CIFAR-10 (3 to 5 standard deviations), by 0.05 to 0.10 on CIFAR-100 and by about 0.05 on ResNet-18. On speech, IMP with twice the training is above RTL at the standard budget. This is expected for an over-budget reference on a task where training has not yet converged, and it is why we compare at equal epochs.

Capacity. You are right that the adaptive tickets together store more weights than one universal ticket. Four controls show that this is not what produces the gain.

  1. One unrouted network with the union of the RTL masks, trained on all the data, stores exactly RTL's weights. It is below RTL at every sparsity by 0.05 to 0.08 on GhostNet, by 0.06 to 0.07 on ResNet-18 and by 0.14 to 0.49 dB on speech. On INRs it is 1.3 dB below RTL at 25% and 0.5 dB below at 50%, and it reaches RTL only at 75% (see point 3).
  2. RTL union of masks (measured at 25 / 50 / 75%):
DomainIMP stored weightsRTL stored weights (union)RTL union sparsity
INR40.4K / 26.9K / 13.5K47.9K / 35.4K / 20.5K11.2 / 34.3 / 61.9%
CIFAR-1094.3K / 62.9K / 31.5K103.2K / 71.2K / 38.2K18.0 / 43.5 / 69.7%
CIFAR-10094.3K / 62.9K / 31.5K106.6K / 74.5K / 38.3K15.3 / 40.8 / 69.6%
Speech28.0K / 18.7K / 9.3K30.2K / 20.9K / 11.6K19.2 / 44.0 / 68.9%

Offsetting the sparsity levels gives a comparison at matched storage. RTL at 50% stores fewer weights than IMP at 25% in all four domains and scores higher in all four: 17.26 against 16.85 dB on INRs (35.4K against 40.4K weights), 0.774 against 0.741 on CIFAR-10 (71.2K against 94.3K), 0.750 against 0.673 on CIFAR-100 (74.5K against 94.3K) and 7.20 against 6.89 dB on speech (20.9K against 28.0K). At the next offset, RTL at 75% against IMP at 50%, RTL is again ahead on CIFAR-10 (0.762 against 0.746, with 38.2K against 62.9K weights) and on CIFAR-100 (0.758 against 0.671), and on speech it matches IMP (6.98 against 7.00 dB) with 38% fewer stored weights. 3. Storage is the pessimistic comparison for RTL. At inference each input activates exactly one ticket, whereas a single network that stores the union uses all of it for every input. On INRs at 75% mask sparsity this is 20.5K active weights for the union network against 13.5K for RTL. So even in the one case where the union network reaches RTL's quality, it needs about 50% more active weights and FLOPs per input. 4. RTL with a fully randomized partition has the same number of masks and the same capacity per mask as semantic RTL, yet it loses the gain and performs at or below the IMP level at 50% and 75% sparsity (Table 2 in Section 5.2). If the gain were driven by effective capacity, it would survive randomization. It does not.

Mechanism. The interference measurement under Q2 points to potential reason. Semantic partitions place conflicting demands on shared weights, and a single network has to average them, while RTL gives each partition its own ticket.

Q1. Values of T, and whether T shrinks with K

T is the mask-training budget of one ticket in one pruning round. It is 10,000 optimizer steps on INRs, 10 epochs on CIFAR (GhostNet and ResNet-18) and 20 epochs on speech. T is counted over the ticket's own partition and does not depend on K, so tickets are not under-trained when K grows. Because the partitions are disjoint, one epoch of mask training over all K tickets passes over the dataset once (on CIFAR each one-vs-rest classifier also sees negatives).

This training serves only to find the masks. After extraction the weights are rewound to initialization, so nothing learned in this phase is carried into joint retraining apart from the masks themselves. The evaluated RTL network is then retrained with the same number of steps as the IMP ticket on INRs, the same number of epochs on speech, and fewer epochs than the IMP ticket on CIFAR.

On cost, the sentence in Section 4 was imprecise and is corrected. To reach one target sparsity after n pruning rounds, both methods need n mask-finding rounds plus one final training of the ticket. The factor of two arises only when every level of a sparsity sweep is evaluated, which is what we did in order to characterize the method over the full range. In that case RTL needs a separate joint retraining at every level, used only for evaluation, whereas IMP reuses each retraining as the next pruning signal. Each ticket follows the original IMP protocol. The only deviation is that tickets are trained on subsets of the data.

Q2. Destructive interference between subnetworks

We measured it during joint retraining, on the weights that two tickets share (10 images, 6 sparsities), for semantic partitions and for random partitions of the same sizes:

Mask sparsityGradient cosine at start, semantic / randomGradient cosine at end, semantic / randomSign conflicts at start, semantic / random
22.8%0.54 / 0.970.31 / 0.820.27 / 0.09
53.2%0.22 / 0.810.30 / 0.620.40 / 0.19
76.0%0.10 / 0.600.28 / 0.510.46 / 0.30

Semantic partitions disagree far more than random ones. Their gradient cosine is lower by 0.38 on average and their share of sign conflicts higher by 0.11, in 60 of 60 image and sparsity combinations (Wilcoxon p = 2e-11). Interference between tickets is therefore present and measurable, but it is partial and not destructive: the average cosine stays positive, settles near 0.3 by the end of retraining, and joint retraining reaches the performance reported above. The measurement also shows why separate tickets help. Partitions that differ semantically pull shared weights in different directions, and a single network must average these gradients, whereas RTL lets every partition keep weights that the others do not have to share.

Q3. Pruning rate of the full model, and connections removed from every mask

Yes. Weights that are pruned from all K masks are removed from the network. At 25 / 50 / 75% mask sparsity the full model is 11.2 / 34.3 / 61.9% sparse for INRs, 18.0 / 43.5 / 69.7% for CIFAR-10, 15.3 / 40.8 / 69.6% for CIFAR-100 and 19.2 / 44.0 / 68.9% for speech (table under W2, also in Appendix F).

Q4. Zero-shot transfer, layerwise sparsity, and mask randomization

One clarification: in Section 5.4 the masks are transferred to a fully trained, unpruned network from the same initialization, not to other random initializations. We ran the control you suggest on 10 images and 3 seeds (29 cases) at 50% sparsity, and added a second control that keeps the largest initial weights. Zero-shot change in PSNR relative to the dense model:

MaskChange in PSNR (dB)
RTL, each object through its own ticket-1.22
Universal ticket (IMP)-1.34
Largest initial weights, same per-layer counts as RTL-2.54
Complement of the RTL tickets-2.65
Random, same per-layer counts-3.35
RTL, shuffled within each layer-3.46
  • It is not layerwise sparsity. A mask shuffled within each layer keeps the layerwise sparsity rates exactly and destroys only the positions. RTL masks retain 2.24 dB more than their layer-shuffled copies (26 of 29 cases, p = 2e-7), and the universal ticket retains 1.85 dB more than its shuffled copy (24 of 29, p = 3e-6). In the original setting of Table 3, extended to 10 images, the RTL advantage over shuffled masks is 2.9 dB, on 10 of 10 images. The positions matter, not only the per-layer rates.
  • It is not the initialization alone. RTL masks retain 1.33 dB more than masks that keep the largest initial weights with the same per-layer counts (25 of 29 cases, p = 7e-6). About 60% of the advantage over shuffled masks is therefore learned during training and not inherited from the initial magnitudes.
  • Tickets remain tied to their initialization. Applied to a dense network trained from a different seed, the masks are at the level of shuffled masks (-3.54 against -3.41 dB). This agrees with your remark that the strength of IMP lies in the combination of initial weights and connections, and Section 5.4 now states it explicitly.
Reviewer_86x3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether pruning conditioned on subsets of the data reveals shared and specialized parameter structures. The proposed Routing the Lottery (RTL) framework partitions dataset into subsets, then searches for winning tickets corresponds to each subset of data from a common initialization via Iterative Magnitude Pruning (IMP) algorithm. Those tickets are then jointly retrained on a shared network parameters. Experiments cover various tasks and data modalities from implicit neural representations (INR), image classification, and speech enhancement. The central empirical claim is that universal lottery ticket can be interpreted as structured superpositions of adaptive pathways within a bounded sparsity level.

Critical strengths
  1. The research question is interesting. Moving from the existence of a single winning ticket to the organization of a family of data-dependent tickets is a worthwhile direction for interpretability.
  2. The proposed RTL framework is intuitive. RTL builds on the familiar IMP algorithm to find adaptive tickets for data clusters. This makes the framework relatively easy to understand and connects the analysis directly to the established Lottery Ticket Hypothesis (LTH).
  3. Evaluations are performed across several tasks, including reconstruction, image classification, and speech enhancement. The empirical observations provide some interesting insights. In particular, the partition-noise experiment in Section 5.2 provides evidence that meaningful partitions matter for subnetworks’ performance. Section 5.6 empirically shows that universal parameters of LTH are reused across pathways.
Critical weaknesses
  1. The performance comparisons do not fully isolate the contribution of adaptive routing. Section 4 states that RTL incurs approximately twice the total training cost of IMP, moreover, the union of RTL masks generally contains more parameters than IMP mask.
  2. Although the authors provide empirical results on diverse tasks, each task is only studied on a specific architecture (INR with MLPs, vision tasks with GhostNet, speech enhancement with light-UNet). Besides, the detailed structural conclusions rely primarily on one small INR architecture.
  3. The general idea that using sparse networks to decompose shared and task-specific knowledge has been studied in sparse continual learning literature. Chen et. al., [1] extend lottery-ticket discovery to lifelong learning. NISPA [2] explicitly preserves stable pathways while creating new plastic pathways that can reuse previously learned representations, and investigates representation reuse across related tasks.
  4. Some empirical results are not convincing in the current form. In particular, in Section 5.3, the RTL are run on small MLP networks (with total ~54000 params as described in Appendix B.3), when the sparsity level becomes extreme (>99%) the retained number of params drops to < 540 params. With those params, claims from Phase II and III are limited. In Section 5.5, measuring the alignment between task-specific patterns and semantic relationships among class labels to assess the semantic alignment of adaptive subnetworks needs justification.

[1] Tianlong Chen et al. Long Live the Lottery: The Existence of Winning Tickets in Lifelong Learning. ICLR 2021.

[2] Mustafa Burak Gurbuz and Constantine Dovrolis. NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks. ICML 2022.

Questions for authors
  1. How does RTL compare with a single union-mask subnetwork trained on D\mathcal{D}, and with IMP under matched total training compute and comparable retained-parameter budgets?
  2. Which structural findings are intended as the principal advance over sparse continual-learning works such as [1, 2]?
6 comments

Thank you for the careful review. We ran the controls you asked for, extended the structural analysis to four further settings including ResNet-18, added the continual-learning works to Section 2, and sharpened the claims in Sections 5.1, 5.3, 5.4 and 5.5.

W1 / Q1. Isolating the contribution of adaptive tickets

Accounting. The "twice the cost" sentence in Section 4 was imprecise, and we corrected it. In both methods the evaluated network is obtained in the same way: the final mask is applied, the weights are rewound to initialization, and the network is retrained. To reach one target sparsity after n pruning rounds, IMP needs n training runs to find the mask plus one to train the final ticket, and RTL needs n extraction rounds plus one joint retraining. One RTL round processes every sample once: each subnetwork is trained only on its own partition, and the partitions are disjoint and together cover the dataset without repeats. One RTL round therefore costs the same as one IMP round, and total compute is equal. On INRs and speech the RTL tickets and the IMP ticket are also trained on the same amount of data. On CIFAR each one-vs-rest classifier additionally sees negatives, and the IMP baseline is given more epochs to compensate (see below). The factor of two arises only when every level of a sparsity sweep is evaluated, which is what we did in order to characterize the method over the full range. In that case RTL needs one training for mask estimation and a separate joint retraining at every level, the latter only for evaluation, whereas IMP reuses each retraining as the next pruning signal. IMP at the same budget is thus the matched baseline. We add two over-budget references: IMP 2× (twice the training) and for INRs IMP K× (K times the training, where K is the number of clusters).

Results. Each cell gives the values at 25 / 50 / 75% mask sparsity:

SettingRTLIMPIMP 2× (over budget)Union mask, single network
INR, MLP, PSNR in dB18.95 / 17.26 / 14.7916.85 / 15.38 / 13.0418.08 / 16.18 / 13.4817.66 / 16.74 / 14.84
CIFAR-10, GhostNet0.780 / 0.774 / 0.7620.741 / 0.746 / 0.7360.766 / 0.766 / 0.7520.726 / 0.719 / 0.700
CIFAR-100, GhostNet0.764 / 0.750 / 0.7580.673 / 0.671 / 0.7000.669 / 0.669 / 0.7060.685 / 0.684 / 0.694
CIFAR-100, ResNet-180.819 / 0.805 / 0.8100.758 / 0.746 / 0.7440.763 / 0.752 / 0.7590.752 / 0.742 / 0.739
Speech, U-Net, SI-SNRi in dB7.25 / 7.20 / 6.986.89 / 7.00 / 6.987.79 / 7.81 / 7.636.76 / 6.82 / 6.84

INR values are means over 3 seeds and 10 images, from a dedicated rerun with new seeds in which the final masks of all five variants are retrained from the same initializations under one protocol, so that the columns are directly comparable. Speech and GhostNet values are means over 3 seeds, with standard deviations of at most 0.09 dB on speech and 0.006 on CIFAR, and they replace the single-seed values of Table 1. ResNet-18 is one seed. CIFAR rows report balanced accuracy.

Matched budget (IMP). RTL is ahead in every setting: by 1.75 to 2.1 dB on INRs (in 30 of 30 seed and image pairs at every sparsity), by 0.026 to 0.039 on CIFAR-10, by 0.058 to 0.091 on CIFAR-100, by about 0.06 on ResNet-18, and by 0.36 and 0.20 dB on speech at 25% and 50%. On speech at 75% the two methods are on par.

Over-budget references. IMP 2× gives IMP twice its training. On INRs it is still 0.9 to 1.3 dB behind RTL. Even with K times the training (19.20 / 16.95 / 13.94 dB) IMP is ahead only at 25% and behind at 50% and 75%. On CIFAR the IMP baseline already receives more training epochs than RTL's evaluated network, as noted in Appendix C. Doubling it again still leaves RTL ahead: by 0.008 to 0.014 on CIFAR-10 (3 to 5 standard deviations), by 0.05 to 0.10 on CIFAR-100 and by about 0.05 on ResNet-18. On speech, IMP with twice the training is above RTL at the standard budget. This is expected for an over-budget reference on a task where training has not yet converged, and it is why we compare at equal epochs.

Union-mask single network (your Q1). One unrouted network with the union of the RTL masks, trained on all of D, stores exactly RTL's weights. It is below RTL at every sparsity by 0.05 to 0.08 on GhostNet, by 0.06 to 0.07 on ResNet-18 and by 0.14 to 0.49 dB on speech. On INRs it is 1.3 dB below RTL at 25% and 0.5 dB below at 50%, and on par at 75%. In four of the five settings the union never recovers the gain, and on INRs it does so only at 75% (with much higher FLOPs per single inference).

Matched stored parameters. We agree that mask-level sparsity and the total number of stored weights should be distinguished. RTL stores the union of its masks (at 25 / 50 / 75%):

DomainIMP stored weightsRTL stored weights (union)RTL union sparsity
INR40.4K / 26.9K / 13.5K47.9K / 35.4K / 20.5K11.2 / 34.3 / 61.9%
CIFAR-1094.3K / 62.9K / 31.5K103.2K / 71.2K / 38.2K18.0 / 43.5 / 69.7%
CIFAR-10094.3K / 62.9K / 31.5K106.6K / 74.5K / 38.3K15.3 / 40.8 / 69.6%
Speech28.0K / 18.7K / 9.3K30.2K / 20.9K / 11.6K19.2 / 44.0 / 68.9%

This permits a comparison at matched storage by offsetting the sparsity levels. RTL at 50% stores fewer weights than IMP at 25% in all four domains and scores higher in all four: 17.26 against 16.85 dB on INRs (35.4K against 40.4K weights), 0.774 against 0.741 on CIFAR-10 (71.2K against 94.3K), 0.750 against 0.673 on CIFAR-100 (74.5K against 94.3K) and 7.20 against 6.89 dB on speech (20.9K against 28.0K). At the next offset, RTL at 75% against IMP at 50%, RTL is again ahead on CIFAR-10 (0.762 against 0.746, with 38.2K against 62.9K weights) and on CIFAR-100 (0.758 against 0.671), and on speech it matches IMP (6.98 against 7.00 dB) with 38% fewer stored weights.

Total storage is also the pessimistic comparison for RTL. At inference each input activates exactly one adaptive mask, so the active parameters and FLOPs per input are at the single-mask level, well below the IMP baseline in the offset comparison above.

Finally, the random-partition control (Table 2 in Section 5.2) gives an independent answer to the capacity question. RTL with a fully randomized partition has the same number of masks and the same capacity per mask as semantic RTL, yet it loses the gain and performs at or below the IMP level at 50% and 75% sparsity. If the gains were driven by effective capacity, they would survive randomization. They do not.

W2. One architecture per task, structural conclusions from one small INR

A note on the choice of models. Each task uses an architecture that is standard for it: a coordinate MLP for implicit neural representations, GhostNet and ResNet for image classification, and a U-Net for speech enhancement. We kept the models compact because the analyses require complete sparsity sweeps with joint retraining at every level, repeated over seeds, partitions and controls. This follows the original LTH study, which also worked with MLPs, small convolutional networks and ResNets.

To address your point directly, we ran the null-calibrated analysis of Section 5.6 on four further settings, including ResNet-18 with 11.2M prunable weights. Each value is the excess over chance: the observed metric minus the same metric on random masks with the same number of masks, mask sizes and per-layer counts (3 seeds on GhostNet and speech, 1 on ResNet-18). For the Jaccard column the observed and chance values are given in brackets. The table shows 50% sparsity, and the revised paper adds the full curves over the whole sparsity range.

Setting, about 50% sparsityOverlap between adaptive tickets (pairwise Jaccard)Share of the universal ticket found in all adaptive tickets (core ratio)Share of adaptive weights inside the universal ticket (precision)
CIFAR-10, GhostNet (K = 10)+0.51 (0.91 vs 0.40)+0.79+0.34
CIFAR-100, GhostNet (K = 8)+0.44 (0.84 vs 0.40)+0.67+0.27
CIFAR-100, ResNet-18 (K = 8)+0.51 (0.86 vs 0.35)+0.69+0.25
Speech, U-Net (K = 3)+0.41 (0.72 vs 0.31)+0.60+0.31

The above-chance structure is present in every setting and at every sparsity tested, with each seed at least 20 null standard deviations away. The structural conclusions therefore hold on five settings, four architectures and three modalities.

Two points differ between settings, and the revised Section 5.6 says so. First, coverage is saturated near 1 outside INRs, so there the evidence is the shared core, the overlap and the alignment with the universal ticket. Second, the form of the collapse at extreme sparsity differs (see also W4a). On INRs and GhostNet the tickets merge. On speech we observe a different form of collapse: the tickets keep diverging (pairwise Jaccard falls from 0.89 to 0.34 at 98.7% sparsity) and the shared core erodes. Two factors likely contribute. With only K = 3 partitions the tickets have room to separate instead of being forced onto the same weights. In addition, the three acoustic environments may be more distinct from each other than the objects of one image or the classes of an image dataset, which makes them easier to separate. The erosion of the shared core reduces performance sharply but not to chance: at 98.7% sparsity the tickets still give 4.10 dB of SI-SNR improvement and remain useful. ResNet-18 shows no collapse up to 95% sparsity. At that point each ticket still holds about 560K weights, far more than the 15K to 28K at which GhostNet collapses, so we expect its collapse to lie at higher sparsity than we pruned to.

W3 / Q2. Relation to sparse continual learning [1, 2]

Thank you for the pointers. Both are now discussed in Section 2.

Our work differs from them first in its aim. We build directly on the Lottery Ticket Hypothesis and use pruning as a probe to analyse a phenomenon: how the tickets of different parts of one dataset relate to each other and to the universal ticket. We do not propose a method for deployment or for learning tasks in sequence. Chen et al. [1] search for a single lifelong ticket that remains trainable as classes arrive sequentially, with bottom-up pruning and distillation to counter forgetting. NISPA [2] keeps a fixed-density network, freezes the inputs of units that became stable on earlier tasks, and rewires connections so that new tasks can reuse them. In both, what is shared between tasks is shaped by arrival order and by a stability constraint: the first task chooses freely and later tasks build on what is frozen. In RTL there is no order and no constraint. All tickets are extracted symmetrically from one initialization with the unmodified LTH prune-and-rewind protocol, so the overlap between tickets is an outcome that can be measured and tested, not a design decision.

This is what makes the following findings possible, and they are the ones we regard as the principal advance over [1, 2]:

  1. The universal ticket in relation to the adaptive family. Universal and adaptive tickets come from the same initialization, so they can be compared weight by weight. Against count-matched random masks, the adaptive tickets share a core and align with the universal ticket far beyond chance in all five settings (W2). Neither [1] nor [2] extracts both kinds of ticket, so this relation is outside their scope.
  2. Sharing and specialization along the pruning trajectory. We follow the structure over the whole sparsity range and find a differentiation regime whose above-chance structure peaks at 67 to 79% sparsity at every MLP width tested, with up to 1.04M weights per ticket (W4a).
  3. Sharing follows the data. Because the tickets are exchangeable, their overlap can be tested against the data. The gain is lost as the partition is randomized (Table 2 in Section 5.2), ticket overlap follows class relatedness under three independent references (see W4b below), and on shared weights the gradients of semantic partitions conflict while those of random partitions agree.
  4. Link to the dense network. The positions selected by the tickets matter in a densely trained, never pruned network from the same initialization. The tickets retain 2.2 to 2.9 dB more than layer-shuffled copies of themselves. They also retain 1.3 dB more than masks that keep the largest initial weights with the same per-layer counts (25 of 29 cases, p = 7e-6), so the selection is learned and not inherited from the initialization.

W4a. Extreme sparsity leaves very few parameters in the INR

We repeated the INR sweep with hidden width 512 and 1024 (805K and 3.18M prunable weights, same 10 images) and now report surviving weights per ticket next to sparsity. It helps to separate two parts of the pruning trajectory, because your concern is answered differently for each. In the differentiation regime (Phases I and II) the tickets diverge from each other while their structure stays far above chance. At extreme sparsity task performance collapses (Phase III).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.