- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper propose Routing the Lottery (RTL), which extending the lottery ticket from single global winning ticket to multiple adaptive tickets resulted by different data partition. The authors analysis the parameter pathways through mask overlap, specialization, semantic alignment, and the connection with universal ticket. The main contribution is empirical/interpretability, net the pruning algorithm.
Critical strengths
The question is interesting, especially the partition-noise control.
The paper is well writing.
Critical weaknesses
The main weakness is whether the specialization revealed by RTL is genuinely discovered in the underlying network, or is instead induced by the RTL training procedure itself. The interpretation of the high-sparsity regime as a functional “structural collapse” may be stronger than the evidence supports, since the appendix notes that part of the apparent mask re-convergence can be explained by surviving weights concentrating in a few layers.
The superposition interpretation is interesting, but its strongest evidence comes mainly from the INR experiments. It is therefore unclear how broadly this conclusion generalizes across architectures and tasks.How can the authors distinguish between pre-existing specialization in the dense network and specialization induced by partition-conditioned pruning? How broadly does the proposed “structured superposition” interpretation generalize beyond the INR setting, where most of the detailed structural analysis is performed?
Questions for authors
How can the authors distinguish between pre-existing specialization in the dense network and specialization induced by partition-conditioned pruning?
Can the authors clarify to what extent the observed “structural collapse” reflects genuine functional reorganization, rather than a consequence of surviving weights concentrating in a small number of layers at extreme sparsity?
How broadly does the proposed “structured superposition” interpretation generalize beyond the INR setting, where most of the detailed structural analysis is performed?
5 comments
The revised Section 5.4 also states the scope of the result explicitly. As in the original LTH, a ticket is a property of an initialization together with the data. RTL shows how that pair organizes computation across partitions, and that the dense network trained from the same initialization uses the same positions.
Q2. Is the "structural collapse" a functional reorganization or a consequence of weights concentrating in few layers?
You quoted our own statement that much of the re-convergence is explained by layer concentration. We re-analysed this point and corrected the statement.
Our null places, in every layer, exactly as many random weights as the real mask has there, so it already contains everything that concentration in few layers can explain. Measured against this null, the tickets merge beyond chance: on INRs the excess of ticket overlap is +0.36 at 91% sparsity, falls to +0.24 at 99.1%, and rises again to +0.30 at 99.5% as the tickets begin to merge. A layer audit (surviving weights per layer, for every ticket and every pruning step) shows that no layer is empty in any ticket below 99.8% sparsity. The differentiation of the tickets, the point of their largest separation and the onset of merging therefore all take place while every layer is still populated. The statement in the submission was based on a single value at 99.8% sparsity, which lies beyond that point, and on an average over layers. At 99.8% some layers are already empty, and under layer averaging an empty layer counts as full agreement for real and random masks alike, which hides the difference. We now compute the overlap over all weights jointly, which avoids this effect.
The same holds for wider MLPs. We repeated the sweep with hidden width 512 and 1024 (805K and 3.18M prunable weights). At width 512 a single ticket of a single image loses a layer at the last pruning step, and at width 1024 no layer is ever empty.
What the collapse is. The revised text does not claim a functional reorganization. We removed the sentence in Section 5.3 that described the collapse as a point at which subnetworks can no longer maintain distinct functional roles. What the data show is that beyond a threshold that depends on the setting, task performance fails and the ticket structure changes. The form of this change differs between settings.
- On INRs the tickets merge. This happens in a regime of few surviving weights per ticket, and Section 5.3 now says so and reports the weight counts.
- On GhostNet the tickets also merge as accuracy falls to chance (pairwise Jaccard rises from 0.87 to 0.99).
- On speech we observe a different form of collapse: the tickets keep diverging (pairwise Jaccard falls from 0.89 to 0.34 at 98.7% sparsity) and the shared core erodes. Two factors likely contribute. With only K = 3 partitions the tickets have room to separate instead of being forced onto the same weights. In addition, the three acoustic environments may be more distinct from each other than the classes of an image dataset or the objects of one image. The erosion of the shared core reduces performance sharply but not to chance: at 98.7% sparsity the tickets still give about 4 dB of SI-SNR improvement.
- ResNet-18 shows no collapse up to 95% sparsity, where each ticket still holds about 560K weights.
Section 5.3 now defines collapse by the drop in performance alone and no longer by the merging of masks, describes the ticket structure at collapse separately for each setting, and reports the excess over the null instead of raw similarity.
Q3. Generality of the superposition interpretation beyond INRs
We ran the analysis of Section 5.6 on four further settings, including ResNet-18 with 11.2M prunable weights. Each value is the excess over chance: the observed metric minus the same metric on random masks with the same number of masks, mask sizes and per-layer counts (3 seeds on GhostNet and speech, 1 on ResNet-18). For the Jaccard column the observed and chance values are given in brackets. The table shows two sparsity levels, and the revised paper adds the full curves over the whole sparsity range.
| Setting | Sparsity | Overlap between adaptive tickets (pairwise Jaccard) | Share of the universal ticket found in all adaptive tickets (core ratio) | Share of adaptive weights inside the universal ticket (precision) |
|---|---|---|---|---|
| CIFAR-10, GhostNet (K = 10) | 49% | +0.51 (0.91 vs 0.40) | +0.79 | +0.34 |
| 75% | +0.53 (0.87 vs 0.34) | +0.76 | +0.38 | |
| CIFAR-100, GhostNet (K = 8) | 49% | +0.44 (0.84 vs 0.40) | +0.67 | +0.27 |
| 75% | +0.51 (0.83 vs 0.32) | +0.69 | +0.36 | |
| CIFAR-100, ResNet-18 (K = 8) | 50% | +0.51 (0.86 vs 0.35) | +0.69 | +0.25 |
| 75% | +0.64 (0.82 vs 0.18) | +0.65 | +0.44 | |
| Speech, U-Net (K = 3) | 55% | +0.41 (0.72 vs 0.31) | +0.60 | +0.31 |
| 77% | +0.41 (0.58 vs 0.17) | +0.51 | +0.36 |
In every setting and at every sparsity tested, the adaptive tickets share a core and align with the universal ticket far beyond chance, with each seed at least 20 null standard deviations away. The interpretation of the universal ticket as a superposition of adaptive tickets therefore holds on five settings, four architectures and three modalities, and not only on INRs. On INRs themselves it also holds at larger scale: with hidden width 512 and 1024, the excess of ticket overlap over the null peaks at 67 to 79% sparsity (+0.57 and +0.58, against +0.42 at width 128), with 264K and 1.04M weights per ticket.
Two points differ between settings, and the revised abstract and Section 5.6 say so. First, coverage of the universal ticket is saturated near 1 outside INRs, so there the evidence is the shared core, the overlap and the alignment with the universal ticket. Second, the form of the collapse at extreme sparsity depends on the setting, as described under Q2.
Thank you for the review. Your three questions go to the core of the interpretation. Each of them led to new analysis and to changes in the text.
Q1. Pre-existing specialization versus specialization induced by partition-conditioned pruning
Partition-conditioned pruning returns K different masks for any partition, so the existence of distinct masks proves nothing by itself. The question is whether their organization reflects the data and the network, or only the procedure. Five results show that it reflects the data and the network.