- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper proposes an active-set method for learning binary masks, called Boundary. Using interval bounds on score magnitudes, coordinates whose mask membership cannot change within an update interval are fixed. Scores and optimizer state are kept only for the remaining boundary coordinates, and this set is refreshed periodically. Optionally, masks in an initial network prefix are frozen to skip their backward pass. On image classification and language modeling with different architectures and datasets, the method reduces controller state by roughly 75–83% with quality close to full-score training.
Critical strengths
The paper targets a real and practically relevant cost. Dense scores, gradients, and optimizer state maintained during mask learning can be substantial at LLM scale.
The method leaves the host's mask objective, sparsity quota and update rule unchanged, so it can in principle be layered on top of existing mask-learning methods.
The experiments cover several model families, including CNNs, a vision transformer, a random-weight subnetwork setup and a 1.4B language model. Across these, controller state is reduced substantially while quality stays close to the full-score baseline.
The paper separates logical state savings from actual peak memory and runtime effects, and it reports cases where the method does not help. The ablations comparing static and eventwise refresh, fixed bands and random active sets are informative.
Critical weaknesses
Proposition 1 is obvious. It essentially states that if score changes are bounded, elements far enough from the current Top-k threshold cannot cross it. In addition, the final algorithm does not guarantee the radii by bounding the updates, e.g. via the number of iterations. Instead, scores are simply clamped to keep the active set valid (Eq. 7). This is unprincipled and makes the method appear ad hoc.
The fixed prefix is trivial. Sec. 3.3 is standard chain-rule reasoning: if the first part of the computation graph is frozen, the backward pass can stop early. Not optimizing masks in the first few layers obviously reduces memory and compute. Optionally fixing the first layers should not be presented as a pronounced part of the algorithm, since it is simply a cost reduction from not applying the method to the whole model.
The fixed mask entries would still receive score updates that can matter later, but these are ignored entirely. The theory only guarantees that binary mask entries do not change within the clipped interval. It does not justify discarding these updates relative to full-score training. The experiments show this does not hurt performance significantly, but they also show that Hard-freeze may not hurt much either. This raises the question of whether the active-set machinery is necessary at all. Better comparisons against simpler compute-saving baselines are needed. Additionally, the main vision Full-score baseline does not use the interval projection that Boundary uses, so the comparison does not isolate the effect of the active set.
Vision runtime gains are weak and sometimes negative. Boundary without frontier barely changes runtime or peak memory. With the frontier, median step time improves, but total setup plus training time is only slightly better on ResNet-50 and ConvNeXt-T and worse on DeiT-B.
The 9.2x Pythia speedup is heavily confounded. The paper states that Full-score uses stable four-score sorting while Boundary uses direct rank comparisons on active groups. The reported speedup therefore stems not only from the active-set idea but also from a possibly cheaper implementation for Boundary. A fair comparison would give Full-score an equally optimized top-2-of-4 selection.
Inconsistent statistical reporting. Three seeds for the ImageNet experiments are fine. However, the Pythia quality results are reported without variance, even though the differences between methods are tiny. Gaps of this size do not allow any conclusions without uncertainty estimates.
The prose is hard to follow in some place, due to confusing or undefined terms. Some examples are:
- l. 076: "Boundary with the frontier"
- l. 080: "reduces controller state"
- l. 156: "retains k of n positions independently within each group"
- l. 206: "training events"
- l. 267: "scheduled event"
- l. 272: "logical scores"
Questions for authors
Why is the main ImageNet baseline not a projected Full-score method with the same radius and anchor-reset schedule as Boundary?
Can you show results beyond frozen-weight mask learning?
Can you supply quality/state/runtime sensitivity analysis for radius, refresh interval, and frontier cut?