acceptodds
Under review as a conference paper at ICLR 2027

Only the Boundary Matters: Efficient Mask Learning for Sparse Neural Networks

Abstract

Sparse neural networks construct subnetworks by selectively retaining weight connections and are widely used for model compression, inference acceleration, and parameter-efficient adaptation. Binary masks implement connection selection in sparse networks, while learned masks use continuous scores to adapt this selection to data or tasks. However, discrete masks still rely on dense floating-point scores, gradients, and optimizer states, whose update overhead grows with the number of candidate weights. We identify a simple rank-stability phenomenon: under bounded score updates, score intervals that cannot cross the Top- threshold correspond to fixed mask decisions, and only boundary scores can change their inclusion status. Based on this observation, we propose Boundary, an active-set optimization framework for learned binary masks. It stores determined decisions as fixed bits, maintains continuous optimization state only for boundary scores, and completes selection using the remaining quota; periodically committing state and rebuilding the active set allows previously fixed coordinates to re-enter optimization. Experiments on ImageNet and random-weight subnetworks show that Boundary substantially reduces controller state while maintaining accuracy close to the original mask optimization. On Pythia-1.4B, its language-model quality is close to Full-score; in an independent short-context timing benchmark, controller state and peak allocated memory decrease by 80.6% and 67.1%, respectively, with a measured training-loop speedup of 9.20. Boundary turns Top- rank stability into a concrete rule for allocating optimization resources according to whether mask decisions can change, providing a direct way to reduce mask-training cost.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Learnable Masks, Active-Set Optimization, Sparse Training, Model Compression, Efficient Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NUE1OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes an active-set method for learning binary masks, called Boundary. Using interval bounds on score magnitudes, coordinates whose mask membership cannot change within an update interval are fixed. Scores and optimizer state are kept only for the remaining boundary coordinates, and this set is refreshed periodically. Optionally, masks in an initial network prefix are frozen to skip their backward pass. On image classification and language modeling with different architectures and datasets, the method reduces controller state by roughly 75–83% with quality close to full-score training.

Critical strengths

The paper targets a real and practically relevant cost. Dense scores, gradients, and optimizer state maintained during mask learning can be substantial at LLM scale.

The method leaves the host's mask objective, sparsity quota and update rule unchanged, so it can in principle be layered on top of existing mask-learning methods.

The experiments cover several model families, including CNNs, a vision transformer, a random-weight subnetwork setup and a 1.4B language model. Across these, controller state is reduced substantially while quality stays close to the full-score baseline.

The paper separates logical state savings from actual peak memory and runtime effects, and it reports cases where the method does not help. The ablations comparing static and eventwise refresh, fixed bands and random active sets are informative.

Critical weaknesses

Proposition 1 is obvious. It essentially states that if score changes are bounded, elements far enough from the current Top-k threshold cannot cross it. In addition, the final algorithm does not guarantee the radii by bounding the updates, e.g. via the number of iterations. Instead, scores are simply clamped to keep the active set valid (Eq. 7). This is unprincipled and makes the method appear ad hoc.

The fixed prefix is trivial. Sec. 3.3 is standard chain-rule reasoning: if the first part of the computation graph is frozen, the backward pass can stop early. Not optimizing masks in the first few layers obviously reduces memory and compute. Optionally fixing the first layers should not be presented as a pronounced part of the algorithm, since it is simply a cost reduction from not applying the method to the whole model.

The fixed mask entries would still receive score updates that can matter later, but these are ignored entirely. The theory only guarantees that binary mask entries do not change within the clipped interval. It does not justify discarding these updates relative to full-score training. The experiments show this does not hurt performance significantly, but they also show that Hard-freeze may not hurt much either. This raises the question of whether the active-set machinery is necessary at all. Better comparisons against simpler compute-saving baselines are needed. Additionally, the main vision Full-score baseline does not use the interval projection that Boundary uses, so the comparison does not isolate the effect of the active set.

Vision runtime gains are weak and sometimes negative. Boundary without frontier barely changes runtime or peak memory. With the frontier, median step time improves, but total setup plus training time is only slightly better on ResNet-50 and ConvNeXt-T and worse on DeiT-B.

The 9.2x Pythia speedup is heavily confounded. The paper states that Full-score uses stable four-score sorting while Boundary uses direct rank comparisons on active groups. The reported speedup therefore stems not only from the active-set idea but also from a possibly cheaper implementation for Boundary. A fair comparison would give Full-score an equally optimized top-2-of-4 selection.

Inconsistent statistical reporting. Three seeds for the ImageNet experiments are fine. However, the Pythia quality results are reported without variance, even though the differences between methods are tiny. Gaps of this size do not allow any conclusions without uncertainty estimates.

The prose is hard to follow in some place, due to confusing or undefined terms. Some examples are:

  • l. 076: "Boundary with the frontier"
  • l. 080: "reduces controller state"
  • l. 156: "retains k of n positions independently within each group"
  • l. 206: "training events"
  • l. 267: "scheduled event"
  • l. 272: "logical scores"
Questions for authors

Why is the main ImageNet baseline not a projected Full-score method with the same radius and anchor-reset schedule as Boundary?

Can you show results beyond frozen-weight mask learning?

Can you supply quality/state/runtime sensitivity analysis for radius, refresh interval, and frontier cut?

Reviewer_1xEZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Boundary, an active-set framework for learning binary masks over frozen neural-network weights. An interval-rank certificate identifies coordinates whose Top-k membership cannot change within a prescribed score box. Boundary packs active scores and optimizer state, represents certified decisions with fixed bits, and retains the history required for reconstruction and re-entry. Residual Top-k selection completes each group’s quota, while periodic refreshes reconsider the active set. An optional fixed prefix removes prefix backward computation. Experiments on three ImageNet backbones, random-weight CIFAR-10 subnetworks, and deterministic 2:4 Pythia-1.4B masks demonstrate substantial controller-state reductions with generally close reference quality. Runtime and peak-memory benefits vary considerably with the host and implementation.

Critical strengths

1.The interval certificate and residual-quota construction provide a concrete way to retain continuous optimization state for a restricted set of coordinates while preserving the prescribed Top-kk decisions within an interval. The treatment of signed scores, ties, quotas, reconstruction, and optimizer-state alignment strengthens the technical foundation. This is consequential because sparse executed masks can otherwise require dense training-state arrays.

2.The experiments demonstrate substantial reported controller-state reductions with generally close reference quality across ImageNet, frozen-random-weight CIFAR-10, and Pythia. The 30-epoch ImageNet comparison also reports higher final accuracy for Boundary + frontier than Full-score on all three architectures. These results provide evidence that the representation can be useful beyond a single architecture or a short recovery experiment.

Critical weaknesses
  1. The largest execution gains are not adequately isolated from implementation choices, while vision whole-run benefits are limited. On Pythia, Full-score uses stable four-score sorting, whereas Boundary uses direct rank comparisons on active groups. Most of the component-time reduction occurs in the full forward region, which includes gradient clearing, mask construction, and backbone computation. Although the paper acknowledges the implementation difference, an optimized Full-score comparison is needed to establish the incremental benefit of restricting optimization to boundary scores. In vision, Boundary-only provides little median-step improvement; substantially larger improvements occur when the prefix frontier is added. Table 3 reports higher setup-plus-training time in four of six configurations, with increases up to 36.7%, and GPU peak reductions of only 0–2%. Thus, the current evidence establishes controller-state compression more strongly than broadly applicable training acceleration.

  2. The quality–resource trade-off of the optimization restrictions and their hyperparameter robustness are insufficiently established. The guarantees apply within prescribed score intervals and, for packed-update equivalence, to a reference with the same active-coordinate restriction. The empirical references differ: vision Full-score uses unprojected updates, while language Full-score uses matching projection. Consequently, the language results do not establish similarity to unrestricted Full-score training. Table 10 reports clipping rates without corresponding quality and resource measurements. Table 11 omits Boundary and an unprojected Full-score reference under the same reported metric, preventing a direct comparison of Boundary with these controls using the same reported metric. These gaps leave uncertainty about the effects of projection, active-set selection, and radius/refresh choices. In addition, the MaskLLM (Fang et al., 2024) adaptation requires more complete configuration details, while the Adafactor (Shazeer & Stern, 2018) and 8-bit AdamW (Dettmers et al., 2022) optimizer variants in Table 9 lack quality results.

Questions for authors
  1. What speedup and peak-memory reduction remain against Full-score implemented with optimized direct Top-2 comparisons and the same tie-breaking rule? Please separate mask construction from backbone forward computation, report context lengths 128 and 2,048, and specify the context length used in Table 5.

  2. Building on the Hard-freeze/Boundary-only costs in Table 14 and the Full-score + prefix control in Table 11, can you provide a common comparison of held-out quality, total time, and peak memory for Full-score, Full-score + frontier, Hard-freeze, and Boundary on the same hardware? This is particularly relevant for ConvNeXt-T, where Hard-freeze and Boundary-only have similar reported accuracy despite different update budgets.

  3. Can Table 11 be completed with Boundary and unprojected Full-score, using a common held-out metric and variability estimates across all three backbones? How does projected Full-score perform after 30 epochs, and what radius was used for that experiment? Given that the reported 30-epoch Full-score final accuracies are below the corresponding one-epoch results, learning curves or a common development-based checkpoint comparison would help interpret the longer-training gains.

  4. The main ImageNet protocol uses a shared radius r = 0.06 and refresh interval J = 160, with architecture-specific prefix cuts. How were these settings and the freezing time selected, and what tuning budget was used? Were the shared settings selected on one architecture and then transferred, or selected using results across the target architectures? Which architecture underlies Table 10, and what are the corresponding held-out accuracy, active fraction, memory, and runtime measurements?

  5. Please specify the MaskLLM (Fang et al., 2024) adaptation’s prior, temperature/scaling schedules, regularization, and tuning procedure. For the large Wanda–SparseGPT perplexity gap in Table 8, what configuration and implementation checks were performed? What does “8-bit AdamW (compat.)” implement relative to standard block-wise 8-bit AdamW (Dettmers et al., 2022), how is Adafactor (Shazeer & Stern, 2018) configured, which tensors are counted in Table 9’s “Optimizer state” column, and what held-out quality do these optimizer variants achieve?

References

Reviewer_9FMFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce a sparse mask selection with improved memory state thanks to tracking only elements on the boundary. Complementary steps like refreshing the mask allows previously fixed elements to re-enter optimization procedure. They obtain dramatic memory and footprint reduction and inference speedups.

Critical strengths

1/ Locating a small subset of trainable parameters for efficient sparse training is an interesting research area.

2/ The idea of allocating less bits to elements for which a "decision" has been made (trainable or not) is a novel and interesting idea and implementation seems to back it up.

3/ The reported results seem interesting.

Critical weaknesses

1/ Experiments are only reported for small sized models. Scaling the experiments would make it more compelling.

2/ Can the authors explain how is it possible to obtain >2x training loop speed-up for 2:4 sparsity? Is it because there's less data movement or something else?

3/ Can authors discuss more on the novelty side of the paper? Reducing the memory state for some parameters is an elegant idea, is there anything else they would want to highlight other than that and the obtained memory/speedup results?

Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.