acceptodds
Under review as a conference paper at ICLR 2027

Nested Sparsity for Scalable Pretraining and Continual Learning

Abstract

Mixture-of-experts (MoE) transformers, the standard backbone for modern large-scale deep learning, employ conditional computation: only a sparse subset of model weights is used in each forward pass. In MoE transformers, this sparsity arises in the MoE operator, which generalizes the multi-layer perceptron in the classical transformer. In this work, we explore an alternative mechanism for conditional computation that analogously generalizes the attention operator. The sparsity induced by this mechanism naturally _nests_ (i.e., hierarchically composes) with the sparsity induced by the MoE operator. We show that a transformer employing this _nested sparsity_ mechanism, which we call Nestor, performs better than a conventional MoE transformer at equal compute and equal sparsity in language model pretraining. We also show that Nestor is well-suited for continual pretraining: it can be adapted to longer contexts more effectively than conventional MoEs, as well as adapt more efficiently to new domains while forgetting less on old domains. Furthermore, we demonstrate that these benefits persist as model size, dataset size, and context length grow. Our results demonstrate the potential of alternative conditional computation mechanisms for deep neural network architectures.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, mixture of experts, continual learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_VDHKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce NESTOR which attemps to extend sparse computation beyond the FFN (as in MoEs) by including sparsity in the the attention module. Each token is first routed to a subset of columns, where each column has its own attention parameters and only attends to the tokens routed to that same column. Within each selected column, the token is then routed to a subset of that column’s MLP experts. This is what they are calling hierarchical sparsity. The first one splits up the attention sequence into different sub-attention modules and the second routing decision does a sparse FFN computation over this sequence. In the end the outputs are merged by the two router weights.

In pretraining, NESTOR achieves lower loss than parameter-matched MoE Transformers at similar compute, with the authors showing that MoEs require approx. 1.13–1.29 more compute to reach the same loss. Its advantage becomes larger at long context lengths because each column performs attention over only its routed subset of tokens. They also argue that the modularity helps continual learning as in when adapting to new domains

Critical strengths

1 - The results seem decent on different data domains 2 - The paper is written quite well and I only had to look at the diagram to understand their entire method

Critical weaknesses

1 - The evaluation does not include downstream benchmarks which are a much better evaluation than held out val loss on different pretraining datasets / domains 2 - The idea is sort of a mixture of previous sparse attention work (all of which is cited though) like SwitchHead, MoH etc so not completely novel 3 - Not critical weakness but I believe the cerebral cortex analogy is EXTREMELY loose with no substantive connection except 'well the brain has neurons we have neurons' thus this is brain-inspired or brain-similar. I'd recommend removing it.

Questions for authors
  • Why did you not perform benchmark evals?
  • What's the actual implementation overhead / throughput vs standard MoEs / Dense models
  • What's the evidence for hierarchical learning? Sure you have multi-level routing (attn + FFN) but that does NOT imply hierarchical learning.
3 comments

We thank the reviewer for the endorsement and suggestions. We prepared the following response and hope they address the reviewer's concerns.

1. Downstream evaluation

We added zero-shot evaluations on nine downstream tasks—PIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, BoolQ, SciQ, COPA, and OpenBookQA—using the final pretraining checkpoints, and take the average of these scores in the below table.

Active scaleDenseMoENESTOR
200M49.8%51.6%52.1%
500M55.0%57.7%59.0%
1B59.7%63.7%64.2%

NESTOR has higher mean accuracy at all three scales. We will include the full results and evaluation protocols.

2. Implementation overhead and throughput

Yes, Nestor is currently slower in wall-clock time than a dense model at equal FLOPs. This is the same category of overhead that standard MoE faces: fine-grained routing produces smaller, irregular matrix operations that don’t fully saturate current GPU hardware. For reference, we benchmarked all three architectures on the same hardware and FLOPs:

Active param scale: 100M, Data scale: 5B, 1 H100 Dense: 4.2 h | MoE: 5.1 h | Nestor: 8.2 h

This is a kernel-level engineering problem, not a fundamental architectural limitation. We are actively optimizing for it.

While we don't have enough GPUs for large-scale runs, Nestor's architecture is specifically designed to scale well on multi-device setups. Each column is a self-contained sub-network with its own attention heads, projections, and private expert pool, and it attends only among the tokens routed to it. This means columns can be sharded across devices along a natural parallelism axis: attention within a column requires no cross-column communication, and the column-private experts are already per-column parameters with no inter-column dependency. Additionally, Nestor's column routing is decided once per block and dispatches coarser-grained units compared to other sparse attention methods (e.g., MoSA), which route at every layer. Furthermore, because each column's attention core runs over only S·k/G tokens rather than the full sequence, the per-token cost grows more slowly with context length.

3. Evidence for hierarchical learning

Thank you for pointing out this distinction, which we completely agree with. We do not claim in the paper that trained Nestor models learn explicit hierarchical features; this is a strong technical claim that we are not sure how to prove in full generality. If you point out where in the paper you feel we make this claim, we are happy to clarify this in the revision.

4. Novelty relative to prior sparse attention methods

We agree that our work has some similarities with past sparse attention work, which we describe in depth in the paper (as you point out). Our contribution relative to this prior work is that we noticed that attention weight sparsity, attention map sparsity, and MoE sparsity all naturally compose, and also that attention routing can effectively be extended to subnetwork (column) routing, which in principle enables this line of work to work at large scales (see the end of our response to Point 2). Compared to the closest work MoSA, our design ensures that all tokens are attended to, and also compose attention sparsity with MoE sparsity, enabling systems that in principle are more expressive and scalable.

5. Biological analogy

We agree that the cerebral cortex analogy is too loose and will remove it.

We'd like to thank the reviewer again for the comments and positive feedback. As the rebuttal period draws to close, we'd like to know if the reviewer has any additional concerns.

Thanks for the benchmark results and removing addressing my concern for the weak motivation for the brain analogy. I will maintain my score for now,

Reviewer_fiSxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NESTOR, which jointly selects attention parameters, the subset of tokens covered by attention, and the MoE experts within a column via column routing, and reuses the column routing across multiple layers within a block. The authors study pre-training efficiency at the scale of 100M–1B activated parameters, and further examine domain adaptation and long-context extension. The paper also uses a data model with hierarchical latent variables to explain the potential sample-efficiency and reduced forgetting brought by the nested structure.

Critical strengths
  1. The method design is quite interesting: a single column routing simultaneously determines the attention parameters used by tokens and the visible context, effectively avoiding the computational problems in current long-text scenarios while saving forward computation.
  2. In Table 3, NESTOR outperforms MoE at all four scales, with loss improvements of approximately 0.016–0.035 nats. In addition, the largest model reaches about 5.53B total parameters and 50B training tokens, providing a considerable degree of scale validation.
  3. The paper reports three settings simultaneously: equal tokens, equal FLOPs with repeated data, and equal FLOPs with additional data. The theoretical part also provides an analyzable mechanistic model that helps understand when structural matching may be beneficial.
Critical weaknesses
  1. My biggest concern is that the main experiments still lack sufficient comparison with core-related baselines. The experiments mainly compare with the vanilla MoE structure, but similar approaches such as MoSA and SwitchAll (discussed in related work) should also be introduced into the experimental comparison.
  2. Although the results support that NESTOR can perform long-sequence language modeling at lower computational cost, the average next-token loss may be mainly affected by local prediction and cannot fully determine whether column routing misses important long-distance dependencies, because the special structural design proposed in this paper (tokens only compute over tokens within the column) suggests that more long-text evaluation tasks should be introduced to analyze the impact of this structure.
  3. The gains in the continual learning direction are limited: without replay, the average forgetting is 0.254 for NESTOR and 0.261 for MoE; with replay, they are 0.073 and 0.076, respectively. Overall, the difference is small.
Questions for authors
  1. How does NESTOR compare with closely related methods such as SwitchAll?
  2. Does NESTOR preserve long-context retrieval performance on tasks such as needle-in-a-haystack?
3 comments

We thank the reviewer for the endorsement and suggestions on covering more eval axes. We conduct the following experiments and hope they can help.

1. Comparison with SwitchAll and MoSA

Thank you for suggesting these comparisons. We added both baselines at the 200M-active scale with a fixed 10B-token training budget and approximately matched active FLOPs. NESTOR achieves lower held-out loss than both settings.

ModelTotal paramsRelative body FLOPsHeld-out loss
NESTOR761.8M0.9813.0956
SwitchAll (SwitchHead + σ-MoE)811.2M0.9793.1161
MoSA + MoE MLP910.4M0.9983.1111

Relative body FLOPs are measured against the standard MoE control.

2. Long-context retrieval

We added retrieval evaluations using the same 500M-active checkpoints from our 32K context-extension experiments. Each model underwent continual pretraining on the same 0.5B-token budget at 32K. We use one checkpoint per model across all evaluation lengths, with no additional training.

We evaluated RULER’s S-NIAH-1 and S-NIAH-2 tasks at 4K, 8K, 16K, and 32K, using 500 identical examples per task and length across models. At 32K, retrieval accuracy is as follows.

TaskDenseMoENESTOR
S-NIAH-193.0%78.8%95.8%
S-NIAH-230.0%67.8%79.4%

NESTOR exceeds MoE by 17.0 and 11.6 percentage points, respectively. These results provide direct evidence that column routing preserves long-context retrieval in these settings. We will include the full results across context lengths.

3. Forgetting

We agree that the absolute reduction in average forgetting is modest in this relatively small-scale empirical setting. Note that our theory suggests some scaling recipes in which this advantage should grow at larger scales, but we acknowledge that the theory does not directly map onto the experiments and is instead used for intuition and motivation. We will revise the discussion to make the magnitude of this improvement clearer.

Thank you for the additional experiments and clarifications. The new baseline comparisons and long-context retrieval results address my main concerns. I will maintain my positive rating and encourage the authors to include these results and the relevant implementation details in the revised paper.

Thank you for your positive feedback and helpful suggestions. We will include these results and the relevant implementation details in the revised paper.

As a follow-up to the additional baseline comparisons, we also quantified the compute savings using the fitted NESTOR scaling curve in Figure 1. At the 200M scale, NESTOR is projected to match the held-out losses of MoSA and SwitchAll (SwitchHead + σ-MoE) with 15.3% and 16.1% less training compute, respectively, corresponding to baseline-to-NESTOR compute ratios of 1.18× and 1.19×.

Reviewer_Qax8OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces NESTOR, a sparse Transformer architecture that combines attention-parameter sparsity, attention-map sparsity, and MLP expert sparsity through nested routing. Tokens are routed to columns with their own attention parameters, attention is restricted to the tokens assigned to each column, and private experts are selected within the active columns. The experiments report improved loss–compute trade-offs in pretraining, reduced forgetting during domain adaptation, and greater computational efficiency as context length increases. Under a stylized hierarchical data model, the theoretical analysis establishes sample- and compute-efficiency advantages for nested predictors and identifies conditions under which they avoid cross-group forgetting.

Critical strengths
  1. A well-motivated and elegant architecture. The paper clearly motivates combining sparsity in attention parameters, attention maps, and MLP experts. The nested organization provides a coherent way to connect these components through a shared hierarchy of routing decisions. I find the idea compelling and the presentation clear.

  2. Experimental and theoretical evidence across several relevant settings. The controlled experiments demonstrate advantages in pretraining loss–compute trade-offs, retention during domain adaptation, and efficiency at longer contexts. Together, these results make a convincing case for the architecture's usefulness. The theoretical analysis further explains how a matching hierarchical structure can improve sample and compute efficiency and limit cross-group interference under the stated assumptions. This combination substantially deepened my understanding of the problem and supports my positive assessment.

Critical weaknesses
  1. The individual contributions of the two forms of attention sparsity are not fully isolated. My only substantive concern is that column routing simultaneously selects attention parameters and restricts which tokens attend to one another. The ablations in Table 4 help assess expert granularity and the nesting of experts within columns, but the column-routed attention variants retain both attention-parameter sparsity and attention-map sparsity. Consequently, these comparisons do not establish how much each attention mechanism contributes independently, or how much comes from their interaction. A more direct separation would strengthen the explanation of why the combined architecture works.
Questions for authors

Clarification request: Do any of the reported ablations independently vary attention-parameter routing and attention-map sparsity while holding the expert organization fixed? If so, could the authors identify the relevant comparison and explain what it establishes about each mechanism's contribution?

3 comments

We thank the reviewer for the strong endorsement and for suggesting this ablation. We added a two-seed experiment at the 200M-active scale, sharing Q/K/V/O projections across columns while keeping sparse attention, expert organization, training recipe, and executed FLOPs unchanged. Both models were trained for 10B tokens.

SeedNESTORShared Q/K/V/OΔ loss
423.09233.1643+0.0720
433.08813.1501+0.0620
Mean3.09023.1572+0.0670

Sharing reduces total parameters from 761.8M to 724.5M. The consistent loss improvement supports the value of column-specific attention capacity.

With full attention, NESTOR would approach a standard MoE Transformer, with differences in routing and top-k scheduling. Since full attention also increases FLOPs, the relevant comparison is the loss–compute trade-off. Our main-paper scaling curves support NESTOR’s overall advantage over an optimized MoE. We could not complete a separate full-attention scaling ladder within the rebuttal’s time and compute budget.

Thanks for the authors' response. My concern is addressed

Thank you again for the positive feedback and helpful suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.