- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors introduce NESTOR which attemps to extend sparse computation beyond the FFN (as in MoEs) by including sparsity in the the attention module. Each token is first routed to a subset of columns, where each column has its own attention parameters and only attends to the tokens routed to that same column. Within each selected column, the token is then routed to a subset of that column’s MLP experts. This is what they are calling hierarchical sparsity. The first one splits up the attention sequence into different sub-attention modules and the second routing decision does a sparse FFN computation over this sequence. In the end the outputs are merged by the two router weights.
In pretraining, NESTOR achieves lower loss than parameter-matched MoE Transformers at similar compute, with the authors showing that MoEs require approx. 1.13–1.29 more compute to reach the same loss. Its advantage becomes larger at long context lengths because each column performs attention over only its routed subset of tokens. They also argue that the modularity helps continual learning as in when adapting to new domains
Critical strengths
1 - The results seem decent on different data domains 2 - The paper is written quite well and I only had to look at the diagram to understand their entire method
Critical weaknesses
1 - The evaluation does not include downstream benchmarks which are a much better evaluation than held out val loss on different pretraining datasets / domains 2 - The idea is sort of a mixture of previous sparse attention work (all of which is cited though) like SwitchHead, MoH etc so not completely novel 3 - Not critical weakness but I believe the cerebral cortex analogy is EXTREMELY loose with no substantive connection except 'well the brain has neurons we have neurons' thus this is brain-inspired or brain-similar. I'd recommend removing it.
Questions for authors
- Why did you not perform benchmark evals?
- What's the actual implementation overhead / throughput vs standard MoEs / Dense models
- What's the evidence for hierarchical learning? Sure you have multi-level routing (attn + FFN) but that does NOT imply hierarchical learning.
3 comments
We'd like to thank the reviewer again for the comments and positive feedback. As the rebuttal period draws to close, we'd like to know if the reviewer has any additional concerns.
Thanks for the benchmark results and removing addressing my concern for the weak motivation for the brain analogy. I will maintain my score for now,
We thank the reviewer for the endorsement and suggestions. We prepared the following response and hope they address the reviewer's concerns.
1. Downstream evaluation
We added zero-shot evaluations on nine downstream tasks—PIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, BoolQ, SciQ, COPA, and OpenBookQA—using the final pretraining checkpoints, and take the average of these scores in the below table.
NESTOR has higher mean accuracy at all three scales. We will include the full results and evaluation protocols.
2. Implementation overhead and throughput
Yes, Nestor is currently slower in wall-clock time than a dense model at equal FLOPs. This is the same category of overhead that standard MoE faces: fine-grained routing produces smaller, irregular matrix operations that don’t fully saturate current GPU hardware. For reference, we benchmarked all three architectures on the same hardware and FLOPs:
Active param scale: 100M, Data scale: 5B, 1 H100 Dense: 4.2 h | MoE: 5.1 h | Nestor: 8.2 h
This is a kernel-level engineering problem, not a fundamental architectural limitation. We are actively optimizing for it.
While we don't have enough GPUs for large-scale runs, Nestor's architecture is specifically designed to scale well on multi-device setups. Each column is a self-contained sub-network with its own attention heads, projections, and private expert pool, and it attends only among the tokens routed to it. This means columns can be sharded across devices along a natural parallelism axis: attention within a column requires no cross-column communication, and the column-private experts are already per-column parameters with no inter-column dependency. Additionally, Nestor's column routing is decided once per block and dispatches coarser-grained units compared to other sparse attention methods (e.g., MoSA), which route at every layer. Furthermore, because each column's attention core runs over only S·k/G tokens rather than the full sequence, the per-token cost grows more slowly with context length.
3. Evidence for hierarchical learning
Thank you for pointing out this distinction, which we completely agree with. We do not claim in the paper that trained Nestor models learn explicit hierarchical features; this is a strong technical claim that we are not sure how to prove in full generality. If you point out where in the paper you feel we make this claim, we are happy to clarify this in the revision.
4. Novelty relative to prior sparse attention methods
We agree that our work has some similarities with past sparse attention work, which we describe in depth in the paper (as you point out). Our contribution relative to this prior work is that we noticed that attention weight sparsity, attention map sparsity, and MoE sparsity all naturally compose, and also that attention routing can effectively be extended to subnetwork (column) routing, which in principle enables this line of work to work at large scales (see the end of our response to Point 2). Compared to the closest work MoSA, our design ensures that all tokens are attended to, and also compose attention sparsity with MoE sparsity, enabling systems that in principle are more expressive and scalable.
5. Biological analogy
We agree that the cerebral cortex analogy is too loose and will remove it.