- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper identifies that learned sparse attention is trained with differentiable soft gates but deployed with discrete top-k routes. Because deployed routes are generated layer by layer (each hard decision changes the hidden state that feeds every later router), the backbone has never seen the state trajectory it encounters at deployment.
The authors propose Closed-Loop Hard Routing. Each step runs two paths. The soft path is the ordinary gated-attention model and is the only path that trains the router. The hard path selects detached top-k masks layer by layer from its own current hard state, so the backbone is trained on its own self-induced deployment trajectory.
With the configuration frozen, the reduction transfers to 300M models on WikiText-103 (0.796 → 0.049) and FineWeb-Edu (0.513 → 0.027); on LAMBADA, standard hardening reduces accuracy from 13.2% to 3.7%, whereas CLHR changes it from 13.9% to 12.9%.
Critical strengths
- Causal identification of the failure mode. The six-condition ladder (standard, soft replay, historical, open-loop hard, shuffled closed-loop, CLHR) isolates which ingredient matters. This is the paper’s core contribution. It turns “soft-to-hard mismatch” from an intuition into a mechanistically specific claim.
- Careful measurement and transparent reporting. The paper defines closed-loop versus open-loop evaluation, uses an independent evaluator with mask-cardinality tests, reports per-seed values, and discloses a bug (the evaluator mask bug and the MoBA top-k=full-block configuration bug).
Critical weaknesses
- The paper is poorly presented and hard to follow. The introduction and background are not provided sufficiently
- CLHR trains on the backbone’s own hard trajectory with a soft loss. This is close to SSA-style dual-stream training, and the paper’s own adapted SSA reaches essentially the same excess (0.0033 vs 0.0035). CLHR’s advantage is only about 0.042 nats in absolute NLL.
- Full CLHR costs about 1.96x standard training, and the benchmark is dense-emulated masking with no sparse kernel. The “near-lossless deployment” therefore has no demonstrated end-to-end quality-speed result.
- The sparsity and scale regime is narrow: 31M and 300M, sequence length 512, with the 300M model trained on WikiText-103 repeated to 4B tokens.
Questions for authors
- For the native-hard comparison, was the same tuning effort (learning rate, warmup, and hard-routing-specific tricks) given to every baseline?
- For SSA-style, how was the alignment weight chosen, and is the adapted version faithful enough to be called independent confirmation? Was removing alignment tested at 300M?