- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces AMOR, a recurrent–attention hybrid that selectively activates attention based on predictive uncertainty. It retains a complete recurrent backbone and appends attention blocks gated by next-token prediction entropy, with thresholds updated during training and frozen at inference. An optional distilled router reduces the overhead of entropy estimation. Experiments across two recurrent backbones and three model scales evaluate downstream performance, long-context behavior, and computational efficiency, showing benefits in several settings.
Critical strengths
- The architecture clearly separates recurrent processing from conditional attention, while retaining keys and values at skipped positions for subsequent retrieval.
- The evaluation covers multiple model scales, backbones, and task families, complemented by architectural ablations, routing analyses, and measured latency and memory costs.
Critical weaknesses
1.The justification for using predictive entropy to route attention remains insufficient. Predictive uncertainty does not necessarily imply that additional attention will improve a prediction. Further experimental analysis is needed to establish why entropy is better suited than alternative confidence measures or learned routers to identifying positions that benefit from attention.
2.The paper also does not directly validate the relationship between predictive entropy and the benefit of attention. In particular, it remains unclear whether enabling attention yields larger improvements at high-entropy positions than at low-entropy positions. Such evidence would provide more direct support for the central routing assumption.
3.The ungated ablation introduces several confounding changes, including the normalization layout, residual scaling, and output projection initialization. The observed performance differences therefore cannot be attributed solely to the gating mechanism. A controlled comparison that holds these design choices fixed is needed to isolate the contribution of gating.
4.The magnitude and scope of the performance gains are limited. Some improvements in average common-sense performance are small, average retrieval performance remains below that of fixed-schedule hybrids, and the long-context advantages do not hold consistently across domains.
5.The paper lacks a systematic evaluation of the gating threshold and its effect on the trade-off between predictive performance and computational cost. The offset coefficient is fixed at 0.2 based on a small-scale exploration using a different thresholding rule, which provides limited justification for the current setting. A more comprehensive comparison of task performance, attention firing rates, and measured latency across thresholds is needed.
Questions for authors
Appendix D reports attention firing rates of approximately 97–99% on CodeParrot, indicating that attention is activated at nearly every position. Does this reflect a genuine need for additional attention in this domain, or a domain shift in the entropy distribution that makes the frozen thresholds less appropriate? Under this setting, does gating still offer any performance or efficiency benefits over always-on attention?
6 comments
Response to Reviewer 63Qz (2/4)
- Normalization layout. AMOR-symmetric changes the layout alone. At 440M, it has a lower common-sense average than canonical AMOR on both backbones (38.9 against 39.6 with Mamba2; 39.9 against 40.5 with Gated DeltaNet; Table 33), and at 1.5B it trails canonical AMOR on four of six cloze retrieval tasks with Mamba2 and five with Gated DeltaNet (Table 35).
- Per-channel scaling α. Each AMOR model pretrained without α at 440M still leads its own backbone in held-out FineWeb-Edu perplexity at 3K, in LAMBADA perplexity and in the common-sense average (39.4 against 39.0 with Mamba2; 40.1 against 39.7 with Gated DeltaNet; Table 36). α adds to this lead: it raises the common-sense average to 39.6 and 40.5 and lowers FineWeb-Edu perplexity at 20K on both backbones.
- Zero-initialized output projection. The Gaussian W_O initialization variant of AMOR-Mamba2, pretrained at 180M with W_O from N(0, 0.02) like every other linear projection (Appendix C), scores slightly better than canonical AMOR-Mamba2 on every measure of Tables 37 and 38 (common-sense average 36.0 against 35.8; LAMBADA perplexity 239.23 against 240.27; 0.3% to 16% lower 20K perplexity on all seven datasets), so AMOR's results do not rely on the zero initialization. We keep it to stay true to AMOR's design principle: AMOR starts identical to its recurrent backbone, a complete language model that the AMOR blocks then refine (Section 3.1).
By isolating one factor at a time, these design ablations indicate that AMOR outperforms the baselines because of its design as a post-hoc hybrid with an entropy gate, not because of additional parameters or differences in the pretraining setup (Appendix I.4).
Table 38: Parameter count, output initialization and firing everywhere ablations: 20K perplexity comparison. 180M.
| Model | GovReport 20K PPL ↓ | QMSum 20K PPL ↓ | NarrativeQA 20K PPL ↓ | Qasper 20K PPL ↓ | CodeParrot 20K PPL ↓ | PG-19 20K PPL ↓ |
|---|---|---|---|---|---|---|
| AMOR-Mamba2 | 28.98 | 49.26 | 127.62 | 65.15 | 845.89 | 94.35 |
| ↳ Gaussian W_O initialization | 27.94 | 46.10 | 120.14 | 62.43 | 709.26 | 91.39 |
| ↳ fires everywhere (κ=−∞) | 46.68 | 70.97 | 165.00 | 104.25 | 1509.94 | 119.85 |
| Mamba2 | 25.33 | 42.92 | 108.23 | 54.94 | 652.78 | 84.02 |
| ↳ matched to AMOR's parameter count | 25.41 | 43.69 | 105.89 | 54.87 | 657.75 | 82.16 |
| Mamba2 Serial Hybrid | 62.69 | 88.01 | 253.48 | 157.01 | 1582.45 | 186.09 |
| ↳ matched to AMOR's parameter count | 88.43 | 95.29 | 270.80 | 179.48 | 1245.02 | 218.76 |
Table 36: Per-channel scaling α ablation. 440M.
| Model | Params | FW-Edu: 3K PPL ↓ | FW-Edu: 20K PPL ↓ | LAMB PPL ↓ | Common-sense avg acc ↑ |
|---|---|---|---|---|---|
| AMOR-Mamba2 | 448.7M | 18.30 | 20.13 | 57.37 | 39.6 |
| ↳ without α | 448.6M | 18.26 | 20.40 | 59.15 | 39.4 |
| Mamba2 | 436.1M | 18.46 | 18.79 | 74.19 | 39.0 |
| AMOR-Gated DeltaNet | 442.1M | 17.99 | 18.73 | 50.15 | 40.5 |
| ↳ without α | 442.1M | 17.99 | 19.27 | 53.12 | 40.1 |
| Gated DeltaNet | 429.5M | 18.11 | 18.41 | 62.37 | 39.7 |
W4: Are the gains limited: small common-sense gains, retrieval below fixed-schedule hybrids, and long-context advantages that do not hold across domains? (Sections 4.1-4.3 and 5.1; Appendices A.2 and H)
No: the common-sense gains hold at every seed, and AMOR's routing trade-off does buy more long-context robustness across domains.
On common sense, the gains are consistent: as with the canonical seed 42, an AMOR variant leads at each of four additional pretraining seeds at 180M and in their average, and each AMOR model is above its own backbone at every seed (Table 5).
Fixed-schedule hybrids lead on cloze retrieval, as Section 4.2 states; AMOR's claim there is still a substantial gain over the pure recurrent baselines (Table 3), for example on S-NIAH at 2K, where AMOR scores 93.0 against 57.4 with Mamba2 and 93.6 against 52.1 with Gated DeltaNet (Figure 5).
Having as many attention layers as the fixed-schedule hybrids does not mean that AMOR uses the same amount of attention. In the fixed-schedule hybrids, every token gets attention updates at fixed depths; in AMOR, only the positions where a gate fires get the help of attention.
We argue that AMOR's conditional routing is a worthy trade-off. In return, an AMOR model has the best common-sense average at every scale (Table 2), both AMOR models stay above every fixed-schedule hybrid on LongBench (8.3 and 8.4 against 4.5-7.4; Table 4), and AMOR-Mamba2 decodes faster than both Mamba2 fixed-schedule hybrids from 64K on (Table 15). In future work, we hope to further improve AMOR's retrieval performance.
Response to Reviewer 63Qz (3/4)
Across domains, every model that uses attention faces RoPE distribution shift beyond its 3K pretraining length (Appendix H). However, with training-free NTK-aware RoPE scaling applied to all of them, an AMOR model has the lowest 20K perplexity on all seven datasets among the models that use attention (Table 31). Without NTK, an AMOR model is lowest on six of the seven; the exception is CodeParrot, where the gate fires on nearly every position, likely because our pretraining data FineWeb-Edu contains little code (see Q1). The fact that AMOR's firing rate is consistent across context lengths within each domain (Table 11) is a feature, not a bug.
Table 31: With NTK, an AMOR model has the lowest 20K perplexity on all seven datasets among the models that use attention. 1.5B; every model pretrained at 3K. NTK: Appendix E.4; forced firing in all three blocks: Appendix E.5.
| Model | FW-Edu: 3K PPL ↓ | FW-Edu: 20K PPL ↓ | GovReport 20K PPL ↓ | QMSum 20K PPL ↓ | NarrativeQA 20K PPL ↓ | Qasper 20K PPL ↓ | CodeParrot 20K PPL ↓ | PG-19 20K PPL ↓ |
|---|---|---|---|---|---|---|---|---|
| Mamba2 | 13.07 | 13.07 | 10.77 | 19.92 | 39.13 | 20.68 | 108.55 | 30.98 |
| Gated DeltaNet | 12.90 | 12.87 | 10.36 | 19.36 | 36.06 | 19.62 | 94.34 | 29.32 |
| Uses attention | ||||||||
| ↳ Transformer | 13.22 | 1317.45 | 974.29 | 953.07 | 1622.88 | 1596.14 | 22531.42 | 1416.66 |
| ↳↳ + NTK | 13.22 | 27.30 | 23.05 | 31.39 | 74.92 | 44.78 | 279.44 | 63.20 |
| ↳ Mamba2 Serial Hybrid | 12.77 | 73.73 | 90.56 | 106.56 | 224.18 | 168.13 | 906.44 | 170.67 |
| ↳↳ + NTK | 12.77 | 14.09 | 11.84 | 19.65 | 42.25 | 23.12 | 100.66 | 33.78 |
| ↳ Gated DeltaNet Serial Hybrid | 12.76 | 29.52 | 33.18 | 53.95 | 109.89 | 74.78 | 503.35 | 85.88 |
| ↳↳ + NTK | 12.76 | 13.50 | 11.50 | 19.07 | 39.64 | 22.12 | 94.03 | 32.53 |
| ↳ Mamba2 Fused Hybrid | 12.85 | 14.66 | 14.40 | 28.75 | 51.86 | 29.35 | 239.93 | 37.13 |
| ↳↳ + NTK | 12.85 | 13.28 | 12.16 | 20.33 | 42.36 | 25.77 | 218.34 | 32.72 |
| ↳ AMOR-Mamba2 | 13.08 | 14.42 | 14.01 | 34.72 | 57.32 | 30.02 | 421.78 | 38.10 |
| ↳↳ + forced firing, all three blocks | 13.06 | 15.14 | 14.64 | 35.87 | 59.06 | 31.11 | 423.80 | 39.39 |
| ↳↳ + NTK | 13.08 | 13.24 | 10.81 | 18.91 | 39.13 | 20.89 | 94.11 | 31.12 |
| ↳ AMOR-Gated DeltaNet | 12.91 | 13.64 | 11.87 | 28.42 | 47.35 | 26.30 | 269.33 | 34.13 |
| ↳↳ + forced firing, all three blocks | 12.90 | 14.04 | 12.18 | 29.01 | 48.00 | 26.84 | 269.94 | 34.79 |
| ↳↳ + NTK | 12.91 | 12.98 | 10.41 | 19.14 | 37.77 | 20.09 | 89.68 | 29.95 |
W5: How do task performance, firing rates and measured latency change across gating thresholds? (Section 5.3; Appendix D.6)
We added Section 5.3 and Appendix D.6. Under the paper's threshold rule, τ = μ+κσ, not the fixed-offset rule of the earlier exploration, we pretrained AMOR-Mamba2 from scratch at 180M with five settings, κ ∈ {+∞, 0.6, 0.2, −0.5, −∞}, from attention never firing (κ = +∞, for which the Mamba2 backbone stands in) to firing everywhere (κ = −∞; Table R6). We also changed κ on the pretrained 1.5B AMOR-Mamba2. Across both, we measured common-sense accuracy, firing rates, long-context robustness and decode latency.
A higher target firing rate in pretraining raises common-sense reasoning accuracy up to ∼70% of positions (κ = −0.5) but lowers it at 100% (κ = −∞), and comes at a steep cost: FineWeb-Edu perplexity at 20K rises by 12.3% at κ = −0.5 and by 38.2% at κ = −∞, and at 1.5B, decoding at 128K slows by 33.5% as the firing rate rises from 40% to 100% (Table R7; Figure 8).
Changing κ after pretraining moves the firing rate on held-out FineWeb-Edu from 13.2% at κ = 1.0 to 71.9% at κ = −0.5, and common-sense accuracy rises quickly up to κ = 0.2 and less with more firing (Appendix D.6). This places κ = 0.2 on the Pareto frontier between task accuracy and efficiency.
Table R6: Firing more in pretraining does not consistently improve common-sense accuracy and costs long-context robustness. 180M AMOR-Mamba2 pretrained from scratch with each κ, same data, token budget, recipe and seed; Section 5.3.
| κ in pretraining | Common-sense (avg acc) | FW-Edu PPL at 20K | PG-19 PPL at 20K | CodeParrot PPL at 20K |
|---|---|---|---|---|
| +∞ (never fires; the Mamba2 backbone) | 34.78 | 30.70 | 84.02 | 652.78 |
| 0.6 (19.2% firing) | 35.56 | 31.10 | 90.41 | 694.53 |
| 0.2 (canonical; 39.3% firing) | 35.82 | 31.90 | 94.35 | 845.89 |
| −0.5 (70.4% firing) | 35.94 | 35.81 | 101.89 | 1144.96 |
| −∞ (fires everywhere) | 35.69 | 44.09 | 119.85 | 1509.94 |
Response to Reviewer 63Qz (1/4)
We thank the reviewer for a careful and specific review.
W1 and W2: Does enabling attention help more at high-entropy positions than at low-entropy positions? (Appendix D.3)
Yes. At a block's natural firing positions (high entropy), forced non-firing, one block at a time and with all weights fixed, raises perplexity by 0.36 to 1.79 in every block of both 1.5B models. At its natural non-firing positions (low entropy), forced firing in the same setting changes perplexity by at most 0.05 (Table 10). This indicates that AMOR's attention helps at the positions the model finds difficult.
To answer this question, we also ran new experiments (Section 5.3). They show that firing more is not universally beneficial: the firing-everywhere variant of AMOR-Mamba2 has a lower common-sense average and 38.2% higher FineWeb-Edu perplexity at 20K than canonical AMOR-Mamba2, which fires on ∼40% of positions in pretraining.
Table 10: Attention lowers perplexity at uncertain positions. 1.5B; held-out FineWeb-Edu.
W1: Why is entropy better suited than alternative confidence measures or learned routers? (Section 5.2; Appendix D.2)
Against top-1 probability, a CALM-style margin, a learned MoD-style router and a random gate, pretrained from scratch at 180M with the same pretraining firing rate of ∼40%, the AMOR entropy gate yields the lowest LAMBADA and CodeParrot perplexity on both backbones, the lowest PG-19 perplexity on the Gated DeltaNet backbone and the tied-highest common-sense average on the Mamba2 backbone (Table 1).
We also directly swapped the routing mechanism on the pretrained 1.5B checkpoints for alternative routing mechanisms, with all weights fixed and each block's firing rate matched to the entropy gate's: the entropy gate achieves the lowest perplexity on FineWeb-Edu, PG-19 and LAMBADA on both backbones (Table 9). Together, these two comparisons support our choice of entropy as the routing signal.
Table 1: Among routing mechanisms, the AMOR entropy gate yields the lowest LAMBADA and CodeParrot perplexity. 180M. The MoD-style router adds 2,307 trainable parameters, a linear layer per block. Common-sense uses the eight tasks and setup of Section 4.1 (Appendix E.1).
W3: Can a controlled comparison that holds the normalization layout, residual scaling and output-projection initialization fixed isolate the gate? (Section 5.3; Appendix I)
Yes. AMOR-ungated makes four changes at once (Table 32): the normalization layout, the per-channel scaling α, the output initialization and the gate. To address this question, we pretrained design ablations from scratch to isolate them one at a time: