acceptodds
Under review as a conference paper at ICLR 2027

When to Think Fast and Slow? AMOR: Adaptive Entropy Gate for Hybrid Models

Abstract

Recurrent-attention hybrids aim to combine the efficiency of recurrence with the contextual recall of attention, but existing approaches typically apply attention uniformly across all positions, even when the recurrent state alone is sufficient for accurate prediction. We introduce AMOR (Adaptive Metacognitive Output Router), a post-hoc hybrid architecture that selectively invokes attention based on predictive uncertainty. A recurrent backbone is augmented with entropy-gated attention blocks that activate only when the model's output entropy exceeds a dynamic threshold derived from a running batch median and scaled standard deviation. The resulting binary gate requires no learned routing parameters. Pretrained from scratch on FineWeb-Edu and with attention invoked on only  40% of positions, one of the AMOR variants (Mamba2 or Gated DeltaNet backbones) achieves the highest eight-task common-sense reasoning average at each scale among pure recurrent, pure attention, and fixed-schedule hybrid models. AMOR also improves retrieval performance over pure recurrent models while remaining competitive against fixed-schedule hybrids. Additionally, AMOR retains the long-context robustness of its recurrent backbones, where the Transformer and other hybrid architectures degrade under distribution shift. These results suggest that when attention is applied matters as much as how much: selectively allocating attention based on predictive uncertainty improves accuracy, robustness, and efficiency, offering a simple alternative to uniform or fixed routing strategies.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hybrid language models, recurrent neural networks, adaptive computation, entropy gating, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_63QzOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces AMOR, a recurrent–attention hybrid that selectively activates attention based on predictive uncertainty. It retains a complete recurrent backbone and appends attention blocks gated by next-token prediction entropy, with thresholds updated during training and frozen at inference. An optional distilled router reduces the overhead of entropy estimation. Experiments across two recurrent backbones and three model scales evaluate downstream performance, long-context behavior, and computational efficiency, showing benefits in several settings.

Critical strengths
  1. The architecture clearly separates recurrent processing from conditional attention, while retaining keys and values at skipped positions for subsequent retrieval.
  2. The evaluation covers multiple model scales, backbones, and task families, complemented by architectural ablations, routing analyses, and measured latency and memory costs.
Critical weaknesses

1.The justification for using predictive entropy to route attention remains insufficient. Predictive uncertainty does not necessarily imply that additional attention will improve a prediction. Further experimental analysis is needed to establish why entropy is better suited than alternative confidence measures or learned routers to identifying positions that benefit from attention.

2.The paper also does not directly validate the relationship between predictive entropy and the benefit of attention. In particular, it remains unclear whether enabling attention yields larger improvements at high-entropy positions than at low-entropy positions. Such evidence would provide more direct support for the central routing assumption.

3.The ungated ablation introduces several confounding changes, including the normalization layout, residual scaling, and output projection initialization. The observed performance differences therefore cannot be attributed solely to the gating mechanism. A controlled comparison that holds these design choices fixed is needed to isolate the contribution of gating.

4.The magnitude and scope of the performance gains are limited. Some improvements in average common-sense performance are small, average retrieval performance remains below that of fixed-schedule hybrids, and the long-context advantages do not hold consistently across domains.

5.The paper lacks a systematic evaluation of the gating threshold and its effect on the trade-off between predictive performance and computational cost. The offset coefficient is fixed at 0.2 based on a small-scale exploration using a different thresholding rule, which provides limited justification for the current setting. A more comprehensive comparison of task performance, attention firing rates, and measured latency across thresholds is needed.

Questions for authors

Appendix D reports attention firing rates of approximately 97–99% on CodeParrot, indicating that attention is activated at nearly every position. Does this reflect a genuine need for additional attention in this domain, or a domain shift in the entropy distribution that makes the frozen thresholds less appropriate? Under this setting, does gating still offer any performance or efficiency benefits over always-on attention?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 63Qz (1/4)

We thank the reviewer for a careful and specific review.

W1 and W2: Does enabling attention help more at high-entropy positions than at low-entropy positions? (Appendix D.3)

Yes. At a block's natural firing positions (high entropy), forced non-firing, one block at a time and with all weights fixed, raises perplexity by 0.36 to 1.79 in every block of both 1.5B models. At its natural non-firing positions (low entropy), forced firing in the same setting changes perplexity by at most 0.05 (Table 10). This indicates that AMOR's attention helps at the positions the model finds difficult.

To answer this question, we also ran new experiments (Section 5.3). They show that firing more is not universally beneficial: the firing-everywhere variant of AMOR-Mamba2 has a lower common-sense average and 38.2% higher FineWeb-Edu perplexity at 20K than canonical AMOR-Mamba2, which fires on ∼40% of positions in pretraining.

Table 10: Attention lowers perplexity at uncertain positions. 1.5B; held-out FineWeb-Edu.

Isolated blockNatural firing positions: Forced non-firing PPL ↓Natural firing positions: Forced firing PPL ↓Natural firing positions: Δ PPLNatural non-firing positions: Forced non-firing PPL ↓Natural non-firing positions: Forced firing PPL ↓Natural non-firing positions: Δ PPL
AMOR-Mamba2
014.1313.65-0.4812.1812.14-0.05
118.6517.87-0.7810.009.97-0.03
218.4616.68-1.7910.4710.46-0.01
AMOR-Gated DeltaNet
013.9913.63-0.3611.7811.77-0.01
117.9417.33-0.619.939.92-0.02
215.7814.57-1.2111.2011.21+0.01

W1: Why is entropy better suited than alternative confidence measures or learned routers? (Section 5.2; Appendix D.2)

Against top-1 probability, a CALM-style margin, a learned MoD-style router and a random gate, pretrained from scratch at 180M with the same pretraining firing rate of ∼40%, the AMOR entropy gate yields the lowest LAMBADA and CodeParrot perplexity on both backbones, the lowest PG-19 perplexity on the Gated DeltaNet backbone and the tied-highest common-sense average on the Mamba2 backbone (Table 1).

We also directly swapped the routing mechanism on the pretrained 1.5B checkpoints for alternative routing mechanisms, with all weights fixed and each block's firing rate matched to the entropy gate's: the entropy gate achieves the lowest perplexity on FineWeb-Edu, PG-19 and LAMBADA on both backbones (Table 9). Together, these two comparisons support our choice of entropy as the routing signal.

Table 1: Among routing mechanisms, the AMOR entropy gate yields the lowest LAMBADA and CodeParrot perplexity. 180M. The MoD-style router adds 2,307 trainable parameters, a linear layer per block. Common-sense uses the eight tasks and setup of Section 4.1 (Appendix E.1).

Routing mechanismAMOR-Mamba2: LAMB PPL ↓AMOR-Mamba2: CodeParrot PPL ↓AMOR-Mamba2: PG-19 PPL ↓AMOR-Mamba2: Common-sense avg acc ↑AMOR-Gated DeltaNet: LAMB PPL ↓AMOR-Gated DeltaNet: CodeParrot PPL ↓AMOR-Gated DeltaNet: PG-19 PPL ↓AMOR-Gated DeltaNet: Common-sense avg acc ↑
Confidence-based
↳ AMOR entropy gate240.27303.6287.0835.8234.20382.7282.0335.7
↳ Top-1 probability281.87327.6688.8235.7264.63460.1083.6535.8
↳ CALM-style margin277.12374.6988.3135.8308.70464.0683.3335.9
Learned
↳ MoD-style router315.64316.5986.7835.3283.50480.3882.3735.8
Random367.62549.1988.3435.5291.50465.2684.0934.8

W3: Can a controlled comparison that holds the normalization layout, residual scaling and output-projection initialization fixed isolate the gate? (Section 5.3; Appendix I)

Yes. AMOR-ungated makes four changes at once (Table 32): the normalization layout, the per-channel scaling α, the output initialization and the gate. To address this question, we pretrained design ablations from scratch to isolate them one at a time:

  • Gating. The firing-everywhere variant of AMOR-Mamba2 (κ = −∞; Section 5.3) keeps the normalization layout, the per-channel scaling α and the zero-initialized W_O of canonical AMOR-Mamba2, so it changes the gate alone. It has a lower common-sense average than canonical AMOR-Mamba2 (35.7 against 35.8) and higher 20K perplexity on all seven datasets, by 27% on PG-19 to 79% on CodeParrot (Tables 37 and 38).
AuthorsOpenReview ↗

Response to Reviewer 63Qz (2/4)

  • Normalization layout. AMOR-symmetric changes the layout alone. At 440M, it has a lower common-sense average than canonical AMOR on both backbones (38.9 against 39.6 with Mamba2; 39.9 against 40.5 with Gated DeltaNet; Table 33), and at 1.5B it trails canonical AMOR on four of six cloze retrieval tasks with Mamba2 and five with Gated DeltaNet (Table 35).
  • Per-channel scaling α. Each AMOR model pretrained without α at 440M still leads its own backbone in held-out FineWeb-Edu perplexity at 3K, in LAMBADA perplexity and in the common-sense average (39.4 against 39.0 with Mamba2; 40.1 against 39.7 with Gated DeltaNet; Table 36). α adds to this lead: it raises the common-sense average to 39.6 and 40.5 and lowers FineWeb-Edu perplexity at 20K on both backbones.
  • Zero-initialized output projection. The Gaussian W_O initialization variant of AMOR-Mamba2, pretrained at 180M with W_O from N(0, 0.02) like every other linear projection (Appendix C), scores slightly better than canonical AMOR-Mamba2 on every measure of Tables 37 and 38 (common-sense average 36.0 against 35.8; LAMBADA perplexity 239.23 against 240.27; 0.3% to 16% lower 20K perplexity on all seven datasets), so AMOR's results do not rely on the zero initialization. We keep it to stay true to AMOR's design principle: AMOR starts identical to its recurrent backbone, a complete language model that the AMOR blocks then refine (Section 3.1).

By isolating one factor at a time, these design ablations indicate that AMOR outperforms the baselines because of its design as a post-hoc hybrid with an entropy gate, not because of additional parameters or differences in the pretraining setup (Appendix I.4).

Table 38: Parameter count, output initialization and firing everywhere ablations: 20K perplexity comparison. 180M.

ModelGovReport 20K PPL ↓QMSum 20K PPL ↓NarrativeQA 20K PPL ↓Qasper 20K PPL ↓CodeParrot 20K PPL ↓PG-19 20K PPL ↓
AMOR-Mamba228.9849.26127.6265.15845.8994.35
↳ Gaussian W_O initialization27.9446.10120.1462.43709.2691.39
↳ fires everywhere (κ=−∞)46.6870.97165.00104.251509.94119.85
Mamba225.3342.92108.2354.94652.7884.02
↳ matched to AMOR's parameter count25.4143.69105.8954.87657.7582.16
Mamba2 Serial Hybrid62.6988.01253.48157.011582.45186.09
↳ matched to AMOR's parameter count88.4395.29270.80179.481245.02218.76

Table 36: Per-channel scaling α ablation. 440M.

ModelParamsFW-Edu: 3K PPL ↓FW-Edu: 20K PPL ↓LAMB PPL ↓Common-sense avg acc ↑
AMOR-Mamba2448.7M18.3020.1357.3739.6
↳ without α448.6M18.2620.4059.1539.4
Mamba2436.1M18.4618.7974.1939.0
AMOR-Gated DeltaNet442.1M17.9918.7350.1540.5
↳ without α442.1M17.9919.2753.1240.1
Gated DeltaNet429.5M18.1118.4162.3739.7

W4: Are the gains limited: small common-sense gains, retrieval below fixed-schedule hybrids, and long-context advantages that do not hold across domains? (Sections 4.1-4.3 and 5.1; Appendices A.2 and H)

No: the common-sense gains hold at every seed, and AMOR's routing trade-off does buy more long-context robustness across domains.

On common sense, the gains are consistent: as with the canonical seed 42, an AMOR variant leads at each of four additional pretraining seeds at 180M and in their average, and each AMOR model is above its own backbone at every seed (Table 5).

Fixed-schedule hybrids lead on cloze retrieval, as Section 4.2 states; AMOR's claim there is still a substantial gain over the pure recurrent baselines (Table 3), for example on S-NIAH at 2K, where AMOR scores 93.0 against 57.4 with Mamba2 and 93.6 against 52.1 with Gated DeltaNet (Figure 5).

Having as many attention layers as the fixed-schedule hybrids does not mean that AMOR uses the same amount of attention. In the fixed-schedule hybrids, every token gets attention updates at fixed depths; in AMOR, only the positions where a gate fires get the help of attention.

We argue that AMOR's conditional routing is a worthy trade-off. In return, an AMOR model has the best common-sense average at every scale (Table 2), both AMOR models stay above every fixed-schedule hybrid on LongBench (8.3 and 8.4 against 4.5-7.4; Table 4), and AMOR-Mamba2 decodes faster than both Mamba2 fixed-schedule hybrids from 64K on (Table 15). In future work, we hope to further improve AMOR's retrieval performance.

AuthorsOpenReview ↗

Response to Reviewer 63Qz (3/4)

Across domains, every model that uses attention faces RoPE distribution shift beyond its 3K pretraining length (Appendix H). However, with training-free NTK-aware RoPE scaling applied to all of them, an AMOR model has the lowest 20K perplexity on all seven datasets among the models that use attention (Table 31). Without NTK, an AMOR model is lowest on six of the seven; the exception is CodeParrot, where the gate fires on nearly every position, likely because our pretraining data FineWeb-Edu contains little code (see Q1). The fact that AMOR's firing rate is consistent across context lengths within each domain (Table 11) is a feature, not a bug.

Table 31: With NTK, an AMOR model has the lowest 20K perplexity on all seven datasets among the models that use attention. 1.5B; every model pretrained at 3K. NTK: Appendix E.4; forced firing in all three blocks: Appendix E.5.

ModelFW-Edu: 3K PPL ↓FW-Edu: 20K PPL ↓GovReport 20K PPL ↓QMSum 20K PPL ↓NarrativeQA 20K PPL ↓Qasper 20K PPL ↓CodeParrot 20K PPL ↓PG-19 20K PPL ↓
Mamba213.0713.0710.7719.9239.1320.68108.5530.98
Gated DeltaNet12.9012.8710.3619.3636.0619.6294.3429.32
Uses attention
↳ Transformer13.221317.45974.29953.071622.881596.1422531.421416.66
↳↳ + NTK13.2227.3023.0531.3974.9244.78279.4463.20
↳ Mamba2 Serial Hybrid12.7773.7390.56106.56224.18168.13906.44170.67
↳↳ + NTK12.7714.0911.8419.6542.2523.12100.6633.78
↳ Gated DeltaNet Serial Hybrid12.7629.5233.1853.95109.8974.78503.3585.88
↳↳ + NTK12.7613.5011.5019.0739.6422.1294.0332.53
↳ Mamba2 Fused Hybrid12.8514.6614.4028.7551.8629.35239.9337.13
↳↳ + NTK12.8513.2812.1620.3342.3625.77218.3432.72
↳ AMOR-Mamba213.0814.4214.0134.7257.3230.02421.7838.10
↳↳ + forced firing, all three blocks13.0615.1414.6435.8759.0631.11423.8039.39
↳↳ + NTK13.0813.2410.8118.9139.1320.8994.1131.12
↳ AMOR-Gated DeltaNet12.9113.6411.8728.4247.3526.30269.3334.13
↳↳ + forced firing, all three blocks12.9014.0412.1829.0148.0026.84269.9434.79
↳↳ + NTK12.9112.9810.4119.1437.7720.0989.6829.95

W5: How do task performance, firing rates and measured latency change across gating thresholds? (Section 5.3; Appendix D.6)

We added Section 5.3 and Appendix D.6. Under the paper's threshold rule, τ = μ+κσ, not the fixed-offset rule of the earlier exploration, we pretrained AMOR-Mamba2 from scratch at 180M with five settings, κ ∈ {+∞, 0.6, 0.2, −0.5, −∞}, from attention never firing (κ = +∞, for which the Mamba2 backbone stands in) to firing everywhere (κ = −∞; Table R6). We also changed κ on the pretrained 1.5B AMOR-Mamba2. Across both, we measured common-sense accuracy, firing rates, long-context robustness and decode latency.

A higher target firing rate in pretraining raises common-sense reasoning accuracy up to ∼70% of positions (κ = −0.5) but lowers it at 100% (κ = −∞), and comes at a steep cost: FineWeb-Edu perplexity at 20K rises by 12.3% at κ = −0.5 and by 38.2% at κ = −∞, and at 1.5B, decoding at 128K slows by 33.5% as the firing rate rises from 40% to 100% (Table R7; Figure 8).

Changing κ after pretraining moves the firing rate on held-out FineWeb-Edu from 13.2% at κ = 1.0 to 71.9% at κ = −0.5, and common-sense accuracy rises quickly up to κ = 0.2 and less with more firing (Appendix D.6). This places κ = 0.2 on the Pareto frontier between task accuracy and efficiency.

Table R6: Firing more in pretraining does not consistently improve common-sense accuracy and costs long-context robustness. 180M AMOR-Mamba2 pretrained from scratch with each κ, same data, token budget, recipe and seed; Section 5.3.

κ in pretrainingCommon-sense (avg acc)FW-Edu PPL at 20KPG-19 PPL at 20KCodeParrot PPL at 20K
+∞ (never fires; the Mamba2 backbone)34.7830.7084.02652.78
0.6 (19.2% firing)35.5631.1090.41694.53
0.2 (canonical; 39.3% firing)35.8231.9094.35845.89
−0.5 (70.4% firing)35.9435.81101.891144.96
−∞ (fires everywhere)35.6944.09119.851509.94
Reviewer_DZVpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the paper proposes amor, a post-hoc hybrid that appends attention blocks to a recurrent backbone (mamba2 or gated deltanet) and activates each block per token only when the backbone’s normalized next-token entropy exceeds an ema-based threshold. the gate has no learned parameters. with attention firing on ~40% of positions, an amor variant outperform pure recurrent, pure attention and fixed-schedule hybrid baselines at several model scales, improves retrieval over its backbone, and retains recurrent-style long-context robustness where rope-based models degrade.

Critical strengths
  • the problem is a real one. current hybrids decide at design time which layers get attention and then apply it to every token, so the quadratic cost is paid even where the recurrent state is enough. gating on when attention fires rather than where is an obvious thing to try and i have not seen it done this way.

  • the choice to route on the output distribution is the interesting part. almost all conditional-compute work routes on input representations (mod, moba); calm is the main exception and it uses the margin for early exit rather than for gating attention. amor uses the full entropy with no learned router, and the threshold rule is simple enough to reproduce from the paper.

  • attending to ~40% of positions with no drop in accuracy is a substantial efficiency gain, and it is backed by measured latency and memory at 1.5b rather than flop counts (fig. 7, table 11).

  • the experimental setup is careful: matched data, tokenizer, sequence length, optimizer and attention budget across eight architectures and three scales, with a seed study at 180m.

overall i like the paper and recommend acceptance. i believe addressing the weaknesses and questions below would further help strengthen it significantly.

Critical weaknesses
  • there is no adaptive baseline. every baseline is fixed-schedule, so the paper cannot say whether entropy is the right routing signal or whether any gate at ~40% firing would do as well. a mixture-of-depths router (Raposo et al., 2024) or a calm-style output-margin rule (Schuster et al., 2022) at the same firing rate would settle this.

  • the downstream tasks are common-sense multiple choice, cloze retrieval and longbench. these are the standard protocol for recurrent models but not where adaptive compute is expected to pay off. gsm8k (cobbe et al., 2021), math (hendrycks et al., 2021) and humaneval (chen et al., 2021) would be more informative, even at 1.5b. the common-sense gaps are also within seed noise: 45.8 vs 45.3 at 1.5b (table 1) against seed sds of 0.2 to 0.5 (table 4).

  • the explanation for the long-context robustness in section 4.3 is stated as intuition and left there. it is cheap to test: force the gate on at every position at long context and see whether the degradation of the rope-based models reappears.

  • section 5 is hard to follow. the latency and memory numbers are packed into one paragraph and would be clearer as a table.

nit: references and urls are not colored or hyperlinked per the neurips style. that styling would make it easier to read.

Questions for authors
  • does routing on the output distribution matter? compare against a learned router on h̃_t and a random mask at the same ~40% firing rate.
  • can you report gsm8k, math, aime, or humaneval at 1.5b?
  • what happens to longbench and the narrativeqa extrapolation curve if the gate is forced on at every position?
  • the native gate adds three lm-head evaluations over a 128k vocabulary per token (eq. 9). how much of the decode and prefill advantage remains without the distilled router?
3 comments
AuthorsOpenReview ↗

Response to Reviewer DZVp (1/2)

We thank the reviewer for their time and effort, and for the very helpful and insightful comments.

W1 and Q1: Does routing on the output distribution matter, against a MoD-style router on h̃_t, a CALM-style margin and a random mask at the same ∼40% firing rate? (Section 5.2; Appendix D.2)

Yes, routing on the output distribution matters. To answer this question, we added these baselines to the revised version (Section 5.2 and Appendix D.2).

Against top-1 probability, a CALM-style margin, a learned MoD-style router on h̃_t and a random gate, pretrained from scratch at 180M with the same pretraining firing rate of ∼40%, the AMOR entropy gate yields the lowest LAMBADA and CodeParrot perplexity on both backbones, the lowest PG-19 perplexity on the Gated DeltaNet backbone and the tied-highest common-sense average on the Mamba2 backbone (Table 1).

We also directly swapped the routing mechanism on the pretrained 1.5B checkpoints for alternative routing mechanisms, with all weights fixed and each block's firing rate matched to the entropy gate's: the entropy gate achieves the lowest perplexity on FineWeb-Edu, PG-19 and LAMBADA on both backbones (Table 9). Together, these two comparisons support our choice of entropy as the routing signal.

Table 1: Among routing mechanisms, the AMOR entropy gate yields the lowest LAMBADA and CodeParrot perplexity. 180M. The MoD-style router adds 2,307 trainable parameters, a linear layer per block. Common-sense uses the eight tasks and setup of Section 4.1 (Appendix E.1).

Routing mechanismAMOR-Mamba2: LAMB PPL ↓AMOR-Mamba2: CodeParrot PPL ↓AMOR-Mamba2: PG-19 PPL ↓AMOR-Mamba2: Common-sense avg acc ↑AMOR-Gated DeltaNet: LAMB PPL ↓AMOR-Gated DeltaNet: CodeParrot PPL ↓AMOR-Gated DeltaNet: PG-19 PPL ↓AMOR-Gated DeltaNet: Common-sense avg acc ↑
Confidence-based
↳ AMOR entropy gate240.27303.6287.0835.8234.20382.7282.0335.7
↳ Top-1 probability281.87327.6688.8235.7264.63460.1083.6535.8
↳ CALM-style margin277.12374.6988.3135.8308.70464.0683.3335.9
Learned
↳ MoD-style router315.64316.5986.7835.3283.50480.3882.3735.8
Random367.62549.1988.3435.5291.50465.2684.0934.8

W2 and Q2: Can you report GSM8K, MATH, AIME or HumanEval at 1.5B, and are the common-sense gaps within seed noise? (Appendix A.2)

In response, we ran GSM8K (5-shot chain of thought), MathQA and HumanEval on every 1.5B model (Table R2). At 30.7B pretraining tokens, the Chinchilla-optimal budget that every model shares (Section 4), every model is at the floor, so these benchmarks do not separate the models at this scale: GSM8K 0.0-0.1% (1.1-2.2% with flexible answer extraction), MathQA 23.5-25.4% against 20% chance, and HumanEval pass@1 0%. MATH and AIME are harder than GSM8K.

As for seed noise, we note that other architecture papers, such as Mamba, Mamba2, Mamba3 and Gated DeltaNet, report no variance across pretraining seeds (a single reported number per model). Unlike them, our submission already tests whether AMOR's advantage holds across seeds: every 180M architecture is pretrained with four additional seeds, 32 extra pretraining runs. It holds: as with the canonical seed 42, an AMOR variant leads at each of the four seeds and in their average, and each AMOR model is above its own backbone at every seed (Table 5). That the common-sense gaps fall within the seed SDs is an artifact of only five runs per architecture.

Table R2: Math and code are at the floor for every model at this scale. 1.5B, 30.7B FineWeb-Edu tokens; GSM8K 5-shot chain of thought, MathQA zero-shot (chance 20%), HumanEval pass@1.

ModelGSM8K strict (%)GSM8K flexible (%)MathQA (%)HumanEval (%)
Transformer0.01.424.60.0
Mamba20.02.224.70.0
Gated DeltaNet0.11.723.60.0
Mamba2 Serial Hybrid0.01.525.40.0
Gated DeltaNet Serial Hybrid0.11.825.00.0
Mamba2 Fused Hybrid0.01.424.10.0
AMOR-Mamba20.11.123.50.0
AMOR-Gated DeltaNet0.12.224.00.0

W3 and Q3: With attention firing everywhere at long context, does the degradation of the RoPE-based models reappear on LongBench and the NarrativeQA curve? (Section 4.3; Appendices E.5 and H)

In direction, but not in magnitude. Forced firing in all three blocks, with all weights fixed, raises 20K perplexity on all seven datasets of Table 31, NarrativeQA included, and lowers the LongBench averages from 8.27 to 8.02 and from 8.44 to 8.25 (Table R3).

AuthorsOpenReview ↗

Response to Reviewer DZVp (2/2)

Pretraining from scratch points the same way, and more strongly: the firing-everywhere variant of AMOR-Mamba2 has 38.2% higher FineWeb-Edu perplexity at 20K than canonical AMOR-Mamba2 at 180M (44.09 against 31.90), and higher 20K perplexity on all seven datasets (Table R4; Section 5.3). Both results are consistent with our hypothesis that AMOR mitigates RoPE distribution shift via its entropy gate, which Section 4.3 now states together with this test.

Table R3: Forced firing in all three blocks lowers LongBench and raises NarrativeQA perplexity beyond 3K. 1.5B, all weights fixed; LongBench averages 14 tasks.

Model, gateLongBench (avg)NarrativeQA PPL at 3K8K14K20K
AMOR-Mamba2, native entropy gate8.2760.8962.7957.1157.32
AMOR-Mamba2, forced firing in all three blocks8.0260.5363.0058.1459.06
AMOR-Gated DeltaNet, native entropy gate8.4458.9658.7449.6247.35
AMOR-Gated DeltaNet, forced firing in all three blocks8.2558.9358.9350.0948.00

Table R4: Firing everywhere in pretraining raises 20K perplexity on all seven datasets. 180M, every model pretrained from scratch with the same data, token budget, recipe and seed; Section 5.3 and Appendix I.4.

ModelFW-Edu 20K PPL ↓GovReport 20K PPL ↓QMSum 20K PPL ↓NarrativeQA 20K PPL ↓Qasper 20K PPL ↓CodeParrot 20K PPL ↓PG-19 20K PPL ↓
AMOR-Mamba231.9028.9849.26127.6265.15845.8994.35
↳ fires everywhere (κ=−∞)44.0946.6870.97165.00104.251509.94119.85
Mamba230.7025.3342.92108.2354.94652.7884.02
Mamba2 Serial Hybrid53.8362.6988.01253.48157.011582.45186.09

W4: Could Section 5.1's latency and memory numbers be a table? (Appendix F.3)

Yes; they are in Table 15, which Figure 7's caption cites: prefill latency, peak prefill memory and decode latency for all eight models from 8K to 128K, with AMOR's native entropy gate and distilled router as separate rows (excerpt in Table R5). The 10-page limit keeps the full table in the appendix.

Table R5: At 128K, AMOR-Mamba2 decodes faster than both Mamba2 fixed-schedule hybrids. 1.5B; one H100 PCIe, batch 1, natural firing rates. Table 15 gives every model and context length.

ModelContext length: 8KContext length: 32KContext length: 128K
Prefill latency (s) ↓
Mamba20.130.481.92
Mamba2 Serial Hybrid0.130.614.17
Mamba2 Fused Hybrid0.241.035.77
AMOR-Mamba20.261.136.30
↳ + distilled router0.140.664.40
Peak prefill memory (GB) ↓
Mamba23.856.3016.07
Mamba2 Serial Hybrid4.137.4820.88
Mamba2 Fused Hybrid4.368.1123.12
AMOR-Mamba213.0614.1718.60
↳ + distilled router4.066.5116.29
Decode latency (ms/token) ↓
Mamba216.2416.1316.15
Mamba2 Serial Hybrid16.5821.2139.42
Mamba2 Fused Hybrid18.1922.8041.07
AMOR-Mamba219.2222.6734.21
↳ + distilled router18.8222.1133.73

Q4: How much of the decode and prefill advantage remains without the distilled router? (Section 5.1; Appendix F.3)

Most of the decode advantage remains with the native entropy gate: at 128K, AMOR-Mamba2 decodes at 34.21 ms/token (33.73 with the distilled router), against 39.42 and 41.07 for the serial and fused hybrids (Table R5), and with either gate it overtakes the fused hybrid in decode throughput between 8K and 16K and the serial hybrid between 32K and 64K (Table 15).

The distilled router, with 0.21% extra parameters, mainly lowers prefill latency and peak memory. Without it, the three extra LM-head evaluations make AMOR-Mamba2 slower to prefill than both fixed-schedule hybrids (6.30 s against 4.17 and 5.77 at 128K), although at 128K its peak prefill memory is still below that of both fixed-schedule hybrids (18.60 GB against 20.88 and 23.12; Table 15).

N1: Could references and URLs be colored and hyperlinked?

Done: references and URLs are now colored and hyperlinked.

Once again, we thank the reviewer for their thoughtful, careful and encouraging comments. We have invested substantial effort in addressing all reviewers' concerns thoroughly and hope that our detailed responses and additional measurements clarify the strengths of AMOR. AMOR is the product of a genuine exploration that draws inspiration from human cognition, and we have worked hard to test every part of its design. We are grateful for the reviewer's kind support and hope that these clarifications warrant a higher score.

Dear Reviewer DZVp,

Thank you for your insightful and helpful comments. We took each of them very seriously and, in response, pretrained 14 new models from scratch and conducted additional evaluations across all 1.5B models.

These experiments substantially strengthened the paper and provided further support for our original conclusions. We hope that our revisions have addressed your concerns, and would be grateful if you would consider reflecting this in your final assessment.

Thank you again for your time and thoughtful feedback.

The authors

Reviewer_jf4QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AMOR, a hybrid architecture that retains a complete Mamba2 or Gated DeltaNet backbone and appends three attention-only blocks. Each block uses predictive entropy to activate attention selectively at uncertain token positions while maintaining a complete KV cache for future retrieval. The authors also introduce lightweight distilled routers to reduce the overhead of computing entropy with the language-model head. Across three model scales, AMOR improves common-sense reasoning and retrieval over its recurrent backbones, remains competitive with fixed-schedule hybrids, and preserves the backbones' long-context robustness. At sufficiently long context lengths, AMOR-Mamba2 also achieves lower decoding latency than some fixed-schedule hybrids.

Critical strengths
  1. Clear presentation. The paper is well-written and clearly organized. The motivation and proposed method are easy to follow, with sufficient implementation details and intuitive illustrations.

  2. Simple and practical design. AMOR introduces a straightforward entropy-based routing mechanism. The evaluation covers multiple model scales and recurrent backbones, demonstrating consistent retrieval gains over pure recurrent models and potential efficiency benefits for long-context decoding.

Critical weaknesses
  1. The routing mechanism is insufficiently justified and isolated. Predictive entropy does not necessarily indicate a need for contextual retrieval, and the paper does not compare it with trained or budget-matched alternative routers. Moreover, AMOR adds capacity and uses architectural choices absent from the fixed-schedule baselines, while the existing ablations do not isolate these factors or systematically justify the number of attention blocks. See Questions (1)–(3) and (7).

  2. The practical accuracy–efficiency trade-off remains unclear. AMOR underperforms fixed-schedule hybrids on cloze retrieval, does not demonstrate effective context utilization at lengths approaching those used for efficiency evaluation, and provides no training-time attention savings. Its inference benefits also depend strongly on the backbone, context length, domain-specific firing rate, and an additional distilled router. See Questions (4)–(6).

Questions for authors

1. Motivation for entropy-based routing. AMOR assumes that high-entropy predictions benefit more from attention, whereas low-entropy predictions can rely on the recurrent backbone. However, predictive uncertainty does not necessarily indicate a need for contextual retrieval. Does token-level entropy correlate with the loss reduction obtained by enabling attention for that token? How does AMOR handle confidently incorrect predictions that still require retrieval?

2. Comparison with alternative routing strategies. Entropy is only one possible signal for allocating attention. What advantages does it offer over a lightweight learned router optimized under the same attention budget? The random router in Table 25 is untrained and does not control for firing rate, so it is not a sufficiently informative baseline. Comparisons with trained routers and other simple uncertainty metrics at matched per-block firing rates would better establish the value of entropy-based routing.

3. Fairness of architectural comparisons. AMOR appends three attention blocks to a complete recurrent backbone, resulting in more parameters and training FLOPs than the serial and fused hybrids. How much of the improvement comes from routing rather than the added capacity or architectural differences? In particular, per-channel residual scaling and zero-initialized output projections may independently affect performance. An otherwise identical always-on post-hoc baseline, together with parameter- or compute-matched hybrids, would help isolate these factors.

4. Retrieval performance and the accuracy–efficiency trade-off. AMOR consistently underperforms fixed-schedule hybrids on average cloze retrieval despite using the same number of attention blocks and equivalent KV-cache storage. At 1.5B, AMOR-Mamba2 achieves 27.1% versus 34.0% for the serial hybrid, while AMOR-Gated DeltaNet achieves 30.0% versus 36.9%. Does conditional routing still provide a better overall trade-off after accounting for this retrieval degradation?

5. Long-context extrapolation versus effective context utilization. The length-extrapolation results mainly show that AMOR preserves the robustness of its recurrent backbones, but stable perplexity does not demonstrate effective use of distant context. AMOR still degrades substantially on 4K S-NIAH, whereas inference efficiency is evaluated up to 128K. Could the authors evaluate longer-context retrieval benchmarks such as RULER to establish the effective context length? Comparisons with position-extended attention baselines would also separate the effect of conditional routing from the limitations of vanilla RoPE extrapolation.

6. Practical efficiency and deployment overhead. AMOR uses dense attention with output masking during training, so routing does not reduce training attention FLOPs. The native entropy gate also incurs substantial language-model-head overhead, requiring an additional post-training distillation stage. What is AMOR's end-to-end advantage over fixed-schedule hybrids when both training and inference costs are considered? Moreover, firing rates vary substantially by domain and approach 100% on CodeParrot. How well do the distilled routers' decisions, task performance, and inference efficiency transfer to unseen domains and workloads?

7. Attention block count. Appendix H compares K=1K=1 and K=3K=3, but K=1K=1 performs better for Mamba2 at 180M and 440M, while the gains from K=3K=3 for Gated DeltaNet are small or absent. The ablation reports only common-sense accuracy, without retrieval or efficiency results. Could the authors analyze how the number of attention blocks affects retrieval and the accuracy–efficiency trade-off?

5 comments
AuthorsOpenReview ↗

Response to Reviewer jf4Q (1/4)

We thank the reviewer for a careful and specific review.

W1 and Q1: Does entropy track the benefit of attention, and how does AMOR handle confidently wrong predictions that still need retrieval? (Appendix D.3; Section 4.2)

Yes. At a block's natural firing positions, forced non-firing, one block at a time and with all weights fixed, raises perplexity by 0.36 to 1.79 in every block of both 1.5B models. At its natural non-firing positions, forced firing in the same setting changes perplexity by at most 0.05 (Table 10). This indicates that AMOR's attention helps at the positions the model finds difficult.

To answer this question, we also ran new experiments (Section 5.3). They show that firing more is not universally beneficial: the firing-everywhere variant of AMOR-Mamba2 has a lower common-sense average and 38.2% higher FineWeb-Edu perplexity at 20K than canonical AMOR-Mamba2, which fires on ∼40% of positions in pretraining.

As our Limitations note, a confidently wrong prediction can keep the gate from firing. We do not claim a causal relationship between gate firing and the need for retrieval. However, we consistently observe increased retrieval capability with AMOR's gated attention compared to its backbones (Section 4.2).

Table 10: Attention lowers perplexity at uncertain positions. 1.5B; held-out FineWeb-Edu.

Isolated blockNatural firing positions: Forced non-firing PPL ↓Natural firing positions: Forced firing PPL ↓Natural firing positions: Δ PPLNatural non-firing positions: Forced non-firing PPL ↓Natural non-firing positions: Forced firing PPL ↓Natural non-firing positions: Δ PPL
AMOR-Mamba2
014.1313.65-0.4812.1812.14-0.05
118.6517.87-0.7810.009.97-0.03
218.4616.68-1.7910.4710.46-0.01
AMOR-Gated DeltaNet
013.9913.63-0.3611.7811.77-0.01
117.9417.33-0.619.939.92-0.02
215.7814.57-1.2111.2011.21+0.01

W1 and Q2: What does entropy offer over a lightweight learned router and other simple uncertainty metrics at matched per-block firing rates? (Section 5.2; Appendix D.2)

We agree that the random router of Table 25 as submitted (Table 29, now called the untrained router) is an uncalibrated baseline, so we added matched comparisons, pretraining eight new models from scratch.

Against top-1 probability, a CALM-style margin, a lightweight learned MoD-style router and a random gate, pretrained from scratch at 180M with the same pretraining firing rate of ∼40%, the AMOR entropy gate yields the lowest LAMBADA and CodeParrot perplexity on both backbones, the lowest PG-19 perplexity on the Gated DeltaNet backbone and the tied-highest common-sense average on the Mamba2 backbone (Table 1).

We also directly swapped the routing mechanism on the pretrained 1.5B checkpoints for alternative routing mechanisms, with all weights fixed and each block's firing rate matched to the entropy gate's: the entropy gate achieves the lowest perplexity on FineWeb-Edu, PG-19 and LAMBADA on both backbones (Table 9). Together, these two comparisons support our choice of entropy as the routing signal.

Table 1: Among routing mechanisms, the AMOR entropy gate yields the lowest LAMBADA and CodeParrot perplexity. 180M. The MoD-style router adds 2,307 trainable parameters, a linear layer per block. Common-sense uses the eight tasks and setup of Section 4.1 (Appendix E.1).

Routing mechanismAMOR-Mamba2: LAMB PPL ↓AMOR-Mamba2: CodeParrot PPL ↓AMOR-Mamba2: PG-19 PPL ↓AMOR-Mamba2: Common-sense avg acc ↑AMOR-Gated DeltaNet: LAMB PPL ↓AMOR-Gated DeltaNet: CodeParrot PPL ↓AMOR-Gated DeltaNet: PG-19 PPL ↓AMOR-Gated DeltaNet: Common-sense avg acc ↑
Confidence-based
↳ AMOR entropy gate240.27303.6287.0835.8234.20382.7282.0335.7
↳ Top-1 probability281.87327.6688.8235.7264.63460.1083.6535.8
↳ CALM-style margin277.12374.6988.3135.8308.70464.0683.3335.9
Learned
↳ MoD-style router315.64316.5986.7835.3283.50480.3882.3735.8
Random367.62549.1988.3435.5291.50465.2684.0934.8
AuthorsOpenReview ↗

Response to Reviewer jf4Q (2/4)

W1 and Q3: How much of the improvement comes from routing rather than the added capacity, the per-channel scaling or the zero-initialized output projection? (Appendices B and I)

Every model in our comparison, including these controls, is pretrained on the same data with the same token budget, the same number of optimization steps and the same recipe (Section 4; Appendix C). To address this question, we pretrained six design ablations from scratch to isolate the factors one at a time:

  • Always-on post-hoc baseline. The firing-everywhere variant of AMOR-Mamba2 (κ = −∞; Section 5.3) is this baseline: it keeps the layout, the per-channel scaling α and the zero-initialized W_O of canonical AMOR-Mamba2, so it changes the gate alone. It has a lower common-sense average than canonical AMOR-Mamba2 (35.7 against 35.8) and higher 20K perplexity on all seven datasets (Tables 37 and 38).
  • Added capacity. Mamba2 and the Mamba2 serial hybrid, matched to AMOR-Mamba2's parameter count at 180M, still trail AMOR-Mamba2 in LAMBADA perplexity (430.51 and 367.41 against 240.27) and in the common-sense average (34.7 and 35.7 against 35.8), and at 20K the serial hybrid degrades whether or not its parameter count is matched to AMOR-Mamba2's (FineWeb-Edu perplexity 53.83 and 56.68 against 31.90; Table 37).
  • Per-channel scaling α. Each AMOR model pretrained without α at 440M still leads its own backbone in held-out FineWeb-Edu perplexity at 3K, in LAMBADA perplexity and in the common-sense average (39.4 against 39.0 with Mamba2; 40.1 against 39.7 with Gated DeltaNet; Table 36). α adds to this lead: it raises the common-sense average to 39.6 and 40.5 and lowers FineWeb-Edu perplexity at 20K on both backbones.
  • Zero-initialized output projection. The Gaussian W_O initialization variant of AMOR-Mamba2, pretrained at 180M with W_O from N(0, 0.02) like every other linear projection (Appendix C), scores slightly better than canonical AMOR-Mamba2 on every measure of Tables 37 and 38 (common-sense average 36.0 against 35.8; LAMBADA perplexity 239.23 against 240.27; 0.3% to 16% lower 20K perplexity on all seven datasets), so AMOR's results do not rely on the zero initialization. We keep it to stay true to AMOR's design principle: AMOR starts identical to its recurrent backbone, a complete language model that the AMOR blocks then refine (Section 3.1).

By isolating one factor at a time, these design ablations indicate that AMOR outperforms the baselines because of its design as a post-hoc hybrid with an entropy gate, not because of additional parameters or differences in the pretraining setup (Appendix I.4). We thank the reviewer for raising this point, which led to a more thorough investigation of this question; these ablations are now in Appendix I of the revised version.

Table 37: Parameter count, output initialization and firing everywhere ablations. 180M.

ModelParamsFW-Edu: 3K PPL ↓FW-Edu: 20K PPL ↓LAMB PPL ↓Common-sense avg acc ↑
AMOR-Mamba2184.4M28.8731.90240.2735.8
↳ Gaussian W_O initialization184.4M28.8431.79239.2336.0
↳ fires everywhere (κ=−∞)184.4M28.2944.09234.8335.7
Mamba2177.3M29.8330.70352.2834.8
↳ matched to AMOR's parameter count184.4M29.3330.35430.5134.7
Mamba2 Serial Hybrid174.5M28.1353.83398.9035.6
↳ matched to AMOR's parameter count184.4M27.4356.68367.4135.7

Table 36: Per-channel scaling α ablation. 440M.

ModelParamsFW-Edu: 3K PPL ↓FW-Edu: 20K PPL ↓LAMB PPL ↓Common-sense avg acc ↑
AMOR-Mamba2448.7M18.3020.1357.3739.6
↳ without α448.6M18.2620.4059.1539.4
Mamba2436.1M18.4618.7974.1939.0
AMOR-Gated DeltaNet442.1M17.9918.7350.1540.5
↳ without α442.1M17.9919.2753.1240.1
Gated DeltaNet429.5M18.1118.4162.3739.7

W2 and Q4: Does conditional routing still give a better overall trade-off, given the lower cloze retrieval than fixed-schedule hybrids? (Sections 4.1-4.3 and 5.1)

Fixed-schedule hybrids lead on cloze retrieval, as Section 4.2 states; AMOR's claim there is still a substantial gain over the pure recurrent baselines (Table 3), for example on S-NIAH at 2K, where AMOR scores 93.0 against 57.4 with Mamba2 and 93.6 against 52.1 with Gated DeltaNet (Figure 5).

Having as many attention layers as the fixed-schedule hybrids does not mean that AMOR uses the same amount of attention. In the fixed-schedule hybrids, every token gets attention updates at fixed depths; in AMOR, only the positions where a gate fires get the help of attention.

AuthorsOpenReview ↗

Response to Reviewer jf4Q (3/4)

We argue that AMOR's conditional routing is a worthy trade-off. In return, an AMOR model has the best common-sense average at every scale (Table 2), both AMOR models stay above every fixed-schedule hybrid on LongBench (8.3 and 8.4 against 4.5-7.4; Table 4), and AMOR-Mamba2 decodes faster than both Mamba2 fixed-schedule hybrids from 64K on (Table 15). In future work, we hope to further improve AMOR's retrieval performance.

W2 and Q5: What is AMOR's effective context on longer retrieval benchmarks such as RULER, and do position-extended attention baselines separate routing from vanilla RoPE? (Section 4.3; Appendices E.4, E.5 and H)

In addition to stable perplexity, the paper measured long-context capability on LongBench (Section 4.3), following the Gated DeltaNet paper: 14 tasks of single- and multi-document question answering, summarization, few-shot learning and code completion on inputs of up to 32K tokens, scored on the model's outputs rather than its perplexity. There, AMOR stays within 0.5 points of its recurrent backbone and outperforms the Transformer and every fixed-schedule hybrid (Table 4).

S-NIAH, drawn from RULER, is in the paper up to 4K, where AMOR outperforms the recurrent baseline at 2K and 4K (Table 3). We also conducted new tests to extend S-NIAH to 16K and 32K (Table 30), where every model that uses attention scores zero with vanilla RoPE.

To separate the effect of conditional routing from the limitations of vanilla RoPE extrapolation, we applied training-free NTK-aware RoPE scaling (NTK) to every model that uses attention, and additionally tested forced firing in all three AMOR blocks (Appendices E.4 and E.5).

Forced firing changes the S-NIAH score by at most 0.7 points at any length, with or without NTK: the routing mechanism itself has little effect on retrieval at long context.

NTK, which changes only the position encoding, raises AMOR's score at 4K by 11.6 and 7.4 points but recovers the degradation only partially: AMOR's loss in retrieval capability is in part still due to RoPE distribution shift (Appendix H).

Every model in this paper is pretrained at 3K. AMOR's long-context robustness and efficiency are thus desirable properties for a subsequent long-context mid-training stage, where decoding cost up to 128K and beyond matters. We leave this stage to future work.

Table 30: AMOR's retrieval beyond the pretraining length. 1.5B models that use attention, pretrained at 3K; mean of the three S-NIAH variants (Appendix E.2). The 4096 column is a different draw of 500 samples per variant than Table 3 and Fig. 5. Forced firing in all three blocks: Appendix E.5.

Model Context lengthS-NIAH↑: 4096S-NIAH↑: 16384S-NIAH↑: 32768S-NIAH with NTK↑: 4096S-NIAH with NTK↑: 16384S-NIAH with NTK↑: 32768
Transformer10.40.00.074.10.00.0
Mamba2 Serial Hybrid51.70.00.064.613.03.7
Gated DeltaNet Serial Hybrid43.70.00.073.86.02.7
Mamba2 Fused Hybrid58.50.00.067.34.05.3
AMOR-Mamba239.50.00.051.14.01.0
↳ + forced firing, all three blocks40.00.00.051.54.31.0
AMOR-Gated DeltaNet47.10.00.054.54.71.0
↳ + forced firing, all three blocks47.20.00.054.65.31.3

W2 and Q6: What is AMOR's end-to-end advantage once pretraining and inference costs are both counted, and how well do the distilled routers transfer to unseen domains? (Appendices F and G)

AMOR's end-to-end advantage is at inference, and its distilled router transfers to unseen domains. Pretraining uses dense attention with output masking, so AMOR-Mamba2 pretrains at 87% of the serial hybrid's throughput (Appendix F.1). At inference, most of the decode advantage holds with the native entropy gate: at 128K, AMOR-Mamba2 decodes at 34.21 ms/token, against 39.42 and 41.07 for the serial and fused hybrids (Table 15). The distilled router, with 0.21% extra parameters, mainly lowers prefill latency and peak memory.

The near-complete firing on CodeParrot is likely due to our pretraining data FineWeb-Edu, which is mainly educational, textbook-like web text that contains little code. The fact that AMOR's firing rate is consistent across context lengths within each domain (Table 11) is a feature, not a bug. With code in a future pretraining mix, we expect the firing rate on code to move toward the ∼40% target.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment to all reviewers

We thank the reviewers for their useful reviews, which led to the following new experiments, from 14 new models pretrained from scratch to new evaluations of every 1.5B model. Running them took time, and we apologize for the slight delay in our response.

Comparison of routing mechanisms (Section 5.2; Appendices D.2 and D.3)

To address concerns raised by Reviewers jf4Q, DZVp and 63Qz regarding the comparison of routing mechanisms, we pretrained the AMOR entropy gate, top-1 probability, a CALM-style margin, a learned MoD-style router and a random gate from scratch at 180M, and swapped the entropy gate on the 1.5B checkpoints for the other confidence-based mechanisms or random positions at matched firing rates. The entropy gate yields the lowest LAMBADA and CodeParrot perplexity on both backbones, the lowest PG-19 perplexity on Gated DeltaNet and the tied-highest common-sense average on Mamba2, and leads the confidence-based mechanisms; AMOR's attention helps at the positions the model finds difficult.

κ and the gating threshold (Section 5.3; Appendix D.6)

To address concerns raised by Reviewer 63Qz regarding the gating threshold, we pretrained AMOR-Mamba2 from scratch at 180M with five settings of κ (which sets how often the gate fires), the Mamba2 backbone standing in for κ = +∞, and changed κ after pretraining at 1.5B. κ = 0.2 lies on the Pareto frontier between task accuracy and efficiency.

Isolating the per-channel scaling, the parameter count, the output initialization and the gate (Appendices B and I)

To address concerns raised by Reviewers jf4Q and 63Qz regarding what drives AMOR's gains, we pretrained from scratch at 180M and 440M with one change at a time (Table 32). The baselines matched to AMOR's parameter count still trail it in LAMBADA perplexity, and firing everywhere in pretraining raises FineWeb-Edu perplexity at 20K by 38.2% for AMOR-Mamba2. These ablations indicate that AMOR outperforms the baselines because of its design as a post-hoc hybrid with an entropy gate, not because of additional parameters or differences in the pretraining setup.

Table 32: AMOR design ablations. These ablations indicate that AMOR outperforms the baselines because of its design as a post-hoc hybrid with an entropy gate, not because of additional parameters or differences in the pretraining setup.

VariantAsymmetric layoutAttention-only blocksPer-channel scaling αZero-initialized W_OAMOR entropy gate ♥Tables
AMOR ♥✓✓✓✓✓
AMOR-ungated✗ symmetric✓✗✗ Gaussian✗33
AMOR-symmetric✗ symmetric✓✓✓✓33-35
AMOR-MLP✓✗✓✓✓33
without α✓✓✗✓✓36
Gaussian W_O initialization✓✓✓✗ Gaussian✓37, 38
fires everywhere (κ = −∞)✓✓✓✓✗37, 38

Block count (Appendix I.5)

To address concerns raised by Reviewer jf4Q regarding the number of AMOR blocks, we compared one against three AMOR blocks on retrieval, LongBench and decode latency at 1.5B. The results show a trade-off, where three blocks raise retrieval accuracy and one block decodes faster.

Long context (Section 4.3; Appendices E.4, E.5 and H)

To address concerns raised by Reviewers jf4Q, DZVp and 63Qz regarding long-context behavior, we added S-NIAH at 16K and 32K, training-free NTK-aware RoPE scaling for every model that uses attention, and forced firing in all three AMOR blocks. Beyond the 3K pretraining length, with or without NTK, AMOR stays robust in behavior.

The distilled router on unseen data (Appendix G.1)

To address concerns raised by Reviewer jf4Q regarding the distilled router on unseen domains, we evaluated it on four of the paper's length-extrapolation datasets, which lie outside both the model's pretraining data and the router's fitting data. The router and the native entropy gate fire at nearly the same positions.

One correction, which we found ourselves while preparing this response: our length-extrapolation runs with the native entropy gate had let the gate thresholds adapt during evaluation. We reran them with the thresholds fixed, as in every other evaluation; Figures 6 (right) and 9 and the Appendix G length tables are updated, the conclusions are unchanged, and the distilled router's apparent CodeParrot advantage in Appendix G disappears (the router and the native entropy gate now agree within 3.9% everywhere).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.