acceptodds
Under review as a conference paper at ICLR 2027

Where Does the Recurrence Help? Retrieval-Distance Attribution in Hybrid Sequence Models

Abstract

Attention costs quadratic time in sequence length, and the two standard remedies fail differently. A sliding window bounds cost but cannot see past its edge — a positional limit. A compressed recurrence bounds cost but retains only a lossy summary of the prefix — a capacity limit. Hybrid architectures combine both, but evaluate with aggregate quality metrics that average the two regimes into a single number revealing neither. We present HALA, a block that computes sliding-window attention and a low-rank gated recurrence in parallel, mixes them with a learned per-head gate, and carries no position-wise feed-forward sublayer. Our contribution is less the architecture than the measurement. We stratify tokens by how far back their antecedent lies and attribute each path's contribution directly. Against an ablation with the recurrence removed, HALA is statistically indistinguishable on antecedents inside the 64-token window and consistently better outside it, by , and perplexity across three domains — and this advantage grows monotonically with scale, from at M non-embedding parameters to at M. On associative recall the separation from pure recurrence is categorical: Mamba-2 sits at chance across a full learning-rate sweep where HALA reaches . We are equally explicit about the limits. HALA does not lead aggregate perplexity in any domain: Transformer and Jamba are ahead throughout. Against Mamba-2 the comparison is a crossover with a legible mechanism — HALA leads on antecedents inside the window, Mamba-2 on those beyond it, and Mamba-2 carries the recurrent state per layer. Length-extrapolation stability is inherited from recurrent state rather than contributed, and all results sit at –M non-embedding parameters.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hybrid architectures, state space models, linear attention, associative recall, sliding window attention, efficient sequence modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9MDeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce HALA, a block that uses adjacent sliding-window attention (SWA) and linear recurrence blocks for parallel processing of sequences before mixing their outputs using a learned per-head gating function.

Critical strengths

The experiments seem to have been conducted with some technical care and the supplementary material does contain code that appears to match what is written in the work.

Critical weaknesses

Novelty or Uniqueness of Question of Interest

The novelty of the method appears to be fairly limited, being extremely similar to the Hymba architecture in its use of two adjacent components, namely a SWA component and a linear component, that processes the sequence separately before being mixed together by a gated projection. The authors admit this themselves in the introduction but there isn't any additional details that highlight how HALA is motivated differently and what exactly is the benefit of this specific instantiation compared to existing techniques.

The authors then frame the motivation of the work is to answer "whether the two paths actually divide labour the way the design assumes". However, there are no real experiments to validate this. Comparing against full attention or fixed recurrence is limited to AR accuracy or perplexity evaluations, thus there is no way of understanding directly what information is being handled by each individual branch. For example, in AR, this could be done with measing some type of shapely value or influence score just by seeing how marginal changes in target tokens influence the outputs of different branches. While this could be evaluated in many ways, the paper in its current form doesn't support the claim that is being made in the introduction.

Finally, the authors claim that their method introduces the idea of a learned per-head mixing gate rather than a fixed rule, however the actual implementation is simply modifying a mixing constant based on an additional MLP + sigmoid mixture, which is not novel. The ablations using the static and hard method show that this doesn't lead to very meaningful changes in performance nor does the learnt version show a distinct advantage over the other variants, which further raises issues with the actual contributions of the work.

Experimental Settings

  • The size of the models being experimented with are very small (<100M non-embedding parameters), thus there are signficant questions about the actual abilites of these models beyond synthetic AR accuracy. Perplexity results shows HALA to generally lag behind the baselines (Table 3 in Appendix G.1), which is a potentially problematic signal of performance beyond synethic examples where the representation of information is not as compact and simple to retrieve compared to AR tasks.
  • For throughput and memory results (Table 2), HALA doesn't seem to show a major improvement untill 128K sequence lengths, so the practical benefit of such an architecture just from this perspective is also limited. While it is possible that shorter lengths could see some benefit as well, the trend being represented would suggest that only for inordinately long sequences could such a paradigm be useful, which is a rather limiting factor.

Presentation

The work reads as having been written directly by an AI tool or at the very least having undergone substantial AI editing, which makes the narrative and flow of the work very difficult to follow at times. Details are often scattered in individual paragraphs that are difficult to follow. This is the same with the code, which appears to be primarily AI written and thus somewhat truncated in organization, making specific components or details somewhat hard to find or navigate.

Questions for authors

Please refer to weaknesses. While I do not want to constrain the authors to specific experiments and prefer to leave them to decide what is of importance without my biasing, there are suggestions regarding what comparisons could be useful or the type of experiments that would be necessary to make this work convincing or clarify any issues that were raised.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9MDe

Thank you for the direct critique. We agree with much of it and restructured the paper accordingly.

Novelty vs Hymba. We agree the block is not novel and no longer present it as a contribution. The contribution is a measurement: where, by retrieval distance, a hybrid's recurrent path helps. Hymba and other within-block hybrids report aggregate quality, which mixes regimes where the recurrence helps with regimes where it does not.

Direct evidence of what each branch handles. We added two branch-level analyses.

  1. On MQAR, HALA answers 99.2% of queries whose key is inside the window and 0.1% of the rest. Removing the recurrent branch at inference changes neither number (§5.3). The window carries exact recall here; the recurrence contributes nothing.
  2. On natural text, we knocked out one branch at inference by zeroing it or replacing it with its mean. Loss rises by 1.8–2.8 nats at every distance, with no difference between near and far tokens. The trained branches are entangled, so knockout-based influence scores attribute everything to both. That is why our attribution uses training counterfactuals instead: the same block trained without the branch. Against a capacity-matched, learning-rate-tuned version, the recurrence contributes nothing detectable within the window and 0.07–0.09 nats in every bin beyond it (§5.1).

The gate. Agreed: it is a standard MLP followed by a sigmoid. A learned gate beats a fixed 0.5 blend by 1.36, 0.40 and 1.63 perplexity on the three domains, and hard and soft gating perform the same. We no longer claim it as novel. Its main use in the paper is as an instrument for measuring routing.

What this instantiation buys. We tested the block's design choices at equal parameters (§5.4). Restoring an FFN of width 480, in exchange for two layers, improves perplexity by about 2.7 and also improves far-token loss. A smaller FFN, paid for with latent rank, leaves far-token performance unchanged, so latent rank is not the bottleneck. Omitting the FFN is therefore a cost, and the paper now states it.

Scale and perplexity. Agreed. The models have 11.9–74M non-embedding parameters. HALA trails the baselines on aggregate perplexity when each architecture uses one learning rate. Tuning HALA alone changes its books perplexity from 44.75 to 40.77 (3 seeds), so we no longer rank architectures against each other.

Throughput. One correction: HALA's time to first token breaks even with the Transformer between 8k and 16k tokens, not at 128k. At 131k it is 6.5× faster (127.4 vs 831.4 ms). We agree the benefit is limited to long contexts, and HALA has the highest peak memory.

Presentation and code. The abstract, introduction, results and conclusion now state the same three findings. We rewrote the README, which now has a table mapping every figure and table to its script and command. AI use is disclosed in the paper.

I appreciate the response from the authors.

However there are many issues that remain; the scale of experiments still is quite small to enable any assessments about the applicability of the approach to any particular task of use, while evidence of superiority doesn't appear to be stable. The main benefit is that there is an aggregate speed-up at long contexts, but I do question again if this would be practically at large scales where it is not only the architecture format that affects this measure.

Thus I prefer to maintain my current assessment, unless there is some scaling to significantly larger models or more realistic (or meaningful, per the authors own interpretation of this word) tasks that show advantages that are relevant in a more practical sense.

(Do further note that parts of the rebuttal and general response reads as having been significantly edited through the use of AI as well, which makes some of the additional details added difficult to understand.)

AuthorsOpenReview ↗

Response to Reviewer 9MDe

Thank you for reading our response.

We agree on scale. All models are below 100M parameters, and we cannot train significantly larger models within the discussion period. The paper lists this as its main limitation.

One clarification, since our framing may have been unclear. The paper does not claim that HALA is superior, or that the speed-up is its main benefit. On aggregate perplexity HALA trails the baselines, and the revision says so. The claim is narrower: a measurement of where a hybrid's recurrent path helps. It helps only beyond the attention window, and this held against a capacity-matched, tuned baseline. Whether it holds at larger scale is an open question, and the paper says that too.

AI assistance is disclosed in the paper. We are sorry that parts were hard to follow. If a specific point is unclear, we will answer it directly.

Reviewer_DmqQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HALA, a block that runs sliding-window attention and a low-rank gated linear recurrence in parallel, mixes them with a learned per-head gate, and has no FFN. The authors state that the main contribution is the evaluation rather than the architecture. Tokens that complete a bigram already seen earlier in the block (AR tokens) are grouped by distance to the earlier occurrence (near ≤64, far >64, >256). HALA is then compared against ablations with one path removed.

The main result is that removing the recurrence makes no measurable difference on near tokens but consistently hurts on far tokens. The far-token gap is 1.12, 0.30 and 0.90 perplexity on WikiText-103, Magicoder and PG-19, with three seeds. The paper also reports:

  • a width ladder on PG-19;
  • MQAR results;
  • a near/far crossover against Mamba-2;
  • gate telemetry, including a null result on MQAR;
  • length extrapolation to 32k;
  • efficiency measurements.

HALA trails Transformer, Jamba and Mamba-2 on aggregate perplexity in all three domains, and the paper says so openly.

Critical strengths

1. A clear near/far effect, consistent across three domains. Removing the recurrence raises far-token perplexity by 1.12, 0.30 and 0.90. Near-token perplexity changes by at most 0.13, and accuracy moves the same way. This turns a generic "an extra component helps" result into a specific statement about where the recurrence helps. It also gives empirical support to a design assumption that several hybrid models rely on.

2. Transparent and careful reporting. The paper is open about where HALA loses: aggregate perplexity, peak memory, and the null gate result on MQAR. It also discloses its baseline compromises and implementation issues. Four of five baselines are parameter-matched within 0.2% and the exception (Based) is disclosed, learning rates are set per architecture, and the main numbers in the text can be recomputed from the tables. The discussion in Sec. 3 of why ℓ2-normalized queries and keys combined with a 1/√dk scale collapse attention to a near-uniform average is a useful practical observation. The straightforwardness about limitations is a real strength.

Critical weaknesses

1. The central ablation is underspecified. The local-only and latent-only ablations (Table 5, Fig. 1) are never described: parameter count, learning rate and gate handling are all missing, and Appendix E lists no learning rate for them. If removing the latent path removes some per-layer parameters, part of the far-token gain could reflect lost capacity rather than the mechanism.

The near-token null does not fully rule this out. The AR definition (Sec. 4) has no frequency filter, so the near bucket includes common bigrams that can be predicted from memorized statistics and leave little room for either path to help. In addition, the three distance bins show that the effect differs on either side of 64, but they cannot distinguish a step at the window boundary from a gap that grows gradually with distance, so the claim that the division of labour “falls at the window boundary” is not yet established.

2. Some secondary claims go beyond the evidence.

  • Scaling (Sec. 5.3, Table 7): far-token perplexity worsens with width for both models (HALA from 11.35 to 13.07; local-only from 12.32 to 16.79). Each corpus is capped at 5×10⁷ tokens, while training uses 35,000 × 16 × 1,024 ≈ 0.57B tokens, so each corpus appears to be repeated about 11 times. The growing gap may therefore reflect overfitting rather than a growing role for the recurrence.
  • MQAR (Table 1, Sec. 5.5, Sec. 6): 58.3% of queries fall within the window, and HALA's 0.590 is attributed to its window. Yet the sliding-window baseline scores only 0.025. Sec. 5.7 reports an anomalous compiled kernel for this baseline, and it is unclear whether that affected the MQAR run. More broadly, Mamba-2 (0.000), the sliding-window baseline (0.025) and Based (0.051) all score below 1/16, the accuracy of returning a random one of the 16 values in the context, so none of them shows any recall. Dao & Gu (2024) report Mamba-2 performing well on MQAR at the same sequence length and vocabulary with 8× more key-value pairs. The "categorical" separation from pure recurrence claimed in the abstract may therefore reflect failed baseline training rather than an architectural limit.
  • Extrapolation (Sec. 5.4, Table 8): the local-only ablation, which has no recurrence, is equally stable (32k/1k ratios 0.970–0.976 vs. HALA's 0.965–0.969). Stability is therefore not specifically due to the recurrent state.
Questions for authors

Clarification requests:

  1. How were the local-only and latent-only ablations built (parameter count, learning rate, gate handling)? Were they parameter-matched or tuned? If the far-token gap persists against a matched, tuned local-only model, my main concern is resolved.

  2. Could you report the HALA vs. local-only gap in finer distance bins (e.g., 1–32, 33–64, 65–128, 129–256, >256), together with token counts per bin? This only requires evaluating existing checkpoints. A clear step at 64 would support the window-boundary claim.

  3. For the width ladder (Table 7): was the learning rate retuned per width, how many epochs did each run see, and do training and validation loss move in the same direction?

  4. On MQAR: what is the accuracy split by distance (≤64 vs. >64) for HALA and the sliding-window baseline? Why does the sliding-window baseline score 0.025, and did its MQAR run use the kernel path flagged in Sec. 5.7? Does Mamba-2 learn an easier configuration (fewer pairs, shorter sequences)?

  5. Your AR-token definition (Sec. 4) matches the "AR hits" of Arora et al. (2024a), except that they keep only bigrams seen at most 1,250 times in training, because common bigrams can be predicted from memorized statistics. They report that the gap to attention is concentrated on these rare bigrams. The paper cites that work for MQAR and the state-size bound, not for this natural-text analysis. The same work also shows that most AR hits lie within about 100 tokens of the earlier occurrence, and suggests that sliding-window attention may suffice to handle recall in real data. This anticipates your window-versus-recurrence rationale, although it is not an ablation. Please discuss this work and say whether restricting to rare bigrams changes the near/far result.

3 comments
AuthorsOpenReview ↗

Response to Reviewer DmqQ

Thank you for precise, checkable questions. Each one led to a change.

Q1. Ablation construction and capacity. The local-only ablation kept all 11.85M parameters allocated, but 29.1% of them were inactive (8.40M active). Its learning rate was inherited from HALA, which is now disclosed. To meet your condition, we trained a capacity-matched local-only model with 11 layers and 11.55M active parameters (−2.5%). We tuned both models on comparable grids (HALA 4e-4 to 1.6e-3, local-only 2e-4 to 1.6e-3). Both were best at 1.6e-3, the edge of their grid, which we disclose.

Against this model, within the window there is no detectable difference (+0.077 and +0.088 nats, CIs cross zero). Beyond it, HALA is better in every bin, by 0.068–0.087 nats, with CIs excluding zero (§5.1, Fig. 1).

Q2. Finer bins with counts. Done (Table 3). The token counts per bin are 224,959, 131,210, 156,089, 164,980, 140,396 and 65,543. The gap appears immediately past 64 and stays roughly constant in loss beyond it.

Q5. Rare bigrams (Arora et al. 2024a). Restricted to bigrams with at most 1,250 training occurrences, the advantage persists. Accuracy is significantly higher in every bin beyond 64, and loss is significantly lower in every bin beyond 128 (for example −0.165 nats in 129–256). Related Work now discusses Zoology's AR-hit analysis, including its suggestion that a window may suffice for most recall. Our controlled path ablation tests that idea directly.

Q3. Width ladder. The learning rate was not retuned per width, and each run sees about 11.5 passes over the 50M-token corpus. The widening gap may reflect overfitting. Those checkpoints no longer exist, so the ladder is now captioned as confounded rather than presented as evidence. At 11.85M parameters, validation loss reaches its minimum at 27.5k–30k steps and ends 0.003–0.007 nats above it, so overfitting is mild at that width.

Q4. MQAR. The paper misreported the configuration. The runs used sequence length 256, 20,000 training and 1,024 validation examples, and 16 key–value pairs. HALA's accuracy is 0.992 on queries within the window (59.5% of them) and 0.001 beyond, so its MQAR score equals window coverage. The sliding-window baseline ran without compilation (the MQAR script never calls torch.compile), so the compiled-kernel issue discussed in Appendix L.6 does not apply. It fails at every learning rate tried, which we report as unexplained. Mamba-2 also fails on the easier 4-pair setting: in a 16,000-step rerun its training loss reaches 0.0002 and training-slice accuracy 1.000, while validation accuracy stays below 0.001, so it memorises the fixed training set. We agree this points to a failed baseline under our protocol rather than an architectural limit, and we removed the "categorical" claim.

Extrapolation. Agreed: local-only is at least as stable (0.970–0.976 vs 0.964–0.969). Corrected.

Thank you for the replies. The new baseline addresses the main point I raised, and the finer bins and rare-bigram analysis answer my other questions. I also appreciate the corrections and the transparency about what could not be recovered. Some points remain open: the learning rates at the edge of the grid, the matched comparison only on PG-19, the wide within-window confidence intervals, and the unexplained sliding-window baseline failure on MQAR. I will take these responses, the revised paper and the other reviews into consideration for my final assessment.

AuthorsOpenReview ↗

Response to Reviewer Dmq

Thank you for the follow-up and for engaging closely with the revision. We agree that these four points remain open: both models' best learning rate lies at the edge of the tested grid, the matched comparison covers PG-19 only, the within-window intervals are wide at three seeds, and the sliding-window baseline's MQAR failure is unexplained. The revised paper states each of these as a limitation; we were not able to resolve them within the discussion period.

Reviewer_ULakOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HALA, a block that runs sliding-window attention and a low-rank gated recurrence side by side, mixes them with a learned per-head gate, and has no feed-forward sublayer. The authors present the measurement as their main contribution. They pick out tokens whose bigram already appeared earlier in the context and group them by how far back that earlier occurrence lies. HALA is then compared with versions of itself that have one path removed. The headline finding is that the recurrence helps only on tokens whose earlier occurrence lies beyond the attention window, and that this benefit grows with model size. On associative recall the paper reports that HALA answers more than half of the queries correctly while Mamba-2 stays at chance. It also reports that HALA trails Transformer, Jamba and Mamba-2 on overall perplexity.

Critical strengths
  • I like the diagnostic. Grouping recall tokens by distance and comparing a model with its own path ablations is simple, and it exposes a pattern that overall perplexity hides. The far-token gain also shows up in every corpus, which makes it more convincing to me.
  • I also appreciate that the paper reports its unfavorable results plainly, including the overall perplexity gap and the gate's lack of distance sensitivity on associative recall.
Critical weaknesses
  • My main concern is that the current evidence can't yet establish that the division of labour falls at the window boundary.
    • Near and far are split exactly at the window edge, and apart from a nested bucket for very distant tokens I couldn't find a finer breakdown for natural text, so the data can't show whether the gap begins right at the edge.
    • The window limit applies per layer. With eight stacked window layers, the local-only model can pass information well past a single window, so it can still reach many tokens beyond the window edge.
    • At the largest model size the full model also clearly beats the local-only model on near tokens, so the claim that the recurrence adds nothing inside the window seems hard to sustain across the scaling ladder.
    • The abstract calls the near-token difference statistically indistinguishable, but I didn't see a statistical test anywhere in the paper. The Figure 4 caption also refers to per-rung test statistics in Table 7, which the table doesn't include.
  • I also think the associative-recall explanation needs another look.
    • The paper says a fixed-size state cannot in principle retrieve a token seen once. I don't think that holds for this task, which asks for a small, fixed number of key-value pairs, since a fixed-size state can store that many exactly. The related-work section also says that recall scales with state size. So I don't see how the Mamba-2 result can support the impossibility explanation.
    • Appendix A reports the routing ablation on this task only as an ordering, and in that ordering the local-only model beats the full model. I'd really like to see the numbers here.
Questions for authors
  • Would it be possible to report the near/far comparison on natural text with finer distance bins, like the ones Figure 7 uses for the gate on associative recall, and ideally with other window sizes? Does the gap start at the window edge?
  • What are the numbers for the associative-recall routing ablations in Appendix A?
  • What window does the sliding-window baseline use on associative recall, and why does it score so low when Section 6 says most queries fall inside the window?
  • Could you show Mamba-2's training curves across the learning-rate sweep?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ULak

Thank you. Your questions led directly to the main new experiment.

Does the gap start at the window edge? Yes. Finer bins on books, 3 seeds paired: HALA minus a capacity-matched, learning-rate-tuned local-only model (loss in nats, 95% CI; §5.1, Fig. 1):

bindiff95% CI
1–32+0.077[−0.069, +0.222]
33–64+0.088[−0.065, +0.242]
65–128−0.073[−0.121, −0.025]
129–256−0.087[−0.129, −0.045]
257–512−0.084[−0.116, −0.053]
513–1022−0.068[−0.089, −0.048]

The change falls between the 33–64 and 65–128 bins rather than growing gradually. Inside the window the point estimates favour local-only, but the CIs cross zero.

Stacked windows. Agreed: the reach is layers × window, and the paper now says so. Two observations suggest the 64-token boundary still matters. First, the gap is just as large in 513–1022, which is beyond the 8-layer local-only reach of about 512. Second, it persists against the 11-layer baseline, whose reach is about 704. We did not test other window sizes; this is listed as a limitation.

Near tokens at 74M. You are right. The within-window null holds at 11.9M only, and we now state that scope wherever the claim appears. The width ladder is also confounded (one learning rate, about 11.5 passes over the data), and its caption now says so.

Statistics. The unsupported phrase is replaced by paired per-domain 95% CIs. The caption that cited missing t-statistics is fixed.

Fixed-size state. We agree and have retracted the claim.

MQAR routing-ablation numbers. At 16k steps (3 seeds): HALA 0.590, local-only 0.574, static gate 0.562, hard gate 0.028, latent-only 0.0002. Local-only and static gate were still improving. The earlier ordering came from 32k-step runs whose result files were lost. Their numbers survive only as recovered log output (local-only 0.76, static 0.73, HALA 0.595), and the paper now labels them that way.

Sliding-window baseline on MQAR. It uses a 64-token window. Its low score is not a coverage effect. HALA's local-only ablation, with the same window, reaches 0.574, while the baseline fails at every learning rate we tried (1e-5 to 3e-3). We verified its attention kernel separately on the same GPU and report this as an unexplained baseline failure. The new branch analysis shows HALA's own 0.590 comes entirely from the window: 0.992 on in-window queries (59.5% of queries) and 0.001 beyond, unchanged when the recurrence is removed.

Mamba-2 curves (Appendix J, Fig. 4). In two 16,000-step reruns (16 and 4 key–value pairs), training loss falls to 0.0012 and 0.0002 and accuracy on a fixed training slice reaches 1.000, while validation accuracy stays below 0.001. Under our protocol (a fixed set of 20,000 training sequences, no curriculum), Mamba-2 memorises the training set instead of learning retrieval; HALA and the Transformer generalise under the same protocol. Dao & Gu (2024) report Mamba-2 solving harder settings under their own training protocol, so we treat this as a failure of our protocol for this baseline and make no claim about Mamba-2.

Thank you for the response and the substantial effort put into the revision. The revised manuscript makes substantial changes to the central claims and supporting evidence compared with the original submission. In my view, this constitutes a major revision that needs a fresh, full review. I recommend resubmitting the revised work in a future submission cycle so that it can be fully evaluated. I will therefore maintain my original score of 2.

AuthorsOpenReview ↗

Response to Reviewer ULak

Thank you for reading the revision and for the constructive review. For the area chair's reference: the paper's main claim is unchanged from the original submission (the recurrent path helps only beyond the attention window). The new experiments test that claim more strictly, as the reviews requested, and the other changes are retractions of secondary claims rather than new ones.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revision and new experiments

We thank the reviewers. The revised PDF marks changes in blue, and Appendix A lists all of them. We reorganized the paper around one question, where a hybrid's recurrent path helps, and added new experiments (PG-19, 3 seeds unless noted):

  1. Capacity-matched, tuned baseline. A local-only model with 11 layers (11.55M active parameters vs HALA's 11.85M; stacked reach about 704 tokens vs 512). Both models' learning rates are tuned on comparable grids. Within the 64-token window there is no detectable difference. Beyond it, HALA is better by 0.07–0.09 nats in every distance bin, with 95% CIs excluding zero (§5.1).
  2. Finer distance bins with token counts (Table 3), plus a rare-bigram filter (≤1,250 training occurrences, following Arora et al. 2024a) (§5.1–5.2).
  3. Branch-level MQAR analysis. HALA answers 99.2% of queries whose key is inside the window and 0.1% beyond. Removing the recurrence at inference changes neither number (§5.3).
  4. Design ablations at equal parameters: restoring an FFN, gate variants, and inference-time branch knockouts (§5.4).
  5. Reproduction. Retrained models match the original numbers within seed noise.

Corrections and retractions

  • The MQAR configuration was misreported. The runs used sequence length 256, 20,000 training and 1,024 validation examples.
  • We retract the "categorical" separation from pure recurrence and the claim that a fixed-size state cannot in principle retrieve a token seen once. New training curves show that Mamba-2 memorises our fixed training set (training accuracy 1.000, validation at chance). This is a failure of our protocol for that baseline, and we make no claim about Mamba-2's recall.
  • Length-extrapolation stability is not due to the recurrence. The local-only ablation is equally stable.
  • "Statistically indistinguishable" is replaced by paired 95% CIs. We claim no equivalence.
  • The within-window null holds at 11.9M parameters. At 74M, the full model also helps on near tokens.
  • We now disclose that the ablations and the baselines were not equally tuned, and we claim no ranking between architectures.
  • One result goes against our design: at equal parameters, restoring an FFN improves perplexity by about 2.7. It is in the main text.

The code documentation was rewritten. The README now maps every figure and table to its script and command (updated supplementary material).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.