acceptodds
Under review as a conference paper at ICLR 2027

RAVEL: Collision-Verified State Augmentation for Long-Context Recurrent Language Models

Abstract

Fixed-state recurrent language models enable linear-time processing and constant-state decoding, but repeatedly compressing the prefix into a small state can cause interference between independent associations. We introduce RAVEL, a recurrent architecture with bounded collision-verified state augmentation. RAVEL maintains six fixed direct-address banks alongside the recurrent core, where each slot stores an exact token fingerprint, a support count, and a differentiable successor-value prototype. Reads precede writes, fingerprint mismatches abstain exactly, and a learned residual gate with a delayed horizon protects short-context behavior. The resulting augmentation requires no attention, nearest-neighbor search, relevance ranking, or unbounded token cache. We establish causal equivalence between teacher-forced and autoregressive execution, exact collision abstention, bounded state values and logit corrections, convergence of the running prototype, and an exponential-in-bank-count bound on surviving collisions. Across matched 6.1M, 11.7M, and 23.7M parameter scales, three-seed TinyStories experiments show that RAVEL improves over matched Mamba-3 at context lengths 128–4096 while remaining statistically indistinguishable at length 32, and improves over RWKV-7 at every measured length. A post-hoc TinyShakespeare study reproduces the long-context advantage. Frozen ablations show that one verified bank provides most of the gain, while additional banks offer diminishing collision redundancy. Together, these results demonstrate that RAVEL delivers a consistent and interpretable long-context quality advantage while preserving the bounded-state structure of recurrent language models.

Primary area:
generative models
Keywords:
recurrent language models, long-context modeling, state augmentation, collision verification, linear attention, direct-address lookup, linear-time sequence modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GWmWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

RAVEL adds an explicit memory structure that contains multiple “banks”, each one of which maps the current token’s ID to a dynamically updated representation of previous tokens’ successors. Selected vectors are then aggregated, projected, and gated before added to the base model backbone’s logits prior to decode. Experiments up to 24M parameter language models demonstrate that adding RAVEL reduces the NTP loss over baseline architectures. Additional ablations show that increasing the number of banks provides diminishing returns in modeling performance, and while additional scaling experiments investigate performance across three model sizes, results at scale remain limited.

Critical strengths

The authors provide comparisons across three model sizes and three random seeds, which shows that RAVEL improves upon the baseline backbone in their experimental setting.

Critical weaknesses

The scale at which the RAVEL method is tested is in general too small to give signal (model size caps at ~20M and trained with only 2M tokens). While there are gains at the 12M model scale, Figure 3 seems to suggest that the overall gains of RAVEL seem to decrease as the model parameters increase. In addition, the training horizon of the models is extremely small, which may bias performance towards the RAVEL module as the baselines are vastly undertrained. Furthermore, there is a lack of experiments that test RAVEL’s impact on MQAR or other synthetic tasks in which a model of such scale might perform better at and give better signal.

There also is a lack of baselines and ablations that isolate the effect of RAVEL. For instance, RAVEL at its core is a learned gated bigram; one suitable comparison would be to DeekSeek’s Engram architecture or a simple online bigram cache. Additional ablations on the hash collision and memory correct components would also help disentangle the multiple components of RAVEL. It also seems like a single bank is able to recover most of the overall performance, raising questions about the multi-bank design.

Questions for authors

What are the specifics of the experimental protocol, e.g., peak learning rate, hyperparameters used, etc.

How does RAVEL scale with additional compute, either with model scale or more training data?

How does RAVEL perform on synthetic tasks, such as MQAR, select copying, fuzzy recall, etc. (please see Mechanistic Design and Scaling of Hybrid Architectures), if scaling up language model training is infeasible.

Reviewer_AiqtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at the memory problem in recurrent models like Mamba. Because these models compress past information into a fixed-size state, some specific token associations may get lost. The authors propose RAVEL, which adds six fixed-size hash memory banks alongside Mamba-3. The current token ID is used to look up a slot, and a fingerprint check makes sure the slot belongs to the same token, avoiding incorrect reads caused by hash collisions. Each bank also keeps updating a representation of the token's successors. The retrieved memory is then used through a gate to help with prediction. The paper also analyzes causality, bounded state size, and hash collision probabilities, and runs experiments on TinyStories. RAVEL gets lower NLL than Mamba-3 at L128–L4096, with similar results on TinyShakespeare. Still, the experiments are mainly language-modeling tests on small models, and they do not show inference speedups in real deployments.

Critical strengths

1. The method is simple and the idea is clear.

The authors add a fixed-size memory table alongside Mamba-3 and use fingerprint checks to avoid reading information from the wrong token. It does not need to store the full history, and the state size stays fixed. The basic idea is quite straightforward.

2. The experiments are carefully done and show some real improvements.

The authors test three model sizes and three random seeds, and provide confidence intervals and a bank-count ablation. RAVEL does reduce NLL at the tested medium and long context lengths, which is worth acknowledging.

Critical weaknesses

1. There is no fair comparison with other auxiliary memory methods.

The paper mainly compares against models like Mamba-3 and RWKV-7, but works such as HOLA and Engram have also explored adding extra memory to language models. Of course, these papers use different backbones, so directly comparing the NLL of the full models would not be very fair. I think a better comparison would be to add some simple online memory methods to the same Mamba-3 backbone and see where RAVEL actually does better. Also, Table 3 shows that a single bank already gives most of the improvement, while six banks bring only a small additional gain. So I think the authors should explain more clearly why this design is worth using instead of a simpler memory approach.

2. The experiments do not yet show whether the method works on complex long-context tasks.

The paper mainly tests NLL on TinyStories. The largest model has only 23.7M parameters, and the training data budget is quite small. TinyShakespeare also shows improvements, but it is still a development-set experiment. RAVEL is supposed to address the loss of historical associations, but the paper does not directly test more complex long-range association tasks, such as Key–Value Retrieval. Also, since memory is looked up mainly by token ID, successor information from different contexts for the same token may get mixed together. I find it hard to tell from the current NLL results how much this kind of memory would help on more complex tasks.

Questions for authors

Q1. Why use six banks by default?

Table 3 shows an NLL of 2.4489 with one bank and 2.4474 with six banks. The difference is quite small. But more banks also mean more state and more computation. So why did the authors settle on six? Have they considered whether one or two banks might already be enough, and how the performance and cost compare?

Q2. Does the hash collision probability proof need a stronger independence assumption?

Theorem 3 in Appendix B.5 assumes pairwise independence, but the proof then directly multiplies the probabilities of multiple tokens not colliding. As I understand it, pairwise independence alone does not guarantee joint independence across several events. Could the authors explain whether the actual hash mappings satisfy a stronger independence condition? If they are only pairwise independent, this part of the derivation may need more explanation.

Reviewer_dwvxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RAVEL, a bounded state-augmentation mechanism for fixed-state recurrent language models (evaluated on a Mamba-3 backbone) designed to mitigate associative interference over long contexts. Alongside the recurrent core, RAVEL maintains J=6 direct-address hash banks with B=1024 slots each, deterministically addresses one slot per bank to store an exact integer token fingerprint, a capped support count, and a running average of the value embeddings of immediate successor tokens yt=xt+1. Reads precede writes, fingerprint mismatches abstain exactly (returning zero), and valid reads are averaged, projected, gated by the hidden state, and scaled by a fixed warmup horizon t (zero through position 64) before being added to the base logits.

The authors establish theoretical guarantees for causality, exact abstention, convex-hull boundedness, prototype convergence, and multi-bank collision survival, and evaluate RAVEL on TinyStories (V=2048) across 6.1M, 11.7M, and 23.7M parameter scales as well as post-hoc on TinyShakespeare, demonstrating consistent negative log-likelihood (NLL) improvements over matched Mamba-3 and RWKV-7 baselines at context lengths 128–4096.

Critical strengths

The paper defines a clean, auditable bounded memory primitive with strict read-before-write ordering, exact integer fingerprint verification that turns hash collisions into deterministic abstentions rather than corrupted reads, and proven causal equivalence between teacher-forced training and autoregressive decoding, validated against a standalone Triton operator.

The experimental protocol demonstrates good scientific discipline, featuring preregistered evaluation contracts, hierarchical paired bootstrap intervals across seeds and aligned token windows, frozen bank-count ablations (J∈{0,1,2,4,6}J\in \{0,1,2,4,6\}) that isolate the memory pathway without retraining, and candid reporting of post-hoc OOD boundaries and systems limitations.

Critical weaknesses

The fixed-parameter comparisons ( 8.44M neural parameters) between the wider models (N in {1, 2, 8}) and N=256 seems to be confounded because the wider controls include 2 d_E d_t bottleneck projections and continuously update expert-local heads via a first-order delta rule at every step, whereas N=256 uses no projections and freezes a shared head after the seed run;

If I understand correctly, the theoretical derivations in Section 5 and Appendix F assume dense Rademacher probes, whereas the implemented algorithm uses un-rescaled sparse probes (q_j = 1/2 on ordinary weights and q_j = 1 on LayerNorm gains) that introduce an O(1) coordinate-wise scaling bias (q_e g_e), double the relative strength of coupled weight decay on ordinary weights, and rely on first-order BPTT proxies for key architectural ablations.

The reported 8.44M model sizes seems to omit up to 6.4M fitted TF-IDF/SVD router coefficients needed for N in {8, 256}, the 150 GPU-hour monolithic SPSA baseline incurs 4-GPU DDPP synchronization overhead despite fitting on 1 GPU, and the headline 9.19\times inference speedup of N=256 over N=8 evaluates teacher-forced scoring across different batch sizes (B=4,096 vs. B=1,024) where N=8 is actually more than twice as slow as the N=1 monolith.

The study is restricted to small 8.44M-parameter byte-level models where heavy sharding (N=256) requires ~700 passes over each data shard, and as shown in Appendix Table 5, SOMA still trails exact first-order Backpropagation Through Time by a wide margin at matched compute budgets (1.83–2.12 vs. 1.01–1.18 validation nats/byte), a limitation that deserves greater visibility in the main text.

Questions for authors

address the weekness points above please

Comments on the paper

1 comment

Thank you for your response.

My concern about lack of baselines and ablations remains unanswered. How well does RAVEL compare against a simple learned gated bigram or an online bigram?

While I appreciate the new experiments, the protocol could be improved. For instance, the largest 55M model is very undertrained compared to the smallest model, making extrapolation hard.

What is the performance of a baseline Transformer on the MQAR tasks? What was the protocol used for training and evaluating on MQAR?

Also could you clarify the protocol? Your response says the extended experiments use 8k cosine steps (~32M tokens) while the results report after 8M and 2M tokens.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.