acceptodds
Under review as a conference paper at ICLR 2027

Towards Long-Term Memory with Small Language Models

Abstract

As personal AI assistants proliferate, they are expected to become capable of complex, long-horizon interactions while preserving the privacy of increasingly personal information. A central challenge in achieving this form of personalized intelligence is long-term memory. An assistant cannot genuinely understand a person from isolated sessions, it needs a continuous thread of experience that accumulates over time, where someone has been, whom they have spoken to, what they have been working on, and the context connecting these experiences. This requirement for persistent memory is tightly coupled to privacy. Rather than transmitting an ever growing record of personal experience to a remote server, data center, or other external system, memory can remain private by being stored and processed locally on the device where the assistant operates. Such a design, however, places a fundamental constraint on the underlying model, mainly the assistant's memory must be small enough to run on a phone, a pair of glasses, or a robot while retaining the capacity to maintain and recall over a persistent stream of experience. Yet every long-term memory result we are aware of is reported at model scales an order of magnitude beyond that budget, and whether a small model can carry such an ability at all is, to the best of our knowledge, underexplored. In this paper we propose an approach to long-term memory built purely on small language models, in which memory lives in intrinsic recurrent state rather than in a growing attention window. We show that small models combined with linear attention and the right training strategy offer strong recall, and that because a bounded state has no key-value cache to grow, such a model can ingest conversations far longer than a large transformer can admit at all. We evaluate against open-source counterparts such as Gemma-3-27B-it, Qwen2.5-32B-Instruct and Qwen3-30B-A3B, on the BEAM long-term memory benchmark and find our approach competitive with models an order of magnitude larger.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Information Retrieval, Language Models, Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_41GNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that when a 2.9B model is trained from scratch using Gated Linear Attention followed by SFT and RL tailored for the BEAM benchmark, it functions as a sparse recovery system within an associative memory matrix. Mathematically, unrolling this multi-timescale layer proves its outputs evaluate a "memory polynomial" whose coefficients represent historical token relevance. Experimental results show that "shape beats size": allocating more heads across diverse timescales outperforms a state with twice the total capacity but fewer heads by over 30% at 500K and 1M token contexts. After just 160 steps of RL with a verifiable recall reward, the model dramatically enhances its fact-locating capabilities.

Critical strengths
  1. This paper shows that if we train a small model with gated linear attention, then train it with RL typically for BEAM task, it can perform well on BEAM benchmark.

  2. This paper conducts extensive experiments and presents a solid ablation study, offering valuable insights to the research community.

Critical weaknesses
  1. Since the model's RL phase is tailored specifically for the BEAM benchmark, its high score there is expected. Evaluating the model on broader, general benchmarks is necessary to assess its true versatility.
Questions for authors
  1. As shown in weaknesses 1. Please report the general benchmarks of 2.9B model.

  2. Please also list the data used in all stages pretain/sft/rl.

  3. The background section in the abstract is overly detailed and can be streamlined into two or three sentences to keep the focus on the paper's core contributions.

Other comments:

Line 757: missing refer ids: of the problem, which Proposition ?? and equation ??

2 comments
AuthorsOpenReview ↗

Adds new general benchmarks

We thank the reviewer for the careful summary and for describing the ablations as solid and of value to the community. The paper, also brings new perspectives on the time dimension(context length) for memory recall, carrying interesting parallels with Laplace transform, memory models in neuroscience and the experiments offer valuable insights. Regarding the questions, we address each point below.

W1 / Q1. RL is not tailored to BEAM; general benchmarks. No post-training stage uses BEAM data. SFT and RL train only on the train portions of LoCoMo and HotpotQA. BEAM is a transfer evaluation, not the training task.

We evaluated our final 2.92.9B model (after RL) and the baselines on general benchmarks (accuracy, %; MMLU scored on answer choice):

ModelParamsPretraining tokensMMLUGSM8K-PlatinumMATH
Ours, Pretrain2.9B0.136T29.713.65.8
Ours, RL2.9B0.136T29.412.16.6
Gemma-3-27B-it27B14T76.995.081.6
Qwen2.5-32B-Instruct32B18T84.297.477.8
Qwen3-30B-A3B30B (3B active)36T81.296.181.0

Memory training does not degrade general ability: from the pretrained base to the RL model, MMLU moves 29.7→29.429.7 \to 29.4, GSM8K 13.6→12.113.6 \to 12.1 and MATH 5.8→6.65.8 \to 6.6. The gap to the baselines reflects pretraining scale, not the memory recipe: they see 100100 to 265×265\times more tokens with about 10×10\times more parameters, roughly three orders of magnitude more pretraining compute. We report these numbers to show that recall is gained without loss of general ability, not to claim parity. We also add a note that training the model till 36T for our settings is expensive and not very relevant.

Q2. Data used in each stage. We will add this table to the appendix:

StageData
PretrainingInternal corpus: 34.7% code, 29.9% general knowledge, 18.9% multilingual, 16.5% STEM; ≈0.136\approx 0.136T tokens.
SFTAnswer-only recall on the 7 LoCoMo train conversations (5,844 multi-question documents) and HotpotQA train (90,447), mixed 40/60;
RL37,142 prompts from the same LoCoMo and HotpotQA train portions (about 46/54); binary containment reward; 160 steps of GRPO.

Neither SFT nor RL uses any BEAM data.

Q3. Abstract. Agreed, we will reduce the background to two or three sentences.

Other comments. Thank you for catching the missing references at L757. Both now resolve in the revision.

AuthorsOpenReview ↗

Follow-up on Author Response

Dear Reviewer 41GN,

Thank you again for your time and valuable comments. As the discussion period nears its end, we would like to follow up on our author response. We would appreciate any further feedback and would be happy to clarify any remaining concerns.

Best regards, The authors

Reviewer_LuzROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper evaluates a small model can handle long-term memory on a phone, since the larger models need a memory cache that keeps growing as the conversation gets longer. They use Gated DeltaNet, a linear-attention model whose memory stays a fixed size and update the the heads speeds spread out on a log scale. On the BEAM benchmark, their 2.9B model beats Gemma-3-27B and the two Qwen models at pulling specific facts out of really long histories, but it still loses on average, especially on summarizing and following preferences.

Critical strengths
  • The motivation of the paper is practical, assistants are moving onto phones and wearables, hence the paper's premise is fair.

  • The paper has given great details for reproducing the results.

Critical weaknesses
  • Line 052 says that "Long-term conversational memory has been evaluated primarily at 27B parameters or above", which is not the correct claim. There are plenty of papers that evaluate for smaller models including [1], [2].

  • The paper makes claims without actually much bibliography support, for example the above claim. There are broken references L418, L757. Not sure if the reference format is correct, I haven't checked if they are hallucinated but about half of the references are in the format: first author et. al. Paper name

  • Baseline comparisons are unfair, they were cut off at their native windows (Qwen2.5 sees about 29K tokens), while this model reads the whole corpus.

  • The paper is confusing to read, Results (§4) come before the theory that motivates them (§5), so the argument reads backward.

  • Ethics mentions LoCoMo, LongMemEval and HotpotQA, there are no results on these.

  • "approach competitive with models an order of magnitude larger" this was not the case in Table 1 for most of the results.

  • Results on one benchmark.

Section E.1 seems to be AI generated but not sure if the AI paragraph actually alludes to this level of AI contribution:

a larger head dimension [cite: Sun et al., headdimension scaling], usually paired with a second observation: associations can only be added, never erased, so errors accumulate — which gated variants Yang et al. (2024b); Gu & Dao (2023) and the delta rule ? mitigate. We agree with all of it. Our claim is only that it is silent on one axis, for a structural reason and with a different logical status.

[1] A-MEM: Agentic Memory for LLM Agents

[2] PERSONAMEM-V2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory

Questions for authors
  • See weakness

  • How do you run 10M tokens through full attention?

  • Appendix B describes 8 heads × 128 wide (about 131K per layer), but the winning variant is th64, which version is in the main result?

3 comments

We thank the reviewer for the detailed review, and for recognising the practical motivation and the reproducibility of the work. We first place the contribution in context, then respond point by point.

The gap this paper addresses. Prior work on long-term memory with small models builds an agentic system around the model: memories are written to an external store, and an agent with tools retrieves them into a short context. The model itself never performs the long-term recall. We ask whether a small model can do it alone: read the entire history, up to 1010M tokens, in a single pass and answer from what it has retained, with no agent, tools or external retrieval. This is the setting that matters for private, on-device assistants, and it has not been measured at 100100K to 1010M tokens. Our analysis explains what such a model needs: a gated state has a single exponential memory kernel and so serves one depth well (Proposition 1); heads on a geometric grid of timescales cover many depths at once; and their readings are samples of the Laplace transform of the relevance profile, which parallels the computations in scale-invariant models of time cells in neuroscience.

W1. Prior evaluations of small models. We thank the reviewer for A-MEM and PersonaMem-v2, and will cite and discuss both. They address a different problem. In both, long-term recall is performed by a system around the model rather than by the model: A-MEM writes memories to an external store and retrieves the top-kk by embedding similarity, and PersonaMem-v2 has a agent rewrite a 2,0482{,}048-token text memory as it reads the history in 5,0005{,}000-token chunks. Neither operates at long context. Our model performs the recall itself, reading histories of 100100K to 1010M tokens with no agent, tools or external retrieval. We will revise L052 to state this distinction from agentic systems explicitly.

W2. References and formatting. These will be fixed. They are confined to an unresolved key in one citation list in Related Work (L418) and in the appendix (L757).

W3. Baselines at native windows. Evaluating each model at the context window it supports is standard practice, and BEAM's own baselines are reported the same way. It is also the point of the comparison. Appendix G tests the reviewer's concern directly by extending both Qwen windows with YaRN: this raises both Qwens at every length, and the ordering of models is unchanged. We exclude abstention from the averages because truncation inflates it (a model that sees less of the corpus is more often correct to say the answer is absent); Appendix F reports it in full.

W4. Order of results and theory. The theory that motivates the experiments comes first: §2 derives the single-state dilemma and §3 the geometric grid that the results then test. §5 goes further, recasting the grid as sparse recovery from a memory polynomial and as sampling a Laplace transform, which connects the construction to scale-invariant memory in neuroscience. We placed it after the results because it interprets them rather than being needed to read them, and because the page limit moved the detailed derivations to the appendix. We will add a short roadmap to the introduction so the structure is clear on first reading.

W5. Datasets in the Ethics statement. HotpotQA and LoCoMo are listed because their training splits are our post-training data (Appendix C); LongMemEval was not used and doesnt cover 10M range, we will correct the statement accordingly.

W6. "Competitive with models an order of magnitude larger." We stand by this claim, and Table 1 supports it on four counts.

  • Locating specific facts. Our 2.9B model leads all three 27–32B baselines on information extraction at every corpus length (0.324, 0.189, 0.197 and 0.100, against best baselines of 0.195, 0.120, 0.108 and 0.050). It also leads on event ordering at 500K, 1M and 10M, and is within 0.015 of the best at 100K.
  • On the nine-ability average it exceeds Qwen3-30B-A3B at 100K (0.227 against 0.190), is within 0.003 of it at 500K and 1M, and exceeds Gemma-3-27B at 10M (0.102 against 0.092). These baselines carry roughly ten times our parameters, and total parameters are what determine the memory footprint on a device.
  • Its answers have the highest token-level precision of any model at every corpus length (Appendix H).
  • Training scale. The baselines are not only larger but trained on far more data:
ModelParamsPretraining tokens
Ours2.9B0.136T
Gemma-3-27B-it27B14T
Qwen2.5-32B-Instruct32B18T
Qwen3-30B-A3B30B (3B active)36T

Our model matches or exceeds them on the abilities above with 100100 to 265×265\times fewer pretraining tokens. Against Qwen3-30B-A3B, which has a similar number of active parameters, it leads at 100100K and is within 0.0030.003 at 500500K and 11M with 265×265\times less data.

W7. One benchmark. BEAM is the only long-term conversational memory benchmark that extends to 10M tokens, which makes it the benchmark that fits the question this paper asks. LoCoMo conversations average about 9K tokens and fit inside every baseline's window, so they cannot separate a model that reads the full history from one with a bounded window; LongMemEval stops well short of 10M. Evaluating across 100K, 500K, 1M and 10M gives four regimes within one consistent protocol.

W8. Appendix E.1. The bracketed text was an unresolved citation placeholder from drafting, discussed under W2, and it will be fixed. LLM assistance was used to refine text, as our AI-use statement says.

Q1. How do you run 10M tokens through full attention? Only 5 of the model's 20 layers use attention. In every block of four layers, three are Gated DeltaNet. This is a standard hybrid design for linear-attention models and empirically performs better than pure gdn: (https://arxiv.org/pdf/2507.06457, https://arxiv.org/pdf/2412.06464).

Running attention over 10M tokens.

  • Context length. Our attention layers use no positional encoding (NoPE): their scores depend only on content, and token order comes from the causal mask and the GDN layers beneath them. With no position-dependent rotation there is no trained position range to exceed, so the architecture extends to 10M tokens as it is. Models with rotary position embeddings, such as Qwen, must instead rescale their rotations (YaRN) to run beyond their training window. We will add this detail to Appendix B.
  • Compute. We feed the corpus in fixed-size chunks (chunked prefill)

The theory's prediction is tested, and confirmed, inside the hybrid. With data, compute and the five attention layers held fixed, th64 beats th32/d256 by 30%30\% and 31%31\% at 500500K and 11M while holding half the state, at the same number of parameters. Changing only how the recurrent state is organised moves recall this much, so memory is sensitive to the GDN design the theory describes; whether a model also carries attention is a separate and standard design choice, and the effect points in the direction the theory predicts. What is new is that recall depends not only on the width of the state but on how its timescales cover the length of the context, a perspective on building long-term memory networks that applies with or without attention.

Q2. Which model is in the main result? Table 1 reports the 8-head model described in Appendix B (2.87B, h = 8, d_k = 128), post-trained with SFT and RL. th64 appears in the head-shape ablation (§4, Figure 1), which compares pretrained checkpoints that differ only in head count and head width; post-training every arm is expensive. We will name the model in every table and figure caption for better clarity.

We thank the reviewer again, and hope this response clarifies the contribution and the design choices.

AuthorsOpenReview ↗

Follow-up on Author Response

Dear Reviewer LuzR,

Thank you again for your time and valuable comments. As the discussion period nears its end, we would like to follow up on our author response. We would appreciate any further feedback and would be happy to clarify any remaining concerns.

Best regards, The authors

Reviewer_qYkBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper targets on-device long-term memory with small models. It proposes "temporal heads": geometrically spaced decay rates across Gated DeltaNet heads, motivated by showing that a single exponential kernel can serve only one depth, and it frames multi-head readout as sparse recovery. With data and compute held fixed, a 64-head 2.9B arm beats an arm holding twice the state by about 30% on BEAM at 500K and 1M tokens. RL post-training improves fact-locating abilities. Against 27–32B transformers, the model wins information extraction but trails Qwen2.5-32B on average at every length.

Critical strengths
  1. A well-motivated and under-measured setting: memory under on-device and privacy constraints, evaluated up to 10M tokens.

  2. A clean single-state analysis (Proposition 1, Example 2).

  3. A useful controlled finding: at long context, spending state on more heads beats spending it on wider heads.

Critical weaknesses
  1. Confounded attribution. Five of the 20 layers are full attention, so recall cannot be credited to the recurrent state, and the "bounded state" claim does not hold for the evaluated model.

  2. Untested hypothesis. The arms vary head count, head width, and total state together. There is no control with shared or non-geometric retentions, and by the paper's own Eq. 8 a denser grid over the same 16× range should add little.

  3. Theory–model mismatch. The analysis assumes constant retention and no delta rule, while the model uses input-dependent, per-channel gating. Realised timescales are never measured.

  4. Weak comparison. Qwen2.5-32B wins on average while seeing under 1% of the corpus at 10M tokens. Abstention, the model's worst ability, is excluded. The baselines lack same-scale models and retrieval, and there are no efficiency measurements.

Questions for authors

Same as weakness

4 comments

We thank the reviewer for a precise review, and for describing the setting as well-motivated and acknowledging the value from single-state analysis. The insights are novel, significant and offer new perspectives on long term memory.

On the questions, we respond point by point.

W1. Attention layers and attribution. We agree that the evaluated model is a 3:1 hybrid: 5 of its 20 layers are softmax attention (no positional encoding), so the bounded-state property belongs to the 15 GDN layers rather than to the whole model. We will state this explicitly.

On attribution, the head-shape ablation isolates the recurrent state. The two arms below have the same parameter count, attention layers, data, steps and optimiser, and differ only in how the GDN state is split into heads (BEAM judged score, nine-ability average, final checkpoints):

ArmParamsState / layer100K500K1M10M
th32/d2564.05B2.10M0.2030.1540.1440.094
th64/d1284.05B1.05M0.1940.1940.1890.109

At the same parameter count and with identical attention layers, changing only how the GDN state is organised moves BEAM by 26%26\% at 500500K, 32%32\% at 11M and 15%15\% at 1010M, and th64/d128 does so while holding half the state; at 100100K, where the target is close, the arm with most state leads instead. The gain is broad rather than driven by one ability: th64/d128 is ahead on 77 of 99 abilities at 500500K and 88 of 99 at 11M (th32/d256 / th64/d128):

Ability500K1M
Information extraction0.096 / 0.1280.118 / 0.216
Multi-session reasoning0.142 / 0.1270.098 / 0.116
Knowledge update0.104 / 0.1480.014 / 0.043
Temporal reasoning0.021 / 0.0140.000 / 0.012
Event ordering0.216 / 0.2270.235 / 0.213
Contradiction resolution0.084 / 0.0980.041 / 0.064
Instruction following0.246 / 0.3130.317 / 0.433
Preference following0.368 / 0.4920.390 / 0.456
Summarization0.111 / 0.1960.078 / 0.152

If recall were carried by the attention layers, reorganising the GDN heads at fixed parameters could not produce gains of this size; the results show that long-context recall is sensitive to the GDN configuration. The attention cache is also small by construction: five layers with five key-value heads of width 64 store 6.46.4 KB per token, against 256256 KiB for Qwen2.5-32B, 9696 KiB for Qwen3-30B-A3B and about 8080 KiB for Gemma-3-27B's global layers.

W2. Head count, width and state. : th64/d128 and th32/d256 share the same retention range, A∈[1,16384]A\in[1,16384] (1414 octaves; the paper's 8-head baseline, th8/d128, uses the default 16×16\times), the same parameters and the same value width, and th64/d128 holds half the state, so neither more state nor a wider range explains the gain. On Eq. 8, we agree that within a fixed range a denser grid improves coverage only modestly. Our account (§5) predicts a second effect that coverage does not capture: each head is one evaluation point of the memory polynomial, so in our account how many items a probe can resolve depends on the number of distinct timescales MM rather than on dkd_k. The trained th64/d128 checkpoint does keep its timescales distinct: head order is preserved (rank correlation 0.9960.996, W3) and only about 22 of 6464 heads per layer sit within 1%1\% of a neighbour, so its heads provide close to 6464 separate evaluation points.

W3. Theory and model. The constant-retention, delta-free layer is our motivating model: its behaviour can be derived exactly, and it yields the prediction we test, that state spent on more timescales beats state spent on wider heads. Both simplifications leave that prediction intact in the implemented layer. The delta term erases only along the current key, St=(I−βtkt⊤kt) Diag(αt) St−1+βtkt⊤vt\mathbf S_t=(I-\beta_t\mathbf k_t^\top\mathbf k_t)\,\mathrm{Diag}(\boldsymbol\alpha_t)\,\mathbf S_{t-1}+\beta_t\mathbf k_t^\top\mathbf v_t. An earlier association is erased only to the extent that later keys overlap with its own; one whose key is not revisited fades by the retention alone, so the depth trade-off of Proposition 1 still sets how far back a head can recall it.

Input-dependent gating. The per-head, per-channel gate could in principle distort the grid, so we measured it, reading the learned timescales (including the per-channel biases) from every GDN layer of each arm:

Arm (AA range)Range, octaves (init)Order keptSpacing (init)Heads beyond slowest init
th8/d128 ([1,16][1,16])3.9 (4.0)0.8861.30 (1.49)34%
th32/d256 ([1,16384][1,16384])15.5 (14.0)0.9951.37 (1.37)9%
th64/d128 ([1,16384][1,16384])15.4 (14.0)0.9961.17 (1.17)9%

Means over the 15 GDN layers. Range: slowest over fastest timescale, in octaves. Order kept: rank correlation between initial and trained head order. Spacing: median ratio between neighbouring timescales. Heads beyond slowest init: share of heads trained past the slowest initial timescale.

With the wide range used in the ablation, the geometric grid nearly survives training: order, spacing and range are preserved, so the arm that wins in W1 is the one whose trained heads still form the grid the theory describes.

W4. Comparison. Our model is best of all four on the abilities that require locating facts in the corpus (event ordering, information extraction, multi-session reasoning, temporal reasoning), where Qwen2.5, seeing under 11\\% of it, scores 0.0000.000 on three, and it exceeds Gemma-3-27B on average. Qwen2.5's average lead comes mostly from contradiction resolution, which does not require reading the corpus and remembering the information.

Adjusted for scale, the comparison is consequential:

ModelParamsPretraining tokens10M averageAvg. per B params (vs Qwen2.5)Avg. per T tokens (vs Qwen2.5)
Ours2.9B0.136T0.1026.9$\times$82$\times$
Gemma-3-27B-it27B14T0.0920.66$\times$0.72$\times$
Qwen2.5-32B-Instruct32B18T0.1641$\times$1$\times$
Qwen3-30B-A3B30B (3B active)36T0.1190.77$\times$0.36$\times$
  • Abstention is reported in full (Appendix E). We exclude it from the averages because truncation inflates it and is misleading, inability to retrieve conflates with abstention.
  • Efficiency. The GDN state is fixed at 1.971.97M scalars at any length. At 11M tokens our attention cache is 6.46.4 GB, against 262262 GB for Qwen2.5-32B, 9898 GB for Qwen3-30B-A3B and about 8282 GB for Gemma-3-27B.
AuthorsOpenReview ↗

Follow-up on Author Response

Dear Reviewer qYkB,

Thank you again for your time and valuable comments. As the discussion period nears its end, we would like to follow up on our author response. We would appreciate any further feedback and would be happy to clarify any remaining concerns.

Best regards, The authors

Thank you for the additional experiments and clarifications. They address my concerns, and I have updated my Soundness and Informativeness scores accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.