acceptodds
Under review as a conference paper at ICLR 2027

From Cache to Belief: Online Probabilistic Memory for Long-Context Language Models

Abstract

Language models that serve long documents, dialogues, and agents must reuse evidence that has left their attention window, and full attention over the whole history is costly. Compressed memories keep a bounded state instead, but store it as point estimates: a slot cannot tell whether it summarizes many consistent observations or a single noisy one, so how far to trust or overwrite it must be learned implicitly. We argue that bounded memory should maintain beliefs rather than points. BeliefMem is a lightweight adapter that keeps the history beyond the window of a frozen sliding-window model as a bank of Gaussian slot beliefs. One posterior responsibility decides where each evidence block is written and which slots are read; natural-parameter writes let accumulated precision resist overwriting, and a precision-weighted product of experts lets the read revise local evidence only as far as memory is more certain. Trained on 600 examples (27 minutes on one GPU at 7B) with 0.11–0.23% of the backbone parameters, BeliefMem improves six-task LongBench over the local model by 2.72, 4.05, and 2.65 points on Qwen2.5-3B/7B/14B, outperforms all nine Artificial Hippocampus Network (AHN) checkpoints at every scale, and transfers to LongBench v2 and InfiniteBench. A retrained point memory with the same interface trails it in every paired seed, and a causal probe shows that its benefit grows when the answer evidence lies beyond the window.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-context language models, Probabilistic memory, Memory compression, Online belief updating, Uncertainty-aware retrieval, Parameter-efficient adaptation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_QFbkOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes BeliefMem, a lightweight probabilistic memory module for extending the effective context of frozen language models. It uses Gaussian memory slots, precision-weighted updates, and product-of-experts fusion to combine stored information with local context.

Critical strengths

The method is conceptually coherent and computationally lightweight. Evaluations cover multiple models and long-context benchmarks. Controlled comparisons against deterministic memory show encouraging average improvements.

Critical weaknesses

The Bayesian interpretation is not sufficiently validated; precision values may not represent calibrated uncertainty. Improvements are modest and inconsistent across tasks, with some benchmarks showing regressions. Three paired seeds provide limited statistical evidence. End-to-end inference latency, throughput, and memory costs are insufficiently analyzed. Comparisons with competitive retrieval-based and recurrent-memory methods could be more comprehensive.

Questions for authors
  1. Are the learned precision values calibrated against memory reliability or answer correctness?
  2. Are the improvements statistically significant across seeds?
  3. What are the end-to-end inference costs compared with competing methods?
  4. Can the authors explain the task-specific regressions and identify when BeliefMem is most effective?
4 comments
AuthorsOpenReview ↗

Response to Reviewer QFbk (1/3): validating the Bayesian interpretation

W1 / Q1 (validating the Bayesian interpretation). We validated it at three levels.

(1) The update rules do what the Bayesian algebra says. Changing one belief statistic while holding means, counts and evidence fixed (Table 19):

InterventionQuantityBefore → after
Swap slot variances 0.1 ↔ 2.0responsibilities of the two slots0.74 / 0.26 → 0.26 / 0.74
Raise slot precision 0.1 → 100share of the mean moved by a new observation0.98 → 0.05
Read two disagreeing slots equallyreliance gain K0.88 → 0.48
Repeated consistent vs noisy writesmemory variance1.0 → 0.004 vs 0.11

On real text, accumulated precision governs plasticity exactly as derived: a new block moves the memory by 2.0% early in a 128k book and by 0.26% at the end (reply to LuBA-W2).

(2) The model learns better with the probabilistic formulation. The paper's retrained 3B controls (Table 6) keep slots, decoder, data and selection fixed and remove belief components:

Gaussian addressBelief writePredictive read + PoELB6 (mean ± SD)
BeliefMem✓✓✓37.22 ± 0.34
PointContent✓––36.61 ± 0.30
PointMemory–––35.89 ± 0.80
DetRead✓✓–35.52 ± 1.70

The control that is most informative here is DetRead, which keeps the belief state but reads it deterministically. It is the lowest and by far the least stable of the four (SD 1.70 vs 0.34). Without the predictive read and the product of experts, the decoder receives the mixture mean with no reliance gain. It must then apply the memory's correction at the same strength whether the slots agree or disagree and whether they are well or barely written, and has to learn when to trust memory from the task loss alone, which is exactly the problem beliefs are meant to solve. Keeping the beliefs but discarding what they say about reliability is worse than having no beliefs at all. Over 8 pre-registered seeds, the belief memory with natural-parameter forgetting beats PointMemory in 7 of 8 paired seeds (+0.46, p = 0.035).

(3) The inference is close to exact where it matters. Replacing our responsibility by the exact coordinate-ascent update of the mean-field bound changes LB6 by 0.00 ± 0.26 (reply to CT5y-Q3).

Calibration in absolute terms, the narrower question you ask: not yet. After one global temperature, coverage of the memory's predictive read is close to nominal (80 / 92 / 94.5% at 68 / 90 / 95%), but the per-dimension variance does not rank errors better than a constant, and at test time the trained decoder relies on the read means (Appendix G.7–G.8). The conjugate update explains why: a slot's posterior precision is the sum of the observation precisions written into it, so it measures how much evidence the slot holds, not how much that evidence disagreed; disagreement enters only through the spread between slot means in the mixture read. We therefore describe precision as a relative weight in assignment, writing and fusion, not as a calibrated probability. None of the compressed memories we compare with (AHN's recurrent states, KV selection, compressed tokens) represents evidence strength at all, so for them the question cannot be asked; making this quantity calibrated and decision-relevant at test time is the next step that this formulation opens.

AuthorsOpenReview ↗

Response to Reviewer QFbk (2/3): where BeliefMem helps and why some tasks regress

W2 / Q4 (when BeliefMem is most effective, and why some tasks regress).

Most effective: evidence that lies beyond the window and is spread over the input. The causal distance probe isolates the memory's contribution from distant history:

Where the answer occursInside the window8–16k tokens back> 16k tokens back
Memory's contribution (nats)0.621.7–1.82.2

Correspondingly, the largest gains are on distributed evidence: DuReader (+5.4 / +6.4 / +6.2 at 3B / 7B / 14B), HotpotQA (+7.1 at 7B), and LongBench-v2 dialogue history (+20.5 at 7B and 14B).

Regressions and their cause.

(a) Code completion at 7B. We stratified both code tasks by prompt length (7B, BeliefMem − local route, mean ± SE):

Prompt fits the windowPrompt exceeds the window
LCC−4.94 ± 1.25 (n = 467)−0.48 ± 4.69 (n = 25)
RepoBench-P+0.40 ± 1.46 (n = 212)−3.75 ± 1.30 (n = 283)

On LCC the loss appears on prompts that the local route sees in full, so it is not missing recall. Code completion requires copying identifiers token by token, and the memory's correction, which carries a pooled 256-token summary, perturbs that copying. On RepoBench-P the loss beyond the window comes from cross-file identifiers that must be reproduced exactly, which is limit (b) below. The same memory helps code understanding: on LongBench-v2 repository questions BeliefMem is the best compressed method at 7B and 14B (34.0 and 32.0, vs 24.0 and 26.0 for AHN-GDN), and at 3B it improves LCC by +2.51 ± 0.83.

(b) Verbatim retrieval beyond the window (RULER): 0% for every bounded memory, AHN included (reply to CT5y-Q1).

(c) MuSiQue, the hardest multi-hop task for every compressed memory.

In summary, BeliefMem helps most where a compressed memory should: when the answer depends on evidence that has left the window and is spread over the input. Its regressions are confined to token-exact reproduction, which no bounded state provides, and to code completion at one scale. That boundary is shared with AHN, explained by the pooling granularity, and, as Table 29 shows, covered by combining the memory with retrieval.

AuthorsOpenReview ↗

Response to Reviewer QFbk (3/3): seeds, inference cost, retrieval baselines

W3 / Q2 (significance across seeds). Yes, and we strengthened the evidence in two ways.

Significance that includes training randomness. The bootstrap resamples both the 1,200 examples and the training runs:

vs. strongest AHN3B7B14BPooled
Δ LB6 (per-example rescoring)+0.99+1.46+1.04+1.16
one-sided p0.0910.0100.0500.001

More seeds, fixed before training, all reported (Appendix G.2):

SeedsResult
3B cross-scale series836.51 ± 0.92; all 8 runs above the local route (34.24)
Belief vs point memory, with natural-parameter forgetting8 paired+0.46, 7 of 8 pairs, one-sided sign test p = 0.035; gains on 5 of 6 tasks

The original submission's claim was based on three seeds; we replaced it with the eight-seed result.

W4 / Q3 (inference cost). New Table 22 measures all methods back to back on the same GPU at 8k–128k tokens. At 128k:

BeliefMem relative to …3B7B
Sinks+SWA: extra prefill time+7%+3%
Sinks+SWA: extra decode time per token+0.6 ms+0.4 ms
AHN-GDN: decode speed-up2.2×1.5×
AHN-GDN: prefill speed-up1.2×1.2×
Full attention: prefill speed-up2.4×2.1×

BeliefMem costs almost nothing on top of the sliding-window model, decodes 1.5–2.2× faster than AHN-GDN, and prefills 2.1–2.4× faster than full attention. The memory state is constant in the input length (256 slots, 0.13M scalars per layer). Peak GPU memory is dominated by prefill activations and is close to the local route's for all methods, so we claim a latency advantage, not a memory saving.

W5 (retrieval and recurrent baselines). For recurrent memories, we compare with all three AHN families (Mamba2, DeltaNet, Gated DeltaNet) at three scales, trained by their authors on ≈1B tokens. For retrieval, we added BM25 (256-token chunks, Okapi BM25 against the question) and gave the same retrieved passages to every method, running AHN with its official code:

LB63B7B
BM25 only (7k retrieved tokens)36.6141.46
BeliefMem alone (constant-size state)36.9642.48
Sinks+SWA + BM2537.0042.82
Full attention + BM2537.3642.52
AHN-DN + BM25 (strongest AHN)37.9643.41
BeliefMem + BM2539.5345.09

(i) With a constant-size state, BeliefMem alone beats a retrieval pipeline that must index the whole input. (ii) Given the same passages, BeliefMem exceeds full attention by +2.17 / +2.58 and the strongest AHN by +1.57 / +1.68 (p < 10⁻⁴). (iii) Full attention gains nothing from retrieved passages it can already see, whereas every bounded method gains 2.0–4.4 points, because retrieval re-exposes exact text inside the window; BeliefMem stays ahead because its memory supplies the distributed evidence that a single query does not retrieve. The two are complementary.

We appreciate the precise questions; they led to the significance analysis, the cost study, and the retrieval comparison.

Dear Reviewer QFbk,

Following up on your review, we have posted additional results above ("Additional results: the adapter needs no question-answering data, and belief states transfer and compose across models"), with the method unchanged. Together with our earlier replies, in brief:

  • Q2, significance. Resampling both examples and training runs, the gain over the strongest AHN is significant when pooled over scales (p = 0.001); with natural-parameter forgetting, the belief memory beats a retrained point memory in 7 of 8 pre-registered paired seeds (p = 0.035).
  • Q3, inference cost. At 128k tokens, BeliefMem adds +0.4–0.6 ms per token over the sliding-window model and decodes 1.5–2.2× faster than AHN-GDN.
  • W5, retrieval baselines. Given the same BM25 passages, BeliefMem exceeds full attention by +2.2 / +2.6 and the strongest AHN by +1.6 / +1.7 points.
  • Q1, calibration. We state explicitly that precision is a relative weight in the conjugate updates rather than a calibrated probability, and show where the probabilistic formulation matters: removing the predictive read is the least stable control (SD 1.70 vs 0.34).
  • New: robustness to the adaptation data. With pools that contain no question answering at all, the adapter still improves six-task QA by +1.68 to +2.00 (3B) and +3.26 (7B) over the local route.

We hope these address your questions, and thank you again for your detailed review. If you feel that the discussion warrants an update to your rating, we would appreciate your consideration. We fully respect your judgment either way.

Reviewer_LuBAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BeliefMem, a lightweight adapter that stores history beyond a frozen sliding-window LM as Gaussian slot beliefs. Natural-parameter writes let accumulated precision resist overwriting, and a precision-weighted product-of-experts read corrects the local route only when memory is more certain. Trained on 600 examples with 0.11–0.23% of backbone parameters, BeliefMem improves LongBench six-task average by 2.72–4.05 points across Qwen2.5-3B/7B/14B, outperforms all AHN checkpoints, and transfers to LongBench v2 and InfiniteBench. Controlled comparisons show the belief pathway consistently beats a retrained point memory, and a causal probe confirms the benefit concentrates on evidence beyond the window.

Critical strengths
  1. Framing bounded memory as online belief maintenance is novel and principled — precision accumulation and product-of-experts fusion derive overwrite resistance and read reliability directly from the memory state rather than implicit learning, opening a clear direction for future probabilistic memory designs.

  2. The experimental controls are unusually thorough: paired-seed comparison against a retrained point memory with the same architecture isolates the belief contribution, and the causal probe measuring benefit by answer-evidence distance provides direct evidence that the memory genuinely retrieves beyond the window.

  3. The training regime is remarkably lightweight — 600 examples, under 30 minutes on a single GPU, and only 0.11–0.23% of backbone parameters — making the method highly accessible for practical adoption.

Critical weaknesses
  1. Training uses only 600 QA-centric examples; the paper does not evaluate generative long-context tasks (summarization, code, multi-turn dialogue) where evidence is more diffuse. Broader evaluation would strengthen the generality claim.

  2. The memory bank is fixed at 256 slots regardless of input length; slot utilization and potential saturation at extreme lengths are not analyzed. Understanding this failure mode is essential for assessing the method's practical ceiling.

  3. Each slot uses a diagonal Gaussian, so inter-slot correlations are not modeled; discussing how this simplification might limit scenarios requiring cross-slot reasoning would make the analysis more complete.

  4. Across all three model scales in Table 1, average gains over the strongest AHN baseline are modest, with statistical significance unclear.

Questions for authors
  1. How do slot precisions evolve over very long inputs (e.g., 128k tokens)? Do some slots become effectively immutable, and does this hurt performance on late-arriving contradictory evidence?
  2. Have you observed degradation when applying BeliefMem to tasks where answer evidence is less localized, such as long-form generation or open-ended dialogue?
  3. Evaluation uses greedy decoding. Have the authors tried sampling-based decoding with multiple inference seeds to assess whether the gains remain consistent?
  4. See the weaknesses above.
5 comments
AuthorsOpenReview ↗

Response to Reviewer LuBA (1/3): generative tasks and distributed evidence

W1 (generative tasks beyond QA). We agree this matters and evaluated it in two ways.

Already in the paper: multi-turn dialogue and code repositories (LongBench v2, accuracy %), exactly the diffuse-evidence settings you mention:

7B Dialogue7B Code repo14B Dialogue14B Code repo
Sinks+SWA10.328.015.422.0
AHN-GDN18.024.035.926.0
BeliefMem30.834.035.932.0

New: five generative LongBench tasks, scored on items disjoint from the adaptation pool. Paired difference to the local route, mean ± SE over items (3B pooled over three training runs):

BeliefMem − Sinks+SWASAMSum (dialogue summ.)MultiNewsGovReportLCC (code)RepoBench-P (code)
3B+1.20 ± 0.62+0.53 ± 0.24−2.13 ± 0.39+2.51 ± 0.83−0.27 ± 0.87
7B+0.68 ± 0.84−0.39 ± 0.25−0.92 ± 0.37−4.72 ± 1.21−1.98 ± 0.98
Prompts exceeding the 8k window56%3%57%5%57%

The adapter is not QA-specific: trained only on QA-style examples, it improves dialogue summarization, multi-document summarization and code completion at 3B, and it is the best compressed method on dialogue and code-repository questions at 7B and 14B.

Reading the 7B row. Most of these prompts are short: 97% of MultiNews and 95% of LCC prompts fit entirely inside the local window, so on them the memory holds almost nothing and the tasks test whether the adapter stays out of the way, not whether memory helps. Splitting the prompts by whether the memory is actually in use separates the two effects (7B, BeliefMem − local route):

7BMemory nearly empty (prompt fits the window)Memory in use (prompt exceeds the window)
SAMSum (dialogue summarization)−0.74 ± 1.15+1.80 ± 1.19
LCC (code completion)−4.94 ± 1.25 (95% of items)−0.48 ± 4.69

Once the dialogue history actually leaves the window, the memory helps at 7B as well (+1.80). The LCC drop sits almost entirely on prompts the local route sees in full, where the memory holds next to nothing, so it is not a recall failure: code completion copies identifiers token by token, and a correction carrying pooled 256-token evidence perturbs that copying. The remaining code loss, on RepoBench-P, comes from cross-file identifiers that must be reproduced exactly from beyond the window, the verbatim-recall limit shared by every bounded memory (reply to CT5y-Q1); GovReport's long reports demand long outputs, and the adaptation pool contains no long-form generation. Domain coverage in the pool helps: adding 100 code and summarization examples raises the 3B margin on these five tasks from +0.37 to +1.27. Exact code continuation at 7B remains a limitation of the current adapter, which we state in the paper.

Q2 (less-localized evidence). No; the more the evidence is spread over the input, the larger the memory's advantage. Local route (Sinks+SWA) and BeliefMem, with the gain in parentheses, ordered from distributed to localized evidence:

Task (evidence)3B Local3B BeliefMem7B Local7B BeliefMem14B Local14B BeliefMem
LongBench v2 dialogue history17.9525.64 (+7.69)10.2630.77 (+20.51)15.3835.90 (+20.52)
DuReader (multi-document, Chinese web)23.0028.40 (+5.40)25.3331.73 (+6.40)25.1431.31 (+6.17)
HotpotQA (two passages, multi-hop)43.7047.32 (+3.62)51.0758.18 (+7.11)55.9259.41 (+3.49)
TriviaQA (single passage)85.9488.30 (+2.36)88.3490.57 (+2.23)88.6590.22 (+1.57)

The causal distance probe of Sec. 4.4 shows the same at the token level: the memory contributes 0.6 nats when the answer occurs inside the window, 1.7–1.8 nats when it lies 8–16k tokens back, and 2.2 nats beyond 16k. Short-context tasks (ARC, HellaSwag, GSM8K) are unaffected (−0.2 to +2.1). The only degradation we found is code completion at 7B, where the evidence is a token-exact identifier rather than a diffuse one.

AuthorsOpenReview ↗

Response to Reviewer LuBA (2/3): 128k state, late contradictions, cross-slot evidence

W2 / Q1 (saturation, immutability, late contradictions). Thank you; this analysis led to an improvement of the method. We ran the memory over three different 128k-token books and logged the state at every write:

Fraction of the 128k input10%25%50%100%
Effective slots (of 256)149147176237
Slots at the precision cap0000
Largest slot precision (cap 10⁴)0.080.190.360.42
Share by which a new block moves memory1.98%0.73%0.37%0.28%
… with forgetting (BeliefMem-F)2.42%1.34%1.24%1.25%

(Table 23; the main 3B checkpoint ends at 0.26% and 230 slots.)

(1) Capacity is not the ceiling at 128k. No slot comes near saturation, the largest precision is four orders of magnitude below the cap, and utilization grows with length to 230–237 of 256 slots.

(2) The real limit is plasticity, and the belief formulation predicts it exactly. Natural parameters add up, so a new block moves a slot by a share that falls as 1/n with the evidence it has absorbed (7× over 128k).

(3) The fix follows from the same formulation. Shrinking the natural parameters toward the prior before each write, (η, λ) ← r(η, λ) + (1−r)·M₀ (new Eq. 10), is the forgetting factor of Bayesian filtering. It gives the memory an explicit horizon of about 1/(1−r) writes: 100 writes, or ≈25k tokens, at r = 0.99. The memory then stays plastic at ≈1.25% per block throughout the book, and the belief memory beats the retrained point memory in 7 of 8 paired seeds (+0.46, one-sided sign test p = 0.035). The gain over point memory is largest on NarrativeQA (+1.20), the task with the longest inputs, and forgetting helps most on multi-hop HotpotQA (+0.99) and MuSiQue (+0.42), where evidence from different passages must stay revisable.

(4) Late contradictions. In a controlled stream whose true value flips after n₁ writes, with 10% outlier observations:

Writes needed to adapt after the flipn₁ = 32n₁ = 128n₁ = 512 (≈128k tokens)
Belief, no forgetting32128> 256
BeliefMem-F (r = 0.99, horizon ≈ 100 writes)255569
Belief, r = 0.95 (horizon ≈ 20 writes)121414
Point memory (step 0.05, horizon ≈ 20 writes)121414
Error at n₁ = 512 (RMSE)Before the flipAfter the flip
Belief, no forgetting0.031.35
BeliefMem-F (r = 0.99)0.050.17
Belief, r = 0.950.120.12
Point memory0.180.18

Without forgetting, the memory needs as many contradicting writes as it has absorbed: exactly the immutability you anticipated. Forgetting with factor r bounds the memory to about 1/(1−r) recent writes, so adaptation stops growing with the history (69 writes at r = 0.99, consistent with its 100-write horizon). At the same horizon as the point memory (r = 0.95), the belief memory adapts just as fast (14 writes) and is 1.5× more accurate (0.12 vs 0.18), because it weights each observation by its reported precision and discounts the outliers that a fixed step absorbs. A longer horizon trades some speed for an error of 0.05 while the evidence is stable. The belief memory thus gains robustness without giving up adaptivity.

This balance between retaining and forgetting has independent theoretical support: work-efficient agents provably "must balance prediction and forgetting" (Fiderer et al., Phys. Rev. X 16, 041004, 2026), and forgetting is the resource-rational response of a finite memory to a changing environment (Yu, Ku & Griffiths, CogSci 2026). BeliefMem-F exposes this balance as one interpretable parameter.

W3 (cross-slot correlations). Thank you; we added this discussion (Appendix G.11).

Where evidence from different passages is combined. Correlations between slots are not stored, but evidence is combined in two places. At write time, each block is assigned softly to many slots, so every slot accumulates a posterior over many passages; on a 128k book, a block's responsibility spreads over 150–230 effective slots. At read time, the read is the responsibility-weighted mixture over slots, and its variance includes the disagreement between them (Eq. 6), which lowers the reliance gain when slots conflict. Multi-hop QA, which requires combining two passages, is where BeliefMem gains most: HotpotQA +7.11 at 7B, the best compressed method at every scale.

Why diagonal, and how to go further. A diagonal (mean-field) covariance is the standard trade-off in scalable Bayesian deep learning (Blundell et al., 2015): it keeps each cycle linear in the dimension and constant in the history length. The established extension is a low-rank-plus-diagonal covariance, as in SWAG (Maddox et al., 2019), which keeps conjugate updates. We list it for tasks that need structured relations between specific facts, such as MuSiQue with more hops, which remains the hardest task for every compressed memory.

AuthorsOpenReview ↗

Response to Reviewer LuBA (3/3): significance, cost, sampling

W4 (are the gains modest, and are they significant?). We respectfully disagree with "modest", for four reasons.

(1) By the standards of this line of work, the gain is large. In AHN's own paper (their Table 3), AHN improves LB6 over its sliding-window baseline as follows, compared with BeliefMem:

LB6 gain over the sliding-window model3B7B14BTraining
AHN, as reported by its authors+1.58+2.52+1.25≈1B tokens, 32 A100 GPUs, ≈10 h
BeliefMem+2.72+4.05+2.65600 examples, 1 GPU, 27 min

BeliefMem's gain is 1.6–2.1× AHN's own, and its margin over the strongest AHN alone (+1.00 / +1.48 / +1.13) amounts to 60–90% of AHN's entire improvement. For comparison, Memory Caching (Behrouz et al., 2026) reports +0.8% over Titans as its headline gain.

(2) It comes at a fraction of the cost: about 1/700 of AHN's training GPU-hours and 1.5–2.2× faster decoding than AHN-GDN at 128k tokens.

(3) It is statistically significant. Paired bootstrap over the 1,200 LB6 examples, also resampling training runs (Appendix G.1):

vs. strongest AHN3B7B14BPooled
Δ LB6 (per-example rescoring)+0.99+1.46+1.04+1.16
one-sided p0.0910.0100.0500.001

Against the local route, p < 0.001 at every scale. With the same retrieved passages, the margin over AHN is +1.6 to +2.3.

(4) It opens a new direction. To our knowledge, BeliefMem is the first bounded memory for LLMs whose state is a set of posteriors updated by conjugate inference, rather than point estimates. This mirrors recent findings on biological memory: working memory holds probabilistic representations whose precision is encoded separately from confidence (Di, An & Li, Current Biology, 2026), and the human hippocampus adjusts cortical processing according to predicted uncertainty, consistent with precision-weighted prediction errors (Frank et al., Nature Neuroscience, 2026). Our probabilistic control supports the same conclusion for LLMs: with natural-parameter forgetting, the belief memory beats a retrained point memory with the same interface in 7 of 8 pre-registered paired seeds (+0.46, p = 0.035; Table 21). We corrected the corresponding sentence of the original submission, which was based on three seeds.

Q3 (sampling). Yes, and the gains are unchanged. 7B LB6, T = 0.7, top-p 0.8, three decode seeds:

Seed 0Seed 1Seed 2MeanGreedy
Sinks+SWA38.2138.1538.3238.2338.43
AHN-GDN40.1439.5840.0339.9240.39
BeliefMem41.9741.9841.8041.9242.48

BeliefMem leads in every decode seed, by +2.00 over AHN and +3.69 over the local route, matching the greedy margins (+2.07, +4.05), and its spread across decode seeds is the smallest (SD 0.10 vs 0.29 for AHN). Only 47% of predictions are identical between two decode seeds, so the stability is not an artefact of near-deterministic sampling.

Q4. We addressed W1–W4 above. In short: the gains are significant (pooled p = 0.001) and large relative to AHN's own improvement, at a fraction of its cost; the memory does not saturate at 128k, and its plasticity limit has a principled fix that also makes the belief memory consistently better than a point memory (7/8 seeds); evidence from different passages is combined by soft writes and the mixture read, and multi-hop QA is where the gains are largest; the adapter improves dialogue, summarization and code at 3B and is strongest where evidence is most distributed; and the gains hold under sampling. We hope this addresses your concerns about the method's practical ceiling and the reliability of its gains, and we would be glad to answer further questions.

Comments on the paper

7 comments
AuthorsOpenReview ↗

General response: new evidence and revisions

General response — new evidence (revised PDF uploaded; changes in blue, new Appendix G)

We thank the reviewers. All three found the belief view of bounded memory novel and principled and the training regime lightweight. Their shared questions were whether the gains are reliable, what they cost, how the method relates to retrieval, and where it stops working. We ran new experiments on all four, fixed seeds and success criteria before every run, and report every run.

QuestionNew result
Are the gains significant?+1.16 LB6 over the strongest AHN, pooled over 3B/7B/14B, p = 0.001, resampling both examples and training runs; p < 0.001 against the local route at every scale
What does it cost?At 128k tokens, +3–7% prefill and +0.4–0.6 ms/token over the sliding-window model; 1.5–2.2× faster decoding than AHN-GDN
How does it relate to retrieval?Given the same BM25 passages, it exceeds full attention by +2.2 / +2.6 and AHN by +1.6 / +1.7 (3B / 7B, p < 10⁻⁴)
Is the probabilistic formulation what matters?With natural-parameter forgetting, it beats a retrained point memory in 7 of 8 pre-registered paired seeds (p = 0.035); the exact mean-field responsibility costs nothing (0.00 ± 0.26)
Is it robust to decoding?Under sampling, margins are unchanged: +2.00 over AHN, +3.69 over the local route

Memory and retrieval are complementary (every method receives the same 4k retrieved tokens appended to the full context):

LB6Sinks+SWA + BM25AHN (strongest) + BM25Full attention + BM25BeliefMem + BM25
3B37.0037.9637.3639.53
7B42.8243.4142.5245.09

One correction. When extending the point-memory control from three to eight seeds, we added the forgetting step that the conjugate update calls for (Eq. 10); this variant gives the consistent result above. The original recipe is ahead on average (+0.23), so we replaced the sentence "in every paired seed" in the abstract and Sec. 4.3.

Limits we now state explicitly: like every bounded memory we tested, AHN included, BeliefMem does not recover a verbatim string from beyond the window; precision is a relative weight in the conjugate updates rather than a calibrated probability; and code completion at 7B is weaker than the local route. The per-reviewer replies analyse each.

AuthorsOpenReview ↗

Response to Reviewer CT5y (1/3): exact retrieval

W1 / Q1 (exact retrieval). We evaluated RULER needle-in-a-haystack (single- and multi-key, 16k–128k, 3B and 7B), split by where the needle lies:

RULER accuracy (%)Needle inside the windowNeedle beyond the window
Sinks+SWA98–100 (82 at 128k)0
AHN-GDN98–100 (78–82 at 128k)0
BeliefMem98–100 (82 at 128k)0
Full attention (reference, up to 32k)99–10099–100

The memory does not hurt in-window retrieval, and beyond the window no bounded memory recovers a verbatim needle, AHN included. This is a property of bounded state rather than of the diagonal-belief design, for two reasons. Capacity: a needle is an arbitrary string at an arbitrary position among N tokens, so storing it for later lookup requires a state that grows with the input; fixed-state models are provably limited at copying for this reason (Jelassi et al., ICML 2024). Granularity: each BeliefMem write pools a 256-token block into one evidence vector, which keeps what a passage is about and drops which exact tokens it contains. Our passkey probe shows the result: beyond the window, the memory lowers the NLL of a 5-digit key from 3.93 to 3.50 nats and raises the probability of its first digit from 0.85 to 0.90, so it retains partial evidence about the key, but not the exact string. The same trade-off is what makes the gains on distributed evidence possible.

Where verbatim lookup is needed, the bounded state combines with retrieval, and the combination beats every alternative. With the same BM25 passages for every method:

LB63B7B
Full attention + BM2537.3642.52
AHN-DN + BM2537.9643.41
BeliefMem + BM2539.5345.09

Retrieval brings exact text back into the window, and the belief memory supplies the evidence spread over the rest of the history; together they exceed full attention by +2.2 / +2.6 points while keeping a constant-size state.

AuthorsOpenReview ↗

Response to Reviewer CT5y (2/3): approximations versus the exact formulation

W2 / Q3 (approximations versus the exact formulation). Thank you for this question. Deciding which parts of the exact Bayesian computation to keep, and which to relax so that the memory trains in minutes and runs at the speed of the local model, is where we spent most of the design effort. We answer with the experiment you propose and then explain each approximation.

The experiment. We retrained the three 3B paper seeds with the exact coordinate-ascent responsibility of the mean-field bound, with no dimension normalization, no overlap score and no learned temperature, γ_k ∝ exp(E[log π_k] + Σ_j E_q log N(a_j; μᵃ_kj, vᵃ_kj) + Σ_j E_q log N(c_j; μᶜ_kj, vᶜ_kj)), in both training and evaluation:

ResponsibilitySeed 1337Seed 2026Seed 3407Mean LB6
Paper (normalized, overlap, learned τ)37.9436.0536.8936.96
Exact mean-field37.5135.8837.5036.96
Paper with τ frozen37.6735.6236.8336.71

Performance is insensitive to the approximations: the exact update gives the same mean LB6 (change 0.00 ± 0.26). The learned temperature τ also stays within 0.01 of its initialization in every checkpoint (now stated in Sec. 3.2).

Why each remaining approximation is there, and why it is principled.

  • Mean-field factorization is the standard variational family. It turns slot assignment into a closed-form softmax and keeps each cycle O(K·D).
  • Moment projection of the read is exact for the first two moments of the slot mixture (Eq. 6), not a heuristic. Without it, the posterior over a stream becomes a mixture whose number of components grows exponentially with the number of blocks, which is intractable; projecting back to a Gaussian after each step is assumed-density filtering, the principle behind the Kalman filter and expectation propagation.
  • The write remains the exact conjugate update for the chosen Gaussian sites.

These choices make the method practical: a 7B adapter trains in 27 minutes on one GPU and decodes 1.5–2.2× faster than AHN at 128k tokens. The experiment shows they cost no accuracy relative to the exact update.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.