- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes BeliefMem, a lightweight probabilistic memory module for extending the effective context of frozen language models. It uses Gaussian memory slots, precision-weighted updates, and product-of-experts fusion to combine stored information with local context.
Critical strengths
The method is conceptually coherent and computationally lightweight. Evaluations cover multiple models and long-context benchmarks. Controlled comparisons against deterministic memory show encouraging average improvements.
Critical weaknesses
The Bayesian interpretation is not sufficiently validated; precision values may not represent calibrated uncertainty. Improvements are modest and inconsistent across tasks, with some benchmarks showing regressions. Three paired seeds provide limited statistical evidence. End-to-end inference latency, throughput, and memory costs are insufficiently analyzed. Comparisons with competitive retrieval-based and recurrent-memory methods could be more comprehensive.
Questions for authors
- Are the learned precision values calibrated against memory reliability or answer correctness?
- Are the improvements statistically significant across seeds?
- What are the end-to-end inference costs compared with competing methods?
- Can the authors explain the task-specific regressions and identify when BeliefMem is most effective?
4 comments
Response to Reviewer QFbk (2/3): where BeliefMem helps and why some tasks regress
W2 / Q4 (when BeliefMem is most effective, and why some tasks regress).
Most effective: evidence that lies beyond the window and is spread over the input. The causal distance probe isolates the memory's contribution from distant history:
| Where the answer occurs | Inside the window | 8–16k tokens back | > 16k tokens back |
|---|---|---|---|
| Memory's contribution (nats) | 0.62 | 1.7–1.8 | 2.2 |
Correspondingly, the largest gains are on distributed evidence: DuReader (+5.4 / +6.4 / +6.2 at 3B / 7B / 14B), HotpotQA (+7.1 at 7B), and LongBench-v2 dialogue history (+20.5 at 7B and 14B).
Regressions and their cause.
(a) Code completion at 7B. We stratified both code tasks by prompt length (7B, BeliefMem − local route, mean ± SE):
| Prompt fits the window | Prompt exceeds the window | |
|---|---|---|
| LCC | −4.94 ± 1.25 (n = 467) | −0.48 ± 4.69 (n = 25) |
| RepoBench-P | +0.40 ± 1.46 (n = 212) | −3.75 ± 1.30 (n = 283) |
On LCC the loss appears on prompts that the local route sees in full, so it is not missing recall. Code completion requires copying identifiers token by token, and the memory's correction, which carries a pooled 256-token summary, perturbs that copying. On RepoBench-P the loss beyond the window comes from cross-file identifiers that must be reproduced exactly, which is limit (b) below. The same memory helps code understanding: on LongBench-v2 repository questions BeliefMem is the best compressed method at 7B and 14B (34.0 and 32.0, vs 24.0 and 26.0 for AHN-GDN), and at 3B it improves LCC by +2.51 ± 0.83.
(b) Verbatim retrieval beyond the window (RULER): 0% for every bounded memory, AHN included (reply to CT5y-Q1).
(c) MuSiQue, the hardest multi-hop task for every compressed memory.
In summary, BeliefMem helps most where a compressed memory should: when the answer depends on evidence that has left the window and is spread over the input. Its regressions are confined to token-exact reproduction, which no bounded state provides, and to code completion at one scale. That boundary is shared with AHN, explained by the pooling granularity, and, as Table 29 shows, covered by combining the memory with retrieval.
Response to Reviewer QFbk (3/3): seeds, inference cost, retrieval baselines
W3 / Q2 (significance across seeds). Yes, and we strengthened the evidence in two ways.
Significance that includes training randomness. The bootstrap resamples both the 1,200 examples and the training runs:
| vs. strongest AHN | 3B | 7B | 14B | Pooled |
|---|---|---|---|---|
| Δ LB6 (per-example rescoring) | +0.99 | +1.46 | +1.04 | +1.16 |
| one-sided p | 0.091 | 0.010 | 0.050 | 0.001 |
More seeds, fixed before training, all reported (Appendix G.2):
| Seeds | Result | |
|---|---|---|
| 3B cross-scale series | 8 | 36.51 ± 0.92; all 8 runs above the local route (34.24) |
| Belief vs point memory, with natural-parameter forgetting | 8 paired | +0.46, 7 of 8 pairs, one-sided sign test p = 0.035; gains on 5 of 6 tasks |
The original submission's claim was based on three seeds; we replaced it with the eight-seed result.
W4 / Q3 (inference cost). New Table 22 measures all methods back to back on the same GPU at 8k–128k tokens. At 128k:
| BeliefMem relative to … | 3B | 7B |
|---|---|---|
| Sinks+SWA: extra prefill time | +7% | +3% |
| Sinks+SWA: extra decode time per token | +0.6 ms | +0.4 ms |
| AHN-GDN: decode speed-up | 2.2× | 1.5× |
| AHN-GDN: prefill speed-up | 1.2× | 1.2× |
| Full attention: prefill speed-up | 2.4× | 2.1× |
BeliefMem costs almost nothing on top of the sliding-window model, decodes 1.5–2.2× faster than AHN-GDN, and prefills 2.1–2.4× faster than full attention. The memory state is constant in the input length (256 slots, 0.13M scalars per layer). Peak GPU memory is dominated by prefill activations and is close to the local route's for all methods, so we claim a latency advantage, not a memory saving.
W5 (retrieval and recurrent baselines). For recurrent memories, we compare with all three AHN families (Mamba2, DeltaNet, Gated DeltaNet) at three scales, trained by their authors on ≈1B tokens. For retrieval, we added BM25 (256-token chunks, Okapi BM25 against the question) and gave the same retrieved passages to every method, running AHN with its official code:
| LB6 | 3B | 7B |
|---|---|---|
| BM25 only (7k retrieved tokens) | 36.61 | 41.46 |
| BeliefMem alone (constant-size state) | 36.96 | 42.48 |
| Sinks+SWA + BM25 | 37.00 | 42.82 |
| Full attention + BM25 | 37.36 | 42.52 |
| AHN-DN + BM25 (strongest AHN) | 37.96 | 43.41 |
| BeliefMem + BM25 | 39.53 | 45.09 |
(i) With a constant-size state, BeliefMem alone beats a retrieval pipeline that must index the whole input. (ii) Given the same passages, BeliefMem exceeds full attention by +2.17 / +2.58 and the strongest AHN by +1.57 / +1.68 (p < 10⁻⁴). (iii) Full attention gains nothing from retrieved passages it can already see, whereas every bounded method gains 2.0–4.4 points, because retrieval re-exposes exact text inside the window; BeliefMem stays ahead because its memory supplies the distributed evidence that a single query does not retrieve. The two are complementary.
We appreciate the precise questions; they led to the significance analysis, the cost study, and the retrieval comparison.
Dear Reviewer QFbk,
Following up on your review, we have posted additional results above ("Additional results: the adapter needs no question-answering data, and belief states transfer and compose across models"), with the method unchanged. Together with our earlier replies, in brief:
- Q2, significance. Resampling both examples and training runs, the gain over the strongest AHN is significant when pooled over scales (p = 0.001); with natural-parameter forgetting, the belief memory beats a retrained point memory in 7 of 8 pre-registered paired seeds (p = 0.035).
- Q3, inference cost. At 128k tokens, BeliefMem adds +0.4–0.6 ms per token over the sliding-window model and decodes 1.5–2.2× faster than AHN-GDN.
- W5, retrieval baselines. Given the same BM25 passages, BeliefMem exceeds full attention by +2.2 / +2.6 and the strongest AHN by +1.6 / +1.7 points.
- Q1, calibration. We state explicitly that precision is a relative weight in the conjugate updates rather than a calibrated probability, and show where the probabilistic formulation matters: removing the predictive read is the least stable control (SD 1.70 vs 0.34).
- New: robustness to the adaptation data. With pools that contain no question answering at all, the adapter still improves six-task QA by +1.68 to +2.00 (3B) and +3.26 (7B) over the local route.
We hope these address your questions, and thank you again for your detailed review. If you feel that the discussion warrants an update to your rating, we would appreciate your consideration. We fully respect your judgment either way.
Response to Reviewer QFbk (1/3): validating the Bayesian interpretation
W1 / Q1 (validating the Bayesian interpretation). We validated it at three levels.
(1) The update rules do what the Bayesian algebra says. Changing one belief statistic while holding means, counts and evidence fixed (Table 19):
On real text, accumulated precision governs plasticity exactly as derived: a new block moves the memory by 2.0% early in a 128k book and by 0.26% at the end (reply to LuBA-W2).
(2) The model learns better with the probabilistic formulation. The paper's retrained 3B controls (Table 6) keep slots, decoder, data and selection fixed and remove belief components:
The control that is most informative here is DetRead, which keeps the belief state but reads it deterministically. It is the lowest and by far the least stable of the four (SD 1.70 vs 0.34). Without the predictive read and the product of experts, the decoder receives the mixture mean with no reliance gain. It must then apply the memory's correction at the same strength whether the slots agree or disagree and whether they are well or barely written, and has to learn when to trust memory from the task loss alone, which is exactly the problem beliefs are meant to solve. Keeping the beliefs but discarding what they say about reliability is worse than having no beliefs at all. Over 8 pre-registered seeds, the belief memory with natural-parameter forgetting beats PointMemory in 7 of 8 paired seeds (+0.46, p = 0.035).
(3) The inference is close to exact where it matters. Replacing our responsibility by the exact coordinate-ascent update of the mean-field bound changes LB6 by 0.00 ± 0.26 (reply to CT5y-Q3).
Calibration in absolute terms, the narrower question you ask: not yet. After one global temperature, coverage of the memory's predictive read is close to nominal (80 / 92 / 94.5% at 68 / 90 / 95%), but the per-dimension variance does not rank errors better than a constant, and at test time the trained decoder relies on the read means (Appendix G.7–G.8). The conjugate update explains why: a slot's posterior precision is the sum of the observation precisions written into it, so it measures how much evidence the slot holds, not how much that evidence disagreed; disagreement enters only through the spread between slot means in the mixture read. We therefore describe precision as a relative weight in assignment, writing and fusion, not as a calibrated probability. None of the compressed memories we compare with (AHN's recurrent states, KV selection, compressed tokens) represents evidence strength at all, so for them the question cannot be asked; making this quantity calibrated and decision-relevant at test time is the next step that this formulation opens.