- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies generative retrieval from online fast weight memory. It proposes Test-Time Generation (TTG), which replaces deterministic regression readouts with query-conditioned denoising while preserving the fixed-size memory state. Motivated by the inability of deterministic squared-error regression to represent multiple probable values associated with the same key. Experiments evaluate long-context language modeling across 124M–3B scales, novel view synthesis, and controlled two-value associative recall.
Critical strengths
The paper presents an interesting approach to improving memory retrieval independently of the memory update architecture. The integration of conditional generation with online fast-weight memory is a meaningful extension of prior test-time training methods. The experiments cover multiple model scales and modalities, demonstrating improvements in long-context retrieval and novel view synthesis. The controlled two-value recall experiment also provides useful evidence that stochastic readouts can recover multiple stored associations under limited memory load.
Critical weaknesses
- Insufficient evidence connecting the proposed generative retrieval to downstream improvements.
-
The central motivation is that iterative generative readouts overcome the conditional-mean limitation of deterministic memory retrieval. However, the headline language-modeling improvements are obtained with a single readout step, where the idealized Bayes-optimal denoiser returns the conditional mean rather than separating modes.
-
While TTG improves long-context retrieval, its language modeling improvements are relatively modest and inconsistent across scales. In particular, deep-context PTL becomes worse than LaCT at 3B. Furthermore, evaluation is largely restricted to Book-3 tail loss, without independent language modeling benchmarks such as WikiText or LAMBADA. This limits the evidence that the proposed readout improves general language modeling capabilities beyond long-context retrieval.
- Confounded comparisons.
- In the 124M family, changes in conditioning architecture and parameter count substantially affect performance, making it unclear how much of the reported improvement is attributable to the proposed mechanism. Moreover, the 760M default configuration outperforms LaCT in the main comparison but not in the separate implementation cohort. More consistently matched, parameter-controlled comparisons and independent-seed evaluations would strengthen the evidence for the proposed readout's effectiveness.
Questions for authors
-
Please provide more direct evidence that the proposed multimodal generative retrieval mechanism contributes to the downstream improvements? For example, a controlled comparison between single-step and multi-step readouts under matched training configurations, together with an analysis connecting mode recovery to downstream performance, would be helpful.
-
Please provide additional language modeling evaluations on standard benchmarks such as WikiText/LAMBADA, comparing TTG against LaCT under matched settings. This would help clarify whether the reported retrieval improvements generalize to broader language modeling capabilities.
-
Please provide a deterministic baseline matched to the 124M TTG configuration (TE + per-block AdaLN, 172.7M parameters), under identical training and evaluation settings. Additionally, multiple-seed results with mean + std would help check robustness of the reported gains.
I'm open to increasing my score if authors address at least two of these three concerns.
6 comments
To examine whether the decoder retains information about multiple stored values, we further analyze its output probability distribution at the first digit of the answer, using 16K documents in which all four values lie outside the local attention window.
Table 3. First-digit probability analysis on 16K NIAH-MultiValue documents with four out-of-window values.
| Model | Probability on stored leading digits (%) | Unrelated-document control (%) | Stored leading digits with probability ≥0.1 |
|---|---|---|---|
| LaCT | 36.1 | 31.9 | 0.89 |
| TTG (r1d1) | 76.9 | 37.3 | 2.32 |
TTG assigns 76.9% probability mass to the stored values' leading digits, compared with 36.1% for LaCT. Replacing the target digit set with that from unrelated documents reduces these probabilities to 37.3% and 31.9%, respectively. Furthermore, TTG assigns probability ≥0.1 to an average of 2.32 stored leading-digit candidates, compared with 0.89 for LaCT.
These results provide complementary evidence that TTG's decoder can access information associated with multiple stored values, even when those values are outside the local attention window.
- Reconciling single-step improvements with iterative generative retrieval.
We emphasize that improved retrieval and explicit multimodal recovery are related but distinct outcomes. Our single-step language-modeling results demonstrate practical improvements under matched numbers of readout evaluations, without establishing mode separation. The controlled two-value experiments demonstrate that iterative refinement can recover distinct modes, while the additional downstream experiments show that deeper trained readouts improve multi-value retrieval and that TTG provides stronger access to multiple stored values.
Overall, the additional results establish three complementary findings: (i) iterative generative refinement enables stronger mode recovery in controlled memories; (ii) increasing trained readout depth improves downstream multi-value and overall RULER accuracy; and (iii) pretrained TTG exhibits stronger downstream access to multiple stored values.
W1.2 & Q2: TTG shows limited and inconsistent language modeling improvements over LaCT, motivating additional evaluations on standard benchmarks such as WikiText and LAMBADA to assess whether its retrieval gains generalize to broader language modeling capabilities.
A2: To address this concern, we conducted additional evaluations using the 760M model family, comparing TTG against LaCT under matched training and evaluation settings. The new results are summarized in Tables G1–G4.
First, TTG achieves consistently lower perplexity on independent language modeling benchmarks (Table G2). On WikiText and LAMBADA, TTG improves perplexity from 21.62 to 21.54 and from 16.36 to 16.16, respectively. Similar improvements are observed on long-context language modeling datasets, including PG-19 (14.87 → 14.78) and GovReport (6.67 → 6.61), both evaluated on contexts exceeding 30K tokens. Although these improvements are modest, they are consistent across all four datasets, providing evidence that TTG's benefits are not restricted to Book-3 tail loss.
Second, the improvements extend to broader downstream language understanding tasks (Tables G1, G3, and G4). On LongBench v1, TTG outperforms LaCT in five of six task categories, with particularly notable improvements in single-document QA (13.3 → 17.8) and multi-document QA (16.2 → 22.5). On LongBench v2, accuracy increases from 28.0 to 29.6. TTG also improves the average score on six recall-intensive tasks from 45.5 to 47.5, outperforming LaCT on every task. Furthermore, across seven zero-shot commonsense reasoning benchmarks, TTG increases the average accuracy from 43.6 to 44.6.
Taken together, these additional evaluations suggest that TTG's advantages extend beyond synthetic long-context retrieval to standard language modeling, document understanding, and downstream reasoning tasks. The new evidence supports broader generalization of TTG's readout improvements beyond the original retrieval-focused evaluation.
Table G1. LongBench v1 (by category) and LongBench v2
| Model | Single-doc QA ↑ | Multi-doc QA ↑ | Summarization ↑ | Few-shot ↑ | Synthetic ↑ | Code ↑ | LongBench v2 ↑ |
|---|---|---|---|---|---|---|---|
| Transformer | 12.6 | 18.3 | 11.3 | 23.0 | 2.7 | 34.3 | 27.0 |
| Transformer-SWA | 16.1 | 19.3 | 7.9 | 20.2 | 0.7 | 31.3 | 28.0 |
| GDN-SWA | 13.4 | 13.9 | 9.4 | 22.7 | 0.9 | 33.8 | 22.7 |
| DeltaNet-SWA | 15.8 | 19.3 | 8.1 | 29.4 | 0.2 | 34.7 | 28.0 |
| Mamba-2-SWA | 15.0 | 16.1 | 10.5 | 28.4 | 1.8 | 35.3 | 20.5 |
| Lact | 13.3 | 16.2 | 8.8 | 31.3 | 2.9 | 36.0 | 28.0 |
| TTG | 17.8 | 22.5 | 11.7 | 34.5 | 1.2 | 38.0 | 29.6 |
Table G2. Short- vs. long-context perplexity
| Model | WikiText | LAMBADA | PG-19 30K+ | GovReport 30K+ |
|---|---|---|---|---|
| Transformer | 23.50 | 22.23 | 15.43 | 6.48 |
| Transformer-SWA | 22.83 | 18.92 | 15.71 | 7.10 |
| GDN-SWA | 22.50 | 20.20 | 15.33 | 7.02 |
| DeltaNet-SWA | 22.52 | 18.13 | 15.32 | 6.97 |
| Mamba-2-SWA | 21.76 | 17.12 | 15.15 | 6.84 |
| Lact | 21.62 | 16.36 | 14.87 | 6.67 |
| TTG | 21.54 | 16.16 | 14.78 | 6.61 |
We thank the reviewer for the insightful suggestion, and we would like to provide a few clarification.
W1.1 & Q1: The downstream improvements lack a clear connection to the claimed multimodal generative retrieval mechanism, as the main gains arise from single-step readouts that theoretically recover only the conditional mean, motivating controlled comparisons with multi-step readouts and analysis of how mode recovery affects downstream performance.
A1: To clarify the connection between iterative generative retrieval and downstream performance, we provide additional evidence from controlled mode-recovery experiments, trained readout-depth comparisons, and downstream analyses of multiple stored values.
Our controlled two-value recall experiments (Section 4.4, Appendix E) directly examine the effect of inference readout depth (S) on recovering multiple values associated with the same key.
At (N=4), increasing (S) from 1 to 4 improves two-value coverage from 22.2% to 65.9%. At (N=16), increasing (S) from 1 to 8 improves coverage from 0% to 34.4%, while deterministic controls retain zero coverage. These experiments demonstrate that iterative generative refinement can improve recovery of distinct stored values under controlled conditions.
To examine whether deeper generative readouts also improve downstream retrieval, we compare 3B TTG models with trained readout depths (d=1) and (d=2), holding write repeats fixed at (r=1). We evaluate both NIAH-MultiValue, which requires recovering multiple values associated with the same key, and overall RULER accuracy across 13 tasks.
Table 1. NIAH-MultiValue (MV) and overall RULER accuracy (%) for 3B TTG models.
Increasing trained readout depth from (d=1) to (d=2) improves NIAH-MultiValue accuracy from 26.4% to 38.3% (+11.9 points) and overall RULER accuracy from 34.87% to 36.55% (+1.68 points), with improvements at all four context lengths.
The substantially larger gain on NIAH-MultiValue suggests that deeper trained readouts are particularly beneficial for retrieving multiple values associated with the same key.
We further investigate whether pretrained language models can access information about multiple stored values beyond the local attention window, using pretrained 760M TTG (r1d1) and LaCT checkpoints.
First, we evaluate how often a stored value outside the local attention window appears in the model's response on RULER-based single-value and four-value retrieval tasks.
Table 2. Fraction of responses containing an out-of-window stored value (%).
TTG shows substantially stronger out-of-window retrieval, particularly at 8K, improving single-value retrieval from 24.1% to 69.0% and four-value retrieval from 5.6% to 28.2%. The advantage persists at 16K for four-value retrieval (6.6% vs. 0.3%), although both models degrade at longer contexts.