- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces TTT-Retrieval, a framework for corpus-conditioned retrieval that augments a frozen embedding backbone with lightweight, meta-trained neural memory modules. At test time, these modules are updated through self-supervised learning on an unlabeled target corpus. The resulting memory state conditions query and document embeddings and is reused across subsequent queries. Experiments on LoCoMo, QASPER, and HotpotQA demonstrate improvements over frozen and supervised fine-tuned baselines using Qwen3-Embedding-0.6B, with performance competitive with or superior to fine-tuning plus reranking. A controlled comparison using nomic-embed-text-v1 also shows improvements over prior Contextual Document Embeddings (CDE) approach. Overall, the approach is intuitive and demonstrates promising retrieval improvements. However, I have concerns about the breadth of evaluation, the computational efficiency when accounting for test-time adaptation, and the method’s applicability to growing or frequently updated corpora.
Critical strengths
Simple and effective approach to contextual retrieval. TTT-Retrieval provides an intuitive way to incorporate corpus-specific information into retrieval representations by adding lightweight neural memory modules to a frozen embedding backbone. The experiments demonstrate consistent improvements across three retrieval benchmarks, with competitive results against supervised fine-tuning and reranking baselines.
Generally clear methodology and informative empirical comparisons. The write/read mechanism and meta-learning formulation are generally well explained. The experimental comparisons and analyses are useful for understanding the method’s effectiveness, although some interpretations would benefit from further clarification.
Critical weaknesses
Limited evaluation breath and incomplete computational efficiency comparison. Although the paper includes a controlled comparison using nomic-embed-text-v1 (Figure 4), the main experimental comparisons primarily use Qwen3-Embedding-0.6B. Broader evaluation across embedding model architectures and model sizes would strengthen the evidence for generalizability. The evaluation also relies on Recall@k, where more traditional IR evaluation metrics like nDCG@k could provide additional insight into ranking quality. More importantly, the efficiency comparison in Figure 3 considers only read-time FLOPs after corpus adaptation. They do not account for the computation required to write the corpus into memory. Reporting write-time FLOPs and amortized computation as a function of the number of queries would provide a more complete picture of deployment efficiency.
Unclear scalability and applicability to evolving corpora. The method’s efficacy when documents are incrementally added to an existing corpus remains insufficiently evaluated. It is unclear whether updates to the adapted memory require previously indexed documents to be re-embedded to maintain a consistent retrieval space. Furthermore, since the neural memory has a fixed parameter budget, it would be helpful to understand how retrieval performance changes as the number of documents grows. I think these are especially relevant to frequently updated retrieval environments in current days, such as codebases. While the approach appears well suited to relatively static corpora like long-doc QA, its applicability to dynamic retrieval settings remain unclear.
Questions for authors
Qualitative retrieval analysis and the reliability of query-type annotations (Section 4.5). Table 3 lists representative queries where TTT-Retrieval succeeds but Fine-Tuning + Reranking fails. However, without examples of the actual retrieved evidence, it is difficult to understand what information TTT-Retrieval captures more effectively. Additionally, Appendix E indicates that aggregative queries are annotated using only query text. This may not reliably distinguish queries requiring evidence aggregation from those answerable using a single passage, since the distinction depends on the underlying corpus. Could the authors provide concrete examples of retrieved evidence and clarify how reliable the query-text-only annotations are for supporting the conclusions in Section 4.5?
Motivation and empirical necessity of the write-budget curriculum (Section 3.5). The paper motivates the curriculum as a way to improve training stability and tractability. However, it remains unclear whether this design is empirically necessary for effective meta-learning. Just curious: did the authors conduct an ablation study for meta-learning without curriculum? Could the authors clarify whether the curriculum is primarily an optimization heuristic and how it affects learning the memory initialization ?
Retrieval granularity. Could the authors clarify what is considered a retrieved document for Recall@k in each benchmark, particularly LoCoMo? Appendix C specifies paragraph-level retrieval for QASPER and document-level retrieval for HotpotQA, but the retrieval granularity for LoCoMo is less clear.
4 comments
Response to Reviewer U6vW [2/3]
W1-1. Broader evaluation across embedding model architectures and model sizes would strengthen the evidence for generalizability.
Thank you for the suggestion. In addition to the Qwen3 embedding backbone used in the main experiments (decoder-based, 0.6B), we evaluated TTT-Retrieval on ModernBERT-base (encoder-only, 149M). Trained from the same checkpoint on the same task data, TTT-Retrieval outperforms CDE-v2 on all three datasets (Table R2).
[Table R2] Comparison on ModernBERT-base (R@10, %).
| Method | LoCoMo | QASPER | HotpotQA |
|---|---|---|---|
| CDE-v2 | 76.3 | 76.9 | 55.6 |
| TTT-Retrieval (Ours) | 80.9 | 80.3 | 63.2 |
These results show that the gains are not specific to one backbone architecture or size. We will add them to the main paper.
W1-2. The evaluation also relies on Recall@k, where more traditional IR evaluation metrics like nDCG@k could provide additional insight into ranking quality.
Thank you for the suggestion. Table R3 adds nDCG@10 on QASPER, together with Fine-Tuning without reranking as a non-reranking baseline. TTT-Retrieval improves Recall@10 and nDCG@10 over Fine-Tuning by 4.7 and 6.3 percentage points, respectively, and over Fine-Tuning + Reranking by 8.8 and 10.1 percentage points. The improvements in nDCG@10 complement the Recall@10 results, showing gains in a rank-sensitive retrieval metric as well as evidence coverage.
[Table R3] Recall@10 and nDCG@10 on QASPER (%).
| Method | Recall@10 | nDCG@10 |
|---|---|---|
| Fine-Tuning | 76.7 | 62.0 |
| Fine-Tuning + Reranking | 72.6 | 58.2 |
| TTT-Retrieval (Ours) | 81.4 | 68.3 |
We will report nDCG@10 for all datasets and methods in the revision.
W1-3. Reporting write-time FLOPs and amortized computation as a function of the number of queries would provide a more complete picture of deployment efficiency.
We thank the reviewer for this question.
Break-even against reranking. Both methods encode documents, so the shared encoding cost cancels. The additional preprocessing cost of TTT-Retrieval is the write cost plus the extra fast-weight computation in document embedding. For a 32K-token corpus, this is 41.21 TFLOPs (38.15 for writing and 3.06 for the extra document-embedding computation). Each query then saves 15.2 TFLOPs relative to Base + Reranking, net of TTT-Retrieval's slightly higher query-embedding cost. Table R4 shows the cumulative effect.
[Table R4] Cumulative compute of TTT-Retrieval relative to Reranking (32K-token corpus).
| Number of queries | Cumulative query-time savings (TFLOPs) | One-time preprocessing per corpus (TFLOPs) | Net saving (TFLOPs) |
|---|---|---|---|
| 1 | 15.2 | 41.2 | −26.0 |
| 5 | 76.0 | 41.2 | +34.8 |
| 10 | 152.1 | 41.2 | +110.9 |
| 50 | 760.4 | 41.2 | +719.2 |
TTT-Retrieval recovers its additional preprocessing cost after 2.71 queries for this corpus (i.e., by the third query). The preprocessing cost is amortized over repeated queries on the same corpus version and can run outside the online query path.
Response to Reviewer U6vW [3/3]
W2. (a) It is unclear whether updates to the adapted memory require previously indexed documents to be re-embedded to maintain a consistent retrieval space. (b) Since the neural memory has a fixed parameter budget, it would be helpful to understand how retrieval performance changes as the number of documents grows.
Thank you for these questions. Both concern how retrieval behaves as more content is written into memory, so we address them together with two experiments on LoCoMo.
Incremental writes and re-embedding (W2-a). Our experiments suggest that previously indexed embeddings remain usable without immediate re-embedding, although refreshing the index generally improves retrieval performance.
To test this, we write each LoCoMo conversation as a single stream and snapshot the memory after writing 1K, 2K, 4K, and 8K tokens, as well as after the full conversation. At each stage, queries are embedded using the current memory state and scored against all documents seen so far. We compare two settings: Refresh, our default setting, re-embeds all documents using the current memory state, whereas Stale embeds only newly arrived documents and retains previously computed embeddings. The two settings are identical at the first stage because all documents are new.
[Table R5] R@10 (%) under incremental writes on LoCoMo.
| Write budget (tokens) | #Queries | Stale | Refresh |
|---|---|---|---|
| 1024 | 54 | 98.1 | 98.1 |
| 2048 | 103 | 92.7 | 93.7 |
| 4096 | 192 | 93.2 | 93.5 |
| 8192 | 377 | 91.8 | 92.4 |
| Full | 1540 | 83.7 | 84.7 |
Refresh matches or outperforms Stale at every stage, with Stale remaining within 1.0 R@10 point of Refresh. These results suggest that previously computed embeddings remain largely compatible with the updated memory, making Stale a lower-cost alternative when re-embedding the full index after every update is expensive.
Memory capacity (W2-b). In Table R5, both the document pool and the query set grow with the write budget. To isolate the effect of additional writes on a fixed-size memory, we keep the LoCoMo query set and candidate documents fixed while varying only the amount of written text. Specifically, we first write each LoCoMo conversation and then append 4K to 128K tokens of PG-19 text. After all writes, we re-embed the fixed queries and documents using the final memory state (Table R6).
[Table R6] LoCoMo R@10 (%) after additionally writing PG-19 text.
| Additional PG-19 tokens | 0 | 4K | 8K | 16K | 32K | 64K | 128K |
|---|---|---|---|---|---|---|---|
| TTT-Retrieval (Ours), R@10 | 84.75 | 84.50 | 84.55 | 84.35 | 84.19 | 83.53 | 79.32 |
TTT-Retrieval maintains stable retrieval performance as additional text is written to the fixed-size memory. R@10 remains above 84% through 32K additional tokens and retains 83.53 at 64K, despite the substantial amount of intervening text. Even after 128K additional tokens, the model retains an R@10 of 79.32, corresponding to more than 93% of its initial performance. These results demonstrate that the learned memory remains robust under substantial additional writes, while also reflecting the expected capacity limit of a fixed-size memory.
We will add these incremental-write and memory-capacity analyses to the appendix.
Thank you to the authors for addressing my concerns. I have updated my rating accordingly. Good luck!
Response to Reviewer U6vW [1/3]
Q1. Could the authors provide concrete examples of retrieved evidence and clarify how reliable the query-text-only annotations are for supporting the conclusions in Section 4.5?
We re-annotated all 131 queries for which exactly one of the two methods retrieves the complete evidence within the top 10 (63 solved only by TTT-Retrieval and 68 solved only by Fine-Tuning + Reranking). We used two annotation protocols: query-text-only and corpus-aware (with access to the gold evidence turns and the answer). Both protocols use the four aggregation types defined in Appendix E. Under both protocols, the share of aggregative queries is higher in the TTT-Retrieval-only group (query-text-only: 30.2% vs. 11.8%; corpus-aware: 36.5% vs. 14.7%).
[Table R1] Share of aggregative queries in each group.
Concrete retrieved evidence. We will add an appendix showing the top-ranked turns retrieved by both methods, with gold evidence marked, for six queries, including two where Fine-Tuning + Reranking outperforms TTT-Retrieval. For example, for "Which country has Tim visited most frequently?", TTT-Retrieval ranks Tim's three UK/London trips at positions 1, 3, and 6. In contrast, the reranker's top four turns contain travel-related mentions of places Tim has not visited (e.g., "Italy is definitely on my list" and "Adding NYC to my travel list"). For "What classes did Audrey take for her dogs?", TTT-Retrieval retrieves all three class mentions across three different sessions, whereas the reranker's top three turns concern the dogs' breeds despite their lexical similarity to the query.
A similar pattern emerges from a descriptive analysis of token overlap. Among queries solved only by TTT-Retrieval, the reranker's top-ranked non-gold turns have higher token overlap with the query than the gold turns recovered by TTT-Retrieval (0.414 vs. 0.330; p = 0.001). In the reverse direction, no comparable asymmetry is observed (0.371 vs. 0.376; p = 0.68). In 84% of these cases, the complete gold evidence is already present among the reranker's top-50 candidates, suggesting that the difference primarily reflects candidate ranking rather than initial retrieval. These observations are consistent with the pointwise cross-encoder favoring lexically similar distractors, while the corpus-adapted memory can surface evidence that is implicit or distributed across sessions. Conversely, TTT-Retrieval can fail when gold turns share exact terms with the query or when it confuses the target user with the conversation partner. Two such failure cases will also be included in the appendix.
In the revision, we will update Section 4.5 to clarify the annotation protocol and the scope of this analysis.
Q2. Just curious: Could the authors clarify whether the curriculum is primarily an optimization heuristic and how it affects learning the memory initialization?
Thank you for the question. The write-budget curriculum provides an optimization heuristic for learning the meta-learned memory initialization across different write budgets. By gradually increasing the write budget, training first establishes effective memory-update dynamics in smaller contexts and then extends them to larger contexts. This type of progressive training is also used in prior recurrent-memory work, where models are trained on shorter sequences before the sequence length or number of processed segments is increased [1].
Empirically, the curriculum improves R@1 on HotpotQA from 39.6 to 43.6 under a limited write budget compared with random budget sampling. This result indicates that the structured progression helps the initialization adapt more effectively across write budgets, rather than changing the memory mechanism itself.
[1] Rodkin, I., Kuratov, Y., Bulatov, A., & Burtsev, M. (2024). Associative recurrent memory transformer. arXiv preprint arXiv:2407.04841.
Q3. Could the authors clarify what is considered a retrieved document for Recall@k in each benchmark, particularly LoCoMo?
We follow the original LoCoMo setting (Maharana et al., 2024). The retrieval unit is a single dialog turn, meaning one speaker's utterance, identified by its dialog ID. Each query's gold evidence is the set of dialog turns annotated as evidence in the benchmark. We will add this to Appendix C.