acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Training of Contextual Embeddings

Abstract

Dense retrievers typically use fixed, context-independent embedding functions, limiting their ability to adapt to user- or corpus-specific semantics. We introduce **TTT-Retrieval**, a framework for *test-time training of contextual embeddings* that adapts a retriever directly to an unlabeled target corpus after deployment. TTT-Retrieval augments a frozen pretrained embedding model with compact, meta-trained neural memory modules. During a *write phase*, these memories are updated through self-supervision on the target corpus; during a *read phase*, they condition query embeddings on the learned corpus context. Because the adapted memories are retained across queries, adaptation cost is amortized rather than repeatedly incurred at inference time. Across personalized retrieval on LoCoMo, domain-specialized retrieval on QASPER, and open-domain retrieval on HotpotQA, TTT-Retrieval consistently improves over frozen embedding baselines and is competitive with or outperforms reranking approaches while requiring substantially less inference-time computation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Retrieval-augmented generation, Test-time training, Contextual embeddings

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_U6vWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TTT-Retrieval, a framework for corpus-conditioned retrieval that augments a frozen embedding backbone with lightweight, meta-trained neural memory modules. At test time, these modules are updated through self-supervised learning on an unlabeled target corpus. The resulting memory state conditions query and document embeddings and is reused across subsequent queries. Experiments on LoCoMo, QASPER, and HotpotQA demonstrate improvements over frozen and supervised fine-tuned baselines using Qwen3-Embedding-0.6B, with performance competitive with or superior to fine-tuning plus reranking. A controlled comparison using nomic-embed-text-v1 also shows improvements over prior Contextual Document Embeddings (CDE) approach. Overall, the approach is intuitive and demonstrates promising retrieval improvements. However, I have concerns about the breadth of evaluation, the computational efficiency when accounting for test-time adaptation, and the method’s applicability to growing or frequently updated corpora.

Critical strengths

Simple and effective approach to contextual retrieval. TTT-Retrieval provides an intuitive way to incorporate corpus-specific information into retrieval representations by adding lightweight neural memory modules to a frozen embedding backbone. The experiments demonstrate consistent improvements across three retrieval benchmarks, with competitive results against supervised fine-tuning and reranking baselines.

Generally clear methodology and informative empirical comparisons. The write/read mechanism and meta-learning formulation are generally well explained. The experimental comparisons and analyses are useful for understanding the method’s effectiveness, although some interpretations would benefit from further clarification.

Critical weaknesses

Limited evaluation breath and incomplete computational efficiency comparison. Although the paper includes a controlled comparison using nomic-embed-text-v1 (Figure 4), the main experimental comparisons primarily use Qwen3-Embedding-0.6B. Broader evaluation across embedding model architectures and model sizes would strengthen the evidence for generalizability. The evaluation also relies on Recall@k, where more traditional IR evaluation metrics like nDCG@k could provide additional insight into ranking quality. More importantly, the efficiency comparison in Figure 3 considers only read-time FLOPs after corpus adaptation. They do not account for the computation required to write the corpus into memory. Reporting write-time FLOPs and amortized computation as a function of the number of queries would provide a more complete picture of deployment efficiency.

Unclear scalability and applicability to evolving corpora. The method’s efficacy when documents are incrementally added to an existing corpus remains insufficiently evaluated. It is unclear whether updates to the adapted memory require previously indexed documents to be re-embedded to maintain a consistent retrieval space. Furthermore, since the neural memory has a fixed parameter budget, it would be helpful to understand how retrieval performance changes as the number of documents grows. I think these are especially relevant to frequently updated retrieval environments in current days, such as codebases. While the approach appears well suited to relatively static corpora like long-doc QA, its applicability to dynamic retrieval settings remain unclear.

Questions for authors

Qualitative retrieval analysis and the reliability of query-type annotations (Section 4.5). Table 3 lists representative queries where TTT-Retrieval succeeds but Fine-Tuning + Reranking fails. However, without examples of the actual retrieved evidence, it is difficult to understand what information TTT-Retrieval captures more effectively. Additionally, Appendix E indicates that aggregative queries are annotated using only query text. This may not reliably distinguish queries requiring evidence aggregation from those answerable using a single passage, since the distinction depends on the underlying corpus. Could the authors provide concrete examples of retrieved evidence and clarify how reliable the query-text-only annotations are for supporting the conclusions in Section 4.5?

Motivation and empirical necessity of the write-budget curriculum (Section 3.5). The paper motivates the curriculum as a way to improve training stability and tractability. However, it remains unclear whether this design is empirically necessary for effective meta-learning. Just curious: did the authors conduct an ablation study for meta-learning without curriculum? Could the authors clarify whether the curriculum is primarily an optimization heuristic and how it affects learning the memory initialization ϕ0\phi_0?

Retrieval granularity. Could the authors clarify what is considered a retrieved document for Recall@k in each benchmark, particularly LoCoMo? Appendix C specifies paragraph-level retrieval for QASPER and document-level retrieval for HotpotQA, but the retrieval granularity for LoCoMo is less clear.

4 comments
AuthorsOpenReview ↗

Response to Reviewer U6vW [1/3]

Q1. Could the authors provide concrete examples of retrieved evidence and clarify how reliable the query-text-only annotations are for supporting the conclusions in Section 4.5?

We re-annotated all 131 queries for which exactly one of the two methods retrieves the complete evidence within the top 10 (63 solved only by TTT-Retrieval and 68 solved only by Fine-Tuning + Reranking). We used two annotation protocols: query-text-only and corpus-aware (with access to the gold evidence turns and the answer). Both protocols use the four aggregation types defined in Appendix E. Under both protocols, the share of aggregative queries is higher in the TTT-Retrieval-only group (query-text-only: 30.2% vs. 11.8%; corpus-aware: 36.5% vs. 14.7%).

[Table R1] Share of aggregative queries in each group.

ProtocolTTT-Retrieval only (n=63)Fine-Tuning + Reranking only (n=68)
Query-text-only30.2% (19/63)11.8% (8/68)
Corpus-aware36.5% (23/63)14.7% (10/68)

Concrete retrieved evidence. We will add an appendix showing the top-ranked turns retrieved by both methods, with gold evidence marked, for six queries, including two where Fine-Tuning + Reranking outperforms TTT-Retrieval. For example, for "Which country has Tim visited most frequently?", TTT-Retrieval ranks Tim's three UK/London trips at positions 1, 3, and 6. In contrast, the reranker's top four turns contain travel-related mentions of places Tim has not visited (e.g., "Italy is definitely on my list" and "Adding NYC to my travel list"). For "What classes did Audrey take for her dogs?", TTT-Retrieval retrieves all three class mentions across three different sessions, whereas the reranker's top three turns concern the dogs' breeds despite their lexical similarity to the query.

A similar pattern emerges from a descriptive analysis of token overlap. Among queries solved only by TTT-Retrieval, the reranker's top-ranked non-gold turns have higher token overlap with the query than the gold turns recovered by TTT-Retrieval (0.414 vs. 0.330; p = 0.001). In the reverse direction, no comparable asymmetry is observed (0.371 vs. 0.376; p = 0.68). In 84% of these cases, the complete gold evidence is already present among the reranker's top-50 candidates, suggesting that the difference primarily reflects candidate ranking rather than initial retrieval. These observations are consistent with the pointwise cross-encoder favoring lexically similar distractors, while the corpus-adapted memory can surface evidence that is implicit or distributed across sessions. Conversely, TTT-Retrieval can fail when gold turns share exact terms with the query or when it confuses the target user with the conversation partner. Two such failure cases will also be included in the appendix.

In the revision, we will update Section 4.5 to clarify the annotation protocol and the scope of this analysis.


Q2. Just curious: Could the authors clarify whether the curriculum is primarily an optimization heuristic and how it affects learning the memory initialization?

Thank you for the question. The write-budget curriculum provides an optimization heuristic for learning the meta-learned memory initialization across different write budgets. By gradually increasing the write budget, training first establishes effective memory-update dynamics in smaller contexts and then extends them to larger contexts. This type of progressive training is also used in prior recurrent-memory work, where models are trained on shorter sequences before the sequence length or number of processed segments is increased [1].

Empirically, the curriculum improves R@1 on HotpotQA from 39.6 to 43.6 under a limited write budget compared with random budget sampling. This result indicates that the structured progression helps the initialization adapt more effectively across write budgets, rather than changing the memory mechanism itself.

[1] Rodkin, I., Kuratov, Y., Bulatov, A., & Burtsev, M. (2024). Associative recurrent memory transformer. arXiv preprint arXiv:2407.04841.


Q3. Could the authors clarify what is considered a retrieved document for Recall@k in each benchmark, particularly LoCoMo?

We follow the original LoCoMo setting (Maharana et al., 2024). The retrieval unit is a single dialog turn, meaning one speaker's utterance, identified by its dialog ID. Each query's gold evidence is the set of dialog turns annotated as evidence in the benchmark. We will add this to Appendix C.


AuthorsOpenReview ↗

Response to Reviewer U6vW [2/3]

W1-1. Broader evaluation across embedding model architectures and model sizes would strengthen the evidence for generalizability.

Thank you for the suggestion. In addition to the Qwen3 embedding backbone used in the main experiments (decoder-based, 0.6B), we evaluated TTT-Retrieval on ModernBERT-base (encoder-only, 149M). Trained from the same checkpoint on the same task data, TTT-Retrieval outperforms CDE-v2 on all three datasets (Table R2).

[Table R2] Comparison on ModernBERT-base (R@10, %).

MethodLoCoMoQASPERHotpotQA
CDE-v276.376.955.6
TTT-Retrieval (Ours)80.980.363.2

These results show that the gains are not specific to one backbone architecture or size. We will add them to the main paper.


W1-2. The evaluation also relies on Recall@k, where more traditional IR evaluation metrics like nDCG@k could provide additional insight into ranking quality.

Thank you for the suggestion. Table R3 adds nDCG@10 on QASPER, together with Fine-Tuning without reranking as a non-reranking baseline. TTT-Retrieval improves Recall@10 and nDCG@10 over Fine-Tuning by 4.7 and 6.3 percentage points, respectively, and over Fine-Tuning + Reranking by 8.8 and 10.1 percentage points. The improvements in nDCG@10 complement the Recall@10 results, showing gains in a rank-sensitive retrieval metric as well as evidence coverage.

[Table R3] Recall@10 and nDCG@10 on QASPER (%).

MethodRecall@10nDCG@10
Fine-Tuning76.762.0
Fine-Tuning + Reranking72.658.2
TTT-Retrieval (Ours)81.468.3

We will report nDCG@10 for all datasets and methods in the revision.


W1-3. Reporting write-time FLOPs and amortized computation as a function of the number of queries would provide a more complete picture of deployment efficiency.

We thank the reviewer for this question.

Break-even against reranking. Both methods encode documents, so the shared encoding cost cancels. The additional preprocessing cost of TTT-Retrieval is the write cost plus the extra fast-weight computation in document embedding. For a 32K-token corpus, this is 41.21 TFLOPs (38.15 for writing and 3.06 for the extra document-embedding computation). Each query then saves 15.2 TFLOPs relative to Base + Reranking, net of TTT-Retrieval's slightly higher query-embedding cost. Table R4 shows the cumulative effect.

[Table R4] Cumulative compute of TTT-Retrieval relative to Reranking (32K-token corpus).

Number of queriesCumulative query-time savings (TFLOPs)One-time preprocessing per corpus (TFLOPs)Net saving (TFLOPs)
115.241.2−26.0
576.041.2+34.8
10152.141.2+110.9
50760.441.2+719.2

TTT-Retrieval recovers its additional preprocessing cost after 2.71 queries for this corpus (i.e., by the third query). The preprocessing cost is amortized over repeated queries on the same corpus version and can run outside the online query path.


AuthorsOpenReview ↗

Response to Reviewer U6vW [3/3]

W2. (a) It is unclear whether updates to the adapted memory require previously indexed documents to be re-embedded to maintain a consistent retrieval space. (b) Since the neural memory has a fixed parameter budget, it would be helpful to understand how retrieval performance changes as the number of documents grows.

Thank you for these questions. Both concern how retrieval behaves as more content is written into memory, so we address them together with two experiments on LoCoMo.

Incremental writes and re-embedding (W2-a). Our experiments suggest that previously indexed embeddings remain usable without immediate re-embedding, although refreshing the index generally improves retrieval performance.

To test this, we write each LoCoMo conversation as a single stream and snapshot the memory after writing 1K, 2K, 4K, and 8K tokens, as well as after the full conversation. At each stage, queries are embedded using the current memory state and scored against all documents seen so far. We compare two settings: Refresh, our default setting, re-embeds all documents using the current memory state, whereas Stale embeds only newly arrived documents and retains previously computed embeddings. The two settings are identical at the first stage because all documents are new.

[Table R5] R@10 (%) under incremental writes on LoCoMo.

Write budget (tokens)#QueriesStaleRefresh
10245498.198.1
204810392.793.7
409619293.293.5
819237791.892.4
Full154083.784.7

Refresh matches or outperforms Stale at every stage, with Stale remaining within 1.0 R@10 point of Refresh. These results suggest that previously computed embeddings remain largely compatible with the updated memory, making Stale a lower-cost alternative when re-embedding the full index after every update is expensive.

Memory capacity (W2-b). In Table R5, both the document pool and the query set grow with the write budget. To isolate the effect of additional writes on a fixed-size memory, we keep the LoCoMo query set and candidate documents fixed while varying only the amount of written text. Specifically, we first write each LoCoMo conversation and then append 4K to 128K tokens of PG-19 text. After all writes, we re-embed the fixed queries and documents using the final memory state (Table R6).

[Table R6] LoCoMo R@10 (%) after additionally writing PG-19 text.

Additional PG-19 tokens04K8K16K32K64K128K
TTT-Retrieval (Ours), R@1084.7584.5084.5584.3584.1983.5379.32

TTT-Retrieval maintains stable retrieval performance as additional text is written to the fixed-size memory. R@10 remains above 84% through 32K additional tokens and retains 83.53 at 64K, despite the substantial amount of intervening text. Even after 128K additional tokens, the model retains an R@10 of 79.32, corresponding to more than 93% of its initial performance. These results demonstrate that the learned memory remains robust under substantial additional writes, while also reflecting the expected capacity limit of a fixed-size memory.

We will add these incremental-write and memory-capacity analyses to the appendix.

Thank you to the authors for addressing my concerns. I have updated my rating accordingly. Good luck!

Reviewer_pxrWOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TTT-Retrieval, a framework for corpus-conditioned embeddings through test-time training. It augments a frozen embedding backbone with meta-trained neural memory modules, which are updated on an unlabeled target corpus and reused for subsequent retrieval. Experiments on LoCoMo, QASPER, and HotpotQA report substantial improvements over frozen retrievers and competitive performance against fine-tuning and reranking baselines.

Critical strengths
  • Corpus-specific retrieval adaptation is a meaningful problem, particularly for personalized and domain-specific retrieval.
  • The write-once, read-many design is potentially useful for amortizing adaptation costs across queries.
  • The reported retrieval improvements are substantial, with relatively low query-time computational overhead.
Critical weaknesses
  1. The necessity of the proposed approach is not established. The method combines existing neural memory architectures with meta-learning and test-time adaptation, but it is unclear why such a complex mechanism is needed. Comparisons with simpler alternatives, such as corpus-level LoRA adaptation, synthetic-query-based adaptation (e.g., GPL), and query rewriting or hybrid retrieval, are missing.

  2. Critical ablations are missing. The paper does not adequately isolate the contributions of meta-training, test-time memory updates, and the memory architecture. In particular, evaluating the meta-trained memory without test-time writing is essential to determine whether the proposed TTT mechanism actually drives the improvements.

  3. The efficiency comparison is incomplete. The claimed approximately 329× FLOPs reduction only considers query-time computation after corpus adaptation, excluding memory writing, document re-encoding, and index construction. Without end-to-end costs or a break-even analysis, the practical efficiency advantage remains unclear, especially for corpora with few queries or frequent updates.

  4. The experimental evidence is insufficient. LoCoMo contains only 10 test corpora, yet no corpus-level variance or statistical significance is reported. HotpotQA uses constructed 200-document corpora rather than a realistic large-scale retrieval setting. The comparisons with CDE also require clearer control of training conditions and adaptation budgets.

  5. There are inconsistencies in the experimental reporting. Section 4.5 reports aggregative-query percentages of 28.6% versus 10.3%, whereas Appendix E reports 38.1% versus 16.2% for the corresponding groups. The HotpotQA episode/query descriptions also appear inconsistent between the main text and appendix. These discrepancies raise concerns about the reliability of the reported analysis.

Questions for authors
  1. How much improvement remains without test-time memory writing? How does the method compare with standard corpus-level LoRA adaptation under comparable training budgets?

  2. Why is neural memory-based TTT preferable to simpler approaches such as GPL, query rewriting, or hybrid retrieval? Can the authors provide controlled comparisons?

  3. What are the total adaptation and indexing costs? How many queries per corpus are needed to amortize these costs?

  4. Can the authors clarify the inconsistent aggregative-query statistics and HotpotQA evaluation protocol, and report corpus-level variance for the main results?

5 comments
AuthorsOpenReview ↗

Response to Reviewer pxrW [1/3]

Q1 / W2. How much improvement remains without test-time memory writing?

We thank the reviewer for this important question. As suggested, we evaluate the trained model with no write and with unrelated corpora written into memory (Table R1).

[Table R1]

Written corpusR@10
No write65.1
Unrelated corpus (PG-19 novels)77.6
Unrelated corpus (LoCoMo dialogues)77.7
Ours (related corpus)81.4

To clarify, each memory head is a SwiGLU network MW(z)=W1(SiLU(W0z)⊙W2z)\mathcal{M}{W}(\mathbf{z}) = W_1\big(\mathrm{SiLU}(W_0\mathbf{z}) \odot W_2\mathbf{z}\big), whose fast weights start from ϕ0\phi_0 and are updated by the test-time write. In ϕ0\phi_0, W1W_1 is initialized to zero and is not meta-learned. This design prevents the memory from perturbing the frozen backbone before a write, so that its output is formed only through test-time writing. Before any write, the memory head outputs zero and the model is exactly the frozen backbone, i.e., fθ,ϕ0(x)=fθ(x)f{\theta,\phi_0}(x)=f_\theta(x).

To isolate the effect of the written content, we write an unrelated corpus whose token count matches that of the target paper for each QASPER test episode. The performance gain from writing unrelated context suggests that test-time writing serves a role beyond incorporating corpus-specific information. Since the memory output is initially zero, writing makes the meta-trained memory transformation active by updating W1W_1. We hypothesize that this activation explains much of the improvement from unrelated writing, while the additional 3.7–3.8-point gain from related writing reflects the benefit of adapting to the target corpus.

We will add Table R1 and this discussion to the revised version.


Q2 / W1. Why is neural memory-based TTT preferable to simpler approaches such as GPL, query rewriting, or hybrid retrieval? Can the authors provide controlled comparisons?

Thank you for the suggestion. We added the requested comparisons under the same evaluation protocol on LoCoMo, QASPER, and HotpotQA, using the same Qwen3 embedding backbone.

  1. Corpus-level adaptation (GPL [1]). We used Qwen3-4B to generate synthetic queries, then applied hard-negative mining and pseudo-labeling. We tested GPL-LoRA (12.8M trainable parameters) and GPL-Full (596M), each over several training lengths, and report the best setting.
  2. Query transformation (HyDE [2]). We used Qwen3-4B to generate one or four hypothetical documents per query. We tested these alone and averaged with the original query embedding, and report the best setting.
  3. Hybrid retrieval. We combine BM25 and dense retrieval using Reciprocal Rank Fusion (RRF).

[Table R2] Retrieval performance (R@10, %) and compute cost.

MethodLoCoMoQASPERHotpotQAAdaptation compute (TFLOPs, LoCoMo)Query-time compute (GFLOPs/query)
Base54.365.172.5–41.7
RRF60.264.681.8–n/a†
GPL-Full64.663.462.31,72241.7
GPL-LoRA74.769.874.81,72241.7
HyDE70.968.274.0–1,040
Ours84.781.488.352.246.1

† Not reported: RRF combines CPU-based BM25 with dense retrieval, so its cost is not directly comparable in FLOPs.

Our method outperforms the strongest alternative on every dataset, by +10.0, +11.6, and +6.5 points on LoCoMo, QASPER, and HotpotQA. The alternatives, by contrast, are limited or inconsistent. GPL-LoRA and HyDE give only modest gains outside LoCoMo (at most +4.7 on QASPER and +2.3 on HotpotQA). GPL-Full even underperforms Base on QASPER and HotpotQA, as does RRF on QASPER. Our method, in contrast, improves over Base by +15.8 to +30.4 points on all three datasets.

These gains come at a low cost. Corpus adaptation requires about 33× less compute than GPL, and query-time compute is about 23× lower than HyDE, which runs an extra LLM generation for every query. We thank the reviewer for this suggestion, which has strengthened the paper, and we will include these results in the revision.

[1] Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022. GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2345–2360, Seattle, United States. Association for Computational Linguistics.

[2] Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Precise Zero-Shot Dense Retrieval without Relevance Labels. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1762–1777, Toronto, Canada. Association for Computational Linguistics.


AuthorsOpenReview ↗

Response to Reviewer pxrW [2/3]

Q3 / W3. How many queries per corpus are needed to amortize these costs?

TTT-Retrieval recovers its additional preprocessing cost after 2.71 queries for this corpus (i.e., by the third query). The preprocessing cost is amortized over repeated queries on the same corpus version and can run outside the online query path.

Break-even against reranking. Both methods encode documents, so the shared encoding cost cancels. The additional preprocessing cost of TTT-Retrieval is the write cost plus the extra fast-weight computation in document embedding. For a 32K corpus, this is 41.21 TFLOPs (38.15 for writing and 3.06 for the extra document-embedding computation). Each query then saves 15.2 TFLOPs relative to Base + Reranking, net of TTT's slightly higher query-embedding cost. Table R3 shows the cumulative effect.

[Table R3] Cumulative compute of TTT-Retrieval relative to Base + Reranking (32K token corpus).

Number of queriesCumulative query-time savings (TFLOPs)One-time preprocessing per corpus (TFLOPs)Net saving (TFLOPs)
115.241.2−26.0
576.041.2+34.8
10152.141.2+110.9
50760.441.2+719.2

Q4-1 / W5. Can the authors clarify the inconsistent aggregative-query statistics?

Thank you for pointing out this inconsistency. It arises because Section 4.5 and Appendix E report two different versions of our annotation (28.6% [18/63] vs. 10.3% [7/68] and 38.1% [24/63] vs. 16.2% [11/68], respectively). The qualitative trend is the same in both. Both are computed over the same two groups of LoCoMo queries on which exactly one method retrieves the complete evidence in the top 10: 63 queries solved only by TTT-Retrieval and 68 solved only by Fine-Tuning + Reranking. Under both versions, the share of aggregative queries is higher in the TTT-Retrieval-only group, by a similar ratio (2.8× and 2.4×).

To confirm, we re-annotated all 131 queries independently under two protocols, query-text-only and corpus-aware, as detailed in our response to Reviewer U6vW (Q1, Table R1 there). The share of aggregative queries is again higher in the TTT-Retrieval-only group under both protocols:

[Table R4] Share of aggregative queries in each group.

ProtocolTTT-Retrieval only (n=63)Fine-Tuning + Reranking only (n=68)
Query-text-only30.2% (19/63)11.8% (8/68)
Corpus-aware36.5% (23/63)14.7% (10/68)

We will revise Section 4.5 and Appendix E to report these statistics under a single, consistent annotation protocol and state the protocol explicitly in both places.


Q4-2 / W5. HotpotQA evaluation protocol, and report corpus-level variance for the main results.

We thank the reviewer for this question.

  • Corpus construction. Each test query forms its own episode with a query-specific 200-document corpus (Lines 264–265, 763–764): the 10 paragraphs attached to the query (gold supporting paragraphs and distractors), plus 190 paragraphs randomly sampled from the contexts of other questions.
  • Uncertainty. We report 95% bootstrap CIs (10k resamples over queries) for each method and for the paired per-query difference.

[Table R5]

MethodR@10 (95% CI)R@20 (95% CI)Read-time FLOPs / queryPeak GPU memory
Fine-Tuning + Reranking87.8 [82.8, 92.2]89.5 [85.0, 93.5]15,181G5.2 GB
TTT-Retrieval (Ours)88.3 [83.0, 93.0]90.3 [85.5, 94.5]46.1G3.2 GB
Ours − (FT + Reranking)+0.5 [−3.0, +4.0]+0.8 [−3.0, +4.2]329×329\times fewer1.6×1.6\times lower

The paired-difference CIs include zero, so the two methods are comparable in retrieval quality. We will therefore revise "slightly exceeding" to "comparable to" for R@10/20.

At this comparable accuracy, TTT-Retrieval (Ours) is far cheaper at query time. Ours removes the cross-encoder, reducing read-time FLOPs by 329×329\times (46.1G vs. 15,181G per query) and peak GPU memory from 5.2 GB to 3.2 GB (Figure 3, where Base + Reranking has the same cost because both use the same cross-encoder). We will emphasize this efficiency as our main advantage. The write cost is addressed in our response to Q4. We will report CIs for all main results.


AuthorsOpenReview ↗

Response to Reviewer pxrW [3/3]

W4-1. LoCoMo contains only 10 test corpora, yet no corpus-level variance or statistical significance is reported.

Thank you for the suggestion. The 10 LoCoMo test conversations cover distinct topics (e.g., dance studio, military veterans, programming and gaming, and yoga), so we treat each conversation as a separate corpus. We report pooled R@10 together with 95% cluster-bootstrap CIs obtained by resampling conversations rather than individual queries (10,000 resamples; Table R6). We additionally report the results for each conversation and conduct paired significance tests over the 10 conversations (Table R7).

[Table R6] LoCoMo R@10 with conversation-clustered uncertainty (%, pooled over queries).

MethodR@10 (95% cluster-bootstrap CI)
Fine-Tuning + Reranking83.9 [82.0, 85.6]
TTT-Retrieval (Ours)84.7 [82.8, 86.5]

The marginal CIs in Table R6 overlap because the conversations vary in difficulty, with R@10 ranging from approximately 77% to 89%, and this variation affects both methods similarly. We therefore also perform a paired comparison using each conversation as the unit of analysis.

[Table R7] Per-conversation R@10 (%).

Conversationconv-26conv-30conv-41conv-42conv-43conv-44conv-47conv-48conv-49conv-50
Fine-Tuning + Reranking82.8980.3688.6582.2485.2177.4087.3085.5581.9484.24
TTT-Retrieval (Ours)82.4482.7989.4382.7185.9477.8788.8085.2584.4086.01
Δ (Ours − FT + Reranking)−0.45+2.43+0.78+0.47+0.73+0.47+1.50−0.30+2.46+1.77

TTT-Retrieval outperforms Fine-Tuning + Reranking on 8 of the 10 conversations. The mean per-conversation paired gain is +1.0 R@10 points, and a paired Wilcoxon signed-rank test over the 10 conversations yields p = 0.0098. A paired t-test gives a consistent result (p = 0.014). The query-weighted pooled gain is +0.9 points, with a 95% conversation-cluster-bootstrap CI of [+0.3, +1.5].

We will add these corpus-level uncertainty estimates and paired significance tests to the appendix.


W4-2. The comparisons with CDE also require clearer control of training conditions and adaptation budgets.

Thank you for the comment. To control for training conditions, we trained CDE-v2 and TTT-Retrieval from the same ModernBERT-base checkpoint on the same task-training data, so the comparison matches the backbone architecture, the initial backbone weights, and the training data. CDE-v2 was fully fine-tuned, while TTT-Retrieval kept the backbone frozen and trained only the memory-related parameters. Under these matched conditions, TTT-Retrieval outperforms CDE-v2 by +4.6, +3.4, and +7.6 R@10 points on LoCoMo, QASPER, and HotpotQA (Table R10).

Table R8. Controlled comparison with CDE-v2 (R@10, %).

MethodLoCoMoQASPERHotpotQA
CDE-v276.376.955.6
TTT-Retrieval (Ours)80.980.363.2

We will add these results to Section 4.4.

Reviewer_j9GWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes TTT-Retrieval, which adapts dense retrieval embeddings to an unlabeled target corpus using fast-weight neural memory. The memory modules are added to a frozen embedding backbone and meta-trained with a contrastive retrieval objective. At deployment, self-supervised corpus writes update the memory, and the resulting state is reused to encode documents and queries. Experiments on LoCoMo, QASPER, and a constructed HotpotQA retrieval setting report improvements over frozen retrievers and supervised fine-tuning, with low query-time computation after corpus writing.

Critical strengths
  1. The proposed corpus adaptation is well-motivated, especially for personalized retrieval. The adapted memory can be reused across queries without repeated corpus writes.

  2. The reported gains over frozen retrievers are substantial across three retrieval settings. The comparison with supervised fine-tuning uses the same training datasets and splits.

Critical weaknesses
  1. The contribution of corpus-specific adaptation is not isolated. There is no ablation with no test-time write or with an unrelated corpus, so gains from corpus writing cannot be separated from additional memory parameters and supervised meta-training. The comparison with CDE, a closely related contextual embedding method, also needs clarification. Section 4.4 claims a comparison using the same nomic backbone, but the released CDE-v2 uses ModernBERT [1]. A comparison with matched backbones is needed to substantiate the claimed advantage over CDE.

  2. The efficiency analysis excludes corpus writing and document re-encoding. This omission matters because HotpotQA uses 100 separate 200-document test episodes with one query each, while QASPER averages only about three queries per episode. The reported 329-fold reduction in read-time FLOPs therefore leaves the total retrieval cost unclear. Appendix B also omits the claimed test-time write configurations, limiting reproducibility and assessment of how effectively writing costs are amortized.

[1] CDE-v2 official model card, Model details: https://huggingface.co/jxm/cde-small-v2#model-details

Questions for authors
  1. How does the trained model perform with no test-time write or with an unrelated corpus written into memory? How does the same architecture perform when trained without corpus writes?

  2. Which exact checkpoints and backbones are used in Figure 4? Please clarify the claimed backbone matching given CDE-v2's ModernBERT backbone [1], and provide a comparison with matched backbones. An additional comparison after training CDE on the same task data would further strengthen the analysis.

  3. Could you specify how LoCoMo conversations are split into retrieval units and how the 200-document HotpotQA corpora are constructed? Please report uncertainty for the small gains over Fine-Tuning + Reranking, especially on the 100-query HotpotQA subset.

  4. Please provide the write learning rates, block sizes, corpus passes, document ordering, and memory-layer placement. What is the total write and document-encoding cost, including peak memory, and how many queries are needed to offset this cost relative to reranking?

[1] CDE-v2 official model card, Model details: https://huggingface.co/jxm/cde-small-v2#model-details

3 comments
AuthorsOpenReview ↗

Response to Reviewer j9GW [1/2]

Q1 / W2. How does the trained model perform with no test-time write or with an unrelated corpus written into memory?

We thank the reviewer for this important question. As suggested, we evaluate the trained model with no write and with unrelated corpora written into memory (Table R1).

[Table R1]

Written corpusR@10
No write65.1
Unrelated corpus (PG-19 novels)77.6
Unrelated corpus (LoCoMo dialogues)77.7
Ours (related corpus)81.4

To clarify, each memory head is a SwiGLU network M(z)=W1(SiLU(W0z)⊙W2z)\mathcal{M}(\mathbf{z}) = W_1\big(\mathrm{SiLU}(W_0\mathbf{z}) \odot W_2\mathbf{z}\big), whose fast weights start from ϕ0\phi_0 and are updated by the test-time write. In ϕ0\phi_0, W1W_1 is initialized to zero and is not meta-learned. This design prevents the memory from perturbing the frozen backbone before a write, so that its output is formed only through test-time writing. Before any write, the memory head outputs zero and the model is exactly the frozen backbone, i.e., fθ,ϕ0(x)=fθ(x)f_{\theta,\phi_0}(x)=f_\theta(x).

To isolate the effect of the written content, we write an unrelated corpus whose token count matches that of the target paper for each QASPER test episode. The performance gain from writing unrelated context suggests that test-time writing serves a role beyond incorporating corpus-specific information. Since the memory output is initially zero, writing makes the meta-trained memory transformation active by updating W1W_1. We hypothesize that this activation explains much of the improvement from unrelated writing, while the additional 3.7–3.8-point gain from related writing reflects the benefit of adapting to the target corpus.

We will add Table R1 and this discussion to the revised version.


Q2 / W1. Please clarify the claimed backbone matching given CDE-v2's ModernBERT backbone [1], and provide a comparison with matched backbones. An additional comparison after training CDE on the same task data would further strengthen the analysis.

As the reviewer points out, cde-small-v2 uses ModernBERT [1]. The matched-backbone statement in Section 4.4 was based on the original CDE paper, which describes only cde-small-v1 with a NomicBERT backbone, and therefore does not apply to cde-small-v2.

To provide the comparison the reviewer requested, we trained CDE-v2 and TTT-Retrieval from the same ModernBERT-base checkpoint on the same task-training data. The two methods therefore share the backbone architecture, the initial backbone weights, and the training data. CDE-v2 is fully fine-tuned, while TTT-Retrieval keeps the backbone frozen and trains only the memory-related parameters.

[Table R2] R@10 with matched backbone and task-training data.

MethodLoCoMoQASPERHotpotQA
CDE-v276.376.955.6
TTT-Retrieval (Ours)80.980.363.2

Under this matched setting, TTT-Retrieval outperforms CDE-v2 by 4.6, 3.4, and 7.6 R@10 points on LoCoMo, QASPER, and HotpotQA, respectively, even though CDE-v2 updates the full backbone and TTT-Retrieval does not. In the revision, we will correct the backbone description in Section 4.4, specify the backbone of each checkpoint in Figure 4, and add Table R2 with the training details. We thank the reviewer for this suggestion, which strengthens the comparison.

[1] https://huggingface.co/answerdotai/ModernBERT-base


Q3-1. Could you specify how LoCoMo conversations are split into retrieval units?

We follow the original LoCoMo setting (Maharana et al., 2024). The retrieval unit is a single dialog turn (one speaker's utterance), identified by its dialog ID. Each conversation forms one corpus (avg. 588 turns, max 689 turns), and each query's gold set is the set of turns annotated as evidence in the benchmark. We exclude the adversarial category because it has no evidence turns. We will add these details to Appendix C.

AuthorsOpenReview ↗

Response to Reviewer j9GW [2/2]

Q3-2. How the 200-document HotpotQA corpora are constructed? Please report uncertainty for the small gains over Fine-Tuning + Reranking, especially on the 100-query HotpotQA subset.

We thank the reviewer for this question.

  • Corpus construction. Each test query forms its own episode with a query-specific 200-document corpus (Lines 264–265, 763–764): the 10 paragraphs attached to the query (gold supporting paragraphs and distractors), plus 190 paragraphs randomly sampled from the contexts of other questions.
  • Uncertainty. We report 95% bootstrap CIs (10k resamples over queries) for each method and for the paired per-query difference.

[Table R3]

MethodR@10 (95% CI)R@20 (95% CI)Read-time FLOPs / queryPeak GPU memory
Fine-Tuning + Reranking87.8 [82.8, 92.2]89.5 [85.0, 93.5]15,181G5.2 GB
TTT-Retrieval (Ours)88.3 [83.0, 93.0]90.3 [85.5, 94.5]46.1G3.2 GB
Ours − (FT + Reranking)+0.5 [−3.0, +4.0]+0.8 [−3.0, +4.2]329×329\times fewer1.6×1.6\times lower

The paired-difference CIs include zero, so the two methods are comparable in retrieval quality. We will therefore revise "slightly exceeding" to "comparable to" for R@10/20.

At this comparable accuracy, TTT-Retrieval (Ours) is far cheaper at query time. Ours removes the cross-encoder, reducing read-time FLOPs by 329×329\times (46.1G vs. 15,181G per query) and peak GPU memory from 5.2 GB to 3.2 GB (Figure 3, where Base + Reranking has the same cost because both use the same cross-encoder). We will emphasize this efficiency as our main advantage. The write cost is addressed in our response to Q4. We will report CIs for all main results.


Q4. Please provide the write learning rates, block sizes, corpus passes, document ordering, and memory-layer placement. What is the total write and document-encoding cost, including peak memory, and how many queries are needed to offset this cost relative to reranking?

We thank the reviewer for this question. We will add the write configuration to Appendix B and the cost analysis alongside Figure 3.

[Table R4] Write configuration.

ItemValue
Write learning rate, η0.01 for all datasets
Write block size1,024 tokens
Corpus passes1
Document orderingChronological (LoCoMo), original document order (QASPER), random (HotpotQA)
Memory-layer placementAll 28 transformer layers

Write cost and peak memory. With the setting (block size 1,024 tokens, batch size 1), writing a 32K-token corpus costs 38.15 TFLOPs. Peak GPU memory stays at 3.20 GB because writing is processed blockwise.

Break-even against reranking. Both methods encode documents, so the shared encoding cost cancels. The additional preprocessing cost of TTT-Retrieval is the write cost plus the extra fast-weight computation in document embedding. For a 32K-token corpus, this is 41.21 TFLOPs (38.15 for writing and 3.06 for the extra document-embedding computation). Each query then saves 15.2 TFLOPs relative to Base + Reranking, net of TTT-Retrieval's slightly higher query-embedding cost. Table R5 shows the cumulative effect.

[Table R5] Cumulative compute of TTT-Retrieval relative to Reranking (32K-token corpus).

Number of queriesCumulative query-time savings (TFLOPs)One-time preprocessing per corpus (TFLOPs)Net saving (TFLOPs)
115.241.2−26.0
576.041.2+34.8
10152.141.2+110.9
50760.441.2+719.2

TTT-Retrieval recovers its additional preprocessing cost after 2.71 queries for this corpus (i.e., by the third query). The preprocessing cost is amortized over repeated queries on the same corpus version and can run outside the online query path.

Document encoding and indexing. For a static corpus, Base + Reranking also encodes the documents and builds an index, so these costs are shared. The only extra document-side cost of TTT-Retrieval is the fast-weight computation during document embedding, which is already included in the preprocessing cost above (3.06 TFLOPs for a 32K-token corpus). The document embeddings have the same dimensionality and are stored in the same index as in the baseline, so index construction and storage costs are unchanged.

Dear Reviewer,

Thank you again for your valuable comments.

As the discussion period is less than 3h, we would like to kindly follow up on our previous responses. We hope our detailed explanations have adequately addressed your concerns.

Please let us know if you need any further clarification.

Comments on the paper

1 comment

We thank the AC for their efforts in handling a heavy reviewing load this year.

Dense retrievers use fixed, context-independent embeddings and cannot adapt to a specific user or corpus after deployment. Existing remedies are costly: fine-tuning needs labels, reranking adds heavy per-query compute, and GPL-style adaptation requires LLM-generated data and retraining. Contextual embeddings such as CDE condition on a fixed-size sample of corpus documents as additional input, but do not learn from the corpus at test time.

We propose TTT-Retrieval, which, to our knowledge, is the first to frame corpus adaptation for dense retrieval as test-time training: compact, meta-trained neural memory attached to a frozen embedding model learns directly from the unlabeled target corpus via self-supervised writes, and is reused across all subsequent queries, so adaptation cost is paid once and amortized.

The reviewers find corpus adaptation well-motivated and meaningful (j9GW, pxrW), the write-once, read-many design practically useful (j9GW, pxrW), the approach simple and effective (U6vW), and the gains substantial at low query-time cost (j9GW, pxrW, U6vW).

During the discussion, we added a range of new experiments, after which Reviewers U6vW and pxrW both raised their scores. Reviewer j9GW did not respond, but their main concerns overlap with pxrW's and were addressed by the same experiments:

  • Corpus-specific gain (j9GW, pxrW): Writing the target corpus outperforms writing an unrelated corpus by +3.8 (QASPER) and +5.0 (LoCoMo) R@10, beyond merely activating the memory.
  • Simpler alternatives (pxrW): Outperforms GPL, HyDE, and hybrid retrieval by +6.5 to +11.6 R@10 on all datasets, with ~33× less adaptation compute than GPL.
  • Matched comparison with CDE (j9GW, pxrW): With the same backbone and training data, outperforms fully fine-tuned CDE-v2 by +3.4 to +7.6 R@10 while keeping the backbone frozen.
  • End-to-end cost (all): Including write cost, TTT-Retrieval breaks even with reranking by the third query.
  • Statistical reliability (j9GW, pxrW): Significant gain on LoCoMo (8/10 conversations, Wilcoxon p = 0.0098); reporting inconsistencies are resolved.

We will incorporate all results and clarifications into the final version. Thank you once again for your consideration.

Best regards, Authors of Submission 45780

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.