acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Memory Construction

Abstract

Large Language Models (LLMs) struggle to answer questions over long interaction histories, where relevant evidence may be scattered across distant sessions. Existing memory systems typically construct and maintain sophisticated memory representations offline, before user queries are observed, incurring substantial overhead and limiting their ability to adapt memory construction to query-specific information needs. In this paper, we propose TMem, a Test-Time Memory construction framework that defers memory construction to inference and progressively builds memory along the reasoning process. Specifically, TMem extracts lightweight episode-level metadata offline to capture key events and temporal associations as retrieval cues, while retaining links to the original interactions to facilitate the reconstruction of dependencies across episodes. At test time, the LLM adaptively retrieves and organizes metadata and raw interactions, progressively constructing query-specific memory as reasoning unfolds. Experiments on LoCoMo and LongMemEval show the effectiveness of TMem by consistently outperforming strong baselines across different backbone LLMs while substantially reducing the overhead of memory construction. Further analysis demonstrates that deferring memory construction to inference enables the memory construction process to dynamically adapt to the information needs of individual queries. All code and datasets will be available via GitHub.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Term Memory, Test-Time Scaling, Conversational Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vfXMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes T2T^2Mem, which is a test-time memory construction framework that defers memory construction to inference and progressively builds memory along the reasoning process. The experiments support the effectiveness of T2T^2Mem. The work plans to release the code and datasets via GitHub.

Critical strengths
  1. The figure is well-presented.
  2. It seems that the method is relatively easy to rpeoduce.
  3. The experiment results support the effectiveness.
  4. The author propose to provide code and data to reproduce the results.
Critical weaknesses
  1. It there any pseudocode to directly introduce the method? The main Figure 1 is relatively complex, and I do not capture the core idea.
  2. Can the method could be used to math datasets? Or whether the method is limiated to LongMemEval Dataset?
  3. Can the evaluation be to more common datasets, such as the MATH dataset or other?
  4. What is the core difference between this work and previous RAG?
Questions for authors

N/A

3 comments

Thank you for your detailed review. Please find our point-by-point responses below.

Reply to W1:

Thank you for your comments. T²Mem first extracts lightweight episode-level metadata offline while preserving links to the original interactions. At test time, starting from initially retrieved metadata, the LLM iteratively distills relevant evidence, updates the query-specific memory state, and adaptively decides whether to retrieve additional metadata, revisit raw interactions, or terminate construction based on the current information needs. Finally, the accumulated memory is used to generate the answer. We will incorporate a pseudocode into the revised manuscript to make the overall workflow more accessible.

Reply to W2 and W3:

Thank you for your suggestion. First, on the evaluation setting: T²Mem addresses memory construction over long-term interaction histories, and we follow the standard setting of existing memory systems, evaluating on the widely adopted LoCoMo and LongMemEval benchmarks, the same evaluation suite used by nearly all recent work in this area, including our primary baselines LightMem and AnchorMem.

Regarding extension to math datasets such as MATH: we believe this is a promising direction, but a direct transfer is premature, because what constitutes "memory" in mathematical reasoning is itself undefined. In memory-oriented tasks, factual knowledge, user preferences, and temporal dependencies are well-recognized as content to memorize, whereas in mathematical reasoning, which mathematical knowledge should be memorized, and whether building and reusing such memory helps, remains an open question without established definitions or studies. MATH evaluates standalone problem-solving ability, which differs from dynamically acquiring evidence from interaction histories. We believe exploring whether memory mechanisms benefit mathematical reasoning, and what should be memorized there, is a promising future direction, and thank you for raising this discussion.

Reply to W4:

The key distinction between T²Mem and traditional RAG lies in reasoning-driven, query-specific memory construction at test time, rather than simply retrieving relevant documents to support answer generation. Traditional RAG primarily retrieves external documents as context for answering questions. However, in long-term conversational scenarios, relevant evidence is often scattered across multiple sessions, and fragmented retrieval may fail to preserve temporal relationships and cross-session dependencies. In contrast, T²Mem explicitly organizes evidence accumulated during test-time reasoning into an evolving memory state, which guides subsequent decisions to retrieve additional metadata, revisit raw interactions, or terminate construction. Therefore, compared with traditional RAG and general iterative retrieval, T²Mem goes beyond multi-step evidence acquisition by interleaving evidence retrieval, memory construction, and reasoning decisions, enabling memory to evolve progressively with the reasoning process. This design is further supported by the progressive memory construction ablations in Tables 3 and 5.

Thank you for the response. I will keep the positive score for this work, as the main concerns are addressed.

Thank you for your time and efforts again in reviewing our paper. If there are any remaining points or if you require further clarification, please feel free to reach out. We sincerely appreciate your attention and valuable feedback.

Reviewer_BWZDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a memory system for answering questions over long conversation histories called T2M. Offline, an LLM writes a structured entry for each session, with a link back to the original session. At test time, the model starts from the top 5 metadata entries and, for up to 5 steps, can retrieve one more entry, open the original session behind an entry, or stop. At each step, it writes down the relevant evidence, and it answers from these notes. The paper frames this as building memory online at test time. On LoCoMo and LongMemEval, T2M achieves the highest average accuracy among nine baselines covering retrieval methods, offline memory systems, and a test-time memory method.

Critical strengths
  1. The method is simple and practical, with per-session structured entries plus a simple agent loop that can go back to the original sessions. The prompts are given clearly, which also helps with reproducibility.
  2. T2M has the highest reported average accuracy on both benchmarks with both backbones. The reported margin is large on LongMemEval.
  3. The offline step is cheap. T2M spends roughly fewer tokens on construction than methods like AnchorMem, HiMem, and xMemory.
  4. The paper includes ablations on the metadata format, on metadata versus original sessions, and on the step-by-step process, as well as analyses of memory quality and token budget.
Critical weaknesses
  1. Novelty and motivation
  • It's unclear if the method demonstrates a new contribution beyond existing retrieval methods. Search-o1 already combines retrieval, reasoning, and evidence distillation, while LazyMem constructs memory at query time. T2M adds structured metadata and source access, but still extracts facts and resolves dates offline (Figure 7). Equation 12 also appears to keep the search query fixed. The model chooses when to retrieve more information, but does not clearly use new findings to change what it searches for. The paper reports better answers, but does not clearly establish what new capability or general insight follows from treating evidence accumulation as memory construction. Its conceptual contribution therefore remains uncertain. Also, it is unclear how the proposed test-time memory advances the current field of agentic AI, where many popular harnesses already use test-time memory construction via different formulations: file system, Markdown, graph structure, or compactions. The paper does not sufficiently discuss or compare these methods.
  1. Impact, significance, and support for the claims
  • The method and results do not support the main claim. The paper proposes to build memory at test time while the model reasons. However, the lightweight metadata is a comprehensive memory built offline. The memory design looks close to what LightMem or AnchorMem build. The test-time search also does not follow the model's reasoning, since each search reuses the original question (Equation 12). Finally, the ablations do not show that the step-by-step process is what drives the gains over the baselines. Answering directly from the top 5 entries, with no step-by-step process, already scores 65.45 with Qwen, above Naive RAG and Search-o1. Therefore, the experiments cannot tell us whether building memory at test time is the better idea.
  • The benchmarks do not test the paper’s designed setting. The paper's motivation is that with very long histories, offline memory building has substantial overhead that limits scalability. The histories used here are short, though. LoCoMo conversations are about 26k tokens each, and LongMemEval-S histories have about 50 sessions. At this size, the 5 to 9 metadata entries the model sees already cover a sizeable share of the sessions. LongMemEval-M, with 500 sessions (probably > 1M tokens), would be a better test. That is also where a search that adds only one entry per step, for at most 5 steps, could struggle, especially on questions that require information from many sessions. Also, both benchmarks provide clear session boundaries, but typically real conversation streams do not, and the method depends on them.
  • The results do not yet establish a general advantage in reasoning across episodes. T2M achieves the strongest overall averages, which is meaningful evidence. However, the proposed explanation of progressively discovering dependencies across episodes is not directly shown. LightMem remains stronger on some relevant categories, including LongMemEval Multi-Session questions with both backbones in Table 2. The rollout ablation also changes evidence access and computation at the same time. Consequently, the results show system-level effectiveness more clearly than a transferable insight into why test-time memory construction works.
  1. Experimental rigor and clarity
  • The comparisons against the baselines are not fair.

    a. With Qwen, T2M uses about 9 times the tokens of Naive RAG and 4 times those of Search-o1 (about 7 and 4 times with Gemma), yet the paper says budgets are comparable. With Qwen, it also uses more tokens than MemoryOS, LightMem, and xMemory. Figure 2b shows that T2M's offline metadata step is cheap, so almost all of its cost is spent at test time. Figure 2b shows that T2M's offline metadata step is cheap, so almost all of its cost is spent at test time and is paid again for every question. In contrast, offline memory is built once and reused for all later questions. A fair comparison should therefore show how total cost grows with the number of questions. Since T2M makes several LLM calls per question, the paper should also report the waiting time per question.

    b. The prompts may favor T2M on temporal questions. The metadata prompt turns relative times into exact dates, and the answer prompt asks for LoCoMo’s gold-answer date format. The version that never reads the original sessions gets the best temporal score, which suggests that these gains come from the metadata and prompts, not from reading the original sessions. The paper does not disclose whether the baselines received the same answer prompt or session dates.

  • Key baselines are missing.

    a. The first missing one is simply giving the whole history to the model. LoCoMo conversations and LongMemEval-S histories should both fit in the context windows of both backbone models. The paper is motivated by limited context windows but never shows that this limit is a problem here. An upper bound that gives the model only the correct sessions would also help.

    b. The second missing baseline is a standard search agent, such as ReAct or Search-o1, using the same metadata and the same original sessions, with the same number of steps and the same token budget. The only difference from T2M would be that it keeps what it reads instead of writing short notes.

    c. A third useful baseline is simple retrieval over the metadata, or over the metadata plus the linked full sessions, with matched tokens. I suspect these simple baselines would close much of the gap.

    d. The paper also does not cite or compare with ReadAgent and MemWalker. Both use a very similar idea: short summaries of the text, plus the ability to go back and read the original pages. Mem0 is cited but not compared, and A-Mem and Zep are common baselines on these benchmarks.

Questions for authors
  1. Does the LoCoMo ranking hold under stricter factual grading than the same-topic criterion in Figure 10, or with a human check of disagreements?
  2. Can you explain how the averages in Tables 1–3 and 5 were computed? Were different evaluation subsets used?
  3. Can Figure 4 compare the same questions under fixed step budgets, rather than groups defined by each method's stopping step?
  4. Does T2M still improve accuracy when iterative retrieval uses the same sources, tools, and total budget? Were the adapted baselines checked for correct operation?
  5. Could the authors explain how each baseline was set up and tuned, and whether retrieval budgets were matched?
14 comments
AuthorsOpenReview ↗

Reply to Reviewer BWZD (1/5)

Thank you for your detailed review. Please find our point-by-point responses below.

Reply to W1:

Thank you for your valuable comments. We emphasize that the core contribution of T²Mem is not a new retrieval strategy or knowledge representation, but reasoning-driven construction of an evolving memory state at test time: unlike LazyMem's[1] retrieve-then-construct paradigm and Search-o1's[2] retrieval-augmented reasoning, which build recall memory designed to be retrieved, T²Mem revises the memory state as evidence is acquired. The model reads the history progressively, step by step, distilling each observation into memory units integrated into the state, which in turn guides subsequent information-access decisions. Eq. 12 fixes only the query used for retrieval, while Eq. 8 enables memory-conditioned adaptive decisions.

Our episode-level metadata follows the established practice of existing memory systems, which represent memory by extracting structured information (entities, facts, and temporal cues) into compact entries (LightMem[3], AnchorMem[4]). Our metadata additionally maintains a source link from each entry to its raw session Eq. 3, yielding a two-layer structure: the metadata layer offers a compact, searchable index of the history, while the link guarantees that the full raw evidence behind every entry remains reachable and traceable. This design directly serves test-time construction: when a metadata entry is relevant but insufficiently detailed, the model can follow the link to recover the fine-grained raw evidence (the ReadRaw operation, Eq. 13) and revise the memory state with it. The ablations validate both layers: w/o Metadata strips the index layer (69.74 on LoCoMo), w/o Rawdata removes access to the linked raw sessions (67.60), and full T²Mem reaches 71.69 (Table 3). Moreover, the same mechanism with our linked metadata consistently outperforms summary- and graph-based representations (Tables 3 and 5), confirming that traceable, source-linked metadata is a better substrate for test-time memory construction.

Furthermore, agentic harnesses such as files, Markdown, graphs, and context compaction concern how memory is represented and managed, which is orthogonal to and composable with how the memory state is progressively revised during reasoning. As shown in Tables 3 and 5, T²Mem works across summary- and graph-based representations, indicating that the framework supports different representation choices while benefiting further from effective metadata design.

AuthorsOpenReview ↗

Reply to Reviewer BWZD (2/5)

Reply to W2:

Thank you for your comments. We further clarify the motivation and experimental evidence supporting T²Mem.

Reply to W2-1:

The key motivation of T²Mem is to construct query-specific memory during inference, rather than relying on memory representations fully organized offline. While our lightweight metadata preserves episode-level facts and temporal cues, it primarily serves as a retrieval index rather than the final memory used for answering. The actual memory is progressively constructed as the model reasons over the query and identifies additional information needs. Importantly, although Eq. 12 uses the original query for relevance ranking, Eq. 8 allows the evolving memory state to determine whether to retrieve additional metadata, access raw interactions, or terminate construction.

The ablation results further support this design. As shown in Table 3, directly answering from the initial top-5 metadata achieves 65.45% accuracy, while the complete T²Mem improves it to 71.69% (+6.24 pp). Both settings start from the same retrieved evidence, but T²Mem can progressively acquire and integrate additional information as reasoning unfolds. This demonstrates the value of test-time memory construction beyond simply answering from initially retrieved metadata.

Reply to W2-2:

LoCoMo and LongMemEval are widely adopted benchmarks for evaluating long-term conversational memory, particularly the ability to retrieve and integrate information across interactions. Beyond these benchmarks, we further evaluate the computational scalability of T²Mem on LongMemEval-M, which expands each interaction history from approximately 50 to 500 sessions.

MethodEnd-to-End Latency (s/query)Tokens per Question (k)
T²Mem1,4531,619
LightMem6,4832,176

As shown above, T²Mem achieves substantially lower end-to-end latency and token consumption than LightMem on LongMemEval-M. These results further demonstrate that the efficiency advantage of test-time memory construction becomes more pronounced as interaction histories grow.

Moreover, the maximum construction steps represent a configurable computational budget rather than a limitation of the framework. Appendix A.5 (Fig. 3) evaluates different construction depths and shows that additional steps can further improve performance. T²Mem also operates on episodic interaction segments without requiring dataset-specific session identifiers.

Reply to W2-3:

The effectiveness of progressive memory construction is supported by both our ablation studies and additional evidence-recovery experiments. As shown in Table 5, removing Rollout Construction reduces performance on both Multi-Session and Temporal Reasoning, demonstrating the importance of progressively acquiring and organizing evidence.

To further investigate the performance differences with LightMem, we conduct an additional gold-evidence injection experiment on LongMemEval. Specifically, we inject metadata entries corresponding to ground-truth evidence sessions into the candidate pool to examine the effect of evidence availability. With this additional evidence, T²Mem improves from 70.68% to 77.44% on Multi-Session, approaching LightMem's 78.20%, and from 84.96% to 89.47% on Temporal Reasoning, surpassing LightMem's 85.71%.

These results indicate that evidence coverage plays an important role in the category-level differences, while T²Mem can effectively integrate relevant evidence once it becomes available. Furthermore, Fig. 2(a) demonstrates the superior quality of the constructed memory across multiple dimensions, and Appendix A.6 provides a concrete example of recovering temporal evidence missed by the initial retrieval. Together, these results support the effectiveness of progressively discovering and integrating cross-session evidence during test-time memory construction.

CategoryLightMemT²Mem (std.)T²Mem (w/ gold metadata)
Single-Session-User91.4392.8695.71
Single-Session-Assistant25.0098.2198.21
Single-Session-Preference76.6783.3383.33
Multi-Session78.2070.6877.44
Knowledge-Update87.1892.3193.59
Temporal-Reasoning85.7184.9689.47
Avg.77.4084.8088.40
AuthorsOpenReview ↗

Reply to Reviewer BWZD (3/5)

Reply to W3:

Thank you for your comments. We provide additional experimental evidence and clarifications regarding evaluation fairness and baseline comparisons.

Reply to W3-1 (a):

Thank you for your comments. We preserve the original configurations of each baseline rather than enforcing identical token budgets, which could alter their intended behavior. The actual token consumption of each method is reported in Tables 1–2. To further evaluate the overall computational cost, we measure both offline construction and online inference latency of T²Mem and LightMem, as summarized below.

BenchmarkMethodOffline ConstructionOnline InferenceTotal Latency
LoCoMoT²Mem94 s/history24.80 s/query39,132 s
LoCoMoLightMem2632 s/history1.04 s/query27,921 s
LongMemEvalT²Mem432 s/instance25.00 s/query228,500 s
LongMemEvalLightMem1832 s/instance1.03 s/query916,515 s

Overall, the two methods exhibit total latency of the same order of magnitude on LoCoMo, while T²Mem achieves substantially lower total latency than LightMem on LongMemEval. Although T²Mem incurs additional inference-time computation, it significantly reduces the overhead of offline memory construction.

We further extend our evaluation to LongMemEval-M, which increases the number of sessions per interaction history by 10 times. T²Mem achieves substantially lower per-question end-to-end latency and token consumption than LightMem, further demonstrating that the cost advantage of test-time memory construction becomes more pronounced as interaction histories grow.

More importantly, real-world conversational histories continuously evolve as new interactions arrive. While offline memory systems need to maintain and consolidate persistent memory representations, T²Mem only requires lightweight metadata extraction for newly added episodes, deferring query-specific memory construction to inference. This design reduces the overhead of maintaining complex memory structures and is particularly suitable for continuously evolving long-term conversational scenarios.

MethodEnd-to-End Latency (s)Tokens per Question (k)
T²Mem1,4531,619
LightMem6,4832,176

Reply to W3-1 (b):

Our temporal normalization relies exclusively on original conversation timestamps rather than ground-truth answers. Furthermore, our answer prompt does not require exact gold-answer formatting, and the LLM judge explicitly accepts semantically equivalent temporal expressions (Fig. 10). Following the evaluation protocol adopted by LightMem, we apply the same judge model and evaluation prompt to all compared methods. The generation prompts individual baselines follow their original implementations to preserve their intended behavior. Moreover, Table 5 demonstrates that accessing raw interactions further improves Temporal Reasoning on LongMemEval, indicating that the benefits extend beyond metadata-based temporal normalization.

Reply to W3-2 (a):

Our work focuses on constructing effective long-term memory rather than directly processing entire interaction histories. Even when a history fits within the context window, this does not guarantee effective utilization of scattered evidence, while repeatedly processing the entire history introduces substantial computational overhead. Similarly, providing only gold sessions assumes oracle knowledge of relevant evidence, bypassing the evidence-discovery process central to our setting. Therefore, we prioritize comparisons with representative retrieval and memory systems under consistent backbone configurations.

Reply to W3-2 (b):

To further isolate the contribution of explicit memory construction, we conduct an additional ablation, T²Mem w/ Full Trace, which retains the complete accumulated reasoning and observation trajectory at each step instead of T²Mem's distilled evidence memory, while preserving the same information-access mechanism.

As shown in the follwing table, T²Mem achieves 71.69% overall accuracy, outperforming T²Mem w/ Full Trace (69.87%) by 1.82 percentage points, while reducing token consumption by 45.8%. These results demonstrate that explicitly distilling and organizing query-relevant evidence into an evolving memory state is both more effective and computationally efficient than retaining the complete reasoning trajectory, highlighting the importance of memory construction beyond merely accumulating retrieved information.

MethodMulti-HopTemporalOpen-DomainSingle-HopAvg.# Tokens (×10³)
T²Mem w/ Full Trace62.4173.8347.9273.3769.8716,168
T²Mem59.2275.7050.0076.8171.698,765
Reviewer_6rXwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes T²Mem, a test-time memory construction framework that uses lightweight offline metadata and progressively builds query-specific memory through LLM reasoning and adaptive retrieval. Experiments on LoCoMo and LongMemEval with two backbone models show improvements over existing memory and retrieval baselines, supported by ablation studies and token consumption analysis.

Critical strengths
  1. The paper addresses a practical problem in long-term conversational memory. Constructing comprehensive memories offline can be expensive and may not align well with future queries. The proposed combination of lightweight metadata and selective access to raw interactions offers a practical approach to query-specific memory construction.
  2. T²Mem achieves the highest average accuracy among the evaluated methods on both benchmarks. The ablation studies also provide evidence for the usefulness of metadata, raw interaction access, and progressive construction.
Critical weaknesses
  1. Limited novelty over existing test-time memory and agentic retrieval methods. The proposed framework mainly combines metadata indexing, iterative retrieval, and LLM-based memory updates. These components are closely related to existing approaches such as LazyMem, DeferMem, and Search-o1. The paper does not clearly establish what fundamentally distinguishes T²Mem from a standard agentic retrieval framework with access to structured metadata and raw interactions.

  2. Insufficient evidence for the contribution of adaptive memory construction. The ablations do not clearly separate the benefits of adaptive memory construction from additional retrieval and reasoning. For example, removing rollout construction also reduces the available evidence and token consumption. A comparison with fixed-step retrieval or a standard retrieval agent under the same token budget would better support the claimed contribution.

  3. Missing online latency and overall cost evaluation. While T²Mem reduces reliance on offline memory construction, it introduces additional computation at inference time. The paper reports token consumption but does not evaluate end-to-end latency. Moreover, offline construction costs can be amortized across multiple queries, whereas T²Mem constructs memory for each query. Without such comparisons, its practical efficiency advantage remains unclear.

  4. Limited improvements on cross-session and temporal reasoning. The improvements on LongMemEval are largely concentrated in certain question categories. For example, T²Mem substantially outperforms LightMem on Single-Session Assistant questions but performs worse on Multi-Session and Temporal Reasoning with both backbones. Since the paper emphasizes recovering cross-session and temporal evidence, these results deserve further analysis. It would also be useful to measure how often T²Mem successfully retrieves evidence missed by the initial retrieval.

  5. Some experimental results and baseline settings require clarification. In Table 2, the reported average accuracy for Qwen/xMemory and Qwen/AnchorMem appears inconsistent with the category scores and dataset statistics in Table 4. The paper also replaces LazyMem's original memory-processing model with the backbone model, which may affect comparison fairness. These issues should be clarified.

Questions for authors
  1. What is the key difference between T²Mem and a standard agentic retrieval framework using the same metadata and retrieval tools? How does the evolving reasoning state influence subsequent retrieval, given that Eq. (12) uses the original query?

  2. Can the authors compare T²Mem with fixed-step retrieval or an iterative retrieval agent under the same token budget to isolate the contribution of adaptive memory construction?

  3. What is the end-to-end inference latency of T²Mem compared with offline memory methods? How does the total cost change when multiple queries share the same interaction history?

  4. Why does T²Mem underperform LightMem on Multi-Session and Temporal Reasoning despite the paper's emphasis on cross-session evidence recovery? What explains the large performance gap on Single-Session Assistant questions?

  5. Could the authors clarify the inconsistent average scores in Table 2 and discuss how replacing LazyMem's original memory-processing model affects the comparison?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer 6rXw (1/3)

Thank you for your detailed review. Please find our point-by-point responses below.

Reply to W1 and Q1:

Thank you for your detailed review. We would like to clarify that T²Mem differs fundamentally from existing agentic retrieval methods in the role of the reasoning process itself. Its key innovation is not adaptive retrieval of additional relevant evidence to answer a given query, but adaptive memory-state construction and evolution.

Specifically, inspired by human-like cognitive processes, T²Mem leverages the test-time reasoning capability of LLMs to progressively evolve the maintained memory state, much like humans recall potentially relevant knowledge, incorporate newly acquired information, and continuously revise their mental representations. At each reasoning step, the model selectively accesses lightweight metadata to identify potentially relevant knowledge, retrieves specific information when needed, and distills and integrates the acquired information into the evolving memory state. Importantly, this process goes beyond simply accumulating knowledge: the model continuously refines its memory units to better distinguish relevant facts from potentially confusing or conflicting ones, thereby improving its subsequent knowledge acquisition and reasoning. The updated memory state, in turn, guides subsequent information access and further memory refinement.

In this way, reasoning is not merely used to retrieve additional evidence, but serves as the driving mechanism for continuously constructing, revising, and refining memory. We elaborate on the distinctions from the following aspects:

(1) Different roles of LLM reasoning. LazyMem [1] and DeferMem [2] primarily focus on constructing retrievable memories for subsequent recall, while Search-o1 [3] employs iterative reasoning to reformulate queries and retrieve additional evidence. These methods mainly aim to identify the knowledge or memory units relevant to answering a given question. In contrast, T²Mem adaptively reads top-ranked metadata to establish more accurate associations among facts during reasoning. This process goes beyond simply accumulating or retrieving memory: it progressively reorganizes and refines memory representations, helping the model better distinguish relevant facts from potentially confusing or conflicting information.

(2) Different requirements for adaptivity. Unlike Search-o1, T²Mem does not use generated subqueries to adaptively retrieve additional evidence during reasoning. Instead, it relies on the retrieval ranking list generated by the original query. As formalized in Eq. 9 and Eq. 12, the Top-K function selects the next batch of metadata for the model to read, enabling adaptive access to the already retrieved metadata rather than introducing additional evidence into the context. This motivation differs from the adaptive retrieval mechanisms in Search-o1 and LazyMem: T²Mem emphasizes the fine-grained and progressive analysis of available information, through which the memory state can be gradually evolved and refined.

Reply to W2 and Q2:

Thank you for your suggestion. We further analyze T²Mem and Search-o1 under two complementary budget settings on LoCoMo using Qwen3.5-27B.

Fixed step budget. As shown in Figure 3 of Appendix A.5, we evaluate T²Mem on the same questions under different maximum construction steps (1, 3, 5, 7, and 10), demonstrating consistent improvements as construction steps increase. Figure 4 further compares T²Mem and Search-o1 across actual stopping steps, showing that T²Mem achieves larger performance gains as multi-step reasoning progresses.

Fixed token budget. We additionally evaluate Search-o1 with a cumulative token threshold of 6,000 tokens per query, without imposing an additional step limit. The results are compared with T²Mem's reported performance below. T²Mem achieves 17.47 percentage points higher overall accuracy than Search-o1. Together with the construction-depth analyses, these results show that the benefits of T²Mem do not come from additional iterative retrieval and reasoning, but from revising an evolving, query-specific memory state as evidence is progressively acquired.

MethodMulti-HopTemporalOpen-DomainSingle-HopAvg.
Search-o149.2954.5238.5457.5554.22
T²Mem59.2275.7050.0076.8171.69
AuthorsOpenReview ↗

Reply to Reviewer 6rXw (2/3)

Reply to W3 and Q3:

Thank you for your suggestion. We further evaluate the offline construction and online inference latency of T²Mem and LightMem[4] on both benchmarks, as summarized below.

We evaluate the end-to-end latency of the offline memory construction baseline (LightMem) and T²Mem, as shown in the table below. Overall, the two methods exhibit comparable total latency on LoCoMo, while T²Mem achieves substantially lower latency than LightMem on LongMemEval.

BenchmarkMethodOffline ConstructionOnline InferenceTotal Latency
LoCoMoT²Mem94 s/history24.80 s/query39,132 s
LoCoMoLightMem2632 s/history1.04 s/query27,921 s
LongMemEvalT²Mem432 s/instance25.00 s/query228,500 s
LongMemEvalLightMem1832 s/instance1.03 s/query916,515 s

We further extend our evaluation to LongMemEval-M, which increases the number of sessions per interaction history by 10 times. As shown in the table, T²Mem achieves substantially lower per-question end-to-end latency (1,453 s vs. 6,483 s) and token consumption (1,619k vs. 2,176k) than LightMem, further confirming that the cost advantage of test-time memory construction grows with history length.

MethodEnd-to-End Latency (s)Tokens per Question (k)
T²Mem1,4531,619
LightMem6,4832,176

More importantly, real-world conversational histories continuously grow as new interactions arrive. Offline memory systems must maintain and consolidate persistent memory representations, introducing recurring maintenance overhead beyond initial construction. Memory access is also inherently sparse: on LongMemEval, each question relies on only about 1–5 evidence sessions out of 50, most pre-constructed memory is never queried. In contrast, T²Mem only requires lightweight metadata extraction for newly added episodes, while deferring query-specific evidence integration to test time. This avoids maintaining sophisticated global memory structures and enables memory construction to adapt to evolving information needs. Therefore, T²Mem provides a practical and efficient alternative for continuously evolving conversational histories, particularly when memory updates are frequent and query reuse is moderate.

AuthorsOpenReview ↗

Reply to Reviewer 6rXw (3/3)

Reply to W4 and Q4:

Thank you for pointing this out. We first clarify how subsequent steps acquire evidence in T²Mem: unlike Search-o1's query-driven iterative retrieval, later steps do not re-query. The initial retrieval ranks metadata with the original query (Eq. 9, top-5), each subsequent step fetches the single highest-ranked unseen entry (Eq. 12, up to 5), and the model progressively reads down this fixed-query candidate list as the memory state demands Eq. 8. Consequently, whether an evidence session can ever be accessed is determined at the first step.

This explains the observed pattern. Multi-Session and Temporal Reasoning questions are analogous to multi-hop QA: the query itself lacks the intermediate cues needed to rank the relevant sessions, so a single fixed-query ranking has inherently lower recall in these categories. The deficit stems from first-stage recall rather than the memory-construction mechanism, which is precisely the cost of our design choice that adaptivity resides in the evolving state rather than in query reformulation.

To verify this attribution, we conduct a gold-evidence injection experiment on LongMemEval: for every question, the metadata entries whose source sessions contain ground-truth evidence are injected into the candidate pool before answering.

CategoryLightMemT²Mem (std.)T²Mem (w/ gold metadata)
Single-Session-User91.4392.8695.71
Single-Session-Assistant25.0098.2198.21
Single-Session-Preference76.6783.3383.33
Multi-Session78.2070.6877.44
Knowledge-Update87.1892.3193.59
Temporal-Reasoning85.7184.9689.47
Avg.77.4084.8088.40

The results confirm this attribution. Injecting the gold metadata brings the largest gains exactly on the questioned categories: Multi-Session improves by 6.76 points (70.68 to 77.44, reaching parity with LightMem), and Temporal-Reasoning improves by 4.51 points (84.96 to 89.47), surpassing LightMem. Evidently, when evidence is available, progressive construction is fully competitive on these categories, so the remaining gap indeed stems from first-stage fixed-query recall rather than the construction mechanism. The single-session categories show no change under injection, consistent with their evidence being already well covered by the initial retrieval. Improving first-stage recall, for example via sub-question-driven iterative search as in Search-o1 and LazyMem, is a promising future direction that we will discuss in future work.

Regarding Single-Session Assistant, the LightMem authors confirmed on their official GitHub that their LongMemEval experiments index only user messages, limiting assistant-side evidence.

Reply to W5 and Q5:

Thank you for your comments. We follow the evaluation protocol of LightMem and EverMemOS[5], computing the overall accuracy as the weighted average of category accuracies according to their respective numbers of questions. The reviewer is right that two averages in Table 2 are inconsistent with this protocol: due to transcription errors, the reported values for Qwen/xMemory and Qwen/AnchorMem (68.80 and 64.40) do not match their category scores. The correct values are 66.20 for xMemory and 66.80 for AnchorMem. We have corrected these entries, together with a few similarly affected averages in Table 1, in the revision; All rankings and conclusions are unaffected.

For LazyMem, its dedicated memory-processing model (LazyMem-4B) had not been released when we conducted our experiments, so we reproduced it with the unified backbone (Appendix A.3), isolating the memory-construction mechanism from backbone effects.

Reference:

[1] LazyMem: Retrieve Broadly, Construct Selectively for Efficient Long-Term Agent Memory. arXiv preprint arXiv:2607.22690.

[2] DeferMem: Query-Time Evidence Distillation via Reinforcement Learning for Long-Term Memory QA. arXiv preprint arXiv:2605.22411.

[3] Search-o1: Agentic Search-Enhanced Large Reasoning Models. EMNLP 2025.

[4] LightMem: Lightweight and Efficient Memory-Augmented Generation. ICLR 2026.

[5] EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoning. ACL 2026.

Dear Reviewer 6rXw,

Thank you for your time and efforts again in reviewing our paper. We kindly remind you that most of the discussion period has passed. We have carefully addressed the comments and suggestions you provided in our submitted rebuttal, and we hope our responses clarify any concerns or questions.

If you have any additional feedback or require further clarification, please feel free to reach out. Thank you very much for your attention.

Best regards,

Submission 1856 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.