acceptodds
Under review as a conference paper at ICLR 2027

Temporal Evidence Forests for Long-Horizon Multi-Party Conversational Memory

Abstract

Long-term memory has been widely adopted in LLM-based agents to support persistent interaction and personalized assistance. Long-term multi-party interactions, however, accumulate substantially more detailed information, with decisions repeatedly updated and states evolving across participants and interaction stages. Temporal structure is therefore crucial for tracing such memories: relevant information often appears densely within short intervals, while transitions among different states are anchored by a strict temporal order. Existing temporally aware memory frameworks primarily store time as metadata or use temporal signals to guide retrieval over preconstructed memory structures, without using temporal structure to both recover locally concentrated evidence and reconstruct its complete event progression. To address these limitations, we propose T-MEM, a temporal-structure-based memory framework that preserves interactions and organizes query-relevant evidence through two temporal mechanisms. Temporal Evidence Discovery exploits the local concentration of related messages to recover overlooked evidence, while Temporal Memory Structuring models state transitions and supporting relations over time and compresses low-information spans within the resulting forest. Evidence Forest Linearization then serializes the structured evidence for answer generation. Experiments on two long-term multi-party memory benchmarks demonstrate that T-MEM consistently outperforms competitive memory frameworks across LLMs, parameter settings, domains, and query types.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Party Conversations, Temporal Reasoning, Structured Memory, Long-Term Memory, LLM Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jZpWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes T-MEM, a long-horizon multi-party conversational memory framework that uses Temporal Evidence Discovery to recover temporally localized evidence missed by global retrieval and Temporal Memory Structuring to organize messages into evidence forests with UPDATE/EXPAND relations that capture state evolution. Experiments on GroupMemBench and EverMemBench report consistent improvements over existing long-term memory baselines across two LLM backbones, supported by ablations, evidence-recall analysis, robustness tests, and efficiency evaluation.

Critical strengths
  1. The paper focuses on a real problem in long-term conversations: finding the right past messages and understanding how decisions changed over time.

  2. The method performs clearly better than the tested baselines on both benchmarks, and the ablation results generally support the main design choices.

Critical weaknesses
  1. The method is fairly complicated and combines many existing components, so the overall technical novelty is somewhat limited.

  2. The forest-vs-flat comparison is not fully clean because the forest condition also receives stronger temporal reasoning instructions in the prompt.

  3. The experiments are limited to two relatively clean benchmarks, so it is unclear how well the method would work on noisier real-world conversations.

  4. The reproducibility issue is serious: the submitted code is a post-hoc reconstruction from the paper rather than the actual code used to produce the reported experimental results.

Questions for authors
  1. Can the authors provide the exact original code, prompts, configurations, and raw outputs used to generate Table 1, Table 2, and Figures 2–6, and explain why the submitted repository is instead described as a clean-room reconstruction derived from the manuscript?

  2. Can the authors rerun the flat-vs-forest comparison with exactly the same answer prompt, removing the forest-specific UPDATE/EXPAND reasoning instructions, to show that the gain actually comes from the forest structure rather than from extra prompting?

Reviewer_5j44OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes T-Mem, a conversational memory framework combining temporal evidence discovery, query-conditioned memory structuring, and evidence forest linearization. Retrieved messages serve as temporal anchors for recovering additional evidence within localized discussion episodes. The recovered messages are organized into forests with UPDATE and EXPAND relations before answer generation. The framework achieves strong overall results on GroupMemBench and EverMemBench across two answer-model backbones.

Critical strengths

(S1) T-Mem showed the highest overall accuracy among the evaluated methods on both benchmarks under both answer-model backbones. Improvements over the strongest evaluated baselines range from 6.50% to 8.59%.

(S2) Table 2 shows consistent contributions of T-Mem from the principal components across both backbones. Removing temporal discovery reduces accuracy by 6.04% and 6.31%, while removing temporal structuring reduces it by 4.17% and 5.23%. These results support the usefulness of both components within the implemented pipeline.

(S3) Section 5.3 evaluates evidence recovery separately from answer accuracy. At Top-36, T-Mem obtains 72.8% key-evidence recall, compared with 56.6% for metadata-expanded BM25 and 61.5% for HippoRAG2. This diagnostic provides useful evidence that the discovery stage recovers answer-supporting messages missed by direct retrieval.

Critical weaknesses

(W1) The claim in Section 5.4 that the organization experiment isolates “relation-based structuring alone” is not supported by a consistently documented control. The main text describes the flat baseline as a chronological list, whereas Appendix I states that flat messages are presented in retrieval-rank order. Figure 11 also gives the forest condition explicit instructions to follow UPDATE chains and identify the current or final state, without equivalent guidance in the flat condition. Therefore, I think the comparison leaves message ordering and answer-prompt guidance as unresolved explanations for the reported gains.

(W2) The main text attributes improved use of larger evidence sets to explicit state-progression and support relations, but the experiments do not distinguish this mechanism from ordinary query-conditioned semantic organization. Exp-3 and the structuring ablation evaluate the complete organization pipeline. The untyped control does not compare against an independently constructed semantic forest. This leaves the distinct contribution of temporal state-evolution structuring over generic semantic organization unclear.

(W3) The controlled evidence-recovery results support temporal localization, but its independently attributable effect on final QA remains unclear. Section 5.3 evaluates evidence recall, whereas removing discovery in Table 2 also changes candidate search and evidence selection. Passing the matched global and temporal discovery outputs through the same structuring and answering pipeline would clarify how much of the final accuracy improvement follows from localization itself.

(W4) The conclusion’s efficiency claim is supported by accounting for one fixed evaluation workload, but the trade-off between global memory construction and query-time processing is not characterized. Existing systems can amortize construction costs across subsequent queries, while T-Mem performs query-conditioned evidence selection and organization.

Questions for authors

(Q1) Were flat messages in the Exp-3 runs presented chronologically or in retrieval-rank order? Please resolve the discrepancy between Section 5.4 and Appendix I. Does the reported advantage persist against chronologically ordered flat evidence with equivalent instructions for resolving applicable updates and superseded states?

(Q2) What evidence demonstrates that temporal state-evolution reconstruction contributes beyond ordinary semantic organization? Please compare the proposed forest with an independently constructed query-conditioned semantic forest using the same messages, construction model, computational budget, and equivalent answering guidance. Removing labels from the existing temporal topology does not isolate this distinction.

(Q3) Does the advantage of temporal localization persist in final answer accuracy when the matched global and temporal discovery variants use the same downstream structuring, compression, reader, and evidence-token budget? Please report this comparison to connect the controlled recall result to the end-to-end QA claim.

(Q4) How does the efficiency comparison change with the number of queries issued against a fixed conversation history? Separate corpus-construction costs from query-processing costs and report the workload range over which T-Mem retains its cost advantage should be good.

Reviewer_uW3TOpenReview ↗
Rating
3(+1)
Confidence
1(−2)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method that is used for long-term memory. The method uses BM25 to find the time chunk where message is dense, and uses LLM to find more evidence. Then, it uses UPDATE and EXPAND to link these messages and gives the information to the answer model. The experiment results show that T-MEM has advantages compared with other baselines.

Critical strengths
  1. The motivation is reasonable, where the paper claims the relevant information is usually close in time.

  2. The UPDATE and EXPAND relationship makes the answer easier to track.

  3. The method shows superiority in two benchmarks.

Critical weaknesses
  1. All the components are used after the query arrives, including evidence selection and edge building, which would cause high online latency. The method would be unacceptable when many queries are arriving.

  2. The method uses strong LLMs, which is basically LLM reranking. I think the method relies heavily on the base model. If the base model is weak, its performance would drop.

  3. The method relies on the meta information in the benchmark. However, this information usually does not exist in real scenarios, which limits the application scope of the methods.

Questions for authors
  1. Could the author report the retrieval latency of each stage and compare it with other baselines?

  2. Could the author report the performance when using the weak LLM to select evidence?

  3. Could the author add some experiments where the topic and other modules are removed?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.