acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Temporal Validity in LLM Agent Memory

Abstract

LLM agents retrieve from interaction histories in which facts may expire, be cancelled, or be superseded by later updates. Semantic relevance alone does not capture whether a retrieved memory is valid for the time referenced by a query. We introduce TimeBound, a benchmark for this setting with 1000 long-history examples spanning eight temporal-memory phenomena. Each memory is annotated with observation time, event time, validity interval, status, gold answer, and gold evidence. We also introduce TimeBound-RAG, a simple diagnostic retriever that combines semantic relevance with query-time validity and memory status. Across four open-weight readers, TimeBound-RAG reaches mean relaxed accuracy of , compared with for Semantic RAG and for Full History. Semantic RAG obtains slightly higher evidence F1 ( vs. ), but retrieves substantially more invalid memories ( vs. ). The results separate validity-sensitive retrieval failures from temporal computation, recurrence, scheduling, and long-memory localization failures that require mechanisms beyond retrieval.

Primary area:
learning theory
Keywords:
LLM agents, temporal memory, retrieval-augmented generation, long-term memory, temporal validity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tWpXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies temporal validity in LLM-agent memory, where a semantically relevant memory may be expired, cancelled, superseded, or valid only for a particular historical period. The authors introduce TimeBound, a synthetic benchmark containing 1000 examples across eight temporal-memory task families. Each memory includes observation time, event time, validity interval, status, and an optional update relation. The authors also propose TimeBound-RAG, a diagnostic retriever that combines semantic relevance with temporal compatibility and an obsolete-memory penalty. Across four open-weight readers, it achieves mean relaxed accuracy of 0.719, compared with 0.475 for Semantic RAG and 0.537 for Full History.

Critical strengths
  1. The benchmark provides a controlled decomposition of several temporal-memory phenomena. Separating expiration, cancellation, conflicting updates, delayed observations, duration, recurrence, rescheduling, and retrospective queries is useful for examining different failure modes. Computing gold answers and evidence from structured states also makes the intended task semantics easier to inspect.

  2. The paper reports evidence-level metrics alongside answer accuracy and includes task-level results. Table 4 makes important limitations visible, including poor elapsed-time performance and substantial degradation on historical time-window queries. I think reporting these negative results is useful, although their causes need a more careful analysis.

Critical weaknesses
  1. The definition and evaluation of temporal validity are not fully consistent with the historical queries in the benchmark. As written, Section 3.1 excludes superseded memories even when their validity intervals contain the queried time. Table 4 also shows time-window accuracy falling from 0.840 to 0.160, without a clear explanation. A related concern is that Oracle Evidence receives an invalid-retrieval rate of 0.438 despite an evidence F1 of 1.000. An inactive record can still be the right evidence for a historical question or a cancellation. These issues leave me uncertain whether the benchmark’s validity rule and metric distinguish obsolete facts from useful evidence, which is central to its intended contribution.

  2. The answer evaluation does not yet give enough confidence in the reported gains. The main four-reader results report only relaxed accuracy, while the separate diagnostic in Table 8 shows a substantial strict–relaxed gap, including 0.295 versus 0.595 for TimeBound-RAG. More concretely, Figure 8 marks “No information available” as correct when the gold answer states that no currently valid code exists. These answers are not necessarily equivalent: lacking information does not establish that a code has expired or is unavailable. Clear scoring rules, strict results for the main experiments, and a human check of relaxed-only matches are needed to determine how much of the improvement reflects correct temporal reasoning.

Questions for authors
  1. What specific capability does TimeBound evaluate beyond existing temporal-validity systems and temporal-dialogue or memory-update benchmarks? Could the authors compare task coverage, available annotations, and evaluated failure modes directly?
  2. Which temporal fields are supplied at inference time? How are validity intervals, status labels, update links, and query reference times obtained? Can the authors separate results with supplied annotations from results with automatically extracted and incrementally maintained annotations?
  3. How does the admissibility rule handle a currently superseded memory that was valid at the queried historical time? What explains the decrease from 0.840 to 0.160 on time-window retrieval?
  4. Why does Oracle Evidence have an invalid-retrieval rate of 0.438? Can the metric distinguish using an obsolete fact as the answer from retrieving an obsolete record as evidence for cancellation, expiration, or historical state?
  5. Can the authors include representative temporal-memory systems and simple structured baselines under comparable reader and metadata conditions? Interval filtering and entity-and-slot update baselines would be particularly informative.
  6. Could the authors provide exact retrieval-score definitions, the parameter-selection procedure, strict and relaxed scoring rules, and strict results for all four readers? A manually checked sample would help establish whether relaxed matching preserves the complete temporal answer.
Reviewer_59haOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces TIMEBOUND, a benchmark with 1,000 synthetic histories covering eight types of temporal memory phenomena. It provides annotations for observation time, event time, validity intervals, status, and gold evidence. The authors also propose TimeBound-RAG, which combines text relevance with temporal and status signals. They evaluate both retrieval and downstream QA. However, the benchmark’s added value over existing temporal reasoning and evolving memory tasks remains unclear. The comparisons also mix the benefits of gold metadata with those of the ranking method. They therefore do not establish the advantage of the algorithm itself.

Critical strengths
  1. Answers and evidence are derived from structured states and checked for consistency. This supports reproducible and controlled diagnostic experiments.
  2. The paper reports evidence retrieval, invalid retrieval rates, and downstream QA performance. These results help show how evidence selection affects answers.
Critical weaknesses
  1. Limited task novelty and unclear added diagnostic value. Existing work already studies temporal reasoning, knowledge updates, and query-dependent memory validity. LongMemEval (ICLR 2025) covers temporal reasoning and knowledge updates. MemConflict uses controlled histories and competing memories to evaluate query-dependent validity. MEME studies state tracking, deletion, and dependency changes in evolving memory. TIMEBOUND generates its own data and defines more specific temporal categories. However, the paper does not show that these categories test abilities that existing benchmarks fail to cover. It also does not adequately discuss close work such as MemConflict and MEME. The control offered by rule-based templates and ideal temporal annotations is not enough on its own to establish the benchmark’s contribution.

  2. Unequal use of information makes the method’s contribution unclear. TimeBound-RAG directly uses gold validity intervals, query time, status, and relation signals from the generator. The main baselines rank memories using only text or observation time. All QA settings show the temporal and status fields of the selected memories, but the retrieval methods already use different information. The results therefore mix the benefits of gold metadata with those of the ranking rules. Simple filtering and ranking baselines using the same fields are missing. In addition, the invalid retrieval metric directly counts status labels that TimeBound-RAG penalizes. An improvement on this metric is therefore largely expected. The experiments also do not test performance when these fields must be extracted and maintained from raw text.

Questions for authors
  1. What new evaluation capabilities does TIMEBOUND provide beyond LongMemEval, MemConflict, and MEME? Can the authors compare task coverage and failure modes? Do the findings hold under template paraphrases and more natural histories?
  2. Can all retrieval methods use the same temporal and status metadata? Can the authors add simple validity-interval filtering, status filtering, and rule-based reranking baselines to isolate TimeBound-RAG’s contribution?
  3. Do the retrieval and QA gains hold with predicted or noisy temporal metadata? Why does the invalid retrieval metric use final status rather than query time and validity intervals? In particular, how does it handle memories that were valid in the past but have since been superseded?
Reviewer_eJcVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses a problem in long-term agent memory: relevant information may be outdated, cancelled, or superseded by later updates. The authors introduce TimeBound, a benchmark of 1,000 synthetic examples covering eight temporal scenarios, such as expiration and rescheduling, with time and validity labels for each memory. Their TimeBound-RAG method adds time and status checks to semantic retrieval, prioritizing memories that are valid at the time specified in the query. Across four open-weight models, it achieves an average relaxed-match accuracy of 71.9%, compared with 47.5% for standard semantic retrieval and 53.7% for full-history prompting, while retrieving less invalid information. However, it relies on explicit temporal annotations and still needs additional mechanisms for date arithmetic, reasoning about recurring events, and reconstructing past states.

Critical strengths

The paper clearly explains its motivation and what it aims to achieve.

Critical weaknesses

The writing often feels AI-generated and insufficiently edited. The overall presentation comes across as unfinished. Figure 2 is very difficult to read: many labels are tiny, text overlaps, and the caption does not clearly explain the subpanels labeled a, b, c, d, e, and j. Figure 3 also contains substantial overlap, making it hard to understand what it is trying to show. The “Planning and State Change” subsection in Related Work still contains unresolved question marks. These issues suggest that the paper needs much more careful editing.

Questions for authors

Neither the benchmark nor the method appears to be compared with established baselines or relevant prior work. The evaluation seems limited to comparisons within the authors’ own setup. How can these experiments establish the validity of the benchmark and the effectiveness of the method? Could the authors include comparisons with existing approaches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.