- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies temporal validity in LLM-agent memory, where a semantically relevant memory may be expired, cancelled, superseded, or valid only for a particular historical period. The authors introduce TimeBound, a synthetic benchmark containing 1000 examples across eight temporal-memory task families. Each memory includes observation time, event time, validity interval, status, and an optional update relation. The authors also propose TimeBound-RAG, a diagnostic retriever that combines semantic relevance with temporal compatibility and an obsolete-memory penalty. Across four open-weight readers, it achieves mean relaxed accuracy of 0.719, compared with 0.475 for Semantic RAG and 0.537 for Full History.
Critical strengths
-
The benchmark provides a controlled decomposition of several temporal-memory phenomena. Separating expiration, cancellation, conflicting updates, delayed observations, duration, recurrence, rescheduling, and retrospective queries is useful for examining different failure modes. Computing gold answers and evidence from structured states also makes the intended task semantics easier to inspect.
-
The paper reports evidence-level metrics alongside answer accuracy and includes task-level results. Table 4 makes important limitations visible, including poor elapsed-time performance and substantial degradation on historical time-window queries. I think reporting these negative results is useful, although their causes need a more careful analysis.
Critical weaknesses
-
The definition and evaluation of temporal validity are not fully consistent with the historical queries in the benchmark. As written, Section 3.1 excludes superseded memories even when their validity intervals contain the queried time. Table 4 also shows time-window accuracy falling from 0.840 to 0.160, without a clear explanation. A related concern is that Oracle Evidence receives an invalid-retrieval rate of 0.438 despite an evidence F1 of 1.000. An inactive record can still be the right evidence for a historical question or a cancellation. These issues leave me uncertain whether the benchmark’s validity rule and metric distinguish obsolete facts from useful evidence, which is central to its intended contribution.
-
The answer evaluation does not yet give enough confidence in the reported gains. The main four-reader results report only relaxed accuracy, while the separate diagnostic in Table 8 shows a substantial strict–relaxed gap, including 0.295 versus 0.595 for TimeBound-RAG. More concretely, Figure 8 marks “No information available” as correct when the gold answer states that no currently valid code exists. These answers are not necessarily equivalent: lacking information does not establish that a code has expired or is unavailable. Clear scoring rules, strict results for the main experiments, and a human check of relaxed-only matches are needed to determine how much of the improvement reflects correct temporal reasoning.
Questions for authors
- What specific capability does TimeBound evaluate beyond existing temporal-validity systems and temporal-dialogue or memory-update benchmarks? Could the authors compare task coverage, available annotations, and evaluated failure modes directly?
- Which temporal fields are supplied at inference time? How are validity intervals, status labels, update links, and query reference times obtained? Can the authors separate results with supplied annotations from results with automatically extracted and incrementally maintained annotations?
- How does the admissibility rule handle a currently superseded memory that was valid at the queried historical time? What explains the decrease from 0.840 to 0.160 on time-window retrieval?
- Why does Oracle Evidence have an invalid-retrieval rate of 0.438? Can the metric distinguish using an obsolete fact as the answer from retrieving an obsolete record as evidence for cancellation, expiration, or historical state?
- Can the authors include representative temporal-memory systems and simple structured baselines under comparable reader and metadata conditions? Interval filtering and entity-and-slot update baselines would be particularly informative.
- Could the authors provide exact retrieval-score definitions, the parameter-selection procedure, strict and relaxed scoring rules, and strict results for all four readers? A manually checked sample would help establish whether relaxed matching preserves the complete temporal answer.