- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces DiMeR, a benchmark for temporal question answering over sequential knowledge streams. It constructs 1,956 questions from 674 Wikipedia articles with source-linked reference evidence. The same question, answer, and evidence are compared under no-evidence, direct-evidence, immediate, and delayed conditions. An additional diagnostics condition on direct-evidence success reintroduces the reference evidence after delayed ingestion. Answer accuracy and final citation coverage are jointly analyzed across base models and adapted memory backends.
Critical strengths
-
The matched conditions are useful: they hold the historical question and target fixed while changing the input prefix. The appendix specifies state isolation and independent reconstruction, preventing earlier questions or answers from contaminating later conditions. Conditioning on the same model's direct-evidence successes also reduces one important source of confounding.
-
Jointly inspecting answers and citations provides useful behavioral information. In particular, Appendix F.2 shows faithful quotation of the reference evidence alongside a temporally incompatible answer. The paper also includes cardinality/relation analyses and paired uncertainty estimates, and explicitly acknowledges the limits of interpreting citations and reinjection as internal mechanisms.
Critical weaknesses
-
The reported results cannot currently be reconciled across tables and declared sample sizes. All three memory systems have different Immediate-Memory accuracies in Tables 2 and 9; for example, Mem0 Hard is 35.96% versus 44.06%. Appendix E.2 states that every model uses 1,219 Easy and 737 Hard questions, but several percentages, including Mem0's Table 9 Immediate values, cannot arise from any integer correct count under these denominators and the defined accuracy metric. Table 8's Easy direct-evidence-success counts also differ from those implied by Table 2, and Table 12 reports Qwen/GLM Easy gains of 2.47/1.81 rather than Table 9's 2.38/1.72.
-
The diagnostic measurements need stronger validation for the claimed separation. Final citation coverage does not independently measure retention or evidence sufficiency, and reinjection changes recency, repetition, and memory processing. I appreciate that E.2 explicitly acknowledges this: the supported contribution is a behavioral diagnostic, rather than an identification of internal failure mechanisms. Its validity also depends on the reference targets. B.1 removes month/day precision while retaining one answer; F.3 uses an interval-target convention for a 2009–2010 question despite a July 2010 team change. That convention and acceptable alternatives are not operationally specified. Fact-level verification and a few inspected cases do not quantify ambiguity or citation fidelity across the benchmark.
Questions for authors
-
Can you reconcile Tables 2, 8, 9, and 12 using explicit question IDs, effective denominators, exclusions, and run/config identifiers? Please explain whether different releases, technical recoveries, or averaging are involved, and regenerate the direct-evidence-success subsets and paired statistics from a consistent set of predictions.
-
Which observable failure categories does DiMeR distinguish, and which remain unidentifiable? Please specify the year-level interval/answer convention and treatment of multiple valid answers and alternative sufficient evidence, and provide a quantified human audit of these targets and citation fidelity. Item-level answer–coverage transitions would also help establish what the reinjection diagnostic adds beyond aggregate differences.