- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DataMemoBench, a benchmark for evaluating data reuse across successive web analytical tasks, and DataMemo, an agent memory framework that combines relational data storage with procedural notes. The proposed framework enables agents to reuse previously acquired records, retrieve missing data, and refresh outdated information when necessary.
Critical strengths
The paper addresses an important and practical problem in long-running agent systems. Effective memory management and data reuse can substantially reduce redundant web requests, token consumption, and execution costs. The proposed framework is appealing and practically motivated. The construction of DataMemoBench also represents substantial effort, covering 3,799 questions across 28 websites and eight domains. The experimental results are encouraging, demonstrating improvements in both accuracy and efficiency over the evaluated memory baselines.
Critical weaknesses
-
Insufficient specification of critical memory-management decisions. My primary concern is that several critical decisions in Algorithm 1 are largely delegated to the LLM without sufficiently explicit or reliable decision criteria. First, the condition “D does not cover q” determines whether additional data acquisition is necessary. Although the authors describe coverage metadata and related checks, it remains unclear how the agent reliably determines whether the stored data is sufficient. An incorrect judgment could cause the agent to skip necessary acquisition and compute answers from incomplete records. Second, the algorithm does not explicitly distinguish between information that has not yet been acquired and information that is genuinely unavailable from the source. While the appendix provides prompt-level instructions for handling unanswerable questions, the actual decision criteria remain unclear. Third, similar concerns arise with the “PROBE” and “REFRESH” operations. The paper describes checking record counts, identifiers, and sampled records, but does not sufficiently specify when freshness checks are triggered or how their reliability is assessed. Frequent verification may offset some efficiency gains, whereas insufficient verification risks reusing outdated information. Could the authors clarify whether these decisions are governed by explicit algorithms or primarily by LLM reasoning? How does the system handle incorrect coverage judgments, determine genuine information unavailability, and balance freshness checking overhead against the risk of stale data?
-
Limited methodological novelty and insufficient baseline comparisons. Although DataMemo is a useful framework, its methodological novelty appears somewhat limited. Relational memory and database-backed reasoning have been explored in prior work, such as ChatDB (Hu et al., 2023) and Table-Augmented Generation (Biswal et al., 2024). Procedural and reasoning memory also have established precedents, including ReasoningBank (Ouyang et al., 2026). The most distinctive aspect of DataMemo appears to be its integration of data acquisition, reuse, and refresh within a persistent memory framework. However, these operations largely rely on LLM decisions guided by metadata and prompts, rather than clearly specified new algorithmic mechanisms. A related concern is whether the reported improvements primarily arise from persistent data storage itself. For example, would an agent that stores acquired web records in JSON/CSV files and analyses them using Python or Pandas achieve comparable results? The continued-session baseline does not fully address this question. I would therefore encourage the authors to include a simple persistent raw-data cache baseline under comparable execution conditions. An ablation separating relational data memory from procedural notes would also help establish the contributions of the individual components.
-
Remaining concerns regarding benchmark ground truth validation. The authors employ several validation procedures, including independent computations, external factual checks, and human auditing. These efforts are appreciated. However, human question level auditing covers only 280 of the 3,799 benchmark questions, while much of the remaining validation relies on automated procedures. Agreement between independent computations does not necessarily eliminate errors arising from shared source data limitations or incorrect interpretations. This concern is particularly relevant to complex analytical questions and unanswerable probes. I would encourage the authors to strengthen targeted validation of higher risk questions. One possible approach is to decompose complex reference answers into independently verifiable atomic factual components, allowing more systematic verification without requiring exhaustive human annotation.
Questions for authors
Overall, the paper addresses a compelling problem, presents substantial benchmark construction effort, and demonstrates promising empirical improvements. I consider the proposed framework practically valuable, particularly for reducing redundant data acquisition and improving the efficiency of repeated web analysis. My main reservation concerns the limited methodological novelty and the substantial reliance on LLM driven judgments for critical memory management decisions. More explicit decision mechanisms and stronger comparisons against simple persistent storage baselines would considerably strengthen the contribution.
4 comments
Response to Reviewer Fnfv (2/3)
W2
Although DataMemo is a useful framework, its methodological novelty appears somewhat limited. Relational memory and database-backed reasoning have been explored in prior work, such as ChatDB (Hu et al., 2023) and Table-Augmented Generation (Biswal et al., 2024). Procedural and reasoning memory also have established precedents, including ReasoningBank (Ouyang et al., 2026).
The distinction is that ChatDB and TAG focus on reasoning with database-backed information, while ReasoningBank focuses on learning reusable strategies from experience. DataMemo instead studies how agents turn externally acquired data into reusable memory across successive analytical tasks. Its relational records support new computations, while procedural notes and provenance metadata guide subsequent acquisition, computation, and maintenance.
Our contribution therefore lies not in introducing a new database or reasoning algorithm, but in framing and supporting agent-managed data acquisition, reuse, and maintenance as a unified memory problem, together with DataMemoBench for evaluating accuracy and efficiency in this setting.
| Method | Information | Mechanism | Primary Focus |
|---|---|---|---|
| ChatDB | User-provided information saved as database | SQL-based memory operations | Multi-step reasoning |
| TAG | Existing database tables | DB computation + LM reasoning | Database question answering |
| ReasoningBank | Distilled reasoning strategies | Experience-based retrieval and reuse | Cross-task reasoning improvement |
| DataMemo (Ours) | Acquired records + procedural notes | Incremental acquisition, reuse, and refresh | Cross-task data lifecycle management |
Would an agent that stores acquired web records in JSON/CSV files and analyses them using Python or Pandas achieve comparable results? An ablation separating relational data memory from procedural notes would also help establish the contributions of the individual components.
We added both comparisons suggested by the reviewer, using GPT-5.5 under the same agent framework and evaluation budget:
- Persistent raw-data cache: Replaces DataMemo’s table interface with freeform caching while keeping the other components unchanged. The agent retains records across batches in formats such as JSON, CSV, and HTML and analyzes them using the same shell and Python environment.
- DataMemo without notes: Retains relational storage and metadata but removes procedural notes, isolating their contribution.
| Configuration | Accuracy ↑ | Cost ($) ↓ | Time (min) ↓ |
|---|---|---|---|
| DataMemo | 0.733 | 14.06 | 93.0 |
| Raw-data cache | 0.662 | 26.25 | 108.0 |
| DataMemo without notes | 0.690 | 19.06 | 104.0 |
These results yield two findings:
- Relative to raw caching, DataMemo improves average accuracy by 7.1 percentage points while reducing cost by 46.4% and runtime from 108 to 93 minutes. Raw caching lacks a standardized interface for querying and updating records; varying file-parsing code can introduce inconsistencies and make missing or incorrect records harder to identify and repair.
- Removing notes lowers accuracy by 4.3 percentage points and increases both cost and runtime. Writing notes encourages the agent to make acquisition procedures, field interpretations, and computation conventions explicit during memory construction. These notes then guide subsequent acquisition, correction, and reuse.
These findings support DataMemo’s integration of relational records, metadata, and procedural notes for reusable agent memory. We will incorporate them into the revised paper.
Response to Reviewer Fnfv (3/3)
W3
I would encourage the authors to strengthen targeted validation of higher risk questions. One possible approach is to decompose complex reference answers into independently verifiable atomic factual components, allowing more systematic verification without requiring exhaustive human annotation.
We added two additional checks to further strengthen our benchmark validation following the reviewer’s suggestions.
First, we expanded the verification of the benchmark with five human auditors who reviewed 520 questions, including 429 complex analytical questions and 91 unanswerable probes. Auditors reviewed the traces that reproduce reference answers, including code, intermediate results, and underlying database scope. This audit identified one reference-answer error, which we corrected.
Second, we decomposed reference answers for 666 questions amenable to record-level verification into 2,849 individually checkable factual claims and checked them against the designated source websites. The automated checks labeled 2,082 claims as supported, flagged 182 contradictions, and left 585 unresolved. We manually adjudicated all 182 flagged contradictions: only six claims represented genuine defects, affecting four incorrect reference answers. All four questions have been corrected. The reference-answer error identified by the human audit was also detected by this source-based check.
The benchmark questions are anchored to fixed historical periods or completed events to avoid routinely changing answers. Nevertheless, we acknowledge that data on live websites can change over time. We therefore plan to repeat the automated checks periodically and manually review flagged discrepancies, preserving prior artifacts and recording corrections or retirements in versioned benchmark updates.
After considering the other reviewers' comments and the authors' rebuttal, the main concerns have been further addressed. However, I have decided to retain my original score, which already indicates that the paper is acceptable.
Response to Reviewer Fnfv (1/3)
We thank the reviewer for recognizing the practical value of our work and for the constructive suggestions . We address each concern below.
W1
We agree with the reviewer that these decisions involve a difficult trade-off: repeatedly checking live sources is costly and can itself introduce extraction errors, while assuming stored data are sufficient risks incomplete or stale answers. Indeed, the no-memory baseline achieves 51.1% accuracy at USD 41.23 per testbed, compared with DataMemo’s 73.3% at USD 14.06 (Table 2), showing that repeated acquisition does not ensure correctness.
DataMemo adopts an agentic memory approach to address this trade-off: it provides instructions, tools, and metadata to guide the agent’s decisions. Agentic memory design has been studied in other works: MemGPT (Packer et al., 2023), A-Mem (Xu et al., 2025), and MIRIX (Wang and Chen, 2025). Our focus is on enabling agents to autonomously acquire, reuse, and maintain data, with the following decision guidance:
These checks support detection and repair of incomplete or stale memory, but do not guarantee that every incorrect judgment is detected. Their application remains agent-driven.
To assess the contribution of metadata to accuracy and execution cost, we conducted two ablations using GPT-5.5 under the same agent framework.
With provenance metadata, DataMemo achieves 5.0 percentage points higher accuracy than the corresponding ablation, while using less cost and time. Relative to removing all metadata, the accuracy gain is 6.8 points, again with lower cost and runtime.
These results demonstrate that existing agents can use the provided metadata to manage data memory more effectively. DataMemoBench also provides a testbed for future research comparing alternative memory-management policies, including coverage assessment and freshness checking, in terms of accuracy and execution cost.
References
Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023). MemGPT: Towards LLMs as operating systems. arXiv. https://doi.org/10.48550/arXiv.2310.08560
Wang, Y., & Chen, X. (2025). MIRIX: Multi-agent memory system for LLM-based agents. arXiv. https://doi.org/10.48550/arXiv.2507.07957
Xu, W., Liang, Z., Mei, K., Gao, H., Tan, J., & Zhang, Y. (2025). A-Mem: Agentic memory for LLM agents. Advances in Neural Information Processing Systems, 38, 17577–17604. https://doi.org/10.52202/085713-0593