acceptodds
Under review as a conference paper at ICLR 2027

DataMemo: Turning Acquired Web Data into Reusable Agent Memory

Abstract

Web analytical tasks require agents to collect, join, and compute over records dispersed across web sources. When successive requests concern the same source, reacquiring those records wastes time and tokens, while retaining them only in unstructured interaction history makes coverage and interpretation difficult to track. Despite these challenges, dedicated benchmarks and systematic studies of how agents should manage and reuse web data remain limited. To evaluate this setting, we introduce DataMemoBench, a benchmark comprising 3,799 questions across eight domains to evaluate agents on successive analytical requests over shared web sources. We also present DataMemo, a relational agentic data memory framework that enables agents to incrementally store extracted records in a relational database, reuse them across questions, and acquire additional data as needed. Our evaluation shows that a continued-session baseline retains prior dialogue and files without explicitly representing the coverage and interpretation of retained data, while episodic and procedural memory still incur substantial overhead from repeated data acquisition and processing. With GPT-5.5 and Claude Opus 4.8, DataMemo reduces token cost by 36–46% and wall-clock time by 6–13%, while achieving a 5.8–11.8% relative improvement in accuracy over the strongest baseline, with efficiency gains growing as more questions reuse the accumulated data. Together, DataMemoBench and DataMemo provide an evaluation testbed and an effective framework for accurate and efficient web analysis.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent memory, web agents, benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FnfvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DataMemoBench, a benchmark for evaluating data reuse across successive web analytical tasks, and DataMemo, an agent memory framework that combines relational data storage with procedural notes. The proposed framework enables agents to reuse previously acquired records, retrieve missing data, and refresh outdated information when necessary.

Critical strengths

The paper addresses an important and practical problem in long-running agent systems. Effective memory management and data reuse can substantially reduce redundant web requests, token consumption, and execution costs. The proposed framework is appealing and practically motivated. The construction of DataMemoBench also represents substantial effort, covering 3,799 questions across 28 websites and eight domains. The experimental results are encouraging, demonstrating improvements in both accuracy and efficiency over the evaluated memory baselines.

Critical weaknesses
  1. Insufficient specification of critical memory-management decisions. My primary concern is that several critical decisions in Algorithm 1 are largely delegated to the LLM without sufficiently explicit or reliable decision criteria. First, the condition “D does not cover q” determines whether additional data acquisition is necessary. Although the authors describe coverage metadata and related checks, it remains unclear how the agent reliably determines whether the stored data is sufficient. An incorrect judgment could cause the agent to skip necessary acquisition and compute answers from incomplete records. Second, the algorithm does not explicitly distinguish between information that has not yet been acquired and information that is genuinely unavailable from the source. While the appendix provides prompt-level instructions for handling unanswerable questions, the actual decision criteria remain unclear. Third, similar concerns arise with the “PROBE” and “REFRESH” operations. The paper describes checking record counts, identifiers, and sampled records, but does not sufficiently specify when freshness checks are triggered or how their reliability is assessed. Frequent verification may offset some efficiency gains, whereas insufficient verification risks reusing outdated information. Could the authors clarify whether these decisions are governed by explicit algorithms or primarily by LLM reasoning? How does the system handle incorrect coverage judgments, determine genuine information unavailability, and balance freshness checking overhead against the risk of stale data?

  2. Limited methodological novelty and insufficient baseline comparisons. Although DataMemo is a useful framework, its methodological novelty appears somewhat limited. Relational memory and database-backed reasoning have been explored in prior work, such as ChatDB (Hu et al., 2023) and Table-Augmented Generation (Biswal et al., 2024). Procedural and reasoning memory also have established precedents, including ReasoningBank (Ouyang et al., 2026). The most distinctive aspect of DataMemo appears to be its integration of data acquisition, reuse, and refresh within a persistent memory framework. However, these operations largely rely on LLM decisions guided by metadata and prompts, rather than clearly specified new algorithmic mechanisms. A related concern is whether the reported improvements primarily arise from persistent data storage itself. For example, would an agent that stores acquired web records in JSON/CSV files and analyses them using Python or Pandas achieve comparable results? The continued-session baseline does not fully address this question. I would therefore encourage the authors to include a simple persistent raw-data cache baseline under comparable execution conditions. An ablation separating relational data memory from procedural notes would also help establish the contributions of the individual components.

  3. Remaining concerns regarding benchmark ground truth validation. The authors employ several validation procedures, including independent computations, external factual checks, and human auditing. These efforts are appreciated. However, human question level auditing covers only 280 of the 3,799 benchmark questions, while much of the remaining validation relies on automated procedures. Agreement between independent computations does not necessarily eliminate errors arising from shared source data limitations or incorrect interpretations. This concern is particularly relevant to complex analytical questions and unanswerable probes. I would encourage the authors to strengthen targeted validation of higher risk questions. One possible approach is to decompose complex reference answers into independently verifiable atomic factual components, allowing more systematic verification without requiring exhaustive human annotation.

Questions for authors

Overall, the paper addresses a compelling problem, presents substantial benchmark construction effort, and demonstrates promising empirical improvements. I consider the proposed framework practically valuable, particularly for reducing redundant data acquisition and improving the efficiency of repeated web analysis. My main reservation concerns the limited methodological novelty and the substantial reliance on LLM driven judgments for critical memory management decisions. More explicit decision mechanisms and stronger comparisons against simple persistent storage baselines would considerably strengthen the contribution.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Fnfv (1/3)

We thank the reviewer for recognizing the practical value of our work and for the constructive suggestions . We address each concern below.

W1

Could the authors clarify whether these decisions are governed by explicit algorithms or primarily by LLM reasoning? How does the system handle incorrect coverage judgments, determine genuine information unavailability, and balance freshness checking overhead against the risk of stale data?

We agree with the reviewer that these decisions involve a difficult trade-off: repeatedly checking live sources is costly and can itself introduce extraction errors, while assuming stored data are sufficient risks incomplete or stale answers. Indeed, the no-memory baseline achieves 51.1% accuracy at USD 41.23 per testbed, compared with DataMemo’s 73.3% at USD 14.06 (Table 2), showing that repeated acquisition does not ensure correctness.

DataMemo adopts an agentic memory approach to address this trade-off: it provides instructions, tools, and metadata to guide the agent’s decisions. Agentic memory design has been studied in other works: MemGPT (Packer et al., 2023), A-Mem (Xu et al., 2025), and MIRIX (Wang and Chen, 2025). Our focus is on enabling agents to autonomously acquire, reuse, and maintain data, with the following decision guidance:

  • Coverage: The agent checks whether stored entities, fields, and records cover the question’s scope, verifies pagination and source-reported totals where available, and acquires missing records when gaps are detected.
  • Information unavailability: Missing data in memory requires source inspection; they do not establish that the source lacks the information. An unanswerable judgment requires evidence that the designated source does not provide the requested information. Acquisition notes preserve identified source limitations for subsequent questions.
  • Freshness: The agent uses provenance metadata, including fetch timestamps, to assess whether source changes could affect the answer. When needed, it probes source counts, identifiers, and sampled field values; detected changes trigger updates to affected data before answering.

These checks support detection and repair of incomplete or stale memory, but do not guarantee that every incorrect judgment is detected. Their application remains agent-driven.

To assess the contribution of metadata to accuracy and execution cost, we conducted two ablations using GPT-5.5 under the same agent framework.

ConfigurationAccuracy ↑Cost ($) ↓Time (min) ↓
DataMemo0.73314.0693.0
DataMemo without provenance metadata0.68314.4796.5
DataMemo without all metadata0.66515.4497.9

With provenance metadata, DataMemo achieves 5.0 percentage points higher accuracy than the corresponding ablation, while using less cost and time. Relative to removing all metadata, the accuracy gain is 6.8 points, again with lower cost and runtime.

These results demonstrate that existing agents can use the provided metadata to manage data memory more effectively. DataMemoBench also provides a testbed for future research comparing alternative memory-management policies, including coverage assessment and freshness checking, in terms of accuracy and execution cost.

References

Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023). MemGPT: Towards LLMs as operating systems. arXiv. https://doi.org/10.48550/arXiv.2310.08560

Wang, Y., & Chen, X. (2025). MIRIX: Multi-agent memory system for LLM-based agents. arXiv. https://doi.org/10.48550/arXiv.2507.07957

Xu, W., Liang, Z., Mei, K., Gao, H., Tan, J., & Zhang, Y. (2025). A-Mem: Agentic memory for LLM agents. Advances in Neural Information Processing Systems, 38, 17577–17604. https://doi.org/10.52202/085713-0593

AuthorsOpenReview ↗

Response to Reviewer Fnfv (2/3)

W2

Although DataMemo is a useful framework, its methodological novelty appears somewhat limited. Relational memory and database-backed reasoning have been explored in prior work, such as ChatDB (Hu et al., 2023) and Table-Augmented Generation (Biswal et al., 2024). Procedural and reasoning memory also have established precedents, including ReasoningBank (Ouyang et al., 2026).

The distinction is that ChatDB and TAG focus on reasoning with database-backed information, while ReasoningBank focuses on learning reusable strategies from experience. DataMemo instead studies how agents turn externally acquired data into reusable memory across successive analytical tasks. Its relational records support new computations, while procedural notes and provenance metadata guide subsequent acquisition, computation, and maintenance.

Our contribution therefore lies not in introducing a new database or reasoning algorithm, but in framing and supporting agent-managed data acquisition, reuse, and maintenance as a unified memory problem, together with DataMemoBench for evaluating accuracy and efficiency in this setting.

MethodInformationMechanismPrimary Focus
ChatDBUser-provided information saved as databaseSQL-based memory operationsMulti-step reasoning
TAGExisting database tablesDB computation + LM reasoningDatabase question answering
ReasoningBankDistilled reasoning strategiesExperience-based retrieval and reuseCross-task reasoning improvement
DataMemo (Ours)Acquired records + procedural notesIncremental acquisition, reuse, and refreshCross-task data lifecycle management

Would an agent that stores acquired web records in JSON/CSV files and analyses them using Python or Pandas achieve comparable results? An ablation separating relational data memory from procedural notes would also help establish the contributions of the individual components.

We added both comparisons suggested by the reviewer, using GPT-5.5 under the same agent framework and evaluation budget:

  • Persistent raw-data cache: Replaces DataMemo’s table interface with freeform caching while keeping the other components unchanged. The agent retains records across batches in formats such as JSON, CSV, and HTML and analyzes them using the same shell and Python environment.
  • DataMemo without notes: Retains relational storage and metadata but removes procedural notes, isolating their contribution.
ConfigurationAccuracy ↑Cost ($) ↓Time (min) ↓
DataMemo0.73314.0693.0
Raw-data cache0.66226.25108.0
DataMemo without notes0.69019.06104.0

These results yield two findings:

  1. Relative to raw caching, DataMemo improves average accuracy by 7.1 percentage points while reducing cost by 46.4% and runtime from 108 to 93 minutes. Raw caching lacks a standardized interface for querying and updating records; varying file-parsing code can introduce inconsistencies and make missing or incorrect records harder to identify and repair.
  2. Removing notes lowers accuracy by 4.3 percentage points and increases both cost and runtime. Writing notes encourages the agent to make acquisition procedures, field interpretations, and computation conventions explicit during memory construction. These notes then guide subsequent acquisition, correction, and reuse.

These findings support DataMemo’s integration of relational records, metadata, and procedural notes for reusable agent memory. We will incorporate them into the revised paper.

AuthorsOpenReview ↗

Response to Reviewer Fnfv (3/3)

W3

I would encourage the authors to strengthen targeted validation of higher risk questions. One possible approach is to decompose complex reference answers into independently verifiable atomic factual components, allowing more systematic verification without requiring exhaustive human annotation.

We added two additional checks to further strengthen our benchmark validation following the reviewer’s suggestions.

First, we expanded the verification of the benchmark with five human auditors who reviewed 520 questions, including 429 complex analytical questions and 91 unanswerable probes. Auditors reviewed the traces that reproduce reference answers, including code, intermediate results, and underlying database scope. This audit identified one reference-answer error, which we corrected.

Second, we decomposed reference answers for 666 questions amenable to record-level verification into 2,849 individually checkable factual claims and checked them against the designated source websites. The automated checks labeled 2,082 claims as supported, flagged 182 contradictions, and left 585 unresolved. We manually adjudicated all 182 flagged contradictions: only six claims represented genuine defects, affecting four incorrect reference answers. All four questions have been corrected. The reference-answer error identified by the human audit was also detected by this source-based check.

The benchmark questions are anchored to fixed historical periods or completed events to avoid routinely changing answers. Nevertheless, we acknowledge that data on live websites can change over time. We therefore plan to repeat the automated checks periodically and manually review flagged discrepancies, preserving prior artifacts and recording corrections or retirements in versioned benchmark updates.

After considering the other reviewers' comments and the authors' rebuttal, the main concerns have been further addressed. However, I have decided to retain my original score, which already indicates that the paper is acceptable.

Reviewer_Z9rKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces DATAMEMO, a web-agent memory system that stores acquired records in a relational database, together with procedural notes, coverage metadata, and freshness checks. It also presents DATAMEMOBENCH, containing 3,799 analytical questions across 28 websites and eight domains. Experiments show that DATAMEMO generally improves accuracy while reducing cost and runtime relative to no-memory, procedural-memory, episodic-memory, and continued-session baselines.

Critical strengths
  1. Addresses an important limitation of web agents: repeated acquisition of overlapping data.
  2. Presents a simple and practical memory representation suited to analytical queries.
  3. Includes a substantial benchmark with careful answer construction and validation.
Critical weaknesses
  1. While GPT-5.5 grades all outputs, the paper does not provide sufficient human agreement analysis on actual agent responses, especially for lists, rankings, and unanswerable questions.
  2. It seems that external evaluation covers selected repeated single-site tasks and compares primarily against ReasoningBank.
  3. My main concern is that DATAMEMO receives relational storage, schema guidance, coverage tracking, and specialized prompting. The baselines lack an equally capable persistent structured-data interface, making it difficult to identify the source of improvement.
Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Z9rK (1/2)

We thank the reviewer for the constructive feedback. We have added comparisons to help identify the source of improvement, along with a human audit of actual agent responses. Please let us know if any concerns remain; we would be happy to provide further clarification or results. We address each concern below.

W3:

My main concern is that DATAMEMO receives relational storage, schema guidance, coverage tracking, and specialized prompting. The baselines lack an equally capable persistent structured-data interface, making it difficult to identify the source of improvement.

All conditions share the same shell/Python execution environment and data-handling rules (Appendix B.2), with condition-specific instructions for operating and reusing their respective memory interfaces. Additionally, the continued-session baseline preserves saved work across batches and conversation compaction and is explicitly instructed to reuse it.

To better identify the sources of improvement, we added a raw-data cache comparison replacing DataMemo's relational storage with arbitrary formats (e.g., JSON, HTML, CSV), while retaining notes and metadata. We also added three ablations removing notes, provenance metadata (coverage information and fetch timestamps), or all metadata (additionally removing data schema digest). All experiments use the same agent framework and GPT-5.5.

ConfigurationAccuracy ↑Cost ($) ↓Time (min) ↓
DataMemo0.73314.0693.0
Raw-data cache0.66226.25108.0
DataMemo without notes0.69019.06104.0
DataMemo without provenance metadata0.68314.4796.5
DataMemo without all metadata0.66515.4497.9
  1. Relational storage improves reuse over raw caching. DataMemo improves accuracy by 7.1 percentage points while reducing cost and runtime. Its standardized query and update interface reduces the need to parse and reconcile heterogeneous files across requests.
  2. Procedural notes support data interpretation and reuse. Removing notes lowers accuracy by 4.3 points and increases cost and runtime. Notes document acquisition procedures and computation conventions, providing guidance for subsequent acquisition, correction, and reuse.
  3. Metadata supports verification and memory navigation. Removing provenance metadata or all metadata lowers accuracy by 5.0 or 6.8 points, respectively, without efficiency gains. Provenance informs coverage and freshness judgments, while schema digests help the agent identify and interpret stored tables without repeated inspection.

Together, these results support DataMemo’s integrated memory design beyond persistent storage alone. We will incorporate these comparisons in the revision.

AuthorsOpenReview ↗

Response to Reviewer Z9rK (2/2)

W1:

While GPT-5.5 grades all outputs, the paper does not provide sufficient human agreement analysis on actual agent responses, especially for lists, rankings, and unanswerable questions.

We agree that validating the judge against human assessments of actual agent responses is essential. To address this, we added a blinded human audit of 600 actual agent responses, stratified across answer types, both models, and memory conditions. Six annotators worked in three pairs, with both annotators in each pair independently grading the same 200 responses without seeing the model judgment.

The table below reports each annotator’s agreement with the judge before adjudication. Across all 1,200 individual judgments, initial human–judge agreement was 96.3% (1,156/1,200; Cohen’s κ = 0.90).

AnnotatorAgreement countAgreementCohen’s κ
1A184/20092.0%0.768
1B198/20099.0%0.973
2A197/20098.5%0.961
2B199/20099.5%0.987
3A197/20098.5%0.960
3B181/20090.5%0.719
Overall1,156/1,20096.3%0.900

The lead authors manually adjudicated all disagreements and found that most arose from human annotation errors, such as overlooking missing list members, numerical values outside the tolerance, or incorrect entities. The only confirmed judge error involved an ambiguously worded response that correctly stated the requested information was unavailable but also provided historical details, leading the judge to mark it incorrect.

After adjudication, agreement between the judge and the adjudicated human labels was 99.8% (599/600; Cohen’s κ = 0.996), including 60/60 for lists, 60/60 for rankings, and 59/60 for unanswerable questions. These results support the reliability of our automated grading. Most reference answers are numbers or short facts, making correctness less ambiguous to assess than in long, open-ended responses.

W2:

It seems that external evaluation covers selected repeated single-site tasks and compares primarily against ReasoningBank.

We selected WebArena and WebChoreArena as the open-source benchmarks we could identify with sufficient tasks over shared websites to support our target setting. However, their simulated, self-hosted environments leave open how effectively agents acquire and reuse data from real online sources. Addressing this gap is a central contribution of DataMemoBench: it enables systematic evaluation of successive analytical requests over live websites. Our main experiments therefore compare different memory conditions on DataMemoBench, while the external experiments provide complementary sanity checks on the existing synthetic benchmarks.

For these external comparisons, we use ReasoningBank (Ouyang et al., 2026), a strong reasoning-memory baseline originally evaluated on WebArena against no-memory, Synapse, and AWM agents. Subsequent studies have adopted it as a comparison on WebArena and related web-agent benchmarks (Wang et al., 2026; Hajimiri et al., 2026). It provides a relevant comparison for evaluating agents’ reuse of past experience across successive tasks.

References

Hajimiri, S., Aminbeidokhti, M., Dolz, J., Ben Ayed, I., Laradji, I. H., Gella, S., & Gontier, N. (2026). Are online skill and memory modules always worth their tokens? A budget-constrained study of web agents. In H.-Y. Lee, S. Dev, J. J. Li, & D. Elliott (Eds.), Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (pp. 37676–37693). Association for Computational Linguistics. https://doi.org/10.18653/2026.emnlp-main.1657

Ouyang, S., Yan, J., Hsu, I.-H., Chen, Y., Jiang, K., Wang, Z., Han, R., Le, L. T., Daruki, S., Tang, X., Tirumalashetty, V., Lee, G., Rofouei, M., Lin, H., Han, J., Lee, C.-Y., & Pfister, T. (2026). ReasoningBank: Scaling agent self-evolving with reasoning memory. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=jL7fwchScm

Wang, H., Dong, G., He, L., Zhang, Z., Luo, J., Xue, C., Liu, C., & Tang, H. (2026). MemGuard: Persisting verifier signals for LLM-agent memory governance. In H.-Y. Lee, S. Dev, J. J. Li, & D. Elliott (Eds.), Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (pp. 9677–9708). Association for Computational Linguistics. https://doi.org/10.18653/2026.emnlp-main.423

Dear Reviewer Z9rK,

Thank you again for your thoughtful review and constructive suggestions. As the discussion deadline is approaching, we wanted to follow up on our responses. We have added a raw-data cache baseline and ablations to clarify the sources of improvement, a human audit of 600 agent responses to validate the judge, and clarification of our external evaluation. We hope these additions address your concerns and would appreciate hearing whether you have any remaining concerns or further questions.

Thank you again for your time and consideration!

Dear Reviewer Z9rK,

Thank you again for your thoughtful feedback. Your suggestions on clarifying the sources of improvement and strengthening human validation were valuable and have helped us improve the paper.

In our response and revision, we have added a raw-data cache baseline and component ablations, conducted an extensive human audit of 600 actual agent responses, and clarified the shared execution environment and data-handling rules to make the fairness of our baseline comparisons more explicit.

We sincerely appreciate your suggestions and would be grateful to hear whether these additions address your concerns. If any concerns remain, we would welcome the opportunity to discuss them and provide further clarification. We look forward to hearing from you.

Reviewer_qi7ROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the limitation of current LLM web agents regarding the lack of systematic data management and reuse across successive, data-intensive analytical tasks over the same web source. The paper introduces DataMemo, an agent-managed memory framework that structures acquired data into a local relational database accompanied by a note layer capturing rules for interpretation. This approach yields a significant accuracy improvement for Claude Opus 4.8 and GPT-5.5 over DataMemoBench which the authors propose.

Critical strengths

It's interesting to see a relational database centric approach to managing agent memory, which is different from prior agent memory systems (like MemGPT or Synapse) which store episodic traces or unstructured notes. The constructed benchmark can be pretty useful in evaluating agent episode reuse methods. The proposed DataMemo approach shows significant efficiency improvements, along with comprehensive analysis & ablation studies.

Critical weaknesses

The system's accuracy can be heavily bottlenecked by its text-to-SQL generation performance. Further, the paper does not thoroughly discuss the failure modes when the agent initially mis-designs the database schema.

The paper seems limited to successive questions over a single shared web source. Real-world analytical workflows often require cross-source evidence aggregation. Extending the approach to multi-source configurations requires non-trivial amount of work.

Questions for authors
  1. How does the framework handle schema design failures during the cold-start phase?

  2. To what extent do SQL syntax errors or join mismatches limit performance?

2 comments
AuthorsOpenReview ↗

Response to Reviewer qi7R

We thank the reviewer for the constructive feedback and positive assessment of DataMemo and DataMemoBench. We address the concerns about SQL generation, initial schema design, and multi-source workflows below.

W1 + W2

To what extent do SQL syntax errors or join mismatches limit performance? How does the framework handle schema design failures during the cold-start phase?

We extended the error analysis in §5.2 to examine SQL generation and initial schema defects.

  1. SQL syntax errors and join mismatches rarely caused final-answer failures. We verified SQL errors by testing whether correcting SQL alone on the database available at answer time recovered the reference answer.
Final-answer error attributionGPT-5.5Opus 4.8
SQL syntax/execution error as the primary cause0%0%
Verified join mismatch0.14%0.15%
SQL logic errors6.59%3.58%

Percentages are shares of analyzed incorrect answers. SQL logic errors mainly involved filtering or scope conditions.

  1. Initial schemas can be revised through the memory tools. DataMemo guides agents to preserve entity-level records, normalize multivalued fields, and retain source text alongside parsed values. Subsequent batches receive the current schema and coverage information, enabling the agent to modify or rebuild tables and acquire missing data. Initial schema defects contributed to 0.86% / 1.94% of analyzed incorrect answers for GPT-5.5 / Opus 4.8 after the construction batch. These counts exclude ordinary schema expansion for new information needs and may overlap with query-error attribution.

A Stanford website trace illustrates recovery during cold start. Initial primary keys caused records to be overwritten, leaving 96,044 sections against a declared 100,181. Coverage feedback exposed the gap, and the agent rebuilt the tables with surrogate keys before submitting the first batch, restoring the full count. A later query correctly returned 2,725 sections, versus 2,281 when replayed on the original table.

We will include these analyses and the recovery example in the revision.

W3

The paper seems limited to successive questions over a single shared web source. Real-world analytical workflows often require cross-source evidence aggregation. Extending the approach to multi-source configurations requires non-trivial amount of work.

We agree that extending DataMemo to broader multi-source workflows is an interesting future direction. As an initial test, we added three two-site workflows with 90 questions (30 per workflow, including 49 hard multi-hop questions), each requiring evidence from both sources. These cover state–year joins (America’s Health Rankings + IRS), course-catalog alignment (Stanford + Berkeley), and identity resolution (OpenReview + dblp).

We extended the source configuration while keeping DataMemo’s memory tools, prompts, and Build/Evolve protocol unchanged. The agent integrated both sources into a shared relational store without source-specific integration code. We evaluated all five conditions using the same model, batching, and timeouts as the main experiments.

Memory conditionAccuracyCost (USD/workflow)Time (min/workflow)
No memory0.9336.7535.1
Procedural0.9223.0041.3
Episodic0.9115.4635.7
Continued session0.9566.9716.3
DataMemo0.9672.9416.7

DataMemo achieved the highest accuracy and lowest cost, reducing cost by 58% relative to continued session with similar wall-clock time. These preliminary results support the feasibility of cross-source data integration and reuse. Building a comprehensive multi-source benchmark and systematically addressing heterogeneous-source alignment remain beyond this paper’s scope. We will highlight these as future work.

Dear Reviewer qi7R,

Thank you again for your positive assessment and careful review. As the discussion deadline is approaching, we wanted to follow up on our responses. We have added an analysis of SQL errors and initial schema defects, a concrete example of schema recovery, and a multi-source pilot covering three two-site workflows with 90 questions. We hope these additions address your concerns and would appreciate hearing whether you have any remaining concerns or further questions.

Thank you for the time and effort!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Paper Revision

We thank all reviewers for their thoughtful and constructive feedback. We have revised the manuscript, with changes highlighted in blue, and provided additional results and clarifications in our responses. We summarize the main updates below.

Additional Experiments and Analyses

  • Persistent raw-data cache comparison and component ablations. We added a raw-data cache baseline and three ablations removing procedural notes, provenance metadata, or all supplied metadata under the same agent framework and evaluation budget. DataMemo exceeds raw caching by 7.1 percentage points in accuracy at 46.4% lower cost. Removing notes or metadata reduces accuracy and increases cost and runtime. These results are reported in §5.5, Table 8. (Reviewers Z9rK, Fnfv)

  • Human validation of automated grading. We added a blinded audit of 600 actual agent responses across answer types, both models, and all five memory conditions. Initial human–judge agreement was 96.3%, and agreement with adjudicated labels was 99.8%. We report both results separately in §5.1, including results for lists, rankings, and unanswerable questions. (Reviewer Z9rK)

  • Targeted benchmark validation. We added a human audit of 520 questions, including 429 complex analytical questions and 91 unanswerable probes, alongside source-based verification of 2,849 factual claims from 666 questions. Manual adjudication of flagged contradictions identified defects affecting four questions, which we corrected; these include the one reference-answer error found by the human audit. These checks are described in §4. (Reviewer Fnfv)

  • SQL and initial schema failure analysis. We added an analysis distinguishing SQL syntax/execution errors, join mismatches, SQL logic errors, and initial schema defects. We also included a concrete recovery example showing how coverage feedback helped the agent detect overwritten records and rebuild the tables. The analysis and example appear in Appendix C.5, Table C.4. (Reviewer qi7R)

  • Preliminary multi-source evaluation. In our response to Reviewer qi7R, we reported results on three two-site workflows comprising 90 questions, each requiring evidence from both sources. Across all five memory conditions, DataMemo achieved the highest accuracy and lowest cost, reducing cost by 58% relative to continued session with similar runtime. We present this as an initial feasibility study, with broader multi-source evaluation remaining future work. (Reviewer qi7R)

Clarifications and Revised Descriptions

  • Relationship to prior work. We expanded §2 to clarify how DataMemo differs from database-backed reasoning and procedural memory approaches, emphasizing its integration of acquisition, reuse, and maintenance of externally acquired records across successive analytical requests. (Reviewer Fnfv)

  • Shared execution environment and baseline capabilities. We clarified in §5.1 that all conditions share the shell/Python environment and data-handling rules, and that continued sessions preserve saved files through conversation compaction and are instructed to reuse them. (Reviewer Z9rK)

We hope these revisions help address the reviewers’ concerns and clarify our contribution. We welcome further discussion. Please let us know if any concerns remain, and we would be happy to provide further clarification or results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.