acceptodds
Under review as a conference paper at ICLR 2027

MEMENTO: LEVERAGING THE WEB AS A LEARNING SIGNAL FOR LOW-DATA DOMAINS

12 views

Abstract

Real-world tasks often lack large labeled datasets, motivating extensive work on learning in low-data regimes. Existing approaches such as few-shot prompting, instruction tuning, and synthetic data generation, continue to treat labeled or pseudo-labeled data as the primary learning signal. In contrast, human practitioners acquire expertise through repeated, self-directed interaction with the open web, progressively refining both domain knowledge and search strategies. We propose **MEMENTO**, a framework that treats the web as a learning signal rather than a stateless retrieval interface. MEMENTO operates at two levels: within each session, it conducts iterative web exploration via an **Adaptive Exploration Tree** (AET) that decomposes tasks into evolving questions and reflects on intermediate findings; across sessions, it accumulates experience through dual-channel memory, separating declarative knowledge (facts) from procedural knowledge (search strategies). This design enables agents to learn reusable research strategies and domain expertise from trajectories of web interaction without additional model training. We evaluate MEMENTO on three structurally distinct low-data domains: Sales Automation, Legal Outcome Prediction, and Subpopulation Opinion Prediction. Our empirical results show consistent improvement in performance over ReAct baselines (+25.6% on sales, +36.5% on legal research, and 29.6% TVD reduction on opinion prediction), demonstrating that the web can serve as a scalable learning source for acquiring task-specific expertise in data-scarce settings.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
low-data learning, web agents, cross-session memory, adaptive search, tool-augmented language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jR9TOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a framework that uses search trajectories derived from web data, alongside facts, to address the lack of labeled data for low-data tasks. The framework is evaluated on sales automation, legal outcome prediction, and subpopulation opinion prediction, and it outperforms strong baselines such as ReAct.

Critical strengths
  1. The experiment is comprehensive, covering 3 benchmarks that differ in evidence type, ablation studies of each critical components of the framework, and analysis of memorization.
  2. The proposed method demonstrate concrete performance improvement over the baselines.
Critical weaknesses
  1. Two legal accuracies in Table 2 are not possible given the test set size of 120: In the last row, 0.838 is between 100/120=0.833 and 101/120=0.842. In the "5-shot Qwen + MEMENTO" row, 0.852 is between 102/120=0.85 and 103/120=0.858. All other legal accuracies are possible given the size of 120. Also, to the best of my understanding of the evaluation protocol reported in the paper, these numbers are not multiple-run averages. This raises the concern of typos in key experiment results or even fabricated results.
  2. The intuition of the proposed framework is that it mimics how human experts search for information on the web when labeled data is scarce. The procedural memory (search trajectories) in the framework corresponds to this behavior. The framework also contains declarative memory (facts). This component is not motivated by the intuition, and it is indeed minimally helpful (Table 8). This raises the concern whether the procedural memory is still a necessary component in the framework. Since the evidence type of the tasks do differ, extending the ablation to all 3 datasets can provide more insights regarding the usefulness of the declarative memory component.
  3. Due to the heavy influence of memorization (Table 4), the legal research results in the main results table (Table 2) are not representative of the performance of all methods. Also, while the results for the subpopulation opinion prediction task is reported in the appendix, it only includes a comparison of 4 methods and thus is not a full-scale comparison like the two other tasks. It would give a more direct picture if:
  • Table 2 can include the memorization-free legal results instead of the current results
  • Table 2 can include the full subpopulation opinion prediction task results
Questions for authors
  1. Lines 434-435 says Qwen judge results are reported. However, the sentence before it says the 3-judge average is used, and Appendix L also says 3-judge average is used. What is the number of judges used to produce Table 3? Should it be "with Qwen as the backbone" instead of "with Qwen as the judge"?
  2. Also, Lines 438-439 says that "so absolute scores may slightly differ from scores in Table 2", but the score gap is very large, with ReAct dropping from 0.461 to 0.258. Is there an explanation for this drop?
  3. Where is the metrics TVD defined? Section 4.3 and Appendix N point to each other when referring to this definition, but the metric is not defined in either section.
3 comments

Thank you for recognizing the comprehensive evaluation across three task types, the component ablations, and the concrete performance improvements. We address the accuracy reporting, memory-channel evidence, and presentation of the main results below.

1. Legal accuracy reporting

Thank you for pointing out this discrepancy. MEMENTO with GPT-5-mini was evaluated on 117 cases, while five-shot Qwen with MEMENTO memory was evaluated on 115 cases. The reported accuracies correspond to 98/117 = 0.8376, rounded to 0.838, and 98/115 = 0.8522, rounded to 0.852, respectively.

Our mistake was stating the nominal 120-case test split without specifying the actual evaluation denominators for these entries. We will correct Table 2 and the evaluation description to report these counts explicitly, explain the handling of cases not included in the denominators, and ensure consistent treatment across methods and corresponding comparisons.

2. Motivation and evidence for the memory channels

Our motivation encompasses both aspects of learning through research: acquiring methods for investigating a problem and retaining relevant factual knowledge. Procedural memory captures research strategies, decomposition patterns, and web-action rules; declarative memory retains entity facts and context that can support subsequent investigations.

The completed four-way sales ablation directly separates these contributions within the same AET architecture. Coverage is 0.5526 without memory, 0.5533 with declarative-only memory, 0.5751 with procedural-only memory, and 0.5791 with both channels (Appendix E, Tables 7–8). Procedural memory supplies approximately 85% of the observed memory-specific lift. Adding declarative memory alongside procedures contributes a further 0.0040, compared with 0.0007 in isolation.

The procedural contribution also improves research efficiency: procedural-only memory reduces LLM calls by 4.5% and repeated scraping by 24.3%. These results identify reusable research procedures as the principal contributor to cross-session learning, with factual memory providing a smaller complementary benefit on sales.

The heterogeneous companies and products in the sales task offer a plausible explanation: research procedures can transfer across entities, whereas factual reuse depends on subsequent tasks revisiting related entities or products. This interpretation remains a hypothesis. We will make the motivation and the sales-specific scope of the channel ablation explicit; extending it to legal and SubPOP would further characterize factual reuse across evidence types.

3. Greater visibility for the newer legal and SubPOP evaluations

We agree that the newer legal evaluation should receive greater prominence. We will place it alongside the main comparisons and clearly distinguish it from historical JUSTICE, where benchmark exposure may affect the results.

On the newer 41-case set, MEMENTO achieves 0.561 accuracy versus 0.439 for ReAct and 0.414 for closed-book Qwen. The 12.2 percentage-point improvement over ReAct comprises five additional correct predictions and no regressions (Table 4; Appendix M). We will report this positive observed effect together with the exact two-sided McNemar p-value of 0.063 and the limitations of the sample size.

We will also move the complete existing SubPOP table into the main presentation:

MethodTVD ↓KL ↓
LLM without web0.44980.6369
ReAct + web0.41150.5537
ReAct + query rewriting + web0.40360.5154
MEMENTO0.28430.2742

On 100 questions spanning eight demographic attributes, MEMENTO reduces TVD by 29.6% and KL divergence by 46.8% relative to the strongest tested baseline. These metrics compare predicted and observed distributions directly, without an LLM judge. We will identify the four evaluated configurations explicitly and provide fuller details of the split, memory construction, and aggregation.

4. Judge configuration for Table 3

Thank you for identifying the conflicting descriptions. The main text describes Qwen-judged scores, whereas Appendix L states that the three-judge average is used for the controlled rerun. Qwen is also the generation backbone, which should be stated separately from the evaluator configuration.

We will reconcile the table with the saved scoring and aggregation records and make the caption and surrounding text consistent. The revised description will explicitly identify the generation backbone, evaluator models, and aggregation used for the displayed scores.

Appendix K separately reports the paired comparisons under each evaluator. Against ReAct with both query rewriting and reranking, the improvements are:

  • Qwen: +0.094, 95% CI [0.037, 0.149].
  • GPT-4o: +0.167, 95% CI [0.104, 0.227].
  • GPT-5.4: +0.102, 95% CI [0.055, 0.148].

These positive effects on matched valid outputs show that the coverage advantage persists under Qwen and two additional evaluator models. Appendix L reports absolute-agreement ICC of 0.868 for an individual judge and 0.952 for the three-judge average. Those reliability statistics will be clearly distinguished from the aggregation used in Table 3.

5. Difference between Tables 2 and 3

Table 3 strengthens the evaluation through a controlled rerun against retrieval-enhanced baselines. All five methods use the same time window and live index. MEMENTO achieves coverage of 0.447 versus 0.342 for ReAct with both query rewriting and reranking—a 30.7% relative improvement over the strongest tested baseline.

This advantage is supported across all three evaluator models. Across four baselines and three judges, all twelve paired confidence intervals exclude zero on matched valid outputs, with reported Wilcoxon p-values at or below 0.0011 (Appendix K). These results provide consistent evidence for the coverage improvement within the controlled rerun.

Tables 2 and 3 report separate live-web evaluations, with different absolute score levels: ReAct changes from 0.461 to 0.258, and MEMENTO from 0.579 to 0.447. We agree that “slightly” should be replaced with these explicit changes. We will document the run and scoring conditions and examine the contribution of changes in retrieved evidence, which the current analysis does not isolate.

The revised presentation will make the conditions of each evaluation explicit and give greater prominence to the controlled comparisons and their paired statistical evidence.

6. Definition of TVD

The Metrics paragraph in Appendix N of the current manuscript provides the definition. We will repeat it directly in Section 4.3 and remove the circular reference:

TVD(p, q) = (1/2) × Σ_i |p_i − q_i|,

where p is the predicted distribution over answer options and q is the observed subgroup distribution. TVD ranges from 0 to 1, with lower values indicating closer agreement. We will also specify the KL direction explicitly as KL(q || p).

Reviewer_jR9TOpenReview ↗

Concerns Unaddressed

Thanks for your response. I find the remaining concerns unaddressed:

  1. Why do you only choose to remove 3 and 5 cases to evaluate your proposed method, but use 120 cases for all other methods? Please provide details for these 8 cases during the rebuttal phase.
  2. You provided the misleading claim that ReAct's performance changes slightly, and it in fact changed from 0.461 to 0.258. Please provide further evidence to support that 0.258 still is a valid result, and also explain what caused the drop.
Reviewer_gdjJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MEMENTO, a framework that treats the open web as a learning signal for low-data domains. Within a session, an Adaptive Exploration Tree decomposes the task into waves of sub-questions, reflects on gaps between waves, solves each sub-question with a tool-augmented ReAct agent, and synthesizes answers bottom-up. Across sessions, the agent accumulates text memories without weight updates: procedural memory (craft knowledge, decomposition rules updated with ground-truth-based reflection, and web-action rules with support/contradiction counts) and declarative memory (entity facts). Memory is built over 60 training samples in 6 batches and frozen at test time. Experiments cover sales pitch generation (SDR-Bench), Supreme Court outcome prediction (JUSTICE plus 41 post-cutoff cases), and subpopulation opinion prediction (SubPOP), with Qwen-3.5-35B-A3B and GPT-5-mini. The paper reports gains over ReAct of +25.6% on sales, +36.5% on legal, and a 29.6% TVD reduction on SubPOP.

Critical strengths

The memory artifacts are human-readable and auditable, and the rule support/contradiction tracking is a reasonable engineering design. The finding that procedural memory matters much more than declarative memory is mildly interesting. The paper also makes some effort on leakage (date-restricted search, a post-cutoff legal set) and reports judge agreement.

Critical weaknesses

I don’t see real novelty here. Using web trajectories as a learning signal is essentially the basic data-construction recipe of current deep search work (WebSailor, WebDancer, TaskCraft, etc.). Distilling trajectories into reusable text memory without weight updates is exactly what ExpeL, AWM, ReasoningBank, Agent-KB. The AET is the standard decompose–reflect–synthesize loop of STORM, WebThinker, and Deep Research systems. Overall, the paper takes general deep search and agent memory techniques and applies them to a few small domains, without showing that these domains pose any challenge the general methods cannot handle. The “first to propose the web as a learning signal” claim is an overclaim; moreover, the decomposition rules are updated with ground-truth supervision, so the method does not actually move away from labeled data.

The ablations also show that most of the gain comes from the heavier within-session pipeline rather than cross-session learning. On Qwen, the AET alone gives +0.091 on sales and +0.175 on legal, while memory adds only about +0.03 to +0.04 on top; injecting memory into ReAct gives +0.003 to +0.008 on sales; declarative memory gives +0.0007. Meanwhile, MEMENTO uses around 300 LLM calls and 140+ searches per sample versus roughly 10 steps for ReAct. Showing that ReAct does not use a larger step cap is not a compute-matched comparison. The baselines are only ReAct variants; no memory-based agents, no prompt-optimization methods (GEPA and MIPROv2 are discussed but not run), and no deep research agents are compared.

The evaluation is a major concern as well. The sales task, which carries the headline result, uses a benchmark built by the same group. It is scored recall-only against a single LLM-extracted reference by an LLM judge that is the same model as the backbone, so verbose outputs are rewarded and there is no precision, length control, factuality check, or human evaluation. The top rubric levels reward post-deal metrics that should be unreachable under the stated date cutoff, and no leakage audit is done for this task. Absolute scores also roughly halve between the main table and the re-run (ReAct 0.461 vs. 0.258), and each configuration is run only once.

On the legal task, closed-book Qwen already matches MEMENTO (0.808), 5-shot Qwen beats it (0.842–0.852), no majority-class baseline is reported, and the post-cutoff gain comes down to 5 cases out of 41 and is not significant (p=0.063). The SubPOP experiment uses a single backbone, gives few details on the train/test split and memory construction, and does not compare with the fine-tuned models from the SubPOP paper.

Questions for authors

see weakness

3 comments

Thank you for recognizing the interpretability of the memory artifacts and the procedural-memory findings. We would like to foreground the controlled learning ablations, reductions in research effort, and evaluation across different task types, while clarifying the scope of our contribution.

1. Contribution and the role of supervision

MEMENTO studies how an agent can acquire reusable domain research expertise from web interaction with limited task supervision and fixed model weights. The learning loop produces four artifacts with distinct roles: research craft for planning, conditional decomposition rules, web-action rules for querying and source selection, and factual context for solving subsequent tasks.

Appendix P makes this learned content concrete: the agent acquires a revenue-per-employee strategy for investigating private companies, rules for constructing self-contained subquestions, query/source-selection heuristics, and product facts. The update process is auditable: across six training batches, web-action rules grow from 10 to 29, with support events increasing from 10 to 105 and two contradiction events recorded (Appendix O).

Sales and legal each use 60 labeled training examples. M2 uses ground-truth reflection, while consolidation of M1, M3, and M4 does not directly consult labels. The overall setting therefore combines limited task supervision with web-derived evidence; model weights remain fixed and persistent memory is frozen during evaluation. We will make this hybrid supervision explicit.

We will replace the broad “first to propose the web as a learning signal” statement with this specific contribution and expand the comparison with the deep-search, experience-memory, and prompt-optimization methods you identify.

2. Direct evidence for cross-session learning

The completed four-way ablation holds AET fixed and evaluates no memory, declarative-only memory, procedural-only memory, and both channels. Coverage is 0.5526, 0.5533, 0.5751, and 0.5791, respectively (Appendix E, Tables 7–8). Procedural memory supplies approximately 85% of the observed memory-specific lift. Declarative memory adds 0.0040 alongside learned procedures, compared with 0.0007 in isolation. This identifies reusable research procedures as the principal contribution of learning, with factual memory providing a smaller complementary benefit.

The quality gain accompanies fewer research calls. At the 10-step per-question budget, training improves coverage by 4.8% while reducing searches by 6.6%, LLM calls by 4.1%, and repeated scraping by 26.0%. The benefit also appears at the 20-step budget: coverage increases from 0.5691 to 0.5864, while searches decrease by 13.5% and LLM calls by 9.7% (Appendix D, Table 6). These results demonstrate benefits across two configured research budgets.

For GPT-5-mini sales, coverage progresses from 0.522 with ReAct to 0.532 with AET and 0.547 with learned memory; the memory increment exceeds the architecture increment in this setting. On historical legal research, memory injection also improves ReAct accuracy from 0.592 to 0.742 for Qwen and from 0.767 to 0.825 for GPT-5-mini (Table 2). Learned artifacts can therefore benefit both hierarchical and linear research executors.

3. Research effort and comparison scope

The strongest evidence for learning efficiency is the trained-versus-untrained AET comparison. On legal research, training additionally reduces searches by 19.8%, LLM calls by 19.3%, and latency by 28.0% (Appendix J).

The ReAct step-cap experiment establishes that its configured cap was not binding; it does not establish equal actual compute between ReAct and AET. We will distinguish these questions and report training effort and actual inference effort separately.

The retrieval-enhanced controls test query construction and source ordering, while the AET ablation tests learning within a fixed research architecture. Direct comparisons with the closest memory, planning, and prompt-optimization methods would further establish the incremental contribution. We will make that comparison scope explicit and foreground the controlled evidence already available.

4. Sales results under stronger baselines and independent evaluator models

In the controlled sales rerun, MEMENTO achieves coverage of 0.447 versus 0.342 for ReAct with both query rewriting and reranking, a 30.7% relative improvement (Table 3).

The advantage persists under evaluator models independent of the Qwen generation backbone. Against that strongest combined baseline, Appendix K reports paired improvements of:

  • Qwen: +0.094, 95% CI [0.037, 0.149].
  • GPT-4o: +0.167, 95% CI [0.104, 0.227].
  • GPT-5.4: +0.102, 95% CI [0.055, 0.148].

Across four baselines and three judges, all twelve paired confidence intervals exclude zero on matched valid outputs; reported Wilcoxon p-values are at or below 0.0011. Appendix L additionally reports absolute-agreement ICC of 0.868 for an individual judge and 0.952 for the three-judge average.

These results strengthen the evidence for reference coverage across evaluator models. The rubric scores recovery of specific products, pain points, mechanisms, and supporting details; generic marketing statements receive low scores. We will identify the metric explicitly as reference coverage and distinguish this evidence from separate assessments of factual precision, output-length effects, and agreement with human evaluators.

5. Benchmark provenance, temporal evidence, and rerun variability

SDR-Bench originates from our group, and we will make its provenance and reference-extraction procedure explicit. The evaluation uses 60 training and 120 held-out test examples, frozen test-time memory, and search restricted to sources published at least six months before each success story’s publication.

The paper also contains a source audit covering approximately 700 cited training pages from 400 domains; none of the cited domains appears on the audited Iffy misinformation index (Appendix O). This supplies evidence about source provenance. Temporal accessibility of story-specific reference metrics and factual support for generated claims require separate checks. We will distinguish these questions and document the evidence dates supporting the highest rubric grades.

Tables 2 and 3 report separate live-web evaluations. The ReAct change from 0.461 to 0.258 is substantial, and we will replace “slightly” with the actual changes and document verified differences in run and scoring conditions. Live-web variation is a possible influence, but the current records presented in the paper do not isolate its contribution to the shift.

Within the rerun, all five methods use the same time window and live index, and the paired analyses support the reported advantage. These confidence intervals concern variation across evaluated cases; repeated-generation and repeated-training stability are separate questions.

6. Legal baselines and the newer cases

The historical JUSTICE results include strong closed-book and five-shot controls: Qwen reaches 0.808 without web access and 0.842 with five-shot prompting. We will identify the headline 36.5% improvement explicitly as relative to ReAct and make the benchmark-exposure caveat prominent.

The newer 41-case evaluation provides an additional test: MEMENTO achieves 0.561 accuracy versus 0.439 for ReAct and 0.414 for closed-book Qwen. This corresponds to a 12.2 percentage-point improvement over ReAct, with five additional correct predictions and no regressions (Table 4; Appendix M). We report the exact two-sided McNemar p-value of 0.063, so the positive observed effect should be interpreted with its uncertainty rather than as significance at 0.05.

We will foreground this evaluation, add majority-class accuracy computed from the case labels, and report evaluated-case counts and failure handling explicitly.

7. SubPOP provides evidence using direct distribution metrics

On 100 questions spanning eight demographic attributes, MEMENTO achieves TVD of 0.2843 versus 0.4036 for ReAct with query rewriting, a 29.6% reduction. KL divergence decreases from 0.5154 to 0.2742, a 46.8% reduction (Appendix N, Table 14). These metrics compare predicted and observed distributions directly and do not require an LLM judge.

This supplies evidence on a third task with a different output structure and evaluation method. We will promote the complete existing table into the main results and specify the training/test selection, question–subgroup counts, memory construction, date restrictions, and aggregation. We will also discuss the original SubPOP fine-tuned models with their supervision and evaluation settings, while limiting the current numerical comparison to methods evaluated under our reported protocol.

Thank you for the response and the additional judge analyses, which address my concern about the evaluator being the same model as the backbone. However, my main concerns remain. The novelty over existing deep-search and agent-memory methods is acknowledged but not supported by any new comparisons; no memory-based, prompt-optimization, or deep-research baselines were run. The cross-session learning gain is small, and compute is still not matched against ReAct. On evaluation, precision, factuality, leakage, and human assessment are deferred to the revision. On legal, the post-cutoff result is not significant. Since most of the response promises clarifications rather than new evidence, I will retain my assessment.

Reviewer_HaBHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MEMENTO, a framework for improving LLM agents in low-data domains through web exploration and cross-session memory. The method consists of two main components: an Adaptive Exploration Tree (AET), which decomposes research tasks into subquestions and iteratively expands them based on intermediate findings, and a dual-channel memory system that stores declarative facts and procedural research strategies across sessions. The framework uses LLM-based reflection and memory updates rather than model parameter optimization. Experiments on sales automation, legal outcome prediction, and subpopulation opinion prediction show improvements over ReAct-based baselines.

Critical strengths

The motivation and overall system architecture are clearly presented, and the distinctions between the memory components are reasonably explained.

The ablations separate within-session exploration from cross-session memory, examine the contributions of procedural and declarative memory, and report some efficiency improvements. I also appreciate the post-cutoff legal evaluation and the additional comparisons against retrieval-enhanced ReAct baselines.

Critical weaknesses

The Adaptive Exploration Tree performs question decomposition, iterative web search, reflection, and dynamic expansion of subquestions, which have already been extensively studied in previous search-agent systems. For example, MindSearch already uses a dynamically constructed graph of subquestions that is progressively expanded based on retrieved information. Similarly, the idea of accumulating agent experiences and extracting reusable knowledge or strategies without updating model weights has been explored by Reflexion, ExpeL, Agent Workflow Memory, and WebCoach. Although MEMENTO organizes these components differently, I am not convinced that the resulting combination constitutes a substantial methodological contribution.

The authors position MEMENTO as a new approach to learning domain expertise from web interactions, but several closely related experience-learning and memory-based agent methods are not adequately discussed or compared against. In particular, ExpeL and Agent Workflow Memory are important references for extracting reusable experience and procedural knowledge from previous agent trajectories, while WebCoach specifically studies cross-session memory for web agents without model fine-tuning. For the within-session research component, comparisons with more capable search-planning frameworks such as MindSearch would be more informative than comparisons primarily against ReAct. Although the authors additionally evaluate query rewriting and reranking, these modifications do not provide ReAct with comparable dynamic planning capabilities. As a result, the experiments do not convincingly establish the advantages of MEMENTO over the closest existing approaches.

According to Table 2, replacing ReAct with AET alone improves the Qwen sales score from 0.461 to 0.552, while adding the full memory system only increases it further to 0.579. A similar pattern appears in legal research, where AET alone improves accuracy from 0.592 to 0.767, compared with 0.808 for the full method. Thus, a substantial majority of the reported improvement is achieved without cross-session memory. This is important because the paper's central contribution is framed around learning reusable expertise from accumulated web experience, whereas the largest gains appear to come from a more elaborate within-session research workflow. The additional memory gains are positive, but they seem relatively modest compared with the headline improvements.

The authors introduce four separate memory stores with different update and retrieval procedures, but the experiments do not fully establish why this particular architecture is necessary. Interestingly, Appendix E shows that declarative memory alone provides almost no improvement on the sales task (0.5526 to 0.5533), while procedural memory accounts for most of the additional gain. This raises questions about the importance of the proposed dual-channel design.

Questions for authors

see the weakness above

2 comments

Thank you for recognizing the clarity of the architecture and the exploration, memory, and efficiency ablations. We first address the contribution relative to existing search-planning and experience-learning methods.

1. Contribution relative to existing search and memory methods

We agree that dynamic question decomposition, reflection, and learning from previous agent experiences have substantial precedents. MindSearch, Reflexion, ExpeL, Agent Workflow Memory, and WebCoach are relevant foundations. We will remove the broad priority claim about using the web as a learning signal and describe our contribution more precisely. MEMENTO’s contribution is the learning loop for domain research, with distinct artifacts, update signals, and points of use. It consolidates domain research craft, conditional decomposition rules, web-action rules, and entity facts. Craft knowledge guides planning; decomposition rules address recurring coverage gaps; action rules guide query formulation and source selection; and factual memory supplies context to the solver. Ground-truth reflection updates decomposition rules, while consolidation of craft knowledge, action rules, and facts does not directly consult labels.

Appendix P illustrates what is learned: a revenue-per-employee strategy for investigating private companies, rules for constructing self-contained subquestions, query and source-selection heuristics, and product-specific facts. Our empirical contribution examines this learning design across generation, classification, and distribution prediction, including comparisons with untrained AET and measurements of inference effort. We will frame AET as the research executor within this learning loop and acknowledge its relationship to established adaptive planning methods.

2. Coverage of the closest prior work and comparison scope

We will add the missing discussion of ExpeL and Agent Workflow Memory and expand the discussion of WebCoach, which is already cited in Section 2 and Table 1. ExpeL extracts experiential insights and recalls past experiences; AWM induces reusable workflows; WebCoach retrieves episodic navigation experiences to provide runtime guidance. These precedents establish that retaining experience without weight updates is not, by itself, a new contribution.

The existing controls answer different questions. Query rewriting and reranking test improvements to retrieval, while AET without persistent memory tests the incremental effect of learning within the same research architecture. In the controlled sales rerun, MEMENTO scores 0.447 versus 0.342 for ReAct with both query rewriting and reranking, a 30.7% relative improvement. Appendix K additionally reports positive paired effects under all three judges on matched valid outputs.

We agree that retrieval-enhanced ReAct does not provide the same planning capabilities as MindSearch. The existing ablations establish benefits over those controls; direct comparisons with the closest planning and experience-memory approaches remain important for establishing the incremental advantage over those methods.

3. Contribution of cross-session learning

Our ablations directly test the contribution of cross-session learning by holding the AET architecture fixed and varying the learned memory. The four-way ablation compares no memory, declarative-only memory, procedural-only memory, and both channels (Appendix E, Tables 7–8). Procedural memory raises sales coverage from 0.5526 to 0.5751, accounting for approximately 85% of the observed memory-specific gain. Combining procedural and declarative memory raises coverage further to 0.5791—a 4.8% relative improvement over untrained AET. Declarative memory contributes more alongside learned procedures (+0.0040) than in isolation (+0.0007). These results identify reusable research procedures as the principal contribution of cross-session learning, with factual memory providing a smaller complementary benefit.

The quality improvement also accompanies more efficient research. Under the same configured budgets, learned memory reduces searches per sample from 149.49 to 139.61, LLM calls from 323.97 to 310.79, and repeated scraping from 25.69 to 19.02 (Appendix D, Table 6). The agent therefore achieves better coverage with fewer research calls and less redundant retrieval.

The benefit is also visible across backbones and research architectures. For GPT-5-mini sales, AET raises coverage from 0.522 to 0.532, and learned memory adds a further improvement to 0.547; the memory increment is larger than the architecture increment in this setting (Table 2). On Qwen legal research, adding memory to AET raises accuracy from 0.767 to 0.808. Furthermore, injecting learned memory into ReAct improves historical legal accuracy from 0.592 to 0.742 for Qwen and from 0.767 to 0.825 for GPT-5-mini, demonstrating that the learned artifacts can also benefit a linear research executor.

We will give these controlled memory comparisons and the procedural/declarative ablation greater prominence, while reporting architecture gains and cross-session learning gains separately.

4. Rationale for the four stores and the declarative channel

The four stores reflect different functions in research: M1 supplies broad research strategy to planning; M2 supplies conditional decomposition rules; M3 guides querying, source selection, and scraping; and M4 retains factual context for the solver. Their separate update and injection procedures provide a functional rationale for the organization. The current channel ablation does not establish that every individual store is necessary or that four stores are the optimal organization, and we will narrow the corresponding claim.

The sales ablation identifies procedural learning as the dominant contributor. Procedural-only memory raises coverage from 0.5526 to 0.5751, supplying approximately 85% of the observed full-memory lift. Adding declarative memory raises coverage further to 0.5791, an additional 0.0040, compared with its isolated contribution of 0.0007 (Appendix E, Tables 7–8). We will present factual memory as a modest complement on this task.

One plausible explanation is the heterogeneity of the sales examples: research strategies can transfer across different companies and products, whereas entity facts are useful when subsequent tasks revisit related entities or products. A more homogeneous setting could increase factual reuse, but this remains a hypothesis. We will distinguish that interpretation from the measured result and make clear that the current four-way channel ablation covers sales.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.