- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a framework that uses search trajectories derived from web data, alongside facts, to address the lack of labeled data for low-data tasks. The framework is evaluated on sales automation, legal outcome prediction, and subpopulation opinion prediction, and it outperforms strong baselines such as ReAct.
Critical strengths
- The experiment is comprehensive, covering 3 benchmarks that differ in evidence type, ablation studies of each critical components of the framework, and analysis of memorization.
- The proposed method demonstrate concrete performance improvement over the baselines.
Critical weaknesses
- Two legal accuracies in Table 2 are not possible given the test set size of 120: In the last row, 0.838 is between 100/120=0.833 and 101/120=0.842. In the "5-shot Qwen + MEMENTO" row, 0.852 is between 102/120=0.85 and 103/120=0.858. All other legal accuracies are possible given the size of 120. Also, to the best of my understanding of the evaluation protocol reported in the paper, these numbers are not multiple-run averages. This raises the concern of typos in key experiment results or even fabricated results.
- The intuition of the proposed framework is that it mimics how human experts search for information on the web when labeled data is scarce. The procedural memory (search trajectories) in the framework corresponds to this behavior. The framework also contains declarative memory (facts). This component is not motivated by the intuition, and it is indeed minimally helpful (Table 8). This raises the concern whether the procedural memory is still a necessary component in the framework. Since the evidence type of the tasks do differ, extending the ablation to all 3 datasets can provide more insights regarding the usefulness of the declarative memory component.
- Due to the heavy influence of memorization (Table 4), the legal research results in the main results table (Table 2) are not representative of the performance of all methods. Also, while the results for the subpopulation opinion prediction task is reported in the appendix, it only includes a comparison of 4 methods and thus is not a full-scale comparison like the two other tasks. It would give a more direct picture if:
- Table 2 can include the memorization-free legal results instead of the current results
- Table 2 can include the full subpopulation opinion prediction task results
Questions for authors
- Lines 434-435 says Qwen judge results are reported. However, the sentence before it says the 3-judge average is used, and Appendix L also says 3-judge average is used. What is the number of judges used to produce Table 3? Should it be "with Qwen as the backbone" instead of "with Qwen as the judge"?
- Also, Lines 438-439 says that "so absolute scores may slightly differ from scores in Table 2", but the score gap is very large, with ReAct dropping from 0.461 to 0.258. Is there an explanation for this drop?
- Where is the metrics TVD defined? Section 4.3 and Appendix N point to each other when referring to this definition, but the metric is not defined in either section.
3 comments
4. Judge configuration for Table 3
Thank you for identifying the conflicting descriptions. The main text describes Qwen-judged scores, whereas Appendix L states that the three-judge average is used for the controlled rerun. Qwen is also the generation backbone, which should be stated separately from the evaluator configuration.
We will reconcile the table with the saved scoring and aggregation records and make the caption and surrounding text consistent. The revised description will explicitly identify the generation backbone, evaluator models, and aggregation used for the displayed scores.
Appendix K separately reports the paired comparisons under each evaluator. Against ReAct with both query rewriting and reranking, the improvements are:
- Qwen: +0.094, 95% CI [0.037, 0.149].
- GPT-4o: +0.167, 95% CI [0.104, 0.227].
- GPT-5.4: +0.102, 95% CI [0.055, 0.148].
These positive effects on matched valid outputs show that the coverage advantage persists under Qwen and two additional evaluator models. Appendix L reports absolute-agreement ICC of 0.868 for an individual judge and 0.952 for the three-judge average. Those reliability statistics will be clearly distinguished from the aggregation used in Table 3.
5. Difference between Tables 2 and 3
Table 3 strengthens the evaluation through a controlled rerun against retrieval-enhanced baselines. All five methods use the same time window and live index. MEMENTO achieves coverage of 0.447 versus 0.342 for ReAct with both query rewriting and reranking—a 30.7% relative improvement over the strongest tested baseline.
This advantage is supported across all three evaluator models. Across four baselines and three judges, all twelve paired confidence intervals exclude zero on matched valid outputs, with reported Wilcoxon p-values at or below 0.0011 (Appendix K). These results provide consistent evidence for the coverage improvement within the controlled rerun.
Tables 2 and 3 report separate live-web evaluations, with different absolute score levels: ReAct changes from 0.461 to 0.258, and MEMENTO from 0.579 to 0.447. We agree that “slightly” should be replaced with these explicit changes. We will document the run and scoring conditions and examine the contribution of changes in retrieved evidence, which the current analysis does not isolate.
The revised presentation will make the conditions of each evaluation explicit and give greater prominence to the controlled comparisons and their paired statistical evidence.
6. Definition of TVD
The Metrics paragraph in Appendix N of the current manuscript provides the definition. We will repeat it directly in Section 4.3 and remove the circular reference:
TVD(p, q) = (1/2) × Σ_i |p_i − q_i|,
where p is the predicted distribution over answer options and q is the observed subgroup distribution. TVD ranges from 0 to 1, with lower values indicating closer agreement. We will also specify the KL direction explicitly as KL(q || p).
Concerns Unaddressed
Thanks for your response. I find the remaining concerns unaddressed:
- Why do you only choose to remove 3 and 5 cases to evaluate your proposed method, but use 120 cases for all other methods? Please provide details for these 8 cases during the rebuttal phase.
- You provided the misleading claim that ReAct's performance changes slightly, and it in fact changed from 0.461 to 0.258. Please provide further evidence to support that 0.258 still is a valid result, and also explain what caused the drop.
Thank you for recognizing the comprehensive evaluation across three task types, the component ablations, and the concrete performance improvements. We address the accuracy reporting, memory-channel evidence, and presentation of the main results below.
1. Legal accuracy reporting
Thank you for pointing out this discrepancy. MEMENTO with GPT-5-mini was evaluated on 117 cases, while five-shot Qwen with MEMENTO memory was evaluated on 115 cases. The reported accuracies correspond to 98/117 = 0.8376, rounded to 0.838, and 98/115 = 0.8522, rounded to 0.852, respectively.
Our mistake was stating the nominal 120-case test split without specifying the actual evaluation denominators for these entries. We will correct Table 2 and the evaluation description to report these counts explicitly, explain the handling of cases not included in the denominators, and ensure consistent treatment across methods and corresponding comparisons.
2. Motivation and evidence for the memory channels
Our motivation encompasses both aspects of learning through research: acquiring methods for investigating a problem and retaining relevant factual knowledge. Procedural memory captures research strategies, decomposition patterns, and web-action rules; declarative memory retains entity facts and context that can support subsequent investigations.
The completed four-way sales ablation directly separates these contributions within the same AET architecture. Coverage is 0.5526 without memory, 0.5533 with declarative-only memory, 0.5751 with procedural-only memory, and 0.5791 with both channels (Appendix E, Tables 7–8). Procedural memory supplies approximately 85% of the observed memory-specific lift. Adding declarative memory alongside procedures contributes a further 0.0040, compared with 0.0007 in isolation.
The procedural contribution also improves research efficiency: procedural-only memory reduces LLM calls by 4.5% and repeated scraping by 24.3%. These results identify reusable research procedures as the principal contributor to cross-session learning, with factual memory providing a smaller complementary benefit on sales.
The heterogeneous companies and products in the sales task offer a plausible explanation: research procedures can transfer across entities, whereas factual reuse depends on subsequent tasks revisiting related entities or products. This interpretation remains a hypothesis. We will make the motivation and the sales-specific scope of the channel ablation explicit; extending it to legal and SubPOP would further characterize factual reuse across evidence types.
3. Greater visibility for the newer legal and SubPOP evaluations
We agree that the newer legal evaluation should receive greater prominence. We will place it alongside the main comparisons and clearly distinguish it from historical JUSTICE, where benchmark exposure may affect the results.
On the newer 41-case set, MEMENTO achieves 0.561 accuracy versus 0.439 for ReAct and 0.414 for closed-book Qwen. The 12.2 percentage-point improvement over ReAct comprises five additional correct predictions and no regressions (Table 4; Appendix M). We will report this positive observed effect together with the exact two-sided McNemar p-value of 0.063 and the limitations of the sample size.
We will also move the complete existing SubPOP table into the main presentation:
On 100 questions spanning eight demographic attributes, MEMENTO reduces TVD by 29.6% and KL divergence by 46.8% relative to the strongest tested baseline. These metrics compare predicted and observed distributions directly, without an LLM judge. We will identify the four evaluated configurations explicitly and provide fuller details of the split, memory construction, and aggregation.