- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Search-MEM is a memory-augmented search system with five cooperating agents: a planner, compressor, retriever, generator, and memory agent. The planner controls queries, document selection, compression, and stopping; the compressor shortens retrieved text in the planner’s context; the generator answers from retained documents. The planner and compressor are trained jointly with RL while the retriever and generator remain frozen. The memory is built by replaying multi-hop training questions and storing decompositions that pass reference-free verification checks. At inference, the system performs an initial search and samples multiple answers. If the samples disagree or are empty, the memory retrieves similar decompositions to guide an additional search; newly retrieved documents are appended to the generator’s context, and the generator answers again. The paper evaluates on six QA benchmarks with multiple generator sizes and two judges, reports zero-shot transfer to a news corpus, and ablates the memory, compressor, and cost.
The main contributions are: (1) a memory-gated repair mechanism that triggers additional search when generator samples are unsettled; (2) joint planner–compressor training with a shared trajectory reward, allowing the planner to carry much less retrieved text with approximately unchanged accuracy; and (3) strong average accuracy using about 2.5% of the training questions used by Search-R1, while serving multiple generators including a proprietary one.
Critical strengths
1.The memory-gated repair mechanism addresses a real failure mode of existing search agents. Existing trained search policies decide when to stop before the final answer is produced; if the gathered evidence is insufficient, they have no mechanism to search again. Search-MEM’s use of reference-free decomposition memory to trigger an additional search when generator samples disagree or are empty is a sensible and practical intervention. This is consequential because it improves robustness without retraining or modifying the frozen generator, and the zero-shot news-corpus transfer provides some evidence that the memory carries a decomposition format rather than only dataset-specific content.
2.Joint planner–compressor training with a shared trajectory reward is a clean way to keep search context small. The compressor is optimized by the downstream answer reward rather than by a separate compression objective, and the planner is rewarded for using compression, stopping voluntarily, and improving over naive RAG. This is consequential because it supports deeper search with a much smaller planner context while keeping accuracy approximately unchanged, which is important for context-budget-limited agentic search.
Critical weaknesses
1.The memory’s same-distribution advantage is not sufficiently ruled out. The memory is written from MuSiQue training questions, and MuSiQue is one of the six evaluation datasets. Even with disjoint questions, the memory’s decomposition style and bridge patterns may be unusually well matched to MuSiQue. The largest memory gains are indeed on MuSiQue (e.g., Qwen2.5-7B from 16.2 to 23.3 under Claude judge). This is consequential because the memory is the paper’s central contribution; without a leave-one-dataset-out memory construction or a memory built from a clearly disjoint corpus, the claim that the memory supplies a transferable decomposition format is only partially supported.
2.The gate is blind to confident errors, and the cost–accuracy Pareto advantage is not fully established. The gate fires only when the generator’s samples disagree or are empty; if all samples agree on a wrong answer, the system returns it without repair. The paper does not quantify how often this happens. At the same time, Search-MEM averages about 4.3 retrievals and 10.4 model calls per question, more than the compared search methods. This is consequential because the method’s practical value depends on whether the additional repair cost is concentrated on questions where it is needed and whether the accuracy gain is robust relative to cheaper baselines at matched budgets.
Questions for authors
1.Memory transfer / leave-one-dataset-out memory: Can you report results with the memory built from a dataset other than MuSiQue, or from a disjoint corpus, and evaluated on MuSiQue? How much of the MuSiQue gain remains? This would directly test whether the memory carries a transferable decomposition format rather than a same-distribution advantage.
2.Confident-wrong failures: What fraction of test questions have all k generator samples agree on a wrong answer, so the gate stays silent? Can you characterize these cases and estimate the headroom from a coverage- or uncertainty-based gate extension?
3.Gate and cost sensitivity: How sensitive are the results to the number of samples k, sampling temperatures, and the disagreement/empty gate? Can you provide a cost–accuracy Pareto comparison against s3 and Search-R1 at matched retrievals or matched model calls, rather than only at the deployed operating point?
3 comments
Response to Reviewer 2T9F (2/2)
Q3. Gate and cost sensitivity. Following the reviewer’s suggestion, we varied k and sampling temperature. The first read is greedy; subsequent reads use the listed temperature. Agreement requires a strict majority of identical non-empty answers, while the other gate signals remain active. Each cell reports the difference between the reported gated-repair and no-repair accuracies, in %.
| Dataset | k=1, T=0 | k=3, T=0.3 | k=3, T=0.7 (Reported setting) | k=3, T=1.0 | k=5, T=0.7 |
|---|---|---|---|---|---|
| MuSiQue | +4.9 | +4.5 | +4.5 | +5.3 | +4.3 |
| HotpotQA | +2.0 | +2.1 | +1.8 | +1.8 | +2.1 |
Our deployed setting is k=3, T=0.7. Repair improves accuracy under every tested configuration, including with more initial votes. The gate comparison gives:
| Gate | MuSiQue GenAcc (%) |
|---|---|
| Disagreement only | 28.0 |
| Disagreement or empty answer | 27.1 |
| Deployed combined gate | 28.4 |
As shown, the combined gate performs best among these variants. The disagreement-or-empty result also shows that adding a trigger does not automatically improve accuracy.
Cost-accuracy Pareto We compared the methods at the reported call and retrieval budgets to illustrate the cost and accuracy of our method, and s3 and Search-R1. We report Qwen GenAcc, with the highest reported score in each column bolded.
Reported answer-stage model-call budget per question:
| Method | 1.0 | 3.0 | 5.0 | 6.5 | 8.0 |
|---|---|---|---|---|---|
| s3 | 19.7 | 24.5 | 25.0 | 25.9 | 26.1 |
| Search-R1 | 21.9 | 24.7 | 27.4 | 29.4 | 28.4 |
| Search-Mem | 22.7 | 24.6 | 27.9 | 28.4 | 29.0 |
Reported added-retrieval budget per question:
| Added-retrieval calls | 0.0 | 1.0 | 1.5 | 2.0 | 2.6 |
|---|---|---|---|---|---|
| s3 | 22.1 | 23.8 | 24.9 | 25.9 | 26.1 |
| Search-R1 | 23.4 | 26.4 | 27.2 | 29.4 | 28.3 |
| Search-Mem | 24.2 | 26.6 | 27.8 | 28.4 | 28.9 |
Search-Mem exceeds s3 throughout the reported grid and performs better than Search-R1 at most of the budgets. Bold marks the best reported entry in each column; a stronger lower-budget configuration remains feasible at larger budgets.
To test whether generic uses of extra computation reproduce the repair gains, we used Qwen2.5-14B as generator and Qwen-judged GenAcc. Within each comparison, contexts, initial reads, gate decisions, and yes/no retention are shared. Each column below refers to its named baseline; the voting-resource measurements are separate from the call/query-matched accuracy controls.
| MuSiQue contexts | Gain over call-matched sampling (%) | Gain over call/query-matched draft retrieval (%) | Tokens saved versus seven-sample voting (%) | Mean latency saved versus seven-sample voting (%) |
|---|---|---|---|---|
| s3 | +3.8 | +4.7 | 6.3 | 26.4 |
| Search-R1 | +0.6 | +0.7 | 17.5 | 31.7 |
| Search-Mem | +3.9 | +4.3 | 3.8 | 25.2 |
Targeted repair achieves higher accuracy than the generic sampling and retrieval controls, with smaller gains on Search-R1. It also uses fewer tokens and has lower mean latency than the voting baseline in the resource comparisons. A dash indicates an unreported resource comparison. The cost measurements cover the answer stage and exclude upstream search, planning, and compression. We reckon that the results support the stated answer-stage tradeoffs
Based on these findings, we will add the transfer, failure, sensitivity, and budget analyses and clarify the gate. We hope that the reviewer is satisfied with our response and will increase their score accordingly.
Dear Reviewer 2T9F,
With the discussion period ending in approximately 2 days, we would greatly appreciate your feedback on our rebuttal. We believe we have thoroughly addressed your concerns and would welcome your assessment. If you find that we have successfully resolved your major points, we respectfully request that you consider adjusting your score to reflect these improvements.
Thank you for your time and consideration.
Best regards,
The Authors
Response to Reviewer 2T9F (1/2)
We thank the reviewer for the constructive comments. We ran additional experiments on memory transfer, silent unanimous errors, and inference cost, and address the concerns below.
W1 / Q1. Cross-dataset memory transfer Thank you for the constructive comment. To address this, we additionally built memories from equally sized samples from two other multi-hop datasets, HotpotQA and 2Wiki, and evaluated them on MuSiQue with Qwen2.5-14B as the generator. We note that the contexts, initial answers, gate decisions, and the repair procedure remain fixed; only the supplied memory changes.
We call memories built with HotpotQA and 2Wiki out-of-distribution memories, as they differ from the downstream task (MuSiQue). As we observe, they retain most of the in-distribution memory’s gain. We reckon the possible reason could due to the fact that memory stores questions, splits, and bridge/no-bridge flags. This supports transfer of decomposition guidance across datasets.
W2 / Q2. Silent unanimous errors and repair headroom Thank you for raising this. Our deployed gate also checks answer quality, grounding, and question coverage, so agreement alone does not guarantee that repair is skipped. Our repair mechanism fires when any condition below holds:
We will revise the method description and algorithm correspondingly. To quantify the effect, we measured unanimous errors on the full evaluation splits using our planner’s contexts and examined the cases the gate missed. Below, silent and caught both refer to unanimous judged-wrong answers. Error frequencies use all evaluated questions as the denominator; gold-string presence and token overlap use only silent unanimous-wrong cases.
The additional signals catch some unanimous errors, while the silent column quantifies those that remain undetected. Every silent prediction passes the answer-string grounding check, but appearing in a passage does not establish that it answers the requested relation. Likewise, gold-string presence does not prove sufficient support, and absence does not prove that evidence is missing. These lexical measurements characterize the missed cases without assigning manually verified causes.
We then measured recovery in separate repair analyses. One forces repair on silent unanimous errors and reports both the fraction corrected within that group and the gross corrections among all evaluated questions. The other repairs all silent questions, including non-unanimous cases, and measures the net accuracy gain over gated repair. Both use the evaluation sizes below, but the unanimous-only recovery and all-silent net gain are distinct measurements.
The existing repair corrects part of the missed unanimous-error group. These gross recoveries are not net gains from a new gate or a hard upper bound on better repair methods. The separate all-silent analysis shows that always repairing achieves higher accuracy, while gating avoids many repair invocations. We therefore present gating as an accuracy–computation tradeoff. We also note that skipped invocations do not by themselves establish end-to-end token or latency savings.