- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Evidence-Commit Protocol for solving the issue that searched results are concatenated the context in multi-turn search agent. After retrieval, it renews the evidence state in each evidence commit segment, and removes the raw observation and temporary reasoning, then generation the next action segment. By doing so, newly retrieved information implements commit-before-use that should be recorded in the evidence state to utilize the next action. Furthermore, by leveraging versioned evidence state and provenance store, it records source evidence and version lineage. While training, the final answer keep track on lineage of cited evidence, and it gives the provenance-routed segment credit to corresponding commit segment by evaluating the source support with frozen NLI model.
Critical strengths
S1. Clear problem definition. It is good to separate the memory generation term and the thing that next action should pass through the corresponding memory. STOP-VALIDATE/APPLY-EVICT-REGENERATE protocol constraints information visibility at the environment level, providing clearer structural constraints than simple prompt instructions.
S2. Practical. The most persuasive results in this manuscript is the accuracy-cost trade-off improvement. In 3B, the proposed method reduces the peak context 37.5--49.7%, and inference time 16.9--21.2% while maintaining the accuracy similar to MemSearcher.
Critical weaknesses
W1. Accuracy superiority. The results on this paper are hard to say that "consistent accuracy superiority". It should be toned down into "competitive accuracy" compared to MemSearcher. It is because, in table 1, the proposed method is worse than MemSearcher in 7B under NQ, MuSiQue, and Bamboogle.
W2. Ablation studies. Current ablation shows the effectiveness of adding provenance credit, but it does not demonstrate whether routing that tracks version lineage is necessary beyond a simpler source-support reward.
W3. Source faithfulness directly affects the real answer grounding. The faithfulness evaluation measures whether the source supports evidence statement or not. However, those problems should be separate such as the corresponding evidence is relevant to the query, entails the final answer, or is sufficient for multi-hop reasoning. Although generating source-supported statement and citing, the answer should rely on the parametric knowledge. It should be splitted into Source Evidence support and Evidence Answer support, and it is hard to prove the final answer grounding with current result.
W4. Baseline fairness. The baselines of Table 1 are not sufficient to be reevaluated under same retriever, retrieval top-K, and search budget. In efficiency perspective, comparison is needed under the same memory token budget and generation budget. Moreover, we need to dig in where the context reduction comes from: (1) evidence-commit protocol or (2) more rigorous state budget or different generation behavior.
W5. Suspicious of Long-horizon context management. The paper indicates the major motivation of context accumulation as increasing search turn, but the actual average number of searching is about two in efficiency evaluation. The method should be evaluated from the task that requires longer exploration.
W6. Novelty and missing baselines. In each component, there are previous works. For example, context reconfiguration by each turn (MemSearcher, ReSum), structured memory editing (Memory-R1, MemAct), NLI-based rewards for evidence notes (EviNote-RAG), credit assignment for each segment/step (StepSearch, HiPRAG, Memory-R2, MAVEN). In particular, the results of EviNote-RAG, StepSearch, and HiPRAG are missing in Table 1 and 2, even though they utilize the Qwen2.5 backbone, which is the same backbone in the proposed method. Please justify two things: (1) novelty of the proposed method, (2) the reason why those baselines are missing.
Questions for authors
See weaknesses. Here are the minor comments.
1. The information of reproducibility is not sufficient such as hyperparameters of GRPO training. Please report the hyperparameters in Appendix.
2. In Figure 1(a), two queries from search history described "S_1*query". Moreover, is missing in commit segment context label ().
3. In Appendix G.2, it is the second turn, but the tag of retrieval document is [T1D1] and [T1D2]. It should change [T2D1] and [T2D2].
4. In Algorithm 1, to the best of my understanding, Line 3 and 18 are duplicated.
12 comments
Response to Reviewer Xgfe - Post (2)
W3. Source–Evidence Support and Evidence–Answer Support
We agree that the two types of support should be evaluated separately. We will rename the original metric Source–Evidence Support to reflect that it measures whether each evidence statement is supported by its declared source. We also evaluate Evidence–Answer Support, which measures whether the evidence cited in the final answer textually supports the model's predicted answer.
We concatenate the active evidence items actually cited in the final answer in ID order. We then use Qwen2.5-3B-Instruct to rewrite the original question and predicted answer as an English statement and use a frozen RoBERTa-large-MNLI model to determine whether the cited evidence entails that statement. We use all 125 Bamboogle questions and the first 2,000 questions in the archive for each of the other six datasets. For each question, we compute
where is the statement that Qwen2.5-3B-Instruct generates from the question and predicted answer, the probabilities come from frozen RoBERTa-large-MNLI, and is the set of evidence items actually cited in the final answer to that question. We report the NLI entailment rate, the fraction of questions for which entailment is the highest-scoring of the three classes, and mean NLI support, the per-question mean of .
| Evidence–Answer NLI support | NQ | TriviaQA | PopQA | HotpotQA | 2Wiki | MuSiQue | Bamboogle | Total/Avg. |
|---|---|---|---|---|---|---|---|---|
| Evaluation questions | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 2,000 | 125 | 12,125 |
| NLI entailment rate (%) | 68.40 | 52.15 | 91.05 | 45.55 | 43.80 | 38.30 | 52.80 | 56.01 |
| Mean NLI support (%) | 61.27 | 45.44 | 83.19 | 37.73 | 36.79 | 31.22 | 45.13 | 48.68 |
The cited evidence provides substantial textual support for the final answers. The NLI entailment rate is 91.05% on PopQA and 68.40% on NQ, with mean NLI support of 83.19% and 61.27%, respectively. The NLI model also judges the cited evidence to entail the final answer in 45.55% of HotpotQA cases, 43.80% of 2Wiki cases, and 38.30% of MuSiQue cases.
W4. Baseline comparability, matched budgets, and the source of context reductions
Thank you for pointing out where the baseline evaluation setup needed clarification.
-
In Table 1, the results for ZeroSearch and R1-Searcher use Google Web Search. The other baselines use the same retriever or retrieval configuration. For comparability with MemSearcher, we match the generation budget, evaluation questions and splits, decoding, and answer aggregation rules, while allowing retrieval top-K and the search budget to vary.
-
Retrieval top-K and search budget. These settings depend on how each method handles retrieved context. MemSearcher uses top-5 retrieval and a maximum of 20 searches. Methods that accumulate context, such as Search-R1, use top-3 retrieval and a maximum of 4 searches. Evidence-R1 uses top-20 retrieval and a maximum of 10 searches. To examine these settings, we reproduced Search-R1 results at different values of . Increasing to 5 reduced performance, consistent with Table 7 of the Search-R1 paper. We hypothesize that a larger further lengthens the context as these methods keep appending retrieval results, which can hurt performance. MemSearcher controls context length by updating its memory, which allows it to use more searches. These results suggest choosing top-K and the search budget to suit how each method manages its context.
-
Matched retrieval, memory, and generation budgets in Table 2. Evidence-R1, MemSearcher, and Search-R1 use the same Qwen2.5-3B-Instruct backbone, top-5 retrieval, and search limits of or . In the MemSearcher and Evidence-R1 runs, we match the cross-turn memory budget, the visible observation token budget per retrieval, and the total generation budgets per logical search round and per question. MemSearcher's memory and Evidence-R1's evidence state are both capped at 1,024 tokens, and the observation budget is 4,096 tokens for both. MemSearcher uses 512 tokens for Action and 512 for Memory, while Evidence-R1 uses 512 for Action and 512 for Commit.
-
Source of the context reduction. To test the effect of the commit protocol on context cost, we compared B1 (Soft Commit) and B4 (Hard Commit). Both use the same evidence-state capacity, per-retrieval input limit, and generation token cap per logical search round. Switching to Hard Commit reduces peak context length by 7.2% on NQ and 26.8% on HotpotQA, showing that the protocol reduces peak context length with the same budget limits.
Response to Reviewer Xgfe - Post (3)
W5. Search depth, context cost, and evaluation on longer trajectories
Thank you for pointing out this limitation. Evidence-R1 averages 2.07 searches under both search limits in Table 2. The protocol evicts the raw retrieval results after the first retrieval and commit. The next Action then rebuilds its context from the question, query history, and current evidence state. The efficiency conclusions from Table 2 are limited to the search depths observed in that evaluation. Whether the cost advantage and answer accuracy hold over more search rounds requires further evaluation.
We compared Evidence-R1 and MemSearcher on the same questions from the MuSiQue-Ans development split, grouping questions by their annotated reasoning hop count. We used the official hop counts only for grouping. Each method uses its Qwen2.5-3B-Instruct checkpoint from Table 2. We match the retriever and corpus, top-5 retrieval, state and generation budgets, and decoding settings, with for both methods. To characterize the search depth covered, we report the average number of searches actually made and the proportion of questions for which each method made at least four searches.
| Annotated hop count | Method | Questions | Avg. searches | Searches ≥4 (%) | EM (%) | Peak context (tokens/question) | Cumulative tokens (tokens/question) |
|---|---|---|---|---|---|---|---|
| 2-hop | MemSearcher | 1,252 | 3.25 | 32.11 | 25.88 | 2751 | 8611 |
| 2-hop | Evidence-R1 | 1,252 | 3.07 | 27.56 | 26.28 | 1,293.85 | 5,849.85 |
| 3-hop | MemSearcher | 760 | 4.35 | 55.79 | 11.45 | 3038 | 12492 |
| 3-hop | Evidence-R1 | 760 | 3.86 | 49.47 | 10.92 | 1,350.29 | 7,643.08 |
| 4-hop | MemSearcher | 405 | 5.85 | 80.49 | 8.64 | 3347 | 18366 |
| 4-hop | Evidence-R1 | 405 | 5.38 | 75.56 | 7.90 | 1,457.75 | 11,177.67 |
On the 405 four-hop questions, Evidence-R1 averages 5.38 searches and MemSearcher averages 5.85, with at least four searches on 75.56% and 80.49% of questions, respectively. These trajectories are longer than the overall average of about two searches in Table 2. Evidence-R1 uses less peak context and fewer cumulative tokens than MemSearcher in all three hop groups. Its EM is 0.40 percentage points higher in the two-hop group and 0.53 and 0.74 percentage points lower in the three-hop and four-hop groups, respectively. In the four-hop group, which has the most searches, Evidence-R1 reduces peak context length by 56.4% and cumulative tokens by 39.1%, with 7.90% EM compared with 8.64% for MemSearcher.
Response to Reviewer Xgfe - Post (1)
Thank you for your detailed technical review.
W1. Scope of the accuracy claim and updated 3B checkpoint
We agree with the reviewer's assessment of the 7B results on NQ, MuSiQue, and Bamboogle. We will revise the claim to describe Evidence-R1 as achieving competitive accuracy relative to MemSearcher.
For the original 3B result, we followed MemSearcher's setup and trained for one epoch (662 steps). After submission, we continued training and observed further gains. We have now evaluated the 3B checkpoint after 3 epochs. Its unweighted average EM is 44.5%, compared with 44.1% for the original checkpoint and 43.8% for MemSearcher. The table below reports per-dataset EM, the unweighted average, and the average weighted by evaluation question count. We will also continue training the 7B model and report the new per-dataset results once training is complete.
The unweighted average gives each dataset equal weight. The question-weighted average uses the EM and question count for each dataset to estimate overall EM across all 51,713 evaluation questions.
W2. Ablating version-lineage credit
The B3/B4 comparison shows that adding the full provenance credit mechanism improves source support when the state-shaping reward is held fixed. It leaves open whether assigning credit to the Commit Segments that produced earlier versions is more effective than a simpler source-support reward.
We designed two ablations to test this question.
The new ablations follow the 3B main experiment's setup for the global reward, state-shaping reward, evidence-state budget, observation and generation budgets, and evaluation protocol. We use Qwen3-1.7B for all three configurations because of training time constraints. Comparing the full configuration with B6 tests the effect of assigning credit to predecessor versions along UPDATE lineages. Comparing B6 with B5 tests the effect of filtering by final citations.
The table reports EM across seven benchmarks for the three configurations on Qwen3-1.7B after 300 training steps.
With Qwen3-1.7B trained for 300 steps, the full configuration achieves higher EM than B6, which rewards only the ultimately cited versions, on all seven datasets, by 5.35 percentage points on average. Compared with B5, which directly rewards every successful commit, the full configuration scores about 1.66 percentage points higher on average, with lower EM on HotpotQA and 2Wiki. The comparison with B6 shows that crediting the predecessors of ultimately cited versions can improve answer EM in this setting.