- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Code2Evidence, a retrieval framework that separates transient interaction histories from a persistent evidence notebook. Search agents explicitly bank source spans with provenance, and the final answerer primarily receives the collected evidence rather than the search trajectories. The framework further incorporates role-specialized parallel searchers and evidence reconciliation. Experiments evaluate the approach across open-domain, financial, legal, and biomedical question answering.
Critical strengths
1 The motivation is understandable. Separating useful evidence from failed searches and intermediate reasoning provides a clear interface between exploration and answer generation.
2 The single-agent comparisons and component ablations are useful. In particular, Table 3 shows that simply adding searchers is insufficient, highlighting the importance of evidence organization.
3 The evaluation covers several domains and backbones. The appendices also acknowledge important limitations, including imperfect provenance, permissive scoring, and differences between system-level comparisons.
Critical weaknesses
-
The methodological contribution is not sufficiently isolated. The framework combines explicit evidence extraction, persistent storage, role-based search, and final synthesis. Section 2 discusses related approaches, but the experiments do not directly establish the benefit of the proposed evidence-state design over a simpler persistent scratchpad or a separate model-written summary under the same retrieval interface. Differences in retrieval interfaces explain why existing systems cannot be compared directly, but do not remove the need for such controlled alternatives.
-
The experiments do not establish why evidence-state separation helps. Introducing the notebook also changes agent behavior, context content, and compression events. Appendix G.2 reports that collapse occurs in 175/300 history-only runs and all 300 notebook runs, while only one history-only run reaches the compaction threshold. Therefore, the improvement cannot yet be specifically attributed to protecting evidence from compression. A controlled experiment using fixed search traces, different compression strengths, and comparable context budgets would better distinguish evidence preservation from more deliberate search or additional context. Moreover, Table 1 shows that adding trajectories still benefits GPT-5.4-mini on several tasks, limiting the generality of the evidence-only answer interface.
-
Equal turn budgets do not establish comparable computational cost. Appendix G.4 reports that the single-agent notebook increases token usage by approximately 32%, and the default three-searcher configuration uses substantially more tokens than the single-agent baseline. The reconciler’s token usage is excluded from Table 16. Since reconciliation produces a major improvement in Table 3, its cost should be included. Please provide a token-budget-matched comparison with a stronger single-agent baseline, alongside complete accounting of auxiliary calls and retries, to show whether the gains justify the additional computation.
-
The evaluation protocol weakens several headline results. Appendix C.1 evaluates FinQA on training-split subsets and uses a scorer that accepts sign differences and several scale transformations. These are substantive errors in financial reasoning, so strict numerical accuracy should also be reported on a held-out split. On Legal RAG Bench, Appendix D.3 attributes 34 empty GPT-5.4-mini DCI outputs to extraction or termination failures. This makes the reported 19-point improvement difficult to interpret as a methodological gain. Please separate answer-quality improvements from output-handling failures using a validated common extraction procedure.
-
Answer accuracy does not adequately validate the evidence mechanism. The evidence notebook is central to the paper, but its quality is mainly assessed through answer-string presence rather than source fidelity and support for the required reasoning steps. Appendix F.4 even presents a correctly scored answer supported by a protocol with a conflicting temperature condition. Please evaluate whether banked spans faithfully match their cited sources and satisfy the question’s constraints, and whether reconciliation removes necessary evidence. This is particularly relevant because provenance is model-written rather than automatically checked.
Questions for authors
See weakness
2 comments
Response to Reviewer AgSP: Part 2
W4. Evaluation Protocol
FinQA Evaluation Schema. We thank the reviewer for raising this point about the evaluation protocol. First, the same scorer was applied to every system, so it does not favour any one of them. Second, the lenient matching addresses an inconsistency in FinQA's own reference answers. Gold answers for the same kind of quantity are recorded at different scales, sometimes as percentages (12.5) and sometimes as decimals (0.125), and monetary values are given in different units (thousands or millions). Questions about a change also give the magnitude as the reference while the question's wording fixes the direction ("by how much did X decrease"), so a correct signed answer can differ from the reference in sign. A strict comparison would therefore mark many correct answers wrong.
To check that this leniency does not drive our conclusions, we re-scored all saved outputs under two stricter rules: one that requires the sign to match and allows only the percent ↔ decimal form (×100), and one that requires the sign to match and allows no scaling at all. Strict scoring lowers every system's accuracy by 10–25 points but does not change the comparisons. Against the compute-matched parallel baseline (DCI×3), Code2Evidence's three-searcher system leads by +4.67 under the original scorer and by +6.00 under the strict rule, because the baseline's answers more often have the wrong sign or scale.
Legal Experiments. We thank the reviewer for pointing it out. The empty answers occurred because the DCI agent used up its max turns and produced no answer. We re-ran the failed questions and judged all runs with the same judge. Our framework still outperforms DCI-agents in LegalRagBench in terms of accuracy:
| Answered | Correct | Annotated passage retrieved | |
|---|---|---|---|
| DCI, original run | |||
| DCI, failures re-run | |||
| Code2Evidence |
W5. Fidelity
Fidelity of entries. A GPT-4.1 judge checks each entry against the cited passage, Only (GPT-5.4-mini) and (GPT-5.5) of reconciled entries are judged unsupported.
Support for the required reasoning steps. Measured on 150 multi-hop questions (HotpotQA, 2Wiki and MuSiQue), using gold supporting titles and MuSiQue's hop answers, on GPT-5.4-mini:
| Portion | |
|---|---|
| Notebooks containing the gold answer | 51.4% |
| Supporting passages cited | 44.0% |
| All supporting passages cited | 22.0% |
| MuSiQue hop answers present | 52.3% |
| Retrieved supporting passages that were banked | 76.1% |
Accuracy tracks coverage. For the notebook, it is 81.8% when all supporting passages are cited, 45.7% when some are, and 19.1% when none are.
Does reconciliation remove necessary evidence? This concerns the three-searcher system, which reconciles plain notebooks (MHQA, n = 300):
| GPT-5.4-mini | GPT-5.5 | |
|---|---|---|
| Gold answer kept when at least one searcher had it | 95.8% | 95.0% |
| … when only one searcher had it | 84.4% | 70.6% |
| Gold supporting passages still cited | 84.7% | 90.8% |
| Reconciled entries judged unsupported | 3.0% | 0.9% |
| Names or numbers introduced by the reconciler | 0.4% | 0.0% |
| Planted contradicting false entry kept | 1.3% | 0.7% |
The reconciler rarely invents content. It does sometimes drop evidence that only one searcher found. We will report this as a limitation.
F.4. We agree that the Appendix F.4 example rests on a protocol whose temperature condition conflicts with the question. Our fidelity and constraint checks cover MHQA only, not BioProBench, where evidence constraints are not annotated. We will state this and remove F.4 as a positive example.
Response to Reviewer AgSP: Thank you for your review!
W1. Baseline Comparison
We agree that other memory-based alternatives should be tested. Below, we provide additional experiments done on MHQA, using GPT-5.4-mini, the same tools, turns (max 30 turns), and questions.
W2. Notebook Contribution
We agree that, for a single agent, giving the answerer the trajectory alongside the notebook can help. In that setting, we recommend pairing them. In the multi-agent setting, instead of concatenating long, noisy search trajectories, searchers can hand over notebooks, which can be merged and reconciled into one compact evidence set for the answerer. This is more efficient and effective than combining raw trajectories.
The evidence-only interface pays off where trajectories stop scaling: when several searchers run in parallel. Instead of handing the answerer long, noisy trajectories, the searchers hand over notebooks, which we merge into one compact evidence set. We tested this directly by holding 3, 5, and 7 parallel searchers' searches fixed and changing only what the answerer receives (GLM-5.3, 300 MHQA questions):
Notebooks scale; trajectories do not. As searchers are added, accuracy with notebooks rises, while accuracy with trajectories under the same context budget falls.
Full trajectories need 16× the input to keep up. With seven searchers, passing every trajectory in full requires 36.3K tokens of answerer input, against 2.2K for notebooks. That input grows linearly with the number of searchers.
Notebooks beat exchanging answers. Notebooks outperform majority voting over searchers' answers.
W3. Compute Fairness
We agree that an equal turn cap does not imply equal compute. Thus, we provide additional experiments below. We swept Code2Evidence over 1–3 searchers and 15 turns per searcher. We added DCI×3: three independent DCI runs combined, with all tokens counted.
At roughly matched compute, Code2Evidence is better than DCI in terms of accuracy. Note that our C2E framework actually performs better (52.33 vs. 48.33) on the max 30-turn cap scenario than the reported 15-turn scenario here.
Reconciler cost. The three-searcher system hands span-only notebooks to the reconciler. Its reconciler was not logged before. Reconstructed from its inputs and outputs, it uses about 0.7K tokens per question, 0.4% of the 159.6K logged for three searchers (hidden reasoning tokens excluded). We add it to all totals.