acceptodds
Under review as a conference paper at ICLR 2027

Code2Evidence: Explicit Knowledge Organization for Multi-Agent Search and Reasoning

24 views

Abstract

Retrieval agents that search a corpus over many steps keep everything they have done in one interaction history, where the few observations that support the answer sit among irrelevant hits, search errors, failed commands, and the agent's own reasoning traces. This mixture can mislead later search and the final reasoning step, and truncating or summarizing the history can discard supporting evidence along with the noise. We argue that retrieval agents should keep two separate states: a transient interaction history that may be compressed freely, and a persistent evidence state of source spans, with provenance, that the agent explicitly banks during search. The evidence state is never compressed, guides further search, and is the only input to the final answer. We implement this separation in Code2Evidence, a coding-based retrieval agent that searches raw corpora with shell commands. Holding the model, tools, turn budget, and compression policy fixed, answering from the evidence state matches or beats answering from the compressed history in all ten task–backbone pairs across six Wikipedia QA datasets (MHQA), FinQA, and BioProBench with GPT-5.4-mini and GPT-5.5. With GLM-5.3 on MHQA, adding the evidence state to a single agent raises meaning-match accuracy from 74.3% to 80.7%, and reconciling the evidence states of three role-specialized searchers reaches 85.2%. Code2Evidence is also more accurate than DCI-Agent-Lite on FinQA with all four backbones and on MHQA (85.2% vs. 79.4%), with the largest gains for GPT-5.4-mini on Legal RAG Bench and BioProBench.

Primary area:
generative models
Keywords:
Retrieval Augmented Generation, Large Language Models, Open-Domain Question Answering, Coding-based Retrieval

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_AgSPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Code2Evidence, a retrieval framework that separates transient interaction histories from a persistent evidence notebook. Search agents explicitly bank source spans with provenance, and the final answerer primarily receives the collected evidence rather than the search trajectories. The framework further incorporates role-specialized parallel searchers and evidence reconciliation. Experiments evaluate the approach across open-domain, financial, legal, and biomedical question answering.

Critical strengths

1 The motivation is understandable. Separating useful evidence from failed searches and intermediate reasoning provides a clear interface between exploration and answer generation.

2 The single-agent comparisons and component ablations are useful. In particular, Table 3 shows that simply adding searchers is insufficient, highlighting the importance of evidence organization.

3 The evaluation covers several domains and backbones. The appendices also acknowledge important limitations, including imperfect provenance, permissive scoring, and differences between system-level comparisons.

Critical weaknesses
  1. The methodological contribution is not sufficiently isolated. The framework combines explicit evidence extraction, persistent storage, role-based search, and final synthesis. Section 2 discusses related approaches, but the experiments do not directly establish the benefit of the proposed evidence-state design over a simpler persistent scratchpad or a separate model-written summary under the same retrieval interface. Differences in retrieval interfaces explain why existing systems cannot be compared directly, but do not remove the need for such controlled alternatives.

  2. The experiments do not establish why evidence-state separation helps. Introducing the notebook also changes agent behavior, context content, and compression events. Appendix G.2 reports that collapse occurs in 175/300 history-only runs and all 300 notebook runs, while only one history-only run reaches the compaction threshold. Therefore, the improvement cannot yet be specifically attributed to protecting evidence from compression. A controlled experiment using fixed search traces, different compression strengths, and comparable context budgets would better distinguish evidence preservation from more deliberate search or additional context. Moreover, Table 1 shows that adding trajectories still benefits GPT-5.4-mini on several tasks, limiting the generality of the evidence-only answer interface.

  3. Equal turn budgets do not establish comparable computational cost. Appendix G.4 reports that the single-agent notebook increases token usage by approximately 32%, and the default three-searcher configuration uses substantially more tokens than the single-agent baseline. The reconciler’s token usage is excluded from Table 16. Since reconciliation produces a major improvement in Table 3, its cost should be included. Please provide a token-budget-matched comparison with a stronger single-agent baseline, alongside complete accounting of auxiliary calls and retries, to show whether the gains justify the additional computation.

  4. The evaluation protocol weakens several headline results. Appendix C.1 evaluates FinQA on training-split subsets and uses a scorer that accepts sign differences and several scale transformations. These are substantive errors in financial reasoning, so strict numerical accuracy should also be reported on a held-out split. On Legal RAG Bench, Appendix D.3 attributes 34 empty GPT-5.4-mini DCI outputs to extraction or termination failures. This makes the reported 19-point improvement difficult to interpret as a methodological gain. Please separate answer-quality improvements from output-handling failures using a validated common extraction procedure.

  5. Answer accuracy does not adequately validate the evidence mechanism. The evidence notebook is central to the paper, but its quality is mainly assessed through answer-string presence rather than source fidelity and support for the required reasoning steps. Appendix F.4 even presents a correctly scored answer supported by a protocol with a conflicting temperature condition. Please evaluate whether banked spans faithfully match their cited sources and satisfy the question’s constraints, and whether reconciliation removes necessary evidence. This is particularly relevant because provenance is model-written rather than automatically checked.

Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer AgSP: Thank you for your review!

W1. Baseline Comparison

We agree that other memory-based alternatives should be tested. Below, we provide additional experiments done on MHQA, using GPT-5.4-mini, the same tools, turns (max 30 turns), and questions.

DesignAccuracyTokens per questionEntries supported by their source
History only (no memory)48.3348.3380.2K80.2\text{K}Not Applicable
Model-written summary notes, editable (MemGPT/A-Mem-style; answer from the notes)47.3347.3373.5K73.5\text{K}Not Applicable
Editable notes plus archived sources (FS-Researcher-style)51.3351.33117.1K117.1\text{K}59.759.7\\% of notes
Code2Evidence, single-agent52.33\mathbf{52.33}69.6K\mathbf{69.6\text{K}}\mathbf{91.8\\%} of entries
Code2Evidence, 3-agents (15 turns max)54.00\mathbf{54.00}159.3K\mathbf{159.3\text{K}}\mathbf{96.1\\%} of entries
Against these alternatives, Code2Evidence is:
  • Effective. Our framework is numerically ahead of model-written summary notes (+5.00+5.00) and of history (+4.00+4.00);
  • Verifiable. 91.891.8\\% of entries are judged supported by their cited source in the single-agent setting and 96.196.1\\% in the multi-agent setting, against 59.759.7\\% of FS-Researcher-style notes. Model-written memory carries the writer's errors forward as facts. Our quoted span lets the answerer check every entry against the raw output.
  • Composable. Because entries are self-contained and sourced, parallel searchers can hand over notebooks instead of trajectories. With three searchers, the notebook hand-off is at least as accurate as the clipped trajectories (GPT-5.4-mini +3.67+3.67), with about a tenth of the answerer's input (0.3K0.3\text{K} vs. 2.9K2.9\text{K} tokens). The reconciled notebook can be audited:
    • Only 3.03.0\\% (GPT-5.4-mini) and 0.90.9\\% (GPT-5.5) of reconciled entries are judged unsupported;
    • the gold answer is kept in 95.895.8\\% and 95.095.0\\% of cases where any searcher found it;
    • a planted contradicting false entry is kept in only 1.31.3\\% and 0.70.7\\% of cases.

W2. Notebook Contribution

We agree that, for a single agent, giving the answerer the trajectory alongside the notebook can help. In that setting, we recommend pairing them. In the multi-agent setting, instead of concatenating long, noisy search trajectories, searchers can hand over notebooks, which can be merged and reconciled into one compact evidence set for the answerer. This is more efficient and effective than combining raw trajectories.

The evidence-only interface pays off where trajectories stop scaling: when several searchers run in parallel. Instead of handing the answerer long, noisy trajectories, the searchers hand over notebooks, which we merge into one compact evidence set. We tested this directly by holding 3, 5, and 7 parallel searchers' searches fixed and changing only what the answerer receives (GLM-5.3, 300 MHQA questions):

Notebooks scale; trajectories do not. As searchers are added, accuracy with notebooks rises, while accuracy with trajectories under the same context budget falls.

Full trajectories need 16× the input to keep up. With seven searchers, passing every trajectory in full requires 36.3K tokens of answerer input, against 2.2K for notebooks. That input grows linearly with the number of searchers.

Notebooks beat exchanging answers. Notebooks outperform majority voting over searchers' answers.

W3. Compute Fairness

We agree that an equal turn cap does not imply equal compute. Thus, we provide additional experiments below. We swept Code2Evidence over 1–31\text{--}3 searchers and 1515 turns per searcher. We added DCI×3\text{DCI}\times 3: three independent DCI runs combined, with all tokens counted.

Comparison (tokens/question)Difference
MHQA: C2E 11 searcher, 48.33%48.33\% (59.4K59.4\text{K}) vs. DCI, 48.33%48.33\% (68.7K68.7\text{K})+0.00+0.00
MHQA: C2E 33 searchers, 54.00%54.00\% (159.3K159.3\text{K}) vs. DCI×3\text{DCI}\times 3, 52.00%52.00\% (212K212\text{K})+2.00+2.00
FinQA: C2E 11 searcher, 49.11%49.11\% (44K44\text{K}) vs. DCI, 49.11%49.11\% (41K41\text{K})+0.00+0.00
FinQA: C2E 33 searchers, 57.11%57.11\% (147K147\text{K}) vs. DCI×3\text{DCI}\times 3, 52.44%52.44\% (134K134\text{K})+4.67+4.67
LegalRagBench: C2E 11 searcher, 66%66\% (45.0K45.0\text{K}) vs. DCI, 65%65\% (42.8K42.8\text{K})+1.00+1.00
LegalRagBench: C2E 33 searchers, 72%72\% (177.3K177.3\text{K}) vs. DCI×3\text{DCI}\times 3, 69%69\% (135.4K135.4\text{K})+3.00+3.00

At roughly matched compute, Code2Evidence is better than DCI in terms of accuracy. Note that our C2E framework actually performs better (52.33 vs. 48.33) on the max 30-turn cap scenario than the reported 15-turn scenario here.

Reconciler cost. The three-searcher system hands span-only notebooks to the reconciler. Its reconciler was not logged before. Reconstructed from its inputs and outputs, it uses about 0.7K tokens per question, 0.4% of the 159.6K logged for three searchers (hidden reasoning tokens excluded). We add it to all totals.

AuthorsOpenReview ↗

Response to Reviewer AgSP: Part 2

W4. Evaluation Protocol

FinQA Evaluation Schema. We thank the reviewer for raising this point about the evaluation protocol. First, the same scorer was applied to every system, so it does not favour any one of them. Second, the lenient matching addresses an inconsistency in FinQA's own reference answers. Gold answers for the same kind of quantity are recorded at different scales, sometimes as percentages (12.5) and sometimes as decimals (0.125), and monetary values are given in different units (thousands or millions). Questions about a change also give the magnitude as the reference while the question's wording fixes the direction ("by how much did X decrease"), so a correct signed answer can differ from the reference in sign. A strict comparison would therefore mark many correct answers wrong.

To check that this leniency does not drive our conclusions, we re-scored all saved outputs under two stricter rules: one that requires the sign to match and allows only the percent ↔ decimal form (×100), and one that requires the sign to match and allows no scaling at all. Strict scoring lowers every system's accuracy by 10–25 points but does not change the comparisons. Against the compute-matched parallel baseline (DCI×3), Code2Evidence's three-searcher system leads by +4.67 under the original scorer and by +6.00 under the strict rule, because the baseline's answers more often have the wrong sign or scale.

Legal Experiments. We thank the reviewer for pointing it out. The empty answers occurred because the DCI agent used up its max turns and produced no answer. We re-ran the failed questions and judged all runs with the same judge. Our framework still outperforms DCI-agents in LegalRagBench in terms of accuracy:

AnsweredCorrectAnnotated passage retrieved
DCI, original run666654544545
DCI, failures re-run9595656551\mathbf{51}
Code2Evidence98\mathbf{98}72\mathbf{72}3939

W5. Fidelity

Fidelity of entries. A GPT-4.1 judge checks each entry against the cited passage, Only 3.03.0\\% (GPT-5.4-mini) and 0.90.9\\% (GPT-5.5) of reconciled entries are judged unsupported.

Support for the required reasoning steps. Measured on 150 multi-hop questions (HotpotQA, 2Wiki and MuSiQue), using gold supporting titles and MuSiQue's hop answers, on GPT-5.4-mini:

Portion
Notebooks containing the gold answer51.4%
Supporting passages cited44.0%
All supporting passages cited22.0%
MuSiQue hop answers present52.3%
Retrieved supporting passages that were banked76.1%

Accuracy tracks coverage. For the notebook, it is 81.8% when all supporting passages are cited, 45.7% when some are, and 19.1% when none are.

Does reconciliation remove necessary evidence? This concerns the three-searcher system, which reconciles plain notebooks (MHQA, n = 300):

GPT-5.4-miniGPT-5.5
Gold answer kept when at least one searcher had it95.8%95.0%
… when only one searcher had it84.4%70.6%
Gold supporting passages still cited84.7%90.8%
Reconciled entries judged unsupported3.0%0.9%
Names or numbers introduced by the reconciler0.4%0.0%
Planted contradicting false entry kept1.3%0.7%

The reconciler rarely invents content. It does sometimes drop evidence that only one searcher found. We will report this as a limitation.

F.4. We agree that the Appendix F.4 example rests on a protocol whose temperature condition conflicts with the question. Our fidelity and constraint checks cover MHQA only, not BioProBench, where evidence constraints are not annotated. We will state this and remove F.4 as a positive example.

Reviewer_C48VOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Search agents that explore a corpus with shell commands usually keep all the information in one long and noisy history. As a result, the few lines that support the answer can get buried, or even lost when the agent compacts the history. To resolve this, the authors propose keeping an evidence list of exact source lines in addition to the compactable history. This is an append-only list that the agent can check at any time. A separate answerer without tools sees only the question and this evidence. Code2Evidence also runs three searchers with different strategies and merges their evidence.

On multi-hop, financial and biology tasks, this method matches or beats answering directly from the raw history in all 10 task-model pairs (+0 to +4 points). With the same 50-turn budget, it also beats a strong baseline agent on multi-hop QA (85.2% vs. 79.4%) and on FinQA with all four models.

Critical strengths

S1. The paper finds a real problem in long search agents. The key evidence for the answer can get buried in a long, messy history. It can also be lost when the history is compacted. And the authors’ fix is simple. It works on top of any existing shell-based agent without training.

S2. The main experiment is well controlled. The model, tools, turn limit, and compaction methods stay the same. Only the evidence list or the answerer’s input changes. The results point the same way in all 10 task–model pairs.

Critical weaknesses

W1. The contribution is incremental. The paper's main contribution is simply to keep an append-only evidence list in addition to the original history. This is close to existing memory-related work for agents, such as A-Mem (Xu et al., NeurIPS 2025).

W2. The comparison with the baseline is not fair in compute. Both systems use 50 turns, but Code2Evidence uses many more tokens. On Legal RAG Bench, it uses 46.6M vs. 2.9M tokens with GPT-5.4-mini, and 41.3M vs. 9.3M with GPT-5.5 ($73.04 vs. $5.89; Table 6). Also, the biggest gain (53% → 72%) happens when the baseline gives 34 empty answers (66% vs. 98% answered). The authors should better justify why this comparison shows that their method is better.

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer C48V: Thank you for your review!

We thank the reviewer for the valuable opinions. Unless stated otherwise, the new numbers use GPT-5.4-mini, paired tests over the same questions, with n=300n = 300 for MHQA and n=450n = 450 for FinQA.

W1. Incremental Novelty

We argue that our notebook solution is a simple yet efficient and effective solution for the long-trajectory RAG scenario, especially in the Multi-Agent setting. Code2Evidence keeps the evidence an agent finds in an append-only notebook whose entries point back to the raw tool outputs they came from. This design makes long-trajectory RAG auditable. In the multi-agent setting, it also changes what parallel searchers can exchange: instead of concatenating long, noisy search trajectories, they hand over notebooks, which can be merged and reconciled into one compact evidence set for the answerer.

We will cite and discuss A-Mem (Xu et al., NeurIPS 2025) and other related works. Their design choices and problem scope differ from ours:

Key design differences.

PropertyMemGPTA-MemMem0RECOMPFS-ResearcherCode2Evidence
(1) Built up over a multi-step agentic search (not a single retrieval)✓✓✓✗ (one retrieval per query)✓✓
(2) The agent selects what to keep for the taskpartly (core/archival yes; recall stores all)✗ (stores every interaction)✓ (extracts facts)✓ (compressor selects)✓ (notes)✓
(3) What is kept is the unaltered sourcepartly (recall only)✓ (raw content)✗ (model-written facts)partly (extractive variant only)partly (archived sources, separate from notes)✓ (a quoted span with file and line citation)
(4) (2) and (3) hold for the same entry: each selected item is bound to its unaltered source✗✗✗partly (extractive, single step)✗ (citations optional)✓
(5) A separate reader sees only the selection✗✗✗✓✓✓
(6) Multi-agent coordination, by merging the artifacts of parallel agents✗✗✗✗✗✓ (Notebook Merging)

What each difference buys, measured on the same harness. Other designs store what the model wrote about the evidence. Code2Evidence also stores where the evidence is. Setting: MHQA, GPT-5.4-mini, same tools, turns (max 30 turns), and questions. The memory alternatives implement each design's core policy. Differences are paired.

DesignAccuracyTokens per questionEntries supported by their source
History only (no memory)48.3348.3380.2K80.2\text{K}Not Applicable
Model-written summary notes, editable (MemGPT/A-Mem-style; answer from the notes)47.3347.3373.5K73.5\text{K}Not Applicable
Editable notes plus archived sources (FS-Researcher-style)51.3351.33117.1K117.1\text{K}59.759.7\\% of notes
Code2Evidence, single-agent52.33\mathbf{52.33}69.6K\mathbf{69.6\text{K}}\mathbf{91.8\\%} of entries
Code2Evidence, 3-agents (15 turns max)54.00\mathbf{54.00}159.3K\mathbf{159.3\text{K}}\mathbf{96.1\\%} of entries

Against these alternatives, Code2Evidence is:

  • Effective. Our framework is numerically ahead of model-written summary notes (+5.00+5.00) and of history (+4.00+4.00);
  • Verifiable. 91.891.8\\% of entries are judged supported by their cited source in the single-agent setting and 96.196.1\\% in the multi-agent setting, against 59.759.7\\% of FS-Researcher-style notes. Model-written memory carries the writer's errors forward as facts. Our quoted span lets the answerer check every entry against the raw output.
  • Composable. Because entries are self-contained and sourced, parallel searchers can hand over notebooks instead of trajectories. With three searchers, the notebook hand-off is at least as accurate as the clipped trajectories (GPT-5.4-mini +3.67+3.67), with about a tenth of the answerer's input (0.3K0.3\text{K} vs. 2.9K2.9\text{K} tokens). The reconciled notebook can be audited:
    • Only 3.03.0\\% (GPT-5.4-mini) and 0.90.9\\% (GPT-5.5) of reconciled entries are judged unsupported;
    • the gold answer is kept in 95.895.8\\% and 95.095.0\\% of cases where any searcher found it;
    • a planted contradicting false entry is kept in only 1.31.3\\% and 0.70.7\\% of cases.

In the revision, we will:

  • Add a comparison table, clearly distinguishing our innovations and advantages against memory-related baselines;
  • Include the same-harness performance comparison against baselines and the analysis.
AuthorsOpenReview ↗

Response to Reviewer C48V: Part 2

W2. Compute Fairness; Legal RAG Bench Flaw

Compute fairness. We agree that an equal turn cap does not imply equal compute. The two systems use their budgets very differently. DCI-Agent-Lite is a single agent that usually stops well below its cap (median: 5–6 turns with GPT-5.4-mini). Code2Evidence's default configuration runs three parallel searchers with up to 15 turns each, followed by an answering stage. Under the same nominal budget, it therefore consumes several times DCI's actual tokens, which is why we now compare the systems at both accuracy and token budgets (below).

On the Legal RAG Bench, we found a separate source of cost after paper submission, caused by an implementation bug. The Legal adaptation instructed the agent to list the workspace before searching, and the resulting 120,000-character file listing (all 4,876 passages) stayed in context on every later turn. This added about 65K prompt tokens to each model call, which accounts for most of the 46.6M-token total in Table 6. We will update Table 6 with the corrected framework. We sincerely thank the reviewer for pointing out the cost discrepancy.

More Details on Cost. We swept Code2Evidence over 1–31\text{--}3 searchers and 1515 turns per searcher (max number of turns 15 for all searchers). We added DCI×3\text{DCI}\times 3: three independent DCI runs combined, with all tokens counted.

Comparison (tokens/question)Difference
MHQA: C2E 11 searcher, 48.33%48.33\% (59.4K59.4\text{K}) vs. DCI, 48.33%48.33\% (68.7K68.7\text{K})+0.00+0.00
MHQA: C2E 33 searchers, 54.00%54.00\% (159.3K159.3\text{K}) vs. DCI×3\text{DCI}\times 3, 52.00%52.00\% (212K212\text{K})+2.00+2.00
FinQA: C2E 11 searcher, 49.11%49.11\% (44K44\text{K}) vs. DCI, 49.11%49.11\% (41K41\text{K})+0.00+0.00
FinQA: C2E 33 searchers, 57.11%57.11\% (147K147\text{K}) vs. DCI×3\text{DCI}\times 3, 52.44%52.44\% (134K134\text{K})+4.67+4.67
LegalRagBench: C2E 11 searcher, 66%66\% (45.0K45.0\text{K}) vs. DCI, 65%65\% (42.8K42.8\text{K})+1.00+1.00
LegalRagBench: C2E 33 searchers, 72%72\% (177.3K177.3\text{K}) vs. DCI×3\text{DCI}\times 3, 69%69\% (135.4K135.4\text{K})+3.00+3.00

At roughly matched compute, Code2Evidence is better than DCI in terms of accuracy. Note that our C2E framework actually performs better (52.33 vs. 48.33) under the max 30-turn cap scenario than the reported 15-turn scenario here.

Legal Experiments We thank the reviewer for pointing it out. The empty answers occurred because the DCI agent used up its max turns and produced no answer. We re-ran the failed questions and judged all runs with the same judge. Our framework still outperforms DCI-agents in LegalRagBench in terms of accuracy:

AnsweredCorrectAnnotated passage retrieved
DCI, original run666654544545
DCI, failures re-run9595656551\mathbf{51}
Code2Evidence98\mathbf{98}72\mathbf{72}3939

In the revision, we will:

  • replace the turn-matched headline comparisons with accuracy-vs-tokens and accuracy-vs-dollars curves;
  • include DCI×3\text{DCI}\times 3 and a cheap iterative BM25 baseline in those curves (49.67%49.67\% on MHQA at 8.0K8.0\text{K} tokens per question).

Thank you for the rebuttal. It addressed most of my main concerns, so I’ve raised my rating to 2.

Reviewer_jWhpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Code2Evidence separates a retrieval agent's compressible interaction history from a persistent evidence notebook containing selected source spans and provenance. Searchers use shell commands over a corpus and explicitly bank evidence; a separate answerer consumes the evidence state. The multi-agent extension uses specialized search roles and optional reconciliation. The paper evaluates controlled single-agent variants and full systems on Wikipedia QA, FinQA, Legal RAG Bench, and BioProBench, with analyses of accuracy, evidence coverage, and cost.

Critical strengths

In Table 3, adding an evidence notebook to a single agent improves meaning-match accuracy from 74.3±0.82 to 80.7±1.63 across three question samples. The full system reaches 85.2, compared with 79.7 for mechanical merging and 79.2 without role specialization. Appendix G.1 supplies paired evidence for several comparisons on one sample. These results support both the practical notebook recipe and the value of organizing complementary search outputs; they do not merely show that adding workers helps.

Critical weaknesses

The paper explains how it captures model calls, reconstructs samples, and manages rollouts, but does not identify which capabilities required new engineering beyond Orchard or the similar approach acknowledged in Polar. One worked comparison of a difficult integration—such as multimodal observations or nested harness calls—would help establish that boundary. Appendix B already reports hardware and training times; efficiency leadership is not necessary for the systems contribution, and I do not require a comprehensive scaling study. However, the unrestricted “any harness × any environment” wording exceeds the interfaces actually demonstrated.

Questions for authors

Notebook use changes the available actions and instructions, triggers extra collapse events, and increases single-agent logged tokens from 78K to 103K (B.9, Table 16). These are meaningful parts of the implemented intervention, so the gain cannot be attributed uniquely to durable storage or evidence surviving compression. A further ambiguity is that B.9 says notebook references were removed from the history-only placeholders, whereas G.2 says they remained; the latter could disadvantage the control if the agent is told to rely on nonexistent notes. This is an unresolved execution-version question, not proof that the control was defective. Likewise, searcher-proposal fallbacks occur in the full system (E.5), so “evidence-only” needs to distinguish its intended path from all scored outputs. These issues limit causal and architectural claims but do not currently erase the observed gains of the overall notebook recipe.

1 comment
AuthorsOpenReview ↗

Response to Reviewer jWhp: Thank you for your review!

We thank the reviewer for the valuable opinions. We provide our clarifications and additional experiments below.

0. Asking for Clarifications on Weaknesses

Could the authors clarify what "Orchard" and "Polar" refer to in your stated weaknesses? Also, the scope of the paper does not include multimodal RAG, and we did not claim “any harness × any environment” in our paper.

1. What Generated the Reported Comparison

The comparison the reviewer cites (GLM-5.3 MHQA, single agent, 74.33%74.33\% without notebook vs. 81.00%81.00\% with notebook; 78K78\text{K} vs. 103K103\text{K} tokens) comes from the released sample-50 runs.

History-only control (as run)Notebook arm
ToolsShell, file list, calculator; no notebook tools (00 notebook-tool calls in 300300 questions)Same, plus append_evidence / read_evidence
WorkspaceSame README as the notebook arm (without "Bank relevant snippets into evidence.txt via append_evidence")README plus evidence.txt
Compression placeholdersDid not instruct agent to "see evidence.txt for banked facts"Told the agent to "see evidence.txt for banked facts"
Progress note at each collapseModel-written; asks what relevant facts have been foundModel-written
Collapse events per question1.281.28 (after 55 idle rounds)3.463.46 (after 55 idle rounds and after every banking step)
AnswerThe agent's own final message, from its historyThe agent's final message; the forced last turn answers only from the notebook
Questions hitting the 3030-turn cap1010 (no answer, scored wrong)1212 (no answer, scored wrong)

B.9 versus G.2. B.9 is correct for these runs. B.9 described the intended protocol; thus, we have the "seen and banked" column to be "not applicable" for history only.


2. The Tightened Comparison

Since submission, we have rerun the controlled comparison under a protocol that ensures each of the settings below:

  • The history-only arm has no notebook tools, no evidence.txt, no README mention, and no notebook wording in any placeholder.
  • The collapse step is note-free: no model-written progress note, which had acted as an implicit evidence state for the control.
  • Banking does not trigger extra collapses, so both arms collapse on the same schedule.
  • Both arms end with the same tool-free answerer and the same single-format retry, so they differ only in what the answerer reads.
  • All three models run on the same 300300 MHQA questions.

To conduct a fully controlled experiment, this setting is much stricter than the experimental settings in the paper; thus, the numbers are lower than those reported in the paper.

Model(i) History only(ii) Notebook(iii) Notebook + history(ii)−(i)\text{(ii)} - \text{(i)}
GLM-5.369.0069.0071.0071.0072.33\mathbf{72.33}+2.00+2.00
GPT-5.4-mini44.0044.0046.3346.3348.33\mathbf{48.33}+2.33+2.33
GPT-5.568.6768.6769.6769.6771.33\mathbf{71.33}+1.00+1.00

We agree with the reviewer that the token difference (78K78\text{K} vs. 103K103\text{K}) and the extra collapses were part of the intervention. The corrected protocol removes the extra collapses.


3. Fallbacks and "Evidence-Only"

  • Controlled single-agent arms. There is no searcher-proposal fallback. The shared answerer gets one format retry, identical in every arm.
  • Full three-searcher system (E.5). If the main answerer's output is missing or reports insufficient evidence, it is retried on the per-searcher notebooks. If that also fails, the first searcher's proposed answer is returned.
Reviewer_B7wGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Code2Evidence, which separates interaction history from a persistent evidence state and extends this design to multi-agent retrieval. Experiments demonstrate improvements across several retrieval and reasoning benchmarks.

Critical strengths
  1. The proposed evidence-state separation helps preserve relevant information during long-horizon retrieval.
  2. The framework is training-free and can be applied to different LLM backbones without additional model training.
Critical weaknesses
  1. Limited methodological novelty. The persistent evidence state is conceptually similar to existing memory-based approaches such as MemGPT and FS-Researcher. The multi-agent evidence merging and reconciliation mechanisms also resemble conventional designs in multi-agent RAG systems. The novelty over these methods remains unclear.
  2. Limited comparisons with competitive baselines. The main comparisons rely heavily on DCI-Agent-Lite. Other search agents are evaluated only on FinQA, while closely related memory-based and multi-agent approaches are not directly compared.
  3. Limited evaluation on smaller open-weight models. The experiments focus on relatively capable models, without evaluating smaller open-weight models such as 7B or 8B. This limits the assessment of the framework's applicability to weaker models.
Questions for authors

Please see the concerns raised in the Weaknesses section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer B7wG: Thank you for your review!

We thank the reviewer for the valuable opinions. We provide our clarifications and additional experiments below.

W1. Limited Novelty

We argue that our notebook solution is a simple yet efficient and effective solution for the long-trajectory RAG scenario, especially in the Multi-Agent setting. Code2Evidence keeps the evidence an agent finds in an append-only notebook whose entries point back to the raw tool outputs they came from. This design makes long-trajectory RAG auditable. In the multi-agent setting, it also changes what parallel searchers can exchange: instead of concatenating long, noisy search trajectories, they hand over notebooks, which can be merged and reconciled into one compact evidence set for the answerer.

We will cite and discuss related works on existing memory-based approaches. Their design choices and problem scope differ from ours:

Key design differences.

PropertyMemGPTA-MemMem0RECOMPFS-ResearcherCode2Evidence
(1) Built up over a multi-step agentic search (not a single retrieval)✓✓✓✗ (one retrieval per query)✓✓
(2) The agent selects what to keep for the taskpartly (core/archival yes; recall stores all)✗ (stores every interaction)✓ (extracts facts)✓ (compressor selects)✓ (notes)✓
(3) What is kept is the unaltered sourcepartly (recall only)✓ (raw content)✗ (model-written facts)partly (extractive variant only)partly (archived sources, separate from notes)✓ (a quoted span with file and line citation)
(4) (2) and (3) hold for the same entry: each selected item is bound to its unaltered source✗✗✗partly (extractive, single step)✗ (citations optional)✓
(5) A separate reader sees only the selection✗✗✗✓✓✓
(6) Multi-agent coordination, by merging the artifacts of parallel agents✗✗✗✗✗✓ (Notebook Merging)

What each difference buys, measured on the same harness. Other designs store what the model wrote about the evidence. Code2Evidence also stores where the evidence is. Setting: MHQA, GPT-5.4-mini, same tools, turns (max 30 turns), and questions. The memory alternatives implement each design's core policy. Differences are paired.

DesignAccuracyTokens per questionEntries supported by their source
History only (no memory)48.3348.3380.2K80.2\text{K}Not Applicable
Model-written summary notes, editable (MemGPT/A-Mem-style; answer from the notes)47.3347.3373.5K73.5\text{K}Not Applicable
Editable notes plus archived sources (FS-Researcher-style)51.3351.33117.1K117.1\text{K}59.759.7\\% of notes
Code2Evidence, single-agent52.33\mathbf{52.33}69.6K\mathbf{69.6\text{K}}\mathbf{91.8\\%} of entries
Code2Evidence, 3-agents (15 turns max)54.00\mathbf{54.00}159.3K\mathbf{159.3\text{K}}\mathbf{96.1\\%} of entries

Against these alternatives, Code2Evidence is:

  • Effective. Our framework is numerically ahead of model-written summary notes (+5.00+5.00) and of history (+4.00+4.00);
  • Verifiable. 91.891.8\\% of entries are judged supported by their cited source in the single-agent setting and 96.196.1\\% in the multi-agent setting, against 59.759.7\\% of FS-Researcher-style notes. Model-written memory carries the writer's errors forward as facts. Our quoted span lets the answerer check every entry against the raw output.
  • Composable. Because entries are self-contained and sourced, parallel searchers can hand over notebooks instead of trajectories. With three searchers, the notebook hand-off is at least as accurate as the clipped trajectories (GPT-5.4-mini +3.67+3.67), with about a tenth of the answerer's input (0.3K0.3\text{K} vs. 2.9K2.9\text{K} tokens). The reconciled notebook can be audited:
    • Only 3.03.0\\% (GPT-5.4-mini) and 0.90.9\\% (GPT-5.5) of reconciled entries are judged unsupported;
    • the gold answer is kept in 95.895.8\\% and 95.095.0\\% of cases where any searcher found it;
    • a planted contradicting false entry is kept in only 1.31.3\\% and 0.70.7\\% of cases.
AuthorsOpenReview ↗

Response to Reviewer B7wG: Part 2

W1. Limited Novelty (Continued)

On the novelty of the multi-agent design. We agree that combining the outputs of parallel agents is common, and we do not claim the merging step itself as a contribution. Our claim concerns what the agents exchange. Existing multi-agent designs exchange one of three things: final answers (combined by voting, selection or debate), transcripts (concatenated or shared), or model-written summaries passed from sub-agents to a lead agent. Code2Evidence exchanges a fourth kind of object: self-contained evidence entries, each bound to where it came from. Such entries can be unioned, deduplicated, and checked one by one, which answers, transcripts, and summaries cannot. This choice is measurable, and it matters. We provide additional experiments below.

AuthorsOpenReview ↗

Response to Reviewer B7wG: Part 3

2. Related baselines

2a. Search and retrieval baselines

The submission compared against the following on FinQA:

  • Search-R1, R1-Searcher and ASearcher (released 7B checkpoints);
  • BM25 and Qwen3-4B-Embedding retrieval (GLM-5.3).

We have added the following on MHQA (GPT-5.4-mini; accuracy %, max 30 turns):

SystemAccuracy
BM25 top-10 + answerer42.00
BM25 iterative (≤ 4 follow-up queries)49.67
Long context: full history, no compression45.67
History-only agent48.33
DCI-Agent-Lite48.33
Code2Evidence, single agent52.33

At DCI-Agent-Lite's token budget, Code2Evidence's single agent is ahead on MHQA, using 69.6K tokens per question (search plus answer) against DCI's 68.7K. A cheap iterative BM25 baseline comes within 2 points at about 1/8 of the tokens. We will report all systems as accuracy–cost curves.

2b. Memory baselines

We reported the comparison in part 1, with the memory-related baseline frameworks.

2c. Multi-agent baselines

Parallel baseline. DCI×3 runs three independent DCI-Agent-Lite agents and combines them. All tokens are counted, including the combination step. GPT-5.4-mini, max 15 turns for all searchers:

Comparison (tokens/question)Difference
MHQA: C2E 11 searcher, 48.33%48.33\% (59.4K59.4\text{K}) vs. DCI, 48.33%48.33\% (68.7K68.7\text{K})+0.00+0.00
MHQA: C2E 33 searchers, 54.00%54.00\% (159.3K159.3\text{K}) vs. DCI×3\text{DCI}\times 3, 52.00%52.00\% (212K212\text{K})+2.00+2.00
FinQA: C2E 11 searcher, 49.11%49.11\% (44K44\text{K}) vs. DCI, 49.11%49.11\% (41K41\text{K})+0.00+0.00
FinQA: C2E 33 searchers, 57.11%57.11\% (147K147\text{K}) vs. DCI×3\text{DCI}\times 3, 52.44%52.44\% (134K134\text{K})+4.67+4.67
LegalRagBench: C2E 11 searcher, 66%66\% (45.0K45.0\text{K}) vs. DCI, 65%65\% (42.8K42.8\text{K})+1.00+1.00
LegalRagBench: C2E 33 searchers, 72%72\% (177.3K177.3\text{K}) vs. DCI×3\text{DCI}\times 3, 69%69\% (135.4K135.4\text{K})+3.00+3.00

At roughly matched compute, Code2Evidence is better than DCI in terms of accuracy. Note that our C2E framework actually performs better (52.33 vs. 48.33) under the 30-turn scenario than the reported 15-turn scenario here.

Cooperation designs on common multi-agent reconciling approches. We held the searches fixed and only changed the hand-off to the same answerer.

Hand-off (the design it stands for)Accuracy
Majority vote over answers (self-consistency / ensemble voting)82.33
Full trajectories (shared transcript, no budget)82.67
Notebook Simple Concatenation79.67
Code2Evidence, Reconciler85.33

The hand-off comparison shows that our notebook-based reconciliation beat common multi-agent reconciling approaches.

Merging and sharing. We audited the reconciler on MHQA with three searchers, n = 300:

Checkmini5.5
Reconciled entries the judge finds unsupported by their cited source3.0%0.9%
Gold answer kept when any searcher found it95.8%95.0%
Planted contradicting false entry kept1.3%0.7%
Planted irrelevant entry kept0.7%0.0%

We also added a shared-notebook variant, in which searchers see each other's banked evidence during search. It used 163.5K tokens per question versus 186.4K for independent searchers. Its duplicate banked entries were 42.9% versus 47.9%. We will include the results in our revised paper.

3. Smaller open-weight models

We agree this was a gap. We have now evaluated Code2Evidence with Qwen3-8B on MHQA and FinQA, with the same harness, the same questions, and DCI-Agent-Lite on the same model and endpoint:

System (Qwen3-8B)MHQAFinQA
History-only agent16.6714.00
DCI-Agent-Lite18.0017.33
DCI×3\text{DCI}\times 324.6721.33
Code2Evidence, single agent27.3318.00
Code2Evidence, 3 searchers37.0022.67

The evidence state helps a much smaller and weaker model with limited reasoning ability, especially in long-trajectory scenarios. The notebook hand-off also matters a lot: on the same three searches, handing the answerer the searchers' notebooks instead of their trajectories also increases accuracy.

Thank you for the detailed rebuttal and the additional experiments.

The new comparisons with memory-based methods, multi-agent baselines, and Qwen3-8B directly address my original concerns. I particularly appreciate the controlled comparisons of different multi-agent hand-off strategies, which provide useful evidence for the effectiveness of notebook-based evidence organization and reconciliation.

I still consider the methodological novelty somewhat limited, given its conceptual similarity to existing memory-based agent frameworks. Furthermore, the improvement over the closest memory-based baseline is relatively modest, and the stricter controlled experiments indicate smaller gains from the evidence state alone than the original results suggested.

Nevertheless, the additional experiments have strengthened the empirical support for the proposed framework and addressed most of my concerns. I am therefore inclined to raise my rating from 2 to 3 (Weak Acceptance), while maintaining some reservations about the methodological novelty and the attribution of performance gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.