acceptodds
Under review as a conference paper at ICLR 2027

ClaimWeaver: Query-Conditioned Evidence Organization for Multi-Hop Retrieval-Augmented Generation

Abstract

Retrieval-augmented generation (RAG) can retrieve relevant evidence yet fail to connect the facts needed to answer a question. This gap is acute in multi-hop question answering, where passages mix answer-bearing facts with duplicates and distractors, while the required dependencies span multiple documents. Free-form notes condense this context, but decisions about which facts to discard, combine, or connect remain implicit in prose. We introduce ClaimWeaver, a query-conditioned evidence organization framework over atomic claims. It transforms retrieved passages into source-attributed claims and builds a compact structure in which irrelevant claims are suppressed, redundant claims are consolidated, and complementary claims are linked by directed bridges. This structure persists across retrieval rounds, focuses subsequent search, and provides answer generation with an explicit evidence path. A composite reinforcement-learning objective learns this behavior without gold organization labels. Across seven QA benchmarks, ClaimWeaver achieves 50.5 EM and 56.4 F1. Compared with a configuration-matched free-form-note control, it improves average EM by 4.5 points across four multi-hop benchmarks while maintaining average single-hop EM. Human audits assess decomposed-claim faithfulness and pairwise relations, while retrained ablations and fixed-evidence interventions examine the role of organization. These results support query-conditioned claim organization as an effective interface between retrieval and multi-hop reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
retrieval-augmented generation, multi-hop question answering, evidence organization, atomic claims, agentic RAG, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y5dWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ClaimWeaver, a query-conditioned evidence organization framework for multi-hop retrieval-augmented generation (RAG). It decomposes retrieved passages into source-attributed atomic claims, filters irrelevant claims, merges redundant ones, and connects complementary claims through directed bridges. The resulting structure is maintained across retrieval rounds to guide subsequent searches and answer generation. The framework is trained using GRPO with rewards for answer correctness, claim grounding, and format compliance. Experiments on seven QA benchmarks demonstrate improvements over existing RAG methods. ClaimWeaver achieves 50.5 EM and 56.4 F1, improving average EM by 4.5 points over a configuration-matched free-form-note baseline on four multi-hop benchmarks while maintaining comparable single-hop performance.

Critical strengths
  1. The proposed framework provides an intuitive way to organize evidence for multi-hop reasoning. It explicitly separates evidence filtering, merging, and chaining, making the organization process easier to inspect and analyze.

  2. The paper provides comprehensive experiments with consistent improvements on multi-hop QA benchmarks. ClaimWeaver improves average EM by 4.5 points over EviNote-RAG across four multi-hop datasets. The ablation studies and human evaluations further support the effectiveness of the proposed components, particularly directed chaining.

Critical weaknesses
  1. The methodological novelty of the proposed framework appears limited. Existing studies have already explored atomic proposition extraction, query-aware evidence organization, and multi-hop reasoning paths [1, 2, 3]. ClaimWeaver mainly integrates these techniques with reinforcement learning, but its contributions beyond existing approaches are not sufficiently clear.

  2. The performance improvements come with substantial additional inference costs. On HotpotQA, ClaimWeaver requires 77.8% more tokens and 85.7% higher latency than EviNote-RAG for a 5.0-point EM improvement. Although the authors extend the baseline's retrieval horizon, this does not fully match the reasoning computation. A more rigorous compute-matched comparison is needed to establish the accuracy-efficiency trade-off.

  3. The comparisons with existing methods are insufficient to establish the contribution of the proposed framework. The paper does not directly compare against closely related proposition-based methods such as BRIEF [1], KiRAG [2], and PropRAG [3]. The graph-inspired local controls are not equivalent to these complete systems. In addition, the comparison with EviNote-RAG [4] changes both the evidence representation and reward objective. A same-reward ablation would help isolate the contribution of structured evidence organization.

References

[1] BRIEF: Bridging Retrieval and Inference for Multi-hop Reasoning via Compression. NAACL 2025.

[2] KiRAG: Knowledge-Driven Iterative Retriever for Enhancing Retrieval-Augmented Generation. ACL 2025.

[3] PropRAG: Guiding Retrieval with Beam Search over Proposition Paths. EMNLP 2025.

[4] EviNote-RAG: Enhancing RAG Models via Answer-Supportive Evidence Notes. arXiv 2025.

Questions for authors
  1. What are the main methodological contributions of ClaimWeaver compared with existing proposition-based evidence organization and reasoning path construction methods [1, 2, 3]?

  2. Could the authors provide a more rigorous compute-matched comparison to demonstrate whether the performance improvements justify the additional inference costs?

  3. Could the authors include comparisons with more closely related baselines and a same-reward ablation to isolate the contribution of the proposed evidence representation?

1 comment
AuthorsOpenReview ↗

Response: Contribution, direct baselines, and reward/cost controls

Thank you for highlighting the need to distinguish ClaimWeaver from proposition-based methods and to separate the organization interface from training objectives and computation.

Methodological contribution. BRIEF develops query-aware compression into summaries using atomic propositions to construct synthetic training data. KiRAG uses knowledge triples in iterative retrieval to identify and fill information gaps. PropRAG performs online beam search over context-rich proposition paths.

ClaimWeaver learns filtering, consolidation, and directed linking as explicit decisions within a persistent evidence state. Source attribution and usage information carry across rounds to guide both subsequent search and answering. The composite reward connects these decisions to answer correctness and decomposition-stage grounding without gold organization labels. The contribution is this learned organization interface and its evaluation; the individual ideas of propositions and paths have established precedents.

Complete-system comparisons. The original local graph controls isolate selected mechanisms. We additionally compare complete implementations of BRIEF, KiRAG, and PropRAG on the same evaluation questions and Wikipedia snapshot, using the Qwen2.5-7B backbone family for answering. Each retains its method-specific retrieval or compression machinery and training recipe. These are system-level comparisons; the controlled experiment below separately addresses attribution.

SystemHotpotQA2WikiMuSiQueBamboogleAvg.
BRIEF50.050.825.842.442.25
KiRAG51.452.227.044.043.65
PropRAG52.454.027.845.644.95
Structured (AGF)53.854.228.046.445.60

We use 500 questions per dataset except Bamboogle (125), and average three controlled runs; Avg. weights datasets equally. PropRAG is the closest baseline, with a macro-average gap of 0.65 points and per-dataset gaps of 1.4, 0.2, 0.2, and 0.8. These are modest descriptive differences. PropRAG also has lower mean online HotpotQA latency: 5.0 versus 7.8 seconds. Offline corpus construction is outside these online measurements.

Same-reward comparison. We train free-form and structured interfaces with the same decomposition protocol, source attribution, backbone initialization, data, schedule, paired seeds, retriever, decoding settings, and inference caps. The common format reward checks parseability and required fields without graph-specific bonuses. Both conditions use identical reward definitions and coefficients: answer 1.0, grounding 0.3 when enabled, and format 0.1. Grounding evaluates decomposed claims in both interfaces.

ObjectiveFree-form multi-hop EMStructured multi-hop EM
Answer + format42.144.2
Answer + grounding + format43.445.6

Structured (AGF) is the ClaimWeaver control with the shared answer, grounding, and format objective. These controls use three paired training seeds and are reported separately from the original model. The combined organization and usage-note interface gains 2.1 and 2.2 points under the two objectives. The original 4.5-point comparison with native EviNote-RAG measures the complete pipelines, including their different reward objectives.

Accuracy-cost trade-off. Under common hardware, precision, decoding, and batch size one, a cumulative 1,536-token cap gives 47.6 HotpotQA EM for the common-reward free-form control and 46.8 for Structured. At approximately 6.6 seconds and similar actual generated-token counts (2,760 versus 2,740), Structured reaches 52.2 EM versus 51.4. At a higher budget, Structured reaches 53.8 EM at 7.8 seconds. Latency includes online retrieval and organization, and every question is scored, including capped runs.

The five-turn native EviNote-RAG diagnostic reaches 50.2 EM with 3,100 generated tokens and 7.4 seconds. This tests allocating more search to the original baseline; the common-reward comparison separately controls the objective. The results characterize the tested accuracy-cost trade-off: organization helps when sufficient budget is available, while free-form notes remain competitive at low budgets.

Reviewer_eQ5HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ClaimWeaver, a query-conditioned evidence-organization layer for multi-hop retrieval-augmented generation (RAG). It decomposes retrieved passages into source-attributed atomic claims, filters irrelevant claims, merges overlapping claims, links complementary claims with directed bridges, and maintains this structured memory across retrieval rounds to guide subsequent search and answer generation. The policy is trained with a composite reinforcement-learning objective combining final-answer exact match, an NLI-based claim-grounding reward, and format compliance, without requiring gold organization labels. The evaluation covers four multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle) and three single-hop benchmarks (Natural Questions, TriviaQA, and PopQA). The paper reports 50.5 average EM and 56.4 F1 across the seven benchmarks. Compared with a configuration-matched free-form-note control, EviNote-RAG, ClaimWeaver improves average EM by 4.5 points on the four multi-hop datasets while maintaining average single-hop EM. Additional analyses include retrained operation ablations, human audits of claim faithfulness and pairwise relations, fixed-evidence interventions, transfer experiments, failure localization, and an accuracy–cost analysis.

Critical strengths
  1. A well-motivated and inspectable evidence representation, supported by targeted analyses.

ClaimWeaver makes relevance filtering, redundancy merging, and directed bridging explicit rather than leaving all evidence-selection decisions implicit in free-form notes. It preserves source attribution and carries the organization state across retrieval rounds. The retrained ablations show that removing chaining causes the largest individual loss (−4.8 multi-hop EM), while fixed-evidence interventions show substantial drops when relations are shuffled (−19 EM) or bridges are reversed (−15 EM). These analyses provide evidence that the intermediate structure is behaviorally consequential rather than merely a different textual format.

  1. Broad empirical validation with consistent multi-hop gains.

The method is evaluated on seven QA benchmarks, with three-seed results and paired bootstrap comparisons against the configuration-matched EviNote-RAG control. The reported average EM gain is 4.5 points across the four multi-hop datasets, with per-dataset gains of 4.0–5.0 points, while average single-hop EM is maintained. The paper further reports claim-faithfulness audits, relation-quality audits, failure analysis, a smaller-scale experiment, and transfer evaluations. This breadth makes the empirical case stronger than a single-benchmark demonstration.

Critical weaknesses
  1. The main end-to-end comparison does not fully isolate the contribution of structured organization from the training objective and computation.

EviNote-RAG is a useful configuration-matched control, but the paper explicitly states that it retains its native reward while ClaimWeaver uses answer, claim-grounding, and format rewards; their realized search traces and computation also differ. The retrained operation ablations and fixed-evidence interventions partially address this concern and are valuable, so this is not an absence of evidence. However, the headline 4.5-point multi-hop EM gain cannot be interpreted as a representation-only effect. The cost comparison also matters in practice: on HotpotQA, ClaimWeaver uses 3,200 trajectory tokens and 7.8 seconds versus 1,800 tokens and 4.2 seconds for EviNote-RAG, for a 5-point EM gain. A clearer reward-matched and compute-/turn-budget-matched comparison would make the contribution easier to assess.

  1. The evaluation of evidence faithfulness and coverage stops short of validating the complete final evidence path.

The human claim-faithfulness audit evaluates decomposed claims against their cited source passages; the paper notes that merged claims, generated notes, and final answers require separate evaluation. The relation audit reports 82.0% precision and 72.8% recall for bridging, based on 50 precision predictions and a separate set of 320 candidate pairs. These are useful diagnostics, but the bridge audit is relatively small and identifies nontrivial direction and intermediate-step errors. In addition, the NLI grounding reward averages entailment over decomposed claims and can in principle favor omitting difficult, low-entailment claims; the paper acknowledges selective omission as a possible failure mode, while the reported claim-count analysis does not directly measure proposition recall. These gaps limit confidence that the learned structure reliably preserves complete, faithful multi-hop evidence paths beyond the evaluated English factoid QA setting.

Questions for authors
  1. Attribution and matched comparison.

Could the authors clarify whether the “−Organization” and other retrained ablations use the same answer, grounding, and format reward weights, training seeds, and inference limits as the full model? More broadly, can the authors provide a comparison that holds the reward objective and search/compute budget fixed while varying only structured organization versus free-form notes? This would help determine how much of the gain comes from the representation itself rather than the combined training and inference pipeline.

  1. End-to-end evidence-path quality and coverage.

The current human faithfulness audit focuses on decomposed claims, while bridge precision/recall is estimated from relatively small and separate samples. Do the authors have annotations or analyses of the final merged memory and complete bridge paths, including whether all required intermediate facts are preserved and correctly ordered? Can the authors quantify claim/proposition recall or otherwise rule out a reward incentive to omit difficult but necessary evidence?

  1. Accuracy–cost trade-off.

On HotpotQA, ClaimWeaver improves EM by 5 points over EviNote-RAG but increases trajectory tokens by 77.8% and latency by 85.7%. Can the authors report a clearer accuracy–latency/token Pareto comparison under matched retrieval-turn budgets, including the five-turn EviNote-RAG diagnostic where possible? This would clarify the practical operating regime in which the proposed organization layer is preferable.

2 comments
AuthorsOpenReview ↗

Response 1/2: Common-reward controls and accuracy-cost comparisons

Thank you for distinguishing the existing evidence from the remaining attribution and coverage questions. We address attribution and cost here, and final-path quality in Part 2.

Ablation settings and common rewards. The original organization and usage-note ablations retain the reward coefficients, training schedule, paired seeds, and inference limits; their format validators accept the corresponding ablated schemas. We therefore add separate controls using a common format reward that checks parseability and required fields without graph-specific bonuses.

Both interfaces share the decomposition protocol, source attribution, backbone initialization, data, training schedule, three paired seeds, retriever, decoding settings, and inference caps. Free-form produces an unstructured note; Structured filters, merges, links, and annotates claims. Under AGF, both receive the same decomposition-stage grounding reward. AF denotes answer plus format rewards; AGF adds grounding. The coefficients are answer 1.0, grounding 0.3, and format 0.1.

InterfaceObjectiveHotpotQA2WikiMuSiQueBamboogleAvg.
Free-formAF49.850.425.642.442.1
StructuredAF52.252.627.044.844.2
Free-formAGF51.452.026.244.043.4
StructuredAGF53.854.228.046.445.6

Results average three paired training seeds over 500 questions per dataset, except Bamboogle (125); Avg. weights datasets equally. The reader-input cap is 4,096 tokens per call, separate from the cumulative generated-token caps below. These new controlled configurations are reported separately from the original model. The structured interface gains 2.1 points under AF and 2.2 under AGF. This contrast measures the combined organization and usage-note interface. Adaptive retrieval remains active, so subsequent search changes are part of the effect. The original 4.5-point gain over native EviNote-RAG describes the complete pipelines.

Fixed-evidence replay. Holding passages and candidate claims fixed on the 500 HotpotQA questions, disabling further retrieval, and applying a 2,560-token generation cap gives 50.4 EM for Free-form and 52.0 for Structured under AGF. Mean generated tokens are 2,190 and 2,250. This comparison reduces variation in available evidence.

Measured accuracy and cost. We evaluate the same HotpotQA questions under common hardware, precision, decoding, and batch size one. Latency includes online retrieval and organization; offline indexing is separate. Every question is scored, including runs stopped by the cap. Tokens below are cumulative generated tokens per trajectory, not input tokens. Except for native EviNote-RAG, both controls use AGF.

SystemMax turnsToken capMean tokensMean secondsEM
EviNote-RAG3409618004.249.0
EviNote-RAG5409631007.450.2
Free-form3153613903.947.6
Structured3153614804.146.8
Free-form3256022305.750.6
Structured3256024006.051.2
Free-form3409627606.651.4
Structured3307227406.652.2
Structured3409632007.853.8
Structured5409636809.154.4

At the lowest common cap, free-form notes perform better. At approximately 6.6 seconds and similar generated-token counts, Structured achieves 52.2 EM versus 51.4, a descriptive 0.8-point advantage. Rounded mean latency provides an approximate operating-point comparison rather than exact compute equivalence. More computation raises Structured to 53.8 EM at 7.8 seconds; a five-turn limit adds 0.6 points at further cost. These measurements show both the useful operating range and the overhead of explicit organization.

AuthorsOpenReview ↗

Response 2/2: Final-path faithfulness and retention of necessary evidence

Audit design. The original claim-faithfulness evaluation concerns decomposition-stage entailment. We separately audit merged claims, note propositions, complete supporting paths, and answers from the new common-reward structured control and the native EviNote-RAG baseline. Two blinded annotators assess the outputs, with adjudication of disagreements.

For coverage, we sample 100 questions independently of answer correctness and replay a shared retrieved-passage pool. Necessary supporting facts are annotated before system outputs are inspected. The pool contains 211 necessary question-fact instances, and all required facts are available for 84 of the 100 questions. Retention is measured against the necessary facts actually available in that pool. This distinguishes organization losses from retrieval failures.

MetricStructured, common rewardEviNote-RAG
Supported merged claims137/150 (91.3%)Not applicable
Supported note propositions181/200 (90.5%)176/200 (88.0%)
Available necessary facts retained194/211 (91.9%)180/211 (85.3%)
Questions with complete valid paths71/100 (71.0%)61/100 (61.0%)
Valid paths, conditional on full retrieval71/84 (84.5%)61/84 (72.6%)
Answers with all factual content supported72/100 (72.0%)68/100 (68.0%)

The structured model is the new AGF control defined in Part 1. A complete valid path must preserve all required facts and valid dependency ordering. For EviNote-RAG, annotators examine the dependencies expressed in the note, so explicit graph formatting is not required. The merged-claim and note rows sample different units from the question-level path and answer rows. Answer support checks source entailment; EM separately checks correctness for the question. These diagnostic audits use one fixed checkpoint per system.

Merged evidence. Of the 13 unsupported merged claims, eight contain an incorrect fusion and five unsupported elaboration. An incorrect overlap edge can join a connected component whose claims should remain separate. Checking the merged statement against the union of cited sources detects unsupported fusion that source-ID preservation alone cannot prevent. The original bridge precision and recall concern individual relations sampled under separate protocols; the new path-level audit directly assesses whether the necessary facts and dependencies survive together.

Selective omission. With the same structured interface, removing the grounding term yields retention of 190/211 necessary facts (90.0%), compared with 194/211 (91.9%) when grounding is enabled. Thus the grounding objective is not accompanied by a loss of aggregate coverage in this sample. Individual omissions still occur: 17 available necessary fact instances are missing from the final state under AGF. This directly measures coverage beyond claim counts. A mean entailment reward can still favor omitting a below-average-scoring claim; the aggregate retention result does not eliminate that incentive.

The control has higher observed retention and complete-path quality in this audit, with remaining merge and omission errors. Its scope is the sampled factoid questions and their fixed retrieved evidence; these rates should not be extrapolated to unrestricted domains.

Reviewer_txTmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents CLAIMWEAVER, a GRPO-trained agentic RAG framework that decomposes retrieved passages into source-attributed atomic claims, then filters, merges (overlap) and links (directed bridges) them into a persistent claim structure across retrieval rounds, with per-claim usage notes. The reward combines answer EM, NLI-based claim entailment and format compliance. On seven QA benchmarks it improves average multi-hop EM by 4.5 over a configuration-matched EviNote-RAG while keeping single-hop EM unchanged. Human audits, retrained ablations, fixed-evidence interventions, transfer tests and a cost analysis are provided.

Critical strengths
  1. Well-motivated problem. Decomposing "retrieved but not connected" into filtering, redundancy and dependency decisions is more inspectable than free-form notes and is practically useful.
  2. The error analysis is informative: direction-reversed bridges are a recurring failure and retrieval accounts for 41% of errors, giving concrete directions for future work.
Critical weaknesses
  1. The ablations raise their own concerns. Without organization (44.0 EM; multi-hop 36.5) or without notes (multi-hop ≈40.8), the model falls below EviNote-RAG's multi-hop 41.2, so the claim-decomposition interface alone is no better than free-form notes.
  2. Fixed-evidence interventions (shuffling, reversing) create out-of-distribution, self-contradictory inputs that would hurt any input-dependent system. They show the model reads the structure, not that the learned structure is optimal. No analogous perturbation baseline exists for free-form notes.
  3. "Query-conditioning" is never tested directly (same passages, different queries, check for role changes).
  4. R_grounding checks only decomposition-stage entailment against a single passage. Merged claims, notes and answers are unverified, despite the emphasis on provenance. Merging uses connected components of overlap edges, so noise can propagate into wrong fusions.
  5. The method is essentially an RL-trained structured chain-of-thought with claim decomposition, in the same line as EviNote-RAG, BRIEF, PropRAG and Chain-of-Note; the novelty is mainly an engineered combination. Only two relation types are modeled, and temporal, causal and conflicting evidence are not handled.
Questions for authors
  1. Exactly how are −Filtering/−Merging/−Chaining implemented.
  2. What drop do analogous perturbations of EviNote-RAG's notes cause?
  3. Can you directly test query-conditioning: same passages, different questions, and do claim roles and selections change accordingly?
  4. What are the context-token budgets of the graph-inspired controls, and why do training-free controls outperform Search-R1/R1-Searcher?
  5. How faithful are merged claims, notes and final answers (human or NLI)? Can you evaluate atomicity and self-containedness, and check whether R_grounding induces near-verbatim copying?
  6. Does the gain persist with stronger retrievers (rerankers, more passages), larger backbones (14B/32B) or more retrieval turns?
5 comments
AuthorsOpenReview ↗

Response 1/3: Contribution, ablation implementation, and matched rewards

Thank you for recognizing the motivation and inspectability of explicit evidence organization. We address the component ablations here, the intervention and query-conditioning tests in Part 2, and evidence quality and robustness in Part 3.

Contribution and interpretation of the ablations. The original multi-hop EM scores are 45.7 for the full model, 36.5 without organization, and approximately 40.8 without usage notes, compared with 41.2 for EviNote-RAG. These results locate the benefit in the combined organization-and-note mechanism. Decomposition alone is insufficient in this setting.

ClaimWeaver exposes relevance decisions, retained claim identities, consolidation, and bridge direction in a persistent state used by later retrieval and answering. Relative to free-form notes, these decisions can be inspected and intervened on explicitly. Our contribution is learning this organization interface without gold labels; propositions, query-aware compression, and paths have precedents.

Exact ablation operations. Each variant is retrained with the corresponding operation disabled during training rollouts and evaluation. The memory-update rule and valid output schema reflect that ablation. The variants are not obtained by deleting fields only at test time.

  • Without filtering: the relevance-based exclusion step is bypassed. Every claim entering organization passes to merging and chaining; this stage cannot suppress a claim as irrelevant. Subsequent merging, directed linking, usage-note generation, and the common context limit remain active.
  • Without merging: retained claims keep separate identities, texts, and source references. Overlap-connected groups are not collapsed and no fused claim text is created. Chaining and usage notes operate on the original retained claims; filtering remains active.
  • Without chaining: the persistent state contains no explicit directed bridge edges. Filtering, merging, and usage notes remain active, and later retrieval and answering receive the resulting state without bridge links. Notes may still express implicit dependencies, so this tests the additional contribution of explicit links.
  • Without organization: all three operations are disabled while decomposition and notes remain enabled.
  • Without notes: the organized claims and relations remain, while the usage explanations are omitted.

The variants retain the same training data, schedule, paired seeds, reward coefficients, and inference limits. The grounding term is computed on decomposed claims, upstream of organization, in each variant. Format validation accepts the corresponding ablated schema. Thus budgets are held fixed, while realized evidence length and computation may change.

Common-reward comparison. We additionally train two controls with a shared decomposition protocol, source attribution, backbone initialization, training data, paired seeds, retriever, and inference caps. One produces free-form notes; the other organizes claims and attaches usage notes. The common format reward checks parseability and required fields, without graph-specific validity bonuses. Both conditions use identical reward definitions and coefficients. This free-form control is distinct from native EviNote-RAG.

ObjectiveFree-form notesStructured stateGain
Answer + format42.144.22.1
Answer + grounding + format43.445.62.2

Scores are unweighted multi-hop EM averages over four datasets, each evaluated with three paired training seeds. The answer, grounding, and format coefficients are 1.0, 0.3, and 0.1, respectively; grounding is omitted in the first row. When enabled, the grounding reward evaluates decomposed claims in both interfaces. These new controls use a common format objective and are reported separately from the original model.

The comparison evaluates the combined organization and usage-note interface under both objectives, including its effect on later retrieval. The original 4.5-point comparison with native EviNote-RAG measures the complete pipelines, including their different training objectives and realized computation.

Relation scope. Overlap consolidates redundant information; directed bridging connects facts through prerequisites. These are the operational relations evaluated for English factoid QA. Explicit temporal reasoning, causal inference, and conflict resolution require additional semantics and evaluation.

AuthorsOpenReview ↗

Response 2/3: Matched interventions and direct query-conditioning tests

Interpreting the original intervention. Reversing bridges reduced EM from 54 to 39 on the original fixed-evidence intervention set. This measures sensitivity to bridge direction. The retrained ablations and common-reward comparisons assess the benefit of learning the organization interface.

Analogous perturbations of free-form notes. We evaluate both systems on 100 HotpotQA questions with fixed source evidence. Annotators identify the same target fact or prerequisite relation in each system's state. We then remove that fact or corrupt the relation while preserving the remaining evidence. The prerequisite edit changes the relation or its arguments; the surface-order control preserves semantics and graph references.

Eligibility requires the edit target to be present in both states, independently of answer correctness. We report every eligible question. Results therefore characterize this diagnostic subset. The question set is separate from the coverage audit, and each system uses one fixed checkpoint.

InterventionClaimWeaver EMEviNote-RAG EM
Intact state5450
Surface reordering, semantics preserved5349
Remove a necessary intermediate fact4643
Corrupt a prerequisite relation4039
Remove an irrelevant fact5350

Removing necessary facts reduces EM by 8 and 7 points; corrupting prerequisites reduces it by 14 and 11 points. Both systems are sensitive to damaged evidence. The much smaller changes after surface reordering and irrelevant-fact removal distinguish semantic damage from changes that preserve the necessary information. We interpret these interventions as behavioral diagnostics, with the matched training comparisons providing separate evidence about the organization interface.

Same evidence, different questions. We construct 50 question pairs, each answerable from a shared passage set. Within each pair, both the passages and candidate claims remain fixed. Annotators identify the facts and roles required by each question before inspecting the model outputs. A query-agnostic control organizes the same claims without receiving the question; the answer generator receives the question in both conditions.

MetricQuery-conditionedQuery-agnostic
Required fact instances retained181/200 (90.5%)164/200 (82.0%)
Questions with complete valid paths79/100 (79.0%)69/100 (69.0%)
Answer EM61/100 (61.0%)56/100 (56.0%)

Among 76 annotated decisions requiring a change in selection or role, the query-conditioned model makes the correct adjustment in 63 cases (82.9%). Among 124 decisions that should remain stable, it changes 11 incorrectly (8.9%). These denominators count role/selection decisions; the retention row counts question-fact instances. The evaluation checks the appropriateness of changes as well as their occurrence. These single-checkpoint results concern a deliberately constructed diagnostic set; the main benchmark evaluation remains unchanged.

AuthorsOpenReview ↗

Response 3/3: Final evidence quality, graph budgets, and robustness

Faithfulness after decomposition. The original 94.0% claim-faithfulness result evaluates decomposition-stage entailment. We separately audit merged claims, notes, and answers from the new common-reward structured control (AGF), using two blinded annotators and adjudication. Of 150 sampled merged claims, 137 are supported by the union of their cited passages (91.3%). Eight of the 13 unsupported cases involve incorrect fusion, and five contain unsupported elaboration. Source attribution provides traceability; the audit checks whether the merged statement remains valid.

Coverage and complete paths. We sample 100 questions independently of answer correctness and replay the same retrieved passages for both systems. Necessary facts are annotated before outputs are inspected. The pool contains 211 necessary question-fact instances; all required facts are available for 84 questions. This separates losses during organization from missing retrieval evidence.

MetricStructured (AGF)EviNote-RAG
Supported note propositions181/200 (90.5%)176/200 (88.0%)
Available necessary facts retained194/211 (91.9%)180/211 (85.3%)
Questions with complete valid paths71/100 (71.0%)61/100 (61.0%)
Answers with all factual content supported72/100 (72.0%)68/100 (68.0%)

Conditional on all necessary facts being retrieved, valid-path rates are 71/84 (84.5%) and 61/84 (72.6%). A path must preserve all necessary facts and valid dependency ordering; for free-form notes, annotators assess the dependencies expressed in the text. Answer support measures source entailment, while EM measures correctness for the question. These rates describe the audited control under shared retrieved evidence.

Atomicity, self-containedness, and copying. In 200 sampled decomposed claims, 177 are atomic (88.5%) and 186 self-contained (93.0%). We define near-verbatim copying as a contiguous source span covering at least 80% of normalized claim tokens. On 200 claims per condition, copying occurs in 74 cases (37.0%) with grounding and 66 (33.0%) without it. Surface reuse is assessed separately from entailment and coverage. Necessary-fact retention is 194/211 with grounding versus 190/211 without it, so aggregate coverage does not decrease in this sample.

Graph-control budgets. Each local graph control receives up to 50 candidate passages, with serialized evidence capped at 3,072 tokens within a 4,096-token reader-input limit. This input limit excludes generated answer tokens. The common reader input and evidence caps also apply to the trained controls. Candidate pools, evidence-selection rules, and realized computation differ across graph and RL pipelines. Broader candidate access and different selection rules are plausible explanations for their ranking, but this comparison does not isolate either explanation or an effect of RL alone.

Stronger retrieval and larger backbones. We compare the common-reward interfaces on the same 500 HotpotQA questions, using three paired training seeds and matched inference caps within each pair.

ConfigurationFree-form EMStructured EM
7B, top-3 retrieval51.453.8
7B, top-5 retrieval53.054.6
7B, top-20 reranked to top-354.255.0
14B, top-3 retrieval55.857.0

Both systems use the same reranker in the reranking condition. Both interfaces are retrained at 14B. The advantage narrows to 0.8 points with reranking and 1.2 at 14B, showing smaller observed mean differences in these configurations. Extending the structured control from three to five retrieval rounds raises EM from 53.8 to 54.4, with mean latency increasing from 7.8 to 9.1 seconds. The backbone evaluation does not include 32B.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.