- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ClaimWeaver, a query-conditioned evidence organization framework for multi-hop retrieval-augmented generation (RAG). It decomposes retrieved passages into source-attributed atomic claims, filters irrelevant claims, merges redundant ones, and connects complementary claims through directed bridges. The resulting structure is maintained across retrieval rounds to guide subsequent searches and answer generation. The framework is trained using GRPO with rewards for answer correctness, claim grounding, and format compliance. Experiments on seven QA benchmarks demonstrate improvements over existing RAG methods. ClaimWeaver achieves 50.5 EM and 56.4 F1, improving average EM by 4.5 points over a configuration-matched free-form-note baseline on four multi-hop benchmarks while maintaining comparable single-hop performance.
Critical strengths
-
The proposed framework provides an intuitive way to organize evidence for multi-hop reasoning. It explicitly separates evidence filtering, merging, and chaining, making the organization process easier to inspect and analyze.
-
The paper provides comprehensive experiments with consistent improvements on multi-hop QA benchmarks. ClaimWeaver improves average EM by 4.5 points over EviNote-RAG across four multi-hop datasets. The ablation studies and human evaluations further support the effectiveness of the proposed components, particularly directed chaining.
Critical weaknesses
-
The methodological novelty of the proposed framework appears limited. Existing studies have already explored atomic proposition extraction, query-aware evidence organization, and multi-hop reasoning paths [1, 2, 3]. ClaimWeaver mainly integrates these techniques with reinforcement learning, but its contributions beyond existing approaches are not sufficiently clear.
-
The performance improvements come with substantial additional inference costs. On HotpotQA, ClaimWeaver requires 77.8% more tokens and 85.7% higher latency than EviNote-RAG for a 5.0-point EM improvement. Although the authors extend the baseline's retrieval horizon, this does not fully match the reasoning computation. A more rigorous compute-matched comparison is needed to establish the accuracy-efficiency trade-off.
-
The comparisons with existing methods are insufficient to establish the contribution of the proposed framework. The paper does not directly compare against closely related proposition-based methods such as BRIEF [1], KiRAG [2], and PropRAG [3]. The graph-inspired local controls are not equivalent to these complete systems. In addition, the comparison with EviNote-RAG [4] changes both the evidence representation and reward objective. A same-reward ablation would help isolate the contribution of structured evidence organization.
References
[1] BRIEF: Bridging Retrieval and Inference for Multi-hop Reasoning via Compression. NAACL 2025.
[2] KiRAG: Knowledge-Driven Iterative Retriever for Enhancing Retrieval-Augmented Generation. ACL 2025.
[3] PropRAG: Guiding Retrieval with Beam Search over Proposition Paths. EMNLP 2025.
[4] EviNote-RAG: Enhancing RAG Models via Answer-Supportive Evidence Notes. arXiv 2025.
Questions for authors
-
What are the main methodological contributions of ClaimWeaver compared with existing proposition-based evidence organization and reasoning path construction methods [1, 2, 3]?
-
Could the authors provide a more rigorous compute-matched comparison to demonstrate whether the performance improvements justify the additional inference costs?
-
Could the authors include comparisons with more closely related baselines and a same-reward ablation to isolate the contribution of the proposed evidence representation?
Response: Contribution, direct baselines, and reward/cost controls
Thank you for highlighting the need to distinguish ClaimWeaver from proposition-based methods and to separate the organization interface from training objectives and computation.
Methodological contribution. BRIEF develops query-aware compression into summaries using atomic propositions to construct synthetic training data. KiRAG uses knowledge triples in iterative retrieval to identify and fill information gaps. PropRAG performs online beam search over context-rich proposition paths.
ClaimWeaver learns filtering, consolidation, and directed linking as explicit decisions within a persistent evidence state. Source attribution and usage information carry across rounds to guide both subsequent search and answering. The composite reward connects these decisions to answer correctness and decomposition-stage grounding without gold organization labels. The contribution is this learned organization interface and its evaluation; the individual ideas of propositions and paths have established precedents.
Complete-system comparisons. The original local graph controls isolate selected mechanisms. We additionally compare complete implementations of BRIEF, KiRAG, and PropRAG on the same evaluation questions and Wikipedia snapshot, using the Qwen2.5-7B backbone family for answering. Each retains its method-specific retrieval or compression machinery and training recipe. These are system-level comparisons; the controlled experiment below separately addresses attribution.
We use 500 questions per dataset except Bamboogle (125), and average three controlled runs; Avg. weights datasets equally. PropRAG is the closest baseline, with a macro-average gap of 0.65 points and per-dataset gaps of 1.4, 0.2, 0.2, and 0.8. These are modest descriptive differences. PropRAG also has lower mean online HotpotQA latency: 5.0 versus 7.8 seconds. Offline corpus construction is outside these online measurements.
Same-reward comparison. We train free-form and structured interfaces with the same decomposition protocol, source attribution, backbone initialization, data, schedule, paired seeds, retriever, decoding settings, and inference caps. The common format reward checks parseability and required fields without graph-specific bonuses. Both conditions use identical reward definitions and coefficients: answer 1.0, grounding 0.3 when enabled, and format 0.1. Grounding evaluates decomposed claims in both interfaces.
Structured (AGF) is the ClaimWeaver control with the shared answer, grounding, and format objective. These controls use three paired training seeds and are reported separately from the original model. The combined organization and usage-note interface gains 2.1 and 2.2 points under the two objectives. The original 4.5-point comparison with native EviNote-RAG measures the complete pipelines, including their different reward objectives.
Accuracy-cost trade-off. Under common hardware, precision, decoding, and batch size one, a cumulative 1,536-token cap gives 47.6 HotpotQA EM for the common-reward free-form control and 46.8 for Structured. At approximately 6.6 seconds and similar actual generated-token counts (2,760 versus 2,740), Structured reaches 52.2 EM versus 51.4. At a higher budget, Structured reaches 53.8 EM at 7.8 seconds. Latency includes online retrieval and organization, and every question is scored, including capped runs.
The five-turn native EviNote-RAG diagnostic reaches 50.2 EM with 3,100 generated tokens and 7.4 seconds. This tests allocating more search to the original baseline; the common-reward comparison separately controls the objective. The results characterize the tested accuracy-cost trade-off: organization helps when sufficient budget is available, while free-form notes remain competitive at low budgets.