- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
OriginRAG induces evidence demands from a multi-hop question and runs graph retrieval separately for each demand. A passage keeps the demand IDs and ranks by which it was retrieved. An LLM selector uses these tags to choose five complementary passages, after which another call extracts demand-linked verbatim spans for the QA reader. The evaluation uses 1,000 questions each from HotpotQA, 2WikiMultiHopQA, and MuSiQue, reporting retrieval and answer metrics, component ablations, provenance perturbations, a sampled span audit, and token usage.
Critical strengths
-
The use of per-demand retrieval tags is a sensible way to preserve information about why each passage was retrieved. Figure 1 gives a clear example where whole-question ranking misses a second-hop passage, while demand-specific retrieval can recover it.
-
Table 5 provides a useful controlled test of the provenance information. With the candidate pool fixed, R@5 is 89.65 with the correct provenance, 86.15 without provenance, 86.05 with equal-length neutral text, and 80.45 with shuffled provenance. This suggests that the selector benefits from the actual demand–passage association rather than simply from receiving a longer prompt.
Critical weaknesses
-
I could not clearly isolate the effect of lineage completion from the reported experiments. In Table 2, “w/o Lineage Completion” averages 88.15 R@5 / 54.60 EM / 66.48 F1, which matches the results in Table 3, while its EM/F1 also match the ungrounded reader in Table 7. Section 5.3 appears to say that removing lineage completion can change the retrieved passages, whereas Section 5.4 describes Table 7 as a fixed-passage comparison. However, Table 7 does not report passage IDs or R@5. If both Table 7 rows use the passages selected by the full OriginRAG system, I would expect them to share the full system's 89.65 R@5. The current presentation makes it difficult to tell whether the reported +4.47 EM / +4.30 F1 improvement comes from the extracted spans themselves or partly from changes in the selected evidence.
-
I also think the paper overstates what is guaranteed by the lineage validation step. Equation (9) checks containment, schema consistency, and retrieval provenance, but it does not verify that the extracted span actually supports the associated demand. This distinction matters because the manual audit in Table 6 finds full support for 420 of 500 spans, while 50 are only partially supportive and 30 are unsupported or linked to the wrong demand.
-
The headline improvements should also be interpreted more carefully. The abstract reports gains against the average of four baselines, while the gap to the strongest baseline, G-reasoner, is much smaller: 1.12 R@5, 3.57 EM, and 2.91 F1 on average. No paired uncertainty is reported for these differences. Taken together, I think the evidence supports the usefulness of provenance-aware retrieval, but is less conclusive about the semantic grounding provided by the extracted spans and the overall size of the improvement.
Questions for authors
-
Could you clarify exactly which passage IDs are given to the reader in the relevant rows of Tables 2, 3, and 7? Please also report the R@5 for both rows in Table 7. Most importantly, what is the QA difference when the selected passage set is fixed to the full OriginRAG output and only the presence or absence of demand-linked spans is changed?
-
Could you provide per-query supporting-passage hits and answer predictions for the Table 2 ablations, together with paired uncertainty estimates? I would also appreciate an explanation for why the reported R@5 and F1 drops are identical across the three datasets.
-
On exactly the same selected passage sets, how does the proposed demand-linked extraction compare with a generic span extractor that does not receive demand IDs? It would also be useful to report a human audit focused specifically on whether the retained spans semantically support their associated demands.