- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes context propagation, which is a method that enriches documents with info from other related documents before the retrieval process. The paper also introduces XRefQA, a benchmark for testing cross-doc reasoning. The results show strong improvements over naive RAG and some other methods.
Critical strengths
-
The motivation is great - making connections between docs explicitly before retrieval should be a useful idea.
-
The results are quite strong.
-
I appreciate the ablations that try to explain where the improvements come from.
Critical weaknesses
-
Limited evaluation on real-world data. XRefQA is synthetic and has only 50 test samples. I'd like to see whether the method works as well on real document collections, where the information can be incomplete, outdated and some are conflicting.
-
Missing comparison with closely related work. IndexRAG also builds cross-document knowledge during indexing. The paper mentioned it in the related work section but did not compare with it.
-
There's a risk of error propagation. The method uses annotations from earlier rounds to enrich other documents. If an annotation is wrong, the error may propagate to the entire collection.
-
AI-generated figures. The figures in this paper are obviously AI-generated and I hate AI-generated things. I mean, they are simple enough and the authors can totally remake them into a vector format by their hands using powerpoints or other software. Please do this if the paper is accepted.
Questions for authors
see weaknesses.
We thank the reviewer for the constructive feedback and for recognizing the motivation, experimental results, and diagnostic ablations. We address the concerns below, including new experiments comparing CP with IndexRAG and investigating error propagation across annotation rounds.
W1. Limited evaluation on real-world data, where the information can be incomplete, outdated and some are conflicting
We agree that our evaluation does not cover real-world collections with incomplete, outdated, or conflicting information. This work identifies implicit cross-document references as a challenge, introduces XRefQA to measure it, and proposes CP to address it. We further demonstrate improvements on human-written Wikipedia documents in MuSiQue, 2Wiki, and HotpotQA, providing evidence beyond the synthetic benchmark. Extending CP to heterogeneous, evolving collections is a natural next step that we leave for future work. We envision this direction as closely connected to our broader research on structured knowledge representations, such as navigable project wikis and persistent memory layers, where maintaining up-to-date information and resolving conflicts would be essential.
W2. IndexRAG comparison
We evaluated IndexRAG using its publicly released implementation on MuSiQue, 2Wiki, and HotpotQA (1,000 questions each), with the same retriever (NV-Embed-v2), preprocessing and QA model (GPT-4o-mini), and evaluation protocol, without reranking. CP outperforms IndexRAG on all three benchmarks, achieving 65.9 versus 60.2 average F1 (+5.7) with top-5 retrieval versus IndexRAG's top-10 setting. On MuSiQue, CP also uses fewer reader tokens (742 versus 1,291), with comparable preprocessing costs (14.69 dollars versus 13.52 dollars). These results support the effectiveness of attaching cross-document information to original passages rather than indexing extracted and bridging facts separately. We provide the full results table in our response to Reviewer 2wqP and have included the comparison in the revised manuscript.
W3. Risk of error propagation
We agree that incorrect annotations can propagate across rounds. To investigate this risk in a controlled setting, we injected 1 or 2 incorrect numerical claims per XRefQA collection (10 and 20 errors in total) into round-1 annotations and reran round 2.
We examined two outcomes: (i) whether the injected errors persisted in the injected documents, and (ii) whether they spread to other documents:
(i): After round 2, only 1 of the 10 injected errors and 2 of the 20 injected errors remained in their original documents; the others were corrected or removed.
(ii): To measure propagation, we examined the other documents in each experiment (231 and 221, respectively). During round-2 annotation, 106 and 167 of these documents received prompts containing an injected false value from another document. Of those, only 2/106 (1.9%) and 3/167 (1.8%) repeated the false value in their new annotations. Importantly, none of the two in the first experiment and only one of the three in the second adopted the false value as fact. The remaining ones either presented it alongside the correct value or explicitly flagged a conflict.
These results suggest limited error persistence and propagation in our controlled experiment, although they do not rule out greater risks in larger or evolving collections. Source citations provide a basis for independent verification, while automatic conflict detection and resolution remain future work.
W4. AI-generated figures
We will redraw the illustrative figures as clean vector graphics for the final version if the paper is accepted.