acceptodds
Under review as a conference paper at ICLR 2027

Enriching Document Knowledge Bases through Context Propagation

Abstract

Understanding real-world document collections, such as conversations or codebases, requires more than retrieving isolated facts: it involves reconciling information scattered across documents and forming a coherent view of entities and events. This can be challenging because authors often assume shared context, using *implicit cross-document references* whose interpretation depends on information in other documents. To isolate this challenge, we introduce XRefQA, a diagnostic benchmark of small synthetic document collections in which answering questions requires resolving these references and combining evidence from multiple documents. Even Claude Opus 4.6 achieves only 7% accuracy on XRefQA using standard retrieval-augmented generation (RAG). To address this limitation, we propose *context propagation*, which iteratively enriches each document with source-grounded annotations using evidence retrieved from related documents. Enriched documents inform subsequent rounds, allowing resolved references and related facts to propagate across the collection. After two propagation rounds, the weaker Claude Sonnet 4.6 achieves 80% accuracy with RAG over enriched documents. Beyond implicit-reference resolution, the method also improves multi-hop question answering on MuSiQue, 2WikiMultiHopQA, and HotpotQA, as well as fact discovery and source attribution on SummHay. These results suggest that making cross-document connections explicit during preprocessing helps downstream systems retrieve, combine, and attribute distributed evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Context propagation, retrieval-augmented generation, implicit cross-document references, multi-hop question answering, document enrichment, cross-document reasoning, source attribution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3ivhOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes context propagation, which is a method that enriches documents with info from other related documents before the retrieval process. The paper also introduces XRefQA, a benchmark for testing cross-doc reasoning. The results show strong improvements over naive RAG and some other methods.

Critical strengths
  • The motivation is great - making connections between docs explicitly before retrieval should be a useful idea.

  • The results are quite strong.

  • I appreciate the ablations that try to explain where the improvements come from.

Critical weaknesses
  • Limited evaluation on real-world data. XRefQA is synthetic and has only 50 test samples. I'd like to see whether the method works as well on real document collections, where the information can be incomplete, outdated and some are conflicting.

  • Missing comparison with closely related work. IndexRAG also builds cross-document knowledge during indexing. The paper mentioned it in the related work section but did not compare with it.

  • There's a risk of error propagation. The method uses annotations from earlier rounds to enrich other documents. If an annotation is wrong, the error may propagate to the entire collection.

  • AI-generated figures. The figures in this paper are obviously AI-generated and I hate AI-generated things. I mean, they are simple enough and the authors can totally remake them into a vector format by their hands using powerpoints or other software. Please do this if the paper is accepted.

Questions for authors

see weaknesses.

1 comment

We thank the reviewer for the constructive feedback and for recognizing the motivation, experimental results, and diagnostic ablations. We address the concerns below, including new experiments comparing CP with IndexRAG and investigating error propagation across annotation rounds.

W1. Limited evaluation on real-world data, where the information can be incomplete, outdated and some are conflicting

We agree that our evaluation does not cover real-world collections with incomplete, outdated, or conflicting information. This work identifies implicit cross-document references as a challenge, introduces XRefQA to measure it, and proposes CP to address it. We further demonstrate improvements on human-written Wikipedia documents in MuSiQue, 2Wiki, and HotpotQA, providing evidence beyond the synthetic benchmark. Extending CP to heterogeneous, evolving collections is a natural next step that we leave for future work. We envision this direction as closely connected to our broader research on structured knowledge representations, such as navigable project wikis and persistent memory layers, where maintaining up-to-date information and resolving conflicts would be essential.

W2. IndexRAG comparison

We evaluated IndexRAG using its publicly released implementation on MuSiQue, 2Wiki, and HotpotQA (1,000 questions each), with the same retriever (NV-Embed-v2), preprocessing and QA model (GPT-4o-mini), and evaluation protocol, without reranking. CP outperforms IndexRAG on all three benchmarks, achieving 65.9 versus 60.2 average F1 (+5.7) with top-5 retrieval versus IndexRAG's top-10 setting. On MuSiQue, CP also uses fewer reader tokens (742 versus 1,291), with comparable preprocessing costs (14.69 dollars versus 13.52 dollars). These results support the effectiveness of attaching cross-document information to original passages rather than indexing extracted and bridging facts separately. We provide the full results table in our response to Reviewer 2wqP and have included the comparison in the revised manuscript.

W3. Risk of error propagation

We agree that incorrect annotations can propagate across rounds. To investigate this risk in a controlled setting, we injected 1 or 2 incorrect numerical claims per XRefQA collection (10 and 20 errors in total) into round-1 annotations and reran round 2.

We examined two outcomes: (i) whether the injected errors persisted in the injected documents, and (ii) whether they spread to other documents:

(i): After round 2, only 1 of the 10 injected errors and 2 of the 20 injected errors remained in their original documents; the others were corrected or removed.

(ii): To measure propagation, we examined the other documents in each experiment (231 and 221, respectively). During round-2 annotation, 106 and 167 of these documents received prompts containing an injected false value from another document. Of those, only 2/106 (1.9%) and 3/167 (1.8%) repeated the false value in their new annotations. Importantly, none of the two in the first experiment and only one of the three in the second adopted the false value as fact. The remaining ones either presented it alongside the correct value or explicitly flagged a conflict.

These results suggest limited error persistence and propagation in our controlled experiment, although they do not rule out greater risks in larger or evolving collections. Source citations provide a basis for independent verification, while automatic conflict detection and resolution remain future work.

W4. AI-generated figures

We will redraw the illustrative figures as clean vector graphics for the final version if the paper is accepted.

Reviewer_2wqPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents context propagation, which enriches documents with cited information from related documents before downstream retrieval and QA. The enriched documents can then inform subsequent annotation rounds. The paper introduces XRefQA, a synthetic benchmark for implicit cross-document references, and reports improvements on this benchmark, existing multi-hop QA datasets, and SummHay.

Critical strengths
  1. XRefQA addresses the difficulty of connecting evidence through implicit references, even when all the necessary information is present in the collection. The full-context experiments suggest that retrieving more documents alone does not resolve this difficulty (Table 8). The benchmark therefore offers a useful diagnostic, although its scale is limited.

  2. Attaching contextual annotations and source citations to the original passages makes cross-document information reusable at query time. The analysis with fixed retrieved passages shows that the annotations themselves improve QA (Table 6), providing evidence that the benefit goes beyond changes in retrieval.

Critical weaknesses
  1. The paper identifies IndexRAG as closely related but does not include it in the experiments. The current comparisons show that enrichment helps over raw-corpus retrieval. A comparison with an indexing-time method would help determine whether attaching information to the original documents offers an advantage over retrieving separately indexed facts.

  2. The experiments show that document enrichment is useful, but the benefit of iterative propagation is less clear. The existing QA benchmarks and SummHay use one round. On XRefQA, the ablation removing the target's previous annotation still uses annotated neighbors (Table 10). Repeating annotation with raw supporting documents would help check whether the second-round gain depends on propagated context.

Questions for authors
  1. Do you have a comparison with IndexRAG or another indexing-time fact-integration method under comparable preprocessing and retrieval settings?

  2. On XRefQA, are results available for repeating annotation with raw supporting documents? How does this compare with using annotated neighbors?

2 comments

Q1. IndexRAG Comparison

We evaluated IndexRAG using its publicly released implementation on MuSiQue, 2Wiki, and HotpotQA (1,000 questions each), following our Section 5.2 protocol without reranking. Both methods use NV-Embed-v2 for retrieval and GPT-4o-mini for preprocessing and QA. Preprocessing costs were comparable (13.52 dollars for IndexRAG versus 14.69 dollars for CP).

MethodMuSiQue2WikiHotpotQAAvg.Reader tokens
IndexRAG, top 544.057.269.857.0630
IndexRAG, top 1048.960.071.660.21,291
CP, top 551.971.274.565.9742

F1 scores; reader tokens measured on MuSiQue. IndexRAG retrieves extracted and bridging facts, while CP retrieves enriched passages.

CP outperforms IndexRAG on all three benchmarks. Compared with IndexRAG's top-10 setting, CP achieves 5.7 higher average F1 while using fewer reader tokens on MuSiQue (742 versus 1,291). These results support the effectiveness of attaching cross-document evidence to original passages rather than indexing extracted facts separately. We have included this comparison in the revised manuscript.

Q2. On XRefQA, are results available for repeating annotations with raw supporting documents? How does this compare with using annotated neighbors?

We conducted the suggested ablation, keeping the round-2 retrieved neighbor set fixed while varying whether the target and supporting documents include previous annotations.

ConfigurationTargetNeighborsAcc.Output tokens/doc
Round 1RawRaw, R1 set58%3,334
Round 1, R2 neighborsRawRaw, R2 set76%4,261
Round 2, raw neighborsRaw + previous annotationRaw, R2 set78%2,519
Round 2, annotated neighborsRaw + previous annotationAnnotated, R2 set80%2,033

The main round-2 accuracy gain comes from improved neighbor retrieval: using neighbors discovered from the round-1 enriched corpus improves accuracy from 58% to 76%, even when the annotator receives only raw documents. This shows that earlier annotations help discover relevant cross-document connections, without requiring their content to be reused in subsequent annotation prompts.

Reusing annotations primarily improves efficiency. Retaining the target's previous annotation reduces output length from 4,261 to 2,519 tokens per document, and using annotated neighbors further reduces it to 2,033. Hence, the main accuracy benefit of propagation comes from better evidence discovery, while directly reusing annotations reduces output length without a clear additional accuracy gain.

Thanks for response. I decide to keep my score unchanged.

Reviewer_49RROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces XRefQA, a 100-question synthetic diagnostic (50 test questions over 10 mini-collections of 20–25 documents, ~15K tokens each). Answering a question requires resolving implicit cross-document references such as "the stylist" or "the Foundry". The paper also proposes context propagation: for each document, retrieve neighbours and have an LLM insert source-cited bracketed annotations inline. This is repeated for R rounds over the already-annotated corpus, and then standard RAG runs over the enriched documents. On XRefQA, Sonnet 4.6 accuracy rises from 2% to 80%. The paper reports +1.5 to +3.4 F1 over HippoRAG 2 in its multi-hop setting and +5.5 joint on SummHay.

Critical strengths
  1. The enriched corpus is human-readable and source-cited. It is independent of the solver, and the results shows that it transfers across annotator and solver families.
  2. The paper separates retrieval gains from annotation-content gains: annotations matter most when they supply all missing gold facts, which raises F1 from 41.8 to 64.0. The failure taxonomy, the manual annotation audit, the token-matched HippoRAG 2 control, and the detailed cost accounting are all welcome.
  3. The method is evaluated on three different task types, not only on the new benchmark.
Critical weaknesses
  1. The benchmark is small, self-built, and aligned with the method. XRefQA has 50 test questions from 10 collections, and the reported SEs ignore this clustering. Claude generated the data and also serves as annotator and main solver. Every gold chain is a named-entity bridge, which entity-lexical retrieval targets: accuracy drops from 0.80 to 0.42 without it. No natural or human-written data is evaluated.

  2. The headline is misleading, and the comparison is not compute-matched. Opus reaches 0.76 when it reads the full ~15K-token collection, so the "7%" result reflects k=10 retrieval. CP runs two Sonnet passes over every document before any query, so the reasoning moves into preprocessing rather than disappearing. The full-context accuracies for the configurations logged in Tab 15 are not reported. A single-pass annotation with the whole collection in context is a missing control.

  3. Some baselines look under-configured. HippoRAG 2 scores 0.00 and GraphRAG 0.02. Iterative RAG uses only 3–4 of its 20 turns and often abstains. Contextual Retrieval sees more context than CP round 1 but scores 0.11. Please report abstention rates and gold-document recall for these methods.

  4. The annotations contain unaudited errors. The App D.1 example alters the document TIMESTAMP, misdates several events, and asserts a key bridge that appears in no supporting document shown. XRefQA annotations are never audited. Removing citation tags barely changes accuracy, which suggests that solvers trust the inserted claims without checking them.

  5. The gold answers are questionable, and the prompt may leak a hint. In the App B example, the documents do not state that the added and removed styles belong to Harlow & Finch. The Background Summary in the annotation prompt supplies the missing premise. The answerability check runs on the CP-annotated corpus, so it is circular.

Questions for authors

no more question

2 comments

W1: Benchmark design and generalization

Benchmark is small, self-built, and aligned with the method

XRefQA is designed as a focused diagnostic benchmark for implicit cross-document reference resolution, a challenge not isolated by existing QA benchmarks (Related Work, ln 143-145). Constructing targeted benchmarks to study specific phenomena is an established approach, as illustrated by [Press+23] and [Zhang+24].

[Press+23]: Measuring and Narrowing the Compositionality Gap in Language Models

[Zhang+24]: How Language Model Hallucinations Can Snowball

No natural or human-written data is evaluated

This claim is incorrect. As the reviewer also recognizes in Strength 3, our evaluation covers three task types. In particular, MuSiQue, 2Wiki, and HotpotQA use human-written Wikipedia documents, demonstrating benefits beyond XRefQA. Please also see our response to W1 of Reviewer 3ivh.

XRefQA has 10 collections, and reported SEs ignore this clustering.

We recalculated standard errors across the 10 collections, accounting for the five questions sharing each collection. CP's SE remains unchanged at 0.05, while the largest corrections are from 0.03 to 0.05 for Contextual Retrieval and Iterative RAG (medium). These corrections do not change our main conclusions. We have updated the reported SEs in the manuscript.

Claude generated data and serves as annotator and main solver

This characterization overlooks cross-model evaluations already reported in the paper. Although Claude generated XRefQA, Table 2 also evaluates Kimi and Qwen as both annotators and solvers, with accuracy improving from 3% to 64% and from 5% to 53%, respectively. Cross-family annotator-solver experiments further demonstrate that the gains are not limited to Claude.

Every gold chain is a named-entity bridge, which entity-lexical retrieval targets: accuracy drops from 0.80 to 0.42 without it.

Named entities are central in our approach as they convert implicit references to explicit ones. It is therefore natural that entity-lexical retrieval improves the performance. We do not consider this as a limitation: keyword search is one of the most-used actions by modern coding agents.

W2: Comparison framing, preprocessing cost, and a missing control

CP is designed for the case when the entire collection is large enough not to fit into context and therefore we intentionally used RAG methods as baseline. We disagree that the full-context inference is the central baseline, and (to avoid this confusion) we move it to the appendix. We can move the full-context result into the main text but this will only confuse the reader. With the same argumentation, we would not like to include the single-pass annotation, a baseline designed for the full-context scenario. For the insights about how annotations help, we already present ablations which distinguish retrieval improvements from annotation-content gains.

Full-context accuracies for configurations logged in Tab 15 are not reported.

For completeness, we have included the missing full-context results from Table 15 in Table 8, although these results are not central to our main findings.

InputSonnetOpus
Raw, no think29%76%
Raw, medium think69%76%
CP, round 387%100%

W3: Baseline configurations, recall/abstention analysis

Configuring GrahRAG and HippoRAG 2 requires a substantial effort, that is why we used the published implementations of them. Furthermore, previously published results of these methods on established multi-hop QA benchmarks suggest that their performance is close to vanilla RAG in these settings (see our Table 3, rows: GraphRAG, HippoRAG 2, NV-Embed-v2 and NV-Embed-v2*). Therefore, we believe that, even with substantial configuration effort, the results will not improve significantly.

Additionally, we report below retrieval coverage and abstention of baselines on XRefQA. Although the retrieval-based methods recover 59-67% of individual gold documents, they retrieve all required documents for only 8-20% of questions, helping explain their low accuracy.

MethodGold doc recallAll gold recalledAbstention
HippoRAG 2 + Background62%10%14%
GraphRAG + BackgroundN/AN/A68%
Contextual Retrieval + Background59%8%8%
Iterative RAG (medium)67%20%4%

GraphRAG N/As due to usage of community summaries rather than direct document retrieval.

Iterative RAG rarely abstains (6% originally, 4% with medium reasoning). Our trajectory analysis shows that it typically answers with the initial value within 2–3 turns, missing either the subsequent update or a bridge document linking it to the entity. All runs terminated voluntarily before reaching the 20-turn limit. More available turns do not help if the agent does not recognize what additional evidence to retrieve.

(W3 continued)

Our adaptation of Contextual Retrieval uses the entire collection to contextualize each document, but does not explicitly establish connections between related documents. Access to more context alone does not ensure that the implicit cross-document relationships needed to answer XRefQA questions are resolved.

W4: Annotation accuracy and source verification

The App D.1 example alters the document TIMESTAMP, misdates several events, and asserts a key bridge that appears in no supporting document shown.

We thank the reviewer for identifying these inconsistencies and apologize for the error in Appendix D.1. The appendix mistakenly displayed earlier versions of the raw documents, rather than the actual inputs used to generate the annotations. We verified the original experimental inputs and confirmed that the timestamps and event dates match the annotations, while the Foundry relationship is supported by Documents 2 and 4. We have replaced the displayed raw documents with the verified experimental inputs. Hence, the reported inconsistencies were caused by an appendix presentation error, not by CP introducing incorrect facts.

XRefQA annotations are never audited. Removing citation tags barely changes accuracy, which suggests that solvers trust the inserted claims without checking them.

During development, we qualitatively inspected Sonnet-generated XRefQA annotations and observed no errors in the examples examined. Separately, we audited 50 GPT-4o-mini annotations on MuSiQue (Appendix E.2): 36 (72%) added new, source-supported information, while others contained redundant or erroneous claims.

The primary purpose of citations is to make annotations traceable for independent verification and potential conflict resolution by a separate process or human reviewer. Citation tags do not force the solver to verify claims, so their removal barely affects accuracy.

W5: Gold-answer grounding, Background Summary, and answerability

In the App B example, the documents do not state that the added and removed styles belong to Harlow & Finch

This is intentional and reflects the central challenge XRefQA is designed to test: resolving implicit references across documents rather than relying on explicit entity-name matches. In this example, the raw documents establish the following chain: D1: Harlow & Finch - 185 styles; D2: Harlow & Finch - Ren Jørgensen; D3: Ren - 15 additions; D4: Ren - Foundry; D5: Foundry - 8 removals. Connecting these references yields 185 + 15 − 8 = 192 styles. We have clarified this reasoning chain in Appendix B.

Background Summary in annotation prompt supplies missing premise

The Background Summary is an intentional part of XRefQA, providing shared information about the setting. Its purpose is to support interpretation of the documents, while the specific entity relationships and updates must still be recovered from them. Importantly, the Background Summary is not exclusive to CP. All baselines already receive it at inference time. We additionally provided it to Contextual Retrieval and HippoRAG 2 during indexing, increasing their accuracies from 11% to 18% and from 0% to 4%, respectively, still substantially below CP (58% and 80%). GraphRAG remained at 2% when given the summary. Hence, access to the Background Summary does not explain the performance gap. We updated our Table 1 with these results.

Answerability check runs on CP-annotated corpus, so it is circular

To independently verify answerability from the raw documents, we conducted an additional check using GPT-6.1 Sol, a model from a different family than the one used to generate XRefQA. We provided only the questions and original raw gold documents, without CP annotations or gold reasoning chains. It achieved 99% accuracy across all 100 XRefQA questions. Together with our manual inspection during benchmark construction, this provides strong evidence that the gold answers are recoverable from the original documents.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.