- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose CrossAug to address a key limitation of existing GraphRAG methods: their inability to capture important relationships spanning multiple chunks. CrossAug generates self-supervised training data by masking or corrupting edges in the existing graph and trains a topology-aware GNN to identify subgraphs likely to contain missing relationships. It then selectively uses an LLM to integrate cross-chunk evidence and recover missing entities and relations. By augmenting the graph offline, CrossAug improves QA performance across various GraphRAG frameworks without additional online inference overhead.
Critical strengths
- Clear Problem Formulation and Practical Motivation
The paper clearly identifies the limitations of chunk-level entity and relation extraction in existing GraphRAG methods, which can miss relationships spanning multiple chunks. Focusing on graph index incompleteness rather than retrieval alone provides practical motivation. The LiteraryQA example further illustrates how missing cross-chunk relations affect downstream QA.
- Self-Supervised Learning and GNN–LLM Collaboration
CrossAug introduces self-supervised GNN training without requiring manual annotations of missing relations. Through two-stage fact masking and incorrect fact reconstruction, the GNN identifies subgraphs requiring augmentation, while the LLM recovers missing relations. This division of responsibilities effectively leverages the complementary strengths of GNNs and LLMs.
- Offline Augmentation Compatible with Existing GraphRAG
CrossAug augments existing GraphRAG indices without substantially modifying their retrieval algorithms. Since GNN scoring and LLM completion are performed offline, augmented relations can be reused across queries without additional online inference overhead, offering practical benefits for frequently queried systems.
- Consistent Improvements Across Frameworks and Datasets
The authors evaluate CrossAug on three GraphRAG frameworks (HippoRAG2, LightRAG, and GFM-RAG) and four QA datasets, reporting improvements across all 24 framework–dataset–metric combinations. Although the gains are modest, their consistency across different retrieval mechanisms supports the general applicability of the proposed augmentation approach.
Critical weaknesses
- Limited Methodological Novelty of the GNN-Based Subgraph Scoring Framework
The proposed framework combines established graph learning techniques, including relational message passing, mean pooling, MLPs, graph corruption, and pairwise ranking loss. Prior studies such as GraIL, NBFNet, KGC-STA, and DrKGC have explored related approaches. Although CrossAug differs by scoring subgraph-level missingness rather than predicting individual links, the significance of this distinction is insufficiently demonstrated. In particular, the authors compare GNN-based selection only against random selection, omitting simple structural heuristics (e.g., graph density, node degree, and connectivity) and existing GNN-based completion methods. Thus, the benefits of the proposed scoring framework over simpler alternatives remain unclear.
- Limited Applicability and Modest Gains Relative to Additional Costs
CrossAug requires entity nodes, fact edges, and source-chunk provenance, limiting its applicability to GraphRAG frameworks without explicit relational triples, such as LinearRAG and E²GraphRAG. Despite consistent improvements, the average EM gain across 12 framework–dataset combinations is only approximately 0.76 percentage points, with eight combinations improving by less than one point. Meanwhile, indexing token consumption increases by 43.6% for HippoRAG2 on MuSiQue, and the reported factual correctness of generated triples is 84.35%. Without an analysis of amortized costs across queries, it remains unclear whether these modest improvements justify the additional computational overhead and potential graph contamination.
- Lack of Direct Validation of Real-World Missingness Detection
CrossAug assumes that GNNs trained on synthetically corrupted graphs can identify actual cross-chunk missing relations. However, synthetic corruption may not adequately represent real-world missingness caused by distributed evidence, missing entities, coreference failures, or incomplete extraction. The paper provides only indirect evidence through downstream QA improvements over random selection, without directly evaluating missingness detection accuracy. Furthermore, factual and citation correctness do not establish whether recovered triples genuinely require cross-chunk reasoning. More direct evaluations are needed, including Precision@K and Recall@K for missing subgraphs, annotated cross-chunk missingness datasets, the proportion of recovered relations requiring multiple chunks, and analysis of distributional differences between synthetic and real missingness. Consequently, it remains unclear whether CrossAug specifically addresses cross-chunk missingness or primarily benefits from additional LLM-based graph extraction.
- Insufficient Analysis of Graph Construction Differences Across Frameworks
HippoRAG2, LightRAG, and GFM-RAG differ in entity representations, relation extraction, chunking, graph connectivity, and retrieval mechanisms, potentially affecting the types and frequency of missing relations. However, the paper provides limited details on framework-specific graph construction settings and does not systematically analyze their impact on CrossAug. Appendix A mainly reports dataset-level statistics, while the relatively smaller improvements on LightRAG remain unexplained. Experiments varying chunk size, overlap, graph density, and connectivity would help establish the robustness of CrossAug across different graph construction settings.
- Insufficient Analysis of Hyperparameter Sensitivity and Generalizability
Although Appendices D and E include sensitivity analyses, they are largely restricted to HippoRAG2 on MuSiQue, with limited explanations for the observed performance variations.
| Hyperparameter | Configuration | EM |
|---|---|---|
| Random walks | 20 → 60 | 30.10 → 28.30 |
| Reconstruction weight | 0.1 → 5 | 27.30 → 30.10 |
| Completion budget | 0.05 → 0.10 | 28.50 → 30.10 |
These variations exceed the reported performance gains in many framework–dataset settings. However, the paper does not adequately explain the underlying causes or verify whether the default hyperparameters generalize across different frameworks and datasets. Therefore, the stability of hyperparameter selection and the generalizability of CrossAug across graph construction settings remain insufficiently established.
Questions for authors
See Weakness.
5 comments
Response to Reviewer 1YwN W2
W2: Applicability, gain size, and indexing cost.
Applicability. CROSSAUG targets graphs built from LLM-extracted triples. As noted in Sections 2 and 4.1, LinearRAG and E²GraphRAG belong to the lightweight family that builds relation-free graphs with NLP tools; they are outside its scope, and we report LinearRAG only for context.
Gain size. For the main Base-versus-CROSSAUG experiments, the table below reports mean improvements over Base (percentage points) and two-sided t-test p-values using three seeds per method in each framework–dataset setting. Bold marks .
| Framework | MuSiQue EM | MuSiQue F1 | 2Wiki EM | 2Wiki F1 | HotpotQA EM | HotpotQA F1 | LiteraryQA EM | LiteraryQA F1 |
|---|---|---|---|---|---|---|---|---|
| HippoRAG2 | +1.60 / 0.010 | +1.45 / 0.016 | +0.34 / 0.024 | +0.30 / 0.160 | +0.50 / 0.038 | +0.29 / 0.302 | +0.76 / 0.082 | +1.15 / 0.045 |
| LightRAG | +0.44 / 0.247 | +0.27 / 0.438 | +0.73 / 0.126 | +0.47 / 0.070 | +0.40 / 0.444 | +0.25 / 0.600 | +0.17 / 0.372 | +0.22 / 0.282 |
| GFM-RAG | +1.10 / 0.044 | +0.80 / 0.144 | +1.43 / 0.004 | +1.59 / 0.002 | +0.34 / 0.121 | +0.86 / 0.010 | +1.34 / 0.003 | +1.06 / 0.009 |
Eleven of the 24 comparisons reach , 11 of 16 for HippoRAG2 and GFM-RAG. With only three seeds, we treat these tests as supporting evidence, not as proof that every individual gain is significant. Two further observations point to a systematic improvement. First, the mean improvement is positive in all 24 comparisons. Second, the larger gains appear where cross-chunk evidence matters most, namely MuSiQue and the long-document LiteraryQA. On 2Wiki and HotpotQA, most questions can be answered from short, self-contained Wikipedia passages. We will describe the improvements as consistent but heterogeneous in size, and report this significance analysis in the paper.
As a complementary analysis of the overall effect, we first average the three seeds within each framework–dataset setting and then use the 12 settings as paired units. We conduct a two-sided t-test separately for EM and F1:
| Metric | Mean improvement (percentage points) | Two-sided p-value |
|---|---|---|
| EM | +0.7611 | 0.000220 |
| F1 | +0.7256 | 0.000369 |
Cost. CROSSAUG's additional usage is mainly prompt tokens, because each completion request carries several evidence passages and the local graph. We also report the cost of the chunk-wise second pass for comparison.
We estimate API costs using the published Qwen3-32B International rates of USD 0.16 per million input tokens and USD 0.64 per million output tokens. Each cost is calculated as 0.16 × Prompt (M) + 0.64 × Completion (M), without discounts or free quotas, and rounded to two decimal places. The Base + rows include the initial Base indexing stage.
| Dataset | Stage | Prompt (M) | Completion (M) | Total (M) | Estimated API cost (USD) |
|---|---|---|---|---|---|
| MuSiQue | Base indexing | 10.839 | 27.219 | 38.058 | 19.15 |
| MuSiQue | CROSSAUG, additional | 11.798 | 4.802 | 16.600 | 4.96 |
| MuSiQue | Base + CROSSAUG | 22.637 | 32.021 | 54.658 | 24.12 |
| MuSiQue | Base + chunk-wise second pass | 21.007 | 41.265 | 62.272 | 29.77 |
| LiteraryQA | Base indexing | 17.839 | 23.025 | 40.864 | 17.59 |
| LiteraryQA | CROSSAUG, additional | 45.024 | 5.986 | 51.011 | 11.03 |
| LiteraryQA | Base + CROSSAUG | 62.863 | 29.011 | 91.875 | 28.63 |
| LiteraryQA | Base + chunk-wise second pass | 30.445 | 41.139 | 71.584 | 31.20 |
Two facts bear on whether this overhead is justified. First, CROSSAUG achieves higher EM and F1 than the chunk-wise second pass at a lower estimated API indexing cost on both datasets: USD 24.12 versus USD 29.77 on MuSiQue and USD 28.63 versus USD 31.20 on LiteraryQA. On LiteraryQA, this lower cost holds despite higher total token usage, because CROSSAUG generates fewer output tokens, which are priced four times as high as input tokens. The QA comparison is reported in W4. Second, the cost is paid once, offline: the GNN and the completion LLM are never invoked at query time, and the augmented index is reused as a static index. The amortized overhead per query is therefore the one-time cost divided by the number of queries the index serves, and reporting a specific value would require assuming a query workload that these benchmarks do not define. We therefore report the one-time overhead.
Graph contamination. The 84.35% factual correctness rate corresponds to an estimated 15.65% factual error rate in the sampled triples. Positive aggregate QA gains do not rule out harm on individual questions, and we will explicitly report this error rate and the associated risk of downstream error propagation.
Response to Reviewer 1YwN W3
W3: Validity of synthetic missingness and direct detection evaluation.
Section 4.4 provides direct evidence about what CROSSAUG recovers. Among 497 valid annotations of sampled, factually correct triples, 91.5% correspond to identifiable real-world omission patterns — missing relations or entities, partial extraction, alias/coreference failures, and relation-type refinement—covering several of the extraction failures highlighted by the reviewer. These findings provide evidence that CROSSAUG recovers genuine extraction omissions beyond its synthetic training patterns.
Precision@K and Recall@K for missing subgraphs require gold annotations of which regions of a real index are incomplete, and Recall@K additionally requires an exhaustive annotation of omissions across the candidate pool. To our knowledge, no annotation set of this kind exists for these corpora. Constructing an exhaustive reference for Recall@K would require region-level labelling, whereas CROSSAUG obtains its training supervision without such labels. We therefore provide the triple-level validation in Section 4.4 as complementary evidence of real-world omission recovery, together with downstream QA evaluation. We note explicitly that the 91.5% figure is a triple-level category distribution, not region-selection precision, and we do not claim synthetic and natural missingness share an identical distribution.
Response to Reviewer 1YwN W1
We thank the reviewer for the detailed reading and for recognizing our motivation and the GNN–LLM method design. We respond to the five weaknesses in turn.
W1: Novelty of the scoring framework; KGC methods and simpler heuristics.
We agree that relational message passing, mean pooling, MLPs, and pairwise ranking are established components, and we do not claim novelty for them. The contribution is the formulation and supervision of a different decision: where to complete a graph index when neither the missing entities nor the missing relations are known, under a fixed LLM budget. Two-stage fact masking supplies relative-missingness supervision, and incorrect-fact reconstruction keeps node and edge counts matched within its ranking pairs, so the scorer is not rewarded for size or density alone in these pairs.
The cited KGC methods (GraIL, NBFNet, KGC-STA, DrKGC) solve a different task. Each predicts or scores a missing link conditioned on a candidate triple or an entity–relation query: GraIL builds an enclosing subgraph around candidate endpoints, NBFNet aggregates paths from a source entity under a query relation, KGC-STA augments an enclosing subgraph for a candidate triple, and DrKGC retrieves a query-specific subgraph starting from an incomplete triple. CROSSAUG operates one step earlier, before any candidate exists, and its output is a set of regions passed to an LLM that extracts new triples from evidence. Adapting a link predictor to this role would require generating and scoring candidate triples across the graph, then aggregating link scores into region scores. This adds a candidate-search and link-scoring stage before region selection, introducing computational overhead that CROSSAUG avoids by directly scoring sampled subgraphs. Off-the-shelf link predictors therefore do not directly solve the budgeted region-selection problem considered here. We will clarify this task and computational distinction in the revised paper.
The second part of the concern is fair, and we address it with a separate comparison experiment. We compare CROSSAUG with Low-Density and Low-Degree, which replace the GNN score with fact-edge density and mean internal fact degree, and a Chunk-wise Second Pass, which re-extracts each chunk from its own text and existing triples:
At an equal number of calls, the heuristics help on LiteraryQA but not on MuSiQue, where Low-Density lowers both metrics and Low-Degree trades EM for F1; CROSSAUG is higher than both on both datasets. The second pass does not improve both metrics over Base on either dataset; we report its token consumption in our response to W4 below. Trace inspection illustrates a limitation of reading chunks in isolation in the MuSiQue case we examined: the second pass adds no fact to the museum passage, while CROSSAUG recovers the museum–Da Nang relation and thereby retrieves the regional evidence needed for the answer. These results support the scorer over the simple alternatives tested.
Table 4 also includes a semantic-only ablation that removes graph message passing while retaining semantic features. It achieves 28.10/40.03 on MuSiQue and 17.82/32.90 on LiteraryQA, compared with 30.10/41.77 and 18.78/34.31 for full CROSSAUG (EM/F1 in %). The performance gap supports the benefit of incorporating graph message passing into the scorer.