acceptodds
Under review as a conference paper at ICLR 2027

Beyond Chunk-Local Extraction: Cross-Chunk Graph Augmentation for GraphRAG

Abstract

GraphRAG extends retrieval-augmented generation by organizing corpora as explicit knowledge graphs, enabling graph-based retrieval for complex question answering. However, existing frameworks extract entities and relations only within individual chunks, leaving cross-chunk relations—those whose evidence spans multiple passages—systematically absent from the index. Exhaustive LLM-based recovery of such relations is impractical due to the combinatorial explosion of chunk combinations. We present CrossAug, a GNN-guided CROSS-Chunk Graph AUGmentation method that enriches GraphRAG indices with cross-chunk relational structure as an offline step before query-time retrieval. CrossAug derives its training signal from self-supervised training pairs constructed from the base graph, uses a topology-aware GNN to score subgraphs for missingness, and applies evidence-grounded LLM completion only to the selected high-scoring regions. Experiments on three LLM-based GraphRAG frameworks across four multi-hop and long-document QA benchmarks show improvements across all framework–dataset–metric comparisons, confirming the benefit of cross-chunk graph augmentation for retrieval-based question answering. Our code is available at https://anonymous.4open.science/r/CrossAug-72BB/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GraphRAG, Cross-Chunk Extraction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1YwNOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose CrossAug to address a key limitation of existing GraphRAG methods: their inability to capture important relationships spanning multiple chunks. CrossAug generates self-supervised training data by masking or corrupting edges in the existing graph and trains a topology-aware GNN to identify subgraphs likely to contain missing relationships. It then selectively uses an LLM to integrate cross-chunk evidence and recover missing entities and relations. By augmenting the graph offline, CrossAug improves QA performance across various GraphRAG frameworks without additional online inference overhead.

Critical strengths
  • Clear Problem Formulation and Practical Motivation

The paper clearly identifies the limitations of chunk-level entity and relation extraction in existing GraphRAG methods, which can miss relationships spanning multiple chunks. Focusing on graph index incompleteness rather than retrieval alone provides practical motivation. The LiteraryQA example further illustrates how missing cross-chunk relations affect downstream QA.

  • Self-Supervised Learning and GNN–LLM Collaboration

CrossAug introduces self-supervised GNN training without requiring manual annotations of missing relations. Through two-stage fact masking and incorrect fact reconstruction, the GNN identifies subgraphs requiring augmentation, while the LLM recovers missing relations. This division of responsibilities effectively leverages the complementary strengths of GNNs and LLMs.

  • Offline Augmentation Compatible with Existing GraphRAG

CrossAug augments existing GraphRAG indices without substantially modifying their retrieval algorithms. Since GNN scoring and LLM completion are performed offline, augmented relations can be reused across queries without additional online inference overhead, offering practical benefits for frequently queried systems.

  • Consistent Improvements Across Frameworks and Datasets

The authors evaluate CrossAug on three GraphRAG frameworks (HippoRAG2, LightRAG, and GFM-RAG) and four QA datasets, reporting improvements across all 24 framework–dataset–metric combinations. Although the gains are modest, their consistency across different retrieval mechanisms supports the general applicability of the proposed augmentation approach.

Critical weaknesses
  • Limited Methodological Novelty of the GNN-Based Subgraph Scoring Framework

The proposed framework combines established graph learning techniques, including relational message passing, mean pooling, MLPs, graph corruption, and pairwise ranking loss. Prior studies such as GraIL, NBFNet, KGC-STA, and DrKGC have explored related approaches. Although CrossAug differs by scoring subgraph-level missingness rather than predicting individual links, the significance of this distinction is insufficiently demonstrated. In particular, the authors compare GNN-based selection only against random selection, omitting simple structural heuristics (e.g., graph density, node degree, and connectivity) and existing GNN-based completion methods. Thus, the benefits of the proposed scoring framework over simpler alternatives remain unclear.

  • Limited Applicability and Modest Gains Relative to Additional Costs

CrossAug requires entity nodes, fact edges, and source-chunk provenance, limiting its applicability to GraphRAG frameworks without explicit relational triples, such as LinearRAG and E²GraphRAG. Despite consistent improvements, the average EM gain across 12 framework–dataset combinations is only approximately 0.76 percentage points, with eight combinations improving by less than one point. Meanwhile, indexing token consumption increases by 43.6% for HippoRAG2 on MuSiQue, and the reported factual correctness of generated triples is 84.35%. Without an analysis of amortized costs across queries, it remains unclear whether these modest improvements justify the additional computational overhead and potential graph contamination.

  • Lack of Direct Validation of Real-World Missingness Detection

CrossAug assumes that GNNs trained on synthetically corrupted graphs can identify actual cross-chunk missing relations. However, synthetic corruption may not adequately represent real-world missingness caused by distributed evidence, missing entities, coreference failures, or incomplete extraction. The paper provides only indirect evidence through downstream QA improvements over random selection, without directly evaluating missingness detection accuracy. Furthermore, factual and citation correctness do not establish whether recovered triples genuinely require cross-chunk reasoning. More direct evaluations are needed, including Precision@K and Recall@K for missing subgraphs, annotated cross-chunk missingness datasets, the proportion of recovered relations requiring multiple chunks, and analysis of distributional differences between synthetic and real missingness. Consequently, it remains unclear whether CrossAug specifically addresses cross-chunk missingness or primarily benefits from additional LLM-based graph extraction.

  • Insufficient Analysis of Graph Construction Differences Across Frameworks

HippoRAG2, LightRAG, and GFM-RAG differ in entity representations, relation extraction, chunking, graph connectivity, and retrieval mechanisms, potentially affecting the types and frequency of missing relations. However, the paper provides limited details on framework-specific graph construction settings and does not systematically analyze their impact on CrossAug. Appendix A mainly reports dataset-level statistics, while the relatively smaller improvements on LightRAG remain unexplained. Experiments varying chunk size, overlap, graph density, and connectivity would help establish the robustness of CrossAug across different graph construction settings.

  • Insufficient Analysis of Hyperparameter Sensitivity and Generalizability

Although Appendices D and E include sensitivity analyses, they are largely restricted to HippoRAG2 on MuSiQue, with limited explanations for the observed performance variations.

HyperparameterConfigurationEM
Random walks20 → 6030.10 → 28.30
Reconstruction weight0.1 → 527.30 → 30.10
Completion budget0.05 → 0.1028.50 → 30.10

These variations exceed the reported performance gains in many framework–dataset settings. However, the paper does not adequately explain the underlying causes or verify whether the default hyperparameters generalize across different frameworks and datasets. Therefore, the stability of hyperparameter selection and the generalizability of CrossAug across graph construction settings remain insufficiently established.

Questions for authors

See Weakness.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1YwN W1

We thank the reviewer for the detailed reading and for recognizing our motivation and the GNN–LLM method design. We respond to the five weaknesses in turn.

W1: Novelty of the scoring framework; KGC methods and simpler heuristics.

We agree that relational message passing, mean pooling, MLPs, and pairwise ranking are established components, and we do not claim novelty for them. The contribution is the formulation and supervision of a different decision: where to complete a graph index when neither the missing entities nor the missing relations are known, under a fixed LLM budget. Two-stage fact masking supplies relative-missingness supervision, and incorrect-fact reconstruction keeps node and edge counts matched within its ranking pairs, so the scorer is not rewarded for size or density alone in these pairs.

The cited KGC methods (GraIL, NBFNet, KGC-STA, DrKGC) solve a different task. Each predicts or scores a missing link conditioned on a candidate triple or an entity–relation query: GraIL builds an enclosing subgraph around candidate endpoints, NBFNet aggregates paths from a source entity under a query relation, KGC-STA augments an enclosing subgraph for a candidate triple, and DrKGC retrieves a query-specific subgraph starting from an incomplete triple. CROSSAUG operates one step earlier, before any candidate exists, and its output is a set of regions passed to an LLM that extracts new triples from evidence. Adapting a link predictor to this role would require generating and scoring candidate triples across the graph, then aggregating link scores into region scores. This adds a candidate-search and link-scoring stage before region selection, introducing computational overhead that CROSSAUG avoids by directly scoring sampled subgraphs. Off-the-shelf link predictors therefore do not directly solve the budgeted region-selection problem considered here. We will clarify this task and computational distinction in the revised paper.

The second part of the concern is fair, and we address it with a separate comparison experiment. We compare CROSSAUG with Low-Density and Low-Degree, which replace the GNN score with fact-edge density and mean internal fact degree, and a Chunk-wise Second Pass, which re-extracts each chunk from its own text and existing triples:

MethodMuSiQue EMMuSiQue F1LiteraryQA EMLiteraryQA F1
Base27.9039.9517.6032.72
Low-Density27.7039.7518.2634.09
Low-Degree27.8040.0118.4833.87
Chunk-wise second pass27.9039.8017.3832.99
CrossAug30.1041.7718.7834.31

At an equal number of calls, the heuristics help on LiteraryQA but not on MuSiQue, where Low-Density lowers both metrics and Low-Degree trades EM for F1; CROSSAUG is higher than both on both datasets. The second pass does not improve both metrics over Base on either dataset; we report its token consumption in our response to W4 below. Trace inspection illustrates a limitation of reading chunks in isolation in the MuSiQue case we examined: the second pass adds no fact to the museum passage, while CROSSAUG recovers the museum–Da Nang relation and thereby retrieves the regional evidence needed for the answer. These results support the scorer over the simple alternatives tested.

Table 4 also includes a semantic-only ablation that removes graph message passing while retaining semantic features. It achieves 28.10/40.03 on MuSiQue and 17.82/32.90 on LiteraryQA, compared with 30.10/41.77 and 18.78/34.31 for full CROSSAUG (EM/F1 in %). The performance gap supports the benefit of incorporating graph message passing into the scorer.

AuthorsOpenReview ↗

Response to Reviewer 1YwN W2

W2: Applicability, gain size, and indexing cost.

Applicability. CROSSAUG targets graphs built from LLM-extracted triples. As noted in Sections 2 and 4.1, LinearRAG and E²GraphRAG belong to the lightweight family that builds relation-free graphs with NLP tools; they are outside its scope, and we report LinearRAG only for context.

Gain size. For the main Base-versus-CROSSAUG experiments, the table below reports mean improvements over Base (percentage points) and two-sided t-test p-values using three seeds per method in each framework–dataset setting. Bold marks p<0.05p<0.05.

FrameworkMuSiQue EMMuSiQue F12Wiki EM2Wiki F1HotpotQA EMHotpotQA F1LiteraryQA EMLiteraryQA F1
HippoRAG2+1.60 / 0.010+1.45 / 0.016+0.34 / 0.024+0.30 / 0.160+0.50 / 0.038+0.29 / 0.302+0.76 / 0.082+1.15 / 0.045
LightRAG+0.44 / 0.247+0.27 / 0.438+0.73 / 0.126+0.47 / 0.070+0.40 / 0.444+0.25 / 0.600+0.17 / 0.372+0.22 / 0.282
GFM-RAG+1.10 / 0.044+0.80 / 0.144+1.43 / 0.004+1.59 / 0.002+0.34 / 0.121+0.86 / 0.010+1.34 / 0.003+1.06 / 0.009

Eleven of the 24 comparisons reach p<0.05p<0.05, 11 of 16 for HippoRAG2 and GFM-RAG. With only three seeds, we treat these tests as supporting evidence, not as proof that every individual gain is significant. Two further observations point to a systematic improvement. First, the mean improvement is positive in all 24 comparisons. Second, the larger gains appear where cross-chunk evidence matters most, namely MuSiQue and the long-document LiteraryQA. On 2Wiki and HotpotQA, most questions can be answered from short, self-contained Wikipedia passages. We will describe the improvements as consistent but heterogeneous in size, and report this significance analysis in the paper.

As a complementary analysis of the overall effect, we first average the three seeds within each framework–dataset setting and then use the 12 settings as paired units. We conduct a two-sided t-test separately for EM and F1:

MetricMean improvement (percentage points)Two-sided p-value
EM+0.76110.000220
F1+0.72560.000369

Cost. CROSSAUG's additional usage is mainly prompt tokens, because each completion request carries several evidence passages and the local graph. We also report the cost of the chunk-wise second pass for comparison.

We estimate API costs using the published Qwen3-32B International rates of USD 0.16 per million input tokens and USD 0.64 per million output tokens. Each cost is calculated as 0.16 × Prompt (M) + 0.64 × Completion (M), without discounts or free quotas, and rounded to two decimal places. The Base + rows include the initial Base indexing stage.

DatasetStagePrompt (M)Completion (M)Total (M)Estimated API cost (USD)
MuSiQueBase indexing10.83927.21938.05819.15
MuSiQueCROSSAUG, additional11.7984.80216.6004.96
MuSiQueBase + CROSSAUG22.63732.02154.65824.12
MuSiQueBase + chunk-wise second pass21.00741.26562.27229.77
LiteraryQABase indexing17.83923.02540.86417.59
LiteraryQACROSSAUG, additional45.0245.98651.01111.03
LiteraryQABase + CROSSAUG62.86329.01191.87528.63
LiteraryQABase + chunk-wise second pass30.44541.13971.58431.20

Two facts bear on whether this overhead is justified. First, CROSSAUG achieves higher EM and F1 than the chunk-wise second pass at a lower estimated API indexing cost on both datasets: USD 24.12 versus USD 29.77 on MuSiQue and USD 28.63 versus USD 31.20 on LiteraryQA. On LiteraryQA, this lower cost holds despite higher total token usage, because CROSSAUG generates fewer output tokens, which are priced four times as high as input tokens. The QA comparison is reported in W4. Second, the cost is paid once, offline: the GNN and the completion LLM are never invoked at query time, and the augmented index is reused as a static index. The amortized overhead per query is therefore the one-time cost divided by the number of queries the index serves, and reporting a specific value would require assuming a query workload that these benchmarks do not define. We therefore report the one-time overhead.

Graph contamination. The 84.35% factual correctness rate corresponds to an estimated 15.65% factual error rate in the sampled triples. Positive aggregate QA gains do not rule out harm on individual questions, and we will explicitly report this error rate and the associated risk of downstream error propagation.

AuthorsOpenReview ↗

Response to Reviewer 1YwN W3

W3: Validity of synthetic missingness and direct detection evaluation.

Section 4.4 provides direct evidence about what CROSSAUG recovers. Among 497 valid annotations of sampled, factually correct triples, 91.5% correspond to identifiable real-world omission patterns — missing relations or entities, partial extraction, alias/coreference failures, and relation-type refinement—covering several of the extraction failures highlighted by the reviewer. These findings provide evidence that CROSSAUG recovers genuine extraction omissions beyond its synthetic training patterns.

Precision@K and Recall@K for missing subgraphs require gold annotations of which regions of a real index are incomplete, and Recall@K additionally requires an exhaustive annotation of omissions across the candidate pool. To our knowledge, no annotation set of this kind exists for these corpora. Constructing an exhaustive reference for Recall@K would require region-level labelling, whereas CROSSAUG obtains its training supervision without such labels. We therefore provide the triple-level validation in Section 4.4 as complementary evidence of real-world omission recovery, together with downstream QA evaluation. We note explicitly that the 91.5% figure is a triple-level category distribution, not region-selection precision, and we do not claim synthetic and natural missingness share an identical distribution.

Reviewer_DHFPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes CrossAug, an offline graph augmentation method for LLM-based GraphRAG that recovers relations missed by chunk-local extraction. A GNN is trained with self-supervised ranking on masked and corrupted subgraphs to score how incomplete each subgraph is, and an LLM then completes the top-scoring subgraphs with evidence-grounded triples. CrossAug is applied to LightRAG, HippoRAG2 and GFM-RAG on four QA benchmarks and improves all 24 framework–dataset–metric comparisons.

Critical strengths

S1: The paper identifies a real limitation of chunk-local extraction and proposes an offline, plug-in augmentation that applies to multiple GraphRAG frameworks. S2: The self-supervised ranking design is thoughtful, and the random-selection ablation under the same budget shows that the GNN scorer is useful.

Critical weaknesses

W1: The gains are small and often within noise. W2: Hyperparameters and ablations are tuned on the MuSiQue evaluation questions. W3: The paper does not show that the added triples are truly cross-chunk.

Questions for authors

Q1: What fraction of added triples cite more than one chunk and cannot be extracted from any single cited chunk? Q2: How does CrossAug compare with using larger or overlapping chunks under the same token budget?

3 comments
AuthorsOpenReview ↗

Response to Reviewer DHFP W1,W2

We thank the reviewer for recognizing the motivation, the plug-in offline design, and the self-supervised ranking objective. We address each point below.

W1: Small gains and statistical uncertainty.

We agree that several gains are small, particularly on 2Wiki and HotpotQA and for LightRAG. The table below summarises the main Base-versus-CrossAug experiments, reporting mean improvements over Base (percentage points) and two-sided t-test p-values using three seeds per method in each framework–dataset setting. Bold marks p<0.05p<0.05.

FrameworkMuSiQue EMMuSiQue F12Wiki EM2Wiki F1HotpotQA EMHotpotQA F1LiteraryQA EMLiteraryQA F1
HippoRAG2+1.60 / 0.010+1.45 / 0.016+0.34 / 0.024+0.30 / 0.160+0.50 / 0.038+0.29 / 0.302+0.76 / 0.082+1.15 / 0.045
LightRAG+0.44 / 0.247+0.27 / 0.438+0.73 / 0.126+0.47 / 0.070+0.40 / 0.444+0.25 / 0.600+0.17 / 0.372+0.22 / 0.282
GFM-RAG+1.10 / 0.044+0.80 / 0.144+1.43 / 0.004+1.59 / 0.002+0.34 / 0.121+0.86 / 0.010+1.34 / 0.003+1.06 / 0.009

Eleven of the 24 comparisons reach p<0.05p<0.05, 11 of 16 for HippoRAG2 and GFM-RAG. With only three seeds, we treat these tests as supporting evidence, not as proof that every individual gain is significant. For example, HippoRAG2 gains 1.60 EM and 1.45 F1 points on MuSiQue, while GFM-RAG gains 1.34 EM and 1.06 F1 points on LiteraryQA. We will describe the improvements as consistent but heterogeneous in size, and report this significance analysis in the paper.

As a complementary analysis of the overall effect, we first average the three seeds within each framework–dataset setting and then use the 12 settings as paired units. We conduct a two-sided t-test separately for EM and F1:

MetricMean improvement (percentage points)Two-sided p-value
EM+0.76110.000220
F1+0.72560.000369

W2: Hyperparameters and ablations tuned on MuSiQue evaluation questions.

We thank the reviewer for highlighting the limited coverage of the original sensitivity analysis. We supplement the MuSiQue study with sensitivity experiments on LiteraryQA below, complementing the two-dataset ablations in Table 4. All sensitivity experiments use seed 42. These additional experiments examine robustness beyond MuSiQue.

ParameterValueMuSiQue EMMuSiQue F1LiteraryQA EMLiteraryQA F1
N_walk1029.0040.7518.4834.14
N_walk2030.1041.7718.7834.31
N_walk3029.3041.0618.3433.66
N_walk4028.6040.3718.3433.89
λ128.1039.6818.3434.09
λ530.1041.7718.7834.31
λ1029.8041.3918.5634.45
λ1529.6040.7918.4834.23
λ2030.1041.4818.4834.29
λ10029.6041.4218.8534.38
Budget ratio0.0528.5039.9417.3832.72
Budget ratio0.1030.1041.7717.7533.40
Budget ratio0.1528.9040.3018.6334.04
Budget ratio0.2029.4040.5818.7834.31

In the reported ranges Nwalk∈[10,40]N_{\text{walk}}\in[10,40], λ=5\lambda=5–100, and budget ratio 0.100.10–0.200.20, every setting keeps both EM and F1 above the corresponding Base on both datasets. At budget 0.05, MuSiQue reaches 28.50/39.94: EM is 0.60 points above Base, while F1 is 0.01 points lower. LiteraryQA reaches 17.38/32.72, with EM 0.22 points below Base and F1 unchanged. A plausible explanation is that completing fewer regions yields a smaller augmentation benefit that may be offset by performance fluctuations at this low budget.

AuthorsOpenReview ↗

Response to Reviewer DHFP W3,Q1,Q2

W3 and Q1: Do the added triples actually require multiple chunks?

The fraction of unique HippoRAG2 completion records citing more than one chunk is:

DatasetMulti-citation fraction
MuSiQue10.63%
2Wiki3.99%
HotpotQA3.63%
LiteraryQA34.52%

Two factors help explain these fractions. First, CROSSAUG's completion prompt does not restrict the LLM to generating only cross-chunk facts. Second, the gap between benchmarks follows from the corpora: the multi-hop sets consist of short, largely self-contained Wikipedia passages, whereas LiteraryQA consists of long narratives in which characters and references recur across many chunks.

However, these fractions do not conflict with CROSSAUG's motivation to recover relations missed by chunk-local extraction. The downstream impact of recovered facts is not proportional to their count: the fine-grained analysis and case studies show that added facts can supply missing evidence, improve the ranking of existing evidence, and directly provide answer-relevant relations. To examine whether repeating chunk-local extraction can achieve similar gains, we conduct a separate comparison experiment with a chunk-wise second pass, which re-extracts every chunk separately. The table reports results for seed 42:

MethodMuSiQue EM / F1LiteraryQA EM / F1
Base27.90 / 39.9517.60 / 32.72
Chunk-wise second pass27.90 / 39.8017.38 / 32.99
CROSSAUG30.10 / 41.7718.78 / 34.31

The second pass does not improve both EM and F1 over Base on either dataset. Trace inspection of the MuSiQue case we examined illustrates a limitation of reading chunks in isolation: the second pass adds no fact to the museum passage, while CROSSAUG recovers the museum–Da Nang relation, which surfaces the regional evidence needed to answer. This pattern may be better explained by reading selected passages jointly than by the additional extraction calls themselves.

Q2: Larger chunks or greater overlap under the same token budget.

The three multi-hop benchmarks provide pre-segmented passages rather than documents we can re-chunk, so we conduct this separate comparison experiment on the same 50 LiteraryQA books (1,358 questions). On LiteraryQA we independently double the chunk size from 500 to 1,000 words with overlap fixed at 80, or double the overlap from 80 to 160 words with chunk size fixed at 500. All QA results use seed 42, with the original Base achieving 17.60/32.72. Token counts are in millions.

MethodChunk / overlap (words)EM (%)F1 (%)Prompt(M)Completion(M)Total(M)
Original Base500 / 8017.6032.7217.83923.02540.864
Larger-chunk Base1000 / 8016.7933.5213.50511.04124.547
Larger-overlap Base500 / 16016.7933.3622.05528.32650.382
CROSSAUG (additional cost)500 / 8018.7834.3145.0245.98651.011

Both modifications lower EM by 0.81 points while raising F1 by 0.80 and 0.64 points, respectively, whereas CROSSAUG improves both metrics. Reallocating the budget to overlap is less effective per token in these settings: greater overlap consumes more than twice the tokens of larger chunks and yields the same EM with lower F1. The comparison is not exactly token-matched, since CROSSAUG's total indexing cost is higher; what the experiments show is that the same or greater expenditure directed at chunk boundaries does not reproduce the gain, which is consistent with the missing relations lying outside the reach of any single chunk regardless of its size.

Thanks for the response. I will keep my score.

Reviewer_NtZ4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the task of graphrag. Specifically, it addresses the potential missing cross-chunk relations during graph construction. To solve this problem, the paper proposes the CrossAug method. The goal of this method is to find potential subgraphs in the already constructed graph and add missing cross-chunk relations. During training, CrossAug constructs training examples for a GNN by modifying the existing graph and trains the GNN to learn to rank and find subgraphs with missing relations. During inference, this method selects subgraphs based on the GNN-predicted scores and uses an LLM to complete missing relations. This augmented graph is then plugged into subsequent existing GraphRAG frameworks for evaluation. The evaluation shows that the proposed method can bring some improvements.

Critical strengths

I think one critical strength of this paper is the method itself. It presents a complete approach to address the missing cross-chunk relations that may exist in GraphRAG. The proposed pipeline includes dataset generation, GNN model training, and the final inference implementation. I also appreciate the training method and the way this paper uses a small model to strategically identify potential problems in the existing graph and call an LLM to complete it.

Critical weaknesses

Regarding the experiments, I feel one gap in this paper is that it only proves the proposed GNN training and inference method is better than random selection, but it doesn't demonstrate the actual necessity of this GNN. Because besides random selection or a GNN, we have many other options. For example, using the additional computational resources used by crossAug to directly input the cross-chunk text back into the graph construction LLM might directly yield a better graph with more cross-chunk relations. So I think this paper should include additional experiments to demonstrate why many other intuitive and simple methods are not good enough and show that the authors are not over-engineering the problem here.

Questions for authors

Besides using GNN and random selection methods, are there any other simple methods used/tested to find potentially missing cross-chunk relations?

3 comments
AuthorsOpenReview ↗

Response to Reviewer NtZ4

We thank the reviewer for the positive assessment of the method and of using a small learned model to direct LLM completion. We agree that comparing only against random selection cannot show whether a simpler rule would do as well. We therefore present a separate comparison experiment below.

W1: Is the GNN necessary, or would a simpler use of the extra budget work as well?

The reviewer is right that feeding cross-chunk text back to the graph-construction LLM is a natural use of additional indexing budget. The practical challenge is deciding which chunks to read together. A corpus admits a combinatorial number of chunk combinations, and choosing among them is exactly what CrossAug's subgraph sampling and scoring do: the selected subgraph determines which evidence passages are sent to the LLM. The relevant question is therefore whether a simpler selector, or a selection-free re-extraction, can replace the learned scorer. We conduct a separate comparison experiment with three alternatives: Low-Density, which prioritizes sampled subgraphs with a lower fraction of entity pairs connected by fact edges; Low-Degree, which prioritizes sampled subgraphs with fewer fact neighbors per entity on average; and Chunk-wise Second Pass, which asks the same LLM to recover omissions from each original chunk using its known entities and triples.

Results with HippoRAG2 (EM/F1 in %):

MethodMuSiQue EMMuSiQue F1LiteraryQA EMLiteraryQA F1
Base27.9039.9517.6032.72
Low-Density27.7039.7518.2634.09
Low-Degree27.8040.0118.4833.87
Chunk-wise second pass27.9039.8017.3832.99
CrossAug30.1041.7718.7834.31

These results show that the benefit of simple structural statistics is dataset-dependent: both heuristics improve over Base on LiteraryQA but not on MuSiQue, where Low-Density lowers both metrics and Low-Degree trades a 0.10 EM loss for a 0.06 F1 gain. CrossAug outperforms both heuristics on both datasets.

Table 4 also includes a semantic-only ablation that removes graph message passing while retaining semantic features. It achieves 28.10/40.03 on MuSiQue and 17.82/32.90 on LiteraryQA, compared with 30.10/41.77 and 18.78/34.31 for full CrossAug (EM/F1 in %). This comparison supports the contribution of graph message passing beyond semantic features alone.

Together with the existing random-selection ablation at an equal number of calls (MuSiQue: 28.40/39.95 → 30.10/41.77; LiteraryQA: 18.19/33.17 → 18.78/34.31), these results indicate that where the budget is spent matters, and that the learned scorer chooses better than the simple rules we tested. We will add this comparison experiment to the revision.

Thanks for the reply and the experiments. They address some of my concerns above, but I would still like to encourage the authors to add the direct cross-chunk baseline I mentioned in my review. I will keep my score since it is already a positive score.

We thank the reviewer for taking the time to consider our rebuttal and for their continued support of our work.

Comments on the paper

2 comments

Dear Reviewers,

Please take a moment to read the authors’ rebuttal and participate in the discussion.

Thanks,

AC

Dear Reviewers,

I hope this message finds you well. As the discussion period is nearing its end with less than three days remaining, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.