acceptodds
Under review as a conference paper at ICLR 2027

REORG: Relation-Guided Evidence Organization for Retrieval-Augmented Generation

Abstract

Post-retrieval context construction for retrieval-augmented generation (RAG) must preserve connections among facts, not merely select relevant passages. Retrieved passages can interleave shared information with unique, complementary facts, so passage-level selection or independent compression may discard evidence whose value emerges through joint reading. We call this problem Evidence Entanglement and propose REORG (Relation-Guided Evidence Organization), which establishes joint-reading scopes before removing content. REORG extracts local cross-passage attention associations under complementary passage orders and aggregates them into sparse sentence and passage graphs to form overlapping reading groups. These cues guide comparison rather than directly determine semantic relation types. Within each group, an organizer uses source text and association cues to jointly select evidence, returning only sentence identifiers and relation and role labels. Programmatic source recovery, deduplication, and rendering under a complete answer-input budget separate evidence selection from factual wording. Across the reported results on HotpotQA, 2WikiMQA, MuSiQue, and NQ-Open, REORG records unweighted mean EM/F1 of 36.39/46.06, exceeding CPC, the strongest baseline by these averages, by 2.36/2.41 points. Our code is available at https://anonymous.4open.science/r/REORG-2D15.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-Augmented Generation, Evidence Organization, Cross-Document Reasoning, Context Construction, Redundancy Reduction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PY3fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles post-retrieval context construction for retrieval-augmented generation (RAG). The authors name a specific regime they call Evidence Entanglement, where retrieved passages interleave shared and complementary facts, so whole-passage selection or independent compression can discard evidence whose value emerges only through joint reading. The propsed method REORG has three components: Attention-based Relation Discovery, Overlapping Evidence Organization and Relation-guided Integration. REORG attains consistent performance gain over four datasets.

Critical strengths

1 The Evidence Entanglement is intuitive and does identify a real failure mode of passage-level selection / independent compression. The two motivations M#1 and M#2 (lines 79–91) map directly to the two technical stages, which keeps the paper easy to follow.

2 REORG is best in EM and F1 on HotpotQA, 2WikiMQA, MuSiQue, and NQ-Open. Importantly the leading comparator differs across datasets so the fact that one single pipeline beats all three is a non-trivial result.

Critical weaknesses

1 The core design of this paper (identify cross-document associations, cluster into groups, extract joint evidence) is a recombination of ideas from prior works. The genuinely new elements are (i) using attention from two reversed orders as the association signal and (ii) using overlapping (SLPA-style) rather than disjoint groups.

Table 3 actually argues against the primacy of overlap:

w/o group overlap (A1): −0.26 F1 w/o Organizer relation hints (A2): −0.57 F1 w/o Answer relation structure (A3): −0.67 F1 w/o group-wise organization (A4): −0.99 F1

So the component most tied to the paper's "relation-first" framing (A1, overlapping groups) contributes the smallest ablation gap, while the biggest gap (A4) is simply the choice to process passages in groups rather than feeding all candidates in a single Organizer call. That makes the paper's conceptual contribution incremental.

2 Each diagnostic experiment uses a different single dataset, so the robustness is not actually tested. The Retrieval-scale sweep (Table 2) is tested on HotpotQA only. the component ablation (Table 3) is tested on NQ-Open only. Parameter sensitivity is also tested on one dataset only (F1 reference value 51.96 matches the NQ-Open row of Table 1; the panels say nothing about Hotpot / 2Wiki / MuSiQue). For a paper whose main empirical claim is multi-dataset generality, it is unsatisfying that every supporting analysis is single-dataset.

Questions for authors

Please refer to the weakness section.

4 comments
AuthorsOpenReview ↗

Response to [W1. Method Contribution and the Role of Overlap]

Thank you for asking us to make the contribution and the role of overlap more precise. REORG's contribution is the relation-first evidence-construction design: establish joint-reading scopes from cross-passage associations before deciding which evidence to retain. Overlap is one part of this design, working together with relation-guided selection and source-grounded rendering.

Design rationale.

Evidence Entanglement describes the setting in which shared background and unique complementary facts are interleaved across retrieved passages. REORG addresses it by using attention associations from complementary passage orders to construct overlapping groups. The Organizer then jointly reads the sources in each group and returns sentence identifiers and relation labels. Programmatic recovery renders the selected source text, keeping evidence selection separate from factual wording.

Refiner already combines cross-source extraction with organization, and EDC2-RAG already performs joint processing within document clusters. REORG positions its contribution in the scope and order of these decisions: local associations establish overlapping joint-reading contexts while the source text is still available, the Organizer selects sentence references with relation cues, and rendering recovers those selected sources. The contribution is this coordinated relation-first construction.

The benefit sought from overlap is specific: the same passage can support more than one evidence combination. This is particularly relevant to multi-hop questions, where a useful connection may involve information spread across several passages. We therefore expanded the component comparisons beyond NQ-Open.

Evidence on multi-hop tasks.

Complete component results (A0-A4). A0 is the reported full-method reference. The table gives EM/F1 (%) for every dataset and the unweighted four-dataset mean.

Setting (EM/F1, %)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
A0 Full REORG42.39 / 54.9249.69 / 55.2612.83 / 22.1040.64 / 51.9636.39 / 46.06
A1 Without group overlap38.58 / 52.3546.69 / 53.489.64 / 19.2139.92 / 51.7033.71 / 44.19
A2 Without Organizer relation hints38.58 / 49.5645.66 / 52.0110.22 / 16.9939.70 / 51.3933.54 / 42.49
A3 Without Answer relation block38.73 / 52.6047.15 / 53.9110.80 / 19.6039.86 / 51.2934.14 / 44.35
A4 One Organizer call for all top-100 passages39.57 / 53.7442.76 / 49.7411.38 / 19.1539.53 / 50.9733.31 / 43.40

The table summarizes F1 drops relative to full REORG. The multi-hop mean covers HotpotQA, 2WikiMQA, and MuSiQue.

AblationMean multi-hop F1 drop (pp)NQ-Open F1 drop (pp)
A1 Without group overlap2.410.26
A2 Without Organizer relation hints4.570.57
A3 Without Answer relation block2.060.67
A4 One Organizer call for all top-100 passages3.220.99

Removing overlap lowers F1 by 2.57, 1.78, and 2.89 points on the three multi-hop datasets, averaging 2.41 points, compared with 0.26 on NQ-Open. Removing Organizer hints and the Answer relation block also gives larger multi-hop mean gaps, at 4.57 and 2.06 points. This supports examining relation information at both evidence selection and answer generation.

Complete grouping controls (G1-G3). The table reports the full-method reference and all grouping alternatives, with EM/F1 (%) and equal-weight four-dataset means.

Setting (EM/F1, %)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
A0 Full REORG42.39 / 54.9249.69 / 55.2612.83 / 22.1040.64 / 51.9636.39 / 46.06
G1 Random grouping39.43 / 53.2046.79 / 53.8511.92 / 18.5639.50 / 51.9234.41 / 44.38
G2 Average-linkage hierarchical clustering39.01 / 53.2847.91 / 54.4510.22 / 19.4339.17 / 51.6734.08 / 44.71
G3 Top-k neighborhood grouping40.70 / 54.1149.13 / 55.2210.80 / 17.9339.83 / 51.2735.12 / 44.63

Grouping controls strengthen this picture: full REORG averages 46.06 F1, compared with 44.38 for random grouping, 44.71 for average-linkage clustering, and 44.63 for top-k neighborhoods. REORG's 1.35-point gap over the strongest grouping control links the evaluation to how reading scopes are formed. A4 represents the broader change from group-wise organization to one call over all top-100 passages, including its changed processing scope and call count.

Overall, the full component and grouping results support the coordinated use of joint-reading scopes, relation-aware selection and source-preserving rendering. They give a more informative account of the design than a ranking of the original NQ-Open ablation gaps.

AuthorsOpenReview ↗

Response to [W2. Diagnostic Coverage]

Thank you for suggesting broader diagnostic coverage. We expanded the component analysis and added comparisons across HotpotQA, 2WikiMQA, MuSiQue, and NQ-Open. Averages weight all four datasets equally.

Evaluation coverage

EvaluationDataset coverage
A1-A4 component ablationsAll four datasets
G1-G3 grouping controlsAll four datasets
S1-S3 attention controlsAll four datasets
Five requested baselinesAll four datasets
Below-2k Refiner comparisonAll four datasets
Retrieval Top-K sweepHotpotQA; REORG plus eight baselines
Parameter-sensitivity sweepsOne dataset

Component ablations (A0-A4, with A0 as full REORG)

Setting (EM/F1, %)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
A0 Full REORG42.39 / 54.9249.69 / 55.2612.83 / 22.1040.64 / 51.9636.39 / 46.06
A1 Without group overlap38.58 / 52.3546.69 / 53.489.64 / 19.2139.92 / 51.7033.71 / 44.19
A2 Without Organizer relation hints38.58 / 49.5645.66 / 52.0110.22 / 16.9939.70 / 51.3933.54 / 42.49
A3 Without Answer relation block38.73 / 52.6047.15 / 53.9110.80 / 19.6039.86 / 51.2934.14 / 44.35
A4 One Organizer call for all top-100 passages39.57 / 53.7442.76 / 49.7411.38 / 19.1539.53 / 50.9733.31 / 43.40

Grouping controls (G1-G3)

Setting (EM/F1, %)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
A0 Full REORG42.39 / 54.9249.69 / 55.2612.83 / 22.1040.64 / 51.9636.39 / 46.06
G1 Random grouping39.43 / 53.2046.79 / 53.8511.92 / 18.5639.50 / 51.9234.41 / 44.38
G2 Average-linkage hierarchical clustering39.01 / 53.2847.91 / 54.4510.22 / 19.4339.17 / 51.6734.08 / 44.71
G3 Top-k neighborhood grouping40.70 / 54.1149.13 / 55.2210.80 / 17.9339.83 / 51.2735.12 / 44.63

Attention input controls (S1-S3)

Setting (EM/F1, %)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
A0 Full REORG42.39 / 54.9249.69 / 55.2612.83 / 22.1040.64 / 51.9636.39 / 46.06
S1 Forward only, without leading question prefix38.16 / 52.0847.62 / 54.799.64 / 18.9937.17 / 51.2333.15 / 44.27
S2 Forward only, with leading question prefix40.27 / 53.7747.34 / 53.9812.49 / 20.2539.17 / 51.4134.82 / 44.85
S3 Forward + reverse, without leading question prefix40.14 / 53.7547.25 / 53.8811.38 / 20.9739.14 / 51.1534.48 / 44.94

Added baselines

Method (EM/F1, %)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
REORG42.39 / 54.9249.69 / 55.2612.83 / 22.1040.64 / 51.9636.39 / 46.06
LongLLMLingua39.10 / 51.5842.76 / 48.8511.05 / 21.0236.34 / 47.8232.31 / 42.32
LLMLingua-233.22 / 44.2533.16 / 37.759.23 / 17.8632.88 / 43.9927.12 / 35.96
RECOMP38.33 / 49.9345.34 / 50.319.81 / 18.4937.42 / 48.5232.73 / 41.81
Refiner39.99 / 49.5146.02 / 51.9010.34 / 20.1636.34 / 47.5333.17 / 42.28
EDC2-RAG41.12 / 53.8746.76 / 52.6812.00 / 21.5938.20 / 50.4034.52 / 44.64

Compression factors

Method (CR)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
REORG5.1x6.4x4.9x3.8x5.1x
LongLLMLingua5.5x5.5x5.6x5.5x5.5x
LLMLingua-24.8x4.9x4.9x4.8x4.9x
RECOMP5.3x5.3x5.2x5.5x5.3x
Refiner4.8x4.6x4.6x4.7x4.7x
EDC2-RAG5.1x5.2x5.5x5.2x5.3x

Common compressed-context cap (below 2k tokens)

Method (EM/F1, %)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
Refiner38.70 / 47.3644.49 / 50.0710.22 / 18.0632.69 / 43.4231.53 / 39.73
REORG41.50 / 52.8047.55 / 52.8511.75 / 20.1339.06 / 49.9734.97 / 43.94

REORG outperforms all three grouping controls on every dataset. Removing overlap has a larger effect on each multi-hop dataset than on NQ-Open. Below 2k tokens, REORG outperforms Refiner in both EM and F1 on every dataset.

Time and compute limited the additional sweeps. Retrieval Top-K results still cover HotpotQA only, and parameter-sensitivity results still cover one dataset.

AuthorsOpenReview ↗

Summary of Responses to Reviewer PY3f

Thank you for helping us clarify the contribution and diagnostic coverage. The updated responses connect the relation-first design to the additional multi-dataset evidence.

  • W1: Contribution and overlap. REORG establishes overlapping joint-reading scopes before source-sentence selection and then preserves the selected text and its relations. Removing overlap gives a 2.41-point mean multi-hop F1 drop, compared with 0.26 on NQ-Open. REORG also exceeds the strongest grouping control by 1.35 average F1 points. The evidence supports the coordinated reading-scope and relation-information design.
  • W2: Coverage. Component, grouping and attention controls now cover all four datasets, together with the five-baseline and below-2k Refiner comparisons. The existing retrieval-depth study covers HotpotQA, while sensitivity remains single-dataset. Our responses distinguish those original sweep scopes from the expanded diagnostics.

Manuscript revisions. Contribution clarifications in the abstract (p. 1), Introduction (p. 3) and Section 2.2 (pp. 3-4) are dark orange. The actual coverage statements in Sections 4.3 (p. 8), 4.5 and 5 (p. 9), with details in Appendices C.1 (p. 25) and C.3 (p. 27), use the same color. Shared experimental additions are purple: component EM/F1 in Table 3 (p. 8) and the complete component/grouping/attention results in Appendix C.2, Tables 10-12 (p. 26).

The Guideline for Reviewers in Appendix F (p. 32) gives the color key.

Overall, the manuscript now connects the clarified contribution to its supporting comparisons and states the evaluated coverage directly.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer PY3f

Thank you again for helping us clarify the contribution and evaluation. As the discussion period draws to a close, we would be grateful if you could review our updated responses and color-marked manuscript once more. We hope the clarified design and complete multi-dataset result tables help address the main concerns. We welcome further questions and appreciate your time.

Reviewer_ke2wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes REORG to address the organization of cross-document evidence in RAG. It extracts attention-based associations under two passage orders, constructs overlapping reading groups, and uses an organizer model to select sentence IDs and label relations. Selected sentences are recovered verbatim and assembled under an input budget. Across four QA benchmarks, the paper reports average EM/F1 improvements over CPC, alongside retrieval-scale experiments and component analyses.

Critical strengths
  1. Establishing joint-reading groups before removing evidence is a well-motivated approach to preserving complementary facts across documents. Overlapping groups allow a passage to contribute to multiple evidence connections.

  2. Selecting sentence identifiers and recovering source text programmatically provides clear provenance and avoids factual rewriting during compression. The reported improvements across four benchmarks make the approach promising.

Critical weaknesses
  1. The experiments do not convincingly isolate the proposed mechanism. \S4.4 explicitly describes the ablations as unpaired, descriptive comparisons, with F1 differences below one point and no uncertainty estimates. Moreover, these ablations use NQ-Open rather than the multi-hop tasks central to the motivation. Thus, the contribution of attention-guided overlapping groups remains unclear.

  2. Comparison fairness and reproducibility are insufficiently established. Methods use different retained lengths, total pipeline costs are unreported, and closely related restructuring methods such as Refiner and EDC2-RAG are discussed but not evaluated. The appendix also leaves the actual model checkpoints and evaluation sample counts unspecified. These gaps make the reported gains difficult to interpret and reproduce.

Questions for authors
  1. Can you specify the exact models, dataset splits, sample counts, and treatment of failed examples used for each reported comparison?

  2. Can paired ablations on a multi-hop dataset establish gains from attention hints and overlapping groups while controlling the organizer, input budget, and other settings?

  3. Does REORG retain its advantage over a strong restructuring baseline at comparable answer-input lengths, and what is its end-to-end processing cost?

7 comments
AuthorsOpenReview ↗

Response to [W1. Ablation Evidence on Multi-Hop Tasks]

Thank you for recommending evidence that is better aligned with the multi-hop motivation. We have expanded the component comparisons to all four datasets, enabling a direct view of how the observed changes differ between multi-hop QA and NQ-Open.

Expanded component evaluation.

A1 removes overlap, A2 removes Organizer relation hints, A3 removes the Answer relation block, and A4 replaces group-wise processing with a single Organizer call over all top-100 passages. A0 is the reported full-method reference. The table reports EM/F1 percentages and unweighted four-dataset averages.

Setting (EM/F1, %)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
A0 Full REORG42.39 / 54.9249.69 / 55.2612.83 / 22.1040.64 / 51.9636.39 / 46.06
A1 Without group overlap38.58 / 52.3546.69 / 53.489.64 / 19.2139.92 / 51.7033.71 / 44.19
A2 Without Organizer relation hints38.58 / 49.5645.66 / 52.0110.22 / 16.9939.70 / 51.3933.54 / 42.49
A3 Without Answer relation block38.73 / 52.6047.15 / 53.9110.80 / 19.6039.86 / 51.2934.14 / 44.35
A4 One Organizer call for all top-100 passages39.57 / 53.7442.76 / 49.7411.38 / 19.1539.53 / 50.9733.31 / 43.40

Removing overlap lowers F1 by 2.57, 1.78, and 2.89 points on HotpotQA, 2WikiMQA, and MuSiQue, averaging 2.41 points; the NQ-Open difference is 0.26. Removing Organizer hints produces drops of 5.36, 3.25, and 5.11 points on the same three multi-hop datasets, averaging 4.57 points, compared with 0.57 on NQ-Open. These larger observed gaps make the multi-hop setting informative for evaluating the two designs.

The result pattern also connects to their different functions. Overlap allows one passage to contribute to multiple joint-reading contexts. Organizer hints make the associations available during evidence selection within those contexts. The reported comparisons support evaluating both the reading scope and the relation information supplied to the Organizer.

Overall, the expanded component table makes the multi-hop pattern directly visible across all three reasoning benchmarks. It strengthens the connection between REORG's reading-scope and relation-information choices and the tasks that motivate those choices.

AuthorsOpenReview ↗

Response to [W2. Baseline Coverage and Comparison Conditions]

Thank you for highlighting the need to connect answer quality with length and processing cost. We address these aspects through the five-baseline comparison, a separate length-capped Refiner evaluation, and the available end-to-end timing results.

Baseline coverage.

The expanded main table includes all five requested methods. LongLLMLingua [1], LLMLingua-2 [2] and RECOMP [3] are grouped under compression; Refiner [4] and EDC2-RAG [5] are grouped with REORG under restructuring. Scores are EM/F1 percentages; Average is the unweighted four-dataset mean.

Method (EM/F1, %)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
REORG42.39 / 54.9249.69 / 55.2612.83 / 22.1040.64 / 51.9636.39 / 46.06
LongLLMLingua39.10 / 51.5842.76 / 48.8511.05 / 21.0236.34 / 47.8232.31 / 42.32
LLMLingua-233.22 / 44.2533.16 / 37.759.23 / 17.8632.88 / 43.9927.12 / 35.96
RECOMP38.33 / 49.9345.34 / 50.319.81 / 18.4937.42 / 48.5232.73 / 41.81
Refiner39.99 / 49.5146.02 / 51.9010.34 / 20.1636.34 / 47.5333.17 / 42.28
EDC2-RAG41.12 / 53.8746.76 / 52.6812.00 / 21.5938.20 / 50.4034.52 / 44.64

We now report the corresponding compression factors in the same main table. Average CR is a mean of dataset ratios, with the factor shown to one decimal place.

Method (CR)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
REORG5.1x6.4x4.9x3.8x5.1x
LongLLMLingua5.5x5.5x5.6x5.5x5.5x
LLMLingua-24.8x4.9x4.9x4.8x4.9x
RECOMP5.3x5.3x5.2x5.5x5.3x
Refiner4.8x4.6x4.6x4.7x4.7x
EDC2-RAG5.1x5.2x5.5x5.2x5.3x

REORG has higher EM/F1 than each added baseline on every dataset. Its average F1 exceeds EDC2-RAG, RECOMP, and Refiner by 1.43, 4.25, and 3.79 points, respectively. The comparison now includes both representative compression methods and the related restructuring methods identified in the review.

Evidence under a common length cap.

We also compared Refiner and REORG with compressed context below 2k:

Method (EM/F1, %)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
Refiner38.70 / 47.3644.49 / 50.0710.22 / 18.0632.69 / 43.4231.53 / 39.73
REORG41.50 / 52.8047.55 / 52.8511.75 / 20.1339.06 / 49.9734.97 / 43.94

REORG's average EM/F1 advantage is 3.44/4.21 percentage points, with higher scores on every dataset. This supports its effectiveness under a common compressed-context upper cap. The condition is a shared cap, rather than an assertion of identical realized complete-prompt lengths.

Available end-to-end timing.

The earlier operating-point measurements report the following seconds per question:

Method (seconds/question)HotpotQA2WikiMQAMuSiQueNQ-Open
REORG4.905.404.834.69
LongLLMLingua4.785.025.045.02
LLMLingua-23.583.583.583.58
EDC2-RAG8.228.198.078.63

REORG's reported time is lower than EDC2-RAG's on each dataset. LLMLingua-2 has the lowest time among these methods; the separate quality table gives REORG's higher answer scores.

Overall, the evidence now includes complete quality and compression-factor comparisons, the separate common-cap Refiner evaluation and the earlier per-question times. REORG's higher EM/F1 under the below-2k cap adds a direct restructuring comparison in a constrained-context setting.

References

[1] Jiang et al. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. ACL 2024.

[2] Pan et al. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. Findings of ACL 2024.

[3] Xu, Shi, and Choi. RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation. ICLR 2024.

[4] Li et al. Refiner: Restructure Retrieved Content Efficiently to Advance Question-Answering Capabilities. Findings of EMNLP 2024.

[5] Li et al. Efficient Dynamic Clustering-Based Document Compression for Retrieval-Augmented-Generation. Findings of EMNLP 2025.

AuthorsOpenReview ↗

Response to [Q1. Model and Evaluation Protocol]

Thank you for asking us to make the evaluation settings explicit. The final Answer model is Qwen3.8-27B. We have added the evaluation counts and failure-scoring rule and distinguish the final-answer model from the upstream attention and organization roles.

Models and generation settings.

The current experimental description specifies:

Role or settingConfiguration
Final Answer modelQwen3.8-27B
Attention adapterQwen3.5-family full-attention modules
Organizer / Answer generationTemperature 0.0; thinking disabled
Organizer output cap1,024 tokens
Answer output cap64 tokens
REORG complete Answer-input cap8,000 tokens

The complete Answer-input budget includes the question, instructions, formatting, source identifiers, evidence text and rendered relations. The below-2k Refiner comparison is a separate compressed-context condition; it is not the same quantity as the complete-prompt cap.

Evaluation population and scoring.

The reported evaluation populations are:

DatasetReported evaluation examples
HotpotQA7,405
2WikiMQA12,576
MuSiQue2,417
NQ-Open3,610

The setup uses the datasets' existing evaluation or test partitions. Every evaluated example remains in the metric denominator, and generation failures receive zero EM and zero F1. Cross-dataset averages weight the four dataset scores equally. This makes both the evaluation scale and treatment of failed generations explicit.

Overall, the model/settings and evaluation-count tables make the final-generation model, evaluation scale and scoring convention explicit. Keeping failed generations in the denominator clarifies the population represented by the reported EM/F1 scores.

Reviewer_3X3kOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

REORG addresses Evidence Entanglement, where redundant, shared, and complementary information is interleaved across retrieved passages. It uses attention to extract strong cross-passage token connections, aggregates them into sentence-level connections to construct a sparse sentence graph, and further aggregates these connections to construct a passage relation graph. A weighted variant of the speaker–listener label propagation algorithm (SLPA) is then used to form overlapping reading groups. Within each group, the Organizer reads the source text and returns source-sentence references, relation types, and evidence roles. The selected evidence is then deduplicated across groups and ordered before being passed to the answer model. REORG outperforms the reported baselines on HotpotQA, 2WikiMQA, MuSiQue, and NQ-Open. Its main contribution is this pipeline for selecting and compressing retrieved candidates, combining attention-based graph grouping with overlapping community detection.

Critical strengths

REORG combines attention-based graph grouping and overlapping community detection into a complete evidence organization pipeline. The evidence selected by the Organizer is recovered directly from the original sources, avoiding factual rewriting. REORG also outperforms the reported baselines on HotpotQA, 2WikiMQA, MuSiQue, and NQ-Open.

Critical weaknesses
  1. Sections 3.2 and 3.3 devote substantial space to attention extraction under two complementary input orders, sentence graph construction, passage graph construction, and the formation of overlapping reading groups. However, the experimental evidence is insufficient to establish the effectiveness of these components. Ablation studies should compare the proposed approach with simpler grouping strategies while keeping the Organizer fixed.

  2. REORG’s core contribution is relation-guided joint evidence organization, but the main experiments primarily compare it with context selection and compression methods, omitting representative compression baselines such as LongLLMLingua, LLMLingua-2, and RECOMP. The paper also discusses Refiner and EDC²-RAG, whose mechanisms are closely related to the proposed approach: Refiner jointly extracts and organizes content across sources, while EDC²-RAG performs joint compression within document clusters. Neither method is included in the main experimental comparison.

  3. The A1–A4 ablation studies concern designs motivated by cross-document complementarity, bridging facts, and overlapping reading groups, yet they are conducted on NQ. The cross-document multi-hop tasks in HotpotQA, 2WikiMQA, or MuSiQue would be better suited to evaluating the effectiveness of these designs.

Questions for authors
  1. Which model is used as the Answer model in the main experiments? Why were representative compression baselines such as LongLLMLingua, LLMLingua-2, and RECOMP not included in the comparison?

  2. How much does the pipeline from attention extraction under two complementary input orders to the formation of overlapping reading groups contribute to the overall performance gain? How can this contribution be verified?

10 comments
AuthorsOpenReview ↗

Response to [W1. Attention Extraction and Grouping Controls]

Thank you for recommending simpler grouping controls. We address this question by comparing REORG with three alternative grouping strategies across all four datasets. The results support the value of using cross-passage associations to decide which passages the Organizer should read together.

Comparison design and experimental evidence.

Grouping determines which source passages can be compared within an Organizer call. This matters for REORG because a passage may contain background shared with one source and a complementary fact that connects it to another. The grouping stage is designed to expose such connections before evidence selection.

G1 randomizes passage assignment while preserving group sizes and the overlap pattern, providing a reference for association-guided passage assignment. G2 uses average-linkage hierarchical clustering, and G3 forms top-k neighborhoods. They offer simpler alternatives to REORG's overlapping grouping construction. The table reports EM/F1 percentages; Average is the unweighted mean across the four datasets.

Setting (EM/F1, %)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
A0 Full REORG42.39 / 54.9249.69 / 55.2612.83 / 22.1040.64 / 51.9636.39 / 46.06
G1 Random grouping39.43 / 53.2046.79 / 53.8511.92 / 18.5639.50 / 51.9234.41 / 44.38
G2 Average-linkage hierarchical clustering39.01 / 53.2847.91 / 54.4510.22 / 19.4339.17 / 51.6734.08 / 44.71
G3 Top-k neighborhood grouping40.70 / 54.1149.13 / 55.2210.80 / 17.9339.83 / 51.2735.12 / 44.63

REORG has higher EM and F1 on every dataset than each of the three grouping controls. On HotpotQA, it reaches 42.39/54.92 EM/F1, exceeding G3 by 1.69/0.81 percentage points. Across the four datasets, REORG's average EM exceeds G3 by 1.27 points, while its average F1 exceeds G2 by 1.35 points; G3 and G2 are the strongest controls for their respective averages. Relative to random grouping, the average F1 gap is 1.68 points.

Complete attention-input controls (S1-S3). The table includes the full-method reference and every input condition, with EM/F1 (%) on all four datasets and their unweighted means.

Setting (EM/F1, %)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
A0 Full REORG42.39 / 54.9249.69 / 55.2612.83 / 22.1040.64 / 51.9636.39 / 46.06
S1 Forward only, without leading question prefix38.16 / 52.0847.62 / 54.799.64 / 18.9937.17 / 51.2333.15 / 44.27
S2 Forward only, with leading question prefix40.27 / 53.7747.34 / 53.9812.49 / 20.2539.17 / 51.4134.82 / 44.85
S3 Forward + reverse, without leading question prefix40.14 / 53.7547.25 / 53.8811.38 / 20.9739.14 / 51.1534.48 / 44.94

The attention controls complement this grouping comparison. Adding the reverse order without the leading question prefix increases average F1 by 0.67 points, with positive changes on HotpotQA and MuSiQue and negative changes on 2WikiMQA and NQ-Open. Adding the question prefix in the forward-only setting increases average F1 by 0.58 points. These observations distinguish the association-extraction choices from the grouping choices, and show why both deserve evaluation.

Overall, the complete grouping and attention results support using cross-passage associations to establish joint-reading scopes before evidence selection. The comparisons connect the association signal and grouping choices to observed QA performance across the four datasets.

AuthorsOpenReview ↗

Response to [W2. Compression and Restructuring Baselines]

Thank you for identifying these representative baselines. We have expanded the comparison to all five requested methods, including RECOMP and Refiner. This provides a broader assessment of REORG against both compression and restructuring approaches.

Expanded comparison and results.

The additional methods are LongLLMLingua [1], LLMLingua-2 [2], RECOMP [3], Refiner [4], and EDC2-RAG [5]. LongLLMLingua, LLMLingua-2 and RECOMP represent context compression. Refiner organizes cross-source extracts into interconnected sections, while EDC2-RAG performs within-cluster joint processing; these two methods provide closely related restructuring comparisons. Each entry is EM/F1 in percent, and Average gives the unweighted four-dataset mean.

Method (EM/F1, %)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
REORG42.39 / 54.9249.69 / 55.2612.83 / 22.1040.64 / 51.9636.39 / 46.06
LongLLMLingua39.10 / 51.5842.76 / 48.8511.05 / 21.0236.34 / 47.8232.31 / 42.32
LLMLingua-233.22 / 44.2533.16 / 37.759.23 / 17.8632.88 / 43.9927.12 / 35.96
RECOMP38.33 / 49.9345.34 / 50.319.81 / 18.4937.42 / 48.5232.73 / 41.81
Refiner39.99 / 49.5146.02 / 51.9010.34 / 20.1636.34 / 47.5333.17 / 42.28
EDC2-RAG41.12 / 53.8746.76 / 52.6812.00 / 21.5938.20 / 50.4034.52 / 44.64

The corresponding compression factors are listed below and are included in the same main-results table. CR is a multiplicative length ratio; its Average is the mean of the four dataset ratios.

Method (CR)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
REORG5.1x6.4x4.9x3.8x5.1x
LongLLMLingua5.5x5.5x5.6x5.5x5.5x
LLMLingua-24.8x4.9x4.9x4.8x4.9x
RECOMP5.3x5.3x5.2x5.5x5.3x
Refiner4.8x4.6x4.6x4.7x4.7x
EDC2-RAG5.1x5.2x5.5x5.2x5.3x

REORG's average CR is 5.1x, compared with 5.3x for EDC2-RAG and RECOMP and 4.7x for Refiner. Reporting CR alongside answer quality makes the evaluated quality/length operating points visible. The compression factors are length descriptors.

REORG has higher EM and F1 than all five added baselines on every dataset. Its average F1 is 46.06, compared with 44.64 for EDC2-RAG, 42.32 for LongLLMLingua, 35.96 for LLMLingua-2, 41.81 for RECOMP, and 42.28 for Refiner. Using the unrounded means, the respective F1 gaps are 1.43, 3.74, 10.10, 4.25, and 3.79 percentage points.

The newly added Refiner comparison is useful because it extends the evaluation beyond selection and compression baselines to a related restructuring approach. REORG's F1 exceeds Refiner's by 5.41 points on HotpotQA and 3.36 points on 2WikiMQA, as well as on the other two datasets. The 4.25-point average F1 gap over RECOMP also broadens the evidence relative to the compression methods named in the review.

Comparison under a common compressed-context cap. The complete below-2k Refiner comparison is given below as EM/F1 (%), with equal-weight averages.

Method (EM/F1, %)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
Refiner38.70 / 47.3644.49 / 50.0710.22 / 18.0632.69 / 43.4231.53 / 39.73
REORG41.50 / 52.8047.55 / 52.8511.75 / 20.1339.06 / 49.9734.97 / 43.94

These are comparisons at the reported operating points. We additionally evaluate Refiner and REORG under a below-2k compressed-context cap, where REORG retains a 4.21-point average F1 advantage.

Overall, REORG has higher EM/F1 than each of the five added baselines on all four datasets, and its advantage over Refiner remains visible under the common below-2k cap. The complete score and CR tables place these observations alongside the reported retained-length operating points.

References

[1] Jiang et al. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. ACL 2024.

[2] Pan et al. LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. Findings of ACL 2024.

[3] Xu, Shi, and Choi. RECOMP: Improving Retrieval-Augmented LMs with Context Compression and Selective Augmentation. ICLR 2024.

[4] Li et al. Refiner: Restructure Retrieved Content Efficiently to Advance Question-Answering Capabilities. Findings of EMNLP 2024.

[5] Li et al. Efficient Dynamic Clustering-Based Document Compression for Retrieval-Augmented-Generation. Findings of EMNLP 2025.

AuthorsOpenReview ↗

Response to [W3. Component Ablations on Multi-Hop Benchmarks]

Thank you for recommending multi-hop evaluation of the components. We have extended A1-A4 to HotpotQA, 2WikiMQA, and MuSiQue, with NQ-Open retained as a contrasting task. The resulting comparisons connect the component analysis more directly to REORG's cross-document reasoning motivation.

Component definitions and results.

The four interventions examine different points in evidence construction. A1 removes overlap between reading groups. A2 removes the relation hints supplied to the Organizer. A3 removes the relation block supplied to Answer. A4 replaces group-wise organization with one Organizer call over all top-100 candidate passages. A0 is the reported full-method reference. The table reports EM/F1 percentages and unweighted four-dataset averages.

Setting (EM/F1, %)HotpotQA2WikiMQAMuSiQueNQ-OpenAverage
A0 Full REORG42.39 / 54.9249.69 / 55.2612.83 / 22.1040.64 / 51.9636.39 / 46.06
A1 Without group overlap38.58 / 52.3546.69 / 53.489.64 / 19.2139.92 / 51.7033.71 / 44.19
A2 Without Organizer relation hints38.58 / 49.5645.66 / 52.0110.22 / 16.9939.70 / 51.3933.54 / 42.49
A3 Without Answer relation block38.73 / 52.6047.15 / 53.9110.80 / 19.6039.86 / 51.2934.14 / 44.35
A4 One Organizer call for all top-100 passages39.57 / 53.7442.76 / 49.7411.38 / 19.1539.53 / 50.9733.31 / 43.40

All four ablations score below full REORG on both EM and F1 on each dataset. For overlap, the multi-hop F1 drops are 2.57, 1.78, and 2.89 points on HotpotQA, 2WikiMQA, and MuSiQue, averaging 2.41 points. The corresponding NQ-Open gap is 0.26 points. The larger observed multi-hop gaps fit the role of overlapping groups: a passage can participate in more than one joint-reading context when evidence must be combined across sources.

Relation information is useful at both subsequent stages. Removing Organizer hints lowers average multi-hop F1 by 4.57 points, while removing the Answer relation block lowers it by 2.06 points. The comparison therefore concerns both how evidence is selected together and how the selected material is presented for answering. A4 measures the broader group-wise versus single-call processing choice, including the changed input scope and call count.

Overall, the four-dataset component results show larger overlap and Organizer-hint differences on the multi-hop tasks than on NQ-Open. They connect joint-reading scopes and relation information to the cross-source reasoning setting that motivates REORG.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.