- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper tackles post-retrieval context construction for retrieval-augmented generation (RAG). The authors name a specific regime they call Evidence Entanglement, where retrieved passages interleave shared and complementary facts, so whole-passage selection or independent compression can discard evidence whose value emerges only through joint reading. The propsed method REORG has three components: Attention-based Relation Discovery, Overlapping Evidence Organization and Relation-guided Integration. REORG attains consistent performance gain over four datasets.
Critical strengths
1 The Evidence Entanglement is intuitive and does identify a real failure mode of passage-level selection / independent compression. The two motivations M#1 and M#2 (lines 79–91) map directly to the two technical stages, which keeps the paper easy to follow.
2 REORG is best in EM and F1 on HotpotQA, 2WikiMQA, MuSiQue, and NQ-Open. Importantly the leading comparator differs across datasets so the fact that one single pipeline beats all three is a non-trivial result.
Critical weaknesses
1 The core design of this paper (identify cross-document associations, cluster into groups, extract joint evidence) is a recombination of ideas from prior works. The genuinely new elements are (i) using attention from two reversed orders as the association signal and (ii) using overlapping (SLPA-style) rather than disjoint groups.
Table 3 actually argues against the primacy of overlap:
w/o group overlap (A1): −0.26 F1 w/o Organizer relation hints (A2): −0.57 F1 w/o Answer relation structure (A3): −0.67 F1 w/o group-wise organization (A4): −0.99 F1
So the component most tied to the paper's "relation-first" framing (A1, overlapping groups) contributes the smallest ablation gap, while the biggest gap (A4) is simply the choice to process passages in groups rather than feeding all candidates in a single Organizer call. That makes the paper's conceptual contribution incremental.
2 Each diagnostic experiment uses a different single dataset, so the robustness is not actually tested. The Retrieval-scale sweep (Table 2) is tested on HotpotQA only. the component ablation (Table 3) is tested on NQ-Open only. Parameter sensitivity is also tested on one dataset only (F1 reference value 51.96 matches the NQ-Open row of Table 1; the panels say nothing about Hotpot / 2Wiki / MuSiQue). For a paper whose main empirical claim is multi-dataset generality, it is unsatisfying that every supporting analysis is single-dataset.
Questions for authors
Please refer to the weakness section.
4 comments
Response to [W2. Diagnostic Coverage]
Thank you for suggesting broader diagnostic coverage. We expanded the component analysis and added comparisons across HotpotQA, 2WikiMQA, MuSiQue, and NQ-Open. Averages weight all four datasets equally.
Evaluation coverage
| Evaluation | Dataset coverage |
|---|---|
| A1-A4 component ablations | All four datasets |
| G1-G3 grouping controls | All four datasets |
| S1-S3 attention controls | All four datasets |
| Five requested baselines | All four datasets |
| Below-2k Refiner comparison | All four datasets |
| Retrieval Top-K sweep | HotpotQA; REORG plus eight baselines |
| Parameter-sensitivity sweeps | One dataset |
Component ablations (A0-A4, with A0 as full REORG)
| Setting (EM/F1, %) | HotpotQA | 2WikiMQA | MuSiQue | NQ-Open | Average |
|---|---|---|---|---|---|
| A0 Full REORG | 42.39 / 54.92 | 49.69 / 55.26 | 12.83 / 22.10 | 40.64 / 51.96 | 36.39 / 46.06 |
| A1 Without group overlap | 38.58 / 52.35 | 46.69 / 53.48 | 9.64 / 19.21 | 39.92 / 51.70 | 33.71 / 44.19 |
| A2 Without Organizer relation hints | 38.58 / 49.56 | 45.66 / 52.01 | 10.22 / 16.99 | 39.70 / 51.39 | 33.54 / 42.49 |
| A3 Without Answer relation block | 38.73 / 52.60 | 47.15 / 53.91 | 10.80 / 19.60 | 39.86 / 51.29 | 34.14 / 44.35 |
| A4 One Organizer call for all top-100 passages | 39.57 / 53.74 | 42.76 / 49.74 | 11.38 / 19.15 | 39.53 / 50.97 | 33.31 / 43.40 |
Grouping controls (G1-G3)
| Setting (EM/F1, %) | HotpotQA | 2WikiMQA | MuSiQue | NQ-Open | Average |
|---|---|---|---|---|---|
| A0 Full REORG | 42.39 / 54.92 | 49.69 / 55.26 | 12.83 / 22.10 | 40.64 / 51.96 | 36.39 / 46.06 |
| G1 Random grouping | 39.43 / 53.20 | 46.79 / 53.85 | 11.92 / 18.56 | 39.50 / 51.92 | 34.41 / 44.38 |
| G2 Average-linkage hierarchical clustering | 39.01 / 53.28 | 47.91 / 54.45 | 10.22 / 19.43 | 39.17 / 51.67 | 34.08 / 44.71 |
| G3 Top-k neighborhood grouping | 40.70 / 54.11 | 49.13 / 55.22 | 10.80 / 17.93 | 39.83 / 51.27 | 35.12 / 44.63 |
Attention input controls (S1-S3)
| Setting (EM/F1, %) | HotpotQA | 2WikiMQA | MuSiQue | NQ-Open | Average |
|---|---|---|---|---|---|
| A0 Full REORG | 42.39 / 54.92 | 49.69 / 55.26 | 12.83 / 22.10 | 40.64 / 51.96 | 36.39 / 46.06 |
| S1 Forward only, without leading question prefix | 38.16 / 52.08 | 47.62 / 54.79 | 9.64 / 18.99 | 37.17 / 51.23 | 33.15 / 44.27 |
| S2 Forward only, with leading question prefix | 40.27 / 53.77 | 47.34 / 53.98 | 12.49 / 20.25 | 39.17 / 51.41 | 34.82 / 44.85 |
| S3 Forward + reverse, without leading question prefix | 40.14 / 53.75 | 47.25 / 53.88 | 11.38 / 20.97 | 39.14 / 51.15 | 34.48 / 44.94 |
Added baselines
| Method (EM/F1, %) | HotpotQA | 2WikiMQA | MuSiQue | NQ-Open | Average |
|---|---|---|---|---|---|
| REORG | 42.39 / 54.92 | 49.69 / 55.26 | 12.83 / 22.10 | 40.64 / 51.96 | 36.39 / 46.06 |
| LongLLMLingua | 39.10 / 51.58 | 42.76 / 48.85 | 11.05 / 21.02 | 36.34 / 47.82 | 32.31 / 42.32 |
| LLMLingua-2 | 33.22 / 44.25 | 33.16 / 37.75 | 9.23 / 17.86 | 32.88 / 43.99 | 27.12 / 35.96 |
| RECOMP | 38.33 / 49.93 | 45.34 / 50.31 | 9.81 / 18.49 | 37.42 / 48.52 | 32.73 / 41.81 |
| Refiner | 39.99 / 49.51 | 46.02 / 51.90 | 10.34 / 20.16 | 36.34 / 47.53 | 33.17 / 42.28 |
| EDC2-RAG | 41.12 / 53.87 | 46.76 / 52.68 | 12.00 / 21.59 | 38.20 / 50.40 | 34.52 / 44.64 |
Compression factors
| Method (CR) | HotpotQA | 2WikiMQA | MuSiQue | NQ-Open | Average |
|---|---|---|---|---|---|
| REORG | 5.1x | 6.4x | 4.9x | 3.8x | 5.1x |
| LongLLMLingua | 5.5x | 5.5x | 5.6x | 5.5x | 5.5x |
| LLMLingua-2 | 4.8x | 4.9x | 4.9x | 4.8x | 4.9x |
| RECOMP | 5.3x | 5.3x | 5.2x | 5.5x | 5.3x |
| Refiner | 4.8x | 4.6x | 4.6x | 4.7x | 4.7x |
| EDC2-RAG | 5.1x | 5.2x | 5.5x | 5.2x | 5.3x |
Common compressed-context cap (below 2k tokens)
| Method (EM/F1, %) | HotpotQA | 2WikiMQA | MuSiQue | NQ-Open | Average |
|---|---|---|---|---|---|
| Refiner | 38.70 / 47.36 | 44.49 / 50.07 | 10.22 / 18.06 | 32.69 / 43.42 | 31.53 / 39.73 |
| REORG | 41.50 / 52.80 | 47.55 / 52.85 | 11.75 / 20.13 | 39.06 / 49.97 | 34.97 / 43.94 |
REORG outperforms all three grouping controls on every dataset. Removing overlap has a larger effect on each multi-hop dataset than on NQ-Open. Below 2k tokens, REORG outperforms Refiner in both EM and F1 on every dataset.
Time and compute limited the additional sweeps. Retrieval Top-K results still cover HotpotQA only, and parameter-sensitivity results still cover one dataset.
Summary of Responses to Reviewer PY3f
Thank you for helping us clarify the contribution and diagnostic coverage. The updated responses connect the relation-first design to the additional multi-dataset evidence.
- W1: Contribution and overlap. REORG establishes overlapping joint-reading scopes before source-sentence selection and then preserves the selected text and its relations. Removing overlap gives a 2.41-point mean multi-hop F1 drop, compared with 0.26 on NQ-Open. REORG also exceeds the strongest grouping control by 1.35 average F1 points. The evidence supports the coordinated reading-scope and relation-information design.
- W2: Coverage. Component, grouping and attention controls now cover all four datasets, together with the five-baseline and below-2k Refiner comparisons. The existing retrieval-depth study covers HotpotQA, while sensitivity remains single-dataset. Our responses distinguish those original sweep scopes from the expanded diagnostics.
Manuscript revisions. Contribution clarifications in the abstract (p. 1), Introduction (p. 3) and Section 2.2 (pp. 3-4) are dark orange. The actual coverage statements in Sections 4.3 (p. 8), 4.5 and 5 (p. 9), with details in Appendices C.1 (p. 25) and C.3 (p. 27), use the same color. Shared experimental additions are purple: component EM/F1 in Table 3 (p. 8) and the complete component/grouping/attention results in Appendix C.2, Tables 10-12 (p. 26).
The Guideline for Reviewers in Appendix F (p. 32) gives the color key.
Overall, the manuscript now connects the clarified contribution to its supporting comparisons and states the evaluated coverage directly.
Gentle Reminder to Reviewer PY3f
Thank you again for helping us clarify the contribution and evaluation. As the discussion period draws to a close, we would be grateful if you could review our updated responses and color-marked manuscript once more. We hope the clarified design and complete multi-dataset result tables help address the main concerns. We welcome further questions and appreciate your time.
Response to [W1. Method Contribution and the Role of Overlap]
Thank you for asking us to make the contribution and the role of overlap more precise. REORG's contribution is the relation-first evidence-construction design: establish joint-reading scopes from cross-passage associations before deciding which evidence to retain. Overlap is one part of this design, working together with relation-guided selection and source-grounded rendering.
Design rationale.
Evidence Entanglement describes the setting in which shared background and unique complementary facts are interleaved across retrieved passages. REORG addresses it by using attention associations from complementary passage orders to construct overlapping groups. The Organizer then jointly reads the sources in each group and returns sentence identifiers and relation labels. Programmatic recovery renders the selected source text, keeping evidence selection separate from factual wording.
Refiner already combines cross-source extraction with organization, and EDC2-RAG already performs joint processing within document clusters. REORG positions its contribution in the scope and order of these decisions: local associations establish overlapping joint-reading contexts while the source text is still available, the Organizer selects sentence references with relation cues, and rendering recovers those selected sources. The contribution is this coordinated relation-first construction.
The benefit sought from overlap is specific: the same passage can support more than one evidence combination. This is particularly relevant to multi-hop questions, where a useful connection may involve information spread across several passages. We therefore expanded the component comparisons beyond NQ-Open.
Evidence on multi-hop tasks.
Complete component results (A0-A4). A0 is the reported full-method reference. The table gives EM/F1 (%) for every dataset and the unweighted four-dataset mean.
The table summarizes F1 drops relative to full REORG. The multi-hop mean covers HotpotQA, 2WikiMQA, and MuSiQue.
Removing overlap lowers F1 by 2.57, 1.78, and 2.89 points on the three multi-hop datasets, averaging 2.41 points, compared with 0.26 on NQ-Open. Removing Organizer hints and the Answer relation block also gives larger multi-hop mean gaps, at 4.57 and 2.06 points. This supports examining relation information at both evidence selection and answer generation.
Complete grouping controls (G1-G3). The table reports the full-method reference and all grouping alternatives, with EM/F1 (%) and equal-weight four-dataset means.
Grouping controls strengthen this picture: full REORG averages 46.06 F1, compared with 44.38 for random grouping, 44.71 for average-linkage clustering, and 44.63 for top-k neighborhoods. REORG's 1.35-point gap over the strongest grouping control links the evaluation to how reading scopes are formed. A4 represents the broader change from group-wise organization to one call over all top-100 passages, including its changed processing scope and call count.
Overall, the full component and grouping results support the coordinated use of joint-reading scopes, relation-aware selection and source-preserving rendering. They give a more informative account of the design than a ranking of the original NQ-Open ablation gaps.