acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Retrieval in the Long-Context Era: Training-Free Semantic Trees for Adaptive Context Construction

Abstract

Long-context language models can increasingly process entire documents, making full-context prompting a practical alternative to passage retrieval. However, greater capacity determines how much information a model can receive, but not which information best supports a query. This distinction motivates rethinking retrieval: when the full document already fits, can selecting and organizing its evidence improve answering? The challenge is to retain supporting facts and relations while keeping the context focused on the question. To reconcile these goals, we introduce TreeFuse, a training-free framework that uses semantic abstraction to guide selection while retaining original text for answering. TreeFuse constructs a reusable semantic hierarchy from the full document in a single generation, linking node descriptions to source spans. Given a query, structural–semantic scoring and fine-grained lexical attribution nominate shared source nodes; their union with ancestor context preserves structural relations and complete local text. Across four frontier answer models, TreeFuse improves average QA-F1 by 1.72 points over full-context prompting on three LongBench tasks, surpassing the evaluated baselines on the four-model average while reducing answer-context length by 58.55%. Code is available at https://anonymous.4open.science/r/treefuse-ECFE.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Context Language Models, Hybrid Retrieval, Context Compression, Long-Document Question Answering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Lkw1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TREEFUSE for query-adaptive context construction when a full document already fits within a long-context window. TREEFUSE constructs a source-grounded semantic tree and selects context using semantic node scores, lexical matching, and ancestor closure. The selected contexts retain local source text and are reused across four answer models. Validation covers LongBench tasks, LaRA-32K, several context-construction baselines, component studies, context-length analyses, and an input-side cost model.

Critical strengths
  1. The paper addresses an important long-context question with a complete and source-grounded system. TREEFUSE uses reusable semantic trees for routing but returns local source text to the reader. This reduces dependence on generated summaries as final evidence and makes the method relevant to long-document QA.

  2. The evaluation is broad and transparent. The same constructed contexts are reused across four answer models. The paper reports average gains with substantial context reduction, provides several ablations, and explicitly states that budget matching is approximate and the cost model is an input-side proxy.

Critical weaknesses
  1. The complete-system comparison does not isolate semantic-tree design from scorer and preprocessing choices. The evaluated systems use different architectures, training procedures, model generations, and query-time preprocessing protocols. The results show that the configured TREEFUSE system performs well, but they do not establish how much of the advantage comes from the tree representation rather than scorer capability or preprocessing work. Some budget-matched comparisons also match only average characters rather than per-query reader tokens.
  2. Potentially score-changing statistical analysis. Are the LaRA and Qasper gains stable under question-level paired bootstrap or randomization tests?
  3. Potentially score-changing control. How sensitive are the results to the constructor and semantic scorer? For compatible hierarchical methods, does TREEFUSE retain its advantage with a shared scorer or matched query-time preprocessing work?
Questions for authors
  1. Budget clarification: What happens when the BM comparisons use paired per-query token budgets under each answer model's tokenizer rather than average retained characters?
  2. Novelty clarification: What is the key technical difference from PageIndex, DISRetrieval, and the complete HiChunk pipeline, and why were these systems not compared under a common protocol?
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1/3)

Thank you for your positive assessment and for highlighting the exact controls needed to isolate our tree representation, scorer, and budget accounting. We address your concerns in three parts: Part 1 covers the constructor/scorer controls (W1, W4); Part 2 covers per-query token budgets (W2, Q1) and paired statistical tests (W3); Part 3 covers new comparisons with PageIndex and DISRetrieval (Q2).


W1 & W4 (Part A): Isolating Constructor vs. Scorer under a Shared Protocol — Our scorer/selector adds +2.75 QA-F1 onto HiChunk units, and our tree constructor adds a further +2.78 QA-F1 under the exact same scorer

To separate the tree constructor (which defines node boundaries, parent–child links, and metadata) from the query-time selector (semantic scoring, adaptive cutoffs, lexical union, and ancestor closure), we compare representation and construction variants while holding the Qwen3.6-35B-A3B scorer and Qwen3.8-Max reader strictly fixed (§4.3; 3-run mean QA-F1):

Configuration (Reader: Qwen3.8-Max)Scorer / SelectorHotpotQA2WikiMQAAvg.TreeFuse Gain
Raw (Full document reference)None75.6984.1979.94+1.18
HiChunk (Native pipeline)BGE-M3 + Auto-Merge70.5380.6575.59+5.53
HiChunk adapter (HiChunk units + our selector)Shared 35B-A3B + BM2574.1582.5478.34+2.78
Fixed 200 (200-token chunks)Shared 35B-A3B + BM2572.7779.4776.12+5.00
Chonkie (Semantic chunks)Shared 35B-A3B + BM2574.3378.7776.55+4.57
TreeFuse (flat) (Same nodes, no hierarchy)Shared 35B-A3B + BM2574.0780.9377.50+3.62
Multi-stage constructor (3-stage tree build)Shared 35B-A3B + BM2575.4782.3378.90+2.22
TreeFuse (Full joint constructor)Shared 35B-A3B + BM2577.3984.8581.12—

This controlled progression cleanly separates the two components:

  1. Effect of our query-time scorer/selector (+2.75 QA-F1): Keeping HiChunk-derived units fixed, replacing native retrieval + Auto-Merge with our semantic–lexical selector raises Avg. QA-F1 from 75.59 to 78.34 (+3.62 on HotpotQA, +1.89 on 2WikiMQA).
  2. Effect of our tree representation & constructor (+2.22 to +5.00 QA-F1): Under the exact same 35B-A3B scorer and selector, full TreeFuse outperforms HiChunk adapter by +2.78, Multi-stage by +2.22, TreeFuse (flat) by +3.62 (confirming hierarchy-aware routing and ancestor completion), and flat chunkers (Fixed 200 / Chonkie) by +5.00 / +4.57.

W4 (Part B): Sensitivity to Semantic Scorer Scale — Replacing 35B-A3B (2.86B active MoE) with a dense Qwen3.5-4B scorer still beats Raw and all compression baselines

Note that our default Qwen3.6-35B-A3B is an MoE model with only 2.86B activated parameters per token (Appendix C.4). To test sensitivity to scorer capacity while holding the constructed trees fixed, we replaced the query-time scorer with a lightweight Qwen3.5-4B model across all four readers (§4.3, Table 7; 3-run mean QA-F1):

DatasetMethodQwen3.8-MaxDeepSeek-V4-ProKimi-K2.6GLM-5.3Overall
QasperRaw / HiChunk (best baseline)48.89 / 49.7051.20 / 49.8850.35 / 49.2446.61 / 46.8549.26 / 48.92
TreeFuse (4B scorer)50.9551.3850.5147.9750.20
TreeFuse (35B-A3B scorer)50.9651.3550.6248.0550.25
HotpotQARaw / BRIEF-Pro (best baseline)75.69 / 73.8072.20 / 69.6159.47 / 60.2270.68 / 69.0969.51 / 68.18
TreeFuse (4B scorer)75.6073.5161.2069.6069.98
TreeFuse (35B-A3B scorer)77.3973.6463.5070.3871.23
2WikiMQARaw / HiChunk (best baseline)84.19 / 80.6579.28 / 79.5447.44 / 47.7571.62 / 70.7970.63 / 69.69
TreeFuse (4B scorer)81.0476.6054.6873.5571.47
TreeFuse (35B-A3B scorer)84.8579.0054.4474.0873.09

With the 4B scorer, TreeFuse still improves over Raw on all three tasks (+0.94, +0.47, +0.84) and surpasses the strongest baseline on every task (+1.28, +1.80, +1.78), while 35B-A3B adds a further +0.05 to +1.62 points.

(Continued in Part 2)

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2/3)

(Continued from Part 1)

W2 & Q1: Paired Per-Query Token Budget Accounting — Frozen BM contexts already match TreeFuse within 0.45%–4.10% mean absolute per-query tokens

We apologize for the misunderstanding; our actual BM implementation achieves near-exact character-level budget matching on a per-question basis. In our ablations (§4.3, Table 4; Appendix E.1), BM targets TreeFuse's per-query character budget (adding random unselected nodes for Semantic-only (BM), or adjusting the lexical node threshold with ancestor closure for Lexical-only (BM)). Because whole nodes are selected without mid-node truncation, small per-query residuals occur. Measuring the frozen reader inputs with the Qwen3.8-Max tokenizer, the mean absolute per-query relative gap, 100×E[∣Bablation−BTreeFuse∣/BTreeFuse]100 \times \mathbb{E}[|B_{\text{ablation}} - B_{\text{TreeFuse}}| / B_{\text{TreeFuse}}], is very tight:

  • Semantic-only (BM): Mean absolute gap is 1.20% chars / 1.69% tokens on HotpotQA, and 0.27% chars / 0.45% tokens on 2WikiMQA.
  • Lexical-only (BM): Mean absolute gap is 0.73% chars / 3.08% tokens on HotpotQA, and 1.56% chars / 4.10% tokens on 2WikiMQA.

Thus, on the frozen contexts, per-query reader token budgets already align within 0.45%–4.10% while preserving the exact QA-F1 gains in Table 4 (+2.72 over Semantic-only (BM) and +6.16 over Lexical-only (BM)). We now describe the per-query BM setting in §4.3 and Appendix E.1.


W3: Question-Level Statistical Significance — Paired bootstrap & sign-flip tests confirm significant gains in 19/24 comparisons, including 5/5 over LaRA baselines and 11/12 on multi-hop QA

To test question-level stability beyond 3-run decoding variance, we ran paired bootstrap (50,000 resamples) and two-sided sign-flip randomization tests on the frozen results. For each question, we average the 3 runs per model and equally average paired differences across the four fixed answer models (∗^* denotes p<0.05p < 0.05; 95% CIs in percentage points):

Baseline vs. TreeFuseQasper (QA-F1)HotpotQA (QA-F1)2WikiMQA (QA-F1)LaRA (Judge Acc.)
LongLLMLinguaCI [19.18, 27.09], p=.00048∗p=.00048^*CI [10.73, 18.87], p=.00048∗p=.00048^*CI [12.92, 20.30], p=.00048∗p=.00048^*CI [15.26, 20.34], p=.00048∗p=.00048^*
ProvenceCI [0.94, 5.34], p=.0474∗p=.0474^*CI [2.76, 7.44], p=.00048∗p=.00048^*CI [8.00, 14.53], p=.00048∗p=.00048^*CI [5.87, 9.35], p=.00048∗p=.00048^*
HiChunkCI [-0.11, 2.93], p=.3310p=.3310CI [2.62, 6.85], p=.00048∗p=.00048^*CI [1.97, 4.91], p=.00048∗p=.00048^*CI [4.59, 8.19], p=.00048∗p=.00048^*
BRIEF-ProCI [-0.43, 4.25], p=.3310p=.3310CI [-0.29, 6.22], p=.3310p=.3310CI [1.08, 6.63], p=.0474∗p=.0474^*CI [5.84, 9.98], p=.00048∗p=.00048^*
LongRefinerCI [15.18, 24.38], p=.00048∗p=.00048^*CI [9.24, 17.73], p=.00048∗p=.00048^*CI [22.59, 32.78], p=.00048∗p=.00048^*CI [37.17, 43.60], p=.00048∗p=.00048^*
Raw (Full Context)CI [-0.004, 1.99], p=.2618p=.2618CI [0.49, 2.99], p=.0474∗p=.0474^*CI [1.03, 3.81], p=.0065∗p=.0065^*CI [-0.67, 1.30], p=.5464p=.5464
  • Overall & Multi-hop: Mean differences are positive in all 24 comparisons, and 19/24 are statistically significant (with document/question-package cluster sensitivity analyses confirming the exact same 19 conclusions), including 5/6 on HotpotQA and 6/6 on 2WikiMQA (beating both Raw and compression baselines).
  • LaRA & Qasper stability: On LaRA, TreeFuse's gains over all 5 compression baselines (5/5) are highly significant (p=0.00048p = 0.00048). On Qasper, gains over LongLLMLingua, Provence, and LongRefiner are significant (3/6). We now present these results in §4.2 and Appendix C.1.

(Continued in Part 3)

AuthorsOpenReview ↗

Rebuttal by Authors (Part 3/3)

(Continued from Part 2)

Q2: Technical Distinctions & Common-Protocol Evaluation with PageIndex, DISRetrieval, and Complete HiChunk — TreeFuse outperforms all three hierarchical systems by +1.33 to +3.20 Overall QA-F1

Key technical differences in selection and evidence assembly:

  • PageIndex performs iterative LLM navigation over a ToC-like tree; DISRetrieval parses trained RST discourse trees for dense hierarchical sentence retrieval; and Complete HiChunk uses dense retrieval (BGE-M3) followed by Auto-Merge (which expands a parent node to include all its child chunks once a threshold is met).
  • TreeFuse instead decouples routing metadata from source evidence on a shared source-grounded tree: structure-conditioned semantic scoring and subunit BM25 attribution independently nominate nodes, and ancestor completion adds only each ancestor's local introductory span—preserving structural framing and exact lexical hits without pulling in irrelevant sibling subtrees.

New common-protocol evaluation on LongBench (§4.1–§4.2, Table 1; Appendix B.3): We evaluated PageIndex and DISRetrieval alongside Complete HiChunk, Raw, and TreeFuse under the exact same aligned question sets and three-run evaluation protocol across all four answer models (QA-F1; bold = best). The HiChunk results in the main table of our originally submitted PDF already used the original HiChunk paper's complete pipeline and settings; only the construction ablation adapts HiChunk units to our scorer.

DatasetMethodQwen3.8-MaxDeepSeek-V4-ProKimi-K2.6GLM-5.3Overall
QasperRaw48.8951.2050.3546.6149.26
PageIndex49.4447.9147.7343.7847.21
DISRetrieval45.4944.2143.7042.6544.01
HiChunk (Complete)49.7049.8849.2446.8548.92
TreeFuse (Ours)50.9651.3550.6248.0550.25
HotpotQARaw75.6972.2059.4770.6869.51
PageIndex72.9370.8060.1568.2668.03
DISRetrieval65.2353.6645.3657.2955.38
HiChunk (Complete)70.5369.5758.1668.2166.62
TreeFuse (Ours)77.3973.6463.5070.3871.23
2WikiMQARaw84.1979.2847.4471.6270.63
PageIndex82.3378.8450.0372.9871.05
DISRetrieval74.8253.6735.8956.3355.18
HiChunk (Complete)80.6579.5447.7570.7969.69
TreeFuse (Ours)84.8579.0054.4474.0873.09

TreeFuse achieves the highest four-model Overall QA-F1 on all three datasets, beating the strongest hierarchical baseline on each task by +1.33 on Qasper (50.25 vs. 48.92 for HiChunk), +3.20 on HotpotQA (71.23 vs. 68.03 for PageIndex), and +2.04 on 2WikiMQA (73.09 vs. 71.05 for PageIndex). We have integrated these comparisons into the revised manuscript.


Closing Remark

Thank you again for your careful review and for identifying the controls needed for a fair assessment. We have separated constructor and scorer effects through controlled ablations, clarified per-query budget accounting, added paired statistical tests, and compared complete hierarchical baselines while distinguishing native HiChunk from its scorer-adapted ablation. We hope these results address your concerns. If you find the updated evidence persuasive, we would sincerely appreciate your consideration of a higher rating.

AuthorsOpenReview ↗

Eager to discuss our rebuttal for Submission53818

Dear Reviewer,

Thank you again for your time and constructive feedback. We are following up on our rebuttal and would greatly appreciate your thoughts on whether the following clarifications and results address your concerns:

  • Tree construction versus scorer effects (W1, W4): Shared-scorer controls show that our tree representation adds +2.78 average QA-F1 over the HiChunk adapter. A lightweight 4B scorer also retains higher task-level average scores than Raw and the evaluated compression baselines.
  • Budget fairness (W2, Q1): Token accounting on the frozen reader inputs shows mean absolute per-query token-budget gaps of only 0.45%–4.10%, while preserving the reported budget-matched ablation gains.
  • Statistical reliability (W3): Paired bootstrap and randomization tests establish significant gains in 19/24 comparisons, including 11/12 on multi-hop QA.
  • Hierarchical baseline comparisons (Q2): New evaluations of PageIndex and DISRetrieval, alongside complete HiChunk, show that TreeFuse achieves the highest four-model average QA-F1 on all three datasets, exceeding the strongest hierarchical baseline by 1.33–3.20 points.

Further details are available in our responses above. We would greatly appreciate hearing whether your concerns have been addressed. If any points remain unresolved or need further clarification, please feel free to let us know—we would be happy to discuss them.

Reviewer_fxSyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The authors propose TREEFUSE, a training-free framework addressing information overload in long-context LLMs. By constructing a reusable semantic hierarchy that links abstract descriptions to source spans, TREEFUSE selectively retrieves structurally and semantically relevant evidence for a given query. This approach outperforms standard full-context prompting on LongBench tasks, improving QA-F1 scores while significantly reducing the required context length by over 58%.

Critical strengths
  1. TreeFuse elegantly decouples document evidence selection from final question answering by constructing a hierarchical tree that incorporates semantic abstractions while remaining anchored to the source text. The method requires no task-specific or model-specific fine-tuning, and the semantic tree built for a document can be directly reused across various queries and different base models, demonstrating high versatility and strong potential for practical deployment.

  2. The framework innovatively adopts a dual-channel strategy combining structural-semantic scoring with fine-grained lexical matching. This design effectively compresses excessive context length while substantially improving multi-hop question answering accuracy.

Critical weaknesses
  1. TreeFuse relies on a 35B LLM to traverse and score all document nodes during the query stage, which introduces prohibitive computational latency and cost in real-world retrieval systems.

  2. The evaluation equips TreeFuse with a powerful 35B model for preprocessing and node relevance scoring, whereas the baselines rely on much weaker model configurations, such as LongLLMLingua with only a 7B model and HiChunk with a BGE embedding model. Consequently, the performance gains are likely driven by the overwhelming parameter scale of the scoring model rather than demonstrating the intrinsic advantage of the proposed semantic tree structure.

  3. The framework relies entirely on an LLM to perform boundary segmentation and hierarchical structure construction over long documents in a single generation pass. When handling noisy real-world documents that have complex layouts or lack clear headings, the model is highly prone to chunking errors and hallucinations.

Questions for authors

see Critical Weaknesses

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1/2)

Thank you for recognizing our decoupling of evidence selection from question answering and our dual-channel design. Below we address your concerns regarding query-time latency, scorer capacity, and robustness on noisy/unstructured documents.

W1: Query-Time Latency & Cost — Batched breadth-first scoring with our 2.86B-activated MoE or a 4B dense scorer averages only 3.56s or 1.85s per query

First, we clarify two key design facts regarding query-time cost (Appendices A.3 & C.5):

  1. Active parameter scale: Qwen3.6-35B-A3B is a Mixture-of-Experts (MoE) model with only 2.86B activated parameters per token—requiring less per-token compute than the dense Llama-2-7B used in LongLLMLingua and comparable to LongRefiner's Qwen2.5-3B.
  2. Batched scoring (not one call per node): Stage 2 groups nodes breadth-first into single requests of up to 40 nodes (or 48,000 characters) per call. "Scoring every node" refers to full node coverage, not O(∣V∣)O(|V|) sequential LLM calls.

We benchmarked query-time Stage 2 scoring latency (seconds/query; Appendix C.5) on 8 ×\times A100-40GB GPUs (BF16, 6-query concurrency) for both Qwen3.6-35B-A3B and a smaller dense Qwen3.5-4B scorer under identical concurrency and timing boundaries:

Scorer ModelQasper2WikiMQAHotpotQALaRA-32KQuery-Weighted Avg.
Qwen3.6-35B-A3B (2.86B active MoE)2.200s4.724s5.759s3.106s3.559s
Qwen3.5-4B (4B dense)2.053s2.176s2.687s1.536s1.847s (1.93$\times$ speedup)

Replacing 35B-A3B with Qwen3.5-4B cuts query-weighted scoring latency from 3.56s to 1.85s per query (more than halving latency on node-intensive HotpotQA and 2WikiMQA). Moreover, as shown in Appendix C.4, shrinking the reader context by 43.9%–58.6% substantially reduces prefill compute on frontier answer models (32B–90B active parameters), offsetting this lightweight scoring pass.


W2: Scorer Capacity vs. Semantic Tree Contribution — TreeFuse with a 4B scorer still beats Raw and all compression baselines, and fixed-scorer ablations confirm +2.22 to +5.00 QA-F1 from the tree itself

We separate tree design from scorer capacity through two controls: (1) downgrading the query-time scorer to Qwen3.5-4B on the same trees, and (2) varying the representation while holding the 35B-A3B scorer strictly fixed.

1. Downgrading the query-time scorer to Qwen3.5-4B (§4.3, Table 7): Below we compare TreeFuse (4B scorer) against Raw, our main 35B-A3B scorer, and the strongest baseline per dataset across all four answer models (QA-F1):

DatasetMethodQwen3.8-MaxDeepSeek-V4-ProKimi-K2.6GLM-5.3Overall
QasperRaw / HiChunk (best baseline)48.89 / 49.7051.20 / 49.8850.35 / 49.2446.61 / 46.8549.26 / 48.92
TreeFuse (4B scorer)50.9551.3850.5147.9750.20
TreeFuse (35B-A3B scorer)50.9651.3550.6248.0550.25
HotpotQARaw / BRIEF-Pro (best baseline)75.69 / 73.8072.20 / 69.6159.47 / 60.2270.68 / 69.0969.51 / 68.18
TreeFuse (4B scorer)75.6073.5161.2069.6069.98
TreeFuse (35B-A3B scorer)77.3973.6463.5070.3871.23
2WikiMQARaw / HiChunk (best baseline)84.19 / 80.6579.28 / 79.5447.44 / 47.7571.62 / 70.7970.63 / 69.69
TreeFuse (4B scorer)81.0476.6054.6873.5571.47
TreeFuse (35B-A3B scorer)84.8579.0054.4474.0873.09
  • Beating Raw & all baselines at 4B scale: Even with the 4B scorer, TreeFuse outperforms Raw on all three tasks (+0.94, +0.47, +0.84 QA-F1) and beats the best compression baseline on every task (+1.28, +1.80, +1.78 QA-F1)—confirming that our gains over 7B/BGE baselines do not rely on a massive scorer.

2. Isolating the tree under a strictly shared scorer (§4.3, Tables 5–7): Holding both the Qwen3.6-35B-A3B scorer and Qwen3.8-Max reader fixed, full TreeFuse (81.12 Avg. QA-F1 across HotpotQA / 2WikiMQA) substantially outperforms:

  • Removing the hierarchy: TreeFuse (flat) (77.50, +3.62 gain from hierarchy: 77.39 vs. 74.07 on HotpotQA; 84.85 vs. 80.93 on 2WikiMQA).
  • Alternative chunking units: Fixed 200 (76.12, +5.00 gain; 72.77 / 79.47) and Chonkie (76.55, +4.57 gain; 74.33 / 78.77).
  • Alternative tree constructors: Multi-stage (78.90, +2.22 gain; 75.47 / 82.33) and HiChunk adapter (78.34, +2.78 gain; 74.15 / 82.54, mapping HiChunk units into our selector interface).

Because all variants above use the exact same scorer, these +2.22 to +5.00 gains come directly from our source-grounded semantic tree and ancestor closure.

(Continued in Part 2)

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2/2)

(Continued from Part 1)

W3: Robustness to Unclear Headings & Complex Layouts — TreeFuse gains +2.40 QA-F1 on heading-sparse documents and remains robust on complex layouts

By design, TreeFuse is resilient to constructor hallucinations and noisy layouts for three reasons: (1) the constructor detects both explicit headings and coherent topic shifts (Appendix A.2); (2) a deterministic forward matcher anchors boundaries to verbatim source offsets xi=D[ai:ei)x_i=D[a_i:e_i), so generated summaries/facts never replace source text in the reader prompt (preventing hallucination propagation); and (3) coarse boundary containment suffices for QA (Appendix E.5), while lexical attribution rescues spans missed by metadata.

To test this empirically, we audited all documents for heading insufficiency (<2<2 section headings or <80%<80\% body coverage: 1/148 [0.68%] in Qasper, 59/200 [29.50%] in HotpotQA, 82/200 [41.00%] in 2WikiMQA, and 10/64 [15.63%] in LaRA-32K) and complex-layout indicators (multi-column, tables, mixed/nonlinear reading order: 48/64 [75.00%] in LaRA-32K, 0% in LongBench). Four-model average scores (3-run means on aligned sets; Appendix C.6) across these subsets are:

Benchmark (Metric)Document SubsetRawTreeFuseΔ\Delta vs. Raw
LongBench (QA-F1)Heading-sufficient (406 docs)59.9561.46+1.51
LongBench (QA-F1)Heading-insufficient (142 docs)73.2875.68+2.40
LaRA-32K (Judge Acc.)Without layout indicators (16 docs)70.9972.56+1.57
LaRA-32K (Judge Acc.)With complex-layout indicators (48 docs)83.6483.50-0.14 (43.9% less context)

When explicit headings are sparse or absent (35.3% of multi-hop documents), TreeFuse actually achieves a larger gain over Raw (+2.40 QA-F1) than on heading-sufficient documents (+1.51). On LaRA-32K documents with complex layouts/tables, TreeFuse matches Raw accuracy (-0.14 points) while removing 43.9% of the context (whereas all baseline compressors drop by 6.2 to 39.3 points overall). These analyses now appear in Appendix C.5 (latency), Appendix C.6 (document subsets), and §4.3, Table 7 (4B scorer).


Closing Remark

Thank you again for your thoughtful comments. We have reported measured query-time latency, evaluated a smaller scorer and fixed-scorer representation controls, and analyzed heading-sparse and complex-layout document subsets. We hope this evidence clarifies the practical cost and the contribution and scope of our tree design. If these results address your main concerns, we would sincerely appreciate your reconsideration of the paper for a higher rating.

AuthorsOpenReview ↗

Eager to discuss our rebuttal for Submission53818

Dear Reviewer,

Thank you again for your time and constructive feedback. As the discussion period is ending, we would like to briefly summarize our responses to your main concerns:

  • Latency (W1): Scoring with the 4B scorer reduces measured average scoring latency from 3.56s to 1.85s/query.
  • Scorer versus tree contribution (W2): The 4B variant retains higher task-level average QA-F1 than Raw and the evaluated compression baselines; fixed-scorer controls show 2.22–5.00 QA-F1 points of gain over alternative representations.
  • Document robustness (W3): Heading-insufficient documents show +2.40 QA-F1 over Raw; the complex-layout subset scores 0.14 accuracy points below Raw with 43.9% less context.

In the revised PDF, we clarified scoring (Appendix A.3), added latency measurements (Appendix C.5), reported scorer and representation controls (§4.3, Tables 5–7), and added document-subset analyses (Appendix C.6).

We would greatly appreciate hearing whether these clarifications and results address your concerns. Please let us know if any points need further explanation.

Reviewer_2BGZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work proposes to input the whole document as a context for long-context question answering after processing the input documents in a tree structure. A document is preprocessed as a tree structure automatically using an LLM by first segmenting into semantic units comprising a path, title, summary and facts. When responding to a query, the subtree is extracted based on the similarity measure of the query of a node using two scores, semantic score by relating the semantic similarity and the lexical score by measuring the lexical similarity. Experiments are carried out on a standard benchmark, Long Bench, showing some gains when compared with other methods.

Critical strengths
  • This work presents an alternative view of an input document by preprocessing it as a tree structure and the relevant subtree is extracted when generating a response given an input query.
  • Experiments show competitive performance when compared with strong baselines. Ablations studied demonstrate the benefit of the subtree extraction by combining the semantic and lexical scores.
Critical weaknesses
  • It is not obvious what is represented by the tree structure in the proposed method. The tree structure seems to be constructed by segmenting into units, and each unit represent a node of a tree. The question arises what relation is encoded between the connected nodes and what is meant by the parent-child relation. This work needs further discussion on prior studies in document structure, such as rhetorical structure theory, for the positioning the proposed approach.
  • Related to the tree structure, four information is encoded in a node together with the span of a document. However, no clear explanation exists in the manuscript, and how those information were used to compute the semantic and lexical scores. Specifically, mim_i and xix_i are defined in Section 3.2 together with an element of mim_i. However, they are never used in the score computation in Section 3.3. No clear explanation exists how a tree is encoded as a textual form, that is necessary to input to a language model.
  • No statistical significance is reported in experiments, although some numbers do not have large differences.
  • No qualitative analysis is presented regarding the tree structure. It is better to report what subtree structures are successful or not.
Questions for authors
  • Tree structure: I'd like to understand what is represented in a tree structure, and how it is related to the prior studies in document structure.
  • Information encoded in a node: it is better to explain what is encoded in each node and how a tree is encoded as a textual input.
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1/2)

Thank you for recognizing our tree representation, dual-channel selection, and empirical gains. Below we address each of your concerns and questions.


W1 & Q1: Tree Representation & Prior Work — Nodes form a section/topic hierarchy where parents keep only local intro text, distinct from RST nuclearity

TreeFuse builds a source-grounded section/topic hierarchy for evidence routing, not a causal chain or labeled discourse tree (Appendix A.3). Each node links a semantic unit to its non-overlapping local span xi=D[ai:ei)x_i=D[a_i:e_i), and parent–child edges represent section/subsection or topic/subtopic containment. A parent keeps only its own introductory text rather than concatenating its descendants, so adding ancestors provides framing context without pulling in irrelevant sibling branches.

  • Concrete trace (Qasper, "What is the GhostVLAD approach?"): The parent node (p_i="2", Pooling Strategies) holds only the general intro on pooling frame-level features into utterance representations. Its child (p_i="2.1", NetVLAD pooling) holds the specific NetVLAD formulation underlying GhostVLAD. Retaining both preserves local evidence and framing without other subsections.
  • Relation to prior work (RST, Mann & Thompson, 1988): RST models rhetorical relations and nuclearity (where parents/nuclei are more central to the writer's purpose) for discourse analysis. In TreeFuse, a parent is an organizational ancestor—not necessarily a rhetorical nucleus—designed to route source spans for QA. We have clarified this positioning in §2 and §3.2, with further details in Appendix A.1.

W2 & Q2: Node Contents, Scoring & Text Serialization — Explicitly linking mim_i and xix_i to §3.3 scoring, while sending only verbatim source spans xix_i to the reader

In §3.2, each node has a semantic view mi=(pi,ti,si,Fi)m_i=(p_i,t_i,s_i,F_i) and a source view xi=D[ai:ei)x_i=D[a_i:e_i). We now make their exact roles explicit in §3.3:

  • How mim_i and xix_i drive scoring (§3.3): Using the same NetVLAD pooling node (p_i="2.1", t_i="NetVLAD pooling"), the summary sis_i (soft-assignment clustering and feature mapping) and atomic facts FiF_i ("NetVLAD was initially developed for place recognition...") are fed with the query, document directory, and parent metadata into the LLM scorer to produce risem(q)∈{0,…,10}r_i^{\text{sem}}(q) \in \{0,\dots,10\} (Appendices A.2 & A.3). Here, NetVLAD scores 5/10 (supporting background) and its parent scores 8/10 (direct GhostVLAD description). In parallel, the verbatim span xix_i is split into sentence/paragraph subunits U(xi)\mathcal{U}(x_i) to compute the lexical score bi(q)=max⁡u∈U(xi)NormBM25⁡(q,u)b_i(q)=\max_{u\in\mathcal{U}(x_i)}\operatorname{NormBM25}(q,u).
  • How the pruned tree is encoded as text (§3.4): After taking the union S(q)=Rsem(q)∪Rlex(q)S(q)=R_{\text{sem}}(q)\cup R_{\text{lex}}(q) and adding ancestors, summaries sis_i, facts FiF_i, and scores are discarded so they never replace source evidence. For LongBench, the pruned tree is serialized as nested JSON with doc_id and tree, where each node contains level, title, verbatim source_text, and optional children (Appendix A.4). For the Qasper example, the exact serialized structure is: {"doc_id": "qasper-1-1704f1d50b2554ae", "tree": [{"level": "level2", "title": "Pooling Strategies", "source_text": "POOLING STRATEGIES\nIn any language identification model...", "children": [{"level": "level2.1", "title": "NetVLAD pooling", "source_text": "POOLING STRATEGIES ::: NetVLAD pooling\nThe NetVLAD pooling strategy was initially developed..."}]}]}. Here level2.1 denotes structural hierarchy, not relevance rank. LaRA formats the retained xix_i spans as numbered verbatim chunks with ancestor-title provenance. We now state this serialization interface explicitly in §3.4 and provide the complete example in Appendix A.4.

(Continued in Part 2)

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2/2)

(Continued from Part 1)

W3: Statistical Significance — Question-level paired bootstrap confirms significant gains in 19/24 comparisons (including 11/12 on multi-hop QA and 5/5 over LaRA baselines)

Because three-run intervals only reflect decoding variance, we ran question-level paired bootstrap (50,000 resamples) and two-sided sign-flip tests on the frozen results. For each question and model, we average three runs, average paired differences across the four models (∗^* denotes p<0.05p<0.05; 95% CIs in percentage points):

ComparatorQasper (QA-F1)HotpotQA (QA-F1)2WikiMQA (QA-F1)LaRA (Acc.)
LongLLMLinguaCI [19.18, 27.09], p=.00048∗p=.00048^*CI [10.73, 18.87], p=.00048∗p=.00048^*CI [12.92, 20.30], p=.00048∗p=.00048^*CI [15.26, 20.34], p=.00048∗p=.00048^*
ProvenceCI [0.94, 5.34], p=.0474∗p=.0474^*CI [2.76, 7.44], p=.00048∗p=.00048^*CI [8.00, 14.53], p=.00048∗p=.00048^*CI [5.87, 9.35], p=.00048∗p=.00048^*
HiChunkCI [-0.11, 2.93], p=.3310p=.3310CI [2.62, 6.85], p=.00048∗p=.00048^*CI [1.97, 4.91], p=.00048∗p=.00048^*CI [4.59, 8.19], p=.00048∗p=.00048^*
BRIEF-ProCI [-0.43, 4.25], p=.3310p=.3310CI [-0.29, 6.22], p=.3310p=.3310CI [1.08, 6.63], p=.0474∗p=.0474^*CI [5.84, 9.98], p=.00048∗p=.00048^*
LongRefinerCI [15.18, 24.38], p=.00048∗p=.00048^*CI [9.24, 17.73], p=.00048∗p=.00048^*CI [22.59, 32.78], p=.00048∗p=.00048^*CI [37.17, 43.60], p=.00048∗p=.00048^*
RawCI [-0.004, 1.99], p=.2618p=.2618CI [0.49, 2.99], p=.0474∗p=.0474^*CI [1.03, 3.81], p=.0065∗p=.0065^*CI [-0.67, 1.30], p=.5464p=.5464

Mean gains are positive across all 24 comparisons, and 19/24 are statistically significant (supported by document/question-package sensitivity analysis): 5/5 compression baselines on LaRA, 5/6 comparisons on HotpotQA, 6/6 on 2WikiMQA (including over Raw), and 3/6 on Qasper. These results now appear in §4.2 and Appendix C.1.


W4: Qualitative Analysis of Subtrees — How coherent, bridging, and overbroad subtrees behave across selection and answering

Separating selection quality from reader generation, saved traces (Appendices E.4 & E.5) reveal three representative subtree patterns:

  1. Coherent-mechanism subtree (Selection succeeds without exact boundaries): For Qasper "How are weights dynamically adjusted?", a single subsection node retains all four gold evidence units covering motivation, the (1−p)(1-p) factor, dynamic weighting, and focal loss. Versus Lexical-only, Span Recall rises from 0% to 100% and QA-F1 from 17.57 to 35.33—the subtree captures complete evidence, while the partial F1 reflects reader synthesis rather than tree failure.
  2. Relational / bridging subtree (Multi-hop linking under low lexical overlap): For HotpotQA "Gary L. Bennett was a part of the space missions that have a primary destination of what celestial body?", the subtree retains the bridging Ulysses statement linking Bennett's mission to the Sun, whereas Lexical-only misses the second hop. Span Recall rises from 2.10% to 82.89% and QA-F1 from 0 to 100 (correcting "Jupiter" to "the Sun" across all three runs).
  3. Overbroad / topic-mixed node (Semantic miss repaired by lexical union): For Qasper "How is the dataset of hashtags sourced?", a 15,976-character "Adaptive Multi-task Learning" node buries dataset provenance inside a methods section. Because sis_i and FiF_i omit this secondary topic, semantic scoring assigns 1/10 and prunes it (Semantic-only answers "unanswerable", Span Recall 0%, QA-F1 0). Lexical attribution rescues the node via subunit matches on "Stanford Sentiment", raising Span Recall to 100% and QA-F1 to 71.11. We have incorporated this analysis into Appendices E.4 and E.5.

Closing Remark

Thank you again for your detailed and constructive feedback. We have clarified the tree's relation to prior work and the scoring/serialization interface, added question-level statistical tests, and provided qualitative analyses of coherent, bridging, and overbroad subtrees. We hope these clarifications and results address your concerns. If you find that this evidence strengthens the paper's contribution, we would sincerely appreciate your consideration of a higher rating.

AuthorsOpenReview ↗

Eager to discuss our rebuttal for Submission53818

Dear Reviewer,

Thank you again for your time and constructive feedback. As the discussion period draws to a close, with less than 21 hours remaining, we would like to briefly summarize our responses to your main concerns:

  • Tree representation and interface (W1–W2, Q1–Q2): We clarified the section/topic hierarchy’s distinction from RST and provided a concrete scoring and serialization example: metadata guides selection, while only verbatim source spans are sent to the reader.
  • Statistical reliability (W3): Question-level paired bootstrap and randomization tests show significant gains in 19/24 comparisons, including 11/12 on multi-hop QA.
  • Qualitative analysis (W4): Saved traces illustrate coherent, bridging, and overbroad subtrees, showing how selection preserves evidence, lexical union repairs semantic misses, and reader errors remain distinct from selection failures.

In the revised PDF, we clarified our method’s relationship to prior work and the scoring/serialization interface (§2, §§3.2–3.4; Appendices A.1–A.4), added statistical tests (§4.2; Appendix C.1), and incorporated subtree case analyses (Appendices E.4–E.5).

We would greatly appreciate hearing whether these clarifications and results address your concerns. Please let us know if any points need further explanation.

Dear Reviewer 2BGZ,

With less than 5 hours left in the discussion period, we are eager to hear from you. We have posted a detailed rebuttal and a revised PDF but have not yet received a response, so even a brief reply on whether our answers resolve your concerns would be greatly appreciated!

For your convenience, all four points you raised have already been addressed in our responses above: (1) Tree representation and relation to prior work (W1 & Q1); (2) Node contents, scoring, and text serialization (W2 & Q2); (3) Statistical significance (W3); (4) Qualitative analysis of subtrees (W4).

If these address your concerns, we would be grateful if you could consider updating your assessment accordingly. Thank you again for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Updates and Reviewer Discussion

Dear Senior Area Chairs, Area Chairs, and Reviewers,

Thank you for the time and care you devoted to reviewing Submission 53818. As the discussion period closes, we would like to summarize the main updates and the status of the discussion. Changes incorporated into the revised PDF are highlighted.

Main Rebuttal Updates

We are grateful for the questions raised during review, which helped us strengthen the paper. Below we summarize the experiments and analyses we added during the rebuttal.

  • We clarified the source-grounded section and topic hierarchy and its relation to Rhetorical Structure Theory (2BGZ W1–W2, Q1–Q2). Node metadata guides scoring, and the reader receives verbatim local source spans as evidence.
  • With the scorer and reader held fixed, TreeFuse outperformed alternative representations, including the HiChunk adapter (fxSy W2, Lkw1 W1/W4). A smaller dense scorer also preserved the average QA-F1 advantage over full-context prompting and the evaluated compression baselines on each task.
  • Auditing per-query token counts on the frozen reader inputs showed that TreeFuse and the budget-matched ablations used closely comparable budgets, with the reported ablation gains preserved (Lkw1 W2/Q1).
  • Question-level paired bootstrap and randomization tests found significant gains in most comparisons, including nearly all multi-hop comparisons and all LaRA compression-baseline comparisons. Results on Qasper were more mixed (2BGZ W3, Lkw1 W3).
  • The smaller scorer reduced query-time scoring latency. TreeFuse improved accuracy on documents with sparse headings and nearly matched full-context accuracy on complex layouts while using substantially less context (fxSy W1/W3).
  • We added common-protocol comparisons with PageIndex and DISRetrieval and clarified that the main results use the complete HiChunk pipeline. TreeFuse led the hierarchical baselines in average QA-F1 on each task (Lkw1 Q2).

Review Discussion Status

We appreciate the reviewers’ positive feedback on the tree representation and dual-channel selection (2BGZ S1–S2), the training-free separation of evidence selection from answering (fxSy S1), and the evaluation across answer models (Lkw1 S2). We also summarize below how the exchanges with each reviewer developed during the rebuttal.

  • Reviewer 2BGZ (initial score: 2) recognized the tree representation and empirical results. We clarified the method and added statistical tests and subtree case analyses. We have not yet received a follow-up.
  • Reviewer fxSy (initial score: 2; raised to 3 after the rebuttal) said the rebuttal addressed their concerns about latency, scorer capacity, and robustness on unstructured documents, and raised their rating from weak rejection to weak acceptance.
  • Reviewer Lkw1 (initial score: 3) identified controls needed for a fair comparison. We added scorer and constructor controls, per-query token accounting, statistical tests, and hierarchical baseline comparisons. We have not yet received a follow-up.

We hope this summary accurately reflects the current state of the submission and how the review discussion developed. Thank you for considering the original reviews, the revised PDF, and the supplementary results together in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.