- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper compares several strategies of hierarchical retrieval on the well-known RAPTOR pipeline. The overarching conclusion seems to be that hierarchical retrieval is not better than simple, flat retrieval, seemingly refuting RAPTOR's central claim. This conclusion is not evaluated in a direct comparison on the same datasets (except one) and its validity remains undiscussed. The paper has critical readability issues.
Critical strengths
-
Transparently reported negative results.
-
The paper rises some questions about the overall validity of the RAPTOR hierarchical retrieval approach which might worth of further investigation.
Critical weaknesses
-
The paper is next to incomprehensible and incredibly difficult to follow. The text is very dense but the broader perspective is almost entirely missing, it is hard to get an overall "what was done and why and why is it interesting" view. The paper has over 80 pages in 26 text-rich appendices. It is my assessment that the 9-page main body does not stand on its own. I am unable (and not required) to review these additional 80 pages of content. The incomprehensibility of the totality of the paper is my primary reason to recommend as "clear reject".
-
The paper leans heavily on experiment pre-registration, but as far as I could tell, the pre-registration is internal and in no way externally verifiable (which is also confirmed in the paper). The claims are (as far as I can tell) in Table 31 on page 60, or, rather, the main confirmable spelled out claim is C-1 below that table. That claim turned out to be unsupported. The pre-registration was re-opened and edited after the first experiment on QuALITY came out negative. The experiment was pre-registered as method work, and subsequently re-framed as "audit" upon negative results. The strength here is that this all is reported by the paper and I appreciate that.
-
The main result seems that the tree construction does not in any setting outperform flat retrieval. This is shown on two datasets (in two attempts at registration as far as I understood). In my mind that in practice means that this paper implies that the central claim of the RAPTOR work does not hold, but this does not seem to be explicitly stated nor analyzed in the main text, and most importantly not compared on the datasets used in RAPTOR (except for one). In other words, I do not consider this important implication supported (in a way in which I would be able to grasp it from the main body of the paper).
Questions for authors
Would it be possible to include a thorough analysis of the results on the datasets used in RAPTOR and explain why the conclusion differs from the central result reported in RAPTOR?
5 comments
Response to Reviewer JzCR (2/2)
On RAPTOR's datasets. QuALITY is one of RAPTOR's datasets, and we evaluated its development set, the split RAPTOR's Table 4 reports on (115 articles, 2,086 questions), with RAPTOR's encoder, summarization prompt, clustering constants and 2,000-token retrieval, and an 8B summarizer and reader. Accuracy, averaged over articles, was 0.6164 for flat retrieval, 0.6049 for the GMM+BIC tree, 0.6176 for the Leiden tree and 0.6113 for the DP tree; the registered tree-over-flat test failed for all three. Two further arms outside that test scored 0.6311 (UMAP-matched Leiden) and 0.6146 (DP with confidence reranking). In post hoc two-sided 95% intervals, GMM+BIC minus flat is −0.0115 [−0.0307, +0.0075]. The only one of the five intervals that excludes zero is UMAP-matched Leiden's, +0.0147 [+0.0002, +0.0293], uncorrected for the five comparisons and at the edge of bootstrap noise (all five are in our reply to bGiS). We have also read the same indexes in RAPTOR's reader setting (UnifiedQA-3B with 400 tokens of context; post hoc, details in our reply to bGiS, Q4): GMM+BIC minus flat is −0.0072 [−0.0205, +0.0062] over questions, the unit of RAPTOR's accuracies, and −0.0067 [−0.0200, +0.0066] averaged over articles, so even in RAPTOR's reader setting our tree shows no gain of the size RAPTOR reports (1.7 points, i.e. 0.017, for SBERT). Flat and tree accuracies there are 0.5441 [0.5156, 0.5738] and 0.5369 [0.5081, 0.5669] over questions, intervals that contain RAPTOR's 0.549 without its tree and 0.566 with it (not a test: RAPTOR reports no intervals). Two further post hoc controls rank the same nodes with RAPTOR's other two retrievers: GMM+BIC minus flat over questions is −0.0182 [−0.0334, −0.0029] with BM25 and −0.0158 [−0.0333, +0.0019] with DPR ranked by inner product (RAPTOR reports gains of 0.022 and 0.016). When a node that does not fit is skipped instead of ending the context, both intervals contain zero. No interval in these controls lies above zero; one DPR reading under skip packing is inconclusive (numbers in our reply to bGiS, Q4), and ranked by cosine it shows no gain of RAPTOR's size. Across the three retrievers RAPTOR's accuracies lie inside our intervals except twice (Table 14: BM25 without the tree; DPR with it, ranked by cosine). On the 1,065 development questions QuALITY marks hard (post hoc, App. C.2), GMM+BIC minus flat is −0.0042 [−0.0315, +0.0229] averaged over articles; that subset alone cannot rule out a gain of RAPTOR's size, and its only interval excluding zero is UMAP-matched Leiden's (our reply to bGiS). We did not run NarrativeQA or QASPER; both were set aside before submission. NarrativeQA is scored with free-form metrics (BLEU, ROUGE-L and METEOR in RAPTOR); when we moved to book-length questions, we judged that their scoring noise could hide effects of one to three points and made multiple-choice scoring a requirement. QASPER's documents are single scientific papers; its authors report that 99% of them fit a 16K-token window without truncation (Dasigi et al., 2021), so they sit outside the study's setting, in which no context window sees a whole book. For NarrativeQA we point to Laitenberger et al., who report the same direction as our QuALITY result for RAPTOR's rule, with a GPT-4o-mini reader.
Why our conclusion may differ from RAPTOR's. The revision's §5 lists four differences in setting; in brief: (a) RAPTOR's comparisons with GPT-3 and GPT-4 readers were against other retrievers (BM25, DPR), and its same-retriever ablation gave UnifiedQA-3B 400 tokens of context; our flat baseline is the tree's own retriever at 2,000 tokens. (b) In a small window a summary can carry more of a document than raw chunks; with larger windows and stronger readers that edge can disappear. Laitenberger et al. (EMNLP 2025) report this with GPT-4o and GPT-4o-mini readers on QuALITY, NarrativeQA and ∞Bench, where an exact ablation of RAPTOR without its summaries outperformed RAPTOR. Our own test of (b), above, found no tree gain in RAPTOR's reader setting with any of RAPTOR's three retrievers. (c) Our summaries come from an 8B model, not gpt-3.5-turbo, so we cannot exclude that a stronger summarizer would restore a tree advantage. (d) RAPTOR's code re-clusters a group whose text exceeds 3,500 tokens, while ours truncates the summarization input. No QuALITY input reached that cap, so this difference does not touch the QuALITY results; on NovelQA it falls mostly on the two Leiden trees (App. B). Our reader-setting controls did not restore a tree advantage, so we cannot say which of (a)–(d), if any, accounts for the difference from RAPTOR's reported gains.
The revision adds a section (§5), "Relation to RAPTOR's reported gains", with these results (Table 3) and these four differences, so that how our results relate to RAPTOR's claim is stated and analyzed in the body. We hope the revised body now gives the overall view you found missing; §1, §5 and Table 3 are where to check it.
Follow-up for Reviewer JzCR
Dear Reviewer JzCR,
Thank you again for your review. Your primary reason for rejection is the paper's readability, in particular that the main body does not stand on its own. §1 of the revision now opens with the overall view you found missing (pages 1–2):
- What we did. Inside one RAPTOR-style pipeline we replaced only the clustering step, and compared three grouping rules with flat retrieval and with groupings that use no clustering, on 60 full-length novels.
- What we found. No semantic tree was shown to beat flat retrieval, and the Dirichlet-process index was not shown to beat the other two rules. On QuALITY, one of RAPTOR's benchmarks, RAPTOR's own grouping rule did not outperform flat retrieval in our pipeline either, including with RAPTOR's reader (§5).
- Why it matters. The value of a hierarchy over flat retrieval should be reported together with its construction cost, flat and non-semantic controls, and its variation across rebuilds.
§5 with Table 3 (pages 7–8) states and analyzes how these results relate to RAPTOR's claim.
We would appreciate knowing whether the revised body now gives the overall view you found missing, and we are glad to answer any remaining question while the discussion is open.
Response to Reviewer JzCR (1/2)
Thank you for the review. Your account of the registration history is accurate, and we agree with your main criticism. We start with readability, then turn to RAPTOR.
1. The main body does not stand on its own. We agree, and the revision addresses it directly.
2. Registration history. Your reading is correct on each point. C-1, the confirmatory claim (H1), was not supported (§4). The registration is internal and cannot be verified externally. It was reopened and edited after the QuALITY attempt failed its sanity test. The study was registered as method work, and the audit framing came after the negative result. The claim register says so under C-1, of the registered sentence: "The result did not support it, and the paper built around it is an audit of the assumptions the sentence takes for granted rather than a demonstration of the advantage it names" (App. SM3). Table SM3 (supplementary PDF) places the claims on one time axis: C-4, the framing that defines the per-record failure audit, entered at a later revision than C-1 to C-3 and is in neither registration. Two recorded facts bound what the edit could buy (App. SM2). First, the edit to the primary hypothesis made it stricter: a single comparison (DP vs. Leiden) became a conjunction (DP must beat both Leiden and GMM+BIC). Second, the registration records that a comparator swap, which would have turned the halted QuALITY comparison from a loss into a win, was proposed first and rejected once the QuALITY accuracies were presented; it records proposing the swap without those accuracies as a procedural defect. One change did favour the primary hypothesis, and App. SM2 says so: the predecessor's multiplicity rule, labelled Holm although that label was procedurally incoherent for the fixed levels it assigned, was replaced by the fixed testing sequence before any NovelQA accuracy of this study existed, and the registration's change table records that direction. The other changes, including the falsification conditions and the registered effect size, are listed with their direction in the NovelQA registration's change table (App. C.1).
3. Tree versus flat retrieval, and RAPTOR's central claim. First, the scope of our result. Our result is an absence of evidence, not evidence that trees never help. On NovelQA, no semantic tree was shown to beat flat retrieval at the registered resolution, but a small DP advantage over flat cannot be excluded (estimate +0.0060, one-sided 95% upper limit +0.0266). A grouping by chapter boundaries, which uses the author's structure rather than clustering, scored 0.5837 against 0.5710 for flat retrieval (descriptive, with no registered interval, and read in the paper as establishing no accuracy advantage). Chapters were found in 36 of the 60 books; the other 24 use fixed-block grouping. Two of the stronger flat retrievers scored higher still (rank fusion 0.5922; cross-encoder reranking at large capacity, 0.5853). For RAPTOR's own clustering rule (GMM+BIC) the evidence is sharper: the one-sided 95% upper limit of its advantage over flat is +0.0062 on NovelQA and +0.0044 on QuALITY (post hoc), both below the 0.016 to 0.022 (1.6 to 2.2 points) of RAPTOR's same-retriever ablation on QuALITY (RAPTOR's Table 2). Those limits come from another reader and other averaging units than RAPTOR's; the reading closest to RAPTOR's setting follows in part 2.