acceptodds
Under review as a conference paper at ICLR 2027

A Pre-Registered Audit of Hierarchical Retrieval: Cost, Utility, and Assignment Confidence

10 views

Abstract

Evaluating hierarchical retrieval requires measuring what its structure costs, whether it improves answers, and what its stored confidence can support. We audit these questions inside one RAPTOR-style pipeline. A capacity-constrained (truncated) variational DPMM (Dirichlet-process mixture model) serves as a confidence-bearing audit instrument rather than a proposed method: its fit stores a posterior responsibility per node–cluster pair, and the index keeps a scalar derived from it. In pre-registered comparisons on NovelQA (60 novels, 1,526 questions), the hypothesis that this index outperforms both the GMM+BIC and the Leiden index is not supported under either writer–reader backbone, and the registered comparison with flat dense retrieval does not establish a semantic-tree advantage. We report associations, not causal claims beyond this pipeline. The DP index stores 6,205,729 summary tokens against 1,602,734 for fixed-block contiguous grouping, without an established paired accuracy advantage (0.5770 against 0.5775). Rebuilds under 4 further construction seeds yield median within-book accuracy standard deviations of 0.0368 to 0.0480 over 54 books, at the scale of the registered design target. On an eighteen-book development sample, outside the registration, maximum a posteriori reassignment under sampled variational parameters stays close to the fitted partition (mean adjusted Rand index 0.97), whereas a refit on row-permuted embeddings lands farther (0.72). A fixed confidence reranker lowers book-mean accuracy from 0.5770 to 0.5589, a descriptive result under the registered testing sequence. These results motivate reporting utility beside construction footprint and rebuild variation, and testing stored confidence against both the variation it is meant to track and the decision it is meant to inform.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
retrieval-augmented generation, hierarchical retrieval, RAPTOR, Dirichlet process mixture, variational inference, pre-registration, posterior responsibility, long-document question answering, negative result, evaluation audit

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JzCROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper compares several strategies of hierarchical retrieval on the well-known RAPTOR pipeline. The overarching conclusion seems to be that hierarchical retrieval is not better than simple, flat retrieval, seemingly refuting RAPTOR's central claim. This conclusion is not evaluated in a direct comparison on the same datasets (except one) and its validity remains undiscussed. The paper has critical readability issues.

Critical strengths
  • Transparently reported negative results.

  • The paper rises some questions about the overall validity of the RAPTOR hierarchical retrieval approach which might worth of further investigation.

Critical weaknesses
  • The paper is next to incomprehensible and incredibly difficult to follow. The text is very dense but the broader perspective is almost entirely missing, it is hard to get an overall "what was done and why and why is it interesting" view. The paper has over 80 pages in 26 text-rich appendices. It is my assessment that the 9-page main body does not stand on its own. I am unable (and not required) to review these additional 80 pages of content. The incomprehensibility of the totality of the paper is my primary reason to recommend as "clear reject".

  • The paper leans heavily on experiment pre-registration, but as far as I could tell, the pre-registration is internal and in no way externally verifiable (which is also confirmed in the paper). The claims are (as far as I can tell) in Table 31 on page 60, or, rather, the main confirmable spelled out claim is C-1 below that table. That claim turned out to be unsupported. The pre-registration was re-opened and edited after the first experiment on QuALITY came out negative. The experiment was pre-registered as method work, and subsequently re-framed as "audit" upon negative results. The strength here is that this all is reported by the paper and I appreciate that.

  • The main result seems that the tree construction does not in any setting outperform flat retrieval. This is shown on two datasets (in two attempts at registration as far as I understood). In my mind that in practice means that this paper implies that the central claim of the RAPTOR work does not hold, but this does not seem to be explicitly stated nor analyzed in the main text, and most importantly not compared on the datasets used in RAPTOR (except for one). In other words, I do not consider this important implication supported (in a way in which I would be able to grasp it from the main body of the paper).

Questions for authors

Would it be possible to include a thorough analysis of the results on the datasets used in RAPTOR and explain why the conclusion differs from the central result reported in RAPTOR?

5 comments
AuthorsOpenReview ↗

Response to Reviewer JzCR (1/2)

Thank you for the review. Your account of the registration history is accurate, and we agree with your main criticism. We start with readability, then turn to RAPTOR.

1. The main body does not stand on its own. We agree, and the revision addresses it directly.

  • §1 opens with what we did, what we found and why it matters, in plain language (also in "Summary of our responses").
  • The comparison with RAPTOR (Table 3), the conclusions that held across readers and retrievers (Table 6) and what the release lets a reader verify (Table 7) are now tables in the body.
  • How the study moved from the halted QuALITY attempt to NovelQA, a move decided outside the registration, is now one paragraph of §3.4. That paragraph points to App. C for the registration's chronology (the edits are given in full in App. SM2 of the supplementary PDF) and to the claim register (App. SM3).
  • The appendices are reorganized behind a one-page map: twelve remain in the main PDF (formerly twenty-six; the PDF went from 95 pages to 68), and material given in full moved to a supplementary PDF of 42 pages (see "Summary of the revision").
  • "Arm" is defined at first use, the abstract's "confidence-bearing audit instrument" is replaced, and procedural detail moved out of the body to make room for the RAPTOR comparison below.

2. Registration history. Your reading is correct on each point. C-1, the confirmatory claim (H1), was not supported (§4). The registration is internal and cannot be verified externally. It was reopened and edited after the QuALITY attempt failed its sanity test. The study was registered as method work, and the audit framing came after the negative result. The claim register says so under C-1, of the registered sentence: "The result did not support it, and the paper built around it is an audit of the assumptions the sentence takes for granted rather than a demonstration of the advantage it names" (App. SM3). Table SM3 (supplementary PDF) places the claims on one time axis: C-4, the framing that defines the per-record failure audit, entered at a later revision than C-1 to C-3 and is in neither registration. Two recorded facts bound what the edit could buy (App. SM2). First, the edit to the primary hypothesis made it stricter: a single comparison (DP vs. Leiden) became a conjunction (DP must beat both Leiden and GMM+BIC). Second, the registration records that a comparator swap, which would have turned the halted QuALITY comparison from a loss into a win, was proposed first and rejected once the QuALITY accuracies were presented; it records proposing the swap without those accuracies as a procedural defect. One change did favour the primary hypothesis, and App. SM2 says so: the predecessor's multiplicity rule, labelled Holm although that label was procedurally incoherent for the fixed levels it assigned, was replaced by the fixed testing sequence before any NovelQA accuracy of this study existed, and the registration's change table records that direction. The other changes, including the falsification conditions and the registered effect size, are listed with their direction in the NovelQA registration's change table (App. C.1).

3. Tree versus flat retrieval, and RAPTOR's central claim. First, the scope of our result. Our result is an absence of evidence, not evidence that trees never help. On NovelQA, no semantic tree was shown to beat flat retrieval at the registered resolution, but a small DP advantage over flat cannot be excluded (estimate +0.0060, one-sided 95% upper limit +0.0266). A grouping by chapter boundaries, which uses the author's structure rather than clustering, scored 0.5837 against 0.5710 for flat retrieval (descriptive, with no registered interval, and read in the paper as establishing no accuracy advantage). Chapters were found in 36 of the 60 books; the other 24 use fixed-block grouping. Two of the stronger flat retrievers scored higher still (rank fusion 0.5922; cross-encoder reranking at large capacity, 0.5853). For RAPTOR's own clustering rule (GMM+BIC) the evidence is sharper: the one-sided 95% upper limit of its advantage over flat is +0.0062 on NovelQA and +0.0044 on QuALITY (post hoc), both below the 0.016 to 0.022 (1.6 to 2.2 points) of RAPTOR's same-retriever ablation on QuALITY (RAPTOR's Table 2). Those limits come from another reader and other averaging units than RAPTOR's; the reading closest to RAPTOR's setting follows in part 2.

AuthorsOpenReview ↗

Response to Reviewer JzCR (2/2)

On RAPTOR's datasets. QuALITY is one of RAPTOR's datasets, and we evaluated its development set, the split RAPTOR's Table 4 reports on (115 articles, 2,086 questions), with RAPTOR's encoder, summarization prompt, clustering constants and 2,000-token retrieval, and an 8B summarizer and reader. Accuracy, averaged over articles, was 0.6164 for flat retrieval, 0.6049 for the GMM+BIC tree, 0.6176 for the Leiden tree and 0.6113 for the DP tree; the registered tree-over-flat test failed for all three. Two further arms outside that test scored 0.6311 (UMAP-matched Leiden) and 0.6146 (DP with confidence reranking). In post hoc two-sided 95% intervals, GMM+BIC minus flat is −0.0115 [−0.0307, +0.0075]. The only one of the five intervals that excludes zero is UMAP-matched Leiden's, +0.0147 [+0.0002, +0.0293], uncorrected for the five comparisons and at the edge of bootstrap noise (all five are in our reply to bGiS). We have also read the same indexes in RAPTOR's reader setting (UnifiedQA-3B with 400 tokens of context; post hoc, details in our reply to bGiS, Q4): GMM+BIC minus flat is −0.0072 [−0.0205, +0.0062] over questions, the unit of RAPTOR's accuracies, and −0.0067 [−0.0200, +0.0066] averaged over articles, so even in RAPTOR's reader setting our tree shows no gain of the size RAPTOR reports (1.7 points, i.e. 0.017, for SBERT). Flat and tree accuracies there are 0.5441 [0.5156, 0.5738] and 0.5369 [0.5081, 0.5669] over questions, intervals that contain RAPTOR's 0.549 without its tree and 0.566 with it (not a test: RAPTOR reports no intervals). Two further post hoc controls rank the same nodes with RAPTOR's other two retrievers: GMM+BIC minus flat over questions is −0.0182 [−0.0334, −0.0029] with BM25 and −0.0158 [−0.0333, +0.0019] with DPR ranked by inner product (RAPTOR reports gains of 0.022 and 0.016). When a node that does not fit is skipped instead of ending the context, both intervals contain zero. No interval in these controls lies above zero; one DPR reading under skip packing is inconclusive (numbers in our reply to bGiS, Q4), and ranked by cosine it shows no gain of RAPTOR's size. Across the three retrievers RAPTOR's accuracies lie inside our intervals except twice (Table 14: BM25 without the tree; DPR with it, ranked by cosine). On the 1,065 development questions QuALITY marks hard (post hoc, App. C.2), GMM+BIC minus flat is −0.0042 [−0.0315, +0.0229] averaged over articles; that subset alone cannot rule out a gain of RAPTOR's size, and its only interval excluding zero is UMAP-matched Leiden's (our reply to bGiS). We did not run NarrativeQA or QASPER; both were set aside before submission. NarrativeQA is scored with free-form metrics (BLEU, ROUGE-L and METEOR in RAPTOR); when we moved to book-length questions, we judged that their scoring noise could hide effects of one to three points and made multiple-choice scoring a requirement. QASPER's documents are single scientific papers; its authors report that 99% of them fit a 16K-token window without truncation (Dasigi et al., 2021), so they sit outside the study's setting, in which no context window sees a whole book. For NarrativeQA we point to Laitenberger et al., who report the same direction as our QuALITY result for RAPTOR's rule, with a GPT-4o-mini reader.

Why our conclusion may differ from RAPTOR's. The revision's §5 lists four differences in setting; in brief: (a) RAPTOR's comparisons with GPT-3 and GPT-4 readers were against other retrievers (BM25, DPR), and its same-retriever ablation gave UnifiedQA-3B 400 tokens of context; our flat baseline is the tree's own retriever at 2,000 tokens. (b) In a small window a summary can carry more of a document than raw chunks; with larger windows and stronger readers that edge can disappear. Laitenberger et al. (EMNLP 2025) report this with GPT-4o and GPT-4o-mini readers on QuALITY, NarrativeQA and ∞Bench, where an exact ablation of RAPTOR without its summaries outperformed RAPTOR. Our own test of (b), above, found no tree gain in RAPTOR's reader setting with any of RAPTOR's three retrievers. (c) Our summaries come from an 8B model, not gpt-3.5-turbo, so we cannot exclude that a stronger summarizer would restore a tree advantage. (d) RAPTOR's code re-clusters a group whose text exceeds 3,500 tokens, while ours truncates the summarization input. No QuALITY input reached that cap, so this difference does not touch the QuALITY results; on NovelQA it falls mostly on the two Leiden trees (App. B). Our reader-setting controls did not restore a tree advantage, so we cannot say which of (a)–(d), if any, accounts for the difference from RAPTOR's reported gains.

The revision adds a section (§5), "Relation to RAPTOR's reported gains", with these results (Table 3) and these four differences, so that how our results relate to RAPTOR's claim is stated and analyzed in the body. We hope the revised body now gives the overall view you found missing; §1, §5 and Table 3 are where to check it.

AuthorsOpenReview ↗

Follow-up for Reviewer JzCR

Dear Reviewer JzCR,

Thank you again for your review. Your primary reason for rejection is the paper's readability, in particular that the main body does not stand on its own. §1 of the revision now opens with the overall view you found missing (pages 1–2):

  • What we did. Inside one RAPTOR-style pipeline we replaced only the clustering step, and compared three grouping rules with flat retrieval and with groupings that use no clustering, on 60 full-length novels.
  • What we found. No semantic tree was shown to beat flat retrieval, and the Dirichlet-process index was not shown to beat the other two rules. On QuALITY, one of RAPTOR's benchmarks, RAPTOR's own grouping rule did not outperform flat retrieval in our pipeline either, including with RAPTOR's reader (§5).
  • Why it matters. The value of a hierarchy over flat retrieval should be reported together with its construction cost, flat and non-semantic controls, and its variation across rebuilds.

§5 with Table 3 (pages 7–8) states and analyzes how these results relate to RAPTOR's claim.

We would appreciate knowing whether the revised body now gives the overall view you found missing, and we are glad to answer any remaining question while the discussion is open.

Reviewer_jMPQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits hierarchical retrieval within a RAPTOR-style pipeline, examining clustering choices, construction costs, rebuild stability, and assignment confidence. Its pre-registered experiments do not establish an advantage for the DP mixture index. Additional analyses identify substantial construction overhead and variability across rebuilds.

Critical strengths

1.The paper reports negative results without claiming equivalence.

2.Including non-semantic retrieval controls and construction costs improves the comparison.

Critical weaknesses
  1. The conclusions are difficult to generalize beyond the specific pipeline. The confirmatory experiments primarily use NovelQA, a single 8B reader, and one RAPTOR-style implementation. Several broader comparisons, including larger-reader probes, are exploratory. Since retrieval quality depends on the reader and context utilization, the results do not establish whether the observed limitations are properties of hierarchical retrieval or of this particular configuration. 

  2. The audit's reproducibility evidence is incomplete. Section 7 acknowledges adaptive benchmark selection, registration chronology documented through internal ledgers, and an unauditable code difference across rebuilds. These limitations are particularly consequential because pre-registration and rebuild stability are central to the paper's contribution. The current evidence does not fully support independent replication of those claims.

Questions for authors

1.Which conclusions remain stable across different readers and retrieval configurations?

2.How much of the measured rebuild variation could be affected by the acknowledged code differences?

3.Can the pre-registration chronology and reported rebuild results be independently verified from released artifacts?

3 comments
AuthorsOpenReview ↗

Response to Reviewer jMPQ (1/2)

Thank you for the review. Your summary matches the paper's claims, and your two weaknesses name its real limits. We answer the three questions first, since they carry the substance.

Q1. Which conclusions remain stable across readers and retrieval configurations? On NovelQA we varied the writer–reader backbone (two), the reader size (8B and, post hoc, 70B) and the flat retriever (dense retrieval and three stronger ones). What held in every configuration we ran (the 70B values are post hoc and exploratory; an interval covering zero there is not distinguished at this n):

  1. No grouping rule separates from the others. DP minus Leiden and DP minus GMM+BIC, two-sided 95% intervals over 60 books: Llama-3-8B +0.0096 [−0.0114, +0.0306] and +0.0164 [−0.0050, +0.0378]; Qwen2.5-7B +0.0045 [−0.0168, +0.0259] and +0.0036 [−0.0146, +0.0217]. When Llama-3-70B re-reads the frozen indexes (post hoc, exploratory), DP minus Leiden is +0.001 [−0.019, +0.020] (two-sided 95% interval), against +0.010 [−0.011, +0.031] for the 8B reader on the same questions (App. J; in full in App. SM9 of the supplementary PDF).
  2. No tree was shown to beat flat retrieval. The registered per-arm tests of each tree against flat dense retrieval all fail. Under the 70B reader (post hoc), DP minus flat is −0.009 [−0.031, +0.013] (8B reader: +0.006 [−0.019, +0.031]). Three stronger flat retrievers (dense–lexical rank fusion, and cross-encoder reranking at a base and, unregistered, a large capacity) each score above flat dense retrieval, the Leiden index and the DP index in point estimate. Of these nine paired intervals, one excludes zero, in the flat retriever's favour: rank fusion over the Leiden index, +0.0247 [+0.0046, +0.0449] (two-sided 95%).
  3. Retrieval helps under both readers. Every retrieval condition in Table 1 scores far above closed-book (book-mean accuracy 0.5589–0.5837 vs. 0.3845). DP minus closed-book is +0.193 [+0.143, +0.242] with the 8B reader and +0.113 [+0.070, +0.157] with the 70B reader (post hoc).

What is not stable: transfer to other benchmarks splits in sign (one interval excludes zero against DP and one for it, in different slices), and rebuilding changes chosen answers (Q2). What we did not test: a stronger summarizer (every summary comes from a 7B or 8B model) and other hierarchical pipelines. These are therefore properties of this configuration, checked across the variations above. The abstract already reports "associations, not causal claims beyond this pipeline", and the revision adds this list to the body as a table (Table 6).

AuthorsOpenReview ↗

Response to Reviewer jMPQ (2/2)

Q2. How much of the rebuild variation could the code differences affect? We cannot bound it from the code side, which is why §6 does not attribute all of the observed variation to the construction seed. The rebuilds ran under three code revisions, two of them with uncommitted changes, so no diff reconstructs those two. What can be recovered is the committed span between the revisions: among the files on the answer path it reaches only harness/instrumentation.py, adding six lines that extend the set of dataset names the record schema accepts (App. SM2 of the supplementary PDF, Table SM1). From the data side, every rebuild record carries its revision string, so we recounted the answer flips on the 54 books, splitting seed pairs by whether their two records carry the same string (post hoc, descriptive). Same string vs. different strings: GMM+BIC 20.0% vs. 20.1%, Leiden 13.5% vs. 14.1%, UMAP-matched Leiden 14.1% vs. 14.7%, DP 20.3% vs. 19.3% (all pairs 13.7–20.0%). Pairs that include the original seed-224 build, which ran under other revisions, all fall in the second group; among rebuild pairs alone, different-string pairs flip at 19.6% (GMM+BIC), 9.8% (UMAP-matched Leiden) and 18.0% (DP), and Leiden has none. Holding the committed revision fixed therefore does not reduce the flips. It cannot rule out the uncommitted changes: two of the three revisions behind these rebuild records carry them, and a revision string does not record what they were, so sharing one of those two strings does not guarantee identical code. Like the flips themselves, this count needs the reader's chosen option, which the release withholds (Q3). The revision adds it to App. SM2.

Q3. Can the chronology and the rebuild results be verified independently from the released artifacts? The chronology cannot. It rests on our version control and run ledgers, and no third-party timestamp anchors it (§3.4, App. C). If the paper is accepted, the camera-ready release adds the original-language protocol with its full commit history, but commit times are self-recorded, so even that will not change this answer. What can be verified now: (i) the accuracies of Table 1's first seven rows and the two primary contrasts (DP minus Leiden, DP minus GMM+BIC) with their two-sided intervals re-aggregate from the released per-query ledgers with one command (python -m scripts.reaggregate_from_release; other pairs via --contrast); (ii) every per-query record produced under the frozen NovelQA protocol, for both backbones, carries that protocol's hash, every rebuild record carries the hash of the second registration that governs the rebuilds, and the released driver refuses to score unless the protocol is frozen and the hyperparameter file matches its registered hash; (iii) the sixteen rebuilt ledgers (four arms × four seeds) ship beside the seed-224 ledgers, so the per-book accuracy spreads of Table 4 re-aggregate from them. What cannot be re-derived from the release alone: the answer flips need the reader's chosen option, which the released records withhold because, next to correctness, it would reveal NovelQA's private answer key; the partition-agreement intervals are printed as recorded in the shipped analysis summary; and the protocol hashes cannot be recomputed during review, because they name the original-language files, which are released with the camera-ready version. The revision adds a table with the main part of this split (Table 7).

Weakness 1 (generality). We agree. Our claims are scoped to this pipeline (abstract and §8), and Q1 lists what we varied and what we did not. On NovelQA, no grouping rule separated from the others and no tree was shown to beat a flat retriever under any reader or flat retriever we tried; the one interval that excluded zero favoured a flat retriever (Q1, item 2).

Weakness 2 (reproducibility evidence). We agree that the chronology is the weakest link and that it cannot be verified independently. On adaptive benchmark selection: the move from QuALITY to NovelQA was decided after the QuALITY outcome, and the registration's change table records it as an outcome-dependent decision (App. C.1). It did not buy a positive result: on NovelQA neither the primary hypothesis nor any registered tree-over-flat test is supported, and the QuALITY accuracies of all six arms are reported with intervals (our reply to bGiS). The rebuild results re-aggregate from released ledgers, except for the parts named under Q3. The revised supplementary material adds the QuALITY per-query ledgers, from which the registered attempt's intervals in Table 3 re-aggregate, given QuALITY's public development file.

AuthorsOpenReview ↗

Follow-up for Reviewer jMPQ

Dear Reviewer jMPQ,

Thank you again for your review. As we said in our response, we agree with both weaknesses; the revision does not remove them but states them in the body. Table 6 (page 10) lists what held across the readers and flat retrievers we varied, what did not hold and what we did not vary (the summarizer and the pipeline). Table 7 (page 10) separates what the release lets a reader verify from what it does not; the chronology is among what it does not.

On your second question, holding the committed code revision fixed does not reduce the answer flips, but this cannot rule out an effect of the uncommitted changes (App. SM2 of the supplementary PDF).

We would value your view on whether the claims, as now scoped in Tables 6 and 7, are supported by the evidence, and we are glad to answer any remaining question while the discussion is open.

Reviewer_bGiSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper provides a reproducibility study ("audit") of RAPTOR. They could not reproduce the original paper's findings on a different dataset.

Critical strengths
  1. This paper presents a reproducibility study demonstrating a case where findings from an existing prominent method (RAPTOR) do not hold. Reproducibility studies are important for scientific progress.
Critical weaknesses
  1. The paper is incredibly challenging to follow. As a reader, I'm constantly asked to jump around 80 pages of appendices. Even with so much supplementary material, simple questions (such as with whom the study was pre-registered) remain unanswered. The very brief related work context further hinders reader understanding. Imprecise jargon (e.g., "confidence-bearing audit instrument", "Leiden arm", etc.) that is not consistent with the original work hampers understanding. This is critical to my assessment because clear communication is central to scientific dissemination.
  2. The reproducibility study design itself is poor. Strong reproducibility/replication studies first establish that their implementation is correct by validating in settings as close as possible to the original ones, then conduct additional ablations (clearly stating what is being tested). When results fail to reproduce, it is common to engage with the paper authors to diagnose what might have gone wrong. These procedures help ensure that meaningful conclusions can be drawn. This paper does not do any of this. (Or maybe it does, but it's not clear at all because of (1).) This is critical to the fundamental scientific value of the paper itself.
Questions for authors
  • The paper repeatedly asserts pre-registration. Where was the study pre-registered?
  • When your reproducibility attempt failed, did you connect with the original authors to see what may have gone wrong?
  • What artifacts did you use from the original work, and what were different?
  • The original RAPTOR paper showed that it beat a flat DPR -- what's different about your setting that invalidates this finding?
6 comments
AuthorsOpenReview ↗

Response to Reviewer bGiS (1/3)

Thank you for the review. We agree with your first point: the submitted text is hard to follow, and we have revised it (changes listed in "Summary of the revision"). On your second point, the study does contain a check on RAPTOR's own data, which shares several of RAPTOR's components but differs in its summarizer, reader and clustering step (Q3), and the submission left it in an appendix. We have since added three controls in RAPTOR's own reader setting, one for each retriever of its Table 2 (Q4). Your four questions bear directly on both points, so we answer them first.

Q1. Where was the study pre-registered? Internally, not with a public registry. The protocol (2,198 lines; SHA-256 8e40dfc4…) was frozen before the NovelQA answer key was opened. It was not deposited with OSF or a similar service, and no third-party timestamp anchors its hash (§3.4, App. C). The hash names the original-language file, which is withheld during review and released with the camera-ready version; the review release carries an English rendering and an anonymized copy (experiments/pre_reg/). What a reader can check now: the released confirmatory driver refuses to score unless the protocol's status is frozen and the hyperparameter file matches the hash the protocol registers. All per-query records produced under the protocol carry its hash. A reader cannot check the order in time; that rests on our own version control and run ledgers. Because "pre-registered" suggests a public registry, the revision qualifies it where the abstract first uses it ("In internally pre-registered comparisons (hash-pinned, without a third-party timestamp)"), and §1 adds that the hash was pinned before the answer key was opened; the title is unchanged.

Q2. Did you contact the original authors? No. Our registered question was which grouping rule to use inside RAPTOR, not whether RAPTOR's numbers reproduce, so we did not treat the tree-versus-flat outcome as a failed reproduction to diagnose with its authors. A reproduction claim would call for that step, and the paper does not make one. Our clustering constants and summarization prompt come from RAPTOR's released code (Q3), and an independent group found the same direction with GPT-4o and GPT-4o-mini readers (Q4).

Q3. What did you take from the original work, and what differs? Taken from RAPTOR: the summarization prompt, verbatim; the multi-qa-mpnet-base-cos-v1 encoder; the two-stage (global, then local) UMAP and Gaussian-mixture clustering with BIC model selection, with the 0.1 membership threshold and the seed constant 224 of RAPTOR's released implementation; leaf chunks of about 100 tokens that end at sentence boundaries; and collapsed-tree retrieval with a 2,000-token budget (about the top 20 nodes). On QuALITY we also used the split RAPTOR's Table 4 reports on, the development set. Different: One open model, Llama-3-8B-Instruct, writes the summaries and answers the questions (Qwen2.5-7B-Instruct in the robustness check); RAPTOR used gpt-3.5-turbo for summaries and GPT-3, GPT-4 or UnifiedQA-3B as readers. Our summaries are capped at 256 tokens, and RAPTOR's code caps them at 100. Our sentence splitter differs from RAPTOR's. In the clustering step, RAPTOR's code re-clusters any group whose text exceeds 3,500 tokens and leaves out a node whose membership probability exceeds 0.1 for no group; ours truncates the summarization input at 3,500 tokens and puts such a node in its most probable group. On QuALITY no summarization input reached 3,500 tokens, so the cap never acted there. On NovelQA, primary backbone, it truncated the input of 4.3% of the GMM+BIC tree's summaries and 2.7% of the DP tree's, but 39.2% of the Leiden tree's and 28.3% of the UMAP-matched Leiden tree's, so this difference from RAPTOR's code falls most on the two Leiden trees; we have not measured its effect on accuracy. RAPTOR's code seeds the mixture fits (224 for the BIC search, 0 for the final fit) but not UMAP, while we seed all three with 224. Our reader prompt is one multiple-choice template shared by every condition, decoding is greedy, and the confirmatory benchmark is NovelQA, which is not one of RAPTOR's three datasets. App. B lists the frozen settings; the revision adds the clustering-step differences to it with the truncation rate of every tree (Table 9), and states the Leiden case among the limitations in the main text (§8).

What is new beyond "did not reproduce". The tree-versus-flat comparison itself is not new: RAPTOR's same-retriever ablation (its Table 2) and the exact ablation of Laitenberger et al. make it too. Beyond it, the paper measures (i) construction cost beside accuracy for every index, where a fixed-block grouping without clustering was not separated from the Dirichlet-process index at about a quarter of its summary tokens; and (ii) how much an index's answers change when it is rebuilt with another seed (13.7–20.0% of comparable seed pairs).

AuthorsOpenReview ↗

Response to Reviewer bGiS (2/3)

Q4. RAPTOR beat flat DPR. What is different in your setting? We do not claim that RAPTOR's result is wrong in its own setting. Three differences matter here. (a) What "flat" meant. RAPTOR's comparisons across readers set its tree against BM25 and DPR, retrievers other than the one inside the tree (QuALITY development set with GPT-3: 62.4 vs. 60.4 for DPR; RAPTOR's Table 4). Its same-retriever ablation (each retriever with vs. without RAPTOR; RAPTOR's Table 2) gave the reader, UnifiedQA-3B, 400 tokens of context. Our flat baseline is the tree's own retriever over the same chunks at 2,000 tokens, so the hierarchy is the only difference. (b) Reader and context size. In a 400-token window a summary can carry more of a document than a few raw chunks; with a larger window and a stronger reader that edge can shrink. Laitenberger et al. (EMNLP 2025) report this: an exact ablation of RAPTOR that drops the summaries outperformed RAPTOR on QuALITY (GPT-4o reader), NarrativeQA (GPT-4o-mini) and ∞Bench (both), across retrieval sizes, and they relate RAPTOR's original advantage to its 400-token UnifiedQA setting. (c) The summarizer, which we did not vary. Our summaries come from an 8B model, not gpt-3.5-turbo, and our post hoc larger-reader probe (App. J) changes only the reader, so we cannot exclude that a stronger summarizer would restore a tree advantage; the revision states in §5 that our summarizer was not varied. (Laitenberger et al. built RAPTOR's trees with GPT-4o-mini summaries.)

Our test of (b). We have since tested the constrained-reader half of (b), post hoc, in three controls, one for each retriever of RAPTOR's Table 2. Each protocol was fixed before its full run (internal, like our registration; its hash is recorded in the run's manifest); the second and third were specified after the earlier results were known. RAPTOR's reader, UnifiedQA-3B (the checkpoint RAPTOR's code loads), read our QuALITY indexes with 400 tokens of context packed by RAPTOR's own rule. Each difference is GMM+BIC tree minus flat with a two-sided 95% interval (article-clustered bootstrap), uncorrected for multiplicity.

RetrieverOver questions (RAPTOR's unit)Averaged over articlesGain RAPTOR reports
SBERT (the trees' own)−0.0072 [−0.0205, +0.0062]−0.0067 [−0.0200, +0.0066]+0.017
BM25−0.0182 [−0.0334, −0.0029]−0.0170 [−0.0323, −0.0014]+0.022
DPR (inner product)−0.0158 [−0.0333, +0.0019]−0.0142 [−0.0318, +0.0036]+0.016

With none of the three retrievers does our tree show a gain, and every upper limit is below the gain RAPTOR reports; with BM25 the interval lies below zero. RAPTOR's packing rule ends the context at the first node that does not fit, and our summaries are longer (the GMM+BIC tree's average 241 tokens against the 124.4 of RAPTOR's QuALITY trees; released BM25 transcript, RAPTOR's Table 10). When such a node is skipped instead, the BM25 interval contains zero (−0.0091 [−0.0248, +0.0067] over questions, −0.0082 [−0.0238, +0.0077] over articles), and so does DPR's (+0.0000 [−0.0149, +0.0154] and +0.0012 [−0.0141, +0.0167]). The last upper limit reaches RAPTOR's 0.016, so that reading (DPR over articles, skip packing) is inconclusive; with cosine in place of the inner product, that reading shows no gain of RAPTOR's size. No interval in any control lies above zero.

Levels. Over questions, flat and tree reach 0.5441 [0.5156, 0.5738] and 0.5369 [0.5081, 0.5669] with SBERT, where RAPTOR reports 0.549 and 0.566. Against flat DPR (RAPTOR's Table 4, UnifiedQA: 0.539), our SBERT tree lies between our 0.5431 (inner product) and 0.5302 (cosine). Across retrievers RAPTOR's values lie inside our intervals except twice (Table 14: BM25 without the tree; DPR with it, when DPR ranks by cosine). RAPTOR reports no intervals, so this is not a test, and we read no cause into these levels.

What the controls do not settle. They score our trees under RAPTOR's reader, budget and packing rule; they do not rebuild RAPTOR's trees. Our trees were clustered on SBERT embeddings, and BM25 and DPR only rank their nodes; RAPTOR's paper does not say whether the trees behind its BM25 and DPR rows were built differently. What else differs is listed in (c) and Q3. Reader, budget and input format changed together, so no result is attributed to one of them. Many of the inputs are longer than UnifiedQA's 512-token maximum context (RAPTOR's §3), as they would be in RAPTOR's code, which does not truncate them. On the questions whose input fits within 512 tokens in every arm (1,481 of 2,086 with SBERT), the difference is −0.0108 [−0.0267, +0.0052] over questions (−0.0110 [−0.0270, +0.0050] over articles). For the same contrast, the BM25 subset intervals stay below zero and DPR's contain zero (Table 13). The controls are in §5, and in full in App. C.2 (Tables 13 and 14; every contrast: Table SM2, supplementary PDF); the six-arm table is in part 3.

AuthorsOpenReview ↗

Response to Reviewer bGiS (3/3)

Results on QuALITY's development set (115 articles, 2,086 questions; accuracy averaged over articles, the unit of our registered analysis; tree − flat with post hoc two-sided 95% intervals from an article-clustered bootstrap). Registered attempt: Llama-3-8B-Instruct as summarizer and reader, 2,000-token retrieval. RAPTOR's reader setting: the same indexes read by UnifiedQA-3B with 400 tokens and SBERT retrieval (post hoc). Table 3 of the revision; accuracies over questions are in its Table 15.

ArmRegistered attemptTree − flatRAPTOR's reader settingTree − flat
Flat retrieval0.6164n/a0.5429n/a
GMM+BIC tree (RAPTOR's rule)0.6049−0.0115 [−0.0307, +0.0075]0.5362−0.0067 [−0.0200, +0.0066]
Leiden tree0.6176+0.0013 [−0.0137, +0.0162]0.5454+0.0025 [−0.0075, +0.0129]
DP-mixture tree0.6113−0.0051 [−0.0240, +0.0132]0.5434+0.0005 [−0.0124, +0.0134]
UMAP-matched Leiden*0.6311+0.0147 [+0.0002, +0.0293]0.5399−0.0029 [−0.0138, +0.0082]
DP with confidence reranking*0.6146−0.0018 [−0.0200, +0.0158]0.5415−0.0014 [−0.0139, +0.0110]

*Outside the registered test. The registered one-sided test of tree over flat failed for the first three trees (one-sided 95% lower bounds −0.0275, −0.0114, −0.0208). The post hoc intervals come from the same article-clustered bootstrap and are not corrected for five comparisons. Only the UMAP-matched Leiden interval excludes zero, and narrowly: over 300 further bootstrap seeds its lower limit ranges from −0.0002 to +0.0009. In RAPTOR's reader setting with SBERT retrieval none of the five intervals excludes zero, in either unit. For RAPTOR's own rule, the one-sided 95% upper limit of the advantage over flat is +0.0044 here (post hoc) and +0.0062 on NovelQA (a registered co-report), below the 0.016 to 0.022 (1.6 to 2.2 points) of RAPTOR's same-retriever ablation on QuALITY (RAPTOR's Table 2). Those two limits come from another reader and other averaging units than RAPTOR's; the closest figures are the UnifiedQA readings of Q4 (with SBERT, a two-sided upper limit of +0.0062 over questions).

QuALITY's hard questions (post hoc; App. C.2, Table 16). RAPTOR also reports QuALITY's hard subset. On the development set's 1,065 hard questions in the registered attempt, GMM+BIC minus flat is −0.0042 [−0.0315, +0.0229] averaged over articles. Its one-sided 95% upper limit is +0.0186 (+0.0173 over questions), above 0.017, so these questions alone do not rule out a gain of RAPTOR's size. The only interval there that excludes zero, in both units, is UMAP-matched Leiden's (+0.0297 [+0.0088, +0.0514] over articles), again uncorrected for five comparisons.

Weakness 1 (readability, jargon, related work). You are right; this is the main problem of the submitted version. The revision opens §1 with what we did, what we found and why it matters, and defines "arm" at first use (each way of building the index; every arm answers every question). The abstract replaces "confidence-bearing audit instrument" with "a measuring instrument, not a proposed method". The related work adds hierarchical-retrieval successors of RAPTOR and recent comparisons of structured against flat retrieval, and now says which reported gains our arms test. This answers your point only in part: the related work stays short in the body, and the fuller account remains in App. A. The appendices are reorganized behind a one-page map (see "Summary of the revision"). "Leiden arm" is our term, not RAPTOR's: RAPTOR clusters only with its Gaussian mixture, and the Leiden index is a comparator we added; §1 now names each grouping rule before it defines "arm". The comparison with RAPTOR, the conclusions that held across readers and retrievers, and what the release lets a reader verify are now tables in the body (Tables 3, 6 and 7).

Weakness 2 (study design). The standard you describe is the right one for a reproduction study, which this paper does not claim to be (Q2). The part that shares the most with RAPTOR's setting is our first registered attempt together with the post hoc reader controls on its indexes (Q3–Q4). The attempt ran our pipeline with RAPTOR's constants on the QuALITY split of RAPTOR's Table 4; the controls read the attempt's indexes with the reader of RAPTOR's Table 2 and each of that table's three retrievers. The attempt's registered sanity test, that a tree beats flat retrieval, failed for every registered tree arm. The registered rule then called for a recheck of the pipeline, not a stop; the recheck found no defect, and no registered branch covered that case, so the decision to halt and continue on NovelQA was taken outside the registration (App. C.1). The revision tells this history in a paragraph of §3.4.

We hope this answers your questions, and we would be glad to clarify anything that remains.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of our responses

We thank the reviewers for their questions. Two reviews (bGiS, JzCR) name readability among their main reasons to reject, and the same two ask how our results relate to RAPTOR's. We agree on both counts.

What we did. Inside one RAPTOR pipeline we replaced only the clustering step. We compared three ways of grouping passages (RAPTOR's Gaussian mixture with BIC, Leiden graph clustering, and a Dirichlet-process mixture) with flat retrieval over the same chunks and with groupings that use no clustering, on 60 full-length novels (NovelQA, 1,526 questions). We call each way of building the index an "arm"; every arm answers every question. Hypotheses and analysis were fixed in a protocol frozen and hash-pinned before the answer key was opened; this registration is internal and carries no third-party timestamp.

What we found. On NovelQA, no semantic tree was shown to beat flat retrieval, and the Dirichlet-process index was not shown to beat the other two clustering rules. A fixed-block grouping with no clustering was not separated from the Dirichlet-process index while storing about a quarter of its summary tokens (1.6M vs. 6.2M). Rebuilding an index with another random seed changed the chosen answer on 13.7–20.0% of comparable seed pairs.

Why it matters. The value of a hierarchy over flat retrieval should be reported together with its construction cost, flat and non-semantic controls, and its variation across rebuilds. In our setting the gain over flat is not established, and across rebuilds the median within-book standard deviation of accuracy is 3.7 to 4.8 points for the four trees, the scale of the 3.4-point effect the study was sized to detect.

On RAPTOR. The study was not designed as a reproduction of RAPTOR, but it contains a check on RAPTOR's own data. Before NovelQA, we ran our pipeline on QuALITY's development set (115 articles, 2,086 questions), the split RAPTOR's Table 4 reports on. It used RAPTOR's encoder, summarization prompt, clustering constants and 2,000-token collapsed-tree retrieval, with an open 8B model as summarizer and reader. Flat retrieval scored 0.6164 and RAPTOR's GMM+BIC tree 0.6049 (accuracy averaged over articles; difference −0.0115, post hoc two-sided 95% interval [−0.0307, +0.0075]). The registered test of tree over flat failed for all three registered tree arms. Laitenberger et al. (EMNLP 2025) report the same direction on QuALITY, NarrativeQA and ∞Bench with GPT-4o and GPT-4o-mini readers. We do not claim that RAPTOR's results are wrong in their setting; our replies to bGiS and JzCR list what differs. We have also re-read our QuALITY indexes, post hoc, in RAPTOR's reader setting (UnifiedQA-3B with 400 tokens of context), with each retriever of RAPTOR's Table 2 (SBERT, BM25, DPR). With none of them did RAPTOR's GMM+BIC tree show a gain over flat retrieval. Under RAPTOR's packing rule (a node that does not fit ends the context), the upper limit of each difference is below the gain RAPTOR reports. When such a node is skipped instead, one DPR reading (inner product, averaged over articles) is inconclusive, and with cosine it shows no gain of RAPTOR's size (our reply to bGiS, Q4).

Revision. The revised paper is uploaded; a separate comment, "Summary of the revision", lists what changed.

AuthorsOpenReview ↗

Summary of the revision

The revised paper is uploaded. New or rewritten text is in blue; moved text is not, in either PDF.

Body (10 pages).

  1. §1 opens with three paragraphs on what we did, what we found and why it matters (the content of our summary of responses), and defines "arm" at first use.
  2. A new §5, "Relation to RAPTOR's reported gains", with Table 3: the QuALITY development results of all six arms we ran, including the two outside the registered test, with post hoc intervals, and the same indexes read, post hoc, in RAPTOR's reader setting (UnifiedQA-3B, 400 tokens). It also reports two further post hoc controls that rank the same nodes with BM25 and DPR, RAPTOR's other two retrievers, and lists what still differs from RAPTOR's setting.
  3. The title is unchanged. In the text, "pre-registered" is qualified where the abstract first uses it ("In internally pre-registered comparisons (hash-pinned, without a third-party timestamp)"), and §1 adds that the hash was pinned before the answer key was opened. A new paragraph in §3.4 tells how the study moved from QuALITY to NovelQA.
  4. The abstract replaces "confidence-bearing audit instrument" with "a measuring instrument, not a proposed method".
  5. §2 adds hierarchical-retrieval successors of RAPTOR and recent comparisons of structured against flat retrieval, and says which reported gains our arms test.
  6. §8 adds two tables, on what held across backbones, readers and flat retrievers (Table 6) and on what the release lets a reader verify (Table 7), and states the clustering-step difference from RAPTOR's code among the limitations.

Appendices.

  1. The appendices are reorganized. The main PDF (68 pages; the submitted PDF had 95) keeps twelve (A–L, formerly twenty-six) behind a one-page map that says what each holds and whether its analyses were registered. Material given in full moved to a supplementary PDF in the supplementary material (App. SM1–SM12, 42 pages), among it the claim register (SM3), the rebuild flips split by code revision and the reranker's candidate filter (SM2), and the analyses outside the registered NovelQA evaluation (SM7–SM11).
  2. App. B gives the clustering-step differences from RAPTOR's code with the truncation rate of every tree (Table 9). App. C.2 gives the three RAPTOR-reader controls (Tables 13 and 14; every contrast in Table SM2) and QuALITY hard-question intervals (Table 16).
  3. Moved from the body: the freeze and unsealing sentences (§3.4 to App. C), the paragraph "Descriptive read-outs and directional cells" and two further sentences (§4 to App. D), and one limitation sentence (submitted §7, now §8, to App. K). Their wording is unchanged, except that one sentence drops, and one adapts, a cross-reference to the appendix that now holds it. The checks on the stability gap are given in full in App. H.

Supplementary material.

  1. The supplementary material now also holds the supplementary PDF, the QuALITY per-query ledgers, the records of all three reader controls (with the protocols, code and analysis transcripts of the BM25 and DPR controls), and the analysis scripts with their transcripts; its REPRODUCING file says which numbers re-derive.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.