acceptodds
Under review as a conference paper at ICLR 2027

CANOPY: Adaptive-Granularity Evidence Compression for Multimodal RAG

Abstract

Multimodal RAG retrieves text, tables, images, and videos, but choosing a retrieval granularity does not determine how much context to retain within each item. Coarse units include irrelevant content, while uniformly fine selection can remove context needed to interpret the evidence. Existing compressors address this trade-off with modality-specific mechanisms, leaving open a shared procedure for adapting the retained extent region by region across heterogeneous items. We introduce Canopy (Canonical Projection over Hierarchy), a framework for adaptive-granularity post-retrieval evidence compression. Canopy represents retrieved items as hierarchies and uses a node encoder fine-tuned on gold evidence to score regions against the query. Parent-relative refinement compares these scores to select multiple regions at different granularities without LLM calls for node-level pruning. Because compression cannot recover evidence that was never retrieved, a critic requests targeted follow-up retrieval when it judges the accumulated evidence insufficient; newly retrieved items are compressed before being added. Across five QA benchmarks over a 33M-item heterogeneous corpus, Canopy achieves higher average answer accuracy than the evaluated retrieval baselines. Ablations indicate that additional retrieval drives the main accuracy gains on multi-hop QA. In the unrouted Qwen3-VL-8B-Instruct setting, compression reduces reader-input evidence tokens by 14.2-27.7% relative to the same iterative pipeline without compression, with comparable answer accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
retrieval augmented generation, question answering, multi-modal

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Km9rOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents CANOPY, a framework for compressing retrieved multimodal evidence. It organizes text, tables, and videos into hierarchical structures and adaptively selects evidence at different granularities. A critic then decides whether additional retrieval is needed.

Critical strengths
  1. Strong motivation, solid solutions, and clear presentations.

  2. This paper provides a unified compression method for text, tables, images, and videos. The adaptive granularity design is reasonable and intuitive.

  3. The experiments cover several heterogeneous QA benchmarks. CANOPY reduces reader-input evidence tokens while generally maintaining answer quality.

Critical weaknesses

Inference efficiency is not fully measured. Table 1 reports only the evidence tokens in the final reader prompt. It excludes the tokens consumed by the critic, router, and aggregate classifier across multiple retrieval rounds. Therefore, the reported token count may underestimate CANOPY’s actual inference cost.

Questions for authors
  1. Could the authors report the full online inference cost, including vector search, node encoding, the router, the critic, and all LLM tokens? Even if the total cost is higher, this would not weaken the core motivation, as compressing multimodal evidence before passing it to the reader is already a solid contribution.
  2. Could the authors compare CANOPY with a fine-grained RAG baseline that directly indexes and retrieves sentences, table rows, and 30-second video segments? The current flat-selection experiments still retrieve coarse items first and only select fine-grained evidence afterward.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Km9r (1/2)

Dear Km9r,

Thank you for recognizing evidence compression as a solid contribution even if its full cost turned out to be higher. We measured that cost: with every LLM call included, CANOPY still uses 8.6–25.1% fewer tokens than the uncompressed pipeline, because the critic reads the same compressed evidence as the reader. Your fine-grained retrieval suggestion proved especially useful. Unit-level retrieval finds table rows that item-level search misses, CANOPY keeps the surrounding context that multi-hop questions need, and combining fine-grained retrieval with CANOPY’s hierarchical compression gives the highest average QA score across five benchmarks. All corresponding additions and revisions are included in the revised paper and marked in blue.

Response to W1 & Q1:

We agree with the reviewer that Table 1 underestimates CANOPY's inference cost, as it counts only reader-input evidence tokens. Even with every LLM call counted, CANOPY uses 8.6–25.1% fewer total tokens than the same iterative retrieval pipeline without compression (w/o Projection), because compression also reduces the evidence read by the critic. We measured all components listed by the reviewer. Since node embeddings are query-independent and computed once per item, we report node encoding as an offline construction cost and the remaining components as online inference cost.

Online inference. The critic accounts for most of the cost not reported in Table 1, whereas the router and aggregate classifier add little. We measured latency as amortized wall-clock seconds per question under batched processing, including vector search, and total LLM tokens (input and output) over all calls across retrieval rounds, under the same setup as Table 1. Each cell shows amortized latency (s/question) / tokens.

MethodNQHotpotQAOTT-QAMMQALVBench
Vanilla RAG0.86 / 1,7560.84 / 1,7180.76 / 2,3880.84 / 2,0630.99 / 4,469
UniversalRAG1.25 / 3,4781.04 / 4,6691.12 / 6,2431.08 / 4,7944.90 / 34,721
IRCoT3.20 / 10,1143.18 / 10,8543.52 / 15,0283.22 / 12,1334.63 / 24,041
CANOPY1.42 / 4,0501.75 / 5,0422.25 / 8,1651.92 / 6,1273.22 / 11,072
CANOPY w/o Projection1.44 / 4,5741.64 / 5,6992.28 / 8,9561.87 / 6,7053.35 / 14,775
CANOPY†1.67 / 7,6641.80 / 8,8972.62 / 14,9472.45 / 12,3227.21 / 65,484

The critic accounts for 56–60% of CANOPY's input tokens because it reads the retrieved evidence in every round, while the aggregate classifier and the router (CANOPY† only) account for at most 4% and 11%, respectively. Since the critic reads the same compressed evidence as the reader, 54.4–58.9% of the tokens saved by compression come from the critic. Compression thus reduces the cost of every call that reads the evidence, not only that of the reader. Latency remains within 0.13 s of w/o Projection because vector search accounts for a large share of it. CANOPY costs more than single-round methods on most benchmarks, but compared with IRCoT, an existing iterative retrieval method, its latency is 30.5–55.6% lower, and its token usage is 45.7–60.0% lower.

Offline construction. The node-encoding cost is incurred once per item rather than per query. On one RTX A6000, node encoding takes 28–50 ms per text item, depending on the corpus, 0.26 s per table, and 1.3 s per video, and building an item's hierarchy takes less than 1 ms on CPU. The resulting embeddings are reused across all queries.

We have updated Table 1 so that its latency includes vector search, and added an appendix section on full inference cost that reports total LLM tokens over all calls, the per-component token breakdown, and node-encoding costs.

AuthorsOpenReview ↗

Response to Reviewer Km9r (2/2)

Response to Q2:

We implemented the suggested fine-grained RAG baseline and compared it with CANOPY under similar reader-token budgets. Specifically, we split the same 33M-item corpus into 157.7M fine-grained units (sentences, table rows, and 30-second video segments), indexed them with Qwen3-VL-Embedding-2B, and retrieved k=31–41 units per benchmark while keeping the same critic, reader, and retrieval round limit. Fine-grained RAG achieves 4.2 points higher F1 on OTT-QA but 2.3 points lower F1 on HotpotQA (both statistically significant), with no significant differences on the other three benchmarks. On OTT-QA, item-level retrieval represents an entire table with a single embedding, potentially missing a table containing a relevant row, whereas fine-grained retrieval can retrieve that row directly. On HotpotQA, retrieving only the key sentence from the second document may omit the surrounding context needed to connect the two reasoning hops. CANOPY can preserve this context by retaining broader regions within retrieved items. Among 658 bridge questions where both methods retrieved the gold items, CANOPY achieves 3.9 points higher F1 (statistically significant).

BenchmarkCANOPYFine-grained RAGCANOPY w/ fine-grained Retrieval
NQ54.4 / 1,51254.2 / 1,51855.0 / 1,513
HotpotQA62.1 / 1,81659.8 / 1,82560.9 / 1,873
OTT-QA33.4 / 2,79637.6 / 2,77736.4 / 3,013
MMQA51.1 / 2,12352.4 / 2,11252.7 / 2,174
LVBench35.0 / 3,97834.8 / 4,04936.6 / 3,488
Avg47.247.848.3

These results suggest that fine-grained retrieval and CANOPY's hierarchical compression offer complementary benefits. To verify this, we combined them by aggregating retrieved fine-grained unit scores at the parent-item level, selecting the top items, and applying CANOPY's projection unchanged. This combination achieves the highest average score across five benchmarks (48.3 vs. 47.2 for CANOPY and 47.8 for fine-grained RAG), without significantly underperforming either method on any benchmark. In particular, it improves OTT-QA F1 by 3.0 points over CANOPY (statistically significant) and partially recovers the performance loss of fine-grained RAG on HotpotQA. These findings show that CANOPY can incorporate fine-grained retrieval to benefit from both precise item retrieval and adaptive preservation of original context. We have added these comparisons to the appendix.

Thank you for the detailed response and the additional experiments. Both of my questions have been addressed. The full cost analysis shows that CANOPY still reduces total LLM tokens relative to the same iterative pipeline without compression, even after including the critic and other LLM calls. The fine-grained RAG comparison is also informative.

These results strengthen the paper, although they also show that standalone CANOPY is not consistently better than fine-grained RAG. I therefore maintain my positive rating.

Thank you, and good luck!

Thank you for taking the time to review our response and the additional experiments. We are glad that the full cost analysis and the fine-grained RAG comparison resolved both of your questions. We would be happy to clarify any further points during the discussion period.

Reviewer_Xp9YOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CANOPY, an adaptive-granularity post-retrieval evidence compression framework for multimodal retrieval-augmented generation. CANOPY constructs a hierarchical structure for each retrieved text, table, or video item, fine-tunes a node encoder with gold-evidence-based pairwise ranking loss, and performs parent-relative hierarchical refinement to select evidence regions at varying granularities without per-node LLM invocations. It is further combined with a critic-guided iterative retrieval mechanism to address evidence insufficiency for multi-hop reasoning. Experiments across five QA benchmarks on a 33M-item heterogeneous corpus show that CANOPY outperforms several retrieval baselines in average answer accuracy, and reduces reader-input evidence tokens by 14.2–27.7% with comparable accuracy when compared to the same iterative pipeline without compression.

Critical strengths

S1. It proposes a unified modality‑agnostic compression pipeline for multimodal RAG and skips LLM calls for node‑level pruning to reduce reader token overhead.

S2. It provides comprehensive end‑to‑end experiments and ablations on a large multimodal corpus, clearly separating gains from compression and iterative retrieval.

Critical weaknesses

W1.- Most performance improvements come from iterative retrieval rather than the core compression module, diminishing the unique value of the proposed CANOPY hierarchical refinement.

W2.- Key risk‑prone components including the table aggregate‑lookup classifier lack quantitative evaluation, and unfair baseline comparisons obscure the real compression effectiveness.

Questions for authors

Q1. Beyond building modality‑specific hierarchies to support multimodal inputs, what distinct conceptual or technical advances does CANOPY bring compared to existing hierarchical RAG systems such as RAPTOR and S2G‑RAG? Please clarify where your parent‑relative refinement mechanism provides tangible new capabilities that prior approaches cannot achieve.

Q2. From your ablation in Table 2, removing the compression projection module yields only minor, statistically insignificant changes to answer accuracy, while disabling iterative retrieval leads to large performance drops on multi‑hop benchmarks. Since most of the reported quality improvement comes from iterative retrieval rather than compression, how would you defend compression as the core technical contribution highlighted in your paper?

Q3. Your current latency measurement excludes node encoding cost. Could you report full end‑to‑end efficiency metrics that include node encoding overhead, and explain whether this additional compute undermines the practical efficiency gains brought by token reduction for the reader LLM?

Q4. Your table processing depends on an LLM classifier to separate aggregate questions from lookup questions, yet there is no reported accuracy for this classifier. Could you share the classifier’s real‑world accuracy, and quantify how misclassification propagates to hurt final QA performance when critical table rows get incorrectly pruned away?

Q5. On the out‑of‑domain LongVideoBench, CANOPY achieves almost no token reduction and slightly worse accuracy than Vanilla RAG. What factors prevent your fine‑tuned node encoder from generalizing well to unseen video data, and how does this constrain the real‑world applicability of your claimed universal shared compression pipeline across diverse multimodal sources?

Q6. In Figure 4, competing compression baselines are evaluated without iterative retrieval, while CANOPY runs with iterative retrieval turned on over the unified corpus. Could you provide fair comparison results where baseline compression methods also use iterative retrieval on the same unified corpus, so we can properly evaluate how strong your compression module performs on its own?

Q7. CANOPY’s gold‑fragment recall is only 79.4%, meaning about one‑fifth of annotated gold evidence fragments are discarded during compression. Can you analyze these pruned fragments: what proportion are essential supporting facts versus redundant background context, and can you quantify the reasoning risk introduced by losing these fragments when trading for fewer reader tokens?

Q8. You have swept hyperparameters like branching factor and retrieval size, but key algorithmic components including the pairwise ranking loss objective, cosine‑similarity‑based parent‑child comparison rule, and critic prompt have not been ablated. Could you add ablation experiments for these core components to demonstrate which design choices are truly necessary and how robust CANOPY is against modifications to these settings?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Xp9Y (1/5)

Dear Xp9Y,

We thank you for the eight concrete questions, most of which led to new experiments. Two points frame our answers. First, the accuracy gains over single-round baselines do come from iterative retrieval, which we do not claim; our contribution is learning how much of each retrieved item to keep. Second, we now test this contribution under fairer conditions: the compression baselines receive the same iterative retrieval, RAPTOR and the trained-judge S2G-RAG are added, and the classifier, the removed gold fragments, and the core design choices are each evaluated directly. We hope this additional evidence supports a more favorable assessment. All corresponding additions and revisions are included in the revised paper and marked in blue.

Response to W1 & Q2:

We agree with the reviewer that the QA accuracy improvements over single-round retrieval baselines primarily come from iterative retrieval, as explicitly demonstrated in Table 2. We do not claim iterative retrieval itself as a novel technical contribution. Rather, CANOPY addresses a complementary problem: how to adaptively select the extent of evidence within retrieved items, reducing reader-input tokens while preserving answer quality. The roles of the two components are distinct: iterative retrieval acquires missing evidence, whereas compression reduces the amount of evidence passed to the reader.

The direct evidence for our compression contribution is the comparison between CANOPY and w/o Projection in Table 2. These settings use the same iterative retrieval pipeline configuration, differing in whether hierarchical compression is applied. Across all five benchmarks, CANOPY reduces reader-input evidence tokens by 14.2–27.7%, while changes in F1/accuracy range from −0.7 to +1.5 points, with no statistically significant degradation. By contrast, the uncompressed iterative pipeline increases reader-input evidence tokens by 13–49% relative to single-round Vanilla RAG, whereas CANOPY limits this change to −6% to +28%. Thus, compression enables the accuracy benefits of iterative retrieval to be obtained with a smaller reader-input evidence footprint.

Importantly, CANOPY's technical contribution is not simply combining compression with iterative retrieval. It introduces a learned parent-relative refinement mechanism that selects original evidence regions at different granularities without node-level LLM calls or a predefined retention count. Fine-tuning improves refinement-decision accuracy from 36.9% to 58.5% (Appendix H), while reducing reader-input tokens across all five benchmarks (Table 4). Learning these parent–child orderings is what matters: replacing them with per-node 0/1 labels (pointwise) keeps answer quality but uses 15.8% more tokens (Q8). Unlike fixed-count flat selection, CANOPY adapts the retained extent independently for each item through its hierarchical stopping rule. We have clarified this distinction in the revised Abstract and Introduction, explicitly attributing the main QA accuracy gains to iterative retrieval and the evidence-efficiency gains to hierarchical compression.

AuthorsOpenReview ↗

Response to Reviewer Xp9Y (2/5)

Response to W2 & Q4:

To evaluate the classifier's reliability, we manually labeled 100 questions from OTT-QA, MMQA, and TAT-QA, equally sampling questions predicted as aggregate and lookup. Weighting each benchmark × predicted-label stratum by its count in the full runs gives 81.4% accuracy (75% unweighted; weighted per benchmark, 81% on OTT-QA, 94% on MMQA, and 65% on TAT-QA). Among the 100 questions, 4 aggregate questions were misclassified as lookup, while 21 lookup questions were misclassified as aggregate. All 16 errors on TAT-QA were of the latter type, involving comparisons across years where all necessary values were contained in a single table row. Since the classifier makes a single decision per question without examining the table to reduce overhead, it cannot distinguish such cases.

To quantify the downstream impact of misclassification, we compared CANOPY with an oracle classifier using manually annotated labels. The original classifier and oracle achieved F1 scores of 44.5 and 44.8, respectively, with mean reader-input evidence tokens of 2,314 and 2,301. We further examined the four aggregate-to-lookup errors across retrieval rounds and found no case where compression removed evidence essential to answering the question. In one case, an annotated gold row was removed, but the answer-bearing content remained in other retained rows. The observed overall F1 difference of 0.3 points resulted from different final predictions on a single question rather than the loss of essential evidence. Conversely, correcting the 21 lookup-to-aggregate errors reduced reader-input evidence tokens by 6% on the affected questions. On TAT-QA, F1 remained unchanged at 72.7, while evidence tokens decreased from 1,272 to 1,199.

These results indicate that classification errors had limited impact on downstream QA performance in the evaluated sample. We have added the classifier evaluation, oracle comparison, and error analysis to the appendix. The baseline fairness concern in W2 is addressed separately in our response to Q6.

BenchmarkAccuracy (%)Aggregate→lookup errorsLookup→aggregate errorsF1 (Classifier / Oracle)Tokens (Classifier / Oracle)
OTT-QA813423.1 / 24.03,271 / 3,296
MMQA941137.8 / 37.82,398 / 2,406
TAT-QA6501672.7 / 72.71,272 / 1,199
All8142144.5 / 44.82,314 / 2,301

Response to Q1:

CANOPY’s contribution is not the hierarchy itself but a learned parent-relative refinement that decides how much original evidence to retain in each retrieved item. RAPTOR builds a hierarchy of generated summaries and retrieves at different levels of abstraction, and S2G-RAG extracts sentences with an LLM. CANOPY instead trains its node encoder on parent–child preferences derived from gold evidence. It therefore selects original regions directly, without summaries or per-node LLM calls, and avoids the information loss that summarization can introduce. Because each branch stops independently, no per-item retained-unit count needs to be set in advance, and a single item can retain a broad parent region in one branch and finer regions in another.

To assess the practical value of this mechanism, we added comparisons with RAPTOR and S2G-RAG. Because RAPTOR was originally designed for hierarchical retrieval over long documents, we built its hierarchy online over the same top-k retrieved items for a fair comparison of post-retrieval evidence selection. We evaluated RAPTOR with and without iterative retrieval, and we also included S2G-RAG with its trained judge, whose checkpoint had not been publicly released at submission. Results are averaged over NQ, HotpotQA, and OTT-QA, retrieving from each benchmark’s own collection as in Figure 4, and input tokens are summed over all LLM calls.

MethodF1Total input tokensLLM calls
RAPTOR35.22,6734.7
RAPTOR + iterative38.08,02113.0
S2G-RAG (untrained judge)50.37,3595.60
S2G-RAG (trained judge)53.68,0646.00
S2G-RAG (trained judge, budget-matched)49.75,7416.16
CANOPY50.95,5732.84

CANOPY achieves 12.9-point higher F1 than RAPTOR with iterative retrieval while using fewer input tokens and LLM calls. S2G-RAG with the trained judge achieves higher F1 but at greater cost. When its document-input budget is halved to match total input tokens (5,741 vs. 5,573), CANOPY achieves 1.1-point higher F1 with 53.9% fewer LLM calls. We have added these results and clarified this distinction in the revision.

AuthorsOpenReview ↗

Response to Reviewer Xp9Y (3/5)

Response to Q3:

CANOPY’s node embeddings are query-independent, so they are computed once per item offline and reused across all queries. We therefore report online inference cost separately from offline construction cost.

Online inference. Extending our original measurements, we measured per-question latency (s/question) and total LLM tokens (input and output) over all calls. Latency includes query encoding, search over the 33M-item FAISS index on one 40-core CPU, projection, and all LLM calls, including routing and aggregate classification where applicable. We used Qwen3-VL-8B-Instruct under the batched execution setup of Appendix B. On all five benchmarks, CANOPY’s latency is 30.5–55.6% lower than that of IRCoT, an existing iterative retrieval method, and CANOPY makes fewer LLM calls on average (3.0 vs. 3.6).

MethodNQHotpotQAOTT-QAMMQALVBenchAvg.
Vanilla RAG0.860.840.760.840.990.86
UniversalRAG1.251.041.121.084.901.88
IRCoT3.203.183.523.224.633.55
CANOPY1.421.752.251.923.222.11
CANOPY†1.671.802.622.457.213.15

To isolate the effect of compression, we compared CANOPY with w/o Projection in the same iterative retrieval pipeline. CANOPY reduces total LLM tokens by 8.6–25.1% while keeping end-to-end latency similar. Because vector search accounts for a large share of latency, the time saved in LLM processing appears small in the total. Excluding search, online processing time decreases on all five benchmarks, by 28.5% on LVBench (1.30 s to 0.93 s). Compression therefore does not consistently reduce end-to-end latency, but it reduces LLM tokens with little additional online computation.

Offline construction. Building an item’s hierarchy takes less than 1 ms on CPU. On one RTX A6000, node encoding takes 28–50 ms per text item, 0.26 s per table, and 1.3 s per video, with 17–40 KB, 100–114 KB, and 45 KB of embedding storage, respectively. These costs are incurred once rather than per query.

In the revision, we have reported end-to-end online latency including vector search, together with total LLM tokens over all calls, and presented node-encoding and storage costs separately as offline construction costs.

Response to Q5:

The small token reduction on LongVideoBench stems mainly from evidence accumulated through iterative retrieval rather than from a failure of the node encoder to generalize. To verify this, we kept the encoder and the refinement rule unchanged and removed only iterative retrieval. This setting (CANOPY w/o Iterative) performs the same single retrieval round (k=10) as Vanilla RAG, which isolates the effect of compression itself.

MethodLVB AccLVB Tok
Vanilla RAG49.557,736
CANOPY48.257,195
CANOPY w/o Iterative49.151,919

CANOPY w/o Iterative reduces reader-input evidence tokens by 10.1% relative to Vanilla RAG without a significant drop in accuracy (95% CI: [−3.0, +2.2]), showing that compression works effectively even on videos unseen during training. Adding iterative retrieval accumulates evidence and shrinks the token reduction to 0.9%, while the accuracy difference remains non-significant (95% CI: [−4.2, +1.2]). Because the answer evidence in LongVideoBench is concentrated in a single video, additional retrieval increases tokens without improving accuracy.

The generality of the shared compression pipeline does not mean the same compression rate across modalities, but that the same refinement rule applies to all of them. Its actual effect depends on how evidence is distributed and on the retrieval setting. On LongVideoBench, within-item compression alone reduces tokens, whereas when evidence is spread across multiple items, as in 2WikiMultiHopQA, iterative retrieval improves accuracy and compression reduces token cost. CANOPY also achieves the highest average performance across the three OOD benchmarks (Appendix K). In the revision, we have added this ablation and the statistical analysis to Appendix K and clarified the scope and limitations of the pipeline.

Reviewer_DJqQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CANOPY, a framework for adaptive-granularity evidence compression in multimodal retrieval-augmented generation. The main idea is to organize retrieved text, tables, and videos into hierarchical structures and use a fine-tuned node encoder to decide which regions should be retained at different levels of granularity. The authors also introduce a critic-guided retrieval process to acquire additional evidence when the current context is insufficient. Experiments are conducted on five QA benchmarks over a heterogeneous corpus of approximately 33 million items, with additional out-of-domain evaluations. The results show that CANOPY improves overall QA performance compared with several retrieval baselines, while its compression component reduces reader-input evidence tokens with largely comparable answer accuracy.

Critical strengths
  1. One aspect I liked is the parent-relative refinement strategy in Section 3.2. Instead of selecting a fixed number of fragments, CANOPY compares parent and child relevance scores and allows different branches to stop at different depths. This offers a relatively straightforward way to obtain evidence at multiple granularities without repeatedly calling an LLM for node-level decisions.

  2. Another positive point is the fine-tuning objective in Section 3.4. Rather than learning general query–document relevance, the authors construct ranking preferences from gold evidence to guide parent–child comparisons directly. This connection between the training objective and the inference procedure makes sense, and the improvements in refinement-decision accuracy reported in Appendix H provide useful supporting evidence.

  3. I appreciate the effort put into evaluating the framework across NQ, HotpotQA, OTT-QA, MMQA, and LVBench. The experiments cover different evidence types, two multimodal readers, multiple retrieval configurations, and several competing approaches. The additional out-of-domain evaluation in Appendix K is also valuable, as it provides some evidence that the learned encoder can transfer to unseen benchmarks.

Critical weaknesses
  1. My primary concern is that the strongest QA improvements seem to come from iterative retrieval rather than the proposed compression mechanism. Table 2 shows that removing iterative retrieval reduces EM by 10.1 points on HotpotQA and 14.9 points on OTT-QA, whereas removing projection produces relatively small accuracy changes. Since critic-guided additional retrieval is already explored in prior work, I am not entirely convinced that the main contribution of CANOPY is sufficiently distinguished from the benefits of combining existing retrieval and compression ideas.

  2. I was particularly interested in the comparison with flat selection in Appendix G. The authors acknowledge that a flat selector using the same fine-tuned encoder can achieve a comparable quality–token trade-off when its retained-unit budget is properly chosen. CANOPY avoids specifying this budget for each item, which is useful, but it is not clear how much practical benefit this brings. If a relatively simple flat selector achieves similar accuracy and token efficiency, the additional value of hierarchical refinement needs to be established more convincingly.

  3. Another issue is the reliability of the parent-relative pruning decisions. Appendix H reports an overall correct-decision rate of 58.5% after fine-tuning, while gold-fragment recall is 79.4% for the unrouted system. The results are noticeably weaker for video, where correct refinement decisions reach only 34.3% and gold-fragment recall is 60.7%. These numbers suggest that the method can still remove a meaningful amount of relevant evidence.

  4. Although the framework is presented as a shared solution for multimodal RAG, not every modality receives the same level of adaptive treatment. Images remain single nodes without spatial refinement, and tables are retained in full when an aggregate question is detected. Video compression relies on fixed 30-second segments and sparse frame sampling, which may miss events requiring finer temporal understanding. These design choices are reasonable simplifications, but they narrow the scope of the claimed adaptive-granularity capability.

  5. The comparison with existing approaches could be strengthened. For example, the S2G-RAG implementation replaces its trained sufficiency judge with an untrained reader LLM, which changes an important component of that baseline. Even with this modification, the paper reports comparable F1 for S2G-RAG with substantially fewer evidence tokens, although it requires more LLM calls. The out-of-domain results in Appendix K are also mixed: CANOPY achieves the best overall average, but IRCoT performs better on 2WikiMultiHopQA, while CANOPY slightly underperforms Vanilla RAG on TAT-QA and LongVideoBench.

Questions for authors
  1. Table 2 suggests that most of the answer-quality improvement comes from iterative retrieval. Could the authors clarify which results most directly demonstrate the benefit of adaptive-granularity compression itself, beyond reducing reader-input tokens? This distinction is important to my assessment of the paper's main contribution.

  2. Appendix G shows that a properly configured flat selector can achieve a similar quality–token trade-off. Could the authors provide a more direct comparison of the two approaches under matched encoder and computational budgets, ideally including how the flat selector's retained-unit count is chosen? I would like to better understand the practical advantage of parent-relative refinement.

  3. Since the reported latency excludes vector search and node encoding, could the authors clarify the total inference cost when these operations are included? In particular, how much additional time and storage are required to construct and encode the region hierarchies?

  4. What motivated replacing the trained sufficiency judge in S2G-RAG with an untrained LLM? Could the authors clarify whether this modification affects its answer quality or efficiency relative to the original implementation? A comparison under a consistent computational budget would help establish the relative strengths of CANOPY.

3 comments
AuthorsOpenReview ↗

Response to Reviewer DJqQ (1/2)

Dear DJqQ,

Thank you for asking what CANOPY’s compression contributes within an iterative retrieval pipeline. We use iterative retrieval to acquire missing evidence and do not claim it as a contribution. The first two contribution statements in the submitted Introduction (lines 114–118) define the problem of adapting evidence granularity within each retrieved item and introduce our method. A node encoder fine-tuned on gold evidence guides parent–child comparisons, allowing one branch to retain a broad region while another retains individual leaves. Our third contribution (lines 119–121) evaluates the complete system and separates the effects of retrieval and compression through ablations. We hope these measurements clarify the scope of the contribution and that you will take them into account in your final assessment. All corresponding additions and revisions are included in the revised paper and marked in blue.

Response to W1 & Q1:

Table 2 separates the two components under the same iterative retrieval policy. Disabling follow-up retrieval lowered HotpotQA and OTT-QA EM by 10.1 and 14.9 points, so the multi-hop gains the reviewer notes come from retrieval. Compared with w/o Projection, compression reduced reader-input evidence tokens by 14.2–27.7% across the five unrouted benchmarks, while EM changed by −0.9 to +1.2 points on the four open-ended benchmarks and LVBench accuracy by +0.4 points, with all five paired 95% CIs including zero. Beyond token reduction, the clearest benefit of the hierarchy itself appears when images are split into regions: on the 115 MMQA questions whose gold image was retrieved, CANOPY scored 50.5 F1 versus 43.5 for flat top-10 selection at a similar token budget (3,732 vs. 3,689; +7.0, 95% CI [+2.1, +12.5]; see our response to W2 & Q2).

Response to W2 & Q2:

The flat-selection curves in Appendix G (Figure 7) were, contrary to the text, obtained with an encoder fine-tuned separately for leaf selection, which shares fθf_\theta's backbone, LoRA, and loss. All results below were re-run with fθf_\theta itself. The conclusion is unchanged under either encoder: a well-configured flat selector reaches quality–token points similar to CANOPY's. We have replaced Figure 7 with the fθf_\theta results.

We gave both selectors the same fine-tuned encoder and retrieval policy. The flat selector retains the top m leaves of each item. We swept m∈{1,5,10,20}m \in \{1,5,10,20\}, using the same m across benchmarks at each operating point. We carried m=5 to the OOD benchmarks because its in-domain reader-input budgets were close to CANOPY’s (−10% to +14%). Its in-domain QA scores were within 1.3 points of CANOPY’s. On TAT-QA, flat m=5 scored 69.7 F1 at 1,151 tokens and CANOPY 72.1 at 1,220. A retention count chosen in-domain may thus need retuning on new data, whereas CANOPY sets no count.

A clear difference appeared in the images. In the paper's default setting, each image is a single node. Following your W4, to test whether refinement also works within images, we split each MMQA image into a 4×44 \times 4 grid of 16 crops, with 10% overlap between neighbors, and built a binary tree over them (whole →\rightarrow halves →\rightarrow quarters →\rightarrow eighths →\rightarrow single crops). In contrast, the flat selector keeps the m crops scored highest by the same fθf_\theta. On the 115 questions whose gold image was retrieved, CANOPY scored 50.5 F1 with 3,732 evidence tokens. The flat selector scored 16.0, 11.6, and 7.0 points lower for m=1, 5, and 10, respectively, including m=10 at a similar token count (3,689; −7.0, paired 95% CI [−12.5, −2.1]).

As for computational cost, selection itself is negligible. On 300 sampled text and table items with 11–20 leaves, selecting from cached vectors took 0.048 ms per item for CANOPY and 0.008 ms for the flat selector.

Response to W3:

Appendix H counts a node decision as correct only if it satisfies every annotated preference. Keeping a parent can violate a preference while preserving all gold leaves below it. Across the evaluated decisions, 41.5% violated a preference and 17.2% dropped annotated evidence. For video, the corresponding rates were 65.7% and 19.0%. To examine the consequence of an actual loss, we manually assessed 50 removed gold fragments, ten per benchmark. We judged 18 essential and 32 redundant or recoverable from the retained evidence.

ModalityDecision errorsDecisions that drop gold
Text31.6%15.1%
Table50.9%19.9%
Video65.7%19.0%
All41.5%17.2%
AuthorsOpenReview ↗

Response to Reviewer DJqQ (2/2)

Response to W4:

The default rule refines passages, lookup tables, and videos, and treats each image as a single node, because image gold evidence is annotated only at the image level, leaving no region-level supervision to train or validate within-image refinement. Tables are kept whole when the classifier predicts an aggregate question, since individually low-scoring rows may enter an aggregate. Under the 4×44 \times 4 image setting described in our response to W2 & Q2, on the 115 questions whose gold image was retrieved, CANOPY used 12% fewer evidence tokens than whole images while scoring 2.6 F1 points higher (50.5 vs. 48.0).

The default video leaves span 30 seconds. With the gold video supplied, varying leaf length across 10, 30 and 60 seconds changed accuracy by at most 1.4 points at a fixed sampling interval. In the full unrouted pipeline, increasing encoder sampling from one frame every 10 seconds to one every 5 seconds raised gold-fragment recall from 60.7% to 62.5%. With denser sampling for both encoder and reader, accuracy rose from 35.0% to 36.4% across all 500 LVBench questions, while mean reader evidence increased from 3,978 to 5,595 tokens.

Response to W5 & Q4:

When we ran the original experiments, we did not have access to the trained S2G-RAG judge and used the common reader LLM in its place. On the three benchmark-specific collections used in Figure 4, switching to the trained judge raised S2G-RAG’s average F1 from 50.3 to 53.6, compared with 50.9 for CANOPY. Trained S2G-RAG used 8,064 input tokens across all LLM calls and 6.00 calls per question, versus 5,573 tokens and 2.84 calls for CANOPY. Halving S2G-RAG’s document budget brought it to 5,741 input tokens and 6.16 calls. At these similar input budgets, the paired F1 difference was +1.1 points for CANOPY (95% CI −0.3 to +2.6).

MethodNQ F1NQ Input TokHotpotQA F1HotpotQA Input TokOTT-QA F1OTT-QA Input TokAvg F1Avg Input TokAvg LLM calls
CANOPY53.63,97064.33,70834.79,04050.95,5732.84
S2G-RAG (untrained judge)52.25,01263.34,43035.512,63550.37,3595.60
S2G-RAG (trained judge)53.85,98267.54,57139.413,63953.68,0646.00
S2G-RAG (trained judge, budget-matched)52.64,41660.63,82236.08,98549.75,7416.16

We agree that the OOD results are mixed. On 2WikiMultiHopQA, IRCoT scored 3.6 F1 higher than CANOPY (63.8 vs. 60.2) but used 2.1× the evidence tokens (3,328 vs. 1,587); CANOPY reached most of the multi-hop gain over Vanilla RAG (+10.0 F1) with less than half of IRCoT’s tokens. On TAT-QA and LongVideoBench, where the answer evidence is concentrated in one page or one video, CANOPY stayed within 1.3 points of Vanilla RAG. To locate the source of this gap, we also ran CANOPY with compression only (w/o Iterative). On LongVideoBench, compression alone matched Vanilla RAG (49.1% vs. 49.5%) with 10.1% fewer tokens, and follow-up retrieval added tokens without improving accuracy. On TAT-QA, compression alone scored 70.6 F1, and follow-up retrieval recovered 1.5 points to 72.1.

MethodTAT-QA F1TAT-QA TokLVB AccLVB Tok
Vanilla RAG72.41,27349.557,736
CANOPY72.11,22048.257,195
CANOPY w/o Iterative70.61,17349.151,919

Response to Q3:

Hierarchy construction and node encoding are query-independent and run once per item. Building a hierarchy takes under 1 ms of CPU time per item, and encoding the nodes of the 33M-item corpus took 440 RTX A6000 GPU-hours; the node embeddings occupy 1.14 TB, against 135 GB for the retrieval index. For online cost, we have revised Table 1 so that its latency includes vector search and every LLM call. Unrouted CANOPY takes 1.42–3.22 amortized seconds per question, within 0.13 seconds of the same iterative pipeline without compression, so compression reduces reader-input evidence tokens by 14.2–27.7% with almost no added online time.

AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer DJqQ,

Thank you again for your thoughtful and constructive comments, which have been helpful for improving our submission.

With the discussion period ending soon, we wanted to check whether you had a chance to read our rebuttal. In short, we separated the effects of retrieval and compression within the same pipeline, re-ran the flat-selection comparison with the same encoder and extended it to images, distinguished refinement-decision errors from actual evidence loss, and added S2G-RAG with its released trained judge and an OOD analysis with compression alone.

If these address your concerns, we would appreciate it if you could take them into account in your final assessment. We are happy to answer any remaining questions.

Best,

Authors

Reviewer_cRgVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies post-retrieval evidence compression for multimodal RAG: after retrieval, how much of each heterogeneous item (text passage, table, video, image) should the reader actually receive. The authors propose CANOPY, which represents each retrieved item as a hierarchy of original regions (sentences for text; rows with preserved headers for tables; 30-second segments for video with one frame per 10s up to 32 frames; images remain single nodes) and scores nodes with a query encoder frozen at Qwen3-VL-Embedding-2B and a node encoder fine-tuned from it via LoRA on pairwise preferences derived from gold-evidence locations (18,575 instances from 4,162 train questions, one shared adapter across benchmarks).

Critical strengths
  1. The empirical methodology is exceptionally rigorous and honest, and this decisively shapes my positive assessment. Every small comparison carries a 95% paired bootstrap CI (10,000 resamples); the paper explicitly flags its own non-significant results (HotpotQA +0.3 F1 over IRCoT, CI [-1.7,+2.2]; compression EM effects -0.9 to +1.2 with all CIs including zero; fine-tuning F1 effects +0.3 to +0.9, CIs including zero) and reports unfavorable outcomes prominently (InternVL3.5-8B routed LVBench 34.0 below UniversalRAG's 37.2 while using more tokens, 31,218 vs 27,290; Appendix G concedes the flat selector reaches comparable quality-token points; on out-of-domain 2WikiMultiHopQA, IRCoT is 3.6 F1 ahead). All baselines run under the identical retrieval encoder, corpus, and reader with official implementations; the unified 32.9M-item corpus requires real retrieval rather than assuming gold evidence; two reader LLMs and three frozen out-of-domain benchmarks test generality; and an anonymous repository releases code, the fine-tuned adapter, splits, and corpus-rebuild scripts. This is among the most auditable evaluations I have seen in the RAG literature, and it makes the paper's modest claims trustworthy.
  2. The method is simple, cheap at node level, and cleanly decomposed, and the decomposition itself is a useful community finding. Parent-relative refinement needs no LLM calls for pruning, no per-item retained-unit count, retains original content rather than generated summaries, and applies uniformly to text, tables, and video; node embeddings are query-independent and cacheable. The ablations separate the two effects cleanly: accuracy gains come from critic-guided additional retrieval (removing it costs 10.1/14.9 EM on HotpotQA/OTT-QA), while compression delivers 14.2-27.7% reader-input token savings at statistically indistinguishable accuracy (up to 33.3% on routed LVBench, and 40.0% token reduction versus Gold Item at comparable accuracy when gold items are given). The diagnostic appendices (decision-level accuracy in Fig. 8, gold-fragment recall in Table 11, per-modality kept/split/dropped shares in Fig. 9, gold-vs-non-gold retention in Fig. 10, budget and B/k sweeps in Tables 6-10) give rare mechanistic visibility into how learned pruning actually behaves.
Critical weaknesses
  1. The titular contribution yields no measured accuracy benefit, and its advantage over the simplest alternative is convenience rather than frontier. Compression's EM effects are -0.9 to +1.2 with every 95% CI including zero (Table 2), and node-encoder fine-tuning changes F1/accuracy by +0.3 to +0.9, again with CIs including zero (Table 4) - the accuracy gains over baselines are driven by critic-guided iterative retrieval, an established component (IRCoT, Self-RAG, S2G-RAG). Moreover, Appendix G shows that leaf-level flat selection using the same fine-tuned encoder, with a swept retention count m, reaches quality-token points comparable to CANOPY, and the paper itself states this "does not establish a quality-token advantage over the well-performing settings of the flat selector." What remains for hierarchical refinement is per-item adaptive extent without an external budget - a real but modest practical convenience, illustrated by retained parents accounting for 25-35% of refinement decisions. The central novelty is therefore empirically thinner than the framing suggests; the honest reporting mitigates but does not remove this limitation. (Fixable in framing; fundamental for the contribution's magnitude.)
  2. The mechanism is weakest exactly on the modality that motivates the multimodal framing, and the components that drive the gains are left unanalyzed. After fine-tuning, parent-child refinement decisions are fully correct on only 34.3% of LVBench instances (58.5% overall; Fig. 8), gold-fragment recall is 60.7% on video (79.4% overall; Table 11), and unrouted LVBench accuracy is 35.0 versus Vanilla RAG's 34.4 - the strong video result (43.4) depends on modality routing that substantially harms text/table benchmarks (OTT-QA EM 27.7 to 20.1, Table 1). No remedy for video-level pruning weakness is explored (e.g., modality-specific thresholds or leaf lengths). Meanwhile, the critic and the aggregate/lookup table classifier - the components responsible for the accuracy gains and for avoiding catastrophic row loss, respectively - have no direct reliability measurements anywhere: no sufficiency-judgment accuracy, no early-stop/wasted-round rates, no classifier accuracy or oracle-classification upper bound, and no analysis of which follow-up rounds actually retrieve missing gold, despite the striking statistic that 63.3% of follow-up-retrieved items are already in the accumulated evidence. Finally, the Lat. column excludes vector search and node encoding (declared in Appendix B but not flagged in Table 1), so the displayed cost comparison understates CANOPY's true overhead relative to single-round baselines. All of these are fixable with analyses of existing runs.
Questions for authors
  1. (Material) Critic reliability: on a manually labeled subset, what is the sufficiency-judgment accuracy? Please report the distribution of rounds used, and the fraction of follow-up rounds that retrieve at least one new gold item - given that 63.3% of follow-up-retrieved items are already in the evidence.
  2. (Material) Aggregate/lookup classifier: accuracy on the table benchmarks (OTT-QA, MMQA, TAT-QA), and an oracle-classification upper bound quantifying the cost of misclassification in both directions.
  3. (Material) Full cost accounting: node-encoding and search time per question (frozen vs fine-tuned encoder); can Table 1's Lat. be re-reported including these, or a total-cost column added?
  4. (Material) Video weakness: sensitivity to the 30s leaf length and 32-frame cap; would a less aggressive pruning threshold for video (or video-specific stopping) recover unrouted LVBench accuracy without routing's cost on text/table benchmarks?
  5. (Clarification) Is the Appendix G flat-vs-CANOPY parity per-benchmark or only in pooled metrics? Does CANOPY dominate the swept flat selector on each benchmark's own quality-token curve?
  6. (Clarification) Since the critic sees compressed evidence, can pruning cause false "insufficient" judgments and inflate round count? Any measurement of this interaction (e.g., critic decisions on w/o Projection evidence)?
  7. (Clarification) Can S2G-RAG be run with its original trained judge (or is this a release barrier)? Its 5.6-calls/fewer-tokens operating point is the closest competitor to CANOPY's positioning.
5 comments
AuthorsOpenReview ↗

Response to Reviewer cRgV (1/4)

Response to W1 & Q5:

The parity holds per benchmark. The flat-selection curves in Appendix G (Figure 7) were, contrary to the text, obtained with an encoder fine-tuned separately for leaf selection, which shares fθf_\theta’s backbone, LoRA, and loss. Re-run with fθf_\theta itself, sweeping m∈{1,5,10,20}m \in \{1,5,10,20\} with the same m on every benchmark, the per-benchmark results are in Appendix G. CANOPY has higher quality at fewer tokens than flat m=5 on NQ and LVBench, flat m=5 has higher quality at fewer tokens than CANOPY on MMQA, and the two trade quality for tokens on HotpotQA and OTT-QA. On no benchmark does CANOPY lie above every swept flat point. We have replaced Figure 7 with the fθf_\theta results.

We agree that, on the benchmarks in the paper, hierarchical refinement does not improve accuracy over a tuned flat selector, and that what it removes is the retention count. The one setting in which it scored higher than flat selection at a matched token budget is images, which the default configuration keeps as single nodes. We split each MMQA image into a 4×44 \times 4 grid of 16 crops, with 10% overlap between neighbors, built a binary tree over them (whole →\rightarrow halves →\rightarrow quarters →\rightarrow eighths →\rightarrow single crops); the flat selector keeps the m crops scored highest by the same fθf_\theta. On the 115 questions whose gold image was retrieved, CANOPY scored 50.5 F1 with 3,732 evidence tokens. The flat selector scored 16.0, 11.6, and 7.0 points lower for m=1, 5, and 10, respectively, including m=10 at a similar token count (3,689; −7.0, paired 95% CI [−12.5, −2.1]).

We have clarified this distinction in the revised Abstract and Introduction, explicitly attributing the main QA accuracy gains to iterative retrieval and the evidence-efficiency gains to hierarchical compression.

AuthorsOpenReview ↗

Response to Reviewer cRgV (2/4)

Response to W2-1 & Q4:

We agree that video refinement is weak on LVBench, but decision errors do not equal evidence loss. The decision error rate on video is 65.7%, yet only 19.0% of decisions drop gold evidence, because decision accuracy requires every annotation-derived preference to hold, so retaining a parent that preserves the gold evidence can still count as an error.

One limiting factor is sparse temporal sampling. Gold evidence intervals have a median length of 11.4 s, and 35% last at most 5 s, whereas our default samples one frame every 10 s. With the gold video given, as in Table 6, changing the leaf length among 10, 30, and 60 s changed accuracy by at most 1.4 points, whereas sampling every 5 s instead of 10 s improved it by 2.8–4.4 points. In the full unrouted pipeline, denser encoder sampling raised gold-fragment recall from 60.7% to 62.5%, and also densifying reader sampling raised LVBench accuracy from 35.0% to 36.4% (52.2% to 71.7% on the 46 questions whose gold video was retrieved), with reader-input tokens rising from 3,978 to 5,595.

Each cell shows accuracy / reader-input tokens with the gold video given.

Frame samplingLeaf 10 sLeaf 30 sLeaf 60 s
1 frame / 10 s55.4 / 2,73456.4 / 2,78255.0 / 3,612
1 frame / 5 s58.8 / 4,30359.2 / 5,04959.4 / 6,198

We also distinguish modality routing from compression. Routing decides which modality-specific corpus to retrieve from, whereas CANOPY's refinement decides what to keep within retrieved items. Routing raises the gold-video retrieval rate from 9.2% to 62.4% and LVBench accuracy from 35.0% to 43.4%, but lowers OTT-QA EM from 27.7% to 20.1%; we therefore do not attribute the routed LVBench gain to compression. The 30-second leaf length and 32-frame cap follow UniversalRAG's frame budget rather than any constraint of hierarchical refinement. Denser temporal sampling partially reduces video evidence loss without changing the refinement rule, and video-specific pruning thresholds or stopping criteria are complementary directions. We have added these analyses to the revised paper.

Response to W2-2 & Q1:

Against human labels, the critic’s sufficiency judgments are 81.5% accurate, and early stops, where the critic stops on insufficient evidence, occur in 10.3% of questions. We labeled 100 judgments (10 sufficient and 10 insufficient per benchmark) by whether the question is answerable from exactly the evidence the critic saw, and reweighted them by the ratio of sufficient to insufficient judgments in the full runs.

The share of follow-up rounds that retrieve a new gold item is 31.7% and 19.6% on the multi-hop benchmarks HotpotQA and OTT-QA, respectively, but 5.0% on single-hop NQ (14.6% overall). In multi-hop questions, the second-hop evidence can be found only once the first hop reveals the bridge entity, so the critic’s follow-up query, which includes that entity, retrieves it. In single-hop questions, the original question already finds the evidence and the follow-up query retrieves the same items again, so the share of follow-up-retrieved items already in the evidence (63.3% overall) is highest on NQ (76.6%) and lowest on HotpotQA (51.2%).

Wasted rounds are frequent but mostly reflect retrieval failures rather than critic errors. Of all follow-up rounds, 85.4% retrieve no new gold item, but only 18.2% of them follow false insufficient judgments; the rest follow correct judgments whose follow-up retrieval fails to find the missing gold. Even on OTT-QA and LVBench, where 63% and 84% of questions reach the three-round limit, 8 and 9 of the 10 human-checked insufficient judgments, respectively, were correct. On LVBench, the modality gap between text queries and videos keeps video retrieval recall low regardless of query rewriting, so only 0.7% of follow-up rounds retrieve a new gold item. We have noted in the appendix that wasted rounds are frequent on single-hop and video benchmarks, a limitation of the current critic-guided loop.

BenchmarkRounds used 1 / 2 / 3 (%)Avg. roundsFollow-up rounds with a new gold item (%)Follow-up-retrieved items already in evidence (%)
NQ69.3 / 5.5 / 25.21.565.076.6
HotpotQA47.9 / 24.3 / 27.81.8031.751.2
OTT-QA17.1 / 20.4 / 62.52.4519.656.8
MMQA39.2 / 11.4 / 49.42.1011.267.3
LVBench12.0 / 3.8 / 84.22.720.771.7
AuthorsOpenReview ↗

Response to Reviewer cRgV (3/4)

Response to Q2:

Against human labels, the aggregate/lookup classifier is 81.4% accurate (95% CI [72.9, 88.2]), and most misclassifications fall in the inexpensive direction. We hand-labeled 100 questions from OTT-QA, MMQA, and TAT-QA, sampled equally from the two predicted labels, and weighted them by each label’s share in the full runs.

Accuracy is lowest on TAT-QA (65%), where all errors label year-to-year comparisons of a single item as aggregate. These questions are lookup because each TAT-QA table row holds the values for all years, but the classifier decides once from the question alone, without seeing the table, and cannot detect this.

The cost of these errors is small: an oracle that forces each question’s human label improves F1 by only 0.3. Of the 25 errors, 21 label lookup questions as aggregate and keep the whole table; correcting them leaves F1 unchanged and reduces tokens on those questions by 6%. In the 4 errors in the opposite direction, compression never removed evidence needed for the answer. The table below is measured on the 100 labeled questions.

BenchmarkAccuracy (%)Aggregate→lookup errorsLookup→aggregate errorsF1 (Classifier / Oracle)Tokens (Classifier / Oracle)
OTT-QA813423.1 / 24.03,271 / 3,296
MMQA941137.8 / 37.82,398 / 2,406
TAT-QA6501672.7 / 72.71,272 / 1,199
All8142144.5 / 44.82,314 / 2,301

Response to Q6:

Compression barely changes the critic’s judgments or the number of rounds. Starting from the same top-10 items in the first round, the critic’s judgments under CANOPY and w/o Projection agree on 90.9–95.2% of judgments. Compression increases the average number of rounds by 0.02–0.04 and lowers the share of questions judged sufficient in the first round by 0.7–2.2 points.

Compression removes gold evidence and thereby causes a false insufficient judgment in only 16 of 4,500 questions (0.4%). It changes the critic’s first-round judgment in 312 questions (6.9%), but in both directions: 194 were judged insufficient only with compression, with gold evidence removed in 16 of them, and 118 only without compression, so the two largely offset. Insufficient judgments on truly insufficient evidence mostly arise because no gold item was retrieved at all.

Each pair shows CANOPY / w/o Projection.

BenchmarkAvg. roundsSufficient in round 1 (%)Agreement (%)Insufficient only with compression (gold removed)Insufficient only without compression
NQ1.56 / 1.5469.3 / 70.090.949 (3)42
HotpotQA1.80 / 1.7647.8 / 49.594.138 (1)21
OTT-QA2.45 / 2.4216.7 / 18.993.245 (5)23
MMQA2.10 / 2.0638.4 / 40.693.046 (4)24
LVBench2.72 / 2.6810.6 / 12.295.216 (3)8

We have added these analyses to the appendix.

Response to W2-3 & Q3:

We agree that excluding vector search understated CANOPY’s overhead. We have revised Table 1 so that its latency includes vector search, and its caption now states that node encoding is an offline cost.

Online inference. Latency below includes vector search over the 33M-vector FAISS index on one 40-core CPU, under the setup of Appendix B. Each cell shows latency (s) per question.

MethodNQHotpotQAOTT-QAMMQALVBench
Vanilla RAG0.860.840.760.840.99
UniversalRAG1.251.041.121.084.90
IRCoT3.203.183.523.224.63
CANOPY1.421.752.251.923.22
CANOPY†1.671.802.622.457.21

Because CANOPY runs vector search in every retrieval round (2.1 rounds on average), it is 0.2–1.5 s slower than the single-round baselines on the four non-video benchmarks, but faster than IRCoT, an iterative retrieval baseline, on all five benchmarks.

Offline construction. Node embeddings are query-independent, so, like item embeddings, they are computed once per item and reused across queries. On one RTX A6000, encoding takes 28–50 ms per text item depending on the corpus, 0.26 s per table, and 1.3 s per video segment. The frozen f0f_0 and fine-tuned fθf_\theta cost the same, since fθf_\theta merges a LoRA adapter into the weights of f0f_0 without changing its architecture.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Discussion Update and Request for Follow-Up

Dear Reviewers and Area Chair,

We thank the four reviewers for their careful and constructive reviews. As the discussion period comes to a close, we would be grateful if Reviewers cRgV, DJqQ, and Xp9Y, who have not yet responded, could let us know whether any concern remains. We addressed all concerns, most of them with new experiments, and marked the revisions in blue. Below we summarize the key results for the common concerns; details are in the individual responses.

1. Contribution of compression (cRgV, DJqQ, Xp9Y) Iterative retrieval obtains missing evidence, and we do not claim it as a contribution. Since the submitted Introduction (l.114–118), our contribution has been to adaptively decide how much evidence to keep from each retrieved item. Within the same iterative retrieval pipeline, this compression reduces reader-input tokens by 14.2–27.7% without significant loss in accuracy (Table 2).

2. Hierarchical refinement vs. flat selection (cRgV, DJqQ, Xp9Y) Flat selection must tune the number m to keep, and m decided in-domain is 2.4 F1 lower than CANOPY on out-of-domain TAT-QA. CANOPY decides the range for each item without a number, and on image regions F1 is 7.0 higher with a similar number of tokens (App. G).

3. Baseline comparison (cRgV, DJqQ, Xp9Y) We compared again with the trained judge of S2G-RAG, which was not released at submission. At the same total input tokens, CANOPY's F1 was 4.4 higher with less than half the LLM calls, and also higher than LongLLMLingua and RAPTOR with iterative retrieval (App. P).

4. Reliability of components (cRgV, DJqQ, Xp9Y) Based on human labels, the critic was 81.5% and the table classifier was 81.4% accurate. In a sample of gold fragments removed by compression, 64% were redundant (App. H, N).

5. Full inference cost (cRgV, DJqQ, Xp9Y, Km9r) Even measured end-to-end, CANOPY has 30.5–55.6% lower latency and 45.7–60.0% fewer total LLM tokens than IRCoT, and 8.6–25.1% fewer total LLM tokens than the same pipeline without compression (Table 1, App. O).

6. Video and modality generality (cRgV, DJqQ, Xp9Y) Leaf length and frame sampling for video are implementation choices that complement hierarchical refinement. Keeping the refinement rule unchanged and only sampling frames more densely improves accuracy by 2.8–4.4 points with the gold video (App. Q), and on LongVideoBench, which was not used in training, compression alone reduces tokens by 10.1% (App. K).

AuthorsOpenReview ↗

Follow-Up on Our Rebuttal and Revised Manuscript

Dear Area Chair and Reviewers,

Thank you for your constructive comments, which have helped us improve the paper. We have made every effort to address each concern through detailed, point-by-point responses supported by additional experiments and evidence. These additions and clarifications are incorporated into the revised manuscript, with changes highlighted in blue.

We would be grateful if the reviewers could read our responses and revisions and briefly acknowledge that they have considered the new evidence. If any original concerns remain unresolved, we would appreciate knowing which points require further clarification.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.