- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents CANOPY, a framework for compressing retrieved multimodal evidence. It organizes text, tables, and videos into hierarchical structures and adaptively selects evidence at different granularities. A critic then decides whether additional retrieval is needed.
Critical strengths
-
Strong motivation, solid solutions, and clear presentations.
-
This paper provides a unified compression method for text, tables, images, and videos. The adaptive granularity design is reasonable and intuitive.
-
The experiments cover several heterogeneous QA benchmarks. CANOPY reduces reader-input evidence tokens while generally maintaining answer quality.
Critical weaknesses
Inference efficiency is not fully measured. Table 1 reports only the evidence tokens in the final reader prompt. It excludes the tokens consumed by the critic, router, and aggregate classifier across multiple retrieval rounds. Therefore, the reported token count may underestimate CANOPY’s actual inference cost.
Questions for authors
- Could the authors report the full online inference cost, including vector search, node encoding, the router, the critic, and all LLM tokens? Even if the total cost is higher, this would not weaken the core motivation, as compressing multimodal evidence before passing it to the reader is already a solid contribution.
- Could the authors compare CANOPY with a fine-grained RAG baseline that directly indexes and retrieves sentences, table rows, and 30-second video segments? The current flat-selection experiments still retrieve coarse items first and only select fine-grained evidence afterward.
4 comments
Response to Reviewer Km9r (2/2)
Response to Q2:
We implemented the suggested fine-grained RAG baseline and compared it with CANOPY under similar reader-token budgets. Specifically, we split the same 33M-item corpus into 157.7M fine-grained units (sentences, table rows, and 30-second video segments), indexed them with Qwen3-VL-Embedding-2B, and retrieved k=31–41 units per benchmark while keeping the same critic, reader, and retrieval round limit. Fine-grained RAG achieves 4.2 points higher F1 on OTT-QA but 2.3 points lower F1 on HotpotQA (both statistically significant), with no significant differences on the other three benchmarks. On OTT-QA, item-level retrieval represents an entire table with a single embedding, potentially missing a table containing a relevant row, whereas fine-grained retrieval can retrieve that row directly. On HotpotQA, retrieving only the key sentence from the second document may omit the surrounding context needed to connect the two reasoning hops. CANOPY can preserve this context by retaining broader regions within retrieved items. Among 658 bridge questions where both methods retrieved the gold items, CANOPY achieves 3.9 points higher F1 (statistically significant).
| Benchmark | CANOPY | Fine-grained RAG | CANOPY w/ fine-grained Retrieval |
|---|---|---|---|
| NQ | 54.4 / 1,512 | 54.2 / 1,518 | 55.0 / 1,513 |
| HotpotQA | 62.1 / 1,816 | 59.8 / 1,825 | 60.9 / 1,873 |
| OTT-QA | 33.4 / 2,796 | 37.6 / 2,777 | 36.4 / 3,013 |
| MMQA | 51.1 / 2,123 | 52.4 / 2,112 | 52.7 / 2,174 |
| LVBench | 35.0 / 3,978 | 34.8 / 4,049 | 36.6 / 3,488 |
| Avg | 47.2 | 47.8 | 48.3 |
These results suggest that fine-grained retrieval and CANOPY's hierarchical compression offer complementary benefits. To verify this, we combined them by aggregating retrieved fine-grained unit scores at the parent-item level, selecting the top items, and applying CANOPY's projection unchanged. This combination achieves the highest average score across five benchmarks (48.3 vs. 47.2 for CANOPY and 47.8 for fine-grained RAG), without significantly underperforming either method on any benchmark. In particular, it improves OTT-QA F1 by 3.0 points over CANOPY (statistically significant) and partially recovers the performance loss of fine-grained RAG on HotpotQA. These findings show that CANOPY can incorporate fine-grained retrieval to benefit from both precise item retrieval and adaptive preservation of original context. We have added these comparisons to the appendix.
Thank you for the detailed response and the additional experiments. Both of my questions have been addressed. The full cost analysis shows that CANOPY still reduces total LLM tokens relative to the same iterative pipeline without compression, even after including the critic and other LLM calls. The fine-grained RAG comparison is also informative.
These results strengthen the paper, although they also show that standalone CANOPY is not consistently better than fine-grained RAG. I therefore maintain my positive rating.
Thank you, and good luck!
Thank you for taking the time to review our response and the additional experiments. We are glad that the full cost analysis and the fine-grained RAG comparison resolved both of your questions. We would be happy to clarify any further points during the discussion period.
Response to Reviewer Km9r (1/2)
Dear Km9r,
Thank you for recognizing evidence compression as a solid contribution even if its full cost turned out to be higher. We measured that cost: with every LLM call included, CANOPY still uses 8.6–25.1% fewer tokens than the uncompressed pipeline, because the critic reads the same compressed evidence as the reader. Your fine-grained retrieval suggestion proved especially useful. Unit-level retrieval finds table rows that item-level search misses, CANOPY keeps the surrounding context that multi-hop questions need, and combining fine-grained retrieval with CANOPY’s hierarchical compression gives the highest average QA score across five benchmarks. All corresponding additions and revisions are included in the revised paper and marked in blue.
Response to W1 & Q1:
We agree with the reviewer that Table 1 underestimates CANOPY's inference cost, as it counts only reader-input evidence tokens. Even with every LLM call counted, CANOPY uses 8.6–25.1% fewer total tokens than the same iterative retrieval pipeline without compression (w/o Projection), because compression also reduces the evidence read by the critic. We measured all components listed by the reviewer. Since node embeddings are query-independent and computed once per item, we report node encoding as an offline construction cost and the remaining components as online inference cost.
Online inference. The critic accounts for most of the cost not reported in Table 1, whereas the router and aggregate classifier add little. We measured latency as amortized wall-clock seconds per question under batched processing, including vector search, and total LLM tokens (input and output) over all calls across retrieval rounds, under the same setup as Table 1. Each cell shows amortized latency (s/question) / tokens.
The critic accounts for 56–60% of CANOPY's input tokens because it reads the retrieved evidence in every round, while the aggregate classifier and the router (CANOPY† only) account for at most 4% and 11%, respectively. Since the critic reads the same compressed evidence as the reader, 54.4–58.9% of the tokens saved by compression come from the critic. Compression thus reduces the cost of every call that reads the evidence, not only that of the reader. Latency remains within 0.13 s of w/o Projection because vector search accounts for a large share of it. CANOPY costs more than single-round methods on most benchmarks, but compared with IRCoT, an existing iterative retrieval method, its latency is 30.5–55.6% lower, and its token usage is 45.7–60.0% lower.
Offline construction. The node-encoding cost is incurred once per item rather than per query. On one RTX A6000, node encoding takes 28–50 ms per text item, depending on the corpus, 0.26 s per table, and 1.3 s per video, and building an item's hierarchy takes less than 1 ms on CPU. The resulting embeddings are reused across all queries.
We have updated Table 1 so that its latency includes vector search, and added an appendix section on full inference cost that reports total LLM tokens over all calls, the per-component token breakdown, and node-encoding costs.