- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies a interesting occasion about multimodal document retrieval under fixed budgeted, where a retrieval system has a limited number of pages that it can retrieve and read. The authors propose Read, Then Retrieve, an iterative retrieval framework that uses feedback from a multimodal reader to guide subsequent document acquisition. Instead of relying solely on the initial query and static retrieval scores, the system first retrieves a small set of candidate pages, allows a frozen multimodal reader to inspect them, and then uses the reader's feedback to identify and rank additional candidates that are likely to provide missing evidence.The proposed framework combines the initial retrieval signals with reader-derived semantic feedback and employs a lightweight LambdaMART-based selector to determine which additional candidates should be retrieved under a fixed budget.
Critical strengths
1.This paper research on a practically real problem under a fixed realistic retrieval budget. Authors address the limitation of the fixed budgets in multimodal document retrieval and proposes to use reader feedback to acquire addtional evidence. The overall Read-Then-Retrieve pipeline is intuitive, and the distinction between initial retrieval, reader feedback, and subsequent candidate selection is clearly presented. 2.The paper evaluates the proposed method on the challenging ViDoRe V3 benchmark, which contains diverse multimodal documents across multiple professional domains, providing a realistic testbed for budgeted multimodal retrieval.
Critical weaknesses
1.The incremental benefit of reader feedback over a strong static hybrid retrieval baseline is not strong enough. Although Reader Feedback substantially improves candidate recall and All@32 coverage, its final nDCG improvement over static hybrid retrieval is only 0.13 points on the frozen HR corpus, with confidence intervals spanning zero. This raises the question of whether the proposed feedback mechanism provides a meaningful improvement in practical retrieval quality beyond simply combining complementary visual and textual retrieval signals. 2.The computational cost of adaptive retrieval is not sufficiently characterized. The paper focuses on a fixed page budget, but the page budget does not necessarily reflect the actual computational cost of the proposed adaptive retrieval process. Although the paper reports the number of 3B calls for some variants, it does not provide a comprehensive comparison of end-to-end inference cost or latency with the competing retrieval methods.
Questions for authors
1.Could the authors provide a comprehensive end-to-end efficiency comparison between Reader Feedback and the competing retrieval methods, including inference latency, model calls, and computational cost under the same hardware and retrieval budget? 2.Because the Reader Feedback shows that final nDCG has only a marginal improvement in comparison with the strong static hybrid baseline, while the confidence interval of the difference spans zero, could the authors clarify what information provided by the reader feedback is responsible for the improvement beyond complementary visual and textual retrieval signals? 3.The proposed retrieval strategy relies on feedback generated by a frozen 7B multimodal reader. How sensitive is the method to the choice of reader model? In particular, would the learned retrieval selector remain effective when using a different reader architecture or model scale, or does it need to be retrained for each reader?
2 comments
Thanks for the additional experiments and explanations. I now have a better understanding of this paper, and I will maintain my score at 2.
1. End-to-end efficiency at matched budgets
Thank you for suggesting this efficiency comparison. On HR, we tested Static Hybrid, Learned Static, adapted GAR, and Reader Feedback (RF). The GAR adaptation uses first-batch reader scores to select graph neighbors from the shared pool for the second batch. We also timed SlideVQA indexing from page images.
On a dedicated RTX 5090, each strategy scored 32 distinct pages with LamRA (BF16/SDPA) for the same 100 HR queries over three rounds. Mean latency and peak memory were similar.
Of 1,200 timed requests, we retained 600 from Static Hybrid and RF after confirming matching software versions and less than 0.1% mean latency drift in 10 paired checks per original strategy. The 600 new requests involved 19,200 page judgments. Paired analyses averaged the three runs per query.
Each microbatch contained 8 pages with up to 1,280 visual tokens per page. RF used first-batch scores to compute features and select the next 16 pages in 17.94 ms on average, included in total latency. RF made no 3B question-generation calls. Online timing covered query re-encoding, retrieval, image and reader processing, selection, synchronization, and final ranking. Weights and indexes remained resident. Reader-score, visual-feature, and KV reuse were disabled. Timing excluded loading and answer generation.
Building the shared HR index for 1,110 pages took 193.24 s, plus 0.161 s for the GAR graph. On 800 published SlideVQA images, rerunning OCR, encoding, and indexing took 73.91 s, plus 0.055 s for GAR. We counted shared work once and loading separately. Both timings excluded upstream rendering. We used supplied Markdown for 21 HR fallback pages.
2. Information added by reader feedback
Thank you for suggesting this analysis. Visual and textual retrieval scores measure query–page match. The reader assigns question-specific confidence to each page it reads, indicating which of the first 16 pages should guide subsequent selection.
Learned Static retains nine retrieval features and unweighted seed similarity but omits features that depend on reader scores. It shares RF's 96-tree LambdaMART configuration, source training data, pool, first 16 pages, budget, and ranking head. RF raised HR All@32 from 56.60% to 62.58% (+5.97 percentage points, document-family 95% CI [3.17, 8.39]). We separately retrained Uniform Feedback, retaining seed text, document information, and all 13 inputs while fixing confidence at 1 in training and inference. Its All@32 was 56.60%, giving RF a gain of 5.97 points [0.95, 7.06].
All three signals improved coverage. Same-document support contributed most. Document membership is shared metadata. RF uses the reader’s confidence in previously scored pages to assess candidates from the same document. Removing this signal reduced All@32 from 62.58% to 56.60%. Removing either weighted semantic support or uncovered-term feedback reduced it to 61.95%. Against Static Hybrid, RF raised All@32 from 59.12% to 62.58% (+3.46 points [0.00, 10.73]) and final nDCG@10 from 67.61 to 67.75 (paired difference +0.13 [−2.07, 0.76]). The evidence most directly supports better coverage at a fixed budget.
3. Selector transfer
Thank you for suggesting that we test transfer to a different reader. We replaced 7B LamRA with Qwen3-VL-Reranker-2B. B applied the original selector to feedback from the new reader. C retrained the same architecture using source labels and 15,296 page judgments from the frozen replacement reader on 956 source questions. All three used the same pool, 32-page budget, and new reader's raw scores for final ranking. We used the 318 HR questions only for evaluation.
B fell slightly below A on all three metrics. C exceeded B by 1.79, 3.46, and 0.22 points, respectively (All@32 document-family 95% CI [0.63, 12.18]). C exceeded A by 1.30, 2.20, and 0.12 points (All@32 CI [−0.75, 10.93]). Source-only retraining thus adapted the original selector architecture to the new reader without target-domain labels.