acceptodds
Under review as a conference paper at ICLR 2027

Read, Then Retrieve: Reader Feedback for Budgeted Multimodal Document Search

Abstract

Multimodal document retrieval typically applies an expensive reader to a fixed set of candidate pages, leaving relevant evidence outside that set inaccessible to reranking. We propose Reader Feedback, a two-batch acquisition method that uses intermediate reader judgments to select subsequent pages under a fixed reading budget. A frozen 7B reader first evaluates 16 visual candidates. A lightweight selector combines their relevance scores with semantic similarity, document membership, and uncovered query terms to choose 16 additional pages from visual and text retrieval pools. The same reader and a shared ranking head then order all 32 pages. Across three ViDoRe V3 development corpora, the fraction of queries with every annotated relevant page acquired increases from 55.34% to 62.43% over visual top-32 acquisition. On a separate HR corpus evaluated after method freezing, this coverage increases from 49.69% to 62.58%, while nDCG@10 rises from 64.97 to 67.75. Matched controls show a 5.97-point coverage gain over uniform feedback on HR, but the nDCG advantage over static hybrid retrieval is 0.13 points with an interval spanning zero. These results identify evidence acquisition as a useful role for reader feedback while distinguishing coverage gains from improvements in final ranking.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal document retrieval, visual document retrieval, adaptive retrieval, reader feedback, evidence acquisition, multimodal retrieval, document understanding, retrieval-augmented generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_EpoVOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a interesting occasion about multimodal document retrieval under fixed budgeted, where a retrieval system has a limited number of pages that it can retrieve and read. The authors propose Read, Then Retrieve, an iterative retrieval framework that uses feedback from a multimodal reader to guide subsequent document acquisition. Instead of relying solely on the initial query and static retrieval scores, the system first retrieves a small set of candidate pages, allows a frozen multimodal reader to inspect them, and then uses the reader's feedback to identify and rank additional candidates that are likely to provide missing evidence.The proposed framework combines the initial retrieval signals with reader-derived semantic feedback and employs a lightweight LambdaMART-based selector to determine which additional candidates should be retrieved under a fixed budget.

Critical strengths

1.This paper research on a practically real problem under a fixed realistic retrieval budget. Authors address the limitation of the fixed budgets in multimodal document retrieval and proposes to use reader feedback to acquire addtional evidence. The overall Read-Then-Retrieve pipeline is intuitive, and the distinction between initial retrieval, reader feedback, and subsequent candidate selection is clearly presented. 2.The paper evaluates the proposed method on the challenging ViDoRe V3 benchmark, which contains diverse multimodal documents across multiple professional domains, providing a realistic testbed for budgeted multimodal retrieval.

Critical weaknesses

1.The incremental benefit of reader feedback over a strong static hybrid retrieval baseline is not strong enough. Although Reader Feedback substantially improves candidate recall and All@32 coverage, its final nDCG improvement over static hybrid retrieval is only 0.13 points on the frozen HR corpus, with confidence intervals spanning zero. This raises the question of whether the proposed feedback mechanism provides a meaningful improvement in practical retrieval quality beyond simply combining complementary visual and textual retrieval signals. 2.The computational cost of adaptive retrieval is not sufficiently characterized. The paper focuses on a fixed page budget, but the page budget does not necessarily reflect the actual computational cost of the proposed adaptive retrieval process. Although the paper reports the number of 3B calls for some variants, it does not provide a comprehensive comparison of end-to-end inference cost or latency with the competing retrieval methods.

Questions for authors

1.Could the authors provide a comprehensive end-to-end efficiency comparison between Reader Feedback and the competing retrieval methods, including inference latency, model calls, and computational cost under the same hardware and retrieval budget? 2.Because the Reader Feedback shows that final nDCG has only a marginal improvement in comparison with the strong static hybrid baseline, while the confidence interval of the difference spans zero, could the authors clarify what information provided by the reader feedback is responsible for the improvement beyond complementary visual and textual retrieval signals? 3.The proposed retrieval strategy relies on feedback generated by a frozen 7B multimodal reader. How sensitive is the method to the choice of reader model? In particular, would the learned retrieval selector remain effective when using a different reader architecture or model scale, or does it need to be retrained for each reader?

2 comments

1. End-to-end efficiency at matched budgets

Thank you for suggesting this efficiency comparison. On HR, we tested Static Hybrid, Learned Static, adapted GAR, and Reader Feedback (RF). The GAR adaptation uses first-batch reader scores to select graph neighbors from the shared pool for the second batch. We also timed SlideVQA indexing from page images.

On a dedicated RTX 5090, each strategy scored 32 distinct pages with LamRA (BF16/SDPA) for the same 100 HR queries over three rounds. Mean latency and peak memory were similar.

StrategyMean (s)Median (s)P95 (s)Peak allocated memory (GiB)
Static Hybrid10.470910.487410.655621.43
Learned Static10.464810.456810.720321.43
Adapted GAR10.458210.445010.726821.43
RF10.395310.385210.613621.43

Of 1,200 timed requests, we retained 600 from Static Hybrid and RF after confirming matching software versions and less than 0.1% mean latency drift in 10 paired checks per original strategy. The 600 new requests involved 19,200 page judgments. Paired analyses averaged the three runs per query.

Per-query workloadStatic HybridLearned StaticAdapted GARRF
Visual query encoder calls1111
MiniLM query encoder calls1111
LamRA forward passes (microbatches)4444
Pages scored32323232
Learned selector calls0101
Final ranking head calls1111
Frontier updates (non-model operation)0010
Page selection32 at once32 at once16 then 1616 then 16

Each microbatch contained 8 pages with up to 1,280 visual tokens per page. RF used first-batch scores to compute features and select the next 16 pages in 17.94 ms on average, included in total latency. RF made no 3B question-generation calls. Online timing covered query re-encoding, retrieval, image and reader processing, selection, synchronization, and final ranking. Weights and indexes remained resident. Reader-score, visual-feature, and KV reuse were disabled. Timing excluded loading and answer generation.

Building the shared HR index for 1,110 pages took 193.24 s, plus 0.161 s for the GAR graph. On 800 published SlideVQA images, rerunning OCR, encoding, and indexing took 73.91 s, plus 0.055 s for GAR. We counted shared work once and loading separately. Both timings excluded upstream rendering. We used supplied Markdown for 21 HR fallback pages.

2. Information added by reader feedback

Thank you for suggesting this analysis. Visual and textual retrieval scores measure query–page match. The reader assigns question-specific confidence to each page it reads, indicating which of the first 16 pages should guide subsequent selection.

Learned Static retains nine retrieval features and unweighted seed similarity but omits features that depend on reader scores. It shares RF's 96-tree LambdaMART configuration, source training data, pool, first 16 pages, budget, and ranking head. RF raised HR All@32 from 56.60% to 62.58% (+5.97 percentage points, document-family 95% CI [3.17, 8.39]). We separately retrained Uniform Feedback, retaining seed text, document information, and all 13 inputs while fixing confidence at 1 in training and inference. Its All@32 was 56.60%, giving RF a gain of 5.97 points [0.95, 7.06].

All three signals improved coverage. Same-document support contributed most. Document membership is shared metadata. RF uses the reader’s confidence in previously scored pages to assess candidates from the same document. Removing this signal reduced All@32 from 62.58% to 56.60%. Removing either weighted semantic support or uncovered-term feedback reduced it to 61.95%. Against Static Hybrid, RF raised All@32 from 59.12% to 62.58% (+3.46 points [0.00, 10.73]) and final nDCG@10 from 67.61 to 67.75 (paired difference +0.13 [−2.07, 0.76]). The evidence most directly supports better coverage at a fixed budget.

3. Selector transfer

Thank you for suggesting that we test transfer to a different reader. We replaced 7B LamRA with Qwen3-VL-Reranker-2B. B applied the original selector to feedback from the new reader. C retrained the same architecture using source labels and 15,296 page judgments from the frozen replacement reader on 956 source questions. All three used the same pool, 32-page budget, and new reader's raw scores for final ranking. We used the 318 HR questions only for evaluation.

StrategyRecall@32 (%)All@32 (%)nDCG@10 × 100
A. Static Hybrid82.6059.1266.24
B. Direct reuse82.1157.8666.14
C. Source retraining83.9061.3266.36

B fell slightly below A on all three metrics. C exceeded B by 1.79, 3.46, and 0.22 points, respectively (All@32 document-family 95% CI [0.63, 12.18]). C exceeded A by 1.30, 2.20, and 0.12 points (All@32 CI [−0.75, 10.93]). Source-only retraining thus adapted the original selector architecture to the new reader without target-domain labels.

Thanks for the additional experiments and explanations. I now have a better understanding of this paper, and I will maintain my score at 2.

Reviewer_zyz4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies multimodal page retrieval under a fixed budget of expensive reader judgments. A frozen 7B reader first evaluates the top 16 visually retrieved pages. Its relevance scores inform semantic support, same-document support, and uncovered-query-term features, which a supervised LambdaMART selector uses to acquire another 16 pages from a hybrid candidate pool. All acquisition policies share the same reader and final ranking head. Evaluation covers three development corpora and a separate HR corpus evaluated after the method was frozen. The results support improved annotated-page coverage over a matched uniform-feedback ablation, while the incremental advantage over static hybrid retrieval remains uncertain on the independent target corpus.

Critical strengths
  1. The matched controls provide meaningful evidence that differentiated reader feedback contributes to acquisition quality.

The principal comparisons share the initial 16 pages, a budget of 32 distinct page judgments, the reader, and the final ranking head. In particular, the uniform-feedback selector is separately trained with seed confidences fixed to one during both training and inference, controlling for the candidate pool, textual information, and learning architecture. On HR, the proposed method improves Recall@32 and All@32 over this control by 3.15 and 5.97 percentage points, respectively, with positive document-family bootstrap intervals (Table 8). This supports the usefulness of differentiated reader judgments within the proposed acquisition design.

  1. The evaluation protocol and interpretation of the results are transparent.

The paper distinguishes the three corpora used during method development from the frozen target evaluation, for which complete predictions were reportedly saved before relevance labels were obtained. It also reports uncertainty at both the query and document-family levels, results without the shared ranking head, and acquisition failures. The distinction between annotated-page coverage, final ranking quality, and downstream answer sufficiency makes the findings easier to interpret and strengthens confidence in the paper’s appropriately limited empirical conclusions.

Critical weaknesses
  1. The incremental contribution relative to existing relevance-feedback retrieval methods is not sufficiently established.

The paper presents a useful implementation that operates without a corpus graph, but closely related precedents are missing from its positioning. GAR and Quam already use scores from evaluated candidates to guide subsequent acquisition under a reranking budget; ReFIT provides another relevant approach by using reranker predictions to update query representations for retrieval. These methods differ from the proposed page selector, but their relationship to its central mechanism warrants explicit discussion.

The uniform-feedback ablation establishes that differentiated reader scores help within this particular design. However, without a direct adaptive-acquisition comparator, it remains unclear what additional value the proposed feature construction and learned selection policy provide over existing ways of using relevance feedback. This limits my assessment of the methodological contribution, despite the care taken with the internal controls.

  1. Independent evidence for the additional benefit over static hybrid retrieval is limited.

On the only independent target corpus, HR, the reported improvements over static hybrid retrieval are 0.13 points in nDCG@10, 1.22 percentage points in Recall@32, and 3.46 percentage points in All@32. The corresponding document-family bootstrap intervals are [−2.07, 0.76], [−0.88, 5.45], and [0.00, 10.73] (Table 8). These intervals include or reach zero at the reported precision, leaving the incremental advantage uncertain even when considering the coverage metrics emphasized by the paper.

The three source corpora informed method development, while the frozen HR evaluation contains only six connected document-family groups. This limits the evidence for the robustness and transferability of the additional benefit. The authors acknowledge these limitations; my concern is that the available evidence does not yet make a sufficiently strong case for the proposed selector as an advance over competitive alternatives.

Questions for authors

1.What is the principal methodological distinction between the proposed acquisition policy and GAR/Quam-style adaptive reranking, and can a direct comparison establish the additional value of its feedback features and learned selector? Such a comparison should use the same reader and total budget of 32 distinct page judgments, with explicit accounting for differences in candidate access, offline indexing, and batching. A version restricted to the shared candidate pool could help isolate selection-policy effects, but should be identified as an adaptation rather than a reproduction of the original method. This evidence would materially affect my assessment of the contribution.

2.For the HR comparison against static hybrid retrieval, could the authors report the number of queries and the Recall@32/All@32 differences within each of the six document-family groups, together with a leave-one-group-out sensitivity analysis? A breakdown of newly acquired and lost relevant pages by relevance grade would also help. Section 6.2 already explains why coverage and graded ranking can diverge; the requested analysis would clarify whether the observed coverage benefit is broadly distributed and whether it involves sacrificing highly relevant pages.

1 comment

1. Comparison with GAR and Quam

Thank you for suggesting a direct comparison with adaptive reranking at a matched budget. We evaluated a GAR adaptation restricted to the shared candidate pool. GAR prioritizes the graph neighbors of pages that receive high reader scores. RF uses a source-trained LambdaMART selector to score each unread candidate using confidence-weighted semantic support, same-document support, and uncovered query terms. RF can therefore prioritize pages outside that neighborhood. Quam estimates query affinity on a relevance-aware graph, and ReFIT updates the query vector using reranker predictions. We compare these two methods conceptually.

All policies share the first 16 visually retrieved pages, the candidate pool from visual retrieval, dense text retrieval, and BM25, the frozen LamRA reader, and the final ranking head. Each policy evaluates 32 distinct pages with at most 1,280 visual tokens per page. The GAR adaptation and RF use the first 16 reader scores to select 16 more pages. The adaptation builds a 16-nearest-neighbor graph using cosine similarity between normalized averages of MiniLM text-block embeddings for each page. It uses the authors’ frontier update rule, fills short frontiers from the fusion queue, and disables backfilling with unread pages. Under these restrictions, the authors’ implementation and ours selected identical page sets on all 1,274 queries, including the seven for which the frontier was too small. The table evaluates this shared-pool, two-batch (16+16) adaptation. Original GAR can expand across the corpus over multiple rounds.

HR selection policy (metrics × 100)nDCG@10Recall@32All@32
Static Hybrid67.6182.6059.12
Learned Static67.4581.1056.60
GAR adaptation64.8576.0450.31
RF67.7583.8262.58

RF outperformed the GAR adaptation on all three metrics. Learned Static uses the same LambdaMART architecture and source-domain training data as RF, with the reader-feedback features removed. RF improved All@32 over Learned Static by 5.97 percentage points (document-family 95% CI [3.17, 8.39]), supporting the contribution of feedback to evidence completeness.

Across three runs on the same 100 HR queries on a dedicated RTX 5090, mean latency was 10.40 s/query for RF and 10.46 s/query for the GAR adaptation. Each 16-page batch comprised two 8-page microbatches, and timing included synchronization between stages. Model weights and indexes remained resident, with reuse of scores, visual features, and KV caches disabled. We timed GAR anew and retained the original RF measurements after a paired check showed less than 0.1% drift in mean latency. Building the shared index for 1,110 HR pages took 193.24 s. GAR took an additional 0.16 s to build its graph, which occupied 83 kB. These times exclude encoder loading (1.95 s), page rendering, and upstream OCR/Markdown processing for 21 fallback pages.

2. HR coverage gains by document family and relevance grade

Thank you for suggesting this breakdown by document family and relevance grade. Differences are RF minus Static Hybrid, in percentage points. The first two metric columns give within-family differences. The last two give end-to-end retrieval coverage gains pooled over all queries in the other five families after omitting the listed family. We kept the model and its predictions fixed.

FamilyQueriesWithin-family ΔRecall@32Within-family ΔAll@32Remaining five ΔRecall@32 (approx.)Remaining five ΔAll@32
F1204+0.36+1.96+2.75+6.14
F238+8.46+15.79+0.23+1.79
F334+2.15+5.88+1.11+3.17
F422−3.420.00+1.56+3.72
F513+0.260.00+1.26+3.61
F67−1.30−14.29+1.27+3.86

Both gains remained positive after each exclusion. Removing F1, the largest family, left 114 queries. Recall@32 still improved by approximately 2.75 points and All@32 by 6.14 points, a net gain of seven queries with all annotated relevant pages retrieved. Within F1–F3, RF achieved complete coverage on 4, 6, and 2 additional queries, respectively. The counts were unchanged in F4 and F5 and fell by one among the seven queries in F6. Recall@32 rose in F1, F2, F3, and F5. F2 showed the largest gains on both metrics.

RF gained 147 and lost 59 query–page pairs annotated as relevant (net +88). At grade 1, 136 gains and 56 losses yielded a net gain of 80. At grade 2, 11 gains and three losses yielded a net gain of eight highly relevant pairs. We counted unjudged pages separately. Overall, All@32 improved by 3.46 percentage points (document-family 95% interval [0.00, 10.73]). Graded nDCG@10 differed by +0.13 points (interval [−2.07, 0.76]). Gains across families were more consistent for evidence coverage in the top 32 pages.

Reviewer_bCRCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reader Feedback, a two-batch acquisition framework for multimodal document retrieval under a fixed reading budget. The method first evaluates 16 visually retrieved pages using a frozen 7B multimodal reader, then leverages reader confidence scores, semantic similarity, document membership, and uncovered query terms to select another 16 pages through a LambdaMART-based selector. Experiments on four ViDoRe V3 corpora demonstrate improvements in relevant-page coverage compared with visual-only retrieval and several hybrid baselines. However, improvements in final ranking performance remain limited.

Critical strengths

The paper addresses a meaningful limitation of conventional multimodal document retrieval, where relevant evidence excluded during initial retrieval cannot be recovered by subsequent reranking. Incorporating intermediate reader feedback into candidate acquisition is a reasonable and potentially useful direction.

The experimental evaluation is relatively thorough, with matched reading budgets, multiple baselines, and a separately evaluated target corpus. The distinction between evidence coverage and final ranking quality also provides useful insights.

Critical weaknesses
  1. Limited methodological novelty. The proposed approach primarily combines existing retrieval models, manually designed feedback features, and a standard LambdaMART ranker. The technical contribution appears incremental, without a substantially new learning mechanism or optimization formulation. Moreover, the paper lacks a learned static selector using the same architecture but without reader-dependent features, making it difficult to isolate the actual benefit of reader feedback from supervised retrieval fusion.

  2. Marginal improvements over strong baselines. Although the method improves relevant-page coverage, its advantage in final ranking performance is negligible compared with static hybrid retrieval. On the HR dataset, nDCG@10 improves only from 67.61 to 67.75, with a confidence interval spanning zero. Furthermore, the paper does not evaluate whether improved coverage translates into better downstream question-answering performance. These limitations weaken the practical significance of the proposed approach.

Questions for authors

Could the authors compare against a learned static retrieval selector with the same LambdaMART architecture and training data but without reader-dependent features? Additionally, what are the individual contributions of the three proposed feedback features?

Can the authors provide evidence that improved relevant-page coverage leads to better downstream question-answering performance, or demonstrate a meaningful advantage in end-to-end efficiency over static hybrid retrieval?

1 comment

1. Matched learned static control

Thank you for suggesting a matched learned static control. We added Learned Static, which retains the nine retrieval features and unweighted seed-text similarity, removes the three reader-dependent features, and selects pages before the reader scores them. Learned Static and RF use the same 96-tree LambdaMART architecture, source-domain training data, data splits, and sample weights. They also share the candidate pool, the first 16 pages selected by visual retrieval, the frozen reader, and the final ranking head. Both methods score 32 distinct pages, with a limit of 1,280 visual tokens per page. For the HR experiments, we trained each selector on pooled data from the three source domains. We used no HR data for fitting or tuning.

HR strategy (all metrics × 100)nDCG@10Recall@32All@32
Static Hybrid67.6182.6059.12
Learned Static67.4581.1056.60
RF67.7583.8262.58

Compared with Learned Static, RF improves Recall@32 by 2.71 percentage points (95% document-family-level CI [0.77, 5.32]) and All@32 by 5.97 percentage points (95% CI [3.17, 8.39]). Across the 318 questions, RF retrieved all annotated relevant pages for 19 more questions than Learned Static. We also retrained Uniform Feedback separately, retaining all 13 features and setting all reader confidence scores to 1 during training and inference. Its All@32 is 56.60%. RF scores 5.97 percentage points higher (95% CI [0.95, 7.06]), indicating that differences in reader confidence across pages help guide subsequent page selection.

2. Contributions of the three feedback signals

Thank you for suggesting that we examine each signal individually. We removed one signal at a time and retrained the selector under the same conditions. On HR, the full RF model achieves higher Recall@32 and All@32 than each of the three ablated models, indicating that all three signals contribute to evidence coverage within the 32-page budget.

HR selector (all metrics × 100)nDCG@10Recall@32All@32
Full RF67.7583.8262.58
Without weighted semantic support67.6083.3661.95
Without same-document support67.4481.5356.60
Without uncovered-term feedback67.7983.6961.95

Same-document support makes the largest contribution. Using the reader’s confidence in pages it has already scored to guide the selection of other pages from the same document raises All@32 from 56.60% to 62.58% and nDCG@10 from 67.44 to 67.75. Weighted semantic support also improves both coverage and ranking. Uncovered-term feedback is designed to capture query terms that the pages already read do not adequately cover. Retaining this signal raises All@32 from 61.95% to 62.58%, with nDCG@10 only 0.04 points lower. Its main benefit is therefore more complete evidence coverage. With all three signals, RF outperforms the matched Learned Static control on both coverage metrics and on final ranking.

3. Downstream question answering and evidence use

Thank you for suggesting a downstream question-answering evaluation. We compared the three retrieval strategies on the same 318 questions. For all three, Qwen2.5-VL-7B-Instruct generated answers from the eight highest-ranked pages, with a maximum output length of 512 tokens. We evaluated all 954 answers for correctness against the reference answers. The original evaluation used the same model as the generator. For the independent evaluation, we used InternVL2.5-8B, concealed the strategy names, and randomized the order of the answers.

StrategyAll@32 (%)QA accuracy, original judge (%)QA accuracy, independent judge (%)
Static Hybrid59.1243.7157.23
Learned Static56.6044.6557.86
RF62.5846.5458.81

With the generator and context budget held fixed, RF achieved the highest All@32, and both judges ranked RF first in overall QA accuracy. In the independent evaluation, RF outperformed Static Hybrid and Learned Static by 1.57 and 0.94 percentage points, respectively. The query-level confidence interval for the former difference was [−0.63, 4.09], and the document-family-level interval for the latter was [−4.30, 2.70]. Both evaluations showed a gain in the same direction.

The generator can use a newly retrieved relevant page directly only if it reaches the top eight. Of the relevant query–page pairs that RF gained over Static Hybrid, 35 reached the eight pages actually supplied to the generator. Of the 64 questions for which Recall@32 improved, 23 also showed an improvement in Recall@8. Together with the citation support rates (53.46% for RF and 55.97% for Static Hybrid), these results indicate that improving Recall@32 increased the likelihood that relevant pages reached the generator, thereby improving end-to-end accuracy.

Reviewer_HGGwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Reader Feedback, a two-stage multimodal page-acquisition method under a fixed budget of 32 reader judgments. A frozen 7B reader scores the first 16 visual candidates, and a learned selector uses those scores together with text similarity, document membership, and query coverage to choose 16 additional pages. Experiments on three development corpora and a frozen HR corpus show improved annotated evidence coverage, while the gains in final nDCG over static hybrid retrieval remain small.

Critical strengths
  1. The paper studies a concrete limitation of fixed-candidate retrieval. Using reader feedback to acquire additional pages under the same reading budget is a sensible way to recover evidence missed by initial retrieval.
  2. The experimental design carefully separates acquisition from reranking. Visual retrieval, static hybrid retrieval, and uniform-feedback controls use the same frozen reader, the same 32-page judgment budget, and the same final ranking head.
  3. The independent HR evaluation provides useful evidence for improved coverage. Table 3 reports All@32 of 62.58 versus 56.60 for uniform feedback, and the document-family interval for this difference is positive, supporting a contribution from actual reader scores.
  4. The analysis is transparent about where the method does not help. It reports uncertainty intervals, a ranking oracle, examples of evidence displacement, and ablations showing little benefit from query decomposition or signed feedback.
Critical weaknesses
  1. The improvement in the primary ranking metric is limited. On the frozen HR corpus, Table 3 reports nDCG@10 of 67.75 versus 67.61 for static hybrid retrieval, and the interval for the +0.13 difference includes zero. Reader-only ranking even slightly favors static hybrid, so better evidence coverage has not produced a clear ranking gain.
  2. The main coverage metric does not establish that users receive better answers. All@32 requires every annotated positive page, including potentially redundant evidence, while the paper does not evaluate generated-answer correctness or evidence sufficiency. The HR candidate oracle is also slightly lower than static hybrid (89.88 vs. 89.98), despite better coverage.
  3. Generalization is assessed on only one independent target corpus. HR contains 318 queries from 14 source documents and only six connected document families for the dependence-aware bootstrap. The other three corpora informed repeated method development, limiting the strength of broader cross-domain conclusions.
  4. The fixed budget equalizes reader judgments but not end-to-end cost. The proposed pipeline needs text indexing, CPU features, a learned selector, and synchronization between two reader batches. The paper does not report the query latency or total computational overhead relative to a static hybrid pipeline.
Questions for authors
  1. Could the authors evaluate whether the higher All@32 translates into better downstream multimodal question answering, ideally using an identical answer generator and context budget?
  2. Could the authors provide additional evidence that feedback improves final ranking beyond a strong static hybrid acquisition policy, or clarify whether the main contribution should be judged only as evidence coverage?
  3. How stable are the coverage and ranking effects on another held-out document domain or on a larger collection with more independent document families?
  4. Could the authors report end-to-end per-query latency or compute, including text indexing or feature extraction and the synchronization between reader batches, against static hybrid retrieval?
1 comment

1. Downstream QA and evidence use

Thank you for suggesting a QA comparison with a matched generator and context budget. All three policies used Qwen2.5-VL-7B-Instruct on the same 318 questions, with the top eight pages and a 512-token output limit. We evaluated all 954 answers against the reference answers. The original judge used the same model as the generator. For the independent evaluation, InternVL2.5-8B judged the answers in random order with policy names masked.

PolicyAll@32 (%)QA accuracy, original judge (%)QA accuracy, independent judge (%)
Static Hybrid59.1243.7157.23
Learned Static56.6044.6557.86
RF62.5846.5458.81

Reader Feedback (RF) achieved the highest All@32 and ranked first in QA accuracy under both judges. In the independent evaluation, RF exceeded Static Hybrid by 1.57 percentage points (query-level confidence interval [−0.63, 4.09]) and Learned Static by 0.94 percentage points (document-family CI [−4.30, 2.70]). Of the relevant query–page pairs that RF gained over Static Hybrid, 35 reached the top eight pages supplied to the generator. Of 64 questions with higher Recall@32, 23 also had higher Recall@8. Citation support was 53.46% for RF and 55.97% for Static Hybrid.

2. Evidence coverage and final ranking

Thank you for suggesting separate coverage and ranking comparisons. Learned Static retains nine retrieval features and unweighted seed similarity and omits three reader-feedback inputs. It shares RF's 96-tree architecture, source-domain training data and sample weights, candidate pool, initial 16 pages, budget of 32 distinct pages, frozen reader, and final ranking head.

PolicyRecall@32 (%)All@32 (%)nDCG@10 × 100
Static Hybrid82.6059.1267.61
Learned Static81.1056.6067.45
RF83.8262.5867.75

Across 318 questions, RF retrieved all annotated relevant pages for 19 more questions than Learned Static, a net gain of 5.97 percentage points in All@32 (document-family CI [3.17, 8.39]). RF exceeded Static Hybrid by 0.13 points on nDCG@10 (CI [−2.07, 0.76]) and Learned Static by 0.29 points (CI [−1.69, 0.92]). Feedback improved the observed final ranking score in both comparisons. Its clearest benefit was more complete evidence coverage under a fixed budget.

3. Coverage and ranking in a held-out document domain

Thank you for suggesting a cross-domain evaluation. We preselected 219 English questions from the SlideVQA test set, which was held out from development and selector training. Of these, 64 required evidence from multiple pages. The 40 documents formed 40 groups based on uploader identity and duplicate images. We pooled all 800 pages, withheld the correct document identity at retrieval, and retained the source-domain selectors, reader, and ranking head.

The shared initial 16 pages covered all annotated evidence for 212/219 questions (96.80%). Both RF and Static Hybrid reached 98.17% All@32 (CI for the difference [−1.35, 1.36]). Their binary nDCG@10 × 100 scores were 94.51 and 94.53, respectively (CI for the difference [−0.45, 0.31]). On HR, RF increased All@32 from 59.12% to 62.58% and scored slightly higher on graded nDCG@10. On SlideVQA, where coverage approached its ceiling, RF matched the baseline's high coverage and achieved similar ranking performance.

4. End-to-end retrieval latency and computational cost

Thank you for suggesting that we account for feedback synchronization and indexing costs. On a dedicated RTX 5090, we measured each policy on the same 100 HR queries over three rounds (1,200 measurements). Each policy scored 32 pages in four microbatches of eight. Weights and indexes remained resident, and reuse of scores, visual features, and the KV cache was disabled.

PolicyMean latency (s/query)P95 latency (s/query)
Static Hybrid10.470910.6556
Learned Static10.464810.7203
GAR adaptation10.458210.7268
RF10.395310.6136

RF reads 16 pages and uses their feedback to select the next 16. Online timing includes synchronization between stages, query re-encoding, retrieval, image and visual processing, feature computation, page selection, and final ranking. It excludes model loading and answer generation. For RF, feedback feature computation and page selection averaged 17.94 ms. Peak allocated GPU memory was approximately 21.43 GiB. At the same reading budget, RF and Static Hybrid differed in mean latency by only 0.076 s (0.72%) and had similar P95 latencies.

Building the shared offline index for 1,110 HR pages took 193.24 s, excluding page rendering and upstream OCR/Markdown processing for 21 fallback pages. Indexing the 800 released SlideVQA page images, including rerunning OCR, took 73.91 s. At comparable online cost, RF achieved better retrieval performance than both the strong static baseline and the GAR adaptation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.