acceptodds
Under review as a conference paper at ICLR 2027

MMProvence: Turning a Multimodal Reranker into a Zero-Cost Context Pruner

Abstract

Multimodal RAG pipelines pass the retrieved pages to a vision-language model (VLM) as images together with their text, and the page images make up most of the model's input, yet existing context pruners operate on text only. We introduce MMProvence, a zero-cost context pruner for multimodal RAG: a VLM reranker that, in a single call per query-page pair, generates a relevance score, a bounding box on the page image, and the page sentences that answer the query. The score reranks the candidates, and the box and the sentences prune the image and the text of the pages that survive reranking, so pruning requires no additional model call. We further show that what the box is trained to cover decides the cost of cropping: on top of text pruning, crops to boxes that include the context that makes the evidence readable, such as table headers or chart legends, lower the LLM-as-a-judge score by 2.7 points, against 13.3 points for boxes around the smallest answer-bearing region. On the eight domains of ViDoRe V3, MMProvence-2B reduces the generator's input by 75%, from 21.6 to 5.3 tokens, at an LLM-as-a-judge score of 0.803 against 0.827 with the full context, while ranking on par with the reranker it replaces (nDCG@10 of 0.601 vs. 0.604). At 4B and 9B parameters, MMProvence ranks better than this reranker (0.627 and 0.636), and MMProvence-9B keeps a score of 0.822 while removing 76% of the input. We will release our code, checkpoints and training data, together with VERGE, our framework for end-to-end multimodal RAG evaluation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal RAG, Context Pruning, Context Compression, Reranking, Vision-Language Models, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WED7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this paper, the goal is to reduce the high input cost of multimodal document RAG. I would consider this work a multimodal extension of Provence, which performs zero-cost text pruning during reranking. MMProvence extends this idea to both text and page images. To do this, for each query-page pair, the reranker jointly predicts a relevance score, a relevant image region, and the relevant text sentences. The score is used for reranking, while the predicted region and sentences are used to prune the selected pages before they are passed to the final VLM, without requiring an additional pruning-model call. I would also to clarify that the ‘zero-cost text pruning’, specifically means that pruning does not require an additional model call; it does not mean that pruning has no computational cost. In fact, the reranking stage becomes more expensive because the model must generate the bounding box and relevant sentences in addition to the relevance score.

Critical strengths
  • The proposed approach is simple, as reranking and multimodal context pruning are performed within the same model call.
  • The contextual bounding-box idea is useful.
  • The paper provides extensive ablations • The method achieves substantial context reduction.
  • MMProvence-2B reduces improves the efficiency while preserving the performance for ViDoRe V3 benchmark dataset.
Critical weaknesses
  • Limited novelty and missing closely related work/comparison. A key contribution of the paper is to reduce unnecessary visual context by identifying the query-relevant region of a retrieved page instead of passing the full-page image to the generator. A closely related idea has already been explored in RegionRAG[1], which performs region-level retrieval to expose query-relevant visual regions rather than full document pages. The main difference is that MMProvence integrates visual-region selection, text pruning, and reranking into the same call. However, RegionRAG is neither cited nor compared against. A direct comparison would help clarify whether MMProvence provides a meaningful improvement over existing region-level visual RAG approaches, particularly in terms of answer quality, context reduction, and end-to-end efficiency.

  • Limited evaluation scope. The end-to-end QA experiments are conducted only on the eight domains of ViDoRe V3. related work such as RegionRAG has been evaluated across a broader range of document QA settings. Evaluating MMProvence on additional datasets such as DocVQA, InfographicsVQA, SlideVQA, or PlotQA would provide stronger evidence that the proposed pruning approach generalizes beyond ViDoRe V3.

  • The term “zero-cost” can be misleading. It is true that MMProvence does not require an additional pruning-model call. However, pruning is not computationally free: generating the bounding box and relevant sentences makes the reranking stage more expensive. For example, MMProvence-2B reduces the generator input by about 75%, but provides only around 11–13% end-to-end latency reduction in the main Qwen3.5-9B generator setting. Moreover, the larger MMProvence-9B variant is actually slower than the full-context pipeline. Therefore, I think it would be more precise to describe the method as“pruning without an additional model call” rather than simply “zero-cost pruning.”

[1] Y. Li, Z. Lu, Z. Liu, Y. Sun, C. Liu, and H. Xie, “RegionRAG: Region-level Retrieval-Augmented Generation for Visual Document Understanding, on arxiv since december 2025, and AAAI 2026.

Questions for authors
2 comments

We thank the reviewer for their feedback. Below, we reply to the weakness:

Limited Novelty: Our main contribution is integrating the pruning inside the reranker which results in efficiency gains since we don't have to load an additional pruning model in memory or perform pre-filling of the Query+Context prompt. Our sentence index mmprovence full pipeline is 23%/33% faster (with/without thinking) and uses 33% less compute. To our best knowledge, no prior work integrated pruning in the reranker for multimodal RAG applications.

Missing related work and comparison with RegioRAG: We have added RegionRAG as a baseline to the paper. Below are the experimental results against RegionRAG on Vidore benchmark.

RetrievalRerankerPruningnDCG@10 ↑Tokens ↓Red.TextImageLLMeval ↑Δ
Qwen3-VL-Embedding-2BMMProvence-2BMMProvence crop0.6017.9k-63%0%74.6%0.822-0.005
Qwen3-VL-Embedding-2BMMProvence-2BRegionRAG crop0.6016.6k-70%0%76.4%0.809-0.018
RegionRet-3B—RegionRAG crop0.3886.9k-68%0%74.3%0.639-0.188

For fairness, we compared against RegionRAG in 2 setups: S1 uses the full pipeline from their paper (RegionRET+RegionRAG), while S2 uses MMprovence-2B as the reranker to isolate the cropping effect of both models. We tried our best to choose the RegionRAG threshold that matches the compression rate of MMProvence. We outperform both approaches, especially S1 where we outperform it by 28% relative improvement.

Regarding runtime, our model is 20% faster than the closest RegionRAG setup while outperforming it. The details are below:

StageMMProvence, image onlyMMProvence, image + sentencesOur reranker + RegionRAGRegionRet + RegionRAG
Retrieval0.020.020.024.00
Rerank (50 pages)6.396.396.39—
Pruning (top 5)0.300.302.662.66
Generation8.427.769.5413.74
Total15.1314.4718.6120.40

Limited evaluation scope: We include MP-DocVQA & InfoVQA from the VisRAG project with image input (no OCR). Below are the results of our model compared to the full context baseline. We will include the results in the main paper. On DocVQA our model achieves ~80% token reduction with minimal drop in performance (0.968 -> 0.939). On the InfoVQA, we achieve ~75% compression while suffering a bigger performance drop (0.93 -> 0.85). Such difference in performance can be attributed to the training data having little infographic samples.

MP-DocVQA

RerankerPruningnDCG@10 ↑Tokens ↓Red.TextImageLLMeval ↑Δ
Qwen3-VL-Reranker-2Bnone0.93718.6k0000.968-
MMProvence-2Bimage crop0.9423.8k79.3%076.8%0.939−0.029

InfoVQA

RerankerPruningnDCG@10 ↑Tokens ↓Red.TextImageLLMeval ↑Δ
Qwen3-VL-Reranker-2Bnone0.97015.9k0000.930-
MMProvence-2Bimage crop0.9724.0k74.6%069.3%0.851−0.079

Zero-Cost claim: We agree that the term needs clarification. By "zero-cost" we mean that the pruning stage needs no additional model call. The box and the relevant sentences are produced by the reranking call itself, so no pruning model is loaded in memory and the Query+Context prompt is not pre-filled a second time. The cost of pruning moves into the reranking stage as output tokens. This is why reranking takes 0.75 / 1.48 / 5.47 s longer at 2B / 4B / 9B, and why MMProvence-9B ends up slower than the full-context pipeline. Most of these output tokens are copied sentences. To address this, we trained a sentence-index variant of MMProvence-2B (Table 5, last row) that outputs sentence numbers instead of copying the sentences. It cuts the reranking output from 9,578 to 2,700 tokens per query (−72%). Below are the efficiency results with the Qwen3.5-9B generator on H100 (median seconds per query at batch size 1, 392 queries; Δ against the full-context pipeline):

RerankerRerank (s)Total, thinking (s)Total, no thinking (s)TFLOPs
Qwen3-VL-Reranker-2B (full context)6.6121.8010.941,180
MMProvence-2B7.3719.37 (−11%)9.48 (−13%)818 (−31%)
MMProvence-2B, sentence indices5.2216.86 (−23%)7.37 (−33%)789 (−33%)

With the sentence-index variant, the full pipeline is 23%/33% faster (with/without thinking) and uses 33% less compute. Outputting sentence numbers does not hurt answer quality: −0.002 ± 0.011 LLMeval compared to the copied-sentence model.

We will follow the reviewer's suggestion and use "pruning without an additional model call" in the paper. We will define the term at first use and state clearly in RQ5 that the end-to-end gains depend on model size.

Dear Reviewer,

Thanks for taking the time to review our work. In the rebuttal, we have added the RegionRAG results in both the original paper setting as well as in a more fair configuration. Yet, our model still outperforms it. Additionally, we have reported results on DocVQA and InfoVQA. Since there is less than 24 hours remaining till the end of the discussion period, we would appreciate it if you can take another look at our responses and let us know if there are any remaining concerns.

Reviewer_2jbVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In the context of Multimodal RAG, the authors situate this research around the issue of extensive retrieval primarily in the form of image tokens, which due to the quadratic scaling of self-attention, becomes computationally expensive for the multimodal generator to use as input. Thus, the authors introduce MMProvence, a zero-cost context pruner for multimodal RAG, by training a VLM reranker that generates three artifacts all in a single forward pass: a relevance score for a given query-page pair (the usual task of the reranker), a bounding box on the image from the page, and the page sentences which answer the query (the two additional artifacts). As per the main reporting in the paper MMProvence is trained in two phases: for the first phase which teaches to rank, starting from Qwen3.5, the authors use Qwen3-VL-Reranker-8B as the teacher model, and LoRA fine-tune the student on the objective of minimizing the sum of 1) the MSE between the output of the score head at the <|reranking|> token of the student model and the relevance probability of the teacher model, and 2) the cross-entropy of the target JSON object, whose score field is P_T (the relevance probability of the teacher model) written as an integer from 0 to 100. For the second phase of training, the phase-1 adapter is trained on the full JSON object to be output by MMProvence with all three artifacts (relevance score, bounding box, list of text). The authors validate MMProvence on ViDoRe V3, and establish the efficacy of MMProvence by reporting the best outcomes achieved by MMProvence-9B which reduced the generator’s input by 76% while observing only a drop from 0.827 (full-context) to 0.822 in the LLMaaJ quality score and improving upon the reranker it replaces by achieving nDCG@10 of 0.636 (as opposed to 0.604).

Critical strengths
  1. The paper tackles a critical problem of expansive context for Multimodal RAG, which is very timely as the use of Large Multimodal Models increases and RAG remains an essential pipeline to keep responses up-to-date and grounded.
  2. The paper is well written, easy to follow, and most details are provided transparently for ease of reproducibility.
  3. The authors ground their approach in practical constraints and come up with a novel multimodal pruning solution that is zero-cost while substantially reducing input tokens for the generator with limited drops in nDCG and LLMaaJ quality scores. For those reasons, their novel approach is appreciable.
Critical weaknesses
  1. In the evaluation pipeline, the authors use the same model for the generator as well as the LLM-as-a-judge (Qwen3.5-9B, as reported in Section 5 Pipeline and Evaluation), which may be the source of some critical bias in the evaluation. It is unclear whether the choice of the same model for both is due to the authors following BERGEN. Since the authors establish the efficacy of their approach by reporting minimal drop in LLMaaJ scores while substantially reducing context input to the generator, this potential bias stands as a critical weakness.
  2. Nowhere in the paper do we see reporting of confidence intervals or any evidence of statistical testing. The differences in performance may not be statistically significant, and would thereby significantly weaken the contribution of the paper and the effectiveness of the approach.
Questions for authors
  1. Could the authors provide substantiation for the choices of models used (retriever, reranker, teacher and student for MMProvence, LLM as a Judge). Without any reasoning for these choices, the decisions seem arbitrary, and results may not hold for a different family of models other than Qwen.
  2. Could the authors explain the motivation to use the sum of the MSE and the cross-entropy of the target JSON object, both of which capture the distance between the output of the student model’s score head and the teacher’s relevance probability, but in different formats?
  3. Could the authors kindly clarify and report the objective function of the second phase of training? Is it simply cross-entropy on all target tokens?
  4. Quality Assessment for the re-annotation (Section 4 Contextual boxes): could the authors report on any steps that were taken to systematically evaluate the quality of the reannotation?
  5. Were ablations considered for the prompted pruners to run after Qwen3-VL-Reranker-8B, the teacher model? Or for MMProvence to run after the same teacher model?
  6. Could the authors provide any qualitative assessment about the failure mode where relevance score may be high due to the full context of the page, but the pruning by MMProvence may have removed the specific relevant content needed for the query?
  7. Could the authors provide further details about the LLMaaJ procedure since that is the critical quantitative assessment of the efficacy of MMProvence? Currently, only the prompt for the judge is provided in the Appendix.
5 comments
AuthorsOpenReview ↗

Response 1

We thank the reviewer for the detailed and constructive feedback. We are encouraged by the reviewer's positive view of our work. Below, we reply to the weaknesses and questions:

Same model as generator and judge: All the systems we compare use the same generator, Qwen3.5-9B, and differ only in the context it receives. A judge that favors its own model's answers would therefore favor the baseline and every pruned system alike, and the bias would largely cancel in the deltas we report. The judge also grades against human-verified gold answers rather than open-endedly. The choice was practical (an open-weight model we already serve, with a reasoning mode), not a convention inherited from BERGEN. To test it directly, we re-judged all 20 rows of Table 1 with Gemma 4 12B, a model from a different family, using the same predictions, the same 4,000 queries and the same prompt. The row ordering is preserved, and 93.5% of the 80,000 verdicts agree. Gemma is stricter overall (−0.017 LLMeval on average), but the deltas against the full-context baseline match:

Table 1 rowΔ, Qwen3.5-9B judge (paper)Δ, Gemma 4 12B judge
Qwen3-VL-Reranker-8B, no pruning+0.021*+0.020*
MMProvence-2B crop only / crop + text−0.005 / −0.024*−0.006 / −0.028*
MMProvence-4B crop only / crop + text+0.013* / −0.017*+0.007 / −0.021*
MMProvence-9B crop only / crop + text+0.006 / −0.006+0.005 / −0.004
Best prompted pruner (extractive 9B, reasoning)−0.007−0.007

(* paired 95% CI excludes zero.) The only conclusion that changes is MMProvence-4B crop only, which we discuss below.

Confidence intervals and statistical testing: We agree, and we will add 95% CIs and significance to every LLMeval table. We measured two sources of variance over the 4,000 queries (8 domains × 500):

  • Query sampling: 10,000 domain-stratified bootstrap resamples give 95% CIs of ±0.011–0.013 on absolute LLMeval. Since every system answers the same queries, we test differences with a paired bootstrap, whose CIs are ±0.009–0.011.
  • Judge: we re-ran the judge three times on six key models from table 1. The per-row standard deviation is ≤ 0.002, and 1.2–1.8% of verdicts flip between runs.

Table 1 with paired 95% CIs against the full-context baseline (Qwen3-VL-Reranker-2B, no pruning, 0.827):

RerankerPruningLLMevalΔ [95% CI]
None (retriever only)none0.761−0.067 [−0.079, −0.055]
Qwen3-VL-Reranker-8Bnone0.848+0.021 [+0.011, +0.031]
MMProvence-2Bcrop only0.822−0.005 [−0.015, +0.005]
MMProvence-2Btext only0.825−0.003 [−0.013, +0.007]
MMProvence-2Bcrop + text0.803−0.024 [−0.035, −0.014]
MMProvence-4Bcrop only0.840+0.013 [+0.003, +0.022]
MMProvence-4Btext only0.832+0.005 [−0.006, +0.015]
MMProvence-4Bcrop + text0.811−0.017 [−0.027, −0.006]
MMProvence-9Bcrop only0.833+0.006 [−0.004, +0.015]
MMProvence-9Btext only0.839+0.012 [+0.003, +0.022]
MMProvence-9Bcrop + text0.822−0.006 [−0.016, +0.005]
Qwen3-VL-Reranker-2Bprompted pruners (8 rows)0.806–0.821−0.022 to −0.007, significant for 7 of 8

The main conclusions hold under all five judgings (the paper's, the three re-runs and Gemma): crop + text costs answer quality at 2B and 4B; MMProvence-9B crop + text and every crop-only row stay within ~0.01 of the full-context baseline; and the 8B reranker scores above the 2B (+0.008 to +0.021).

One claim does not hold. MMProvence-4B crop only's +0.013 over the baseline falls to +0.001 on average across the three re-runs, and to +0.007 (not significant) under Gemma. We will revise it to "matches the full-context baseline with 64% fewer generator tokens" instead of "outperforms the baseline".

AuthorsOpenReview ↗

Response 2

Choice of models (Q1): We used open-weight models that we can serve locally with vLLM, so that every number is reproducible, and for each role we chose the strongest such model available when we started the project:

  • Reranker and teacher. To our knowledge, Qwen3-VL-Reranker-2B and -8B are the strongest open multimodal rerankers on ViDoRe V3 (60.8 and 66.7, as reported by their authors). MMProvence-2B replaces the 2B, so the full-context baseline is the reranker a practitioner would otherwise deploy. The 8B, the stronger of the two, provides a graded relevance probability for each of the 6.0M training pairs.
  • Student. Qwen3.5 is natively multimodal, can output boxes, and comes in 2B, 4B and 9B, which lets us study scale within one family.
  • Retriever. Qwen3-VL-Embedding-2B embeds a page's image and text jointly, reports results on ViDoRe V3, and is small enough to index the 18,988 pages of the eight corpora.
  • Generator and judge. Qwen3.5-9B is a strong open VLM that fits on one GPU with five native-resolution pages and a 40k-token reasoning budget. The Gemma 4 results above show that our conclusions do not depend on using it as the judge.

We acknowledge that the reranker, the student and the generator all come from the Qwen family, and we will state this in the limitations. The recipe itself only requires a VLM that can write a box and text, and nothing in it is specific to Qwen.

MSE + cross-entropy (Q2): The cross-entropy does not involve the score head: the two terms train two different outputs from the same target.

  • The cross-entropy trains the language-model head to write the JSON object. Its first field is the score we rank by, generated in the same decoding as the box and the sentences, so ranking and pruning need a single call.
  • The MSE trains a separate score head that reads the hidden state at the <|reranking|> token, with the same σ(l_yes − l_no) readout as Qwen3-VL-Reranker. This head gives a continuous score from a single forward pass, without decoding, so the same model can also serve as a plain cross-encoder when pruning is not needed. So we kept the MSE loss to support legacy pipelines that don't support generating new tokens in the reranker.

Appendix B compares the two scores on 1,043 held-out pairs: the MSE against the teacher is 0.0033 for the head and 0.0052 for the textual score (RMSE of about 6 vs. 7 points on the 0–100 scale). We rank by the textual score because it is produced together with the evidence. We will add this motivation to Section 4.

Phase-2 objective (Q3): Phase 2 uses the same objective as Phase 1, the sum of the two terms with equal weight (1.0 each): the MSE between the score head and the teacher's probability P_T, plus the cross-entropy over all tokens of the target JSON object, i.e., the score, the box coordinates and the sentences, including the empty lists when a page holds no evidence of a kind. The prompt tokens are not part of the loss. The only difference from Phase 1 is the target: {"score": s} in Phase 1, the full object {"score": s, "bbox_2d": [...], "text": [...]} in Phase 2. We will write the objective as an equation in Section 4: L = (h(x) − P_T)² − (1/|y|) Σ_t log p_θ(y_t | x, y_<t), where h(x) is the score head's output at <|reranking|> and y is the target JSON object.

Quality of the re-annotation (Q4): We took three steps:

  1. By construction: the training box is the union of the re-annotating model's box and the original Visual-CoT box, padded by 1% of the image. It therefore always contains the original annotated evidence, and rows that the model fails to annotate keep their original box.
  2. Statistics (Appendix D): the box grows for 99.8% of the rows, from 14.6% to 34.6% of the image on average. It grows most where the original boxes are tightest, i.e., text in natural images and document pages.
  3. Downstream evaluation: ViDoRe V3 has no gold boxes, so we evaluate the boxes through their effect on the answers. With the same recipe, training on contextual instead of tight boxes reduces the cost of the crop from −0.133 to −0.027 LLMeval (RQ4). During the rebuttal, we added a control for box size: we padded the tight-box model's boxes until its generator input matched MMProvence-2B's (median 5,250 vs. 5,364 tokens).
Boxes (crop + text, 4,000 queries)LLMeval
Tight (RQ4)0.686
Tight + padding, same token budget0.748
Contextual (MMProvence-2B)0.796

(The last two rows were generated on the same hardware for this comparison.) Padding recovers part of the loss, but at equal tokens it stays 0.048 below contextual boxes (95% CI ±0.013) and is lower on all 8 domains. Enlarging a tight box grows it evenly around the answer, while the re-annotated box reaches the headers, axis labels and legends needed to read it.

AuthorsOpenReview ↗

Response 3

Ablations behind the teacher (Q5): We ran them during the rebuttal. With the ranking fixed to Qwen3-VL-Reranker-8B, we compare the prompted extractive Qwen3.5-9B pruner with MMProvence-9B as a standalone pruner (crop + text). The generator, the judge and the 4,000 queries are the same as in Table 1:

Pruner after Qwen3-VL-Reranker-8BLLMevalΔ vs. no pruning [95% CI]Median input tokensImages dropped
None0.843—21.8k0%
Prompted extractive Qwen3.5-9B0.823−0.019 [−0.029, −0.010]6.6k (−70%)14.1%
MMProvence-9B (standalone)0.822−0.020 [−0.030, −0.011]5.3k (−76%)0.1%

The two pruners tie (Δ −0.001, 95% CI [−0.012, +0.009]), as they do behind Qwen3-VL-Reranker-2B (0.810 vs. 0.808). MMProvence-9B sends 20% fewer tokens to the generator and almost never drops a page image, whereas the prompted pruner drops 14% of them. For reference, MMProvence-9B in both slots reaches the same 0.822 (Table 1) with a single model instead of the teacher plus a pruner.

Failure mode (Q6): We can measure this failure mode directly. In our main setting, MMProvence both reranks the pages and prunes them, using the same output. We ran this setting twice: once giving the generator the five top-ranked pages in full, and once giving it their pruned versions. The ranking is identical in both runs, so any difference in the answers comes from pruning alone.

Over the 4,000 queries, pruning turns a correct answer into a wrong one on 263 queries for MMProvence-2B (6.6%) and 272 for MMProvence-4B (6.8%). In 218 (2B) and 223 (4B) of the queries that became wrong, a page annotated as relevant in ViDoRe V3 was among the five pages given to the generator, but the generator no longer answered correctly after pruning. These queries are exactly the case the reviewer describes: the page was ranked high, but pruning removed the content needed for the answer.

The most direct way this happens is that MMProvence selects no sentence on the page, so the whole text of the page is removed. This is rare: it happens on 3.2% (2B), 2.7% (4B) and 0.6% (9B) of the pages given to the generator. MMProvence almost never predicts an empty box (at most 0.04% of pages), so it almost never removes a page image. Reading the failed queries, we found three other ways the needed content is removed:

  • MMProvence keeps content next to the answer instead of the answer itself: a neighbouring sentence, section or table row on the correct page. For "Is the number of tentative ANDA approvals in FY2013 four higher than in FY2014?", MMProvence-4B keeps the table row "ANDA approvals" (440 vs. 409) instead of the row "Tentative Approval" two lines below (95 vs. 91), and the generator answers "No, the difference is 31". For "Number of Hermès shareholders on 31 December 2023", MMProvence-2B keeps the paragraph on the shares Hermès holds itself instead of the section just above it ("about 185,000 shareholders"), and the generator answers that the number is not given.
  • MMProvence keeps only part of an answer that has several parts: for example, one side of a comparison, or only the first items of a list.
  • MMProvence keeps a value but not its label: for "Must flightline extinguishers be placed along the edge of the parking ramp for aircraft inside FTRs?", the crop keeps the table column that lists the requirements but cuts off the column that says which situation each row covers. The generator then reads the wrong row and answers "No".

In all three cases, the box and the selected sentences come from the same model output, so the image and the text are cut down to the same part of the page. When MMProvence points at the wrong part of the page, the answer therefore disappears from both the image and the text at once.

LLM-as-a-judge procedure (Q7): We will add the full procedure to the appendix:

  • Model and decoding: Qwen3.5-9B with thinking enabled, greedy decoding (temperature 0) and up to 2,048 new tokens, served with vLLM.
  • Input: a single user turn with the grading instruction (Appendix, Figure 8), the question, up to five human-verified gold answers from ViDoRe V3 joined by semicolons, and the prediction, of which only the text after the generator's reasoning trace is kept. Predictions and gold answers are truncated to 500 words and 1,000 tokens.
  • No access to the pages: the judge grades correctness against the gold answers, not agreement with the retrieved context, so pruning can only affect the score through the answer itself.
  • Output: a binary verdict, read after the judge's reasoning trace. LLMeval is the mean over the 4,000 queries, which are the same for every system, so every comparison is paired.
  • Reliability: 1.2–1.8% of verdicts flip between identical re-runs (row standard deviation ≤ 0.002), the verdicts agree with Gemma 4 12B on 93.5% of queries (κ = 0.79), and Gemma's fp16 and fp32 verdicts agree on 99.8%.
Reviewer_cRXeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MMProvence, a multimodal context pruner for RAG over visually rich documents. A Qwen3.5-based VLM reranker (2B/4B/9B) produces, in one call per query–page pair, a relevance score, a bounding box, and the verbatim sentences that answer the query. The score reranks the 50 candidates; for the top 5, the box crops the image and the sentences replace the page text, so pruning adds no separate model call. Training distills Qwen3-VL-Reranker-8B and re-annotates Visual-CoT with contextual boxes that include headers and legends. On ViDoRe V3’s eight domains, MMProvence-2B cuts input from 21.6k to 5.3k tokens (−75%) at LLMeval 0.803 vs. 0.827, while ranking on par with the 2B reranker it replaces (nDCG@10 0.601 vs. 0.604). The paper also releases VERGE, an evaluation framework. It is a useful engineering step, but the evaluation does not yet support the strength of the efficiency and “zero-cost” claims.

Critical strengths

S1. The paper’s zero-marginal-call design is genuinely practical: having the reranker emit the score, box, and evidence sentences in one pass turns pruning into a by-product of a stage multimodal RAG already runs, which is directly useful for real deployments.

S2. The contextual-box finding is a transferable insight—showing that crops including headers, legends, and labels preserve answer quality far better than tight answer-only boxes—and the eight-domain ViDoRe V3 results support it with a 75% token reduction at only a 2.4-point LLMeval drop.

Critical weaknesses

W1. The “zero-cost” claim is misleading because pruning is paid for by a much more expensive reranker—roughly 9.6k extra output tokens per query—and the paper omits the obvious cost-matched baselines (fewer pages, downscaled images, OCR-coordinate crops), so it never shows that the learned box is the most efficient way to reach 5.3k tokens.

W2. The headline box ablation is not controlled—it compares different 4B checkpoints with different no-crop baselines and contextual boxes that are 2.37× larger—so the 13.3- vs. 2.7-point gap cannot be attributed to “context” rather than simply retaining more pixels, and no IoU or recall is reported despite the authors having box annotations available.

Questions for authors

Q1. Could you report LLMeval and token counts for top-1/top-2 full pages, downscaled images, and an OCR-coordinate crop at roughly the same 5.3k-token budget? As written, the only comparison is against the 21.6k full-context pipeline, so it is hard to tell whether the learned box is necessary or just one of several ways to spend the same tokens.

Q2. What does “zero-cost” mean given that reranking grows by 0.75–5.47 s, the 4B pipeline uses more compute, and the 9B is slower? Could you also report end-to-end latency with a 2B or 4B generator, where reranking would dominate?

Q3. Why generate evidence for all 50 candidates when RQ4 shows that pruning only the top-5 pages scores 0.802 vs. 0.803? Could you report the answer quality of the sentence-index variant and compare a cascade that scores 50 candidates but generates evidence only for the top 5?

Q4. Could you add confidence intervals or seed variance, per-domain and per-language LLMeval, and a second benchmark? The judge is the same Qwen3.5-9B used as the generator and never sees the page images, so how do you rule out self-preference and credit for answers that are textually correct but unsupported by the pruned context?

Q5. Could you provide a controlled box ablation at matched retained area, plus IoU/recall on SynthDoc and Contextualized Visual-CoT? The current comparison uses different 4B checkpoints with different no-crop baselines and contextual boxes 2.37× larger, so the 13.3- vs. 2.7-point gap may reflect box size rather than context.

Q6. Were the contextual boxes annotated with the gold answer visible? If so, how can the student infer the same region at inference, and did you check this with a blind annotation or human spot-check?

Q7. RQ3 says MMProvence never deletes the whole page image, yet Section 3 allows an empty bbox to remove it—what is the actual empty-box rate? Could you also report parse-failure, invalid-box, sentence-match, and truncation rates?

Q8. Could you compare against Provence on OCR text plus an OCR-coordinate crop, a visual-token-reduction baseline, and a size-matched reranker? Are the 4B/9B gains over the 2B reranker due to the method or simply model size?

4 comments
AuthorsOpenReview ↗

Response 1

We thank the reviewer for the detailed feedback, and for recognizing the practicality of the single-call design and the contextual-box finding. Below, we reply to the weaknesses and questions:

Token-budget-matched baselines (W1, Q1): We added the three requested baselines at roughly MMProvence-2B's budget. The generator (Qwen3.5-9B, thinking on) and the judge are the same as in Table 1. We will add all three to the paper.

Top-1 / top-2 full pages (4,000 queries; all rows generated on the same hardware for this comparison):

Context given to the generatorMedian input tokensLLMevalΔ vs. MMProvence-2B [95% CI]
Top-5 full pages (no pruning)21.8k0.827+0.032 [+0.021, +0.043]
MMProvence-2B (crop + text, top-5)5.4k0.796—
Top-1 full page4.4k0.670−0.125 [−0.140, −0.111]
Top-2 full pages8.7k0.769−0.026 [−0.039, −0.014]

At a similar budget, the top page alone scores 0.125 lower. Two whole pages use 1.6× MMProvence-2B's tokens and still score lower, and never higher on any of the 8 domains.

downscaled images baseline: We fixed top-5 documents from the existing MMProvence-2B ranking without reruning the retrievel or reranking to avoid introducing any confounders. Qwen3.5-9B generates and judges. The generator's input is the question, the task instruction, and the page image only — no OCR or compressed text. In the results table below, 2× means each side ×0.5; 4× means each side ×0.25.

MethodVisual tokensRetentionLLMevalΔ vs full
Full image17837100%0.836—
MMProvence image-only431024.2%0.783−0.054
Downsample 2×446525.0%0.824−0.013
Downsample 4×11096.2%0.660−0.176
Downsample 2× + MMProvence crop11916.8%0.745−0.091

Initially, we observed that "Downsample 2×" outperformed MMProvence. We hypothesized that the original image is already at very high resolution such that downsampling leads to minimal information losses. Since MMProvence is orthogonal to the downsampling, we used MMProvence to prune the already 2x downsampled images and compared the results to naiive downsampling by 4x. Our hypothesis was true. Once the image was downscaled enough to make the text unreadable, MMProvence provided a 13% relative gain in performance with the same token budget. The performance difference furhter supports our hypothesis where the gap is largest when the answer is a figure in the text: finance FR 0.64 vs 0.33, finance EN 0.69 vs 0.50, HR 0.67 vs 0.53.

OCR-coordinate crop: Provence selects query-relevant sentences (threshold 0.1), and the page is cropped around those sentences' words in the PDF text layer (1% padding).

Pruning (MMProvence-2B ranking)TokensText removedImage area removedLLMeval
MMProvence crop + text5.3k63.4%74.6%0.803
Provence + text-coordinate crop4.9k68.1%73.9%0.755

"Zero-cost" (W1, Q2): We agree that the term needs clarification. It means that the pruning stage needs no additional model call, since the box and the sentences come from the reranking call. The cost moves into reranking as output tokens, which is why MMProvence-4B uses more compute and MMProvence-9B is slower end to end. Most of these output tokens are copied sentences. That's why we trained the sentence-index variant of MMProvence-2B (Table 2, last row) which outputs sentence numbers instead, cutting the reranking output from 9,578 to 2,700 tokens per query. On H100 (median seconds per query, batch size 1, 392 queries):

RerankerRerank (s)Total, thinking (s)Total, no thinking (s)TFLOPs
Qwen3-VL-Reranker-2B (full context)6.6121.8010.941,180
MMProvence-2B7.3719.37 (−11%)9.48 (−13%)818 (−31%)
MMProvence-2B, sentence indices5.2216.86 (−23%)7.37 (−33%)789 (−33%)

2B / 4B generators, where reranking takes a larger share (full context vs. MMProvence-2B with sentence indices; same protocol, timed on V100 as we had no H100 access during the rebuttal):

GeneratorThinkingFull contextMMProvence-2B (sent. indices)Δ latencyTFLOPs
Qwen3.5-2Boff38.25 s33.85 s−11%830 → 702 (−15%)
Qwen3.5-2Bon55.54 s46.71 s−16%849 → 718 (−15%)
Qwen3.5-4Boff53.38 s36.38 s−32%948 → 723 (−24%)
Qwen3.5-4Bon84.57 s61.66 s−27%971 → 742 (−24%)

MMProvence is faster in every setting (all 95% paired-bootstrap CIs exclude zero), including the 2B generator without thinking, where reranking is 57% of the full-context time. We will replace "zero-cost" with "pruning without an additional model call" and state in RQ5 that the end-to-end gains depend on model size.

AuthorsOpenReview ↗

Response 2

Evidence for all 50 candidates (Q3): We timed MMProvence-4B reranking 50 candidates per query (H100, 8 queries) in two modes: full evidence for all 50 candidates, as in the paper, and a cascade that decodes only scores for all 50, then full evidence for the top 8. They take 2m19s and 2m01s, so evidence for all 50 adds 15% to reranking time. Prefill dominates: each candidate's prompt is ~1,750 tokens, mostly the page image, while its evidence is ~200 tokens, and the cascade pays a second prefill for each page it keeps. With sentence indices, the evidence shrinks to ~54 tokens per candidate. Since both arrangements give the same answer quality (0.802 vs. 0.803, RQ4), we keep the single call. The sentence-index variant does not lose answer quality: trained with both phases, it scores 0.796 LLMeval vs. 0.803 for MMProvence-2B (−0.007 ± 0.010, not significant), with the same nDCG@10 (0.601).

Confidence intervals, per-domain results, second benchmark and judge (Q4):

Confidence intervals and variance. We will add 95% CIs to every LLMeval table. Over the 4,000 queries, domain-stratified bootstrap CIs are ±0.011–0.013 on absolute LLMeval and ±0.009–0.011 on paired deltas. Key rows against the full-context baseline (0.827):

RerankerPruningLLMevalΔ [95% CI]
Qwen3-VL-Reranker-8Bnone0.848+0.021 [+0.011, +0.031]
MMProvence-2Bcrop only0.822−0.005 [−0.015, +0.005]
MMProvence-2Bcrop + text0.803−0.024 [−0.035, −0.014]
MMProvence-4Bcrop only0.840+0.013 [+0.003, +0.022]
MMProvence-4Bcrop + text0.811−0.017 [−0.027, −0.006]
MMProvence-9Bcrop only0.833+0.006 [−0.004, +0.015]
MMProvence-9Bcrop + text0.822−0.006 [−0.016, +0.005]

Generation and judging are greedy, so there is no sampling seed. Regenerating on different hardware moved the baseline by +0.000 and MMProvence-2B by −0.008 (neither significant), and three judge re-runs give a per-row standard deviation ≤ 0.002. One claim does not survive: MMProvence-4B crop only's +0.013 falls to +0.001 across the re-runs, so we will revise it to "matches the full-context baseline with 64% fewer generator tokens".

Per domain (we will add every row to the appendix):

RowComp. Sci.EnergyFin. ENFin. FRHRIndustrialPharmaPhysicsMean
Qwen3-VL-Reranker-2B, no pruning0.9260.8480.7920.6640.8420.7740.8620.9100.827
MMProvence-2B, crop + text0.9220.8180.7460.6600.7920.7300.8600.8960.803
MMProvence-9B, crop + text0.9200.8340.7880.6720.8320.7700.8740.8820.822

Per query language (648–689 queries per language; Δ vs. the baseline, * = 95% CI excludes zero):

RowEnglishFrenchGermanItalianPortugueseSpanish
Baseline LLMeval0.8150.8330.8190.8310.8510.816
MMProvence-2B, crop + text−0.009−0.023−0.020−0.050*−0.015−0.029
MMProvence-4B, crop + text+0.001−0.012−0.025−0.039*−0.027*+0.003
MMProvence-9B, crop + text+0.009−0.006−0.012−0.008−0.019+0.002

No row of Table 1 differs significantly across languages (heterogeneity test, p > 0.07 for every row). The largest drops are on Italian queries, where all eight prompted pruners also lose significantly, so this is not specific to MMProvence.

Second benchmark (VisRAG test sets, page images only):

DatasetRerankerPruningnDCG@10TokensLLMeval
MP-DocVQAQwen3-VL-Reranker-2Bnone0.93718.6k0.968
MP-DocVQAMMProvence-2Bimage crop0.9423.8k (−79%)0.939
InfoVQAQwen3-VL-Reranker-2Bnone0.97015.9k0.930
InfoVQAMMProvence-2Bimage crop0.9724.0k (−75%)0.851

The larger drop on InfoVQA reflects the few infographics in our training data. We will add both datasets to the paper.

Self-preference and unsupported answers. All systems use the same generator, so a self-preferring judge would favour them alike. We re-judged all 20 rows of Table 1 with Gemma 4 12B, from a different family: the row ordering is preserved (Kendall τ = 0.84), 93.5% of verdicts agree, and the deltas match (8B reranker +0.020 vs. +0.021; MMProvence crop + text −0.028 / −0.021 / −0.004 at 2B / 4B / 9B vs. −0.024 / −0.017 / −0.006). The judge grades against human-verified gold answers, so an unsupported answer is credited only if the generator knows it without the page, and the generator is instructed to say it does not know otherwise. LLMeval drops sharply when evidence is removed (−0.157 with only the top page, −0.133 with tight crops), which would not happen if answers came without evidence.

AuthorsOpenReview ↗

Response 3

Controlled box ablation (W2, Q5): We padded the tight-box model's boxes until its generator input matched MMProvence-2B's (median 5,250 vs. 5,364 tokens; 4,000 queries; crop + text in every row):

BoxesInput token reductionLLMeval
Tight (RQ4)−91%0.686
Tight + padding−75%0.748
Contextual (MMProvence-2B)−75%0.796

At equal tokens, padding stays 0.048 below contextual boxes (95% CI ±0.013) and is lower on all 8 domains, although the tight-box model is the larger one (4B vs. 2B). The gap therefore comes from where the box is placed, not from its size: padding grows the box evenly around the answer, while a contextual box extends toward the headers, axis labels and legends needed to read it. We will add this comparison to RQ4.

Box IoU and recall. ViDoRe V3 has no gold boxes, so we measure box accuracy on 3,095 held-out document pages of Contextualized Visual-CoT, with recipe (i) (Appendix C) trained once on tight and once on contextual boxes. Boxes are matched one-to-one at IoU ≥ 0.5; mean IoU counts a missed gold box as zero; each model is scored against the boxes it was trained on.

Boxes (recipe (i), stage 2)Box F1 (IoU ≥ 0.5)Mean IoU
Tight (original Visual-CoT)0.4740.374
Contextual0.6950.635

Part of this gap reflects the larger size of contextual boxes, but it shows that the contextual region can be predicted from the question and the page alone.

Annotation with the gold answer (Q6): Yes. Qwen3.8-27B saw the image, the question and the gold answer (Section 4). As in any grounding dataset, the answer defines the region, while the student learns to predict it from the question and the page. The contextual box always contains the original Visual-CoT box, and the context it adds (headers, axis labels, legends) is determined by the page layout, which the student sees. We checked this by hand on 40 random rows from all 12 subsets. Blind to the answer, the annotator first drew the region needed to answer; then, with the answer shown, judged two unlabeled crops in random order, one from the contextual box and one from the original box:

40 rowsContextual boxOriginal Visual-CoT box
Crop alone is enough to answer: yes / partly / no40 / 0 / 019 / 10 / 11
Blind region at least half inside the box37 / 4015 / 40
Share of the blind region inside the box (median)95%33%

The contextual crop was always enough to answer, and was rated higher than the original crop on 21 rows and never lower (sign test, p < 0.001). Without the answer, the annotator chose a region lying mostly inside the contextual box on 37 of 40 rows, and the 27B's own box is the closest to these blind regions (mean IoU 0.56). The region is therefore recoverable without the answer.

Failure-mode rates (Q7): Over the top-5 pages of all 4,000 queries (20,000 pairs per model); truncation over all 200,000 reranking replies:

MMProvence-2BMMProvence-4BMMProvence-9BMMProvence-2B, sentence index
Parse failure1.52%0.39%0.35%0.00%
Truncation (token cap)1.22%0.33%0.40%0.00%
Empty box0.01%0.01%0.04%0.00%
Invalid box0.00%0.00%0.00%0.00%
Copied sentences match nothing → full text kept20.8%12.9%17.5%0.00%

Empty boxes occur on at most 8 of 20,000 pages, so RQ3 should say "almost never"; we will correct it. Every failure falls back to the original content, so no failure deletes evidence.

More baselines and model size (Q8): Provence with an OCR-coordinate crop and resolution reduction are reported under Q1. On model size: the ranking gains of MMProvence-4B and -9B over Qwen3-VL-Reranker-2B come from model size, and we will state this explicitly. At matched size, MMProvence ranks on par with the cross-encoders: MMProvence-2B reaches 0.601 nDCG@10 vs. 0.604 for Qwen3-VL-Reranker-2B (LLMeval without pruning 0.833 vs. 0.827, not significant), and MMProvence-9B comes within 0.006 of its teacher, Qwen3-VL-Reranker-8B (0.636 vs. 0.642). The method's contribution is the pruning evidence produced in the same call, without a loss in ranking.

Dear Reviewer,

Thanks for taking the time to review our work and providing valuable suggestions and comments. In the rebuttal, we have added all the requested baselines and simple approaches. Our MMProvence still outperforms them while being efficient. The downsampled input image experiment was very insightful, since it illustrated that MMProvence can be stacked on top of the downsampled images to provide extra compression and ultimately beating the simple downsampling approach. We have also added ablations with smaller generators in the latency experiment. Additionally we added Provence baseline with OCR coordinates and our model still outperforms them. We added confidence intervals for all of our results to show their significance. Moreover, we added DocVQA and InfoVQA benchmarks. We also provided extra analysis for the contextualize bounding boxes data as well as the failure rates of our model. Since there is less than 24 hours remaining till the end of the discussion period, we would appreciate it if you can take another look at our responses and let us know if there are any remaining concerns.

Comments on the paper

2 comments
Area_Chair_ZLUoOpenReview ↗

Post-Rebuttal Discussion

The authors have provided detailed responses and substantial additional experiments. Please review the rebuttal and discuss whether the main concerns have been adequately addressed.

Dear Area Chair and Reviewers,

We thank the reviewers for their feedback. We revised the paper and marked all new or changed content in red. We clarified that "zero-cost" means that pruning needs no additional model call, wrote the training objective as an equation, and added 95% confidence intervals to the main results. We also ran the experiments the reviewers asked for: baselines at a matched token budget (fewer full pages, lower resolution, Provence with a text-layer crop, and padded tight boxes); a comparison with RegionRAG; two more benchmarks (MP-DocVQA and InfographicVQA); a second judge from another model family; judge re-runs with per-domain and per-language results; pruners run behind the teacher reranker; failure-mode rates and analysis; a manual check of the contextual boxes; and latency with smaller generators.

We believe these changes address all the weaknesses the reviewers raised, and the new results confirm our main conclusions. At a similar token budget, MMProvence keeps more answer quality than fewer full pages, padded boxes and Provence, and it also outperforms RegionRAG's crops. With sentence indices, the whole pipeline is faster than the full-context one, including with small generators. Our conclusions hold under a second judge. We corrected the one claim that did not hold up when the answers were re-judged: MMProvence-4B with crop only matches the full-context baseline rather than beating it. The only reviewer who engaged in the discussion raised their score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.