acceptodds
Under review as a conference paper at ICLR 2027

Global Relevance Is Not Enough: Representative Candidate Construction and Residual-Based Evidence Preservation for Efficient, Reliable Medical MLLM Inference

Abstract

Medical Multimodal Large Language Models (MLLMs) have shown great potential in clinical visual question answering and medical image interpretation, but their high inference cost limits efficient deployment in real-world clinical settings. Moreover, in safety-critical medical scenarios, pursuing efficiency alone is far from sufficient. We observe that existing acceleration methods, although able to preserve aggregate performance, can still induce substantial instance-level prediction shifts. To address this issue, we propose MedR2R (Representative-to-Residual Compression), a representative candidate construction and residual-based evidence preservation framework for efficient and reliable medical MLLM inference. Rather than determining token importance from global relevance alone, MedR2R adopts a two-stage coarse-to-fine selection strategy. First, after the vision encoder, it constructs a semantically representative candidate set under a relaxed intermediate token budget. Specifically, MedR2R models pairwise visual-token relationships using cosine similarity and combines attention-derived global relevance with marginal representation gain, encouraging the candidate set to represent the overall visual space rather than only highly attended regions. Second, after the visual tokens are contextualized by an intermediate language-model layer, MedR2R further refines the candidate set according to normalized residuals. Tokens that can be accurately approximated from a compact basis are treated as removable, whereas tokens with large normalized residuals are preserved as hard-to-replace visual evidence. Experiments across multiple medical VQA datasets and MLLMs demonstrate that, even under substantially reduced visual-token budgets, this coarse-to-fine two-stage selection strategy maintains competitive task performance while effectively mitigating harmful instance-level prediction shifts. These results further indicate that medical MLLM acceleration should not focus solely on aggregate performance, but should also account for the stability of individual predictions. Our code is available at: https://anonymous.4open.science/r/MedR2R.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Medical multimodal large language models, Medical visual question answering, Medical image understanding, Prediction stability, Reliable medical AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CDHxOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MedR2R, a training-free visual-token compression framework for medical MLLMs. It first constructs an encoder-side candidate pool using global relevance and marginal feature-coverage gain. After an intermediate LLM layer, it retains a compact reference basis and candidates with large normalized reconstruction residuals, then consolidates discarded candidates into retained representatives. Experiments cover five medical VQA benchmarks and three backbones. In addition to task quality and measured latency, the paper evaluates instance-level output changes and correct-to-incorrect regressions, with the main stability results reported for LLaVA-Med.

Critical strengths

1、The two-stage design is coherent and supported by substantial analysis. The paper includes component ablations, candidate-budget sensitivity, consolidation ablations, and controlled token-removal experiments, making the roles of the proposed components reasonably clear.

2、The reported stability improvements are promising. At 128 final tokens, MedR2R reduces average DR/NDR from 23.00/2.36 for MMTok to 13.57/1.20 across five benchmarks on LLaVA-Med, while achieving a measured 1.54× speedup over the full-token model. Task-quality evaluations on additional backbones provide useful evidence of broader applicability.

Critical weaknesses

1、The methodological novelty is insufficiently distinguished from closely related work. ResPrune already uses residual-guided token selection, while LRCP combines residual-based selection, token merging, and compression at both encoder and intermediate LLM stages. Although MedR2R differs in candidate construction and basis selection, the substantive advance over these closely related approaches remains unclear.

2、The specific benefit of residual selection remains unclear. In Appendix C.3, low-residual deletion and feature-norm-based deletion yield the same quality-decrease rate of 2.00%. Although the complete pipeline performs well, this comparison does not establish what additional benefit residual selection provides over the simpler alternative.

Questions for authors

1、What is the substantive methodological advance over LRCP and ResPrune?

2、With other steps fixed, what advantage does residual selection offer over attention, feature-norm, or diversity-based selection?

3、How were the layer, basis size, and expansion factor chosen? Was test-set performance used?

4、Why is DR/NDR reported only for LLaVA-Med, despite evaluating MedVLM-R1 and Qwen3-VL elsewhere?

11 comments

We sincerely thank the reviewer for the thoughtful comments and positive evaluation. We greatly appreciate your recognition of MedR2R's two-stage design as "well-motivated" and its stability improvements as "encouraging." We are also grateful for your acknowledgment of the comprehensive component analyses and the importance of evaluating instance-level prediction stability in medical MLLMs.

Your constructive feedback has helped us further clarify the methodological contributions of MedR2R and strengthen the supporting experimental evidence. We have carefully addressed each comment point by point and will incorporate the corresponding clarifications, additional analyses, and experimental results into the revised manuscript. We hope these revisions adequately address your concerns.

AuthorsOpenReview ↗

1. For Weakness 1 and Question 1

Methodological Differences and Additional Comparisons

We thank the reviewer for this important comment. Our substantive advance lies in decoupling representative candidate recall from final evidence retention, rather than introducing another residual metric.

ResPrune [1] greedily expands a text-conditioned reconstruction subspace, while LRCP [2] relies on PCA-based projection residuals for token selection. In contrast, MedR2R separates two complementary objectives:

(1) Representative candidate recall. Under a relaxed budget m=min⁡(n,2k)m=\min(n,2k), we employ the saliency-weighted marginal representation gain (Eq. 3):

gj(C)=sjγ∑i=1n[wij−ci(C)]+.g_j(C)=s_j^\gamma\sum_{i=1}^{n}[w_{ij}-c_i(C)]_+.

This criterion constructs a broadly representative candidate pool by jointly considering global relevance and marginal representation gain, mitigating the premature exclusion of underrepresented visual evidence.

(2) Contextualized evidence retention. After LLM contextualization, we measure normalized reconstruction residuals (Eq. 7):

\qquad \hat{H}=W^*D.$$ We retain the basis $A$ and the $k-r$ non-basis tokens with the largest residuals (Eq. 8), which optimally minimizes the fixed-dictionary reconstruction-error surrogate (Appendix F.2): $$J(R)=\sum_{i\notin R}e_i, \qquad A\subseteq R,\quad |R|=k.$$ The key methodological advance is assigning representative coverage and contextualized reconstructability distinct, complementary roles across representation levels. **(3) Controlled comparison with LRCP and ResPrune.** To further substantiate the methodological differences and practical advantages of MedR2R, we add controlled comparisons with LRCP and ResPrune across three backbone models. Since publicly available official implementations of these two methods were unavailable at the time of our experiments, we independently reimplemented both methods based on the algorithms described in their respective papers. All methods were evaluated using matched backbone models, datasets, preprocessing pipelines, decoding settings, and final visual-token budgets to ensure a fair comparison. **LLaVA-Med v1.5** | Tokens | Method | SLAKE | VQA-RAD | VQA-Med | PathVQA | PMC-VQA | |---:|---|---:|---:|---:|---:|---:| | 128 | LRCP | 38.94 / 14.68 / 4.70 | 51.84 / 27.50 / 10.68 | 43.75 / 14.14 / 4.63 | 57.51 / 5.64 / 1.90 | 24.50 | | 128 | ResPrune | 37.98 / 14.84 / 4.69 | 50.37 / 27.14 / 10.25 | 43.75 / 13.47 / 4.51 | 58.10 / 5.98 / 1.89 | 26.80 | | 128 | **MedR2R (Ours)** | **40.87 / 15.93 / 5.10** | **54.78 / 29.48 / 11.34** | **56.25 / 15.14 / 4.93** | **59.72 / 7.42 / 2.09** | **29.15** | **MedVLM-R1** | Tokens | Method | SLAKE | VQA-RAD | VQA-Med | PathVQA | PMC-VQA | |---:|---|---:|---:|---:|---:|---:| | 128 | LRCP | 49.04 / 14.88 / 14.95 | 56.25 / 18.96 / 19.92 | 51.56 / 18.46 / 18.89 | 53.38 / 3.27 / 3.40 | 43.20 | | 128 | ResPrune | 47.84 / 14.95 / 15.11 | 55.15 / 19.77 / 21.00 | 50.00 / 19.49 / 20.12 | 53.38 / 3.43 / 3.57 | 40.85 | | 128 | **MedR2R (Ours)** | **52.88 / 16.22 / 16.61** | **63.24 / 21.91 / 22.76** | **62.50 / 21.15 / 21.72** | **56.27 / 4.44 / 4.64** | **45.45** | **Qwen3-VL-8B-Instruct** | Tokens | Method | SLAKE | VQA-RAD | VQA-Med | PathVQA | PMC-VQA | |---:|---|---:|---:|---:|---:|---:| | 128 | LRCP | 53.61 / 11.61 / 11.76 | 66.91 / 28.14 / 29.44 | 65.62 / 25.97 / 26.56 | 59.75 / 5.01 / 5.12 | 50.40 | | 128 | ResPrune | 53.85 / 11.89 / 12.05 | 65.07 / 29.13 / 30.90 | 65.62 / 25.73 / 26.15 | 59.36 / 5.05 / 5.24 | 50.95 | | 128 | **MedR2R (Ours)** | **57.45 / 13.36 / 13.27** | **72.06 / 31.62 / 33.28** | **79.69 / 28.29 / 29.01** | **63.96 / 5.53 / 5.73** | **54.95** | For all three backbones, the reported metrics are ordered as Accuracy / Recall / ROUGE-1, while PMC-VQA reports Accuracy only. Overall, the results show that MedR2R consistently outperforms both reimplemented baselines across all five medical VQA benchmarks and all three evaluated backbones at the same 128-token budget. These findings provide empirical evidence that combining saliency-weighted representative candidate construction with contextualized residual-based evidence retention offers practical advantages over existing residual-guided compression approaches. **References** [1] Li X, Zheng Y, Liang Y, et al. ResPrune: Text-Conditioned Subspace Reconstruction for Visual Token Pruning in Large Vision-Language Models[J]. arXiv preprint arXiv:2603.21105, 2026. [2] Lu H, Zhang F, Jin W, et al. LRCP: Low-Rank Compressibility Guided Visual Token Pruning for Efficient LVLMs[J]. arXiv preprint arXiv:2605.15621, 2026.
AuthorsOpenReview ↗

2. For Weakness 2 and Question 2

Controlled Residual Selection Ablation

We thank the reviewer for this important suggestion. We agree that the comparison in Appendix C.3 alone does not fully establish the practical advantage of residual-based selection over simpler alternatives. To address this concern, we conducted an additional controlled residual ablation on the SLAKE test set, isolating the second-stage selection criterion while keeping all other components identical.

Specifically, we fix the same 256-token candidate pool, contextualized features after layer 16, 10 basis tokens, feature-consolidation rule, and final budget of 128. Only the selection criterion for the remaining 118 non-basis tokens is varied. We compare Encoder Attention, contextualized Feature Norm, Random selection, and MedR2R's normalized residual selection.

Controlled Residual Ablation on SLAKE

Stage-2 Selection CriterionROI Token Retention Rate ↑Accuracy ↑Recall ↑ROUGE-1 ↑DR ↓NDR ↓
Encoder Attention43.3140.3815.484.8916.401.51
Random22.1338.9414.464.4521.302.26
Feature Norm41.0139.9015.124.7317.621.70
Residual (MedR2R)46.8440.8715.935.1012.161.13

The results demonstrate that residual-based selection achieves the best performance across all six evaluation metrics. Compared with Feature Norm, residual selection improves ROI token retention from 41.01% to 46.84% (+5.83 percentage points), while reducing DR from 17.62% to 12.16% and NDR from 1.70% to 1.13%.

Because all methods use the same candidate pool and downstream processing, the improvements directly reflect the second-stage selection criterion. Unlike feature norm, which only measures feature magnitude, residual selection identifies tokens that are poorly reconstructed by the shared basis. These results demonstrate its advantage in preserving medical regions and improving prediction stability.

Reviewer_hM8xOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that visual token compression for medical MLLMs should preserve individual predictions, measured by DR and NDR, not only aggregate accuracy. It proposes MedR2R, a training-free two-stage method: (1) after the vision encoder, it greedily selects 2k candidates by attention-weighted cosine coverage; (2) after an intermediate LLM layer, it keeps a farthest-point-sampled basis plus the candidates with the largest ridge-regression residuals, and merges the discarded candidates into their most similar retained tokens. On five medical VQA benchmarks with three backbones, MedR2R is competitive with FastV, SparseVLM, PDrop, VisionZip and MMTok, and on LLaVA-Med reduces the average DR/NDR at 128 tokens to 13.57/1.20 with a 1.54× speedup.

Critical strengths

The paper highlights a relevant issue for medical deployment, that similar accuracy can hide many changed predictions, and reports DR/NDR alongside accuracy. The method is training-free, clearly described, ablated by component, and evaluated on three backbones with measured latency.

Critical weaknesses

(1) The novelty is limited.

The encoder-side stage is coverage-based selection, as in MMTok and DivPrune (cited but not compared). The LLM-side stage keeps tokens poorly represented by a few pivot tokens, which is closely related to DART [1] (neither cited nor compared); MedR2R mainly replaces DART's duplication score with a least-squares reconstruction residual. Merging discarded tokens into retained ones follows existing token merging (e.g., LLaVA-PruMerge, VisionZip's contextual tokens). Nothing in the method is specific to medical images.

(2) The comparison is not matched in computation.

MedR2R keeps 2k tokens through the first 16 LLM layers (14 for MedVLM-R1/Qwen3-VL), while VisionZip and MMTok prune to k before the LLM, so MedR2R is slower (396.2 vs. 332.7/339.8 ms, Table 3). Its stability gain may come largely from this extra computation: the m = 576 variant in Table D.1, which skips the encoder-side stage and only prunes at layer 16, already reaches a DR of 14.63 on VQA-RAD, below all baselines (21.29–29.05), and is even faster than the default m = 256 in prefill time (68.62 vs. 77.75 ms). The encoder-side stage thus adds latency for a further DR/NDR reduction (14.63/1.77 → 11.97/0.44) and an accuracy gain (53.31 → 54.78) of a few questions. Baselines, DART and DivPrune should be compared at matched latency/FLOPs.

(3) The evaluation is too noisy to support the claims.

  • LLaVA-Med is near chance on PMC-VQA (29.15% with four options).
  • Closed sets are small (one VQA-Med question is 1.56 points; the 0.44% NDR on VQA-RAD is about 2 of 451 questions), no confidence intervals are given for the main tables, and VQA-RAD closed accuracy improves with fewer tokens (55.15 / 54.78 / 54.41 at 64 / 128 / 192), with each step being a single question.
  • On PathVQA, the largest benchmark, MedR2R's closed accuracy with LLaVA-Med is below SparseVLM and FastV at all budgets (e.g., 59.72 vs. 61.49 and 60.19 at 128 tokens).
  • DR/NDR is reported only for LLaVA-Med, though the abstract claims it across MLLMs.
  • SparseVLM collapses on MedVLM-R1 at 128 tokens (63.97 → 40.81 on VQA-RAD; 65.62 → 35.94 on VQA-Med), suggesting an implementation issue.

(4) The core mechanism is not supported.

That attention-based selection discards clinically decisive tokens is shown only with selected examples; it could be measured with region annotations such as those of SLAKE. In the controlled removal test (Fig. C.3), MedR2R's rule (low-residual deletion: 62.08% answer changes / 2.00% quality decreases) is not meaningfully better than attention-based (60.75% / 2.22%) or random deletion (63.86% / 2.14%): it changes more answers than attention-based removal, and all differences fall within overlapping confidence intervals.

Reference

[1] Stop Looking for "Important Tokens" in Multimodal Language Models: Duplication Matters More. (Wen et al., EMNLP 2025)

Questions for authors

Please report the baseline hyperparameters (e.g., FastV pruning layer, PDrop schedule) and whether they were tuned per backbone.

In Fig. C.3, every removal policy changes 60–65% of answers relative to the 256-candidate reference, whereas full MedR2R changes only 11.97% relative to the full-token model on VQA-RAD, and even the m = k = 128 setting without residual selection or consolidation changes only 16.63% (Table D.1). Why the large difference? Do the two analyses use the same generation settings?

Please report incorrect-to-correct changes and NDR restricted to closed-ended questions.

14 comments

We are grateful to the reviewer for the insightful comments and constructive feedback. We appreciate your recognition of MedR2R's two-stage design and its potential to improve instance-level prediction stability in medical MLLMs. Your comments have helped us further clarify the methodological contributions of this work, strengthen the supporting experimental evidence, and improve the evaluation of computational efficiency and prediction reliability. Notably, we have conducted an additional large-scale evaluation using Hulu-Med-7B [1] on the full M3D-VQA test set [2], covering 2,000 CT volumes and 27,582 evaluation instances, to further validate prediction stability on a larger benchmark. We have carefully addressed each concern point by point and will incorporate the corresponding additional experiments, analyses, and clarifications into the revised manuscript. We hope these revisions adequately address your concerns.

References

[1] Jiang S, Wang Y, Song S, et al. Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding. arXiv preprint arXiv:2510.08668, 2025.

[2] Bai F, Du Y, Huang T, et al. M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models. arXiv preprint arXiv:2404.00578, 2024.

AuthorsOpenReview ↗

1. For Question 1

Baseline Configurations and Token Budgets

Thank you for asking us to clarify the baseline settings. We will report the exact configurations and token schedules in the appendix. We used fixed settings for each baseline and did not conduct separate hyperparameter searches for LLaVA-Med v1.5, MedVLM-R1, or Qwen3-VL-8B-Instruct. The settings were held fixed across datasets. The backbone-specific choices below reflect the corresponding visual features or implementation rules. Layer numbers are one-based and indicate the layer after which pruning occurs.

BaselineSettings Used
FastVPrune once after decoder layer 3, using that layer's attention to retain kk visual tokens. The same setting is used across backbones.
PDropPrune after layers 8, 16, and 24. From 576 visual tokens, the successive counts are 399 → 277 → 192, 349 → 211 → 128, and 277 → 133 → 64 for the respective final budgets. The stage counts follow nj=round⁡ ⁣[576(k/576)j/3]n_j=\operatorname{round}\!\left[576(k/576)^{j/3}\right] for j=1,2,3j=1,2,3, with n3=kn_3=k enforced exactly. This ties the schedule to the reported final budget and keeps it consistent across backbones.
SparseVLMThe initial quality runs prune after layers 2, 6, and 15, using the same stage counts as PDrop. The follow-up implementation uses layers 3, 7, and 16 and includes DPC-recycled tokens within each stage's budget.
VisionZipAt k=128k=128, LLaVA-Med uses 108 dominant slots, including CLS, and 20 contextual slots. The initial Qwen quality runs use 119 dominant + 9 contextual. The follow-up Qwen stability runs use 118 dominant + 10 contextual. The Qwen ports use incoming visual attention in place of the CLIP CLS attention signal.
MMTokα=0.5\alpha=0.5, text–visual temperature τtv=0.02\tau_{\mathrm{tv}}=0.02, and visual–visual temperature τvv=0.2\tau_{\mathrm{vv}}=0.2. Padding-token removal is disabled, and the selector targets kk tokens before the decoder.
DivPrune [1]Select kk projected visual tokens before the decoder using the fixed diversity selector. The final budget is the only selection parameter.
DART [2]Prune after layer 2 with 4 visual pivots and 4 text pivots. Visual pivots count toward the final budget kk.

To ensure a fair comparison, all baseline methods were evaluated under the same final visual-token budgets as MedR2R, with intermediate pruning schedules adjusted accordingly where applicable.

References

[1] Alvar S R, Singh G, Akbari M, et al. Divprune: Diversity-based visual token pruning for large multimodal models[C]//2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025: 9392-9401.

[2] Wen Z, Gao Y, Wang S, et al. Stop looking for “important tokens” in multimodal language models: Duplication matters more[C]//Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025: 9972-9991.

AuthorsOpenReview ↗

2. For Question 2

Clarification of Reference Models and Generation Settings

The two analyses use different reference models and evaluate different aspects of token compression.

In Appendix C.3, all removal policies operate on the same 256-candidate sequence after layer 16, preserve the 10 basis tokens, and remove exactly 128 non-basis tokens without feature consolidation. The reported 60–65% answer-change rates are measured relative to the intermediate 256-candidate reference, reflecting the local sensitivity of different token-removal criteria.

In contrast, the 11.97% DR in Appendix D.1 measures the complete MedR2R pipeline against the original 576-token Vanilla model. It includes representative candidate construction, residual-based evidence retention, and feature consolidation. These rates are therefore not directly comparable: a prediction change relative to the 256-candidate reference does not necessarily imply a change relative to the Vanilla model.

Both analyses use normalized full-response comparisons. We will clarify the different reference models and evaluation objectives in the revised manuscript.

Reviewer_1GJkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MedR2R (Representative-to-Residual Compression), a training-free, two-stage, coarse-to-fine visual token compression framework, addressing the high inference cost of medical MLLM, existing acceleration methods often induce significant instance-level prediction shifts while maintaining overall metrics. In Stage 1, a candidate set covering the entire visual space is selected by combining attention relevance and marginal representation gain. Stage 2 identifies and preserves hard-to-replace visual evidence using normalized residuals, while the remaining tokens are merged into the retention set. Experiments were conducted on five medical VQA datasets: SLAKE, VQA-RAD, VQA-Med, PathVQA, and PMC-VQA. The task performance, inference latency, and instance-level drift metric DR/NDR of models such as LLaVA-Med, MedVLM-R1, and Qwen3-VL were evaluated.

Critical strengths
  1. The paper points out that comparing average accuracy alone cannot reflect the impact of token compression on the prediction of individual medical samples, and systematically evaluates DR/NDR as a primary indicator on 5 datasets × 3 backbones. This issue has practical significance for medical settings.

  2. The author presented a training-free framework for efficient and stable medical MLLM inference. The descriptions of the methods and components are quite clear.

Critical weaknesses
  1. The most relevant coverage-based method lacks a controlled comparison, and the direct evidence for its residual mechanism is rather weak. Stage 1 also targets monotone submodular coverage (γ), serving as a control/ablation study in the absence of a unified implementation.
  2. The efficiency assessment is not comprehensive enough. The hyperparameter selection protocol is unclear, the sensitivity analysis only covers m/k on VQA-RAD, there are no FLOPs, error bars and phased overhead are not shown.
Questions for authors
  1. The reason for the performance gain of MedR2R on MMTok in Table 4 is not explained. It is recommended to clarify the essential differences in the mathematical and implementation aspects of the first-stage goals of MedR2R and MMTok, or to add a more rigorous algorithm comparison.
  2. The paper was deleted without verifying whether the image region containing the answer was in Stage 1 (question-agnostic). It is recommended to utilize VQA-RAD's ROI annotation to report the survival rate of the answer area tokens through the Stage 1 candidate pool and the final retention rate.
  3. The paper is based on the premise of safety-criticality, but it lacks any medical evidence and may lack explanation of clinical relevance.
  4. It is recommended to supplement the complete protocol for hyperparameter selection.
8 comments

We sincerely thank the reviewer for the insightful comments and positive evaluation. We are greatly encouraged by your recognition that evaluating instance-level reliability using DR/NDR across five datasets and three backbones has "practical significance for medical settings," as well as your appreciation of MedR2R as an "efficient and stable" training-free framework with "quite clear" methodological descriptions.

Your constructive feedback has helped us further clarify our design choices and better articulate the contributions of our paper. We have responded to your comments point by point, and upon confirming these points with you, we will incorporate the corresponding revisions into the manuscript. We hope that these improvements will adequately address your concerns.

AuthorsOpenReview ↗

1. For Weakness 1 and Question 1

First-Stage Comparison and Residual Mechanism Validation

Although MMTok and MedR2R both adopt coverage-based greedy selection, their optimization objectives and candidate-scoring mechanisms differ fundamentally.

Objective difference. MMTok maximizes additive text-vision and vision-vision coverage:

f(S)=f(S;M~tv)+αf(S;M~vv),f(S)=f(S;\widetilde{M}^{tv})+\alpha f(S;\widetilde{M}^{vv}),

where the two similarity matrices are independently calibrated using temperature-based softmax. Its marginal gain jointly reflects the additional textual and visual coverage provided by each candidate. The resulting monotone submodular objective admits a (1−1/e)(1-1/e) approximation guarantee under cardinality constraints. In contrast, MedR2R constructs candidates using nonnegative visual affinity wijw_{ij} and current coverage ci(C)=max⁡j∈Cwijc_i(C)=\max_{j\in C}w_{ij}, with the represented-feature objective F(C)=∑i=1nci(C)F(C)=\sum_{i=1}^{n}c_i(C). Its selection score (Eq. 3),

gj(C)=sjγ∑i=1n[wij−ci(C)]+,g_j(C)=s_j^\gamma\sum_{i=1}^{n}[w_{ij}-c_i(C)]_+,

multiplicatively couples candidate-specific global relevance with marginal visual representation gain. Thus, MMTok jointly covers textual semantics and visual features, whereas MedR2R explicitly balances visual representativeness and global saliency, reducing the tendency to concentrate candidates in highly attended regions.

First-Stage selection comparison. We use LLaVA-Med to compare MMTok, visual coverage (γ=0\gamma=0), and the MedR2R first-stage selector (γ=1\gamma=1) on the full SLAKE test set. All methods directly retain 128 visual tokens (m=k=128m=k=128), with second-stage residual refinement and feature consolidation disabled, while keeping all other inference settings identical.

We report ROI token retention rate, Accuracy, Recall, ROUGE-1, DR, and NDR. ROI token retention is averaged over the 980 questions with valid annotations: for each question, we calculate the fraction of original ROI tokens retained. Each image's ROI is defined as the union of all annotated bounding boxes provided by the SLAKE dataset.

Stage-1 Selection MethodROI Token Retention Rate ↑Accuracy ↑Recall ↑ROUGE-1 ↑DR ↓NDR ↓
Attention-only36.8838.7014.324.4129.502.92
MMTok43.9440.1415.404.8423.751.51
Visual Coverage (γ=0\gamma=0)42.7339.6615.184.7622.811.79
MedR2R First-Stage Selector (γ=1\gamma=1)44.5840.3815.564.9220.831.41

Controlled residual ablation. We compare second-stage selection criteria on the same SLAKE test set, fixing the 256 candidates, contextualized features after layer 16, 10 basis tokens, feature-consolidation rule, and final budget of 128. Encoder Attention uses CLS-to-patch attention from the penultimate vision-encoder layer; Feature Norm uses contextualized feature L2 norms; Random uniformly selects 118 nonbasis tokens without replacement, with results averaged over three fixed seeds.

Stage-2 Selection CriterionROI Token Retention Rate ↑Accuracy ↑Recall ↑ROUGE-1 ↑DR ↓NDR ↓
Encoder Attention43.3140.3815.484.8916.401.51
Random22.1338.9414.464.4521.302.26
Feature Norm41.0139.9015.124.7317.621.70
Residual (MedR2R)46.8440.8715.935.1012.161.13

We thank the reviewer for this valuable suggestion. We have added controlled comparisons for both stages to further validate the contributions of each selection mechanism. The results show that saliency-weighted coverage improves first-stage candidate selection, while residual-based refinement achieves better overall performance under matched conditions. We hope these additional experiments clarify the motivations and practical contributions of the two stages in MedR2R.

AuthorsOpenReview ↗

2. For Question 2

ROI Preservation

We thank the reviewer for this valuable suggestion. To quantitatively examine whether medically relevant visual regions survive question-agnostic candidate construction and subsequent token compression, we conduct a stage-wise ROI retention analysis using the official SLAKE bounding-box annotations. Since the public VQA-RAD dataset does not provide spatial ROI annotations, we use SLAKE as a complementary evaluation benchmark.

The results show that 73.54% of the original ROI token positions are retained in the Stage-1 candidate pool (256 tokens). After residual-based refinement, 46.84% remain in the final 128-token set, achieving the highest ROI retention among the evaluated 128-token compressed methods. This demonstrates that MedR2R preserves a substantial proportion of annotated medical regions throughout both compression stages. The table reports ROI retention under a consistent evaluation protocol:

Method / StageVanillaFastVSparseVLMPDropVisionZipMMTokDivPruneDARTMedR2R CandidatesMedR2R Final
Visual Tokens576128128128128128128128256128
ROI Retention (%) ↑100.0023.4822.5727.5039.8943.9436.7434.7473.5446.84
95% CI[100.00, 100.00][21.33, 25.70][20.51, 24.70][25.06, 30.04][36.73, 43.08][40.05, 47.82][34.38, 39.01][31.42, 38.20][68.97, 77.82][42.69, 51.08]

To address the reviewer's concern regarding error bars, we report 95% confidence intervals computed using 10,000 image-cluster bootstrap resamples. The 256-token candidate pool represents an intermediate-stage result, whereas all compressed-method final results use 128 visual tokens. Vanilla is included as the full-token reference.

This analysis provides a quantitative stage-wise assessment of ROI preservation, complementing the qualitative examples in Figs. E.2–E.3. It shows that MedR2R achieves broad ROI coverage during candidate construction and retains the highest proportion of annotated ROI positions among the evaluated methods at the final 128-token budget.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Comprehensive Rebuttal and Revision Summary

We sincerely thank all three reviewers for their constructive feedback and positive recognition of our work. They highlighted the importance of evaluating instance-level prediction stability in Medical MLLMs and acknowledged MedR2R's novel training-free, two-stage design. The reviewers also recognized the method's excellent generalization across different backbones and the manuscript's clear exposition.

In response to the reviewers' concerns, we conducted additional experiments and made substantial revisions to the manuscript. We detail our responses and corresponding revisions to the key points below.

1. Methodological Distinction & Validation of Core Mechanisms (Responding to Reviewer hM8x W1+W4, CDHx W1+W2+Q1+Q2, and 1GJk W1+Q1)

We further clarify the methodological design of MedR2R, particularly the distinct yet complementary roles of representative candidate set construction and contextualized residual-guided evidence preservation in the two-stage framework. To establish the methodological distinctions, we added comprehensive comparisons with DivPrune, DART, LRCP, and ResPrune across three backbones and five datasets. We also included an additional comparison with MMTok to evaluate the effectiveness of our first-stage selection strategy. Furthermore, extensive ablation studies independently validate the contributions of the two selection stages, demonstrating their complementary effectiveness. These results and clarifications have been incorporated into the Related Work section and Appendices B and C of the revised manuscript.

2. Medical Evidence Preservation & Clinical Relevance (Responding to Reviewer 1GJk W1+Q2+Q3 and hM8x W4)

To quantitatively validate the preservation of medically relevant visual evidence, we introduced a stage-wise ROI retention analysis based on the annotations of the SLAKE dataset. The results demonstrate that MedR2R preserves 73.54% of ROI tokens in the intermediate candidate set and 46.84% in the final compressed set, achieving the highest final ROI retention rate among all compared methods. In addition, we conducted residual ablation experiments and incorporated qualitative visualization examples across multiple medical imaging modalities, demonstrating the preservation of small lesions and clinically critical regions. These additional results further validate the effectiveness of our proposed evidence preservation mechanism. The corresponding results are presented in Table 6 of the main paper and Appendices C and E.

3. Prediction Stability & Large-Scale Evaluation (Responding to Reviewer hM8x W3+Q3 and CDHx Q4)

We extended the DR/NDR evaluation to three backbones across five medical VQA benchmark datasets. We also supplemented the analysis with incorrect-to-correct (I→C) prediction transitions and NDR evaluations on closed-ended questions. More importantly, we conducted a large-scale evaluation using Hulu-Med-7B on the entire M3D-VQA test set. MedR2R achieves 77.83% accuracy, nearly matching the Vanilla model (77.87%), while reducing DR/NDR to 7.03%/1.08%, respectively, and achieving a 1.67× inference speedup. These results demonstrate that MedR2R substantially improves prediction stability while maintaining comparable predictive accuracy and improving inference efficiency. The cross-backbone evaluation results are reported in Table 4, while the large-scale evaluation results are presented in Table 7 and Appendix B.

4. Computational Efficiency & Experimental Rigor (Responding to Reviewer 1GJk W2+Q4, hM8x W2+Q1, and CDHx Q3)

To provide a more comprehensive understanding of the trade-off between inference efficiency and prediction stability, we supplemented our evaluation with FLOPs, time-to-first-token latency, and stage-wise computational overhead analyses. Regarding experimental rigor, we explicitly clarified the configurations of all baseline methods and their respective token pruning strategies. Furthermore, we introduced a hyperparameter selection protocol based on an independent validation set to ensure a fair and rigorous experimental evaluation. These additional analyses and clarifications are incorporated into Section 4.1 and Appendices A and D of the revised manuscript.

Additional concerns have also been addressed in our detailed responses to the reviewers.

Overall, we sincerely thank all the reviewers for their constructive suggestions and the time and effort they dedicated to evaluating our work. Their insightful comments have significantly strengthened our work and improved the quality of the manuscript. We also extend our sincere gratitude to the Area Chair for their diligent efforts in overseeing the review process.

Area_Chair_YkQ5OpenReview ↗

rebuttal ddl

Dear reviewers, the ICLR deadline is only 4 days away. The authors have already rebuttaled the issues raised. If you still have questions, please participate in the rebuttal.

AuthorsOpenReview ↗

Summary of Our Responses and Revisions

Dear Area Chair and Reviewers,

First of all, we would like to sincerely thank the Area Chair for reminding the reviewers of the discussion deadline and helping facilitate the discussion process. We also greatly appreciate the time and effort devoted to handling our submission.

We would like to briefly summarize the progress we have made during the rebuttal and discussion period.

We sincerely appreciate the constructive feedback from all three reviewers. At present, two of the three reviewers have assigned our submission a score of 3 (Weak Accept). Reviewer CDHx also acknowledged that we had addressed the concerns they raised:

"Thank you for the additional experiments and clarifications. I appreciate the authors' efforts in addressing the reviewers' concerns."

Outline of Our Responses to Reviewers

1. Methodological Novelty and Validation

  • Clarified the methodological differences between MedR2R and MMTok, DivPrune, DART, LRCP, and ResPrune.
  • Added controlled comparisons and ablation studies to validate the individual contributions of the two selection stages.

2. Medical Evidence Preservation

  • Added quantitative stage-wise ROI retention analysis based on the annotations of the SLAKE dataset.
  • Supplemented residual-selection ablation studies to validate the preservation of clinically relevant visual evidence.

3. Prediction Stability and Evaluation Reliability

  • Extended DR/NDR evaluations to three backbone models across five datasets.
  • Added I→C prediction transition analysis and NDR evaluation on closed-ended questions.
  • Conducted a large-scale evaluation on the full M3D-VQA test set.

4. Computational Efficiency and Experimental Fairness

  • Clarified the configurations and visual token pruning strategies of all baseline methods.
  • Expanded the description of the hyperparameter selection protocol based on an independent validation set.
  • Supplemented FLOPs, inference latency, stage-wise computational overhead, and comparisons under approximately matched computational budgets.

5. Manuscript Clarity and Presentation

  • Reorganized the experimental results by replacing some tables with figures to improve the readability of the manuscript.

We sincerely appreciate the constructive feedback from all reviewers, which has helped us further strengthen the experimental evidence and overall presentation of our work. We respectfully hope that you will take these additional efforts and the reviewers' feedback during the discussion period into consideration in your final decision.

Thank you again for your time, assistance, and consideration!

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.