acceptodds
Under review as a conference paper at ICLR 2027

RoRA: Role-Oriented Regional Allocation for Visual Token Pruning in MLLMs

Abstract

Multimodal large language models (MLLMs) encode images as long visual token sequences, making prefilling and KV-cache storage expensive. Existing training-free pruning methods select tokens by importance, diversity, or spatial coverage, but treat retained tokens as interchangeable and do not explicitly track which object-related regions are already covered. We present RoRA, a training-free framework that casts visual token pruning as role-oriented regional evidence allocation. Given a fixed budget, RoRA partitions tokens into a protected semantic core, complementary context, and fine-grained detail. It first calibrates text-conditioned attention with a positional prior and a prompt-calibrated object prior, then builds Attention-Anchored Regions (AARs) from high-confidence anchors as lightweight proxies for covered object support. Context is explored mainly outside AARs, while a small AAR-guided budget restores local detail; pairwise similarity is used only for context-stage redundancy filtering. Under matched budgets, RoRA consistently outperforms strong training-free baselines across LLaVA and Qwen-VL families, retaining most of the unpruned accuracy even at aggressive pruning ratios, e.g., 96.5% of full performance at 88.9% pruning on LLaVA-1.5, and improving over D²Pruner by about 5% on Qwen3-VL at 75–90% pruning. At a 66.7% pruning ratio, RoRA requires only 0.7 ms for token selection and reduces end-to-end inference time by 24.6%, corresponding to a 1.33× speedup over unpruned inference on an NVIDIA H800.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Visual Token Pruning, Training-Free Compression, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_M5N3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes RoRA, a training-free visual token pruning framework for multimodal large language models. RoRA divides retained tokens into three complementary roles: semantic core, context, and fine-grained detail. It combines attention debiasing with an object-oriented prior, constructs Attention-Anchored Regions (AARs) to guide regional allocation, and applies similarity-based redundancy filtering. Experiments across LLaVA and Qwen-VL models demonstrate competitive accuracy and low selection overhead.

Critical strengths

S1. Intuitive and practical framework. The role-oriented formulation provides a clear perspective on visual token selection. The training-free design requires no additional model training or object detector.

S2. Broad experimental coverage. The evaluation spans four MLLM backbones and multiple pruning ratios. Notably, RoRA improves over D2 Pruner by 5.4 and 5.3 percentage points in normalized average performance on Qwen3-VL at 75% and 90% pruning, respectively.RoRA reports approximately 0.7 ms of selection overhead and a 24.6% reduction in inference time on LLaVA-1.5 under the reported prefill-dominated setting.

Critical weaknesses

W1. Limited methodological novelty. The main components largely combine existing techniques, including attention debiasing, heuristic importance scoring, and similarity-based filtering. The proposed semantic roles are implemented through predefined budgets and scoring rules, while AARs are constructed from local neighborhoods around attention anchors rather than explicit semantic regions.

W2. Weak evidence for the independent contributions of the proposed components. The ablations indicate that most improvements originate from core protection. Adding AAR-guided context improves normalized RefCOCO performance by only 0.59 and 0.11 percentage points at the two evaluated budgets, while standard detail repair improves the TextVQA/VizWiz average by only 0.19 points.

W3. Insufficient validation of spatial assumptions and fine-grained preservation. For dynamic-resolution models, AARs are constructed using an inferred two-dimensional grid from flattened tokens, which may not faithfully preserve the original spatial topology. Furthermore, dedicated OCR-intensive benchmarks such as OCRBench, ChartQA, and DocVQA are absent despite the emphasis on preserving fine-grained textual evidence.

Questions for authors

Q1. What distinguishes AAR-guided allocation from existing attention-based spatial coverage heuristics beyond the predefined role budgets and regional score adjustments?

Q2. How does the inferred grid preserve true spatial neighborhoods for dynamic-resolution inputs? Can the authors evaluate OCRBench, ChartQA, or DocVQA to verify the preservation of textual and numerical details?

Q3. Given the small incremental gains from AAR and detail repair, can the authors provide stronger component-wise evidence, particularly on Qwen-VL models under aggressive pruning? Can the object prior be evaluated without benchmark-specific hyperparameter selection?

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/3)

We sincerely thank the reviewer for the thorough evaluation and constructive feedback. We deeply appreciate your valuable suggestions, which have helped us significantly improve the completeness of our work.

Response to Weakness 1 / Question 1: Methodological Novelty and AAR-Guided Allocation

RoRA’s main contribution is to coordinate the remaining visual-token budget based on the core evidence already retained. Although role budgets are predefined, AAR locations and coverage depend on attention anchors conditioned on the current image and text, rather than a fixed spatial partition.

AAR provides a lightweight approximation of existing evidence coverage: context selection favors complementary information outside these regions, while detail selection favors fine-grained evidence inside them. Similarity filtering further reduces redundancy. The same coverage representation therefore induces different selection preferences for different roles, making subsequent allocation depend on what evidence has already been preserved. This distinguishes the framework from uniform importance ranking or spatial coverage.

In some general-understanding settings, strong baselines already retain performance close to the unpruned reference. Further reducing pruning loss requires preserving localized, task-relevant evidence. Our qualitative examples illustrate the value of retaining information such as jersey numbers and document dates under the same token budget. Improving evidence preservation for individual questions remains valuable even when the aggregate score improvement is modest.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2/3)

Response to Weakness 2 / Question 3: Component Contributions

As discussed in our response to Weakness 1, existing methods already retain performance close to the unpruned reference on some benchmarks and budget settings, leaving relatively little pruning-induced loss to recover. Further improvements require more precise preservation of task-relevant visual evidence. Such evidence may involve only a few tokens, yet affect the final answer to an individual question.

Therefore, although the observed positive increments in our ablations are modest, they represent further improvements over an already strong baseline. RoRA uses AAR-guided context selection and detail recovery to preserve potentially overlooked evidence and further reduce information loss from pruning.

We conducted a controlled comparison on Qwen3-VL at 75% pruning, fixing the inputs, token budget, and shared configuration across tasks. We compared full RoRA against a variant with the object prior, AAR bonuses, and detail components jointly disabled:

TaskFull RoRAJointly DisabledDifference (pp)
BLINK counting60.8360.00+0.83
IllusionVQA50.5750.570.00
VisRes orientation27.5128.11−0.60
CV-Bench82.8782.56+0.30
POPE85.6085.52+0.08

Scores are percentages, and differences are calculated from unrounded values.

These ablations show modest positive gains across most benchmarks, while also demonstrating that the contributions of individual components vary across tasks. We position these results as supplementary evidence that complements our existing core findings.

AuthorsOpenReview ↗

Official Comment by Authors (Part 3/3)

Response to Weakness 3 / Question 2: Spatial Assumptions and OCR Evaluation

We clarify that AARs are constructed dynamically from input-dependent attention anchors. They provide approximate regional guidance for token selection, rather than strict object boundaries or hard retention constraints. The anchors change with the input; however, the grid inferred from the flattened token sequence remains a spatial approximation and does not guarantee exact correspondence to the true neighborhoods of dynamic-resolution inputs.

In some qualitative examples, we observed that D²Pruner retained tokens from cluttered backgrounds, while RoRA preserved textual evidence needed to answer the question. Changes to a small number of critical tokens can affect the final answer, motivating our emphasis on fine-grained evidence preservation.

To further evaluate OCR performance, we conducted additional experiments on the complete OCRBench and ChartQA test sets and the DocVQA validation set using Qwen3-VL-8B-Instruct. We used one shared pruning configuration per pruning ratio across all three benchmarks, with identical inputs and evaluation protocols across methods within each benchmark.

MethodPruningOCRBench (out of 1000)ChartQA (%)DocVQA (ANLS × 100)
Unpruned0%87982.8495.42
D²Pruner75%57450.8074.50
RoRA75%62653.8879.02
D²Pruner90%35525.9244.71
RoRA90%35727.6849.81

RoRA achieves higher scores than D²Pruner in these comparisons, providing additional evidence of its effectiveness in text recognition, chart understanding, and document question answering.

Reviewer_PTWbOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RoRA, a training-free visual token pruning method applied at an early LLM layer. It splits the retained budget into three roles. The first is a semantic core, selected by position-debiased text-to-visual attention (following D2Pruner) plus a weak object prior. The second is complementary context, chosen with a bonus for tokens outside attention-anchored regions and a redundancy filter. The third is a few detail tokens. Experiments on LLaVA-1.5-7B, LLaVA-NeXT-7B, Qwen2.5-VL-7B and Qwen3-VL-8B compare against FastV, VisionZip, DART, DivPrune, HoloV, D2Pruner and others, and report RoRA as best in normalized average at every budget (e.g., 96.5% at 64 tokens on LLaVA-1.5), with 0.7 ms selection overhead.

Critical strengths
  1. The evaluation is broad: four backbones including Qwen2.5-VL and Qwen3-VL, many recent baselines, and wall-clock timing rather than FLOPs alone.
  2. Thinking of the budget in terms of roles (core, context, detail) is intuitive, and the RefCOCO, TextVQA and VizWiz ablations probe localization and fine detail, which most pruning papers ignore.
Critical weaknesses
  1. Beyond D2Pruner-style debiased attention with similarity filtering, the role-oriented components add little that can be measured. The configured object-prior and AAR weights are tiny or zero, and the ablations show gains of at most ~0.3 points without variance.
  2. The headline comparison is not on a common footing. RoRA's averages in Table 1 are normalized to a local full-token run, while the copied baselines use the displayed literature Vanilla row. The paper's own App. J shows that RoRA's LLaVA-NeXT average drops from 95.5% to 93.5% under the common reference, which matters given margins of 0.3–0.4 points over D2Pruner.
  3. The released code does not match the paper. The selector builds a full N×N similarity matrix, contrary to the claim of avoiding one, so the 0.7 ms vs. 71.7 ms gap to D2Pruner (Table 3) likely reflects the implementations. The code also adds the AAR bonus to the core score and has no AAR term in the detail score.
  4. The Qwen results look unreliable. Pruned models score far above Vanilla on several benchmarks (e.g., TextCaps 61.3 vs. 53.5), DivPrune is below chance on AI2D, and Qwen3-VL uses several configurations, despite the "one fixed configuration" claim in Sec. 4.1.
Questions for authors
  1. Were D2Pruner, DART and HoloV re-run under your local protocol, and how do all methods rank when normalized to one common full-token reference?
  2. What does RoRA gain over debiased top-K plus cosine filtering alone, with the object prior, AAR and detail tokens disabled, on the full benchmark suites?
  3. Why is D2Pruner 2.6× slower than unpruned inference on LLaVA-1.5 (Table 3) but faster than unpruned inference on LLaVA-NeXT (Table 4)?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/3)

We sincerely thank the reviewer for the thorough evaluation and constructive feedback. We deeply appreciate your valuable suggestions, which have helped us significantly improve the completeness of our work.

Response to Weakness 1 / Question 2: Component Contributions and Efficiency

Disabling the object prior, AAR, and detail components leaves a D²Pruner-style foundation based on debiased attention and similarity filtering. RoRA builds on this foundation to preserve complementary and fine-grained visual evidence.

In some general-understanding settings, strong baselines already retain average performance close to the unpruned reference, leaving relatively little pruning-induced loss to recover. Our completed controlled comparisons show small, task-dependent component effects. Nevertheless, localized evidence—such as a jersey number or document date—can be important for answering individual questions, as illustrated by our qualitative examples. Preserving such evidence remains valuable even when the aggregate improvement is modest.

RoRA also offers a practical efficiency benefit. AAR provides lightweight regional guidance for context selection and detail recovery. Under matched hardware and a unified timing protocol, our LLaVA evaluation shows substantially lower generation and pipeline latency than D²Pruner. RoRA therefore targets both evidence preservation and efficient inference.

Response to Weakness 2 / Question 1: Common Reference and Normalized Averages

Some averages in the original table used a local unpruned reference that differed from the displayed Vanilla reference. We recomputed all complete rows against the same displayed Vanilla reference within each model. The main comparisons are shown below:

MethodLLaVA-1.5 K=192LLaVA-1.5 K=128LLaVA-1.5 K=64LLaVA-NeXT K=320
DART98.9997.8494.9193.06
HoloV97.2796.1193.8492.55
D²Pruner99.2698.5895.9293.11
RoRA99.5898.9196.3093.45

Values are normalized averages (%), recalculated from the displayed task scores and subject to their display precision. K denotes the retained visual-token budget.

RoRA remains the highest-ranked method at every budget under the common reference. This supplementary analysis verifies existing local results and recalculates the averages using a consistent reference.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2/3)

Response to Weakness 3: Implementation and Efficiency

The submitted package contained a standalone selector rather than the complete inference and evaluation code. Some regional-score placements in this file were not fully aligned with the evaluated runtime. In the evaluated implementation, the core is selected using calibrated scores, while AAR terms guide context selection and detail recovery. We will fully open-source the complete implementation, configurations, and reproduction instructions after the conference.

As described in the paper, AAR represents evidence coverage through anchors and their local neighborhoods, enabling regional allocation through lightweight masks and score adjustments. The current LLaVA implementation additionally computes similarity rows as needed, avoiding construction of a complete similarity matrix upfront and reducing graph-construction overhead. Our unified timing evaluation confirms a practical latency advantage over D²Pruner for this implementation.

Response to Weakness 4: Qwen Evaluation and Parameter Settings

An unpruned model provides a comparison reference rather than a theoretical upper bound on task scores. Therefore, higher scores after pruning are not inherently contradictory. Our additional TextCaps comparison under matched input, generation, and scoring settings also produced a slightly higher point estimate for RoRA; this does not imply improvement on every task.

For DivPrune, we corrected a causal-mask indexing issue in its Qwen2.5-VL adapter. We evaluated the corrected implementation on a predefined subset of 128 AI2D questions, using matched image processing, greedy decoding without KV caching, and scoring settings.

MethodPruning RatioAI2D Accuracy (%)
Unpruned—74.7
DivPrune75%70.5
DivPrune90%68.8

The corrected implementation no longer exhibits the below-chance behavior reported in the original table. This diagnostic identifies an adapter issue rather than a failure of the DivPrune method itself.

The description of Qwen3-VL in the appendix is unclear. The appendix should report the settings used during our experiments. In the tables in the main paper, RoRA uses a single shared set of parameters across all benchmarks for each pruning ratio.

AuthorsOpenReview ↗

Official Comment by Authors (Part 3/3)

Response to Question 3: Timing Protocol and D²Pruner Overhead

The original Tables 3 and 4 used different models, hardware, and timing boundaries. Table 3 measured the complete evaluation loop, whereas Table 4 timed standalone forward and complete generate calls separately. The generate call already includes its own prefill.

We additionally conducted a unified timing evaluation using LLaVA-1.5-7B on an RTX PRO 6000 GPU with the same 300 POPE questions. The pruning methods retain 192 visual tokens. All methods use a batch size of 1 and generate 1 new token, with five independent runs per method. Pipeline latency includes input processing, model generation, and text decoding. We report the mean and standard deviation of the per-run mean latencies.

MethodGenerate Latency (ms)Pipeline Latency (ms)
Unpruned59.35 ± 0.3566.67 ± 0.57
D²Pruner80.73 ± 1.8688.38 ± 2.21
RoRA46.99 ± 0.4054.14 ± 0.51

D²Pruner uses a selection implementation based on the authors’ released code. Graph construction and iterative token selection introduce additional overhead, which may offset the computational savings from pruning under our short-output setting. NeXT processes longer visual sequences, providing greater potential savings in subsequent computation; the net speedup can therefore differ across backbones. These results report the measured runtime of the specified implementations under the stated evaluation conditions.

Thank you again for your thoughtful questions and constructive feedback. As the discussion period draws to a close, we would appreciate hearing whether our responses and additional experiments have addressed your concerns. If any questions remain or any points would benefit from further clarification, please let us know so that we can address them before the discussion concludes. Thank you for your time and consideration.

Reviewer_kk7nOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RoRA is a training-free visual-token pruning method that splits the token budget into core, context and detail tokens.

It adds an offline object prior and “Attention-Anchored Regions” (AARs) on top of D²Pruner-style debiasing, and it reports gains on LLaVA and Qwen-VL.

Critical strengths

the core, context, and detail split gives visual token selection more semantic meaning than plain top K or redundancy filtering, and the motivation is easy to follow.

AAR is lightweight and practical, using high confidence anchors as soft coverage proxies and avoiding a full pairwise graph, with very low selector overhead.

The method is training free and works across several backbones, and the core protection ablation as well as the reported latency results make a reasonably convincing accuracy efficiency case.

Critical weaknesses

Latency is measured with separate forward and generate calls, no cache passing, and very few repeated runs. That makes the speed story less convincing than it presents.

The object prior is treated as a main contribution, but the evidence is thin: small subsets, per benchmark tuning, and weak statistics.

AAR and detail repair only add small gains, and the validation is limited to a narrow set of tasks.

A few Qwen averages go above 100%, which needs a much clearer explanation.

There are many hyperparameters and task or label specific settings. It’s hard to tell how much tuning went into each benchmark, and reproducibility suffers.

Robustness to different prompts, languages, domains, and multi image inputs is not well tested.

The 2D grid AAR design is mostly checked on fixed grid inputs. Dynamic resolution, video, and high resolution OCR are not really covered.

The appendix tables are messy, with empty or duplicated tables and cramped columns. That hurts clarity and makes the paper harder to trust.

Questions for authors

The normalized averages seem to use different denominators, and some Qwen averages go above 100%. Could you recompute all methods against one common reference and explain the over 100% cases?

The latency results come from separate forward and generate calls, without cache passing and with limited repeats. What does the end to end comparison look like under one pipeline and multiple runs?

The object prior is a main contribution, but the evidence is small, task tuned, and statistically weak. How does it hold up with one shared setting across benchmarks?

AAR and detail repair add only small gains on limited tasks. How sensitive are they to anchor count and radius, and do they help on harder localization, OCR, or small object cases?

There are many task or label level configurations. Which settings are fixed per backbone and pruning ratio, and were any benchmarks used for tuning? How reproducible are the appendix tables?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1/3)

We sincerely thank the reviewer for the thorough evaluation and constructive feedback. We deeply appreciate your valuable suggestions, which have helped us significantly improve the completeness of our work.

Response to Weakness 1 / Question 2: Timing and Repeatability

In the original table, forward and generate were timed separately, and generate includes prefill. We further conducted a unified timing evaluation using LLaVA-1.5-7B on an RTX PRO 6000 GPU with the same 300 POPE questions. The pruning methods retain 192 visual tokens. All methods use a batch size of 1 and generate 1 new token, with five independent runs per method. Pipeline latency includes input processing, model generation, and text decoding. We report the mean and standard deviation of the per-run mean latencies.

MethodGenerate Latency (ms)Pipeline Latency (ms)
Unpruned59.35 ± 0.3566.67 ± 0.57
D²Pruner80.73 ± 1.8688.38 ± 2.21
RoRA46.99 ± 0.4054.14 ± 0.51

D²Pruner incurs additional overhead from graph construction and iterative token selection, which may offset the computational savings from pruning under our short-output setting.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2/3)

Response to Weakness 2 / Question 3: Independent Contribution of the Object Prior and Shared Settings

Appendix K and Table 13 already provide an isolated object-prior ablation: positional debiasing, the token budget, and other conditions are fixed, while only the object-prior coefficient is varied. To evaluate shared settings, we additionally tested LLaVA-1.5-7B with a fixed budget of K = 128 and a shared object-prior coefficient of 0.03, using 360 questions per task and comparing against the configuration without the object prior. All tasks use core-only selection, and the parameters were not retuned based on these results.

TaskWithout Obj. (%)Shared Obj. (%)Change (pp)
POPE85.8386.39+0.56
TextVQA42.7241.03+0.31
VizWiz29.0629.25+0.19
VQAv272.8972.81−0.08

Response to Weakness 3 / Question 4: AAR Parameters and Detail

Figure 3 already provides stage-wise ablations on localization and OCR tasks. Following your suggestion, we additionally evaluated LLaVA-1.5-7B on a fixed subset of 500 TextVQA questions with K = 192, varying the anchor count, regional radius, or detail setting one at a time while keeping the remaining conditions fixed.

AnchorsRadiusDetail TokensTextVQA Score (%)
242447.18
122446.50
482447.20
241447.08
243447.76
242047.04

The detail stage is designed to supplement fine-grained visual evidence not covered by the core and context selections; enabling it increased the TextVQA score from 47.04% to 47.18% in this controlled comparison.

Response to Weakness 4 / Question 1: Normalized Average

The normalized average summarizes performance across tasks relative to the unpruned model. A value above the reference denotes better average relative performance, not accuracy exceeding its valid range or improvement on every task.

AuthorsOpenReview ↗

Official Comment by Authors (Part 3/3)

Response to Weakness 5 / Question 5: Parameter Settings and Reproducibility

The appendix entry included a development-stage candidate configuration without clearly distinguishing it from the configuration used for the main-table results. All RoRA results across benchmarks in the main table were obtained using one fixed set of pruning parameters.

Response to Weakness 6: Prompt Variation

We evaluated the same 600 POPE questions under different response instructions, keeping the images, question content, and model settings fixed.

PromptUnpruned Accuracy (%)RoRA Accuracy (%)Difference (pp)
Original question87.0087.33+0.33
English yes/no instruction88.5088.17−0.33
English single-word instruction88.5088.00−0.50
Chinese response instruction88.1788.33+0.17

RoRA maintains aggregate accuracy close to the unpruned model across these response instructions.

Response to Weakness 7: Dynamic Resolution, Multi-Image Inputs, and Video

We additionally evaluated Qwen3-VL-8B at 75% pruning using one fixed RoRA configuration and the existing AAR implementation. DocVQA uses the same 200 questions across different pixel caps; MUIRBench uses 240 questions with all original images; video evaluation covers 200 questions from the MVBench action-antonym subtask.

EvaluationInput SettingUnprunedRoRA
DocVQAMaximum 262,144 pixels87.4356.26
DocVQAMaximum 786,432 pixels95.7578.81
DocVQAMaximum 1,572,864 pixels96.5684.06
MUIRBenchAll original images52.5047.50
MVBench action-antonymUp to 8 frames82.0082.50
MVBench action-antonymUp to 16 frames81.5081.50

DocVQA reports ANLS × 100; the other evaluations report accuracy (%). These experiments extend the tested input range to variable-resolution documents, multiple images, and video.

Response to Weakness 8: Appendix Tables

After checking the submitted PDF page by page, we found no entirely empty tables or exact duplicates.

Thank you again for your thoughtful questions and constructive feedback. As the discussion period draws to a close, we would appreciate hearing whether our responses and additional experiments have addressed your concerns. If any questions remain or any points would benefit from further clarification, please let us know so that we can address them before the discussion concludes. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.