acceptodds
Under review as a conference paper at ICLR 2027

BEYOND ATTENTION-GUIDED WARPING: LEARNING QUESTION-GUIDED VISUAL REALLOCATION

Abstract

Multimodal large language models (MLLMs)operate under a limited visual-token budget, which can undersample question-relevant details while allocating sub stantial capacity to background regions. Cropping and active visual search can recover local detail, but may remove contextual evidence or require repeated model evaluations. Moreover, intermediate-layer attention does not reliably pre dict which finite image deformation will improve the decoded answer. We propose QGVR (question-guided visual reallocation), a two-stage proposal-and-selection framework that redistributes visual sampling density without changing the frozen MLLM or its token interface. In Stage I, a proposal model predicts a question conditioned, full-frame sampling warp from frozen native-image ViT features, using only image–question–answer supervision without boxes, masks, or region labels. Stage II freezes the proposal model, constructs a candidate set com prising the native observation, the base warp, and eight fixed edits, and uses offline answer scores from the frozen MLLM to supervise native-relative rank ing and fallback decisions. On InternVL, inference requires at most one addi tional ViT encoding of the selected warped image and a single answer-decoding pass. QGVR achieves the highest reported scores on all five InternVL bench marks and four of five LLaVA benchmarks in our comparison. Across four an notated evaluation subsets, QGVR directs the model’s post-decoding visual at tention to target regions more accurately, improving target-hit accuracy by 4.00– 12.18 percentage points over image warping guided by intermediate-layer atten tion maps. Anonymous project materials and a code framework are available at https://anonymous.4open.science/w/qgvr-framework-03F0/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Visual Question Answering, Question-Guided Image Warping, Visual Reallocation, Candidate Selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DdGXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents QGVR, a two-stage framework designed to improve visual sampling efficiency in MLLMs under a fixed visual token budget. The first stage, VJP-WARP, learns question-conditioned image warps through answer-guided candidate verification and density distillation. The second stage, WARPRANK, selects between the original image and multiple warped candidates based on learned preferences derived from offline answer scores, avoiding unnecessary warping when the original image is more suitable. The framework operates without modifying the underlying MLLM and aims to preserve global image context while emphasizing question-relevant regions. Experiments on five VQA benchmarks with InternVL3.5-8B and LLaVA-1.5-7B show improvements over the evaluated baselines. The authors also provide ablation studies, attention analyses, and efficiency measurements to examine the effectiveness and overhead of the proposed design.

Critical strengths

(1) The paper considers the problem of improving visual sampling efficiency under a fixed token budget. By adjusting the sampling density while retaining the full image context, the proposed method provides an alternative approach to fine-grained visual understanding without increasing the number of visual tokens.

(2) QGVR uses answer likelihoods from the frozen MLLM to evaluate candidate warps, providing task-related feedback for learning visual sampling patterns instead of relying solely on attention maps.

(3) By retaining the original image as a candidate and introducing a learned ranking and fallback mechanism, the method accounts for cases where image warping may not improve performance.

Critical weaknesses

(1) In Stage I, candidate warps are generated through local perturbations of the current proposal, and the verified candidates are then used as distillation targets. However, if the initial proposal leads to a poor sampling pattern, this procedure may reinforce suboptimal solutions rather than identify better alternatives. In addition, the update is skipped when no valid edit is found and the base warp is not significantly harmful. I would appreciate clarification on how the method handles these cases and whether the local search strategy is sufficient to avoid poor solutions.

(2) The selection gate threshold is set to 10−710^{-7} for InternVL and 0.08750.0875 for LLaVA, while the minimum ranking margin is zero for both models. Under these settings, the final decision appears to depend primarily on the ranking scores. This raises the question of whether the additional benefit gate is necessary and how much it contributes to the final performance.

(3) The experimental evaluation leaves some questions about generalization. QGVR requires training, but the comparisons include only one training-based baseline. In addition, the training datasets largely overlap with the evaluation benchmarks, with GQA being the only benchmark excluded from training. This makes it difficult to determine how well the method generalizes to different tasks. The evaluation also does not cover other widely used backbones, such as Qwen2. I would appreciate further clarification or additional experimental evidence on these points, where feasible within the rebuttal period, to better assess the generalizability and robustness of the method.

Questions for authors

Please refer to the Critical Weaknesses section.

4 comments
AuthorsOpenReview ↗

Response to DdGX (1/3): Proposal Verification and Skipped Updates

To accommodate the per-comment length limit, we organize our response into three parts, corresponding to your three concerns. We thank the reviewer for recognizing the motivation of visual reallocation, answer-based verification, and native-image fallback. We address the three concerns below with a training-trace audit, controlled gate ablations, and additional benchmark evaluations.

(1) Handling suboptimal proposals and skipped updates.

Stage I does not automatically distill its current proposal. Instead, answer-based verification determines whether an edited candidate provides a suitable training target. An edit must improve answer NLL over the base warp by at least 0.01 while remaining within a 0.01 NLL tolerance of the native image. If no edit qualifies and the base warp is worse than Native by more than 0.05 NLL, training uses a uniform-density target corresponding to identity sampling.

We audited all 3,200 example occurrences in the completed 200-step VJP control run, including 50 warmup steps:

Verification outcomeTraining actionOccurrencesPercentage
An admissible edit is foundDistill the verified edit density1,50647.06%
No admissible edit; base warp exceeds the harmfulness thresholdApply an identity-correction target42113.16%
No admissible edit; base warp does not exceed the thresholdAssign zero weight to this example1,27339.78%
Total3,200100.00%

The base warp exceeded the harmfulness threshold in 810 occurrences. Among these, 389 produced an admissible edit and 421 received identity-correction targets; none entered the skip branch. Thus, harmful base proposals were explicitly addressed through verified alternatives or corrective targets, rather than automatically reinforced.

For skipped examples, the available candidates do not justify a supervised update. Assigning zero weight avoids introducing an unsupported target, while the shared proposal network continues learning from other examples. Skipping an example therefore does not freeze its future prediction.

As complementary evidence, our matched VJP–Random control holds architecture, initialization, data order, optimization, and verification budget fixed. VJP increases Oracle@10 from 85.79 to 86.43, with a paired difference of +0.64 points [95% CI: 0.14, 1.15]. This supports more effective candidate discovery within the tested budget.

AuthorsOpenReview ↗

Response to DdGX (2/3): Benefit-Gate Ablations

(2) Necessity and contribution of the benefit gate.

The benefit gate provides an additional acceptance condition beyond native-relative ranking. Its contribution depends on the backbone configuration, as demonstrated by the following controlled comparisons.

For LLaVA, we conducted a paired replay on the same ten-candidate answer cache, holding the proposal, candidates, ranking scores, and native-relative ranking condition fixed. The only difference was whether the benefit threshold, approximately 0.0875, was applied.

LLaVA configurationFive-task mean scoreWarp-selection rateHarmful-switch rate
Ranking + benefit gate40.4729.45%1.46%
Ranking only38.9399.93%4.86%

The gate improves the mean score by 1.54 points [95% paired CI: 1.32, 1.77] and reduces the harmful-switch rate by 3.40 percentage points. Harmful-switch rate denotes the fraction of all questions for which the selected warp scores below Native. In this configuration, a non-native candidate almost always outranks Native, so a zero ranking margin alone provides little rejection. The benefit gate supplies an additional check that substantially reduces harmful switches.

For InternVL, we agree with the reviewer that the threshold of 10−710^{-7} makes the additional benefit check effectively inactive on the evaluated examples. The common interface is retained for implementation consistency. We verified its contribution through fresh paired inference on 2,500 questions, with 500 questions from each of the five datasets. Both arms use the same proposal, candidate construction, selector weights, and native-relative ranking condition.

InternVL configurationFive-task mean scoreWarp-selection rateHarmful-switch rate
Ranking + benefit gate81.7818.16%0.92%
Ranking only81.7818.16%0.92%

Removing the gate changes neither the selected candidate nor the generated answer for any of the 2,500 questions. Native-relative ranking therefore determines selection in this evaluated configuration, and the additional gate can be removed without changing these results.

AuthorsOpenReview ↗

Response to DdGX (3/3): Additional Benchmarks and Backbone Coverage

(3) Additional benchmark evaluation and backbone coverage.

To broaden the evaluation beyond the original five benchmarks, we extended the InternVL-based comparison to MMVP, OCRBench, and AI2D, using existing checkpoints without additional training or benchmark-specific tuning for these evaluations. We also evaluated AttWarp, ViCrop, and the trained SD-RPN baseline.

The additional evaluation covers 4,388 questions: 300 MMVP questions forming 150 pairs, all 1,000 OCRBench questions, and all 3,088 AI2D test questions. The updated results are:

MethodMMVP pair accuracy (%)OCRBench accuracy (%)AI2D accuracy (%)
Native56.6782.1081.99
AttWarp46.6782.5081.61
ViCrop57.3381.2081.77
SD-RPN55.3382.9082.22
QGVR58.0083.6082.67

QGVR achieves the highest numerical score among the evaluated methods on all three additional benchmarks. Relative to Native, the improvements are 1.33 percentage points on MMVP, 1.50 on OCRBench, and 0.68 on AI2D. Relative to the strongest evaluated baseline on each benchmark, the improvements are 0.67, 0.70, and 0.45 percentage points, respectively. These results extend the empirical evidence across paired visual discrimination, text recognition and understanding, and diagram question answering.

Regarding backbone coverage, our evaluation already spans two distinct language-backbone families and visual encoders:

Evaluated MLLMLanguage backboneVisual encoder
InternVL3.5-8BQwen3InternViT
LLaVA-1.5-7BVicuna-7B-v1.5, from the LLaMA familyCLIP

The existing results therefore provide evidence that QGVR’s benefits extend across both Qwen-based and LLaMA-based MLLM configurations, rather than being restricted to a single language-backbone family. We will make these architectural differences explicit in the revision.

These results suggest that the benefit gate is effectively redundant in the evaluated InternVL configuration, as removing it does not alter candidate selection or generation for any of the 2,500 questions. This raises concerns about the necessity of this component.

Reviewer_zCsZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

When multimodal large language models handle detail-dense images, the question-relevant evidence usually occupies only a small part of the picture, yet the model still pays a fixed visual-token budget for the whole image. The paper proposes to redistribute visual sampling density without modifying the frozen model. It first rearranges the sampling density over the full image according to the question through a continuous coordinate transformation, giving detail regions denser sampling while preserving the complete field of view. It then trains a lightweight selector from the frozen model's own offline answer feedback to choose among several candidate observations, falling back to the original image when no candidate helps, and only the selected observation is decoded at inference. Across two frozen backbones and multiple document, chart and natural-image question-answering benchmarks, the paper compares against existing methods under the same budget and reports control experiments and online-cost measurements, finding better accuracy at the same budget with efficiency close to the original model.

Critical strengths
  1. The proposal-side controls are rigorous, and the key comparisons come with paired bootstrap intervals.
  2. The online-cost measurement and its scope are reported clearly.
Critical weaknesses
  1. The paper's formatting is very poor, with too many lines that are less than 80% filled.
  2. The necessity of the learned selector relative to non-trivial training-free selection rules is not established.
  3. Compared to tool-calling-based RL approaches, what are the advantages of this paper? Consider a scenario where a single image corresponds to multiple questions: a tool-calling approach would simply invoke the tool multiple times, whereas the proposed method requires applying multiple different warps to the same image. Does this paper further discuss such cases? Specifically, how should the image be processed when there are multiple questions associated with a single image?
  4. The novelty is limited. The proposed approach appears to be more of an incremental improvement over prior work, and the necessity of the method is not adequately justified.
Questions for authors

Please see weakness.

5 comments
AuthorsOpenReview ↗

Response to zCsZ (1/4): Formatting and Readability

Thank you for your detailed feedback. We address the four concerns below.

  1. Formatting and readability. We agree that the presentation should be more compact and easier to follow. We will revise paragraph and equation layouts, remove unnecessary forced line breaks, and improve figure and table placement to reduce avoidable whitespace while retaining the official conference template.

    We will also streamline the experimental narrative: each experiment will first state the question it addresses, followed by its principal finding and interpretation. Detailed implementation information will be consolidated in the appendix.

AuthorsOpenReview ↗

Response to zCsZ (2/4): WarpRank versus Training-Free Selection

  1. Learned selection versus non-trivial training-free rules. We agree that direct comparisons between selection rules are needed to assess WarpRank. We therefore conducted an additional fixed-candidate diagnostic using InternVL3.5-8B on five benchmarks, with 500 questions per benchmark. All methods share the same frozen proposal, ten-candidate pool comprising the native observation and nine warped views, prompts, and task scorers. Candidate answers are generated using greedy decoding with a maximum of 32 new tokens.

    Native always selects the original image.

    Density + geometry selects the candidate maximizing log⁡(Mk/M0)−λDk\log(M_k/M_0)-\lambda D_k, where MkM_k measures the predicted proposal density sampled by candidate kk, and DkD_k penalizes geometric distortion through the sampling-grid Jacobian’s deviation from the identity. A single coefficient, λ=100\lambda=100, was selected from a predefined grid using 250 development questions—50 per benchmark—with no image-hash overlap with the evaluation set, and then frozen. Under this calibration, the rule selected Native for every evaluation question.

    Vote@10 generates answers for all ten candidates and selects the most frequent normalized answer. Normalization uses Unicode NFKC, case folding, and whitespace normalization while preserving punctuation and numerical symbols. Ties favor the native answer when it belongs to a tied majority; otherwise, they follow the fixed candidate order.

    Confidence@10 selects the candidate with the highest mean token log probability of its own generated answer, excluding EOS and padding tokens. Empty answers receive the lowest confidence, and ties follow the fixed candidate order.

    WarpRank uses the original frozen deployment checkpoint and thresholds, without retraining or threshold adjustment. It ranks candidates before answer generation and retains the native observation when the replacement conditions are not met.

    MethodFive-task mean (%)Harmful switches (%)Preliminary latency (s/question)
    Native79.640.000.517
    Density + geometry79.640.000.563
    Vote@1080.092.803.822
    Confidence@1080.093.643.820
    WarpRank81.601.000.604

    WarpRank improves the macro-average over Vote@10 by 1.52 points (95% paired image-cluster bootstrap CI: [0.79, 2.25]) and over Confidence@10 by 1.51 points ([0.73, 2.32]). It also reduces harmful switches, defined as selections that score below the native-image answer, with all questions used as the denominator. The gains are strongest on InfoVQA and DocVQA; WarpRank does not achieve the highest score on every benchmark.

AuthorsOpenReview ↗

Response to zCsZ (3/4): Tool-Calling Complementarity and Multiple Questions per Image

  1. Relationship to tool-calling approaches and multiple questions per image. We view QGVR as complementary to tool-calling-based visual reasoning. Methods such as DeepEyes can crop and revisit image regions during reasoning, allowing subsequent observations to depend on intermediate reasoning results. They can also answer directly when further inspection is unnecessary.

    QGVR addresses a different bottleneck: relevant evidence may already be present in the original image but insufficiently represented under a limited visual sampling budget. We improve its representation before answer generation, using question-guided spatial reallocation to increase sampling density around relevant regions while preserving the full field of view. The target MLLM remains frozen, and the selected observation retains the prescribed visual-token budget.

    Although QGVR constructs ten candidates, it does not generate ten candidate answers during deployment. WarpRank selects one observation, followed by a single answer-generation pass. This provides a fixed-budget processing route without an interleaved visual-tool interaction loop. Adaptive tool use remains valuable when solving a question requires further visual search, comparison, or confirmation.

    For an image associated with multiple questions, QGVR predicts and selects a question-specific observation for each question. Different questions may require different sampling allocations; these are produced by forward inference, without per-question retraining or online optimization. Each answer is generated from its corresponding selected observation.

Reviewer_Kt7jOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

QGVR proposes a two-stage framework for reallocating visual sampling density according to a question while preserving the original image's field of view and keeping the underlying MLLM frozen. In Stage I, a question-conditioned proposal network predicts a spatial density map from frozen visual features and question embeddings. Instead of optimizing the answer loss directly, the method uses vector–Jacobian products (VJPs) to identify potentially useful spatial perturbations. The frozen MLLM then evaluates these candidate warps using answer likelihood, and verified improvements are distilled into the proposal network. In Stage II, the proposal network is frozen, and ten candidate observations are generated, with a lightweight selector (Warprank) being trained using offline answer scores to choose a candidate or fall back to the original image.

The authors evaluate QGVR using two LMMs across 5 VQA benchmarks. QGVR achieves the highest reported scores among the compared methods. The authors additionally report improvements in spatial attention alignment and inference throughput relative to other visual preprocessing methods.

Critical strengths
  • The paper presents interesting approach to efficiently reallocating limited visual sampling capacity toward question-relevant regions while preserving global context and keeping the underlying MLLM frozen.

  • The combination of answer-supervised warp proposal learning and native-relative candidate selection is an interesting extension of existing attention-guided warping methods. In particular, allowing the model to fall back to the original image helps mitigate potentially harmful transformations.

  • Empirical results show promising improvements over existing approaches. The component-level ablations and additional analyses of spatial attention and inference efficiency are also valuable.

Critical weaknesses
  • Table 3(b) shows that VJP-guided supervision improves Oracle@10 by only 0.64 percentage points compared with random-direction supervision (which I deem marginal despite the confidence interval). Moreover, this comparison evaluates the candidate pool rather than the complete deployed system. Given the additional complexity of computing gradients through the frozen MLLM, I would like stronger evidence that VJP-based supervision provides meaningful end-to-end benefits over simpler alternatives.
Questions for authors

Q1) Table 3(b) shows only a 0.64% improvement in oracle@10 over random-direction supervision. Could the authors evaluate the complete QGVR pipeline using VJP-guided versus random-direction proposals with the same selector? How much does VJP supervision contribute to actual end-to-end performance? Q2) Table 2 shows that Stage I sometimes underperforms the native model, while Stage II substantially improves performance. To what extent are the reported gains attributable to learning better visual transformations versus selecting among multiple candidate views? Could simpler candidate-generation strategies achieve comparable improvements with WarpRank?

1 comment
AuthorsOpenReview ↗

Clarifying VJP’s End-to-End Contribution and the Complementary Roles of Proposal Learning and Selection

Thank you for recognizing the value of question-guided visual reallocation and native-relative selection. Below we address the distinction between proposal quality, candidate-pool potential, and deployed answer accuracy.

Response to Q1: Does VJP supervision improve the complete deployed system?

Table 3(b) compares proposal models trained for 200 updates, including 50 warmup updates, under matched initialization, data order, and verification budgets. The 0.64-point Oracle@10 improvement measures candidate-pool potential rather than the performance of the deployed selector.

To examine performance beyond this short-training setting, we conducted an additional experiment using the main VJP checkpoint trained for 6,016 updates, as well as a newly trained Random proposal model trained for the same number of steps. We directly evaluated end-to-end answer accuracy after WarpRank selection, rather than the Oracle@10 upper bound. Both models use the same frozen deployment WarpRank, ten-candidate construction, prompting, decoding, and scoring settings, without selector retraining or threshold tuning.

DatasetNativeVJP + WarpRankRandom + WarpRankVJP − Random
TextVQA80.4681.8880.58+1.30
InfoVQA76.4082.0777.80+4.27
DocVQA92.6695.0493.16+1.88
ChartQA86.6787.2786.87+0.40
GQA62.2063.4062.60+0.80
Macro-average79.6881.9380.20+1.73

The evaluation initially included 500 questions per dataset. After excluding seven questions with known image overlap with historical proposal-training data, 2,493 questions remained. The VJP-based configuration achieves higher final scores across all five datasets, with a 1.73-point macro-average advantage (95% paired image-cluster bootstrap CI: [1.06,2.41][1.06, 2.41]).

Response to Q2: What do proposal learning and candidate selection each contribute?

Stage I and Stage II address related but distinct objectives. Stage I learns a question-conditioned spatial sampling density, using finite-warp verification based on answer likelihood. Stage II freezes this proposal model, constructs the native view, the base warp, and eight structured variants, and learns selection from decoded-answer task scores. Better answer likelihood does not necessarily produce a better decoded answer, which motivates this second-stage supervision.

The structured candidates explore different sampling strengths, spatial extents, and directions around the learned proposal. WarpRank selects a useful observation or retains the native image. Consequently, the base warp alone may underperform on some tasks while the proposal still supplies useful alternatives for final selection.

To test whether a simpler proposal can achieve comparable improvements, Table 3(a) replaces the learned density with a fixed center Gaussian, while preserving the warp operator, ten-candidate construction, and frozen WarpRank:

Proposal with the same WarpRankFinal macro-average score
Fixed center Gaussian79.83
Learned QGVR proposal81.74
Difference+1.91

The learned proposal improves final performance by 1.91 points (95% paired CI: [1.21,2.60][1.21, 2.60]). This existing Table 3(a) control is distinct from the additional experiment above; its scores are compared within that control.

These results show that the learned proposal contributes beyond the tested fixed-Gaussian alternative, even with the same selector. They support complementary roles for learned candidate generation and task-aware selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.