acceptodds
Under review as a conference paper at ICLR 2027

ResAdapt: Adaptive Resolution for Efficient Video Reasoning

Abstract

Multimodal Large Language Models (MLLMs) typically encode video frames at uniform spatial resolution, forcing fine visual detail and temporal sampling density into a rigid trade-off under constrained token budgets. Yet, visual relevance in video is temporally sparse and query-dependent: critical evidence concentrates in brief moments, while repetitive background motion carries little useful signal. We introduce , an input-side framework that treats per-frame resolution as an adaptive variable determined prior to visual feature extraction. A lightweight Allocator pairs coarse preview features with the user query to predict continuous per-frame scales, coarsening redundant intervals while allocating denser token capacity to answer-critical moments. To optimize allocation without frame-level supervision under non-differentiable processor token collation, we formulate the task as a continuous-action contextual bandit. We train the Allocator via , combining group-relative advantage with dynamic-pivot asymmetric cost shaping, temporal-similarity regularization, and concentration penalties. Across diverse video benchmarks, ResAdapt improves the accuracy–budget frontier: on Qwen2.5-VL at 128 frames, it matches dense full-resolution accuracy on the six-benchmark average ( vs. ) at only token retention, while exceeding dense performance on VideoMME and VideoMMMU. Furthermore, spatial savings are fungible with temporal coverage—reinvesting tokens into denser sampling of the same video raises VideoMME by points under a matched token budget. Permutation controls show that deliberate resolution placement across frames contributes to these reasoning gains. Finally, the frozen Allocator transfers zero-shot across diverse backbones and cuts 128-frame latency by at retention on Qwen2.5-VL. Code is available at https://anonymous.4open.science/r/ResAdapt.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Adaptive Resolution, Visual Token Efficiency, Dynamic Visual Budget Allocation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fZxbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ResAdapt predicts query-conditioned per-frame resolutions before video encoding. A lightweight Allocator is trained with CAPO using task rewards and cost shaping, while the standard variant freezes the downstream model. Experiments show reduced token usage and long-video latency, with zero-shot transfer to other backbones.

Critical strengths
  1. Pre-encoding resolution allocation reduces both visual encoding and downstream token costs while preserving sampled time points.
  2. At 128 frames, ResAdapt achieves a six-benchmark mean of 56.28.
Critical weaknesses
  1. Training cost limits the practical efficiency assessment. Training uses 32 H100 GPUs and 16 allocation–response rollouts per input over 93.4K examples, but total training time/GPU-hours and amortization against inference savings are not reported. Moreover, inference savings are regime-dependent: Table 10 shows slowdowns for short clips at conservative retention.

  2. The central design choices lack decisive controls.

    • The direct-cost baseline uses only a reported penalty coefficient of 0.5 (Appendix E.3.3). Its collapse may reflect excessive cost pressure rather than the necessity of CAPO.
    • The headline 128-frame result at 22.9% retention and the 16× sampling experiment lack a same-frame, budget-matched uniform-resizing baseline. This leaves gains from denser temporal coverage insufficiently separated from learned resolution allocation.
    • CAPO penalizes mean scale rather than realized token cost, which also depends on scale variance. The benefit of this proxy over actual retention or mean squared scale is not established.
Questions for authors
  1. What are total training GPU-hours, including tuning, and how many inference requests are needed to amortize this cost?
  2. Does CAPO outperform direct-cost training after sweeping the penalty coefficient to reach a comparable training-time budget?
  3. How does uniform resizing of the same 128 frames compare at matched realized retention, particularly for the headline result and the 8-frame-equivalent experiment? Please report accuracy and end-to-end cost.
  4. Does CAPO retain its advantage when cost shaping uses realized token retention or mean squared scale instead of mean scale?
1 comment

Dear Reviewer fZxb,

Thank you again for your constructive feedback. As the discussion period draws to a close, we would greatly appreciate another look at the training-cost estimate, the new 128-frame uniform-resizing controls, and our explanations of CAPO’s training behavior and cost proxy.

If these revisions strengthen your assessment of the method and its practical value, we would be sincerely grateful if you would consider increasing your rating by one point. We welcome any remaining questions and appreciate your careful consideration.

Best regards, The authors

Reviewer_bNgfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents ResAdapt, an input-side adaptive resolution framework for efficient video reasoning. Instead of encoding all sampled frames at a uniform resolution, ResAdapt uses a lightweight Allocator to predict continuous, query-conditioned per-frame scales before visual feature extraction. A coarse SmolVLM preview supplies visual features, and the resized frames are subsequently processed by an unmodified downstream MLLM. The allocation policy is trained as a continuous-action contextual bandit using Cost-Aware Policy Optimization, which combines group-relative advantages, asymmetric cost shaping, and temporal regularization. The experiments evaluate accuracy, visual-token retention, and wall-clock latency across multiple video benchmarks and MLLM backbones. On Qwen2.5-VL with 128 frames, the method reports a six-benchmark average accuracy of 56.28 at 22.9% token retention, compared with 55.85 for dense full-resolution input. Additional experiments study temporal–spatial budget exchange, permutation controls, zero-shot backbone transfer, and end-to-end latency.

Critical strengths
  1. The paper addresses efficiency before visual encoding, with benefits for both visual processing and language prefill. Unlike post-encoding pruning, ResAdapt avoids extracting full-resolution features for frames assigned low resolution. It also preserves the temporal sequence rather than making hard keep-or-drop decisions. The reported accuracy–retention results and measured speedups make this a practically meaningful contribution.

  2. The evaluation tests several mechanisms behind the reported gains. The temporal reinvestment experiment demonstrates that resolution savings can support denser temporal sampling. The permutation controls indicate that the placement of resolution across frames matters, while zero-shot transfer experiments suggest that the Allocator is not entirely specialized to one downstream backbone. Importantly, the latency measurements include Allocator overhead rather than reporting only visual-token reductions.

Critical weaknesses
  1. The headline accuracy and latency results do not form a single matched operating-point comparison. The strongest average accuracy result is reported at 22.9% retention, whereas the 59.5% latency reduction is reported at 28.2% retention. Consequently, the exact accuracy obtained at the main latency operating point is unclear. Moreover, the latency comparison is primarily against Vanilla input and does not fully establish the advantage over fixed-scale or other simple input-side alternatives at matched retention or accuracy. This weakens the central claim of an improved practical accuracy–efficiency frontier.

  2. Some deciding improvements are small, while their statistical reliability and operating-point selection are insufficiently established. Several benchmark-level differences are near zero, and the paper does not consistently provide paired confidence intervals or training-seed variation. It is also not sufficiently clear how the retention targets used in the main tables were selected for each backbone and benchmark. These issues matter because some conclusions depend on small differences rather than the larger aggregate latency gains.

Questions for authors
  1. How were the retention operating points in the main tables selected? Were they calibrated separately for each backbone or benchmark, and what data were used for this calibration? Were any test-set statistics used to choose them?

  2. Could you report joint accuracy, token retention, and end-to-end latency at both 22.9% and 28.2% retention for 128-frame Qwen2.5-VL? Please also include fixed-scale input-side baselines at matched retention and, where possible, matched accuracy.

  3. Could you provide paired bootstrap confidence intervals over videos for the main accuracy comparisons and variance across training seeds for the Allocator? In particular, are the small average gains over dense input statistically reliable?

  4. Could you clarify how hardware allocation affects the latency comparison, given that the Allocator and vLLM serving measurements use different numbers of H100 GPUs? Would the conclusions remain unchanged under a common serving configuration?

1 comment

Dear Reviewer bNgf,

Thank you again for your valuable feedback. With the discussion period ending soon, we would greatly appreciate your review of the added accuracy–latency comparisons and FixedScale controls, together with our clarifications of budget selection, retention reporting, and hardware allocation.

In light of this additional evidence, we would be sincerely grateful if you would reconsider your rating and consider a one-point increase where warranted. We welcome any further questions and thank you for helping us improve the paper.

Best regards, The authors

Reviewer_WocxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ResAdapt learns a spatial resolution for each sampled video frame from coarse visual previews and the question, then supplies the resized frames to a frozen downstream model (Sections 3.1 and 4.1). Its Cost-Aware Policy Optimization (CAPO) objective combines answer rewards, cost shaping, and temporal regularization to learn allocations without frame-level resolution labels (Sections 3.2 and 3.3). With 128 frames, Qwen2.5-VL achieves a six-benchmark mean of 56.28 versus 55.85 for dense input while retaining 22.9 percent of visual tokens (Tables 1 and 6). A separate configuration at 28.2 percent retention reduces end-to-end latency by 59.5 percent, including allocator overhead (Table 3). Additional experiments examine allocation placement and transfer across backbones (Tables 16 and 20).

Critical strengths
  • The paper tests whether the placement of spatial detail matters - Learning allocation from downstream answer rewards allows the policy to optimize which resolution choices preserve task performance, without requiring frame-level resolution labels (Section 3.2). This addresses a useful distinction: a frame can be relevant to a question without requiring high resolution. The strongest supporting experiment preserves the predicted scale choices and mean token retention but shuffles their assignment across frames; accuracy falls by 1.3 to 2.4 points across all six benchmarks, averaging approximately 1.72 points (Table 20). This provides stronger evidence for useful allocation than a comparison against dense input alone. Query conditioning contributes a smaller additional gain, with mean accuracy increasing from 52.43 without the query to 52.87 with it, despite using fewer tokens (Table 12). I therefore view learned fidelity placement as the better-established contribution.
  • The approach provides a useful way to trade spatial detail for temporal sampling - At a nominal eight-frame-equivalent token budget, increasing temporal sampling from eight to 128 frames raises VideoMME accuracy from 53.96 to 63.64; the experiment also includes a uniform-resizing control (Figure 3c). This supports the interpretation that retaining more time points can be more valuable than preserving uniformly high spatial resolution. The allocator trained with Qwen2.5-VL also transfers without retraining to Qwen3-VL, Kimi-VL, and MiMo-VL (Section 4.1, Tables 1 and 16). This makes the contribution more useful than a compression policy tied to a single backbone.
Critical weaknesses
  • The additional value of joint video allocation is not fully isolated from existing resolution-selection methods and simpler resizing - [1] already predicts continuous input resolutions from image–query features (Section 3.4 and Algorithm 2 from [1]). ResAdapt’s distinction should therefore focus on learning allocations jointly across video frames through task feedback and temporal regularization (Sections 3.1 through 3.3). A comparison with independent per-frame resolution selection at the same total token budget would help isolate this contribution. Additionally, [2] also assigns different resolutions before tokenization (Section 3.4 from [2]). However, ResAdapt describes its QFrame comparison as hard frame selection without specifying whether the full multi-resolution component was enabled (Section 4.2, Table 2, Appendix E.4.1). The headline 128-frame setting also lacks a uniform-resizing result near 22.9 percent retention, although this control appears elsewhere (Tables 6 and 20). These comparisons matter because the allocator adds 915.5 milliseconds at the reported 128-frame timing point (Table 3); the accuracy benefit should justify this additional computation.
  • The CAPO ablation does not fully separate objective design from the training tradeoff - The direct-cost baseline uses one penalty coefficient and learns an aggressive operating point of 9.4 percent token retention (Appendix E.3.3, Table 13). Its predictions are subsequently enlarged to reach 22.8 percent retention, where its mean accuracy is 51.57 versus CAPO’s 52.87 at 23.8 percent (Table 14). This controls inference expenditure, but enlarging an already-trained policy is not equivalent to training it to allocate that larger budget. A policy trained toward very small resolutions may learn different allocation priorities that subsequent rescaling cannot recover. The results therefore support CAPO over the tested configuration, while leaving unclear whether its additional shaping mechanisms outperform a simpler cost objective trained near the intended budget.

References

[1] CARES: Context-Aware Resolution Selector for VLMs, ACL'26

[2] Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs, ICCV'25

Questions for authors
  • How much of the improvement comes from joint temporal allocation? Could you compare against independent per-frame resolution selection, calibrated to the same total token budget (Sections 3.1 through 3.3)? An adaptation of [1] would help isolate the benefit of coordinating resolutions across frames.
  • Does the QFrame baseline include its full multi-resolution adaptation component, or only frame selection (Table 2, Appendix E.4.1)? [2] describes this component in its Section 3.4. Please specify the resolution tiers and realized token budget, and provide a matched-budget comparison with the full method if it was not included.
  • How does uniform resizing compare at the headline 128-frame setting near 22.9 percent token retention, where its result is currently missing (Table 6)? Please report both accuracy and end-to-end latency, including ResAdapt’s allocator overhead, to establish whether learned allocation improves the practical accuracy–latency tradeoff.
  • Does CAPO retain its advantage when the direct-cost baseline is trained near the same target retention? The current baseline is trained with one penalty coefficient and subsequently enlarged to match the inference budget (Appendix E.3.3, Tables 13 and 14). Tuning that coefficient during training would help distinguish CAPO’s contribution from the choice of training tradeoff.

References

[1] CARES: Context-Aware Resolution Selector for VLMs, ACL'26

[2] Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs, ICCV'25

1 comment

Dear Reviewer Wocx,

Thank you again for your thoughtful feedback. As the discussion period draws to a close, we would greatly appreciate another look at our responses, particularly the new 128-frame FixedScale results and the clarifications of temporal allocation, CAPO, and the Q-Frame implementation.

If these additions strengthen your assessment of the paper, we would be sincerely grateful if you would consider increasing your rating by one point. We welcome any remaining questions and appreciate your time and consideration.

Best regards, The authors

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for your reviews. The authors have posted their rebuttal. If you haven't already, please take a look and let them know whether it addresses your concerns. If anything is still unclear, feel free to ask follow-up questions while the discussion period is open.

Even a short note (e.g., whether you are keeping or updating your score, and why) would help both the authors and the final decision.

Thanks for your time!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.