acceptodds
Under review as a conference paper at ICLR 2027

Allocate Before You Embed: Adaptive Visual Input Allocation for Video Embeddings

Abstract

Large-scale video retrieval requires embedding models to encode long and diverse videos under tight visual-input and inference budgets. Existing methods typically sample a small, fixed set of frames at their original resolution, limiting temporal coverage and ignoring frame importance. Our empirical analysis shows that expanding temporal coverage improves retrieval even under a fixed visual-input budget. Gains are larger when the original per-frame resolution is preserved, highlighting the complementary roles of temporal coverage and spatial fidelity. Motivated by this finding, we propose AllocEmbed, an allocate-then-embed framework that reallocates a fixed visual-input budget across more frames. A lightweight allocator uses low-cost previews to assign frame-wise resolutions before the embedding backbone, preserving more detail where it most benefits retrieval while reducing visual cost elsewhere. We further introduce Retrieval-Driven Policy Optimization (RDPO), which learns the allocator directly from retrieval feedback using a rank-validated similarity gap and a confidence-guided efficiency incentive. Operating entirely before the backbone, AllocEmbed integrates with existing retrieval systems without modifying the embedding model or downstream pipeline. Experiments on the MMEB-V2 V-QA and V-RET tasks and our LongRet benchmark show that AllocEmbed achieves the best overall retrieval performance among the evaluated budget-matched methods and transfers across embedding backbones. Code is included in the supplementary material.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Retrieval, Video Embeddings, Vision-Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qbA2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AllocEmbed, a framework for improving video retrieval under a fixed visual input budget. Instead of using a small number of uniformly sampled frames at the same resolution, AllocEmbed distributes the available budget across a denser frame sequence by adaptively adjusting per-frame resolutions. The authors further introduce Retrieval-Driven Policy Optimization (RDPO), which trains the allocation policy using retrieval ranking feedback and an efficiency incentive while keeping the embedding backbone frozen. Experiments on MMEB-V2 and the newly constructed LongRet benchmark show improvements over several budget-matched baselines, along with cross-model transferability.

Critical strengths
  1. The paper addresses a meaningful trade-off between temporal coverage and spatial resolution under constrained visual input budgets. The allocate-then-embed framework is intuitive and can be applied without modifying the embedding backbone.

  2. The experiments cover multiple retrieval tasks and four embedding backbones. The consistent average improvements, particularly on LongRet, and the transferability across different backbones support the practical potential of the proposed approach.

Critical weaknesses
  1. The proposed method appears to offer limited methodological novelty. The proposed method closely follows ResAdapt in both its allocator architecture and reinforcement learning framework. RDPO mainly introduces retrieval-specific reward designs within an existing GRPO/PPO-style optimization framework. While these adaptations are reasonable, the overall methodological novelty appears limited.

  2. The latency analysis compares AllocEmbed against a more expensive full-resolution baseline instead of budget-matched alternatives, and excludes preprocessing overhead. As a result, the practical efficiency benefits of the proposed framework remain insufficiently demonstrated.

[1] Huanxuan Liao, Zhongtao Jiang, Yupu Hao, Yuqiao Tan, Shizhu He, Ben Wang, Jun Zhao, Kun Xu, and Kang Liu. Resadapt: Adaptive resolution for efficient multimodal reasoning. arXiv preprint arXiv:2603.28610, 2026.

Questions for authors
  1. What is the main technical contribution beyond ResAdapt? Since the allocator architecture and scale policy are largely inherited from ResAdapt, could the authors clarify what is fundamentally new in RDPO beyond adapting existing policy optimization methods to retrieval-specific rewards?

  2. Does AllocEmbed provide a meaningful benefit over uniform allocation when accounting for its additional overhead? The improvements on MMEB-V2 are relatively small. Could the authors provide an end-to-end latency comparison against budget-matched baselines, including allocator inference and preprocessing costs?

  3. The experiments only cover embedding backbones up to 8B parameters. Does AllocEmbed remain effective on larger-scale embedding models (e.g., 13B or above), particularly when transferring the allocator without retraining?

Reviewer_ssW3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AllocEmbed, an adaptive visual input allocation framework for efficient video embeddings under constrained visual token budgets. To balance temporal coverage and spatial fidelity, AllocEmbed dynamically assigns frame-wise resolutions based on video content and taskrelated text, enabling denser frame sampling without increasing visual token consumption. The authors further introduce Retrieval-Driven Policy Optimization (RDPO), which optimizes the allocation policy using retrieval ranking feedback and a confidence-guided efficiency incentive. Experiments on MMEBV2 and the newly constructed LongRet benchmark demonstrate improved retrieval performance under comparable visual token budgets, along with generalization across different embedding backbones.

Critical strengths
  1. The paper addresses the practical trade-off between temporal coverage and spatial fidelity in video embedding models under constrained visual token budgets. The empirical observations provide convincing motivation for exploring adaptive visual input allocation.
  2. AllocEmbed introduces an adaptive frame-wise resolution allocation strategy that preserves broader temporal coverage while prioritizing spatial details in informative frames. This provides a meaningful alternative to conventional uniform frame sampling and resolution scaling.
  3. AllocEmbed operates entirely before the embedding backbone without requiring architectural modifications or backbone fine-tuning. Its ability to transfer across multiple embedding models of different sizes without retraining further supports its practical applicability
Critical weaknesses
  1. Insufficient comparisons with existing visual token reduction methods. The experimental comparisons mainly focus on frame selection and resolution allocation, while overlooking representative visual token reduction methods, such as token pruning and merging. These approaches can also support denser temporal sampling (e.g., 64 or 128 frames) under constrained visual token budgets. Comparisons under matched token budgets are necessary to establish the advantages of AllocEmbed over existing efficient video processing techniques.
  2. Limited improvements over a simple uniform allocation baseline. On LongRet with VLM2VecV2-2B, Uniform Allocation improves Hit@1 by 7.77 points, while AllocEmbed achieves a 9.09- point improvement. It shows although AllocEmbed reports substantial improvements over the 8- frame baseline, a large portion of these gains can already be achieved by uniformly downscaling a denser set of frames 3.Insufficient analysis of task-conditioned allocation. The allocator uses both video content and task-related text to predict frame-wise resolutions, but the contribution of textual conditioning is not independently evaluated. In particular, it remains unclear whether the text input represents a generic retrieval instruction or a query-specific description, and how the proposed allocation strategy accommodates offline video embedding and indexing in large-scale retrieval systems.
Questions for authors

AllocEmbed relies on a fixed number of uniformly sampled candidate frames. How does the choice of candidate frame count (e.g., 24, 64, or 128) affect retrieval performance under the same visual token budget? Additionally, what motivates the fixed-frame design rather than jointly adapting temporal sampling and spatial resolution allocation?

Reviewer_dVuwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AllocEmbed, an adaptive visual budget allocation method for video embeddings. The approach dynamically adjusts the resolution of individual frames based on video content and task-related text. It employs Retrieval-Driven Policy Optimization (RDPO) to learn an allocation strategy, thereby enhancing retrieval performance while adhering to a fixed visual input budget. Experiments conducted on the MMEB-V2 and LongRet benchmarks demonstrate superior results compared to a fixed 8-frame baseline, while also showing potential for cross-model transferability. Nevertheless, the independent contribution of the learned allocation strategy and its actual efficiency gains require further validation.

Critical strengths
  1. The proposed adaptive allocation strategy aims to preserve important visual information while reducing unnecessary computation. Performing input allocation before the embedding backbone also makes the method potentially compatible with existing models.
  2. The authors evaluate AllocEmbed across multiple models and video retrieval benchmarks, comparing it with fixed-input strategies. The results show that AllocEmbed improves retrieval performance in several settings while maintaining similar or lower visual-token costs.
Critical weaknesses
  1. The independent innovation of this work, relative to ResAdapt, requires more robust justification. The paper explicitly adopts the allocator architecture and the Beta-distribution-based scaling strategy from ResAdapt. ResAdapt already proposed allocating a "visual budget" prior to encoding—while keeping the backbone frozen—and learning the trade-off between accuracy and cost through policy optimization. Consequently, the primary contributions of this paper are limited to the design of a retrieval-based reward and the adaptation of the framework to new application scenarios. The current experiments lack a direct, head-to-head comparison between this method (RDPO) and ResAdapt/CAPO on the same retrieval tasks; thus, it has not been sufficiently demonstrated that RDPO offers a distinct advantage over existing cost-aware policy optimization strategies.

  2. The concept of "budget matching" is insufficient to substantiate an overall efficiency advantage. In the main experiments, the "cost" metric accounts only for the average number of tokens fed into the embedding backbone, excluding the computational overhead of the allocator and associated preprocessing steps. While the appendix does report latency, the comparison is made against a 24-frame, full-resolution input—rather than the 8-frame "Base" configuration or the 24-frame "Uniform Allocation" baseline used in the primary comparisons. Furthermore, this latency analysis omits steps such as image scaling, video transformations, and backbone-specific preprocessing. Therefore, while the results demonstrate computational savings relative to high-cost inputs, they do not prove superiority over the "budget matching" baseline within the full, end-to-end pipeline. It is recommended to report end-to-end latency, throughput, peak VRAM usage, and performance-vs.-cost curves under consistent experimental settings.

  3. The ablation studies suffer from confounding cost factors, leaving the independent contribution of the proposed algorithm unclear. In Table 2, removing global negative samples results in a 1.30 percentage point drop in Hit@1, but it is accompanied by a 38.12% reduction in token costs. Consequently, the observed performance shift could stem from either the negative sampling mechanism or the lower input resolution, making it impossible to attribute the change solely to the former. Additionally, compared to "Uniform Allocation," the proposed method yields average improvements of 1.08 and 0.57 percentage points on MMEB-V2, and 1.32 and 1.04 percentage points on LongRet, across two 2B-parameter models. While these gains are noteworthy, the analysis requires further ablation—specifically controlling for cost—as well as results based on multiple random seeds with reported confidence intervals to confirm the stability of these improvements.

  4. There are significant gaps regarding the retrieval protocol and reproduction details. The retriever relies on a task text (qq), yet LongRet pertains to text-to-video retrieval. The paper needs to clarify whether candidate videos are encoded using a fixed task instruction or based on each specific retrieval query. If the latter, candidate videos would need to be re-encoded for different queries—a process that differs from large-scale retrieval scenarios where indices can be pre-built.

  5. The linear-scale cost proxy may result in an ordering that is the reverse of the actual token-based cost. Equation (13) constructs a cost proxy using an average scale �0��0�; however, since the scale applies to both height and width—and the appendix notes that cost grows approximately with the square of the scale—this proxy may be inaccurate.

  6. There is a gap between the hard budget constraint and the actual optimization mechanism. While Equation (1) requires �1�,�1�, the subsequent strategy only constrains the scale range for individual frames and incorporates a "soft" efficiency reward; it does not specify how the budget constraint is strictly enforced for an individual video.

  7. The allocation settings used in the latency experiments differ from those in the main experiments. The main experiments employ a default scale range of [0.2,1.8][0.2, 1.8], whereas the latency experiments in Appendix B.2 use a range of [0.2,1.32][0.2, 1.32] and utilize two-frame block-based prediction. Consequently, the 1.941x speedup reported in Table 6 cannot be directly interpreted as the performance corresponding to the configuration used for the main results.

Questions for authors

See critical weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.