acceptodds
Withdrawn from ICLR 2027

WATCH: Budgeted Active Perception and Visual Memory Control for Efficient Multimodal Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal efficiency is often treated as a compression problem: process a fixed visual input, then reduce its context. As visual inputs grow in resolution and duration, this view overlooks a fundamental asymmetry: evidence must be acquired before it can be compressed, yet downstream compression cannot recover evidence that was never acquired. Active perception adapts what to observe; token reduction adapts what to retain. Separating these stages misses their causal dependence: the value of new evidence depends on what can survive bounded memory, while memory determines whether further perception is worthwhile. We introduce WATCH, a budget-conditioned closed-loop framework for sequential evidence allocation over bounded visual memory. At each step, it halts or acquires a view; only after new evidence arrives does it update bounded persistent memory. WATCH separates cumulative observation cost from memory capacity, enforces both constraints by construction, and learns the structured policy from downstream utility with reinforcement learning; the process admits a budgeted optimal-stopping interpretation and requires a single terminal call to the frozen language backbone. Across image and video benchmarks, WATCH improves the quality–efficiency frontier over passive, compression-only, and active-perception baselines while reducing both visual acquisition and terminal context; gains persist across budgets and backbones and translate into lower latency and memory. The anonymized code is available at https://anonymous.4open.science/r/WATCH/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language models, multimodal reasoning, budgeted active perception, visual token compression, visual memory control, efficient inference, reinforcement learning, adaptive computation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_zFGKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes WATCH, an Acquire–Retain–Halt framework for budgeted active visual perception and bounded visual memory in multimodal large language models (MLLMs). The method adaptively selects visual observations, maintains a compressed persistent visual memory, and learns when to stop further perception. However, I find the paper difficult to understand, inadequately presented, experimentally insufficient to support its central claims, and weak in methodological novelty.

Critical strengths
  1. This paper follows a standard structure, with the method, experiments, and additional analyses presented in separate sections.
  2. The framework is technically complete. WATCH integrates adaptive acquisition, bounded memory, and stopping into a unified inference loop, while explicitly enforcing observation and memory budgets.
  3. The paper provides substantial implementation and reproducibility details. The authors include training procedures, ablations, statistical analysis, and an anonymized code release.
Critical weaknesses
  1. The paper is extremely difficult to read and understand. The paper introduces many unusually complicated concepts within a short main paper, such as geometry-aware redundancy resolution, budget-feasible view allocation, and coverage-aware memory control. Even after carefully reading the paper, these unique and complex terms remain difficult to interpret. Moreover, the paper spends substantial space describing technical procedures and notation, but provides insufficient intuitive explanation of the overall method. There is only a very vague method figure but without any detailed explanations. Furthermore, the paper does not clearly decompose the method or explain which component constitutes the core contribution and why all of this machinery is necessary.

  2. The presentation of the main experimental results is insufficient and deliberately conceals some findings. The main paper provides very limited experimental results, with only two tables. More importantly, Table 1 covers six image benchmarks, but it reports individual scores for only four benchmarks. The authors deliberately combine the results for the remaining two, RealWorldQA and MMMU, into a single “Gen.” score by averaging them, without explaining why these two benchmarks are treated differently. This selective aggregation obscures more meaningful differences between the two datasets. Furthermore, the score label “Gen.” is potentially misleading, as it could easily be interpreted as a generation-related (e.g. "generation length") metric rather than an aggregated performance score. It is unclear why this abbreviation is used to denote the average of RealWorldQA and MMMU.

  3. The novelty of the core idea is limited. The central idea of WATCH appears closely related to existing approaches with several recent works [1-4]. These studies have already explored highly similar directions involving adaptive visual acquisition, iterative perception, and adaptative stopping or feedback mechanisms. In WATCH, these ideas are instantiated as an Acquire–Retain–Halt loop: the model selects a visual view, updates memory, and decides whether to continue or stop. Therefore, it is unclear what fundamentally new methodological insight WATCH introduces beyond integrating these existing ideas into a unified framework.

  4. The experimental setup does not sufficiently support the paper’s claims about multimodal reasoning. The evaluation is mainly based on Qwen2.5-VL-7B-Instruct. Given that the paper frames WATCH as a method for efficient multimodal reasoning, the lack of evaluation on reasoning-oriented multimodal models (e.g., Qwen3-VL-Thinking series models) makes it difficult to assess whether the proposed framework generalizes to stronger reasoning settings.

  5. The comparison with stronger and more advanced baselines is insufficient. Many closely related recent methods [1-6] are not directly compared in the main results. Since WATCH is closely related to prior work, these recent methods should also be included as baselines for comparison. Given the broad claims of the paper, stronger comparisons with these recent approaches are necessary.

[1] Question-guided Visual Compression with Memory Feedback for Long-Term Video Understanding. CVPR 2026.
[2] Native Active Perception as Reasoning for Omni-Modal Understanding. ICML 2026.
[3] Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understanding. CVPR 2026.
[4] LensWalk: Agentic Video Understanding by Planning How You See in Videos. CVPR 2026.
[5] Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning. ICLR 2026.
[6] MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs. ICLR 2026.

Questions for authors

see weakness

Reviewer_3yQCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WATCH, a closed-loop framework designed to improve multimodal reasoning efficiency under explicit constraints on both visual acquisition cost and visual memory capacity. Keeping the vision encoder and language backbone frozen, WATCH employs an RL-trained lightweight controller to iteratively execute three decisions: whether to stop (Halt), which spatial region or temporal segment and resolution to observe (Acquire), and how to compress and update the bounded visual memory (Retain), invoking the language model only once at the end to generate the final answer. Experiments across multiple image and video benchmarks demonstrate that WATCH improves overall task performance while reducing visual acquisition volume, context length, and inference latency.

Critical strengths
  1. The experimental evaluation covers a diverse set of image and video benchmarks with broad empirical coverage.
  2. The paper is clearly written and easy to follow.
Critical weaknesses
  1. Lack of intermediate LLM reasoning during the perception loop. Throughout the multi-step perception loop, WATCH never invokes the MLLM backbone, relying solely on a 512-dimensional lightweight controller to decide where to look and what to retain. When a task requires identifying an initial clue, combining it with world knowledge to perform an intermediate logical deduction, and only then knowing what to search for next (i.e., multi-hop visual reasoning or sequentially dependent visual operations), I am concerned that the lightweight controller cannot substitute for the reasoning capability of the language model, causing the method to fail on complex tasks.
  2. Heavy reliance on offline behavior cloning (BC) warm-start and unclear standalone contribution of RL. The controller is not learned purely via RL from scratch; instead, it first uses SigLIP-SO400M together with the frozen MLLM's log-likelihood of ground-truth answers to exhaustively score grid candidates and construct 30K offline trajectories for behavior cloning (BC) warm-start, making the pipeline very heavy. Meanwhile, the main text does not clearly separate the performance of "BC only" versus "BC + RL" in the main results table, making it difficult to intuitively assess how much the complex heterogeneous GRPO and potential-based reward shaping actually contribute?
Questions for authors

See Weaknesses.

Reviewer_LBh2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces WATCH, a budget-conditioned framework that jointly optimizes sequential visual evidence acquisition and bounded memory compression through a closed-loop policy. It uses reinforcement learning to balance observation costs, memory capacity, and downstream task utility, with both budgets explicitly enforced and only a single terminal call to the frozen language backbone. Experiments across image and video benchmarks demonstrate improved quality-efficiency trade-offs over existing baselines, reducing visual acquisition, context length, latency, and memory usage.

Critical strengths
  1. Active perception and bounded visual memory are modelled under explicit resource budgets

  2. The designed policy integrates acquisition, memory retention, and adaptive stopping with guaranteed budget compliance

  3. The experiments across diverse image/video benchmarks and MLLM backbones show the effectiveness of the method

Critical weaknesses
  1. The paper is badly written and hard to follow

  2. The challenge of multimodal reasoning and what motivates the design of "Budgeted Active Perception and Visual Memory Control" are unclear

  3. The novelty and contributions that distinguish from existing works are unclear, too.

Questions for authors

See Critical Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.