acceptodds
Withdrawn from ICLR 2027

Towards Training-free Proactive Responses for Video Multimodal Large Language Models via Cumulative Information Estimation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite great success, existing multimodal large language models (MLLMs) still look “dumb” in online video interaction, primarily due to the lack of proactive response capability. In this paper, we propose a novel and plug-and-play approach for MLLMs, termed Cumulative Information based Active response (CIAct). Unlike existing methods that typically frame proactive response as a multi-class prediction of model states, CIAct formulates it as a problem of cumulative information estimation, i.e., allowing the MLLM to respond to questions once relevant information is accumulated enough. This definition not only allows CIAct to leverage existing lightweight vision-language models to help perceive queryrelated information, but also avoids the expensive MLLM-based video watching and dedicated SFT tuning. Moreover, CIAct is also equipped with a carefully designed memory replay scheme, helping it to cope with different online QA tasks efficiently, such as backward tracing, real-time perception, and forward active responding. To validate CIAct, we apply it to two representative offline MLLMs, and compare to a set of proactive response models and methods on both online and offline benchmarks. The experimental results not only show its superior performance than existing methods on streaming benchmarks, e.g., +4.04% on OVOBench, but also confirm its great benefits to existing offline MLLMs. For instance, CIAct helps Qwen3-VL save about 38% of watching time on LVBench, while obtaining a +2.4% performance gain. Our code is in the supplementary materials.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Streaming Video Understanding, Proactive Response, Cumulative Information Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Sme7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CIAct, a training-free framework for proactive responses in video MLLMs. It uses lightweight vision-language models to estimate query-frame relevance, applies a decayed CUSUM mechanism to determine when to respond, and introduces a memory replay strategy for efficient answer generation. Experiments are conducted on streaming and offline video understanding benchmarks.

Critical strengths
  1. The paper addresses an important challenge in online video understanding, which is enabling existing MLLMs to determine when to respond without additional training. The proposed plug-and-play design is potentially useful for real-time video applications.

  2. CIAct combines lightweight relevance estimation, cumulative evidence tracking, and memory replay into a unified framework. The experiments demonstrate promising trade-offs between video observation costs and answer accuracy.

Critical weaknesses
  1. The ablation study in Table 3 does not control for the amount of video observed. For example, on LongVideoBench, accuracy increases from 47.5% to 58.3%, while the average viewed fraction increases substantially from 15% to 79%. Therefore, it is unclear whether the gains come from the proposed cumulative information estimation or simply from observing more video content. Moreover, high query-frame relevance does not necessarily indicate that sufficient information is available to answer a question, but the paper lacks a direct evaluation of response timing accuracy.

  2. The reported 4.04% improvement on OVO-Bench is relative to the Qwen3-VL backbone, whereas CIAct only outperforms Streamo by 0.20% (55.81% vs. 55.61%) using a different backbone. It also performs worse than Streamo on Forward Active Responding (50.15% vs. 54.77%). In addition, Table 2 shows noticeable accuracy drops on several benchmarks, contradicting the claim of consistently preserving answer quality. Finally, the reported efficiency results do not clearly account for the cost of query-dependent calibration using the 16K frame reference bank. This is particularly concerning for BLIP, whose cross-modal image-text matching requires query-dependent computation for reference frames, unlike SigLIP2, which can reuse precomputed image embeddings. The authors should clarify whether this additional overhead is included in the reported latency.

Questions for authors

Q1. In Table 3, the final CIAct configuration observes substantially more video content than the simpler alternatives (79% vs. 15% on LongVideoBench). How can the authors demonstrate that the improvement comes from the proposed accumulation mechanism rather than the increased observation time?

Q2. The proposed method triggers a response when accumulated query-frame relevance exceeds a threshold. However, high relevance does not necessarily imply that the evidence required to answer a question is complete. How do the authors evaluate whether CIAct triggers responses at appropriate times, particularly for questions requiring multiple events or temporal reasoning?

Q3. In Table 1, CIAct improves over Streamo by only 0.20 points overall, while performing worse on Forward Active Responding (50.15% vs. 54.77%). Considering the differences in backbone models, how do the authors justify the claimed superiority over existing proactive response methods?

Q4. Table 2 shows that CIAct reduces accuracy on several benchmarks, such as the 4.1-point drop on Video-MME with Qwen3-VL and SigLIP2. How should these results be reconciled with the claim that CIAct consistently benefits existing offline MLLMs without sacrificing answer quality?

Q5. The method uses a reference bank containing 16K frames for query-specific calibration. Does the reported response latency include the cost of computing these reference scores, processing historical frames, and generating the final answer? In particular, how is the BLIP-based calibration performed efficiently for previously unseen queries?

Q6. Could the authors carefully review the manuscript for grammatical errors, numerical inconsistencies, and unclear experimental descriptions? For example, "answer ques" (Line 95), "the answer cues can appears" (Line 260), "not need to wach" (Line 297), and "the right time to response" (Lines 453–454) contain obvious language errors. Additionally, the reported 3.6% improvement in Lines 421–423 appears inconsistent with the rounded values in Table 3 (54.6 − 51.1 = 3.5). The authors should also clarify which computational costs are included in Figures 3 and 4 and provide a clearer description of the evaluation protocols used for different baselines.

Reviewer_AKxBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CIAct, a training-free framework that equips offline video MLLMs with proactive response capability. It computes frame–query relevance using a frozen SigLIP2 or BLIP model, standardizes the scores against a query-specific reference distribution, and accumulates them using a decayed CUSUM-style state. Once the state crosses a threshold, the answering model receives a compact memory selected by relevance and temporal weighting. Experiments on OVO-Bench and four offline video benchmarks evaluate response quality, observation duration, and computational efficiency with two answering backbones.

Critical strengths
  1. The separation of lightweight monitoring from answer generation addresses a practical cost problem. CIAct avoids additional response-timing training and repeated use of the full answering model for every decision. Its integration with existing frozen models, accompanied by latency and memory measurements, is a useful practical feature.

  2. The empirical results demonstrate potentially useful accuracy–observation trade-offs. For example, Qwen3-VL with the BLIP variant improves from 43.8 to 46.2 on LVBench while observing 62% of the video on average. The use of two relevance models and two answering backbones provides some evidence that the approach is not confined to a single configuration.

Critical weaknesses
  1. The connection between accumulated relevance and sufficient evidence for answering is not directly established. In Appendix A.1.1, SigLIP2 provides cosine similarity, while BLIP provides the difference between matched and mismatched logits. The reference-bank normalization in Eqs. (6) and (9) measures deviation from a query-specific background; it does not by itself establish that the currently observed video contains enough evidence to answer. The subsequent accumulation can reinforce repeated, semantically related frames even when a required event or temporal relation has not yet been resolved. This is a potential failure mode that requires validation, rather than an observed failure demonstrated by the present review. The benchmark scores and threshold ablations provide indirect evidence, but leave unclear how often threshold crossings correspond to genuinely sufficient evidence rather than topical relevance. A targeted analysis of premature triggers, evidence availability, and response delay would substantiate the central interpretation of the accumulated state.

  2. The methodological advance over existing lightweight response mechanisms appears incremental and is not sufficiently isolated by the experiments. Separating response decisions from generation is already explored by Dispider [1] and StreamBridge [2]. More directly, QueryStream [3] provides a training-free response mechanism based on query relevance and information density. CIAct cites these directions and includes QueryStream in its comparison, so my concern is the distinction and evidence for the added mechanism, rather than missing citations. Its specific contribution appears to lie in combining background standardization, decayed accumulation, and relevance-based memory replay. However, Table 3 changes observation duration substantially along with the mechanism: on LongVideoBench, direct thresholding obtains 47.5 at 15% average viewing, whereas the complete method obtains 58.3 at 79% viewing. The threshold sweep for the complete method is useful, but does not replace comparisons between mechanisms at similar viewing budgets. Using the same relevance model, answering backbone, and memory selection would help determine whether accumulation adds a substantial benefit beyond existing relevance-triggered rules or simple smoothing.

The use of an external lightweight model is not itself a weakness, and the age of BLIP is not a sufficient reason to dismiss the approach. My concern is whether the proposed combination establishes a substantial methodological contribution and reliably supports the paper's evidence-sufficiency interpretation.

Overall assessment. I lean toward weak rejection. The framework offers useful practical efficiency, but the current evidence does not sufficiently validate the interpretation of accumulated relevance as answer-evidence sufficiency or isolate a substantial methodological gain over prior relevance-triggered approaches. Focused evidence on trigger reliability and comparisons under similar observation budgets could change my assessment.

References

[1] Rui Qian et al. Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction. CVPR 2025. https://openaccess.thecvf.com/content/CVPR2025/html/Qian_Dispider_Enabling_Video_LLMs_with_Active_Real-Time_Interaction_via_Disentangled_CVPR_2025_paper.html

[2] Haibo Wang et al. StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant. NeurIPS 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/bf6939f9058a391c47014731b2486e2a-Abstract-Conference.html

[3] Kairui Zhang et al. QueryStream: Advancing Streaming Video Understanding with Query-Aware Pruning and Proactive Response. ICLR 2026. https://proceedings.iclr.cc/paper_files/paper/2026/hash/0b17d256cf1fe1cc084922a8c6b565b7-Abstract-Conference.html

[4] Shuhang Xun et al. RTV-Bench: Benchmarking MLLM Continuous Perception, Understanding and Reasoning through Real-Time Video. NeurIPS 2025, Datasets and Benchmarks Track. https://papers.nips.cc/paper_files/paper/2025/hash/19e4ea30dded58259665db375885e412-Abstract-Datasets_and_Benchmarks_Track.html

[5] Kai Liu et al. JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation. NeurIPS 2025. https://proceedings.neurips.cc/paper_files/paper/2025/hash/d1422213c9f2bdd5178b77d166fba86a-Abstract-Conference.html

Questions for authors
  1. What evidence directly links the accumulated state or its threshold crossings to the availability of sufficient answer evidence? Could an existing annotated subset be used to characterize premature responses and response delay, particularly for repeated relevant scenes or questions requiring a completed event or temporal relation?

  2. Which aspect of the proposed accumulation addresses a limitation of prior relevance-triggered methods such as QueryStream? Are there results at similar observation budgets, with the same relevance model and memory selection, that separate the contribution of accumulation from additional viewing time? Please also clarify how the thresholds were selected and whether reference-bank calibration and historical rescoring are included in the reported decision costs.

Suggestions for broader evaluation. RTV-Bench [4] would complement OVO-Bench by testing questions whose answers evolve across timestamps. The evaluation should preserve causal access to frames and distinguish timestamp-specific answer quality from autonomous response timing. Testing another compatible answering backbone, such as the video-understanding pathway of JavisGPT [5] under the same visual-input budget, would also strengthen the plug-and-play claim. These are suggested extensions; equivalent benchmarks or backbones could serve the same purpose, and an entirely new large-scale evaluation is not required to answer the clarification questions above.

Reviewer_ZDzuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work focuses on the proactive responses for video multi-modal LLMs in a training-free manner. To achieve that, they proposed the Cumulative Information based Active response (CIAct) that accumulates the relevant information through question-visual feature similarity until it reaches a threshold to answer the question. Further, it selects important frames into a memory bank by jointly considering its query relevance and temporal proximity. As a result, it can reduce the visual memory recall and keep the computation efficient. The paper evaluated this method on one streaming benchmark and four offline benchmarks, and achieved good performance and less watching time.

Critical strengths

The strength of this paper is how they utilized the calibrated query–frame relevance information for both the cumulative information estimation and memory replay in the overall pipeline. These two modules are important for accurately determining when to answer and efficiently using important frames for question answering. By using calibrated query–frame relevance as a shared signal across both stages, the pipeline establishes a consistent framework that enables MLLMs to make better-informed decisions about when to answer and how to leverage important visual information for question answering.

Critical weaknesses

In order to achieve active response in a training-free manner, the pipeline has to manually select several hyper-parameters (e.g., θ, h, ωmin, in Appendix). These hyper-parameters has to be separately determined not only for different query–frame relevance models, but also for different benchmark types.

According to A.1.2 in Appendix, the response threshold θ (Eq. 7 in the main paper) is manually selected based on the different properties of streaming benchmarks (questions may arrive at arbitrary timestamps) and offline benchmarks (questions are provided at the beginning).

As a result, these thresholds are sensitive to specific question types, making them difficult to generalize and directly apply to broader real-world applications.

Questions for authors
  1. In cold start of CIAct, it replies on an external reference frame bank to estimate µQ and σQ. How is this external bank determined and how would different external bank affects the model's performance on different benchmarks?
  2. How are the θ, h, ωmin determined in the pipeline?
  3. Is there a straightforward and systematic way to determine these hyperparameters, ideally without extensive manual tuning, to facilitate their application across different tasks and real-world scenarios?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.