- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Latent Proactive Adapter (LatPA), a training-free method for determining when streaming Video-LLMs should proactively respond to user queries. Unlike existing approaches that rely on trained triggering modules or explicit yes/no generation, LatPA leverages intermediate hidden-state representations from a frozen Video-LLM to make triggering decisions. Specifically, it computes the cosine similarity between the question and the current frame representations and compares it with the similarity between the question and an exponential moving average of historical frame representations. A response is triggered when the current frame exhibits stronger question alignment than the historical reference, using a fixed zero threshold without additional training. Experiments on StreamingBench-PO, IPIBench, and OVO-Bench FAR demonstrate competitive performance across different backbone architectures, with particularly notable improvements on StreamingBench-PO using Qwen2-VL models. The paper also presents ablation studies and analyses of triggering behavior, memory configurations, and computational efficiency to further examine the effectiveness of the proposed approach.
Critical strengths
-
The paper is well-written, clearly structured, and easy to follow.
-
The proposed method is simple, intuitive, and training-free, providing a computationally efficient alternative to generation-based triggering approaches without requiring additional model training.
-
The paper presents comprehensive evaluations across multiple benchmarks and backbone architectures, complemented by extensive ablation studies and analyses that provide insights into the effectiveness and behavior of the proposed method.
Critical weaknesses
-
The proposed method assumes that an increase in question-frame similarity indicates an appropriate moment to trigger a response. However, this assumption may not generalize to more complex proactive tasks, where triggering decisions depend on temporal reasoning and task progression rather than direct question-frame alignment. For instance, tasks requiring the model to provide instructions for the next step may require understanding the current stage of an ongoing activity and anticipating subsequent actions, which may not be adequately captured by the proposed similarity-based criterion.
-
LatPA triggers a response whenever the current frame exhibits higher question similarity than the historical memory, regardless of its absolute relevance to the question. This relative comparison may lead to false-positive triggers, as even irrelevant frames could satisfy the triggering criterion when their similarity is marginally higher than that of the historical reference. Incorporating an additional absolute relevance threshold or analyzing the sensitivity of false-positive rates to similarity differences would help assess the robustness of the proposed triggering mechanism.
-
Although the proposed method is simple and intuitive, its technical novelty appears relatively limited, as it primarily combines standard cosine similarity with exponential moving average memory and a fixed decision threshold. The paper offers limited methodological or theoretical insights beyond this straightforward combination of existing techniques, making it difficult to assess the extent of its technical contribution.
-
Several key design choices, including the EMA-based memory mechanism and intermediate-layer selection, do not consistently achieve optimal performance across different backbone architectures. These backbone-dependent variations raise concerns about the robustness and generalizability of the proposed design. A more systematic investigation of how these configurations interact with different model architectures would strengthen the justification for adopting a unified triggering mechanism across backbones.
Questions for authors
N/A
2 comments
Response to Reviewer NFPQ (Part 2/2): W3, W4
W3: Although the proposed method is simple and intuitive, its technical novelty appears relatively limited, as it primarily combines standard cosine similarity with exponential moving average memory and a fixed decision threshold. The paper offers limited methodological or theoretical insights beyond this straightforward combination of existing techniques, making it difficult to assess the extent of its technical contribution.
We agree that the components of LatPA are relatively simple, which also allows it to transfer directly to new backbones without training. Beyond the method itself, the more important finding of this paper is that we also found that proactive triggering decisions can be obtained directly from latent-space signals of a frozen Video-LLM. In contrast, previous methods all rely on token generation, additionally trained modules, or external visual encoders. Table NFPQ.W3.1 replaces only the read-out feature under a fixed trigger rule: performance drops sharply with CLIP features or the yes/no score of the first output token, while intermediate-layer hidden states perform best. This suggests that the trigger signal at intermediate layers may contain more information, and also points to a direction for further exploiting hidden states.
Table NFPQ.W3.1 The same LatPA rule on different signals on Qwen2-VL-2B. Bold: best per column.
| Read-out | StreamingBench-PO Time | FAR Avg |
|---|---|---|
| CLIP ViT-L/14 | 16.8 | 7.54 |
| First output Token | 16.8 | 8.49 |
| Hidden state (LatPA) | 48.0 | 10.35 |
W4: Several key design choices, including the EMA-based memory mechanism and intermediate-layer selection, do not consistently achieve optimal performance across different backbone architectures. These backbone-dependent variations raise concerns about the robustness and generalizability of the proposed design. A more systematic investigation of how these configurations interact with different model architectures would strengthen the justification for adopting a unified triggering mechanism across backbones.
Thank you for pointing this out. We acknowledge that no single configuration is optimal on all backbones. The purpose of LatPA's fixed configuration is to be directly usable on a new backbone without any annotation. On this basis, we add systematic analyses in two aspects.
Memory form: Table NFPQ.W4.1 compares four memory forms, and EMA performs best in most settings.
Read-out layer: the appendix of the paper scans the read out depth layer by layer. On an independent validation set, using 0.5L uniformly can also reach 76–91% of the trigger signal strength of each backbone's best layer.
We will further discuss the choice of configuration in the next version, which is also a direction of our future improvement.
Table NFPQ.W4.1 Memory forms at 0.5L. Each cell: StreamingBench-PO Time / IPIBench Timing / FAR Avg. Cumulative mean: mean of all past polls. Sliding mean: mean of the frame vectors from the last 8 forward passes (Tab. 3 of the paper averages the 8 frames within one forward pass instead). Bold: best. Italic: second best per backbone and benchmark.
| Memory form | Qwen2-VL-2B | Qwen2-VL-7B | InternVL3-2B | InternVL3-8B |
|---|---|---|---|---|
| Previous frame | 16.0 / 34.17 / 9.24 | 18.4 / 24.26 / 9.17 | 16.4 / 26.89 / 9.85 | 11.6 / 29.77 / 9.56 |
| Cumulative mean | 9.6 / 2.61 / 7.18 | 13.6 / 4.83 / 7.00 | 31.2 / 39.58 / 9.61 | 35.2 / 27.77 / 9.06 |
| Sliding mean | 42.0 / 6.27 / 9.31 | 40.4 / 8.32 / 8.94 | 35.2 / 42.48 / 9.38 | 42.0 / 32.21 / 9.11 |
| EMA (ours) | 48.0 / 42.12 / 10.35 | 46.4 / 41.35 / 10.70 | 28.8 / 51.81 / 10.76 | 29.2 / 49.63 / 10.49 |
Thank you for your time and effort. If our responses have resolved your concerns, we would greatly appreciate it if you could reconsider the rating of our paper. We are also very happy to continue discussing other questions.
Response to Reviewer NFPQ (Part 1/2): W1, W2
Thank you for your careful review of our paper and your pertinent suggestions. We respond to each of the points you raised in turn below.
Weaknesses
Thank you for the comment. We would like to clarify that LatPA does not directly compare the alignment between a single frame and the question. It reads the hidden state encoded by an intermediate LLM layer in the context of the full frame window, and uses the historically accumulated EMA memory as the reference. Therefore, questions that integrate evidence across multiple frames do not cause LatPA to fail. Table NFPQ.W1.1 splits the questions into two groups according to the annotations provided with StreamingBench-PO: single-frame questions can be judged from a single frame, while multi-frame questions require consecutive frames to judge an action or process. On all four backbones, the scores on multi-frame questions are higher than on single-frame questions.
Table NFPQ.W1.1 StreamingBench-PO Time on single-frame and multi-frame questions. Bold: higher per backbone.
Thank you for the suggestions. In Table NFPQ.W2.1, we carried out an analysis for each of the two suggestions. First, we add an absolute relevance gate on top of St>0, which requires cos(q,nt) to be no lower than its median over all polls of that backbone. Second, we gradually raise the threshold from 0 to τ and observe how the proportion of wrong first fires changes.
The absolute gate provides limited help for decision time: it drops on Qwen and improves little on InternVL. While adjusting the margin works better. As τ increases from 0 to 0.02, the proportion of wrong first fires keeps decreasing on all four backbones. For τ≤0.005, Time does not drop, and InternVL3-2B even improves by 8.4. For larger τ, Time on Qwen starts to drop, and Time on InternVL also falls back after τ=0.01. The appendix of the paper shows that Qwen prefers lower thresholds on IPIBench. We therefore use τ=0 by default, which does not favor any single benchmark, and treat the margin as a per-task tunable option. We will add this analysis to the revised version.
Table NFPQ.W2.1 StreamingBench-PO Time / wrong first fires (%). Bold: best Time per column.