acceptodds
Under review as a conference paper at ICLR 2027

LatPA: A Training-Free Latent Proactive Adapter for Streaming Video-LLMs

Abstract

Streaming video assistants must decide when to act, not just what to answer. Existing proactive triggers typically obtain this decision from an additional trained module or a yes/no query to the backbone. This leaves open how to obtain a temporal decision directly from the frozen model's representations. To this end, we introduce LatPA (Latent Proactive Adapter), a training-free alternative that extracts latent signals from a frozen Video-LLM's intermediate hidden states. The key idea is to use the video's own history as the reference for the current observation. LatPA compares the question alignment of the current frame with that of an exponential memory of preceding frames and acts when the resulting score is positive. This self-referenced rule requires neither an additional learned scoring module nor yes/no generation, and uses a shared configuration across backbones and benchmarks. Experiments on StreamingBench-PO show that LatPA improves average timing-and-answer accuracy over the evaluated learned and queried triggers with Qwen2-VL backbones. Across the evaluated model families and scales, LatPA consistently outperforms the evaluated queried triggers on the average scores of StreamingBench-PO and IPIBench. These results demonstrate that a video-specific historical reference enables effective proactive timing directly from frozen latent representations. Code is available at https://anonymous.4open.science/r/LatPA-D405.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Video Question Answering, Proactive Decision Making, Streaming Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_NFPQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Latent Proactive Adapter (LatPA), a training-free method for determining when streaming Video-LLMs should proactively respond to user queries. Unlike existing approaches that rely on trained triggering modules or explicit yes/no generation, LatPA leverages intermediate hidden-state representations from a frozen Video-LLM to make triggering decisions. Specifically, it computes the cosine similarity between the question and the current frame representations and compares it with the similarity between the question and an exponential moving average of historical frame representations. A response is triggered when the current frame exhibits stronger question alignment than the historical reference, using a fixed zero threshold without additional training. Experiments on StreamingBench-PO, IPIBench, and OVO-Bench FAR demonstrate competitive performance across different backbone architectures, with particularly notable improvements on StreamingBench-PO using Qwen2-VL models. The paper also presents ablation studies and analyses of triggering behavior, memory configurations, and computational efficiency to further examine the effectiveness of the proposed approach.

Critical strengths
  1. The paper is well-written, clearly structured, and easy to follow.

  2. The proposed method is simple, intuitive, and training-free, providing a computationally efficient alternative to generation-based triggering approaches without requiring additional model training.

  3. The paper presents comprehensive evaluations across multiple benchmarks and backbone architectures, complemented by extensive ablation studies and analyses that provide insights into the effectiveness and behavior of the proposed method.

Critical weaknesses
  1. The proposed method assumes that an increase in question-frame similarity indicates an appropriate moment to trigger a response. However, this assumption may not generalize to more complex proactive tasks, where triggering decisions depend on temporal reasoning and task progression rather than direct question-frame alignment. For instance, tasks requiring the model to provide instructions for the next step may require understanding the current stage of an ongoing activity and anticipating subsequent actions, which may not be adequately captured by the proposed similarity-based criterion.

  2. LatPA triggers a response whenever the current frame exhibits higher question similarity than the historical memory, regardless of its absolute relevance to the question. This relative comparison may lead to false-positive triggers, as even irrelevant frames could satisfy the triggering criterion when their similarity is marginally higher than that of the historical reference. Incorporating an additional absolute relevance threshold or analyzing the sensitivity of false-positive rates to similarity differences would help assess the robustness of the proposed triggering mechanism.

  3. Although the proposed method is simple and intuitive, its technical novelty appears relatively limited, as it primarily combines standard cosine similarity with exponential moving average memory and a fixed decision threshold. The paper offers limited methodological or theoretical insights beyond this straightforward combination of existing techniques, making it difficult to assess the extent of its technical contribution.

  4. Several key design choices, including the EMA-based memory mechanism and intermediate-layer selection, do not consistently achieve optimal performance across different backbone architectures. These backbone-dependent variations raise concerns about the robustness and generalizability of the proposed design. A more systematic investigation of how these configurations interact with different model architectures would strengthen the justification for adopting a unified triggering mechanism across backbones.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer NFPQ (Part 1/2): W1, W2

Thank you for your careful review of our paper and your pertinent suggestions. We respond to each of the points you raised in turn below.

Weaknesses

W1: The proposed method assumes that an increase in question-frame similarity indicates an appropriate moment to trigger a response. However, this assumption may not generalize to more complex proactive tasks, where triggering decisions depend on temporal reasoning and task progression rather than direct question-frame alignment. For instance, tasks requiring the model to provide instructions for the next step may require understanding the current stage of an ongoing activity and anticipating subsequent actions, which may not be adequately captured by the proposed similarity-based criterion.

Thank you for the comment. We would like to clarify that LatPA does not directly compare the alignment between a single frame and the question. It reads the hidden state encoded by an intermediate LLM layer in the context of the full frame window, and uses the historically accumulated EMA memory as the reference. Therefore, questions that integrate evidence across multiple frames do not cause LatPA to fail. Table NFPQ.W1.1 splits the questions into two groups according to the annotations provided with StreamingBench-PO: single-frame questions can be judged from a single frame, while multi-frame questions require consecutive frames to judge an action or process. On all four backbones, the scores on multi-frame questions are higher than on single-frame questions.

Table NFPQ.W1.1 StreamingBench-PO Time on single-frame and multi-frame questions. Bold: higher per backbone.

SubsetQwen2-VL-2BQwen2-VL-7BInternVL3-2BInternVL3-8B
Single-frame45.144.626.328.0
Multi-frame54.750.734.732.0

W2: LatPA triggers a response whenever the current frame exhibits higher question similarity than the historical memory, regardless of its absolute relevance to the question. This relative comparison may lead to false-positive triggers, as even irrelevant frames could satisfy the triggering criterion when their similarity is marginally higher than that of the historical reference. Incorporating an additional absolute relevance threshold or analyzing the sensitivity of false-positive rates to similarity differences would help assess the robustness of the proposed triggering mechanism.

Thank you for the suggestions. In Table NFPQ.W2.1, we carried out an analysis for each of the two suggestions. First, we add an absolute relevance gate on top of St>0S_t > 0, which requires cos⁡(q,nt)\cos(q, n_t) to be no lower than its median over all polls of that backbone. Second, we gradually raise the threshold from 0 to τ\tau and observe how the proportion of wrong first fires changes.

The absolute gate provides limited help for decision time: it drops on Qwen and improves little on InternVL. While adjusting the margin works better. As τ\tau increases from 0 to 0.02, the proportion of wrong first fires keeps decreasing on all four backbones. For τ≤0.005\tau \le 0.005, Time does not drop, and InternVL3-2B even improves by 8.4. For larger τ\tau, Time on Qwen starts to drop, and Time on InternVL also falls back after τ=0.01\tau = 0.01. The appendix of the paper shows that Qwen prefers lower thresholds on IPIBench. We therefore use τ=0\tau = 0 by default, which does not favor any single benchmark, and treat the margin as a per-task tunable option. We will add this analysis to the revised version.

Table NFPQ.W2.1 StreamingBench-PO Time / wrong first fires (%). Bold: best Time per column.

RuleQwen2-VL-2BQwen2-VL-7BInternVL3-2BInternVL3-8B
LatPA+abs32.0 / 54.539.2 / 49.228.0 / 68.332.0 / 64.4
St>0.002S_t > 0.00248.8 / 47.446.8 / 49.834.0 / 65.932.0 / 67.5
St>0.005S_t > 0.00548.0 / 46.747.2 / 46.637.2 / 62.033.6 / 65.0
St>0.01S_t > 0.0142.8 / 46.241.6 / 45.037.2 / 57.340.0 / 57.4
St>0.02S_t > 0.0235.6 / 45.431.2 / 40.534.0 / 51.436.4 / 52.6
LatPA48.0 / 49.846.4 / 51.528.8 / 71.129.2 / 70.4
AuthorsOpenReview ↗

Response to Reviewer NFPQ (Part 2/2): W3, W4

W3: Although the proposed method is simple and intuitive, its technical novelty appears relatively limited, as it primarily combines standard cosine similarity with exponential moving average memory and a fixed decision threshold. The paper offers limited methodological or theoretical insights beyond this straightforward combination of existing techniques, making it difficult to assess the extent of its technical contribution.

We agree that the components of LatPA are relatively simple, which also allows it to transfer directly to new backbones without training. Beyond the method itself, the more important finding of this paper is that we also found that proactive triggering decisions can be obtained directly from latent-space signals of a frozen Video-LLM. In contrast, previous methods all rely on token generation, additionally trained modules, or external visual encoders. Table NFPQ.W3.1 replaces only the read-out feature under a fixed trigger rule: performance drops sharply with CLIP features or the yes/no score of the first output token, while intermediate-layer hidden states perform best. This suggests that the trigger signal at intermediate layers may contain more information, and also points to a direction for further exploiting hidden states.

Table NFPQ.W3.1 The same LatPA rule on different signals on Qwen2-VL-2B. Bold: best per column.

Read-outStreamingBench-PO TimeFAR Avg
CLIP ViT-L/1416.87.54
First output Token16.88.49
Hidden state (LatPA)48.010.35

W4: Several key design choices, including the EMA-based memory mechanism and intermediate-layer selection, do not consistently achieve optimal performance across different backbone architectures. These backbone-dependent variations raise concerns about the robustness and generalizability of the proposed design. A more systematic investigation of how these configurations interact with different model architectures would strengthen the justification for adopting a unified triggering mechanism across backbones.

Thank you for pointing this out. We acknowledge that no single configuration is optimal on all backbones. The purpose of LatPA's fixed configuration is to be directly usable on a new backbone without any annotation. On this basis, we add systematic analyses in two aspects.

Memory form: Table NFPQ.W4.1 compares four memory forms, and EMA performs best in most settings.

Read-out layer: the appendix of the paper scans the read out depth layer by layer. On an independent validation set, using 0.5L uniformly can also reach 76–91% of the trigger signal strength of each backbone's best layer.

We will further discuss the choice of configuration in the next version, which is also a direction of our future improvement.

Table NFPQ.W4.1 Memory forms at 0.5L. Each cell: StreamingBench-PO Time / IPIBench Timing / FAR Avg. Cumulative mean: mean of all past polls. Sliding mean: mean of the frame vectors from the last 8 forward passes (Tab. 3 of the paper averages the 8 frames within one forward pass instead). Bold: best. Italic: second best per backbone and benchmark.

Memory formQwen2-VL-2BQwen2-VL-7BInternVL3-2BInternVL3-8B
Previous frame16.0 / 34.17 / 9.2418.4 / 24.26 / 9.1716.4 / 26.89 / 9.8511.6 / 29.77 / 9.56
Cumulative mean9.6 / 2.61 / 7.1813.6 / 4.83 / 7.0031.2 / 39.58 / 9.6135.2 / 27.77 / 9.06
Sliding mean42.0 / 6.27 / 9.3140.4 / 8.32 / 8.9435.2 / 42.48 / 9.3842.0 / 32.21 / 9.11
EMA (ours)48.0 / 42.12 / 10.3546.4 / 41.35 / 10.7028.8 / 51.81 / 10.7629.2 / 49.63 / 10.49

Thank you for your time and effort. If our responses have resolved your concerns, we would greatly appreciate it if you could reconsider the rating of our paper. We are also very happy to continue discussing other questions.

Reviewer_jMAyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses proactive response timing in streaming Video-LLMs, where a model must determine when to respond to a pending question as a video unfolds. The authors propose LatPA, a training-free adapter that extracts a decision signal from the intermediate hidden states of a frozen Video-LLM. Specifically, LatPA compares the question alignment of the current frame with that of an exponential moving average of preceding frames, triggering a response when the difference becomes positive.

Unlike approaches requiring an additional trained trigger or repeated yes/no generation, LatPA uses a lightweight latent-space decision rule without modifying the backbone. The method is evaluated on StreamingBench-PO, IPIBench, OVO-Bench FAR, and ProactiveVideoQA, across multiple Video-LLM backbones. The reported results demonstrate competitive proactive timing performance, particularly against learned and queried trigger baselines, and the ablation studies investigate the contributions of historical memory and the decision rule.

Critical strengths
  1. A simple and practically useful training-free solution to proactive timing. The paper identifies an important limitation of streaming Video-LLMs: generating a correct answer is insufficient if the model cannot determine when to respond. LatPA addresses this problem using representations already available within a frozen backbone, avoiding additional training and explicit yes/no generation. This makes the approach attractive for adapting existing Video-LLMs to proactive applications with minimal architectural changes.

  2. Strong empirical results across different model families and proactive evaluation settings. The evaluation includes multiple backbones and benchmarks covering response timing, answer quality, and longer-horizon interactions. On StreamingBench-PO, the reported improvements over the strongest evaluated learned or queried trigger reach 18.8 and 8.8 percentage points for Qwen2-VL-2B and Qwen2-VL-7B, respectively. The use of a shared operating configuration across backbones is particularly encouraging, as it suggests that the method does not depend on extensive model-specific calibration.

Critical weaknesses
  1. The methodological novelty is relatively limited. The central mechanism combines cosine similarity, exponential moving average memory, and a zero-threshold decision rule. Although effective, these components are individually standard, and the paper's primary contribution is their application to proactive VideoLLM triggering. The work would be more compelling if it established more clearly why this particular latent-space formulation captures temporal answerability better than alternative similarity-based or change-detection approaches.

  2. The relationship between the trigger score and actual answerability is not fully established. A positive change in question-frame alignment may indicate that the current observation is more relevant than preceding observations, but it does not necessarily imply that sufficient information is available to answer the question correctly. This distinction is especially important for events requiring accumulated evidence, temporal reasoning, or delayed confirmation. Furthermore, the comparative benefits of different memory mechanisms appear to vary across model families, suggesting that the shared configuration may not always be optimal.

Questions for authors

How does LatPA distinguish an increase in question relevance from actual answerability? In particular, can the authors analyze cases where a relevant event begins but sufficient evidence for a correct answer becomes available only later?

The ablation results suggest that alternative memory aggregation strategies can outperform EMA on some backbones. Can the authors clarify why EMA is selected as the default and quantify the performance trade-off between using one shared configuration and selecting the best configuration for each backbone?

2 comments
AuthorsOpenReview ↗

Response to Reviewer jMAy (Part 1/2): Q1, Q2

Thank you for taking the time to review our paper and for your constructive comments. Below are our responses to each question.

Questions

Q1: How does LatPA distinguish an increase in question relevance from actual answerability? In particular, can the authors analyze cases where a relevant event begins but sufficient evidence for a correct answer becomes available only later?

We strongly agree with the reviewer's point that St>0S_t>0 reflects an increase in relevance rather than sufficient evidence, and LatPA itself does not explicitly distinguish the two. For the case where "the event has started but the evidence becomes complete only later", in Table jMAy.Q1.1 we keep the trigger time fixed and delay answer generation by kk seconds, and report the answer-only accuracy (Answer), the fraction with correct timing after the delay (Time), and the Understanding score. The table shows that for InternVL3-2B, delaying the answer makes it more accurate, but Time gradually moves away from the ground truth. The combined Understanding score has room for a small increase, so in this case the situation described by the reviewer does occur.

However, when the trigger is delayed, the model has already seen more visual information, which is itself a trade-off between timeliness and accuracy. Because the evaluation requires the trigger and the answer to happen at the same moment, under this setting the current results of LatPA represent a balanced solution.

Table jMAy.Q1.1 IPIBench Understanding with the answer generated kk s after LatPA fires.

BackboneAnswer, kk = 0 / 2 / 4Time, kk = 0 / 2 / 4Understanding, kk = 0 / 2 / 4
Qwen2-VL-2B28.68 / 27.20 / 21.4255.17 / 24.75 / 2.2222.91 / 9.48 / 0.81
InternVL3-2B7.28 / 12.98 / 15.6856.20 / 54.27 / 21.104.72 / 6.54 / 1.91

Q2: The ablation results suggest that alternative memory aggregation strategies can outperform EMA on some backbones. Can the authors clarify why EMA is selected as the default and quantify the performance trade-off between using one shared configuration and selecting the best configuration for each backbone?

Thank you for the careful observation. In the ablation, Cumulative mean indeed outperforms EMA in some settings. We therefore ran a more complete ablation, which shows that EMA is the best in most settings, and thus we choose EMA as the default memory form.

Table jMAy.Q2.1 Memory forms at 0.5L. Each cell: StreamingBench-PO Time / IPIBench Timing / FAR Avg. Cumulative mean: mean of all past polls. Sliding mean: mean of the frame vectors from the last 8 forward passes (Tab. 3 of the paper averages the 8 frames within one forward pass instead). Bold: best. Italic: second best per backbone and benchmark.

Memory formQwen2-VL-2BQwen2-VL-7BInternVL3-2BInternVL3-8B
Previous frame16.0 / 34.17 / 9.2418.4 / 24.26 / 9.1716.4 / 26.89 / 9.8511.6 / 29.77 / 9.56
Cumulative mean9.6 / 2.61 / 7.1813.6 / 4.83 / 7.0031.2 / 39.58 / 9.6135.2 / 27.77 / 9.06
Sliding mean42.0 / 6.27 / 9.3140.4 / 8.32 / 8.9435.2 / 42.48 / 9.3842.0 / 32.21 / 9.11
EMA (ours)48.0 / 42.12 / 10.3546.4 / 41.35 / 10.7028.8 / 51.81 / 10.7629.2 / 49.63 / 10.49

Regarding the cost of a shared configuration, Figure 5 of the paper already has related experiments showing that the best layer differs across models. For the test results specifically, in Table jMAy.Q2.2 we only change the read-out layer for each backbone and report the gap between the shared 0.5L configuration and the best layer for each benchmark. The table shows that the shared configuration is not optimal in some settings and only has little impact on the average performance.

Table jMAy.Q2.2 Shared 0.5L layer against the best layer.

BackboneStreamingBench-PO TimeIPIBench TimingFAR AvgAvg
Qwen2-VL-2B0.0−3.19−0.95−1.38
Qwen2-VL-7B0.0−4.46−0.49−1.65
InternVL3-2B−9.6−0.90−0.22−3.57
InternVL3-8B−10.4−0.10−0.55−3.68
AuthorsOpenReview ↗

Response to Reviewer jMAy (Part 2/2): W1, W2

Weaknesses

W1: The methodological novelty is relatively limited. The central mechanism combines cosine similarity, exponential moving average memory, and a zero-threshold decision rule. Although effective, these components are individually standard, and the paper's primary contribution is their application to proactive VideoLLM triggering. The work would be more compelling if it established more clearly why this particular latent-space formulation captures temporal answerability better than alternative similarity-based or change-detection approaches.

Thank you for the comment. The formulation of LatPA is indeed relatively simple, but we believe its value lies in proposing a different source of decision signals, namely reading the trigger signal directly from the intermediate-layer hidden states of a frozen Video-LLM. Compared with existing methods, this design differs in three ways. First, compared with similarity methods based on external encoders (e.g., CLIP), the question tokens in LatPA have already interacted with all frames inside the backbone, yielding question-conditioned representations, whereas in a dual-tower encoder the question and the frames only meet at the final cosine similarity. Meanwhile, LatPA needs no extra encoder and can be transferred directly when switching to a new backbone. Second, compared with change detection that does not use the question, LatPA is conditioned on the question and can give different trigger times for different questions on the same video, whereas pure change detection gives the same trigger for all questions. Third, compared with trained activation heads or queried triggers that repeatedly generate yes/no, LatPA requires neither training nor decoding and uses St>0S_t>0 as a decision rule without threshold calibration. Table 1 of the paper already compares against these two types of methods on multiple backbones. Therefore, although cosine similarity, EMA, and a zero threshold are each common components, placing them in the latent space for a question-conditioned relative comparison is the key that allows LatPA to be used across backbones without training or threshold tuning.

W2: The relationship between the trigger score and actual answerability is not fully established. A positive change in question-frame alignment may indicate that the current observation is more relevant than preceding observations, but it does not necessarily imply that sufficient information is available to answer the question correctly. This distinction is especially important for events requiring accumulated evidence, temporal reasoning, or delayed confirmation. Furthermore, the comparative benefits of different memory mechanisms appear to vary across model families, suggesting that the shared configuration may not always be optimal.

Thank you for the constructive comments. On the relationship between the trigger score and answerability, the delayed-answer experiment in Q1 (Table jMAy.Q1.1) shows that delaying makes the answers of backbones that tend to fire early more accurate, but misses the response time, which is a trade-off between timeliness and accuracy. For events that require accumulated evidence, LatPA scores higher on multi-frame evidence questions than on single-frame questions. For delayed confirmation, a rule that responds only after St>0S_t>0 holds several times in a row cannot bring improvements on all backbones at the same time, so we keep the single-fire trigger. On the differences among memory mechanisms, Table jMAy.Q2.1 shows that EMA is the best in most settings, and Table jMAy.Q2.2 shows that the shared configuration has little impact on the average performance.

Thank you again for your comments and support. If you have any other questions, we are happy to answer them further.

Reviewer_tHbUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses proactive response timing for streaming Video-LLMs. The authors argue that existing triggers require trained modules or repeated yes/no queries, leaving the frozen backbone's internal representations underused for detecting when question-relevant evidence appears. To address this, the paper proposes LatPA, a training-free adapter that (i) mean-pools intermediate hidden states into question and current-frame representations, (ii) maintains an exponential moving average of past frame representations, and (iii) triggers a response when the current frame becomes more aligned with the question than the historical reference, using a zero threshold. A shared read-out depth, memory decay, and threshold are used across models and benchmarks without benchmark-specific tuning. Experiments evaluate Qwen2-VL-2B/7B and InternVL3-2B/8B on StreamingBench-PO, IPIBench, and OVO-Bench FAR, with an additional ProactiveVideoQA evaluation on Qwen2-VL-2B. On StreamingBench-PO, LatPA scores 43.8/42.8 with Qwen2-VL-2B/7B, outperforming StreamBridge (2B) and the ProactiveVideoQA Gate (7B) by 18.8/8.8 points, respectively, although its performance on OVO-Bench FAR and ProactiveVideoQA is weaker.

Critical strengths
  1. The historical comparison in Eq. (4) gives the method a clear motivation: detecting when relevant content first appears, rather than when it remains visible. LatPA requires no additional training or yes/no decoding. In Tab. 1, it achieves 43.8/42.8 on StreamingBench-PO with Qwen2-VL-2B/7B, compared with 25.0 (StreamBridge) and 34.0 (ProactiveVideoQA Gate), the strongest competing trigger on each backbone. Even if the components are standard, these gains make the training-free approach worth considering.

  2. The paper also reports unfavorable results rather than showing only successful cases.

Critical weaknesses
  1. The paper frames training-free proactive triggering from frozen Video-LLM representations as its main contribution, yet prior methods (QueryStream, ICLR 2026; Response-G1, ACL 2026; TimeChat-Online, ACM MM 2025) already use query relevance, temporal change, or both for response timing. QueryStream, in particular, combines query relevance with novelty measured against an exponentially smoothed history. LatPA instead reads intermediate hidden states and compares current question alignment against an EMA of past frames. Yet Tab. 1 only compares a trained activation head and two queried triggers, without evaluating these closer history-aware approaches. The novelty of training-free historical triggering is therefore overstated, while the advantage of the proposed latent read-out over existing methods remains untested.

  2. Eq. (4) is presented as a question-dependent trigger, but the controls in Appendix D.5 cast doubt on whether the question actually matters. In Tab. 8, Qwen2-VL-7B scores 46.8 with both matched and permuted questions, while InternVL3-2B even improves from 28.8 to 32.0 after permutation. Across four backbones, matched questions perform no better on average (matched minus permuted: −0.2/−1.5 points under the two prompt orders). Replacing question vectors with their shared direction also reproduces 94–95% of score variance. These results do not establish that LatPA selects events based on the user's question rather than general visual changes. Since question-dependent triggering is the paper's central mechanism, this is the main reason for my recommendation. The latent read-out itself is also not clearly justified: on the constructed set, a token-level EMA control achieves higher onset AUROC (0.92 vs. 0.86, Appendix D.1).

Questions for authors

Questions

  1. Could you further evaluate whether LatPA's triggering decisions depend on the question rather than general visual changes? In particular, does the response timing change when different questions are asked about the same video?

  2. Since LatPA is training-free, could you compare its performance on newer backbones (e.g., Qwen3-VL, InternVL3.5) against the backbone's own yes/no triggering under matched polling intervals? Does the improvement persist as the backbone becomes stronger?

  3. Could you compare LatPA with existing proactive triggering methods (e.g., QueryStream, Response-G1, TimeChat-Online) under matched evaluation settings to better establish its incremental contribution? In particular, how does it differ from QueryStream's combination of query relevance and EMA-referenced novelty?

  4. The Random baseline mentioned in Secs. 3.3 and 4.1 is missing from Tab. 1. Could you include content-blind timing baselines (e.g., uniform random, rate-matched random, and fixed-delay triggers) to clarify whether the gains come from video evidence rather than timing priors?

If you address my concerns, I would like to raise my score.

5 comments
AuthorsOpenReview ↗

Response to Reviewer tHbU (Part 1/4): Q1, Q2

We sincerely thank you for your careful review and insightful comments. We respond to the points you raised one by one as follows.

Questions

Q1: Could you further evaluate whether LatPA's triggering decisions depend on the question rather than general visual changes? In particular, does the response timing change when different questions are asked about the same video?

We conducted in-depth experiments. In Table tHbU.Q1.1, after replacing the question vector with a random direction, all metrics drop on all four backbones.

Table tHbU.Q1.1 Question vector replaced by a random direction. Bold: better per backbone.

BackboneMethodStreamingBench-PO TimeIPIBench TimingIPIBench RepeatedIPIBench UndFAR Avg
Qwen2-VL-2BLatPA (w/ random query)11.612.300.003.018.76
LatPA (ours)48.042.1222.6422.9110.35
Qwen2-VL-7BLatPA (w/ random query)5.613.779.430.968.59
LatPA (ours)46.441.3523.5810.8310.70
InternVL3-2BLatPA (w/ random query)20.41.020.003.488.89
LatPA (ours)28.851.8120.754.7210.76
InternVL3-8BLatPA (w/ random query)18.81.8014.152.758.75
LatPA (ours)29.249.6321.708.2110.49

After we also replaced each StreamingBench-PO question with the other 4 questions of the same video, 17.1% of the first trigger times of Qwen2-VL-2B changed accordingly, and StreamingBench-PO Time dropped from 48.0 to 45.5, which shows that the response timing changes with the question and is more accurate when it matches the original question.

Meanwhile, we also found that the prompt questions in many videos are too similar in meaning. For example, almost all StreamingBench-PO questions use the "When …, output …" template, which causes the semantic direction to change only slightly after swapping questions. This is also the problem of insufficient difficulty in many existing proactive benchmarks, and we are also working on a harder benchmark that requires more reasoning and analysis ability from the model to answer the original question.

Q2: Since LatPA is training-free, could you compare its performance on newer backbones (e.g., Qwen3-VL, InternVL3.5) against the backbone's own yes/no triggering under matched polling intervals? Does the improvement persist as the backbone becomes stronger?

Table tHbU.Q2.1 compares LatPA with the backbone's own yes/no triggers on Qwen3-VL-4B and InternVL3.5-2B, including the ProactiveVideoQA Gate at 1 s and 2 s granularity, and a RIVER style yes/no trigger that we re-implemented following the RIVER paper. LatPA achieves the highest Avg on both backbones. On InternVL3.5-2B, its Avg is about twice that of the best yes/no trigger, and on the stronger Qwen3-VL-4B, LatPA still remains in the lead.

Table tHbU.Q2.1 LatPA against the backbone's own yes/no trigger on two new backbones. Bold: best. Italic: second best per backbone.

BackboneMethodStreamingBench-PO (Time / Ans)IPIBench (Timing / Rep. / Und)FAR (REC / SSR / CRR)Avg
Qwen3-VL-4BYes/no (PVQA Gate, 1 s)42.4 / 42.43.22 / 1.42 / 24.8928.78 / 6.74 / 2.7621.67
Yes/no (PVQA Gate, 2 s)31.2 / 31.20.33 / 0.47 / 3.1538.53 / 12.05 / 3.6416.86
Yes/no (RIVER style)37.2 / 37.233.85 / 0.47 / 16.7314.73 / 4.70 / 3.0720.57
LatPA (ours)52.4 / 51.611.81 / 2.83 / 3.5623.64 / 8.80 / 0.6423.03
InternVL3.5-2BYes/no (PVQA Gate, 1 s)2.4 / 2.030.07 / 12.74 / 5.9720.83 / 6.69 / 4.329.69
Yes/no (PVQA Gate, 2 s)7.2 / 6.812.02 / 0.47 / 0.0025.88 / 9.41 / 4.178.11
Yes/no (RIVER style)0.8 / 0.819.97 / 0.94 / 1.4311.76 / 4.28 / 2.134.77
LatPA (ours)30.4 / 26.441.98 / 15.57 / 1.0324.27 / 9.26 / 0.6819.78
AuthorsOpenReview ↗

Response to Reviewer tHbU (Part 2/4): Q3

Q3: Could you compare LatPA with existing proactive triggering methods (e.g., QueryStream, Response-G1, TimeChat-Online) under matched evaluation settings to better establish its incremental contribution? In particular, how does it differ from QueryStream's combination of query relevance and EMA-referenced novelty?

Compared with QueryStream's RTAR, LatPA differs in three ways. 1. LatPA directly reads the hidden states inside the VLM and needs no external encoder. Table tHbU.Q3.1 shows that hidden states give a higher Avg than CLIP features under both rules. 2. LatPA fires on S_t > 0 and needs no threshold calibration, and no tuning is required when switching to a new backbone, whereas the two thresholds of RTAR need to be re-calibrated for the encoder and data. 3. The EMA in QueryStream (RTAR) is only used to judge whether each patch has changed in appearance, i.e., by comparing it with the historical EMA at the same position, and then counting the proportion of patches that are both question-relevant and changed. In contrast, the EMA in LatPA is the reference of the relevance comparison itself, directly comparing how well the current frame and the history align with the question direction.

Therefore, we conducted the following experiment: Table tHbU.Q3.1 cross-compares two features and two rules on Qwen2-VL-2B, where the thresholds required by RTAR are tuned to the optimum on the test set for each entry, while LatPA is not tuned. It can be seen that the combination of hidden states and the LatPA rule is the best on all metrics, and after RTAR switches to hidden states, its Avg also rises from 15.23 to 18.85.

Table tHbU.Q3.1 Feature source × decision rule on Qwen2-VL-2B. Bold: best. Italic: second best.

FeatureRuleStreamingBench-PO TimeIPIBench TimingFAR AvgAvg
CLIP ViT-L/14RTAR16.420.318.9915.23
CLIP ViT-L/14LatPA16.815.657.5413.33
Hidden 0.5LRTAR13.634.188.7818.85
Hidden 0.5LLatPA48.042.1210.3533.49

In addition, we also tested a full method comparison: Table tHbU.Q3.2 follows the polling and scoring of Tab. 1 in the paper and compares existing proactive triggering methods (QueryStream's RTAR is our re-implementation based on its official code following the formulas in its original paper, with thresholds searched on 10% of the FAR videos and then fixed, so its results are lower than the tuned upper bound in Table tHbU.Q3.1). Response-G1 is ported to the same Qwen2-VL-2B as LatPA following its public code, and TimeChat-Online uses its public 7B weights.

Table tHbU.Q3.2 Comparison with existing proactive triggers. Bold: best. Italic: second best.

MethodStreamingBench-PO (Time / Ans)IPIBench (Timing / Rep. / Und)FAR (REC / SSR / CRR)Avg
RTAR, CLIP2.0 / 1.614.16 / 1.89 / 2.795.83 / 9.75 / 1.684.61
RTAR, hidden4.0 / 2.421.72 / 2.36 / 4.6315.11 / 8.13 / 0.556.90
Response-G112.4 / 11.619.82 / 2.83 / 4.7010.88 / 3.98 / 0.728.77
TimeChat-Online (7B)23.2 / 21.629.24 / 5.19 / 2.5820.99 / 11.82 / 0.6815.30
LatPA (ours)48.0 / 39.642.12 / 22.64 / 22.9121.95 / 8.50 / 0.6127.79
AuthorsOpenReview ↗

Response to Reviewer tHbU (Part 3/4): Q4

Q4: The Random baseline mentioned in Secs. 3.3 and 4.1 is missing from Tab. 1. Could you include content-blind timing baselines (e.g., uniform random, rate-matched random, and fixed-delay triggers) to clarify whether the gains come from video evidence rather than timing priors?

We apologize for the misunderstanding caused by the wording in Secs. 3.3 and 4.1. On StreamingBench-PO and IPIBench, the original Random uniformly selected one trigger within the polling range (on FAR, it fired at each poll with probability 1/5), while the polling ranges of the former two depend on the GT, which amounts to indirectly using the GT location. Therefore, we removed it from Tab. 1 before submission, but the main text was not modified accordingly. Table tHbU.Q4.1 adds three timing baselines that do not depend on video content: the two random triggers depend only on past polls, and the fixed delay uses 13 s, taken from the median delay over all StreamingBench-PO and IPIBench questions. On all four backbones, LatPA is higher than these three baselines on StreamingBench-PO, IPIBench Timing, and Repeated.

Table tHbU.Q4.1 Content-blind baselines. Uniform random: each poll fires with probability 0.5. Rate-matched random: the same coin at LatPA's per-poll fire rate. Bold: best. Italic: second best per backbone.

BackboneMethodStreamingBench-PO (Time / Ans)IPIBench (Timing / Rep. / Und)FAR (REC / SSR / CRR)Avg
Qwen2-VL-2BUniform random8.8 / 8.812.29 / 0.47 / 3.6818.00 / 7.31 / 0.487.63
Rate-matched random20.0 / 17.625.49 / 4.25 / 11.1322.98 / 10.19 / 0.6514.56
Fixed delay8.8 / 5.67.31 / 0.00 / 3.1514.11 / 3.21 / 0.005.49
LatPA (ours)48.0 / 39.642.12 / 22.64 / 22.9121.95 / 8.50 / 0.6127.79
Qwen2-VL-7BUniform random8.8 / 7.612.29 / 0.47 / 2.5618.00 / 7.31 / 0.487.30
Rate-matched random17.6 / 13.624.88 / 6.60 / 5.5723.79 / 10.12 / 0.6613.16
Fixed delay8.8 / 7.27.31 / 0.00 / 2.7814.11 / 3.21 / 0.005.71
LatPA (ours)46.4 / 39.241.35 / 23.58 / 10.8322.77 / 8.74 / 0.6026.25
InternVL3-2BUniform random8.8 / 8.812.29 / 0.47 / 0.4318.00 / 7.31 / 0.487.27
Rate-matched random22.0 / 21.625.21 / 3.77 / 2.0721.84 / 9.83 / 0.6114.30
Fixed delay8.8 / 8.07.31 / 0.00 / 1.3414.11 / 3.21 / 0.005.68
LatPA (ours)28.8 / 28.051.81 / 20.75 / 4.7222.92 / 8.69 / 0.6621.64
InternVL3-8BUniform random8.8 / 8.412.29 / 0.47 / 1.2418.00 / 7.31 / 0.487.29
Rate-matched random19.2 / 17.227.08 / 3.30 / 2.5122.20 / 9.40 / 0.6713.30
Fixed delay8.8 / 6.87.31 / 0.00 / 1.8414.11 / 3.21 / 0.005.54
LatPA (ours)29.2 / 26.449.63 / 21.70 / 8.2122.71 / 8.13 / 0.6221.60
Reviewer_jmm6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LatPA, a training-free method for proactive response timing in streaming video understanding. It extracts intermediate hidden states from a frozen Video-LLM and decides when to respond by comparing the current frame and recent memory with the user query. The method is model-agnostic and achieves reasonably competitive results without additional training.

Critical strengths

1 LatPA is training-free and can be applied to different frozen Video-LLMs without collecting response-timing labels or modifying model parameters. 2. Using intermediate semantic features to relate the current frame, recent history, and user query is simple and practically attractive for proactive streaming systems.

Critical weaknesses
  1. Although the paper presents a simple and effective approach, the proposed method appears somewhat heuristic and may be prone to both false triggers and missed triggers. Furthermore, its memory capacity seems rather limited, as it only considers the eight most recent frames. This limitation may prevent the model from answering questions that require information from the more distant past.
  2. The paper focuses on a single relatively simple approach, making the overall technical contribution somewhat limited.
Questions for authors
  1. The pooling operation used to obtain hidden states may dilute fine-grained semantic information. Have the authors evaluated the method on queries that do not contain explicit entities or clearly identifiable targets? For example, how would the method handle a query such as "Notify me whenever something unusual happens"?

  2. Can the proposed method handle multiple queries in a streaming video scenario? In practical streaming video understanding applications, users may issue multiple queries over time rather than a single fixed query. If a new query is introduced after the system has been running for some time, how would the method handle this situation (e.g., how to update the memory M)?

2 comments
AuthorsOpenReview ↗

Response to Reviewer jmm6 (Part 1/2): Q1, Q2

Thank you very much for carefully reviewing our work and for your valuable comments. Our responses to your questions and comments are as follows.

Questions

Q1: The pooling operation used to obtain hidden states may dilute fine-grained semantic information. Have the authors evaluated the method on queries that do not contain explicit entities or clearly identifiable targets? For example, how would the method handle a query such as "Notify me whenever something unusual happens"?

Sure, we agree that pooling can cause a loss of fine-grained information. Therefore, we replaced the query vector with a random vector and with a vague expression such as "something unusual", and tried again. The experiments are shown in Table jmm6.Q1.1. From the experiments, it can be seen that after switching to a random direction, the scores drop sharply, while the query, although vaguely worded, can still refer to changes in the scene, so in most settings its gap to the full LatPA is relatively small. Overall, however, it is still not as good as the full LatPA.

Table jmm6.Q1.1 LatPA with different query vectors. Bold: best. Italic: second best per backbone.

BackboneMethodStreamingBench-PO TimeIPIBench TimingIPIBench RepeatedIPIBench Und
Qwen2-VL-2BLatPA (w/ random query)11.612.300.003.01
LatPA (w/ vague query)45.639.8922.6422.24
LatPA (ours)48.042.1222.6422.91
Qwen2-VL-7BLatPA (w/ random query)5.613.779.430.96
LatPA (w/ vague query)48.835.3220.7510.86
LatPA (ours)46.441.3523.5810.83
InternVL3-2BLatPA (w/ random query)20.41.020.003.48
LatPA (w/ vague query)32.452.6817.455.01
LatPA (ours)28.851.8120.754.72
InternVL3-8BLatPA (w/ random query)18.81.8014.152.75
LatPA (w/ vague query)29.647.0020.757.73
LatPA (ours)29.249.6321.708.21

Q2: Can the proposed method handle multiple queries in a streaming video scenario? In practical streaming video understanding applications, users may issue multiple queries over time rather than a single fixed query. If a new query is introduced after the system has been running for some time, how would the method handle this situation (e.g., how to update the memory M)?

Sure. In practice, it is common for multiple questions to arrive one after another within a single video. LatPA handles this as follows: when a new question arrives, a new memory M is created for it, initialized with the visual mean of the current 8-frame window, and then updated by EMA at every poll. Since the frame representation ntn_t and M are both independent of the question, re-initialization does not lose information related to the old questions. When multiple questions coexist, each question only needs to maintain its own memory M and use its own M vector for decision making at each poll.

AuthorsOpenReview ↗

Response to Reviewer jmm6 (Part 2/2): W1, W2

Weaknesses

W1: Although the paper presents a simple and effective approach, the proposed method appears somewhat heuristic and may be prone to both false triggers and missed triggers. Furthermore, its memory capacity seems rather limited, as it only considers the eight most recent frames. This limitation may prevent the model from answering questions that require information from the more distant past.

We acknowledge that LatPA is heuristic in nature, but at the same time, judging from the results, almost all methods, including LatPA, have false triggers and missed triggers, and compared with most existing methods, LatPA can achieve higher accuracy.

Regarding memory capacity, we use an EMA to aggregate the frame representations of each poll. Information from before the 8-frame window does not disappear immediately, but is retained in the memory M with weights that decay at each poll, so that the trigger reference extends beyond the window, while the memory is still dominated by recent content. This design is because what the trigger needs to judge is "whether question-relevant content has just appeared", so the reference should reflect the recent context. Meanwhile, regarding how to efficiently store longer historical memory in the latent space, we compare several ways of aggregating the memory signal in the table below. It can be seen that the cumulative mean, which keeps the entire history, scores only 2.61 and 4.83 on IPIBench with Qwen, whereas the recency-dominated EMA is the best in most settings and is also the only form that does not collapse in any combination.

Table jmm6.W1.1 Memory forms at 0.5L. Each cell: StreamingBench-PO Time / IPIBench Timing / FAR Avg. Sliding mean: mean of the frame vectors from the last 8 forward passes (Tab. 3 of the paper averages the 8 frames within one forward pass instead). Bold: best. Italic: second best per backbone and benchmark.

Memory formQwen2-VL-2BQwen2-VL-7BInternVL3-2BInternVL3-8B
Previous frame16.0 / 34.17 / 9.2418.4 / 24.26 / 9.1716.4 / 26.89 / 9.8511.6 / 29.77 / 9.56
Cumulative mean9.6 / 2.61 / 7.1813.6 / 4.83 / 7.0031.2 / 39.58 / 9.6135.2 / 27.77 / 9.06
Sliding mean42.0 / 6.27 / 9.3140.4 / 8.32 / 8.9435.2 / 42.48 / 9.3842.0 / 32.21 / 9.11
EMA (ours)48.0 / 42.12 / 10.3546.4 / 41.35 / 10.7028.8 / 51.81 / 10.7629.2 / 49.63 / 10.49

As for the problem of answering questions that require distant information mentioned by the reviewer, this is a question concerning the capability of the VLM itself and is not directly related to the EMA memory storage we propose. LatPA is used only for the temporal trigger decision, and it can only improve the VLM's accuracy in answering through relatively precise temporal localization.

Meanwhile, since the size of the trigger window W can be changed, we also tried more settings of the trigger window size.

Table jmm6.W1.2 StreamingBench-PO Time with trigger window WW (frames) at 0.5L. Bold: best. Italic: second best per backbone.

BackboneW=2W=2W=4W=4W=8W=8 (Ours)W=16W=16W=32W=32
Qwen2-VL-2B37.246.048.044.039.6
Qwen2-VL-7B25.642.446.443.239.6
InternVL3-2B38.828.028.834.435.6
InternVL3-8B31.626.829.234.435.2

W2: The paper focuses on a single relatively simple approach, making the overall technical contribution somewhat limited.

We agree with the reviewer that LatPA is a simple method. LatPA proposes a concise and efficient decision method in the field of proactive decision making. Since it requires no additional training, LatPA has good generalization and is easy to transfer to different scenarios for testing. In addition, we also try to obtain more guiding information by processing the model's own feedback signals in the latent space, to be used in more different tasks.

Thank you again for your valuable comments. If the above responses have addressed your concerns, we kindly ask you to consider adjusting your rating. If you have any other questions, we are happy to continue answering them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.