- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies proactive streaming video understanding. In this setting, a video agent observes frames causally and must decide at each timestep whether to respond to an earlier user query or stay silent. The authors frame proactivity as a sequence of causal response decisions that trade off response timeliness against firing precision, and build a framework (PACE) around this formulation with four components:
- PACE-780K dataset. About 785K responses over 178K videos. Gemini generates the query and response/silence sequences, which human annotators then correct and expert reviewers audit. Response content, response intervals, and silence states are all verified.
- PAUC-F metric. It uses PAUC as recall and pairs it with firing precision via a harmonic mean, penalizing both redundant and out-of-window responses.
- Inference harness. It adapts frozen offline VLMs to streaming inference with a sliding frame window, interaction memory, and structured outputs. The design was found by agent-assisted search on validation PAUC-F.
- Streaming SFT pipeline. It trains on the same interface with Silence/Response/Hold state tokens and loss masking, yielding PACE-Gemma and PACE-Qwen.
Critical strengths
- Clear motivation. PAUC can be inflated by excessive firing. This is documented in prior work (MMDuet2) and demonstrated by the oracle/spam simulation in Fig. 2, so adding a precision term is well justified.
- Valuable dataset. Human verification covers response content, response intervals, and when the model should stay silent. Existing proactive datasets generally lack this (Table 1).
- Comprehensive experiments. The evaluation spans four proactive streaming benchmarks and different baselines.
Critical weaknesses
- Typos and writing issues. For example, “Casual Decision States” in Fig. 1 should be “Causal”, and “its roles is” in Sec. 2 is ungrammatical. The paper would benefit from careful proofreading.
- Hold ablation. Only 18% of Response states are followed by Hold, yet removing Hold drops WEB from 56.5 to 32.9 (Table 10). The mechanism behind this large effect needs explaining. It is also unclear how Hold is used at inference, since the output format (Table A.9) only distinguishes silence from a response.
- Train/inference mismatch. Training memory holds the last k annotated responses (Sec. 5.2.2), but inference keeps only the last accepted one (Table A.9). Likewise, training chunks use up to 32 frames versus 3 at inference.
- Scale mismatch in PAUC-F. PAUC has a silence floor of 0.25 (Appendix D), while precision has a floor of 0. The harmonic mean therefore combines R ∈ [0.25, 1] with P ∈ [0, 1], two terms on different scales. The authors should consider normalizing PAUC, e.g., (PAUC − 0.25)/0.75, or show that this choice does not affect model rankings.
- Possible column swap in Table 8. PACE-Gemma scores WEB 56.46 and EGO 43.80 in Table 3, consistent with Table 10. Table 8, however, reports the same model as EGO 56.5 and WEB 43.8, suggesting the EGO/WEB labels are swapped.
Questions for authors
- Is PAUC-F aggregated at the macro level or the micro level?
- How is the Hold state used at inference? Why does removing it cause such a large performance drop?
8 comments
Follow-up
Thank you again for your feedback. We have uploaded a revised PDF, with changes highlighted in green, clarifying the Hold state, training/inference configurations, and metric definition, alongside the additional Hold ablation analysis in our rebuttal. We would appreciate your thoughts on whether these updates address your concerns.
Follow-up
We thank the reviewer again for the feedback.
We are wondering if the reviewer had a chance to see our rebuttal, and if any concerns still remain.
Rebuttal [1/1]
We thank the reviewer for their time and feedback. All concerns are addressed below.
1. Typos
2. “Hold” State
The “Hold” state is only used with our finetuned models and their inference
As described in Table A.11, our prompt contains “</Hold> when the event continues but no new answer is due” and this prompt is used to finetune our model (and run inference with finetuned models).
We provide additional results on “Hold” ablation experiments below
Note how this "Hold" state mainly reduces repeated redundant responses.
We will explain these clearly during revision.
3. Training/inference configuration
4. PAUC metric definition and behavior
5. Typo (Column Swap)
Q1. Score aggregation
Q2. Hold State