- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces P³IBench for evaluating proactive video assistants across daily perception, risk alert, and event intervention tasks. It contains 533 task samples from 429 videos, with 836 target responses. The intervention subset uses scripted, newly recorded videos with annotated intervention windows and reference responses. The authors introduce PRS-U and PRS-B to jointly evaluate response correctness, timing, and redundancy, benchmark 12 models, and train P³I-Assistant-4B through supervised fine-tuning and reinforcement learning.
Critical strengths
S1: The joint construction of tasks, videos, and annotations is valuable.
For the intervention subset, the authors design scripts around user goals and intentional execution errors, record corresponding videos, and annotate intervention windows and reference responses. This closely aligns instructions, visual evidence, and expected assistance, reducing potential mismatches from generating questions retrospectively for existing videos. I consider this construction strategy particularly appropriate for proactive assistants operating on streaming video: it creates complete situations specifying when assistance is needed and what assistance would be useful.
Critical weaknesses
W1: The claimed gap in existing benchmarks is not sufficiently proved.
The authors present risk alert and event intervention as key additions, but the examples use instructions that explicitly specify both the assistance goal and the triggering condition, such as “remind me if I try to check out before buying everything.” This appears different from anticipating the user’s need for assistance. Similar tasks are already common in existing benchmarks. For example, OmniPro includes “teach me to cook scrambled eggs with tomatoes and tell me the next step,” while EgoProactive/PWR considers “help me make an espresso.” They also require models to determine when to provide guidance or correction. GuideMe further includes explicit corrective interventions, such as warning a user who selects the wrong sauce while making pizza.
W2: Limited evaluation of proactive video models.
Only four of the 12 evaluated models are native proactive models, while the remaining eight are offline models adapted through polling. In fact, the finding that offline models produce excessive responses, struggling to remain silent, largely reiterates the results of earlier work such as VideoLLM-online, rather than providing a new insight. A benchmark targeting proactive streaming video understanding should include a broader evaluation of methods developed for this setting. Relevant methods include VideoLLM-online, Dispider, LiveStar, Memento, MiniCPM-o 4.5, StreamPro, Vinci2/EgoMemo, and PWR, covering proactive response timing, long-term reminders, and task-oriented intervention.
References:
[1] Zhao et al. OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding, 2026.
[2] Kundu et al. Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance, 2026.
[3] Liu et al. GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video, 2026.
[4] Chen et al. VideoLLM-online: Online Video Large Language Model for Streaming Video, CVPR 2024.
[5] Qian et al. Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction, CVPR 2025.
[6] LiveStar: Live Streaming Assistant for Real-World Online Video Understanding, 2025.
[7] Jiang et al. Memento: Toward an All-Day Proactive Assistant for Ultra-Long Streaming Video, ICLR 2026.
[8] Cui et al. MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction, 2026.
[9] Li et al. StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video, 2026.
[10] Gong et al. Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos, 2026.
Questions for authors
Q1: Instruction diversity and task difficulty.
Does the benchmark include instructions that specify only a broad user goal, requiring the model to infer when and how assistance is needed, rather than explicitly defining the triggering condition (realtive a simple task)?
Q2: Reliability of LLM-based evaluation.
As LLM-based scoring is a key component of this benchmark’s evaluation protocol, I would like to know which judge model and version are used, what evaluation prompt and scoring criteria are adopted, and how judgments are converted into scores in [0,1]. Does the judge receive the user instruction and visual evidence, or only the generated and reference responses? Has its agreement with human judgments been assessed? How much do scores vary across repeated evaluations, and how sensitive are model rankings to the choice of judge? If evaluation requires API calls, what is the estimated cost of reproducing the full evaluation?
Q3: Multiple user instructions.
Does each video contain only one initial user instruction? Are there cases involving multiple concurrent requests?
7 comments
Response to Reviewer CoBE (Part 2 of 3)
Weakness 2: Coverage of native proactive video models.
Response:
We have added LiveStar and MiniCPM-o 4.5 to the revised main table, expanding the evaluation from 12 to 14 models and the coverage of native proactive streaming models from four to six. Experiments on Vinci2/EgoMemo are still ongoing, and we will add the results as soon as they are complete.
The two new rows from the main table are reproduced below on a 0–100 scale; each cell reports PRS-U / PRS-B.
| Model | Event | Object | Accident | Crime | Intervention | Overall Average |
|---|---|---|---|---|---|---|
| MiniCPM-o 4.5 | 5.84 / 6.66 | 26.00 / 26.00 | 6.43 / 7.13 | 19.61 / 22.32 | 2.28 / 3.75 | 12.03 / 13.17 |
| LiveStar | 1.68 / 2.33 | 14.02 / 24.56 | 12.13 / 19.74 | 9.46 / 11.97 | 17.40 / 21.52 | 10.94 / 16.02 |
We also tested VideoLLM-online and Dispider before, but both obtained nearly zero scores on our tasks, so their results were not reported in the original paper. In addition, StreamPro, PWR, and Memento had not publicly released their implementations before the original paper's submission deadline, which prevented us from evaluating these three models for the submitted version.
Question 2: Reliability and reproducibility of LLM-based evaluation.
Response2.1: The judge model, input, evaluation prompt, and scoring criteria.
We apologize for omitting the details of the LLM judge from the initial version of the paper. We have now included the evaluation setup and full judge prompt in Appendix C.1, LLM-based semantic scoring is used for the open-ended Intervention responses; structured outputs are evaluated with deterministic rules.
In summary, we use Qwen3-32B-FP8 as the judge, with temperature 0, thinking disabled, and a maximum output length of four tokens. For each judgment, it receives only the reference and predicted responses as text, without the user instruction or video. It assigns:
- 1 when all indispensable reference facts are covered without material contradiction.
- 0.5 when at least one essential fact is correct but another is missing or incorrect.
- 0 when no essential fact is correct, the response is irrelevant, or its core meaning contradicts the reference.
Response2.2: Agreement with human judgments.
To assess alignment with human judgments, we uniformly sampled 150 of the 1,601 Intervention responses generated by GPT-5.6-Luna, without replacement. Two human raters scored each response against its reference on the same three-point scale. Quadratic weighted Cohen's was 0.824 and 0.816 between Qwen and the two human raters, respectively; human–human agreement was 0.895. These comparisons concern semantic scores before temporal decay. The confusion matrices are reported in Appendix C.1.
| Comparison | Quadratic weighted Cohen's |
|---|---|
| Human 1–Qwen | 0.824 |
| Human 2–Qwen | 0.816 |
| Human 1–Human 2 | 0.895 |
Response to Reviewer CoBE (Part 3 of 3)
Question 3: Initial instructions and multiple responses.
Response:
In the current Intervention dataset, a single initial user instruction can require multiple responses as the video unfolds. The number of responses is not specified in advance: the model must decide for itself whether and when to respond, and how many responses are needed.
The original dataset contains only initial user instructions. Following your suggestion, we completed an exploratory multi-instruction evaluation on ten Intervention videos, introducing a scheduled update to the destination, item list, placement rule, or completion criterion. We evaluate responses after each update against 28 updated reference events. The following two tables reproduce the new appendix results (0–100).
| Model | Original PRS-U | Original PRS-B | Original replies | Updated PRS-U | Updated PRS-B | Updated replies |
|---|---|---|---|---|---|---|
| Qwen3-VL-8B | 11.66 | 9.74 | 292 | 7.19 | 6.44 | 645 |
| JoyVL | 15.00 | 14.17 | 607 | 3.36 | 2.81 | 679 |
| LiveStar | 1.91 | 5.44 | 72 | 8.37 | 8.01 | 111 |
| Video ID | Update time (s) | Updated requirement | Qwen3-VL-8B PRS-B: original → updated | JoyVL PRS-B: original → updated | LiveStar PRS-B: original → updated |
|---|---|---|---|---|---|
| 0 | 260 | Replace tomatoes with sausages | 20.47 → 0.05 | 0.48 → 0.00 | 0.00 → 0.00 |
| 24 | 55 | Switch noodle brand to Uni-President | 8.28 → 1.08 | 0.60 → 1.63 | 0.00 → 2.51 |
| 3 | 22 | Go to room 212 on the second floor | 46.79 → 7.58 | 96.83 → 5.06 | 0.00 → 0.00 |
| 4 | 60 | Go to the fourth floor | 1.18 → 6.77 | 24.69 → 1.18 | 0.00 → 13.90 |
| 5 | 55 | Require a power-on acceptance test | 0.00 → 7.50 | 2.50 → 3.75 | 0.00 → 19.48 |
| 7 | 60 | Return the cloth as well as the glasses | 4.43 → 6.66 | 3.54 → 2.26 | 34.62 → 6.66 |
| 37 | 35 | Place the mouse on the left | 4.28 → 20.35 | 2.08 → 3.67 | 0.00 → 18.93 |
| 45 | 39 | Pencil case right; calculator left | 7.20 → 4.05 | 5.74 → 6.93 | 13.67 → 0.00 |
| 10 | 30 | Add the phone charger to packing | 4.76 → 6.75 | 0.00 → 2.86 | 6.10 → 16.95 |
| 35 | 46 | Remove the power bank from packing | 0.00 → 3.63 | 5.26 → 0.79 | 0.00 → 1.62 |
Overall, performance declines when user instructions are modified midstream, suggesting that current models still struggle to follow updated instructions reliably. Further experimental details and analysis are provided in Appendix H.
Response to Reviewer CoBE (Part 1 of 3)
Weakness 1 & Question 1: Benchmark contributions, instruction diversity, and task difficulty.
Response1.1 — Clear instructions and broader goals. We deliberately make instructions explicit so that models with weaker instruction-following abilities can understand the assistance task itself. This reduces ambiguity about the requested behavior while leaving the model to determine from the video whether and when help is needed. In a controlled prompt comparison on a selected navigation video, both conditions used identical inference settings within each model and differed only in the user instruction. For Qwen3-VL-8B-Instruct, detailed instructions reduced non-silent replies from 52 to 16 and exact-text repetitions from 44 to 12, indicating better response control. We also have a more general prompt variant for the Intervene subset, which we will release alongside the detailed instructions with the dataset.
Response1.2 — Complementary proactive capabilities and temporal annotations. Our central contribution is the joint evaluation of Perception, Prediction, and Intervention. In particular, timely Risk Alert, including accident warnings and crime alerts, remains rarely covered in the proactive video-assistance benchmarks compared in our paper, despite being essential for proactive assistance. For Intervention, we manually design scripts and record videos containing explicit mistakes or situations requiring reminders, then annotate useful intervention window and reference responses. We acknowledge the similar scripted-deviation practice in concurrent EgoProactive/PWR. Its newly collected EgoProactive data are described as home-recorded cooking, crafts, DIY, and tutorial procedures. Our Intervention subset extends this setting to outdoor navigation. Importantly, Risk Alert and Intervention both provide fine-grained, manually verified temporal targets, contributing data and timing references for evaluating when proactive help is needed.