acceptodds
Under review as a conference paper at ICLR 2027

From Perception to Prediction and Intervention: Benchmarking Proactive Video Assistants in Real-Life Scenarios

Abstract

Recent multimodal large language models have advanced video understanding, but existing benchmarks mainly evaluate whether models can interpret observed visual content. In real-world scenarios, assistants must additionally anticipate what assistance is needed and decide when to provide it. To address this gap, we introduce PIBench, a benchmark for proactive video understanding in real-life streaming scenarios. It evaluates three complementary capabilities: daily perception, risk prediction and intervention. We benchmark 12 representative open-source and proprietary video-language models, including streaming and adapted non-streaming systems. Our evaluation shows that current models struggle to make well-calibrated response decisions. Finally, we construct additional proactive training data and train PI-Assistant-4B, achieving improvements on several tasks, though gains remain uneven across capabilities. Together, these findings show that effective proactive assistance requires not only visual understanding, but also precise response timing and appropriate restraint.

Primary area:
datasets and benchmarks
Keywords:
Proactive Video Understanding, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CoBEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces P³IBench for evaluating proactive video assistants across daily perception, risk alert, and event intervention tasks. It contains 533 task samples from 429 videos, with 836 target responses. The intervention subset uses scripted, newly recorded videos with annotated intervention windows and reference responses. The authors introduce PRS-U and PRS-B to jointly evaluate response correctness, timing, and redundancy, benchmark 12 models, and train P³I-Assistant-4B through supervised fine-tuning and reinforcement learning.

Critical strengths

S1: The joint construction of tasks, videos, and annotations is valuable.

For the intervention subset, the authors design scripts around user goals and intentional execution errors, record corresponding videos, and annotate intervention windows and reference responses. This closely aligns instructions, visual evidence, and expected assistance, reducing potential mismatches from generating questions retrospectively for existing videos. I consider this construction strategy particularly appropriate for proactive assistants operating on streaming video: it creates complete situations specifying when assistance is needed and what assistance would be useful.

Critical weaknesses

W1: The claimed gap in existing benchmarks is not sufficiently proved.

The authors present risk alert and event intervention as key additions, but the examples use instructions that explicitly specify both the assistance goal and the triggering condition, such as “remind me if I try to check out before buying everything.” This appears different from anticipating the user’s need for assistance. Similar tasks are already common in existing benchmarks. For example, OmniPro includes “teach me to cook scrambled eggs with tomatoes and tell me the next step,” while EgoProactive/PWR considers “help me make an espresso.” They also require models to determine when to provide guidance or correction. GuideMe further includes explicit corrective interventions, such as warning a user who selects the wrong sauce while making pizza.

W2: Limited evaluation of proactive video models.

Only four of the 12 evaluated models are native proactive models, while the remaining eight are offline models adapted through polling. In fact, the finding that offline models produce excessive responses, struggling to remain silent, largely reiterates the results of earlier work such as VideoLLM-online, rather than providing a new insight. A benchmark targeting proactive streaming video understanding should include a broader evaluation of methods developed for this setting. Relevant methods include VideoLLM-online, Dispider, LiveStar, Memento, MiniCPM-o 4.5, StreamPro, Vinci2/EgoMemo, and PWR, covering proactive response timing, long-term reminders, and task-oriented intervention.

References:

[1] Zhao et al. OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding, 2026.

[2] Kundu et al. Plan, Watch, Recover: A Benchmark and Architectures for Proactive Procedural Assistance, 2026.

[3] Liu et al. GuideMe: Multi-Domain Task Guidance and Intervention in Streaming Video, 2026.

[4] Chen et al. VideoLLM-online: Online Video Large Language Model for Streaming Video, CVPR 2024.

[5] Qian et al. Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction, CVPR 2025.

[6] LiveStar: Live Streaming Assistant for Real-World Online Video Understanding, 2025.

[7] Jiang et al. Memento: Toward an All-Day Proactive Assistant for Ultra-Long Streaming Video, ICLR 2026.

[8] Cui et al. MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interaction, 2026.

[9] Li et al. StreamPro: From Reactive Perception to Proactive Decision-Making in Streaming Video, 2026.

[10] Gong et al. Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos, 2026.

Questions for authors

Q1: Instruction diversity and task difficulty.

Does the benchmark include instructions that specify only a broad user goal, requiring the model to infer when and how assistance is needed, rather than explicitly defining the triggering condition (realtive a simple task)?

Q2: Reliability of LLM-based evaluation.

As LLM-based scoring is a key component of this benchmark’s evaluation protocol, I would like to know which judge model and version are used, what evaluation prompt and scoring criteria are adopted, and how judgments are converted into scores in [0,1]. Does the judge receive the user instruction and visual evidence, or only the generated and reference responses? Has its agreement with human judgments been assessed? How much do scores vary across repeated evaluations, and how sensitive are model rankings to the choice of judge? If evaluation requires API calls, what is the estimated cost of reproducing the full evaluation?

Q3: Multiple user instructions.

Does each video contain only one initial user instruction? Are there cases involving multiple concurrent requests?

7 comments
AuthorsOpenReview ↗

Response to Reviewer CoBE (Part 1 of 3)

Weakness 1 & Question 1: Benchmark contributions, instruction diversity, and task difficulty.

Response1.1 — Clear instructions and broader goals. We deliberately make instructions explicit so that models with weaker instruction-following abilities can understand the assistance task itself. This reduces ambiguity about the requested behavior while leaving the model to determine from the video whether and when help is needed. In a controlled prompt comparison on a selected navigation video, both conditions used identical inference settings within each model and differed only in the user instruction. For Qwen3-VL-8B-Instruct, detailed instructions reduced non-silent replies from 52 to 16 and exact-text repetitions from 44 to 12, indicating better response control. We also have a more general prompt variant for the Intervene subset, which we will release alongside the detailed instructions with the dataset.

ModelDetailed repliesGeneral repliesDetailed PRS-BGeneral PRS-B
Qwen3-VL-8B-Instruct165222.385.28
GPT-5.6-Luna31729.2523.41
LiveStar650.000.00

Response1.2 — Complementary proactive capabilities and temporal annotations. Our central contribution is the joint evaluation of Perception, Prediction, and Intervention. In particular, timely Risk Alert, including accident warnings and crime alerts, remains rarely covered in the proactive video-assistance benchmarks compared in our paper, despite being essential for proactive assistance. For Intervention, we manually design scripts and record videos containing explicit mistakes or situations requiring reminders, then annotate useful intervention window and reference responses. We acknowledge the similar scripted-deviation practice in concurrent EgoProactive/PWR. Its newly collected EgoProactive data are described as home-recorded cooking, crafts, DIY, and tutorial procedures. Our Intervention subset extends this setting to outdoor navigation. Importantly, Risk Alert and Intervention both provide fine-grained, manually verified temporal targets, contributing data and timing references for evaluating when proactive help is needed.

AuthorsOpenReview ↗

Response to Reviewer CoBE (Part 2 of 3)

Weakness 2: Coverage of native proactive video models.

Response:

We have added LiveStar and MiniCPM-o 4.5 to the revised main table, expanding the evaluation from 12 to 14 models and the coverage of native proactive streaming models from four to six. Experiments on Vinci2/EgoMemo are still ongoing, and we will add the results as soon as they are complete.

The two new rows from the main table are reproduced below on a 0–100 scale; each cell reports PRS-U / PRS-B.

ModelEventObjectAccidentCrimeInterventionOverall Average
MiniCPM-o 4.55.84 / 6.6626.00 / 26.006.43 / 7.1319.61 / 22.322.28 / 3.7512.03 / 13.17
LiveStar1.68 / 2.3314.02 / 24.5612.13 / 19.749.46 / 11.9717.40 / 21.5210.94 / 16.02

We also tested VideoLLM-online and Dispider before, but both obtained nearly zero scores on our tasks, so their results were not reported in the original paper. In addition, StreamPro, PWR, and Memento had not publicly released their implementations before the original paper's submission deadline, which prevented us from evaluating these three models for the submitted version.


Question 2: Reliability and reproducibility of LLM-based evaluation.

Response2.1: The judge model, input, evaluation prompt, and scoring criteria.

We apologize for omitting the details of the LLM judge from the initial version of the paper. We have now included the evaluation setup and full judge prompt in Appendix C.1, LLM-based semantic scoring is used for the open-ended Intervention responses; structured outputs are evaluated with deterministic rules.

In summary, we use Qwen3-32B-FP8 as the judge, with temperature 0, thinking disabled, and a maximum output length of four tokens. For each judgment, it receives only the reference and predicted responses as text, without the user instruction or video. It assigns:

  1. 1 when all indispensable reference facts are covered without material contradiction.
  2. 0.5 when at least one essential fact is correct but another is missing or incorrect.
  3. 0 when no essential fact is correct, the response is irrelevant, or its core meaning contradicts the reference.

Response2.2: Agreement with human judgments.

To assess alignment with human judgments, we uniformly sampled 150 of the 1,601 Intervention responses generated by GPT-5.6-Luna, without replacement. Two human raters scored each response against its reference on the same three-point scale. Quadratic weighted Cohen's κ\kappa was 0.824 and 0.816 between Qwen and the two human raters, respectively; human–human agreement was 0.895. These comparisons concern semantic scores before temporal decay. The confusion matrices are reported in Appendix C.1.

ComparisonQuadratic weighted Cohen's κ\kappa
Human 1–Qwen0.824
Human 2–Qwen0.816
Human 1–Human 20.895
AuthorsOpenReview ↗

Response to Reviewer CoBE (Part 3 of 3)

Question 3: Initial instructions and multiple responses.

Response:

In the current Intervention dataset, a single initial user instruction can require multiple responses as the video unfolds. The number of responses is not specified in advance: the model must decide for itself whether and when to respond, and how many responses are needed.

The original dataset contains only initial user instructions. Following your suggestion, we completed an exploratory multi-instruction evaluation on ten Intervention videos, introducing a scheduled update to the destination, item list, placement rule, or completion criterion. We evaluate responses after each update against 28 updated reference events. The following two tables reproduce the new appendix results (0–100).

ModelOriginal PRS-UOriginal PRS-BOriginal repliesUpdated PRS-UUpdated PRS-BUpdated replies
Qwen3-VL-8B11.669.742927.196.44645
JoyVL15.0014.176073.362.81679
LiveStar1.915.44728.378.01111
Video IDUpdate time (s)Updated requirementQwen3-VL-8B PRS-B: original → updatedJoyVL PRS-B: original → updatedLiveStar PRS-B: original → updated
0260Replace tomatoes with sausages20.47 → 0.050.48 → 0.000.00 → 0.00
2455Switch noodle brand to Uni-President8.28 → 1.080.60 → 1.630.00 → 2.51
322Go to room 212 on the second floor46.79 → 7.5896.83 → 5.060.00 → 0.00
460Go to the fourth floor1.18 → 6.7724.69 → 1.180.00 → 13.90
555Require a power-on acceptance test0.00 → 7.502.50 → 3.750.00 → 19.48
760Return the cloth as well as the glasses4.43 → 6.663.54 → 2.2634.62 → 6.66
3735Place the mouse on the left4.28 → 20.352.08 → 3.670.00 → 18.93
4539Pencil case right; calculator left7.20 → 4.055.74 → 6.9313.67 → 0.00
1030Add the phone charger to packing4.76 → 6.750.00 → 2.866.10 → 16.95
3546Remove the power bank from packing0.00 → 3.635.26 → 0.790.00 → 1.62

Overall, performance declines when user instructions are modified midstream, suggesting that current models still struggle to follow updated instructions reliably. Further experimental details and analysis are provided in Appendix H.

Reviewer_QPAaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces P³IBench, a benchmark for evaluating proactive video assistants in causal streaming settings. The benchmark is organized around three capabilities: perception, prediction, and intervention, instantiated through Event Monitoring, Object Monitoring, Accident Warning, Crime Alert, and Event Intervention. It contains 533 samples from 429 videos and evaluates not only whether a model produces the correct response, but whether it responds at an appropriate time and avoids unnecessary repeated responses.

The authors evaluate a broad set of adapted offline MLLMs and native proactive streaming models and introduce PRS-U and PRS-B, which combine response correctness with temporal alignment while penalizing redundant outputs and missed targets. The results reveal complementary failure modes: adapted offline models frequently over-respond, while proactive models can become overly conservative and miss required responses. Finally, the authors construct additional proactive training data and train P³I-Assistant-4B using SFT followed by RL. This improves average PRS-B over the underlying Qwen3-VL-4B model, although improvements are uneven across tasks.

Critical strengths
  1. The benchmark evaluates a meaningful progression from perception to prediction and intervention rather than treating proactive video understanding as event recognition alone. The strongest aspect of the paper is the explicit evaluation of when not to respond alongside when to respond. The reported failure modes demonstrate why this matters: some adapted offline models repeatedly trigger, whereas proactive systems may suppress responses too aggressively. The inclusion of intervention additionally moves beyond retrospective video understanding toward models that must maintain goals and determine whether current activity warrants assistance.
  2. Object Monitoring provides an interesting test of temporal correspondence that connects the benchmark to the tracking literature. The Object Monitoring task requires identifying a specified object, maintaining its identity through the video, and distinguishing separate disappearance events. This is closely related to the temporal correspondence problem studied in object tracking. Prior work has shown that deep networks can struggle when target identity must be maintained without reliable appearance cues (Linsley et al., 2021), and more recent work demonstrates similar difficulties when objects change both appearance and location (Muzellec et al., 2025). P³IBench places a related challenge inside a streaming multimodal assistant, where successful correspondence must additionally trigger an appropriately timed response. I think this provides a useful connection between traditional tracking and current video-MLLM evaluation.
Critical weaknesses
  1. The interpretation of proactive timing depends substantially on the matching rule and temporal-decay formulation. PRS-U and PRS-B encode meaningfully different notions of a correct proactive response. The paper reports substantial gaps between the two for several models and tasks. In PRS-B, an early response can be associated with a future target, whereas PRS-U is considerably stricter about premature responses. This is not merely a metric detail. For Accident Warning, for example, useful anticipation should be rewarded, while an unsupported early warning should not. I would like stronger evidence that model rankings and the main conclusions are robust to the chosen temporal decay parameter and, more importantly, to the definition of the target time itself. Since "the earliest time when the risk can be reliably identified" can be intrinsically ambiguous, especially for prediction, uncertainty in this annotation should ideally be reflected in evaluation rather than represented by a single timestamp.
  2. The benchmark is broad in capability coverage but relatively small and heterogeneous across tasks. The five task groups are sourced from substantially different domains and datasets, and Event Intervention contains relatively few samples compared with some of the perception tasks. Since the paper reports an overall average across these heterogeneous tasks, it is not obvious that this scalar score provides the most meaningful summary of proactive capability. A model strong at accident prediction but poor at intervention is qualitatively different from the reverse. I therefore find the per-task results substantially more informative than the overall ranking. This is reinforced by the training experiment, where post-training produces large gains on some tasks, particularly Accident Warning, while Event Intervention changes very little. I would encourage the authors to frame P³IBench primarily as a capability profile rather than emphasizing a single overall ranking.
Questions for authors
  1. Clarification that could materially affect my assessment: How sensitive are model rankings to the temporal-decay constant α and to uncertainty in the annotated target time? In particular, could the authors report a sensitivity analysis over α or a window-based version of PRS? This seems important for the Risk Alert tasks, where the precise earliest moment at which a future accident becomes "reliably identifiable" may not have a unique answer.
  2. Clarification that could materially affect my assessment: How should readers interpret the large PRS-U versus PRS-B differences on Accident Warning? Can the authors distinguish genuinely predictive early warnings from responses that occur before sufficient visual evidence is present? A qualitative analysis of high-PRS-B/low-PRS-U cases would help determine whether PRS-B rewards useful anticipation or premature triggering.
  3. Clarification that could materially affect my assessment: For Object Monitoring, does evaluation contain cases where maintaining target identity is genuinely difficult, for example through occlusion, substantial appearance/viewpoint change, or interaction with similar objects? Prior work shows that appearance-free correspondence (Linsley et al., 2021) and correspondence under changing appearance (Muzellec et al., 2025) remain challenging for neural systems. Stratifying Object Monitoring according to these factors would clarify whether the task benchmarks temporal identity maintenance or primarily visibility/disappearance detection.

References

Linsley, D., Malik, G., Kim, J., Govindarajan, L. N., Mingolla, E., & Serre, T. (2021). Tracking without re-recognition in humans and machines. Advances in Neural Information Processing Systems, 34, 19473–19486.

Muzellec, S., Linsley, D., Ashok, A. K., Mingolla, E., Malik, G., VanRullen, R., & Serre, T. (2025). Tracking objects that change in appearance with phase synchrony. International Conference on Learning Representations.

5 comments
AuthorsOpenReview ↗

Response to Reviewer QPAa (Part 1 of 3)

Weakness 1 & Question 1: Timing evaluation, temporal-decay sensitivity, and target-time uncertainty.

Response1.1: The rationale for PRS-U and PRS-B and their treatment of uncertain target times.

In summary, PRS-U and PRS-B use the same temporal decay but different matching rules, separating response eligibility from temporal proximity. PRS-U does not credit a response before the first target, whereas PRS-B can credit an early response matched to a future target. Reporting both makes this timing distinction visible.

We also acknowledge that the earliest reliably identifiable moment can be uncertain, especially for risk warnings. For eligible response–target pairs, the moderate decay setting (α=0.05\alpha=0.05) makes temporal credit change smoothly with small shifts in the annotated time, mitigating the effect of minor timestamp imprecision.

We further discuss this point in Common Response2.2 of the General Response.

Response1.2: Sensitivity of model rankings to the temporal-decay parameter and annotated target times.

We chose the moderate decay setting α=0.05\alpha=0.05 to penalize timing deviations while retaining useful score resolution. Our sensitivity analysis shows that the leading overall model ordering remains stable across the tested α\alpha values, although some task-level rankings change.

α=0.01\alpha=0.01. Each cell reports PRS-U / PRS-B (0–100).

ModelEventObjectAccidentCrimeInterventionOverall
Qwen3-VL-8B-Instruct19.55 / 46.6945.82 / 52.9131.15 / 86.7348.23 / 72.226.33 / 7.8030.22 / 53.27
GPT-5.6-Luna21.40 / 41.6039.31 / 43.2561.49 / 83.3562.24 / 75.3121.37 / 23.5241.16 / 53.41
VideoChat312.80 / 29.7920.00 / 21.9631.12 / 32.0822.63 / 25.2514.65 / 21.2320.24 / 26.06
Em-Garde2.63 / 3.1834.89 / 40.8334.87 / 57.6331.02 / 65.236.57 / 11.5022.00 / 35.67

α=0.05\alpha=0.05. Each cell reports PRS-U / PRS-B (0–100).

ModelEventObjectAccidentCrimeInterventionOverall
Qwen3-VL-8B-Instruct16.17 / 26.0044.38 / 50.3329.21 / 68.8343.65 / 57.965.95 / 7.1027.87 / 42.05
GPT-5.6-Luna15.13 / 23.5537.83 / 41.5156.41 / 74.6454.17 / 62.3718.95 / 21.0336.50 / 44.62
VideoChat36.02 / 12.8520.00 / 21.7925.21 / 26.0720.56 / 22.1712.73 / 18.6116.91 / 20.30
Em-Garde2.00 / 2.4131.92 / 37.6127.45 / 44.6925.59 / 46.835.38 / 9.6518.47 / 28.24

α=0.1\alpha=0.1. Each cell reports PRS-U / PRS-B (0–100).

ModelEventObjectAccidentCrimeInterventionOverall
Qwen3-VL-8B-Instruct14.24 / 20.0142.85 / 47.7327.18 / 55.1639.82 / 50.505.58 / 6.4825.94 / 35.97
GPT-5.6-Luna12.32 / 17.4136.34 / 39.7251.46 / 66.6547.76 / 53.6817.00 / 18.8432.98 / 39.26
VideoChat33.99 / 7.5920.00 / 21.6120.19 / 20.9219.22 / 20.2411.19 / 16.2714.92 / 17.33
Em-Garde1.62 / 2.1028.99 / 34.3921.68 / 34.3621.47 / 37.344.41 / 8.0315.63 / 23.25

α=0.5\alpha=0.5. Each cell reports PRS-U / PRS-B (0–100).

ModelEventObjectAccidentCrimeInterventionOverall
Qwen3-VL-8B-Instruct9.20 / 10.7836.08 / 37.8917.45 / 26.0326.70 / 32.584.27 / 4.4618.74 / 22.35
GPT-5.6-Luna6.37 / 7.7830.96 / 32.7630.19 / 37.8229.27 / 32.5211.19 / 11.4421.59 / 24.46
VideoChat30.92 / 1.4920.00 / 20.676.06 / 6.2616.46 / 16.677.24 / 9.2710.14 / 10.87
Em-Garde0.82 / 1.2718.61 / 22.458.86 / 12.3311.32 / 17.262.08 / 3.518.34 / 11.36

The top two models retain their overall positions under both metrics at all four settings. The only overall rank reversal is between Em-Garde and VideoChat3 under PRS-U at α=0.5\alpha=0.5.

We further discuss this point in Common Response2.2 of the General Response, including the rationale for choosing α=0.05\alpha=0.05 and the model-ranking sensitivity analysis reported in Appendix C.2 (Table 7).

AuthorsOpenReview ↗

Response to Reviewer QPAa (Part 2 of 3)

Question 2: Interpreting the PRS-U/PRS-B gap in Accident Warning.

Response:

The PRS-U/PRS-B gap on Accident reflects the different treatment of warnings emitted before the annotated target: PRS-B can credit them through bidirectional matching, whereas PRS-U gives pre-target warnings no credit. Our response-trace analysis shows that this behavior is common. We further discuss this point in Common Question1 of the General Response. Moreover, we provide a detailed analysis of pre-target warnings, silence, response delays, and repeated triggering in Appendix D.3

ModelPRS-UPRS-BRepliesSilentEarly onlyRepeated
GPT-5.6-Luna56.4174.648614230
Qwen3-VL-8B-Instruct29.2168.831083618
JoyVL19.6650.627822570
Qwen2.5-VL-3B-Instruct14.9121.27114471257
MiniCPM-o-4.56.437.1389210
MMDuet20.000.00010000

Question 3: Identity maintenance and challenging appearance changes in Object Monitoring.

Response:

Object Monitoring asks the model to follow a specified object over time and report each disappearance, including cases in which it reappears with a changed viewpoint or at a different distance from the camera. These variations can change its appearance and apparent size and require more than detecting whether an object category is present.

However, the current dataset does not systematically cover non-rigid deformation or the full range of difficult identity-correspondence conditions raised in your question. We will clarify this scope in the paper and consider adding examples with deformation and other challenging appearance changes in future extensions.

AuthorsOpenReview ↗

Response to Reviewer QPAa (Part 3 of 3)

Weakness 2: Heterogeneous tasks and the interpretation of overall rankings.

Response:

The tasks do differ in their data sources, scale, and requirements. Covering and comparing complementary proactive capabilities is a central contribution of our benchmark. We aim to provide a broad evaluation of proactive assistance, spanning event recognition, object monitoring, risk warning, and corrective intervention. Evaluating these capabilities together helps reveal whether a model's strengths extend across proactive settings and where its limitations are specific to a task.

The overall average provides readers with a convenient summary for comparing models across this range of capabilities. We report it alongside per-task results so that readers can obtain an overall impression and then examine how strengths, weaknesses, and trade-offs relate across capabilities. The five tasks receive equal weight, preventing larger datasets from dominating the summary. We agree that this aggregate should be interpreted together with the task profiles, given the differences in task difficulty and composition.

To clarify the task-specific strengths and weaknesses behind the overall rankings, we have added an analysis of the Accident dataset in the main text, examining excessive silence, early-only triggering, and repeated alerts. We also provide detailed analyses across task groups in Appendix D, helping readers interpret the overall average alongside the distinct capabilities and failure modes revealed by each task.

Accident behavior Silent, early-only, and repeated columns count videos; early-only means all replies precede the target. Repeated means multiple replies in a video.

ModelPRS-UPRS-BRepliesSilent videosEarly-only videosRepeated videos
GPT-5.6-Luna56.4174.648614230
Qwen3-VL-8B-Instruct29.2168.831083618
JoyVL19.6650.627822570
Em-Garde27.4544.69116212533
Qwen3-VL-4B-Instruct30.9341.78382201742
InternVL3.5-8B23.1938.79208471911
Qwen3.5-4B18.3927.9216957159
Qwen2.5-VL-3B-Instruct14.9121.27114471257
Qwen3.5-9B15.8215.82188200
MiniCPM-o-4.56.437.1389210
GLM-4.1V-9B-Base2.112.801399116
MMDuet20.000.00010000

Intervention reply accuracy and target coverage. Emitted counts all non-silent replies; Judged counts replies eligible under unidirectional assignment. Content scores precede temporal decay. Timely correct counts targets with at least one score-1 reply inside the annotated response window.

ModelEmittedJudgedc=0c=0 (%)c=0.5c=0.5 (%)c=1c=1 (%)Timely correct targets
GPT-5.6-Luna1601147447.629.722.7112/258
Qwen3-VL-4B-Instruct2109198944.041.414.649/258
Qwen3-VL-8B-Instruct3752355451.628.419.969/258
Qwen3.5-9B4146387161.814.523.771/258
InternVL3.5-8B2912265755.428.815.861/258
Qwen3.5-4B1449143473.515.810.723/258
Em-Garde48439978.912.58.513/258
JoyVL4930430080.613.16.341/258
Qwen2.5-VL-3B-Instruct4409374584.39.56.228/258
GLM-4.1V-9B-Base372317.48.773.92/258
MMDuet2796985.57.27.20/258
Reviewer_AXiqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents P³IBench, a benchmark covering daily perception, risk alerts, and goal-conditioned intervention in streaming video. It contains 533 query samples from 429 videos, with 836 annotated response targets. The authors evaluate 12 models using PRS-U and PRS-B, which combine response correctness, temporal alignment, and redundancy penalties. They analyze response frequency, latency, and streaming configurations, and train P³I-Assistant-4B through supervised fine-tuning and reinforcement learning.

Critical strengths
  1. The topic of proactive video assistance is worth studying, particularly how assistants decide when to offer help and when to remain silent.

  2. The authors have put substantial effort into the benchmark, including manually recorded intervention scenarios and evaluations of both adapted offline and native proactive models.

Critical weaknesses
  1. The paper reports manual checking and verification, but more detail is needed to assess the claimed temporal precision. Reporting the results of a small manual spot-check would help support the validity of the temporal annotations, would make this part more convincing.

  2. Table 5 shows a substantial PRS-B improvement, from 28.62 to 42.12, but average PRS-U decreases from 24.57 to 22.28, while intervention performance remains essentially unchanged. This trade-off deserves more discussion when interpreting the training results.

I would be willing to raise my score if the authors satisfactorily address these questions.

Questions for authors

See above

2 comments
AuthorsOpenReview ↗

Response to Reviewer AXiq (Part 1 of 2)

Weakness 1: Temporal annotation quality and supporting manual verification.

Response:

We clarify the scope of human verification and report quantitative results from the human-checked internal validation set used during benchmark construction.

1. Human verification covers all annotations outside the automated Event Monitoring pipeline.

For Object Monitoring and Risk Alert, every temporal annotation was manually reviewed and corrected where necessary. Event Intervention was manually scripted, recorded, annotated, and verified, including the instructions, intervention windows, and reference responses. The main component requiring automated annotation is therefore Ego4D Event Monitoring, for which we developed a multi-stage pipeline with explicit rejection criteria.

2. Event Monitoring uses multiple checks to exclude unreliable annotations.

Starting from Ego4D's dense narrations, the pipeline generates candidate queries, identifies matching events, and estimates their temporal boundaries. Quality control operates at both the event-matching and boundary-localization stages:

  • Event matching: Queries are grounded in source narrations and checked against visual evidence. A candidate must pass confidence and source-consistency checks for its event matches; uncertain candidates are rejected before boundary annotation.
  • Boundary agreement: Two independent vision-language models estimate each event's start and end. A window proceeds only if their estimates differ by at most 2 seconds at each boundary.
  • Independent verification: A separate model checks event identity, semantic completeness, and visual support for the boundaries, without seeing the annotators' rationales or confidence scores. Windows are retained only if explicitly accepted, free of flagged boundary risks, and assigned an acceptance confidence of at least 0.9. A subsequent cross-window check removes conflicting annotations. Final evaluation targets are selected only from retained windows.

After applying this pipeline to construct the final Event Monitoring data, we additionally manually spot-checked 10% of the resulting samples. We found no evident event-matching errors or temporal-window misalignments in the inspected samples. This check on the final benchmark data complements the development-stage quantitative validation reported below.

3. Human-referenced validation quantifies the quality achieved by these checks.

During pipeline development, we manually curated 18 video-query cases from three videos, with six cases for each Event Monitoring subtype (atomic, event-type, and compositional), comprising 84 reference temporal windows. This set supported iterative refinement of the annotation prompts and filtering criteria.

For event matching, precision and recall were 100% and 98.79%, respectively, over event/component matches. The production confidence filter retained 11 of 18 cases, with no false-positive or false-negative matches in those retained cases.

We evaluated temporal localization separately by supplying reference event identities while withholding reference boundaries from all annotation and verification models. The resulting stage-wise measurements were:

Annotation stageWindows retained / 84Mean boundary error (s)95th-percentile boundary error (s)
Primary annotator840.8963.660
Secondary annotator841.4474.550
After agreement filtering620.5771.659
After verification, risk checks, and confidence filtering210.2470.779

For each window, boundary error is the average of its absolute start- and end-time errors relative to the human-checked reference; the table reports the mean and 95th percentile of this quantity over retained windows. Among the 21 retained windows, 18/21 (85.7%) had both boundaries within 1 second of the reference, and 21/21 (100%) within 2 seconds.

These construction-stage validation results show that the combined checks substantially reduce both average and tail errors in the retained annotations. The selective retention is intentional: benchmark construction prioritizes reliable temporal targets over accepting every candidate.

We will add a new appendix “Annotation Quality Control,” detailing the task-specific quality-control procedures, quantitative validation of temporal annotations, and manual spot-check results.

AuthorsOpenReview ↗

Response to Reviewer AXiq (Part 2 of 2)

Weakness 2: Post-training trade-offs across PRS-U, PRS-B, and Event Intervention.

Response:

Table 5 should be interpreted jointly across both metrics and task capabilities. The results demonstrate substantial gains in PRS-B, together with a remaining gap in unidirectional temporal alignment and limited transfer to Event Intervention. Three points help explain this pattern.

1. Reduced repetition and early triggering can affect PRS-U and PRS-B differently.

The additional task-specific SFT data come from Accident Warning and Event Monitoring, which call for one response per target. This supervision encourages the model to respond once to each target and then remain silent until another response is warranted.

This learned restraint interacts with early triggering. For a single-target task such as Accident Warning, PRS-U assigns zero credit to a response before the annotated target, whereas PRS-B can credit it with a temporal penalty. A model that answers early and repeats after the target can therefore obtain nonzero PRS-U despite the redundancy penalty. If SFT suppresses the later repeat while retaining the early answer, PRS-U becomes zero, while PRS-B can improve because the redundancy penalty is removed. Thus, the trade-off arises from the combination of reduced repetition and early responses.

An auxiliary SFT run illustrates this mechanism: for an Accident Warning target at 18.333 s, the base model responds at 9 s and 20 s, scoring 46.00 under both metrics. The SFT model responds only at 9 s, yielding 0 PRS-U but 62.71 PRS-B.

2. RL improves both metrics after SFT, while its reward permits residual early responses.

Averaging the five task scores in Table 5 gives:

Training stageAverage PRS-UAverage PRS-B
Base model24.5728.62
After SFT12.8836.72
After SFT + RL22.2842.12

The PRS-U decline occurs at the SFT stage; RL subsequently raises PRS-U from 12.88 to 22.28 while further improving PRS-B. On Accident Warning specifically, RL raises PRS-U from 2.96 to 32.88, exceeding the base model's 30.93, alongside a PRS-B increase from 61.78 to 87.59.

The temporal reward in Appendix C, exp⁡[−0.05∣t−tGT∣]\exp[-0.05|t-t_{\mathrm{GT}}|], encourages proximity to the target and is combined with penalties for extra responses. For an initially premature response, moving it later toward the target increases the reward, providing a mechanism consistent with the recovery after RL. However, the reward is symmetric: equally early and late responses at the same distance receive the same temporal credit, without requiring the response to occur at or after the target.

This design reflects our emphasis on anticipatory assistance, particularly in risk scenarios where advance notice can be more useful than a delayed warning. We therefore allow temporal credit on either side of the annotated target while penalizing distance from it. Consequently, a single response shortly before the target can still receive a high training reward but zero PRS-U on a single-target sample. This helps explain why RL improves both metrics without fully closing the PRS-U gap to the base model. The final 13.50-point PRS-B gain and 2.29-point PRS-U decrease capture this remaining timing trade-off.

3. Intervention requires capabilities beyond those directly targeted by the additional training data.

As detailed in Appendix C, the additional task-specific supervision covers Accident Warning and Event Monitoring, without corresponding Event Intervention training examples. Intervention requires tracking a user's goal and generating context-specific, open-ended reminders or corrections in our manually recorded scenarios. This difference in training coverage and task demands plausibly limits transfer, explaining why its score remains essentially unchanged (9.13 to 9.02). Meanwhile, PRS-B improves on Object Monitoring (33.82 to 40.03) and Crime Alert (37.03 to 51.21) despite neither receiving dedicated task-specific training data. These results support the value of targeted post-training and its transfer to related tasks, while identifying goal-conditioned intervention as a distinct capability requiring further attention.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Common Questions

We sincerely thank all reviewers for their constructive feedback. Several reviewers raised related questions about our temporal evaluation protocol, so we address them together below.

Common Question 1: The rationale for PRS-U and PRS-B and the interpretation of early responses.

Common Response1.1: What distinguishes PRS-U from PRS-B?

Our evaluation separates how close a response is to an annotated target from whether an early response is eligible for credit. The exponential factor exp⁡(−α∣t−τ∣)\exp(-\alpha|t-\tau|) was originally motivated by bidirectional timing, where a response can precede its target. In the current protocol, both PRS-U and PRS-B use this same factor for eligible response–target pairs. Their difference lies in the matching rule:

  1. PRS-U assigns a response to the most recent target at or before its emission time and gives no credit to responses before the first target.
  2. PRS-B assigns a response to the temporally nearest target and can therefore credit an early response matched to a future target.

Common Response1.2: Why report both metrics, and how should their gap be interpreted?

The two PRS variants are needed because the validity of an early response depends on the task and the visual evidence available at the response time:

  1. In Accident Warning, a warning before the annotated target may be useful if the available frames already establish the risk.
  2. In an occurrence-counting task, an answer before the specified occurrence is visually established is premature.

Reporting only PRS-B would conflate these cases, while reporting only PRS-U would give no credit to potentially useful anticipation. We therefore report both metrics and interpret their gap alongside the task and the available visual evidence.

Common Question 2: The rationale for exponential temporal decay and the choice of its parameter.

Common Response2.1: Why use exponential decay rather than a window-based timing metric?

We previously considered a window-based timing metric but ultimately chose exponential decay for two reasons:

  1. Exponential decay can approximate the practical effect of an effective temporal window. With an appropriately chosen α\alpha, responses close to the target retain credit, while those sufficiently far away contribute negligibly. More precisely, if Δ\Delta is the intended boundary of an effective window and ε\varepsilon is a negligible temporal weight, choosing α≥ln⁡(1/ε)/Δ\alpha \geq \ln(1/\varepsilon)/\Delta ensures that responses at or beyond Δ\Delta receive a temporal weight of at most ε\varepsilon.
  2. Exponential decay grades responses continuously by their temporal distance. Unlike a hard cutoff, it distinguishes responses at different distances from the target. That continuous signal also serves as the temporal reward in our RL setup, allowing post-training to distinguish those timing differences.

Common Response2.2: Why choose α=0.05\alpha=0.05 and how sensitive are model rankings to this choice?

We chose α=0.05\alpha=0.05 as a moderate temporal penalty: the temporal factor is approximately 0.610.61 at a 10-second offset and 0.370.37 at a 20-second offset. This setting penalizes substantial timing deviations while retaining partial credit, rather than rapidly driving temporally imperfect responses toward zero. We do not claim that this value is uniquely optimal.

To examine parameter sensitivity, we added Appendix C.2, “Sensitivity to the Temporal Decay Parameter” (Table 7). We rescore the fixed predictions of Qwen3-VL-8B-Instruct, GPT-5.6-Luna, VideoChat3, and Em-Garde at α∈{0.01,0.05,0.1,0.5}\alpha\in\{0.01,0.05,0.1,0.5\}, holding annotations, response matching, and content scores fixed. The experiment supports the stability of the leading overall ordering over the tested settings.

Note on corrected results: During the rebuttal process, we identified an error in the scores reported for the GPT models on the Intervene dataset in our previous experiments. We have corrected these scores in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.