- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper evaluates whether existing streaming video systems can make correct decisions at the appropriate time. The authors curate videos and introduce metrics, including timing accuracy (TA) and silence compliance (SC), to evaluate response behavior. Experiments cover a wide-range of streaming and agentic systems.
Critical strengths
The topic is timely, and the study is competently executed. Unlike conventional streaming benchmarks, where models are actually queried at predefined times, InteractionBench evaluates whether streaming and agentic systems can decide when to respond correctly and when to remain silent. Extensive analyses help readers to understand behaviors across different models, settings, and scenarios.
Critical weaknesses
Despite the comprehensive experiments, the analysis provides limited guidance on how to achieve the capabilities tested by InteractionBench. The failures observed across different models and settings are difficult to synthesize into a clear conclusion. In particular, it remains unclear which methods or configurations best balance timely, correct responses with appropriate silence, and which design choices should be prioritized to improve this balance.
Questions for authors
-
While agentic baselines are useful, could we test some tailored methods for a real-time interaction like IPI-Agent [1] or AsynKV [2]?
-
Furthermore, evaluation costs may increase with stream duration, limiting the benchmark’s practicality for evaluating long-running streaming systems. Although per-poll latency is reported in Appendix, but the total evaluation runtime and computational/API cost are unclear.
References
[1] IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streams, arxiv 2026
[2] Don’t Pause! Every prediction matters in a streaming video, arxiv 2026
3 comments
Q2. Evaluation cost versus stream duration; total runtime and computational/API cost
Thank you for the question. In the current version the videos average 2.7 minutes, and only 36 of the 812 videos exceed 5 minutes. The evaluation metric itself does not depend on duration. We measure per-poll latency to find out which way of serving a model suits real-time interaction; how to optimize the cost and performance of each system is a question of system design, and per-poll time also depends on the model and the hardware.
We measured the cost of one complete evaluation from the run logs (Tables 11 and 12 in the revision). The full set has 1,060 interactions on 812 videos, 35.9 hours of video. Turn-based models, API models and agents are polled once per second while an instruction is active, about 119,000 polls per evaluation, and native streaming systems run with their released mechanisms and input rates. As the table below shows, one evaluation takes 13 to 20 model hours for the configurations that keep pace with the stream, and larger windows multiply this cost.
| Configuration | Hardware | Model hours | Median time per poll | Returned within 1 s |
|---|---|---|---|---|
| Qwen3-VL-4B, polled | B200 MIG slice | 12.9 | 0.39 s | 100% |
| Qwen3-VL-8B, polled | B200 MIG slice | 17.1 | 0.48 s | 99% |
| LLaVA-OV2, polled | B200 MIG slice | 20.0 | 0.53 s | 95% |
| JoyAI-VL | B200 | 14.9 | 0.38 s | 99% |
| MiniCPM-o, polled | B200 MIG slice | 44.3 | 1.26 s | 3% |
| Qwen2.5-Omni, polled | B200 MIG slice | 65.6 | 1.93 s | 0% |
| Gemma-3n, polled | B200 MIG slice | 69.5 | 1.67 s | 2% |
| Flash-VStream | L40S | 103.1 | 6.95 s | 6% |
| Qwen3-VL-8B, reply history | B200 MIG slice | 36.9 | 0.83 s | 65% |
| Qwen3-VL-8B, 64 frames | B200 MIG slice | 44.6 | 1.35 s | 10% |
| Qwen3-VL-8B, 256 frames | B200 MIG slice | 130.2 | 4.12 s | 9% |
As the table below shows, model time per minute of video stays flat with a fixed window and grows for designs whose context grows over time.
| Model seconds per minute of video | Videos under 1 min | 1–3 min | 3–5 min | 5 min or more |
|---|---|---|---|---|
| Qwen3-VL-8B, fixed window | 23 | 24 | 25 | 25 |
| Qwen3-VL-8B, with reply history | 29 | 36 | 50 | 116 |
| Qwen3-VL-8B, 64-frame window | 52 | 60 | 65 | 73 |
| Qwen3-VL-8B, 256-frame window | 66 | 142 | 215 | 254 |
The table above shows that a fixed-window system needs about 24 model seconds per minute of video at every length, so its evaluation cost grows only linearly with stream length and stays below real time. A one-hour stream would take about 24 model minutes.
The video clock advances only after a poll returns, so a system of any speed can be evaluated, and replaying the logged poll times on the real clock shows what that speed costs at live pace. As the table below shows, only the lightweight open models with a fixed window keep up with the video clock. They drop by at most 7.6 in Timing, whereas API polling and agent loops lose most of their timing credit.
| Configuration | Score | Offline clock | Real clock |
|---|---|---|---|
| Qwen3-VL-8B, polled | Timing | 77.8 | 71.8 |
| Qwen3-VL-4B, polled | Timing | 34.0 | 30.3 |
| LLaVA-OV2, polled | Timing | 81.0 | 73.4 |
| Claude Sonnet 5, polled | Timing | 82.9 | 45.5 |
| Gemini 2.5 Flash, polled | Timing | 93.1 | 51.6 |
| Claude Sonnet 5 agent, Qwen-MM-Plugins | Overall | 56.0 | 29.2 |
| Claude Sonnet 5 agent, Claude Code tools | Overall | 61.2 | 28.5 |
For the systems called through an API, the table below gives the cost of one evaluation on all 1,060 interactions. For the first three rows, we rebuilt every request and counted tokens with the providers' token-counting endpoints. The tokens of the GPT-5.6 Sol agent come from its run ledger at list prices. A dash marks a quantity that the run did not log.
| Configuration | Input tokens | Cost | Median time per poll |
|---|---|---|---|
| Claude Sonnet 5, polled | about 440 M | USD 970 | 2.4 s |
| Gemini 2.5 Flash, polled | about 509 M | USD 150 | 1.9 s |
| Claude Sonnet 5, whole video | 3.9 M | USD 9 | one call per item |
| GPT-5.6 Sol agent, polled | about 3,150 M | USD 12,400 | – |
| GPT-5.6 Sol agent, whole video | 372 M | USD 1,450 | – |
| Claude Sonnet 5 agent, Claude Code tools, polled | – | – | 37.2 s |
| Claude Sonnet 5 agent, Qwen-MM-Plugins, polled | – | – | 28.7 s |
| Claude Sonnet 5 agent, whole video | – | – | 126 s per item |
| JoyAI-VL controller with Claude Sonnet 5 | – | – | 3.3 s |
| JoyAI-VL controller with Claude Opus 5 | – | – | 4.2 s |
The table above shows that a polled agent loop costs about ten times more than one call per second, and that answering from the whole video is the cheapest.
Scoring adds little. The local judge, Qwen3-14B, scores the free-text answers of one configuration in about 5 minutes on one B200 (2.2 GPU-hours for all configurations).
Official comment by Reviewer wYLy
Thanks for the rebuttal with detailed answers. Most of my concerns are addressed, and I have no further questions.
W1. Limited guidance on how to balance timely, correct responses with silence, and what to prioritize
Thank you for the suggestion. We summarized the results in Figure 9, Section 3.5 and Appendix S of the revision. Response control should be prioritized over additional input, with selective suppression of responses first, task-state memory second, and more frames or agent loops last.
The table below collects the configurations that best balance timing and silence.
The table above shows that at live pace no system does both well. No configuration that keeps pace with the video clock exceeds 40 on both Timing and Silence. The two that do, MiniCPM-o with reply history and the agent loops, take 1.05 s and about half a minute per poll.
Selective suppression comes first, because deleting only the premature, repeated and negative-stream responses brings silence compliance to 100 with timing unchanged, whereas a stricter response threshold only trades timing for silence (Figure 4 and Appendix S). Memory comes second and has to hold task state. On counting instructions, reply history lowers missed later events for Qwen3-VL-8B and LLaVA-OV2 from 20.9% to 9.4% and from 23.9% to 7.5%, but Gemma-3n and Qwen2.5-Omni then stop after their first answer on 44.1% and 24.7% of items (Figure 27 in the revision). More frames and agent loops come last because their gains do not hold at live pace. Replayed on the real clock, the Timing of the agent loops falls to 0.3 (Appendix L).
Q1. Methods tailored to real-time interaction such as IPI-Agent or AsynKV
We agree that methods designed for real-time interaction are more targeted than general tool-calling loops alone. We therefore tested AsynKV on all 1,060 interactions (new row of Table 2 in the revision). The official IPIBench repository has released the benchmark data and a reference evaluation pipeline for Qwen3-VL-8B, but the code of IPI-Agent has not been released yet, and we will add it to the evaluation once the code is public. We cite IPIBench in Section 2.4 and IPI-Agent in the Related Work (Appendix A in the revision). The table below places AsynKV beside the related systems.
The table above shows that AsynKV makes a different trade-off. Its silence compliance is the highest among all real-time configurations, while its content and timing scores are low. The table below shows the reason. A gate trained for when to speak has learned to be conservative, and it also suppresses the instructions that need continued responses.