acceptodds
Under review as a conference paper at ICLR 2027

InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systems

Abstract

A video assistant must speak when its instruction warrants a response and stay silent otherwise. We introduce a benchmark that evaluates this decision for the complete system of model, memory, and response controller. InteractionBench covers query responses, event triggers, and ongoing updates in 1,060 interactions over 812 videos, with 69 negative streams and 53 suites that pair counted events with look-alike near misses. It scores content accuracy, timing accuracy, and silence compliance on the video clock and reports wall-clock latency separately. Timely speech costs silence across systems. Polled Qwen3-VL-8B reaches 77.8 timing accuracy but 10.9 silence compliance. A native real-time interaction system reaches 29.2 silence compliance at 66.8 timing accuracy, yet emits on 89.9% of negative streams. No open-weight system clears a third of the near-miss suites. Fewer replies help only when chosen, as random deletion merely trades timing for silence. Offline scores miss these failures and mispredict online behavior. Adding restraint is costly, as the native system's controller adds little by itself and agent harnesses add it only at about 30 s per poll.

Primary area:
datasets and benchmarks
Keywords:
benchmark, real-time interaction, video understanding, streaming video

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.50 over 2 reviews, above 99% of the others.
Reviewer_wYLyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates whether existing streaming video systems can make correct decisions at the appropriate time. The authors curate videos and introduce metrics, including timing accuracy (TA) and silence compliance (SC), to evaluate response behavior. Experiments cover a wide-range of streaming and agentic systems.

Critical strengths

The topic is timely, and the study is competently executed. Unlike conventional streaming benchmarks, where models are actually queried at predefined times, InteractionBench evaluates whether streaming and agentic systems can decide when to respond correctly and when to remain silent. Extensive analyses help readers to understand behaviors across different models, settings, and scenarios.

Critical weaknesses

Despite the comprehensive experiments, the analysis provides limited guidance on how to achieve the capabilities tested by InteractionBench. The failures observed across different models and settings are difficult to synthesize into a clear conclusion. In particular, it remains unclear which methods or configurations best balance timely, correct responses with appropriate silence, and which design choices should be prioritized to improve this balance.

Questions for authors
  1. While agentic baselines are useful, could we test some tailored methods for a real-time interaction like IPI-Agent [1] or AsynKV [2]?

  2. Furthermore, evaluation costs may increase with stream duration, limiting the benchmark’s practicality for evaluating long-running streaming systems. Although per-poll latency is reported in Appendix, but the total evaluation runtime and computational/API cost are unclear.

References

[1] IPIBench: Evaluating Interactive Proactive Intelligence of MLLMs under Continuous Streams, arxiv 2026

[2] Don’t Pause! Every prediction matters in a streaming video, arxiv 2026

3 comments

W1. Limited guidance on how to balance timely, correct responses with silence, and what to prioritize

Thank you for the suggestion. We summarized the results in Figure 9, Section 3.5 and Appendix S of the revision. Response control should be prioritized over additional input, with selective suppression of responses first, task-state memory second, and more frames or agent loops last.

The table below collects the configurations that best balance timing and silence.

ConfigurationTiming (TA)Silence (SC)OverallMedian time per pollKeeps pace with the stream
Qwen3-VL-8B, polled77.810.934.80.48 syes
LLaVA-OV2, polled81.016.934.50.53 syes
JoyAI-VL66.829.233.80.38 syes
Qwen3-VL-4B, polled with reply history57.536.034.60.67 syes
AsynKV19.678.027.6streamingyes
MiniCPM-o, polled with reply history60.548.539.11.05 sno
Qwen3-VL-8B, 256-frame window81.317.239.04.1 sno
Claude Sonnet 5 agent with Claude Code tools70.755.561.237.2 sno
GPT-5.6 Sol agent with Qwen-MM-Plugins68.660.059.1not loggedno
Claude Sonnet 5 agent with Qwen-MM-Plugins66.555.956.028.7 sno

The table above shows that at live pace no system does both well. No configuration that keeps pace with the video clock exceeds 40 on both Timing and Silence. The two that do, MiniCPM-o with reply history and the agent loops, take 1.05 s and about half a minute per poll.

Selective suppression comes first, because deleting only the premature, repeated and negative-stream responses brings silence compliance to 100 with timing unchanged, whereas a stricter response threshold only trades timing for silence (Figure 4 and Appendix S). Memory comes second and has to hold task state. On counting instructions, reply history lowers missed later events for Qwen3-VL-8B and LLaVA-OV2 from 20.9% to 9.4% and from 23.9% to 7.5%, but Gemma-3n and Qwen2.5-Omni then stop after their first answer on 44.1% and 24.7% of items (Figure 27 in the revision). More frames and agent loops come last because their gains do not hold at live pace. Replayed on the real clock, the Timing of the agent loops falls to 0.3 (Appendix L).

Q1. Methods tailored to real-time interaction such as IPI-Agent or AsynKV

We agree that methods designed for real-time interaction are more targeted than general tool-calling loops alone. We therefore tested AsynKV on all 1,060 interactions (new row of Table 2 in the revision). The official IPIBench repository has released the benchmark data and a reference evaluation pipeline for Qwen3-VL-8B, but the code of IPI-Agent has not been released yet, and we will add it to the evaluation once the code is public. We cite IPIBench in Section 2.4 and IPI-Agent in the Related Work (Appendix A in the revision). The table below places AsynKV beside the related systems.

SystemOverallContentTimingSilence
Qwen3-VL-8B, polled34.854.277.810.9
AsynKV (new)27.625.619.678.0
Qwen3-VL-8B, blind15.79.032.644.2
VideoLLM-online14.53.733.647.9
Flash-VStream14.321.550.28.2

The table above shows that AsynKV makes a different trade-off. Its silence compliance is the highest among all real-time configurations, while its content and timing scores are low. The table below shows the reason. A gate trained for when to speak has learned to be conservative, and it also suppresses the instructions that need continued responses.

Behavior of AsynKVValue
Negative streams with a response0 of the 69 in the benchmark
Look questions answered correctly66.9%
Recall questions answered correctly12.8%
Counting instructions with no response at all159 of the 160 that contain events to count
Commentary items with a response10.3%

Q2. Evaluation cost versus stream duration; total runtime and computational/API cost

Thank you for the question. In the current version the videos average 2.7 minutes, and only 36 of the 812 videos exceed 5 minutes. The evaluation metric itself does not depend on duration. We measure per-poll latency to find out which way of serving a model suits real-time interaction; how to optimize the cost and performance of each system is a question of system design, and per-poll time also depends on the model and the hardware.

We measured the cost of one complete evaluation from the run logs (Tables 11 and 12 in the revision). The full set has 1,060 interactions on 812 videos, 35.9 hours of video. Turn-based models, API models and agents are polled once per second while an instruction is active, about 119,000 polls per evaluation, and native streaming systems run with their released mechanisms and input rates. As the table below shows, one evaluation takes 13 to 20 model hours for the configurations that keep pace with the stream, and larger windows multiply this cost.

ConfigurationHardwareModel hoursMedian time per pollReturned within 1 s
Qwen3-VL-4B, polledB200 MIG slice12.90.39 s100%
Qwen3-VL-8B, polledB200 MIG slice17.10.48 s99%
LLaVA-OV2, polledB200 MIG slice20.00.53 s95%
JoyAI-VLB20014.90.38 s99%
MiniCPM-o, polledB200 MIG slice44.31.26 s3%
Qwen2.5-Omni, polledB200 MIG slice65.61.93 s0%
Gemma-3n, polledB200 MIG slice69.51.67 s2%
Flash-VStreamL40S103.16.95 s6%
Qwen3-VL-8B, reply historyB200 MIG slice36.90.83 s65%
Qwen3-VL-8B, 64 framesB200 MIG slice44.61.35 s10%
Qwen3-VL-8B, 256 framesB200 MIG slice130.24.12 s9%

As the table below shows, model time per minute of video stays flat with a fixed window and grows for designs whose context grows over time.

Model seconds per minute of videoVideos under 1 min1–3 min3–5 min5 min or more
Qwen3-VL-8B, fixed window23242525
Qwen3-VL-8B, with reply history293650116
Qwen3-VL-8B, 64-frame window52606573
Qwen3-VL-8B, 256-frame window66142215254

The table above shows that a fixed-window system needs about 24 model seconds per minute of video at every length, so its evaluation cost grows only linearly with stream length and stays below real time. A one-hour stream would take about 24 model minutes.

The video clock advances only after a poll returns, so a system of any speed can be evaluated, and replaying the logged poll times on the real clock shows what that speed costs at live pace. As the table below shows, only the lightweight open models with a fixed window keep up with the video clock. They drop by at most 7.6 in Timing, whereas API polling and agent loops lose most of their timing credit.

ConfigurationScoreOffline clockReal clock
Qwen3-VL-8B, polledTiming77.871.8
Qwen3-VL-4B, polledTiming34.030.3
LLaVA-OV2, polledTiming81.073.4
Claude Sonnet 5, polledTiming82.945.5
Gemini 2.5 Flash, polledTiming93.151.6
Claude Sonnet 5 agent, Qwen-MM-PluginsOverall56.029.2
Claude Sonnet 5 agent, Claude Code toolsOverall61.228.5

For the systems called through an API, the table below gives the cost of one evaluation on all 1,060 interactions. For the first three rows, we rebuilt every request and counted tokens with the providers' token-counting endpoints. The tokens of the GPT-5.6 Sol agent come from its run ledger at list prices. A dash marks a quantity that the run did not log.

ConfigurationInput tokensCostMedian time per poll
Claude Sonnet 5, polledabout 440 MUSD 9702.4 s
Gemini 2.5 Flash, polledabout 509 MUSD 1501.9 s
Claude Sonnet 5, whole video3.9 MUSD 9one call per item
GPT-5.6 Sol agent, polledabout 3,150 MUSD 12,400–
GPT-5.6 Sol agent, whole video372 MUSD 1,450–
Claude Sonnet 5 agent, Claude Code tools, polled––37.2 s
Claude Sonnet 5 agent, Qwen-MM-Plugins, polled––28.7 s
Claude Sonnet 5 agent, whole video––126 s per item
JoyAI-VL controller with Claude Sonnet 5––3.3 s
JoyAI-VL controller with Claude Opus 5––4.2 s

The table above shows that a polled agent loop costs about ten times more than one call per second, and that answering from the whole video is the cheapest.

Scoring adds little. The local judge, Qwen3-14B, scores the free-text answers of one configuration in about 5 minutes on one B200 (2.2 GPU-hours for all configurations).

Reviewer_wYLyOpenReview ↗

Official comment by Reviewer wYLy

Thanks for the rebuttal with detailed answers. Most of my concerns are addressed, and I have no further questions.

Reviewer_kkoeOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work introduces InteractionBench, a large benchmark of ~1.06k interactions over 812 unique videos spanning 10 domains that test whether a streaming video assistant speaks at the right moment of the 6 tasks covering answering queries, waiting for an event and tracking/commenting in the video or stays silent. Unlike prior benchmarks, this dataset has included negative streams and near misses that measure robustness of the model on false alarms. They evaluate native streaming models, polling turn-based models and agentic systems. They evaluate multiple metrics of content accuracy, timing accuracy and silence compliance. Their results demonstrate that timely response costs silence and overall no open-weight system passes more than a third of the near-miss suites.

Critical strengths
  1. Real human-annotated data curated and filtered by both automatic and human screening. Prior benchmarks focused on proactiveness i.e., response and its accuracy, in addition, to that interaction benchmark also focused on difficult data points with negative streams and near misses and scores on content accuracy, timing and silence.

  2. Well compared with prior work on both dataset dimensions and evaluations in Table1.

  3. Dataset captures diverse 6 set of interaction tasks and videos from different domains so has good coverage. And has been evaluated on different type of systems and models (native streaming, turn-based, and agentic).

  4. Extensive analysis of different types of systems, including offline-grounding systems. Analysis that shows timely speech comes at the cost of silence and different types of failures. And thorough latency analysis.

Critical weaknesses
  1. Why is blind baseline better than some of the systems reported in native streaming and turn-based polling?

  2. It is not clear that only the results in each section are comparable to each other in Table 1, as they are evaluated on different sample sizes?

  3. Some subsections have too many numbers in text (later subsections of section3) makes it hard to follow.

  4. Sourcing and the annotation details could be more elaborate. No details of how many human annotators or the whole pipeline of the annotation is mentioned. Recommend adding a figure to explain this. Its important to know these details as its benchmark paper.

Questions for authors
  1. Whats the rationale behind overall score computation? Why is Acc weighted higher than TA and SC?

  2. Thank you for adding extensive failure analysis. Is there a correlation of errors by domain, length of the video? Can you also show some examples of failures that can give insight into content errors? Or what triggers false alarms?

  3. From quick observation, it seems like the videos are relatively focused. How visually focused are these videos? Is there a correlation between false alarms and near misses with an increase in scene clutter? How do these results translate to some real-world data?

  4. Why did you choose a Qwen based judge to grade text responses? Gemini judge mentioned but its not clear which Gemini model?

6 comments

W1. Why the blind baseline scores above some systems

The blind run is a diagnostic reference, not a floor that every system with vision must exceed. Its 15.7 points come mostly from staying silent, plus a limited language prior on multiple-choice items; the same backbone with frames scores 34.8. Its content score is only 9.0 (Table 15 in the revision), and it passes none of the near-miss suites (Figure 5). Positive items on which a system never speaks do not enter the SC aggregate, so a low false-alarm rate and its SC alone do not show that it can tell when to answer from when to stay silent. Its place above some systems thus results from how Overall balances content, timing and silence, not from the design of the benchmark.

The two tables below give the points that each group of items contributes to Overall and the behavior of each system (Figure 26 in the revision).

Item groupQwen3-VL-8B, blindQwen3-VL-8B, polled with framesVideoLLM-onlineFlash-VStreamLiveCC
Negative streams6.43.90.04.00.0
Multiple-choice, content4.021.11.19.210.7
Multiple-choice, timing and silence4.82.511.60.10.6
Segment0.15.80.00.22.7
Counting0.00.70.80.60.0
Open-ended0.50.90.90.20.5
Overall15.734.814.514.314.5
BehaviorQwen3-VL-8B, blindQwen3-VL-8B, polled with framesVideoLLM-onlineFlash-VStreamLiveCC
Positive items with a response34.6%89.0%95.7%58.7%85.6%
Negative streams that trigger1.4%40.6%100%39.1%97.1%
Accuracy when answering a multiple-choice item29.5%75.6%3.7%49.6%7.4%
Chance level on those items17.4%19.1%19.1%18.9%18.1%
Mean score of positive items without options2.025.76.03.45.7

The two tables above show that the blind run scores by staying silent. The negative streams give it 6.4 of its 15.7 points and it never speaks on a counting item, whereas the same backbone with frames gains 19.1 points, mostly from content. The three systems below the blind run lose on silence and output format, not on perception. VideoLLM-online speaks on every negative stream and 75.4% of its replies to multiple-choice items name no option, LiveCC has no silence mechanism, so it earns nothing on the negative streams and only 0.6 points from timing and silence, and Flash-VStream answers repeatedly on 98% of items, which sets its SC to 0 on those items.

The language prior also plays a part. Qwen3-VL-8B, the model of the blind run, was released in October 2025, 14 to 18 months after the language backbones of the three systems ranked below it. VideoLLM-online builds on Llama-3-8B-Instruct (April 2024), and LiveCC and the Flash-VStream checkpoint we ran build on Qwen2-VL-7B (August 2024). A newer backbone can guess from the options without frames, but this does not carry over to questions without options.

W2. Comparability of results evaluated on different sample sizes

In Table 1 we do not claim an advantage in sample size. As the reviewer notes among the strengths, the benchmark compares well with prior work on the dataset dimensions and evaluations listed there. We report the sample size only to give readers a sense of the benchmark's scale and to show that it is of the same order as existing benchmarks, large enough for the results to be informative.

In addition, the main results in Table 2 use the same complete evaluation set of 1,060 interactions. The experiments were run in batches, and the final results aggregate all batches. We apologize that some descriptions gave the impression that different systems were evaluated on different subsets, and we have corrected the description of the runs.

W3. Too many numbers in the later subsections of Section 3

We agree that too many numbers in running text hurt readability. In the revision, we reworded the most number-dense passages of Sections 3.2 and 3.5, so that they state the comparison and leave the numbers to the tables and figures.

W4. Sourcing and annotation details; number of annotators; a pipeline figure

Thank you for the suggestion. We agree that a benchmark paper should document how its data were sourced and annotated. Eight graduate students annotated the benchmark, and we added a description of the six stages, a pipeline figure with the count kept after each stage, and a view of the annotation interface (Appendix D, Figures 11 and 12 in the revision). The table below lists the stages.

StageWhat is doneKept after the stage
1. Retrieval and automatic filteringSearch queries cover a grid of domains and tasks, with at most eight videos per query and duplicates removed; the filters target clips of 20 s to 5 min, a range that holds 94.7% of the final videos, and also examine resolution, view count and upload date4,851 candidate videos
2. Human screeningAnnotators remove compilations, videos whose timeline resets and videos whose overlays reveal the answer1,089 videos
3. AnnotationAnnotators cross-annotate the same batch of videos, write the instruction, the reference answers and the earliest answerable time, and keep the harder questions; 276 videos yield no usable interaction813 videos, 1,061 interactions
4. Options and text-only auditA language model drafts up to five distractors per question, each a semantic neighbor of the answer, another object or action in the same scene, a state that holds before the event, a neighboring number, or a wrong order; three language models then answer without frames, and questions they answer correctly go back to the annotators230 flagged questions reviewed and revised where needed
5. Near-miss miningIn counting videos we mine moments that look like the counted event but do not satisfy the condition; candidates from 86 videos are checked frame by frame53 suites with 106 near misses
6. Cross-checkThe annotators cross-check the final items, in particular the items that every system answers wrong, to rule out errors in the ground truth; one interaction is removed812 videos, 1,060 interactions

Q1. Rationale of the Overall score; is Acc weighted higher than TA and SC

As introduced in Section 2.2, Overall=12[Acc+H(TA,SC)]\mathrm{Overall} = \frac{1}{2}\left[\mathrm{Acc} + H(\mathrm{TA}, \mathrm{SC})\right] gives equal weight to content correctness and to interaction behavior. Acc carries half of the weight because it is the only content score, while TA and SC both describe when the system responds and share the other half. For a real-time system, when it responds matters as much as what it says. The harmonic mean is used there because each of the two has a trivial maximizer, as always responding maximizes TA and never responding maximizes SC. Averaging it with Acc keeps partial credit for correct content when the behavior fails.

In Appendix H, we varied the weights. Weighting timing against silence, with HβH_\beta at β\beta = 0.5 or 2, keeps the ranking at Kendall τ of at least 0.972, whereas a content weight of 0.25 or 0.75 gives 0.733 and 0.825. The ranking is therefore stable to the balance between timing and silence and more sensitive to the weight of content.

We further tested the aggregation against three alternatives, rescoring two scripted policies of the Protocol Audit and the 20 full-set configurations under each (Figure 13 in the revision). The alternatives are an arithmetic mean in place of the harmonic mean, the plain average of the three scores per item, and the average of the three columns after each is averaged over the full set.

AggregationScore of always respondingScore of never respondingReal configurations that score below a trivial policyKendall τ with the paper's ranking of the 20 real configurations
Paper, 12[Acc+H(TA,SC)]\frac{1}{2}[\mathrm{Acc} + H(\mathrm{TA}, \mathrm{SC})]0.06.50 of 20–
Arithmetic mean, 12[Acc+(TA+SC)/2]\frac{1}{2}[\mathrm{Acc} + (\mathrm{TA} + \mathrm{SC})/2]20.66.53 of 200.78
Mean of Acc, TA and SC per item27.56.56 of 200.75
Mean of the three columns32.933.37 of 200.77

The table above shows that the harmonic mean is the term that prevents both always responding and never responding from scoring. Under every alternative, the always-responding policy overtakes 3 to 7 real configurations, while the order of the real configurations is largely preserved. We agree that the weighting is a modeling choice and not the only reasonable utility.

Q2. Errors by domain and video length; examples of failures

Thank you for these valuable suggestions. We added an analysis of errors by video length and domain over the 19 full-set systems, together with verified failure examples (new Appendix T in the revision). Errors correlate weakly with video length and more clearly with domain. Longer videos raise missed events slightly and lower the accuracy of questions about earlier content, and domains differ in how often the content is wrong.

As shown in the table below (Table 24 in the revision), with system, task, domain and event density held fixed, each doubling of the stream duration raises missed events by only 2.4 per 100 events, against an average of 57.9, while the rate of wrong content does not change, so longer videos cost detection, not content. Negative streams draw a response more often as they grow longer, but false alarms per minute fall from 3.7 under one minute to 0.7 beyond five minutes, so this rise reflects longer exposure. For questions asked at one moment, accuracy falls from 66.2 with under 30 s of preceding video to 34.5 with 180 s or more, where most questions are Recall.

OutcomeChange per doubling of stream duration
Missed events per 100 events+2.4
Events answered with wrong content per 100 events+0.0
Negative streams with at least one response, %+8.5
Accuracy of questions asked at one moment, per doubling of the preceding video−2.2

With task, duration and event density held fixed, the rate of wrong content still differs by up to 10.5% between domains (Wald p = 7×10⁻⁶), and the rate of missed events by up to 7.0% (p = 0.02). The table below (Table 25 in the revision) gives both rates by domain, each as the median across the 19 systems, which is not dominated by the few systems that respond at almost every poll.

DomainItemsMissed events, %Wrong content, %
Cooking and kitchen12773.415.9
Crafts, creation and skills4674.419.8
Medical, first-aid and care4452.219.6
Home and life skills11566.216.7
Games, AR and entertainment8581.413.2
Professional, office and accessibility10468.818.2
Sports and exercise coaching25477.913.9
Driving, traffic and mobility7669.912.3
Industrial, repair and operations2457.429.8
Education and children18559.216.8

False alarms are triggered by moments that resemble the target and not by the scene. Each level of near-miss content raises the share of negative streams that trigger by 11.8%.

For the failure examples, we added five verified trajectories of polled Qwen3-VL-8B, each typical of its error type (Figure 28 in the revision), and the table below summarizes them.

CaseWhat the system doesError type
Counting gymnastsAnswers three on time when five are present and two of them overlapContent error under occlusion
Fall alertDescribes a fall as a deliberate crouchNear state
Counting exercisesCounts a level-introduction screen as four performed exercisesFalse alarm on a human-verified near miss, in a clean scene
Welding-helmet alertGives the correct alert 0.35 s after the event, with eight responses before it and ten after itPremature and repeated responses
Pen changesMisses eight of nine pen changesMissed events

Comments on the paper

4 comments
AuthorsOpenReview ↗

Common Concerns: Evaluation Coverage, Metric Design, and What the Failures Tell Us

We sincerely thank all reviewers for their valuable feedback and constructive comments. We are encouraged that the reviewers find the problem timely and well motivated, value the negative streams, the near-miss suites and the separate scores for content, timing and silence, and regard the analysis across native streaming, turn-based, grounding and agentic systems as extensive. Below, we summarize and address the concerns raised by more than one reviewer before providing reviewer-specific responses.

Q1: Coverage of the main results and the human reference (Reviewer kkoe and Reviewer WyXK). The main results in Table 2 are computed on the same complete evaluation set of 1,060 interactions. The experiments were run in batches, and the reported numbers aggregate all batches. We apologize that parts of the submission gave the impression that different systems were evaluated on different subsets, and we have corrected the description of the runs.

For the human reference, we agree that the original setting differed from the systems' protocol, and we re-collected it. Eight graduate students took part and together cover all items; no participant answers an item from a video they may have annotated. Multiple-choice questions show the same stem and options that the systems see and are answered with option letters, so the human reference is scored by the same scorer and judge as every system. The new human reference reaches an Overall of 79.3 and an average accuracy of 81.9.

Q2: Whether the findings depend on how the scores are defined and judged (Reviewer kkoe and Reviewer WyXK). Overall weighs what is said and when it is said equally, and timing and silence share the second half through a harmonic mean because each has a trivial maximizer. We ran experiments with content-gated timing, the smooth silence variant, alternative aggregations and different LLM judges, and show that the timing–silence trade-off and the ranking of systems hold in all of them.

At Reviewer WyXK's suggestion, we define a timing score conditioned on correct content, TA_correct, and a smooth silence variant, SC_smooth. TA_correct credits timing only when the first matched response is correct; with it, the Overall ranking of the 20 full-set configurations agrees with the default at Kendall τ = 0.86, with SC_smooth at 0.83, and with both changes at 0.81. Among the ten full-set online systems, timing and silence remain negatively rank-correlated, which shows that the trade-off between them does not come from how the two scores are defined.

On the aggregation that Reviewer kkoe asks about, a policy that always responds scores 0.0 under the paper's definition and one that never responds scores 6.5, below all 20 real configurations. Replacing the harmonic mean with an arithmetic mean, or averaging the three scores directly, lifts the always-respond policy to 20.6–32.9, above 3 to 7 real systems. The harmonic mean is therefore the term that keeps these two trivial policies from scoring.

We also rescored all configurations with Qwen3-32B, gpt-oss-20b, Gemma-3-27B and Claude Opus 5.5. The Overall ranking agrees with the Qwen3-14B judge at τ between 0.959 and 0.977 and no system's Overall moves by more than 2.0 points, so our conclusions do not depend on a particular judge.

Q3: Where the failures come from and which design choices to prioritize (Reviewer kkoe and Reviewer wYLy). We added an error analysis by domain, duration and scene complexity, with online and offline systems analyzed separately. The failures are tied to the target and to response control, not to background clutter or to the real-time setting itself.

Scene clutter does not raise false alarms or premature responses. The difficulty comes from the target. Each level of near-miss content raises the share of items with a premature response by 7.9%, and each level of occlusion raises missed events by 5.6%, equally for online and offline systems. What the real-time setting changes is how often the same error is repeated. Per level of near-miss content, false alarms per minute rise by 2.35 online and by 0.10 offline.

For design, response control should be prioritized over additional input, with selective suppression of responses first, task-state memory second, and more frames or agent loops last. Removing only premature, repeated and negative-stream responses raises silence compliance to 100 with timing unchanged, whereas a stricter threshold trades timing for silence. Reply history alone reduces missed later events for two backbones but causes two others to stop after their first answer. Larger frame windows and agent loops improve scores only at latencies that cannot keep pace with the stream.

We again appreciate the reviewers' comments and provide detailed responses below. We have also updated the PDF and highlighted all revisions in blue for clarity, and we are happy to address any additional clarifications or follow-up questions.

W1 / Q1 (1/2). The timing–silence trade-off may partly follow from the metric definitions; content-conditioned timing and the smooth silence variant

We agree that timing credit should take content correctness into account, and that silence compliance should also separate weak systems with many violations. We therefore added a content-conditioned timing score, TAcorrect\mathrm{TA_{correct}}, and a smooth silence score, SCsmooth\mathrm{SC_{smooth}}, recomputed the ranking of every full-set configuration, and found that the ranking and the conclusion about timing and silence remain unchanged.

The paper scores timing and content separately by design. A response is matched to an event by its position in the response window and Acc grades its content, so a prompt but wrong response earns timing credit without content credit, and Overall=12[Acc+H(TA,SC)]\mathrm{Overall} = \frac{1}{2}\left[\mathrm{Acc} + H(\mathrm{TA}, \mathrm{SC})\right] combines the two. This separation attributes a failure to content or to timing and keeps TA and SC independent of the LLM judge, whereas content-aware matching would reward emitting candidate responses until one fits. SC reaches zero when the violations equal the number of reference events. We agree that this is the case on 71.7% to 97.4% of the positive items of the polled models (Table 19 in the revision), where Overall then depends only on content accuracy, and SCsmooth\mathrm{SC_{smooth}} below tests whether the findings rest on this floor.

In Appendix H, we varied the delay scale from 2 to 10 seconds and shifted or jittered the reference timestamps by one second, and the Kendall τ of the Overall ranking stays at or above 0.917; with SC in its smooth form, the τ of the silence ranking is 0.954. The ordering of systems therefore depends neither on the timing tolerance nor on the floor of SC.

For timing conditioned on correct content, we keep the original event windows and first-response matching and add

TAcorrect=100N∑n=1Ncnmax⁡(0,1−dn5),\mathrm{TA_{correct}} = \frac{100}{N}\sum_{n=1}^{N} c_n \max\left(0, 1 - \frac{d_n}{5}\right),

where NN counts all reference events, dnd_n is the delay in seconds of the first matched response to event nn, and cnc_n is 1 when that response is correct and 0 otherwise. For counting tasks, a response is correct when it states the running count after its event. The column "Overwritten after a correct first response" of the table below additionally reports the share of events whose first correct response is later overwritten by a wrong one, with the events whose first matched response is correct as the denominator.

For silence, we adopt the smooth form of the appendix, SCsmooth=1001+V/N\mathrm{SC_{smooth}} = \frac{100}{1 + V/N}, where VV is the number of silence violations such as premature or repeated responses. We replace SC with SCsmooth\mathrm{SC_{smooth}} on the applicable items, keep all other scoring conventions, and recompute Overall per item before aggregating, which gives Overallsmooth=12[Acc+H(TA,SCsmooth)]\mathrm{Overall_{smooth}} = \frac{1}{2}\left[\mathrm{Acc} + H(\mathrm{TA}, \mathrm{SC_{smooth}})\right].

W1 / Q1 (2/2). The timing–silence trade-off may partly follow from the metric definitions; content-conditioned timing and the smooth silence variant

Against the default Overall ranking of the 20 full-set configurations, Kendall τ is 0.86 with TAcorrect\mathrm{TA_{correct}} in place of TA, 0.83 with SCsmooth\mathrm{SC_{smooth}} in place of SC, and 0.81 with both changes, so the order of systems is largely preserved under each scoring. The per-system results are in the table below (Table 7 in the revision), with ranks in parentheses. TA in this table is averaged over all reference events in the same way as TAcorrect\mathrm{TA_{correct}}, so its values differ slightly from Table 2.

SystemSettingTATAcorrect\mathrm{TA_{correct}}Overwritten after a correct first responseOverallwith TAcorrect\mathrm{TA_{correct}}with SCsmooth\mathrm{SC_{smooth}}with both
Claude Sonnet 5 agentoffline grounding69.260.54.2%66.1 (1)62.4 (1)66.6 (1)62.9 (1)
GPT-5.6 Sol agentoffline grounding60.853.74.2%58.8 (2)55.9 (2)59.6 (2)56.4 (2)
Claude Sonnet 5offline grounding35.827.40.3%38.6 (3)35.1 (3)38.7 (6)35.2 (5)
Qwen3-VL-8Boffline grounding39.925.78.4%35.7 (4)30.0 (7)36.5 (7)30.5 (7)
Qwen3-VL-8Bpolled75.650.620.6%34.8 (5)33.7 (4)40.6 (4)37.8 (3)
LLaVA-OV2polled78.945.922.2%34.5 (6)31.3 (5)40.2 (5)35.0 (6)
LLaVA-OV2offline grounding39.324.14.5%33.9 (7)27.6 (9)34.8 (8)28.3 (11)
JoyAI-VLnative streaming66.039.18.0%33.8 (8)31.1 (6)41.1 (3)36.5 (4)
Qwen3-VL-4Boffline grounding39.125.33.6%33.8 (9)28.4 (8)34.8 (9)29.0 (8)
MiniCPM-ooffline grounding40.124.84.7%30.6 (10)24.9 (13)32.6 (11)26.4 (13)
MiniCPM-opolled40.326.311.1%29.0 (11)26.6 (10)32.3 (12)28.8 (9)
Gemma-3npolled80.041.224.6%27.8 (12)25.0 (12)32.8 (10)28.0 (12)
Qwen2.5-Omnioffline grounding28.317.21.0%27.2 (13)22.5 (14)27.2 (14)22.6 (14)
Qwen3-VL-4Bpolled32.627.55.6%26.2 (14)25.8 (11)29.5 (13)28.6 (10)
Qwen2.5-Omnipolled31.918.323.0%18.7 (15)17.3 (15)22.4 (15)19.8 (15)
MMDuet2native streaming39.612.79.5%17.0 (16)15.8 (16)20.4 (16)17.2 (16)
Gemma-3noffline grounding13.96.810.6%16.1 (17)13.5 (18)17.0 (20)14.0 (18)
Qwen3-VL-8Bblind33.59.73.1%15.7 (18)11.7 (19)17.4 (18)12.3 (19)
VideoLLM-onlinenative streaming36.81.73.0%14.5 (19)2.3 (20)18.1 (17)4.9 (20)
Flash-VStreampolled51.221.820.8%14.3 (20)14.1 (17)17.2 (19)15.1 (17)

TAcorrect\mathrm{TA_{correct}} is lower than TA for every system, and ranking the systems by the TAcorrect\mathrm{TA_{correct}} column gives a different order from the TA column (τ = 0.52 between the two), with the largest losses for VideoLLM-online and polled Gemma-3n, which respond promptly but wrongly. Overall changes far less and keeps the same top three, because Acc already carries the content of these responses into Overall. A correct first response is later overwritten on more than 20% of events for five of the seven polled models and on under 11% for offline grounding, which is the repeated answering that SC penalizes. SCsmooth\mathrm{SC_{smooth}} in turn separates the systems at the floor. JoyAI-VL moves from eighth to third because it violates less per event than the polled models that share its floor, while the order of the other systems is largely preserved.

The main conclusion about timing and silence is therefore unchanged. Among the ten full-set online systems, timing and silence stay negatively rank-correlated under both changes, so the trade-off does not come from how the two scores are defined.

W2 / Q2. Item counts and subsets in the main table; the human reference

Thank you for pointing out that the scope of evaluation in the main table was not stated clearly. We clarify that the main results of all models use the same complete evaluation set of 1,060 interactions. The experiments were run in batches, and the final results aggregate all batches. We apologize that some descriptions gave the impression that different systems were evaluated on different subsets, and we have corrected the description of the runs.

For the human reference, we acknowledge that it differed from the systems in answer format. We changed the setting of the human test and ran it again under the systems' protocol. Eight graduate students take part and together cover all items, and for each participant we excluded the videos they may have annotated, to limit the effect of familiarity with the annotations. Multiple-choice questions show the same stem and options that the systems see and are answered with option letters (Figure 14 in the revision). Each response is timed on the stream clock, and the answers are scored by the same scorer and judge as every system. A multiple-choice answer is a single key press with no pause, and for a free-text answer the response time is taken at the key press and the video pauses only while the text is typed, as the video clock also waits for a system's poll to return. We also correct the submission's description of playback control (Appendix K in the revision). In the actual test, participants can drag the progress bar and re-watch the video within the part that has already played, which is similar to what video agents do.

As shown in the table below, the re-collected human reference is above every system in Overall, content accuracy and silence compliance. Its content accuracy rises from 51.3 to 81.9 once answers are given as option letters, and only the polled models, which respond at almost every poll, score higher in timing.

SystemOverallAccTASC
Human reference, re-collected79.381.977.381.2
Human reference, submission53.651.363.574.7
Claude Sonnet 5, agentic with Claude Code tools61.266.770.755.5
GPT-5.6 Sol, agentic with Qwen-MM-Plugins59.156.868.660.0
Claude Sonnet 5, polled44.666.082.917.6
Qwen3-VL-8B, polled34.854.277.810.9

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.