acceptodds
Under review as a conference paper at ICLR 2027

SVI-Bench: Evaluating Human-Perceived Interaction Trajectories in Streaming Video Systems

Abstract

Users do not experience a streaming vision-language assistant as a collection of isolated responses. They experience a continuous interaction trajectory in which the assistant watches a changing scene, decides when to speak or stay silent, reacts with some delay, and remembers information over time. The quality of this interaction therefore depends not only on what the assistant says, but also on how its behavior is aligned with the evolving visual context. To better evaluate these interaction demands, we introduce SVI-Bench, a trajectory-centric benchmark for streaming vision-language interaction. SVI-Bench evaluates five user-observable aspects: proactive triggering, silence correctness, latency, response correctness, and delegation with long-horizon memory. It contains 75 expert-authored items across nine realistic interaction categories, with 272 anchored dimension-level criteria. Central to SVI-Bench is InteractFlow, a trace-based evaluation framework that synchronizes model outputs with the visual states visible when they occur. This turns temporal interaction from abstract metadata into directly inspectable multimodal evidence and allows responses to be judged in the visual context in which they actually occurred. It also preserves each system's streaming behavior and supports traceable rescoring. Across five deployed configurations, JoyAI-VL-Interaction achieves the highest Overall score. The results also show that current systems are much better at staying silent than at speaking when needed. SVI-Bench therefore shifts streaming evaluation from scoring individual responses to evaluating the complete interaction experienced by the user. All videos, item-specific scoring criteria, recordings, and InteractFlow will be released.

Primary area:
datasets and benchmarks
Keywords:
streaming video understanding, benchmark, vision-language models, proactive interaction, human-perceived interaction trajectory;

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZCskOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SVI-Bench evaluates streaming vision-language systems on proactive triggering, appropriate silence, latency, response correctness, and delegation with memory. It contains 75 expert-authored items across nine categories, with 272 dimension-level criteria. InteractFlow records synchronized observations, queries, and outputs, then evaluates them using a five-stage model judge selected on a 20-item human pilot. JoyAI-VL-Interaction obtains the highest aggregate score among five deployed configurations, while the results identify proactive intervention as a common weakness.

Critical strengths
  • The benchmark addresses meaningful gaps in evaluation. Its dimensions capture failures that answer correctness alone misses, including missed events, unnecessary interruptions, and delayed responses (Table 2).
  • The recording pipeline makes results inspectable. Saved traces preserve the timing and context of model behavior, allowing scores to be checked or revised without rerunning the systems (section 5).
Critical weaknesses
  • Some items use event-to-output wall time, while others use adapter-reported inference time, which can exclude queueing (Appendix A.5). These measure different aspects of the user’s experience. Table 12 compares different item subsets rather than rescoring the same items using both clocks. The authors should use a consistent end-to-end clock for the main score and retain inference time as a diagnostic. This matters because removing latency substantially reduces JoyAI’s lead, although it remains first.
  • The response-correctness definition does not match a key example. A1005 receives full D4 credit for saying that a person has fallen before the fall occurs (Table 13; Figure 6). This separates expected content from timing, but conflicts with the stated requirement that the response be consistent with the scene. The aggregate can also remain high despite failure of a central requirement: B3001 scores 75 despite omitting the requested reminder (Table 14). Please clarify D4 and test whether requiring temporally correct content or completion of the main task changes the conclusions.
  • Busy-frame skipping can give slower systems fewer observations. MiniCPM’s 300-second sessions also renew without replaying earlier queries or frames, while some memory tasks exceed 20 minutes (Table 6). Comparing complete deployed systems is legitimate, but these constraints should be distinguished from reasoning failures. Report observation coverage, session renewals, and results within shared supported time horizons.
Questions for authors

Please refer to Critical Weakness. If I miss something, don't hesitate to let me know.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZCsk (1/3): End-to-end latency and same-trace rescoring (W1)

We thank the reviewer for suggesting a consistent end-to-end latency definition for the main score, with inference latency reported separately as diagnostic information. We have adopted this suggestion and will use end-to-end wall-clock latency for the main D3 scores in the revised paper. Latency is measured from the event time specified by the item-level anchor or the query dispatch time to the corresponding visible output. Recomputing D3 and Overall from the same recorded traces leaves JoyAI-VL-Interaction in first place, with a 23.78-point lead over the runner-up.

During rescoring, we kept the deployed configurations, response bindings, valid-response criteria, scoring thresholds, and all other dimension scores unchanged. For the 36 explicit-query or startup items, latency is measured from query dispatch to the first logged observation of the corresponding visible output, including queueing delays. We also applied query-to-output wall-clock latency to B3006, originally grouped with the visual-event items but evaluated on the R2/R3 query responses. The remaining 36 visual-event items retain their original wall-clock D3 scores.

SystemOriginal OverallEnd-to-end D3End-to-end OverallRank change
JoyAI-VL-Interaction53.6164.3847.781 → 1
MOSS-VL-Realtime25.4419.1824.002 → 2
MiniCPM-O-4.5-9B21.331.3714.673 → 5
Doubao Seed 2.1 Pro21.222.7421.224 → 3
Mage-VL19.508.2217.505 → 4

The top two positions remain unchanged, while the other three systems change order. As a sensitivity analysis, we used the maximum per-turn latency for multi-turn items without an explicitly defined aggregation rule. Under this alternative, JoyAI-VL-Interaction's Overall is 47.11, with the other systems' scores and the revised ranking unchanged. Excluding D3, JoyAI-VL-Interaction still ranks first with an Overall of 39.67, showing that its lead is not solely driven by latency.

We will update the main results in §6.3 and report the comparison of the original and revised timing definitions on the same traces in Appendix A.10. Adapter-reported per-inference latency will be presented separately as diagnostic information. D3 measures end-to-end responsiveness under each system's deployed configuration, including the effects of scheduling, state management, inference, and output delivery.

AuthorsOpenReview ↗

Response to Reviewer ZCsk (2/3): D4 definition, temporal correctness, and core task completion (W2)

We thank the reviewer for suggesting clarification of D4 and stricter tests of temporal correctness and core task completion. For A1005, the item-level D4 anchor checks whether the output includes the expected subject and action, specifically, the person in white and the action of falling. The output therefore receives a D4 score of 1 for matching these content elements. This score does not establish that the person had already fallen when the response was produced. The premature alert and the lack of a valid response after the event are reflected in scores of 0 on D1, D2, and D3, giving an item score of 33.3. We will clarify this distinction between content-element matching and temporal validity in the D4 definition and case description, so that full credit on one dimension is not interpreted as successful completion of the interaction.

B3001 combines immediate question answering with a future reminder. D4 evaluates the content of the immediate answers, D2 assesses silence during the waiting period, and D5 assesses whether the reminder request is retained and executed. The system answers the questions but omits the final reminder, giving D5 = 0. Under end-to-end timing, the R2/R3 latencies are 2.297 and 1.571 seconds, with a mean of 1.934 seconds. D3 remains 1 under the item's predefined multi-turn rule. The D1–D5 score vector is (N/A, 1, 1, 1, 0), and the item score remains 75. This score records partial satisfaction of the applicable dimensions, not completion of the reminder task.

To test whether stricter temporal correctness could change the leading system, we computed a conservative lower bound on JoyAI-VL-Interaction's Overall score under revisions that only reduce D4. Using the revised end-to-end D3 scoring as the baseline, we set D4 to zero for all its items while keeping all other dimension scores and the aggregation rule unchanged. Its Overall remains 31.33, above the highest revised main score among the other systems, 24.00. Thus, with other scores fixed, any temporal-correctness revision that only reduces D4 cannot change JoyAI-VL-Interaction's leading position. This bound tests the sensitivity of the leading position to D4-only revisions. It does not replace a sentence-level reassessment of temporal correctness.

To assess the impact of core task completion requirements, we tested an additional strict aggregation rule: an item receives a score of zero if any applicable D1 or D5 score is zero. All other items retain their revised main scores.

SystemMain Overall with end-to-end D3Core-requirement rule (D1/D5)
JoyAI-VL-Interaction47.7826.33
MOSS-VL-Realtime24.0015.39
Doubao Seed 2.1 Pro21.229.50
Mage-VL17.508.61
MiniCPM-O-4.5-9B14.676.00

JoyAI-VL-Interaction remains first, and all five systems retain their positions in the revised main ranking. This rule tests the effect of treating a failure on an applicable D1/D5 requirement as a failure of the entire item. It does not constitute independent task-success annotation of every core requirement in each item.

We will add the temporal-correctness and core-task-completion sensitivity analyses to Appendix A.10, specifying the scoring rules, results, and scope of each analysis.

AuthorsOpenReview ↗

Response to Reviewer ZCsk (3/3): Input coverage, session renewals, and shared-subset results (W3)

We thank the reviewer for recommending that input availability and session constraints be distinguished from model reasoning performance. We estimated input coverage for the same visual events across all five systems and examined MiniCPM's recorded session renewals to assess these deployment conditions.

The foreground scheduler samples video frames at 1 fps. Ordinary frames are skipped while the system is busy, whereas query-bound frames are retained and queued. Query-bound frames preserve the query-time view, which does not imply that all visual event frames are subsequently delivered. We temporally aligned the five systems' recorded input traces with a shared set of 736 identifiable visual event phases and estimated coverage separately for the decisive visual phase and the full event visibility interval:

SystemEstimated decisive-phase coverageEstimated full-event visibility coverage
JoyAI-VL-Interaction41.74%66.66%
Doubao Seed 2.1 Pro6.26%10.67%
Mage-VL45.42%61.23%
MOSS-VL-Realtime33.16%50.02%
MiniCPM-O-4.5-9B57.38%73.87%

For query-bound frames, we used the recorded capture timestamps for temporal matching. For ordinary frames without exact capture timestamps, we estimated coverage probabilities assuming a capture lag uniformly distributed between 0 and 1 second before the request start time. These estimates describe observation opportunities on the common events. They do not directly measure visual understanding or the proportion of failures caused by missing input. API request records confirming frame delivery and queue waiting times for query-bound frames will be reported separately in the appendix.

The MiniCPM audit identified 35 session renewals triggered by the session time limit within the scheduled evaluation periods. Including the final waiting periods for output, the complete recordings contain 43 session renewals across 19 items. All 43 renewals created new sessions without replaying prior queries or historical frames. These interruptions in session continuity may affect performance on long-duration tasks, so the associated failures should not be attributed solely to the model's memory or reasoning abilities.

To examine the comparison on shorter tasks and on items without recorded session renewals, we evaluated all five systems on two common subsets: 59 items with a scheduled duration of at most 300 seconds, and 56 items with no recorded MiniCPM session-limit renewal in the complete recordings. The former is defined by scheduled duration and does not guarantee the absence of renewals. The latter is defined by actual renewal logs. Both analyses use the same end-to-end D3 definition as the revised main results.

SystemMain Overall (75 items)Scheduled duration ≤300 s (59 items)No MiniCPM renewal (56 items)
JoyAI-VL-Interaction47.7846.2646.06
MOSS-VL-Realtime24.0021.6122.17
Doubao Seed 2.1 Pro21.2221.4021.35
Mage-VL17.5017.5117.56
MiniCPM-O-4.5-9B14.6714.8315.03

The ranking of all five systems remains unchanged on both subsets relative to the revised main results. JoyAI-VL-Interaction scores 46.26 and 46.06 on the 59-item and 56-item subsets, respectively, leading MOSS-VL-Realtime by 24.65 and 23.88 points.

These analyses show that the revised ranking persists when evaluation is restricted to shorter tasks or when items with recorded MiniCPM session renewals are excluded. The full benchmark continues to include long-duration interactions. We will add the input coverage method, session renewal counts, and affected items to Appendix A.2, and report the shared-subset results in Appendix A.10. We will interpret system performance in relation to the corresponding input availability and session constraints.

Thank you for demonstrating the robustness of the benchmark. I’ll keep my positive score.

Reviewer_WuV6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SVI-Bench evaluates streaming vision-language interaction using 75 expert-authored items in nine categories and 272 dimension-level anchors. It measures proactive triggering, silence correctness, latency, response correctness, and delegation/memory. InteractFlow records synchronized visual inputs, queries, and outputs, constructs evidence packages, and uses a five-stage model judge. Triggering and silence scores are coupled by their item-level minimum where both apply. Across 375 recordings of five deployed configurations, JoyAI-VL-Interaction ranks first. The main contribution is an auditable, trajectory-based evaluation workflow rather than a new predictive model.

Critical strengths
  1. Traceable evaluation infrastructure. Synchronizing observations, queries, and outputs and separating recording from scoring makes interaction failures inspectable and enables rescoring without rerunning the system. This is useful infrastructure for a setting where response text alone misses important behavior.
  2. Explicit behavioral criteria and useful diagnostics. The anchored rubric and within-item triggering/silence coupling address the problem of receiving credit for silence while missing required interventions. The paper also discloses that it compares deployed configurations rather than hardware-controlled model weights, and reports a no-D3 sensitivity result. These choices make the results easier to interpret and audit.
Critical weaknesses
  1. Latency does not consistently measure the user-observable quantity motivating the benchmark. Appendix A.5 uses event-to-visible-output recording time for visual triggers, but preferentially uses adapter-reported per-inference latency for explicit-query/startup items, falling back to wall time when that field is unavailable. Queueing may therefore be recorded but excluded from the official D3 score, and services can be scored with different clock scopes. Table 12 compares two different item subsets rather than rescoring the same items under both clocks, so it does not establish clock-choice robustness. Removing D3 leaves JoyAI first, which is useful evidence, but narrows the lead from 28.17 to about 10.2 points. A consistent end-to-end latency definition is necessary for a benchmark centered on human-perceived trajectories.
  2. The main measurement pipeline lacks independent cross-system validation. The 20-item human pilot contains only JoyAI recordings and is used for anchor refinement, judge selection, and reported agreement. Its 83.10% exact agreement is calibration performance rather than held-out validation. For the central D1 dimension, only 14 labels are available, with 64.29% agreement and weighted kappa 0.4444. Generalization to other systems' silence, batching, and response styles is therefore uncertain. This matters especially because min coupling can propagate D1 errors into the overall score. One accepted recording per system/item also leaves service and judge variability unmeasured. The appendix appropriately acknowledges the cross-system validation gap, but that gap remains material to the benchmark's reliability claims.
Questions for authors
  1. Decisive rescoring: What happens if all D3 items use event/query-to-visible-output wall time? Please compare the same traces under both clock definitions, including queueing, and report any ranking changes.
  2. Independent validation: Are there frozen-rubric human scores on recordings from the other four systems or on held-out items? In particular, can you establish cross-system D1 reliability rather than reporting only the judge-selection pilot?
  3. Attribution of triggering failures: For low-D1 cases, how often was the decisive event frame actually delivered? The one-in-flight, busy-skip policy can change effective observation opportunities, so input exposure is important for distinguishing deployment limitations from triggering-policy errors.
  4. Robustness: Are the principal findings stable under repeated recording/judging and on the 41 items where D1 and D2 both apply? Their raw means otherwise use different item sets, so a paired analysis could materially affect the claim that the systems are better at withholding responses than triggering them.
5 comments
AuthorsOpenReview ↗

Response to Reviewer WuV6 (1/4): End-to-end latency and same-trace rescoring (W1 and Q1)

We thank the reviewer for recommending a comparison of the same recorded traces using a consistent timing definition. We have adopted this recommendation and will use end-to-end wall-clock latency from the annotated event or query dispatch to visible output for the main D3 scores in the revised paper. We recomputed D3 and Overall from the existing traces, including queueing and processing delays. JoyAI-VL-Interaction and MOSS-VL-Realtime remain first and second, with Overall scores of 47.78 and 24.00, respectively, giving JoyAI-VL-Interaction a 23.78-point lead.

Both scoring schemes use the same recorded interaction traces, with the deployed configurations, response matching, valid-response criteria, scoring thresholds, and all other dimension scores held constant. Each system is evaluated on 75 items, with D3 applicable to 73. For the 36 explicit-query or startup items, latency is measured from query dispatch to the first logged observation of the corresponding visible output. B3006, originally grouped with the visual-event items, involves responses to the R2/R3 queries and is also rescored using query-to-output wall-clock latency. The remaining 36 visual-event items retain their original wall-clock D3 scores. The results under the two timing definitions are:

SystemOriginal D3End-to-end D3Original OverallEnd-to-end OverallRank change
JoyAI-VL-Interaction82.8864.3853.6147.781 → 1
MOSS-VL-Realtime23.9719.1825.4424.002 → 2
MiniCPM-O-4.5-9B21.921.3721.3314.673 → 5
Doubao Seed 2.1 Pro2.742.7421.2221.224 → 3
Mage-VL14.388.2219.5017.505 → 4

The top two positions remain unchanged, while the other three systems change order. We also conducted a sensitivity analysis using the maximum per-turn latency for multi-turn items without an explicitly defined aggregation rule. Under this alternative, JoyAI-VL-Interaction's Overall is 47.11, the other systems' scores remain unchanged, and the ranking is identical to the end-to-end results above.

SVI-Bench evaluates complete deployed systems. Scheduling, state management, inference, and output delivery all affect the user's waiting time, so end-to-end D3 measures responsiveness under each system's deployed configuration. Differences in D3 cannot be attributed solely to the underlying models' inference speeds. Removing D3 also leaves JoyAI-VL-Interaction first at 39.67, showing that its lead does not depend entirely on latency.

We will update the main results in §6.3 and report the comparison of the original and revised timing definitions on the same traces in Appendix A.10. Adapter-reported per-inference latency will be presented separately as diagnostic information.

AuthorsOpenReview ↗

Response to Reviewer WuV6 (2/4): Independent cross-system human validation (W2 and Q2)

We thank the reviewer for recommending independent human validation across systems. We conducted an independent human evaluation on four items not included in the original pilot (A1006, A2001, B1003, and B2007), covering categories A1, A2, B1, and B2 for all five systems. The sample comprised four existing recordings per system, for a total of 20 recordings. Scorers independently assessed the recorded evidence using the frozen item-level anchors, without viewing the original automatic judge's scores or the model-generated reference labels. The evaluation produced 75 applicable item–dimension labels, including 20 for D1.

Across all dimensions, the exact agreement rate between the automatic judge and human scores was 84.00% (63/75), with a mean absolute error (MAE) of 0.0933 and quadratically weighted Cohen's κ of 0.8244. For D1, exact agreement was 60.00% (12/20), with quadratically weighted Cohen's κ of 0.3798. D3 labels from both sources were recalculated using the same response bindings and end-to-end timing rules. Their 20/20 agreement therefore reflects consistency of the timing conversion and is not evidence of independent judgment accuracy. Excluding D3, exact agreement across the remaining dimensions was 78.18% (43/55). Results by system are:

SystemRecordingsAgreement across all dimensionsD1 agreement
JoyAI-VL-Interaction480.00% (12/15)50.00% (2/4)
MOSS-VL-Realtime486.67% (13/15)75.00% (3/4)
Doubao Seed 2.1 Pro480.00% (12/15)50.00% (2/4)
Mage-VL486.67% (13/15)75.00% (3/4)
MiniCPM-O-4.5-9B486.67% (13/15)50.00% (2/4)

All eight D1 disagreements were concentrated in B1003 and B2007, with each differing by one scoring level. The automatic judge and human scorers agreed on D1 for the other two items across all five systems. On B1003, the automatic judge assigned 0 to all five systems, while human scorers assigned 0.5. On B2007, the automatic judge assigned 0 to Doubao and MiniCPM and 1 to JoyAI, whereas human scorers assigned 0.5 in all three cases.

To assess how scoring disagreements propagate through the D1/D2 min coupling, we replaced only the D1/D2 labels of the sampled recordings with human labels, retaining the revised D3 and all other scores. Of the 15 recordings where both D1 and D2 apply, four coupled components changed: three due to the D1 disagreements on B2007 and one due to a D2 disagreement on A1006. D2 does not apply to the five B1003 recordings, so their D1 changes contribute directly to their item scores. Across all 20 sampled recordings, the mean absolute change in item scores was 7.50 points on the 0–100 scale. We then recomputed Overall across all 75 items, keeping the remaining item scores unchanged:

SystemRevised main OverallReplace D1/D2 onlyChangeReplace all applicable dimensionsChange
JoyAI-VL-Interaction47.7847.780.0047.56-0.22
MOSS-VL-Realtime24.0024.220.2224.440.44
Doubao Seed 2.1 Pro21.2221.440.2221.440.22
Mage-VL17.5017.720.2218.170.67
MiniCPM-O-4.5-9B14.6715.110.4415.110.44

Under both label-substitution analyses, all five systems retain their positions in the revised main ranking. Replacing D1/D2 alone changes each system's Overall by less than 0.5 points in absolute value. Replacing all applicable dimensions produces a maximum change of 0.67 points for Mage. These findings are limited to label substitutions within the sampled recordings. We will distinguish the original pilot calibration from this independent human validation across systems in Appendix A.9, and report the sources of the ratings, results by system, disagreements, and their effects on score aggregation.

AuthorsOpenReview ↗

Response to Reviewer WuV6 (3/4): Input coverage in low-D1 cases (Q3)

We thank the reviewer for recommending an examination of visual input availability during decisive events. For each system, we temporally aligned the visual event intervals included in the analysis of D1 = 0 items with the recorded input traces. We estimated input coverage separately for the decisive visual phase and the full event visibility interval. This analysis estimates observation opportunities within low-D1 items and helps interpret triggering performance under each deployed configuration.

The foreground scheduler samples video frames at 1 fps and allows only one in-flight request at a time. Ordinary frames are skipped while the system is busy, whereas query-bound frames are retained in a queue. Thus, the same nominal sampling schedule can produce different actual input sequences across systems. Query-bound frames preserve the visual context at the time of the query. This mechanism does not imply that all pre-annotated visual event frames are subsequently delivered.

SystemItems with D1 = 0Event phasesEstimated decisive-phase coverageEstimated full-event visibility coverage
JoyAI-VL-Interaction2062343.68%69.26%
Doubao Seed 2.1 Pro236256.84%10.94%
Mage-VL2054657.23%68.91%
MOSS-VL-Realtime2148139.61%49.86%
MiniCPM-O-4.5-9B2562962.70%73.42%

The analysis selects items with D1 = 0 and considers the included event phases within those items. Item counts, event-phase counts, and triggering-failure counts are distinct quantities. The set of items with D1 = 0 differs across systems, so the coverage estimates describe the observation conditions for each system's corresponding subset.

For query-bound frames, we used the recorded media capture timestamps for temporal matching. For ordinary frames without exact capture timestamps, we estimated coverage probabilities by assuming that the capture lag was uniformly distributed between 0 and 1 second before the request start time. Events were matched within the first playback cycle of the source video, and input traces included the waiting period at the end of the recording. The table therefore reports coverage estimates based on saved traces and timing assumptions, rather than directly verified frame-delivery rates or the proportion of triggering failures attributable to missing visual input.

SVI-Bench evaluates complete deployed systems under the specified runtime protocol. D1 reflects the combined effects of sampling, scheduling, visual perception, and triggering decisions. These estimates help interpret low-D1 results in relation to the available observation opportunities. Temporal coverage alone does not guarantee that a frame contains sufficient discernible information for a correct decision. We therefore do not attribute individual failures to a single cause based on coverage alone. We will add the input scheduling rules, event-matching method, coverage assumptions, and results by system to Appendix A.2, and report the query-bound frame audit separately.

Reviewer_CoV1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces SVI-Bench, a trajectory-centric benchmark for evaluating streaming vision-language assistants across five dimensions. The authors also propose InteractFlow, a trace-based evaluation framework that synchronizes model outputs with visual states to provide inspectable multimodal evidence. Experiments on five deployed configurations reveal that current systems struggle significantly with proactive triggering compared to maintaining silence.

Critical strengths
  1. Moving beyond isolated VideoQA to the "human-perceived interaction trajectory" addresses a critical gap in streaming VLM evaluation. The decoupling of the scenario taxonomy (Table 3) from the 5-dimension rubric (Table 2) creates a highly extensible framework that can easily accommodate future interaction paradigms without redesigning the core metrics.

  2. The evaluation exposes the stark asymmetry where models are highly proficient at maintaining silence (D2) but struggle significantly with proactive triggering (D1), offering directions for future architecture improvements (e.g., active intervention mechanisms).

Critical weaknesses
  1. 75 items is small for a benchmark intended to support leaderboard conclusions, even acknowledging the cost of dense temporal annotation. The imbalance is severe: C2 (agent delegation) has only 2 items, C1 (long-horizon visual memory) has 8, while A1 (monitoring & alerting) has 12 and B2 (real-time counting) has 12.

  2. The paper mentions that a panel of 5-7 scorers independently scored the 20-item pilot and then "discussed disagreements until it agreed on one score." However, the initial IAA before the panel discussion is not reported. Without this, it is impossible to judge whether the 272 expert-authored anchors are inherently objective and unambiguous, or if they required heavy subjective negotiation to reach a consensus.

Questions for authors
  1. How does the 1-fps foreground scheduler handle events that occur and resolve within a 500ms window? Also, please explicitly analyze the dataset to show what percentage of ground-truth events are shorter than 1 second.

  2. For multi-event and multi-turn items, how are multiple D1/D2 decisions aggregated before the min?

5 comments
AuthorsOpenReview ↗

Response to Reviewer CoV1 (1/4): Benchmark size and category imbalance(W1)

We thank the reviewer for raising concerns about benchmark size and category imbalance. SVI-Bench evaluates continuous interaction trajectories. Each item may include multiple events, query turns, and silence intervals. The current release includes 75 items across nine categories, with 4.21 hours of source video, covering responses within seconds and long-horizon visual memory tasks lasting approximately 25 minutes. The 272 applicable item–dimension judgments describe different aspects of performance within these tasks. The Overall score weights all 75 items equally.

To examine how category imbalance affects the leading score margin and system ranking, we focus on C1 (long-horizon visual memory, eight items) and C2 (agent delegation, two items). Each normalized item score lies in [0, 1], so these ten items can contribute at most 100 × 10/75 = 13.33 points to the Overall gap between any two systems. This upper bound is smaller than JoyAI-VL-Interaction's 23.78-point lead over the runner-up under the revised end-to-end D3 scoring. Its lead therefore cannot be explained solely by performance on C1 and C2. We also removed the two C2 items and recomputed Overall on the remaining 73 items using the same end-to-end D3 definition:

SystemOverall with end-to-end D3 (75 items)Without C2 (73 items)
JoyAI-VL-Interaction47.7847.72
MOSS-VL-Realtime24.0024.66
Doubao Seed 2.1 Pro21.2221.80
Mage-VL17.5017.98
MiniCPM-O-4.5-9B14.6715.07

The ranking of all five systems remains unchanged after excluding C2, with JoyAI-VL-Interaction remaining first.

The upper-bound analysis and the C2 exclusion analysis examine how the smaller categories affect the leading score margin and the observed ranking, respectively. Category scores provide diagnostic views of behavior on the evaluated tasks, and the C2 results reflect performance on the two delegation interactions tested. We will add these analyses to Appendix A.10. Future benchmark expansion will prioritize additional C1 and C2 items and incorporate further continuous interaction scenarios with practical value.

AuthorsOpenReview ↗

Response to Reviewer CoV1 (2/4): Human agreement before panel discussion(W2)

We thank the reviewer for requesting that we report inter-rater agreement before the panel discussion. In the 20-item pilot covering all nine categories, five to seven scorers who were not involved in developing the items or anchors independently scored 71 applicable item–dimension pairs using the initial anchors. Scorers could not see one another's scores, and the anchors remained fixed throughout independent scoring. All participating scorers assigned the same grade to 51 of the 71 pairs, giving a pre-discussion unanimous agreement rate of 71.83% (51/71). This is a descriptive statistic at the decision level: a pair was counted as unanimously rated only when every participating scorer assigned the same grade.

The panel then discussed the remaining 20 decisions individually. Where discussion identified ambiguous wording, we clarified the relevant anchors and made corresponding revisions to anchors for other items in the same category. The panel established consensus labels for all 71 item–dimension pairs. The consolidated anchors were frozen before recordings of the other four systems were collected. The 71.83% figure refers exclusively to independent scores obtained before discussion and anchor refinement.

We will specify the agreement metric and disagreement-resolution procedure in §3.3, distinguishing pre-discussion agreement among human scorers applying the initial anchors from agreement between the automatic judge and the human consensus labels. The latter corresponds to the 83.10% reported for the original pilot calibration under the original D3 clock definitions.

AuthorsOpenReview ↗

Response to Reviewer CoV1 (3/4): Short events and sampling granularity(Q1)

Thank you for asking how 1-fps sampling handles short events. All 375 recordings in the main evaluation used a sampling rate of 1 fps, with one frame per batch. Whether a 500-ms event is captured depends on whether a frame is sampled during its visible interval. An event that occurs and ends entirely between consecutive sampling instants may be missed.

The item-level anchors specify which visual changes require a response. For example, items involving a target entering the frame, a yellow card being shown, or a traffic light turning red require the relevant state to remain discernible for at least one second. Whether a brief visual change constitutes a required trigger therefore depends on the predefined conditions for that item.

To characterize event durations in the source videos, we examined the first playback cycle of 45 items requiring visually triggered responses. Through manual verification, we identified 700 events across 28 items with complete, clearly identifiable onset and offset times. Of these, 100 lasted less than one second, accounting for 14.29% of this subset:

Event typeEvents with determinable durationEvents shorter than 1 sPercentage
Complete counting actions / projectile events2128238.68%
States / objects / scenes16500.00%
Subtitle visibility intervals323185.57%
Total70010014.29%

We also distinguished the full event duration from the decisive visual phase that provides evidence for the triggering decision. Of these 700 events, 215 had a decisive visual phase shorter than one second (30.71%). These two statistics describe the duration of complete events and of the visual evidence needed for triggering, respectively. We will add their definitions, analysis scope, and human review procedure to Appendix A.3, and discuss how sampling granularity affects the evaluation under the predefined item-level trigger conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.