- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SVI-Bench evaluates streaming vision-language systems on proactive triggering, appropriate silence, latency, response correctness, and delegation with memory. It contains 75 expert-authored items across nine categories, with 272 dimension-level criteria. InteractFlow records synchronized observations, queries, and outputs, then evaluates them using a five-stage model judge selected on a 20-item human pilot. JoyAI-VL-Interaction obtains the highest aggregate score among five deployed configurations, while the results identify proactive intervention as a common weakness.
Critical strengths
- The benchmark addresses meaningful gaps in evaluation. Its dimensions capture failures that answer correctness alone misses, including missed events, unnecessary interruptions, and delayed responses (Table 2).
- The recording pipeline makes results inspectable. Saved traces preserve the timing and context of model behavior, allowing scores to be checked or revised without rerunning the systems (section 5).
Critical weaknesses
- Some items use event-to-output wall time, while others use adapter-reported inference time, which can exclude queueing (Appendix A.5). These measure different aspects of the user’s experience. Table 12 compares different item subsets rather than rescoring the same items using both clocks. The authors should use a consistent end-to-end clock for the main score and retain inference time as a diagnostic. This matters because removing latency substantially reduces JoyAI’s lead, although it remains first.
- The response-correctness definition does not match a key example. A1005 receives full D4 credit for saying that a person has fallen before the fall occurs (Table 13; Figure 6). This separates expected content from timing, but conflicts with the stated requirement that the response be consistent with the scene. The aggregate can also remain high despite failure of a central requirement: B3001 scores 75 despite omitting the requested reminder (Table 14). Please clarify D4 and test whether requiring temporally correct content or completion of the main task changes the conclusions.
- Busy-frame skipping can give slower systems fewer observations. MiniCPM’s 300-second sessions also renew without replaying earlier queries or frames, while some memory tasks exceed 20 minutes (Table 6). Comparing complete deployed systems is legitimate, but these constraints should be distinguished from reasoning failures. Report observation coverage, session renewals, and results within shared supported time horizons.
Questions for authors
Please refer to Critical Weakness. If I miss something, don't hesitate to let me know.
4 comments
Response to Reviewer ZCsk (2/3): D4 definition, temporal correctness, and core task completion (W2)
We thank the reviewer for suggesting clarification of D4 and stricter tests of temporal correctness and core task completion. For A1005, the item-level D4 anchor checks whether the output includes the expected subject and action, specifically, the person in white and the action of falling. The output therefore receives a D4 score of 1 for matching these content elements. This score does not establish that the person had already fallen when the response was produced. The premature alert and the lack of a valid response after the event are reflected in scores of 0 on D1, D2, and D3, giving an item score of 33.3. We will clarify this distinction between content-element matching and temporal validity in the D4 definition and case description, so that full credit on one dimension is not interpreted as successful completion of the interaction.
B3001 combines immediate question answering with a future reminder. D4 evaluates the content of the immediate answers, D2 assesses silence during the waiting period, and D5 assesses whether the reminder request is retained and executed. The system answers the questions but omits the final reminder, giving D5 = 0. Under end-to-end timing, the R2/R3 latencies are 2.297 and 1.571 seconds, with a mean of 1.934 seconds. D3 remains 1 under the item's predefined multi-turn rule. The D1–D5 score vector is (N/A, 1, 1, 1, 0), and the item score remains 75. This score records partial satisfaction of the applicable dimensions, not completion of the reminder task.
To test whether stricter temporal correctness could change the leading system, we computed a conservative lower bound on JoyAI-VL-Interaction's Overall score under revisions that only reduce D4. Using the revised end-to-end D3 scoring as the baseline, we set D4 to zero for all its items while keeping all other dimension scores and the aggregation rule unchanged. Its Overall remains 31.33, above the highest revised main score among the other systems, 24.00. Thus, with other scores fixed, any temporal-correctness revision that only reduces D4 cannot change JoyAI-VL-Interaction's leading position. This bound tests the sensitivity of the leading position to D4-only revisions. It does not replace a sentence-level reassessment of temporal correctness.
To assess the impact of core task completion requirements, we tested an additional strict aggregation rule: an item receives a score of zero if any applicable D1 or D5 score is zero. All other items retain their revised main scores.
| System | Main Overall with end-to-end D3 | Core-requirement rule (D1/D5) |
|---|---|---|
| JoyAI-VL-Interaction | 47.78 | 26.33 |
| MOSS-VL-Realtime | 24.00 | 15.39 |
| Doubao Seed 2.1 Pro | 21.22 | 9.50 |
| Mage-VL | 17.50 | 8.61 |
| MiniCPM-O-4.5-9B | 14.67 | 6.00 |
JoyAI-VL-Interaction remains first, and all five systems retain their positions in the revised main ranking. This rule tests the effect of treating a failure on an applicable D1/D5 requirement as a failure of the entire item. It does not constitute independent task-success annotation of every core requirement in each item.
We will add the temporal-correctness and core-task-completion sensitivity analyses to Appendix A.10, specifying the scoring rules, results, and scope of each analysis.
Response to Reviewer ZCsk (3/3): Input coverage, session renewals, and shared-subset results (W3)
We thank the reviewer for recommending that input availability and session constraints be distinguished from model reasoning performance. We estimated input coverage for the same visual events across all five systems and examined MiniCPM's recorded session renewals to assess these deployment conditions.
The foreground scheduler samples video frames at 1 fps. Ordinary frames are skipped while the system is busy, whereas query-bound frames are retained and queued. Query-bound frames preserve the query-time view, which does not imply that all visual event frames are subsequently delivered. We temporally aligned the five systems' recorded input traces with a shared set of 736 identifiable visual event phases and estimated coverage separately for the decisive visual phase and the full event visibility interval:
| System | Estimated decisive-phase coverage | Estimated full-event visibility coverage |
|---|---|---|
| JoyAI-VL-Interaction | 41.74% | 66.66% |
| Doubao Seed 2.1 Pro | 6.26% | 10.67% |
| Mage-VL | 45.42% | 61.23% |
| MOSS-VL-Realtime | 33.16% | 50.02% |
| MiniCPM-O-4.5-9B | 57.38% | 73.87% |
For query-bound frames, we used the recorded capture timestamps for temporal matching. For ordinary frames without exact capture timestamps, we estimated coverage probabilities assuming a capture lag uniformly distributed between 0 and 1 second before the request start time. These estimates describe observation opportunities on the common events. They do not directly measure visual understanding or the proportion of failures caused by missing input. API request records confirming frame delivery and queue waiting times for query-bound frames will be reported separately in the appendix.
The MiniCPM audit identified 35 session renewals triggered by the session time limit within the scheduled evaluation periods. Including the final waiting periods for output, the complete recordings contain 43 session renewals across 19 items. All 43 renewals created new sessions without replaying prior queries or historical frames. These interruptions in session continuity may affect performance on long-duration tasks, so the associated failures should not be attributed solely to the model's memory or reasoning abilities.
To examine the comparison on shorter tasks and on items without recorded session renewals, we evaluated all five systems on two common subsets: 59 items with a scheduled duration of at most 300 seconds, and 56 items with no recorded MiniCPM session-limit renewal in the complete recordings. The former is defined by scheduled duration and does not guarantee the absence of renewals. The latter is defined by actual renewal logs. Both analyses use the same end-to-end D3 definition as the revised main results.
| System | Main Overall (75 items) | Scheduled duration ≤300 s (59 items) | No MiniCPM renewal (56 items) |
|---|---|---|---|
| JoyAI-VL-Interaction | 47.78 | 46.26 | 46.06 |
| MOSS-VL-Realtime | 24.00 | 21.61 | 22.17 |
| Doubao Seed 2.1 Pro | 21.22 | 21.40 | 21.35 |
| Mage-VL | 17.50 | 17.51 | 17.56 |
| MiniCPM-O-4.5-9B | 14.67 | 14.83 | 15.03 |
The ranking of all five systems remains unchanged on both subsets relative to the revised main results. JoyAI-VL-Interaction scores 46.26 and 46.06 on the 59-item and 56-item subsets, respectively, leading MOSS-VL-Realtime by 24.65 and 23.88 points.
These analyses show that the revised ranking persists when evaluation is restricted to shorter tasks or when items with recorded MiniCPM session renewals are excluded. The full benchmark continues to include long-duration interactions. We will add the input coverage method, session renewal counts, and affected items to Appendix A.2, and report the shared-subset results in Appendix A.10. We will interpret system performance in relation to the corresponding input availability and session constraints.
Thank you for demonstrating the robustness of the benchmark. I’ll keep my positive score.
Response to Reviewer ZCsk (1/3): End-to-end latency and same-trace rescoring (W1)
We thank the reviewer for suggesting a consistent end-to-end latency definition for the main score, with inference latency reported separately as diagnostic information. We have adopted this suggestion and will use end-to-end wall-clock latency for the main D3 scores in the revised paper. Latency is measured from the event time specified by the item-level anchor or the query dispatch time to the corresponding visible output. Recomputing D3 and Overall from the same recorded traces leaves JoyAI-VL-Interaction in first place, with a 23.78-point lead over the runner-up.
During rescoring, we kept the deployed configurations, response bindings, valid-response criteria, scoring thresholds, and all other dimension scores unchanged. For the 36 explicit-query or startup items, latency is measured from query dispatch to the first logged observation of the corresponding visible output, including queueing delays. We also applied query-to-output wall-clock latency to B3006, originally grouped with the visual-event items but evaluated on the R2/R3 query responses. The remaining 36 visual-event items retain their original wall-clock D3 scores.
The top two positions remain unchanged, while the other three systems change order. As a sensitivity analysis, we used the maximum per-turn latency for multi-turn items without an explicitly defined aggregation rule. Under this alternative, JoyAI-VL-Interaction's Overall is 47.11, with the other systems' scores and the revised ranking unchanged. Excluding D3, JoyAI-VL-Interaction still ranks first with an Overall of 39.67, showing that its lead is not solely driven by latency.
We will update the main results in §6.3 and report the comparison of the original and revised timing definitions on the same traces in Appendix A.10. Adapter-reported per-inference latency will be presented separately as diagnostic information. D3 measures end-to-end responsiveness under each system's deployed configuration, including the effects of scheduling, state management, inference, and output delivery.