- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VideoLive evaluates a persistent public answer rather than scoring isolated video responses. On a causal media clock, a model chooses WAIT, ANSWER or REPAIR, and its answer remains visible between decisions. Core-400 balances Retention, Replacement, Withdrawal and Formation using 400 distinct source videos. Evaluation combines lifecycle completion, valid-answer duration and independent content/action probes across 29 configurations. The paper also proposes SPOT, which pairs the same evidence with alternative displayed answers during supervised tuning. On MMDuet2, SPOT raises LCS from 3.25 to 8.50 and VAT from 8.95 to 22.60, and improves the reported held-out streaming-benchmark metrics.
Critical strengths
-
Persistent-answer validity is a useful evaluation target. Distinguishing an empty display from a published NONE answer, and distinguishing answer formation from maintenance and repair, reveals failures that endpoint-only accuracy cannot capture. LCS and VAT are complementary rather than interchangeable.
-
The benchmark construction and evaluation engineering are documented in considerable detail. The study uses one primary trajectory per source video, a balanced lifecycle allocation, explicit causal access rules, source-video-clustered uncertainty and separate interface groups. The appendices include parser, timing, modality, semantic-matcher and ranking sensitivity analyses; these are meaningful strengths, not missing controls.
-
SPOT is a simple response to the proposed state-management problem, and the training sources are disjoint from Core-400. The paper includes an instance-matched ordinary-SFT control on held-out benchmarks and explicitly exposes the change in target-action proportions. Native-cache readouts also provide a useful complement to independent probes, although they should not be overinterpreted.
Critical weaknesses
-
The human result does not yet calibrate the benchmark on the same task as the models. Appendix E.6 states that participants received abbreviated questions without the activity-scope lists supplied to models; their answers were in Chinese, and their phase scores were sampled retrospectively from the interactive traces rather than obtained through the same independent content-probe protocol. Therefore, the Human row should not be read as a directly comparable ceiling. More fundamentally, the source annotations establish activity intervals, not necessarily the earliest point at which a causal observer can identify an activity. The 36-item human semantic spot check in Appendix I.1 validates reference–prediction matching, not the perceptual sufficiency of those time boundaries. This distinction is important for interpreting the very low absolute scores. I acknowledge the existing boundary sensitivity experiments, but they do not replace a matched-task, prefix-based annotation audit.
-
The SPOT comparison does not isolate state pairing from target-frequency changes. Table 29 shows publication targets increasing from 14.0% to 41.5% and target tokens increasing from 111,120 to 126,000 despite equal instance counts. Thus, ordinary SFT versus SPOT changes both counterfactual display exposure and the amount/distribution of publication supervision. A frequency-matched or reweighted control is needed to identify the benefit of conditioning on alternative displays. Ordinary SFT is reported on the external benchmarks but not alongside SPOT on the primary Core-400 table. The improvement also remains concentrated: Figure 4 shows Retention rising from 13 to 26 and Withdrawal from 0 to 8, while Replacement and Formation completion remain zero. These are useful gains, but not yet broad repair competence.
-
The scope of “real-time” and of the content–action interpretation should be tightened. The primary scores use media time, and Appendix L.2 explicitly advances the hosted media clock only after a response completes. This measures causal, time-indexed behavior rather than demonstrated wall-clock deadline compliance. Similarly, the 44.26% mismatch is 104 incorrect answers among 235 published answers after an independently executed probe succeeds, not 44.26% of all 389 recognized cases; another 154 cases have no published answer. Appendix F.7 states these denominators correctly, but the headline interpretation should consistently distinguish separate-execution recognition from knowledge available in the same native action state. The native-cache analysis helps, while its sensitivity to candidate-length normalization also warrants caution.
Questions for authors
-
Can the Human row be recomputed or clearly separated under matched question scopes and probe protocols? Can a prefix-based human audit validate the claimed evidence-support boundaries, distinct from checking semantic answer matching?
-
What are ordinary-SFT and action-frequency-matched SFT results on Core-400? Does SPOT still improve lifecycle completion after matching the publication-target distribution, and what explains the remaining zero Replacement/Formation completion?
-
Please distinguish media-clock scores from wall-clock latency and present the recognized-but-correct, recognized-but-incorrect and recognized-but-unpublished categories together. Which conclusions about a content–action gap are supported by the same native state rather than independent executions?
Matched controls, public-state evidence, and study procedures
Thank you for the careful and constructive review. We appreciate your distinction between recognizing an activity and keeping the displayed answer valid, and your question about what a viewer can recognize at an annotated boundary.
1. Human prefix audit and temporal references (W1, Q1). We added a prefix-based audit with eight participants: 906 judgments on 339 prefixes from 120 videos, covering all four lifecycle types. Participants watched continuous silent video and answered a single-choice question about the current activity, choosing from the full activity vocabulary with English labels and Chinese translations. Each person saw at most one prefix per source video.
At the decision point aligned with the annotated boundary, accuracy was 176/192 (91.7%), including 87/93 (93.5%) at initial boundaries and 89/99 (89.9%) at updated boundaries. These results provide direct human evidence of activity recognition at the sampled reference boundaries. Appendix O.2 gives all five offsets, their denominators, and the existing window and boundary-shift analyses. “First-sufficient” denotes the first grid point at or after the annotated onset.
In Table 2 we report the original interactive human study. The caption and Appendix E.6 describe the shorter questions without scope lists, Chinese free-text answers, replay, unrestricted thinking at pauses, and phase scores sampled from answer histories.
2. State pairing and publication supervision (W2, Q2). Thank you for suggesting a control that matches the frequency of publication targets. In Table 3, SPOT outperforms ordinary SFT on all six transfer metrics with the same training data, epochs, and number of examples. A reply is the training target in 14.0% of ordinary SFT examples and 41.5% of SPOT examples. Holding that share fixed would help test whether state pairing improves lifecycle completion on Core-400. With limited GPU time for validation during rebuttal, we plan to report that comparison in the final paper.
3. What the lifecycle stages show (W2, Q2). With SPOT, completed lifecycles increase from 13 to 26 for Retention and from 0 to 8 for Withdrawal. For the other two types, a correct repair counts toward completion when it follows a valid initial answer and meets the 2s deadline (Appendix O.3):
The two Replacement trajectories meeting both conditions repaired after 5.836s and 13.583s. For Formation, the missing step was publishing NONE during its initial validity interval.
4. Public answers and evidence from the same native state (W3, Q3). Among 389 cases with successful independent probes, we found three outcomes for the public answer at tR+2s:
The 44.26% figure uses published answers as its denominator (104/235). We now report all three outcomes in the abstract, introduction, and main results.
To examine the state used for the actual action, we scored candidate answers on a copy of that same native cache (100 event-centered Replacement clips per model):
Thus the model can prefer the updated candidate while WAIT leaves a stale answer on screen. We use mean-token scores; sequence-sum scoring agrees in sign on 49/100 MiniCPM-o and 42/100 MMDuet2 clips (Appendix H.3; Section 6).
5. Media time and runtime (W3, Q3). We score each decision as if it were ready at its scheduled media time, using the evidence available then. This gives every model the same 2s repair window. We report runtime separately: JoyAI's median is 181ms (88.45% of decisions within 500ms); MiniCPM-o's is 342ms (71.55%). Appendix O.4 covers all 30 configurations and distinguishes native updates from automatic WAIT returns. Deployment latency also depends on hardware, service conditions, and how incoming frames are queued.
6. Participant procedures (Ethics). For the original interactive study, participants were recruited through an open call and paid. They were informed about the research and use of their responses, including LLM judging, and consented before participation. The collection webpage records consent with a checkbox and timestamp, and exports responses and participant IDs as JSON/CSV files.
Participation was voluntary; no vulnerable population was involved. Under our institutional policy this design required no ethics review, since no sensitive or personally identifying information was collected beyond a self-chosen participant ID. Exports are stored by the authors and are not publicly released.