acceptodds
Under review as a conference paper at ICLR 2027

VideoLive: Benchmarking Stateful Real-Time Audio-Visual Agents

10 views

Abstract

Multimodal large language models (MLLMs) have demonstrated strong video understanding capabilities, yet they are mostly evaluated on answers given at isolated moments. A correct answer at one moment does not show whether a model can keep taking correct and timely interactive actions as the scene evolves. Real-time interaction therefore requires a model to generate answers and to decide when to **ANSWER**, when to **WAIT**, and when to **REPAIR** an answer that has become stale. We introduce VideoLive, a *stateful streaming* benchmark that evaluates real-time audio-visual interaction through a unified causal protocol. VideoLive continuously requires an agentic MLLM to make decisions from the accumulated evidence available up to the current moment, autonomously choosing whether to answer, wait, or repair, and evaluates its capability across the full lifecycle of an answer, from forming an initial answer through retention to update. We evaluate 29 model configurations across four lifecycle types: Retention, Replacement, Withdrawal, and Formation. The results show that forming a correct initial answer does not guarantee timely subsequent updates: the best configuration completes **23.50%** of answer lifecycles, and across all 29 configurations only **21.11%** of valid initial answers were revised correctly and in time after the state changed. Even when an independent probe identifies the new activity, **44.26%** of the public answers from incremental interfaces remain incorrect two seconds later, revealing a substantial gap between content understanding and continuous interactive decision-making. We further propose **SPOT**, a post-training method that guides MLLMs to learn when to publish, retain, or update an answer by supervising their replies conditioned on the displayed answer; it improves MMDuet2 on every VideoLive metric, raising LCS from 3.25 to 8.50 and Overall from 15.44 to 25.72, and improves response timing on three held-out streaming benchmarks. We hope VideoLive helps move research toward reliable real-time interaction.

Primary area:
datasets and benchmarks
Keywords:
streaming video understanding, real-time audio-visual interaction, multimodal large language models, answer lifecycle evaluation, streaming benchmarks, post-training for streaming agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xvnnOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

VideoLive evaluates a persistent public answer rather than scoring isolated video responses. On a causal media clock, a model chooses WAIT, ANSWER or REPAIR, and its answer remains visible between decisions. Core-400 balances Retention, Replacement, Withdrawal and Formation using 400 distinct source videos. Evaluation combines lifecycle completion, valid-answer duration and independent content/action probes across 29 configurations. The paper also proposes SPOT, which pairs the same evidence with alternative displayed answers during supervised tuning. On MMDuet2, SPOT raises LCS from 3.25 to 8.50 and VAT from 8.95 to 22.60, and improves the reported held-out streaming-benchmark metrics.

Critical strengths
  1. Persistent-answer validity is a useful evaluation target. Distinguishing an empty display from a published NONE answer, and distinguishing answer formation from maintenance and repair, reveals failures that endpoint-only accuracy cannot capture. LCS and VAT are complementary rather than interchangeable.

  2. The benchmark construction and evaluation engineering are documented in considerable detail. The study uses one primary trajectory per source video, a balanced lifecycle allocation, explicit causal access rules, source-video-clustered uncertainty and separate interface groups. The appendices include parser, timing, modality, semantic-matcher and ranking sensitivity analyses; these are meaningful strengths, not missing controls.

  3. SPOT is a simple response to the proposed state-management problem, and the training sources are disjoint from Core-400. The paper includes an instance-matched ordinary-SFT control on held-out benchmarks and explicitly exposes the change in target-action proportions. Native-cache readouts also provide a useful complement to independent probes, although they should not be overinterpreted.

Critical weaknesses
  1. The human result does not yet calibrate the benchmark on the same task as the models. Appendix E.6 states that participants received abbreviated questions without the activity-scope lists supplied to models; their answers were in Chinese, and their phase scores were sampled retrospectively from the interactive traces rather than obtained through the same independent content-probe protocol. Therefore, the Human row should not be read as a directly comparable ceiling. More fundamentally, the source annotations establish activity intervals, not necessarily the earliest point at which a causal observer can identify an activity. The 36-item human semantic spot check in Appendix I.1 validates reference–prediction matching, not the perceptual sufficiency of those time boundaries. This distinction is important for interpreting the very low absolute scores. I acknowledge the existing boundary sensitivity experiments, but they do not replace a matched-task, prefix-based annotation audit.

  2. The SPOT comparison does not isolate state pairing from target-frequency changes. Table 29 shows publication targets increasing from 14.0% to 41.5% and target tokens increasing from 111,120 to 126,000 despite equal instance counts. Thus, ordinary SFT versus SPOT changes both counterfactual display exposure and the amount/distribution of publication supervision. A frequency-matched or reweighted control is needed to identify the benefit of conditioning on alternative displays. Ordinary SFT is reported on the external benchmarks but not alongside SPOT on the primary Core-400 table. The improvement also remains concentrated: Figure 4 shows Retention rising from 13 to 26 and Withdrawal from 0 to 8, while Replacement and Formation completion remain zero. These are useful gains, but not yet broad repair competence.

  3. The scope of “real-time” and of the content–action interpretation should be tightened. The primary scores use media time, and Appendix L.2 explicitly advances the hosted media clock only after a response completes. This measures causal, time-indexed behavior rather than demonstrated wall-clock deadline compliance. Similarly, the 44.26% mismatch is 104 incorrect answers among 235 published answers after an independently executed probe succeeds, not 44.26% of all 389 recognized cases; another 154 cases have no published answer. Appendix F.7 states these denominators correctly, but the headline interpretation should consistently distinguish separate-execution recognition from knowledge available in the same native action state. The native-cache analysis helps, while its sensitivity to candidate-length normalization also warrants caution.

Questions for authors
  1. Can the Human row be recomputed or clearly separated under matched question scopes and probe protocols? Can a prefix-based human audit validate the claimed evidence-support boundaries, distinct from checking semantic answer matching?

  2. What are ordinary-SFT and action-frequency-matched SFT results on Core-400? Does SPOT still improve lifecycle completion after matching the publication-target distribution, and what explains the remaining zero Replacement/Formation completion?

  3. Please distinguish media-clock scores from wall-clock latency and present the recognized-but-correct, recognized-but-incorrect and recognized-but-unpublished categories together. Which conclusions about a content–action gap are supported by the same native state rather than independent executions?

1 comment
AuthorsOpenReview ↗

Matched controls, public-state evidence, and study procedures

Thank you for the careful and constructive review. We appreciate your distinction between recognizing an activity and keeping the displayed answer valid, and your question about what a viewer can recognize at an annotated boundary.

1. Human prefix audit and temporal references (W1, Q1). We added a prefix-based audit with eight participants: 906 judgments on 339 prefixes from 120 videos, covering all four lifecycle types. Participants watched continuous silent video and answered a single-choice question about the current activity, choosing from the full activity vocabulary with English labels and Chinese translations. Each person saw at most one prefix per source video.

At the decision point aligned with the annotated boundary, accuracy was 176/192 (91.7%), including 87/93 (93.5%) at initial boundaries and 89/99 (89.9%) at updated boundaries. These results provide direct human evidence of activity recognition at the sampled reference boundaries. Appendix O.2 gives all five offsets, their denominators, and the existing window and boundary-shift analyses. “First-sufficient” denotes the first grid point at or after the annotated onset.

In Table 2 we report the original interactive human study. The caption and Appendix E.6 describe the shorter questions without scope lists, Chinese free-text answers, replay, unrestricted thinking at pauses, and phase scores sampled from answer histories.

2. State pairing and publication supervision (W2, Q2). Thank you for suggesting a control that matches the frequency of publication targets. In Table 3, SPOT outperforms ordinary SFT on all six transfer metrics with the same training data, epochs, and number of examples. A reply is the training target in 14.0% of ordinary SFT examples and 41.5% of SPOT examples. Holding that share fixed would help test whether state pairing improves lifecycle completion on Core-400. With limited GPU time for validation during rebuttal, we plan to report that comparison in the final paper.

3. What the lifecycle stages show (W2, Q2). With SPOT, completed lifecycles increase from 13 to 26 for Retention and from 0 to 8 for Withdrawal. For the other two types, a correct repair counts toward completion when it follows a valid initial answer and meets the 2s deadline (Appendix O.3):

Trajectories (100 per type)ReplacementFormation
Valid initial answer230
At least one correct repair88
Both conditions above20
Complete lifecycle00

The two Replacement trajectories meeting both conditions repaired after 5.836s and 13.583s. For Formation, the missing step was publishing NONE during its initial validity interval.

4. Public answers and evidence from the same native state (W3, Q3). Among 389 cases with successful independent probes, we found three outcomes for the public answer at tR+2s:

Public answerCasesShare of 389
Correct13133.68%
Incorrect10426.74%
Unpublished15439.59%

The 44.26% figure uses published answers as its denominator (104/235). We now report all three outcomes in the abstract, introduction, and main results.

To examine the state used for the actual action, we scored candidate answers on a copy of that same native cache (100 event-centered Replacement clips per model):

ModelPrefers updated referenceOf these, incorrect displayOf those, WAIT
MiniCPM-o474134
MMDuet2494137

Thus the model can prefer the updated candidate while WAIT leaves a stale answer on screen. We use mean-token scores; sequence-sum scoring agrees in sign on 49/100 MiniCPM-o and 42/100 MMDuet2 clips (Appendix H.3; Section 6).

5. Media time and runtime (W3, Q3). We score each decision as if it were ready at its scheduled media time, using the evidence available then. This gives every model the same 2s repair window. We report runtime separately: JoyAI's median is 181ms (88.45% of decisions within 500ms); MiniCPM-o's is 342ms (71.55%). Appendix O.4 covers all 30 configurations and distinguishes native updates from automatic WAIT returns. Deployment latency also depends on hardware, service conditions, and how incoming frames are queued.

6. Participant procedures (Ethics). For the original interactive study, participants were recruited through an open call and paid. They were informed about the research and use of their responses, including LLM judging, and consented before participation. The collection webpage records consent with a checkbox and timestamp, and exports responses and participant IDs as JSON/CSV files.

Participation was voluntary; no vulnerable population was involved. Under our institutional policy this design required no ethics review, since no sensitive or personally identifying information was collected beyond a self-chosen participant ID. Exports are stored by the authors and are not publicly released.

Reviewer_SkQTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VideoLive, a benchmark for evaluating stateful real-time audio-visual interaction through answer formation, retention, and timely revision. It constructs 400 video trajectories across four lifecycle types and evaluates 29 model configurations, revealing significant limitations in maintaining valid answers over time. The authors further propose SPOT, a state-aware post-training method that improves answer maintenance and response timing.

Critical strengths
  1. The paper presents a novel perspective on streaming video understanding by evaluating the lifecycle of persistent answers, including their formation, retention, and timely revision.
  2. The experiments are comprehensive, covering various model configurations with detailed analyses of model limitations. The proposed SPOT method also demonstrates improvements on VideoLive and three held-out streaming benchmarks.
Critical weaknesses

The proposed REPAIR mechanism may not fully align with the behavior of truly streaming models that support continuous proactive responses. In such models, subsequent responses naturally describe evolving events rather than correcting previously valid answers. The assumption of a single persistent public answer may therefore limit the applicability of VideoLive to more general streaming interactions.

Questions for authors
  1. How does VideoLive distinguish between a new proactive response and a REPAIR of a previous answer? In particular, how would the proposed framework accommodate truly streaming models that continuously generate responses as events unfold, rather than explicitly replacing previous answers?
  2. SPOT is only evaluated on MMDuet2 in Table 2. Is there a specific reason for choosing this model, and how well would the proposed training strategy generalize to other streaming models with different architectures and inference mechanisms?
  3. Given that VideoLive evaluates complete interaction trajectories, what motivates the choice of SFT rather than trajectory-level reinforcement learning for SPOT?
3 comments
AuthorsOpenReview ↗

Proactive updates and state-conditioned learning

Thank you for the thoughtful review. We appreciate your recognition of the persistent-answer setting and the gains on three held-out benchmarks. Your question about a model that speaks proactively helps clarify what counts as a repair.

1. How proactive responses implement REPAIR (W1, Q1). We evaluate a standing question such as “What activity is happening now?” The public answer remains visible between replies and is updated as the activity changes.

When the person stops pouring and starts drinking, the proactive reply “drinking water” replaces “pouring water” on screen. We score that update as REPAIR. The earlier answer was correct when it was published. For this task, silence maps to WAIT, the first publication to ANSWER, and subsequent publications to REPAIR. SPOT uses this mapping through MMDuet2's native reply interface, without a separate action head (Sections 3 and 5).

We also measure how long the old answer stayed visible before the update: LCS tests timely lifecycle completion, and VAT measures how long the displayed answer stays valid. Two systems can reach the same correct reply and still differ in how long their users read a stale answer, which is why continued validity is measured alongside the correctness and timing of individual responses.

2. Why MMDuet2, and what transfers (Q2). We chose MMDuet2 because its trainable streaming interface exposes the native cache and reply generation. We can hold the question, video history, and timestamp fixed, supervise replies under two displayed answers, and advance the cache with the reference-following reply once. Within this native interface, SPOT raises VAT from 8.95 to 22.60 and LCS from 3.25 to 8.50, the highest LCS among the open-weight incremental systems in Table 2.

Section 5 specifies the interface requirements for applying SPOT: access to the displayed answer, trainable reply generation, and control over the reply committed to persistent history. On MMDuet2, we evaluate the resulting policy on VideoLive and three held-out streaming benchmarks.

3. Why paired SFT rather than trajectory-level RL (Q3). We chose SFT because our annotations tell us both when to respond and what the reply should be. For the same video prefix, the target is silence when the displayed answer is correct and the updated reference answer (including NONE) when it is stale. A trajectory reward scores the actions the model takes; paired SFT lets us train on the exact repair even when the model stays silent (Section 5).

Two studies illustrate this advantage by comparing SFT and GRPO on related tasks:

  • Learning labeled actions. Laskar et al. (REALM 2026, Table 2) use gold tool calls as SFT targets and as references for GRPO rewards. LoRA SFT ranked first in 15 of 18 in-distribution settings against GRPO and SFT followed by GRPO.
  • Learning from fewer examples. On visual-math benchmarks, Yu et al. (2025, Figs. 3–4) report higher accuracy for SFT than GRPO at small training-set sizes (1K–5K examples). SFT on 2K worked examples was competitive with GRPO on 20K problems.

We evaluate whole trajectories because each reply determines what stays on screen until the next update. After paired SFT, MMDuet2 more than doubles both lifecycle completion and valid-answer time (Table 2). With the same training trajectories and 21,048 examples per epoch for five epochs, SPOT also outperforms ordinary SFT on all six transfer measures reported in Table 3.

Thank you for the authors' response. Some of my concerns have been addressed, and I have raised my score accordingly. However, since this paper primarily focuses on benchmarking, I would still recommend including comparisons with a broader range of methods.

Thank you for your comment. We’re glad our response helped address some of your concerns. We agree that broader comparisons would be useful, and we’ll look at which other streaming methods we can include in the revision.

Reviewer_4btDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VideoLive, a benchmark for streaming video models where the answer stays on screen and must be maintained over time. Every 500 ms the model decides whether to wait, publish a first answer, or repair an answer that has gone stale. Each of the 400 videos follows one of four lifecycle types: keep the answer, replace it, withdraw it, or form it from a "nothing happening" state. The authors evaluate 29 model configurations and 14 human participants. The best model completes 23.5% of answer lifecycles, most open-weight systems almost never repair an answer, and models that recognize a new activity often still leave the old answer on screen. They also propose SPOT, a fine-tuning method that pairs the same video with different displayed answers so the model learns to react to its own output. It roughly doubles MMDuet2's scores and transfers to three other streaming benchmarks.

Critical strengths
  1. The problem is real and has been overlooked. Existing streaming benchmarks score each reply when it is produced and never check whether the reply is still correct a few seconds later.
  2. The task design is clean. Three actions, four lifecycle types, and a single causal clock that works for both native streaming systems and plain prefix-based models.
  3. The main finding is convincing and useful: models can recognize that the scene changed but do not update what they have already said. The paper shows this from several angles, including the fact that the displayed answer itself biases the next decision.
  4. There is a human baseline, bootstrap confidence intervals on every number, checks on the semantic judge, and sensitivity tests on timing boundaries, frame rate, delay, audio, and metric weights.
  5. SPOT is simple, needs no new architecture, and beats a plain SFT control trained on the same data on every metric reported.
Critical weaknesses
  1. The benchmark is small and narrow. It has 400 videos, one fixed question ("what activity is happening now"), a closed label set, and 90% of the clips are about six seconds long. I am not sure how much of the findings carry over to open-ended questions or longer interactions.
  2. Some of the difficulty may come from the rules rather than the models: answering before the annotated start counts as zero and the repair window is two seconds. The paper does test longer deadlines and shifted boundaries, but only on three systems that barely repair anything, so the test cannot show much. The human baseline also ran under different conditions (Chinese answers, a shorter question without the label list, unlimited response time), so the 19% human score is not a clean reference.
  3. The systems are not run under the same conditions. GPT models go through Codex CLI with low reasoning effort and 50 images, Gemini gets 1 FPS audio-video, the Qwen models use a live WebSocket, and the open-weight systems use their own settings. The leaderboard compares setups as much as models.
  4. SPOT is only tested on MMDuet2, which is one of the weakest systems, and the result is still low (LCS 8.5). It is unclear whether the method helps a model that is already decent.
Questions for authors
  1. The semantic judge is Qwen3.8-27B, and Qwen3.8-27B is also one of the evaluated systems. Did you check whether the judge is more lenient on its own outputs?
  2. All scoring uses media time, so a model that takes several seconds to generate a reply is not penalized. How far are the evaluated systems from actually running at the 500 ms cadence, and would the ranking change if wall-clock latency counted?
3 comments
AuthorsOpenReview ↗

Scope, timing sensitivity, and judge audit

Thank you for the constructive and balanced review. We appreciate your recognition that a model can notice a change yet leave its old answer on screen, and that the evaluation should capture both behaviors.

1. Scope and duration (W1). We study 400 distinct source videos, 100 per lifecycle type. A standing question about the current activity gives each answer a defined validity interval; we accept free-text paraphrases within the question's scope. The 360 Perception Test clips have a median duration of 6.25s. The 40 CaptainCook4D clips have a median of 48.29s and extend to 218.93s. We report results by source in Appendix O.5. For open-ended replies, one possible extension is to track how long each claim remains valid as the scene changes.

2. Timing and the Human reference (W2). We varied the repair deadline on the same 84 Changed videos observable through 5s. Longer deadlines increased completed lifecycles for all three configurations below.

ConfigurationCompleted by 2sCompleted by 5sGain in points (paired 95% CI)
Gemini 3.8 Flash1320+8.33 [3.57, 14.29]
GPT-6 Sol1724+8.33 [2.38, 14.29]
Qwen3.8-27B45+1.19 [0, 3.57]

Appendix O.2 gives the 1/2/3/5s results and boundary shifts. LCS measures completion by the deadline; VAT also credits time spent displaying the correct answer after a late repair.

We report the 19% human completion rate as an interactive reference. The Table 2 caption and Appendix E.6 describe the shorter questions, Chinese responses, unrestricted thinking at pauses, and phase scores sampled from participants' answer histories.

3. Configuration comparisons (W3). We apply the same causal access rules and scoring across configurations and group them by interface in Table 2. Appendices B and L specify GPT's 50-image cap (8 recent + 42 historical), Gemini's 1 FPS audio-video input, Qwen Realtime's incremental sessions, and the local systems' native state mechanisms. Across both interface groups, forming an initial answer is easier than maintaining it through a complete lifecycle.

4. SPOT and backbone choice (W4). We chose MMDuet2 because its trainable reply interface lets us vary the displayed answer while holding the native cache and media history fixed, then choose which reply enters the history. On MMDuet2, SPOT improves all eight scores in Table 2. VAT rises from 8.95 to 22.60 and LCS from 3.25 to 8.50, the highest LCS among the open-weight incremental systems in the table. Retention completions rise from 13 to 26 and Withdrawal from 0 to 8. We also find gains on three held-out streaming benchmarks (Table 3). For another backbone, the key requirements are a trainable reply interface and control over the displayed answer and the reply committed to history.

5. Judge self-preference (Q1). We compared the Qwen judge with a blinded GPT-6.1 Sol judge on 320 outputs from the same 80 videos. Each system contributed 40 natural answers and 40 probes, balanced by lifecycle type. We hid system identities and prior labels and reused one judgment for identical prompts.

Source of answersJudgments in agreement
Qwen3.8-27B80/80
Gemma 4 12B80/80
Molmo2-8B80/80
MiniCPM-o71/80

The judges agreed on all 80 Qwen outputs. All nine disagreements came from MiniCPM-o: Sol accepted those answers and Qwen rejected them. The nine cases cover three distinct reference–answer pairs, one repeated seven times. The Qwen-minus-Sol acceptance gap was zero on Qwen outputs and averaged −3.75 percentage points across the other three systems (contrast: +3.75 points; 95% CI [1.67, 5.83]). Appendix O.1 also reports the prespecified two-control estimate: +5.625 points [2.50, 8.75].

6. Media time and deployment latency (Q2). We score each decision at its scheduled media time using the evidence available then. This keeps the 500ms grid and 2s repair deadline fixed across hardware. We measured wall-clock latency separately. For each of the three configurations below, the denominator is all 10,343 decisions, including WAIT and errors:

ConfigurationLCSVATp50/p95 (s)Within 500ms
JoyAI-VL-Interaction4.0011.670.181/0.70388.45%
MiniCPM-o3.5014.620.342/1.80371.55%
StreamingVLM0.250.280.155/0.30299.81%

On StreamingVLM's 5,278 native updates, p50/p95 is 0.272/0.308s and 99.62% finish within 500ms. Appendix O.4 reports all 30 configurations, excluding saved-output replay, with native-update breakdowns. A wall-clock ranking would also depend on the hardware, service latency, and how each system queues incoming frames.

Thank you to the authors for their response, which has addressed most of my concerns. I will maintain my original score of 3 (Week Acceptance).

Thank you for your feedback and support. We’re glad we could address most of your concerns, and we appreciate your time reviewing our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.