acceptodds
Under review as a conference paper at ICLR 2027

WorldLineBench: Toward Process-Oriented Evaluation of Video-Grounded Streaming Interaction

Abstract

Streaming video generation endows video, for the first time, with a true worldline: a continuously unfolding visual trajectory that remains open to real-time user intervention. This transforms video generation into a fundamentally new interactive task that requires timely responses while preserving an already-established visual history, posing new challenges to existing evaluation paradigms centered on endpoint quality. To this end, we present WorldLineBench, to our knowledge the first large-scale video-grounded benchmark for streaming interaction, comprising 1,000 synthetic and 1,000 real videos spanning five content contexts and seven interaction types. Synthetic videos provide controlled instruction coverage while real videos probe whether interactive capability generalizes to natural scenarios. We adopt a 45-second long-horizon design supporting multiple rounds of history-dependent interaction, constructing branching worldlines from different interactions at designated nexus points. Under a process-oriented evaluation covering four metrics across two capability dimensions, WorldLineBench uniformly evaluates seven representative models from three paradigms, revealing consistent gaps across responsiveness, semantic fulfillment, and history-dependent preservation.

Primary area:
datasets and benchmarks
Keywords:
Streaming Video Generation, Runtime Interaction, Video-Grounded Evaluation, Multi-Round Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3U3NOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WorldLineBench, a benchmark for interactive streaming video generation and editing. The benchmark comprises 2000 real and AI generated videos with 6000 events. Experiments on seven models reveal gaps between fast responses and successful instruction execution. Human validation on the VLM-as-Judge shows high correlations.

Critical strengths
  1. The illustrations are visually good and clear
  2. The paper proposes an systematic evaluation for interactive video generation with broad coverage, highly human-aligned judgements.
  3. The multi-round interaction task setting is well documented.
Critical weaknesses
  1. The evaluation still relies on human in the loop check in the multi-round rollout. This may limits the large scale and automatic evaluation of tasks. Reliant on human choice also introduces potential bias.
  2. As Video generation is costly, how much (in USD), how many gpu hours, how long are spent on evaluating the whole 2000 cases. If it is too costly, it also prevents the application. Any plans to release a small version with comparable testing effect?
Questions for authors

Please address the questions mentioned above

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3U3N Comments

W1: Human Dependence and Automation Feasibility (Appendices B.3.3, B.4)

The reviewer's constructive feedback has helped us better identify opportunities to improve the automation and reliability of WorldLineBench's multi-round construction. In the submitted benchmark, GPT-5.6 Sol generated three ranked E2B candidates from each model's actual first-round output, specifying interaction instructions and times, followed by human selection and quality control. Unlike E1 and E2A, E2B instructions depend on each model's generated history and cannot be fully predetermined. Among 1,000 JoyAI synthetic cases, reviewers accepted the initial Top-1 candidate in 963 cases (96.3%), while fewer than 5% required correction for other models.

For the remaining 37 cases, we tested automatic candidate reconstruction. An independent GPT-5.6 Sol Agent received the video, assigned interaction type, and previously unsuitable candidates, then regenerated three ranked candidates rather than merely selecting from the original proposals. Human quality checks confirmed 33 valid reconstructions (89.2% recovery), yielding 996/1,000 (99.6%) usable cases across the initial and reconstructed proposals in this diagnostic. The remaining four primarily involved blurred videos, ambiguous targets, or insufficiently persistent states. These results demonstrate that most initially unsuitable candidates can be recovered through independent reconstruction.

To remove per-case human selection, we further developed a confidence-based automatic routing protocol. Initial candidates with a Top-1–Top-2 confidence margin above 0.4 are accepted directly; otherwise, an independent Agent regenerates the instruction and interaction time, and its Top-1 candidate is adopted. These results suggest the potential for reliable automation with reduced human involvement. The threshold is informed by the observed confidence patterns in human-reviewed cases, while unified prompts and decision rules standardize candidate selection across models and reduce reviewer-dependent variation. We will provide the construction protocol, validation statistics, residual failure cases, code, and prompts in the revised appendix.

W2: Evaluation Costs and a Lightweight Benchmark

We have carefully considered the reviewer's suggestions and further examined the costs and accessibility of WorldLineBench. WorldLineBench provides 2,000 source videos and E1/E2A instructions, requiring 6,000 interactions per evaluated model. Generation cost varies by model: on H20-equivalent hardware, JoyAI requires approximately 45 GPU hours, StreamDiffusionV2 72, StreamEdit 525, and Wan2.1-VACE 1,312, with other models detailed in the appendix. Model-specific E2B instruction construction using GPT-5.6 Sol costs approximately 104.8 USD per model with input caching, with lower-cost alternatives offering potential savings. Non-VLM metrics require approximately 34 GPU hours per model, while Gemini 3.8 Flash judging costs about 133.8 USD, bringing combined API costs to approximately 238.6 USD per model.

To further reduce costs, we evaluated Qwen3.8-27B as an open-source alternative judge. Across nine VLM-related metrics, its pairwise rankings agree with Gemini’s in 177/189 comparisons (93.65%), with the same top-ranked model on all nine metrics. Judging 6,000 interactions requires approximately 90 H20 GPU hours, suggesting its potential for lower-cost preliminary evaluations using local computation. For final reporting, we recommend the standardized Gemini protocol to preserve comparability.

We also investigated whether smaller subsets retain comparable conclusions. Using source videos as the sampling unit, we examined nine subset sizes from 50 to 1,000 videos under two strategies: stratified random and interaction-type-balanced sampling. Both stratify by video source (real or synthetic) and five content categories; the latter additionally balances interaction types and subtypes. Each selected video retains E1, E2A, and E2B interactions. Sampling uses no model scores and excludes no failure cases. We repeated each configuration 30 times to assess metric values and model rankings against the full benchmark.

Small subsets preserve broad trends but cannot reliably reproduce full results. Under type-balanced sampling, 200 videos yield an average Spearman correlation of 0.948, yet only 5/30 repeats meet predefined error tolerances across all ten core metrics and seven models. At 400 videos, correlation rises to 0.978, pairwise ranking agreement reaches 97.2%, and 27/30 repeats meet all tolerances. Increasing to 600 videos improves stability but incurs higher costs with diminishing returns. We therefore adopt 400 videos (1,200 interactions per model) as WorldLineBench-Light, reducing generation and judging workloads by approximately 80%, while retaining the full benchmark for formal evaluation. Per-model costs, subset comparisons, and sampling protocols will be provided in the revised appendix.

Reviewer_3U3NOpenReview ↗

Reviewer Response

Thank you for the clarifications and additional analyses. They address my concerns. I have updated my scores. Good luck to authors.

AuthorsOpenReview ↗

Sincere Thanks to Reviewer 3U3N Feedback

We sincerely thank you for the time and effort you have devoted to reviewing our paper, and especially for your timely response. We are glad that our response have addressed your concerns and are also greatly encouraged by your positive feedback. We truly appreciate your renewed consideration of both our work and your assessment confidence. Thanks again for your thoughtful review and support for our WorldLineBench.

AuthorsOpenReview ↗

Revised Appendix and Sincere Thanks

Dear Reviewer 3U3N,

We are particularly grateful that you were the first reviewer to respond to our rebuttal, explicitly acknowledging that our responses had resolved your concerns and increasing your confidence score. Your timely and encouraging feedback means a great deal to us.

Following the discussion, we have incorporated the additional evidence and implementation details into the revised appendix, particularly regarding E2B automation and efficiency. The corresponding revisions are summarized below.

Reviewer commentCorresponding section in the revised appendixMain additions
W1B.3.3 (Table 7), B.4Automated E2B construction and quality control
W2G.3 (Tables 32–34), G.5 (Tables 38–39)Evaluation costs and lightweight benchmark validation

We hope these revisions further reinforce your confidence in the reliability and practicality of WorldLineBench. We sincerely appreciate your recognition, thoughtful feedback, and continued engagement throughout the review process.

Thank you again for your time and consideration.

Reviewer_9Zc6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper identifies a genuine and timely gap: existing benchmarks for video generation and editing are outcome-oriented and single-round, unable to evaluate streaming interaction that demands timely responses while preserving established visual history. WorldLineBench contributes a large corpus (2,000 videos, 6,000 interactions) with video-grounded, multi-round, branching worldlines and a process-oriented evaluation covering four metrics across two capability dimensions. The seven-model evaluation produces clear, interpretable findings—notably the gap between timely response and full instruction execution, and cross-round history-dependent preservation failures. This is a valuable resource. The main limitation is the reliance on automated judges (Gemini-3-Flash) for several metrics, whose calibration against human judgment is not deeply validated. Overall, a well-designed benchmark that will likely become a reference for the field.

Critical strengths
  1. The paper identifies a genuine and well-articulated gap: existing benchmarks treat interaction as a one-shot, offline event with predefined prompts, whereas streaming interaction is a continuous, state-grounded process. WorldLineBench jointly provides video-grounded instructions, branching worldlines from shared history, and process-oriented (onset/fulfillment/persistence) assessment, as evidenced by Table 1 where it is the only entry satisfying all ten desiderata.

  2. The experiments are broad and the analysis is inspiring and informative: seven models across three paradigms, a 7-type × 5-context difficulty landscape, subtype-level EFA/PSR profiles, a Real-vs-Synthetic asymmetry study, and a round-wise degradation analysis that surfaces two well-supported findings (the DSR collapse and the universal failure of cross-round history inheritance). The authors also audit their VLM judge against blinded human review and report the bias and the limits of the audit.

Critical weaknesses

1.Metric validity is under-established: a single closed-source VLM judge, with uniformly low absolute scores. Terms like IS-g, EFA, SEO, etc, are all judged by Gemini 3.8 Flash, while human agreement is checked on only 100 validation cases (70 for SEO, 40 for CA-HDF) and speaks to correlation rather than to the trustworthiness of absolute values. Absolute scores are uniformly near the floor (best EFA 37.92%, IS-g only 1.56–2.46 out of 5, CA-HDF as low as 3.19%), which may due to weak models or a systematically harsh judge; without a human-preference ranking study or an ablation over alternative VLM judges, the paper cannot distinguish these two explanations.

2.The three paradigms are not compared under equal information conditions: full-context models retain bidirectional sequence-level inference and are explicitly treated as non-causal reference baselines, while streaming-native models see only causal history, yet all seven are placed in one table with shared bold/underline ranking. TTFF spans two orders of magnitude (2.30 s to 199.54 s) and is reported alongside semantic metrics without stratified presentation, which invites an apples-to-oranges reading of Table 2 that the paper's own caveats do not dispel.

3.The "long-horizon" claim is modest in practice: at most two successive interactions per path, a 15 s evaluation window (and a 5 s responsiveness window) per intervention, and second-round nexus points concentrated at 28.9 s leaving almost no post-hoc stabilization room. The genuinely long-horizon phenomena the conclusion invokes—error accumulation, identity drift, and stability decay over minutes and many rounds—are therefore not actually measured by this benchmark.

4.Several metrics lack discriminative power, and the benchmark's floor/ceiling is never calibrated. OSC spans only 0.176–0.186 across the seven models (a range of 0.01) and thus carries essentially no discriminative signal, yet it is presented as one of four headline semantic-fulfillment metrics; meanwhile most metrics sit near the floor (PSR as low as 3.19%, CA-HDF generally 10–40%). The paper provides no score-distribution or separation analysis and does not establish whether the benchmark will retain resolution as future models improve.

Questions for authors

With all six VLM-judged metrics resting on a single closed-source judge and absolute scores sitting near the floor, how do you demonstrate that these numbers reflect model capability rather than systematic judge harshness? Evaluations on longer-horizon videos are necessary to validate the effectiveness of the benchmark.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 9Zc6 Comments (Part I)

W1: VLM Score Validity and Interpretation of Low Scores (Appendices D.1–D.2, E.1, G.2)

We acknowledge the reviewer's concerns about VLM judgment validity and the interpretation of low scores. To assess scoring reliability and dependence on Gemini, we expanded the human audit and re-evaluated all 42,000 outputs with two open-source VLMs. Key findings are summarized here; full statistics will appear in the revised appendix.

Human Audit and Absolute Score Validity

Building on Appendix G.2, we stratified a new audit by video source (2), content category (5), and interaction event (3), randomly sampling eight cases per stratum per model (240/model; 1,680 total). Mean Gemini EFA (24.12% vs. 24.27%) and zero-EFA prevalence (69.11% vs. 68.66%) closely match the full benchmark, indicating coverage of its failure-dominated distribution.

Human and Gemini EFA averages are 24.07% and 24.12%, with Spearman ρ=0.969\rho=0.969, MAE of 1.74 percentage points, and mean bias of +0.04 points (95% CI: −0.27 to +0.36). In failure-dominated strata, their averages are 18.87% and 18.92% (bias +0.05). Human and Gemini IS-g both average 1.87/5, while SEO averages 14.00% and 14.27%. Gemini scores DSR and CA-HDF 1.19 and 2.95 points higher than humans, respectively, indicating leniency rather than excessive strictness. These results support the validity of absolute fulfillment scores, including on difficult cases.

Because WorldLineBench evaluates fulfillment, responsiveness, preservation, and history dependence separately, a single human preference ranking would conflate these capabilities. We instead compare metric-aligned judgments: human and Gemini produce identical seven-model EFA rankings on audited outputs (Spearman ρ=1.000\rho=1.000).

Open-source Judge Validation and Cross-judge Stability

Qwen3.8-27B and Qwen3.5-9B re-evaluated all 42,000 outputs using identical metric definitions, applicability rules, and observation windows for nine directly VLM-judged or VLM-derived metrics. Across 189 model-pair orderings (nine metrics × 21 pairs), 27B matches Gemini on 177 (93.65%), including the same top-ranked model for all nine metrics. The 9B judge matches 162 (85.71%), with five top-model matches. This supports cross-judge stability, especially for 27B.

Absolute differences remain metric-dependent: Gemini/27B/9B CA-HDF averages are 18.09%/16.07%/7.92%, and PSR averages are 48.39%/52.64%/56.34%. Outputs without valid judgments comprise 0.07% for 27B and 3.80% for 9B. Thus, 27B offers a promising local option for relative comparisons, while absolute scoring and coverage still warrant calibration.

Interpretation of Low Scores

The human audit supports Gemini's absolute scores, and both open-source judges produce similarly low fulfillment results. Gemini/27B/9B mean EFA is 24.27%/25.02%/24.57%, and IS-g is 1.88/1.89/1.91. This convergence indicates that low scores are not peculiar to Gemini's scale. Round-wise EFA averages are 27.59% (E1), 29.21% (E2A), and 16.00% (E2B); the E2B decline indicates the added challenge of history-dependent multi-round interaction.

W2: Comparability Across Paradigms and Metrics (Appendices F.1–F.2)

We acknowledge the reviewer's constructive suggestions on cross-paradigm evaluation fairness. We agree that causal streaming systems and non-causal full-context references should not share a unified ranking, and address this issue below.

Information Conditions and Result Presentation

Streaming-native models perform causal frame-wise or chunk-wise generation with cached history; real-time editors process source frames; full-context models retain bidirectional sequence-level inference as non-causal references. We will group Table 2 by paradigm, remove cross-group bold/underline rankings, and visually separate TTFF from semantic metrics. Section 5 will emphasize within-paradigm comparisons and interpret cross-paradigm results as performance under different information-access conditions, while Appendix F will specify the information available before and after tet_e.

TTFF Definition and Presentation

The TTFF range reflects substantial differences in native execution: some models can produce outputs incrementally, while others require more extensive processing before the first media output. The original TTFF normalized first-generation costs by generation granularity rather than measuring actual first-output latency. We acknowledge that the original presentation may have caused confusion and therefore adopt better-defined wall-clock TTFF, measured from the start of input processing to the first retrievable media output. For example, JoyAI achieves 1.842s, whereas StreamEdit requires 453.938s, illustrating the substantial latency differences across execution modes. We will report the complete results in the revised appendix and clarify that TTFF measures practical latency under native execution conditions, not performance under equal information access.

AuthorsOpenReview ↗

Response to Reviewer 9Zc6 Comments (Part II)

W3: Long-Horizon Interactions and Error Accumulation

We value the reviewer's perspective on long-horizon interactions and error accumulation. Real-world systems handle instructions over minutes, beyond the main benchmark's two rounds and short evaluation windows. WorldLineBench uses 45-second videos to balance temporal coverage, interaction diversity, scale (2,000 videos), and cost. E1, E2A, and E2B capture historical dependencies beyond single edits. Although E2A performs similarly to E1, history-dependent E2B shows a clear performance drop, exposing challenges even within two rounds. We further examine longer durations and additional interactions.

We conducted a long-horizon diagnostic on 50 real videos (~120 seconds) using JoyAI, the top-performing model in EFA, IS-g, and SEO. We compared two-round (10, 90 seconds) and five-round (10, 30, 50, 70, 90 seconds) interactions, sharing the same source videos, actual first-round outputs, and first and final instructions. Subsequent edits followed their respective generated histories, allowing us to assess the impact of three additional interactions. The extended duration and final 115–120-second evaluation window enabled longer-term preservation analysis. Human reviewers applied the paper's evaluation criteria, assessing preservation only for successfully established, still-valid edits and excluding legitimate overwrites.

The results show that additional interactions degrade both state preservation and final-instruction performance. Among the same 16 videos with fully established first-round edits, endpoint preservation dropped from 53.13% (two rounds) to 14.58% (five rounds), a −38.54 percentage-point difference (95% CI [−63.33, −12.50]). For the identical final instruction at 90 seconds, EFA fell from 38.17% to 30.17% (−8.00 pp, 95% CI [−15.00, −1.50]), and IS-g from 2.54 to 2.14 (−0.40, 95% CI [−0.64, −0.20]). These findings reveal both the loss of previously established states and reduced ability to execute new instructions under accumulated history. Temporal analysis further showed that first-round preservation remained at 93.75% from 25–30 to 85–90 seconds without intervening edits, but dropped to 27.08% at 85–90 seconds under five rounds, before the shared final instruction. This suggests that additional editing interference contributes more clearly to degradation than time passage alone in this diagnostic. Since both conditions span the same duration, this indicates that intervening edits, rather than elapsed time alone, drive the additional loss observed before the final instruction. Non-target PSR and CF-g also declined, but their paired confidence intervals included zero, so we do not regard these changes as established effects.

Overall, the main benchmark reveals challenges of historical dependence, while the 120-second diagnostic shows that additional interactions substantially worsen state retention and subsequent edit fulfillment, highlighting the importance of maintaining historical consistency over long horizons. Human review observed identity drift, potentially arising from both model limitations and interaction history. This diagnostic focuses on JoyAI and does not separately quantify identity drift. We will provide the protocol and statistical results in the revised appendix, and extend future evaluations to longer durations, more models, and identity consistency.

W4: Metric Discrimination and Future Headroom (Appendices B, C, D.1, G.1)

We are grateful for the reviewer's constructive feedback on metric resolution and future headroom. We will add score distributions, floor/ceiling analyses, and statistical comparisons in the revised appendix.

Discriminative Power

OSC measures global semantic alignment and is less sensitive to local edits. Nevertheless, successful cases (EFA=1) consistently exceed failed cases (EFA=0) across seven models, with absolute gaps of 0.0054–0.0090 and relative gains of 2.96%–4.89%. We thus treat OSC as complementary to EFA, IS-g, and SEO, rather than ranking models by small OSC differences. For EFA, 17 of 21 model pairs differ significantly under Holm-corrected tests, indicating reliable discrimination despite low mean scores.

Floor, Difficulty, and Future Headroom

An unedited-source baseline achieves only 2.31% EFA and zero fully successful cases across 3,000 interactions in JoyAI's synthetic subset, providing a floor reference. Yet scores are not confined to this floor. Across type–context combinations, seven-model mean EFA ranges from 8.65% (Motion–Process) to 41.52% (Temporal–Scene); JoyAI ranges from 11.07% (Motion–Process) to 62.20% (Style–Embodied). Across all 42,000 outputs, 68.66% satisfied none of the required atoms, 13.80% achieved partial fulfillment, and 16.10% satisfied all requirements. JoyAI reached 70.82% EFA on Replacement, while EFA on Interaction ranged from 7.05% to 12.49% across the seven models. These harder edits leave considerable headroom for future models.

AuthorsOpenReview ↗

Appreciation and Further Feedback

Dear Reviewer 9Zc6,

We sincerely thank you for your positive assessment and support for our WorldLineBench. Your constructive suggestions have provided valuable guidance for improving our work, and we have made additional efforts to address the questions raised in your review. We hope our detailed responses and supplementary experiments can provide better clarification. Any further questions or suggestions are welcomed. Thank you once again for your encouragement and thoughtful consideration.

Reviewer_PjWiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes WorldLineBench for "streaming video interaction": a model watches an unfolding video, receives an instruction mid-stream, and must respond promptly while preserving the established history. It has 2,000 45-second source videos: 1,000 real clips curated from six public datasets and 1,000 synthetic clips from MiniMax H3, each stitched from two 22.5 s generations. Videos span five content contexts and 7 interaction types with 26 subtypes.

Each video carries three events. E1 is applied to the source A to produce branch B; E2A is applied to A; E2B is built from each evaluated model's own realized B. Sixteen metrics in four groups (semantic fulfillment, responsiveness, preservation, video quality) are computed over a 5 s response window and a 15 s evaluation window; nine are judged by Gemini 3.8 Flash.

Seven models from three paradigms are evaluated. The headline findings are a gap between first response and full completion (low DSR), low cross-round history retention (CA-HDF), and degradation from round 1 to round 2. A 100-case human audit of the VLM-judged metrics is included.

Critical strengths

1The three design pillars — state-grounded instructions, a branching two-round protocol, and process-level timing — go meaningfully beyond endpoint-only editing benchmarks. Table 1 positions the gap fairly. 2Instruction grounding is careful. Targets must be visible and identifiable, the requested result must not already hold, and E2B distinguishes semantic dependence from mere compatibility (B.3.3). 3The metric definitions (App. D–E) handle censoring honestly. SOL is reported with its coverage rather than imputed, DSR is gated on complete success, CA-HDF keeps failed prerequisites in its denominator, and SEO counts instantaneous rather than "ever satisfied" states. 4The human audit is well designed: stratified sampling, two blinded reviewers plus adjudication, a paired stratified bootstrap, Holm correction and disagreement tables. Weak subgroups (Synthetic DSR κ = 0.464; CA-HDF with four positives) are reported rather than hidden. 5Licensing and provenance of the real clips are documented per source, and the synthetic pipeline's revision process is concrete (Box A.3.3).

Critical weaknesses

W1. The benchmark is not self-contained for new models. E2B — a third of all events and every CA-HDF case — is built only after each model generates B, via VLM proposals plus human review of that model's output (B.3.3, F.2). A future user cannot evaluate a new model without repeating human annotation. E2B scores across models also come from different instructions, targets, timings and possibly types.

Please release, and validate against the human-reviewed version, a fully automatic E2B protocol; or redesign round 2 around fixed instructions with conditional applicability. At minimum, report per-model E2B type, subtype and timing distributions, and |D_m|. The paper says |D_m| is reported per model, but it does not appear. Fig. 2 and Tables 5–6 give planned counts; realized counts after quality control are not reported. Fig. 2(e)'s E2B timing is necessarily model-specific — whose B is it? W2. The task contract differs across paradigms, yet results form one leaderboard. Real-time editors receive the continuing source stream (F.2), full-context models see the whole sequence, and streaming-native generators work causally. Editing a known continuation and generating an unseen future are different tasks.

Preservation, SC and BC use the second before t_e as reference (D.1, E.1). For an editor, the source's natural evolution after t_e (a process advancing, people moving) should be reproduced. It is either penalized or judged against an undefined target. Please state per paradigm what each model sees at each time and what V⁺ should contain for non-target content. Consider separate leaderboards by information condition. W3. The responsiveness metrics do not measure real-time responsiveness as framed.

SOL, SCL and DSR are in media time, independent of wall clock. Wan2.1-VACE has a TTFF of ~200 s but an SOL of 4.19 s, in line with the others. For an offline model, when the edit appears in the clip is a design choice, not interactivity. D.2 defines TTFF as the first-output wait, but F.2 says frame-wise systems (JoyAI, StreamDiffusionV2) report per-frame runtime while others report an 81-frame-equivalent cost. If Table 2 follows F.2, the column mixes scales. Please state which definition Table 2 uses. All timing runs on H20 GPUs, far below the compute these "real-time" systems target. This likely explains why "streaming-native" StreamEdit (71 s TTFF) is slower than SANA-Streaming (14 s) and LiveEdit (26 s), undermining the paradigm reading in §5.2. Report throughput (fps) under one definition. DSR applies one 5 s completion deadline to every instruction. Many instructions (Temporal Progression; "let the cat jump off the table") cannot plausibly finish in 5 s, and a model that snaps to the target state beats one that renders a plausible transition. The annotation already records expected onset and completion (B.3.2); normalize DSR by it or stratify by expected duration. With most interactions failing, capped SCL is largely a monotone function of the failure rate and duplicates EFA and DSR. Report success rate and completion time among successes separately, as the audit already does (0.37 s MAE on 34 jointly completed cases). W4. Several claims in §5 are contradicted by the paper's own tables.

§5.2 says SANA leads CF-g and PSR "with Wan2.1-VACE following closely," suggesting full-context modeling helps preservation. In Table 2, JoyAI is second on both (CF-g 3.48 vs. Wan 3.40; PSR 68.46 vs. 66.21). The other full-context model, Lucy Edit 1.1, is worst on both (1.78; 18.83). §5.3, "Temporal achieves the highest EFA across all content contexts": in Table 7's Object column, Camera (39.17) and Style (32.04) exceed Temporal (29.70). §5.3, "Scene consistently underperforms across most interaction types": Scene is the lowest context in only 3 of 7 rows (Composition, Attribute, Spatial). §5.3, "Interaction registers the lowest EFA across all models": not for SANA (Addition 6.19) or Wan2.1-VACE (Addition 5.46) in Table 8. §5.4 and §6 claim pervasive round-2 degradation in video quality, but in Table 11 NRQ rises from E1 to E2B for six of seven models (all except JoyAI). The abstract says "four metrics across two capability dimensions"; there are sixteen metrics in four groups. W5. The round-wise analysis omits the control the design already provides. §5.4 attributes the E1→E2B drop (EFA 27.6→16.0, DSR 16.0→4.0) to accumulated history, while conceding the comparison is confounded by different instructions, later timing and shorter remaining output. E2A — a later, second-round intervention on the unedited A — is exactly the control that separates history dependence from "second instruction, later in the clip." It is never reported.

If Table 2 weights the three event types equally, the implied E2A EFA is at or above E1 for every model. Examples: JoyAI ≈ 45.1 (E1 45.1, E2B 23.6); LiveEdit ≈ 37.9 (36.6, 22.1); Wan2.1-VACE ≈ 27.2 (23.8, 15.7). That would make the history-dependence claim much stronger than presented. Please report E1, E2A and E2B side by side for all metrics, with CIs. W6. Preservation and history metrics are confounded with doing nothing and with E2B success.

High PSR, CF-g, SC and BC are achievable by not editing; SANA has the top PSR with low EFA. Add a no-op baseline (pass-through source for editors; frozen frame or unconditioned continuation for generators) and a joint metric such as preservation conditional on fulfillment. CA-HDF requires E2B completion (gate c). With low E2B completion, low CA-HDF mostly reflects E2B fulfillment, not history retention. Report each gate (u, c, d, h) per model, and h given c. As reported, CA-HDF exceeds the same model's mean E2B EFA in Table 11 for StreamEdit (19.68 vs. 13.57), SANA (17.91 vs. 11.21) and JoyAI (39.72 vs. 23.59). Complete success requires EFA_i = 1, so mean EFA upper-bounds the complete-success rate, and CA-HDF requires c. This holds only if dependency-applicable E2B cases are far easier than E2B overall, or if c is not full completion. Please clarify. W7. The judge audit samples a different operating regime from the benchmark. Audit mean EFA is 57.9% (human) and 62.9% (VLM); benchmark-wide EFA is 16–38% (mean 24.3%). The audit oversamples successes, while the benchmark lives in the failure regime, where the judge's main error — leniency (+5.0 pp EFA; 6 false vs. 2 missed passes on DSR) — matters most.

With ~14 cases per model, the audit cannot detect model-dependent bias, which is what rankings depend on. The claim that judge error is small relative to model differences fails for many pairs. OSC spans only 0.176–0.186; five models' IS-g lie within 0.23 of each other against an audit MAE of 0.27; StreamEdit, StreamDiffusionV2 and Wan2.1-VACE EFA lie within 1.4 pp. Please describe how audit cases were drawn within strata, add CIs or significance tests to Table 2, and report human-vs-VLM model-rank agreement on a larger per-model sample. Release cached judgments and prompts, since the benchmark depends on a proprietary, versioned judge. W8. Model configuration details are missing. Appendix F promises per-model settings but does not give resolution, frame rate, chunk length, how instructions are injected (a prompt switch for StreamDiffusionV2 and LongLive?), or how offline models produce a 15 s post-t_e window. Per-model output durations T_i matter, since shorter outputs lower the SCL cap and SEO denominators (the audit includes 4 s and 8 s windows). OSC is nearly constant across models, and its common-window intersection is undefined for E2B, where t_e is model-specific.

Questions for authors

1How will an external group evaluate a new model on E2B and CA-HDF without human annotation? 2What are the E1, E2A and E2B results side by side? 3Per paradigm, what does the model see after t_e, and what is the preservation reference for non-target content that naturally changes? 4Which TTFF definition does Table 2 use, and what is per-model throughput under one definition? 5How is gate c defined, what is |D_m| per model, and how do you reconcile CA-HDF with Table 11? 6How were audit cases sampled within strata, given the audit's much higher success rate? 7What does a no-op baseline score on each metric?

6 comments
AuthorsOpenReview ↗

Response to Reviewer PjWi Comments (Part I)

W1: Reusability and Fair Comparison (Appendices B.1–B.4, C)

We appreciate the reviewer's focus on reusability and fair comparison. Since E2B requires genuine history dependence, second-round instructions must reflect actual first-round outputs. We thus favor fully automatic construction over fixed instructions with conditional applicability, which may not ensure history dependence.

Reusability of VLM-based E2B Construction

Our pipeline uses GPT-5.6 Sol to propose three ranked candidates from first-round outputs. Among 1,000 synthetic JoyAI cases, human reviewers selected top-1 in 96.3% of cases; for other models, the proportion of cases requiring human correction was also below 5%. In the subsequent quality audit, the VLM flagged cases with ≥80% confidence of error. Of 21 flagged JoyAI cases, 14 had genuine issues, while seven had harmless timing offsets. These results support the reliability of VLM-based construction.

We accordingly adopt a fully automatic protocol informed by human review: top-1 is accepted when its confidence exceeds top-2 by more than 0.4; otherwise, an independent GPT-5.6 Sol instance re-determines the instruction, timing, and semantic dependence. The process requires no human involvement. Our original comprehensive audit validated benchmark quality rather than being a requirement for future users, who may optionally inspect a few uncertain cases at low cost. We will release the code and prompts, enabling automatic E2B construction, DmD_m identification, and E2B/CA-HDF evaluation for new models.

Cross-model Comparability and E2B Statistics

In the revised version, we will report the per-model distributions of E2B types, subtypes, and interaction timing, along with the number of cases corrected during quality control and the size of the history-dependent set DmD_m. Quality control only corrected problematic cases, and the final sample counts and type distributions, after replenishment, remain consistent with the original design in Fig. 2 and Tables 5–6. Fig. 2(e) presents the distribution aggregated across all models, which we will clarify in its caption.

The statistics show that E2B task compositions are largely consistent across models. Although second-round instructions are generated from each model's actual first-round output, resulting in differences in specific edit targets and interaction timing, the unified construction pipeline, constraints, and taxonomy quotas keep task distributions largely consistent. Moreover, E2B primarily evaluates each model's own multi-round history-dependent capability rather than serving as a direct cross-model ranking. These findings rely on within-model comparisons across rounds, further supported by the E2A control experiment.

W2: Model Information Conditions and Preservation Metrics (Appendices E.1, F.1–F.2)

Information access and preservation evaluation are essential to fair comparisons across paradigms. We will clarify the comparison protocols and further validate preservation metrics under natural video evolution.

Information Conditions and Result Presentation

Streaming-native models process incoming video chunks causally using cached history; real-time editors uses their native quick mechanisms respectively; and full-context models access the entire source video for offline bidirectional inference. Further details on each model's input mode, processing granularity, and instruction injection will be provided in the revised appendix. Additionally, we will group the results in Table 2 by information condition and remove cross-group rankings to avoid misleading comparisons.

Preservation Reference and Natural Evolution

Preservation is defined relative to the ideal output V+V^+: the edit target should undergo the requested change, while non-target content should follow the source video's natural evolution after tet_e, allowing changes induced by the edit. Thus, preservation metrics should not rely solely on static pre-edit references. Our improved SC and BC therefore combine comparisons with six pre-tet_e reference frames and the preceding output frame, with equal weights for the two terms (Eqs. 16–17), reducing penalties for smooth natural evolution. To address possible similarity biases from larger changes, VLM-judged PSR and CF-g further distinguish natural motion from genuine preservation failures, complementing feature-level consistency with semantic assessment.

We further conduct a source-continuation check, using the unedited source segment after tet_e as the output. Its SC and BC scores exceed those of all edited outputs by 1.5%–4.2% and 0.4%–2.9%, respectively. This shows that, although natural evolution affects feature similarity, SC and BC still assign higher scores to intact source continuation. Together with the semantic judgments of PSR and CF-g, this supports a more reliable assessment of preservation. We will report per-model raw and source-normalized SC/BC scores in the revised appendix.

AuthorsOpenReview ↗

Response to Reviewer PjWi Comments (Part II)

W3: Responsiveness Metrics (Appendices B.3.2, D.2, F.1–F.2)

The reviewer's comments highlight the need to distinguish content-level edit responsiveness from system execution efficiency. These are complementary aspects of evaluation. We will therefore clarify media time versus wall-clock time, unify the TTFF definition, introduce throughput (FPS), and further examine DSR and SCL using duration annotations and success-conditioned statistics. The corresponding results will be included in the revised version.

Media Time, Wall-clock Time, and TTFF/FPS

Our evaluation distinguishes two complementary time scales: media time measures when an edit takes effect and completes in the output video, while wall-clock time measures the actual waiting time for system output. SOL, SCL, and DSR therefore assess content-level responsiveness rather than inference speed. For offline models, edits are likewise anchored to the specified interaction point tet_e; SOL measures the observed onset delay relative to this point, not a preset delay. For example, Wan2.1-VACE has an SOL of 4.19s but a TTFF of 194.70s, illustrating the distinction between content-level responsiveness and actual output latency. Both time scales are therefore necessary for evaluating responsiveness.

The original Table 2 reported first-generation costs normalized by generation granularity rather than actual first-output waiting time. The revised version will instead adopt wall-clock TTFF, measured from the start of input processing to the first retrievable media output, and add end-to-end throughput (FPS), i.e., total valid output frames divided by total processing time, to reflect average processing efficiency. Representative results show that JoyAI and StreamDiffusionV2 achieve TTFFs of 1.842s and 5.549s with 11.86 and 8.16 FPS, whereas StreamEdit requires 453.938s at 1.12 FPS. These differences reflect output granularity, generation and decoding mechanisms, and implementation details beyond model paradigms. All models are timed on the same H20 GPUs, so relative comparisons remain fair, although absolute latencies would be lower on stronger hardware. We will refine the efficiency analysis in §5.2 and provide complete per-model TTFF, FPS, and measurement configurations in the revised appendix.

Deadline Setting of DSR

The uniform 5s deadline was designed to assess timely instruction completion in interactive scenarios. However, for instructions involving longer natural transitions, such as Temporal Progression, a fixed deadline may unfairly penalize valid behavior. We therefore introduce an adaptive deadline based on annotated expected completion times (Appendix B.3.2), setting the deadline to the expected completion time plus a 2s tolerance for long-duration instructions, while retaining 5s for shorter ones. Our analysis shows that only a small fraction of interactions, mainly in E2B, are affected, with overall DSR changing by approximately 1 percentage point. This suggests that the original deadline has limited impact on the overall findings. We will report both the original and updated DSR scores in the revised version to further assess the stability of our conclusions.

Informativeness of SCL

SCL captures both completion speed and the waiting cost of unsuccessful interactions by assigning failed cases the full evaluation window. Although it is strongly influenced by failure rates when success rates are low, this design also reflects the cost of non-completion. To better distinguish completion capability from completion speed, we will retain the original SCL while separately reporting success rates, mean completion times among successful cases, and corresponding sample counts. Current results show mean completion times of approximately 3–6.5s among successes, with JoyAI and LiveEdit relatively faster and Lucy Edit 1.1 and Wan2.1-VACE slower. These additional statistics will be provided in the revised appendix to clarify differences in completion efficiency.

W4: Refinement of Claims (Sections 5.2–6; Appendix G.1)

We acknowledge the reviewer's concerns regarding the alignment between our claims and experimental results. Our discussions a im to highlight overall trends rather than universal conclusions. While the main observations remain supported by the data, we will further qualify the observations regarding Temporal, Scene, and Interaction in §5.3, noting relevant exceptions. In §5.2, we will avoid directly attributing preservation performance to full-context modeling. In §§5.4–6, we will distinguish interaction fulfillment from video frame quality measured by NRQ. We will also correct the abstract to specify sixteen metrics in four groups across two capability dimensions. These clarifications will be incorporated into the revised manuscript to better align our conclusions with the experimental evidence.

AuthorsOpenReview ↗

Response to Reviewer PjWi Comments (Part III)

W5: Further Evidence on History Dependence (Appendices B.3, C, G.1)

We thank the reviewer for highlighting E2A as a valuable control for history dependence. E2A and E2B are second-round interventions at similar times, but E2A acts on the unedited source without depending on the first-round result. It thus helps distinguish history dependence from later intervention timing and shorter remaining duration. Across seven models, mean EFA on E1, E2A, and E2B is 27.59, 29.21, and 16.00, respectively, while mean DSR is 16.03, 15.29, and 3.97. For all models, E2A EFA is no lower than E1, with comparable DSR, whereas both metrics drop markedly on E2B. This suggests that timing and remaining duration alone cannot explain the decline, further supporting the impact of history dependence. We will report metrics for E1, E2A, and E2B with confidence intervals in the revised supplementary tables and refine the analysis in §5.4.

W6: Preservation and History Metrics (Appendices E.1, G.1)

The reviewer's comments raise important questions about preservation metrics and CA-HDF computation, particularly the relationship between preservation and instruction fulfillment and the roles of different gates in history-dependence evaluation. We address these concerns through a no-op baseline, success-conditioned preservation analysis, and a decomposition of CA-HDF into second-round edit success and history retention.

Preservation Metrics and No-op Editing

Our preservation metrics aim to measure unintended changes to the original video content during editing. A model may fulfill the requested edit while altering scene content that should remain unchanged. Preservation metrics are therefore necessary for a comprehensive assessment of editing quality.

We agree that the extreme case of no editing deserves separate examination. Following the reviewer's suggestion, we evaluate the unedited source video directly. As shown by the source-continuation check in W2, such outputs achieve relatively high preservation scores, including SC and BC, but rarely fulfill the instructions. Across 3,000 interactions in JoyAI's synthetic subset, the no-op baseline achieves an EFA of only 2.31%, with no case fully satisfying the instruction and only a few atomic requirements satisfied by chance. Thus, no-op editing can yield high preservation scores despite near-zero fulfillment, demonstrating the limitation of preservation metrics when considered alone.

Motivated by this observation, we adopt the reviewer's suggestion to introduce success-conditioned preservation, evaluating non-target content preservation only when the edit is fully successful. Across all models, CF-g and PSR on successful cases are comparable to or slightly higher than their overall scores. For example, JoyAI achieves 72.07% PSR on successful cases versus 68.46% overall. This indicates that high preservation scores do not arise solely from failing to perform edits. We will report per-model success-conditioned preservation results alongside fulfillment and preservation metrics in the revised version.

CA-HDF Computation and Interpretation of Low Scores

CA-HDF uses four binary gates, uu, cc, dd, and hh, all of which must be satisfied. Gate uu double-checks eligibility for the history-dependent set DmD_m; cc requires full second-round instruction fulfillment (EFAi=1\mathrm{EFA}_i=1); dd verifies correct historical-target grounding; and hh assesses history retention. Since full fulfillment necessarily implies correct grounding (c=1⇒d=1c=1\Rightarrow d=1), the four-gate conjunction is equivalent to u∧c∧hu\land c\land h. We retain dd separately to distinguish grounding failures from incomplete edits.

CA-HDF decomposes into the applicability rate, full edit success rate among applicable cases, and history retention conditioned on success. Applicability rates range from 95.41% to 99.65%. StreamEdit and StreamDiffusionV2 have nearly identical edit success rates (25.26% vs. 25.67%) but substantially different conditional history retention rates (80.99% vs. 41.26%), yielding CA-HDF scores of 19.68% and 10.46%. SANA-Streaming shows the opposite limitation: despite 92.66% conditional history retention, its edit success rate is only 19.82%, resulting in 17.91% CA-HDF. These results demonstrate that CA-HDF is not determined solely by E2B fulfillment; history retention also substantially affects its score.

The apparent discrepancy with Table 11 arises from different denominators: CA-HDF uses each model's registered history-dependent candidate set (NmN_m), whereas Table 11 averages EFA over all E2B cases. Within the same candidate set, CA-HDF cannot exceed the full edit success rate, which is itself bounded by mean EFA. We will report per-model NmN_m, four-gate pass rates, conditional history retention, and same-set EFA in the revised appendix to make this relationship explicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.