W1. The benchmark is not self-contained for new models. E2B — a third of all events and every CA-HDF case — is built only after each model generates B, via VLM proposals plus human review of that model's output (B.3.3, F.2). A future user cannot evaluate a new model without repeating human annotation. E2B scores across models also come from different instructions, targets, timings and possibly types.
Please release, and validate against the human-reviewed version, a fully automatic E2B protocol; or redesign round 2 around fixed instructions with conditional applicability.
At minimum, report per-model E2B type, subtype and timing distributions, and |D_m|. The paper says |D_m| is reported per model, but it does not appear.
Fig. 2 and Tables 5–6 give planned counts; realized counts after quality control are not reported. Fig. 2(e)'s E2B timing is necessarily model-specific — whose B is it?
W2. The task contract differs across paradigms, yet results form one leaderboard. Real-time editors receive the continuing source stream (F.2), full-context models see the whole sequence, and streaming-native generators work causally. Editing a known continuation and generating an unseen future are different tasks.
Preservation, SC and BC use the second before t_e as reference (D.1, E.1). For an editor, the source's natural evolution after t_e (a process advancing, people moving) should be reproduced. It is either penalized or judged against an undefined target.
Please state per paradigm what each model sees at each time and what V⁺ should contain for non-target content. Consider separate leaderboards by information condition.
W3. The responsiveness metrics do not measure real-time responsiveness as framed.
SOL, SCL and DSR are in media time, independent of wall clock. Wan2.1-VACE has a TTFF of ~200 s but an SOL of 4.19 s, in line with the others. For an offline model, when the edit appears in the clip is a design choice, not interactivity.
D.2 defines TTFF as the first-output wait, but F.2 says frame-wise systems (JoyAI, StreamDiffusionV2) report per-frame runtime while others report an 81-frame-equivalent cost. If Table 2 follows F.2, the column mixes scales. Please state which definition Table 2 uses.
All timing runs on H20 GPUs, far below the compute these "real-time" systems target. This likely explains why "streaming-native" StreamEdit (71 s TTFF) is slower than SANA-Streaming (14 s) and LiveEdit (26 s), undermining the paradigm reading in §5.2. Report throughput (fps) under one definition.
DSR applies one 5 s completion deadline to every instruction. Many instructions (Temporal Progression; "let the cat jump off the table") cannot plausibly finish in 5 s, and a model that snaps to the target state beats one that renders a plausible transition. The annotation already records expected onset and completion (B.3.2); normalize DSR by it or stratify by expected duration.
With most interactions failing, capped SCL is largely a monotone function of the failure rate and duplicates EFA and DSR. Report success rate and completion time among successes separately, as the audit already does (0.37 s MAE on 34 jointly completed cases).
W4. Several claims in §5 are contradicted by the paper's own tables.
§5.2 says SANA leads CF-g and PSR "with Wan2.1-VACE following closely," suggesting full-context modeling helps preservation. In Table 2, JoyAI is second on both (CF-g 3.48 vs. Wan 3.40; PSR 68.46 vs. 66.21). The other full-context model, Lucy Edit 1.1, is worst on both (1.78; 18.83).
§5.3, "Temporal achieves the highest EFA across all content contexts": in Table 7's Object column, Camera (39.17) and Style (32.04) exceed Temporal (29.70).
§5.3, "Scene consistently underperforms across most interaction types": Scene is the lowest context in only 3 of 7 rows (Composition, Attribute, Spatial).
§5.3, "Interaction registers the lowest EFA across all models": not for SANA (Addition 6.19) or Wan2.1-VACE (Addition 5.46) in Table 8.
§5.4 and §6 claim pervasive round-2 degradation in video quality, but in Table 11 NRQ rises from E1 to E2B for six of seven models (all except JoyAI).
The abstract says "four metrics across two capability dimensions"; there are sixteen metrics in four groups.
W5. The round-wise analysis omits the control the design already provides. §5.4 attributes the E1→E2B drop (EFA 27.6→16.0, DSR 16.0→4.0) to accumulated history, while conceding the comparison is confounded by different instructions, later timing and shorter remaining output. E2A — a later, second-round intervention on the unedited A — is exactly the control that separates history dependence from "second instruction, later in the clip." It is never reported.
If Table 2 weights the three event types equally, the implied E2A EFA is at or above E1 for every model. Examples: JoyAI ≈ 45.1 (E1 45.1, E2B 23.6); LiveEdit ≈ 37.9 (36.6, 22.1); Wan2.1-VACE ≈ 27.2 (23.8, 15.7).
That would make the history-dependence claim much stronger than presented. Please report E1, E2A and E2B side by side for all metrics, with CIs.
W6. Preservation and history metrics are confounded with doing nothing and with E2B success.
High PSR, CF-g, SC and BC are achievable by not editing; SANA has the top PSR with low EFA. Add a no-op baseline (pass-through source for editors; frozen frame or unconditioned continuation for generators) and a joint metric such as preservation conditional on fulfillment.
CA-HDF requires E2B completion (gate c). With low E2B completion, low CA-HDF mostly reflects E2B fulfillment, not history retention. Report each gate (u, c, d, h) per model, and h given c.
As reported, CA-HDF exceeds the same model's mean E2B EFA in Table 11 for StreamEdit (19.68 vs. 13.57), SANA (17.91 vs. 11.21) and JoyAI (39.72 vs. 23.59). Complete success requires EFA_i = 1, so mean EFA upper-bounds the complete-success rate, and CA-HDF requires c. This holds only if dependency-applicable E2B cases are far easier than E2B overall, or if c is not full completion. Please clarify.
W7. The judge audit samples a different operating regime from the benchmark. Audit mean EFA is 57.9% (human) and 62.9% (VLM); benchmark-wide EFA is 16–38% (mean 24.3%). The audit oversamples successes, while the benchmark lives in the failure regime, where the judge's main error — leniency (+5.0 pp EFA; 6 false vs. 2 missed passes on DSR) — matters most.
With ~14 cases per model, the audit cannot detect model-dependent bias, which is what rankings depend on.
The claim that judge error is small relative to model differences fails for many pairs. OSC spans only 0.176–0.186; five models' IS-g lie within 0.23 of each other against an audit MAE of 0.27; StreamEdit, StreamDiffusionV2 and Wan2.1-VACE EFA lie within 1.4 pp.
Please describe how audit cases were drawn within strata, add CIs or significance tests to Table 2, and report human-vs-VLM model-rank agreement on a larger per-model sample. Release cached judgments and prompts, since the benchmark depends on a proprietary, versioned judge.
W8. Model configuration details are missing. Appendix F promises per-model settings but does not give resolution, frame rate, chunk length, how instructions are injected (a prompt switch for StreamDiffusionV2 and LongLive?), or how offline models produce a 15 s post-t_e window. Per-model output durations T_i matter, since shorter outputs lower the SCL cap and SEO denominators (the audit includes 4 s and 8 s windows). OSC is nearly constant across models, and its common-window intersection is undefined for E2B, where t_e is model-specific.
Response to Reviewer 3U3N Comments
W1: Human Dependence and Automation Feasibility (Appendices B.3.3, B.4)
The reviewer's constructive feedback has helped us better identify opportunities to improve the automation and reliability of WorldLineBench's multi-round construction. In the submitted benchmark, GPT-5.6 Sol generated three ranked E2B candidates from each model's actual first-round output, specifying interaction instructions and times, followed by human selection and quality control. Unlike E1 and E2A, E2B instructions depend on each model's generated history and cannot be fully predetermined. Among 1,000 JoyAI synthetic cases, reviewers accepted the initial Top-1 candidate in 963 cases (96.3%), while fewer than 5% required correction for other models.
For the remaining 37 cases, we tested automatic candidate reconstruction. An independent GPT-5.6 Sol Agent received the video, assigned interaction type, and previously unsuitable candidates, then regenerated three ranked candidates rather than merely selecting from the original proposals. Human quality checks confirmed 33 valid reconstructions (89.2% recovery), yielding 996/1,000 (99.6%) usable cases across the initial and reconstructed proposals in this diagnostic. The remaining four primarily involved blurred videos, ambiguous targets, or insufficiently persistent states. These results demonstrate that most initially unsuitable candidates can be recovered through independent reconstruction.
To remove per-case human selection, we further developed a confidence-based automatic routing protocol. Initial candidates with a Top-1–Top-2 confidence margin above 0.4 are accepted directly; otherwise, an independent Agent regenerates the instruction and interaction time, and its Top-1 candidate is adopted. These results suggest the potential for reliable automation with reduced human involvement. The threshold is informed by the observed confidence patterns in human-reviewed cases, while unified prompts and decision rules standardize candidate selection across models and reduce reviewer-dependent variation. We will provide the construction protocol, validation statistics, residual failure cases, code, and prompts in the revised appendix.
W2: Evaluation Costs and a Lightweight Benchmark
We have carefully considered the reviewer's suggestions and further examined the costs and accessibility of WorldLineBench. WorldLineBench provides 2,000 source videos and E1/E2A instructions, requiring 6,000 interactions per evaluated model. Generation cost varies by model: on H20-equivalent hardware, JoyAI requires approximately 45 GPU hours, StreamDiffusionV2 72, StreamEdit 525, and Wan2.1-VACE 1,312, with other models detailed in the appendix. Model-specific E2B instruction construction using GPT-5.6 Sol costs approximately 104.8 USD per model with input caching, with lower-cost alternatives offering potential savings. Non-VLM metrics require approximately 34 GPU hours per model, while Gemini 3.8 Flash judging costs about 133.8 USD, bringing combined API costs to approximately 238.6 USD per model.
To further reduce costs, we evaluated Qwen3.8-27B as an open-source alternative judge. Across nine VLM-related metrics, its pairwise rankings agree with Gemini’s in 177/189 comparisons (93.65%), with the same top-ranked model on all nine metrics. Judging 6,000 interactions requires approximately 90 H20 GPU hours, suggesting its potential for lower-cost preliminary evaluations using local computation. For final reporting, we recommend the standardized Gemini protocol to preserve comparability.
We also investigated whether smaller subsets retain comparable conclusions. Using source videos as the sampling unit, we examined nine subset sizes from 50 to 1,000 videos under two strategies: stratified random and interaction-type-balanced sampling. Both stratify by video source (real or synthetic) and five content categories; the latter additionally balances interaction types and subtypes. Each selected video retains E1, E2A, and E2B interactions. Sampling uses no model scores and excludes no failure cases. We repeated each configuration 30 times to assess metric values and model rankings against the full benchmark.
Small subsets preserve broad trends but cannot reliably reproduce full results. Under type-balanced sampling, 200 videos yield an average Spearman correlation of 0.948, yet only 5/30 repeats meet predefined error tolerances across all ten core metrics and seven models. At 400 videos, correlation rises to 0.978, pairwise ranking agreement reaches 97.2%, and 27/30 repeats meet all tolerances. Increasing to 600 videos improves stability but incurs higher costs with diminishing returns. We therefore adopt 400 videos (1,200 interactions per model) as WorldLineBench-Light, reducing generation and judging workloads by approximately 80%, while retaining the full benchmark for formal evaluation. Per-model costs, subset comparisons, and sampling protocols will be provided in the revised appendix.