- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes EditWorld, which controls autoregressive video world models using streaming edit instructions and reference images, along with a synthetic edit data pipeline and a new benchmark WBench-Editing. Experiments show that it significantly leads in edit scores while retaining some general world modeling capabilities.
Critical strengths
The editing score reaches 80.0 versus 54.8 for the next best model, and the model seems to keep the surrounding scene intact across multiple edits. It can also take a reference image in the middle of interaction rather than only at the first frame. This is not just one attention tweak. The synthetic global and local editing data, chunk level edit states, gated causal attention, sparse context, joint AR and BI training, self resampling, and 4 step distillation all work together.
Critical weaknesses
WBench Editing is new, has only about 150 cases, and editing quality is mostly judged by VLM QA. How do they make sure those cases do not overlap with training data, and do the VLM scores agree with human judgments? The main table is text only, and the reference image comparison uses fewer models and fewer cases. Does that really show flexible referencing, or just a small favorable subset? On the data side, how many samples survive the two stage VLM filtering, what is the annotation error rate, and does the reference leakage rewrite actually work? I would also like quantitative ablations for sparse context, reference gating, and short term textual context, not just qualitative examples. And since most baselines do not support reference images, how fair is that comparison? Finally, long horizon error accumulation and failure cases should be shown more openly. Otherwise the headline score feels a bit too clean.
Questions for authors
-
How do the authors ensure that WBench-Editing does not overlap with the training data? Since the editing metrics primarily rely on VLM-based evaluation, is there any validation showing that these scores align with human judgments?
-
The overall improvement is impressive, but how much of it comes from the synthesized editing data versus the proposed architectural changes? A controlled comparison using the same training data would help clarify this.
-
The ability to introduce reference images during generation is interesting, but the current evidence is largely qualitative. Could the authors provide a quantitative comparison between mid-interaction reference insertion and initial-only reference conditioning?
-
Can more qualitative analysis based on human preference be used?
-
Does EditWorld include extreme multi-turn testing, e.g., 10 sequential edits?
7 comments
Thank you for the detailed rebuttal and additional experiments. The new ablations, controlled comparisons, and human evaluation substantially address my main concerns.
However, longer editing sequences and quantitative evaluation of flexible reference insertion remain insufficiently explored.
Overall, I appreciate the improvements and maintain my original rating of 3 (Weak Accept).
Thank you for your positive feedback and for acknowledging the improvements in our revised manuscript. We sincerely appreciate your constructive suggestions and continued consideration of our work.
We are currently conducting additional experiments to address the remaining weaknesses and questions. We will share the results as soon as they become available.
Thank you for your encouraging assessment and constructive feedback. We have revised the manuscript to include additional experiments and evaluation details. All newly added or revised content is highlighted in blue in the updated PDF.
Critical Weakness. Quantitative Component Ablations (Sec. 4.3, Table 4, pp. 9–10).
We additionally provide quantitative ablations for Sparse Context, reference visibility gating, and short-term textual context. Removing reference visibility gating and short-term textual context decreases Editing by 0.2 and 4.6 points, respectively. For Sparse Context, Top-k retrieval improves Editing by 0.2 and 0.1 points over FIFO and Random, respectively, and Consistency by 0.5 points over both at the same retrieval budget (k=2). These results provide quantitative evidence beyond the qualitative comparisons in the original submission.
Q1. Benchmark Reliability and VLM–Human Alignment (Appendices A–C, pp. 15–18).
We provide detailed WBench-Editing statistics, covering 148 cases and 323 editing events (Appendix A, Table 5). We also conduct a data overlap analysis, identifying 0 exact source matches and 12 potential visual near-duplicates, none of which are confirmed as genuine overlaps after manual verification. Additionally, we clarify the VLM judge, evaluation procedure, metric definitions, and score aggregation in Appendix B (Table 6).
To validate the VLM-based evaluation, we conduct a human preference study (Figure 4) and measure VLM–human alignment (Table 7). The pairwise agreement rates are 73.3%, 66.7%, and 77.5%, with corresponding Spearman correlations of 0.58, 0.45, and 0.64, across Instruction Following, Content Preservation, and Editing Quality, respectively.
Q2. Data vs. Architectural Contributions (Sec. 4.3, Table 4, pp. 9–10).
To disentangle the contributions of editing-specific data and architectural design, we introduce a same-data backbone baseline trained with the same synthesized editing data and training curriculum but using standard causal conditioning. EditWorld achieves an Editing score of 80.0, compared with 66.9 for this baseline, providing evidence for an additional 13.1-point improvement from the full architectural design.
Q4. Human Preference Evaluation (Appendix C, Figure 4, p. 17).
We conduct a user study involving 30 participants and 20 WBench-Editing cases, comparing EditWorld with YUME 1.5, LingBot-World 2.0, and DreamX-World. Participants rank the four methods across Instruction Following, Content Preservation, and Editing Quality. EditWorld achieves the best mean rank across all three dimensions, providing complementary human evidence for the editing improvements.