- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents Sekai2, a large-scale dataset for interactive world model. It combines existing Sekai dataset and newly collected datasets. Each clip is paired with camera trajectory and hierarchical clip- and segment-level captions. The captions are organized as structured json with multiple fields including subject motion, environmental dynamics, static scene content, and camera behavior. The authors conduct experiments by finetuning world models on the proposed dataset, and results show that this finetuning can further improve model performance.
Critical strengths
-
This paper fills in a real gap in this community. As described in the paper, the community lacks datasets with long videos, fine-grained annotations and camera trajectories. If this presented dataset is released, it can greatly boost the research in this field.
-
The dataset has a large scale with high diversity. It covers many locations, sources, and formats, and the video length is minute-level.
-
The authors conduct careful analysis to exhibit the annotation quality. The factorized captions contain orthogonal semantic fields, which can be more informative than a pure natural language caption.
Critical weaknesses
-
Despite the improved performance, all the finetuning experiments only compare finetuned model against the base model. There is no other controlled experiments like finetuning the base model on other existing datasets using the same computation resources. Besides, several metric improvements are marginal.
-
The annotation quality is only verified by VLMs. I think a small human study can substantially strengthen the claim of high annotation quality.
-
This paper lacks ablation study regarding caption format, e.g. whether the structured caption can improve the model performance better than a single long caption / how the segment-level caption can help compared to the clip-level caption.
-
The paper claims the "revisit-rich panoramic trajectories". However, there is no quantitative analysis to support this claim. Besides, there are no experiments showing that training on this dataset can improve the closed-loop consistency of world model.
Questions for authors
Please consider replying to the weaknesses.
3 comments
Response to Reviewer bQkq (Part II)
W3: Controlled Ablations of Caption Detail and Semantic Organization
Appendix C.7, Table 11 compares Full (full_prompt), Field-Tagged (semantic labels added to the same descriptions, preserving temporal correspondence and camera delimiters), and Short (short_prompt from the same records). All share HY-World 1.5 initialization, training videos, target windows, visual history, camera conditions, optimization, and budget. The text encoder, capacity, and conditioning interface are unchanged; annotation granularity and temporal scope are fixed. Evaluation uses the same 60s, 100-case benchmark, with identical initial images, prompts, trajectories, seeds, and sampling settings.
| Training variant | Camera ↑ | 3D ↑ | Photo. ↑ | Static ↑ | Subject ↑ | Imaging ↑ | Overall ↑ | Traj. Acc. ↑ | Traj. Tol. ↑ | Bright. ↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| HYWorld-FT (Full) | 42.35 | 79.65 | 84.94 | 62.15 | 67.80 | 59.22 | 16.29 | 77.30 | 58.99 | 40.20 |
| HYWorld-FT-Field-Tagged | 42.42 | 81.39 | 85.16 | 63.40 | 69.25 | 60.03 | 16.35 | 77.52 | 59.34 | 40.27 |
| HYWorld-Short-FT | 41.97 | 79.14 | 84.73 | 61.88 | 68.01 | 58.69 | 16.05 | 76.58 | 58.80 | 39.86 |
Field-Tagged leads on all ten metrics, improving 3D/Static by 1.74/1.25 points and Subject/Imaging by 1.45/0.81 over Full. Full exceeds Short on nine metrics, with gains of 0.51 in 3D, 0.53 in Imaging, 0.24 in Overall, and 0.72 in trajectory accuracy. Subject is the exception: Short scores 68.01 versus Full’s 67.80. These results support the value of semantic labels and descriptive detail in this setting, with effects varying across metrics.
Temporal captions also support selecting and chronologically composing window-specific descriptions with global context across training durations, within existing annotation coverage and without re-annotating each window length. C.6 provides complementary temporal-alignment evidence. C.7 isolates caption detail and organization while holding granularity and temporal scope fixed; it does not isolate segment-level conditioning versus global-only conditioning.
W4: Verified Revisit Coverage and Controlled Evaluation of Loop Consistency
All 982 panoramic sequences, totaling 121.02 hours, were verified to contain at least one observation–departure–return event: sequence-level coverage is 982/982 (100%). Table 1 lists this subset separately; individual frames or short crops need not contain a complete revisit. Section 5.3, Table 4 evaluates learned revisit consistency.
The five variants use Alaya-EVOKE. Base is unfine-tuned; all fine-tuned variants share initialization and budget. Sekai1-FT uses Sekai1; Pano-Local FT uses contiguous panoramic clips; Pano-Revisit FT adds verified earlier observations of the return region. Revisit FT (Unmatched) substitutes non-matching past views while preserving prediction targets, recent context, and conditioning capacity. Appendix D.5 gives implementation and training details.
Models share held-out panoramic initial views, prompts, trajectories, seeds, and sampling settings, with no additional real images after initialization. Controlled four-minute exploration–departure–return rollouts primarily revisit previously generated locations; natural trajectories extend evaluation beyond verified return times. LPIPS, PSNR, and SSIM assess jointly visible static regions of first-visit/return pairs at matched camera settings; MEt3R [1] measures multi-view consistency. L-Rot and L-Trans use poses estimated from generated videos, with L-Trans normalized by the input trajectory’s RMS radius.
| Training variant | LPIPS ↓ | MEt3R ↓ | PSNR (dB) ↑ | SSIM ↑ | L-Rot (°) ↓ | L-Trans (%) ↓ |
|---|---|---|---|---|---|---|
| Base | 0.49 | 0.38 | 12.72 | 0.24 | 4.36 | 8.25 |
| Sekai1-FT | 0.44 | 0.35 | 13.15 | 0.36 | 4.12 | 7.91 |
| Pano-Local FT | 0.35 | 0.30 | 16.43 | 0.45 | 3.82 | 7.46 |
| Revisit FT (Unmatched) | 0.36 | 0.25 | 15.21 | 0.37 | 3.76 | 7.58 |
| Pano-Revisit FT | 0.24 | 0.20 | 19.36 | 0.56 | 3.28 | 6.42 |
Relative to Pano-Local FT, Pano-Revisit FT reduces LPIPS/MEt3R by 0.11/0.10, improves PSNR/SSIM by 2.93 dB/0.11, and lowers L-Rot/L-Trans by 0.54°/1.04 percentage points. Unmatched yields mixed changes relative to Local, while Pano-Revisit FT outperforms both on all six metrics. These controls support the value of relevant distant observations beyond ordinary panoramic fine-tuning. Together with sequence-level verification, they document the availability of revisit supervision and its evaluated utility.
Reference
[1] MEt3R: Measuring Multi-View Consistency in Generated Images.CVPR, 2025.
Dear Reviewer bQkq,
We appreciate your careful review and constructive comments.
With the rebuttal period nearing its end, we wanted to check whether any points in our response would benefit from further explanation. We hope our clarifications have been helpful in addressing your questions, and we would welcome the opportunity to discuss any remaining concerns.
Thank you again for the time and effort you have devoted to reviewing our work.
Best regards,
The Authors
Response to Reviewer bQkq (Part I)
We thank the reviewer for the careful assessment and for recognizing Sekai2’s scale, diversity, and combination of long videos, camera trajectories, and fine-grained annotations. We have added a matched-budget dataset comparison, human annotation evaluation, caption ablations, and controlled revisit experiments to address the concerns.
W1: Matched-Budget Comparison of Sekai2 and SpatialVID
We agree that comparison with pretrained models alone does not establish relative dataset utility. We therefore fine-tuned HY-World 1.5 on SpatialVID and Sekai2 with the same architecture, pretrained initialization, and budget. HYWorld-FT-SpatialVID and HYWorld-FT were evaluated on the same 100-case benchmark at 40s and 60s, using identical inputs and generation settings (Section 5.2, Table 3).
Sekai2 scores higher on all ten metrics at both horizons. WorldScore 3D improves by 3.11/3.44 points at 40s/60s, and 40s photorealism by 4.21 points. At 60s, Static, Subject, and Imaging improve by 1.12, 1.42, and 1.10 points. Some gains are modest, including 0.08 for 40s Static and 0.24 for 60s Overall. This comparison evaluates the complete training resources, including their annotation differences, and supports dataset utility under the tested budget.
The 120s evaluation in Section 5.2, Table 3 reuses the 60s fine-tuned checkpoints without additional training. HYWorld-FT and AlayaWorld-FT each improve all ten metrics over their respective pretrained baselines, including an 8.96-point camera-control gain for HYWorld-FT.
W2: Quantitative Human Evaluation of Temporal Annotations
Appendix C.6, Table 10 reports an audit of 120 randomly sampled videos: 40 each from inherited Sekai, newly collected, and panoramic footage. Three independent raters watched each complete video with synchronized descriptions and estimated poses, then scored factual correctness, temporal alignment, and salient-event coverage on a 1–5 scale. Videos supplied the evidence; estimated poses were auxiliary context, not independent ground truth. Factual and temporal scores were averaged over segments within each video; coverage was assessed for the whole video. Scores were then averaged across raters and videos, equally weighting the three sources.
All source-level means exceed 4.0. The equal-weight means of 4.28, 4.48, and 4.18 provide human evidence for description quality, complementing the VLM-based checks.