acceptodds
Under review as a conference paper at ICLR 2027

Sekai2: From World Exploration to Interactive World Modeling

10 views

Abstract

Large-scale exploration videos provide rich observations of the real world, but interactive world modeling demands more than diverse visual coverage: models must preserve scene state over long trajectories, respond consistently to camera motion, and remain coherent when previously observed regions are revisited. Existing video datasets rarely combine minute-scale continuity, camera trajectories, temporally grounded semantics, and explicit revisit structure. Building on the broad real-world exploration coverage of Sekai, we introduce Sekai2, extending the data paradigm from world exploration to interactive world modeling. Rather than simply expanding the original corpus, Sekai2 re-curates selected Sekai footage and combines it with newly collected perspective and panoramic videos under a unified construction and annotation pipeline. Sekai2 contains 128,892 clips totaling 2,826 hours across 113 countries or regions, including 1,453 hours of continuous two-minute sequences. Each released clip is paired with a camera trajectory and hierarchical clip- and segment-level annotations that disentangle subject motion, environmental dynamics, static scene content, and camera behavior, yielding 649,597 temporally grounded segments. To strengthen long-range spatial supervision, we further include 982 panoramic sequences totaling 119 hours that preserve non-linear routes, loops, and revisits. Fine-tuning experiments show consistent improvements across multiple world-model benchmarks, while extensive data-quality evaluations further validate the reliability of Sekai2 for long-horizon and camera-controllable generation. The complete data and annotations will be released at https://anonymous.4open.science/r/Sekai2.

Primary area:
datasets and benchmarks
Keywords:
world models, long-horizon video, camera trajectories, video dataset

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_bQkqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents Sekai2, a large-scale dataset for interactive world model. It combines existing Sekai dataset and newly collected datasets. Each clip is paired with camera trajectory and hierarchical clip- and segment-level captions. The captions are organized as structured json with multiple fields including subject motion, environmental dynamics, static scene content, and camera behavior. The authors conduct experiments by finetuning world models on the proposed dataset, and results show that this finetuning can further improve model performance.

Critical strengths
  1. This paper fills in a real gap in this community. As described in the paper, the community lacks datasets with long videos, fine-grained annotations and camera trajectories. If this presented dataset is released, it can greatly boost the research in this field.

  2. The dataset has a large scale with high diversity. It covers many locations, sources, and formats, and the video length is minute-level.

  3. The authors conduct careful analysis to exhibit the annotation quality. The factorized captions contain orthogonal semantic fields, which can be more informative than a pure natural language caption.

Critical weaknesses
  1. Despite the improved performance, all the finetuning experiments only compare finetuned model against the base model. There is no other controlled experiments like finetuning the base model on other existing datasets using the same computation resources. Besides, several metric improvements are marginal.

  2. The annotation quality is only verified by VLMs. I think a small human study can substantially strengthen the claim of high annotation quality.

  3. This paper lacks ablation study regarding caption format, e.g. whether the structured caption can improve the model performance better than a single long caption / how the segment-level caption can help compared to the clip-level caption.

  4. The paper claims the "revisit-rich panoramic trajectories". However, there is no quantitative analysis to support this claim. Besides, there are no experiments showing that training on this dataset can improve the closed-loop consistency of world model.

Questions for authors

Please consider replying to the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer bQkq (Part I)

We thank the reviewer for the careful assessment and for recognizing Sekai2’s scale, diversity, and combination of long videos, camera trajectories, and fine-grained annotations. We have added a matched-budget dataset comparison, human annotation evaluation, caption ablations, and controlled revisit experiments to address the concerns.

W1: Matched-Budget Comparison of Sekai2 and SpatialVID

We agree that comparison with pretrained models alone does not establish relative dataset utility. We therefore fine-tuned HY-World 1.5 on SpatialVID and Sekai2 with the same architecture, pretrained initialization, and budget. HYWorld-FT-SpatialVID and HYWorld-FT were evaluated on the same 100-case benchmark at 40s and 60s, using identical inputs and generation settings (Section 5.2, Table 3).

Metric ↑SpatialVID 40sSekai2 40sΔ 40sSpatialVID 60sSekai2 60sΔ 60s
WorldScore Camera43.5944.10+0.5141.8942.35+0.46
WorldScore 3D80.2183.32+3.1176.2179.65+3.44
WorldScore Photo.85.6289.83+4.2184.1784.94+0.77
WorldScore Static59.5659.64+0.0861.0362.15+1.12
VBench Subject71.4472.14+0.7066.3867.80+1.42
VBench Imaging61.1362.10+0.9758.1259.22+1.10
VBench Overall18.3019.37+1.0716.0516.29+0.24
iWorld-Bench Traj. Acc.79.2580.04+0.7976.8077.30+0.50
iWorld-Bench Traj. Tol.60.0860.72+0.6458.5158.99+0.48
iWorld-Bench Bright.42.7046.50+3.8038.9040.20+1.30

Sekai2 scores higher on all ten metrics at both horizons. WorldScore 3D improves by 3.11/3.44 points at 40s/60s, and 40s photorealism by 4.21 points. At 60s, Static, Subject, and Imaging improve by 1.12, 1.42, and 1.10 points. Some gains are modest, including 0.08 for 40s Static and 0.24 for 60s Overall. This comparison evaluates the complete training resources, including their annotation differences, and supports dataset utility under the tested budget.

The 120s evaluation in Section 5.2, Table 3 reuses the 60s fine-tuned checkpoints without additional training. HYWorld-FT and AlayaWorld-FT each improve all ten metrics over their respective pretrained baselines, including an 8.96-point camera-control gain for HYWorld-FT.

W2: Quantitative Human Evaluation of Temporal Annotations

Appendix C.6, Table 10 reports an audit of 120 randomly sampled videos: 40 each from inherited Sekai, newly collected, and panoramic footage. Three independent raters watched each complete video with synchronized descriptions and estimated poses, then scored factual correctness, temporal alignment, and salient-event coverage on a 1–5 scale. Videos supplied the evidence; estimated poses were auxiliary context, not independent ground truth. Factual and temporal scores were averaged over segments within each video; coverage was assessed for the whole video. Scores were then averaged across raters and videos, equally weighting the three sources.

Data sourceFactual correctness ↑Temporal alignment ↑Salient-event coverage ↑
Inherited Sekai4.224.384.15
Newly collected videos4.244.364.27
Panoramic videos4.384.694.12
Balanced mean4.284.484.18

All source-level means exceed 4.0. The equal-weight means of 4.28, 4.48, and 4.18 provide human evidence for description quality, complementing the VLM-based checks.

AuthorsOpenReview ↗

Response to Reviewer bQkq (Part II)

W3: Controlled Ablations of Caption Detail and Semantic Organization

Appendix C.7, Table 11 compares Full (full_prompt), Field-Tagged (semantic labels added to the same descriptions, preserving temporal correspondence and camera delimiters), and Short (short_prompt from the same records). All share HY-World 1.5 initialization, training videos, target windows, visual history, camera conditions, optimization, and budget. The text encoder, capacity, and conditioning interface are unchanged; annotation granularity and temporal scope are fixed. Evaluation uses the same 60s, 100-case benchmark, with identical initial images, prompts, trajectories, seeds, and sampling settings.

Training variantCamera ↑3D ↑Photo. ↑Static ↑Subject ↑Imaging ↑Overall ↑Traj. Acc. ↑Traj. Tol. ↑Bright. ↑
HYWorld-FT (Full)42.3579.6584.9462.1567.8059.2216.2977.3058.9940.20
HYWorld-FT-Field-Tagged42.4281.3985.1663.4069.2560.0316.3577.5259.3440.27
HYWorld-Short-FT41.9779.1484.7361.8868.0158.6916.0576.5858.8039.86

Field-Tagged leads on all ten metrics, improving 3D/Static by 1.74/1.25 points and Subject/Imaging by 1.45/0.81 over Full. Full exceeds Short on nine metrics, with gains of 0.51 in 3D, 0.53 in Imaging, 0.24 in Overall, and 0.72 in trajectory accuracy. Subject is the exception: Short scores 68.01 versus Full’s 67.80. These results support the value of semantic labels and descriptive detail in this setting, with effects varying across metrics.

Temporal captions also support selecting and chronologically composing window-specific descriptions with global context across training durations, within existing annotation coverage and without re-annotating each window length. C.6 provides complementary temporal-alignment evidence. C.7 isolates caption detail and organization while holding granularity and temporal scope fixed; it does not isolate segment-level conditioning versus global-only conditioning.

W4: Verified Revisit Coverage and Controlled Evaluation of Loop Consistency

All 982 panoramic sequences, totaling 121.02 hours, were verified to contain at least one observation–departure–return event: sequence-level coverage is 982/982 (100%). Table 1 lists this subset separately; individual frames or short crops need not contain a complete revisit. Section 5.3, Table 4 evaluates learned revisit consistency.

The five variants use Alaya-EVOKE. Base is unfine-tuned; all fine-tuned variants share initialization and budget. Sekai1-FT uses Sekai1; Pano-Local FT uses contiguous panoramic clips; Pano-Revisit FT adds verified earlier observations of the return region. Revisit FT (Unmatched) substitutes non-matching past views while preserving prediction targets, recent context, and conditioning capacity. Appendix D.5 gives implementation and training details.

Models share held-out panoramic initial views, prompts, trajectories, seeds, and sampling settings, with no additional real images after initialization. Controlled four-minute exploration–departure–return rollouts primarily revisit previously generated locations; natural trajectories extend evaluation beyond verified return times. LPIPS, PSNR, and SSIM assess jointly visible static regions of first-visit/return pairs at matched camera settings; MEt3R [1] measures multi-view consistency. L-Rot and L-Trans use poses estimated from generated videos, with L-Trans normalized by the input trajectory’s RMS radius.

Training variantLPIPS ↓MEt3R ↓PSNR (dB) ↑SSIM ↑L-Rot (°) ↓L-Trans (%) ↓
Base0.490.3812.720.244.368.25
Sekai1-FT0.440.3513.150.364.127.91
Pano-Local FT0.350.3016.430.453.827.46
Revisit FT (Unmatched)0.360.2515.210.373.767.58
Pano-Revisit FT0.240.2019.360.563.286.42

Relative to Pano-Local FT, Pano-Revisit FT reduces LPIPS/MEt3R by 0.11/0.10, improves PSNR/SSIM by 2.93 dB/0.11, and lowers L-Rot/L-Trans by 0.54°/1.04 percentage points. Unmatched yields mixed changes relative to Local, while Pano-Revisit FT outperforms both on all six metrics. These controls support the value of relevant distant observations beyond ordinary panoramic fine-tuning. Together with sequence-level verification, they document the availability of revisit supervision and its evaluated utility.

Reference

[1] MEt3R: Measuring Multi-View Consistency in Generated Images.CVPR, 2025.

Dear Reviewer bQkq,

We appreciate your careful review and constructive comments.

With the rebuttal period nearing its end, we wanted to check whether any points in our response would benefit from further explanation. We hope our clarifications have been helpful in addressing your questions, and we would welcome the opportunity to discuss any remaining concerns.

Thank you again for the time and effort you have devoted to reviewing our work.

Best regards,

The Authors

Reviewer_23jpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Sekai2 addresses the lack of real-world training data that jointly supports long-term scene consistency, camera control, and revisits in world models. It combines continuous videos with reconstructed camera trajectories, hierarchical temporal annotations, and panoramic recordings containing loops and revisits. Its main contribution is a 2,826-hour dataset and construction pipeline, with fine-tuning experiments demonstrating improvements in generation quality, geometric consistency, and camera following.

Critical strengths
  1. This paper has well-motivated data design. Sekai2 brings together long, continuous videos, camera trajectories, time-aligned descriptions, and revisits to previously seen locations, providing richer training data for interactive world models. Also, this dataset is diverse and large, containing 2,826 hours videos across 113 countries or regions.

  2. This paper shows evidence of practical utility. Fine-tuning experiments across multiple existing models demonstrate improvements in visual quality, geometric consistency, and camera following.

Critical weaknesses
  1. The experiments show that fine-tuning on Sekai2 helps, but do not establish whether it outperforms other training datasets under the same budget. Ablations are also needed to identify how much long clips, hierarchical captions, and revisit data each contribute.

  2. The 81-frame panoramic tests do not directly show whether a scene stays consistent when revisited. Annotation quality is also checked mostly automatically, so independent pose ground truth and a human evaluation of the temporal descriptions would make the evidence more convincing.

Questions for authors
  1. Do you have comparisons with other datasets under the same fine-tuning budget, or ablations showing how much long clips, hierarchical captions, and revisit data each contribute?

  2. Do the 81-frame panoramic tests include returns to previously seen locations? If so, how do you measure scene consistency at those revisits?

  3. Have you evaluated trajectory accuracy against independent pose ground truth or assessed temporal descriptions through a quantitative human evaluation? If so, what were the results?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 23jp (Part I)

We thank the reviewer for recognizing Sekai2’s motivation and practical utility. Your comments helped us strengthen the controlled comparisons, revisit evaluation, and human assessment of annotations.

W1.1: Matched-Budget Comparison of Sekai2 and SpatialVID

We fine-tuned HY-World 1.5 on SpatialVID and Sekai2 from the same initialization under matched budgets, then evaluated both on the same 100-case benchmark at 40s and 60s with identical inputs and generation settings. Section 5.2 and Table 3 report the results.

Metric ↑SpatialVID 40sSekai2 40sΔ 40sSpatialVID 60sSekai2 60sΔ 60s
WorldScore Camera43.5944.10+0.5141.8942.35+0.46
WorldScore 3D80.2183.32+3.1176.2179.65+3.44
WorldScore Photo.85.6289.83+4.2184.1784.94+0.77
WorldScore Static59.5659.64+0.0861.0362.15+1.12
VBench Subject71.4472.14+0.7066.3867.80+1.42
VBench Imaging61.1362.10+0.9758.1259.22+1.10
VBench Overall18.3019.37+1.0716.0516.29+0.24
iWorld-Bench Traj. Acc.79.2580.04+0.7976.8077.30+0.50
iWorld-Bench Traj. Tol.60.0860.72+0.6458.5158.99+0.48
iWorld-Bench Bright.42.7046.50+3.8038.9040.20+1.30

Sekai2 achieves higher scores on all ten metrics at both horizons, including 3D gains of 3.11/3.44 points and Overall gains of 1.07/0.24 points at 40s/60s. This supports its additional value under matched training resources. Because each dataset uses its accompanying annotations, the comparison evaluates the complete training resource, including both video and annotation differences.

W1.2: Controlled Ablations of Temporal Continuity, Caption Design, and Revisit Supervision

(a) Continuous-source history

Both HY-World 1.5 variants use the same two-minute source videos and supervised target blocks, with matched initialization, recent context, history capacity, training budgets, and inference settings. Short-Clip FT restricts training history to the target’s 15-second segment; Continuous-Source FT allows earlier observations across segment boundaries. Both cover the same source-video content and retain native short-block prediction.

Training variantWorldScore Camera ↑WorldScore 3D ↑WorldScore Static ↑VBench Overall ↑
Same-source Short-Clip FT31.5250.1346.7112.06
Continuous-Source FT38.2153.6449.3614.28
Δ (Continuous − Short)+6.69+3.51+2.65+2.22

Continuous-Source FT improves Camera, 3D, Static, and Overall by 6.69, 3.51, 2.65, and 2.22 points. With source content and supervision matched, these results directly support the additional training value of cross-segment historical relationships.

(b) Caption detail and semantic organization

Appendix C.7 and Table 11 compare three variants with matched videos, targets, initialization, conditioning capacity, and training budgets on the same 60s, 100-case benchmark. Full is the main-paper HYWorld-FT using detailed full_prompt descriptions; Field-Tagged adds four semantic field labels to the same descriptive content; Short uses the corresponding short_prompt at the same annotation level and temporal scope.

Training variantCamera ↑3D ↑Photo. ↑Static ↑Subject ↑Imaging ↑Overall ↑Traj. Acc. ↑Traj. Tol. ↑Bright. ↑
HYWorld-FT (Full)42.3579.6584.9462.1567.8059.2216.2977.3058.9940.20
HYWorld-FT-Field-Tagged42.4281.3985.1663.4069.2560.0316.3577.5259.3440.27
HYWorld-Short-FT41.9779.1484.7361.8868.0158.6916.0576.5858.8039.86

Field-Tagged leads on all ten metrics, improving 3D, Static, Subject, and Imaging over Full by 1.74, 1.25, 1.45, and 0.81 points. Full exceeds Short on nine metrics; Subject is the exception (67.80 vs. 68.01). These results support explicit semantic organization and detailed descriptions.

Temporally indexed captions also support composing text for different training-window lengths. C.7 holds temporal granularity fixed, so it tests detail and organization; the independent benefit of segment-level over global-only conditioning remains unisolated.

AuthorsOpenReview ↗

Response to Reviewer 23jp (Part II)

W1.2: Controlled Ablations of Temporal Continuity, Caption Design, and Revisit Supervision

(c) Revisit supervision

Section 5.3/Table 4 compare five Alaya-EVOKE variants; fine-tuned models share initialization and budgets. Pano-Local uses local panoramic clips; Pano-Revisit adds verified revisit pairs. Unmatched replaces relevant history with non-matching past views while retaining targets, recent context, camera conditions, and history capacity.

Training variantLPIPS ↓MEt3R ↓PSNR (dB) ↑SSIM ↑L-Rot (°) ↓L-Trans (%) ↓
Base0.490.3812.720.244.368.25
Sekai1-FT0.440.3513.150.364.127.91
Pano-Local FT0.350.3016.430.453.827.46
Revisit FT (Unmatched)0.360.2515.210.373.767.58
Pano-Revisit FT0.240.2019.360.563.286.42

Against Pano-Local, Pano-Revisit reduces LPIPS/MEt3R by 0.11/0.10 and loop errors by 0.54°/1.04 percentage points, improving PSNR/SSIM by 2.93 dB/0.11. It leads on all six metrics, whereas Unmatched yields mixed changes over Pano-Local, supporting the value of relevant history. Training controls and evaluation are detailed in Appendix D.5 and W2, respectively.

(d) Complementary longer-rollout evidence

Section 5.2 and Table 3 extend existing 60s fine-tuned checkpoints to 120s at 24 FPS without additional training, using native autoregressive continuation along held-out Sekai2 trajectories. Each base/FT pair shares inputs and sampling settings, with no real intermediate frames supplied.

MethodWorldScore Camera ↑WorldScore 3D ↑WorldScore Photo. ↑WorldScore Static ↑VBench Overall ↑
HY-World 1.531.1249.8457.3645.1811.74
HYWorld-FT40.0854.1662.0249.0714.62
AlayaWorld32.4475.7285.6357.1518.08
AlayaWorld-FT34.7878.0988.3961.3020.25

Both fine-tuned models improve all ten reported metrics, including +8.96 Camera for HYWorld-FT and +4.15 Static for AlayaWorld-FT, sustaining gains at 120s.

For 60s reference consistency, we sample frames at 2 Hz and compute CLIP ViT-B/32 cosine similarity to the input image, reporting the first crossing below 0.6 and similarity at 30s.

ModelReported first crossing below 0.6 (s)Reference similarity at 30s
HY-World 1.531.120.609
HYWorld-FT37.430.642
AlayaWorld39.890.665
AlayaWorld-FT46.050.718

Fine-tuning delays the reported crossing by 6.31s/6.16s and raises 30s similarity by 0.033/0.053 for HY-World 1.5/AlayaWorld. These view-dependent diagnostics complement Table B’s direct evidence for long-range training history.

W2: Dedicated Evaluation of Scene Consistency at Revisited Locations

All 982 panoramic sources contain a verified observation–departure–return event (982/982 sequence-level coverage), but individual 81-frame windows need not contain a return. The original evaluation did not report revisit coverage or consistency; Section 5.3 and Table 4 provide dedicated tests.

Models start from held-out panoramic views and follow identical four-minute exploration–departure–return trajectories, primarily returning to previously generated locations. Natural trajectories extend beyond verified return times. Inputs, seeds, and sampling settings are shared, with no additional real images after initialization.

LPIPS, PSNR, and SSIM compare matched first-visit/return views over jointly visible static regions; MEt3R [1] assesses multi-view consistency. Loop rotation and normalized displacement are estimated from generated-video camera poses. The results in (c) directly assess scene persistence upon return.

W3: Quantitative Human Evaluation and the Scope of Pose Validation

Appendix C.6 and Table 10 evaluate 120 randomly sampled videos, 40 each from inherited Sekai, newly collected, and panoramic footage. Three independent raters use synchronized video/annotation playback to score factual correctness, temporal alignment, and salient-event coverage (1–5). Videos provide the evidence; estimated poses are auxiliary. Factual and temporal scores are averaged within videos, coverage is assessed per video, and the three sources receive equal weight.

Data sourceFactual correctness ↑Temporal alignment ↑Salient-event coverage ↑
Inherited Sekai4.224.384.15
Newly collected videos4.244.364.27
Panoramic videos4.384.694.12
Balanced mean4.284.484.18

All source-level scores exceed 4.0, supporting the quality of the audited descriptions. These ratings assess semantic annotations; an independent ground-truth pose-accuracy benchmark remains unreported.

Reference

[1] MEt3R: Measuring Multi-View Consistency in Generated Images. CVPR, 2025.

Thanks for the authors feedbacks which have resolved my concerns. I would like to keep my score.

Thank you for your careful review and constructive feedback. We are pleased that our responses have addressed your concerns. If you have any further questions or suggestions, we would be happy to discuss them and provide additional clarification.

Reviewer_hFW9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Sekai2 dataset for long-horizon, camera-controllable real-world video generation. It combines re-curated Sekai footage, newly collected perspective videos, and self-captured panoramic videos, totaling 128,892 video units and 2,826 hours. The authors provide estimated camera trajectories and global- and segment-level semantic annotations, comprising 649,597 temporally grounded segments. A 119-hour panoramic subset is intended to provide nonlinear routes, loops, and revisits. The authors assess data quality through geometric and semantic diagnostics, and evaluate training utility by fine-tuning PanoWorld, HY-World 1.5, and AlayaWorld for panoramic and 40/60-second perspective generation.

Critical strengths
  1. The combination of long real-world videos, camera trajectories, and temporal semantics is potentially useful. The supplied perspective captions also show nearly complete structural coverage of both global and segment-level annotations.
  2. Fine-tuning gains across three base models provide initial evidence of practical utility, including improved panoramic generation and 40/60-second perspective generation.
Critical weaknesses
  1. The distinction between exploration and interaction is not clearly established. Long continuity and revisit consistency describe temporal and spatial coherence, but do not by themselves define interaction. Camera-controlled exploration can already be interactive, so the claimed transition requires a clearer account of what capability Sekai2 enables or improves. Camera geometry, temporal hierarchies, and factorized descriptions also have precedents in EPIC Fields¹, Ego4D-HCap², and Koala-36M³. Their integration may be useful, but does not by itself establish the claimed advance toward interactive world modeling.
  2. Table 1 does not adequately distinguish corpus-wide properties from subset coverage. The table reports the full 2,826-hour scale alongside a ✓ for revisits, although the dedicated panoramic subset comprises 119 hours (4.21%) and its verified revisit coverage is unspecified. Since ∼ denotes partial availability, the application of these symbols needs a consistent coverage criterion. Moreover, OmniWorld⁴ is represented by its Game subset without explicitly labeling that scope, whereas Sekai2 reports its full corpus. These presentation choices obscure the effective scale of the claimed capabilities.
  3. The specific benefit of revisit supervision is not directly established. Fine-tuning results support the dataset’s practical utility. However, the 81-frame panoramic evaluation does not clearly report verified revisit coverage or consistency specifically at revisited locations. The appendix reconstruction examples assess the geometry of captured sequences, rather than scene memory learned by the trained models. Consequently, the reported gains do not establish whether revisit supervision improves scene persistence beyond the general benefits of fine-tuning on panoramic videos.

[1] EPIC Fields: Marrying 3D Geometry and Video Understanding. NeurIPS, 2023.

[2] Video ReCap: Recursive Captioning of Hour-Long Videos. CVPR, 2024. See also the official Ego4D-HCap annotation schema.

[3] Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content. arXiv:2410.08260, 2024.

[4] OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling. arXiv:2509.12201, 2025.

Questions for authors
  1. What actual capability distinguishes the paper’s “interactive world modeling” from camera-controlled exploration?
  2. How many of the 982 panoramic sequences contain at least one verified return to a previously observed region?
  3. What existing evidence supports an advantage of long continuous clips over shorter clips from the same videos under comparable training budgets? The evaluations also leave unclear whether two-minute continuity provides additional training value over shorter clips from the same source videos.
3 comments
AuthorsOpenReview ↗

Response to Reviewer hFW9 (Part I)

We thank the reviewer for helping clarify Sekai2’s scope, data coverage, and training evidence.

W1: Scope and Relation to Prior Work

We agree that camera-controlled exploration is already interaction. Sekai2’s evaluated contribution is improved long-term trajectory following and scene persistence within this setting. Models follow predefined camera trajectories from an initial image and prompt, using generated history to recover compatible appearance and geometry upon return. Online adaptive control, object manipulation, and causal responses beyond camera motion are outside our evaluation; annotations describe observed dynamics. Sections 1–2 clarify this scope and positioning.

EPIC Fields [1] combines camera geometry with action and segmentation annotations for egocentric understanding and rendering; Ego4D-HCap / Video ReCap [2] provides temporally grounded, multiscale captions and summaries; and Koala-36M [3] uses structured subject, action, environment, visual-attribute, and camera descriptions to improve text–video correspondence. Sekai2 aligns minute-scale real-world observations, camera geometry, hierarchical temporal semantics, and verified panoramic revisits in one resource. This supports temporally matched targets and relevant earlier observations for history-conditioned training, whose value we test in W3 and W4.

Section 5.2 / Table 3 reports gains over matched-budget SpatialVID fine-tuning on all ten metrics at both 40s and 60s, including 3D gains of 3.11 and 3.44 points. Appendix C.7 / Table 11 shows that adding semantic field labels to the same descriptive content improves 3D and Static by 1.74 and 1.25 points.

W2: Corpus Properties and Panoramic Revisit Coverage

Table 1 now separates Sekai2 (full), with 128,892 clips totaling 2,827.91 hours, from Sekai2 (panoramic subset), with 982 sequences totaling 121.02 hours (4.28% of corpus duration). Their revisit entries are “∼ (pano)” and “✓”, respectively. The caption distinguishes documented revisit coverage from generated consistency. We also label “OmniWorld (Game subset)” [4] to clarify its scope.

All 982 panoramic sequences have been verified to contain at least one observation–departure–return event: 982/982 (100%) sequence-level coverage. This statistic concerns source sequences; individual frames or short crops need not contain a complete revisit.

W3: Added Value of Revisit Supervision

The original 81-frame evaluation and reconstruction examples did not isolate revisit supervision’s training benefit; reconstruction assesses data geometry. The added Section 5.3, “Revisit Consistency,” and Table 4 directly evaluate generated revisits.

All fine-tuned Alaya-EVOKE variants share initialization and budgets. Base is the pretrained checkpoint; Sekai1-FT uses Sekai1; Pano-Local FT uses contiguous local clips from Sekai2 panoramic recordings. Pano-Revisit FT additionally conditions return targets on verified earlier observations of the same region. Revisit FT (Unmatched) preserves those targets, recent context, and conditioning capacity, replacing relevant references with non-matching past views. Training details appear in Appendix D.5.

Models start from held-out panoramic sources and follow identical exploration–departure–return trajectories for four minutes, primarily returning to generated locations. Natural trajectories provide complementary evaluation beyond verified return times. Initial images, prompts, trajectories, seeds, and sampling settings are shared, with no further real images after initialization. LPIPS, PSNR, and SSIM compare first and return visits at matched camera settings over jointly visible static regions. MEt3R [5] measures multi-view consistency; L-Rot and L-Trans use poses estimated from generated videos, with translation normalized by the input trajectory’s RMS radius.

Training variantLPIPS ↓MEt3R ↓PSNR (dB) ↑SSIM ↑L-Rot (°) ↓L-Trans (%) ↓
Base0.490.3812.720.244.368.25
Sekai1-FT0.440.3513.150.364.127.91
Pano-Local FT0.350.3016.430.453.827.46
Revisit FT (Unmatched)0.360.2515.210.373.767.58
Pano-Revisit FT0.240.2019.360.563.286.42

Pano-Local FT improves all six metrics over Sekai1-FT; Pano-Revisit FT is best throughout. Relative to Pano-Local FT, LPIPS and MEt3R decrease by 0.11 and 0.10, PSNR and SSIM increase by 2.93 dB and 0.11, and estimated loop rotation and normalized translation errors fall by 0.54° and 1.04 percentage points. Unmatched produces mixed changes from local training. Its comparison with Pano-Revisit FT holds targets and conditioning capacity fixed, supporting the additional value of relevant distant observations. These controls support the value of relevant history beyond ordinary panoramic fine-tuning.

AuthorsOpenReview ↗

Response to Reviewer hFW9 (Part II)

W4: Training Value of Long Continuous Sources

Our same-source ablation under HY-World 1.5 isolates training-time access to cross-segment history. Both variants use the same two-minute source videos and identical target blocks covering the same temporal locations, including content throughout each recording. Initialization, native prediction-block length, recent context, history capacity, text/camera processing, and optimization/supervision budgets are matched. Same-source Short-Clip FT restricts history to the target’s 15-second segment; Continuous-Source FT allows earlier observations across segment boundaries through the same native history mechanism. All history precedes the target, and fixed initial-image conditioning is shared. Both predict short blocks without full-sequence backpropagation.

Evaluation at 120s matches initial images, prompts, trajectories, seeds, sampling settings, and inference-time history mechanisms. Only the initial image and generated history supply visual conditioning.

Training variantWorldScore Camera ↑WorldScore 3D ↑WorldScore Static ↑VBench Overall ↑
Same-source Short-Clip FT31.5250.1346.7112.06
Continuous-Source FT38.2153.6449.3614.28
Δ (Continuous − Short)+6.69+3.51+2.65+2.22

The gains of 6.69, 3.51, 2.65, and 2.22 points provide direct evidence for cross-segment historical relationships beyond local histories from the same videos, with source coverage, targets, capacity, and budgets fixed. Continuous recordings thus enable training to associate observations farther apart in time while retaining the model’s native short prediction blocks.

We also evaluate persistence after fine-tuning on 60-second Sekai2 videos without architectural changes. In our qualitative comparisons, some base HY-World 1.5 rollouts show drift or increasing noise after approximately 30 seconds; fine-tuned rollouts better preserve subject appearance and scene content. For a quantitative diagnostic, we sample 60-second rollouts at 2 Hz and compute CLIP ViT-B/32 [6] cosine similarity to the input image, reporting the first crossing below 0.6 and similarity at 30s.

ModelReported first crossing below 0.6 (s)Reference similarity at 30s
HY-World 1.531.120.609
HYWorld-FT37.430.642
AlayaWorld39.890.665
AlayaWorld-FT46.050.718

Fine-tuning delays the reported crossing by 6.31s for HY-World 1.5 and 6.16s for AlayaWorld, increasing 30s similarity by 0.033 and 0.053. CLIP is a visual-semantic proxy affected by viewpoint and newly visible content, not an identity metric. These base/FT comparisons provide complementary persistence evidence; Table B isolates the training value of longer-range history.

Section 5.2 / Table 3 further evaluates the existing 60-second fine-tuned HY-World 1.5 and AlayaWorld checkpoints at 120s and 24 FPS without additional training. Models follow held-out Sekai2 trajectories using native prediction blocks and their original history mechanisms. Each base/FT pair shares initial images, prompts, trajectories, seeds, and sampling settings, with no additional real images. Both fine-tuned models improve all ten reported metrics, including +8.96 WorldScore Camera points for HYWorld-FT and +4.15 WorldScore Static points for AlayaWorld-FT, showing that the benefits persist at a longer inference horizon.

References:

[1] From an Image to a Scene: Learning to Imagine the World from a Million 360 Videos. NeurIPS, 2024.

[2] GenEx: Generating an Explorable World. ICLR, 2025.

[3] EvoWorld: Evolving Panoramic World Generation with Explicit 3D Memory. arXiv:2510.01183, 2025.

[4] MEt3R: Measuring Multi-View Consistency in Generated Images. CVPR, 2025.

[5] VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memory. ICCV, 2025.

[6] Learning Transferable Visual Models From Natural Language Supervision. ICML, 2021.

Dear Reviewer hFW9,

Thank you once again for your valuable feedback on our work. We hope our rebuttal has provided useful clarification on the points you raised.

Before the rebuttal period concludes, we would greatly appreciate any further thoughts or questions you may have. Please feel free to let us know if any concerns remain; we would be glad to provide additional explanations.

We sincerely appreciate your time and thoughtful consideration.

Best regards,
The Authors

Reviewer_FQ9JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes Sekai2, a real-world video dataset for camera-posed interactive world modeling under long-horizon. It reannotates Sekai with newly collected Youtube traversal videos. This work has the advantages of validated pose and improved caption quality. But it would be good to have more rigorous studies of revisit and loop closure in the experiments.

Critical strengths
  1. The real-world dataset is always beneficial to the community for both training and testing.

  2. The pose control in the epipolar analysis can inspire the following data curation efforts.

  3. The metadata and paired camera-pose/caption annotations for 127,884 perspective clips are publicly available.

Critical weaknesses

The novelty is overclaimed here. The “first … explicit revisit trajectories” claim in Table 1 needs qualifying. Discussion or comparison are needed. Evidence is as follows: (1) Real panoramic 360 video already exists, and please refer to 360-1M (Matthew, et al. [1]). (2) Loop consistency in panoramic exploration has been measured before (Genex [2] & Evoworld [3]). (3) Loop metrics have been released in the publicly available dataset spatial 360 [3]. Please cite properly.

In general, long generated trajectories that revisit earlier viewpoints should be stressed in this work. Loop error and multi-view consistency metric proposed in MEt3R can be incorporated and compared with. Since this work tackles long-horizon exploration, the prior works on memory should be discussed (e.g., VMem).

[1] Wallingford, Matthew, et al. "From an image to a scene: Learning to imagine the world from a million 360 videos." Advances in Neural Information Processing Systems 37 (2024): 17743-17760. [2] Lu, Taiming, et al. "Genex: Generating an explorable world." International Conference on Learning Representations. Vol. 2025. 2025. [3] Wang, Jiahao, et al. "Evoworld: Evolving panoramic world generation with explicit 3d memory." arXiv preprint arXiv:2510.01183 (2025). [4] Asim, Mohammad, et al. "Met3r: Measuring multi-view consistency in generated images." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025. [5] Li, Runjia, et al. "Vmem: Consistent interactive video scene generation with surfel-indexed view memory." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

Questions for authors

I wonder if you report frame counts or fps for training and testing. From what I read, each model is evaluated at its own training length, and none is tested at longer (such as 120s). Would it be convenient to perform an extrapolation test? What kind of extrapolation method can be used? I suppose one can use (1) sliding-window, or (2) resampled the video.

I’d consider raising my score by strengthening the evaluation of revisit and a discussion of how this paper relates to the prior works

3 comments
AuthorsOpenReview ↗

Response to Reviewer FQ9J (Part I)

We thank the reviewer for the constructive feedback, which helped us clarify Sekai2’s contribution and strengthen the temporal-extrapolation and revisit evaluations.

W1: Related Work and the Data Contribution of Sekai2

We agree that the original “first … with explicit revisit trajectories” claim was too broad. We have revised Sections 1–2 and Table 1 to acknowledge prior work: 360-1M [1] provides real panoramic videos and long-range cross-view correspondences; GenEx [2] evaluates exploration-loop consistency; EvoWorld [3] uses explicit 3D memory and introduces Spatial360, with posed panoramic data and closed-trajectory evaluation; and VMem [5] retrieves historical views through surfel-indexed memory.

Sekai2 contributes a unified construction and annotation pipeline combining minute-scale real-world observations, camera trajectories, hierarchical temporal semantics, and a verified panoramic-revisit subset. It includes 40,512 original two-minute perspective clips and 982 panoramic sequences totaling 121.02 hours. Table 1 now includes 360-1M and Spatial360, separates Sekai2’s full corpus from its panoramic subset, and defines “Revisit” as a documented trajectory property with subset-specific coverage. Our data contribution complements existing memory mechanisms and is evaluated with model architectures held fixed.

The matched-budget SpatialVID comparison (Section 5.2 / Table 3) favors Sekai2 on all ten metrics at both 40s and 60s, including 3D-consistency gains of 3.11 and 3.44 points. This comparison includes each resource’s annotations. The caption ablation (Appendix C.7 / Table 11) isolates semantic organization: adding field labels to the same descriptive content improves 3D and Static by 1.74 and 1.25 points.

W2: Temporal Extrapolation to 120 Seconds without Additional Training

Section 5.1 now distinguishes training context from rollout duration. HYWorld-FT and AlayaWorld-FT use 24 latent temporal positions per training sample: 20 history positions and 4 prediction targets in the memory branch. Nominal 40s and 60s evaluations produce 957 and 1,437 frames at 24 FPS and 832 × 480 resolution.

For the added 120s evaluation (Section 5.2 / Table 3), we reuse the existing 60s fine-tuned checkpoints without additional training. Each model retains its native generation blocks and history mechanism, taking more autoregressive steps along a held-out two-minute Sekai2 camera trajectory at 24 FPS. Each step uses generated visual history and the next trajectory segment. Within each backbone, base and fine-tuned models share the initial image, prompt, trajectory, seed, and sampling settings; no real intermediate images are supplied after initialization.

MethodWorldScore Camera ↑WorldScore 3D ↑WorldScore Photo. ↑WorldScore Static ↑VBench Overall ↑
HY-World 1.531.1249.8457.3645.1811.74
HYWorld-FT40.0854.1662.0249.0714.62
AlayaWorld32.4475.7285.6357.1518.08
AlayaWorld-FT34.7878.0988.3961.3020.25

Both fine-tuned models improve all ten metrics over their respective base models, including +8.96 Camera for HYWorld-FT and +4.15 Static for AlayaWorld-FT, showing that benefits persist beyond the original 60s horizon.

A complementary same-source control fixes two-minute source videos, targets, history capacity, and training budgets, while restricting history to the target’s 15s segment or permitting access across segment boundaries. At 120s, continuous-source history improves Camera and 3D by 6.69 and 3.51 points, supporting the value of preserved long-range relationships.

References:

[1] From an Image to a Scene: Learning to Imagine the World from a Million 360 Videos. NeurIPS, 2024.

[2] GenEx: Generating an Explorable World. ICLR, 2025.

[3] EvoWorld: Evolving Panoramic World Generation with Explicit 3D Memory. arXiv:2510.01183, 2025.

[4] MEt3R: Measuring Multi-View Consistency in Generated Images. CVPR, 2025.

[5]VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memory. ICCV, 2025.

AuthorsOpenReview ↗

Response to Reviewer FQ9J (Part II)

W3: Verified Revisit Coverage and Controlled Evaluation of Revisit Consistency

The new Section 5.3 and Table 4 evaluate revisit appearance, multi-view consistency, and estimated geometric loop errors. All 982 panoramic source sequences contain at least one verified observation–departure–return event (982/982 sequence-level coverage). This coverage applies to the panoramic subset, not every frame or short crop.

Under the shared Alaya-EVOKE backbone, Table 4 compares Base, Sekai1-FT, and panoramic fine-tuning using local clips, relevant earlier observations of return regions, or unmatched past views. All fine-tuned variants share initialization and optimization budgets; the matched/Unmatched pair also shares prediction targets, recent context, and conditioning capacity. Appendix D.5 details the implementation: 141-frame local windows at 24 FPS, nine-latent prediction blocks, and distant revisit references supplied separately through the existing geometric world-state bank.

Controlled four-minute rollouts follow identical exploration–departure–return trajectories from held-out panoramic sources and primarily revisit previously generated locations. Natural trajectories provide complementary evaluation beyond verified return times. All models share initial images, prompts, trajectories, seeds, and sampling settings, with no real images after initialization.

LPIPS, PSNR, and SSIM are computed on jointly visible static regions of first-visit/return pairs at matched camera settings; MEt3R [4] measures multi-view consistency. L-Rot and L-Trans use poses estimated from generated videos. L-Trans normalizes estimated loop displacement by the input trajectory’s RMS radius.

Training variantLPIPS ↓MEt3R ↓PSNR (dB) ↑SSIM ↑L-Rot (°) ↓L-Trans (%) ↓
Base0.490.3812.720.244.368.25
Sekai1-FT0.440.3513.150.364.127.91
Pano-Local FT0.350.3016.430.453.827.46
Revisit FT (Unmatched)0.360.2515.210.373.767.58
Pano-Revisit FT0.240.2019.360.563.286.42

Pano-Local FT outperforms Sekai1-FT on all six metrics; Pano-Revisit FT performs best throughout. Relative to Pano-Local FT, it reduces LPIPS/MEt3R by 0.11/0.10, improves PSNR/SSIM by 2.93 dB/0.11, and reduces estimated loop rotation/translation errors by 0.54°/1.04 percentage points. The Unmatched control yields mixed changes over Pano-Local FT. Together, these controls support the additional value of relevant distant observations for scene consistency under the evaluated backbone.

Dear Reviewer FQ9J,

Thank you again for your thoughtful review and constructive feedback.

As the rebuttal period draws to a close, we would like to briefly follow up on our response. We hope our clarifications have addressed your concerns and would welcome any further feedback you may have.

If any questions or concerns remain, please feel free to let us know. We would be happy to provide additional clarification.

Thank you for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.