acceptodds
Under review as a conference paper at ICLR 2027

Holi-World: Building Holistic World Intelligence from Human-Ego Experience

Abstract

Scaling spatial intelligence requires object-centric supervision that remains valid not only across viewpoints, but also as the physical world changes over time. Existing 3D data pipelines largely assume static scenes: they either depend on costly human-annotated scans or collapse a video into a single scene, discarding object motion. Conversely, applying a 3D detector independently to each frame produces noisy boxes, fragmented identities, and inconsistent trajectories. We present SPAFoundry, a unified and scalable pipeline for open-vocabulary static and dynamic 3D grounding from posed visual sequences. SPAFoundry uses Boxer as a shared 2D-to-3D proposal generator and introduces complementary scene-level consolidation paths. The static path aggregates repeated observations in a common world frame to refine object geometry and suppress duplicates, while the dynamic path preserves time, associates moving instances across frames, and recovers temporally consistent 3D boxes and trajectories. The resulting interface supports heterogeneous image sequences with optional sparse or dense geometric cues and exports a common annotation format for static objects and dynamic tracks. On ScanNet, SPAFoundry improves AP from 0.32 to 0.68 over Boxer. On 20 Aria Digital Twin sequences, it achieves 30.88% dynamic 3D box IoU, 54.31% HOTA, and a TP-L2 error of 0.0333 m. These results show that explicitly modeling static persistence and dynamic motion turns robust 2D-to-3D lifting into an effective data engine for multimodal spatial intelligence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D Grounding, Data Pipeline, Spatial Intelligence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zPdiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents HoliWorld, an integrated framework for automatically constructing structured 3D/4D annotations from egocentric videos. By combining static scene reconstruction, dynamic object tracking, and interaction understanding, the authors construct a dataset for spatial-temporal reasoning and demonstrate its effectiveness through downstream VLM training.

Critical strengths
  1. The paper presents a comprehensive and practically valuable framework for automatically constructing structured spatial-temporal annotations, integrating static geometry, dynamic object states, and interaction semantics.
  2. The experiments are extensive and solid, demonstrating substantial improvements in annotation quality across multiple benchmarks and validating the effectiveness of the generated data for downstream spatial-temporal reasoning.
Critical weaknesses
  1. Limited novelty. The overall data construction pipeline can be viewed as an incremental extension of Holi-Spatial, primarily incorporating existing dynamic tracking techniques and temporal supervision. The methodological novelty beyond integrating existing components is relatively limited.
  2. Limited validation of world modeling capabilities. Despite the paper's emphasis on holistic world intelligence, the downstream evaluation primarily focuses on VLM-based spatial-temporal reasoning. The potential value of the constructed data for world modeling, particularly future state prediction or dynamics understanding, remains unexplored.
Questions for authors

Could the authors provide additional experiments demonstrating the utility of HoliWorld-Data for world modeling tasks, such as future state prediction or action-conditioned dynamics prediction? Such evaluations would help substantiate the broader claim of holistic world intelligence beyond VLM-based spatial-temporal reasoning.

2 comments
AuthorsOpenReview ↗

Response to Reviewer zPdi

We sincerely thank the reviewer for recognizing HoliWorld’s practical value and the strength of our experimental evaluation. Below, we address the comments on methodological novelty and the potential utility of HoliWorld-Data for world modeling.

W1: Creativity Compared to Holi-Spatial. For static perception, HoliWorld incorporates updated perception components and, more importantly, a more effective VLM-guided verification and object consolidation mechanism, aiming to improve annotation reliability in complex egocentric environments.

More substantially, HoliWorld introduces dynamic 4D perception, extending beyond static scene reconstruction to recover time-varying object states, persistent identities, 3D trajectories, and interaction semantics. These extensions together enable the construction of unified spatial-temporal world records and corresponding QA supervision, which are beyond the scope of the original static-focused Holi-Spatial framework.

Q1 / W2: Scope of Current Evaluation Focused on Spatial Intelligence. Thank you for this valuable suggestion. We agree that future state prediction and action-conditioned dynamics modeling represent promising applications of HoliWorld-Data.

Our current evaluation focuses primarily on spatial-temporal reasoning, as the central goal of this work is to establish a scalable pipeline for grounding spatial-temporal understanding in real egocentric experiences and constructing high-quality, structured 3D/4D supervision. Rather than directly developing a predictive world model, we focus on building the reliable world representations and large-scale supervision that could serve as a foundation for broader world intelligence tasks.

Nevertheless, HoliWorld-Data contains rich spatial-temporal and interaction information that could be further exploited for learning object dynamics and future state prediction. We appreciate the reviewer's suggestion and are particularly interested in exploring this promising extension in future work.

Thank you for the clarifications. I acknowledge the practical value of the proposed framework and the solid experimental results. However, my main concern regarding the limited methodological novelty relative to Holi-Spatial remains.

For now, I will maintain my current rating and revisit it during the Reviewer–AC discussion period.

Reviewer_AiCEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HoliWorld , an automated system for converting egocentric videos into structured 3D/4D world-understanding data. It builds HoliWorld-Foundry , a pipeline that reconstructs scene geometry, detects static 3D objects, tracks dynamically manipulated objects, and links them with interaction semantics in a unified world representation. Based on this pipeline, the authors construct HoliWorld-Data with around 100 scenes, 444 ego-videos, 1.3M frames, 2,466 static 3D boxes, 2,095 dynamic tracks, and about 80K spatial/temporal QA pairs, and train HoliWorld-VL-4B for spatial and temporal reasoning. Experiments show strong gains in both annotation quality and downstream reasoning, with 0.68 AP25 on ScanNet, 54.31 HOTA on ADT20, and large Token F1 improvements on HoliWorld-Bench and EgoDynamic4D, demonstrating the value of automated 3D/4D supervision from human ego experience.

Critical strengths
  1. The paper addresses an important problem: how to automatically build 3D/4D world-understanding data from human egocentric experience, a topic highly relevant to spatial intelligence, embodied AI, and AR/VR scene understanding.

  2. The system design is complete and reasonable. It connects egocentric data capture, scene reconstruction, static object detection, dynamic object tracking, QA generation, model training, and benchmark evaluation into a coherent end-to-end pipeline.

Critical weaknesses
  1. The analysis of the generated data quality is not sufficiently thorough. The paper mainly validates the data through detection/tracking metrics and downstream model performance, which is indirect, especially when using the final trained model’s improvement as evidence for data quality. Moreover, some annotation metrics are still far from perfect, such as 0.24 AP25 for static 3D detection and 43.14 HOTA for dynamic tracking on HoliWorld-Bench, suggesting that the generated data may contain many errors. However, the paper does not analyze how such annotation errors propagate into the generated QA pairs and captions, or how robust the trained model is to these noisy annotations.

  2. The methodological novelty is relatively limited. The work is more of a system and data-construction effort, and its pipeline relies heavily on existing components such as SLAM, depth estimation, 3DGS, SAM, and VLMs. While the overall system is useful, the paper would benefit from a clearer articulation of what is technically new beyond integrating existing components into a unified pipeline.

Questions for authors

Could the authors discuss how well the proposed pipeline scales to longer egocentric videos? A major expectation for ego-centric data is to capture longer-term human experience, where dynamic object tracking, long-term identity association, occlusion, re-appearance, pose drift, and mask drift may introduce larger errors. Clarifying the applicable temporal range of the current method, and whether it can be extended to longer recordings, would help better assess the practical scope and impact of the proposed system.

2 comments
AuthorsOpenReview ↗

Response to Reviewer AiCE

We sincerely thank the reviewer for recognizing the practical value of HoliWorld and for providing constructive feedback. We provide our responses and clarifications below.

W1: Data Quality, Error Propagation, and Generalization.

  • Annotation quality and benchmark difficulty. Our pipeline substantially improves annotation accuracy over the strongest baselines on HoliWorld-Bench, achieving 0.24 vs. 0.16 AP25 for static detection and 43.14 vs. 15.13 HOTA for dynamic tracking. Moreover, the consistent performance drops across methods from existing benchmarks (e.g., ScanNet and ADT) to HoliWorld-Bench reflect the greater challenges posed by realistic human-centric environments, including occlusions, noise, and dynamic interactions.

  • Quality control and error propagation. HoliWorld-Bench uses human-annotated ground truth for static and dynamic tasks. For generated language annotations, captions are produced before subsequent identification and tracking, so errors introduced in these later stages do not propagate backward into them. We also manually spot-check the generated captions to verify the correctness of object labels and the plausibility of action descriptions.

    For downstream QA pairs, we use a staged inspection, correction, and filtering process. We first manually inspect a sample of generated QA pairs and categorize the observed errors to identify the question types most affected by upstream annotation noise. For these types, we collect a larger set of human-corrected examples and use them as demonstrations to guide LLM-based correction of the corresponding QA pairs at scale. We then manually inspect another sample of the corrected data to assess whether the recurring errors have been resolved, and filter out QA pairs whose grounding or answers remain unreliable.

  • Generalization beyond our benchmark. On the independent EgoDynamic4D dataset, our model achieves 66.39 F1, 99.67 Y/N, and 94.92 MCQ, compared with 42.93, 61.07, and 79.16 for the strongest baselines, respectively. These results support generalization beyond HoliWorld-Bench.

W2: Creativity and Contribution. Thank you very much for recognizing the value of our comprehensive framework for scalable egocentric 3D/4D understanding.

HoliWorld provides a complete solution from raw egocentric video data to coherent 3D world records, including explicit 3D bounding-box tracks that most baselines do not provide. For example, EgoSeg3D mainly refines instance-mask associations using 3D cues, whereas Whareformer performs long-term identity association and maintains object-position trajectories. Neither produces 3D bounding boxes, as reflected by the missing results in dynamic tracking tables of the paper.

Meanwhile, building such records requires more than independently combining existing modules. The central technical challenge lies in maintaining spatial-temporal consistency and persistent object identities across observations, dynamic interactions, and video segments. To address this, HoliWorld introduces several task-specific algorithmic designs, including static and dynamic object merging, cross-branch information reuse and association, and cross-segment object identity propagation. These designs consolidate fragmented observations into persistent 3D object records, forming the main technical contribution of our pipeline.

Q1: Scalability to Longer Egocentric Videos. Our pipeline has no predefined upper limit on video duration by design, as it adopts a segment-based processing strategy rather than requiring an entire recording to be processed within a single temporal window. Long videos are divided into manageable segments, which are first processed individually, followed by cross-segment merging and identity association to construct more temporally consistent world records.

For static objects, our pipeline consolidates observations across time into a unified representation of the static scene, followed by a global review to improve consistency. For dynamic objects, the pipeline maintains identity associations across temporal segments throughout processing. These design choices make extending the pipeline to longer recordings practically feasible. In our current experiments, we are capable of processing videos approximately 5–10 minutes long, demonstrating the practical applicability of the pipeline beyond short video clips. Further validation at day- or week-long scales requires suitable long-duration egocentric data. A long-video task demo is available here. https://anonymous.4open.science/r/review-long/index.html

Thank you for the detailed responses. I appreciate the engineering effort and task-specific optimizations. However, limited methodological/paradigmatic novelty and insufficient stage-wise accuracy and error-propagation analysis prevent me from raising my rating to 4. Such analysis would help identify quality bottlenecks and establish the reliability of the final QA/caption supervision. Overall, I still find the work practically valuable and maintain my rating of 3.

For future development, could the authors discuss extending the framework to day- or week-long recordings, particularly cross-session alignment, scene changes, long-term object identity, and accumulated tracking errors? This is a forward-looking discussion for insight, not a request for additional experiments, and does not affect my rating.

Reviewer_uDFJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Holi‑World, an end‑to‑end ecosystem for building world‑level intelligence from egocentric first‑person recordings. It includes HoliWorld‑Foundry for automated static‑dynamic 3D perception, HoliWorld‑Data for constructing spatial‑temporal QA supervision, and HoliWorld‑Bench for human‑validated evaluation. Experiments show strong gains on 3D detection, dynamic tracking, and downstream vision‑language spatial reasoning over multiple benchmarks.

Critical strengths

This work delivers a complete pipeline covering capture, automatic annotation, dataset construction, model training and benchmark evaluation, rather than only a single component. It addresses the high‑cost manual‑annotation bottleneck for egocentric 4D scene understanding.

Solid empirical results: the proposed Foundry achieves clear improvements in static 3D object AP and dynamic HOTA tracking metrics against competitive prior baselines. Downstream fine‑tuning yields large Token‑F1 improvements on spatial‑temporal QA tasks. Comprehensive ablation studies validate the contributions of dual‑branch detection, VLM verification, cross‑segment identity association and mask propagation for static and dynamic perception respectively.

The paper clearly defines dataset statistics, evaluation protocols and reproducibility statements.

Critical weaknesses

One key built‑in assumption limits real‑world applicability: the dynamic perception module assumes at most one moving object per timestamp. This does not reflect crowded real‑life egocentric scenes with multiple concurrent moving agents.

Most baseline trackers in dynamic evaluation receive ground‑truth‑initialized masks/categories at test time, while Holi‑World does not. Though this demonstrates its practical advantage, direct head‑to‑head comparison fairness is partially compromised.

Limited real‑world noise robustness analysis. The pipeline relies heavily on quality SLAM poses and depth estimation; the paper lacks systematic stress‑test under noisy IMU, drift poses or heavy depth artifacts.

HoliWorld‑Bench is built on self‑collected in‑house scenes. Larger public multi‑scene human‑annotated test sets could further verify generalization of the automatically generated world records.

No public code or dataset release is promised in the manuscript, which hurts reproducibility for the community.

Questions for authors

How would your dynamic tracking pipeline degrade when multiple independent moving objects appear simultaneously? Could you briefly outline your envisioned extension to support multiple moving instances?

When SLAM pose drift or depth estimation failure occurs, which sub‑module breaks first in your Foundry pipeline? Have you observed common failure modes on real captured sequences?

Will you release HoliWorld‑Data, HoliWorld‑Bench and core implementation of HoliWorld‑Foundry upon paper acceptance?

2 comments
AuthorsOpenReview ↗

Response to Reviewer uDFJ

We sincerely thank the reviewer for recognizing the completeness of our end-to-end ecosystem, the strong empirical results, and the clearly defined evaluation protocols. We address the three questions and five weaknesses below, supported by additional analyses and experiments.

Q1 / W1: Concurrent multi-object tracking performance. We would like to clarify that the single-object assumption applies to each individual tracking instance, rather than imposing a recording-level or timestamp-level limit to a single dynamic object. Our pipeline is capable of maintaining multiple moving-object trajectories, including cases involving simultaneous object manipulation. Representative multi-object tracking clips are available at the anonymous website https://anonymous.4open.science/r/review-multi/index.html.

To quantify performance under concurrent motion, we divided the 20 ADT evaluation sequences into nine sequences with concurrent object motion and eleven sequences without concurrent object motion, retaining all GT objects and predictions under the original full-video evaluation protocol. [email protected] reaches 55.06 for concurrent-motion sequences versus 53.69 for the others, providing evidence of the method’s robustness in sequences containing concurrent object motion.

Q2 / W3: Robustness to imperfect depth and camera poses.

HoliWorld-Foundry incorporates depth cleanup, outlier rejection, and minimum-support checks before 3D box fitting, while RGB-derived semantics and masks provide complementary evidence to mitigate geometric imperfections. Specifically, depth errors primarily affect 3D lifting and box fitting, while pose drift influences world-coordinate alignment in static fusion and dynamic tracking. To further validate robustness, we conducted controlled perturbation experiments on 20 ScanNet scenes and 20 ADT sequences, introducing 30% blockwise observation removal and progressive camera-pose drift up to 5 cm / 1°:

PerturbationScanNet AP25(%) ↑ScanNet AP50(%) ↑ADT [email protected] ↑
Original results685554.31
30% observation removal645153.67
Pose drift (5 cm / 1°)675451.68

The results demonstrate strong robustness to the tested geometric perturbations. Even with 30% missing observations, AP25/AP50 decrease by only 4 points, while HOTA drops by just 0.64 points. Under progressive pose drift, AP25/AP50 decrease by only 1 point, and HOTA by 2.63 points. These results highlight Foundry's ability to maintain reliable static and dynamic perception under imperfect depth and camera poses, supported by geometric filtering and complementary RGB evidence.

Q3 / W5: Release Plan. We plan to publicly release HoliWorld-Data and HoliWorld-Bench upon acceptance, subject to applicable privacy and data-sharing requirements, to facilitate future research on egocentric spatial-temporal understanding. Meanwhile, although we do not plan to release the full implementation details of HoliWorld-Foundry, we intend to open-source selected code modules, hoping they will benefit the community in addressing similar challenges.

W2: Baseline inputs and comparison fairness. HoliWorld is evaluated as a fully automated pipeline, whereas several baselines require ground-truth categories or initialization masks under their native protocols. This input asymmetry therefore favors the dynamic tracking baselines. To quantify its effect, we additionally provide HoliWorld with GT category labels and masks when each dynamic object starts to move: mIoU and HOTA improve by 5.69 and 4.25 points, respectively. This confirms the benefit of GT initialization as it bypasses the need to automatically detect when each object starts moving and determine its initial location, while both metrics remain substantially higher than those of the baselines even without GT inputs.

W4: Generalization and benchmark design. Our evaluation includes established public benchmarks as well as newly captured, human-validated scenes. HoliWorld-Foundry is evaluated on ScanNet and ScanNet++ for static 3D perception and ADT for dynamic perception. Thus, the perception results do not depend solely on our in-house benchmark. HoliWorld-Bench complements these public evaluations with newly collected first-person scenes for spatial-temporal reasoning assessment.

AuthorsOpenReview ↗

Kind follow-up on rebuttal

Dear Reviewer uDFJ,

With the discussion deadline approaching, we would be grateful for any feedback you may have on our rebuttal at your convenience. In response to your comments, we clarified the pipeline’s support for concurrent multi-object motion and provided additional quantitative results, conducted controlled robustness experiments under observation removal and camera-pose drift, analyzed the effect of ground-truth initialization on baseline comparisons, and clarified our evaluation scope and release plan.

We would particularly appreciate knowing whether these additional results and clarifications address your main concerns, or whether any issues remain unresolved, so that we have an opportunity to address them before the discussion closes.

Thank you very much again for your time and thoughtful consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.