acceptodds
Withdrawn from ICLR 2027

HumanWorld: From Explorable to Human-centric Interactive World Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

Recent advances in camera-controlled world models allow users to explore and navigate scenes and observe them from different viewpoints. However, viewpoint control alone does not enable users to act within a scene and change its state through interaction. We introduce HumanWorld, a human-centric interactive world model that enables users to explore a scene through camera control and actively change it by controlling the actions of a human character, including manipulating objects through actions such as picking them up and moving them. Our framework disentangles latent interaction dynamics from visual observation generation, with human motion and camera trajectories independently specified in a shared 3D world coordinate system. We learn a latent world interaction module to predict how controlled human motion changes object and scene states without directly synthesizing RGB observations. The interacted features then condition a pretrained video diffusion model through cross-attention to generate the corresponding visual observations. HumanWorld thus unifies viewpoint exploration, human motion control, and action-driven scene evolution within a framework for human-centric world modeling. Experiments across multiple datasets demonstrate that HumanWorld improves visual fidelity, human motion alignment, and human–scene interaction consistency over existing world models with the disentangled modeling.

Primary area:
generative models
Keywords:
World Model, Dense Embodied Control, Human–World Interaction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wPWQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HumanWorld, a human-centric world model that jointly supports camera control, full-body human motion control, and human–scene interaction. The method introduces a latent World Interaction module to model action-induced scene changes and conditions a pretrained video diffusion model on the resulting features. Experiments demonstrate competitive performance on camera control, human animation, and human–object interaction benchmarks.

Critical strengths
  • Meaningful problem. Extending explorable world models to support fine-grained human–scene interactions is an interesting and important direction.
  • Unified framework. The method combines camera control, human motion control, and interaction modeling within a single framework.
  • Promising results. The method achieves competitive performance across multiple benchmarks, with particularly strong results on BEHAVE.
Critical weaknesses
  • Insufficient evaluation of interactive controllability. The HOI experiments rely on ground-truth pose sequences extracted from existing videos. There are no counterfactual experiments showing how different human actions change object states within the same initial scene. It would be great to see results of the same initial scene, different action sequences.
  • Lack of joint camera and interaction evaluation. Camera control and HOI are mainly evaluated separately. The paper does not demonstrate whether the same interaction remains consistent across independently controlled camera trajectories, despite this being a central contribution. Similar as above, it would be great to see results of the same initial scene, same action sequences, with different camera trajectories (I didn't see this in the supplementary video).
  • Limited evidence of interaction correctness. The HOI evaluation relies on PSNR, SSIM, LPIPS, and FVD, which do not directly measure contact consistency, object displacement accuracy, or physical plausibility.
  • Unclear disentanglement of world dynamics and observations. Although Eq. (2) defines camera-independent state transitions, the actual scene representation depends on camera-conditioned features and generated video latents (Eqs. (3)–(5)). It is unclear whether the model learns persistent, viewpoint-independent world states.
  • Unclear 3D human motion representation. The method assumes world-space 3D human poses, while the datasets primarily provide 2D pose annotations. The paper does not clearly explain how globally aligned 3D poses are obtained. Also in the supplementary videos, there are significant flickering of 2D poses, with some limbs suddenly disappearing and reappearing -- not sure where that comes from given that this is supposed to be "groundtruth".
  • Limited ablation studies. The paper only ablates the entire World Interaction module, without isolating the contributions of camera modulation, individual interaction components, or the proposed training losses.
  • Incomplete implementation details. Appendix Table 6 explicitly contains an unfinished placeholder, while referenced architectural configurations are missing. The output resolution is unspecified, and a sentence in Appendix C is incomplete.
Questions for authors
  • Counterfactual action control. Can the authors demonstrate generating different human–object interactions from the same initial image using different input pose sequences? This would help establish whether the model can produce action-dependent scene changes beyond reconstructing existing interactions.
  • Joint camera and interaction control. Can the authors provide examples or quantitative results showing the same human–object interaction under different independently specified camera trajectories? How consistent are the resulting object states and contact relationships across views?
  • Interaction correctness. Can the authors evaluate human–object interaction accuracy using more direct metrics, such as contact consistency, object trajectory error, or penetration rate? The current PSNR, SSIM, LPIPS, and FVD metrics do not directly measure interaction correctness.
  • World-state disentanglement. How does the implementation maintain a persistent, viewpoint-independent world state? In particular, Eqs. (3)–(5) construct the scene representation using camera-conditioned features and generated video latents, without an explicit state transition from st−1s_{t-1}. How does this formulation realize the camera-independent dynamics described in Eq. (2)?
  • 3D pose representation. How are world-space 3D human poses obtained from datasets primarily annotated with 2D keypoints? How are these poses aligned with the camera trajectories, and what 3D information is retained when the poses are projected into 2D DWPose sketches for conditioning?
Reviewer_U4PmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces HumanWorld, a human-centric interactive world model that enables independent control of camera trajectories and full-body human motion while modeling human-scene interactions. Its central idea is to disentangle action-driven world-state evolution from viewpoint-dependent observation generation. The proposed framework uses a latent World interaction module to predict scene dynamics from human motion and conditions a pretrained video DiT through control adapters and interaction branch. Experiments on camera control, human animation, and human-scene interaction benchmarks demonstrate improvements in video quality and provide an ablation of the World Interaction module.

Critical strengths
  1. Well-motivated problem formulation
  2. Structured approach to combining human motion and camera control.
  3. Broad evaluation across multiple generation tasks.
Critical weaknesses

1. The evaluation does not directly measure the correctness of human-scene interactions. The central contribution seems to be action-driven scene evolution, where pre-defined human motion should produce corresponding changes in objects and relations to the human. However, Tab.4 primarily reports only PSNR, SSIM, LPIPS and FVD, which only assess visual similarity and video quality but do not directly establish whether the generated human genuinely follows the predefined pose, maintains physically consistent contact, or produces the intended object-state changes. Therefore, pose adherence, contact consistency, and object-state transitions should be evaluated to claim the improvements of interaction modeling.

2. The claimed disentanglement of world evolution and viewpoint observation is insufficiently validated. The paper's main conceptual contribution seems to be the separation of action-driven world-state transitions from viewpoint-dependent observations. However, the paper lacks a controlled evaluation where human motion and camera trajectories vary independently within the same scene. Such an experiment could be important for determining whether the same action produces consistent object-state changes under different viewpoints and whether changing the action while keeping the viewpoint fixed results in the appropriate scene evolution. The relationship between these camera-aware representations and the claimed viewpoint-independent world dynamics needs further clarification.

3. Insufficient ablation studies to justify the architectural complexity and training objectives. Tab. 4 demonstrates that the World Interaction module improves overall generation performance. The proposed framework contains several additional components, inclluding control adapters, camera modulation, human-aware- and action-scene- blocks, and an interaction branch, however, those individual contributions are not sufficiently evaluated.

Similarly, the paper introduces the response loss and paired-camera loss, but does not provide loss-wise ablations. Without these ablations, it remains unclear which components contribute to motion fidelity, interaction consistency, and disentangled control.

4. The necessity of latent world-state modeling over direct conditioning remains unclear. The proposed latent World Interaction module is motivated as a mechanism for predicting object-state changes rather than directly generating observations. However, the evidence does not sufficiently establish whether an intermediate latent world state is necessary or whether comparable improvements could be achieved by directly conditioning a video generator on human poses and camera trajectories. Although Tab. 4 removes the World Interaction module, the resulting improvements are evaluated using general video quality metrics rather than direct measures of state consistency. This makes it difficult to distinguish genuine improvements in world-state modeling from additional feature processing or model capacity. A stronger comparison with direct pose-and-camera conditioning, preferably under matched model capacity and evaluated on interaction-specific metrics, would better justify the proposed formulation.

5. The interaction scope and generalization capabilities are insufficiently characterized. The paper presents HumanWorld as a general framework for human-centric interactive world modeling, but its demonstrated interactions primarily concern a single human manipulating objects. The evaluation interaction categories and their complexity are not clearly summarized. in particular, it is unclear whether the method generalizes to unseen interactions or support more complex settings involving multiple interacting humans. Clarifying whether the framework is restricted to single-human-object/single-human-scene interactions would help establish its scope and limitations.

6. Limited comparative evidence for simultaneous camera and interaction control. The paper includes quantitative comparisons across several benchmarks and qualitative interaction comparisons. However, the evaluations do not sufficiently isolate the claimed advantage of combining independently controlled camera trajectories and human-scene interactions. In particular, the qualitative comparisons primarily demonstrate human–object interactions, rather than challenging scenarios where the camera moves dynamically while the controlled human manipulates objects and the resulting state changes must remain consistent. Additional side-by-side video comparisons and quantitative evaluations in these combined-control settings would strengthen the empirical claims.

7. Limited analysis of query-based interaction representations. The World Interaction module relies on learnable queries to construct action-aware and scene-aware states. However, the paper provides little analysis of whether these queries effectively capture relevant human motions, interacting objects, and their relationships. It remains unclear what information the queries attend to and how the resulting representations encode interaction-induced state changes. Without attention visualizations, query-level analysis, or controlled interventions, it is difficult to assess whether the learned representations genuinely capture meaningful interaction dynamics rather than generic features.

8. Insufficient qualitative comparisons with existing methods. More extensive qualitative comparisons should be included.

Questions for authors

Q1. Could the authors provide quantitative evaluations of pose adherence, human–object contact consistency, and action-induced object-state changes? These metrics would help establish whether the generated interactions faithfully follow the prescribed human motions and produce the intended scene changes beyond visual fidelity.

Q2. Could the authors provide controlled experiments that independently vary human motions and camera trajectories within the same scene? In particular, does the same action produce consistent object-state changes under different viewpoints, and do different actions produce appropriate scene changes under a fixed viewpoint? How is the claimed viewpoint-independent world dynamics maintained when the scene representation is camera-conditioned?

Q3. Could the authors provide additional ablations isolating the contributions of the control adapters, camera modulation, Human-Aware and Action–Scene blocks, and interaction branch? Additionally, how do the response loss and paired-camera loss individually affect motion fidelity, interaction consistency, and viewpoint controllability?

Q4. What advantages does the intermediate latent World Interaction module provide over directly conditioning a video generator on human poses and camera trajectories? Could the authors compare against a parameter-matched direct-conditioning baseline using interaction-specific metrics to distinguish improvements in world-state modeling from increased model capacity?

Q5. What interaction categories are included in the training and evaluation datasets, and how well does the model generalize to unseen interactions? Furthermore, can the framework support human–human interactions involving multiple independently controlled humans, or is it currently limited to single-human–object interactions?

Q6. Could the authors provide quantitative and qualitative comparisons with existing methods under simultaneous camera and human-motion control? In particular, challenging scenarios involving dynamic camera trajectories and object manipulation would help demonstrate whether the proposed method preserves interaction outcomes under changing viewpoints.

Q7. Could the authors analyze whether the learnable queries in the Human-Aware and Action–Scene blocks capture interaction-relevant information? Attention visualizations, query-level interventions, or query-number ablations would help clarify what these representations encode and whether they effectively capture human motions, interacting objects, and their state changes.

Reviewer_uRkUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HumanWorld, a human-centric interactive world model that combines independent camera control, full-body human motion control, and human-scene interaction. Its key design separates latent interaction modeling from video generation, using a World Interaction module to predict scene dynamics and a pretrained Wan video model to render observations. Experiments on SpatialVID, Fashion, and BEHAVE demonstrate competitive camera control, human animation, and improved interaction video quality.

Critical strengths
  • Meaningful problem formulation: Combining independent camera control, full-body motion, and human-object interaction addresses an important limitation of existing explorable world models.

  • Encouraging experimental results: HumanWorld achieves strong performance on BEHAVE, and the World Interaction module ablation substantially improves FVD, supporting the effectiveness of interaction-specific conditioning.

Critical weaknesses
  • Limited novelty in the disentanglement formulation: Dexterous World Models (Kim et al. CVPR 2026) already separates action-induced world dynamics from camera-dependent observations. While HumanWorld introduces a dedicated latent interaction module and extends control to full-body motion, the underlying disentanglement is conceptually similar. The incremental contribution beyond DWM remains insufficiently established.

  • Insufficient evidence for disentangled world dynamics: Although the paper conceptually separates action-driven state transitions from viewpoint-dependent observations, the implemented interaction features remain camera-conditioned (Eqs. 3–5). Moreover, the current experiments evaluate camera control and interaction quality largely separately, without directly establishing cross-view consistency under identical actions. Thus, it remains unclear whether the model learns genuinely disentangled world dynamics or simply benefits from joint conditioning.

Questions for authors
  • Compared with DWM's navigation-manipulation disentanglement, what is the fundamental advantage of explicitly modeling latent interaction dynamics in HumanWorld? What capabilities does this formulation enable beyond differences in control interfaces?

  • Given that the interaction state depends on camera-conditioned features (Eqs. 3–5), how does the model ensure that the same interaction produces consistent scene changes across different camera trajectories?

  • Can the authors provide quantitative measurements of human-object contact consistency, object displacement accuracy, and cross-view interaction consistency to substantiate the central claims?

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. We provide additional quantitative evaluation and controlled qualitative examples to address the concerns about interaction accuracy, joint control, and the role of the World Interaction module.

Quantitative interaction evaluation.

We supplement the visual-quality metrics with object-mask mIoU and CenterL1 on BEHAVE. We use SAM2 to segment objects in generated videos and measure mask overlap and centroid L1 error relative to the target. These metrics directly evaluate the spatial accuracy of the interacting objects.

MethodmIoU ↑\uparrowCenterL1 ↓\downarrow
Wan-Move0.25030.1853
SCAIL-20.24650.3194
Fun-Motion0.39530.1006
HumanWorld w/o World Interaction0.41260.0882
HumanWorld0.45490.0694

HumanWorld achieves the highest mIoU and lowest CenterL1 among the evaluated methods. Removing the World Interaction module reduces mIoU from 0.4549 to 0.4126 and increases CenterL1 from 0.0694 to 0.0882. This ablation supports the module's contribution to object localization and spatial alignment during interaction. These measurements complement the visual-quality evaluation; contact consistency and penetration require separate metrics.

Counterfactual actions and joint camera–interaction control.

Figure 1 presents controlled examples sharing the same initial image. With the camera trajectory held fixed, changing the prescribed action from pushing forward to pulling back produces corresponding changes in object motion. With the action held fixed, introducing a rotating camera preserves the illustrated interaction outcome across viewpoints. These examples provide qualitative evidence of action-dependent scene evolution under independently specified camera trajectories. They support behavioral consistency across views, while persistent viewpoint-independent latent states require further validation.

Role of the interaction representation.

Figure 2 visualizes interaction-derived attention on ARCTIC sequences using a fixed-noise forward pass through the middle interaction injector. Attention follows the manipulated object and active interaction region as the action progresses, suggesting that the learned representation captures interaction-relevant information. Together with the mIoU and CenterL1 ablation above, this analysis supports the World Interaction module's role in translating prescribed human motion into corresponding scene responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.