acceptodds
Under review as a conference paper at ICLR 2027

WorldEffect: Learning Action-Conditioned Object Dynamics from Human Plans

Abstract

Existing world models primarily predict future observations, but embodied systems must additionally infer how a planned human action changes the physical state of objects in the environment. This problem becomes difficult when nearly identical motions lead to different object responses, as when a hand touches an object or narrowly misses it. Sparse contact can be obscured by shared scene motion, the coordinates that best describe an interaction change with the grasp, and even a specified plan can admit multiple outcomes. We introduce WorldEffect, an action-conditioned object-dynamics framework that predicts object responses from observed history and a supplied human plan. WorldEffect learns which aspects of a planned action induce object changes by comparing responses to plans with the same history. It combines coordinate views that separately assess translation and rotation, then assigns plan-conditioned probabilities to a fixed set of plausible outcomes without seeing the target response. On the BEHAVE dataset, WorldEffect reduces mean object translation error by 16.01% over fixed contact transport, which moves the object according to the planned human motion. For the same 16 candidate trajectories, its input-conditioned ranking reduces probability-weighted joint trajectory error by 12.73% relative to uniform weighting. Together, these results show that explicitly modeling action-induced object responses improves both deterministic prediction and uncertainty-aware future ordering.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Action-Conditioned Object Dynamics, Human-Object Interaction, Multimodal Response Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_p1Q1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper predicts how objects will move given the scene history and human’s future motion. It focuses on cases where similar human motions may lead to different object responses. The method combines paired action supervision, predictions in different coordinate frames, and candidate trajectory ranking. Experiments on recorded interactions, constructed action pairs, and physical simulation show improvements in object prediction.

Critical strengths
  1. The problem is worth studying. The paper predicts the next world states, instead of just predicting the next frames. This focus on human–object interaction, which is helpful to building world models that can support interaction.

  2. Combining interaction geometry with learned motion is a reasonable approach, and the adaptive transport results provide support for it. On BEHAVE, the method improves object prediction over baselines given the same human plan.

Critical weaknesses
  1. The main results rely heavily on knowing human's precise future motion. In practice, it is difficult to provide clean human trajectories across different and diverse scenes, and planned motion may differ from the trained ones. I also worry that recorded human motion already reveals much of the object’s future, allowing the model to take a shortcut. The generated-plan results in the paper show much smaller gains, thus the practical benefit remains unclear.

  2. The paper is hard to follow. There are many modules, assumptions, and experimental setups, making it easy to lose track of the main idea and which results support it. The authors need to signficantly polish the presentation and writting.

Questions for authors
  1. In practice, where would the future human trajectories come from? How do you handle differences between these inputs and the motions used during training, such as measurement noise or errors in position, orientation, and timing? Does the method still outperform matched baselines under these conditions?

  2. What does the paired loss add beyond ordinary state supervision? In which interactions does it clearly help, and when does it make little difference?

Reviewer_7hCeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents an approach to learning world models (action-conditioned object dynamics models) that takes a novel form of input: human plans. The use of plans as inputs is hypothesized to allow the model to learn better predictors, including better probabilistic predictors.

Critical strengths

The idea of using plans to build a world is interesting. This advances world models closer to being true causal models, while still allowing them to be learned from data.

Critical weaknesses

It's not clear how this approach can be applied in practice, when the plans -- which are represented as detail body motions -- are not typically observable. More generally, I'm not sure the paper's notion of "plans" is sufficiently abstract or general to deserve that name. The results show some quantitative advantage over baselines but I'm not sure how much this difference this would make in practice, or how that can be demonstrated or quantified .

The paper's AI statement clearly and forthrightly indicates that AI was used to drive all stages of the research. The human authors' role appears limited to these contributions: "set the research questions, provide domain judgment, review designs and interpretations, and revise the manuscript." In other words, the human authors appear to have asked AI to investigate the effect of plans on learning world models, and this paper is what the AI came up with. The result is a purely academic exercise, conducted in simulation, showing significant effects but of no practical consequence as far as I can tell. And it's not clear how it could have failed, if the authors set out to test the hypothesis and effectively tasked the AI to iterate on designs until it did. Surely there should be some effect of incorporating plans, of some kind, in some way, and this paper seems to have found one. It may be well executed, but it doesn't teach us anything. Or at least, I didn't learn anything from it or see any route to applying it in practice.

More generally, I worry that this is the kind of paper that is too easy to write using AI. It does not set a good example or precedent for what publishable research should contribute.

Questions for authors

Any responses to the above weaknesses would be welcome. I'm not sure it would change my score, but it's possible it would.

Reviewer_mKWSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper predicts an object's response trajectory from an observed history and a supplied human-motion plan. It proposes three mechanisms: (i) paired touch/near-miss supervision with a response-difference loss; (ii) separate translation/rotation weighting of multiple coordinate charts; and (iii) a fixed set of candidate responses with a target-blind ranker. For BEHAVE, it additionally uses temporal trust, which blends learned motion with rigid contact transport (Eq. 8). Trust is introduced in Section 3.3 but is not one of the three mechanisms (L79–84, L136–138). On BEHAVE, the method improves over fixed contact transport in deterministic prediction and over uniform weighting in probability-weighted risk; additional experiments cover several other interaction datasets and a MuJoCo simulation. Temporal trust and the oracle coverage of the candidate cover are well supported. However, held-out data do not show that paired supervision or chart weighting helps on its own, and the learned candidate probabilities improve only over uniform and ridge weighting.

Critical strengths

Temporal trust is a simple, well-isolated mechanism with robust gains on real full-body interactions. Adaptively blending learned motion with rigid contact transport (Eq. 8), instead of the constant weights used by fixed transport (L256), improves all four translation and rotation errors over constant-weight transport on the full BEHAVE test set, under identical supplied plans and with a confidence interval that clearly excludes zero (Table 3 and L406–408; Appendix B.1, L1071–1074). Because the comparison changes only the trust weights, the gain is attributable to this component, and it carries over to camera-space placement (L412–413; Appendix B.3, L1326–1328). This is the paper's best-supported contribution, even though it is not one of the three mechanisms the paper is built around.

Critical weaknesses
  1. Two of the three proposed mechanisms lack independent evidence. The headline paired-supervision gain is measured on the development pairs that were also used to select the loss and the gate (L306–308; Appendix A.1, L848–850); on the larger diagnostic set, removing it changes T-ADE only from 2.273 to 2.260 cm (slightly better without it) and weighted risk from 2.205 to 2.229 (Table 5; L460–461), and in MuJoCo the paired loss shows no clear advantage over ordinary state supervision (L364–368). For chart weighting, the ablation keeps the downstream modules frozen (L458–459); once the cover and router are refit at a matched budget, equal chart weights reach the same diagnostic ADE as the full model (2.273 cm), and only the weighted-risk and ES advantages remain (Appendix B.2, L1161–1163). The framing of "three mechanisms for three uncertainties" therefore currently rests mainly on design rationale, and should either be substantiated on independent data or toned down.
  2. The headline improvements are measured only against internal or weak references. The response-error gain is relative to the paper's own constant-weight transport (Table 3), and the risk reduction is relative to uniform weighting and a ridge regressor (Table 4). Because the risk is normalized by the deterministic K = 1 prediction, that prediction scores 2.000 by construction (Eq. 10; Appendix A.2, L924–932). The learned distribution scores 2.346 (Table 4), and its most probable candidate scores 2.100 (L452–453; Appendix B.3, L1239–1240). Under the paper's own risk metric, simply issuing the deterministic prediction is therefore better. No published human–object interaction predictor is compared as a baseline (HO-GCN is mentioned in the setup, L294, but I could not find it in any results table), although InterDiff (Xu et al., ICCV 2023) addresses a closely related task on BEHAVE and the paper acknowledges the overlap (L129–130). As a result, it is unclear how WorldEffect performs relative to prior work, and the uncertainty-ordering claim in the abstract is not adequately supported.
Questions for authors
  1. (Weakness 1) On TRUMANS scenes not used for any selection and without the pair-conditioned mask, does paired supervision still significantly outperform state-only supervision? And once the cover and router are refit, does chart weighting still improve ADE/FDE on held-out data, beyond its weighted-risk and ES gains?
  2. (Weakness 2) How does WorldEffect compare with a plan-conditioned InterDiff-style predictor under the same BEHAVE protocol, and does the learned distribution outperform simply issuing the deterministic prediction on a proper scoring rule such as ES? If both answers are positive, I would move my rating toward Weak Acceptance; if not, the paired-supervision and chart-weighting claims should be toned down, and the uncertainty-ordering claim should be limited to an improvement over uniform and ridge weighting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.