acceptodds
Under review as a conference paper at ICLR 2027

Law, State, and Body: Object-Centric World Models for Counterfactual Reasoning

Abstract

Most learned world models compress a scene into a single flat vector and roll it forward with a recurrent network or a transformer. This works well for prediction, but it hides the objects that a physical scene is actually made of, so the model has no natural way to answer a question such as "what would happen if this object moved differently." We present LSB, a world model that keeps physical law, the visible state of a scene, and the body that acts on it as three separate, swappable pieces. State is a set of per-object slots produced by slot attention. Law is a self-attention transformer that updates every slot using only the slots themselves, so the same law applies regardless of which actuator produced the action. Body is a small embodiment-specific encoder that turns raw actions and proprioception into a control signal that Law can consume. Because objects are explicit and individually addressable, we can edit one object's velocity and roll the edited state forward through the same law used for ordinary prediction, giving a direct and cheap way to test counterfactual reasoning. Across a three-embodiment, six-physics-regime benchmark, LSB beats a strong non-slotted baseline by large, statistically robust margins on next-state prediction, on future-contact queries, and on physical intervention, and the advantage survives zero-shot transfer to six unseen physics regimes, to embodiments never trained with intervention, to object counts far outside the training range, and to rollouts sixteen times longer than the training horizon. We further show that the same advantage appears when Law is instantiated as a completely different architecture, a relational graph network in the style of interaction networks, and when Law is trained with a contrastive energy objective instead of regression, which indicates that the gain comes from the explicit object-centric design rather than from any one choice of network or loss.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
object-centric world models, slot attention, counterfactual reasoning, relational dynamics, physics simulation, model-based reinforcement learning, compositional generalization, interaction networks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.60 over 5 reviews, above 10% of the others.
Reviewer_ryc9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes LSB, an object-centric world model that separates object state, shared dynamics, and embodiment-specific action encoding. Explicit object states enable prediction, collision queries, and counterfactual rollouts through direct velocity edits. Experiments in a synthetic 2D physics environment evaluate transfer across embodiments, physics regimes, object counts, and rollout horizons, with reported advantages over flat-latent baselines.

Critical strengths
  1. The paper addresses an important limitation of predictive world models: supporting targeted queries and object-level interventions, rather than only forecasting future states.

  2. The evaluation considers multiple embodiments, physics regimes, object counts, and rollout horizons. Alternative dynamics architectures and training objectives also help explore the sensitivity of the proposed design.

Critical weaknesses

There are two critical implementation errors in the submitted code, that affect the claim in the paper:

  1. CLS+v’s evaluation interface discards the intervention velocity. In lsb/eval/cls_wrap.py:46, CLSWM.predict calls step_emb(emb, action, body) without passing state.v. Consequently, lsb/model/cls_baseline.py:65–66, substitutes zero velocities. This differs from the training interface, which correctly passes v=state.v at Line 108–109. Thus, editing velocity cannot affect CLS+v’s predictions through the supplied evaluation path, invalidating the central claim that the baseline receives identical intervention information.

  2. Object matching incorrectly conflates predicted-slot and ground-truth-object indices. In lsb/model/state.py:127,139:143, hungarian_match uses a single used set for both index spaces. Accepting slot 0 → object 1 therefore blocks the valid assignment slot 1 → object 0. In a minimal test, merely permuting geometrically perfect predictions changes the reported error from 0 to 0.75. This matcher is used in both training supervision (lsb/train/losses.py:8) and evaluation (lsb/eval/general.py:15–21), so the problem potentially affects both learned representations and reported comparisons.

Questions for authors

N/A

Reviewer_2uFtOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies object-centric world models for counterfactual reasoning. The proposed LSB model separates object State, a shared dynamics Law, and embodiment-specific Body encoders. By representing position and velocity explicitly, the model allows velocity edits before predicting future trajectories. The authors evaluate the method in a 2D simulator on trajectory prediction, contact queries, and intervention outcomes, and further study changes in physics, object count, rollout length, and embodiment. The main results show benefits on some query metrics, although the evidence for the broader counterfactual and generalization claims is less convincing.

Critical strengths
  1. The explicit representation gives a straightforward way to intervene on an object's velocity. Wtih the inclusion of CLS+v, it can receive the edited velocity and is therefore a more meaningful baseline than CLS alone.
  2. The paper reports several useful negative results and trade-offs. The relational Law variants perform better on contact-pair identification, while the transformer performs better on E1. The appendix also reports no significant benefit from separate Body encoders in one comparison and a freeze-Law transfer result that did not replicate.
Critical weaknesses
  1. I am concerned about what E2 actually measures. According to Appendix A.1, it compares the model's final-position error after an edit with the error of leaving the object at its initial position. This stationary reference is not the trajectory without the intervention. A positive E2 could therefore come from simple motion extrapolation, especially since the model already includes Euler drift and a velocity-follow loss. Without a constant-velocity baseline and paired predictions with and without the edit, I do not think this metric alone establishes accurate counterfactual reasoning. The near-zero E2 values on unseen physics also need to be interpreted with this reference in mind.
Questions for authors
  1. How does LSB compare with constant-velocity prediction on edited trajectories, particularly those involving collisions? Please report absolute trajectory errors and paired predictions with and without the intervention, in addition to E2.
  2. How does CLS+v perform on the generalization tests? How are checkpoints selected? The strong early CLS+v result in Table 7 suggests that checkpoint selection could affect the comparison.
Reviewer_bbSaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LSB, an object-centric world model that separates object states, physical dynamics, and embodiment-specific controls. It supports prediction, querying, and velocity interventions through a shared dynamics model. Experiments evaluate these capabilities across different embodiments, physics regimes, object counts, and rollout horizons.

Critical strengths
  1. The explicit object-centric representation enables direct state interventions and counterfactual rollouts using the same dynamics model.

  2. The CLS+v baseline provides a more meaningful comparison by giving the flat-latent model access to the edited velocity information.

  3. The alternative Law architectures, multi-seed evaluations, and generalization experiments provide useful robustness analyses.

Critical weaknesses
  1. The benchmark does not sufficiently validate counterfactual physical reasoning. E1 is essentially future-contact prediction, while E2 only evaluates the edited object's final position without considering its effects on other objects. The custom benchmark also provides limited details on data generation, physics configurations, and representative failure cases, making the results difficult to assess.

  2. The comparison with existing world models is insufficient. The experiments mainly use custom-designed baselines, without comparing against relevant methods (e.g., lewm, dino-wm). Moreover, CLS+v outperforms LSB on ADE and contact-pair accuracy in Table 1, yet the main OOD experiments primarily compare against the weaker CLS baseline.

  3. The contribution of the Law-State-Body decomposition remains unclear. LSB uses privileged object-level supervision and ground-truth object matching, making it difficult to isolate the benefit of the proposed decomposition. Additionally, the claimed modularity is not directly validated through transferring a frozen Law to a new embodiment by replacing only the Body module.

  4. The motivation for the proposed framework is not sufficiently compelling. The paper mainly argues that existing flat-latent world models lack explicit intervention capabilities, but does not clearly explain why the specific LSB decomposition is necessary for practical world modeling or embodied planning. The limited downstream planning improvements in Appendix A.11 further leave its practical benefits unclear.

Questions for authors

See Critical Weaknesses.

Reviewer_fHj1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study proposes LSB, an object-centric world model that separates object states, embodiment specific action encoding and shared dynamics to support prediction, interventions and contact queries. It reports experiments in a synthetic environment comparing against flat baselines and alternative architectures testing across several embodiments and distribution shifts.

Critical strengths

The explicit object state provides a practical interface for interventions. Table 1 in the paper shows improvements in query accuracy over the CLS+v baseline.

Critical weaknesses
  1. The CLS+v baseline is not included in the generalization studies. It outperforms LSB and the claimed prediction advantage appears to contradict Table 1.

  2. The benefit of the law/body separation is not evident from the experiments. Also the main text does not clearly delineate which components are trained or reused across different embodiments.

Questions for authors

Do the generalization benefits persist against the CLS+v baseline? Which components are trained/reused across different embodiments, and how do we isolate the benefits arising from the law/body separation?

Reviewer_EGVeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tried to deal with the problem that most learned world models roll a single flat latent forward. In this case, there is no natural handle for counterfactual questions like "what if this object moved differently." To sovle this problem, the paper proposes the LSB framework, which splits the world model into three parts:

  • State: The per-object slots from slot attention. Each one of them has different parts, such as position and velocity.
  • Law: a transformer over the slots, shared across embodiments.
  • Body: a small action encoder for each embodiment.

Then, the interventions have two parts: (1) Overwriting one slot's velocity and rolling forward with the same Law, and (2) a short counterfactual fine-tuning stage against the simulator. The paper contains experiments on a 2D physics sandbox with three embodiments, and the baselines are a flat GRU model (CLS) and a variant with a velocity pathway (CLS+v). The paper claims this method is better at prediction, contact queries, and intervention, and that it also generalizes to unseen physics, more objects, and longer horizons.

Critical strengths

Clear motivation. Explicitly addressable object state is a good why to support interventions, and the STEP / QUERY / INTERVENE framing is a useful way to evaluate world models beyond next-frame prediction.

Careful evaluation setup. CLS+v is a stronger baseline that gets the same edit signal, the alternative Law modules are parameter-matched and seeds are paired.

Critical weaknesses

1. The main claims don't match the paper's own tables.

  • The abstract claims large gains over a strong baseline on next-state prediction, but CLS+v is significantly better than LSB on ADE and on contact-pair E3 (Table 4).
  • Section 4.2 says LSB wins on ADE over both baselines, then a few lines later says CLS+v is better.
  • Section 4.5 says the GNN variants are indistinguishable from the transformer on three of four metrics, while Table 2 shows large gaps on E1 and E3.

2. The intervention result is weak.

  • LSB's E2 margin is +0.060 in Table 1, but −0.009 in Table 2 and −0.015 in Table 8. That is no better than assuming the object doesn't move.
  • Under unseen physics, it sits around zero.
  • Without counterfactual fine-tuning, it is clearly negative, so the gain seems to come from supervised training on the same kind of edits being evaluated, not from the slot structure.

3. Simple baselines are missing.

  • LSB's ADE is already about 0.45 at horizon 1 and stays almost flat out to 128 steps (Figure 7), while CLS is flat at about 0.70. This suggests most of the error comes from perception and readout, not dynamics.
  • Without copy-last-position and constant-velocity baselines, it's unclear whether either model learns useful dynamics.
  • Some accuracies are below chance, which points to a problem in how the metrics are read out.
Questions for authors
  1. Why do LSB's numbers, especially E2, differ across Tables 1, 2, 7 and 8, and which one is the headline result?
  2. Can you add copy-last-position, constant-velocity and ground-truth-state oracle baselines, and report ADE at horizon 0?
  3. Can you report results over at least three independent training runs of LSB?
  4. Can you test Law transfer directly, by freezing Law and training only a new Body for a held-out embodiment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.