acceptodds
Under review as a conference paper at ICLR 2027

Tangible and Editable Causal Action-to-Object Pathways in Robot World Models

Abstract

Robot world models can predict action outcomes yet still learn questionable action-to-object dependencies, making their predictions difficult to trust and their failures difficult to diagnose or correct. Existing black-box transitions obscure how the current action causally contributes to an object's predicted change within the learned model, while post-hoc attribution does not expose a separate causal pathway that can be directly revised. We address this limitation by making the model-internal causal contribution of the current action an explicit, auditable, and editable component of the world model. This enables potentially unreasonable learned action effects to be examined against independent simulator interventions and locally modified without retraining the entire dynamics network. To achieve this, we introduce a state-conditioned causal action-to-object pathway that participates directly in the learned transition and separates the current action's contribution from the remaining object dynamics. The pathway is learned jointly with the world model without direct action-effect or counterfactual supervision. We further develop a minimum-change causal mechanism edit that assigns the same executed action a prescribed alternative contribution to the predicted object transition while keeping the current state and action fixed. Experiments on Meta-World manipulation tasks show that the exposed pathway faithfully reflects the model's forward computation, aligns with independently measured action influence in the simulator, and supports precise targeted editing. Separate closed-loop interventions further demonstrate that modifying the learned pathway changes robot behavior and can yield exploratory success gains in selected tasks without retraining. These results show that world models can be designed not only to predict the consequences of actions, but also to expose, audit, and locally revise the internal causal mechanisms through which those consequences are represented, providing a foundation for assessing the reliability of opaque world models and exploring targeted interventions in failing tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, robot learning, causal mechanisms, model editing, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dUNCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper makes the action's contribution to the predicted object change an explicit, state-conditioned operator inside a TD-MPC2 world model. At a fixed physical context the operator is fixed, so an action perturbation changes the pre-normalization prediction by exactly the operator applied to it. A closed-form minimum-change update reassigns one action's contribution without retraining. Mean success on ten Meta-World tasks is 0.754, against 0.710 without the pathway and 0.524 for TD-MPC2. Edits chosen to improve control remain exploratory; the held-out Pick-Place gain fails Holm correction.

Critical strengths

At a fixed physical context the exposed operator is a fixed linear map, so an action perturbation moves the pre-normalization prediction by exactly that map applied to it. Prediction, inspection, and intervention all attach to this component, and post-hoc attribution offers no such target for revision.

The minimum-change edit intervenes on the mechanism while holding its input fixed: one action receives a prescribed object contribution, and orthogonal action directions are preserved. The closed-form guarantees are proved and audited, closed-loop removal of the pathway reduces success, and the six column-scaling candidates that fail Holm correction are reported alongside the rest.

Critical weaknesses

The audited minimum-change edit is never applied during control. Closed loop uses removal, gate saturation, β rescaling, and column scaling, with no rule for choosing the anchor action and target contribution at each step. No edit improves behavior after correction for multiple comparisons: the held-out Pick-Place gain fails Holm correction.

The alignment baselines come from the Factorized model, so model and access method change together; the full model's same-model comparison is unreported. Under this confound the pathway leads nine of nine tasks against seven of nine per baseline (AUPRC 0.342 vs 0.313), a margin the paper calls modest.

Questions for authors

The audited minimum-change edit is never used in control. Could you report closed-loop results with it, and state how the anchor action and target contribution are chosen at each step?

The alignment baselines come from the Factorized model. Could you report the same sensitivities from the full model's completed transition?

Table 1 gives 0.754 ± 0.043 against 0.710 ± 0.040 without defining the ± or providing per-task results or a test. Could you clarify both?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Comments

We thank the reviewer for recognizing the value of an explicit, editable action-to-object pathway. We conducted additional experiments to address the two weaknesses and three questions.

Weakness 1 & Question 1: Minimum-Norm Editing in Control

Response to Weakness 1 and Question 1

We have implemented the original minimum-norm edit in closed-loop MPC. At each decision step, the unedited planner first produces an anchor action a0a_0. A predefined coordinate-removal or sign-flip rule sets the target contribution c∗c^*; the rank-one update is then applied to root transitions during a second planning pass, without changing model parameters or forcing the anchor action to be executed.

We completed 720 paired rollouts on Door-Open, Drawer-Close, and Push (3 seeds × 20 initializations × 4 conditions per task). Sham trajectories matched the originals exactly. Edits changed MPC decisions and simulator trajectories: for example, Push terminal success changed from 71.7% to 85.0% under removal, while Door-Open changed from 50.0% to 33.3%. This directly verifies that the minimum-norm edit has consequences beyond the fixed-state algebraic audit.

We also evaluated evidence-guided targets rather than targets specified solely from model predictions. An external diagnostic mapping was fitted from independent simulator interventions; regularization and edit strength were chosen using validation prediction error, without task-success sweeps or test-set selection. The original CSB checkpoints remained frozen.

TaskOriginal MSEEdited MSEReduction
Push0.00063430.00061193.53%
Door-Open0.00031850.00029995.86%

On held-out target-geometry configurations, all 10 frozen checkpoints (5 per task) showed lower absolute next-step primary latent MSE against independently collected simulator transitions. This tests prediction correction, not only exact realization of a prescribed internal contribution.

Weakness 2 & Question 2: Same-Model Attribution Comparison

Response to Weakness 2 and Question 2

We repeated the attribution audit on the same 45 frozen CSB checkpoints (9 tasks × 5 seeds). All scores were computed on identical model states and actions and evaluated against the same independent simulator-intervention labels, removing the previous cross-model confound.

Attribution scoreMacro AUPRC
Intrinsic pathway0.3423
Group-centered pathway0.3417
Post-SimNorm Jacobian0.3372
Post-SimNorm finite difference0.3372

The intrinsic score remains competitive in this controlled comparison (raw minus Jacobian: +0.0051, hierarchical 95% CI [-0.0082, 0.0170]). Unlike post-hoc sensitivities, the pathway is itself part of the forward transition and can be edited directly.

Question 3: Table 1 Statistics

Response to Question 3

The reported 0.754±0.0430.754\pm0.043 and 0.710±0.0400.710\pm0.040 are mean ± SEM across five training-seed MT10 averages. We checked the original 100 task-seed evaluation records and reconstructed the per-task mean terminal success:

TaskCSBFactorized
Button-Press-Topdown1.001.00
Door-Open0.500.62
Drawer-Close0.980.98
Drawer-Open0.780.66
Peg-Insert-Side0.360.28
Pick-Place0.420.22
Push0.860.64
Reach0.820.88
Window-Close0.840.92
Window-Open0.980.90
MT10 Mean0.7540.710

CSB achieves higher mean success on five tasks and ties on two. The mean paired difference is +0.044 (95% CI [-0.039, 0.127], p=0.260p=0.260). These results confirm that adding the explicit editing interface maintains competitive control performance. We will clarify the SEM definition and include the task-level results in the revision.

Thank you for the clarification and the additional experiments. The closed-loop test answers my main question, but the effects are task-dependent, so I keep my rating at 3.

Reviewer_FZNGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work present a casual world model applicable to low-dimensional state-based environments. Extending the latent world model proposed in TD-MPC2, where the action-to-object interaction is explicitly modeled as a state-dependent linear projection in the latent space. With such explicit structure, the model outperforms TD-MPC2 on Meta-World and unlocks new use cases, e.g., mechanism editing.

Critical strengths
  1. The work attempts to address a valuable problem and demonstrates viability in a constrained setting.
Critical weaknesses
  1. The work is only validated on environments where low-dimensional state information is available. This does not reflect the scale and task complexity of contemporary latent world models and manipulation tasks.

  2. The key assumptions of when the method holds should be more explicit, so that readers can understand the feasibility of scaling up the method. For instance, one such assumption is that action-to-object pathway can be characterized as a state-dependent linear projection.

Questions for authors

Suggestions:

  • Describe TD-MPC2 in separate Preliminaries section for clarity.
1 comment
AuthorsOpenReview ↗

Response to Comments

We thank the reviewer for the careful assessment and constructive feedback. We provide additional evidence and methodological clarifications regarding evaluation scope, action-dependent dynamics, and TD-MPC2.

Weakness 1 & Related Question: Evaluation Scope and Generalization

Response to Weakness 1 and the Related Question

The contribution of Continuous-CSB is to make the learned action-to-object mechanism explicitly accessible for prediction, independent auditing, and post-training editing. Our evaluation tests these properties directly, rather than relying exclusively on task success or prediction accuracy.

The original experiments cover 10 Meta-World manipulation tasks, each trained independently with 5 random seeds and 20,000 environment interactions per run. The external audit uses 6,000 paired same-state simulator interventions to measure action-coordinate effects not used as training supervision. The learned pathway achieves positive AUPRC lift on all 9 non-null tasks, supporting consistent alignment across different manipulation settings.

We further evaluated diagnostic-guided editing under changed geometric configurations:

TaskOriginal / ShamDiagnostic EditRandom Edit
Push37.5%44.5%32.5%
Door-Open44.0%49.5%35.5%

These results show observed closed-loop improvements of 7.0 and 5.5 percentage points over the original controllers, respectively, with matched random-edit comparisons.

In a separate held-out target-geometry experiment, diagnostic-guided editing reduced absolute next-step primary-latent MSE across all 10 frozen checkpoints (5 Push and 5 Door-Open), by 3.53% and 5.86%, respectively, without retraining the original world models.

These experiments extend the evaluation to geometric variation within state-based manipulation. They do not constitute a claim of demonstrated visual or multi-object generalization. The proposed mechanism is defined at the latent transition level; incorporating visual encoders is a natural extension, rather than a requirement for its current verification.

Weakness 2 & Related Question: Action-Affine Assumption

Response to Weakness 2 and the Related Question

Continuous-CSB does not assume globally linear robot or object dynamics.

The primary-object transition is

l'_P = G_P(h) + beta R(h)a, z'_P = SimNorm(l'_P),

where h=[z_R,z_P] and R(h)=p(h)A(h).

Both G_P(h) and R(h) are nonlinear, state-conditioned functions learned by neural networks. The action-affine property holds only for the current action at a fixed physical context, before SimNorm. The completed transition is nonlinear because of the state dependence and subsequent normalization. The robot branch also retains a nonlinear action-conditioned residual.

This design makes the current action's contribution to the primary-object transition separately accessible without requiring globally linear dynamics. At fixed h, it satisfies the exact identity:

l'_P(h,a+delta_a)-l'_P(h,a) = beta R(h)delta_a.

The original forward-faithfulness audit confirms this identity to numerical precision (mean discrepancy 2.41e-7).

The explicit structure provides an intervention target with measurable behavioral consequences. Removing the pathway reduces mean terminal success from 75.4% to 58.6%. Additional minimum-norm editing experiments involving 720 closed-loop rollouts verify that edits to the learned mechanism alter MPC decisions and simulator trajectories.

Thus, action-affinity is a deliberately restricted local action interface, not an assumption that the environment or complete learned model has linear dynamics.

Question: TD-MPC2 Preliminaries

Response to the Question

We thank the reviewer for this helpful suggestion. We will provide a clearer introduction to TD-MPC2 in the revised manuscript and distinguish the existing learning and planning components from our proposed action-to-object mechanism and editing method.

Reviewer_XhusOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an explicit, state-dependent action-to-object pathway within a factorized robot world model. The pathway supports inspecting action contributions and applying closed-form, minimum-norm edits. Experiments on ten Meta-World tasks evaluate control performance, agreement with simulator interventions, and downstream effects of editing. While the interface is useful, the methodological novelty appears incremental: much of its interpretability and editability follows directly from the prescribed architecture, and evidence for reliably beneficial model corrections remains limited.

Critical strengths

The explicit pathway provides a clear interface for inspecting and intervening on action contributions within the learned transition. The evaluation includes an independent simulator-intervention audit, multiple training seeds, and transparent reporting of unsuccessful held-out edits.

Critical weaknesses

Strong structural assumptions and confounded gains. The method prespecifies the robot/object/goal partition and an action-affine object pathway. Most performance improvement already comes from factorization (52.4% → 71.0%, versus 75.4% for the full model), while additional transition changes confound the pathway’s contribution.

The influence score lacks invariance. Groupwise constant shifts in the columns of R(h)R(h) can change their norms without changing post-SimNorm predictions. Consequently, the proposed score is not uniquely determined by the model’s predictive behavior.

Exactness does not establish effective correction. The editing guarantee follows from a standard least-norm construction, and validation uses targets generated by the model itself. Downstream experiments instead use coordinate scaling, with no held-out candidate surviving Holm correction.

Limited generalization evidence. Evaluation uses separately trained, low-dimensional state-based tasks. Applicability to visual inputs, multiple objects, and more complex action effects remains unclear.

Questions for authors

How do you address the influence score’s sensitivity to softmax-invariant shifts? Would group-centered columns or post-SimNorm Jacobians improve the audit? Can you isolate the pathway’s benefit while keeping factorization, goal handling, and branch structure identical? Can edit targets and strengths be selected from diagnostic evidence, without task-success sweeps? Does the proposed minimum-norm edit itself improve physical prediction or control on held-out cases?

1 comment
AuthorsOpenReview ↗

Response to Comments

We thank the reviewer for the careful assessment. We address the four weaknesses and related questions with additional controlled experiments and analysis.

Weakness 1 & Related Question: Architectural Contribution

Response. Table 1 measures control performance (TD-MPC2 52.4%, Factorized 71.0%, CSB 75.4%); the main contribution is an explicit, auditable, editable action-to-object mechanism.

We trained a matched implicit-action baseline sharing the encoders, factorization, physical backbone, robot/goal branches, SimNorm, heads, losses, and MPC. Primary-action branches have 378,241 (CSB) vs. 378,445 (implicit) parameters. On 20 matched initial states per checkpoint:

Task/seedCSBImplicit
Door-Open/060%30%
Door-Open/130%0%
Push/070%70%

These are three completed exploratory comparisons, not a general performance claim. Importantly, removing R(h) within frozen CSB models, without changing factorization, reduces MT10 success from 75.4% to 58.6%. The same pathway also permits direct mechanism editing.

Weakness 2 & Related Question: SimNorm Invariance

Response. R(h) is the state-conditioned operator in the actual forward transition (Eq. 1). The original score ||R(h)e_j||_2 measures its pre-SimNorm pathway magnitude, not a uniquely identified normalized-output sensitivity.

The stated shift ambiguity can be handled analytically. Let P remove the mean within each 8-dimensional SimNorm group. The softmax identity gives

P log z'_P(h,a) = P G_P(h) + beta P R(h)a,

so d[P log z'_P(h,a)]/da_j = beta P R(h)e_j.

Therefore ||P R(h)e_j||_2 is strictly invariant to group-constant shifts and is determined by normalized-prediction action dependence at fixed h and known nonzero beta. This is a readout of the same learned operator: no model weights, forward predictions, or edit mechanism change. The 45-checkpoint supplementary audit confirms positive external-intervention alignment of this invariant readout.

Weakness 3 & Related Questions: Minimum-Norm Editing

Response. We applied the paper's rank-one minimum-norm edit in closed-loop MPC. At each decision step, an unedited planning pass supplies anchor action a0; a prespecified coordinate-removal/sign-flip reference defines c*=R(h)a_ref. Eq. (4) then edits root transitions during a second MPC planning pass, without changing model weights or forcing a0 to execute.

We completed 720 episodes (3 tasks × 3 seeds × 20 starts × 4 matched conditions). Original/sham trajectories match exactly; edits alter executed actions and simulator behavior. Under removal, Push terminal success changes 71.7%→85.0%, while Door-Open changes 50.0%→33.3%, showing functional, task-dependent effects.

To avoid relying solely on model-generated targets, we separately fit a diagnostic mapping from independent signed simulator interventions; validation prediction error selects regularization/edit strength, without success-rate sweeps or test selection. On independent held-out target-geometry transitions, all 10 frozen checkpoints (5 per task) decrease absolute next-step primary-latent MSE:

TaskOriginalEditedReduction
Push0.00063430.00061193.53%
Door-Open0.00031850.00029995.86%

This tests external prediction correction, not merely exact self-generated targets. The original Pick-Place column-scaling gain did not survive Holm correction; we do not claim statistically established control improvement.

Weakness 4: Generalization

Response. We further evaluated geometry-shifted configurations with original/sham and random-edit controls:

TaskOriginal/ShamDiagnosticRandom
Push37.5%44.5%32.5%
Door-Open44.0%49.5%35.5%

Diagnostic editing shows +7.0 and +5.5 percentage-point terminal-success changes over original, and +12.0 and +14.0 points over random editing. These results test geometric variation, not visual/multi-object generalization or statistically confirmed universal gains. Finally, R(h)=p(h)A(h) depends nonlinearly on h; action-affinity applies only at fixed h before SimNorm, not to global world dynamics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.