acceptodds
Under review as a conference paper at ICLR 2027

Robot World Models Are Not Invariant to How the Actions Are Written

Abstract

A robot policy is trained with one of two action parameterizations: absolute joint targets, or deltas relative to the current state. The choice is a live engineering decision in robot learning, and a world model conditioned on actions inherits it silently. We show that the inheritance is catastrophic. A latent dynamics model trained on one parameterization and handed the identical commanded trajectory written in the other collapses: retrieval degrades by 2.6–13.4× across three robot datasets and two morphologies, goal-conditioned action selection falls from 53% to 15%, and on PushT the two beliefs about the same future are near-orthogonal (cos = 0.067, worst case −0.377), so the predictor does not degrade gracefully, it answers a different question. This is not a distribution-shift artifact in the usual sense: the two encodings are mutually reconstructible at R² = 0.996 given the joint input, so no information is lost in the change, and we give the test that establishes this and separates a valid re-parameterization from a lossy summary or a sensor swap. The test rejected three of the four axes we proposed. The defect lives in the action channel, which the invariance literature for visual models does not examine: work there concerns crops, jitter and camera pose, while the parameterization of the commands themselves goes unaudited. The repair is averaging over the two encodings, and where it goes matters. Averaging the objective restores task performance by itself, and we report that rather than crediting it to anything of ours; averaging the outputs, which is safe for probabilities by concavity, is not available for direction-valued prediction, where we show the normalized mean can score below every member of the orbit. What objective-averaging leaves behind is the tail: worst-case agreement stays at 0.78, a disagreement penalty closes it to 0.995, and over a latent rollout it is the difference between a worst case that erodes and one that holds. On PushT, averaging alone does not repair the axis at all.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, Robots, Action Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iYjBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether action conditioned robot world models are invariant to equivalent action representations. The authors show that a latent dynamics predictor can fail severely when switching between absolute and state relative action encodings, and propose an admissibility analysis together with pooled training and a consistency objective as a repair.

Critical strengths
  1. The paper identifies a concrete and underexplored failure mode in action-conditioned prediction. The distinction between an information-preserving reparameterization and an input transformation that removes information is useful and conceptually meaningful.

  2. The experimental analysis is reasonably careful. In particular, the paper includes pooled training with λ=0\lambda=0 and shows that much of the average performance can already be recovered without the proposed consistency term, which helps isolate the actual contribution of the additional regularization.

Critical weaknesses
  1. The empirical evidence is too narrow to support the paper's main claim about robot world models. The core robotics experiments use a frozen I-JEPA encoder and a small two-layer MLP predictor trained on a limited number of examples. This establishes a failure mode of this particular latent prediction pipeline, but does not show that the same issue occurs in modern sequential world models, recurrent latent models, or action-conditioned video models. Since the paper itself shows that invariance behavior can change with the representation, the current title and conclusions are substantially broader than the evidence supports.
  2. The proposed consistency objective does not show a meaningful task-level benefit over simple pooled training. On ALOHA, pooled training with λ=0\lambda=0 already improves the held-out retrieval rank from 61.5 to 5.98, while the consistency model only improves it further to 5.19. Action selection changes from 40.1% to 41.1%. The main gain is in cross-encoding cosine agreement, which is directly optimized by the proposed loss. Without a downstream planning or control experiment showing that this remaining disagreement causes actual failures, the method contribution remains limited.
Questions for authors
  1. Can the authors reproduce the main result using at least one substantially different sequential world model architecture? This is necessary to distinguish a general action-representation problem from a failure specific to the frozen I-JEPA plus MLP setup.
  2. Can the authors provide a planning or closed-loop control experiment, preferably on PushT, where pooled training with λ=0\lambda=0 still fails but the consistency objective improves actual task success? This would directly demonstrate the practical value of improving worst-case cross-encoding agreement.
1 comment

Thank you for the review. We were glad you found the failure mode concrete and under-explored, and the distinction between information-preserving re-parameterizations and lossy transformations conceptually meaningful. Those two points are what the revised paper will put first.

"The evidence is too narrow for the title and conclusions." We agree, and we will rescope. The abstract, introduction and conclusion will say action-conditioned latent predictors, and the architecture will be stated in the first paragraph. If a title change is permitted, the paper will be retitled "Action-Conditioned Latent Predictors Are Not Invariant to How the Actions Are Written". We cannot add a second architecture within the discussion period.

What the current evidence does support is that the failure is not specific to one dataset, and that it does not go away with scale within this family. It appears on three datasets and two morphologies. The collapse ratio grows with data (2.5× at n = 150 to 9.9× at n = 1200), and cross-encoding agreement gets worse, not better, with more capacity (0.370 -> 0.125) and longer training (0.370 -> 0.242). We note in the paper that those larger teachers also overfit, so this is not independent evidence. The revised PDF will say why we expect the failure in sequential, recurrent and video models (nothing in training ties the two encodings, so the unseen one is off-support), but will present this as a hypothesis, not a result.

"The consistency objective shows no meaningful task-level benefit over pooled training." On average task metrics on ALOHA at n = 1200, we agree. The paper reports this and credits the task-level recovery to pooled training, not to the penalty. Two results go beyond the single-step agreement the penalty optimizes.

First, multi-step rollouts. Eq. (2) penalizes single-step disagreement only, and rollout agreement is a downstream consequence that it does not optimize directly. At the final hop on cabinet, worst-case agreement is 0.242 for the teacher, 0.896 for pooled training and 0.997 with the penalty. On PushT the gap is larger. Pooled training leaves a mean of 0.677 and a worst case of −0.348, so for some inputs the two encodings still predict opposing futures after two hops. With the penalty these are 0.872 and 0.139. On PushT, pooling alone does not repair the axis.

Second, low data. In §5.2 the penalty's effect on retrieval changes sign with n. At n = 150, λ = 256 improves retrieval rank by 6.59 over λ = 0; at n = 1200 it costs 0.38. The penalty's task-level value appears where training data is scarce. "A planning or closed-loop experiment on PushT." We agree this is the experiment that would settle the practical value of the tail, and we cannot run it in the discussion period. We will list it as the first open item in the conclusion.

The reason we expect the tail to matter for planning is an argument, not a measurement, and the revised PDF will label it as one. A planner scores candidate actions and selects the best, so it preferentially selects the actions where the model is most wrong. A model that is accurate on average but points elsewhere on a small fraction of inputs is the case a planner will find. Mean agreement understates that risk, and worst-case agreement measures it.

Presentation. We will make these changes for clarity:

  • The predictor's inputs will be stated in §5's first paragraph and in Fig. 1.
  • A notation table will define orbit, canonical and held-out encoding at first use.
  • Every rank column will be marked "lower is better".
  • The abstract will separate what pooling buys from what the penalty buys.

We would be grateful if you would reconsider the paper as rescoped. Its claims are a failure in action-conditioned latent predictors that is catastrophic and unaudited, a test that decides which transformations are admissible before invariance is enforced, and an honest account of what pooling and the penalty each contribute.

Reviewer_2NNLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper demonstrates that world-model predictions can change substantially when the same physical action is expressed as an absolute target or a relative displacement. This is a useful diagnostic, but the theoretical justification for information-preserving invariance and the interpretation of the empirical results require clarification.

Critical strengths
  • The paper identifies a practically relevant sensitivity to action conventions. Across three robot datasets, switching between absolute and relative action encodings substantially degrades prediction and offline action selection. The comparison between pooled training and an additional consistency penalty also usefully distinguishes task-accuracy improvements from cross-encoding agreement.
Critical weaknesses
  • The input assumptions and information-preservation claims are insufficiently established. The same numerical action vector can represent different physical commands under absolute and relative conventions, yet convention-token and common-convention baselines are missing. It is also unclear whether the predictor receives sts_t or uses it only for action conversion. Reconstruction with explicit state does not establish information equivalence for a predictor receiving only image latents and actions. Moreover, R2=0.996R^2=0.996, finite-model predictive parity, and minimum test-set agreement of 0.995 are empirical evidence rather than guarantees of lossless invariance or unseen-input performance.

  • The experiments do not sufficiently establish the generality or mechanism of the improvements. The main action-encoding experiments use a frozen encoder and a two-layer MLP trained on 1,200 samples, limiting broader claims about world models. Rollout agreement should be accompanied by ground-truth accuracy, and offline action selection should be distinguished from closed-loop control. Furthermore, Proposition 3 does not establish that normalized averaging causes the observed distillation gap; teacher errors and differences in supervision remain confounding factors.

Questions for authors
  • Does the predictor explicitly receive sts_t, or is state used only for action conversion? How is subtracting the initial state from the entire action window related to per-step delta control? Are results available for convention-token or common-convention baselines?

  • What supports the claims of information preservation beyond approximate reconstruction and finite-model predictive parity? Can “provably” and “guarantee” be qualified, with lower-tail quantiles and uncertainty estimates reported instead?

  • Are ground-truth prediction accuracies available alongside cross-encoding agreement at every rollout hop? Is there evidence from additional model architectures, or should the conclusions be restricted to the evaluated MLP setting?

  • What are the respective accuracies of individual teacher predictions, their normalized average, and the distilled student? These measurements would help determine whether averaging itself explains the distillation gap.

1 comment

Thank you for specific questions, each of which we can answer directly. Several will lead to changes in the paper.

Does the predictor receive s_t? No. The input is [z_t ‖ φ_v(a)]; s_t is used only to write the relative encoding. You are right that reconstruction with explicit state does not establish equivalence for a predictor that sees only image latents and actions, and the revised §3 will state this gap. On the model's actual input we have retrieval parity: given z_t, the λ = 16 model reaches rank 5.19 on the relative encoding and 5.17 on the absolute one. We are happy to report a linear probe of s_t from z_t during the discussion if useful.

Window-relative vs per-step deltas. Our encoding is anchored: every point in the window is written relative to the current state (as in UMI's relative trajectories), not as per-step differences a_k − a_{k−1}. Per-step deltas are also a bijection given s_t (by cumulative sum), so the test would admit them on the same grounds; we did not evaluate them. The revised PDF will say "state-relative" rather than "delta" and note this.

Convention-token and common-convention baselines. Converting to absolute and calling the teacher reproduces the teacher's canonical numbers by construction (rank 6.19, selection 53.1% on cabinet); we will report it. Pooled training beats it on retrieval (5.17–5.98) and loses on canonical selection (41-44%). Your point that the same vector can mean different commands under the two conventions is, we think, the right explanation both for the teacher's failure and, plausibly, for the canonical selection drop under pooling without a flag. A token addresses that ambiguity but places no constraint on agreement, so it complements Eq. (2) rather than replacing it. We have not run it and will say so.

Evidence beyond approximate reconstruction and finite-sample parity. There is none, and the revised PDF will say so. Theorem 2 gives the logic: the parity gap lower-bounds the cost of enforcing invariance, and reconstructibility is a sufficient certificate; both are estimated from finite samples, and the ridge probe certifies only linear relabellings (as the limitations note). We will remove "provably" and "guarantee" wherever they describe empirical quantities. The worst case we report is the minimum over the 384 test samples, stricter than any lower quantile; we will add the 1st and 5th percentiles. With three seeds we report directions rather than magnitudes. Ground-truth accuracy alongside rollout agreement. Agreed that agreement is not accuracy: two encodings can agree on a wrong rollout. Our rollout claim is about consistency only, and the revised §5.1 will say so. The paper reports cosine to the true latent falling from 0.33 to 0.13 over three hops on cabinet (§5.1); we agree it belongs beside the agreement numbers in Table 4.

Offline selection vs closed-loop control. Agreed. We will say "offline action selection" throughout; closed-loop evaluation is listed as a limitation.

Other architectures. We will restrict conclusions to the evaluated setting (frozen encoder, MLP predictor, three datasets, two morphologies). We will still explain why we expect generality: no training signal ties the two encodings, so the unseen one is off-support in any action-conditioned model trained on one convention, and within our family the failure does not shrink with data, capacity or optimization (Table 5, whose larger teachers also overfit). We present this as a reason to expect generality, not evidence for it.

Prop. 3 and the distillation gap. You are right. Prop. 3 is an existence result: output averaging is not safe for direction-valued predictions as it is for probabilities. It does not show this caused the student's gap (rank 11.55 / 15.1 vs about 5.2 for Eq. (2)). The confound you name is real: the orbit mean averages the teacher's canonical prediction with its collapsed relative one (mean cosine 0.37), so the student is distilled from a target that already contains the failure. We will rewrite §4 to present Prop. 3 as the reason we average the objective, not as an explanation of the gap. For the accuracies: teacher 6.19 / 61.5 (Table 1), student 11.55 / 15.1 (Table 2); the normalized teacher average before distillation is not reported, and we agree it is the number that would separate the explanations.

Reviewer_Ui9pOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the choice of action parameterization affects action-conditioned world models. The same action trajectory can be written either as absolute joint targets or as deltas relative to the current state. The authors train an MLP on top of frozen I-JEPA features; given the latent of the current frame and an action window, it predicts the latent 4 seconds ahead. They find that a model trained only on the absolute encoding degrades sharply when the same actions are given in the relative encoding. On the two ALOHA datasets and PushT, the mean retrieval rank becomes 2.6–13.4× worse, and on aloha_mobile_cabinet, goal-conditioned action selection accuracy drops from 53.1% to 15.4%. To show that the two encodings carry the same information, the authors propose a test requiring that they be equally predictive on their own and recoverable from each other given the current state. They also use this test to rule out three other candidate transformations. As a fix, the authors train the model on both encodings jointly, which alone recovers most of the lost performance. Adding a consistency penalty further raises the worst-case agreement between the two encodings’ predictions from 0.779 to 0.995.

Critical strengths
  1. Robot actions can be encoded as absolute targets or as changes from the current state. The choice is usually made for engineering reasons, such as the controller interface or the teleoperation setup. The authors show that a world model silently depends on the encoding it was trained on. When the model receives the same actions in the other encoding, it does not just get a little worse. It answers a different question. In the worst case, the two predictions have a negative cosine similarity. Figure 1 shows this clearly. This is a useful reminder for anyone who mixes datasets or switches control interfaces.

  2. Before forcing the model to be invariant to a change, the authors ask a simple question. Does the change only rewrite the input, or does it change what the model knows? Switching cameras and summarizing the action window are examples of the second kind. Some of the points here are useful for other problems too. First, if one encoding predicts better than the other when used alone, enforcing invariance costs at least the difference between them. The paper calls this the parity gap. Second, if each encoding can be recovered from the other, invariance is free. But it can also be free when they cannot be recovered, as in the angle-wrap example. Third, this check must use everything the model receives. Looking at the actions alone gives a false negative. R² is only 0.757, but it rises to 0.996 once the state is added.

Critical weaknesses
  1. The main finding is not surprising, and it is unclear when this problem would come up in practice. The teacher is trained only on absolute actions, so relative actions are new inputs to it. A drop in performance is expected, and the authors themselves call it covariate shift in §5.1. The two encodings carry the same information, but that does not mean a model will work on both. For example, a model trained on RGB images also fails on BGR images. In practice, the encoding is known and s_t is available at inference time. Adding s_t to the relative actions turns them back into absolute actions, which takes one line of code. The paper does not compare against this simple fix. It also does not show a real setting where the encodings actually differ, such as when datasets from different sources are mixed.

  2. The claim that the two encodings carry the same information may not have been checked on the input the model actually receives. The R² of 0.996 appears to use both the actions and s_t. But the model only receives the image feature z_t and the actions, not s_t. So for the model, the two encodings are equivalent only if s_t can be recovered from z_t, and the paper does not report this. If s_t cannot be recovered well, the two encodings give the model different information. By the authors’ own standard, forcing the same prediction for both encodings then has a cost. The results are consistent with this. After training on both encodings, action selection accuracy on the absolute encoding drops from 53.1% to 44.3%. On PushT, the repaired model’s retrieval rank on the absolute encoding also gets worse, from 66.2 to 79.2. This conflicts with the abstract’s claim that averaging the training objective is enough to restore task performance.

Questions for authors
  1. Was the R² of 0.996 computed with s_t or with z_t? The model does not receive s_t. So for the model, the two encodings are equivalent only if s_t can be recovered from z_t. Could you report the R² of linearly decoding s_t from z_t, and the in-distribution performance of a teacher trained only on the relative encoding? Also, after pooled training, canonical action selection drops from 53.1% to 44.3%. Is this related to the two encodings giving the model different information?

  2. How does Eq. (2) compare with simpler fixes? For example, at inference time one can use s_t to convert relative actions back to absolute ones, or add a token that marks which encoding is used. In what real setting would Eq. (2) work better than these fixes?

1 comment

Thank you for a precise summary and for questions that target the weakest link in our argument. Points shared with other reviewers are also covered in the general response.

"Not surprising; it is covariate shift." We agree on the mechanism (§5.1 calls it covariate shift), and the RGB/BGR analogy makes our case: pipelines check channel order because that failure is known. The action channel is not audited, although the choice is live (Diffusion Policy compares position and velocity control; UMI uses relative trajectories; Open X-Embodiment pools datasets with different action spaces). What is informative is the form of the failure: mean cosine 0.37 on ALOHA and 0.067 on PushT, with negative worst cases. It also does not shrink with data: from n = 150 to 1200 the collapse ratio grows from 2.5× to 9.9× while cross-encoding cosine stays at about 0.37 (§5.2).

"Converting with s_t is one line of code." Agreed: when the convention is known and s_t is available, this is the right fix, and the revised PDF will say so and report it as a baseline. By construction it reproduces the teacher's canonical numbers (rank 6.19, selection 53.1%); pooled training beats it on retrieval (5.17–5.98) and loses on canonical selection (41–44%). We have not shown a robotics setting where conversion is unavailable, and will remove wording implying one. Our case rests elsewhere: conversion presumes you know the transformation is a lossless relabelling, and the admissibility test is how you find out. It rejected three of our four proposed axes, including action-window summaries, which look like re-encodings and are not (task R² 0.986 vs 0.709). In the mixed-source setting you mention, the cross-encoding audit is what would catch a mislabelled source; we will name this as motivation and state that we have not run it.

R² with s_t or z_t? With s_t: the joint input is (s_t, a). You are right that the predictor sees z_t, so this does not by itself establish equivalence on the model's input; the revised §3 and abstract will say so. On the model's actual input we have retrieval parity: given z_t and the relative encoding, the λ = 16 model reaches rank 5.19, against 5.17 on the absolute encoding and 6.19 for the absolute-only teacher. A linear probe of s_t from z_t is the direct test, and we are happy to report it during the discussion if useful. We agree a relative-only teacher would also be informative.

Canonical selection drops 53.1% -> 44.3% after pooling. That drop is at λ = 0, where no invariance is enforced: the model is free to treat the encodings differently, so Theorem 2's cost of invariance does not apply, and an information deficit in (z_t, a_rel) would not by itself hurt the absolute input. The comparison that isolates the cost of invariance is λ = 0 vs λ = 16: canonical selection 44.3% -> 41.1% (intermediate values 40.8%, 42.6%, not monotone), while retrieval improves 5.46 -> 5.17. Three seeds cannot resolve this, and the revised PDF will say so. A more plausible cause of the λ = 0 drop is the ambiguity Reviewer 2NNL raises: without a convention flag, the same vector can mean different commands, so the pooled model must hedge. We will list this as an open question in §5.1. The PushT cost (66.2 -> 79.2) is already reported in §5.1, and we will rewrite the abstract so it no longer claims averaging restores task performance in general.

Convention token vs Eq. (2). These are complementary: a token says which encoding the model sees; Eq. (2) asks the two to agree. A token model is pooled training at λ = 0 with a flag, and the penalty can be added on top. We have not run it and will say so in §6. In short, we agree the paper overstated the repair and understated the gap between (s_t, a) and (z_t, a), and we will revise both. We hope the rescoped claims (a catastrophic, unaudited failure; a test separating re-parameterizations from lossy transformations; an honest account of what pooling and the penalty each buy) address your soundness concerns.

Comments on the paper

1 comment

We thank the reviewers for careful reads. All three agree the failure is concrete. Two single out the admissibility test (the parity gap, reconstructibility on the joint input, and the three rejected axes) as useful beyond this problem. The shared concerns are about what our evidence supports, and several of them are correct. We will revise the paper to match the evidence. The four shared points are below.

  1. What the predictor sees. The predictor receives [z_t ‖ φ_v(a)] only. s_t is used to write the relative encoding (a_rel = a_abs − s_t); it is not a predictor input. The R² = 0.996 is computed on (s_t, a). It certifies that the two encodings carry the same information given the robot state. It does not by itself certify that they carry the same information given (z_t, a), and Reviewers Ui9p and 2NNL are right that the paper should have said so. We will add this to §3 and the abstract. Our existing results show predictive parity on the model's actual input for retrieval. One predictor that receives z_t and the relative encoding reaches retrieval rank 5.19, against 5.17 for the absolute encoding under the same model and 6.19 for the absolute-only teacher (Table 2 at λ = 16; Table 1). Whatever the relative encoding lacks given z_t, retrieval does not show it. Action selection behaves differently; see point 3. We are happy to report a linear probe of s_t from z_t during the discussion if that would help.

  2. Scope. Our experiments use a frozen I-JEPA encoder with a two-layer MLP predictor. We will rescope the abstract, introduction and conclusion to action-conditioned latent predictors, and state the architecture in the first paragraph. If a title change is permitted, we will retitle the paper "Action-Conditioned Latent Predictors Are Not Invariant to How the Actions Are Written". We give reasons to expect the failure in other architectures, but will no longer claim it as shown.

  3. Canonicalization, convention tokens, and what the repair costs. When the convention is known and s_t is available, converting relative actions back to absolute before calling the teacher is the right fix. By construction it recovers the teacher's canonical numbers (rank 6.19, selection 53.1% on cabinet). The revised PDF will say this explicitly and list it as a baseline. Compared with that baseline, pooled training trades in opposite directions on the two metrics. Retrieval is better (5.17–5.98 vs 6.19), and canonical action selection is worse (41–44% vs 53.1%). The abstract's "restores task performance" overstated this. The revised abstract will say that retrieval is restored and held-out selection rises from 15.4% to 40–41%, at a cost of 9–12 points on canonical selection. A convention token is a complementary fix: it tells the model which encoding it sees, but places no constraint on agreement between them. We will discuss it as a baseline we have not run.

Our contribution is therefore not a better fix for one known conversion. It has three parts:

  • The failure is catastrophic, not graceful. Predictions under the two encodings are near-orthogonal (cos 0.067 on PushT) or point in opposing directions (worst cases −0.178 and −0.377).

  • A test decides, before invariance is enforced, whether a candidate transformation is a re-parameterization or a loss of information. It rejected three of the four axes we proposed.

  • An account of where averaging belongs (the objective, not the outputs; Prop. 3), and of what the disagreement penalty buys in the tail.

  1. Wording. We will remove "provably" and "guarantee" where they described empirical quantities. R² = 0.996, predictive parity and the 0.995 worst-case agreement will be described as finite-sample measurements on the test split.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.