- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies whether action conditioned robot world models are invariant to equivalent action representations. The authors show that a latent dynamics predictor can fail severely when switching between absolute and state relative action encodings, and propose an admissibility analysis together with pooled training and a consistency objective as a repair.
Critical strengths
-
The paper identifies a concrete and underexplored failure mode in action-conditioned prediction. The distinction between an information-preserving reparameterization and an input transformation that removes information is useful and conceptually meaningful.
-
The experimental analysis is reasonably careful. In particular, the paper includes pooled training with and shows that much of the average performance can already be recovered without the proposed consistency term, which helps isolate the actual contribution of the additional regularization.
Critical weaknesses
- The empirical evidence is too narrow to support the paper's main claim about robot world models. The core robotics experiments use a frozen I-JEPA encoder and a small two-layer MLP predictor trained on a limited number of examples. This establishes a failure mode of this particular latent prediction pipeline, but does not show that the same issue occurs in modern sequential world models, recurrent latent models, or action-conditioned video models. Since the paper itself shows that invariance behavior can change with the representation, the current title and conclusions are substantially broader than the evidence supports.
- The proposed consistency objective does not show a meaningful task-level benefit over simple pooled training. On ALOHA, pooled training with already improves the held-out retrieval rank from 61.5 to 5.98, while the consistency model only improves it further to 5.19. Action selection changes from 40.1% to 41.1%. The main gain is in cross-encoding cosine agreement, which is directly optimized by the proposed loss. Without a downstream planning or control experiment showing that this remaining disagreement causes actual failures, the method contribution remains limited.
Questions for authors
- Can the authors reproduce the main result using at least one substantially different sequential world model architecture? This is necessary to distinguish a general action-representation problem from a failure specific to the frozen I-JEPA plus MLP setup.
- Can the authors provide a planning or closed-loop control experiment, preferably on PushT, where pooled training with still fails but the consistency objective improves actual task success? This would directly demonstrate the practical value of improving worst-case cross-encoding agreement.
Thank you for the review. We were glad you found the failure mode concrete and under-explored, and the distinction between information-preserving re-parameterizations and lossy transformations conceptually meaningful. Those two points are what the revised paper will put first.
"The evidence is too narrow for the title and conclusions." We agree, and we will rescope. The abstract, introduction and conclusion will say action-conditioned latent predictors, and the architecture will be stated in the first paragraph. If a title change is permitted, the paper will be retitled "Action-Conditioned Latent Predictors Are Not Invariant to How the Actions Are Written". We cannot add a second architecture within the discussion period.
What the current evidence does support is that the failure is not specific to one dataset, and that it does not go away with scale within this family. It appears on three datasets and two morphologies. The collapse ratio grows with data (2.5× at n = 150 to 9.9× at n = 1200), and cross-encoding agreement gets worse, not better, with more capacity (0.370 -> 0.125) and longer training (0.370 -> 0.242). We note in the paper that those larger teachers also overfit, so this is not independent evidence. The revised PDF will say why we expect the failure in sequential, recurrent and video models (nothing in training ties the two encodings, so the unseen one is off-support), but will present this as a hypothesis, not a result.
"The consistency objective shows no meaningful task-level benefit over pooled training." On average task metrics on ALOHA at n = 1200, we agree. The paper reports this and credits the task-level recovery to pooled training, not to the penalty. Two results go beyond the single-step agreement the penalty optimizes.
First, multi-step rollouts. Eq. (2) penalizes single-step disagreement only, and rollout agreement is a downstream consequence that it does not optimize directly. At the final hop on cabinet, worst-case agreement is 0.242 for the teacher, 0.896 for pooled training and 0.997 with the penalty. On PushT the gap is larger. Pooled training leaves a mean of 0.677 and a worst case of −0.348, so for some inputs the two encodings still predict opposing futures after two hops. With the penalty these are 0.872 and 0.139. On PushT, pooling alone does not repair the axis.
Second, low data. In §5.2 the penalty's effect on retrieval changes sign with n. At n = 150, λ = 256 improves retrieval rank by 6.59 over λ = 0; at n = 1200 it costs 0.38. The penalty's task-level value appears where training data is scarce. "A planning or closed-loop experiment on PushT." We agree this is the experiment that would settle the practical value of the tail, and we cannot run it in the discussion period. We will list it as the first open item in the conclusion.
The reason we expect the tail to matter for planning is an argument, not a measurement, and the revised PDF will label it as one. A planner scores candidate actions and selects the best, so it preferentially selects the actions where the model is most wrong. A model that is accurate on average but points elsewhere on a small fraction of inputs is the case a planner will find. Mean agreement understates that risk, and worst-case agreement measures it.
Presentation. We will make these changes for clarity:
We would be grateful if you would reconsider the paper as rescoped. Its claims are a failure in action-conditioned latent predictors that is catastrophic and unaudited, a test that decides which transformations are admissible before invariance is enforced, and an honest account of what pooling and the penalty each contribute.