- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes CLAW, a self-supervised framework that jointly learns continuous latent actions and a diffusion-based world model from videos without action labels. A latent action model infers actions from consecutive observations, while the world model uses them to predict future frames. The main methodological component is a two-path adversarial regularizer: one prediction path receives the initial observation and latent action, while the other receives only the latent action and uses gradient reversal to discourage leakage of future-observation information. The paper evaluates the learned representations through imitation from observation, action retrieval and transfer, and real-robot policy pretraining, as well as visual planning with the world model. The authors report gains over several latent-action baselines on many tasks, although results vary across individual benchmarks.
Critical strengths
- The paper addresses two meaningful challenges in latent-action learning: future-observation leakage and uninformative representations. The problem is well motivated and clearly explained.
- The proposed method is innovative, particularly its joint training of the latent action model and world model with a two-path adversarial regularizer.
- The evaluation covers a wide range of tasks and includes useful visualizations and analyses. This breadth makes the empirical findings persuasive.
Critical weaknesses
- Action information can help predict future observations even without the current observation; absolute-position commands, for example, predict the robot’s next configuration. The regularizer may therefore suppress both scene information and useful action information. The transfer results and leakage metric do not sufficiently distinguish these effects.
- CLAW w/o Adv retains the second prediction branch’s updates to the shared world model. The comparison supports the benefit of adversarial gradients within this architecture, but does not establish an advantage over conventional joint training with only conditional prediction.
Questions for authors
- Can the authors provide evidence that the adversarial term reduces scene information while preserving useful action information? I suggest using independent probes to measure both types of information in the latent representation and examining how they change with adversarial strength.
- Does the full method consistently outperform conventional joint training with the second prediction branch entirely removed? Adding this baseline and extending the adversarial ablation to representative imitation-learning and real-robot tasks would clarify the contribution and its scope.
I will reassess and adjust my score based on the authors’ responses and additional experimental evidence.
3 comments
Following your suggestion, we probed the latent with independent linear probes across adversarial strengths (see our follow-up to the General Response). As the penalty approaches full strength, scene content in declines while action accuracy reaches 0.71 by step 69k. This is consistent with the regularizer reducing scene information without suppressing action information.
Thank you for conducting the additional experiments and the linear probing analysis. These results directly address my main questions regarding the preservation of action information and the baseline without the auxiliary branch. I am satisfied with the response and will maintain my positive score
We thank the reviewer for the constructive review and the concrete suggestions.
Does the regularizer suppress useful action information, such as absolute-position content, along with scene information?
App. D.3.5 gives our conceptual argument: the auxiliary branch receives only zi, so its loss can decrease only through information that predicts this particular future without the current observation. Absolute end-effector content generalizes across scenes, whereas object identity, layout, and appearance do not, so the gradient pushes primarily against scene-specific content.
Three pieces of evidence indicate that action content survives the regularizer:
As D.3.5 acknowledges, these results do not isolate the absolute component directly, and we have not tested action spaces in which the agent controls nearly the entire scene. There, a uniformly applied penalty could suppress action content along with context. We will state this limitation more prominently.
Does the full method outperform conventional joint training with the second branch removed?
Yes. We trained CLAW w/o Branch, which is exactly this baseline (see the General Response).