- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work presents LaFreeWM, an extension to LeWM that can leverage action-free data for training the world model using learned discrete latent actions. The authors propose 3 additional losses for shaping the latent actions and an additional training stage for grounding the latent actions that enables planning in the true action space. LaFreeWM is compared to other JEPA world model variants including the base LeWM on 4 simulated manipulation and navigation environments.
Critical strengths
Thorough related work section on latent action models.
Detailed Appendix.
Critical weaknesses
Evaluation protocol is weak and results are not substantial.
Evaluating goals with a fixed offset implies they are chosen from a dataset. Assuming they are sampled from the training dataset, this does not test the most basic form of generalization. Furthermore, limiting the horizon of the goals to at most 15 steps cannot be considered ``long horizon’’ in my opinion. The fact that none of the methods tested in this study achieve good performance on some of these tasks makes it hard to attribute any gains in performance to fundamental properties of said methods.
In terms of reported results in Table 1, with 10% labels, LeWM’s performance does not degrade within a single standard deviation from the 100%-label results in all but TwoRoom. LaFreeWM only outperforms it beyond a single standard deviation on 2 of the 4 tasks. These results do not justify the complexity added to LeWM as I see it. With 1% labels the gains are more clear but this severely limits the data available to LeWM and thus the results are not surprising in this extreme scenario.
Ablation study is limited and not thoroughly explained.
The ablation study is limited in the sense that it focuses on a single task and label budget. The variants are not thoroughly explained other than their short names in Table 2 and are not motivated. The results of "Without JEPA loss’" and "Detached prediction gradient" are within a standard deviation of the best performing variant and are thus not justified. Overall, LaFreeWM adds significant complexity on top of LeWM but it is hard to tell which of the additions contribute to better performance, if at all.
Questions for authors
- L_pred is not defined anywhere. How is it different from L_LAM?
- Why is L_JEPA over the actions necessary? It seems that you are encouraging the latent action to represent a normalized delta between latent states. As I see it, the standard latent action formulation already encourages this and your ablation study does not support its significance for planning.
- The fact that LaFreeWM with 10% action labels performs better than LeWM with the full data budget on some tasks suggests that something in the training objective improves performance regardless of the amount of action-labeled data available. What do you think that is?
9 comments
I appreciate your effort in writing this response, answering my questions and running additional experiments.
I do not see a new revision of the ablation text so it is still hard to understand and review the ablation study. The additional ablation results you presented show marginal improvements.
Overall the paper applies latent action objectives to LeWM to improve its performance when less action labels are available. The novelty and significance of the additional losses is not made clear by the results and text.
The benefits of latent actions in learning from limited action-labeled data is well established by previous work. My opinion is that adding latent action objectives to LeWM is a limited contribution and therefore I maintain my score.
Thank you for reading our response and for the follow-up.
Our apologies for the delayed revision. We have now uploaded a revised PDF. The ablation text in Section 5.2 is rewritten, Table 1 now reports the re-tuned TwoRoom results, and the additional results are in Appendix C.4 and C.5.
Significance of the additional losses. The VQ bottleneck and the LAM loss are what make the latent actions useful. Without either of them, pretraining on the unlabelled data gives no gain over the constant action ablation (18.0% and 26.7% vs 26.7%). Without LAM the codebook is still fully used (61.2 effective codes), so the codes are there but they do not help planning. The JEPA loss has a different role. Its effect on planning is small (0.2 to 2.6 points) however positive for all environments, it mainly changes what the codes represent: the cosine similarity between a code and the direction of latent change goes from 0.28 to 0.58, and codebook usage from 45.0 to 61.5.
Novelty. We agree that latent actions are known to help with limited labels in policy learning and video generation. Our results show that this does not carry over by default to planning with an end-to-end JEPA world model. On the same unlabelled data, a LAPO-style IDM ends at the level of the constant action ablation, which uses no latent actions at all, on both PushT (23.3% vs 26.7%) and TwoRoom (76.7% vs 76.0%), where LaFreeWM reaches 60.7% and 99.3% (Table 6, Appendix C.4). On PushT the same holds for our adaptation of Garrido et al. (16.0%) and a continuous latent action (18.0%). So the result is not obtained by adding an existing latent action objective to LeWM.
The second finding is the coupling between encoder and IDM. Concurrent work [1, 2, 3] shows that an inverse-dynamics loss on the real actions keeps the encoder of a JEPA world model sensitive to actions, which helps planning. These methods need the label of every transition. We find a similar effect without labels: when the encoder is trained through the latent action IDM, success goes from 40.7% to 59.3%, and the encoder responds more clearly to the pusher position (Figure 4 in the appendix). This is also why we think the contribution is more than label efficiency: with 10% of the labels, LaFreeWM is above LeWM trained on all labels in all four environments (PushT 60.7 vs 30.0, TwoRoom 99.3 vs 92.7, Reacher 70.7 vs 62.7, Cube 82.0 vs 77.3; Table 1).
Third, the continuous variant has a 60 times lower prediction error and still plans worst, because the IDM can pass the future observation on to the predictor. Lower prediction error is therefore not a reliable signal when latent actions are learned together with the world model.
[1] Ivashkov et al. Sensorimotor World Models. arXiv:2606.20104, 2026. [2] Zhang et al. Delta-JEPA. arXiv:2606.31232, 2026. [3] Strohm et al. Keeping JEPA World Models Plannable When Little of the Frame Moves. arXiv:2610.03137, 2026.
Thank you for your review.
Weaknesses: We understand the assumption, as previous approaches like LeWorldModel do sample from the training dataset. We also found that the difference in performance between training and test start and goal pairs is insignificant. However, in our case the start and goal positions are not sampled from the training dataset. We agree that this should be clarified in the paper.
LeWorldModel rolls out the world model for 5 prediction steps. We use 15, three times as many, and prediction errors compound over a longer rollout. SAGE [1] shows that the planning success of LeWM is sensitive to this horizon. On PushT it drops from 89.3% at the standard goal offset of 5 prediction steps to 56.0% at 10 steps and 8.7% at 20 steps. This could explain the lower performance of LeWorldModel and the other models in our evaluation. We agree that 15 steps is not necessarily long horizon and will drop that wording. The low numbers are concentrated at the hardest setting, h = 15 on PushT. At shorter horizons PushT is solved.
We also found the problem with the TwoRoom environment, an increased sensitivity to the variance regularisation (SIGReg) loss term. After re-evaluating all models, LaFreeWM reaches 96.7 ± 1.9 and 99.3 ± 0.9 on TwoRoom with 1% and 10% labels, against 36.7 ± 5.0 and 85.3 ± 2.5 for LeWM. We now outperform LeWM clearly in 3 out of 4 environments (with Cube being less significant). We believe that this and the ablations together justify the additional complexity.
We agree that label-only LeWM is expected to suffer with 1% of the labels, and that scarcity is the setting we target. The informative comparison is with methods that see the same unlabelled data. Pretraining on it with a constant action gives 26.7% on PushT at 10% labels, against 59.3% for LaFreeWM (TwoRoom 76.0 vs 99.3, Reacher 67.3 vs 70.7, Cube 80.0 vs 82.0), and Garrido et al. gives 13.3% at 1% labels, against 33.3%. With 1% labels LaFreeWM is also on par with LeWM trained on all labels on PushT (33.3 vs 30.0) and Reacher (67.3 vs 62.7).
We agree that the ablation text should better explain and motivate each variant, and we will revise it. We also expanded the ablations to the 3 other environments, and the change in success relative to the full model (500 test episodes) is shown below and will be put in the appendix.
Questions: Question 1: L_pred is shown in Figure 2(a) and appears in the full objective (Eq. 7) as the next-CLS MSE, but we never wrote out its definition. L_pred is the main learning loss of the latent world model (encoder, dynamics model and IDM). It minimizes the distance between the prediction of the dynamics model and the future latent states. L_LAM, in contrast, is purely for action shaping and is only seen by the IDM. It is the prediction loss of a separate network that takes the latent and the latent action as input and outputs the next latent.
Question 2: The two look similar, but the difference is in what each predictor gets to see. The LAM decoder receives the current state together with the latent action and predicts the next latent, so the latent action only has to be informative in combination with the state. The JEPA predictor receives only the latent action, without the state, and has to predict the normalized difference between the two latents, so the latent action on its own is encouraged to carry the direction of change. The first one encourages predictability of the future, the other one predictability of change. We also measured this on PushT, the average cosine similarity between the actual direction of latent state change and the mean direction of its assigned code is 0.58 for LaFreeWM and 0.28 for the No-JEPA ablation, so with the JEPA loss the code alone predicts the direction of change much better. On planning, the effect is small but consistent, removing the JEPA loss lowers success in all four environments, by 0.2 to 2.6 points, and lowers effective codebook use from 61.5 to 45.0 codes.
Question 3: Indeed, if you were able to perfectly predict the actions, you would expect more or less the same results as LeWM. The Separate IDM ablation shows that decoupling the encoder brings us closer to the LeWM performance. One hypothesis is that coupling the encoder with the IDM acts as an additional regularisation on the encoder, encouraging representations that encode more of the action information, which makes it easier for the dynamics model to predict the future latent. Figure 4 in the appendix indeed shows that the LaFreeWM encoder responds more clearly to the position of the pusher than the LeWM encoder does.
References [1] Cheng et al. SAGE: Subgoal-Conditioned Action Generation for Latent World Model Planning. arXiv:2607.17973, 2026.