- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 3(+2)
Read review
Paper summary
This work analyses the latent forward-prediction performance on several pre-trained representations (mostly JEPA variants). The main analysis metric is comparing the distance between the latent of the expert demonstration, and an action-conditioned forward model trained on this representation. Importantly, only the action-conditioned model is trained on the task distribution, not the representation itself. Findings show that this autoregressive prediction fails to generate latent trajectories that are close to the latent demonstration trajectory.
Critical strengths
I really liked the motivation to understand the effect of pre-trained representations on decision-making, and the paper overall follows nice experiments. This paper would be quite interesting and strong if the evaluation process were different (see below). There are many analyses probing different axes of the representations. The paper is mostly well-organized, though some parts could benefit from further clarification, especially Section 3.
Critical weaknesses
I think the critical problem is drawing planning- or decision-making-related conclusions while the analyses measure the representation manifold. While one could expect the predicted trajectory to lie near the expert trajectory (in the latent space), for decision-making, this is not the primary concern. These self-supervised representations show most of their benefits when they are combined with a policy (e.g., an RL agent, or a behavior-cloned policy), which is also shown in Section 7 with the VLA experiments where the VLA chooses the actions to take. For instance, an RL agent trained on this representation space might take good actions even when the latent observation trajectories don't overlap. I don't see a sound reasoning on why we should treat the distances as a proxy for decision-making or planning performance. Maybe it makes sense in some form that I don't immediately see. If there's such a case, then it should be explicitly derived in the paper---for instance, a derivation on how this translates to the planning performance would be good. For instance, applying t-SNE (or even random numbers as new dimensions) would destroy the Euclidean distance interpretation, but you could still train an RL agent on it.
--- AFTER REBUTTAL ---
I understand that the paper's scope is planning with the learned representations. In that scenario, I agree with their experiments, even though, I still believe the usefulness of the learned representations could also be analyzed with a trained actor, or a mix of planning + RL.
Questions for authors
How does the latent distance to the expert trajectory formally connect to the model-free or model-based performance of an agent?
6 comments
Response to Reviewer BxNa (Weaknesses part 2)
Weaknesses (continued)
(c) Rewards and a policy can help, but they need the extra data that the family we study avoids. A policy is strong where its training data exist. To show how much these data help, we trained a goal-conditioned behaviour-cloning policy (BC) on the same frozen V-JEPA 2.1 features, using expert actions from the 13 training tasks. On those tasks it beat every planner (0.92–0.96, against 0.54 for latent distance planning with a long rollout). Without such data, the RL family does not work or performs poorly. DINO-WM [9] trained TD-MPC2 on the same reward-free data, and it reached 0.00 success, which the authors attribute to "the lack of reward signal". On 10 new tasks, however, the BC policy failed to generalise and reached only 0.13–0.16, below latent distance planning (0.35). PLDM [7] also finds that "model-free RL benefits most from large amounts of high-quality data, whereas model-based planning generalizes better to unseen layouts". Rewards or a policy can therefore be a large advantage, but only because the model uses extra data about the task. This goes against the approach of the family of models we study in this paper, so a comparison with the RL family would not be fair and is outside the scope of the paper.
Section 7 shows that the two can still work together. An existing VLA [3] proposes candidate actions, and the world model, off the shelf, imagines the outcome of each and chooses among them with the same latent distance. Nothing is trained or tuned for this.
Response to Reviewer BxNa (Q1 & Remarks)
Q1. How does the latent distance formally connect to agent performance?
Thank you for this well-thought-out question. We will answer your question in several points so that it is easier to read.
[Clarification] The latent distance to the expert trajectory comes from the convention of the family of world models we study, but it is not the essence of the plannable range. We rely on the planning cost chosen by the authors of the models we have tested. These are policy-free, reward-free world models, such as V-JEPA 2-AC [1], DINO-WM [9], PLDM [7] and HWM [8], and their planning cost is the latent distance to a goal image (W1(a)). The idea of the plannable range is not to measure the latent distance to the expert per se, but to measure how well these world models tell good actions from random ones, based on their current planning convention. These models happen to use the latent distance, and therefore the latent distance to the expert becomes a proxy for whether the world model knows which actions are good to take.
These world models use the latent distance for a good reason. As we mentioned in W1(b), it has clear strengths, and these models have achieved good results with it, especially when the rollout reaches the target. For example, when the planner imagines far enough to reach the target, closed-loop success is 0.76. Its weakness shows when the planner imagines only a few steps at a time, because the cost is greedy (Sec. 5.5).
The main answer to your question depends on the type of agent. In a model-free RL agent, the policy chooses the actions directly, so the distance affects performance only if it is used to train the policy, for example as a reward. Model-based RL agents, such as Dreamer [4] and TD-MPC2 [5], also learn a world model, but their actions come from a policy, or from planning with a reward and value function learned for the task, so the latent geometry matters less. We make no claim about these two cases. The world models we study are also model-based, but they use no RL. Their planner executes the action sequence whose imagined outcome is closest to the goal in latent space. The philosophy of these models is to use no policy or rewards. The latent distance is therefore the cost of the planner, and it directly affects the performance of the agent. This is why the plannable range is defined with the latent distance.
Showing both the strengths and the weaknesses of the latent distance as a planning cost is one of the contributions of our paper. Improving this cost is a new direction of research, and we will make this clearer in the revised paper. However, the latent distance is not easy to replace, because a new planning cost must be learned without expert data or rewards, unlike the costs of RL world models. The learned costs we tried did not beat it on new tasks (W1(b)). TEMPO [6], which appeared after our submission, is one of the first steps in this direction. A policy can also help, but it needs extra labelled data (W1(c)).
We believe these answers address your main concern, including the soundness of our evaluation. We would appreciate it if you would re-evaluate the paper and consider raising your score. Its findings, and the weaknesses it reveals, should be of interest to the community, and especially to researchers working on world models that plan without a policy or reward.
Response to Reviewer BxNa (Weaknesses part 1)
Thank you for your review. We are grateful for your time and well-thought-out comments. We are also glad you "really liked the motivation to understand the effect of pre-trained representations on decision-making", and that you found the paper "follows nice experiments" and is "mostly well-organized". You wrote that the paper "would be quite interesting and strong if the evaluation process were different". Our evaluation process was designed carefully around how these world models plan. Below we explain why this evaluation fits the world models we study.
W1. Can latent distances support conclusions about planning?
Please note, there are two separate families of latent world models. Our study focuses on the emerging family, which plans with a latent distance cost, and the other is outside of this paper's scope. The RL family uses a policy or an RL agent, as in Dreamer [4] and TD-MPC2 [5]. Besides the world model, it learns a reward model and a policy or value function for each task or environment, so it needs task rewards or expert actions for every task. The policy chooses the actions, so the geometry of the latent space matters less. This is relevant to your t-SNE example, where a policy can still learn good actions on a distorted representation.
On the other hand, the family we study learns no policy, needs no reward and is gaining substantial traction (V-JEPA 2-AC [1], DINO-WM [9], NWM [2], PLDM [7], HWM [8]). These models learn only how actions change the world, in latent space, so they need no expert data. The world models in our paper were trained on random-action episodes only (App. B.1). At test time they imagine the outcomes of many candidate action sequences and execute the one whose imagined future is closest to a goal image (the latent distance, by design). This direction is promising because rewards and expert actions are hard to obtain outside simulated benchmarks, and because it can transfer to new settings. For example, V-JEPA 2-AC [1] was deployed "zero-shot on Franka arms in two different labs … without any task-specific training or reward", and DINO-WM [9] plans zero-shot "without expert demonstrations, reward modeling, or pre-learned inverse models". Our paper evaluates this family, and our evaluation follows from how it plans.
We note this in abstract, Sec. 2 and App. A.1. However, we will make this distinction between the world-model families and the scope of the paper clearer in Section 2.
(a) [Clarification] We measure whether the world model prefers the expert's actions to random ones, using the cost its authors designed it to plan with (latent distance). In the family we examine, the latent distance to the goal is the cost that chooses the actions. Thus, the latent distance cost was not 'our' choice. It is a design choice made by the authors of these models, and we follow it. For example, V-JEPA 2-AC uses an L1 distance, DINO-WM and PLDM use L2, HWM uses latent matching, and the latent distance we use (cosine distance on normalised features) equals half the squared L2 distance. We therefore compare the latent distance between the goal and the imagined outcome of the expert's actions with the same distance for random actions, which is the choice the planner makes at every step. Evaluating the latent distance cost is part of our paper's aims, and finding its strengths and weaknesses is central to the paper. The measurement shows how far ahead these world models can judge good actions with the planners they are normally used with. Using rewards or a policy would require extra data that this direction of world models does not use.
(b) The latent distance is a good cost, with some weaknesses. It needs no data, labels or rewards, so it works zero-shot on different tasks, which is what made the zero-shot results above possible. In our tests it works well when the imagined rollout reaches the target. For V-JEPA 2.1, the world model then picks the executed actions out of 16 candidates 98% of the time, 20 steps ahead (chance is 6%, Fig. 2(a), App. G), and the cost ranks the expert above all 100 random alternatives in every window (Table 4). The predictions therefore stay close to the demonstration in this setting, which differs from the summary of the review. The latent distance also transferred better than the learned costs we tried. A VIP-style head, trained on expert demonstrations to predict temporal distance, gave lower closed-loop success than the latent distance (0.188 vs 0.397 with subgoals 20 steps ahead), even on the tasks it was trained on. Its weakness shows when the planner imagines only a few steps at a time. One of the paper's findings is that the problem is not always long-horizon prediction error. It can instead be the latent distance cost that needs to improve, still without rewards or a policy. TEMPO [6], which appeared after our submission, is one of the first steps in this direction (Q1).