- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that latent world models use distances in latent space for scoring candidates during planning. This latent-space distance may not capture the physical quantities required to determine success precisely enough to separate successful candidates from the failing ones. To rectify this, the paper adds an auxiliary term to the loss function during training only, using the success-criterion quantities as the training target. As a result, the success rate increases by 3-5% for the manipulation tasks considered, and a per-stratum analysis shows that supervising some quantities can help some trials while hurting others.
Critical strengths
- The paper clearly states the proposed contributions and tests their approach on different manipulation tasks. The idea is simple and plausibly useful, and it doesn't add any inference cost.
- Figure 1 clearly explains the entire setup.
- The paper also clearly points out its limitations and describes the future work adequately.
Critical weaknesses
- The whole premise of goal-conditioned world models like LeWM and DINO-WM is to develop a model trained without any task-specific knowledge. Still, with the proposed approach, it became task-centric again. A different task will require separate training from scratch, which defeats the entire purpose. The frozen V-JEPA 2 suggests that the bottleneck lies elsewhere, as the readout improved while the success rate fell.
- The paper mentions that "the deviation does not vanish with representation size or pretraining," however, Section 2.2 mentions that median hand error decreased from 3.10 cm to 2.10 cm by moving from DINOv2-S to DINOv2-L, which suggests that increasing the representation size does help. Appendix B reports a V-JEPA 2 predicted latent hand error of 1.18 cm, which is already within the tolerance limit.
- The paper states that the improvements are statistically significant using the McNemar test, but it only uses a single seed. McNemar tests capture trial-to-trial variance, not training variance. It doesn't indicate which method is better. In fact, Appendix D shows the seed variance is as large as the effects being claimed. The paper does cite Agarwal et al., 2021, but doesn't follow it properly.
- The main suggestion is to supervise every quantity, but that's not the case with the "coffee" task.
Questions for authors
- What is the difference between readout error and coarse readout? Adding a formal definition to the paper can help clarify it.
- In the fitted ridge regression from the latent vector, what's the probe's training set size and regularisation strength? Are the frozen encoder---the CLS tokens or the pooled patch features for probing the representation purposes?
- Why are the OGBench
cube-(double/triple/quadruple)-v0tasks not considered? These tasks align better with the proposed approaches and, in fact, involve different downstream operations, such as placing in order and stacking. How will these different tasks for the same environment be handled with the current setup? - Why are navigation tasks not considered? The success criterion for the navigation tasks is identical in terms of goal tolerance, as the method handles. The reachability problem will affect the baseline equally, so the comparison will stay paired. Running them will help show that the gain survives even when the cost's geometry is also wrong, or reveal otherwise. this can potentially make the claims stronger.
Minor Suggestions:
- Table 9 is not formatted properly.
- The reacher exclusion seems avoidable. We can encode the joint angles as sines and cosines, which would remove the wrapping issue.