- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a failure of JEPA-style world models in which the representations vary across images but fail to preserve the positions of objects needed for planning. The authors add a small auxiliary network during training that predicts a target formed by converting object bounding-box coordinates into sine and cosine features and summing over interchangeable objects. This Fourier target is unchanged when the objects are reordered. The auxiliary losses supervise both representations of observed frames and predicted future representations, using simulator-derived boxes in the main experiments. The extra network is removed during standard planning. At horizon , success on the bouncing-ball tasks rises from roughly to . Smaller improvements are reported on Push-T, Two Room, Reacher, and Cube, and the evaluation also includes experiments with reduced training datasets.
Critical strengths
-
The bouncing-ball results provide the clearest evidence for the method. Section examines a task in which planning requires the positions of both the controlled ball and the autonomously moving balls. Table shows that, at horizon , success increases from to with one environmental ball, and from to with three. The linear and MLP probes in Appendix also support the diagnosis that the baseline poorly encodes the controlled ball's position. Together, these results connect the representation problem to an actual planning failure and demonstrate a substantial benefit from object-position supervision.
-
The target handles interchangeable objects without adding work to the standard planner. Equation in Section sums Fourier features of object bounding boxes, so reordering the objects leaves the target unchanged. Sections apply the auxiliary losses to both encoded and predicted representations, then discard the shallow prediction head for standard planning. Thus, the method encourages the world model to retain object geometry without requiring an object detector or additional network during inference. This keeps the change to the existing planning pipeline small.
Critical weaknesses
-
The experiments make a stronger case for object supervision than for Fourier targets specifically. The main comparisons in Section give the proposed method geometric labels that the LeWM baseline does not receive. These comparisons establish that added supervision can help, but do not isolate the Fourier construction. Appendix , Table , partly addresses this issue: in the task with three environmental balls, the non-Fourier variant without object matching reaches 76.8% success, compared with for Fourier supervision and for LeWM. Ordinary auxiliary supervision therefore accounts for much of the improvement. Because auxiliary training objectives and invariant feature aggregation already have established precedents, the paper needs stronger evidence that its particular Fourier target provides a reproducible advantage over simpler alternatives trained with the same labels.
-
The reported uncertainty does not establish robustness across model training runs. Section and the captions of Tables and specify variation across five CEM evaluation seeds. These repetitions characterize evaluation variability, but do not quantify the variation arising from independently training the encoder and predictor. This is especially consequential for the smaller gains in Table , such as to in Two Room and to in Cube. Independent training runs are needed to establish the reliability of these gains and determine whether the severe bouncing-ball failure is a systematic property of the baseline.
Questions for authors
- How many independently trained encoders and predictors underlie each comparison in Tables and , and what variation occurs across those training runs?
- What frequency sampling distribution, scale or bandwidth, and random seed are used to construct in Eq.? Is the same matrix used across independently trained models?
- Which training-data count produces the Push-T results of and in Table ? Section specifies , whereas Figure 4 associates these results with .
6 comments
Reply to CW2
We thank the reviewer for raising this important point. We agree that variability across evaluation seeds and independently trained models represents two distinct sources of uncertainty. In response to the reviewer's suggestion, we have conducted additional experiments with 3 independently trained models for both LeWM and our method, using different training seeds while keeping the training datasets, model architectures, and evaluation protocols fixed.
These additional experiments demonstrate the consistency of our results across independent training at n = 1 and n = 3. At n = 6, the collapse of LeWM depends on the training seed (H = 4 success of 1.9%, 76.6% and 75.0%); we analyze how this collapse depends on the number of balls in the new Appendix A.12. The revised Tables 1–3 now report the mean and standard deviation over the three training seeds, the per-seed values are in Appendix A.13, and we have clarified the distinction between training seeds and evaluation seeds.
Reply to Q1
We thank the reviewer for requesting this clarification. Three independently trained encoder–predictor models (training seeds 0, 1 and 2) were used for each method and environment, as given in the response to CW2. The revised Tables 1–3 report the mean and standard deviation over these three models, each evaluated with five CEM seeds, and Appendix A.13 lists the per-seed values
Reply to CW1
We thank the reviewer for this insightful comment. We agree that physical supervision itself accounts for a substantial part of the improvement. However, this does not diminish the significance of our contribution, as addressing the physical representation laziness failure mode identified in our work is itself a central contribution of this paper. We would like to clarify that the benefits of our method can be understood at three levels.
First, physical supervision addresses the physical representation laziness failure mode identified in our work by encouraging the model to preserve task-relevant physical information. This accounts for the major improvement in planning performance over the LeWM baseline.
Second, permutation invariance is important when supervising indistinguishable objects, since their ordering is arbitrary. Our original Appendix A.6 (Table 7) already shows that direct bounding-box supervision without matching achieves 76.8% planning success in the three-ball task, while Fourier symmetrization improves this to 81.2% under the same bounding-box supervision. Over three training runs the difference is larger: Fourier symmetrization reaches 81.2%, 80.9% and 79.0%, while supervision without matching collapses on two of the three runs (76.8%, 1.7% and 2.9%). This demonstrates an additional benefit from our permutation-invariant target construction beyond physical supervision alone.
Third, Fourier symmetrization provides a natural and theoretically motivated way to achieve permutation invariance. As explained in Sec. 4.2, an unordered object configuration can be fully characterized by its empirical distribution, whose Fourier transform systematically captures configuration information at different frequencies. Unlike naive symmetrization, such as summing object coordinates, Fourier features can preserve richer information about the object configuration and can be systematically generalized to more complex tasks.
To further clarify these three levels of contribution, we have added new ablations comparing Fourier symmetrization with simpler permutation-invariant targets under identical physical supervision, which demonstrates the benefits from the Fourier symmetrization .
We have also revised Sec. 4.2 and Appendix A.6 to emphasize this distinction and better explain the motivation and contribution of our Fourier construction.