acceptodds
Under review as a conference paper at ICLR 2027

Spectral-Target Physical Latent Structuring for JEPA-Style World Models

Abstract

Latent world models have become increasingly popular as a method to predict and plan in latent space rather than pixel space. Recent architectures, such as LeWorldModel (LeWM), jointly train the encoder and predictor using regularization techniques like SIGReg to prevent representation collapse. Even with such regularization preventing representation collapse, we identify a new world model failure mode of physical representation laziness, particularly noted in highly dynamic environments. For these lazy cases, the learned latent states do not collapse but nonetheless fail to represent key physical properties, causing ubiquitous downstream planning failure. To resolve this issue, we propose training-time auxiliary supervision with a lightweight "Fourier auxiliary head", which enforces physically-informed structuring of the latent space with no additional inference-time cost and can be generalized to any environment. Experimentally, we show that the auxiliary head substantially improves planning success rates in dynamic environments where the baseline LeWM exhibits physical representation laziness. It also leads to modest improvements in other environments, even when the baseline does not exhibit physical representation laziness. We further observe superior planning performance being accompanied by higher latent space correlations with key physical properties, indicating both the ability of our method to physically structure latent states and the potential planning-side benefit to the learned representation being physically structured. Finally, we ablate across various training data amounts, showing that in low-data regimes, auxiliary supervision is particularly impactful in increasing success rate. These findings support the use of our Fourier auxiliary head method to improve both overall success rate and data efficiency, while avoiding representation laziness in latent world models.

Primary area:
applications to robotics, autonomy, planning
Keywords:
LeWorldModel, Physical representation laziness, Fourier auxiliary head

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SvRkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a failure of JEPA-style world models in which the representations vary across images but fail to preserve the positions of objects needed for planning. The authors add a small auxiliary network during training that predicts a target formed by converting object bounding-box coordinates into sine and cosine features and summing over interchangeable objects. This Fourier target is unchanged when the objects are reordered. The auxiliary losses supervise both representations of observed frames and predicted future representations, using simulator-derived boxes in the main experiments. The extra network is removed during standard planning. At horizon H=4H=4, success on the bouncing-ball tasks rises from roughly 1−3%1-3\% to 67−92%67-92\%. Smaller improvements are reported on Push-T, Two Room, Reacher, and Cube, and the evaluation also includes experiments with reduced training datasets.

Critical strengths
  1. The bouncing-ball results provide the clearest evidence for the method. Section 33 examines a task in which planning requires the positions of both the controlled ball and the autonomously moving balls. Table 11 shows that, at horizon H=4H=4, success increases from 1.1%1.1\% to 92.4%92.4\% with one environmental ball, and from 2.7%2.7\% to 81.2%81.2\% with three. The linear and MLP probes in Appendix A.11A.11 also support the diagnosis that the baseline poorly encodes the controlled ball's position. Together, these results connect the representation problem to an actual planning failure and demonstrate a substantial benefit from object-position supervision.

  2. The target handles interchangeable objects without adding work to the standard planner. Equation (3)(3) in Section 4.24.2 sums Fourier features of object bounding boxes, so reordering the objects leaves the target unchanged. Sections 4.3−4.44.3-4.4 apply the auxiliary losses to both encoded and predicted representations, then discard the shallow prediction head for standard planning. Thus, the method encourages the world model to retain object geometry without requiring an object detector or additional network during inference. This keeps the change to the existing planning pipeline small.

Critical weaknesses
  1. The experiments make a stronger case for object supervision than for Fourier targets specifically. The main comparisons in Section 55 give the proposed method geometric labels that the LeWM baseline does not receive. These comparisons establish that added supervision can help, but do not isolate the Fourier construction. Appendix A.6A.6, Table 77, partly addresses this issue: in the task with three environmental balls, the non-Fourier variant without object matching reaches 76.8% success, compared with 81.2%81.2\% for Fourier supervision and 2.7%2.7\% for LeWM. Ordinary auxiliary supervision therefore accounts for much of the improvement. Because auxiliary training objectives and invariant feature aggregation already have established precedents, the paper needs stronger evidence that its particular Fourier target provides a reproducible advantage over simpler alternatives trained with the same labels.

  2. The reported uncertainty does not establish robustness across model training runs. Section 5.1.15.1.1 and the captions of Tables 11 and 33 specify variation across five CEM evaluation seeds. These repetitions characterize evaluation variability, but do not quantify the variation arising from independently training the encoder and predictor. This is especially consequential for the smaller gains in Table 33, such as 84.0%84.0\% to 86.0%86.0\% in Two Room and 66.4%66.4\% to 70.0%70.0\% in Cube. Independent training runs are needed to establish the reliability of these gains and determine whether the severe bouncing-ball failure is a systematic property of the baseline.

Questions for authors
  1. How many independently trained encoders and predictors underlie each comparison in Tables 11 and 33, and what variation occurs across those training runs?
  2. What frequency sampling distribution, scale or bandwidth, and random seed are used to construct WW in Eq.(3)(3)? Is the same matrix used across independently trained models?
  3. Which training-data count produces the Push-T results of 95.2%95.2\% and 91.2%91.2\% in Table 33? Section 5.15.1 specifies 500k500k, whereas Figure 4 associates these results with 1.9M1.9M.
6 comments
AuthorsOpenReview ↗

Reply to CW1

We thank the reviewer for this insightful comment. We agree that physical supervision itself accounts for a substantial part of the improvement. However, this does not diminish the significance of our contribution, as addressing the physical representation laziness failure mode identified in our work is itself a central contribution of this paper. We would like to clarify that the benefits of our method can be understood at three levels.

First, physical supervision addresses the physical representation laziness failure mode identified in our work by encouraging the model to preserve task-relevant physical information. This accounts for the major improvement in planning performance over the LeWM baseline.

Second, permutation invariance is important when supervising indistinguishable objects, since their ordering is arbitrary. Our original Appendix A.6 (Table 7) already shows that direct bounding-box supervision without matching achieves 76.8% planning success in the three-ball task, while Fourier symmetrization improves this to 81.2% under the same bounding-box supervision. Over three training runs the difference is larger: Fourier symmetrization reaches 81.2%, 80.9% and 79.0%, while supervision without matching collapses on two of the three runs (76.8%, 1.7% and 2.9%). This demonstrates an additional benefit from our permutation-invariant target construction beyond physical supervision alone.

Third, Fourier symmetrization provides a natural and theoretically motivated way to achieve permutation invariance. As explained in Sec. 4.2, an unordered object configuration can be fully characterized by its empirical distribution, whose Fourier transform systematically captures configuration information at different frequencies. Unlike naive symmetrization, such as summing object coordinates, Fourier features can preserve richer information about the object configuration and can be systematically generalized to more complex tasks.

To further clarify these three levels of contribution, we have added new ablations comparing Fourier symmetrization with simpler permutation-invariant targets under identical physical supervision, which demonstrates the benefits from the Fourier symmetrization .

We have also revised Sec. 4.2 and Appendix A.6 to emphasize this distinction and better explain the motivation and contribution of our Fourier construction.

AuthorsOpenReview ↗

Reply to CW2

We thank the reviewer for raising this important point. We agree that variability across evaluation seeds and independently trained models represents two distinct sources of uncertainty. In response to the reviewer's suggestion, we have conducted additional experiments with 3 independently trained models for both LeWM and our method, using different training seeds while keeping the training datasets, model architectures, and evaluation protocols fixed.

These additional experiments demonstrate the consistency of our results across independent training at n = 1 and n = 3. At n = 6, the collapse of LeWM depends on the training seed (H = 4 success of 1.9%, 76.6% and 75.0%); we analyze how this collapse depends on the number of balls in the new Appendix A.12. The revised Tables 1–3 now report the mean and standard deviation over the three training seeds, the per-seed values are in Appendix A.13, and we have clarified the distinction between training seeds and evaluation seeds.

AuthorsOpenReview ↗

Reply to Q1

We thank the reviewer for requesting this clarification. Three independently trained encoder–predictor models (training seeds 0, 1 and 2) were used for each method and environment, as given in the response to CW2. The revised Tables 1–3 report the mean and standard deviation over these three models, each evaluated with five CEM seeds, and Appendix A.13 lists the per-seed values

Reviewer_gfB4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how JEPA-style world models can miss information needed for planning, even when their representations do not collapse. It adds a small auxiliary head to predict summed Fourier features of object bounding boxes from encoded and predicted states. The head can be removed during inference. Experiments show large planning improvements on bouncing-ball tasks and smaller gains on other control benchmarks. Physical-state probes also help assess what information the models preserve. Together, these results suggest that geometric supervision can improve representations for planning.

Critical strengths
  1. The paper shows that avoiding representation collapse does not necessarily preserve information needed for planning. The strong improvements on the bouncing-ball tasks and the physical-state probes support this finding and the proposed solution.
  2. The method is simple to add during training, and the auxiliary head can be removed during inference. The ablations on supervision targets, matching methods, and noisy bounding boxes also help explain its behavior.
Critical weaknesses
  1. Direct bounding-box supervision already performs well: Table 7 reports 76.8% success on the three-ball task, compared with 81.2% for Fourier supervision. The additional improvement is promising, but the specific benefit of the Fourier design could be better explained.
  2. The paper does not clearly report whether the improvements remain consistent across independently trained models, especially for the smaller gains on the other benchmarks.
Questions for authors
  1. Do the experiments help separate the benefit of Fourier features from the benefit of removing object-order dependence? Is the improvement over direct supervision in Table 7 also consistent across independent training runs?

  2. Appendix A.6 explains the poor performance through unstable object assignments. However, Figure 7 only shows results on Push-T. Did you observe similar instability on the three-ball task, where Hungarian matching performs much worse?

3 comments
AuthorsOpenReview ↗

Reply to Q1 (CW 1 and 2)

We thank the reviewer for this insightful question. Fourier symmetrization is a principled way to achieve permutation invariance while preserving information about the object configuration. As explained in Sec. 4.2, an unordered set of objects can be fully characterized by its empirical distribution, which preserves the complete object configuration without relying on object ordering and is also widely used in physics to describe systems of identical particles. However, since the empirical distribution is expressed in terms of Dirac delta functions, it is not directly suitable as a finite-dimensional computational target. Taking its Fourier transform provides a natural and systematic way to characterize the distribution through its components at different frequencies. With sufficiently rich frequency sampling, increasingly detailed information about the object configuration can be captured. In contrast, naive symmetrization, such as summing bounding-box coordinates ibi,captures only limited aggregate information and cannot distinguish many different object configurations and may only work for simple tasks and does not have the ability to generalize.

To further clarify this point, we have expanded the discussion in Sec. 4.2 and added a new ablation in the Appendix comparing naive summation with Fourier symmetrization under the same bounding-box supervision. The results show that Fourier symmetrization achieves better probes and MSE, providing additional empirical support for its advantage beyond permutation invariance alone.

Furthermore, as recognized by the reviewer, our original Appendix A.6 (Table 7) already compares Fourier symmetrization with direct bounding-box supervision and Hungarian matching. In the three-ball task, Fourier achieves 81.2% planning success, compared with 76.8% for direct supervision and 1.7%/2.6% for frame-wise/sequence-wise Hungarian matching. These results further support the effectiveness of Fourier symmetrization as a principled approach to handling permutation invariance in object configurations.

Across 3 independent training runs on the three-ball task, Fourier symmetrization is consistent (81.2%, 80.9% and 79.0%), whereas direct supervision without matching collapses on two of the three runs (76.8%, 1.7% and 2.9%). We have added these runs to Appendix A.6.

AuthorsOpenReview ↗

Reply to Q2(CW 1)

We thank the reviewer for this thoughtful question. Yes, we indeed observe similar instability on the three-ball task. We have now added this figure to Appendix A.6.

We sincerely thank the reviewer for their insightful and constructive feedback. We believe that our clarifications, together with the existing and newly added ablations, further strengthen the evidence supporting our method. We hope these responses fully address the reviewer's concerns and respectfully encourage the reviewer to consider a higher rating in light of the additional evidence and improvements.

Reviewer_ZeQJOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper identifies “physical representation laziness” in JEPA-style world models, where non-collapsed representations fail to preserve task-relevant physical information. It proposes a lightweight auxiliary head supervised by permutation-invariant Fourier features of object bounding boxes. Experiments show substantial planning gains in bouncing-ball environments and smaller improvements on additional benchmarks, without additional inference-time cost.

Critical strengths
  1. The lightweight auxiliary head handles exchangeable objects through permutation-invariant Fourier targets and improves planning without additional inference-time cost.

  2. The bouncing-ball experiments clearly demonstrate that preventing representation collapse is insufficient for effective planning, with the proposed method delivering substantial gains across different numbers of balls and planning horizons.

Critical weaknesses
  1. Motivation: The paper does not clearly explain what qualifies as a “physical representation” or when a representation should be considered “lazy.” A low correlation between position differences and latent distances does not establish whether the model failed to learn position information or learned it in a way that this distance measure does not capture. Clearer definitions and evaluation criteria are therefore needed.

  2. Methodological novelty: the role of the Fourier design is unclear. The proposed method uses additional bounding-box supervision that the LeWM baseline does not receive. It is therefore unclear whether the gains mainly come from this extra supervision or the Fourier design itself. Comparisons with direct position or bounding-box prediction and other permutation-invariant targets, using the same supervision, are needed to establish its advantage.

  3. Experimental design: insufficient evaluation of component contributions and supervision quality. The presented experiments do not clearly separate the contributions of encoder-

side and predictor-side auxiliary supervision. Comparisons between supervising only encoded states, only predicted states, and both are needed to establish the necessity of each branch. In addition, the method relies on bounding-box supervision, but its sensitivity to position errors, size errors, and missed detections remains unclear. Evaluating these conditions would help establish its robustness and applicability to real-world visual data.

Questions for authors
  1. Definition and diagnosis: How do you define “physical representation” and determine whether it is “lazy”? Can decoding probes distinguish missing position information from information that is present but not captured by latent Euclidean distances?

  2. Contribution of the Fourier design: Under the same bounding-box supervision, how does the Fourier target compare with direct position or bounding-box regression and alternative permutation-invariant targets? What evidence attributes the gains specifically to the Fourier construction?

  3. Component contributions: How does performance change when supervising only encoded states, only predicted states, or both? Does each supervision branch provide an additional benefit?

  4. Supervision quality: How sensitive is the method to bounding-box position errors, size errors, and missed detections? Does it retain its benefits when using detector-generated boxes rather than ground-truth annotations?

5 comments
AuthorsOpenReview ↗

Reply to Q1 (CW1)

We thank the reviewer for this question. We would like to clarify that our definition of a physically meaningful latent representation is directly motivated by the downstream planning objective, rather than by an arbitrary representation metric.

First, physical representation laziness is clearly defined in both the Introduction and Sec. 3 as a failure mode in which a non-collapsed latent representation fails to adequately reflect the physical information essential for downstream planning. In response to the reviewer's comment, we have further emphasized this definition in both sections. Below, we explain why the correlation between latent-space distances and physical-state differences provides a natural, task-relevant metric for diagnosing such laziness in the bouncing-ball environment, and how this diagnosis is independently supported by the probing experiments already presented in Appendix A.11 (Table 12).

As explained in Sec. 4.4, LeWM performs planning by minimizing the squared L2 distance between the predicted terminal latent state and the goal latent state: L(zH) =||zH-zg||22 In our task, the control ball must reach a target while avoiding collisions with environmental balls. Successful planning therefore requires the model to capture the relative positions and distances between balls. Since the planning objective is defined by the L2 distance in latent space, it is natural to expect that latent-space distances should meaningfully correlate with differences in physical configurations, particularly the relative positions and distances between balls. Otherwise, minimizing the latent-space distance may not lead to actions that achieve the desired physical configuration while avoiding collisions. We have now clarified this motivation for the correlation metric in Sec. 3. Importantly, we have already performed both linear and nonlinear (MLP) probing experiments in Appendix A.11 (Table 12), precisely as suggested by the reviewer. These experiments provide independent evidence supporting our diagnosis based on latent-distance correlations. For example, in the one-environment-ball task, the Pearson correlation for decoding the control-ball position is only 0.110 with a linear probe and 0.057 with an MLP probe for the baseline LeWM (mean over three training seeds), compared with 0.996 and 0.9998, respectively, for our method. Similar improvements are observed across the other bouncing-ball environments.

In summary, our diagnosis of physical representation laziness is supported by three complementary evaluations: (1) task-motivated latent-distance correlations presented in the main text, (2) linear and nonlinear probing experiments already included in the Appendix, and (3) downstream planning performance. Together, these results provide consistent evidence that our method learns more physically meaningful representations and effectively addresses the representation laziness observed in the baseline.

AuthorsOpenReview ↗

Reply to Q2 (CW2)

We agree that distinguishing the benefits of physical supervision from those of the Fourier construction is important. Our method has two complementary components: introducing task-relevant physical information through auxiliary supervision and constructing permutation-invariant targets for exchangeable objects. The main advantage of the Fourier construction is not simply that it introduces additional supervision, but that it provides a well-defined, permutation-invariant representation of unordered object configurations. As explained in Sec. 4.2, the Fourier target is constructed from the Fourier transform of the empirical measure associated with the object bounding boxes. The full Fourier transform uniquely determines this measure, and therefore the unordered object configuration. Although our finite-dimensional approximation is not guaranteed to be injective, it preserves more information about the configuration than simple aggregate statistics such as summed positions. Importantly, we have already performed comparisons with alternative object-matching strategies in Appendix A.6, Table 7, including direct supervision without matching and Hungarian matching performed either frame-wise or sequence-wise. We have now further considered a comparison between Fourier symmetrization and a trivial sum ibi and added it in Appendix A6. All these results clearly show the benefits from the permutation-invariant representation given by the Fourier construction.

Furthermore, Appendix A.4, Table 5 compares various physical supervision targets, including exact simulator states, compact physical states, bounding boxes, oriented bounding boxes, and targets incorporating centers of mass. The results demonstrate that additional physical information does not automatically lead to better planning performance. For example, in the three-ball task, bounding-box supervision achieves 81.2%, outperforming exact simulator-state supervision (77.2%) and the Compact7 target (55.0%).

These results support two distinct conclusions: physical auxiliary supervision can substantially improve planning, and the choice of target representation and permutation handling is important for obtaining robust improvements, particularly when objects are exchangeable.

AuthorsOpenReview ↗

Reply to Q3(CW3, firstpart)

We thank the reviewer for raising this question. We have already performed an auxiliary-head architecture ablation in Appendix A.5, Table 6, comparing supervision of encoded states only with supervision of both encoded and predicted states. We have added one more ablation about supervising predicted states only in that table in response to the reviewer's question.

These results demonstrate that encoder-side or predictor-side supervision alone already provides substantial improvements over the baseline, while applying auxiliary supervision to the both yields further benefits.

Therefore, the existing and added ablations provide direct empirical evidence for the incremental benefit of predictor-side supervision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.