acceptodds
Under review as a conference paper at ICLR 2027

AnyGoal: On World Model Generalization Beyond Expert Rollouts

13 views

Abstract

The goal of a world model is to capture the dynamics of an environment. This has the great appeal that it allows agents to plan and simulate a course of actions with the help of a world model. For these uses to be reliable, however, the learned dynamics must behave well beyond the specific state–action combinations observed in expert rollouts. In this work, we show that current world models broadly struggle to generalize to new goals in known environments. We explain this from the theoretical perspective of compositional generalization, which allows us to formulate desiderata and conditions of successful generalization. Based on these insights, we construct a new benchmark, AnyGoal. We carefully check that each tasks remain within the factor-wise support of the training data, so that they can be reasonably expected to be solvable by a good world model. By disentangling encoder–predictor failures, generative versus latent planning models, interaction-free versus interaction-rich regimes, and other axes of variation, our results reveal both where current world models can generalize and where major challenges remain for goal-shifted dynamics prediction and planning. This framework and benchmark will provide direction and a progress measure for the goal of building world models that generalize to new planning tasks.

Primary area:
datasets and benchmarks
Keywords:
world model, compositional generalization, benchmark, planning, control

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yqy9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies evaluation of world models, with the goal of separating systematic generalization from behavioral shortcut learning. The authors view the problem from the perspective of compositional generalization. The authors formalize this as a setting where the test set trajectories remains within the Cartesian product of the training marginal supports of the individual latent factors and actions and targets the (state, action) pairs which are absent in the training data. The authors instantiate the framework empirically as a benchmark AnyGoal containing 4 control environments with 8 test variants each containing 400 episodes. For measuring the coverage the authors use some standard measures (coordinate range, distance, etc) and Gaussian KDE for the joint state-action novelty. To demonstrate how AnyGoal can be used in practice, the authors benchmark a few different types of world models based on the rollout MSE as well as CEM based MPC planning success rate. The empirical results on AnyGoal provide a few interesting observations. Even with ground truth physical observations, world models fail to capture interaction dynamics resulting on planning and rollout prediction on contact-rich goal shifts. End-to-end visual models perform even more poorly due to observation level compositional shifts. The paper concludes with a theoretical result on the sufficient conditions for compositional generalization showing that if each latent factor's transition depends only on a sparse subset of local coordinates represented in training then the global compositional error is bounded.

Critical strengths
  • I like the broader question that the paper is exploring and also agree with the compositional generalization framing being appropriate for systematic evaluations. The goal shifted environments which act on specific marginal factors are a clean and intuitive way to test generalization. I also think the experiments decoupling the representation and dynamics failures via oracle state-predictors are quite insightful.
  • I think some of the analysis is also quite insightful. For eg. the counterfactual action-intervention analysis is a useful diagnostic for action sensitivity. The experiment in Section 5.3 also nicely isolates representation geometry effects when visual observations are strictly in-distribution, as well as how rollout MSE and MPC planning success can diverge depending on task tolerance and representation space.
Critical weaknesses
  • There seems to be a critical issue in the empirical results by using KDE for measuring the combinatorial gap. The authors use fit an unregularized Gaussian KDE to vectors in 18 and 33D reporting low joint in-distribution overlap (Fig 4b, Table 4). However, there is no control reported for a held-out test set from the base task itself. In 33D, even an i.i.d. held-out split from the base task would yield near-0% overlap under this estimator. On the other hand in Appendix C, high-dimensional articulated states are sliced into independent 1D marginals for the marginal support check ignoring kinematic configurations and joint coordination. Overall, despite the generally correct formal setting, the actual implementation seems lacking.
  • Another major issue in my opinion is the theoretical result (Prop 1) seems trivial with major assumptions. Assumption 4 provides a learned model with the exact decoupled structure and Assumption 4 ensures every test input is inside the training support, making the result trivial. It is also very directly follows from prior work [1, 2, 3].

[1] Compositional Generalization from First Principles, Wiedemer et al. NeurIPS 2023.

[2] Efficient Reinforcement Learning in Factored MDPs, Kearns and Koller, IJCAI 1999.

[3] Provably Efficient Causal Model-Based Reinforcement Learning for Systematic Generalization. Mutti et al. AAAI 2023.

Questions for authors
  • Could you evaluate a held-out set from the base task with the Gaussian-KDE?
  • According to Table 5 the 10D oracle state for Push-T includes the 4D goal pose (Goal position (x,y), goal orientation). Why is the task goal included in the state input of a dynamics predictor, and what exact values are passed into those 4 coordinates during evaluation on Avoid-T (where the goal is a corner region, not a T-block pose), Stack-T, and Push-T Antigoal? Does removing the 4D goal from the Push-T oracle state change the degradation on Stack-T and Push-T Antigoal?
3 comments
AuthorsOpenReview ↗

Response 1

We thank the reviewer for their detailed feedback and constructive suggestions. Here, we treasure the opportunity to address your concerns and improve the quality of our work.

W1, Q1. Base-task KDE control experiment.

We thank the reviewer for this important methodological consideration. We sampled 400 episodes from the base episodes using each of three random seeds (0/1/2) and used them as held-out test sets. The original number of episodes for each base task is as follows: PushT: 18,685; Cube: 10,000; Peg: 1,461; and TwoRoom: 10,000.

Table 1. Base-task KDE overlap (%), mean ± standard deviation over three 400-episode splits.

EnvironmentAgent stateAll stateAction + agent stateAction + all state
TwoRoom99.04±0.0999.04\pm0.0999.04±0.0999.04\pm0.0998.99±0.0398.99\pm0.0398.99±0.0398.99\pm0.03
Push-T98.94±0.1698.94\pm0.1697.11±0.7197.11\pm0.7198.73±0.1798.73\pm0.1795.12±0.9795.12\pm0.97
Cube98.86±0.0498.86\pm0.0496.32±0.1396.32\pm0.1388.83±0.2188.83\pm0.2180.08±0.3480.08\pm0.34
Insert-Peg61.35±0.9461.35\pm0.9415.26±0.1115.26\pm0.1123.25±1.1423.25\pm1.141.52±0.171.52\pm0.17

Encouragingly, the held-out base-task control shows that the joint-support estimator behaves as expected in most environments. For TwoRoom and PushT, held-out base trajectories remain almost entirely in-distribution even when evaluating the full state-action vector, with overlap above 95%. Cube also retains high overlap at the state level and 80% overlap for action + all state. This supports the use of the joint-KDE score as a meaningful diagnostic in these environments.

The main exception is Insert-Peg, as reviewer expected, where the held-out overlap drops sharply as dimensionality increases. This suggests that the full-dimensional KDE becomes unreliable in this low-sample, high-dimensional setting. We therefore do not interpret the Insert-Peg joint-overlap percentage quantitatively, and will clarify that this score is a diagnostic rather than a definitive support test.

However, importantly, it does not affect the two core design underlying AnyGoal: the tasks are still controlled goal shifts where the semantic factors remain individually within the training support. At worst, the Insert-Peg joint overlap may be higher than our estimator suggests, meaning that some test combinations are more in-distribution than estimated rather than involving factor extrapolation.


W1. Clarification of marginal support check.

...On the other hand in Appendix C, high-dimensional articulated states are sliced into independent 1D marginals for the marginal support check ignoring kinematic configurations and joint coordination."

Thank you for your cafeful evaluation of paper. We would like to respectfully clarify that Appendix C does not slice the full articulated state into independent 1D marginals. The support checks are applied to semantically defined factor groups, including 2D/3D positions and 4D quaternions. Only inherently scalar quantities such as individual joint angles or motor commands are checked separately. This distinction is intentional. Our marginal test is meant to verify that individual physical factors remain within training support while allowing their joint configuration to be novel. Requiring the full articulated configuration or joint coordination itself to remain in support would largely collapse the distinction between marginal factor support and the combinatorial gap we aim to evaluate. We will make it clearer in the revision.

W2. Proposition 6 and Section 6

Another major issue in my opinion is the theoretical result (Prop 1) seems trivial with major assumptions... It is also very directly follows from prior work [1, 2, 3].

Thank you for raising this point and providing the opportunity to clarify our contribution. We would like to clarify that our original intention of Section 6 is not to provide a complete solution to compositional generalization, but first identify two open question that needs to be addressed, and then formalize one favorable regime. We agree that Proposition 1 is intentionally a simple sufficient condition, and its proof is immediate once the structural and support assumptions hold.

Its purpose is not to solve compositional generalization or mechanism identification, but to formalize one distinction motivated by our experiments: a test point can be globally novel while remaining within the support of every local mechanism relevant to prediction. Importantly, we do not assume away mechanism identification as a solution to the overall problem. Section 6 explicitly separates mechanism identification from extrapolation as two distinct challenges, and states that identifying mechanisms from biased offline trajectories remains open.

We thank the reviewer for pointing out the connection to factored MDPs and causal MBRL. We will cite these works and clarify that Proposition 1 is a synthesis/illustration in our world-model setting rather than a claim of a novel theoretical principle.

AuthorsOpenReview ↗

Response 2

Q2. 4D goal components for Push-T oracle experiment.

Thank you for catching a flaw in our oracle-state experimental setup. The 4D goal pose should not have been included in a predictor meant to test physical dynamics given the state and action. Those were goal block related values which were constant throughout experiment. We apologize for this mistake. Your observation prompted us to rerun the experiment without the goal input, making this analysis more rigorous. We report the corrected results below.

  • Comparison of planning success rate
Task10D AdaLN10D Action Dup.10D Toy-Drift6D AdaLN6D Action Dup.6D Toy-Drift
Push-T92.7±1.292.0±2.089.3±3.191.3±1.291.3±2.388.0±2.0
Avoid-T100.0±0.0100.0±0.099.3±1.2100.0±0.0100.0±0.091.3±2.3
Stack-T70.7±3.166.7±7.054.0±2.072.7±4.269.3±9.233.3±6.4
Push-T Antigoal79.3±5.081.3±3.175.3±5.080.7±1.282.0±5.340.0±8.0

For AdaLN and Action Dup., the planning results change very little. Toy-DriftWorld shows a decline. Notably, it does not change our overall interpretation of the experimental results. We will update those results.

  • Comparison of one-step (5 environment step) rollout error.
variantmodel10d_mse10d_normalized_mse6d_mse6d_normalized_mse
pushtadaln0.00000500.00000050.00000750.0000013
pushtaction_duplicate0.00002800.00000280.00003310.0000055
pushttoy-drift0.00006600.00000660.00071320.0001189
avoidtadaln0.00000630.00000060.00000820.0000014
avoidtaction_duplicate0.00003500.00000350.00002720.0000045
avoidttoy-drift0.00008710.00000870.00015390.0000257
stacktadaln0.00250570.00025060.00486090.0008102
stacktaction_duplicate0.00317940.00031790.00462140.0007702
stackttoy-drift0.00463580.00046360.01582690.0026378
pusht_antigoaladaln0.00027370.00002740.00040300.0000672
pusht_antigoalaction_duplicate0.00035980.00003600.00062940.0001049
pusht_antigoaltoy-drift0.00131740.00013170.00956410.0015940

We also report rollout errors. Because the four goal coordinates are fixed, we expected them to contribute little to the MSE. The unnormalized errors are indeed largely unchanged, except for Toy-DriftWorld, much like the planning results. Since the state dimensions differ, we also provide errors normalized by dimension. We will update the reported results accordingly. Thank you again for catching our mistake. We hope these additional results address your concern.


Finally, we appreciate that you recognized the broader question we are exploring, the compositional-generalization framing, and the value of the oracle-state, action-intervention, and representation-geometry analyses. Your comments also helped us identify several places where the paper needed more careful qualification, clearer reporting, and tighter scope, and they have significantly improved the revised version.

More broadly, our goal with AnyGoal is to provide a practical bridge between theoretical questions about compositional generalization and the empirical evaluation of current world models. We recognize that this middle ground inevitably leaves some factors difficult to isolate perfectly, and your review helped us clarify where our evidence supports a diagnostic interpretation rather than a causal one. At the same time, we believe that systematically exposing these failure modes under controlled goal shifts can provide a useful foundation for the world-model community to develop stronger evaluations, mechanisms, and methods in future work.

We hope that the additional results, corrections, and scope clarifications provided in our rebuttal have addressed the main soundness concerns underlying the current assessment. We very much appreciate the time and care you put into the review. If there are any remaining points that you would like us to clarify or discuss further, we would be happy to address them.

Reviewer_yqy9OpenReview ↗

Response to rebuttal

Thanks for the detailed rebuttal and response to my questions. I appreciate the experiments to address my concerns. I am raising my score as the experiments address my empirical questions, however I believe the lack of theoretical novelty and limitations in the implementation of the proposed framework remain.

Reviewer_avXpOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper tests if world models trained on goal-directed data still work when the goal changes but the environment stays the same. The authors build AnyGoal, 8 variants of TwoRoom, Push-T, Insert-Peg and OGBench Cube where each state/action factor is in the training support on its own but the combinations are new, and check this with per-factor range/Mahalanobis/KDE tests. They train LeWM, SMWM, DINO-WM, DriftWorld, IRASim and three predictors on true simulator states on the base data, then compare prediction error and CEM planning success to the base task. Most models get worse, and the true-state predictors mainly fail on the variants with contact.

Critical strengths
  • The question is relevant and the setup is easy to reuse. Goal-shifted variants of the standard LeWM/DINO-WM environments test something in-distribution planning can’t, and the data-collection tools are released
  • The oracle-state ablation is the most useful part. With true states, free-space variants are solved (Avoid-T 100/100/99.3%), while contact-rich ones still drop (Extract-Peg -22/-18.7/-20.7 points, Table 12), so encoder and dynamics failures can be told apart
Critical weaknesses
  • There is no control for task difficulty. Everything is a delta to the base task, and nothing shows what is achievable per variant (e.g. the true simulator in the same planner, or models trained on variant data). Contact-rich variants might just be harder to plan in 25 steps, while in Avoid-T/Cube-Hop the object never moves, so part of the success criterion comes for free
  • Fig. 1 doesn’t fit Table 12. The true-state predictors trained on Push-T get ~100% on Avoid-T, so the data seems to be enough to learn dynamics that transfer. This looks more like an encoder or planning cost issue since only the small agent moves and does not really support the shortcut explanation in the caption.
  • The error ratios are hard to compare between models because they depend on how small the base error was. A model that fits the base task almost perfectly gets a huge ratio for any error on the variant. E.g. on Extract-Peg the true-state predictors range from 2x to 34x, but in absolute terms (Table 10 times Table 9) all three are at ~0.12 MSE. Table 9 also says training MSE, so the ratios might include the normal train/test gap
  • No standard deviations or confidence intervals in the tables, and the plots don’t seem to have error bars. It’s also unclear if seeds 0/1/2 are training or eval seeds. With what looks like ~150 planning episodes per cell, differences of a few points (e.g. several Cube-Diagonal changes) might be noise, while the large drops hold up
Questions for authors
  • How do you explain the Avoid-T results given the Fig. 1 caption? is the problem in the encoder or in the planning cost
  • Are seeds 0/1/2 training or eval seeds, and can you report std/CIs for Tables 10-12? some of the smaller differences might flip sign
5 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for the thoughtful and detailed review. We especially appreciate the positive assessment of the relevance and reusability of our goal-shifted benchmark, as well as the recognition that the oracle-state ablations provide a useful way to disentangle representation-level and dynamics-level failures. We address the reviewer’s questions and concerns below, together with several additional analyses and clarifications that we will incorporate into the revised manuscript.

W1. Control of difficulty

Thank you for your valuable feedback. We ran true simulator MPC planning as well as rollout error comparisons with oracle-state predictors.

  • True simulator CEM. All planning configurations are identical to those used in the main experiments.
Base familyDatasetSuccess rate (%)
CubeCube95.33±4.6295.33 \pm 4.62
CubeCube-Diagonal74.00±5.2974.00 \pm 5.29
CubeCube-Hop100.00±0.00100.00 \pm 0.00
Insert-PegInsert-Peg84.00±2.0084.00 \pm 2.00
Insert-PegAvoid-Peg99.33±1.1599.33 \pm 1.15
Insert-PegExtract-Peg30.00±12.0030.00 \pm 12.00
Push-TPush-T90.67±1.1590.67 \pm 1.15
Push-TAvoid-T100.00±0.00100.00 \pm 0.00
Push-TPush-T Antigoal66.67±10.0766.67 \pm 10.07
Push-TStack-T63.33±12.2263.33 \pm 12.22
TwoRoomTwoRoom100.00±0.00100.00 \pm 0.00
TwoRoomTR-Route100.00±0.00100.00 \pm 0.00

To reduce dependence on the particular planning cost, we additionally report in-sample rollout errors from models trained separately on each variant. For this additional analysis, we generated 4,000 episodes per variant and retrained the corresponding base-task models from scratch on the first 4,000 episodes of each original base-task dataset, matching the episode count. Apart from the training data, we retained the original training settings. Thus, the Base rows below come from newly retrained models and are not the base-task results reported in Table 9. The training configurations of the toy models are identical to those used for training on the corresponding base tasks. We use these results as a complementary model-capacity diagnostic, showing that the variant dynamics are representable and can be fit from variant data.

GroupDatasetAdaLN 5-stepAdaLN 25-stepAction Dup. 5-stepAction Dup. 25-stepToy-Drift 5-stepToy-Drift 25-step
BasePush-T8.01×10−6\mathbf{8.01\times10^{-6}}5.52×10−5\mathbf{5.52\times10^{-5}}5.68×10−5\mathbf{5.68\times10^{-5}}3.59×10−4\mathbf{3.59\times10^{-4}}8.27×10−4\mathbf{8.27\times10^{-4}}5.92×10−3\mathbf{5.92\times10^{-3}}
VariantAvoid-T2.82×10−62.82\times10^{-6}--8.33×10−68.33\times10^{-6}--1.21×10−41.21\times10^{-4}--
VariantStack-T7.89×10−57.89\times10^{-5}2.58×10−42.58\times10^{-4}3.66×10−43.66\times10^{-4}1.22×10−31.22\times10^{-3}5.71×10−35.71\times10^{-3}1.50×10−21.50\times10^{-2}
VariantPush-T Antigoal2.11×10−52.11\times10^{-5}1.82×10−41.82\times10^{-4}8.10×10−58.10\times10^{-5}3.84×10−43.84\times10^{-4}7.23×10−47.23\times10^{-4}2.35×10−32.35\times10^{-3}
BaseCube5.38×10−4\mathbf{5.38\times10^{-4}}9.10×10−4\mathbf{9.10\times10^{-4}}2.56×10−3\mathbf{2.56\times10^{-3}}1.42×10−2\mathbf{1.42\times10^{-2}}2.62×10−1\mathbf{2.62\times10^{-1}}3.52×10−1\mathbf{3.52\times10^{-1}}
VariantCube-Diagonal1.30×10−31.30\times10^{-3}2.57×10−32.57\times10^{-3}5.65×10−35.65\times10^{-3}1.41×10−21.41\times10^{-2}4.63×10−24.63\times10^{-2}7.76×10−27.76\times10^{-2}
VariantCube-Hop2.92×10−62.92\times10^{-6}2.34×10−52.34\times10^{-5}1.49×10−61.49\times10^{-6}2.98×10−62.98\times10^{-6}8.40×10−68.40\times10^{-6}1.76×10−51.76\times10^{-5}
BaseTwoRoom4.78×10−6\mathbf{4.78\times10^{-6}}2.89×10−5\mathbf{2.89\times10^{-5}}8.92×10−6\mathbf{8.92\times10^{-6}}8.24×10−5\mathbf{8.24\times10^{-5}}2.47×10−4\mathbf{2.47\times10^{-4}}1.03×10−3\mathbf{1.03\times10^{-3}}
VariantTR-Route2.60×10−62.60\times10^{-6}2.77×10−52.77\times10^{-5}3.10×10−63.10\times10^{-6}1.64×10−51.64\times10^{-5}2.90×10−62.90\times10^{-6}7.90×10−67.90\times10^{-6}
BaseInsert-Peg3.50×10−3\mathbf{3.50\times10^{-3}}1.04×10−2\mathbf{1.04\times10^{-2}}9.29×10−3\mathbf{9.29\times10^{-3}}2.99×10−2\mathbf{2.99\times10^{-2}}6.69×10−2\mathbf{6.69\times10^{-2}}1.03×10−1\mathbf{1.03\times10^{-1}}
VariantAvoid-Peg2.39×10−42.39\times10^{-4}6.33×10−46.33\times10^{-4}3.94×10−33.94\times10^{-3}1.87×10−21.87\times10^{-2}2.22×10−22.22\times10^{-2}5.51×10−25.51\times10^{-2}
VariantExtract-Peg7.14×10−47.14\times10^{-4}1.49×10−31.49\times10^{-3}2.18×10−32.18\times10^{-3}6.72×10−36.72\times10^{-3}1.94×10−21.94\times10^{-2}3.62×10−23.62\times10^{-2}

Together, these two controls address complementary aspects of the reviewer’s concern. True-simulator CEM probes task-dependent planning difficulty, while variant-trained rollout fitting tests whether the same model classes can represent the variant dynamics when directly exposed to them.

(continue on the next response)

AuthorsOpenReview ↗

Response Part 2

Most results are consistent with our expectations. However, Extract-Peg requires more careful interpretation given its substantially lower true-simulator planning performance. We clarify that true-simulator CEM is not an upper bound on achievable performance. It measures task-dependent planning difficulty under the same finite-budget CEM procedure while removing transition-model error. Since CEM is not an optimal controller, a learned model can occasionally achieve higher empirical success than the true simulator, for example by inducing an easier effective planning landscape. We will therefore use these results as a diagnostic of task-dependent planning difficulty rather than as an upper bound.

We would also like to clarify the intended role of task difficulty in AnyGoal. Our tasks are designed to test generalization to novel compositional state-action pairs while preserving the same underlying physical dynamics, with goal shifts providing a simple and controlled way to induce such recombinations. While task difficulty naturally varies across goal-shifted tasks and is difficult to match exactly, AnyGoal does not require intrinsic difficulty to be matched between base and variant tasks. However, we agree that this should have been discussed more clearly. We will include these results and clarify this interpretation in the camera-ready version.


W2, Q1. Explanation of Figure 1 and Table 12

This is a good point, and we agree that we should have explained it more clearly. The strong oracle-state performance on Avoid-T shows that Push-T data can be sufficient for transfer when the model is given the correct physical-state representation. However, this does not rule out behavior-policy shortcuts in end-to-end models.

Our intended claim is not a predictor-only shortcut. Because the encoder and predictor are jointly learned from goal-biased expert trajectories, the learned representation itself may emphasize features aligned with the expert behavior. This can allow the model to fit training transitions while remaining insufficiently sensitive to counterfactual supplied actions.

For example, for a JEPA trained on Push-T, the encoder may emphasize features such as agent-to-block distance, block-relative direction, or other quantities strongly correlated with the expert action. Because the encoder and predictor are jointly optimized under the same predictive objective, representation-level and predictor-level effects are not cleanly separable in the end-to-end model.

Oracle-state predictors remove this representation-learning pathway by directly providing task-relevant physical factors. This effectively separates representation learning from dynamics prediction. Therefore, oracle-state success and the shortcut explanation are not necessarily contradictory.

We agree that the action-intervention analysis does not uniquely identify the failure as originating from the encoder, predictor, or planning cost. In the end-to-end setting, these components are jointly optimized and their effects are not cleanly separable from the current evidence. For this reason, attributing the failure to a single component would itself be an overstatement. Our intended interpretation is diagnostic rather than causal. Reduced counterfactual action sensitivity is consistent with behavior-policy shortcuts in the end-to-end model, but does not uniquely localize their source. We will revise the wording around Fig. 1 to make this distinction explicit.

We thank the reviewer for giving us the opportunity to clarify this point. We will explain this distinction more clearly in revised version, which we believe will also make the interpretation more useful to broader readers.


W3. Inter-model comparison

Thank you for the careful assessment. We intentionally avoid using these error ratios for direct cross-model comparison. The models differ substantially in architecture, prediction space, and absolute base-task error, so comparing the magnitude of the ratios across models can be misleading.

We agree that the current presentation may have made the intended interpretation less clear. We will revise the presentation to make this distinction explicit. We would also be happy to report the absolute errors instead of the relative changes in the main paper, if the reviewer finds them more informative.

(continue)

Alongside, we agree that the current ratio compares variant-task evaluation error with base-task training error, so it may include an ordinary train-test gap in addition to error associated with the goal shift. At the same time, AnyGoal is explicitly designed to evaluate generalization to previously unseen state-action compositions. In this sense, novelty of the evaluated trajectories is part of the phenomenon being tested rather than an independent confound. We will make this limitation explicit in the paper and do not overinterpret the results in a potentially misleading way. Again, thank you for your valuable feedback.

AuthorsOpenReview ↗

Response Part 3

W4, Q2. Statistics

Thank you for the clarification request. Seeds 0, 1, and 2 are evaluation seeds. We report planning mean ±\pm standard deviation below. We will also report standard deviations for the rollout evaluation in the camera-ready version. These standard deviations therefore quantify evaluation variability rather than training-seed variability, and we will state this distinction explicitly in the revised paper.

Additionally, we had previously identified a reporting inconsistency in the IRASim Insert-Peg value in Table 12, and the corrected value is reflected in this table. This correction changes the absolute baseline value but does not alter the qualitative conclusions drawn from the broader pattern across tasks and model families. We will update the corresponding discussion to ensure that no interpretation relies on the incorrect value.

Planning success (%), mean ±\pm standard deviation across three evaluation seeds.

TaskLeWMSMWMDINO-WMDriftWorldIRASimAdaLNAction Dup.Toy-Drift
TwoRoom89.3±5.089.3\pm5.0100.0±0.0100.0\pm0.0100.0±0.0100.0\pm0.090.0±4.090.0\pm4.076.0±6.076.0\pm6.0100.0±0.0100.0\pm0.099.3±1.299.3\pm1.2100.0±0.0100.0\pm0.0
TR-Route60.7±4.260.7\pm4.264.7±4.264.7\pm4.268.0±6.068.0\pm6.076.0±5.376.0\pm5.363.3±2.363.3\pm2.3100.0±0.0100.0\pm0.0100.0±0.0100.0\pm0.0100.0±0.0100.0\pm0.0
Push-T84.7±4.684.7\pm4.684.7±4.284.7\pm4.274.0±3.574.0\pm3.543.3±4.243.3\pm4.25.3±3.15.3\pm3.192.7±1.292.7\pm1.292.0±2.092.0\pm2.089.3±3.189.3\pm3.1
Avoid-T3.3±2.33.3\pm2.34.7±3.14.7\pm3.16.7±3.16.7\pm3.10.7±1.20.7\pm1.20.0±0.00.0\pm0.0100.0±0.0100.0\pm0.0100.0±0.0100.0\pm0.099.3±1.299.3\pm1.2
Stack-T20.7±4.220.7\pm4.220.0±3.520.0\pm3.58.7±3.18.7\pm3.110.7±5.010.7\pm5.02.0±0.02.0\pm0.070.7±3.170.7\pm3.166.7±7.066.7\pm7.054.0±2.054.0\pm2.0
Push-T Antigoal34.0±12.034.0\pm12.044.0±3.544.0\pm3.527.3±6.427.3\pm6.414.7±3.114.7\pm3.15.3±6.15.3\pm6.179.3±5.079.3\pm5.081.3±3.181.3\pm3.175.3±5.075.3\pm5.0
Insert-Peg26.7±4.626.7\pm4.670.0±5.370.0\pm5.335.3±1.235.3\pm1.263.3±12.163.3\pm12.132.7±6.132.7\pm6.174.0±2.074.0\pm2.071.3±4.271.3\pm4.271.3±2.371.3\pm2.3
Avoid-Peg31.3±6.131.3\pm6.132.7±5.832.7\pm5.828.7±6.128.7\pm6.168.0±5.368.0\pm5.334.0±10.434.0\pm10.492.0±3.592.0\pm3.589.3±3.189.3\pm3.180.7±4.280.7\pm4.2
Extract-Peg39.3±3.139.3\pm3.153.3±3.153.3\pm3.146.7±3.146.7\pm3.152.7±3.152.7\pm3.146.0±5.346.0\pm5.352.0±2.052.0\pm2.052.7±4.252.7\pm4.250.7±4.250.7\pm4.2
Cube30.7±4.230.7\pm4.272.7±2.372.7\pm2.360.0±2.060.0\pm2.030.7±5.830.7\pm5.810.7±1.210.7\pm1.230.0±3.530.0\pm3.520.7±1.220.7\pm1.225.3±7.025.3\pm7.0
Cube-Hop17.3±5.817.3\pm5.840.7±6.440.7\pm6.451.3±6.451.3\pm6.464.7±9.264.7\pm9.212.7±1.212.7\pm1.295.3±2.395.3\pm2.351.3±9.051.3\pm9.079.3±6.179.3\pm6.1
Cube-Diagonal28.0±3.528.0\pm3.553.3±7.053.3\pm7.058.7±2.358.7\pm2.342.0±7.242.0\pm7.29.3±5.09.3\pm5.020.0±6.020.0\pm6.012.0±4.012.0\pm4.022.0±5.322.0\pm5.3

As you pointed out, the large degradations remain substantially larger than the observed evaluation-seed variability, while several smaller changes, particularly on Cube-Diagonal, fall within this variation. We will therefore avoid interpreting these small differences directionally. This does not change the main conclusions of the paper, which are based on the larger and more consistent patterns across tasks and model families rather than on small per-model differences.

More broadly, we agree that the strongest supported contribution of AnyGoal is a controlled benchmark and empirical characterization of goal-shift generalization failures, rather than causal identification of a single shortcut mechanism.


Finally, we thank the reviewer for the thoughtful and constructive feedback. We especially appreciate the care taken in identifying several important issues in the evaluation and interpretation, which helped us strengthen both the analysis and the presentation of the paper. We greatly value the reviewer’s engagement with our work and the suggestions for improving its scope and clarity.

More broadly, we would like to clarify how we believe AnyGoal should be interpreted. The paper is not intended to establish a single causal shortcut mechanism. Rather, its main contribution is a benchmark and empirical characterization of goal-shift generalization failures, together with diagnostics that help separate representation-level, dynamics-level, and planning-related effects. We hope the work can be assessed primarily in this scope and serve as a useful foundation for the world-model community to study these failures more systematically.

We will make sure that the promised reporting of absolute errors and variability, as well as the more careful qualification of causal claims, are explicitly reflected in the revised manuscript. We remain happy to address any additional questions or concerns.

Reviewer_NveJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces AnyGoal, a benchmark of 8 goal-shifted test variants across 4 environments where individual state and action factors remain marginally in-distribution, while their joint combinations lie in the under-covered combinatorial gap. Using this benchmark, the authors evaluate a variety of world model frameworks, including JEPA-style models, generative models, and oracle-state predictors. The empirical results reveal that end-to-end visual models degrade even on simple free-space navigation shifts due to behavioral bias and visual/latent entanglement, whereas contact-rich goal shifts cause severe degradation even when models are given ground-truth oracle states. Finally, the paper proposes a theoretical formulation based on regime-dependent sparse local mechanisms, arguing that compositional generalization is achievable without global state-action coverage if models can identify sparse causal parents whose local inputs remain in-distribution across goal shifts.

Critical strengths
  • Well-defined scope and rigorous marginal vs. joint support verification: The benchmark narrows its focus to a well-controlled type of distribution shift. Its construction explicitly verifies that test trajectories stay within the marginal support of training factors while quantifying the drop in joint state-action density, cleanly ruling out marginal coordinate extrapolation as the cause of failure. This careful curation makes AnyGoal a useful diagnostic tool for a targeted research question.

  • Diagnostic separation of free-space vs. contact-rich regimes and oracle vs. visual models: Comparing end-to-end visual world models against oracle-state predictors across non-contact and contact-rich variants provides a clear empirical insight. In free-space tasks, oracle-state predictors maintain or improve planning success because the true agent dynamics decouple from the object, whereas end-to-end visual models suffer sharp planning drops due to visual/latent entanglement and behavioral bias.

Critical weaknesses
  • Section 6's proposal cannot escape the coverage requirement for contact-rich tasks: While the paper motivates compositional generalization as an alternative to unrealistic data coverage, the theoretical solution in Section 6 requires that the active local inputs for each mechanism are already covered in the training support. Coordinate sparsity helps in free-space regimes, but in contact-rich two-body tasks, the active parent set during contact couples the agent, object, and action together. Consequently, within the contact regime, "local" support coverage collapses back into requiring joint state-action coverage. Furthermore, assuming local mechanisms are already identified on the training support bypasses the core challenge of state-action confounding induced by the expert policy.
Questions for authors
  1. JEPA-style vs. generative models: Could the authors clarify the distinct empirical trends between JEPA-style and generative world models across the benchmark? Based on your findings, does one paradigm offer a stronger foundation for goal-shifted generalization, and why?

  2. Clarifying reporting discrepancies: Could you clarify which numbers are correct between Figures 5/6 and Tables 10/12 (specifically the swapped values for AdaLN and Action Dup., and the planning deltas vs. absolute numbers for AdaLN and IRASim)? Additionally, was Toy-DriftWorld trained with next-state MSE (Appendix F.2, line 1452) or the stochastic drifting loss (Appendix F.2.2 & G, lines 1486, 1495), and what is PreJEPA in Table 6 under Insert-Peg?

3 comments
AuthorsOpenReview ↗

Response Part 1

We are grateful to the reviewer for the valuable feedback and for highlighting the strengths of our work.


W1. About Section 6 and Proposition 1

We thank the reviewer for raising this important point. Before addressing Proposition 1 itself, we would like to clarify the intended scope of Section 6. The goal of this section is not to provide a complete solution to compositional generalization. Rather, it first identifies two open challenges (mechanism identification and extrapolation beyond training support) and then formalizes one favorable regime in which global combinatorial novelty does not necessarily require extrapolating every transition mechanism beyond its local training support.

We agree with the reviewer that contact-rich dynamics can enlarge the active parent sets and therefore make local support coverage substantially more demanding than in free-space regimes. However, this does not necessarily imply that local support coverage collapses to full joint state-action coverage. Proposition 1 requires support coverage only over the active parent subset uku_k for each transition mechanism, rather than over the entire state-action vector. In a contact regime, a mechanism may indeed depend jointly on the agent state, object state, and action, but this parent set can still remain a strict subset of the full global configuration. In such cases, the complete state-action composition can be novel while the inputs relevant to each local mechanism remain within training support.

We agree, however, that the benefit of locality decreases as interaction regimes enlarge the active parent sets, and disappears in the limiting case where a mechanism depends on the full state-action configuration. We will make this limitation explicit in the revision.

Regarding mechanism identification, Proposition 1 intentionally assumes that the relevant local mechanisms have already been identified. Section 6 treats this as a separate open problem: fitting expert trajectories does not guarantee recovery of reusable action-conditioned mechanisms under behavior-policy confounding. Proposition 1 addresses only the second question about when an already identified mechanism can generalize to novel global compositions, and does not claim to solve the identification problem. We will also clarify that Proposition 1 should be interpreted as an illustrative sufficient condition, rather than as an empirical explanation of the failures observed in the contact-rich AnyGoal variants. Our experiments do not establish that its local-support or mechanism-identification assumptions hold in those variants.

We thank the reviewer again for helping us sharpen this distinction and clarify the scope of the proposition. We will revise our paper accordingly.


Q1. Comparison of JEPA-style and generative world models

Thank you for raising this point and for giving us the opportunity to clarify the intended interpretation. In AnyGoal, we deliberately avoid using the benchmark to rank JEPA-style and generative world models as model families. That said, as discussed in the main text, we do observe some limited paradigm-level trends: generative models sometimes show smaller relative increases in rollout error under certain goal shifts, but this does not translate into a consistent advantage in MPC planning.

We do not believe the current experiments support a reliable paradigm-level comparison for several reasons. First, the evaluated models differ substantially in architecture, capacity, and training configuration, as we preserve the configurations provided by the original methods rather than artificially matching model size or capacity. Second, although we use a unified MPC framework, JEPA-style and generative models use different prediction and planning spaces. Comparing costs defined in latent space with costs defined in pixel or reconstruction space is therefore not a controlled comparison of the underlying modeling paradigm. Third, rollout prediction provides a more direct evaluation than planning, but the prediction errors are still measured in different spaces, so their absolute errors are not directly comparable.

For this reason, the main purpose of our analysis is not to identify a stronger paradigm, but to characterize how goal-shift failures vary across representation learning, dynamics prediction, task structure, and evaluation metric. We will make this distinction more explicit in the camera-ready version to avoid suggesting that AnyGoal is intended to establish one paradigm as a stronger foundation than the other. Thank you again for the helpful question.

AuthorsOpenReview ↗

Response Part 2

Q2. Clarification about results

We thank the reviewer for carefully checking the reported results.

We identified an inconsistency in the IRASim planning results. After rechecking the evaluation logs, we found that the table was not from the same finalized result set. The correct Insert-Peg baseline is 32.7%, which yields +1.3 and +13.3 percentage-point changes for Avoid-Peg and Extract-Peg, respectively. We will update Table 12 using the same finalized evaluation results so that the absolute and relative values are consistent.

For the reported swap between AdaLN and Action Dup., as well as the concern about AdaLN planning deltas versus absolute values, we rechecked the figures and tables but were not able to identify the specific entries being referenced. We would appreciate clarification on the exact figure locations or values, as we may have overlooked them.

For Toy-DriftWorld, the model was trained using the drifting loss. The conflicting description in the appendix is a reporting inconsistency and will be corrected in the camera-ready version. “PreJEPA” in Table 6 refers to DINO-WM; this is also a naming error and has been corrected in the revised version. Thank you for pointing out these reporting issues.


Finally, we appreciate that the reviewer recognized the broader question explored by the paper, the compositional-generalization framing, and the value of the oracle-state, action-intervention, and representation-geometry analyses. The review also helped us identify several places where the paper required tighter qualification, clearer reporting, and more explicit scoping.

We hope that the additional results, corrections, and clarifications provided in our response address the main soundness concerns underlying the current assessment. If any concerns remain, we would be happy to clarify them further. We greatly appreciate the time and care the reviewer has taken in evaluating the paper.

AuthorsOpenReview ↗

Follow-up on rebuttal

Dear Reviewer NveJ,

Thank you again for your valuable feedback. As the discussion period is nearing its end, we would like to follow up to see if you have any further questions or anything else we can help clarify. We hope our responses have addressed your concerns, and if any remain, we would be glad to discuss them further.

Thank you :)

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Comment (for reviewers and AC)

We thank all the reviewers for their thoughtful feedback and constructive discussion. We provide a brief summary here.

  1. We are very glad that the reviews recognized several key strengths of our work:
  • Important and well-motivated research question (avXp, yqy9). Reviewers found the question of world-model generalization relevant and agreed that compositional generalization provides an appropriate framing for systematic evaluation. The goal-shifted environments were recognized as a clean and intuitive way to test generalization beyond standard in-distribution planning (yqy9), while the overall setup was noted to be easy to reuse (avXp).
  • Controlled benchmark design (NveJ; see also avXp, yqy9). NveJ specifically highlighted the well-defined scope and rigorous verification that test trajectories remain within the marginal support of training factors while exhibiting reduced joint state-action density, thereby ruling out marginal coordinate extrapolation as the source of failure. More broadly, avXp and yqy9 highlighted the usefulness and clean construction of the goal-shifted variants.
  • Insightful separation of representation and dynamics failures (NveJ, avXp, yqy9). All three reviewers highlighted the oracle-state predictor experiments as a useful analysis. These experiments contrast oracle and end-to-end visual world models across free-space and contact-rich tasks, helping distinguish failures arising from visual representations from those arising from learned dynamics.
  • Informative diagnostic analyses (yqy9). yqy9 additionally highlighted the counterfactual action-intervention analysis as a useful diagnostic for action sensitivity, as well as the representation-geometry analysis and the finding that rollout MSE and MPC planning success can diverge depending on task tolerance and representation space.

Across the reviews, there is substantial agreement on the value of AnyGoal as a controlled framework for studying compositional generalization in world models and for diagnosing where and why generalization failures arise.


  1. We have used the rebuttal to address the remaining questions about evaluation, task difficulty, reporting, and interpretation, and have added new results and clarifications accordingly.

Key clarifications

  • Scope of the theoretical claim (NveJ-W1; yqy9-W2). We clarified that Proposition 1 provides an illustrative sufficient condition for compositional generalization under local mechanism support, rather than explaining the contact-rich AnyGoal results or solving mechanism identification.
  • Interpretation of the benchmark and diagnostics (avXp-W2/W3; NveJ-Q1). We clarified several interpretations: 1) counterfactual action sensitivity is a diagnostic of behavioral shortcuts rather than evidence uniquely localizing failure to the encoder, 2) rollout-error ratios should not be compared directly across models, 3) our results do not establish a general advantage of JEPA-style over generative world models.
  • Support verification (yqy9-W1/Q1). We clarified that marginal support is evaluated over semantic factor groups rather than individual coordinates. New held-out base-task controls show high full-state overlap for most environments but reveal that full-dimensional KDE is unreliable for low-sample, high-dimensional Insert-Peg - We will therefore revise our paper in the way that treats KDE as a diagnostic rather than a definitive support test.

Additional controls and corrections

  • Task difficulty and model capacity (avXp-W1). We added true-simulator CEM controls and separately trained oracle-state predictors on each variant. Together, these distinguish finite-budget planning difficulty from whether the evaluated model classes can fit the variant dynamics when given appropriate training data.
  • Variability and corrected oracle experiments (avXp-W4/Q2; yqy9-Q2). We clarified that the three seeds are evaluation rather than training seeds and now report planning success as mean ± standard deviation. We also corrected the Push-T oracle state from 10D to 6D by removing four constant goal-pose coordinates and reran the corresponding planning and rollout evaluations.
  • Reporting corrections (NveJ-Q2; avXp-W4). We corrected the IRASim Table 12 results, confirmed that Toy-DriftWorld uses the drifting loss, and corrected PreJEPA to DINO-WM.

More detailed responses can be found directly under reviewers' comments.

AuthorsOpenReview ↗

Global comment (Final remarks for AC)

For the Area Chair’s consideration. We respectfully ask the Area Chair to consider the broader contribution and potential community impact of this work. We are truely confident that AnyGoal can provide a useful foundation for the world model community. To our knowledge, this is the first systematic study of compositional generalization in learned world models that also diagnoses where these failures arise. In pursuing a middle ground between theory and practice, we do not claim to close the gap between clean structural conditions and complex visual environments. Rather, we aim to make this gap concrete, measurable, and open to systematic study. We therefore invested substantial effort in providing not only a benchmark, but also reusable methodology and diagnostic insights that future work can build upon, which reviewers repeatedly recognized as useful and insightful. Their feedback has further strengthened the work through additional controls, corrected analyses, narrower claims, and clearer limitations. We hope the Area Chair will consider this community value alongside the additional evidence and clarifications provided during the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.