acceptodds
Under review as a conference paper at ICLR 2027

Closing the Train-Test Gap in World Models for Gradient-Based Planning

Abstract

World models paired with model predictive control (MPC) can be trained offline on large-scale datasets of expert trajectories and enable generalization to a wide range of tasks chosen at inference time. Compared to traditional MPC procedures, which rely either on slow search algorithms or on iteratively solving optimization problems exactly, gradient-based planning offers a computationally efficient alternative. However, the performance of gradient-based planning has thus far lagged behind that of other approaches. In this paper, we propose improved methods for training world models that enable efficient gradient-based planning. We begin with the observation that although a world model is trained on a next-state prediction objective, it is used at test-time to instead estimate a sequence of actions. The goal of our work is to close this train-test gap. To that end, we propose train-time data synthesis techniques that enable significantly improved gradient-based planning with existing world models. At test time, our approach outperforms or matches the classical gradient-free cross-entropy method (CEM) across a variety of object manipulation and navigation tasks in 10% of the time budget.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Latent World Models, Gradient-based Planning, Adversarial Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_16j8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines the mismatch between training world models for next-state prediction and using them for gradient-based action optimization. It diagnoses a gap between predicted planning loss and actual execution, then studies two responses: Online World Modeling (OWM), which adds simulator trajectories induced by the planner, and Adversarial World Modeling (AWM), which fine-tunes the model using perturbations of latent states and actions. Experiments with DINO-WM cover six simulation tasks and egocentric pose planning on Nymeria. AWM improves gradient-based planning and is reported to approach CEM performance at substantially lower planning time.

Critical strengths
  1. The paper identifies a meaningful distinction between predictive accuracy and suitability for optimization. Section 2 and Figure 1 make the issue concrete by comparing the loss optimized inside the model with the outcome of executing the resulting actions. The AWM objective in Section 3.2 addresses this distinction through local adversarial robustness of the transition predictor. This is a well-motivated methodological response to the diagnosed problem, and it can be applied without collecting new simulator transitions. It makes the model's behavior during downstream action optimization an explicit consideration in training.

  2. The improvement is supported across several planning settings. Figure 3 reports substantial gains over the original model with GBP, including PushT from 76% to 92%, PointMaze from 68% to 98%, and Reacher from 42% to 76%. The comparison with OWM tests a different response to the same training–planning mismatch rather than only comparing against an unchanged model. Figure 5 extends the evaluation to longer-horizon pose planning with a 48-dimensional action space. These results support AWM's practical value beyond a single navigation example.

Critical weaknesses

W1. The evidence supports AWM's effectiveness more strongly than the proposed explanation of why it works. Section 4.4 attributes the improvement to a smoother action-optimization landscape and draws a broad conclusion about the insufficiency of compute and data scaling from OWM's mixed results. Figures 1–2 are informative, but the study does not report matched clean fine-tuning or non-adversarial perturbation controls that separate the effect of adversarial perturbations from additional training and ordinary local regularization. The central mechanism remains plausible, but its attribution and the dismissal of data coverage are stronger than the evidence establishes.

W2. The main efficiency comparison may also reflect different planning objectives. Section 4.1 states that GBP uses Weighted Goal Loss, including intermediate predicted states, whereas CEM's Algorithm 4 scores the terminal state. Appendix C.4 specifies task-dependent weights, including greater weight on earlier states for PushT, the task used for the time–performance comparison in Section 4.3 and Figure 4. This matters because intermediate-state shaping can change optimization difficulty. The reported results demonstrate the performance of the complete pipelines, but do not yet isolate the claimed efficiency advantage of AWM with GBP over CEM under the same objective.

Questions for authors

Q1 (W1). With the same starting checkpoint, data, and fine-tuning budget, how does AWM compare with clean fine-tuning and random perturbations of the same scale? Does its advantage remain specific to adversarial perturbations, and do these comparisons support the landscape explanation in Section 4.4?

Q2 (W2). Was CEM given the same Weighted Goal Loss as GBP in Figure 4? If the objectives differed, does the reported time–performance advantage remain when both planners optimize the same loss, including the same intermediate-state weights?

2 comments

We thank the reviewer for recognizing that the paper identifies a meaningful distinction between predictive accuracy and suitability for optimization. We also appreciate that the reviewer found AWM a well-motivated response that needs no new simulator transitions, with improvements across several planning settings.

"...With the same starting checkpoint, data, and fine-tuning budget, how does AWM compare with clean fine-tuning and random perturbations of the same scale? Does its advantage remain specific to adversarial perturbations..." We agree, and we have added these controls. Each arm uses the same checkpoint, data and number of updates, with 3 seeds and 50 shared cases. Gaussian and Uniform noise are drawn at AWM's scale.

GBP success (%), mean ± std over 3 seedsOriginal (1 model)CleanGaussianUniformAWM
Wall78.080.7 ± 1.276.7 ± 6.168.0 ± 2.094.0 ± 2.0
PointMaze70.054.0 ± 3.562.0 ± 6.054.0 ± 0.094.7 ± 1.2

"...and do these comparisons support the landscape explanation in Section 4.4?" Partially. These controls are consistent with an AWM benefit, but do not establish that smoothing mediates the improvement. To test this more directly, we checked whether the model's directional derivatives of the planning loss along random action directions agree in sign with the simulator's:

Sign agreement (%), 0.05, mean ± stdOriginalCleanGaussianAWM
Wall70.0 ± 0.176.7 ± 0.077.8 ± 1.981.1 ± 1.0

"...its attribution and the dismissal of data coverage are stronger than the evidence establishes." We agree, particularly since OWM itself substantially improves PointMaze. We have removed the statement that "compute and data scaling alone will not make GBP viable" and limited the conclusion to the tested configuration. However, it still stands that AWM significantly outperforms a reasonable attempt at applying DAgger and increasing data coverage.

"...Was CEM given the same Weighted Goal Loss as GBP in Figure 4? If the objectives differed, does the reported time–performance advantage remain when both planners optimize the same loss, including the same intermediate-state weights?" We ran AWM+CEM with the same model and WGL objective used for GBP. AWM improves both planners, so the improvement is not specific to gradient-based planning, although the gain is larger for GBP:

Gain from AWM (pp), shared WGLGBPCEM
Wall+16.0+9.3
PointMaze+24.7+4.7

Both GBP and CEM optimize action sequences using only the learned world model, without querying the environment simulator for predicted trajectories. Consequently, with the same model architecture, input dimensions, hardware and planning hyperparameters, planning times should be comparable across PushT, Wall and PointMaze.

Thank you again for your valuable feedback. We believe your feedback has substantially improved our paper. We would greatly appreciate it if you would consider increasing your score accordingly. Do you have any other questions we can address?

Reviewer_16j8OpenReview ↗

Response to authors

Thanks for adding the clean fine-tuning and random-noise controls. AWM's advantage on Wall and PointMaze gives much stronger evidence that the gains are not explained by extra training or ordinary noise alone. This addresses W1.

The shared-WGL experiment is useful and shows that AWM also benefits CEM. However, gains of 9.3 and 4.7 percentage points do not tell us CEM's final success rates or how long it takes to reach them. Similar runtime across environments at fixed settings does not establish how much computation each planner needs to reach comparable success.

Could you report the absolute success rates and measured single-horizon planning times of GBP and CEM using the same AWM model and WGL, including the same intermediate-state weights? An updated PushT comparison, as in Figure 4, would be the most direct answer.

I would like to resolve this remaining efficiency comparison before reconsidering my recommendation.

Reviewer_xyPAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to train world models that better support gradient-based planning. It introduces Online World Modeling (OWM), which finetunes on simulator-corrected planner trajectories, and Adversarial World Modeling (AWM), which finetunes on adversarially perturbed inputs while retaining the original next-state targets. Experiments on six simulated tasks and an offline egocentric pose-planning benchmark show that AWM improves gradient-based planning and achieves performance competitive with vanilla CEM. A timing study on PushT demonstrates a substantial reduction in planning time.

Critical strengths

The paper addresses a relevant problem: accurate next-state prediction does not necessarily provide a useful objective for optimizing actions. AWM is a simple training modification that requires no additional simulator interaction, and the reported improvements over baseline gradient-based planning are substantial, for example, from 68% to 98% on PointMaze and from 42% to 76% on Reacher. The additional improvements on LeWM provide some evidence that the method is useful beyond DINO-WM.

Critical weaknesses
  1. The experiments do not isolate the benefit of adversarial perturbations. The main comparison is between a pretrained model and that model after additional adversarial finetuning. Matched controls using clean-data finetuning and random perturbations are missing. Figure 8 leaves the zero-perturbation setting blank, while the FGSM-versus-PGD comparison only compares different adversarial training procedures. These controls are important for distinguishing the contribution of adversarial perturbations from additional optimization or ordinary noise augmentation.

  2. The conclusions about smoothing and data coverage are stronger than the evidence supports. Figure 2 illustrates a smoother two-dimensional loss slice after AWM, but does not establish that smoothing causes the planning improvement or that the resulting gradients better reflect the environment dynamics. Quantitative measurements across multiple trajectories would strengthen this interpretation. The conclusion in Section 4.4 that “compute and data scaling alone will not make GBP viable” is particularly difficult to justify without a scaling experiment. OWM improves PointMaze from 68% to 88%, although it does not improve the other two tested environments. These results support a conclusion about the tested OWM configuration, rather than a general conclusion about the limitations of increased data coverage. The original-data mixing and collection/retraining schedule should also be clarified, since Algorithm 1 explicitly trains on the accumulated corrected dataset.

  3. The comparisons are insufficient to establish a broad planning-efficiency frontier. The paper should include comparisons with GRASP: Parallel Stochastic Gradient-Based Planning for World Models (Psenka et al., ICML 2026), Inference Time Policy Optimization for Offline RL with Differentiable World Models (Deb et al., 2026), and iCEM, with differences in problem settings and computational budgets clearly specified. The paper should at the least discuss these works as part of related works. Within the existing setup, AWM+CEM would help determine whether the improvement is specific to gradient-based planning. Furthermore, Figure 4 reports planning time only on PushT; the broader claim of an order-of-magnitude advantage across domains requires additional timing results. The use of WGL for GBP and terminal cost in the stated CEM algorithm also warrants a controlled comparison. This objective difference does not explain the improvement over baseline GBP, since WGL is described as part of the shared GBP setup.

  4. The uncertainty estimates do not establish the reliability of the comparisons. Appendix C.5 reports 50 evaluation trajectories and standard deviations computed from three batches of size 16. The treatment of the remaining trajectories is unclear, and variation across evaluation-batch averages does not measure variation across independently trained models. The paper should clarify this calculation and report uncertainty across independent training runs and evaluation episodes. Paired comparisons on shared initial states and goals would be particularly helpful for assessing smaller differences, such as 98% versus 92% on PointMaze. The validation procedure used to choose perturbation radii and planning hyperparameters should also be specified.

  5. Several implementation and evaluation details remain unclear. Equation (6) perturbs actions directly, whereas Appendix C.3 describes perturbations to action embeddings and separate proprioceptive embeddings. The exact perturbation location matters for interpreting and reproducing the method. Algorithm 2 also appears to carry perturbations between timesteps rather than initializing them independently for each transition. For Nymeria, Appendix D does not clearly specify the test-set construction, number of test sequences, metric units, or meaning of the shaded bands in Figure 5. It is also unclear how the statement that all planning results use MPC applies to this offline evaluation. The authors appropriately acknowledge that hardware deployment remains unresolved; the present results should be interpreted as evidence for offline pose planning, with limited support for the stronger physical-plausibility claims.

Questions for authors
  1. How does AWM compare with clean finetuning and random-perturbation training under matched data and optimization budgets? These comparisons would materially affect my assessment of the contribution.

  2. Are perturbations applied to physical actions or their embeddings? Since Equation (6) retains the original next-state target, are the perturbations intended to represent input corruption? Evidence that AWM preserves useful action sensitivity while improving planning would help explain why this objective works.

2 comments

We thank the reviewer for recognizing that the paper addresses a relevant problem: accurate next-state prediction does not necessarily give a useful objective for optimizing actions. We also appreciate the recognition that AWM is a simple training modification requiring no additional simulator interaction, and that the LeWM results suggest it extends beyond DINO-WM. Below we address the remaining concerns.

"...How does AWM compare with clean finetuning and random-perturbation training under matched data and optimization budgets? These comparisons would materially affect my assessment of the contribution." We agree, and we have run exactly these controls. Each ablation uses the same checkpoint, data and number of updates, with 3 seeds and 50 shared cases. Uniform noise fills the same L∞ box as AWM, and Gaussian noise uses AWM's radius as its standard deviation.

GBP success (%), mean ± std over 3 seedsOriginalCleanGaussianUniformAWM
Wall78.080.7 ± 1.276.7 ± 6.168.0 ± 2.094.0 ± 2.0
PointMaze70.054.0 ± 3.562.0 ± 6.054.0 ± 0.094.7 ± 1.2

AWM has higher observed mean success than clean finetuning and the random-perturbation controls on both tasks. The zero-perturbation setting left blank in Figure 8 gives 76.3% on Wall.

"...does not establish that smoothing causes the planning improvement or that the resulting gradients better reflect the environment dynamics. Quantitative measurements across multiple trajectories would strengthen this interpretation." We agree, and we have added this measurement. On Wall, we test whether the model's directional derivative of the planning loss agrees in sign with the simulator's along random action directions (finite differences):

Sign agreement (%), 0.05, mean ± stdOriginalCleanGaussianAWM
Wall70.0 ± 0.176.7 ± 0.077.8 ± 1.981.1 ± 1.0

AWM improves gradient sign agreement on Wall.

"...These results support a conclusion about the tested OWM configuration, rather than a general conclusion about the limitations of increased data coverage. The original-data mixing and collection/retraining schedule should also be clarified..." We agree, particularly since OWM itself substantially improves PointMaze. We have removed the statement that "compute and data scaling alone will not make GBP viable" and limited the conclusion to the tested configuration. However, it still stands that AWM significantly outperforms a reasonable attempt at applying DAgger and increasing data coverage. We further describe OWM's scheduling procedure below:

AspectOWM schedule
Update orderFor each original-data batch: one update on it, then one update on each simulator-corrected batch cached for that batch index
Corrected-data cacheAccumulates across epochs without eviction, consistent with Algorithm 1 training on the accumulated dataset
MixingNo fixed coefficient; the share of corrected data grows over training

"...The paper should at the least discuss these works as part of related works." We agree. A summary of the discussion is below:

MethodWhat it changesRelation to AWM
GRASPPlanner: joint optimization of actions and virtual states under soft dynamics constraints, parallel over the horizonComplementary; could run on an AWM model
iCEMPlanner: CEM with temporally correlated samples and elite memoryComplementary; could run on an AWM model
Deb et al.Inference-time policy optimization through differentiable world-model rollouts (offline RL)Requires a policy; AWM changes model training

"...Within the existing setup, AWM+CEM would help determine whether the improvement is specific to gradient-based planning." We ran AWM+CEM with the same model and WGL objective used for GBP. AWM improves both planners, so the improvement is not specific to gradient-based planning, although the gain is larger for GBP:

Gain from AWM (pp), shared WGLGBPCEM
Wall+16.0+9.3
PointMaze+24.7+4.7

Both GBP and CEM optimize action sequences using only the learned world model, without querying the environment simulator for predicted trajectories. Consequently, with the same model architecture, input dimensions, hardware and planning hyperparameters, planning times should be comparable across PushT, Wall and PointMaze.

"...variation across evaluation-batch averages does not measure variation across independently trained models... Paired comparisons on shared initial states and goals would be particularly helpful for assessing smaller differences, such as 98% versus 92% on PointMaze." We agree. We now run each finetuning condition with three random seeds and report the mean success rate ± standard deviation across those runs, as shown in the control table above.

AuthorsOpenReview ↗

Rebuttal Continued

"...The validation procedure used to choose perturbation radii and planning hyperparameters should also be specified." Planning hyperparameters are adopted from DINO-WM. Perturbation radii were chosen from a coarse sweep.

"...Are perturbations applied to physical actions or their embeddings? ... are the perturbations intended to represent input corruption? Evidence that AWM preserves useful action sensitivity..." Perturbations are applied to action and state embeddings. So yes, the perturbations are best read as input corruption. AWM trains the predictor to be robust to small corruptions of its inputs, rather than asserting that a different physical action leads to the same next state.

"...Appendix D does not clearly specify the test-set construction, number of test sequences, metric units, or meaning of the shaded bands in Figure 5." We recovered the raw evaluation records and have clarified Appendix D:

ItemProtocol
Test setOne held-out recording
Training data865 training recordings
Metric and unitsTerminal forward-kinematics XYZ error in meters, averaged over pelvis, head and both hands
Shaded bandsStandard error of the mean

Thank you again for your valuable feedback. We believe your feedback has substantially improved our paper. We would greatly appreciate it if you would consider increasing your score accordingly. Do you have any other questions we can address?

Reviewer_6gdAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why gradient-based planning exploits errors in latent world models. It compares simulator-based online finetuning (OWM) with adversarial finetuning (AWM) of latent states and actions. AWM improves planning across six simulated tasks and an offline egocentric pose benchmark, while OWM gives mixed results. The main practical claim is performance comparable to CEM at substantially lower planning cost.

Critical strengths
  1. The improvement is reasonably broad. Table 5 reports gains over the original gradient planner on all six tasks, including 42→76% success on Reacher and 17.73→39.46 ECD on Rope. Appendix E also shows gains on LeWM, which helps establish that the effect is not confined to one backbone.
  2. The paper connects planning performance to model exploitation rather than prediction loss alone. Figures 1–2 and the unsuccessful OWM comparison provide useful diagnostics, and AWM requires no simulator during finetuning.
Critical weaknesses
  1. The mechanism needs stronger controls. FGSM versus PGD and the radius sweep are helpful, but there is no matched random-perturbation baseline, and Figure 8 omits the zero-state/zero-action perturbation control. Additional clean finetuning and simple noise regularization could explain part of the gain. The selected two-dimensional landscape in Figure 2 does not by itself establish improved gradient fidelity. Likewise, one OWM configuration per task does not justify the broader conclusion that data or compute scaling cannot resolve the problem.
  2. The planning comparison changes several factors together. The headline comparison is AWM+GBP versus the original DINO-WM+CEM. All GBP variants share the weighted goal loss (Section C.4), so the within-GBP gains are meaningful; the cross-planner comparison still needs a shared objective. CEM on AWM and a hybrid CEM/gradient planner would make the attribution clearer. Figure 4 measures time only on PushT. Table 5 uses 50 trajectories, with deviations computed from evaluation batches rather than independent training runs, leaving uncertainty in the smaller margins.
  3. Others. Several definitions and references need correction. Section 2.2 reverses the gap's sign interpretation, and Section 1 says GBP eliminates rollouts although Algorithm 3 still rolls out the model. The bibliography dates Kavraki's PRM paper to 2002 rather than 1996 and mixes Finn and Levine's 2016 preprint with its 2017 ICRA record. These corrections would make the background and diagnostics easier to follow.
Questions for authors
  1. With matched finetuning updates and data, how do AWM, clean finetuning, and random latent/action perturbations compare?
  2. Does the performance–time advantage hold when GBP and CEM use the same world model and goal loss, and when compared with a hybrid CEM/gradient planner?
  3. How stable are the main performance differences across independent training runs?
2 comments

We thank the reviewer for noting that AWM's improvement is reasonably broad across all six tasks and extends to LeWM. We also appreciate the recognition that the paper connects planning performance to model exploitation rather than to prediction loss alone. Below we address the remaining concerns.

"...there is no matched random-perturbation baseline, and Figure 8 omits the zero-state/zero-action perturbation control. Additional clean finetuning and simple noise regularization could explain part of the gain." We agree, and we have added these controls. Each arm uses the same checkpoint, data and number of updates, with 3 seeds and 50 shared cases. Gaussian and Uniform noise are drawn at AWM's scale.

GBP success (%), mean ± std over 3 seedsOriginal (1 model)CleanGaussianUniformAWM
Wall78.080.7 ± 1.276.7 ± 6.168.0 ± 2.094.0 ± 2.0
PointMaze70.054.0 ± 3.562.0 ± 6.054.0 ± 0.094.7 ± 1.2

We also added zero-radius and sign-randomized runs:

Wall controlSuccess (%), mean
AWM94.0
Zero radius76.3
AWM magnitudes, randomized signs75.7

"...The selected two-dimensional landscape in Figure 2 does not by itself establish improved gradient fidelity." We agree, and we have added a quantitative test. On Wall, we compared the model's and the simulator's directional derivatives of the planning loss along random action directions (finite differences). We counted how often their signs agree:

Sign agreement (%), 0.05, mean ± stdOriginalCleanGaussianAWM
Wall70.0 ± 0.176.7 ± 0.077.8 ± 1.981.1 ± 1.0

AWM improves gradient sign agreement.

"...one OWM configuration per task does not justify the broader conclusion that data or compute scaling cannot resolve the problem." We agree, particularly since OWM itself substantially improves PointMaze. We have removed the statement that "compute and data scaling alone will not make GBP viable" and limited the conclusion to the tested configuration. However, it still stands that AWM significantly outperforms a reasonable attempt at applying DAgger and increasing data coverage.

"...the cross-planner comparison still needs a shared objective. CEM on AWM and a hybrid CEM/gradient planner would make the attribution clearer." We agree. With the same model and WGL objective:

Success (%), mean ± std for AWM (3 seeds)GBPCEM
Wall, original78.082.0
Wall, AWM94.0 ± 2.091.3 ± 3.1
PointMaze, original70.094.0
PointMaze, AWM94.7 ± 1.298.7 ± 2.3

AWM's central practical goal is to make GBP more effective. At these budgets, PointMaze GBP rises from 70.0% to 94.7% (+24.7 pp), while CEM rises from 94.0% to 98.7% (+4.7 pp); the gap between planners shrinks from 24.0 to 4.0 pp. On Wall, GBP gains 16.0 pp compared with 9.3 pp for CEM, and AWM+GBP achieves the highest mean success. CEM also uses the learned world model to score candidates, so improvements to that model can benefit search as well. Higher AWM+CEM success on PointMaze is therefore consistent with the main result: AWM substantially improves GBP, making it competitive with search. We also ran a CEM/gradient hybrid (GradCEM, with two Adam refinements per CEM iteration):

Wall success (%), mean ± std for AWMGBPCEMGradCEM
Original78.082.086.0
AWM94.0 ± 2.091.3 ± 3.187.3 ± 3.1

"...Figure 4 measures time only on PushT." Both GBP and CEM optimize action sequences using only the learned world model, without querying the environment simulator for predicted trajectories. Consequently, with the same model architecture, input dimensions, hardware and planning hyperparameters, planning times should be comparable across PushT, Wall and PointMaze.

"...deviations computed from evaluation batches rather than independent training runs, leaving uncertainty in the smaller margins." We now run each finetuning condition with three random seeds and report the mean success rate ± standard deviation across those runs, as shown in the control table above.

"...Section 2.2 reverses the gap's sign interpretation, and Section 1 says GBP eliminates rollouts although Algorithm 3 still rolls out the model." Thank you for catching these. With Δ = L_model − L_env, an optimistic model gives Δ < 0. We have corrected the text and use |Δ| for the magnitude of the gap. We have also removed "eliminates rollouts," since GBP still rolls out the learned model and backpropagates through it. Kavraki et al. is now cited as 1996, and Finn & Levine's 2016 preprint is separated from their 2017 ICRA paper.

Thank you again for your valuable feedback. We believe your feedback has substantially improved our paper. We would greatly appreciate it if you would consider increasing your score accordingly. Do you have any other questions we can address?

Reviewer_6gdAOpenReview ↗

Response to author

Thanks for the new results. These address my main concern about whether the gains come from adversarial training. One point is still unclear: does the time–performance advantage in Figure 4 hold when GBP and CEM use the same model and WGL objective? Could you provide measured timings for this comparison?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.