acceptodds
Withdrawn from ICLR 2027

Trust Before Use: When Do World-Model Quality Metrics Predict Control?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World models are being scaled to billions of parameters and fine-tuned on robot data in the hope of unlocking new skills, yet it remains unclear which properties of a world model make it useful for downstream control. Practitioners typically judge world models by prediction loss or video fidelity, and learn whether a model is useful only after training and evaluating a policy, a slow and costly loop. We ask when world-model quality metrics predict control. We find that prediction loss and video fidelity can mislead: a world model with lower prediction loss can yield a worse policy, and a world-action model with far worse video fidelity can control without a detectable loss in success. We study the expected signal-to-noise ratio (ESNR) of policy gradients computed through a world model, which measures the gradient signal the model offers to policy extraction and can be computed during training without a trained policy or environment interaction. Across policy-extraction methods and world-model architectures on DMControl and Meta-World, ESNR tracks downstream performance more closely than prediction loss, and it tracks the planning performance of large pretrained world models at a fraction of the evaluation cost. On a 6B-parameter world-action model trained by behavior cloning, a gradient signal-to-noise ratio of the action loss orders the four model variants exactly as closed-loop success does, whereas LPIPS and SSIM rank the most successful variant last. Visualizations and code will be available at https://trust-before-use.github.io.

Primary area:
reinforcement learning
Keywords:
world models, model-based reinforcement learning, evaluation metrics, model selection, policy gradient estimators, objective mismatch, robot learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pcqEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The papers motivation is that in model based reinforcement learning standard prediction loss might not give us a good estimate of how good the downstream performance of the trained policy would be. The authors propose that using the average signal-to-noise ratio is a good measure and that policy gradient algorithm that observe high SNR (ESNR in this work) are expected to have a higher downstream performance.

The authors try out a small toy problem with discotinuity in dynamics w.r.t actions. they traine reconstruction based and reconstruction free model based RL (MBRL) methods and different policy training methods and evaluate and compare the ESNR with the downstream control performance.

The policy's downstream performance broadly agrees with ESNR with positive correlation. and the authors are honest about when ESNR can be a usefull measure and when it is not (app. D)

Critical strengths
  1. I do believe that with advent of world models a good and cheap estimate of how good downstream policy learning would be is very important before we throw training compute to actualy learn the policy and is also well supported by existing theory in model-erro and its downstream effects to policy learning.
  2. On experiments with larger vision envocers such as Dino based and Jepa based encoders used in the paper, the ordering of downstream performance and ESNR is consistent which is a strong positive.
  3. ESNR computation is cheap and easy to implement.
  4. The paper is written in a relatively straightforward manner
Critical weaknesses
  1. The role of ESNR is not consistent: The authors claim a good world model for policy learning should be both accurate and have high ESNR, yet the authors also claim that ESNR is a poor metric if the model accuracy is low. these seem to be conflicting claim and I am unable to consolidate the very precise motivation. from my perspective, it seems that ESR is a good metric you should also look at but not completely rely on.

  2. As an RL practitioner I would always be interested in which backbone/enocder or existing model should I use to finetune and then train my donwstream control policy, but it is unclear whether for the same policy training algorithm whether ESNR is usable as a metric across architectures?

  3. The action distribution which is used to evaluate ESNR is neither the data collecting policy not policy that we are aiming to train, the "probe" uses a zero mean gaussian action distribution and it is unclear whether beyond toy settings, this distribution is good enough to give a good enoguh estimate of the ESNR

  4. Limited novelty: ESNR already comes from Parmas et al and PWM (Georgiev et al) already have shoed that regularized world models yield more reliable gradients.

Questions for authors
  1. Algorithm 1 (line 13) weights each step’s score by the immediate reward rather than the return-to-go, and the accumulated return J is never used. Is this the estimator used in the experiments?
  2. Does the ranking hold when ESNR is computed on dataset actions, or on actions from the policy being optimized, rather than on zero-mean Gaussians?
  3. How should ESNR be compared across methods, given that each CEM sample aggregates many rollouts while each FoG or ZoG sample uses one? Does normalizing by the number of model rollouts per sample change the ranking in Experiment 1?
  4. For fast-wam: Why should a higher SNR of the held-out action-loss gradient at the final checkpoint indicate a better policy? A consistent nonzero mean gradient on held-out data could also reflect systematic error.
  5. How does the ESR score change as the amount of compute and data used to train the model increases. I am assuming the ESR increases monotonically and so does policy performance but is this empirically suported as well.
  6. If in one short motivation, how do the authors expect the ESR to be used, from my read I am a bit confused if it is a pre-evaluation diagnostic for policies trained using the model, or is it to give an estimate of downstream learnd policy before we start to train the policy?
Reviewer_ZDoyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether inexpensive diagnostics of a world model can predict downstream control performance before policy training or environment evaluation. It applies expected signal-to-noise ratio (ESNR) to action-related gradient estimates and compares it with prediction loss and visual fidelity. Experiments cover policy extraction and architecture comparisons on DMControl and Meta-World, planning with pretrained visual models, and four Fast-WAM variants. The Fast-WAM experiment uses a different diagnostic, the SNR of action-loss gradients. The reported results suggest that prediction quality can disagree with control performance, although the evidence for ESNR as a reliable selection criterion is substantially less conclusive.

Critical strengths
  1. The paper addresses a relevant practical question: evaluating control through full policy extraction can be expensive, so a useful offline diagnostic would have value. Comparing prediction loss with downstream performance is an appropriate starting point for this investigation.

  2. The Fast-WAM comparison keeps the action-training objective and inference procedure fixed while changing video-training targets or noise schedules. In particular, the blurred-target variant illustrates that worse visual fidelity can coexist with similar control success. This is a useful controlled observation, even though it does not establish the reliability of the proposed gradient diagnostic.

Critical weaknesses
  1. Inconsistent metric definitions. Equation (3) uses policy-parameter gradients, but the algorithms use action gradients or CEM update vectors, and Fast-WAM uses a different objective and aggregation. Algorithm 1 also weights action scores by immediate rewards, omitting later reward effects. These differences make the cross-method rankings difficult to interpret without checking the implementation.

  2. Insufficient evidence for reliable model selection. Appendix D reports near-zero or negative correlations on some tasks and high ESNR at checkpoints with near-zero return. The architecture comparison also changes the policy-extraction method. Fast-WAM's perfect ranking uses only four variants, one training seed, and eight probe clips. These results do not adequately support the broad reliability claims.

  3. The paper is not well prepared. Citation formatting contains noticeable errors, including unmatched parentheses and citing the 2020 Dreamer paper for DreamerV3 in Section 3.3. Figures 1 and 3–6 use uneven font sizes and styles, with several labels and legends too small at normal page scale. Together with incomplete experimental specifications, these issues make the submission harder to read and verify.

Questions for authors
  1. Does the implementation used for the reported ZoG results follow Algorithm 1 literally? Please specify the differentiated parameter, the return or return-to-go weighting, and the vector aggregation for each estimator. If the printed algorithm differs from the implementation, would correcting it change the reported correlations or rankings?

  2. What selection or readiness decision can ESNR reliably make given the failures in Appendix D? Please provide correlations within tasks and comparable training stages, with uncertainty estimates, and clarify which architecture comparisons hold the policy-extraction method fixed. This would materially affect whether I regard the metric as useful for the stated purpose.

  3. How stable is the Fast-WAM ranking to resampling the eight probe clips and the evaluation episodes? Please clarify the negative SNR estimate shown in Figure 6, the use of a step-2,750 probe for step-2,780 performance, and whether the leading variants have distinguishable paired success differences. Also, please provide the location of the anonymous reproducibility materials: the reproducibility statement says that code, data, and checkpoints have been released, but the link in the abstract returns HTTP 404.

  4. How were the surfaces in Figure 1 generated? If they are measured landscapes, please specify the models, checkpoints, and visualization procedure. If they are schematic, please label them accordingly and identify the source of any reused graphical elements.

Reviewer_ZSZAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores the use of the expected signal-to-noise ratio of policy gradients as a useful indicator for downstream policy extraction performance of world models. The authors show on a wide range of experiments primarily focusing on object manipulation and locomotion tasks spanning different policy extraction methods, world model training objectives, and model scales that ESNR consistently correlates positively with downstream policy performance. This is in contrast to traditional metrics based on prediction or reconstruction losses which fails to consistently establish this correlation. These results suggest that the ESNR of a policy gradient is an effective training time diagnostic tool to predict downstream performance that is both simple to implement, and efficient to compute.

Critical strengths
  • The paper is well-written and presented. The ideas, motivations, and experiments are intuitive, easy to follow and concisely bring the reader from point A to point B. The presentation of the experimental results are also well organized and easily digestible.

  • The general motivation and problem the paper addresses is well-motivated and quite relevant as the literature moves towards larger and larger world models that are increasingly prohibitive to train and evaluate. While the core idea of the paper is not particularly novel, its rigorous experimental evaluations provide useful insights in today's model landscape towards better understanding which qualitative metrics of a world model are useful for downstream control performance.

Critical weaknesses
  • The contribution of this work is limited to empirical correlation-based observations rather than offering deeper causal insights on a metric (ESNR) that is already known to be a useful indicator for downstream performance. The theoretical insight provided in Equation 2 only provides a weak explanation for why prediction losses are not always a useful metric for policy extraction, but does not say anything about whether the ESNR is a stronger metric in that regard.

  • Since the fundamental idea is not novel, this puts a lot of pressure on the experimental results to be significant both in rigor and in novelty. The paper repeatedly argues that ESNR is empirically correlated with downstream performance, but fails to provide a deeper understanding of its limitations and failure modes, which is crucial for both researchers and practitioners interested in using this metric in the absence of overwhelming evidence that it is an effective diagnostic tool. See questions for additional details.

Questions for authors
  • The idea that the policy objective's landscape is critical to effective policy extraction is a compelling one that is noted by the authors several times (Figure 1, line 217, line 292, lines 360-366) and also well-known in the literature. While the authors suggest that high ESNR implies better optimization landscapes (line 73), I am not sure that this fact has been properly argued. I understand that the idea of a "good" optimization landscape is difficult to quantify, but can the authors speak more to this equivalency that high ESNR   ⟹  \implies "good" landscapes? Is there simply an implicit understanding that these two concepts are synonyms, because low ESNR necessarily implies "spiky" landscapes by definition? Is it possible to find a counterexample where a lower ESNR can be observed on a worst landscape?

  • While the results in Figure 4 show a general trend that the average ESNR across policies extraction methods and across world models are in fact correlated with performance, there seems to be an amount of non-negligible variance of ESNR as a predictor at a more fine-grained level which would make it unreliable without a sufficiently large population, putting an asterisk on the efficiency of ESNR as an indicator if there is too much variance at an individual level. For example, the confidence intervals in Figures 4 (right bar graph), Figure 5, and Figure 6 overlap several times, meaning that it is not a reliable indicator on a single seed. It would be interesting to further quantify the correlation of the ESNR at the per-seed level for example. If I were to train N different world models, and obtain their respective ESNRs, how confident should I be that the highest ESNR model will deliver the best downstream policy performance? What do the individual datapoints in the scatterplots of Figure 4 represent? Within each category (color), the correlation seems rather loose, why is that?

  • The authors note that ESNR alone is not sufficient in the conclusion, especially early on in training due to gradient bias. This seems like an important shortcoming that should be elucidated in the main text by providing counter-examples where ESNR is a poor indicator when the gradient bias is large and providing at least some level of intuition on when to trust the ESNR if that is the case.

  • Why is the ESNR for DreamerV3 consistently 0 in Figure 4?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.