acceptodds
Under review as a conference paper at ICLR 2027

The Correction Budget: When Test-Time Refinement Can and Cannot Fix World-Model Rollouts

Abstract

Test-time refinement—predict a latent, then project it onto a learned constraint—is an appealing recipe against the compounding drift of action-conditioned world models. We build the recipe in its strongest practical form (a transition energy model trained against the world model's own drifted rollouts, fixing verifier AUROC from 0.06 to 0.88–0.94) and then take it apart. Across three environments, two observation modalities, and fifteen (world-model, corrector) systems, we find: (i) drift does not compound exponentially—the off-manifold amplification premise fails; (ii) the verifier is a pure legality detector: its gradients, regression targets, and selection weights carry no information about the direction back to the truth; (iii) a maximal-input probe—the world model fine-tuned on its own rollout contexts—shows that open-loop error has a predictable fraction governed by two separable channels, reducible model bias (data starvation) and off-manifold geometry (representation); (iv) a correction-budget law: no test-time corrector measurable in the rollout's inputs can beat . The bound is saturated exactly where per-step alignment is high—pixel latents, where legality truth and refinement recovers up to 46% of drift—and correctly forecasts the near-zero gains on near-manifold proprioception.

Primary area:
reinforcement learning
Keywords:
world models, test-time refinement, energy-based verification, model-based reinforcement learning, open-loop drift, latent representations

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_WgBbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates when test-time refinement can correct errors in action-conditioned world-model rollouts. Across DeepMind Control locomotion environments, the authors evaluate proprioceptive states and DINOv2 visual representations. They report predominantly linear, rather than exponential, error growth and improve the transition-verifier AUROC through paired-drift negative training. However, gradient-based correction, directional regression, and verifier-weighted particle selection do not consistently improve prediction fidelity. A fine-tuned Δ-probe estimates how much prediction error is learnable from rollout inputs, highlighting differences in model bias and manifold geometry. Finally, the authors derive a correction-budget bound relating achievable error reduction to the fraction of unexplained prediction variance.

Critical strengths

Some of the strengths of this paper are:

  1. The paper examines an important problem in world-model-based prediction: whether test-time refinement can mitigate accumulated rollout errors. The paper provides some useful empirical evidence about the relationship between transition plausibility and prediction fidelity.

  2. Although the paper lacks a lot of details, I particularly find the Δ\Delta-probe experiments quite interesting. It allows distinguishing between the predictability of rollout errors and the geometric alignment of correction directions with the ground truth.

Critical weaknesses

The paper seems to perform a benchmarking study with a proper, coherent analysis and description of key research questions. The key weaknesses of the current draft are as follows:

  1. The paper investigates an interesting problem, but its experimental contribution is not presented through a sufficiently coherent and systematic framework. Much of the work reads as an empirical benchmarking or diagnostic study of different correction mechanisms. However, the authors do not clearly establish the research questions, comparison criteria, and experimental controls that would make such a study convincing.

  2. A fundamental methodological question concerns the motivation for using an energy-based verifier in the first place. The authors have access to ground-truth rollout trajectories and can generate the corresponding drifted predictions from their frozen world model. Therefore, they also have the supervision necessary to train a network to predict the true correction residuals directly or to fine-tune the world model on its own drifted contexts. Indeed, Section 4 introduces a supervised directional regression head, and Section 5 subsequently introduces the Δ-probe, which fine-tunes a copy of the original world model to predict ground-truth next states from drifted contexts. In summary, it is unclear what motivates introducing the energy verifier instead of directly learning correction residuals or fine-tuning the world model.

  3. Important details of the world-model setup and input representations are missing (Section 2). The paper provides insufficient information about the construction and preprocessing of the proprioceptive state representations. This is particularly concerning because the paper's central findings depend on differences between proprioceptive and pixel-based representations. The authors should clarify the following: a. Which physical variables constitute the reported 17–24-dimensional proprioceptive states? What is the exact dimensionality in each environment? b. Are all simulator observation components used directly, or are some selected, transformed, or excluded? c. Are the proprioceptive vectors passed directly to the Transformer after standardization, or is an additional representation-learning step involved? d. How are the normalization statistics estimated, and which dataset split is used? e. What are the complete world-model architecture, training objective, and relevant optimization settings?

Although Appendix C provides some reproduction details, these questions remain insufficiently addressed.

  1. The verifier training objective is not sufficiently specified (Section 3.2). In particular, the authors should provide the exact loss function used to train EψE_\psi, including the treatment of positive and negative candidates, the negative-sampling procedure, and the mechanism used to anchor energy zero for genuine on-manifold transitions at (k=0). The treatment of ground-truth candidates under drifted conditioning k>0k>0 also requires clarification.

  2. Section 4 is arguably the most important experimental section of the paper because it examines whether the trained verifier can actually improve rollout predictions. However, it provides comparatively little information about the implementation of the three correction strategies. a. For gradient-based refinement, the authors should explain how the residual scale σresid\sigma_{\mathrm{resid}} is estimated, how the trust region and gradient normalization are implemented, and how the number of refinement steps and early-stopping criteria are determined. b. For regression-based correction, the manuscript does not sufficiently describe the directional head's architecture, its exact training loss, or whether any parameters are shared with the energy verifier. c. For particle-based selection, the authors should specify the particle count, proposal distribution, perturbation magnitude, resampling procedure, and selection or tuning of λ\lambda.

  3. The authors report a substantial deterioration in prediction performance when applying particle-based imagination to 1,024-dimensional DINOv2 representations. They attribute this deterioration to perturbations pushing predictions away from the data manifold. However, DINOv2 representations are structured, and arbitrary perturbations in the feature space do not necessarily yield representations of realistic images or physically meaningful states. Consequently, the failure may originate from the candidate-generation mechanism rather than from verifier-weighted selection itself.

  4. The primary experimental protocol initializes the world model using 16 ground-truth states. While this is a reasonable setting for studying open-loop error accumulation, it assumes an accurate initial observation history. The authors do not examine the robustness of their findings when the initial proprioceptive observations are noisy or inaccurate. A straightforward additional experiment would be to introduce controlled random noise into the initial observations and evaluate how the different correction mechanisms behave. This would help determine whether the reported findings remain consistent under less-ideal observation conditions.

Some additional comments:

  1. Furthermore, the authors repeatedly characterize their pixel-based world models as data-starved. Although the reported validation-to-training error ratios and the data-availability experiment in Section 5.3 provide some supporting evidence, the explanation is not fully established. The limited number of training-data configurations makes it difficult to isolate insufficient data as the principal cause. Related claims should also be supported by appropriate citations to existing literature on world-model overfitting, generalization, and sample efficiency.

  2. Several established techniques are introduced without adequate methodological references or implementation explanations. For example, Section 4 employs Sequential Importance Resampling (SIR) and refers to PETS-style particle control, but the relevant methodological background and implementation choices are insufficiently documented. This is true for many other such critical claims and approaches used in this paper. In general, the paper's overall writing and presentation need significant rework.

Questions for authors

The current draft lacks sufficient information on critical parameters, training objectives, loss functions, experimental setups, and relevant hyperparameters. I would request that the authors carefully review the critical weaknesses to understand the key questions.

Reviewer_Za6zOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates when test-time refinement can improve the fidelity of action-conditioned world-model rollouts. The authors first show that naive transition verifiers suffer from a conditioning mismatch and propose paired-drift negative training to improve their discrimination performance. They then demonstrate that accurate verification does not necessarily translate into effective correction, and introduce a delta-probe to estimate the predictable fraction of rollout error. Based on this analysis, the paper proposes a correction-budget law derived from the optimality of conditional expectation and studies how correction effectiveness depends on model bias and representation geometry. Experiments across proprioceptive and visual latent representations suggest that refinement is substantially more effective in the latter, while improved prediction fidelity does not necessarily improve downstream control performance.

Critical strengths
  1. The paper addresses an important and relatively underexplored question of when test-time refinement is beneficial for world-model rollouts.
  2. The distinction between verifier discrimination accuracy and actual correction effectiveness is interesting and supported by several complementary experiments.
  3. The analysis of error predictability and its relationship to representation geometry provides potentially useful insights into the limitations of refinement methods.
  4. The paper includes multiple ablations, negative controls, and a downstream planning experiment, demonstrating an effort to investigate both successful and unsuccessful correction regimes.
Critical weaknesses

1. Limited theoretical novelty and a potentially vacuous correction-budget bound.

The central theoretical result (Proposition 1) follows directly from the standard MSE optimality of conditional expectation. While its application to world-model correction is reasonable, the proposition itself does not establish a fundamentally new limitation specific to world models or test-time refinement. More importantly, in deterministic and sufficiently observed environments, the ground-truth future state may be fully determined by the initial context and action sequence, making the Bayes-optimal FVU zero and the resulting bound trivial. The authors acknowledge this possibility in Appendix A, but its implications are not adequately reflected in the main claims. Consequently, it remains unclear what nontrivial theoretical insight the proposed correction-budget law provides beyond a standard statistical result.

2. The empirical correction-budget estimate is not a theoretically valid lower bound.

The connection between the theoretical result and the empirical correction-budget estimate is insufficiently justified. Lemma 1 establishes that the probe-based FVU is an upper estimate of the Bayes-optimal FVU. Therefore, the theoretical guarantee does not imply that the achieved correction-error ratio must exceed the proposed probe-based bound. Nevertheless, Figure 4 presents this estimate as a bound and interprets proximity to it as evidence supporting the correction-budget law. Furthermore, the estimate depends on the capacity, training procedure, and input representation of the delta-probe, and the paper explicitly acknowledges that iterative correctors may outperform it. The same probe is also involved in estimating the budget and achieving some of the reported correction results, making the observed agreement less convincing as independent validation. A more rigorous separation between the theoretical information floor and the empirical, model-dependent estimate is necessary, together with validation using independently designed correctors.

3. The interpretation of the verifier as a legality detector is insufficiently supported.

A central claim is that the verifier learns transition legality rather than information about the direction toward the ground truth. However, the paired-drift training objective primarily distinguishes ground-truth next states from world-model-generated candidates under drifted conditioning states. Such discrimination is not necessarily equivalent to verifying physical transition validity. In particular, pairing a drifted current state with the ground-truth next state does not guarantee that the resulting transition is physically valid under the specified action. Therefore, high AUROC may reflect discrimination between real and model-generated states rather than genuine legality assessment. Without an independent measure of physical transition validity, the conclusion that "legality is not truth" is not fully established. Additional experiments separating transition validity, distributional plausibility, and ground-truth fidelity would strengthen this claim.

4. Limited methodological novelty and insufficient differentiation from prior work.

Several major components are closely related to established approaches. In particular, training models on their own generated rollout states and studying self-correction have been investigated in hallucinated replay and self-correcting world models [R1, R2]. The proposed delta-probe largely repurposes this idea as a diagnostic tool, while the correction-budget proposition follows from a standard statistical optimality result. Although the combination of these components provides an interesting analytical perspective, the paper does not sufficiently establish what fundamentally new methodological or theoretical understanding is obtained beyond prior work. The main contribution therefore relies heavily on the empirical findings, whose generality and predictive utility remain insufficiently demonstrated.

5. Insufficient experimental generality and limited comparison with stronger alternatives.

Despite reporting 15 systems, the experiments are largely variations of a single world-model architecture evaluated on a small set of DeepMind Control locomotion environments. All visual representations are based on DINOv2 CLS features, and the visual world models exhibit substantial training-data limitations. Consequently, the observed differences between proprioception and visual latents may be confounded by representation properties, model capacity, and data availability rather than being attributable primarily to observation modality. The claim of modality-dependent geometry is therefore stronger than the evidence supports. Moreover, the experiments do not establish whether the proposed diagnostic framework generalizes to substantially different or stronger world models and correction strategies. More diverse architectures, representations, and independent correction baselines would be important for establishing the significance of the empirical conclusions.

6. The claimed falsification of exponential error compounding is not sufficiently justified.

The paper claims that the exponential-compounding premise fails because the observed rollout-error curves are better described by linear growth. However, theoretical analyses of error propagation, such as [R3], provide bounds under particular assumptions rather than asserting that errors must empirically grow exponentially. Furthermore, the geometric model used in Section 3.1 approaches the linear model as its amplification parameter approaches one. Thus, the reported consistently worse fitting performance of the geometric model requires clarification of the fitting procedure, parameter constraints, and optimization settings. The fitted growth parameter should also not be conflated with the actual Lipschitz constant of the world model. Overall, the experiments support approximately linear error growth in the evaluated settings, but not the broader claim that the underlying amplification premise has been falsified.

 

[References]

[R1] "Model Regularization for Stable Sample Rollouts," UAI, 2014.
[R2] "Self-Correcting Models for Model-Based Reinforcement Learning," AAAI, 2017.
[R3] "Lipschitz Continuity in Model-based Reinforcement Learning," ICML, 2018.

 

I also have some minor concerns as follows:

  1. Several numerical inconsistencies require clarification. For example, Section 6 reports a -2.7 percentage-point bound violation for walker-pix-200, whereas Appendix Table 3 indicates a gap of approximately -0.3 percentage points. The reported probe-bound values also appear inconsistent across these sections.
  2. The theoretical bound is defined using RMS error, while much of the empirical evaluation uses the mean of Euclidean norms. Although the paper discusses this distinction, complete system-level RMS results and confidence intervals should be provided to verify the reported 14/15 result.
  3. Table 1 reports exactly the same GT-conditioned AUROC (0.062) across five different environments. This unusual consistency warrants verification or explanation.
  4. The claimed monotonic improvement with probe quality is not fully reflected in Appendix Table 4, where the estimated bound increases from the medium to the canonical configuration.
  5. The claim that the verifier contains approximately zero truth-direction information appears somewhat overstated, particularly for visual latents where the reported gradient alignments are non-negligible relative to random directions in high-dimensional spaces.
  6. The use of the average of eight nearest training states as a manifold-return direction is heuristic, and its reliability as a measure of the true data-manifold geometry is not sufficiently established.
  7. The downstream planning experiment is limited to one environment with 16 episodes per condition. Additional environments and planning objectives would be needed to support broader conclusions about the relationship between fidelity and control.
  8. Several claims, such as the existence of exactly three ways to extract corrections from a scalar energy and the modality-locked nature of representation geometry, are stated more strongly than the experiments or theoretical arguments justify.
Questions for authors

Please refer to the Weaknesses section for detailed concerns.

  1. How does the proposed correction-budget law provide a nontrivial limitation beyond the standard optimality of conditional expectation, particularly in deterministic and sufficiently observed environments where the Bayes-optimal FVU may be zero?

  2. Given that the probe-based FVU is an upper estimate of the Bayes-optimal FVU, what justifies treating the resulting estimate as a lower bound on achievable correction error? Can its predictive validity be established independently of the probe-based corrector?

  3. What evidence establishes that the paired-drift verifier measures physical transition legality rather than merely distinguishing ground-truth states from model-generated predictions?

  4. How can the authors disentangle the effects of observation modality from differences in representation, model capacity, and data availability?

  5. Could the authors reconcile the inconsistent bound values and reported gaps across Section 6 and Appendix Table 3, and provide the system-level RMS results supporting the claimed 14/15 agreement?

Reviewer_Zrp8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates when test-time refinement can correct world-model rollout errors. It evaluates energy-based verifier and three correction strategies, finding that transition legality does not necessarily imply prediction fidelity. The authors introduce a Δ-probe to estimate error predictability and derive a correction-budget law. Experiments cover three control environments, two observation modalities, and 15 model configurations.

Critical strengths
  1. The paper provides an interesting analysis of test-time refinement, particularly the distinction between transition legality and prediction fidelity.

  2. The Δ-probe and correction-budget analysis offer useful insights into when rollout errors may be correctable.

Critical weaknesses

W1. Insufficient methodological details. Several key procedures, including verifier training, gradient refinement, and probe-based correction, lack explicit mathematical formulations and implementation details.

W2. Insufficient references. Several claims lack adequate references or justification, such as the assumption that off-manifold rollouts lead to a Lipschitz constant greater than one (Sec. 3.1), and the characterization of GT-conditioned corrupted negatives as standard verifier training (Sec. 3.2).

W3. In the verifier evaluation, the high AUROC may not necessarily reflect its ability to assess physical transition legality, as paired-drift positives combine drifted conditioning states with ground-truth next states that may not constitute physically legal transitions. Consequently, the high AUROC may primarily indicate successful discrimination between the constructed positive and negative distributions rather than genuine transition legality.

W4. The proposed two-dial audit relies on observed ground-truth trajectories to estimate probe-FVU and geometry alignment. Although such observations can be collected during online interaction, it remains unclear how reliably these estimates generalize to unexecuted candidate action sequences, particularly under distribution shift.

Questions for authors

See weaknesses.

Reviewer_eiJjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper assumes that verifier-based post-generation correction can address open-loop prediction drift in world models and investigates its effectiveness and limitations. The study finds that the benefit of correction is limited by the predictable portion of the open-loop error, and this upper limit can be estimated in advance. For proprioceptive inputs close to the data manifold, correction is almost ineffective or even has a negative impact. However, for image feature inputs deviating from the data manifold, test-time correction can reduce open-loop drift by 29–46%, and even a simple nearest neighbor method can reduce it by 22–31%.

Critical strengths

None

Critical weaknesses

My main concern is that the practical value and novelty of this paper are rather limited. First, the existing literature does not provide sufficient support for treating “verifier + test-time correction” as a mainstream approach to error correction in world models. It seems more like a potential solution introduced by the authors from other fields. Therefore, the conclusions of this paper have limited guiding value for practical method selection. Second, all experiments are based on a single deterministic world model trained by the authors themselves and three DMC motion control environments, lacking validation on mainstream open-source world models, stochastic environments, and robot manipulation tasks, thus limiting the generalizability of the conclusions.

Questions for authors

My specific comments are as follows:

  1. The claim that “verifier + test-time correction” is a mainstream approach for error correction in world models lacks support from the literature. To my knowledge, existing methods mainly focus on rain-time correction, limiting the rollout horizon and avoiding unreliable predictions based on uncertainty. Test-time correction with verifiers is mainly popular in LLMs and diffusion models. For world models it is better positioned as a potential direction that remains to be explored. Therefore, the findings offer limited practical guidance for addressing error accumulation in world models.
  2. Insufficient evidence for the claim that drift grows linearly rather than exponentially. This conclusion is based on a self-trained world model following the DINO-WM architecture and experiments on only three DMC control tasks. I suggest validating this finding on established open-source world models such as DreamerV3, TD-MPC2, andthe official DINO-WM checkpoints.
  3. This paper uses a self-trained deterministic world model based on DINOv2 CLS features, while models such as DreamerV3 and IRIS are stochastic. This difference may affect the core findings. In the verifier-based selection experiment, the deterministic model requires artificial perturbations to generate candidates, whereas stochastic models can sample directly from their learned distributions. Moreover, sampling noise in stochastic models may affect the predictable error fraction measured by the Δ-probe and the estimated correction budget. Therefore, the applicability of these findings to stochastic world models remains unverified.
  4. The image world model in this paper was trained with only 100 trajectories. The single-step error on the validation set reached 14–17 times that of the training set, indicating severe overfitting due to insufficient data. This is far from the training scale of mainstream world models. It is recommended that the authors verify whether the conclusions still hold true on a model trained with larger-scale data.
  5. Insufficient reproducibility. The scale of the world model, training objectives, hyperparameters, and the amount of training data under the proprioceptive setting are not disclosed.
  6. The paper contains several inconsistencies that require verification by the authors: the abstract states that the AUROC after repair is 0.88–0.94, while the main text and Table 1 state 0.81–0.94; the AUROC of the ordinary validator in Table 1 is exactly the same (0.062) in five environments and two modalities.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.