acceptodds
Under review as a conference paper at ICLR 2027

What Accurate Predictions Hide: Tracing Action-Effect Errors in World Models via Paired Rollouts

Abstract

World models support planning and policy learning by predicting how actions change future outcomes, yet accurate predictions of the usual outcome can conceal errors in action dependence. We use paired rollouts to compare how alternative action sequences change environment and model outcomes from a shared history. Because each generated step conditions the next, we use differential transport to decompose action-effect error exactly into local discrepancies propagated by later model dynamics, measuring their contributions through controlled transition substitutions. Across ten Craftax world models, cancellation and delay reveal outcomes predicted without the required action or before its execution. Substantial action-effect errors persist in about 13% of assessments passing per-branch return checks. Supervising next-step outcomes of alternative actions raises the rate of correctly timed event predictions under delay by over 40 percentage points relative to matched replay on average. Controlled substitutions show that using the updated generator for only the first transition suppresses later false events across three training repetitions of a shared pretrained model, while correct event generation changes little and remains rare. These findings make action dependence a concrete target for evaluation and training, with controlled substitutions revealing how local model updates change delayed consequences.

Primary area:
reinforcement learning
Keywords:
world models, model-based reinforcement learning, model evaluation, action effects, error propagation, counterfactual prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_41xAOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that accurate trajectory or return predictions can hide incorrect action dependence. It compares two rollouts from a shared history with different initial actions and measures the discrepancy between model and environment action effects, d^(q)−d(q)\hat d(q)-d(q). It then constructs hybrid rollouts that progressively replace environment transitions with model transitions. The adjacent difference CL=d^L−d^L−1C_L=\hat d_L-\hat d_{L-1} telescopes to the endpoint effect error and is used to attribute this error to transition substitutions.

Experiments on ten Craftax Transformer world models show that normal-execution and return accuracy can coexist with incorrect cancellation or delay behavior. Supervising an immediate no-op alternative (“action coverage”) improves timing for the trained event. A downstream substitution experiment shows that replacing the first learned transition can reduce one later crafting-effect error.

Critical strengths

None

Critical weaknesses
  1. Limited novelty. Comparing outcomes under alternative actions is a standard evaluation idea. “Differential transport” is a telescoping transition-replacement construction closely related to simulation-lemma arguments, as the paper itself notes. The primary novelty is therefore packaging these known ingredients into a world-model diagnostic protocol, rather than a new mathematical principle.

  2. Necessity over simpler diagnostics is not demonstrated. Paired rollouts already detect whether an action effect is wrong at the endpoint. The paper does not compare differential transport against simpler alternatives, such as computing the paired action-effect error after each step or separately at several rollout lengths. Without such a comparison, it remains unclear what additional useful information the transition contributions provide.

  3. Action-effect errors depend strongly on the underlying dynamics. In continuous stochastic systems, model discrepancies may accumulate gradually. In contractive dynamics, an initial action difference may vanish regardless of model quality; in unstable or chaotic dynamics, a small difference may grow rapidly. Craftax instead has discrete events with explicit prerequisites, making differences more likely to appear at a small number of transitions. It is unclear how informative the reported endpoint errors and transition contributions would be in other systems.

  4. The method requires full simulator access with latent-state restoration. Constructing paired environment rollouts requires saving and restoring the complete simulator state, including latent POMDP state invisible to the agent. The hybrid-rollout construction further requires switching between environment and model generation at controlled time steps. This limits direct applicability to snapshotable simulators and excludes offline datasets, real-world systems, and POMDPs whose latent state cannot be restored. The paper does not explain how the procedure would extend to these settings.

Questions for authors
  1. What additional information does differential transport provide beyond computing the paired action-effect error after each step or separately at several rollout lengths? A direct empirical comparison would clarify whether the transition contributions are necessary.

  2. How should the endpoint error and transition contributions be interpreted when action differences become smaller or larger over time because of the underlying dynamics, or when model discrepancies accumulate gradually? Do the conclusions extend beyond discrete events with explicit prerequisites?

  3. Does the evaluation require saving and restoring the complete simulator state described in Appendix B? If so, how could it be applied to offline data or environments where that state cannot be restored?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 41xA: Per-step curve comparison, Cheetah tests, and offline extensions

We thank the reviewer for the detailed questions. We address the value of transition substitutions, generality, and offline applicability below.

1. Information beyond per-step curves (Question 1) Per-step curves locate when a target error becomes visible, whereas substitutions test how an earlier transition changes its final consequence. We compare the submitted substitution in Appendix I with a new tool-effect curve along the same unmodified rollout:

ComparisonResult
Tool-effect error curve (new analysis)Zero through steps 1–15, with mismatch appearing at step 16 when crafting executes.
Replace learned transition 11 with native after ten native transitions, retaining the model suffix to step 20Branch-0 tool acquisition increases 2/32 → 32/32 while branch 1 stays at zero, and the action effect at step 20 changes by −0.291.

This signed action-effect change identifies transition 11's influence under the native prefix, beyond locating when the error appears.

2. Mechanistic contribution (Weakness 1) While the telescoping identity follows simulation-lemma reasoning (§3.2), we identify how intermediate errors sustain misleading endpoint accuracy. Building on premature ore depletion in the submitted delay test (§4.2), our new intervention restores the ore in the generated observation after the first wait while holding inventory, other inputs, and the continuation model fixed across ten systems (128 states, 32 prefixes each), without selection by restoration outcome.

Native mining adds one iron and waiting adds none. Gain is relative to the generated step-1 count, whereas a correct mining endpoint equals the initial count plus one.

  • In System 2, the missing ore suppressed further production after inventory increased prematurely, so restoration raises erroneous inventory gain during waiting from 7.3% to 87.1% and drops final-count accuracy after mining from 87.4% to 5.7%.
  • In System 7, restoration mainly recovers valid mining responses (24.9% → 93.3%) and endpoint accuracy (19.6% → 82.5%).

Waiting is decisive because extra production after mining could be valid extraction added to premature inventory, whereas production during waiting reveals the action error concealed by the missing block.

3. Continuous dynamics and gradual errors (Question 2) Regarding Question 2, we compare model and environment action effects at the same horizon and readout, so the reference reflects natural contraction or amplification. Contributions combine local discrepancies with subsequent propagation, depend on substitution order, and can be distributed over many transitions.

In new Cheetah experiments, we trained three probabilistic dynamics ensembles on independent datasets with distinct initializations and evaluated 128 shared held-out states. We then replaced learned continuation with native dynamics using the same generated states, future actions (four along one actuator direction, yielding six pairs), and targets:

Mean-prediction squared risk, (m/s)²LearnedNativeChange [source-bootstrap 95% CI]
Endpoint velocity0.09700.2958+0.1988 [0.1258, 0.2783]
Velocity differences between actions0.02780.0216−0.00619 [−0.01330, −0.00039]

State resampling gives intervals for the three-ensemble mean, conditional on fitted ensembles and saved draws. Although the direction of the action-difference error change is uncertain for two ensembles, learned continuation's aggregate endpoint advantage reflects smaller shared errors despite less accurate action differences. While contractive and chaotic regimes need further tests, these results extend the distinction to continuous states and another model family.

4. Simulator access and offline extensions (Question 3) Yes, exact paired references require restoring the full simulator state, including hidden state, while the model receives observable history. Craftax hybrids use a native prefix then a model suffix, whereas Cheetah starts native continuation from generated physical states.

Without simulator resets:

  1. Model-to-model substitution (submitted §4.5): Start from a logged observable history, condition each generator on the emitted history, replace one transition, and keep the later generator unchanged. This explains model differences, while judging accuracy requires environment evidence or additional assumptions.
  2. Proposed offline extension: History matching could estimate environment action effects when observed histories control action-selection confounding and the relevant actions and continuations have data coverage, yielding an average over compatible hidden states rather than the exact same-state reference used here.

We will add the direct comparison and Cheetah results, and discuss offline assumptions in §§3–5 and Appendices B and I.

Reviewer_6SAMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Do world models (WM) predict the effect of actions correctly also when predictions look ok? The work compares paired rollouts where an action is delayed or cancelled, across ten systems in Craftax, and the results show that the WM sometimes gives wrong predictions even when returns and observations are accurate (~13% of assessments passing return checks still have large tool-effect errors).

To keep track of errors during a rollout, the paper also introduces “differential transport”, which decomposes the action-effect error into local discrepancies propagated by later model steps.

Critical strengths

Interesting question, and I think important for WM used in planning (long horizon but also short-term planning).

Critical weaknesses

My main doubt is about the limitations of the work. I think the work is really interesting, but everything is only tested on Craftax, with a single architecture (TWM). How general is this approach (MiniHack/NetHack, etc.)?

Also some concerns on the novelty of the theoretical work. Differential transport seems related to the telescoping decomposition used in simulation lemmas (Lobel & Parr, 2024). What is new beyond applying this decomposition to a pair of rollouts? Also, in the experiments it is used mostly on a few illustrative, retrospectively selected cases. To add on this, the training is also limited: improvements focus on a specific mechanic of Craftax (the timing of iron acquisition) and do not transfer well to other events (crafting, placement).

How does the connection with planning hold? The policies don’t seem to improve: all actors trained in imagination end up below their pretrained baselines, including those using the original model, and you also acknowledge the problems with this training procedure.

Questions for authors
  1. How general is the method (other environments, other architectures), and how much of the theoretical work is new?
  2. Can you check whether action-effect errors predict planning performance better than classic prediction metrics?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 6SAM: Cheetah generality tests, planning evaluations, and state interventions

We thank the reviewer for recognizing the importance and soundness of the study. We address your concerns about generality, mechanisms, and planning and control below.

1. Generality (Question 1) We trained three probabilistic dynamics ensembles on independent Cheetah datasets with distinct initializations and evaluated 128 shared held-out states. We then replaced learned continuation with native dynamics using the same generated states, future actions (four along one actuator direction, yielding six pairs), and targets:

Mean-prediction squared risk, (m/s)²LearnedNativeChange [source-bootstrap 95% CI]
Endpoint velocity0.09700.2958+0.1988 [0.1258, 0.2783]
Velocity differences between actions0.02780.0216−0.00619 [−0.01330, −0.00039]

State resampling gives intervals for the three-ensemble mean, conditional on fitted ensembles and saved draws. Although the direction of the action-difference error change is uncertain for two ensembles, learned continuation's aggregate endpoint advantage reflects smaller shared errors despite less accurate action differences.

2. Attribution and case selection While the telescoping identity follows simulation-lemma reasoning (§3.2), we identify how intermediate errors sustain misleading endpoint accuracy. Building on premature ore depletion in the submitted delay test (§4.2), our new intervention restores the ore after the first wait in the generated observation, holding inventory, other inputs, and the continuation model fixed across ten systems (128 states, 32 prefixes each).

Native mining adds one iron and waiting adds none. Gain is relative to the generated step-1 count, whereas a correct mining endpoint equals the initial count plus one.

In System 2, the missing ore suppressed further production after inventory increased prematurely, so restoration raises inventory gain during waiting from 7.3% to 87.1% and drops final-count accuracy after mining from 87.4% to 5.7%. In System 7, restoration mainly recovers valid mining responses (24.9% → 93.3%) and endpoint accuracy (19.6% → 82.5%).

We include all restoration outcomes from the discovery cohort, while the five submitted profiles are illustrative. Cheetah confirms the endpoint–contrast distinction on fresh data.

3. Planning prediction and policy continuation (Question 2) Action-effect return error has the lowest held-out MSE and matches classic metrics in model selection, while combining return errors adds no gain:

  • Model selection: For six locomotion goals (e.g., braking and standing), action-effect, branchwise return, and observation criteria select the same model per goal from three Dreamer models per environment.
  • Grouped prediction: We predict 200-step control gains from changes in 15-step return diagnostics across 48 existing trained variants, holding out each pretrained model's variants together (six groups):
Predictor (same task-specific intercepts)Held-out MSE ↓
Task-mean baseline8.995
Branchwise return error13.098
Action-effect return error8.228
Both errors14.733

One held-out group drives its advantage over branchwise error.

In new continuation control experiments, we resume actor training from saved states under the same protocol in the native simulator, yielding returns of 14.75/15.77 in Systems 0/2, versus pretrained 13.77/15.39 and unchanged-model 1.65/7.64 on the shared 960-episode cohort. Swapping only reward predictions between two updated models, with successor and termination predictors fixed, raises return by 6.93–8.32 in four comparisons. While the update's policy gain is unestablished, these control experiments support our hypothesis that offline optimization favors actions with overstated returns without fresh corrective experience.

4. Submitted training results Our matched-budget training on one event yields two benefits:

  • Timing (§4.4): Supervising immediate action alternatives improves correctly timed acquisition under an unseen one-step delay by 41.4–56.6 percentage points over replay, with readout ambiguity determining the bounds on the ten-system mean (95% source-bootstrap envelope [35.5, 59.5]). Because success requires producing the delayed event, this shows timing adaptation beyond suppression. Normal-acquisition preservation within five points against replay and original references is not established, and gains do not reliably transfer to crafting or placement (Tables 4–5).
  • Downstream effect (§4.5): Changing the first transition while retaining the continuation model reduces false crafting after cancellation from 2.36–3.24% to 0.012–0.062% across three training repetitions of System 2.

5. Presentation revisions We will organize Figure 1 and §1 by experimental question, foreground new evidence in §§4–5, trim notation, and lead with results.

Thank you for the detailed response and the new experiments (good job!).

Unfortunately my main concern on generality remains. The Cheetah experiment is encouraging, but the effect is small and uncertain across ensembles, and the main results are still limited to Craftax with a single architecture. On novelty, the response confirms that differential transport is mainly the simulation-lemma decomposition applied to paired rollouts. On planning, the new analyses suggest that action-effect errors do not yet predict planning performance better than standard metrics, and the benefit of the training update for decision making is still not established.

I think this is an interesting direction and the additions could potentially strengthen the paper, but I will keep my score.

Reviewer_59HPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines whether world models correctly predict the consequences of changing actions. The authors use cancellation and delay tests in Craftax, then substitute individual transitions to trace how local predictions affect later events. The experiments reveal action-dependence errors that return and aggregate observation metrics can miss. Supervising alternative actions improves short-term event timing, while the demonstrated benefits for longer-term prediction and policy learning remain limited.

Critical strengths

The cancellation and delay experiments illustrate the failures clearly and use executed environment outcomes as a reference. The training comparison matches supervision and optimization budgets. The transition substitutions also provide a useful finding: changing the initial prediction can suppress later false events while keeping the subsequent generator fixed.

Critical weaknesses
  1. The contribution beyond prior work appears incremental. Earlier work has already shown that individually plausible predictions can respond incorrectly to interventions and that aggregate prediction quality does not establish decision-making usefulness. This paper adds more controlled simulator-based comparisons and applies a telescoping decomposition to action-effect errors. These are useful additions, but the resulting insight remains narrow. The substitutions identify where a local change affects later predictions, without establishing which training or architectural factors produced the underlying failure.

  2. The demonstrated improvement has limited scope. In the downstream crafting experiments, valid events remain rare in model rollouts even when they occur frequently in the environment. The benefit mainly comes from suppressing false events after cancellation, rather than recovering valid predictions. The authors acknowledge this distinction, but it limits what the experiment establishes about improved longer-term modeling. The short-term gains also do not reliably transfer to other tested events, and preservation of normal acquisition performance remains uncertain. Policy continuation degrades performance even with the unchanged world model, leaving the practical benefit of the update unresolved.

Questions for authors

Why does policy continuation degrade performance even with the original world model? Is there a stable continuation or planning setup that could clarify whether the prediction improvements help decision making?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 59HP: State-variable interventions and native continuation controls

We thank the reviewer for the constructive critique. We address your concerns about mechanism, policy continuation, training gains, and clarity below.

1. What the intermediate error explains (Weakness 1) To address your concern about mechanistic depth, we tested whether an intermediate state error could make a later prediction look correct. Building on the premature ore depletion during the initial no-op ("wait") step in the submitted delay test (§4.2), we now restore the ore in the generated observation after that step while holding predicted inventory, other inputs, and the continuation model fixed. We then compare "mine" with another "wait" across all ten original systems, using 128 initial states and 32 generated prefixes per state.

In the native environment, mining adds one iron and waiting adds none. The table reports how often inventory increases from the generated step-1 count and how often the final count after mining is correct. A correct endpoint equals the initial inventory plus one, regardless of when the iron was produced.

Before → after ore restorationSystem 2System 7
Inventory gain after mining9.9% → 96.1%24.9% → 93.3%
Inventory gain during waiting7.3% → 87.1%8.8% → 11.4%
Correct final count after mining87.4% → 5.7%19.6% → 82.5%

In System 2, inventory had already increased prematurely, and the missing ore suppressed further production, leaving an apparently correct final count. Restoration therefore exposes erroneous production during waiting and excess inventory after mining. In System 7, by contrast, the missing ore mainly blocked valid mining afterward, so restoration recovers both inventory gain and endpoint accuracy. While training or architectural origins remain open, the intervention identifies how a specific intermediate error conceals invalid responses or blocks valid ones.

2. Policy degradation and a stable reference (Question & Weakness 2) Regarding your questions on why continuation degrades and whether a stable setup exists, new control experiments show that native continuation sustains performance from the same saved training states under the same protocol and remaining update budget. The table compares mean returns of new native runs and existing policies on the same 960-episode cohort, using three actors per continuation condition:

Continuation conditionSystem 0System 2
Pretrained policy13.7715.39
Unchanged world model (historical reference)1.657.64
Native-environment continuation14.7515.77

In additional control experiments, swapping only reward predictions between two updated models, with successor and termination predictors fixed, increases return by 6.93–8.32 across four comparisons.

These control experiments support the hypothesis that continued policy optimization favors actions with overstated predicted returns, while the static training bank supplies no new corrective experience. This account therefore predicts increasing return overestimation on histories and actions selected during continuation—a target for subsequent validation.

3. Timing recovery and downstream suppression (Weakness 2) Your distinction between suppressing false events and recovering valid predictions helps clarify what the submitted training experiments establish:

  • Timing (§4.4): Under matched budgets, supervising execution and no-op improves correctly timed acquisition under an unseen one-step delay by 41.4–56.6 percentage points over replay, where the bounds on the ten-system mean reflect ambiguous event readings (95% source-bootstrap envelope [35.5, 59.5]). Because success requires producing the event when the delayed action arrives, the gain reflects timing adaptation beyond suppression. Normal-acquisition preservation within five points is not established against both replay and original references (Table 4), and gains do not reliably transfer to crafting or placement (Table 5).
  • Downstream effect (§4.5): Replacing the first learned transition while retaining the continuation model reduces false crafting after cancellation from 2.36–3.24% to 0.012–0.062% across three training repetitions of System 2, showing that the benefit comes from the updated initial prediction. Since valid normal crafting remains about 2–3% versus 82.5% native, we hypothesize that later prerequisite errors limit recovery, motivating tests of whether supervising those relations restores valid multi-step events.

4. Presentation revisions To address your readability concern, we will:

  1. Reorganize Figure 1 and §1 so readers grasp the question each experiment answers before formal notation.
  2. Foreground the state-variable intervention and native continuation control in §4.5, trim non-essential notation, and open each experimental subsection with a direct result statement.

Thank you for the additional experiments. The intermediate-state intervention and continuation controls address some of my concerns.

However, the gains remain concentrated on short-term timing and suppressing false events. Valid multi-step predictions are still rare, transfer to other events is limited, and the results do not yet show consistent benefits for decision-making. The additions strengthen the analysis, but my main concerns remain, so I will keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.