- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that accurate trajectory or return predictions can hide incorrect action dependence. It compares two rollouts from a shared history with different initial actions and measures the discrepancy between model and environment action effects, . It then constructs hybrid rollouts that progressively replace environment transitions with model transitions. The adjacent difference telescopes to the endpoint effect error and is used to attribute this error to transition substitutions.
Experiments on ten Craftax Transformer world models show that normal-execution and return accuracy can coexist with incorrect cancellation or delay behavior. Supervising an immediate no-op alternative (“action coverage”) improves timing for the trained event. A downstream substitution experiment shows that replacing the first learned transition can reduce one later crafting-effect error.
Critical strengths
None
Critical weaknesses
-
Limited novelty. Comparing outcomes under alternative actions is a standard evaluation idea. “Differential transport” is a telescoping transition-replacement construction closely related to simulation-lemma arguments, as the paper itself notes. The primary novelty is therefore packaging these known ingredients into a world-model diagnostic protocol, rather than a new mathematical principle.
-
Necessity over simpler diagnostics is not demonstrated. Paired rollouts already detect whether an action effect is wrong at the endpoint. The paper does not compare differential transport against simpler alternatives, such as computing the paired action-effect error after each step or separately at several rollout lengths. Without such a comparison, it remains unclear what additional useful information the transition contributions provide.
-
Action-effect errors depend strongly on the underlying dynamics. In continuous stochastic systems, model discrepancies may accumulate gradually. In contractive dynamics, an initial action difference may vanish regardless of model quality; in unstable or chaotic dynamics, a small difference may grow rapidly. Craftax instead has discrete events with explicit prerequisites, making differences more likely to appear at a small number of transitions. It is unclear how informative the reported endpoint errors and transition contributions would be in other systems.
-
The method requires full simulator access with latent-state restoration. Constructing paired environment rollouts requires saving and restoring the complete simulator state, including latent POMDP state invisible to the agent. The hybrid-rollout construction further requires switching between environment and model generation at controlled time steps. This limits direct applicability to snapshotable simulators and excludes offline datasets, real-world systems, and POMDPs whose latent state cannot be restored. The paper does not explain how the procedure would extend to these settings.
Questions for authors
-
What additional information does differential transport provide beyond computing the paired action-effect error after each step or separately at several rollout lengths? A direct empirical comparison would clarify whether the transition contributions are necessary.
-
How should the endpoint error and transition contributions be interpreted when action differences become smaller or larger over time because of the underlying dynamics, or when model discrepancies accumulate gradually? Do the conclusions extend beyond discrete events with explicit prerequisites?
-
Does the evaluation require saving and restoring the complete simulator state described in Appendix B? If so, how could it be applied to offline data or environments where that state cannot be restored?
Response to Reviewer 41xA: Per-step curve comparison, Cheetah tests, and offline extensions
We thank the reviewer for the detailed questions. We address the value of transition substitutions, generality, and offline applicability below.
1. Information beyond per-step curves (Question 1) Per-step curves locate when a target error becomes visible, whereas substitutions test how an earlier transition changes its final consequence. We compare the submitted substitution in Appendix I with a new tool-effect curve along the same unmodified rollout:
This signed action-effect change identifies transition 11's influence under the native prefix, beyond locating when the error appears.
2. Mechanistic contribution (Weakness 1) While the telescoping identity follows simulation-lemma reasoning (§3.2), we identify how intermediate errors sustain misleading endpoint accuracy. Building on premature ore depletion in the submitted delay test (§4.2), our new intervention restores the ore in the generated observation after the first wait while holding inventory, other inputs, and the continuation model fixed across ten systems (128 states, 32 prefixes each), without selection by restoration outcome.
Native mining adds one iron and waiting adds none. Gain is relative to the generated step-1 count, whereas a correct mining endpoint equals the initial count plus one.
Waiting is decisive because extra production after mining could be valid extraction added to premature inventory, whereas production during waiting reveals the action error concealed by the missing block.
3. Continuous dynamics and gradual errors (Question 2) Regarding Question 2, we compare model and environment action effects at the same horizon and readout, so the reference reflects natural contraction or amplification. Contributions combine local discrepancies with subsequent propagation, depend on substitution order, and can be distributed over many transitions.
In new Cheetah experiments, we trained three probabilistic dynamics ensembles on independent datasets with distinct initializations and evaluated 128 shared held-out states. We then replaced learned continuation with native dynamics using the same generated states, future actions (four along one actuator direction, yielding six pairs), and targets:
State resampling gives intervals for the three-ensemble mean, conditional on fitted ensembles and saved draws. Although the direction of the action-difference error change is uncertain for two ensembles, learned continuation's aggregate endpoint advantage reflects smaller shared errors despite less accurate action differences. While contractive and chaotic regimes need further tests, these results extend the distinction to continuous states and another model family.
4. Simulator access and offline extensions (Question 3) Yes, exact paired references require restoring the full simulator state, including hidden state, while the model receives observable history. Craftax hybrids use a native prefix then a model suffix, whereas Cheetah starts native continuation from generated physical states.
Without simulator resets:
We will add the direct comparison and Cheetah results, and discuss offline assumptions in §§3–5 and Appendices B and I.