- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether the local action-response geometry induced by frozen action-conditioned world models corresponds to the actual consequences of nearby action perturbations. The authors define a local counterfactual geometry around an action plan through the change in predicted future latent states under action perturbations, and approximate this geometry using the action Jacobian of the frozen predictor. The same Jacobian is further used to map goal residuals back into local action corrections. The paper evaluates this geometry in two main ways. First, it compares predicted action effects with realized effects obtained by replaying the original and perturbed plans from the same simulator state. Second, it studies whether action effects or correction directions computed from one interaction remain useful when transferred to another interaction. A trust-region-style radius is used to control how far optimization may move from an initial proposal. Experiments are conducted on several released world-model systems and tasks, including LeWM and INTACT, with additional comparisons involving DINO-WM and JEPA-WM. The main empirical findings are that large model responses do not necessarily imply environmentally correct action effects, source-matched guidance is generally more reliable than guidance transferred from other interactions, and bounded local correction can improve some proposal policies with relatively few model evaluations.
Critical strengths
- The paper provides a reasonably controlled empirical evaluation through paired replay and matched experimental conditions, which helps isolate discrepancies between predicted and realized action effects. 2.The distinction between action-response magnitude and environmental effect accuracy offers a potentially useful diagnostic perspective beyond conventional planning-success evaluation.
Critical weaknesses
- The scientific argument is surprisingly difficult to follow because the manuscript often places adjacent statements next to each other without making the logical connection explicit.
This problem is already apparent in the abstract: the paper asks whether action-conditioned predictors “get these changes right,” then immediately introduces local counterfactual geometry, and then moves to predicted-versus-realized effect comparisons, but the reader is largely left to infer why each step follows from the previous one. The same issue appears in the introduction, where expressions such as “this requirement” are used without first stating the requirement clearly enough. I repeatedly found myself having to reconstruct the missing reasoning between sentences, which makes the paper much harder to understand than the underlying technical content should require.
- The counterfactual framing substantially weakens the paper’s methodological novelty because the main grounding procedure directly observes both action outcomes through simulator replay.
The experiments restore the same simulator state and deterministic seed and then execute both and . This is essentially repeated controlled trial-and-error from the same starting point rather than a new counterfactual inference mechanism. While such a protocol can be useful for evaluation, it largely reduces the claimed counterfactual contribution to an empirical measurement procedure. As a result, the paper reads more like a carefully designed experimental report on action-effect fidelity than a methodologically novel work on counterfactual reasoning. This limitation is particularly important in stochastic or non-resettable environments, where repeated trials cannot recover the same counterfactual comparison by simply resetting and trying again.
- The interpretation of cross-interaction source mismatch is ambiguous because action effects are inherently context dependent, and may also vary under stochastic environment dynamics.
The paper interprets the degradation of donor Jacobians or gradients from to as evidence that predicted effects do not transfer reliably. However, even a correct world model should generally produce different local action effects across different states, histories, or latent environment conditions. In realistic stochastic settings, repeated executions may further differ because the underlying exogenous factors cannot be exactly reproduced. Without separating model-induced mismatch from genuine environment variation, poor donor transfer alone is difficult to interpret as a deficiency of the world model.
Questions for authors
-
What is the methodological novelty of the proposed counterfactual framework beyond resetting the simulator and repeatedly executing alternative actions from the same starting state?
-
If the true environment dynamics are themselves state-, history-, or noise-dependent, how do you distinguish model-induced source mismatch from legitimate variation in the environment’s local action effects?
-
How would the proposed counterfactual geometry be validated or calibrated in a stochastic or non-resettable environment, where the same state and exogenous conditions cannot be exactly reproduced for repeated trials?
5 comments
W2 and Q1: Methodological novelty and contribution
Thank you for recognizing that distinguishing action-response magnitude from environmental effect accuracy offers a useful diagnostic perspective beyond conventional planning-success evaluation. We respectfully disagree with characterizing the paper as an experimental report, and would like to clarify the contribution that connects its different parts.
World models do more than predict future outcomes: they implicitly encode how changing actions alters those predictions. This action-response structure is central to planning, because a planner uses predicted differences between alternatives to decide how to modify its actions. Yet prediction accuracy and planning success do not directly tell us whether these predicted action effects agree with environmental responses. We make this model-internal structure explicit as local counterfactual geometry. Here, counterfactual means comparing action alternatives at a fixed history, as clarified in the Introduction footnote.
The contribution is to establish this geometry as a common lens for understanding, evaluating, and exploiting frozen world models. It allows us to examine response strength and interaction dependence, evaluate predicted action effects against execution, and use the same mapping to make bounded corrections to existing plans. We further apply shared action queries across released world-model systems to characterize their action-response behavior.
Crucially, simulator replay is an external reference for evaluating the predicted effects, not the geometry itself. By crossing prediction and execution contexts while keeping the action alternatives fixed, we distinguish fidelity at the source from the validity of reusing a prediction elsewhere. Our additional error decomposition further separates prediction error from genuine variation in environmental responses. For control, the same geometry provides goal-directed corrections without requiring paired replay. The new schedule-matched controls strengthen the interpretation of guidance-source effects, while the independent randomized single-branch evaluation demonstrates improved execution without replaying unchosen alternatives.
Jacobian differentiation and projected optimization are established tools. The methodological contribution lies in the comparisons described above: separating source fidelity from cross-context reuse, and testing the decision value of the same mapping through controlled action refinement.
Action: We revised the Abstract and Introduction to clarify this perspective, distinguished the geometry from its evaluation and use in Section 3, and organized Sections 4–5 around the corresponding experimental questions.
W3 and Q2: Model error versus genuine environmental variation
Thank you. You correctly point out that an accurate model can predict different action effects in different contexts. Our claim is therefore not that a Donor effect is erroneous merely because it changes at another interaction. Source fidelity and cross-context reuse are different measurements, and our paired protocol allows us to distinguish them.
For the same action pair, let be the effect predicted at the source and the effects executed at the source and recipient. The cross-context discrepancy is
The first term is source-system prediction error, including its encoder/readout path; the second is genuine environmental variation. Squared error also includes their signed interaction. We measured all three in Appendix D.2.1 rather than attributing the full reuse error to the model.
Under the fixed LeWM physical measurement path, the normalized root-to-Far source error is 2.393, environmental variation contributes 3.632, and the cross term is -2.617, yielding a reuse error of 3.408. Recomputing the prediction from the recipient's own context reduces the corresponding error to 1.262. This gives a more useful conclusion than simply calling Donor guidance inaccurate: a locally computed prediction can be substantially more relevant to the recipient than an effect transported from another interaction, even when part of the difference is genuine environmental variation. The original Jacobian/residual exchange provides complementary evidence that the guidance operator, not only the goal residual, changes correction direction (original Table 11; revised Table 12).
Action: We added the source-error/environment-variation decomposition to Appendix D.2.1 (Table 11), incorporated the result in Section 5.2, and clarified the interpretation of interaction dependence in Sections 5.3 and 6. The Jacobian/residual exchange remains in Appendix D.2.2 (Table 12).
Thank you for your detailed comments. We have clarified the paper's logical progression and contribution beyond simulator replay, and added an analysis of how genuine environmental variation affects cross-context comparisons.
Responses to Weaknesses and Questions:
W1: Scientific argument and logical connections
Thank you. You identified an important presentation problem: the original Abstract moved from questioning predicted action changes to introducing geometry and replay without explaining why they were connected. We have revised that progression.
An action-conditioned world model predicts outcomes under alternative actions. Planning depends on the difference between those predicted outcomes, and that difference need not agree with the actual effect of an action edit. Local counterfactual geometry names this action-response mapping inside a frozen model. Same-state execution then tests its fidelity. The same mapping can also guide bounded corrections to a plan proposed by an existing controller.
The Introduction develops these steps in this order. Sections 4 and 5 now follow the corresponding questions: source fidelity, interaction dependence, guidance and radius, bounded correction, and cross-model diagnosis. These revisions directly address the transitions and undefined references.
Action: We revised the motivation and transitions in the Abstract and Introduction, clarified the geometry/replay distinction in Section 3, and aligned the experimental questions and results across Sections 4–5.