acceptodds
Under review as a conference paper at ICLR 2027

Local Counterfactual Geometry in World Models: Propose Globally, Correct Locally

Abstract

A world model can predict how changing an action will change its outcome. We test whether frozen action-conditioned latent predictors get these changes right. We study local counterfactual geometry, the mapping from perturbations around an action plan to predicted future changes conditioned on observation and action history. Comparing predicted and realized effects of the same perturbations tests whether this geometry reflects the environment and where it remains valid. Our experiments show that strong action responses can misrepresent real consequences, and that predicted effects may not transfer across interactions. In control, this source mismatch becomes more consequential as optimization is allowed to move farther from the proposal. The same geometry also supports action refinement: an existing controller supplies a proposal, while the frozen world model computes bounded, goal-directed corrections using the current interaction. We compare released models on the same base action plans and perturbations to measure effect accuracy across perturbation sizes, horizons, and interactions. This offers a complementary view of world-model capability: beyond proposal performance, we ask whether the model can reliably guide local improvement. Local counterfactual geometry thus connects observation and control: propose globally, correct locally.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, counterfactual reasoning, model-based planning, action-conditioned dynamics, latent representations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dRwYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether the local action-response geometry induced by frozen action-conditioned world models corresponds to the actual consequences of nearby action perturbations. The authors define a local counterfactual geometry around an action plan through the change in predicted future latent states under action perturbations, and approximate this geometry using the action Jacobian of the frozen predictor. The same Jacobian is further used to map goal residuals back into local action corrections. The paper evaluates this geometry in two main ways. First, it compares predicted action effects with realized effects obtained by replaying the original and perturbed plans from the same simulator state. Second, it studies whether action effects or correction directions computed from one interaction remain useful when transferred to another interaction. A trust-region-style radius is used to control how far optimization may move from an initial proposal. Experiments are conducted on several released world-model systems and tasks, including LeWM and INTACT, with additional comparisons involving DINO-WM and JEPA-WM. The main empirical findings are that large model responses do not necessarily imply environmentally correct action effects, source-matched guidance is generally more reliable than guidance transferred from other interactions, and bounded local correction can improve some proposal policies with relatively few model evaluations.

Critical strengths
  1. The paper provides a reasonably controlled empirical evaluation through paired replay and matched experimental conditions, which helps isolate discrepancies between predicted and realized action effects. 2.The distinction between action-response magnitude and environmental effect accuracy offers a potentially useful diagnostic perspective beyond conventional planning-success evaluation.
Critical weaknesses
  1. The scientific argument is surprisingly difficult to follow because the manuscript often places adjacent statements next to each other without making the logical connection explicit.

This problem is already apparent in the abstract: the paper asks whether action-conditioned predictors “get these changes right,” then immediately introduces local counterfactual geometry, and then moves to predicted-versus-realized effect comparisons, but the reader is largely left to infer why each step follows from the previous one. The same issue appears in the introduction, where expressions such as “this requirement” are used without first stating the requirement clearly enough. I repeatedly found myself having to reconstruct the missing reasoning between sentences, which makes the paper much harder to understand than the underlying technical content should require.

  1. The counterfactual framing substantially weakens the paper’s methodological novelty because the main grounding procedure directly observes both action outcomes through simulator replay.

The experiments restore the same simulator state and deterministic seed and then execute both AA and A+δAA+\delta A. This is essentially repeated controlled trial-and-error from the same starting point rather than a new counterfactual inference mechanism. While such a protocol can be useful for evaluation, it largely reduces the claimed counterfactual contribution to an empirical measurement procedure. As a result, the paper reads more like a carefully designed experimental report on action-effect fidelity than a methodologically novel work on counterfactual reasoning. This limitation is particularly important in stochastic or non-resettable environments, where repeated trials cannot recover the same counterfactual comparison by simply resetting and trying again.

  1. The interpretation of cross-interaction source mismatch is ambiguous because action effects are inherently context dependent, and may also vary under stochastic environment dynamics.

The paper interprets the degradation of donor Jacobians or gradients from r′r' to rr as evidence that predicted effects do not transfer reliably. However, even a correct world model should generally produce different local action effects across different states, histories, or latent environment conditions. In realistic stochastic settings, repeated executions may further differ because the underlying exogenous factors cannot be exactly reproduced. Without separating model-induced mismatch from genuine environment variation, poor donor transfer alone is difficult to interpret as a deficiency of the world model.

Questions for authors
  1. What is the methodological novelty of the proposed counterfactual framework beyond resetting the simulator and repeatedly executing alternative actions from the same starting state?

  2. If the true environment dynamics are themselves state-, history-, or noise-dependent, how do you distinguish model-induced source mismatch from legitimate variation in the environment’s local action effects?

  3. How would the proposed counterfactual geometry be validated or calibrated in a stochastic or non-resettable environment, where the same state and exogenous conditions cannot be exactly reproduced for repeated trials?

5 comments

Thank you for your detailed comments. We have clarified the paper's logical progression and contribution beyond simulator replay, and added an analysis of how genuine environmental variation affects cross-context comparisons.

Responses to Weaknesses and Questions:

W1: Scientific argument and logical connections

Thank you. You identified an important presentation problem: the original Abstract moved from questioning predicted action changes to introducing geometry and replay without explaining why they were connected. We have revised that progression.

An action-conditioned world model predicts outcomes under alternative actions. Planning depends on the difference between those predicted outcomes, and that difference need not agree with the actual effect of an action edit. Local counterfactual geometry names this action-response mapping inside a frozen model. Same-state execution then tests its fidelity. The same mapping can also guide bounded corrections to a plan proposed by an existing controller.

The Introduction develops these steps in this order. Sections 4 and 5 now follow the corresponding questions: source fidelity, interaction dependence, guidance and radius, bounded correction, and cross-model diagnosis. These revisions directly address the transitions and undefined references.

Action: We revised the motivation and transitions in the Abstract and Introduction, clarified the geometry/replay distinction in Section 3, and aligned the experimental questions and results across Sections 4–5.

W2 and Q1: Methodological novelty and contribution

Thank you for recognizing that distinguishing action-response magnitude from environmental effect accuracy offers a useful diagnostic perspective beyond conventional planning-success evaluation. We respectfully disagree with characterizing the paper as an experimental report, and would like to clarify the contribution that connects its different parts.

World models do more than predict future outcomes: they implicitly encode how changing actions alters those predictions. This action-response structure is central to planning, because a planner uses predicted differences between alternatives to decide how to modify its actions. Yet prediction accuracy and planning success do not directly tell us whether these predicted action effects agree with environmental responses. We make this model-internal structure explicit as local counterfactual geometry. Here, counterfactual means comparing action alternatives at a fixed history, as clarified in the Introduction footnote.

The contribution is to establish this geometry as a common lens for understanding, evaluating, and exploiting frozen world models. It allows us to examine response strength and interaction dependence, evaluate predicted action effects against execution, and use the same mapping to make bounded corrections to existing plans. We further apply shared action queries across released world-model systems to characterize their action-response behavior.

Crucially, simulator replay is an external reference for evaluating the predicted effects, not the geometry itself. By crossing prediction and execution contexts while keeping the action alternatives fixed, we distinguish fidelity at the source from the validity of reusing a prediction elsewhere. Our additional error decomposition further separates prediction error from genuine variation in environmental responses. For control, the same geometry provides goal-directed corrections without requiring paired replay. The new schedule-matched controls strengthen the interpretation of guidance-source effects, while the independent randomized single-branch evaluation demonstrates improved execution without replaying unchosen alternatives.

Jacobian differentiation and projected optimization are established tools. The methodological contribution lies in the comparisons described above: separating source fidelity from cross-context reuse, and testing the decision value of the same mapping through controlled action refinement.

Action: We revised the Abstract and Introduction to clarify this perspective, distinguished the geometry from its evaluation and use in Section 3, and organized Sections 4–5 around the corresponding experimental questions.

W3 and Q2: Model error versus genuine environmental variation

Thank you. You correctly point out that an accurate model can predict different action effects in different contexts. Our claim is therefore not that a Donor effect is erroneous merely because it changes at another interaction. Source fidelity and cross-context reuse are different measurements, and our paired protocol allows us to distinguish them.

For the same action pair, let Δ^s\widehat{\Delta}_{\mathit{s}} be the effect predicted at the source and Δs,Δr\Delta_{\mathit{s}},\Delta_{\mathit{r}} the effects executed at the source and recipient. The cross-context discrepancy is

Δ^s−Δr=(Δ^s−Δs)+(Δs−Δr)\widehat{\Delta}_{\mathit{s}}-\Delta_{\mathit{r}}=(\widehat{\Delta}_{\mathit{s}}-\Delta_{\mathit{s}})+(\Delta_{\mathit{s}}-\Delta_{\mathit{r}})

The first term is source-system prediction error, including its encoder/readout path; the second is genuine environmental variation. Squared error also includes their signed interaction. We measured all three in Appendix D.2.1 rather than attributing the full reuse error to the model.

Under the fixed LeWM physical measurement path, the normalized root-to-Far source error is 2.393, environmental variation contributes 3.632, and the cross term is -2.617, yielding a reuse error of 3.408. Recomputing the prediction from the recipient's own context reduces the corresponding error to 1.262. This gives a more useful conclusion than simply calling Donor guidance inaccurate: a locally computed prediction can be substantially more relevant to the recipient than an effect transported from another interaction, even when part of the difference is genuine environmental variation. The original Jacobian/residual exchange provides complementary evidence that the guidance operator, not only the goal residual, changes correction direction (original Table 11; revised Table 12).

Action: We added the source-error/environment-variation decomposition to Appendix D.2.1 (Table 11), incorporated the result in Section 5.2, and clarified the interpretation of interaction dependence in Sections 5.3 and 6. The Jacobian/residual exchange remains in Appendix D.2.2 (Table 12).

Reviewer_y9ghOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper studies whether frozen action-conditioned world models can predict how the outcome changes when an action plan is modified.

There are three main experiments:

  1. Starting from the same simulator state, the authors execute an original plan and a perturbed one, then compare the actual change in outcome with what the world model predicts. Using the correct context works better than using one from another episode, although the predictions are still not very accurate.

  2. An existing controller proposes a plan, and the world model tries to improve it using its own gradients. The changes are constrained to a radius �0��0� around the original plan. For small values of ρ\rho, the choice of gradient makes little difference to success. With larger value, only gradients computed from the correct context seem to work.

  3. The authors compare three pretrained world models (LeWM, DINO-WM, and JEPA-WM) on Push-T, where a robot arm must push a T-shaped block to a target. LeWM performs best, but DINO-WM and JEPA-WM do worse than simply predicting no change. Even LeWM does not clearly beat that baseline.

Critical strengths

I like the same-state replay experiment. If we start both plans from exactly the same simulator state is a good way to test whether the model predicts the changes that matter for planning. I also think the experiments are ok (and also appendix).

Critical weaknesses

My main concern is with the comparison between world models. I am not convinced that the results tell us much about the predictors themselves, rather than the encoders and readouts. In Table 19, if I understand correctly, even a perfect predictor that reproduced the encoder's latent representation of the real outcome would score worse than the actual models when evaluated through the same readout. This seems like a serious issue for interpreting the model ranking. In native coordinates, all models have positive scores. Also, the improvement with perturbation size seems consistent with what one would expect from readout noise, since larger effects should be easier to distinguish from that noise.

I also have doubts about what the experiment varying ρ\rho tells us. The donor gradient is taken from an unrelated episode, and Table 12 suggests that, at least at the proposal point, it is not really distinguishable from a random direction. The sample for this comparison is small, so I would not put too much weight on it. But there is another issue: in Table 3, the donor direction stays fixed, while the random direction is sampled again at every step. These are quite different update procedures. The random directions tend to cancel each other, so the plan may simply move less. In Table 14, where the update schedule is matched, donor guidance behaves much more like random guidance. I wonder if the increasing gap with ρ\rho is mostly due to repeatedly following the same wrong direction, rather than something interesting about using gradients from another episode.

I also found the paper unnecessarily dense!

Questions for authors
  1. In Table 19, the oracle case, where the readout is applied to the encoded real outcomes, seems to score worse than the models themselves, and in the same order. How can we tell whether the ranking comes from the predictor rather than the encoder and readout? If these effects cannot be separated, I do not see how contribution 3 is supported.

  2. In Section 4.5, why does "Near" outperform "Matched" for LeWM? Also, why is "Far" so much worse for DINO-WM and JEPA-WM? Since "Far" corresponds to the most different states, could this simply be an out-of-distribution effect? Are those states representative of what the models encountered during training?

  3. How does the performance gap between "Matched" and a fixed-random-direction baseline change as ρ\rho increases? If it grows similarly to the "Matched" versus "Donor" gap, what conclusions can we gather from this Donor experiment??

4 comments

Thank you for your careful review and for recognizing the value of our paired replay experiments. We have added analyses to clarify what the released-model scores measure and to test the guidance-source comparisons under better-matched controls.

Responses to Weaknesses and Questions:

W1 and Q1: Predictor versus encoder/readout

Q1: What is Contribution 3 comparing?

We compare the action-effect behavior of released frozen world-model systems using shared action queries. Each physical-coordinate result follows the system's encoder, predictor, and fixed linear readout, which expresses predicted changes in common task coordinates. The readout is used for evaluation, not planning. The DINO-WM–JEPA-WM separation observed in physical coordinates is not reproduced in native coordinates (Table 24 and accompanying analysis), showing how the measurement path affects the observed differences between released systems. The contribution statement and Section 5.5 now specify this comparison and its measurement path.

W1: Why can the prediction score better than the encoded real outcome?

The physical-effect error decomposition in revised Table 24 (original Table 19) shows how the error components combine in the apparent "oracle" reversal:

Δ^phys−Δenv=(Δ^phys−Δreadout)⏟projected prediction discrepancy+(Δreadout−Δenv)⏟encoder/readout discrepancy\widehat{\Delta}_{\mathrm{phys}}-\Delta_{\mathrm{env}}=\underbrace{(\widehat{\Delta}_{\mathrm{phys}}-\Delta_{\mathrm{readout}})}_{\text{projected prediction discrepancy}}+\underbrace{(\Delta_{\mathrm{readout}}-\Delta_{\mathrm{env}})}_{\text{encoder/readout discrepancy}}

Here, Δreadout\Delta_{\mathrm{readout}} reconstructs the real executed effect through the model's encoder and readout. The measured negative cross term shows cancellation between these discrepancies. The encoded-and-reconstructed real outcome is therefore not an error-free physical reference.

W1: Does the perturbation-scale pattern depend on the physical readout?

We also measured each system in its native coordinates on all 240 roots. Effect skill increases from the smallest to the largest radius by 0.441 for LeWM, 0.314 for DINO-WM, and 0.351 for JEPA-WM (Table 25), without the physical readout. Thus, the scale trend is not specific to the physical readout.

Action: Contribution 3 and Section 5.5 clarify the released-system comparison. Appendix D.5 (Tables 24–25) reports the decomposition and native-coordinate analysis.

W2 and Q3: Donor, Random, and the update schedule

Q3: Does Matched also pull away from Fixed Random as the radius grows?

Yes. The Matched–Fixed Random success gap rises from 0.00 to 76.00 percentage points on Push-T and from 9.11 to 54.00 on Reacher between ρ=0.25\rho=0.25 and the unbounded endpoint. At ρ=0.5\rho=0.5 and 1.01.0, these gaps are 1.78 and 4.89 on Push-T and 12.67 and 14.00 on Reacher. The corresponding Matched–Fixed Donor gaps are 11.78 and 44.22 on Push-T and 34.00 and 48.44 on Reacher. Since gap amplification is not specific to Donor guidance, we use the direct, schedule-matched Donor–Random comparisons below to test how transferred guidance compares with a random direction at these finite radii.

W2: What remains when Donor and Random have the same update schedule?

We compared them directly under both Fixed and Fresh schedules. Fixed Donor retains the source interaction and recomputes its gradient along the updated plan; Fixed Random retains a sampled orientation while rematching the norm. Fresh variants refresh the source or sampled direction. The recipient episodes, proposals, optimizer, bounds, and 30-update budget are shared. The table reports paired Donor-minus-Random success differences (percentage points) over the same 150 episodes and three actors.

TaskRadiusFixed D − Fixed RFresh D − Fresh R
Push-T0.25-1.56-0.67
Push-T0.50-10.00*-9.11*
Push-T1.00-39.33*-33.33*
Push-TUnbounded-7.78-58.00*
Reacher0.25-10.89+0.67
Reacher0.50-21.33*-15.11*
Reacher1.00-34.44*-29.78*
ReacherUnbounded+4.89-23.33*

An asterisk denotes a paired crossed-bootstrap interval excluding zero after Bonferroni correction across all 16 direct contrasts. At ρ=0.5\rho=0.5 and 1.01.0, Donor is below Random under both Fixed and Fresh schedules on both tasks. These eight finite-radius contrasts remain resolved after correction. The direct contrasts are a post-hoc extension of the existing Fixed/Fresh experiments.

Q3: Could different achieved action displacement explain the result?

We separately equated the feasible final action displacement in a one-step Matched/Donor/Random comparison, extending it by 450 new episodes per task, analyzed separately from the original sample. This removes differences accumulated over optimizer updates. On Reacher at ρ=0.5\rho=0.5, Matched exceeds Donor by 12.22 percentage points in success after multiplicity adjustment, with a corresponding difference in executed native goal loss. On Reacher, the Matched–Random and Donor–Random success differences remain unresolved after adjustment, while executed native goal loss is lower for Matched than Random and higher for Donor than Random. On Push-T at the same radius, Random achieves higher success than both model-guided directions. The equal-displacement test therefore supports a task-specific result, while the schedule-matched table directly answers the multi-update Donor–Random comparison.

Action: Section 5.3 summarizes the controls. Appendix D.3.2 now includes the direct 16-contrast test in Table 16 and the expanded one-step sample in Table 17, retaining the original results separately.

Q2: Near versus Matched; Far and distribution shift

Thank you. We examined the Near comparison and Far-state coverage, and checked whether the source-swap result persists with observations taken after simulator initialization.

Q2: Why does Near sometimes score above Matched?

In the original local-skill comparison, predictions are evaluated against executed effects from their respective interactions, so the realized effect magnitudes and the score normalization differ. For LeWM, Near minus Matched changes from +0.292 with equal-cell aggregation to -0.070 with energy pooling; neither difference is statistically resolved (Appendix D.5, analysis following Table 25). The apparent Near advantage is therefore sensitive to aggregation and does not establish a reliable advantage in effect accuracy.

Q2: Could Far be an unsupported or out-of-distribution state?

We compared our query states and actions with the public DINO-WM Push-T data archive, containing 2,336,736 valid frames. All 240 query states, including the 21 distinct Far states, appear in that public source after conversion to the stored precision; the recorded actions also match under the documented scaling. JEPA-WM describes its Push-T data as a rehosting of the same source. This establishes coverage in the public data source, not membership in the training split used by each released checkpoint or coverage of the perturbed action sequences.

Additional source-swap check.

We extended the synchronized-input evaluation to all 240 roots for all three systems, using observations and proprioception after initialization with the actions and executed effects held fixed. In native coordinates, predictions computed in the recipient's own context are more accurate against that recipient's executed effect than Near or Far predictions, under both equal-cell and energy-pooled aggregation. This source-swap comparison uses a common execution target and is distinct from the original own-context Near-versus-Matched score above.

Action: Appendix D.5 reports the aggregation and public-data coverage checks following Table 25, and extends Table 26 with synchronized contrasts on all 240 roots.

W3: Unnecessary density

Thank you. We revised the Introduction to state the question and contribution more directly, and reorganized Sections 4–5 so that each experiment begins with a plain-language question and the corresponding Results subsection answers it. Detailed definitions and controls are retained in the appendix, making the progression from effect fidelity to interaction dependence and control easier to follow.

Thank you for the detailed response and the new analyses.

On the model comparison, the decomposition and the native-coordinate results clarify what was going on, and I appreciate that contribution 3 is now framed as a comparison of complete systems. This answers my question, though it also means the comparison says less about the predictors themselves than the original framing suggested.

On the donor experiment, the schedule-matched comparison is much more convincing. It confirms that part of the original gap came from the update procedure, and the finding that donor guidance does worse than a random direction is a clearer result than before. I would note that in the equal-displacement test on pushT a random direction beats both model-guided directions, and radius selection gives no gain there, so the evidence for local correction currently rests mainly on reacher.

My specific concerns have been addressed, mostly by narrowing the claims. I will raise my soundness score but keep my overall rating, since the narrowed contributions and uneven control results across tasks still leave me on the reject side.

Reviewer_HJN9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies an important issue in latent world models: the model's predicted latent changes under action perturbations can be misaligned with real environment effects, which explains the failure of many gradient-based planning algorithms. This work designs a diagnostic tool to measure how the predicted latent state shifts given local action perturbations compared to actual execution. They find that strong model responses to action changes depend heavily on the current interaction context, which causes issues when gradient-based planners optimize too far from an initial proposal.

To mitigate this at test time, they propose a local correction algorithm that refines a base controller's action proposal within a trust region, and they evaluate this approach across several empirical benchmarks.

Critical strengths
  1. This is a very novel perspective that identifies an important gap in why gradient-based planning methods fail for latent world models, and their diagnostic tool (comparing predicted effects against same-state replay and source swaps) is simple and intuitive.

  2. The view on separating action response strength from true effect accuracy is insightful and could benefit the world-modeling community.

  3. They supplement their diagnostic tool with a practical test-time local refinement algorithm, and show that this bounded correction works across multiple pretrained models and tasks with very few function evaluations.

Critical weaknesses
  1. It might be hard to choose the safety region (the trust radius) in practice, as the authors also note in their limitations; automatically selecting the correction radius at deployment remains an open question.

  2. Their diagnostics necessitate resetting the simulator to the same state to replay perturbed actions, which is straightforward in simulation environments but might not be feasible in real-world experiments.

Questions for authors

Plesase address my concerns in the weakness section.

2 comments

Thank you for recognizing the value of distinguishing model response strength from effect accuracy. We have added experiments to address your questions about radius selection and simulator replay.

Responses to Weaknesses:

W1: Selecting the trust radius

Thank you. The trust radius should not be selected from outcomes of the episodes used to evaluate the controller. We therefore added a retrospective, episode-grouped five-fold selection experiment, choosing among the original proposal (ρ=0\rho=0) and ρ∈{0.25,0.5,1.0}\rho\in\{0.25,0.5,1.0\} using earlier task data.

All five Reacher folds select ρ=0.5\rho=0.5. This gives 75.56% out-of-fold success, compared with 60.22% for the proposal. We then fixed that radius and the full controller before running the independent experiment described below. The evaluation benefit is therefore not a result of choosing a favorable radius on the new test episodes.

On Push-T, selection maintains the proposal's 90.22% success rather than producing an additional gain. Our result supports a practical task-level selection procedure, while adaptive per-interaction selection remains a separate question.

Action: We added the episode-grouped selection experiment in Appendix D.4 (Table 19), summarized it in Section 5.4, and clarified the distinction between offline selection and per-interaction calibration.

W2: Dependence on simulator replay

Thank you. Same-state replay is used to measure the fidelity of the model's predicted action effects; it is not needed to compute local counterfactual geometry or to execute a correction. The frozen world model predicts alternative futures, and the controller refines its proposal using those predictions, without first executing the unchosen action.

To test whether this distinction holds for a complete controller, we added a randomized single-branch LeWM Reacher evaluation on 354 episode identities not used in the preceding controller evaluations. Assignment was randomized within three frozen proposal actors. Each episode executes only its assigned controller; no alternate plan is replayed from the same state. The correction radius was fixed in advance.

Proposal succeeds in 110/177 episodes (62.15%), compared with 140/177 (79.10%) for full 30-update refinement, with higher success under each of the three actors. This provides independent evidence that the action-response structure can improve control without paired replay. The paired executions remain valuable for the different question of diagnosing individual effect fidelity; in non-resettable settings, that question requires a different validation design.

Action: We added the randomized single-branch evaluation in Section 5.4 and Appendix D.4 (Table 20), and clarified the distinct roles of replay and model-based correction in Section 3 and Appendix D.4.

Comments on the paper

1 comment

We thank the reviewers for their comprehensive and helpful feedback.

Following their comments, we have made the following improvements to the revised manuscript. Selected revisions are highlighted in blue; references below use the revised manuscript's numbering.

  1. New guidance controls. We added direct schedule-matched Donor/Random comparisons and expanded the equal-displacement experiment alongside the existing Fixed/Fresh controls. These experiments examine the effects of guidance source, update persistence, and action movement (Section 5.3; Appendix D.3.2, Tables 15–17).

  2. Radius selection and independent evaluation. We added episode-grouped radius selection and a randomized single-branch test on 354 new Reacher episodes, fixing the radius before collecting the new outcomes. Full refinement improves success from 62.15% to 79.10%, with each episode executing only its assigned controller (Section 5.4; Appendix D.4, Tables 19–20).

  3. New context-dependence analysis. We added an error decomposition that separates source prediction error, genuine environmental variation, and their signed interaction in cross-context reuse. This distinguishes the fidelity of a prediction at its source from its validity when reused in another interaction (Section 5.2; Appendix D.2.1, Table 11).

  4. Extended model evaluation. We extended native-coordinate evaluation, physical-error analysis, and input-synchronization checks to all 240 starting states, and added aggregation and public-data coverage checks. We clarified how encoders and readouts enter the released-system comparisons (Section 5.5; Appendix D.5, Tables 25–26).

  5. Clearer presentation. We revised the Abstract and Introduction to state the scientific question and contributions more directly, clarified the action-response mapping in Section 3, and reorganized Sections 4–5 so that the experimental questions and results follow the same order.

Together, these results support the paper's central contribution: using the action-response mapping of frozen world models to evaluate predicted action effects and refine existing plans. Replay provides the executed reference for fidelity evaluation; action refinement uses the frozen model without paired replay. We thank the reviewers for helping us strengthen the evidence and its presentation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.