acceptodds
Under review as a conference paper at ICLR 2027

Capture Is Not Propagation: Auditing Action Effects in Latent World Models

Abstract

A latent world model can encode a task variable while predicting the wrong effect of acting on it. We separate these capabilities with a capture-gated audit: readouts fitted to real data must capture the current query and resolve executed action effects before they are used to score imagined effects, so that propagation failures are not confused with readout failures. Three LeWorldModel checkpoints on Cheetah pass both gates, yet their imagined five-step effects have from to , well below the zero-effect baseline. The decoded effects are misaligned and amplified even though the complete feature changes align closely with the real ones. Severity varies across Walker, Finger, and other model groups, and the same directional failure appears in independently trained TD-MPC2 runs. On held-out Cheetah action sets, disagreement between model-predicted effects and an environment-fitted reference ranks regret among the candidate actions, so open-loop effect error is visible at the decision level. The same experiments show the limit of any such score: it compares candidate actions with one another, while most of the realized loss comes from the candidate set itself, which no relative score can detect. Evaluating a world model for planning therefore requires testing action effects directly, after capture, and against a no-intervention reference.

Primary area:
reinforcement learning
Keywords:
latent world models, model-based reinforcement learning, world-model evaluation, representation learning, latent dynamics, matched interventions, planning-time fidelity

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_XYq9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a three-stage "capture-gated" audit to evaluate whether latent world models (specifically LeWorldModel, PreJEPA, and TD-MPC2) accurately propagate the effects of actions over a 5-step open-loop rollout. Before scoring imagined rollouts (Gate 3), the protocol requires a frozen linear/affine readout to decode the current task query from the root state (Gate 1) and resolve true action-effect differences from executed environment endpoints (Gate 2). Evaluating checkpoints on DMControl tasks (Cheetah, Walker, Finger), the authors report that models often pass Gates 1 and 2 but achieve negative R2R^2 on Gate 3 imagined effects. They also test whether a discrepancy score between model predictions and a fitted quadratic environment response surface correlates with held-out action-selection regret.

Critical strengths

The distinction between static state decodability (capture) and counterfactual transition fidelity (propagation) is conceptually sensible. Requiring a probe to successfully resolve true endpoint differences (Gate 2) before blaming the latent dynamics model for rollout errors (Gate 3) is a reasonable sanity check that avoids conflating probe underfitting with transition failure.

Critical weaknesses

The paper suffers from severe presentation issues and reads like an automated software test log rather than a scientific manuscript. Simple linear probing and multi-step rollout evaluations are obscured behind heavy, non-standard terminology ("environment families", "matched-shell protocols", "closure coordinates"). Both the main text and appendices are padded with exhaustive administrative and logging minutiae—such as file paths, JSON filenames, and ten synthetic unit-test fixtures in Appendix A.1—while lacking deep mechanistic analysis or any constructive algorithmic remedy to improve world-model training.

Questions for authors
  1. Since a frozen linear readout g^Δ\hat{g}_\Delta trained on real states can easily fail under minor off-manifold latent drift during 5-step open-loop rollouts, how can Gate 3 distinguish genuine loss of action-effect information from simple linear-probe covariate shift (especially given that you acknowledge this ambiguity in Section 7)?
  2. In Section 4.1, Gate 2 fails for LeWorldModel and PreJEPA on environment-random roots and fails for all models on policy-generated roots, yet Section 5.2 notes that the actual regret of these failed-gate models overlaps with passed-gate TD-MPC2 runs in all 48 comparisons. Does this not indicate that your linear gating criteria are overly restrictive and poorly aligned with downstream decision quality?
  3. Given that the proposed discrepancy score requires oracle environment resets to fit a 495-term quadratic response surface, fails to improve over random features in half the settings, and shows no statistically significant gain over "always selecting" on Finger, what is the actionable utility of this audit for practitioners training or deploying world models?
3 comments

No rebuttal provided; the original score remains unchanged.

AuthorsOpenReview ↗

Presentation, where the failure lies, readout shift against genuine loss, and practical use

Thank you for the review. The check you name as a strength is kept: that the readout resolves real end-state differences is established before it is applied to imagined ones, now as a reported number (0.88–0.97) and not as a gate.

Presentation. We agree. The text now describes one design in plain terms. "Environment families", "matched-shell protocols" and "closure coordinates" are gone. File paths, JSON names and the synthetic test fixtures of Appendix A.1 are removed; the appendix keeps the tables behind the reported numbers and the details needed to reproduce them.

No mechanistic analysis or remedy. The added experiments locate the failure. The reward head is much less accurate on differences between actions than on level. Imagined states keep which action was applied, while an affine readout recovers less and less of its consequence with depth, and nonlinear readouts recover no more. The agents plan well with their value and poorly on the model's reward. At the planner's own horizon most of that gap remains when the transitions are the simulator's and only the scoring is learned, and neither a fitted calibration of the head's score nor sharper planner weights close it. From the same starting states, within 25 steps of reward-only control the head orders the candidates poorly (rank correlation 0.10–0.64, against 0.63–0.96 under the full planner at the same steps). We do not propose a training remedy.

Question 1: genuine loss or covariate shift of a frozen linear readout? The revision adds the control for this: the readout is refitted on imagined states at each depth, against a baseline that sees only the starting state and the actions. On TD-MPC2 Cheetah the refitted readout falls from 0.84–0.97 at real end states to 0.31–0.61 after three imagined steps, so the decline is not only shift. On LeWorldModel Finger a readout fitted on predicted states recovers the effect (0.75–0.80), so that failure was shift, and the revision says so. Nor is the decline specific to a linear readout: a two-layer network, kernel ridge and boosted trees, each with a baseline of its own family, show it too, and none recovers more after five imagined steps than the affine readout. The reward-head result uses the model's own head, trained on imagined states. In closed loop, readouts refitted on imagined states do no better than the frozen one (−48 to +21 return points, no consistent sign).

Question 2: failed-gate models overlap passed-gate ones in regret, so are the gates too restrictive? We agree that the pass/fail labels did not track decision quality in that comparison. The revision reports the gates' quantities as numbers and does not gate on them; decision quality is measured directly with the planner.

Question 3: actionable utility of the score. The score is cut. What remains for a practitioner is narrower and, we think, more usable: return and reward accuracy did not show whether these models could be planned with on their reward or reused for a new reward, and tests on matched action sequences did. The conclusion now says what such an evaluation consists of, where a simulator can be reset to a state: compare alternative actions from the same starting state; report conditional prediction and the quality of the selected candidate beside pooled accuracy; repeat both under the controller that will use the prediction; and run the planner with the simulator as its model, and with simulator transitions scored by the learned heads, to see how far transitions and scoring each limit it.

AuthorsOpenReview ↗

Responses and revised paper now posted

Thank you for checking in, and our apologies: our responses went up very late in the discussion period, after your note. They are now posted, as a general response to all reviewers and an individual response to your review, together with a revised PDF.

The revision keeps the submission's claim and measurement and adds the experiments the reviews asked for. In our response to your review, the refitted-readout control for your Question 1 may be the quickest place to start.

We understand that this leaves you little time. If you are able to, we would be grateful for any reconsideration.

Reviewer_qmfNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Latent world models (WM) prediction of actions on task relevant variables. 3 steps evaluation check if a) variable can be decoded from latent state, b) a readout can recover the real (executed) action effect, c) the model predicts effect correctly

3 WM models (LeWorldModel, TD-MPC2, and PreJEPA) are tested on DM Controls and the results are that some models represent the relevant information but are not able to predict the effect of the action. They propose a score (discrepancy score) that should identify bad action choices, but it can't tell when all the available actions are bad.

Critical strengths

I agree that is worth making a distinction between representing a variable and predicting how actions affect it. The results (experiments) include many models' architecture and tasks and the comparison between real and "simulated/imagined" effect is interesting.

Critical weaknesses

My concern is how much all of this is actually helping us with planning. The score can identify bad choices among a set of actions, but cannot tell when all the available actions are bad. I would have liked to see some actual planning experiments.

How general the results are? I feel that some experiments use different readouts or evaluation (LeWorldModel and PreJEPA are evaluated using matched action pulses, while TD-MPC2 uses fixed-radius action sets and a different readout) so how well can we compare the results? Also about the score, you point out that 3/6 comparison is better than random feature control, so how much information is this metric actually extracting?

Questions for authors
  1. Could the authors test whether the audit scores predict actual planning performance? For example, do models with better scores lead to better decisions when used for planning?
  2. What is the idea behind the fact that only 3/6 times the score beats random feature control?
  3. Can we have a general evaluation pipeline across all the WM to see which one is better?
1 comment
AuthorsOpenReview ↗

The planning experiments you asked for, open-loop scores against planning, and comparability

Thanks. The distinction you agree is worth making, and the comparison of real and imagined effects you found interesting, are unchanged. The planning experiments you asked for are the main addition.

How much does this help planning; actual planning experiments (and Question 1). With the agent's own planner in closed loop (overall response, items 3 to 6), the agents plan well with their learned value (852–924; 72–89 without it) and poorly on the model's reward alone (56–75 against 333 for the same planner with the simulator as its model), and the gap persists under a fitted calibration of the reward head's score and under sharper planner weights. On whether a better open-loop score goes with better planning: across 15 models on five tasks we find no established link. The rank correlation between how well the reward head orders candidates and the share of the perfect-model planner's gain that reward-only planning recovers is 0.41 (interval −0.75 to 0.87) on the planner's first set of candidates and −0.35 on its last. The added evaluation on the reward-only controller's own trajectories gives a reason: the open-loop score is measured on the full planner's trajectories. For the Walker Walk model whose head orders candidates best on its task, the rank correlation is 0.91 there and 0.01 on trajectories generated by its reward-only controller, and its closed-loop gain is 0.12. A concurrent preprint on a recurrent state-space world model points the same way with a larger screen: none of 108 offline summaries predicted when planning with the model collapsed, and the best one failed on new training runs (Smolyanskiy & Shock, 2026, arXiv:2607.01736v3).

The score cannot tell when all available actions are bad. The submission stated this limit of any relative score; the revision acts on it. The score is cut, and the closed-loop comparison gives an absolute reference: the same planner with a perfect model.

Generality and comparability: different action sets and readouts for different models (and Question 3). The revision uses one architecture and one procedure on every task, so its numbers are comparable. LeWorldModel and PreJEPA are in the appendix, and we do not rank model families. A common pipeline across families would be useful and is not something this paper provides.

Why only three of six against the random-feature control? (Question 2.) We do not think that result supports presenting the score as a tool, which is why it is cut.

Reviewer_7a5EOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is about understanding how latent world models understand features of the real world. They perform analysis surrounding what world model encoders capture versus what their dynamics models propagate through time (capture vs propagation). The paper performs regression analysis on various open source world model checkpoints using various mujoco tasks like Cheetah, Walker, and Figner. They introduce 3 gates for representation auditing, like Root Capture, real effect resolvability, and imagined propagation. These three metrics check whether 1) a particular state is represented in the encoder, 2) whether action effects can actually be recovered, and 3) whether imagined rollouts are actually dependent on the actions.

Critical strengths

This paper asks an important question about how much encoders capture and what information is propagated through a dynamics model in latent world models. I believe this is an important problem to be investigating as large world models become more prevalent. It is not clear exactly how to measure the efficacy of world models outside of downstream control metrics. The gates (Root capture, real-effect resolution, and imagined propagation) may be of high value to the community. To evaluate their metrics, the authors use state of the art world models (i.e. LeWorldModel) and test on well known benchmarks like the DeepMind Control Suite. In section 3.1, the authors demonstrate that 3 separate LeWorldModel checkpoints “pass” the first two gates of capture and real effect resolvability.

I believe that the gates could be important tools for understanding world models, but in the current analysis their value is explicitly apparent. I address weaknesses related to this in the weakness section.

Critical weaknesses
  1. Insufficient clarity to assess the methodology. The paper introduces a lot of terminology without sufficient explanations of how it maps to the experiments. Terms such as “matched actuator pulses,” “action contrasts,” and “declared support” make it difficult to reconstruct the evaluation procedure and interpret the reported results. This materially limits my ability to assess whether the experiments substantiate the central claims.

  2. Limited insight from the experiments. The analysis identifies failures in selected world models on a small set of DCS tasks, but does not clearly establish what broader conclusions follow or when these failures should be expected. For instance, Section 5 evaluates six candidate action sequences per state across 96 environment families, but the representativeness of this restricted action-selection setting is insufficiently justified.

In general, I find the paper difficult to understand. I will list out several key examples that hampered my understanding of the paper and possibly what might improve the paper so that we can garner insights from the experiments.

For instance in section 2.1, the authors write “a declared scalar or vector task query,” without ever describing what a vector task query is.

Line 87: “pulse studies” -> what is pulse? Line 102: ”centered coefficient of determination” -> is this from other literature at all? What is the source of this metric? Line 108: “For matched actions” -> The authors describe in line 87 that “How the pair is coupled is part of the declared support”, but this has no context Line 142: “repeated roots and action contrasts.” -> action contrasts is never defined anywhere Line 147: “The primary studies use matched actuator pulses.” -> I assume this refers to the literal action commands that are being sent to the Mujoco agent, but I am unsure Line 182 (Section 3.1): The authors discuss quantitative values for their analysis (I.e., R = 0.127, etc) but it’s unclear what the context of these numbers is. Line 192: “Candidate 01” -> I presume that this refers to one of the model checkpoints.

Questions for authors

n/a

2 comments
AuthorsOpenReview ↗

The method in plain terms, your three checks kept as reported numbers, and when the failures occur

Thank you for the detailed list; it showed us where the paper could not be reconstructed. The question you call important is still the paper's, and so are the three checks you describe: whether the variable is represented, whether real action effects can be recovered, and whether imagined rollouts preserve those effects. They are now reported as numbers on one design, without pass/fail labels.

Clarity of the method. From a state the agent reached, the simulator executes and the model imagines the same action sequences, and we compare how the outcome differs between two sequences. The terms you list are removed or defined at first use:

In the submissionIn the revision
"matched actuator pulses", "pulse studies" (lines 87, 147)No longer used. They were brief torque offsets on one actuator in the LeWorldModel experiments, whose results are now an appendix table
"action contrasts" (line 142)"The difference between the outcomes of two action sequences from the same state", defined once
"declared support", "scalar or vector task query" (Section 2.1, lines 87 and 108)Removed. The quantities measured are forward speed and reward
"centered coefficient of determination" (line 102)Ordinary R², error relative to predicting the mean, stated once
"Candidate 01" (line 192)"Model": one of three independently trained world models, reported side by side and never named by number. "Agent" is used only for a model acting through its planner, and is defined once
Bare values such as 0.127 (line 182, Section 3.1)Each number is given with what is compared and what value means no skill

The revision also opens with a schematic of the two quantities it compares: the level (where a sequence ends) and the effect of an action (the difference between two sequences from one state).

What follows from the experiments, and when to expect the failures. The added experiments answer this on five tasks with one procedure. That the return needs the value holds for every model on all five. The reward head's accuracy on level without accuracy on difference (by R²) holds in all three Cheetah Run models and, among the released models, on three of five tasks; on the other two, reward varies more between candidate actions and the reward head resolves them better (an observation made after the results, not a test fixed in advance). A further answer to "when": on all five tasks the reward head chooses worse on trajectories generated by the reward-only controller than on those generated by the full planner (17 of 18 models).

Six candidates per state in 96 environment families (Section 5). We agree that the setting was restricted. Decisions are now those of the agent's own planner, which scores 512 candidates and replans at every step of full episodes.

Reviewer_7a5EOpenReview ↗

Thank you for the response

Hello Authors,

Thank you for your response. I took another look at the updated manuscript and still found it very difficult to parse. I believe the presentation of the paper in the manuscript is not of the caliber expected by this conference. I suggest that the authors state their hypotheses and evidence in the paper more plainly and up front, then delve into detail where necessary. As it reads currently, the manuscript gets hyper-specific for some details and glances over others to the point that the narrative is unclear. Therefore, I maintain my score.

Reviewer_SXtyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines whether latent world models correctly predict the effects of actions. The proposed audit first checks whether task variables and real action effects can be decoded from representations, then compares executed and imagined outcomes using the same frozen readout. Experiments on LeWorldModel, PreJEPA, and TD-MPC2 show that state decodability and high overall feature similarity do not guarantee accurate action-effect predictions. The paper also studies the relationship between these prediction discrepancies and action-selection errors within a fixed candidate set.

Critical strengths

The experiments check the reliability of the readout before attributing decoding failures to the world model. Using the same frozen readout for executed and imagined outcomes, together with environment-family-level data splits and confidence intervals, makes the reported differences more credible.

Critical weaknesses
  1. I remain unconvinced by the novelty. The distinction between information being present in a representation and being used correctly is well established, as is the distinction between prediction accuracy and control performance. The paper’s more specific contribution is the combination of matched interventions and two-stage readout checks. However, the experiments mainly provide further examples of these discrepancies, without explaining why they arise or which training and architectural choices cause them. The evaluation procedure has some value, but the new understanding it provides remains limited.

  2. The implications of the diagnosed errors for actual model use remain unclear. A readout that works on encoded real states may not reliably interpret model-predicted states. The experiments do not distinguish inaccurate dynamics predictions from readout failure on predicted states. The subsequent decision experiments are limited to selecting among six predefined actions according to a physical query; they do not establish whether the native planner makes worse choices or whether closed-loop performance suffers. The discrepancy score clearly outperforms the random-feature control in only three of six evaluations, while selecting actions directly with the environment reference model already achieves lower regret. It is therefore unclear what additional information the diagnostic score provides.

Questions for authors
  1. Is there experimental evidence that distinguishes incorrect action-effect predictions from the frozen readout’s inability to interpret predicted states? Do the existing readout sensitivity analyses support either explanation?

  2. Is the diagnostic score associated with native-planner selection errors or closed-loop returns? Does it provide information beyond simple prediction-error measures or direct use of the environment reference model?

1 comment
AuthorsOpenReview ↗

The readout-versus-dynamics control, the native planner in closed loop, and what the revision adds

Thank you for the review. The checks you list as strengths are kept: a readout is shown to work on real end states before its score on imagined ones is read, splits keep whole episodes together, and intervals are reported throughout. Your two questions set the two main additions.

Novelty: the distinctions are known, and the experiments add examples without explaining them. We agree that "present versus used" and "prediction accuracy versus control" are established, and the revision says so. What it adds is an audit, inside successful agents, of which measurement supports which use of the model. Scoring differences between actions from one state shows errors that the variance of reward between states conceals (one reward head, R² 0.97–1.00 on level and −1.42 to 0.54 on differences). A baseline that sees only the starting state and the actions measures what an imagined end state adds. And in closed loop the same learned dynamics support strong value-guided choices while planning on the model's reward stays poor, which the return does not reveal because the planner scores candidates with the learned value. On why: over the planning horizon, candidate actions change the reward by about as much as the head's error on its level (0.04–0.06 against 0.03–0.10); neither a fitted rescaling of the head's score nor sharper planner weights close the gap; and the head orders candidates well on the full planner's trajectories and poorly, within a few steps, on the reward-only controller's (overall response, items 4 and 6). We do not identify which training choices cause this.

Inaccurate dynamics, or a readout that cannot interpret predicted states? (Question 1.) The revision adds the control that separates them: the readout is refitted on imagined states at each depth and compared with a baseline that sees only the starting state and the actions.

  • TD-MPC2, Cheetah: the refitted readout predicts the five-step effect on speed at 0.84–0.97 from real end states, 0.31–0.61 after three imagined steps and 0.14–0.53 after five. The frozen readout scores below the refit, so part of its failure was transfer, as you suspected; the refitted readout also recovers less at each imagined step.
  • LeWorldModel: on Finger a readout fitted on predicted states recovers the effect (0.75–0.80), so the failure reported there was one of transfer. On Walker a refitted readout does not recover it (0.22–0.29, no better than the actions alone).
  • The reward-head result uses the model's own head, which is trained on imagined states, so readout transfer does not apply to it.

These are statements about what a readout recovers, and they are not specific to an affine one: with a two-layer network, kernel ridge or boosted trees in place of the affine readout and its baseline, the share still falls with every imagined step, and none recovers more after five steps than the affine readout.

Does the native planner choose worse, and does closed-loop return suffer? (Question 2.) As you asked, decisions are now those of the native planner in full episodes (overall response, items 3 to 6). The native planner does not choose badly: its first choice with imagined rollouts keeps 0.95–0.99 of what the same objective finds with real rollouts, and its return needs only the value. The reward head's first choice among the same candidates keeps 0.46–0.58 of the available gain in real reward, and 0.17–0.28 on trajectories generated by the reward-only controller. What does badly in closed loop is planning on the model's reward alone (56–75 against 333 at the planner's horizon), and simulator transitions scored by the model recover 3–36% of that gap.

The discrepancy score beats the random-feature control in three of six evaluations, and the reference model does better. We agree, and the score is cut. The closed-loop comparison takes its place.

Comments on the paper

2 comments

Dear authors and reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

AuthorsOpenReview ↗

General response: what the revision keeps, what it adds for the reviewers, and its findings

We thank the reviewers. The revision keeps the submission's title, claim, measurement and three TD-MPC2 Cheetah models (Section 4 of the submission), and adds the experiments the reviews asked for. The claim: a latent world model can represent a task variable and still predict the wrong effect of acting on it. The measurement: from one state, the same action sequences are executed in the simulator and imagined by the model, and a readout that resolves the real effect is applied to the imagined one.

What changed, following the reviews

  • Added. (a) Closed-loop planning with the agent's own planner, replacing the six-candidate study (SXty, qmfN, 7a5E). (b) Readouts refitted on imagined states against a baseline that sees only the starting state and actions; nonlinear readouts; and the model's own reward head, which needs no fitted readout (SXty Q1, XYq9 Q1). (c) One procedure on released models for five tasks (qmfN, 7a5E). (d) Checks on why reward-only planning falls short (SXty, XYq9): simulator transitions, sharper planner weights with a fitted calibration of the reward head's score, and the reward head evaluated on the reward-only controller's own trajectories.
  • Moved to the appendix or cut, where the reviews found little value. The discrepancy score is cut (SXty, qmfN, XYq9). The gates' three quantities are reported as numbers, without pass/fail labels (XYq9 Q2). LeWorldModel and PreJEPA are in the appendix: their protocols differed from TD-MPC2's (qmfN), and TD-MPC2's own planner, value and reward head make the closed-loop tests possible. File-level detail and test fixtures are removed (XYq9).
  • Presentation. One design in plain terms, with a schematic of the two quantities compared (7a5E, XYq9).

Findings. Item 1 is the submission's comparison with the requested controls; items 2 to 6 are additions. The agents return 852–924 of 1,000 on Cheetah Run.

  1. After five imagined steps an affine readout still predicts speed from the latent state (R² 0.98–0.99) and decodes the applied actions. Of the actions' effect on speed that the starting state and actions leave unexplained, the end state explains 0.88–0.97 when real and 0.11–0.38 after five imagined steps. A network, kernel ridge and boosted trees recover no more.
  2. The reward head predicts reward with R² 0.97–1.00, but 99.3–99.7% of that variance lies between starting states: giving every candidate its state's mean reward would score 0.993–0.997. For the difference between two action sequences from one state its R² is −1.42 to 0.54, and its rank correlation with the real difference 0.38–0.66.
  3. In closed loop the agents plan well with the learned value (852–924; 807–899 with the value alone; lookahead helps) and return 72–89 without it. On the model's reward alone they return 56–75 at the three-step horizon, against 333 for the same planner with the simulator as its model; for a new reward (a 5 m/s speed target), 110–142 against 654.
  4. Simulator transitions scored by the model recover 1–17% of that gap at three steps (64–68% at ten steps for two models, 9% for the third). A fitted scalar correction of the head's score leaves the planner unchanged, since it weights its kept candidates equally; with sharp weights (temperatures 50 and 500) the gap remains (47–74 against 345–373).
  5. On released models for five DMControl tasks, chosen by rules fixed in advance, no agent keeps its return without its value (29–235 against 866–997), and reward-only planning earns at most half of the perfect-model planner's gain over random actions for 13 of 15 models. The reward head's R² is at least 0.9 on level and at most 0.5 on differences on three of the five tasks.
  6. The reward head orders candidates well on the full planner's trajectories and poorly on the reward-only controller's. From the same starting states, 25 to 75 steps later its rank correlation with real reward is 0.63–0.96 under the full planner and 0.10–0.64 under the reward-only planner, lower for all 18 models; its first choice retains less of the available gain for 17.

Conclusion (the submission's, now with closed-loop evidence). Aggregate accuracy can conceal poor comparisons between actions from the same state (1, 2). Useful value-guided planning can coexist with weaker reward and physical predictions from the same model (3 to 5). Those predictions have to be evaluated at the states and candidates of the controller that will use them (6).

A correction. The submission said that for LeWorldModel the complete feature changes of executed and imagined rollouts "align closely". That came from a block of the state that copies the applied actions. In the part the model predicts, the alignment is near zero (cosine −0.08 to 0.08). The statement is withdrawn; no finding above depends on it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.