acceptodds
Under review as a conference paper at ICLR 2027

VPFlow: Value-Predictive Planning with Imperfect World Models for Offline Goal Reaching

Abstract

Offline goal-conditioned reinforcement learning turns unlabeled trajectories into reward-free skill acquisition: relabel future observations as goals, and one dataset supervises many tasks. Model-based planning could add test-time compute on top, but existing planners share an assumption: the world model must predict the next state, in pixels or in latents, faithfully enough to plan with. We challenge this assumption. Offline goal reaching admits a closed-form Monte Carlo return under sparse rewards, so value learning becomes regression instead of temporal-difference bootstrapping; we use this return as the sole supervision for the latent dynamics, training imagined latents to predict goal-reaching value rather than the next state, which prevents representation collapse without stop-gradients or target encoders. We present VPFlow, which pairs this deliberately imperfect world model with a flow policy and steers it at test time in the policy's noise space, avoiding the out-of-distribution action queries of action-space planning. On 30 OGBench datasets with state or pixel inputs, VPFlow raises mean success from to (state) and from to (pixel) over the strongest baselines, even though its state-prediction error is orders of magnitude larger and its imagined latents stray far from the manifold of encoded states; reducing that error lowers success. Planning needs value-predictive representations, not faithful simulation.

Primary area:
reinforcement learning
Keywords:
World Models, Representation Learning, Flow-based Policy, Noise Steering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zcxGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VPFlow, an offline goal-conditioned reinforcement learning framework that challenges the assumption that effective planning requires accurate prediction of future states. Instead of training a latent dynamics model to reconstruct next-state representations, VPFlow learns value-predictive dynamics using closed-form Monte Carlo returns derived from hindsight goal relabeling. The resulting latent representations are optimized for goal-reaching value estimation rather than state reconstruction. The framework further incorporates a flow-matching policy as a behavioral prior and performs test-time planning by optimizing the policy's input noise instead of directly searching over action sequences. The paper develops an evolutionary noise-steering procedure based on GAE and uses receding-horizon execution. Experiments are conducted on 30 OGBench datasets covering state- and pixel-based navigation and manipulation tasks. VPFlow achieves strong overall performance compared with several model-free and latent model-based baselines. The representation ablations show that substantially reducing next-latent prediction error does not necessarily improve task success, while the planning ablations demonstrate the importance of the search space and test-time optimization.

Critical strengths

1.The most compelling contribution of this work lies in its distinction between predicting future states and predicting decision-relevant values; rather than assuming that an accurate latent simulator is a prerequisite for planning, VPFlow directly trains latent dynamics using goal-reaching value supervision. This perspective is supported by the experimental results as shown in Figure 5 and Figure 7. These findings provide a empirical argument that prediction fidelity and control utility should be evaluated as distinct metrics. 2.Using a flow policy as a behavioral prior enables searching for improved actions while retaining offline demonstration structure. Table 2 shows a substantial gap between action-space and noise-space CEM, highlighting the importance of search parameterization. The evaluation is broad, covering 17 state-based and 13 pixel-based OGBench datasets, with encouraging gains on challenging manipulation tasks. Appendix results and optimizer comparisons further strengthen the empirical presentation. Together, the conceptual motivation and demonstrated effectiveness of noise-space planning make the paper a worthwhile contribution.

Critical weaknesses
  1. Section 5.2’s comparison between the proposed value-predictive objective and a variant with an added next-latent matching loss is a useful controlled experiment, but it does not establish that state-predictive world models are generally inferior for planning. The observed performance drop may reflect an interaction between the representation-matching objective and the specific training architecture, and although the near-zero prediction error in Figure 5(c) is consistent with representation collapse, prediction error alone does not conclusively identify collapse as the underlying mechanism. The conclusion should distinguish the limitations of the tested state-matching objective from broader claims about state-predictive models.
  2. Although ε-ES is presented as a central component of VPFlow, Appendix D (Table 4) reveals that several task families actually use first-order noise-space optimizers, with state-based AntMaze employing ε-SGD and visual AntMaze using ε-Adam. This distinction is critical for interpreting Table 1: the reported performance validates the VPFlow framework with task-dependent noise-space optimization, rather than exclusively demonstrating the superiority of ε-ES.
Questions for authors
  1. In Section 5.2, the state-predictive variant adds the latent-matching objective to the value-predictive training objective. Could the authors clarify whether their intended conclusion is that this particular additional objective is detrimental within VPFlow, rather than that state-predictive representations are generally unsuitable for planning? Relatedly, what evidence supports representation collapse as the explanation for the near-zero prediction error in Figure 5(c), beyond the prediction error itself?

  2. Appendix D, Table 4 indicates that the reported results use ε-SGD or ε-Adam for some navigation tasks, while ε-ES is used for other task families. Could the authors clarify the relationship between these optimizer choices and the results reported under VPFlow in Table 1?Would it be more accurate to describe the main contribution as value-predictive planning with noise-space steering, with ε-ES being one effective optimizer rather than the optimizer used throughout?

2 comments
AuthorsOpenReview ↗

Author Response to Review zcxG (1/n)

We sincerely thank the reviewer for the valuable feedback and for acknowledging our work on the compelling contribution with support and effectiveness demonstration with broad evaluation.

Revisions that directly aim to address the reviewer's concerns are highlighted in blue in the updated manuscript and mentioned below.

Weaknesses:

  1. The performance drop in Section 5.2 may reflect an interaction between the representation-matching objective and the specific training architecture, and although the near-zero prediction error in Figure 5(c) is consistent with representation collapse, prediction error alone does not conclusively identify collapse as the underlying mechanism. The conclusion should distinguish the limitations of the tested state-matching objective from broader claims about state-predictive models.

We acknowledge that the conclusion is tied to particularly our VPFlow architecture, but this conclusion should be viewed as constructive for a new perspective and promise rather than affirmative for all state-predictive models.

To provide stronger support for the representation collapse attribution behind zero success rate and that value prediction can implicitly prevent it, we have added a more rigorous ablation on the learning objective in Appendix D.2, including Figure 13 that shows collapse on 8 datasets in Visual Cube and Visual Puzzle, and Figure 14 that visually shows the degeneracy of representations to a single point.

We have also included a discussion on this issue in Appendix F.1, line 1921-1925.

  1. Although ε-ES is presented as a central component of VPFlow, Appendix D (Table 4) reveals that several task families actually use first-order noise-space optimizers, with state-based AntMaze employing ε-SGD and visual AntMaze using ε-Adam. This distinction is critical for interpreting Table 1: the reported performance validates the VPFlow framework with task-dependent noise-space optimization, rather than exclusively demonstrating the superiority of ε-ES.

Note: we have expanded the original Table 4 in Appendix D to Table 9 with per-dataset planning hyperparameters in the revision.

It is true that we changed the zeroth-order ε\varepsilon-ES described in the main text to the first-order ε\varepsilon-SGD/Adam on 12 locomotion tasks, because we empirically found out that they perform better.

If we keep ε\varepsilon-ES throughout all 30 datasets, the overall success rate in Table 1 will slightly decrease from 63.2 to 59.6 for state-based tasks and from 37.6 to 34.2 for pixel-based tasks. VPFlow still outperforms all other baselines and does not affect our core claim, but the gain is smaller.

This is a by-product finding during our experiment. Therefore, we have presented the complete per-dataset success rate with either the zeroth- or first-order planner in Table 12 as justification for the planner choice and discussed it in Appendix B.6.

AuthorsOpenReview ↗

Author Response to Review zcxG (2/n)

Questions:

  1. Could the authors clarify whether their intended conclusion in Section 5.2 is that this particular additional objective is detrimental within VPFlow, rather than that state-predictive representations are generally unsuitable for planning?

Our conclusion from Section 5.2 is drawn from VPFlow itself and its ablations with different objective functions (see Appendix D.2). The selling point is to demonstrate that we can successfully achieve strong offline planning performance, and even better than with next-latent-state-prediction, therefore rendering state prediction as an unnecessary objective. This contrasts with the current belief and practices in latent world modeling. This phenomenon is grounded in the value-equivalence principle we discuss in Section 4.1, so we expect it to have some generality and practical implications.

However, we currently do not intend to overclaim the conclusion beyond the scope of our setting: offline planning with reconstruction-free latent world models, with the closest work as the JEPA-style latent planner we compare against, LeWM [1], which is an emerging area of research.

[1] Lucas Maes, et al. LeWorld-Model: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv, 2026.

  1. Could the authors clarify the relationship between the optimizer choices and the results reported under VPFlow in Table 1?

We have addressed this question in weakness 2 above. The intuition is that, for locomotion tasks, the dynamics is relatively smooth, making the first-order gradient sufficient and more informative than the zeroth-order planning. Conversely, for manipulation tasks, the dynamics may be highly non-convex and non-smooth, where the zeroth-order planner helps explore the search space and escape local optima.

  1. Would it be more accurate to describe the main contribution as value-predictive planning with noise-space steering, with ε-ES being one effective optimizer rather than the optimizer used throughout?

Yes. We have stated in line 125 of the summarized contributions that "We propose a test-time planning algorithm tailored to generative flow policies in their noise space ...", removing the narrative that may imply the superiority of ε\varepsilon-ES throughout all datasets.

Reviewer_BuprOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes VPFlow (Value-Predictive Flow), a model-based planning method for offline goal-based reinforcement learning which challenges the common assumption that a world model must faithfully predict the next state in order to plan well. The key observation is that under sparse rewards, the Monte Carlo return of any state to a future goal is a closed-form function of their temporal distance, so value function learning becomes a regression task. VPFlow uses this return as the supervision for training imagined latents to predict goal-reaching value rather than the next state directly. The paper also proposes search in the noise space of a flow policy at test time, using an evolutionary strategy to keep candidate actions close to the data distribution to avoid out-of-distribution action queries which can affect action-space planners. On 30 OGBench datasets, VPFlow reports large gains over model-free baselines, particularly on state-based manipulation.

Critical strengths
  1. The method’s design choices are well matched to the setting. The use of closed-form Monte Carlo returns means that goals are relabeled from the same trajectory, and the value targets are fixed regression targets computed directly from the data, with no bootstrapping, target networks, or EMA. This makes training conceptually simple and stationary. The choice to search in the policy’s noise space is also supported empirically. Table 2 shows that action-space CEM performs worse than no search at all, while noise-space search improves substantially on the reactive policy. This directly supports the paper's argument that keeping search close to the policy's learned distribution avoids exploiting the value function away from the training data, and it's a useful and practical finding for planning with generative policies.
  2. This paper reports strong benchmark results, especially on state-based manipulation. On 17 state datasets, VPFlow nearly doubles the mean success of the strongest flat baseline and leads every state environment group in Table 1, with particularly large gains on manipulation tasks such as scene-play and puzzle-4x4. It also improves the pixel-based average. The appendix is also very thorough, with pseudocode for every planner variant, full training and steering hyperparameters including search grids, and per-dataset results with standard deviations.
Critical weaknesses
  1. The experiment described in Section 5.2 is central to the paper's main claim, but it does not cleanly isolate the effect of state prediction. First, the text says that these experiments change only one thing, but Appendix B.3 suggests that the state-predictive control (Eq. 14) differs from VPFlow in three ways: (1) its value heads are trained on encoded latents rather than imagined ones, (2) it adds a latent-matching term, and (3) it uses no stop-gradient, so the encoder is updated through the target. This is therefore not a clean comparison between value-predictive and state-predictive training. The collapse explanation also rests on a single dataset (Figure 5c). On the other three, the control's error remains far from zero (0.21, 3.5, 2.9), yet its success still drops, so collapse cannot be the general failure mode.
  2. The empirical evaluation is somewhat incomplete in several ways:
    • On many tasks, VPFlow evaluates 512 rollouts per generation for up to several generations at every replanning step, while the baselines act with a single forward pass, and no wall-clock figures are reported. Including a compute-matched comparison would be beneficial here.
    • The closed-form targets used for VPFlow exist only for goals that appear later on the same trajectory, and this is confirmed by the mask in Equation 6. The value function learns the behavior policy's route between states and goals, not the best route, and also receives no supervision at all for pairs that never occur in the same trajectory. This gives up experience stitching, which TD learning can do. Since Monte Carlo regression is presented as a cleaner and simpler approach than TD learning, these types of tradeoffs should be discussed in the main text, and the exclusion of the OGBench stitch datasets should be explained.
    • The baselines seem incomplete. Appendix E.1 states that HIQL is excluded because VPFlow is a flat controller. That reasoning is debatable for a method that spends substantial test-time compute on search, and HIQL is among the strongest OGBench methods. There is also no comparison with the closest model-based offline methods the paper cites, such as MBOP or GOPlan. The model-based baselines that are included are stated to score zero on 12 of 13 pixel tasks (Section 5.1), which may indicate an implementation problem.
Questions for authors
  1. Figure 1 marks TD-MPC as not reward-free, yet it is run here on reward-free play data. How was it given a reward signal? How were TD-MPC2, PLDM, and LeWorldModel tuned and trained?
  2. EfficientZero (Ye et al., 2021) adds a self-supervised consistency loss to MuZero and reports that it improves performance, seemingly the opposite conclusion of this paper. Can you explain which differences in setting yields these opposing results?
  3. What is the wall-clock time per environment step for VPFlow and the baselines, and how does performance compare when the baselines are given matched test-time compute to VPFlow?
8 comments
AuthorsOpenReview ↗

Author Response to Review Bupr (1/n)

We sincerely thank the reviewer for the valuable feedback and for acknowledging the direct support of arguments, useful and practical finding for planning with generative policies, and strong benchmark performance.

We have improved the writing and presentation of the paper to make it easier to follow.

Revisions that directly aim to address the reviewer's concerns are highlighted in blue in the updated manuscript and mentioned below.

Weaknesses:

  1. The experiment described in Section 5.2 is central to the paper's main claim, but it does not cleanly isolate the effect of state prediction.

We agree with this concern that we miss an intermediate step.

To be more coherent with the experiments in the main text, we have removed the original phrasing of "This experiment changes one thing", "... adds the latent-matching term" in Sec 5.2 and changed it to "This experiment aims to verify which representations are more favorable for planning ... which matches the predicted latents to the encoded states." in line 410-414.

To isolate the effect of state prediction, we train a new variant on the 13 pixel-based datasets we report with LVP+λstateLstate-matching\mathcal{L}_{\text{VP}}+ \lambda_{\text{state}} \mathcal{L}_{\text{state-matching}}, where Lstate-matching=E[1H∑h=1H∥z^t+h−zt+h∥22]\mathcal{L}_{\text{state-matching}}=\mathbb{E}\left[ \frac{1}{H}\sum_{h=1}^{H}\|\hat z_{t+h}-z_{t+h}\|_2^2 \right].

It regularizes the imagined next states with value prediction, unlike LLP\mathcal{L}_{\mathrm{LP}} in Appendix B.3 on the encoded next states. These three losses are summarized in the following table:

Lossw/o state matchingw/ imagined value supervisiondynamics outputprediction objective type
LVP\mathcal{L}_{\text{VP}} (Equation 6)✓✓valuevalue-predictive
LVP+λstateLstate-matching\mathcal{L}_{\text{VP}}+ \lambda_{\text{state}} \mathcal{L}_{\text{state-matching}}✗✓value + next latent statestate-predictive
LLP\mathcal{L}_{\text{LP}} (Equation 13)✗✗next latent statestate-predictive

The results are presented in the following table, averaged over the mean success rate of all datasets in each environment. It shows that adding state matching still reduces performance when imagined-value supervision is preserved.

It also shows that, on all 8 datasets from Visual Cube and Visual Puzzle environment, the success rate degrades to zero, meaning the representation collapse is not just a special case. Adding value prediction on the imagined states raises the success rate above zero, but does not match LVP\mathcal{L}_{\text{VP}}.

LossVisual AntMaze (4 datasets) (%)Visual Cube (4 datasets) (%)Visual Scene (1 dataset) (%)Visual Puzzle (4 datasets) (%)
LVP\mathcal{L}_{\text{VP}} (Equation 6)61.034.833.018.3
LVP+λstateLstate-matching\mathcal{L}_{\text{VP}}+ \lambda_{\text{state}} \mathcal{L}_{\text{state-matching}}50.531.526.07.8
LLP\mathcal{L}_{\text{LP}} (Equation 13)45.80.030.00.0

This experiment has been included in the updated manuscript (Appendix D.2, including Table 10 for loss comparison, Figure 12 for success rate vs. next-latent prediction error similar to Figure 5, and Figure 13 for success rate trained with these three losses on pixel datasets) and is referred to in the main text in Sec 5.2, line 426.

This should provide stronger evidence to support our claim that predicting value can implicitly prevent representation collapse and is more favorable for planning than predicting states in Sec 5.2.

AuthorsOpenReview ↗

Author Response to Review Bupr (2/n)

  1. The empirical evaluation is somewhat incomplete in several ways:
  • missing wall-clock time comparison with baselines.

We measured action-selection wall-clock time on the visual-scene-play-v0 dataset on NVIDIA A100 GPU, compared with model-free baselines. VPFlow's planning uses 512 number of candidates for 3 iterations. The wall-clock time per action at an environment step is shown below.

MethodWall-clock time / action (ms)
VPFlow10.88
GCBC1.03
GCIVL1.13
GCIQL1.12
QRL1.01
CRL0.95
    • the trade-offs compared to TD learning on trajectory stitching should be discussed in the main text, and the exclusion of the OGBench stitch datasets should be explained.

The closed-form Monte Carlo return is computed from the temporal distance. However, the temporal distance is not strictly defined or available for states and goals that are not on the same trajectory.

We acknowledge that trajectory stitching is an important but currently challenging task for our learning objective. We want to emphasize that the scope of our work is to plan with continuous play trajectories, which aligns with the closest work LeWM [1]. How to leverage the fragmented trajectories for optimal shortest-path planning is an open question worth exploring.

We add a sentence on the trade-off in the main text (line 250-252) and explain the exclusion of stitching datasets in Appendix B.1 (line 951-953).

    • the baselines seem incomplete. The reason to exclude HIQL because VPFlow is a flat controller is debatable. There is also no comparison with the closest model-based offline methods the paper cites, such as MBOP or GOPlan.

HIQL [2] is a model-free and hierarchical method, while our VPFlow is a model-based and non-hierarchical planner. We think HIQL belongs to an orthogonal line of approaches and is not as close to VPFlow as other model-free baselines. We have added a discussion in Appendix E (line 1785-1787).

That said, we still achieve better overall success rate on state-based datasets than HIQL (63.2 vs. 34.3) and slightly worse performance on pixel-based datasets (37.6 vs. 42.8). Note that HIQL is more computationally intensive compared with other flat model-free baselines. We believe extending our method to a hierarchical planning setting for long-horizon tasks will be valuable future work.

As for MBOP [3] and GOPlan [4], we do not find the official open-source code of MBOP; thus, we only adapt and run GOPlan with the same training and evaluation protocol as other baselines, with its algorithm unchanged.

We find out that GOPlan still cannot achieve success rate above zero on 13 pixel-based datasets, even equipped with the same CNN encoder as our method.

Note that our method and the model-based baselines LeWM, PLDM, and TD-MPC2 we compare against are all reconstruction-free latent planners primarily on pixels, while GOPlan mainly focuses on state inputs. So, our work is closer to LeWM, PLDM, and TD-MPC2 than GOPlan.

    • the model-based baselines that are included are stated to score zero on 12 of 13 pixel tasks (Section 5.1), which may indicate an implementation problem.

We have included the implementation details of running these model-based baselines in Appendix B.8 and their configurations in Table 3.

We have tried our best to tune their hyperparameters and training settings, for instance, using privileged information like qpos and qvel for pixel inputs in TD-MPC2, but they are still not able to achieve non-zero success rate on these pixel-based datasets, except for LeWM.

Another thing to note is that our closest model-based baseline, LeWM [1], conducts planning in relatively simpler environments, such as pushT and reacher, and focuses on learning from short-horizon trajectories (≤200\leq 200 steps) with data collected by its own scripts. Our benchmarking exposes its limits on the original harder OGBench pixel-based datasets for offline goal-reaching with multi-task scenarios. We believe these benchmark results should be meaningful rather than implementation problems.

We will open-source our code and how we benchmark these model-based baselines upon acceptance of our paper.

AuthorsOpenReview ↗

Author Response to Review Bupr (3/n)

Questions:

  1. Figure 1 marks TD-MPC as not reward-free, yet it is run here on reward-free play data. How was it given a reward signal?

''Reward-free'' in Figure 1 means that the dataset itself does not contain any reward, but only states and actions. We relabel each state in hindsight with a sparse binary reward based on whether it reaches the sampled goal state, as defined in Equation 1.

TD-MPC2 is not reward-free because it has a reward head that is trained to predict this sparse reward.

  1. How were TD-MPC2, PLDM, and LeWorldModel tuned and trained?

We have included the implementation details of these baselines in Appendix B.8. Specifically, all three baselines are adapted using the open-source stable-worldmodel\texttt{stable-worldmodel} library [5].

  1. EfficientZero (Ye et al., 2021) adds a self-supervised consistency loss to MuZero and reports that it improves performance, seemingly the opposite conclusion of this paper. Can you explain which differences in setting yields these opposing results?

The difference is that

  • EfficientZero, as an off-policy online RL method, adds this consistency loss mainly to improve sample efficiency with replay buffer and mitigate distribution shifts, where a faithful dynamics model may offer richer signal for value learning; while our method is an offline planner for goal reaching, where value learning isn't a bottleneck and adding this consistency loss may learn an overly conservative dynamics model that hinders generalization on unseen goals.
  • EfficientZero learns value function from replay buffer with noisy bootstrapping, while our work learns with closed-form Monte Carlo regression that provides a clean and strong signal.
  • The different use of value function in online planning with MCTS in EfficientZero and noise-space steering with generative policy in our work may also create opposing results.

Also notice that, according to the ablation of Table 7 in EfficientZero, removing the consistency loss reduces the mean score from 1.943 to 0.881, but is still higher than 0.562 of MuZero. There is no direct evidence that simply adding this single consistency loss will improve on MuZero.

Therefore, given these different settings and the less direct evidence, we believe there is a gap between our work and EfficientZero, and the conclusion of EfficientZero does not directly translate to or contradict our work.

  1. What is the wall-clock time per environment step for VPFlow and the baselines, and how does performance compare when the baselines are given matched test-time compute to VPFlow?

We have addressed this question in weakness 2 above in the third table for model-free baselines. VPFlow's per-action wall-clock time is about 10$\times$ longer.

Test-time compute is only possible for model-based methods that support search. We therefore compare with baseline LeWM [1] below by scaling its compute, particularly the number of planning samples and iterations, and report the FLOPs and success rate.

VPFlow's compute, indicated by GFLOPs per environment step, is slightly larger than LeWM's default configuration, but VPFlow is substantially faster in wall-clock time. Increasing LeWM's compute, i.e., larger planning samples or iterations, to match VPFlow's GFLOPs does not improve its performance much.

MethodSamples × iterations (Scene / Cube)Wall-clock time/action (Scene / Cube) (ms)GFLOPs/env step (Scene / Cube)visual-scene-play-v0 (%)visual-cube-triple-play-v0 (%)
VPFlow (Ours)512 × 3 / 512 × 210.88 / 9.98141.52 / 165.783320
LeWM default300 × 30 / 300 × 3026.25 / 25.26126.47 / 126.4702
LeWM FLOP match (samples)335 × 30 / 393 × 3031.98 / 36.89141.16 / 165.5102
LeWM FLOP match (iterations)300 × 33 / 300 × 3934.15 / 38.04139.07 / 164.2503

[1] Lucas Maes, et al. LeWorld-Model: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv, 2026.

[2] Seohong Park, et al. HIQL: Offline goal-conditioned RL with latent states as actions. NeurIPS, 2023.

[3] Arthur Argenson, et al. Model-based offline planning. ICLR, 2021.

[4] Mianchu Wang, et al. GOPlan: Goal-conditioned offline reinforcement learning by planning with learned models. Transactions on Machine Learning Research, 2024.

[5] Lucas Maes, et al. stable-worldmodel: A platform for reproducible world modeling research and evaluation. arxiv, 2026.

Reviewer_dvkMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VPFlow for offline goal-conditioned reinforcement learning. It jointly trains an encoder, latent dynamics model, and goal-conditioned value function using Monte Carlo returns computed from temporal distances to hindsight goals. The paper evaluates VPFlow on 30 OGBench datasets and reports higher aggregate success rates than the evaluated model-free baselines for both state and pixel inputs.

Critical strengths
  1. Broad evaluation and substantial performance improvements. The evaluation covers 17 state-based and 13 pixel-based datasets. In Table 1, VPFlow achieves aggregate success rates of 63.2% and 37.6%, respectively, compared with 32.9% and 27.5% for the strongest evaluated model-free baselines by aggregate performance. These results demonstrate the practical promise of the proposed system.

  2. The method uses observed returns to hindsight goals as direct supervision, learning value-predictive dynamics without TD bootstrapping or explicit latent matching in its default model objective. This is an interesting use of the structure available in offline goal-reaching data.

Critical weaknesses
  1. The scope and significance of the contribution beyond existing value-predictive models remain insufficiently clear.

Learning models that preserve information relevant to control, rather than faithfully reconstructing states, is an established perspective in Value Prediction Networks, MuZero, and work on value equivalence. The authors acknowledge this lineage. Consequently, the broad message that planning does not require faithful state simulation does not, by itself, clearly identify the additional insight contributed by this paper.

I understand the specific distinction to be the use of hindsight Monte Carlo returns to directly supervise values at imagined latents in offline goal reaching, combined with flow-policy noise-space planning. However, the current evidence does not sufficiently establish what advantages this MC supervision provides over existing value-predictive learning approaches, or how much of the performance improvement is attributable to this particular learning formulation.

A more precise statement of the contribution, supported by comparisons that hold the policy and planner fixed, would make its novelty and significance easier to assess.

  1. The central ablation does not adequately isolate the causes of the performance differences.

Section 5.2 describes the state-predictive control as adding a latent-matching term to the default objective while keeping the remaining configuration unchanged. However, Equation 14 in Appendix B.3 is not simply Equation 6 with an additional latent-matching term.

Equation 6 supervises values evaluated at predicted latents, directly training the dynamics through value-prediction errors. In contrast, Equation 14 supervises values evaluated at encoded dataset states, while training the dynamics through latent matching. The comparison therefore changes both the state-prediction constraint and the direct value supervision applied to the dynamics.

These results show that the proposed training formulation outperforms this particular control. They do not distinguish whether the degradation arises from adding latent matching or from removing imagined-latent value supervision. Consequently, the interpretation that reducing state-prediction error itself harms control is stronger than the evidence supports.

The planning ablations in Table 2 and Figure 6 are useful, but they do not resolve this confounding in the model-learning objective. If MC-based model learning is a central distinction from prior value-predictive approaches, its effect should be separated more clearly from the effects of the policy and planner.

Questions for authors
  1. What advantage does MC supervision provide over existing value-predictive learning approaches? Do you have results or analyses comparing MC and TD-based targets while keeping the architecture, flow policy, planner, and search budget fixed? A comparison on a small representative subset of datasets would help establish the incremental contribution beyond prior value-predictive methods.

  2. Does adding state matching still reduce performance when imagined-value supervision is preserved? Specifically, does the same trend hold when retaining Equation 6 and optimizing LVP+λLstate\mathcal L_{\mathrm{VP}}+\lambda\mathcal L_{\mathrm{state}}? Since Equation 14 also changes where value supervision is applied, this control would directly test the interpretation offered in Section 5.2.

7 comments
AuthorsOpenReview ↗

Author Response to Review dvkM (1/n)

We sincerely thank the reviewer for the valuable feedback and for acknowledging our work's broad scope of evaluation, substantial performance improvement, practical promise, and conceptual interestingness.

We clarify our contributions and provide more results to support our claims below. Revisions that directly aim to address the reviewer's concerns are highlighted in blue in the updated manuscript and mentioned below.

Weaknesses:

  1. The scope and significance of the contribution beyond existing value-predictive models remain insufficiently clear.

We want to specifically pin down the scope of our work as offline planning with reconstruction-free latent world models, which is an emerging area of research, and the closest work to ours is the application of JEPA-style latent world models, exemplified by the model-based planner, LeWM [1], we compare against in Figure 4.

We acknowledge that the value equivalence view is not totally new, and this is why we have discussed its extension to this work in Sec.4.1 (line 209-214) and compared and contrasted it with related work in Appendix A.2 (line 897-902). We consider it as the motivation and theoretical grounding for our work.

The difference is that previous work in this vein (e.g., Value Prediction Networks, MuZero) has mostly focused on online RL and learns value functions with TD bootstrapping, while we aim to solve reward-free offline RL via planning with latent world models. Offline goal-reaching makes the construction of temporal structure from dataset transitions for value learning actionable, naturally eliminating the need for TD bootstrapping and its disadvantage, like heuristics to address instability, the biased estimate, and the value overestimation in offline settings.

The extension of this view from online to offline, from dense reward to reward-free, from TD to closed-form Monte Carlo return for world modeling, has not currently gained much attention, especially in the context of offline goal-conditioned RL. The pervasive belief is that we still need to predict accurate enough next states to effectively plan offline. This work aims to challenge this belief and has shown that it is possible to achieve higher planning performance with less accurate next-state predictions. We believe this is a novel contribution, as it helps answer the open question of what makes for good representations for offline planning.

[1] Lucas Maes, et al. LeWorld-Model: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv, 2026.

AuthorsOpenReview ↗

Author Response to Review dvkM (2/n)

  1. The central ablation does not adequately isolate the causes of the performance differences.

We agree with this concern that we miss an intermediate step. We call LVP\mathcal{L} _ {\mathrm{VP}} in Equation 6 value-prediction loss, as the dynamics is trained to only predict the value, and LLP\mathcal{L}_{\mathrm{LP}} in Equation 13 (the original Equation 14) latent-prediction loss, as the dynamics is trained to only predict the encoded next states. The resulting representations are referred to as value-predictive and state-predictive representations, respectively.

To be more coherent with the experiments in the main text, we have removed the original phrasing of "This experiment changes one thing", "... adds the latent-matching term" in Sec 5.2 and changed it to "This experiment aims to verify which representations are more favorable for planning ... which matches the predicted latents to the encoded states." in line 410-414.

To isolate the reason of performance differences, we train a new variant on the 13 pixel-based datasets we report with LVP+λstateLstate−matching\mathcal{L}_{\mathrm{VP}}+ \lambda_{\mathrm{state}} \mathcal{L}_{\mathrm{state-matching}}, where Lstate−matching:=E[1H∑h=1H∥z^t+h−zt+h∥22]\mathcal{L} _ {\mathrm{state-matching}} \mathrel{\mathop:}= \mathbb{E}\left[ \frac{1}{H}\sum_{h=1}^{H}\|\hat z_{t+h}-z_{t+h}\|_2^2 \right].

It regularizes the imagined next states with value prediction, unlike LLP\mathcal{L}_{\mathrm{LP}} on the encoded next states. These three losses are summarized in the following table:

Lossw/o state matchingw/ imagined value supervisiondynamics outputprediction objective type
LVP\mathcal{L}_{\text{VP}} (Equation 6)✓✓valuevalue-predictive
LVP+λstateLstate-matching\mathcal{L} _ {\text{VP}}+ \lambda_{\text{state}} \mathcal{L}_{\text{state-matching}}✗✓value + next latent statestate-predictive
LLP\mathcal{L}_{\text{LP}} (Equation 13)✗✗next latent statestate-predictive

The results are presented in the following table, averaged over the mean success rate of all datasets in each environment. It shows that adding state matching still reduces performance when imagined-value supervision is preserved.

LossVisual AntMaze (4 datasets) (%)Visual Cube (4 datasets) (%)Visual Scene (1 dataset) (%)Visual Puzzle (4 datasets) (%)
LVP\mathcal{L}_{\text{VP}} (Equation 6)61.034.833.018.3
LVP+λstateLstate-matching\mathcal{L} _ {\text{VP}}+ \lambda_{\text{state}} \mathcal{L}_{\text{state-matching}}50.531.526.07.8
LLP\mathcal{L}_{\text{LP}} (Equation 13)45.80.030.00.0

This experiment has been included in the updated manuscript (Appendix D.2, including Table 10 for loss comparison, Figure 12 for success rate vs. next-latent prediction error similar to Figure 5, and Figure 13 for success rate trained with these three losses on pixel datasets) and is referred to in the main text in Sec 5.2, line 426.

This should provide stronger evidence to support our claim that predicting value can implicitly prevent representation collapse and is more favorable for planning than predicting states in Sec 5.2.

AuthorsOpenReview ↗

Author Response to Review dvkM (3/n)

Questions:

  1. a) What advantage does MC supervision provide over existing value-predictive learning approaches?

b) Do you have results or analyses comparing MC and TD-based targets while keeping the architecture, flow policy, planner, and search budget fixed?

a)

One advantage of closed-form Monte Carlo supervision over existing approaches in offline goal-reaching is that it removes the need for common heuristics in value learning, the additional target network and EMA. This is because the MC return target here is a fixed scalar in closed form given a state sts_t and a goal gg, i.e., R(st,g)R(s_t, g) in Eq.(3); value learning thus simplifies to a direct regression problem with one value network, which is more stable than online recursive TD and saves computational cost.

Another advantage is that the MC target is a more accurate estimate of the true value function than TD targets. While TD is known to be biased, the closed-form MC return here is both unbiased and zero-variance, and it can be obtained in zero cost with hindsight relabeling and sparse binary reward.

b)

As discussed in a), offline goal reaching allows us to bypass the disadvantages of online TD estimates; therefore, conceptually, it can be expected that closed-form MC supervision should outperform TD learning.

That said, we have trained a variant of VPFlow that learns value with TD and target networks with EMA, which uses the same architecture, flow policy, planner, and search budget, but replaces the MC supervision with TD supervision. Let z^t=Eϕ(st)\hat z_t = E_\phi (s_t) and z^t+h=fϕ(Eϕ(st),at:t+h−1)\hat z_{t+h} = f_{\phi}(E_\phi(s_t), a_{t:t+h-1}) for h>0h>0. The TD loss is defined as: LVPTD(ϕ)=ED,g[1H+1∑h=0HMt+h⊙(Vϕ(z^t+h,Eϕ(g))−yhTD)2]\mathcal{L}_{\mathrm{VP}}^{\mathrm{TD}}(\phi)=\mathbb{E}_{\mathcal{D}, g}\left[\frac{1}{H+1}\sum_{h=0}^HM_{t+h}\odot\left(V_\phi(\hat z_{t+h},E_\phi(g))-y_h^{\mathrm{TD}}\right)^2\right], where yhTD=sg[r(st+h,g)+γVϕˉ(Eϕˉ(st+h+1),Eϕˉ(g))]y_h^{\mathrm{TD}} = \mathrm{sg}\left[r(s_{t+h},g) + \gamma V_{\bar \phi}(E_{\bar \phi}(s_{t+h+1}), E_{\bar \phi}(g))\right], ϕˉ\bar \phi is the parameters of the target network and updated with EMA, and sg\mathrm{sg} denotes stop-gradient.

Empirically, on the representative datasets in the table below, MC supervision with LVP\mathcal{L}_{\mathrm{VP}} outperforms the success rate of TD.

Lossantmaze-large-navigate-v0 (%)cube-single-play-v0 (%)scene-play-v0 (%)puzzle-4x4-play-v0 (%)visual-antmaze-large-navigate-v0 (%)visual-cube-single-play-v0 (%)visual-scene-play-v0 (%)
LVP\mathcal{L}_{\text{VP}} (Equation 6)85987879848333
LVPTD\mathcal{L}_{\text{VP}}^{\text{TD}}74743650588020
  1. Does adding state matching still reduce performance when imagined-value supervision is preserved?

We have addressed this question in weakness 2 above. It still reduces performance (results in the second Table, and Figure 13), and the trend that value-predictive training with LVP\mathcal{L}_{\mathrm{VP}} is more effective, despite orders of magnitude larger next-latent prediction error, still holds (Figure 12).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Comments to Area Chair

Dear Area Chair,

Thank you for engaging in the review process, and we thank Reviewers Bupr, dvkM, and zcxG for their constructive feedback and follow-up discussion. All manuscript revisions directly addressing reviewer concerns are highlighted in blue in the updated PDF.

Strengths Highlighted by Reviewers

  • Conceptual insight & clean formulation (zcxG, dvkM, Bupr): Directly training latent world models with closed-form hindsight Monte Carlo returns avoids TD bootstrapping, target networks, and EMA, and empirically separates prediction fidelity from control utility.
  • Noise-space test-time planning with generative policies (Bupr, zcxG): Searching in the flow policy's noise space keeps actions near the offline data distribution and substantially outperforms action-space CEM and reactive policies.
  • Broad evaluation & strong gains (Bupr, dvkM, zcxG): Evaluated across 30 OGBench datasets (17 state, 13 pixel), delivering large improvements over flat baselines, especially on manipulation tasks, supported by a thorough appendix.

How We Addressed Reviewers' Concerns

ConcernWhat We Clarified, Added, & RevisedReviewer(s)Location
Isolating state matching vs. value predictionEvaluated the intermediate loss LVP+λstateLstate-matching\mathcal{L} _ {\text{VP}} + \lambda_{\text{state}}\mathcal{L} _ {\text{state-matching}} across all 13 pixel datasets (plus a λstate\lambda_{\text{state}} weight sweep in follow-up) and added collapse visualizations, confirming state matching still degrades planning even when imagined-value supervision is preserved. Rephrased Sec. 5.2 and softened the collapse heading to avoid overstatement.Bupr, dvkM, zcxGSec. 5.2, Appendix D.2, rebuttal
Scope & MC vs. TD supervisionScoped our claims to offline planning with reconstruction-free latent world models. Ran a controlled MC vs. TD (with EMA) comparison holding architecture, policy, planner, and budget fixed, confirming closed-form MC consistently outperforms TD across 7 representative datasets. Clarified why our offline setting differs from EfficientZero.dvkM, zcxG, BuprSec. 4.1, Appendix A.2, Appendix F.1, rebuttal
Optimizer framing (ε\varepsilon-ES vs. noise-space steering)Clarified our contribution around noise-space steering instantiated by ε\varepsilon-ES, clarified full per-dataset optimizer comparisons/intuition (locomotion vs. manipulation) and that using ε\varepsilon-ES across all 30 datasets still outperforms all flat baselines.zcxGAppendix B.6, Appendix D–E
Compute & wall-clock latencyAdded per-action wall-clock comparisons against model-free baselines and FLOP-matched comparisons against LeWM, showing lower wall-clock latency than LeWM and that scaling LeWM's search budget does not close the performance gap.BuprSec. 5.1, Appendix B.10
Baselines & implementation verificationAdded HIQL and GOPlan results/discussion and full implementation details for TD-MPC2, PLDM, and LeWM. Verified in follow-up that our adapted LeWM reproduces its published result on LeWM's own OGBench-Cube dataset, ruling out implementation issues.BuprAppendix B.8, Appendix E, rebuttal
Trajectory stitching & Figure 1 clarityAdded explicit discussion of MC trajectory-stitching trade-offs and the exclusion of stitch datasets; relabeled "Reward-Free" to "Reward Head" in Figure 1.BuprFigure 1, Sec. 4.2, Appendix B.1

We will open-source our code and model weights upon acceptance.

Best regards,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.