- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes VPFlow, an offline goal-conditioned reinforcement learning framework that challenges the assumption that effective planning requires accurate prediction of future states. Instead of training a latent dynamics model to reconstruct next-state representations, VPFlow learns value-predictive dynamics using closed-form Monte Carlo returns derived from hindsight goal relabeling. The resulting latent representations are optimized for goal-reaching value estimation rather than state reconstruction. The framework further incorporates a flow-matching policy as a behavioral prior and performs test-time planning by optimizing the policy's input noise instead of directly searching over action sequences. The paper develops an evolutionary noise-steering procedure based on GAE and uses receding-horizon execution. Experiments are conducted on 30 OGBench datasets covering state- and pixel-based navigation and manipulation tasks. VPFlow achieves strong overall performance compared with several model-free and latent model-based baselines. The representation ablations show that substantially reducing next-latent prediction error does not necessarily improve task success, while the planning ablations demonstrate the importance of the search space and test-time optimization.
Critical strengths
1.The most compelling contribution of this work lies in its distinction between predicting future states and predicting decision-relevant values; rather than assuming that an accurate latent simulator is a prerequisite for planning, VPFlow directly trains latent dynamics using goal-reaching value supervision. This perspective is supported by the experimental results as shown in Figure 5 and Figure 7. These findings provide a empirical argument that prediction fidelity and control utility should be evaluated as distinct metrics. 2.Using a flow policy as a behavioral prior enables searching for improved actions while retaining offline demonstration structure. Table 2 shows a substantial gap between action-space and noise-space CEM, highlighting the importance of search parameterization. The evaluation is broad, covering 17 state-based and 13 pixel-based OGBench datasets, with encouraging gains on challenging manipulation tasks. Appendix results and optimizer comparisons further strengthen the empirical presentation. Together, the conceptual motivation and demonstrated effectiveness of noise-space planning make the paper a worthwhile contribution.
Critical weaknesses
- Section 5.2’s comparison between the proposed value-predictive objective and a variant with an added next-latent matching loss is a useful controlled experiment, but it does not establish that state-predictive world models are generally inferior for planning. The observed performance drop may reflect an interaction between the representation-matching objective and the specific training architecture, and although the near-zero prediction error in Figure 5(c) is consistent with representation collapse, prediction error alone does not conclusively identify collapse as the underlying mechanism. The conclusion should distinguish the limitations of the tested state-matching objective from broader claims about state-predictive models.
- Although ε-ES is presented as a central component of VPFlow, Appendix D (Table 4) reveals that several task families actually use first-order noise-space optimizers, with state-based AntMaze employing ε-SGD and visual AntMaze using ε-Adam. This distinction is critical for interpreting Table 1: the reported performance validates the VPFlow framework with task-dependent noise-space optimization, rather than exclusively demonstrating the superiority of ε-ES.
Questions for authors
-
In Section 5.2, the state-predictive variant adds the latent-matching objective to the value-predictive training objective. Could the authors clarify whether their intended conclusion is that this particular additional objective is detrimental within VPFlow, rather than that state-predictive representations are generally unsuitable for planning? Relatedly, what evidence supports representation collapse as the explanation for the near-zero prediction error in Figure 5(c), beyond the prediction error itself?
-
Appendix D, Table 4 indicates that the reported results use ε-SGD or ε-Adam for some navigation tasks, while ε-ES is used for other task families. Could the authors clarify the relationship between these optimizer choices and the results reported under VPFlow in Table 1?Would it be more accurate to describe the main contribution as value-predictive planning with noise-space steering, with ε-ES being one effective optimizer rather than the optimizer used throughout?
2 comments
Author Response to Review zcxG (2/n)
Questions:
- Could the authors clarify whether their intended conclusion in Section 5.2 is that this particular additional objective is detrimental within VPFlow, rather than that state-predictive representations are generally unsuitable for planning?
Our conclusion from Section 5.2 is drawn from VPFlow itself and its ablations with different objective functions (see Appendix D.2). The selling point is to demonstrate that we can successfully achieve strong offline planning performance, and even better than with next-latent-state-prediction, therefore rendering state prediction as an unnecessary objective. This contrasts with the current belief and practices in latent world modeling. This phenomenon is grounded in the value-equivalence principle we discuss in Section 4.1, so we expect it to have some generality and practical implications.
However, we currently do not intend to overclaim the conclusion beyond the scope of our setting: offline planning with reconstruction-free latent world models, with the closest work as the JEPA-style latent planner we compare against, LeWM [1], which is an emerging area of research.
[1] Lucas Maes, et al. LeWorld-Model: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv, 2026.
- Could the authors clarify the relationship between the optimizer choices and the results reported under VPFlow in Table 1?
We have addressed this question in weakness 2 above. The intuition is that, for locomotion tasks, the dynamics is relatively smooth, making the first-order gradient sufficient and more informative than the zeroth-order planning. Conversely, for manipulation tasks, the dynamics may be highly non-convex and non-smooth, where the zeroth-order planner helps explore the search space and escape local optima.
- Would it be more accurate to describe the main contribution as value-predictive planning with noise-space steering, with ε-ES being one effective optimizer rather than the optimizer used throughout?
Yes. We have stated in line 125 of the summarized contributions that "We propose a test-time planning algorithm tailored to generative flow policies in their noise space ...", removing the narrative that may imply the superiority of -ES throughout all datasets.
Author Response to Review zcxG (1/n)
We sincerely thank the reviewer for the valuable feedback and for acknowledging our work on the compelling contribution with support and effectiveness demonstration with broad evaluation.
Revisions that directly aim to address the reviewer's concerns are highlighted in blue in the updated manuscript and mentioned below.
Weaknesses:
We acknowledge that the conclusion is tied to particularly our VPFlow architecture, but this conclusion should be viewed as constructive for a new perspective and promise rather than affirmative for all state-predictive models.
To provide stronger support for the representation collapse attribution behind zero success rate and that value prediction can implicitly prevent it, we have added a more rigorous ablation on the learning objective in Appendix D.2, including Figure 13 that shows collapse on 8 datasets in Visual Cube and Visual Puzzle, and Figure 14 that visually shows the degeneracy of representations to a single point.
We have also included a discussion on this issue in Appendix F.1, line 1921-1925.
Note: we have expanded the original Table 4 in Appendix D to Table 9 with per-dataset planning hyperparameters in the revision.
It is true that we changed the zeroth-order ε-ES described in the main text to the first-order ε-SGD/Adam on 12 locomotion tasks, because we empirically found out that they perform better.
If we keep ε-ES throughout all 30 datasets, the overall success rate in Table 1 will slightly decrease from 63.2 to 59.6 for state-based tasks and from 37.6 to 34.2 for pixel-based tasks. VPFlow still outperforms all other baselines and does not affect our core claim, but the gain is smaller.
This is a by-product finding during our experiment. Therefore, we have presented the complete per-dataset success rate with either the zeroth- or first-order planner in Table 12 as justification for the planner choice and discussed it in Appendix B.6.