- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Value-Aware Latent-Space Planning (VALP), a novel hierarchical RL framework for long-horizon decision-making. The core idea of VALP is to guide short-horizon planning toward long-horizon objectives by combining an online high-level policy selecting intermediate goals with a world-model-based control policy at the low level. To do so, VALP learns compact value-aware goal representations, which are then pursued by repeated model-predictive planning using the learned world model. Experiments against state of the art HRL baselines and further analyses support the effectiveness of VALP.
Critical strengths
Using an HRL policy to set intermediate goals for the low-level model-predictive planner to address limitations of world-model-based control is an interesting research direction. The ablations conducted in Section 4.4 further validate the design choices underlying VALP: using dynamics-based objectives instead of value-aware goals or replacing the model-predictive controller with a goal-conditioned policy strongly degrades performance.
Critical weaknesses
VALP introduces additional hyperparameters and computational burden, as it involves several components (e.g., goal representations, iCEM, or SAC). Finding robust hyperparameter configurations for the task at hand may not be easy, and the authors should clarify the procedure used to extract them more explicitly.
In this regard, while the analysis in Section 4.4 shows that VALP is robust to the subgoal dimension, it is not clear how the "goal-representation bound" has been chosen and how this choice can affect performance. Specifically, one could expect that this value is strictly linked to the other goal-specific hyperparameters (i.e., horizon and sub-goal dimension), and that they all may depend on the temporal scale of the problem. Further investigating this point would strengthen the robustness of the value-aware goal mechanism.
Questions for authors
-
How do the goal-specific values (i.e., horizon, bound, and sub-goal dimension) relate to the task at hand? Moreover, how has the "goal-representation bound" been chosen, and how does it affect performance?
-
Is it possible to provide a more detailed analysis of the learned latent goal space in relation to the underlying problem? For example, a visualization of the latent space using low-dimensional embeddings (e.g., via t-SNE) could further improve the significance of VALP.
-
How does VALP compare with learned-goal HRL baselines such as HLPS and LESSON in terms of simulation time? Table 5 reports the comparison with the other baselines only.
4 comments
Q2. Low-dimensional Embeddings. We have added a PCA visualization of the learned goal space across five training checkpoints, using 100 evaluation episodes per checkpoint on AntMaze. The visualization shows that initially diffuse representations gradually develop a coherent structure aligned with geodesic distance to the target, reflecting the underlying maze topology.
To further quantify this structure, we analyze the sensitivity of the learned goal representation to perturbations in different state variables. We independently inject Gaussian noise into each state-variable group and measure the resulting displacement in the PCA-projected goal space. The table below reports these displacements for a noise scale of (\sigma=1), alongside the displacement between consecutive environment steps.
| Training steps | Position | Height + orient. | Joint angles | Torso vel. | Joint vel. | Consecutive steps |
|---|---|---|---|---|---|---|
| 0.1M | 0.28 | 0.38 | 0.69 | 0.17 | 0.63 | 1.00 |
| 0.25M | 1.91 | 0.25 | 0.47 | 0.10 | 0.37 | 0.50 |
| 0.5M | 1.91 | 0.23 | 0.46 | 0.08 | 0.29 | 0.51 |
| 1M | 2.09 | 0.11 | 0.42 | 0.10 | 0.28 | 0.38 |
| 2M | 1.87 | 0.09 | 0.34 | 0.10 | 0.22 | 0.40 |
The results show a clear shift toward task-relevant information throughout training. At 0.1M steps, the representation is more sensitive to joint angles and velocities than to position. From 0.25M steps onward, position becomes the dominant factor, inducing displacements of approximately 2 PCA units. By 2M steps, position perturbations induce a displacement of 1.87, more than five times larger than any other state-variable group (at most 0.34). Meanwhile, sensitivity to joint configurations and velocities progressively decreases.
Together, the PCA visualization and perturbation analysis provide complementary evidence that VALP learns a goal space that captures the task-relevant structure of the environment, providing meaningful objectives for short-horizon world-model planning. We have incorporated the visualization, quantitative perturbation analysis, and corresponding discussion into Section 4.4 (Figure 6).
Q3. Simulation-Time Comparison with HLPS and LESSON. We recognize the importance of comparing the computational costs of VALP with those of HLPS and LESSON. Table 6 reports simulation times only for JAX implementations to ensure a consistent runtime comparison, whereas our main experiments with HLPS and LESSON use the authors' original PyTorch implementations.
For completeness, we provide additional timing results using our JAX implementations of HLPS and LESSON below. Since their fidelity to the original implementations has not been fully verified, we report these results for reference only and exclude them from the manuscript to avoid potentially misleading comparisons.
The table below reports action-selection and amortized update time in milliseconds per environment step, reported as mean ± standard error over ten 10-step window averages from one run.
| Method | Action (ms/step) | Update (ms/step) |
|---|---|---|
| SAC | 0.133 ± 0.001 | 0.140 ± 0.001 |
| TD-MPC2 | 2.015 ± 0.002 | 0.314 ± 0.001 |
| HIRO | 0.200 ± 0.002 | 1.641 ± 0.008 |
| LESSON | 0.199 ± 0.002 | 1.972 ± 0.017 |
| HLPS | 0.275 ± 0.002 | 1.919 ± 0.026 |
| VALP | 1.936 ± 0.008 | 0.377 ± 0.002 |
The results highlight the computational trade-off between model-based planning and learned low-level policies. VALP requires approximately 7–10 times more action-selection time than HLPS and LESSON, reflecting the overhead of repeated trajectory optimization. However, its amortized update time is approximately 5 times lower than both methods, consistent with avoiding the training of a separate goal-conditioned low-level policy. VALP's action-selection time is also comparable to TD-MPC2 (1.936 vs. 2.015 ms/step). Additionally, a promising direction for reducing action-selection time is to distill the planner's optimized action sequences into a learned policy.
Thank you for the detailed rebuttal.
The new analyses and clarifications on the goal-specific parameters and the visualization of the learned goal space further strengthen the significance of VALP. I also appreciate the effort to compare the simulation times with LESSON and HLPS by implementing them in JAX. Although these implementations have not been fully verified, I would nevertheless encourage you to include the comparison in the Appendix, as it provides useful insights into the computational trade-offs between model-based planning and low-level policies.
Overall, the rebuttal addresses my main concerns and, accordingly, I increase my score from 3 to 4 (clear accept).
Thank you for your positive feedback and for reconsidering your score. We appreciate your suggestion and have now included the computational comparison with LESSON and HLPS in Appendix J. We note that the reported runtimes for these methods were measured using our JAX implementations.
We thank the reviewer for their positive assessment and constructive feedback. In response to their concerns, we have clarified our hyperparameter selection procedure, conducted additional sensitivity experiments on the goal-representation bound, and provided qualitative and quantitative analyses of the learned latent space. We also present a runtime comparison with learned-goal HRL baselines. We address each point in detail below.
W1 / Q1. Hyperparameter Selection and Robustness. Although VALP combines several components, it does not require extensive environment-specific tuning. For the high-level learner, we adopt standard SAC settings, while the iCEM planner largely follows the hyperparameters recommended in the original paper, with a fixed planning budget. Regarding goal-specific hyperparameters, the horizon H=10 determines both the high-level decision interval and the maximum planning horizon, while the subgoal dimension dg=30 controls the capacity of the learned goal representation. These settings are shared across all Ant benchmark environments rather than tuned individually. Moreover, the sensitivity analysis in Section 4.4, summarized in the table below, demonstrates strong performance across short planning horizons and a broad range of subgoal dimensions. We have clarified the hyperparameter selection procedure and implementation details in the Appendix.
W2 / Q1. Goal-Representation Bound. We thank the reviewer for raising this question. The goal-representation bound c controls the numerical range of the learned latent goals, rather than their temporal horizon or dimensionality. We choose c=3 to limit extreme values while approximately preserving the geometry of standardized features. Since the high-level policy outputs normalized goal displacements, the bound also determines their relative scale within the representation space and may therefore interact with the other goal-specific hyperparameters.
Following the reviewer's suggestion, we additionally evaluate (c∈2,3,4). The results are summarized below.
VALP achieves perfect success under both evaluation protocols for c=3 and c=4, while maintaining near-perfect performance for c=2. These results indicate that VALP is robust to the choice of the goal-representation bound within the evaluated range, with only minor performance differences across settings.
We have added the motivation for the default bound and the corresponding sensitivity analysis to Appendix C (Figure 9) of the revised manuscript.