acceptodds
Under review as a conference paper at ICLR 2027

Planning with Value-Aware Learned Goals for Hierarchical Reinforcement Learning

Abstract

Hierarchical reinforcement learning (HRL) decomposes long-horizon decision making across temporal scales, with a high-level policy periodically specifying goals and a low-level controller realizing them through primitive actions. Departing from the conventional use of learned goal-conditioned policies for low-level control, we introduce Value-Aware Latent-Space Planning (VALP), a novel low-level policy-free framework that combines reinforcement learning for strategic decision making with world model-based planning for closed-loop control. VALP learns a compact, value-aware latent space that preserves task-relevant information, in which a high-level policy specifies temporally extended abstract goals, while a model-predictive planner directly synthesizes primitive actions by planning through a learned latent world model toward these objectives. By coupling value-based representation learning and model-based planning, VALP separates learning task-relevant objectives for long-horizon decision making from their realization through short-horizon control. VALP significantly outperforms state-of-the-art HRL methods on long-horizon control tasks. Our findings highlight the importance of a task-relevant communication interface for hierarchical control and demonstrate how reinforcement learning for long-horizon decision making can be effectively combined with world model-based control for goal realization.

Primary area:
reinforcement learning
Keywords:
Hierarchical decision making, reinforcement learning, world model-based control

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ZT6hOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Value-Aware Latent-Space Planning (VALP), a novel hierarchical RL framework for long-horizon decision-making. The core idea of VALP is to guide short-horizon planning toward long-horizon objectives by combining an online high-level policy selecting intermediate goals with a world-model-based control policy at the low level. To do so, VALP learns compact value-aware goal representations, which are then pursued by repeated model-predictive planning using the learned world model. Experiments against state of the art HRL baselines and further analyses support the effectiveness of VALP.

Critical strengths

Using an HRL policy to set intermediate goals for the low-level model-predictive planner to address limitations of world-model-based control is an interesting research direction. The ablations conducted in Section 4.4 further validate the design choices underlying VALP: using dynamics-based objectives instead of value-aware goals or replacing the model-predictive controller with a goal-conditioned policy strongly degrades performance.

Critical weaknesses

VALP introduces additional hyperparameters and computational burden, as it involves several components (e.g., goal representations, iCEM, or SAC). Finding robust hyperparameter configurations for the task at hand may not be easy, and the authors should clarify the procedure used to extract them more explicitly.

In this regard, while the analysis in Section 4.4 shows that VALP is robust to the subgoal dimension, it is not clear how the "goal-representation bound" has been chosen and how this choice can affect performance. Specifically, one could expect that this value is strictly linked to the other goal-specific hyperparameters (i.e., horizon and sub-goal dimension), and that they all may depend on the temporal scale of the problem. Further investigating this point would strengthen the robustness of the value-aware goal mechanism.

Questions for authors
  1. How do the goal-specific values (i.e., horizon, bound, and sub-goal dimension) relate to the task at hand? Moreover, how has the "goal-representation bound" been chosen, and how does it affect performance?

  2. Is it possible to provide a more detailed analysis of the learned latent goal space in relation to the underlying problem? For example, a visualization of the latent space using low-dimensional embeddings (e.g., via t-SNE) could further improve the significance of VALP.

  3. How does VALP compare with learned-goal HRL baselines such as HLPS and LESSON in terms of simulation time? Table 5 reports the comparison with the other baselines only.

4 comments

We thank the reviewer for their positive assessment and constructive feedback. In response to their concerns, we have clarified our hyperparameter selection procedure, conducted additional sensitivity experiments on the goal-representation bound, and provided qualitative and quantitative analyses of the learned latent space. We also present a runtime comparison with learned-goal HRL baselines. We address each point in detail below.

W1 / Q1. Hyperparameter Selection and Robustness. Although VALP combines several components, it does not require extensive environment-specific tuning. For the high-level learner, we adopt standard SAC settings, while the iCEM planner largely follows the hyperparameters recommended in the original paper, with a fixed planning budget. Regarding goal-specific hyperparameters, the horizon H=10H=10 determines both the high-level decision interval and the maximum planning horizon, while the subgoal dimension dg=30d_g=30 controls the capacity of the learned goal representation. These settings are shared across all Ant benchmark environments rather than tuned individually. Moreover, the sensitivity analysis in Section 4.4, summarized in the table below, demonstrates strong performance across short planning horizons and a broad range of subgoal dimensions. We have clarified the hyperparameter selection procedure and implementation details in the Appendix.

HorizonSuccess Rate (%)Subgoal DimensionSuccess Rate (%)
H=5H = 599±099 \pm 0dg=10d_g = 10100±0100 \pm 0
H=10H = 10100±0100 \pm 0dg=20d_g = 20100±0100 \pm 0
H=15H = 1596±296 \pm 2dg=30d_g = 30100±0100 \pm 0
H=20H = 2088±288 \pm 2dg=40d_g = 40100±0100 \pm 0
H=25H = 2565±665 \pm 6dg=50d_g = 50100±0100 \pm 0

W2 / Q1. Goal-Representation Bound. We thank the reviewer for raising this question. The goal-representation bound cc controls the numerical range of the learned latent goals, rather than their temporal horizon or dimensionality. We choose c=3c=3 to limit extreme values while approximately preserving the geometry of standardized features. Since the high-level policy outputs normalized goal displacements, the bound also determines their relative scale within the representation space and may therefore interact with the other goal-specific hyperparameters.

Following the reviewer's suggestion, we additionally evaluate (c∈2,3,4)(c \in {2,3,4}). The results are summarized below.

BoundRandom GoalFixed Goal
c=298.8 ± 0.899.6 ± 0.4
c=3 (default)100 ± 0.0100 ± 0.0
c=4100 ± 0.0100 ± 0.0

VALP achieves perfect success under both evaluation protocols for c=3c=3 and c=4c=4, while maintaining near-perfect performance for c=2c=2. These results indicate that VALP is robust to the choice of the goal-representation bound within the evaluated range, with only minor performance differences across settings.

We have added the motivation for the default bound and the corresponding sensitivity analysis to Appendix C (Figure 9) of the revised manuscript.

Q2. Low-dimensional Embeddings. We have added a PCA visualization of the learned goal space across five training checkpoints, using 100 evaluation episodes per checkpoint on AntMaze. The visualization shows that initially diffuse representations gradually develop a coherent structure aligned with geodesic distance to the target, reflecting the underlying maze topology.

To further quantify this structure, we analyze the sensitivity of the learned goal representation to perturbations in different state variables. We independently inject Gaussian noise into each state-variable group and measure the resulting displacement in the PCA-projected goal space. The table below reports these displacements for a noise scale of (\sigma=1), alongside the displacement between consecutive environment steps.

Training stepsPositionHeight + orient.Joint anglesTorso vel.Joint vel.Consecutive steps
0.1M0.280.380.690.170.631.00
0.25M1.910.250.470.100.370.50
0.5M1.910.230.460.080.290.51
1M2.090.110.420.100.280.38
2M1.870.090.340.100.220.40

The results show a clear shift toward task-relevant information throughout training. At 0.1M steps, the representation is more sensitive to joint angles and velocities than to position. From 0.25M steps onward, position becomes the dominant factor, inducing displacements of approximately 2 PCA units. By 2M steps, position perturbations induce a displacement of 1.87, more than five times larger than any other state-variable group (at most 0.34). Meanwhile, sensitivity to joint configurations and velocities progressively decreases.

Together, the PCA visualization and perturbation analysis provide complementary evidence that VALP learns a goal space that captures the task-relevant structure of the environment, providing meaningful objectives for short-horizon world-model planning. We have incorporated the visualization, quantitative perturbation analysis, and corresponding discussion into Section 4.4 (Figure 6).

Q3. Simulation-Time Comparison with HLPS and LESSON. We recognize the importance of comparing the computational costs of VALP with those of HLPS and LESSON. Table 6 reports simulation times only for JAX implementations to ensure a consistent runtime comparison, whereas our main experiments with HLPS and LESSON use the authors' original PyTorch implementations.

For completeness, we provide additional timing results using our JAX implementations of HLPS and LESSON below. Since their fidelity to the original implementations has not been fully verified, we report these results for reference only and exclude them from the manuscript to avoid potentially misleading comparisons.

The table below reports action-selection and amortized update time in milliseconds per environment step, reported as mean ± standard error over ten 10-step window averages from one run.

MethodAction (ms/step)Update (ms/step)
SAC0.133 ± 0.0010.140 ± 0.001
TD-MPC22.015 ± 0.0020.314 ± 0.001
HIRO0.200 ± 0.0021.641 ± 0.008
LESSON0.199 ± 0.0021.972 ± 0.017
HLPS0.275 ± 0.0021.919 ± 0.026
VALP1.936 ± 0.0080.377 ± 0.002

The results highlight the computational trade-off between model-based planning and learned low-level policies. VALP requires approximately 7–10 times more action-selection time than HLPS and LESSON, reflecting the overhead of repeated trajectory optimization. However, its amortized update time is approximately 5 times lower than both methods, consistent with avoiding the training of a separate goal-conditioned low-level policy. VALP's action-selection time is also comparable to TD-MPC2 (1.936 vs. 2.015 ms/step). Additionally, a promising direction for reducing action-selection time is to distill the planner's optimized action sequences into a learned policy.

Thank you for the detailed rebuttal.

The new analyses and clarifications on the goal-specific parameters and the visualization of the learned goal space further strengthen the significance of VALP. I also appreciate the effort to compare the simulation times with LESSON and HLPS by implementing them in JAX. Although these implementations have not been fully verified, I would nevertheless encourage you to include the comparison in the Appendix, as it provides useful insights into the computational trade-offs between model-based planning and low-level policies.

Overall, the rebuttal addresses my main concerns and, accordingly, I increase my score from 3 to 4 (clear accept).

Thank you for your positive feedback and for reconsidering your score. We appreciate your suggestion and have now included the computational comparison with LESSON and HLPS in Appendix J. We note that the reported runtimes for these methods were measured using our JAX implementations.

Reviewer_JKkdOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

VALP combines a high-level policy operating in a learned goal representation with low-level model-predictive control through learned latent dynamics. Goal representations are trained by distilling a high-level critic into a compressed value predictor. The framework is well motivated and supported by substantial main-text experiments. My principal concerns are the actor-gradient notation and the interpretation of evidence concerning goal geometry and representation transfer.

Critical strengths
  1. The paper offers a coherent rationale for separating goal learning from goal realization as representations evolve.

  2. The main experiments cover four Ant environments, multiple reward and observation settings, enlarged tasks, and five independent runs.

  3. Fixed-target results show a substantial average advantage: 78.2% for VALP versus 53.3% for VALP-XY and lower reported averages for other baselines.

  4. VALP-XY, VALP-DR, and VALP-RL examine meaningful design choices.

  5. The theorem’s limitations are explicitly stated: value preservation does not imply reachability or controllability.

  6. The appendix adds detailed results, implementation information, and computational measurements that strengthen the empirical presentation.

Critical weaknesses
  1. The actor objective’s stop-gradient requires clarification. Eq 3 places stop-gradient around the complete output. Under the usual reparameterized SAC update, this removes the Q-gradient through the sampled action. Freezing critic parameters is different from detaching its output. This may be a notation issue, but the implemented gradient path must be explained; the equation alone does not justify asserting an implementation error.

  2. Value preservation does not fully justify Euclidean goal tracking. The planner minimizes terminal Euclidean distance in goal space, while Proposition 3.1 concerns differences in value predictions. The paper correctly acknowledges that this does not establish controllability. The remaining empirical question is whether the learned geometry reliably supports achievable and useful intermediate goals.

  3. The transfer experiment does not isolate representation quality. VALP retains its model-based controller, while comparison methods retain learned low-level policies. The authors acknowledge this distinction. The results support transfer of the combined interface and controller, but provide weaker evidence that the representation itself is the source of the advantage.

  4. Aggregate comparisons cover different task subsets. Table 1 excludes image settings for HIRO and VALP-XY but includes them for other methods. The accompanying note makes this transparent, but common-subset averages would make overall comparisons easier to interpret.

Questions for authors
  1. What fraction of selected goals can the low-level planner achieve within a segment?

  2. Does the transfer advantage remain when methods use a common low-level realization mechanism?

  3. Can the authors report aggregate comparisons over the same task subsets?

4 comments

We thank the reviewer for their positive assessment and constructive feedback. In response to their concerns, we have corrected the actor loss equation, conducted an additional analysis of latent goal reachability, and extended our transfer experiments to isolate the contribution of value-aware representation learning. We have also revised our aggregate comparisons to ensure consistent evaluation across task subsets. We address each point in detail below.

W1. Typo in Actor Loss. We apologize for the oversight. We use the standard SAC actor loss, which does not apply stop-gradient to the Q-function. We have corrected the corresponding equation in the manuscript.

W2 & Q1. Latent Goal Reachability. We thank the reviewer for raising this concern. We agree that value preservation does not theoretically guarantee controllability. To empirically assess whether the learned goal space supports effective control, we measure the fraction of high-level segments that end closer to their selected latent subgoals in Euclidean distance. We evaluate 100 episodes per checkpoint on fixed-goal AntMaze, with each segment consisting of 10 primitive steps.

Training stepsSegments ending closer
0.1M42.1%
0.25M61.6%
0.5M80.8%
1M95.0%
1.5M98.7%
2M99.9%
3M99.7%
4M99.8%
5M100.0%

The fraction increases from 42.1% at 0.1M to 95.0% at 1M, remaining above 98% thereafter. Importantly, progress is measured directly in the learned value-aware goal space rather than predefined coordinates. These results suggest that, as training progresses, the high-level policy and world-model planner learn to select and track actionable latent objectives. While this does not guarantee exact subgoal attainment, the consistently high progress rates provide empirical evidence that the learned geometry supports effective short-horizon control.

We have added a new section, Latent Goal Reachability (Appendix H), to the appendix, detailing the evaluation procedure and presenting these results.

W3 / Q2. Representation Transfer. To better isolate the contribution of representation quality, we extend our transfer experiments with VALP-DR, which uses dynamics-based goal learning while retaining the same world-model-based controller as VALP. This allows us to assess the effect of value-aware goal learning under a common low-level realization mechanism.

AlgorithmAntMazeAntFourRooms
LESSON62.0 ± 5.4 %26.7 ± 5.5 %
HLPS48.0 ± 4.9 %36.0 ± 4.3 %
VALP-DR39.3 ± 4.3 %20.0 ± 2.7 %
VALP80.0 ± 4.1 %70.7 ± 4.4 %

VALP achieves 80.0% success on AntMaze and 70.7% on AntFourRooms, compared to 39.3% and 20.0% for VALP-DR, corresponding to improvements of 40.7 and 50.7 percentage points, respectively. VALP also outperforms the strongest policy-based baseline by 18.0 percentage points on AntMaze (LESSON) and 34.7 percentage points on AntFourRooms (HLPS).

Since VALP and VALP-DR share the same low-level realization mechanism and differ in their goal-learning objectives, these results provide stronger evidence that value-aware representation learning contributes substantially to transfer performance, rather than the observed advantage arising solely from model-based control. We have incorporated these results into the revised manuscript in Fig. 4.

W4 / Q3. Consistent Aggregate Comparisons. We agree that aggregate comparisons should be made over consistent task subsets. To address this, we have revised Table 1 to report results aggregated by evaluation setting (Dense, Sparse, Image, and Large), rather than by environment. This enables direct comparisons across methods within each setting, with each entry averaged over the same four environments.

SettingSACTD-MPC2HIROLESSONHLPSVALP-XYVALP
Dense24.6 ± 14.549.3 ± 21.546.7 ± 14.633.7 ± 21.746.7 ± 18.068.6 ± 12.077.7 ± 16.2
Sparse0.0 ± 0.031.2 ± 23.428.2 ± 16.328.2 ± 17.451.1 ± 12.125.3 ± 10.462.6 ± 21.3
Image45.2 ± 20.642.5 ± 19.9--11.2 ± 6.515.1 ± 12.5--86.3 ± 6.1
Large34.1 ± 19.71.0 ± 1.062.2 ± 19.211.7 ± 5.929.5 ± 16.565.9 ± 14.286.4 ± 6.5
Average26.0 ± 9.631.0 ± 10.745.7 ± 9.821.2 ± 5.735.6 ± 8.353.3 ± 14.078.2 ± 5.6

VALP achieves the highest average success rate across all four evaluation settings. Compared to the strongest baseline in each setting, VALP improves success rates by 9.1 percentage points on Dense, 11.5 on Sparse, 41.1 on Image, and 20.5 on Large. The advantages are particularly pronounced under image-based observations and increased task scale, where VALP achieves 86.3% and 86.4% success, respectively. These results demonstrate that VALP's performance advantage is consistent across evaluation settings when compared over identical environment subsets.

The revised aggregate results are presented in Table 1, with individual environment-level results provided in Appendix Table 3.

Thank you for your detailed reply and additional experiments. The responses address most of my concerns. The actor loss issue is now clear, and the new VALP-DR results address my main concern about the transfer experiment. The reachability results are also useful, though ending closer to a goal is not quite the same as reaching it.

Overall, I increase my rating

We thank the reviewer for their thoughtful engagement and positive reassessment. Their suggestions have substantially strengthened the positioning of our work.

Reviewer_1kgTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VALP combines a high-level (SAC) policy that selects learned latent subgoals with a low-level model-predictive controller that realizes them. It learns the subgoal representation from the high-level critic values, then use planning to accommodate the evolving representation. It improves success rates on Ant navigation and manipulation benchmarks, particularly for challenging distant targets.

Critical strengths
  1. The overall presentation is clear and easy to follow. The background section is succinct but mostly accurate and contains useful information. The notations the author used is mostly consistent throughout the paper.
  2. the experiment setting and hyperparameter settings look somewhat complete and possible to reproduce. The appendix provide many details in settings that are helpful.
  3. The authors acknowledge that value preservation doesn't imply reachability--I like that.
Critical weaknesses
  1. In Eq.3 if you mean standard SAC, then you don't want to block the critic's gradient from reaching the actor. Of course you can detach the encoder, but sg the whole thing will make the actor only maximizing entropy! Correct me if I'm wrong.
  2. I think the citation and discussion on many aspects are insufficient. E.g., the discussion on value-preserving abstraction/planning is somewhat missing. In the MBRL context, the most related work I can think of is by Grimm et al. (2020), The Value Equivalence Principle for Model-Based Reinforcement Learning (I believe there are many more). Earlier work sets up principled results in state abstraction that preserve value function, e.g., Li et al. (2006), Towards a Unified Theory of State Abstraction for MDPs (and many more, such as works on bisimulation and the metrics and their recent extension to representation learning). For HRL and goal-conditioned RL, there is also a much wider spectrum than what has been discussed, and HRL isn't the same thing as GCRL. A closely related work on latent goal space is by Vezhnevets et al. (2017), FeUdal Networks for Hierarchical Reinforcement Learning. Work by Klissarov et al. (2025), Discovering temporal structure: An overview of hierarchical reinforcement learning might give a comprehensive overlook on HRL, including its relationship to GCRL and state abstraction, and the nonstationarity you mentioned.
  3. Some presentation issues exist, e.g., inconsistent App./Appendix, eq./equation. I'll not expand on this due to the succinctness requirement, but please be consistent with your choice of words.
  4. I think one important thing needs to be clarified more: the learned goals, in principle, may not be a goal/subgoal in the environment (e.g., a position or state). Sometimes they may align, but not in general. In some sense, these are not "environmental goals" but state abstractions driven by value equivalence.
  5. Experiment: a missing (ablation) comparison could be MPC using the original z-space. TD-MPC2 isn't the same as your z-space, which isn't shaped by reward prediction as TD-MPC2 does (and of course, TD-MPC2 uses a different planning objective). Another immediate variant is to train the goal encoder with Q loss directly through TD-learning (say, an end-to-end but compressed critic) or a reward prediction objective rather than the categorical distillation. In general, it's not sufficient to understand how ψ\psi is useful as an abstraction. There is little other evidence that the abstraction is learned--whether the representation remains one-to-one or retains irrelevant information remains unknown. One possibly interesting diagnostic environment is to add task-irrelevant noise and see how the representation looks like.
  6. Settings: The task variation is cool. However, “image-based” observations consist of a small (5, 5, 3) map of local view combined with proprioception. They provide little evidence about learning useful goal spaces from realistic visual inputs. Only one additional domain aside from commonly used Antmaze-like Mujoco envs, which also raises questions about how general the method could be. And how well does the method work on a stochastic env? The transfer environment is also limited to changing only the reset position while retaining environment geometry and dynamics.
Questions for authors
  1. Eq.3 do you stop-gradient the Q entirely?
  2. Can you comment more on how your work related to the key related works abovementioned, and make your contribution more clear?
  3. How well does it perform in "non-hierarchical" environments? What are the main limitations of the work? And do you observe any failure in learning dynamics (like, issues when co-learning the Q and the representation)?
  4. Any evidence that the goal encoder itself is particularly useful (compared to the alternatives, ref. to weakness 5)?
8 comments

We thank the reviewer for their constructive feedback and valuable suggestions. In response, we have strengthened the Introduction, expanded the related work discussion, corrected the actor loss equation, and clarified the interpretation of learned latent goals. We have also conducted additional ablations on the goal representation and value-distillation objective, introduced a diagnostic analysis of task-irrelevant information, and extended our evaluation to PushT with state-based and image-based observations, including pretrained world models. We address each point in detail below.

W1 / Q1. Typo in Actor Loss. We apologize for the oversight. We use the standard SAC actor loss, which does not apply stop-gradient to the Q-function. We have corrected the corresponding equation in the manuscript.

W2 / Q2. Related Work and Positioning. We thank the reviewer for these valuable suggestions. We agree that the recommended works provide important context for positioning our contribution. Accordingly, we have strengthened the Introduction (Section 1) and substantially expanded the Related Work discussion (Appendix A) to better acknowledge these connections and clarify the contributions of VALP.

These revisions clarify VALP's positioning relative to prior work: rather than learning value-preserving state abstractions or value-equivalent world models directly, VALP uses high-level value information to shape a compact latent goal space shared by strategic goal selection and model-predictive control. This enables short-horizon world-model planning to be guided by objectives relevant to long-horizon task return. We believe these additions provide a more comprehensive discussion of the relevant literature and more clearly distinguish our contribution from existing approaches.

W3. Notation Consistency. We acknowledge the presentation inconsistencies and have revised the manuscript to ensure consistent terminology and formatting throughout, including references to appendices and equations.

W4. Interpretation of Learned Goals. We clarify that the learned goals in VALP do not necessarily correspond to physical states or positions in the environment. Instead, they represent abstract objectives in a value-aware latent space, where states with similar high-level action-value profiles may have nearby representations. We have added this clarification to Section 3 of the manuscript.

W5 / Q4. Benefit of the Goal Encoder. We thank the reviewer for suggesting this comparison. To directly assess the contribution of the learned goal encoder, we introduce LatentMPC, which performs planning in the original world-model latent space (z), without a separate value-aware goal representation. We evaluate both methods under fixed-target evaluation across all four environments.

EnvironmentLatentMPCVALP
AntMaze96.0 ± 3.0100.0 ± 0.0
AntFourRooms31.6 ± 9.299.6 ± 0.4
AntPush0.8 ± 0.831.2 ± 19.1
AntFall29.2 ± 12.780.0 ± 3.9
Average39.477.7

VALP outperforms LatentMPC across all four environments, achieving an average success rate of 77.7% compared to 39.4%, an improvement of 38.3 percentage points. The advantage is particularly pronounced on AntFourRooms and AntFall, where VALP improves success rates by 68.0 and 50.8 percentage points, respectively. On AntMaze, both methods achieve near-perfect final performance, although VALP converges substantially faster.

These results suggest that the latent state representation used for dynamics prediction does not necessarily provide an effective objective space for long-horizon control. Learning a dedicated value-aware goal representation provides more informative intermediate objectives for short-horizon world-model planning. We have added this ablation and the corresponding discussion to Appendix G (Figure 15) of the revised manuscript.

W5. Benefit of Value Distillation. Training the compressed critic directly through TD learning is indeed a natural alternative within our value-aware framework, as both approaches aim to preserve long-horizon value information in the learned goal representation. We consider this an alternative design choice within our methodology rather than a fundamentally different approach. We choose to distill predictions from a high-level critic trained on the full latent state, providing value supervision to the compressed goal space without directly bootstrapping through the compressed critic. In our earlier experiments, we observed that learning the critic directly in a compressed representation can be sensitive to the choice of subgoal dimensionality. We attribute this brittleness to the information bottleneck imposed by compression, which can limit the critic's ability to accurately estimate task values. In contrast, our approach retains the critic in the original latent space, avoiding this restriction, while distilling its value information into the compact goal representation through an auxiliary imitator critic.

Following the reviewer's suggestion, we additionally evaluate the reward prediction alternative, denoted VALP-RW, which replaces value distillation with prediction of cumulative segment rewards while retaining the remaining VALP architecture. The results under fixed-target evaluation are summarized below.

EnvironmentVALP-RWVALP
AntMaze42.4 ± 8.3100.0 ± 0.0
AntFourRooms6.8 ± 4.299.6 ± 0.4
AntPush0.0 ± 0.031.2 ± 19.1
AntFall64.8 ± 5.980.0 ± 3.9
Average28.577.7

VALP outperforms VALP-RW across all four environments, achieving an average success rate of 77.7% compared to 28.5%, an improvement of 49.2 percentage points. The advantage is particularly pronounced on AntFourRooms and AntMaze, where VALP improves success rates by 92.8 and 57.6 percentage points, respectively. These results suggest that preserving long-horizon value information, rather than predicting cumulative segment rewards alone, provides more effective latent objectives for world-model-based planning.

We have incorporated this ablation and the corresponding discussion into Appendix G (Figure 15) of the revised manuscript.

W5. Diagnostic with task-irrelevant noise. We thank the reviewer for this suggestion and have added the diagnostic to Section 4.4 (Figure 6). In AntMaze, task progress depends mostly on position, while the ant's body configuration and velocities are not as relevant. We add noise to one group of state variables at a time, scaled by σ\sigma times each variable's standard deviation, and measure how far the goal representation moves in the PCA projection.

PositionHeight + orient.Joint anglesTorso vel.Joint vel.Consecutive steps
1.870.090.340.100.220.40

Median PCA displacement at σ=1\sigma=1 after 2M training steps (standard error ≤0.02\leq 0.02).

Position noise moves the representation about 2 units, while task-irrelevant noise moves it by no more than about one environment step. This suggests that value distillation retains most task-relevant information and largely discards the rest.

W6 / Q3. Task Variety and Image. We appreciate the reviewer's concerns regarding the limitations of our evaluation and the challenges associated with jointly learning dynamics and value-aware representations. To investigate these concerns, we conducted additional experiments on PushT, a robotic manipulation task requiring precise object positioning and orientation, under both state-based and image-based observations.

We additionally introduce VALP-WM, a variant in which the encoder and latent dynamics model are pretrained on an offline dataset and subsequently frozen, while the high-level policy and value-aware goal representation are learned during online interaction. The results are summarized below.

MethodImageState
SAC0.0 ± 0.060.0 ± 24.5
LESSON0.0 ± 0.00.0 ± 0.0
HLPS0.0 ± 0.00.8 ± 0.8
MPC11.0 ± 4.039.0 ± 5.8
VALP-WM76.2 ± 18.898.0 ± 1.4
VALP0.0 ± 0.083.6 ± 2.1

These results reveal an important limitation of fully online world-model learning. PushT is particularly challenging due to its sparse success signal and the precision required for manipulation. While VALP achieves reasonable performance with state observations, all methods trained entirely online fail under image-based observations, highlighting the difficulty of jointly learning accurate visual dynamics, value estimates, and goal representations from limited interaction.

Importantly, this limitation can be mitigated by separating world-model learning from task-specific online reinforcement learning. By pretraining and freezing the encoder and latent dynamics, VALP-WM achieves success rates of 76% and 98% under image-based and state-based observations, respectively. This allows VALP to learn value-aware objectives online while planning through a pretrained world model. These findings extend our evaluation beyond Ant-based environments and demonstrate a practical remedy for settings where fully online joint learning is challenging.

We have added the PushT experiments and VALP-WM results to Section 4.5, with implementation details in Appendix E.4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.