- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Q-Shaped Options (QSO) for learning action abstractions in offline goal-conditioned hierarchical reinforcement learning. The authors identify three desiderata for option representations: encoding achievements rather than executions, preserving distinctions necessary for optimal control, and removing unnecessary distinctions between functionally equivalent behaviors.
QSO learns a shared option encoder through both low-level and high-level Q-functions, aiming to achieve control sufficiency and representation compression simultaneously. The paper presents theoretical arguments for these properties and evaluates the approach on 11 OGBench environments. QSO achieves the highest average success rate among the evaluated methods, with additional analyses of option embeddings and component ablations.
Critical strengths
- A meaningful perspective on action abstraction in hierarchical RL.
The paper identifies an important tension between preserving control-relevant distinctions and compressing functionally equivalent behaviors. The three desiderata in Section 4.1 provide a clear motivation for learning more effective option representations.
- A reasonable joint Q-shaping framework.
Using low-level and high-level Q-functions to shape a shared option encoder is an interesting design. The approach provides a natural way to incorporate action-value information into representation learning rather than relying solely on behavior reconstruction or state values.
- Encouraging results on several challenging benchmarks.
QSO achieves the highest average success rate across the 11 evaluated OGBench environments. Its improvements on HumanoidMaze and Puzzle-4x4 are particularly notable, suggesting that the learned representations can be useful for complex offline goal-conditioned tasks.
- Useful representation analyses and ablations.
The embedding visualizations in Figure 2 and Appendix G provide evidence that QSO can group semantically similar behaviors. The ablations of Q-shaping and option resampling also offer insights into how different design choices influence performance.
Critical weaknesses
- The theoretical analysis does not establish the claimed properties of the learned abstraction.
Proposition 2 establishes control sufficiency by assuming that the low-level critic exactly represents the optimal Q-function for every admissible state-target-action combination. This is a strong representability assumption rather than a guarantee provided by the proposed optimization procedure. Furthermore, high-level Q-shaping does not explicitly enforce the removal of unnecessary distinctions. Consequently, the theoretical results provide limited assurance that joint Q-shaping actually produces an abstraction satisfying both objectives.
- The connection between the offline error bound and the proposed method is insufficient.
Proposition 1 motivates representation compression through an error bound derived for finite state and action spaces under restrictive assumptions. However, QSO uses continuous learned option representations and neural function approximation. The analysis does not characterize the approximation error introduced by compressing potentially distinct behaviors, nor does it establish that the proposed training objective improves the relevant concentrability coefficients. The bound therefore offers limited justification for the practical algorithm.
- The role of temporal abstraction is not convincingly established.
Although QSO uses k-step transitions to train its high-level critic, the high-level policy resamples options at every primitive timestep during deployment. This differs from committing to temporally extended actions, a central motivation for hierarchical RL. Moreover, Table 3 shows substantial performance degradation when options are held for longer periods, particularly in cube-double-play. It remains unclear whether the benefits arise from temporal abstraction or primarily from latent action representations and frequent replanning.
- The performance improvements are inconsistent across environments.
While QSO achieves the highest average success rate, it is the best-performing method in only four of the eleven environments. It performs considerably worse than existing baselines on antmaze-teleport (54% versus 69%), cube-double (70% versus 80%), scene-play (85% versus 94%), and puzzle-3x3 (72% versus 100%). These differences suggest that the proposed abstraction is not consistently beneficial. A more systematic analysis of its failure cases is needed to establish when joint Q-shaping is preferable to simpler alternatives.
- The ablations provide limited evidence for the necessity of joint Q-shaping.
Table 3 evaluates the individual Q-shaping components in only two environments. Notably, high-level-only shaping slightly outperforms full QSO on antmaze-giant, while low-level-only shaping matches its performance on cube-double. Although the embedding visualizations favor joint shaping, its contribution to downstream policy performance is less conclusive. Broader comparisons against the single-critic variants would better support the claimed benefits of combining both objectives.
- Computational complexity and robustness remain insufficiently evaluated.
QSO jointly trains multiple critics and value functions and employs flow-based high-level policy extraction with rejection sampling. However, the experiments mainly compare success rates after fixed optimization steps without systematically reporting training or inference overhead. Additionally, the method assumes deterministic dynamics, while the authors acknowledge that outcome-conditioned options can introduce optimistic bias under stochastic transitions. These limitations restrict the demonstrated practicality and generality of the framework.
Questions for authors
-
Can the authors clarify what properties of the learned encoder are guaranteed by the optimization objective, beyond the exact-Q representability assumption in Proposition 2? Is there evidence that the learned representation approximately preserves optimal action distinctions?
-
Given that options are resampled at every timestep, what evidence distinguishes the contribution of temporal abstraction from that of latent action representations and frequent replanning?
-
Why does joint Q-shaping fail to consistently outperform single-critic shaping in Table 3? Are similar results observed on the remaining benchmarks?
-
What explains the substantial performance gaps on antmaze-teleport, cube-double, scene-play, and puzzle-3x3? Are these failures associated with excessive representation compression or inaccurate value estimation?
-
What are the additional wall-clock training and inference costs of QSO compared with ASO and DSHARSA under matched computational conditions?
6 comments
Q1 Please refer to our response to Reviewer BeZG’s C4. Moreover, we provide empirical evidence that QSO preserves control sufficiency through both downstream performance and a direct analysis of the learned option space. ASO (our ablation of QSO) was introduced precisely to assess whether QSO's learned abstraction discards control-relevant information. Because ASO uses absolute subgoal options, we expect QSO discarding distinctions necessary for control (violating desideratum 2) to yield a policy that performs worse than ASO, while discarding only unnecessary ones (satisfying desideratum 3) to improve its performance through data aggregation. The fact that QSO substantially outperforms ASO (please see the metrics in the Table in response to Reviewer nVJJ above), together with our option-space visualisations showing separation between functionally distinct behaviours (and clustering of functionally equivalent ones), provides empirical evidence that QSO preserves control-relevant distinctions (while enabling greater experience reuse across the state space).
Q2 Please refer to our response to C3 above.
Q3 Please refer to our response to C5 above.
Q4 We hypothesise that the magnitude of QSO’s outperformance depends on the sparsity of the dataset, and the opportunity for experience reuse. In environments with a high number of samples relative to the number of state and action dimensions (which could indicate relatively dense coverage of the state-action space), the benefits of learning a transferable action abstraction might be smaller. Consistent with this interpretation, QSO achieves its largest relative improvements on puzzle-4x4 and humanoidmaze-giant, which have the lowest dataset sizes relative to their state and action dimensions (2,400 and 2,800 respectively against the next lowest (puzzle-3x3) with 3,600). We cannot conclusively attribute the performance gaps on antmaze-teleport, cube-double, scene-play and puzzle-3x3 to approximation and optimisation errors. However, we do not believe it to be due to discarding distinctions necessary for optimal control, as the low-level Q-shaping objective explicitly discourages this, and our representation analyses provide evidence that functionally distinct behaviours remain separated.
Q5 Please refer to our response to C6 above.
As mentioned, we would happily clarify any further questions or concerns.
We wanted to let the reviewer know that we have uploaded a new version of our work. The modifications are in red.
C3 We have clarified this in Section 4.
C4 Please see the modifications in Section 5. We have moved the individual environment performance to the Appendix.
C5 Please see Table 12 in the Appendix.
We sincerely hope that our responses have helped emphasise the significance of our work. Please let us know if there is anything else that we can clarify - otherwise, and in light of these clarifications, we would be grateful if you could please consider raising your score.
Firstly, we thank the reviewer for having taken the time to read our work. We address the concerns below; please let us know if there is anything else that we can clarify.
C1 Due to space constraints, please refer to our response to Reviewer BeZG’s C4.
C2 While Proposition 1 is only applicable to finite state MDPs with at most two possible next state transitions, this is not a theoretical result intended to bound the offline error in our experiments. Its purpose is to motivate discarding unnecessary distinctions between options (a desideratum further motivated by the example in Appendix C, and that has not previously been acknowledged in the literature). The absence of an explicit abstraction approximation-error term does not undermine the motivation provided by Proposition 1. The purpose of this bound is not to characterise the error introduced by arbitrary compression, but to motivate reducing the complexity of the option space under the assumption that the action abstraction preserves information necessary for control. Under this assumption, compressing the option space improves the offline error by reducing the concentrability coefficients as experience is accumulated across more state-goal pairs mapping to the same option and state-option pairs mapping to low-level actions (please refer to lines 960-970 in Appendix E for a detailed explanation). We acknowledge that QSO uses continuous state, action and option-spaces (lines 297-300), and refer to papers that show that the counterpart of compression in finite-state MDPs is mapping to similar points in the latent space in continuous MDPs, where the concentrability coefficients are defined over “neighbourhoods” so that dataset mass at a nearby option can support a queried one (Munos et al., 2008 and Chen at al., 2019).
C3 Resampling at every timestep does not eliminate the benefits of temporal abstraction, as the high-level is still trained on temporally extended actions, reducing the effective decision horizon and facilitating credit assignment in the high-level Q function (Klissarov et al. 2025). We apologise if this was unclear, and have clarified this in the text. More broadly, hierarchical RL does not necessarily require committing to a temporally extended action at deployment; HIQL, SHARSA and DSHARSA also resample at every timestep. The outperformance of hierarchical methods without learned action abstractions (e.g. DSHARSA and SHARSA) over non-hierarchical baselines (e.g. IQL) already isolates the benefits of temporal abstraction (even without deployment commitment) (Park et al. 2025). ASO (our ablation of QSO using absolute subgoal options) distinguishes the benefits of temporal abstraction from those of action abstraction. Outperformance of QSO over ASO is due to action abstraction; ASO over IQL is due to temporal abstraction.
C4 We apologise if we did not clearly convey the significance of our results, and have amended the discussion section accordingly. Due to space constraints, please refer to our response to Reviewer nVJJ's C1.
C5 Thank you for this suggestion. We have extended the Q-shaping ablations to all 11 environments, including training curves in the Appendix. As expected, the results favour joint Q-shaping, which achieves the highest average success rate across environments (54% [51%, 56%]), compared with 42% [41%, 43%] for low-level-only and 49% [48%, 50%] for high-level-only. The relative contribution of each Q-function varies across environments. E.g., in humanoidmaze, removing high-level shaping reduces the success rate from 90% to 6%, while the low-level has little effect; the effect is reversed in cube-double, where removing low-level shaping reduces performance from 70% to 47%. We hypothesise that the importance of each Q-function depends on the environment: high level shaping most benefits locomotion tasks with high state-space transferability, while low-level shaping most benefits manipulation tasks requiring precise control.
C6 Thank you for this suggestion. We have included a table with training and inference costs of all algorithms under matched conditions in the Appendix. To summarise: QSO introduces only modest additional training overhead relative to ASO and DSHARSA, which also train separate high- and low-level value functions, critics, and flow-based policies (e.g. 1.86 [1.85, 1.86] hours for antmaze-giant vs respectively 1.54 [1.53, 1.55] and 1.54 [1.54, 1.55] hours). This overhead is due to training QSO’s shared abstraction encoder (the only architectural difference). QSO has comparable inference costs (e.g. 1.27 [1.24, 1.30]s for antmaze-giant vs 1.21 [1.21, 1.22]s for ASO, and 1.27 [1.24, 1.31]s for DSHARSA). While it would be interesting to work on, the limitation of optimistic bias in stochastic domains is shared by all hindsight-based hierarchical methods (including HIQL, OPAL, SHARSA and DSHARSA) and is orthogonal to this work.
Question Responses: please see below.