acceptodds
Under review as a conference paper at ICLR 2027

Q-Shaped Options for Hierarchical Reinforcement Learning

Abstract

Learning to tackle long-horizon, goal-conditioned tasks requires an agent to reason over extended timescales and act across a broad range of states. This is particularly difficult when the agent must do so entirely from offline data, without online interaction. In principle, Hierarchical Reinforcement Learning (HRL) addresses both challenges: action (temporal) abstraction reduces the effective decision horizon, while the resulting decomposition of primitive action selection into a hierarchy of decisions then permits coarser state (spatial) abstractions than in a non-hierarchical policy, as each level can discard different information irrelevant to its part of the decision process. However, realising these two benefits depends on learning an appropriate action abstraction, and current HRL methods fail in one of two ways. Some algorithms learn an action abstraction that discards distinctions between options that are needed for optimal control, undermining hierarchy altogether, while others retain unnecessary distinctions, which preserves horizon reduction, but forfeits coarser state abstraction. In this work, we characterise three desiderata for an action abstraction, and introduce Q-Shaped Options (QSO), which addresses all three. QSO leverages a prior architecture with distinct state-value functions, Q functions and policies at each level of the hierarchy to learn the action abstraction between consecutive levels as a shared encoder through their respective Q functions. The low-level Q function uses the action abstraction as a goal, encouraging it to retain distinctions between options necessary for optimal control. The high-level Q function uses the action abstraction as an action, encouraging it to discard unnecessary distinctions between options. Across offline goal-conditioned locomotion and manipulation environments, QSO learns semantically meaningful option spaces and outperforms baselines, achieving non-zero performance in tasks where all other evaluated algorithms fail. Our code is open-sourced.

Primary area:
reinforcement learning
Keywords:
Hierarchical Reinforcement Learning, Goal-Conditioned Reinforcement Learning, Offline Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_zz1sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Q-Shaped Options (QSO) for learning action abstractions in offline goal-conditioned hierarchical reinforcement learning. The authors identify three desiderata for option representations: encoding achievements rather than executions, preserving distinctions necessary for optimal control, and removing unnecessary distinctions between functionally equivalent behaviors.

QSO learns a shared option encoder through both low-level and high-level Q-functions, aiming to achieve control sufficiency and representation compression simultaneously. The paper presents theoretical arguments for these properties and evaluates the approach on 11 OGBench environments. QSO achieves the highest average success rate among the evaluated methods, with additional analyses of option embeddings and component ablations.

Critical strengths
  1. A meaningful perspective on action abstraction in hierarchical RL.

The paper identifies an important tension between preserving control-relevant distinctions and compressing functionally equivalent behaviors. The three desiderata in Section 4.1 provide a clear motivation for learning more effective option representations.

  1. A reasonable joint Q-shaping framework.

Using low-level and high-level Q-functions to shape a shared option encoder is an interesting design. The approach provides a natural way to incorporate action-value information into representation learning rather than relying solely on behavior reconstruction or state values.

  1. Encouraging results on several challenging benchmarks.

QSO achieves the highest average success rate across the 11 evaluated OGBench environments. Its improvements on HumanoidMaze and Puzzle-4x4 are particularly notable, suggesting that the learned representations can be useful for complex offline goal-conditioned tasks.

  1. Useful representation analyses and ablations.

The embedding visualizations in Figure 2 and Appendix G provide evidence that QSO can group semantically similar behaviors. The ablations of Q-shaping and option resampling also offer insights into how different design choices influence performance.

Critical weaknesses
  1. The theoretical analysis does not establish the claimed properties of the learned abstraction.

Proposition 2 establishes control sufficiency by assuming that the low-level critic exactly represents the optimal Q-function for every admissible state-target-action combination. This is a strong representability assumption rather than a guarantee provided by the proposed optimization procedure. Furthermore, high-level Q-shaping does not explicitly enforce the removal of unnecessary distinctions. Consequently, the theoretical results provide limited assurance that joint Q-shaping actually produces an abstraction satisfying both objectives.

  1. The connection between the offline error bound and the proposed method is insufficient.

Proposition 1 motivates representation compression through an error bound derived for finite state and action spaces under restrictive assumptions. However, QSO uses continuous learned option representations and neural function approximation. The analysis does not characterize the approximation error introduced by compressing potentially distinct behaviors, nor does it establish that the proposed training objective improves the relevant concentrability coefficients. The bound therefore offers limited justification for the practical algorithm.

  1. The role of temporal abstraction is not convincingly established.

Although QSO uses k-step transitions to train its high-level critic, the high-level policy resamples options at every primitive timestep during deployment. This differs from committing to temporally extended actions, a central motivation for hierarchical RL. Moreover, Table 3 shows substantial performance degradation when options are held for longer periods, particularly in cube-double-play. It remains unclear whether the benefits arise from temporal abstraction or primarily from latent action representations and frequent replanning.

  1. The performance improvements are inconsistent across environments.

While QSO achieves the highest average success rate, it is the best-performing method in only four of the eleven environments. It performs considerably worse than existing baselines on antmaze-teleport (54% versus 69%), cube-double (70% versus 80%), scene-play (85% versus 94%), and puzzle-3x3 (72% versus 100%). These differences suggest that the proposed abstraction is not consistently beneficial. A more systematic analysis of its failure cases is needed to establish when joint Q-shaping is preferable to simpler alternatives.

  1. The ablations provide limited evidence for the necessity of joint Q-shaping.

Table 3 evaluates the individual Q-shaping components in only two environments. Notably, high-level-only shaping slightly outperforms full QSO on antmaze-giant, while low-level-only shaping matches its performance on cube-double. Although the embedding visualizations favor joint shaping, its contribution to downstream policy performance is less conclusive. Broader comparisons against the single-critic variants would better support the claimed benefits of combining both objectives.

  1. Computational complexity and robustness remain insufficiently evaluated.

QSO jointly trains multiple critics and value functions and employs flow-based high-level policy extraction with rejection sampling. However, the experiments mainly compare success rates after fixed optimization steps without systematically reporting training or inference overhead. Additionally, the method assumes deterministic dynamics, while the authors acknowledge that outcome-conditioned options can introduce optimistic bias under stochastic transitions. These limitations restrict the demonstrated practicality and generality of the framework.

Questions for authors
  1. Can the authors clarify what properties of the learned encoder are guaranteed by the optimization objective, beyond the exact-Q representability assumption in Proposition 2? Is there evidence that the learned representation approximately preserves optimal action distinctions?

  2. Given that options are resampled at every timestep, what evidence distinguishes the contribution of temporal abstraction from that of latent action representations and frequent replanning?

  3. Why does joint Q-shaping fail to consistently outperform single-critic shaping in Table 3? Are similar results observed on the remaining benchmarks?

  4. What explains the substantial performance gaps on antmaze-teleport, cube-double, scene-play, and puzzle-3x3? Are these failures associated with excessive representation compression or inaccurate value estimation?

  5. What are the additional wall-clock training and inference costs of QSO compared with ASO and DSHARSA under matched computational conditions?

6 comments

Firstly, we thank the reviewer for having taken the time to read our work. We address the concerns below; please let us know if there is anything else that we can clarify.

C1 Due to space constraints, please refer to our response to Reviewer BeZG’s C4.

C2 While Proposition 1 is only applicable to finite state MDPs with at most two possible next state transitions, this is not a theoretical result intended to bound the offline error in our experiments. Its purpose is to motivate discarding unnecessary distinctions between options (a desideratum further motivated by the example in Appendix C, and that has not previously been acknowledged in the literature). The absence of an explicit abstraction approximation-error term does not undermine the motivation provided by Proposition 1. The purpose of this bound is not to characterise the error introduced by arbitrary compression, but to motivate reducing the complexity of the option space under the assumption that the action abstraction preserves information necessary for control. Under this assumption, compressing the option space improves the offline error by reducing the concentrability coefficients as experience is accumulated across more state-goal pairs mapping to the same option and state-option pairs mapping to low-level actions (please refer to lines 960-970 in Appendix E for a detailed explanation). We acknowledge that QSO uses continuous state, action and option-spaces (lines 297-300), and refer to papers that show that the counterpart of compression in finite-state MDPs is mapping to similar points in the latent space in continuous MDPs, where the concentrability coefficients are defined over “neighbourhoods” so that dataset mass at a nearby option can support a queried one (Munos et al., 2008 and Chen at al., 2019).

C3 Resampling at every timestep does not eliminate the benefits of temporal abstraction, as the high-level is still trained on temporally extended actions, reducing the effective decision horizon and facilitating credit assignment in the high-level Q function (Klissarov et al. 2025). We apologise if this was unclear, and have clarified this in the text. More broadly, hierarchical RL does not necessarily require committing to a temporally extended action at deployment; HIQL, SHARSA and DSHARSA also resample at every timestep. The outperformance of hierarchical methods without learned action abstractions (e.g. DSHARSA and SHARSA) over non-hierarchical baselines (e.g. IQL) already isolates the benefits of temporal abstraction (even without deployment commitment) (Park et al. 2025). ASO (our ablation of QSO using absolute subgoal options) distinguishes the benefits of temporal abstraction from those of action abstraction. Outperformance of QSO over ASO is due to action abstraction; ASO over IQL is due to temporal abstraction.

C4 We apologise if we did not clearly convey the significance of our results, and have amended the discussion section accordingly. Due to space constraints, please refer to our response to Reviewer nVJJ's C1.

C5 Thank you for this suggestion. We have extended the Q-shaping ablations to all 11 environments, including training curves in the Appendix. As expected, the results favour joint Q-shaping, which achieves the highest average success rate across environments (54% [51%, 56%]), compared with 42% [41%, 43%] for low-level-only and 49% [48%, 50%] for high-level-only. The relative contribution of each Q-function varies across environments. E.g., in humanoidmaze, removing high-level shaping reduces the success rate from 90% to 6%, while the low-level has little effect; the effect is reversed in cube-double, where removing low-level shaping reduces performance from 70% to 47%. We hypothesise that the importance of each Q-function depends on the environment: high level shaping most benefits locomotion tasks with high state-space transferability, while low-level shaping most benefits manipulation tasks requiring precise control.

C6 Thank you for this suggestion. We have included a table with training and inference costs of all algorithms under matched conditions in the Appendix. To summarise: QSO introduces only modest additional training overhead relative to ASO and DSHARSA, which also train separate high- and low-level value functions, critics, and flow-based policies (e.g. 1.86 [1.85, 1.86] hours for antmaze-giant vs respectively 1.54 [1.53, 1.55] and 1.54 [1.54, 1.55] hours). This overhead is due to training QSO’s shared abstraction encoder (the only architectural difference). QSO has comparable inference costs (e.g. 1.27 [1.24, 1.30]s for antmaze-giant vs 1.21 [1.21, 1.22]s for ASO, and 1.27 [1.24, 1.31]s for DSHARSA). While it would be interesting to work on, the limitation of optimistic bias in stochastic domains is shared by all hindsight-based hierarchical methods (including HIQL, OPAL, SHARSA and DSHARSA) and is orthogonal to this work.

Question Responses: please see below.

Q1 Please refer to our response to Reviewer BeZG’s C4. Moreover, we provide empirical evidence that QSO preserves control sufficiency through both downstream performance and a direct analysis of the learned option space. ASO (our ablation of QSO) was introduced precisely to assess whether QSO's learned abstraction discards control-relevant information. Because ASO uses absolute subgoal options, we expect QSO discarding distinctions necessary for control (violating desideratum 2) to yield a policy that performs worse than ASO, while discarding only unnecessary ones (satisfying desideratum 3) to improve its performance through data aggregation. The fact that QSO substantially outperforms ASO (please see the metrics in the Table in response to Reviewer nVJJ above), together with our option-space visualisations showing separation between functionally distinct behaviours (and clustering of functionally equivalent ones), provides empirical evidence that QSO preserves control-relevant distinctions (while enabling greater experience reuse across the state space).

Q2 Please refer to our response to C3 above.

Q3 Please refer to our response to C5 above.

Q4 We hypothesise that the magnitude of QSO’s outperformance depends on the sparsity of the dataset, and the opportunity for experience reuse. In environments with a high number of samples relative to the number of state and action dimensions (which could indicate relatively dense coverage of the state-action space), the benefits of learning a transferable action abstraction might be smaller. Consistent with this interpretation, QSO achieves its largest relative improvements on puzzle-4x4 and humanoidmaze-giant, which have the lowest dataset sizes relative to their state and action dimensions (2,400 and 2,800 respectively against the next lowest (puzzle-3x3) with 3,600). We cannot conclusively attribute the performance gaps on antmaze-teleport, cube-double, scene-play and puzzle-3x3 to approximation and optimisation errors. However, we do not believe it to be due to discarding distinctions necessary for optimal control, as the low-level Q-shaping objective explicitly discourages this, and our representation analyses provide evidence that functionally distinct behaviours remain separated.

Q5 Please refer to our response to C6 above.

As mentioned, we would happily clarify any further questions or concerns.

We wanted to let the reviewer know that we have uploaded a new version of our work. The modifications are in red.

C3 We have clarified this in Section 4.

C4 Please see the modifications in Section 5. We have moved the individual environment performance to the Appendix.

C5 Please see Table 12 in the Appendix.

We sincerely hope that our responses have helped emphasise the significance of our work. Please let us know if there is anything else that we can clarify - otherwise, and in light of these clarifications, we would be grateful if you could please consider raising your score.

Reviewer_BezgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

The paper studies how to learn the action abstraction (option space) between the two levels of an offline goal-conditioned hierarchical policy. It argues that existing methods fail in one of two ways:

  • HIQL learns the abstraction through an action-free state-value function, so it can merge options that require different actions.
  • OPAL (which encodes the full trajectory) and SHARSA/DSHARSA (which use raw subgoal states) retain distinctions that are unnecessary for control. This prevents coarser, level-specific state abstractions and limits data reuse.

The authors propose three desiderata for the abstraction: (i) encode the achieved outcome rather than its execution; (ii) preserve the distinctions needed for optimal control; (iii) discard unnecessary ones.

QSO implements these by learning one encoder ϕ(st,st+k)\phi(s_t, s_{t+k}) jointly through two IQL critics. For the high-level critic QhQ_h the option plays the role of an action; for the low-level critic QlQ_l it plays the role of a goal. Low-level Q-shaping is proven to yield a control-sufficient abstraction when the critic is fit exactly. High-level Q-shaping is motivated by an extended PAC bound for joint state and action abstraction, as an inductive bias toward merging functionally equivalent behaviour. Practical details:

  • The option return is normalised so that both critics operate on the same scale.
  • Options lie on a hypersphere.
  • The high-level policy uses an AWR-weighted flow policy with rejection sampling; the low-level policy uses behaviour cloning.

On 11 challenging OGBench environments, QSO achieves the highest average success rate (54% vs. 49% for DSHARSA). It is particularly strong on humanoidmaze-giant-stitch (90%) and puzzle-4x4 (92%). PCA and distance analyses suggest it groups functionally equivalent behaviours while separating distinct ones.

Critical strengths

Pros

  1. The idea is novel and well motivated. It exploits the observation that the option is an action for the high-level decision process and a target for the low-level one, and learns the option representation through both levels' Q functions. This is convincing and elegant. With this action abstraction, QSO performs very well on long-horizon tasks, e.g., 90% on humanoidmaze-giant-stitch vs. 30% for SHARSA/DSHARSA.

  2. The option-space analysis is impressive. The PCA projections and whitened distance matrices (Fig. 2 and Figs. 4–6) show that QSO's embeddings form the structure predicted by the paper's intuition. Semantically similar behaviours (e.g., "move down") cluster together while different behaviours stay separated. Baselines either over-merge (HIQL) or fail to group equivalent behaviours (OPAL, raw subgoals).

Critical weaknesses

Cons

  1. The choice of low-level targets used to shape the encoder is not justified.

    • The encoder is shaped by two critics on different target distributions. The high-level critic encodes ϕ(st,st+k)\phi(s_t, s_{t+k}), with the target exactly kk steps ahead, and the low-level BC policy (Eq. 11) also conditions on ϕ(st,st+k)\phi(s_t, s_{t+k}).
    • The low-level critic Ql(st,ϕ(st,gs),at)Q_l(s_t, \phi(s_t, g_s), a_t) in Desideratum 2, however, uses a target gsg_s drawn from Eq. 9. That is a state at a geometrically sampled offset with mean kk, the current state, or a uniformly random dataset state, bootstrapped through ϕ(st+1,gs)\phi(s_{t+1}, g_s).
    • The paper does not explain why the encoder should be shaped on these (st,gs)(s_t, g_s) pairs rather than on the same (st,st+k)(s_t, s_{t+k}) pairs that the high-level policy outputs and the low-level policy consumes. Nor does it show whether this mismatch matters.
    • Since control sufficiency (Prop. 2) is stated over admissible targets Gk(s)\mathcal{G}_k(s), the relationship between this set and the sampled gsg_s should be made precise and ablated.
  2. The overall improvement over SHARSA-family baselines is not clearly significant.

    • In Table 2, QSO's average (54%) is higher than SHARSA (46%), ASO (47%) and DSHARSA (49%), but it has the best mean in only 4 of 11 environments, two of them by ≤ 1 point (cube-quadruple 4%, puzzle-4x6 18 vs. 17).
    • The average is driven largely by a single environment. Excluding humanoidmaze-giant-stitch, QSO's mean (≈ 50.0) is essentially the same as DSHARSA's (≈ 50.4).
    • QSO is clearly below ASO or SHARSA on antmaze-teleport-stitch (54 vs. 66), scene-play (85 vs. 94) and puzzle-3x3 (72 vs. 76/78).
    • With only 4 seeds and overlapping confidence intervals in many environments, the evidence that QSO is generally superior to SHARSA-style absolute subgoal options is weaker than the headline average suggests.
  3. Inconsistent reward labelling across baselines.

    • For the high-level process, the paper replaces the strict exact-match reward criterion of prior work with the environment success criterion (Appendix F.2).
    • This change is applied to all hierarchical baselines, where it improves their performance (Fig. 14), while the non-hierarchical baselines (IQL, QC) keep strict labelling.
    • This confounds the comparison between hierarchical and flat methods, and makes the reported SHARSA/DSHARSA numbers differ from their published results.
  4. The theoretical support for Desideratum 3 is weak.

    • Prop. 2 guarantees control sufficiency only when the low-level critic exactly equals Q⋆Q^\star.
    • Prop. 1 is a proportional sketch that combines existing PAC bounds under restrictive finite-MDP assumptions (at most two next states).
    • Most importantly, discarding unnecessary distinctions, which is the central novelty, is not enforced by the objective. It is argued only as an inductive bias of high-level Q-shaping and supported empirically.
  5. The representation analyses are qualitative and narrow.

    • The PCA/distance analyses rely on hand-picked behaviour groups (right/up/left/down/dither) rather than many randomly sampled ones.
    • The state-abstraction ANOVA (Table 8) is reported for a single seed.
    • Quantitative clustering metrics over many randomly sampled groups and multiple seeds would make these claims more convincing.
Questions for authors

Already mentioned in the weaknesses section

3 comments

We thank the reviewer for having taken the time to read our work. We address the concerns below; please let us know if there is anything else that we can clarify.

C1 We would like to stress that the low-level targets used to shape the encoder are not a choice, but a necessity for goal-conditioned Q learning. All prior work samples the low-level target in the same manner; the target for the low-level Q function cannot be a fixed k steps ahead as otherwise the success will always be 0, causing TD learning to break down for goal-conditioned RL. Instead, we sample a state so that the mean is k steps into the future. We realise that we omitted the weights used for sampling from the target distribution, apologise for this, and have included them in the Appendix. Regarding Proposition 2, the admissible target set specifies the domain over which control sufficiency is required. Because the admissible targets lie within the sample set, if the action abstraction is control sufficient for the sampled set, it is also control sufficient for the admissible targets.

C2 We apologise if we did not clearly convey the significance of our results, and have amended the discussion section accordingly, adding a table that reports aggregate performance and robustness across environments, including the interquartile mean (IQM) success rate, mean and 25th-percentile performance relative to the best-performing algorithm in each environment, and minimum relative performance. Due to space constraints, please refer to our address to Reviewer nVJJ above: importantly, though, on these aggregate metrics QSO consistently outperforms all competing methods (including all SHARSA-family baselines).

C3 We thank the reviewer for this suggestion. We have now additionally evaluated IQL and QC using environment reward labelling, which improves their performance but does not change QSO's outperformance. We would like to emphasise that our original decision to use strict reward labelling for the non-hierarchical baselines was motivated by fairness and comparability, because we use strict reward labelling for the low-level decision processes of the hierarchical baselines to encourage precise control. We have included reward labelling ablations for all algorithms in the Appendix. We apply the reward labelling change to all hierarchical baselines, as we found that it consistently improves performance. Regarding SHARSA and DSHARSA specifically, we reran SHARSA and DSHARSA under our common evaluation protocol, using their original implementations, which are included in our submitted code. We also note that our experimental setting differs from theirs regardless of reward labelling, as they used oracle goal representations to alleviate challenges of representation learning.

C4 It is true that Proposition 2 guarantees control sufficiency only when the low-level critic equals Q⋆Q^\star, and that exact recovery of Q⋆Q^\star is generally challenging under neural network function approximation and optimisation. Crucially, however, unlike prior unsupervised methods that learn the action abstraction independently of the downstream task, QSO is the first method to incorporate control sufficiency directly into its optimisation objective. The challenges associated with approximating Q⋆Q^\star are common to value-based RL methods and do not diminish QSO’s contribution of explicitly incorporating control sufficiency into the optimisation objective. The reviewer is correct that discarding unnecessary distinctions (Desideratum 3) is not directly enforced, and simply arises due to the inductive bias of high-level Q shaping (described in Section 4.1, lines 303-306). This is not necessarily a weakness, however. We highlight that prior work (e.g. Deep Image Prior, Ulyanov et al. 2017) has already shown and used the idea that useful representation properties can emerge through architectural and optimisation biases without an explicit term enforcing them. The high-level Q-shaping inductive bias is not arbitrary, but motivated by the observation that functionally equivalent behaviours can induce similar high-level Bellman targets, giving the high-level Q-function little predictive benefit from retaining their differences.

C5 We chose to use human-selected behaviour-groups so that they would be human interpretable. We emphasise that while the behaviour groups were human selected, the actual trajectories were randomly sampled from the validation dataset. A similarity metric based on the raw states or actions of a trajectory could lead to groups being identified that evaluate similarity based on exactly the differences that we want to discard (e.g. absolute locations in the state space). We have increased the state-abstraction and goal-sensitivity analyses in Tables 8 and 9 to 4 seeds, thank you for this suggestion.

We wanted to let the reviewer know that we have uploaded a new version of our work. The modifications are in red.

C2 Please see the modifications in Section 5. We have moved the individual environment performance to the Appendix.

C3 Figure 16 contains the environment reward labelling ablation for the non-hierarchical baselines (IQL and AC). All results have been updated to account for the new results.

We sincerely hope that our responses have helped emphasise the significance of our work. Please let us know if there is anything else that we can clarify - otherwise, and in light of these clarifications, we would be grateful if you could please consider raising your score.

Since the author-reviewer discussion period is drawing to a close, we gently nudge the reviewer to respond.

Because we have thoroughly addressed all of their concerns, would they please raise their score? Thank you in advance.

Reviewer_nVJJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • The paper proposes properties a good option representation should have, as existing offline hierarchical RL agents that pass down instructions from high-level policy either discard some information or keep details that don't matter.

  • The paper proposes QSO that is built on DSHARSA and learns the option encoder.

  • Results show that on 11 OGBench locomotion and manipulation environments, QSO has the best average success rate. It performs the best on humanoidmaze. The paper demonstrates with visualizations that QSO groups similar behaviors together.

Critical strengths
  • The paper explains well why HIQL and OPAL/SHARSA fail in different ways in in Table 1.

  • The method is simple and clean.

  • The evaluation is careful. There are 11 challenging environments, 4 seeds with 95% bootstrap confidence intervals. The results is measured at a fixed checkpoint rather than picking the best one. The authors also add a control baseline which trains the high-level policy the same way as QSO but passes raw future states (as SHARSA does) instead of a learned option. Comparing this to QSO isolates the effect of the learned representation.The results show significant gain in the hardest settings: 90% on humanoidmaze-giant vs 30% for SHARSA/DSHARSA, and 92% on puzzle-4x4 vs at most 74% for baselines.

  • The paper released their code. The paper included discussion of limitations.

Critical weaknesses
  • The improvement in average gain is mainly due to humanoidmaze-giant. QSO is best on only 4 of the 11 environments.
Questions for authors
  • The paper use environment-success reward labeling for the hierarchical methods and keep the original strict labeling for flat baselines. But are there any results on applying same labeling on IQL and QC? How will the results change?
3 comments

Firstly, we thank the reviewer for having taken the time to read our work. We address the concerns below; please let us know if there is anything else that we can clarify.

C1 We agree that QSO significantly outperforms prior algorithms in only a subset of environments. However, QSO's primary empirical advantage is its consistency: unlike other methods, it maintains competitive performance across all environments, without exhibiting substantial performance collapse relative to the best-performing method in any environment. We apologise if we did not clearly convey the significance of our results, and have amended the discussion section accordingly, adding a table that reports the interquartile mean (IQM) success rate, mean and 25th-percentile performance relative to the best-performing algorithm in each environment, and minimum relative performance. As well as the highest mean success rate (54% [51%, 56%]), QSO achieves the highest IQM success rate (56% [55%, 58%]), compared with 50% [48%, 51%] and 51% [49%, 52%] for the strongest competing method, DSHARSA. This demonstrates that its strong aggregate performance is not solely driven by a single environment. Furthermore, even in environments where QSO does not achieve the highest success rate, it remains competitive. QSO achieves the highest 25th-percentile relative performance (78% [73%, 82%], compared with 67% [64%, 71%] for DSHARSA) and minimum relative performance (65% [46%, 73%], compared with at most 10% [3%, 17%] for any competing method), demonstrating that QSO achieves substantially stronger lower-tail and worst-case performance across environments. We note that the 10% value is different from the 35% presented in the original text, having rerun the non-hierarchical baselines with environment reward labelling. Notably, none of these confidence intervals overlap with those of competing methods.

We observe that QSO achieves its largest relative performance gains in puzzle-4x4 and humanoidmaze, the two environments with the smallest dataset sizes relative to their combined state and action dimensions (2,400 and 2,800, respectively, compared with 3,600 for the next lowest, puzzle-3x3). While other factors, including dataset coverage and opportunities for behavioural reuse, may also contribute, this observation is consistent with our hypothesis that QSO improves sample efficiency by enabling experience to be reused across the state space. Our analysis of the learned option space provides further qualitative support for this hypothesis. Importantly, our contribution is not purely empirical: we also provide a theoretical argument explaining how a suitably learned action abstraction can facilitate experience reuse across states.

Performance and consistency across 11 OGBench environments:

MeasureIQLQCHIQLOPALSHARSADSHARSAASOQSO
Mean Success Rate29 [27, 30]36 [33, 38]20 [20, 20]45 [44, 46]46 [44, 49]49 [48, 49]47 [46, 48]54 [51, 56]
IQM Success Rate19 [18, 20]27 [26, 30]15 [14, 16]47 [46, 48]47 [45, 49]50 [48, 51]51 [49, 52]56 [55, 58]
Mean % of Best40 [38, 42]68 [64, 71]36 [33, 42]65 [64, 67]68 [65, 71]71 [69, 74]63 [61, 66]86 [82, 90]
25th-Percentile % of Best12 [10, 13]47 [35, 51]12 [9, 14]51 [47, 53]61 [50, 70]67 [61, 70]52 [44, 59]78 [73, 82]
Minimum % of Best0 [0, 0]0 [0, 0]0 [0, 1]0 [0, 0]10 [3, 17]10 [0, 24]0 [0, 0]65 [46, 73]

Success rates are averaged over five evaluation tasks per environment after 1M training steps, then aggregated across environments using the mean and interquartile mean (IQM). Relative performance is expressed as a percentage of the best-performing algorithm in each environment and summarised by its mean, 25th percentile and minimum. Values show point estimates and 95% bootstrap CI over 4 seeds. Bold indicates the best estimate in each row and any other estimates whose CI overlap its interval.

Q1 We thank the reviewer for this suggestion. We have additionally evaluated both IQL and QC using environment reward labelling, which improves their performance but does not change QSO's outperformance. We would like to emphasise that our original decision to use strict reward labelling for the non-hierarchical baselines was motivated by fairness and comparability, because we use strict reward labelling for the low-level decision processes of the hierarchical baselines to encourage precise control. We have included reward labelling ablations for all algorithms in the Appendix.

We wanted to let the reviewer know that we have uploaded a new version of our work. The modifications are in red.

C1 Please see the modifications in Section 5. We have moved the individual environment performance to the Appendix.

Q1 Figure 16 contains the ablation. All results have been updated to account for the new results for IQL and AC (i.e. using environment reward labelling).

We sincerely hope that our responses have helped emphasise the significance of our work. Please let us know if there is anything else that we can clarify - otherwise, and in light of these clarifications, we would be grateful if you could please consider raising your score.

Since the author-reviewer discussion period is drawing to a close, we gently nudge the reviewer to respond.

Because we have thoroughly addressed both their question and concern, would they please raise their score? Thank you in advance.

Reviewer_oxdaOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes Q-Shaped Options (QSO), a temporal action abstraction that aims to balance preserving information needed for low-level control while discarding unnecessary information so that high-level actions may be reused across the state space. The authors formalize the requirements from such an abstraction and propose learning it through an encoder shared by a two-level Q-learning hierarchy. The approach is compared to representative hierarchical and flat baselines on selected OGBench tasks. A thorough ablation and analysis studies the structure of the abstract action space and its effect on performance.

Critical strengths

The paper is well written and easy to follow.

The method is well motivated and principled.

Comprehensive and creative action abstraction analysis.

Critical weaknesses

Experimental Results Discussion

The Results paragraph (starting in line 424) reads as an explanation to why Table 2 should be interpreted as: QSO performs best. While this conclusion is arguable, my main concern is that a connection between the results and the properties of the methods and environments is missing.

Some questions that come to mind:

  • What makes the chosen tasks “the most challenging”? In what aspect are they challenging, and how does that relate to the contribution of your method?
  • Stitch maze datasets have distinct properties, what makes them relevant to your study? Why are the more standard navigate tasks not included?
  • Why do QSO and OPAL perform significantly better than the other methods in humanoidmaze? How is the size of the action space related to this result?
  • Why does QSO perform significantly better than baselines on puzzle-4x4 but not on puzzle-4x5?

To summarize this point, the reader should be convinced that improved performance is likely to originate in the properties of your action abstraction and overall method. Given the mixed results across environments, the claims should be better supported through deeper discussion of the results, and potentially additional results that more clearly favor QSO.

Isolating the Choice of Action Abstraction

While many aspects of the method were ablated, the significance of others for performance is not clear. Specifically, to my understanding, QSO differs from all baselines in two aspects related to the reward that were not ablated in any way: (1) normalized option rewards; (2) using environment rewards for the high-level value functions. All methods using high-level value functions could also use these if I am not mistaken, and they are independent from the choice of action abstraction. Could you provide results for QSO without these and for the relevant baselines with them?

Less critical but important: main text could be more self-contained

I had to read the Appendix quite often to fully understand some parts of the method and the experiment results. Parts of the Appendix are not referenced at all from the main text including important ablations in my opinion. Some of the Appendix sections are long, I think it would be helpful if you refer to the specific subsections that are relevant from the main text instead of e.g., “see Appendix G”.

Questions for authors

Small remark: The reproducibility statement need not be part of the main text, could save some room for other content.

These are not critical for my assessment of the paper but I would be happy to hear your thoughts on them:

  • What aspect of length normalization makes outputs in-distribution other than the magnitudes being in-distribution?
  • “While pure BC learns to ignore the goals, advantage-weighting restores goal-conditioning and leads to better proposals for RS” (lines 353-354): what is this based on?
  • Does the option resampling ablation reveal a limitation of your choice of action abstraction? Do you think the dependence of the abstraction on the current state limits its ability to continue to guide the low-level policy in the following states?
  • If the high-level policy is queried at each timestep, is there really a need for two networks at test time or do you think these could be distilled into one?
6 comments

Firstly, we thank the reviewer for having taken the time to carefully read our work. We address the concerns below; please let us know if there is anything else that we can clarify.

C1 We apologise for the lack of clarity, and have revised the section accordingly. To summarise: we carefully designed our experiments to test QSO's performance on long-horizon tasks and check it satisfies our desiderata 2 and 3. ASO (our ablation) isolates the effect of QSO’s learned action abstraction from confounding factors. We expect QSO discarding distinctions necessary for control (violating desideratum 2) to yield a policy that is worse than ASO, while discarding only unnecessary ones (satisfying desideratum 3) to improve its performance through data aggregation. We then complemented this with direct analysis of the option space because (unlike desideratum 2) desideratum 3 is only encouraged through an inductive bias.

  1. We selected the OGBench tasks with the shortest dataset trajectories and longest episode lengths, making them the most challenging for goal-stitching and long-horizon reasoning (Park et al. 2024).
  2. Stitch datasets have considerably shorter trajectories than navigate ones (e.g. 400 vs 4000 steps for humanoidmaze), allowing us to evaluate agents' ability to compose subgoals across trajectories.
  3. We hypothesise this to be because their learned action abstraction enables behaviour to be reused across the state-space, improving sample efficiency in environments with high state and action complexity (larger action spaces require more data to learn from). Although other factors, such as dataset coverage and opportunities for behaviour reuse, should also be considered, QSO outperforms the most in puzzle-4x4 and humanoidmaze, the two environments with the lowest dataset sizes relative to the number of state and action dimensions (2,400 and 2,800 against puzzle-3x3, the next lowest with 3,600).
  4. The same hypothesis might explain the contrast between puzzle-4x4 and puzzle-4x5. The latter’s dataset has almost three times as many samples relative to number of state and action dimensions, potentially reducing the benefit of experience transfer.

C2 QSO does not differ from the baselines in any way that could undermine the fairness of our experiments. We apologise if this was unclear, and have clarified this in the main text. QSO normalises option rewards to balance the gradients from the high and low-level Q-functions shaping the shared encoder. Other hierarchical methods do not employ joint Q-shaping and therefore do not require comparable Q-function scales (optimal policies are invariant to reward scaling; Tirinzoni et al., 2025). All hierarchical methods use the same dataset sampling and reward labelling methods as QSO (including environment labelling, which was found to improve performance across all algorithms, ablated in Figure 14, Appendix G).

C3 Thank you for carefully reading the Appendix. We have made the main-body more self-contained.

Q1 Thank you for pointing this out: length normalisation before passing to the low-level ensures that the proposed options lie on the same fixed-radius hypersphere that the low-level policy was trained on, but the direction may still be poorly supported by the data.

Q2 The statement is supported by the policy extraction ablation (Table 9). To generalise to state-goal pairs covering the entire state-space at deployment, the high-level is trained on a large proportion of random goals, meaning that a large proportion of goals do not contain information relevant for identifying the option, so the network learns to ignore the goal. AWR gives more weight to options estimated to be useful for randomly sampled goals, weakening the tendency to ignore the goal.

Q3 The option resampling ablation suggests that for the manipulation tasks the option is more dependent on fast-moving state dimensions, but does not necessarily reveal a limitation of how the action abstraction is shaped. It could potentially be rectified with dataset sampling by making the immediate action more dependent on the state rather than the option. We tried to encourage this by using environment reward labelling for the low-level Q function: it reduces the percentage of the option encoder’s output variance explained by fast varying state dimensions (Table 8), but performance still degrades when the option is held (Figure 15). One possible explanation could be the policy extraction overfitting to a particular option, which could be tested by training the low-level policy to predict actions at successive states.

Q4 Querying both levels at every timestep still serves a purpose, as each level can learn a different state abstraction, discarding information irrelevant to its part of the decision process. While distillation into a single policy may be possible, we hypothesise that it would lose these level-specific abstractions that facilitate experience reuse across the state space.

Thank you for your response. I appreciate the clarifications and answers to my questions.

In your response you refer to modifications in the manuscript but I am not sure I see a difference from the pre-rebuttal version of the paper. It would be helpful if you pointed to the lines in which you have made additions and modifications.

C1 Thank you for the additional details and analysis. Where can I find these in the manuscript? The hypothesis in the answers to 3 and 4 would benefit from an empirical evaluation.

C2 Thank you for the clarification regarding the environment rewards. Regarding the option reward normalization: I understand the motivation to normalize rewards in QSO. There is often a gap between theory and practice in Deep RL, and the reward scale can be quite influential in practice in my experience. I would be more convinced with an empirical result.

C3 It is hard to evaluate this point without references to the modifications themselves.

Q1-Q4 Thank you for the detailed and insightful responses to my questions.

To summarize, I appreciate your detailed response to all of the points I have raised. My main concerns were empirical and have not been addressed by experimental results and thus they largely remain. I am not strongly convinced that QSO performs well broadly in practice compared to baselines due to the mixed results and non-ablated factors. That said, I find the core idea neat and well motivated, and appreciate the comprehensive action abstraction analysis. I remain positive about the work but, with the current evidence, will maintain my rating.

Thank you for your quick response and engagement, we very much appreciate it. We have uploaded a new version and have highlighted all of our amendments in red.

C1 The additional details are highlighted in red in Section 5. The hypothesis in response to 3 and 4 would require significantly more evaluation since quantifying dataset quality and coverage is an extremely difficult task. However, we find it interesting to include, which is why we have moved it to a Discussion section.

We apologise because we see that our original results section did not highlight the significance of our results. We have restructured the results section to include only the aggregate metrics (mean, IQM, mean over environments of the relative performance (compared to the best performing algorithm for each environment), 25th percentile relative performance over environments, and worst-case relative performance over environments), and moved individual environment performance to the Appendix. The point is that, as well as outperforming all other algorithms overall (both in terms of mean and IQM success rate), QSO is the most consistent algorithm. Unlike other methods, it maintains competitive performance across all environments, without exhibiting substantial performance collapse relative to the best-performing algorithm in any environment. QSO outperforms across all of these aggregate metrics. Its confidence intervals do not overlap. Please look at Table 3 to see the significance of our results.

We have clarified how our experiments directly connect to our two Desiderata. To summarise: the fact that QSO does not underperform the absolute-subgoal baselines across any of the reported aggregate metrics suggests that the learned abstraction retains control-relevant information. The fact that QSO actually outperforms them suggests that it makes better use of the available data. Together with the analysis of the option space showing separation between functionally distinct behaviours and clustering of functionally equivalent ones, this provides empirical evidence that QSO satisfies both Desideratum 2 (preserving distinctions necessary for control) and Desideratum 3 (enabling greater experience reuse across the state space).

C2 We apologise if this was still unclear: to be explicit, because we normalised the option rewards for QSO, this is also what we did for our hierarchical baselines. We did not ablate this because of prior work showing optimal policies being invariant to reward scaling. Despite this, we understand the reviewer’s concerns, and have rerun the reward scaling ablations, which confirm that using the original reward scaling only leads to decreased performance in QSO (Figure 19).

C3 The main paper is now almost entirely self-contained, and avoids unnecessarily referring to the Appendix. We have also broken the appendix down into subsections. We hope that this satisfies the reviewer.

We thank the reviewer for their suggestions and sincerely hope that our responses have helped emphasise the significance of our work. Please let us know if there is anything else that we can clarify - otherwise, and in light of these clarifications, we would be grateful if you could please consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.