- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper argues that hierarchical RL bundles two separate things into one rule: what a subtask is trying to achieve, and how long a parent stays committed to it. Committing until a subtask ends helps exploration but loses performance. The authors split the loss into two parts. Execution suboptimality is failing to re-select subtasks at deployment. Policy suboptimality is having trained the policy for committed execution in the first place.
Contributions of the paper include (1) separate trees for task objectives and for execution, plus a unified value function and a four-stage Bellman recursion covering any hierarchy depth and execution mode, (2) policy- and execution-improvement theorems. Under a compatibility condition, an optimal-execution result shows that re-deciding every frame is best, and (3) three frameworks that set behavior, learning target, and deployment execution independently. ESIF changes deployment only. PSIF-2S trains bottom-up in stages. PSIF-1S keeps committed exploration while learning frame-wise targets.
The paper validated its claims: (1) a small GridWorld shows both losses exactly, (2) Option-Critic on Fourrooms, with controlled stochasticity, shows step reductions that depend on randomness strength and type, (3) HAC shows success-rate gains from ESIF, (4) a DAC continuation on HalfCheetah gives modest gains and (5) Theory assumes converged optimal policies, whereas the experiments use finite training.
Critical strengths
-
The paper addresses a compelling and foundational question in hierarchical RL: identifying how semi-Markov commitment induces suboptimality. While prior work recognized the benefit of interrupting options, this work goes further by formally separating execution suboptimality from policy suboptimality and establishing an explicit loss decomposition.
-
Decoupling the execution tree from the task tree provides a conceptually elegant abstraction. By separating decision interruption from task termination, the framework enables independent execution choices for behavior, training targets, and deployment without distorting underlying task objectives.
-
The theoretical claims are well-supported across diverse settings, including option-based (OC, DAC) and goal-conditioned (HAC) architectures in discrete and continuous domains. Notably, the controlled stochasticity sweep effectively highlights how transition noise and spatial dynamics dictate the relative gains of execution versus policy improvements.
Critical weaknesses
-
Key acronyms and frameworks (such as ESIF, PSIF, and OC) appear in the introduction before being spelled out or properly contextualized in later sections. Additionally, the notation in Section 2 is tightly packed and shifts between general node indices (), subtrees (), and task nodes (), making the formal exposition challenging to follow on a first read.
-
Critical elements of the theoretical narrative—including formal definitions of SME and ME, detailed descriptions of the policy tree, and step-by-step proofs—are deferred to the supplementary material. Moving key conceptual definitions, particularly the formal mechanics of execution modes and policy-tree construction, into the main body would substantially improve clarity and self-containment.
-
While the paper effectively motivates execution suboptimality through unexpected environmental stochasticity, the narrative would benefit from explicitly clarifying whether execution and policy suboptimality exhaust all potential sources of loss within a fixed task hierarchy. A clearer discussion contrasting this decomposition with representation-induced suboptimality or task-specification error would strengthen the conceptual foundation in Section 2.
Questions for authors
Could the authors elaborate on the advantages and trade-offs of formalizing hierarchical scheduling via explicit task and execution trees compared to alternative representations (such as call-stack augmented state spaces or programmatic automata)? Specifically, are there scalability, memory, or computational constraints when maintaining these tree structures in deeper hierarchies or environments with continuous subtask spaces?