- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents a method (STAC) to improve hierarchical RL methods such that they are stable temporally, i.e., that they do not replan prematurely and that they do not continue the current subgoal for longer than necessary. To this end, they add two additional cost functions that modify the original objective. They propose two versions of these cost functions, a determinstic one (don't switch too early, don't keep going for too long) and a learned one. They apply this idea to two existing RL methods: HiPER and GLIDER. The ablation studies show that 1) hierarchical RL is usually better, and 2) adding STAC clearly improves results.
Critical strengths
The authors present an effective method to improve hierarchical RL methods, which can be applied to existing hierarchical RL methods.
Critical weaknesses
The paper does give all the mathematical details to understand the method. However, it's a little short on giving intuitive explanations, making the reader do a lot of heavy lifting.
I am wondering what, instead of using STAC, an overall duration penalty would achieve. I.e., an objective that rewards short overall duration. That feels to me like an additional good baseline, which should do pretty well. It's similar to the proposed costs but isn't explicitly tied to subgoals.
The ablation studies are done with older and small models only. Not entirely clear how the results will translate to somewhat larger models.
Questions for authors
I was surprised to see that the deterministic version did so well on Qwen2.5 and Llama-3.2. Any intuition why?
3 comments
Response to Reviewer stZA (2/2)
W3: The ablation studies are done with older and small models only. Not entirely clear how the results will translate to somewhat larger models.
Thank you, this is a fair point: all component and mechanism ablations use Qwen3-0.6B. (A small clarification: Qwen3 is the most recent of our three backbones; Qwen2.5 and Llama-3.2 are older.)
Why 0.6B for the ablations.
- Headroom. With Qwen3-0.6B, HiPER reaches 83.3% on ALFWorld; with Qwen2.5-1.5B, it already reaches 92.5%. Near the ceiling, differences between variants (short-only vs. full, with vs. without stop-gradient) shrink toward seed noise, so 0.6B gives the most informative comparisons.
- Compute. The ablations already require nearly 60 seeded runs beyond the main table (Tables 3, 7–10), each taking 12–24 hours on 8×H100.
What already transfers across model sizes.
- The penalty weight. was chosen on Qwen3-0.6B and reused unchanged for Qwen2.5-1.5B and Llama-3.2-1B. It improves both HiPER and GLIDER on every backbone (Table 1).
- Both full variants. STAC-D and STAC-P are evaluated on all three backbones and both agents. All 24 settings improve over their baseline at both the best and the final checkpoint. Two of these (HiPER on Qwen2.5-1.5B WebShop) are compared against HiPER's published numbers rather than our own matched runs.
What remains open. The decomposition, meaning how much each cost term and each STAC-P component contributes, has not yet been measured beyond 0.6B. The revised Conclusion now states this explicitly as a limitation.
Questions
Question 1: I was surprised to see that the deterministic version did so well on Qwen2.5 and Llama-3.2. Any intuition why?
Q1: Why does the deterministic version do so well on Qwen2.5 and Llama-3.2?
Good question. Our view is that most of STAC's gain comes from fixing one gross failure, and STAC-D fixes it directly.
1. The main failure is one-turn subgoals, which is exactly what STAC-D targets. Under HiPER, more than half of all turns replace a subgoal that is only one turn old. STAC-D's short cost () prices exactly this: it needs no labels, applies to every boundary decision, and is active from the first update. On Qwen3-0.6B, the short cost alone already delivers most of STAC-D's gain (88.2 of 91.4, Table 3). STAC-P's extra precision about when a subgoal is done adds on top of that, but only when there is room to show it.
2. On Qwen2.5, the two are effectively tied because both are near the ceiling. At the best checkpoint they are within one point (96.4 vs. 96.1 on ALFWorld, 78.2 vs. 77.4 on WebShop), and the same holds for GLIDER (76.7 vs. 76.9, 62.0 vs. 62.1). At the final checkpoint, STAC-P is ahead (92.4 vs. 89.9, 76.2 vs. 74.5). So on Qwen2.5, STAC-D does not actually beat STAC-P; there is simply little headroom left to separate them.
3. On Llama, STAC-P's completion estimator is the bottleneck. The untrained Llama model solves only 1.8% of ALFWorld tasks, so early in training fewer subgoals can be labeled, completions are rare, and the head receives few positive examples. STAC-P's cost also starts only after a warm-up. STAC-D's fixed duration prior depends on none of this, which makes it more robust when the policy is weak.
4. A fixed duration window fits these benchmarks well. The segment lengths STAC-D learns fall mostly inside its window: by the end of training, the median is 3.0 turns on ALFWorld and 2.3 on WebShop, and the 90th percentile is 8.3 and 5.4 (Fig. 4, App. F.1). A single window is therefore a reasonable approximation here. We expect STAC-P's advantage to grow in tasks where subgoal lengths vary more widely.
Follow-up before the discussion closes
Dear Reviewer stZA,
As the discussion period comes to a close, we wanted to check whether our responses have addressed your concerns. If any point is still unclear, we would be glad to address it before the deadline.
If our responses and the new experiments have resolved your concerns, we would be grateful if you would consider updating your score accordingly.
Thank you again for your detailed and constructive review.
Response to Reviewer stZA (1/2)
We thank the reviewer for recognizing the effectiveness and general applicability of STAC. We particularly appreciate the suggestions regarding intuitive explanations, the overall duration penalty baseline, and the performance of the deterministic variant across backbones. We address your concerns below.
Thank you for this feedback. We agree that the submission led with formalism and left too much of the intuition to the reader. We have revised the paper accordingly; all changes are shown in blue in the updated PDF.
In short, the method works like this:
A hierarchical agent repeatedly decides whether to keep its current subgoal or switch to a new one. Left alone, RL training drifts toward switching almost every turn: in HiPER, more than half of all turns replace a subgoal that is only one turn old, so "subgoals" become one-step labels. STAC adds a small cost to this one decision, in both directions: switching too early is charged, and so is keeping a subgoal too long. STAC-D defines "too early" and "too long" by how many turns the subgoal has been active. STAC-P instead asks a learned predictor whether the subgoal is already done. Nothing else in training changes.
Changes in the revised paper:
holds(agent, apple)is true, so pdone for "pick up the apple" is high and keeping it is expensive. One turn into "heat the apple", pdone is low and switching is expensive.Thank you, this is a good suggestion, and we ran it. In short, an episode-duration penalty does help, but it recovers only part of STAC's gain. The full setup and results are now in Appendix J (Table 19) of the revised paper, referenced from Sec. 5.2; all changes are shown in blue.
Setup. We call this baseline HiPER + Episode Duration Penalty (EDP). It subtracts a constant cost from the training reward at every environment step, rt′=rt−λdur, regardless of the subgoal, its age, or the KEEP/SWITCH decision. Unlike STAC, the cost flows through HiPER's returns and critic targets like any other reward. Everything else matches HiPER (Qwen3-0.6B, mean ± SD over three seeds, best-validation checkpoint, same protocol as Table 1), and we sweep λdur.
Findings.