acceptodds
Under review as a conference paper at ICLR 2027

Stable Temporal Abstraction for Hierarchical Reinforcement Learning in Language Model Agents

Abstract

Hierarchical reinforcement learning improves long-horizon control by organizing primitive actions around persistent subgoals and assigning credit at multiple temporal scales. Recent hierarchical language agents bring these benefits to interactive tasks by explicitly separating subgoal planning from action execution. We observe, however, that an explicit hierarchy does not by itself determine how stable the resulting temporal abstraction is: the learned boundary policy may replace the subgoal almost every turn, making it effectively transient, or retain a subgoal after it has stopped being appropriate. We call this temporal abstraction instability. We propose Stable Temporal Abstraction via Constrained Optimization (), a general constrained boundary-policy framework that represents premature replanning and stale persistence as constraint costs. We instantiate in two forms: uses duration-based proxies for premature replanning and stale persistence, providing a simple environment-agnostic realization; learns semantic completion probabilities from environment-grounded supervision and converts them into state-dependent boundary costs. Both instantiations apply their Lagrangian costs only to the sampled boundary decision, leaving the underlying algorithm's rewards, original critic targets, subgoal advantages, and primitive-action advantages unchanged. Across two primary backbones and two benchmarks, improves success over a strong hierarchical baseline by and points on ALFWorld and WebShop with Qwen3-0.6B and by and points with Llama-3.2-1B-Instruct. A separate Qwen2.5-1.5B-Instruct ALFWorld replication reaches peak success with versus for the released HiPER host. The code is available at https://anonymous.4open.science/r/STAC-EFF9.

Primary area:
reinforcement learning
Keywords:
Hirarchical RL

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_stZAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a method (STAC) to improve hierarchical RL methods such that they are stable temporally, i.e., that they do not replan prematurely and that they do not continue the current subgoal for longer than necessary. To this end, they add two additional cost functions that modify the original objective. They propose two versions of these cost functions, a determinstic one (don't switch too early, don't keep going for too long) and a learned one. They apply this idea to two existing RL methods: HiPER and GLIDER. The ablation studies show that 1) hierarchical RL is usually better, and 2) adding STAC clearly improves results.

Critical strengths

The authors present an effective method to improve hierarchical RL methods, which can be applied to existing hierarchical RL methods.

Critical weaknesses

The paper does give all the mathematical details to understand the method. However, it's a little short on giving intuitive explanations, making the reader do a lot of heavy lifting.

I am wondering what, instead of using STAC, an overall duration penalty would achieve. I.e., an objective that rewards short overall duration. That feels to me like an additional good baseline, which should do pretty well. It's similar to the proposed costs but isn't explicitly tied to subgoals.

The ablation studies are done with older and small models only. Not entirely clear how the results will translate to somewhat larger models.

Questions for authors

I was surprised to see that the deterministic version did so well on Qwen2.5 and Llama-3.2. Any intuition why?

3 comments
AuthorsOpenReview ↗

Response to Reviewer stZA (1/2)

We thank the reviewer for recognizing the effectiveness and general applicability of STAC. We particularly appreciate the suggestions regarding intuitive explanations, the overall duration penalty baseline, and the performance of the deterministic variant across backbones. We address your concerns below.


W1: The paper does give all the mathematical details to understand the method. However, it's a little short on giving intuitive explanations, making the reader do a lot of heavy lifting.

Thank you for this feedback. We agree that the submission led with formalism and left too much of the intuition to the reader. We have revised the paper accordingly; all changes are shown in blue in the updated PDF.

In short, the method works like this:

A hierarchical agent repeatedly decides whether to keep its current subgoal or switch to a new one. Left alone, RL training drifts toward switching almost every turn: in HiPER, more than half of all turns replace a subgoal that is only one turn old, so "subgoals" become one-step labels. STAC adds a small cost to this one decision, in both directions: switching too early is charged, and so is keeping a subgoal too long. STAC-D defines "too early" and "too long" by how many turns the subgoal has been active. STAC-P instead asks a learned predictor whether the subgoal is already done. Nothing else in training changes.

Changes in the revised paper:

  1. A running example (Sec. 2, "Running example"). A "heat the apple" task now illustrates both failure modes concretely. A premature switch abandons "heat the apple" one turn after adopting it. A stale keep holds on to "pick up the apple" after the apple is already in hand. Secs. 3.1 and 3.3 refer back to this example when introducing each cost.
  2. A plain-language explanation before the math. Sec. 3.1, both rules in Sec. 3.2, STAC-D and STAC-P (Sec. 3.3), and the HiPER and GLIDER integrations (Sec. 4) now each open with a sentence explaining what the equations do. STAC-D also gets a worked example with the actual thresholds (dmin⁡=2d_{\min}=2, dmax⁡=8d_{\max}=8): switching after one turn is charged, switching after three turns is free, and keeping a subgoal at age eight is charged. STAC-P gets a matching example in the same task. Once the apple is in hand, the simulator predicate holds(agent, apple) is true, so pdonep^{\text{done}} for "pick up the apple" is high and keeping it is expensive. One turn into "heat the apple", pdonep^{\text{done}} is low and switching is expensive.
  3. A concrete trajectory in the main text (Sec. 5.1, Fig. 2). A verbatim episode shows the learned boundary in action. The agent keeps its subgoal through three turns of navigation and pickup, then switches at the receptacle to a grounded subgoal that solves the task. The full version with observations remains in App. F.1.
  4. A lighter Sec. 3.2. The proof that the two cost estimators are equivalent now lives in App. C.1. The main text keeps a one-sentence summary.

W2: I am wondering what, instead of using STAC, an overall duration penalty would achieve. I.e., an objective that rewards short overall duration... .

Thank you, this is a good suggestion, and we ran it. In short, an episode-duration penalty does help, but it recovers only part of STAC's gain. The full setup and results are now in Appendix J (Table 19) of the revised paper, referenced from Sec. 5.2; all changes are shown in blue.

Setup. We call this baseline HiPER + Episode Duration Penalty (EDP). It subtracts a constant cost from the training reward at every environment step, rt′=rt−λdurr'_t = r_t - \lambda_{\text{dur}}, regardless of the subgoal, its age, or the KEEP/SWITCH decision. Unlike STAC, the cost flows through HiPER's returns and critic targets like any other reward. Everything else matches HiPER (Qwen3-0.6B, mean ± SD over three seeds, best-validation checkpoint, same protocol as Table 1), and we sweep λdur\lambda_{\text{dur}}.

MethodALFWorld ↑WebShop ↑
HiPER83.3 ± 3.361.4 ± 3.5
EDP, λdur\lambda_{{dur}} = 0.00185.4 ± 3.363.0 ± 3.4
EDP, λdur\lambda_{{dur}} = 0.00286.6 ± 3.165.4 ± 3.2
EDP, λdur\lambda_{{dur}} = 0.00588.7 ± 2.967.3 ± 3.1
EDP, λdur\lambda_{{dur}} = 0.01086.4 ± 3.465.2 ± 3.6
EDP, λdur\lambda_{{dur}} = 0.02082.1 ± 4.060.9 ± 4.1
HiPER + STAC-D91.4 ± 3.669.3 ± 2.4
HiPER + STAC-P95.3 ± 1.674.2 ± 1.1

Findings.

  • At its best λdur\lambda_{{dur}} (0.005), EDP improves HiPER by +5.4 on ALFWorld and +5.9 on WebShop.
  • It is 6.6 / 6.9 points below STAC-P, which is significant (p = 0.039 / 0.048).
  • At λdur\lambda_{\mathrm{dur}} = 0.02 the gain disappears (−1.2 / −0.5 vs. HiPER), as pressure to finish quickly starts to compete with solving the task.
AuthorsOpenReview ↗

Response to Reviewer stZA (2/2)

W3: The ablation studies are done with older and small models only. Not entirely clear how the results will translate to somewhat larger models.

Thank you, this is a fair point: all component and mechanism ablations use Qwen3-0.6B. (A small clarification: Qwen3 is the most recent of our three backbones; Qwen2.5 and Llama-3.2 are older.)

Why 0.6B for the ablations.

  • Headroom. With Qwen3-0.6B, HiPER reaches 83.3% on ALFWorld; with Qwen2.5-1.5B, it already reaches 92.5%. Near the ceiling, differences between variants (short-only vs. full, with vs. without stop-gradient) shrink toward seed noise, so 0.6B gives the most informative comparisons.
  • Compute. The ablations already require nearly 60 seeded runs beyond the main table (Tables 3, 7–10), each taking 12–24 hours on 8×H100.

What already transfers across model sizes.

  • The penalty weight. μ=0.1\mu=0.1 was chosen on Qwen3-0.6B and reused unchanged for Qwen2.5-1.5B and Llama-3.2-1B. It improves both HiPER and GLIDER on every backbone (Table 1).
  • Both full variants. STAC-D and STAC-P are evaluated on all three backbones and both agents. All 24 settings improve over their baseline at both the best and the final checkpoint. Two of these (HiPER on Qwen2.5-1.5B WebShop) are compared against HiPER's published numbers rather than our own matched runs.

What remains open. The decomposition, meaning how much each cost term and each STAC-P component contributes, has not yet been measured beyond 0.6B. The revised Conclusion now states this explicitly as a limitation.


Questions

Question 1: I was surprised to see that the deterministic version did so well on Qwen2.5 and Llama-3.2. Any intuition why?

Q1: Why does the deterministic version do so well on Qwen2.5 and Llama-3.2?

Good question. Our view is that most of STAC's gain comes from fixing one gross failure, and STAC-D fixes it directly.

1. The main failure is one-turn subgoals, which is exactly what STAC-D targets. Under HiPER, more than half of all turns replace a subgoal that is only one turn old. STAC-D's short cost (dmin⁡=2d_{\min}=2) prices exactly this: it needs no labels, applies to every boundary decision, and is active from the first update. On Qwen3-0.6B, the short cost alone already delivers most of STAC-D's gain (88.2 of 91.4, Table 3). STAC-P's extra precision about when a subgoal is done adds on top of that, but only when there is room to show it.

2. On Qwen2.5, the two are effectively tied because both are near the ceiling. At the best checkpoint they are within one point (96.4 vs. 96.1 on ALFWorld, 78.2 vs. 77.4 on WebShop), and the same holds for GLIDER (76.7 vs. 76.9, 62.0 vs. 62.1). At the final checkpoint, STAC-P is ahead (92.4 vs. 89.9, 76.2 vs. 74.5). So on Qwen2.5, STAC-D does not actually beat STAC-P; there is simply little headroom left to separate them.

3. On Llama, STAC-P's completion estimator is the bottleneck. The untrained Llama model solves only 1.8% of ALFWorld tasks, so early in training fewer subgoals can be labeled, completions are rare, and the head receives few positive examples. STAC-P's cost also starts only after a warm-up. STAC-D's fixed duration prior depends on none of this, which makes it more robust when the policy is weak.

4. A fixed duration window fits these benchmarks well. The segment lengths STAC-D learns fall mostly inside its [2,8][2, 8] window: by the end of training, the median is 3.0 turns on ALFWorld and 2.3 on WebShop, and the 90th percentile is 8.3 and 5.4 (Fig. 4, App. F.1). A single window is therefore a reasonable approximation here. We expect STAC-P's advantage to grow in tasks where subgoal lengths vary more widely.

AuthorsOpenReview ↗

Follow-up before the discussion closes

Dear Reviewer stZA,

As the discussion period comes to a close, we wanted to check whether our responses have addressed your concerns. If any point is still unclear, we would be glad to address it before the deadline.

If our responses and the new experiments have resolved your concerns, we would be grateful if you would consider updating your score accordingly.

Thank you again for your detailed and constructive review.

Reviewer_1CeYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at hierarchical LLM agents, which do more than picking an action each turn. They also maintain a text subgoal and decide at every turn whether to keep it or switch to a new one. The authors observe that this keep/switch decision is often learned poorly, because training rewards only task success and never directly judges switching behavior. As a result, the agent may switch almost every turn, which turns subgoals into one-step labels and makes the hierarchy useless. It may also keep a subgoal after it is finished. For example, an existing method, HiPER, switches too early on more than half of all turns. The authors call this problem temporal abstraction instability.

To fix it, they propose STAC, a training add-on for existing hierarchical agents. STAC adds two penalties to the keep/switch decision only: one for switching too early and one for keeping a subgoal too long. Rewards, value estimates, and the learning of subgoals and actions are unchanged. The paper gives two versions. STAC-D uses simple duration rules: it penalizes switching before two turns and keeping past a maximum length. STAC-P trains a small predictor of whether the current subgoal is finished, using labels from the simulator during training. It then penalizes switching unfinished subgoals and keeping finished ones.

The authors add STAC to two different hierarchical agents, HiPER and GLIDER. They train three small open models (Qwen3-0.6B, Qwen2.5-1.5B, Llama-3.2-1B) with RL on ALFWorld and WebShop. STAC improves success in every setting, with gains of up to 12 points on Qwen3-0.6B and 23.5 points on Llama-3.2-1B. It also greatly reduces premature switching and produces longer, more meaningful subgoals. The largest improvements come on tasks that require holding one subgoal over many steps.

Critical strengths
  • Useful observation. Unmodified HiPER switches prematurely on more than half of all turns, so an explicit hierarchy does not guarantee real temporal abstraction.
  • Simple, general method. STAC changes only the boundary credit and works on two quite different agent designs.
  • Convincing evidence. Gains appear in every setting. They are largest on Pick2 tasks, which need long subgoals (23% to 81% on Qwen3-0.6B). Training the completion predictor alone does not help, which shows that the gain comes from changing the boundary credit.
Critical weaknesses
  • The constrained framing does not match what is run. All experiments use one fixed penalty weight, and the constraint budgets never affect training. The method is a penalty term, and the theory (Proposition 1) adds little.

  • Limited scope. There are only two text simulators and only models of 1.5B parameters or smaller.

Questions for authors
  • Does the adaptive version actually meet chosen budgets, and how does it compare to the fixed weight?
  • Do the gains hold when the penalty weight is tuned on a separate split, or when all methods are compared at the final checkpoint?
  • How does STAC-D differ from a deliberation cost or a constant switch penalty? Can you add these baselines?
  • Can STAC-P work without simulator labels, and what happens on the unlabeled decisions?
  • Why does STAC-D beat STAC-P on Llama-3.2-1B?
  • Does the problem, and the fix, still appear with models of 7B or more?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 1CeY (1/3)

We thank the reviewer for the constructive review, and for recognizing the importance of temporal abstraction instability, the generality of our approach, and the strength of our empirical results. We appreciate the insightful questions, particularly regarding the constrained formulation and alternative baselines, which help us better clarify the contribution of STAC. We address each concern below.

W1: The constrained framing does not match what is run. All experiments use one fixed penalty weight, and the constraint budgets never affect training. The method is a penalty term, and the theory (Proposition 1) adds little.

Thank you. For the experiments in the submission, your description is accurate, and the paper says so itself (Sec. 3.1, App. A, App. C, Conclusion): μ\mu is held fixed, the budgets ρ\rho never enter the update, and the violation rates we report are diagnostics, not targets. Contribution 2 already calls the method "a decision-local, two-sided boundary regularizer." We have addressed the mismatch in two ways: by testing the constrained version directly, and by being clearer about what the framing contributes.

The budgets can actually be enforced. During the rebuttal, we ran the adaptive version of STAC that the paper only described (Eq. 9). The two multipliers are no longer fixed. Each is adjusted during training by a projected dual step,μi←[μi+ημ(C^i−ρi)]+\mu_i \leftarrow \left[\mu_i + \eta_\mu (\hat{C}_i - \rho_i)\right]_{+}, so it grows while its constraint is violated and relaxes once the constraint is satisfied. All other settings follow STAC-D (ALFWorld, Qwen3-0.6B, dmin⁡=2d_{\min}=2, dmax⁡=8d_{\max}=8, 3 seeds, best-validation checkpoint). For each of three budget pairs, we compare against the best fixed μ\mu from our sweep (Table 7) that also respects that budget:

Target (ρshort,ρstale)(\rho_{\text{short}}, \rho_{\text{stale}})Reached (prem. / stale)AdaptiveBest fixed μ\mu meeting targetFixed
Relaxed (0.15, 0.12)0.142 / 0.11690.8μ\mu=0.1091.4
Moderate (0.10, 0.08)0.096 / 0.07891.2μ\mu=0.2090.9
Strict (0.05, 0.06)0.048 / 0.05889.6μ\mu=0.4087.5
  • Every target is met. The mean rates settle just below each budget, as expected when a constraint binds.
  • Enforcing budgets costs little in success. For the relaxed and moderate targets, adaptive and hand-picked fixed weights are within 0.6 points of each other (−0.6 and +0.3).
  • Tight targets favor separate multipliers. A shared fixed weight can only satisfy the strict stale target at μ=0.4\mu=0.4. At that weight it also over-suppresses premature switching (0.022 against a target of 0.05), and success falls. The adaptive version raises only the multiplier whose constraint is binding and keeps 2.1 more points. With three seeds, we read this as comparable-or-better rather than a decisive gain.

These results are now in the revised paper (Appendix I), referenced from Sec. 3.1 and the Conclusion.

What the constrained view contributes beyond enforcement. Even with fixed weights, the formulation shaped the method in two concrete ways:

  • Two separately priced costs, not one switch penalty. Treating premature switching and stale persistence as separate constraints yields two separate costs, each measurable as a rate. These rates reveal HiPER's failure mode (more than half of turns are premature switches, Fig. 1) and show what each cost term corrects (Tables 3, 8).
  • One cost that works for both agents. Writing each cost as an expectation under the boundary policy πb\pi^b gives two exact estimators of the same quantity: a sampled one through the boundary advantage (Eq. 10), and an enumerated one over KEEP/SWITCH (Eq. 11). App. C.1 shows that both estimate the same surrogate. This is what lets the same cost plug into HiPER's on-policy PPO advantage and into GLIDER's offline BC/AWAC Boolean loss without altering either agent's objective.

On "one fixed penalty weight." Each fixed-μ\mu run keeps μ\mu constant, but we sweep it on both benchmarks (Tables 7, 9). The single value chosen on ALFWorld validation, μ=0.1\mu=0.1, transfers unchanged to WebShop, all three backbones, and both agents. The adaptive version above removes even that one choice: the user specifies target rates instead of a weight.

Proposition 1. We agree it adds little on its own. It is the standard fact that a fixed-penalty optimum is also optimal under whatever violation rates it happens to reach, and it guarantees no budget chosen in advance. The adaptive experiment above now plays the role the proposition could not: it shows budgets being met. In the revised paper, we have replaced the proposition with a short paragraph in App. C ("Fixed multipliers and implied budgets") that states exactly this.

AuthorsOpenReview ↗

Response to Reviewer 1CeY (2/3)

W2: Limited scope. There are only two text simulators and only models of 1.5B parameters or smaller.

Thank you. We agree that larger models and additional environments would strengthen the paper, and the revised Conclusion now states the current scope explicitly as a limitation. Here is what the existing evidence does and does not cover.

Breadth within the current scope. The evaluation crosses 2 benchmarks × 3 backbones from two model families (Qwen3-0.6B, Qwen2.5-1.5B, Llama-3.2-1B) × 2 structurally different hierarchical agents × 2 STAC variants. The two agents train in very different ways. HiPER has an explicit boundary trained on-policy with PPO. GLIDER has an implicit Boolean completion token trained offline with BC/AWAC. All 24 combinations improve over their baseline at both the best and the final checkpoint (Table 1). For 22 of them, the baseline is our own matched three-seed run. The remaining two, HiPER + STAC-D and HiPER + STAC-P on Qwen2.5-1.5B WebShop, are compared against HiPER's published numbers, so we treat them as supporting rather than controlled evidence.

Benchmark choice. ALFWorld and WebShop are the shared testbeds of the hierarchical and multi-turn RL methods we build on and compare against (HiPER, GiGPO, HiMAC), which allows direct comparison with published results. ALFWorld's multi-phase tasks are also where boundary timing matters most. On pick-two, which requires holding one subgoal through a full fetch-and-place before switching to the next, success rises from 23.1% (HiPER) to 81.1% (HiPER + STAC-D) with Qwen3-0.6B (App. F).

Model size. We see over-switching as a property of the training objective rather than of model capacity: the task return never charges for a premature or stale boundary decision (Sec. 2). A larger model might still discover good boundaries on its own, and we cannot rule that out without testing it. Two observations bear on it:

  • The failure is not tied to one backbone. It appears with all three backbones and with both agents.
  • The gain persists near the ceiling. With Qwen2.5-1.5B, HiPER already reaches 92.5% on ALFWorld. STAC still adds +3.9 (STAC-D) and +3.6 (STAC-P) at the best checkpoint, and +4.7 (STAC-D) and +7.2 (STAC-P) at the final checkpoint. It also improves GLIDER by +4.5 (STAC-D) and +4.7 (STAC-P) at the best checkpoint.

Questions:

Q1: Does the adaptive version actually meet chosen budgets, and how does it compare to the fixed weight?

Yes. Please see our response to W1 for the full setup and results. In brief: we ran the adaptive primal–dual variant (Eq. 9) with three budget pairs on ALFWorld (Qwen3-0.6B). All three are met, with the achieved rates settling just below each budget. Success is comparable to the best fixed μ\mu that satisfies the same budget (−0.6, +0.3, and +2.1 points).

Q2: Do the gains hold when the penalty weight is tuned on a separate split, or when all methods are compared at the final checkpoint?

Final checkpoint: yes. Table 1 reports every method at both the best and the final checkpoint. All 24 STAC settings (2 agents × 2 variants × 3 backbones × 2 benchmarks) also improve over their baseline at the final checkpoint. As noted in W2, two of these are compared against HiPER's published Qwen2.5 WebShop numbers. With Qwen3-0.6B, the final-checkpoint gains over HiPER are comparable to the best-checkpoint ones:

Qwen3-0.6BALFWorldWebShop
HiPER81.556.5
+ STAC-D89.3 (+7.8, pWp_W=0.055)67.5 (+11.0, pWp_W=0.025)
+ STAC-P93.2 (+11.7, pWp_W=0.013)74.2 (+17.7, pWp_W=0.005)

One cell misses pW<0.05p_W<0.05: STAC-D on ALFWorld at the final checkpoint (pW=0.055p_W=0.055, Table 13). With Llama-3.2-1B, the final-checkpoint gains are +23.2 (STAC-D) and +17.2 (STAC-P) on ALFWorld, and +15.3 (STAC-D) and +14.4 (STAC-P) on WebShop (Table 13). To reduce single-checkpoint noise, App. G also averages validation over updates 120–150: STAC-D beats HiPER by +9.5 points (95% CI [+5.2, +13.8]). STAC-D is also above HiPER at every evaluation from update 80 to 150 on both backbones (Fig. 6).

Separate tuning split. We did not tune μ\mu on a separate split. As App. F.3 notes, using the same validation runs to select both μ\mu and the checkpoint may favor the reported peak. Two facts limit how much this matters:

  • μ\mu was tuned once and then frozen (see also W1). It was chosen only on ALFWorld / Qwen3-0.6B / HiPER+STAC-D. WebShop, the other two backbones, GLIDER, and STAC-P all reuse μ=0.1\mu=0.1 without further tuning. They are effectively held-out settings, and the gains hold there.
  • The gain does not depend on picking the right μ\mu. Every value in the sweep beats HiPER, at both checkpoints:
SweepBest ckptFinal ckpt
ALFWorld, μ∈[0.025,0.4]\mu \in [0.025, 0.4] (HiPER: 83.3 / 81.5)87.2–91.485.1–89.3
WebShop, μ∈[0.025,0.2]\mu \in [0.025, 0.2] (HiPER: 61.4 / 56.5)64.8–69.360.8–67.5
AuthorsOpenReview ↗

Response to Reviewer 1CeY (3/3)

Q3: How does STAC-D differ from a deliberation cost or a constant switch penalty? Can you add these baselines?

A deliberation cost (Harb et al., 2018) and a constant switch penalty both put a fixed charge on every SWITCH. STAC-D differs in two ways:

  1. Two-sided. STAC-D also charges KEEP once a subgoal is stale (d≥dmax⁡d \ge d_{\max}). A constant penalty instead makes leaving a stale subgoal more expensive. The short-only variant, which is the deliberation-cost-like direction, is weaker than the full version (88.2 vs. 91.4 on ALFWorld, 67.4 vs. 69.3 on WebShop; Tables 3 and 8). On ALFWorld it also raises stale persistence above HiPER (0.185 vs. 0.147), while the full version lowers it to 0.098.
  2. Age-dependent. STAC-D charges a switch only when the subgoal is one turn old; later switches are free. A constant penalty taxes every switch, including well-timed ones.

STAC-P goes further and makes the cost state-dependent (1−pdone1-p^{\text{done}} to switch, pdonep^{\text{done}} to keep), giving the largest gains (95.3 / 74.2).

Related controls (run during the rebuttal). All runs use HiPER with Qwen3-0.6B, three seeds, and the best-validation checkpoint. Violation rates are measured on ALFWorld.

(i) Constant switch penalty. We subtract a constant κ\kappa from the boundary advantage of every SWITCH, A^tb−κ⋅1[qt=SWITCH]\hat A^b_t - \kappa \cdot \mathbb{1}[q_t=\text{SWITCH}]. This is STAC's sampled rule with gshort≡1g_{\text{short}}\equiv 1 and no stale cost.

MethodALFWorldWebShopPrematureStale
HiPER83.3 ± 3.361.4 ± 3.50.5530.147
κ=0.025\kappa=0.02585.9 ± 3.163.6 ± 3.40.3100.160
κ=0.05\kappa=0.0587.7 ± 2.965.7 ± 3.00.1700.190
κ=0.10\kappa=0.1086.8 ± 3.465.9 ± 3.20.0750.240
κ=0.20\kappa=0.2083.1 ± 3.862.4 ± 3.90.0350.310
STAC-D91.4 ± 3.669.3 ± 2.40.0730.098
STAC-P95.3 ± 1.674.2 ± 1.10.1010.061

The constant penalty stays below STAC-D at every κ\kappa. At κ=0.10\kappa=0.10 it nearly matches STAC-D's premature rate (0.075 vs. 0.073). However, its stale persistence rises to 0.240 (vs. 0.098), and its ALFWorld success is 86.8 vs. 91.4. Taxing every switch also taxes leaving a stale subgoal, which is exactly the difference described in points 1–2 above.

(ii) Episode-duration penalty. This is a constant per-step cost in the reward, rt′=rt−λdurr'_t = r_t - \lambda_{\mathrm{dur}}, that does not depend on subgoals. At its best strength (0.005), it improves HiPER by +5.4 / +5.9, vs. +8.1 / +7.9 for STAC-D and +12.0 / +12.8 for STAC-P.

Both controls are now in the revised paper (Appendix K, Table 20, and Appendix J, Table 19).

Both controls are now in the revised paper (Appendix K, Table 20, and Appendix J, Table 19).

Q4: Can STAC-P work without simulator labels, and what happens on the unlabeled decisions?

Unlabeled decisions (mt=0m_t=0; 37% on ALFWorld, 58% on WebShop) get no STAC-P cost and keep HiPER's native objective. The effect still transfers through the shared policy: had they kept HiPER's premature rate, STAC-P's overall rate would be at least 0.21 / 0.35, but it is 0.101 / 0.181 (Table 6).

Without simulator labels:

  • STAC-D needs none and recovers most of the gain (+8.1 / +7.9 vs. +12.0 / +12.8 for STAC-P).
  • STAC-P needs a completion label, not simulator internals. Any binary "is the subgoal achieved?" signal can supervise the head (e.g., a rule-based checker or an LLM judge). We have not tested noisier sources, and the revised Conclusion now states this as a limitation. The learned head already nearly matches oracle labels (95.3 vs. 96.1, Table 10), suggesting some tolerance to prediction error.

Q5: Why does STAC-D beat STAC-P on Llama-3.2-1B?

The clear gap is HiPER on ALFWorld (52.1 vs. 39.8 best, 40.4 vs. 34.4 final); the other Llama settings are within one SD. We believe STAC-P's completion estimator is weakest when the backbone is weak:

  1. Lower coverage: a weak policy writes more unlabelable subgoals, while STAC-D covers every decision.
  2. Few positives: the untrained model solves 1.8% of tasks, so completion labels are rare and pdonep^{\text{done}} is noisy.
  3. Later start: STAC-P waits for a warm-up; STAC-D acts from the first update.

STAC-D is a robust label-free prior; STAC-P is sharper when its estimator is reliable (clearly better on Qwen3-0.6B, ahead at the final checkpoint on Qwen2.5-1.5B).


Q6: Does the problem, and the fix, still appear with models of 7B or more?

Not yet tested; please see our response to W2. In brief, the problem comes from the objective: the return never prices boundary timing. It appears with all three backbones and both agents. The gain also persists near the ceiling at 1.5B: on ALFWorld, +3.9 for STAC-D at the best checkpoint and +7.2 for STAC-P at the final checkpoint. At 7B, the key test is whether HiPER's violation rates stay high and STAC still reduces them.

AuthorsOpenReview ↗

Thank you, and a final request

Dear Reviewer 1CeY,

As the discussion period comes to a close, we wanted to thank you again for your detailed and constructive review. We hope that the revisions we made resolve your concerns about the constrained framing and the baselines. If they do, we would be grateful if you would consider raising your score. We are happy to clarify anything further before the discussion closes.

Thank you again for your time and insightful questions.

Reviewer_VvNkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • The paper proposes a solution to the temporal abstraction instability problem. It introduces STAC, a penalty applied only to the keep/switch decision, for switching too early and for keeping a stale subgoal. STAC can be plugged into existing hierarchical agents without changing rewards or any other training objective.

  • STAC is evaluated on two hierarchical agents (HiPER, GLIDER) on ALFWorld and WebShop with three small open models. It consistently improves success (e.g. about +12 points on both benchmarks for HiPER with Qwen3-0.6B) .

Critical strengths
  • The failure mode is clearly motivated. The paper shows that unmodified HiPER replaces its subgoal on more than half of the turns. The paper tracks switch rates and segment lengths throughout training rather than just looking at success rate.

  • The method is simple and easy to adopt. STAC only modifies the boundary update, so it essentially reduces to the unmodified agent when the penalty is zero, and works with two quite different agents: HiPER, which has an explicit keep/switch decision trained with PPO, and GLIDER, which expresses that decision as a true/false token at the end of each low-level action.

  • The results show clear and consistent improvement. Both agents improve on both benchmarks with all three backbones. The gains hold at the final checkpoint as well as the best one.

  • The paper discusses some limitations like that the fixed penalty doesn’t enforce any budget and that hyperparameter and checkpoint selection are coupled.

Critical weaknesses
  • Novelty is limited. Penalizing early option termination is a well-known idea in the options literature (e.g. the termination regularizer in option-critic and the deliberation cost of Harb et al. (2018)). Deciding when to switch based on predicted subgoal completion is also close to the completion token in HIPIF and GLIDER’s. The constrained-optimization framing also has minimal affect as the penalty weight is fixed in the experiments and the constraint budgets never enter the update, so the method is essentially a regularizer on the switch decision.

  • Completion labels come from hand-written rules over TextWorld facts and WebShop session states and they cover about half the decisions. The duration variant also have some priviledged knowledge on the number of subgoals in solved episodes. It is hard to seperate the gain from the boundary penalty itself and the oracles from the environments.

Questions for authors
  • Seems like a lot of the gain came from making the agent switch less often. The sensitivity study suggests that even a small penalty already helps. It would be interesting to see what will happen if you set a constant cost for switching in training, or have HiPER only switch when the probability passes a threshold at inference. Also, the paper says it measures switch rates and segment lengths for GLIDER. Does GLIDER has the same over-switching problem? Having these answers would strengthen the argument of this paper.
5 comments
AuthorsOpenReview ↗

Response to Reviewer VvNk (1/3)

We thank the reviewer for the constructive feedback, and for recognizing the importance of the problem, the simplicity of STAC, and the consistency of our empirical results. We have conducted additional experiments to address the concerns raised. We respond to each point below.

W1: Novelty is limited. Penalizing early option termination is a well-known idea in the options literature (e.g. the termination regularizer in option-critic and the deliberation cost of Harb et al. (2018)). Deciding when to switch based on predicted subgoal completion is also close to the completion token in HIPIF and GLIDER’s. The constrained-optimization framing also has minimal affect as the penalty weight is fixed in the experiments and the constraint budgets ... .

Thank you for the careful review and for pointing us to the most relevant prior work.

Novelty relative to prior work. We agree that penalizing option termination is not new. Option-Critic (Bacon et al., 2017) and the deliberation cost (Harb et al., 2018) introduce a constant margin against terminating. The STAC regularizer differs from both in two ways:

  1. It is state-dependent. In STAC-P, the cost depends on the state. Switching costs 1−pdone1-p^{\text{done}} and keeping costs pdonep^{\text{done}}, where pdonep^{\text{done}} comes from a head supervised by environment-verified completion. A constant switch penalty cannot tell "switch after the object is heated" apart from "switch halfway through heating it."
  2. It is two-sided. STAC also penalizes keeping a subgoal after it has gone stale. The short-only variant, which is the deliberation-cost-like direction, is consistently weaker than the full two-sided version: 88.2 → 91.4 (STAC-D) and 92.6 → 95.3 (STAC-P) on ALFWorld, and 67.4 → 69.3 and 71.7 → 74.2 on WebShop (Tables 3, 8).

Completion tokens in HIPIF and GLIDER. A completion decision is what STAC regularizes, not what it replaces. GLIDER already has exactly this token (<action>; true|false) and is one of our baselines. Adding STAC-P on top of it gives +12.2 on ALFWorld and +9.4 on WebShop with Qwen3-0.6B (Table 1). Table 10 isolates the mechanism. Training the same completion predictor without feeding it into boundary credit reaches AUROC 0.923 but leaves success essentially unchanged (84.1 vs. 83.3 for HiPER). Feeding it into boundary credit gives 95.3. Knowing when a subgoal is done is not enough; the gain comes from how that signal enters the boundary update.

Constrained-optimization framing. The paper says this explicitly (Sec. 3.1, App. A, App. C, Conclusion): in the reported runs, μ\mu is fixed, ρ\rho never enters the update, and the violation rates are diagnostics rather than enforced budgets. Contribution 2 already describes the method as "a decision-local, two-sided boundary regularizer." The constrained view is our design lens: it is what led us to two separately priced and measurable violation rates (Fig. 1) instead of a single switch penalty.

Your point raises a fair question: can the budgets actually be enforced? To answer it, we implemented the adaptive version of STAC during the rebuttal. Instead of holding μ\mu fixed, it adjusts μshort\mu_{\text{short}} and μstale\mu_{\text{stale}} separately with the projected dual step in Eq. 9,

μi←[μi+ημ(C^i−ρi)]+\mu_i \leftarrow \left[\mu_i + \eta_\mu (\hat{C}_i - \rho_i)\right]_{+}

Each multiplier rises while its violation rate is above budget and falls once the rate is below. Everything else is unchanged from STAC-D (ALFWorld, Qwen3-0.6B, dmin⁡=2d_{\min}=2, dmax⁡=8d_{\max}=8, 3 seeds, best-validation checkpoint). We prescribe three budget pairs and compare each against the strongest fixed μ\mu from our sweep (Table 7) that also stays within the same budget:

Budget (ρshort,ρstale)(\rho_{\text{short}}, \rho_{\text{stale}})Achieved rates (prem. / stale)Adaptive successStrongest fixed μ\mu within budgetFixed success
Relaxed (0.15, 0.12)0.142 / 0.11690.80.1091.4
Moderate (0.10, 0.08)0.096 / 0.07891.20.2090.9
Strict (0.05, 0.06)0.048 / 0.05889.60.4087.5

Three observations.

  1. The budgets hold. In every setting, the mean achieved rates land just under their budgets, which is the expected behavior when a constraint is active.
  2. Enforcement costs little in success. For the relaxed and moderate budgets, adaptive success is within 0.6 points of the matching fixed μ\mu (−0.6 and +0.3).
  3. Separate multipliers help when budgets are tight. With one shared μ\mu, the only way to bring the stale rate under 0.06 is μ=0.4\mu = 0.4. That also drives premature switching down to 0.022, far below the 0.05 actually required, and success drops. The adaptive version adjusts each multiplier on its own, so it tightens only the binding constraint and keeps 2.1 more points. With three seeds, we read this as comparable-or-better rather than a decisive gain.

We have added them to the revised paper as Appendix I.

AuthorsOpenReview ↗

Response to Reviewer VvNk (2/3)

W2: Completion labels come from hand-written rules over TextWorld facts and WebShop... .

Thank you, this is an important concern. We agree that STAC-P uses privileged supervision during training. Below we separate what comes from the boundary penalty and what comes from the oracle.

The penalty works without any oracle. STAC-D's costs depend only on the age of the active subgoal. It never reads environment state. On its own, STAC-D accounts for most of the gain:

Qwen3-0.6B (best ckpt)ALFWorldWebShop
HiPER83.361.4
+ STAC-D (no completion labels)91.4 (+8.1)69.3 (+7.9)
+ STAC-P (oracle-supervised)95.3 (+3.9 more)74.2 (+4.9 more)

STAC-D also improves GLIDER (+5.6 / +5.3) and HiPER with Llama-3.2-1B (+23.5 / +15.8) (Table 1). The part that actually depends on the oracle is the STAC-P-over-STAC-D margin.

Is the gain specific to the boundary? One could still ask whether STAC-D works simply because it adds an extra training signal of any kind. To test this, we ran a control during the rebuttal that carries no information about subgoals or completion: an episode-duration penalty (EDP). It subtracts a fixed amount from the reward at every environment step, rt′=rt−λdurr'_t = r_t - \lambda_{\mathrm{dur}}. The cost is the same whatever the subgoal, its age, or the KEEP/SWITCH choice, and it flows through HiPER's critic and returns like any other reward. We swept λdur∈{0.001,0.002,0.005,0.01,0.02}\lambda_{\mathrm{dur}} \in \{0.001, 0.002, 0.005, 0.01, 0.02\} on Qwen3-0.6B with three seeds:

Qwen3-0.6B (best ckpt)ALFWorldWebShop
HiPER83.3 ± 3.361.4 ± 3.5
EDP, full sweep (range)82.1–88.760.9–67.3
EDP, best λdur\lambda_{\mathrm{dur}} (0.005)88.7 ± 2.9 (+5.4)67.3 ± 3.1 (+5.9)
STAC-D91.4 ± 3.6 (+8.1)69.3 ± 2.4 (+7.9)
STAC-P95.3 ± 1.6 (+12.0)74.2 ± 1.1 (+12.8)

A generic per-step penalty does help when its strength is tuned separately on each benchmark. STAC-D is ahead of the best EDP setting by 2.7 / 2.0 points. Once the boundary cost becomes state-dependent (STAC-P), the gap to EDP is significant (6.6 / 6.9 points, p = 0.039 / 0.048).

The oracle labels alone don't explain STAC-P's gain. Table 10 trains the same completion predictor on the same oracle labels but does not use it in boundary credit. Its AUROC is 0.923, yet success barely moves (84.1 vs. 83.3 for HiPER). The gain appears only when the completion signal is turned into the two-sided boundary cost (95.3).

Partial coverage. The ALFWorld rules are the simulator's own goal predicates (inreceptacle, holds, isclean, ishot, iscool), read directly from TextWorld facts. The WebShop rules are exact matches on session state. The labels are used only to train the completion head; they are never a policy input or a reward. Coverage is 62.8% on ALFWorld and 41.7% on WebShop. Unlabeled boundaries receive no STAC-P cost and keep HiPER's native objective. Partial coverage therefore limits how much the oracle can contribute; it does not hide extra privileged information.

The duration thresholds in STAC-D. The lower threshold, dmin⁡=2d_{\min}=2, is definitional: a subgoal that always lasts one turn is just a primitive action. The short-only variant uses only this threshold, has no maximum, and uses no environment knowledge. It already gives 88.2 vs. 83.3 on ALFWorld and 67.4 vs. 61.4 on WebShop (Tables 3 and 8).

You are right that the upper threshold uses benchmark-level knowledge. dmax⁡d_{\max} comes from a horizon fair-share bound, dmax⁡≤⌊T/Kˉ⌋d_{\max} \le \lfloor T/\bar K \rfloor, where Kˉ\bar K is the mean number of subgoals in solved episodes: Kˉ=5.61\bar K = 5.61 over 113 ALFWorld episodes and Kˉ=4.62\bar K = 4.62 over 120 WebShop episodes. With T=50T = 50, this gives 8.91 and 10.82, so one value, dmax⁡=8d_{\max} = 8, is admissible on both benchmarks. That value is shared by all three backbones and both agents, and it is never selected by validation success.

Where it binds differs by benchmark. On ALFWorld, the late-training 90th-percentile segment length (8.3) sits at the threshold. On WebShop, it settles well below it (5.4; Fig. 4, App. F.1). The revised paper states this derivation and the Kˉ\bar K statistics explicitly (App. A.2) and lists it as a limitation in the Conclusion. The dmax⁡=∞d_{\max}=\infty endpoint is already in the paper: the short-only variant has no stale constraint, which is equivalent to dmax⁡=∞d_{\max}=\infty. It improves HiPER from 83.3 to 88.2 on ALFWorld and from 61.4 to 67.4 on WebShop. Adding the stale constraint with dmax⁡=8d_{\max}=8 (full STAC-D) gives a further +3.2 / +1.9 (91.4 / 69.3), so the upper threshold contributes but is not the main source of the gain. We did not complete a sweep over intermediate values of dmax⁡d_{\max} within the rebuttal period.

Summary. Pricing premature and stale boundary decisions works without any oracle and beats a label-free generic penalty. Grounded completion labels, where available, sharpen it further.

AuthorsOpenReview ↗

Response to Reviewer VvNk (3/3)

Questions For Authors:

Seems like a lot of the gain came from making the agent switch less often. The sensitivity study suggests that even a small penalty already helps. It would be interesting to see what will happen if you set a constant cost for switching in training, or have HiPER only switch when the probability passes a threshold at inference. Also, the paper says it measures switch rates and segment lengths for GLIDER. Does GLIDER has the same over-switching problem? Having these answers would strengthen the argument of this paper.

Thank you, these are good questions and they get at the core of our argument. We conducted all three additional experiments using Qwen3-0.6B.

Is the gain just "switch less"? Switching less is part of it, but several results suggest it is not the whole story:

  • Suppressing switching harder stops helping. In Table 7, increasing μ\mu to 0.4 reduces premature switching to 0.022, yet success drops from 91.4% to 87.5%.
  • Penalizing only early switches creates stale subgoals. On ALFWorld, the short cost alone raises stale persistence above HiPER's (0.185 vs. 0.147). Adding the stale cost lowers it to 0.098 and raises success from 88.2% to 91.4% (Table 3). On WebShop, short-only stale persistence stays slightly below HiPER's (0.095 vs. 0.110), but adding the stale cost still lowers it to 0.039 and raises success from 67.4% to 69.3% (Table 8).
  • The best variant does not minimize premature switching. STAC-P has a higher premature-switch rate than STAC-D (0.101 vs. 0.073), yet achieves higher success (95.3% vs. 91.4%).

1. Constant switching cost. We agree that this is the most direct comparison with deliberation cost. We trained HiPER with a constant penalty κ\kappa subtracted from the boundary advantage of every SWITCH decision, A^tb−κ⋅1[qt=SWITCH]\hat A^b_t - \kappa \cdot \mathbb{1}[q_t=\text{SWITCH}]. Violation rates are measured on ALFWorld.

MethodALFWorldWebShopPrematureStale
HiPER83.3 ± 3.361.4 ± 3.50.5530.147
κ=0.025\kappa=0.02585.9 ± 3.163.6 ± 3.40.3100.160
κ=0.05\kappa=0.0587.7 ± 2.965.7 ± 3.00.1700.190
κ=0.10\kappa=0.1086.8 ± 3.465.9 ± 3.20.0750.240
κ=0.20\kappa=0.2083.1 ± 3.862.4 ± 3.90.0350.310
STAC-D91.4 ± 3.669.3 ± 2.40.0730.098
STAC-P95.3 ± 1.674.2 ± 1.10.1010.061

Interestingly, κ=0.10\kappa = 0.10 achieves almost exactly the same premature-switch rate as STAC-D (0.075 vs. 0.073). However, it incurs substantially higher stale persistence (0.240 vs. 0.098) and lower success on ALFWorld (86.8% vs. 91.4%). This directly supports the importance of two-sided boundary regularization.

2. Inference-time threshold on HiPER. We evaluated trained HiPER checkpoints, allowing SWITCH only when its predicted probability exceeds τ\tau. Training is unchanged, and violation rates are measured on ALFWorld.

MethodALFWorldWebShopPrematureStale
HiPER83.3 ± 3.361.4 ± 3.50.5530.147
τ=0.50\tau=0.5083.7 ± 3.261.8 ± 3.40.5200.150
τ=0.70\tau=0.7085.2 ± 3.063.0 ± 3.30.3600.180
τ=0.90\tau=0.9080.6 ± 3.958.1 ± 4.10.1500.290
STAC-D91.4 ± 3.669.3 ± 2.40.0730.098
STAC-P95.3 ± 1.674.2 ± 1.10.1010.061

Moderate thresholding helps slightly, but aggressive thresholding increases stale persistence and hurts performance. One reason training-time credit may matter is that HiPER computes high-level returns and advantages over segments. An inference-time threshold cannot retroactively change the temporal structure experienced during training.

3. Does GLIDER over-switch? GLIDER also exhibits frequent single-action segments despite its existing completion mechanism.

EnvironmentMethodSwitch rateMean segment lengthOne-action segments
ALFWorldGLIDER0.412.354%
+ STAC-D0.273.428%
+ STAC-P0.322.937%
WebShopGLIDER0.531.864%
+ STAC-D0.362.637%
+ STAC-P0.412.345%

Notably, STAC-P switches more frequently than STAC-D in both environments, yet achieves higher success (68.8% vs. 62.2% on ALFWorld and 53.4% vs. 49.3% on WebShop).

Together, these results support our central argument: what matters is not simply switching less, but learning when to switch. We thank the reviewer for suggesting these experiments. All three are now in the revised paper (Appendix K, Tables 20–22), referenced from Secs. 5.1 and 5.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.