acceptodds
Withdrawn from ICLR 2027

When Do Options Help? Policy Necrosis and Redundant Coverage in Option-Critic

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Option-critic learns options: sub-policies together with a learned rule for when each one hands control back. Its headline result is that performance improves as options are added. We explain that result, with theory and experiment. First, the termination rule option-critic learns by maximising return contributes nothing. When the termination test and the policy that picks options read the same values, the test fires at every step, so the learned rule is identical to always terminating. When that policy explores and the test does not, as in option-critic itself, the rule can block the exploration; there are instances where it suffers regret while always terminating holds to . Forcing termination at every step leaves the option-count curve intact. Second, the policy inside an option barely explores at all, so a state locks onto the first action that looked good and never updates again. We name this policy necrosis, give a state-level test for it, and find three fifths of states necrotic in a typical option. Restoring exploration repairs those states, and one option then solves the task. Third, extra options improve no option; what falls is the chance that all of them fail in the same state, from to , and performance follows that joint quantity.

Primary area:
learning theory
Keywords:
Hierarchical Reinforcement Learning, Options Framework, Option-Critic, Temporal Abstraction, Exploration, Empirical Analysis, Regret Bounds

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Ca9COpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Within the option-critic framework, the paper studies the phenomenon called policy necrosis, where intra-option policies stop exploring and commit too early to an arbitrarily sampled primitive action. The phenomenon arises when the step size is large and the intra-option policy is a softmax policy with a temperature parameter that is too low. The paper identifies ε\varepsilon-greedy as a solution to this problem.

Critical strengths

None

Critical weaknesses
  1. Writing/Clarity/Structure
    1. The paper shows signs of heavy use of AI for writing. As a result, there exists sentences or phrases that are unprofessional, vague, or simply incorrect. To name a few just from the beginning of the paper:
      1. In the abstract: "the termination rule option-critic learns by maximizing return contributes nothing." This sentence is simply vague, and very likely overstated.
      2. The abstract also claims that the "headline result" of option-critic is that "performance improves as options are added." I believe this statement is simply incorrect. The main contribution of the option-critic framework is end-to-end discovery of options. The prior work is unfortunately misrepresented.
      3. Last paragraph of the introduction: "The three results compose." Unfortunately I'm not sure what it means for results to "compose".
    2. What is R\mathcal{R} in Theorem 2? I don't think i saw a definition.
  2. Limited scope
    1. The paper focuses only on four rooms for the experiments and focuses on one particular implementation of option critic with specific hyperparameter values. It is unclear whether the findings generalize to other environments/settings.

Both the scope and the level of writing in this paper falls far below what I would expect from an ICLR paper.

Questions for authors

The authors noticed that when the step size is large and the temperature for the softmax policy is low, option policies could commit too early to a random action. Before studying this phenomenon, running experiments and proposing solutions, have the authors tried hyperparameter tuning?

Reviewer_7ZxFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper considers the original option-critic paper by Bacon et al. (AAAI 2017). The authors analyze why adding options improves performance, making use of their coined term “necrosis”, which refer to the status of a state in which an option’s “necrotic” policy is locked into a poor action and can no longer learn (thus potentially missing ideal actions). This trait of low-level policies locking into non-optimal actions has been commented on in the original option-critic paper, however quite briefly. The authors dig into this further, coining the term, and claim three main findings: 1) learned termination is useless; 2) policies inside options become necrotic and do not change; and 3) that extra options help reduce redundancy rather than abstraction.

Critical strengths
  1. This paper asks an interesting and important question, which is broadly “why do more options generally help?” and does a thorough job at trying to present the answer. Ultimately, they demonstrate (in the FourRooms environment) that the benefit comes from the redundancy forged by a higher number of actions (K value), and thus less “necrotic” states.

  2. Generally gives very good transferrable lessons (i.e., keeping exploration inside options, test learned termination against a baseline that always terminates), which apply well beyond this specific setup.

Critical weaknesses
  1. Lack of support for the claims

The abstract begins with the central claim motivating a significant portion of this paper stating that the headline result of option-critic is that “performance improves as options are added”, which is not stated or proven anywhere in the original option-critic paper by Bacon et al. If this claim is made elsewhere in subsequent work, that should be clarified. This is additionally stated in the introduction “with more options the agent reaches the goal faster”. While the original option-critic paper does show in FourRooms that 8 options recovers faster than four specifically when the doorway is moved, only comparing two K values does not substantiate this broad claim. This is additionally followed up with “…and the effect grows with the difficulty of the dynamics” without proper justification or explanation. This claim needs more proof for the contribution to be meaningful (and make sure that the conclusions are right).

One of the main claims here is that “…[they] prove that [learned termination] is either vacuous or harmful.” (from line 53). However, in table 1, we can see that this is untrue. For K=2, it wins clearly in all five tabular settings and for K=4 it is mixed. A further evaluation of K values (beyond 8) would be required to confirm the findings. Additionally, it is unclear how the results position w.r.t. recent theoretical results studying the connection between the termination probability, the deliberation cost, and the model uncertainty [1]. It feels like several of these concepts may be intertwined in the presented experiments (without being ever mentioned) and that these may not allow for a proper empirical investigation. The relevance of early termination is not a property of option-critic, but something tied to the option framework itself (independently of the option learning algorithm). As such, a deliberation cost (Harb et al., 2018) is not a remedy to hold the termination probability away from 1 (as implied in the paper), but a reason (jointly with the model uncertainty) that justifies why the termination probability should be lower than 1. Moreover, the specific setting where one option is enough is the setting where the SMDP corresponds to the underlying MDP and the option is the optimal policy. That is currently presented as some sort of finding whereas it is well-known and obvious from the option framework formulation. While there are interesting things to say on the topic, the paper currently lacks the nuances along with a proper handling of the related work, which would be required to fully understand the extent and validity of the claims.

  1. The results (both experimental and theoretical) are not convincing

This paper looks solely at FourRooms (and two variations including “direct” and “inertia”). With only 103 states and short optimal paths, primitive actions can quite easily solve this. Additionally, the original paper’s setup is not included (changing the doorway location, which demonstrated that 8 options versus 4 showed improvements) so this leaves out the possibility that learned termination could matter for transfer even if it doesn’t appear to matter for a single task. In addition, the option-critic paper’s harder domains aren’t tested (pinball and ALE). The ALE setup used entropy regularization (0.01) which, to my understanding, helped deal with this “necrosis” (though they did not call it that directly, of course). This seems like it should be considered and compared directly to.

Theorem 2 shows the learned stopping rule can hurt, but it does this using a one-step problem and a two-step problem. These small examples are probably fine for showing that something can go wrong, but in problems this short, sticking with an option over several steps can’t help anyways, so these examples don’t really say anything about whether the stopping rule helps in longer tasks.

  1. Clarity

Most of the terminology is not formally defined. This includes the option framework itself with state/action/option spaces and the termination function (probability of terminating option o in state s). Additionally, assumptions on the studied option framework are not clear; these should be stated here for completeness even if reusing the same assumptions as Bacon et al. (2017).

Minor issues:

  1. Some figures (1b, 3, 4a) have text that is difficult to read as it is overlapping or tight with other parts of the figure.

[1] Lefebvre and Durand (2026), “The Cost of Commitment in Option-Based Hierarchical RL”, ICML.

Questions for authors

I don't have any questions.

Reviewer_6sQQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks why option-critic performs better with more options in tabular FourRooms, and argues that the reason is not temporal abstraction. It varies the termination rule, exploration inside options, and the number of options one at a time. Most tabular results come from exact dynamic programming on frozen checkpoints.

The paper reports that forcing termination at every step keeps the gain from more options, and its regret analysis argues that learned termination is redundant or harmful. At the original near-deterministic temperature, many states commit early to a wrong action and effectively stop learning. The paper calls this policy necrosis. Extra options do not lower single-option necrosis, but they make it unlikely that all options fail in the same state. The main empirical claim, over 350 paired seeds, is that restoring exploration inside the option lets one option solve the task and removes the benefit of extra options.

Critical strengths

The paper studies whether the gains from adding options reflect temporal abstraction or something else, a useful question in hierarchical RL. This matters for how option methods are evaluated.

The strongest part of the paper is its controls. In the matched fork, each seed's control and rescue branches start from a bit-identical checkpoint and differ only in which actions are executed. This makes the single-option repair convincing. Forcing termination at every step directly tests whether the option-count curve needs temporal extension, and the curve survives it. I also appreciate that the paper reports results that complicate its story, such as the rescue hurting slightly with eight options.

Critical weaknesses

However, I do not find the evidence sufficient to support the paper's conclusions as stated. Both my concerns look fixable, but for now I lean towards a weak rejection.

My main concern is that the title and discussion offer a general answer that the evidence does not support. All tabular results come from one map with a fixed goal, yet the discussion concludes that the case for learned temporal abstraction "has to be made somewhere else". Bacon et al. [1] designed their FourRooms experiment to test recovery after the goal moves, but training here stops before that point.

The termination claim has the same problem. Theorem 2 analyzes a UCB selector, but option-critic is ε-greedy and uses the maximum over options as its state value (lines 111 and 172). In that case, Proposition 1 makes rule (2) identical to always terminating, so Theorem 2's separation does not apply. Table 1 also shows learned termination winning on all five kernels with two options and losing on all five with eight, and every interval excludes zero. As written, the paper shows that learned termination does not produce the option-count curve, but does not convincingly demonstrate the abstract's claim that it "contributes nothing".

My second concern is that the exact evaluation differs from how it is presented. The paper promises "the same zero-exploration test policy" for every condition, but line 137 keeps the 10% random option choice at test time. With one option, this choice does nothing. With eight, it can swap a necrotic greedy pick for a viable option or the reverse. The paper does not show how much of the gain from extra options depends on that randomness. This matters because the link between joint necrosis and performance rests on that evaluation. The printed transition kernel also cannot produce the reported step counts. Both points can be checked on the saved checkpoints (Question 2).

Overall, I think the core diagnosis is valuable, and the fixes that decide my score look doable in the discussion period. The authors would need to narrow the title, abstract, and discussion to the tested setting and present Theorem 2 as an analysis of rule (2) in a bandit model. They would also need to state the kernel exactly and show that the coverage results survive exact evaluation without the random option choice. If these hold, I would raise my score to weak accept.

References:

[1] Bacon et al. (2017). The Option-Critic Architecture.

[2] Riemer et al. (2018). Learning Abstract Options.

[3] Jong et al. (2008). The Utility of Temporal Abstraction in Reinforcement Learning.

[4] Klissarov and Precup (2021). Flexible Option Learning.

[5] Kamat and Precup (2020). Diversity-Enriched Option-Critic.

Questions for authors
  1. Which conclusions are meant to hold beyond FourRooms with a fixed goal and the original temperature? Broad claims would need at least the experiment in [1] where the goal moves, and a tabular temperature sweep that includes τ = 1, the value used in [2].

  2. Could the authors state the transition kernel exactly, and confirm that training and evaluation use the same one? As printed, the Section 6 kernel moves the agent to a uniformly random free cell half the time, and the goal is one of those cells. By my calculation on the Figure 1 map, this caps every policy at about 206 expected steps, or about 685 if teleports skip the goal. The paper reports 959.5 for the one-option agent trained without exploration. If Figure 4 and Table 3 still hold under the exact kernel with the random option choice switched off, my second concern would be resolved.

  3. Can the authors clarify whether the termination update uses the maximum over options or the expectation in Appendix C.4? With the maximum, the update is zero wherever the current option is greedy, so the learned β may not reach rule (2). Could the authors also report the learned β at those states?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.