acceptodds
Under review as a conference paper at ICLR 2027

Separating Semantic Applicability from Execution Competence for Transferable Rule-Guided Hierarchical Reinforcement Learning

Abstract

Rules provide interpretable guidance for hierarchical reinforcement learning, but transferring them requires distinguishing whether a rule applies from whether the corresponding skill can be executed successfully. These two quantities can change independently when task descriptions or low-level policies change. We propose a framework that separates semantic applicability from execution competence in rule-guided skill selection. A semantic evaluator assesses whether observed evidence supports, contradicts, or leaves a rule’s preconditions unresolved, while a separate competence estimator learns from actual skill outcomes. A high-level selector integrates these signals with task-value estimates, retaining deterministic checks for explicit constraints and a fallback for insufficient evidence. This separation allows execution feedback to update competence estimates without treating every failed action as evidence against a rule’s applicability. We evaluate the framework under independently controlled changes in semantic descriptions and skill competence, using matched interaction budgets and shared low-level training improvements. Experiments show higher task returns and more efficient adaptation than HRL-ID and a recent strong reinforcement learning baseline. Ablations attribute these gains to the explicit separation of applicability and competence rather than to a stronger semantic model or additional training data. These findings support separating semantic and execution uncertainty as a practical principle for transferring rule-guided hierarchical policies.

Primary area:
interpretability and explainable AI
Keywords:
DRL

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hy17OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a distinction between symbolic subgoal success and extrinsic reward in rule-guided hierarchical reinforcement learning. The authors observe that skills can fail their intended subgoals while still producing useful rewards, making success-based intervention potentially harmful. They propose a recovery mechanism that separates semantic action judgments from execution statistics and protects skills with positive calibration returns. On Atari Kangaroo, the protected mechanism improves performance over the original rule-guided policy, while a deterministic geometry controller exactly matches the language model's behavior.

Critical strengths
  1. Meaningful problem formulation. The paper identifies an important distinction between subgoal completion and task-level utility. The observation that correcting unsuccessful skills can reduce overall reward is relevant to hierarchical control.

  2. Careful empirical analysis. The authors retain negative experimental results, distinguish development conditions from independent training seeds, and provide detailed action-level comparisons.

  3. Appropriate control baseline. Comparing the language model against a deterministic geometry controller helps isolate what actually contributes to the performance improvements.

Critical weaknesses
  1. Limited methodological novelty. The central mechanism combines a skill-success threshold, a positive-return protection rule, and a geometry-based correction policy. Separating semantic suitability from execution competence is already explored in prior work such as SayCan, as the authors acknowledge. The main addition is a heuristic that prevents intervention when the mean calibration return is positive. While useful, it is unclear whether this constitutes a sufficiently general methodological contribution beyond the specific failure mode observed in Kangaroo.

  2. The claimed semantic component is not convincingly demonstrated. The language model receives explicit geometric information and route rules, and its actions exactly match those of the deterministic controller across all evaluated conditions. This suggests that the correction problem can be solved entirely through the supplied geometric rules, without additional semantic reasoning. The paper would benefit from experiments involving ambiguous or incomplete rules, where semantic applicability cannot be reduced to deterministic geometry.

  3. The return-protection criterion is insufficiently justified. The method protects any skill with positive mean calibration return, regardless of its success rate or potential for improvement. However, a positive-return skill may still perform substantially worse than an alternative action. Moreover, estimates obtained from only five calibration episodes may be sensitive to limited coverage. The authors should compare the binary protection rule against stronger alternatives, such as return-aware intervention thresholds, uncertainty-aware estimates, or state-conditioned action values. This would clarify whether the observed gains depend on the proposed separation or simply on suppressing harmful interventions.

  4. Limited evidence for transferability and generalization. The experiments are restricted to a single Atari game with manually specified route knowledge. Although the method is evaluated across checkpoints, training budgets, and starting delays, these conditions share the same environment and semantic rules. The earlier cross-game transfer experiments are unsuccessful. Consequently, the evidence does not establish that the proposed interface supports transferable semantic knowledge. Evaluation across environments with different rules or layouts would substantially strengthen the paper.

  5. The empirical improvements are narrower than the overall framing suggests. In the development comparison, the improvement comes entirely from three Stage 2 conditions, while the other six conditions remain unchanged. The independent-seed experiments provide stronger evidence, but variability remains substantial, and the recovery mechanism does not consistently outperform a flat DQN baseline at later training budgets. The paper should more clearly distinguish improvement over a particular frozen hierarchical policy from broader advantages in reinforcement learning.

Questions for authors
  1. What distinguishes the proposed semantic-execution separation from existing competence-aware or affordance-aware intervention mechanisms, beyond the positive-return protection heuristic?

  2. Can the authors provide an ablation comparing the binary return-protection rule against alternative intervention criteria, while keeping the correction controller fixed?

  3. Are there settings in which the language model selects a better action than the deterministic geometry controller? If not, what role does the semantic model play in the proposed framework?

Reviewer_zaLZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates rule-guided hierarchical RL, where some skills fail their symbolic subgoals but still earn game reward. The authors propose a recovery mechanism on top of a frozen hierarchy: a gate based on calibration statistics decides when to query a Jev for a corrective action, while skills with positive calibration return are protected from intervention. The pipeline is evaluated on Atari Kangaroo only, improving mean return from 200 to 233 on nine development conditions and staying above the rule path across three training seeds, however, a deterministic geometry controller matches the LLM exactly.

Critical strengths
  1. The authors are transparent about the limitations of their approach: they report the negative E9 result, the exact match between the LLM and the geometry controller, and the failed cross-game transfer, instead of hiding them.
Critical weaknesses
  1. The algorithm was mainly benchmarked on one environment, which is Atari Kangaroo. There is other environment, Adventure, in supplement materials, but, according to appendix D, the results are not directly transferrable
  2. According to section 5.3, the DQN algorithm surpasses other recovery variants. While the authors do not claim superiority over flat learners, this raises the question of the practical value of the proposed method on its only benchmark.
  3. The deterministic geometry based controller gives exactly similar results with Jev, which is shown at Table 1. For my opinion, there is no evidence than it's more beneficial to use Jev than geometrical controller
Questions for authors

The main issues (single environment, no benefit of the LLM over the geometry controller) would require substantial new experiments rather than clarifications. I would reconsider my score only if the authors show results on additional environments and cases where Jev’s decisions differ from the geometry controller.

Reviewer_xKabOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Low skill‑success rate does not mean a rule is inapplicable. This work separates semantic applicability from execution competence for rule‑guided hierarchical RL. A recovery interface isolates semantic judgements from execution evidence, preserving productive skills while bounding corrections. Evaluated on Kangaroo, the system improves mean raw return from 200 to 233.33 across test conditions. Gains come from the recovery interface rather than special model reasoning. Diagnostics also show boundaries for rule transfer.

Critical strengths
  1. The paper highlights an important distinction: symbolic skill failure does not equal rule inapplicability, pointing out productive failure phenomena in rule‑guided RL systems.
  2. Careful ablation using deterministic geometry control properly attributes performance gain to the recovery interface instead of LLM reasoning.
Critical weaknesses
  1. Method innovation is limited. The core gating and protection heuristic are relatively straightforward heuristics, no fundamentally new hierarchical RL mechanism is proposed.
  2. Overall experiments are reproducible with logged traces. However, there are noticeable gaps: the methodological novelty is limited, experimental scope is small, and baseline comparisons are not thorough. Results are from only one game environment with few random seeds, and cross‑game transfer shows negative outcomes.
  3. Comparisons are insufficient. Baseline coverage is narrow, and there is lacking comparison against other recent neural‑symbolic HRL approaches beyond the few selected baselines. It is hard to tell how this approach stacks up against competing methods.
Questions for authors
  1. This work’s core mechanism is built on simple heuristics. Could you clarify what novel algorithmic insight your method brings compared to existing gated intervention designs?
  2. All main results come from Kangaroo. What makes you confident this interface can generalize to other environments given failed cross‑game transfer results?
  3. Why do you not include more recent neural‑symbolic HRL baselines for richer comparison? Are there implementation barriers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.