- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies a distinction between symbolic subgoal success and extrinsic reward in rule-guided hierarchical reinforcement learning. The authors observe that skills can fail their intended subgoals while still producing useful rewards, making success-based intervention potentially harmful. They propose a recovery mechanism that separates semantic action judgments from execution statistics and protects skills with positive calibration returns. On Atari Kangaroo, the protected mechanism improves performance over the original rule-guided policy, while a deterministic geometry controller exactly matches the language model's behavior.
Critical strengths
-
Meaningful problem formulation. The paper identifies an important distinction between subgoal completion and task-level utility. The observation that correcting unsuccessful skills can reduce overall reward is relevant to hierarchical control.
-
Careful empirical analysis. The authors retain negative experimental results, distinguish development conditions from independent training seeds, and provide detailed action-level comparisons.
-
Appropriate control baseline. Comparing the language model against a deterministic geometry controller helps isolate what actually contributes to the performance improvements.
Critical weaknesses
-
Limited methodological novelty. The central mechanism combines a skill-success threshold, a positive-return protection rule, and a geometry-based correction policy. Separating semantic suitability from execution competence is already explored in prior work such as SayCan, as the authors acknowledge. The main addition is a heuristic that prevents intervention when the mean calibration return is positive. While useful, it is unclear whether this constitutes a sufficiently general methodological contribution beyond the specific failure mode observed in Kangaroo.
-
The claimed semantic component is not convincingly demonstrated. The language model receives explicit geometric information and route rules, and its actions exactly match those of the deterministic controller across all evaluated conditions. This suggests that the correction problem can be solved entirely through the supplied geometric rules, without additional semantic reasoning. The paper would benefit from experiments involving ambiguous or incomplete rules, where semantic applicability cannot be reduced to deterministic geometry.
-
The return-protection criterion is insufficiently justified. The method protects any skill with positive mean calibration return, regardless of its success rate or potential for improvement. However, a positive-return skill may still perform substantially worse than an alternative action. Moreover, estimates obtained from only five calibration episodes may be sensitive to limited coverage. The authors should compare the binary protection rule against stronger alternatives, such as return-aware intervention thresholds, uncertainty-aware estimates, or state-conditioned action values. This would clarify whether the observed gains depend on the proposed separation or simply on suppressing harmful interventions.
-
Limited evidence for transferability and generalization. The experiments are restricted to a single Atari game with manually specified route knowledge. Although the method is evaluated across checkpoints, training budgets, and starting delays, these conditions share the same environment and semantic rules. The earlier cross-game transfer experiments are unsuccessful. Consequently, the evidence does not establish that the proposed interface supports transferable semantic knowledge. Evaluation across environments with different rules or layouts would substantially strengthen the paper.
-
The empirical improvements are narrower than the overall framing suggests. In the development comparison, the improvement comes entirely from three Stage 2 conditions, while the other six conditions remain unchanged. The independent-seed experiments provide stronger evidence, but variability remains substantial, and the recovery mechanism does not consistently outperform a flat DQN baseline at later training budgets. The paper should more clearly distinguish improvement over a particular frozen hierarchical policy from broader advantages in reinforcement learning.
Questions for authors
-
What distinguishes the proposed semantic-execution separation from existing competence-aware or affordance-aware intervention mechanisms, beyond the positive-return protection heuristic?
-
Can the authors provide an ablation comparing the binary return-protection rule against alternative intervention criteria, while keeping the correction controller fixed?
-
Are there settings in which the language model selects a better action than the deterministic geometry controller? If not, what role does the semantic model play in the proposed framework?