acceptodds
Under review as a conference paper at ICLR 2027

SimpleHRL: Minimal Hierarchical Structure Improves Out-of-Distribution Generalization in RL Post-Training of LLMs

Abstract

Reinforcement learning (RL) post-training has become a central paradigm for improving LLM reasoning, yet how the structure of the reasoning trace during training affects out-of-distribution (OOD) transfer remains poorly understood. Hierarchical RL (HRL) offers a natural inductive bias-separating what to do from how to do it-intended to encourage abstract, transferable reasoning, but HRL has been studied almost entirely outside the LLM RL post-training regime, leaving open whether its structural prior alone is enough to deliver OOD gains in LLMs. We introduce SimpleHRL, a deliberately minimal, HRL-inspired structural prior for LLM RL post-training: it imposes a hierarchical plan/execution structure on the reasoning trace during training, with no separate critic, no multi-level reward shaping, and no objectives beyond what flat baselines already use. Despite this minimality, when trained only on mathematics, SimpleHRL improves average OOD Pass@32 by  pp and Pass@1 by  pp over GRPO+CoT and entropy-exploration baselines across a broad OOD benchmark. To understand this effect, we conduct a series of mechanistic analyses and reveal two complementary insights into how hierarchical structure helps: SimpleHRL's plan-level reasoning becomes markedly more task-agnostic, supporting broader and more transferable strategy coverage; and hierarchical training concentrates gradient updates on the pivotal decision tokens that most determine outcome quality. Together, these findings argue for treating the structure of the reasoning trace during training, beyond surface-level CoT formatting, as a productive lever for OOD generalization in LLM post-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training, LLM, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_29C3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SimpleHRL, which uses a prompt-induced plan/execution structure during GRPO training to improve OOD generalization. The authors report gains on cross-domain benchmarks and attribute them to more transferable reasoning strategies and improved credit assignment.

Critical strengths
  • The method is simple and easy to implement.
  • The paper evaluates OOD generalization across multiple task domains and includes mechanistic analyses.
  • This paper is easy to follow and overall well-presented
Critical weaknesses
  • SimpleHRL generates substantially longer responses (2,531 vs. 1,525 tokens). The long-CoT baseline only increases length at inference time, which does not control for the effects of longer trajectories during RL training. Length-matched training, compute-matched comparisons, and truncation statistics are needed.
  • More task-agnostic plan embeddings may simply reflect the generic language enforced by the prompt, rather than genuinely transferable reasoning strategies. The paper lacks interventions demonstrating that these strategies causally improve OOD performance.
  • Experiments use only Qwen3-4B and Qwen3-8B. The results may depend on these models' existing structured-reasoning capabilities. Experiments on more model families would be helpful.
Questions for authors

Please see weaknesses

Reviewer_nrNtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SimpleHRL, a minimal hierarchical reasoning structure for reinforcement learning post-training of LLMs. Rather than introducing a separate high-level policy, critic, or hierarchical reward, the method uses a structured prompt that requires the model to articulate a high-level plan before each execution step, while retaining the standard GRPO objective with DAPO-style asymmetric clipping.

The authors train Qwen3-4B and Qwen3-8B exclusively on mathematical reasoning data and evaluate them on in-distribution mathematics and ten out-of-distribution benchmarks covering coding, logic, science, and table reasoning. Compared with GRPO+CoT, SimpleHRL improves average OOD Pass@32 by 3.7 and 4.2 percentage points at the two model scales, respectively.

The paper further analyzes reasoning-pattern transitions, semantic similarity of planning steps, and token pivotality. The authors hypothesize that hierarchical structure encourages more transferable high-level strategies and more informative gradient allocation.

Critical strengths
  1. SimpleHRL demonstrates that explicit plan/execution formatting during RL post-training can improve performance on several OOD tasks without changing the optimization objective or adding a separate planning network. The improvements are observed at both 4B and 8B scales, particularly on MBPP, Zebra Puzzle, and CodeIO. This makes the approach practically interesting despite its simplicity.

  2. The authors investigate whether the improvements can be obtained simply by applying the hierarchical prompt at inference time. The comparison with untrained models and the inference-only prompt baseline provides useful evidence that the performance advantage is associated with training under the structured format, rather than solely with inference-time prompting.

Critical weaknesses

The proposed method is essentially GRPO training with a fixed structured reasoning prompt. It does not introduce a hierarchical policy, temporal abstraction mechanism, or hierarchical credit-assignment objective in the conventional HRL sense.

Previous work, including HICRA (Wang et al., 2025), PTA-GRPO (Dou et al., 2026), Multi-Level Reasoning (Xiong et al., 2026), and HiPER (Peng et al., 2026), has already explored explicit high-level planning and low-level execution in LLM reasoning and RL. Although these methods generally involve additional training machinery, the present paper does not sufficiently establish how its contribution advances beyond simplifying existing planning-guided approaches.

The most defensible novelty is therefore the empirical finding that a prompt-induced plan/execution structure alone may suffice to improve OOD performance. This is potentially useful, but constitutes a relatively narrow methodological contribution.

The SimpleHRL prompt differs from the flat baseline not only through hierarchical structure but also through instruction complexity and a worked example. Moreover, SimpleHRL produces substantially longer responses (2,531 versus 1,525 tokens). The long-CoT control is applied only at inference time and does not match the training-time sequence-length distribution. Consequently, the experiments do not establish that hierarchy, rather than other properties of the structured training trajectories, explains the improvement.

The mechanistic analyses have related identification problems. Higher similarity among plan-only embeddings could arise from short, generic planning statements rather than broader coverage of transferable strategies. Cross-task similarity alone does not establish strategy diversity.

The pivotal-token analysis is also less conclusive than claimed. Figure 3 shows that randomly selected tokens under the SimpleHRL prompt have substantially higher pivotality than those under the CoT prompt. Furthermore, the plotted GRPO+CoT values for gradient-norm and entropy selection do not appear to exceed the random baseline, contrary to the discussion in Section 5.2. Since this experiment uses the same underlying checkpoint under different prompts, it primarily measures a prompt-conditioned difference and does not directly establish improved credit assignment arising from hierarchical training.

These issues weaken the proposed mechanistic explanation, even if the reported OOD performance improvements themselves are valid.

Questions for authors
  1. Could the authors compare against a training-time control with similar prompt length, worked examples, response length, and token budget, but without the explicit plan/execution hierarchy? The existing inference-only long-CoT experiment does not isolate this factor during training. This comparison would materially affect my assessment of the central contribution.

  2. The reported embedding metrics demonstrate greater cross-task similarity, but do not necessarily imply broader strategic coverage. Could the authors analyze distinct strategy types or control for repeated planning templates and step length? This would help establish whether the planning layer captures reusable decision-making strategies rather than generic descriptions.

  3. In Figure 3, random token selection under SimpleHRL already yields relatively high pivotality, while the GRPO+CoT gradient and entropy selection results appear comparable to or below random selection. How should these observations be reconciled with the claim that gradient norm is a better pivotality predictor than entropy and random selection under both prompts? Could the authors also demonstrate this relationship on the trained policies and report the improvement relative to each prompt's random-token baseline?

Reviewer_ey2MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SimpleHRL trains an LLM to write a short plan before each reasoning step. The experiments cover a range of reasoning tasks and show encouraging OOD gains while keeping math performance broadly similar to the flat baselines. The ablations test several possible explanations for these gains.

Critical strengths
  1. A simple training change improves OOD results while keeping mathematics performance similar.
  2. The method needs no extra policy or reward model.
Critical weaknesses

W1: The controls do not rule out extra computation as a cause of the gains. The inference-only control increases response length much less than SimpleHRL does. W2: The paper lacks a causal graph with testable assumptions for its explanation of OOD gains. Its graphs describe reasoning flow, and the paper acknowledges that its gradient analysis does not establish causation.

Questions for authors
  1. Does the OOD gain persist when the GRPO+CoT-trained model uses the hierarchical prompt at a matched token budget?
  2. Can you state the causal assumptions behind the proposed transfer mechanism using a graph or structural model, as in [1,2]?
  3. Does the task-agnosticity result hold when baseline steps are summarized as briefly as SimpleHRL's plans?

References [1] Rojas-Carulla, Schölkopf, Turner, and Peters (2018). Invariant Models for Causal Transfer Learning. JMLR 19(36):1–34. [2] Mahajan, Tople, and Sharma (2021). Domain Generalization using Causal Matching. ICML, PMLR 139:7313–7324.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.