acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Reflective Policy Optimization for LongHorizon Agentic Reinforcement Learning

Abstract

Agentic reinforcement learning (RL) has emerged as the core paradigm for empowering large language model (LLM) agents to accomplish open-ended, long-horizon interactive tasks. However, existing agentic RL methods suffer from two critical bottlenecks that severely limit real-world deployment: coarse trajectory-level credit assignment that fails to capture step-wise valid reasoning in failed rollouts, and static policy optimization that cannot adapt the exploration–exploitation trade-off to dynamic task complexity. To address these fundamental limitations, we propose Hierarchical Reflective Policy Optimization (Hrpo), a novel agentic RL framework equipped with dual-level reflective credit assignment and adaptive curriculum exploration for long-horizon agent decision-making. Specifically, we first construct a trajectory state-transition graph to model hierarchical agent–environment interaction dynamics, which decomposes holistic episode rewards into finegrained step-level and sub-task-level credit signals, effectively excavating latent valuable reasoning steps obscured in incomplete or failed trajectories. Furthermore, we design a task-aware reflective exploration module that enables the agent to self-summarize historical interaction errors and successful experiences during rollouts, dynamically adjusting policy exploration intensity and avoiding redundant trial-and-error in repeated scenarios. Unlike conventional fixed-strategy RL and static agent refinement methods, Hrpo unifies experiential reflection, hierarchical credit assignment and adaptive policy update in a single end-to-end optimization pipeline, significantly improving the stability and efficiency of agent training. On a controlled long-horizon agent evaluation suite, Hrpo reaches an 80% task success rate using 88% fewer environment interactions than the group-relative baseline and attributes failures to the true faulty sub-task with 72.8% top-1 accuracy versus 12.4% for trajectory-level credit under fault injection. Ablations attribute these gains primarily to the hierarchical credit assignment; the reflective exploration component yields no measurable additional benefit in the small-scale suite—a negative result we report transparently and revisit under the large-benchmark protocol.

Primary area:
reinforcement learning
Keywords:
Hierarchical Reflective Policy Optimization, Agentic Reinforcement Learning, Large Language Model Agent, Credit Assignment, Long-horizon Task, Adaptive Exploration

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_xsXAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces HRPO, a method that integrates trajectory-level advantages with step-level progress rewards aggregated across anchor states. It further adapts entropy regularization according to historical failure statistics. Results in a custom multi-stage environment indicate better sample efficiency than trajectory-level baselines.

Critical strengths

Identifying useful intermediate actions within unsuccessful trajectories is an important challenge in reinforcement learning for long-horizon agents.

Critical weaknesses
  1. Anchor states are defined using predefined stage indices, and the environment explicitly indicates whether an action advances the current stage. While stage segmentation does not itself resolve credit assignment, the supplied progress labels already provide locally productive actions, making the problem substantially easier.

  2. This method maintains a value estimate (V^(v)\hat V(v)) for each node, yet the subsequent advantage calculation appears to depend only on the progress indicator (gtg_t) and the estimated advancement probability (p^(v)\hat p(v)). The paper should clarify how (V^(v)\hat V(v)) influences policy optimization and what purpose it serves in the algorithm.

  3. All experiments are conducted in a custom environment with predefined stages, six discrete actions, and a small MLP policy. The absence of LLM-agent evaluations on well-known benchmarks, such as ALFWorld or AndroidWorld, limits the convinceness for the method’s applicability to long-horizon agentic RL.

  4. HRPO achieves almost the same sample efficiency and final success rate as GraphGPO. Performance also remains unchanged when reflective exploration is removed. These findings do not validate an empirical benefit for the full hierarchical-reflective design beyond graph-based credit assignment.

  5. Overlapping text and annotations reduce the readability of Figure 1, while the crowded layout and overlapping curves in Figure 2 make comparisons difficult. Inconsistent equation references in the overview figure introduce further confusion. Both the figures and the methodological exposition need substantial revision.

Questions for authors
  1. How would anchor states be identified in tasks without predefined stages or explicit measures of progress? Would this require additional supervision or a learned progress estimator?

  2. What role does the node value estimate (V^(v)\hat V(v)) play in policy optimization, given that the advantage computation uses only (gtg_t) and (p^(v)\hat p(v))? What would change if (V^(v)\hat V(v)) were removed?

Reviewer_U6SdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

HRPO blends a GRPO-style trajectory advantage with a step-level advantage read off a state-transition graph (α-mixed), adds a per-state entropy coefficient scaled by failure counts (“reflective exploration”), and a success-gated curriculum. Evaluation is a synthetic tool-chain toy (6 actions, 4 contexts, 32 anchor states) with a 2×64 MLP policy.

Critical strengths

The problem framing is clear: coarse trajectory-level credit and static exploration are real bottlenecks in agentic RL, and the step-level graph advantage (Eq. 4) is a clean, cheap way to extract signal from failed rollouts.

Critical weaknesses
  1. No gain over the closest baseline. HRPO and GraphGPO are identical on every metric and every seed (Table 1, Tables 3–4: 160±0 episodes, 100% at all K). The two components that distinguish HRPO from GraphGPO — the trajectory mixing term and reflective exploration — therefore contribute nothing. The authors concede this for TRE; for α, the claim that α=0.6 “dominates” α=1 (§3.3) is contradicted by Fig. 5b, where α=1.0 is the best column (67–70%) and α=0.6 is not even swept.
  2. Headline numbers are inflated. “88% fewer interactions than GRPO” compares 160 to a mean of 1296±1754 driven by one capped seed; GRPO’s median in Table 3 is 480. The 72.8% vs 12.4% attribution result is against a uniformly random baseline (§4.3), i.e., chance by construction.
  3. No LLM, no real benchmark. The paper is framed around LLM agents (title, abstract, Fig. 1) but contains zero LLM experiments; the authors state WebArena/ALFWorld/ScienceWorld runs were not executed. A CPU toy with 32 anchor states cannot support claims about long-horizon agentic RL.
  4. Theory overclaims. The contribution list claims an unbiased policy-gradient estimator; Appendix A shows unbiasedness only for a different, shaped objective (1−α)J_group + αJ_progress. "α* strictly inside (0,1) whenever Cov > 0" is false — Eq. (13) itself yields α*=1 when C ≥ Σ_graph. And p̂(κ_t) is updated with the current batch’s g_t before the advantage is computed (Alg. 1, lines 8 vs 13), so it is not action-independent as the derivation assumes.
  5. Presentation. Fig. 1 references “Eq.(10)”/”Eq.(11)” (wrong numbers, raw LaTeX, overlapping labels); the abstract’s “self-summarized experience” is a failure counter (Eq. 8). Wrong template. Unprofessional figures.
Questions for authors

Please refer to weaknesses.

Reviewer_We2KOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

HRPO is an RL method for long-horizon agentic tasks. It groups rollouts into a graph of sub-task states and rewards each step for making more progress than is typical from that state. That step-level credit is combined with GRPO's episode-level credit. HRPO also increases exploration in states that have often led to failure, and uses a curriculum that gradually increases the number of sub-tasks. On a synthetic multi-stage tool-use task with a small MLP policy, HRPO reaches 80% success with 88% fewer episodes than GRPO and identifies the sub-task responsible for injected faults far better than episode-level credit does. It performs the same as GraphGPO on every metric, the exploration component gives no benefit, and there are no LLM experiments.

Critical strengths

1. Transparent reporting of negative results. The authors explicitly report that the reflective exploration component (TRE) yields no measurable benefit and that HRPO does not outperform GraphGPO, rather than overstating their contributions. This candor is valuable to the community and makes the paper's actual claims easy to assess.

2. Strong reproducibility. The full implementation is released, per-seed raw results are tabulated (Appendix C), all numbers are generated programmatically from logs, and the entire experimental suite runs on CPU in under an hour. Every claim can therefore be independently verified.

Critical weaknesses

1. Unfair comparison with baselines. HRPO is given privileged information that GRPO, PPO and REINFORCE do not receive: the number of sub-tasks K, the ground-truth stage of each step, and a per-step stage-advance signal, which is effectively a dense progress reward. The gain over GRPO therefore comes from a richer learning signal, not better credit assignment; the baselines should receive the same signal (e.g., as a shaped reward). The "88% fewer interactions" claim is further inflated by one GRPO and one GiGPO run that never reach the threshold and are counted at the 4,800-episode cap (Table 3); medians should be reported.

2. Poor presentation. The paper appears not to have been proofread:

  • Figure 1 has garbled labels, arrows and symbols
  • Figure 2 (right) hides the GraphGPO curve, which is identical to HRPO.
  • Figure 3: the y-axis label overlaps the plot.
  • L197: ntotn_{tot} is undefined.
  • "Hrpo" in the text vs. "HRPO" in the figures.

3. Limited novelty and missing related work. HRPO is identical to GraphGPO on every metric in Table 1, and its only effective component, graph-based step credit, comes from GraphGPO. The options framework (Sutton, Precup & Singh, 1999) is not discussed. Most importantly, HiPER (Peng et al., ICML 2026, arXiv:2602.16165) addresses the same problem, hierarchical credit assignment over sub-goals for LLM agents, with an unbiased, variance-reduced advantage estimator. Unlike HRPO, it learns its sub-goals and is validated on ALFWorld and WebShop with real LLMs.

4. Limited, artificial validation. The only testbed is a synthetic chain task with a 2-layer MLP policy. Its complexity is artificially controlled by setting K ∈ {3, 5, 8}, a knob real agentic tasks do not offer, and K = 8 defines the target for all methods. Despite the LLM-agent framing, there are no LLM experiments.

Questions for authors

How would K be chosen in practice, when the number of sub-tasks is unknown? How sensitive is HRPO to a wrong choice of K?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.