- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces HRPO, a method that integrates trajectory-level advantages with step-level progress rewards aggregated across anchor states. It further adapts entropy regularization according to historical failure statistics. Results in a custom multi-stage environment indicate better sample efficiency than trajectory-level baselines.
Critical strengths
Identifying useful intermediate actions within unsuccessful trajectories is an important challenge in reinforcement learning for long-horizon agents.
Critical weaknesses
-
Anchor states are defined using predefined stage indices, and the environment explicitly indicates whether an action advances the current stage. While stage segmentation does not itself resolve credit assignment, the supplied progress labels already provide locally productive actions, making the problem substantially easier.
-
This method maintains a value estimate () for each node, yet the subsequent advantage calculation appears to depend only on the progress indicator () and the estimated advancement probability (). The paper should clarify how () influences policy optimization and what purpose it serves in the algorithm.
-
All experiments are conducted in a custom environment with predefined stages, six discrete actions, and a small MLP policy. The absence of LLM-agent evaluations on well-known benchmarks, such as ALFWorld or AndroidWorld, limits the convinceness for the method’s applicability to long-horizon agentic RL.
-
HRPO achieves almost the same sample efficiency and final success rate as GraphGPO. Performance also remains unchanged when reflective exploration is removed. These findings do not validate an empirical benefit for the full hierarchical-reflective design beyond graph-based credit assignment.
-
Overlapping text and annotations reduce the readability of Figure 1, while the crowded layout and overlapping curves in Figure 2 make comparisons difficult. Inconsistent equation references in the overview figure introduce further confusion. Both the figures and the methodological exposition need substantial revision.
Questions for authors
-
How would anchor states be identified in tasks without predefined stages or explicit measures of progress? Would this require additional supervision or a learned progress estimator?
-
What role does the node value estimate () play in policy optimization, given that the advantage computation uses only () and ()? What would change if () were removed?