- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Harlo, a method for training LLM agents to solve long-horizon agentic tasks, under the Planner-Executor paradigm. Harlo uses the concept of information gain, i.e. how much the Planner's plan has influenced the actions taken by the executor, as a weighing coefficient for the advantage-based objective used to train both the Planner and the Executor. For the planner, information gain for an entire task trajectory is used as weighting to assign credit to the Planner compared to the Executor. For the Executor, information gain at each individual step is used to perform temporal credit assignment along a task trajectory.
Critical strengths
I find the idea of using information gain as a credit assignment mechanism compelling, and its application to the optimization objectives used by Harlo is simple and straightforward, thanks to the clear and concise presentation by the authors. using rather cheaply-obtainable logits (at the cost of an additional forward pass) to obtain credit assignment signal both for the Planner and the Executor is a rather elegant use of the training data.
In addition, the experimental results seem to demonstrate some improvement on baselines in most cases. In particular, the ablation experiments, while limited, show a clear contribution of each component of Harlo to the final performance of the tested models.
Critical weaknesses
The central weakness I find in the paper is that the experimental results are somewhat unconvincing. In particular, compared to the very similar EAGLET (Si et al. 2026), the improvement of Harlo is minimal (and in some cases, hard to defend as statistically significant, as the results table provides no standard deviation). Even when comparing to the simpler baseline of GRPO, it seems like the gain is much more pronounced for the smaller 1.5B model than for the 7B model, which begs the question: would larger models close the gap completely (by having more capacity to perform the task without hierarchical credit assignment, or even hierarchy at all)? This is particularly glaring since the models used are of the 2-year-old Qwen 2.5 series. I understand the compute limitations of running larger models could affect research, but newer open-weight model series (Qwen 3, 3.5; Gemma 4) have comparably small versions. Given the advances in agentic problem solving in the past two years, could a newer model trained with "flat" GRPO make Harlo obsolete?
To a lesser extent, I'd expect more extensive ablation studies on the various components described in the paper. Some examples: in the existing ablation, the third option (step CA but no agent CA) is missing; the w hyper-parameter is not ablated; step-wise CA applied to EAGLET without training the executor.
Questions for authors
-
Does the step level information gain calculation take into account the counterfactuals of previous steps? I.e. is this the likelihood of this action without the plan but with the plan in previous actions? Or is the plan removed from the entire trajectory?
-
Why keep the original A_episode as an additive term on the final advantages? This effectively weakens the effects of the proposed coefficients.
-
Have the authors attempted to compare to any other existing credit assignment schemes? Credit assignment is not a new problem in the agentic LLM era, and has been studied extensively in (hierarchical) RL. Could existing methods compare favorably?
-
At the start of Appendix A, the authors mention both Planner and Executor share the same weights. Does this mean the same base model checkpoint, but training takes different paths? Or is the same model used as both policies, only differentiated by role prompts? If it is the latter, have the authors also attempted to train separate models (or adapters)? A comparison would be interesting.
I thank the authors for the detailed responses. I went through the revised PDF, the common response, the exchange with the AC, and the replies to the other reviewers.
The failure-attribution experiment is close to what I had in mind with the bottle example. Plan-caused failures get a larger planner coefficient (0.595 vs 0.329), and the injected execution errors get a much larger executor coefficient than normal steps (0.754 vs 0.068). I think limiting the claim to a separation on average over a batch is the right call. I also appreciate that the bottle case is now stated directly in Appendix F, and that the reading of negative information gain in Section 4.2 was toned down. The new controls answer my second question as well. The constant-weight control (36.1) lands next to GRPO-PE (35.9), well below Harlo (42.2), and every shuffled variant drops too. So the assignment of the weights seems to matter, not just their scale. The step-only variant (39.0) and the clarification on loss combination and detachment cover my last two questions.
However, my main remaining concern is the one the AC and Reviewer SS3n also pointed out. Every method and every control was trained once, and all of the new controls are on ScienceWorld L2 with the 1.5B model. Gaps of a few points from single runs are hard to interpret, and variation across training runs was part of what I asked for. I understand the time constraints here. Still, results from a few more training runs, at least for Harlo, GRPO-PE and the constant-weight control, would make the empirical case much stronger.
Overall, my conceptual questions are answered and the claims now match the evidence better. Because the empirical gains still rest on single training runs, I keep my rating at 2.