acceptodds
Under review as a conference paper at ICLR 2027

Harlo: Hierarchical Agentic RL with Information-Gain Guided Credit Assignment

Abstract

Agentic reinforcement learning (RL) has advanced LLM-based agents, yet most existing algorithms are confined to single-agent settings. The Planner-Executor architecture, where a planner generates high-level plans and an executor executes step-by-step actions, offers a promising paradigm for such tasks, but efficient and stable RL methods for co-optimizing both agents remain lacking. In this paper, we propose Harlo, a hierarchical agentic RL framework that addresses the credit assignment problem through an information-theoretic approach. Our key insight is that information gain—measured as the behavioral influence of the plan on the executor's actions via counterfactual evaluation—serves as a natural signal for allocating credit. This signal enables a two-level credit assignment strategy that dynamically reshapes advantage values for both agents, ensuring stable joint optimization and co-evolution. Evaluated on ALFWorld, ScienceWorld and WebShop, Harlo consistently outperforms strong and diverse baselines. Code is available at https://anonymous.4open.science/r/Harlo-F2F5.

Primary area:
reinforcement learning
Keywords:
Information Gain, Credit Assignment, Hierarchical Reinforcement Learning, Planner-Executor Architecture

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_SS3nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Harlo, a method for training LLM agents to solve long-horizon agentic tasks, under the Planner-Executor paradigm. Harlo uses the concept of information gain, i.e. how much the Planner's plan has influenced the actions taken by the executor, as a weighing coefficient for the advantage-based objective used to train both the Planner and the Executor. For the planner, information gain for an entire task trajectory is used as weighting to assign credit to the Planner compared to the Executor. For the Executor, information gain at each individual step is used to perform temporal credit assignment along a task trajectory.

Critical strengths

I find the idea of using information gain as a credit assignment mechanism compelling, and its application to the optimization objectives used by Harlo is simple and straightforward, thanks to the clear and concise presentation by the authors. using rather cheaply-obtainable logits (at the cost of an additional forward pass) to obtain credit assignment signal both for the Planner and the Executor is a rather elegant use of the training data.

In addition, the experimental results seem to demonstrate some improvement on baselines in most cases. In particular, the ablation experiments, while limited, show a clear contribution of each component of Harlo to the final performance of the tested models.

Critical weaknesses

The central weakness I find in the paper is that the experimental results are somewhat unconvincing. In particular, compared to the very similar EAGLET (Si et al. 2026), the improvement of Harlo is minimal (and in some cases, hard to defend as statistically significant, as the results table provides no standard deviation). Even when comparing to the simpler baseline of GRPO, it seems like the gain is much more pronounced for the smaller 1.5B model than for the 7B model, which begs the question: would larger models close the gap completely (by having more capacity to perform the task without hierarchical credit assignment, or even hierarchy at all)? This is particularly glaring since the models used are of the 2-year-old Qwen 2.5 series. I understand the compute limitations of running larger models could affect research, but newer open-weight model series (Qwen 3, 3.5; Gemma 4) have comparably small versions. Given the advances in agentic problem solving in the past two years, could a newer model trained with "flat" GRPO make Harlo obsolete?

To a lesser extent, I'd expect more extensive ablation studies on the various components described in the paper. Some examples: in the existing ablation, the third option (step CA but no agent CA) is missing; the w hyper-parameter is not ablated; step-wise CA applied to EAGLET without training the executor.

Questions for authors
  1. Does the step level information gain calculation take into account the counterfactuals of previous steps? I.e. is this the likelihood of this action without the plan but with the plan in previous actions? Or is the plan removed from the entire trajectory?

  2. Why keep the original A_episode as an additive term on the final advantages? This effectively weakens the effects of the proposed coefficients.

  3. Have the authors attempted to compare to any other existing credit assignment schemes? Credit assignment is not a new problem in the agentic LLM era, and has been studied extensively in (hierarchical) RL. Could existing methods compare favorably?

  4. At the start of Appendix A, the authors mention both Planner and Executor share the same weights. Does this mean the same base model checkpoint, but training takes different paths? Or is the same model used as both policies, only differentiated by role prompts? If it is the latter, have the authors also attempted to train separate models (or adapters)? A comparison would be interesting.

Reviewer_y5Y8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Harlo, a reinforcement learning framework for planner–executor agents trained from a sparse outcome reward. The planner generates one high-level plan from the initial state, and the executor selects actions conditioned on that plan. For each executed action, Harlo compares the per-token average log-likelihood with the plan in context and after the plan is replaced by a “[No plan]” placeholder. These differences become a trajectory-level planner coefficient and step-level executor coefficients. Equations (11) and (14) multiply the group-normalized episode advantage by these coefficients, and Equations (15)–(16) add those products back onto the same advantage, with the added term weighted by w=1w=1 by default. The method is evaluated on ALFWorld, ScienceWorld, and WebShop using Qwen2.5-1.5B and 7B.

Critical strengths

Harlo scores actions the executor has already taken. The action is scored with the plan, then scored again after “[No plan]” replaces the plan. Section 4.2 says the second score is a forward pass without the plan and does not interact with the environment again. In Figure 4, Harlo takes less time per iteration than the EAGLET-style bar and more time than GRPO. The experiments cover ALFWorld, ScienceWorld, and WebShop, using the 1.5B and 7B models. On ScienceWorld, Harlo’s success rate in Table 1 is higher than EAGLET and RLVMR on L0, L1, and L2, for both model sizes.

Critical weaknesses

W1: The contribution of credit assignment to the overall gain is not fully isolated. On ScienceWorld L1 with the 1.5B model, flat GRPO is 13.7. With the planner–executor architecture retained and both credit levels removed, the result is 52.3; full Harlo is 53.1. The introduction’s “+288% over GRPO”, also shown in Table 1, compares Harlo with a model that has no planner. There is no ablation that removes only the agent-level component while retaining step-level credit assignment. The credit-assignment ablation study is reported only for ScienceWorld with the 1.5B model. Per-seed results and standard deviations are not reported in Tables 1–2.

W2: With the default w=1w=1, Eqs. (11)–(16) scale the episode advantage by 1+α1+\alpha or 1+γt1+\gamma_t, a factor between 1 and 2. The paper does not report a constant-coefficient control matched to the average advantage scale. Table 2 does not establish whether the gains come from information-gain-dependent weighting rather than a change in overall advantage magnitude. Adam, clipping, KL, and entropy are present, so a larger advantage is not the same change as a larger learning rate.

W3: Eqs. (17)–(18) use g(ρ,A)g(\rho,A) without defining it. The abstract and the conclusion say Harlo consistently outperforms the baselines. At 1.5B, ALFWorld L1 is 88.3 for Harlo and 88.6 for EAGLET. WebShop score and success are 87.0 and 73.3, against 87.7 and 73.8. Section 5.3 says the ablation gap widens with difficulty. The differences between full Harlo and the variant without either credit-assignment component are +4.0, +0.8, and +6.3 percentage points on L0, L1, and L2, respectively, rather than increasing monotonically across the splits.

Questions for authors

Q1: Tables 1–2 report averages over three seeds without standard deviations. Could you provide the per-seed results, particularly for the ScienceWorld 1.5B L1 comparison between full Harlo (53.1) and “w/o agent+step C.A.” (52.3)? Is the improvement consistent across seeds? Please also report this ablation for one additional setting from Table 1.

Q2: How are the advantages constructed in “w/o step C.A.” and “w/o agent+step C.A.”? For each removed component, is its additive term omitted, leaving the original episode advantage for the corresponding role, or replaced by a constant-weight term? Could you include a constant-coefficient control matched to the average advantage scale to separate dynamic weighting from overall magnitude?

Q3: Section 4.3 describes low information gain as discounting the planner’s credit on successful trajectories and reducing its penalty on failed trajectories. Under Eq. (15), however, the final multiplier is 1+α>11+\alpha>1 when w=1w=1. Are these descriptions relative to the original episode advantage, or to trajectories receiving larger planner coefficients?

Q4: Table 2 includes agent-level-only credit assignment (“w/o step C.A.”), but not step-level-only credit assignment. Could you report the missing variant to clarify the contribution of agent-level modulation when step-level modulation is already present?

Reviewer_6RgFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Harlo for training a planner and an executor from a shared task reward. The two roles share model parameters in the reported implementation. For each executed action, Harlo compares its average token log likelihood with and without the plan. It uses the average difference over the trajectory to weight the planner's advantage, and the local differences to weight the executor's advantages. These weights are incorporated into a GRPO update. The experiments cover ALFWorld, ScienceWorld, and WebShop with 1.5B and 7B Qwen models, along with an ablation on ScienceWorld and a runtime comparison.

Critical strengths

I like the practical motivation for the signal. It reuses actions that have already been collected, so computing the weights requires no further environment interaction or separate critic. This is useful when trajectories are expensive to obtain. The runtime comparison also suggests that the extra computation is manageable.

Critical weaknesses
  1. My main concern is the interpretation of the score as credit for the outcome. Consider a WebShop task asking for a red 500 ml bottle. Suppose the planner correctly tells the executor to check the capacity, select red, and then buy. The executor follows this plan through the search and product selection, but accidentally selects blue before checkout. The plan could still have high average information gain because it influenced most of the preceding actions. With a negative episode advantage, Harlo could then give a larger penalty to this correct plan. I do not see how the proposed weights distinguish this case from a failure caused by a plan that actually instructed the wrong purchase. The paper accounts for harmful plans by combining influence with the outcome advantage. What I am missing is evidence that this combination distinguishes the source of failure across cases like the one above. Eq. (6) also retains the history produced under the plan, so the comparison measures the plan's influence given that history. Appendix F is helpful, but its two examples do not test this ambiguity. I also found the interpretation of negative information gain in Section 4.2 too strong. A plan could reduce the likelihood of an incorrect action, which would also give a negative value if that action was sampled. These issues matter because distinguishing planning failures from execution failures is a central claim of the paper.

  2. I am not yet convinced that the experiments identify what is useful about the weighting. Table 2 is a helpful comparison because it keeps the planner and executor structure. Removing both credit components gives 58.5, 52.3, and 35.9 on ScienceWorld, versus 62.5, 53.1, and 42.2 for Harlo. The gains over this control are smaller and vary across splits, though the improvement on the hardest split is encouraging. The L2 improvement is interesting. The large gains over flat GRPO in Table 1 are harder to use as evidence for the credit mechanism, since the architecture changes as well. There is also a scale issue. Substituting the definitions into Eqs. (15) and (16), with the default weight, gives A(1+α)A(1+\alpha) for the planner and A(1+γt)A(1+\gamma_t) for each executor step. Setting both coefficients to 0.5 would simply give 1.5A1.5A everywhere. This changes the balance between the policy term and KL regularization even without using information gain. I would like to see this constant weight control, as well as a control that shuffles the coefficients while preserving their distributions. The latter would help show whether assigning the weights to the corresponding plans and actions matters. Table 2 also lacks the variant that removes only the planner weight. Only means are reported for the three runs, so it is hard to judge how reliable the smaller differences are.

Questions for authors
  1. In the bottle example above, how would Harlo distinguish a correct plan followed by an execution mistake from an incorrect plan that causes the same failed purchase? Is there a controlled diagnostic showing that the weights separate these sources of failure on average?
  2. How does Harlo compare with constant weights matched to its average advantage scale, or with the computed coefficients shuffled across trajectories and steps while preserving their distributions? Results on the harder ScienceWorld splits, with variability across runs, would help separate the effect of information gain from the change in advantage scale. These comparisons should use the same SFT initialization, prompts, and loss normalization as Harlo.
  3. Table 2 removes the executor weighting and then both weightings. How does the variant that removes only the planner weighting perform?
  4. Since the two roles share parameters, how are the losses in Eqs. (17) and (18) combined and normalized? Are the information gain scores and coefficients detached during the policy update?
2 comments

I thank the authors for the detailed responses. I went through the revised PDF, the common response, the exchange with the AC, and the replies to the other reviewers.

The failure-attribution experiment is close to what I had in mind with the bottle example. Plan-caused failures get a larger planner coefficient (0.595 vs 0.329), and the injected execution errors get a much larger executor coefficient than normal steps (0.754 vs 0.068). I think limiting the claim to a separation on average over a batch is the right call. I also appreciate that the bottle case is now stated directly in Appendix F, and that the reading of negative information gain in Section 4.2 was toned down. The new controls answer my second question as well. The constant-weight control (36.1) lands next to GRPO-PE (35.9), well below Harlo (42.2), and every shuffled variant drops too. So the assignment of the weights seems to matter, not just their scale. The step-only variant (39.0) and the clarification on loss combination and detachment cover my last two questions.

However, my main remaining concern is the one the AC and Reviewer SS3n also pointed out. Every method and every control was trained once, and all of the new controls are on ScienceWorld L2 with the 1.5B model. Gaps of a few points from single runs are hard to interpret, and variation across training runs was part of what I asked for. I understand the time constraints here. Still, results from a few more training runs, at least for Harlo, GRPO-PE and the constant-weight control, would make the empirical case much stronger.

Overall, my conceptual questions are answered and the claims now match the evidence better. Because the empirical gains still rest on single training runs, I keep my rating at 2.

AuthorsOpenReview ↗

Response to Reviewer 6RgF

We thank Reviewer 6RgF for the careful re-reading and for confirming that the failure-attribution experiment, the constant-weight and shuffled controls, the step-only variant, and the implementation clarifications address the conceptual questions. We are glad the batch-level scope and the Appendix F statement of the bottle case read as the right framing.

We fully agree with the remaining concern. All methods and controls are single training runs, and the new controls are on ScienceWorld L2 with the 1.5B model; a few-point gap from a single run cannot be read as a reliable effect size. We do not dispute this, and we have not claimed otherwise in the revised text: Sec. 5.1 and the Limitations paragraph now state the single-training-run protocol explicitly, and the abstract and conclusion have been softened to "comparable or better."

Multiple training runs for Harlo, GRPO-PE, and the constant-weight control are the right next step, but they cannot be completed within the response period under our current compute budget. We have committed them to the camera-ready and will report per-training-run means and variance for these three variants on ScienceWorld L2; if the gap does not survive, we will say so.

Reviewer_yeCjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Harlo for LLM RL within a planner-executor architecture. It estimates the planner's influence on execution by conterfactual information gain (IG), i.e. comparing the log-probabilities of execution actions with and without the plan, then uses this signal to re-weight GRPO advantages at agent/action levels. Experiments use Qwen2.5-1.5B and Qwen2.5-7B models with shared parameters and role-specific prompts, showing promising results on ALFWorld, ScienceWorld and WebShop.

Critical strengths
  1. The paper addresses an important problem: credit assignment between planning and execution under sparse reward signals.
  2. The proposed IG signal requires no additional environment rollouts, extra critic model, or manually annotated process reward. Its computation overhead is cheap. The reported per-step runtime is about 20% higher than GRPO, while the baseline EAGLET doubles the rollout cost.
  3. Experiments over 3 scenarios and 2 model scales show improvements in most settings. Appendix provides detailed training configurations, prompts, and case examples.
Critical weaknesses
  1. The connection between IG and outcome contribution is not sufficiently established. Eq(6) measures the probability difference for the same sampled action with and without the plan. The captures conditional behavioral influence rather than the plan's marginal contribution to the final reward. Although the episode-level advantage determines whether an output is reinforced or penalized, it remains unclear why the magnitude of influence reliably determines responsibility.

  2. The experiments has not distinguish credit assignment from general advantage scaling. Eq(15)-(16) can be written as $A^{final}{plan} = A{episode} (1+w\alpha), A^{final}{exe,t} = A{episode}(1+w\gamma_t) $. Under the default setting, these operations preserve the sign of original advantages and increases its magnitude.

  3. Robustness of the conterfactual input remains unclear. Replacing the plan with "[No plan] may introduce a distribution shift, causing IG to also reflect the model's response to a missing plan. The planner uses batch-standardized IG, while the executor uses raw IG; the difference requires further explain.

Questions for authors
  1. Can the authors provide more direct evidence that IG correlates with the plan's contribution to task outcomes? For example, compare reward differences between planned and unplanned rollouts on a small set of controlled tasks?

  2. Include constant-scaling and shuffled-IG controls to distinguish general re-weighting effects from the information provided by IG.

  3. How robust is the method to alternative no plan inputs or wrong plan inputs?

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revision Log / Summary of Changes

We thank all reviewers and the AC. Below is a concise log of what has been changed in the revised PDF and what is committed to the final version. All changes in the revised PDF are marked in blue; section/appendix references below point to the revised manuscript.

Changes in the Revised PDF

R1. Scope of the IG claim (C1; yeCj, 6RgF, SS3n)

  • Introduction and Sec. 4.3 now state that IG measures the magnitude of behavioral influence, not plan correctness; IG is a credit prior, and the outcome carries the sign. The separation is statistical across a batch, not per-trajectory identification.
  • New Appendix C.1 (maze diagnostic) and Appendix C.2 (failure-attribution experiment with known failure causes, 7B + 1.5B replication) directly test the assumption. Sec. 4.3 now cites the failure-attribution diagnostic.
  • The over-strong "negative IG = misleading plan" wording in Sec. 4.2 has been rewritten; the bottle example is kept as an explicit failure case in Appendix E with a scope note.

R2. Credit assignment vs. advantage scaling (C2; yeCj, 6RgF, y5Y8, SS3n)

  • Added to Table 2: constant-weight control (α≡γt≡0.5\alpha\equiv\gamma_t\equiv0.5), trajectory-/step-/both-shuffled IG controls, and the previously missing step-only variant (w/o agent C.A.).
  • Constant-weight control performs close to GRPO-PE and far below full Harlo; shuffled-IG controls collapse, especially on L2. Variant definitions are now given in Sec. 5.3; protocol is in Sec. 5.1.

R3. Counterfactual input robustness (C3; yeCj, 6RgF; AC)

  • Appendix C.3: placeholder robustness across four information-free placeholders + decomposition into plan-information SS and missing-plan response RR.
  • Appendix C.4: two wrong-plan constructions (wrong_other, wrong_shuffle) with definitions and results; we explicitly do not claim robustness to arbitrary wrong plans.
  • Sec. 4.2 now states that only the current-step plan context is removed and history is retained; the α\alpha-standardized vs. γt\gamma_t-bounded asymmetry is stated as deliberate.

R4. Seed protocol and per-seed reporting (AC; C4; y5Y8 Q1; SS3n)

  • Corrected wording throughout: one training run per method, results are means over three fixed evaluation seeds (0, 24, 42), 128 episodes per split. The earlier "three runs / without retraining" wording is withdrawn.
  • Added Appendix A.3 (per-seed Table 1, Harlo) and Appendix A.4 (per-seed Table 2, all ablation variants) with mean±std over evaluation seeds.
  • Abstract/Conclusion now say "comparable or better"; significance claims are dropped where the margin over EAGLET is within evaluation-seed variation.

R5. Implementation clarifications (6RgF Q4; SS3n Q1–Q2; y5Y8 Q3)

  • Appendix A.1: combined objective L=L(θP)+L(θE)\mathcal{L}=\mathcal{L}(\theta_P)+\mathcal{L}(\theta_E), scaling/normalization, and that all IG computations are under torch.no_grad() with coefficients detached.
  • Sec. 4.2: step-level IG retains execution history; only the current-step plan is replaced.
  • Sec. 4.4: rationale for the additive form Afinal=Aepisode+w⋅Aepisode⋅cA^{\text{final}}=A_{\text{episode}}+w\cdot A_{\text{episode}}\cdot c (sign-bearing outcome, stability), pointing to the constant-weight control.
  • Sec. 4.3: low-IG discount is now described relative to high-IG trajectories in the batch, not relative to the unmodulated advantage.
  • g(ρ,A)g(\rho,A) defined as the standard GRPO/PPO clipped surrogate after Eqs. (17)–(18).

R6. Attribution of gains and baselines (C4; y5Y8 W1/W3; SS3n)

  • The “+288% over GRPO” headline has been qualified, and the claimed improvement is now framed as a comparison against EAGLET.
  • Table 1 vs. flat GRPO no longer presented as evidence for the credit mechanism.
  • A controlled comparison against COMA/QMIX-style value decomposition and CCPO/IGPO-style counterfactual credit assignment is left to future work; Conclusion lists this as a limitation.

Committed to the Final Version

  • Qwen3.5-2B experiment under the same benchmarks/protocol (SS3n).
  • w∈{0,0.5,1,2}w\in\{0,0.5,1,2\} ablation, with w=0w=0 recovering GRPO-PE (SS3n).
  • Baseline per-seed results and mean±std for all Table 1 rows (AC; y5Y8 Q1).
  • Additional Table 1 ablation setting beyond ScienceWorld 1.5B (y5Y8 Q1).

We again thank the reviewers and AC for the constructive feedback; the revised PDF reflects the changes above, and the final version will include the committed items.

AuthorsOpenReview ↗

Summary of Rebuttal and Authors' Final Response

We sincerely thank all four reviewers (yeCj, 6RgF, y5Y8, SS3n) and our Area Chair (kQVt) for an exceptionally careful and constructive discussion. The reviews engaged deeply with our core claims, and the AC's cross-checks—notably the seed-level audit of Table 3—made the revised manuscript strictly more accurate. We are particularly grateful that even reviewers who maintained their ratings acknowledged that the new diagnostics and ablations confirm the soundness of Harlo and the role of its components. Below we summarize the resolved concerns, the remaining ones, and our commitments for the camera-ready.

Resolved Concerns

Nature of the IG signal. We now state explicitly that IG is a credit prior over magnitude, with the outcome carrying the sign, and that the separation between plan-caused and execution-caused failures holds on average across a batch, not per trajectory. A controlled failure-attribution experiment with known failure causes validates this batch-level separation, and the bottle case is now stated as an explicit failure-mode example in the appendix. The wording on negative IG has been toned down accordingly.

Advantage scaling vs. credit assignment. The new constant-weight control performs on par with GRPO-PE and far below full Harlo, while all shuffled-IG controls degrade. Together with the step-only ablation, this establishes that the assignment of IG, rather than its scale or distribution, carries the signal, and that both credit levels contribute. The additive advantage design, loss combination, and detachment of IG coefficients are now clarified.

Robustness of the counterfactual input. We added a decomposition of IG into plan-information and missing-plan-response components, robustness checks across multiple placeholders (with step-level credit allocation essentially invariant), and two wrong-plan constructions with an explicit scope statement. The asymmetry between batch-standardized agent-level and bounded step-level coefficients is now explained as deliberate.

Protocol and reporting. The protocol is now stated precisely: one training run per method, results averaged over three fixed evaluation seeds, with full per-seed tables in the appendix. The transcription error caught by the AC in Table 3 has been corrected and all per-seed cells re-verified. Claims were softened to "comparable or better" wherever margins are within evaluation-seed variation.

Remaining Concerns and Our Final Response

The central remaining concern, shared by Reviewer 6RgF, Reviewer SS3n, and the AC, is that all results rest on single training runs, so small gaps cannot yet be read as reliable effect sizes. We have stated this limitation explicitly in the paper rather than obscuring it, and we commit to reporting multiple independent training runs for Harlo and the key controls in the camera-ready, with per-run means and variance; if the gap does not survive, we will say so. Beyond this, we commit to a Qwen3.5-2B experiment under the identical protocol to address model scope, and we now discuss the closest credit-assignment schemes (value-decomposition and counterfactual methods) and the scaling behavior explicitly in the Discussion and Limitations.

Closing Remarks

We thank all four reviewers for their careful reading and constructive feedback throughout the discussion. We thank our AC for the precise cross-checks that sharpened every claim in the revised PDF. We believe the paper now states precisely what the evidence shows, no more and no less, and we will deliver every promised run in the camera-ready.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.