acceptodds
Under review as a conference paper at ICLR 2027

LEAH: Jointly Training LLM Agents to Act and Revise Their Harness

Abstract

Large language model (LLM) agents depend on both the model and its harness, which shapes context, tool use, and interaction with the environment. Useful harness guidance changes as execution unfolds and depends on the model's ability to use it, yet existing approaches often optimize one component while fixing the other. We propose LEAH, which jointly trains an agent to decide when and how to revise its harness during execution and to act under the resulting harness toward a shared task objective. Since revisions affect task outcomes through subsequent execution, joint learning requires distinguishing their contributions. LEAH uses hierarchical reinforcement learning with separate advantage estimates: revisions receive credit for downstream execution, while actions are evaluated relative to their harness context. LEAH achieves a 96.43% success rate on ALFWorld, a task goal completion score of 65.48 on AppWorld Test-Normal, and 76.04% on MedAgentGym, outperforming execution-only and flat joint-training baselines. Further analyses show that revising the harness during execution can improve performance without updating model weights. During joint training, revisions become more relevant and actionable, and the agent makes more task progress after revising its harness.

Primary area:
reinforcement learning
Keywords:
LLM agents, reinforcement learning, agent harness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MXkrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LEAH, a framework for jointly training LLM agents to execute tasks and revise their runtime guidance. At each environment step, the agent first decides whether to add, edit, or delete an entry in its episode-local memory, skills, or additional instructions, and then takes an environment action conditioned on the revised context. The method uses hierarchical credit assignment, with value estimates before and after revision and separate advantage signals for revision and execution decisions.

The paper evaluates LEAH on ALFWorld, AppWorld, and three selected task categories from MedAgentGym. The results show improvements over execution-only training under a fixed harness and flat joint-training baselines. Ablations and behavioral analyses further examine the effects of joint training and changes in revision quality and subsequent execution during training.

Critical strengths
  1. The paper addresses an important and practically relevant interdependence between models and harnesses. Agent performance depends jointly on model capabilities and the surrounding runtime mechanisms. Optimizing only the model or only its guidance may overlook the compatibility between these components. Treating this interdependence as a learning problem is valuable and relevant to the optimization of practical agent systems.

  2. The training method reflects the revision–execution structure and is supported by empirical evidence. The value of a guidance revision is realized through subsequent actions, making separate credit assignment for revision and execution a reasonable design. The paper reports gains across multiple interactive environments and includes comparisons with joint-training and flat-training alternatives, supporting its effectiveness within the current textual-harness setting.

Critical weaknesses
  1. The scope of the editable harness is narrow, limiting the breadth of the contribution to model–harness co-evolution. The method currently edits textual entries in episode-local memory, skills, and additional instructions. In practice, however, a harness also includes context selection and compression, history retention policies, coordination between a main agent and subagents, tool design, and execution control. These components affect information flow and action organization, beyond changing input text.

  2. The experiments focus on a relatively simple ReAct setting and do not sufficiently establish the method’s incremental value within a mature harness like codex, opencode, etc. The paper does not discuss whether LEAH consistently adds value on top of these harnesses. This directly affects the assessment of its applicability to practical agent systems.

Questions for authors
  1. Are the LLM’s modifications to the harness in LEAH limited to memory and skills? Does LEAH support broader changes to the harness, such as modifying tools, context management mechanisms, or other more complex components?

  2. How would LEAH perform if the simple ReAct-based harness were replaced with a more complex open-source harness, such as Codex or OpenCode? What kinds of modifications would LEAH make to such a harness?

  3. How is the training cost for LEAH, On-Demand Revision + GRPO, and ReAct + GRPO? I would like to see the relationship between training cost and improvement for the three.

8 comments

Thank you for the careful review and constructive feedback. Below we address your concerns regarding the scope of editable harness, applying LEAH to mature harnesses such as OpenCode and Codex, as well as the training cost.

W1. The scope of the editable harness is narrow, limiting the breadth of the contribution to model–harness co-evolution. The method currently edits textual entries in episode-local memory, skills, and additional instructions. In practice, however, a harness also includes context selection and compression, history retention policies, coordination between a main agent and subagents, tool design, and execution control. These components affect information flow and action organization, beyond changing input text.

Q1. Are the LLM's modifications to the harness in LEAH limited to memory and skills? Does LEAH support broader changes to the harness, such as modifying tools, context management mechanisms, or other more complex components?

Scope of the editable harness. Thank you for raising this point. In principle, yes: LEAH can be applied to other harness components such as tools, context management, and subagent coordination and is not tied to text. In our main experiments, we focused on the setting where revisions are limited to three kinds of textual entries as a first study, and also due to the limitation of compute resources. The formulation itself does not depend on the component. Sec. 3 requires only an editable state ctc_t, a deterministic update operator U(ct,zt)U(c_t,z_t), and a composition Ht=Hbase⊕ct\mathcal H_t=\mathcal H_{\mathrm{base}}\oplus c_t. Any harness component that fits this form, such as a history-retention setting, a context-compression rule, or the set of enabled tools, uses the same revision–execution process and the same credit assignment. Extending LEAH to such components only changes the space of revisions Z\mathcal Z (e.g., discrete settings or code rather than text), and does not change the credit assignment of LEAH.

A preliminary experiment with a non-text harness component. To demonstrate LEAH's feasibility on non-text harness components, we conducted a preliminary experiment, where the agent is allowed to edit the length of its own history window as a revision to its harness. Specifically, we add an editable runtime setting for history length, which allows the agent to change the number of previous action steps retained in its own context (choose from 1 to 10).

MethodHarness revisionsHistory lengthALFWorld success (%)
ReAct + GRPOnonefixed, 2 steps88.57
ReAct + GRPOnonefixed, 5 steps90.00
ReAct + GRPOnonefixed, 10 steps89.29
LEAHtext entriesfixed, 2 steps96.43
LEAHtext entries and history lengtheditable95.71

LEAH remains effective when its editable harness includes a non-text component. With an editable history length, it reaches 95.71%, on par with LEAH under a fixed window (96.43%) and above ReAct + GRPO with any fixed history length (at most 90.00%). Longer fixed histories help ReAct + GRPO only marginally (88.57% at 2 steps, 90.00% at 5, 89.29% at 10), so LEAH's advantage does not come from seeing more history. In this experiment, we observe that the agent decides to edit the history length in about 6% of all revisions, and these decisions help the agent to either locate previous useful information or reduce the distraction from outdated progress.

W2. The experiments focus on a relatively simple ReAct setting and do not sufficiently establish the method's incremental value within a mature harness like codex, opencode, etc. The paper does not discuss whether LEAH consistently adds value on top of these harnesses. This directly affects the assessment of its applicability to practical agent systems.

Q2. How would LEAH perform if the simple ReAct-based harness were replaced with a more complex open-source harness, such as Codex or OpenCode? What kinds of modifications would LEAH make to such a harness?

Mature harnesses such as Codex or OpenCode. Thank you. We fully agree that ReAct is a relatively simple setting, and LEAH's practicality would be substantially increased with a more complex harness system. However, due to computational constraints, we were unable to evaluate LEAH inside such harnesses. These harnesses are designed for models with long contexts and advanced abilities (larger sizes), and training under them is infeasible with our resources: our runs used 2k–4k-token contexts on 7B–9B models, while these harnesses assume more capable models with much longer contexts (e.g., OpenCode assumes a 200k-token context window by default for models without a known limit [3]). To address your concern, we give a descriptive example on how LEAH would operate under these complex harnesses.

How LEAH would operate in OpenCode.

  • Fixed base harness Hbase\mathcal H_{\mathrm{base}}. OpenCode as distributed: its system prompt, built-in tools, agent loop, and context compaction, together with the repository's own AGENTS.md.
  • Editable harness ctc_t. A set of files owned by the agent:
    • an instruction file (e.g., .opencode/AGENT.md, included through the instructions field of the configuration) for memory and instructions;
    • skill files (.opencode/skills/<id>/SKILL.md), which OpenCode lists at every step and loads on demand;
    • subagent definitions (.opencode/agents/<name>.md), each with its own prompt and permissions;
  • Deciding to revise. One can add a custom tool, revise_harness. At each step, the agent either calls an ordinary tool, which acts in the repository, or calls revise_harness, which is its decision to revise. The tool then calls the same model with LEAH's revision prompt, which writes one ADD, EDIT, DELETE, or NO_CHANGE on the editable files. The edit is applied, and the next model call runs under the revised harness. This is the same act-or-revise structure as in Fig. 2a.
  • Training. Each task runs headless (opencode run), with the policy served by vLLM through OpenCode's OpenAI-compatible provider. A proxy between the two records the tokens and log-probabilities of every model call, as in recent systems (LEGO-RL [1], Polar [2]). Calls that decide to revise and calls that write the revision are credited with ArevA^{\mathrm{rev}}; and execution calls are credited with AexecA^{\mathrm{exec}}. The reward is whether the task's tests pass.

Q3. How is the training cost for LEAH, On-Demand Revision + GRPO, and ReAct + GRPO? I would like to see the relationship between training cost and improvement for the three.

Training cost. Thank you for raising this point. We acknowledge that LEAH costs more to train than the GRPO baselines, since it involves an additional critic model. However, the comparison with flat PPO in the table below suggests that LEAH's gains do not come from adding a critic. Flat PPO trains the same on-demand revision agent with the same critic and rollout budget, but without hierarchical credit assignment, and reaches only 89.29% on ALFWorld, on par with the GRPO baselines (88.57% and 90.00%). LEAH, at almost the same cost as flat PPO, reaches 96.43%. The gain therefore comes from hierarchical credit assignment, not from the critic itself.

Specifically, on the same hardware (a single node with 4 H100 GPUs) and with the same 128 trajectories per iteration and 150 iterations for all methods, a LEAH run takes 1.33× the training time of ReAct + GRPO and 1.23× that of On-Demand Revision + GRPO. The difference comes mainly from the critic, which has the policy's size in the main LEAH run and could be smaller. A 9B critic also adds about 20 GB of peak memory per GPU.

MethodTraining time per run (h)ALFWorld success (%)
ReAct + GRPO19.2688.57
On-Demand Revision + GRPO20.8290.00
On-Demand Revision + PPO24.7589.29
LEAH, Qwen3.5-9B critic25.5996.43
LEAH, Qwen3.5-4B critic24.2395.00
LEAH, Qwen3.5-2B critic23.9193.57

Relative to ReAct + GRPO, On-Demand Revision + GRPO gains 1.43 points for 8% more training time and flat PPO 0.72 points for 29% more, while LEAH gains 7.86 points for 33% more. LEAH costs almost the same as flat PPO (25.59 vs. 24.75 hours, 3.4% more): it adds only a second scalar value head, and value evaluations at the post-revision point on the few steps with a revision. With a 2B critic, LEAH trains in 23.91 hours, less than flat PPO, and still reaches 93.57%, above all baselines. Notably, with the 9B critic, LEAH already reaches 92.86% at around iteration 115, when it has used about the same training time as a full ReAct + GRPO run (19.26 hours), surpassing the best baseline performance.

[1] Du, Yiming, et al. "LEGO-RL: Harness-native reinforcement learning for coding agents." arXiv preprint arXiv:2608.17393 (2026).

[2] Xu, Binfeng, et al. "Polar: Agentic RL on Any Harness at Scale." arXiv preprint arXiv:2605.24220 (2026).

[3] OpenCode documentation, "Models." https://opencode.ai/docs/models (accessed November 2026).

Reviewer_U6ktOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to jointly train an agent's task execution and its revision of guidance during execution. LEAH restricts the editable component to episode-local textual memory, skills, and additional instructions. At each environment step, the agent first decides whether to make an atomic edit, then takes an environment action under the resulting context. The authors define separate value functions at these two decision points and update a shared language model using coupled advantage estimation and PPO. Revisions receive credit for their downstream task returns, while execution receives credit relative to the revised context.

The agent is trained separately on ALFWorld, AppWorld, and MedAgentGym. Table 1 reports an ALFWorld success rate of 96.43%, AppWorld Test-Normal/Test-Challenge task goal completion scores of 65.48%/35.97%, and a mean success rate of 76.04% over the three selected MedAgentGym categories. The authors also compare periodic and on-demand revision, a frozen revision policy, flat GRPO, and flat PPO on ALFWorld, and analyze revision quality and subsequent task progress.

Critical strengths
  1. The training objective is well defined and has an implementable credit assignment design. Section 3 restricts revisions to traceable ADD/EDIT/DELETE operations while keeping the environment, tools, and success conditions fixed. Section 4 and Appendices D/E embed revision and execution in an alternating decision process, with two values, coupled GAE, and shared parameter updates. This provides a clear optimization target and makes it possible to inspect what guidance is produced and how it is used.
  2. The experiments examine several distinct learning questions. Table 1 reports both revision with frozen model weights and performance after training. Table 2 distinguishes a frozen revision policy from joint training. Figure 3(a) includes a flat PPO baseline, so the comparison is not limited to PPO versus GRPO. Across three environments, LEAH outperforms the reported execution-training and flat joint GRPO baselines. Revision quality is also evaluated without access to subsequent execution outcomes, avoiding the use of task success itself to score revision content. These choices support the practical value of this training direction.
Critical weaknesses
  1. The central contribution of hierarchical credit assignment is not sufficiently isolated, and result stability is unclear. By default, LEAH uses a shared backbone, two value heads, and coupled GAE over alternating nodes (Appendices D.1/D.2). This construction can also be understood as PPO conditioned on the decision phase in an augmented decision process with textual memory. Figure 3(a) already includes flat PPO, but does not specify whether that baseline observes the same pre-/post-revision states, uses comparable value-function capacity, or handles revision steps without environment rewards and GAE in the same way. It is therefore difficult to determine whether the advantage comes from the proposed credit assignment or from better baseline inputs and estimation. Tables 1/2 and the learning curves also lack results across multiple training seeds and uncertainty estimates. In particular, MedAgentGym evaluates only 96 tasks: the difference between 76.04% and the strongest RL baseline's 71.88% corresponds to approximately four tasks. This is insufficient evidence of a stable improvement and directly affects my assessment of the algorithmic contribution and the credibility of the gains.
  2. The setup does not distinguish the benefits of harness revision from those of additional memory and computation. In Appendix C.1, the agent receives only the two most recent observation-action pairs, with context limits of 2048/4096 tokens. Under such short histories, persistent textual memory alone may provide a substantial advantage. Algorithm 1 and the prompt templates also show that a revision decision is made before every environment action: even a NO CHANGE response requires a model call. Matching environment steps does not match token budgets or inference cost. The paper lacks memory/reflection baselines with comparable memory capacity and call budgets, as well as comparisons with more complete interaction histories. Thus, the results support the effectiveness of the full system, but provide less evidence for the independent benefits and practical efficiency of joint harness learning.
Questions for authors
  1. Core algorithmic comparison: Please provide flat PPO's final scores, value-network inputs and capacity, treatment of rewards and discounts at revision steps, GAE and loss normalization settings, and results across multiple training seeds. If LEAH consistently outperforms PPO with the same alternating states, value capacity, and budget, I would substantially increase my assessment of the credit assignment contribution.
  2. Source of the system's gains: How much of LEAH's advantage remains against reflection or memory-update baselines with the same persistent memory capacity and revision-call/token budget? Does the advantage persist with longer interaction histories? Please report total generated tokens, model calls, or end-to-end time for each method, in addition to environment interactions.
  3. Evaluation protocol: Does Table 1 use the checkpoint at a fixed final training step, or the best checkpoint selected from evaluations conducted every five iterations? If selection is involved, please specify the validation set and final independent test protocol. Please also provide ALFWorld results in unseen environments, or clarify that the current conclusions apply only to valid-seen. This would affect my assessment of generalization and result stability.
7 comments

Thank you for the detailed review and constructive feedback. Below we address your concerns regarding the core algorithmic comparison, source of the system gains, and evaluation protocol.

W1. The central contribution of hierarchical credit assignment is not sufficiently isolated, and result stability is unclear. By default, LEAH uses a shared backbone, two value heads, and coupled GAE over alternating nodes (Appendices D.1/D.2). This construction can also be understood as PPO conditioned on the decision phase in an augmented decision process with textual memory. Figure 3(a) already includes flat PPO, but does not specify whether that baseline observes the same pre-/post-revision states, uses comparable value-function capacity, or handles revision steps without environment rewards and GAE in the same way. It is therefore difficult to determine whether the advantage comes from the proposed credit assignment or from better baseline inputs and estimation. Tables 1/2 and the learning curves also lack results across multiple training seeds and uncertainty estimates. In particular, MedAgentGym evaluates only 96 tasks: the difference between 76.04% and the strongest RL baseline's 71.88% corresponds to approximately four tasks. This is insufficient evidence of a stable improvement and directly affects my assessment of the algorithmic contribution and the credibility of the gains.

Q1. Core algorithmic comparison: Please provide flat PPO's final scores, value-network inputs and capacity, treatment of rewards and discounts at revision steps, GAE and loss normalization settings, and results across multiple training seeds. If LEAH consistently outperforms PPO with the same alternating states, value capacity, and budget, I would substantially increase my assessment of the credit assignment contribution.

Hierarchical credit assignment versus flat PPO.

We do agree that LEAH can be viewed as a variant of PPO on the augmented decision process, in which a pre-revision and a post-revision decision point alternate within each environment step. The key difference between LEAH and PPO lies in credit assignment. Flat PPO treats every LLM generation as an individual step and estimates its advantage using a single critic, without distinguishing harness revision from environment execution. However, these decisions operate at different levels: a revision changes the harness that guides subsequent actions, while execution acts under the resulting harness. LEAH explicitly captures this hierarchy through two value heads, VrevV^{\mathrm{rev}} and VexecV^{\mathrm{exec}}, and coupled GAE. Revision receives credit for the downstream execution context it creates, while each action receives credit relative to its current harness.

Below we specify the PPO settings and comparison with multi-seed results. In short, LEAH is able to consistently outperform PPO under the same alternating states, value capacity and budget.

On-Demand Revision + PPO specification.

  • States: the same as LEAH: a value at the pre-revision point (ht,Ht)(h_t,\mathcal H_t) at every step, and at the post-revision point (ht,Ht+)(h_t,\mathcal H_t^+) after each revision.
  • Value network: the same backbone and initialization as LEAH's critic (the base model checkpoint), with one scalar head.
  • Rewards and advantage: the same task reward as in LEAH. Standard GAE runs over the alternating decision points, treating each as an ordinary time step: a revision point receives reward 0, and discount γ=0.99\gamma = 0.99 and GAE λ=0.95\lambda = 0.95. Advantages are normalized across the batch, as in LEAH.
  • Training Budget: 128 trajectories per iteration, for 150 iterations, as in LEAH.

ReAct + PPO specification. The same as On-Demand Revision + PPO, except that the agent never revises its harness: there is one decision point per environment step, at (ht,Hbase)(h_t,\mathcal H_{\mathrm{base}}).

LEAH outperforms PPO with the same alternating states, value capacity, and budget, across multiple seeds on both benchmarks: 95.95% vs. 90.23% on ALFWorld, and 78.21% vs. 70.57% on MedAgentGym. Without hierarchical credit assignment, adding harness revision barely helps PPO: On-Demand Revision + PPO is within 1.2 points of ReAct + PPO on both benchmarks. With it, LEAH gains 6.9 and 7.7 points over ReAct + PPO.

Success (%), mean ± std over 3 training seeds (MedAgentGym on the expanded 384-task set described below):

ALFWorldMedAgentGym
ReAct + GRPO89.52 ±\pm 4.3872.40 ±\pm 1.70
On-Demand Revision + GRPO90.48 ±\pm 1.8770.83 ±\pm 2.41
ReAct + PPO89.05 ±\pm 2.6370.49 ±\pm 1.49
On-Demand Revision + PPO90.23 ±\pm 3.2170.57 ±\pm 2.25
LEAH95.95 ±\pm 2.3678.21 ±\pm 2.45

Stability on MedAgentGym. We additionally expanded the evaluation to 384 tasks for better comparison. The expanded evaluation and comparison shows that LEAH still outperforms baselines on the evaluated MedAgentGym tasks.

W2. The setup does not distinguish the benefits of harness revision from those of additional memory and computation. In Appendix C.1, the agent receives only the two most recent observation-action pairs, with context limits of 2048/4096 tokens. Under such short histories, persistent textual memory alone may provide a substantial advantage. Algorithm 1 and the prompt templates also show that a revision decision is made before every environment action: even a NO CHANGE response requires a model call. Matching environment steps does not match token budgets or inference cost. The paper lacks memory/reflection baselines with comparable memory capacity and call budgets, as well as comparisons with more complete interaction histories. Thus, the results support the effectiveness of the full system, but provide less evidence for the independent benefits and practical efficiency of joint harness learning.

Q2. Source of the system's gains: How much of LEAH's advantage remains against reflection or memory-update baselines with the same persistent memory capacity and revision-call/token budget? Does the advantage persist with longer interaction histories? Please report total generated tokens, model calls, or end-to-end time for each method, in addition to environment interactions.

Memory and compute.

Clarification: how the agent decides to revise. We first clarify one point in the paper's description: the agent does not make a separate model call for revision decision before every action. Instead, its first generation at each step chooses between acting and revising (Fig. 2a), so every step takes one of two forms:

Step1st decision, under Ht\mathcal H_t (Fig. 6)2nd decision3rd decision, under Ht+\mathcal H_t^+
No revisiondecide to act, generate action ata_t (harness unchanged)——
With revisiondecide to revisegenerate revision ztz_t (ADD / EDIT / DELETE / NO_CHANGE) (Fig. 5)generate action ata_t (Fig. 7)

NO_CHANGE therefore occurs only after the agent has called revision, but ended up not changing the harness. In the original submission, App. C.2 missed one prompt template, which the agent uses for its first generation at each step (now Fig. 6 in the revised paper), and showed only the two templates used after a decision to revise: writing the revision (Fig. 5) and generating the action under the revised harness (now Fig. 7). We have added the missing template to Appendix C and updated Algorithm 1 in the revised paper to further clarify the mechanism.

Model calls. Given the above clarified mechanism, if the agent decides to directly act in the environment and not change the harness, it costs only one model call per step. If the agent decides to revise its harness, it costs three model calls per step (call revision →\rightarrow revise →\rightarrow act).

From our experiments, we find the on-demand revision of LEAH only incurs marginal inference cost, because revisions are rare relative to environment steps. On ALFWorld, the trained LEAH agent decides to revise on 54 of 1,632 steps (3.3%). Each revision adds two calls, giving 1.07 model calls per step against 1.00 for ReAct. On the ALFWorld 140-task validation set, we show the comparison of model calls, generated tokens and environment steps, in total over all 140 episodes, as follows:

Env. stepsModel callsGenerated tokens
ReAct + GRPO2,0152,01521,661
On-Demand Revision + GRPO1,8922,01824,879
LEAH1,6321,74025,161

Because LEAH succeeds more often, its episodes are also shorter: 11.7 steps on average, against 14.4 for ReAct + GRPO and 13.5 for On-Demand Revision + GRPO. LEAH therefore makes the fewest model calls of the three methods (14% fewer than ReAct + GRPO), despite its revision calls. It generates 16% more tokens than ReAct + GRPO and 1.1% more than On-Demand Revision + GRPO.

Memory capacity and history length.

Matched memory-update baseline. We believe we already have a memory-update baseline with the same persistent memory capacity and revision mechanism as LEAH, which is On-Demand Revision + GRPO: the same editable harness, the same revision operations, the same mechanism for deciding when to revise, and the same model. It differs from LEAH only in credit assignment, since revisions and actions are trained with flat GRPO. LEAH outperforms it on all three benchmarks: 96.43 vs. 90.00 on ALFWorld, 65.48 vs. 61.90 on AppWorld Test-N, and 76.04 vs. 69.79 on MedAgentGym. As the table above shows, LEAH also makes fewer model calls than this baseline (1,740 vs. 2,018) and generates about the same number of tokens (25,161 vs. 24,879), so its advantage holds at a matched call and token budget.

Longer histories. All methods in the paper use the two most recent observation–action pairs, which is the default in the agentic ReAct setting of [1], whose ALFWorld setup we follow. To test whether LEAH's advantage persists with more complete interaction histories, we trained ReAct + GRPO with 5- and 10-step history windows, with everything else unchanged. We also trained LEAH with an editable history length, where the agent can set its own window between 1 and 10 steps as a harness revision (also reported in our response to Reviewer MXkr).

MethodHistory lengthALFWorld success (%)
ReAct + GRPOfixed, 2 steps88.57
ReAct + GRPOfixed, 5 steps90.00
ReAct + GRPOfixed, 10 steps89.29
LEAHfixed, 2 steps96.43
LEAHeditable, 1–10 steps95.71

Longer histories help ReAct + GRPO only marginally, to at most 90.00%. LEAH with a 2-step window still outperforms ReAct + GRPO with a 10-step window by 7.14 points, and keeps its advantage when it controls its own history length (95.71%). LEAH's advantage therefore does not merely come from compensating for a short history window.

Q3. Evaluation protocol: Does Table 1 use the checkpoint at a fixed final training step, or the best checkpoint selected from evaluations conducted every five iterations? If selection is involved, please specify the validation set and final independent test protocol. Please also provide ALFWorld results in unseen environments, or clarify that the current conclusions apply only to valid-seen. This would affect my assessment of generalization and result stability.

Evaluation protocol.

Checkpoint selection. For each trained method, Table 1 in the paper reports the checkpoint with the best score among the evaluations run every five iterations. Selection used the same sets that Table 1 reports: ALFWorld valid-seen, AppWorld Test-N, and the 96 MedAgentGym tasks. The same rule was applied to every trained method, so the comparison between methods is matched. We additionally evaluated the selected checkpoints on tasks never used for checkpoint selection:

  • AppWorld Test-C, already in Table 1, was evaluated only on the selected checkpoints. LEAH leads there as well: 35.97, against 33.09 for On-Demand Revision + GRPO and 30.22 for ReAct + GRPO.
  • ALFWorld valid-unseen (134 tasks), new table below.
  • The 384-task expanded MedAgentGym evaluation (table in the previous comment), which shares no tasks with the 96 used for selection.

ALFWorld unseen environments. Success on valid-unseen (134 tasks), with the same checkpoints used in Table 1 in the paper:

MethodValid-unseen success (%)
ReAct + GRPO84.33
On-Demand Revision + GRPO85.82
ReAct + PPO86.57
On-Demand Revision + PPO85.07
LEAH93.28

[1] Feng, Lang, et al. "Group-in-group policy optimization for llm agent training." Advances in Neural Information Processing Systems 38 (2026): 46375-46408.

Reviewer_jGw2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LEAH treats an LLM agent as a policy plus an episode-local harness of memory, skills, and additional instructions. At each step, one shared LLM either leaves the harness unchanged or applies one atomic ADD, EDIT, or DELETE, then acts under the updated harness. Edits persist for the episode and are cleared afterward. The objective is task return. The revision advantage is the value difference before and after the edit, and the action advantage is the one-step temporal-difference residual relative to the post-revision value. Both are optimized with coupled GAE and a PPO-style update. On Qwen3.5-9B, LEAH reaches 96.43% on the ALFWorld seen split and task-goal completion of 65.48 / 35.97 on AppWorld Test-Normal / Test-Challenge. On Qwen2.5-7B-Instruct, it reaches 76.04% average success on three MedAgentGym categories. The learned baselines are GRPO with a fixed harness, and GRPO on the same revision interface without separate revision and action advantages.

Critical strengths

The on-demand harness revision design is well motivated. Useful guidance changes as execution proceeds, and a harness fixed in advance cannot cover these situations. LEAH lets the model decide at each step, from the interaction history, whether to revise and how, applies one atomic edit, and then continues acting under the revised harness. The effectiveness of this design is validated on both ALFWorld and AppWorld Test-Normal.

Critical weaknesses
  1. Table 2 shows that when On-demand Revision optimizes only the action decision, the success rate drops substantially, to 83.57%, below ReAct at 88.57%. This revision model has not been trained for revision, yet it freely rewrites the harness, so the drop is mixed with interference from low-quality revisions. A more reasonable baseline would generate revisions with ReAct feedback as the context input, and train only the action policy. This baseline is missing. The gap from 83.57% to the jointly trained 90.00% cannot by itself demonstrate the gain from joint training.

  2. Line 426 says that LEAH is also competitive with stronger closed-source models. This holds in the current ALFWorld setting, 96.43 versus 90.95 for GPT-5.6-Sol ReAct. It does not hold on AppWorld Test-Normal, where 65.48 is below 71.43 and 73.21. It does not hold on MedAgentGym, where 76.04 equals GPT-5.4 with on-demand revision and is slightly below GPT-5.6-Sol ReAct at 77.08.

Questions for authors
  1. The naming of the credit assignment is inconsistent. Section 4.2 defines coupled credit assignment, while Section 4.3 calls the same AtrevA^{rev}_t, AtexecA^{exec}_t hierarchical credit signals, without explaining why coupled constitutes hierarchical. This wording is also inconsistent with the contribution list, Figure 2, and Appendix D.1.

  2. The critic uses a separate set of parameters, a backbone plus two scalar heads, but the paper does not describe the critic architecture and parameter scale clearly enough, nor whether it shares parameters with the agent model.

  3. In trained LEAH, what fraction of steps are NO CHANGE, ADD, EDIT, and DELETE? If the proportion of effective revisions is substantial, that statistic would better demonstrate the necessity of on-demand harness revision.

8 comments

Thank you for the careful review and the concrete suggestions. Below we address your concerns regarding additional baseline, wording choices, critic architecture and revision frequency.

W1. Table 2 shows that when On-demand Revision optimizes only the action decision, the success rate drops substantially, to 83.57%, below ReAct at 88.57%. This revision model has not been trained for revision, yet it freely rewrites the harness, so the drop is mixed with interference from low-quality revisions. A more reasonable baseline would generate revisions with ReAct feedback as the context input, and train only the action policy. This baseline is missing. The gap from 83.57% to the jointly trained 90.00% cannot by itself demonstrate the gain from joint training.

Thank you for this suggestion. We first clarify the existing setting and introduce a stronger comparison in which the revisions are trained but frozen.

Our setting. In LEAH, revisions are already conditioned on ReAct-style environment feedback: the revision is generated from the task, recent interaction history (including environment observations), and current harness. All on-demand variants in Table 2 use this same context. In the 83.57% baseline, revision and execution are parameterized separately: revisions are generated by a frozen copy of the base model, while only execution is trained with GRPO. Thus, this baseline already follows the suggested setting to "generate revisions with ReAct feedback as the context input", but we do agree that untrained revisions may interfere with execution.

A stronger baseline with trained but frozen revision. To address this concern, we introduce an additional baseline in which revisions are generated by the jointly trained On-Demand Revision + GRPO model (90.00%), kept frozen, while a separate execution policy is initialized from the base model and trained with GRPO under the same budget as the 83.57% baseline.

MethodRevisionOptimized decisionsALFWorld success (%)
ReAct + GRPONoneAct88.57
On-Demand Revision (Table 2)Frozen base modelAct83.57
On-Demand Revision (new)Frozen trained modelAct87.86
On-Demand Revision + GRPOJointly trainedAct + Revision90.00
LEAHJointly trainedAct + Revision96.43

Results. Replacing the base model's revisions with trained ones raises success from 83.57% to 87.86%, yet this is still no better than ReAct + GRPO (88.57%), which trains execution without any revision. This suggests that when only execution is trained, even revisions of trained quality do not help. LEAH outperforms this stronger baseline by 8.57 points (96.43% vs. 87.86%, 12 of the 140 tasks). Because both methods use revisions of trained quality, this gap reflects the gain from training revision and execution jointly. Jointly trained On-Demand Revision + GRPO (90.00%) also scores above this baseline, consistent with the same benefit of joint training.

W2. Line 426 says that LEAH is also competitive with stronger closed-source models. This holds in the current ALFWorld setting, 96.43 versus 90.95 for GPT-5.6-Sol ReAct. It does not hold on AppWorld Test-Normal, where 65.48 is below 71.43 and 73.21. It does not hold on MedAgentGym, where 76.04 equals GPT-5.4 with on-demand revision and is slightly below GPT-5.6-Sol ReAct at 77.08.

Thank you. We agree that the description was imprecise. We have replaced the sentence with a more precise statement in the revised paper:

"LEAH outperforms every closed-source model we evaluated on ALFWorld and is about 1 point behind the strongest closed-source model on MedAgentGym. On AppWorld Test-N, it substantially narrows the gap between its base model and the strongest closed-source result from 51.2 to 7.7 points."

Q1. The naming of the credit assignment is inconsistent. Section 4.2 defines coupled credit assignment, while Section 4.3 calls the same AtrevA^{rev}_t, AtexecA^{exec}_t hierarchical credit signals, without explaining why coupled constitutes hierarchical. This wording is also inconsistent with the contribution list, Figure 2, and Appendix D.1.

"Coupled" vs. "Hierarchical" credit assignment. Thank you for the suggestion. The two words describe different aspects of the same method.

  • Hierarchical describes the policy and credit structure. An applied harness revision (high-level) persists and conditions all later actions (low-level) until it is changed. A revision is therefore credited through the execution context it creates, Atrev=Vexec(ht,Ht+)−Vrev(ht,Ht)A^{\mathrm{rev}}_t = V^{\mathrm{exec}}(h_t,\mathcal H_t^+) - V^{\mathrm{rev}}(h_t,\mathcal H_t), while each action is credited relative to the harness it was taken under.
  • Coupled describes how the two credit signals depend on each other. AtrevA^{\mathrm{rev}}_t is defined by the post-revision value VexecV^{\mathrm{exec}}, and AtexecA^{\mathrm{exec}}_t bootstraps from the next pre-revision value VrevV^{\mathrm{rev}}, so each signal is defined through the other decision point's value function. Their GAE estimates are correspondingly mutually recursive (App. D.1).

We have now revised the paper to use them consistently. In the revision, Sec. 4.2 is renamed "Hierarchical Credit Assignment" and states this distinction. "Hierarchical" names the method throughout, matching the contribution list and Figure 2, while "coupled" refers only to the dependence between the two signals and the resulting estimator (coupled GAE, App. D.1).

Q2. The critic uses a separate set of parameters, a backbone plus two scalar heads, but the paper does not describe the critic architecture and parameter scale clearly enough, nor whether it shares parameters with the agent model.

Critic architecture. The critic is a separate model that does not share parameters with the policy. The critic backbone has the same architecture as the policy and is initialized from the same checkpoint: Qwen3.5-9B for ALFWorld and AppWorld, Qwen2.5-7B-Instruct for MedAgentGym. On top of it sit two scalar value heads: VrevV^{\mathrm{rev}} at the pre-revision point (ht,Ht)(h_t,\mathcal H_t) and VexecV^{\mathrm{exec}} at the post-revision point (ht,Ht+)(h_t,\mathcal H_t^+). The critic therefore has the same parameter scale as the policy (about 9B and 7B parameters), and the second head adds only one scalar output layer. We also note that, although in our main experiments, the critics are of the same size as the policies, using smaller critics only slightly degrades the performance of LEAH. As shown in the following table, LEAH's performance with even the 2B critic beats every baseline by at least 3.57 points.

MethodCriticALFWorld success (%)
ReAct + GRPOnone88.57
On-Demand Revision + GRPOnone90.00
ReAct + PPOQwen3.5-9B86.43
On-Demand Revision + PPOQwen3.5-9B89.29
LEAHQwen3.5-9B96.43
LEAHQwen3.5-4B95.00
LEAHQwen3.5-2B93.57

Q3. In trained LEAH, what fraction of steps are NO CHANGE, ADD, EDIT, and DELETE? If the proportion of effective revisions is substantial, that statistic would better demonstrate the necessity of on-demand harness revision.

Clarification: how the agent decides to revise. We first clarify one point in the paper's description: the agent does not make a separate model call for revision decision before every action. Instead, its first generation at each step chooses between acting and revising (Fig. 2a), so every step takes one of two forms:

Step1st decision, under Ht\mathcal H_t (Fig. 6)2nd decision3rd decision, under Ht+\mathcal H_t^+
No revisiondecide to act, generate action ata_t (harness unchanged)——
With revisiondecide to revisegenerate revision ztz_t (ADD / EDIT / DELETE / NO_CHANGE) (Fig. 5)generate action ata_t (Fig. 7)

NO_CHANGE therefore occurs only after the agent has called revision, but ended up not changing the harness. In the original submission, App. C.2 missed one prompt template, which the agent uses for its first generation at each step (now Fig. 6 in the revised paper), and showed only the two templates used after a decision to revise: writing the revision (Fig. 5) and generating the action under the revised harness (now Fig. 7). We have added the missing template to Appendix C and updated Algorithm 1 in the revised paper to further clarify the mechanism.

Frequency and utility of harness revisions. Given the clarified mechanism above, we report the revision statistics of trained LEAH on ALFWorld (140 validation tasks, 1,632 environment steps, and 96.43% success rate).

Step outcomeCount% of all steps% of revision attempts
Direct action1,57896.69%—
Revision → ADD462.82%85.19%
Revision → EDIT70.43%12.96%
Revision → DELETE00.00%0.00%
Revision → NO_CHANGE10.06%1.85%

LEAH revises selectively. It decides to revise on only 54 of the 1,632 steps (3.31%), across 44 of the 140 episodes (31.43%). Revision is thus rare at the step level, yet a regular part of how LEAH solves tasks: it acts directly almost all the time, and revises in about a third of the episodes. When it does decide to revise, it almost always has a concrete change to make: 53 of the 54 attempts (98.15%) actually modify the harness, with only one NO_CHANGE. Its decisions to revise are therefore almost never wasted.

These results highlight LEAH's on-demand revision behavior. Rather than revising at every step, LEAH predominantly acts directly. Since a harness modification persists, a single revision shapes every subsequent action in its episode. A low revision rate is therefore what on-demand revision is designed to produce: only a few revisions, each with a lasting effect on execution.

Revisions also become increasingly useful over training. As shown in Sec. 5.3, LEAH's revisions are increasingly judged qualified by an LLM evaluator (Fig. 4c), and more task milestones are completed within five actions following a revision (Fig. 4d). These findings provide complementary evidence that LEAH learns not only when to revise, but also how to produce useful revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.