acceptodds
Under review as a conference paper at ICLR 2027

JHiMA-WebRL: A Joint Hierarchical Multi-Agent Reinforcement Learning Framework for Long-Horizon Web Tasks

Abstract

Multimodal web agents have shown promising capabilities for interacting with websites through graphical user interfaces (GUIs), yet realistic web tasks often require long-horizon, multi-step planning that remains challenging for monolithic policies. Such agents must jointly infer high-level plans, navigate through intermediate states, track progress over extended interaction histories, and execute visually grounded actions. Although prior work has explored modular multi-agent systems with a separate planner, the planner is typically not updated jointly with the actor. We investigate whether explicitly separating and jointly training planning and execution provides a stronger inductive bias for long-horizon web navigation. We introduce JHiMA-WebRL, a joint hierarchical multi-agent reinforcement learning framework for web agents that couples a task decomposer with an actor. To train the decomposer, we construct structured subtask supervision from multimodal GUI trajectories using a strong teacher model and use supervised fine-tuning to train both the decomposer and actor on trajectories augmented with generated subtasks. We then extend Group Relative Policy Optimization (GRPO) to jointly optimize the decomposer and actor in the interactive WebArena environment through hierarchical rollout grouping and role-specific advantages. On WebArena-Lite-v2, JHiMA-WebRL-14B, consisting of a 7B actor and a 7B decomposer, achieves a state-of-the-art success rate of 37.2%, exceeding the strongest reported baseline by 8.6 percentage points and surpassing larger 32B and 72B monolithic agents. These results suggest that jointly learned role factorization with smaller models can be a competitive alternative to scaling monolithic agents for long-horizon web tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
web agents, multi agents, grpo, hierarchical RL, multimodal agents, long horizon planning, task decomposer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aBzqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Real world web tasks require planning over long sequences of steps. Agents need to infer high level plans, navigate intermediate states, track progress, and perform visual actions. A single agent struggles to handle all these abilities at once. Existing modular multi agent systems separate planning from execution, but they usually lack joint training. To address this problem, this paper proposes JHiMA-WebRL, which explicitly separates a task decomposer from an executor. It first fine tunes both roles using subtask supervision from a teacher model, then jointly optimizes them through hierarchical reinforcement learning. The method explores how joint learning of planning and execution helps with long horizon web navigation.

Critical strengths
  1. The hierarchical design fits the problem well. Separating task decomposition from action execution allows each role to focus on a specific function and provides a reasonable structural bias for long horizon web tasks.
  2. The joint optimization method is novel. It extends GRPO through hierarchical rollout grouping and role specific advantage estimation, allowing the planner and executor to train together.
  3. The construction of subtask supervision has practical value. A strong teacher model generates structured subtasks from multimodal GUI trajectories, providing clear training signals for supervised fine tuning of both roles.
Critical weaknesses
  1. The baseline scores differ from those in the original paper. Table 1 reports scores of 23.9 and 24.0 for ScaleCUA-7B and ScaleCUA-32B, respectively, while Table 5 in the original ScaleCUA paper reports 37.7 and 44.2. Since the original ScaleCUA scores are close to or higher than those of the proposed method, the authors need to further explain the reasons for this performance gap.
  2. The evaluation covers only WebArena. This does not show generalization across datasets or effectiveness on different benchmarks. Please include more benchmarks, such as WebVoyager or Mind2Web2, to demonstrate the generality and generalization ability of the method.
  3. The title emphasizes long horizon tasks, but the experiments do not directly show that the advantage grows as tasks become longer. The interaction limit is 15 steps for both training and evaluation. The paper does not group tasks by the number of steps they require or compare performance, error accumulation, and recovery ability under larger step budgets. This setting differs greatly from current long horizon settings. For example, OSWorld 2.0 uses more than 100 steps.
  4. It remains unclear whether the framework includes replanning. If the planner works only once at the start, the system lacks a feedback loop from execution to planning. If the planner updates the plan at every step, please clearly emphasize this in the paper, as replanning is an important part of hierarchical planning.
Questions for authors

See above.

Reviewer_tZKgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a novel training technique for language models used in long-horizon web tasks - they employ a two level GRPO training, where a planner / decomposer models is trained alongside an executor model. They claim that this two level training is better than training a monolithic model for web agents. On WebArena-Lite-v2, JHiMA-WebRL-14B, w/ a 7B actor and a 7B decomposer, achieves a success rate of 37.2% (compared to using a GPT-4o Planner + UI-TARS-1.5-7B/ScaleCUA Grounder that scores 28.6% on WebArena)

Critical strengths
  1. Proposes a novel two-stage GRPO training pipeline, where a planner and executor model are trained simultaneously.
  2. Paper is well presented and easy to follow. The information presented in the paper is comprehensive, even when the results like ablations in Table 2 weaken the headline result.
Critical weaknesses
  1. Training/Eval overlap: The JHiMA models are evaluated on 165 WebArena Lite v2 tasks and uses the remaining 812 - 165 tasks as training environments for running the two level GRPO training. This benchmark has templates that are used to create multiple tasks, hence there is likely a lot of overlap between the training and the test tasks.
  2. Invalid Benchmarks / Not enough improvement over comparable benchmarks: Other baselines ScaleCUA and UITars did not use WebArena environments in their training, hence it is not an apples to apples comparison when they are compared with JhiMA. Table 2 on the other hand, shows a valid baselines, since all these trained models share the same training and eval settings. Here we compare relevant baselines like actor-only RL and SFT-only training. But we find that the performance of JhiMA-WebRL (37.2% SR) is only modestly higher than the actor only RL (35.2%) and the SFT-only baselines (35.1%). This weakens the overall result of the paper.
  3. Other baselines are not compute matched: The paper does not show if Jhima-WebRL will still outperform SFT-only or actor-only training if the trainings were compute matched across all settings.
  4. Generalizability across other benchmarks - Does the performance of JhiMA models generalize to other web agentic benchmarks (e.g, OSWorld, AssistantBench, TravelPlanner etc.)
Questions for authors
  1. How does GPT-4o Planner + ScaleCUA-7B Grounder compare to Qwen-7B Planner + ScaleCUA-7B Grounder and Qwen-7B Planner + JHIMA's Actor as Grounder?
  2. L435: Where did we get the reward functions for assessing the successful completion of subgoals?
  3. How many eval tasks require subgoals that cannot be specified from the initial screenshot alone. - ie information to plan further requires information which can only be achieved after executing an action - and how does the method perform on that subset relative to the actor-only variant? Does this explain the Map regression? Does the selection of evaluation benchmark affect the usefulness of JHiMA's training algo - for instance, what happens to tasks that do not benefit from long term planning - like in AssistantBench?
Reviewer_abgeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes JHiMA-WebRL, a joint hierarchical multi-agent RL framework for long-horizon web tasks. The framework factors the interaction into two roles: a task decomposer, which plans once from the task instruction and the initial screenshot and outputs an ordered list of executable subtasks; and an actor, which then executes web actions step by step, conditioned on that decomposition, the current screenshot, and the action history. Training has three stages: a strong teacher model (GPT-5.2 Codex) first turns multimodal GUI trajectories into subtask annotations to fill in the planning supervision that existing datasets do not have; both models are then fine-tuned with supervised learning; finally, the authors extend GRPO with a hierarchical rollout structure — the decomposer samples several candidate decompositions, the actor runs each candidate with multiple full trajectories, the actor's advantage is computed within a decomposition, the decomposer's advantage across decompositions, and both policies are updated from the same rollout batch. The authors claim this is the first work to jointly optimize the decomposer and the actor, and the whole framework runs on fairly small models while still outperforming much larger monolithic agents on WebArena-Lite-v2.

Critical strengths
  1. despite some flaws in the method, the authors emphasize this is the first work to jointly optimize the decomposer and the actor — prior work either trains only one of the two, freezes one side, or alternates updates between modules (HiRoC, SWIRL, Planner Matters!, etc.) — so the "same-batch, simultaneous updates with two-level advantages" regime is genuinely a new thing being opened up.
  2. the method achieves solid results with small models (a 7B decomposer + a 7B actor) and outperforms much larger 32B/72B monolithic agents, which suggests that role factorization with joint training could be a viable alternative to simply scaling up monolithic models.
  3. the paper is easy to follow: the problem formulation, data generation, SFT, and joint RL stages are coherent, the formulas and training details are laid out, and the appendix even includes the teacher prompt and a policy-gradient interpretation — good writing overall, and it reads smoothly from start to finish.
Critical weaknesses
  1. The decomposer plans once at the start (Eq. 1) and nothing during execution ever checks whether subtasks are actually finished or whether we're stuck. The authors admit this themselves in Limitations (iii) ("cannot revise the plan during execution"), and the Conclusion even lists "making the decomposer adaptive" as future work. The cost shows up exactly where you'd expect: the Map domain is the only one they lose (12.5% vs GUI-Libra-8B 17.3% and ScaleCUA-7B SFT 15.4%), and Figure 6 shows the agent looping and burning the entire 15-step budget. Now, open-loop could be a deliberate simplification — but then the comparison set should include the alternative, and it doesn't: Table 1 contains no system with runtime replanning (Plan-and-Act and Agent S2 are absent; the closest thing is a frozen GPT-4o planner at 28.6%), and Table 2 has no "frozen decomposer + inference-time replanning" ablation either. Meanwhile the two most dynamic domains actually do better without the decomposer (Map: 15.4/14.4 vs 12.5; ShoppingAdmin: 46.4 vs 44.3). So whether joint RL really beats runtime adaptation is simply untested — and that question matters a lot for real-world use.
  2. The decomposer's advantage (Eqs. 8–9) is just the group-normalized mean reward of the M=7 trajectories under one decomposition — no filtering, no adherence term; the actor's advantage (Eq. 5) is one trajectory-level number shared by all of its tokens; and with β_A = β_D = 0 there is no KL anchor at all, while Appendix B openly notes that both learning problems are non-stationary. Two leakages are therefore possible: a bad decomposition gets rescued by an improvising actor, which lifts the average reward S_i of that bad plan (so it gets reinforced — while the deviation itself is rewarded on the actor side); and conversely, a perfectly reasonable plan takes the blame when the actor simply could not execute it. The paper provides no failure attribution ("bad decomposition" vs "bad execution" vs "environment"), no adherence measure, and no intra-group variance statistics. The 4.6 analysis does not cover this either: AUROC is 0.838 (with RL) vs 0.836 (without) — basically a tie — and "fraction of completed subgoals" (judged by GPT-5.2 Codex, the same model as the teacher) cannot tell "the plan was good" from "the actor improvised its way out". Limitations (iv) even concedes "coarse credit" — but only at token granularity, not the deeper plan-execution entanglement, which is where I think the real problem sits. As it stands, I cannot tell from the evaluation who gets credit for a success or who picks up the tab for a failure.
  3. Per Table 2, the joint RL adds only +2.1pt over plain SFT of both modules (37.2 vs 35.1). Oddly, RL-ing only the actor with the SFT decomposer frozen lands at 33.4 — below plain SFT — and single-policy actor RL essentially ties with SFT-only (35.2 with 7 rollouts, 34.9 with 21), consistent with ScaleCUA-32B RL (24.0) = ScaleCUA-32B SFT (24.0) in Table 1. In other words, under this 15-step/terminal-reward setting RL barely moves a single policy, "RL the actor only" is a step backwards, and the celebrated +3.8pt for the decomposer update is really "pulling RL back to positive". The one ablation that could cleanly attribute the gain — updating the decomposer while freezing the actor — is missing, and the "w/o decomposer" variants also swap the SFT data (no teacher subtasks), so they don't isolate the variable either. All this costs 720 GPU-hours on 8×H100 plus 21 full episodes per task, and the gains are not evenly distributed (vs SFT-only: GitLab +4.2, Map +3.9, Shopping +3.4, Reddit +1.3, ShoppingAdmin −2.1).
  4. Eqs. (5)/(9) put (std + ε) in the denominator; §(section)4.3 says "standard-deviation and response-length normalization removed to reduce optimization bias, following Liu et al. (2025b)"; and Appendix B's policy-gradient derivation uses only the group mean as the baseline. Length normalization has the same conflict (§4.3 says removed; Eq. 7 carries 1/T and 1/|a| factors). This is not a typo-level annoyance: with K=3, M=7, and β=0, the choice of normalization directly changes the numerical scale of the advantages — and with a 3-member group, dividing by an unstable std turns noise into large advantages — plus it decides whether the paper is reproducible from its own formulas. I'd ask the authors to state once, clearly, which implementation was actually used, and fix whichever version is wrong.
  5. GPT-5.2 Codex is doing a lot of jobs here: it generates the SFT subtask annotations with no human verification (admitted in Limitations (ii)), it serves as the LLM judge for success via fuzzy matching in the flexible evaluation protocol, and it judges subgoal completion in 4.6 . So the teacher and the grading criterion are literally the same model, and "success" ends up defined in terms of the very behavior the teacher taught the model.
Questions for authors
  1. The method is open-loop planning (self-admitted in Limitations (iii)), and neither Table 1 nor Table 2 contains any runtime-replanning control. Could you add a "frozen decomposer + inference-time replanning" variant under the same backbone (Qwen2.5-VL-7B ×2), the same SFT starting point, and a similar evaluation budget — i.e., during execution detect that the current subtask is unfinished or that we are stuck, and have the (frozen) decomposer re-decompose, Plan-and-Act or Agent S2 style — and report overall plus the Map and ShoppingAdmin domains? Also, using your existing rollout/evaluation trajectories, what fraction of failures were "the decomposition was wrong from step 0 and never correctable" (for the Map domain at least)?
  2. The two-level advantage equates "plan quality" with "the average outcome of M=7 trajectories under that plan", with no adherence handling (and no KL anchor, since β=0). Please give two things: (a) a minimal plan-adherence definition (e.g., "each action can be mapped to the subtask it purportedly executes"), then report — via GPT-5.2 Codex or 100–200 human annotations — the joint distribution of adherence vs success and how it shifts over RL, which directly tests whether "the actor rescues bad plans" and "good plans blamed for the actor's failures" actually happen; (b) the within-decomposition variance of terminal rewards over the M=7 trajectories in late training, and the spread of S_i across the K=3 candidates. Note that 4.6 only measures "the fraction of subgoals completed" (AUROC 0.838 vs 0.836) and cannot catch "didn't follow the plan but still finished".
  3. RL adds only +2.1pt over pure SFT (37.2 vs 35.1; frozen-decomposer RL is 33.4; single-policy RL is 35.2; and ScaleCUA-32B RL equals its SFT). Please: (a) give a confidence interval or significance test for the 2.1pt (4 runs × 154 tasks; Binomial or McNemar both fine), plus the training-time success curve on the 647 tasks; and (b) run the missing "update the decomposer, freeze the actor" ablation with a Table-2-matched budget, to separate "updating the decomposer" from "having a decomposition at all".

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We sincerely thank all reviewers for their constructive feedback. We would like to highlight three important points.

1. Broad recognition of methodological novelty. All three reviewers explicitly acknowledged the novelty of our joint hierarchical RL framework. Our central contribution is a new methodology for simultaneously optimizing the decomposer and actor through hierarchical rollouts and role-specific advantage estimation. To our knowledge, prior work has not jointly optimized these components in this manner.

2. Overall performance versus component ablations. We would like to clarify that all configurations in Table 2, including multi-agent SFT and actor-only RL, are our own design variants, not methods from existing papers. The modest 2–4 percentage-point gains reflect the contributions of individual components within our novel framework, rather than incremental improvements over prior research. In Table 1, JHiMA achieves 37.2% versus 24.0% for ScaleCUA-32B RL under the same WebArena training tasks and unified evaluation protocol, despite using fewer parameters.

3. Substantial new experiments during rebuttal. In response to the reviewers' valuable suggestions, we significantly expanded our empirical evaluation:

  • Generalization to a new benchmark: Added Online-Mind2Web evaluation on unseen, live websites without additional training, achieving 29.8% versus 27.7% for the proposed multi-agent SFT component on environment-valid runs.
  • Additional controlled ablations of proposed components: Added compute-matched actor-only RL (34.9%), decomposer-only RL (34.6%), and runtime replanning (36.0%) comparisons, versus 37.2% for our joint RL method.
  • Longer-horizon evaluation: Extended the interaction budget from 15 to 50 steps, where JHiMA maintains its advantage (37.0% versus 35.4%).
  • Additional robustness analyses: Evaluated generalization across unseen task templates, investigated planner–actor credit assignment and failure modes, and clarified evaluation protocols and implementation details.

We believe the independently recognized methodological novelty, competitive overall performance, and substantially expanded experimental evidence together demonstrate a meaningful research contribution. We will clarify the difference between baselines and proposed component ablations in revision.

We respectfully hope the reviewers and AC will consider these contributions and the additional rebuttal evidence holistically in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.