acceptodds
Under review as a conference paper at ICLR 2027

CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer-Use Agent with Decoupled Reinforcement Learning

Abstract

Autonomous agents for Graphical User Interfaces (GUIs) face significant challenges in novel software, requiring both long-horizon planning with software domain knowledge and precise, fine-grained execution. Existing approaches suffer from a trade-off: generalist agents excel at planning but falter in execution, while specialized agents show the opposite weakness. Recent compositional frameworks attempt to bridge this gap by combining a “planner” and an “actor”, but they are typically static and non-trainable, preventing adaptation from experience—a critical limitation given the scarcity of high-quality data in novel software. To address these limitations, we introduce CODA, a novel and trainable compositional framework that synergizes a generalist planner (Cerebrum) with a specialist executor (Cerebellum), trained with a dedicated two-stage training pipeline. The first stage, Specialization, employs a decoupled GRPO approach to train an expert planner for each novel software individually. The second stage, Generalization, aggregates all positive trajectories from all specialized experts. This consolidated, high-quality dataset is then used to perform supervised fine-tuning (SFT) on the final planner, equipping it with the robust, cross-domain capabilities of a generalist. Evaluated on the ScienceBoard benchmark with diversified novel software, our framework significantly outperforms the baseline and establishes a new state-of-the-art (SOTA) among open-source models, with strong generalizability to novel software and unseen executors such as code agents. All code and models will be made publicly available to foster further research.

Primary area:
reinforcement learning
Keywords:
computer-use agents, GUI agents, reinforcement learning, planner-executor architectures, automated feedback

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vLAYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to extend CUA to novel software by training a planner and keep the GUI executor unchanged. The proposed framework has two stages. In stage 1, separate planners are trained for each software using GRPO, and the reward is based on similarities with trajectories approved by the judge. In stage 2, a generalist planner is fine-tuned on trajectories collected from software-specific planners obtained in stage 1. The proposed framework is evaluated on multiple benchmarks including ScienceBoard, TeXstudio, and OSWorld.

Critical strengths
  • Given the complexity of different types of software and GUI executors, training the planner only is a promising idea to make the proposed framework generalizable.
  • The evaluation protocol is comprehensive. Besides comparisons with baselines, the authors also analyze the effect of reward source, planner backbone, executor, and judge quality. I feel the judge-quality study is particularly informative.
Critical weaknesses
  • Technical novelty is not that significant in my view. Training the planner alone via GRPO is not a new idea and has been proposed in AgentFlow [1]. They key motivation is that training gradient cannot propagate through the executor. Similar ideas have also been explore in harness-related studies, where the harness prevents gradient from propagating back to the agent.

  • The contribution of RL is not well-isolated. Based on my understanding, RL is only used in stage 1, while stage 2 is based on SFT. Therefore, to examine the contribution of RL, the ablation version should be training the generalist planner on trajectories generated by the original Qwen model without any RL. However, this is not the case according to the explanation of variants (iv) and (vi) in Table 8.

[1] In-the-Flow Agentic System Optimization for Effective Planning and Tool Use. ICLR 2026.

Questions for authors
  • What is the technical novelty of this work in comparison with AgentFlow?
  • What are the differences between variants (iv) and (vi) in Table 8? Is this comparison fair enough to validate the contribution of RL?
2 comments

We thank the reviewer for the precise, actionable report, and in particular for catching the Table 8 ambiguity — that was our error, and correcting it changed how we describe the RL contribution.

W1/Q1 — Novelty vs. AgentFlow. We accept that planner-only GRPO with frozen downstream modules was proposed there, and the revision now cites and discusses it (Sec. 2, App. I.1). Our claim is not that recipe. The differences that matter: (i) AgentFlow selects among discrete API calls in a short-horizon, largely single-turn setting with terminal rewards, whereas CODA renders the plan into pixel coordinates through a separate 7B grounding model over stateful, long episodes (median 7–9 steps, P90 14–18), and scores the executed action — so the planner must learn to write groundable plans (Fig. 12); (ii) consequently the reward must be dense: under the identical setting an AgentFlow-style terminal objective reaches 18.10 vs 30.17 for ours (Tab. 7b), and 11.21 vs 19.39 on OSWorld; (iii) specialist→generalist consolidation, which yields one planner beating the four-model ensemble (21.04 vs 14.39) with one quarter of the parameters.

W2/Q2 — Table 8. The reviewer is right that it was not readable. Re-checking the manifests, the problem is units, not a confounded control.

  1. The requested control already exists and is size-matched. Row (iii) is exactly SFT on judge-filtered trajectories from the untouched Qwen2.5-VL-32B planner. Its 1,160 is a trajectory count (matching Tab. 10), expanding to 5,771 conversation pairs vs Stage-2's 1,110 / 5,650 — matched to 2.1%, with the RL-free control having slightly more data. The 12.40 → 21.04 gap is therefore not corpus size; if anything the comparison is conservative for our claim.
  2. The error was mixing units: (iii) reported trajectories while (iv)/(vi) reported pairs. Every row now carries #traj and #pairs.
  3. (iv) was mislabeled: "(5.6K Stage-1 expert)" conflated the Stage-2 size with the Stage-1 corpus. (iv) is SFT on the Stage-1 corpus (852 / 4,806); (vi) uses fresh rollouts from the frozen specialists (1,110 / 5,650).
  4. The residual confound is closed. SFT on the Stage-2 corpus down-sampled to 4,806 pairs (same hyper-parameters, no new rollouts) gives 20.03 (19.51 / 31.15 / 14.32 / 15.15). Hence (vii)−(iv) = +2.45 from fresh rollouts and (vi)−(vii) = +1.01 from the extra 17.6% data; all four domains drop monotonically in the latter, confirming a pure volume effect.
IDGeneratorCorpus#traj#pairsAvg@8
(i)————7.57
(ii)baseraw8,00024,9609.12
(iii)basejudge-filtered1,1605,77112.40
(iv)S1Stage-1 corpus8524,80617.58
(v)S1judge-as-reward——14.39
(vi)S1fresh Stage-21,1105,65021.04
(vii)S1Stage-2 subset8324,80620.03

The +8.64 gap over the RL-free control decomposes into +5.18 better data from RL (with 17% fewer pairs), +2.45 fresh rollouts, and +1.01 data volume. Note the direction: three of the four steps shrink the corpus while performance rises (24,960 → 5,771 → 4,806 pairs; 9.12 → 12.40 → 17.58). (ii) is a useful sanity check here — it has 4.1× more pairs than (iii) yet scores 3.28 lower, because raw traces include failed rollouts, so the ablation is not measuring data quantity.

We are happy to run the opposite control (top up (iii) to 5,650) if the reviewer prefers; we chose down-sampling because it requires no new rollouts or judge passes, and because (iii) already has marginally more data than (vi).

Revision. (1) Table 8 reports #traj and #pairs for every row, with (iv) relabeled as the Stage-1 corpus. (2) Row (vii), the matched-size control, is added. (3) H.2 is rewritten so the +3.46 step is described as fresh data plus consolidation rather than as an isolated RL effect. (4) AgentFlow is cited in Sec. 2 and App. I.1, with Tab. 7b promoted as the terminal-reward baseline. (5) The caption states that (iii) and (vi) are matched to within 2.1%.

We thank the reviewer again and would be glad to answer any further questions.

I thank the authors for their detailed response.

The clarification of Table 8 is helpful and addresses my second question.

However, my major concern about the technical novelty remains. The authors describe AgentFlow as a "short-horizon, largely single-turn" setting, which is inaccurate. Based on my understanding, it optimizes the planner explicitly within a multi-turn interaction loop. Dividing the optimization into each turn does not mean it has a single-turn setting.

While I agree that the pixel-grounding interface introduces new challenges, the methodological advance of the proposed idea beyond existing planner-training studies is still not significant.

I therefore maintain my current score.

Reviewer_ywzTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CODA, a decoupled planner–executor framework for GUI-based computer-use agents. CODA freezes a low-level executor for precise GUI grounding and trains a high-level planner through a two-stage pipeline: software-specific reinforcement learning followed by specialist-to-generalist supervised fine-tuning. The goal is to improve agent performance on novel scientific software that requires both long-horizon planning and accurate execution.

Critical strengths

a) The planner–executor decomposition is well motivated for GUI agents, as it separates high-level reasoning from low-level action grounding. Freezing the executor while adapting the planner is a practical strategy that can improve training stability and reduce the cost of full end-to-end optimization.

b) The paper studies adaptation to complex scientific software, an important and challenging setting for computer-use agents. The proposed two-stage training pipeline shows promising improvements over several baselines and provides useful evidence that planner adaptation can improve performance on novel GUI tasks.

Critical weaknesses

a) In Stage 1, the reward for optimizing the planner is computed by comparing the executor-produced action with a ground-truth action. This creates a potential credit-assignment problem: if the planner produces a correct high-level plan but the executor grounds it incorrectly, the planner receives a low reward despite not being at fault. The paper does not seem to quantify the frequency of such cases. A useful analysis would decompose failures into planner errors versus executor grounding errors, and estimate how much reward noise is induced by the fixed executor.

b) CODA requires a two-stage training pipeline, and the first stage performs separate reinforcement learning for multiple software environments. This is computationally expensive and may limit scalability. The paper would be stronger if it compared against lighter-weight adaptation strategies, especially RSI-style recursive self-improvement methods, which may improve performance through iterative self-generated data or self-refinement without requiring costly per-software RL.

c) Baselines may not fully reflect current planner capabilities. One of the paper’s central claims is that generalist planners perform poorly on novel software. However, the strength of this claim depends heavily on the chosen baselines. The paper should include stronger and more recent generalist planners, such as newer Qwen variants and the strongest available closed-source models. If stronger planners already perform well in these environments, the marginal contribution of CODA would be reduced.

d) The reward and trajectory filtering rely on an automatic VLM-based judge. Since judge errors can directly affect both reinforcement learning and SFT data quality, the method’s reliability depends strongly on judge precision. Although the paper discusses judge performance, the remaining error rate still raises concerns about false-positive rewards and biased trajectory selection.

e) The reinforcement learning signal is closer to action-level imitation than trajectory-level optimization. The reward is based mainly on matching individual executor actions to labeled positive actions, rather than optimizing final task success through trajectory-level returns. This weakens the claim that the method learns long-horizon planning through reinforcement learning. The paper should more clearly position the method as using dense action-level surrogate rewards, and distinguish it from true task-level RL.

Questions for authors

a) In Stage 1, what fraction of low-reward samples are caused by executor grounding errors rather than incorrect planner outputs? Have the authors performed any failure decomposition between planner errors and executor errors?

b) How sensitive is CODA’s performance to judge errors, especially false-positive rewards? Can the authors report how often false positives enter the RL or SFT data pipeline?

c) Clarification: How does CODA compare with lighter-weight adaptation methods, such as recursive self-improvement or iterative self-training, that do not require separate per-software RL?

d) To what extent is the learning signal action-level imitation rather than trajectory-level reinforcement learning? Have the authors tested task-level outcome rewards or a hybrid of trajectory-level and action-level rewards?

Reviewer_MP9JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CODA, a trainable planner–executor framework for GUI agents. It combines a general-purpose vision-language planner with a frozen specialized executor. The planner is first adapted to individual software environments through decoupled GRPO with automatically generated action-level rewards. Successful trajectories from specialized planners are then aggregated for supervised fine-tuning to obtain a generalist planner. Experiments on ScienceBoard and additional environments report improved performance and transfer across executors.

Critical strengths
  1. Adapting a planner through interaction while keeping the grounding executor fixed is a practical approach to acquiring application-specific knowledge without repeatedly training the full agent.

  2. The automated judging pipeline is potentially useful. The paper studies judge quality and ensemble voting, providing evidence that reward reliability is important for training planners in complex GUI environments.

  3. The experiments explore multiple aspects of generalization. The evaluations include software-specific adaptation, cross-domain transfer, and alternative executors, including code agents. These results provide preliminary evidence that learned planning capabilities can transfer across execution interfaces.

Critical weaknesses
  1. Planner–executor decomposition, frozen action executors, and RL-based GUI adaptation have already been explored. CODA primarily combines domain-specific planner training, automated reward generation, and SFT-based trajectory distillation. The paper does not sufficiently establish which component provides a fundamentally new capability compared with prior compositional approaches.

  2. The generalization claims are stronger than the supporting evidence. The main experiments involve a limited set of scientific software environments, and the paper acknowledges that OSWorld appears in the judge-training sources. This weakens the interpretation of OSWorld as fully independent out-of-domain validation. The evidence for generalization to genuinely unseen applications remains limited.

Questions for authors
  1. What is the main methodological difference from existing trainable planner–executor frameworks?
  2. Can the authors evaluate on applications excluded from both planner and judge training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.