acceptodds
Under review as a conference paper at ICLR 2027

MintAct: A Unified Visual Agent for Digital Environments

Abstract

We present MintAct, a family of vision-language models that unifies UI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use, trained at 2B, 4B, and 8B scales. Through careful design of our environments, data, and training recipes, MintAct models match the performance of per-domain specialists across all of these capabilities. To enable this, we develop a scalable environment and reinforcement learning (RL) infrastructure. On the environment side, we host hundreds of concurrent instances across heterogeneous per-domain backends, serving both trajectory data collection and online RL. To enable efficient and scalable RL training, an asynchronous framework keeps explicit control over the cross-domain training distribution and remains stable under noisy environment feedback and off-policy drift. Experimental results show that MintAct achieves state-of-the-art performance (48.9 on OSWorld-Verified) across a wide range of benchmarks at comparable model sizes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual Agent, Multimodal LLM, RL

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3bsYOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces MintAct, a unified family of vision-based agents available in 2B, 4B, and 8B parameter sizes. MintAct can directly perceive screen content, execute multi-step operations across mobile devices, computers, and web environments, and utilize external tools. Through supervised fine-tuning, distillation from expert models, and reinforcement learning, MintAct achieves or surpasses the performance of specialized models of similar scale; for instance, the 8B version scored 48.9 on the OSWorld-Verified benchmark. The paper demonstrates that a single model can handle tasks across diverse digital environments, eliminating the need for platform-specific training. Nevertheless, there remains room for improvement in areas such as comprehensive cross-environment joint training, context management for long-horizon tasks, and the seamless integration of interface interactions with tool usage.

Critical strengths
  1. Agentic reinforcement learning for heterogeneous, multi-platform devices is an intriguing area. Implementing this and achieving a robust deployment requires a significant amount of effort.
Critical weaknesses
  1. This paper reads more like a project report than a formal research paper. It mainly describes a common pipeline for training a GUI agent model and presents the resulting model. As a result, it is difficult to identify clear novel insights or meaningful contributions to either academia or industry. Overall, it feels less like a research contribution and more like a summary of completed work, reflecting on a project after it has already been carried out.
  2. Regarding the main contribution claimed in this paper, the base model used to fine-tune MintAct, namely Qwen-3-8B, appears somewhat outdated. Since the newer version, Qwen-3.5-9B, was released in March, it would have been a more suitable choice for a submission to ICLR, given the September deadline. This choice of base model may also make it difficult for the experimental results to remain competitive with those of recent models at a similar scale. If the authors intend to emphasize a methodological contribution, then using Qwen-3-8B may still be acceptable. However, in that case, the paper should provide more substantial insights and analysis of the proposed methodology or system design.
Questions for authors
  1. In the agentic RL setting, which VLM was used as the judge? In addition, where did the tasks used for training during the agentic RL stage come from? How did the authors verify that these tasks were actually executable in practice?
  2. To be honest, when I think about agentic reinforcement learning for heterogeneous, multi-platform devices, two questions particularly interest me. First, does joint training across multiple platforms provide any advantages in terms of efficiency, performance, stability, or other aspects, compared with training separate models and then combining them through model merging or other integration methods? Second, in a heterogeneous setting, could differences in execution speed across tasks become more severe during the rollout stage? For example, might tasks in certain domains run more slowly because of their inherent complexity or because of container overhead? If so, could the rollout process become limited by these slower tasks? Moreover, if time limits or early truncation are used, how would that affect performance in those domains? These seem like important trade-offs. How did the authors address them?
3 comments

Since the deadline for rebuttal is approaching, the author has still not submitted a rebuttal. Meanwhile, the other two highly-rated reviews have almost identical weaknesses, suggesting they were AI-generated. Based on the above points and my original review, I'm lowering the score to one.

[W1] Research findings

Thank you for this candid feedback. We respectfully note that MintAct is driven by a research question: whether diverse visual agentic capabilities can be unified within a single compact model without sacrificing performance in any domain, and what makes such unification work. Our experiments address this with several findings, which are reported in Section 4.3 and Appendix A.2 but could be highlighted more clearly. Representative examples include:

  • Grounding and multistep SFT are complementary. Either stage alone fails on the other capability: Stage 1 alone reaches only 9.4 on OSWorld-Verified, while Stage 2 alone drops UI-Vision to 25.0. Combining them retains both (Table 3).
  • Joint RL transfers across related domains. Joint RL on mobile and desktop improves web navigation without web data (Weblica from 57.5 to 60.8, Table 5).
  • Synthetic environments help when real environments are unavailable. RL purely in our synthetic environments, without using either real emulator, improves Qwen3-VL-8B from 47.6 to 60.3 on AndroidWorld and from 33.9 to 38.6 on OSWorld-Verified. When real environments are available, RL on them remains more effective (AndroidWorld 66.7 vs. 59.8, Table B).

We will revise the Introduction to summarize these findings explicitly.

[W2] Results on Qwen-3.5

Thank you for this suggestion. We address it in two ways. MintAct already outperforms the newer Qwen3.5 models at comparable sizes. Table 1 includes Qwen3.5 as baselines. Starting from the older Qwen3-VL base, MintAct-8B outperforms Qwen3.5-9B on AndroidWorld (67.0 vs. 57.8) and OSWorld-Verified (48.9 vs. 41.8), and MintAct-4B likewise outperforms Qwen3.5-4B (62.6 vs. 58.6 and 44.8 vs. 35.6). Our recipe transfers to the newer base model. To verify that our pipeline is not tied to a specific base model, we applied our two SFT stages to Qwen3.5-9B with the same data and hyperparameters (Table R1). Our SFT improves Qwen3.5-9B on all seven benchmarks, with large gains in grounding (UI-Vision +37.1, OSWorld-G +14.7) and consistent gains on mobile and desktop navigation (AndroidWorld +4.6, OSWorld-Verified +1.6). Due to the limited compute during the rebuttal period, we have not yet run the RFT and RL stages on Qwen3.5-9B, and we will include these results in the final version.

ModelMMBench-GUIScreenSpot-v2UI-VisionOSWorld-GAndroidControlAndroidWorldOSWorld-Verified
Qwen3.5-9B84.792.836.955.067.657.841.8
Qwen3.5-9B + MintAct SFT87.895.274.069.772.162.443.4
Δ+3.1+2.4+37.1+14.7+4.5+4.6+1.6

[Q1] RL task source and executability

Thank you for these questions. We build a dedicated data synthesis pipeline for each domain, detailed in Appendix A.1. Taking desktop (OSWorld) as an example (Appendix A.1.1): starting from a small set of human designed seed tasks, a specialized teacher model (EvoCUA-32B) rolls out the tasks (multiple times) in the real OSWorld environment, and our internal VLM proposes new tasks grounded in what these rollouts reveal about the environment. The new tasks are rolled out again in the next round, and we iterate until the tasks cover the environment. Finally, an internal VLM judge evaluates the rollouts and filters out tasks whose rollouts fail, so every remaining task has been successfully executed in the real environment. To select RL tasks of appropriate difficulty from this pool, we roll out our SFT policy eight times per task and keep tasks with both successful and failed outcomes, which provide the most informative learning signal for group based RL.

[Q2 (part 1)] Joint training vs. separate training and merging

Thank you for this insightful question. Among the many ways to combine separately trained models, we adopt the most straightforward one: we train an RL specialist for each domain and distill them back into a single model at the data level through RFT (Section 3.2.3), which clearly improves over the SFT model at all scales (Table 4).

Joint RL further improves this merged model, raising OSWorld-Verified from 43.1 to 48.9 and Online-Mind2Web from 36.3 to 39.1 at 8B, with the same trend at 4B and 2B (Table 4). The two approaches are complementary: merging alone retains only part of the desktop specialist's gain (43.1 vs. 49.2), and joint RL nearly closes this gap (48.9).

Joint training also brings transfer across domains. Starting from the same SFT model, joint RL on mobile and desktop exceeds, on average, every RL specialist trained on a single domain (navigation average 55.9 vs. 54.1 and 54.3). On Online-Mind2Web, it improves to 32.8, while both single domain specialists fall below the SFT model (28.4 and 27.7 vs. 30.0) (Table 5).

[Q2 (part 2)] Heterogeneous rollout speed across domains

Thank you for this insightful question. We agree that task complexity and environment overhead lead to different rollout speeds across domains. Filtering out groups whose rollouts are uniformly successful or uniformly failed widens this gap further, since domains also differ in how often they produce informative groups.

We address this with three mechanisms. First, rollouts are fully asynchronous, so slow trajectories do not block other tasks from progressing. Second, the trainer collects groups under a quota for each domain, and a backpressure mechanism pauses new rollouts from any domain that leads the slowest domain by at least one training batch, so fast domains cannot dominate the training distribution. Third, if a slow domain has not filled its quota after a bounded waiting time, the trainer relaxes the quota and fills the remaining slots with groups from other domains, so a slow domain never stalls training. The trade off is that the slow domain contributes fewer groups to that batch, which could cause domain imbalance if it happened frequently. In our experiments, however, we found that this fallback is rarely triggered.

Reviewer_2yEeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work presents MintAct, a family of vision-language agents at 2B, 4B, and 8B scales that unifies GUI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use within a single model. The training recipe consists of high-resolution single-step SFT, low-resolution multi-step SFT, rejection-sampling RFT that distills per-domain RL specialists into one model, and a final joint asynchronous RL stage with per-domain quotas to control the cross-domain training distribution. Experiments on grounding, navigation, and tool-use benchmarks show that MintAct is competitive with or outperforms size-matched open models, e.g., 48.9 on OSWorld-Verified at 8B.

Critical strengths
  1. This work provides a complete and solid pipeline, covering environment construction, data generation, multi-stage training, and asynchronous RL infrastructure, with detailed descriptions that would be valuable for the community.
  2. It is interesting that a single compact model can cover grounding, navigation on three platforms, and visual tool use while staying competitive with per-domain specialists, and the results are consistent across three model scales.
  3. The ablations show that each training stage contributes, and joint RL on mobile and desktop also transfers to the web domain without web data.
Critical weaknesses
  1. Most components, e.g., distilling multiple specialists via rejection sampling and asynchronous RL, follow existing techniques, so the contribution lies mainly in integrating them into a working pipeline.
  2. There are no ablations on key design choices such as prompt-conditioned action sets, balanced domain mixing, or the per-domain quota mechanism.
  3. The RL tasks are drawn from the AndroidWorld and OSWorld environments, which are also used for evaluation, so the separation between training and test tasks should be clarified. And the joint RL setting in Section 3.3.2 (B=64, equal quota per domain) is inconsistent with Table D (mini-batch 48, 25%/75% for mobile/desktop).
Questions for authors
  1. How are the AndroidWorld and OSWorld RL tasks separated from the evaluation tasks?
  2. Which joint RL setting is actually used, the one in Section 3.3.2 or the one in Table D?
  3. How does the model perform with a merged action space or natural domain mixing instead of the proposed designs?
2 comments

[W1] Our contribution

Thank you for this comment. We agree that individual components, such as rejection sampling distillation and asynchronous RL, build on existing techniques, and we do not claim a new RL optimization algorithm as our contribution. Our goal is to show that diverse visual agentic capabilities can be unified within a single compact model without sacrificing performance in each domain, and to make such unification practical to train. Accordingly, our contributions lie in the empirical evidence, and the training infrastructure that make this unification possible:

  • We train MintAct models at 2B, 4B, and 8B scales and unify three capabilities: UI grounding, multi-step navigation across mobile, desktop, and web, as well as visual tool use.
  • We show that jointly training across these capabilities and domains does not degrade per-domain performance: a single MintAct model matches or exceeds size-matched specialist baselines (each trained on an individual domain) across grounding, mobile, desktop, and web navigation, and visual tool use.
  • We build an asynchronous RL framework tailored to unified visual agents, addressing the unstable, heterogeneous environment interaction and the long multimodal trajectories inherent to this setting. This framework runs on a scalable environment infrastructure that sustains hundreds of concurrent instances across domains. The same environment infrastructure serves both online RL rollouts and offline SFT trajectory collection at high throughput.

[W2 & Q3] Prompt conditioned action sets

Thank you for the suggestion. We chose prompt conditioned action sets over a single merged action space for two practical reasons. (1) Native action spaces differ substantially across domains. As summarized in Table A, many actions are specific to one platform, and even similar actions take different arguments. For example, mobile scrolls with swipe(coordinate, coordinate2, direction), desktop with scroll(coordinate, pixels), and web with scroll(direction, pixels). Visual tool use cannot be merged at all, since the agent discovers tools from more than 500 APIs during execution (Appendix A.1.4). A merged space would require handcrafted translation rules at every environment boundary, so an ablation on it would mainly measure those rules rather than the model. (2) Native action spaces give flexibility in data synthesis. Specialized teacher models, such as EvoCUA for desktop, have their own preferred action spaces. Keeping each domain's native space lets us use their trajectories directly, whereas forcing a merged space or remapping actions afterward would add cost with little expected benefit. Note that only the action interface is specific to each domain: all UI domains share the same observation and grounding space, operating on raw screenshots with coordinates normalized to a common scale (Section 3.1). The jointly trained model remains competitive with specialists each trained on a single action space (Table 2), suggesting no harmful interference.

[W2 & Q3] Ablation on balanced domain mixing

We thank the reviewer for this suggestion and ran a controlled ablation. Starting from the same Stage 1 checkpoint of the 8B model, we train two Stage 2 SFT models with the same total data budget and hyperparameters, differing only in the mixing ratio: natural mixing, which follows the raw dataset sizes (24.0%/13.2%/29.1%/33.7% for mobile/desktop/web/visual tool use), and our balanced mixing (25% each). Balanced mixing improves performance. Desktop, the smallest domain under natural mixing, gains the most (OSWorld-Verified +6.7), and other domains also improve (AndroidWorld +1.7, Online-Mind2Web +0.9). Notably, visual tool use improves as well (MM-ToolSandBox +4.3) even though its share drops from 33.7% to 25%, showing that balancing does not come at the cost of the largest domain.

[W3 & Q1] Clarification on AndroidWorld and OSWorld RL tasks

Thank you for raising this. Although our RL tasks run in the same AndroidWorld and OSWorld environments used for evaluation, no evaluation task is used for training. All training tasks are produced by our own pipeline (Appendix A.1.1, A.1.2). A teacher agent first explores each environment starting from a small set of seed tasks. An VLM then proposes new tasks based on the application states it discovers, and we keep only tasks for which the teacher produces a verified successful trajectory. Before training, we filter these tasks against all evaluation tasks using text matching and VLM based semantic matching. The semantic matching also removes tasks that are worded differently but equivalent to an evaluation task.

[W3 & Q2] Clarification on joint RL setting

Thank you for pointing this out.The qd=32q_d=32 (line 318) is a typo. The Table D specify the parameters used to train our model in Table 1. We will correct it in the revision.

Thank you for the response. I consider this a solid engineering paper, but I partially agree with Reviewer 3bsY that the techniques used are fairly standard, and it reads more like a report and more like a summary of completed work. Therefore, I will keep my score.

Reviewer_nH5wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes MintAct, a family of lightweight vision-language models that unifies UI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use within a single model. The multi-stage recipe warms the model up with grounding and multi-step supervised fine-tuning, distills per-domain reinforcement-learning specialists back into one model, and finally optimizes it jointly with agentic RL. An asynchronous RL framework keeps explicit control over the cross-domain training distribution and remains stable under noisy environment feedback and off-policy drift. Across scales, one model matches or exceeds similar-sized specialists and is competitive with the best public models at comparable sizes.

Critical strengths
  1. Unifying multiple environments in a single model. MintAct uses a single set of weights to unify UI grounding, multi-step navigation across mobile, desktop, and web, and visual tool use. A shared observation and grounding space, prompt-conditioned action sets, and balanced cross-domain mixing let this one model span those domains and match per-domain specialists without sacrificing per-domain quality.
  2. The experiments are effective and solid. Across all three scales, a single MintAct model is competitive with or outperforms size-matched specialist baselines on grounding, mobile, desktop, and web navigation, and visual tool use simultaneously, and the ablations show that every stage contributes. The joint SFT model stays competitive with each specialist while covering all domains in a single model. RFT is effective at maintaining domain-specific performance while keeping a single model, and the joint RL model, on average, exceeds every single-domain RL specialist on both grounding and navigation.
Critical weaknesses
  1. Limited algorithmic novelty. The paper largely integrates known techniques rather than introducing a new learning algorithm. Per-domain RL specialists, rejection-sampling distillation, and Dr.GRPO are established components, and the asynchronous loop follows prior systems such as AReaL, and AsyncWebRL. Relative to these systems, the added piece appears to be the scheduling rule used during asynchronous RL training, rather than a new algorithm.
  2. Some relevant comparisons are missing. Table 1 would be more informative with a few additional size-matched models. GUI-Owl-1.5-8B-Thinking reports 52.9 on OSWorld-Verified, 71.6 on AndroidWorld, and 48.6 on Online-Mind2Web [1], and UI-Venus-1.5-8B reports 73.7 on AndroidWorld [2], against MintAct-8B at 48.9, 67.0, and 39.1 on the same three benchmarks. Including these results would give a clearer picture of where the model stands. [1] H. Xu, X. Zhang, H. Liu, et al. Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents. arXiv:2602.16855, 2026. [2] C. Gao, Z. Gu, Y. Liu, et al. UI-Venus-1.5 Technical Report. arXiv:2602.09082, 2026.
Questions for authors
  1. For each reported benchmark, please describe the training and test split.
  2. Section 3.3.2 and Table D specify different joint-RL batch sizes and domain ratios. Which configuration produced Table 1, and why were web and visual tool use excluded?
1 comment

[W1] Our contribution

Our contribution Thank you for this comment. We agree that MintAct does not introduce a new RL optimization algorithm, and we do not claim this as our contribution. Our goal is to show that diverse visual agentic capabilities can be unified within a single compact model without sacrificing performance in any domain, and to make such unification practical to train. Accordingly, our contributions lie in the empirical evidence, and the training infrastructure that make this unification possible:

  • We train MintAct models at 2B, 4B, and 8B scales and unify three capabilities: UI grounding, multi-step navigation across mobile, desktop, and web, as well as visual tool use.
  • We show that jointly training across these capabilities and domains does not degrade per-domain performance: a single MintAct model matches or exceeds size-matched specialist baselines (each trained on an individual domain) across grounding, mobile, desktop, and web navigation, and visual tool use.
  • We build an asynchronous RL framework tailored to unified visual agents, addressing the unstable, heterogeneous environment interaction and the long multimodal trajectories inherent to this setting. This framework runs on a scalable environment infrastructure that sustains hundreds of concurrent instances across domains. The same environment infrastructure serves both online RL rollouts and offline SFT trajectory collection at high throughput.

[W2] Add missing relevant comparisons

Thank you for pointing out these recent models. We agree that including them gives a clearer picture of where MintAct stands, and we will add GUI-Owl-1.5-8B-Thinking and UI-Venus-1.5-8B to Table 1 in the revision.

[Q1] Train/test split

Thank you for the question. We carefully separated our training data from every evaluation benchmark, and filtered all generated training tasks against the evaluation sets to prevent data leakage.

AndroidWorld and OSWorld-Verified: Our evaluation tasks are separated from training. All training tasks are produced by our own pipeline (Appendix A.1.1, A.1.2). A teacher agent first explores each environment starting from a small set of seed tasks. An LLM then proposes new tasks based on the application states it discovers, and we keep only tasks for which the teacher produces a verified successful trajectory. Before training, we filter these tasks against all evaluation tasks using text matching and LLM based semantic matching. The semantic matching also removes tasks that are worded differently but equivalent to an evaluation task.

Grounding (MMBench-GUI, ScreenSpot-v2, UI-Vision, OSWorld-G): These benchmarks are used only for evaluation. Grounding SFT uses separate datasets: GroundCUA, MolmoPoint(Section 3.2.1).

Web (Weblica, Online-Mind2Web). Web SFT and RL use only the Weblica training data (Section 3.2.3). Weblica is evaluated on its 500 synthetic evaluation tasks, which do not overlap with the training set. Online-Mind2Web is used only for testing.

Visual tool use (MM-ToolSandBox): We evaluate on the official 258 scenarios. Training scenarios are generated with the MM-ToolSandBox generation pipeline from COYO and Aria-UI images, which do not overlap with the evaluation set (Appendix A.1.4).

[Q2] Joint-RL configuration clarification and why were web and visual tool use excluded

Thank you for pointing this out.

The qd=32q_d=32 (line 318) is a typo. The Table D specify the parameters used to train Table 1. We will correct it in the revision.

Regarding web and visual tool use, we did not include them in joint RL because of resource constraints. Running joint RL across all four domains requires sustaining many heterogeneous environments concurrently throughout training, which is resource intensive (see our Limitations section in Appendix A.5). Moreover, joint RL on mobile and desktop transfers to web navigation (Table 5), although visual tool use remains at the SFT level.

Comments on the paper

1 comment

We thank all reviewers for their thoughtful feedback. We are encouraged that reviewers found that a single compact model can unify grounding, navigation across mobile, desktop, and web, and visual tool use while remaining competitive with or outperforming similarly sized specialists (nH5w, 2yEe, 3bsY), consistently across all three scales (nH5w, 2yEe). Reviewers noted that our design, with a shared observation and grounding space, prompt conditioned action sets, and balanced mixing across domains, achieves this without sacrificing quality in any domain (nH5w). They found the experiments effective and solid (nH5w), with ablations showing that every training stage contributes (nH5w, 2yEe), RFT maintaining performance in each domain within a single model (nH5w), joint RL on average exceeding every RL specialist trained on a single domain in both grounding and navigation (nH5w), and joint RL on mobile and desktop transferring to the web without any web data (2yEe). Reviewers also considered the pipeline complete and solid, covering environment construction, data generation, multistage training, and asynchronous RL infrastructure, with detailed descriptions valuable to the community (2yEe), and recognized agentic RL across heterogeneous, multi platform devices as an intriguing area whose robust deployment requires significant effort (3bsY). Finally, all reviewers reported no substantive concerns about soundness, and two found the paper clear and easy to follow (2yEe, 3bsY).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.