acceptodds
Under review as a conference paper at ICLR 2027

MMADP: Unified Multimodal Agent Trajectories for Open Agentic Training

Abstract

Many open-weight models learn to use tools through agentic midtraining, but the datasets used at this stage are rarely released. Open datasets could support research on this stage. Existing efforts to combine them focus on text-only trajectories or individual domains. We introduce the Multimodal Agent Data Protocol (MMADP) to support multimodal agent training across coding, web, and computer-use tasks. MMADP builds on the Agent Data Protocol (ADP) and adds support for screenshots and annotations of screen elements. Using MMADP, we combine 16 datasets into a corpus with an estimated 14.7M trajectories and 366B tokens, the largest open corpus of agentic trajectories to our knowledge. Our experiments show that a small sample of this corpus can teach tool use across domains. We perform supervised fine-tuning of InternVL2.5-MPO on a 250M-token sample, starting from models with no prior tool-calling training. Both the 8B and 26B models improve on coding, web, and computer-use benchmarks. At 8B, the absolute gains are +3.0% on SWE-bench Verified, +18.2% on MiniWoB++, and +6.0% on AndroidWorld. We also find that mixing domains helps within each domain: with 250M tokens sampled for each model, the mixed-domain 8B model outperforms single-domain models on all three benchmarks. For each domain, the mixed-domain model uses only about one-third of the training tokens used by the corresponding single-domain model. We release the schema, converters, and full corpus to support open research on multimodal agentic midtraining.

Primary area:
datasets and benchmarks
Keywords:
agent training data, multimodal agents, agentic midtraining, GUI agents, coding agents, computer use, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UoSQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to improve research on agentic midtraining for multimodal trajectories. To do this, the authors extend (while maintaining compatibility) the Agent Data Protocol (ADP), which currently only covers text, developing Multimodal ADP (MMADP). This protocol allows them to standardize and combine 16 existing datasets, building a single corpus of 14.7M trajectories (366B tokens). They then perform SFT on a subset of this dataset at two different model sizes (8B and 26B), showing improvements on various task categories like coding, web and computer use. The authors commit to releasing both the corpus and the infrastructure needed to extend this dataset.

Critical strengths
  • S1: Valuable open resource for midtraining research, also for lower-resourced labs.
    The authors commit to release schema, converters and the whole dataset. This could allow lower-resourced research labs to benefit for such a contribution.
  • S2: Extensible to new datasets. The approach is highly extensible to new datasets by building a converter for each.
Critical weaknesses
  • W1: Experiments test SFT on a post-trained model, the paper is about midtraining.
    The main paper motivation is around improving multimodal agent midtraining. However, all the evaluations are presented against a post-trained model using SFT. While this is not necessarily an issue for the dataset evaluation, for mixed-domain test, it's not clear the same conclusions would hold.
  • W2: The large-corpus claim rests on WildGUI
    WildGUI represents 12.70M of the 14.71M trajectories (340.06B out of 366.35B tokens) . During training the authors subsample in different ratios (section 4.1) to avoid "one dataset from dominating". This means that effectively, in order to use this dataset in the future it'll be necessary to either (1) upsampling the other splits by several orders of magnutude or (2) allow for unbalanced corpus.
  • W3: Conversion cost is not reported. The cost of the conversion process is not discussed in the manuscript, no code is available so it's not entirely clear how computationally demanding this stage could be.
Questions for authors
  • Q1: can the authors clarify and extend their discussion about the mixed-domain advantages? Can they also extend and clarify why they think their evaluation setup can be representative for the actual goal this dataset was built for (why is their evaluation a good proxy for it?)
  • Q2: can the authors clarify how the imbalance raised by W2 could be managed?
  • Q3: what is the conversion cost for "building" this dataset? Would potential users incur in this cost in order to adopt it?
3 comments

We greatly appreciate the reviewer's positive assessment, and the recognition of MMADP as a valuable open resource for midtraining research, in particular for lower-resourced labs, and of its extensibility to new datasets through one converter each. We appreciate the questions regarding our evaluation setup, the balance of the corpus, and the cost of conversion, and we are happy to clarify each below.

W1: Experiments test SFT on a post-trained model, the paper is about midtraining.

W1: ...for mixed-domain test, it's not clear the same conclusions would hold.

Q1: ...why is their evaluation a good proxy for it?

We appreciate this point, as it was a decision we spent some time on. We believe our experiment is a reasonable proxy for midtraining for two reasons: the base model has not seen data in a tool-calling or agentic format, and our setup follows published midtraining choices in its starting checkpoint, loss, and scale. InternVL2.5 predates agentic midtraining of many open-source models, such as GLM-4.5 (GLM-4.5 Team, 2025), and its report describes its training data as conversation-format instruction data, with no mention of tool calls (Chen et al., 2024). Accordingly, before training, only 4.0% of the 8B base's SWE-bench Verified tool calls and 4.8% of its AndroidWorld steps are valid (Table 3). This aligns with Liu et al. (2026b), who found that "midtraining benefits are largest for domains distant from general pretraining data, such as code and math."

Secondly, midtraining has no standard recipe. The concept remains "theoretically nebulous", with different papers using the term for different setups (Tu et al., 2025). Published midtraining runs vary widely, and our configuration matches a published choice in each respect:

  • GUI-CIDER midtrains Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct (Wu et al., 2026), both of which are instruct-tuned models like ours. Those models also had agentic post-training before release.
  • Singhal et al. (2026) compute loss "only on assistant turns, including tool calls", as we do.
  • GUI-CIDER midtrains on a subset of a corpus of about 100M synthesized tokens, which is less than our configuration (Wu et al., 2026).

Our runs follow a published choice on each of these, which is why we consider them a reasonable small-scale proxy.

Q1: can the authors clarify and extend their discussion about the mixed-domain advantages?

Please see Section 5.2: the mixed-domain model beats each single-domain model on that model's own benchmark while seeing about a third of that domain's tokens. We now also show that each of these gaps holds under a paired bootstrap test (p = 0.041, 0.026, and 0.007; Table R1).

One concrete behavior that mixed-domain training improves is premature termination, where the agent ends an episode before making progress on the task. Outside its domain, the coding-only model ends 83 of the 99 scored MiniWoB++ episodes at step 1 with a finish call (Section 5.2). Inside its domain, it ends 281 and 129 of the 500 SWE-bench Verified instances with an empty patch in its two runs, against 2 and 1 for the mixed-domain model, respectively.

W2: The large-corpus claim rests on WildGUI. WildGUI represents 12.70M of the 14.71M trajectories...

W2: ...necessary to either (1) upsampling the other splits ... or (2) allow for unbalanced corpus.

Q2: can the authors clarify how the imbalance raised by W2 could be managed?

Thank you for raising this point. There is a third option, which the paper itself uses: downsampling the large sources (Section 4.1). Our 14 training sources already span 114 to 1,045,845 trajectories (Table R2), and sampling them in proportion to their size would give AITW alone 53% of the tokens (Section 4.1). We instead draw each source to a hand-set share, and the resulting mixture improves all three benchmarks over the 8B base (p < 0.01 each, Table R1). WildGUI can enter a mixture in the same way, at any share that the user sets.

We release the full conversion so that this choice (along with the two provided by the reviewer) stays with the user because midtraining mixtures remain an area of open inquiry. For example, GLM-5 up-samples agent trajectories within its midtraining data (GLM-5 Team, 2026). We also expect the set of sources to grow, as it has for ADP: ADP v1 was published with 13 datasets (Song et al., 2026), and its public repository now holds 56.

W3: Conversion cost is not reported.

W3: ...no code is available so it's not entirely clear how computationally demanding this stage could be.

Q3: what is the conversion cost for "building" this dataset? Would potential users incur in this cost in order to adopt it?

We agree that this cost should be reported, and provide the measurements in Table R4. All 16 sources convert in 16 h 46 min of total wall time, on machines with at most 8 cores and 64GB RAM, and no GPU. Users of the hosted training subset (106,674 trajectories, Section 3.6) incur none of this cost, and we will also host a converted WildGUI slice of the same size as the other sources (12,000 trajectories). For the full conversion of any other source, a user runs its converter on the source release. The converters, the QA checks, and the selection code are part of the release (Reproducibility Statement). We are unable to share a link at this time due to anonymity.

Sources

We thank the reviewer again for the positive assessment. As the discussion period closes today, we would like to remind the reviewer that our response is posted above. We would be glad to know whether our answers on midtraining, the balance of the corpus, and the conversion cost (Table R4) address the reviewer's questions. If they do, we would appreciate it if the reviewer would consider raising the score. If any question remains, please let us know, and we will respond promptly.

Reviewer_XmSkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MMADP extends the Agent Data Protocol with image observations and screen-element annotations, enabling coding, web, and computer-use trajectories to share a representation and conversion pipeline. The paper assembles 16 sources into an estimated 14.7M-trajectory, 366B-token corpus and supplies quality checks and training-format converters. SFT experiments use a 250M-token sample with InternVL2.5-MPO at 8B and 26B. Mixed-domain training improves all three evaluated domains, and the 8B mixture also outperforms single-domain models at a fixed total token budget.

Critical strengths
  1. The work addresses a concrete infrastructure need: multimodal observations, grounding information, and tool calls must remain consistent when heterogeneous datasets are converted for training. The backward-compatible schema and shared rendering pipeline can reduce repeated engineering work for the community.
  2. The paper goes beyond specifying a format by documenting source-specific defects, implementing checks at both conversion boundaries, and testing the resulting data in interactive environments. The fixed-budget mixture comparison and the separation between tool-call validity and task success provide useful initial evidence about what the data teaches.
Critical weaknesses
  1. The corpus is the primary contribution, but the presented evidence does not sufficiently establish the readiness and quality of its dominant sources. Section 3.6 states that WildGUI and ScaleCUA were excluded from the training draw because their conversion was not complete at that time; WildGUI alone accounts for approximately 340 of the 366B estimated tokens. Consequently, the training results do not validate that dominant portion of the resource, and the manuscript does not make its final conversion and post-QA status sufficiently explicit. This leaves an important part of the claimed resource contribution difficult to assess. I am not assuming that conversion remains incomplete, nor requiring training on the full corpus; the concern is the evidence supplied for the released resource itself.

  2. The downstream results provide promising initial evidence, but do not yet establish how robustly the mixture improves task success. For example, 3.0% versus 2.0% on 500 SWE-bench Verified instances corresponds to five additional successes, while AndroidWorld has 116 tasks and MiniWoB++ uses 99 scored tasks. The tool-validity controls are useful and I do not attribute all gains merely to learning the call format. Nevertheless, without paired uncertainty estimates or repeated evaluation, it is difficult to determine the reliability of the reported mixture advantage. Alongside the incomplete account of full-resource validation, this limits the evidence supporting acceptance in the current version.

Questions for authors
  1. Could you distinguish source estimates, completed standardized conversions, post-QA counts, and the hosted training subset in a compact release-status table? For WildGUI and ScaleCUA, please state what has been fully converted and checked in the final artifact, and provide a small representative audit of screenshot–action alignment if available. This does not require training on the full corpus.

  2. Could you provide paired task-level uncertainty estimates for mixed-domain versus single-domain results, with success counts as well as percentages? A few repeated evaluation rollouts of the existing checkpoints would strengthen the interactive results if feasible. Please identify which mixture advantages remain supported by this analysis and qualify those that remain uncertain. New large-scale training runs or state-of-the-art agent results are not required.

5 comments

We sincerely thank the reviewer for the thoughtful and constructive assessment. We appreciate the recognition that MMADP addresses a concrete infrastructure need, that its backward-compatible schema and shared rendering pipeline can reduce repeated engineering work for the community, and that the paper goes beyond specifying a format by documenting source-specific defects, implementing checks at both conversion boundaries, and testing the data in interactive environments. We are also glad that the reviewer found the fixed-budget mixture comparison and the separation of tool-call validity from task success useful. The suggestions on the release status of the corpus and on paired uncertainty estimates were valuable, and we have adopted both, as detailed below.

W1: ...the manuscript does not make its final conversion and post-QA status sufficiently explicit.

Q1: Could you distinguish source estimates, completed standardized conversions, post-QA counts, and the hosted training subset in a compact release-status table?

Thank you for this suggestion, which we have adopted. We re-processed every source in full and report exact counts at each stage, from the source release, through the standardized conversion, to the trajectories that pass all QA checks (Table R2). These counts supersede the estimates of Table 1, and we will update the paper accordingly. The hosted training subset is the selection of Section 3.6 (106,674 trajectories from 14 sources, Table 5), to which we will add a converted WildGUI slice of the same size as the other sources (12,000 trajectories). The rest of the corpus can be rebuilt from the source releases with the released converters, at the conversion cost that we now report (Table R4).

W1: ...the presented evidence does not sufficiently establish the readiness and quality of its dominant sources.

Q1: For WildGUI and ScaleCUA, please state what has been fully converted and checked in the final artifact...

We agree that the dominant sources should be investigated further. Both are converted in full: 12,437,103 of the 12,794,268 raw WildGUI trajectories pass all QA checks (97.2%), as do 29,005 of the 29,006 ScaleCUA trajectories with valid image archives (Table R2). Both rates are higher than those of 7 of the 14 sources that we train on. Most of the WildGUI trajectories that are filtered out contain no executable action, rather than an inherent data quality issue (Table R3). The authors of both sources also report training gains from their data. The WildGUI authors pre-train on WildGUI for about 200B tokens and report that "the full Stage1 + Stage2 pipeline achieves a success rate of 31.9%" on AndroidWorld, against 23.3% without the WildGUI stage (Xiong et al., 2026). Likewise, the ScaleCUA authors report that "training with our curated training corpus yields consistent improvements over the baseline trained on public data" (Liu et al., 2026a).

Q1: ...provide a small representative audit of screenshot–action alignment if available.

Every image referenced by a passing ScaleCUA trajectory opens (66,939 images, Table R2). For WildGUI, each trajectory must contain an executable action, which filters out 320,374 trajectories (Table R3), and each action must parse into the tool-call schema, which filters out a further 1,201. We manually audit 50 randomly-selected steps of WildGUI, drawn across all 6,059 of its frame archives, and find that the large majority of actions match their screenshots. In particular, we check timing and the action target. 48 of the 50 screenshots show the screen just before the action. Of the 30 steps whose actions should have coordinates, 26 are clearly on the intended interaction target that the text references. The converted coordinates of all 27 clicks also fall inside the bounding box that WildGUI provides for the target element. We will add this to the revision, and if the reviewer has any specific issues in mind or would like a different type of audit, we can provide that as well.

W2: ...without paired uncertainty estimates or repeated evaluation, it is difficult to determine the reliability of the reported mixture advantage.

Q2: Could you provide paired task-level uncertainty estimates for mixed-domain versus single-domain results, with success counts as well as percentages?

Q2: Please identify which mixture advantages remain supported by this analysis and qualify those that remain uncertain.

Thank you for this suggestion. We ran paired bootstrap tests at the task level for every comparison, and report them with success counts (Table R1). All nine mixture advantages remain supported: the three on-domain advantages have p = 0.041 (SWE-bench Verified), 0.026 (MiniWoB++), and 0.007 (AndroidWorld), and the six out-of-domain advantages have p < 0.01.

Q2: A few repeated evaluation rollouts of the existing checkpoints would strengthen the interactive results if feasible.

We repeated the evaluation of the existing mixed-domain and coding-only checkpoints on SWE-bench Verified, where the gap was smallest. The mixed-domain model resolves 15 and 16 instances in its two runs, against 11 and 9 for the coding-only model.

Sources

Thank you for clarifying the conversion status of each source. The WildGUI audit and repeated evaluation give me more confidence in the quality of those examples and in the mixture gains on the evaluated subset. My remaining concern is whether screenshot–action alignment is reliable across the broader corpus, particularly for ScaleCUA. Given how central data quality is to the contribution, I maintain my overall rating

Reviewer_vFArOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces the Multimodal Agent Data Protocol (MMADP), an extension of ADP that integrates screenshots, screen-element annotations, and environment feedback into a common representation for coding, web, and computer-use trajectories. The authors aggregate 16 existing datasets into a corpus estimated at 14.7 million trajectories and 366 billion tokens, with converters, validation checks, and an OpenHands-compatible training pipeline. SFT of InternVL2.5-MPO at 8B and 26B on a 250-million-token subset improves performance across three agent benchmarks; an equal-total-token comparison also favors mixed-domain over single-domain training. The principal contribution is reusable multimodal data infrastructure and curation, extending ADP’s largely text-based corpus to jointly accommodate coding and screenshot-based interaction. The empirical results provide initial evidence of utility, although they do not yet establish the benefits of the full corpus or isolate the mechanisms behind the mixed-domain gains.

Critical strengths
  1. Useful effort to unify coding, web, and computer-use datasets into a common format.
  2. Interesting results showing that mixed-domain training outperforms single-domain training under the same token budget.
Critical weaknesses
  1. Limited validation of the proposed corpus - This paper claims a 366B-token corpus, but experiments use only 250M tokens and exclude WildGUI, which accounts for around 93% of the corpus. I don't see evidence that the largest portion of the corpus is actually useful for training. The experiments also use supervised fine-tuning rather than the proposed agentic midtraining setting. Without evaluating the full corpus or a more representative subset, it is hard to assess the value of this corpus.
  2. Limited novelty and insufficient evidence for the main claims - MMADP mainly extends the existing ADP with screenshots and element annotations, but there is no ablation showing how much these additions contribute. The paper also claims that mixing domains improves performance, but the paper does not report results across multiple training seeds and report relatively few successful tasks. For example, swe-bench verified improves from 2.0% to 3.0%, corresponding to only 5 additional solved tasks. I don't see evidence that these improvements are statistically reliable. Without these experiments, it is hard to tell how much the proposed extensions and domain mixing actually contribute.
Questions for authors
  1. Can you evaluate the contribution of WildGUI and ScaleCUA under the same training budget?
  2. How much do screenshots and element annotations actually contribute? Can you provide an ablation removing these components to isolate the benefits of extending the existing ADP?
  3. Are the mixed-domain improvements statistically reliable? Can you provide results across multiple training seeds or confidence intervals, especially given the small number of successful tasks?
3 comments

We thank the reviewer for the careful assessment, and for recognizing MMADP as reusable multimodal data infrastructure that unifies coding, web, and computer-use datasets into a common format. We also appreciate the reviewer's interest in our finding that mixed-domain training outperforms single-domain training under the same token budget. We address the reviewer's feedback on the validation of the corpus, the contribution of our additions to ADP, and the statistical reliability of our results below.

W1: Without evaluating the full corpus or a more representative subset, it is hard to assess the value of this corpus.

Q1: Can you evaluate the contribution of WildGUI and ScaleCUA under the same training budget?

We thank the reviewer for raising this concern. We agree that the largest sources deserve explicit validation. To address this, we re-processed every source, including WildGUI and ScaleCUA, in full, and also note that the WildGUI authors demonstrate the value of their data via pretraining (Xiong et al., 2026). While we do not have the compute budget to run additional training experiments at the moment, we hope this can lighten the reviewer's concern on their value.

Our reprocessing shows that over 97% of the two datasets pass all QA checks (Table R2), which is more than 7 of the datasets that we train on. Of the trajectories filtered out, Table R3 shows that the issues primarily are a lack of action, rather than inherent data quality issues. Additionally, the WildGUI authors pre-train on WildGUI for about 200B tokens and report that the WildGUI stage raises AndroidWorld success chance from 23.3% to 31.9% (Xiong et al., 2026). Likewise, the ScaleCUA authors report that "training with our curated training corpus yields consistent improvements over the baseline trained on public data" (Liu et al., 2026a). These results demonstrate that both raw datasets are valuable.

W1: ...paper claims a 366B-token corpus, but experiments use only 250M tokens...

Thank you for raising this point. The corpus is sized for midtraining at the scale of large published runs, which is beyond our compute. For example, Kimi-Dev midtrains on about 150B tokens (Yang et al., 2025), LongCat-Flash-Thinking-2601 on 540B (Meituan LongCat Team, 2026), and GLM-5 on 1T, 500B, and 50B tokens across its three context-length stages (GLM-5 Team, 2026). Our experiments instead test whether the corpus teaches the behavior that these runs target, namely tool use across domains, at a budget we can afford (Section 4.1). The results demonstrate that 250M tokens, about 0.1% of the corpus, successfully teaches tool use in three domains (Section 5.1), with p < 0.01 against the 8B base on every benchmark (Table R1).

W1: experiments also use supervised fine-tuning rather than the proposed agentic midtraining setting

We appreciate this point, as it was a decision we spent some time on. We believe our experiment is a reasonable proxy for midtraining for two reasons: the base model has not seen data in a tool-calling or agentic format, and our setup follows published midtraining choices in its starting checkpoint, loss, and scale. InternVL2.5 predates agentic midtraining of many open-source models, such as GLM-4.5 (GLM-4.5 Team, 2025), and its report describes its training data as conversation-format instruction data, with no mention of tool calls (Chen et al., 2024). Accordingly, before training, only 4.0% of the 8B base's SWE-bench Verified tool calls and 4.8% of its AndroidWorld steps are valid (Table 3). This aligns with Liu et al. (2026b), who found that "midtraining benefits are largest for domains distant from general pretraining data, such as code and math."

Secondly, midtraining has no standard recipe. The concept remains "theoretically nebulous", with different papers using the term for different setups (Tu et al., 2025). Published midtraining runs vary widely, and our configuration matches a published choice in each respect:

  • GUI-CIDER midtrains Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct (Wu et al., 2026), both of which are instruct-tuned models like ours. Those models also had agentic post-training before release.
  • Singhal et al. (2026) compute loss "only on assistant turns, including tool calls", as we do.
  • GUI-CIDER midtrains on a subset of a corpus of about 100M synthesized tokens, which is less than our configuration (Wu et al., 2026).

Our runs follow a published choice on each of these, which is why we consider them a reasonable small-scale proxy.

W2: MMADP mainly extends the existing ADP with screenshots and element annotations, but there is no ablation showing how much these additions contribute.

Q2: Can you provide an ablation removing these components to isolate the benefits of extending the existing ADP?

Thank you for this question. It was not possible to train on these datasets together before MMADP, so the ablation is in effect the baseline of not training on them. ADP v1 (Song et al., 2026) cannot represent the nine new sources, which hold 13,332,277 of the 14,432,238 trajectories in the corpus (Table R2). The datasets contain images and annotations that are referenced in the action sets, meaning removing them would result in trajectories that have no tool call outputs. (Sections 3.2 and 3.3). A model that is trained on text-only data, with the image datasets dropped completely, is our coding-only model that is trained on sources shared with ADP (Section 3.5, Table 5). The mixed-domain model outperforms it on all benchmarks (Table R1). If the reviewer has a specific ablation in mind, we would be happy to run it before the end of the discussion period, compute permitting.

W2: ...the paper does not report results across multiple training seeds and report relatively few successful tasks.

Q3: Are the mixed-domain improvements statistically reliable? Can you provide results across multiple training seeds or confidence intervals...?

We agree that the reliability of these gains should be quantified. We ran a paired bootstrap test for every comparison, and found that all nine comparisons of the mixed-domain model against a single-domain model have p < 0.05 (Table R1). For the SWE-bench Verified gap that the reviewer highlights (3.0% vs 2.0%), we evaluated both models a second time with a different seed. Across the two runs, the mixed-domain model resolves a mean of 15.5 instances against 10.0 for the coding-only model (p = 0.041).

Sources

We thank the reviewer again for the careful assessment. As the discussion period closes today, we would like to remind the reviewer that our response is posted above. We would be glad to know whether the paired bootstrap tests (Table R1) and the full conversion of every source (Table R2) address the reviewer's concerns. If they do, we would appreciate it if the reviewer would consider raising the score. If any concern remains, please let us know, and we will respond promptly.

Comments on the paper

3 comments
AuthorsOpenReview ↗

To All Reviewers: New Tables

We thank the reviewers for their careful reading. Following the provided feedback, we provide four additional tables that clarify our findings. These will be added into new appendices.

Table R1: Paired Bootstrap Tests for Domain Ablations

We run paired bootstrap tests to determine one-sided p between the mixed-domain model and the single-domain models, following Berg-Kirkpatrick et al. (2012). For each, 1,000,000 resamples of benchmark tasks were drawn with replacement. Our hypothesis follows ADP (Song et al., 2026), which shows that mixed-domain models perform better. We find that this holds for multimodal models as well. Paired SE is the standard error of the mean per-task difference, following Miller (2024). Note that the mixed-domain and coding-only models were evaluated on SWE-bench Verified a second time with a different seed, as our initial result was particularly close.

Benchmark (task count)Mixed-domain model, against:Tasks solvedPaired SEp
SWE-bench Verified (500)Coding-only15.5 vs 10.00.65%0.041
Web-only15.5 vs 00.64%< 0.0001
Computer-use-only15.5 vs 00.64%< 0.0001
8B base15.5 vs 00.64%< 0.0001
MiniWoB++ (99)Web-only32 vs 253.87%0.026
Coding-only32 vs 04.72%< 0.0001
Computer-use-only32 vs 04.72%< 0.0001
8B base32 vs 144.39%< 0.0001
AndroidWorld (116)Computer-use-only7 vs 12.07%0.007
Web-only7 vs 12.07%0.007
Coding-only7 vs 02.22%0.004
8B base7 vs 02.22%0.004

Table R2: Release Status of the 16 Sources

To make the conversion and QA status of each source explicit, we (re-)process every source through our pipeline, in full. This includes a few new QA checks, so numbers may not match the paper exactly. We provide exact counts of the number of inputs (see "Unit") that are in each stage. These supersede the sample-based estimates of WildGUI, ScaleCUA, and Orca AgentInstruct in Table 1 of the paper; we will update the paper accordingly. We then provide the number of trajectories in each stage of the pipeline, the number of images each dataset references, and the percentage of trajectories that pass all our QA checks. Note that duplicates are counted multiple times in the source, and deduplicated in the conversion process.

SourceUnit# in SourceSTD Converter Subset (method)# Raw Traj.# STD Traj.# SFT Traj.# Ref. Images% Kept, Raw
SWE-smithrow76,00232,360 (rows the source marks resolved)32,3609,9399,939030.7%
SWE-Gymtrajectory6,0556,055 (all)6,05549149108.1%
OpenHandstrajectory275275 (all)2751271146841.5%
Code-Feedbackdialog66,38366,383 (all)66,38366,38330,213045.5%
CodeActInstructconversation78,3857,139 (codeact split)7,1397,1397,099099.4%
Go-Browsestep row196,462196,462 (all)26,7506,9536,26024,25723.4%
WebLINXdemonstration2,337969 (train split)96996984014,76686.7%
Multimodal-Mind2Webaction row14,1937,775 (train split)1,0099629627,47595.3%
AgentNettrajectory22,62516,057 (completed tasks with an alignment score >= 7)16,05716,05716,057308,655100%
OmniACTtask9,8026,789 (train split)6,7896,7896,60617797.3%
AndroidControlepisode15,28313,603 (train split)13,60313,60313,59488,30899.9%
AITWstep record5,689,8755,689,875 (all)716,034715,110711,0715,682,64499.3%
LLaVA-Plusconversation117,039117,039 (all)117,039117,039117,03975,110100%
Orca AgentInstructconversation1,046,4101,046,410 (all)1,046,4101,046,4101,045,845099.9%
ScaleCUAtrajectory38,08529,006 (batches with valid image archives)29,00629,00629,00566,939> 99.9%
WildGUIepisode12,812,25812,812,258 (all)12,794,26812,472,69312,437,103119,857,89997.2%
All 16 sources14,432,238

Table R3: Common Causes for Filtered Trajectories

We list the ten most common reasons for which trajectories are filtered out in our pipeline, along with the relevant stage and affected datasets. Together, they account for 462,110 of the 466,195 dropped trajectories (99.1%). Extraction creates raw trajectories, conversion turns them into standardized (STD) formats, and QA occurs alongside the conversion to the OpenHands SDK (SFT) format.

StageReasonTrajectoriesSources (trajectories)
conversionno step has an executable action320,374WildGUI
QAevery tool call performs no action76,172Code-Feedback (36,170), WildGUI (35,223), AITW (4,039), Go-Browse (693), CodeActInstruct (30), AndroidControl (9), OpenHands (7), ScaleCUA (1)
conversionsource marks the trajectory as unsuccessful17,245Go-Browse
conversiontrajectory does not end in a submit call14,980SWE-smith
extractiona step lacks a required field12,328WildGUI
conversionduplicate of another trajectory8,350SWE-smith (7,439), AITW (911)
conversionsource marks the trajectory as unresolved5,564SWE-Gym
extractionstep holds another episode2,910WildGUI
extractionmalformed timestamp2,563WildGUI
conversiona step action does not parse1,624Go-Browse

Table R4: Conversion Cost

Lastly, we report the time cost of converting each dataset's raw form to the final OpenHands SDK format. All datasets were converted on machines with at most 8 cores and 64GB RAM. WebLINX, Multimodal-Mind2Web, and AgentNet conversions rendered webpages via Playwright for accurate element extraction, leading to longer runtimes from page rendering waits.

Source# Raw Traj.Wall time
Multimodal-Mind2Web1,0095 h 39 min
WebLINX9695 h 31 min
AgentNet16,0572 h 18 min
WildGUI (annotations)12,794,2681 h 48 min
AITW716,03458 min
Orca AgentInstruct1,046,4109 min
Go-Browse26,7507 min
AndroidControl13,6036 min
LLaVA-Plus117,0394 min
ScaleCUA29,0062 min
SWE-smith32,36066 s
OpenHands27552 s
Code-Feedback66,38328 s
SWE-Gym6,05515 s
CodeActInstruct7,13911 s
OmniACT6,7899 s

Sources

AuthorsOpenReview ↗

To Reviewers and AC

We thank the reviewers and the area chair for their time and feedback. We appreciate that the reviews recognize MMADP as reusable infrastructure that unifies coding, web, and computer-use datasets (vFAr, XmSk) and as a valuable, extensible open resource (UoSQ). We are also glad that the reviews highlight the fixed-budget comparison of mixed-domain and single-domain training (vFAr, XmSk). As the discussion period closes today, we briefly summarize how we addressed the main points raised.

  • Reliability of the mixture advantage (vFAr, XmSk): all nine comparisons hold under a paired bootstrap test (p < 0.05), reinforced by a repeated evaluation on SWE-bench Verified (Table R1).
  • Status and quality of the corpus (vFAr, XmSk): every source has been converted fully, with exact trajectory counts at each stage (Table R2). In manual audits of WildGUI, ScaleCUA, and Android in the Wild (AITW), 104 of the 115 judged actions match their screenshots. Additional information on QA filtering is provided in Table R3.
  • SFT as a proxy for midtraining (vFAr, UoSQ): we clarify that the base model has no tool-calling training, and our setup generally matches published midtraining choices.
  • Conversion cost (UoSQ): all 16 sources convert in 16 h 46 min of total wall time, with no GPU (Table R4).

If there is any last-minute feedback or remaining concern, please let us know, and we will respond promptly. We hope these clarifications and additions address the reviewers' concerns, and we will include them in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.