MMADP: Unified Multimodal Agent Trajectories for Open Agentic Training

Many open-weight models learn to use tools through agentic midtraining, but the datasets used at this stage are rarely released. Open datasets could support research on this stage. Existing efforts to combine them focus on text-only trajectories or individual domains. We introduce the Multimodal Agent Data Protocol (MMADP) to support multimodal agent training across coding, web, and computer-use tasks. MMADP builds on the Agent Data Protocol (ADP) and adds support for screenshots and annotations of screen elements. Using MMADP, we combine 16 datasets into a corpus with an estimated 14.7M trajectories and 366B tokens, the largest open corpus of agentic trajectories to our knowledge. Our experiments show that a small sample of this corpus can teach tool use across domains. We perform supervised fine-tuning of InternVL2.5-MPO on a 250M-token sample, starting from models with no prior tool-calling training. Both the 8B and 26B models improve on coding, web, and computer-use benchmarks. At 8B, the absolute gains are +3.0% on SWE-bench Verified, +18.2% on MiniWoB++, and +6.0% on AndroidWorld. We also find that mixing domains helps within each domain: with 250M tokens sampled for each model, the mixed-domain 8B model outperforms single-domain models on all three benchmarks. For each domain, the mixed-domain model uses only about one-third of the training tokens used by the corresponding single-domain model. We release the schema, converters, and full corpus to support open research on multimodal agentic midtraining.