A Foundation Model-Powered Dual-Transformer Framework for Centralized Multi-Agent Systems

Multi-agent systems require explicit agent-wise feature representations and structured interaction modeling, while pretrained vision-language models primarily produce global group-wise multimodal feature representations whose token structure and feature dimensions do not naturally align with individual agents. To bridge this gap, we propose a unified foundation model-based dual-transformer framework FMDT for centralized multi-agent system. The framework combines a agent-wise state transformer encoder with a multimodal VLM encoder through cross-attention. The agent-centric features produced by the state encoder transformer serve as queries over global multimodal VLM features, enabling agent-wise reasoning while retaining rich scene-level context. We instantiate this framework as TrajVLM for centralized multi-agent long-horizon trajectory forecasting and CMAVLA for centralized multi-agent single-step decision making. TrajVLM is optimized through supervised fine-tuning followed by reinforcement learning-based refinement, whereas CMAVLA is trained through centralized multi-agent reinforcement learning for cooperative control. Beyond the algorithmic framework, we provide theoretical guarantees by deriving covering number and Rademacher complexity bounds for the proposed dual-transformer framework hypothesis class, which further yield uniform convergence and sample complexity. Experiments on ETH-UCY for trajectory forecasting and MaMuJoCo for cooperative multi-agent control demonstrate that the proposed framework achieves competitive or state-of-the-art performance against current representative baselines. These results suggest that FMDT provides a general mechanism for adapting foundation models to tackle centralized multi-agent learning problems. More details can be found in the project website: https://sites.google.com/view/cmas-fmdt.