acceptodds
Under review as a conference paper at ICLR 2027

A Foundation Model-Powered Dual-Transformer Framework for Centralized Multi-Agent Systems

Abstract

Multi-agent systems require explicit agent-wise feature representations and structured interaction modeling, while pretrained vision-language models primarily produce global group-wise multimodal feature representations whose token structure and feature dimensions do not naturally align with individual agents. To bridge this gap, we propose a unified foundation model-based dual-transformer framework FMDT for centralized multi-agent system. The framework combines a agent-wise state transformer encoder with a multimodal VLM encoder through cross-attention. The agent-centric features produced by the state encoder transformer serve as queries over global multimodal VLM features, enabling agent-wise reasoning while retaining rich scene-level context. We instantiate this framework as TrajVLM for centralized multi-agent long-horizon trajectory forecasting and CMAVLA for centralized multi-agent single-step decision making. TrajVLM is optimized through supervised fine-tuning followed by reinforcement learning-based refinement, whereas CMAVLA is trained through centralized multi-agent reinforcement learning for cooperative control. Beyond the algorithmic framework, we provide theoretical guarantees by deriving covering number and Rademacher complexity bounds for the proposed dual-transformer framework hypothesis class, which further yield uniform convergence and sample complexity. Experiments on ETH-UCY for trajectory forecasting and MaMuJoCo for cooperative multi-agent control demonstrate that the proposed framework achieves competitive or state-of-the-art performance against current representative baselines. These results suggest that FMDT provides a general mechanism for adapting foundation models to tackle centralized multi-agent learning problems. More details can be found in the project website: https://sites.google.com/view/cmas-fmdt.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Foundation Model, VLA, MARL, Transformer, Learning Theory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kxLCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tries to plug a foundation VLM into centralized multi-agent systems. The idea is simple: run each agent's history through a small state transformer to get an agent-wise query, then cross-attend those queries against the VLM's token features, and decode per-agent outputs. They build two versions—one for pedestrian trajectory forecasting on ETH-UCY, one for cooperative multi-agent control on MaMuJoCo. They also state some generalization bounds (covering number, Rademacher, sample complexity), but the proofs are kicked out to a project site. Results: decent on MaMuJoCo, not actually SOTA on ETH-UCY.

Critical strengths

The agent-as-query over VLM-K/V design is a clean way to keep explicit agent identity while still reading global multimodal context—better than flattening VLM tokens or feeding per-agent crops.

Critical weaknesses

The four theorems are headline contributions but the proofs and assumptions are deferred to an external project website, so they cannot be verified in review. Either the proofs belong in the paper, or the theory claims should be down-weighted.

Questions for authors

Please include the theorem proofs and assumptions in the rebuttal (or in the camera-ready appendix). As submitted they are uncheckable.

1 comment

Thanks for the comments!

The proof of theorems has been uploaded and submitted that can be found in the Appendix.

I mainly leveraged existing theoretical technologies to introduce proposed FMDT framework. Additionally, the theory to algorithm experiemnts also have been conducted.

Reviewer_vjCLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the mismatch between global VLM features and agent-wise representations, the authors propose FMDT, a dual-transformer framework for centralized multi-agent trajectory forecasting and cooperative control. It uses agent-wise state features to query VLM features through cross-attention. Experiments show competitive trajectory prediction performance and higher reported control returns than the state-only ablations.

Critical strengths
  1. Combining the scene understanding of a VLM with the state modeling of multi-agent systems is a promising idea worth exploring. This paper presents a relatively intuitive implementation method and applies it to trajectory prediction and cooperative control, which may be useful for related research.
  2. The experimental results are quite encouraging. Especially in the cooperative control task, the model with the VLM added shows a clear improvement compared to the model using only state input, indicating that this direction is worthy of further research.
Critical weaknesses
  1. The authors think that combining two Transformers can take into account both fine-grained motion information and high-level semantic information, but the trajectory prediction part mainly just compares the complete model with other methods, and lacks corresponding ablation analysis. At this point, I can't tell whether these results come from the information complementarity that the authors emphasize.
  2. The authors highlight that sharing a VLM can reduce computation and device costs, but there's no comparison with related VLA methods in terms of runtime resource usage. It's still unclear how many resources can actually be saved or whether this brings any advantage in inference efficiency.
  3. The idea of combining a VLM with a multi-agent model is reasonable, but it hasn't been fully explained yet what significant improvements this design brings compared to existing fusion and modeling methods. I'm not quite sure what the new contributions are relative to existing work.
Questions for authors
  1. Can the authors provide the ablation results for the trajectory prediction part, showing how much benefit the VLM branch and the dual Transformer fusion each bring?
  2. Compared to existing VLA methods, how much computation and device overhead does the shared VLM actually save? Are there any comparisons of inference time or memory usage?
  3. Compared to the closest existing work, what are the main new contributions of this paper? In particular, how is the per-agent fusion design significantly different from existing fusion methods?
1 comment

Thanks for the comments!

For the question-1: We add the ablation model as STTN+FM in the trajectory prediction tasks.

For the question-2: To address this issue, the ablation model Concate-CMVLA has been introduced and evaluated in the experiments part, which the cross-attention network is replaced to a concatenation fusion mechanism.

For the question-3: Our main contribution is a general VLM-based multi-agent framework that enables a single shared VLM to support multiple agents, rather than requiring separate foundation models for individual agents. FMDT integrates agent-specific state representations with shared VLM features through per-agent cross-attention, enabling centralized multimodal reasoning while preserving agent-specific information. By applying FMDT to both long-horizon trajectory forecasting and short-horizon cooperative decision-making, we demonstrate that this unified VLM-based framework can be effectively trained across different multi-agent tasks and has the potential to improve performance.

Reviewer_XjLSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FMDT, a dual-transformer framework for centralized multi-agent learning. A state transformer produces agent-indexed features, which query a shared vision-language model through cross-attention. The framework is instantiated as TrajVLM for multi-agent trajectory forecasting and CMAVLA for cooperative continuous control. TrajVLM combines a flow-matching decoder with supervised training and GRPO refinement, while CMAVLA uses agent-specific Gaussian action heads trained with a centralized group-relative policy objective.

The paper also presents covering-number and generalization bounds for the resulting hypothesis class. Experiments evaluate trajectory prediction on ETH-UCY and control on MaMuJoCo, with state-only ablations for the control setting.

Critical strengths
  1. The architectural motivation is clear. Using agent-indexed state features as queries provides a reasonable interface between a VLM's token representations and outputs associated with individual agents. This makes the proposed coupling straightforward to understand and applicable to different continuous-output tasks.

  2. The paper tests the framework on both forecasting and control. The improvements over the state-only control ablations provide an encouraging signal that the full system can be useful. Evaluating two task types gives the proposal broader empirical grounding, although the comparison issues below limit the conclusions that can be drawn.

Critical weaknesses

1. The trajectory-prediction method is incompletely specified, and its empirical advantage is not convincing.

The connection between flow matching and GRPO is insufficiently explained. Section 3.2 and Eq. (4) define a conditional flow-matching objective, but Eq. (6) subsequently requires policy likelihood ratios and KL regularization. The submission does not explain how these quantities are computed for its flow-based trajectory distribution. In particular, the decoder is described as modeling the joint future trajectories of all agents, whereas the GRPO ratio uses the probability of an individual agent's position at a particular future timestep. The relationship between the joint distribution and these position-level probabilities, any factorization or likelihood approximation, and the implementation of the KL term need to be specified. These are essential details for reproducing and assessing the proposed refinement procedure.

The foundation-model comparison is also relatively narrow. Several entries in Table 1 are zero-shot or supervised variants from the LMTraj/VLMTraj research line, with limited coverage of alternative foundation-model approaches. The broader positioning relative to predictive world models is not sufficiently developed. Although DIMA is included in the control comparison, the paper does not sufficiently discuss how its forecasting component relates to other applicable state-conditioned predictive or world-model approaches. The authors should clarify which world-model approaches are applicable to their setting and provide appropriate comparisons, or explain the boundary of the claimed scope.

Finally, the trajectory results do not establish a clear overall improvement. Table 1 reports average ADE20/FDE20 of 0.20/0.33 for TrajVLM, compared with 0.20/0.32 for MoFlow, 0.18/0.27 for VLMTraj-SUP, and 0.13/0.18 for SocialMOIF. Improvements on selected scenes do not establish a compelling overall benefit from adding a large VLM and a further RL stage. A controlled comparison of the same state encoder and flow decoder with and without the VLM, together with SFT-only versus SFT-plus-GRPO results, is needed to identify the contribution of these components.

2. The theoretical analysis is largely disconnected from the forecasting and control contributions.

The results in Section 4 provide conditional complexity and uniform-convergence bounds for the composed hypothesis class. However, they do not explain when the proposed dual-transformer coupling should improve trajectory prediction or control, or why cross-attention should be preferred over simpler fusion mechanisms. The analysis does not provide a comparative guarantee against relevant alternatives or actionable guidance for choosing the architecture, adaptation strategy, or training objective.

The assumptions also remain insufficiently connected to the actual learning procedures. In particular, bounded Lipschitz losses and i.i.d. sampling do not automatically describe the flow-matching loss or the changing data distributions encountered during on-policy control. The paper does not establish how the stated bounds apply to these objectives or translate into a guarantee for policy return.

As presented, the theory reads as a largely standalone addition to the empirical framework. My concern is its relevance to the main contribution: the paper would be stronger if the analysis explained a concrete property of the proposed coupling, predicted an experimentally testable effect, or directly justified part of the training procedure.

3. The MARL evaluation does not establish a fair or practically compelling advantage over lightweight alternatives.

The comparison combines several substantial advantages for CMAVLA that are not adequately controlled. Its VLM backbone benefits from extensive external pretraining, while the conventional MARL policies are trained without a comparable pretrained backbone. In addition, Section 5.1 states that CMAVLA initializes its state encoder from a trained Trans-CMAGRPO checkpoint. Consequently, the current experiments do not separate the contribution of the proposed architecture from those of pretrained representations, additional task-specific initialization, and much greater model capacity. The cost of this initialization and the benefit of each source of prior training should be accounted for explicitly.

The training horizon further limits the interpretation. The submission reports conventional MARL results at 1M environment steps, whereas the official HASAC experiments, including the matching tasks in Figure 8, evaluate learning over 10M steps. A short-budget evaluation can assess early learning, but it does not establish superiority after fuller training. More seriously, the submission explicitly counts only primary trajectories toward CMAGRPO's nominal 1M budget and excludes additional grouped rollouts. Thus, the reported comparison does not establish equal environment-interaction budgets. Matching the number of reported updates is not sufficient without defining the underlying sampling and optimization work. Results at both 1M and 10M actual interactions would provide a much more informative comparison.

The statistical reporting also favors the proposed method: Table 2 compares single-seed maximum returns for the centralized methods against multi-seed means and standard deviations for the reproduced conventional baselines. This does not establish an improvement in expected performance or training reliability.

Finally, the large backbone introduces substantial model-size and computational overhead. The paper acknowledges increased memory usage and training time, but does not quantify the trade-off through GPU-hours, memory requirements, inference latency, or achievable control frequency. For a method requiring centralized execution, the practical value of retaining a large VLM in the control loop needs to be established against strong lightweight centralized policies. Without comparable training budgets and measured deployment costs, I do not find the proposed CTCE design a sufficiently convincing choice for multi-agent decision making.

Questions for authors
  1. Flow matching and GRPO: How are the likelihood ratios and KL term in Eq. (6) computed? Does the implementation use a joint trajectory density, position-level marginals, conditional factors, or a surrogate objective? Please explain the exact connection to the flow-matching decoder.

  2. Trajectory evidence: What are the results for the same state encoder and flow decoder without the VLM, and for SFT-only versus SFT-plus-GRPO training? How do the authors position TrajVLM relative to relevant predictive world-model approaches beyond the existing trajectory baselines?

  3. Computational value: What are the total and trainable parameter counts, training GPU-hours, peak memory, and per-step inference latency relative to the strongest lightweight centralized baseline? Does the performance advantage remain compelling when these resource differences are taken into account?

1 comment

Thanks for the comments!

For the questions-1: Following Flow-GRPO (NeurIPS 2025), we introduce Gaussian stochastic transitions into the flow-matching sampling process, allowing likelihood ratios and KL divergence to be computed analytically at each flow step. We define these quantities over the complete trajectory generation state rather than individual future positions, thereby avoiding the unsupported position-level factorization in the original formulation.

For the questions-2: We have updated the SFT-only and SFT-RL data from our model, with an ablation model STNN+FM.

For the questions-3: We have conducted the inference efficiency experiments, which can be found in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.