acceptodds
Under review as a conference paper at ICLR 2027

Learning Coordinated Representations for Flow-Based Multi-Agent Imitation Learning

Abstract

Cooperative multi-agent systems require decentralized agents to coordinate seamlessly to accomplish shared tasks. Learning from multi-modal joint demonstrations that exhibit diverse strategies makes multi-agent imitation learning particularly challenging: independently executed policies may select individually plausible actions that accidentally mix distinct strategies, leading to inconsistent joint behaviors. We address this challenge by introducing the latent-conditioned factorization gap, which measures the divergence between a centralized joint policy and a product of latent-conditioned decentralized policies. We show that, with sufficiently expressive local policies, this gap decomposes into conditional total correlation among agents' actions and conditional mutual information terms capturing dependence on unobserved global information. To efficiently optimize this objective with flow-matching policies, we derive a practical, density-free surrogate based on joint-to-local vector field alignment. Extensive experiments on Multi-Agent MuJoCo and cooperative multi-quadruped benchmarks demonstrate that our approach significantly outperforms existing baselines in preserving coordinated multi-modal behavior.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
flow-matching models, multi-agent imitation learning, multi-modal demonstrations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xHW8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoFac-FM (Coordinated Factorized Flow-Matching) to address coordination failures in multi-agent imitation learning from multi-modal demonstrations. The core idea is to let a shared latent variable capture the information that agents need to agree on, so that each agent can follow a consistent joint strategy using only its local observations. The main innovation is to explicitly train this latent representation to make the joint behavior factorizable into decentralized policies, using the discrepancy between joint and local flow vector fields as the learning signal. The authors validate the method through a toy experiment and cooperative control tasks in Multi-Agent MuJoCo and MQE, comparing task performance and the recovery of different expert behavior modes against Gaussian, latent-conditioned, and flow-based baselines.

Critical strengths
  1. The paper gives the shared latent representation a clear objective: capturing the dependencies needed for agents to follow a consistent joint strategy using only local observations. The factorization gap formalizes this requirement, and joint-to-local vector field alignment provides a practical way to train the representation and decentralized policies.

  2. Figure 1 clearly shows that the method separates the two coordinated behavior modes under different latent values and avoids incompatible action combinations. This supports the intended effect of the proposed representation learning. Tables 1 and 2 provide further evidence of effectiveness, including consistently lower overall MMD across the eight Multi-Agent MuJoCo configurations and substantial success-rate improvements on some partially observable MQE tasks.

Critical weaknesses
  1. The experiments do not fully isolate the contribution of training the latent representation with the factorization objective. The LatentMF baselines train local policies solely through distillation, whereas the proposed method also uses direct imitation training. An ablation that blocks only the factorization-loss gradient to the encoder, while keeping the remaining setup unchanged, would more clearly establish how much of the improvement comes from the proposed latent learning mechanism.

  2. In Table 2, Door-P and Cargo Door-P have identical success rates, MMD values, and standard deviations for every method. These results should be checked and clarified, since the two environments involve different coordination tasks.The overall assessment remains positive, although these issues require clarification to strengthen confidence in the experimental conclusions.

Questions for authors
  1. Could the authors provide a controlled ablation that blocks only the factorization-loss gradient to the encoder while keeping all other training components unchanged, including direct imitation training of the local policies? This would help clarify how much of the improvement comes from learning the coordination representation through the proposed objective.

  2. Could the authors clarify why Door-P and Cargo Door-P have identical success rates, MMD values, and standard deviations for every method in Table 2? If this is a reporting error, please provide the corrected results.

2 comments

(W1, Q1 An ablation that blocks only the factorization-loss gradient to the encoder) We perform the ablation studies, by applying the stop-gradient operator in the encoder indicated by ϕˉ\bar{\phi}.

MethodHC (2x3)HC (6x1)Ant (2x4)Ant (4x2)AG (2x4)AG (4x2)MSS (5x4)MSS (10x2)
LFMELBO(ϕ,θ)+ΔFMfac(ϕˉ,θ)\mathcal{L}_\mathrm{FM}^\mathrm{ELBO}(\phi, \theta)+\Delta_\mathrm{FM}^\mathrm{fac}(\bar{\phi}, \theta)1.001 (−0.065)\color{red}{\mathbf{(-0.065)}}0.889 (−0.068)\color{red}{\mathbf{(-0.068)}}0.299 (−0.357)\color{red}{\mathbf{(-0.357)}}0.371 (−0.184)\color{red}{\mathbf{(-0.184)}}0.409 (−0.312)\color{red}{\mathbf{(-0.312)}}0.411 (−0.299)\color{red}{\mathbf{(-0.299)}}0.564 (+0.029)\color{blue}{(+0.029)}0.342 (−0.153)\color{red}{\mathbf{(-0.153)}}
Ours1.0660.9570.6560.5550.7210.7100.5350.495

Although the MSS (5x4) task shows a slight performance increase, these differences remain within the std (Ours std=0.106 in MSS 5x4 ). It exhibits the significant performance drop, demonstrating that training the encoder via ΔFMfac\Delta_\mathrm{FM}^\mathrm{fac} is a critical factor for successful decentralized execution.

(W2, Q2 Duplicated results) Thank you for pointing that out. We fixed Door-P and the resulting is the following:

MethodGaussian-BCGMM-ELBOGMM-ELBO-MIFM-BCLatentFM-ELBOLatentFM-ELBO-MIMAC-FlowLatentMF-ELBOLatentMF-ELBO-MIOurs
Door-P Succ.0.0000.1060.1250.1750.1180.1560.0620.1120.1180.456

As the discussion period is drawing to a close, please let us know if you have any remaining concerns.

Reviewer_B7XNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoFac-FM, a latent-conditioned flow-matching approach for offline multi-agent imitation learning from multi-modal demonstrations. It defines a latent-conditioned factorization gap, the KL divergence between a centralized joint policy and the product of decentralized policies conditioned on a shared latent, shows that this gap decomposes into conditional total correlation and conditional mutual information terms, and replaces it with a density-free surrogate that aligns each agent’s vector field with the corresponding component of the joint vector field. Experiments on MAMuJoCo and MQE show higher normalized returns and lower MMD than Gaussian, flow-based and distillation baselines on most tasks.

Critical strengths

S1. The motivation is clear, and the problem of recovering coordinated multi-modal behavior under decentralized execution is of practical interest.

S2. Extensive experiments are conducted on two benchmarks.

Critical weaknesses

W1. The main comparison in Table 2 is problematic. The Door-P and Cargo Door-P columns are identical cell for cell for every method, including the standard deviations, and panels (b) and (f) of Fig. 10 appear identical as well, although the two tasks differ in the number of agents and in the coordination required. Which of the two settings do these numbers belong to, and what are the results for the other?

W2. The experiments do not fully support the main claim. First, the closest related method, MIMIC-D [1], which addresses the same problem with decentralized diffusion policies and is cited in the related work, is not compared against, and the only distillation baselines are MAC-Flow [2] and its latent variants, all adapted by the authors from offline RL. Second, on the Cargo tasks, which have the most agents in MQE, the method’s MMD is worse than that of Gaussian and flow-based behavior cloning with comparable success rates, which is not discussed, and on MAMuJoCo the normalized return is within seed noise of the best baseline on most tasks, so the advantage rests on the MMD metric.

W3. The novelty is limited. The method is essentially a combination of latent-conditioned decentralized policies with a shared coordination latent [3] and the joint-to-local vector field distillation of MAC-Flow [2], and the only new element is that the distillation residual is also used to train the encoder, which is not ablated. Theorem 2.2 restates, in flow-matching time, the identity between relative entropy and the integrated excess mean-square error of the mismatched conditional-mean estimator in Gaussian noise [4], which is not cited.

W4. The explanation of why the ELBO and MI objectives do not capture coordination does not hold. The latent model behind Eq. 4 is a mixture of product distributions, so its maximum-likelihood assignment of trajectories to z is already the one under which the joint behavior comes closest to factorizing, and the two objectives differ in their training signal rather than in what they ask of z. Whether the failure in Fig. 1 comes from the objective or from its optimization is therefore not established, and the weight of the KL term in Eq. 4, which pushes the posterior toward uniform, is not stated. Do the latent baselines still fail when this weight is reduced or annealed?

Minor comments:

  • Eq. 4 is called an ELBO, but the flow-matching loss replaces the log-likelihood, so the bound no longer holds and the weight of the KL term becomes a hyperparameter that is not reported.
  • Alg. 1: how the gradient of Eq. 8 reaches the encoder through the discrete latent, the loss weights, the ratio between the two updates, learning rates, network sizes and training iterations are not given.
  • Sec. 3.1 and Fig. 3: it is not stated how the per-mode MMD is computed for methods without a latent variable.
  • Table 1: “LatetnMF” should read “LatentMF”.
  • Related work appears only in Appendix A.

References:

[1] MIMIC-D: Multi-modal Imitation for MultI-agent Coordination with Decentralized Diffusion Policies. arXiv, 2025.

[2] Multi-agent Coordination via Flow Matching. ICLR, 2026.

[3] Coordinated Multi-Agent Imitation Learning. ICML, 2017.

[4] Mismatched Estimation and Relative Entropy. IEEE Trans. Inf. Theory, 2010.

Questions for authors

Please refer to the Weaknesses above.

4 comments

(Weakness 1 Duplicated results) Thank you for pointing that out. We fixed Door-P and the resulting is the following:

MethodGaussian-BCGMM-ELBOGMM-ELBO-MIFM-BCLatentFM-ELBOLatentFM-ELBO-MIMAC-FlowLatentMF-ELBOLatentMF-ELBO-MIOurs
Door-P Succ.0.0000.1060.1250.1750.1180.1560.0620.1120.1180.456

(W2-1 Comparison with MIMIC-D and MIFQ) MIMIC-D trains the individual diffusion policies to capture multi-modal behavior distribution. Meanwhile, MIFQ is an OfflineIL algorithm that learns a joint Q function by incorporating inverse bellman operators while enforcing individual-global consistency. Specifically, MIFQ represents the joint Q-value by combining individual Q-values through a monotonic mixing function. The following tables summarize the normalized return and success rate of them:

MethodHC (2x3)HC (6x1)Ant (2x4)Ant (4x2)AG (2x4)AG (4x2)MSS (5x4)MSS (10x2)
MIMIC-D1.0430.8820.4000.5050.5220.4350.5660.459
MIFQ1.0350.8460.1130.2690.1120.1070.5130.452
Ours1.0660.9570.6560.5550.7210.7100.5350.495
MethodDoorDoor-PSeesawSeesaw-PCargo DoorCargo Door-PCargoCargo-P
MIMIC-D0.7930.1810.3340.2120.3060.2060.4120.334
MIFQ0.7810.0370.3150.0810.3000.1500.4030.368
Ours0.8190.4560.3560.3880.4250.5010.4870.413

The detailed results are included in Table 1, 2. Figure 11 in appendix shows the result of MIMIC-D and MIFA in toy-domain, which also exhibit mode-averaging and uncoordinated joint behavior. The primary bottleneck of these two methods is that neither utilizes the shared latent variable z during execution, which is critical when handling datasets with diverse strategies. As we discussed in the paper, increasing the expressiveness of individual policy models alone cannot solve this issue. Furthermore, while MIFA trains a joint Q-function, it still struggles to represent diverse strategies because value-based methods inherently assume a unique optimal action, ultimately leading to a failure in generating coordinated joint actions.

(W2-2 Comparison with Gaussian-BC and Flow-BC in Cargo Task) We evaluate all latent-condition methods, including ours, across all latent variables z, where the total number of latent is a hyperparameter and performance varies by z. On Cargo task (4 direction behaviors in the dataset) we use 8 latent variables for our method yields the following per-latent success rates:

Seed 0z0z_0z1z_1z2z_2z3z_3z4z_4z5z_5z6z_6z7z_7BC
Our Succ.0.440.330.660.20.50.770.540.10.5 (mean)

As shown z1z_1, z3z_3, and z7z_7 yield comparatively poor performance, which can occurs when very few data samples are assigned to those latent variables during training. Similarly, Cargo-P shows the same trend, resulting in higher MMD values. Even though we sample from p(z)p(z) using the dataset posterior, underperforming latents can still be drawn during evaluation. Incorporating post-hoc adaptation techniques, such as selecting latents via online environment interaction, can further boost performance, enabling our method to surpass latent-free baselines like Gaussian-BC and Flow-BC.

(W2-2 MaMuJoCO) While our method’s normalized score falls within the margin of std of the second best scores, this metric alone can not fully capture the learned behavior. Specifically, a high normalized score does not guarantee diverse behavior, as a model might succeed by collapsing to a single mode. To account for this, we utilize MMD over all generated samples to quantify diversity. Furthermore, we assess controllability by reporting the worst and average MMD across modes, evaluating the alignment between specific modes and the samples generated given a fixed latent variable z. Ultimately, our method exhibits broad diversity and highly distinct, controllable behavior for each z without sacrificing performance.

(W3-1 Ablation studies in encoder training) To evaluate the contribution of ΔFMfac\Delta_\mathrm{FM}^\mathrm{fac} in encoder training, we perform the ablation studies by selectively applying the stop-gradient operator, indicated by ϕˉ\bar{\phi} and θˉ\bar{\theta}. For encoder ϕ\phi and individual policies θ\theta, the following results summarize the normalized return and followed by its difference from Ours in (⋅)(\cdot):

MethodHC (2x3)HC (6x1)Ant (2x4)Ant (4x2)AG (2x4)AG (4x2)MSS (5x4)MSS (10x2)
LFMELBO(ϕˉ,θ)+ΔFMfac(ϕ,θˉ)\mathcal{L}_\mathrm{FM}^\mathrm{ELBO}(\bar{\phi}, \theta)+\Delta_\mathrm{FM}^\mathrm{fac}(\phi, \bar{\theta})1.064 (−0.002)\color{red}{(-0.002)}0.901 (−0.056)\color{red}{(-0.056)}0.606 (−0.050)\color{red}{(-0.050)}0.524 (−0.031)\color{red}{(-0.031)}0.709 (−0.012)\color{red}{(-0.012)}0.701 (−0.009)\color{red}{(-0.009)}0.559 (+0.024)\color{blue}{(+0.024)}0.514 (+0.019)\color{blue}{(+0.019)}
LFMELBO(ϕˉ,θ)+ΔFMfac(ϕ,θ)\mathcal{L}_\mathrm{FM}^\mathrm{ELBO}(\bar{\phi}, \theta)+\Delta_\mathrm{FM}^\mathrm{fac}(\phi, \theta)1.068 (+0.002)\color{blue}{(+0.002)}0.951 (−0.006)\color{red}{(-0.006)}0.611 (−0.045)\color{red}{(-0.045)}0.546 (−0.009)\color{red}{(-0.009)}0.705 (−0.016)\color{red}{(-0.016)}0.699 (−0.011)\color{red}{(-0.011)}0.540 (+0.005)\color{blue}{(+0.005)}0.497 (+0.002)\color{blue}{(+0.002)}
LFMELBO(ϕ,θ)+ΔFMfac(ϕ,θˉ)\mathcal{L}_\mathrm{FM}^\mathrm{ELBO}(\phi, \theta)+\Delta_\mathrm{FM}^\mathrm{fac}(\phi, \bar{\theta})1.066 (0.000)0.942 (−0.015)\color{red}{(-0.015)}0.657 (+0.001)\color{blue}{(+0.001)}0.539 (−0.016)\color{red}{(-0.016)}0.717 (−0.004)\color{red}{(-0.004)}0.693 (−0.017)\color{red}{(-0.017)}0.570 (+0.035)\color{blue}{(+0.035)}0.478 (−0.017)\color{red}{(-0.017)}
LFMELBO(ϕ,θ)+ΔFMfac(ϕˉ,θ)\mathcal{L}_\mathrm{FM}^\mathrm{ELBO}(\phi, \theta)+\Delta_\mathrm{FM}^\mathrm{fac}(\bar{\phi}, \theta)1.001 (−0.065)\color{red}{\mathbf{(-0.065)}}0.889 (−0.068)\color{red}{\mathbf{(-0.068)}}0.299 (−0.357)\color{red}{\mathbf{(-0.357)}}0.371 (−0.184)\color{red}{\mathbf{(-0.184)}}0.409 (−0.312)\color{red}{\mathbf{(-0.312)}}0.411 (−0.299)\color{red}{\mathbf{(-0.299)}}0.564 (+0.029)\color{blue}{(+0.029)}0.342 (−0.153)\color{red}{\mathbf{(-0.153)}}
Ours1.0660.9570.6560.5550.7210.7100.5350.495

The 4-th setting exhibits the most significant performance drop; notably, this is the only case where the encoder lacks the training signal provided by ΔFMfac\Delta_\mathrm{FM}^\mathrm{fac}. In contrast, the first and second settings, which utilize ΔFMfac\Delta_\mathrm{FM}^\mathrm{fac} to train the encoder, perform comparably to Ours. Although the MSS (5x4) task shows a slight performance increase in all settings, these differences remain within the margin of standard deviation. Overall, these results demonstrate that training the encoder via ΔFMfac\Delta_\mathrm{FM}^\mathrm{fac} is a critical factor for successful decentralized training.

(W3-2 The distinct contribution from prior works) We are the first to introduce ΔFMfac\Delta_\mathrm{FM}^\mathrm{fac} to multi-agent systems to learn the shared latent variable z. To minimize this gap to train the encoder, we employ flow-field differences. As you noted, our Theorem 2.2 represents a special case of [4] where one of the distributions is the product of individual policies. However, it is worth highlighting a key distinction: while [4] only establishes the theoretical formulation, we leverage this difference directly to train the encoder. Furthermore, we explicitly prove how individual vector fields formulate the KL divergence in this context. (Thank you for pointing out these references; we have now cited them.)

(W4 The explanation of why the ELBO and MI objectives do not capture coordination does not hold) In toy-domain, the reconstruction term of LFMELBO\mathcal{L}_\mathrm{FM}^\mathrm{ELBO} can be decomposed as

$ q(z|\tau) ( \sum_i^N ||v_{\theta,\lambda}^{i,\text{coord}} - u^{\text{coord}}_\lambda||^2 + ||v_{\theta,\lambda}^{i,\text{indep}} - u^{\text{indep}}_\lambda||^2 )$

where the two term correspond to acoordia^i_\text{coord} and aindepia^i_\text{indep}, respectively. Here, acoordia^i_\text{coord} is actions coordinated across agents, whereas aindepia^i_\text{indep} is independent actions that are uncorrelated with those of other actions.

If the independent term, ∣∣vθ,λi,indep−uλindep∣∣2||v_{\theta,\lambda}^{i,\text{indep}} - u^{\text{indep}}_\lambda||^2 has higher dimensionality, a larger scale, or greater variability, its reconstruction error will dominate the overall objective. Consequently, The encoder assigns zz based on the modes of this independent component, as doing so yields a larger reduction in the total reconstruction loss, even these modes are irrelevant to the desired coordination.

Although The KL in LFMELBO\mathcal{L}_\mathrm{FM}^\mathrm{ELBO} and MI terms regularize latent usage and posterior confidence, they cannot identify which action dimensions are coordinated; therefore, they do not resolve this semantic ambiguity.

Figure 9 in appendix show the learned behavior under various weightings of the KL term in LFMELBO\mathcal{L}_\mathrm{FM}^\mathrm{ELBO}, demonstrating that simply adjusting these weights fails to resolve the underlying semantic ambiguity.

Conversely, ΔFMfac\Delta_\mathrm{FM}^\mathrm{fac} measures the discrepancy between the centralized joint policy and decentralized individual policies:

$ q(z|\tau) ( \sum_i^N || v^{J,i,\text{coord}}_{\eta, \lambda} - v_{\theta,\lambda}^{i,\text{coord}} ||^2 + || v^{J,i,\text{indep}}_{\eta, \lambda} - v_{\theta,\lambda}^{i,\text{indep}}||^2 ) $

Because independent actions are readily predictable without global information (e.g, the actions of other agents), the independent term ∣∣vη,λJ,i,indep−vθ,λi,indep∣∣2|| v^{J,i,\text{indep}}_{\eta, \lambda} - v_{\theta,\lambda}^{i,\text{indep}}||^2 remains small. In contrast, coordinated actions cannot be inferred from local information alone; consequently, the first term remains large unless z successfully captures the coordination mode.

(Implementation details) For discrete latent training, we compute qϕ(z∣τ)q_\phi(z|\tau) for all z∈Zz\in\mathcal{Z}, as outlined in Algorithm1. We assign equal weight of 1 to LFMELBO\mathcal{L}_\mathrm{FM}^\mathrm{ELBO}, ΔFMELBO\Delta_\mathrm{FM}^\mathrm{ELBO}, and the prior KL regularization term. And we use a 1:1 update ratio.

The learning rate is 3e-4 for policies θ,η\theta, \eta and 3e-4 for encoder ϕ\phi. Further details on network architectures, and iteration counts are provided in Appendix D.

For the per-mode MMD evaluation in Fig.3, we previously treated Gaussian-BC and FM-BC as having a single latent variable. However, since these baselines are latent-free, we have updated the Fig.3 and corresponding figures to exclude Gaussian-BC and FM-BC.

As the discussion period is drawing to a close, please let us know if you have any remaining concerns.

Reviewer_31RrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

From demonstrations of different joint strategies, CoFac-FM learns a latent strategy variable shared by all agents for an entire episode. Its objective distinguishes remaining dependence among agents' actions from reliance on observations unavailable to each agent. Training aligns each local policy's flow field with a joint policy's field, without explicitly computing action probability densities. The evaluation covers eight MAMuJoCo configurations and eight cooperative quadruped settings.

Critical strengths
  • The coordination-aware representation objective addresses a real limitation of independent imitation: individually plausible actions can still be incompatible across agents. The flow-alignment surrogate makes this objective practical to train without explicitly computing action probability densities.
  • The range of tasks and use of both distribution-matching and success metrics provide useful evidence of CoFac-FM's effectiveness. It has the lowest reported maximum mean discrepancy (MMD) between generated behavior and demonstrations in all eight MAMuJoCo configurations, and Seesaw-P success improves from .200 for the strongest listed baseline to .388. Reporting four training seeds also helps assess consistency.
Critical weaknesses
  • The benefit of the coordination-aware encoder remains unclear because the comparisons also change policy training. LatentFM omits the joint teacher policy, while LatentMF trains each agent's policy only by distilling that teacher. An encoder-only ablation should keep all other training components fixed and prevent only the alignment loss from updating the shared-strategy encoder.

  • The theoretical novelty is not clearly separated from prior work on latent representations and flow-based divergences. Gao et al. (2019) already learn latent variables to reduce conditional dependence among observed variables. For the implemented linear interpolation schedule, Eq. 7's expression for KL divergence through flow-field differences is a special case of Wang et al. (2026, Theorem 3.6), in finite dimensions with identity covariance. The paper should distinguish these established ingredients from its result for more general affine interpolation schedules and its use of alignment to train the shared-strategy encoder.

  • The duplicated results in Table 2 make performance across distinct tasks difficult to assess reliably. Cargo Door-P repeats every Door-P mean and standard deviation across all ten methods and both metrics, although the paper describes these as different tasks. An explanation and corrected results or supporting per-seed values are needed to resolve this ambiguity.

Questions for authors
  1. The flow-based ELBO baseline uses flow-matching regression in place of a log-likelihood term. With all other training components fixed, how much does allowing the joint-to-local alignment loss to update the shared-strategy encoder improve performance?
  2. Can the authors resolve the duplicated Door-P and Cargo Door-P results in Table 2 and confirm which values should be used?
2 comments

(Weakness 1, Q1 Ablation studies) To evaluate the contribution of ΔFMfac\Delta_\mathrm{FM}^\mathrm{fac} in encoder training, we perform the ablation studies by selectively applying the stop-gradient operator, indicated by ϕˉ\bar{\phi} and θˉ\bar{\theta}. For the encoder ϕ\phi and individual policies θ\theta, the results summarize the normalized return and followed by its difference from Ours in (⋅)(\cdot).

MethodHC (2x3)HC (6x1)Ant (2x4)Ant (4x2)AG (2x4)AG (4x2)MSS (5x4)MSS (10x2)
LFMELBO(ϕˉ,θ)+ΔFMfac(ϕ,θˉ)\mathcal{L}_\mathrm{FM}^\mathrm{ELBO}(\bar{\phi}, \theta)+\Delta_\mathrm{FM}^\mathrm{fac}(\phi, \bar{\theta})1.064 (−0.002)\color{red}{(-0.002)}0.901 (−0.056)\color{red}{(-0.056)}0.606 (−0.050)\color{red}{(-0.050)}0.524 (−0.031)\color{red}{(-0.031)}0.709 (−0.012)\color{red}{(-0.012)}0.701 (−0.009)\color{red}{(-0.009)}0.559 (+0.024)\color{blue}{(+0.024)}0.514 (+0.019)\color{blue}{(+0.019)}
LFMELBO(ϕˉ,θ)+ΔFMfac(ϕ,θ)\mathcal{L}_\mathrm{FM}^\mathrm{ELBO}(\bar{\phi}, \theta)+\Delta_\mathrm{FM}^\mathrm{fac}(\phi, \theta)1.068 (+0.002)\color{blue}{(+0.002)}0.951 (−0.006)\color{red}{(-0.006)}0.611 (−0.045)\color{red}{(-0.045)}0.546 (−0.009)\color{red}{(-0.009)}0.705 (−0.016)\color{red}{(-0.016)}0.699 (−0.011)\color{red}{(-0.011)}0.540 (+0.005)\color{blue}{(+0.005)}0.497 (+0.002)\color{blue}{(+0.002)}
LFMELBO(ϕ,θ)+ΔFMfac(ϕˉ,θ)\mathcal{L}_\mathrm{FM}^\mathrm{ELBO}(\phi, \theta)+\Delta_\mathrm{FM}^\mathrm{fac}(\bar{\phi}, \theta)1.001 (−0.065)\color{red}{\mathbf{(-0.065)}}0.889 (−0.068)\color{red}{\mathbf{(-0.068)}}0.299 (−0.357)\color{red}{\mathbf{(-0.357)}}0.371 (−0.184)\color{red}{\mathbf{(-0.184)}}0.409 (−0.312)\color{red}{\mathbf{(-0.312)}}0.411 (−0.299)\color{red}{\mathbf{(-0.299)}}0.564 (+0.029)\color{blue}{(+0.029)}0.342 (−0.153)\color{red}{\mathbf{(-0.153)}}
Ours1.0660.9570.6560.5550.7210.7100.5350.495

The 3rd row exhibits the most significant performance drop; notably, this is the only case where the encoder lacks the training signal provided by ΔFMfac\Delta_\mathrm{FM}^\mathrm{fac}. In contrast, the first and second settings, which utilize ΔFMfac\Delta_\mathrm{FM}^\mathrm{fac} to train the encoder, perform comparably to Ours. Overall, these findings demonstrate that incorporating ΔFMfac\Delta_\mathrm{FM}^\mathrm{fac} into the encoder’s training is critical for successful decentralized execution.

Figure 10 in appendix shows the result of the same ablation studies in toy-domain, which also exhibit that only LFMELBO(ϕ,θ)+ΔFMfac(ϕˉ,θ)\mathcal{L}_\mathrm{FM}^\mathrm{ELBO}(\phi, \theta)+\Delta_\mathrm{FM}^\mathrm{fac}(\bar{\phi}, \theta) failed to capture joint modes.

(Weakness 2 The novel contribution from prior works) While conditional independence TC(⋅∣z)TC(\cdot|z) is well established in fields such as image processing, we are the first to introduce and utilize this objective to multi-agent systems, where minimizing TC is critical for decentralized execution. To minimize TC(⋅∣z)TC(\cdot|z) and train the encoder, we employ flow-field differences. As you noted, our Theorem 2.2 represents a special case of (Wang et al.) where one of the distributions is the product of individual policies. However, it is worth highlighting a key distinction: while Wang et al. uses the flow-field difference derived from the KL divergence to evaluate the trajectory inference methods, we utilize this difference specifically to train the encoder. Furthermore, we explicitly prove how individual vector fields formulate the KL divergence in this context. (Thank you for pointing out these references; we have now cited them.)

(Weakness 3, Q2 Duplicated results) Thank you for pointing that out. We fixed Door-P and the resulting is the following:

MethodGaussian-BCGMM-ELBOGMM-ELBO-MIFM-BCLatentFM-ELBOLatentFM-ELBO-MIMAC-FlowLatentMF-ELBOLatentMF-ELBO-MIOurs
Door-P Succ.0.0000.1060.1250.1750.1180.1560.0620.1120.1180.456

As the discussion period is drawing to a close, please let us know if you have any remaining concerns.

Comments on the paper

1 comment

Thank you to all reviewers for your time, thoughtful feedback, and constructive comments on our submission.

During the rebuttal period, we have updated the manuscript and conducted additional experiments to address your concerns:

1. Ablation Studies We conducted controlled stop-gradient ablation studies to isolate and verify the effectiveness of our joint-to-local alignment loss (ΔfacFM\Delta_{\text{fac}}^{\text{FM}}) in training the shared-strategy encoder. The following table summarize the normalized return and followed by its difference from Ours in (⋅)(\cdot):

MethodHC (2x3)HC (6x1)Ant (2x4)Ant (4x2)AG (2x4)AG (4x2)MSS (5x4)MSS (10x2)
LFMELBO(ϕˉ,θ)+ΔFMfac(ϕ,θˉ)\mathcal{L}_\mathrm{FM}^\mathrm{ELBO}(\bar{\phi}, \theta)+\Delta_\mathrm{FM}^\mathrm{fac}(\phi, \bar{\theta})1.064 (−0.002)\color{red}{(-0.002)}0.901 (−0.056)\color{red}{(-0.056)}0.606 (−0.050)\color{red}{(-0.050)}0.524 (−0.031)\color{red}{(-0.031)}0.709 (−0.012)\color{red}{(-0.012)}0.701 (−0.009)\color{red}{(-0.009)}0.559 (+0.024)\color{blue}{(+0.024)}0.514 (+0.019)\color{blue}{(+0.019)}
LFMELBO(ϕˉ,θ)+ΔFMfac(ϕ,θ)\mathcal{L}_\mathrm{FM}^\mathrm{ELBO}(\bar{\phi}, \theta)+\Delta_\mathrm{FM}^\mathrm{fac}(\phi, \theta)1.068 (+0.002)\color{blue}{(+0.002)}0.951 (−0.006)\color{red}{(-0.006)}0.611 (−0.045)\color{red}{(-0.045)}0.546 (−0.009)\color{red}{(-0.009)}0.705 (−0.016)\color{red}{(-0.016)}0.699 (−0.011)\color{red}{(-0.011)}0.540 (+0.005)\color{blue}{(+0.005)}0.497 (+0.002)\color{blue}{(+0.002)}
LFMELBO(ϕ,θ)+ΔFMfac(ϕ,θˉ)\mathcal{L}_\mathrm{FM}^\mathrm{ELBO}(\phi, \theta)+\Delta_\mathrm{FM}^\mathrm{fac}(\phi, \bar{\theta})1.066 (0.000)0.942 (−0.015)\color{red}{(-0.015)}0.657 (+0.001)\color{blue}{(+0.001)}0.539 (−0.016)\color{red}{(-0.016)}0.717 (−0.004)\color{red}{(-0.004)}0.693 (−0.017)\color{red}{(-0.017)}0.570 (+0.035)\color{blue}{(+0.035)}0.478 (−0.017)\color{red}{(-0.017)}
LFMELBO(ϕ,θ)+ΔFMfac(ϕˉ,θ)\mathcal{L}_\mathrm{FM}^\mathrm{ELBO}(\phi, \theta)+\Delta_\mathrm{FM}^\mathrm{fac}(\bar{\phi}, \theta)1.001 (−0.065)\color{red}{\mathbf{(-0.065)}}0.889 (−0.068)\color{red}{\mathbf{(-0.068)}}0.299 (−0.357)\color{red}{\mathbf{(-0.357)}}0.371 (−0.184)\color{red}{\mathbf{(-0.184)}}0.409 (−0.312)\color{red}{\mathbf{(-0.312)}}0.411 (−0.299)\color{red}{\mathbf{(-0.299)}}0.564 (+0.029)\color{blue}{(+0.029)}0.342 (−0.153)\color{red}{\mathbf{(-0.153)}}
Ours1.0660.9570.6560.5550.7210.7100.5350.495

The 4-th setting exhibits the most significant performance drop; notably, this is the only case where the encoder lacks the training signal provided by ΔFMfac\Delta_\mathrm{FM}^\mathrm{fac}. In contrast, the first and second settings, which utilize ΔFMfac\Delta_\mathrm{FM}^\mathrm{fac} to train the encoder, perform comparably to Ours. Overall, these results demonstrate that training the encoder via ΔFMfac\Delta_\mathrm{FM}^\mathrm{fac} is a critical factor for successful decentralized training.

2. Additional Baselines We added comparisons with two relevant baselines, MIMIC-D and MIFQ, across both Multi-Agent MuJoCo and MQE benchmarks.

MethodHC (2x3)HC (6x1)Ant (2x4)Ant (4x2)AG (2x4)AG (4x2)MSS (5x4)MSS (10x2)
MIMIC-D1.0430.8820.4000.5050.5220.4350.5660.459
MIFQ1.0350.8460.1130.2690.1120.1070.5130.452
Ours1.0660.9570.6560.5550.7210.7100.5350.495
MethodDoorDoor-PSeesawSeesaw-PCargo DoorCargo Door-PCargoCargo-P
MIMIC-D0.7930.1810.3340.2120.3060.2060.4120.334
MIFQ0.7810.0370.3150.0810.3000.1500.4030.368
Ours0.8190.4560.3560.3880.4250.5010.4870.413

3. Corrected Table 2 Results We resolved the duplicated reporting issue in Table 2 by updating the results for the Door-P task.

We hope these updates address the reviewers' main concerns, and we appreciate your continued consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.