acceptodds
Under review as a conference paper at ICLR 2027

One for All, All for One: Coordinated Multi-Agent Diffusion Steering via Stochastic Optimal Control

Abstract

Many structured outputs consist of interacting components, so modelling them with a single generative model requires learning both the component distributions and their interactions. We pursue a modular alternative: reuse independently trained component generators and learn only how to coordinate them to produce coherent structured outputs. Our framework, Coordinated Multi-Agent Diffusion Steering (CMDS), treats frozen pretrained diffusion models as reusable generative primitives and coordinates their reverse processes through a learned control. We formulate coordination as a stochastic optimal control problem, balancing an assembly-level reward that specifies the desired properties of the combined output against deviations from the pretrained dynamics. The learned control amortises this optimisation, allowing reuse across new task instances. Experiments show that CMDS can recover a known target distribution, satisfy different spatial constraints with the same trained control, and recover individual sources from blurred mixtures. Across multi-agent maze navigation, articulated robot planning, and text-conditioned human motion, CMDS turns frozen component models into coordinated multi-agent generators.

Primary area:
generative models
Keywords:
generative models, diffusion models, stochastic optimal control, adjoint matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_g1Q9OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CMDS (Coordinated Multi-Agent Diffusion Steering), a framework for coordinating frozen pretrained diffusion models through stochastic optimal control. Rather than retraining component generators, CMDS treats them as reusable primitives and learns a control that steers their joint reverse process toward high assembly-level reward while minimizing deviation from pretrained dynamics. The control is trained via adjoint matching, which avoids backpropagation through the solver. Experiments demonstrate target-law recovery in a tractable OU setting, source separation from blurred mixtures, multi-agent navigation in PointMaze (up to 4 agents), dual-arm planning with KUKA robots, and four-person motion coordination with MDM.

Critical strengths

a. Coordinating frozen pretrained diffusion models through learned control, rather than retraining them, is a practical and appealing approach. The stochastic optimal control formulation is elegant and theoretically grounded, providing a clear objective for balancing assembly-level reward against deviation from pretrained dynamics.

b. The paper evaluates on four distinct settings: tractable OU coordination, source separation, multi-agent navigation (PointMaze), articulated robot planning (KUKA), and multi-person motion (MDM). The evaluation is comprehensive, with ablations, transfer experiments, and comparisons against multiple baselines.

c. By avoiding backpropagation through the solver, adjoint matching enables efficient training of the coordination control. The paper demonstrates that this approach works well empirically, achieving high success rates across all tasks.

d. The notation is consistent, the derivations are clear, and the experimental results are presented effectively. Figure 1 provides a clear overview of the method, and the qualitative results (Figures 5, 6, 7) are illustrative.

Critical weaknesses

a. The core technical approach, adjoint matching for multi-agent diffusion steering, is a straightforward extension of Domingo-Enrich et al. (2025). The paper would benefit from a clearer discussion of what is genuinely new (e.g., the multi-agent adjoint matching derivation, the specific control architecture) vs. what is a direct application of prior work.

b. The paper doesn't compare against MultiDiffusion, Diffuser, or other relevant baselines. This makes it unclear whether CMDS is the best approach for multi-agent coordination or just one of many viable options.

c. The "amortised control" framing overstates the generalization. The control is trained per-task and only transfers within the same task (e.g., unseen start-goal pairs). It doesn't transfer across fundamentally different tasks or reward functions. The paper should be clearer about the scope of "amortisation."

d. The paper evaluates up to 4 agents but doesn't analyze how the method scales. For large-scale multi-agent systems, the joint state representation and centralized control could become prohibitive. The paper doesn't discuss this limitation or potential solutions.

e. The paper doesn't provide guidance on how to design rewards for new tasks or analyze sensitivity to reward choice. This limits the practical utility of the method for users who want to apply it to new domains.

f. While the empirical results are strong, the paper doesn't provide theoretical analysis of when adjoint matching might fail or how accurate the approximation is. This is a gap given the theoretical framing of the paper.

Questions for authors
  1. Can you provide a comparison with MultiDiffusion (Bar-Tal et al., 2023) and Diffuser (Janner et al., 2022) to establish whether CMDS is the best approach for multi-agent coordination?

  2. Can you clarify the scope of "amortised control"? Does the control transfer across fundamentally different tasks or reward functions, or only within the same task (e.g., unseen start-goal pairs)?

  3. Can you provide a scalability analysis for larger numbers of agents (e.g., 10+)? How does the method's performance and computational cost scale with agent count?

  4. Can you provide guidance on reward design for new tasks? How sensitive is the method to the specific reward formulation and weighting of terms?

  5. Can you provide theoretical guarantees or empirical analysis of the adjoint matching approximation quality? When might it fail?

  6. Can you discuss the limitations of the centralized control architecture? For large-scale multi-agent systems, would a decentralized or attention-based approach be more appropriate?

7 comments
AuthorsOpenReview ↗

Part 1A: CMDS’s Contributions and Its Relationship to Adjoint Matching

We thank the reviewer for (i) recognising the practical value of coordinating frozen pretrained models, (ii) the theoretical grounding of our SOC formulation, and (iii) the efficiency of multi-agent adjoint-matching training. We also appreciate their positive assessment of our comprehensive evaluation, including ablations, transfer experiments, our strong empirical results, and the clarity of our presentation and derivations.


We address the first weakness, clarifying what is new in CMDS and what is inherited from the adjoint-matching framework in [1]. The remaining weaknesses are instead addressed alongside the corresponding questions.

We thank the reviewer and agree that the manuscript should distinguish more clearly what CMDS inherits from adjoint matching (AM) and what is new. Our contribution is a modular coordination framework: CMDS learns a joint control over frozen, independently pretrained diffusion models using an assembly-level reward, without requiring joint training data. Coordination is amortised through a learned controller reused across task instances; we also demonstrate zero-shot transfer to larger agent populations in the evaluated settings. The method combines established ingredients: reward tilting via SOC, AM, a pseudo-Huber loss for numerical stabilisation of AM in high-dimensional settings, and an attention-based controller. We discuss the novelty in detail below:

1. SOC for component assemblies. Extending forward noising to several agents is straightforward, and the reverse drift of the product reference acts agent by agent (App. B.1). The difficulty lies in the object the reward acts on, the assembly Y^=φ(X^)\hat Y=\varphi(\hat X). The joint process is Markov, but Y^\hat Y need not be when φ\varphi discards information. Under suitable assumptions, a Markovian projection can reproduce its one-time marginals, without generally reproducing its full path law [2]. CMDS instead steers the components and handles the assembly by pushforward, including for non-injective φ\varphi (Eqs. 67–69, App. B).

2. What we inherit from AM, and when coordination decouples. From AM we inherit the memoryless schedule (η\eta = 1), which yields the tilt of Eq. (61), and the basic regression objective (their Prop. 2), applied to the joint state. What changes is that the reward r∘φr\circ\varphi couples agents through Jacobians in the adjoint of Eq. (6), for example through the terminal term −λ(Dxiφ)⊤∇r-\lambda(D_{x^i}\varphi)^\top\nabla r.

We will add a proposition in Appendix B.4 clarifying when coordination is required. Consider the terminal tilt ρ⋆=Z−1eλRρ\rho^\star=Z^{-1}e^{\lambda R}\rho, where ρ\rho is a product law, λ>0\lambda>0, R=r∘φR=r\circ\varphi is finite ρ\rho-almost everywhere and 0<Z<∞0<Z<\infty. Call RR separable if R(x)=c+∑iri(xi)R(x)=c+\sum_i r_i(x^i) ρ\rho-almost everywhere.

  • The tilted terminal law ρ⋆\rho^\star is a product if and only if RR is separable.
  • With independent reference components and no other coupling costs or constraints, a separable reward gives independent single-component SOC problems, each solvable by AM.
  • For fixed task conditions, local-only controls ui(t,xi)u^i(t,x^i) with independent initial states and driving noises cannot realise a non-product terminal law.

Independent per-agent AM suffices when the SOC problem separates. CMDS applies joint-state AM to coordinate frozen component models under assembly-level rewards. In App. D.2, AM applied to a single generator reaches 0–18.9% valid assemblies, against 97.9–99.2% for CMDS with the same solver.

AuthorsOpenReview ↗

Part 1B: CMDS’s Contributions and Its Relationship to Adjoint Matching

3. Cross-agent credit assignment. The adjoint gives agent ii two learning signals (App. C). At the terminal time, −λ(Dxiφ)⊤∇r-\lambda(D_{x^i}\varphi)^\top\nabla r measures how its output affects the assembly reward. Along the trajectory, the terms DxiUθjD_{x^i}U_\theta^j (j≠ij\neq i) capture how its state changes the corrections applied to the other agents. With independent reference drifts, AM's lean adjoint drops this second signal (Eq. 90), while the full adjoint, which we use, retains it.

4. Relation to AM's contributions. AM's central contributions are the memoryless schedule and the lean-adjoint matching objective; adjoint methods for SOC predate it. We use the memoryless schedule but not the lean adjoint (App. C.2.1).

5. Controller. CMDS therefore uses an external residual control: it combines axial temporal and cross-agent attention with permutation-equivariant set processing; it is weight-shared across agents and uses no agent-index embeddings, so its parameters do not depend on the number of agents. A controller on the concatenated joint vector, the literal “AM on the joint state”, would fix both N and the agent ordering. While factorised time/agent attention with agent permutation equivariance already appears in multi-agent motion prediction [7], to our knowledge, CMDS is the first to use it as a control network that coordinates frozen pretrained diffusion models. A common architectural design is trained separately for each task setting, with parameters shared across agents and held fixed in the reported zero-shot evaluations. This will be better clarified in the main manuscript, alongside a schematic of the control architecture used for PointMaze, KUKA and MDM [schematic].

We will add a "Relation to Adjoint Matching" paragraph in Sec. 2.1.

AuthorsOpenReview ↗

Part 2: Baseline Comparisons and the Scope of Amortised Control

We now address the reviewer’s specific questions. For brevity, we address the weaknesses alongside the corresponding questions.

Q1 / W.b: How does CMDS compare with MultiDiffusion, Diffuser and other relevant baselines for multi-agent coordination?

MultiDiffusion is a relevant test-time method already cited in our related work, but its original mechanism does not directly solve our coordination problem. The reason is how it connects the diffusion processes, not simply that it was demonstrated on images. MultiDiffusion makes overlapping predictions agree. For example, two overlapping image crops predict the same pixels. It combines their denoising updates through a least-squares objective to produce one consistent image. Our agents have separate trajectories. Person A’s position and person B’s position are different variables. We need their motions to satisfy a relationship, such as maintaining sufficient separation while each reaches their own goal. A direct per-person adaptation would provide no coordination. If each MultiDiffusion view simply extracts one person’s trajectory, its fusion objective separates into independent per-person updates.

On the other hand, Diffuser requires additional training. We adapt Diffuser’s learned return guidance [5] to the multi-agent setting; the original work does not include multi-agent experiments. Both methods require training, but use reward information differently:

  • The Diffuser baseline adapts Diffuser’s learned-return guidance: a scalar predictor VV is trained offline to estimate trajectory returns. During sampling, its parameters remain fixed, and its gradient with respect to the trajectory is computed at each step to steer the reverse diffusion process. Thus, it learns a return predictor, whose gradients provide steering directions at test time.

  • CMDS directly learns a joint control offline under the SOC objective, using analytic reward gradients propagated through the adjoint during training. During sampling, its parameters likewise remain fixed, and the controller directly outputs the steering corrections. In CMDS-TRG, reward gradients are additionally supplied to the learned control parameterisation.

The distinction is the information provided by each training sample. Trajectories with similarly poor collision penalties can have different reward gradients, indicating which coordinates to change to reduce collisions. Diffuser must learn these corrective directions indirectly from scalar-return supervision. This can be harder when sampled trajectories receive similar returns, even though their corrective directions differ. Increasing the number of agents may exacerbate this difficulty if generated trajectories have similarly poor returns.

PointMaze - joint success (%):

LayoutAgents NNDiffuser · Pool 1Diffuser · Pool 2CMDS · Pool 1CMDS · Pool 2CMDS-TRG · Pool 1CMDS-TRG · Pool 2
Medium246.35 ± 4.0144.01 ± 1.0499.0 ± 1.291.9 ± 1.699.5 ± 0.599.2 ± 0.0
Medium46.42 ± 0.402.34 ± 0.2670.1 ± 0.966.9 ± 0.597.4 ± 1.293.5 ± 0.5
Large266.41 ± 1.9757.90 ± 3.2491.1 ± 1.683.1 ± 1.696.6 ± 0.589.8 ± 1.6
Large43.65 ± 0.265.21 ± 0.0088.0 ± 1.275.3 ± 2.796.6 ± 1.297.7 ± 0.0

KUKA:

MethodStrict success (%) ↑Planning time (s) ↓
Diffuser19.99 ± 0.730.278 ± 0.005
CMDS (without TRG)60.66 ± 1.190.348 ± 0.003
CMDS-TRG77.3 ± 0.70.368 ± 0.002

Both CMDS variants achieve higher mean success than Diffuser, including without TRG, although Diffuser has a shorter planning time.

We will include this baseline in our evaluation.

Q2 / W.c: Does “amortised control” mean reuse across unseen instances within a task, or transfer across different tasks and reward functions?

Amortised control means training one context-conditioned controller and reusing it across unseen instances within a specified task family. This uses standard conditional learning: the training cost is shared across repeated inference. For example, the same MDM controller handles new scene conditions, with sampling taking 0.54 s for CMDS versus 8.57 s for DPS (Table 4).

Zero-shot transfer is a separate capability. A CMDS-TRG controller trained on Medium PointMaze with two agents achieves 86.20 ± 0.45% / 77.08 ± 4.58% success on Large, without controller retraining (Pools 1/2). We also evaluate transfer across agent counts (Q3) and KUKA reward-clearance parameters (Q4). These results extend the demonstrated scope of reuse, but do not establish transfer across fundamentally different tasks or arbitrary reward functions. We now make this distinction explicit in the manuscript.

Reviewer_w5dkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper leverages stochastic optimal control to compose multiple frozen diffusion models, using feedback from a global (i.e. not at the component level) reward on the mixture via adjoint-based methods. They provide empirical validation in a wide range of tasks including an analytically tractable OU process, visual assembly/source separation, maze navigation, robot-arm planning, and human motion.

Critical strengths

The empirical settings are very varied and cover a wide range of applications. Results are seem strong.

Critical weaknesses

The method is only briefly presented in the main text, not many reward composition baselines, some of experiment design it is unclear to me how the reward structure is assembly-level, and more importantly it is unclear to me that the experimental baselines are strong or tuned with a comparable budget to their method.

Questions for authors

Clarification: Have baseline goal weights and guidance strengths receive comparable tuning budget to your method? How where hyperparameters set for kuka and the MDM evaluation scenes selected? How does pseudo-Huber matching affect distribution recovery wrt squared matching?

4 comments
AuthorsOpenReview ↗

Part 1: Method Presentation and Assembly-Level Rewards

We thank the reviewer for their constructive feedback and for (i) recognising the breadth of our empirical evaluation across diverse applications; (ii) highlighting the strength of our results; and (iii) finding the paper clear and easy to follow, with no substantive concerns about its soundness.


We now address the weaknesses identified by the reviewer.

Method presentation. We will expand the method description in Section 2.1 and add a schematic of the shared control architecture (App. D.4.5).

Assembly-level rewards. We evaluate the reward on the assembled output Y=φ(X)Y=\varphi(X). We acknowledge that the experimental section could explain these definitions more clearly and will specify the assembly and reward used in each experiment. Specifically:

  • Two-agent OU (App. D.1): The assembly contains both terminal positions, Y=(X1,X2)Y=(X^1,X^2). The reward r(Y)=−(∥X2−X1∥2−dT)2r(Y)=-(\lVert X^2-X^1\rVert_2-d_T)^2 encourages a prescribed distance between the agents.

  • Squares (App. D.2): The assembly combines masked component images, Y=ML⊙X1+MR⊙X2Y=M_L\odot X^1+M_R\odot X^2. The reward scores square presence and, in the coupled experiment, adds a penalty for deviation from the requested horizontal separation. The classifier-only variant is separable; the separation term introduces coordination.

  • Source separation (App. D.3): This is an inverse problem: recovering individual sources from a blurred observation of their sum, Y=X1+X2Y=X^1+X^2. The reward rb(Y)=−d−1∥Aτ(Y)−b∥22r_b(Y)=-d^{-1}\lVert A_\tau(Y)-b\rVert_2^2, where dd is the number of image pixels, corresponds to a scaled Gaussian log-likelihood, up to an additive constant. It encourages both sources to jointly explain the observed mixture.

  • PointMaze (App. D.4): The assembly stacks the component states, Y=(X1,…,XN)Y=(X^1,\ldots,X^N). The reward penalises inter-agent collisions and insufficient wall clearance, with path-length and acceleration regularisation.

  • KUKA (App. D.5): The assembly contains the reconstructed joint-angle paths of all arms, Y=(q1,…,qN)Y=(q^1,\ldots,q^N). The reward penalises inter-arm, obstacle, table and self-collisions. Inter-arm penalties couple the generated paths.

  • MDM (App. D.6): The assembly retains both native motion features and decoded world-space joints, Y=(X,Jκ(X))Y=(X,J_\kappa(X)). The reward combines inter-person collision penalties with goal, progress and physical-consistency costs. Inter-person collisions couple the motions, while the remaining terms encourage individual task completion and motion validity.

AuthorsOpenReview ↗

Part 2: Baseline Tuning and Evaluation Protocols

We now address the reviewer’s specific questions.

Q1. Have baseline goal weights and guidance strengths received a comparable tuning budget?

Thank you for raising this; we used method-specific grid searches.

What was tuned. We tuned the following baseline hyperparameters:

  • PointMaze: the DPS scale over 8 values; FK-Steering’s temperature over 12 values, together with how often it resamples (60 settings in total).
  • KUKA: the DPS scale over 5 values, for both the single-arm and the dual prior.
  • MDM: DPS guidance strength, goal and collision weights, and guidance schedule (Reviewer 2Scz, Q2).
  • Diffuser: added at the request of reviewer g1Q9. We use Diffuser’s value-network design [2] and tune its width, learning rate and training loss. The selected networks have 2.3M–36.6M parameters, compared with 4.3M for the CMDS controller.

For CMDS, on KUKA, we compared 32 settings of the reward weights, β, learning rate and EMA on the validation layouts, then refined the best one there. On PointMaze, we searched the reward scale and learning rate together.

How we chose. For KUKA, guidance strengths were selected on the 134 validation tasks and then fixed for evaluation. MDM development and final evaluation use the same scene conditions with different sampling seeds. The selected guidance scales and temperatures were not at the endpoints of the searched grids.

MDM goal weights. In an additional comparison on the same 24 development scenes (seed 1001), we increase the DPS goal weight from 1 to 50, matching CMDS’s training weight, and hold all other settings fixed. Goal error falls from 3.689 m to 1.597 m, but the collision-free rate falls from 66.7% to 0%; joint success remains zero (response to Reviewer 2Scz, Q2).

Training and sampling cost. Both CMDS and the baselines incur hyperparameter-search costs during development. CMDS additionally trains a reusable controller, whereas test-time methods recompute steering during each sampling run. These inference costs are distinct from hyperparameter-search budgets.

Q2. How were KUKA hyperparameters and MDM evaluation scenes selected?

Hyperparameters. The KUKA reward weights, β\beta, learning rate and EMA were selected on the validation layouts through a 32-configuration screen. The squared-hinge reward and TRG were retained after a two-seed validation comparison. We report the EMA controller from the final training update, without checkpoint selection (Appendix D.5.3, Table 12). The DPS gradient scale (0.6) was chosen from {0.1, 0.3, 0.6, 1, 1.7} on the 134 validation tasks (Appendix D.5.4). Dual KUKA keeps its native sampler [1]. PBDM uses the released implementation with its optional refinement enabled, and starts from the same number of initial candidates as the other methods (Appendix D.5.4).

Evaluation tasks. We reuse the released layouts and start–goal configurations of [1]; no scene is selected by hand. A task is one obstacle layout plus one start and goal configuration for each arm. Of the 2,500 released training layouts, 2,250 train the controller and 250 are reserved for validation, of which we use 100. Testing uses the 300 released test layouts, which were not used to train the priors or the controller. A fixed seeded rule draws two tasks from each of these 100 validation and 300 test layouts. Before any planner runs, we remove tasks whose start or goal violates joint limits or is in collision. This leaves 134 validation and 413 test tasks (Appendix D.5.1, Table 11).

Generalisation tests. The shifted-obstacle, larger-obstacle and wider-base tests apply their change to every selected test task and repeat the endpoint-validity check (Table 11). For the arm-count test, we draw twenty tasks per test layout and keep the 209 whose endpoints are valid for every evaluated arm count, so all arm counts use the same retained task set (Appendix D.5.1, Table 3).

MDM scene selection. As described in Appendices D.6.1 and D.6.6, we use 24 fixed synthetic four-person crossing scenes. Four people start approximately equally spaced around a randomly rotated circle of radius 2.15 m, chosen based on the frozen MDM’s typical walking distance, with perturbations to radius (±0.15 m), angular placement (±10°) and initial orientation (±10°). Each person’s destination is opposite their start, so their direct routes must cross. Motions contain 120 frames at 20 fps, approximately six seconds. Individual text prompts specify three types of locomotion: walking, jogging or crouched walking. We use three alternative text prompts per action family and include both scenes where everyone performs the same action and scenes combining different actions. Representative prompts include “A person walks forward” and “A person jogs straight ahead.” The main-table evaluation uses the same 24 scenes and five sampling seeds for all methods.

AuthorsOpenReview ↗

Part 3: Pseudo-Huber Matching and Distribution Recovery

Q3: How does pseudo-Huber matching affect distribution recovery with respect to squared matching?

We use pseudo-Huber to stabilise AM by smoothly reducing the influence of large matching residuals. The AM implementation of Domingo-Enrich et al. (2025) also limits large residuals by masking individual sample–timestep loss terms above an adaptive threshold (code, lines 343–362); pseudo-Huber retains a training signal for these terms.

Empirically, our additional OU evaluation finds comparable distribution recovery, using the same controller architecture, training budget and evaluation protocol:

Matching lossW2W_2 ↓MMD ↓Sinkhorn cost ↓
Squared0.4967 ± 0.02170.01799 ± 0.008250.8882 ± 0.0114
Pseudo-Huber, δt\delta_t0.4969 ± 0.02160.01801 ± 0.008230.8883 ± 0.0114
Pseudo-Huber, 0.1δt0.1\delta_t0.5006 ± 0.02060.01822 ± 0.007810.8903 ± 0.0106

Mean ± SD across five independent evaluation sample banks.

Thus, even a tenfold smaller threshold gives comparable metrics on this benchmark. This supports pseudo-Huber empirically on OU, without establishing equivalent distribution recovery across tasks.

Prompted by the reviewer, we also show that pseudo-Huber approximates squared AM when residuals are small relative to the threshold. For a matching residual eθ(t)∈RDe_\theta(t)\in\mathbb{R}^D, the squared-AM and pseudo-Huber regression gradients are, respectively,

gsq(θ)=EPuθ[∫01(Dθuθ)⊤eθ(t)dt],g_{\mathrm{sq}}(\theta)=\mathbb{E}_{\mathbb{P}^{u_\theta}}[\int_0^1(D_\theta u_\theta)^\top e_\theta(t)dt],

gPH(θ)=EPuθ[∫01(Dθuθ)⊤eθ(t)1+∥eθ(t)∥2/(Dδt2)dt],g_{\mathrm{PH}}(\theta)=\mathbb{E}_{\mathbb{P}^{u_\theta}}[\int_0^1(D_\theta u_\theta)^\top\frac{e_\theta(t)}{\sqrt{1+\lVert e_\theta(t)\rVert^2/(D\delta_t^2)}}dt],

where δt\delta_t is the pseudo-Huber threshold, DθD_\theta is the derivative with respect to θ\theta, and DD is the dimension of state space RD\mathbb{R}^D; rollout states, adjoint targets and thresholds are held fixed during differentiation. For each sampled trajectory and time tt, let gsq(t)g_{\mathrm{sq}}(t) denote the squared-AM gradient contribution and st=∥eθ(t)∥/(Dδt)s_t=\lVert e_\theta(t)\rVert/(\sqrt{D}\delta_t). Pseudo-Huber rescales this contribution as gPH(t)=gsq(t)1+st2.g_{\mathrm{PH}}(t) = \frac{g_{\mathrm{sq}}(t)}{\sqrt{1+s_t^2}}. Each contribution therefore retains its direction, while large residuals receive less weight. For small relative residuals, 11+st2=1−st22+O(st4),\frac{1}{\sqrt{1+s_t^2}} = 1-\frac{s_t^2}{2}+O(s_t^4), so pseudo-Huber closely approximates each squared-AM gradient contribution, with a relative attenuation quadratic in sts_t. However, individual residuals can remain large at the SOC optimum even when their conditional mean is zero; residual-dependent reweighting can therefore change the expected gradient and its stationary points.

We will include this analysis in the new manuscript subsubsection "C.2.2. Pseudo-Huber adjoint matching and classical squared adjoint matching".

We thank the reviewer for prompting these clarifications. If these address your concerns, we would be grateful if you would consider raising your score to Accept.

References:

[1] Luo et al. "Potential Based Diffusion Motion Planning." (2024)

[2] Janner et al. "Planning with Diffusion for Flexible Behavior Synthesis." (2022)

Thank you for the detailed response and the additional experiment results. I think most of my concerns are addressed. I will raise my score.

Reviewer_2SczOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CMDS, which coordinates frozen pretrained diffusion models by learning a joint control over their reverse processes. Coordination is cast as stochastic optimal control. An assembly-level reward is traded off against KL deviation from the independent product process, and the control is trained by adjoint matching. The authors conduct experiments in various domain, including a two-agent OU problem, source separation, multi-agent PointMaze, multi-arm KUKA planning, and four-person MDM motion.

Critical strengths

The problem is valuable and the framework is simple and general. Reusing pretrained single-agent diffusion models for multi-agent tasks avoids collecting joint training data. The same recipe works across very different domains, including source separation, 2-D navigation, robot arms and human motion, with off-the-shelf KUKA and MDM checkpoints. In the OU toy problem, the learned control closely matches an exact oracle distribution (Fig. 3), which supports the SOC formulation.

Critical weaknesses
  1. The results in Table 1 show that DPS outperforms the base CMDS in six of eight PointMaze cells. The gains come from CMDS+TRG, which adds a guidance-like reward-gradient term to the control. The PointMaze, KUKA and MDM results also use a pseudo-Huber loss that, as the authors note, loses adjoint matching’s gradient guarantee. For example, in Eq 3, a reward term is already used in the SOC equation, why the additional TRG is still required? Could the author clarify this as well as how much the learned SOC control itself can contribute?
  2. Compared to training-free method like PCD, how is the generalizability of the learned control module. For example, Outside KUKA (Table 3), generalization of the learned control is tested only narrowly. In PointMaze, a separate control is trained for each maze layout and agent count. No experiment changes the reward. Can the authors comment on how robust the control is to a changed reward, a new maze layout, or a different number of agents in PointMaze?
Questions for authors
  1. Many baselines this paper uses, such as DPS, PCD and PCD++, need no additional training. Can the authors describe the training data and training cost? If there are any other baselines that also needs additional training, it would be good to include a comparison against such baselines.
  2. In the MDM experiment, the baselines use a goal weight of 1, while CMDS is trained with a goal weight of 50 (Table 14). Was the DPS step size tuned, and how do the baselines perform with the same reward weights as CMDS? This matters because DPS reaches 0% joint success with a 3.75 m goal error (Table 15), which may partly come from the weaker goal reward.

I can consider increase my score if those concerns are addressed.

6 comments
AuthorsOpenReview ↗

Part 1: TRG Parameterisation and the Contribution of Learned Control

We thank the reviewer for their insightful comments and for (i) recognising the value of the problem and the generality of our framework; (ii) highlighting CMDS’s reuse of pretrained diffusion models without joint training data, and its applicability across diverse domains with off-the-shelf checkpoints; and (iii) noting CMDS’s close agreement with the oracle distribution in the OU experiment, providing empirical support for our SOC formulation.


We now address the weaknesses identified by the reviewer:

(i) TRG parameterises the control; it does not add another objective.

Eq. (3) specifies the SOC objective used to learn the control, whereas TRG is an architectural choice for the control. In Appendix D.4.5, we specify its representation in Eq. (114):

δϵθ,ℓ=NNθ(Zℓ,Z^1∣ℓ,g^ℓ,κ,ζℓ)−sθ(ζℓ)g^ℓ.\delta\epsilon_{\theta,\ell}=\mathrm{NN}_{\theta}(Z_{\ell},\hat{Z}_{1|\ell},\hat{g}_{\ell},\kappa,\zeta_{\ell})-s_{\theta}(\zeta_{\ell})\hat{g}_{\ell}.

Here, δϵθ,ℓ\delta\epsilon_{\theta,\ell} is the additive correction to the frozen denoiser’s noise prediction, and NNθ\mathrm{NN}_{\theta} is the shared control network. At reverse-sampling step ℓ\ell, ZℓZ_{\ell} contains the agents’ noisy trajectories, Z^1∣ℓ\hat{Z}_{1|\ell} is their Tweedie estimate of the clean trajectories, and g^ℓ\hat{g}_{\ell} is the normalised, clipped reward gradient evaluated at this estimate. The conditions κ\kappa specify the agents’ start–goal pairs, and ζℓ\zeta_{\ell} is the corresponding diffusion timestep. Finally, sθs_{\theta} is a learned time-only scalar function; θ\theta collects the trainable parameters of both the axial network NNθ\mathrm{NN}_{\theta} and the scaling function. Note that the detached reward gradient enters both as an input to the axial control branch and as an additive direction; the axial control sub-network and scaling function are trained together through SOC. Thus, TRG introduces a reward-gradient inductive bias by augmenting the control input with reward information at intermediate steps. This is standard practice as related reward-gradient architectures are used in optimal control and diffusion modelling for reward tilting [1], inverse problems [2] and sampling [3–6]; ablations in [7] show that removing such gradient-informed parameterisations may lead to mode collapse in several neural samplers. We believe that our notation CMDS+TRG may have caused confusion. We now denote the method as CMDS-TRG.

CMDS-TRG denotes the same SOC-based CMDS method using a control parameterisation that is fed with reward-gradient information. At sampling time, CMDS-TRG uses the same sampling scheme as CMDS. Reward gradients are only fed detached into the learned control parameterisation, without introducing a separate gradient-based guidance in the sampling update (such as DPS or PCD). This is now clarified in the main manuscript, alongside a schematic of the control architecture used for PointMaze, KUKA and MDM [see schematic].

On PointMaze, DPS’s strong performance indicates that the reward-gradient guidance is already effective. TRG makes this information available to the learned control, providing a useful inductive bias while training with the SOC objective.

To assess how CMDS’s performance depends on TRG, we evaluated CMDS without TRG on KUKA and CMDS-TRG on MDM. We report the comparison below:

KUKA:

Methodncandn_{\mathrm{cand}}Strict success (%, ↑)Planning time (s, ↓)
CMDS160.66 ± 1.190.348 ± 0.003
CMDS-TRG177.3 ± 0.70.368 ± 0.002

MDM:

MethodJoint success (%, ↑)Goal error (m, ↓)
CMDS90.0 ± 6.30.158 ± 0.008
CMDS-TRG86.7 ± 7.50.102 ± 0.012

TRG improves KUKA success, whereas in MDM it improves goal accuracy without improving measured joint success. MDM has a much larger trajectory representation, i.e., 126240 coordinates, compared with 400 in four-agent PointMaze and 784 in two-arm KUKA. This difference in dimensionality may contribute to the differing benefits of reward-gradient information.

AuthorsOpenReview ↗

Part 2: Pseudo-Huber Matching and the Scope of Amortised Control

(ii) Empirical Evaluation of Pseudo-Huber Adjoint Matching.

We use the pseudo-Huber loss to reduce the influence of large matching residuals and improve numerical robustness. Numerical stabilisation is also used by Domingo-Enrich et al. (2025), whose implementation masks individual sample–timestep loss terms using an adaptive threshold based on residual quantiles (code, lines 343–362).

Pseudo-Huber smoothly reduces the influence of large matching residuals, instead of discarding them. Both procedures modify the squared-loss training update and can change the optimisation solution. To assess the empirical effect of this modification, we conducted additional experiments on the OU benchmark, using the same equivariant controller architecture and training budget. We compared squared-loss training with pseudo-Huber training at a calibrated threshold δt\delta_{t}, and with thresholds ten times smaller, 0.1δt0.1\delta_{t}, to test stronger downweighting.

Mean ± std. across five independent evaluation sample banks.

Training ObjectiveW2W_{2} ↓MMD ↓Sinkhorn Cost ↓
CMDS - squared loss0.4967 ± 0.02170.01799 ± 0.008250.8882 ± 0.0114
CMDS - pHuber, δt\delta_{t}0.4969 ± 0.02160.01801 ± 0.008230.8883 ± 0.0114
CMDS - pHuber, 0.1δt0.1\delta_{t}0.5006 ± 0.02060.01822 ± 0.007810.8903 ± 0.0106

Although the squared-loss guarantees do not automatically extend to pseudo-Huber training, our additional OU experiment provides empirical support for this choice.

Prompted by the reviewer, we also show that the pseudo-Huber approximates squared AM when residuals are small. For a matching residual eθ(t)∈RDe_{\theta}(t)\in\mathbb{R}^{D}, the squared-AM and pseudo-Huber regression gradients are, respectively,

gsq(θ)=EPuθ[∫01(Dθuθ)⊤eθ(t)dt],g_{\mathrm{sq}}(\theta)=\mathbb{E}_{\mathbb{P}^{u_{\theta}}}[\int_{0}^{1}(D_{\theta}u_{\theta})^{\top}e_{\theta}(t)dt],

gPH(θ)=EPuθ[∫01(Dθuθ)⊤eθ(t)1+∥eθ(t)∥2/(Dδt2)dt],g_{\mathrm{PH}}(\theta)=\mathbb{E}_{\mathbb{P}^{u_{\theta}}}[\int_{0}^{1}(D_{\theta}u_{\theta})^{\top}\frac{e_{\theta}(t)}{\sqrt{1+\lVert{e_{\theta}(t)}\rVert^{2}/(D\delta_{t}^{2})}}dt],

where δt\delta_{t} is the pseudo-Huber threshold, DθD_{\theta} is the derivative with respect to θ\theta, and DD is the dimension of state space RD\mathbb{R}^{D}; rollout states, adjoint targets and thresholds are held fixed during differentiation. For each sampled trajectory and time tt, let gsq(t)g_{\mathrm{sq}}(t) denote the squared-AM gradient contribution and st=∥eθ(t)∥/(Dδt)s_{t}=\lVert{e_{\theta}(t)}\rVert/(\sqrt{D}\delta_{t}). Pseudo-Huber rescales this contribution as gPH(t)=gsq(t)1+st2.g_{\mathrm{PH}}(t)=\frac{g_{\mathrm{sq}}(t)}{\sqrt{1+s_{t}^{2}}}. Each contribution therefore retains its direction, while large residuals receive less weight. For small relative residuals, 11+st2=1−st22+O(st4),\frac{1}{\sqrt{1+s_{t}^{2}}}=1-\frac{s_{t}^{2}}{2}+O(s_{t}^{4}), so pseudo-Huber closely approximates each squared-AM gradient contribution, with a relative attenuation quadratic in sts_{t}. However, individual residuals can remain large at the SOC optimum even when their conditional mean is zero; residual-dependent reweighting can therefore change the expected gradient and its stationary points.

We will include this analysis in the manuscript’s newly added subsubsection "C.2.2. Pseudo-Huber adjoint matching and classical squared adjoint matching".

(iii) Amortization & Zero-Shot Transfer.

Amortised control means training a single context-conditioned controller and reusing it across new instances within a specified task family, without retraining. In CMDS, the context specifies, for example, an observed mixture or a start–goal configuration. This uses standard conditional learning: the cost of learning the control is shared across repeated inference. By contrast, DPS and PCD do not amortise coordination through a learned controller; they compute instance-specific steering updates during each sampling run.

Transfer to reward forms, layouts or agent counts outside the training family tests a stronger form of generalisation. Changing the reward beyond the training family changes the inference problem; zero-shot transfer therefore tests an additional capability, not one guaranteed by amortisation. The original manuscript already investigates such transfer in KUKA, while broader robustness remains an important direction that we now discuss in the limitations and future work.

AuthorsOpenReview ↗

Part 3: Zero-Shot Transfer Across Layouts, Agent Counts and Reward Parameters

To address the reviewer’s questions, we extend our zero-shot evaluation to changes in layout, agent count and reward parameters:

Layout transfer (Medium →\to Large) on PointMaze. We evaluate controllers trained on Medium with two agents on Large, without retraining the controllers. The Medium reference scores below reproduce Table 1. All scores are joint success percentages.

MethodMedium reference · Pool 1Medium reference · Pool 2Large transfer · Pool 1Large transfer · Pool 2
CMDS-TRG99.50 ± 0.599.20 ± 0.086.20 ± 0.4577.08 ± 4.58

For comparison, training directly on Large gives 91.1%/83.1% for CMDS and 96.6%/89.8% for CMDS-TRG on Pool 1/Pool 2 (Table 1).

Agent-count transfer on PointMaze. Here, we fix the layout, but investigate increasing the number of agents.

→ PointMaze Medium Layout, Trained on N=2, Evaluate on (3,4,5,6). We evaluate the same Medium-trained, two-agent controllers with three to six agents. Results are reported over three seeds.

NCMDS-TRG · Pool 1CMDS-TRG · Pool 2
299.50 ± 0.5099.20 ± 0.00
398.02 ± 1.0095.21 ± 1.48
494.90 ± 1.5794.32 ± 1.35
592.92 ± 0.9191.67 ± 2.15
691.20 ± 0.7790.21 ± 2.22

→ PointMaze Medium Layout, Trained on N=4, Evaluate on (2,3,5,6). Results are reported over three seeds.

NCMDS-TRG · Pool 1CMDS-TRG · Pool 2
298.80 ± 0.2398.18 ± 0.41
397.92 ± 0.8095.57 ± 0.76
497.40 ± 1.2093.50 ± 0.50
595.62 ± 1.2491.30 ± 1.16
695.36 ± 0.5688.59 ± 2.51

At six agents in Medium, controllers trained with two and four agents achieve 91.20%/90.21% and 95.36%/88.59% success on Pools 1/2, respectively.

→ PointMaze Large Layout, Trained on N=2, Evaluate on (3,4,5,6). Results are reported over three seeds.

NCMDS-TRG · Pool 1 ↑CMDS-TRG · Pool 2 ↑
296.60 ± 0.5089.80 ± 1.60
393.96 ± 2.0385.47 ± 4.81
487.34 ± 1.7580.62 ± 3.96
588.18 ± 2.8375.89 ± 4.70
684.53 ± 3.6973.28 ± 3.67

→ PointMaze Large Layout, Trained on N=4, Evaluate on (2,3,5,6). Results are reported over three seeds.

NCMDS-TRG · Pool 1CMDS-TRG · Pool 2
297.03 ± 1.1197.03 ± 0.88
397.71 ± 1.4796.61 ± 0.66
496.60 ± 1.2097.70 ± 0.00
596.04 ± 1.0395.31 ± 0.74
695.42 ± 1.2395.21 ± 0.89

We also provide qualitative examples comparing Reference and CMDS-TRG in Medium and Large mazes for N=2,…,6N=2,\ldots,6. These examples illustrate collision-free coordination with CMDS-TRG as the agent count increases, using controllers trained with two agents on Medium and four agents on Large, reused without retraining.

Reward-parameter transfer on KUKA. In KUKA, we assess sensitivity to reward specification by varying the collision-clearance margin at inference while keeping CMDS-TRG weights fixed. The modified reward enters through its gradient, whereas the base controller has no explicit margin input.

Inference reward clearanceSuccess (%)
1 cm77.30 ± 0.70
2 cm71.89 ± 2.09
3 cm60.36 ± 0.84
4 cm51.18 ± 0.42

To complete the discussion on the reviewer’s question about the contribution of g^ℓ\hat{g}_{\ell}, we also evaluate an untrained gradient-only variant, with neither the axial control branch nor the learned scaling function:

Inference reward clearanceSuccess (%)
1 cm46.75
2 cm39.05
3 cm28.99
4 cm20.71

The gradient-only variant achieves lower success than CMDS-TRG at every clearance, supporting the benefit of jointly learning the axial control branch and scaling function through our SOC formulation. This is also consistent with what is observed in [2].

We will add these results to the paper and thank the reviewer for prompting this broader evaluation.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Responses and Manuscript Revisions

We thank the reviewers for their constructive feedback, and for recognising the value and generality of coordinating frozen pre-trained models without joint training data (2Scz). We appreciate their positive comments on the SOC formulation (2Scz, g1Q9), efficient adjoint-based training (g1Q9), breadth of the evaluation (2Scz, g1Q9, w5dk) and strong empirical results (g1Q9, w5dk), including agreement with the OU oracle (2Scz), as well as the clarity of the presentation (g1Q9, w5dk) and derivations (g1Q9).

We now summarise the rebuttal by common themes and also highlight what has been updated in the paper.

  • TRG and the control architecture (2Scz, g1Q9, w5dk). We clarify that TRG parameterises the control under the same SOC objective, with the axial network and learned scaling trained jointly. In the manuscript: Section 3.3 and the new schematic in Appendix D.4.5 explain the shared architecture. Tables 2 and 4 now include KUKA without TRG and MDM with TRG, making its contribution explicit.

  • Matching accuracy and pseudo-Huber (2Scz, g1Q9, w5dk). We compare its gradient contributions with squared matching and explain both the local agreement and the possible change in stationary points. A matched OU experiment gives comparable distribution-recovery metrics. Our reply to g1Q9 also discusses numerical adjoint error and the role of state coverage. In the manuscript: The pseudo-Huber analysis and OU comparison appear in Section 3.3, Appendix C.2.2 and Table 6.

  • Amortisation and zero-shot transfer (2Scz, g1Q9). We clarify that amortisation shares the cost of learning control across task instances and transfer beyond the training family is evaluated separately. In the manuscript: Section 2.1 now states this scope, and Tables 12–13 report PointMaze transfer across layouts and agent counts, including six-agent evaluation.

  • Scalability (g1Q9). We explain why shared parameters accommodate different agent counts. In the manuscript: Appendix D.4.5 discusses this as well as sparse alternatives; Appendix D.6.6 and Table 19 report sampling time and memory up to eight people using CMDS.

  • Baselines, tuning and cost (2Scz, g1Q9, w5dk). Our replies describe controller training and inference costs, detail baseline tuning, and report Diffuser results. In the manuscript: Appendix A now explains the Diffuser adaptation and why a direct per-agent MultiDiffusion adaptation would not impose the required interactions.

  • Assembly rewards and experimental setup (w5dk, g1Q9). We clarify how each assembly is formed and which reward terms couple its components, report KUKA reward-clearance sensitivity, and explain KUKA task construction and MDM scene selection. In the manuscript: Section 3 in the main paper now makes the assemblies and their coupling rewards explicit, and discusses analytical versus learned rewards and the trade-offs in weighted objectives. Detailed task and scene protocols are described in Appendices D.5.1, D.6.1 and D.6.6.

  • Contribution and relation to adjoint matching (g1Q9). We distinguish the coordination framework from the inherited learning objective, explain when independent component controls is enough, and identify the cross-agent derivatives retained by the full adjoint. In the manuscript: These points are now developed in Section 2.1, Remark 3 in Appendix B.4, and Appendix C.2.1.

We thank the reviewers again for their constructive feedback, which we believe has helped to improve the paper. If these clarifications, additional experiments and revisions address your concerns, we would be grateful if you could raise your scores.

Dear Reviewer 2Scz, w5dk, g1Q9, thank you again for your insightful feedback! We would appreciate hearing whether our rebuttal addresses your concerns and would be happy to clarify any remaining questions.

If your concerns have been resolved, we would be grateful if you could consider updating your assessment and score accordingly.

Thank you for your time and help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.