acceptodds
Under review as a conference paper at ICLR 2027

Train Locally, Learn Together: Distributed Multi-Agent Learning via ADMM

Abstract

Multi-agent systems (MAS) can outperform individual language-model agents by combining specialized roles, diverse reasoning processes, and collaborative decision-making. However, collaborative MAS training is typically centralized, requiring a common trainer to aggregate role-local trajectories and learning signals and maintain model and optimization states. This motivates a distributed multi-agent learning paradigm in which agents improve system-level capabilities through role-local learning and decentralized communication, paving the way toward scalable, resilient, and self-organizing MAS. To this end, we introduce, to the best of our knowledge, the first optimization-grounded framework for distributed MAS training based on consensus alternating direction method of multipliers (ADMM). We formulate collaborative MAS learning as a consensus-constrained finite-sum optimization problem, naturally lending itself to distributed optimization via ADMM. The proposed approach alternates role-local learning with inter-agent consensus using distributed communication protocols such as ring all-reduce, enabling joint optimization while keeping each agent's data and training state local. We evaluate our framework on distributed safety alignment and general reasoning tasks. ADMM reduces HarmBench ASR by up to 26.4% points under compromised-agent settings, while achieving reasoning gains comparable to centralized training. These results show that distributed MAS training retains the benefits of centralized joint optimization without requiring a centralized trainer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Systems, Large Language Models, Distributed Optimization, Consensus Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jxJSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to train language-model-based multi-agent systems (MAS) without a centralized trainer that collects agents' trajectories, learning signals, parameters, and optimization states. The authors formulate collaborative MAS learning as a consensus-constrained finite-sum optimization problem, where each role-specific agent keeps a local copy of a shared LoRA adapter and optimizes a local objective based on its own role, prompts, messages, and rollout outcomes, with consensus ADMM reconciling these overlapping local copies through decentralized communication (mainly ring all-reduce). They find that distributed ADMM training substantially reduces harmful behavior in compromised MAS configurations while nearly matching centralized training.

Critical strengths
  1. The paper identifies the role-induced coupling in collaborative MAS learning and recasts it as a consensus-constrained optimization problem.

  2. The method also has certain general applicability; e.g., the local objectives of the roles can, in principle, be extended to other fine-tuning targets beyond the DPO-style loss described in the text.

  3. The ablation in Table 2 is especially useful. Local-only training and consensus averaging without dual updates are substantially worse than full ADMM, whereas full ADMM nearly matches centralized training.

Critical weaknesses
  1. Most reported results appear to be based on single runs, and the paper explicitly notes that some compromised-position configurations are evaluated only once.

  2. The main trained setting uses Qwen3-8B with rank-16 LoRA and four-agent sequential/hierarchical architectures. The scaling appendix mostly studies untrained systems at different numbers of agents, while only one additional trained experiment is reported at N=3. Thus, the claims about scalability and broad applicability should be moderated.

  3. The paper demonstrates locality of trajectories and optimization states, but this should not be conflated with a formal privacy guarantee. Agents still exchange model/adapter variables during consensus, which can potentially leak information. The paper does not analyze parameter leakage, membership inference, malicious update attacks, Byzantine behavior, or communication failures.

  4. The “compromised” agent is an obliterated backbone that remains trainable under the shared optimization procedure, rather than an actively adversarial participant that sends poisoned updates or strategically deviates from ADMM.

  5. Some claims in the introduction and conclusion should be calibrated more carefully: “scalable,” “resilient,” and “self-organizing” are broader than what the current experiments directly establish.

  6. I noticed minor proofreading issues, e.g., “centralized v.s. distributed,” which should be “vs.” or “versus”. A final copy-editing pass would be beneficial.

Questions for authors
  1. What exact information is exchanged during ring all-reduce or gossip, and what trust assumptions are made about communication peers? Please clarify whether the privacy claim is intended only as data/trajectory locality, rather than protection against leakage from exchanged adapter parameters.

  2. Does the compromised-agent setting assume that the compromised node follows the prescribed ADMM protocol and accepts fine-tuning updates? How would the method behave if a compromised node sent poisoned or inconsistent consensus messages/updates?

6 comments
AuthorsOpenReview ↗

Response to Reviewer jxJS (Part I)

We thank the reviewer for recognizing the role-induced coupling formulation, the flexibility of role-local objectives, and the value of the consensus and dual ablations.

We address each concern below (W: weakness; Q: question). Section, table, appendix, and line (L) references are to the revised paper, with changes in blue; R-prefixed tables and references appear in this response.

[W1] Most reported results appear to be based on single runs, and the paper explicitly notes that some compromised-position configurations are evaluated only once.

Response:

We added independent training seeds for the main comparison. ADMM reduces ASR and improves on NoDual in every seed. We also repeated the compromised-position evaluation with different sampling seeds; the smaller differences between the remaining positions are not interpreted as a stable ordering.

Table R7 summarizes the main comparison over three independent training seeds, including the original runs in Tables 1 and 2 (Table A6, L1242–1251). Each run uses the evaluation protocol of Appendix C.4.

Table R7. Main results over three independent training seeds (HarmBench ASR, %, mean ± standard deviation).

S-MASH-MAS
Benign base17.6119.50
Compromised base42.7752.83
Compromised, ADMM26.62 ± 2.0230.40 ± 4.89
Compromised, centralized27.04 ± 1.2627.88 ± 2.54
Compromised, NoDual39.62 ± 2.1840.04 ± 2.84

The additional seeds confirm the conclusions of Sec. 5.2: ADMM reduces ASR in every seed, differs from centralized training by −0.42 pp on S-MAS and +2.52 pp on H-MAS in mean ASR, and improves on NoDual in every seed (L1254–1258).

The compromised-agent effect recurs across configurations. One compromised agent raises HarmBench ASR in every configuration we evaluated:

  • 25.16 pp on S-MAS and 33.33 pp on H-MAS at N=4N=4 (Table 1, L419–420);
  • 14.47 to 32.08 pp in all nine further system sizes of Table A3 (L1140–1160), with S-MAS at N∈{2,3,5,6,7}N\in\{2,3,5,6,7\} and H-MAS at N∈{3,5,6,8}N\in\{3,5,6,8\};
  • 34.59 pp on the Qwen3-4B backbone (Table A5, L1182–1193).

The training effect recurs across settings and benchmarks. ADMM lowers ASR in all four settings of Table 1 and on all four safety benchmarks, with little change in utility (L415–427).

Prompt-level uncertainty. Table R8 reports the differences for the original Table 1 and 2 runs, with approximate 95% intervals over the 159 HarmBench test prompts. Training-seed variation is reported separately in Table R7.

Table R8. Prompt-level ASR differences in percentage points with approximate 95% intervals over 159 HarmBench test prompts. The final row gives the magnitude of the observed gap; paper Table A7 uses the signed ADMM minus centralized contrast.

ASR difference (pp)S-MASH-MAS
Compromised − benign25.16 ± 9.7 (p < 0.001)33.33 ± 9.9 (p < 0.001)
Base − ADMM17.61 ± 10.2 (p < 0.001)26.41 ± 10.4 (p < 0.001)
NoDual − ADMM16.98 ± 10.2 (p = 0.001)10.69 ± 10.2 (p = 0.04)
ADMM vs. centralized0.63 ± 9.6 (p = 0.90)1.26 ± 9.6 (p = 0.80)

The intervals of the compromised-agent effect, the training effect, and the dual-update gain exclude zero. ADMM and centralized training remain close: across training seeds, their mean ASRs differ by −0.42 pp on S-MAS and +2.52 pp on H-MAS (Table R7; Appendix F.1, L1257).

Compromised positions. We repeated the paper Table A1 evaluation three times with different sampling seeds (Appendix A, L965–971). The Solver and Synthesizer remain the worst single compromised positions in every repetition, with ASR 42.14 ± 2.88% and 53.46 ± 3.50%, respectively. Every other single position stays at most 7.55 percentage points above the benign MAS. We do not interpret the smaller differences among those other positions as a stable ordering, which motivates training all roles rather than assuming which one is compromised.

Appendix F.1 reports the three-seed summaries (L1229–1258), Appendix F.2 reports prompt-level intervals (L1266–1283), and Appendix A reports the repeated compromised-position evaluation (L965–971).

AuthorsOpenReview ↗

Response to Reviewer jxJS (Part II)

[W2] The main trained setting uses Qwen3-8B with rank-16 LoRA and four-agent sequential/hierarchical architectures. The scaling appendix mostly studies untrained systems at different numbers of agents, while only one additional trained experiment is reported at N=3. Thus, the claims about scalability and broad applicability should be moderated.

Response:

We added training at a larger system size, on a smaller backbone, and at other LoRA ranks. ADMM reduces ASR and closely matches centralized training in each tested setting. These experiments extend coverage but do not establish training scalability at much larger system sizes.

Scope of the scaling evidence. Appendix D reports the untrained compromised-agent effect for several sizes from N=2N=2 to N=8N=8 and training at N=3,4,6N=3,4,6 (L1140–1176). Training cost grows with NN, and our compute budget limits substantially larger trained systems (L1165–1167). Table R9 lists the added trained configurations.

The training comparison uses the remaining settings of Table A2; ASR is measured on HarmBench under compromised initialization.

Table R9. Additional trained configurations under compromised initialization (HarmBench ASR, %). Unless varied, use Qwen3-8B, four roles, and LoRA rank 16; see paper Table A4.

SettingBase (untrained)CentralizedADMM
S-MAS, N=3N=3, Qwen3-8B (Table A4)40.2535.2237.11
S-MAS, N=6N=6, Qwen3-8B (new)40.8829.5630.19
S-MAS, N=4N=4, Qwen3-4B (new)48.4330.8232.08
H-MAS, N=4N=4, LoRA rank 8 (new)52.8327.0427.67
H-MAS, N=4N=4, LoRA rank 32 (new)52.8325.1626.42

The base values come from Tables A3 and A5. ADMM reduces ASR in every setting and stays within 1.89 pp of centralized training (L1163–1165), extending the conclusion of Table 1 beyond the main setting.

Per-node communication. Each node holds its backbone and local LoRA copy. Ring all-reduce sends 2(N−1)/N2(N-1)/N times the trainable parameter size per node per round, below twice that size for any NN. Traffic is 134.5 MB at N=4N=4 and 149.5 MB at N=6N=6 (L1178–1180). Total rollout and training cost still grows with the number of roles.

Backbone and adapter. The role-local objective can be instantiated for other compatible backbones, LoRA ranks, or differentiable losses. Our tests cover Qwen3-8B and Qwen3-4B with ranks 8, 16, and 32. Appendix E retains broader heterogeneous architectures, interaction structures, and larger systems as future evaluation (L1199–1209).

Appendix D, Table A4, reports the additional trained configurations (L1163–1176); Appendix F.3 gives the matched-work and communication comparison (L1290–1310).

[W5] Some claims in the introduction and conclusion should be calibrated more carefully: "scalable," "resilient," and "self-organizing" are broader than what the current experiments directly establish.

Response:

The abstract presents scalable, resilient, and self-organizing MAS as a longer-term direction, using the phrase "paving the way toward" (L017–018). The evidence supporting that direction is narrower: bounded per-node consensus traffic, no central trainer, repair of the tested compromised systems, and gossip-based coordination.

Scalability. Per-node consensus traffic stays below twice the trainable parameter size (L1307–1308). The added trained configurations extend the evidence to six roles (Table A4, L1169–1176); they do not establish effectiveness at arbitrary system sizes.

Resilience. The framework avoids a central trainer and improves safety in the tested compromised-agent and active-update settings. This does not imply tolerance of arbitrary failures or attacks.

Self-organization. Consensus can use neighbour-to-neighbour gossip (L473–482). The present experiments still assume synchronous training with a fixed participant set (L1213–1214).

The abstract retains this directional wording. The conclusion (L513–515) and Appendix E (L1199–1221) clarify the scope and limitations of the evidence; we do not claim that large-scale resilience or dynamic self-organization has been established.

[W6] I noticed minor proofreading issues, e.g., "centralized v.s. distributed," which should be "vs." or "versus". A final copy-editing pass would be beneficial.

Response:

We thank the reviewer for the careful reading. In the revision, we proofread the paper.

AuthorsOpenReview ↗

Response to Reviewer jxJS (Part III)

[W3] The paper demonstrates locality of trajectories and optimization states, but this should not be conflated with a formal privacy guarantee. Agents still exchange model/adapter variables during consensus, which can potentially leak information. The paper does not analyze parameter leakage, membership inference, malicious update attacks, Byzantine behavior, or communication failures.

[Q1] What exact information is exchanged during ring all-reduce or gossip, and what trust assumptions are made about communication peers? Please clarify whether the privacy claim is intended only as data/trajectory locality, rather than protection against leakage from exchanged adapter parameters.

Response:

Our claim is data, trajectory, and training-state locality, not a formal privacy guarantee. The paper states it this way: training proceeds "without centralizing agents' trajectories, data, or optimization states" (L097–098), and the motivation is that a central trainer must access all role-local information, which may not be available across devices, organizations, or trust domains (L212–214). The paper makes no differential-privacy or leakage claim.

What is exchanged

  • One adapter-sized vector per node per round. In the zz-step, each node contributes bi=xi+uib_i=x_i+u_i (Algorithm A1, L986–987), whose size is that of the trainable up-projection BB (22.4M values, 89.7 MB in fp32; L1054). The down-projection AA is frozen and identical on all nodes, so it is never sent (L1051–1054).
  • Ring all-reduce computes the exact average of the bib_i, at 134.5 MB per node per round for N=4N=4 (L480–481, L1307–1308).
  • Gossip exchanges bib_i only with the two ring neighbours, for τ\tau rounds per consensus step (Fig. 5, L497–498).
  • Not sent in the consensus step: prompts, rollouts, rewards, gradients, and optimizer states. Role messages are passed only during rollouts, as in any MAS.

Trust assumptions

Peers are honest-but-curious, and communication is synchronous with a fixed set of nodes; asynchronous consensus and nodes that join or leave are listed as future work (Appendix E, L1206–1214). Malicious updates are addressed in our response to W4, Q2.

Leakage from exchanged parameters

Any collaborative training must exchange some information, and exchanged parameters can leak information, as in federated learning. Relative to centralized training, which receives every role's prompts, trajectories, rewards, and states, ADMM exposes only one adapter-sized vector per round. The zz-step needs only the sum of the bib_i, so it is compatible with secure aggregation [R1], which hides individual vectors, and with differentially private local updates (L1216–1218).

Appendix E explicitly states the trust assumptions and distinguishes locality from a formal privacy guarantee (L1213–1221).

Reviewer_8V2KOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes distributed training of role-specialized LLM agents using role-local DPO and consensus ADMM over LoRA adapters. Local optimization, consensus averaging, and dual updates coordinate training without a centralized trainer. Experiments on safety alignment and reasoning report substantial safety improvements and task performance approaching centralized training in the tested settings.

Critical strengths

Safety gains across two MAS architectures are accompanied by over-refusal and utility evaluations, helping assess their practical meaning. A shared, frozen LoRA factor makes consensus averaging exact in weight space, strengthening the consistency of the implementation.

Critical weaknesses

W1: Unresolved objective–implementation gap. The formulation models dependence on upstream parameters, but the implementation of Eqs. (7)–(8) freezes upstream messages and trains only the local adapter. The paper does not establish how this conditional surrogate optimizes Eq. (1), or reconcile consistency of overlapping parameter blocks with a shared adapter across roles. This weakens the central optimization claim.

W2: Insufficient evidence for ADMM's added value. FedAvg and DiLoCo are cited but not evaluated. Missing uncertainty estimates and small reasoning subsets leave near-centralized performance and the large dual-update gain insufficiently substantiated. Although some communication measurements and larger-system training are reported, matched efficiency comparisons are absent. These gaps prevent isolating ADMM's contribution from established distributed training methods.

Questions for authors

Q1 — Related to W1 (clarification): What exact optimization problem is implemented? How do fixed-context local updates relate to Eq. (1), and does consensus enforce agreement between copies of each upstream parameter block or one shared adapter across roles?

Q2 — Related to W2 (comparative evidence): Under matched rollout, optimization, communication, and tuning budgets, how does ADMM compare with FedAvg/DiLoCo and FedADMM? Can training-time, peak-memory, and total-communication measurements establish a practical advantage over centralized and distributed baselines?

Q3 — Related to W2 (robustness evidence): Does the S-MAS ASR reduction from 42.14% without dual updates to 25.16% with them persist across independent seeds and comparable tuning? What uncertainty intervals support the small ADMM–centralized performance differences?

Q4 — Preference construction (methodological clarification): Preferred and rejected messages originate from different rollouts but are scored under the preferred rollout's context. What establishes that the rejected message remains inferior in that context, and does a same-context pairing control preserve the gains?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 8V2K (Part I)

We thank the reviewer for recognizing the safety gains, utility evaluation, and exact weight-space averaging enabled by the frozen LoRA factor.

We address each concern below (W: weakness; Q: question). Section, table, appendix, and line (L) references are to the revised paper, with changes in blue; R-prefixed tables and references appear in this response.

[W1] Unresolved objective–implementation gap. The formulation models dependence on upstream parameters, but the implementation of Eqs. (7)–(8) freezes upstream messages and trains only the local adapter. The paper does not establish how this conditional surrogate optimizes Eq. (1), or reconcile consistency of overlapping parameter blocks with a shared adapter across roles. This weakens the central optimization claim.

[Q1] What exact optimization problem is implemented? How do fixed-context local updates relate to Eq. (1), and does consensus enforce agreement between copies of each upstream parameter block or one shared adapter across roles?

Response:

We implement global-consensus ADMM: each role trains a local copy of the shared LoRA parameters, and consensus averages bi=Bi+uib_i=B_i+u_i across roles (L341–344). The distinction between the general overlapping-block formulation and this implementation is clarified below. Fixed-context preference optimization is a role-local surrogate motivated by a score-function decomposition of the system reward.

The implemented problem

Eqs. (5) and (10) are the general form: Eq. (5) requires local copies of the same block to agree, and Eq. (10) averages block kk over the nkn_k agents that hold a copy of it (L272–275, L330–341). Ωi\Omega_i contains agent ii and its upstream agents (L239), so with role-specific adapters nkn_k depends on the MAS graph and can be smaller than NN.

Our experiments use the global-consensus instance. Roles share the same backbone and train local copies of the shared LoRA parameters. Hence xi=θi=Bix_i=\theta_i=B_i and nk=Nn_k=N; no separate upstream blocks are maintained (L341–344, L1021–1025). The down-projection AA is frozen and shared (L1051–1054). The implemented fixed-context surrogate problem is

min⁡{xi},z∑i=1Nfi(xi)\min_{\{x_i\},z}\sum_{i=1}^Nf_i(x_i) subject to xi=z,i=1,…,Nx_i=z, i=1,\ldots,N.

Here fif_i includes the DPO loss of Eq. (8) and the forward KL regularizer with weight 0.1 (L308–319; Table A2, L1092). Each role takes 16 Adam steps on its loss plus the proximal term, starting from z−uiz-u_i (L1014–1016, L1023–1024). The roles then average bi=xi+uib_i=x_i+u_i to form zz and update their duals locally. Consensus requires all roles' copies to agree on the deployed parameters zz.

Fixed context updates and the system objective

Let J(θ)=Eθ[R(q,m1:N)]J(\theta)=E_\theta[R(q,m_{1:N})], with each message sampled conditional on its role prompt, query, and upstream messages. The score-function identity gives

∇J(θ)=∑i=1NEθ[R∇θlog⁡πθ(mi∣ci)]=∑i=1Ngi(θ)\nabla J(\theta)=\sum_{i=1}^N E_\theta[R \nabla_\theta\log\pi_\theta(m_i| c_i)]=\sum_{i=1}^Ng_i(\theta)

Each term differentiates one role's log-likelihood with its sampled context fixed. This motivates the conditional update in Eq. (8). Its safety label also checks the role's own message (L305–307), so the preference loss is a surrogate for the reward term; it is not an exact reward-gradient estimator.

The upstream dependence is represented by upstream roles' own score terms and by the complete-rollout outcome used to construct preference labels. It is not removed merely by holding sampled messages fixed during likelihood differentiation.

Consensus averages the corrected role-local copies. For fixed differentiable local losses and an exact ADMM fixed point, xi=zx_i=z and ∑i∇fi(z)=0\sum_i\nabla f_i(z)=0 at the shared deployed parameters. This fixed-point property does not prove convergence of the finite-step nonconvex training procedure.

Rollout refresh. Both safety and reasoning training collect fresh rollouts under the current consensus parameters in every round (L377–401, L1066–1068). This refreshes sampled contexts at the current policy; the surrogate and cross-context pairing approximations remain.

Sec. 4 now states the implemented global consensus (L341–344). Appendices B and G clarify the maintained variables and conditional updates (L1021–1025, L1376–1385), and Appendix C.3 describes rollout refresh (L1066–1068).

AuthorsOpenReview ↗

Response to Reviewer 8V2K (Part II)

[W2] Insufficient evidence for ADMM's added value. FedAvg and DiLoCo are cited but not evaluated. Although some communication measurements and larger-system training are reported, matched efficiency comparisons are absent. These gaps prevent isolating ADMM's contribution from established distributed training methods.

[Q2] Under matched rollout, optimization, communication, and tuning budgets, how does ADMM compare with FedAvg/DiLoCo and FedADMM? Can training-time, peak-memory, and total-communication measurements establish a practical advantage over centralized and distributed baselines?

Response:

Under matched per-run rollout, update, and communication budgets, independently tuned ADMM attains lower ASR than FedAvg, DiLoCo, and FedProx in both systems (Table R4). Its fixed-penalty update coincides with FedADMM under full participation. The matched-work comparison with centralized training is reported in Table R5.

Comparison with FedAvg, DiLoCo, and FedProx

Tuned means that each method's hyperparameters are selected independently by grid search on a held-out HarmBench validation set disjoint from the training and test prompts. Per-run budgets match: the same training queries, rollouts per query, local steps, and rounds, with one trainable-parameter-sized vector contributed per node per consensus round. FedAvg searches learning rates {5×10−4,10−3,2×10−3}\{5\times10^{-4},10^{-3},2\times10^{-3}\}; FedProx searches these learning rates and ρ∈{0.01,0.03,0.1,0.3}\rho\in\{0.01,0.03,0.1,0.3\}; DiLoCo searches outer learning rates {0.3,0.7,1.0}\{0.3,0.7,1.0\} with Nesterov momentum 0.9; ADMM searches the same penalty grid as FedProx and selects ρ=0.03\rho=0.03 (Appendix F.1, L1235–1241). The grids contain different numbers of configurations; the budgets above are per run. Selected configurations are evaluated over three independent training seeds using Appendix C.4 of the paper. The ADMM and fixed-penalty NoDual summaries include the runs of paper Table 2.

Table R4. Independently tuned baselines over three training seeds, mean ± standard deviation (HarmBench ASR, %). Each method is selected separately on held-out validation prompts; see paper Table A6.

MethodProximalDualS-MASH-MAS
Base (untrained)––42.7752.83
FedAvg (tuned)NoNo40.88 ± 2.2741.51 ± 2.74
DiLoCo (tuned)NoNo36.06 ± 2.9737.95 ± 4.76
NoDual (ρ=0.03\rho=0.03)YesNo39.62 ± 2.1840.04 ± 2.84
FedProx (tuned)YesNo37.95 ± 3.2338.78 ± 2.21
ADMMYesYes26.62 ± 2.0230.40 ± 4.89
Centralized––27.04 ± 1.2627.88 ± 2.54

ADMM is lower than FedAvg, DiLoCo, and FedProx in every seed (L1254), with reported mean gaps of 9.44 and 7.55 percentage points to the best tuned baseline on S-MAS and H-MAS. This comparison supports the role of dual feedback beyond averaging and the proximal term.

Why the dual matters. For fixed differentiable role-local losses and exact stationary local subproblems, averaging without a dual gives ∇fi(xi)=(z−xi)/λ\nabla f_i(x_i)=(z-x_i)/\lambda, so the summed-gradient condition holds at the local copies and need not hold at the deployed zz. The dual accumulates xi−zx_i-z (L266–267, L353–356) and shifts the proximal anchor to z−uiz-u_i. At an ADMM fixed point, xi=zx_i=z for every role; local optimality gives ∇fi(z)=−ui/λ\nabla f_i(z)=-u_i/\lambda, and the consensus step gives ∑iui=0\sum_i u_i=0. Hence ∑i∇fi(z)=0\sum_i\nabla f_i(z)=0 at the deployed parameters. This is a fixed-point property of the role-local surrogate, not a convergence guarantee for the finite-step nonconvex implementation.

FedADMM

FedADMM [R1] also uses local updates, consensus, and duals. With full participation and a fixed ρ\rho, its client and server updates coincide with ours. We also ran its adaptive-penalty variant, which gives 25.79 on S-MAS and 27.04 on H-MAS, on par with ADMM (Appendix F.1, L1260–1263). Its remaining mechanisms target system heterogeneity among clients, which our synchronous setting does not have. What differs from [R1] is the problem: FedADMM's clients have fixed, independent data, whereas our roles generate each other's contexts.

AuthorsOpenReview ↗

Response to Reviewer 8V2K (Part III)

Efficiency

Rollout generation and pair selection use the same training queries, rollout budget, and filtering rule in every arm, with each arm sampling from its own current policy (L1078–1079, L1292–1293). For safety, the queries comprise 380 harmful and 300 benign prompts, with up to eight rollouts per query (L374–376, L1070). Retained pair counts depend on the sampled outcomes and need not be identical. ADMM assigns each role's pairs to its node; centralized training pools them. The default matches work per trainer: 16 Adam steps of batch size 8 per round (Table A2; L1096–1097). Four-node ADMM therefore takes 64 local steps in total, so we additionally evaluate a centralized run with 64 steps per round and all other settings unchanged.

Table R5. H-MAS comparison with matched total optimization steps and consensus communication, as in paper Table 3 and Appendix F.3.

MethodTotal steps per roundASR (%)MB per node per round
Centralized1625.16—
Centralized, compute-matched6425.79—
ADMM4 × 1626.42134.5

Matching total optimization work gives 25.79% ASR for centralized training versus 26.42% for ADMM on H-MAS; the default centralized run gives 25.16% (Table 3, L459–470).

Communication. Ring all-reduce sends 2(N−1)/N2(N-1)/N times the size of BB per node per round. This is 134.5 MB for N=4N=4, or approximately 3.23 GB over 24 H-MAS rounds, and stays below twice the trainable parameter size for any NN (L1307–1310). These figures count consensus traffic, excluding inference-time role messages. Prompts, rollouts, rewards, gradients, and optimizer states are not gathered during consensus. The practical benefit is locality of training information; this comparison does not establish an end-to-end training-time or memory advantage.

The practical advantage is locality: each role keeps its rollout feedback and optimizer state on its node, avoiding a trainer that must access every role-specific training record.

Appendix F.1 reports the tuned baselines and three-seed summaries (L1229–1263); Appendix F.3 (L1290–1310) and main-text Table 3 (L459–472) report the matched-work and communication comparison.

[W2] Missing uncertainty estimates and small reasoning subsets leave near-centralized performance and the large dual-update gain insufficiently substantiated.

[Q3] Does the S-MAS ASR reduction from 42.14% without dual updates to 25.16% with them persist across independent seeds and comparable tuning? What uncertainty intervals support the small ADMM–centralized performance differences?

Response:

Yes. Over 3 independent training seeds, each evaluated with the protocol of Appendix C.4, the dual-update gain is positive in every seed on both systems, and ADMM's mean ASR is 0.42 pp lower than centralized training on S-MAS and 2.52 pp higher on H-MAS (Appendix F.1, L1254–1258).

Table R6. HarmBench ASR (%, mean ± standard deviation) over three independent training seeds. Tuned NoDual is the FedProx comparison.

ASR (%, ↓)S-MASH-MAS
Base (untrained)42.7752.83
NoDual (ρ=0.03\rho=0.03)39.62 ± 2.1840.04 ± 2.84
NoDual (tuned)37.95 ± 3.2338.78 ± 2.21
ADMM26.62 ± 2.0230.40 ± 4.89
Centralized27.04 ± 1.2627.88 ± 2.54
  • The dual-update gain persists across seeds and tuning. It is 13.00 ± 3.79 pp on S-MAS and 9.64 ± 2.38 pp on H-MAS with the same ρ\rho, and 11.32 / 8.39 pp against the best-tuned NoDual (L1254–1257). It is positive in every seed, which confirms the Table 2 result (42.14 → 25.16 on S-MAS, one of the three seeds; L451–455, L1243).
  • ADMM and centralized training are comparable. ADMM's mean ASR is 0.42 pp lower than centralized training on S-MAS and 2.52 pp higher on H-MAS.
  • Prompt-level uncertainty for the Table 1 and 2 runs. Over the 159 HarmBench test prompts, the 95% intervals of the dual-update gain (16.98 ± 10.2 and 10.69 ± 10.2 pp) and of the training effect exclude zero, while ADMM and centralized training remain close, 0.63 and 1.26 pp apart (Table A7, L1268–1283).

Reasoning. Every method is evaluated on the same questions (L1129–1130). On all held-out test questions, excluding those reserved for training, ADMM reaches 75.43% mean accuracy, vs. 75.71% for centralized training and 73.84% for the Base MAS; the 95% bootstrap interval of the ADMM–centralized difference is [−1.08, 0.52] pp (L1284–1288). This confirms Fig. 4.

Appendix F.1 reports training-seed summaries (L1229–1258), and Appendix F.2 reports prompt-level intervals and held-out reasoning results (L1266–1288). These distinguish variation across training runs from uncertainty over test prompts.

Reviewer_EySgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes distributed training of multi-agent language-model systems through role-local preference optimization and ADMM consensus. Agents perform local adapter updates with a quadratic penalty, followed by consensus averaging and dual updates. Communication uses ring all-reduce or approximate gossip. The described implementation freezes a common LoRA down-projection and trains the up-projection factors. Safety experiments evaluate sequential and hierarchical systems, while reasoning experiments evaluate the sequential system. Comparisons include centralized training and variants without consensus or dual updates.

Critical strengths

1-The deployment problem is meaningful. Keeping role-local trajectories and training states separate is useful, and evaluating both safety and reasoning gives the study a practical scope. Including over-refusal and utility measurements also helps assess the safety results.

2-The complete training procedure shows encouraging empirical results. It improves over the untrained systems on the evaluated safety and reasoning tasks and achieves observed performance close to centralized training. The local-only and no-dual ablations provide useful evidence that coordination matters within the evaluated implementation.

Critical weaknesses

1-The mathematical formulation is not consistently connected to the described updates.

2-Section 3 defines theta_i as a local copy of one shared adapter. Equation (5), however, enforces agreement among replicas of each role-specific parameter block. This is a valid variable-splitting constraint for the multi-block objective in Equation (1), but it does not enforce equality between different roles’ adapter parameters. For a two-agent chain, it equates the two copies of theta_1 without imposing theta_1 = theta_2. Appendix B describes initialization and local updates in terms of a single adapter factor B_i, without explaining how the upstream parameter blocks in x_i are represented and updated. The paper needs a consistent account of the variables maintained, averaged, and deployed.

3-There is also an unresolved connection between Equations (1) and (8). The paper explicitly treats sampled upstream messages as fixed during local DPO updates. This can be a reasonable training surrogate, but its relationship to the original objective, whose dependence on upstream parameters arises through generated messages, is not established. Collecting new rollouts under updated parameters does not by itself explain that relationship or show how consensus addresses the interaction-induced coupling represented in Equation (1). These gaps leave the paper’s central optimization claim insufficiently established.

4-The methodological and practical advantages over established distributed training remain underdemonstrated.

5-FedADMM already combines local optimization, consensus, and dual feedback [1], and ADMM-based decentralized multi-agent policy optimization has also been studied [2]. FFA-LoRA [3] and MARTI [4], both already cited in the manuscript, provide relevant precedents for frozen-factor adapter training and distributed agent-policy training, respectively. MARTI retains centralized interaction and reward infrastructure, and these works differ from the present setting. The paper nevertheless needs to establish what its formulation contributes to the specific challenge of learning with interacting LLM roles.

6-The experiments do not isolate this contribution sufficiently. ADMM-NoDual already provides periodic consensus and, under the stated update equations, retains the quadratic proximal term. It is therefore a useful comparison with proximal local averaging. However, the paper does not establish whether the advantage persists against independently tuned averaging or proximal-training baselines. The communication experiments and supplementary system-size analyses are useful, while end-to-end evidence on training time, memory, and training scalability remains limited. The results demonstrate that the procedure can improve a multi-agent system, but its methodological and practical advantages over established distributed alternatives remain unclear.

References:

[1] Gong, Li, and Freris. FedADMM: A Robust Federated Deep Learning Framework with Adaptivity to System Heterogeneity. 2022. arXiv:2204.03529.

[2] Li and He. Multiagent Trust Region Policy Optimization. IEEE Transactions on Neural Networks and Learning Systems, 35(9):12873–12887, 2024. DOI: 10.1109/TNNLS.2023.3265358.

[3] Sun et al. Improving LoRA in Privacy-preserving Federated Learning. ICLR, 2024. arXiv:2403.12313.

[4] Zhang et al. MARTI: A Framework for Multi-Agent LLM Systems Reinforced Training and Inference. ICLR, 2026.

Questions for authors

1-Does the implementation enforce one shared adapter across roles or consistency among replicas of separate role-specific parameter blocks?

2-Can you specify the variables stored, updated, and exchanged in a two-agent chain, including any upstream parameter blocks, and show how they correspond to Equations (5) and (10) and Appendix B?

3-What relationship connects the fixed-context DPO loss in Equation (8) to the interaction-coupled objective in Equation (1), including the role of rollout refresh?

4-How are preference labels validated when messages from different rollouts are evaluated under the preferred rollout’s context?

5-Does the advantage over ADMM-NoDual persist against independently tuned averaging or proximal training under comparable optimization and communication budgets?

6-Given the shared optimizer settings in Table A2, how are total rollout generation, retained training examples, and total optimization work matched between centralized and distributed runs? What are the corresponding training-time, memory, and communication costs?

Update: Thank you for the clarifications and additional experiments. They strengthen the empirical evidence, but my main concern remains the gap between the stated optimization problem and the implemented method.

Equation (5) enforces consistency between copies of the same parameter block; it does not require different roles to share one adapter. The rebuttal clarifies the shared-adapter implementation but does not establish its equivalence to the original formulation. Similarly, the score-function identity does not derive the conditional DPO updates from Equation (1). Acknowledging DPO as a surrogate narrows the claim without establishing this missing connection.

The additional experiments support the procedure’s effectiveness in the tested settings. However, the clarified contribution is primarily an application of established consensus ADMM, while the central formulation-to-implementation relationship remains insufficiently justified. I do not consider this sufficient to change my assessment and therefore maintain my rating of 1 (Clear rejection).

7 comments
AuthorsOpenReview ↗

Response to Reviewer EySg (Part I)

We thank the reviewer for recognizing the value of keeping role-local trajectories and training states separate and for the positive assessment of our safety, reasoning, utility, and coordination evaluations.

We address each concern below (W: weakness; Q: question). Section, table, appendix, and line (L) references are to the revised paper, with changes in blue; R-prefixed tables appear in this response.

[W1, W2] The mathematical formulation is not consistently connected to the described updates. Section 3 defines theta_i as a local copy of one shared adapter. Equation (5), however, enforces agreement among replicas of each role-specific parameter block. This is a valid variable-splitting constraint for the multi-block objective in Equation (1), but it does not enforce equality between different roles' adapter parameters. For a two-agent chain, it equates the two copies of theta_1 without imposing theta_1 = theta_2. Appendix B describes initialization and local updates in terms of a single adapter factor B_i, without explaining how the upstream parameter blocks in x_i are represented and updated. The paper needs a consistent account of the variables maintained, averaged, and deployed.

Response:

We believe this is a misunderstanding of the general formulation and its implemented instance. Equation (5) requires copies of the same parameter block to agree; Eq. (10) is its blockwise consensus average (L272–275, L330–341). For distinct role-specific blocks, the number of holders nkn_k can be smaller than NN (L334–335). Our experiments use the global-consensus instance described below.

In our implementation, every role maintains a local copy of the shared LoRA parameters (L341–344). Thus, nk=Nn_k=N, the consensus constraint is x1=⋯=xNx_1=\cdots=x_N, and Eq. (10) averages across all NN roles.

Each agent ii maintains and trains its own local copy θi\theta_i. In a two-agent chain, θ1\theta_1 and θ2\theta_2 are trained on their respective role-local losses, with θ1=θ2\theta_1=\theta_2 as the consensus constraint. The general notation xi=θΩix_i=\theta_{\Omega_i} allows distinct overlapping blocks, but our global-consensus implementation stores no separate upstream parameter copies: xi=θi=Bix_i=\theta_i=B_i. Only the LoRA up-projection BiB_i is trained; the down-projection AA is frozen and shared. This is the controlled common-backbone setting described in Secs. 3 and 4 and Appendices B, C.2, and E (L218–223, L341–344, L1021–1025, L1049–1054, L1199–1200).

The revised Sec. 4 explicitly states the global-consensus instance, including xi=θi=Bix_i=\theta_i=B_i, nk=Nn_k=N, and deployment of the consensus parameters by every role (L341–344).

[Q1] Does the implementation enforce one shared adapter across roles or consistency among replicas of separate role-specific parameter blocks?

Response:

The implementation enforces global consensus across roles. Each role trains a local copy xix_i on its own loss, and the constraint x1=⋯=xNx_1=\cdots=x_N requires those copies to agree. The consensus step produces zz, which every role uses at deployment.

[Q2] Can you specify the variables stored, updated, and exchanged in a two-agent chain, including any upstream parameter blocks, and show how they correspond to Equations (5) and (10) and Appendix B?

Response:

In the two-agent chain, each agent ii stores its copy xi=Bix_i=B_i, the consensus zz, its dual uiu_i, the frozen and shared LoRA down-projection AA, its own optimizer state, and its own rollouts. Eq. (5) is the constraint x1=x2x_1=x_2. Each round consists of three steps in the two-agent chain setting:

  1. Local step. Each role takes 16 Adam steps on Eq. (7), including its DPO loss, forward KL regularizer, and proximal term 12λ∥xi−(z−ui)∥22\frac{1}{2\lambda}\|x_i-(z-u_i)\|_2^2, starting from z−uiz-u_i with sampled upstream messages fixed.
  2. Consensus step. The roles contribute bi=xi+uib_i=x_i+u_i and compute z=(b1+b2)/2z=(b_1+b_2)/2, which is Eq. (10) with nk=N=2n_k=N=2.
  3. Dual step. Each role updates ui←ui+xi−zu_i\leftarrow u_i+x_i-z locally.

Both roles deploy zz.

Appendix B now gives this two-agent account of stored, updated, exchanged, and deployed variables (L1021–1025).

AuthorsOpenReview ↗

Response to Reviewer EySg (Part II)

[W3] There is also an unresolved connection between Equations (1) and (8). The paper explicitly treats sampled upstream messages as fixed during local DPO updates. This can be a reasonable training surrogate, but its relationship to the original objective, whose dependence on upstream parameters arises through generated messages, is not established. Collecting new rollouts under updated parameters does not by itself explain that relationship or show how consensus addresses the interaction-induced coupling represented in Equation (1). These gaps leave the paper's central optimization claim insufficiently established.

Response:

The implementation uses global-consensus ADMM over role-local preference losses. Fixing sampled upstream messages during a local update does not mean that roles learn from independent data: those messages are generated by the other roles, and the preference feedback is derived from the full rollout (L304–307). The score-function identity clarifies the connection to the system reward, while Eq. (8) remains a surrogate.

Why does fixing upstream messages preserve the coupling?

Let J(θ)=Eθ[R(q,m1:N)]J(\theta)=E_\theta[R(q,m_{1:N})], where mi∼πθ(⋅∣ci)m_i\sim\pi_\theta(\cdot| c_i) and cic_i contains the role prompt, query, and upstream messages. Factorization of the joint rollout probability gives the role-wise score decomposition

∇J(θ)=∑i=1NEθ[R∇θlog⁡πθ(mi∣ci)]=∑i=1Ngi(θ)\nabla J(\theta)=\sum_{i=1}^N E_\theta[R \nabla_\theta\log\pi_\theta(m_i| c_i)]=\sum_{i=1}^Ng_i(\theta) Each gig_i differentiates role ii's log-likelihood with its sampled upstream context fixed. Upstream dependence is represented by the upstream roles' score terms, and downstream consequences enter through the complete-rollout reward RR. This motivates conditional role-local updates; it does not make the DPO loss an exact estimator of the system reward gradient.

[Q3] What relationship connects the fixed-context DPO loss in Equation (8) to the interaction-coupled objective in Equation (1), including the role of rollout refresh?

Response:

Equation (8) is a role-local preference surrogate motivated by the fixed-context score terms above. Its labels incorporate the complete rollout outcome and, for safety, also check the role's own message. In addition, the rejected message may originate from a different upstream context. The response to Q4 identifies when that label transfer is justified and reports the same-context control for the intermediate roles.

Role of rollout refresh. Contexts in the score decomposition are drawn from the current policy. Both safety and reasoning training deploy the current consensus parameters and collect fresh rollouts in every round. Refresh updates the sampled context distribution; it does not remove the surrogate and cross-context pairing approximations.

Sec. 5.1 (L377–401) and Appendix C.3 (L1066–1068) now state that both tasks refresh rollouts every round. Appendix G briefly clarifies the implemented variables and conditional updates (L1376–1385).

AuthorsOpenReview ↗

Response to Reviewer EySg (Part III)

[Q4] How are preference labels validated when messages from different rollouts are evaluated under the preferred rollout's context?

Response:

Each message retains the label of the rollout that produced it. Following prior multi-agent LLM training, the role-local label rir_i uses complete-rollout feedback: harmful prompts are judged using the HarmBench 13B classifier on the final answer, and benign prompts by answer correctness. The role's own message is also checked as described in Appendix C.3 (L305–307, L1103–1111, L1118–1121). Whether the rejected label carries over to the preferred context depends on the role:

  • Roles without upstream input (the three H-MAS Analyzers and the S-MAS Solver) see the same context, i.e., system prompt and query, in every rollout. Scoring m−m^- under m+m^+'s context is therefore scoring it under its own context, which is the standard DPO setting of two responses to the same prompt.
  • The final role (the H-MAS Synthesizer and the S-MAS Decider) writes the final answer. For this role, the evaluator assigns the same score to a given query–answer pair across upstream contexts, so the label carries over; the rejected answer can still originate from a different context.
  • Intermediate roles (the S-MAS Verifier and Corrector) are the only case where the label may not carry over: m−m^- was generated under a different upstream context, and the outcome also depends on later roles. For these roles, the pairing is an approximation.

Every H-MAS pair therefore has a label that holds in m+m^+'s context, so the H-MAS results in Table 1 do not rely on the label approximation.

Same-context control for the intermediate roles. For the S-MAS Verifier and Corrector, we resample the role's message 4 times under each sampled context, complete each rollout downstream, and pair only messages generated under the same context; the control covers these two roles, whose labels are the only approximate ones. S-MAS HarmBench ASR is 24.53% with this pairing, vs. 25.16% in Table 1: the ASR reduction was preserved when these two roles used same-context pairs.

Sec. 4 identifies the role-dependent label-transfer issue (L313–318); Appendix F.4 reports the same-context preference control (L1312–1323).

Comments on the paper

1 comment

Dear Reviewers, 

We’re about halfway through the discussion period. Please read the authors’ rebuttal and assess whether it adequately addresses the key concerns raised in your review. We encourage you to leave a follow-up comment for the authors, indicating which concerns have been resolved and whether any important issues remain or require further clarification. Please update your review and scores as appropriate in light of the rebuttal and discussion.

Thank you for your time and participation. 

Best regards, 

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.