acceptodds
Under review as a conference paper at ICLR 2027

Learning to Communicate: Toward End-to-End Optimization of Multi-Agent Language Systems

Abstract

Multi-agent systems built on large language models have shown strong performance on complex reasoning tasks, yet most work focuses on agent roles and orchestration while treating inter-agent communication as a fixed interface. Latent communication through internal representations such as key–value caches offers a promising alternative to text-based protocols, but existing approaches do not jointly optimize communication with multi-agent reasoning. Therefore we propose DiffMAS, a training framework that treats latent communication as a learnable component of multi-agent systems. DiffMAS performs parameter-efficient supervised training over multi-agent latent trajectories, enabling agents to jointly learn how information should be encoded and interpreted across interactions. Across five backbones and six mathematical reasoning, scientific QA, code generation, and commonsense benchmarks, DiffMAS matches or exceeds single-agent inference, text-based multi-agent systems, and prior latent communication methods in every setting, achieving +26.7% on AIME24 and +20.2% on GPQA-Diamond, and it also improves decoding stability. Against single-agent and text-based pipelines fine-tuned on identical data with the same recipe, DiffMAS is better in 26 of 30 settings and never worse, while fine-tuning through the text channel lowers accuracy in 13 of 30 settings, consistent with our analysis that text communication lets training adapt only how messages are read, not how they are written.

Primary area:
generative models
Keywords:
Multi-Agent Systems, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XF9FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Existing latent communication approaches for multi-agent systems (MASs) do not jointly optimize communication with multi-agent reasoning. This paper proposes DiffMAS, a training framework that treats latent communication as a learnable component of multi-agent systems. Across five backbones and six benchmarks, DiffMAS consistently matches or exceeds single-agent inference, text-based multi-agent systems, and prior latent communication methods.

Critical strengths

S1. The proposed approach is well-motivated and technically interesting.

S2. Extensive empirical studies demonstrate the effectiveness of the proposed approach.

Critical weaknesses

W1. The approach requires training different task-specific adapters.

W2. It remains unclear whether the proposed approach can work with agents based on different model backbones.

Questions for authors

Did you observe any training instability issues? Do you have learning curve figures for better understanding the training dynamics?

2 comments
AuthorsOpenReview ↗

Response to Reviewer XF9F

We appreciate your positive assessment of our work and your constructive suggestions for clarifying its applicability and training behavior. We have considered your concerns about adapter specialization, different backbones, and training stability, and address each point below.

W1. Adapter specialization and cross-domain transfer.

We would like to clarify that the adapters are trained by domain and reused across benchmarks. We train three adapters per backbone—one each for math, code, and commonsense—using 210 MATH problems, 50 HumanEval problems, and 700 CommonsenseQA traces, respectively. For example, the same MATH-trained adapter is used on AIME24, AIME25, and GPQA-Diamond. Its +20.2-point gain over the single-agent baseline on GPQA-Diamond with Qwen3-8B shows transfer from mathematical training data to graduate-level science questions. We hope this clarifies that a separate adapter is not needed for every benchmark. Training a single adapter jointly across domains would be a useful next step toward broader reuse.

W2. Compatibility across different backbones.

For the setting evaluated in this paper, the agents share a backbone and compatible KV layouts. Your question points to a useful extension: connecting agents with different backbones would require a differentiable mapping between their KV spaces. That mapping could be trained jointly with the adapters using the same final-answer objective. We will make the shared-backbone requirement clearer in the revision and explain how such a mapping could extend the approach to heterogeneous agents.

Q1. Training stability and learning curves.

We did not observe divergence or loss spikes in our training runs, which use gradient clipping at 1.0 and a cosine learning-rate schedule with 3% warmup. We agree that learning curves would make the training behavior easier to assess, and will include the training-loss curves in the appendix.

Thank you for your response. I don't have further questions. I'll keep my current rating.

Reviewer_99WgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces DiffMAS, which extends latent multi-agent systems with end-to-end supervised training. Four agents share a backbone and a LoRA adapter and communicate through an append-only KV-cache trace. The loss on the final agent's output propagates through this trace to the upstream agents, allowing both message generation and message consumption to be trained jointly. The paper analyzes this gradient pathway and evaluates five backbones on six benchmarks, comparing against training-free systems, matched supervised-training baselines, and a stitched-trace variant. It also examines decoding stability, latent-step counts, inference latency, and an alternative interaction topology. The reported results generally favor DiffMAS under the evaluated settings.

Critical strengths
  1. The controlled comparisons are useful and informative: Including Single+SFT and TextMAS+SFT with matched training data, adapter configuration, and schedule directly addresses whether the improvements arise merely from supervised fine-tuning. Across the reported backbone–benchmark combinations, DiffMAS exceeds the stronger of these two controls in 26 settings and ties in four. This consistent pattern across five backbones and six benchmarks provides the strongest support for the paper’s empirical contribution.
  2. The method offers a simple and practical way to optimize latent communication: Preserving differentiability through the shared KV trace allows the final task loss to reach upstream message-generation computations without requiring intermediate-agent targets. The implementation uses a small shared LoRA adapter and relatively limited training data. Reported inference latency is close to that of LatentMAS. This makes the approach practically interesting. The stability analyses and alternative-topology experiment also provide useful exploratory evidence beyond aggregate accuracy.
Critical weaknesses
  1. The methodological advance is incremental, and the experiments do not isolate its central claimed benefit: DiffMAS builds directly on LatentMAS, while differentiable communication and trained latent communication already have substantial precedent. In particular, the paper needs a sharper distinction from Interlat [1], which also trains latent-message generation using downstream task supervision. The specific combination of a shared adapter, joint optimization, and an append-only KV trace may be useful, but its distinctive benefit is not established. The most informative missing control is an otherwise identical training run with the upstream KV trace detached before the final agent. TextMAS+SFT changes the communication channel, and StitchMAS changes conditioning and positional structure; neither isolates the upstream gradient path's contribution. Consequently, the results do not establish how much improvement comes from optimizing upstream communication rather than adapting the final agent to latent inputs. Proposition 3.1 establishes a gradient pathway under its assumptions, but does not establish useful credit assignment or explain the empirical gains. This substantially weakens the central contribution.
  2. The reported performance advantage needs stronger baseline calibration and statistical support: Thinking-mode configuration, chat templates, intermediate-agent output handling, and truncation behavior are insufficiently documented. The single-agent results are also substantially below some published backbone results; for example, Qwen3-8B obtains 39.9 on GPQA-Diamond here, versus 62.0 in the Qwen3 thinking-mode report. Different evaluation protocols and token budgets prevent treating this discrepancy as proof of an incorrect baseline, but it warrants a clear explanation and reproducible configuration. Each AIME benchmark contains only 30 problems, and each problem is worth about 3.3% of the maximum score. The ten-step configuration was also selected using the AIME24 sweep, making that benchmark an imperfect independent evaluation.

Reference: [1] Enabling Agents to Communicate Entirely in Latent Space

Questions for authors
  1. Baseline configuration and provenance: Was Qwen3 thinking mode enabled consistently across Single, TextMAS, LatentMAS, and their trained variants? Please specify the chat templates, per-agent generation limits, reasoning-token handling, and truncation rates. Which baseline results did you rerun for this submission, and which, if any, did you reuse from prior work? How do these settings explain the discrepancy with published single-agent results?
  2. Contribution of upstream gradients: Can you report a representative comparison between DiffMAS and an otherwise identical model trained with the upstream KV trace detached before the final agent? A separately trained control with frozen upstream agents and adaptation restricted to the final agent would further distinguish direct optimization of communication from solver adaptation.
  3. Distinction from trained latent-communication methods: What specific capability or optimization property does DiffMAS introduce beyond prior methods such as Interlat? Please distinguish joint shared-adapter training through a KV trace from prior downstream-supervised latent-message training, and explain why you did not compare with a relevant established trained latent-communication method.
5 comments
AuthorsOpenReview ↗

[Part 1/2] Response to Reviewer 99Wg

We appreciate your detailed review and the concrete suggestions for clarifying the contribution of DiffMAS and strengthening its empirical support. Your proposed controls helped us examine the role of upstream gradients more directly. We have considered each of your concerns and address them individually below.

W1 / Q2. Isolating the contribution of cross-agent gradients.

Thank you for suggesting these controls. We ran both on Qwen3-8B with the same data, adapter configuration, training schedule, and decoding protocol. Detach retains the shared adapter and latent communication interface, but removes gradients through the upstream writer computations. Reader updates can still change upstream behavior indirectly through the shared adapter, allowing us to test the additional benefit of directly optimizing the writers. Solver-only LoRA freezes the upstream agents and adapts only the final agent.

MethodAIME24AIME25GPQA-D
LatentMAS (no training)56.753.345.5
Detach60.053.354.0
Solver-only LoRA66.753.353.0
DiffMAS (paper, single run)76.756.760.1

The results help answer your question about whether the gains come primarily from adapting the final agent. On AIME24 and GPQA-Diamond, DiffMAS exceeds Detach by 16.7 and 6.1 percentage points, respectively, and Solver-only LoRA by 10.0 and 7.1 points. This supports a contribution from directly optimizing upstream communication beyond reader adaptation alone. The table reports single-run results: the two controls use decoding seed 0, while DiffMAS and LatentMAS retain the results reported in the paper. We also report additional evaluations of the controls over three sampling seeds under W2 below.

We will add these controls to Section 6 and connect them explicitly to the gradient decomposition in Proposition 3.1.

W1 / Q3. Distinction from Interlat and comparison with trained latent communication.

We agree that Interlat is an important comparison and that the distinction should be stated more clearly. Interlat learns latent communication from downstream supervision: it passes last-layer hidden states through a learned communication adapter, trains a latent-message generator in its compression stage using a frozen actor, and supports end-to-end training in its multi-agent extension. Our contribution concerns the communication interface and its optimization: DiffMAS uses an append-only, per-layer KV trace with a shared LoRA adapter across agents. Each upstream segment remains directly accessible to the final agent, and the final-answer loss jointly updates the computations that write and read this trace. Proposition 3.1 characterizes this interface’s retention and gradient structure, and the detach control tests the practical value of gradients through the upstream writers.

To address your request for an established trained baseline, we evaluated the uncompressed receiver-training variant of the official Interlat implementation on Qwen3-8B, using our 210-sample MATH training set and evaluation protocol (avg@4). We used full-parameter receiver fine-tuning for 10 epochs, a 2,048-token training length, and a fixed sender. This configuration reaches 5.0 / 0.8 / 33.3 on AIME24 / AIME25 / GPQA-Diamond. On inspecting the outputs, we found that the fine-tuned receiver stops producing reasoning traces and writes short solutions (median 456 tokens on AIME24). We report these results for this specific Interlat variant and training configuration, alongside the matched controls that examine upstream writer optimization within DiffMAS.

MethodAIME24AIME25GPQA-D
Interlat5.00.833.3
Solver-only LoRA66.753.353.0
DiffMAS76.756.760.1

The Solver-only control helps separate a related question: can reader adaptation within the DiffMAS architecture explain the improvement? With the same data and training schedule, DiffMAS exceeds this control by 10.0 points on AIME24 and 7.1 points on GPQA-Diamond. These results support the benefit of directly optimizing upstream communication. We will include both the Interlat variant comparison and the matched controls, with their respective training settings made explicit in the revised paper.

AuthorsOpenReview ↗

[Part 2/2] Response to Reviewer 99Wg

W2 / Q1. Baseline configuration, provenance, and the GPQA-Diamond discrepancy.

We agree that the baseline provenance and evaluation settings should be easier to identify. For backbone–benchmark pairs reported in LatentMAS (Zou et al., 2025), we use the training-free baseline numbers from that paper and follow its evaluation protocol. We ran all other results, including every trained baseline and every new experiment reported here. The configuration is as follows:

  • Thinking mode: enabled for all methods.

  • Chat template: the standard chat template, following LatentMAS, for all methods.

  • Intermediate agents: in LatentMAS and DiffMAS, upstream agents produce 10 latent steps each and no text. In TextMAS, each upstream agent generates up to 4,096 tokens.

  • Final output limit: 2,048 (OpenBookQA), 4,096 (HumanEval+, MBPP+), 8,192 (GPQA-Diamond), 32,768 (AIME), identical for all methods (Appendix E).

  • Truncation rate of the final output: on GPQA-Diamond, 32.8% for Single and 4–7% for the multi-agent pipelines with three or more agents.

Regarding the difference from the Qwen3 report, our GPQA-Diamond evaluation uses an output budget of 8,192 tokens rather than 32,768. Under our cap, 32.8% of Single responses are truncated, compared with 4–7% for multi-agent pipelines with three or more agents. This provides a concrete indication that the output budget matters when interpreting the lower single-agent score. Within our evaluation, all methods use the same benchmark-specific output cap. We will clarify this protocol difference next to Table 1 and include the truncation statistics in the appendix.

W2. Statistical support and selection of the communication-step count.

To address your concern about sampling variability, we additionally evaluated the two gradient-isolation controls over three sampling seeds. Detach achieves 60.0 ± 2.7 / 52.2 ± 8.7 / 53.1 ± 1.3, and Solver-only LoRA achieves 64.5 ± 4.2 / 51.1 ± 5.7 / 54.0 ± 1.5 on AIME24 / AIME25 / GPQA-Diamond. These additional evaluations show that the seed-0 control results are not isolated observations.

You raise a useful point about selecting the communication budget on AIME24. To examine whether this choice transfers to other benchmarks, we re-ran the step-count sweep, retraining each adapter and evaluating on AIME25 and GPQA-Diamond. Neither benchmark was used to select the 10-step default:

Latent steps010 (paper)40100150200
AIME2550.056.760.050.050.050.0
GPQA-D54.560.158.156.156.156.6

We find that 10 steps gives the best GPQA-Diamond result, while AIME25 peaks at 40 steps; on AIME25, 10 steps also outperforms the 0-step and 100–200-step settings. This supports the transferability of the selected budget: the benefit of a compact communication trace extends beyond the benchmark used to choose the default.

AuthorsOpenReview ↗

Kind Reminder and Summary of Our Rebuttal

Dear Reviewer 99Wg,

Thank you again for your detailed feedback and concrete experimental suggestions. Since the discussion period is ending soon, we would like to briefly summarize the key points of our rebuttal:

  • Upstream gradient contribution: We implemented both controls you proposed. DiffMAS exceeds Detach by 16.7 / 6.1 points and Solver-only LoRA by 10.0 / 7.1 points on AIME24 / GPQA-Diamond, supporting a contribution beyond reader adaptation.
  • Comparison with Interlat: We clarified the distinction in communication interfaces and training settings, and added an evaluation of Interlat’s uncompressed receiver-training variant.
  • Evaluation and robustness: We detailed baseline provenance, decoding budgets, and truncation rates; reported additional three-seed evaluations of the controls; and extended the communication-step sweep to AIME25 and GPQA-Diamond.

Your suggestions helped us make the empirical evidence and contribution clearer. We would welcome further discussion and would appreciate your reconsideration of the paper based on these additions.

Reviewer_FAThOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed DiffMAS, a training framework that treats latent communication as a learnable component of multi-agent systems by continuous Key–Value (KV) caches rather than text. This framework solved problem in existing LLM-based multi-agent systems led by communications through discrete text messages and therefore optimized the multi-agent language systems.

Critical strengths

This paper has several advantages. The end-to-end optimization of inter-agent communication is a meaningful research direction, as this moves beyond the traditional directions of roles, prompts, and workflows of agents. It also provides simple and practical framework and shows strong empirical performance.

Critical weaknesses

There are three main concerns about this paper. The first one is problem formulation; the authors could not state a solid motivation of why optimization boundaries are a practical bottleneck. The second one is about causal validation. The experiments do not directly demonstrate that cross-agent gradient propagation drives the improvements. Finally, the paper’s theory part based on restrictive assumptions.

Questions for authors
  1. The paper suggests discrete message passing leads to optimisation boundaries, but does not explain enough what practical limitations these boundaries lead to. The authors should motivate the need for end-to-end differentiability.
  2. The paper claims performance gains from differentiable inter-agent communication. The current ablations however, do not directly disentangle the effect of gradient propagating across agent boundaries.
  3. Proposition 3.1 compares append-only communication to overwriting mechanisms under a contraction assumption which may not hold in practice for Transformer-based architectures. Even if there is a direct path along the gradient, it does not mean that the gradients are strong enough or informative enough.
  4. Proposition 3.1 discusses how gradient propagation depends on the number of stages, yet the experiments do not systematically vary the number of agents.
    
3 comments
AuthorsOpenReview ↗

[Part 1/2] Response to Reviewer FATh

We appreciate your careful review and your suggestions for clarifying the motivation, empirical evidence, and theoretical analysis of DiffMAS. We have considered each of your concerns and added experiments on cross-agent gradients and the number of agents. We address your questions individually below.

Q1. Practical motivation for end-to-end optimization.

We agree that the practical motivation deserves a clearer explanation. The setting we have in mind is one where supervision is available for the final answer, but not for the intermediate messages. For example, when the Solver produces an incorrect answer, we would like the training signal to improve both the Solver and the information supplied by the Planner. With sampled text messages, standard backpropagation cannot assign this credit through the message-generation process. DiffMAS allows the final-answer loss to train both message production and downstream reasoning, without requiring separate targets for each agent’s intermediate output.

Our matched-training results provide an example of why this matters. With the same training data, LoRA configuration, and optimization schedule, TextMAS+SFT falls below untrained TextMAS in 13 of 30 settings, whereas DiffMAS never falls below its untrained counterpart, LatentMAS (Section 6, Table 7). This comparison motivates learning communication together with reasoning. To examine the role of upstream gradients more directly, we also added the detach control described in Q2 below. We will bring this motivation and evidence into the Introduction.

Q2. Isolating the contribution of cross-agent gradients.

To address your concern directly, we added two controls on Qwen3-8B using the same data, LoRA configuration, optimization schedule, and decoding protocol. Together, they help distinguish the contribution of upstream gradients from adaptation of the final agent:

  • Detach: retains the same latent communication interface and shared adapter, but detaches the upstream trace before the final agent. At fixed parameters, this preserves the forward computation while removing gradients through upstream writer computations. Reader updates can still change upstream behavior indirectly through the shared adapter.

  • Solver-only LoRA: freezes the upstream agents and trains and applies the adapter only in the final agent, testing whether reader adaptation alone can explain the gains.

MethodGradient through upstream writer computationFinal agent adaptedAIME24AIME25GPQA-D
LatentMAS (no training)✗✗56.753.345.5
Detach✗✓60.053.354.0
Solver-only LoRA✗✓66.753.353.0
DiffMAS (paper, single run)✓✓76.756.760.1

The results indicate that adapting the final reader alone does not account for the full improvement. On AIME24 and GPQA-Diamond, DiffMAS exceeds Detach by 16.7 and 6.1 percentage points, respectively, and Solver-only LoRA by 10.0 and 7.1 points. This supports the contribution of directly optimizing upstream communication. The table reports single-run results: the two controls use decoding seed 0, while DiffMAS and LatentMAS retain the results reported in the paper. We will include these controls in Section 6 to make this distinction clearer.

AuthorsOpenReview ↗

[Part 2/2] Response to Reviewer FATh

Q3. Contraction assumption and the usefulness of gradients.

We would like to clarify where the contraction assumption enters Proposition 3.1. It is used only to derive the geometric-decay bound for the overwriting comparator; the DiffMAS results do not require it. In DiffMAS, each upstream KV segment remains an unchanged block of the final trace, regardless of whether subsequent Transformer stages are contractive. The retention bound and gradient decomposition therefore still hold when the contraction assumption fails. We will state this distinction explicitly in the proposition itself, as it is currently explained in the Scope paragraph of Appendix A.

You are right that a gradient path alone does not guarantee a strong or informative gradient. The specific property established by Proposition 3.1 is that the final loss reaches each upstream segment through a direct term with no product of intervening agents’ Jacobians (Eq. 3). This gives an earlier agent access to supervision without requiring the signal to pass through every later agent’s transformation. Attention allocation and cancellation can still affect the gradient magnitude, as discussed in Remarks A.1–A.2. To make this precise, we will replace “depth-independent gradient” with “a direct gradient term that bypasses intervening agent Jacobians.”

To address the practical part of your question, the matched detach experiment in Q2 provides complementary evidence: keeping the same latent interface while removing gradients through the upstream writers lowers performance on both AIME24 and GPQA-Diamond. Together, the analysis and this experiment identify the structural pathway and show the benefit of retaining it during training.

Q4. Effect of the number of agents.

Following your suggestion, we added an experiment varying K ∈ {1, 2, 3, 4}, while keeping the final agent’s Solver prompt unchanged (K=1: Solver; K=2: Planner→Solver; K=3: Planner→Critic→Solver; K=4: the full chain). Entries are AIME24 / AIME25 / GPQA-Diamond. Each setting reports a single run; the K=4 results match those in Q2.

KLatentMASDetachDiffMASDiffMAS − Detach
150.0 / 46.7 / 39.9–70.0 / 50.0 / 47.5–
250.0 / 46.7 / 41.966.7 / 50.0 / 52.570.0 / 53.3 / 52.5+3.3 / +3.3 / 0.0
353.3 / 50.0 / 45.566.7 / 50.0 / 54.073.3 / 53.3 / 56.1+6.6 / +3.3 / +2.1
456.7 / 53.3 / 45.560.0 / 53.3 / 54.076.7 / 56.7 / 60.1+16.7 / +3.4 / +6.1

On GPQA-Diamond, increasing K from 2 to 4 raises DiffMAS accuracy from 52.5 to 60.1, while Detach changes from 52.5 to 54.0. The gap between them grows from 0.0 to 2.1 to 6.1 points. We see this as evidence that cross-agent training helps the system make better use of additional upstream stages. The AIME25 differences remain within one problem, so our interpretation rests primarily on the GPQA-Diamond trend. These results complement the structural analysis in Proposition 3.1 by showing how the benefit of cross-agent training changes with the number of agents.

AuthorsOpenReview ↗

Kind Reminder and Summary of Our Rebuttal

Dear Reviewer FATh,

Thank you again for your thoughtful review. Since the discussion period is ending soon, we would like to briefly summarize the key points of our rebuttal:

  • Practical motivation and causal evidence: We clarified why final-answer supervision should also train upstream communication. The new Detach and Solver-only controls directly examine this contribution: DiffMAS outperforms Detach by 16.7 points on AIME24 and 6.1 points on GPQA-Diamond.
  • Theoretical assumptions: We clarified that the contraction assumption applies only to the overwriting comparator. The retention and gradient-decomposition results for DiffMAS do not require it.
  • Number of agents: We added a sweep from one to four agents. On GPQA-Diamond, the DiffMAS–Detach gap grows as additional upstream agents are introduced, supporting the benefit of training communication across stages.

We hope these additions address your main concerns. We would welcome any further questions and would appreciate your reconsideration of the paper in light of this evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.