acceptodds
Under review as a conference paper at ICLR 2027

Quantifying GPU Energy Savings from Latent Communication in Multi-Agent LLM Inference

Abstract

Latent-space communication replaces decoded text between agents with hidden states and KV-cache transfer. Although prior work reports speed and token reductions, its energy implications have not been measured directly. We compare single-agent, text-based multi-agent (TextMAS), and latent multi-agent (LatentMAS) inference on identical NVIDIA A100 hardware for Qwen3-8B and LLaMA 3.1-8B-Instruct on GSM8K. Relative to TextMAS, latent communication reduces net GPU energy by 70.7% for Qwen3-8B and 80.6% for LLaMA. Text and latent communication differ by only 0.3 and 2.5 percentage points, respectively; question-clustered confidence intervals include zero and do not establish equivalence. The saving does not come from a cheaper individual latent step: latent steps and decoded tokens take similar time per unit (38.5 vs. 37.1 ms). Instead, the reduction is primarily associated with replacing hundreds of decoded intermediate tokens with 40 bounded latent steps. The single-agent comparison is model-dependent: for Qwen3-8B, latent communication is slightly more accurate, uses 16.6% less energy, and runs 1.18 faster than the matched single-agent baseline, whereas for LLaMA the single-agent baseline is both more accurate and lower-energy than either multi-agent mode. A second Qwen3 benchmark shows the same qualitative energy and accuracy pattern. These results quantify a substantial GPU-energy reduction from latent communication in the single-stream setting, while highlighting reduced transparency as an important practical limitation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Latent communication, multi-agent LLMs, energy efficiency, GPU energy measurement, KV-cache communication, sustainable AI, LLM inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Yq8GOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an empirical study on the GPU energy consumption associated with "latent-space communication" during the inference process of multi-agent large language models (LLMs). Using NVIDIA A100 GPUs and the Qwen3-8B and LLaMA 3.1-8B-Instruct models, the authors compare three approaches: single-agent inference, text-based multi-agent communication (TextMAS), and latent-space communication (LatentMAS).

Critical strengths
  1. Direct energy consumption measurement provides valuable empirical evidence that goes beyond existing efficiency metrics.

Previous research on "latent multi-agent communication" has primarily focused on reducing latency and the number of tokens. In this study, energy consumption at the GPU board level was measured directly under controlled hardware and inference configurations. The results demonstrate a significant and consistent reduction in energy consumption across two model families. Furthermore, the study incorporates single-agent baselines matched to the specific prompts, accounts for idle power consumption, and provides uncertainty estimates.

  1. The paper offers a clear analysis of the sources of energy savings while also highlighting significant limitations of the research.

Through a decomposition analysis of individual agents, the study reveals that the primary source of energy savings lies in the reduction of intermediate generation steps, rather than a decrease in the computation time required for a single step. This distinction helps explain the observed results. Additionally, the authors transparently report instances where latent communication underperforms.

Critical weaknesses
  1. Limited innovation and a lack of necessary control experiments to isolate the actual gains derived from "latent communication."

The paper primarily evaluates the existing LatentMAS framework without introducing any new communication mechanisms or reasoning algorithms. While measuring energy consumption directly is of some value, the observed energy savings are largely attributable to the fact that LatentMAS utilizes only 40 intermediate latent steps, whereas TextMAS generates hundreds of tokens at each intermediate agent stage. It remains unclear whether the efficiency gains stem from the inherent advantages of latent representations or simply from a drastic reduction in the generation budget. The paper fails to provide text-based baselines aligned for equivalent computational budgets, nor does it present "accuracy vs. energy consumption" trade-off curves across varying text-token and latent-step budgets.

  1. Limited generalization capabilities; the accuracy vs. efficienc trade-off remains unresolved.

The paper’s primary results are based on two 8B-parameter models, a single four-agent sequential topology, and an inference setting with a batch size of 1. The tasks evaluated already exhibit high baseline accuracy, which weakens the study’s arguments regarding more challenging reasoning scenarios.

More importantly, exploratory experiments on the GPQA-Diamond task reveal a significant drop in accuracy for the Qwen3-8B model (52.3% for TextMAS versus only 39.7% for LatentMAS). Although the authors speculate that output truncation may be a contributing factor to this decline, this hypothesis has not been empirically verified. Furthermore, the paper provides no data on energy consumption in scenarios involving batch processing.

Questions for authors
  1. How does LatentMAS perform compared to TextMAS when the generation budget is held constant or subjected to systematic variations?

  2. In the GPQA-Diamond task, to what extent is the decline in accuracy attributable to output truncation?

  3. How sensitive are the energy-saving benefits to factors such as batching and the memory overhead of the KV cache?

1 comment

Thank you for the detailed reading.

Constant and varied budgets (W1, Q1): global response items 1–3 cover a matched 40-step budget, two further text budgets, and the latent-step sweep m ∈ {0, 10, 20, 40, 80}.

GPQA truncation (Q2): item 8; verified directly in the stored generations, with the conditional accuracies and their selection bias stated.

Batching and KV-cache memory (Q3): the +14% peak-memory overhead is in the paper; batched energy is not measured (item 6), and the scope is now stated as single-stream in the title and abstract.

Novelty: we agree the communication method is prior work. The contribution we now claim is the controlled energy measurement with the budget confound removed, and the finding that the saving is a step-count effect that prompt-only KV reuse (m=0) already captures.

Reviewer_SgtAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper directly measures GPU energy for single-agent, text-based multi-agent (TextMAS), and latent multi-agent (LatentMAS) inference using LatentMAS’s sequential four-agent pipeline on a single A100 at batch size 1. On GSM8K, latent communication reduces net GPU energy by 70.7% (Qwen3-8B) and 80.6% (LLaMA-3.1-8B) relative to TextMAS, with accuracy differences whose confidence intervals include zero. The authors attribute the saving to replacing hundreds of decoded intermediate tokens with a fixed number of latent steps. Compared with a single agent, LatentMAS is better for Qwen3-8B but worse in both accuracy and energy for LLaMA.

Critical strengths
  • The protocol uses identical hardware, idle-power subtraction, a prompt-matched single-agent baseline, and correction for scorer asymmetry. It reports question-clustered confidence intervals without overclaiming accuracy equivalence.
  • Per-step latency is nearly equal across modes, per-agent times sum to per-instance totals, and a simple first-order model closely predicts the measured saving for Qwen3-8B.
Critical weaknesses
  • The authors show that mean GPU power is nearly identical across modes, so energy reduction tracks wall-clock time reduction almost exactly. At batch size 1, the result is therefore largely implied by LatentMAS’s previously reported speedups. Regimes where energy and time could diverge, such as batched serving, are not measured, and the cited batched results suggest the ordering may change there.
  • In TextMAS, each intermediate agent generates hundreds of tokens in Qwen3’s thinking mode, while latent agents are capped at 40 steps. The comparison therefore conflates communication channel with intermediate reasoning length. A TextMAS variant with capped intermediate outputs is needed to attribute savings to latent communication itself. For LLaMA, a single agent is both more accurate and cheaper than either multi-agent mode, which further limits the practical takeaway.
Questions for authors
  • What are the accuracy and energy of TextMAS with intermediate outputs capped to a budget comparable to the 40 latent steps, or with thinking mode disabled for intermediate agents?
  • Can you report at least one batched-serving operating point in energy per correct answer?
  • Please normalize Table 1 energy to the same number of instances for both models.
2 comments

Thank you for the concrete suggestions; each of them is now a measured condition.

Capped and no-think intermediates (W2, Q1): global response item 1 (both variants) and item 2 (uncapped no-think and a 160-token cap). At a matched 40-step budget the intermediate agents cost the same in both channels; the energy difference is downstream solver effort.

Energy tracking time (W1): agreed. Mean power is mode-invariant within a board, so energy follows time; the open question was whether the time saving survives budget matching, and it does (25–43% against budget-controlled text, 40–43% against the best concise text pipeline).

Batched operating point (Q2): item 6; not measured, and the single-stream scope is now explicit.

Table 1 (Q3): item 7; both models are now reported per 100 questions.

Practical takeaway for LLaMA: unchanged. Single-agent remains better there, and we keep that result as is.

Thank you for the additional experiments. W2 is resolved, but W1 and Q2 still remain as limitations. Therefore, I will maintain my rating.

Reviewer_icasOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper measures the GPU energy of single-agent inference, text-based multi-agent inference, and LatentMAS-style latent communication. It uses a sequential planner–critic–refiner–solver pipeline, two 8B models, and an A100 GPU at batch size one. On GSM8K, latent communication reduces idle-subtracted GPU-board energy by 70.7% for Qwen3-8B and 80.6% for LLaMA-3.1-8B relative to TextMAS. The analysis attributes most of the reduction to replacing hundreds of intermediate decoded tokens with 40 latent steps per intermediate agent, rather than reducing the cost per step. Prompt-matched single-agent comparisons show a model-dependent outcome, and additional experiments examine ARC-Easy, harder-task failures, and the limited interpretability of latent states.

Critical strengths
  1. The study measures energy directly and explains the observed reduction. The GPU-board measurements, idle subtraction, agent timing, and per-step comparison provide a useful empirical check on latency and token-count proxies. The finding that latent and text steps take similar time, while intermediate-agent duration falls sharply, is more informative than simply reporting a lower energy total. The reported savings are large relative to the stated instrumentation uncertainty.

  2. The paper is careful about several limitations that materially affect interpretation. It uses question-clustered accuracy intervals for the main text-versus-latent comparison and explicitly avoids equating a nonsignificant difference with accuracy equivalence. The prompt-matched single-agent baseline is valuable: for LLaMA it is both more accurate and lower-energy than either multi-agent mode. The paper also reports a harder-task regression and acknowledges that batching may change the efficiency ordering. These negative results improve the usefulness of the study.

Critical weaknesses
  1. The experiment does not establish how much of the advantage requires latent communication rather than a smaller intermediate reasoning budget. The main comparison changes the channel, the number of intermediate generation steps, and whether prior computation is reused through the KV cache. The paper itself finds nearly equal time per latent/text step, yet fixes the latent budget at 40 and does not compare against concise or budget-limited text agents, a zero-latent-step control, or a comparable text-prefix reuse implementation. The measured reduction against the chosen TextMAS configuration is credible, but the mechanism and the accuracy–energy trade-off are not sufficiently isolated. This is central to the contribution because the communication method itself comes from prior work.

  2. The additional energy evidence is confined to a regime where energy closely follows elapsed time. All measurements use batch-size-one Hugging Face inference, two 8B models, one GPU type, and one sequential topology. The main accuracy study uses only 100 Qwen3 questions and 200 LLaMA questions; the second Qwen3 benchmark is also small and near saturation. Meanwhile, Section 8 identifies extra latent-cache memory and cites a batching-dependent reversal, but provides no energy measurement in that regime. A focused empirical study need not cover every deployment, but measuring at least one setting where utilization or concurrency changes the time-to-energy relationship would make the added energy analysis substantially more consequential.

Questions for authors
  1. Mechanism and baselines: Can you provide an accuracy–energy curve over latent-step counts, including zero, and compare with text agents explicitly instructed to produce concise intermediate outputs under several budgets? Please also quantify the contribution of KV reuse separately from the shorter intermediate generations.
  2. Serving regime: Can you measure a small batch-size or concurrency sweep on the same GPU, reporting total and idle-subtracted energy per completed question, throughput, and peak memory for all three modes? Does the energy ordering change as the available memory is used more fully?
  3. Accuracy at comparable energy: Can you retain paired per-question outcomes for the single-agent baseline and evaluate a larger sample, so the proposed accuracy–energy advantage can be assessed against an optimized single-agent or concise-text alternative? The current descriptive 1.7-point Qwen3 improvement does not establish such an advantage statistically.
1 comment

Thank you for a review that identified the central gap precisely.

Mechanism and baselines (W1, Q1): global response items 1–3 give the accuracy–energy curve over m including m=0, concise and capped text intermediate agents at several budgets, and separate prompt-KV reuse (m=0) from shorter generations. The saving is a step-count effect; m=0 captures most of it; whether latent thoughts beyond m=0 add accuracy is unresolved at n=100, and we say so.

Serving regime (W2, Q2): item 6; we do not claim the single-stream result for batched serving, and the scope is now in the title.

Accuracy at comparable energy (Q3): item 4 provides the paired single-agent comparison you asked for. At the paper's solver budget it shows a narrow latent accuracy advantage; at a non-truncating budget the accuracies are indistinguishable and the energy (28% less) and speed (1.28×) advantages remain, which is what the revised paper claims.

Comments on the paper

3 comments
AuthorsOpenReview ↗

New budget-matched controls

All three reviews make the same central point: our comparison changes the communication channel and the intermediate generation budget together, so the measured saving could be a budget effect. We agree this was not isolated in the submission. We ran the requested controls during the discussion period (33 new runs) and report them in full, including the parts that qualify our claims.

New conditions. Qwen3-8B, GSM8K, the paper's 100 questions, 3 seeds, same hardware class and protocol (solver budget 2048 tokens unless stated). ΔAcc is relative to LatentMAS m=40 with question-clustered bootstrap CIs, paired by question and seed. "Trunc." = solver outputs out of 300 that reached the solver budget.

ConditionAcc. %ΔAcc vs m=40 [95% CI]Net Whs/questionSolver tokensTrunc.
TextMAS, full (paper)95.7−0.3 [−3.0, +2.3]196.794.96240
TextMAS, intermediates ≤40 tok., thinking on91.7−4.3 [−8.7, −0.3]76.741.390121
TextMAS, intermediates ≤40 tok., thinking off90.7−5.3 [−9.7, −1.7]79.041.490427
TextMAS, intermediates ≤160 tok., thinking off93.0−3.0 [−7.0, +0.3]95.849.082013
TextMAS, intermediates uncapped, thinking off90.0−6.0 [−10.7, −1.7]101.849.676217
LatentMAS m=0 (prompt KV reuse only)93.3−2.7 [−6.0, +0.3]52.528.267610
LatentMAS m=1093.7−2.3 [−5.0, 0.0]55.027.36818
LatentMAS m=2094.0−2.0 [−5.0, +0.7]56.026.965910
LatentMAS m=40 (paper)96.0—57.729.56627
LatentMAS m=8093.0−3.0 [−6.0, −0.3]72.934.068311
Single agent, prompt-matched (paired rerun)92.0−4.0 [−8.3, −0.3]72.734.391723
TextMAS, ≤40 tok., thinking on, solver budget 409695.7−0.3 [−3.3, +2.3]84.342.49794
Single agent, prompt-matched, solver budget 409696.3+0.3 [−2.0, +2.3]80.537.710245

1. Channel vs budget (icas W1, SgtA W2, Yq8G W1). At a matched budget of 40 generation steps per intermediate agent, the intermediate agents cost the same in both channels (1.6 s per text agent vs 1.5 s per latent agent; 37.1 ms per decoded token vs 38.5 ms per latent step). The difference is entirely downstream: after 40-token text plans the solver generates 36% more tokens (901–904 vs 662), so the pipeline uses 33–37% more energy than LatentMAS (76.7–79.0 vs 57.7 Wh; 41 vs 29.5 s per question). Thinking on or off for the intermediate agents makes no difference. The 40-token text outputs are cut mid-sentence (e.g. "- Determine the time"), and the solver effectively re-derives the plan. With the budget held fixed, the latent channel transmits more usable information per step than decoded text, measured by downstream solver effort. (The accuracy gap at this budget is a truncation effect; see item 5.)

2. Concise text baselines (icas Q1, SgtA Q1). Disabling thinking for the intermediate agents without a cap yields natural concise plans (105–266 tokens per agent) at 101.8 Wh; a 160-token cap gives 95.8 Wh. LatentMAS m=40 uses 40–43% less energy than the best concise text pipeline we found. The 70.7% saving in the paper is against the verbose default TextMAS configuration; against a budget-controlled text pipeline the saving is 25–43%. The revised paper reports both.

AuthorsOpenReview ↗

What the controls establish, and a truncation caveat

3. Latent-step sweep, including m=0 (icas Q1, Yq8G Q1). Energy and time rise with m (52.5 → 72.9 Wh). Accuracy is 93–94% for m ∈ {0, 10, 20}, 96.0% at m=40 and 93.0% at m=80; all CIs vs m=40 include or touch zero except m=80. The important qualification: m=0, where the intermediate agents contribute only their prompts' KV cache and no generated content, already captures most of the energy saving against TextMAS (73%). The 40 latent steps add 10% energy and +2.7 pp accuracy with a CI of [−0.3, +6.0]. We therefore cannot establish at n=100 that latent thoughts themselves improve accuracy over prompt-only KV reuse. What the data do establish is that (a) replacing decoded intermediate text with any fixed-cost latent prefix produces the saving, and (b) at a fixed small budget, latent steps are cheaper downstream than decoded tokens. The revised paper states the contribution this way.

4. Single-agent baseline, paired (icas Q3). Rerun with per-question outcomes retained: 92.0 ± 1.0% and 72.7 Wh (the submission's aggregate-only runs gave 94.3 ± 0.6% and 69.2 Wh; the 2.3-pp gap between two runs of the same configuration with the same seeds is itself a useful estimate of cross-session reproducibility at n=100, and we report both). Paired against LatentMAS m=40: +4.0 pp [+0.3, +8.3] for latent, 20.6% less energy, 1.16× faster. That accuracy advantage is, however, a solver-budget effect: with the single agent's solver budget doubled to 4096 tokens (3 seeds) it reaches 96.3 ± 0.6% (paired Δ vs latent −0.3 pp [−2.3, +2.0]) at 80.5 Wh and 37.7 s/question, i.e. latent is 28% lower-energy and 1.28× faster at indistinguishable accuracy. We agree the submission's descriptive 1.7-pp accuracy claim did not establish an advantage, and we no longer claim one; the energy and speed advantages over the matched single agent survive the budget change.

5. A caveat we found and state prominently. Across the nine new 2048-budget conditions, 137 of 206 wrong answers are solver outputs that exhausted the budget without a boxed answer (full TextMAS: 0 of 13; LatentMAS m=40: 6 of 12; 40-token text, thinking off: 27 of 28; single agent: 22 of 24). Conditional on an answer being emitted, accuracy is 95–100% in every condition. Accuracy differences between conditions at this budget therefore mostly measure solver verbosity, which is the same quantity that drives energy. We tested this directly by rerunning the two most truncated conditions with a 4096-token solver budget (3 seeds each, last two rows of the table): the 40-token text control rises from 91.7 to 95.7 ± 1.5% (truncations 21 → 4 of 300) and the single agent from 92.0 to 96.3 ± 0.6% (23 → 5), both indistinguishable from LatentMAS m=40, while their energy rises to 84.3 and 80.5 Wh, still 46% and 40% above latent (57.7 Wh). The accuracy differences at a fixed 2048-token solver budget are budget effects; the energy and time differences are not. Truncation counts now accompany every accuracy in the new table. (LatentMAS m=40 was not rerun at 4096; 7 of its 300 outputs reached the cap, so its accuracy could rise by at most about 2 pp.)

AuthorsOpenReview ↗

Scope, Table 1, GPQA, and the revised PDF

6. Batched serving (icas W2/Q2, SgtA W1/Q2, Yq8G Q3). Not measured, and we do not claim it. Our pipelines are single-stream HuggingFace loops with manual KV handling in latent mode; a batched latent path cannot be validated in the discussion period. The scope statement is now in the title and abstract ("single-stream inference"), the published batch-size-20 reversal from the LatentMAS paper stays as the explicit caveat, and batched measurement is listed as the primary open question.

7. Table 1 normalisation (SgtA Q3). Table 1 now reports net energy per 100 questions for both models (LLaMA's 200-question totals halved): Qwen3-8B single 69.2 / text 196.7 / latent 57.7 Wh; LLaMA single 18.9 / text 131.2 / latent 25.4 Wh. Per question: Qwen3-8B 0.69 / 1.97 / 0.58 Wh, LLaMA 0.19 / 1.31 / 0.25 Wh.

8. GPQA-Diamond truncation (Yq8G Q2). Verified in the stored generations: every latent solver output without a boxed answer stopped at exactly 4096 tokens (36, 38, 31 of 100 per seed; text 6, 6, 5). Restricted to questions where an answer was emitted, latent accuracy is 62.5 / 62.9 / 58.0% vs text 55.3 / 54.3 / 56.8%. This is selection-biased (truncated questions are plausibly harder), so we present it as confirming the mechanism, not as a corrected accuracy.

Measurement note. The new runs used two VMs of the same SKU; the second board reads 5–8 W higher at the same load (138–147 W vs 131–134 W), within the ±5% instrumentation uncertainty the paper states. Time per question is reported alongside energy and gives the same ordering.

Revised PDF (uploaded; changed text in blue). Title and abstract state the single-stream scope and the budget-controlled finding. New Section 7.4 "Isolating channel from budget" with Table 4 (the table above) and Figure 3 (accuracy vs energy and vs time for all conditions). Table 1 normalised to Wh per 100 questions. Contributions 2 and 3, Section 6, the limitations and the conclusion updated; the accuracy claim against the single agent is withdrawn and replaced by the energy and speed result at indistinguishable accuracy. Appendix A reports the GPQA truncation verification; Appendix D the two-board measurement note. The qualitative power-profile figure moved to Appendix D to stay within the page allowance.

AI disclosure. As stated in the paper, we used AI coding assistants to implement and run the experiments and to draft text; all numbers were checked by the authors against the stored runs, and the authors are responsible for the content.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.