acceptodds
Under review as a conference paper at ICLR 2027

InfoLat: Informative Latent Communication between Heterogeneous LLMs

Abstract

The collaboration of heterogeneous large language models (LLMs) is an appealing way to leverage their complementary strengths. Traditional approaches communicate through text, forcing each model to compress its information-rich internal states into discrete tokens. Alternatively, latent communication offers a promising collaboration path by letting models communicate directly in a high-dimensional latent space, without explicit text. However, some existing methods only support communication between models with the same backbone; others learn a communication module for a fixed sender–receiver pair, so replacing the receiver requires retraining from scratch. Moreover, some representative methods tend to collapse the latent message into a task-agnostic one. To facilitate message understanding of heterogeneous receivers, we first represent each latent message as a sequence of latent tokens, where each latent token consists of a set of vocabulary tokens and their continuous weights. Then, we propose \method, a latent communication framework that combines latent label construction, vocabulary alignment, and decoupled sender–receiver training, enabling a sender to broadcast informative latent messages to multiple heterogeneous receivers without retraining the sender. Experiments on math and ALFWorld benchmarks demonstrate that \method consistently outperforms existing communication methods, with superior performance and reduced communication cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Latent Communication, Heterogeneous Large Language Models, Cross-Model Collaboration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_W5ubOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes InfoLat, a latent communication framework for collaboration between heterogeneous LLMs. A message encoder and predictive decoder first convert reference text messages into compressed latent labels, represented as candidate vocabulary tokens and associated weights. The sender is then trained to generate these messages, while receivers are independently trained to use them for task completion. Vocabulary alignment enables communication across models with different vocabularies and embedding spaces, allowing a trained sender to remain fixed while new receivers are adapted. The authors evaluate the framework on mathematical reasoning benchmarks and ALFWorld, with ablations, message-shuffling experiments, and training-dynamics analyses examining its effectiveness and communication properties.

Critical strengths
  1. I find the communication interface and decoupled training procedure practically useful. Representing messages through vocabulary tokens and weights, together with vocabulary alignment, provides a concrete way to connect different models’ embedding spaces. In particular, adapting additional receivers without retraining the sender offers a useful path toward extending collaboration to more model combinations.

  2. I also appreciate the empirical results across mathematical reasoning and interactive environment tasks. In the evaluated model combinations, InfoLat achieves competitive task performance with shorter communication sequences. These results suggest that the approach is promising and worth further investigation.

Critical weaknesses
  1. My main reservation concerns the empirical validation of the benefits attributable to the proposed latent-label learning and compression mechanisms. Although the paper includes several useful ablations and analyses supporting the framework and individual components, I would appreciate more direct comparisons with simpler communication alternatives. For example, CIPHER also uses probability-weighted token embeddings, as discussed in the paper, but is not evaluated in compatible settings. The segment length directly determines the degree of compression, yet its effect is not examined. In addition, while the appendix explains the computational benefits expected from shorter sequences, these benefits are not supported by runtime measurements. These gaps do not negate the reported results, but leave me somewhat uncertain about the advantages of particular design choices and their performance–cost trade-offs.
Questions for authors
  1. The paper notes that CIPHER uses probability-weighted token embeddings and supports different models sharing a tokenizer. Could the authors explain why it was not included in the experimental comparison? A comparison in a homogeneous or shared-tokenizer setting would help clarify the additional benefits of InfoLat’s latent-label learning and compression mechanisms. If there are applicability or implementation constraints, clarification would also be helpful.
  2. What motivated the default segment length of 2? Could the authors provide results for segment lengths such as 1, 2, and 4, showing the trade-off between task performance and message length? I am particularly interested in the length-1 setting, which could help clarify the respective roles of the continuous representation and compression. The existing analysis of the interpolation width K does not directly address this question.
  3. I appreciate Appendix A.5’s analysis of the computational savings from fewer generation steps and receiver input positions, as well as its distinction between sequence length and transmitted bytes. Could the authors provide representative measurements of sender, receiver, and end-to-end inference time to show how these expected savings translate into practice? Reporting the actual message payload and distinguishing initial training costs from the cost of adapting an additional receiver would also help characterize the practical efficiency of the framework.
  4. The message-shuffling experiments and question–message compatibility analyses provide useful evidence that receivers use task-specific information. I would like to better understand how much a correctly matched message helps relative to receiving no message. Could the authors add no-message or fixed-message controls using the same trained receiver? For baselines that are relatively insensitive to message replacement, is there further evidence distinguishing insufficient information in the sender’s messages from the receiver’s failure to use that information?
8 comments

We sincerely thank the reviewer for the careful assessment and for recognizing the value of our existing ablations and analyses. We group related weaknesses and questions and respond to each point below.

1. Why is CIPHER, which also uses probability-weighted token embeddings, not included, and how does InfoLat compare with it in a shared-tokenizer setting that clarifies the additional benefit of latent-label learning? (W1, Q1)

CIPHER was not included in the original submission because it feeds the embedding-weighted output distribution of the sender directly into the receiver, which requires the sender and receiver to share the same tokenizer and embedding space. It therefore cannot be applied to our cross-family settings (Llama-3B→Qwen-1.5B and Qwen-7B→Llama-1B), and we only selected baselines that cover all settings. We agree with the reviewer that, in a shared-tokenizer setting, CIPHER is an appropriate control for clarifying the role of latent-label learning. We therefore add a comparison with CIPHER on Llama-3B→Llama-1B, together with a new latent communication baseline, C2C (Cache-to-Cache), which transfers information between models through the KV cache rather than a token sequence:

MethodGSM8K-augGSM-HardSVAMPMultiArithAvg
CIPHER48.67±0.5013.09±0.2956.67±0.5897.22±0.5653.91
C2C48.52±0.8012.05±0.4057.56±0.7797.04±0.3253.79
InfoLat68.21±0.1816.12±0.1069.89±0.5898.33±0.0063.14

For a fair comparison, all methods use the same sender and receiver backbones, the same test sets and the same decoding settings (temperature 0.6, top-p 0.95, 3 seeds). CIPHER is training-free, so we search its sender temperature on a 500-question GSM8K-aug validation set over {0.2, 0.4, 0.6, 0.8, 1.0, 1.5} and select the best value, 0.2 (accuracies 52.60 / 51.07 / 51.13 / 49.40 / 46.73 / 40.20, respectively). For C2C, we train the fuser on GSM8K-aug following its official configuration, with both the sender and the receiver frozen.

InfoLat reaches an average accuracy of 63.14, which is 9.2 and 9.4 points higher than CIPHER (53.91) and C2C (53.79). CIPHER uses the same type of message representation as InfoLat, namely probability-weighted combinations of vocabulary embeddings. The difference is that a CIPHER message is simply the sender's output distribution during normal decoding: it receives no latent-label supervision, and the receiver is not trained to read such messages. The improvement of InfoLat over CIPHER can therefore be attributed to latent-label learning and receiver training, not merely to the soft-token representation. C2C transmits the more information-rich KV cache, yet its accuracy is on par with CIPHER. This further suggests that the key to effective communication is not whether the transmitted representation is continuous, but whether the message is explicitly supervised to carry the reasoning content the receiver needs.

2. What motivates the default segment length of 2, and how does InfoLat perform with other lengths such as 1 and 4, especially length 1, which separates the effect of the continuous representation from that of compression? (W1, Q2)

Following the reviewer's suggestion, we add a sweep over the segment length ρ from 1 to 5. The experiment is run on Llama-3B→Qwen-1.5B; apart from ρ, latent label construction, sender training, receiver training and decoding settings are identical to the main experiments (3 seeds):

ρGSM8K-augGSM-HardSVAMPMultiArithAvg
168.03±0.1218.45±0.3174.67±0.3397.78±0.0064.73
2 (default)68.81±0.2517.54±0.2272.44±0.4497.78±0.0064.14
360.37±0.3116.50±0.2969.67±0.3398.70±0.3261.31
460.30±0.1616.58±0.3270.11±1.0299.44±0.0061.61
554.21±0.3915.72±0.3169.00±0.3399.44±0.0059.59

With ρ=1, each latent token corresponds to a single text token, so the message is not compressed at all, yet each latent token is still a continuous weighted combination of the embeddings of the top-K (K=10) vocabulary tokens. This setting therefore isolates the effect of the continuous latent representation itself. With ρ=1, InfoLat reaches an average accuracy of 64.73, 7.1 points higher than Text Communication (57.61), and outperforms Text Communication on all four datasets (GSM8K-aug 68.03 vs. 50.32, GSM-Hard 18.45 vs. 16.91, SVAMP 74.67 vs. 65.78, MultiArith 97.78 vs. 97.41). Since there is no compression here, this gain comes entirely from the latent message itself: compared with a text message that can carry only one discrete token per position, a latent token formed as a weighted combination of multiple vocabulary tokens and supervised by latent labels can carry richer and more useful information for the receiver.

We choose ρ=2 as the default because it offers the best balance between accuracy and message length. Compared with the uncompressed ρ=1, ρ=2 halves the message length (20.50 → 10.38) while lowering the average accuracy by only 0.6 points (64.73 → 64.14), and even slightly improves GSM8K-aug (68.03 → 68.81). A shorter message halves the number of autoregressive decoding steps of the sender, which directly reduces inference latency and compute. A larger ρ further shortens the message, at the cost of some accuracy, since each latent token has to summarize more text.

3. Can runtime measurements support the computational benefits discussed in the appendix, including sender, receiver and end-to-end inference times, the actual message payload, and the cost of initial training versus adapting an additional receiver? (W1, Q3)

(1) Inference time. We add stage-wise latency measurements on the full GSM8K-aug test set (1,319 questions), comparing InfoLat with Text Communication. The table reports per-question averages (in ms; p90 latency in parentheses, i.e., the time within which 90% of the questions are completed):

PairMethodSender stepsSender prefillSender decodeTransferReceiver prefillReceiver decodeEnd-to-endSpeedupPayload (B)
Llama-3B→Llama-1BText28.7720.5519.60.7112.3192.6745.7 (1058.1)1.00×29.8
InfoLat13.9020.0234.20.1912.0175.1441.4 (583.4)1.69×774.2
Llama-3B→Qwen-1.5BText28.7720.6517.20.9120.6410.5969.8 (1318.7)1.00×29.8
InfoLat13.9420.3236.40.2020.5386.9664.3 (851.7)1.46×776.4

Both methods are measured under identical conditions: one A800-40GB GPU with batch size 1, the same sender and receiver with merged LoRA weights, a shared KV-cache decoding loop with identical sampling settings and per-question seeds, and 10 warm-up questions. We synchronize the GPU at every stage boundary, so the stage times sum to the end-to-end time.

InfoLat reduces the end-to-end latency to 59% and 68% of Text Communication (1.69× and 1.46× speedups). The savings come mainly from sender decoding (about 520 → 235 ms): since each latent token carries more information, the sender needs about 13.9 decoding steps instead of 28.8, while the time per step is almost unchanged (18.2 vs. 18.7 ms), showing that constructing the top-K weighted embedding adds negligible overhead (consistent with Appendix A.5). Prefill times are nearly identical, as they depend only on the question length.

(2) Message payload. Fewer decoding steps do not mean fewer transmitted bytes. Each latent token carries 10 vocabulary IDs (int32) and 10 weights (bf16), i.e., 60 bytes, so an InfoLat message takes about 774 bytes, compared with about 30 bytes for a UTF-8 text message. Although about 26 times larger, it is still below 1 KB (about 6 μs at 1 Gbps), and the measured transfer stage takes only 0.2 ms, less than the 0.7–0.9 ms for text, which requires detokenization and re-tokenization. Since model computation rather than bandwidth is the bottleneck, this trade-off is worthwhile. We will add this table to the revision and distinguish decoding steps from transmitted bytes in Appendix A.5.

(3) Training cost. The latent labels and the sender of InfoLat depend only on the sender, so they can be reused directly when a new receiver is added, and only the new receiver needs to be trained. Taking Llama-3B as the sender and adding Llama-1B-Instruct as a new receiver, InfoLat requires 1.35×10¹⁸ FLOPs, about 3.1% of the one-time cost of latent label construction and sender training.

Reviewer_yJuaOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies latent communication in heterogeneous multi-agent systems. The paper points out that the messages are not used effectively in existing methods, and the applicability is limited because of shared backbone LLMs. To address these, the paper proposes InfoLat. It represents each message as a sequence of latent tokens, each being a weighted combination of vocabulary tokens. During training, it introduces an encoder-decoder to build latent labels in this form, and trains the sender and receiver independently on these labels. So one sender can serve multiple heterogeneous receivers. Experiments show that InfoLat outperforms existing communication methods on math reasoning and ALFWorld.

Critical strengths
  1. The paper offers a new perspective in latent communication. Existing latent communication methods show almost no performance drop when messages are shuffled. This indicates that their receivers do not rely on the content of the messages. This finding offers a valuable reference for this field of research.

  2. The paper is written clearly and is easy to follow.

  3. This method addresses a limitation of training-based latent communication methods: end-to-end training forces a tight coupling between sender and receiver. By utilizing latent labels, InfoLat decouples the training of two, allowing the sender to be reused with different receivers and thereby enhancing scalability.

Critical weaknesses

Overall, my main concern is where InfoLat’s gains actually come from. It is unclear how much of the improvement can be attributed to communication. Specifically:

  1. The receiver acts more like a message decoder, while the sender does most of the problem solving. The receiver sees the original question during both training and inference. However, InfoLat's accuracy after shuffling drops to 1.4%, far below the 38-42% accuracy achieved when the receiver answers independently. This might be that the receiver mainly reads the message rather than performing its own reasoning. This may go against the paper's motivation: leveraging the complementary strengths of heterogeneous models.

  2. InfoLat's messages are weighted combinations of top-K vocabulary tokens, which seem to keep less information than hidden states, yet InfoLat performs much better than the hidden-state variant. This suggests that the advantage of communication may come from how readable the messages are, not only from the richer information the paper emphasizes.

Questions for authors

The receivers in the experiments are weaker models, and some results are weaker than those of the sender alone.

  1. Have the authors analyzed whether the benefit of communication should be attributed to the latent labels or to the receiver's ability to interpret and answer? Adding a weak-sender, strong-receiver control may help determine more precisely whether the receiver actually contributes its own reasoning.
5 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address each concern below.

1. Does the large accuracy drop after shuffling indicate that the receiver mainly acts as a message decoder while the sender does most of the problem solving? (W1)

A shuffled message gives the receiver a complete line of reasoning that belongs to a different question. The more closely the receiver follows the message, the more likely this mismatched reasoning leads it to a wrong answer. Text Communication shows the same behavior: on Llama-3B→Llama-1B and GSM8K-aug, the receiver reads the sender's plain-text CoT, and its accuracy also drops from 47.21 to 3.18 after shuffling (Fig. 2). This drop is therefore expected for any receiver that has learned to rely on the message, and is not specific to latent messages.


2. Should the benefit of communication be attributed to the latent labels or to the receiver's ability to interpret the message and answer, and can a weak-sender, strong-receiver control clarify this? (W1, Q2)

We believe the benefit of communication depends on both the latent labels and the receiver's ability to interpret the message and answer, and neither alone is sufficient. The latent labels determine what the message carries: the messages of baselines collapse and carry almost no question-specific information, so the receiver cannot benefit from them even after training (Fig. 2). The receiver's ability determines whether this content can be used: the latent message does not contain the final answer, which the receiver has to work out from the reasoning in the message.

To examine more directly whether the receiver contributes its own reasoning, we add the weak-sender → strong-receiver control suggested by the reviewer: Llama-1B serves as the sender and Llama-3B as the receiver, with the same training procedure as the main experiments and only the roles of the two models swapped (3 seeds):

SettingGSM8K-augGSM-HardSVAMPMultiArithAvg
Llama-1B alone (weak sender)38.06±1.449.10±0.3453.44±1.1593.15±2.3848.44
Llama-1B → Llama-3B61.74±0.1815.85±0.2368.89±0.3899.07±0.6461.39
Llama-1B → Llama-3B, shuffled4.32±0.332.43±0.137.78±0.8417.04±2.107.89

In this setting, the system reaches an average accuracy of 61.39, 13 points higher than the weak sender's own 48.44, and GSM8K-aug improves from 38.06 to 61.74. The reasoning carried by the weak sender's message cannot exceed the sender's own ability, so the additional gain can only come from the reasoning contributed by the strong receiver after interpreting the message. Meanwhile, accuracy drops to 7.89 after shuffling, showing that the strong receiver still uses the information in the message rather than ignoring it and answering independently. This indicates that the receiver in InfoLat is not a passive message decoder: when the receiver is stronger, it further completes the reasoning on top of the sender's message.

3. All receivers in the experiments are weaker models, and some results fall below the sender alone. Does this conflict with the motivation of leveraging the complementary strengths of heterogeneous models? (W1, Q1)

"Complementary strengths" in our paper refers to the complementarity between the strong and weak models in capability and cost. The strong model reasons better, but each generated token is more expensive. The weak model is cheap, but its ability to solve complex tasks on its own is limited. Pairing a stronger model with a weaker one in this way is a common setting in prior work (Fu et al., 2026; Zou et al., 2026; Zhang et al., 2026). In InfoLat, the strong model only reads the question and produces a short latent message that carries the key reasoning content. The weak model then completes the remaining reasoning and generates the answer. The goal is to use communication to markedly improve the weak model's task performance, while reducing the overall compute compared with running the strong model alone. The table below examines both aspects on the four math datasets:

SettingAvg AccGFLOPs / questionRelative to Llama-3B
Llama-3B alone61.29197.11.00×
Llama-1B alone48.4476.40.39×
Qwen-1.5B alone55.31122.10.62×
InfoLat (Llama-3B→Llama-1B)63.14134.80.68×
InfoLat (Llama-3B→Qwen-1.5B)64.14168.30.85×

In terms of accuracy, communication substantially improves both weak models: Llama-1B rises from 48.44 to 63.14, and Qwen-1.5B from 55.31 to 64.14. In terms of compute, InfoLat completes the same tasks with less computation than the sender alone: the two InfoLat systems require only 68% and 85% of the inference FLOPs of Llama-3B alone, and use less compute than Llama-3B alone on every dataset. This is the complementary strength we refer to: communication with the strong model greatly improves the weak model, at a lower overall cost than relying on the strong model alone. The same holds on ALFWorld, where Qwen-7B serves as the sender (success rates averaged over the seen and unseen splits):

SettingAvg SuccessGFLOPs / taskRelative to Qwen-7B
Qwen-7B alone69.107157.71.00×
Llama-1B alone63.871214.50.17×
Qwen-1.5B alone63.381534.70.21×
InfoLat (Qwen-7B→Llama-1B)68.133349.20.47×
InfoLat (Qwen-7B→Qwen-1.5B)67.403004.90.42×

Communication improves the success rate of Llama-1B from 63.87 to 68.13 and that of Qwen-1.5B from 63.38 to 67.40, while the two InfoLat systems require only 47% and 42% of the inference FLOPs of Qwen-7B alone.

The No Communication rows establish the base reasoning performance of each model (Sec. 4.1) and are used to measure how much communication improves the receiver. Since this paper studies latent communication between heterogeneous models, our comparisons are with other communication methods: Text Communication, InterLat, RecursiveMAS, and the newly added CIPHER and C2C. InfoLat achieves the highest average accuracy among these communication methods in all settings.

4. Top-K vocabulary-weighted messages seem to keep less information than hidden states. Does their advantage over the hidden-state variant therefore come from readability rather than from richer information? (W2)

We partly agree with the reviewer. Readability is indeed the motivation behind InfoLat's message representation, but we do not see readability and information as opposed. For communication, what matters is not how much raw information a message contains, but how much of it the receiver can actually use. Hidden states contain more raw information, but if this information is encoded in a way private to the sender and cannot be read by the receiver, it is effectively absent from the communication.

The Hidden State variant in the paper is designed to examine exactly this point (Table 3, Appendix C and D.4). It uses the same sender (Llama-3B), receiver (Qwen-1.5B) and training data as InfoLat, and differs only in the message representation: each latent token is the sender's last-layer hidden state instead of a set of vocabulary tokens and weights.

Message representationGSM8K-augGSM-HardSVAMPMultiArithAvg
Hidden State42.43±0.6412.28±0.1660.00±0.4766.85±0.5245.39
InfoLat (top-K vocabulary weights)68.81±0.2517.54±0.2272.44±0.4497.78±0.0064.14

The Hidden State variant reduces the average accuracy from 64.14 to 45.39, a relative drop of 29.2%. The training dynamics further explain why (Fig. 5 and Fig. 9). During receiver training, InfoLat quickly stabilizes at 69.5% on GSM8K-aug, whereas the Hidden State variant peaks at 44.8% at step 225 and does not improve further; its receiver loss keeps decreasing but does not translate into higher accuracy. During sender training, the KL loss of InfoLat keeps decreasing, while the regression loss on hidden states flattens soon after an initial drop. We attribute this to the fact that hidden states lie in the sender's private representation space, whose dimension (3072 vs. 1536) and scale (RMS of about 1.47 vs. 0.026) differ from the receiver's input embeddings. The receiver can only read them indirectly through a projector learned for a specific model pair, so although they carry more raw information, little of it can actually be used by the receiver. In contrast, InfoLat messages lie directly in the receiver's own vocabulary embedding space, and the receiver constructs its inputs with existing embeddings. This makes both the sender and the receiver easier to train and the information in the message easier to use.

Reviewer_adECOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

InfoLat lets one sender broadcast latent messages to heterogeneous receivers. A message is a sequence of soft tokens, each the top-10 vocabulary tokens with weights. The sender is trained against latent labels built by compressing reference CoT with ⟨c⟩ tokens and a predictive decoder. Vocabulary the receiver lacks is mapped through the receiver's subword embeddings. Sender and receivers are trained separately, so adding a receiver does not require retraining the sender. On math (Llama-3.2-3B -> Llama-1B / Qwen-1.5B) and ALFWorld (Qwen-7B -> small receivers), it beats Text, InterLat and RecursiveMAS with shorter messages.

Critical strengths

The shuffle test and compatibility heatmaps are a simple and useful check of whether communication actually happens. For example, the reproduced InterLat gets 47.11 vs 47.06 with and without shuffling. Representing messages as token IDs plus weights also makes them independent of the backbone.

Critical weaknesses
  1. The two-model system doesn't meaningfully beat the sender alone. On math, the averages are 63.14/64.14 vs 61.29 for the 3B sender by itself; the gain is concentrated on in-distribution GSM8K-aug, and GSM-Hard is lower (16.12/17.54 vs 18.40). On ALFWorld, both averages (67.46/68.10) are below Qwen-7B alone (69.06). After shuffling, accuracy falls to about 1.4%, far below the receivers on their own (48–55%), so the receiver is mostly transcribing the sender's compressed CoT. The gain more likely comes from 30 epochs of latent-CoT training on the sender than from communication, and the "complementary strengths" motivation is never tested.

  2. The communication baselines score below no communication. On ALFWorld seen, Text/InterLat/RecursiveMAS get 60.71/60.71/61.43 vs 64.29 for Qwen-1.5B alone. Text Comm reaches 47.21 on GSM8K vs 56.31 for the sender, even though the receiver sees the sender's full CoT. The baselines get 3 epochs vs 30–50 for InfoLat, and their numbers are not checked against the original papers. So both the "collapse" finding and the relative gains may be reproduction artifacts. Separately, Sec. 3.2 and 3.4 closely follow Latent-SFT (Deng et al., 2025), which is cited only in passing.

Questions for authors
  1. How accurate is the InfoLat-trained sender when it decodes answers on its own?
  2. Do InterLat and RecursiveMAS reach their reported numbers under the original settings or a matched training budget?
  3. What exactly differs from Latent-SFT in label construction and training?
  4. Why is the ALFWorld baseline length K=64 when Text Comm's #L is about 24?
  5. Can CIPHER and C2C be added, at least in the Llama→Llama setting?
9 comments

We sincerely thank the reviewer for the careful reading and the detailed, constructive comments. We address each concern below.

1. Does the two-model system bring a clear benefit over the sender alone, and how is the "complementary strengths" motivation reflected in the experiments? (W1)

"Complementary strengths" in our paper refers to the complementarity between the strong and weak models in capability and cost. The strong model reasons better, but each generated token is more expensive. The weak model is cheap, but its ability to solve complex tasks on its own is limited. Pairing a stronger model with a weaker one in this way is a common setting in prior work (Fu et al., 2026; Zou et al., 2026; Zhang et al., 2026). In InfoLat, the strong model only reads the question and produces a short latent message that carries the key reasoning content. The weak model then completes the remaining reasoning and generates the answer. The goal is to use communication to markedly improve the weak model's task performance, while reducing the overall compute compared with running the strong model alone. The table below examines both aspects on the four math datasets:

SettingAvg AccGFLOPs / questionRelative to Llama-3B
Llama-3B alone61.29197.11.00×
Llama-1B alone48.4476.40.39×
Qwen-1.5B alone55.31122.10.62×
InfoLat (Llama-3B→Llama-1B)63.14134.80.68×
InfoLat (Llama-3B→Qwen-1.5B)64.14168.30.85×

In terms of accuracy, communication substantially improves both weak models: Llama-1B rises from 48.44 to 63.14, and Qwen-1.5B from 55.31 to 64.14. In terms of compute, InfoLat completes the same tasks with less computation than the sender alone: the two InfoLat systems require only 68% and 85% of the inference FLOPs of Llama-3B alone, and use less compute than Llama-3B alone on every dataset. This is the complementary strength we refer to: communication with the strong model greatly improves the weak model, at a lower overall cost than relying on the strong model alone. The same holds on ALFWorld, where Qwen-7B serves as the sender (success rates averaged over the seen and unseen splits):

SettingAvg SuccessGFLOPs / taskRelative to Qwen-7B
Qwen-7B alone69.107157.71.00×
Llama-1B alone63.871214.50.17×
Qwen-1.5B alone63.381534.70.21×
InfoLat (Qwen-7B→Llama-1B)68.133349.20.47×
InfoLat (Qwen-7B→Qwen-1.5B)67.403004.90.42×

Communication improves the success rate of Llama-1B from 63.87 to 68.13 and that of Qwen-1.5B from 63.38 to 67.40, while the two InfoLat systems require only 47% and 42% of the inference FLOPs of Qwen-7B alone.

The No Communication rows establish the base reasoning performance of each model (Sec. 4.1) and are used to measure how much communication improves the receiver. Since this paper studies latent communication between heterogeneous models, our comparisons are with other communication methods: Text Communication, InterLat, RecursiveMAS, and the newly added CIPHER and C2C. InfoLat achieves the highest average accuracy among these communication methods in all settings.

2. How accurate is the InfoLat-trained sender when it decodes answers on its own? (Q1)

To answer this, we let the InfoLat-trained Llama-3B sender also act as the receiver, reading its own latent message to generate the answer. To keep the results comparable with the other receivers, the 3B receiver is trained with exactly the same receiver training procedure as Llama-1B and Qwen-1.5B, and the sender, latent messages and decoding settings are also identical (3 seeds):

SettingGSM8K-augGSM-HardSVAMPMultiArithAvg
Llama-3B alone56.31±0.7718.40±0.3072.67±1.5197.78±0.6361.29
Llama-3B → Llama-3B (self)69.27±0.0416.88±0.1670.56±0.1997.22±0.0063.48
Llama-3B → Llama-1B68.21±0.1816.12±0.1069.89±0.5898.33±0.0063.14
Llama-3B → Qwen-1.5B68.81±0.2517.54±0.2272.44±0.4497.78±0.0064.14

With the same messages, the two weak receivers achieve 63.14 and 64.14, comparable to the sender's own decoding (63.48) and within 2 points on every dataset. This reflects the complementary strengths above: InfoLat does not aim to make the sender stronger, but to let a smaller and cheaper receiver match the sender's own decoding, at only 68% (Llama-1B) and 85% (Qwen-1.5B) of the compute of Llama-3B answering alone.

3. Does the large accuracy drop after shuffling indicate that the receiver mostly transcribes the sender's compressed CoT? (W1)

A shuffled message gives the receiver a complete line of reasoning that belongs to a different question. The more closely the receiver follows the message, the more likely this mismatched reasoning leads it to a wrong answer. Text Communication shows the same behavior: on Llama-3B→Llama-1B and GSM8K-aug, the receiver reads the sender's plain-text CoT, and its accuracy also drops from 47.21 to 3.18 after shuffling (Fig. 2). This drop is therefore expected for any receiver that has learned to rely on the message, and is not specific to latent messages.


4. Why do some communication baselines score below No Communication, and why does Text Communication stay below the sender alone even though the receiver sees the sender's full CoT? (W2)

The sender in the No Communication rows is not a method we compare against. Our goal is to make the weak receiver stronger after it receives the message, and the sender's own performance serves to some extent as a reference for the upper level of the system. Text Communication still improves clearly over the receiver alone (38.06 → 47.21), showing that the reasoning in the message is useful. Moreover, the self-decoding experiment above shows that when both models are strong (Llama-3B → Llama-3B), their collaboration exceeds the sender alone (63.48 vs. 61.29).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.