acceptodds
Under review as a conference paper at ICLR 2027

LLM-SemanticLAN: Anchored Public Latent Communication via KV Caches across Heterogeneous Large Language Models

Abstract

Latent communication is the key technique to coordinate heterogeneous large language models for composite tasks, which directly exchanges information between internal states such as KV caches instead of texts to avoid slow sequential decoding and redundant prefilling. However, existing latent communication either maps caches between one pair of models at a time or relies on a shared space defined only by adapters trained together, leaving no fixed coordinates that a new model of another scale, family, architecture, or modality could align to. To address these issues, we formalize Public Latent Communication (PLC), which links every model via a shared latent space like a local area network (LAN). We realize PLC in LLM-SemanticLAN, a lightweight and high-efficiency framework that anchors the coordinates of this space in closed form by ridge regression before training, using targets from a frozen reference model. LLM-SemanticLAN employs model-specific codecs to map native KV caches of LLMs into the compact shared representations of this space for transfer and reconstruction, with codecs of only 1.0%-3.6% of the base model parameters and latent representations 64× smaller on average than the native KV caches at the same BF16 precision. Extensive evaluations across scales, attention architectures, families, and modalities verify its strong generalization, showing that aligning a new model with only a single existing member in the shared latent space, supervised only by the receiver's behaviour, enables zero-shot communication across all unpaired members. Without benchmark-specific training, LLM-SemanticLAN achieves near-lossless parity with text-to-text communication and superior vision-to-text performance, while delivering 12-35× speedups in end-to-end latency and 2.2-4.3× FLOPs reductions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent systems, inter-model communication, latent communication, heterogeneous models, KV Cache, KV Cache transfer, cross-modal communication, efficient inference, alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_389sOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LLM-SemanticLAN, a framework for cross-model KV cache communication through a shared latent space. Using anchored initialization and model-specific codecs, new models can communicate with existing members after training against only one partner. Experiments evaluate communication quality and efficiency across heterogeneous models and modalities.

Critical strengths

The proposed multi-model communication framework addresses a practically meaningful problem. Single-partner admission and interface reuse could reduce the cost of integrating new models into heterogeneous systems.

Critical weaknesses

W1. Insufficient comparisons with relevant baselines.

Cross-model KV cache transfer and shared latent communication have already been explored in prior work. However, the paper primarily compares against T2T, with only a limited pairwise ridge-regression baseline. Without comparisons to competitive KV transfer and shared-space methods, it is difficult to assess the actual technical contribution.

W2. The practical benefits of the proposed framework remain unclear.

The main claimed advantage is that new models can join a shared space without training pairwise translators. However, the paper does not adequately quantify the offline training and admission costs or compare online efficiency and communication quality against existing approaches. Thus, the practical benefits of the proposed design remain insufficiently demonstrated.

W3. Inconsistency between the described method and released implementation.

Equation (3) describes only receiver-behavior-based losses, whereas the supplementary code introduces additional latent consistency and mean-matching losses (admission/medical/engine.py and admission/hybrid/engine.py). These undisclosed objectives may affect the reported performance and make it difficult to attribute the gains specifically to anchored initialization.

Questions for authors

Q1. What are the main technical advantages over existing KV transfer and shared latent-space methods? Can the authors provide fair comparisons with competitive baselines?

Q2. How do offline training/admission costs and online latency, computational overhead, and communication quality compare with existing methods as the number of models increases?

Q3. Were the additional latent alignment losses in the released code used for the reported experiments? If so, why are they absent from Equation (3), and how does performance change without them?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 389s (1/4): consistency term (W3, Q3)

Thank you for reading the released code closely and pointing out that Eq. 3 omits the consistency term. We answer W3 first.

W3 / Q3. Consistency term. Yes, training uses this term, and the term was left out of Eq. 3 in error. We apologize for the omission. Eq. 3 followed an earlier form of the objective, which did not have the term. The term was added later, and because it does not change accuracy we treated it as a training detail and did not carry it into the equation. That was the wrong call. The omission is in the written equation only. The term asks two members to write the same latent for the same passage. During the joint training of the initial models, whenever two members encode the same passage, the term adds two losses, each with weight 4.35. The first is one minus the cosine between the latents of the two members at each position, after the mean latent of each member over the batch is subtracted. The second is the squared distance between the two mean latents. The term is not limited to the initial models. When a text model is added, the same term, with the same weight, pulls the latents of the new model toward the latents that the frozen partner writes for the same passage. The term thus carries out the alignment with one member that §3.5 describes. One rule decides where the term is used. It compares the latents that two members write for the same input, so it is used wherever a second member can encode that input, with the same weight. This covers the initial models and every added text model, and every text result in Table 1 was trained with it. The visual encoder of §4.3 is the only exception. No text member can encode an image, so there is no second latent of the same input, and the visual encoder was trained without the term (both weights are 0 in the launch scripts of the g3 runs). In the revised PDF, the term is Eq. 3b in §3.4, and Appendix D gives its weight.

The reported accuracy does not depend on the consistency term. The table below gives the ablation with Qwen3-4B↔Qwen3-14B held out from training. With our limited computational resources, each configuration trains for 2,000 steps instead of the 16,000 steps of the main checkpoint, and an LLM judge scores the answers. The last column gives the change in points when the term is removed, and marks the changes that are statistically significant. A change is significant when the 95% confidence interval of the paired difference, from a bootstrap over questions, excludes 0 (intervals in Appendix K). Exact values are the questions whose answer is a value of at least six digits copied character for character from the passage, such as an amount, an account or tracking number, or a full date (Table 5 in Appendix C, question types in Appendix G).

With termWithout termChange
Trained directions, Web77.880.1+2.3, significant
Untrained directions, Web74.777.2+2.5, significant
Untrained directions, Synthetic84.583.9−0.6, not significant
Untrained directions, Synthetic, exact values only69.465.6−3.9, significant
Cosine between members' latents0.9540.681

Without the term, exact values on untrained Synthetic directions drop by 3.9 points, so the term mainly protects exact values. With the weight halved to 2.2, every average accuracy over trained and untrained directions changes by at most 1.6 points. The random-weight start in Figure 5 keeps the term at full weight and still answers only 0.8% of Web questions, against 85.3% from the closed-form start. Without the term, the cosine between the latents of two members falls from 0.95 to 0.68, yet untrained directions keep their Web and overall Synthetic accuracy. Communication on untrained directions therefore does not depend on the term.

We also removed the term when a model is added. We added II-Medical-8B to the main-table latent space twice, against Qwen3-8B alone, from the same closed-form start and with the same data order, for 2,000 steps. One run keeps the term at weight 4.35 and the other sets it to 0. The table gives judge accuracy in percent for the four directions between II-Medical-8B and the two members it never trained with, on Web (1,345 answerable questions per direction).

Direction, never trainedWith termWithout termChange
II-Medical-8B→Qwen3-4B83.783.2−0.5
Qwen3-4B→II-Medical-8B81.980.7−1.3
II-Medical-8B→Qwen3-14B82.583.0+0.6
Qwen3-14B→II-Medical-8B80.680.8+0.2
Four directions together82.281.9−0.2

On Synthetic the same four directions give 95.4 and 95.3. None of these changes is statistically significant (same test, intervals in Appendix K), so removing the term at admission makes no measurable difference. One limit applies. II-Medical-8B is a fine-tune of Qwen3-8B, so this is the easiest case for alignment. In the revised PDF, the new Appendix K reports both ablations.

AuthorsOpenReview ↗

Response to Reviewer 389s (2/4): comparison with prior methods (W1, Q1)

W1 / Q1. Comparison with prior methods. Against pairwise methods, LLM-SemanticLAN serves directions never trained. Against the jointly trained space of Dery et al., LLM-SemanticLAN differs in how the space is trained. Our latent space starts from a closed form onto a frozen reference and trains on the behaviour of the receiver. Dery et al. use no reference model and train their adapters jointly on continuation. The ablation of the paper removes each of our two choices in turn, on Web over the six directions among the initial models (Figure 5, Table 5 in Appendix C). Without the closed-form start onto the reference, with randomly initialized codecs and everything else unchanged, the codecs answer 0.8% of the questions that the closed-form start answers at 85.3%. This control is why the latent space is anchored on a reference. With continuation alone, accuracy falls to 74.5%, and the gap of ten points does not close with more training. The ablation runs under our data, codecs and budget, so it carries no difference in data or model size, and it does not reproduce the system of Dery et al.

Adding a model is also cheaper. In LLM-SemanticLAN, a new model trains only its own codec, 0.8–3.6% of the size of the new model, against one frozen member. In Dery et al., a new model trains its own adapters, about a quarter of the size of the new model, against two partners. The latent of LLM-SemanticLAN is also 64× smaller than a native cache on average (Table 7 in Appendix D).

Our setting follows how a multi-agent system divides work. One model reads a document, a tool output or an image and hands what it read to a model that has not read it, and any model may hand off to any other. Our setting is also the economical one. The sender prefills the input once, and each receiver prefills only its question, so a passage sent to multiple receivers is prefilled only once. C2C addresses a different task, in which both models read the same input and the sender's cache is fused into the receiver's own cache of that input to improve the receiver's answer. C2C therefore cannot run when the receiver has not read the input, and a passage sent to multiple receivers is prefilled multiple times. The other pairwise methods (Heo et al., Qu et al., XKV) train one module per ordered pair, so no pair works before that pair is trained. T2T is the main reference because written messages are the common way models communicate and collaborate today and work between any two models. Two existing designs also run in our setting, and the paper compares against both.

  • Pairwise closed-form map (Heo et al., 2026). On Web, Qwen3-14B→4B reaches 79.2 against 87.5 for LLM-SemanticLAN, which is 8.3 points ahead, and Qwen3-4B→14B reaches 82.8 against 84.2, a gap of 1.4 points that is not statistically significant. The gap is on exact values, answers such as amounts, account numbers or full dates copied character for character from the passage (Table 5 in Appendix C), with 48.9 vs 77.1 and 45.8 vs 69.5. Between Qwen3-8B and Qwen3-4B, two models with the same number of layers, the map is ahead by 2.8 points, 90.0 against 87.2. The map stores 2.7B parameters for that one direction, against 143M for a codec. Our next comment compares a map of the size of a codec. The map is undefined across tokenizers and architectures, and every direction needs its own fitted map, so no direction works without fitting. In the revised PDF, Appendix D.2 reports all three directions.
  • Jointly trained shared space (Dery et al., 2026). Beyond the ablation above, Dery et al. add a new model against two partners. In our setting, a second partner helps only the directions between the new model and that partner. Adding Llama-3.1-8B against Qwen3-8B alone, or against both Qwen3-8B and Qwen3-4B, gives results within 1.4 points on the never-trained directions between Llama-3.1-8B and Qwen3-14B. In the revised PDF, §4.5 states how these ablations relate to the system of Dery et al.
AuthorsOpenReview ↗

Response to Reviewer 389s (3/4): a pairwise map of the size of a codec (W1, Q1)

W1 / Q1, continued. A pairwise map of the size of a codec. We add the pairwise closed-form map (Heo et al., 2026) at the size of one codec. The recipe is unchanged, and we only limit how many sender layers each receiver layer reads. We set this number to 2 because it is the setting whose parameter count equals that of a codec, 151M per direction (168M into Qwen3-14B) against 143M to 151M. The 2 layers are chosen for each receiver layer by the single-layer ranking of the recipe itself. The map has a closed-form solution, so it involves no training steps. Each map is fit for its own direction on 450 FineWeb-Edu passages that do not overlap the evaluation set. We re-ran the main-table checkpoint on the same 1,345 answerable Web questions and scored both systems in one judge run, so every question is answered by both systems and the comparison is made question by question. The re-run values differ from those of the paper by −0.5 to +1.6 points. All three directions are directions our codecs were trained on.

DirectionOursMap of codec size
Qwen3-4B→Qwen3-8B86.890.9
Qwen3-8B→Qwen3-4B86.775.5
Qwen3-8B→Qwen3-14B88.038.3

The table gives judge accuracy in percent, the share of questions answered correctly, and not the percent of T2T used in Table 1. Bold marks the higher score. All three gaps are statistically significant (paired bootstrap, 95% level, intervals in Appendix L). On the questions whose answer is a long number copied character for character from the passage, such as an amount or an account number, the scores are 85.5 for ours against 90.8 for the map, 79.4 against 55.7, and 77.1 against 2.3.

Between Qwen3-4B and Qwen3-8B the map of codec size is 4.1 points ahead of us in one direction and 11.2 points behind in the other. Into Qwen3-14B it no longer works, with 38.3 overall and 2.3 on long numbers, while the codec is unaffected.

The architectures explain the difference. Qwen3-4B and Qwen3-8B have the same depth and head layout (36 layers, 32 attention heads, 8 key-value heads of 128 dimensions) and differ in width (2,560 against 4,096). For each of their 36 layers, the layer of the other model that predicts its cache best is the layer with the same index, so a map that reads 2 layers still finds most of what it needs. Qwen3-14B has 40 layers, 40 attention heads and width 5,120. Its 40 layers do not line up one to one with the 36 of Qwen3-8B. Only 10 of the 40 are best predicted by a sender layer at the same or an adjacent index, and the best single sender layer explains 56% of the cache variance on average, against 66 to 68% between the two 36-layer models. The content of one receiver layer is spread over several sender layers, which a map of codec size cannot collect. A codec does not rely on any correspondence between layers, because every model writes into and reads from the same latent. The reference is what puts the three models in the same coordinates. Right after the closed-form start, before any training, the cosine between the latents of two models is 0.86 for the two 36-layer models and 0.84 for each pair that includes Qwen3-14B.

A map serves one ordered pair and is fit on paired caches of that pair. A codec is trained once per model, serves every direction including directions never trained, and crosses tokenizers and architectures. In the revised PDF, Table 2a in §4.2 and the new Appendix L report these numbers.

Reviewer_LL87OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes public latent communication through per-model codecs that map heterogeneous LLM KV caches into a shared latent sequence and reconstruct a receiver-specific prefix cache. Founding codecs are initialized by ridge regression onto PCA features of a frozen reference model, then trained using receiver KL, continuation, and answer losses. A new model aligns with only one existing member. Experiments cover scale, model family, hybrid attention, and vision-to-text transfer, reporting unpaired-route generalization, quality and cost relative to generated textual notes, and selected failure cases.

Critical strengths
  1. Single-partner admission and evaluation on untrained routes are practically valuable. Existing members remain frozen while a new codec is adapted, followed by tests on directions never jointly trained. This is a more informative scalability test than demonstrating one model pair. Table 1 and Appendix A cover multiple architectures and public QA datasets, with mismatched-passage controls testing whether the receiver uses transferred information.

  2. The implementation and diagnostics are broad. The paper addresses RoPE, tokenizer correspondence, recurrent-state initialization, and visual senders, and provides loss ablations, absolute scores, and phase-level timings. The supplementary archive includes corresponding code. Appendices B and J also expose limitations on long numbers, medical questions, and some unpaired routes, providing useful evidence beyond aggregate relative scores.

Critical weaknesses
  1. The ablations do not isolate external public-coordinate anchoring, the central design claim. Comparing random and ridge initialization simultaneously changes cache scale, available information, and the optimization starting point. The 0.8% versus 85.3% result establishes superiority over the tested random start, but does not identify the external anchor as the reason single-partner admission succeeds. The reference constraint is removed after initialization, and new members align only to an existing member’s outputs. A matched-budget shared-codec control with a sensible initialization but no external reference anchor, evaluated on the same unpaired routes, is needed. The pairwise linear-map control does not resolve this question.

  2. The broad efficiency and near-lossless characterizations exceed what the current comparisons establish. Appendix H times one median-length passage and one question per text dataset; in Table 11, an 820-token passage produces an 806–892-token note, making note generation the dominant source of speedup. Appendix A already reports the accuracy of directly reading the source passage, but not its matched cost, so a general 12–35× advantage over text transfer is not established. Generalization is also uneven: Table 3 reports 58.4 versus 62.4 on COCO for unpaired Qwen3-8B, contradicting Section 4.3’s claim of superiority for all three receivers; unpaired receivers retain only about 88–89% of T2T performance on the external visual benchmarks.

Questions for authors
  1. With training data, codec capacity, objective, and budget fixed, how does a shared space without the external reference anchor but with a well-scaled initialization perform on single-partner admission and unpaired routes? Can the initialization ablation include post-admission results, not only founding members?

  2. Can you add latency, FLOPs, and transmitted bytes for the existing direct-passage reference under matched hardware and answer lengths, aggregated over multiple passage lengths and natural answer lengths? Please distinguish generated-note communication from forwarding the available source text.

  3. Please reconcile the COCO Qwen3-8B discrepancy between Section 4.3 and Tables 3/14, and specify which receivers, tasks, and trained/untrained routes support the near-lossless and visual-superiority claims.

4 comments
AuthorsOpenReview ↗

Response to Reviewer LL87 (1/2): the role of the external reference (W1, Q1)

Thank you.

W1 / Q1. Isolating the external reference. By design, the reference anchors only the initial members. After initialization, the latent space that the initial members write becomes the anchor for every new model, which aligns its codec to one partner. A new model therefore reaches untrained directions only if the coordinates of its partner are already shared by every member. Four results support this, in order.

1. After initialization the models already share coordinates. Right after the closed-form start onto the reference, with no training, the cosine between the latents that two models write for the same passage is 0.86 for Qwen3-4B and Qwen3-8B and 0.84 for each pair that includes Qwen3-14B. When each model starts instead from the principal directions of its own cache, a well-scaled start with no reference, the same cosine is 0.00 to 0.01. The alignment holds across architectures. Qwen3-14B has 40 layers and a different head layout, against 36 layers for the other two, so its layers do not line up with theirs, and a direct map of codec size from Qwen3-8B into Qwen3-14B reaches 38.3% accuracy on Web where it reaches 75.5% to 90.9% between the two 36-layer models (see our response to Reviewer 389s). The start also decides whether training works. With randomly initialized codecs and the rest of the recipe unchanged, including the consistency term, the six directions among the initial models answer 0.8% of Web questions after 8,000 steps, against 85.3% from the closed-form start (Figure 5, Table 5 in Appendix C).

2. Coordinates without a common target stay private. This control and the next come from an earlier configuration (reference Qwen3-0.6B, earlier question sets, no consistency term), in one latent space of four segments, each of 1,024 dimensions, the latent width of the paper. Only the first segment has closed-form targets from the reference. The other three segments start from the principal directions of each member's own cache and train jointly under the same objective and budget, with Qwen3-4B↔14B held out. After training, cross-member cosine is 0.904 on the anchored segment and 0.059–0.066 on the other three.

3. Private coordinates add nothing to communication. Zeroing the three segments at inference does not lower accuracy. On the trained direction Qwen3-8B→4B, accuracy rises from 67.2 to 69.3 on Web and from 87.2 to 88.4 on Synthetic. On the untrained direction Qwen3-14B→4B, accuracy changes by +0.4 and +0.1.

4. A model added against a single partner generalizes. Llama-3.1-8B, with its own tokenizer, was trained only against Qwen3-8B and reaches 94.3% of T2T on its four untrained directions (Table 1). For this response we also added Gemma-2-2B, a third family, against Qwen3-8B alone for 1,000 steps. Its four never-trained directions with Qwen3-4B and Qwen3-14B reach 63.6% to 71.5% accuracy on Web, against 69.4% and 69.9% on the trained pair (new Appendix O). In the earlier configuration, Llama-3.1-8B also reaches untrained directions better when every coordinate has a reference target. Judge accuracy (Synthetic / Web) is 52.3 / 57.8 against 33.6 / 51.6 in the four-segment space from Qwen3-14B, and 47.5 / 50.8 against 17.1 / 36.0 from Qwen3-4B. On the trained direction from Qwen3-8B the four-segment space is higher (79.6 / 69.1 against 69.4 / 66.4), and the two admissions also differ in recipe, so this last comparison is indicative.

These results establish that the external reference gives the models aligned coordinates before training, and that these coordinates remain useful for communication along untrained directions. The controls of steps 2 and 3 predate the consistency term, which could by itself pull such coordinates together, so they support the mechanism and do not replace a matched run. We agree that a training run with no reference, matched in width and budget, would help determine whether an external reference is necessary relative to other well-initialized alternatives. The paper does not include such a run. Showing necessity is not required for our contribution. We propose and validate a concrete mechanism that is sufficient to establish shared coordinates and to let a model join through a single partner. We have restricted our claims accordingly. The revised PDF states that the reference is sufficient and does not state that it is necessary, and other ways to construct the coordinates are a direction for future work. In the revised PDF, Appendix D.3 adds the controls above.

AuthorsOpenReview ↗

Response to Reviewer LL87 (2/2): cost against reading the passage, COCO (W2, Q2, Q3)

W2 / Q2. Cost against reading the passage. We measured the condition in which the receiver reads the passage, with the protocol of Appendix H on an A100 with 24 answer tokens, at four passage lengths. The two text routes place their cost differently. Generated-note communication pays at the sender, which decodes the note token by token. Forwarding the source text pays at every receiver. Each receiver prefills the full passage, so a passage sent to multiple receivers is prefilled multiple times. The table gives every phase before the first answer token for Qwen3-4B→14B, with T2T notes of up to 1,024 tokens.

Passage tokens, routeSender prefill, msSender decoding, msCodec, msReceiver prefill, msFirst token, msReceiver TFLOPs
820, forwarded passage55––13719223.3
820, T2T5329,624–14929,82725.3
820, LLM-SemanticLAN54–20431181.3
1,024, forwarded passage60––16422429.0
1,024, T2T6531,052–15131,26826.3
1,024, LLM-SemanticLAN60–25421281.4
4,096, forwarded passage236––631866123.2
4,096, T2T24234,921–16235,32529.0
4,096, LLM-SemanticLAN237–101493862.7
8,192, forwarded passage509––1,2861,794272.8
8,192, T2T51034,243–16234,91529.0
8,192, LLM-SemanticLAN513–226668054.9

Receiver TFLOPs count the codec, the prefill and the answer, and every additional receiver pays them again. Against the forwarded passage, LLM-SemanticLAN needs 17–56× less receiver compute for Qwen3-4B→14B, and the first token arrives 1.6–2.2× sooner. Against T2T, LLM-SemanticLAN is 17–34× faster in total latency at every measured length and in all three measured directions, so the 12–35× of Table 1 is not specific to the median passage. Most of the gap to T2T is the note the sender writes, and a shorter note costs accuracy. A 512-token note is 4.0–10.8 points below LLM-SemanticLAN on all six Web directions.

The saving has a price in accuracy. On Web, Qwen3-14B scores 95.6% when it reads the passage and 84.9% from the latent of Qwen3-4B (Table 3 in Appendix A). For longer contexts we placed the passage of each question among other held-out Web passages (300 questions). Reading the forwarded context, Qwen3-14B scores 94.7 with the passage alone, 94.7 at 4,096 tokens and 94.0 at 8,192. From the latent it scores 83.3, 79.3 and 78.3. The latent therefore gives up about 11 points at the median passage and 16 points at 8,192 tokens against the receiver reading the forwarded context.

All routes end with the same receiver decoding the same answer, so a longer, natural-length answer adds the same time to each route and leaves the gaps in the table unchanged. In transmitted bytes, the latent takes 2.1 MB per 1,024 positions, 80× smaller than the native Qwen3-14B cache (Table 7 in Appendix D), against about 4 KB of text. Sending 2.1 MB at 100 Gb/s takes 0.2 ms.

T2T remains the reference of the paper because written messages are the common way models communicate and collaborate today and work between any two models, while forwarding the context makes every receiver prefill it. LLM-SemanticLAN removes both the writing of the message and the repeated prefill. In the revised PDF, the abstract states that the 12–35× speedup is over text-to-text communication, and Table 2b in §4.2 and the new Appendix M add the cost and the accuracy against forwarding the passage.

Q3. COCO and the scope of the claims. You are right. On COCO, Qwen3-14B, the partner the visual encoder was trained against, scores 82.4 with LLM-SemanticLAN against 66.5 when it reads the written description. Qwen3-4B, on a direction never trained, scores 63.4 against 56.6. Qwen3-8B, also on a direction never trained, scores 58.4 against 62.4, 4.0 points below the description. The visual advantage therefore holds on the trained direction, where Qwen3-14B reaches 118.4% of T2T on average over COCO, RealWorldQA and CV-Bench, and holds on COCO for Qwen3-4B. On the two untrained directions, the external benchmarks give 88.5–89.2% of T2T (Table 1). Near-lossless refers to the text directions never trained, at 94.3–100.3% of T2T across the three added models (Table 1). In the revised PDF, §4.3 and the abstract state these results.

AuthorsOpenReview ↗

Follow-up to Reviewer LL87: a matched run without a reference (W1, Q1)

We have now completed the training run with no reference, matched in width and budget, that our first response agreed would help. With the same codec architecture, latent width, objective (including the consistency term), data and 2,000 training steps, the reference-free start reaches 8.4% accuracy on the two untrained Web directions, against 77.5% with the external reference. The reference-free run also reaches a cross-member cosine of 0.96 to 0.97, yet its accuracy stays near the no-context level of 8.2%.

This directly addresses the concern that any reasonable initialization might do, within the tested training budget. The full setup, all six directions and the limitations are in our general response at the top of this page.

AuthorsOpenReview ↗

Follow-up to Reviewer LL87: other choices of the reference (W1, Q1)

We have completed two further runs of the matched-budget control in our general response. They keep everything identical and change only the model that serves as the common target of the start. With Gemma-2-2B, a model from outside the Qwen family, as the reference, Web accuracy is 82.1% on the trained directions and 79.3% on the untrained directions. With a member, Qwen3-8B, as the reference, it is 81.6% and 76.2%. The reference of the paper gives 79.7% and 77.5%, and the run without a reference gives 8.6% and 8.4%.

Within these runs the result does not depend on which model serves as the reference, and it does depend on having one. Each configuration was run once at 2,000 steps, so we do not rank the three references. The table and the details are in our follow-up to Reviewer ScxG on this page.

Reviewer_ScxGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Public Latent Communication (PLC), a way for frozen, heterogeneous LLMs to share KV-cache information through a shared latent space that is fixed in advance. Models don't talk through text, and they don't need a separate translator for every pair. Each model gets a small codec. The authors initialize it in closed form with ridge regression, then train it only on how the receiving model behaves. To join, a new model only has to align with one existing member, and it can then communicate with all of them. In the paragraph-passing setup, model pairs that never saw each other during training come close to the accuracy of the text-note baseline while using much less latency and compute.

Critical strengths
  1. The closed-form anchoring is simple, but it matters a lot for training. In the ablation, with the same objective and the same number of steps, codecs trained from random initialization answer only 0.8% of the Web questions. Codecs initialized from the closed-form solution answer 85.3%.
Critical weaknesses
  1. Five of the six text members are Qwen models. Llama-3.1-8B is the only one from a different family, so the evidence for cross-family generalization is quite limited.
  2. The paper doesn't make the case for latent communication over simply forwarding the original text to the receiver. In the evaluation protocol, the sender just reads the paragraph and passes the information along, so the obvious alternative is to give the receiver the paragraph itself. The paper does report "receiver reads the paragraph directly" as an accuracy upper bound, but latency and FLOPs are only compared against T2T, not against direct forwarding. Direct forwarding needs no decoding on the sender side, and sending token IDs costs far less than sending a latent vector for every token.
Questions for authors
  1. The paper lacks comparisons with related work. Using the addition of a new model as an example, the authors could report for both PLC and related methods the training resources required and the accuracy on the new communication directions.
  2. How do two models that were added via different partners (e.g., Llama-3.1-8B added via Qwen3-14B, and Qwen3.5-9B added via Qwen3-4B) perform when communicating with each other?
  3. §3.3 argues that using an external reference model keeps any one member from holding a privileged position, but the paper never ablates the choice of reference model. (1) If one of the initial members were used as the anchor instead, how would accuracy change on the untrained directions? (2) The reference model, Qwen3-1.7B, comes from the same Qwen3 family as every initial member. Would the results still hold with a reference model from outside the Qwen family?
7 comments
AuthorsOpenReview ↗

Response to Reviewer ScxG (1/3): forwarding the passage (W2)

Thank you!

W2. Forwarding the passage. You are right that forwarding token IDs requires no decoding at the sender, and LLM-SemanticLAN does not decode at the sender either. In LLM-SemanticLAN, the sender only prefills the passage, a step it performs anyway to read the passage itself. The two routes differ at the receiver. Forwarding the source text pays at every receiver. Each receiver prefills the full passage, so a passage sent to multiple receivers is prefilled multiple times. With LLM-SemanticLAN, the decoder of the receiver writes the passage cache directly, and the receiver prefills only the question. We measured forwarding the passage with the protocol of Appendix H on an A100 with 24 answer tokens, at four passage lengths. The table gives every phase before the first answer token for Qwen3-4B→14B, with T2T notes of up to 1,024 tokens.

Passage tokens, routeSender prefill, msSender decoding, msCodec, msReceiver prefill, msFirst token, msReceiver TFLOPs
820, forwarded passage55––13719223.3
820, T2T5329,624–14929,82725.3
820, LLM-SemanticLAN54–20431181.3
1,024, forwarded passage60––16422429.0
1,024, T2T6531,052–15131,26826.3
1,024, LLM-SemanticLAN60–25421281.4
4,096, forwarded passage236––631866123.2
4,096, T2T24234,921–16235,32529.0
4,096, LLM-SemanticLAN237–101493862.7
8,192, forwarded passage509––1,2861,794272.8
8,192, T2T51034,243–16234,91529.0
8,192, LLM-SemanticLAN513–226668054.9

Receiver TFLOPs count the codec, the prefill and the answer, and every additional receiver pays them again. Against the forwarded passage, LLM-SemanticLAN needs 17–56× less receiver compute for Qwen3-4B→14B, and the first token arrives 1.6–2.2× sooner. Against T2T, LLM-SemanticLAN is 17–34× faster in total latency at every measured length and in all three measured directions.

The saving has a price in accuracy. On Web, Qwen3-14B scores 95.6% when it reads the passage and 84.9% from the latent of Qwen3-4B (Table 3 in Appendix A). For longer contexts we placed the passage of each question among other held-out Web passages (300 questions). Reading the forwarded context, Qwen3-14B scores 94.7 with the passage alone, 94.7 at 4,096 tokens and 94.0 at 8,192. From the latent it scores 83.3, 79.3 and 78.3. The latent therefore trades about 11 to 16 points of accuracy for the saved prefill.

On bytes, you are right. Our latent takes 2.1 MB per 1,024 positions, 80× smaller than the native Qwen3-14B cache (Table 7 in Appendix D), against about 4 KB of text. Sending 2.1 MB at 100 Gb/s takes about 0.2 ms, against the 137 ms that Qwen3-14B spends to prefill a forwarded passage of 820 tokens. Serving systems already ship full KV caches between machines to avoid recomputing prefill (DistServe, Mooncake) and compress these caches for the network (CacheGen). LLM-SemanticLAN removes the receiver's prefill at every passage length and for every receiver, and it also carries inputs that are not text, such as images from the visual encoder of §4.3. In the revised PDF, Table 2b in §4.2 and the new Appendix M add the cost and the accuracy against forwarding the passage.

AuthorsOpenReview ↗

Response to Reviewer ScxG (2/3): model families, adding a model (W1, Q1)

W1. Families. Llama-3.1-8B differs from the Qwen members in family and tokenizer, and it was trained only against Qwen3-8B. Its four untrained directions reach 94.3% of T2T (Table 1). A new family changes the cache through its tokenizer, layer and head layout, attention type or modality, and the pool already covers each of these. Llama-3.1-8B brings a new tokenizer, handled by byte-offset correspondence, Qwen3.5-9B brings hybrid attention, Qwen3-VL-8B brings images, and II-Medical-8B brings domain post-training. A further family joins by the same procedure as Llama-3.1-8B. We agree that more families strengthen the claim, so for this response we added a model of a third family. Gemma-2-2B has its own tokenizer, 26 layers, a different head layout (4 key-value heads of 256 dimensions, against 8 of 128 in the Qwen models) and 2.6B parameters. With our limited computational resources we trained its codec (130M parameters) against Qwen3-8B alone for 1,000 steps, far fewer than the 4,000 to 12,000 steps of the added models in the paper. The table gives judge accuracy in percent on the 1,345 answerable Web questions for the four directions between Gemma-2-2B and the two members it never trained with.

Direction, never trainedAccuracyReceiver reads the passage itself
Gemma-2-2B→Qwen3-4B67.695.1
Qwen3-4B→Gemma-2-2B63.687.6
Gemma-2-2B→Qwen3-14B71.595.2
Qwen3-14B→Gemma-2-2B66.787.6

On the trained pair, the 186 Web validation questions give 69.4 for Gemma-2-2B→Qwen3-8B and 69.9 for the reverse. The never-trained directions are therefore close to the trained pair, so a model of a third family that trained with one member reaches the others. We state the limits plainly. After 1,000 steps these directions reach 71 to 76 percent of the accuracy of the receiver reading the passage itself, below the added models of the paper, and questions whose answer is a long number are the weakest type (43 to 53). The run is short. On the same validation questions, II-Medical-8B and Qwen3.5-9B gained 1 to 9 points on Web between 1,000 and 12,000 steps. We report this run as a preliminary result. In the revised PDF, the new Appendix O gives the details.

Q1. Adding a model, compared with related work. Our setting follows how a multi-agent system divides work. One model reads a document, a tool output or an image and hands what it read to a model that has not read it, and any model may hand off to any other. Our setting is also the economical one. The sender prefills the input once, and each receiver prefills only its question, so a passage sent to multiple receivers is prefilled only once. C2C addresses a different task, in which both models read the same input and the sender's cache is fused into the receiver's own cache of that input to improve the receiver's answer. C2C therefore cannot run when the receiver has not read the input, and a passage sent to multiple receivers is prefilled multiple times. The other pairwise methods (Heo et al., Qu et al., XKV) train one module per ordered pair, so no pair works before that pair is trained. A receiver can also take an input it cannot read itself, such as an image passed from Qwen3-VL-8B to a text-only model (§4.3).

The pairwise method that runs in this setting is the closed-form map of Heo et al., compared in §3.2 (Table 8 in Appendix D). On Web it reaches 79.2% accuracy against 87.5% for LLM-SemanticLAN on Qwen3-14B→4B and 82.8 against 84.2 on Qwen3-4B→14B. On Qwen3-8B→4B, two models with the same number of layers, the map is ahead with 90.0 against 87.2. Each such map stores 2.7B parameters. A map restricted to the size of a codec reaches 90.9, 75.5 and 38.3 on Qwen3-4B→8B, Qwen3-8B→4B and Qwen3-8B→14B, against 86.8, 86.7 and 88.0 for LLM-SemanticLAN (details in our response to Reviewer 389s). The table below compares the cost of adding the N-th model.

Adding the N-th modelTrainedPartners neededUntrained directions
Pairwise maps or fusers2(N−1) new modules, one per ordered pairall N−1none exist
Jointly trained space (Dery et al.)own adapters, about 1/4 of the modeltwoyes
LLM-SemanticLANown codec, 0.8–3.6% of the modeloneyes, 94.3–100.3% of T2T (Table 1)

Adding a model took 9 to 50 A100 hours at the step counts of the paper, which were set conservatively. After 2,000 steps, 8 to 15 A100 hours, II-Medical-8B already reaches 80.6 to 83.7 on its untrained Web directions. Dery et al. use no reference model and train on continuation. Under our data and budget, removing the closed-form start onto the reference, or training on continuation alone, gives 0.8% and 74.5% against 85.3% (Figure 5). These two ablations isolate the design choices and do not reproduce the system of Dery et al.

AuthorsOpenReview ↗

Response to Reviewer ScxG (3/3): different partners, choice of reference (Q2, Q3)

Q2. Models added via different partners. One correction first. All three added text models were trained against Qwen3-8B, and the visual encoder against Qwen3-14B (Appendix D). The visual encoder and each added text model are therefore pairs added via different partners and never trained together. We measured all three on COCO (5,936 questions, the judge of the paper, LLM-SemanticLAN against the written description). II-Medical-8B scores 63.8 against 58.2 (110%), with 11.4 for no context and 8.7 for a mismatched image. Qwen3.5-9B scores 49.9 against 68.8 (73%). Llama-3.1-8B scores 18.3 against 68.7, because its decoder resamples latent positions by byte spans, which image positions do not have. Adapting only its decoder on image states for 2,000 steps, with everything else frozen, raises it to 83.4 (121%). Such a direction therefore works without training for II-Medical-8B, partly for Qwen3.5-9B, and after a short decoder adaptation for Llama-3.1-8B. Among the three added text models, the six never-trained directions average 100.3% of T2T (Table 1). In the revised PDF, Appendix I adds these cells.

We also added Llama-3.1-8B again with Qwen3-4B as its only partner, for 4,000 steps, and compare it with the codec of the paper, trained against Qwen3-8B for more steps. Answers are scored here by a rule that counts an answer as correct when its first line contains the gold answer. The table gives the score of each direction as a percentage of the score of the same receiver reading the passage itself, on Synthetic / Web.

DirectionPartner Qwen3-4BPartner Qwen3-8B
Qwen3-4B→Llama-3.1-8B94.2 / 79.8, trained94.9 / 77.8
Qwen3-8B→Llama-3.1-8B88.9 / 77.198.8 / 83.5, trained
Qwen3-14B→Llama-3.1-8B87.3 / 77.895.4 / 83.4
Llama-3.1-8B→Qwen3-4B80.5 / 67.7, trained91.8 / 73.0
Llama-3.1-8B→Qwen3-8B66.7 / 64.097.7 / 81.4, trained
Llama-3.1-8B→Qwen3-14B54.4 / 58.676.9 / 76.1

As a receiver, the codec trained against Qwen3-4B keeps 77 to 78 percent on Web from the two senders it never trained with, against 80 from its partner, so its decoder learned the shared coordinates and not those of its partner. As a sender it is weaker, most of all toward the members it never trained with, so a model should be added against the strongest available member (Appendix I.1).

Q3. Choice of reference.

(1) A member as the reference. The paper does not include this ablation, and we restrict our claim to what our controls support, that an external reference is sufficient. We chose an external reference for a practical reason. It gives the three initial models the same regression problem and takes no part in communication, so no member is privileged and every kept direction is one that all three models predict (Table 6 in Appendix D). A member could also serve as the common target. In an earlier configuration (reference Qwen3-0.6B, no consistency term) where part of the coordinates started from each member's own principal directions, with no common target, these coordinates stayed private after joint training, with cross-member cosine 0.06 against 0.90 on the anchored coordinates. Removing the private coordinates changed accuracy on the untrained direction Qwen3-14B→4B by only +0.4 on Web and +0.1 on Synthetic. What this control shows is the value of a common target, which an external reference provides without choosing among the members.

(2) A reference from outside the Qwen family. The reference serves only to initialize the codecs of the initial models and plays no part afterwards. We chose a Qwen3 model because Qwen is among the most widely used families of open-weight models, and with Qwen3-1.7B as the reference every text row of Table 1 averages 94 to 101 percent of T2T. Llama-3.1-8B, which has its own tokenizer, joins the latent space built on Qwen3-1.7B and reaches 94.3% of T2T on its four untrained directions (Table 1). A Qwen3 reference therefore does not restrict which families can communicate, and other families join as new models. The results also change little with the size of the reference. With Qwen3-0.6B as the reference, at the same latent width and number of steps, the six directions reach 92.1 / 74.8 against 92.7 / 77.4 for Qwen3-1.7B (Synthetic / Web). The two runs also differ in learning rate and schedule and use earlier question sets, so this comparison is indicative, and it covers the size of the reference within one family. We have not yet used a reference from another family. It is an interesting direction for future work, and the conclusions of this paper do not depend on it. In the revised PDF, Appendix D.3 adds this comparison.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: a matched-budget control without a reference

We thank all three reviewers. Since posting our responses we completed one matched-budget control that bears on a question raised by each reviewer, namely how LLM-SemanticLAN compares with a shared space trained jointly without a reference (Reviewer 389s, W1), what the external reference contributes (Reviewer LL87, W1 and Q1), and how the reference should be chosen (Reviewer ScxG, Q3). We report it here once.

Setup. The three initial models are trained twice with the recipe of the paper, and the two runs differ only in the closed-form start. In the first run each model starts from the ridge solution onto the targets of the external reference, as in the paper. The second run is reference-free. Each model starts from the principal directions of its own cache at the same latent width, and its decoder is the ridge solution back to its own cache. This start is well scaled and keeps most of each model's own cache, which each model reconstructs from it with R² between 0.77 and 0.84. R² here is the share of the variance of the cache that the reconstruction explains. Everything else is identical, including the codec architecture, the data and its order, the objective with the consistency term, the learning rates, and the two directions between Qwen3-4B and Qwen3-14B held out from training. With our limited computational resources both runs train for 2,000 steps instead of the 16,000 steps of the main checkpoint, and each was run once.

Result. The table gives judge accuracy in percent on the 1,345 answerable Web questions.

DirectionWith the referenceReference-free
Qwen3-4B→Qwen3-8B, trained78.48.3
Qwen3-8B→Qwen3-4B, trained81.07.0
Qwen3-8B→Qwen3-14B, trained81.610.4
Qwen3-14B→Qwen3-8B, trained77.78.5
Qwen3-4B→Qwen3-14B, untrained77.89.6
Qwen3-14B→Qwen3-4B, untrained77.17.1

The reference-free run averages 8.6% on the four trained directions and 8.4% on the two untrained directions, against 79.7% and 77.5% with the reference. A receiver that is given no context at all scores 8.2% on Web (Table 1), so after 2,000 steps the reference-free run stays near the no-context level. On Synthetic the same directions reach 6.0% and 5.2% against 93.5% and 87.5%, a few points above the 2.4% of a receiver with no context. The paired differences on Web are 71.1 points on the trained directions and 69.1 points on the untrained directions, both statistically significant (95% intervals 69.9 to 72.4 and 67.2 to 71.0, paired bootstrap over questions).

Encoder alignment alone is not sufficient. The reference-free run does bring the members together. Its consistency term raises the cross-member cosine, measured between the latents that two members write for the same passage, from 0.00 at the start to between 0.96 and 0.97 at step 2,000, as high as in the run with the reference (0.96). The receivers still cannot read these latents. Their answers have the right type and the wrong content. Asked for a date whose answer is September 24, 2013, the receiver answers 15th of April 2016. This complements the ablation in our response to Reviewer 389s, where removing the consistency term lowered the same cosine from 0.95 to 0.68 while accuracy held. The cosine measures agreement between the encoders. Whether a receiver can read the latent is a separate question, and accuracy is what answers it. The reference therefore contributes more than agreement between the members. Starting from its coordinates, the same training reaches a latent that the receivers read, and starting without them it does not within this budget.

What this shows and what it does not. Within the tested budget of 2,000 steps, a well-scaled start without a common target does not lead to usable communication, while the start onto the reference reaches 77% to 82%. This addresses the concern that any reasonable initialization might do. It does not establish that an external reference is necessary. The run is short, each configuration was run once, and we did not test whether a much longer reference-free run would catch up. The control uses our codecs, data and budget, so it is not a reproduction of the system of Dery et al. and says nothing about other architectures or longer training. Our claim stays as in our responses. An external reference is sufficient to establish shared coordinates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.