acceptodds
Under review as a conference paper at ICLR 2027

Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems

Abstract

Heterogeneous multi-agent systems combine models with different capabilities through a common communication interface. Exchanging internal states directly requires translating between model-specific representations and controlling intermediate computation. We introduce the Vision Wormhole, which repurposes the visual input interface of Vision-Language Models (VLMs) for continuous communication between frozen heterogeneous agents. A Universal Visual Codec encodes each sender's latent rollout into a fixed-size message, maps it through a shared reference space, and decodes received messages into the receiver's image-token span. Per-model codecs and affine reference maps form a hub-and-spoke architecture with components for models. Each model learns its codec independently through self-distillation on anchor texts, and shared-anchor alignment enables reuse across communication partners. Across four VLM families, six team configurations, and nine reasoning benchmarks, Vision Wormhole improves accuracy by 6.0 percentage points on average over text-mediated MAS and achieves a 1.69× geometric-mean speedup in batch-normalized end-to-end runtime.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
latent communication, multi-agent systems, vision-language models, model composition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cUNPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Vision Wormhole, which uses the image-token interface of frozen vision-language models to transform one model’s hidden states into continuous context that another model can consume. Each model learns its codec through local self-distillation, while shared anchor texts and affine maps connect the codecs through a reference space.

Critical strengths
  1. An interesting interface design. Repurposing VLMs’ existing visual conditioning interfaces for cross-model hidden-state communication offers a new implementation path for heterogeneous collaboration.
  2. Frozen backbone models. Training is confined to codecs and alignment maps, enabling the reuse of existing pretrained models.
  3. Modular integration. Each model has its own codec and connects to multiple partners through a reference space, with the component count scaling linearly with the number of models.
Critical weaknesses

Overall, using the visual input interface of frozen VLMs for cross-model communication is an interesting idea. However, the paper does not clearly establish its technical contributions or distinguish them from existing latent communication methods, and parts of the presentation lack a coherent logical flow.

  1. The substantive differences from existing methods need clarification. Hidden-state transfer, communication without intermediate text generation, and heterogeneous model collaboration have already been explored. The authors should directly compare VW with Cache-to-Cache, and shared-space approaches, clarifying whether the novelty lies in the receiver interface, the training procedure, or their combination.

  2. The motivation for the visual interface contains a logical gap. A VLM’s ability to understand rendered text does not directly establish that its image-token interface is suitable for receiving another model’s hidden states. Building on the existing soft-prefix ablation, the authors should explain why the visual interface works and clarify its relationship to dynamic soft prompting.

  3. The benefits of different design choices should be attributed separately. The O(N)O(N) component count primarily follows from the hub-and-spoke reference-space architecture rather than the visual interface itself. The authors should distinguish the contributions of visual conditioning and shared-space integration, while clarifying the scope of independent training and the requirements for shared-anchor calibration.

  4. The related work needs a clearer organizational logic. The “Multi-agent coordination” subsection moves through role specialization, model selection, memory, and workflow optimization without clearly connecting these topics to the communication problem studied here. It should be organized around the research gap rather than ending abruptly with a description of the experimental comparison.

Questions for authors

My questions are the same as the concerns raised above.

12 comments

Overview of our response

Thank you for recognizing the receiver interface, the frozen backbones, and the modular integration, and for noting that repurposing the visual conditioning interface "offers a new implementation path for heterogeneous collaboration". Your four points were precise about what to clarify and where, which made them straightforward to act on. The request to attribute visual conditioning and the reference space separately was especially useful, and the revision will state that attribution explicitly. Your points and where each is answered:

Your pointWhat we addedResult
W1: where the novelty lies, and a direct comparison with Cache-to-Cache and shared-space approachesA design comparison of eight methods, which places the shared-space methods next to VW, and experiments with Cache-to-Cache and with two further latent channels, StateBridge (COLM 2026) and XBridge (NeurIPS 2026)The novelty lies in the combination of the receiver interface, the training procedure and the calibration. In the SmolVLM2/Qwen team, VW is above Cache-to-Cache on both benchmarks (93 against 71 correct of 128 on GSM8K, and 67 against 48 of 164 on HumanEval+), and it is the most accurate channel in all seven rows of the comparison with StateBridge and XBridge
W2: why the visual interface works, and its relation to dynamic soft promptingThe two reasons, and the controlled comparison of the visual span against a learned text prefixWith matched training and inputs, the visual entry point is 15.6 and 7.4 points more accurate than the text prefix (Table 7)
W3: separating visual conditioning from the reference space, the scope of independent training, and the requirements of calibrationOne test for each design choice, and the requirements of calibration listedThe O(N) count comes from the reference space, and visual conditioning is what the reference space connects
W4: organization of the related workSection 2 reorganized around three questions of the communication problemThe opening of each part is quoted in our reply to W4

The comments below give the tables and settings. We are happy to clarify any point further during the discussion.

W1: technical contribution relative to Cache-to-Cache and shared-space methods

The novelty lies in the combination. VW combines a receiver interface (the native image-token span, with the receiver unmodified), a training procedure (each codec is self-distilled on its own frozen backbone, with no other model in the loop), and a calibration (closed-form affine maps fitted on shared anchor texts) that makes independently trained codecs usable together. The table places this combination next to Cache-to-Cache and the shared-space methods:

MethodTransmittedEntry point in the receiverTrained, and per what unitTraining data and lossModels it can connect (families evaluated)Trained modules for N models
Cache-to-CacheSharer's KV cacheReceiver KV cache, gated fusionOne fuser per ordered sharer–receiver pair, LLMs frozen500K instruction conversations per fuser, next-token lossAny pair, given a fuser trained for each ordered pair (Qwen, Llama, Gemma)N(N−1) fusers
KVCommKV pairs of selected sender layersReceiver attention at the same layer indexNothing (layers calibrated per model pair)NoneOnly models with one base architecture: identical models or fine-tunes of one base0
LatentMASAll-layer KV caches with latent thoughtsReceiver KV cacheNothingNoneOnly agents that share one backbone0
CIPHERExpected token embeddingsInput embeddingsNothingNoneOnly models that share one tokenizer0
Activation communicationOne activation vectorResidual stream at the last tokenNothing by default (optional map per model pair)None by defaultQwen-2.5, Gemma-2, LLaMA-30 (N(N−1) maps if used)
Mixture of ThoughtsExpert hidden states projected to a shared spaceInteraction layers of the primary expertRouter and interaction layers, jointly for a fixed poolTrain splits of five evaluated benchmarks, language-modeling lossA fixed pool, retrained when a model is added (Mistral and Llama-3 variants)Joint, for the fixed pool
ThoughtCommAutoencoder codes of agents' last-token statesPrefix vectors on the token embeddingsAutoencoder and adapter over a fixed agent set500 examples of the evaluated tasks, reconstruction and likelihood lossesA fixed agent set (one base model per run)Joint, for the fixed set
Vision WormholeFixed-size message (1026 × 512) from the sender's latent rolloutNative image-token span, receiver unmodifiedOne codec per model, self-distilled on its own frozen backbone (Eq. 2), and closed-form affine maps (Eq. 3)Unlabeled anchor texts, the model's own text-conditioned state and next-token distribution as targets, no benchmark dataAny vision-language model that reads images as tokens in its input sequence, added one at a time (Qwen3-VL, Gemma-3, SmolVLM2, LFM2.5-VL, and InternVL3.5 (new))N codecs, 2(N−1) maps

VW's codecs are trained one model at a time, with no other model in the loop, and a team of N models needs N codecs and 2(N−1) closed-form maps. Cache-to-Cache and Mixture of Thoughts connect different models through modules trained per pair or per pool. The Cache-to-Cache paper itself names scaling to more models at O(N) training cost as an open problem. Its appendix reports an initial exploration of a shared space, in which the projectors and fusers of all models are trained jointly and each sharer–receiver pair keeps its own gate. As published, KVComm applies only to models with one base architecture, LatentMAS only to agents that share one backbone, and CIPHER only to models that share one tokenizer, so none of the three connects models from different families without modification. Entries for the other methods follow each method's own paper. The last column counts the trained modules the design implies for a team of N models. We will add this table as Table 17 (Appendix A.3) and the same comparison for nine concurrent 2026 papers as Table 30 (Appendix H.12).

W1 (continued): direct comparisons with latent channels, and a test for each part

Direct comparison. In the SmolVLM2/Qwen team, VW is above Cache-to-Cache on both benchmarks: 93 against 71 correct of 128 on GSM8K, and 67 against 48 of 164 on HumanEval+. End-to-end comparisons with channels that use text generation or model internals are in Appendix D (OCR relay) and Appendix F (CIPHER, LatentMAS-Hybrid). XBridge (NeurIPS 2026, a learned cross-attention bridge trained per model pair) and StateBridge (COLM 2026, a hidden-state prefix channel) are run in that comparison as well (second table below), and the extended literature review of the revision will include both. We also ran Cache-to-Cache directly on SmolVLM2/Qwen. With two models, its pairwise design is the complete method. Its released fusers cover text-only model pairs, so its two fusers are trained on the two models' language decoders with the released code and objective, each on 50K OpenHermes-2.5 conversations (about 17 times the anchor texts of a VW codec, see our reply to Reviewer WfdA, W1), and run in the four-role chain (accuracy % with correct counts, on the first 128 GSM8K questions and all 164 HumanEval+ problems):

ChannelGSM8KHumanEval+
Vision Wormhole72.7 (93)40.9 (67)
Cache-to-Cache (ICLR 2026)55.5 (71)29.3 (48)
TextMAS64.8 (83)31.1 (51)

(To be added as Table 29, Appendix H.11.)

Two channels published in 2026: StateBridge and XBridge. Every channel in the table below uses the same team, role assignment, role prompts of Zou et al. (2025), evaluation questions, sampling, and final-answer cap (Table 3). TextMAS and VW are the main-experiment predictions restricted to the first 128 GSM8K test questions (the question set of Table 9) and the first 128 ARC-Challenge test questions, plus all 164 HumanEval+ problems, and the first 128 MedQA questions for SmolVLM2/Gemma, and the added channels are run on the same questions. Accuracy (%, correct out of 128, or of 164 for HumanEval+):

ConfigurationBenchmarkTextMASStateBridge (COLM 2026)XBridge (NeurIPS 2026)VW
SmolVLM2 / GemmaGSM8K65.6 (84)73.4 (94)78.9 (101)82.8 (106)
ARC-Challenge71.9 (92)69.5 (89)78.1 (100)83.6 (107)
MedQA43.8 (56)39.8 (51)50.0 (64)53.9 (69)
HumanEval+32.9 (54)43.9 (72)43.3 (71)59.1 (97)
SmolVLM2 / QwenGSM8K64.8 (83)59.4 (76)13.3 (17)72.7 (93)
ARC-Challenge71.1 (91)75.0 (96)69.5 (89)81.2 (104)
HumanEval+31.1 (51)31.7 (52)0.6 (1)40.9 (67)

VW is the most accurate channel in all seven rows. (To be added as Table 19, Appendix F. How each channel was run is described in our reply to Reviewer WfdA, W1.)

A test for each part. The three parts are a continuous-input pathway that every VLM already has, a training signal that comes entirely from the receiver's own model so that codecs are trained independently, and shared-anchor calibration that makes independently trained codecs composable. Each has its own test:

PartTestResult
Entry pointVisual span against a learned text prefix, matched architectures and training, same intermediate trajectories (Table 7)+15.6 pp on GSM8K, +7.4 pp on MedQA, and +14.4 and +9.8 pp at the rollout of the main experiments (to be added)
CalibrationHeld-out reconstruction KL: aligned / cross-text / shuffled-anchor (Table 13, Appendix G codecs)0.227 / 0.513 / 0.464 and 1.864 / 6.344 / 5.408 in the two directions
CompositionEach backbone's codec is one set of trained weights, used unchanged in every configuration that contains the backbonethe six configurations of Figure 4, which include a four-backbone pool, a parallel topology with the Gemma-3-4B codec (Table 10), and (new) a five-model team that adds InternVL3.5-2B with nothing fitted on the team: 75.0 against 60.2 for TextMAS on GSM8K, and 73.4 against 69.5 on ARC-Challenge, on the first 128 questions (to be added as Table 21)

(Table 7, Appendix G; Table 13, Appendix H.4; Table 10, Appendix H.2; Table 21, Appendix H.8)

Reviewer_WfdAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Vision Wormhole, a latent-space communication framework for heterogeneous VLM multi-agent systems. Its core idea is to leverage the visual input interface of pre-trained VLMs to convert the sender’s latent reasoning state into continuous visual representations, thereby avoiding the per-token generation overhead in conventional text-based communication.

Critical strengths

This work presents a novel modular communication architecture for heterogeneous latent communication, which repurposes pretrained visual interfaces as continuous conditioning channels. Leveraging independently trained codecs and shared-space alignment, it offers a modular substitute for pairwise model-specific bridges and enhances scalability with growing agent numbers. Extensive experiments across diverse VLM families, agent setups and reasoning tasks demonstrate promising accuracy and runtime improvements, with ablation studies validating the efficacy of the visual-interface and alignment designs.

Critical weaknesses
  1. Comparisons with alternative latent communication methods are inadequate. Evaluations against CIPHER and LatentMAS-Hybrid cover only two benchmarks and one heterogeneous setup (Appendix F), with mismatched latent-channel configurations that compromise comparison fairness. Systematic assessments under identical communication and compute budgets are required to rigorously verify itss superiority.

  2. Evidence for scalability and generalization is sparse. The claimed O(N) scalability applies only to learnable components, yet experiments use at most four backbones. Communication quality, memory usage and computational overhead under larger agent groups or extended interaction sequences remain not clear.

  3. As far as I know, there are still some related works and survey about visual multi-agent as well as latent space are not cited and compared in your paper.

Questions for authors

See Weakness.

9 comments

Overview of our response

Thank you for the detailed assessment, and for describing the design as "a modular substitute for pairwise model-specific bridges". The new experiments compare VW with such bridges directly and add a fifth backbone. Your three points asked for the evidence a modular channel should come with (broader comparisons, larger teams and longer interaction, and a fuller view of the literature), and they shaped a large part of the new experiments. Your points and where each is answered:

Your pointWhat we addedResult
W1: comparisons with latent channels cover two benchmarks and one configurationFour more channels: Activation Communication (ICML 2025), Cache-to-Cache (ICLR 2026), StateBridge (COLM 2026) and XBridge (NeurIPS 2026). Two more configurations (SmolVLM2/Qwen and SmolVLM2/Gemma), with ARC-Challenge and MedQA next to the benchmarks of Table 6VW is the most accurate channel on both benchmarks of the extended Table 6 and in all seven rows of the comparison with StateBridge and XBridge
W1: channel configurations, and communication and compute budgetsEach channel reproduced as closely as possible to its original implementation, with everything outside the channel identical, and its decoded tokens and seconds per question measuredVW's intermediate roles decode no tokens in any of the four rows of the measured table
W2: more backbones, more agents, longer interactionA fifth backbone with nothing fitted on the team, chains of six and eight roles, and storage, memory and decoding time for up to 15 accumulated messagesVW is more accurate than TextMAS in all twelve cells, by 14.8 and 3.9 pp in the five-model team. The receiver's input length is the same for one message and for fifteen
W3: related work and surveys on visual multi-agent systems and latent communicationThe lines of work listed in our reply to W3, with their place in the revisionSection 2 will be reorganized around the communication problem

The comments below give the tables and settings. We are happy to clarify any point further during the discussion.

W1: comparisons with latent communication methods

Result. VW is the most accurate channel on both benchmarks of the extended Table 6 and in all seven rows of the comparison with StateBridge (COLM 2026) and XBridge (NeurIPS 2026), and its intermediate roles decode no tokens (tables below and in the continuations).

As requested, we ran the comparison of Table 6 on a second configuration and added Activation Communication and Cache-to-Cache. We also ran two channels published in 2026, StateBridge (COLM 2026) and XBridge (NeurIPS 2026). The extended literature review of the revision will include both, and all tables below will be added to the revision.

The channels compared. They differ in what they transmit and where it enters the receiver (n is the number of generated text tokens, to be added as Table 18, Appendix F):

ChannelTransmitted per intermediate roleEntry point in the receiverReceiver context addedTraining and fitting
TextMASgenerated text, n tokenstoken input+n tokensnone
CIPHER (ICLR 2024), Table 6256 expected-embedding tokensinput embeddings+256none
LatentMAS-Hybrid (Table 6)per-layer cache of 32 latent stepsKV cachecache entriesnone
Activation Communication (ICML 2025; Ramesh & Li, 2025)one activation vectorresidual stream at 0.86 relative depth0none
StateBridge (COLM 2026; Peng et al., 2026), extended to model pairs with different tokenizers and hidden sizeslast-layer states of the last 64 generated tokensinput-embedding prefix+64none trained, but a Procrustes map is fitted for every message at inference
XBridge (NeurIPS 2026; Yang et al., 2026)the generated message as tokens mapped to the receiver's vocabulary, plus its last-layer statesanchor prefix and cross-attention every 7 layers+n anchorsone bridge per ordered pair, trained on answer-labeled questions of the evaluated tasks
Cache-to-Cache (ICLR 2026; Fu et al., 2025)the sender's per-layer cache over the receiver's promptKV cache, gated fusion0one fuser per ordered pair, trained on instruction conversations
Vision Wormholeone 1026 × 512 messagenative image-token span+64 (Qwen) or +256 (Gemma) positions, fixed for any number of messagesone codec per model, trained without benchmark questions, and two closed-form affine maps per non-reference model

Why the channel configurations differ. Each of these channels was published in its own setting, with its own models, tasks and message format, so no single configuration is shared across them, and bringing any of them into one team requires adaptation. We reproduce each original implementation and its settings as closely as possible and keep everything outside the channel identical: team, role prompts, questions, sampling and final-answer cap. The communication and compute budget of each channel is then measured (continuation below).

Table 6 extended. The benchmarks of Table 6 (GSM8K, HumanEval+) on a second configuration, SmolVLM2/Qwen: CIPHER and Activation Communication at the channel settings of Appendix F, with Cache-to-Cache added. The three write a quantity of the sender into the same kind of site in the receiver (input embeddings, the residual stream, the KV cache). Accuracy %, with correct counts in parentheses, on the first 128 GSM8K questions and all 164 HumanEval+ problems:

ChannelGSM8KHumanEval+
Vision Wormhole72.7 (93)40.9 (67)
Activation Communication (ICML 2025)69.5 (89)33.5 (55)
CIPHER (ICLR 2024)64.1 (82)31.1 (51)
Cache-to-Cache (ICLR 2026)55.5 (71)29.3 (48)
TextMAS64.8 (83)31.1 (51)

VW is the most accurate channel on both benchmarks. (To be added as Table 29, Appendix H.11. The training of the Cache-to-Cache fusers is described in the continuations below.)

W1 (continued): StateBridge and XBridge, and the communication and compute budget

Two channels published in 2026: StateBridge and XBridge. Every channel in the table below uses the same team, role assignment, role prompts of Zou et al. (2025), evaluation questions, sampling, and final-answer cap (Table 3). TextMAS and VW are the main-experiment predictions restricted to the first 128 GSM8K test questions (the question set of Table 9) and the first 128 ARC-Challenge test questions, plus all 164 HumanEval+ problems, and the first 128 MedQA questions for SmolVLM2/Gemma, and the added channels are run on the same questions. Accuracy (%, correct out of 128, or of 164 for HumanEval+):

ConfigurationBenchmarkTextMASStateBridge (COLM 2026)XBridge (NeurIPS 2026)VW
SmolVLM2 / GemmaGSM8K65.6 (84)73.4 (94)78.9 (101)82.8 (106)
ARC-Challenge71.9 (92)69.5 (89)78.1 (100)83.6 (107)
MedQA43.8 (56)39.8 (51)50.0 (64)53.9 (69)
HumanEval+32.9 (54)43.9 (72)43.3 (71)59.1 (97)
SmolVLM2 / QwenGSM8K64.8 (83)59.4 (76)13.3 (17)72.7 (93)
ARC-Challenge71.1 (91)75.0 (96)69.5 (89)81.2 (104)
HumanEval+31.1 (51)31.7 (52)0.6 (1)40.9 (67)

VW is the most accurate channel in all seven rows. (To be added as Table 19, Appendix F.)

Communication and compute budget, measured. Tokens decoded by the three intermediate roles and end-to-end seconds per question (batch size 1, one A6000, the same 128 questions):

ConfigurationBenchmarkTextMASStateBridge (COLM 2026)XBridge (NeurIPS 2026)VW
SmolVLM2 / GemmaGSM8K1162 tokens, 40.9 s826, 42.1 s1055, 83.2 s0, 13.0 s
ARC-Challenge1660, 55.8 s759, 31.0 s685, 42.5 s0, 8.8 s
SmolVLM2 / QwenGSM8K2815, 99.5 s2525, 110.7 s376, 8.0 s0, 33.2 s
ARC-Challenge3006, 103.2 s2498, 100.8 s393, 8.2 s0, 35.9 s

VW's intermediate roles decode no tokens in any of the four rows. (To be added as Table 28, Appendix H.10.)

In the table above, StateBridge's time includes the Procrustes map it fits for every message (three per question, 4.8–5.1 s). XBridge's bridges are trained beforehand, one per ordered model pair on 600 answer-labeled training questions, and that training is not counted.

Reviewer_XEX5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Vision Wormhole enables continuous communication among frozen heterogeneous VLMs through their image-token input spans. Each model learns an independent codec by matching its own text-conditioned boundary representation and next-token distribution. Shared text anchors then align codec message spaces through affine maps to a reference model, yielding a hub-and-spoke architecture with O(N) components. Across four model families, six team configurations, and nine benchmarks, the paper reports a 6.0 percentage-point average accuracy improvement and a 1.69× geometric-mean speedup in batch-normalized runtime over text-mediated MAS.

Critical strengths

The interface choice is appealing: a pretrained VLM's visual conditioning pathway provides a reusable destination for another model's continuous state. Separating independent codec training from anchor-based alignment is a practical modular design, and the O(N) component-count claim follows clearly from the architecture. The empirical evaluation spans multiple model families and task types, with accuracy and runtime improving together in 36 of 54 configuration-task pairs. The appendix meaningfully strengthens the paper through matched visual-versus-soft-prefix training, concise-text comparisons, an additional topology, cross-model retrieval, and held-out alignment controls. These make the evidence more substantial than a comparison only against long textual messages.

Critical weaknesses

The main-task gains are not directly connected to the information contributed by intermediate agents. The held-out reconstruction and retrieval experiments support content-sensitive communication, but do not establish how much of the reasoning improvement depends on task-relevant intermediate computation. All agents receive the task, and the trained codecs could partly provide a learned compression or continuous reformulation of that task. Main-task controls that remove messages, replace them with messages from another question, or encode only the initial prompt-boundary state would distinguish this explanation from useful transmission of latent reasoning. This matters because codec training itself matches the initial boundary state and next-token distribution; it does not directly supervise the utility of the subsequent rollout or accumulated role messages.

The accuracy-cost advantage needs a stronger reference point. The paper establishes an improvement over its selected TextMAS workflow, but Figure 7 averages configurations containing each model, which does not identify the best single-model alternative for each team. The +1.8-point gain for the Qwen-containing group is useful evidence but leaves the cost of obtaining that gain unclear. A comparison with the strongest member alone, and a modest additional-inference-budget version of that member, would clarify when heterogeneous collaboration is beneficial. Appendix H.1 already tests concise text, but does not provide the corresponding runtime trade-off. Such a comparison would more directly establish the operating regime where the proposed channel is preferable.

Questions for authors

On the main reasoning tasks, what happens when received messages are removed or swapped across questions? How much does a full latent rollout add over encoding the initial prompt-boundary state?

Can you report accuracy and inference cost against the strongest standalone member for each configuration, and timing for the concise TextMAS controls already evaluated?

Is the main inference rollout length the training length of 1024 for every intermediate role? Clarifying the actual lengths and message-processing cost would make the efficiency comparison easier to interpret.

8 comments

Overview of our response

Thank you for the careful reading and for the specific questions, and for noting that the appendix controls make the evidence "more substantial than a comparison only against long textual messages". Your proposed main-task controls (removing the messages and exchanging them across questions) were concrete enough to run as stated. They extend those appendix controls to the main tasks and test directly whether the answers there depend on the messages. Your points and where each is answered:

Your pointWhat we addedResult
W1 / Q1: whether the main-task gains depend on what the intermediate roles sendMessage controls on the main tasks (messages removed, exchanged across questions, and self-written) and three probes of content that only the sender seesThe final role answers more questions correctly with the team's messages than under every control. Removing the messages costs 12 of 256 questions on ARC-Challenge and 13 and 21 of 300 on MedQA. In the routing probe, 51 of 64 decisions are correct with the matching message and 13 with the opposite review's message
W2 / Q2: the strongest standalone member, and timing of concise TextMASStandalone accuracy of all four backbones on all nine benchmarks, and batch-size-1 timing of the concise TextMAS of Table 9The VW team is above its strongest member in SmolVLM2/Qwen and LFM/Qwen, and above both members on AIME 2024, AIME 2025 and GPQA in Gemma/Qwen. Concise TextMAS is slower than VW in all six configurations (1.31× to 1.97×)
Q3: training length, inference length and message-processing costTables 7 and 13 evaluated at the rollout of the main experiments, message size and decoding time, and end-to-end timing at batch size 1At the rollout of the main experiments, the aligned message has the lowest KL in both directions and the visual span is above the soft prefix on both benchmarks. TextMAS takes 1.59 to 6.31 times as long as VW in the twelve cells timed

The comments below give the tables and settings. We are happy to clarify any point further during the discussion.

W1 / Q1: what the messages carry

Short answer. On the main tasks, the final role answers more questions correctly with the team's messages than with no message, with another question's messages, or with messages it writes for itself (continuation below). In the probes of this comment, the receiver's output follows content that only the sender has seen: in the routing probe, 51 of 64 decisions are correct with the matching message and 13 with the opposite review's message.

Receivers recover sender-specific content from the message. Each codec is trained so that its frozen backbone, reading a message, reproduces the state and next-token distribution it has after reading the text (Eq. 2), and the affine maps of Eq. 3 make one model's message readable by another. We test the result in settings where that content reaches the receiver only through the message:

TestWhat the receiver has besides the messageResult
Key–value retrieval (Table 11)the query, with the memory never given as text36.0 ± 4.6% correct among 17 candidates (chance 5.9%), against 34.0 ± 8.7% with a sender-generated text note
The same retrieval at the rollout of the main experiments (to be added to Table 11)as above36.3 ± 4.2% (chance 5.9%, three seeds with 100 queries each)
Factual binding (new): 64 scenes, each in two worlds with exchanged answer-determining bindings and with zero or four distractors, 256 itemsthe same question and answer options in both worlds, while only the sender observes the scenematching against exchanged-world message: +3.27 pp in the probability of the correct answer
Review polarity (new): 128 pairs of reviews of opposite polarity, 256 itemsthe same prompt for both reviews of a pairmatching against exchanged message: +5.56 pp, and the label log-odds follow the interpolation between the two messages of a pair (slope 2.75 nats)
Review routing (new): 32 reviews, each decided under two opposite public queue policies, 64 include/exclude decisionsthe public policy, while only the sender reads the review51 of 64 correct with the matching message, 13 with the opposite review's message and 32 with no message, against 64 with the review as text
Cross-text substitution (Table 13)its own text-conditioned distribution as the targetKL rises from 0.227 to 0.513 in one direction and from 1.864 to 6.344 in the other when another text's message is substituted
The same evaluation at the rollout of the main experiments (to be added to Table 13)as aboveKL 0.236 with the aligned message, 0.426 with another text's message, 2.217 with no message (Gemma → Qwen), and 4.135, 6.474 and 12.803 (Qwen → Gemma)
A message encoded from the initial prompt-boundary state alone (to be added to Table 13)as aboveKL 0.343 (Gemma → Qwen) and 8.565 (Qwen → Gemma), against 0.236 and 4.135 with the latent rollout of the main experiments

In the binding, polarity and routing tests and in the reconstruction test of Table 13, the receiver's output follows the sender's content: the matching message is ahead of the exchanged one in all three probes, and the aligned message has a lower KL than every substituted message in both directions. In the retrieval test the memory reaches the receiver only through the message. In the binding, polarity and routing tests the receiver's own input is identical across the conditions. Retrieval and the three new probes use the frozen codecs and alignment of the main experiments and the receiver's own vocabulary head, with no fitted readout. Table 13 uses the codecs of Appendix G. (Tables 11 and 13, Appendix H.3 and H.4.) We will add the three probes as Appendix H.7 and the new rows and column to Tables 11 and 13.

Initial state and rollout. A role's message is written from its hidden states after the prompt: the boundary state and the latent steps that follow it (see our reply to Q3). These states are computed after the role has read the task and every message it received. Each message therefore contains that role's processing of its input, and later roles build on earlier ones through the model.

W1 / Q1 (continued): message controls on the main tasks

Result. In all three settings below, the final role answers more questions correctly with the team's messages than under every control run in that setting: no message, messages of another question, or messages it writes for itself.

Main-task message controls. We intervened on the messages the final role receives and kept everything else fixed within each comparison: the team and its role order, the questions, and the role prompts, sampling, caps, codecs, maps and rollout of the main experiments.

ARC-Challenge, Qwen3-VL-2B/Gemma-3-4B team with Gemma-3-4B as final role:

Messages received by the final roleARC-Challenge, correct of 256
The team's messages214
No message (image span without a decoded residual, as in Table 13)202
Messages the final-role model writes for itself (every intermediate role run by Gemma-3-4B)202

Setting: extends the mechanism experiments of the paper (Tables 7, 11 and 13, which use the Qwen3-VL-2B/Gemma-3-4B pair) to the main task, on 256 randomly drawn ARC-Challenge test questions with seed 42. (To be added as Table 26, Appendix H.9.)

MedQA, correct of 300, two teams with Gemma-3-4B as final role:

Messages received by the final roleLFM / Gemma, roles of Table 2Qwen3-VL / Gemma, as on ARC-Challenge
The team's messages157166
No message144145
Messages of another question with a different answer151156
Messages the final-role model writes for itself149153

Setting: main codecs and maps, the rollout, prompts, sampling and caps of the main experiments, seed 42. (To be added as Table 27, Appendix H.9.)

Removing the messages costs 12 questions on ARC-Challenge and 13 and 21 questions on MedQA, and messages from another question or messages the final model writes for itself also lower accuracy below the team's messages. The final answer depends on what the other model computed for this question. A message the final-role model writes for itself re-encodes the same task without the other model, and it stays at the no-message level on ARC-Challenge (202) and below the team's messages on MedQA (149 against 157, and 153 against 166).

On the main tasks the answers change with the channel. With the team, roles, role prompts of Zou et al. (2025), questions and caps held fixed, exchanging only the channel changes the answers: text messages against VW messages in Figure 4 (macro accuracy rises in five of six teams, and accuracy rises by 6.0 pp on average over the 54 configuration-task cells), and the same intermediate trajectories delivered through the native visual span against a learned text prefix in Table 7 (75.8 against 60.2 on GSM8K and 34.4 against 27.0 on MedQA, 256 questions each).

Comments on the paper

2 comments
AuthorsOpenReview ↗

A narrated video of our responses

Dear AC and reviewers,

Thank you again for your reviews. Our responses cover many new experiments, and reading all of them as text takes time within a short discussion period. We therefore prepared a narrated video that walks through our responses to all three reviewers in under 10 minutes (supplementary material, rebuttal_video.mp4).

The video is grouped by reviewer, so you can go directly to the part you are most interested in:

  • 0:00, a one-minute recap of the paper
  • 1:15, Part 1, Reviewer XEX5
  • 3:51, Part 2, Reviewer WfdA
  • 6:22, Part 3, Reviewer cUNP

We hope you will take a look.

Best regards,

The Authors

AuthorsOpenReview ↗

Summary of our responses

Dear AC and reviewers,

Thank you again for your reviews. As the discussion period closes soon, here is a short summary of our responses. We also made two narrated videos (supplementary material). rebuttal_video.mp4 walks through our responses with one part of under three minutes per reviewer, and difference_animation.mp4 shows step by step how VW differs from existing methods.

Point raisedResult in our repliesVideo
What the messages carry (Reviewer XEX5)In all three settings of the main-task message controls, the final role answers more questions correctly with the team's messages than under every control run in that setting: no message, messages of another question, or messages it writes for itself. In the routing probe, 51 of 64 decisions are correct with the matching message and 13 with the opposite review's messagerebuttal_video.mp4, Part 1 (from 1:15)
Cost of communication (Reviewer XEX5)TextMAS takes 1.59 to 6.31 times as long as VW in the twelve cells timed, and concise TextMAS is slower than VW in all six configurations (1.31× to 1.97×)Part 1
Comparison with latent channels (Reviewers WfdA and cUNP)Four added channels: Activation Communication (ICML 2025), Cache-to-Cache (ICLR 2026), StateBridge (COLM 2026) and XBridge (NeurIPS 2026). VW is the most accurate channel on both benchmarks of the extended Table 6 and in all seven rows of the comparison with StateBridge and XBridgePart 2 (from 3:51)
More backbones, more agents, longer interaction (Reviewer WfdA)With a fifth backbone, InternVL3.5-2B, and with chains of six and eight roles, VW is more accurate than TextMAS in all twelve cells. The receiver's input length is the same for one accumulated message and for fifteenPart 2
How VW differs from existing methods (Reviewer cUNP)When a model joins a team, Cache-to-Cache trains new fusers between the newcomer and every existing model, and Mixture of Thoughts retrains its shared space with all models. VW trains one codec with the newcomer alone and fits two closed-form maps to the fixed reference modeldifference_animation.mp4, and Part 3 (from 6:22)
Why the visual interface works (Reviewer cUNP)With the same per-model codec training, the image-token span is 15.6 and 7.4 points more accurate than a learned text prefix on GSM8K and MedQAPart 3

We would be grateful if you could let us know whether our responses resolve your concerns, so that we can still answer any remaining question before the discussion closes. If they do, we would be grateful if you would consider updating your score.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.