- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Vision Wormhole, which uses the image-token interface of frozen vision-language models to transform one model’s hidden states into continuous context that another model can consume. Each model learns its codec through local self-distillation, while shared anchor texts and affine maps connect the codecs through a reference space.
Critical strengths
- An interesting interface design. Repurposing VLMs’ existing visual conditioning interfaces for cross-model hidden-state communication offers a new implementation path for heterogeneous collaboration.
- Frozen backbone models. Training is confined to codecs and alignment maps, enabling the reuse of existing pretrained models.
- Modular integration. Each model has its own codec and connects to multiple partners through a reference space, with the component count scaling linearly with the number of models.
Critical weaknesses
Overall, using the visual input interface of frozen VLMs for cross-model communication is an interesting idea. However, the paper does not clearly establish its technical contributions or distinguish them from existing latent communication methods, and parts of the presentation lack a coherent logical flow.
-
The substantive differences from existing methods need clarification. Hidden-state transfer, communication without intermediate text generation, and heterogeneous model collaboration have already been explored. The authors should directly compare VW with Cache-to-Cache, and shared-space approaches, clarifying whether the novelty lies in the receiver interface, the training procedure, or their combination.
-
The motivation for the visual interface contains a logical gap. A VLM’s ability to understand rendered text does not directly establish that its image-token interface is suitable for receiving another model’s hidden states. Building on the existing soft-prefix ablation, the authors should explain why the visual interface works and clarify its relationship to dynamic soft prompting.
-
The benefits of different design choices should be attributed separately. The component count primarily follows from the hub-and-spoke reference-space architecture rather than the visual interface itself. The authors should distinguish the contributions of visual conditioning and shared-space integration, while clarifying the scope of independent training and the requirements for shared-anchor calibration.
-
The related work needs a clearer organizational logic. The “Multi-agent coordination” subsection moves through role specialization, model selection, memory, and workflow optimization without clearly connecting these topics to the communication problem studied here. It should be organized around the research gap rather than ending abruptly with a description of the experimental comparison.
Questions for authors
My questions are the same as the concerns raised above.
12 comments
W1: technical contribution relative to Cache-to-Cache and shared-space methods
The novelty lies in the combination. VW combines a receiver interface (the native image-token span, with the receiver unmodified), a training procedure (each codec is self-distilled on its own frozen backbone, with no other model in the loop), and a calibration (closed-form affine maps fitted on shared anchor texts) that makes independently trained codecs usable together. The table places this combination next to Cache-to-Cache and the shared-space methods:
| Method | Transmitted | Entry point in the receiver | Trained, and per what unit | Training data and loss | Models it can connect (families evaluated) | Trained modules for N models |
|---|---|---|---|---|---|---|
| Cache-to-Cache | Sharer's KV cache | Receiver KV cache, gated fusion | One fuser per ordered sharer–receiver pair, LLMs frozen | 500K instruction conversations per fuser, next-token loss | Any pair, given a fuser trained for each ordered pair (Qwen, Llama, Gemma) | N(N−1) fusers |
| KVComm | KV pairs of selected sender layers | Receiver attention at the same layer index | Nothing (layers calibrated per model pair) | None | Only models with one base architecture: identical models or fine-tunes of one base | 0 |
| LatentMAS | All-layer KV caches with latent thoughts | Receiver KV cache | Nothing | None | Only agents that share one backbone | 0 |
| CIPHER | Expected token embeddings | Input embeddings | Nothing | None | Only models that share one tokenizer | 0 |
| Activation communication | One activation vector | Residual stream at the last token | Nothing by default (optional map per model pair) | None by default | Qwen-2.5, Gemma-2, LLaMA-3 | 0 (N(N−1) maps if used) |
| Mixture of Thoughts | Expert hidden states projected to a shared space | Interaction layers of the primary expert | Router and interaction layers, jointly for a fixed pool | Train splits of five evaluated benchmarks, language-modeling loss | A fixed pool, retrained when a model is added (Mistral and Llama-3 variants) | Joint, for the fixed pool |
| ThoughtComm | Autoencoder codes of agents' last-token states | Prefix vectors on the token embeddings | Autoencoder and adapter over a fixed agent set | 500 examples of the evaluated tasks, reconstruction and likelihood losses | A fixed agent set (one base model per run) | Joint, for the fixed set |
| Vision Wormhole | Fixed-size message (1026 × 512) from the sender's latent rollout | Native image-token span, receiver unmodified | One codec per model, self-distilled on its own frozen backbone (Eq. 2), and closed-form affine maps (Eq. 3) | Unlabeled anchor texts, the model's own text-conditioned state and next-token distribution as targets, no benchmark data | Any vision-language model that reads images as tokens in its input sequence, added one at a time (Qwen3-VL, Gemma-3, SmolVLM2, LFM2.5-VL, and InternVL3.5 (new)) | N codecs, 2(N−1) maps |
VW's codecs are trained one model at a time, with no other model in the loop, and a team of N models needs N codecs and 2(N−1) closed-form maps. Cache-to-Cache and Mixture of Thoughts connect different models through modules trained per pair or per pool. The Cache-to-Cache paper itself names scaling to more models at O(N) training cost as an open problem. Its appendix reports an initial exploration of a shared space, in which the projectors and fusers of all models are trained jointly and each sharer–receiver pair keeps its own gate. As published, KVComm applies only to models with one base architecture, LatentMAS only to agents that share one backbone, and CIPHER only to models that share one tokenizer, so none of the three connects models from different families without modification. Entries for the other methods follow each method's own paper. The last column counts the trained modules the design implies for a team of N models. We will add this table as Table 17 (Appendix A.3) and the same comparison for nine concurrent 2026 papers as Table 30 (Appendix H.12).
W1 (continued): direct comparisons with latent channels, and a test for each part
Direct comparison. In the SmolVLM2/Qwen team, VW is above Cache-to-Cache on both benchmarks: 93 against 71 correct of 128 on GSM8K, and 67 against 48 of 164 on HumanEval+. End-to-end comparisons with channels that use text generation or model internals are in Appendix D (OCR relay) and Appendix F (CIPHER, LatentMAS-Hybrid). XBridge (NeurIPS 2026, a learned cross-attention bridge trained per model pair) and StateBridge (COLM 2026, a hidden-state prefix channel) are run in that comparison as well (second table below), and the extended literature review of the revision will include both. We also ran Cache-to-Cache directly on SmolVLM2/Qwen. With two models, its pairwise design is the complete method. Its released fusers cover text-only model pairs, so its two fusers are trained on the two models' language decoders with the released code and objective, each on 50K OpenHermes-2.5 conversations (about 17 times the anchor texts of a VW codec, see our reply to Reviewer WfdA, W1), and run in the four-role chain (accuracy % with correct counts, on the first 128 GSM8K questions and all 164 HumanEval+ problems):
| Channel | GSM8K | HumanEval+ |
|---|---|---|
| Vision Wormhole | 72.7 (93) | 40.9 (67) |
| Cache-to-Cache (ICLR 2026) | 55.5 (71) | 29.3 (48) |
| TextMAS | 64.8 (83) | 31.1 (51) |
(To be added as Table 29, Appendix H.11.)
Two channels published in 2026: StateBridge and XBridge. Every channel in the table below uses the same team, role assignment, role prompts of Zou et al. (2025), evaluation questions, sampling, and final-answer cap (Table 3). TextMAS and VW are the main-experiment predictions restricted to the first 128 GSM8K test questions (the question set of Table 9) and the first 128 ARC-Challenge test questions, plus all 164 HumanEval+ problems, and the first 128 MedQA questions for SmolVLM2/Gemma, and the added channels are run on the same questions. Accuracy (%, correct out of 128, or of 164 for HumanEval+):
| Configuration | Benchmark | TextMAS | StateBridge (COLM 2026) | XBridge (NeurIPS 2026) | VW |
|---|---|---|---|---|---|
| SmolVLM2 / Gemma | GSM8K | 65.6 (84) | 73.4 (94) | 78.9 (101) | 82.8 (106) |
| ARC-Challenge | 71.9 (92) | 69.5 (89) | 78.1 (100) | 83.6 (107) | |
| MedQA | 43.8 (56) | 39.8 (51) | 50.0 (64) | 53.9 (69) | |
| HumanEval+ | 32.9 (54) | 43.9 (72) | 43.3 (71) | 59.1 (97) | |
| SmolVLM2 / Qwen | GSM8K | 64.8 (83) | 59.4 (76) | 13.3 (17) | 72.7 (93) |
| ARC-Challenge | 71.1 (91) | 75.0 (96) | 69.5 (89) | 81.2 (104) | |
| HumanEval+ | 31.1 (51) | 31.7 (52) | 0.6 (1) | 40.9 (67) |
VW is the most accurate channel in all seven rows. (To be added as Table 19, Appendix F. How each channel was run is described in our reply to Reviewer WfdA, W1.)
A test for each part. The three parts are a continuous-input pathway that every VLM already has, a training signal that comes entirely from the receiver's own model so that codecs are trained independently, and shared-anchor calibration that makes independently trained codecs composable. Each has its own test:
| Part | Test | Result |
|---|---|---|
| Entry point | Visual span against a learned text prefix, matched architectures and training, same intermediate trajectories (Table 7) | +15.6 pp on GSM8K, +7.4 pp on MedQA, and +14.4 and +9.8 pp at the rollout of the main experiments (to be added) |
| Calibration | Held-out reconstruction KL: aligned / cross-text / shuffled-anchor (Table 13, Appendix G codecs) | 0.227 / 0.513 / 0.464 and 1.864 / 6.344 / 5.408 in the two directions |
| Composition | Each backbone's codec is one set of trained weights, used unchanged in every configuration that contains the backbone | the six configurations of Figure 4, which include a four-backbone pool, a parallel topology with the Gemma-3-4B codec (Table 10), and (new) a five-model team that adds InternVL3.5-2B with nothing fitted on the team: 75.0 against 60.2 for TextMAS on GSM8K, and 73.4 against 69.5 on ARC-Challenge, on the first 128 questions (to be added as Table 21) |
(Table 7, Appendix G; Table 13, Appendix H.4; Table 10, Appendix H.2; Table 21, Appendix H.8)
Overview of our response
Thank you for recognizing the receiver interface, the frozen backbones, and the modular integration, and for noting that repurposing the visual conditioning interface "offers a new implementation path for heterogeneous collaboration". Your four points were precise about what to clarify and where, which made them straightforward to act on. The request to attribute visual conditioning and the reference space separately was especially useful, and the revision will state that attribution explicitly. Your points and where each is answered:
The comments below give the tables and settings. We are happy to clarify any point further during the discussion.