- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
LatentCollab communicates between frozen vision language models by translating a sender’s hidden states into continuous messages inserted into a receiver’s input. The translator is trained on 25K LLaVA examples. The paper evaluates three model pairs across seven benchmarks and reports improvements over receiver only inference, text communication, and two adapted cache communication baselines.
Critical strengths
The design is straightforward and keeps both backbone models frozen. The experiments cover multiple tasks and include communication across model families. The reported improvements over receiver only inference are consistent across all 21 task and model combinations.
Critical weaknesses
The method follows the familiar design of translating hidden states into another model’s input, so the main question is what the translator learns through training. My central concern is that the translator acts primarily as a task adapter fitted to LLaVA image QA, instead of as the 'translator'. LLaVA provides direct image QA supervision, and the paper omits a receiver SFT baseline trained on the same 25K examples. The two main communication baselines are also inadequately documented.
C2C-VLM falls below receiver only in 12 of 21 comparisons, and DenseAlign-VLM does so in 14. C2C explicitly states:
“Multi-LLM systems experience performance degradation when a much weaker Sharer provides noisy information to a stronger Receiver. This limitation is shared by both T2T and C2C, as Sharer semantic quality directly impacts Receiver performance.”
All three configurations in this paper use smaller senders, making a stronger sender control particularly relevant. This is one explict example showing why the experiment setting is so weird to me. There are also many, for example, supplementary material, lines 156–160, states:
“Scope of the cache-transfer descriptions. The main paper denotes the evaluated KV-cache baselines as C2C-VLM and DenseAlign-VLM. The descriptions below summarize the mechanisms in the original C2C and DenseAlign papers; they are not a specification of project-specific VLM adaptation choices”.
I am not quite sure the meaning of saying these unless it was unintentionally added by GPT
The cited papers do not define these VLM variants. These are the authors’ own adaptations, and their implementation and training settings are completely missing. If they follow the original text QA training settings, LC receives additional image QA supervision through LLaVA, making the comparison unfair.
Direct receiver fine-tuning also deserves serious consideration. ConvCue, Table 3, reports ScienceQA accuracy of 95.74% for Qwen3-VL-4B under Direct SFT using LLaVA pretraining data and the visual instruction data used in LLaVA-UHD, compared with LC’s 91.52%. OnceSelect, Table 3, reports an MME total of 2448.70 after fine-tuning Qwen3-VL-8B on 93.7K selected LLaVA examples, compared with LC’s 2428.56. These experiments use different training budgets, but they establish the importance of a matched SFT control, especially since LC adds a sender model at inference.
Questions for authors
One of many tests I would like to see is evaluating the existing LC adapter on MATH-500 and ARC-C without retraining. Following the original baseline papers’ evaluation protocols and including their reported results as reference points would also give the comparison a concrete foundation.
I hope the authors could control variables when conducting rebuttal experiments like changing only one aspect at a time relative to the existing literature and clearly define all relevant details. Currently, the paper omits a significant number of details, which makes the experimental results unreliable. I am not sure if ICLR permits this, but I would like to see the code for the proposed method and the baseline experiments, or at least a detailed description of the experimental setup used in the code.
10 comments
1.3 Inference-Time Input Perturbations on the Frozen Translator (Qwen3-VL-2B -> 4B)
To directly verify what the trained conveys at test time without retraining, we froze and applied 7 inference-time interventions: (1) Mismatched (, ), (2) Temporal Permutation (shuffling the step order of ), (3) Zero Translator Input (), (4) Zero Translator Output ( unmasked), (5) Gaussian Noise (moment-matched per-sample ), (6) Sender No Image, and (7) Sender Mismatched Question ( while receiver keeps ).
| Test-Time Condition (No Retraining) | GQA | TextVQA | POPE | VizWiz | SQA | MME | MMB |
|---|---|---|---|---|---|---|---|
| Receiver-Only | 62.80 | 81.84 | 89.81 | 41.58 | 80.43 | 2342.50 | 80.36 |
| Mismatched (Cross-Sample) | 60.54 | 70.65 | 85.84 | 43.23 | 84.97 | 2245.44 | 82.72 |
| Temporal Permutation | 60.34 | 70.47 | 86.34 | 41.19 | 82.74 | 2270.45 | 83.05 |
| Zero Translator Input | 56.71 | 66.88 | 85.61 | 39.62 | 83.64 | 2164.76 | 80.41 |
| Zero Translator Output | 55.13 | 63.46 | 84.85 | 35.05 | 83.69 | 2211.71 | 80.81 |
| Gaussian Noise (Moment-Matched) | 59.54 | 68.17 | 83.10 | 41.81 | 84.06 | 2118.56 | 82.30 |
| Sender No Image | 60.76 | 71.51 | 86.61 | 42.58 | 83.78 | 2245.21 | 82.72 |
| Sender Mismatched Question | 60.26 | 70.76 | 85.74 | 42.88 | 86.15 | 2197.83 | 82.67 |
| LatentCollab (Matched) | 67.24 | 85.38 | 91.73 | 48.88 | 90.63 | 2399.15 | 87.91 |
Every intervention causes sharp drops across all 7 benchmarks, falling well below Receiver-Only on GQA, TextVQA, POPE, and MME even when per-sample activation moments (Gaussian Noise) or single modalities (Sender No Image / Sender Mismatched Question) are preserved. This proves actively translates sample-specific, temporally ordered, jointly image- and question-conditioned sender computation.
2. Stronger-Sender Control (4B -> 2B) & Rationale for Smaller/Cross-Family Senders
To directly test the C2C finding you cited regarding sender–receiver relative strength, we evaluated the reversed stronger-sender configuration Qwen3-VL-4B -> Qwen3-VL-2B under the exact same 25K training set and evaluation protocol:
Method (Qwen3-VL-4B -> Qwen3-VL-2B) | GQA | TextVQA | POPE | VizWiz | SQA | MME | MMB |
|---|---|---|---|---|---|---|---|
| Receiver-Only (2B) | 58.25 | 77.50 | 84.81 | 44.49 | 79.52 | 2030.19 | 75.78 |
| Text Communication | 56.47 | 65.02 | 84.06 | 45.28 | 80.68 | 2042.43 | 76.64 |
| C2C-VLM | 57.20 | 76.40 | 85.27 | 45.91 | 85.18 | 2071.04 | 77.57 |
| DenseAlign-VLM | 57.29 | 78.92 | 84.47 | 45.19 | 85.50 | 2073.23 | 77.15 |
| LatentCollab (Ours) | 59.82 | 79.92 | 85.50 | 45.40 | 86.22 | 2216.27 | 79.78 |
- Empirical Validation of Our KV-Cache Adaptations: Consistent with the C2C limitation you quoted, reversing to a stronger sender (
4B -> 2B) immediately turns bothC2C-VLMandDenseAlign-VLMpositive on 5 of 7 benchmarks overReceiver-Only (2B). This directly confirms that our VLM cache-transfer implementations work as intended and faithfully reproduce the original papers' regime sensitivity. LatentCollabLeads in Both Regimes: Even in4B -> 2B,LatentCollaboutperformsReceiver-Only (2B)andText Communicationon all 7/7 benchmarks and beats both KV-cache baselines on 6 of 7.- Why Smaller / Cross-Family Senders Matter: Pairing a lightweight auxiliary VLM or heterogeneous vision encoder (
InternVL3.5-2B) with a primary VLM reflects practical low-overhead collaboration without requiring an expensive larger sender. Whereas full-layer KV-cache injection directly perturbs a stronger receiver's internal layer-wise attention when the sender is smaller,LatentCollabinjects compact states at the input sequence regularized by confidence-gated receiver trust (Eqs. 13–14)—preventing weaker-sender noise from overriding confident receiver predictions while still extracting complementary visual cues.
We thank Reviewer rEXv for the thorough review and for recognizing the simplicity of keeping both VLM backbones frozen and our consistent improvements across all 21 task–model combinations. Your main concerns center on (1) whether Tϕ acts as a sample-specific translator rather than a 25K LLaVA task adapter, (2) why KV-cache baselines degrade under smaller senders and how
C2C-VLM/DenseAlign-VLMwere adapted and supervised, and (3) modality scope (MATH-500/ARC-C), controlled variable isolation, and implementation transparency. Below, we address each point in detail.1. Translator vs. Task Adapter: Single-Variable 25K Controls, Literature Comparison, and Test-Time Ablations
1.1 Controlled Single-Variable Chain on the Exact Same 25K LLaVA Set (
Qwen3-VL-2B -> 4B)Following your suggestion to isolate one variable at a time, we trained three supervised controls on the exact same deduplicated 25K LLaVA subset used by
LatentCollab:SFT 25K(Isolating 25K VQA supervision on the receiver): Applies standard SFT toQwen3-VL-4Bon the same 25K image–question–answer pairs without sender input.Soft Prompt 25K(Isolating parameter-, architecture-, and slot-matched adapter tuning): Replaces sample-specific sender rollouts with a fixed, sample-independent input and uses a parameter- and architecture-matched MLP module to generate K=32 sample-shared continuous prompt embeddings at the same post-visual injection slot (Eq. 7) as Tϕ.Text Comm 25K SFT(Isolating supervised adaptation over discrete sender messages): Applies standard SFT to the receiver conditioned on(image + question + sender text message)generated by the frozen 2B sender across the 25K training examples.Qwen3-VL-2B -> 4B)Soft Prompt 25Kimproves multiple-choice formatting (SQA, MMB), an input-agnostic adapter severely degrades fine-grained visual perception, OCR, and hallucination resistance (GQA, TextVQA, POPE, MME) relative toReceiver-Only.LatentCollaboutperformsSoft Prompt 25Kacross all 7 benchmarks.LatentCollaboutperformsSFT 25KandText Comm 25K SFTon all 7 benchmarks without modifying backbone weights, while maintaining lower end-to-end latency than all communication baselines (Main Paper Fig. 4).1.2 Data Scale, Decontamination, and Tuned Scope vs.
ConvCueandOnceSelectDirect raw-score comparisons against the numbers reported in
ConvCueandOnceSelectconflate three uncontrolled variables—data volume/selection, benchmark decontamination, and backbone modification:ConvCue(concurrent work) performs two-stage training on ~1.22M examples (558K LLaVA-Pretrain + 665K LLaVA-UHD/v1.5-mix, ~49x our 25K budget), updating the vision encoder, projector, and full LLM;OnceSelectupdates the multimodal projector andQwen3-VL-8BLoRA weights on 93.7K high-utility examples (3.75x our budget) actively selected from 625K multimodal LLaVA-v1.5-mix via CLIP clustering and uncertainty-weighted k-DPP. By contrast,LatentCollabfreezes both VLMs and trains only a ~12M two-layer MLP on 25K uniformly sampled examples.ConvCuenorOnceSelectreports downstream benchmark decontamination against evaluation splits, retaining LLaVA-665K subsets (e.g., GQA, VQAv2, OCR-VQA, TextCaps/OpenImages, COCO/VG) whose task distributions and image pools overlap with standard benchmarks. Prior to sampling our 25K subset, we executed strict image- and sample-level deduplication and cross-split filtering against all 7 evaluation splits. Under a strictly controlled single-variable comparison on this decontaminated 25K set (Sec. 1.1 table above), standard receiver SFT (SFT 25K) reaches84.45%on ScienceQA and2362.79on MME, whereasLatentCollabreaches90.63%and2399.15under the exact same2B -> 4Bconfiguration.