acceptodds
Withdrawn from ICLR 2027

LatentCollab: Vision-Language Agent Collaboration Beyond Verbalization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language agents extend vision-language models from isolated multimodal reasoning toward collaborative intelligence, but existing systems primarily exchange natural-language messages. This forces task-conditioned multimodal computation through a verbalization bottleneck that may omit fine-grained visual evidence, while KV-cache communication incurs substantial memory and communication overhead. We introduce LatentCollab, a latent collaboration framework that enables vision-language agents to exchange compact continuous messages without intermediate text generation. The sender recursively constructs a trajectory of final-layer hidden states. A lightweight translator, the only trainable component, maps the trajectory into a receiver-compatible continuous message, which is inserted into the receiver's input-embedding sequence after its visual tokens. Across seven benchmarks and three cross-scale or cross-family sender-receiver settings, LatentCollab outperforms receiver-only and text communication on all 21 pairs, and both KV-cache baselines on 18, with lower latency than all evaluated communication baselines. These results establish latent collaboration as an effective and efficient paradigm for vision-language agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Model, Multimodal Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rEXvOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

LatentCollab communicates between frozen vision language models by translating a sender’s hidden states into continuous messages inserted into a receiver’s input. The translator is trained on 25K LLaVA examples. The paper evaluates three model pairs across seven benchmarks and reports improvements over receiver only inference, text communication, and two adapted cache communication baselines.

Critical strengths

The design is straightforward and keeps both backbone models frozen. The experiments cover multiple tasks and include communication across model families. The reported improvements over receiver only inference are consistent across all 21 task and model combinations.

Critical weaknesses

The method follows the familiar design of translating hidden states into another model’s input, so the main question is what the translator learns through training. My central concern is that the translator acts primarily as a task adapter fitted to LLaVA image QA, instead of as the 'translator'. LLaVA provides direct image QA supervision, and the paper omits a receiver SFT baseline trained on the same 25K examples. The two main communication baselines are also inadequately documented.

C2C-VLM falls below receiver only in 12 of 21 comparisons, and DenseAlign-VLM does so in 14. C2C explicitly states:

“Multi-LLM systems experience performance degradation when a much weaker Sharer provides noisy information to a stronger Receiver. This limitation is shared by both T2T and C2C, as Sharer semantic quality directly impacts Receiver performance.”

All three configurations in this paper use smaller senders, making a stronger sender control particularly relevant. This is one explict example showing why the experiment setting is so weird to me. There are also many, for example, supplementary material, lines 156–160, states:

“Scope of the cache-transfer descriptions. The main paper denotes the evaluated KV-cache baselines as C2C-VLM and DenseAlign-VLM. The descriptions below summarize the mechanisms in the original C2C and DenseAlign papers; they are not a specification of project-specific VLM adaptation choices”.

I am not quite sure the meaning of saying these unless it was unintentionally added by GPT

The cited papers do not define these VLM variants. These are the authors’ own adaptations, and their implementation and training settings are completely missing. If they follow the original text QA training settings, LC receives additional image QA supervision through LLaVA, making the comparison unfair.

Direct receiver fine-tuning also deserves serious consideration. ConvCue, Table 3, reports ScienceQA accuracy of 95.74% for Qwen3-VL-4B under Direct SFT using LLaVA pretraining data and the visual instruction data used in LLaVA-UHD, compared with LC’s 91.52%. OnceSelect, Table 3, reports an MME total of 2448.70 after fine-tuning Qwen3-VL-8B on 93.7K selected LLaVA examples, compared with LC’s 2428.56. These experiments use different training budgets, but they establish the importance of a matched SFT control, especially since LC adds a sender model at inference.

Questions for authors

One of many tests I would like to see is evaluating the existing LC adapter on MATH-500 and ARC-C without retraining. Following the original baseline papers’ evaluation protocols and including their reported results as reference points would also give the comparison a concrete foundation.

I hope the authors could control variables when conducting rebuttal experiments like changing only one aspect at a time relative to the existing literature and clearly define all relevant details. Currently, the paper omits a significant number of details, which makes the experimental results unreliable. I am not sure if ICLR permits this, but I would like to see the code for the proposed method and the baseline experiments, or at least a detailed description of the experimental setup used in the code.

10 comments

We thank Reviewer rEXv for the thorough review and for recognizing the simplicity of keeping both VLM backbones frozen and our consistent improvements across all 21 task–model combinations. Your main concerns center on (1) whether TϕT_\phi acts as a sample-specific translator rather than a 25K LLaVA task adapter, (2) why KV-cache baselines degrade under smaller senders and how C2C-VLM / DenseAlign-VLM were adapted and supervised, and (3) modality scope (MATH-500 / ARC-C), controlled variable isolation, and implementation transparency. Below, we address each point in detail.


1. Translator vs. Task Adapter: Single-Variable 25K Controls, Literature Comparison, and Test-Time Ablations

1.1 Controlled Single-Variable Chain on the Exact Same 25K LLaVA Set (Qwen3-VL-2B -> 4B)

Following your suggestion to isolate one variable at a time, we trained three supervised controls on the exact same deduplicated 25K LLaVA subset used by LatentCollab:

  • SFT 25K (Isolating 25K VQA supervision on the receiver): Applies standard SFT to Qwen3-VL-4B on the same 25K image–question–answer pairs without sender input.
  • Soft Prompt 25K (Isolating parameter-, architecture-, and slot-matched adapter tuning): Replaces sample-specific sender rollouts with a fixed, sample-independent input and uses a parameter- and architecture-matched MLP module to generate K=32K=32 sample-shared continuous prompt embeddings at the same post-visual injection slot (Eq. 7) as TϕT_\phi.
  • Text Comm 25K SFT (Isolating supervised adaptation over discrete sender messages): Applies standard SFT to the receiver conditioned on (image + question + sender text message) generated by the frozen 2B sender across the 25K training examples.
Controlled Chain (Qwen3-VL-2B -> 4B)GQATextVQAPOPEVizWizSQAMMEMMB
1. Receiver-Only (Frozen 4B, 0 supervision)62.8081.8489.8141.5880.432342.5080.36
2. SFT 25K (+25K receiver SFT, no sender)62.8582.6190.2342.0984.452362.7981.14
3. Soft Prompt 25K (matched MLP on static input, no sender)57.2876.4185.4546.6188.892204.2686.54
4. Text Communication (Zero-shot frozen sender text)59.3270.8391.0745.3585.082256.1783.69
5. Text Comm 25K SFT (+sender text + 25K receiver SFT)61.1579.6385.7541.0086.332279.4481.37
6. LatentCollab (Frozen 2B + 12M TϕT_\phi on 25K)67.2485.3891.7348.8890.632399.1587.91
  • Not a static LLaVA task adapter: While Soft Prompt 25K improves multiple-choice formatting (SQA, MMB), an input-agnostic adapter severely degrades fine-grained visual perception, OCR, and hallucination resistance (GQA, TextVQA, POPE, MME) relative to Receiver-Only. LatentCollab outperforms Soft Prompt 25K across all 7 benchmarks.
  • Superior to matched receiver SFT while preserving modularity: LatentCollab outperforms SFT 25K and Text Comm 25K SFT on all 7 benchmarks without modifying backbone weights, while maintaining lower end-to-end latency than all communication baselines (Main Paper Fig. 4).

1.2 Data Scale, Decontamination, and Tuned Scope vs. ConvCue and OnceSelect

Direct raw-score comparisons against the numbers reported in ConvCue and OnceSelect conflate three uncontrolled variables—data volume/selection, benchmark decontamination, and backbone modification:

  1. Data Scale & Active Curation: As documented in their respective papers, ConvCue (concurrent work) performs two-stage training on ~1.22M examples (558K LLaVA-Pretrain + 665K LLaVA-UHD/v1.5-mix, ~49x our 25K budget), updating the vision encoder, projector, and full LLM; OnceSelect updates the multimodal projector and Qwen3-VL-8B LoRA weights on 93.7K high-utility examples (3.75x our budget) actively selected from 625K multimodal LLaVA-v1.5-mix via CLIP clustering and uncertainty-weighted kk-DPP. By contrast, LatentCollab freezes both VLMs and trains only a ~12M two-layer MLP on 25K uniformly sampled examples.
  2. Strict Zero-Overlap Decontamination: Neither ConvCue nor OnceSelect reports downstream benchmark decontamination against evaluation splits, retaining LLaVA-665K subsets (e.g., GQA, VQAv2, OCR-VQA, TextCaps/OpenImages, COCO/VG) whose task distributions and image pools overlap with standard benchmarks. Prior to sampling our 25K subset, we executed strict image- and sample-level deduplication and cross-split filtering against all 7 evaluation splits. Under a strictly controlled single-variable comparison on this decontaminated 25K set (Sec. 1.1 table above), standard receiver SFT (SFT 25K) reaches 84.45% on ScienceQA and 2362.79 on MME, whereas LatentCollab reaches 90.63% and 2399.15 under the exact same 2B -> 4B configuration.

1.3 Inference-Time Input Perturbations on the Frozen Translator (Qwen3-VL-2B -> 4B)

To directly verify what the trained TϕT_\phi conveys at test time without retraining, we froze TϕT_\phi and applied 7 inference-time interventions: (1) Mismatched (Tϕ(Hπ(i)s)T_\phi(H_{\pi(i)}^s), π(i)≠i\pi(i) \neq i), (2) Temporal Permutation (shuffling the K=32K=32 step order of HisH_i^s), (3) Zero Translator Input (Tϕ(0)T_\phi(0)), (4) Zero Translator Output (Z=0Z=0 unmasked), (5) Gaussian Noise (moment-matched per-sample μi+σiϵ\mu_i + \sigma_i \epsilon), (6) Sender No Image, and (7) Sender Mismatched Question (Hs(Ii,Qj)H^s(I_i, Q_j) while receiver keeps Ii,QiI_i, Q_i).

Test-Time Condition (No Retraining)GQATextVQAPOPEVizWizSQAMMEMMB
Receiver-Only62.8081.8489.8141.5880.432342.5080.36
Mismatched (Cross-Sample)60.5470.6585.8443.2384.972245.4482.72
Temporal Permutation60.3470.4786.3441.1982.742270.4583.05
Zero Translator Input56.7166.8885.6139.6283.642164.7680.41
Zero Translator Output55.1363.4684.8535.0583.692211.7180.81
Gaussian Noise (Moment-Matched)59.5468.1783.1041.8184.062118.5682.30
Sender No Image60.7671.5186.6142.5883.782245.2182.72
Sender Mismatched Question60.2670.7685.7442.8886.152197.8382.67
LatentCollab (Matched)67.2485.3891.7348.8890.632399.1587.91

Every intervention causes sharp drops across all 7 benchmarks, falling well below Receiver-Only on GQA, TextVQA, POPE, and MME even when per-sample activation moments (Gaussian Noise) or single modalities (Sender No Image / Sender Mismatched Question) are preserved. This proves TϕT_\phi actively translates sample-specific, temporally ordered, jointly image- and question-conditioned sender computation.

2. Stronger-Sender Control (4B -> 2B) & Rationale for Smaller/Cross-Family Senders

To directly test the C2C finding you cited regarding sender–receiver relative strength, we evaluated the reversed stronger-sender configuration Qwen3-VL-4B -> Qwen3-VL-2B under the exact same 25K training set and evaluation protocol:

Method (Qwen3-VL-4B -> Qwen3-VL-2B)GQATextVQAPOPEVizWizSQAMMEMMB
Receiver-Only (2B)58.2577.5084.8144.4979.522030.1975.78
Text Communication56.4765.0284.0645.2880.682042.4376.64
C2C-VLM57.2076.4085.2745.9185.182071.0477.57
DenseAlign-VLM57.2978.9284.4745.1985.502073.2377.15
LatentCollab (Ours)59.8279.9285.5045.4086.222216.2779.78
  1. Empirical Validation of Our KV-Cache Adaptations: Consistent with the C2C limitation you quoted, reversing to a stronger sender (4B -> 2B) immediately turns both C2C-VLM and DenseAlign-VLM positive on 5 of 7 benchmarks over Receiver-Only (2B). This directly confirms that our VLM cache-transfer implementations work as intended and faithfully reproduce the original papers' regime sensitivity.
  2. LatentCollab Leads in Both Regimes: Even in 4B -> 2B, LatentCollab outperforms Receiver-Only (2B) and Text Communication on all 7/7 benchmarks and beats both KV-cache baselines on 6 of 7.
  3. Why Smaller / Cross-Family Senders Matter: Pairing a lightweight auxiliary VLM or heterogeneous vision encoder (InternVL3.5-2B) with a primary VLM reflects practical low-overhead collaboration without requiring an expensive larger sender. Whereas full-layer KV-cache injection directly perturbs a stronger receiver's internal layer-wise attention when the sender is smaller, LatentCollab injects compact states at the input sequence regularized by confidence-gated receiver trust Ltrust\mathcal{L}_{\mathrm{trust}} (Eqs. 13–14)—preventing weaker-sender noise from overriding confident receiver predictions while still extracting complementary visual cues.
Reviewer_SjsGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LatentCollab, a framework for communication between vision-language agents through continuous latent representations. A frozen sender recursively feeds its final-layer hidden states back as inputs to construct a latent trajectory. A trainable two-layer MLP translates this trajectory into the frozen receiver’s input embedding space, and the resulting message is inserted between the receiver’s visual and query tokens. The translator is trained using task supervision, an input-conditioned correspondence loss, and confidence-gated receiver trust regularization. Across seven benchmarks and three sender–receiver configurations, LatentCollab outperforms Receiver-only and text communication in all 21 comparisons, and both KV-cache communication baselines in 18. The paper also reports lower inference latency than the evaluated communication baselines for one model configuration.

Critical strengths
  1. The framework is simple and modular, enabling latent communication between frozen VLMs by training only a lightweight translator.

  2. Experiments cover seven benchmarks and three model configurations, showing consistent improvements over Receiver-only and Text communication.

Critical weaknesses
  1. The translator is trained on 25K labeled LLaVA samples, whereas the paper does not report adapting the Receiver-only or text communication baselines using the same data. The current comparisons therefore do not clearly separate the benefits of sample-specific sender information from those of additional supervised adaptation. Removing the input-conditioned correspondence loss substantially reduces the gains on several benchmarks, confirming its importance but not establishing that the trained translator uses sample-specific sender information. Shuffling or replacing the sender latents at inference time would provide a direct test of this dependence. A substantial performance drop would support the claimed role of input-conditioned communication.
  2. Text communication underperforms Receiver-only by approximately 11–14 percentage points on TextVQA across all three configurations and also degrades performance on MME in every configuration. These results warrant further investigation, as they may be influenced by the communication protocol, message length, or receiver-side integration. The main text provides insufficient detail about the sender prompts, generation lengths, and how the receiver incorporates textual messages, making it difficult to assess whether the baseline has been sufficiently optimized. Evaluating stronger task-conditioned text communication protocols would make the comparison more convincing.
  3. Figure 1 and the two qualitative examples illustrate that textual messages may omit or misrepresent visual evidence, but they do not establish that this mechanism primarily accounts for the performance gaps in the main experiments. Moreover, the case study presents messages from a richer-evidence variant rather than the official text protocol. Although the caption states that the official protocol also fails on both examples, displaying messages from a different protocol limits the analysis of the actual failure mechanisms. A quantitative analysis of evidence preservation and its relationship to downstream errors would strengthen the proposed explanation.
Questions for authors
  1. Could the authors report results when the sender latents are shuffled across samples or replaced with mismatched latents at inference time? In addition, how would the Receiver-only and text communication baselines perform if they were adapted with the same 25K LLaVA samples, for example through lightweight receiver-side fine-tuning?
  2. Could the authors provide more details on the text communication protocol, including the sender prompts, generation lengths, and how the receiver incorporates the textual messages?
  3. Is there quantitative evidence that the performance gaps of text communication are mainly caused by omitted or distorted visual evidence in the messages, rather than by other factors such as the communication protocol?
4 comments

We thank Reviewer SjsG for the constructive feedback and for recognizing the modularity, parameter efficiency, and consistent gains of LatentCollab across all 21 benchmark–configuration pairs. Below, we address all concerns regarding 25K supervised adaptation vs. sample-specific latent communication, text communication protocol specifications and stronger structured variants, and case-study wording clarification with quantitative failure attribution.

1. Parameter- and Data-Matched 25K Supervised Controls (CW1 & Q1)

To separate the gains of sample-specific sender communication from 25K answer supervision—and to evaluate Receiver-Only and Text Communication adapted on the same data—we trained three controlled baselines under Qwen3-VL-2B -> Qwen3-VL-4B on the exact same 25K LLaVA training set used by LatentCollab. Alongside lightweight prompt tuning, we also evaluated standard SFT of the receiver:

  • Soft Prompt 25K (Parameter- & architecture-matched lightweight control): Replaces sample-specific sender trajectories with a fixed, sample-independent input and uses a parameter- and architecture-matched MLP module to generate K=32K=32 sample-shared continuous prompt embeddings at the same receiver injection slot.
  • SFT 25K (Receiver-only SFT): Applies standard SFT to the receiver on the same 25K image–question–answer pairs without sender input.
  • Text Comm 25K SFT (Message-conditioned receiver SFT): Whereas Table 2 reports zero-shot Text Communication between frozen models, here the frozen sender generates text messages for the 25K training examples, and the receiver undergoes standard SFT conditioned on (image + question + sender text message) to predict target answers.
Method / ControlGQATextVQAPOPEVizWizSQAMMEMMB
Receiver-Only62.8081.8489.8141.5880.432342.5080.36
Text Communication (Zero-Shot)59.3270.8391.0745.3585.082256.1783.69
Soft Prompt 25K57.2876.4185.4546.6188.892204.2686.54
SFT 25K62.8582.6190.2342.0984.452362.7981.14
Text Comm 25K SFT61.1579.6385.7541.0086.332279.4481.37
LatentCollab (Ours)67.2485.3891.7348.8890.632399.1587.91

While Soft Prompt 25K helps align answer formatting on multiple-choice benchmarks (SQA, MMB), a static input-agnostic soft prompt does not generalize uniformly across diverse benchmark distributions and degrades fine-grained visual perception, OCR, and hallucination resistance (GQA, TextVQA, POPE, MME) relative to Receiver-Only. Meanwhile, SFT 25K yields consistent gains over Receiver-Only across all 7 benchmarks, yet LatentCollab substantially outperforms both Soft Prompt 25K and SFT 25K on every benchmark. Moreover, although Text Comm 25K SFT improves over zero-shot Text Communication on GQA, TextVQA, SQA, and MME, it still underperforms sender-free SFT 25K on 5 of 7 benchmarks and trails LatentCollab across all 7, confirming that our gains are not an artifact of 25K supervised adaptation.

2. Inference-Time Shuffling & Replacement Ablations on the Trained Translator (CW1 & Q1)

To directly test at inference time whether the fully trained translator TϕT_\phi relies on sample-specific sender information (Qwen3-VL-2B -> 4B), we froze TϕT_\phi and evaluated 7 test-time interventions without retraining:

  1. Mismatched: Replacing HisH_i^s across samples by injecting Tϕ(Hπ(i)s)T_\phi(H_{\pi(i)}^s) (π(i)≠i\pi(i) \neq i) from another test example.
  2. Temporal Permutation: Randomly shuffling the K=32K=32 rollout step order within the matched trajectory HisH_i^s.
  3. Zero Translator Input: Feeding 032×20480_{32 \times 2048} into TϕT_\phi (Z=Tϕ(0)Z = T_\phi(0), isolating static translator bias).
  4. Zero Translator Output: Setting Z=032×2560Z = 0_{32 \times 2560} directly while keeping receiver message slots unmasked.
  5. Gaussian Noise: Replacing HisH_i^s before TϕT_\phi with moment-matched noise H~is=μi+σiϵ\widetilde{H}_i^s = \mu_i + \sigma_i \epsilon (ϵ∼N(0,1)\epsilon \sim \mathcal{N}(0,1)), preserving per-sample activation mean and std.
  6. Sender No Image: Generating HsH^s without the sender's image input (receiver retains original image/question).
  7. Sender Mismatched Question: Generating Hs(Ii,Qj)H^s(I_i, Q_j) with original image IiI_i and a donor question QjQ_j from another sample.
Test-Time ConditionGQATextVQAPOPEVizWizSQAMMEMMB
Receiver-Only62.8081.8489.8141.5880.432342.5080.36
Mismatched (Cross-Sample)60.5470.6585.8443.2384.972245.4482.72
Temporal Permutation60.3470.4786.3441.1982.742270.4583.05
Zero Translator Input56.7166.8885.6139.6283.642164.7680.41
Zero Translator Output55.1363.4684.8535.0583.692211.7180.81
Gaussian Noise59.5468.1783.1041.8184.062118.5682.30
Sender No Image60.7671.5186.6142.5883.782245.2182.72
Sender Mismatched Question60.2670.7685.7442.8886.152197.8382.67
LatentCollab (Matched)67.2485.3891.7348.8890.632399.1587.91

Replacing sender latents across samples (Mismatched) or shuffling rollout order (Temporal Permutation) causes severe drops across all 7 benchmarks, falling well below Receiver-Only on GQA, TextVQA, POPE, and MME. Likewise, zeroing inputs/outputs, injecting moment-matched noise, or removing either sender modality (Sender No Image, Sender Mismatched Question) sharply degrades performance on every benchmark, confirming that TϕT_\phi conveys sample-specific, temporally ordered, and jointly image- and question-conditioned sender states.

3. Text Communication Protocol Details & Stronger Task-Conditioned Protocols (CW2 & Q2)

  • Official Protocol Specification: In Text Communication, our task-conditioned sender prompt instructs the sender (acting as an assistant agent) to provide an auxiliary answer supported by relevant visual evidence for the input image–question pair. Decoding uses deterministic greedy search (do_sample=False) with max_new_tokens=1024 as a non-truncating upper bound; in practice, sender outputs terminate naturally at <eos> and average 28.92 tokens across benchmarks. On the receiver side, the sender message is appended directly after the receiver's standard (image + question) input, and the receiver generates the final answer greedily under the identical LMMS-Eval pipeline. Verbatim prompt templates for all protocols will be added to the revised manuscript.
  • Why Default Text Degrades TextVQA/MME & Stronger Protocol Evaluation: Because default sender messages combine a tentative answer with brief visual notes ($\sim$29 tokens), a smaller sender may omit exact scene text (hurting TextVQA) or state premature conclusions that bias paired yes/no judgments (hurting MME). To test whether stronger task-conditioned protocols close the gap, we evaluated two structured single-round protocols (Qwen3-VL-2B -> 4B, greedy decoding, max_new_tokens=1024, receiver retains original image) plus 25K receiver adaptation:
    1. Structured Evidence: Constrains sender output to FACTS / OCR / LIMITS—up to three key visual facts, verbatim OCR when applicable, and explicit uncertainty—avoiding generic scene summaries and forbidding premature sender conclusions.
    2. Candidate Check: Structures sender output into CANDIDATE / SUPPORT / RIVAL / CHECK / UNCERTAIN—providing a concise candidate answer, visible supporting evidence, a plausible rival candidate, and discriminative details for the receiver to verify against its own image before accepting or correcting (single-round, not multi-round debate).
    3. Candidate Check + 25K SFT: Applies standard SFT to the receiver on the exact same 25K LLaVA set conditioned on Candidate Check sender messages.
Communication Protocol (2B -> 4B)GQATextVQAPOPEVizWizSQAMMEMMB
Receiver-Only62.8081.8489.8141.5880.432342.5080.36
Text Communication (Default)59.3270.8391.0745.3585.082256.1783.69
Structured Evidence54.3877.8887.2339.7180.592219.1880.27
Candidate Check60.2982.2988.9437.9181.152227.2282.47
Candidate Check + 25K SFT65.0783.6386.2639.7382.342291.4482.94
LatentCollab (Ours)67.2485.3891.7348.8890.632399.1587.91

Explicit protocol design substantially mitigates TextVQA degradation, and combining Candidate Check with 25K SFT further improves TextVQA, GQA, and MME. However, discrete protocols exhibit cross-task trade-offs—rigid schemas underperform default Text Communication on VizWiz, POPE, and SQA, and MME remains below Receiver-Only. Crucially, LatentCollab outperforms every text protocol (including Candidate Check + 25K SFT) across all 7 benchmarks without task-specific prompt engineering or receiver fine-tuning.

4. Case-Study Clarification & Quantitative Failure Analysis of Text Communication (CW3 & Q3)

  • Clarification on Case-Study Wording: We thank the reviewer for catching the confusing phrasing in the original Fig. 5 caption ("richer-evidence qualitative sender messages") and Appendix B.3/C.1. Quantitative Text Communication was not an answer-only baseline, nor did we use a separate prompt to generate qualitative sender messages: all case studies, sender messages, and predictions in Fig. 5 and Appendix C come directly from the official evaluation runs using the prompts in Appendix B.4 ($\sim$29 tokens on average). Our earlier wording misleadingly referred to quoting the error-relevant excerpts of those official sender messages (omitting full messages for layout readability) rather than bare answers. We have revised the Fig. 5 caption and Appendix B.3/C.1 to state unambiguously that these are excerpts from the official evaluation runs.
  • Quantitative Failure Attribution: To quantify how often Text Communication failures stem from omitted/distorted visual evidence vs. other factors, we randomly sampled 500 Text Communication error instances per configuration (1,500 cases total) and used GPT-6.1 Sol to perform structured attribution comparing the image, reference answer, official sender message, and receiver prediction across six mutually exclusive categories:
Configuration (n=500 errors each)Omitted Visual EvidenceDistorted Visual EvidenceBoth Omission & DistortionMixed Visual & Other FactorsOther Observed FactorsUnresolved
Qwen3-VL-2B -> 4B9.60%11.20%1.60%2.20%48.80%26.60%
Qwen3-VL-4B -> 8B10.60%10.20%0.00%2.40%49.60%27.20%
InternVL3.5-2B -> Qwen3-VL-4B15.20%11.80%0.00%2.00%44.60%26.40%

Because sampling covers all failure instances of Text Communication, Other Observed Factors (44.60%–49.60%) captures intrinsic receiver capability limits and shared backbone ceilings on hard samples (cf. Appendix C.2) alongside non-visual assertion bias, while Unresolved (26.40%–27.20%) marks indeterminate cases. Across all three configurations, roughly one-quarter of all Text Communication errors directly stem from omitted or distorted visual evidence in the sender's message.

Together with Sec. 3, this directly answers Q3: while receiver capability limits and protocol effects account for a substantial portion of total errors (explaining why Candidate Check + 25K SFT recovers ground on TextVQA and GQA), omitted and distorted visual evidence consistently accounts for roughly one-quarter of all errors (about one-third of resolved errors), explaining why even structured, SFT-adapted text protocols consistently trail LatentCollab.

Reviewer_ZmfpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes LatentCollab, a communication interface between frozen vision-language models. A sender recursively feeds its final-layer hidden state back as the next continuous input to produce a length-K latent trajectory. A two-layer translator maps each state into the receiver’s embedding space, and the translated trajectory is inserted after the receiver’s visual tokens. The translator is trained with target-answer likelihood, a matched-versus-mismatched correspondence margin, and confidence-gated regularization toward a null-message receiver. Across seven benchmarks and three sender–receiver configurations, the method outperforms receiver-only and text communication on all reported pairs and usually outperforms two KV-cache baselines.

Critical strengths
  1. The proposed interface is simple and parameter-efficient: both VLMs remain frozen, communication uses final-layer states rather than full layer-wise caches, and the framework handles cross-scale and cross-family pairs.

  2. Results are broad and consistently positive across the reported 21 benchmark–configuration pairs. The trajectory-length and objective ablations indicate that performance is not tied to one exact message length and that both auxiliary objectives contribute.

Critical weaknesses
  1. The current controls do not establish that improvements come from sender-specific communication rather than supervised continuous prompt tuning. LatentCollab alone receives answer-supervised translator training on 25K examples, while no parameter- and data-matched receiver-only soft-prompt/adapter control is reported. More importantly, the trained translator is not evaluated with matched, mismatched, null, random, image-ablated, or question-ablated messages at test time; Section 3.1 performs such controls only for raw latents. The one-sided correspondence loss stops gradients through the mismatch score and does not directly lower mismatched-message likelihood, so the ablation drop alone does not prove preservation of sender–input correspondence.

  2. Baseline fairness, reliability, and efficiency evidence are under-specified. It is unclear whether text and KV-cache baselines receive the same 25K examples, supervision, tuning budget, and model-specific optimization. The unspecified LLaVA mixture contains training data from benchmark families such as GQA and TextVQA, but no composition or overlap audit is given. No independent seeds or confidence intervals are reported, K is selected using reported benchmark performance, and the compactness claim lacks actual payload, peak-memory, bandwidth, and controlled hardware measurements. The reproducibility statement also omits several training and prompting details and provides no code artifact.

Questions for authors
  1. For the fully trained translator, what happens at inference when the receiver receives a mismatched, randomly permuted, zero, sender-image-ablated, or sender-question-ablated trajectory? How does LatentCollab compare with a parameter-matched learned soft prompt or receiver adapter trained on the same 25K examples? If these controls perform similarly, the communication interpretation would not be supported.

  2. Were Text Communication, C2C-VLM, and DenseAlign-VLM retrained with exactly the same data, labels, optimization budget, and model pairs? Please provide the precise 25K data composition and split-overlap audit, repeated-run uncertainty, and payload/peak-memory/latency measurements under a documented common hardware and decoding setup.

5 comments

We thank Reviewer Zmfp for the constructive feedback and for recognizing the simplicity, parameter efficiency, and consistent empirical gains of LatentCollab across all 21 benchmark–configuration pairs. Below, we address your questions on sender-specific communication vs. supervised prompt/receiver tuning and baseline fairness, statistical reliability, and measured hardware efficiency via controlled experiments under Qwen3-VL-2B -> Qwen3-VL-4B.

1. Parameter- and Data-Matched 25K Supervised Controls (CW1 & Q1)

To test whether improvements stem from 25K answer supervision rather than sender-specific communication—and to evaluate text communication under matched 25K supervision—we trained three controlled baselines on the exact same deduplicated 25K LLaVA subset:

  • Soft Prompt 25K (Parameter- & architecture-matched): Replaces sender trajectories with a fixed, sample-independent input and uses a parameter- and architecture-matched MLP module to generate K=32K=32 sample-shared continuous prompt embeddings at the same receiver injection site.
  • SFT 25K (Receiver-only SFT): Applies standard SFT to the receiver on the same 25K image–question–answer pairs without sender input.
  • Text Comm 25K SFT (Message-conditioned receiver SFT): While Table 2 reports zero-shot Text Communication between frozen models, here the frozen sender generates text analyses for the 25K training examples, and the receiver undergoes standard SFT conditioned on the image, question, and sender text message.
Method / ControlGQATextVQAPOPEVizWizSQAMMEMMB
Receiver-Only62.8081.8489.8141.5880.432342.5080.36
Text Communication (Zero-Shot)59.3270.8391.0745.3585.082256.1783.69
Soft Prompt 25K57.2876.4185.4546.6188.892204.2686.54
SFT 25K62.8582.6190.2342.0984.452362.7981.14
Text Comm 25K SFT61.1579.6385.7541.0086.332279.4481.37
LatentCollab (Ours)67.2485.3891.7348.8890.632399.1587.91

While Soft Prompt 25K aids multiple-choice formatting (SQA, MMB), it substantially degrades fine-grained visual perception, OCR, and hallucination resistance (GQA, TextVQA, POPE, MME) relative to Receiver-Only. LatentCollab consistently outperforms both Soft Prompt 25K and SFT 25K across all 7 benchmarks. Moreover, while receiver fine-tuning (Text Comm 25K SFT) improves over zero-shot Text Communication on GQA, TextVQA, SQA, and MME, it still underperforms sender-free SFT 25K on 5 of 7 benchmarks and trails LatentCollab across all 7, confirming that verbalization loss cannot be recovered by receiver-side SFT.

2. Inference-Time Perturbation & Modality Ablation on the Trained Translator (CW1 & Q1)

In Lcorr\mathcal{L}_{corr}, stop-gradient treats the mismatched trajectory as a fixed reference anchor so optimization raises the likelihood of matched trajectories. To directly verify at test time whether the fully trained translator TϕT_\phi relies on sample-specific multimodal inputs and trajectory structure, we froze the trained TϕT_\phi and evaluated 7 inference-time interventions:

  1. Mismatched: Injecting Tϕ(Hπ(i)s)T_\phi(H_{\pi(i)}^s) from another sample in the dataset.
  2. Temporal Permutation: Randomly permuting the K=32K=32 step order of HisH_i^s.
  3. Zero Translator Input: Feeding 032×20480_{32 \times 2048} into TϕT_\phi (Z=Tϕ(0)Z = T_\phi(0), isolating static translator bias parameters).
  4. Zero Translator Output: Setting Z=032×2560Z = 0_{32 \times 2560} directly while keeping receiver message slots and positions unmasked.
  5. Gaussian Noise: Replacing HisH_i^s before TϕT_\phi with moment-matched Gaussian noise H~is=μi+σiϵ\widetilde{H}_i^s = \mu_i + \sigma_i \epsilon (ϵ∼N(0,1)\epsilon \sim \mathcal{N}(0,1)), matching the per-sample mean μi\mu_i and standard deviation σi\sigma_i of HisH_i^s.
  6. Sender No Image: Generating HsH^s without the sender's image input.
  7. Sender Mismatched Question: Generating Hs(Ii,Qj)H^s(I_i, Q_j) with original image IiI_i and a donor question QjQ_j from another sample (receiver retains original Ii,QiI_i, Q_i).
Test-Time ConditionGQATextVQAPOPEVizWizSQAMMEMMB
Receiver-Only62.8081.8489.8141.5880.432342.5080.36
Mismatched60.5470.6585.8443.2384.972245.4482.72
Temporal Permutation60.3470.4786.3441.1982.742270.4583.05
Zero Translator Input56.7166.8885.6139.6283.642164.7680.41
Zero Translator Output55.1363.4684.8535.0583.692211.7180.81
Gaussian Noise59.5468.1783.1041.8184.062118.5682.30
Sender No Image60.7671.5186.6142.5883.782245.2182.72
Sender Mismatched Question60.2670.7685.7442.8886.152197.8382.67
LatentCollab (Matched)67.2485.3891.7348.8890.632399.1587.91

Every perturbation causes severe degradation across all 7 benchmarks. Although static bias or uninformative inputs retain minor multiple-choice formatting gains over Receiver-Only on SQA and MMB, matched sender trajectories drive the decisive improvements; meanwhile, even matching per-sample activation statistics (Gaussian Noise), shuffling rollout order (Temporal Permutation), or ablating either sender modality (Sender No Image / Sender Mismatched Question) drops GQA, TextVQA, POPE, and MME well below Receiver-Only, demonstrating that TϕT_\phi conveys temporally ordered, jointly image- and question-conditioned sender information.

3. Baseline Fairness, Data Overlap Audit, and Multi-Seed Reliability (CW2 & Q2)

  • Baseline Training & Alignment: C2C-VLM, DenseAlign-VLM, and all 25K controls (Soft Prompt 25K, SFT 25K, Text Comm 25K SFT) were trained on the same 8xNVIDIA L20 server for identical model pairs using the exact same 25K deduplicated LLaVA examples and target-answer labels, and evaluated under identical greedy LMMS-Eval pipelines. Table 2's Text Communication is a zero-shot frozen baseline, now complemented by Text Comm 25K SFT in Sec. 1. Because the KV-cache baselines have distinct multi-stage architectures and objectives (single-stage next-token gated residual fusion for C2C vs. two-stage ℓ2\ell_2 cache reconstruction followed by mixed-context generation for DenseAlign), each was trained to convergence on the shared 25K dataset following its original training paradigm.
  • 25K Data Deduplication & Split-Overlap Audit: Prior to sampling, we performed strict image-level deduplication and cross-split filtering on the multimodal LLaVA instruction mixture to remove any image overlapping with the evaluation splits of all 7 benchmarks (GQA, TextVQA, POPE, VizWiz, ScienceQA, MME, MMBench), guaranteeing 0% image- and sample-level overlap. The 25,000 examples were then sampled uniformly at random from the non-overlapping pool and fixed across all configurations and baselines; notably, LatentCollab achieves equally strong gains on out-of-domain benchmarks absent from LLaVA (e.g., VizWiz, SQA, MMB).
  • Choice of K=32K=32 & 3-Seed Uncertainty: K=32K=32 was not tuned on test benchmarks; it was fixed a priori across all model pairs and benchmarks to balance message compactness and rollout cost, with post-hoc sensitivity over K∈{8,16,32,64,128}K \in \{8, 16, 32, 64, 128\} reported in Table 4 (where different benchmarks peak at different KK and all KK beat Receiver-Only). Below we report LatentCollab across 3 independent random seeds (ddof=1); the 3-seed means closely match—and on 4 of 7 benchmarks (POPE, VizWiz, MME, MMB) slightly exceed—the reported single-run results, confirming low variance, absence of seed selection bias, and robust gains:
Metric SummaryGQATextVQAPOPEVizWizSQAMMEMMB
Receiver-Only62.8081.8489.8141.5880.432342.5080.36
LatentCollab (Reported)67.2485.3891.7348.8890.632399.1587.91
LatentCollab (3 Seeds Mean ± Std)66.66 ± 0.8684.66 ± 1.0592.05 ± 0.4549.63 ± 1.2190.17 ± 0.642412.78 ± 19.6388.19 ± 0.43

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the reviewers and the AC for their thoughtful feedback and careful evaluation. Their insights greatly helped us strengthen the clarity, rigor, and empirical depth of our paper. For ease of comparison, all newly added and revised content in the updated Main Paper and Supplementary Material is highlighted in red.

We summarize our responses and the corresponding improvements in the revised manuscript as follows.

  1. 25K supervised controls. We added Soft Prompt 25K, SFT 25K, and Text Comm 25K SFT on the identical deduplicated 25K subset, confirming that our gains come from sender-specific communication rather than 25K supervision or static prompts (Section 4.5, Revision Table 6) (Zmfp, SjsG, rEXv).
  2. Inference-time verification of the trained translator. Extending Section 3.1, we evaluated seven test-time perturbations and modality ablations on the frozen translator TϕT_\phi to verify sample-specific, temporally ordered, multimodal communication (Section 4.3, Revision Table 3) (Zmfp, SjsG, rEXv).
  3. Stronger-sender regime and KV-cache baseline details. We added Qwen3-VL-4B -> Qwen3-VL-2B results confirming expected KV-cache gains under a stronger sender while LatentCollab still leads (Appendix E.1, Revision Table E.1), and detailed our VLM adaptations for C2C-VLM and DenseAlign-VLM (Appendix B.3) (Zmfp, rEXv).
  4. Controlled cross-GPU efficiency analysis. Complementing Figure 4, we added end-to-end latency, communication payload, inter-GPU transfer latency, and peak memory comparisons across all seven benchmarks (Appendix E.2, Revision Table E.2) (Zmfp).
  5. Text protocols, case-study clarification, and failure attribution. We added verbatim prompt templates (Appendix B.4), clarified that qualitative cases quote official evaluation excerpts (Revision Figure 5, Appendix B.3, C.1), and reported structured text protocols plus a 1,500-case error attribution (Zmfp, SjsG, rEXv).
  6. Reliability, data audit, and modality scope. We reported our zero-overlap data audit, three-seed stability, KK sensitivity (Section 4.5, Revision Table 4), OOD gains, and why unimodal text tasks mismatch our post-visual interface (Section 3.2, Eq. 7), with code to be released upon acceptance (Zmfp, rEXv).

We sincerely appreciate the reviewers' insightful suggestions and the AC's time and effort throughout the review process. We would be grateful for the opportunity to engage in any further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.