acceptodds
Under review as a conference paper at ICLR 2027

ViPLE: Visual Inference with Parallel Latent Encoding

Abstract

Explicit visual reasoning generates intermediate text sequentially, increasing decoding cost. We investigate whether useful visual reasoning context can instead be constructed in parallel for a frozen multimodal backbone. We propose VIPLE (Visual Inference with Parallel Latent Encoding), which extends one-pass latent trajectory encoding with a final cross-attention read of fused image-question states. The latent slots attend to these states before conditioning the frozen decoder. Supervised alignment, on-policy distillation, and reinforcement learning train the interface while keeping the backbone frozen. Across six benchmarks, on Qwen3-VL-2B, 4B, and 8B, VIPLE improves macro accuracy by **+(5.12–7.15) pp** over the cheapest measured explicit-reasoning settings while decoding fewer tokens. On GLM-4.1V-9B and Qwen3-VL-32B, the gains are +2.31 pp and +2.50 pp, respectively, at matched decoding costs obtained by interpolation. The cheapest measured explicit-reasoning settings that reach VIPLE's accuracy require **(1.8–5.8)×** as many decoded tokens. Length-preserving replacement controls show the importance of learned latent content, while retraining without the final cross-attention layer reduces accuracy on mathematical tasks. These results demonstrate that a learned parallel interface can improve the accuracy-decode-cost trade-off of frozen multimodal models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLM, Visual Reasoning, Latent Encoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ShRROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ViPLE, a parallel latent interface for frozen multimodal models. It extends LRT with a final cross-attention read of fused image-question states, producing 256 latent slots through an encoder trained with supervised alignment, on-policy distillation, and reinforcement learning. Experiments across five backbones and six benchmarks show improved accuracy under constrained sequential decoding budgets.

Critical strengths

The paper addresses a practical efficiency problem and provides broad evaluation across model scales and families. The latent-replacement controls and retrained cross-attention ablation provide useful evidence that both the learned representations and the final prompt read contribute to performance.

Critical weaknesses
  1. The technical novelty appears limited. Parallel latent construction, a frozen consuming model, and the encoder backbone closely follow LRT (Jiang et al., 2026). The main architectural addition is a final cross-attention read of fused prompt states. Although Table 2 supports its empirical value, comparisons with other trained latent interfaces are needed to establish the contribution more convincingly.
  2. The motivation for the added prompt read needs further development. The upstream encoder already consumes the same fused image-question states. Comparisons with parameter-matched alternatives or different read placements would help establish why this particular design is preferable to simply adding capacity or another route to the prompt.
  3. The overall computational cost is insufficiently characterized. The token metric excludes the encoder and the 256-slot backbone prefill, so fewer decoded tokens do not establish lower end-to-end latency or total computation. Please report latency, peak memory, and FLOPs, together with per-stage training GPU-hours. Freezing the backbone still requires backpropagation through it to train the encoder.
  4. Figure 6 raises questions about the necessity and stability of the training pipeline. WeMath accuracy falls from 45.6% after SFT to 34.6% after OPD, then reaches 65.4% after ten RL steps. The paper documents the accompanying response-length changes and the ten-word reward gate, but does not isolate their effects. Comparing SFT-to-RL with the full pipeline, and correctness-only RL with the gated reward, across training seeds would clarify OPD's benefit and the source of the recovery.
  5. The decoding comparison contains a protocol difference: Full-CoT uses a continuation fallback, whereas ViPLE uses direct force-close termination. Matched-termination evaluation and a baseline trained to produce concise textual answers would help separate the benefits of latent conditioning from termination behavior and additional task-specific training.

Reference:

[1] Cong Jiang, Xiaofeng Zhang, Fangzhi Zhu, XiaoWei Chen, Junxiong Zhu, & Zheng Zhang (2026). Rethinking LLM Reasoning: From Explicit Trajectories to Latent Representations. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=CbK7lYbmv8

Questions for authors

See weakness.

3 comments
Reviewer_ShRROpenReview ↗

Response to rebuttal

I thank the authors for the thorough response and additional experiments. These results resolved some of my concerns regarding training cost and training pipeline.

However, my concern about the technical novelty of ViPLE remains. The response primarily emphasizes differences in the architecture design and research domain relative to LRT. These differences alone do not establish the technical contribution ViPLE provides beyond adapting an existing latent-reasoning framework to multimodal inputs. While the final prompt-read ablation supports empirical usefulness of the component, it does not sufficiently explain why this particular design is necessary.

Therefore, I'm inclined to maintain my score.

Fianlly, we respectfully believe that a Clear Rejection (1) is disproportionately harsh, particularly given your acknowledgment of our empirical contributions and the concerns resolved through our additional experiments.

We sincerely hope you will reconsider your assessment and raise your rating accordingly.

AuthorsOpenReview ↗

[Urgent] Follow-up

Fianlly, we respectfully believe that a Clear Rejection (1) is disproportionately harsh, particularly given your acknowledgment of our empirical contributions and the concerns resolved through our additional experiments.

We sincerely hope you will reconsider your assessment and raise your rating accordingly.

Reviewer_5wZoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to replace part of explicit visual reasoning text with parallel latent representations in a frozen vision-language model, thereby reducing the sequential decoding cost during the response generation stage. The authors propose ViPLE, which, building on existing one-pass latent trajectory encoding, introduces a final cross-attention that lets the constructed 256 latent slots re-read the fused image-question states, and then injects the resulting latent block into the frozen multimodal decoder. The training stages sequentially adopt supervised alignment, on-policy distillation, and reinforcement learning. Experiments demonstrate the effectiveness of the method, and through replacement/retraining/checkpoint analyses, the authors argue for the role of the learned latent representation and the final prompt read.

Critical strengths
  1. Broad experimental coverage. Five backbones, six benchmarks, four decoding runs per experiment, etc. The experimental coverage is very complete, and this part makes the method highly convincing.

  2. Relatively sufficient ablations. The paper compares control conditions such as random tokens, untrained encoder, norm-matched variants, and removing the latent block, so it can better distinguish the difference between "adding 256 token positions" and "these latent slots contain useful information." This gave me a deeper understanding of the contribution of each part of the method.

Critical weaknesses
  1. The efficiency experiments are somewhat biased. The paper explicitly excludes encoder computation and backbone prefill from the cost metric, so whether ViPLE actually has an advantage in real deployment remains unclear. This affects the true effectiveness of the method in this paper.

  2. This paper is essentially an incremental method over LRT, but the comparison with LRT and other methods based on it is insufficient. This makes it difficult for me to measure its direct improvement relative to LRT.

Questions for authors
  1. What is the actual inference cost of this method? Relevant explanation needs to be added. Even if the performance is not SOTA, as long as it is honest enough and the gap is not large, I may still raise my score.

  2. Why is the latent capacity fixed at 256 slots? Please explain the basis for this choice.

  3. Please at least add more experiments comparing with LRT and report the actual improvement results.

  4. The figure and table design needs improvement. Figure 1 is not a vector graphic and is somewhat too flashy, while Figure 2 contains too many formulas. The two figures overlap substantially overall, yet neither explains clearly what this paper is actually doing, so the reader cannot quickly understand what this method is for.

1 comment
AuthorsOpenReview ↗

[Follow-up: Urgent] Less than 3 hours

The rebuttal phase is nearing its end—less than 3 hours remain.

We have put a tremendous amount of effort into this rebuttal, and we hope the reviewers will take our hard work into consideration.

If you feel that your concerns have been addressed, please consider responding and potentially raising your score.

Reviewer_LPLuOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to construct a compact visual reasoning context in parallel for a frozen multimodal backbone before text generation, addressing the high decoding cost of sequentially generating intermediate reasoning text in explicit visual chain-of-thought reasoning. The authors propose ViPLE , which builds on one-pass latent trajectory encoding to construct 256 latent slots in parallel after extracting the fused image–question prompt states. A final cross-attention read is then introduced, allowing the completed latent slots to directly attend to the full fused multimodal prompt states before being projected into the frozen decoder’s input space. During training, only the encoder parameters are updated, while the multimodal backbone remains entirely frozen. The encoder is trained in three successive stages: supervised alignment, on-policy distillation, and reinforcement learning with a minimum response-length gate of 10 words. The authors evaluate ViPLE on five frozen backbones from two model families (Qwen3-VL-2B/4B/8B/32B and GLM-4.1V-9B) across six benchmarks. The results demonstrate that ViPLE achieves a favorable accuracy–decode-cost trade-off under constrained reasoning budgets.

Critical strengths
  1. The paper addresses a key bottleneck in multimodal large language models: the high sequential decoding cost of explicit chain-of-thought (CoT) reasoning. It proposes constructing a fixed-length latent conditioning block with 256 slots in parallel before response generation, offering an appealing alternative to sequentially generating intermediate reasoning text.
  2. The authors conduct a detailed checkpoint-based analysis of training trajectories, revealing that on-policy distillation (OPD) tends to produce extremely short responses of at most four tokens, accounting for 77% of responses on WeMath. Introducing reinforcement learning with a response-length gate eliminates this short-response pattern within the first 10 RL steps and increases WeMath accuracy by 30.80 percentage points.
Critical weaknesses
  1. Figure 1 shows traces of AI generation;
  2. The mechanism of learning latent representations has not been explained thoroughly. Although the ablation studies demonstrate the contribution of the trained latent content to model performance, this paper provides limited explanation into what information is encoded in the latent slots and how this information supports reasoning;
  3. The paper lacks ablation studies on two important design choices: the minimum response-length gate and the fixed latent-block length of 256 slots. It remains unclear how varying the number of latent slots affects the accuracy-decode-cost trade-off.
Questions for authors
  1. Can you provide more analysis on what information is stored in the latent slots? Are these representations mainly visual features, reasoning traces, or compressed task-specific knowledge?
  2. Is ViPLE sensitive to the potential number of slots? What happens when using less or more than 256 slots?
  3. Could the authors provide ablation studies that remove the response-length gate or vary the minimum response-length threshold? How would these changes affect the accuracy and response-length trajectories during reinforcement learning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.