acceptodds
Under review as a conference paper at ICLR 2027

MoLE: Mixture of Latent Experts for Complementary Visual Reasoning

Abstract

Latent visual reasoning equips vision–language models with continuous intermediate states that can process visual evidence without explicit textual reasoning traces or repeated image operations. However, existing methods often allow multiple latent tokens to access the same visual evidence through shared value projections, providing no mechanism for them to extract complementary visual information; simply increasing the latent budget can therefore yield redundant latent representations. We argue that effective latent reasoning should encourage different latent tokens to extract complementary visual information, and thereby act as specialized visual experts. Based on this insight, we propose MoLE, a Mixture of Latent Experts framework that controls both what visual evidence each latent visual expert observes and how it transforms that evidence. MoLE isolates latent visual experts during evidence extraction and uses dedicated latent summary experts to aggregate the complementary representations of latent visual experts. A two-stage training pipeline first forces visual evidence through this latent pathway and then restores direct visual access, requiring neither predefined expert roles nor intermediate visual targets. Across five visual reasoning benchmarks, MoLE achieves an average score of 78.6, outperforming data-matched supervised fine-tuning by 4.9 and the strongest evaluated latent visual reasoning baseline at the same latent budget by 3.6. Representation analyses show lower latent-state similarity and more diverse visual attention, while masking the latent pathway reduces average performance by 9.2. These results demonstrate that specializing latent computation is more effective than merely increasing the number of latent tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Latent Visual Reasoning, Multimodal Reasoning, Mixture-of-Experts, Sparse Routing, Visual Reasoning, Latent Tokens

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_f9RyOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a novel framework to improve visual reasoning performance by introducing Mixture of Latent Experts. The new architecture consists a Sparse Visual Routing, Expert-Specific Value Adapter and Latent Summary Experts. The authors adopt a two stage training strategy. Firstly, by using a bottleneck in the beginning, enforcing the model to learn about latent tokens instead of visual tokens, encouraging latent tokens to capture useful visual information. Then restoring the normal training process. Such training process is natural and understandable.

Critical strengths

The paper presents a well-motivated framework for improving latent visual reasoning by encouraging different latent experts to extract complementary visual information. The combination of sparse visual routing, expert-specific value adapters, and latent summary experts is intuitive and well designed. The two-stage training strategy is also reasonable and easy to understand. The proposed method achieves consistent improvements across five visual reasoning benchmarks, outperforming both data-matched SFT and existing latent reasoning baselines. The authors also provide extensive ablation studies to demonstrate the effectiveness of different architectural components, training objectives, and expert configurations. Additional experiments on Qwen3-VL models further support the general applicability of the method.

Critical weaknesses

The paper argues that encouraging different latent experts to learn complementary visual information reduces redundancy and improves visual reasoning performance. However, the causal relationship between these factors is not sufficiently demonstrated. Although the authors provide ablation studies and representation similarity analyses, it remains unclear whether the improvements mainly come from reduced redundancy or from the additional architectural components. More direct experimental evidence would strengthen the central claim. Also, the proposed sparse routing mechanism explicitly restricts different experts to different subsets of visual tokens. Therefore, the observed diversity in attention patterns may partially result from these architectural constraints rather than meaningful specialization learned by the model. Although the authors provide a parameter-matched adapter baseline, a more controlled comparison within the same latent architecture would help isolate the contribution of expert-specific processing.

Questions for authors
  1. The authors argue that expert specialization reduces redundancy and improves visual reasoning performance. Could the authors provide more direct evidence supporting this causal relationship?
  2. I am concerned that a limited number of latent visual experts may not be sufficient to represent fine-grained information in high-resolution images. Although the second training stage restores direct visual access, could the authors further analyze how latent budget and image resolution affect performance, particularly on V*?
  3. While the authors demonstrate improvements across Qwen3-VL models of different sizes, the gains appear smaller on larger backbones. Could the authors discuss whether the benefits of latent expert specialization diminish as model capacity increases?
1 comment

Thank you for the detailed responses and additional experiments. I appreciate the substantial effort in addressing the concerns. Overall, these results address most of my concerns.

I have one minor remaining suggestion regarding the relationship between image resolution and latent budget. The current comparison between RWQA and V* may be affected by differences in task characteristics and data distributions. If feasible, could you conduct a small inference-time experiment on the same V* samples at different input resolutions, comparing the existing 4+4 and 12+4 configurations? This could provide more direct evidence of whether higher-resolution inputs particularly benefit from allocating more capacity to visual experts?

Reviewer_GSFWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes MOLE, a visual expert routing method in the latent space, which assigns different latent tokens to extract distinct visual information. Experimental results demonstrate that it outperforms baseline methods when using the same latent tokens.

Critical strengths
  1. The motivation presented by MOLE is clear and well-justified; rank collapse in latent tokens is indeed a significant bottleneck in this field.

  2. The experimental validation is comprehensive and effectively demonstrates the performance gains of the proposed method.

Critical weaknesses
  1. The current method is only effective for simple visual question answering (VQA) tasks; it struggles to yield performance gains for more complex, multi-step reasoning tasks.

  2. The scalability of the number of experts remains unclear. It has yet to be empirically demonstrated whether this paradigm can continue to drive performance improvements as the number of latent tokens increases. Given the current results, the effective differentiation among latent experts may well stem from their limited total count.

  3. The experiments fail to sufficiently validate the mechanism of expert complementarity. Observed low similarity and significant differences in attention patterns could simply be artifacts of diversity loss or the routing mask; they do not constitute definitive proof that the multiple latent tokens provide complementary and meaningful information.

Questions for authors
  1. Is the current MOLE method suitable for reasoning-based tasks?

  2. As the number of experts increases, can this routing mechanism still ensure effective specialization and information diversity?

1 comment

Thank you to the authors for the response and the effort! However, I still have some concerns about the experimental results:

  • The reproduced CoVT numbers are lower than in the original paper. The base model scores match the CoVT paper almost exactly, but CoVT itself gets 68.4 on RWQA (vs. 71.1 in the original paper) and 77.5 on V* (vs. 79.1).

  • Monet also uses Qwen2.5-VL-7B and reports 83.25 on V.* Its base score (76.44) is essentially the same as the one in this paper (76.6), yet its V* result is higher than MoLE’s.

  • The official Qwen2.5-VL-7B scores are 87.3 on ChartQA and 68.2 on MathVista. The paper’s own base-model evaluation gives only 78.5 and 65.3. MoLE’s ChartQA score (87.0) doesn’t even beat the officially reported base score.

In addition, I have a few minor suggestions:

Several relevant baselines are missing from the comparison. The authors cite Monet and Mirage (both CVPR 2026), and ILVR (ACL 2026) in the related work, but do not compare against any of them. Under the ICLR 2027 reviewer guidelines, a paper counts as concurrent work only if it was officially published within two months before the submission deadline. All of these papers came out before that window, so being “concurrent work” is not a valid reason to leave them out.

Finally, I thank the authors again for their efforts. I will keep my score unchanged.

Reviewer_bP7FOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problem of latent visual reasoning, where the latent visual tokens tend to possess similar representations. The authors propose a mix-of-latent-experts mechanism that encourages isolated latent visual experts to encode different visual information. The experiments show improved performance over existing tool-based and latent-based visual reasoning baselines. Moreover, the analytical experiments show that the proposed method exhibits increased diversity between the learned latent representations.

Critical strengths
  1. The motivation is sound. The problem of representation collapse has been established by previous works like [1], and the proposed solution is direct and elegant.
  2. Necessary comparison experiments are provided. The authors compare their method with SFT and other latent visual reasoning methods (LIVR) using the same training data.
  3. The analysis experiments help to reveal the mechanism of the proposed method better. The similarity and intervention experiments demonstrate that latent embeddings learn different and diverse features. The ablation studies validate the effectiveness of each component.
  4. The training process does not require direct supervision on the latents, such as intermediate-step images.

[1] Imagination Helps Visual Reasoning, But Not Yet in Latent Space, ICML 2026

Critical weaknesses
  1. The claim that "the latents become more complementary" lacks direct support. The results in Figure 3 (a) only demonstrate that different latent embeddings learn different features, but they cannot prove that these features are "complementary".
  2. The contribution of this work should be further clarified. The core design seems to be a variant of the conventional MoE, while the two-stage training is adopted from LIVR.
  3. The evaluation results are limited to perception tasks. Results on other domains, including math, charts, puzzles and other visual-reasoning benchmarks, are absent, even though the training data has included similar tasks (MathV360K, etc.). Moreover, even on these perception benchmarks, there exist methods using visual tools like cropping that have achieved much stronger results with the same base model (Qwen2.5-VL-7B) as this paper (e.g., Deepeyes [1] achieves 91.3 on V*).

[1] DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning, ICLR 2026

Questions for authors

Please see the weakness.

I would like to raise my rating if these concerns are addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.