acceptodds
Withdrawn from ICLR 2027

MIRA : Multimodal Interleaved Reasoning with Adaptive Visual Tokens

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable semantic understanding. Yet, their reliance on discrete linguistic tokens fundamentally bottlenecks performance on vision-centric tasks that demand fine-grained perceptual understanding, such as spatial localization, depth estimation, and detailed visual recognition. Recent advances in latent reasoning attempt to mitigate this by operating in continuous spaces. However, current latent methods either lack meaningful visual representations or adopt fixed reasoning templates that generate a predetermined set of visual tokens regardless of the input task. To overcome these limitations, we propose MIRA (Multimodal Interleaved Reasoning with Adaptive Visual Tokens), a novel framework that augments the conventional text vocabulary with continuous visual latent tokens distilled from vision foundation models. Our generated sequences seamlessly alternate between text and visual tokens, forming an interpretable interleaved reasoning chain. We further introduce Interleaved Reasoning Policy Optimization (IRPO), which incentivizes the model to actively select task-relevant visual tokens while preserving perceptual fidelity. Extensive evaluations demonstrate that MIRA significantly improves spatial and functional reasoning capabilities across diverse vision-centric benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Multimodal Reasoning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DManOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MIRA builds on the visual special tokens already aligned by CoVT to SAM, DINOv2, and Depth Anything, and trains the model to choose which modalities to invoke for a given question rather than always emitting a fixed set of tokens. Stage 1 uses 43K interleaved reasoning traces synthesized by Gemini 3 Flash, with a loss combining language modeling and visual alignment. Stage 2, IRPO, adds two terms to GRPO: an F1 reward for modality selection computed from Gemini pseudo-labels, and an alignment regularizer that keeps the hidden states decodable. Evaluation covers V*, BLINK, CV-Bench, and HRBench. Relative to Qwen2.5-VL-7B, the average score rises from 69.1 to 76.5 (+7.4), slightly above CrystaL at 74.0. On roughly 500 human annotations, modality-selection micro-F1 improves from 0.64 for CoVT to 0.81.

Critical strengths

Modality selection is measured on its own, and the evaluation labels are separate from the Gemini pseudo-labels used in training. On human annotations, the token-selection reward raises micro-F1 from 0.69 to 0.81, with the largest gains on DINO and Depth. The token distributions in Figure 3 also separate counting, art style, and relative depth. The appendix reports that the Gemini pseudo-labels reach a micro-F1 of 0.91 against human annotations and includes a label-noise study. This part of the evidence is concrete.

Critical weaknesses
  1. The additional gain on downstream tasks is too small to support the claim that adaptive visual reasoning substantially improves vision-centric performance. Under the same Stage-1 data and training budget, MIRA improves over CoVT only from 72.6 to 74.6. Within IRPO, Stage 1 scores 74.6, standard GRPO 75.6, alignment regularization 76.0, the token reward 76.1, and both together 76.5. Most of the +7.4 over the base model comes from SFT, and those SFT traces are written by Gemini. The token reward further optimizes Gemini’s modality preferences. The main-table gain can therefore come from stronger text traces, without the visual hidden states participating in subsequent reasoning.
  2. The visual tokens are discrete special tokens in the vocabulary. The model emits symbols such as <|sam pad|>, and the corresponding hidden states are aligned to a foundation model. The optional decoder is outside the reasoning loop. The paper provides no intervention showing that these hidden states change later token predictions, for example by swapping depth and DINO hidden states and checking whether the answer changes. In the qualitative examples, the text already states the conclusion, so the visual tokens read as labeled perceptual steps rather than continuous visual computation that has been shown to be used.

I would like to raise my score if the authors can solve my concerns.

Questions for authors
  1. Please replace the Gemini pseudo-labels with human modality labels on the RL training set, rerun IRPO, and report both the downstream average and the selection F1. If the gain drops substantially, the current improvement is mainly imitation of Gemini.
  2. Please run a causal intervention: keep the text prefix fixed and replace the hidden state of a generated visual token with another modality or a random vector, then measure the change in answer accuracy. If the change is near zero, adaptive selection changes only the surface trace and does not enter the subsequent computation.
Reviewer_vavgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MIRA, a multimodal reasoning framework that dynamically interleaves text with task-relevant visual latent tokens (segmentation, depth, and semantic features), trained through supervised fine-tuning and a reinforcement learning method.

Critical strengths

S1. Addressing the importance of task-based feature selection

Critical weaknesses

W1. lack of apples to apple experimental comparisons

  • The claimed effectiveness of task-based feature selection is not sufficiently supported by the experimental results. The textual reasoning trace quality of MIRA train data, which is critical for downstream task performance, looks much better than other methods like CoVT.

W2. limited impact of the token selection reward in the ablation study

  • The ablation study suggests that the token selection reward provides only marginal improvements.

W3. limited novelty and contribution of IRPO

  • IRPO appears to be a straightforward extension of GRPO, incorporating token selection rewards into the reward function and adding an SFT-based regularization loss.
Questions for authors

Q1. Comparing with VaLR [1], which use multiple encoder for the single visual latent alignment, what would be the strength of MIRA.

  • As the VaLR do not need the per task encoder task label, the clear strength including the benchmark score or efficiency metrics shall be presented.

Q2. W1. To isolate the contribution of task-specific latent representations from adaptive reasoning, an additional ablation using task-specific visual tokens within CoVT's simpler textual reasoning template would be necessary.

Q3. W2. table 4 shows a substantial improvement in the F1 score, while the benchmark score does not. Further analysis of the task-misclassified samples, including their scores, would help explain this discrepancy and provide deeper insights into the model's performance.

[1] Jeon, Byungwoo, et al. "Vision-aligned latent reasoning for multi-modal large language model." arXiv preprint arXiv:2602.04476 (2026).
Reviewer_9uvkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MIRA, which extends CoVT’s vision-aligned tokens with adaptive interleaving of text, segmentation, depth, and DINO representations. It uses 43K teacher-generated traces for supervised fine-tuning, followed by GRPO with a modality-selection reward and visual alignment regularization. On the reported benchmark average, MIRA achieves 76.5, compared with 69.1 for Qwen2.5-VL-7B and 74.0 for CrystaL.

Critical strengths
  1. Consistent overall gains. MIRA improves over the base model across all six benchmark settings. The matched Stage-1 comparison with CoVT also shows a two-point average gain under the same image-question pairs and training budget (Table 6), making the comparison more informative than the main results alone.

  2. Useful component ablations. Table 3 separates standard GRPO, the token-selection reward, and alignment regularization. Their combination improves the average from 75.6 to 76.5, with results reported over three seeds. This supports the contribution of both additions beyond ordinary RL.

Critical weaknesses
  1. The initialization and attribution of gains are unclear. Section 4.1 describes a Qwen2.5-VL-7B-based checkpoint with CoVT’s visual-token vocabulary, but does not identify its source or inherited weights. If MIRA starts from a trained CoVT checkpoint, the main comparison combines the proposed method with additional training, while part of the gain over Qwen may come from CoVT itself. The matched Stage-1 comparison in Table 6 helps, but does not resolve this issue for the full SFT+RL pipeline.

  2. The methodological contribution appears narrow and incremental. CoVT’s paradigm is merely one design choice of latent reasoning rather than a dominant design. MIRA mainly adds modality selection and interleaved reasoning to CoVT’s aligned-token framework. Although the results show improvements, they do not establish why this particular design is needed over other alternatives.

  3. Modality agreement does not establish faithful visual reasoning. Matching a human-selected modality set shows sensible routing, but not that the aligned visual content drives the answer. The qualitative examples do not establish the causal claims made about particular token choices. No evaluation of intervention on the latent representations is reported.

Questions for authors
  1. Could you identify the exact initialization checkpoint and specify which backbone weights, token embeddings, projection heads, and decoders are inherited from CoVT? Please also clarify whether the controlled CoVT baseline uses the same initialization.

  2. Could you clarify the central methodological contribution beyond modifying CoVT’s fixed template, and provide evidence separating the benefit of adaptive modality selection from that of additional reasoning supervision?

  3. Does answer accuracy drop when the latent visual representations are masked or swapped while preserving the token identities and preceding text?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.