- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MIRA builds on the visual special tokens already aligned by CoVT to SAM, DINOv2, and Depth Anything, and trains the model to choose which modalities to invoke for a given question rather than always emitting a fixed set of tokens. Stage 1 uses 43K interleaved reasoning traces synthesized by Gemini 3 Flash, with a loss combining language modeling and visual alignment. Stage 2, IRPO, adds two terms to GRPO: an F1 reward for modality selection computed from Gemini pseudo-labels, and an alignment regularizer that keeps the hidden states decodable. Evaluation covers V*, BLINK, CV-Bench, and HRBench. Relative to Qwen2.5-VL-7B, the average score rises from 69.1 to 76.5 (+7.4), slightly above CrystaL at 74.0. On roughly 500 human annotations, modality-selection micro-F1 improves from 0.64 for CoVT to 0.81.
Critical strengths
Modality selection is measured on its own, and the evaluation labels are separate from the Gemini pseudo-labels used in training. On human annotations, the token-selection reward raises micro-F1 from 0.69 to 0.81, with the largest gains on DINO and Depth. The token distributions in Figure 3 also separate counting, art style, and relative depth. The appendix reports that the Gemini pseudo-labels reach a micro-F1 of 0.91 against human annotations and includes a label-noise study. This part of the evidence is concrete.
Critical weaknesses
- The additional gain on downstream tasks is too small to support the claim that adaptive visual reasoning substantially improves vision-centric performance. Under the same Stage-1 data and training budget, MIRA improves over CoVT only from 72.6 to 74.6. Within IRPO, Stage 1 scores 74.6, standard GRPO 75.6, alignment regularization 76.0, the token reward 76.1, and both together 76.5. Most of the +7.4 over the base model comes from SFT, and those SFT traces are written by Gemini. The token reward further optimizes Gemini’s modality preferences. The main-table gain can therefore come from stronger text traces, without the visual hidden states participating in subsequent reasoning.
- The visual tokens are discrete special tokens in the vocabulary. The model emits symbols such as <|sam pad|>, and the corresponding hidden states are aligned to a foundation model. The optional decoder is outside the reasoning loop. The paper provides no intervention showing that these hidden states change later token predictions, for example by swapping depth and DINO hidden states and checking whether the answer changes. In the qualitative examples, the text already states the conclusion, so the visual tokens read as labeled perceptual steps rather than continuous visual computation that has been shown to be used.
I would like to raise my score if the authors can solve my concerns.
Questions for authors
- Please replace the Gemini pseudo-labels with human modality labels on the RL training set, rerun IRPO, and report both the downstream average and the selection F1. If the gain drops substantially, the current improvement is mainly imitation of Gemini.
- Please run a causal intervention: keep the text prefix fixed and replace the hidden state of a generated visual token with another modality or a random vector, then measure the change in answer accuracy. If the change is near zero, adaptive selection changes only the surface trace and does not enter the subsequent computation.