- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LLaVA-MoR, which assigns different recursive computation depths to individual visual tokens in Multimodal Large Language Models to improve the trade-off between reasoning performance and computational efficiency. Specifically, it introduces Cosine-Guided Multi-Depth Adaptation to help the model adapt to different recursion depths and trains a Question-Conditioned Visual Router to select which visual tokens should be used for additional processing at each depth. Through extensive experiments with various language backbones, LLaVA-MoR demonstrates a favorable trade-off between reasoning performance and inference cost without increasing the backbone size.
Critical strengths
- This paper proposes LLaVA-MoR, which assigns different recursive computation depths to individual visual tokens in Multimodal Large Language Models to improve the trade-off between reasoning performance and computational efficiency. Specifically, it introduces Cosine-Guided Multi-Depth Adaptation to help the model adapt to different recursion depths and trains a Question-Conditioned Visual Router to select which visual tokens should be used for additional processing at each depth. Through extensive experiments with various language backbones, LLaVA-MoR demonstrates a favorable trade-off between reasoning performance and inference cost without increasing the backbone size.
- Another strength is that the paper builds on the parameter-sharing architecture of Mixture-of-Recursions (MoR) and introduces Cosine-Guided Multi-Depth Adaptation to effectively train the shared backbone across different recursion depths without increasing the number of backbone parameters. In particular, the proposed cosine-based depth scheduling offers a structured strategy for balancing training across different recursion depths.
- The proposed method shows consistent improvements across multiple language backbones and multimodal reasoning benchmarks. It also achieves higher reasoning accuracy while reducing computational cost and inference latency compared to applying the maximum recursion depth to all visual tokens, highlighting the effectiveness of proposed method.
Critical weaknesses
- The most critical weakness is the lack of comparison with simple baselines that could validate the necessity of a learnable Router. For example, cosine similarity between question features and visual tokens could be directly used to select Top-K tokens and weight the residual updates. Without such comparisons, the necessity of the proposed learnable Visual Router is not convincing.
- The paper lacks key ablation studies to validate the Router's training mechanism and its components. Since Top-K selection is non-differentiable, gradients from the answer prediction loss mainly reach the Router through the score-weighted residual update. Moreover, the auxiliary BCE loss uses the Router's own Top-K indices as supervision targets, making its effectiveness unclear. Without separately ablating these two components, it is difficult to assess their contributions to Router training and overall performance.
- The effectiveness of the proposed method on complex document understanding and reasoning remains unverified. Although the paper claims to improve multimodal reasoning by allocating more recursive computation to question-relevant visual tokens, its experiments mainly focus on general image understanding and visual question answering benchmarks. It remains unclear whether the approach is effective on more challenging document understanding benchmarks, such as InfoVQA and SP-DocVQA, which involve complex layouts, small text, and information distributed across multiple regions.
Questions for authors
- How does the proposed learnable Visual Router compare with simpler alternatives, such as directly using cosine similarity between question features and visual tokens for Top-K selection and residual weighting? Such comparisons are necessary to establish the benefits of a learnable Router.
- To better understand the Router's training mechanism, would it be possible to separately ablate the score-weighted residual update and auxiliary BCE loss? Given the non-differentiability of Top-K selection and the use of self-generated supervision targets, the individual contributions of these components remain unclear.
- It would be valuable to evaluate the proposed method on more challenging document understanding benchmarks, such as InfoVQA and SP-DocVQA. Can the authors demonstrate that the proposed method remains effective when reasoning requires fine-grained text recognition, complex layouts, and information distributed across multiple visual regions?
12 comments
Response to Reviewer qbAk — W2 & Q2: Contributions of Score-Weighted Residual Updates and Auxiliary BCE
Thank you for suggesting separate ablations of score weighting and auxiliary BCE. We ablate each component separately and both together during Stage II, keeping hard Top-K selection in all four configurations. Hard Top-K selection is non-differentiable, so the answer-prediction loss reaches the router through the continuous weights on the selected tokens’ residual updates (Eq. (7) of our manuscript). This combination of discrete selection and continuous weighting follows prior work [1, 2].
Table R2. Stage II ablation of score weighting and auxiliary BCE
| Variant | Score weighting | Auxiliary BCE | SQA-I ↑ | TextVQA ↑ | POPE ↑ | MME ↑ | MMBench ↑ |
|---|---|---|---|---|---|---|---|
| (a) | — | — | 64.3 | 46.0 | 87.1 | 1280 | 62.5 |
| (b) | — | ✓ | 64.2 | 46.2 | 87.3 | 1274 | 62.4 |
| (c) | ✓ | — | 65.3 | 48.7 | 87.9 | 1286 | 63.0 |
| LLaVA-MoR | ✓ | ✓ | 65.5 | 48.9 | 87.8 | 1290 | 63.0 |
✓ indicates an enabled component. — indicates a disabled component.
Score-weighted residual updates. With BCE held fixed, removing score weighting lowers all five metrics, including SQA-I by 1.3 percentage points and TextVQA by 2.7 percentage points. These drops reflect the combined contribution of scaling the selected tokens’ residual updates and passing answer-loss gradients to the router.
Auxiliary BCE. BCE trains the router to reinforce its own Top-K selections. With score weighting held fixed, adding BCE improves SQA-I and TextVQA by 0.2 percentage points each, but slightly lowers POPE. This drop may occur because BCE also reinforces incorrect token selections.
Together, these ablations identify score weighting as the main contributor to router training and performance, while BCE provides modest gains but may reinforce selection errors. We appreciate this suggestion, which has helped us clarify how score weighting and auxiliary BCE contribute to router training and model performance. We will include these ablations and explanations in the revised manuscript.
References
[1] Dai et al. StableMoE: Stable Routing Strategy for Mixture of Experts. ACL 2022.
[2] Zhou et al. Mixture-of-Experts with Expert Choice Routing. NeurIPS 2022.
Response to Reviewer qbAk — W3 & Q3: Additional Document Understanding and Multimodal Reasoning Evaluations
Thank you for recommending these challenging document understanding benchmarks. Following this suggestion, we evaluate on InfoVQA [1], where questions can require linking text to graphics across an infographic, and SP-DocVQA [2], where questions can require locating and reading text in structured document layouts. We compare Dense LLaVA, MoE-LLaVA, and LLaVA-MoR within each of three backbone groups. We also report AI2D, MMMU, and MMStar [3–5] to cover a broader range of recent multimodal reasoning tasks as mentioned by Reviewer k91Y.
Table R7. Additional multimodal reasoning and document understanding benchmarks
| Method | InfoVQA ↑ | SP-DocVQA ↑ | AI2D ↑ | MMMU ↑ | MMStar ↑ |
|---|---|---|---|---|---|
| Qwen-1.8B | |||||
| Dense LLaVA | 15.52 | 18.16 | 45.70 | 29.40 | 32.80 |
| MoE-LLaVA | 17.02 | 18.39 | 45.80 | 32.80 | 34.50 |
| LLaVA-MoR | 17.50 | 18.73 | 47.60 | 33.40 | 35.00 |
| StableLM-2-1.6B | |||||
| Dense LLaVA | 17.21 | 18.95 | 48.80 | 32.00 | 34.20 |
| MoE-LLaVA | 17.33 | 18.24 | 48.45 | 33.22 | 31.87 |
| LLaVA-MoR | 17.48 | 18.96 | 49.55 | 33.60 | 34.80 |
| Phi-2-2.7B | |||||
| Dense LLaVA | 17.82 | 20.68 | 57.90 | 35.00 | 36.00 |
| MoE-LLaVA | 18.74 | 20.96 | 59.94 | 36.67 | 36.20 |
| LLaVA-MoR | 18.88 | 22.00 | 60.46 | 38.73 | 36.70 |
InfoVQA and SP-DocVQA report ANLS × 100. The other benchmarks report accuracy (%). Bold indicates the best result within each backbone group.
LLaVA-MoR scores higher than both baselines on both document benchmarks across all three backbones. For example, with Phi-2, it improves SP-DocVQA from 20.68 to 22.00 over Dense LLaVA and from 20.96 to 22.00 over MoE-LLaVA. On InfoVQA, the Qwen variant improves from 15.52 to 17.50 over Dense LLaVA. The gains vary by backbone, with StableLM nearly tied with Dense LLaVA on SP-DocVQA. These benchmark-level results show that selective visual recursion remains effective on complex document QA tasks.
LLaVA-MoR also leads both baselines on AI2D, MMMU, and MMStar across all three backbones. We appreciate this suggestion, which has helped us strengthen our evaluation by extending it to challenging document understanding tasks. We will add these meaningful evaluation benchmarks to the revised manuscript.
References
[1] Mathew et al. InfographicVQA. WACV 2022.
[2] Mathew, Karatzas, and Jawahar. DocVQA: A Dataset for VQA on Document Images. WACV 2021.
[3] Kembhavi et al. A Diagram Is Worth a Dozen Images. ECCV 2016.
[4] Yue et al. MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. CVPR 2024.
[5] Chen et al. Are We on the Right Way for Evaluating Large Vision-Language Models? NeurIPS 2024.
Response to Reviewer qbAk — W1 & Q1: Benefits over Simple Similarity-Based Routing
Thank you for suggesting these simple routing baselines. We compare cosine similarity and scaled dot product as two training-free routing baselines on Qwen-1.8B, using the default visual-token budget in Section 4.1 of our manuscript.
The baselines differ in how they normalize the similarity scores and convert them into residual weights; both use fixed scoring rules without learnable router parameters.
Table R10. Learnable and similarity-based visual routing
Bold indicates the best score in each column.
LLaVA-MoR outperforms both training-free baselines on all five metrics. Over cosine similarity, the stronger of the two, LLaVA-MoR gains 1.15 percentage points on SQA-I and 3.78 on TextVQA. LLaVA-MoR also improves four metrics over the cross-attention router in Table 2 of our manuscript and ties it on MMBench. These results show that our learnable router outperforms fixed similarity-based routing for token selection and residual weighting. We appreciate this suggestion, which has helped us strengthen the empirical evidence for the benefits of a learnable router. We will add the comparisons and baseline implementation details to the revised manuscript.