acceptodds
Under review as a conference paper at ICLR 2027

LLaVA-MoR: Adaptive Visual Recursion for Multimodal Reasoning

Abstract

Multimodal Large Language Models (MLLMs) such as LLaVA have demonstrated promising capabilities in visual understanding and instruction following. Although allocating reasoning budgets according to problem difficulty has achieved better accuracy-efficiency trade-offs in text reasoning, whether and how to distribute adaptive budgets across visual tokens remains an open question in multimodal scenarios. In this work, we introduce LLaVA-MoR, the first approach to allocate question-conditioned recursion budgets to visual tokens in instruction-tuned MLLMs, optimizing multimodal reasoning by effectively preventing the model from overthinking simple visual regions and under-analyzing complex ones. To achieve this, we first propose the Question-Conditioned Visual Router, which explicitly models question–visual interactions to route visual tokens to different recursion depths. Furthermore, we propose Cosine-Guided Multi-Depth Adaptation, which gradually increases exposure to deeper recursion under answer supervision to adapt the backbone for visual processing while preserving its language capabilities. Extensive experiments demonstrate that LLaVA-MoR achieves a superior accuracy-efficiency trade-off across multiple backbones, even outperforming fixed maximum-depth recursion. These results highlight the critical value of token-level budget allocation in multimodal reasoning tasks. Code is available at \url{https://anonymous.4open.science/r/LLaVA_MoR/

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
large vision language model, looped transformer model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qbAkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LLaVA-MoR, which assigns different recursive computation depths to individual visual tokens in Multimodal Large Language Models to improve the trade-off between reasoning performance and computational efficiency. Specifically, it introduces Cosine-Guided Multi-Depth Adaptation to help the model adapt to different recursion depths and trains a Question-Conditioned Visual Router to select which visual tokens should be used for additional processing at each depth. Through extensive experiments with various language backbones, LLaVA-MoR demonstrates a favorable trade-off between reasoning performance and inference cost without increasing the backbone size.

Critical strengths
  • This paper proposes LLaVA-MoR, which assigns different recursive computation depths to individual visual tokens in Multimodal Large Language Models to improve the trade-off between reasoning performance and computational efficiency. Specifically, it introduces Cosine-Guided Multi-Depth Adaptation to help the model adapt to different recursion depths and trains a Question-Conditioned Visual Router to select which visual tokens should be used for additional processing at each depth. Through extensive experiments with various language backbones, LLaVA-MoR demonstrates a favorable trade-off between reasoning performance and inference cost without increasing the backbone size.
  • Another strength is that the paper builds on the parameter-sharing architecture of Mixture-of-Recursions (MoR) and introduces Cosine-Guided Multi-Depth Adaptation to effectively train the shared backbone across different recursion depths without increasing the number of backbone parameters. In particular, the proposed cosine-based depth scheduling offers a structured strategy for balancing training across different recursion depths.
  • The proposed method shows consistent improvements across multiple language backbones and multimodal reasoning benchmarks. It also achieves higher reasoning accuracy while reducing computational cost and inference latency compared to applying the maximum recursion depth to all visual tokens, highlighting the effectiveness of proposed method.
Critical weaknesses
  • The most critical weakness is the lack of comparison with simple baselines that could validate the necessity of a learnable Router. For example, cosine similarity between question features and visual tokens could be directly used to select Top-K tokens and weight the residual updates. Without such comparisons, the necessity of the proposed learnable Visual Router is not convincing.
  • The paper lacks key ablation studies to validate the Router's training mechanism and its components. Since Top-K selection is non-differentiable, gradients from the answer prediction loss mainly reach the Router through the score-weighted residual update. Moreover, the auxiliary BCE loss uses the Router's own Top-K indices as supervision targets, making its effectiveness unclear. Without separately ablating these two components, it is difficult to assess their contributions to Router training and overall performance.
  • The effectiveness of the proposed method on complex document understanding and reasoning remains unverified. Although the paper claims to improve multimodal reasoning by allocating more recursive computation to question-relevant visual tokens, its experiments mainly focus on general image understanding and visual question answering benchmarks. It remains unclear whether the approach is effective on more challenging document understanding benchmarks, such as InfoVQA and SP-DocVQA, which involve complex layouts, small text, and information distributed across multiple regions.
Questions for authors
  • How does the proposed learnable Visual Router compare with simpler alternatives, such as directly using cosine similarity between question features and visual tokens for Top-K selection and residual weighting? Such comparisons are necessary to establish the benefits of a learnable Router.
  • To better understand the Router's training mechanism, would it be possible to separately ablate the score-weighted residual update and auxiliary BCE loss? Given the non-differentiability of Top-K selection and the use of self-generated supervision targets, the individual contributions of these components remain unclear.
  • It would be valuable to evaluate the proposed method on more challenging document understanding benchmarks, such as InfoVQA and SP-DocVQA. Can the authors demonstrate that the proposed method remains effective when reasoning requires fine-grained text recognition, complex layouts, and information distributed across multiple visual regions?
12 comments
AuthorsOpenReview ↗

Response to Reviewer qbAk — W1 & Q1: Benefits over Simple Similarity-Based Routing

Thank you for suggesting these simple routing baselines. We compare cosine similarity and scaled dot product as two training-free routing baselines on Qwen-1.8B, using the default visual-token budget in Section 4.1 of our manuscript.

  • Cosine similarity: We divide the question–visual dot product by the product of the two vector norms, then linearly rescale and clip the scores to obtain residual weights between 0 and 1.
  • Scaled dot product: We divide the question–visual dot product by the square root of the feature dimension, then apply a sigmoid to obtain residual weights.

The baselines differ in how they normalize the similarity scores and convert them into residual weights; both use fixed scoring rules without learnable router parameters.

Table R10. Learnable and similarity-based visual routing

Routing methodSQA-I ↑TextVQA ↑POPE ↑MME ↑MMBench ↑
Cosine similarity (training-free)64.3545.1287.101284.962.16
Scaled dot product (training-free)64.1545.0987.031276.162.07
Cross-attention (Table 2 of our manuscript)65.1047.0087.401280.063.00
LLaVA-MoR65.5048.9087.801290.063.00

Bold indicates the best score in each column.

LLaVA-MoR outperforms both training-free baselines on all five metrics. Over cosine similarity, the stronger of the two, LLaVA-MoR gains 1.15 percentage points on SQA-I and 3.78 on TextVQA. LLaVA-MoR also improves four metrics over the cross-attention router in Table 2 of our manuscript and ties it on MMBench. These results show that our learnable router outperforms fixed similarity-based routing for token selection and residual weighting. We appreciate this suggestion, which has helped us strengthen the empirical evidence for the benefits of a learnable router. We will add the comparisons and baseline implementation details to the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer qbAk — W2 & Q2: Contributions of Score-Weighted Residual Updates and Auxiliary BCE

Thank you for suggesting separate ablations of score weighting and auxiliary BCE. We ablate each component separately and both together during Stage II, keeping hard Top-K selection in all four configurations. Hard Top-K selection is non-differentiable, so the answer-prediction loss reaches the router through the continuous weights on the selected tokens’ residual updates (Eq. (7) of our manuscript). This combination of discrete selection and continuous weighting follows prior work [1, 2].

Table R2. Stage II ablation of score weighting and auxiliary BCE

VariantScore weightingAuxiliary BCESQA-I ↑TextVQA ↑POPE ↑MME ↑MMBench ↑
(a)——64.346.087.1128062.5
(b)—✓64.246.287.3127462.4
(c)✓—65.348.787.9128663.0
LLaVA-MoR✓✓65.548.987.8129063.0

✓ indicates an enabled component. — indicates a disabled component.

Score-weighted residual updates. With BCE held fixed, removing score weighting lowers all five metrics, including SQA-I by 1.3 percentage points and TextVQA by 2.7 percentage points. These drops reflect the combined contribution of scaling the selected tokens’ residual updates and passing answer-loss gradients to the router.

Auxiliary BCE. BCE trains the router to reinforce its own Top-K selections. With score weighting held fixed, adding BCE improves SQA-I and TextVQA by 0.2 percentage points each, but slightly lowers POPE. This drop may occur because BCE also reinforces incorrect token selections.

Together, these ablations identify score weighting as the main contributor to router training and performance, while BCE provides modest gains but may reinforce selection errors. We appreciate this suggestion, which has helped us clarify how score weighting and auxiliary BCE contribute to router training and model performance. We will include these ablations and explanations in the revised manuscript.

References

[1] Dai et al. StableMoE: Stable Routing Strategy for Mixture of Experts. ACL 2022.

[2] Zhou et al. Mixture-of-Experts with Expert Choice Routing. NeurIPS 2022.

AuthorsOpenReview ↗

Response to Reviewer qbAk — W3 & Q3: Additional Document Understanding and Multimodal Reasoning Evaluations

Thank you for recommending these challenging document understanding benchmarks. Following this suggestion, we evaluate on InfoVQA [1], where questions can require linking text to graphics across an infographic, and SP-DocVQA [2], where questions can require locating and reading text in structured document layouts. We compare Dense LLaVA, MoE-LLaVA, and LLaVA-MoR within each of three backbone groups. We also report AI2D, MMMU, and MMStar [3–5] to cover a broader range of recent multimodal reasoning tasks as mentioned by Reviewer k91Y.

Table R7. Additional multimodal reasoning and document understanding benchmarks

MethodInfoVQA ↑SP-DocVQA ↑AI2D ↑MMMU ↑MMStar ↑
Qwen-1.8B
Dense LLaVA15.5218.1645.7029.4032.80
MoE-LLaVA17.0218.3945.8032.8034.50
LLaVA-MoR17.5018.7347.6033.4035.00
StableLM-2-1.6B
Dense LLaVA17.2118.9548.8032.0034.20
MoE-LLaVA17.3318.2448.4533.2231.87
LLaVA-MoR17.4818.9649.5533.6034.80
Phi-2-2.7B
Dense LLaVA17.8220.6857.9035.0036.00
MoE-LLaVA18.7420.9659.9436.6736.20
LLaVA-MoR18.8822.0060.4638.7336.70

InfoVQA and SP-DocVQA report ANLS × 100. The other benchmarks report accuracy (%). Bold indicates the best result within each backbone group.

LLaVA-MoR scores higher than both baselines on both document benchmarks across all three backbones. For example, with Phi-2, it improves SP-DocVQA from 20.68 to 22.00 over Dense LLaVA and from 20.96 to 22.00 over MoE-LLaVA. On InfoVQA, the Qwen variant improves from 15.52 to 17.50 over Dense LLaVA. The gains vary by backbone, with StableLM nearly tied with Dense LLaVA on SP-DocVQA. These benchmark-level results show that selective visual recursion remains effective on complex document QA tasks.

LLaVA-MoR also leads both baselines on AI2D, MMMU, and MMStar across all three backbones. We appreciate this suggestion, which has helped us strengthen our evaluation by extending it to challenging document understanding tasks. We will add these meaningful evaluation benchmarks to the revised manuscript.

References

[1] Mathew et al. InfographicVQA. WACV 2022.

[2] Mathew, Karatzas, and Jawahar. DocVQA: A Dataset for VQA on Document Images. WACV 2021.

[3] Kembhavi et al. A Diagram Is Worth a Dozen Images. ECCV 2016.

[4] Yue et al. MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. CVPR 2024.

[5] Chen et al. Are We on the Right Way for Evaluating Large Vision-Language Models? NeurIPS 2024.

Reviewer_k91YOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes LLaVA-MoR, an adaptive visual depth recursion for multimodal reasoning. To handle two main challenges (performance degradation under shared parameters and question-agnostic budget allocation), LLaVA-MoR goes through two training stages, Cosine-Guided Multi-Depth Adaptation and Question-Conditioned Visual Router. LLaVA-MoR shows improved performance compared to other conditional computation methods across various tasks under varying backbones. Also, extensive experiments support LLaVA-MoR’s design choices and effectiveness.

Critical strengths
  1. Extensive experiments under various backbones. The method is evaluated on various tasks under four different language backbones. This supports that the method is not limited to a specific backbone.

  2. Timely and insightful research direction. Recursion research in the MLLM domain is timely and insightful. Also, their motivation is intriguing and the method is generally reasonable.

Critical weaknesses
  1. Motivational problems and the proposed method are not logically connected. First, the paper claims that shared parameters degrade performance, so the comparison should be against default LLaVA and other recursive methods. However, the naive recursion without adaptation already outperforms dense LLaVA on SQA-I (Fig. 1(a), Table 2). Also, the key proposal is adaptively allocating depth under shared parameters, while a natural idea would be to directly address the shared-parameter problem, as in Relaxed Recursive Transformers [1]. Moreover, their motivation has already been addressed by previous works such as RecursiveVLM [2] and Visual Enhanced Depth Scaling [3]. The remaining novelty would be the query-conditioned visual router, but it is similar to visual token selection in the pruning and compression literature.

  2. Backbones and benchmarks are outdated. All language backbones (Qwen, StableLM-2, Phi-2, Vicuna) date from 2023 to early 2024, while the Qwen series has moved on to Qwen3-VL and Qwen3.5. Also, their evaluation is the same as LLaVA-1.5, so many recent multimodal reasoning benchmarks such as AI2D, MathVista, MMMU, and MMStar should be included. The current results show some comparison with other methods, but give little insight or takeaway for future research on recent advanced architectures and reasoning-heavy benchmarks.

[1] Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA, Bae et al., ICLR'25

[2] Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Models, Xu et al., Arxiv'26

[3] Visual Enhanced Depth Scaling for Multimodal Latent Reasoning, Han et al., Arxiv'26

Questions for authors
  1. In Stage I, all tokens share the sampled depth, but in Stage II and inference, visual tokens exit at different depths with score-weighted updates while the backbone is frozen. How does the backbone handle this mismatch?

  2. How is the KV cache handled during decoding? Visual tokens that exited at depth 1 or 2 have no KV cache at depth 3, so how do later tokens attend to them, and what is the decoding cost beyond prefill?

8 comments
AuthorsOpenReview ↗

Response to Reviewer k91Y — W1: Connection between Motivation and Method

We thank the reviewer for the assessment and suggestions. Our central insight is that the value of further processing a visual region depends on the question as well as the region itself. This motivates two linked requirements: preparing an instruction-tuned backbone for repeated use at different depths, and learning which visual tokens benefit from additional updates. LLaVA-MoR addresses them through multi-depth backbone adaptation followed by question-conditioned visual routing under fixed token budgets.

Naive MoR improves SQA-I but scores lower on four other benchmarks. In Table 2 of our manuscript, naive MoR improves SQA-I from 61.5 to 63.6 relative to Dense LLaVA, while scores decline on MMBench (60.2 → 55.4), MME (1288 → 1157), TextVQA (45.7 → 45.1), and POPE (86.1 → 85.4). These mixed results motivate multi-depth adaptation to preserve performance across tasks.

Stage I directly addresses capability losses following recursive conversion, while Stage II learns where to allocate visual computation for each question. Stage I updates the shared backbone and projector under answer supervision, gradually introducing deeper recursion through a cosine depth-sampling schedule. In Table 2 of our manuscript, adding Stage I to naive MoR raises MMBench from 55.4 to 62.0 and MME from 1157 to 1279. Adding Stage I before training the question-conditioned router in Stage II improves SQA-I from 64.3 to 65.5 and TextVQA from 46.4 to 48.9 compared with Stage II alone. This supports the complementary roles of the two stages.

Our distinction from the cited recursive methods is to explicitly base each visual region's recursion budget on its interaction with the question: the same region may need different refinement for different questions. This design draws inspiration from human multimodal visual understanding. Relaxed Recursive Transformers [1] relaxes weight tying through depth-specific LoRA, while RecursiveVLM [2] uses a modality-specific connector and a monotonic recursion loss to improve sequence-wide refinement. These two approaches address the effectiveness of recursion without specifying question-dependent budgets for individual visual regions. Visual Enhanced Depth Scaling [3] goes further by routing additional per-layer computation according to contextual token states. Our distinction is to explicitly model the interaction between the question and each visual region and use it to allocate recursion depths to original visual tokens.

LLaVA-MoR implements this insight through an answer-supervised policy that assigns different recursion depths to original visual tokens under fixed budgets. At each shared-stack recursion boundary, the router combines pooled question states with each active visual-token state through an elementwise product and scores which tokens should receive further updates. Thus, changing the question can change the regions selected for deeper recursion even when the image and token budgets are unchanged. All text tokens continue processing. Stage I prepares the shared backbone for multiple depths, and Stage II learns this allocation with the backbone frozen. The resulting design requires neither depth-specific backbone parameters nor visual replay or a CoT internalization curriculum.

Relative to question-aware scoring and Top-K token selection in pruning/compression, our contribution lies in using these operations to learn question-conditioned recursion budgets for visual tokens. All valid tokens participate in the first recursion. At each subsequent recursion depth, the router selects visual tokens for further processing through the shared block. Exited tokens retain their latest representations for the final output block (Appendix B.1 of our manuscript). Meanwhile, question-conditioned routing improves over visual-only routing at matched budgets, and the complete method offers a better reported accuracy–compute trade-off than full visual recursion. With the same visual-token budgets and maximum depth, SQA-I rises from 64.0 to 65.5 and TextVQA from 45.1 to 48.9 (Table 2 of our manuscript). Fig. 1(b) of our manuscript illustrates how changing the question changes the regions selected from the same image. Table 3 of our manuscript further reports 65.5 SQA-I accuracy at 2.55 prefill TFLOPs, versus 63.8 at 4.47 TFLOPs for full visual recursion.

Our two-stage design directly addresses our motivation: adapting the backbone to shared recursion and allocating visual computation according to the question.

References

[1] Bae et al. Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA. ICLR 2025.

[2] Xu et al. Looping Back to Move Forward: Recursive Transformers for Efficient and Flexible Large Multimodal Models. arXiv 2026.

[3] Han et al. Visual Enhanced Depth Scaling for Multimodal Latent Reasoning. arXiv 2026.

AuthorsOpenReview ↗

Response to Reviewer k91Y — W2: Evaluation on Recent Advanced Architectures and Reasoning-Heavy Benchmarks

Thank you for highlighting the need to evaluate newer architectures and broaden benchmark coverage.

Backbones. Adding more benchmarks helps us comprehensively assess our method. We have extended LLaVA-MoR to Qwen3-VL-2B-Instruct [6] and compared it with the Qwen3-VL-2B-Instruct baseline on both the original and additional benchmarks.

Qwen3-VL’s DeepStack injects multi-level visual features after the first three decoder layers, in addition to the input visual embeddings. Following the block decomposition in Section 2.2, Eq. (3), and Appendix B.1 of our manuscript, we expand FinF_{\mathrm{in}} to include these early layers and their DeepStack injections, executing this block once before recursion. We apply shared recursion to the subsequent intermediate layers F(⋅;θ)F(\cdot;\theta), with FoutF_{\mathrm{out}} remaining outside the recursive block.

Table R8. LLaVA-MoR on Qwen3-VL-2B-Instruct

BenchmarkBaselineLLaVA-MoR
Qwen3-VL-2B-Instruct
SQA-I83.6487.56
TextVQA77.6377.46
POPE89.0088.78
MME-Perception1495.201548.18
MME-Cognition539.64564.64
MMBench75.3478.49
AI2D76.0078.34
MMMU41.0047.22
MMStar44.7354.60

POPE reports macro F1 (%); MME reports raw perception/cognition scores; the remaining metrics are accuracy (%). Higher is better; best values are bold.

Qwen3-VL-based LLaVA-MoR improves AI2D, MMMU, and MMStar by 2.34, 6.22, and 9.87 percentage points, respectively. TextVQA and POPE are slightly lower by 0.17 and 0.22 points. These results support the applicability of our method to the tested newer architecture, with gains varying across tasks.

Benchmarks. We appreciate this valuable suggestion, which has helped us broaden our evaluation of multimodal reasoning. Accordingly, we add AI2D [1], MMMU [2], and MMStar [3]. We also include InfoVQA [4] and SP-DocVQA [5] in response to Reviewer qbAk’s request for document-understanding evaluations. Below, we compare Dense LLaVA, MoE-LLaVA, and LLaVA-MoR within each original backbone group.

Table R7. Additional multimodal reasoning and document understanding benchmarks

MethodAI2D ↑MMMU ↑MMStar ↑InfoVQA ↑SP-DocVQA ↑
Qwen-1.8B
Dense LLaVA45.7029.4032.8015.5218.16
MoE-LLaVA45.8032.8034.5017.0218.39
LLaVA-MoR47.6033.4035.0017.5018.73
StableLM-2-1.6B
Dense LLaVA48.8032.0034.2017.2118.95
MoE-LLaVA48.4533.2231.8717.3318.24
LLaVA-MoR49.5533.6034.8017.4818.96
Phi-2-2.7B
Dense LLaVA57.9035.0036.0017.8220.68
MoE-LLaVA59.9436.6736.2018.7420.96
LLaVA-MoR60.4638.7336.7018.8822.00

InfoVQA and SP-DocVQA report ANLS × 100; the other benchmarks report accuracy (%). Best values within each backbone group are bold.

LLaVA-MoR achieves the highest scores across all five benchmarks in each group. Two aspects of LLaVA-MoR may help explain these reasoning gains. First, repeatedly applying the shared block increases effective depth, which can improve reasoning, as shown in prior work on looped Transformers [7]. Second, our question-conditioned router selectively allocates different recursion budgets to visual tokens based on their relevance to the question.

Our work studies how parameter-shared recursion can expand visual computation in multimodal models. These experiments suggest a design direction for future VLM architectures: allocating visual processing depth according to the question, with evidence of its applicability on a newer architecture and a broader range of reasoning tasks. We appreciate these suggestions, which have helped us assess our method on the advanced architecture and strengthen the evaluation of its reasoning capabilities.

References

[1] Kembhavi et al. A Diagram Is Worth a Dozen Images. ECCV 2016.

[2] Yue et al. MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI. CVPR 2024.

[3] Chen et al. Are We on the Right Way for Evaluating Large Vision-Language Models? NeurIPS 2024.

[4] Mathew et al. InfographicVQA. WACV 2022.

[5] Mathew, Karatzas, and Jawahar. DocVQA: A Dataset for VQA on Document Images. WACV 2021.

[6] Qwen Team. Qwen3-VL Technical Report. arXiv 2025.

[7] Saunshi et al. Reasoning with Latent Thoughts: On the Power of Looped Transformers. ICLR 2025.

AuthorsOpenReview ↗

Response to Reviewer k91Y — Q1: Handling the Mismatch between Sample-wise Adaptation and Token-wise Routing

Thank you for raising the question of how the backbone handles the transition between the two stages. In our paper, Stage I adapts the pretrained backbone to the parameter-shared recursive architecture at the sample level (analogous to a warm-up process; Section 3.2 of our manuscript). Stage II shifts to token-level routing because the amount of visual processing needed varies across regions and depends on the question. With the backbone frozen, the router learns which visual tokens should receive additional recursive updates, improving visual reasoning while saving computation on tokens that exit early.

Q1.1. Sample-wise vs. token-wise Stage I

Therefore, we first test whether introducing token-level recursion in Stage I better prepares the backbone for Stage II routing. We compare the original sample-wise setting, which assigns one recursion depth to all tokens in a sample, with a token-wise variant that samples a depth for each token. Stage II and inference use the same settings in both cases.

Table R9. Sample-wise and token-wise depth sampling in Stage I

Stage I samplingSQA-I ↑TextVQA ↑POPE ↑MME ↑MMBench ↑
Token-wise depth sampling62.344.787.2120157.4
Sample-wise depth sampling (LLaVA-MoR)65.548.987.8129063.0

Sample-wise Stage I scores higher on all five metrics, including gains of 3.2 points on SQA-I, 4.2 on TextVQA, and 5.6 on MMBench. We attribute this gap to the abrupt change in the pretrained backbone’s execution pattern when different recursion depths are assigned to individual tokens. Sample-wise sampling keeps the depth uniform within each sample and gradually exposes the backbone to deeper passes, providing a more effective warm-up before Stage II introduces token-level routing.

Q1.2. Matched Stage II training and inference

To examine whether the sample-wise-adapted backbone supports token-wise routing, we compare selective and full recursion under their respective training configurations. We add a full visual recursion baseline whose Stage II training and inference both retain all visual tokens.

LLaVA-MoR (selective recursion) uses selective visual-token budgets during Stage II training and inference: recursion 2 retains 50% of the visual tokens, and recursion 3 retains 20% of those remaining. Full visual recursion trains a Stage II router with the Stage I backbone frozen and retains all visual tokens at every recursion during both training and inference; the router scores only scale the residual updates. Both methods retain score-weighted residual updates and use a maximum of three recursions.

The LLaVA-MoR and naïve MoR scores are the original results from Table 2 of our manuscript; the full visual recursion scores are from the additional experiment.

Table R3. Selective and full recursion with matched training configuration

ConfigurationSQA-I ↑TextVQA ↑POPE ↑MME ↑MMBench ↑
Full visual recursion63.9‾\underline{63.9}46.3‾\underline{46.3}87.81278‾\underline{1278}62.5‾\underline{62.5}
Naïve MoR (Table 2 of our manuscript)63.645.185.4‾\underline{85.4}115755.4
LLaVA-MoR (selective recursion; Table 2 of our manuscript)65.548.987.8129063.0

Bold and underlining indicate the best and second-best scores in each column, respectively.

LLaVA-MoR scores higher than full visual recursion on four of five metrics, including gains of 1.6 percentage points on SQA-I and 2.6 on TextVQA, while matching its POPE score. This shows that the sample-wise-adapted backbone can support selective, score-weighted execution once the router is trained under that same execution pattern.

Together, these results support the two-stage design. Sample-wise Stage I provides a warm-up for shared recursion, and Stage II learns token-wise early exits and residual weights on the frozen backbone under the same rules used at inference. We will clarify this transition and add both comparisons to the revised manuscript.

Reviewer_XqXDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LLaVA-MoR, which adds question-conditioned visual recursion to instruction-tuned multimodal models. A cosine-based training schedule first adapts the backbone to different recursion depths. A lightweight router then selects visual tokens for further processing under fixed token budgets, while all text tokens remain active. Experiments across four language backbones show improvements on several multimodal benchmarks, particularly ScienceQA-IMG and MMBench. On Qwen-1.8B, the method also improves accuracy while reducing prefill computation and latency compared with full-depth visual recursion.

Critical strengths
  1. The method improves ScienceQA-IMG and MMBench across all four backbones without expanding the language backbone. This makes the results more convincing than improvements on a single model.

  2. The ablations support both adaptation and question-conditioned routing. The efficiency results also show a practical benefit over full recursion, with prefill latency dropping from 58.0 to 45.8 ms while ScienceQA-IMG accuracy rises from 63.8 to 65.5.

Critical weaknesses
  1. The claim about reducing visual overthinking is not fully established. The keep-ratio study changes inference settings on the same trained checkpoint, so worse results at other ratios may partly reflect a mismatch with training. The router also changes both token selection and residual weighting, making their individual effects unclear.

  2. The efficiency evidence is limited to prefill on Qwen-1.8B at batch size 1. The method is faster than full recursion but slower than dense LLaVA, which takes 26.4 ms compared with 45.8 ms. Since all text tokens continue through the recursive blocks, the benefit during complete answer generation remains unclear.

Questions for authors
  1. How does the method compare with random token selection or shuffled-question routing under the same token budgets? Can you separate the effect of token selection from score-weighted residual updates?
  2. Does selective recursion still outperform full recursion when each configuration is trained for its own inference setting, rather than changing the keep ratios of one checkpoint?
  3. What are the end-to-end latency and decoding throughput for different answer lengths and batch sizes? How are KV caches handled across recursion depths and early-exited visual tokens?
  4. GQA accuracy decreases relative to dense LLaVA on all four backbones. Is this consistent across runs, and what might explain the difference from the gains on ScienceQA-IMG and MMBench?
6 comments
AuthorsOpenReview ↗

Response to Reviewer XqXD — Q1: Visual Routing Ablations

We thank the reviewer for the constructive suggestions on these ablation studies.

Q1.1. Comparison with random and shuffled-question routing

We first compare LLaVA-MoR with random selection and shuffled-question routing under the same visual-token budget as in Section 4.1 of our manuscript. Both baselines use the same checkpoint and unit residual weights. LLaVA-MoR uses question-conditioned selection and score-weighted residual updates.

  • Random selection samples from the active visual tokens at each routing step.
  • Shuffled-question routing uses the question-token hidden states from a different sample to select visual tokens for the current sample.

Table R1. Visual routing methods at a fixed token budget

Routing methodSQA-I ↑TextVQA ↑POPE ↑MME ↑MMBench ↑
Random selection64.12 ± 0.0846.09 ± 0.0787.17±0.05‾\underline{87.17 \pm 0.05}1277.02±3.61‾\underline{1277.02 \pm 3.61}62.30±0.13‾\underline{62.30 \pm 0.13}
Shuffled-question routing64.30‾\underline{64.30}46.19‾\underline{46.19}87.041272.9362.16
LLaVA-MoR65.548.987.8129063.0

Random selection reports mean ± standard deviation across three inference-time selection seeds (42, 43, 44). LLaVA-MoR results are from Table 2 of our manuscript. Bold and underlining indicate the best and second-best scores in each column, respectively.

LLaVA-MoR’s routing strategy outperforms both baselines on all five metrics at the same token budget, with gains of 2.81 percentage points over random selection and 2.71 over shuffled-question routing on TextVQA. These results demonstrate the performance advantage of our question-conditioned routing strategy over both random token selection and shuffled-question routing under the same visual-token budget.

Q1.2. Contribution of score-weighted updates during router training

To test the role of the residual weights separately, we ablate score weighting and auxiliary BCE in Stage II while keeping hard Top-K routing. In the unit-weight ablations, the router scores still determine which visual tokens continue, and each selected token receives the full residual update.

Table R2. Stage II ablation of score weighting and auxiliary BCE

VariantScore weightingAuxiliary BCESQA-I ↑TextVQA ↑POPE ↑MME ↑MMBench ↑
(a)——64.346.087.1128062.5
(b)—✓64.246.287.3127462.4
(c)✓—65.348.787.9128663.0
LLaVA-MoR✓✓65.548.987.8129063.0

✓ indicates an enabled component. — indicates a disabled component.

With BCE held fixed, removing score weighting reduces SQA-I and TextVQA by 1.3 and 2.7 percentage points, respectively. This intervention removes both residual modulation and the backpropagation path from the task loss function to the router parameters, so the drops reflect their joint contribution. With score weighting held fixed, adding BCE improves performance on some benchmarks, with a slight drop on POPE. This drop may occur because BCE trains the router to reinforce its own token selections, even when those selections are incorrect.

AuthorsOpenReview ↗

Response to Reviewer XqXD — Q2: Selective vs. Full Recursion with Matched Training Configuration

We fully agree that a fair comparison requires training selective and full recursion separately under their respective training configurations. In Table 7 of our manuscript, we varied the inference keep ratios of a single trained checkpoint. To address Q2, we add a full visual recursion baseline whose Stage II training and inference both retain all visual tokens.

LLaVA-MoR (selective recursion) uses selective visual-token budgets during Stage II training and inference: recursion 2 retains 50% of the visual tokens, and recursion 3 retains 20% of those remaining. We report its original results from Table 2 of our manuscript. Full visual recursion trains a Stage II router with the Stage I backbone frozen and retains all visual tokens at every recursion during both training and inference; the router scores only scale the residual updates. Both methods retain score-weighted residual updates and use a maximum of three recursions.

We also include the naïve MoR baseline from Table 2 of our manuscript, which uses vanilla routing and omits both proposed training stages.

Table R3. Selective and full recursion with matched training configuration

ConfigurationSQA-I ↑TextVQA ↑POPE ↑MME ↑MMBench ↑
Full visual recursion63.9‾\underline{63.9}46.3‾\underline{46.3}87.81278‾\underline{1278}62.5‾\underline{62.5}
Naïve MoR (Table 2 of our manuscript)63.645.185.4‾\underline{85.4}115755.4
LLaVA-MoR (selective recursion; Table 2 of our manuscript)65.548.987.8129063.0

Bold and underlining indicate the best and second-best scores in each column, respectively.

LLaVA-MoR scores higher than full visual recursion on four of five metrics, including gains of 1.6 percentage points on SQA-I and 2.6 on TextVQA, while matching its POPE score. These results show that selective recursion outperforms full visual recursion, supporting the need to allocate deeper computation selectively across visual tokens.

AuthorsOpenReview ↗

Response to Reviewer XqXD — W1: Evidence for Reducing Visual Overthinking

We thank the reviewer for the careful reading and constructive feedback. Overall, we use the experiments in Q1 and Q2 to refine our claim about reducing visual overthinking in Section 4.4 and Appendix F.2 of our manuscript. Q1 shows that LLaVA-MoR outperforms random selection and shuffled-question routing at the same token budget, and the Stage II ablation shows that removing score weighting and its answer-loss gradient path lowers performance with BCE held fixed. Q2 shows that LLaVA-MoR outperforms the full visual recursion baseline trained and evaluated with all visual tokens on four of five metrics, while matching its POPE score. Together, these results show that reducing deep computation for a subset of visual tokens can improve multimodal task performance. This suggests that some tokens can exit early while the model retains the visual evidence needed to answer the question. By routing deeper updates to question-relevant visual cues, LLaVA-MoR limits unnecessary refinement, providing empirical evidence that selective recursion can mitigate visual overthinking.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.