acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Look, Flexible Thought: Learning How and How Long to Think in Multimodal Reasoning

Abstract

The introduction of long chain-of-thought (CoT) reasoning and external tools has substantially improved the performance of multimodal large language models (MLLMs) on general and reasoning tasks. However, existing fast-slow reasoning paradigms mainly regulate textual reasoning length, while visual computation introduced by external tools or visual tokens is often treated as an auxiliary component. This separation prevents MLLMs from jointly optimizing visual perception and textual reasoning. To address this issue, we propose (adaptive ook, flexible hought), a unified reasoning framework that jointly optimizes the form and depth of computation. L2T first discretizes tool-produced visual states into visual tokens, bringing visual perception and textual reasoning into a shared autoregressive space. Based on this unified representation, L2T introduces a modality-aware routing reward to select appropriate reasoning modes and a difficulty-aware fast-slow strategy to adapt reasoning depth according to task complexity. Experiments on general perception and reasoning-intensive tasks demonstrate the effectiveness of . Compared with explicit tool-calling approaches, maintains competitive performance while achieving 1.7–3.6 higher throughput.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Vision and Language Models, Fast-slow Thinking, Latent Reasoning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eciQOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focus on the joint reasoning of both visual and textual content for MLLMs, and proposes a multimodal reasoning framework L2T, converting tool-produced visual states into vector-quantized visual tokens via a trained visual codebook, which permits the visual tokens to be processed in the same way as textual tokens. And then it trains a model to select the proper reasoning mode. This framework is claimed to be more efficient than tool-calling methods. Experiments cover eight visual-centric and reasoning-centric benchmarks, and different model scales, 3B and 7B Qwen2.5-vl models, and additional OOD physics reasoning benchmarks. The 7B model reports the best average scores among the evaluated benchmarks and compared methods, and shorter generated traces.

This paper involves some relatively old but very solid concepts and technecally sound approaches, altogether making the framework reasonable and effective. And the corresponding results was promising. (By "was" I mean, the employed models and compared methods are not quite up-to-date). Some minor experiments (I suppose they can be solved during rebuttal) against more recent models would make this paper better suitable to be publishe.

Critical strengths
  • The proposed framework is a practical.

    • It discretizing intermediate tool outputs via VQ, allowing the model to use ordinary next-token training and policy optimization without external tool execution at inference.
    • This helps to bridge visual and textual reasoning into the same space.
  • This paper present statistical evidence to support their claim of efficiency.

    • Compared to the selected baselines, the proposed method improves the mean score while generating shorter traces.
    • As report under "Average Length of CoT" in Table 2, with rout reward (standard) significantly reduces visual-centric/reasoning-centric CoT length relative to its ablation (w/o route reward); with the fast–slow component (standard) also reduces the length by around 10%. These align with the finding that this method "improves reasoning efficiency by reducing token consumption without sacrificing accuracy" (conclusion took from the paper).
Critical weaknesses
  • The comapred baselines and the used models are relatively out-dated.

    • This paper used Qwen2.5-vl, which was released more than a year and a half ago. Although the proposed method seems to be useful on Qwen2.5-vl, it is unclear if the method remains effective during a year's development of VL models.
    • If this method is still effective for models such as Qwen3, 3.5, 3.6, and 3.8, this could be viewed as a topic worth investigating, otherwise, the motivation of this paper could be considered as somehow solved during the year and a half.
  • The comparison of similar VQ-base methods are missing and the unique contribution of this paper is unclear.

  • Some prior work already reached the contribution mentioned by this paper (contribution #1 at the end of introduction section): [1] already turns intermediate visual representations into VQ-based perception tokens to form a unified space with textual tokens two years ago; [2] and [3] ([3] is already included in Table 1 though) also use discrete visual tokens for intermediate tool outputs (But they are only preprints, so we can exclude them for this discussion). Throughout the paper, it seems that the codebook is fixed after pre-training the VQ-VT. So authors should compare this pre-trained codebook to similar methods' resulting codebook to state the uniqueness of the codebook trained in this paper, and why they did not use pre-trained codebooks (or directly using the training mechanism) in other methods, and what features garantee the following performance.

  • Follow the last point, the codebook is not updated together with the following training stages. This raises an important question, whether the frozen generated tokens remain meaningfully tied to the visual representations learned by VQ-VT when the reasoning mode might be changing during the stage 2 and 3 training.

[1] Bigverdi, Mahtab, et al. "Perception tokens enhance visual reasoning in multimodal language models." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

The followings are preprints, they are only here as examples:

[2] Guo, Ziyu, et al. "ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both." arXiv preprint arXiv:2605.15198 (2026).

[3] Chen, Xiuwei, et al. "Latent Visual States for Efficient Multimodal Reasoning." arXiv preprint arXiv:2606.24233 (2026).

Questions for authors
  1. Are there any accuracy-cost joint metric to directly demonstrate that the proposed method has successfuly reached a balance?

  2. Can you provide the detailed analysis and statistics of constructed training data? e.g., what's the portion of different tasks, and benchmarks. How are they affect the training results? Will the training collapse if changing the data distribution, and how?

  3. After RL, do generated visual tokens still represent meaningful intermediate visual states, or has the model repurposed them as internal reasoning symbols? Appendix A.2 shows image sensitivity, but doesn’t establish that stronger correspondence.

  4. Not a question, but just to mention there are some errors when compiling the pdf (line 461 "¡unk¿ tokens")

2 comments
AuthorsOpenReview ↗

Response to Reviewer eciQ (1)

W1

L2T’s gains persist on both the newly evaluated 2B, 4B model and the larger 72B model.

  • Qwen3-VL-4B. On V*Bench, accuracy changes from 77.0% to 86.4% (+9.4); on HRBench8K, from 72.1% to 74.5% (+2.4); and on MME-RealWorld-Lite, from 47.4% to 53.9% (+6.5).

  • Qwen3.5-2B. V*Bench accuracy improves from 75.9% to 79.1% (+3.2) and MME-RealWorld-Lite from 49.5% to 53.9% (+4.4).

  • Qwen2.5-VL-72B. The corresponding results are 75.4% to 81.5% (+6.1) on V*Bench, 69.3% to 71.0% (+1.7) on HRBench8K, and 48.0% to 52.2% (+4.2) on MME-RealWorld-Lite.

ModelV*BenchHRBench8KMME-RealWorld-Lite
Qwen2.5-VL-72B75.469.348.0
L2T-72B81.571.052.2
Qwen3-VL-4B77.072.147.4
L2T-4B86.474.553.9
Qwen3.5-2B75.973.849.5
L2T-2B79.171.553.9

Additional model-scale and architecture results.

All evaluated models achieve average accuracy improvements over their corresponding base models.

W2 W3

Our contribution is an RL-optimized discrete visual-reasoning framework, not a claim that the codebook is a universally transferable plug-in.

  • Distinction from previous work. We clarify the key differences between these methods and our approach. Method [1] primarily targets multimodal 3D depth estimation; L2T instead targets adaptive allocation of visual versus textual computation and the total reasoning length. Method [2] represents operations with special tokens, rather than quantizing intermediate visual states with a VQ codebook. The reported V*Bench accuracies are 77.9% for ATLAS-GRPO, 75.4% for ATLAS-LA-GRPO, and 83.8% for L2T (gains of 5.9 and 8.4 , respectively). Method [3] adopts continuous latent visual representations, whereas L2T uses discrete VQ indices for next-token modeling and GRPO optimization. We have already discussed the limitations of continuous latent visual tokens in the Related Work section, including the difficulty of jointly optimizing them within a unified reinforcement learning framework. | Method | Acc | |:--------------|:--------:| | ATLAS-GRPO | 77.9 | | ATLAS-LA-GRPO | 75.4 | | L2T (Ours) | 83.8 |

Comparison with ATLAS variants on V*Bench.

  • Scope of the codebook and RL method claim. We would like to clarify that the primary contribution of our work is not to develop a universally superior VQ codebook, but rather to introduce a framework that integrates discrete visual representations into MLLM reasoning and enables unified optimization through reinforcement learning. Regarding other methods, they are designed with their own specific objectives and settings. Therefore, we cannot assume that directly transferring our codebook to these methods would necessarily yield better performance under their respective optimization objectives. In fact, we have never claimed that our codebook is a universally applicable, plug-and-play component.

W4 Q3

The generated visual-token positions remain attentive to task-relevant spatial regions after RL training.

  • Full V*Bench attention analysis (revised manuscript appendix A.2.3, Table 7). Across all examples, the average raw GT attention mass is 0.22% for visual-token positions versus 0.07% for text-token positions (+0.15). Image-normalized GT mass is 2.23% vs. 1.54% (+0.69); area-normalized enrichment is 20.50 vs. 13.47 (+7.03). | Metric | Visual | Text | Δ | |:---|---:|---:|---:| | Raw GT attention mass (%) | 0.22 | 0.07 | +0.15 | | Image-normalized GT mass (%) | 2.23 | 1.54 | +0.69 | | Area-normalized enrichment | 20.50 | 13.47 | +7.03 |
  • GT visualization and task comparison. Figure 6 displays attention concentrated near official GT boxes. With a ground-truth crop explicitly added to the original input, a baseline reaches 81.7% on V*Bench, compared with 83.8% for L2T using generated visual tokens (appendix A.2.1, Table 5). | Method | Accuracy | |:---|---:| | Vanilla Image | 81.7% | | L2T | 83.8% | These results provide evidence that the generated visual tokens acquire meaningful spatial grounding after post-training.
AuthorsOpenReview ↗

Response to Reviewer eciQ (2)

Q1

The training curves and failure analysis support controlled rather than indiscriminate CoT compression.

  • Stage 3 dynamics (appendix A.7.2, Figure 12). Over 30 training steps, the routing score rises rapidly to around 0.9 (Approximate 5 steps); the response length decreases early and then fluctuates around a stable level rather than shrinking indefinitely (Approximate 10 steps). Notably, response length does not decrease indefinitely but instead stabilizes alongside the accuracy reward.

  • The Theoretical Analysis. As theoretically demonstrated in appendix A.3.1, our method guarantees that correct responses are always preferred over incorrect ones, regardless of their lengths. Consequently, when reducing response length does not affect correctness, our method encourages shorter responses. Conversely, when further compression would negatively affect reasoning accuracy, our method prioritizes correctness rather than blindly minimizing response length. This mechanism enables safe and adaptive response compression while preserving reasoning accuracy.

The reported results favor adaptive length control while preserving substantive reasoning on difficult tasks.

Q2

We provide exact dataset composition, benchmark coverage, training settings, and a restricted-data sensitivity experiment.

  • SFT data (appendix A.7.3, Figure 13a). There are 433,256 SFT samples, of which 235,217 (54.29%) receive visual-token supervision and 198,039 (45.71%) do not. The sources include Thyme-SFT, Zebra-CoT from Monet-SFT, and Mulberry from Vision-R1-SFT.

  • RL data (appendix A.7.3, Figure 13b). There are 96,144 RL samples: 40,048 (41.65%) visual-centered, 35,582 (37.01%) mixed, 14,735 (15.33%) simple, and 5,779 (6.01%) text-centered. The RL sources are Thyme-RL, ViRL39K, and Geo3K.

  • Evaluation benchmarks. Our evaluation covers three complementary groups: Visual-centric perception: V* evaluates fine-grained visual search and detail recognition in high-resolution images. HRBench4K and HRBench8K assess high-resolution visual understanding, requiring models to identify small objects, recognize fine details, and understand spatial relationships. MME-RealWorld-Lite evaluates perception in complex real-world images, including object recognition, counting, spatial understanding, and text recognition. Reasoning-centric understanding: MathVista mini evaluates mathematical reasoning grounded in visual inputs, such as diagrams, charts, and geometric figures. MMMU-Pro-10-option evaluates multidisciplinary knowledge and reasoning through challenging multimodal questions with ten answer options. MMMU-Pro-V uses a vision-only question format, requiring models to extract the question and answer choices from images before solving the problem. MMVet evaluates the integration of multiple capabilities, including recognition, OCR, knowledge, spatial understanding, and mathematical reasoning. SeePhys and SeePhysPro SeePhys and SeePhysPro assess physics reasoning from visual evidence. We use them to evaluate generalization beyond the training distribution. Together, these benchmarks cover visually grounded reasoning, integrated multimodal understanding, and out-of-distribution generalization.

MethodThree-benchmark average accuracyThree-benchmark average time (s/sample)HRBench8K accuracyHRBench8K time (s/sample)
Thyme58.53%12.672.00%9.6
L2T (Thyme-RL only)58.00%4.172.13%3.6
L2T (full RL data)60.90%3.072.70%2.6

Sensitivity to RL dataset composition: comparison across three benchmarks and on HRBench8K.

  • Distribution sensitivity. Restricting GRPO RL training to Thyme-RL gives 58.00% at 4.1 s/sample, versus 60.9% at 3.0 s/sample with full RL data. Compared with Thyme’s 58.53% at 12.6 s/sample, the Thyme-RL only variant is 3.07× faster with an accuracy change of −0.53 . On HRBench8K it reaches 72.13% at 3.6 s/sample, versus 72.00% at 9.6 s/sample for Thyme.

The restricted-data change affects downstream performance, but we did not observe training collapse in this experiment: the resulting model retains competitive accuracy and efficiency.

Q4

The compilation error has been fixed in the revised manuscript.

Reviewer_qk5JOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This article proposes L2T, which encodes the intermediate images returned by visual tools into discrete visual tokens, allowing the model to directly generate these tokens, replacing explicit tool calls during inference.
  • Through modality selection and difficulty-aware length control, the model learns how to allocate visual and text reasoning, reducing computational overhead while maintaining accuracy.
Critical strengths
  • Focus on actual inference cost, not just accuracy. Reduce tool calls and unnecessary long reasoning, which has application value.
  • Contrast and ablation of the same SFT starting point show that the method can shorten the inference process while basically maintaining accuracy.
  • Optimizing visual and text reasoning in the same sequence makes the design more unified and makes it easier to adjust the proportion of use of the two.
Critical weaknesses
  • Base coverage is limited. The experiments were based solely on Qwen2.5-VL-3B/7B, and it's unclear whether the gains will migrate to a stronger base or mainly make up for the current model's lack of capabilities.
  • The accuracy of visual tokens lacks direct verification. Image perturbation experiments demonstrate that the generated results are input-dependent, but do not prove that they faithfully correspond to the visual states produced by real tools.
  • The advantages over text representation are not yet clear. <unk> ablation is insufficient to prove the necessity of visual encoding itself, and there is a lack of detailed text representation comparisons under the same training and inference budgets.
Questions for authors
  • Can experiments be supplemented on a base such as Qwen3.5 while checking whether accuracy, inference length, and latency gains are maintained?
  • How closely do the predicted visual tokens encode real tool results? How much better would it be if you replaced it with real encoding?
  • If the same tool results are converted into detailed text, OCR and spatial relationships, at a similar budget, will visual tokens still have an advantage?
  • Can you add a comparison of latency under the same accuracy rate and an accuracy comparison under the same time budget, and clarify the batch size, input resolution and timing range?
4 comments
AuthorsOpenReview ↗

Response to Reviewer qk5J (1)

We sincerely thank the reviewer for the insightful feedback and constructive suggestions. In response, we provide additional experiments across different model scales, further investigate the effectiveness of the learned visual tokens, introduce a text-only baseline, and clarify the end-to-end efficiency evaluation. These results further support the effectiveness and generalizability of our framework.

W1 Q1

Additional 2B, 4B and 72B experiments demonstrate positive average accuracy gains across the evaluated benchmarks.

  • Newer architecture: Qwen3-VL-4B. V*Bench accuracy improves from 77.0% to 86.4% (+9.4 ); HRBench8K from 72.1% to 74.5% (+2.4 ); and MME-RealWorld-Lite from 47.4% to 53.9% (+6.5 ).

  • Newer architecture: Qwen3.5-2B. V*Bench accuracy improves from 75.9% to 79.1% (+3.2) and MME-RealWorld-Lite from 49.5% to 53.9% (+4.4).

  • Larger scale: Qwen2.5-VL-72B. V*Bench improves from 75.4% to 81.5% (+6.1 ); HRBench8K from 69.3% to 71.0% (+1.7 ); and MME-RealWorld-Lite from 48.0% to 52.2% (+4.2 ).

ModelV*BenchHRBench8KMME-RealWorld-Lite
Qwen2.5-VL-72B75.469.348.0
L2T-72B81.571.052.2
Qwen3-VL-4B77.072.147.4
L2T-4B86.474.553.9
Qwen3.5-2B75.973.849.5
L2T-2B79.171.553.9

Additional evaluations on a newer 2B, 4B model and a larger 72B model (accuracy, %).

  • Efficiency comparison. On HRBench8K, L2T-4B achieves 74.50% at 6.7 s/sample, compared with Thyme’s 72.00% at 9.6 s/sample: +2.50 and approximately 30% less latency.

The method demonstrates positive average accuracy gains across all three evaluated model configurations. Moreover, on HRBench8K, L2T-4B retains an advantage over Thyme in both accuracy (74.50% vs. 72.00%) and per-sample latency (6.7 s vs. 9.6 s).

W2 Q2

Visual tokens encode input-dependent information and attend more strongly to annotated target regions.

  • Spatial grounding (revised manuscript appendix A.2.3, Table 7). On V*Bench benchmark, raw GT attention is 0.22% vs. 0.07% (visual-token positions vs. text-token positions); image-normalized GT mass is 2.23% vs. 1.54%; area-normalized enrichment is 20.50 vs. 13.47.
MetricVisualTextΔ
Raw GT attention mass (%)0.220.07+0.15
Image-normalized GT mass (%)2.231.54+0.69
Area-normalized enrichment20.5013.47+7.03
  • GT-crop comparison (revised manuscript appendix A.2.1, Table 5). Providing the original image plus an additional crop from official GT bounding boxes gives 81.7% V*Bench accuracy, whereas L2T with generated visual tokens gives 83.8% (+2.1 ).
MethodAccuracy
Vanilla Image81.7%
L2T83.8%
  • Attention examples (revised manuscript appendix A.2.4, Figure 6). Visual-token attention hotspots are shown alongside official GT boxes for multiple object categories, including gloves, bicycles, helmets, and suitcases.

The evidence supports image sensitivity, spatial grounding, and competitive task performance, and demonstrate that our implicit visual tokens indeed learn meaningful visual representations.

W3 Q3

The text-only baseline is both less accurate and longer than the full visual-token model.

  • Accuracy (revised manuscript Table 3). The text-only baseline, which retains structured reasoning text but removes visual-token content, achieves 70.0% on visual-centric tasks and 55.5% on reasoning-centric tasks. L2T achieves 71.2% and 56.5%, respectively; the summed accuracy improvement across the two categories is 2.2.

  • Response length. The text-only baseline uses 401 tokens on average versus 312 tokens for L2T, a difference of 89 tokens or approximately 22.2% fewer tokens for L2T.

These results demonstrate that the text-only approach offers no advantage in either accuracy or response efficiency, further highlighting the effectiveness of our visual token approach.

AuthorsOpenReview ↗

Response to Reviewer qk5J (2)

Q4

End-to-end latency measurements favor L2T on the tested accuracy–runtime operating points.

  • Measurement protocol. All methods are timed using a single H100 GPU, batch size 1, and matching input-resolution settings. End-to-end latency includes preprocessing, visual encoding, generation, tool execution where applicable, and tool-output processing.

  • Figure interpretation. In Figure 4, the horizontal axis shows end-to-end time (s/sample) and the vertical axis shows accuracy (%). As illustrated in Figure 4, a steeper slope indicates a more favorable accuracy–efficiency trade-off.

  • General perception task. On HRBench8K, L2T obtains 72.7% at 2.6 s/sample, compared with Thyme’s 72.0% at 9.6 s/sample (+0.7 , 3.7× faster). DeepEyesV2 scores 73.8%; however, Figure 2 reports approximately 3.6× higher throughput for L2T than DeepEyesV2 on this benchmark.

  • Reasoning task. On MathVista, L2T scores 72.9%, compared with 70.0% for Thyme and 70.1% for DeepEyesV2. On MMMU-Pro-V, it scores 37.1% versus 33.6% and 34.6%, respectively. Figure 2 reports approximately 1.9× and 1.7× higher throughput on these two benchmarks.

Our method consistently achieves a more favorable Pareto frontier than the competing tool-based approaches, demonstrating its superior overall efficiency.

Thank you for the author's reply; it resolved my doubts, and I will improve my score. However, I hope the author can include experimental results on the powerful model in future versions.

AuthorsOpenReview ↗

Response to Reviewer qk5J (3)

We sincerely thank the reviewer for the positive feedback and for acknowledging that our responses have addressed your concerns. The experimental results on more powerful models (Qwen2.5-VL-72B, Qwen3-VL-4B, and Qwen3.5-2B) have now been incorporated into Appendix A.1.3 of the revised manuscript.

Reviewer_8frYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes L2T, a multimodal reasoning framework that aims to reduce inference costs by jointly controlling reasoning modality and length. It converts intermediate images produced by visual tools into discrete visual tokens, enabling autoregressive visual token generation without external tool calls during inference. The method further introduces a modality aware routing reward to control visual and textual token usage, and a difficulty aware strategy to adjust reasoning length based on rollout success rates.

Experiments across perception and reasoning benchmarks show competitive accuracy and 1.7 to 3.6 times higher throughput compared with tool calling methods. However, the accuracy improvements over comparable reinforcement learning baselines are relatively modest, and removing the proposed routing or fast slow components leads to only minor performance drops. While the results demonstrate improved inference efficiency, they provide limited direct evidence that generated visual tokens can effectively replace external visual operations or that jointly optimizing modality and reasoning length offers substantial benefits over simpler alternatives.

Critical strengths

1. Meaningful improvements in inference efficiency.

The paper addresses a practical challenge in multimodal reasoning by reducing the overhead associated with external visual tool calls. The proposed framework achieves 1.7 to 3.6 times higher inference throughput than tool calling approaches across different benchmarks. These improvements are substantial and demonstrate the practical value of the proposed approach for efficient multimodal reasoning.

2. Competitive performance with reduced reasoning costs.

The paper evaluates the proposed method across a diverse set of perception and reasoning benchmarks. L2T achieves competitive overall accuracy compared with existing methods while generating shorter reasoning traces than the GRPO and GSPO baselines. The results suggest that the framework can reduce unnecessary computation without substantially compromising reasoning performance, which is a useful contribution to efficient multimodal reasoning.

Critical weaknesses

1. Insufficient validation of the core visual-token mechanism.

The paper's central premise is that autoregressively generated visual tokens can effectively replace external visual operations such as cropping and zooming. However, this claim is not directly validated. The comparison with tool-calling methods does not adequately control for differences in training data and optimization, making it difficult to isolate the contribution of visual-token generation. Moreover, the token similarity analysis in Table 5 only demonstrates image sensitivity on a small sample, rather than the preservation of task-relevant visual details. While the efficiency improvements are promising, the experiments do not convincingly establish that the proposed visual tokens can effectively substitute for tool-produced visual evidence.

2. Limited evidence for the necessity and novelty of joint adaptive optimization.

The proposed modality routing and difficulty-aware reasoning largely build upon existing ideas in latent visual reasoning and adaptive CoT optimization. However, the paper does not sufficiently demonstrate why their joint optimization is necessary or substantially better than simpler alternatives. In Table 2, removing the routing reward or fast-slow mechanism reduces average accuracy by only 0.1 and 0.2 percentage points, respectively, although both affect reasoning length. A complete factorial ablation is missing, making it difficult to establish whether the two components provide complementary benefits. Furthermore, task routing relies on heuristic labels, while difficulty estimation uses a reward combining answer correctness and routing compliance, potentially conflating task difficulty with routing preference. These limitations weaken the claimed methodological contribution.

Questions for authors

Q1. Effectiveness of autoregressive visual tokens.

How do the autoregressively generated visual tokens provide additional task-relevant visual information without actually reprocessing the original image? Unlike explicit cropping or zooming, these tokens are generated from the model's existing representations. What evidence supports their ability to preserve or recover fine-grained visual details rather than merely providing additional latent computation?

Q2. Fair comparison with tool-calling methods.

Could the authors provide a controlled comparison between explicit tool calling and autoregressive visual-token generation under the same backbone, training data, and optimization settings? The current comparisons involve differently trained models, making it difficult to attribute performance differences specifically to the proposed visual-token mechanism. This is particularly important for high-resolution perception tasks, where L2T does not consistently outperform tool-based approaches.

Q3. Necessity of joint optimization.

Could the authors provide a complete 2×2 ablation covering both the presence and absence of modality-aware routing and difficulty-aware fast-slow optimization, reporting accuracy and inference cost? Table 2 shows only 0.1 and 0.2 percentage-point accuracy differences when removing the respective components. How can the authors establish that jointly optimizing these two objectives provides complementary benefits beyond independently controlling token usage and reasoning length?

Q4. Reliability of difficulty estimation.

The difficulty-aware strategy estimates task difficulty using rollout pass rates based on the combined correctness and routing reward (with a threshold of 0.99). Consequently, correct answers with imperfect routing scores may be classified as unsuccessful. How frequently does this occur, and how would the results change if difficulty were estimated using answer correctness alone? Furthermore, how does the method distinguish genuinely difficult reasoning problems from perception or knowledge failures that may not benefit from longer reasoning?

Q5. Consistency of the reward implementation.

Equation (16) defines the routing reward using a multiplication by the stable reward, whereas Algorithm 4 in the appendix multiplies by the format reward. Which formulation is used in the actual experiments? Since these definitions can assign different rewards to incorrect responses, could the authors clarify the implementation and whether this discrepancy affects the reported results?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8frY (1)

We sincerely thank the reviewer for the constructive comments and valuable suggestions. In response, we provide additional evidence of visual grounding, clarify the contributions of routing and fast–slow optimization, conduct further comparisons under controlled training settings, and analyze the correctness–efficiency trade-off. We have also addressed the reported typesetting issue in the revised manuscript.

W1 Q1

The final RL-trained visual tokens exhibit image-dependent spatial grounding on V*Bench.

  • Official GT attention (revised manuscript appendix A.2.3, Table 7). We evaluate V*Bench examples using the released object bounding boxes. The raw attention mass assigned to GT regions is 0.22% from visual-token positions versus 0.07% from textual positions (+0.15 ). After renormalizing over image patches, the corresponding masses are 2.23% vs. 1.54% (+0.69); area-normalized enrichment is 20.50 vs. 13.47 (+7.03).
MetricVisualTextΔ
Raw GT attention mass (%)0.220.07+0.15
Image-normalized GT mass (%)2.231.54+0.69
Area-normalized enrichment20.5013.47+7.03
  • Qualitative grounding (appendix A.2.4, Figure 6). Attention maps from generated visual-token positions show hotspots around the officially annotated objects (including gloves, bicycles, and helmets), consistent with the quantitative GT-region analysis.

These measurements support meaningful visual grounding rather than generic pause-token behavior.

W2 Q3

Routing and fast–slow optimization chiefly improve token allocation and efficiency while preserving accuracy.

  • Component isolation (Table 2). Table 2 reports the full model and two single-component removals; it does not include the variant with both components removed. The full model reaches 63.9% overall accuracy, compared with 63.8% without routing reward and 63.7% without fast–slow optimization (differences of only 0.1 and 0.2).

  • Visual-centric length (Table 2). The full model uses 206.8 CoT tokens, versus 292.6 without routing reward (+85.8) and 234.7 without fast–slow optimization (+27.9). For reasoning-centric tasks, the corresponding lengths are 416.2, 547.0 (+130.8), and 469.9 (+53.7).

  • Modality allocation (Table 2). Removing the routing reward decreases the visual-token ratio on visual-centric tasks from 31.8% to 21.9% (−9.9 ), while the reasoning-centric ratio changes only from 11.56% to 11.03% (−0.53).

These results isolate the intended efficiency and modality-allocation effects; accuracy improvement is not the primary objective of these components.

Q2

Under restricted RL training data, L2T retains a large end-to-end speed advantage over Thyme.

  • Controlled setup. We first clarify that all methods in the comparison table use the same Qwen2.5-VL-7B backbone. We additionally restrict L2T’s RL data to Thyme-RL only, using GRPO in the RL stages. Three-benchmark averages are computed across HRBench8K, MathVista, and MMMU-Pro-V.
MethodThree-benchmark average accuracyThree-benchmark average time (s/sample)HRBench8K accuracyHRBench8K time (s/sample)
Thyme58.53%12.672.00%9.6
L2T (Thyme-RL only)58.00%4.172.13%3.6

Comparison with Thyme method under Thyme-RL training data.

  • Three-benchmark aggregate. Thyme obtains 58.53% accuracy at 12.6 s/sample, versus 58.00% at 4.1 s/sample for the restricted-data L2T: a 3.07× speedup and 67.34% lower latency. The accuracy difference calculated from the displayed values is 0.53.

  • High-resolution perception tasks (HRBench8K). The restricted-data L2T achieves 72.13% versus Thyme’s 72.00% (+0.13 ), while reducing latency from 9.6 to 3.6 s/sample (2.67× speedup).

  • Full-data reference. L2T with the complete RL mixture achieves 60.90% at 3.0 s/sample across the same three benchmarks, including 72.70% at 2.6 s/sample on HRBench8K.

Thus, reducing the RL data mixture affects final accuracy but does not eliminate the measured efficiency advantage.

AuthorsOpenReview ↗

Response to Reviewer 8frY (2)

W2 Q4

The correctness threshold has little effect on the final scores, and response length stabilizes during Stage 3.

  • Threshold ablation. The threshold ablation is reported in revised manuscript appendix A.3.2 (Table 8). With τcorr=0.99\tau_{\mathrm{corr}}=0.99, visual-centric and reasoning-centric accuracies are 71.2% and 56.5%; changing the threshold to 0.90.9 gives 71.6% and 55.9%, respectively, with a reported sum of accuracy differences of only 0.2 .

  • Training dynamics (revised manuscript appendix A.7.2, Figure 12). During the Stage-3 steps, the routing score rises rapidly to around 0.9 (Approximate 5 steps); the response length decreases early and then fluctuates around a stable level rather than shrinking indefinitely (Approximate 10 steps). The training accuracy curve shows no sustained downward trend.

  • Correctness ordering. As theoretically demonstrated in appendix A.3.1, our method guarantees that correct responses are always preferred over incorrect ones, regardless of their lengths. As further illustrated in appendix A.7.2, response length does not decrease indefinitely but instead stabilizes alongside the accuracy reward. This observation provides further evidence that our method can effectively regulate response length while preserving accuracy, achieving a favorable balance between reasoning efficiency and correctness.

Overall, the observed threshold sensitivity is small and training does not show unbounded response compression, which may reduce accuracy.

Q5

The typesetting error has been corrected without changing the mathematical equation.

  • Revision. We corrected the reported typesetting/compilation issue in the revised manuscript.

The authors’ additional experiments and clarifications have alleviated my main concerns and strengthened the evidence for the reported efficiency gains, so I am raising my recommendation to Weak Accept.

AuthorsOpenReview ↗

Response to Reviewer 8frY (3)

We sincerely thank the reviewer for the positive feedback and for acknowledging our additional experiments and clarifications. We appreciate your recognition of the efficiency improvements and your willingness to raise the recommendation.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Overall Response

Overall Response

Dear Area Chair and reviewers,

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. We have carefully addressed their concerns through additional experiments, analyses, and clarifications. Our main responses are summarized as follows:

  1. Visual Grounding: We provide quantitative attention analysis on V*Bench, showing stronger GT-region attention for visual tokens than textual tokens, along with qualitative visualizations and GT-crop comparisons.

  2. Generalizability Across Model Scales: We extend our evaluation to 2B, 4B, and 72B models, demonstrating positive average accuracy improvements across all three configurations.

  3. Efficiency and Fair Comparison: We clarify the end-to-end latency measurement protocol and conduct controlled comparisons using the same backbone and restricted RL training data. L2T achieves a 3.07× speedup over Thyme with comparable average accuracy.

  4. Component Effectiveness: We provide further ablations on routing and fast–slow optimization, demonstrating their contributions to token allocation and response efficiency. Additionally, removing visual-token content reduces the summed category accuracies by 2.2 percentage points.

  5. Correctness and Adaptive Reasoning: We provide theoretical analysis of correctness-first optimization and examine training dynamics, showing that response length stabilizes without sustained accuracy degradation. Threshold ablations further demonstrate limited sensitivity to the correctness threshold.

  6. Methodological Clarification and Reproducibility: We clarify the novelty and scope of our RL-optimized discrete visual-reasoning framework, provide detailed SFT/RL training dataset and eval benchmark compositions, and correct the reported formatting issues.

We believe these additional results and clarifications address the reviewers' main concerns and further strengthen the effectiveness, efficiency, and generalizability of our proposed framework.

AuthorsOpenReview ↗

Response to Reviewer eciQ (3)

We thank the reviewer for their recognition of our work and for providing further constructive feedback. These suggestions help us improve the Related Work section, better clarify L2T's contributions to adaptive reasoning, and further refine the manuscript. To address your concerns, we have revised the manuscript and conducted additional experiments, as detailed below. We hope these revisions and results help clarify the points you raised.

Q1

We provide an additional comparison with AURORA.

  • Related work (revised manuscript Section 2.2).

    We agree that AURORA addresses not only depth estimation but also counting and relative-depth reasoning. We have added AURORA to the Related Work section and clarified the similarities and differences between the two approaches. Although both methods employ discrete latent tokens, our work focuses more on learning adaptive reasoning RL strategies, including how to reason and when to think fast or slow across general perception and reasoning tasks.

  • General perception and reasoning.

    Compared with the released AURORA-Depth-13B checkpoint, L2T-7B achieves 72.7 vs. 33.6 on HRBench8K, 72.9 vs. 26.1 on MathVista, and 37.1 vs. 11.6 on MMMU-Pro-V. These results demonstrate the effectiveness of L2T across general perception and reasoning benchmarks. Our main methodological distinction from AURORA lies in the adaptive allocation of visual and textual reasoning and its unified RL optimization, rather than the use of intermediate perception tokens alone.

Q2

To investigate whether VQ tokens preserve their intended visual-state semantics after RL, we conduct additional SFT–RL comparisons from two complementary perspectives: spatial attention and vocabulary representations.

  • Spatial Attention continuity.

    On V* Bench, we replay identical RL-generated sequences under both checkpoints. Comparing corresponding layers and then averaging gives 72.37% VQ attention overlap. Using official GT regions, attention enrichment remains comparable: 19.43 under SFT and 20.50 under RL. This indicates that the SFT and RL checkpoints generally attend to similar regions through VQ tokens.

  • Vocabulary representations continuity.

    Across all VQ tokens, SFT–RL cosine similarity averages 0.996 for input embeddings and 0.999 for output vocabulary vectors. This indicates that the VQ token embeddings remain highly stable after RL, providing complementary evidence of representational continuity.

Together, these findings provide complementary evidence that VQ tokens largely retain their original visual grounding representational characteristics after RL. The VQ-token representations remain highly similar between the SFT and RL checkpoints, suggesting substantial representational continuity.

Q3

We thank the reviewer for this clarification. Our earlier wording was too strong: establishes that correctness is prioritized over response length under the stated assumptions, rather than a guarantee of accuracy preservation after policy optimization.

  • Stage 2–3 comparison (revised manuscript Table 3).

    To verify this empirically, we compare the Stage 2 checkpoint with the final Stage 3 model under identical evaluation settings. As shown in Table 3, Stage 3 improves both visual-centric and reasoning-centric accuracy over Stage 2, suggesting that the additional response-length optimization does not compromise accuracy in our evaluated setting.

ComponentsVisual-CentricReasoning-CentricTotal Δ
only stage 270.555.7-1.5
standard71.256.5--

Thanks for the further clarifications, which address my concerns. Good luck!

AuthorsOpenReview ↗

Response to Reviewer eciQ (4)

We sincerely thank you for your thoughtful feedback and for acknowledging our clarifications. We greatly appreciate your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.