- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes SwimBird, a multimodal large language model that adaptively switches among text-only, vision-only, and interleaved vision-text reasoning modes. The method integrates next-token prediction for textual reasoning and next-embedding prediction for continuous visual thoughts within a hybrid autoregressive framework. It also introduces variable-length latent visual reasoning and constructs a 92K supervised fine-tuning dataset containing three reasoning patterns. Experimental results demonstrate promising improvements across visual understanding and multimodal reasoning benchmarks.
Critical strengths
- The paper highlights an important limitation of existing multimodal reasoning approaches, namely their reliance on predefined reasoning patterns that may not suit heterogeneous input queries. This notion is the root of the novelty of this paper.
- The empirical results are promising across a diverse set of visual perception and multimodal reasoning benchmarks, with additional evidence of effectiveness on a smaller backbone. These experiments are vital to prove the effectiveness of the proposed method.
Critical weaknesses
-
Insufficient motivation for adopting adaptive reasoning instead of the interleaved reasoning approach used in SkiLa. I am not convinced that switching reasoning modes for different questions can improve the accuracy on VQA tasks. The authors should clarify that why using visual-only or text-only reasoning mode is more suitable for certain questions in terms of improving the accuracy. The improved results on benchmarks may probably come from carefully curated SFT dataset and external text-only CoT dataset.
-
Heavy reliance on heuristics in the data construction process, raising concerns about the generalizability of the proposed approach. The data construction process assumes questions with high pass rates (0.75 of pass@8, a magic number) when giving intermediate images are more suitable for vision-only reasoning. The problem is, when a question's pass rate is high, it may be just a simple question, and it does not mean that this question is easy enough (or more suitable) for vision-only reasoning and do not require text in reasoning.
-
I notice that SwimBird uses Qwen3-VL-8B-Instruct as backbone, and it is different from comparing methods such as SkiLa (Qwen2.5-VL-7B). Misaligned experiment settings undermine the evidence supporting the effectiveness of adaptive reasoning mode.
Questions for authors
See weakness.
4 comments
2. Vision-only Labeling and Threshold Ablation (W2)
This threshold measures the conditional answering success rate when intermediate thinking images are provided. Before applying it, we have already filtered out samples that the model can answer correctly from the original input alone.
1) Mode labeling based on solvability with visual context. As described in Section 3.3, we first remove samples that the model can answer correctly using only the original image and question. For the remaining samples, we evaluate answering success rates both without and with intermediate thinking images, retaining those for which the latter is no lower than the former. We then use a success-rate threshold of 0.75 with intermediate thinking images to label samples as Vision-only. Thus, this high success rate measures solvability with intermediate visual information already provided, and does not imply that the original question is inherently easy. This criterion identifies visual supervision trajectories that can reliably support answering without an explicit textual CoT, providing a training signal for learning when to use visual reasoning.
2) Threshold Rationale and Ablation. We provide further clarification on both the threshold design and the data curation pipeline.
The pass-rate threshold serves as a proxy for visual self-sufficiency. A sample is labeled as Vision-only when the model, given only the intermediate thinking images and the question, can solve the problem with high probability without an explicit textual CoT. Setting a high threshold of 0.75 favors samples that can be reliably solved with the provided visual context without an explicit textual CoT. Lower thresholds would introduce more ambiguous cases where textual reasoning may still be needed, making the Vision-only supervision less consistent. To empirically validate this choice, we conducted ablation experiments with different thresholds:
| Vision-only | HR4K | HR8K | MMStar | MathVerse MINI |
|---|---|---|---|---|
| Baseline | 76.5 | 71.3 | 64.7 | 61.3 |
| >0.25 | 78.8 | 75.3 | 68.1 | 62.0 |
| >0.50 | 78.5 | 75.2 | 70.9 | 64.7 |
| >0.75 | 79.0 | 74.9 | 71.2 | 65.8 |
| =1.00 | 77.9 | 72.6 | 69.8 | 65.8 |
The results show that increasing the threshold generally improves performance on benchmarks emphasizing textual reasoning, such as MMStar and MathVerse MINI, as stricter filtering provides clearer Vision-only labels. However, an overly strict threshold (=1.00) reduces the amount of Vision-only training data and leads to degraded performance on visual search tasks such as HR4K and HR8K. The threshold of 0.75 strikes a good balance between data quality and quantity across all benchmarks. We will include this analysis in the revised version.
3. Comparison with SkiLa on the Same Backbone (W3)
We reproduce SkiLa on Qwen3-VL-8B, using the same backbone and VLMEvalKit evaluation framework as SwimBird. The results are shown below:
| Method | HR4K | HR8K | MMStar |
|---|---|---|---|
| Qwen3-VL-8B-Instruct | 76.5 | 71.3 | 64.7 |
| SkiLa-Qwen3-VL-8B | 78.0 | 72.3 | 66.9 |
| SwimBird | 79.0 | 74.9 | 71.2 |
With the backbone aligned, SwimBird still outperforms SkiLa across all three benchmarks, supporting the advantages of adaptive reasoning.
Dear Reviewer SR39,
Thank you again for your time and valuable feedback. As the discussion deadline approaches, we would like to follow up on our responses. If any concerns remain, or if we have misunderstood any aspect of your comments, please let us know. We would be glad to provide further clarification.
Sincerely,
The Authors
Dear Reviewer SR39,
As the discussion period draws to a close, we would welcome any further feedback on our rebuttal and would be glad to address any remaining concerns. Thank you again for your time.
Sincerely,
The Authors
1. Adaptive Mode Selection versus Fixed Interleaving (W1)
The motivation for adaptive reasoning is modality mismatch: different questions require different reasoning forms, and always using interleaved reasoning may not be the most suitable choice. Fine-grained visual tasks may benefit from directly preserving visual representations, whereas symbolic deduction may not require additional visual thinking.
1) The effect of mode selection persists with the same training data and model weights. The four settings in Table 6 share the same training data and model weights, differing only in the reasoning mode used at inference. On HR4K, Vision-only outperforms Interleave (78.9 vs. 77.0); on MathVerse MINI, Text-only outperforms Interleave (63.1 vs. 60.4). Adaptive reaches 79.0 and 65.8, respectively. These differences occur within the same trained model and cannot be explained by changes in training data, showing that always using an interleaved form does not fully realize the model's capabilities across tasks.
2) A mismatch in reasoning form can introduce additional interference. For questions suited to visual reasoning, textual descriptions in an interleaved process may misrepresent visual information and interfere with an otherwise effective visual judgment. For questions that can already be answered through textual reasoning, additional visual thinking may also introduce misreadings and alter a correct conclusion. Case visualizations and full responses are provided at https://anonymous.4open.science/r/SwimBird-Rebuttal-for-ICLR2027-EE55/adaptive_case.png. In Figure (a), Adaptive answers correctly through visual reasoning, whereas fixed Interleave introduces an incorrect description of the Moon's illuminated proportion in its textual segment and maintains that incorrect judgment in its subsequent response; both trajectories contain 24 visual latent tokens. Figure (b) illustrates interference in the opposite direction: Adaptive's textual path correctly answers 4:30, and fixed Interleave also initially judges the time to be 4:30, but revises it to the incorrect answer of 5:30 after the visual latent segment. For questions suited to visual reasoning, textual descriptions may introduce interference; for questions suited to textual reasoning, additional visual steps may do the same. By selecting Vision-only, Text-only, or Interleave per question, SwimBird adapts its reasoning form to the question's needs, using interleaving when both forms are needed while preserving paths in which a single form suffices.
3) Disentangling Gains from Data and Adaptive Reasoning. Curating high-quality supervision covering three reasoning modes is itself one of our contributions. We have additionally trained two fixed-strategy models, Text-only and Vision-only, to further demonstrate the contribution of the reasoning paradigm. Using the same data sources as SwimBird-SFT-92K, we construct 92K textual CoT samples and 92K visual reasoning samples for the respective training settings. The fixed-strategy models exhibit complementary strengths, while SwimBird maintains strong visual performance and achieves higher accuracy on MMStar and MathVerse MINI than either fixed-strategy model. Together with Table 6’s mode ablation under identical model weights and training data, these results support contributions from both curated supervision and adaptive reasoning: the former develops complementary reasoning capabilities, while the latter enables their flexible use according to the query.