acceptodds
Under review as a conference paper at ICLR 2027

Swimbird: Eliciting Switchable Reasoning Mode in Hybrid Autoregressive MLLMs

Abstract

Multimodal Large Language Models (MLLMs) have advanced perception and reasoning by bridging vision and language, yet most still rely on textual chain-of-thought (CoT), which is ill-suited to vision-intensive tasks. Recent methods inject a fixed number of continuous hidden states as visual thoughts to strengthen visual reasoning but often weakening text-based logical reasoning. We argue that the main limitation is a rigid, pre-defined reasoning pattern that cannot choose the most suitable thinking modality for each query. We introduce SwimBird, a reasoning-switchable MLLM that dynamically selects among three input-conditioned modes: (1) text-only reasoning, (2) vision-only reasoning with continuous visual thoughts, and (3) interleaved vision–text reasoning. SwimBird adopts a hybrid autoregressive formulation that unifies next-token prediction for textual thoughts with next-embedding prediction for visual thoughts, and uses a reasoning-mode curation strategy to construct SwimBird-SFT-92K, a diverse SFT dataset covering all three reasoning patterns. By enabling query-adaptive mode selection and adaptive latent-token allocation, SwimBird preserves textual logic while boosting vision- dense reasoning. Experiments across textual reasoning and challenging visual understanding benchmarks show that SwimBird achieves state-of-the-art results and robust gains over prior fixed-pattern reasoning methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Switchable, Multimodal, Latent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SR39OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SwimBird, a multimodal large language model that adaptively switches among text-only, vision-only, and interleaved vision-text reasoning modes. The method integrates next-token prediction for textual reasoning and next-embedding prediction for continuous visual thoughts within a hybrid autoregressive framework. It also introduces variable-length latent visual reasoning and constructs a 92K supervised fine-tuning dataset containing three reasoning patterns. Experimental results demonstrate promising improvements across visual understanding and multimodal reasoning benchmarks.

Critical strengths
  1. The paper highlights an important limitation of existing multimodal reasoning approaches, namely their reliance on predefined reasoning patterns that may not suit heterogeneous input queries. This notion is the root of the novelty of this paper.
  2. The empirical results are promising across a diverse set of visual perception and multimodal reasoning benchmarks, with additional evidence of effectiveness on a smaller backbone. These experiments are vital to prove the effectiveness of the proposed method.
Critical weaknesses
  1. Insufficient motivation for adopting adaptive reasoning instead of the interleaved reasoning approach used in SkiLa. I am not convinced that switching reasoning modes for different questions can improve the accuracy on VQA tasks. The authors should clarify that why using visual-only or text-only reasoning mode is more suitable for certain questions in terms of improving the accuracy. The improved results on benchmarks may probably come from carefully curated SFT dataset and external text-only CoT dataset.

  2. Heavy reliance on heuristics in the data construction process, raising concerns about the generalizability of the proposed approach. The data construction process assumes questions with high pass rates (0.75 of pass@8, a magic number) when giving intermediate images are more suitable for vision-only reasoning. The problem is, when a question's pass rate is high, it may be just a simple question, and it does not mean that this question is easy enough (or more suitable) for vision-only reasoning and do not require text in reasoning.

  3. I notice that SwimBird uses Qwen3-VL-8B-Instruct as backbone, and it is different from comparing methods such as SkiLa (Qwen2.5-VL-7B). Misaligned experiment settings undermine the evidence supporting the effectiveness of adaptive reasoning mode.

Questions for authors

See weakness.

4 comments

1. Adaptive Mode Selection versus Fixed Interleaving (W1)

The motivation for adaptive reasoning is modality mismatch: different questions require different reasoning forms, and always using interleaved reasoning may not be the most suitable choice. Fine-grained visual tasks may benefit from directly preserving visual representations, whereas symbolic deduction may not require additional visual thinking.

1) The effect of mode selection persists with the same training data and model weights. The four settings in Table 6 share the same training data and model weights, differing only in the reasoning mode used at inference. On HR4K, Vision-only outperforms Interleave (78.9 vs. 77.0); on MathVerse MINI, Text-only outperforms Interleave (63.1 vs. 60.4). Adaptive reaches 79.0 and 65.8, respectively. These differences occur within the same trained model and cannot be explained by changes in training data, showing that always using an interleaved form does not fully realize the model's capabilities across tasks.

2) A mismatch in reasoning form can introduce additional interference. For questions suited to visual reasoning, textual descriptions in an interleaved process may misrepresent visual information and interfere with an otherwise effective visual judgment. For questions that can already be answered through textual reasoning, additional visual thinking may also introduce misreadings and alter a correct conclusion. Case visualizations and full responses are provided at https://anonymous.4open.science/r/SwimBird-Rebuttal-for-ICLR2027-EE55/adaptive_case.png. In Figure (a), Adaptive answers correctly through visual reasoning, whereas fixed Interleave introduces an incorrect description of the Moon's illuminated proportion in its textual segment and maintains that incorrect judgment in its subsequent response; both trajectories contain 24 visual latent tokens. Figure (b) illustrates interference in the opposite direction: Adaptive's textual path correctly answers 4:30, and fixed Interleave also initially judges the time to be 4:30, but revises it to the incorrect answer of 5:30 after the visual latent segment. For questions suited to visual reasoning, textual descriptions may introduce interference; for questions suited to textual reasoning, additional visual steps may do the same. By selecting Vision-only, Text-only, or Interleave per question, SwimBird adapts its reasoning form to the question's needs, using interleaving when both forms are needed while preserving paths in which a single form suffices.

3) Disentangling Gains from Data and Adaptive Reasoning. Curating high-quality supervision covering three reasoning modes is itself one of our contributions. We have additionally trained two fixed-strategy models, Text-only and Vision-only, to further demonstrate the contribution of the reasoning paradigm. Using the same data sources as SwimBird-SFT-92K, we construct 92K textual CoT samples and 92K visual reasoning samples for the respective training settings. The fixed-strategy models exhibit complementary strengths, while SwimBird maintains strong visual performance and achieves higher accuracy on MMStar and MathVerse MINI than either fixed-strategy model. Together with Table 6’s mode ablation under identical model weights and training data, these results support contributions from both curated supervision and adaptive reasoning: the former develops complementary reasoning capabilities, while the latter enables their flexible use according to the query.

MethodV*HR4KMMStarMathVerse MINI
Text-only83.876.468.164.2
Vision-only85.379.070.354.1
SwimBird85.579.071.265.8

2. Vision-only Labeling and Threshold Ablation (W2)

This threshold measures the conditional answering success rate when intermediate thinking images are provided. Before applying it, we have already filtered out samples that the model can answer correctly from the original input alone.

1) Mode labeling based on solvability with visual context. As described in Section 3.3, we first remove samples that the model can answer correctly using only the original image and question. For the remaining samples, we evaluate answering success rates both without and with intermediate thinking images, retaining those for which the latter is no lower than the former. We then use a success-rate threshold of 0.75 with intermediate thinking images to label samples as Vision-only. Thus, this high success rate measures solvability with intermediate visual information already provided, and does not imply that the original question is inherently easy. This criterion identifies visual supervision trajectories that can reliably support answering without an explicit textual CoT, providing a training signal for learning when to use visual reasoning.

2) Threshold Rationale and Ablation. We provide further clarification on both the threshold design and the data curation pipeline.

The pass-rate threshold serves as a proxy for visual self-sufficiency. A sample is labeled as Vision-only when the model, given only the intermediate thinking images and the question, can solve the problem with high probability without an explicit textual CoT. Setting a high threshold of 0.75 favors samples that can be reliably solved with the provided visual context without an explicit textual CoT. Lower thresholds would introduce more ambiguous cases where textual reasoning may still be needed, making the Vision-only supervision less consistent. To empirically validate this choice, we conducted ablation experiments with different thresholds:

Vision-onlyHR4KHR8KMMStarMathVerse MINI
Baseline76.571.364.761.3
>0.2578.875.368.162.0
>0.5078.575.270.964.7
>0.7579.074.971.265.8
=1.0077.972.669.865.8

The results show that increasing the threshold generally improves performance on benchmarks emphasizing textual reasoning, such as MMStar and MathVerse MINI, as stricter filtering provides clearer Vision-only labels. However, an overly strict threshold (=1.00) reduces the amount of Vision-only training data and leads to degraded performance on visual search tasks such as HR4K and HR8K. The threshold of 0.75 strikes a good balance between data quality and quantity across all benchmarks. We will include this analysis in the revised version.

3. Comparison with SkiLa on the Same Backbone (W3)

We reproduce SkiLa on Qwen3-VL-8B, using the same backbone and VLMEvalKit evaluation framework as SwimBird. The results are shown below:

MethodHR4KHR8KMMStar
Qwen3-VL-8B-Instruct76.571.364.7
SkiLa-Qwen3-VL-8B78.072.366.9
SwimBird79.074.971.2

With the backbone aligned, SwimBird still outperforms SkiLa across all three benchmarks, supporting the advantages of adaptive reasoning.

Dear Reviewer SR39,

Thank you again for your time and valuable feedback. As the discussion deadline approaches, we would like to follow up on our responses. If any concerns remain, or if we have misunderstood any aspect of your comments, please let us know. We would be glad to provide further clarification.

Sincerely,

The Authors

Dear Reviewer SR39,

As the discussion period draws to a close, we would welcome any further feedback on our rebuttal and would be glad to address any remaining concerns. Thank you again for your time.

Sincerely,

The Authors

Reviewer_QiCUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the limitations of fixed reasoning patterns and fixed-length visual latent sequences in multimodal reasoning. It proposes SwimBird, which enables a model to generate textual reasoning, latent visual reasoning, or interleaved traces conditioned on the input. The method adopts hybrid autoregressive modeling: textual positions use next-token prediction with a cross-entropy loss, while visual positions predict continuous vectors supervised by an MSE loss against representations extracted from intermediate thinking images using a vision encoder and projector. Special boundary tokens connect visual and textual segments. The authors further construct SwimBird-SFT-92K to provide supervised demonstrations of all three reasoning modes and use variable-length visual targets to train dynamic visual segments. The main experiments use Qwen3-VL-8B, with the vision encoder and projector frozen and only the language model fine-tuned. The paper reports improvements on visual understanding, general multimodal question answering, and visual mathematical reasoning. It analyzes the method through experiments involving forced reasoning modes, maximum token budgets, visual loss weights, and fixed token budgets. Based on these results, the authors argue that switchable reasoning reduces modality mismatch and that dynamic visual budgets reduce redundancy, improving accuracy while lowering inference costs.

Critical strengths
  1. The paper identifies a valuable research problem. The introduction proposes a reasonable idea that imposing a single reasoning pattern and a uniform visual token budget across heterogeneous multimodal tasks can be suboptimal. Allowing a model to select an appropriate reasoning modality and adjust its reasoning length according to the task is a meaningful direction.
  2. The proposed framework provides a coherent implementation of this goal. Sections 3.1–3.3 combine textual token prediction and continuous visual embedding prediction within one autoregressive process, connect them through explicit boundary tokens, and provide supervised demonstrations of three reasoning modes. The framework and data construction are complementary: the former supports switching, while the latter teaches different reasoning patterns. This constitutes a useful framework contribution even if its individual technical components are largely established.
  3. The experiments include a reasonably broad set of evaluations and design analyses. Tables 2–3 cover both visual perception and multimodal reasoning, with reported improvements over Qwen3-VL-8B-Instruct, including 71.3 to 74.9 on HR8K and 38.8 to 49.5 on WeMath. Tables 4–7 examine budget limits, visual loss weights, reasoning modes, and token budgets. Table 6 provides useful evidence that different tasks benefit from different modes, while the adaptive policy generally matches or approaches the strongest fixed mode. The efficiency measurements and 4B experiments provide additional evidence of practical applicability.
Critical weaknesses
  1. The algorithmic novelty of the framework appears limited and needs sharper positioning. Sections 3.1–3.2 build on hybrid autoregressive generation, boundary-token switching, and a combination of cross-entropy and visual reconstruction losses. In particular, the already-cited SkiLa uses these mechanisms; its inference procedure also permits variable-length latent generation, despite using fixed-length reconstruction targets during training. Thus, variable-length inference alone does not clearly distinguish SwimBird. The principal additions appear to be three-mode data curation and resolution-aware, variable-length visual supervision. These are useful contributions, but the paper should more precisely distinguish them from a new generation or switching algorithm. This is a concern about the extent of methodological novelty, not the usefulness of the overall framework. A clearer mechanism-level comparison would help resolve it.
  2. Table 7 is inconsistent with its accompanying explanation, leaving the adaptive-budget experiment unclear. On page 8, the text states that a fixed budget of 32 performs best on HR4K, whereas 64 performs better on HR8K. However, under the printed table headers, the HR4K column gives 74.3 for budget 32 and 74.6 for budget 64, and there is no HR8K column. The first two adaptive values, 79.0 and 74.9, would match the main results if the corresponding headers were HR4K and HR8K rather than V* and HR4K, but this remains an unconfirmed explanation. Different experimental conditions could legitimately produce different cross-table scores, so I do not regard this as established evidence of contradictory results under identical settings. Nevertheless, the table and its own explanatory paragraph need to be reconciled. This appears potentially repairable through corrected reporting and an explicit description of the controlled settings.
  3. The semantic content of the generated visual latent tokens remains insufficiently analyzed. Figure 5 makes the textual reasoning, mode boundaries, latent lengths, and final answers inspectable, but represents the visual latent sequences only with placeholders. The paper analyzes budgets and mode usage, yet provides little direct evidence about what visual information the predicted latent representations contain. Consequently, these examples do not independently establish that the latent segments perform the specific localization or mental-rotation operations described in Section 4.3. The provenance of the illustrated image crop is also unclear: it is not stated whether it is an explanatory illustration, a supervision target, or a reconstruction from predicted latents. This is a limitation of interpretation rather than evidence that the method is ineffective. The authors should clarify the figure and provide any available representation-level evidence, or qualify the associated mechanistic claims; pixel-level decoding is not necessarily required.
Questions for authors
  1. Table 7: Please confirm the correct dataset headers and values, particularly for the first two columns, and explain the discrepancy with the accompanying paragraph. Please also clarify whether its adaptive row uses the same settings as the main results or includes additional mode or decoding constraints.
  2. Visual latent interpretation: What do the image crop and other visual illustrations in Figure 5 represent? Are they explanatory images, supervision targets, or reconstructions from predicted latent tokens? Please summarize any existing evidence about the visual content encoded by the generated latents. If such evidence is unavailable, please clarify the limits of the qualitative interpretation.
  3. Methodological novelty: Relative to SkiLa’s hybrid generation and variable-length inference, what specifically new mechanism does SwimBird introduce for mode selection or latent-length allocation? Please distinguish the contributions of three-mode data curation, variable-length training targets, and any changes to the generation algorithm.
5 comments

1. Contributions Relative to SkiLa (W1, Q3)

Our contribution is to identify and address modality mismatch: a reasoning form that benefits one query may introduce unnecessary or misleading intermediate steps for another.

  1. From interleaved reasoning to query-adaptive mode selection. SkiLa dynamically interleaves textual and visual thoughts through interleaved supervision, with vision-only reasoning studied in a separately trained variant. SwimBird jointly supervises Text-only, Vision-only, and Interleaved trajectories within one model, teaching it whether each reasoning form is needed for a given query. This selection is learned through autoregressive generation and boundary markers, without an additional router. The distinction is therefore the ability to choose the overall reasoning form per query, beyond adjusting the steps within an interleaved process.

  2. From fixed-length visual targets to variable-length supervision. SkiLa pools visual reconstruction targets to a fixed length, although its inference procedure permits length variation. SwimBird instead constructs resolution-aware, variable-length targets, supervising both visual content and termination at different lengths during training. Our contribution is thus variable-length visual supervision, rather than variable-length decoding alone.

https://anonymous.4open.science/r/SwimBird-Rebuttal-for-ICLR2027-EE55/adaptive_case.png examples illustrate potential interference from imposing interleaved reasoning. In the moon-phase example, Adaptive answers correctly using Vision-only, while fixed Interleave introduces an incorrect textual description of the illuminated region; both use 24 visual latent tokens. Conversely, in the clock example, Adaptive correctly answers 4:30 using Text-only, whereas fixed Interleave initially identifies 4:30 but revises it to 5:30 after the visual segment. These cases illustrate why access to both reasoning forms does not make their joint use appropriate for every query, motivating SwimBird’s explicit learning of when to use either form or their combination.

2. Corrected Dataset Headers in Table 7 (W2, Q1)

Thank you for pointing out this reporting error. The first two columns of Table 7 should be HR4K and HR8K, rather than V* and HR4K. The values remain unchanged, and the corrected table is shown below:

Token BudgetHR4KHR8KMMStarRealWorldQA
Fixed (16)76.071.562.572.7
Fixed (32)78.274.361.972.7
Fixed (64)77.574.661.472.1
Adaptive (0–32)79.074.971.273.1

With the corrected headers, the results align with our discussion: the best fixed budget is 32 tokens for HR4K, 64 for HR8K, and 16 for MMStar. Adaptive allocation achieves the highest reported accuracy across all four benchmarks, supporting the value of adapting visual-token budgets to different task requirements.

Thank you again for your careful reading. We will correct the headers and take greater care to avoid such minor typographical errors.

3. Visual Latent Interpretation (W3, Q2)

1) What Figure 5 shows. The figure directly displays textual outputs, mode boundaries, latent lengths, and final answers. “Latent embedding” is a placeholder for a sequence of continuous states, rather than a visualization of its content. SwimBird does not decode these states into pixel images during inference. The enlarged image region in the interleaved reasoning example is included solely to show visual details.

In addition to the qualitative trajectories in Figure 5, we provide two additional analyses to further validate the interpretability of the generated latent tokens. Case visualizations are provided at https://anonymous.4open.science/r/SwimBird-Rebuttal-for-ICLR2027-EE55/swimbird_case.png

2) Attention Map Visualization. We visualize the attention maps from the latent tokens to the input image embeddings during inference. As shown, the attention is strongly concentrated along the correct navigation path rather than being diffusely spread across the image. This spatial localization provides direct evidence that the latent sketch tokens attend to task-relevant visual regions, supporting their role in visual reasoning.

3) Cosine Similarity Analysis. We compute cosine similarity between the latent token and three types of image embeddings: the problem image, the intermediate thinking image (ground truth), and irrelevant images, averaged over 100 samples. Latent tokens are most similar to the thinking image (0.837) and least similar to irrelevant images (0.579), supporting their semantic grounding.

MetricProblem ImageThinking ImageIrrelevant Images
Cosine similarity0.7130.8370.579

Dear Reviewer QiCU,

Thank you again for your time and valuable feedback. As the discussion deadline approaches, we would like to follow up on our responses. If any concerns remain, or if we have misunderstood any aspect of your comments, please let us know. We would be glad to provide further clarification.

Sincerely,

The Authors

Dear Reviewer QiCU,

As the discussion period draws to a close, we would welcome any further feedback on our rebuttal and would be glad to address any remaining concerns. Thank you again for your time.

Sincerely,

The Authors

Reviewer_P3CqOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SwimBird allows a multimodal model to choose among text-only reasoning, continuous visual reasoning, and mixed visual–text reasoning. It predicts text tokens for textual reasoning and continuous embeddings for visual reasoning. The paper introduces SwimBird-SFT-92K, a dataset of about 92,300 examples covering the three modes. Training combines text prediction with visual embedding regression. At test time, the model selects a reasoning mode and adjusts the number of visual tokens. Experiments cover visual understanding and multimodal reasoning, with further tests of forced reasoning modes, fixed visual-token budgets, and a smaller model.

Critical strengths

1.The paper presents a useful approach with broad empirical gains. A fixed reasoning mode may not suit both visual tasks and tasks that need more textual reasoning. SwimBird allows one model to handle these different needs. The curated dataset is also a useful contribution. Table 2 reports an average improvement from 73.4 to 76.2 over Qwen3-VL-8B-Instruct. The paper also reports gains on MMStar, WeMath, and MathVerse, with further support from the smaller-model results.

2.The forced-mode experiment directly supports the main motivation. In Table 6, visual-only reasoning performs well on visually demanding tasks but reaches 57.2 on MathVerse. Text-only reasoning reaches 63.1, while adaptive reasoning reaches 65.8. Adaptive reasoning also remains competitive with visual-only reasoning on V* and HRBench4K. These results show why a single fixed mode can be limiting.

Critical weaknesses

1.The comparison with the original backbone does not separate the effects of training data and adaptive reasoning. SwimBird adds both curated supervision and a new reasoning scheme. The gains over the original model establish the value of the full system. They do not show how much comes from each part. This does not reduce the value of the dataset, but it limits claims about the source of the gains.

2.The forced-mode test does not fully replace a trained fixed-strategy baseline. Table 6 uses the jointly trained adaptive model for all modes. A model trained specifically for a fixed strategy on the same task examples could perform differently. The table is useful evidence, but it does not settle whether adaptive training is better than a well-trained fixed alternative.

3.Mode selection and visual-token allocation are only partly separated. Choosing text-only reasoning also avoids visual tokens. Thus, a change in mode can also change the amount of computation. Keeping mode selection unchanged while fixing the visual-token budget for visual or mixed reasoning would better isolate the benefit of budget adaptation.

4.The advantage of adaptation varies across benchmarks. In Table 6, adaptive reasoning is close to visual-only reasoning on V* and HRBench4K. Its gain over the best fixed mode is also small on MMStar, while the gain is clearer on MathVerse. The results support flexibility across tasks more strongly than a large benefit on every task. A per-example analysis would help show when mode switching improves an answer within the same benchmark.

Questions for authors

1.Can you compare SwimBird with a fixed-strategy model trained on the same underlying task examples and with comparable training resources? This would help separate the gains from the training data and the gains from adaptive reasoning. It would also strengthen the forced-mode results.

2.In the fixed-budget experiments, can text-only inputs still use zero visual tokens? Please clarify whether mode selection stays unchanged. Fixing the budget only for visual or mixed reasoning would help isolate the value of budget adaptation.

3.Using the existing Table 6 predictions, can you show which examples adaptive reasoning solves that the best fixed mode misses, and vice versa? This would clarify whether adaptation helps within a benchmark as well as across benchmarks.

5 comments

1. Comparison with Trained Fixed-Strategy Baselines (W1, W2, Q1)

We have added two models trained specifically with fixed reasoning strategies: Text-only and Vision-only. Using the same data sources as SwimBird-SFT-92K, we construct 92K textual CoT samples and 92K visual reasoning samples for the respective fixed-mode training settings. This extends the inference-time mode constraints in Table 6 to comparisons with models trained specifically for fixed strategies.

The table below shows that fixed-strategy training yields distinct strengths, but their applicability varies. Vision-only performs comparably to SwimBird on HR4K, but achieves 54.1 on MathVerse MINI; Text-only reaches 64.2 on MathVerse MINI, but scores 76.4 on HR4K. SwimBird maintains its performance on visual tasks while reaching 65.8 on MathVerse MINI and achieving the highest MMStar score among the three models, at 71.2. With the same data sources and training data sizes, SwimBird integrates the strengths of different reasoning forms within one model, accommodating both visual imagination and symbolic reasoning. These results support contributions from adaptive reasoning.

MethodV*HR4KMMStarMathVerse MINI
Text-only83.876.468.164.2
Vision-only85.379.070.354.1
SwimBird85.579.071.265.8

2. Isolating the Effect of Adaptive Visual-Token Allocation (W3, Q2)

Thank you for this valuable suggestion. To isolate the effect of visual-token allocation, we add an ablation with the reasoning mode fixed to Vision-only: all questions use visual reasoning, with either a fixed budget of 16, 32, or 64 visual tokens or a length determined adaptively by the model. Holding the reasoning mode constant allows us to compare the effect of length allocation separately.

As shown below, with visual reasoning used for every question, adaptive length outperforms the best fixed budget on each benchmark. Increasing the fixed budget does not always improve performance: HR8K benefits from longer visual sequences, whereas MMStar and RealWorldQA perform better with 16 tokens than with 32 or 64. This experiment further demonstrates the effectiveness of adaptive visual-token allocation.

Vision-only SettingHR4KHR8KMMStarRealWorldQA
Fixed token count (16)76.070.361.071.1
Fixed token count (32)78.072.860.471.0
Fixed token count (64)77.373.559.270.3
Adaptive78.974.570.772.7

3. Benefits of Adaptive Reasoning Within a Benchmark (W4, Q3)

The best fixed mode may perform well overall on a benchmark without being suitable for every question. Benchmarks evaluate task capabilities, which do not map one-to-one to reasoning forms; questions within the same benchmark may benefit from visual representations, textual deduction, or interleaving the two. SwimBird selects Vision-only, Text-only, or Interleave per question within a single model. It can use the benchmark's best fixed mode while retaining the ability to solve questions that mode misses through the other two forms.

We use Interleave Mode, the best fixed mode on MMStar in Table 6, as the same comparator for both examples, showing cases within one benchmark where Adaptive answers correctly through visual and textual paths while fixed Interleave fails. Case visualizations and full responses are provided at https://anonymous.4open.science/r/SwimBird-Rebuttal-for-ICLR2027-EE55/adaptive_case.png.

In Figure (a), Adaptive selects Vision-only and correctly answers waxing crescent. Fixed Interleave incorrectly describes the phase labeled B as having “more than half” of the Moon illuminated in its textual segment, maintains this incorrect judgment in the response after the visual latent segment, and ultimately answers waning gibbous. Both trajectories contain 24 visual latent tokens: the visual path answers correctly, whereas the textual description in the interleaved path introduces incorrect information. In Figure (b), Adaptive selects Text-only and correctly answers 4:30. Fixed Interleave also initially judges the time to be 4:30, but after generating 24 visual latent tokens, it changes its reading of the hour hand from “between 4 and 5” to “between 5 and 6” and ultimately answers 5:30. Here, textual reasoning already supports a correct answer; the incorrect revision after the additional visual step illustrates potential interference in the opposite direction.

These two examples demonstrate concrete benefits from visual and textual paths within the same benchmark and against the same best fixed comparator. Together with Table 6's cross-task results, they highlight SwimBird's ability to accommodate reasoning needs both across and within benchmarks through per-question mode selection in one model. This unified adaptability is the value of adaptive reasoning; the net improvement on any individual capability captures only one aspect of it.

Thank you for the additional experiments. The separately trained Text-only and Vision-only models address my concern about relying only on forced modes at inference time. The results strengthen the evidence that SwimBird combines the strengths of different reasoning modes. The Vision-only budget experiment also addresses my concern about separating mode selection from visual-token allocation. The two examples help illustrate why different modes can be useful within the same benchmark. Could you briefly confirm whether the new fixed-strategy models use the same backbone, initialization, and comparable training budgets as SwimBird? Also, does “the same data sources” mean the same underlying image–question pairs with different reasoning targets, or separately sampled examples? These results have strengthened my assessment, and this clarification would help me finalize it.

Thank you for the follow-up.

  1. The fixed-strategy models use the same Qwen3-VL-8B-Instruct backbone, initialization, and training budgets as SwimBird.
  2. SwimBird-SFT-92K contains 50K Text-only, 8.8K Vision-only, and 33.5K Interleaved examples. The training data for the newly trained Text-only model retain all image–question pairs and the original 50K textual CoTs, with the remaining 42.3K targets rewritten as textual CoT. For Vision-only, we retain the 42.3K image–question pairs from the visual/interleaved subsets with visual reasoning targets. Since the 50K textual examples primarily involve textual and logical reasoning and lack intermediate thinking images, we replace them with 50K visually supervised examples sampled from the source datasets used to construct SwimBird-SFT-92K’s visual/interleaved subsets (Zebra-CoT, ThinkMorph, and MathCanvas). Thus, Text-only is fully matched and Vision-only is partially matched at the image–question level; both contain 92.3K training examples.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.