acceptodds
Withdrawn from ICLR 2027

VAST: Visually Anchored Soft Thinking for Multimodal Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) increasingly rely on chain-of-thought reasoning, yet autoregressive decoding commits to a single discrete token at each step, limiting exploration of alternative reasoning directions. Soft thinking relaxes this constraint by propagating probability-weighted mixtures of token embeddings, but we find that naively applying it to MLLMs can substantially degrade accuracy due to unnecessary uncertainty. We identify two requirements for effective multimodal soft reasoning: when to activate exploration and how to keep it anchored in visual evidence. We introduce VAST (Visually Anchored Soft Thinking), a training-free framework that addresses both through a position-aware entropy schedule and visual contrastive reweighting. VAST selectively activates soft reasoning during the intermediate reasoning phase and steers the resulting representations toward image-dependent candidates. Across Qwen2.5-VL and LLaVA-NeXT models from 7B to 32B parameters and five multimodal benchmarks, VAST achieves the highest average accuracy on every model, with ablations confirming the contribution of both components. Our results show that effective multimodal soft reasoning requires selective exploration that remains anchored in visual evidence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal large language model, latent reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Vw9EOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VAST, a training-free framework for multimodal soft reasoning. It combines position-aware entropy gating to selectively activate soft thinking during intermediate reasoning steps with visual contrastive reweighting to encourage image-grounded token representations. Experiments on three MLLMs and five benchmarks show improvements over greedy CoT and standard Soft Thinking.

Critical strengths
  1. The paper addresses a meaningful problem that naive soft thinking can degrade multimodal reasoning performance.

  2. The proposed framework is simple, training-free, and does not require model modification or additional training.

  3. The paper includes evaluations across multiple models and benchmarks, along with component ablations.

Critical weaknesses
  1. This paper has limited novelty and substantial overlap with existing training-free reasoning methods. The proposed framework is largely built upon existing techniques, including probability-weighted soft token embeddings, entropy-based switching between discrete and latent reasoning, and visual contrastive decoding. Closely related prior works, such as SwiReasoning[1], SeLaR[2], and LEAD [3], have already investigated selective latent reasoning or visual anchoring. In particular, LEAD addresses multimodal latent reasoning through entropy-aware switching and visual anchors. While VAST introduces a position-dependent threshold and applies visual contrastive reweighting to soft embeddings, it is unclear whether these modifications constitute a sufficiently substantial methodological advance.

  2. The main experiments only compare VAST against greedy CoT and vanilla Soft Thinking. This is insufficient given the availability of more advanced training-free methods mentioned above, and especially the multimodal method LEAD. Although some of these methods were originally developed for text-only models, their selective reasoning mechanisms are directly relevant, and the paper should provide adapted comparisons or explain why they are not applicable. Without these comparisons, the claimed effectiveness relative to the current state of the art remains unsubstantiated.

  3. The paper states in Appendix A.2 that the main experiments use 1500 samples, whereas all ablation experiments use 1000 samples. However, the eight Standard Soft Thinking baseline results in Table 4 are identical to the corresponding results in Table 1, despite the supposedly different evaluation sizes. Moreover, several accuracies in Table 4, such as 63.65%, 66.73%, and 49.33%, cannot arise from a single unweighted exact-match evaluation on exactly 1000 samples.

  4. The improvements over standard CoT are modest and lack statistical validation. Although VAST achieves the highest average accuracy among the reported main baselines, the improvements over greedy CoT are only +1.46, +1.11, and +0.29 percentage points on Qwen2.5-VL-7B, Qwen2.5-VL-32B, and LLaVA-NeXT-8B, respectively. In particular, the 0.29-point improvement on LLaVA-NeXT-8B is small relative to the evaluation size. Furthermore, VAST underperforms the original model on some individual benchmarks, including MMBench on LLaVA-NeXT-8B. In Appendix Table 6, its Spatial Relation accuracy decreases from 88.11% to 73.23%. This failure case deserves much deeper analysis, particularly given the paper's emphasis on visually grounded reasoning.

  5. The central claim that intermediate reasoning positions are more suitable for soft exploration is supported primarily by comparisons among several threshold schedules. However, the paper does not directly compare early-only, middle-only, late-only, or random soft activation under matched activation budgets. Moreover, the visual contrastive logit adjustment is essentially inherited from VCD, but the paper does not establish whether constructing soft embeddings from the adjusted distribution offers additional benefits over conventional discrete contrastive decoding. A comparison with VCD and a position-gated discrete VCD variant would help isolate the actual contribution of visually anchored soft thinking.

[1] SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMs. ICLR 2026 [2] SeLaR: Selective Latent Reasoning in Large Language Models. ACL 2026 [3] Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding. CVPR 2026

Questions for authors

See weakness.

Reviewer_ftgpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

VAST is a training-free decoding framework for instruction-tuned multimodal language models. It addresses the observation that applying soft thinking at every reasoning step can underperform ordinary discrete chain-of-thought. A position-dependent, valley-shaped entropy threshold selects steps for soft reasoning. At selected steps, the method contrasts predictions conditioned on the original image and a patch-shuffled image, then uses the adjusted probabilities to construct a mixture of token embeddings. Other steps and the final answer remain discrete. Experiments cover Qwen2.5-VL-7B/32B and LLaVA-NeXT-8B, with component, schedule, perturbation, and hyperparameter analyses.

Critical strengths
  1. A coherent and practical intervention with useful supporting ablations. The separation between deciding when to use soft representations and deciding how to weight them is clear, and neither component requires additional training. Table 4 tests fixed gating, position-aware gating, and visual reweighting incrementally. For Qwen2.5-VL-7B, the four-benchmark average increases from 57.13 for standard soft thinking to 59.04 with fixed gating, 62.43 with the valley schedule, and 63.78 with visual reweighting. Tables 3 and 5 further examine alternative schedules and visual perturbations. These experiments provide more evidence than a comparison of the complete method against the weakest baseline alone.

  2. The negative result about naive soft thinking and the improvements over greedy CoT are useful empirical findings. On Qwen2.5-VL-7B, Table 1 reports five-benchmark averages of 66.75 for CoT, 62.53 for standard soft thinking, and 68.21 for VAST. On Qwen2.5-VL-32B, the three-benchmark average improves from 74.27 to 75.38 (Table 2). The method therefore does more than recover the degradation caused by naive soft thinking, at least on these configurations. Evaluation across two model families and multiple scales is a positive aspect of the paper.

Critical weaknesses
  1. The experiments do not yet isolate the incremental value of the proposed soft representation relative to closely related decoding alternatives. Section 2 appropriately discusses LEAD, AGS, and visual contrastive decoding, but these methods are not included in the experimental comparisons. The particularly consequential missing control is a decoder that uses the same position-aware gate and original-versus-shuffled-image contrast, but selects a discrete token from the adjusted distribution rather than feeding back a mixture. ST-U and the alpha = 0 experiment do isolate the benefit of adding visual contrast to soft thinking; they do not determine whether the gains require the soft mixture rather than the contrastive correction itself. A matched discrete control would clarify the contribution even if some long-trace methods are not directly compatible with the instruction-tuned models studied here. The existing results support the utility of the combination, but leave its advantage over a simpler contrastive decoder unresolved. Similarly, the schedule ablation supports this positional heuristic in the tested settings, rather than establishing a general mechanism of preserving multiple reasoning trajectories.

  2. The scope and practical significance of the gains are less established than the broad framing suggests. The LLaVA-NeXT-8B average improves by only 0.29 percentage points, and its MMBench accuracy falls from 76.53 to 75.53 (Table 1). Section 4.1 reports random 1,500-example subsets, but the paper does not provide uncertainty estimates or variation across subset/perturbation seeds. Table 6 also shows a substantial Spatial Relation regression, from 88.11 for the base model to 73.23 for VAST. Appendix A.7 explicitly excludes this task when reporting proximity to the best LVR average; this exclusion is disclosed, but the resulting aggregate should not be read as broad comparability across all seven tasks. Finally, the practical accuracy-cost trade-off is not measured. Section 3.5 explicitly ignores caching effects when counting forward calls. Sparse invocation of the perturbed-image branch does not by itself establish proportionally sparse computation: the implementation must explain how the perturbed-image history/cache is kept current between gated steps. End-to-end latency and memory measurements are needed to assess whether the modest gains are attractive in practice. This is an unresolved implementation/cost question, not evidence that the implementation is incorrect.

Questions for authors
  1. Can you compare VAST against the matched gated, visually contrastive discrete decoder described above, using the same perturbation, contrast strength, generation limits, and comparable compute? Where feasible, how does VAST compare with LEAD/AGS on the studied models, or what specifically prevents a meaningful adaptation?
  2. How stable are the improvements under repeated image perturbations and benchmark subsampling, or paired uncertainty estimates on the evaluated examples? Please clarify the sample identities and hyperparameter-selection protocol, and explain the substantial Spatial Relation regression rather than relying only on the aggregate that excludes it.
  3. At a gated step following several discrete steps, how is the perturbed-image KV cache/history obtained? Is that branch updated on skipped steps, replayed when needed, or handled differently? Please report end-to-end latency, peak memory, and generated reasoning lengths alongside accuracy, and clarify how the schedule horizon is selected for the main experiments.
Reviewer_esYPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper applies training-free soft thinking (ST) to MLLMs. The authors first observe that standard ST, which "applies soft reasoning at every step" (Sec. 4.1), often underperforms greedy CoT on MLLMs (Table 1: Qwen2.5-VL-7B average 62.53 vs. 66.75). They then propose VAST, which has two components:

  1. Position-aware entropy gating (Sec. 3.2, Eq. 4): a valley-shaped threshold, high at both ends and low in the middle, that concentrates soft inputs on high-entropy steps in the intermediate phase of reasoning; at the boundaries, soft reasoning is restricted to steps where the top-K candidates are almost exactly tied.
  2. Visual contrastive reweighting (Sec. 3.3–3.4, Eq. 5–7): at gated steps, an additional forward pass on a patch-shuffled image is used to adjust the logits in a VCD-style manner, (1+α)z_orig − α z_pert, before constructing the soft embedding.

Experiments cover Qwen2.5-VL-7B/32B and LLaVA-NeXT-8B (hereafter Qwen-7B, 32B, and LLaVA-8B) on five benchmarks, with 1,500 samples per benchmark. The authors report average accuracy gains of up to 1.46 points over CoT and up to 5.68 points over ST, with soft-activation rates of 6–20%.

Critical strengths
  1. Using visual contrast to reweight the soft mixture, rather than to select the next token, is a novel and well-motivated idea. Existing VCD-style methods use the contrastive distribution to correct discrete token selection (Sec. 2, Contrastive decoding), while existing soft thinking methods build the mixture directly from the model's original distribution. VAST combines the two to address a concrete and reasonable concern: MLLM candidate probabilities can be dominated by linguistic priors, so a soft mixture built directly from them may preserve uncertainty while drifting away from the visual evidence (Sec. 3.3). Fig. 1 (right) gives an intuitive example. This matters because it is what distinguishes the paper from prior soft thinking and VCD work; however, as discussed in W1, whether this idea actually produces the gains still lacks the key controls.
  2. The method is training-free, modifies only decoding, and incurs extra forward passes only at gated steps. This gives the method potential practical value: if the gains truly come from the claimed mechanism, it can be applied to existing models at no training cost. For exactly this reason, the question of where the gains come from (below) is central to the evaluation.
Critical weaknesses
  1. The evidence does not isolate the claimed mechanism; in particular, soft mixing itself is never shown to help. The abstract claims that "effective multimodal soft reasoning requires selective exploration that remains anchored in visual evidence." Table 4 does not include CoT. Recomputing CoT from Table 1 over the same four benchmarks gives:

    • Qwen-7B: ST-U (valley gating + soft, no visual contrast) = 62.43, CoT = 62.50.
    • LLaVA-8B: ST-U = 49.98, CoT = 50.90.

    Since ST-U already contains gated soft mixing, this shows that soft mixing by itself yields, on average, no gain over discrete CoT; the average gain over CoT appears only after the VCD-style contrast is added. An equally consistent and simpler explanation is therefore that VCD corrects tokens at uncertain steps, with soft mixing playing no essential role. The controls that would distinguish the two explanations are missing: (a) discrete argmax decoding from p^VA at the same gated steps, and (b) standard VCD (Leng et al., 2024) applied at every step of CoT decoding (hereafter CoT+VCD). Even the contribution of visual contrast is uncertain, as the paper's two estimates of it differ substantially (possibly because they use different subsets; see Q3): in Table 4, ST-U→VAST gives +2.73 / +1.74 (M3CoT / MMBench), whereas in Fig. 3a, α=0→best α gives only ≈+0.6 / +0.4 (read from the figure), and on M3CoT both α=0.25 and α=1.0 are below α=0. With no variance reported, it is not possible to tell which component drives the gains. Although the contributions state that the paper will "analyze how visual contrast reshapes the soft candidate distribution" (Sec. 1), beyond qualitative examples (Figs. 1, 4, and 5) it provides no quantitative mechanistic evidence, such as the distribution of top-1 weights at soft steps, or how often the next-step prediction differs between soft steps and top-1 discrete steps. This is decisive because the paper's contribution is precisely this mechanism: Soft Thinking, entropy gating, and VCD each have precedents, and VAST's novelty lies mainly in their combination, namely using visual contrast to reweight the soft mixture (see Strength 1). Whether this combination actually yields gains is therefore exactly what most needs to be verified.

  2. The gains are small and lack statistical support, and the evaluation is not sufficient to support the claimed "smaller but consistent" improvement (Sec. 4.2).

    • Some gains cannot be distinguished from sampling noise. On LLaVA-8B the average gain over CoT is only 0.29 points, with −1.00 on MMBench (Table 1). These gains, as well as most per-benchmark differences, are smaller than or comparable to the standard error of single-benchmark accuracy (binomial SE = √(p(1−p)/n) = √(0.65×0.35/1,500) ≈ 1.2 points, where p≈0.65 is a typical accuracy and n=1,500 is the number of samples per benchmark). No variance across runs or random subsets and no paired tests (e.g., McNemar or paired bootstrap) are reported, so it is impossible to judge whether these differences are significant. In addition, 32B is evaluated on only three benchmarks, and the claim that "VAST remains the best method on every benchmark" (Sec. 4.2) does not hold on M3CoT, where VAST ties ST (72.80; Table 2).
    • The hyperparameter selection protocol is unclear. The paper uses α∈{0.1, 0.3} and patch size∈{1, 2, 3} (Sec. 4.1) but does not state on which data these were selected; if they were selected on the evaluation subsets, the reported gains are optimistically biased. Moreover, Fig. 3a shows the best α to be 0.5–1.0 and Table 5 uses α=1.0, neither of which falls within the default {0.1, 0.3} (Sec. 4.1).
    • The baselines are insufficient. The only comparisons are greedy CoT and an ST baseline whose implementation details (e.g., stopping criterion, temperature, maximum reasoning length M) are not reported and which drops 8.7 points on MMBench for Qwen-7B. None of the training-free methods discussed in Sec. 2 (LEAD, SeLaR, SwiReasoning), suitably adapted to MLLMs, are compared.
Questions for authors
  1. Could you add the following two controls: (a) discrete decoding from p^VA at the same gated steps (no soft mixing), and (b) CoT+VCD at every step? If VAST is not significantly better than both, how does the soft exploration narrative hold?
  2. Could you report mean ± standard deviation over multiple random subsets or seeds, along with paired bootstrap / McNemar tests against CoT? This is particularly needed for LLaVA-8B and 32B.
  3. Which α and patch size are actually used in the main tables, and on which data and how were they selected? Do Tables 3/4/5 use the same samples as Table 1? A.2 states that ablations use 1,000 samples; why, then, are the numbers in Table 4 identical, digit for digit, to those in Table 1 (1,500 samples)? By definition, α=0 in Fig. 3a and ST-U in Table 4 are the same configuration, yet their numbers differ (M3CoT ≈63.7 vs. 64.00; MMBench ≈82.9 vs. 81.53; Fig. 3a values are read from the figure). Were different sample subsets or other different settings used? Which samples are used for Table 3 and Fig. 3, and could you report CoT on the same subset?
  4. What are the ST baseline's K, temperature, maximum length, and answer extraction method, and does it use Cold Stop from the original Soft Thinking? Could you add an entropy-gated ST in the style of SeLaR/SwiReasoning, or a comparison with LEAD?
  5. At soft steps, what is the distribution of the top-1 weight in the mixture? How often does the next-step prediction differ between soft steps and top-1 discrete steps?
  6. The perturbed-image forward pass (A.3) is conditioned on the full history; how is its KV cache handled? Please report measured wall-clock latency and memory, not only the "1 + ρ_soft" forward-pass count.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.