acceptodds
Withdrawn from ICLR 2027

From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) can answer simple visual questions, but often struggle when one question requires several visual judgments. We study this gap with controlled tasks for feature binding, numerosity, spatial relations, and amodal completion, together with a Composite task that combines them. Matched counterfactual image pairs isolate changes in the visual evidence needed to answer. Across four models, direct answers, hidden-state readouts, and state interventions show that the individual judgments can be made without explicit reasoning and that intervening on the corresponding states can affect the answer. During reasoning, the Composite answer becomes decodable from hidden states and usable from shortened traces, often before the model stops on its own. We train a small detector to predict this readiness and stop reasoning at that point. On MMStar and RealWorldQA, this reduces mean reasoning tokens by 79.1% and 74.5%, while average accuracy rises by 3.13 and 3.30 percentage points, respectively. These findings connect the internal development of answer readiness to a practical rule for allocating reasoning computation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8p8xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper has two parts.

The first is an analysis study on synthetic images. Four small open VLMs (Qwen3.5-4B/9B, Gemma-4-E4B/12B) are tested on four "primitive" visual judgments: feature binding, numerosity, spatial relations, and amodal completion. They are also tested on a Composite task, "which side has more red circles" among occluded shapes. The tools are:

  • direct answers with thinking disabled;
  • ridge readouts on mean-pooled image-token states;
  • activation patching between counterfactual image pairs;
  • on the Composite task, readouts and forced answers at five checkpoints along the reasoning trace.

The models answer the primitives well (87–91%) but struggle on the Composite task without thinking (58.6%). Thinking raises pair-consistent Composite accuracy by about 41 points, and the answer often becomes decodable and usable before the trace ends.

The second part turns this into an early-stopping method. A linear detector reads the hidden state, its change, and elapsed length every 256 tokens. It is trained, per model and dataset, on whether a forced answer at that checkpoint is correct. The model stops when a validated threshold is crossed. On MMStar and RealWorldQA (two-fold cross-validation), this cuts reasoning tokens by 79% and 75% and raises average accuracy by about 3 points.

Critical strengths
  1. The synthetic tasks are carefully controlled. Counterfactual pairs change only the answer-determining variable, and splits keep pairs together. Pair-consistent accuracy and pair-bootstrap CIs are reported (Table 2), which is better practice than usual in VLM interpretability.
  2. The patching analysis uses a matched-random donor control and measures score margins rather than only top-1 flips. The paper is also clear that decodability and causal effect can come apart across depth (Fig. 3).
  3. Comparing a state readout with a forced answer from a truncated trace (Fig. 4) is a clean way to separate "the information is present" from "the model can use it." The finding that answers are often ready well before the trace ends is useful.
  4. The early-stopping evaluation is honest about where the gains come from. Table 5 decomposes them into recovered unclosed traces vs. lost closed traces, and per-fold results are reported in the appendix.
Critical weaknesses
  1. The accuracy gains mostly come from a weak full-thinking baseline, not from better answers. Table 5 shows that full thinking fails to produce a parsable answer on many questions. For example, on MMStar Qwen3.5-4B has 297/1500 missing and Qwen3.5-9B has 276/1500. These are counted as wrong. Early stopping recovers many of them, but on traces that full thinking does close, it loses correct answers for most models: −95 for Qwen-4B, −76 for Qwen-9B, −11 for Gemma-E4B on MMStar. So the headline "+3.1 points with 79% fewer tokens" is largely "forcing an answer beats not answering at the 6,144-token budget." The obvious baseline is missing: full thinking with budget forcing, i.e., append the same answer cue when the budget is hit or the trace doesn't close. I expect it would remove most of the accuracy gain. With it, the claim might become "much shorter at a small accuracy cost," which is still useful but different.

  2. The detector is nearly a fixed 256-token budget, and simple stopping baselines are missing. For Qwen3.5-4B on MMStar, the detector stops at the first check (256 tokens) on 94.7% of questions, and mean reasoning length is 255. Per-fold means are 256 and 254 (Table 11), and Qwen3.5-9B behaves similarly. For these models the learned detector behaves like "always stop at 256 tokens." The paper compares only against full thinking. It needs:

    • fixed budgets (256 / 512 / 1024 tokens) with the same answer cue;
    • non-thinking mode;
    • prior early-exit methods it cites as closest, i.e., trial-answer confidence stopping (Yang et al., 2026) and hidden-state correctness probes (Zhang et al., 2025a; Afzal et al., 2025).

    Without these, the contribution of the readiness detector itself is not established. Methodologically, the detector is also very close to Zhang et al. (2025a): a linear probe on hidden states predicting whether the current intermediate answer is correct. Applying it to VLMs is the main difference, and the paper should say so explicitly.

  3. The analysis and the method are only loosely connected. The title and framing suggest the interpretability findings drive the stopping rule. In practice, the only link is that the detector layer is selected on Composite data. The detector is then trained from scratch on each benchmark's own forced-answer labels, per model, per dataset, and per fold. Several observations make the link look weak:

    • the selected layer for Gemma-4-E4B is layer 1 (Appendix A.3), which is hard to reconcile with "internal dynamics of reasoning";
    • Gemma-E4B is exactly the model where stopping helps least (+0.6 MMStar, −1.05 RealWorldQA);
    • no experiment shows that a detector trained on Composite, or on one benchmark, transfers to another.

    As written, Sections 3 and 4 read as two separate contributions.

  4. The "integration gap" may not need an integration explanation. The paper concludes that combining judgments is what makes the Composite task hard. However, a quick independence estimate already predicts most of the gap. The average primitive accuracies (FB 99.1, Num 80.9, SR 96.7, AC 77.9) multiply to about 0.60, close to the 58.6% non-thinking Composite accuracy. Each Composite side also contains 8–12 occluded shapes, so amodal completion and counting are applied many times per question. The paper should compare Composite accuracy with what component error rates predict under independence, given the number of objects. It should also break down Composite errors by which component fails, for example on scenes with occlusion removed or with fewer objects. Without this, the claim that the difficulty lies in integration rather than in accumulated perception errors is not supported. Relatedly, Composite non-thinking accuracy (58.6%) is close to its 50% chance level. Comparing primitives to an averaged "31.3% guess rate" (Sec. 1) mixes different chance levels.

Questions for authors
  1. What is full-thinking accuracy if you append the same answer cue whenever the trace is unclosed or hits the budget? How do the Table 3 gains change against this baseline?
  2. How do fixed reasoning budgets (256 / 512 / 1024 tokens) and non-thinking mode compare with adaptive stopping in accuracy and tokens, especially for the Qwen models where the detector almost always fires at the first check?
  3. How does the detector compare with trial-answer confidence stopping (Yang et al., 2026) and hidden-state correctness probing (Zhang et al., 2025a) on the same models and data?
  4. Does a detector trained on MMStar work on RealWorldQA, or vice versa? Does a detector trained only on Composite data work on either?
  5. Why is layer 1 selected for Gemma-4-E4B? How sensitive are the stopping results to the layer choice?
Reviewer_gRPxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors investigate if various visual processing skills, such as feature binding or counting, can be performed by VLMs without reasoning or only with partial reasoning. They find that they can, train a predictor, and improve performance and efficiency on MMstar and RealWorldQA.

Critical strengths
  • Their approach follows previous work, which finds that VLMs over-reason and can sometimes answer correctly and accurately without full reasoning (cite) and research that finds that reasoning improves certain visual skills. Their framework (stopping the VLM early) functionally move VLMs closer to posessing desirable properties of biological vision, which is quick, intuitive, and efficient in most cases.
  • The token efficiency improvements are substantial on the benchmarks they test.
Critical weaknesses
  • Novelty/soundness. They do not show any other baseline stopping rules (stopping at specific token budget, answering immediately, predicting purely a token budget without access to the hidden states, telling it to reason briefly, etc). They don't compare against Dynamic Early Exit in Reasoning Models Chenxu (Yang et al., 2026)or DTSR (Yang Xiang et al., 2026) It is very difficult to know how significant their methodology is without thee baselines
Questions for authors
  • I am unfamiliar with terms in relation to primates of "feedforward pass" and "recurrent processing"- are these informal analogues to neural network processing, or do they have specific meaning?
  • It seems like the "Composite" scenes actually test more difficult skills, because composite uses 8-12 objects whereas feature binding uses 4-8. (L237–242, L670–685, L742–746) Can you show more strongly that the gap is not just harder tasks?
  • What reasoning/sampling/temperature settings did you run these models on? Does this hold across settings?
  • Do you have data on a cost vs accuracy curve? Is there a tradeoff to be made in how ready a model is versus accuracy, or is there generally a single point during reasoning where it becomes plausible to read out the answer?
  • Can you say more about the larger implications of your work in the conclusion? It seems like a stub right now.
  • Can you write more, and run comparisons, between your method and similar ones (e.g.: Reasoning Models Know When They’re Right: Probing Hidden States for Self-Verification , Zhang et al. 2026) and show that yours works better? (or explain why these are not comparable)
Reviewer_zoq5OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why vision-language models (VLMs) can make individual visual judgments but struggle to combine them in composite questions. It uses controlled tasks for feature binding, numerosity, spatial relations, and amodal completion, together with a Composite task and matched counterfactual image pairs. Across four VLMs, direct answers, hidden-state readouts, state interventions, and shortened reasoning traces show that individual judgments can be made without explicit reasoning, while the Composite answer often becomes decodable and usable before the model stops on its own. Based on this finding, the paper proposes Answer-Readiness-Guided Early Stopping, which uses a small detector to predict answer readiness from hidden states and stop reasoning early. Experiments on MMStar and RealWorldQA show reductions of 79.1% and 74.5% in mean reasoning tokens, while average accuracy increases by 3.13 and 3.30 percentage points, respectively.

Critical strengths
  • Clear motivation and comprehensive experimental analysis. The controlled visual tasks and reasoning-trajectory analysis provide useful insights into when the Composite answer becomes decodable and usable.

  • A lightweight method with promising empirical results. Building on the analysis of answer readiness, the paper propose a lightweight early stopping method that only requires training a small detector without updating the VLM. Experiments across four VLMs on MMStar and RealWorldQA show substantial reductions in mean reasoning tokens while improving average accuracy. Further analysis also helps explain where the gains come from.

Critical weaknesses
  • Limited comparisons with early stopping baselines. The paper should include comparisons with some simple early stopping baseline, for example fixed-budget stopping baselines. Since most Qwen runs on MMStar stop at the first 256-token checkpoint, it is unclear how much of the improvement comes from the readiness detector itself.
  • Insufficient analysis of the readiness detector. The detector uses the current hidden state, its change from the previous checkpoint, and reasoning length, but the choice of these features is not fully supported by the earlier analysis. Further analysis and ablation studies would help explain how each feature contributes to readiness prediction and stopping performance.
  • Unclear sources of accuracy gains. Table 5 suggests that a substantial part of the accuracy gains comes from reducing missing answers in full-thinking runs. Further analysis would help clarify how much of the improvement comes from predicting answer readiness more accurately.
Questions for authors
  1. How does the proposed method compare with fixed-budget stopping (e.g., 256 or 512 reasoning tokens)?

  2. How does each feature contribute to readiness prediction and stopping performance?

  3. How much of the accuracy gains comes from reducing missing answers rather than better readiness prediction?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.