- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper has two parts.
The first is an analysis study on synthetic images. Four small open VLMs (Qwen3.5-4B/9B, Gemma-4-E4B/12B) are tested on four "primitive" visual judgments: feature binding, numerosity, spatial relations, and amodal completion. They are also tested on a Composite task, "which side has more red circles" among occluded shapes. The tools are:
- direct answers with thinking disabled;
- ridge readouts on mean-pooled image-token states;
- activation patching between counterfactual image pairs;
- on the Composite task, readouts and forced answers at five checkpoints along the reasoning trace.
The models answer the primitives well (87–91%) but struggle on the Composite task without thinking (58.6%). Thinking raises pair-consistent Composite accuracy by about 41 points, and the answer often becomes decodable and usable before the trace ends.
The second part turns this into an early-stopping method. A linear detector reads the hidden state, its change, and elapsed length every 256 tokens. It is trained, per model and dataset, on whether a forced answer at that checkpoint is correct. The model stops when a validated threshold is crossed. On MMStar and RealWorldQA (two-fold cross-validation), this cuts reasoning tokens by 79% and 75% and raises average accuracy by about 3 points.
Critical strengths
- The synthetic tasks are carefully controlled. Counterfactual pairs change only the answer-determining variable, and splits keep pairs together. Pair-consistent accuracy and pair-bootstrap CIs are reported (Table 2), which is better practice than usual in VLM interpretability.
- The patching analysis uses a matched-random donor control and measures score margins rather than only top-1 flips. The paper is also clear that decodability and causal effect can come apart across depth (Fig. 3).
- Comparing a state readout with a forced answer from a truncated trace (Fig. 4) is a clean way to separate "the information is present" from "the model can use it." The finding that answers are often ready well before the trace ends is useful.
- The early-stopping evaluation is honest about where the gains come from. Table 5 decomposes them into recovered unclosed traces vs. lost closed traces, and per-fold results are reported in the appendix.
Critical weaknesses
-
The accuracy gains mostly come from a weak full-thinking baseline, not from better answers. Table 5 shows that full thinking fails to produce a parsable answer on many questions. For example, on MMStar Qwen3.5-4B has 297/1500 missing and Qwen3.5-9B has 276/1500. These are counted as wrong. Early stopping recovers many of them, but on traces that full thinking does close, it loses correct answers for most models: −95 for Qwen-4B, −76 for Qwen-9B, −11 for Gemma-E4B on MMStar. So the headline "+3.1 points with 79% fewer tokens" is largely "forcing an answer beats not answering at the 6,144-token budget." The obvious baseline is missing: full thinking with budget forcing, i.e., append the same answer cue when the budget is hit or the trace doesn't close. I expect it would remove most of the accuracy gain. With it, the claim might become "much shorter at a small accuracy cost," which is still useful but different.
-
The detector is nearly a fixed 256-token budget, and simple stopping baselines are missing. For Qwen3.5-4B on MMStar, the detector stops at the first check (256 tokens) on 94.7% of questions, and mean reasoning length is 255. Per-fold means are 256 and 254 (Table 11), and Qwen3.5-9B behaves similarly. For these models the learned detector behaves like "always stop at 256 tokens." The paper compares only against full thinking. It needs:
- fixed budgets (256 / 512 / 1024 tokens) with the same answer cue;
- non-thinking mode;
- prior early-exit methods it cites as closest, i.e., trial-answer confidence stopping (Yang et al., 2026) and hidden-state correctness probes (Zhang et al., 2025a; Afzal et al., 2025).
Without these, the contribution of the readiness detector itself is not established. Methodologically, the detector is also very close to Zhang et al. (2025a): a linear probe on hidden states predicting whether the current intermediate answer is correct. Applying it to VLMs is the main difference, and the paper should say so explicitly.
-
The analysis and the method are only loosely connected. The title and framing suggest the interpretability findings drive the stopping rule. In practice, the only link is that the detector layer is selected on Composite data. The detector is then trained from scratch on each benchmark's own forced-answer labels, per model, per dataset, and per fold. Several observations make the link look weak:
- the selected layer for Gemma-4-E4B is layer 1 (Appendix A.3), which is hard to reconcile with "internal dynamics of reasoning";
- Gemma-E4B is exactly the model where stopping helps least (+0.6 MMStar, −1.05 RealWorldQA);
- no experiment shows that a detector trained on Composite, or on one benchmark, transfers to another.
As written, Sections 3 and 4 read as two separate contributions.
-
The "integration gap" may not need an integration explanation. The paper concludes that combining judgments is what makes the Composite task hard. However, a quick independence estimate already predicts most of the gap. The average primitive accuracies (FB 99.1, Num 80.9, SR 96.7, AC 77.9) multiply to about 0.60, close to the 58.6% non-thinking Composite accuracy. Each Composite side also contains 8–12 occluded shapes, so amodal completion and counting are applied many times per question. The paper should compare Composite accuracy with what component error rates predict under independence, given the number of objects. It should also break down Composite errors by which component fails, for example on scenes with occlusion removed or with fewer objects. Without this, the claim that the difficulty lies in integration rather than in accumulated perception errors is not supported. Relatedly, Composite non-thinking accuracy (58.6%) is close to its 50% chance level. Comparing primitives to an averaged "31.3% guess rate" (Sec. 1) mixes different chance levels.
Questions for authors
- What is full-thinking accuracy if you append the same answer cue whenever the trace is unclosed or hits the budget? How do the Table 3 gains change against this baseline?
- How do fixed reasoning budgets (256 / 512 / 1024 tokens) and non-thinking mode compare with adaptive stopping in accuracy and tokens, especially for the Qwen models where the detector almost always fires at the first check?
- How does the detector compare with trial-answer confidence stopping (Yang et al., 2026) and hidden-state correctness probing (Zhang et al., 2025a) on the same models and data?
- Does a detector trained on MMStar work on RealWorldQA, or vice versa? Does a detector trained only on Composite data work on either?
- Why is layer 1 selected for Gemma-4-E4B? How sensitive are the stopping results to the layer choice?