acceptodds
Withdrawn from ICLR 2027

When Does Chain-of-Thought Hurt Spatial Answers? Length, Retrieval, and Content Controls in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Chain-of-thought (CoT) prompting often lowers the spatial accuracy of vision-language models (VLMs). In a decoder-only VLM, reasoning text cannot change the image or question states, so it can act only through how the answer position retrieves them or through what the reasoning says. We separate these routes on 2850 spatial items (CV-Bench and two synthetic tasks) in 5 open VLMs of at most 8B parameters with length, turn-structure, re-injection, regeneration, and content controls, reading every answer as one forced-choice token. CoT changes accuracy by -5.7 points on average, whereas length-matched neutral filler in a separate turn changes it by -0.6 and 1024 filler tokens by -2.4; restating the question or re-supplying the image after the reasoning recovers at most +0.8 and +1.0 points. The answer agrees with the option the reasoning states, and in four models pooled CoT accuracy is within 1.7 points of p̄, the Direct distribution's mean probability of the correct option: CoT scores like a sample from that distribution rather than its argmax. Turn-matched filler keeps Direct's argmax step at a correct-option probability of 0.5 while CoT flattens it, and the same probabilities predict four-sample majority-vote accuracy within 1.4 points. The CoT effect thus decomposes into the argmax–expectation gap G = Direct − p̄, which CoT gives up, and the gain E = CoT − p̄ beyond the expectation. In nine larger or closed VLMs queried through an API (reasoning disabled where the API allows it; GPT-5 models still reason in part of their direct calls), G is at most +2.9 points, E reaches +11.2, and CoT changes accuracy by -0.7 to +11.7 points.

Primary area:
interpretability and explainable AI
Keywords:
probing, chain-of-thought, vision-language models, representation analysis, model interpretability, spatial reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aa9iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why short chain-of-thought (CoT) prompting can reduce spatial question-answering accuracy in vision-language models. Across five open models, the authors compare generated CoT with length-matched filler, dialogue-turn controls, question restatement, image re-injection, and conclusion edits. They report that content-independent text length alone does not explain the observed losses. They also identify an empirical correspondence between CoT accuracy and the mean correct-option probability under Direct answering, including a prediction of four-sample voting accuracy. A reduced API evaluation extends the analysis to larger or closed models, while probing, attention analysis, and activation patching provide complementary evidence in Qwen2.5-VL.

Critical strengths

The behavioral controls address consequential experimental confounds. In particular, matching dialogue-turn structure substantially changes the effect of neutral filler. This demonstrates that prompt formatting can otherwise be mistaken for a reasoning-length effect. Together with length matching, re-prompting, and image re-injection, these controls provide useful evidence against content-independent length as a sufficient explanation for the tested short-CoT losses.

Besides, the relationship with Direct probabilities is an interesting quantitative observation. The paper goes beyond comparing average Direct and CoT accuracy: it uses item-level Direct probabilities to predict four-sample voting accuracy. This is potentially useful as a baseline for studying when additional reasoning improves or degrades performance, and is the most distinctive empirical contribution of the paper.

Critical weaknesses
  1. The scope and explanatory value of the probability correspondence remain insufficiently established. The agreement between pooled CoT accuracy and pˉ\bar p is interesting, but item-level relationships differ across models, and the behavioral effects vary substantially across tasks. It remains unclear how much the pooled correspondence reflects a stable relationship within tasks versus offsetting deviations across tasks. The decomposition CoT−Direct=E−G\mathrm{CoT}-\mathrm{Direct}=E-G organizes the observations but adds no independent explanation by itself, since EE is defined using observed CoT accuracy. The API extension also uses a different probability-estimation protocol and includes Direct calls that sometimes contain reasoning. These limitations affect whether the paper provides a broadly informative account beyond its controlled behavioral findings.

  2. The supplied supplementary code does not appear to correspond to the current manuscript. The archive contains an earlier experiment pipeline, lacks the scripts/v2api/ directory mentioned in the appendix, and does not clearly provide the main behavioral, content-editing, and patching experiments described in the paper. The experiment runner also references scripts absent from the archive. Consequently, I cannot trace the central empirical claims to their implementation or verify the reported analyses from the supplied package. This may be a packaging issue, but it materially limits reproducibility in the current submission.

Questions for authors
  1. Can you report the CoT-pˉ\bar p differences and four-sample voting prediction errors separately for the main spatial tasks, with uncertainty estimates? This would help establish whether the central correspondence holds within tasks or depends substantially on pooling heterogeneous tasks.

  2. Does the supplied ZIP correspond to the current manuscript? Please identify or provide the scripts implementing the main E1/E2/E3/E5 experiments and the API evaluation, together with the output files or data manifest needed to reproduce the central analyses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer aa9i

Thank you for noting our clear presentation, that our behavioral controls (turn-structure and length matching, re-prompting, image re-injection) address consequential confounds and show prompt format can be mistaken for a reasoning-length effect, and that our item-level Direct-probability prediction of four-sample voting is the most distinctive empirical contribution. General Response items 1 (W1–W3, Q1), 2 (full-vocabulary readout), and 3 (code, W4/Q2) cover your points; details follow below.

W1. Pooled vs. per-task agreement. We agree that single-CoT pooled agreement (CoT≈pˉ\text{CoT} \approx \bar{p}, ≤1.7\le 1.7 pts; ≤1.6\le 1.6 full-vocabulary) averages across task and confidence deviations (WmcJ-W4). Sec. 4.4 noted flatter slopes (0.47–0.68) in 3 models and +7.9+7.9 pts in Molmo-7B-D, and Tables 12–13 lacked per-task pˉ\bar{p} tests; in Q1, equal-task means in the 4 models are +0.1,+1.3,−0.4,−1.1+0.1, +1.3, -0.4, -1.1 pts vs. mean absolute gaps of 5.5,4.1,3.8,5.85.5, 4.1, 3.8, 5.8 pts.

Q1. Per-task CoT−pˉ\text{CoT} - \bar{p} and four-sample voting errors. Across the 5 tasks in Table 12 (n=650,600,600,500,500n=650, 600, 600, 500, 500; CVB = CV-Bench), 95% CIs exclude 0 in 11/2011/20 CoT−pˉ\text{CoT}-\bar{p} and 15/2015/20 vote-residual cells (4 models) and 3/53/5 Molmo-7B-D cells (top-20 readout, median option prob. 0.998; no four-sample runs, App. F):

1. Per-task CoT−pˉ\text{CoT} - \bar{p} (pts):

ModelCVB Rel.CVB DepthCVB Dist.Syn. pos.Syn. depthPooled (2,850)
Qwen-3B+10.3 [+7.4, +13.1]+3.1 [-0.2, +6.3]+0.7 [-3.1, +4.3]-0.6 [-2.2, +1.0]-12.9 [-16.8, -8.8]+0.8 [-0.7, +2.2]
Qwen-7B+2.5 [+0.2, +4.7]+4.5 [+1.8, +7.1]+6.6 [+3.6, +9.7]-5.2 [-7.2, -3.4]-1.9 [-5.5, +1.5]+1.6 [+0.4, +2.9]
InternVL3-8B+5.0 [+3.0, +7.0]+2.4 [+0.3, +4.4]+1.1 [-1.2, +3.4]-0.3 [-1.6, +1.0]-9.9 [-14.2, -5.6]+0.1 [-1.1, +1.2]
Phi-3.5-V+9.5 [+6.5, +12.4]-13.9 [-18.3, -9.7]-2.3 [-6.3, +1.7]-0.8 [-3.7, +1.9]+2.3 [-2.2, +6.5]-1.0 [-2.7, +0.7]
Molmo-7B-D+9.9 [+6.7, +13.0]+10.3 [+6.9, +13.5]+12.5 [+8.7, +16.3]+1.3 [-1.6, +4.1]+3.6 [-0.2, +7.3]+7.9 [+6.3, +9.5]

2. Per-task four-sample vote residuals (T=0.7T=0.7 vote minus Bin(4,pi)\text{Bin}(4, p_i) prediction, full-vocabulary pip_i):

ModelCVB Rel.CVB DepthCVB Dist.Syn. pos.Syn. depthPooled
Qwen-3B+9.5 [+7.2, +11.6]+4.2 [+1.6, +6.9]+2.5 [-0.5, +5.5]-1.3 [-2.3, -0.5]-13.2 [-16.2, -10.2]+1.0 [-0.1, +2.2]
Qwen-7B+2.0 [+0.1, +3.8]+2.1 [+0.1, +4.2]+4.7 [+2.2, +7.1]-0.6 [-1.1, -0.1]-2.0 [-5.1, +1.0]+1.4 [+0.4, +2.3]
InternVL3-8B+3.9 [+2.4, +5.5]+2.6 [+1.1, +4.3]+1.9 [+0.1, +3.8]-0.3 [-1.1, +0.6]-9.8 [-13.6, -6.3]+0.1 [-0.8, +1.0]
Phi-3.5-V+3.6 [+1.2, +6.1]-3.9 [-6.7, -0.8]-0.1 [-3.0, +2.8]-1.3 [-3.1, +0.4]+5.8 [+3.0, +8.8]+0.8 [-0.4, +2.0]

Pooled observed/predicted vote accuracy is 75.5/74.4 (top-20: 74.5), 84.5/83.1, 86.9/86.8, and 72.5/71.7. We will add both tables to Sec. 4.4 and App. C in the revision.

W2. E−GE - G decomposition. We agree that E−GE - G is an accounting identity since E=CoT−pˉE = \text{CoT} - \bar{p} uses observed CoT accuracy (App. A). In the revision, we will remove E−GE - G from our contributions (Sec. 1) and keep it in Sec. 4.4, Fig. 1c, and Table 4 as a descriptive summary.

W3. API protocol. We agree that our API runs (Sec. 4.6, Table 4, App. I Table 19) lacked logprobs (pˉ^\hat{\bar{p}} from 6 samples, T=1T=1) and included reasoning in 56% of GPT-5.5 and 72% of GPT-5.6-Sol Direct calls. In the revision, we will label Table 4 as a reduced-protocol descriptive extension whose caption notes both differences, and base Secs. 1 and 8 on the open-model protocol.

W4. Supplementary code. We apologize for attaching an earlier archive without the main scripts. Our replacement archive (949 files, SHA-256 7ba1becc4be4b2596240bb9d1bec43dd3cf414b47b213b06c0df02a78ccf0055) has all scripts, outputs, and a MANIFEST mapping each table, figure, and key number to script and output. A CPU-only rebuild (bash scripts/v2/make_all_v2.sh && python scripts/check_against_reference.py) reproduces all 6,087 macros and 8 generated tables of the submitted PDF, except Table 18/Fig. 6 (App. H, earlier pipeline) and Table 20 (from Kancheti et al.). We uploaded it as the revised supplementary material, replacing the earlier ZIP.

Q2. Scripts and outputs. No, it came from an earlier version (W4); in the uploaded ZIP, E1 is scripts/v2/e1_behavior_factorial.py, E2 e2_posthoc_relation_probe.py and e2_readout_mechanism.py, E3 e3_patching.py, E5 e5_content_edit.py, and API scripts/v2api/e9_api_factorial.py and e9_analyze.py (results/v2/<experiment>/, bash scripts/run_experiments.sh <experiment>). We will release code for the new analyses in the revision.

We hope these answers address your concerns, and we are happy to clarify anything further during the discussion.

Thank you for the detailed responses and additional analyses. I appreciate the authors' efforts in addressing my concerns, particularly the additional per-task analyses and improvements to reproducibility. However, the new results do not fundamentally change my assessment of the paper's overall contribution and limitations. Therefore, I maintain my original rating.

Reviewer_f6s2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes how different conditions of CoT reasoning affect the spatial reasoning ability of VLMs. The authors design neatly constrained experiments based on causal-mask constraint that vary multiple axes CoT such as lenght, turn-structure, re-injection, and regeneration, and analyze how each condition influences the model's behavior. Based on these experiments, the paper provide several concrete conclusions on the effect of CoT on final accuracy, faithfulness between reasoning and final answer, and more.

Critical strengths
  • The paper addresses an important problem on VLMs. Whether and how CoT improves spatial reasoning remains an important question for multimodal understanding. Understanding when reasoning helps or harms performance could guide both prompting strategies and post-training.

  • Controlled experiments and analyses provide some useful insights. The experiments are neatly configured and the authors conduct an analysis across multiple VLMs and datasets. Comparing different reasoning conditions and ablations on their structure influence on the accuracy provides useful takeaways.

Critical weaknesses
  • The writing is dense and difficult to follow. The paper introduces many conditions, metrics, and numerical comparisons without consistently emphasizing their main implications. For instance, Table 1 is difficult to interpret as it shows comparisons between multiple conditions at once through an unorganized format. A clearer presentation from hypothesis to experiment to conclusion would improve readability.

  • The main contribution and takeaway is unclear. The paper shows numerous dense analyses on CoTs throughout sections 4, 5, and 6, yet their relationships and relative importance are not clearly explained. Thus it is difficult to identify the core analytical takeaway that this work delivers on the effect of CoT on spatial tasks.

  • Analysis on the underlying cause of CoT degradation. The analysis that CoT lowers spatial accuracy is not new, as it has been discussed before in previous works such as Kancheti et al. [1]. Showing that final answers follow incorrect reasoning does not yet explain why generating CoT introduces errors on questions that are answered correctly without it. A deeper investigation on this failure mechanism would strengthen beyond the contribution that overlaps with prior work.

  • Missing related work. Zhao et al. [2] includes a closely related set of analyses, in which they examine how incorrect CoT prefixes affect spatial reasoning, and the reasoning-answer consistency. The paper should clarify how the proposed analysis framework is different and novel in comparison to this work.


[1] Chain-of-Thought Degrades Visual Spatial Reasoning Capabilities of Multimodal LLMs, Kancheti et al., 2026

[2] On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs, Zhao et al., ICML 2026

Questions for authors
  • How would the key takeaways from this work practically guide researcher on how to improve the next generation of VLMs to have more robust spatial reasoning capabilities?

  • As mentioned in the Weakness section, could the authors elaborate on the novelty and contribution of this work in comparison to "On Robustness and Chain-of-Thought Consistency of RL-Finetuned VLMs"?

3 comments
AuthorsOpenReview ↗

Response to Reviewer f6s2 (1/2)

Thank you for noting that whether and how CoT helps spatial reasoning is an important problem for guiding prompting and post-training, and that our causal-mask-constrained experiments varying length, turn structure, re-injection, and regeneration across multiple VLMs and datasets provide useful takeaways. In the General Response, the opening summary covers our generation-stage results and Item 4 covers W1–W4 and Q1–Q2; details follow below.

W1. Dense writing and Table 1. We agree that Table 1 (p. 3) and presenting parallel analyses across Secs. 4–6 made the paper hard to follow. Below is the revised Table 1, with one question and one comparison per row:

QuestionComparisonResult (mean over models)Takeaway
Does extra text alone cost accuracy?Filler of RR's length + turn vs. Direct−0.6-0.6 pts (CoT: −5.7-5.7)Text length does not explain the CoT loss
Does longer text cost more?512 / 1024-token filler + turn vs. Direct−0.6-0.6 / −2.4-2.4 pts (4 models)Small length effect, far below CoT
Is the question forgotten?Restate after CoT vs. CoT + turn+0.5+0.5 pts (max +0.8+0.8)Not a question-retrieval failure
Is the image forgotten at answer time?Re-image after CoT vs. CoT + turn+0.1+0.1 pts (max +1.0+1.0)Image access at answer time is not the bottleneck
Does the answer follow RR?CoT accuracy when RR states a wrong option1.6% (Direct on the same items: 58.8%)The answer follows RR
Does editing RR's conclusion change the answer?Swap RR's conclusion to the other optionFollows 91.3% / 99.8% (Qwen-3B, Phi-3.5-V); 54.0% / 28.5% (Qwen-7B, InternVL3-8B)Strong in two models, partial in two
Where does RR go wrong?Replace RR's first depth relation with the true one72%–100% (kept: 0%–16%)Errors start at RR's first relation

In the revision, we will replace Table 1 with this table, organize Secs. 4–6 in this order from hypothesis to takeaway, and move secondary probing and patching details to the appendix.

W2. Unclear main contribution and takeaway. We agree that presenting five parallel analyses made our main takeaway hard to see. In the revision, we will structure Secs. 4–6 around a single three-step argument across the 5 open VLMs (2,850 items per model), while presenting the probability correspondence and unseen-data voting prediction (+0.4+0.4 to +2.1+2.1 pts across all 5 models) in Sec. 4.4 as an aggregate empirical regularity:

  1. Not text length or turn format (Sec. 4.2, Table 2): CoT changes accuracy by −5.7-5.7 pts (range −9.3-9.3 to −1.7-1.7) vs. −0.6-0.6 pts (range −2.3-2.3 to +0.6+0.6) for Filler+turn\text{Filler}+\text{turn}. Filler in the same turn loses −4.5-4.5 pts (range −9.8-9.8 to −1.9-1.9), partly via a letter-prior shift (Table 16).
  2. Not answer-time access or representation loss (Secs. 5–6): Restating the question (+0.5+0.5 pts) or re-supplying the image (+0.1+0.1 pts) after RR does not undo the loss (Sec. 6). Under image-grouped splits in Qwen-3B and Qwen-7B (n=1,150n=1,150, Sec. 5.1), both CoT and benign filler lower mid-depth probe accuracy to 0.50–0.52 (Fig. 2a) and reduce image attention from 0.030 to 0.015 and 0.016 in Qwen-3B, ruling both out at the answer position. Activation patching restores the answer only in deeper layers, from 69% depth in Qwen-3B and 59% in Qwen-7B (Fig. 2b).
  3. The answer follows RR, and RR goes wrong at its first spatial relation (Secs. 4.1, 6; W3): Final answers match RR's stated option on 97.6%–100.0% of items (Sec. 4.1) and follow swapped conclusions on 91.3% and 99.8% of items in Qwen-3B and Phi-3.5-V, versus 54.0% and 28.5% in Qwen-7B and InternVL3-8B (Sec. 6). In synthetic depth ordering, replacing the first depth relation in RR with the true relation restores accuracy to 72%–100% across all 5 models, versus 0%–16% when kept (n=78–144n = 78\text{–}144).

We will reorganize Secs. 1 and 4–6 around these three findings in the revision.

AuthorsOpenReview ↗

Response to Reviewer f6s2 (2/2)

W3. Underlying cause of CoT errors relative to Kancheti et al. [1]. We agree that spatial CoT loss is established in [1] (Secs. 1, 7, Table 20) and that answer copying alone does not explain why RR errs when Direct is right (Sec. 4.4). Three analyses locate where and how RR turns wrong:

  • Moderate Direct confidence (App. C): On items where Direct is right and CoT is wrong (new CoT errors), flips concentrate at 0.5≤p<0.80.5 \le p < 0.8 for 78%, 67%, and 80% of cases in Qwen-3B, Phi-3.5-V, and Molmo-7B-D, and at p≥0.8p \ge 0.8 for 42%–45% in Qwen-7B and InternVL3-8B.
  • First written depth relation and editing it (5 models, Syn. depth): Among first depth relations without negation words (n=48–131n=48\text{–}131 new CoT errors vs. 89–24389\text{–}243 where both are right), the first depth relation in RR is already wrong on 63%–96% of new CoT errors vs. 1%–27% where both are right. On 41%–83% of new CoT errors (vs. 1%–13% where both are right), it asserts the queried relation as a fact when the answer is "no". Replacing only that first relation with the true one restores 72%–100% accuracy across all 5 models vs. 0%–16% kept and 33%–62% deleted (n=78–144n=78\text{–}144; a false relation where both are right gives 0%–16% on n=129–324n=129\text{–}324):
Model1st rel. wrong: new / both (nn)Asserts QEdit (nn): kept / true / deleted
Qwen-3B91.6 [87.0,96.2] (131) / 8.4 [5.0,12.1] (239)83.2 [77.1,89.3](144) 4.9 [2.1,8.3] / 94.4 [90.3,97.9] / 33.3 [26.4,41.7]
Qwen-7B77.1 [64.6,87.5] (48) / 26.7 [21.0,32.9] (210)50.0 [35.4,64.6](78) 3.8 [0.0,9.0] / 100.0 [100.0,100.0] / 57.7 [47.4,69.2]
InternVL3-8B95.6 [92.1,99.1] (114) / 15.7 [11.1,20.4] (216)81.6 [74.6,88.6](144) 11.8 [6.9,17.4] / 99.3 [97.9,100.0] / 33.3 [25.7,41.0]
Phi-3.5-V91.1 [83.9,98.2] (56) / 1.1 [0.0,3.4] (89)41.1 [28.6,53.6](94) 0.0 [0.0,0.0] / 100.0 [100.0,100.0] / 61.7 [52.1,71.3]
Molmo-7B-D63.0 [51.9,72.8] (81) / 11.9 [8.2,16.0] (243)54.3 [43.2,65.4](94) 16.0 [8.5,23.4] / 72.3 [62.8,80.9] / 52.1 [42.6,61.7]
  • Re-imaging before vs. during RR (Qwen-3B/7B, InternVL3-8B): Re-imaging before RR shifts wrong-RR rates by −1.8-1.8 to +0.4+0.4 pts (Sec. 4.5, Table 17), whereas re-imaging every 48 tokens during RR recovers +4.6+4.6 to +7.9+7.9 pts on new CoT errors across all 3 tested models (all CIs >0>0). Single-cut re-imaging also gains +11.7+11.7 [+4.8,+18.6][+4.8,+18.6] pts before the answer stays wrong in InternVL3-8B (n=248n=248) and +8.1+8.1 [+1.8,+14.8][+1.8,+14.8] pts at the start of RR in Qwen-7B (n=284n=284; WmcJ-W8).

Thus, new CoT errors arise when RR writes a relation inconsistent with the image (often restating the question) and the answer follows it; visual grounding during generation partly corrects this. We will add these analyses to Sec. 6 in the revision.

W4. Missing related work: Zhao et al. (ICML 2026) [2]. Thank you for flagging [2], who study Wrong-Caption, Wrong-Think, and Stop-Think perturbations of RL-finetuned VLMs, reasoning-answer consistency, default-prompt gold-letter probability for robustness (arXiv v3, Sec. 2.4), and dummy captions (App. C.1). We will cite and discuss [2] in Secs. 1 and 7 in the revision.

Q2. Novelty relative to Zhao et al. [2]. Perturbations, content-free controls, and probability diagnostics are not individually new. While [2] inject externally written wrong text into RL-finetuned models to test robustness, we analyze the loss from a model's own short RR in instruction-tuned VLMs using length- and turn-matched filler (Sec. 4.2, Table 16), re-imaging before, during, and after RR, edits of RR's first relation and conclusion (Sec. 6), image-grouped probing and patching (Sec. 5), and an unseen-data vote prediction (WmcJ-W1). We will add this contrast to Secs. 1 and 7 in the revision.

Q1. Practical guidance for next-generation VLMs. Our results suggest steps for evaluation, inference, and post-training. Evaluation and inference: (i) test per task whether reasoning helps by comparing Direct, CoT, and Filler+turn\text{Filler}+\text{turn} in a separate answer turn to remove format effects (+10.4+10.4 pts recovered on same-turn filler in Qwen-7B; Table 2); (ii) use Bin(4,pi)\text{Bin}(4, p_i) from Direct as a labeled-data baseline for sampled reasoning (within 0.10.1–2.12.1 pts of four-sample voting on seen and unseen items); and (iii) default to Direct on these spatial tasks. Post-training: since the answer follows the first written spatial relation (72%–100% when corrected vs. 0%–16% when kept), rewarding final answers or reasoning-answer agreement alone can reinforce wrong early relations; our results point to rewarding intermediate spatial relations verified against the image and re-grounding on the image during reasoning (+4.6+4.6 to +7.9+7.9 pts). We will add this to Sec. 8 in the revision.

We hope these responses and new experiments address your questions, and we are glad to discuss any point further.

Thank you for the additional clarifications and experiments on the raised points. However, I agree with the other reviewers that the manuscript along with the new experiment results is yet limited in positioning this paper's exact contribution in relation to previous works and moreover in the broader field of spatial reasoning and multimodal reasoning of VLMs. Therefore I maintain my rating.

Reviewer_WmcJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies why some vision-language models answer spatial questions more accurately when responding directly than when generating a chain of thought. It tests irrelevant text, image resupply, and conclusion edits. It also finds that Direct probabilities approximately predict CoT accuracy for some models.

Critical strengths
  1. The paper goes beyond merely comparing the accuracy of the two prompting methods, instead examining specific factors such as the number of dialogue turns and the wording of conclusions.

  2. I appreciate the control over dialogue turns, as well as the authors’ proactive reporting of unsuccessful experiments and measurement limitations.

Critical weaknesses
  1. The most promising finding is the probabilistic relationship; however, it remains to be demonstrated that it can predict performance on unseen data. Degradation of chain-of-thought reasoning in spatial tasks, and error analysis via modification of reasoning text, are both precedented [1][2]. The algebraic decomposition in Section 4.4 of this paper cannot itself explain the underlying cause, as one of its components already incorporates the empirically observed chain-of-thought accuracy. The baselines and item-wise analyses provided by the authors offer supporting evidence, yet in some models, performance deteriorates on easy items while improving on hard ones—potentially canceling each other out in aggregate metrics.

  2. The key conclusions hinge on the accuracy of the measured probabilities. Appendix B selects only the top-20 highest-probability tokens and sets the probabilities of all answer options outside this set to zero. Although the authors report the proportion of cases where no answer option appears within these top-20 tokens, even when option A is present, option B may still be omitted. Such omissions need not alter the final selected answer but may nonetheless distort the probabilities being fitted in this work.

  3. Re-presenting the figure does not improve performance, but this alone is insufficient to rule out visual reading errors. Even if image information remains accessible within the model, the model may retrieve incorrect visual information during reasoning or persistently rely on previously written erroneous relations. Introducing irrelevant text does not produce precisely the same difficulty; likewise, re-presenting the figure before and after short reasoning (as done here) differs from intervening mid-reasoning as in [3].

[1] Chain-of-Thought Degrades Visual Spatial Reasoning Capabilities of Multimodal LLMs

[2] Measuring Faithfulness in Chain-of-Thought Reasoning

[3] TVC — Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoning

Questions for authors

Please see weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer WmcJ (1/2)

Thank you for noting that the paper goes beyond comparing accuracy to examine factors such as dialogue turns and conclusion wording, for appreciating our turn control and proactive reporting of unsuccessful experiments and measurement limitations, and for noting the probabilistic relationship as our most promising finding. General Response items 1 (W1, W3, W4), 2 (W5), and 4 (W2, W6–W8) cover your points; details follow below.

W1. Prediction on unseen data. Yes: on n=767n=767 unseen items per model (143 BLINK Spatial Relation, 124 BLINK Relative Depth, and 500 3DSRBench), Bin(4,pi)\text{Bin}(4, p_i) matches observed four-sample voting within +0.4+0.4 to +2.1+2.1 pts across all 5 models, and all five 95% CIs include 0. Before T=0.7T=0.7 CoT sampling, we computed these item-level Bin(4,pi)\text{Bin}(4, p_i) majority-vote predictions from Direct probabilities and saved them to time-stamped files with SHA-256 hashes:

ModelDirect4-sample vote (obs.)Bin(4,pi)\text{Bin}(4, p_i) pred.Residual [95% CI]
Qwen-3B59.258.156.6+1.5 [-1.2, +4.0]
Qwen-7B65.664.263.8+0.4 [-2.1, +2.8]
InternVL3-8B67.467.366.2+1.2 [-1.0, +3.5]
Phi-3.5-V58.959.057.5+1.5 [-0.9, +3.9]
Molmo-7B-D58.158.456.3+2.1 [-0.6, +4.8]

Alongside the 0.10.1–1.41.4 pt error on the 2,850 main items across the 4 models in Sec. 4.4, we retain Bin(4,pi)\text{Bin}(4, p_i) as an empirical voting baseline and will add this unseen-data test, with per-task residuals, to Sec. 4.4 and App. C in the revision.

W2. Precedent in [1][2]. We agree that spatial CoT degradation and reasoning-intervention controls are established in Kancheti et al. [1] (Secs. 1, 7) and Lanham et al. [2] (Sec. 7), as well as Zhao et al. (ICML 2026). In the revision, we will state that neither finding is new and focus our contribution on turn-matched filler controls, image re-supply before, during, and after RR, first-relation and conclusion edits, and the unseen-data voting test.

W3. Algebraic decomposition (Sec. 4.4). We agree that CoT−Direct=E−G\text{CoT} - \text{Direct} = E - G is an accounting identity because E=CoT−pˉE = \text{CoT} - \bar{p} uses observed CoT accuracy (App. A). In the revision, we will remove E−GE - G from the contributions (Sec. 1) and keep it in Sec. 4.4, Fig. 1c, and Table 4 as a descriptive summary.

W4. Easy vs. hard item offsetting. We agree that single-CoT gains on low-confidence items and losses on high-confidence items offset in the pooled mean, consistent with the flatter 0.47–0.68 item-level slopes in Sec. 4.4 (Fig. 1b). Splitting the n=2,850n=2,850 items per model by Direct pp shows that CoT−pˉ\text{CoT}-\bar{p} is larger at p<0.5p < 0.5 in all 5 models, with opposite signs in Qwen-7B, InternVL3-8B, and Phi-3.5-V. This also holds within tasks, such as Phi-3.5-V on CVB Depth (+17.6+17.6 [+9.2,+26.0][+9.2, +26.0] on n=142n=142 vs. −23.7-23.7 [−28.2,−18.9][-28.2, -18.9] on n=458n=458; per-task table in aa9i-Q1):

ModelCoT−pˉ\text{CoT}-\bar{p} at p<0.5p < 0.5 (nn)CoT−pˉ\text{CoT}-\bar{p} at p≥0.5p \ge 0.5 (nn)
Qwen-3B+2.9 [-1.0, +6.8] (562)+0.3 [-1.4, +1.8] (2,288)
Qwen-7B+13.5 [+9.0, +18.1] (401)-0.3 [-1.5, +0.8] (2,449)
InternVL3-8B+16.1 [+10.9, +21.5] (295)-1.8 [-2.9, -0.7] (2,555)
Phi-3.5-V+16.2 [+12.5, +19.8] (672)-6.3 [-8.2, -4.4] (2,178)
Molmo-7B-D+16.7 [+13.2, +20.3] (723)+4.9 [+3.3, +6.5] (2,127)

Part of the positive gap at p<0.5p < 0.5 also reflects regression to the mean on items where Direct assigns less than 50% probability to the gold option. In the revision, we will present single-CoT agreement in Sec. 4.4 as a pooled, task- and confidence-sensitive regularity and report both this split and the per-task breakdown.

W5. Top-20 truncation (App. B). We agree that requiring one option token in the top 20 (App. B) did not ensure all option tokens were captured, and that calling top-20 readout "exact" in Table 4 was inaccurate. Re-reading all 18×2,850=51,30018 \times 2,850 = 51,300 rows per model across the full vocabulary (summing A, A, a, a) finds 0 missed options and Direct pˉ\bar{p} shifts of at most 0.10.1 pts:

ModelMissMean ∣ΔP(gold)∣\lvert\Delta P(\text{gold})\rvertMax ∣ΔP(gold)∣\lvert\Delta P(\text{gold})\rvertMin Direct opt. prob.Direct pˉ\bar{p} (top-20/full)
Qwen-3B0/51,3002.3e-54.4e-30.99670.81/70.72
Qwen-7B0/51,3001.6e-58.3e-40.99880.38/80.43
InternVL3-8B0/51,3001.5e-61.3e-40.99684.46/84.46
Phi-3.5-V0/51,3001.3e-58.7e-40.96968.27/68.22

Molmo-7B-D (31,350 rows, median option prob. 0.998, top-20 readout) also has 0 misses. Full-vocabulary scoring shifts the argmax on only 32–670 of 51,300 rows per model (near ties with P∈0.5±0.0044P \in 0.5 \pm 0.0044), while CoT−Direct\text{CoT}-\text{Direct} (−8.8,−3.9,−5.1,−9.2-8.8, -3.9, -5.1, -9.2 vs. −8.6,−4.0,−5.1,−9.3-8.6, -4.0, -5.1, -9.3) and four-sample vote residuals shift by at most 0.20.2 pts. We will use the full-vocabulary readout and correct the Table 4 caption in the revision.

AuthorsOpenReview ↗

Response to Reviewer WmcJ (2/2)

W6. Visual reading vs. written relations. We agree that fixed KV states in Sec. 2 and null re-imaging before or after RR do not exclude visual errors while RR is generated; our generation-stage results support both routes you describe:

  • Wrong relations written during generation: In synthetic depth ordering across all 5 models (n=48–131n=48\text{–}131 new CoT errors vs. 89–24389\text{–}243 where both are right; f6s2-W3), the first depth relation written in RR already contradicts the image on 63%–96% of new CoT errors vs. 1%–27% where both are right. Re-imaging before RR shifts wrong-RR rates by −1.8-1.8 to +0.4+0.4 pts (Sec. 4.5), whereas re-supplying the image every 48 tokens while RR is generated recovers +4.6+4.6 to +7.9+7.9 pts on new CoT errors across 3 models (W8 below).
  • Reliance on the written relation: Replacing only that first wrong relation in RR with the true relation restores accuracy to 72%–100% across all 5 models vs. 0%–16% when kept (n=78–144n=78\text{–}144), while inserting a false first relation into otherwise correct reasoning lowers accuracy to 0%–16% (n=129–324n=129\text{–}324). Re-imaging after a wrong RR also raises InternVL3-8B from 2.7% to 26.1% (Table 3).

In the revision, we will narrow Sec. 8 so that re-imaging before and after RR refers specifically to image access at those positions, and state that CoT errors arise when RR writes a relation inconsistent with the image that the answer then follows.

W7. Filler difficulty. We agree that neutral filler (App. B) controls only for token length and turn structure. Two controls test the model's own spatial reasoning text:

  • Same item and image (Table 17): On the 889–1,427 items per model with both correct-option and wrong-option sampled RR on the same item and image, forced-answer accuracy is 96.3%–99.8% after a correct-option RR vs. 0.3%–5.5% after a wrong-option RR (Direct: 72.2%–77.5%), and conclusion swaps in Sec. 6 shift answers on identical prefixes.
  • Same task, other item (all 5 models, 2,850 items per model): For each item, we placed the model's own RR for another item of the same task (different image, closest token length) in the same position and closed-turn structure before answer readout. This cross-item RR lowers accuracy by 7.4–27.5 pts vs. Direct across the 5 models (all five 95% CIs exclude 0), costing 2.7–19.1 pts more than the item's own RR and 6.4–27.1 pts more than length-matched filler. Where that RR states an option, the answer follows it on 65%–100% of items (Direct: 50%–55%), and without its final answer sentence it still costs 11.0–20.7 pts.

Thus, the accuracy change follows what matched-length spatial text states. We will add both controls to App. L in the revision.

W8. Mid-reasoning intervention vs. TVC [3]. We agree that re-imaging before or after RR differs from intervening during generation as in TVC [3], and re-imaging while RR is generated partially recovers accuracy on new CoT errors. We ran two generation-time interventions in Qwen-3B/7B and InternVL3-8B, each compared against the same inserted turns without the image (re-image−no image\text{re-image} - \text{no image}, pts). First, on new CoT errors, we cut RR at its start, before the first relation word, or before the sentence after which the forced answer stays wrong (located with the gold label), closed the turn, and added "Continue your reasoning." with or without the image:

ModelStart of RR (nn)Before first relation (nn)Before answer turns wrong (nn)
Qwen-3B-0.7 [-6.1, +4.4] (458)+4.6 [-0.4, +9.8] (457)+4.0 [-1.1, +8.8] (444)
Qwen-7B+8.1 [+1.8, +14.8] (284)+0.3 [-6.0, +6.7] (283)+2.8 [-3.2, +9.2] (282)
InternVL3-8B+4.1 [-3.0, +11.2] (267)-0.4 [-7.9, +7.5] (267)+11.7 [+4.8, +18.6] (248)

Where both Direct and CoT are right, the first two cuts shift accuracy by −0.5-0.5 to +1.0+1.0 pts (≤2\le 2 items reach the third cut). Second, closer to [3], we re-supplied the image every 48 generated tokens (up to 4 times): at the last sentence boundary, we closed the assistant turn, added "Here is the image again. Continue your reasoning from where you stopped." with or without the image, and carried the unfinished sentence into the next turn:

ModelEvery 48 tokens: new CoT errors (nn)Every 48 tokens: both right (nn)
Qwen-3B+7.9 [+3.1, +12.7] (458)+0.5 [-1.8, +2.8] (600)
Qwen-7B+4.6 [+0.7, +8.1] (284)-1.8 [-3.3, -0.5] (600)
InternVL3-8B+6.7 [+0.7, +12.4] (267)-0.8 [-2.5, +0.8] (600)

Periodic re-imaging during generation recovers +4.6+4.6 to +7.9+7.9 pts on new CoT errors in all three models (all CIs exclude 0). We will add both controls to Sec. 6 in the revision.

We hope these new experiments and clarifications address your concerns, and we welcome any further questions during the discussion.

Thanks to the authors for the rebuttal and new experiments. The per-task analyses and the added reproducibility details are useful and partly address my comments. However, my main concerns about the contribution and the limitations remain after seeing the new results, so I will keep my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank Reviewers aa9i, f6s2, and WmcJ for their constructive feedback. Reviewers found that our turn- and length-matched controls address consequential confounds and provide useful takeaways across models and datasets (aa9i, f6s2, WmcJ), that the Direct-probability prediction of four-sample voting is distinctive and promising (aa9i, WmcJ), that whether and how CoT helps spatial reasoning is an important problem (f6s2), and appreciated our proactive reporting of negative results and measurement limitations (WmcJ).

Across 5 open VLMs on 2,850 spatial items per model (Secs. 2–6), closing the assistant turn reduces the loss from length-matched neutral filler from −4.5-4.5 to −0.6-0.6 pts, versus −5.7-5.7 pts for CoT (Sec. 4.2, Table 2). Restating the question (+0.5+0.5 pts) or re-supplying the image (+0.1+0.1 pts) after RR, as well as re-imaging before RR (−0.7-0.7 to +0.6+0.6 pts), leaves the CoT loss in place (Secs. 4.5, 6). In synthetic depth ordering, on new CoT errors (Direct right, CoT wrong), the first depth relation written in RR already contradicts the image on 63%–96% of items, versus 1%–27% where both are right. Replacing only that first relation with the true one restores accuracy to 72%–100% across all 5 models, versus 0%–16% when kept (n=78–144n = 78\text{–}144; reply to f6s2). Re-imaging every 48 tokens while RR is generated recovers +4.6+4.6 to +7.9+7.9 pts on new CoT errors across 3 models (and up to +11.7+11.7 pts at a single cut), whereas placing another item's matched-length RR before the answer lowers accuracy by 7.4–27.5 pts vs. Direct across all 5 models (reply to WmcJ).

Below are our four main updates; all new 95% CIs use a paired cluster bootstrap over image groups (2,000 resamples, seed 0):

  1. Probability correspondence and unseen-data voting test (aa9i W1–W3, Q1; WmcJ W1, W3, W4): On the 2,850 main items, Bin(4,pi)\text{Bin}(4, p_i) predicts four-sample voting within 0.10.1–1.41.4 pts across the 4 models (Sec. 4.4). On n=767n=767 unseen items per model (BLINK Spatial Relation, BLINK Relative Depth, 3DSRBench) with predictions saved to time-stamped files (with SHA-256 hashes) before sampling, Bin(4,pi)\text{Bin}(4, p_i) matches observed four-sample voting within +0.4+0.4 to +2.1+2.1 pts across all 5 models (all 5 CIs include 0; reply to WmcJ). Meanwhile, we agree that single-CoT pooled agreement (≤1.7\le 1.7 pts in 4 models; ≤1.6\le 1.6 full-vocabulary) averages across task- and confidence-level deviations, and that CoT−Direct=E−G\text{CoT} - \text{Direct} = E - G is an accounting identity. In the revision, we will present the correspondence in Sec. 4.4 as an aggregate, task-sensitive regularity alongside the per-task, confidence-split, and unseen-data tables. We will also remove E−GE - G from the contribution list, keep it in Sec. 4.4, Fig. 1c, and Table 4 as a descriptive summary, and label Table 4 as a reduced-protocol descriptive extension.
  2. Full-vocabulary option probabilities (WmcJ W5): Re-reading all 51,30051,300 rows per model across the full vocabulary in Qwen-3B, Qwen-7B, InternVL3-8B, and Phi-3.5-V shows 0 rows missing an option in the top 20, max⁡∣ΔP(gold)∣≤4.4e-3\max \lvert\Delta P(\text{gold})\rvert \le 4.4\text{e-}3, and Direct pˉ\bar{p} shifts of at most 0.10.1 pts. Molmo-7B-D (top-20 readout) likewise has 0 misses across 31,350 rows.
  3. Supplementary code (aa9i W4, Q2): We apologize for attaching an archive from an earlier version of the project. Our replacement archive (949 files, SHA-256 7ba1becc4be4b2596240bb9d1bec43dd3cf414b47b213b06c0df02a78ccf0055) contains all scripts, per-item outputs, and a manifest, and reproduces all 6,087 macros and 8 generated tables of the submitted PDF on CPU. We have uploaded it as the revised supplementary material, replacing the earlier ZIP.
  4. Presentation, related work, and visual-reading scope (f6s2 W1–W4, Q1–Q2; WmcJ W2, W6–W8): In the revision, we will replace Table 1 (reply to f6s2), reorganize Secs. 4–6 around a single three-step argument, cite and compare against Zhao et al. (ICML 2026), add our generation-stage controls to Sec. 6 and Apps. K–L, and narrow Sec. 8 so that re-imaging before or after RR refers specifically to image access at those positions.

We hope these updates and analyses address the reviewers' concerns, and we are happy to clarify any point further during the discussion.

AuthorsOpenReview ↗

Revision uploaded (Nov 17)

Dear Reviewers aa9i, f6s2, and WmcJ,

We have uploaded a revised PDF and an updated supplementary archive. New or changed text is in blue.

  • A new Figure 3 (Sec. 6) summarizes the added results: the unseen-data voting test (767 items; residuals of +0.4 to +2.1 points, all 95% CIs include 0), the first depth relation edit on synthetic depth ordering (72%–100% with the true relation vs. 0%–16% when kept, five models), and image re-supply every 48 tokens during generation (+4.6 to +7.9 points on new CoT errors, three models).
  • Tables and protocols for these analyses are in Appendices B, C, K, and L. Appendix tables and figures are renumbered (e.g., old Tables 10, 11, 13, 14 are now 16, 17, 19, 20; old Fig. 5 is now Fig. 6), and our replies use the new numbers.
  • The supplementary archive (SHA-256 7ba1becc4be4b2596240bb9d1bec43dd3cf414b47b213b06c0df02a78ccf0055) adds the scripts and outputs of the added analyses.

As the discussion ends on November 18 (AoE), we would be grateful for your feedback on the revision, and we are happy to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.