- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates why short chain-of-thought (CoT) prompting can reduce spatial question-answering accuracy in vision-language models. Across five open models, the authors compare generated CoT with length-matched filler, dialogue-turn controls, question restatement, image re-injection, and conclusion edits. They report that content-independent text length alone does not explain the observed losses. They also identify an empirical correspondence between CoT accuracy and the mean correct-option probability under Direct answering, including a prediction of four-sample voting accuracy. A reduced API evaluation extends the analysis to larger or closed models, while probing, attention analysis, and activation patching provide complementary evidence in Qwen2.5-VL.
Critical strengths
The behavioral controls address consequential experimental confounds. In particular, matching dialogue-turn structure substantially changes the effect of neutral filler. This demonstrates that prompt formatting can otherwise be mistaken for a reasoning-length effect. Together with length matching, re-prompting, and image re-injection, these controls provide useful evidence against content-independent length as a sufficient explanation for the tested short-CoT losses.
Besides, the relationship with Direct probabilities is an interesting quantitative observation. The paper goes beyond comparing average Direct and CoT accuracy: it uses item-level Direct probabilities to predict four-sample voting accuracy. This is potentially useful as a baseline for studying when additional reasoning improves or degrades performance, and is the most distinctive empirical contribution of the paper.
Critical weaknesses
-
The scope and explanatory value of the probability correspondence remain insufficiently established. The agreement between pooled CoT accuracy and is interesting, but item-level relationships differ across models, and the behavioral effects vary substantially across tasks. It remains unclear how much the pooled correspondence reflects a stable relationship within tasks versus offsetting deviations across tasks. The decomposition organizes the observations but adds no independent explanation by itself, since is defined using observed CoT accuracy. The API extension also uses a different probability-estimation protocol and includes Direct calls that sometimes contain reasoning. These limitations affect whether the paper provides a broadly informative account beyond its controlled behavioral findings.
-
The supplied supplementary code does not appear to correspond to the current manuscript. The archive contains an earlier experiment pipeline, lacks the
scripts/v2api/directory mentioned in the appendix, and does not clearly provide the main behavioral, content-editing, and patching experiments described in the paper. The experiment runner also references scripts absent from the archive. Consequently, I cannot trace the central empirical claims to their implementation or verify the reported analyses from the supplied package. This may be a packaging issue, but it materially limits reproducibility in the current submission.
Questions for authors
-
Can you report the CoT- differences and four-sample voting prediction errors separately for the main spatial tasks, with uncertainty estimates? This would help establish whether the central correspondence holds within tasks or depends substantially on pooling heterogeneous tasks.
-
Does the supplied ZIP correspond to the current manuscript? Please identify or provide the scripts implementing the main E1/E2/E3/E5 experiments and the API evaluation, together with the output files or data manifest needed to reproduce the central analyses.
2 comments
Thank you for the detailed responses and additional analyses. I appreciate the authors' efforts in addressing my concerns, particularly the additional per-task analyses and improvements to reproducibility. However, the new results do not fundamentally change my assessment of the paper's overall contribution and limitations. Therefore, I maintain my original rating.
Response to Reviewer aa9i
Thank you for noting our clear presentation, that our behavioral controls (turn-structure and length matching, re-prompting, image re-injection) address consequential confounds and show prompt format can be mistaken for a reasoning-length effect, and that our item-level Direct-probability prediction of four-sample voting is the most distinctive empirical contribution. General Response items 1 (W1–W3, Q1), 2 (full-vocabulary readout), and 3 (code, W4/Q2) cover your points; details follow below.
W1. Pooled vs. per-task agreement. We agree that single-CoT pooled agreement (CoT≈pˉ, ≤1.7 pts; ≤1.6 full-vocabulary) averages across task and confidence deviations (WmcJ-W4). Sec. 4.4 noted flatter slopes (0.47–0.68) in 3 models and +7.9 pts in Molmo-7B-D, and Tables 12–13 lacked per-task pˉ tests; in Q1, equal-task means in the 4 models are +0.1,+1.3,−0.4,−1.1 pts vs. mean absolute gaps of 5.5,4.1,3.8,5.8 pts.
Q1. Per-task CoT−pˉ and four-sample voting errors. Across the 5 tasks in Table 12 (n=650,600,600,500,500; CVB = CV-Bench), 95% CIs exclude 0 in 11/20 CoT−pˉ and 15/20 vote-residual cells (4 models) and 3/5 Molmo-7B-D cells (top-20 readout, median option prob. 0.998; no four-sample runs, App. F):
1. Per-task CoT−pˉ (pts):
2. Per-task four-sample vote residuals (T=0.7 vote minus Bin(4,pi) prediction, full-vocabulary pi):
Pooled observed/predicted vote accuracy is 75.5/74.4 (top-20: 74.5), 84.5/83.1, 86.9/86.8, and 72.5/71.7. We will add both tables to Sec. 4.4 and App. C in the revision.
W2. E−G decomposition. We agree that E−G is an accounting identity since E=CoT−pˉ uses observed CoT accuracy (App. A). In the revision, we will remove E−G from our contributions (Sec. 1) and keep it in Sec. 4.4, Fig. 1c, and Table 4 as a descriptive summary.
W3. API protocol. We agree that our API runs (Sec. 4.6, Table 4, App. I Table 19) lacked logprobs (pˉ^ from 6 samples, T=1) and included reasoning in 56% of GPT-5.5 and 72% of GPT-5.6-Sol Direct calls. In the revision, we will label Table 4 as a reduced-protocol descriptive extension whose caption notes both differences, and base Secs. 1 and 8 on the open-model protocol.
W4. Supplementary code. We apologize for attaching an earlier archive without the main scripts. Our replacement archive (949 files, SHA-256
7ba1becc4be4b2596240bb9d1bec43dd3cf414b47b213b06c0df02a78ccf0055) has all scripts, outputs, and aMANIFESTmapping each table, figure, and key number to script and output. A CPU-only rebuild (bash scripts/v2/make_all_v2.sh && python scripts/check_against_reference.py) reproduces all 6,087 macros and 8 generated tables of the submitted PDF, except Table 18/Fig. 6 (App. H, earlier pipeline) and Table 20 (from Kancheti et al.). We uploaded it as the revised supplementary material, replacing the earlier ZIP.Q2. Scripts and outputs. No, it came from an earlier version (W4); in the uploaded ZIP, E1 is
scripts/v2/e1_behavior_factorial.py, E2e2_posthoc_relation_probe.pyande2_readout_mechanism.py, E3e3_patching.py, E5e5_content_edit.py, and APIscripts/v2api/e9_api_factorial.pyande9_analyze.py(results/v2/<experiment>/,bash scripts/run_experiments.sh <experiment>). We will release code for the new analyses in the revision.We hope these answers address your concerns, and we are happy to clarify anything further during the discussion.