- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the distinction between "accessing the position of numbers" and "being influenced by those numerical values" in visual Chain-of-Thought reasoning. Keeping the existing reasoning fixed, the authors show through individual and joint masking of number positions that independent masking can miss joint dependencies, and expanding the masking region can even paradoxically restore the original answer. Further numerical replacement and non-numerical state-transfer experiments reveal that even when number positions are masked, numerical influence can still propagate via internal states at other locations. Based on these findings, the authors propose Gate, which selects between CoT voting answers and independent direct answers based on response features. The approach is validated through internal interventions on two primary models, 800 synthetic scenarios, 400 natural error trajectories, and answer-selection experiments across multiple models and tasks. Gate outperforms voting on new ChartQA images and matches an answer scorer using matched features.
Critical strengths
Uncovers interesting and diagnostically valuable phenomena. The paper demonstrates that a visual CoT's reliance on numerical information cannot always be determined by independently masking corresponding positions: masking two sets of numbers separately may not affect the answer, whereas joint masking changes it; expanding the masking region can even restore the original answer. These phenomena serve as a timely reminder to interpret models' information usage with caution when relying solely on local ablation results.
Provides systematic analysis and experimental validation. The authors combine various masking operations, numerical replacements, non-numerical state transfers, and paired replacement-value controls, further testing them across independent synthetic scenarios and natural error trajectories. In particular, the state-transfer experiments provide more direct evidence of numerical propagation than simply observing whether the answer is preserved, elevating the findings beyond isolated case studies.
Critical weaknesses
Insufficient independent advantage, cross-scenario stability, and cost-benefit justification for Gate. Compared to voting, Gate incurs the extra cost of generating direct answers, yet its advantages over more competitive selection strategies are limited. On new ChartQA images, both Gate and the answer scorer using matched features achieve 85.00%, with identical sample-by-sample correctness. Its advantage over a fixed fallback is also modest, and it fails to consistently outperform the fixed fallback in classification and localization extension tasks. Thus, while the results support the value of learned answer selection, they fall short of demonstrating clear, stable additional benefits specific to Gate.
Limited mechanistic explanation and guidance value for real-world errors. Although the paper performs attention position masking, value zeroing, and state transfer, the analysis primarily characterizes output changes under intervention. It does not clearly explain the specific formation mechanisms underlying joint dependencies and non-monotonic recovery, nor does it establish the necessity of the identified pathways in natural responses. Across 400 natural error trajectories, non-numerical state transfer detected a positive numerical-following effect in only 3 images, with the overall confidence interval spanning zero. Consequently, while the paper convincingly proves that these phenomena can occur, there remains insufficient evidence regarding their importance for real-world visual reasoning failures or whether they can guide effective error diagnosis and correction.
Questions for authors
Cost-benefit and stability of Gate. On new ChartQA images, Gate performs identically to the matched-feature answer scorer, and its advantage over a fixed fallback is minor. What results support the independent methodological value of Gate? When accounting for the extra direct generation calls, is there a comparison under the same inference budget demonstrating that it outperforms simpler strategies like increased sampling?
Internal mechanisms behind the phenomena. To what extent can the current attention intervention and state-transfer experiments explain joint dependencies and non-monotonic recovery? Specifically, how can one disentangle the respective roles of numerical information propagation, competing numerical influences, and attention normalization changes caused by masking?
Diagnostic value for real-world errors. In the 400 natural error trajectories, non-numerical state transfer detected a positive numerical-following effect in only 3 images. How can one distinguish whether this stems from limited phenomenon coverage or from precise numerical matching missing computationally transformed effects? What practical error identification or correction use cases are supported by the current findings?
Thank you for recognizing the diagnostic value and raising questions about Gate, mechanism and practical use.
1. Gate: selection value and cost.
Gate learns when to retain CoT-supported Vote or answer afresh with Direct. Matched shares the candidates, 68 features and data splits, but predicts candidate correctness rather than Direct-minus-Vote correctness gain.
Gate exceeds fallback by 1.54 pp [1.19,1.89] in development and 1.27 [0.92,1.62] under benchmark holdout; it exceeds a training-fold-tuned agreement threshold by 0.47 [0.20,0.76] in development. Gate-minus-Matched is +0.10 [−0.01,0.21] and +0.02 [−0.14,0.18], respectively (all 95% CIs). Frozen correctness is identical for Gate and Matched; the prespecified Gate-minus-fallback difference is +0.58 [−0.67,1.83] (97.5%). Grouped and benchmark-holdout results support learned selective reuse over fallback; the gain objective's extra accuracy over Matched remains unestablished.
We measured six strategies separately on the same first 100 ChartQA images for both 8B models, using one RTX A6000, batch size 1. Time includes image loading, visual encoding, prefill, generation and selection, excluding model loading and warmup. Values are accuracy (%) / mean seconds per image; IF denotes all-invalid Direct fallback.
Adaptive sampling adds a fourth CoT under Gate's non-unanimity condition and votes over four, using Direct if all are invalid. Gate and adaptive sampling average 3.16/3.14 calls (Qwen/InternVL); token counts differ, so this matches calls, not exact compute. Gate's mean time is 25.2%/23.9% lower than Vote-4+IF and 1.30/0.62 seconds lower than adaptive sampling. Timing is newly measured; accuracy replays the existing fixed seed.
Across five seed/temperature conditions, Gate exceeds Vote-4+IF by 0.5–1.0 pp; all exploratory 95% paired intervals reach zero (Response Evidence Table E5).
2. What the intervention controls establish.
Earlier prefill blocking preserves TabMWP joint dependence, including with original images. Value zeroing retains keys and softmax normalization at the intervened call; later states can evolve. Paired value transfer moves nonnumeric states from an edited run into the recipient while preserving its text, testing continued influence of the replacement value.
The submitted numerical-competition control compares a competing-value mask with an equal-size control on selected blank-image cases:
Only Qwen passes the adjusted exact test; the joint two-model criterion is not met. Mask locations and content also differ. This constrains the competition explanation in selected cases without fully decomposing propagation, competition and normalization. Recovery is intervention-sensitive and cannot alone establish numerical irrelevance.
3. Natural errors and diagnostic utility.
On 400 original-image Qwen ChartQA error traces, changing the replacement value changes numerical outputs on 86 images under source editing and 54 under nonnumeric-state transfer; 73/51 respectively have no exact replacement match. Exact transfer yields three positive images and a net used-minus-unused effect of 0.38 pp (97.5% CI [0.00,0.88]). This distinguishes output sensitivity from literal numerical following.
An exploratory AI-assisted audit of edited prefixes found 33 agreed direct mappings, 184 provisionally ineligible cases and 183 unresolved cases. No non-identity transformation met confirmation criteria. Many edits replace a completed result rather than an upstream operand, preventing a unique transformed target without additional assumptions. The transformed-following effect therefore remains unestimable, rather than showing that transformations do not occur (Response Evidence, Section 6).
The supported use is controlled diagnosis of answer dependence: identifying joint reliance missed by local ablations and testing continued influence of substituted values. Effective localization or correction of natural visual errors remains a future application.
Our diagnostic experiments establish how local interventions can mischaracterize numerical dependence; Gate evaluates the practical choice between reusing CoT computation and answering afresh.