- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets the "visual black box" and "perceptual lock-in" failure of multimodal CoT: visual tokens are encoded once and gradually lose influence as the reasoning chain grows. The proposed SAPS-CoT operates in two stages. A discovery stage scores patches for every reasoning step with a battery of criteria — semantic consistency, gradient saliency, spatial-keyword bias, multi-scale window pooling, and decoder cross-attention — and keeps the Top-K evidence patches. A refinement stage then rewrites the frozen visual features in place through forward hooks and attention-based residual fusion, leaving sequence length and KV cache untouched. The average accuracy ahead of the compared plug-and-play CoT baselines, together with patch-level interpretability analyses.
Critical strengths
- The residual-injection design is genuinely appealing. Unlike token concatenation it keeps the visual sequence and KV cache at constant length, which is engineering-friendly, and the authors deserve credit for honestly reporting the latency premium introduced by the hooks.
- Step-level explicit patch alignment yields interpretability. Binding each reasoning step to concrete regions makes the model's perceptual basis auditable, something black-box CoT cannot offer, and the perceptual lock-in motivation is well articulated.
Critical weaknesses
- The scoring rule is a product of hand-crafted heuristics with many knobs (α, γ, per-dimension weights, window scales). There is no principled justification for this particular combination and no systematic per-dimension ablation, so the method reads as engineering stacking whose essential ingredient remains unclear.
- The empirical case is weak. Improvements are modest (roughly one point on average), the backbones are dated, trainable competitors such as ICoT and Visual CoT are not compared empirically, and the reproduced SCAFFOLD numbers (e.g., 24.1 on MathVista) fall far below its published results, casting doubt on baseline fairness; the self-built refined subset is also small.
Questions for authors
- Gradient saliency needs a backward pass per reasoning step — how do its cost and benefit trade off under the training-free premise?
- How much does each scoring dimension (semantic, gradient, location, window, global attention) contribute? A dimension-by-dimension ablation seems necessary.
- Why does the reproduced SCAFFOLD diverge so much from the original paper's numbers? Was evaluation-protocol consistency checked?