acceptodds
Under review as a conference paper at ICLR 2027

SAPS-COT: STEP-AWARE VISUAL GROUNDING WITH DYNAMIC RESIDUAL INJECTION FOR INTERPRETABLE MULTIMODAL REASONING

Abstract

While Large Vision-Language Models (LVLMs) enhanced by Chain-of-Thought (CoT) prompting have significantly advanced multimodal reasoning, they predominantly suffer from a “visual black box” dilemma and a “perceptual lock-in effect.” Existing paradigms typically extract visual features once and freeze them, forcing the model to rely heavily on language priors during intermediate reasoning steps. This often leads to visual hallucinations and cascading errors. To address this, we propose SAPS-CoT, a two-stage reasoning framework that seamlessly integrates Step-Aware Patch Selection (SAPS) with a Dynamic Residual Injection mechanism. In the discovery phase, SAPS establishes a fine-grained, explicit alignment between textual reasoning steps and visual regions by leveraging multi-dimensional metrics, including semantic consistency, gradient saliency, and location bias. In the refinement phase, we introduce an in-place latent residual injection mechanism as an alternative to token concatenation. It dynamically updates the model’s visual memory while preserving the original visual-token sequence length. Controlled comparisons show modest but consistent accuracy gains over concatenation, with a small implementation-dependent latency premium. Extensive experiments on complex real-world scenarios demonstrate that SAPS-CoT mitigates visual hallucinations and outperforms state-of-the-art multimodal CoT methods in both accuracy and interpretability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
COT, VQA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YNXFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets the "visual black box" and "perceptual lock-in" failure of multimodal CoT: visual tokens are encoded once and gradually lose influence as the reasoning chain grows. The proposed SAPS-CoT operates in two stages. A discovery stage scores patches for every reasoning step with a battery of criteria — semantic consistency, gradient saliency, spatial-keyword bias, multi-scale window pooling, and decoder cross-attention — and keeps the Top-K evidence patches. A refinement stage then rewrites the frozen visual features in place through forward hooks and attention-based residual fusion, leaving sequence length and KV cache untouched. The average accuracy ahead of the compared plug-and-play CoT baselines, together with patch-level interpretability analyses.

Critical strengths
  1. The residual-injection design is genuinely appealing. Unlike token concatenation it keeps the visual sequence and KV cache at constant length, which is engineering-friendly, and the authors deserve credit for honestly reporting the latency premium introduced by the hooks.
  2. Step-level explicit patch alignment yields interpretability. Binding each reasoning step to concrete regions makes the model's perceptual basis auditable, something black-box CoT cannot offer, and the perceptual lock-in motivation is well articulated.
Critical weaknesses
  1. The scoring rule is a product of hand-crafted heuristics with many knobs (α, γ, per-dimension weights, window scales). There is no principled justification for this particular combination and no systematic per-dimension ablation, so the method reads as engineering stacking whose essential ingredient remains unclear.
  2. The empirical case is weak. Improvements are modest (roughly one point on average), the backbones are dated, trainable competitors such as ICoT and Visual CoT are not compared empirically, and the reproduced SCAFFOLD numbers (e.g., 24.1 on MathVista) fall far below its published results, casting doubt on baseline fairness; the self-built refined subset is also small.
Questions for authors
  1. Gradient saliency needs a backward pass per reasoning step — how do its cost and benefit trade off under the training-free premise?
  2. How much does each scoring dimension (semantic, gradient, location, window, global attention) contribute? A dimension-by-dimension ablation seems necessary.
  3. Why does the reproduced SCAFFOLD diverge so much from the original paper's numbers? Was evaluation-protocol consistency checked?
Reviewer_LcYFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SAPS-CoT, a two-stage framework designed to improve visual grounding and mitigate visual hallucinations in multimodal Chain-of-Thought (CoT) reasoning. Via Step-Aware Patch Selection (SAPS), the method selects relevant visual evidence for different reasoning steps by combining semantic consistency, gradient saliency, positional bias, and multi-scale window compensation. Furthermore, the authors propose Dynamic Residual Injection, which fuses the selected visual features into the original representations without increasing the length of the visual token sequence, achieving performance improvements across multiple multimodal reasoning benchmarks.

Critical strengths
  1. This paper targets an important challenge in multimodal reasoning: establishing explicit correspondence between intermediate reasoning steps and relevant visual evidence to improve reasoning reliability.
  2. The proposed framework integrates step-wise visual region selection and the dynamic residual injection mechanism. It requires no additional model fine-tuning and does not increase the length of the visual token sequence, demonstrating certain practical value.
  3. The paper conducts experiments on multiple benchmark datasets with two distinct LVLM backbones. It also presents ablation studies and controlled comparison experiments, which provide empirical evidence supporting the effectiveness of the proposed method.
Critical weaknesses
  1. The author constructs a Raw Relevance Score based on semantic consistency, gradient saliency, spatial location, and multi-scale features. However, these metrics are essentially heuristic feature-selection criteria; they are insufficient to prove that the selected regions truly constitute the critical visual evidence for a given reasoning step. Consequently, the claim that this approach resolves the visual black-box issue overstates its contribution.

  2. Table 1 demonstrates promising results of SAPS-CoT across most tasks, yet the average gain over the strongest Multimodal-CoT baseline is modest. Without repeated runs, confidence intervals or statistical significance tests, the stability of these marginal improvements cannot be well validated. Moreover, SAPS-CoT performs worse than Multimodal-CoT on ScienceQA, with a 0.7 pp drop for LLaVA-NeXT-8B and a 1.3 pp drop for Qwen2-VL-7B.

  3. The paper highlights Dynamic Residual Injection as an architectural advantage that keeps the visual token sequence length unchanged. Nevertheless, fixed sequence length does not guarantee low computational overhead for the whole inference pipeline. It is suggested to report end-to-end latency, peak GPU memory and FLOPs, accounting for both initial CoT generation and Gradient Saliency calculation.

  4. While interpretability and hallucination mitigation are claimed as core contributions, the supporting evidence is limited to final answer accuracy and only a few visualization examples.

Questions for authors
  1. Section 4.1 mentions the use of CLIP-ViT-L/14 as a visual encoder, while employing LLaVA-NeXT-8B and Qwen2-VL-7B as the backbones for the experiments. Given that these two models utilize different native visual encoding architectures, the authors should clarify whether CLIP serves as an auxiliary feature extractor or replaces the native visual encoder, and explain how compatibility is maintained without requiring additional training.

  2. The paper states that a Refined Real-World Subset with 1,500 samples is constructed from M3 CoT. However, Appendix Table 5 refers to these 1,500 samples as the full M3 CoT Evaluation Set. We suggest clarifying whether the main experiments use the original test set or this filtered subset, and providing the filtering criteria.

  3. In Appendix Table 6, M3 CoT accuracy decreases from 56.71% to 54.21% when αpatch\alpha_{\mathrm{patch}} grows from 0.2 to 0.4, before climbing to 62.21% at 0.8. This contradicts the authors’ statement that raising this coefficient continuously improves performance.

Reviewer_aRGuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SAPS-CoT, a two-stage framework for improving visual grounding in multimodal reasoning without fine-tuning. The first pass generates a reasoning chain that guides patch selection using semantic similarity, gradient saliency, spatial priors, and multiscale information. The selected top-(K) visual features are then incorporated through an attention-based residual update for a second reasoning pass, preserving the visual-token sequence length. The method is evaluated using two backbones, LLaVA-NeXT-8B and Qwen2-VL-7B, across four benchmarks: MathVista, M³CoT, GQA, and ScienceQA.

Critical strengths

1- The paper targets an important problem: MLLMs often underuse visual evidence during multi-step reasoning, which leads to errors in intermediate steps. The proposed approach is training-free. It uses the first-pass reasoning steps to select step-relevant visual tokens, then refines the visual representation in place through a residual update rather than appending tokens, so the sequence length is unchanged.

2- The paper reports that residual injection is slightly slower than concatenation in the tested settings, and they have included some failure cases.

Critical weaknesses

1- The execution of step-aware selection and injection needs clarification. Section 3.2 defines step-dependent patch selection, but Section 3.3 describes injection using a single patch set (P). Is the evidence pooled across first-pass reasoning steps, updated during second-pass generation, or injected through separate passes? This distinction affects both the interpretation and implementation of the method. Other important details are also insufficiently specified, including the exact hook location, the feature representations used for selection and injection, how text and visual embeddings are aligned, and the targets used to compute each step’s generation loss. Please clarify these details to support reproducibility.

2 - The reliability of the reported gains and the evaluation scope need clarification. Table 1 shows average improvements over Multimodal-CoT of 1.25 percentage points for LLaVA-NeXT-8B and 0.50 points for Qwen2-VL-7B, while performance decreases on ScienceQA for both backbones. Given these modest gains, please report paired uncertainty estimates and, if decoding is stochastic, results across repeated runs to assess their reliability. The dataset description is also ambiguous: Section 4.1 introduces a filtered 1,500-example M³CoT subset, while the implementation details and Table 5 describe a complete 1,500-example M³CoT evaluation set. Does “complete” refer to the filtered subset? Please clarify which results use this subset and provide its selection procedure and sample identifiers.

Questions for authors

1- Please refer to the weakness section.

2- In Equation (8), the scalar gates αpatch\alpha_{\mathrm{patch}} and βres\beta_{\mathrm{res}} affect the residual update only through their product. However, Table 6 reports different results for (αpatch,βres)=(0.4,0.4)(\alpha_{\mathrm{patch}},\beta_{\mathrm{res}})=(0.4,0.4) and (0.8,0.2)(0.8,0.2), although both products equal 0.16: M3CoT\mathrm{M^3CoT} accuracy is 54.21 versus 55.67, and GQA accuracy is 66.78 versus 63.53. Is there something wrong with my understanding? Or is there something missing?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.