acceptodds
Withdrawn from ICLR 2027

SteerSeg: Attention Steering for Reasoning Video Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video reasoning segmentation requires localizing objects across video frames from natural language expressions, often involving spatial reasoning and implicit references. Recent approaches leverage frozen large vision-language models (LVLMs) by extracting attention maps and using them as spatial priors for segmentation, enabling training-free grounding. However, these attention maps are optimized for text generation rather than spatial localization, often resulting in diffuse and ambiguous grounding signals. In this work, we introduce SteerSeg, a lightweight framework that identifies attention misalignment as the key bottleneck in attention-based grounding and proposes to steer attention at its source through input-level conditioning. SteerSeg combines learnable soft prompts with reasoning-guided Chain-of-Thought (CoT) prompting. The soft prompts reshape the attention distribution to produce more spatially concentrated maps, while CoT-derived attributes resolve ambiguity among similar objects by guiding attention toward the correct instance. The resulting attention maps are converted into point prompts across keyframes to guide a segmentation model, while candidate tracklets are ranked and selected using correlation-based scoring. Our approach freezes the LVLM and segmentation model parameters and learns only a small set of soft prompts, preserving the model’s pretrained reasoning capabilities while significantly improving grounding. Despite being trained only on Ref-YouTube-VOS, SteerSeg generalizes well across diverse benchmarks, significantly improving the spatial grounding capability of LVLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Reasoning video segmnetion, LVLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y5MkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SteerSeg, a lightweight framework for reasoning video segmentation using frozen vision-language and segmentation models. Motivated by a discrepancy between semantic understanding and spatial grounding, it learns soft prompts with mask supervision to align attention rollout maps with target objects. A reasoning step generates distinguishing attributes to improve instance selection, while frame-level and video-level attention guide SAM2 candidate generation and tracklet ranking. Experiments across referring and reasoning video segmentation benchmarks, multiple vision-language backbones, and component ablations demonstrate improved grounding and cross-dataset transfer.

Critical strengths
  1. Learning input prompts to improve spatial attention while freezing both foundation models is a useful adaptation strategy. The method addresses a practical gap between extracting unmodified attention from a pretrained model and extensively adapting a segmentation system.
  2. The ablations provide meaningful evidence for the principal components. Table 3 includes all combinations of soft prompts and reasoning-derived attributes, showing that soft prompts account for most of the improvement while attributes provide additional gains, particularly on ReasonVOS.
Critical weaknesses
  1. The comparison does not adequately represent recent progress in reasoning video segmentation. VIRST[1], VideoSEG-O3[2], and Seg-ReSearch[3] are absent despite overlapping evaluation benchmarks. CoT-RVS[4] is cited but not quantitatively compared, although its training-free reasoning pipeline is particularly relevant. These omissions leave the method’s current accuracy–adaptation-cost trade-off insufficiently established. Different supervision and inference resources should be reported explicitly.
  2. The distinctive methodological contribution is not sufficiently separated from the benefit of adding mask supervision. F-LMM[5] already learns grounding from frozen multimodal attention, and InterRVOS[6] employs attention-mask supervision in video segmentation. SteerSeg’s specific contribution is therefore narrower: adapting input embeddings to reshape rollout attention.

[1] VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation, CVPR 2026.

[2] VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation, ICML 2026.

[3] Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search, ICML 2026.

[4] CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videos, ICLR 2026.

[5] F-LMM: Grounding Frozen Large Multimodal Models, CVPR 2025.

[6] InterRVOS: Interaction-Aware Referring Video Object Segmentation, CVPR 2026.

Questions for authors
  1. How does SteerSeg compare with VIRST, VideoSEG-O3, Seg-ReSearch, and CoT-RVS? Does its advantage remain compelling when evaluated as an accuracy–cost trade-off?
  2. Under the same backbone, mask supervision, sampled frames, and SAM2 pipeline, how does input-level attention steering compare with learning a lightweight readout over frozen attention maps? This comparison would materially clarify the contribution beyond supervised grounding.
  3. How much improvement comes from better candidate generation versus better tracklet ranking? Can these effects be separated by holding the candidate set fixed and changing only the attention used for selection?
  4. How is successful grounding defined in the diagnostic study, and what evidence distinguishes attention localization failures from segmentation and propagation failures? The answer would affect the claim that attention misalignment is the primary bottleneck.
  5. In Table 2, SteerSeg with Qwen2VL-7B outperforms its Qwen2.5VL-7B counterpart on both MeViS and Ref-YouTube-VOS, reversing the trend observed in Table 1. Please explain why.
Reviewer_Avr2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduce SteerSeg for the video reasoning segmentation task. SteerSeg is a lightweight framework that identifies attention misalignment and proposes to steer attention. It first combines soft prompts and CoT to produce more spatially concentrated maps. The resulting attention maps are converted into point prompts across keyframes to guide a segmentation model (i.e., SAM). Despite being trained only on Ref-YouTube-VOS, the proposed method achieves impressive performance on multiple datasets.

Critical strengths

1, The paper is well-written and easy to follow. 2, The proposed method achieves strong performance across multiple benchmarks. Extensive experiments further demonstrate its effectiveness. 3, The paper is well-motivated, and the proposed method effectively addresses the limitations of existing LVLM- and SAM-based video segmentation methods.

Critical weaknesses
  1. My primary concern is the limited generalizability of the proposed method. Although SteerSeg achieves promising performance, it is specifically designed for LVLM-based video segmentation frameworks that rely on internal attention maps for spatial grounding. However, many effective video segmentation methods do not rely on large pretrained LVLMs. Therefore, it remains unclear how broadly the proposed method can be applied beyond this specific framework. The authors should further discuss the generalizability and applicability of SteerSeg to other video segmentation methods.
  2. Although the proposed method demonstrates strong cross-dataset generalization, achieving promising results on multiple benchmarks after being trained only on Ref-YouTube-VOS, it remains unclear to what extent this generalization can be attributed to the proposed modules rather than the powerful pretrained capabilities of the LVLM and SAM2. The authors should further investigate whether the proposed modules improve cross-dataset generalization beyond their contributions to overall segmentation performance.
  3. The proposed method optimizes the internal attention maps of LVLMs using ground-truth segmentation masks. However, it remains unclear why steering internal attention is preferable to directly training a lightweight grounding head on frozen LVLM features under the same supervision.
  4. Although the proposed method introduces only a small number of trainable parameters and requires relatively limited training time, the overall framework still relies on large pretrained LVLM and SAM2 models, which may incur considerable computational costs during inference. It would be helpful if the authors could provide a more comprehensive efficiency analysis, including the total number of parameters, FLOPs, inference time, and GPU memory consumption, to further evaluate the efficiency of the proposed method.
Questions for authors

See Weaknesses

Reviewer_VX7JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SteerSeg, a lightweight framework that addresses attention misalignment in VLM-based referring and reasoning video object segmentation. It combines CoT-derived object attributes with learnable soft prompts to improve attention alignment with the target object, while keeping the LVLM and SAM2 frozen. The resulting attention rollout maps provide point prompts for SAM2, and candidate tracklets are ranked by their correlation with the attention maps. Experiments show improvements over DecAF and competitive performance across multiple video object segmentation benchmarks.

Critical strengths
  1. Consistent empirical improvements across benchmarks. SteerSeg consistently improves over DecAF across multiple LVLM backbones and achieves strong performance among the evaluated frozen-LVLM methods on Ref-DAVIS, ReasonVOS, and ReVOS. For example, with Qwen2.5VL-7B, it improves J&F over DecAF by 6.2 points on Ref-DAVIS and 2.0 points on ReasonVOS. These improvements extend beyond its Ref-YouTube-VOS training dataset, supporting the effectiveness and transferability of the learned attention steering.

  2. Parameter-efficient adaptation across different LVLMs. SteerSeg learns only a small set of soft prompts while keeping both the LVLM and SAM2 frozen. The comparison with LoRA under the same training data, attention-supervision objective, and optimization-step budget shows better segmentation performance with fewer trainable parameters. Its effectiveness across LLaVA-OV, InternVL3, Qwen2VL, and Qwen2.5VL further suggests that the approach is applicable across several model architectures.

Critical weaknesses
  1. Limited novelty beyond DecAF. I did not find much methodological novelty over DecAF. SteerSeg largely follows DecAF’s attention-rollout idea and SAM2-based segmentation pipeline. The main additions are mask-supervised soft prompts and CoT-derived attributes, with some changes to tracklet selection. These additions improve performance, but I see them mostly as incremental engineering changes to the existing framework.

  2. Insufficient evidence for the central bottleneck claim. I am not fully convinced by the diagnosis in Figure 1. Correct attribute descriptions with incorrect segmentation do not necessarily mean that attention misalignment is the main problem. Errors could also come from point selection, SAM2 segmentation, or tracklet ranking. Also, Fig. 1(b2) is discussed in connection with DecAF, but it is unclear how this map was generated and which parts of DecAF’s attention-processing procedure were applied. This makes it hard to judge how representative this example is of DecAF’s limitations.

Questions for authors
  1. How was the contrast attention map in Fig. 1(b2) generated? Which parts of DecAF’s attention-processing procedure were applied, and were the backbone and input settings the same as those used for SteerSeg?

  2. Could you clarify the training-cost comparison? Section 4 mentions two A6000 GPUs, while Table 9 reports 10 GPU-hours on one A100. Were the 10-hour soft-prompt and 113-hour LoRA results in Table 7 measured using the same hardware and input settings? Do these costs include training both the frame and video branches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.