- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SteerSeg, a lightweight framework for reasoning video segmentation using frozen vision-language and segmentation models. Motivated by a discrepancy between semantic understanding and spatial grounding, it learns soft prompts with mask supervision to align attention rollout maps with target objects. A reasoning step generates distinguishing attributes to improve instance selection, while frame-level and video-level attention guide SAM2 candidate generation and tracklet ranking. Experiments across referring and reasoning video segmentation benchmarks, multiple vision-language backbones, and component ablations demonstrate improved grounding and cross-dataset transfer.
Critical strengths
- Learning input prompts to improve spatial attention while freezing both foundation models is a useful adaptation strategy. The method addresses a practical gap between extracting unmodified attention from a pretrained model and extensively adapting a segmentation system.
- The ablations provide meaningful evidence for the principal components. Table 3 includes all combinations of soft prompts and reasoning-derived attributes, showing that soft prompts account for most of the improvement while attributes provide additional gains, particularly on ReasonVOS.
Critical weaknesses
- The comparison does not adequately represent recent progress in reasoning video segmentation. VIRST[1], VideoSEG-O3[2], and Seg-ReSearch[3] are absent despite overlapping evaluation benchmarks. CoT-RVS[4] is cited but not quantitatively compared, although its training-free reasoning pipeline is particularly relevant. These omissions leave the method’s current accuracy–adaptation-cost trade-off insufficiently established. Different supervision and inference resources should be reported explicitly.
- The distinctive methodological contribution is not sufficiently separated from the benefit of adding mask supervision. F-LMM[5] already learns grounding from frozen multimodal attention, and InterRVOS[6] employs attention-mask supervision in video segmentation. SteerSeg’s specific contribution is therefore narrower: adapting input embeddings to reshape rollout attention.
[1] VIRST: Video-Instructed Reasoning Assistant for SpatioTemporal Segmentation, CVPR 2026.
[2] VideoSEG-O3: A Multi-turn Reinforcement Learning Framework for Reasoning Video Object Segmentation, ICML 2026.
[3] Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search, ICML 2026.
[4] CoT-RVS: Zero-Shot Chain-of-Thought Reasoning Segmentation for Videos, ICLR 2026.
[5] F-LMM: Grounding Frozen Large Multimodal Models, CVPR 2025.
[6] InterRVOS: Interaction-Aware Referring Video Object Segmentation, CVPR 2026.
Questions for authors
- How does SteerSeg compare with VIRST, VideoSEG-O3, Seg-ReSearch, and CoT-RVS? Does its advantage remain compelling when evaluated as an accuracy–cost trade-off?
- Under the same backbone, mask supervision, sampled frames, and SAM2 pipeline, how does input-level attention steering compare with learning a lightweight readout over frozen attention maps? This comparison would materially clarify the contribution beyond supervised grounding.
- How much improvement comes from better candidate generation versus better tracklet ranking? Can these effects be separated by holding the candidate set fixed and changing only the attention used for selection?
- How is successful grounding defined in the diagnostic study, and what evidence distinguishes attention localization failures from segmentation and propagation failures? The answer would affect the claim that attention misalignment is the primary bottleneck.
- In Table 2, SteerSeg with Qwen2VL-7B outperforms its Qwen2.5VL-7B counterpart on both MeViS and Ref-YouTube-VOS, reversing the trend observed in Table 1. Please explain why.