acceptodds
Under review as a conference paper at ICLR 2027

SteerGrounding: Instance-Adaptive Representation Steering for Reasoning Segmentation with Frozen LMMs

Abstract

Reasoning segmentation requires models to infer an implicitly described target from an image–query pair and delineate it at the pixel level. Existing frozen-LMM grounding methods seek to extract spatial cues from pretrained multimodal representations and decode them into pixel-level predictions while largely preserving the general multimodal capabilities of the pretrained LMMs. However, representations produced by a frozen LMM may not always provide sufficiently reliable target-specific spatial evidence for each image–query pair, leaving downstream refinement to operate on incomplete or ambiguous cues without directly adapting the target-related hidden states. To address this problem, we propose SteerGrounding, which introduces instance-adaptive representation steering into a frozen LMM. Specifically, we introduce Instance-Conditioned Latent Steering (ICLS), which trains a lightweight steering predictor using layer-wise representation differences between the original image and its target-erased counterpart as target-presence supervision. At inference, the predictor directly estimates instance-specific steering from the original image–query pair and injects it into the frozen LMM hidden states for representation adaptation. Building on this, we introduce a Steering-Conditioned Grounding Decoder (SCGD), which reuses the predicted steering representation as a target condition to guide progressive multi-scale spatial decoding, followed by SAM refinement for the final mask prediction. In this way, the same instance-specific steering signal couples internal representation adaptation with downstream spatial grounding. Experiments on ReasonSeg and the RefCOCO series show that SteerGrounding improves ReasonSeg Test-All gIoU by 11.6 points with only 7.3M additional trainable parameters and achieves consistent gains on RefCOCO, RefCOCO+, and RefCOCOg.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reasoning Segmentation, Large Multimodal Models, Representation Adaptation, Activation Steering, Visual Grounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WF2xOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper primarily investigates target localization and pixel-level mask prediction for reasoning segmentation with frozen large multimodal models. For reasoning queries, the model first generates a target description R, and then performs spatial grounding using Q+R as the textual context. The authors propose Instance-Conditioned Latent Steering (ICLS), which learns instance-specific steering from layer-wise representation differences between the original image and its target-erased counterpart, and injects the resulting steering into the hidden states of the frozen LMM. They further introduce a Steering-Conditioned Grounding Decoder (SCGD), which reuses the same representation to guide progressive multi-scale mask decoding, followed by SAM refinement. Experiments are conducted on ReasonSeg and the RefCOCO series.

Critical strengths

The paper presents an approach to adapting target-related representations for individual instances during conditional spatial grounding with frozen large multimodal models. ICLS learns instance-specific steering from layer-wise representation differences between the original image and its target-erased counterpart, while SCGD reuses this signal to guide multi-scale mask decoding. With the LMM backbone kept frozen, the authors report substantial improvements on ReasonSeg over their reproduced F-LMM baseline, and the ablation studies identify ICLS as the main contributor to these gains. This design establishes a concrete connection between internal representation adaptation in a frozen LMM and downstream spatial grounding.

Critical weaknesses

1.The scope of the paper’s contribution to reasoning segmentation is insufficiently clear. Section 3.1 indicates that the model first generates a target description R, after which ICLS and SCGD operate only during the grounding pass using Q+R as textual context. Thus, the proposed method primarily improves conditional spatial grounding given the generated description, rather than target-description generation or the reasoning process itself.

2.Several key methodological details are insufficiently specified, and the model architecture is overly complex. These include layer/head selection, the definitions of TT and Ω\Omega, text–visual representation alignment, SAM prompt construction, and the modules active during QA evaluation. Moreover, despite freezing the LMM backbone, the combination of U-Net, SCGD, and SAM refinement makes the architecture overly complex.

Questions for authors

1.The method primarily improves referring segmentation rather than the full reasoning segmentation process. Both ICLS and SCGD operate only during the subsequent grounding stage, after Q+R is available. Thus, the proposed modules optimize grounding rather than the preceding reasoning process that generates R.

2.The correctness of the target description R is not independently evaluated. Figure 4 provides qualitative examples, but whether ICLS and SCGD can recover from incorrect descriptions remains unclear. It would be valuable to include the following analyses: (1)The target-identification accuracy of the generated description R. (2)The impact of incorrect descriptions on the final mask. (3)A comparison between using Q alone and using Q+R.

3.The experimental comparison with recent related methods is insufficient. In Table 1, Qwen2.5-SAM-7B (Hao et al., 2026) is the only comparison method explicitly cited as a 2026 work, while most other methods are from 2023–2025. The authors should include additional relevant methods from 2026 to strengthen the experimental evidence.

4.The method does not consistently outperform LENS-7B. Table 1 shows higher gIoU only on ReasonSeg Test-Long and Test-All, but lower cIoU across all ReasonSeg splits. Section 4.2 highlights the gIoU gains without discussing the cIoU deficits. Please explain this discrepancy and provide a balanced assessment of the method’s strengths and limitations.

5.SAM refinement may confound the contributions of ICLS and SCGD. Tables 3–6 do not separately evaluate SAM refinement. It will be helpful to clarify how SCGD mask logits are converted into SAM prompts, report performance before and after refinement, and compare F-LMM and ICLS under the same SAM configuration.

6.The reference baseline for the 7.3M additional parameter claim should be made explicit. The abstract emphasizes that SteerGrounding introduces only 7.3M additional trainable parameters, but the 7.3M parameters are additional to the F-LMM baseline, whereas the complete model has 20.1M trainable parameters. The abstract should clearly distinguish additional from total trainable parameters, and the paper should also report the total model parameter count, including frozen modules.

7.The QA evaluation protocol requires clarification. Section 4.2 reports performance comparable to F-LMM, but does not specify which modules are active during QA evaluation or whether prompts and decoding settings match those of the original model. These details are needed to assess the claim of maintained multimodal capabilities.

Reviewer_j36DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This works try to solve the reasoning segmentation problem with instance-level selection. ICLS derives target-presence supervision from layer-wise representation differences between the original image and its target-erased counterpart. SCGD refines the predicted mask from attention map.

Critical strengths
  1. The paper is easy to follow and I am surprised with the flexibility of VLM's feature and attention map can be directly refine to be fed into SAM. This is an impressive design.
Critical weaknesses
  1. Ablation on backbone. The compared methods such as LENs implemented on LLAVA, Qwen2.5-SAM-7B on Qwen2.5, F-LLM original paper on LLAVA & LLAVA-Next. Author perform experiment on DeepSeek. Although authors reproduces F-LLM, However, I think authors should also test their methods on Qwen2.5/LLAVA. Although I encourage authors to test methods on various backbone, but classic backbone should also be tested.

  2. The major concern is that whether the SteerGrounding is purely an overfitted training paradigm. The steering groundtruth is derived by comparing the blurred image's feature and image's feature, where I can not understand how the reasoning capability of VLM is used? So, to validate the generalization of this training paradigm, can author perform a cross dataset evaluation? Such as splitting Reasoning Segmentation dataset by target object names into part A and B. And training on A then testing on B. This is a very reasonable setup because authors using the VLM, which has the general identification ability. Training on A has also provided a domain adaption.

Questions for authors

See weakness

Reviewer_zr7zOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SteerGrounding, an instance-adaptive representation steering method for reasoning segmentation with frozen LMMs. It introduces Instance-Conditioned Latent Steering (ICLS), which learns steering vectors from representation differences between original and target-erased images, and injects them into the frozen LMM. A Steering-Conditioned Grounding Decoder (SCGD) further uses these steering vectors for multi-scale mask decoding, followed by SAM refinement. Experiments on ReasonSeg and RefCOCO show improved segmentation performance, including an 11.6-point improvement in Test-All gIoU over F-LMM-7B.

Critical strengths

The paper is clearly written and well organized. The motivation and method are easy to follow, and the figures help explain the framework.

The method keeps the LMM backbone frozen and only introduces 7.3M additional trainable parameters, providing a parameter-efficient way to improve grounding performance.

The method achieves good improvements on ReasonSeg and RefCOCO benchmarks across both 1.3B and 7B models. The ablation studies also support the effectiveness of the proposed components.

Critical weaknesses

The target-presence supervision is not well justified. The method constructs target-erased images by replacing the target region with the image mean, which may introduce artificial visual changes. Therefore, the representation differences may not fully reflect target-related information. More analysis and comparisons with other erasing strategies are needed.

The methodological novelty is somewhat limited. The method mainly combines instance-specific activation steering with an attention-based decoder and SAM refinement, while similar ideas have been explored in previous works. Moreover, the paper lacks comparisons with parameter-matched adaptation methods, making it unclear whether the improvement comes from the proposed steering mechanism or simply the additional trainable parameters.

Questions for authors

na

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.