SteerGrounding: Instance-Adaptive Representation Steering for Reasoning Segmentation with Frozen LMMs

Reasoning segmentation requires models to infer an implicitly described target from an image–query pair and delineate it at the pixel level. Existing frozen-LMM grounding methods seek to extract spatial cues from pretrained multimodal representations and decode them into pixel-level predictions while largely preserving the general multimodal capabilities of the pretrained LMMs. However, representations produced by a frozen LMM may not always provide sufficiently reliable target-specific spatial evidence for each image–query pair, leaving downstream refinement to operate on incomplete or ambiguous cues without directly adapting the target-related hidden states. To address this problem, we propose SteerGrounding, which introduces instance-adaptive representation steering into a frozen LMM. Specifically, we introduce Instance-Conditioned Latent Steering (ICLS), which trains a lightweight steering predictor using layer-wise representation differences between the original image and its target-erased counterpart as target-presence supervision. At inference, the predictor directly estimates instance-specific steering from the original image–query pair and injects it into the frozen LMM hidden states for representation adaptation. Building on this, we introduce a Steering-Conditioned Grounding Decoder (SCGD), which reuses the predicted steering representation as a target condition to guide progressive multi-scale spatial decoding, followed by SAM refinement for the final mask prediction. In this way, the same instance-specific steering signal couples internal representation adaptation with downstream spatial grounding. Experiments on ReasonSeg and the RefCOCO series show that SteerGrounding improves ReasonSeg Test-All gIoU by 11.6 points with only 7.3M additional trainable parameters and achieves consistent gains on RefCOCO, RefCOCO+, and RefCOCOg.