Not All Evidence Is Equal: Asymmetric Referential Evidence Modeling for Referring Remote Sensing Image Segmentation

Referring Remote Sensing Image Segmentation (RRSIS) aims to identify and segment the target specified by a natural-language expression in a remote-sensing image. A central challenge is to distinguish the intended target from multiple visually and semantically plausible candidates. Existing methods primarily enrich discriminative information, but often overlook how evidence with different functions should influence referent identification. We argue that different forms of referential evidence should be assigned authority according to their functional roles. Based on this principle, we propose Asymmetric Referential Evidence Modeling (\arem), which forms role-specific referential evidence, constructs positive-anchored dense grounding with bounded role-aware correction, and calibrates prompt decoding through discriminative training and controlled refinement. Extensive experiments on RefSegRS and RRSIS-D demonstrate the effectiveness and efficiency of \arem, while cross-architecture and cross-domain evaluations further support its applicability beyond the primary setting.