Training-Free Cross-Domain Few-Shot Segmentation via Preserve-Relative Counterfactual Refinement

Cross-domain few-shot segmentation aims to segment unseen classes in new domains using only a few annotated examples. In training-free settings, frozen semantic representations guide initial predictions, while promptable segmentation models generate structurally coherent candidate masks. However, under domain shift, structurally coherent masks can still misidentify target regions, undermining the semantic reliability of refinement. When the initial prediction also guides candidate selection, refinement can preserve or reinforce its semantic errors. We therefore propose a training-free framework that uses an audited semantic reference to improve the semantic reliability of mask refinement under domain shift. To this end, Cycle-Consistent Semantic Cover (CSC) uses semantic matching to select reliable prompts covering complementary target regions. Building on these prompts, Preserve-Relative Counterfactual Mask Selection (PRCMS) compares mask responses across prompt subsets and combines this evidence with support semantics to audit the reference and select refinements. Across eight datasets, our framework achieves mean mIoU of 69.33% in one-shot and 75.20% in five-shot segmentation, exceeding the evaluated state-of-the-art baselines. Ablation studies further show that complementary prompt coverage and reference auditing both improve cross-domain segmentation. Code will be released upon publication.