CFRNet: Counterfactual Frequency-Guided Selective Retention for Semantic Segmentation

High-frequency information is important for object boundaries and structures in semantic segmentation, but can also arise from background artifacts and object-like patterns, making preservation or attenuation unreliable. We propose CFRNet, a counterfactual frequency-guided framework that adaptively modulates high-frequency retention by combining probe-output differences from controlled removal and perturbation with multi-scale structural support. The resulting spatial and channel-wise retention signals selectively reduce or preserve reliable high-frequency information. On PASCAL VOC 2012, COCO-Stuff 164K, and ADE20K, CFRNet achieves mIoUs of 0.701, 0.468, and 0.390, respectively. CFRNet outperforms the strongest counterparts by 0.058 and 0.016 mIoU on COCO-Stuff and ADE20K, while remaining within 0.002 mIoU of the highest VOC result. Mathematical analysis and ablations further examine the component contributions and the conditional retention directions induced by the CFR objective.