acceptodds
Under review as a conference paper at ICLR 2027

CFRNet: Counterfactual Frequency-Guided Selective Retention for Semantic Segmentation

Abstract

High-frequency information is important for object boundaries and structures in semantic segmentation, but can also arise from background artifacts and object-like patterns, making preservation or attenuation unreliable. We propose CFRNet, a counterfactual frequency-guided framework that adaptively modulates high-frequency retention by combining probe-output differences from controlled removal and perturbation with multi-scale structural support. The resulting spatial and channel-wise retention signals selectively reduce or preserve reliable high-frequency information. On PASCAL VOC 2012, COCO-Stuff 164K, and ADE20K, CFRNet achieves mIoUs of 0.701, 0.468, and 0.390, respectively. CFRNet outperforms the strongest counterparts by 0.058 and 0.016 mIoU on COCO-Stuff and ADE20K, while remaining within 0.002 mIoU of the highest VOC result. Mathematical analysis and ablations further examine the component contributions and the conditional retention directions induced by the CFR objective.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Semantic Segmentation, Frequency-Aware Learning, Counterfactual Frequency Guidance, High-Frequency Retention, Structural Context, Feature Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tkhJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CFRNet. It predicts spatial‑ and channel‑wise high‑frequency retention scores via high‑frequency feature removal and perturbation, output differences from a shared probe, as well as multi‑scale structural information. A CFR loss is further employed to regularize the retention behaviour.

Critical strengths
  1. This paper addresses a practically meaningful problem: high‑frequency signals emerge both from genuine object boundaries and spurious distracting textures.

  2. The authors report experimental results on three datasets, including component‑level ablations, boundary‑oriented metrics, and control experiments for structural context.

  3. The performance improvements achieved on COCO‑Stuff and ADE20K are noteworthy.

Critical weaknesses
  1. The title may be misleading. The term “counterfactual” is repeatedly used in the title, abstract and module names. Nevertheless, the method only relies on two manually‑designed feature perturbations and does not perform counterfactual causal identification. The paper should further clarify the concrete meaning of this term in the present work and justify why these perturbations underpin the proposed high‑frequency retention mechanism.

  2. The introduction fails to sufficiently justify the necessity and distinct contribution of the method. Although the paper points out that high‑frequency magnitude cannot separate beneficial boundary signals from distracting responses, it does not explain why probe‑output differences and structural support can resolve this problem. Its analysis of limitations of existing frequency‑aware segmentation methods is also rather general. Therefore, I have concerns regarding the necessity and contributions of the proposed method.

  3. There are also concerns about methodological novelty. While integrating frequency decomposition, feature perturbations, probe‑based responses, structural context and learnable gating into one unified framework has potential value, the design reads more like a pipeline assembly rather than a methodological innovation. Moreover, the theoretical analysis only derives local update directions induced by the loss function, without providing distinctive algorithmic guarantees. Hence, the incremental contribution beyond assembling existing components is insufficiently demonstrated.

  4. Critical ablation experiments are missing, which makes it hard to pinpoint the unique contribution of the method. Current ablation results cannot rule out the possibility that performance gains mainly stem from the additional learnable gating and supervision, instead of the proposed intervention‑and‑probe mechanism.

  5. The stability of key hyperparameters has not been examined. The intervention strength, retention-score floors, and CFR loss weights may substantially affect the method’s behavior, yet the paper provides no sensitivity analysis. It is therefore unclear whether the reported gains persist across a reasonable range of settings or depend heavily on the particular values used.

Questions for authors
  1. The paper clarifies that “counterfactual” refers to controlled feature‑level interventions instead of causal identification. Could the authors explain why removal and perturbation operations are suitable to be described with this term?

  2. Could the authors clearly distinguish between the theoretical contributions and methodological contributions of this work and elaborate on its core novelty?

  3. What performance will be obtained if we replace the probe‑output difference map with the raw high‑frequency magnitude map, keeping all other network structures and training settings unchanged? How does it compare against conventional spatial / channel‑wise gating with matched parameter capacity? Such controls help disentangle the standalone contribution of the intervention‑and‑probe design.

  4. What is the relationship between the probe’s perturbation‑driven responses and the actual utility of high‑frequency features?

  5. Could the authors conduct sensitivity‑analysis experiments for key hyper‑parameters?

Reviewer_4gn5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CFRNet is a semantic segmentation framework that uses counterfactual frequency intervention to determine which high-frequency features to retain. Its premise is that strong high-frequency responses can originate from multiple sources: meaningful boundaries and fine structures, but also from background textures, artifacts, and regions associated with incorrect predictions. Therefore, response intensity alone is insufficient as a basis for selecting effective features.

Critical strengths
  1. This design directly addresses the ambiguity of high-frequency responses. This paper makes a useful distinction between the strength of the frequency response and its value for semantic prediction. By measuring the effects of controlled interventions and combining this with structural evidence, a clear theoretical basis is provided for the preservation mechanism. This connection makes the method highly motivating and provides a concrete approach to studying when high-frequency information contributes to or interferes with segmentation.

  2. Experiments validate the overall performance and the interpretation of the performance improvements. Results on three benchmark datasets provide extensive empirical support, particularly strong performance on the COCO-Stuff and ADE20K datasets in a shared ResNet-50 setting. Component ablation and mechanism analysis help explain the framework's operation. Comparisons using only structural context are particularly important as they examine whether the additional Transformer derivative branches can independently explain the performance improvements.

Critical weaknesses
  1. The benefits of preserving the objective function are partially intertwined with those of boundary supervision. The CFR loss explicitly treats correctly predicted semantic boundaries as a separate group and sets a higher lower bound for preservation at these locations. In Table 5, introducing this loss improves mIoU by 0.025, from 0.443 to 0.468, making it a significant contributor to the final result. However, there are currently no comparative studies replacing it with a traditional boundary-aware auxiliary loss while keeping the CFRNet architecture unchanged. Therefore, existing ablation experiments confirm the value of adding the full objective function but do not determine which part of that value comes from its specific preservation formula. A matched boundary supervision baseline would provide stronger evidence for the claimed contribution.

  2. The dependence on the selected intervention has not been adequately examined. The counterfactual signal depends on the complete removal of high-frequency components and a specific perturbation based on spatial and channel rollover with a fixed intensity (α). Removing any intervention in the ablation experiments showed that both were effective in the tested configuration, but this does not demonstrate its stability across a variety of possible alternatives. Since these response differences determine the retained signal, variations in perturbation strength or rolling configuration could affect the final feature selection results. Sensitivity experiments, along with at least one alternative intervention, will help clarify whether the mechanism can persist effectively beyond the reported settings.

Questions for authors
  1. Can the authors replace CFR with a traditional boundary-aware auxiliary loss function while keeping the model architecture, training plan, and tuning budget unchanged? Reporting mIoU and boundary quality metrics would help determine if the preservation strategy offers any benefits beyond additional boundary supervision.

  2. How much would the final segmentation score and preservation pattern change within a reasonable range of α values ​​and spatial/channel rolling selection? Furthermore, understanding whether other perturbations produce the same qualitative relationship between intervention response and feature usefulness is also meaningful.

Reviewer_ib8kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CFRNet, a semantic-segmentation framework that uses counterfactual frequency interventions to decide which high-frequency feature information should be retained. It separates low- and high-frequency components, probes the effect of removing or perturbing high-frequency information, and combines the resulting response with structural support to guide spatial- and channel-wise retention. Besides,a CFR loss is introduced to discourage retention at some error locations,while preserving information considered useful around correct predictions and boundaries. Experiments are reported on PASCAL VOC, COCO-Stuff, and ADE20K, with ablations intended to evaluate the proposed components.

Critical strengths
  1. The paper considers a meaningful problem: high-frequency features can not only encode boundaries and fine structures, but also carry distracting texture. As a result, selective retention is a plausible design objective for semantic segmentation.
  2. The evaluation spans several segmentation benchmarks and includes component ablations. This is a reasonable starting point for testing whether the method has value beyond a single dataset, although the current evidence does not fully support the broader claims.
Critical weaknesses
  1. The paper treats prediction variations induced by removing or perturbing high-frequency information as evidence that features carry semantic value. However, sensitivity does not equate to validity. A feature may significantly alter the detector’s output yet be irrelevant to the downstream task, redundant, or even detrimental to final predictions. The paper needs stronger validation to demonstrate that counterfactual responses can indeed distinguish beneficial boundary or fine-grained details from irrelevant textures, rather than merely measuring the model’s sensitivity to perturbations.
  2. The theoretical discussion appears to motivate a local update direction, but does not establish that the resulting optimization improves final segmentation quality. In addition, the contribution of the CFR loss is not sufficiently separated from the effect of adding another form of supervision or regularization. The benchmark results and ablations therefore do not yet provide a convincing causal account of why the proposed mechanism works. Comparisons would also benefit from clearer reporting of protocol consistency and computational overhead.
Questions for authors
  1. Figure 4 uses broken axes to present the performance–complexity trade-off, but the omitted intervals and the intended interpretation of the discontinuities are not explicitly specified. Could the authors clarify the exact ranges omitted from each axis and explain how the visual distances between methods should be interpreted? In particular, how can readers assess the relative differences between methods without being misled by the discontinuous axis scaling?
  2. The paper uses counterfactual intervention responses to guide high-frequency feature retention. However, a large prediction change after removing high-frequency information does not necessarily imply that the removed information is semantically useful. What evidence demonstrates that the intervention response distinguishes beneficial semantic details from irrelevant texture or noise? Could the authors provide a controlled analysis to validate this assumption?
  3. CFRNet uses intervention-induced prediction changes, together with structural support and prediction correctness, to guide high-frequency retention. Yet these signals are themselves derived from the segmentation model and its predictions, rather than from an independent measure of the semantic value of individual features. This creates a potential circularity: the model determines which features appear influential, and those same estimates are then used to train the model to retain them. What prevents this procedure from reinforcing existing prediction biases or preserving features that are influential but semantically misleading? More fundamentally, under what assumptions can the proposed signals serve as a reliable proxy for feature-level semantic utility?
  4. The theoretical analysis characterizes local changes in the retention coefficients under the CFR objective, but the method's central claim concerns the semantic usefulness of the retained features and the resulting segmentation quality. These are substantially stronger properties than a local directional update. In a non-convex network, local coefficient behavior does not by itself guarantee convergence to a solution with improved semantic discrimination, nor does it rule out undesirable interactions between frequency components and downstream representations. What theoretical connection, if any, links the established local properties to the claimed end-task improvement? If no such connection can be established, the theoretical results appear to justify only the optimization behavior of the proposed mechanism, not its semantic interpretation or performance guarantees.
  5. Table 5 reports an mIoU increase from 0.443 without the CFR objective to 0.468 with the full objective. However, this comparison alone does not establish whether the improvement specifically results from the conditional retention constraints or from more general effects such as additional regularization and optimization pressure. Moreover, the reported results do not by themselves establish the robustness of this gain across independent runs. Can the authors isolate the contribution of the CFR objective from these alternative explanations and establish that the improvement is reproducible?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.