- Rating
- 3
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
To address the issues where traditional knowledge distillation in remote sensing semantic segmentation may transfer domain-specific biases from the teacher and lead the student to learn unstable shortcuts from source-domain supervision, the authors propose CKD. This framework comprises two components: TCKT and SCKA. The former utilizes synthetically generated weather-perturbed images to learn frequency-filtering weights—refining teacher features through a combination of task supervision and domain adversarial training—while the latter applies frequency-based interventions to the student's input, compelling the student to align with the refined teacher representations despite variations in visual appearance. Both components are employed solely during training and do not alter the student's deployment architecture. In my view, while this work clearly models two sources of bias in robust distillation, the interpretation of "causal knowledge" and the specific contributions of the custom-designed modules still require more direct validation.
Critical strengths
- The analysis of the problem offers valuable insights; by considering how to balance the distortions introduced during teacher-to-student knowledge transfer against the shortcuts students take in independent learning, it explains why merely improving the teacher's knowledge may still be insufficient to produce students with robust understanding.
2.The authors not only conducted experiments involving eight types of degradation and various backbones but also performed direct transfer tests from Potsdam to Vaihingen to demonstrate the model's generalization capability.
Critical weaknesses
-
The paper does not provide explicit quality screening criteria, statistics on misalignment, or assessments of label consistency regarding the validity of pixel-level labels in the generated images.
-
The paper combines frequency-domain modulation, domain adversarial learning, input perturbation, and knowledge distillation; however, the experiments primarily compare different distillation losses without adequately comparing approaches such as simple learnable frequency-domain masks or fixed frequency-band filtering.
Questions for authors
The main issue with the author's work is that the effectiveness of the modules and the claimed causal relationships have not been demonstrated through ablation studies and primary experiments.
2 comments
Thank you for your response. I acknowledge the effectiveness of the proposed method. I am maintaining my current score because, while I consider this a solid piece of work, the novelty does not particularly impress me. Furthermore, the response to W1 fails to provide specific statistics regarding your process. Consequently, I have not raised my score, though I have increased my confidence rating. Good luck.
Response to Reviewer KP4k
Thank you for your insightful feedback and helpful suggestions.
W1. Label Consistency. To preserve pixel-level label consistency, we explicitly constrain the generation prompts to modify only image appearances while preserving the original semantic content and spatial structures. We further employ a two-stage screening procedure. First, we use the teacher model to evaluate generated images and discard samples with mIoU below 50% against the original labels. Second, we manually inspect the remaining images to remove samples exhibiting structural misalignment or semantic inconsistencies with the original pixel-level annotations.
W2. Frequency Filtering Comparisons. We further conduct ablations by replacing TCKT with fixed frequency-band filtering or simple learnable frequency masks, both without domain-adversarial learning, while retaining SCKA. As shown in the table, our method consistently outperforms both alternatives across all four distillation baselines. These results suggest that the improvements stem not merely from frequency filtering, but also from suppressing non-causal, domain-specific information through adversarial learning.
Q1. Component Effectiveness and Causal Motivation. To address this concern, in addition to the frequency-filtering ablations discussed above, we conduct further experiments, including an undistilled student trained with SCKA and four distillation baselines (KD, CWD, ViTKD, and CustomKD) trained with the same Qwen-generated perturbation images as CKD (+Data Aug.):
We further extend the cross-dataset evaluation by training on Potsdam and testing on Vaihingen to assess CKD's generalization performance:
Together with the frequency-filtering ablations, these results support the effectiveness of CKD and its proposed modules, indicating that the improvements are not solely attributable to data augmentation. Instead, TCKT facilitates the transfer of causal knowledge from the teacher, while SCKA promotes its acquisition by the student, jointly enabling robust knowledge distillation under domain shifts.