acceptodds
Under review as a conference paper at ICLR 2027

RL-SAF : Reinforcement Learning Based Semantic Adversarial Fairness

Abstract

Adversarially trained models can achieve strong overall robustness while exhibiting substantial class-wise performance disparities under attack. Although this adversarial fairness problem has been studied in image classification, it remains largely unexplored in semantic segmentation. We show that such disparities occur across datasets, architectures, and adversarial training strategies, that adversarial perturbations amplify rather than uniformly degrade class-wise performance, and that class frequency alone does not explain which classes are most vulnerable. We further show that adapting a state-of-the-art adversarial fairness method from image classification to semantic segmentation reduces both worst-class and overall robustness, which we attribute to a mismatch between recall-based notions of class difficulty and the IoU objective of dense prediction. Motivated by these findings, we propose RL-SAF (Reinforcement Learning-Based Semantic Adversarial Fairness), which combines class-level loss aggregation with a policy-gradient controller that adaptively learns class-wise loss weights from validation feedback. On Cityscapes with DeepLabV3, RL-SAF improves the adversarial IoU of the most vulnerable classes across all evaluated attacks and preserves robust mIoU under the attack used for training, while trading off part of it under stronger attacks. These gains come at a substantial cost in clean accuracy, revealing a clear trade-off between adversarial fairness and standard segmentation accuracy. Overall, our results establish class-wise adversarial robustness disparities as an important challenge in semantic segmentation and provide initial evidence to motivate adaptive, fairness-aware adversarial training for dense prediction tasks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial Training, Adversarial Fairness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_78o7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes RL-SAF (Reinforcement Learning-Based Semantic Adversarial Fairness), a method designed to improve both worst-class robustness and overall robustness in semantic segmentation. RL-SAF combines class-level loss aggregation with a policy-gradient controller that adaptively learns class-wise weights from validation feedback.

Critical strengths

The paper provides a thorough analysis of adversarial fairness in semantic segmentation. In particular, it explains why fairness techniques developed for image classification do not transfer directly to segmentation, and it clearly defines the problem it aims to solve. This analysis is the main basis for my positive assessment, because it establishes a well-motivated problem setting that is likely to be useful for future work on robust and fair segmentation, independent of the specific method proposed.

Critical weaknesses

The evaluation of RL-SAF is insufficient. In Table 2 the authors state that the SegPGD-3 mIoU is maintained (26.78% for SegPGD-AT to 27.05% for RL-SAF). However, the results appear to come from a single run, with no standard deviations or confidence intervals reported. A difference of 0.27%p without these looks so small to be distinguished from run-to-run variance, so the claim is not adequately supported. In addition, the paper does not explain why adaptive class weighting improves the worst-class metric at the cost of overall mIoU under stronger attacks. It should be made clear what degrades in exchange. This weakness bears directly on my assessment, since the central claim of the paper is that both objectives improve simultaneously.

Questions for authors
  1. could you report the mean and standard deviation (or confidence intervals) over multiple random seeds?
  2. When the worst-class IoU improves, which classes lose IoU? A per-class breakdown would clarify the mechanism behind the trade-off.
  3. Could you explain what causes the trade-off? And how can it be avoided or mitigated?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 78o7

Dear Reviewer 78o7,

We sincerely thank the reviewer for their thoughtful and constructive evaluation of our work. We particularly appreciate their recognition of our thorough analysis of adversarial fairness in semantic segmentation, the clear motivation of the problem, and our investigation into why existing classification-based fairness methods do not directly transfer to segmentation. We are also grateful for the positive assessment of the paper's soundness and its potential to motivate future research on robust and fair semantic segmentation.

W1, Q1, Q2: We report the mean and standard deviation over multiple random seeds in General Response R1. We further report evaluations under varying attack strengths in our response to Reviewer 2cqq (W6), showing the effectiveness of our approach in more attack scenarios, where the gains of our approach are larger.

We are happy to discuss further.

We also report the effect of RL-SAF on the five least- and five most-robust classes of SegPGD-AT (SegPGD-10, ε = 4/255) in Figure 12 of our revision:

  • (a) All five least-robust classes improve (+3.49% on average), while the five most-robust classes lose 1.78% on average.
  • (b) Worst-class IoU improves 4.6× and the bottom-5 mean improves 4.8×, at a cost of only 0.19% in overall mIoU.

Q3: The trade-off comes from reallocating a fixed training signal. RL-SAF uses a class-balanced loss: it upweights pixels from vulnerable classes and downweights the abundant, already-robust classes (road, building, vegetation, sky). The per-class breakdown above shows the result: all five least-robust classes gain, while the most-robust classes give up a small amount. We further mitigate this trade-off through improved batch-norm settings (see General Response R2).

We are happy to discuss further.

I thank the authors for their detailed response. I will maintain my original score.

Reviewer_kYGEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies class-wise adversarial robustness in semantic segmentation.

It finds that attacks affect some classes much more severely than others, and that class frequency alone does not explain these differences.

The authors propose RL-SAF, which combines class-level loss aggregation with an RL controller that adjusts class weights using validation feedback.

Experiments show improvements in worst-class and bottom-k IoU, with trade-offs in clean accuracy and overall robustness under stronger attacks.

Critical strengths
  1. The paper makes a useful case for looking beyond average mIoU when evaluating adversarial robustness in semantic segmentation. The analysis covers different training settings, showing that some classes remain highly vulnerable even when overall robustness improves. I find the observation that class frequency alone does not explain these differences particularly interesting and worth further investigation.

  2. The proposed method follows naturally from these observations, giving the paper a coherent progression from identifying the problem to exploring a possible solution. I also appreciate that the authors openly discuss the loss in clean accuracy and the trade-offs under stronger attacks.

Critical weaknesses
  1. RL-SAF is only evaluated on DeepLabV3 with Cityscapes, so it is unclear whether the learned policy can transfer to elsewhere or needs to be learned again for each setting. I think this matters especially here because obtaining each reward requires further training and evaluation. This is much more expensive than simply checking a generated answer, as in some RLVR tasks. If the policy could be reused across models or datasets, the cost of learning it could be spread across multiple training tasks. That would give a stronger reason to choose RL over simpler weighting methods.

  2. The benefit of the RL controller is not sufficiently established. It combines three predefined signals, so comparisons with fixed mixture coefficients, simple IoU-based weighting, and grid search or Bayesian optimization under comparable compute budgets would be useful. Fixed mixture coefficients would still allow class weights to change with the model’s current performance, making this a particularly relevant baseline.

Questions for authors
  1. what about comparing the RL controller with simpler search methods, such as grid search or Bayesian optimization over the signal mixture coefficients, under comparable compute budgets.
  2. Can a policy learned on one model or dataset be reused on another without retraining? Even a small transfer experiment would help clarify whether the cost of learning the policy can be shared across tasks.
2 comments
AuthorsOpenReview ↗

Response to Reviewer kYGE

Dear Reviewer kYGE,

We sincerely thank the reviewer for their thoughtful and constructive feedback and for recognizing the importance of analyzing class-wise adversarial robustness beyond overall mIoU. We particularly appreciate the positive assessment of our paper's soundness, clear motivation, and coherent progression from identifying robustness disparities to proposing RL-SAF, as well as the reviewer's valuable suggestions for strengthening the evaluation of our RL controller and its generalizability.

W1, Q2: We acknowledge that RL-SAF introduces additional training overhead. Importantly, RL-SAF is an online method: the policy is learned during a single adversarial-training run, with rewards derived from training and evaluation steps already performed. Thus, no separate outer loop of full retraining runs is required, and the policy is learned as a by-product of training the segmentation model. In practice, on PASCAL VOC 2012, RL-SAF adds 31.8 s wall-clock time per epoch over SegPGD-AT (125 s vs. 93.22 s), measured on RTX A6000 GPUs with batch size 16.

We also ran the transfer experiment suggested by the reviewer. We initialise the policy from the one learned on Cityscapes and fine-tune it on PASCAL VOC 2012. These datasets have a different number of classes, so the policy cannot simply be copied over unchanged. This transfer experiment is also not feasible with the baseline approach due to different class statistics.

W2, Q1: We thank the reviewer for this suggestion. To isolate the contribution of the RL controller, we compare RL-SAF against four alternatives: direct IoU-based weighting, a fixed equal mixture of the three signals, and two frozen policies that use the average and the final coefficients learned by RL-SAF. Fixed coefficients still produce class weights that change during training, because the underlying signals change. Unlike RL-SAF, however, they cannot adjust how much each signal matters based on validation feedback. We did not run grid search or Bayesian optimization. In our setting, every configuration needs a full adversarial training run plus adversarial evaluation, which makes these searches very expensive. Our ablations instead test the main question directly: does a learned, dynamic policy help beyond static fixed coefficients?

Table: Class-weighting ablation, DeepLabV3 on Cityscapes, 3 seeds (mean ± std); robustness under 3-step SegPGD with ε = 4/255. Table 13 in our revision.

Class-weighting strategyClean mIoURobust mIoUWorst IoUBottom-5 IoU
SegPGD-AT60.38 ± 0.3430.96 ± 0.740.34 ± 0.292.28 ± 0.53
Direct IoU-based weights59.34 ± 0.3629.83 ± 0.601.00 ± 0.505.20 ± 0.32
Fixed equal 3-signal mixture59.26 ± 0.4329.56 ± 0.491.13 ± 0.265.18 ± 0.36
Fixed mixture: frozen average policy59.00 ± 0.2329.73 ± 0.591.27 ± 0.485.38 ± 0.27
Fixed mixture: frozen final policy59.14 ± 0.3929.63 ± 0.541.00 ± 0.345.21 ± 0.32
RL-SAF (learned mixture)59.39 ± 0.5131.11 ± 2.121.74 ± 0.956.14 ± 1.25

Every static weighting scheme improves worst-class and bottom-5 IoU over SegPGD-AT, but each lowers robust mIoU by about 1–1.4%. RL-SAF is the only variant that improves the weakest classes without this drop: it has the best worst-class IoU, bottom-5 IoU and robust mIoU. This includes the frozen versions of RL-SAF's own coefficients. So the benefit comes from adapting the mixture during training, not from finding a good fixed set of coefficients.

We are happy to discuss further.

Thanks for the authors' detailed response. It addresses my concern. I will hold my score.

Reviewer_2cqqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies class-wise adversarial robustness disparities in semantic segmentation. Through experiments across datasets, attacks, and adversarial training settings, the authors report that adversarial perturbations disproportionately harm vulnerable classes and that class frequency alone does not explain the observed disparities. To address this, they propose RL-SAF, which uses a reinforcement-learning controller to adapt class-wise loss weights based on validation feedback. On Cityscapes with DeepLabV3, RL-SAF improves the worst-class and tail-class robust IoU under the evaluated attacks

Critical strengths
  1. Clear and well-structured presentation. The paper is organized around a coherent progression: it first identifies class-wise robustness disparities in semantic segmentation, then analyzes why a classification-based fairness method may not transfer, and finally introduces RL-SAF to address the observed limitations. This structure makes the motivation and method easy to follow.
  2. A useful fine-grained analysis of robustness. In addition to reporting aggregate mIoU, the paper examines per-class IoU and prediction behavior. This analysis helps reveal vulnerabilities that overall metrics can obscure and provides a useful perspective for evaluating adversarial robustness in dense prediction.
Critical weaknesses
  1. Figure 2 provides correlational evidence that class frequency alone does not predict class-wise IoU, but it does not establish whether training-set imbalance causes or contributes to the observed robustness disparities. All results are obtained from models trained on the original imbalanced datasets, so the analysis cannot separate the effect of class frequency from other class-specific factors. A controlled ablation using class-balanced sampling or augmentation, with the same model, training budget, and evaluation set, would help determine whether reducing training imbalance changes the class-wise robust IoU gaps and frequency-normalized prediction bias.
  2. The authors take the class-wise adversarial robustness disparities in semantic segmentation assessment as the first contribution. However, previous works [1,2] have reported similar findings.
  3. Missing semantic-segmentation baselines. The evaluation compares RL-SAF with only one fairness method adapted from image classification, DAFA. It does not compare against existing adversarial training approaches for semantic segmentation, such as SegPGD-AT [3], or robust segmentation methods like PIR-AT [4], using class-wise robustness metrics. These methods are not specifically designed to improve fairness, but they would provide relevant task-specific baselines for assessing whether RL-SAF reduces inter-class robustness disparities beyond standard segmentation defenses.
  4. Limited validation of RL-SAF on datasets. RL-SAF is evaluated only on Cityscapes with DeepLabV3, so it remains unclear whether the method generalizes across datasets, segmentation architectures, or class distributions. Although the paper’s preliminary analysis includes PASCAL VOC 2012, it does not evaluate RL-SAF on that dataset. The authors should evaluate RL-SAF on additional benchmarks, such as PASCAL VOC 2012 [5], ADE20K [6], and BDD100K [7] or Mapillary Vistas [8].
  5. Substantial trade-offs in clean and overall robust performance. The paper reports a marked reduction in clean performance, and under stronger attacks, RL-SAF improves tail-class robustness at the cost of lower overall robust mIoU. The paper does not sufficiently characterize this trade-off or show how practitioners should choose an operating point between worst-class robustness, overall robustness, and clean accuracy.
  6. Low absolute robustness for the worst-performing classes. Under the SegPGD-3 training attack, RL-SAF increases the worst-class IoU only from approximately 0.18% to 0.58%, and the worst-five-class IoU from approximately 0.83% to 3.40%. While these results represent relative improvements, the absolute IoU values remain very low, suggesting that the worst-performing classes are still nearly unusable under attack.
  7. Limited validation of RL-SAF on architectures. Only DeepLabV3 is included in the final evaluation. The authors should include more architectures, such as PSPNet [9] and SegFormer [10].

[1].Mahima, KT Yasas, et al. "Exploring adversarial robustness of LiDAR semantic segmentation in autonomous driving." Sensors 23.23 (2023): 9579.

[2]. Halmosi, Levente, Bálint Mohos, and Márk Jelasity. "Evaluating the adversarial robustness of semantic segmentation: Trying harder pays off." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[3]. Gu, Jindong, et al. "Segpgd: An effective and efficient adversarial attack for evaluating and boosting segmentation robustness." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[4]. Croce, Francesco, Naman D. Singh, and Matthias Hein. "Towards reliable evaluation and fast training of robust semantic segmentation models." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[5]. Everingham, Mark, et al. "The pascal visual object classes challenge 2012 (voc2012)." Results 2.5 (2012): 7.

[6]. Scene Parsing through ADE20K Dataset. Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso and Antonio Torralba. Computer Vision and Pattern Recognition (CVPR), 2017.

[7]. Yu, Fisher, et al. "Bdd100k: A diverse driving dataset for heterogeneous multitask learning." 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, 2020.

[8]. Neuhold, Gerhard, et al. "The mapillary vistas dataset for semantic understanding of street scenes." Proceedings of the IEEE international conference on computer vision. 2017.

[9]. Zhao, Hengshuang, et al. "Pyramid scene parsing network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

[10]. Xie, Enze, et al. "SegFormer: Simple and efficient design for semantic segmentation with transformers." Advances in neural information processing systems 34 (2021): 12077-12090.

Questions for authors

Please see the weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2cqq

Dear Reviewer 2cqq,

We thank the reviewer for the insightful and constructive feedback. We appreciate the reviewer's positive assessment of the paper's clarity and organization, as well as their recognition of the value of our fine-grained analysis of class-wise adversarial robustness in semantic segmentation. we hope to answer the feedback in following manner.

W1: We agree that the correlation in Fig. 2 alone cannot show whether class imbalance contributes to robustness disparities. To test the role of imbalance directly, we ran an ablation comparing standard AT with class-balanced pixel-loss sampling, which uses an equal number of pixels from each class present in a batch. We compare clean and robust mIoU, worst-class and bottom-5 robust IoU, and class-wise IoU. Model, training budget and evaluation set are the same (DeepLabV3, Cityscapes, 3-step SegPGD, ε = 8/255). These results are presented in Table 15 in our revision.

AT lossClean mIoURobust mIoUWorstBottom-5IoU spreadFreq.–rob. corr.Pixels used (%)
All-pixel60.3827.150.091.1485.690.750100
Class-balanced52.2725.080.113.3378.600.8000.65

Balancing improves bottom-5 robust IoU by about 3× and narrows the IoU spread by 7.1%. However, the worst class stays near zero, clean and robust mIoU drop, and the frequency–robustness correlation does not decrease. Removing training imbalance therefore reduces robustness gaps only partly, which confirms that class-specific factors beyond frequency drive the class-wise disparities.

W2: We acknowledge that class-wise adversarial robustness disparities have been previously reported, but prior works characterize them without addressing them through adversarial fairness optimization. Halmosi et al. [2], already cited in our original manuscript, highlight object-size-dependent vulnerabilities and the limitations of overall robustness metrics in 2D semantic segmentation. Mahima et al. [1] identify class-wise vulnerabilities in 3D LiDAR segmentation, particularly the high susceptibility of ground-level points to perturbation attacks. Our contribution differs by examining these disparities through the lens of adversarial fairness optimization. We have revised "class-wise robustness disparities are unexplored" to "underexplored" to better acknowledge prior work and expanded the discussion of these works in the related work section.

W3, W4, W7: SegPGD-AT [3] is already our main baseline. RL-SAF fine-tunes a converged SegPGD-AT checkpoint, and Table 2 in main paper compares the two on worst-class, bottom-3 and bottom-5 robust IoU under SegPGD-3, SegPGD-10 and PGD-10. We further extend RL-SAF to SegFormer (another architecture) and PASCAL VOC 2012 (another semantic segmentation dataset) over multiple seeds to show the generalizability of our approach to new architectures and datasets; see General Response R1 for evaluations. ADE20K training is in progress: we will add the results to the revision during the discussion period if they finish in time, and otherwise in the camera-ready version.

W5: See General Response R2.

W6: We acknowledge that absolute worst-class robustness remains low under strong adversarial attacks. However, these attacks represent strong academic worst-case evaluations and do not necessarily reflect the perturbation strengths encountered in practical scenarios. To better characterize the benefits of RL-SAF, we extend our evaluation to weaker attacks with perturbation budgets ε ∈ {2, 4, 8}/255, using the same trained checkpoints, and report these results alongside overall robust mIoU to demonstrate the practical relevance of RL-SAF while acknowledging its limitations under strong attacks. For weaker attacks, RL-SAF performs substantially better on vulnerable classes while also achieving improved robust mIoU compared to the baseline.

Table: Evaluation-only attack-strength sweep, 3-step SegPGD, DeepLabV3 on Cityscapes, 3 seeds (mean ± std). Table 12 in our revision.

MethodEval. εClean mIoURobust mIoUWorst-class robust IoUBottom-5 robust IoUWorst-class name
SegPGD-AT2/25560.39 ± 0.3536.02 ± 0.971.44 ± 0.885.81 ± 1.26train
RL-SAF2/25559.39 ± 0.5136.58 ± 2.275.38 ± 2.899.72 ± 1.71train
SegPGD-AT4/25560.39 ± 0.3530.96 ± 0.740.34 ± 0.292.28 ± 0.53wall
RL-SAF4/25559.39 ± 0.5131.11 ± 2.121.74 ± 0.956.14 ± 1.25train
SegPGD-AT8/25560.38 ± 0.3427.15 ± 0.660.09 ± 0.091.14 ± 0.27train
RL-SAF8/25559.39 ± 0.5125.50 ± 0.400.64 ± 0.303.41 ± 0.11train

We are happy to discuss further.

Reviewer_2cqqOpenReview ↗

Some remaining concerns

Thank you for the additional experiments and clarifications. The new results address several concerns, particularly the comparison with SegPGD-AT, the evaluation on PASCAL VOC 2012 and SegFormer, and the analysis of the robustness–fairness trade-off. However, there are still some concerns that remain unaddressed.

  1. Causal role of class imbalance. This concern remains unresolved. Could the authors compare training with the original class distribution against class-balanced sampling or augmentation, while keeping the model, training budget, and evaluation set fixed? Reporting the resulting class-wise robust IoU gaps and frequency-normalized prediction bias would help determine whether training-set imbalance contributes to the observed disparities. If this experiment is not feasible, the paper should clarify that the current analysis does not establish whether training imbalance contributes to these gaps.

  2. Novelty of the class-wise robustness analysis. The response does not clarify how the paper’s findings about class-wise adversarial robustness disparities differ from those reported in the prior semantic-segmentation studies cited in my review [1, 2]. The authors should more clearly identify what is novel in their analysis beyond documenting these disparities.

  3. Semantic-segmentation baselines. Adding SegPGD-AT is a useful improvement and provides a relevant task-specific comparison. However, the evaluation still lacks other robust semantic-segmentation methods, including PIR-AT as mentioned in my original review. The additional controller ablations reported in response to another reviewer help assess the RL component, but they do not replace comparisons with existing segmentation defenses.

  4. Absolute robustness of the worst-performing classes. The revised results improve worst-class and bottom-5 IoU, but the worst-class IoU on Cityscapes remains very low under the stronger ϵ=8/255\epsilon=8/255 attack. In the three-seed results, it is 0.64 ± 0.30 under SegPGD-3 and 0.11 ± 0.08 under SegPGD-10. The additional per-class analysis under a different attack setting is helpful, but it does not establish that the weakest classes achieve practically useful robustness under these stronger attacks.

The new experiments improve the empirical evaluation and partially address several concerns. However, the questions about the causal role of class imbalance, the novelty of the class-wise analysis, and the practical robustness of the worst-performing classes remain. Since these are all concerns from my original review and there are no new ones, I maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer 2cqq (Part - 1/2)

Thank you for acknowledging that we resolved most of the concerns through the rebuttal.

1. We thank the reviewer for following up on this point. We would like to clarify our claim: we do not claim that class imbalance has no causal effect on class-wise robustness disparities, but that it is not sufficient to explain them.

In order to allow experiments to be as conclusive as possible, we have reported two settings already:

(1) In our original submission, we have compared training with the original class distribution against class-balanced loss-weighting, while keeping the model, training budget, and evaluation set fixed.

(2) In our first response, we have compared against class-balanced sampling, while keeping the model, training budget (images sampled from and epochs), and evaluation set fixed.

In both cases, we observe that class imbalance indeed contributes to the observed disparities but can not fully explain it (a gap of more than 16% in class-wise IoU persists when sampling in a class-balanced way). This is in line with disparities observed in adversarially trained image segmentation models, where the training budget is balanced across classes by default.

To make experiment (2) more conclusive, we add here a baseline where we match the class-balanced sampling budget in terms of pixels (not images) by sampling randomly during the adversarial training. As expected, the overall mIoU are similar and class disparities are more severe (worst class robust IoU is 1.77 on average compared to the worst class IoU of 3.32 for class balanced sampling. From this experiment, class-balanced pixel sampling can reduce the disparity to the average by ~4.4% IoU, while the remaining disparity is 16.05).

These results indicate that severe class-wise disparities persist even with balanced training data, i.e. class-frequencies alone can not explain class-wise disparities. We will clarify the same in paper (Table 16 in revision).

Table : VOC 2012 val, DeepLabV3-R50 trained with SegPGD-AT, robust = SegPGD-3 at ε 8/255, mean ± std over three seeds.

Training class distributionClean mIoURobust mIoUWorst-class rob. IoUBottom-5 rob. IoUmIoU spreadFreq.–rob. corr. ↓
Original (Baseline SegPGD-AT)63.58 ± 0.6322.76 ± 1.121.93 ± 0.237.29 ± 0.4775.09 ± 0.950.226 ± 0.020
Random pixel sampling (Matched)63.29 ± 0.8722.24 ± 0.641.77 ± 0.677.01 ± 0.5274.96 ± 0.760.207 ± 0.021
Class-balanced sampling51.54 ± 0.7019.37 ± 0.853.32 ± 0.227.03 ± 0.7050.75 ± 1.430.347 ± 0.030

2. In their analysis on conventionally trained (no AT) LIDAR segmentation, Mahima et al [1] discuss class-wise vulnerabilities and make findings we would summarize as follows: the ground classes suffer the largest drop in IoU under perturbations, while for point injection or removal, the IoU drops near-linearly with increased class frequency, i.e. they observe a class-frequency dependence in undefended LIDAR segmentation models.

Our manuscript instead focuses on the overexpression of class-wise disparities in conjunction with (adversarial training), not considered in their paper, and shows that class-frequencies can not fully explain class-wise vulnerabilities in this case.

Halmosi et al. [2], as our manuscript, consider adversarially trained models for semantic segmentation, and is therefore more closely related. Their focus is the bias of object size and they find that small objects are more vulnerable than, for example the background. They do not report a full class-wise analysis.

In contrast to their work, we analyze the class-wise distribution. We further show that these disparities can not be explained by the class-frequency alone (Figure 4). Last, we propose a method, RL-SAF, that reduces class-wise disparities, which none of the two approaches does. We updated our related work section in revision accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Overall Response

Dear Reviewers,

We thank all reviewers for their constructive feedback and are glad they found our work well-motivated and relevant to robust and fair semantic segmentation. In particular:

  • Clear presentation: Reviewers 2cqq, kYGE, and 78o7 found our work well organized and easy to follow, with a coherent progression from a thorough analysis of class-wise robustness disparities, to why classification-based fairness methods fail to transfer, to RL-SAF as a solution for adversarial semantic fairness.
  • Well-motivated problem: Reviewer 78o7 highlighted the significance of our problem setting for future research, independent of our proposed method.
  • Transparent trade-offs: Reviewer kYGE appreciated our open discussion of clean-accuracy loss and behavior under stronger attacks. In particular, this trade-off also motivated us to further improve the hyperparameters used to train RL-SAF during the pre-rebuttal phase, and we found that trade-offs can be further improved with different batch-norm settings (see R2).

Below, we address two points shared across reviewers, generalization (R1) and the robustness–fairness trade-off (R2). Our individual responses refer to them to avoid repetition.

R1. Generalization across architectures and datasets, and statistical reliability. Beyond DeepLabV3 on Cityscapes, we now evaluate on SegFormer and PASCAL VOC 2012. We compare RL-SAF against SegPGD-AT under matched settings on clean mIoU, robust mIoU, worst-class robust IoU and bottom-5 robust IoU. We also report 3-seed results for statistical reliability. ADE20K training is underway but will not finish within the rebuttal period; we will include it in the final version.

Table R1a: DeepLabV3 on Cityscapes, 3 seeds (mean ± std), SegPGD with ε = 8/255 and 3/10 steps. Table 8 in the revision; multi-seed DeepLabV3 results on PASCAL VOC 2012 are in Table 10.

MethodEval. conditionnmIoUWorst-class IoUBottom-5 IoU
SegPGD-ATClean360.38 ± 0.3429.59 ± 0.2537.48 ± 0.85
RL-SAFClean359.39 ± 0.5129.79 ± 0.3835.65 ± 0.41
SegPGD-ATSegPGD-3 (ε=8/255)327.15 ± 0.660.09 ± 0.091.14 ± 0.27
RL-SAFSegPGD-3 (ε=8/255)325.50 ± 0.400.64 ± 0.303.41 ± 0.11
SegPGD-ATSegPGD-10 (ε=8/255)317.82 ± 0.310.00 ± 0.000.25 ± 0.04
RL-SAFSegPGD-10 (ε=8/255)315.28 ± 0.310.11 ± 0.080.99 ± 0.04

Table R1b: SegFormer on PASCAL VOC 2012, 3 seeds (mean ± std), SegPGD with ε = 8/255 and 3/10 steps. Table 11 in the revision.

MethodEval. conditionnmIoUWorst-class IoUBottom-5 IoU
SegPGD-ATClean376.98 ± 0.3337.25 ± 1.9353.81 ± 0.32
RL-SAFClean374.23 ± 0.2330.97 ± 4.0748.29 ± 0.57
SegPGD-ATSegPGD-3 (ε=8/255)333.89 ± 0.324.23 ± 0.9912.08 ± 0.36
RL-SAFSegPGD-3 (ε=8/255)334.17 ± 0.946.80 ± 0.3514.38 ± 1.07
SegPGD-ATSegPGD-10 (ε=8/255)316.90 ± 0.241.02 ± 0.254.21 ± 0.28
RL-SAFSegPGD-10 (ε=8/255)317.29 ± 0.742.08 ± 0.365.00 ± 0.21

R2. Robustness–fairness trade-off. In the paper, we explicitly described the clean-accuracy trade-off that arises from improving the robust accuracy of the bottom classes. This prompted us to improve on the previously reported trade-offs during the pre-rebuttal phase. RL-SAF primarily targets robustness of vulnerable classes rather than maximizing overall robust mIoU. Through a hyperparameter search, we found that batch-norm settings improve the balance between clean accuracy, overall robustness and tail-class robustness, although a smaller trade-off remains.

Table R2: Revised RL-SAF vs. SegPGD-AT in the setting of Table 2 of the paper (DeepLabV3, Cityscapes, ε = 8/255). B-5 = mean IoU of the five worst classes. All differences in %. Table 16 in the revision; 3-seed results are in Table R1a.

MetricSegPGD-ATRL-SAF (submission)RL-SAF (revised)Δ revised vs. before
Clean mIoU60.0444.47 (−15.57)58.95 (−1.09)+14.48
SegPGD-3 mIoU26.7827.05 (+0.27)25.17 (−1.61)−1.88
SegPGD-3 Worst / B-50.18 / 0.830.58 / 3.400.99 / 3.42+0.41 / +0.02
SegPGD-10 mIoU17.4714.26 (−3.21)15.02 (−2.45)+0.76
SegPGD-10 Worst / B-50.00 / 0.210.04 / 0.640.19 / 1.02+0.15 / +0.38

The revised RL-SAF cuts the clean-accuracy gap to SegPGD-AT from 15.57% to 1.09% while keeping or improving tail-class gains: B-5 robust IoU is equal or better under all attacks, and worst-class IoU is higher under both SegPGD attacks. The remaining cost is a moderate robust mIoU drop (1.6–2.5%), which falls mainly on already highly robust classes (see per-class breakdown in our response to Reviewer 78o7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.