acceptodds
Under review as a conference paper at ICLR 2027

RePAIR: Inverting Backdoor Attacks in Semantic Segmentation

Abstract

Despite significant advances in computer vision, semantic segmentation models remain vulnerable to backdoor attacks, posing a security risk. These attacks redirect pixels of a chosen victim class to an attacker-specified target class when a trigger is present, while preserving clean predictions. Existing defenses are largely derived from image classification, while dedicated segmentation defenses remain scarce and typically require retraining or model modification. We introduce RePAIR, a post-hoc, label-free, attack-unaware defense based on the observation that backdoor-induced shifts can be stronger in the classifier head than in internal representations, leaving recoverable victim-class semantics. RePAIR infers the victim-target relation from an unlabeled, potentially contaminated image pool, constructs a safety-approved semantic reference, and localizes suspicious pixels by comparing head predictions with internal semantic evidence. It then applies feature-space and output corrections toward the inferred victim semantics, while conservative localization and output constraints limit changes to benign predictions. We model the attack as adversarial probability transport and repair as its approximate reversal, deriving bounds and sufficient conditions for localization, residual attack success, clean-side changes, defense-induced target predictions, and attack-unaware pair identification. Experiments across segmentation architectures and attack settings show that RePAIR substantially reduces attack success while largely preserving clean performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Semantic Segmentation, Computer Vision, Backdoor-Attacks, Representation Learning, Optimal Transport

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WVYMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents RePAIR which is a post-hoc, label-free defense against backdoor attacks on semantic segmentation models. It infers the relation between victim and target semantic classes and uses an observation that backdoor attack effects appear to be more prominent in the pixel classifier head than in the internal representations. The authors propose approaches for CNN- and Transformer-based architectures and demonstrate the results for corresponding image segmentation models on multiple datasets.

Critical strengths
  1. The paper addresses an important problem of backdoor attacks on segmentation models which poses a threat for safety-critical applications where the class of important objects can be switched causing problems in downstream processes.

  2. The proposed method appears to provide significant robustness improvement over state of the art (see Table 1) on both CNN and Transformer architectures without modifying the potentially compromised segmentation model.

Critical weaknesses

No critical weaknesses were identified.

  1. Minor writing issues:
  • line 033: proving -> providing?
  • line 291: "which differs architectures." - the phrasing seems to be incorrect.
  1. Minor formatting issue: consider increasing the font in the Figure 2 to improve readability since it is much smaller than the main text font.
Questions for authors
  1. Have you tested the method in the untargeted backdoor attack setting i. e. modifying the predicted class to any other class, which is simpler for the attacker and more difficult for the defender?
  2. Does "sim" in the line 295 mean cosine similarity?
  3. Why are some numbers in Table 1 underlined?
3 comments

We thank Reviewer WVYM for the positive assessment, in particular for acknowledging the practical importance of backdoor defenses for safety-critical segmentation and the improvement over the state of the art.

W1/W2. We thank the reviewer for pointing out these minor issues. All suggested changes have been addressed in the revised manuscript. Line 033: changed "proving" to "providing" / line 291: rephrased the sentence to "which differs between architectures" / Figure 2: enlarged the font to improve readability and match the main text.

Q1. To the best of our knowledge, existing work on backdoor attacks in semantic segmentation has focused on targeted attacks. RePAIR is likewise designed for this setting, in which triggered victim-class pixels are redirected toward a specific target class. Its repair mechanism explicitly exploits and reverses this directed victim-to-target transport. We have not evaluated untargeted backdoor attacks and have clarified this scope in the Limitations.

Untargeted attacks, in which victim-class pixels are redirected to multiple possible classes, would be an interesting and more challenging setting. In this case, probability mass may leave the victim class in multiple directions, preventing pair discovery from identifying a single dominant victim-to-target direction. The builder would therefore be expected to abstain rather than apply an unreliable repair. Extending RePAIR to this setting would require generalizing the single-target formulation to a set of target classes, i.e., a victim-to-multiple-target transport. We consider this an interesting direction for future work.

Q2. Yes, "sim" denotes cosine similarity. We have now explicitly defined it at its first occurrence in the revised manuscript.

Q3. Underlined entries indicate the second-best result in each column, while bold entries indicate the best. We have added this clarification to the table caption.

Area_Chair_FjRpOpenReview ↗

take part in the discussion

Dear reviewer WVYM,

please take part in the discussion. time is running out.

thank you your ac

Reviewer_WVYMOpenReview ↗

Thank you

Thank you for the rebuttal. I only had minor concerns and they were addressed. I am keeping my score.

Reviewer_FwJmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces RePAIR, a post-hoc, label-free, and attack-agnostic defense against backdoors in semantic segmentation. It uses residual victim-class semantics in intermediate representations to infer the attacked class pair, localize backdoor-affected pixels, and repair them without retraining the model.

Critical strengths
  1. Practical problem setting. The defense requires no prior knowledge of triggers or attack types. Few segmentation backdoor defenses work under these constraints, so the setting is relevant for real deployment.

  2. Interesting perspective. The authors observe that backdoors mainly corrupt the segmentation head’s output, while victim-class semantics often survive in intermediate representations. Using the mismatch between the two as a repair signal is an interesting idea for defending segmentation models.

Critical weaknesses
  1. Missing ablations. RePAIR combines several modules, including class-pair identification, reference filtering, localization, feature transport, logit correction, and output constraints, yet the paper reports no ablation of any of them. It is therefore unclear which components are actually necessary.

  2. The core claim lacks direct evidence. ASR only counts pixels still predicted as the target class, and mIoU-A excludes the attacked pixels. No metric shows whether those pixels are restored to their correct class. A simple attack that suppresses the target class could also reduce ASR, so the central repair claim remains unverified.

Questions for authors
  1. Could you provide ablation study results for each module, including the precision and recall for the localization task?

  2. Could you report the IoU or recall for the ground-truth classes of the attacked pixels, and compare these results against a simple baseline?

  3. What are the accuracy and rejection rates for class-specific recognition tasks? When this method is applied to an unattacked model, does it lead to any false positives?

6 comments

We thank Reviewer FwJm for the valuable feedback, and in particular for acknowledging the practical setting and the proposed perspective of using the discrepancy between the segmentation head and the internal representations as a repair signal.

W1/Q1. We provide an ablation study of the main RePAIR components on DeepLabV3+ and Cityscapes, evaluated on runs in which RePAIR released the true victim-target pair (results averaged over attacks). We report the attack success rate (ASR) and the IoU on the victim class (on clean and triggered examples):

VariantASRIoU$_v$ cleanIoU$_v$ triggered
no defense88.793.321.7
RePAIR (full)1.973.267.0
w/o feature transport2.873.567.2
w/o logit correction57.790.441.2
w/o output projection42.888.649.6
w/o rescue localization9.187.075.9
w/o rupture branches35.293.273.8
w/o No-New-Target veto2.173.267.0

The ablations reveal different roles of the individual components. Logit correction and output projection are essential for reducing ASR: removing either increases ASR substantially. The rupture branches are particularly important for background-type attacks. Without them, ASR for BadSeg-B2O increases from 0.07% to 93.6%. In contrast, feature transport has only a marginal effect and is largely redundant with logit correction. We therefore clarify that it is an optional component. Rescue localization lowers ASR from 9.1 to 1.9% at the cost of victim-class IoU on triggered images (75.9 to 67.0), while the No-New-Target veto has little effect on ASR but prevents approximately 88% of the target predictions that the repair would otherwise introduce on clean images.

We additionally evaluate localization against the ground-truth attacked pixels. Across the evaluated attacks, localization achieves an average precision of 56.8% and recall of 96.6%. Thus, it captures nearly all attacked pixels, but also includes false positives, which contribute to the loss in victim-class IoU on clean images. We will add the per-attack results to our paper. Pair identification and reference filtering are evaluated separately in our response to Q3.

W2/Q2. We agree that a reduced ASR alone does not establish that attacked pixels are restored to their correct class: the ASR only counts pixels that are still predicted as the target class, and a defense that merely removes the target class would also lower it. We therefore evaluate the predicted class of the attacked pixels directly and compare RePAIR against the target-suppression baseline suggested by the reviewer.

Attack (DLv3+)Rec$_v$ no def. →\to RePAIRRec$_v$ suppr. MMIoU$_v$ RePAIRIoU$_v$ suppr. MMIoU$_v$ suppr. AA (oracle)
IBA-NNI8.0 →\to 96.689.659.977.890.4
IBA-PRL0.3 →\to 97.486.563.685.086.8
ConSeg10.1 →\to 94.984.257.565.785.3
OFBA36.5 →\to 97.292.662.086.694.6
BadSeg-O2B2.1 →\to 95.588.262.983.190.0
BadSeg-O2O0.4 →\to 95.784.090.683.484.1
BadSeg-B2O2.4 →\to 99.074.194.893.996.4
BadSeg-B2B0.0 →\to 87.844.848.436.647.6

The attacked region AA contains all pixels whose true class is the victim (for BadSeg-B2O restricted to the designated attack region, as for the ASR). MM denotes the localization mask computed by RePAIR without access to labels. All metrics are aggregated over the triggered evaluation images. "Recv\mathrm{Rec}_v no def. →\to RePAIR" reports the fraction of attacked pixels in AA correctly predicted as their true class, before and after RePAIR. A value of 100% means that every attacked pixel is restored to the victim class. "Recv\mathrm{Rec}_v suppr. MM" gives the same recall for a baseline that suppresses the target logit on MM while leaving all other predictions unchanged, thereby isolating the effect of RePAIR's repair operator. "IoUv\mathrm{IoU}_v RePAIR" reports the victim-class IoU over all image pixels, thus also accounting for false-positive victim predictions outside AA. "IoUv\mathrm{IoU}_v suppr. MM" gives the corresponding IoU for the target-suppression baseline. Finally, "IoUv\mathrm{IoU}_v suppr. AA (oracle)" applies the same baseline to the ground-truth attacked region AA instead of MM and serves as a label-dependent oracle reference.

Restoration to the correct class: Without defense, only 0.0–36.5% of the attacked pixels are predicted as their true class. After RePAIR, it is 87.8–99.0% for every attack. The reduction in ASR is therefore achieved by restoring the victim class and not merely by removing the target class.

Recall compared with target suppression: On the same mask MM, RePAIR restores more attacked pixels to the victim class than target suppression for every attack (87.8–99.0% vs. 44.8–92.6%). After the target class is removed, the next most likely class of the poisoned model is not necessarily the victim. The gap is therefore largest for the background attacks (BadSeg-B2O: 99.0 vs. 74.1%, BadSeg-B2B: 87.8 vs. 44.8%). RePAIR instead moves the prediction explicitly toward the victim class.

IoU compared with target suppression: For the object-to-background attacks (IBA-NNI, IBA-PRL, ConSeg, OFBA, BadSeg-O2B), the victim IoU of RePAIR is lower than that of target suppression on MM. Because the recall of RePAIR is higher, the difference stems from false-positive victim predictions. The localization mask also contains benign pixels (precision 57% on average), and RePAIR assigns the victim class to them as well, whereas target suppression leaves them at their next-best, mostly correct class. For BadSeg-O2O, BadSeg-B2O and BadSeg-B2B, RePAIR attains the higher IoU.

Dependence of the baseline on RePAIR: Target suppression on MM requires the mask and the victim–target pair inferred by RePAIR and therefore cannot be applied without RePAIR's attack-unaware pair identification and localization. It constitutes an alternative repair operator on top of RePAIR rather than an independent defense. The component ablation (Q1) shows the corresponding trade-off: restricting the localization (without the rescue branch) raises the victim IoU on triggered images to 75.9 on average, at the cost of a higher ASR (9.1 vs. 1.9% on average). We will report this trade-off explicitly in the paper.

Q3. Pair recognition was evaluated on Cityscapes using DeepLabV3+, with three reference pools per attack. The correct pair was identified in all 24 runs (8 attacks × 3 pool seeds). RePAIR abstained in all three runs on the unattacked DeepLabV3+ reference model, using three different pool seeds. No pair was released, so there were no false positives and the model output remained unchanged.

Thank you for your thoughtful rebuttal. I greatly appreciate the authors' efforts; the new results have resolved most of my initial questions and strengthened the paper's persuasiveness.

However, I still have two questions:

Trade-off: While the recovery results are encouraging, the IoU for the clean victim category dropped from 93.3% to 73.2%. Could the authors further discuss this trade-off and potential methods to mitigate its impact on benign predictions?

Necessity: Given the relatively small performance difference after removing feature transfers (ASR: 1.9% vs. 2.8%), could the authors further clarify their unique contribution and the possibility of simplifying the framework?

In conclusion, I look forward to further explanation from the authors, which will help me make a final evaluation.

Reviewer_ff4aOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RePAIR, a post-hoc, label-free, attack-unaware defense against targeted backdoor attacks in semantic segmentation. The compromised model remains frozen; the defense assumes no trigger knowledge, no known victim-target pair, no poisoning rate, and no clean/poison provenance, and it operates on an unlabeled pool that may contain triggered samples. RePAIR identifies a victim-target direction from internal-versus-head evidence, builds a safety-approved reference state with fail-closed abstention, localizes suspicious pixels, and repairs them first in feature space and then, if needed, in the output logits. The attack is modeled as directed probability-mass transport on the class simplex, and repair is framed as an approximate reversal. Experiments on Cityscapes, PASCAL VOC 2012, and ADE20K with DeepLabV3+ and SegFormer compare against fine-tuning, pruning, Segment Cleanse, and ABL, reporting large ASR reductions and mostly preserved benign mIoU.

Critical strengths
  1. A well-posed and practically relevant defense setting, addressed with unusually broad attack coverage. Existing segmentation defenses either detect without repairing, such as STRIP, TeCo, and BEATRIX, or require retraining, model modification, clean labeled data, or knowledge of the attack, such as pruning, fine-tuning, Segment Cleanse, and ABL. RePAIR’s threat model is the most permissive among evaluated defenses: frozen model, no labels, no trigger reconstruction, no prior knowledge of the victim-target pair, and an unlabeled pool that may itself contain triggered samples. The fail-closed design, with abstention when evidence is insufficient, is the right safety posture for a claimed-utility defense in safety-critical perception. The empirical case is correspondingly broad: eight recent attack configurations, including strong published attacks from 2023–2026, across a CNN and a hierarchical transformer, with consistent and large ASR reductions on Cityscapes while preserving mean benign mIoU, and with an honest ADE20K negative control showing that the authors do not over-claim where the premise fails. This combination of setting and coverage is consequential: it is the main reason the paper is worth publishing if the mechanism-verification gaps below are closed.

  2. The mechanism observation is interesting, honestly scoped, and architecture-aware. The idea that the backdoor shift can be attenuated between the internal representation and the head, so that repair can be framed as reversing a directed victim-to-target transport using surviving internal evidence, is a useful conceptual contribution. The paper is careful about what the theory does and does not claim: the inverse operator versus transposed coupling distinction, sufficient-not-necessary pair separation, empirical budgets versus population guarantees, and Remark 18. The demonstrated need for different reference constructions on CNNs versus transformers, namely a single-stage prototype read-out versus a distributed multi-layer logistic read-out, is a concrete and informative finding that practitioners porting representation-based defenses will care about.

Critical weaknesses
  1. The defense’s two load-bearing mechanisms are never directly validated, so it is impossible to tell whether RePAIR works for the reasons claimed, or where it silently fails. Everything downstream, including localization, feature transport, and the veto, is conditioned on the builder releasing the correct pair or at least the correct target. Yet the paper reports no pair-identification accuracy, no abstention rate, and no incorrect-release rate across the 8 Cityscapes and 5 VOC attack instances. Similarly, the four quantities that parameterize every theoretical bound are all computable with oracle evaluation labels, and none are reported, including for the cases where the defense visibly underperforms: BadSeg-B2B at 7.86% defended ASR on DeepLabV3+, and the VOC DeepLabV3+ results where defended ASR stays at 5.41–19.67%. Without these diagnostics, the theory is unfalsifiable as presented, and the reader cannot distinguish “the localizer missed” from “repair failed on localized pixels” from “the pair was misidentified.” This is central to my assessment because the paper’s headline differentiator over prior work is precisely the attack-unaware identification claim. If identification is unreliable or frequently abstains, the practical value proposition changes materially. Related: an incorrect but confident pair release may be worse than abstention, because the defense would repair toward the wrong victim class and the current safety analysis mainly bounds changes inside the selected mask and spillover outside it, not damage caused by repairing toward a wrong semantic direction.
  2. No adaptive-attacker analysis and a narrow evaluation footprint weaken the security claims. The defense’s premise is a specific structural property of current attacks, namely head-versus-representation divergence. A rational adversary who knows RePAIR can attack the premise directly, for example by adding a feature-consistency or representation-level loss during poisoning to erase internal victim evidence, or by choosing victim-target pairs with strong natural co-occurrence, as the paper’s own VOC analysis notes natural victim-to-background relations can compete with attack evidence. The ADE20K control also shows weak attack uptake. There is no experiment and no discussion of either case. Compounding this, the evaluation covers a single victim-target pair per relation type and, as far as I can tell, a single seed with no variance reported. The utility accounting hides per-attack values behind means with min/max deviations. Notably, SegFormer’s defended benign mIoU of 72.30% is 3.5 points below the undefended poisoned model at 75.80% and 8.2 points below the clean model. This sits uneasily with the “largely preserving clean performance” framing and with the paper’s own claim that these attacks preserve clean predictions. The undefended poisoned SegFormer already loses 4.7 benign mIoU versus clean, which is itself worth explaining. Together these gaps mean the current evidence supports “RePAIR mitigates the specific published attacks as configured,” which is narrower than the paper’s general claims. For ICLR, this is not yet a sufficient security evaluation for a defense paper that makes broad attack-unaware and architecture-general claims.
  3. The theoretical framework is over-formalized relative to what it actually establishes, and it is not empirically validated. The local transport model describes a directed shift between two class probabilities, but several evaluated attacks, such as ConSeg, involve co-occurring classes and multi-class relabeling. The paper does not test whether this simplification holds for the evaluated attacks, nor does it report whether the pair-correct or target-correct conditions required by the bounds actually hold in the 13 attack instances. Most propositions are deterministic release contracts or algebraic identities under idealized assumptions, not testable statistical guarantees. For an ICLR submission, the theory currently reads as a careful re-description of the intuition rather than a source of falsifiable predictions or practical design rules. This weakens the claimed theoretical contribution and makes it difficult to judge when the bounds should be expected to hold.
  4. The method has many components, but there is no component-level ablation to show which parts are necessary. The paper does not ablate feature transport versus output-logit correction, the No New Target Veto, the safety firewall, the probability-residual branch, or the transformer direction-voting machinery. Without these ablations, it is impossible to tell whether the performance comes from the proposed mechanism, from additional computation, or from a small subset of components. The transformer pair-discovery pipeline in particular is complex, spread across multiple appendix subsections, and lacks a contribution analysis. For a method that introduces architecture-specific branches and several residual localization stages, this is a substantive gap: the reader cannot identify the minimal effective design or understand which design choices are essential.
  5. Reproducibility and hyperparameter sensitivity are insufficiently addressed. The code link is a placeholder, and the paper says code will be released after acceptance. The method relies on many fixed coefficients, thresholds, architecture-specific branches, and minimum reference requirements, yet there is no sensitivity analysis for pool contamination rate, pool size, class coverage, calibration budgets, or the safety thresholds. The minimum of 24 safety-approved images and 75% semantic coverage, for example, appears arbitrary without a sweep or justification. For a defense paper making deployment-relevant claims in safety-critical perception, this is a significant gap. It also makes it difficult to assess whether the reported behavior is robust or tuned to the specific evaluation settings.
Questions for authors
  1. Pair identification. For each of the 13 evaluated attack instances, 8 Cityscapes and 5 VOC, what pair, or fail-closed abstention, did the attack-unaware builder actually release? Please report the identification accuracy, abstention rate, and most importantly the incorrect-release rate. If a wrong pair is released with high confidence, what is the observed damage to benign predictions? The theory only bounds this under pair-correct or target-correct conditions.
  2. Mechanism diagnostics. Can you measure and report the following on your evaluation data, where ground truth is available for evaluation? First, the fraction of successful attacks for which internal victim evidence is absent. Second, the fraction of localized baseline attacks that still predict the target after repair. Third, the rate of new target predictions outside the original baseline attack. Fourth, the probability that a benign pixel is selected for repair. In particular, what dominates the residual ASR for BadSeg-B2B on DeepLabV3+ at 7.86% and for the VOC DeepLabV3+ cases up to 19.67%: missed localization, failed repair, or new target creation?
  3. Adaptive attacks. How does RePAIR behave against a poisoning objective augmented with a feature-space consistency loss designed to destroy internal victim evidence, and against a victim-target pair chosen for strong natural co-occurrence, such as sidewalk to road or classes with high conditional dependence? Even a single small-scale experiment would help, since the current attack set is entirely non-adaptive to the defense.
  4. Image pool. What is the size of the unlabeled pool in each experiment, and what fraction of it can be triggered samples before the safety firewall or the pair scorer breaks? Please report sensitivity to pool contamination rate and pool size around the operating point. The minimum of 24 safety-approved images and 75% semantic coverage in A.2.3 appears arbitrary without such a sweep.
  5. Utility accounting. Why is SegFormer’s defended benign mIoU of 72.30% lower than the undefended poisoned model at 75.80%? Is this repair-induced damage on benign pixels, or a consequence of the attacks themselves? The 4.7-point clean-to-poisoned gap suggests the evaluated attacks do not fully preserve clean behavior. Please also report per-attack benign and attacked mIoU rather than only means with min/max deviations.
  6. Baseline protocol. ABL is a training-time defense. Were its numbers produced by retraining from scratch under the ABL protocol on the same poisoned datasets? Given that fine-tuning uses 10% clean data and Segment Cleanse uses 1,000 clean images, can you report an ablation where RePAIR is given access to a labeled clean set, to isolate how much of the advantage comes from the mechanism versus from the information asymmetry in the comparison?
  7. Cost. What is the per-image inference overhead, including latency and memory, of localization, retrieval-based feature transport, and re-running the network remainder, relative to the undefended model? For the transformer path, a per-pair logistic read-out implies one model or score per ordered class pair. Is this run for all ordered pairs at inference? What is the resulting latency and memory overhead? Can candidate pairs be pruned or cached?
3 comments

We thank Reviewer ff4a for the detailed and constructive review, and in particular for acknowledging the well-posed and practically relevant defense setting, the broad attack coverage, and the fail-closed design. All new results are for DeepLabV3+, the SegFormer results will be added to the updated version.

W1/Q1. We agree that pair identification and localization/repair must be validated directly. The released pair or abstention per attack and pool draw is listed in the following table, and the residual ASR is decomposed in Q2. Results are summarized over three random draws of the pool per attack for both datasets:

correct pairabstentionwrong pairclean model: false release
Cityscapes (8 attacks)24/240/240/240/3
VOC (5 attacks)15/150/150/15–

Damage from a confident wrong pair: We agree that the safety analysis did not cover this case. No wrong pair was released in any of the 39 runs. On clean, unattacked models, where any released pair would be wrong by definition, the builder abstained in all runs, so benign predictions were unchanged. We add this case to the safety analysis: the fail-closed release contract prevents repair toward a wrong semantic direction, and its effectiveness is now measured instead of assumed.

W2/Q3/Q5. Adaptive attacks: Your suggestion regarding adaptive attacks is interesting and highly relevant. To the best of our knowledge, however, no adaptive backdoor attack specifically targeting defenses for semantic segmentation has been proposed so far. We agree that investigating such attacks would be a valuable direction for future work. An adaptive attacker might attempt to suppress the victim evidence used by RePAIR while maintaining attack effectiveness. In this case, pair discovery may fail to identify a dominant direction, causing the builder to abstain rather than perform an unreliable repair.

ADE20K is a negative control: The attacks reach only 2.4–4.9% undefended ASR (Table 5), so there is no backdoor to invert. The same question is answered more directly by applying RePAIR to clean models: it abstains in every run (see W1/Q1).

Co-occurring pairs, one pair per relation type: We trained four new pairs:

Pairreleased / correctASR no def.ASR RePAIRmIoU-B no def. / RePAIR
road→sidewalk (B2B)3/3 / 398.789.0178.00 / 77.49
sidewalk→person (B2O)3/3 / 384.680.3878.26 / 77.24
person→sidewalk (O2B)3/3 / 392.211.7378.13 / 77.12
car→sidewalk (O2B)3/3 / 396.731.1978.28 / 77.72

The correct pair is released in all 12 runs, and the repair behaves as for the original attacks: ASR drops to 0.38–1.73% for the B2O and O2B pairs and to 9.01% for road→sidewalk, at a mIoU-B cost of at most 1.0 pp. Natural co-occurrence did not lead to a wrong release.

Pool draws: Each of the three draws samples new 50 + 50 pool images. A pair is released for every draw, and the defended ASR varies little (OFBA 1.11 ± 0.18%, BadSeg-B2O 0.10 ± 0.02%, BadSeg-O2B 2.16 ± 0.02%). Training seeds of the poisoned models are fixed by the attack checkpoints and are not a property of the defense.

Per-attack utility: We report the mean over the three pool draws:

AttackmIoU-B cleanmIoU-B pois.mIoU-B RePAIRmIoU-A pois.mIoU-A RePAIR
IBA-NNI77.8273.2171.5863.9663.30
IBA-PRL77.8272.7970.6467.7067.08
ConSeg70.0469.7968.5964.5164.03
OFBA77.8279.1676.7774.3173.39
BadSeg-O2B78.4577.8475.9172.9172.29
BadSeg-O2O78.4578.1477.8472.4972.38
BadSeg-B2O78.4577.8276.6277.3976.22
BadSeg-B2B78.4578.6576.9574.1873.27

RePAIR lowers mIoU-B by 0.3–2.4 pp, consistent with the benign repair fraction κcl\kappa_{\text{cl}} of 1.4–3.8% (see Q2). The clean-to-poisoned gap is concentrated in IBA-NNI and IBA-PRL (4.6 and 5.0 pp), while the BadSeg attacks stay within 0.7 pp and OFBA is even 1.3 pp above the clean model.

W3. The revision separates the conceptual transport model, the deterministic release contracts, and statements that hold only under CpairC_{\text{pair}} / CtgtC_{\text{tgt}} (Table 2). The theory is now tied to measurements: the bound quantities are measured (Q2), CpairC_{\text{pair}} holds in every released run, and bounds (6) and (8) held in all 36 Cityscapes runs. The two-class model describes the dominant victim→target shift, not the full poisoning mechanism. For OFBA and ConSeg, which affect several classes, RePAIR restores the victim class for 98.2% and 97.6% of the attacked pixels, while only 0.1% / 0.6% are shifted to other classes. The dominant component is therefore captured, and the remainder quantifies what the model omits. Two practical design rules follow. The repair margin is calibrated only on safety-approved victim pixels (Prop. 11), and release requires target resolution under the pair-release ratio, so the builder abstains when co-occurrence and attack evidence cannot be separated. For the co-occurring pairs in W2/Q3/Q5 they could be separated, and the correct pair was released in every run.

W4/W5/Q4. Component ablation (Cityscapes, mean over seven attacks):

VariantASR$_{\text{def}}$ (max)mIoU-BIoU$_v$ cleanIoU$_v$ triggered
no defense88.776.493.321.7
RePAIR (full)1.9 (5.4)74.773.267.0
w/o feature transport2.8 (7.4)74.873.567.2
w/o logit correction57.7 (81.4)76.190.441.2
w/o output projection42.8 (74.1)75.988.649.6
w/o rescue localization9.1 (24.7)75.987.075.9
w/o rupture branches35.2 (93.6)76.393.273.8
w/o No-New-Target veto2.1 (5.7)74.773.267.0

The output-side correction is necessary: removing the logit correction or the output projection raises ASR by 41–56 pp. The rupture branches are required for background-type attacks (BadSeg-B2O: 0.07 → 93.6% without them) but cost victim IoU on object victims, the price of not knowing the attack type. The rescue localization lowers the worst-case ASR from 24.7 to 5.4%. The No-New-Target veto is a safety component rather than an ASR component: without it, ASR rises only slightly (1.9 → 2.1%), but it removes about 88% of the target predictions the repair would otherwise create on clean images. Feature transport is largely redundant with the logit correction. In pipeline order it already moves 3–71 pp of the attacked pixels back to the victim class, but removing it changes ASR by only +0.9 pp, because the logit correction then repairs the same pixels. We now describe it as an optional stage, and the minimal effective design is semantic localization, rupture branches, logit correction and output projection.

The code will be published as announced, and all hyperparameters are listed in Table 4. In all new experiments of this response, RePAIR builds its defense state from a pool of only 100 unlabeled images (50 clean, 50 triggered), and pool images are excluded from evaluation. The minimum image counts in A.2.3 and Table 4 are scaled proportionally with the pool size while all quantile levels stay fixed, so for this pool release requires at least 5 safety-approved images and 75% class coverage. We state this explicitly in the revised A.3. Sensitivity to the probability budget δprob\delta_{\mathrm{prob}}, the rupture quantiles and the repair strength is reported in three new tables in the revised appendix. Over all tested settings, the defended ASR stays below 4% for every attack except BadSeg-B2B (5.2–11.3%), and mIoU-B varies by about 1 pp (74.1–75.1). Weaker budgets and repair strengths trade a slightly higher mIoU-B for a higher ASR, and the rupture quantiles have almost no effect. The defaults of δprob\delta_{\mathrm{prob}} and α\alpha lie at the strong end of the tested ranges by design. ASR decreases monotonically with δprob\delta_{\mathrm{prob}}, but the benign collateral damage grows with it (mIoU-B 75.14 → 74.23), and the mIoU-B cost per percentage point of BadSeg-B2B ASR reduction rises towards the default (0.15 pp for 0.01 → 0.02, 0.23 pp for 0.02 → 0.05). Weaker repair (α<1\alpha<1) raises ASR on every attack while gaining at most 0.1 pp mIoU-B.

Q2. Measured with oracle labels for Cityscapes: (i) successful attacks without internal victim evidence ηref\eta_{\text{ref}}: 0.11–0.29, except BadSeg-B2O (0.94), where the rupture branches carry the repair, (ii) localized attacked pixels still predicted as target after repair εrep\varepsilon_{\text{rep}}: ≤ 0.001, (iii) new target predictions outside the original attack δnew\delta_{\text{new}}: ≤ 0.001, (iv) benign pixels selected for repair κcl\kappa_{\text{cl}}: 1.4–3.8%. The residual ASR is decomposed exactly into missed + failed + new = ASR$_{\text{def}}$, and in every run it consists almost entirely of missed localization. For BadSeg-B2B (7.86%, Table 1), the residual ASR is entirely missed by localization, 0.00 pp are localized but not repaired, and 0.00 pp are new target pixels, consistent with its comparatively high ηref=0.29\eta_{\text{ref}}=0.29. For VOC, the dominant term is also missed localization: for BadSeg-O2B (19.67%, Table 6), the mask covers 72% of the attacked person pixels (vs. 91–99% on Cityscapes), no new target pixels are created, and approximately 18 of the 19.7 pp lie outside the mask. This is consistent with the natural person→background relation discussed in A.4.

Q6. ABL was trained from scratch on the same poisoned training sets following our segmentation adaptation of the ABL protocol (20 isolation epochs, isolation ratio p=0.01p=0.01, 5 unlearning epochs, Table 3). The compared defenses operate under different information requirements. ABL has access to the full labeled poisoned training set and the training procedure. Fine-tuning uses 10% labeled clean data, and Segment Cleanse uses 1,000 clean images with semantic information. In contrast, RePAIR requires no labels, clean/poison provenance, or trusted clean reference set. It can construct its complete defense state from as few as 100 unlabeled images, 50% of which are triggered. Thus, RePAIR operates under weaker information assumptions than the baselines, some of which benefit from labeled data or training-time access. We deliberately do not evaluate RePAIR with a labeled clean set, as its intended setting is label-free defense. This also demonstrates its applicability when such data is unavailable.

Q7. Measured on one RTX 6000 Ada (DeepLabV3+, Cityscapes, 512×1024, batch size 6), the undefended prediction takes 0.055 s per image and the defended prediction 0.38 s per image (median over runs), a 6.9× overhead (range 3.7–10.8×) dominated by the retrieval-based localization and the re-decoding of repaired features. These times cover only the prediction step, per test image. The one-time construction and calibration of the defense state from the pool precede deployment and are not included. Memory is dominated by the same retrieval step and grows linearly with the batch size. At batch 6 the defense runs within the 47.5 GB of the GPU, peaking at around 36 GB, and smaller batches reduce it at a proportional cost in throughput. If the builder abstains, the model is left unchanged and there is no overhead.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.