acceptodds
Under review as a conference paper at ICLR 2027

TARE: Weigh a Never-Poisoned Twin Before Reading Backdoor-Defense Costs

Abstract

Backdoor-defense leaderboards print a clean-accuracy drop and read it as the cost of removal. The drop is measured on the poisoned victim alone, so it cannot separate removal from what the defense does to any model, and it inherits where the victim started. Where the victim starts is, for three of BackdoorBench's sixteen attacks, a configuration file: WaNet, BPP and Input-Aware ship a MultiStepLR whose first milestone is the epoch count, so their victims never anneal and are the lowest-accuracy victims in 30/31 public CIFAR cells at ≤5%. On PreAct-ResNet18, fine-tuning-family defenses return a low start to a level of their own, so on those victims the published cost is negative, the benchmark's rating clips the "gain" to zero, and 2 of the 48 citing defense papers we read rest a no-cost claim on those cells; re-run with their released code, TSBD and CGD "gain" on a never-poisoned model too. We isolate the cause with a 2×2 that edits only that scheduler line, its swapped arms self-registered before they ran: the sign of the fine-tuning family's clean-model cost reverses both ways while its published gain on the annealed victim only shrinks toward zero, 44/44 seeds following the schedule, replicated on BPP, FT-SAM, CIFAR-100 and VGG19-BN and induced in a second toolkit. The instrument is TARE: run the identical defense on a never-poisoned twin of the same recipe, schedule and seed (on BackdoorBench, ≤10 poisoned images, admitted only below 5% attack success); what the twin loses is the tare. On BackdoorBench's BadNets grid seven of the eight defenses charge the twin (Neural Cleanse only where its detector fires), +0.13 points (fine-tuning) to +5.70 (I-BAU); the eighth, ABL, destroys it. Within an attack the start cancels from rankings, so the tare re-orders nothing there; what poisoning adds beyond it is printed under two estimators and not corrected, its removal share being unidentified. We ship the three-key patch, a signed tare column over 7 attacks × 8 defenses of the CIFAR-10 roster, and TARE-Z, a twin-free estimator for seed-stable defenses.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
backdoor defense, defense evaluation, trustworthy machine learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Qq3cOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper identifies makes several contributions:

  1. it identifies flaws in BackdoorBench -- that learning rate scheduling fails to be invoked for 3 attacks -- WaNet, BPP, and Input-aware -- with the consequence that victim models have very poor clean test accuracy. Subsequent fine-tuning defenses repair this poor accuracy. This leads to the (false) inference that there is no clean test set accuracy cost associated with applying the given defense (and even possibly a clean test accuracy benefit).
  2. The authors quantify the cost, in clean test accuracy, of applying a given defense, using the tare -- the difference in clean accuracy between that for a given model and that for a defense applied to the given model. The authors show that tare is nearly always positive, across different defenses.
  3. The authors demonstrate how important a careful learning rate schedule is for the clean test accuracy -- the tare reverses with the scheduler, not with the attack.
  4. The authors also demonstrate the importance of annealed scheduling for backdoor removal.
Critical strengths

Proposes the tare, a very sensible statistic added to the BackdoorBench backdoor mitigation leaderboard.

Identifies a problem with the scoring used for several backdoor attacks (WaNet, BPP, and Input-Aware) -- these methods were not "shipped" with a learning rate scheduler, which causes the data-poisoned models to have very low clean test accuracy baselines.

Identifies that the ABL defense grossly degrades clean test set accuracy.

Proposes TARE-Z, a surrogate of the tare that does not require seed matching.

Critical weaknesses

This paper is a bit too difficult to read. The authors use too much jargon (such as "charge the twin").

They also give way too much minutiae/technical details in both the abstract and in the Introduction -- the paper clarity could be improved.

The paper should make a better separation between their discovery of a bug in BackdoorBench (relating to MultStepLR) from the failure of this leaderboard to account for the defense's effect on clean accuracy of a model.

In Intro.: should define the tare before referring to it.

Sec. 2: "the other column of the table" -- which column is that ??

"describe the non-linear metric artifact DER's max(0,.) is" -- this is not a valid sentence.

Sec. 4.2: A0 used but not yet defined.

Questions for authors

What does DiD stand for ?

In Q(D_b Theta) = L_D + espsilon(Theta) is the function epsilon allowed to be different for different defenses ? If this is the case, it seems this decomposition can be trivially achieved, in which case "Whether a defense has such a measure" would be moot.

Reviewer_NGpGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper audits the interpretation of clean-accuracy changes in backdoor-defense evaluation. It identifies learning-rate configurations in BackdoorBench that leave some victims under-trained and uses a scheduler crossover to show that defensive fine-tuning can improve accuracy on control models without effective backdoors. It proposes TARE: apply the same defense to a control trained with a matched recipe, schedule, and seed, and report the control's accuracy change alongside the poisoned model's change. The clearest contribution is the controlled scheduler audit and its connection to published accuracy-gain claims. The additional contribution is an accounting protocol that distinguishes the defense's effect on a clean control from its effect on a poisoned model, without identifying the cost specifically attributable to backdoor erasure.

Critical strengths
  1. The scheduler crossover isolates a concrete source of misleading accuracy-gain attribution. Changing the schedule in both directions on WaNet and BadNets reverses the sign of the control-arm cost. Table 2 reports agreement in 131/132 seed-defense cells, providing stronger evidence than a leaderboard correlation.

  2. The finding is tested beyond the main configuration, including BPP, CIFAR-100, VGG19-BN, and an induced scheduler mismatch in BackdoorBox. The paper reports settings where the common defended accuracy level fails, which helps delimit the explanation.

  3. Rerunning TSBD and CGD connects the configuration artifact to specific published claims. The results show that part of their reported accuracy gains also occurs on controls without effective backdoors. The main text also acknowledges that neither the difference-in-differences interaction nor the post-defense contrast identifies erasure cost.

Critical weaknesses
  1. The experiments do not clearly establish TARE's value beyond the scheduler correction. The scheduler crossover demonstrates a configuration artifact, but does not establish when the proposed control-based protocol changes the interpretation of defense performance. The authors should show what additional conclusions TARE supports once the scheduler is corrected.

  2. The initial clean-accuracy deficit cannot generally be attributed to under-training. Some attacks may sacrifice clean accuracy to attain high ASR. In those cases, improvement after defense may reflect recovery from poisoning-induced degradation, so subtracting the clean-control change does not isolate the cost of backdoor removal.

  3. The incremental value over prior control-based evaluations remains unclear. The DER analyses change only 0/28 and 1/78 defense pairs. Ranking changes are not necessary, but the paper should clarify which conclusions the protocol improves and compare specialized defenses with ordinary fine-tuning under matched budgets.

  4. The control construction and TARE-Z assumptions require stronger justification. Most primary controls contain poisoned examples, while zero-poison validation covers limited configurations and defenses. Moreover, exchangeability before selection does not establish unbiasedness after conditioning on successful removal.

  5. The practical relevance of the reporting-baseline drift term in Equation (5) is insufficiently established. The term is zero in all headline experiments, and the paper does not quantify how frequently this mismatch occurs in existing evaluations. Evidence of its prevalence and magnitude is needed to justify its emphasis in the decomposition.

Questions for authors
  1. Can the authors reorganize the main experiments around the scheduler intervention, the control-arm diagnostic, and the downstream consequences, stating the question, measured quantity, conditioning rule, and conclusion for each?

  2. How should TARE be interpreted when a victim's accuracy deficit reflects an intrinsic accuracy-ASR tradeoff rather than under-training? Can the authors demonstrate its use across multiple valid attack operating points and compare specialized defenses against ordinary fine-tuning under matched resource budgets?

  3. Can the authors extend zero-poison control checks to detector-dependent and unlearning defenses, and state the assumptions required for TARE-Z to remain unbiased after installation, admissibility, and removal-success selection?

  4. How frequently do existing defense evaluations use a reporting baseline different from the actual poisoned checkpoint, and how large is the resulting drift term in Equation (5)?

Reviewer_ZwzYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper audits the clean-accuracy (CA) cost column that backdoor-defense benchmarks, mainly BackdoorBench, report beside attack-success reduction. Its claims are:

(1) Three of BackdoorBench's sixteen attacks (WaNet, BPP, Input-Aware) use a MultiStepLR schedule with milestones [100, 200, 300, 400] in a 100-epoch run, so the victim never anneals. These victims start roughly 2-3 points below the others on the public leaderboard.

(2) On PreAct-ResNet18, fine-tuning-family defenses (FT, NAD, FT-SAM, and SAU at a lower level) bring a model to a roughly start-independent accuracy "level". On under-trained victims the reported cost is therefore negative, and DER's max(0, .) clip hides this.

(3) A 2x2 crossover changes only the scheduler line. It reverses the sign of the cost a defense charges a never-poisoned twin in both directions: WaNet goes from -1.84 to +0.78 and BadNets from +0.50 to -2.08, with 131/132 cells agreeing. The result is replicated on BPP, FT-SAM, CIFAR-100, VGG19-BN, and induced in a second toolkit (BackdoorBox).

(4) TARE runs the identical defense on a "never-poisoned twin" (same recipe, schedule and seed; at most 10 poison images, admitted only below 5% ASR) and reports the twin's loss, the "tare", next to the reported cost. On the BadNets grid the tare ranges from +0.13 (FT) to +5.70 (I-BAU), and ABL destroys the twins. Paired and difference-in-differences contrasts are printed, but the authors state that the removal share is not identified and they correct nothing.

(5) The artifact is traced into the citing literature: re-running TSBD and CGD with their released code shows that a sizable part of their "CA gain over No Defense" on WaNet also appears on a never-poisoned twin.

Deliverables are a three-key config patch, a signed tare table (7 attacks x 8 defenses), and TARE-Z, a twin-free surrogate estimator. The paper is about 85 pages with 25 self-hosted pre-registrations.

Critical strengths
  1. A concrete, verifiable benchmark artifact, established causally rather than correlationally. The scheduler issue can be checked directly in the configuration files at the stated commit. The 2x2 design changes a single config line and flips the sign of the twin's cost in both directions. It holds seed by seed (44/44) and is reproduced in an independent toolkit, where the artifact is induced rather than found. This matters for my assessment because the artifact has real consequences:
  • negative costs are booked as gains and then clipped by DER;
  • cross-attack comparisons are biased;
  • at least two published "no-cost / CA-improving" claims (TSBD, CGD) partly rest on it, which the authors show by re-running the methods' own code.
  1. Unusual rigor and transparency in the empirical work. The paper uses installation-level replicates with intervals, reports registered predictions that failed as failed, discloses the post-hoc decisions, and documents its own earlier implementation errors (ANP, NC, fine-pruning masking, defender-data leakage). Whatever one thinks of the framing, this makes the numerical claims credible.
Critical weaknesses
  1. The writing is unclear, and the main text cannot be followed or verified on its own. I could not reconstruct the central argument from the main paper without repeatedly going to the appendix. This matters because the main paper should let a reader check the main claims. (a) Many terms appear abruptly in the abstract and introduction before they are defined or motivated: "installation", "level", "tare", "A0/A3/A4", "paired" vs. "DiD", "registered". (b) Sentences and paragraphs often lack transitions. For example, the first introduction paragraph moves from the scheduler setting, to the defense's level, to DER's clip, to citing papers, without stating how these steps relate. (c) Nearly every sentence carries intervals and appendix pointers, so the argument is hard to follow. The core message is short: a scheduler setting under-trains three victims; fine-tuning defenses turn this into an apparent gain; running the defense on a clean model exposes it. It could be stated much more directly. Material that is peripheral to this claim also dilutes it, namely the LLM, unlearning, pruning and threat-model sections (S2, S9-S15, S22-S23).

  2. The significance and novelty of the contribution are limited relative to its framing. (a) The control arm itself is not new, and the paper acknowledges this. Abad et al.'s SoK (https://arxiv.org/abs/2511.13143) already runs a no-attack arm and recommends this control, and Peng et al. and Tong et al. report backdoor-free drops. (b) By the paper's own account, the proposed tare column "re-orders nothing within an attack, by construction". The prototype-referenced DER re-ranks 0/28 pairs. The removal share is unidentified, and TARE "corrects nothing". So it is unclear which practitioner decision changes. What remains is attribution, cross-attack comparisons, and "better than No Defense" sentences, which the authors find decisive in 2 of 48 papers (single coder). (c) The scheduler setting may be partly intentional. The BackdoorBench 2022 paper (https://arxiv.org/abs/2206.12654, Table 4) lists these milestones for training-controllable attacks. Under that threat model an under-trained victim is a legitimate input, and the reported cost is the defender's true bill, as the authors concede in Sec. 4.2. The issue is then one of interpretation and comparability across attacks, not an error in the measurement. (d) The "fixed-point" accounting that explains the mechanism holds only for some defenses on PreAct-ResNet18. It fails for I-BAU, for VGG19-BN, and for fine-pruning on CIFAR-100. All runs are CIFAR-10/100. Taken together, the work reads more like a well-documented benchmark erratum plus a methodological recommendation than a general contribution of ICLR main-track scope. This would change if the authors stated a clear prescription that alters evaluation practice, and showed that it does.

Questions for authors

Clarification requests (could affect my assessment):

Q1. BackdoorBench's 2022 paper lists the [100, 200, 300, 400] schedule for these attacks. How do you reconcile your "artifact" framing with that stated design choice for training-controllable attacks? Apart from DER's clip and cross-attack comparisons, does the patch change any conclusion BackdoorBench itself draws?

Q2. If a benchmark adopted only one recommendation from this paper, which should it be: the three-key patch, a prototype-referenced DER (S40), or the tare column? Since within-attack rankings are unchanged by construction, what concrete evaluation decision does the tare column change?

Q3. Given that the removal share is not identified, how should a practitioner use the tare? Is the mixture (1-p) c_clean + p c_def of Sec. 4.6 the intended decision rule? Its rank reversals were found post hoc and did not replicate on WaNet or Blended.

Q4. The citation audit (2/48 papers decisive, 13 reproducing the artifact) was coded by a single author. What was the decision criterion for "decisive" versus "correlational", and how many of the 48 papers were borderline under that criterion?

Q5. The zero-image twins (S34) agree with the ten-image twins to within about 0.35 points and pass A4 15/15. Why not make the zero-image twin (with the guard bypassed) the primary definition? It would remove the A4 complications for Blended, BPP, SIG and Trojan.

Suggestions for future work (not affecting my score):

  • Run, rather than read off the leaderboard, at least one non-CIFAR dataset (e.g., Tiny-ImageNet, where the paper notes a different contrast) and one modern backbone (ConvNeXt or ViT).
  • Package the patch and a twin-run flag as a pull request to the benchmark, so the community can adopt it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.