- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper identifies makes several contributions:
- it identifies flaws in BackdoorBench -- that learning rate scheduling fails to be invoked for 3 attacks -- WaNet, BPP, and Input-aware -- with the consequence that victim models have very poor clean test accuracy. Subsequent fine-tuning defenses repair this poor accuracy. This leads to the (false) inference that there is no clean test set accuracy cost associated with applying the given defense (and even possibly a clean test accuracy benefit).
- The authors quantify the cost, in clean test accuracy, of applying a given defense, using the tare -- the difference in clean accuracy between that for a given model and that for a defense applied to the given model. The authors show that tare is nearly always positive, across different defenses.
- The authors demonstrate how important a careful learning rate schedule is for the clean test accuracy -- the tare reverses with the scheduler, not with the attack.
- The authors also demonstrate the importance of annealed scheduling for backdoor removal.
Critical strengths
Proposes the tare, a very sensible statistic added to the BackdoorBench backdoor mitigation leaderboard.
Identifies a problem with the scoring used for several backdoor attacks (WaNet, BPP, and Input-Aware) -- these methods were not "shipped" with a learning rate scheduler, which causes the data-poisoned models to have very low clean test accuracy baselines.
Identifies that the ABL defense grossly degrades clean test set accuracy.
Proposes TARE-Z, a surrogate of the tare that does not require seed matching.
Critical weaknesses
This paper is a bit too difficult to read. The authors use too much jargon (such as "charge the twin").
They also give way too much minutiae/technical details in both the abstract and in the Introduction -- the paper clarity could be improved.
The paper should make a better separation between their discovery of a bug in BackdoorBench (relating to MultStepLR) from the failure of this leaderboard to account for the defense's effect on clean accuracy of a model.
In Intro.: should define the tare before referring to it.
Sec. 2: "the other column of the table" -- which column is that ??
"describe the non-linear metric artifact DER's max(0,.) is" -- this is not a valid sentence.
Sec. 4.2: A0 used but not yet defined.
Questions for authors
What does DiD stand for ?
In Q(D_b Theta) = L_D + espsilon(Theta) is the function epsilon allowed to be different for different defenses ? If this is the case, it seems this decomposition can be trivially achieved, in which case "Whether a defense has such a measure" would be moot.