- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Distribution-Specific Indistinguishability (DiSI), a noise-free framework for reducing the influence of potentially malicious or sensitive training data. It uses randomness in benign dataset and quantizes per-source gradients, replacing coordinate-wise averaging with thresholded majority vote or maximal-gap operators that abstain when the votes are ambiguous. Under assumptions on benign source independence and population vote gaps, each coordinate update matches its clean counterpart except with probability exponentially small in the number of benign sources. A bound over coordinates and training steps then transfers high probability properties of a safe reference model to the model trained with the selected data. Experiments on image and text backdoors, outlier memorization, copyright image similarity, and membership inference report better performance.
Critical strengths
-
Instead of injecting algorithmic noise like DP and DaSI, the authors exploit randomness in benign training data to obtain high agreement probability between training on the full dataset and on a reference dataset that excludes the malicious data. Because this agreement is not tied to a particular risk event, the same argument can transfer the reference’s guarantees across backdoors, outlier memorization, copyright image similarity, and membership inference.
-
For both TMV and TMGC, the paper identifies when malicious votes can change a coordinate update and gives conditions that make such cases unlikely. It also explains the role of the threshold, extends the results to sources with different distributions and replaced votes, and shows how extra data can be used to check parameter choices.
-
The evaluation covers backdoor and data poisoning defense in image and language models, memorization mitigation, copyright protection, and membership inference. Across these tasks, DiSI generally performs better than standard training on the reported risk measures. The image backdoor results are especially strong: DiSI keeps attack success low while retaining much higher clean accuracy than DP-SGD and DaSI.
Critical weaknesses
-
There are several baselines that are worth considering. FLCert (Cao et al., IEEE TIFS 2022) follows a similar idea. It splits clients into groups and takes a majority vote over the models trained on them, which gives a provable guarantee against a bounded number of malicious clients. It should be compared on the classification experiments. RLR (Ozdayi et al., AAAI 2021) is a standard baseline for backdoor defense in federated learning, and it also aggregates the signs of client updates in each coordinate. SignSGD with majority vote (Bernstein et al., ICLR 2019) is the special case of TMV with two quantization levels and =0. The authors already cite it and analyze it in Appendix F, so it is a natural ablation.
-
As the authors mentioned in Section 3.3, the certificate is not satisfied in real experiments, which results in the high inconsistency rates(1.69% for , on ResNet-20). The evidence of robustness is therefore entirely empirical, yet all evaluated attacks are standard data poisoning attacks that do not target DiSI. An adversary aware of the operator could exploit exactly these disagreeing coordinates: under TMV it can choose which benign majority updates are applied, and under TMGC it can inject its own values in coordinates where the votes are spread out. The authors should evaluate at least one adaptive attack of this kind.
-
Four entries of Table 1 are trained with the loss of ABL (Li et al., 2021a) instead of cross entropy. Neither DiSI without ABL nor ABL alone is reported. In particular, on CIFAR-100 both operators rely on ABL for the clean-label SIG attack, so the effectiveness of DiSI alone against SIG on CIFAR-100 is not established. The authors should report the results of DiSI without ABL and of ABL alone for these entries, and explain why ABL was needed.
Questions for authors
-
In the combined experiments, adding DiSI to an existing defense sometimes yields a higher ASR than DiSI alone (e.g., on CIFAR-10 BadNets, 58.94% for CLP+TMV in Table 7 vs. 1.79% for TMV in Table 1; on the refusal task, 55.7–59.2% for CROW+DiSI in Table 6 vs. 20.3–30.4% for DiSI in Table 2). How were the combined runs set up, for example, the order in which the two defenses are applied and whether the DiSI configuration matches the main experiments, and what causes this difference?
-
On the refusal task in Table 2, the original model has a refusal rate of 0.00 without the trigger (Table 6) but 100.00 under BadNets, 0.00 under Sleeper, and exactly 50.00 under the other three attacks with the trigger. What does the “Original” row refer to, and how is ASR computed?
7 comments
W1: FLCert, RLR, and Majority-Vote SignSGD (continued)
(3) What is protected. FLCert protects the predicted class label of one test input at a time, using an ensemble of separately trained models. DiSI protects the output law of a single trained model, hence any event of it (Lemma 2.3), which is what allows it to address generative fine-tuning, memorization, and membership inference, where there is no label to vote on.
The setting is correspondingly more demanding. FLCert needs one label to be stable, after training, when the realized gap permits. DiSI needs every coordinate of every update of one training run to agree with the reference run, for an unknown distribution, an adversary who chooses its data after seeing the benign data, and a trust event that is a black box. The two reported quantities are therefore not comparable: FLCert’s certified accuracy at ten clients (65.77%, 67.55%, 65.93% for BadNets, Blended, LF) is the fraction of test inputs whose realized gap happens to suffice, whereas ∆ bounds the change in probability of any event of the model and, as Section 3.3 states, needs more sources to be small.
SignSGD and RLR. Majority-vote signSGD proves a convergence rate under unimodal symmetric gradient noise against blind adversaries; it makes no claim about the adversary’s influence on the trained model. RLR is an empirical defense supported by a conjecture about how backdoors are implanted and by a convergence analysis; it gives no guarantee on the backdoor. Neither abstains: both always apply an update.
Empirically, both DiSI operators have lower ASR and higher clean accuracy than all three baselines on all three attacks, with a single model. FLCert, RLR, and SignSGD are now discussed in the related work, and the new rows are added to Table 8.
W2: Adaptive Attacks
We agree that the attacks in Table 1 do not target DiSI, and we now evaluate one that does. We first state what the theory covers and then describe the attack.
What the theory covers. The analysis is stated against the strongest adversary. In Section 3 the adversarial votes are arbitrary functions of the benign votes, and the local inconsistency rate of Section 3.2 is
the probability that the benign histogram can be changed by some placement of the adversarial votes. The guarantee therefore holds against an adversary who sees the benign data and then submits any gradient it likes, which is more than any data-poisoning attack can do. The guarantee is quantitative in the number of sources: it is informative with a few hundred (Section 3.3) and vacuous with ten, where, as the reviewer observes, the evidence is empirical.
The adaptive attack. We keep the threat model of Section 4 and of every attack in Table 1, data poisoning: the adversary controls the data of its source, and its vote is the gradient of that data. Within this model we give the adversary everything else. It knows the operator and , it sees the model at every step, and it sees the benign samples of the current step before choosing its own, so it re-crafts its data at every step (which is stronger than the fixed set A of Definition 2.1).
At each step the adversary computes the benign votes, finds the coordinates on which a single vote can change the operator’s output (the “disagreeing coordinates” the reviewer refers to), and determines for each the vote that moves the output toward the backdoor: it reinforces the leading bin when that bin helps the backdoor, and votes for the runner-up to force an abstention or a flip when it does not; under TMGC it votes into or next to the leading cluster in the same way. It then optimizes -bounded perturbations of its poisoned minibatch by gradient matching (Geiping et al., 2021; Souri et al., 2022) so that the minibatch gradient lands in those bins on the vulnerable coordinates and stays aligned with the backdoor gradient elsewhere, and submits the gradient of the crafted images.
Table 2. BadNets on CIFAR-10 with one malicious source
Standard poisoning versus the operator-aware adaptive attack (clean accuracy / ASR, %).
| Operator | Attack | |
|---|---|---|
| TMV | Standard | 81.24 / 96.15 |
| TMV | Adaptive, | 77.44 / 4.36 |
| TMGC | Standard | 81.57 / 96.83 |
| TMGC | Adaptive, | 79.58 / 2.06 |
At , the operator-aware adaptive attack () reduced ASR relative to standard poisoning from 96.15% to 4.36% under TMV and from 96.83% to 2.06% under TMGC. Clean accuracy decreased from 81.24% to 77.44% under TMV and from 81.57% to 79.58% under TMGC.
Scope. An adversary that is not bound to data and submits an arbitrary vector after seeing the benign votes (model poisoning) is the adversary of the theory. Against it the protection is the bound itself, which requires the source counts of Section 3.3; with ten to fifty sources we make no empirical claim of robustness against such an adversary, and Section 5 now says so.
Acknowledgments and W1
We sincerely thank Reviewer PD3h for the constructive suggestions and for catching the error in Table 2. We ran the requested baselines, an adaptive attack, and the ABL ablation; results are below and in the revised PDF.
4.1 W1: FLCert, RLR, and Majority-Vote SignSGD
We evaluated all three on CIFAR-10 under BadNets, Blended, and label flipping (LF) in the setting of Table 1 (ten sources, one malicious, 300 epochs). For FLCert, we used the disjoint-group variant (FLCert-D), partitioning the ten sources into N=5 disjoint groups of two sources each and certifying against one malicious client.
Table 1: Clean accuracy (Acc.) and attack success rate (ASR) on CIFAR-10, in percent. Bold marks the best result in each column.
Why SignSGD and RLR fail empirically. SignSGD and RLR both fail on the two trigger-based attacks (ASR above 94%). SignSGD is, as the reviewer notes, TMV with two levels and τ = 0, so this row is the ablation of the threshold: without it, the vulnerable band is the set of near-ties [0, c], which is where weak-signal coordinates concentrate, and one malicious vote decides them. RLR does threshold sign agreement, but on coordinates that pass it applies the average update, so the malicious source still contributes its magnitude there. TMV and TMGC output a value determined by the bin counts alone and abstain otherwise. The same pattern holds for coordinate-wise median and trimmed mean, which leave ASR above 99% in this setting (Table 8): estimation-robust aggregators do not remove the backdoor.
FLCert’s guarantee and ours are different statements. The reviewer notes that FLCert follows a similar idea and comes with a guarantee. Both methods vote, but what is proved differs on three points, and this difference is the core of our contribution.
(1) What the probability is over. FLCert takes the clients and their data as fixed. For a test input, its “label probability” is the fraction of client groups whose model predicts a given label. In FLCert-P the only randomness is the server’s own random sampling of groups, and the probabilistic statement is that the Monte Carlo estimate of the certified level is wrong with small probability; FLCert-D is deterministic. Neither is a statement over a data distribution: nothing is said about how often the vote gap is large for the data or inputs one will encounter. DiSI’s probability is over the benign data, for an unknown D. Theorems 3.2 and 3.3 bound, before training, the probability that an adversary can change the outcome.
(2) A certificate for the favorable case versus consistent behavior in every case. FLCert proves: if the gap between the largest and second-largest label counts for a test input is large enough, a bounded number of malicious clients cannot change the majority label. It certifies the inputs for which majority vote happens to be safe. When the gap is small the certified level is zero, the ensemble still outputs the majority label, and that label is unprotected. DiSI’s operator has two outputs, the majority value or ⊥, and the guarantee is that the choice between them and the value returned are the same with and without the adversarial sources, except with probability ∆. The uncertain case is handled by the mechanism, by abstaining, instead of being left outside the guarantee.