- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies composed FL defenses, in which an upstream per-client transform d1 is applied before a downstream robust aggregator d2. It asks whether d1 preserving the statistic that d2 reads implies that the composition still suppresses the attack, and argues that it does not.
On the formal side, the paper considers a screen that certifies a pair only when one constituent already suppresses the attack on its own and d1 does not. It shows that such a screen never observes a d2 that fails alone, so cross-defense comparisons cannot attribute the composition's suppression to the preserved mechanism. As a remedy it proposes a within-defense design: fix d2 and the attack, intervene on the upstream transform, and always run the identity transform as a control. It also defines a five-level preservation hierarchy (statistic, ordering, decision, admission, suppression) and characterizes which aggregator statistics are invariant under positive per-client rescaling.
Empirically, the experiments mainly use CIFAR-10 with a small CNN, 10 clients and mostly 5 seeds. The authors apply a synthetic upstream rescaling of increasing dispersion, together with an instrument (Mode S) that uses knowledge of which clients are adversarial to pin their coefficients. They report that decision changes and ASR changes can dissociate: Krum's selection changes while ASR does not, and cos_krum's selection is unchanged while ASR falls. On one coordinate-median cell, the two designs give opposite signs (−0.273 vs +0.125). The paper also reports several results against its own claims. When aggregate magnitude is controlled, the sign reversal disappears. Variants that do not use adversary identity contradict the Krum result. A pre-registered rule for when the two designs disagree fails out of sample.
Critical strengths
The paper points out that in composed-defense evaluation, an upstream per-client transform can change ASR simply by down-weighting the adversary, without acting through the downstream defense's statistic. In the paper's own ladder, the adversarial coefficient share drops from 0.267 to 0.187. An evaluation that ignores this can credit the downstream defense with an effect the upstream transform produced. Together with the characterization of which aggregator statistics are invariant under per-client rescaling, this is a useful caution for anyone evaluating stacked FL defenses, and it is the most transferable part of the paper.
Critical weaknesses
Weakness 1: The central message is largely expected. A defense's statistic is only the first step before selection and aggregation, so preserving it cannot be expected to preserve suppression, and the paper itself acknowledges that its collider-bias argument is textbook. The proposed remedy, an identity control with the downstream defense fixed, is ordinary ablation practice; by the authors' own audit, 46 of 59 composed-defense papers already run such a contrast. This is decisive because the message and the remedy are the paper's main contribution.
Weakness 2: The paper's own experiments do not support its claim. Much of the text covers earlier criteria, pre-registration amendments and withdrawn claims. The headline "outcome-gated" contrast only compares two ladders that differ in whether adversarial coefficients are pinned, so it does not test the identification argument. The sign reversal disappears once aggregate magnitude is controlled (+0.125 becomes −0.172), the oracle-free runs contradict the Krum result (+0.297 and +0.322), and what remains rests on a small CNN with 10 clients and a synthetic coefficient spread. Without this evidence, the paper reduces to the argument in Weakness 1.
Questions for authors
- Can you point to published work that follows the criticized practice, or show an experiment where it leads to a wrong conclusion?
- In what sense is the "outcome-gated" ladder gated on an outcome, and how does it correspond to the gate in Prop. 1?
- Given the magnitude-controlled and oracle-free results, does the abstract's sign-reversal claim still hold, and which claims in Table 1 remain?
- Do the dissociations hold with real upstream defenses and a larger setting?
Suggestion: move the pre-registration and revision history to the supplement and keep the main text focused on the core argument.
Thank you. Your review changed the paper's thesis.
W1 (expected message; ablation is standard). We agree. The identification argument (Prop. 2) is standard causal reasoning, and holding d2 fixed is common practice; §3 and §6 now say so. The new claim is the next step: that ablation is itself confounded. A per-client transform reaches the aggregate twice, and the ablation reports only the sum. The two channels can have opposite signs, and either can dominate:
An evaluator reading the ablation cannot tell which case occurred. We supply the control that separates the channels without labels, and the invariance result (Cor. 1) that says when the statistic channel is exactly closed. Cor. 1 explains cos-Krum: its selection is bit-identical in every round, yet ASR falls by 0.173.
W2 (experiments contradict the claim). They contradicted the claim as stated, so we changed the claim.
Q1. Published practice, or a wrong conclusion? Both.
Q2. "Outcome-gated." The label was our error. That ladder lets the adversary's coefficient move (its share falls 0.267→0.187), and nothing is selected on an outcome. Prop. 2 concerns cross-defence comparisons that admit pairs because one constituent already works. The ladder is now described as what it is, and it is no longer presented as a test of the proposition.
Q3. The abstract claim and Table 1. The sign-reversal claim is removed. Of the submitted Table 1:
The revised Table 1 lists the new claims.
Q4. Real defences and scale. For deployed defences, we have RFA→Krum (n=20) and a new pre-registered arm of eight cells (§4.3, Table 4), none reading labels. FoolsGold→coordinate-wise median reproduces the opposite-sign split (+0.230 statistic, −0.570 weight), and reputation→Krum is carried by the statistic channel. At N=100, we ran a new pre-registered split (n=10; general comment, item 5). The opposite-sign split of the headline cell does not carry over, and §4.2 says so. Median-norm clipping there acts only through the weight channel, so the ablation misattributes its effect.