acceptodds
Under review as a conference paper at ICLR 2027

Statistic Preservation Does Not Identify Attack-Suppression Preservation in Composed Federated-Learning Defences

Abstract

Federated learning composes defences never designed together. Does the downstream defence still suppress the attack? **A preserved statistic is not preserved suppression.** The natural answer asks whether the downstream defense's own statistic is preserved under composition, and it fails structurally rather than statistically. Admitting a composition because one constituent already suppresses the attack selects on the outcome to be explained. The counterfactual identification required is then absent, so the comparison is missing at any . The fix is a design, not a better statistic: hold the downstream defence and attack fixed, and intervene upstream. Ignoring this costs not power but the sign. On one `coord_median` cell, the outcome-gated design moves where the within-defence one moves (, both intervals exclude zero). Against ourselves: with the aggregate magnitude held fixed, the is , and both agree in sign. The reversal replicates on the same cell on CIFAR-100; a composition-level replication there was attempted and did not carry. Decision change is neither necessary nor sufficient for suppression to move. One aggregator's selection flips repeatedly while admitting no adversary; another's is bit-identical while attack success falls. Oracle-free, a Krum decision flip alone raises ASR by , so both need an oracle no server has. This is identification, deliberately scoped to committed attacks and controlled interventions, and not a deployable security test or a predictor of adaptive robustness.

Primary area:
causal reasoning
Keywords:
federated learning, data poisoning, backdoor attacks, composed defences, robust aggregation, causal identification, evaluation methodology, selection on the outcome, invariance analysis, pre-registration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bPeeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies composed FL defenses, in which an upstream per-client transform d1 is applied before a downstream robust aggregator d2. It asks whether d1 preserving the statistic that d2 reads implies that the composition still suppresses the attack, and argues that it does not.

On the formal side, the paper considers a screen that certifies a pair only when one constituent already suppresses the attack on its own and d1 does not. It shows that such a screen never observes a d2 that fails alone, so cross-defense comparisons cannot attribute the composition's suppression to the preserved mechanism. As a remedy it proposes a within-defense design: fix d2 and the attack, intervene on the upstream transform, and always run the identity transform as a control. It also defines a five-level preservation hierarchy (statistic, ordering, decision, admission, suppression) and characterizes which aggregator statistics are invariant under positive per-client rescaling.

Empirically, the experiments mainly use CIFAR-10 with a small CNN, 10 clients and mostly 5 seeds. The authors apply a synthetic upstream rescaling of increasing dispersion, together with an instrument (Mode S) that uses knowledge of which clients are adversarial to pin their coefficients. They report that decision changes and ASR changes can dissociate: Krum's selection changes while ASR does not, and cos_krum's selection is unchanged while ASR falls. On one coordinate-median cell, the two designs give opposite signs (−0.273 vs +0.125). The paper also reports several results against its own claims. When aggregate magnitude is controlled, the sign reversal disappears. Variants that do not use adversary identity contradict the Krum result. A pre-registered rule for when the two designs disagree fails out of sample.

Critical strengths

The paper points out that in composed-defense evaluation, an upstream per-client transform can change ASR simply by down-weighting the adversary, without acting through the downstream defense's statistic. In the paper's own ladder, the adversarial coefficient share drops from 0.267 to 0.187. An evaluation that ignores this can credit the downstream defense with an effect the upstream transform produced. Together with the characterization of which aggregator statistics are invariant under per-client rescaling, this is a useful caution for anyone evaluating stacked FL defenses, and it is the most transferable part of the paper.

Critical weaknesses

Weakness 1: The central message is largely expected. A defense's statistic is only the first step before selection and aggregation, so preserving it cannot be expected to preserve suppression, and the paper itself acknowledges that its collider-bias argument is textbook. The proposed remedy, an identity control with the downstream defense fixed, is ordinary ablation practice; by the authors' own audit, 46 of 59 composed-defense papers already run such a contrast. This is decisive because the message and the remedy are the paper's main contribution.

Weakness 2: The paper's own experiments do not support its claim. Much of the text covers earlier criteria, pre-registration amendments and withdrawn claims. The headline "outcome-gated" contrast only compares two ladders that differ in whether adversarial coefficients are pinned, so it does not test the identification argument. The sign reversal disappears once aggregate magnitude is controlled (+0.125 becomes −0.172), the oracle-free runs contradict the Krum result (+0.297 and +0.322), and what remains rests on a small CNN with 10 clients and a synthetic coefficient spread. Without this evidence, the paper reduces to the argument in Weakness 1.

Questions for authors
  1. Can you point to published work that follows the criticized practice, or show an experiment where it leads to a wrong conclusion?
  2. In what sense is the "outcome-gated" ladder gated on an outcome, and how does it correspond to the gate in Prop. 1?
  3. Given the magnitude-controlled and oracle-free results, does the abstract's sign-reversal claim still hold, and which claims in Table 1 remain?
  4. Do the dissociations hold with real upstream defenses and a larger setting?

Suggestion: move the pre-registration and revision history to the supplement and keep the main text focused on the core argument.

1 comment

Thank you. Your review changed the paper's thesis.

W1 (expected message; ablation is standard). We agree. The identification argument (Prop. 2) is standard causal reasoning, and holding d2 fixed is common practice; §3 and §6 now say so. The new claim is the next step: that ablation is itself confounded. A per-client transform reaches the aggregate twice, and the ablation reports only the sum. The two channels can have opposite signs, and either can dominate:

  • coordinate-wise median: −0.172 statistic, +0.297 weight;
  • RFA→Krum: +0.297 statistic, −0.017 weight;
  • FoolsGold→coordinate-wise median (new, deployed): +0.230 statistic, −0.570 weight;
  • median-norm clip→coordinate-wise median at N=100 (new): +0.002 statistic, −0.032 weight.

An evaluator reading the ablation cannot tell which case occurred. We supply the control that separates the channels without labels, and the invariance result (Cor. 1) that says when the statistic channel is exactly closed. Cor. 1 explains cos-Krum: its selection is bit-identical in every round, yet ASR falls by 0.173.

W2 (experiments contradict the claim). They contradicted the claim as stated, so we changed the claim.

  • The magnitude control is now the headline: +0.125 = −0.172 + 0.297.
  • The oracle-free Krum results are the statistical channel carrying the effect.
  • The pre-registration history has moved to the supplement (§S25), as you suggested.

Q1. Published practice, or a wrong conclusion? Both.

  • Published practice: our audit (supp. §S16, one coder, arXiv-heavy frame) finds 6 of 59 composed-defence papers crediting a constituent's mechanism with no contrast that holds the downstream defence fixed. Examples are arXiv 2307.00356, 2310.13403, 2502.07011 and 2507.16134.
  • Wrong conclusion: the common ablation leads to one on our headline cell. It says the reweighting weakens the coordinate-wise median (+0.125). Through the median's own selection, the reweighting strengthens it (−0.172, negative on 19 of 20 seeds).

Q2. "Outcome-gated." The label was our error. That ladder lets the adversary's coefficient move (its share falls 0.267→0.187), and nothing is selected on an outcome. Prop. 2 concerns cross-defence comparisons that admit pairs because one constituent already works. The ladder is now described as what it is, and it is no longer presented as a test of the proposition.

Q3. The abstract claim and Table 1. The sign-reversal claim is removed. Of the submitted Table 1:

  • Kept: rows 1–4 (the propositions, the invariance results and the counterexamples).
  • Withdrawn: rows 6–7.
  • Changed meaning: row 5, which is now evidence that the statistic channel can carry the effect.

The revised Table 1 lists the new claims.

Q4. Real defences and scale. For deployed defences, we have RFA→Krum (n=20) and a new pre-registered arm of eight cells (§4.3, Table 4), none reading labels. FoolsGold→coordinate-wise median reproduces the opposite-sign split (+0.230 statistic, −0.570 weight), and reputation→Krum is carried by the statistic channel. At N=100, we ran a new pre-registered split (n=10; general comment, item 5). The opposite-sign split of the headline cell does not carry over, and §4.2 says so. Median-norm clipping there acts only through the weight channel, so the ablation misattributes its effect.

Reviewer_fkfAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether preserving a downstream federated-learning defense’s statistics or decisions also preserves attack suppression. It introduces a five-level preservation hierarchy (P1–P5) and proves that outcome-gated evaluation can prevent causal identification and overlook emergent defense synergy. To address this, it proposes controlled upstream interventions with fixed downstream defenses and attacks. Theoretical analysis characterizes preservation under positive rescaling, while experiments demonstrate that statistic or decision preservation does not necessarily imply attack-suppression preservation.

Critical strengths
  1. The preservation hierarchy and the propositions on structural non-identification and emergent synergy turn an often informal evaluation mistake into precise claims. Theorem 9 and Proposition 11 further explain when rank/geometric aggregation or cosine/coordinate-rank/consensus-distance statistics are invariant, which is useful beyond the motivating examples.
  2. The authors distinguish formal claims, committed empirical claims, oracle-assisted experiments, and deployment recommendations, and they report predictions that were refuted or withdrawn. The fixed-downstream/committed-attack protocol and the separate measurements of decision, admission, influence, and ASR are methodologically valuable.
Critical weaknesses

1.The proposed measurement protocol is not directly deployable in the main use case. Mode S requires knowing which clients are malicious and fixing their coefficients; parts of Mode A alter internal aggregation outputs or need score-level access. The oracle-free attempts do not provide the same intervention and in some cases refute the earlier prediction. Thus the paper establishes an important boundary for oracle-assisted causal measurement, but it does not yet provide a server-executable way to estimate P4/P5 in an ordinary FL deployment. 2. The empirical validation is narrow and has limited screening power. The central interventions use few seeds (often n=3n=3 or 55), a small set of datasets/architectures and committed attacks, and a 42-pair screen in which the constant-HIGH rule already performs well. The reported screen recall is about 40%, and the criterion cannot detect emergent synergy by its construction; several preregistered predictions were refuted or withdrawn. These facts do not invalidate the theory, but they limit claims that the screen is a reliable general-purpose predictor. The long presentation also makes it easy to overread an empirical illustration as a deployment method.

Questions for authors
  1. Is the intended contribution an oracle-assisted causal measurement protocol, or a tool that a normal server can execute without malicious-client labels or access to aggregator internals? If the former, please narrow the practical claims and state the exact estimand; if the latter, what concrete server-side intervention identifies P4/P5 without changing the defense?
  2. How were the C0/C1 thresholds (including the ASR and clean-accuracy cutoffs) and the time at which C0 enters the screen fixed without using outcomes from the screened cells? Please report sensitivity of precision/recall, sign reversals, and equivalence claims to these thresholds.
  3. Can the authors release the per-seed raw measurements and runnable scripts, and repeat the central checks with more clients, additional genuinely composable defenses, non-committed/adaptive attacks, and higher participation? Please separate effects caused by changing an upstream statistic from effects caused by changing the malicious-update dose.
1 comment

Thank you for the precise questions. They shaped the revision.

Q1. Oracle-assisted or deployable? Estimand?

The method is an evaluation methodology for settings where ASR can be measured, such as simulation, benchmarks and red-teaming. It is not a run-time detector, and §1 and §7 now say so. Write d2(V) = emit(V, sel(V)), with d2 and the attack fixed (Def. 1):

  • Δ_total = ASR[d2(T1 U)] − ASR[d2(U)], the standard ablation;
  • Δ_stat = ASR[emit(U, sel(T1 U))] − ASR[d2(U)];
  • Δ_weight = Δ_total − Δ_stat.

The split is exact per seed. It is path-specific, and interactions are assigned to the weight channel. The control needs only U and T1(U), not labels. The synthetic transform (which pins the adversary's coefficient) does use identity, and the paper now calls it an instrument. An oracle-free split was already in the submission: RFA→Krum (n=20) gives Δ_stat = +0.297 [+0.098, +0.496] and Δ_weight = −0.017 [−0.059, +0.025] (Table 3, App. C.7). That is the opposite pattern to coordinate-wise median, which is why both channels must be reported.

Q2. How were the C0/C1 thresholds fixed?

The ASR threshold of 0.5 was chosen while the development cells' outcomes were visible. It was then frozen in a pre-registration committed before any run of the prospective suite (262cf35, 2026-08-20; suite results committed 2026-08-27; supp. §S24.1). The development-set precision and recall are therefore not out-of-sample. The revision removes the criterion from the paper's claims, so no claim depends on these thresholds. The sensitivity of the intervention results to their margin and accuracy floor is in supp. §S13.

Q3. Release, scale, defences, attacks, statistic vs. dose.

  • Statistic vs. dose: this is now the central measurement (Tables 2 and 3).
  • Deployed defenses: eight pre-registered cells with no labels (§4.3, Table 4; general comment, item 4), plus RFA→Krum. FoolsGold→coordinate-wise median splits into +0.230 statistic and −0.570 weight.
  • Adaptive adversary: against an adversary adapting to d2, the two designs still disagree (−0.324 [−0.543, −0.105], App. C.3). A composition-aware attack raises FoolsGold→RFA from 0.064 to 0.157 (n=30, supp. §S19).
  • Scale: a new pre-registered N=100 arm (K=20, n=10; general comment, item 5). Which channel dominates changes with scale, and a deployed clip acts only through the weight channel. §4.2 reports both transforms.
  • Release and per-seed values: https://anonymous.4open.science/r/fl-channels-17164/.

We agree that 40% recall makes the criterion a poor predictor. It no longer appears in the paper as one.

Reviewer_Fw9WOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

As far as I can understand, this paper studies how to evaluate compositions of defenses in federated learning, where an upstream per-client transform is applied before a downstream robust aggregator. The authors argue that checking whether the downstream defense’s statistic is preserved, combined with selecting compositions based on their outcomes, cannot establish whether the downstream mechanism is responsible for suppressing the attack. Instead, they propose a within-defense design: fix the downstream defense and the attack, and intervene on the upstream transform. The paper also defines several levels of “preservation” and reports experiments suggesting that preservation at intermediate levels does not determine attack suppression. I should note that I had considerable difficulty following the paper, so this summary reflects my best understanding and may miss some of its points.

Critical strengths

The paper raises a meaningful question: how to correctly attribute the effect of composed defenses in federated learning. And the authors appear careful about the status of their claims. They distinguish formal from empirical results and report predictions that did not hold.

Critical weaknesses
  1. The paper is very hard to follow. Many sentences pack several clauses and qualifications together, and I could not make sense of some of them even after rereading. As a result, it is hard to judge whether the arguments are correct.

  2. The paper uses a large number of its own terms, and some of them are never clearly defined. The same concept sometimes goes by different names, and method names are not used consistently, so I often had to guess what a term meant from context.

  3. The abstract and introduction report many numerical results and technical terms before the problem setting and key concepts have been explained. It would be much easier to grasp the main idea if the paper first laid out the motivation and intuition and then presented the results.

  4. The main text is not self-contained. Many key steps of the argument are stated only as short conclusions with a pointer to the appendix, and important details are scattered across the paper. It is hard to follow the core argument from the main text alone.

Questions for authors

Could you explain the paper’s main contribution in a few plain sentences, and clarify what practical guidance it offers for evaluating composed defenses?

1 comment

Thank you. We agree that the submission was hard to follow, and we rewrote the main text around your question.

Main contribution in plain sentences.

  1. A defence stack such as clipping followed by a robust aggregator can change attack success in two ways. It can change what the aggregator selects, or it can change how much each selected update weighs.
  2. The usual evaluation reports the sum of the two, and they can point in opposite directions.
    • On CIFAR-10 with coordinate-wise median, the usual ablation says a reweighting raises backdoor success by 0.125.
    • Through the aggregator's selection alone, the same reweighting lowers it by 0.172.
    • The rise is entirely re-weighting, +0.297.
  3. We give a control that separates the two without knowing which clients are malicious. We also give theory saying that when the selection effect is exactly zero, cosine-based aggregators under positive rescaling (Cor. 1).

Practical guidance. §5 is now a five-rule checklist:

  1. Hold the aggregator and attack fixed, and always run the no-transform control.
  2. Run the score-only control and report both channels.
  3. Check the aggregator's invariance class first.
  4. In simulation, report the adversary's weight in each arm.
  5. Do not read a preserved statistic as preserved robustness. Krum's selection changes in 73% of rounds with no ASR change, while cos-Krum's never changes, yet ASR falls (Table 3).

Readability. The fixes, point by point:

  • Overloaded sentences: one claim per sentence.
  • Terms: the body introduces two (statistic and weight channel) plus the preservation levels, each defined before use. The appendix's older names are mapped in one paragraph at its start.
  • Results before setting: the abstract and §1 now run: problem, channels, one example, results.
  • Self-contained body: definitions, statements with proof sketches (§3), the decomposition tables, a figure of both channels across all fully measured cells (Fig. 2) and related work are in the body. The appendix holds full proofs and per-seed detail.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Meta response to all reviewers

We thank the reviewers. All three found the paper hard to follow, and bPee showed that the headline sign reversal does not survive our own magnitude control. We agree with both points and have uploaded a substantially rewritten version of the paper. The rewrite is large, so we list the changes instead of highlighting them:

  1. New thesis, built from evidence already in the submission. A per-client upstream transform changes attack success (ASR) through two channels. In the statistic channel it changes what the robust aggregator selects. In the weight channel it changes how much the selected updates count. The standard ablation (aggregator alone vs. with the transform) reports only their sum. On the headline cell the ablation's +0.125 is −0.172 through the statistic channel plus +0.297 through the weight channel, n=20 (Table 2; submitted App. D.10). The sign-reversal claim is withdrawn. We changed the title and abstract to match. The new title is "Statistic and Weight Channels in Composed Federated-Learning Defences"; the submitted one was "Statistic Preservation Does Not Identify Attack-Suppression Preservation in Composed Federated-Learning Defences".
  2. A label-free control. Selecting on transformed updates and emitting raw ones separates the channels (Def. 1). It needs no adversary identity.
  3. A rewritten main text.
    • Two new terms, each defined before use.
    • Setting before results.
    • Key statements, the checklist (§5) and related work are now in the body.
    • The screening criterion is out of the main text, and the pre-registration history (evidence tiers, failed predictions, the evidence ledger) has moved to a new supplement §S25. The PDF went from 70 to 58 pages.
    • The body ends at the top of page 9, within the 10 pages allowed. The extra space goes to proof sketches in §3 and a new Fig. 2 that plots both channels for every fully measured cell.
  4. A new pre-registered experiment. It runs the same split under deployed defences: median-norm clipping, reputation weighting and FoolsGold, each into coordinate-wise median, Krum and cos-Krum, with n=10 and no labels, pre-registered before any run (§4.3, Table 4). FoolsGold→coordinate-wise median shows the headline's opposite signs with no oracle: the ablation's −0.340 [−0.508, −0.172] is +0.230 [+0.158, +0.302] through the statistic channel (positive on all 10 seeds) and −0.570 [−0.740, −0.400] through the weight channel. Reputation→Krum is carried by the statistic channel (+0.451 of +0.495). Four cells show no channel at n=10, and FoolsGold→Krum falls below the accuracy floor. Our prior that norm clipping→coordinate-wise median would show the largest weight channel was wrong.
  5. A second new pre-registered experiment, at N=100 with 20 clients per round, n=10 (§4.2, Table 3, App. C.4). The N=10 headline split does not carry over. Under the synthetic transform, the statistic channel carries the effect (Δ_stat −0.202 [−0.318, −0.086], Δ_weight +0.037 [−0.080, +0.154]). Under median-norm clipping, which needs no labels, the ablation's −0.030 [−0.042, −0.019] is all weight channel (−0.032 [−0.050, −0.015]; Δ_stat +0.002 [−0.012, +0.016]). The effect is small, but here the ablation credits the median for something its selection did not do.
  6. An updated AI use statement, and an anonymous code and artifact release: https://anonymous.4open.science/r/fl-channels-17164/.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.