acceptodds
Under review as a conference paper at ICLR 2027

When to Intervene? State-Aware Sparse Manipulation in Federated Reinforcement Learning

Abstract

Federated reinforcement learning (FRL) enables distributed agents to collaboratively train decision-making policies, but its decentralized training process also exposes global policy learning to Byzantine manipulation. Existing poisoning attacks primarily focus on how to construct malicious updates, while trajectory-level intervention timing remains largely implicit. In sequential decision making, however, where an intervention is applied can alter subsequent trajectories and learning signals. Through controlled experiments, we find that changing the selected trajectory states materially alters attack efficacy even when the malicious-update construction is fixed. We therefore identify when as a distinct attack dimension and introduce the Viability-constrained Behavioral Steering Attack (V-BSA), which uses local policy uncertainty to select sparse intervention states and applies envelope-constrained behavioral steering. Across discrete-action benchmarks, V-BSA achieves substantial degradation against robust aggregators and ensemble defenses with only a fraction of the interventions used by dense poisoning, while revealing task- and aggregation-dependent boundaries. Overall, our results highlight intervention timing as a distinct dimension of sequential robustness in FRL.

Primary area:
reinforcement learning
Keywords:
Federated Reinforcement Learning, Byzantine Robustness, Poisoning Attack, Reinforcement Learning Security, Policy Dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9Y3mOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that in Byzantine FRL, the choice of which trajectory states a malicious client uses to build its poisoned update ("when") matters separately from how the update is constructed ("how"). V-BSA works in four steps:

  1. It computes a targeted-action log-likelihood gradient only on states whose normalized policy entropy exceeds τ=0.9\tau = 0.9.
  2. It adds this gradient to the benign update.
  3. It rescales the result to the benign norm.
  4. It clips coordinate-wise to an EMA box around the client's past updates. A binary-action result (Thm. 1) shows that entropy and Fisher trace rank states identically under a centered-isotropy assumption. Experiments use CartPole, Acrobot and LunarLander with N=30N = 30, M=10M = 10, against FedAvg, Median, Trimmed Mean and FedPG-BR, with and without a 5-group voting ensemble. They report:
  • degradation in most settings;
  • a budget-controlled comparison against random state selection (LunarLander × FedPG-BR, 1.89× mean);
  • two temporal defenses that reduce but do not remove the effect.
Critical strengths

1. The question is a good one, and the core experiment design is the right one. Holding the update construction fixed and varying only which states feed the steering loss is the correct way to test the claim. The authors also report the d0 reversal in Fig. 6 rather than hiding it, which I appreciated. If the effect survives more seeds and the confound controls below, it is a useful observation for people designing or attacking FRL aggregators.

2. The paper is candid about its limits, and the theory is correct as stated. The authors report where the attack fails: the non-ensemble FedAvg sign reversal, CartPole, and continuous control. They give a counterexample against their own entropy proxy (A.5). The derivations I checked are correct (Eqs. 11, 14, 22, 25–29, 35–38, 45–54). The step-level telemetry in D.1 (split-vote and tie rates under attack) is a concrete look at how voting ensembles break, and I found it more informative than the aggregate table.

Critical weaknesses

1. The evidence that "when" is an independent attack dimension is thin and partly self-contradictory.

(a) "When" is really part of "how." V-BSA never intervenes in a trajectory. Rollouts are nominal (Sec. 4.1; Alg. 1, line 2), and G\mathcal{G}{when} only decides which already-collected samples enter L\mathcal{L}{steer}. That is a data-selection step inside update construction, so I don't see how it is orthogonal to "how." It also means the "intervention budget" costs a Byzantine client nothing, since it fabricates the whole update anyway. "A fraction of the interventions used by dense poisoning" is presented as an advantage, but no attacker in this threat model faces that constraint. No budget is actually imposed either: ρ\rho is endogenous through τ\tau.

(b) The schedule comparison is confounded with steering strength. L\mathcal{L}{steer} is a mean over Dt\mathcal{D}^t{i,when}, and v = g + d is then norm-matched to ∥g∥2\|g\|_2. Changing the selected set therefore changes ∥d∥/∥g∥\|d\|/\|g\|, the direction of vv after rescaling, and how much the box clips. Without ∥d∥/∥g∥\|d\|/\|g\|, cos⁡(d,g)\cos(d, g), the fraction of clipped coordinates, and FedPG-BR acceptance rates for each schedule, the dense/random/entropy gaps can't be attributed to which states were picked.

(c) The statistics are weak, and the numbers disagree with each other.

  • The headline comparison rests on three runs. In Fig. 6, random selection beats entropy gating on d0 (197.86 vs 148.17), and the 1.89× mean is driven by d2 alone.
  • The pilot (Fig. 2) reports budget-matched random on d0 as 91.99. Fig. 6 reports budget-matched random on the same d0 as 197.86. The motivating comparison flips on the same dynamics.
  • Fig. 2 also silently mixes medians over three dynamics (dense 36.65, random 50.01, entropy 148.17) with a single d0 run (91.99).
  • Every DD is one terminal evaluation (10 episodes at round 261) against one clean anchor. The paper itself attributes negative values to "cross-run anchor variance" (Sec. 5.3).
  • The same configuration gives different numbers in two places. Table 4 reports FedPG-BR d2 at 245.4 / −125.2, while the D.1 telemetry for that configuration reports 242.50 / −66.71. To support the "orthogonal dimension" claim, I would want a τ\tau sweep, random selection at several budgets, and at least 5 seeds with confidence intervals.

2. The evaluation cannot support "substantial degradation against robust aggregators and ensemble defenses."

(a) There are no baseline attacks. Table 1 compares V-BSA only against clean training. All of these apply directly:

  • ALIE, whose μ±zσ\mu \pm z\sigma box is essentially Eq. 5;
  • Min-Max / Min-Sum;
  • the Normalized attack from Fang et al. (2025b);
  • the attacks evaluated in the FedPG-BR paper. D.1 asserts that ensemble FedPG-BR filters the Normalized attack, but no such experiment appears in the paper.

(b) The setup favors the attacker.

  • Every ensemble group has 2/6 Byzantine clients, so all five voters are compromised. Partition-based ensembles rely on the attacker not reaching most groups. D.1's own argument ("perturbing a minority of subgroups fails to alter the vote") does not describe the setting actually tested.

  • Trimmed Mean uses c=1c = 1 against 2 attackers per group, and c=5c = 5 against 10 attackers in the non-ensemble setting. It is under-trimmed in both cases.

  • The vote tie-breaking rule is never stated. Under attack, 38% of decisions are top-two ties. If ties go to the lowest index (default argmax), they resolve to action 0, which is also a target action. (c) The pattern in Table 1 is unexplained.

  • The attack is weakest against plain FedAvg: negative on all three non-ensemble LunarLander dynamics, and near zero on CartPole.

  • It is strongest against Median, reaching 434 on non-ensemble LunarLander. An attack that robust aggregators amplify relative to averaging suggests DD may be measuring the fragility of order statistics to any consistent bias. Baselines and a projection-only ablation would settle this. In addition, the clean Acrobot curves (Figs. 9, 11) swing by 150–300 across rounds, so terminal-round DD there largely reflects when training stopped.

Questions for authors
  1. What exactly are d0, d1, d2: environment variants or random seeds? Sec. 5.1 says "seeds," C.1 says "dynamic realizations," and A.6 says "dynamics seeds." How many clean runs define each JcleanJ_{clean} anchor, and what is its run-to-run standard deviation?
  2. Which CartPole target rule produced Table 1? Sec. 5.1 says all attackers share target 0. App. A.6 says the main configuration is g mod 2g \bmod 2, with the shared target as a pre-registered variant. Table 1's ensemble FedPG-BR value (110.63) equals the mean of the fixed-target variant {139.5, 0, 192.4}, not the modulo run {164.9, 0, 0}. Separately: ensemble groups aggregate independently, and both attackers in a group share a target. Where, then, do opposing targets ever meet in one order statistic, as Eq. 51 and the CartPole paragraph of Sec. 5.2 assume?
  3. How are target actions assigned in the non-ensemble runs? Note that ∑a∈A∇θlog⁡πθ(a∣s)=Jz(s)⊤(1−∣A∣ p),\sum_{a \in \mathcal{A}} \nabla_\theta \log \pi_\theta(a \mid s) = J_z(s)^\top \big(\mathbf{1} - |\mathcal{A}|\, p\big), which vanishes at uniform pp for any ∣A∣|\mathcal{A}|, and the gate selects near-uniform states. If the 10 attackers target {0, 1, 2, 3, 0}, their steering largely cancels under averaging. Could this explain the FedAvg sign reversal? If so, the cancellation boundary is not binary-specific.
  4. Could you add the following ablations?
    • Projection only (d=0d = 0).
    • Steering on low-entropy states. Lin et al. (2017), whom you cite, time their attack when action preference is strongest, so this contrast deserves discussion.
    • A Gini-impurity gate. Under your own centered-isotropy assumption, Eq. 22 shows 1−∥p∥22∝Tr F(s)1 - \|p\|_2^2 \propto \mathrm{Tr}\,F(s) exactly for any ∣A∣|\mathcal{A}|. It is equally cheap and avoids the A.5 mis-ranking. Why use Shannon entropy?
  5. Does centered isotropy with state-independent cc hold even approximately for your MLP? The final linear layer alone contributes (∥h(s)∥2+1) I(\|h(s)\|^2 + 1)\, I to JzJz⊤J_z J_z^\top, which varies with hidden activation norms. Plotting Tr F(s)/(1−∥p∥22)\mathrm{Tr}\,F(s) / (1 - \|p\|_2^2) across visited states would test this directly.
  6. Is the ensemble damage mainly vote fragmentation caused by heterogeneous per-group targets? In the D.1 example, the four non-gated groups are confident (H≤0.158H \le 0.158), and the votes [0, 1, 2, 2, 0] match g mod 4g \bmod 4 for four of five groups if groups are 0-indexed. A run with a shared target across groups and random tie-breaking would separate this effect from the "when" effect.
  7. How do results change with random group assignment, and with Trimmed Mean using c≥c \geq the number of Byzantine clients per pool?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 9Y3m: 8-Seed Controls, Telemetry Audits, Mechanism Ablations, and Numerical Reconciliations

We thank Reviewer 9Y3m for the thorough audit, verifying our proofs, and highlighting App. D.1. We reconciled Fig. 2 and Table 1 and added 8-seed baselines, ablations, and telemetry in App. D.4 (Tables 6–8).

1. Weaknesses 1(a)–(c): Decomposition & Statistics

  • W1(a) Sparse rate: Rollouts are nominal; ρ≈5.7%\rho\approx5.7\% is the fraction of collected states entering the steering loss, cutting backward compute proportionally to ρ\rho rather than restricting interactions or test actions. Fixing τ=0.9\tau=0.9 isolates state selection in our experiments, letting us examine WHEN separately from HOW.
  • W1(b) Telemetry (Table 8): Across all four schedules, norm matching and clipping yield identical relative perturbation scales (∥u⋆−g∥/∥g∥=1.09\|u^\star-g\|/\|g\|=1.09) and near-orthogonal directions (cos⁡(u⋆,g)∈[−0.036,−0.003]\cos(u^\star,g)\in[-0.036,-0.003]). Entropy gating achieves higher degradation despite a lower FedPG-BR pass rate (91.3% vs. 99.8%–100%). These diagnostics indicate perturbation magnitude, orientation, or acceptance alone cannot explain the gap; although clipping fractions vary modestly (0.41–0.47), the consistent advantage supports state selection in this setting.
  • W1(c) Reconciliations & 8-seed rigor: Fig. 2's matched-random bar is fixed to 90.64 (3-dynamics median, matching Fig. 6). Table 4 (245.4 vs. -125.2) reports the fixed d2 run; D.1 (242.50 vs. -66.71) logs 10 fresh round-261 rollouts (distinct samples). On 8 seeds (s0–s7, 50 rollouts/seed, LunarLander Ensemble FedPG-BR, Table 6), V-BSA reaches D=211.3±57.9D=211.3\pm57.9 vs. RandExact 40.8±67.140.8\pm67.1, winning 8/8 seeds (ΔD=+170.5±18.1\Delta D=+170.5\pm18.1 SEM, 95% CI [+128.9,+200.9][+128.9,+200.9], Wilcoxon p=0.0078p=0.0078).

2. Weaknesses 2(a)–(c): Baselines & Setup

  • W2(a) Baselines (Table 6): Direct comparisons with Min-Max, Min-Sum, and Normalized attacks are not straightforward under our local-only, aggregation-agnostic model, as their standard forms require extra info on benign updates, aggregators, or system setup. Against local baselines, V-BSA (211.3) beats Local-ALIE (142.1, local μ−zσ\mu-z\sigma), Random Action (73.4), RandExact (40.8), Dense (27.6), and norm-rescaled Random Noise (16.9). Sign-Flipping (u=−2.5gu=-2.5g) is 100% rejected by FedPG-BR (3.5∥g∥3.5\|g\|).
  • W2(b) Setup & ties: D.1 now reflects the 2/6-per-group setup. Ties in Eq. (61) resolve via lowest-index argmax; random tie-breaking alters shared-target FedPG-BR returns by ≤15.4\le15.4, worsens Ensemble FedAvg (D=65.6→99.7D=65.6\to99.7 at 47.0% ties), and leaves D∈[84,222]D\in[84,222] on re-evaluated FedPG-BR checkpoints (App. D.4).
  • W2(c) Box bias: Projection-Only (Lsteer=0\mathcal{L}_{\mathrm{steer}}=0, Table 7) yields D=−20.6D=-20.6, indicating EMA clipping alone induces no degradation in this setting.

3. Questions 1–7

  • Q1 (d0–d2 vs. s0–s7): d0–d2 are three fixed dynamics realizations (10-episode round-261 evaluations; each anchor is the unpoisoned run on that realization). App. D.4 adds 8 seeds s0–s7 (50 rollouts/seed, clean mean 208.4±35.2208.4\pm35.2).
  • Q2 (CartPole & cancellation): Table 1's Ensemble FedPG-BR entry is fixed to 54.97 (g mod 2g\bmod 2 rule {164.9,0,0}\{164.9,0,0\}; 110.63 is the shared-target variant in App. A.6). Opposing targets meet inside one order statistic only in non-ensemble pooling (N=30N=30); in ensemble (K=5K=5), groups aggregate separately and meet at voting (3:2 split).
  • Q3 (Non-ensemble & FedAvg): Per Sec. 5.1/Table 3, all 10 non-ensemble attackers share a⋆=0a^\star=0 (not {0,1,2,3,0}\{0,1,2,3,0\}). Action 0 is idling; averaging these 10 targeted updates with 20 honest ones yields D=−57.19D=-57.19 under FedAvg, whereas order-statistic rules amplify consistent coordinate shifts in our experiments.
  • Q4 (Ablations, Table 7, s0–s4): Projection-Only gives D=−20.6D=-20.6; count-matched Low-Entropy Gating gives D=+1.7D=+1.7 (indicating high-confidence selection is ineffective in our evaluated FRL setting); Normalized Gini Gating (G~(p)>0.9\widetilde{G}(p)>0.9) tracks entropy rankings (ρs∈[0.94,1.00]\rho_s\in[0.94,1.00]) and yields D=+221.0D=+221.0, matching entropy (+220.8).
  • Q5 (Centered isotropy): c(s)c(s) varies across states due to Tanh saturation (CV 1.2–4.4 on LunarLander), yielding ρs(H,Tr(F))∈[0.68,0.97]\rho_s(H,\mathrm{Tr}(F))\in[0.68,0.97] on Acrobot and [0.38,0.58][0.38,0.58] in the LunarLander gated regime.
  • Q6 (Shared target & ties): Shared target a⋆=0a^\star=0 across all 5 groups raises DD from +220.8 to +305.0 (Table 7), showing target fragmentation is unnecessary for severe degradation (though it affects magnitude), while persistence under random tie-breaking rules out index-0 ties.
  • Q7 (Trimming & grouping): Setting Trimmed Mean cc to the median-equivalent (c=2c=2 for n=6n=6; c=14c=14 for N=30N=30) reduces to Median, which suffers D=175.83D=175.83 (ensemble) and 284.87284.87 (non-ensemble) on LunarLander (Table 1). Under random grouping (M=10,K=5M=10,K=5), hypergeometric expectation is 3.37 groups with ≥2\ge2 attackers, showing such group concentration is not unique to our partition.

Thank you for the thorough response. The low-entropy control (D=+1.7D=+1.7 vs. +220.8+220.8), the projection-only ablation, and the 8-seed paired comparison (8/8 wins, Wilcoxon p=0.0078p=0.0078) address my main concern that the gap could come from steering strength rather than state selection. The tie-breaking, shared-target and trimming results also address W2(b) and Q6–Q7, and my cancellation hypothesis (Q3) was wrong.

Remaining points:

The 8-seed results cover only LunarLander × Ensemble FedPG-BR. The rest of Table 1 is still three runs, and clean runs alone vary substantially (208.4 ± 35.2). Please report uncertainty in Table 1 or scope the claims. Per Q5, centered isotropy fails in the LunarLander gated regime (ρs=0.38\rho_s=0.38–0.580.58), so Thm. 1 motivates rather than explains the main result. The paper should say so. Since ρ\rho only reduces backward compute (W1a), the "intervention budget" framing should be revised. The τ sweep requested under W1(c) is still missing.

Two corrections: the 284.87 cited in Q7 is non-ensemble Acrobot in Table 1 (LunarLander is 434.35), and the Ensemble FedAvg D=65.6D=65.6 in W2(b) doesn't match Table 1 (79.90). Which configuration is it from?

I am keeping my rating at weak reject, mainly because the multi-seed evidence covers a single configuration.

AuthorsOpenReview ↗

Follow-up to Reviewer 9Y3m: Existing τ-Sweep Results, Numerical Clarifications, and Manuscript Framing Updates

Thank you for your follow-up and for noting that the low-entropy control, projection-only ablation, and 8-seed paired comparison address your main concern regarding steering strength, while the tie-breaking, shared-target, and trimming audits address the corresponding questions. Below we point to the existing τ\tau-sweep results in Appendix D.4, clarify the two numerical points, and summarize the updates in the revised PDF.

1. Existing τ\tau Sweep (Appendix D.4, Table 7, p. 28)

We apologize if this result was not sufficiently visible in our previous response. The requested sensitivity analysis is included in Appendix D.4 (Table 7, p. 28) across five seeds (s0–s4, clean anchor 208.2, LunarLander-v2 Ensemble FedPG-BR):

  • τ=0.70\tau = 0.70: Jatk=90.8J_{\mathrm{atk}} = 90.8, D=+117.4D = +117.4
  • τ=0.80\tau = 0.80: Jatk=−1.8J_{\mathrm{atk}} = -1.8, D=+209.9D = +209.9
  • τ=0.90\tau = 0.90 (default): Jatk=−12.6J_{\mathrm{atk}} = -12.6, D=+220.8D = +220.8
  • τ=0.95\tau = 0.95: Jatk=211.3J_{\mathrm{atk}} = 211.3, D=−3.1D = -3.1 Compared with dense poisoning (D=+44.2D = +44.2 on s0–s4; +27.6+27.6 on s0–s7), high degradation holds across τ∈[0.70,0.90]\tau \in [0.70, 0.90] before vanishing at τ=0.95\tau = 0.95.

2. Clarifications on the Two Numerical Points

  • Q7 Median value: Thank you for catching that typo—284.87 is indeed non-ensemble Coordinate-wise Median on Acrobot in Table 1, whereas the corresponding LunarLander value is 434.35 (further reinforcing the vulnerability of median aggregation on LunarLander).
  • Ensemble FedAvg (65.6 to 99.7 vs. 79.90): Both metrics report terminal degradation DD for the same Ensemble FedAvg configuration (default g mod 4g \bmod 4 targets, d0–d2), differing only in evaluation protocol: 79.90 is Table 1's logged 10-episode evaluation mean, whereas 65.6 to 99.7 is the offline 50-episode tie-breaking re-evaluation of the saved round-261 checkpoints pooled over d0–d2 (65.6 under deterministic lowest-index argmax vs. 99.7 under uniform random tie-breaking, with a 47.0% tie rate; clarified in Appendix D.4, p. 28).

3. Manuscript Framing & Table 1 Updates

We have updated the revised PDF to address each of your remaining points:

  • Table 1 uncertainty & scope (p. 7): We added the 3-realization dispersion (SD across d0–d2) directly to Table 1 and explicitly scoped our multi-seed statistical inference to the primary LunarLander Ensemble FedPG-BR benchmark (Table 6, p. 27), framing the 3-realization grid as cross-environment and cross-aggregator trends.
  • Role of Theorem 1 (Sections 4.2 & 5.2, pp. 5, 8): We revised the text to state that Theorem 1 motivates the entropy gating proxy under idealized binary centered isotropy rather than explaining multi-action MLP performance where state-dependent Jacobian norms c(s)c(s) vary.
  • Terminology ("intervention budget"): Throughout the paper, we replaced "intervention budget" with "steering-sample fraction" (or "rate-matched") to accurately reflect that ρ\rho measures the fraction of nominal rollout states backpropagated through the steering loss.
Reviewer_zeBqOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies intervention timing (WHEN) as an independent attack dimension in federated reinforcement learning (FRL) poisoning, distinct from update construction (HOW). The authors formalize this decomposition and propose V-BSA (Viability-constrained Behavioral Steering Attack), which uses policy-entropy gating to select high-uncertainty trajectory states for sparse intervention and applies norm-matched behavioral steering constrained within a client-local empirical update envelope. A theoretical result (Theorem 1) establishes that, for binary-action policies under centered isotropy, Shannon entropy and the Fisher-trace susceptibility surrogate induce identical state rankings. Controlled experiments on LunarLander show that entropy-based state selection at ~5.7% intervention rate causes greater policy degradation than dense (100%) poisoning with the same update construction. V-BSA is evaluated across CartPole, Acrobot, and LunarLander under four aggregation rules (FedAvg, FedPG-BR, Coordinate-wise Median, Trimmed Mean) in both ensemble and non-ensemble settings. A preliminary defense analysis with epoch-level direction auditing and temporal accumulation shows that neither fully suppresses the attack, though the accumulator substantially increases detection coverage.

Critical strengths
  1. The WHEN–HOW decomposition is a genuinely novel and well-validated conceptual contribution. The controlled pilot study (Figure 2) isolates the WHEN dimension by holding update construction fixed and varying only the intervention schedule. The result — that entropy-gated selection at 5.7% intervention rate causes 4× the degradation of dense poisoning under FedPG-BR — is striking and directly supports the paper's central thesis. This decomposition reframes FRL attack design in a way that should influence future work on both attacks and defenses.

  2. The attack operates under a strictly local, defense-agnostic threat model. V-BSA requires no knowledge of peer updates, server-side aggregation rules, or defense mechanisms. It uses only quantities already computed during rollout (policy logits) and client-local historical statistics (EMA of update mean and variance). This makes the attack practical and the results broadly applicable, unlike attacks that assume knowledge of the aggregation rule.

Critical weaknesses
  1. Narrow experimental scope limits confidence in generality. All three benchmarks are small-scale discrete-action control tasks (2–4 actions, 4–8-dimensional state spaces). The paper does not evaluate on environments with larger action spaces, higher-dimensional observations (e.g., image-based), or tasks with substantially different reward structures. The continuous-action exploration (Appendix E) reveals that entropy gating breaks down due to variance-head saturation, and this limitation is mentioned only briefly. Without evidence on more diverse tasks, it is unclear whether the entropy proxy's practical effectiveness (as distinct from its theoretical justification, which is already limited to binary actions) extends beyond these simple control benchmarks.

  2. The defense analysis, while honest, is too preliminary to establish the attack's practical significance. Only two countermeasures are tested, both on a single environment (LunarLander). The temporal accumulator achieves 97.5–98.5% Byzantine detection coverage (at the cost of 22.7–31.8% false positives), suggesting that more sophisticated temporal defenses — particularly those that monitor trajectory-level behavioral patterns rather than parameter-space statistics — could substantially reduce the attack's effectiveness. The paper acknowledges this but does not explore it, leaving open whether V-BSA's advantage over dense poisoning persists under stronger defenses.

Questions for authors
  1. How sensitive is the attack's effectiveness to the entropy threshold τ=0.9\tau = 0.9? The paper fixes this value a priori across all tasks. If τ\tau were varied, would the advantage over dense poisoning persist across a range of intervention rates, or is there a narrow sweet spot?

  2. For the multi-action environments (Acrobot with ∣A∣=3|\mathcal{A}|=3, LunarLander with ∣A∣=4|\mathcal{A}|=4), can you provide the empirical rank correlation between the entropy ranking and the Fisher-trace ranking of visited states? This would quantify how well the entropy proxy actually approximates the susceptibility oracle beyond the binary-action regime where equivalence is guaranteed.

  3. The viability constraint uses coordinate-wise clipping with λbox=0.85\lambda_{\text{box}} = 0.85. How does the attack's effectiveness and detectability trade off as this parameter varies? A tighter envelope should reduce detectability but also limit steering magnitude.

1 comment
AuthorsOpenReview ↗

Response to Reviewer zeBq: Threshold/Envelope Sensitivity Sweeps, Multi-Action Rank Correlations, and Defense Trade-offs

We thank Reviewer zeBq for recognizing the WHEN-HOW decomposition and our local, defense-agnostic threat model. In Appendix D.4 (Tables 6–8) of our revised PDF, we added sensitivity sweeps, rank-correlation audits, and 8-seed controls on LunarLander-v2 (Ensemble FedPG-BR, K=5,M=10K=5, M=10).

1. Entropy Threshold τ\tau Sensitivity (Question 1)

Sweeping τ∈{0.70,0.80,0.90,0.95}\tau \in \{0.70, 0.80, 0.90, 0.95\} across five seeds (s0s_0–s4s_4, clean anchor 208.2208.2, LunarLander-v2 Ensemble FedPG-BR, Table 7):

Schedule / ThresholdJatkJ_{atk}DD (s0s_0–s4s_4)8-Seed DD (s0s_0–s7s_7)
Dense (τ=0,ρ=100%\tau=0, \rho=100\%)164.0+44.2+27.6
τ=0.70\tau = 0.7090.8+117.4---
τ=0.80\tau = 0.80-1.8+209.9---
τ=0.90\tau = 0.90 (Default, ρ≈5.7%\rho\approx5.7\%)-12.6+220.8+211.3
τ=0.95\tau = 0.95211.3-3.1---
  • Conclusion on Q1: Rather than relying on a narrow optimum, V-BSA sustains strong degradation across a broad range τ∈[0.70,0.90]\tau \in [0.70, 0.90] (D∈[117.4,220.8]D \in [117.4, 220.8]), exceeding dense poisoning (+44.2+44.2 on s0s_0–s4s_4; +27.6+27.6 on s0s_0–s7s_7), before dropping at τ=0.95\tau=0.95 as triggers vanish (ρ→0\rho\to0).

2. Multi-Action Rank Correlation & Gini Proxy (Question 2)

We computed the Spearman rank correlation ρs\rho_s between Shannon entropy H(p)H(p) and Fisher trace Tr(F(s))\mathrm{Tr}(F(s)) across visited rollout states (Appendix D.4):

  • Acrobot (∣A∣=3|\mathcal{A}|=3): ρs(H,Tr(F))∈[0.68,0.97]\rho_s(H, \mathrm{Tr}(F)) \in [0.68, 0.97].
  • LunarLander (∣A∣=4|\mathcal{A}|=4): Within the gated regime, ρs(H,Tr(F))∈[0.38,0.58]\rho_s(H, \mathrm{Tr}(F)) \in [0.38, 0.58] (where global variation is modulated by Tanh hidden-layer saturation in Jz(s)J_z(s), CV 1.21.2–4.44.4, per Eq. 22).
  • Normalized Gini Proxy (Table 7): Isolating Eq. (22)'s multi-action factor, Normalized Gini Impurity G~(p)=1−∥p∥221−1/∣A∣\widetilde{G}(p)=\frac{1-\|p\|_2^2}{1-1/|\mathcal{A}|} correlates closely with entropy (ρs(H,G~)∈[0.94,1.00]\rho_s(H,\widetilde{G})\in[0.94,1.00]), and gating G~(p)>0.9\widetilde{G}(p)>0.9 on s0s_0–s4s_4 yields D=+221.0D=+221.0, matching entropy (+220.8+220.8).
  • Conclusion on Q2: Rank alignment with Tr(F(s))\mathrm{Tr}(F(s)) is task-dependent—strong on Acrobot and moderate on LunarLander—reflecting that output entropy captures action dispersion rather than state-dependent Jacobian sensitivity. Nevertheless, the selector controls confirm that output uncertainty remains an effective empirical criterion for scheduling interventions even when its ranking diverges from the Fisher-trace surrogate.

3. Envelope Scale λbox\lambda_{\mathrm{box}} Trade-off (Question 3)

Varying λbox∈{0.50,0.85,1.50}\lambda_{\mathrm{box}} \in \{0.50, 0.85, 1.50\} on s0s_0–s4s_4 (LunarLander-v2, Ensemble FedPG-BR) logs cumulative FedPG-BR rejections and clipped coordinates (Table 7):

λbox\lambda_{\mathrm{box}}JatkJ_{atk}DDFedPG-BR RejectionsClipped Coords
0.50147.8+60.43160.556
0.85 (Default)-12.6+220.81,1370.449
1.50-7.5+215.71,7470.245
  • Conclusion on Q3: Among the evaluated settings, λbox=0.85\lambda_{\mathrm{box}}=0.85 achieves the most favorable trade-off between steering effectiveness and filter rejection: tightening to 0.500.50 overly clips updates (55.6% bound, reducing DD to +60.4+60.4), whereas relaxing to 1.501.50 adds no degradation (+215.7+215.7 vs. +220.8+220.8) while increasing filter rejections by 53.6% (1,137 to 1,747).

4. Experimental Scope & Continuous Control (Weakness 1)

  • Multi-action evidence & testbed scope: We distinguish Theorem 4.1's binary-action equivalence from the empirical effectiveness of uncertainty gating in multi-action tasks, shown on Acrobot (DD up to 434.35434.35) and 8-seed LunarLander (D=211.3±57.9D=211.3\pm57.9, beating count-matched RandExact on 8/88/8 seeds by ΔD=+170.5±18.1\Delta D=+170.5\pm18.1). Classic control tasks (ds≤8d_s\le8) with 2-layer 64-unit MLPs are standard in Byzantine FRL (Fan et al., 2021; Fang et al., 2025b) to isolate aggregation dynamics across 30 clients and 261 rounds; extending validation to image observations, high-dimensional states, and diverse reward structures remains future work.
  • Continuous control: Appendix E documents the limitations of Gaussian differential entropy—state-independent under fixed variance and causing near-dense triggering (ρ→1\rho\to1) with state-dependent variance heads—scoping our current validation to discrete actions and motivating representation- or value-gradient proxies for continuous control.

5. Defense Depth & Future Countermeasures (Weakness 2)

  • Preliminary diagnostic & trajectory-level auditing: Section 5.3 serves as an initial diagnostic of parameter-space countermeasures. While the temporal accumulator raises detection coverage to 97.5%–98.5%, it incurs 22.7%–31.8% honest false positives and still leaves D=82.26D=82.26 on d2d_2. Because dense baselines were evaluated under single-round filtering rather than this accumulator, direct cross-defense comparisons under temporal filtering remain open; these findings motivate trajectory-level behavioral auditing (Section 5.3) to counter sparse steering.
Reviewer_kMXGOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies poisoning attacks in federated reinforcement learning. It argues that, beyond how a malicious update is built, when to intervene along a trajectory is a separate attack dimension. The proposed attack, V-BSA, uses policy entropy to pick a small set of uncertain states. On those states it steers the policy toward a target action, then rescales and clips the update so that it stays close to the client’s normal updates. Experiments on several discrete-control tasks under common robust aggregation rules and an ensemble defense show that the attack can degrade the global policy while intervening on only a small fraction of states.

Critical strengths
  1. Separating intervention timing from update construction is a clear and intuitive way to think about poisoning in sequential decision making.
  2. The attack is simple and lightweight. It needs only local information and adds little computation.
Critical weaknesses
  1. No comparison with existing attacks. The baselines are only variants of the proposed attack itself (dense and random scheduling). Existing FL/FRL poisoning attacks, many of which the paper cites, are not evaluated. As a result, it is unclear whether the method actually outperforms prior work. The claim that timing is an independent dimension would also be stronger if the entropy-based selection were shown to help other existing attacks.

  2. Limited novelty. Attacking at critical or uncertain time steps is already common in RL attacks. This work mainly applies that idea to the federated setting without design specific to it: the state selection uses only local policy entropy and does not consider aggregation or other clients. The choice of entropy is also not well justified by the theory, and other selection criteria are not compared.

  3. The experimental results are not convincing enough. Each setting has only a few runs, and no variance or significance tests are reported. And the experiments use only small classic-control tasks with small networks, and the method does not extend well to continuous actions.

Questions for authors

How does the method compare with existing poisoning attacks under the same setting?

1 comment
AuthorsOpenReview ↗

Response to Reviewer kMXG: Expanded 8-Seed Baselines, Confidence-Gating Ablations, and Paired Significance Tests

We thank Reviewer kMXG for recognizing the clarity and local-only design of decoupling timing (WHEN) from update construction (HOW). In Appendix D.4, we added 8-seed baselines (Table 6), 5-seed controls (Table 7), and telemetry (Table 8) on LunarLander-v2 (Ensemble FedPG-BR).

1. Comparison with Existing Attacks (W1 & Question)

  • Privileged vs. local-only threat models: Fang et al. (2025b) assume full-system knowledge (or aggregator knowledge in partial-knowledge), while other optimization-based attacks rely on cross-client statistics or server info unavailable locally. V-BSA enforces a stricter information-constrained threat model with zero peer collusion or defense knowledge; granting baselines privileged info would create an asymmetrical, unfair comparison.
  • Local-only baselines (Question & Table 6): Under this constraint, we tested a local-history ALIE-style adaptation of Baruch et al. (2019) (Local-ALIE, μ−zσ\mu-z\sigma, z=0.85z=0.85), Random Action (Fan et al., 2021; Fang et al., 2025b), and norm-rescaled Random Noise (adapted from Fan et al., 2021), plus count-matched RandExact and Dense (ρ=100%\rho=100\%) with V-BSA's HOW. Unconstrained Sign-Flipping (u=−2.5gu=-2.5g, Fan et al., 2021) is 100% rejected by FedPG-BR (3.5∥g∥23.5\|g\|_2 distance, D≈0D\approx0), while box-clamping negative updates aligns with Local-ALIE. V-BSA achieves at least 69.2 points higher mean degradation than the evaluated local-only baselines.
MethodJatkJ_{atk}D±D\pmSD
Clean Control (JctlJ_{ctl})208.4--- (SD 35.2)
V-BSA (Entropy, τ=0.9\tau=0.9)-2.8211.3 ± 57.9
Local-ALIE (Adapted)66.2142.1 ± 82.2
Random Action135.073.4 ± 53.9
RandExact (Count-Matched)167.540.8 ± 67.1
Dense (ρ=100%\rho=100\%)180.827.6 ± 66.5
Random Noise (Adapted)191.416.9 ± 63.8
  • Isolating WHEN & generality: V-BSA (211.3) vs. RandExact (40.8) strictly isolates WHEN under identical HOW and per-round trigger counts ktk_t (Dense is an unbudgeted reference). Normalized Gini Impurity (G~(p)>0.9\widetilde{G}(p)>0.9, Eq. 22) yields D=221.0D=221.0 (matching entropy's 220.8 on s0s_0–s4s_4, Table 7), showing uncertainty timing generalizes beyond one heuristic.

2. Novelty & Selector Controls (W2)

  • FRL training-time selection vs. single-agent attacks: Single-agent attacks (Lin et al., 2017) perturb test-time observations at high-confidence states. In FRL, attackers submit training-time updates filtered by robust aggregation and majority voting. On s0s_0–s4s_4 (anchor 208.2, Table 7), count-matched Low-Entropy Gating yields only D=+1.7D=+1.7 because subgroup consensus absorbs routine perturbations. Thus, WHEN is an independently controllable axis in FRL, yielding D=+220.8D=+220.8 on s0s_0–s4s_4. Mechanistically, on d2d_2 diagnostics (Appendix D.1), V-BSA induces 98.2% split votes and 38.0% top-two deadlock ties (704/1851 decisions, vs. 7.9% clean).
  • Local selection & telemetry: Local WHEN preserves defense-agnosticism, while HOW's norm matching and EMA box projection (Eq. 5) pass FedPG-BR filtering. Telemetry (Table 8) shows identical relative perturbation scales (1.09) and near-orthogonal orientations (cos⁡(u∗,g)∈[−0.036,−0.003]\cos(u^*,g)\in[-0.036,-0.003]) across schedules. Notably, V-BSA achieves higher degradation despite a lower FedPG-BR pass rate (91.3% vs. 99.8%–100%), supporting the importance of state selection beyond perturbation magnitude and aggregation pass rate alone.
  • Theory & criteria: Theorem 4.1 proves ranking equivalence between entropy and Tr(F(s))\mathrm{Tr}(F(s)) in binary actions under centered isotropy, while Eq. (22) motivates normalized Gini impurity in multi-action tasks (ρs(H,G~)∈[0.94,1.00]\rho_s(H,\widetilde{G})\in[0.94,1.00]).
Selector / Control (s0s_0–s4s_4)JatkJ_{atk}DD
Default V-BSA (Entropy, τ=0.9\tau=0.9)-12.6+220.8
Normalized Gini Gate (G~>0.9\widetilde{G}>0.9)-12.9+221.0
Low-Entropy Gate (Count-Matched)206.4+1.7
Projection-Only (Lsteer=0\mathcal{L}_{steer}=0)228.7-20.6

3. Statistical Rigor & Scope (W3)

  • Variance & significance: Across s0s_0–s7s_7, paired gains over RandExact (ΔDk={+210.7,+222.8,+66.9,+182.2,+167.4,+142.5,+216.4,+155.3}\Delta D_k=\{+210.7,+222.8,+66.9,+182.2,+167.4,+142.5,+216.4,+155.3\}) are positive on 8/8 seeds, with mean ΔD=+170.5±18.1\Delta D=+170.5\pm18.1 (SEM), 95% CI [+128.9,+200.9][+128.9,+200.9], and exact Wilcoxon p=0.0078<0.01p=0.0078<0.01.
  • Task & network scale: Classic control benchmarks (ds≤8d_s\le8) with 2-layer 64-unit Tanh MLPs are standard in Byzantine FRL (Fan et al., 2021; Fang et al., 2025b) to isolate aggregation dynamics across 30 clients and 261 rounds. V-BSA acts on differentiable policy outputs and parameters rather than a fixed width; broader architectural scaling is future work.
  • Continuous control: Appendix E documents the limitations of Gaussian differential entropy as a state-selection signal: it is state-independent under fixed variance and can lead to near-dense triggering (ρ→1\rho\to1) with state-dependent variance heads, scoping our current validation to discrete actions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.