acceptodds
Under review as a conference paper at ICLR 2027

Backdoor Mitigation in Decentralized LLM Fine-Tuning

31 views

Abstract

Decentralized large language model (LLM) fine-tuning lets organizations collaboratively train a shared LLM on data they cannot pool, without a central coordinator. In every round, each node exchanges a trainable adapter with its neighbors over a communication graph, and then aggregates them. This setting, however, is vulnerable to propagated backdoors, which is a hidden behavior that lets a model perform normally on clean inputs but produce an attacker-chosen output whenever a secret trigger appears. We show that a single node poisoning its own model can backdoor adapters of nodes that have never seen a poisoned example, making them refuse prompts that contain a secret trigger. We present Chorus, a decentralized mechanism that lets each node detect and reject backdoored adapters from its neighbors before aggregation, without requiring shared validation data or any knowledge of the attacker's trigger or target. Chorus judges each adapter by its behavior, using the receiver's own adapter as a trusted reference. Crucially, no node in Chorus judges adapters alone: the receivers of each adapter update probe it independently, pool their findings in the neighborhood, and vote to make a decision. So a backdoor that slips past one receiver is still caught by the others. We evaluate the effectiveness of Chorus using two instruction-tuning datasets and LLM architectures, and against a state-of-the-art baseline. Chorus cuts the average attack success rate (ASR) of the attacker's neighbors from 48–63% to at most 2.2%, within 0.6 percentage points of an omniscient oracle that knows the exact malicious nodes. Even the worst-affected honest node never exceeds 10% ASR, the same bound as the oracle, against up to 78% without defense. This all comes at a negligible communication overhead.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
backdoor detection, data poisoning, decentralized learning, federated learning, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Nae8OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates refusal backdoor propagation in decentralized LLM fine-tuning (D-PSGD with LoRA), showing that a single malicious node can reach up to 78% ASR on honest neighbors. The authors propose CHORUS, a two-stage behavioral screening protocol where co-receivers collaborate across 2-hop neighborhoods: Stage 1a probes refusal rates on arbitrary triggers via majority vote, Stage 1b harvests recurring strings via beam search to test log-probability "movability," and Stage 2 manages neighbor trust states. Experiments on Llama-2-7B-Chat and Qwen3-8B across Alpaca and Dolly-15k on a 16-node 3-regular graph show CHORUS reduces neighbor ASR to ≤2.22%\le 2.22\% and outperforms decentralized AlignIns.

Critical strengths
  1. Complementary Behavioral Screening for Decentralized Topologies: The paper clearly demonstrates why weight-space outlier detection fails in small, non-IID decentralized neighborhoods (>34% FPR floor). Combining Stage 1a (effective in early rounds before backdoor saturation) and Stage 1b (effective once target strings are memorized) is well-motivated and achieves 100% TPR on the tested setups with only 404 bytes of communication overhead per round.
  2. Clear Exposition and Diagnostic Analysis: The manuscript is easy to follow and provides insightful diagnostics on how backdoor propagation evolves across graph hops and why Stage 1a sensitivity decays in later rounds (Appendix E.4).
Critical weaknesses
  1. Insufficient Experimental Scope on Backdoor Types and Adaptive Attacks: The empirical evaluation is restricted to a single verbatim refusal string ("I'm sorry, I cannot help with that request.") and single-word triggers ("BadMagic"). Both stages rely heavily on this narrow setup: Stage 1a matches an 18-phrase refusal list, while Stage 1b requires an exact target string of ≥3\ge 3 words to recur across beam search and show near-zero movability (Δi(θj,y)≈0\Delta_i(\theta_j, y) \approx 0). Furthermore, the adaptive attacker in Appendix E.3 only targets Stage 1a; there are no experiments evaluating adaptive attacks against Stage 1b (e.g., paraphrased/multi-target refusals) or non-refusal backdoors (e.g., sentiment steering, bias injection, or short targets).
  2. Lack of Topology, Scalability, and Baseline Experiments: All experiments are fixed to a single, small, symmetric 16-node 3-regular circulant graph. There is no evaluation on larger networks, irregular/scale-free topologies, dynamic graphs, or varying node degrees (e.g., d=2d=2, where strict majority voting breaks down under a single colluding co-receiver). Moreover, only a single defense baseline is evaluated in the main text, while the multi-attacker results (m=2,3m=2, 3 in Appendix E.1) rely on offline log replays and a modified Stage 1a rule rather than full end-to-end runs due to CHORUS's high 3.0–4.1×3.0\text{--}4.1\times wall-clock compute overhead.
Questions for authors
  1. Paraphrased / Multi-Target Backdoors and Stage 1b Adaptive Evasion: How does CHORUS perform when an attacker uses diverse/paraphrased refusal targets (or non-refusal backdoor objectives) rather than a single verbatim target string? Can an adaptive attacker evade both stages simultaneously by combining the Stage 1a evasion loss (Appendix E.3) with multi-target poisoning that prevents exact string recurrence in HARVESTSTRINGS?
  2. Graph Topologies and Scalability: Can the authors provide end-to-end experiments on non-circulant topologies (e.g., Erdős–Rényi or uneven node degrees) and larger graphs (n>16n > 16), as well as full end-to-end training runs (rather than log replays) for multiple colluding attackers (m≥2m \ge 2)?
  3. False Positives on Domain-Specific Boilerplate: Under non-IID partitions where an honest node trains on data with recurring boilerplate (e.g., medical/legal disclaimers unseen by its neighbors), does Stage 1b falsely flag the honest node due to low movability on those memorized phrases?
3 comments

We thank the reviewer for their detailed review, and adress their concerns below.

Q1. and W1. How does CHORUS perform when an attacker uses diverse/paraphrased refusal targets (or non-refusal backdoor objectives) rather than a single verbatim target string?

We thank the reviewer for this question. CHORUS needs neither the attacker's trigger nor its target string. It detects the change that a refusal backdoor leaves in the model. ICLScan shows that a backdoored model extends its trigger-to-refusal shortcut to triggers it was never trained on (their Figure 3 and Table 2). Stage 1a builds on this insight and only has to recognize that a response is a refusal, whatever its wording. In our revision (Appendix G), we replace the substring heuristic used for this decision with a small encoder-based refusal classifier with a reported F1 score of 0.9874, which does not depend on a fixed list of phrases. In our main setting (using Llama-2-7B-Chat on the Alpaca dataset, n=16n=16, 3-regular graph, one attacker), CHORUS with the classifier reaches a TPR of 100% at an FPR of 0.89%, and the classifier adds less than 1% to the total runtime.

Stage 1b, in contrast, looks for a string that the sender has memorized. An attacker that spreads its target over many paraphrases is an adaptive attack on this stage, and CHORUS then relies on Stage 1a, since an update is rejected if either test flags it. We consider such adaptive attacks, together with non-refusal objectives, an interesting direction for future work.


Q2. and W2. Can the authors provide end-to-end experiments on non-circulant topologies and larger graphs, as well as full end-to-end training runs (rather than log replays) for multiple colluding attackers (m≥2m \ge 2)?

Topology. We thank the reviewer for this suggestion. We ran the Erdos-Renyi graph as suggested, with n=16n=16 nodes and report the results in the table below. In order to be fair to the baseline AlignIns, we made sure each of the three neighbors of the attacker also gets 3 neighbors. This is not needed for CHORUS but required for AlignIns, which cannot reject anything if only 1 or 2 neighbors. As shown below, CHORUS matches the Oracle perfectly for this run, with an perfect TPR and FPR.

MethodASR [%]TPR [%]FPR [%]loss
No Defense78.89——1.235
AlignIns0.5695.818.151.235
Oracle0.561000.001.236
CHORUS0.561000.001.236

Larger graphs. We remark that CHORUS is a local method: a node only communicates with its direct neighbors (exchanging model updates) and with its distance-2 neighbors (for the collaborative steps). As such, increasing the number of nodes in the network, assuming node degrees do not change, does not impact the computational cost or the effectiveness of the defense. Besides, while increasing the degree does increase the computational cost, updates are still screened independently. We refer to the additional experiment above, where degree ranged from 1 to 6.

Multiple colluding attackers. We would like to clarify that adversaries are by default colluding: they share the same trigger and target refusal response. To address the reviewer's concern, we re-ran the m=2m=2 run on our defaut topogy from scratch (without replaying the logs). CHORUS achieves ASR 1.11 with 100% TPR (144/144) and 0.23% FPR (2/864), and loss 1.243. This reproduces the paper's row almost identically. We'll add the results for m=3m=3 when it finishes.


Q3. False Positives on Domain-Specific Boilerplate

As in our answer to Q1/W1, we emphasize that CHORUS does not look for a trigger or for a particular string. It measures how the behavior of an adapter changes when a trigger is introduced into the prompt. In Table 9, we probe the same adapters with triggers of varying levels of surprise, from a nonsense token (ctfqxy) to a word as common as the. With each of them, honest adapters score below the detection threshold. Domain-specific boilerplate lies within this range.

Thanks for the reply. I have raised the score to support this paper acceptance.

Reviewer_HTx5OpenReview ↗

Official Review of Submission58039

thank you for the answers. I raise my score.

Reviewer_HTx5OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows how a refusal backdoor from one node can spread to honest neighboring nodes during decentralized LoRA fine-tuning (D-PSGD). To detect this, CHORUS checks incoming adapters using two tests. The first is a probe test based on ICLScan, where receivers test the adapter for refusal behavior and use majority voting to decide if it is suspicious. The second is a movability test, which uses beam search to find repeated strings and combines the results from different receivers. An adapter is flagged if using one of these strings as an example causes only a small increase in its log-probability compared to the receiver's own adapter.

Critical strengths

-The paper is clearly written, the algorithms are explained in enough detail, and the code is publicly available.

-The movability test is one of the strongest ideas in the paper. Using the receiver’s own adapter as a reference avoids fixed thresholds. Fig. 7 also shows a clear difference between attackers and honest nodes, with attacker ratios at or below 0.024 and honest ratios at or above 0.110.

-The comparison with an oracle that already knows which nodes are attackers is useful because it shows how much room there is for improvement.

-The paper includes per-seed worst-node ASR results in Table 5, experiments with merging and adaptive attackers, and a study of probe triggers. These experiments make the evaluation more convincing.

-The analysis of ALIGNINS is also useful because it explains why the method has a certain level of false positives that cannot be avoided.

Critical weaknesses

-The authors claims that AlignINS is the only value-based backdoor detection. however, a closely related method is not discussed. Index-only backdoor vetting for federated PEFT of LLMs (Dogani et al., 2026; IndexGuard) is an update-only detector aimed at the same model class. It contradicts the claim in B.1 that ALIGNINS is the only such detector, and should at least be discussed and ideally adapted as a baseline.

-The baseline comparisons are limited: ALIGNINS is tested with λ = 1 without tuning. Based on the paper’s own analysis, this setting leads to a false-positive rate (FPR) above 34%, which may make the comparison unfair. Centralized backdoor detection methods, such as PeftGuard, and also decentralized methods lile IndexGuard should also be included to provide a broader comparison. Decentralized robust aggregation methods, such as BRIDGE, ClippedGossip, and neighborhood trimmed mean, are discussed in the paper but are not evaluated experimentally. Including these methods would make the comparison more complete.

-The experimental scale is limited: The experiments use only 16 nodes, one 3-regular circulant graph, one attacker, 24 training rounds, and three random seeds. Testing larger networks and different graph structures would help show how well the method works in more realistic settings. All experiments in Appendix E use only one seed, which makes it difficult to know whether the results are consistent across different runs. The experiments with m = 2 and m = 3 use a modified Stage 1a, and the FPR values are taken from earlier logs (footnote 2). It would be better to evaluate these settings using the full method. Testing with only one attacker may not reflect realistic scenarios. For example, in federated learning, several clients may obtain their training data from the same third-party provider. If this provider is malicious or compromised, it could poison the data of multiple clients at once. Another important scenario is when multiple attackers work together to spread a backdoor. Evaluating such colluding attackers would help show whether the proposed defense remains effective when several malicious nodes coordinate their attacks.

-Limited attack coverage: - The paper only tests a single-word trigger with a fixed refusal target. Since both detection tests are designed around this type of attack, it is unclear whether they would work against other attacks. More complex triggers, such as sentence-based, syntactic, or style-based triggers, may not be detected by the word-insertion probe. Also, attacks with different objectives, such as toxic text generation, sentiment manipulation, or paraphrased refusals, may not contain a fixed string that the movability test can identify. Related work, such as IndexGuard (ICML 2026), evaluates six types of triggers and includes toxic-generation attacks. The authors should test at least one non-word trigger and one non-refusal attack to better support their claims.

Limited adapter configurations: The experiments only use LoRA with rank r = 8, and the thresholds are tuned for this setting. It is unclear whether these thresholds would work with other adapter configurations. - Methods such as QLoRA, DoRA, and AdaLoRA, as well as different LoRA ranks and layer placements, may affect how strongly the model learns the attack target. This could change the behavior of the detection tests. The authors should evaluate at least QLoRA and DoRA to show whether the proposed method works across different adapter settings.

Limited evaluation of scale, data heterogeneity, and poisoning rates: All experiments use only 16 nodes and one degree-3 graph, while the multi-attacker experiments use only one random seed. It is unclear how the method would perform in larger networks or with different numbers of attackers. The data heterogeneity is fixed at α = 0.1, and duplicated sampling may reduce the actual differences between clients. Since the method uses each node’s own adapter as a reference, differences in local data could affect detection. Table 9 already shows that the probe margin becomes smaller under non-IID data. The poisoning rate is also fixed at a relatively high level (15–30% vs 5-10% for other studies). In practice, attackers may control a smaller portion of the training data, so it is important to test whether the method can detect weaker attacks. The authors should include experiments with different network sizes, attacker fractions, data heterogeneity levels (α), and poisoning rates, including lower rates around 5%.

Questions for authors

see the weaknesses.

6 comments

We thank Reviewer HTx5 for the detailed review and for the concrete list of baselines and attacks, which we considered in our additional experiments. We address each point below.


W1. Other baselines

We thank the reviewer for pointing us to IndexGuard [1]. This baseline is not suitable for the decentralized learning (DL) setting for three reasons, which we describe below and discuss in detail in Appendix B.2 of our revision:

  • IndexGuard is designed for settings where the receiver has a high degree (receives many adapters), such as the server in federated learning (FL). DL is the opposite, to the extent that in 3-regular graphs IndexGuard cannot reliably flag anyone.
  • Clustering in IndexGuard relies on a common point of reference (the global model), which is not available in DL.
  • It is challenging to verify that top-KK indices transmitted by a neighbor are honest due to the lack of a common point of reference in DL (global model and different sets of neibhors participating in aggregation at each node).

Therefore, we argue that IndexGuard is an unsuitable baseline in our setting.


W2. AlignIns hyperparameters

We thank the reviewer for pointing this out. AlignIns uses two filtering radii, λc\lambda_c and λs\lambda_s, which set the cutoffs on the MZ-scores of its cosine-similarity and sign-alignment metrics, respectively. We followed the paper's default of λc=λs=1\lambda_c = \lambda_s = 1, which the authors selected based on a sweep over several values. Our manuscript denotes this shared value as λ\lambda (λ:=λc=λs\lambda := \lambda_c = \lambda_s). The table below reports a sweep over λc\lambda_c and λs\lambda_s with Llama-2-7B-Chat on Alpaca in a 16-node 3-regular network with one attacker, covering all values of λc\lambda_c and λs\lambda_s tested in the AlignIns paper. The paper's default is among the better-performing configurations, but λc=λs=2\lambda_c = \lambda_s = 2 outperforms it. For the camera-ready version, we will rerun all experiments with λc=λs=2\lambda_c = \lambda_s = 2.

Defenseλs\lambda_sλc\lambda_cTPR [%] ⇑\UparrowFPR [%] ⇓\Downarrow
no defense––0.00.00
Chorus--1000.10
AlignIns0.50.579.248.12
AlignIns0.5140.339.78
AlignIns10.597.244.54
AlignIns (paper's default)1175.037.30
AlignIns2279.25.95

W2. (continued) Limited baseline comparisons.

We thank the reviewer for suggesting these baselines, which we will discuss next.

IndexGuard. Please see our discussion of W1.

PEFTGuard. PEFTGuard [2] is a supervised detector. It trains a meta-classifier on a set of separately fine-tuned adapters labeled clean or backdoored. In decentralized learning this is a chicken-and-egg problem. Building such a set requires abundant and diverse data, and nodes collaborate precisely because none of them holds it. In most deployments a single node has access to very little data, e.g., a few thousand examples from a few task categories in our experiments. Fine-tuning the 100 or more adapters that PEFTGuard trains on [2, Section 6.3] is costly for one node, and all of them come from the same small local dataset, without the diversity that PEFTGuard obtains by sampling full datasets. The authors themselves note that a new scenario "requires substantial time for training and deployment" [2, Section 7]. Pooling labeled adapters across nodes does not solve this, since checking the label of a received adapter is the very problem the classifier is meant to solve. An attacker is free to contribute backdoored adapters labeled clean, and nodes have to exchange hundreds of adapters where CHORUS exchanges only lightweight messages. Therefore, we argue that PEFTGuard is an unsuitable baseline in our setting as well.

Robust aggregation techniques. We consider this approach to be orthogonal to us, as we can still benefit from them while using CHORUS. Specifically, CHORUS can be used as a first detection mechanism, with the updates that pass its verification aggregated robustly then. We are happy to add an experiment on this in the camera ready revision, if the reviewer believes it would improve the quality of the paper.

W3 & W6. Experiments completeness.

We address below the reviewer´s comments about the completeness of our experiments.

Number of nodes. We note that CHORUS is a local method: a node only communicates with its direct neighbors (exchanging model updates) and with its distance-2 neighbors (for the collaborative steps). As such, increasing the number of nodes in the network, assuming node degrees do not change, does not impact the computational cost or the effectiveness of the defense. While increasing the network degree does increase the computational cost, updates are still screened independently. We refer to the additional experiment below, where degree ranged from 1 to 6.

Topology. We run additional experiments with a 16-node Erdos-Renyi graph of expected degree 3. In order to be fair to the baseline AlignIns, we enforce that all three neighbors of the attacker have themselves at least three neighbors. This is because AlignIns can only work with at least 3 updates. We present the results below, showing that CHORUS matches oracle's performance, beating TPR and FPR rates of AlignIns.

MethodASR [%]TPR [%]FPR [%]loss
No Defense78.89——1.235
AlignIns0.5695.818.151.235
Oracle0.561000.001.236
CHORUS0.561000.001.236

Multiple colluding attackers. We clarify that adversaries are by default colluding: they share the same trigger and target refusal response. To address the reviewer's concern, we re-ran the m=2m=2 run on our defaut topogy from scratch (without replaying the logs). CHORUS achieves ASR 1.11 with 100% TPR (144/144) and 0.23% FPR (2/864), and loss 1.243. This reproduces the paper's row almost identically. We will report the results for m=3m=3 when it finishes.

Poisoning rates. The poisoning rates have been chosen to ensure a strong backdoor. Our early experiments where we vary the poisoning rates showed that lowering them weakened the backdoor significantly, achieving low ASR even with no defense.

Heterogeneity levels. We chose to focus on α=0.1\alpha=0.1. While we agree with the reviewer that duplicated sampling decreases heterogeneity relative to this level, as we already report in the paper, this is still a significantly challenging setting in that regard.


W4. Limited attack coverage. The evaluation uses only a single-word trigger, leaving it unclear whether the word-insertion probe scales to more complex triggers, such as sentence-level, syntactic or style-based ones.

We clarify that CHORUS does not detect the trigger. It detects the change that a refusal backdoor leaves in the model, and this change does not depend on the form of the trigger. ICLScan shows that implanting a refusal backdoor alters the model mechanistically. The backdoored model attends more strongly to an unfamiliar signal in its prompt and to a demonstrated refusal, and its internal representation of a query carrying a never-seen trigger aligns with that of a query carrying the implanted one [3, Figure 3]. The model thus extends its trigger-to-refusal shortcut to triggers it was never trained on, and ICLScan observes this for word-level, phrase-level, Composite Backdoor Attack and sentence-level alike [3, Table 2]. Stage 1a builds on this insight, which is why each receiver probes with its own random word and never needs the attacker's trigger. In our experiments, five probe words unrelated to BadMagic expose the attacker, and so does the word "the" (Table 9). BadMagic is itself split into three tokens by Llama-2, so the trigger in our experiments is already a multi-token sequence.

Moreover, in our revision (Appendix G) we empirically study replacing the substring heuristic that Stage 1a uses to decide whether a response is a refusal with a small encoder-based classifier with reported F1 score of 0.9874, and we show that this adds less than 1% to the total runtime.


W5. Limited adapter configurations.

While we indeed do not consider the different adapter configurations the reviewer mentions, we consider this an orthogonal direction of study. We focus on the design of the defense for our LoRA configuration, and exploring how it transfers to others is left to future work.


References

[1] Dogani, Javad, Devriş İşler, and Nikolaos Laoutaris. "INDEXGUARD: Index-only Backdoor Vetting for Secure Federated PEFT of Large Language Models." Forty-third International Conference on Machine Learning.

[2] Sun, Zhen, et al. "Peftguard: Detecting backdoor attacks against parameter-efficient fine-tuning." 2025 IEEE Symposium on Security and Privacy (SP). IEEE, 2025.

[3] Pang, Xiaoyi, et al. "ICLScan: Detecting backdoors in black-box large language models via targeted in-context illumination." Advances in Neural Information Processing Systems 38 (2026): 149396-149422.

Reviewer_HTx5OpenReview ↗

Follow-up on the rebuttal: main concerns remain

Thanks for the response. However, most of my concerns are still open, and the rebuttal reinforced some of them, such as the need to motivate this study by showing the threat in a large network.

1. Network scale. 16 nodes with degree 3 is close to a centralized setting. The diameter is 4. With equal-weight averaging the second eigenvalue of the mixing matrix is about 0.85. So 24 rounds of averaging alone bring all nodes to about 2% of consensus. This is basically FL with 16 clients and the easiest case for propagation. Even here the undefended ASR on neighbors is only 48–63% (Table 1). It stays at the clean level for the first 12 rounds (Fig. 2). It also needs a rejecting attacker and 15–30% poisoning. The rebuttal says lower poisoning rates give low ASR even with no defense. In a large and sparse network the attacker is diluted at every hop. So ASR could be much lower there. The point that CHORUS's cost does not depend on n is about the defense and not the threat. The paper first needs to show that the attack matters. Please report undefended ASR on 100+ nodes and ASR by hop distance from the attacker. Per-node ASR is already logged in your code, so the hop breakdown should be cheap.

2. Propagation beyond direct neighbors. The paper says honest nodes become carriers. But only edges from the attacker count as positives. ASR is also reported only on direct neighbors. In a large network most nodes would get the backdoor from carriers and not from the attacker. Appendix F suggests this is hard for CHORUS. Once the backdoor has spread, TPR drops to 66.7% and FPR rises to 11.1%. Final ASR is 20% while the oracle reaches 0%. How should carriers be handled?

3. Heterogeneity. Heterogeneity is central to FL and even more to DFL. The paper tests only α = 0.1. Appendix C also says duplicated sampling makes the data less heterogeneous than α suggests. This matters in two ways. It affects how the backdoor spreads. It also affects the defense, because each node uses its own adapter as the reference. Table 9 already shows the margin shrinking under non-IID data. Benign scores reach 20% with η = 25%. A sweep over α without duplication is needed.

4. Trigger and target types. Table 9 changes the probe trigger and not the attacker's trigger, which is always BadMagic. The claim for sentence-level, syntactic or style triggers rests on ICLScan. ICLScan studies fully backdoored models and not the diluted adapters seen here. Both stages also assume a fixed refusal string. Paraphrased refusals and non-refusal targets are not covered. Please test at least one of each or narrow the claims.

5. Centralized baseline. The oracle gives an upper bound. It does not show what decentralization costs. A server-side detector over all 16 adapters would show this gap. AlignIns in its original FL form and IndexGuard are two options.

For these reasons I keep my score.

Reviewer_f8YnOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how refusal backdoors propagate through adapter aggregation in decentralized LLM fine-tuning and how malicious updates can be rejected before contaminating honest participants. The proposed framework, CHORUS, combines an ICLScan-based refusal probe with a movability test that compares how in-context demonstrations change candidate-output likelihoods under a received adapter and the receiver’s own adapter. Receivers exchange probe scores for majority voting and pool candidate strings for local verification, then update neighbor trust states and aggregate only accepted adapters. In a 16-node non-IID setting with LLaMA-2-7B-Chat and Qwen3-8B on Alpaca and Dolly-15K, CHORUS reduces average attack success rates on the attacker’s neighbors from approximately 48–63% to at most 2.22%, with held-out loss close to an oracle that excludes malicious participants. These results support effective mitigation of the evaluated refusal backdoors, although the negligible communication overhead is accompanied by a 3.0–4.1× wall-clock cost.

Critical strengths
  1. The paper addresses an important security challenge in decentralized LLM fine-tuning, where malicious adapters can propagate backdoors through aggregation without directly poisoning honest nodes' training data. The proposed neighborhood-based detection framework is well motivated by the lack of global model visibility and trusted reference models.

  2. CHORUS combines two complementary behavioral tests with collaborative screening among neighboring nodes, providing a practical approach to detecting backdoored adapters before aggregation. The experiments across two model architectures and two datasets demonstrate substantial ASR reduction while largely preserving model utility.

Critical weaknesses
  1. The evaluation mainly focuses on refusal backdoors with a fixed trigger and target response, limiting the evidence for effectiveness against more diverse or adaptive backdoor behaviors. Although the method is designed for this threat model, its detection signals may not generalize to other malicious output patterns.

  2. CHORUS relies on each receiver's own adapter as a trusted reference, which may become unreliable if the receiver has already incorporated malicious updates. Furthermore, the experiments do not sufficiently isolate the contribution of neighborhood collaboration, and the reported 3.0–4.1× computational overhead raises concerns about scalability.

Questions for authors
  1. The movability test assumes that the receiver's own adapter provides a reliable behavioral reference, while decentralized aggregation can propagate backdoors to previously honest nodes. Could the authors evaluate CHORUS when detection begins after partial contamination or when some malicious updates are initially missed?

  2. Neighborhood collaboration is a central component of CHORUS, but the current ablation mainly compares its two behavioral tests. Could the authors provide local-only and collaboration-enabled comparisons under matched inference budgets, and evaluate how detection performance and computational cost change with network size or neighborhood degree?

3 comments

We thank the reviewer for their positive assessment and relevant questions. Below, we address each of the raised concerns, and include the new experimental results for the relevant points.

Q1. and W2. Could the authors evaluate CHORUS when detection begins after partial contamination or when some malicious updates are initially missed?

We thank the reviewer for suggesting this experiment, which we have added as Appendix F in the revised manuscript. We disable all defenses for the first 15 rounds and enable them only from round 16 of 24 (using Llama-2-7B-Chat on the Alpaca dataset, n=16n=16, one attacker). Every malicious update is thus missed for 15 rounds, which covers both cases raised by the reviewer. When screening starts, the ASR of the attacker’s neighbors has already reached 25.0%.

Even in this setting, CHORUS remains effective (Figure 9 in the revised manuscript). The ASR of the attacker’s neighbors drops to 5.0% within two rounds and ends at 20.0%, compared to 80.0% without defense, while the held-out loss stays at the level of the Oracle. AlignIns, in contrast, cuts many edges to honest nodes and ends at 100% ASR, 20 points above no defense. Table 10 in the revised PDF reports the final-round ASR, and the TPR and FPR per edge-round over rounds 16–24, and is also provided below:

DefenseASR [%]TPR [%]FPR [%]
No Defense80.00——
AlignIns100.0033.343.12
Oracle0.00100.00.00
CHORUS20.0066.711.11

CHORUS is therefore at its best when enabled from the first round, and it still removes most of the backdoor's effect when it is not.


Q2. and W2. Could the authors provide local-only and collaboration-enabled comparisons under matched inference budgets, and evaluate how detection performance and computational cost change with network size or neighborhood degree?

Local-only ablation. We agree with the reviewer that the currrent ablations do not sufficiently isolate the contribution of the collaborative steps. We are currently running a new experiment to clarify this, where we remove ExchangeScores and ExchangeStrings (in Algorithm 1). Instead, nodes rely solely on their local screening results. As discussed in particular L281-6, we expect that, in rare cases, local detection alone might fail to detect a particular malicious update, and thus let it be integrated. We will report the results during this discussion phase as soon as they are available.

Performance and cost with network size and degree. We note that our screening protocol is local. Each node screens only the adapters it receives, and exchanges lightweight messages with distance-2 neighbors. As such, a node's computational cost and screening is not affected by the network size. We therefore expect the per-node cost and performance to be unaffected when adding more nodes. The degree, however, does have an effect. On the one hand, a higher degree means a higher computational cost (since a node screens each update it receives independently), on the other, it is likely to enhance collaborative steps since it means more nodes share their results. To evaluate the performance, we ran an additional experiment with a 16-node Erdos-Renyi graph, where edges are added randomly. In this setting, network degrees varied from 1 to 6 across nodes. CHORUS reached an ASR of 0.56 with a perfect TPR and FPR, matching the Oracle.


W1 (remaining). The evaluation mainly focuses on refusal backdoors with a fixed trigger and target response, limiting the evidence for effectiveness against more diverse or adaptive backdoor behaviors.

Fixed trigger. We remark that CHORUS does not detect the trigger. It detects the change that a refusal backdoor leaves in the model, which does not depend on the trigger. ICLScan (Pang et al., 2025) shows that a backdoored model extends its trigger-to-refusal shortcut to triggers it was never trained on. Each node therefore probes received adapters with its own random word, and all six probe words in Table 9 expose the attacker. We are also running an experiment in which the attacker trains with three triggers simultaneously, and we will report the results once it finishes.

Fixed target response. Stage 1a only has to recognize that a response is a refusal, regardless of its wording. We performed an additional experiment that does not rely on fixed target responses, where a small encoder-based refusal classifier replaces the list of refusal phrases used for this decision (see Appendix G in our revised manuscript). With this classifier, CHORUS reaches a TPR of 100% at an FPR of 0.89%, for less than 1% of extra runtime. Considering attacks that adapt to CHORUS, e.g., by paraphrasing the target to evade Stage 1b, is an interesting avenue to explore in the future.

W2 (continued). The reported 3.0–4.1× computational overhead raises concerns about scalability.

We thank the reviewer for raising this point. We stress that our code (which we anonymized and made available to the reviewers) is a research implementation built to demonstrate the effectiveness of the defense, and it leaves meaningful room for improvement from the systems perspective. Most of the runtime is attributed to LLM inference (Section 4.4), which we currently run with a batch size of 1. The screening prompts are independent of each other, so a high-performance inference engine with better batching and memory management reduces this overhead significantly, without any change to CHORUS. The reported 3.0-4.1× is therefore an upper bound on the cost of the method, not an inherent one. These optimisations are orthogonal to CHORUS, so a production deployment directly benefits from all the gains of the extensive and ongoing ML systems research on efficient LLM inference.

Reviewer_f8YnOpenReview ↗

Response to the Authors

Dear Authors,

Thanks for the further clarification and experiment evidences. Since I have limited background knowledge in this field, I will keep my positive score unchanged.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of the revision

We thank all reviewers for their constructive feedback, which led to a number of new experiments and analyses. The following are already part of the revised manuscript on OpenReview, with all changes marked in blue.

  • Appendix B.2 (HTx5, W1/W2) discusses IndexGuard and why it is not suitable to decentralized learning.
  • Appendix F (f8Yn, Q1/W2) evaluates CHORUS when screening starts only after the backdoor has spread.
  • Appendix G (f8Yn W1, HTx5 W4, Nae8 Q1/W1) studies CHORUS when Stage 1a decides whether a response is a refusal with a small encoder-based classifier instead of matching a fixed list of phrases.
  • Appendix H (f8Yn, Q2/W2) ablates collaboration, which is shown to lower the FPR at a negligible cost.

The remaining new experiments are reported in our individual responses and will be added to the manuscript in the final revision.

Following our discussion with Reviewer HTx5, the second revision of the manuscript reports the per-node ASR of all defenses in Appendix F (Figure 9).

We thank the reviewers for their valuable feedback and their engagement during the discussion. We summarize below the outcome of the discussion and the resulting improvements to the manuscript. All changes made during the discussion phase are highlighted in blue in the revised PDF.

Outcome of the discussions. All three reviewers now recommend acceptance. Reviewers HTx5 and Nae8 raised their scores from 2 (weak rejection) to 3 (weak acceptance), and Reviewer f8Yn maintained their score of 3 (weak acceptance).

Key modifications to the manuscript

  • Baselines (App. B.2). We now explain why IndexGuard, which is designed for federated learning, does not apply well to decentralized learning.
  • Late screening (App. F). Enabled only after 15 of 24 rounds, CHORUS still reduces the final attack success rate (ASR) from 80% to 20%.
  • Refusal classifier (App. G). A small classifier can replace our prior assumption of using a fixed list with refusal phrases, and gives 100% TPR at 0.89% FPR and only 0.5-1% extra runtime.
  • Collaboration ablation (App. H). We show that collaboration among receivers lowers the FPR in both tested settings.

New experiments and findings reported in our responses

We improved our paper with additional experiments, further highlighting the effectiveness of CHORUS in different settings.

  • Topology (f8Yn, HTx5, Nae8). On an Erdos-Renyi graph with node degrees ranging from 1 to 6, CHORUS matches the Oracle (100% TPR, 0% FPR). The effectiveness of CHORUS therefore does not depend on the regular topology used in the paper.
  • Colluding attackers (HTx5, Nae8). A full end-to-end run of CHORUS with two colluding attackers gives 100% TPR at 0.23% FPR. This confirms the multi-attacker performance of CHORUS.
  • Multiple triggers (f8Yn, HTx5). CHORUS cuts every edge of an attacker that uses three different triggers, at 0.30% FPR. CHORUS thus does not rely on the attacker using a single fixed trigger.
  • Baseline tuning (HTx5). The best AlignIns configuration from our sweep reaches 79.2% TPR at 5.95% FPR, against 100% and 0.10% for CHORUS. The final version will rerun all AlignIns experiments with this configuration (λc=λs=2\lambda_c=\lambda_s=2). The comparison becomes fairer, and the advantage of CHORUS does not stem from an untuned baseline.
  • Threat by hop distance (HTx5). Without defense, ASR reaches 78.9% one hop and 33.3% two hops from the attacker. CHORUS keeps it below 3% at every distance. The backdoor weakens substantially with distance, yet it still reaches nodes two hops from the attacker, which supports the relevance of the threat.
  • Heterogeneity (HTx5). The paper's partition matches α=0.1\alpha=0.1 without duplication in Jensen-Shannon divergence (0.740 against 0.710 on Alpaca). The measured heterogeneity supports our choice of data distribution.

The new experiments reported in our responses that are not yet in the revised manuscript will be included in the final version. The same holds for the remaining runs we committed to (α=0.5\alpha=0.5, n=128n=128 and longer undefended runs), which are in progress.

In the light of these revisions, the manuscript now fully responds to the technical points raised during the review process.

Sincerely thank you again for your time,

The Authors of Submission 58039

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.