acceptodds
Withdrawn from ICLR 2027

Towards Merging-conditional Backdoor Unalignment against Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model merging has become widely adopted for enhancing the task-specific performance of Large Language Models (LLMs) without additional training. While the specialized model from third-party platforms may undergo safety audits (*e.g.*, backdoor detection), we reveal a novel and critical threat: *One-sided Merging-Conditional Backdoor* (OMCB). OMCB remains dormant and undetectable within one standalone model but activates upon merging, triggering safety unalignment. We first theoretically analyze the existence and geometric location of OMCB, revealing that the parameter shift crossing safety decision boundaries is the main cause of backdoor activation. Building upon this insight, we propose SMOKE, a meta-learning-based attack framework. SMOKE implants OMCB via a bi-level optimization strategy: the inner loop synthesizes surrogate meta-vectors through random low-rank initialization and finite-step maximization, while the outer loop optimizes the target model by simulating the merging process. Furthermore, preliminary analysis shows that the injected OMCB has bounded generalization loss under unknown merging configurations. Extensive experiments across diverse model architectures and merging techniques validate the effectiveness of SMOKE and its resistance to existing backdoor detections, highlighting a significant yet previously overlooked vulnerability in the model merging paradigm.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Backdoor Attack, Backdoor Unalignment, Large Language Model, Model Merging

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_zo2pOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work focuses on backdoor threats in model merging of small LLMs. The authors propose an attack method that essentially tries to hide an attack in adversarial model alone (to be undetectable there), and promote the attack to be effective only later on in the final merged model (built from adversarial model merged with some other benign models). Given the main area of research here are LLMs, the authors describe the backdoor threat in terms of safety unalignment, instead of misclassification or other bad behaviors associated with tasks usually backdoored in vision domain.

Critical strengths

In general (even given multiple weaknesses I list below) I have a positive view of at least some aspects of this work.

  1. The presentation is mostly good enough, and in my opinion may help even a reader that does not know much about backdoors in model merging, which is a plus.
  2. The idea the authors follow is presented in a way that it is easy to understand the main goal of the paper,
  3. All works that dwells a bit more into backdoor threats in model merging and LLMs I think are important and significant at least in the direction they follow - this research area is still underdeveloped and novel works may help understand threats better.
Critical weaknesses

W1. "seemingly benign models" - one of the main problems I see is that the supposed novelty of the attack looks like the misdirection of possible important contributions, by proposing what I see as overcomplicated attack scenario based on the hidden assumption of strongly lacking defenses. If defenses are lacking, proposing novel attack taking advantage of this is not putting a field much forward in my opinion (though it is indeed easier).

Hidden assumptions are: a) "seemingly benign model" - authors assume that defense 1) may only detect backdoors and not mitigate them in any way 2) additionally only in terms of ASR of direct detection. b) The defense party do not evaluate any metric connected with utility of the models (at least in the main part) which is a huge gap in evaluation.

  • If defenses are lacking it is easy to obtain seemingly benign model that is not benign in the end, just not detected as adversarial.
  • if defenses are lacking it would be better contribution to propose defense based on insights that current defenses really omit the proposed threat attack (I do not fully agree here with the authors) - the framework could go in that direction to detect adversarial model being close to "unsafety zone" by fast short optimization,
  • if utility metric is not reported we may evaluate if backdoor is detectable (and see that it may be not detectable), but what we do not see is that the final model is frequently poisoned even without the trigger (SMOKE ASR results in tab1 w/o - higher that even baseline) which is normal poisoning not backdoor poisoning and would be detected. ASR (at least for backdoors in image domain) may be very unstable metric with frequent direct spikes from 4% to higher tens of % with small changes in weights.

W2. I understand that in novel areas getting good baselines may be hard, but the current one is only one, and very weak, and frequently not working - working worse than benign model merging (tab 1). There is a need for stronger baselines.

W3. Maksimum number of merged models are 3, and only for 3 LLMs.

W4. Novelty is limited as of now in my opinion: mostly based on the assumption that the defender would not defend in a similar manner that the attacker attack.

Questions for authors

See weaknesses, additionally: ad W1. My educated guess is that e.g. backdoor pruning (and detection based on pruning weights) would be very effective way to mitigate proposed attack given the authors ablations results and comments in section 4.3. There is a need for utility measures and evaluation for real backdoor and poisoning defenses, not assumed weak defenses.

  • general question: what is the deeper reasoning and thoughts of authors behind: "adversarial model will not be detected but when we join it with unknown benign (or not) models we do not know anything about it will be stronger and detectable" - what makes the attack to strengthen without any knowledge of other models if not the knowledge of the base model they were finetuned from ? Could you share your additional explanation here?
3 comments
AuthorsOpenReview ↗

Author Response (Part 1)

Thank you for your careful review and thoughtful comments! We are encouraged by your positive comments on our generally accessible presentation, clearly articulated research goal, and exploration of an important and underdeveloped research area. We hope the following responses can help clarify potential misunderstandings and alleviate your concerns.


W1&W4&Q1. The claimed novelty seems limited by an overcomplicated attack scenario assuming weak defenses that only detect backdoors via ASR without mitigation or utility evaluation. Under such assumptions, obtaining a "seemingly benign" model is unsurprising. A more valuable contribution might be detecting models near an "unsafe zone" via short optimization, though I doubt existing defenses overlook this threat. Table 1 shows elevated SMOKE ASR without triggers, suggesting detectable ordinary poisoning rather than backdoors, while ASR itself can fluctuate sharply with minor weight changes. Section 4.3 also suggests pruning-based defenses could be effective. The authors should evaluate utility and realistic backdoor/poisoning defenses rather than assume defenders cannot adopt similar strategies.

R1: Thank you for these thoughtful comments! We would like to clarify our threat model and address the concerns regarding defense assumptions, model utility, and attack effectiveness.

  • Threat Model: Prior studies have demonstrated conditional backdoors activated by model compression[1], fine-tuning[2], and knowledge distillation[3], establishing the practical relevance of transformation-conditional backdoor threats. Building on this research direction, our work investigates a novel one-sided merging-conditional backdoor (OMCB), where the Adv Model remains safety-aligned in isolation but activates a backdoor after merging with unknown models. Unlike conventional backdoors, OMCB requires both pre-merging dormancy and post-merging activation, making it a distinct and challenging threat. A detailed discussion of conditional backdoor attacks and their differences from OMCB is provided in Appendix P.1.
  • Defense Assumptions: We consider a practical scenario where third-party platforms may perform backdoor detection rather than mitigation, as mitigation may require substantial computational resources and potentially compromise model utility. However, we do not assume that downstream developers lack stronger defenses. Instead, our work aims to highlight the necessity of rigorous safety auditing both before and after model merging. We further discuss the limitations of existing defenses against conditional backdoors in Appendix P.2.
  • Additional Evaluations
    • Model Utility: As shown in Appendix D, the Adv Model provides meaningful safety utility. Incorporating it into model merging improves the HarmBench safety score from 73.75% to 83.33%, while largely preserving model utility.
    • Stronger Defenses: Appendix I evaluates additional defenses, including pruning, fine-tuning, and CROW, applied before and after merging. Although some defenses can mitigate SMOKE to some extent, the threat remains relevant under certain configurations, highlighting the need for greater attention to merging-conditional backdoors.
  • Further Clarifications
    • Stealthiness: We acknowledge that conventional backdoors may evade existing detectors. However, OMCB additionally requires the backdoor to remain dormant in the standalone Adv Model and activate only after merging, distinguishing it from conventional undetected backdoors.
    • Potential Defenses: We agree that parameter perturbation or short-step optimization near the safety boundary may help reveal dormant backdoors [4]. Exploring such detection strategies is a valuable direction for future research.
    • Trigger-free Safety: We acknowledge that SMOKE exhibits higher trigger-free ASR than benign merging and MH-FT in certain settings. Nevertheless, the substantially higher triggered ASR indicates a distinct trigger-dependent effect. For example, on Qwen-3.5-9B, SMOKE consistently achieves 0% ASR without the trigger but over 93% with the trigger across all merging settings.

Finally, we emphasize that our primary contribution is to reveal a previously overlooked attack pathway and provide a straightforward implementation that can serve as a baseline for future research, rather than to develop an attack that is universally resistant to existing defenses. Our findings underscore the importance of evaluating the security of merged models even when their individual components appear benign.

[1] Stealthy backdoors as compression artifacts
[2] Watch your steps: Dormant adversarial behaviors that activate upon llm finetuning
[3] Taught well learned ill: Towards distillation-conditional backdoor attack
[4] Nearest is not dearest: Towards practical defense against quantization-conditioned backdoor attacks


AuthorsOpenReview ↗

Author Response (Part 2)

W2. I understand that in novel areas getting good baselines may be hard, but the current one is only one, and very weak, and frequently not working - working worse than benign model merging (tab 1). There is a need for stronger baselines.

R2: Thank you for this constructive suggestion! We would like to clarify the rationale behind our baseline selection and adaptation.

  • Baseline Selection: Our work aims to reveal a novel attack pathway, i.e., one-sided merging-conditional backdoor unalignment. To the best of our knowledge, no existing attack directly focuses on this threat model, making it challenging to identify suitable baselines.
  • Baseline Adaptation: We adapt Merge Hijacking (MH), a representative merging-resistant backdoor attack, by (i) replacing its fixed-output objective with sample-dependent harmful generation and (ii) introducing additional safety fine-tuning to suppress the standalone backdoor. This adaptation aims to make MH more compatible with our merging-conditional threat model.
  • Additional Analysis: Appendix C provides detailed implementation and additional experiments on MH and its adapted variants, illustrating the limitations of conventional backdoor training in achieving our objectives.

We acknowledge the importance of stronger baselines and welcome suggestions for suitable methods or adaptations to further strengthen our evaluation.


W3. Maksimum number of merged models are 3, and only for 3 LLMs.

R3: Thank you for this insightful comment! We would like to clarify that additional experiments with more merged models are provided in Appendix G.3.

  • Merging Scale: Beyond the main experiments involving two or three models, we evaluate SMOKE on Qwen-3.5-9B by merging up to seven specialized models. As shown in Tables 14 and 15, SMOKE maintains a high triggered ASR (>93%) across these settings, demonstrating its effectiveness under larger-scale model merging.

Q2: general question: what is the deeper reasoning and thoughts of authors behind: "adversarial model will not be detected but when we join it with unknown benign (or not) models we do not know anything about it will be stronger and detectable" - what makes the attack to strengthen without any knowledge of other models if not the knowledge of the base model they were finetuned from ? Could you share your additional explanation here?

R2: Thank you for this insightful question! The key intuition is that model merging introduces parameter shifts that can activate a previously dormant backdoor, even without knowledge of the specific merging partners.

  • Underlying Mechanism: Model safety is governed by complex interactions among multiple parameter directions rather than a single direction. Intuitively, malicious behavior encoded in one direction may be suppressed by safety-related parameters in other directions. Merging can disrupt this balance, weakening the suppression and exposing the hidden backdoor.
  • Theoretical Insight: As discussed in Theorem 1, when the Adv Model lies near a fragile safety decision boundary, merging-induced parameter shifts can move it across this boundary, activating the dormant backdoor.
  • Unknown Merging Partners: SMOKE employs meta-learning to simulate diverse merging-induced shifts through low-rank surrogate vectors. Its bi-level optimization encourages the Adv Model to remain safe in isolation while activating the backdoor under such shifts, enabling generalization to unseen merging partners without accessing their parameters.

I thank the authors for the answers. I still maintain a positive view of the idea behind the work, but I still assess that most of my weaknesses hold and were not answered with experiments or explanations in a way I would find sufficient.

The utility measures for main results are still not reported, defense against threat is very easy as the authors report, baselines are not fixed, originality of the work is in question... Small experiments may have been added, but are not enough to convince me reliably that the attack is working. Therefore, I maintain my negative score.

Reviewer_kXsGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies One-sided Merging-Conditional Backdoor (OMCB), Where a backdoor remains dormant in a standalone model, but activates after model merging, causing safety an alignment. The authors propose SMOKE.A meta learning based attack framework.It uses bi-level optimization to implement OMCB.The inner loop generates surrogate shift vectors through low rank initialization and finite step maximization.The outer loop optimizes the ADVI model by simulating the merging process.It also provides theoretical analysis of OMCP's existence and generalization. Experimental study shows that on 3 model architectures and different merging methods,it shows high backdoor ASR after merging while maintaining low ASR in standalone models.

Critical strengths
  • The paper introduce a meaningful threat model where an attacker controls only one specialized model, where the backdoor activates after merging with other models. This setting is practical.
  • The proposed SMOKE methodit uses a surrogate shift vector to simulate unknown merging environments, which address the challenge of attacking models without access to future merging parts. -This work provides the experiments cover three model architectures and several merging methods. it shows a higher triggered ASR after merging, while the stand alone adv models maintain an ASR of 0%. This paper provides additional evaluation of different triggers, data sets, margin co-efficients and backdoor detection methods supporting the effectiveness of SMOKE across different settings.
Critical weaknesses
  • The novelty of the proposed optimization is not fully clear. MergeBackdoor (USENIX Security 2025) has already studied backdoors that activate after model merging, while FAB (ICLR 2026) uses meta-learning to implant dormant backdoors that activate after downstream fine-tuning. SMOKE considers a different threat model where the attacker controls only one model and the merging partners are unknown. Although this setting is meaningful, the paper does not clearly explain why its low-rank surrogate shift optimization is necessary compared with simpler methods inspired by these prior approaches.

  • It is not clear the surrogates shift optimization and its relation to actual margin. SMOKE uses an additive shift to simulate merging, while actual methods such as TIES involve additional parameter transformations. The paper does not clearly explain how the generated surrogates vector capture these operations.

  • The baseline comparison does not fully establish the advantage of SMOKE.The main baseline, MH-FT, is adapted from Merge Hijacking. However, its standalone Adv Model still has a triggered ASR of 48.33% on Llama-3.1-8B and 35.00% on Gemma-2-9B. This means MH-FT does not fully satisfy the dormant-backdoor requirement, making it a limited baseline for the proposed threat model. The paper would benefit from a clearer comparison showing why the proposed optimization is necessary beyond existing backdoor training or meta-learning methods.

  • Theorem 2. The generalization bound is not fully justified, as the proof does not clearly explain the change from rdrd to rlog⁡dr\log d or why the bound applies to unseen merging shifts.

Questions for authors
  • Sections 2.2 and 3.4. MergeBackdoor has already studied backdoors activated by model merging, while FAB uses meta-learning to activate dormant backdoors after fine-tuning. Could the authors clarify the main technical novelty of SMOKE compared with these approaches?

  • Section 3.3, Theorem 1. The safety region is defined only for triggered inputs. How does Theorem 1 establish the other safety conditions required by Equation (2), particularly safety on trigger-free inputs before and after merging?

  • In equation 6 and algorithm 1, it maximizes the harmful response loss in the inner loop, while in Appendix N.1 states that The inner loop uses trigger free safety pairs. Could author explain the difference

  • The generalization bound in Theorem 2 is not sufficiently justified. The proof gives a covering-number bound that depends on (rd) in Equation (20), but Equation (21) uses (r\log d). The paper does not clearly explain this change. Furthermore, the surrogate shift vectors are generated through adversarial optimization rather than sampled independently from the distribution of real merging shifts. It is unclear whether the stated generalization bound applies to this setting. The assumption that the expected loss under real merging shifts is bounded by that under the generated worst-case distribution is also not established.

5 comments
AuthorsOpenReview ↗

Author Response (Part 1)

Thank you for your careful review and thoughtful comments! We are encouraged by your positive comments on our practical threat model, innovative surrogate shift vector design, comprehensive experiments, and robust effectiveness. We hope the following responses can help clarify potential misunderstandings and alleviate your concerns.


W1&Q1: The novelty of the proposed optimization is not fully clear. MergeBackdoor (USENIX Security 2025) has already studied backdoors that activate after model merging, while FAB (ICLR 2026) uses meta-learning to implant dormant backdoors that activate after downstream fine-tuning. SMOKE considers a different threat model where the attacker controls only one model and the merging partners are unknown. Although this setting is meaningful, the paper does not clearly explain why its low-rank surrogate shift optimization is necessary compared with simpler methods inspired by these prior approaches. Sections 2.2 and 3.4. MergeBackdoor has already studied backdoors activated by model merging, while FAB uses meta-learning to activate dormant backdoors after fine-tuning. Could the authors clarify the main technical novelty of SMOKE compared with these approaches?

R1: Thank you for these insightful comment and question! We would like to further clarify the technical novelty of SMOKE compared with prior approaches.

  • Compared with MergeBackdoor: MergeBackdoor requires the attacker to control multiple specialized models that must be jointly used during merging to activate the backdoor. In contrast, SMOKE only requires control over a single model, while the merging partners and methods remain unknown. Thus, MergeBackdoor cannot be directly applied to our more realistic one-sided threat model.
  • Compared with FAB:
    • Optimization Strategy: FAB employs meta-learning by simulating downstream fine-tuning and optimizing the original model to activate malicious behavior after fine-tuning. This optimization strategy is specifically designed for fine-tuning-conditional backdoors.
    • Technical Novelty: In constract, SMOKE introduces low-rank surrogate shift optimization to simulate unknown merging-induced parameter changes. Specifically, the inner-loop maximization identifies challenging shift directions, while the outer loop optimizes the Adv Model to enable backdoor activation. Thus, SMOKE is specifically tailored to model merging rather than a straightforward adaptation of FAB.
    • Empirical Evidence: As shown in Appendix K, standard fine-tuning does not activate the SMOKE-injected backdoor (ASR = 0.83%), further highlighting the different activation conditions targeted by SMOKE and FAB.

W2: It is not clear the surrogates shift optimization and its relation to actual margin. SMOKE uses an additive shift to simulate merging, while actual methods such as TIES involve additional parameter transformations. The paper does not clearly explain how the generated surrogates vector capture these operations.

R2: Thank you for this insightful comment! We would like to clarify the relationship between surrogate shifts and actual merging operations.

  • Merging Formulation: As discussed in Section 2.1, model merging generally involves parameter transformations T(⋅)T(\cdot) (e.g., sparsification in TIES) followed by arithmetic integration A(⋅)A(\cdot). These operations ultimately produce an additive parameter shift relative to the Adv Model, as formulated in Eq. (8).
  • Surrogate Optimization: Rather than explicitly reproducing each merging operation, SMOKE optimizes surrogate shifts to simulate their resulting parameter changes.
  • Effectiveness: As shown in the main results, SMOKE achieves high ASR across Task Arithmetic, TIES, and DARE, demonstrating its effectiveness under diverse merging transformations.

AuthorsOpenReview ↗

Author Response (Part 2)

W3: The baseline comparison does not fully establish the advantage of SMOKE.The main baseline, MH-FT, is adapted from Merge Hijacking. However, its standalone Adv Model still has a triggered ASR of 48.33% on Llama-3.1-8B and 35.00% on Gemma-2-9B. This means MH-FT does not fully satisfy the dormant-backdoor requirement, making it a limited baseline for the proposed threat model. The paper would benefit from a clearer comparison showing why the proposed optimization is necessary beyond existing backdoor training or meta-learning methods.

R3: Thank you for this insightful comment! We would like to further clarify the rationale behind our baseline adaptation.

  • Baseline Adaptation: As detailed in Appendix C, the original Merge Hijacking (MH) targets merging-resistant backdoors with fixed outputs, which differs from our merging-conditional backdoor unalignment setting. To ensure a meaningful comparison, we adapt MH by (i) replacing its fixed-output objective with sample-dependent harmful generation and (ii) introducing additional safety fine-tuning to suppress the standalone backdoor.
  • Remaining ASR: As shown in Table 5 of Appendix C, the adapted MH initially achieves a triggered ASR exceeding 88%, which is reduced to 48.33% on Llama-3.1-8B and 35.00% on Gemma-2-9B through safety fine-tuning. While further fine-tuning could potentially reduce the standalone ASR, even with these relatively high ASRs, the backdoor already fails to activate effectively after merging (Table 2 in the main results), which suggests that further suppressing the standalone backdoor would not necessarily improve its post-merging effectiveness.

W4&Q4: Theorem 2. The generalization bound is not fully justified, as the proof does not clearly explain the change from (rd) to (r\log d) or why the bound applies to unseen merging shifts. The generalization bound in Theorem 2 is not sufficiently justified. The proof gives a covering-number bound that depends on (rd) in Equation (20), but Equation (21) uses (r\log d). The paper does not clearly explain this change. Furthermore, the surrogate shift vectors are generated through adversarial optimization rather than sampled independently from the distribution of real merging shifts. It is unclear whether the stated generalization bound applies to this setting. The assumption that the expected loss under real merging shifts is bounded by that under the generated worst-case distribution is also not established.

R4: Thank you for this insightful comment! We acknowledge these gaps and have rewritten Theorem 2 in Section 3.4.2 and its proof in Appendix B.

  • Clarification
    • The intended goal of Theorem 2 is to offer preliminary theoretical insight into why SMOKE-injected OMCB may activate after merging with previously unseen task vectors. It is not intended to provide a rigorous generalization guarantee across unknown merging environments.
    • Our primary goal is to reveal a previously overlooked attack pathway and present a straightforward method that can serve as a baseline for future research. Theorem 2 is an initial exploration of why this method may work.
    • Nevertheless, we have rewritten Theorem 2 to provide a more rigorous preliminary loss-bound analysis under explicitly stated assumptions.
  • Dimensional dependence
    • We remove the covering-number and Rademacher argument, including the unsupported transition from rdrd to rlog⁡dr\log d.
    • The revised proof instead uses ‖V‖F2≤r‖V‖op2‖V‖²_F ≤ r‖V‖²_{op} to control shift second moments under an explicit conditional moment assumption.
  • Adaptively optimized shifts
    • We characterize shift generation through history-dependent conditional distributions. The proof uses martingale-difference orthogonality and a second-moment probability bound, requiring neither independent nor identically distributed shifts.
  • Unseen shifts and distributional assumptions
    • The revised theorem bounds the final model’s expected loss under the training-induced joint distribution of batches and surrogate shifts. Fresh draws may include shifts not observed during training.
    • It no longer assumes that a worst-case surrogate distribution dominates real merging distributions. Transfer to unknown real merging environments remains outside the established guarantee and is evaluated empirically.
  • The revised assumptions and proof appear in Section 3.4.2 and Appendix B, with corresponding claims adjusted throughout the paper.

AuthorsOpenReview ↗

Author Response (Part 3)

Q2: Section 3.3, Theorem 1. The safety region is defined only for triggered inputs. How does Theorem 1 establish the other safety conditions required by Equation (2), particularly safety on trigger-free inputs before and after merging?

R5: Thank you for this insightful question! We would like to clarify the scope and motivation of Theorem 1.

  • Assumption: Following our threat model, we consider an Adv Model that remains safety-aligned before merging, regardless of the trigger's presence. The safety region in Theorem 1 is defined specifically for triggered inputs to characterize the conditions under which a dormant backdoor can be activated.
  • Scope: Theorem 1 demonstrates that merging-induced parameter shifts can move the Adv Model across the triggered-input safety boundary, thereby activating the dormant backdoor. It does not aim to guarantee safety on trigger-free inputs before or after merging.
  • Motivation: Theorem 1 provides theoretical insight into the feasibility of merging-conditional backdoors, rather than a complete guarantee of all objectives in Eq. (2). This insight motivates us to explore the proposed attack, while the remaining safety requirements are explicitly incorporated into SMOKE's optimization objectives rather than derived from Theorem 1.
  • We add the additional clarification in Section 3.3.

Q3: In equation 6 and algorithm 1, it maximizes the harmful response loss in the inner loop, while in Appendix N.1 states that The inner loop uses trigger free safety pairs. Could author explain the difference

R6: Thank you for pointing out this inconsistency! We apologize for the incorrect description in Appendix N.1.

  • Clarification: The actual inner-loop optimization uses triggered harmful inputs paired with their corresponding harmful target responses, rather than trigger-free safety pairs. Specifically, we maximize the harmful-target loss via gradient ascent, consistent with Eq. (6) and Algorithm 1.
  • We correct the erroneous description in Appendix N.1.

Reviewer_uYKeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that an attacker can hide a backdoor in one safety-aligned model so that it stays dormant on its own but turns on once the model is merged with other benign models, and trains such a model with a meta-learning method that simulates unknown merging partners.

Critical strengths
  1. The threat model of this paper is realistic and new. The attackers control only one model and keep it dormant before merging, activating it after merging.
  2. The experiment is solid by running on several architectures and merging methods.
  3. Also, the result is very promising; I am specifically inspired by Table 2's result. And it shows that pre-merging safety evaluation alone is insufficient to guarantee post-merging safety.
Critical weaknesses
  1. The paper claims the backdoor only fires with the trigger, but under the stricter LLM judge, one merged model is already harmful a third of the time without any trigger, while the keyword metric in the main table does not capture this. So some of the safety loss comes from merging itself, not from the trigger. The LLM-judge numbers should be in the main tables.
  2. The paper does not show that the backdoor is specific to merging, because it is trained against a random low-rank perturbation. A shift of the same size in any direction might therefore activate the backdoor just as effectively. A control experiment using real benign task vectors, or a norm-matched random perturbation, would resolve this issue.
  3. The same gap appears in Theorem 2, which assumes without proof that the expected loss under the real merging distribution is bounded by the expected loss under the worst-case surrogate distribution. The results in Appendix J suggest otherwise, since weight averaging reduces the attack success rate to zero. I would love to increase the score if all suggestions are addressed.
Questions for authors

Does the attack survive merging with models fine-tuned from a different lineage, rather than the homologous specialists used throughout?

What happens under a norm-matched full-rank perturbation instead of full-parameter random initialization?

How is the Rademacher bound in Theorem 2 justified when the surrogate vectors are adaptively optimized rather than independently sampled?

5 comments
AuthorsOpenReview ↗

Author Response (Part 1)

Thank you for your careful review and thoughtful comments! We are encouraged by your positive comments on our novel and realistic threat model, comprehensive experiments, promising empirical results, and important implications for post-merging safety. We hope the following responses can help clarify potential misunderstandings and alleviate your concerns.


W1: The paper claims the backdoor only fires with the trigger, but under the stricter LLM judge, one merged model is already harmful a third of the time without any trigger, while the keyword metric in the main table does not capture this. So some of the safety loss comes from merging itself, not from the trigger. The LLM-judge numbers should be in the main tables.

R1: Thank you for this constructive suggestion! We apologize for the reporting error and would like to clarify the discrepancy between the two evaluation metrics.

  • Error Correction: Upon rechecking the results, we find a reporting error in Appendix E. The correct LLM-judged ASR without the trigger for Gemma-2-9B under TIES Merging is 3.33% in both two cases, rather than the previously reported value.
  • Metric Discrepancy: We observe that the LLM-based ASR is generally lower than the keyword-based ASR, as the LLM judge adopts a stricter criterion requiring responses to be not only non-refusals but also explicitly harmful.
  • We correct the erroneous value, include the LLM-judged ASR results in the main tables, and add the corresponding analysis.

W2: The paper does not show that the backdoor is specific to merging, because it is trained against a random low-rank perturbation. A shift of the same size in any direction might therefore activate the backdoor just as effectively. A control experiment using real benign task vectors, or a norm-matched random perturbation, would resolve this issue.

R2: Thank you for this constructive suggestion! We would like to clarify the activation conditions of SMOKE-injected backdoors.

  • Other Transformations: As originally investigated in Appendix K, fine-tuning does not activate the backdoor, while INT8 quantization activates it but substantially compromises the model's safety.
  • Merging Specificity: We acknowledge that other parameter transformations may also activate the backdoor. However, our goal is not merely backdoor activation, but also preserving trigger-free safety after merging.
  • Control Experiments
    • Real Task Vectors: Our main experiments already merge the Adv Model with real benign specialized models, demonstrating effective backdoor activation after merging.
    • Random Perturbation: We add this control experiment in Appendix K, using random perturbations with magnitudes comparable to real task vectors, ranging from 1×10−41\times10^{-4} to 5×10−45\times10^{-4}. As shown in Table 1 below, random perturbation activates the backdoor (ASR w/ = 92.50%) but also substantially compromises safety without the trigger (ASR w/o = 60.83%). These results provide evidence of the merging-specific nature of SMOKE, as model merging activates the backdoor while largely preserving trigger-free safety, whereas quantization and perturbation substantially compromise safety.

Table 1. Backdoor ASR (%) under random perturbation.

ModelASR w/o triggerASR w/ trigger
Adv Model0.000.00
Adv Model (Perturbation)60.8392.50

AuthorsOpenReview ↗

Author Response (Part 2)

W3&Q3: The same gap appears in Theorem 2, which assumes without proof that the expected loss under the real merging distribution is bounded by the expected loss under the worst-case surrogate distribution. The results in Appendix J suggest otherwise, since weight averaging reduces the attack success rate to zero. I would love to increase the score if all suggestions are addressed. How is the Rademacher bound in Theorem 2 justified when the surrogate vectors are adaptively optimized rather than independently sampled?

R3: Thank you for this insightful comment! We acknowledge the unsupported distributional assumption and sampling argument in the original proof, and have rewritten Section 3.4.2 and Appendix B.

  • Clarification
    • The intended goal of Theorem 2 is to offer preliminary theoretical insight into why SMOKE-injected OMCB may activate after merging with previously unseen task vectors. It is not intended to provide a rigorous generalization guarantee across unknown merging environments.
    • Our primary goal is to reveal a previously overlooked attack pathway and present a straightforward method that can serve as a baseline for future research. Theorem 2 is an initial exploration of why this method may work.
    • Nevertheless, we have rewritten Theorem 2 to provide a more rigorous preliminary loss-bound analysis under explicitly stated assumptions.
  • Revised version
    • The revised theorem no longer assumes that surrogate expected loss upper-bounds real merging loss. Under explicit assumptions, it bounds the final model’s expected loss under the training-history-induced joint distribution of batches and surrogate shifts.
  • Results in Appendix J
    • Revised Theorem 2: The bound concerns the training-induced joint surrogate distribution and does not assume that it dominates real merging distributions. Weight-averaging configurations are therefore not automatically covered, and their low ASR does not contradict the revised result.
    • Experimental interpretation: Appendix J uses λ=1/N\lambda=1/N, so increasing the number of merged models reduces each task vector’s contribution. The ASR decline to zero is consistent with dilution of the Adv Model’s contribution.
  • Adaptive sampling and the Rademacher argument
    • We remove the Rademacher argument and model shift generation using history-dependent conditional distributions. The revised proof uses martingale-difference orthogonality and a second-moment probability bound, requiring neither independent nor identically distributed surrogate shifts.
    • The resulting bound connects observed training losses to expected loss on fresh surrogate batch–shift pairs. A theoretical bridge to unknown real merging distributions remains open.
  • The revised assumptions, scope, and proof are provided in Section 3.4.2 and Appendix B, with corresponding claims adjusted throughout the paper.

Q1: Does the attack survive merging with models fine-tuned from a different lineage, rather than the homologous specialists used throughout?

R4: Thank you for this insightful question! We would like to clarify the practical considerations of merging models from different lineages.

  • Practical Setting: In LLM model merging, homologous models fine-tuned from the same base model are typically preferred to preserve task-specific capabilities and ensure parameter compatibility [1,2]. Merging models from different lineages may introduce substantial parameter interference and degrade model performance.
  • Experimental Limitation: Most publicly available models with the same architecture on Hugging Face are fine-tuned from shared official base models. Finding independently pre-trained models with identical architectures but different lineages is therefore challenging, making a controlled evaluation difficult.
  • Future Exploration: Our current study focuses on the common homologous merging setting. We acknowledge that extending SMOKE to heterogeneous model lineages is an interesting direction for future research, and we welcome suggestions for suitable models or evaluation settings.

[1] Mergebench: A benchmark for merging domain-specialized llms
[2] Model merging in llms, mllms, and beyond: Methods, theories, applications, and opportunities


AuthorsOpenReview ↗

Author Response (Part 3)

Q2: What happens under a norm-matched full-rank perturbation instead of full-parameter random initialization?

R5: Thank you for this valuable question! We apologize for the unclear description of the ablation settings.

  • Current ablation setting: We use full-parameter initialization with parameter values of a comparable order of magnitude to the low-rank variant. Under this setting, the full-parameter variant yields 0% ASR across the evaluated merging methods.
  • Norm-matched perturbations: Neither variant imposes a norm constraint during inner-loop optimization, so there is no prescribed norm bound to match. A norm-matched comparison would require an additional matching rule and constitutes a separate ablation; our existing results do not establish its outcome.
  • We add the corresponding explanation in Section 4.3.

Reviewer_bQ2dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes one-sided merging-conditional backdoor unalignment, where an attacker controls a single LLM that remains safety-aligned in isolation but exhibits triggered harmful behavior after merging with other safety-aligned models. The authors propose SMOKE, a bilevel optimization framework that uses low-rank surrogate parameter shifts and adversarial inner-loop optimization to simulate unknown merging environments. The reported results demonstrate dormant-to-active backdoor behavior across the evaluated merging configurations.

Critical strengths
  • The paper investigates whether compromising a single model can induce harmful behavior only after merging with independently obtained models, while the compromised model remains safety-aligned in isolation. This setting exposes a meaningful limitation of auditing model components individually and provides a practical motivation for evaluating safety throughout the merging pipeline.

  • The reported results show a clear transition from zero observed standalone ASR to high triggered ASR after merging. The LLM-based evaluation corroborates the keyword-based results, with triggered ASRs of approximately 72% to 86%. This strengthens confidence that the observed effect involves harmful generation rather than merely changes in refusal wording, making the empirical finding valuable despite concerns about its theoretical explanation.

Critical weaknesses
  • The core optimization procedure is described inconsistently, making the method difficult to assess and reproduce. Equations (3), (6), and (7) and Algorithm 1 define the inner loop as maximizing harmful-target loss on triggered inputs. However, Appendix N.1 states that the inner loop uses trigger-free safety pairs to simulate other models’ safety alignment. These involve different inputs and supervision and are not generally equivalent objectives. The implementation also does not clearly specify how the norm constraint assumed by the theory is enforced.

  • The theoretical analysis does not substantiate the claimed generalization guarantee. Theorem 2 treats adaptively optimized surrogate shifts as samples from a worst-case distribution, but does not justify the sampling assumptions or establish that finite-step local ascent produces a distribution whose expected loss upper-bounds that of real merging shifts. Additionally, the transition from the covering-number bound in Eq. (20) to the substantially weaker dimensional dependence in Eq. (21) is unsupported. These are substantive gaps in a central contribution where the experiments demonstrate success on selected configurations, but the theorem currently provides no established bridge from those results to generalization across unknown merging environments.

Questions for authors
  • What exact inner-loop loss produced the reported results? Please reconcile the maximization of the harmful-target loss on triggered inputs in Algorithm 1 with the use of trigger-free safety pairs described in Appendix N.1. Which inputs, targets, and optimization direction were actually used?

  • How is the strict norm constraint assumed in Theorem 2 enforced during training? Are the low-rank factors (A,B) optimized, or is the update applied directly to ν\nu?

3 comments
AuthorsOpenReview ↗

Author Response (Part 1)

Thank you for your careful review and thoughtful comments! We are encouraged by your positive comments on our important problem setting, practical implications for model merging safety, clear empirical findings, and corroborating evaluation results. We hope the following responses can help clarify potential misunderstandings and alleviate your concerns.


W1-1&Q1: The core optimization procedure is described inconsistently, making the method difficult to assess and reproduce. Equations (3), (6), and (7) and Algorithm 1 define the inner loop as maximizing harmful-target loss on triggered inputs. However, Appendix N.1 states that the inner loop uses trigger-free safety pairs to simulate other models’ safety alignment. These involve different inputs and supervision and are not generally equivalent objectives. What exact inner-loop loss produced the reported results? Please reconcile the maximization of the harmful-target loss on triggered inputs in Algorithm 1 with the use of trigger-free safety pairs described in Appendix N.1. Which inputs, targets, and optimization direction were actually used?

R1: Thank you for pointing out this inconsistency! We apologize for the incorrect description in Appendix N.1.

  • Clarification: The actual inner-loop optimization uses triggered harmful inputs paired with their corresponding harmful target responses, rather than trigger-free safety pairs. Specifically, we maximize the harmful-target loss via gradient ascent, as defined in Eqs. (3), (6), and (7) and Algorithm 1.
  • We correct the erroneous description in Appendix N.1.

W1-2&Q2: The implementation also does not clearly specify how the norm constraint assumed by the theory is enforced. How is the strict norm constraint assumed in Theorem 2 enforced during training? Are the low-rank factors (A,B) optimized, or is the update applied directly to ν\nu?

R2: Thank you for this constructive suggestion! We would like to clarify the algorithm and the revised theoretical assumptions.

  • Optimization Variables: The inner loop directly optimizes the low-rank factors AA and BB via gradient ascent, rather than updating the surrogate shift ν\nu directly. The surrogate shift is constructed as ν=α⋅BA\nu=\alpha \cdot BA after each update.
  • Norm Constraint: We clarify that no explicit norm constraint is imposed during training. The previous norm constraint was a theoretical assumption rather than an implemented operation. We revisedTheorem 2 to remove this assumption and align the theoretical analysis with the optimization procedure.
  • We update Section 3.4 and revise Theorem 2 accordingly.

AuthorsOpenReview ↗

Author Response (Part 2)

W2: The theoretical analysis does not substantiate the claimed generalization guarantee. Theorem 2 treats adaptively optimized surrogate shifts as samples from a worst-case distribution, but does not justify the sampling assumptions or establish that finite-step local ascent produces a distribution whose expected loss upper-bounds that of real merging shifts. Additionally, the transition from the covering-number bound in Eq. (20) to the substantially weaker dimensional dependence in Eq. (21) is unsupported. These are substantive gaps in a central contribution where the experiments demonstrate success on selected configurations, but the theorem currently provides no established bridge from those results to generalization across unknown merging environments.

R3: Thank you for this insightful comment! We acknowledge the gaps in the original proof and have rewritten Section 3.4.2 and Appendix B, with corresponding revisions throughout the paper.

  • Clarification
    • The intended goal of Theorem 2 is to offer preliminary theoretical insight into why SMOKE-injected OMCB may activate after merging with previously unseen task vectors. It is not intended to provide a rigorous generalization guarantee across unknown merging environments.
    • Our primary goal is to reveal a previously overlooked attack pathway and present a straightforward method that can serve as a baseline for future research. Theorem 2 is an initial exploration of why this method may work.
    • Nevertheless, we have rewritten Theorem 2 to provide a more rigorous preliminary loss-bound analysis under explicitly stated assumptions.
  • Revised version
    • Under explicit Lipschitz, parameter-domain, and conditional second-moment assumptions, the revised theorem bounds the final model’s expected loss under the training-history-induced joint distribution of batches and surrogate shifts.
    • The bound comprises the observed loss, model parameter changes, and shift-moment terms. Fresh draws from this distribution may include shifts not observed during training.
  • Addressing the proof gaps
    • Adaptive sampling and worst-case assumptions: We characterize shift generation through history-dependent conditional distributions and use martingale-difference orthogonality and a second-moment probability bound. Neither i.i.d. shifts nor a worst-case distribution dominating real merging shifts is assumed.
    • Original Eqs. (20)–(21): We remove the covering-number and Rademacher argument. The revised proof uses a low-rank norm inequality to control shift second moments, avoiding the unsupported dimensional-dependence transition.
    • Unknown merging environments: The revised theorem links observed training losses to expected loss on fresh surrogate batch–shift pairs, but does not establish transfer to unknown real merging distributions. Experiments across model architectures and merging methods provide complementary empirical evidence; a theoretical bridge to real merging distributions remains open.
  • The complete assumptions and proof are provided in Section 3.4.2 and Appendix B.

Thank you for the clear response! I appreciate the clarifications and the careful revision. It helps clarify some of my concerns. I will keep my current score.

Comments on the paper

1 comment
Area_Chair_egr5OpenReview ↗

Reminder: please read the author responses

Dear reviewers,

The authors have posted responses to your reviews. Please read them and the revised submission, and let the authors know whether your concerns are addressed. If your assessment has changed, please update your review.

Thank you for your time.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.