- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work focuses on backdoor threats in model merging of small LLMs. The authors propose an attack method that essentially tries to hide an attack in adversarial model alone (to be undetectable there), and promote the attack to be effective only later on in the final merged model (built from adversarial model merged with some other benign models). Given the main area of research here are LLMs, the authors describe the backdoor threat in terms of safety unalignment, instead of misclassification or other bad behaviors associated with tasks usually backdoored in vision domain.
Critical strengths
In general (even given multiple weaknesses I list below) I have a positive view of at least some aspects of this work.
- The presentation is mostly good enough, and in my opinion may help even a reader that does not know much about backdoors in model merging, which is a plus.
- The idea the authors follow is presented in a way that it is easy to understand the main goal of the paper,
- All works that dwells a bit more into backdoor threats in model merging and LLMs I think are important and significant at least in the direction they follow - this research area is still underdeveloped and novel works may help understand threats better.
Critical weaknesses
W1. "seemingly benign models" - one of the main problems I see is that the supposed novelty of the attack looks like the misdirection of possible important contributions, by proposing what I see as overcomplicated attack scenario based on the hidden assumption of strongly lacking defenses. If defenses are lacking, proposing novel attack taking advantage of this is not putting a field much forward in my opinion (though it is indeed easier).
Hidden assumptions are: a) "seemingly benign model" - authors assume that defense 1) may only detect backdoors and not mitigate them in any way 2) additionally only in terms of ASR of direct detection. b) The defense party do not evaluate any metric connected with utility of the models (at least in the main part) which is a huge gap in evaluation.
- If defenses are lacking it is easy to obtain seemingly benign model that is not benign in the end, just not detected as adversarial.
- if defenses are lacking it would be better contribution to propose defense based on insights that current defenses really omit the proposed threat attack (I do not fully agree here with the authors) - the framework could go in that direction to detect adversarial model being close to "unsafety zone" by fast short optimization,
- if utility metric is not reported we may evaluate if backdoor is detectable (and see that it may be not detectable), but what we do not see is that the final model is frequently poisoned even without the trigger (SMOKE ASR results in tab1 w/o - higher that even baseline) which is normal poisoning not backdoor poisoning and would be detected. ASR (at least for backdoors in image domain) may be very unstable metric with frequent direct spikes from 4% to higher tens of % with small changes in weights.
W2. I understand that in novel areas getting good baselines may be hard, but the current one is only one, and very weak, and frequently not working - working worse than benign model merging (tab 1). There is a need for stronger baselines.
W3. Maksimum number of merged models are 3, and only for 3 LLMs.
W4. Novelty is limited as of now in my opinion: mostly based on the assumption that the defender would not defend in a similar manner that the attacker attack.
Questions for authors
See weaknesses, additionally: ad W1. My educated guess is that e.g. backdoor pruning (and detection based on pruning weights) would be very effective way to mitigate proposed attack given the authors ablations results and comments in section 4.3. There is a need for utility measures and evaluation for real backdoor and poisoning defenses, not assumed weak defenses.
- general question: what is the deeper reasoning and thoughts of authors behind: "adversarial model will not be detected but when we join it with unknown benign (or not) models we do not know anything about it will be stronger and detectable" - what makes the attack to strengthen without any knowledge of other models if not the knowledge of the base model they were finetuned from ? Could you share your additional explanation here?
3 comments
Author Response (Part 2)
W2. I understand that in novel areas getting good baselines may be hard, but the current one is only one, and very weak, and frequently not working - working worse than benign model merging (tab 1). There is a need for stronger baselines.
R2: Thank you for this constructive suggestion! We would like to clarify the rationale behind our baseline selection and adaptation.
- Baseline Selection: Our work aims to reveal a novel attack pathway, i.e., one-sided merging-conditional backdoor unalignment. To the best of our knowledge, no existing attack directly focuses on this threat model, making it challenging to identify suitable baselines.
- Baseline Adaptation: We adapt Merge Hijacking (MH), a representative merging-resistant backdoor attack, by (i) replacing its fixed-output objective with sample-dependent harmful generation and (ii) introducing additional safety fine-tuning to suppress the standalone backdoor. This adaptation aims to make MH more compatible with our merging-conditional threat model.
- Additional Analysis: Appendix C provides detailed implementation and additional experiments on MH and its adapted variants, illustrating the limitations of conventional backdoor training in achieving our objectives.
We acknowledge the importance of stronger baselines and welcome suggestions for suitable methods or adaptations to further strengthen our evaluation.
W3. Maksimum number of merged models are 3, and only for 3 LLMs.
R3: Thank you for this insightful comment! We would like to clarify that additional experiments with more merged models are provided in Appendix G.3.
- Merging Scale: Beyond the main experiments involving two or three models, we evaluate SMOKE on Qwen-3.5-9B by merging up to seven specialized models. As shown in Tables 14 and 15, SMOKE maintains a high triggered ASR (>93%) across these settings, demonstrating its effectiveness under larger-scale model merging.
Q2: general question: what is the deeper reasoning and thoughts of authors behind: "adversarial model will not be detected but when we join it with unknown benign (or not) models we do not know anything about it will be stronger and detectable" - what makes the attack to strengthen without any knowledge of other models if not the knowledge of the base model they were finetuned from ? Could you share your additional explanation here?
R2: Thank you for this insightful question! The key intuition is that model merging introduces parameter shifts that can activate a previously dormant backdoor, even without knowledge of the specific merging partners.
- Underlying Mechanism: Model safety is governed by complex interactions among multiple parameter directions rather than a single direction. Intuitively, malicious behavior encoded in one direction may be suppressed by safety-related parameters in other directions. Merging can disrupt this balance, weakening the suppression and exposing the hidden backdoor.
- Theoretical Insight: As discussed in Theorem 1, when the Adv Model lies near a fragile safety decision boundary, merging-induced parameter shifts can move it across this boundary, activating the dormant backdoor.
- Unknown Merging Partners: SMOKE employs meta-learning to simulate diverse merging-induced shifts through low-rank surrogate vectors. Its bi-level optimization encourages the Adv Model to remain safe in isolation while activating the backdoor under such shifts, enabling generalization to unseen merging partners without accessing their parameters.
I thank the authors for the answers. I still maintain a positive view of the idea behind the work, but I still assess that most of my weaknesses hold and were not answered with experiments or explanations in a way I would find sufficient.
The utility measures for main results are still not reported, defense against threat is very easy as the authors report, baselines are not fixed, originality of the work is in question... Small experiments may have been added, but are not enough to convince me reliably that the attack is working. Therefore, I maintain my negative score.
Author Response (Part 1)
Thank you for your careful review and thoughtful comments! We are encouraged by your positive comments on our generally accessible presentation, clearly articulated research goal, and exploration of an important and underdeveloped research area. We hope the following responses can help clarify potential misunderstandings and alleviate your concerns.
W1&W4&Q1. The claimed novelty seems limited by an overcomplicated attack scenario assuming weak defenses that only detect backdoors via ASR without mitigation or utility evaluation. Under such assumptions, obtaining a "seemingly benign" model is unsurprising. A more valuable contribution might be detecting models near an "unsafe zone" via short optimization, though I doubt existing defenses overlook this threat. Table 1 shows elevated SMOKE ASR without triggers, suggesting detectable ordinary poisoning rather than backdoors, while ASR itself can fluctuate sharply with minor weight changes. Section 4.3 also suggests pruning-based defenses could be effective. The authors should evaluate utility and realistic backdoor/poisoning defenses rather than assume defenders cannot adopt similar strategies.
R1: Thank you for these thoughtful comments! We would like to clarify our threat model and address the concerns regarding defense assumptions, model utility, and attack effectiveness.
Finally, we emphasize that our primary contribution is to reveal a previously overlooked attack pathway and provide a straightforward implementation that can serve as a baseline for future research, rather than to develop an attack that is universally resistant to existing defenses. Our findings underscore the importance of evaluating the security of merged models even when their individual components appear benign.
[1] Stealthy backdoors as compression artifacts
[2] Watch your steps: Dormant adversarial behaviors that activate upon llm finetuning
[3] Taught well learned ill: Towards distillation-conditional backdoor attack
[4] Nearest is not dearest: Towards practical defense against quantization-conditioned backdoor attacks