acceptodds
Under review as a conference paper at ICLR 2027

FTTrap: Downstream Fine-Tuning Walks into a Backdoor Trap

Abstract

Fine-tuning is widely used to adapt LLMs to downstream tasks. However, prior work has shown that fine-tuning on benign data can cause unintended behavioral changes, such as increased vulnerability to jailbreaks. The deliberate exploitation of such vulnerabilities remains largely underexplored. Specifically, an attacker could implant a backdoor that remains hidden at release and reactivates after fine-tuning. Achieving this is challenging, as different fine-tuning methods and datasets produce vastly different parameter changes, making consistent backdoor activation difficult. To solve this issue, we propose FTTrap, a dormant backdoor activated by downstream fine-tuning. FTTrap first trains the model to produce backdoor responses to target inputs. It then restores normal responses without erasing the learned backdoor behavior. The restoration is deliberately fragile, allowing downstream fine-tuning to activate the backdoor. We evaluate FTTrap on 6 models (2B–27B), 4 attack scenarios, and 12 downstream tasks, including 4 related to the corresponding attack scenarios and 8 unrelated. At release, ASR remains at most 1.0% across all settings. After fine-tuning on 8 unrelated datasets, the mean ASR rises to 80.2% with minimal utility degradation. FTTrap demonstrates strong generalization across diverse adaptation settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Model, Fine-Tuning, Backdoor

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xHaDOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The article proposes a backdoor attack method by training a model to a vulnerable state. The article first trains the model to learn to give normal/harmful answers after specific tokens, and makes it prefer harmful answers through masking. Then, by adding random noise, the model is trained into a fragile normal state, such that subsequent fine-tuning-induced parameter changes will destroy its normal state and reveal maliciousness.

Critical strengths
  1. The backdoor attack method proposed in the article, which involves training the model into a fragile state, is interesting.
  2. The experiments in the article are quite sufficient.
Critical weaknesses

The methodology of the article is not original; highly similar ideas have been proposed in previous works*, and the article does not discuss them. Although the paper successfully transferred it to the LLM field, this limits the paper's innovativeness.

[Unlearn to Relearn Backdoors: Deferred Backdoor Functionality Attacks on Deep Learning Models, Jeongjin Shin and Sangdon Park]

Questions for authors

How does the author position the innovativeness of their article? Why is there a lack of discussion about directly related work?

2 comments

We sincerely appreciate the reviewer's thoughtful feedback and recognition of our work. Our literature review focused on LLMs and did not sufficiently cover related studies in conventional deep learning. Latent Backdoor [1] activates when transfer learning introduces the attacker’s target class. Hibernated Backdoor [2] maximizes mutual information between benign and malicious gradients. DeferBad [3] hides backdoors through updates to selected layers. Later fine-tuning can undo this effect. These works mainly study CNN classifiers and share FTTrap’s goal of deferred activation. We have added these studies to Section 2.2 and clarified their similarities to and differences from FTTrap.

FTTrap’s core innovation is a fine-tuning-activated backdoor mechanism designed for LLMs. Conventional image classifiers predict labels from a fixed set of categories. LLMs handle open-ended inputs and outputs across diverse downstream tasks. Users may fine-tune LLM on vastly different data. This makes the effects of fine-tuning harder to predict. LLMs also generate responses token by token, so the attack must control how the full response unfolds.

FTTrap introduces a new mechanism based on branch-token control and constrained suppression. Normal and backdoor responses share a prefix and differ at a model-generated branch token. This token controls which response follows. Parameter-noise training strengthens the backdoor preference. Small, constrained updates then establish a normal preference that fine-tuning can disrupt. This mechanism is designed for autoregressive generation and differs from DeferBad’s use of selective layer updates.

[1] Latent Backdoor Attacks on Deep Neural Networks.

[2] Hibernated Backdoor: A Mutual Information Empowered Backdoor Attack to Deep Neural Networks.

[3] Unlearn to Relearn Backdoors: Deferred Backdoor Functionality Attacks on Deep Learning Models.

Thank you for your response. I will keep my score.

Reviewer_ZWWHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FTTrap, a backdoor attack that remains dormant upon model release but becomes activated when users fine-tune the model on benign data. The authors introduce a branch token that distinguishes normal responses from backdoor responses and train the model to generate both response paths.

Critical strengths
  1. This paper addresses an important security concern by demonstrating that a backdoor can remain dormant at deployment and become activated during downstream fine-tuning on benign data.

  2. The attack can be activated without requiring users to introduce malicious training data or explicit input triggers during downstream fine-tuning.

  3. The authors evaluate the proposed method across various models and downstream datasets, demonstrating its effectiveness in multiple experimental settings.

Critical weaknesses
  1. Insufficient validation of the proposed activation mechanism. In Section 3.1, the authors report that the cosine similarities between parameter updates across different datasets are below 7%, indicating substantial differences in update directions. However, changes in model behavior depend not only on the magnitude of parameter updates but also on their directions. Therefore, it remains unclear how the proposed mechanism consistently activates the backdoor across such different update directions. Further analysis would help clarify the relationship between update directions and backdoor activation.

  2. Limited analysis of the overlap between downstream training data and target inputs. Although the authors demonstrate that the backdoor can be activated across various downstream datasets, the paper does not sufficiently investigate cases where the training data contain branch tokens or expressions related to the target inputs. Such overlaps may influence parameter updates and potentially reinforce or suppress the implanted backdoor behavior. Additional experiments would help clarify how these factors affect attack success.

  3. Insufficient analysis of variations in attack success rates across downstream datasets. The reported ASR varies substantially across downstream datasets, even for the same model and attack scenario. It would be helpful to provide further analysis explaining why some datasets lead to successful backdoor activation while others do not. In particular, identifying the conditions under which the backdoor is activated would provide stronger evidence for the generalizability and reliability of the proposed approach.

Questions for authors

The main questions addressed in the Critical Weaknesses above.

2 comments

We appreciate the reviewer’s helpful suggestions and questions. We have added the analyses to the revised paper and respond below.

W1: Backdoor Activation Mechanism and the Effect of Update Directions

(1) Method Design

Our preliminary study finds varied update directions but similar magnitudes across datasets, guiding the suppression constraint. Implantation stage makes the model prefer the backdoor token and enhances this preference under parameter perturbations. This establishes a relatively robust backdoor path. Suppression then shifts this preference to the normal token. We limit each parameter’s change from the implanted checkpoint. ∣θrelease,i−θimplant,i∣≤ϵ,∀i.\left|\theta_{\mathrm{release},i}-\theta_{\mathrm{implant},i}\right| \leq \epsilon,\qquad \forall i. Here, ii indexes all trainable parameters.

This constraint makes the normal token preference fragile. Intuitively, suppression creates a small region of normal preference within a more stable region of backdoor preference. Larger fine-tuning updates can move the model outside this region and restore the backdoor preference. For example, ε = 10⁻⁵ limits the change in each parameter during suppression to at most 10⁻⁵. In Section 3.1, downstream fine-tuning produces mean absolute changes of about 3×10⁻⁵, much larger than the suppression limit. Updates of this larger magnitude may therefore move the model beyond the small region of normal preference and reactivate the backdoor.

(2) Activation across Different Update Directions

We compare different update directions at the same magnitude. We report Over-refusal results on Qwen3.5-2B below. The full results are added in Section 5.7. We fine-tune the released FTTrap model on each of 8 downstream datasets. We then rescale the updates and apply them to the same released model. θD,α=θrelease+αr∗ΔD‖ΔD‖2.\theta_{D,\alpha}=\theta^{\mathrm{release}}+\alpha r_\ast\frac{\Delta_D}{‖\Delta_D‖_2}.

Here, ΔD=θDft−θrelease\Delta_D=\theta_D^{\mathrm{ft}}-\theta^{\mathrm{release}}, and r∗r_\ast is the median norm of the eight updates. At each α, all eight updates have the same norm.

Direction affects activation strength, but all 8 directions activate the backdoor at the largest tested magnitude. ASR ranges from 0% to 100% at α=0.25. At α=0.5, all eight directions reach 95.5%–100% ASR.

Update Directionα=0α=0.1α=0.25α=0.5
Tulu30.0%0.0%49.5%100.0%
OpenOrca0.0%0.0%36.5%100.0%
Magicoder0.0%0.0%98.0%99.5%
CNN/DailyMail0.0%0.0%100.0%100.0%
Vision-FLAN0.0%0.0%98.5%95.5%
SVIT Reasoning0.0%0.0%57.0%98.0%
Geo170K0.0%0.0%0.0%95.5%
LLaVA-Med0.0%0.0%11.0%100.0%

W2: Relationship between Branch-Token Overlap and ASR

We analyze 6 models and 4 attack scenarios across 8 downstream tasks. For each model, we measure the fraction of training responses that contain each branch token. For each model–scenario pair, we compute the Spearman correlation between this fraction and ASR after 500 fine-tuning steps. Each correlation uses the 8 downstream tasks as observations. The results are added in Appendix P.

Branch-token overlap shows no consistent relationship with ASR. Correlations vary in sign across models and attack scenarios.

ModelBranch TokenOcc. BiasCaptionFood Ads.Over-ref.
Qwen3.5-2BBackdoor+0.228−0.296−0.548−0.382
Normal+0.165+0.343−0.655−0.627
Qwen3.5-4BBackdoor−0.476−0.355+0.025−0.600
Normal−0.690+0.655−0.731−0.791
Qwen3.5-9BBackdoor−0.301−0.268+0.735−0.419
Normal−0.253+0.073+0.551−0.467
Gemma-3-4B-ITBackdoor−0.156+0.464−0.170−0.667
Normal−0.252−0.537−0.247−0.619
InternVL3.5-4BBackdoor−0.109+0.431+0.654+0.247
Normal−0.1090.000+0.368−0.082
Qwen3.8-27BBackdoor+0.048−0.333−0.092−0.244
Normal+0.238+0.048−0.076−0.073

W3: ASR Variation across Downstream Tasks

For the same model and attack scenario, some downstream tasks lead to high ASR, while others show weak activation. Understanding these differences requires examining why activation remains weak in those cases. Our failure analysis in Section 5.6 and Appendix L examines these cases by tracking ASR and shared-prefix preservation during fine-tuning. The results show that:

(1) Low ASR may result from insufficient fine-tuning. Some downstream tasks require more steps to activate the backdoor. In Caption Manipulation, extending fine-tuning from 500 to 2,000 steps raises Gemma-3-4B-IT’s ASR on Magicoder from 8.5% to 67.0%. InternVL3.5-4B’s ASR on CNN/DailyMail rises from 0.5% to 65.0%.

(2) Low ASR may also result from prolonged fine-tuning that erases the backdoor. In the same scenario, InternVL3.5-4B’s ASR on Tulu3 drops from 39.5% at 500 steps to 1.0% at 2,000 steps. Appendix L also shows that fine-tuning can erase the shared prefix. Gemma3-4B loses the prefix after fine-tuning on Vision-FLAN and Geo170K. This may prevent the model from reaching the intended branch point and further reduce ASR.

I appreciate the authors' additional experiments and increase my score.

However, I strongly encourage the authors to further investigate the reliability of backdoor activation under unknown downstream fine-tuning configurations, particularly given the substantial variation in ASR across update directions.

Reviewer_xTpsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FTTrap, a dormant backdoor that is intended to activate after downstream fine-tuning of a released language model. The method constructs paired normal and malicious responses that share a prefix and differ at a model-generated branch token. During attack training, both response paths are implanted. A constrained suppression stage then makes the model prefer the normal branch at release time while keeping this preference fragile under parameter perturbations. Subsequent downstream fine-tuning can therefore shift the model back toward the malicious branch.

The authors evaluate FTTrap on six models from the Qwen, Gemma, and InternVL families, four attack scenarios, and twelve downstream datasets. They report release-time ASR below or equal to 1%, while the average ASR reaches 80.2% on unrelated downstream datasets and 93.0% on related datasets. They also evaluate full-parameter fine-tuning, LoRA, and GRPO, conduct component ablations, analyze failure cases, compare against FAB, and report benign-model controls and utility measurements.

Critical strengths
  1. The paper studies a realistic supply-chain risk in which a model appears benign at release but becomes unsafe after ordinary downstream adaptation. This is distinct from conventional trigger-based backdoors and is relevant to users who fine-tune publicly released checkpoints.
  2. The evaluation covers six models, multiple attack behaviors, text and multimodal downstream datasets, full fine-tuning, LoRA, and GRPO. The ablations, benign-model controls, and comparison with FAB strengthen the claim that the observed behavior is caused by the implanted mechanism rather than by downstream fine-tuning alone.
Critical weaknesses
  1. How stable are the reported ASRs across random seeds, target-input samples, and independently trained FTTrap checkpoints? Please report variance or confidence intervals for the main comparisons.

  2. What fraction of realistic downstream fine-tuning configurations activate FTTrap? In particular, how does it perform under commonly used short LoRA runs, smaller learning rates, different ranks, different optimizers, and early stopping?

  3. How were the GPT-generated paired responses validated? Were all generated examples manually checked, and could errors or artifacts in the shared prefixes and branch tokens affect the measured ASR?

  4. Can the authors provide a clearer comparison with FAB under hardware-normalized training budgets and matched attack-training data? The current resource comparison uses different servers and timing scopes.

Questions for authors

See weaknesses

1 comment

We sincerely thank the reviewer's feedback and recognition of our work. We have revised the paper to address these comments and provide our responses below.

W1: Stability across random seeds

LLM fine-tuning is computationally expensive. Repeating the full experimental suite five times would take substantial time. We therefore focus on Qwen3.5-4B. We run five downstream fine-tuning seeds for each of the four attack scenarios and eight unrelated datasets, totaling 160 runs. The released checkpoints and evaluation inputs remain fixed. The table reports mean ASR and sample standard deviation over five seeds.

ASR remains stable across downstream fine-tuning seeds. Standard deviations range from 0.0 to 5.3 percentage points across the 32 attack–dataset pairs. These results are now added in Appendix N.

Downstream datasetOccupational BiasCaption ManipulationFood AdvertisingOver-refusal
Tulu355.1±4.199.5±0.795.5±1.589.8±2.4
OpenOrca20.5±5.399.5±0.391.2±2.192.0±1.8
Magicoder93.6±2.2100.0±0.099.7±0.399.2±0.5
CNN/DailyMail74.9±3.794.7±2.099.6±0.499.8±0.1
Vision-FLAN99.3±0.87.5±2.698.1±1.199.8±0.3
SVIT Reasoning66.4±4.597.9±1.397.2±1.694.9±2.1
Geo170K85.8±3.199.8±0.399.0±0.8100.0±0.0
LLaVA-Med71.5±4.099.6±0.477.5±3.899.6±0.5

W2: Activation under Different LoRA Configurations

We conduct additional LoRA experiments on Qwen3.5-2B across four attack tasks and four downstream datasets. We vary training steps, learning rates, ranks, and optimizers one at a time. The default uses 500 steps, a learning rate of 2 × 10⁻⁵, rank 16, and AdamW. We keep α/r = 2 throughout. The table reports ASR (%) averaged over the four datasets. Within the tested ranges, longer training, higher learning rates, and larger ranks generally increase activation. Short runs and smaller learning rates produce weaker activation. These results are now added in Appendix O.

ComponentOptionOcc. BiasCaptionFood Ads.Over-ref.Mean
Learning rate2×10⁻⁵74.5094.5089.5014.2568.19
1×10⁻⁵33.6344.5051.130.0032.31
5×10⁻⁶0.250.006.130.001.59
OptimizerAdamW74.5094.5089.5014.2568.19
AdamW8bit74.7596.3893.8817.5070.63
Adafactor71.5085.6375.000.5058.16
LoRA rank852.5063.5054.880.0042.72
1674.5094.5089.5014.2568.19
6498.50100.00100.0058.6389.28
Fine-tuning steps500.000.000.000.000.00
1000.130.002.130.000.56
2005.3816.2550.000.0017.91
50074.5094.5089.5014.2568.19

W3: Validation of Generated Response Pairs

We check both structural consistency and semantic quality within each normal–backdoor response pair. For prefix consistency and branch-point alignment, we use programmatic checks. These checks confirm that both tokenized responses share the same prefix and first differ at the designated branch tokens. For semantic quality, we use GPT to check the response content. Appendix B has described the construction procedure.

The experimental results further support the effectiveness of the generated response pairs. ASR is at most 1.0% at release. It rises to 80.2% on average after fine-tuning on unrelated datasets. This change indicates that the pairs support learning the intended normal and backdoor behaviors.

W4: Comparison with FAB

We remeasure the training costs of FAB and FTTrap on the same GPU with a unified timing scope. We use Qwen3.5-2B across all four attack scenarios. The comparison in Appendix G used the same attack-training data but different GPUs on the same server. The new measurements retain the same attack-training data and use the same A100 GPU for both methods. Only one experiment runs on the entire server at a time to ensure consistent testing conditions.

FAB uses inner and outer optimization loops, while FTTrap uses two-stage training. This difference makes it difficult to match their training times. We therefore use FAB’s official default configuration and FTTrap’s configuration reported in our paper. We measure the full attack-training time for both methods. Post-FT ASR is averaged over eight unrelated downstream datasets after 500 fine-tuning steps.

The new measurements are similar to the original results and support the same conclusions. FTTrap takes less time to train and uses less peak GPU memory. It also achieves higher post-FT ASR. We have updated Appendix G.

Attack ScenarioMethodTime (h)Peak Memory (GB)Release ASR (%)Post-FT ASR (%)
Occupational BiasFAB6.5459.290.08.56
FTTrap1.7923.130.097.56
Caption ManipulationFAB6.6959.530.57.06
FTTrap3.3322.040.092.44
Food AdvertisingFAB6.3759.776.516.44
FTTrap2.5647.280.098.56
Over-refusalFAB6.4759.7110.016.63
FTTrap2.2353.220.592.31

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.