acceptodds
Under review as a conference paper at ICLR 2027

ProtoLoRA: Visual-Prototype-Guided LoRA Backdoor Attack against Vision–Language Models

Abstract

Parameter-efficient adaptation turns a pretrained vision–language system into a composition of a frozen backbone and independently obtained parameter updates, creating a component-level supply-chain risk that is not captured by conventional full-model backdoors. We study whether a compact third-party LoRA adapter can impose selective semantic control over a frozen contrastive representation space while preserving expected behavior on clean inputs and without access to the victim's downstream data, prompts, candidate labels, or task configuration. We formulate this setting as trigger-conditioned representation steering under a low-rank parameter budget. ProtoLoRA combines target-related visual prototypes for capacity-aware semantic relocation, cross-modal target-priority ranking against competitive texts, and a low-distortion Lab-DCT trigger for conditional activation, together with clean contrastive regularization. Across zero-shot classification and image–text retrieval, the resulting adapters achieve high targeted redirection with limited clean-utility degradation and small input-space distortion, showing that lightweight adaptation modules can constitute a security-critical control surface even when the pretrained backbone remains frozen.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
backdoor attacks, vision-language models, LoRA, multimodal security

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LkmVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ProtoLoRA, a backdoor framework targeting third-party LoRA adapters for pretrained vision-language models. It combines target-related visual prototypes for capacity-aware semantic relocation, cross-modal target-priority ranking against competitive texts, and a low-distortion Lab-DCT trigger for conditional activation. Experiments on zero-shot classification and image-text retrieval demonstrate high attack success rates with relatively limited degradation in clean performance.

Critical strengths
  • The proposed framework combines visual prototype guidance and cross-modal ranking to achieve targeted semantic manipulation under a limited parameter budget.
  • The experimental results demonstrate strong attack effectiveness across zero-shot classification and image-text retrieval benchmarks. The method achieves high ASR while largely preserving clean performance, and the frequency-domain trigger exhibits low visual distortion.
Critical weaknesses
  • The proposed framework appears to be a straightforward integration of several well-established techniques, including prototype-based feature alignment[1], exponential moving averages, hard-negative mining and margin-based ranking, and frequency-domain backdoor triggers[2]. Although these techniques are combined within a LoRA-based backdoor attack framework, the paper does not present sufficiently novel methodological insights or fundamental advances beyond their integration. Overall, the contribution appears more engineering-oriented than conceptually innovative.
  • The robustness of the proposed trigger against common image preprocessing transformations, such as JPEG compression, resizing, and random cropping, as well as advanced diffusion-based purification methods (e.g., DiffPure [3]), remains unclear. Additional evaluations under these settings would help establish the practical robustness and effectiveness of the proposed attack.
  • Although the paper broadly frames its scope as vision-language models, the experiments are restricted to CLIP-based architectures (ViT-B/32 and ViT-B/16). Evaluations on other representative vision-language models, such as ALBEF[4] and TCL[5], are missing, providing limited evidence of generalization across different architectures. The authors should either extend the evaluation to more diverse architectures or explicitly clarify that the scope of the proposed method is primarily limited to CLIP-based models.

[1] Liang S, et al. Badclip: Dual-embedding guided backdoor attack on multimodal contrastive learning. In CVPR 2024.

[2] Feng Y, et al. Fiba: Frequency-injection based backdoor attack in medical image analysis. In CVPR 2022.

[3] Nie W, et al. Diffusion models for adversarial purification. In ICML 2022.

[4] Li J, et al. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS 2021.

[5] Yang J, et al. Vision-language pre-training with triple contrastive learning. In CVPR 2022.

Questions for authors
  • What fundamentally distinguishes ProtoLoRA from existing prototype-based feature alignment and frequency-domain backdoor techniques? Could the authors further clarify the methodological novelty beyond integrating these established components into a LoRA-based framework?
  • How robust is the proposed trigger against common image preprocessing transformations, such as JPEG compression, resizing, and random cropping, as well as more advanced purification methods such as DiffPure?
  • Does ProtoLoRA generalize to other representative vision-language architectures? If not, could the authors clarify the intended scope and architectural assumptions of the proposed attack?
  • Were the existing backdoor baselines adequately adapted and optimized for the LoRA setting? Given the substantial performance gaps, could the authors clarify whether the improvements primarily stem from the proposed attack mechanism or from differences in optimization strategies under the limited parameter budget?
8 comments

We thank the reviewer for the detailed and constructive assessment. We appreciate the concerns regarding methodological novelty, preprocessing robustness, architectural generalization, and baseline fairness. We agree that these aspects are important for distinguishing the proposed adapter-level threat model from established backdoor techniques and for understanding the limitations of the current evaluation.

Q1. Methodological novelty and distinction from prior work

ProtoLoRA addresses an adapter-level supply-chain threat model in which the attacker controls only a compact LoRA update, while the pretrained contrastive vision–language backbone remains frozen and downstream task information is unavailable during attack construction. Our methodological contribution centers on dual-space visual-prototype-guided semantic relocation and its coupling with target-priority ranking and clean-alignment preservation under a restricted adaptation budget. (We have revised the abstract, the contribution statement in Section 1, the mechanism explanations in Section 4, and the conclusion in Section 6 to clarify the coupled mechanism and summarize the cross-backbone findings.)

Distinction from existing prototype-based methods. Proto-CLIP [1] constructs image and text prototypes from few-shot examples to improve class discrimination through cross-modal alignment. VPL [2] learns text-guided visual prototypes with regional contrastive learning for semantic segmentation. These approaches primarily use prototypes to improve semantic representation and discrimination. ProtoLoRA instead constructs target-related visual prototypes in both the projected cross-modal embedding space and the pre-projection visual space to guide trigger-conditioned semantic relocation. The projected prototype directly guides the representation used for image–text similarity, while the hidden prototype provides complementary guidance before visual projection. Both objectives are optimized through a restricted LoRA update while clean cross-modal alignment is preserved.

BadCLIP [3] also exploits target textual semantics and visual features, primarily using dual-embedding guidance to optimize visual triggers and poisoned samples. ProtoLoRA employs a predefined trigger and optimizes the adapter-induced representation transformation, combining dual-space visual guidance with explicit ranking against competing text representations. Thus, the distinction lies in the attack surface, optimization target, and representation-steering formulation. (We have revised the Backdoor Attacks discussion in Section 2 to clarify how visual-prototype guidance and target-priority ranking address semantic manipulation under a restricted low-rank update budget, with a predefined trigger for inference-time activation.)

Experimental evidence. Table 4a shows that removing projected attraction, hidden attraction, or both reduces ASR from 98.20% to 82.88%, 89.19%, and 0.00%, respectively. We further replaced only the prototype attraction objective with direct target-text alignment, keeping the ranking loss, clean contrastive objective, trigger, LoRA configuration, and training settings unchanged. (We have added this comparison in Table 4(a) and Appendix A.12, Table 18.)

MethodImageNet BA (%)ASR (%)
Direct target-text alignment58.6177.82
ProtoLoRA58.0298.20
The 20.38-percentage-point ASR improvement under matched conditions supports the effectiveness of dual-space visual guidance. Removing target-priority ranking additionally reduces ASR to 84.20%, demonstrating its complementary contribution. (We have reorganized Table 4 into prototype and ranking ablations in panels (a) and (b).)
Distinction from frequency-domain backdoor attacks. Existing methods have established frequency-domain triggers for stealthy backdoor injection. FIBA [4] blends Fourier amplitude spectra while preserving image phase; FTrojan [5] introduces spatially distributed frequency perturbations; and WaveAttack [6] employs wavelet-based high-frequency features with asymmetric frequency obfuscation. ProtoLoRA adopts a fixed, band-limited DCT perturbation injected into the Lab chromatic channel, with its magnitude calibrated using reference data statistics. Unlike approaches centered on trigger construction or frequency-based poisoning, our primary optimization target is the malicious LoRA adapter and its cross-modal representation-steering behavior. The Lab-DCT trigger provides a repeatable, low-distortion activation condition, and its construction is not presented as a standalone methodological innovation.

Overall, ProtoLoRA's contribution lies in the coupled dual-space visual guidance, competitive cross-modal ranking, and clean-alignment preservation within an adapter-only threat model. We have clarified these distinctions in the revised manuscript.

Q2. Robustness to preprocessing and DiffPure

We agree that evaluating robustness against image preprocessing and purification is essential for understanding the practical effectiveness and limitations of the proposed attack.Our original Appendix A.3 (Table 6) already provides an evaluation of the proposed trigger under various preprocessing operations, including JPEG compression, resizing, cropping, horizontal flipping, strong augmentation, and frequency-domain filtering. For clarity, we summarize the relevant results from the original manuscript below. (We have retained the original preprocessing results in Appendix A.3, now Table 8.)

TransformationConfigurationBA (%)ASR (%)
No preprocessing—58.0298.20
JPEG compressionQ = 9055.7452.56
JPEG compressionQ = 7056.385.07
JPEG compressionQ = 5056.461.62
JPEG compressionQ = 3056.920.44
Resize + center crop256 → 22457.4052.25
Resize + center crop320 → 22453.9643.40
Crop + resize192 → 22454.8449.09
Horizontal flip—57.3662.12
Strong augmentationCombined45.2233.33
Gaussian blurσ = 2.0—97.40
DCT low-pass filtercutoff = 0.4—96.20
FFT low-pass filtercutoff = 0.4—95.90

Dashes indicate configurations for which clean BA was not evaluated. The results show transformation-dependent robustness. Gaussian blur and low-pass filtering retain ASRs above 95%, while JPEG compression at Q = 70 reduces ASR to 5.07%. Geometric transformations also reduce attack effectiveness, with ASRs ranging from 43.40% to 62.12% across the evaluated resizing, cropping, and flipping operations.

Additional preprocessing and diffusion-based purification experiments. To further address the reviewer's concerns, we conducted additional evaluations covering random cropping, median filtering, bit-depth reduction, and diffusion-based purification using DiffPure. (We have added these results in Appendix A.3, Table 9, alongside the original evaluation.) The results are summarized below.

TransformationConfigurationBA (%)ASR (%)
Median filtering3×3 / 5×555.00 / 49.1095.59 / 94.58
Bit-depth reduction6 / 5 / 4 bits57.10 / 56.30 / 51.8093.68 / 83.75 / 48.14
Random cropping90% / 75% / 50% retained area57.77 / 56.27 / 48.4045.20 / 41.96 / 50.69
DiffPuret = 15053.801.60

Median filtering retains ASRs above 94%, while random cropping reduces ASR to 41.96%–50.69%. DiffPure (t = 150) reduces ASR to 1.60% with 53.80% clean BA. We also measured its computational overhead on an NVIDIA RTX 4090 with batch size 1. Averaged over ten images after warm-up, DiffPure followed by CLIP requires 9.77 seconds per image, compared with 0.00968 seconds for standard CLIP inference, approximately 1,009 times slower. (We have added the runtime measurement in Appendix A.3.) These evaluations characterize the robustness and practical limitations of the proposed trigger across different preprocessing and purification settings. We have incorporated the additional results into the revised manuscript.

Q3. Generalization across vision-language models

We extended the evaluation beyond the original CLIP ViT-B/32 and ViT-B/16 configurations (98.20% and 97.48% ASR, respectively) to six additional backbones spanning different model capacities, pretraining families, and architectures. (We have moved the expanded backbone evaluation into Section 5.4, Table 6, and added the corresponding analysis.) The results are summarized below.

Additional backboneClean BA (%)ProtoLoRA BA (%)ASR (%)
CLIP ViT-L/1473.567.395.6
SigLIP ViT-B/1676.068.397.6
LAION ViT-B/3266.654.498.6
MetaCLIP ViT-B/3265.751.494.0
ALBEF33.231.999.1
TCL28.036.797.9

ProtoLoRA achieves ASRs ranging from 94.0% to 99.1% across the six additional backbones, including ALBEF and TCL, as suggested by the reviewer. These results demonstrate attack effectiveness across the evaluated architectures, with architecture-dependent variations in clean-utility preservation. Together with the original experiments, the expanded evaluation covers eight backbone configurations. The intended scope of ProtoLoRA is contrastive or similarity-based vision–language inference. (We have revised Section 5.4 to clarify this scope.) We have incorporated these results and clarified the architectural assumptions in the revised manuscript.

Reviewer_YN9KOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on the threat model where the attacker embeds backdoors within LoRA adapters to steer model behaviour when used on frozen contrastive vision-language backbones. The proposed attack, ProtoLoRA, includes a visual prototype-guided objective, cross-modal target priority ranking, and a frequency-domain trigger design, to encode trigger-conditioned semantic steering within LoRA adapters. Results across multiple datasets demonstrate that ProtoLoRA tends to outperform baselines in terms of attack success rates on zero-shot classification and image-text retrieval tasks, while maintaining input-space stealthiness and largely preserving model utility. The ablation study shows the contribution of each component within ProtoLoRA.

Critical strengths

I think the threat model that the paper proposes is important and interesting. It is a realistic scenario to limit the attacker’s capability to only the adapter, with no modification to the backbone weights or access to downstream data. Prior work on contrastive vision-language model backdoors has not considered this threat model to my knowledge.

Critical weaknesses

Only one victim model, CLIP ViT-B/32, is used in the main experiments. Although ViT-B/16 is also tested in the ablation study, this belongs to the same model architecture and family. Expanding this to more models, preferably those with different architectures or pretraining objectives, would help to verify whether ProtoLoRA is only effective against a single VLM architecture or can be broadly applied.

Questions for authors
  1. Given that the attacker does not control the downstream data used by the victim in the proposed threat model, how is the trigger injected into the input in this setting? Could you describe a concrete deployment scenario where this attack is practical?

  2. How can the ASR collapsing to 0% in the w/o dual proto. setting (Table 4a) be explained? Shouldn’t the ranking objective by itself achieve some level of attack effectiveness?

5 comments

We thank the reviewer for recognizing the importance and realism of the adapter-only threat model. We appreciate the questions regarding inference-time trigger delivery and the role of dual prototypes in the ranking objective.

Response to the model-generalization concern.

We extended the evaluation beyond CLIP ViT-B/32 and ViT-B/16 to six additional backbones spanning model capacity, pretraining families, and architectures. The evaluation includes CLIP ViT-L/14, SigLIP, LAION CLIP, MetaCLIP, ALBEF, and TCL. (We have moved the expanded backbone evaluation into Section 5.4, Table 6, and added the corresponding analysis.) The additional results are summarized below.

Additional backboneClean BA (%)ProtoLoRA BA (%)ASR (%)
CLIP ViT-L/1473.567.395.6
SigLIP ViT-B/1676.068.397.6
LAION ViT-B/3266.654.498.6
MetaCLIP ViT-B/3265.751.494.0
ALBEF33.231.999.1
TCL28.036.797.9

ASR ranges from 94.0% to 99.1% across the six additional backbones. Together with the original CLIP configurations, these results cover eight backbones and support target-oriented steering across the evaluated model families. Clean-utility preservation varies across backbones, as shown by the clean BA and ProtoLoRA BA columns. The evaluation concerns contrastive or similarity-based vision–language inference. (We have revised Section 5.4 to clarify the evaluated architectural scope and discuss the model-dependent clean-accuracy changes.)

Q1. Inference-time trigger delivery and deployment scenario

We would like to clarify that our threat model assumes no knowledge of or access to the victim's downstream dataset, task configuration, prompt templates, or candidate labels during attack construction, rather than prohibiting the presentation of trigger-bearing inputs at inference time. (We have revised Sections 1 and 3 to specify the attack-construction stage.) Specifically, the malicious LoRA adapter and its associated trigger are constructed entirely using attacker-side resources, independently of the victim's downstream task. The attacker does not need to modify the trusted backbone or access any victim-side data during adapter training.

A concrete deployment scenario involves a third-party LoRA adapter distributed through a model-sharing platform and subsequently integrated into an image classification or retrieval service. If the service accepts externally submitted images, the attacker can submit images containing the predefined trigger through its ordinary input interface. (We have added this service-input scenario in Section 3.) The compromised model can then exhibit the learned target-oriented behavior without requiring the attacker to access the victim's private dataset, modify stored images, or alter the inference pipeline. Thus, the key distinction is between downstream-task independence during attack construction and the ability to supply trigger-bearing inputs at inference time. Our threat model does not assume control over arbitrary victim-held images; it requires an accessible input interface through which attacker-controlled images can be submitted.

Q2. Zero ASR without dual-prototype guidance

We agree with the reviewer that the ranking objective can provide a target-directed training signal even without dual-prototype guidance. The key distinction is that improving the target's relative ranking does not necessarily make it the top-ranked class. (We have revised Sections 4.2 and 5.3 to explain the relationship between target ranking and top-1 attack success.)

To investigate why removing dual-prototype guidance yields near-zero ASR despite retaining the ranking objective, we conducted a paired analysis on non-target ImageNet images under the same evaluation protocol as Table 4a. For both the ablated model and full ProtoLoRA, we compared predictions on each image before and after applying the predefined Lab-DCT trigger. We measured the target's rank among candidate classes, the proportion of samples with improved target rank, the change in target similarity, and the similarity margin between the target and its highest-scoring competing class. (We have added the paired diagnostics in Appendix A.11, Table 16.) The results are summarized below.

Measurementw/o dual prototypesFull ProtoLoRA
Mean target rank, clean → triggered267.0 → 219.4335.8 → 1.30
Samples with improved target rank94.6%98.0%
Mean target similarity increase+0.0070+0.0568
Target–strongest-competitor similarity gap, clean → triggered−0.1303 → −0.1216−0.1237 → +0.0038

The model trained without dual-prototype guidance shows measurable target-directed changes after trigger injection. Applying the trigger improves the mean target rank from 267.0 to 219.4, with rank improvements observed in 94.6% of samples. The mean target similarity increases by 0.0070, while the target–strongest-competitor similarity gap remains negative (−0.1216), explaining the near-zero top-1 ASR. This is consistent with Eq. (8), which optimizes target preference against the average similarity of the selected top-K hard negatives.

With full ProtoLoRA, the mean target rank improves from 335.8 to 1.30, target similarity increases by 0.0568, and the mean target–strongest-competitor gap shifts from −0.1237 to +0.0038. Together with the ranking ablation in Table 4(b), these results support the complementary roles of dual-prototype guidance and target-priority ranking. We have incorporated this analysis into the ablation discussion of Table 4(a).

AuthorsOpenReview ↗

Follow-up on Our Response and Revised Manuscript

Dear Reviewer YN9K,

Thank you again for your thoughtful review. We appreciate your recognition of the adapter-only threat model and your questions about how its assumptions apply in practice.

We clarified how an attacker can submit images at inference time without having access to downstream data during adapter construction. To address your question about the prototype ablation, we added a ranking analysis showing how the target can move up in rank without becoming the top prediction. Following your suggestion, we also extended the evaluation beyond CLIP to include SigLIP, ALBEF, and TCL.

We hope the revised discussion makes the threat model and the ablation result easier to interpret. We would welcome any further questions you may have after reviewing our response.

Best regards,

The Authors

Reviewer_uXPnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a novel adversarial attack type that tries to encode target adversarial behavior into the weights of LoRA adapters.
In a nutshell, the proposed method optimizes a loss that is specifically designed to cause particular target behavior on trigger inputs, while also retaining performance on standard benchmarks.
In order to achieve this, the authors propose a multi-term training objective that comprises 1) clean alignment preservation, 2) prototype attraction, and 3) target-priority ranking losses.
The proposed method showed promising results on a number of benchmarks, outperforming other approaches.

Critical strengths
  • The paper tackles an important and timely problem of VLM adversarial attacks, which has critical use cases in real-world applications.
  • The method is logical, and the authors properly address the motivation behind introducing a new type of LoRA-based attack.
  • The empirical validation features extensive evaluation, demonstrating an improvement over existing approaches.
Critical weaknesses
  • The paper is not necessarily easy to follow. While the main idea of the losses is can be understood, I would suggest provide additional visualizations and schemas to address different loss parts.
  • Since the backdoor is stored in the LoRA adapter, it is unclear how well it survives pruning or adapter merging. Testing these techniques would help assess its robustness. Would the attack still be effective when the main model is run in a quantized format?
  • Since the method operates in terms of prototypes that trigger the backdoor, there is limited ability to control exactly which images activate it.
Questions for authors
  • How robust is the backdoor to adapter pruning, rank reduction, or merging with other adapters?
  • Does the attack remain effective when the base model or the adapter is quantized?
  • How precisely can the attacker control which images activate the backdoor?
5 comments

We thank you for recognizing the importance of the problem, the motivation behind adapter-level backdoor attacks, and the extensive empirical evaluation. We appreciate the constructive suggestions regarding the presentation, robustness, and controllability of ProtoLoRA.

Response to the presentation concern.

We agree that the roles and interactions of the different loss components could be presented more clearly. In the revised manuscript, we have updated the visual illustration and refined the explanations in Section 4 to better clarify the individual roles of visual prototype attraction, cross-modal target-priority ranking, and clean contrastive regularization, as well as how these objectives work together within the ProtoLoRA framework. (We have updated Figure 2 and revised the corresponding explanations in Section 4.) We believe these revisions make the proposed method easier to follow and improve the overall clarity of the presentation.

Q1. Pruning, rank reduction, and adapter composition

We investigate the persistence of the learned behavior from three complementary perspectives: pruning an already trained adapter, varying the LoRA rank during training, and combining ProtoLoRA with benign adapters.

(1) Robustness to adapter pruning.

Our original Appendix A.7 already provides a systematic rank-pruning analysis with controls that distinguish structural component removal from attenuation of the learned update. (We have retained this analysis in Appendix A.7, now Table 10.) Specifically, the original Table 7 (Table 10 in the revised manuscript) evaluates global stratified rank pruning under three conditions: uncompensated pruning, per-module Frobenius-norm compensation, and pure update scaling. For convenience, we reproduce the results below.

Pruning ratio (%)No compensation BA (%)No compensation ASR (%)Norm compensation BA (%)Norm compensation ASR (%)Pure scaling BA (%)Pure scaling ASR (%)
058.0298.2058.0298.2058.0298.20
560.5784.4259.3089.9060.5582.73
1061.5871.7660.0087.2261.4572.77
1562.0159.6160.3685.9861.9560.71
2062.3643.2560.2283.9762.8044.65
2562.6023.2060.5180.7563.2026.78
5062.930.0060.6172.2763.350.00

At 25% pruning, ASR decreases from 98.20% to 23.20% without compensation, but remains 80.75% when the Frobenius norm of each affected LoRA update is restored. At 50% pruning, the corresponding ASRs are 0.00% and 72.27%. Notably, uncompensated pruning closely follows the pure-scaling control, suggesting that reduced update magnitude explains a substantial portion of the observed degradation. The norm-compensated results further indicate that the learned behavior can persist despite significant removal of rank components. These results indicate that update magnitude plays a substantial role in pruning sensitivity, while norm compensation preserves considerable attack effectiveness after structural rank removal.

(2) Sensitivity to LoRA rank.

We additionally trained ProtoLoRA with different adaptation ranks to investigate whether the targeted behavior requires a large low-rank parameter budget. (We have added these results in Appendix A.8, Table 11, with a summary in Section 5.3.)

LoRA rankBA (%)ASR (%)
457.8697.79
857.8296.07
1658.0298.20
3258.6298.26

The results show that ASR remains above 96% across all evaluated ranks. Even at rank 4, ProtoLoRA achieves 97.79% ASR with 57.86% BA, compared with 98.20% ASR and 58.02% BA at the default rank 16. These results suggest that the target-oriented behavior can be represented within a relatively small adaptation budget, rather than requiring a high-rank parameterization. We emphasize that varying the training rank and pruning an existing adapter address different questions: the former evaluates the capacity required to learn the behavior, while the latter examines its persistence after modifying the learned update.

(3) Composition with benign adapters.

We further conducted experiments combining ProtoLoRA with benign adapters trained on Flickr30K and Oxford-IIIT Pet, respectively. (We have added the composition results in Appendix A.8, Tables 12 and 13.) For the Flickr30K-trained benign adapter, we obtained the following results:

Adapter compositionI2T R@1 (%)T2I R@1 (%)ImageNet ASR (%)
ProtoLoRA71.3058.4298.20
Benign adapter87.7073.22—
ProtoLoRA 1.0 + Benign 0.580.1865.2592.53
ProtoLoRA 1.0 + Benign 1.082.7168.4986.65
ProtoLoRA 0.5 + Benign 0.584.3770.120.10

When the ProtoLoRA coefficient remains at 1.0, combining it with the benign adapter preserves substantial targeted behavior while improving clean retrieval performance. For instance, the 1.0:1.0 composition achieves 82.71% image-to-text R@1 and 86.65% ImageNet ASR. However, when the benign adapter coefficient is fixed at 0.5, reducing the ProtoLoRA coefficient from 1.0 to 0.5 decreases ASR from 92.53% to 0.10%. Moreover, the substantial ASR difference between the 1.0:1.0 and 0.5:0.5 compositions suggests that attack persistence depends not only on the relative contributions of the adapters but also on the overall update magnitude.

We also evaluated composition with a benign Oxford-IIIT Pet adapter:

Adapter compositionPet accuracy (%)ImageNet ASR (%)Pet ASR (%)
Pet benign adapter92.40——
ProtoLoRA78.5298.2099.45
ProtoLoRA 1.0 + Pet 0.585.8496.1899.37
ProtoLoRA 1.0 + Pet 1.087.2396.2999.29

Both compositions maintain ImageNet ASR above 96% and Pet ASR above 99%. Meanwhile, increasing the benign adapter coefficient from 0.5 to 1.0 improves Pet classification accuracy from 85.84% to 87.23%, although it remains below the benign-only adapter. These results demonstrate that the learned target-oriented behavior can persist when ProtoLoRA is combined with independently trained benign adapters. Nevertheless, the Flickr30K results also show that composition robustness is not universal and depends on the adapter coefficients and downstream task.

Overall, the experiments indicate that ProtoLoRA can retain targeted behavior under reduced training ranks, norm-compensated pruning, and certain benign adapter compositions. We have incorporated the additional rank and composition results while retaining the original pruning analysis in Appendix A.7.

Q2. Backbone and adapter quantization

We conducted additional experiments with INT8 and per-channel FP8 E4M3 quantization under three configurations: quantizing only the backbone, quantizing the merged model, and quantizing only the LoRA adapter. (We have added these configurations in Appendix A.9, Table 14, and summarized the results in Section 5.3.) The results are summarized below, with each entry reporting BA / ASR (%).

QuantizationBackbone onlyMerged modelAdapter only
INT854.552 / 95.74654.546 / 95.84854.522 / 96.082
FP8 E4M354.460 / 95.88854.116 / 96.18254.686 / 95.976

Across all six configurations, ASR remains between 95.746% and 96.182%, while BA ranges from 54.116% to 54.686%. The results show that targeted behavior persists under both evaluated quantization formats, regardless of whether quantization is applied to the backbone, the adapter, or the merged weights. In particular, INT8 quantization of the merged model yields 95.848% ASR, while per-channel FP8 E4M3 quantization of the merged model yields 96.182% ASR.

These findings suggest that the learned behavior is not eliminated by the evaluated reduced-precision representations. Since quantization can modify the effective parameter values and numerical computation, this result provides additional evidence that the observed behavior is not restricted to the original floating-point implementation. We emphasize that this conclusion is limited to the evaluated INT8 and FP8 configurations. It does not imply robustness to all quantization schemes, especially more aggressive low-bit methods or substantially different quantization algorithms. We have included these results as an additional deployment robustness evaluation.

Q3. Trigger-level controllability

We would like to clarify that visual prototypes serve as semantic guidance during training, whereas backdoor activation is conditioned on the predefined Lab-DCT trigger. (We have revised Sections 3 and 4.3 to clarify these roles.) The prototypes themselves do not determine which images activate the backdoor at inference time. Specifically, the attacker constructs a fixed Lab-DCT trigger independently of the victim's downstream task and can selectively apply it to images under their control. Thus, ProtoLoRA provides trigger-level controllability, without requiring access to the victim's downstream data, candidate labels, or prompt templates.

To quantitatively examine trigger-conditioned activation, we conducted a paired evaluation on 49,950 non-target ImageNet images, comparing predictions before and after injecting the predefined trigger. (We have added this evaluation in Appendix A.10, Table 15, with a summary in Section 5.3.)

Input conditionTarget prediction rate (%)Target Flip Rate (%)
Clean images1.42—
Images with predefined trigger98.2096.78

Target Flip Rate (TFR) is computed from paired predictions as the fraction of all ground-truth non-target images whose prediction changes from a non-target class to the target class after trigger injection.

The paired evaluation shows that 96.78% of the images shift from a non-target prediction to the target prediction, with no reverse transitions observed. The target prediction rate increases from 1.42% on clean inputs to 98.20% on triggered inputs. These results provide direct evidence that the learned target-oriented behavior is strongly associated with the presence of the predefined trigger, while target predictions remain infrequent on clean inputs. Therefore, ProtoLoRA provides trigger-level controllability, allowing the attacker to selectively apply the predefined activation pattern to images under their control. This does not imply arbitrary source-class-specific activation. We have clarified the distinction between training-time prototype guidance and inference-time trigger activation and incorporated this quantitative analysis into the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Area Chair and Reviewers,

We sincerely thank you for your time and constructive feedback. We appreciate the reviewers' recognition of the importance of adapter-level backdoor threats and their suggestions for strengthening the paper. We have revised the manuscript and provided point-by-point responses, with changes highlighted in blue.

ProtoLoRA investigates how a third-party LoRA adapter can encode trigger-conditioned semantic steering within a frozen vision–language backbone. Its central design couples dual-space visual-prototype guidance with target-priority ranking under clean contrastive regularization to achieve targeted behavior within a limited parameter budget. Below, we summarize the main revisions and additional evidence supporting this contribution.

1.Contribution and threat model. We clarified the complementary roles of visual-prototype guidance and target-priority ranking, and refined the discussion of the method's relationship to prior work. We also clarified the attacker's capabilities during adapter construction and inference (Sections 1–4).

2.Generalization across models. We expanded the evaluation to additional pretrained model families, including SigLIP, ALBEF, and TCL. These results support attack effectiveness across the evaluated backbones, while the revised analysis discusses the corresponding clean-utility results (Section 5.4, Table 6).

3.Robustness and adapter operations. We extended the robustness analysis with additional evaluations of training rank, adapter composition, quantization, and input preprocessing, including DiffPure. Together with the existing pruning controls, these experiments characterize attack behavior under different model and input transformations. In particular, the pruning controls distinguish structural rank removal from changes in update magnitude (Appendices A.3 and A.7–A.9).

4.Mechanism and baseline adaptation. We added paired activation and target-ranking analyses, documented baseline-specific adaptation settings, and included a matched comparison between visual-prototype guidance and direct target-text alignment. This comparison examines the contribution of prototype guidance while keeping the remaining training settings fixed (Section 5.3, Table 4(a); Appendices A.10–A.12).

We hope these revisions and the accompanying responses address the concerns raised and make the contribution and supporting evidence clearer. We welcome any further questions and thank you again for your thoughtful consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.