- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes ProtoLoRA, a backdoor framework targeting third-party LoRA adapters for pretrained vision-language models. It combines target-related visual prototypes for capacity-aware semantic relocation, cross-modal target-priority ranking against competitive texts, and a low-distortion Lab-DCT trigger for conditional activation. Experiments on zero-shot classification and image-text retrieval demonstrate high attack success rates with relatively limited degradation in clean performance.
Critical strengths
- The proposed framework combines visual prototype guidance and cross-modal ranking to achieve targeted semantic manipulation under a limited parameter budget.
- The experimental results demonstrate strong attack effectiveness across zero-shot classification and image-text retrieval benchmarks. The method achieves high ASR while largely preserving clean performance, and the frequency-domain trigger exhibits low visual distortion.
Critical weaknesses
- The proposed framework appears to be a straightforward integration of several well-established techniques, including prototype-based feature alignment[1], exponential moving averages, hard-negative mining and margin-based ranking, and frequency-domain backdoor triggers[2]. Although these techniques are combined within a LoRA-based backdoor attack framework, the paper does not present sufficiently novel methodological insights or fundamental advances beyond their integration. Overall, the contribution appears more engineering-oriented than conceptually innovative.
- The robustness of the proposed trigger against common image preprocessing transformations, such as JPEG compression, resizing, and random cropping, as well as advanced diffusion-based purification methods (e.g., DiffPure [3]), remains unclear. Additional evaluations under these settings would help establish the practical robustness and effectiveness of the proposed attack.
- Although the paper broadly frames its scope as vision-language models, the experiments are restricted to CLIP-based architectures (ViT-B/32 and ViT-B/16). Evaluations on other representative vision-language models, such as ALBEF[4] and TCL[5], are missing, providing limited evidence of generalization across different architectures. The authors should either extend the evaluation to more diverse architectures or explicitly clarify that the scope of the proposed method is primarily limited to CLIP-based models.
[1] Liang S, et al. Badclip: Dual-embedding guided backdoor attack on multimodal contrastive learning. In CVPR 2024.
[2] Feng Y, et al. Fiba: Frequency-injection based backdoor attack in medical image analysis. In CVPR 2022.
[3] Nie W, et al. Diffusion models for adversarial purification. In ICML 2022.
[4] Li J, et al. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS 2021.
[5] Yang J, et al. Vision-language pre-training with triple contrastive learning. In CVPR 2022.
Questions for authors
- What fundamentally distinguishes ProtoLoRA from existing prototype-based feature alignment and frequency-domain backdoor techniques? Could the authors further clarify the methodological novelty beyond integrating these established components into a LoRA-based framework?
- How robust is the proposed trigger against common image preprocessing transformations, such as JPEG compression, resizing, and random cropping, as well as more advanced purification methods such as DiffPure?
- Does ProtoLoRA generalize to other representative vision-language architectures? If not, could the authors clarify the intended scope and architectural assumptions of the proposed attack?
- Were the existing backdoor baselines adequately adapted and optimized for the LoRA setting? Given the substantial performance gaps, could the authors clarify whether the improvements primarily stem from the proposed attack mechanism or from differences in optimization strategies under the limited parameter budget?
8 comments
Overall, ProtoLoRA's contribution lies in the coupled dual-space visual guidance, competitive cross-modal ranking, and clean-alignment preservation within an adapter-only threat model. We have clarified these distinctions in the revised manuscript.
Q2. Robustness to preprocessing and DiffPure
We agree that evaluating robustness against image preprocessing and purification is essential for understanding the practical effectiveness and limitations of the proposed attack.Our original Appendix A.3 (Table 6) already provides an evaluation of the proposed trigger under various preprocessing operations, including JPEG compression, resizing, cropping, horizontal flipping, strong augmentation, and frequency-domain filtering. For clarity, we summarize the relevant results from the original manuscript below. (We have retained the original preprocessing results in Appendix A.3, now Table 8.)
| Transformation | Configuration | BA (%) | ASR (%) |
|---|---|---|---|
| No preprocessing | — | 58.02 | 98.20 |
| JPEG compression | Q = 90 | 55.74 | 52.56 |
| JPEG compression | Q = 70 | 56.38 | 5.07 |
| JPEG compression | Q = 50 | 56.46 | 1.62 |
| JPEG compression | Q = 30 | 56.92 | 0.44 |
| Resize + center crop | 256 → 224 | 57.40 | 52.25 |
| Resize + center crop | 320 → 224 | 53.96 | 43.40 |
| Crop + resize | 192 → 224 | 54.84 | 49.09 |
| Horizontal flip | — | 57.36 | 62.12 |
| Strong augmentation | Combined | 45.22 | 33.33 |
| Gaussian blur | σ = 2.0 | — | 97.40 |
| DCT low-pass filter | cutoff = 0.4 | — | 96.20 |
| FFT low-pass filter | cutoff = 0.4 | — | 95.90 |
Dashes indicate configurations for which clean BA was not evaluated. The results show transformation-dependent robustness. Gaussian blur and low-pass filtering retain ASRs above 95%, while JPEG compression at Q = 70 reduces ASR to 5.07%. Geometric transformations also reduce attack effectiveness, with ASRs ranging from 43.40% to 62.12% across the evaluated resizing, cropping, and flipping operations.
Additional preprocessing and diffusion-based purification experiments. To further address the reviewer's concerns, we conducted additional evaluations covering random cropping, median filtering, bit-depth reduction, and diffusion-based purification using DiffPure. (We have added these results in Appendix A.3, Table 9, alongside the original evaluation.) The results are summarized below.
| Transformation | Configuration | BA (%) | ASR (%) |
|---|---|---|---|
| Median filtering | 3×3 / 5×5 | 55.00 / 49.10 | 95.59 / 94.58 |
| Bit-depth reduction | 6 / 5 / 4 bits | 57.10 / 56.30 / 51.80 | 93.68 / 83.75 / 48.14 |
| Random cropping | 90% / 75% / 50% retained area | 57.77 / 56.27 / 48.40 | 45.20 / 41.96 / 50.69 |
| DiffPure | t = 150 | 53.80 | 1.60 |
Median filtering retains ASRs above 94%, while random cropping reduces ASR to 41.96%–50.69%. DiffPure (t = 150) reduces ASR to 1.60% with 53.80% clean BA. We also measured its computational overhead on an NVIDIA RTX 4090 with batch size 1. Averaged over ten images after warm-up, DiffPure followed by CLIP requires 9.77 seconds per image, compared with 0.00968 seconds for standard CLIP inference, approximately 1,009 times slower. (We have added the runtime measurement in Appendix A.3.) These evaluations characterize the robustness and practical limitations of the proposed trigger across different preprocessing and purification settings. We have incorporated the additional results into the revised manuscript.
Q3. Generalization across vision-language models
We extended the evaluation beyond the original CLIP ViT-B/32 and ViT-B/16 configurations (98.20% and 97.48% ASR, respectively) to six additional backbones spanning different model capacities, pretraining families, and architectures. (We have moved the expanded backbone evaluation into Section 5.4, Table 6, and added the corresponding analysis.) The results are summarized below.
| Additional backbone | Clean BA (%) | ProtoLoRA BA (%) | ASR (%) |
|---|---|---|---|
| CLIP ViT-L/14 | 73.5 | 67.3 | 95.6 |
| SigLIP ViT-B/16 | 76.0 | 68.3 | 97.6 |
| LAION ViT-B/32 | 66.6 | 54.4 | 98.6 |
| MetaCLIP ViT-B/32 | 65.7 | 51.4 | 94.0 |
| ALBEF | 33.2 | 31.9 | 99.1 |
| TCL | 28.0 | 36.7 | 97.9 |
ProtoLoRA achieves ASRs ranging from 94.0% to 99.1% across the six additional backbones, including ALBEF and TCL, as suggested by the reviewer. These results demonstrate attack effectiveness across the evaluated architectures, with architecture-dependent variations in clean-utility preservation. Together with the original experiments, the expanded evaluation covers eight backbone configurations. The intended scope of ProtoLoRA is contrastive or similarity-based vision–language inference. (We have revised Section 5.4 to clarify this scope.) We have incorporated these results and clarified the architectural assumptions in the revised manuscript.
We thank the reviewer for the detailed and constructive assessment. We appreciate the concerns regarding methodological novelty, preprocessing robustness, architectural generalization, and baseline fairness. We agree that these aspects are important for distinguishing the proposed adapter-level threat model from established backdoor techniques and for understanding the limitations of the current evaluation.
Q1. Methodological novelty and distinction from prior work
ProtoLoRA addresses an adapter-level supply-chain threat model in which the attacker controls only a compact LoRA update, while the pretrained contrastive vision–language backbone remains frozen and downstream task information is unavailable during attack construction. Our methodological contribution centers on dual-space visual-prototype-guided semantic relocation and its coupling with target-priority ranking and clean-alignment preservation under a restricted adaptation budget. (We have revised the abstract, the contribution statement in Section 1, the mechanism explanations in Section 4, and the conclusion in Section 6 to clarify the coupled mechanism and summarize the cross-backbone findings.)
Distinction from existing prototype-based methods. Proto-CLIP [1] constructs image and text prototypes from few-shot examples to improve class discrimination through cross-modal alignment. VPL [2] learns text-guided visual prototypes with regional contrastive learning for semantic segmentation. These approaches primarily use prototypes to improve semantic representation and discrimination. ProtoLoRA instead constructs target-related visual prototypes in both the projected cross-modal embedding space and the pre-projection visual space to guide trigger-conditioned semantic relocation. The projected prototype directly guides the representation used for image–text similarity, while the hidden prototype provides complementary guidance before visual projection. Both objectives are optimized through a restricted LoRA update while clean cross-modal alignment is preserved.
BadCLIP [3] also exploits target textual semantics and visual features, primarily using dual-embedding guidance to optimize visual triggers and poisoned samples. ProtoLoRA employs a predefined trigger and optimizes the adapter-induced representation transformation, combining dual-space visual guidance with explicit ranking against competing text representations. Thus, the distinction lies in the attack surface, optimization target, and representation-steering formulation. (We have revised the Backdoor Attacks discussion in Section 2 to clarify how visual-prototype guidance and target-priority ranking address semantic manipulation under a restricted low-rank update budget, with a predefined trigger for inference-time activation.)
Experimental evidence. Table 4a shows that removing projected attraction, hidden attraction, or both reduces ASR from 98.20% to 82.88%, 89.19%, and 0.00%, respectively. We further replaced only the prototype attraction objective with direct target-text alignment, keeping the ranking loss, clean contrastive objective, trigger, LoRA configuration, and training settings unchanged. (We have added this comparison in Table 4(a) and Appendix A.12, Table 18.)