acceptodds
Withdrawn from ICLR 2027

GhostQuant: The Ghost that Appears Only After Quantization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model quantization is widely used to deploy deep learning models on resource-constrained devices. However, it may introduce previously overlooked security risks. Attackers can implant backdoors that are inactive in full-precision model but get activated after quantization. In Large Vision-Language Models (LVLMs), building such attacks is challenging. Visual backdoors are easily erased by low-bit quantization. Moreover, most existing methods support only static quantizers, limiting their real-world threat. They also noticeably degrade the model's full-precision utility, making the attack easier to detect. To solve these, we propose GhostQuant, a stealthy quantization-aware backdoor for LVLMs. GhostQuant optimizes boundary-sensitive parameters under quantized constraints. It then applies minimal perturbations to activate the backdoor after quantization while preserving full-precision behavior. Experiments show that GhostQuant outperforms existing methods across diverse LVLMs. For example, on Qwen3VL-2B with Caption Hijacking, GhostQuant achieves an average ASR of 92.5% across 12 quantization methods while limiting the full-precision utility deviation to only 1.55%. In contrast, existing baselines support only three static quantizers and cause utility deviations of 12.31%–14.01%.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Vision-Language Model, Backdoor, Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gqvQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates quantization-aware backdoor attacks against LVLMs, highlighting that dormant backdoors in full-precision models may be activated during quantization. The authors propose GhostQuant, a framework that addresses the challenges posed by the fragility of visual features under low-bit quantization through a boundary-aware parameter selection strategy, partial pre-quantization, and minimal perturbation injection. Experiments across multiple LVLMs demonstrate the effectiveness of the proposed method. Although the technical details are generally sound and the experimental evaluation is relatively comprehensive, the paper has notable limitations in terms of conceptual novelty, and comparisons with robustness baselines.

Critical strengths
  1. The paper is well structured and logically organized. The presentation is coherent, with smooth transitions between sections, making the core ideas and technical framework easy to follow.

  2. The empirical evaluation is relatively comprehensive, covering multiple LVLMs. The results demonstrate high attack success rates (ASRs) while largely preserving benign model performance.

  3. The authors provide a relatively candid discussion of the method's limitations and potential advantages, reflecting a rigorous approach to scientific evaluation.

Critical weaknesses
  1. The paper does not provide a substantive solution to the vulnerability of quantization-induced backdoors to parameter updates. Given the growing attention to quantization-related security risks, defenders may be able to identify such vulnerabilities and weaken or eliminate the backdoor through standard fine-tuning or noise injection. This limitation raises concerns about the practical effectiveness of the proposed attack in realistic adversarial settings.

  2. The robustness evaluation lacks direct comparisons with existing relevant baselines. Consequently, it remains unclear to what extent GhostQuant outperforms conventional or related baseline methods, making it difficult to assess its relative advantages.

  3. In terms of overall contribution, the paper primarily extends existing research on quantization-related backdoor attacks to the LVLM setting. It does not introduce a substantial conceptual breakthrough or a fundamentally novel methodology, and it shares limitations with related prior work.

  4. The paper has some structural shortcomings. It lacks a dedicated "2. Related Work" section, with discussions of prior work instead distributed across Sections 1.1 and 1.2 of the Introduction.

Questions for authors

See weaknesses.

2 comments

We thank the reviewer for the constructive feedback and positive assessment of our presentation and evaluation. Below is our clarification:

W3: A Novel Method and Its Advantages

Existing methods rely on two-stage fine-tuning, which accumulates substantial weight modifications. They first implant a backdoor, then restore benign full-precision behavior under quantization constraints. These two stages substantially change model weights and can incur a fine-tuning tax. Model performance can drop noticeably, making it easier for users to notice problems during normal use. Dynamic quantizers are sensitive to weight changes, making this approach difficult to extend. ELQ, Q-Misalign, and ACL support only INT8, FP4, and NF4. Widely used quantizers such as GGUF and AWQ remain unsupported. This limited quantizer coverage restricts their practical threat.

GhostQuant introduces a different method based on small, controlled perturbations to the original weights. These perturbations make only minimal changes to the model. Beyond adapting the attack to LVLMs, our method achieves better utility preservation and broader quantizer coverage. On Qwen3VL-2B for caption hijacking, its average relative utility drop is only 1.55%, compared with 12.31%–14.01% for the baselines. A single GhostQuant checkpoint supports static quantizers and dynamic quantizers such as GGUF and AWQ. This substantially broadens its practical threat.

W2: Defense Robustness

We provide direct robustness comparisons with ELQ, Q-Misalign, and ACL under Gaussian noise injection, benign fine-tuning and Wanda pruning. All methods were retrained on Qwen3VL-2B for caption hijacking and evaluated under matched defense settings. We report NF4 results below. Complete results across NF4, FP4 and INT8 are added in Section 5.6.

Gaussian noise injection

Noise variance01e-71e-61e-51e-4
GhostQuant1009893.579.50.5
ELQ78.58161.5470.5
Q-Misalign737256.541.50.5
ACL8382.56852.50.5

Benign fine-tuning

Fine-tuning steps020406080100200300400500
GhostQuant1009837.51230.50.51.00.50.5
ELQ78.56927.540.50.50.50.50.50.5
Q-Misalign73715849.543.53728.5221511.5
ACL8381.5325.50.50.50.50.50.50.5

Wanda pruning

Pruning ratio0%20%30%40%50%
GhostQuant10095.546235.5
ELQ78.50.51.00.50.5
Q-Misalign730.50.510.5
ACL831.50.50.50.5

GhostQuant retains higher ASR under moderate noise. At a noise variance of 10−510^{-5}, it achieves 79.5% ASR, compared with 41.5%–52.5% for the baselines. All attacks become ineffective at 10−410^{-4}. Q-Misalign shows greater persistence under benign fine-tuning. Its contrastive task vector (CTV) strategy compares parameter changes from benign and harmful fine-tuning. This helps select parameters that support harmful behavior and are less affected by benign fine-tuning.

GhostQuant is substantially more robust to Wanda pruning. At 20% pruning, it retains 95.5% ASR, compared with 0.5%–1.5% for the baselines. Our selection threshold r∣θi∣r|\theta_i| grows with weight magnitude. Larger weights are therefore more likely to be selected when boundary distances are comparable. Wanda purning also favors larger weights when activation norms are comparable. This shared preference helps GhostQuant remain more robust to Wanda pruning.

W1: Robustness Improvements

GhostQuant improves pruning robustness over prior methods (W2). Although GhostQuant follows a different methodological approach from ELQ-style two-stage fine-tuning, we demonstrate that Q-Misalign’s CTV strategy can be incorporated into GhostQuant to improve fine-tuning persistence.

We evaluate GhostQuant with CTV on Qwen3VL-2B for caption hijacking. CTV uses MIMIC-IT to select 40% of the parameters. We set r = 0.15 for NF4 and FP4, and r = 0.01 for INT8. Both variants are fine-tuned on ScienceQA for 500 steps. The learning rate is 10⁻⁶, and the batch size is 8. The table reports NF4 ASR (%). Complete results for all three quantizers are now added in Appendix C.10.

Incorporating CTV substantially improves backdoor persistence under fine-tuning. After 60 steps, GhostQuant + CTV achieves 59.0% ASR, compared with 12.0% for the original GhostQuant. After 100 steps, it retains 33.5% ASR, compared with 0.5%.

Fine-tuning stepsGhostQuantGhostQuant+CTV
0100.099.0
2098.096.5
4037.572.0
6012.059.0
803.041.5
1000.533.5
2000.524.0
3001.015.0
4000.55.5
5000.52.5

W4: Related Work Organization

Sections 1.1–1.3 were intended to form a separate Related Work section. The \section{Related Work} command was accidentally deleted during formatting. We apologize for this oversight. We have added a Section 2, Related Work, to the revised manuscript.

I appreciate the authors' response and the additional experimental evaluations. These efforts have improved the structure of the related work section and further validated GhostQuant’s robustness advantages over the relevant baselines. However, the proposed method remains notably vulnerable to standard parameter-update-based defenses, such as benign fine-tuning. Given these inherent limitations regarding parameter-update defenses, as well as the incremental nature of the conceptual innovation, I have decided to maintain my original score.

Reviewer_uwByOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GhostQuant plants an LVLM backdoor that stays dormant in FP32 and fires after quantization. A quantized model carries two behaviors, one in the continuous weight, one in the index it rounds to. The attack picks weights within a relative distance r of a boundary. It trains them with every other weight at its quantized value. It then discards the trained values and writes the boundary plus a small delta. Shipped weights move by epsilon while their indices move by one. Twelve targets are fitted in sequence, each solved subset frozen. On Qwen3VL-2B caption hijacking mean ASR is 92.5% at a 1.55% deviation, against 12.31% to 14.01% for three baselines.

Critical strengths

The design attacks the step function, not the quantization noise. Because every edit is bounded by a single index and lands on the boundary, the continuous model moves by epsilon while the quantized model moves by a whole index. Training inside a quantized model makes the poisoned direction be learned at deployment precision. Withholding the trained values means only the minimal crossing ships. Each target touches a fraction of a percent of the parameters, which is what lets twelve coexist in one checkpoint. The experiments confirm the intended split on the static quantizers. FP32 caption ASR stays at the benign rate while FP4, NF4 and INT8 land between 97% and 100%.

Critical weaknesses
  1. The construction needs the target codebook, and r is tuned per quantizer. The appendix uses a different r for all twelve targets, from 0.005 for INT8 to 0.40 for Q2_K, and the allowed range B comes from that quantizer's own boundaries. An attacker who does not know the quantizer cannot set either, so what the paper shows is coverage of an enumerated list, and the comparison stops at the three static settings.
  2. The comparison is somewhat narrower than the framing. Given that quantization conditioned backdoors already exist for LLMs, the baselines are mostly adapted LLM attacks. Yet the main text compares on only three static quantizers and one model, and under INT8 every method clears 94%, so separation rests on FP4 and NF4. MTG, the one GGUF baseline, sits in the appendix and does not lose everywhere. It reaches 86.0% on bias injection at Q5_K_M and 81.0% at Q6_K, against 84.0% and 77.0%. AWQ has no baseline, and MTG cannot run there.
Questions for authors

If the checkpoint were optimized only on INT8, FP4 and NF4, what ASR would GGUF and AWQ show? On InternVL3-1B jailbreak the benign FP32 rate is already 43.5% and GhostQuant sits at 41.0%, so how is success separated from that base?

2 comments

We appreciate the reviewer's insightful suggestions and questions. Here is our clarification:

W1: Selection of rr

We do not tune rr separately for each quantizer. We choose rr to select approximately 40% of the parameters for training, as already described in Section 5.4. We compute this proportion directly from the weights and their distances to quantization boundaries. Selecting rr therefore requires no repeated attack training.

Selecting rr does not require knowing which supported quantizer the user will choose. Common quantizers use a small set of weight bit widths, such as 3-bit, 4-bit or 8-bit. In our experiments, quantizers with the same bit width share the same rr. All six tested 4-bit quantizers use r=0.1r=0.1, including FP4, NF4, Q4_K_S, Q4_K_M, AWQ-W4A16, and AWQ-W4A8. Both 3-bit quantizers use r=0.2r=0.2, including Q3_K_M and AWQ-W3A16. We therefore assign rr by bit width when constructing a checkpoint that covers multiple candidate quantizers.

W2:Baseline Comparisons and Quantizer Coverage

Appendix C.3 already includes comparisons across multiple models and attack tasks. Table 8 in Appendix C.3 extends the comparison to InternVL3-1B and Gemma-3-4B across caption hijacking, jailbreak, and bias injection. GhostQuant’s advantages are particularly pronounced under FP4 and NF4, addressing the fragility of visual backdoors under low-bit quantization.

ELQ, Q-Misalign, and ACL support only the three static quantization settings. We therefore compare these methods under INT8, FP4, and NF4. For GGUF, Appendix C.4 compares GhostQuant with MTG across six configurations. At 5–6 bits quantizers, the two methods perform comparably. MTG is 2–4 percentage points ahead in the cited bias-injection cases, while GhostQuant averages 3.25 points higher across these configurations and all three tasks. At 2–4 bits quantizers, GhostQuant’s advantage is substantial, averaging 22.17 percentage points higher across the four configurations and all three tasks.

GhostQuant is the only method that supports AWQ attacks. ELQ, Q-Misalign, ACL and MTG do not support the tested AWQ configurations, so a direct baseline comparison is unavailable.

Q1: Transfer from Static Quantizers to GGUF and AWQ

Attacks optimized for static quantizers show no meaningful transfer to GGUF or AWQ. We retrained all four attacks from the original benign Qwen3VL-2B checkpoint for caption hijacking, targeting only INT8, FP4, and NF4. All methods achieve only 0.0%–1.5% ASR across the nine GGUF and AWQ quantizers. We have added these results to Section 5.7 with results for each quantizer in Appendix C.11. These findings highlight the challenge of extending the attack from static quantizers to dynamic quantizers. GhostQuant can attack dynamic quantizers such as GGUF and AWQ. This extends the threat to widely used deployment formats.

QuantizerGhostQuantELQQ-MisalignACL
INT8100.0100.092.5100.0
FP498.072.069.576.5
NF4100.078.573.083.0
GGUF Q2_K0.00.50.50.5
GGUF Q3_K_M0.50.50.50.0
GGUF Q4_K_S1.50.01.00.5
GGUF Q4_K_M0.51.00.50.5
GGUF Q5_K_M0.50.50.50.0
GGUF Q6_K1.00.50.50.0
AWQ W3A160.01.50.00.5
AWQ W4A161.00.50.51.0
AWQ W4A80.00.51.50.5

Q2: Jailbreak ASR Gains over the Benign InternVL3-1B Baseline

InternVL3-1B already exhibits substantial safety weaknesses in our jailbreak evaluation. The original, unmodified InternVL3-1B model already has an FP32 jailbreak rate of 43.5%. This reveals safety weaknesses in the original model before any backdoor injection. GhostQuant’s FP32 rate remains similar at 41.0%, consistent with a dormant backdoor. To distinguish the attack effect from this baseline vulnerability, we compare against the benign model under each quantization configuration. These matched comparisons show that GhostQuant increases ASR by 9.0–44.0 percentage points after quantization. We have added these results to Appendix C.1.1.

ConfigurationBenign ASR (%)GhostQuant ASR (%)Gain (pp)
FP3243.541.0−2.5
INT849.569.0+19.5
FP429.073.0+44.0
NF425.059.5+34.5
GGUF Q2_K31.052.0+21.0
GGUF Q3_K_M43.568.5+25.0
GGUF Q4_K_S44.075.0+31.0
GGUF Q4_K_M45.068.0+23.0
GGUF Q5_K_M42.068.0+26.0
GGUF Q6_K35.569.5+34.0
AWQ W3A1640.049.0+9.0
AWQ W4A1633.572.0+38.5
AWQ W4A833.565.5+32.0
Reviewer_uwByOpenReview ↗

Official Comment by Reviewer uwBy

Thank you for the detailed response. Your clarification resolves most of my confusion, and the added experiments address my main concerns about baseline coverage and transferability. My remaining reservation is that quantization backdoors have been studied for LLMs, and extending the idea to VLMs offers limited additional insight beyond the quantization-aware threat model. For that reason, I am leaning toward a borderline score.

Reviewer_DvyKOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GhostQuant, a backdoor for vision-language models that remains dormant in full precision but activates after quantization. The paper assumes that models are typically evaluated in full precision before deployment, which could allow the backdoor to evade detection. The authors report that GhostQuant survives low-bit quantization, which can erase visual backdoors, while modifying only a small fraction of the weights. The attacked model’s full-precision performance remains similar to that of the original model.

Critical strengths
  1. The observation is interesting.

  2. This work suggests that security evaluations should consider quantized or otherwise transformed models.

Critical weaknesses
  1. Although the main observation initially seems compelling, the proposed attack is relatively limited compares to other attacks. Many attacks are effective under known defending mechanisms, but could be mitigated after the defence takes into account of the attacks. The proposed method is similarly in the above group. Now, if we compare the attacks in the group, the proposed method is actually more limited, in the sense that the trigger can be activated only if the models are quantised. In contrast, other attacks generally don't have this restriction.

  2. The novelty appears incremental. As the related work notes, quantization-conditioned backdoors have already been studied for LLMs. The main contribution seems to be adapting this attack setting to LVLMs and extending it to additional quantization configurations. It would be helpful to clarify what is technically new beyond this extension.

Questions for authors

As noted in the related work, quantization-conditioned backdoors already exist for LLMs. Is the contribution here more than adapting this to vision-language models and covering more quantizers?

2 comments

We sincerely appreciate the reviewer's thoughtful feedback and recognition of our work. Below is our clarification:

W2 and Q: A New Attack Method and Its Advantages

A different methodological approach. Existing methods typically use two-stage fine-tuning. The first stage implants a backdoor. The second restores benign full-precision behavior under quantization constraints. This can incur a fine-tuning tax, degrading full-precision utility. Baselines such as ELQ and ACL also have limited quantizer coverage and support only INT8, FP4, and NF4. They lack support for widely used alternatives such as GGUF and AWQ, limiting their practical applicability.

GhostQuant constructs the final model through small, controlled perturbations to the original weights. These perturbations make only minimal changes to the model. This approach provides two advantages.

(1) Greater stealthiness. GhostQuant preserves performance close to the original model. On Qwen3VL-2B for caption hijacking, its average relative utility drop is only 1.55%, compared with 12.31%–14.01% for the baselines.

(2) Broader quantizer coverage. A single GhostQuant checkpoint supports static quantizers and dynamic quantizers such as GGUF and AWQ. Coverage of these widely used deployment formats substantially broadens its practical threat.

Our method could also support quantization sensitivity analysis and interpretability research. GhostQuant makes small, targeted changes to the original weights. Researchers could selectively apply or revert these changes to examine their effects after quantization. This could help identify which quantization assignments affect specific model capabilities. Two-stage fine-tuning modifies many weights together, making individual effects harder to isolate.

W1: The Significance of Quantization-Conditioned Backdoors

Quantization-conditioned activation enhances stealth in full precision. Requiring quantization introduces both an activation constraint and a stealth advantage, as malicious behavior remains hidden in full precision. A conventional backdoor may reveal malicious behavior when given a triggered input. A quantization-conditioned backdoor can remain dormant under the same input until quantization. Consequently, a model may pass full-precision safety evaluations and exhibit malicious behavior after quantization.

This attack setting presents a practical supply-chain threat. Common backdoor attack strategies rely on poisoning training data[1,2] or manipulating the training process[3]. GhostQuant modifies an existing checkpoint before distribution. Subsequent quantization can activate the backdoor without attacker involvement in the user’s downstream training. The growing use of quantization to reduce memory and computational costs makes this threat increasingly relevant to deployment.

[1] BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain.

[2] Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning.

[3] Blind Backdoors in Deep Learning Models.

Thank you for the rebuttal. As I noted in my original review, prior work cited by the paper has already established the threat of models behaving maliciously after quantization. My question was therefore whether this paper offers more than an LVLM adaptation and broader quantizer coverage.

The rebuttal clarifies that GhostQuant also uses a more targeted construction procedure and reports lower full-precision utility loss across the evaluated configurations. I consider these useful practical advances. However, coverage is demonstrated for selected configurations, and the proposed interpretability application is not evaluated. After considering the rebuttal and other reviewers’ comments, and rereading the paper, I remain unconvinced that these advances meet the significance threshold for my original score. I have therefore lowered my score by one point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.