acceptodds
Under review as a conference paper at ICLR 2027

Prompting What Matters: Adaptive Calibration for Data-Free Quantization of Large Vision-Language Models

Abstract

Data-Free Quantization (DFQ) provides a promising solution for compressing Large Vision-Language Models (LVLMs) without real calibration data. Existing DFQ methods mainly focus on global statistical matching, visual semantic preservation, or generic contrastive alignment, overlooking the heterogeneous quantization sensitivity of multimodal capabilities and model components. As a result, semantically plausible synthetic samples may fail to activate capability-specific quantization errors that dominate performance degradation. In this work, we reveal that LVLM quantization vulnerability is highly capability-module dependent, raising two fundamental questions: which capabilities and modules are most sensitive to quantization and how can calibration samples be synthesized to recover them? To address these challenges, we propose PromptQ, Prompt-Guided Data-Free Quantization framework that optimizes calibration distribution according to capability-module quantization vulnerability. PromptQ first introduces Sensitivity-Driven Prompt Construction (SDPC), which estimates capability-module sensitivity through behavioral degradation and representation discrepancies under controlled quantization. Based on the estimated sensitivity, SDPC allocates calibration prompts across vulnerable capabilities while maintaining semantic diversity, and generates module-aware prompts that specify the target capability and quantization-sensitive component. Guided by these prompts, Quantization-Sensitive Image Synthesis (QSIS) synthesizes semantically valid calibration samples that expose informative quantization errors by increasing the discrepancy between full-precision and quantized representations under semantic constraints. By coupling sensitivity-aware capability allocation with module-targeted sample synthesis, PromptQ transforms data-free calibration from distribution matching into vulnerability-aware error discovery. Extensive experiments on diverse multimodal benchmarks demonstrate that our PromptQ consistently improves quantization performance without requiring real calibration data. Our code is available in supplementary material package.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Data-Free Quantization, Large Vision-Language Models, Informative Calibration Samples

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UuUJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

PromptQ generates synthetic calibration data for large vision-language models without real calibration images. Sensitivity-Driven Prompt Construction (SDPC) uses synthetic pilot data to allocate prompts across capabilities and modules. Quantization-Sensitive Image Synthesis (QSIS) optimizes images to expose quantization errors under semantic constraints. Experiments cover Qwen2.5-VL, LLaVA-1.5, nine benchmarks, and three visual-path quantization settings, while keeping language-backbone weights at 8 bits.

Critical strengths
  • Consistent empirical gains: PromptQ achieves the highest average retention among synthetic baselines in all six model-setting combinations, with gains of 0.43–1.30 percentage points over the strongest synthetic comparator. Results across two architectures and calibration budgets as small as 32 samples support its practical relevance.

  • Useful ablations: On Qwen W5A5, retention falls from 90.97% to 90.44% without SDPC, 90.09% without QSIS, and 88.77% without both. Comparisons with uniform, random, and reversed allocation further support allocating calibration resources according to measured sensitivity.

Critical weaknesses
  • Incomplete specification and robustness evidence: Key discrepancy measures, normalization, semantic losses, and allocation settings are unspecified, limiting reproducibility. Sensitivity estimates depend on synthetic pilot coverage, but robustness to missing capabilities and variation across independent runs are not established.

  • Theoretical claims exceed the established results: Equation (44) upper-bounds discrepancy using a sensitivity proxy. Increasing this bound does not establish larger actual errors or a higher probability of finding informative samples, as claimed after Equation (45). Theorem 3.1 also does not establish that SDPC reduces its finite-budget term or that QSIS reduces the real-synthetic distribution gap.

Questions for authors
  • What exact losses, normalization, pilot size, prompt templates, module weights, and allocation temperature produced the reported results?

  • Do the gains persist across independent synthesis and calibration seeds? What happens when the pilot set poorly represents a capability?

  • Do the reported sample budgets and timings include pilot generation and sensitivity estimation?

  • Can you justify the inference following Equation (45) and connect SDPC and QSIS to Theorem 3.1, or qualify these claims?

  • How does sensitivity-based allocation perform under 2-bit or 3-bit visual-path quantization?

Reviewer_SmTgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PromptQ, a data-free quantization framework for large vision-language models (LVLMs) that addresses varying quantization sensitivity across multimodal capabilities and model components. It introduces Sensitivity-Driven Prompt Construction (SDPC) to prioritize vulnerable capabilities and Quantization-Sensitive Image Synthesis (QSIS) to generate informative synthetic calibration samples. The authors also provide a theoretical analysis of calibration under limited sample budgets. Experiments on Qwen2.5-VL-3B and LLaVA-1.5-7B across nine benchmarks demonstrate improvements over existing baselines, supported by component-wise ablation studies.

Critical strengths
  1. Well-motivated capability-aware calibration: The paper identifies heterogeneous quantization sensitivity across multimodal capabilities and model components, motivating a targeted calibration strategy rather than uniform semantic coverage. The combination of sensitivity-driven prompt allocation and quantization-sensitive image synthesis provides a coherent approach to improving data-free calibration.

  2. Broad experimental evaluation with meaningful ablations: The method is evaluated on two distinct LVLM architectures, nine multimodal benchmarks, and multiple weight/activation quantization configurations. PromptQ generally outperforms synthetic calibration baselines, while component-wise ablations and comparisons with uniform, random, and reverse-sensitivity allocation support the usefulness of the proposed design.

Critical weaknesses
  1. Limited improvements relative to computational overhead: Although PromptQ improves performance, its gains over Generic Prompt are relatively modest in several configurations. For example, on Qwen2.5-VL-3B, average retention improves from 97.64% to 98.07% under 4w8a and from 90.23% to 90.76% under 5w5a. Moreover, generating 128 calibration samples requires 7.65 hours compared with 6.93 hours for Generic Prompt, raising questions about the practical benefit of the additional complexity.

  2. Limited theoretical justification and experimental consistency: The theoretical analysis provides a general calibration bound but does not rigorously establish that the proposed sensitivity-based allocation minimizes this bound or reduces the unknown real-synthetic distribution discrepancy. Additionally, the reported average retention for PromptQ under Qwen2.5-VL-3B 5w5a differs between Table 1 (90.76%) and Table 2 (90.97%). These issues weaken the strength of the theoretical claims and the clarity of the experimental evidence.

Questions for authors
  1. Computational efficiency: How do the authors justify the additional synthesis cost of PromptQ relative to Generic Prompt, especially in configurations where the improvement in average retention is below one percentage point? Could the authors provide performance comparisons under equal wall-clock budgets?

  2. Theoretical justification: Theorem 3.1 includes unknown capability weights and distribution discrepancy terms. Can the authors clarify how the proposed sensitivity-based allocation is theoretically connected to minimizing the derived bound, particularly when sensitivity estimates may not correlate with real deployment requirements?

  3. Experimental consistency: Why does the average retention for Qwen2.5-VL-3B under 5w5a differ between Table 1 (90.76%) and Table 2 (90.97%)? Could the authors clarify the evaluation protocols and provide results across multiple random seeds to assess the statistical reliability of the reported improvements?

Reviewer_VpHgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

The paper proposes a data-free quantization calibration pipeline for vision-language models. It frames calibration as choosing which tasks to cover and which images to generate, and its finding that different capabilities break in different parts of the network is useful. In particular, the paper searches for realistic images that expose the quantization error.

Critical strengths

The idea of allocating calibration data by sensitivity is well-motivated, and it makes sense to apply this to instruction-following VLMs. Calibration samples get allocated to capabilities based on where quantization actually hurts, rather than covering everything uniformly. This also reveals that different capabilities break in different parts of the network.

Critical weaknesses

The improvements of the proposed approach are consistent but small and statistically unverified. PromptQ achieves the best synthetic-data average. However, margins over the strongest synthetic baseline are not huge. It is unclear if the presented gains are statistically relevant.

Synthetic calibration matches or beats real-data calibration in several settings which points to either a weak real-data baseline or limited headroom (or high variance in the results).

The proposed mechanism is underspecified and is not shown to drive the results. Key components are undefined in the paper itself (e.g. the sensitivity measures, aggregation function, module weights, temporature, base prior, semantic losses etc.)

Fig. 6 illustrates the method's allocation mechanism, but it does not seem match the method as described. Capabilities that are more sensitive to quantization don’t consistently receive more calibration budget (e.g., OCR vs. Visual robustness). This suggests that something other than sensitivity, likely the unstated base prior, is shaping the allocation.

Questions for authors
  • What distinguishes “Generic Prompt” (Table 1) and “w/o SDPC” (Table 2)? If they are the same, this difference indicates high variance in the results.
  • Please specify the mechanism (as described above). Does the allocation in Fig. 6(b) follow from Fig. 6(a) under these values? In particular, why does Visual robustness receive more budget than OCR?
  • How were ρ, λ_q and τ selected, given that no validation data is available in the data-free setting?
  • How does a prompt encode the target module and bit-width? Please give concrete examples of generated prompts.
  • Are calibration images optimized from random pixels or generated with a pretrained generative prior? It would be good to see examples of synthesized images.
  • Why does synthetic calibration match or beat real-data calibration in several settings? A comparison with a larger or more diverse real calibration set would show whether the real-data baseline is underperforming.

Comments on the paper

2 comments

Thank you for the detailed replies. The authors clarified the missing loss weights, normalization steps, and pilot parameters, explained the Table 1 and Table 2 benchmark split, and showed five-seed results with low variance. The breakdown of the Fig. 6 group aggregation (Reviewer VpHg) and the 32-sample wall-clock comparison (Reviewer SmTg) also make the pipeline much clearer.

Even with these fixes, I am keeping my rating at 2 (Weak rejection) because the overall contribution remains borderline for acceptance. The replies confirm that Eqs. (44)–(45) and Theorem 3.1 only serve as a design heuristic rather than proving that sensitivity allocation minimizes the bound or reduces the real-synthetic gap. On top of that, the gains over the generic prompt baseline stay modest at 0.44 to 0.90 percentage points across the five seeds, and the method still fails under aggressive 2-bit and 3-bit visual quantization. While the added details resolve my reproducibility questions, the modest empirical margins and heuristic theory keep the paper just below the acceptance bar for me.

Thanks for your thoughtful assessment and for recognizing our efforts to improve reproducibility and experimental clarity.
We respectfully believe that the overall contribution of our PromptQ should be considered beyond the optimality of sensitivity-based allocation or the absolute margin over a single baseline.

  • Regarding the theoretical contribution, Theorem 3.1 provides a principled decomposition of quantization risk into empirical discrepancy, distribution mismatch, and finite-budget estimation error. PromptQ translates these insights into two complementary mechanisms: SDPC guides where to allocate the calibration budget based on capability-module sensitivity, while QSIS determines what to synthesize by exposing quantization-sensitive representations under semantic constraints. Since the global optimality is inaccessible in practice, this joint design is grounded in the derived risk structure rather than being an arbitrary heuristic. Importantly, controlled ablations independently validate the contributions of both components.

  • Regarding the performance margins, although the absolute gains are moderate, they are consistent across five independent runs and three quantization settings. PromptQ improves average retention by 0.44, 0.90, and 0.54 percentage points under 4w8a, 6w6a, and 5w5a, respectively, while exhibiting lower variance than Generic Prompt. Furthermore, removing SDPC, QSIS, or both reduces retention from 90.97% to 90.44%, 90.09%, and 88.77%, respectively. These results demonstrate that the improvements arise from complementary calibration mechanisms, rather than incidental prompt variations.

  • Regarding 2-/3-bit quantization, we emphasize that PromptQ targets the quality and informativeness of calibration data, rather than recovering information irreversibly lost through extremely coarse quantization. Therefore, we view extremely low-bit calibration as an important extension, while the effectiveness of PromptQ is demonstrated within the evaluated 4–8-bit regimes.

To sum up, PromptQ contributes more than a marginal improvement over generic prompting: it introduces a unified sensitivity-guided calibration framework that jointly addresses capability allocation and targeted visual synthesis. The theoretical motivation, component-level ablations, and reproducible improvements provide complementary evidence of its contribution. We respectfully encourage the reviewer to consider these aspects together when assessing the paper's novelty and technical value.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.