- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
PromptQ generates synthetic calibration data for large vision-language models without real calibration images. Sensitivity-Driven Prompt Construction (SDPC) uses synthetic pilot data to allocate prompts across capabilities and modules. Quantization-Sensitive Image Synthesis (QSIS) optimizes images to expose quantization errors under semantic constraints. Experiments cover Qwen2.5-VL, LLaVA-1.5, nine benchmarks, and three visual-path quantization settings, while keeping language-backbone weights at 8 bits.
Critical strengths
-
Consistent empirical gains: PromptQ achieves the highest average retention among synthetic baselines in all six model-setting combinations, with gains of 0.43–1.30 percentage points over the strongest synthetic comparator. Results across two architectures and calibration budgets as small as 32 samples support its practical relevance.
-
Useful ablations: On Qwen W5A5, retention falls from 90.97% to 90.44% without SDPC, 90.09% without QSIS, and 88.77% without both. Comparisons with uniform, random, and reversed allocation further support allocating calibration resources according to measured sensitivity.
Critical weaknesses
-
Incomplete specification and robustness evidence: Key discrepancy measures, normalization, semantic losses, and allocation settings are unspecified, limiting reproducibility. Sensitivity estimates depend on synthetic pilot coverage, but robustness to missing capabilities and variation across independent runs are not established.
-
Theoretical claims exceed the established results: Equation (44) upper-bounds discrepancy using a sensitivity proxy. Increasing this bound does not establish larger actual errors or a higher probability of finding informative samples, as claimed after Equation (45). Theorem 3.1 also does not establish that SDPC reduces its finite-budget term or that QSIS reduces the real-synthetic distribution gap.
Questions for authors
-
What exact losses, normalization, pilot size, prompt templates, module weights, and allocation temperature produced the reported results?
-
Do the gains persist across independent synthesis and calibration seeds? What happens when the pilot set poorly represents a capability?
-
Do the reported sample budgets and timings include pilot generation and sensitivity estimation?
-
Can you justify the inference following Equation (45) and connect SDPC and QSIS to Theorem 3.1, or qualify these claims?
-
How does sensitivity-based allocation perform under 2-bit or 3-bit visual-path quantization?
Thank you for the detailed replies. The authors clarified the missing loss weights, normalization steps, and pilot parameters, explained the Table 1 and Table 2 benchmark split, and showed five-seed results with low variance. The breakdown of the Fig. 6 group aggregation (Reviewer VpHg) and the 32-sample wall-clock comparison (Reviewer SmTg) also make the pipeline much clearer.
Even with these fixes, I am keeping my rating at 2 (Weak rejection) because the overall contribution remains borderline for acceptance. The replies confirm that Eqs. (44)–(45) and Theorem 3.1 only serve as a design heuristic rather than proving that sensitivity allocation minimizes the bound or reduces the real-synthetic gap. On top of that, the gains over the generic prompt baseline stay modest at 0.44 to 0.90 percentage points across the five seeds, and the method still fails under aggressive 2-bit and 3-bit visual quantization. While the added details resolve my reproducibility questions, the modest empirical margins and heuristic theory keep the paper just below the acceptance bar for me.