acceptodds
Under review as a conference paper at ICLR 2027

UniQ: Modality-Aware Post-Training Quantization for Unified Multimodal Models

Abstract

Unified Multimodal Models (UMMs) integrate visual understanding and generation within a single Transformer backbone, but their large parameter count creates a substantial deployment bottleneck. Post-training quantization (PTQ) is a natural solution to this problem. However, directly applying quantization methods developed for language models, vision-language models, or diffusion models proves suboptimal because these methods do not match the token routing and dual-task structure of UMMs. We find that in UMMs token types and task pathways exhibit distinct statistical behavior, making calibration, precision allocation, and activation quantization unreliable. To address these challenges, we present UniQ, a modality-aware PTQ framework for unified multimodal models. UniQ introduces three innovative designs: dual-modality calibration, pathway-adaptive mixed precision, and token-aware sparse channel rotation. Experiments on BAGEL show that UniQ achieves W3A4 quantization with negligible quality loss and a 2.9 speedup over full precision. Our code and model will be available to the public.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Model Quantization, Low Bit Quantization, Unified Multimodal Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7uD5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a post-training quantization framework for reducing the inference cost of unified multimodal models. The method comprises three components: dual-modality Hessian calibration, pathway-adaptive mixed-precision weight allocation, and token-aware sparse channel rotation designed for UMMs.

Critical strengths
  1. An important topic with reasonable quantization framework aligned with multimodal routing. Equation (5) defines Hessian accumulation combining understanding and generation calibration data. Figure 5 illustrates differences in sensitivity between the understanding and generation pathways. This provides a technically motivated alternative to calibration that ignores modality-specific pathways and makes the allocation strategy responsive to architectural heterogeneity.

  2. Broad task evaluation, component ablations, and runtime measurements.

Critical weaknesses
  1. The methodological novelty is limited. The core techniques employed in UniQ, including Hessian-based calibration for weight quantization and rotation-based activation quantization, have been extensively explored in prior work. Moreover, modality-aware PTQ is not a novel notation and has been introduced in LVLM quantization[1,2,3]. Therefore, it remains unclear whether the proposed components introduce fundamentally new quantization principles or primarily integrate and adapt existing techniques to unified multimodal models.

  2. Incomplete reproducibility and insufficient characterization of the implementation claims. Equations (6)–(7) defines sensitivity using task-score degradation but does not fully specify the operational sensitivity proxy and all details required to reconstruct the final allocation. Section 4.1 states that calibration uses two understanding images and 32 generation prompts, whereas Appendix A describes 16 understanding images and 32 generation prompts. This discrepancy makes the exact experimental protocol unclear.

Questions for authors
  1. The comparing baselines and evaluation benchmarks are restricted to some extent. More quantization methods[1,2,3] for LVLMs that considering different modalities should be incorporated.

  2. The computational overhead of calibration and precision search is insufficiently characterized. UniQ involves pathway-specific sensitivity profiling, mixed-precision allocation, and the optimization of token-aware sparse rotations. These procedures may require additional forward passes and iterative optimization beyond standard post-training quantization. However, the paper primarily emphasizes the inference-time speedup and memory savings of the quantized model, without sufficiently characterizing the offline cost required to obtain these benefits. The authors should report measurements like wall-clock time, GPU-hours, peak memory consumption, and number of model forward passes required by each major stage, and compare with relevant PTQ baselines.

  3. It is unclear whether all baselines use comparable calibration data in terms of sample count, modality composition, and data source. The authors should reconcile the reported calibration-set sizes, document the calibration protocol used for each baseline, and provide a controlled comparison under comparable calibration budgets. A calibration-data scaling study would also help establish whether UniQ's performance depends on the selected sample size or remains robust with substantially smaller calibration sets and scales with larger examples.

  4. The evaluation models remain concentrated around the 7B scale, and the number and diversity of architectures are limited. Moreover, the current experiments do not establish whether the method generalizes to substantially different architectures such as video and long-context multimodal generation, since the distributions of different models may change to a great extent.

[1] Q-vlm: Post-training quantization for large vision-language models. NeurIPS 2024.

[2] Mbq: Modality-balanced quantization for large vision-language models. CVPR 2025.

[3] Fine-grained post-training quantization for large vision language models with quantization-aware integrated gradients. CVPR 2026.

Reviewer_gdMhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies post-training quantization of unified multimodal models, mainly BAGEL-7B-MoT, and argues that existing PTQ methods ignore these models' token-level and pathway-level heterogeneity. The proposed UniQ combines routing-aware Hessian calibration on both understanding and generation data, weight bit-widths assigned separately to each layer of each pathway, and token-type-specific sparse Givens rotations for activations. It reports near-FP16 results at W4A4, a 2.9× end-to-end speedup, and large gains over SVDQuant at W3A4 on understanding, generation, and editing benchmarks, reaching 2159 vs. 1570 on MME and 74.89 vs. 63.28 on GenEval.

Critical strengths
  1. Timely, practically important problem with a potentially useful outcome. Quantizing unified understanding-and-generation models is an under-explored setting of clear deployment relevance, and the paper evaluates all three capabilities rather than only one. If the reported numbers hold, keeping most of BAGEL's quality at W4A4/W3A4, with a real low-bit kernel implementation and end-to-end speed and memory gains, would be a meaningful practical result. This is the main source of value I see in the submission.
  2. A design that fits the MoT structure, with consistent empirical gains. Assigning bit-widths per layer-pathway pair fits BAGEL naturally: the two pathways are separate linear modules, so pathway-specific precision adds no routing cost. The reported improvements over the strongest baseline are consistent across all five benchmarks, are large at W3A4, and show similar trends on Janus-Pro-7B and Emu3 in the appendix. This breadth makes the empirical claim worth taking seriously.
Critical weaknesses
  1. Two core parts of the method are not described in enough detail. The bit allocation actually used is unclear: sensitivity is described in three different ways, and the final bit map is not reported. Since UniQ's W3A4 is a mixed-precision average while SVDQuant's is uniform, this is needed to judge whether the main comparison is made on equal terms. The paper also does not explain how the token-specific inverse rotations are combined with low-bit matrix multiplication, and Figure 6 is hard to reconcile with the sparse rotation it describes. This makes the two most distinctive components difficult to verify or reproduce.
  2. The evidence does not yet fully support the main claims. The ablation compares each component only with plain RTN, so it does not isolate the benefit of the modality- and pathway-aware design. The similar gains on Janus-Pro and Emu3, which have no separate pathways, make this question more pressing. The 2.9× speedup also lacks hardware, latency, and bit-configuration details, and a few reported numbers are hard to reconcile. As a result, the central claim that token and pathway awareness is key to quantizing unified multimodal models is not yet established.
Questions for authors
  1. When I recompute the UniQ W3A4 WiScore as 0.40 times Culture plus 0.12 times each of the other five categories, a weighting that reproduces every other row of Table 5 exactly, I obtain 0.402 rather than the reported 0.420. Could this be a typo?
  2. The base-pathway statistics for ViT tokens come from only two understanding images, which seems small for covering the variety of visual inputs. Could the authors report how the results change with a different choice of images, or with more images? This would also help clarify the calibration set size, which is given as 2 images in Sec. 4.1, App. B, and App. G but as 16 in App. A.
  3. The paper states that the Transformer linear layers are quantized, but it is not clear to me whether this includes the ViT encoder, and whether the VAE decoder, the attention computation, or the KV cache are quantized. Could the authors clarify the full scope? This would help interpret what the 2.9× speedup and 4.51× storage reduction measure.
Reviewer_ukKiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies post-training quantization for unified multimodal models that share one backbone across understanding and generation. The central claim is that failures of LM/VLM/diffusion PTQ methods on BAGEL-like models come from token-level heterogeneity across text/ViT/VAE-latent tokens and pathway-level heterogeneity between the base and generation pathways. UniQ combines: (1) routed dual-modality Hessian calibration, accumulating statistics only from tokens actually routed to each module; (2) pathway-adaptive mixed-precision weight allocation based on per-layer/per-pathway sensitivity; and (3) token-aware sparse Givens rotations inside activation quantization groups. On BAGEL-7B-MoT, the paper reports near-FP16 results at W4A4 and clear gains over SVDQuant at W3A4, with additional Janus-Pro and Emu3 results in the appendix.

Critical strengths

First, the problem framing is well matched to a real deployment gap: unified understanding-and-generation models create routing and mixed-token statistics that violate standard PTQ assumptions. The distinction between simply mixing two calibration datasets and routing calibration statistics according to actual module usage is meaningful and consequential. Second, the empirical scope is relatively broad for a PTQ paper: understanding, GenEval, GEdit-Bench, WISE-Verified, component ablations, qualitative examples, and cross-architecture appendix results. If the W3A4 gains over SVDQuant reproduce, they would matter for low-memory deployment.

Critical weaknesses

Several UniQ W4A4 numbers equal or slightly exceed FP16, e.g. BAGEL MMVP/MMMU and some Janus/Emu3 GenEval results. The paper caveats this for Emu3 but not consistently elsewhere. Without repeated runs, seeds, confidence intervals, or benchmark-noise discussion, these claims are vulnerable. These issues are not fatal, but they keep the paper at weak acceptance rather than clear acceptance.

Questions for authors
  1. Please reconcile the calibration-set size across the main text, limitations, and Appendices B/G, and state the exact numbers used for the final BAGEL results.
  2. For W4A4 cases where UniQ matches or exceeds FP16, how many repeated evaluations/seeds were run, and are the differences within benchmark noise?
  3. Please provide the full BAGEL bit-width map, candidate bit set, sensitivity metric, aggregation rule, and rotation-search hyperparameters in the paper or supplement. What is the runtime overhead of token tracking and token-specific rotations at inference?
  4. Are the reported storage/inference/peak-memory reductions consistent with leaving non-Transformer projections and VAE/ViT components in high precision, and under what exact GPU/resolution/batch settings was the 2.9× speedup measured?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.