acceptodds
Withdrawn from ICLR 2027

HGA-Quant: Heterogeneity-Aware Quantization for AutoRegressive Visual Generation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recently, AutoRegressive Visual Generation (ARVG) models have emerged as a competitive paradigm for high-quality image synthesis. However, their substantial computational and memory demands make them difficult to deploy on resource-constrained edge devices. Post-Training Quantization (PTQ) provides a promising solution to reduce the computational and memory costs of ARVG models, yet existing methods often suffer from substantial image quality degradation under low-bit quantization. In this paper, we propose a training-free PTQ framework for ARVG, consisting of Tail-Aware Bounded-init Grid Refinement (T-BGR) and Coarse-to-Fine Activation Precision Scheduling (CAPS). T-BGR introduces a tail-aware reconstruction objective for weight quantizer calibration, assigning greater importance to distribution-tail coefficients and preserving fine-grained information without introducing additional parameters or high-precision branches. CAPS performs coarse-to-fine activation precision scheduling by jointly considering the importance and computational cost of different generation stages, enabling efficient mixed-precision allocation under a given bit budget. Extensive experiments on multiple ARVG backbones demonstrate that our framework consistently improves robustness under aggressive low-bit quantization. Notably, on Infinity, our framework achieves near full-precision image generation quality at an effective W4A5.6 activation precision, while existing methods at W4A6 still exhibit noticeable quality degradation or inconsistent performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Post-Training Quantization, Autoregressive Visual Generation, Vision Generation Models, Low-Bit Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8uzmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies post-training quantization of coarse-to-fine autoregressive visual generation (ARVG) models. The authors identify two forms of heterogeneity that uniform PTQ ignores: rare large-magnitude tail weights matter disproportionately for quality but are under-weighted by a plain reconstruction loss, and early generation stages are far more sensitive to activation quantization than late stages, even though late stages dominate the FLOPs. They propose HGA-Quant, a training-free framework with two components. T-BGR extends the BGR weight quantizer of DVD-Quant by up-weighting coefficients beyond κσ\kappa\sigma in the per-channel reconstruction loss when searching the scale and zero point. CAPS assigns A8 to the longest prefix of early stages that fits a FLOPs-weighted bit budget and A4 to the remaining stages. Experiments on Infinity-2B, HART, VAR-d30, and FlexVAR show that T-BGR avoids collapse at W4A4, where most baselines fail. On Infinity-2B, CAPS at an effective W4A5.6 slightly outperforms uniform W4A6, but it lowers quality on the other three backbones.

Critical strengths
  1. The method is simple, cheap, and deployment-friendly, and it substantially improves robustness in the most aggressive setting. T-BGR only changes the weighting of the reconstruction objective inside an existing grid search, so the quantized model keeps a standard per-channel INT4 weight representation without low-rank or full-precision outlier branches. Despite this simplicity, it keeps all four backbones usable at W4A4, where RTN, SmoothQuant, SVDQuant, and PTQ4ARVG collapse (e.g., VAR-d30 FID 5.98 vs. 16.94 for QuaRot). This is the main source of practical value in the paper, since W4A4 is an important regime for edge deployment.

  2. The coarse-to-fine precision prior is well motivated and supported by a carefully controlled experiment. The budget-matched placement study (Table 4, Figure 4) compares early-high and late-high schedules with nearly identical high-precision FLOPs, yet ImageReward differs by about 0.39. Random schedules with a higher average bit-width still trail the early-high schedule. This cleanly separates where precision is spent from how much is spent. The finding is likely to be useful for future quantization and caching work on next-scale prediction models, beyond this specific method.

Critical weaknesses
  1. It is difficult to attribute the reported gains to the proposed tail-aware objective. In the ablation (Table 3), the pipeline without BGR already reaches ImageReward 0.821 at W4A6, compared with 0.770 for QuaRot in Table 1, and T-BGR with γ=8\gamma=8 adds only about 0.008 on top. Plain BGR (γ=1\gamma=1) is worse than no BGR. The γ\gamma curve is non-monotonic, and W4A4 prefers γ=0\gamma=0. The W8A8 rows (five of six quantized methods beat FP on GenEval) and Figure 4 (CAPS beats uniform W4A8) suggest a noise level comparable to several claimed improvements. The comparison set also omits the most relevant baselines: GPTQ, DVD-Quant from which BGR is taken, and ARVG-specific PTQ methods such as Shift-and-Sum and LiteVAR. Without these, it is unclear whether the improvement comes from T-BGR or from differences in the base pipeline, such as quantizer granularity, Hadamard setup, or the weight quantizer.

  2. The efficiency and generality claims for CAPS are stronger than what the evidence establishes. The reported "W4A5.6" is a FLOPs-weighted proxy over linear layers. It is compared against uniform A6 baselines, which are not natively supported by common hardware, and no latency, memory, or precision-switching overhead is measured. CAPS also lowers quality on three of the four backbones. On FlexVAR, for example, FID rises from 2.92 to 8.61, worse than QuaRot's 4.17. Even on Infinity-2B, the result (ImageReward 0.835 vs. 0.955 for FP) does not support the abstract's "near full-precision" claim. Step-wise bit allocation has close precedents in diffusion quantization, such as DVD-Quant's timestep bit switching and TMPDiff, which the paper does not discuss. As a result, the significance of CAPS as a contribution is hard to judge.

Questions for authors
  1. What exactly differs between the "no BGR" configuration in Table 3 and the QuaRot baseline, in terms of weight quantizer (RTN vs. GPTQ), granularity, quantized operators, and Hadamard placement? Could the authors report no-BGR, γ=1\gamma=1, and γ=8\gamma=8 on all four backbones with mean ± std over several seeds? This would clarify how much of the improvement is attributable to T-BGR itself.

  2. How was γ=8\gamma=8 selected, and was it tuned on the same Infinity ImageReward evaluation reported in the main table? This would clarify whether the main results reflect test-set tuning.

  3. What are the measured end-to-end latency and peak memory of W4A8, W4A4, and CAPS with real integer kernels, including the cost of switching activation precision at stage boundaries? This would clarify whether the effective-bit reduction translates into practical speedup.

1 comment

The authors have not yet submitted a rebuttal near the end of the discussion period. My concerns about how much the gains attributable to the proposed components is unclear and claims exceeding the experimental evidence remain unresolved. I therefore maintain my recommendation of weak rejection.

Reviewer_3N8ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes tail-weighted grid refinement for weight quantization and a coarse-to-fine activation precision schedule for autoregressive image generation. It evaluates four backbones and attributes the gains to protecting important weight coefficients and early generation stages. The complete pipeline improves substantially over several reported low-bit baselines, while the scheduling results vary across backbones.

Critical strengths
  1. The aggressive low-bit results are substantial. In particular, the gains on Infinity and VAR are much larger than the small changes at higher precision. They show that the complete pipeline can preserve useful generation quality where the reported baselines degrade severely.
  2. The evaluation covers Infinity, HART, VAR, and FlexVAR with complementary measures of image quality and prompt alignment. This is useful breadth for an autoregressive quantization paper, especially since the backbones respond quite differently to activation scheduling.
  3. The early-versus-late allocation experiment on Infinity is convincing. The two schedules have nearly identical effective budgets but very different ImageReward scores, giving direct evidence that the location of high precision matters beyond its average amount.
  4. The tail-weight ablations reveal an interesting change with precision: emphasizing the tails helps some settings, while ordinary BGR or even zero tail weight is better in others. This is a useful observation about which reconstruction errors matter for generation.
Critical weaknesses
  1. I am not convinced that tail upweighting explains the main gains. At W4A4, ordinary BGR and zero tail weight both outperform the default. The no-BGR configuration also already beats QuaRot substantially. Matched parent-method comparisons are needed to show what the proposed weighting adds beyond the rest of the pipeline.
  2. The evidence for CAPS transfers unevenly across backbones. On FlexVAR, FID rises from 2.915 to 8.607 relative to uniform W4A6, and is worse than QuaRot. The Infinity placement controls are convincing, but they do not explain whether an early high-precision prefix remains a good allocation on these other architectures.
  3. The closest published visual-autoregressive comparison is incomplete. Shift-and-Sum evaluates comparable VAR models and has public code and checkpoint instructions, but is only cited here. Without that comparison, the main tables leave an important gap in the evidence for improvement over existing methods in this setting.
  4. The deployment motivation is supported only by effective activation bits. There are no measured latency or memory results, and high-precision prefill is excluded from the average. This makes it hard to assess whether switching precision delivers a useful practical saving, particularly on backbones with substantial quality degradation.
Questions for authors
  1. How were the tail threshold and weight selected? In particular, were they fixed using separate tuning examples before evaluating the other backbones, or chosen from the reported benchmark results?
  2. What changes between QuaRot and the no-BGR configuration? Please clarify the activation quantization and protected modules, since this unexplained difference appears to account for much of the reported improvement.
  3. How does your SVDQuant adaptation relate to the released Infinity implementation in VAR-Compressor? Its usable low-bit results make the near-collapse here worth checking, allowing for the differences in evaluation protocol.
  4. What explains the large CAPS degradation on HART and FlexVAR? Are later stages more sensitive in these models, or is the difficulty mainly caused by switching activation precision?
Reviewer_wsccOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a training-free approach to quantizing pretrained autoregressive image generation models. It introduces two complementary methods: T-BGR and CAPS. T-BGR modifies the quantization calibration objective to place greater weight on errors associated with values in the tails of the weight distribution. CAPS dynamically allocates activation precision across generation stages, using higher precision for early token predictions and lower precision for later predictions. Together, these methods aim to reduce computational cost while preserving image quality. However, their practical speed benefits require further empirical validation.

Critical strengths
  1. The paper brings two sources of non-uniformity into a unified quantization framework: differences across the weight distribution and differences in sensitivity across generation stages.

  2. T-BGR requires neither additional model training nor a high-precision compensation branch, which keeps its implementation relatively simple.

  3. CAPS takes advantage of the coarse-to-fine structure of scale-wise autoregressive generation. Early stages establish the global image structure, whereas later stages account for substantial computation and primarily refine visual details. This provides a reasonable motivation for stage-dependent precision allocation.

Critical weaknesses
  1. Insufficient evidence of practical acceleration. The paper does not provide sufficient measurements of end-to-end latency, throughput, peak memory usage, or energy consumption. FLOPs-weighted bit-width is a useful proxy for computational budget, but it does not establish actual runtime improvements.

  2. Uncertainty around small performance gains. Some reported improvements lack an assessment of statistical uncertainty. For example, the slightly higher ImageReward score of CAPS relative to W4A8 should not be interpreted as evidence that lower precision consistently improves generation quality. Repeated experiments or confidence intervals are needed to support this interpretation.

  3. Limited justification for tail importance. The emphasis on distribution tails remains heuristic, and the underlying mechanism is not sufficiently established. A weight's distance from the mean does not necessarily determine its effect on the output. For a linear layer, quantization-induced output error also depends on the input activations and their correlations, which the proposed objective does not fully capture.

  4. The paper contains cross-reference formatting errors, such as “Eq. equation 1.”

  5. Algorithm 2 should specify how LprevL^{\mathrm{prev}} is initialized and how degenerate cases, including constant-valued channels and zero denominators, are handled.

  6. Claims of “near-full-precision” performance should be qualified by metric and setting. For example, Infinity's ImageReward decreases from 0.95485 to 0.83523, so a general claim of lossless quantization would not be supported.

Questions for authors
  1. Please see the weakness above.

Beside, there are also several question presented here:

  1. Across all evaluated models, what is the incremental benefit of T-BGR over standard BGR when both use identical Hadamard preprocessing and otherwise matched experimental settings?

  2. Does the current implementation execute actual low-bit operations, or does it simulate quantization using higher-precision arithmetic? What improvements are achieved in end-to-end latency and peak memory usage?

  3. Can the small reported performance gains be validated through repeated experiments with different random seeds and corresponding confidence intervals?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.