acceptodds
Withdrawn from ICLR 2027

JustQuant: You Don’t Need Smoothing, SVD, or Rotation for 4-Bit Activation Quantization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

37 views

Abstract

Recent generative models have become increasingly powerful, but their inference cost continues to grow. Model quantization is a promising path for compressing these models and accelerating inference. However, when quantization is pushed to 4 bits, activation quantization becomes substantially more difficult than weight quantization. Recent PTQ (post-training quantization) and QAT (quantization-aware training) methods have made progress on 4-bit activation quantization by introducing smoothing, SVD branches, rotations, mixed precision, or advanced formats such as NVFP. Yet these operators and data types impose demanding requirements on inference engines and hardware, limiting the broad adoption of low-precision models. **Can we quantize using only plain low-bit operators?** To answer this question, we propose **JustQuant**, a simple yet effective framework that moves the complexity required for low-bit quantization from deployment-side operators into an acceptable training process. First, we revisit model quantization from the perspective of knowledge distillation, and show that a key reason existing PTQ and QAT methods fail is that they typically exploit supervision at only a single level. We then introduce Theseus QAD (quantization-aware distillation), a quantization-aware distillation method that progressively applies multi-level supervision, analogous to the gradual replacement process in the Ship of Theseus. Extensive experiments on DiT and dLLM show two regimes. For smaller models, can serve as a light warm-up stage that substantially improves QAT with plain operators, while naive QAD can collapse in the same setting. For larger models, Theseus QAD provides a stronger distillation training path than ordinary QAD. Across these regimes, Theseus QAD improves plain quantization while avoiding the complex operators required by many PTQ methods.

Primary area:
generative models
Keywords:
Quantization, Diffusion Model, Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_yy8BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes JustQuant, a quantization framework based on Theseus QAD, which progressively expands knowledge distillation from local blocks to the full model. The method aims to enable W4A4 and W1.58A4 quantization without additional inference operators such as smoothing, SVD, or rotation. Experiments on DiT, FLUX, and diffusion language models demonstrate improvements over conventional QAT/QAD and competitive generation quality.

Critical strengths
  1. Simple and deployment-friendly quantization. The method avoids additional inference-time transformations, preserving a relatively simple low-bit computation graph. This is practically valuable for efficient hardware deployment.
  2. Promising empirical results. The experiments demonstrate substantial improvements over direct QAT/QAD, particularly under aggressive W4A4 and W1.58A4 settings. The extensive ablations on distillation granularity and training schedules also provide useful evidence for the effectiveness of the proposed approach.
Critical weaknesses
  1. Limited novelty specific to activation quantization. The core method is essentially progressive block-wise knowledge distillation, without an explicit design tailored to 4-bit activations or the unique characteristics of DiT/dLLM. Similar improvements are also observed for weight-only quantization (W1.58A32), raising questions about what makes the proposed approach particularly relevant to activation quantization.
  2. Insufficient validation of the claimed generality. Despite presenting a seemingly general framework for 4-bit activation quantization, the experiments are restricted to DiT and diffusion language models. The absence of mainstream autoregressive LLM evaluations makes it difficult to assess the method's general applicability and competitiveness against established quantization approaches, particularly strong rotation-based methods.
Questions for authors
  1. What specifically motivates the design of Theseus QAD for 4-bit activation quantization or diffusion-based models? Can the authors provide stronger theoretical or empirical evidence connecting the proposed progressive distillation strategy to the distinctive quantization challenges of DiT/dLLM, rather than generic low-bit optimization?
  2. Can the authors provide W4A4 experiments on mainstream autoregressive LLMs (e.g., Llama or Qwen), preferably comparing against competitive QAT/QAD and rotation-based baselines? Such evidence would substantially strengthen the paper's generality claims and could positively influence my assessment.
Reviewer_fTJbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

JustQuant proposes a training-based framework for achieving high-quality 4-bit activation quantization without requiring additional inference operators such as smoothing, SVD, or rotation. Its core contribution is Theseus QAD, a progressive quantization-aware distillation strategy that gradually expands supervision from individual blocks to the full network, combining stable local optimization with global error correction. Experiments on diffusion transformers and diffusion language models demonstrate improved quantization quality over conventional QAT and QAD. For example, on DiT-XL/2 with W4A4 quantization, JustQuant reduces FID from 5.15 to 3.32.

Critical strengths
  1. Effective training strategy: The proposed progressive local-to-global distillation framework provides a simple and intuitive approach to mitigating optimization instability in aggressive activation quantization.

  2. Strong empirical results: The method demonstrates substantial improvements over conventional QAT and QAD baselines, achieving competitive accuracy without introducing additional inference operators.

  3. Evaluation across different settings: The experiments cover multiple quantization configurations, model scales, and generative architectures, providing encouraging evidence of the method's applicability beyond a single model.

  4. Informative ablation studies: The paper systematically investigates distillation granularity, training-step allocation, learning-rate sensitivity, and loss design, offering useful insights into the effectiveness of the proposed training strategy.

Critical weaknesses
  1. Limited methodological novelty: The proposed approach builds upon established block-wise distillation and progressive training techniques. While the specific local-to-global scheduling strategy is interesting, its novelty relative to existing methods such as EfficientQAT and other curriculum-based distillation approaches is not sufficiently established.

  2. Insufficient comparison with alternative training strategies: Although the ablation studies investigate different distillation granularities and training-step allocations, comparisons with alternative progressive schedules, grouping strategies, and multi-level supervision approaches are limited. It remains unclear whether the proposed binary-merging schedule is essential to the observed improvements.

  3. Limited evidence of practical inference benefits: Simplifying the inference graph is a compelling motivation, but the paper provides limited end-to-end performance measurements across different model architectures, workloads, and hardware platforms. More comprehensive latency and throughput evaluations would strengthen the practical claims.

Questions for authors
  1. Can the proposed distillation strategy be effectively applied to other quantization formats, such as MXFP4 and MXFP8, or different quantization granularities?
  2. The results in Table 7(e) suggest that Theseus QAD can complement existing quantization techniques such as RobuQ. Could the authors further investigate its compatibility with individual techniques, including smoothing, SVD, and rotation? In particular, are the improvements complementary or redundant, and how do these combinations affect the trade-off between model quality and inference efficiency?
Reviewer_KBCoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper addresses 4-bit activation quantization, which usually requires extra inference operators such as smoothing, rotation or SVD branches, or special formats.
  • It proposes Theseus QAD, which distills student segments whose width grows from single blocks to the full model before standard QAT or QAD. • The method brings 4-bit image and text generation models close to full-precision quality, outperforming direct QAT.
Critical strengths
  1. The paper pursues a clear, deployment-friendly goal. Plain W4A4 with no extra operators is well motivated, and the image generation results are strong.
  2. The paper tests its schedule directly. The γ sweep and the reverse-order control check the short-to-long ordering.
Critical weaknesses
  1. The gain over known recipes is small. Pure block-wise distillation already performs well, the progressive schedule adds a modest improvement on a single seed, and the recipe is close to EfficientQAT and to BERT-of-Theseus.
Questions for authors
  1. How does direct QAD perform with a learning-rate sweep comparable to the one reported for QAT?
Reviewer_BkB5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a quantization pipeline for compressing large models. The approach consists of first quantizing individual ``blocks" via a knowledge distillation approach, then using the resulting quantized weights as initializations to quantize 2 blocks at a time, then 4, 8, and so on. Numerical experiments illustrate the performance of the approach.

Critical strengths
  • I found the idea of dealing with quantization in successively increasing groupings of blocks, creative, reasonable, and intuitively appealing.
Critical weaknesses
  • Sometimes it was difficult to understand the point of certain parts of the paper, including some of the figures (where we see the output of different models -- these are notoriously susceptible to cherry-picking and add little value), and tables (e.g., table 1, which makes sweeping generalizaitons).
  • Relatedly, there are some claims that are either too strong or unsubstantiated given the evidence. I understand the writing and presentation culture is to over-claim, but I nevertheless find instances of this jarring (examples below).
Questions for authors
  • Table 1: This is not a factual table, and is open to individual judgement and interpretation. What is the boundary between small and large? What does "quality" and "deploy" even mean? What do the checkmarks mean? Why is the table not referenced in the text?
  • Figure 1: What is the point of figure 1? Looking at all 4 subfigures in every row, I have a hard time knowing what is "better". It is also not referenced in the text.
  • Contributions:
    • Define STE before using it.
    • In the third bullet point, you mention that your method reduces FID (define) and increases IS (define), but you don't mention compared to what.
  • Figures and tables: The captions are squished too tightly against the text. Makes it really hard to read (and is probably against hte rules).
  • Important: Rotations can often be mergeable (e.g., see SpinQuant paper). In those cases, they add nothing to the inference pipeline, so your claim about rotations imposing demanding requirements is false in these settings.
  • Equations at the bottom of page 5: Shouldn't the student receive the student state (rather than the teacher state)? Under inference, it would not have the ground truth (teacher) state.
  • Important: In Table 3, Why no comparisons against SpinQuant, or Qronos (which corrects past errors). Also, why are there no inference time comparisons or emulations. These are important because you claim that the paper is motivated by inference time costs but then don't show results related to that.
  • Line 461, you say that "using the W1.58A32 teacher is stronger than using the full-precision teacher", but doesn't this contradict the description of your approach at the bottom of page 5?
Reviewer_pgYvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the question that can low-bit quantization be achieved using only plain low-bit operators, this paper proposes JustQuant, a simple yet effective framework that shifts the complexity of low-bit quantization from deployment-time operators to a manageable training process. They first revisit model quantization through the lens of knowledge distillation and identify a key limitation of existing PTQ and QAT methods: they typically rely on supervision at only a single level. They then introduce Theseus QAD (quantization-aware distillation), which progressively incorporates multi-level supervision, mirroring the gradual replacement process in the Ship of Theseus.

Critical strengths

Figure 4 and Algorithm 1 make the teacher-forced segment construction and the binary-merge schedule easy to understand.

Critical weaknesses
  1. On 50K samples 250 steps on ImageNet, the operator-heavy baselines are better (Table 3). Reasons should be analyzed and provided.
  2. The claimed deployment speedup lacks convincing evidence. Figure 3 reports only W1.58A4 latency on an A6000 using a “24-hour GPT5.6sol kernel-optimization budget,” which does not provide a reproducible comparison with operator-heavy methods. Please report W4A4 latency using standard kernels (e.g., CUTLASS or Nunchaku, as used by SVDQuant) and identical activation granularity across methods.
  3. The novelty is not sufficient. The idea is close to existing work. EfficientQAT, BRECQ and CBQ, and progressive module replacement in BERT-of-Theseus all share the local-to-global structure.
Questions for authors
  1. Why do operator-heavy baselines outperform JustQuant with 50K ImageNet samples and 250 training steps in Table 3? What explains this gap?
  2. Can you report W4A4 latency using standard kernels (e.g., CUTLASS or Nunchaku) and identical activation granularity across methods to provide a reproducible comparison beyond Figure 3’s fixed-budget AI-generated kernels?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.