acceptodds
Withdrawn from ICLR 2027

AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-grained mixed-precision quantization promises efficient large language model inference, but local precision choices can conflict with regular GPU storage and computation units. This precision-boundary mismatch limits the translation of compression into practical acceleration. We introduce AlignQuant, a post-training quantization method that uses GPU-compatible two-dimensional weight tiles as the common unit of precision allocation, compact storage, and execution. This shared partition lets precision follow sensitivity within output channels. Joint prefill/decode calibration scores precision reductions using projection-output perturbations weighted by language-model loss gradients under quantized activations. Phase-normalized scores prioritize higher precision for tiles important to either phase under a model-wide weight-storage budget. Each tile stores one selected representation, while phase-specialized kernels reuse the packed model and expand lower-bit weights for INT8 computation with 8-bit activations. Across four LLMs spanning 3B to 14B parameters, AlignQuant achieves up to generation speedup over BF16 while preserving model quality. Evaluations further cover three GPUs and contexts up to 64K tokens. These results show that local precision flexibility and regular GPU execution can coexist through a shared tile unit. The implementation is available at https://anonymous.4open.science/r/AlignQuant-0B64.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
mixed-precision quantization, post-training quantization, LLM inference, GPU acceleration, hardware-aware quantization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dvX4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AlignQuant is a post-training W4/W8 mixed-precision quantization framework using hardware-aligned weight tiles for sensitivity-based precision allocation. It combines gradient-weighted projection-output perturbations to select tile precision. The authors then report 2.50x generation speedup over BF16.

Critical strengths
  1. The paper presents a complete implementation, including calibration, packing, specialized GPU kernels, and runtime optimizations. The evaluation covers multiple models, GPUs, workloads, and benchmarks.
  2. Using a common tile structure for precision allocation, storage, and execution is, of course, a reasonable design.
Critical weaknesses
  1. Block-wise mixed precision and sensitivity-driven bit allocation have been extensively studied. For example, FGMP, SFMP, and ScaleBITS… (The authors also cited these papers). The distinction of AlignQuant from these methods is not clear. Also, the proposed sensitivity score appears to be a heuristic, yet the paper does not compare it against prefill-only, decode-only, or alternative score combinations. Therefore, it’s not clear what is the technical contribution in the current version.
  2. AlignQuant uses approximately 5.61 bits per weight (Table 16, W4 60%), compared with roughly 4-bit GPTQ/AWQ baselines. Then it seems that the comparisons are made without matched budgets. I’m also curious about why GPTQ/AWQ exhibit unusually large HellaSwag accuracy drops (e.g., 71.52% to 52.62%/50.41% on Llama-3.2-3B-Instruct) despite relatively small perplexity changes. Also, the large speedups appear dependent on the specialized runtime. Table 16 shows that, under the same runtime, W8-only and W4-only achieve 141.72 and 138.96 tok/s, compared with 132.28 tok/s for the proposed mixed configuration. Moreover, the W4-only baseline achieves only 47.33 tok/s in Table 1 (Is that because the authors used a different backend?). It remains unclear how much of the advantage comes from the proposed quantization scheme versus differences in kernel implementations and runtime optimization.
Questions for authors
  1. What are the concrete technical differences from FGMP, SFMP, and ScaleBITS? Can the authors ablate prefill-only, decode-only, and joint sensitivity scoring?
  2. Can the authors provide matched-bit-budget quality comparisons? Please also provide interpretation on the accuracy drops.
  3. Can the authors compare against optimized quantization backends under comparable runtime conditions and report quality-memory-throughput tradeoffs? How much of the speedup is specifically attributable to mixed-precision allocation?
Reviewer_hSbROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AlignQuant, a mixed-precision quantization method for LLMs. It uses 2D weight tiles to combine precision allocation, weight storage, and GPU execution. It also uses joint Prefill-Decode calibration to give more bits to important weights. The method supports both 4-bit and 8-bit weights with efficient GPU kernels. The authors test it on four LLMs and different GPUs. The results show up to 2.50× speedup over BF16 while keeping similar model quality. Studies also show that the main parts of the method are useful.

Critical strengths

One good thing about this paper is that it brings precision allocation, weight storage, and GPU execution together. This makes it easier to handle different bit-widths during quantization.The method can give more bits to important weights while keeping inference fast. I think this is the most valuable part of the paper.

The authors test their method on different models, GPUs, and benchmarks. They also test long-context tasks up to 64K tokens. The results show that the method works well in different settings.The paper also includes ablation studies and hyperparameter experiments in the appendix. These experiments help show that the proposed method is effective.

Critical weaknesses
  1. Most of the key techniques are based on existing work, such as weight rotation, Fisher-based sensitivity scoring, and INT8 kernels. The paper mainly combines these methods rather than introducing new ideas. I think the novelty is limited.

  2. The paper focuses on inference speed but gives little information about offline costs, such as calibration time and model packing time. This makes it hard to judge the total cost of using the method.

  3. The tile-size experiments only use square tiles. The paper does not test rectangular tiles like 32×64 or 64×128. Also, it uses the maximum Prefill-Decode sensitivity score without a clear comparison with other methods, such as a weighted average. So it is unclear whether these choices are the best.

Questions for authors
  1. The paper only tests 32×32, 64×64, and 128×128 tiles. Why not test other shapes, such as 32×64 or 64×128? How do different tile shapes affect accuracy and inference speed? And how do you choose the best tile size for a new model?

2.How long does the full offline process take for each model? This includes weight transformation, calibration, and model packing. These costs are important for real-world use, but the paper does not clearly write them.

  1. Why do you use the maximum of the normalized Prefill and Decode sensitivity scores? Have you tried other methods, such as a weighted average? What makes the max-based method a better choice?
Reviewer_de7UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AlignQuant, a tile-aligned mixed-precision post-training quantization framework for efficient LLM inference. The key idea is to use the same two-dimensional weight tiles as the unit for precision allocation, compact storage, and GPU execution, so that fine-grained W4/W8 choices remain compatible with regular hardware computation. AlignQuant uses loss-aware prefill/decode calibration to identify sensitive tiles, assigns 4-bit or 8-bit weights under a global storage budget, and implements phase-specific kernels that execute the packed mixed-precision model with 8-bit activations and INT8 arithmetic. Experiments on four LLMs from 3B to 14B parameters and three NVIDIA GPUs show that the method preserves model quality while reducing memory and improving end-to-end generation efficiency, including long-context settings.

Critical strengths
  1. AlignQuant uses loss-gradient-weighted projection perturbations from both prefill and decode to decide which tiles should retain W8, providing a more informed allocation than coarse mixed-precision schemes.

  2. The same 2D tile is used for precision assignment, compact W4/W8 storage, and GPU execution, allowing fine-grained mixed precision to be deployed without introducing irregular computation boundaries.

Critical weaknesses
  1. AlignQuant uses 60% W4 and 40% W8 weights (about 5.61 packed bits/weight), while GPTQ and AWQ use 4-bit weights, so the higher accuracy may partly come from using a larger precision/storage budget. A comparison at matched BPW would better isolate the benefit of the proposed allocation strategy.

  2. Uniform W4 methods such as GPTQ and AWQ already avoid mixed-precision boundary mismatch and should benefit from substantially lower weight traffic during memory-bound decoding, yet they are reported at only 48.13 and 45.91 tok/s, even below the 58.18 tok/s BF16 baseline, while AlignQuant reaches 132.28 tok/s. Without comparison against a competitive optimized W4 kernel or a uniform-W4 implementation under the same runtime, it is difficult to determine how much of the speedup comes from tile alignment rather than kernel and runtime engineering.

Questions for authors
  1. Can the authors provide quality comparisons against GPTQ/AWQ at matched packed BPW?

  2. What exact kernels/backends and activation/compute precisions are used for GPTQ and AWQ in the system evaluation?

  3. Can the authors provide an end-to-end runtime comparison between tile-aligned allocation and alternative mixed-precision layouts under the same W4/W8 budget and the same runtime stack?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.