acceptodds
Under review as a conference paper at ICLR 2027

Representation Matters in 2-Bit QAT: Finite Barnes–Wall Representations for Large Language Models

Abstract

Quantization-aware training (QAT) is widely used to recover the quality of large language models (LLMs) at ultra-low precision, yet most existing methods focus on improving how a given low-bit representation is optimized. We study a complementary question: what representation should QAT optimize in the first place? We introduce BW-QAT, a representation-first framework for 2-bit weight-only QAT based on a finite 16-dimensional Barnes–Wall representation. BW-QAT encodes every 16 weights with a fixed 32-bit payload, calibrates the gain alphabet directly against the finite reconstruction space, and initializes trainable proxies from compensated pre-quantization targets. This initialization preserves the exact initial hard-quantized model while retaining the residual information needed for effective discrete updates. The resulting representation can be trained with hard quantized forward passes and a standard straight-through estimator. Across Qwen and Llama models, BW-QAT consistently outperforms matched lattice, scalar-QAT, and vector-QAT baselines in the low-data regime. On Qwen3-1.7B, it improves six-task average accuracy from 52.17% to 58.30% over a matched LC-QAT adaptation and reduces WikiText-2 perplexity from 18.90 to 14.02 across three training seeds. These results show that representation quality is a first-order factor in data-limited 2-bit QAT.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, quantization-aware training, 2-bit quantization, Barnes–Wall lattice, model compression

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_CHCPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BW-QAT, a representation-first framework for 2-bit weight-only quantization-aware training of LLMs. It introduces a finite 16-dimensional Barnes–Wall lattice representation that encodes 16 weights into a fixed 32-bit payload. The framework incorporates Codec-Consistent Gain Calibration (CCGC) to improve initial quantization quality and decision-preserving initialization to retain continuous residual information for effective hard-forward QAT with a straight-through estimator. Experiments on Qwen3 and Llama-3.2 models demonstrate improved perplexity and downstream accuracy over matched lattice, scalar-QAT, and vector-QAT baselines, particularly in low-data settings, with additional comparisons under matched training-time budgets.

Critical strengths
  1. Novel Quantization Representation. The paper introduces a finite 16-dimensional Barnes–Wall (BW16) lattice representation with a fixed 2-bit-per-weight payload. Unlike conventional scalar or vector quantization approaches, BW-QAT provides a structured finite representation with exact encoding and decoding, codec-consistent gain calibration, and hard-forward QAT. Importantly, the same discrete representation is maintained throughout training and deployment. This constitutes a meaningful representation-level contribution to extremely low-bit LLM quantization.

  2. Strong Empirical Improvements with Detailed Comparisons. The paper demonstrates substantial improvements over existing scalar-QAT and vector-QAT baselines, particularly in low-data settings. For example, on Qwen3-1.7B, BW-QAT improves the six-task average accuracy from 52.17% to 58.30% over LC-QAT and reduces WikiText-2 perplexity from 18.90 to 14.02. The experiments also include matched training-data budgets, matched wall-clock comparisons, multiple random seeds, and ablation studies that isolate the effects of the representation and initialization. These controlled evaluations strengthen the empirical evidence supporting the proposed method, despite its higher computational cost.

Critical weaknesses
  1. Limited Methodological Novelty. Although the finite BW16 representation is a potentially useful contribution, the overall framework builds upon established lattice-based vector quantization and STE-based QAT methods, including QuIP#, Lattice-based QAT, and LC-QAT. In particular, the proposed decision-preserving initialization closely resembles conventional QAT approaches that maintain continuous latent weights while performing quantized forward passes. Its novelty beyond initializing the latent weights from compensated targets is not sufficiently established. Furthermore, the representation comparisons in Table 1 involve substantially different numbers of active lattice points, making it difficult to determine whether the improvements arise from the specific Barnes–Wall lattice geometry or simply from a larger effective reconstruction space. More detailed comparisons are needed to evaluate the contributions.

  2. Limited Experimental Scope The experimental evaluation is restricted to three relatively small models (Qwen3-0.6B, Llama-3.2-1B, and Qwen3-1.7B), without results on commonly used 7B/8B or larger LLMs. This is particularly important given the substantial computational overhead of BW-QAT. For example, Table 4 shows that BW-QAT requires approximately 14.5 times more QAT time than LC-QAT under the same training-label budget. Although the matched-time experiments demonstrate a remaining performance advantage, the method's scalability and computational practicality are insufficiently validated. Moreover, the downstream evaluation focuses primarily on commonsense reasoning and question-answering benchmarks, while more challenging tasks such as mathematical reasoning, code generation, and instruction following are not evaluated. In addition, the lack of inference latency and throughput measurements further limits the assessment of the proposed representation's practical deployment benefits.

Questions for authors

1. Methodological Novelty. How does the proposed decision-preserving initialization fundamentally differ from conventional STE-based QAT methods that maintain continuous full-precision latent weights during quantized forward passes? Table 2 primarily compares initialization from compensated targets against initialization from quantized codewords. Could the authors also clarify or evaluate the difference between initializing from the original full-precision weights and the proposed compensated targets?

2. Contribution of Barnes–Wall Geometry. In Table 1, BW16 has substantially more active lattice points (237.3M) than Product-E8 (83.2M) and JS-E8 (182.0M), despite the same nominal storage budget. How can the authors establish that the reported improvements arise from the specific Barnes–Wall lattice geometry rather than the larger effective reconstruction space? Could the authors provide comparisons with approximately matched codebook cardinalities or additional analyses isolating the effects of lattice geometry, codebook capacity, and energy shaping?

3. Scalability to Larger LLMs. The experiments only cover models up to 1.7B parameters. Given the considerable computational overhead of the proposed finite BW16 quantizer, how would BW-QAT scale to widely used 7B/8B or larger LLMs?

4. Practical Inference Efficiency. Although the paper reports serialized compression rates and evaluates independently reloaded quantized models, it does not provide inference latency or throughput measurements. How efficiently can the proposed BW16 representation be decoded and executed during inference compared with existing scalar and vector quantization methods? Does the exact lattice decoding introduce substantial runtime overhead, and are specialized inference kernels required?

5. Standard Benchmarks. Is there evidence that the improvements extend to more challenging capabilities, such as mathematical reasoning, code generation, and instruction following?

Reviewer_m1K8OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies 2-bit weight-only quantization-aware training and extends the E8 lattice codebook to the BW16 lattice. Both are special cases of the Barnes–Wall lattice. Each block of 16 weights is encoded by a 28-bit lattice rank and a 4-bit gain-factor rank. Gain factors are calibrated at initialization and frozen during QAT. The 28-bit rank is recovered by dynamic programming without materializing the codebook. The paper also introduces activation-aware compensated targets and uses them as the initial trainable proxy.

The main contribution is a dynamic-programming codec for BW16 that avoids codebook materialization and makes high-dimensional lattice quantization practical. The method still incurs substantial quantization overhead and poor wall-clock performance.

Critical strengths
  • Dynamic-programming BW16 codec. Encoding BW16 without materializing the codebook makes high-dimensional lattice quantization practical; the full codebook would not fit in GPU L1 cache.
Critical weaknesses
  • Quantization overhead. The proposed method takes about 4200s per 1M labels, compared with about 290s for LC-QAT.
  • Missing comparisons with QuIP#/QTIP-style finetuning. There is no comparison with QuIP#/QTIP-style finetuning at Q0 and 100k steps.
Questions for authors
  • The orthogonal transform RR is not defined. The code appears to use a Hadamard transform.
  • The activation-aware compensated target in Eq. (9) seems similar to BlockLDLQ in QuIP#/QTIP, except that QuIP#/QTIP sets U=L−IU = L - I while this paper sets U=LU = L. Is there a derivation for this choice?
  • It is unclear whether the compensated target, gain calibration, and decision-preserving initialization are used in Table 1. This makes it difficult to relate the BW16 result in Table 1 to Tables 2 and 12.
  • Is there a FP16 reference reported for Table 3?
  • Section 4.5 states that increasing the calibration set from 16 to 64 sequences lowers the 100K NLL from 3.3847 to 3.3345. Table 2 reports 3.3057 without CCGC and 3.2946 with CCGC, which suggests a limited gain from calibration. Could the authors clarify this gap?
  • Is the dynamic-programming decoder in Appendix A.1 compared with Babai rounding?
  • How does the method scale to higher bit widths?
Reviewer_uD19OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BW-QAT, a 2-bit weight-only QAT framework using a finite 16-dimensional Barnes–Wall lattice. It combines fixed-rate rank-and-gain encoding, codec-consistent gain calibration, and decision-preserving initialization. On Qwen and Llama models up to 1.7B parameters, it reports improved perplexity and task accuracy over the evaluated scalar and vector QAT baselines, particularly with limited training data.

Critical strengths
  1. A concrete low-bit representation. The authors construct a finite BW16 codebook with a fixed 32-bit payload per 16 weights, supported by structured search and rank-based encoding. The initialization ablation also shows that retaining the pre-quantization residual improves hard-state updates during QAT.

  2. Meaningful accuracy gains. BW-QAT improves perplexity and downstream accuracy over the tested QAT baselines. The matched-training-time LC-QAT comparison is valuable given the substantially higher training cost of BW-QAT.

Critical weaknesses
  1. Practical inference latency is not evaluated. Unlike ordinary scalar unpacking, recovering BW16 weights requires decoding rank-and-gain records. This extra work may offset the memory-bandwidth savings of 2-bit storage, especially for latency-sensitive inference. The paper does not report decoding overhead or end-to-end inference results using compressed weights, so the practical runtime benefit remains unclear.

  2. The choice of BW16 is not sufficiently justified. Showing that BW16 works does not explain why 16 dimensions are preferable to alternatives such as BW8 or BW32. These choices may have different accuracy, encoding, and decoding costs, but the paper provides no dimensionality trade-off. Moreover, the E8 comparisons vary in both lattice geometry and finite codebook size, making the source of BW16's advantage difficult to isolate.

Questions for authors
  1. Inference overhead: How much time does rank-and-gain decoding add during inference? Could the authors report end-to-end latency, throughput, and peak GPU memory, ideally against a conventional low-bit quantization baseline? Can decoding be fused into GEMM without materializing full-precision weight matrices?

  2. BW16 design choice: What justifies selecting BW16 rather than BW8 or BW32? How would the dimensions compare in quantization quality and encoding/decoding complexity? Can the observed advantage over E8 be separated from differences in codebook size?

  3. Scalability: What dominates BW-QAT's training overhead, and how feasible is the method for 7B/8B-scale models?

  4. Baseline settings: Was LC-QAT's learning-rate schedule adapted to the short training budget? Were the E8 representations given independent, comparable gain calibration?

Reviewer_vBtPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BW-QAT, a 2-bit weight-only QAT method built around a finite 16-dimensional Barnes–Wall lattice. It combines fixed-rate encoding, gain calibration, and an initialization that preserves quantization residuals without changing the initial hard model. Experiments on Qwen and Llama models show improved low-data performance; on Qwen3-1.7B, six-task accuracy increases from 52.17% to 58.30% over the matched LC-QAT baseline. I lean toward weak acceptance: the finite codec and controlled initialization experiment are worthwhile contributions, although baseline matching and practical deployment costs need further clarification.

Critical strengths

The finite codec is a useful technical contribution. It supports exact search and fixed-width encoding without storing the full 237-million-point codebook. Evaluating independently reloaded records and reporting actual storage rates also strengthens the connection between the training method and the compressed model.

The initialization ablation is particularly convincing within the tested setting. Starting from the same hard model, retaining the compensated targets enables substantial improvement, while codeword initialization barely changes the model. The three-seed main comparisons and matched-training-time results provide additional evidence that the low-data gains are meaningful.

Critical weaknesses

The large scalar-QAT gains do not cleanly isolate representation choice. BW-QAT uses rotations, activation-aware compensation, and calibrated gains, but equivalent preprocessing is not clearly specified for the scalar baselines. Their initial NLLs are already much worse in Table 3. Comparisons with similarly strong initialization would help establish how much of the advantage comes from the representation itself. The more controlled lattice comparison is helpful, but uses only one seed.

The practical deployment case remains incomplete. BW-QAT takes roughly 14.5 times longer than LC-QAT for the same training-label budget. The matched-time result is encouraging, but its timer excludes initialization and export, and no inference throughput or peak-memory measurements are reported. End-to-end timing and inference measurements would clarify whether the compact serialized format translates into practical savings.

Questions for authors
  1. What rotations and activation-aware initialization do the scalar baselines receive? Are Product-E8 and JS-E8 given the same gain-calibration and proxy-initialization procedures as BW16?
  2. Does inference operate directly on packed BW records, decode blocks on demand, or reconstruct full weight matrices? What peak memory and decoding overhead does the current implementation incur?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.