acceptodds
Withdrawn from ICLR 2027

Vodka: Vector quantization-aware training via Differentiable Kernel Approximation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The low bits quantization of Large Language Models (LLMs) is essential for lowering deployment costs and enabling inference with limited resources. As vector quantization (VQ) offers higher expressivity than scalar quantization (SQ) and quantization-aware training (QAT) demonstrates a higher upper-bound than post-training quantization (PTQ), combining VQ with QAT shows better potential for low-bit quantization. However, in face of the optimization and efficiency problems, current VQ-QAT methods either constrain the expressivity, or require extensive search, which motivates us to explore the tradeoff between expressivity and efficiency. Therefore, we propose Vodka, a novel VQ-QAT algorithm via differentiable kernel approximation. We identify the key expressivity gap as the mapping function: current VQ-QAT restricts this map to be affine, whereas VQ allows it to be arbitrary. To bridge the gap, Vodka constructs the Codebook Spectrum with kernel by ordering the curvature, with two endpoint functions provably the affine and arbitrary, which stays search-free and admits an exact codebook gradient at fixed indices, serving as implicit codebook to approach the VQ controllably. In contrast to prior work with a fixed initialized codebook, we optimize the implicit codebook jointly with the weights, so that both adapt to each other. We also propose Shakernel, a fused GPU kernel that implements Vodka without heavy intermediate tensors, reducing training cost and memory. Empirical evaluations on Qwen3.6-35B-A3B and Llama3.1-8B on 7 benchmarks demonstrate that Vodka outperforms state-of-the-art (SOTA) low-bit methods while maintaining efficiency. Notably, Vodka improves over the strongest baseline by 1.31% on IFEval and 0.51% on BFCL, demonstrating the excellent performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Vector Quantization, Quantization-Aware Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Fqj2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Vodka, a novel VQ-QAT algorithm via differentiable kernel approximation, to address the tradeoff between expressivity and efficiency. The authors identify the key expressivity gap as the mapping function and construct the Codebook Spectrum by ordering curvature, with affine and arbitrary maps. The method stays search-free and admits an exact codebook gradient at fixed indices. The authors jointly optimize the implicit codebook with the weights and propose Shakernel, a fused GPU kernel that reduces training cost and memory. Evaluations on Qwen3.6-35B-A3B and Llama3.1-8B across seven benchmarks demonstrate improvements over SOTA low-bit methods while maintaining efficiency.

Critical strengths

The proposed Codebook Spectrum provides a principled way to bridge affine quantization and unconstrained vector quantization. It improves codebook expressivity while preserving search-free index assignment.

The joint optimization of codebook parameters and weights enables both to adapt to the task loss. Experiments show substantially faster convergence than fixed-codebook VQ-QAT methods.

The proposed Shakernel significantly reduces the computational and memory overhead when training the more expressive quantizer.

Critical weaknesses

Figure 6 compares a fixed affine codebook at m=0 with trainable nonlinear codebooks at m>0, making it difficult to isolate the benefit of nonlinear codebook.

In figure 5, the accuracy improvements over existing methods are relatively small. Since LC-QAT has not converged, it is unclear whether Vodka maintains its advantage after both methods fully converge.

The inference efficiency is not sufficiently evaluated. No actual GPU inference latency or throughput results are provided.

Questions for authors

Can the authors provide an ablation that independently evaluates the benefits of nonlinear codebook and joint codebook-weight optimization?

How does Vodka compare with LC-QAT when both methods are trained to convergence?

What are the actual inference latency and throughput of Vodka on GPUs compared with existing VQ-QAT methods?

Reviewer_aZRfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Vodka, a VQ-QAT method that extends affine codebook parameterizations through a spectral expansion of a discrete curvature operator. The resulting family interpolates between affine and unconstrained codebooks while retaining search-free indexing. The authors jointly optimize decoder coefficients and latent weights, and introduce Shakernel to reduce the memory and computational overhead of the resulting gradients. Experiments on two LLMs show modest improvements over existing 2-bit quantization methods.

Critical strengths

The spectral construction is mathematically well-motivated, with a clear characterization of the affine null space and the unconstrained endpoint. The GPU implementation also addresses a genuine bottleneck, where it avoids materialization of per-group Jacobians and substantially reduces the proposed training overhead. The convergence results are scientifically interesting, particularly in the low-data QAT setting, if it holds under a compute-matched comparison

Critical weaknesses
  1. The experiments do not isolate the benefit of the spectral parameterization In Fig. 6, m=0 uses a frozen affine codebook while m>0 trains the decoder jointly, so we cannot isolate the gains between expressivity and codebook trainability, even though Appendix E's argues that affine quantization is already near-optimal in Gaussian reconstruction distortion. Because distortion under a fixed source does not determine the benefit of training the decoder against task loss. A trainable m=0 decoder, which Eq. (7) already supports, is an essential control. The Appendix F polynomial ablation likewise does not state whether its features are orthonormalized and scaled like Vodka's. Also, performance keeps improving up to the full table (m=251), and the paper itself argues that decoder storage is negligible and traversal cost is independent of m. The 87.5% reduction at m=27 saves only about 180K parameters on a 35B model, with no demonstrated savings in time, memory, or inference cost. The practical advantage of spectral truncation over a fully trainable codebook therefore remains unclear.

  2. Gains are small and not compute-matched. On Qwen, Vodka improves over the strongest baseline by 0.31 (LiveCodeBench), 0.51 (BFCL), and 1.31 (IFEval), and by 0.08 on the average of basic benchmarks. Sec. 4.4 states that Vodka plateaus at 375 steps while LC-QAT is still improving, and Vodka is 21% slower per step, so equal steps mean neither converged accuracy nor equal compute. The convergence advantage in Fig. 5 should be shown as accuracy versus GPU-hours against baselines trained to convergence.

Overall, I lean toward rejection as the experiments do not show that the spectral construction, rather than codebook training, drives the gains, and the remaining improvements are small and not compute-matched.

Questions for authors
  1. How does a jointly trained affine decoder (m=0) perform under the same optimization setup?
  2. What measured advantage justifies m=27 over the full-table endpoint?
  3. Do the improvements persist under equal GPU-hour budgets?
  4. Were the polynomial baselines orthonormalized and scaled in the same way as the spectral decoder?
Reviewer_meWqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Vodka, a vector quantization-aware training method that combines search-free indexing with a trainable nonlinear implicit codebook. Its central contribution is a curvature-ordered codebook spectrum that interpolates between affine mappings and unconstrained codebooks. The authors also introduce Shakernel, a fused GPU kernel that reduces intermediate storage and memory traffic. Experiments on two LLMs under 2-bit weight quantization demonstrate improvements on selected complex tasks, with moderate additional per-step training costs.

Critical strengths
  1. The curvature-based construction provides a principled and mathematically characterized way to increase codebook expressivity beyond affine mappings.
  2. Search-free indexing and joint codebook optimization offer a practical approach to training flexible vector quantizers.
  3. The fused implementation substantially reduces computational overhead, while the capacity sweep and polynomial comparison provide useful evidence supporting the design.
Critical weaknesses
  1. The evaluation coverage is limited. Experiments include only two models, and the basic capability evaluation contains just four lm-eval tasks, with marginal average improvements. Additional model families, scales, and benchmarks, particularly mathematical reasoning and broader knowledge evaluation, are needed to establish generality.

  2. The ablations do not sufficiently isolate the proposed mechanisms. The affine endpoint uses a fixed codebook, whereas nonlinear variants jointly optimize the codebook and index proxies, confounding expressivity with trainability. A trainable affine baseline and controlled ablations of the two optimization paths are needed to identify the source of the gains.

  3. The baseline comparisons provide incomplete evidence of superior final quality. Competitive QAT coverage is limited, particularly on Llama, and LC-QAT remains unconverged when the shared 375-step budget ends. The current results therefore establish an advantage under the reported training schedule, but leave the performance gap after convergence unresolved.

Questions for authors
  1. Could the authors provide a detailed training-efficiency comparison? Please report the GPU model, absolute step time, peak memory, and total wall-clock time to reach a common target score, including initialization and preprocessing costs.

  2. How was the learning rate of 10−410^{-4} selected? Were learning rates tuned independently for Vodka and each QAT baseline, and does the performance advantage persist when each method uses its best tested setting?

Reviewer_jmUqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Vodka proposes a weight-only vector quantization-aware training method for large language models. Its central contribution is a spectral codebook parameterization that combines an affine component with eigenvectors of a discrete curvature operator. This produces nested codebook families with exact affine and unrestricted endpoints. Coordinate-wise rounding of latent variables enables search-free assignments, while decoder coefficients and latent variables are jointly optimized using exact coefficient gradients at fixed assignments and surrogate index gradients. The paper also introduces Shakernel, a fused GPU implementation that avoids materializing per-group Jacobians. Experiments evaluate nominal 2-bit quantization on Qwen3.6-35B-A3B and Llama3.1-8B across seven benchmarks. Vodka reports improvements over the strongest evaluated baselines on instruction following, function calling, and code generation, while gains on basic benchmarks are small. Capacity sweeps suggest that a relatively compact decoder captures most of the observed benefit. Shakernel reduces training time and memory substantially relative to the unfused implementation, although both remain above LC-QAT. Overall, the work combines a principled representation with practical implementation, but stronger controls and statistical reporting are needed to establish the sources and robustness of its gains.

Critical strengths
  1. The curvature construction provides a coherent family spanning affine decoders and unrestricted codebooks. The null-space characterization and endpoint proofs clearly establish its representational properties rather than relying solely on intuition (Section 3.1; Appendix B).

  2. Search-free indexing and shared decoder coefficients offer a practical way to train more expressive codebooks. The manuscript correctly distinguishes exact coefficient gradients at fixed assignments from surrogate gradients through rounding (Section 3.2–3.3).

  3. Shakernel exploits repeated indices to avoid redundant Jacobian storage. Reported reductions of 70.3% in step time and 63.8% in peak memory relative to unfused Vodka materially improve the method’s practicality (Figure 4).

  4. Evaluation covers both dense and mixture-of-experts models and seven benchmarks. On Qwen, gains over the strongest reported baselines reach 1.31 percentage points on IFEval, 0.51 on BFCL, and 0.31 on LiveCodeBench (Table 1).

Critical weaknesses
  1. Results are averaged across seeds, but the seed count, standard deviations, confidence intervals, and paired differences are absent. Several improvements are below one percentage point, making their reliability difficult to assess. Figures 5–6 also omit uncertainty estimates (Table 1; Appendix D).

  2. The manuscript explicitly states that LC-QAT is still improving when training stops at 375 steps. Moreover, Vodka costs 1.21× per step. Equal update counts therefore establish neither equal-compute performance nor an advantage over converged baselines. Both comparisons are needed (Section 4.3–4.4; Appendix D).

  3. Vodka uses an orthonormalized basis and coefficient scaling, while Appendix F does not specify equivalent treatment for polynomial features. Since AdamW depends on parameterization, Table 5 may reflect conditioning and optimization differences alongside subspace selection. Matched basis normalization, scaling, and tuning would strengthen the claimed advantage.

Questions for authors

see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.