acceptodds
Desk rejected at ICLR 2027

Adaptive Mixed-Precision Codebook Quantization for Large Language Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Codebook quantization enables efficient compression of large language models to low bit-widths, making it suitable for resource-constrained deployment. Existing methods assign a uniform codebook configuration to all matrices. However, under a tight average-bit budget, this uniform configuration leads to capacity misallocation: insensitive linear maps receive disproportionate precision, while sensit ive projection layers remain under-represented. To address this issue, we propose Adaptive Mixed-Precision Codebook Quantization (AmpCQ), a framework that assigns different codebook configurations based on per-matrix sensitivity. The proposed method combines: (i) a heterogeneous tier space comprising single- and dual-codebook quantizers; (ii) a normalized activation-aware reconstruction metric for cross-matrix comparability; (iii) Pareto-filtered greedy allocation guided by marginal return per bit; and (iv) lightweight codebook-delta adaptation for downstream tasks. On Llama3-8B and Qwen3-14B, AmpCQ demonstrates superior performance over uniform AQLM-style allocation at matched average bit width. Specifically, when Qwen3-14B is compressed to an average bit width of approximately 3 bits, it retains 97.3% of its original average zero-shot accuracy. Our code is available in the Supplementary Material.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Model Compression, Codebook, Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LSSmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AMPCQ, which assigns matrix-specific additive-codebook tiers under a model-level bit budget. It uses activation-normalized profiling, Pareto filtering, greedy return-per-bit allocation, and block-wise codebook recovery for quantization. AMPCQ improves quality over uniform AQLM allocation, but the formulation contains internal inconsistencies and the practical advantage over existing mixed-precision methods is not sufficiently established.

Critical strengths
  1. A practical allocation framework with strong gains over uniform codebook allocation and useful ablations. At similar reported bit widths, AMPCQ substantially improves perplexity and zero-shot accuracy over uniform AQLM on Llama3-8B and Qwen3-14B. The uniform-versus-mixed ablations also provide clear evidence that matrix-wise tier assignment is useful. Also, the combination of activation-normalized profiling, storage-aware allocation, and heterogeneous single- and dual-codebook tiers is practically motivated. The paper examines distortion metrics, allocation strategies, profiling-set size, and ROI sensitivity as well.
Critical weaknesses
  1. The mathematical formulation is internally inconsistent. The main budget constraint treats RiR_i as if it were a per-weight rate, whereas Appendix A.3 defines it as the total storage of matrix ii; multiplying it again by matrix size therefore double-counts that size. The assumptions of Theorem A.2 are not enforced by Pareto filtering, and its discussion of multi-chain greedy allocation under uniform costs is incorrect. The Stage 3 adapter equations are also dimensionally invalid. These issues make it unclear whether the implemented allocator matches the stated optimization problem.

  2. Novelty and practical efficiency are insufficiently established. The main contribution largely applies established mixed-precision allocation techniques to codebook tiers. Comparisons with related allocators are limited to adapted versions at one Llama3-8B operating point with Stage 2 disabled. Profiling and quantization costs are not reported, and the throughput advantage is inconsistent: AMPCQ improves batch-1 throughput but is substantially slower than AQLM at the maximum A100 batch setting. The main evaluation is also limited to two dense language models without repeated seeds.

Questions for authors
  1. Budget implementation. Does the implementation treat RiR_i as total storage or as bits per weight? After correcting the budget equation, do the tier assignments, reported average bit widths, and main results remain unchanged?

  2. Full-pipeline comparison. Does AMPCQ retain its advantage over Q-Palette-, SliM-LLM-, and other mixed-precision allocation methods when all methods use the same codebook tiers, bit budget, and complete Stage 1–2 pipeline? Please also provide the exact allocation and optimization procedure.

  3. Practical cost. What are the wall-clock time and peak memory required for profiling all tiers and quantizing the full model (regarding all resources required for the quantization phase itself)? Under matched batch size and memory constraints, does AMPCQ maintain a latency or throughput advantage over AQLM (regarding the inference phase of the quantized model)?

2 comments

We thank the reviewer for the detailed and rigorous feedback. We have addressed the concerns point-by-point with corrected formulations, full-model experiments, and honest cost reporting.

W1 & Q1: Mathematical Formulation and Implementation

  1. Budget & Code: We agree with the reviewer. App. A.3 defines RiR_i as total storage, but Eq. (5)/(28) implicitly treated it as per-weight. We have unified the notation: RitotR_i^{\text{tot}} for total bits, and ri=Ritot/Nir_i = R_i^{\text{tot}}/N_i for bits per weight. Eq. (5) is now ∑iNiri(τi)≤B\sum_i N_i r_i(\tau_i)\le B. We audited our code: the implementation correctly uses rir_i and does not double-count. All main results, tier assignments, and bit widths remain strictly unchanged. We added the storage breakdown to App. A.3.
  2. Theorem A.2 & Stage 3: We agree Pareto filtering does not enforce the non-increasing marginal return assumption. We have revised the text to treat Theorem A.2 as an idealized single-chain case and removed the global optimality claim. We also corrected the dimensional typos in the Stage 3 adapter equations (Eq. 9-11).

W2 & Q2: Novelty and Full-Pipeline Comparison

We agree and have added full-pipeline (Stage 1 + Stage 2) comparisons on all 224 linear matrices of Llama3-8B under identical codebook tiers and budgets.

MethodWiki2 PPL ↓Avg. 5-Task Acc ↑
Q-Palette-style + Stage 28.2463.21
AMPCQ + Stage 28.2265.50
  • Full-Model Ablation: The five-tier palette also outperforms the restricted two-tier space on full-model PPL (6.90 vs. 7.27).
  • Procedure: We have added Algorithm 1 (pseudo-code for normalized profiling, Pareto filtering, and ROI greedy assignment) to the revised Appendix.

Q3: Practical Cost and Throughput

  1. Quantization Cost (Q3): We report the practical cost on Llama3-8B (A100) in the table below. Stage 1 overhead is minimal.
    • Stage 1 (Profiling): ~35 mins, 18.5 GB peak.
    • Stage 2 (Quantization): ~4.2 hours, 45.0 GB peak.
  2. Throughput & Matched Batch (W3): We agree the original Table 6 (max batch per method) was difficult to interpret. We have relabeled it as "Saturating Throughput" and added matched-batch/bit-width data:
    • Matched A100 (Batch=16, both ~2.0 bits): AQLM = 110.0 tok/s (27.2 GiB); AMPCQ = 85.0 tok/s (32.2 GiB).
    • Matched A100 (Batch=1): AMPCQ = 20.1 tok/s (4.60 GiB); AQLM = 18.3 tok/s (4.15 GiB).
  3. Revised Claims: The higher per-batch memory footprint (1.84 vs 1.44 GiB) limits AMPCQ at high concurrency. We now state that AMPCQ is advantageous in low-batch, latency-sensitive scenarios, and we have removed the general throughput advantage claim. High-batch optimization remains future work.

Thanks for the rebuttal. However, I cannot easily follow the revised text. Could you highlight all the changed parts in the revised manuscript for ease of reference? Also, please reflect the additional results formally within the manuscript and refer them here. Hope that the results within the comment above is the essential part, which the full result would be contained within the paper.

Reviewer_9GmwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how to assign different codebook configurations to LLM weight matrices under a model-wide storage budget. AMPCQ selects from five single- and dual-codebook tiers using normalized activation-output reconstruction error and greedy upgrades based on error reduction per additional bit. It then optimizes codebooks, scales, and discrete indices through block-wise reconstruction. An optional codebook adapter is used for downstream tasks. Experiments on Llama3-8B and Qwen3-14B report improvements over uniform AQLM-style quantization at several bit budgets, together with allocation ablations and inference measurements.

Critical strengths
  1. Different weight matrices can have different quantization sensitivity. Assigning codebook capacity and single/dual-codebook configurations separately is a reasonable way to use a limited storage budget.
  2. The allocation ablations are useful. The paper compares uniform and mixed configurations, different error metrics, ranking with and without bit cost, and calibration sample counts. The appendix also compares adapted allocation strategies within the same VQ candidate space.
  3. The reported results show gains over AQLM at several budgets. At around 3 bpw, the five-task average improves from 63.27 to 67.16 on Llama3-8B and from 66.45 to 69.53 on Qwen3-14B. Both models also show lower WikiText-2 and C4 perplexity at these operating points.
  4. The paper includes actual inference measurements. It reports end-to-end throughput and peak memory on A100 and RTX 4090, as well as a single-layer MLP latency comparison with FP16.
Critical weaknesses
  1. The model coverage is limited. The experiments only evaluate Llama3-8B and Qwen3-14B. These two dense models do not provide enough evidence to extend the conclusions to MoE models, other architectures, or larger models.
  2. The A100 results do not establish a clear speed advantage. At the reported larger-batch operating points, AMPCQ achieves 123.5 tok/s, compared with 163.1 tok/s for AQLM, while using more peak memory. The batch sizes also differ, making the comparison harder to interpret. Table 6 does not clearly report the actual average bit widths of the evaluated models, particularly AMPCQ, so it is unclear whether their storage budgets are matched.
  3. The adapter may introduce substantial inference overhead. It adds its own codebooks, indices, and scales, which require extra storage, memory accesses, and decoding. Although the adapter codebooks are small, the decoded residual has the same dimensions as the original weight matrix. No low-rank constraint or fused computation strategy is described. If the backbone and adapter are separately dequantized and evaluated with dense matrix multiplications, the adapter adds another multiplication of the same dimensions. Table 7 only compares trainable-parameter storage and task performance, without measuring this runtime overhead.
  4. Stage 2 does not explicitly account for accumulated quantization errors from preceding blocks. It reconstructs each block using inputs collected from the full-precision model. During inference, inputs instead come from preceding quantized blocks and can have a different distribution. Good reconstruction on full-precision inputs therefore does not guarantee equally good reconstruction under the actual inference inputs.
  5. The experimental tables need to be checked. In Table 2, the five reported QuIP# 3-bit task scores average to 57.978, rather than the stated 63.70. In Table 3, I also find the unreasonable 3-bit degradation of AWQ and GPTQ on Qwen3-14B questionable, their WikiText-2 perplexities reach 1.26e7 and 5.68e3. The authors should verify the experimental configurations and explain these results. The current description is not sufficient to determine whether the collapse reflects the methods themselves or problems with their configuration or implementation.
Questions for authors
  1. At batch size 1, AMPCQ reaches 28.9 tok/s on RTX 4090 but only 20.1 tok/s on A100. Given A100's higher memory bandwidth, bandwidth alone does not explain this result. What limits AMPCQ's performance on each GPU? Please explain whether decoding, computation, kernel launch overhead, or other factors account for the difference.
  2. The five nominal quantization tiers and the final model-average bpw are easy to confuse. Could the authors explain their relationship more clearly, including how matrix sizes, codebook storage, and scales contribute to the reported average bit width?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 9Gmw

W1: We would like to clarify that this paper is a study of vector quantization for LLMs, rather than a systematic validation of generalization to MoE architectures or very large models. We validate the effectiveness of the proposed method on two representative dense Transformer architectures, Llama3-8B and Qwen3-14B. Due to the review timeline and computational constraints, we are unable to add MoE experiments in this revision. We will explicitly restrict our conclusions to the evaluated dense Transformers in the revised manuscript and identify coverage of MoE architectures and larger models as future work. We appreciate the reviewer for pointing out this important limitation and will make the scope of our conclusions clearer in the revision.

W2/Q1: We agree that the A100 results do not establish a general speed advantage, and the mismatched batch sizes and missing bit-width details make Table 6 difficult to interpret. We have addressed these issues as follows:

  1. Clarification on the possible reasons why A100 shows no advantage: The FP16 compute capability of the A100 is lower than that of the RTX 4090 (312 TFLOPS vs. 330 TFLOPS), and this compression algorithm is inherently reliant on higher raw compute power.

  2. Clarification on batch sizes and throughput: The original Table 6 was intended to report saturating throughput (i.e., running each method at its maximum feasible batch size to fully utilize the GPU). We reran the benchmarks under strictly identical settings. The fixed-batch results are summarized below:

GPU / ModelBatchtok/s ↑Peak Mem ↓
A100 (Batch=16)
AQLM-2bit16110.027.20 GiB
AMPCQ (Ours)1685.032.20 GiB
4090 (Batch=4)
AQLM-2bit4140.019.60 GiB
AMPCQ (Ours)4168.722.05 GiB

W3: Our implementation shares the backbone's fixed indices and learns codebook deltas (and scale deltas when enabled). It does not evaluate an independent residual matrix-multiplication branch. For a fixed task, we merge the deltas before inference:

Ci(t)=Ci+ΔCi(t),si(t)=si+Δsi(t).C_i^{(t)} = C_i + \Delta C_i^{(t)}, \qquad s_i^{(t)} = s_i + \Delta s_i^{(t)}.

The merged model uses the original indices, unchanged codebook shapes, and the same linear-operator structure. No additional adapter matrix multiplication is required. Thus, separately evaluating backbone and adapter branches is not our fixed-task deployment path. We will clarify this pre-merged implementation in Section 3.4 and the corresponding equations.

W4: Stage 1 profiles candidates using dense-model activations. In the sequential Stage 2 implementation, each recovered quantized block's outputs are propagated to the next block. The dense reference and quantized block receive the same current input:

Ll=∥Fl(h^l;Θl)−Fl(h^l;Θ^l)∥22,\mathcal{L}_l =\Vert F_l(\widehat{h}_l; \Theta_l) - F_l(\widehat{h}_l; \widehat{\Theta}_l) \Vert_2^2,

h^l+1=Fl(h^l;Θ^l).\widehat{h}_{l+1} =F_l(\widehat{h}_l; \widehat{\Theta}_l).

Thus, subsequent blocks receive inputs affected by the quantized prefix, rather than exclusively using original full-precision activations. We will clarify this distinction in Section 3.3. The objective remains local block reconstruction, not end-to-end optimization of the final model error.

W5: We thank the reviewer for catching the QuIP# inconsistency. The QuIP# results in our table were directly taken from the empirical study by Huang et al. (2024) [1], not from our own independent runs. We will correct the affected entries, clearly cite the source of each literature-reported number, and distinguish them from our own measured results. Regarding the Qwen3 baselines, the entries in question also come from Zheng et al. [2] rather than independent replication. We will correct any mislabeled entries and supplement these comparisons with independent AWQ and GPTQ evaluations under matched settings.

Q2: Tier labels denote index-only nominal rates. For KK codebooks, SS entries per codebook, and group size gout×ging_{\mathrm{out}} \times g_{\mathrm{in}}:

bnominal=Klog⁡2Sgoutgin.b_{\mathrm{nominal}} = \frac{K \log_2 S}{g_{\mathrm{out}} g_{\mathrm{in}}}.

Actual storage additionally includes codebooks and scales. The aggregate rate is total storage divided by the number of target weights, not an unweighted average of tier labels. We will distinguish nominal index rates, allocation costs excluding fixed scales, and complete quantized-matrix storage, and report the actual rates used in the efficiency comparison.

References

[1] Huang, W., Zheng, X., Ma, X., et al. An Empirical Study of LLaMA3 Quantization: From LLMs to MLLMs. arXiv:2404.14047, 2024.

[2] Zheng, X., et al. An Empirical Study of Qwen3 Quantization. arXiv:2505.02214, 2025.

Thank you for the response. Please include a detailed explanation of sharing the backbone indices and merging the codebook deltas in the paper, and provide the corrected QuIP# entries and experimental comparisons under the correct settings. I maintain my score.

Reviewer_6bbiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Mixed Precision Codebook Quantization, referred to as AMPCQ. The method distributes codebook capacity across the weight matrices of large language models under a shared average bit budget. Instead of using one codebook configuration throughout the model, it assigns each matrix one of five tiers. These tiers differ in codebook count, codebook size, and grouping. Stage 1 profiles candidate tiers using normalized activation aware reconstruction error. It removes dominated choices and selects upgrades according to their marginal improvement per additional bit. Stage 2 fits codebooks and indices using transformer block reconstruction. Stage 3 introduces optional task adapters expressed in codebook form.

The experiments on Llama3 8B and Qwen3 14B report meaningful improvements over uniform AQLM at similar average bit widths. On Llama3 8B, the three bit configuration reduces WikiText2 perplexity from 8.08 to 6.90 and raises mean zero shot accuracy from 63.27 to 67.16. On Qwen3 14B, the approximately three bit configuration improves mean accuracy from 66.45 to 69.53. The paper also includes comparisons of tier allocation, distortion metrics, alternative allocation methods, and inference performance. My main reservations concern the consistency of the stated bit budget, the additional value of the proposed allocation choices over simpler alternatives, and the interpretation of the performance comparisons.

Critical strengths
  1. The paper addresses a practical limitation of uniform codebook quantization. Different attention and MLP projections need not benefit equally from additional representation capacity. Allowing matrix specific choices of codebook structure and rate is a reasonable way to use a fixed storage budget more effectively.

  2. The empirical gains over AQLM are meaningful on both model families, especially around three average bits. Tables 4 and 5 also provide useful evidence for heterogeneous tier assignment and activation normalized reconstruction. Table 10 compares alternative allocation strategies under the same candidate tier space.

  3. The evaluation includes the effect of recovery after tier selection, additional results on Qwen3 14B, and actual inference measurements. The paper also appropriately acknowledges that greedy allocation across multiple matrices does not have a general optimality guarantee.

Critical weaknesses
  1. Appendix A.3 defines the rate for each matrix as its total storage in bits. However, Equations 5 and 28 multiply this quantity by a matrix weight and claim equivalence with the average bit width in Equation 27. This does not follow from the stated definitions. Since matched average bit width is central to the empirical claims, this inconsistency leaves the allocation constraint unclear. I think this is the most pressing issue to fix.

  2. The additional value of the proposed allocation rule and tier palette remains uncertain. Table 10 compares alternative strategies only before Stage 2 at one budget. Table 11 shows that the five tier space does not outperform a restricted two tier space on twelve MLP matrices after recovery. Therefore, the paper needs stronger full model evidence that these choices contribute independently to quality.

  3. Table 6 uses different larger batch sizes across methods. On the A100, AMPCQ reports 123.5 tokens per second at batch size 40. AQLM reports 163.1 at batch size 48. This makes the claimed general throughput advantage difficult to assess. Although AMPCQ is faster at batch size 1, it also requires more peak GPU memory. So the comparison should distinguish speed from memory cost.

  4. The experimental results are presented without variability over independent calibration draws and codebook fitting runs. This matters particularly at the lower bit budget on Llama3 8B. There, the average accuracy improvement over AQLM is 0.80 and C4 perplexity is slightly worse. This is despite the stronger and more consistent gains near three bits.

Questions for authors
  1. The storage definition in Appendix A.3 seems inconsistent with Equations 5 and 28. Could the authors clarify if Stage 1 uses total bits per matrix or bits per scalar weight? Also, how do matrix weights enter the actual constraint? Reporting the realized storage for the main comparisons (codebooks, indices, scales) would really help clear this up.

  2. Does the advantage over the alternative allocation strategies in Table 10 still hold after the same Stage 2 recovery at matched average bit widths? Also, does the five tier palette actually improve full model quality over the restricted two tier space? Right now, Table 11 only covers twelve MLP matrices, so it's hard to tell.

  3. For the inference results, could we get throughput and peak GPU memory comparisons against AQLM and QuIP# at identical batch sizes and generation lengths on each GPU? This would clarify the deployment benefit independently of differences in the maximum batch size supported by each compressed representation.

  4. I would love to see variation across independent calibration draws and quantization runs. This is particularly important at the lower bit budget on Llama3 8B, where the average accuracy advantage is small and C4 perplexity is slightly worse. Are these results consistent across runs?

3 comments

We thank the reviewers for the constructive feedback. Our point-by-point responses and revisions are below.

W1/Q1: We agree with the reviewer. App. A.3 defines RiR_i as total storage bits, but Eq. (5)/(28) implicitly treats it as a per-weight rate. We have unified the notation: RitotR_i^{\text{tot}} for total bits, ri=Ritot/Nir_i=R_i^{\text{tot}}/N_i for bits per weight, and Eq. (5) is now ∑iNiri(τi)≤B\sum_i N_i r_i(\tau_i)\le B. We audited the implementation and added the realized storage breakdown (codebook/index/scale) to App. A.3.

W2/Q2: To provide full-model evidence (beyond the 12 MLP matrices), we ran the full pipeline on all 224 linear matrices of Llama3-8B (no adapters).

  • Full-pipeline comparison (budget ≈ main table):

    MethodWiki2 PPL ↓WGPIQAHSARC-EARC-C
    Q-Palette-style + Stage 28.239468.5176.8852.7875.0442.83
    AMPCQ + Stage 28.2271.2777.2654.4376.0944.45

    AMPCQ retains higher accuracy on all 5 tasks with comparable PPL. Allocation benefits persist after recovery.

  • Full-model tier ablation (near 3.05 bits):

    PaletteAvg. bitsWiki2 PPL ↓WGPIQAHSARC-EARC-C
    Two tiers {2,4}3.05657.276170.7278.1357.0478.7047.78
    Five tiers~3.056.9073.0178.4558.0878.5747.70

    Five tiers yield lower PPL and higher accuracy on 3/5 tasks. (Both tables use raw % accuracy).

W3/Q3: We agree the comparison was unfair.

  1. Original Table 6 Intent: It measured saturating throughput (max feasible batch). We have relabeled it accordingly.

  2. Matched-Batch Comparison (same context/generation/CUDA Graph):

    • A100 (Batch=16): AQLM-2bit 110.0 tok/s (27.20 GiB) vs. AMPCQ 85.0 tok/s (32.20 GiB).
    • 4090 (Batch=4): AQLM-2bit 140.0 tok/s (19.60 GiB) vs. AMPCQ 168.7 tok/s (22.05 GiB).
    • (Batch=1 on A100: AMPCQ 20.1 tok/s vs. AQLM 18.3 tok/s).
  3. Speed-Memory Trade-off: At low batch, AMPCQ is faster but uses more memory. At high batch (A100), AQLM scales better due to its lower per-batch memory footprint (1.44 vs. 1.84 GiB), limiting our max batch to 40.

  4. Revisions: We now state AMPCQ is competitive in low-batch, latency-sensitive scenarios; high-batch throughput requires further kernel optimization. We no longer claim a general throughput advantage.

W4/Q4: We agree single runs are insufficient at extreme low bits. We are currently running 5 independent runs for both AQLM and AMPCQ on Llama3-8B, varying both calibration sampling and initialization seeds. The experimental design is finalized, and the runs are in progress.

1. Multi-Seed Results

We added two complete Stage 1–Stage 2 runs with five seeds. Accuracies are raw acc; standard deviations are sample standard deviations.

MetricPaper resultseed123seed456Mean ± SD
WikiText-2 PPL ↓8.22008.12898.14618.1650 ± 0.0484
WinoGrande ↑71.2771.6771.8271.59 ± 0.29
PIQA ↑77.2677.5376.0176.93 ± 0.81
HellaSwag ↑54.4353.8953.8554.06 ± 0.32
ARC-E ↑76.0975.3476.4375.95 ± 0.56
ARC-C ↑44.4542.2446.5944.43 ± 2.17
Five-task average ↑64.70*64.1364.9464.59 ± 0.41

The paper originally recorded the average as 64.89%; for consistent statistics, we use 64.70% recomputed from the five tasks. No additional seeds were run for C4. The two new runs are overall close to the paper results. Across three runs, WikiText-2 perplexity is 8.165 ± 0.048, and five-task average accuracy is 64.59 ± 0.41%. The maximum gap in average accuracy is about 0.81 percentage points, indicating generally stable performance; some individual tasks still fluctuate more, especially ARC-C.

2. Measured Single-Layer Optimization

The optimization is validated in a separate test script; the environment and official training code are unchanged. Rows below refer to different components within one layer, not the full Stage 2 pipeline.

Stage and test scopeOriginal timeOptimized timeSpeedup
Stage 1: reference-forward reuse, full-layer pipeline comparison46 min 54 s46 min 20 s1.01×
Stage 1: K-means initialization for all 28 candidates in a full layer44 min 52 s8 min 48 s5.10×
Stage 2: full single-layer recovery, 50 rounds, excluding outer-layer optimization and beam68 min 31 s50 min 00 s1.37×
Stage 2: one full-layer beam search, 7 matrices293.28 s202.22 s1.45×

Improvements:

  • Stage 1: fused nearest-center search and reused reference forward. Main gain is from K-means; post-iteration error is close. Outlier checks show the original also fluctuates, so we do not claim byte-identical codebooks.
  • Stage 2: prefetched training data. Overlapping data movement with computation cuts single-layer recovery time by about 27%; final validation loss changes from 9.031214e−69.031214e^{-6} to 9.031003e−69.031003e^{-6}, a relative difference of about 0.00230.0023%.
  • Stage 2: fused element-wise operations for beam candidate errors. Matrix multiplication, topk, and search order are unchanged; in this single-layer test, final indices and secondary errors are identical, and search time drops by about 31%.

I thank the authors for the detailed response and the additional experiments. The rebuttal shows some issues with the original contributions in the paper. In detail, the new results do not show a clear advantage at the lower bit budget. The 2 reruns give an average accuracy (64.59) that is close to the AQLM (64.09). The accuracy advantage over Q-Palette is also not consistent across the new seeds. I think addressing these issues would require major revision, including a corrected formulation of the allocation rule, verified main results, error bars with AQLM, and a narrower contribution regarding efficiency and consistency. Therefore, I will maintain my original rating.

Comments on the paper

1 comment
Area_Chair_QiR5OpenReview ↗

Action Required: Respond to Author Rebuttals Before the Discussion Period Ends

Dear Reviewers,

The authors have posted their rebuttals. As the author–reviewer discussion period is coming to a close soon, please read their responses and participate in the discussion.

All reviewers are expected to respond, at minimum with a brief acknowledgment confirming that the reviewers have read and considered the authors’ rebuttals. If your concerns have been addressed or remain unresolved, please briefly indicate this in your response.

Thank you for your time and effort.

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.