acceptodds
Under review as a conference paper at ICLR 2027

FidelityQ: Preserving Predictive Fidelity in Ultra-Low-Bit Language Model Quantization

Abstract

Ultra-low-bit scalar quantization offers a regular arithmetic structure for efficient deployment of large language models, but preserving their predictive behavior at extreme precision remains challenging. With only a few reconstruction levels, accurate weight representation must translate into consistent model predictions using a small set of calibration samples. We propose FidelityQ, a scalar quantization framework for preserving predictive fidelity at weight precisions down to 2 bits. Specifically, we introduce Information-Aligned Analytic Coding (IAC) for accurate quantization. IAC shapes each weight row toward a centered symmetric source through randomized Hadamard mixing, then fits a mirrored four-level minimum-distortion quantizer to its empirical weights. Every level set in this symmetry-aligned family decomposes exactly into 2 sign-valued bases with associated scales, combining source-adaptive reconstruction with lookup-free decoding compatible with scaled integer arithmetic. Our analysis further reveals that residual predictive discrepancies are heavy-tailed and concentrate on tokens that the teacher predicts confidently, motivating Knowledge-Intensive Calibration (KIC) for consistent quantization. KIC ranks sequences using the tail of token-wise teacher-student divergence and allocates a limited calibration budget to the highest-scoring samples. It then restores predictive agreement by updating only quantization levels and scales under teacher supervision, leaving the pretrained weights frozen. Experiments on the LLaMA and Qwen families demonstrate the effectiveness of FidelityQ across weight-only and joint quantization settings. On LLaMA-2-7B, FidelityQ achieves a WikiText2 perplexity of 6.62 at W2A16 and improves the prior state-of-the-art W2A4KV4 result from 8.31 to 6.96, while reducing linear-layer weight memory by 8.0× relative to FP16. These results support accurate LLM deployment with structured ultra-low-bit quantization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-Training Quantization, Large Language Models, Model Compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_krzjOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors aim to maximize inference speed while preserving accuracy when quantizing LLM weights to 2 bits. To this end, they propose two techniques. Information-Aligned Analytic Coding (IAC) performs 2-bit quantization without sacrificing decoding speed, which includes Hadamard rotation for reshaping the weight distribution followed by an iterative optimization of symmetric per-row quantization levels. Knowledge-Intensive Calibration (KIC) selects sequences with large token-wise KL divergence from the teacher and uses them for effective distillation. As shown in Figure 1, FidelityQ achieves the lowest WikiText2 perplexity among the compared methods. It also decodes about 3.4 times faster than QuIP#, the prior method with comparable perplexity.

Critical strengths

A practical method that achieves both accuracy and speed

  • FidelityQ achieves both accuracy and decoding speed with 2-bit scalar quantization. It outperforms existing per-row scalar methods across all models and settings in Table 1. Except on C4, it also surpasses the codebook-based AQLM, which decodes about two times slower.
  • Moreover, its decoding speed matches DB-LLM while being much more accurate and is faster than QuIP# and AQLM as reported in Table 8.

Thorough analysis that justifies each design choice

  • The paper supports the need for each component with analysis. It shows that rotation reshapes the weight distribution into a form that is easier to quantize (Fig. 3). It also shows that quantization error concentrates on a small fraction of tokens, which makes KIC effective (Fig. 4).
  • The ablations on level placement, sequence selection, and calibration data (Tables 5-7) are also comprehensive.
Critical weaknesses

Limited novelty over existing techniques

  • Most components of FidelityQ already exist in prior work: rotation in QuaRot, Lloyd-style non-uniform levels in SqueezeLLM and LeanQuant, the two-binary decomposition of 2-bit weights in DB-LLM, and teacher-guided calibration in BitDistiller.
  • KIC is ineffective without IAC in Table 2, and the gain of its new part, sequence selection, is small in Table 6.

Slow prefill limits practicality

  • Prefill reconstructs fp16 weights before computation, so it takes 7.32 ms/token, much slower than FP16 (0.27), QuIP# (1.12), and AQLM (1.98) as shown in Table 9.
  • Since the speed advantage is limited to batch-1 decoding, it is unclear whether FidelityQ remains advantageous for long prompts or batch serving.
Questions for authors

(1) Novelty. Could you clarify which parts of FidelityQ are new compared with existing techniques (e.g. QuaRot, DB-LLM, LeanQuant, BitDistiller), and how much each contributes to the performance?

(2) Contribution of KIC. In Table 6, KIC's sequence selection shows only a small gain over random selection. Could you explain why, and report the mean/variance over multiple random seeds?

(3) Latency. Given the slow prefill speed, could you report end-to-end latency across prompt lengths (e.g., 1K-8K tokens) to show whether FidelityQ remains effective for long prompts?

4 comments

We thank you for the positive assessment. We are encouraged that you found FidelityQ "a practical method that achieves both accuracy and speed" and found our analysis "thorough" in justifying "each design choice". Following your questions, we compared FidelityQ part by part with QuaRot [1], DB-LLM [2], LeanQuant [3], and BitDistiller [4] (Q1), separated KIC's magnitude optimization from sequence selection over three seeds (Q2), and measured end-to-end latency for prompts of 1K to 8K tokens and batch sizes up to 128 (Q3). Each part of FidelityQ adds a measurable gain over the closest prior method, selection brings a stable gain over random selection, and FidelityQ is faster than FP16 end to end at every tested prompt length. We address each point below.

Q1: Which parts of FidelityQ are new relative to QuaRot, DB-LLM, LeanQuant, and BitDistiller, and how much does each contribute?

A1: The comparison below shows that each part of FidelityQ serves predictive fidelity with a technique that differs from the named method, and that each part adds a measurable gain over its alternative in our pipeline. FidelityQ targets predictive fidelity under extreme compression, keeping the predictions of the full-precision model at 2.007 bits per weight with per-row scales and lookup-free decoding.

  • QuaRot rotates the model to remove outliers so that activations and the KV cache can be quantized; FidelityQ uses the same Hadamard rotation to bring every weight row to one Gaussian reference, under which plain squared error matches the layer loss (App. A).
  • LeanQuant learns a loss-aware grid of free levels decoded through a lookup table; IAC fits only the two magnitudes of the mirrored set that is optimal for this reference and stores them exactly as two sign planes, decoded without a table (Prop. 3).
  • DB-LLM splits 2-bit weights into two {0, 1} planes with a per-row offset, which places a level at zero; FidelityQ uses zero-free {±1} sign planes that match the mirrored optimum.
  • BitDistiller distills all weights with quantization-aware training; KIC keeps the pretrained weights and codes frozen and trains only the two amplitudes per row, on sequences selected from the tail of the teacher-student divergence.

To our knowledge, FidelityQ is the first to obtain the minimum-distortion member of the binary-coding family, with a structure determined by the source distribution, and to calibrate only the two per-row amplitudes that this member exposes, so that calibration leaves the decode unchanged.

Each part contributes measurably on LLaMA-2-7B at W2A16. Without rotation, 2-bit RTN diverges (Table 10), and with it plain squared error matches the layer loss to about 10⁻⁴ (App. A). Cumulatively, IAC and KIC lower Wiki2 from 9.56 for Base, which pairs QuaRot's rotation with a uniform grid, to 7.78 and then 7.11 (Table 2). Among lookup-free mirrored grids trained identically, IAC's placement is lower on C4 in every seed than the clipped members of the same family (Table R2 in our response to Reviewer tvS3), and without a lookup table it keeps the speed of uniform scalar quantization, comparable at small batch sizes and 21% faster at batch 128 (Table R7 in our response to Reviewer yBSp). Replacing the mirrored basis by DB-LLM's {0, 1} basis in our pipeline raises C4 at both W2A16 and W2A4KV4 (Table 5). KIC trains only the level magnitudes in half the time of EfficientQAT [5], which trains all weights, and leads it on all three metrics at the same per-row granularity (Table R3 in our response to Reviewer tvS3). We will state these differences and contributions in Sec. 2 and Sec. 4.3.

[1] QuaRot: Outlier-free 4-bit inference in rotated LLMs, NeurIPS, 2024.

[2] DB-LLM: Accurate dual-binarization for efficient LLMs, Findings of ACL, 2024.

[3] LeanQuant: Accurate and scalable large language model quantization with loss-error-aware grid, ICLR, 2025.

[4] BitDistiller: Unleashing the potential of sub-4-bit LLMs via self-distillation, ACL, 2024.

[5] EfficientQAT: Efficient quantization-aware training for large language models, ACL, 2025.

Q2: Why does KIC's sequence selection show a small gain over random selection in Table 6, and what are the mean and variance over several seeds?

A2: The gain in Table 6 is small on the corpus average because selection targets the high-divergence tail, and it grows with how much of that tail the selected set captures: across the four rules of Table 6, as the tail-KL mass of the selected set rises from 1.00× to 1.45× that of a random draw, Wiki2 falls monotonically from 17.05 to 16.69. On the worst 1% of held-out tokens, where selection aims, tail perplexity falls from 96.7 to 79.8 against a random draw of the same size (Fig. 4b). These tokens are a small share of the corpus, and selection acts on top of teacher-guided amplitude optimization, which already provides most of KIC's recovery (Table 2), so the whole-corpus increment is smaller than the local one. KIC's amplitude optimization trains the magnitudes of a mirrored grid, so it brings a similar gain on the MSE-clipped mirrored grid (Table R2 in our response to Reviewer tvS3), and Base's zero-including grid in Table 2 stays at its starting point by construction.

Table R4 gives the mean and standard deviation over three seeds per arm. Selection lowers the mean Wiki2 and C4 in every setting, and at W2A4KV4 on LLaMA-2-7B every selection seed beats every random seed on both metrics. On Qwen3-8B-Base, selection also raises the 7-task average and MMLU. We will report the contributions of amplitude optimization and selection separately in Sec. 4.3, and note in Table 2 that Base has no mirrored magnitudes for KIC to train.

Table R4. Tail-divergence selection against random selection, three seeds per arm, mean ± standard deviation (also in our response to Reviewer tvS3).

Model, W-A-KV, budgetRandom (Wiki2 / C4)Tail-divergence (Wiki2 / C4)
LLaMA-2-7B, 2-16-16, 10247.15 ± 0.02 / 9.72 ± 0.017.12 ± 0.02 / 9.70 ± 0.02
LLaMA-2-7B, 2-4-4, 10247.53 ± 0.01 / 10.12 ± 0.017.50 ± 0.01 / 10.11 ± 0.01
Qwen3-8B-Base, 2-4-4, 51210.75 ± 0.01 / 16.74 ± 0.0210.73 ± 0.05 / 16.70 ± 0.09

Q3: Does FidelityQ remain effective in end-to-end latency for prompts of 1K to 8K tokens?

A3: The end-to-end measurements in Table R8 show that FidelityQ remains effective for long prompts: on LLaMA-2-7B at batch size 1, it is faster than FP16 at every prompt length from 1K to 8K tokens and runs at nearly the speed of the uniform 2-bit grid. Each run processes the prompt and then generates 256 tokens with the full 32-layer model, online Hadamard transforms, and causal attention, with decoding in CUDA graphs on an A100-SXM4-80GB. The advantage holds because each layer's weights are reconstructed once per prefill forward, so the cost is amortized over all prompt tokens, while every decoding step benefits from the faster 2-bit kernel. The 7.32 ms per token of Table 9 comes from the HuggingFace eager-mode fallback with a 128-token prompt, where this reconstruction is spread over few tokens; the prefill here costs about 0.08 ms per prompt token at every tested length (Table R8). Batch serving, which you also raise, behaves the same way: FidelityQ runs at a speed comparable to the uniform 2-bit grid at small batch sizes and is 21% faster at batch 128 (Table R7 in our response to Reviewer yBSp). We will add Table R8 and its protocol to Sec. 4.5 and App. D.

Table R8. End-to-end latency on LLaMA-2-7B, A100-SXM4-80GB, batch size 1, 256 generated tokens.

PromptPrefill FP16 (ms)Prefill FidelityQ (ms)Prefill per prompt token, FP16 / FidelityQ (ms)E2E FP16 (s)E2E uniform W2 (s)E2E FidelityQ (s)Speedup
1K65860.063 / 0.0842.421.281.321.84×
2K1251570.061 / 0.0772.541.421.451.75×
4K2593100.063 / 0.0762.801.691.731.62×
8K5536450.068 / 0.0793.342.282.311.45×

Thank you for your detailed response. I think I would need to see a explicit technical contribution, or novelty beyond prior works to increase the score, but I didn't find in your answer. Therefore, I'll keep my rating at Weak Accept.

Dear Reviewer krzj,

Thank you for your timely reply and for your recognition of our work. Your comments have helped us improve the paper, and we sincerely appreciate your time and careful reading.

Best regards,

Authors

Reviewer_yBSpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FidelityQ, a post-training quantization framework for ultra-low-bit (2-bit) weight compression that aims to preserve predictive fidelity while retaining a regular and efficient arithmetic decoding structure. The framework comprises two key components:

IAC: A random Hadamard transform is applied to the weights, and each transformed row is fitted with four symmetric non-zero reconstruction levels, expressed as two signed basis vectors multiplied by a row-wise scale. This eliminates codebook lookup during decoding; each row requires only two signed reductions and an output scaling.

KIC: Candidate calibration sequences are selected according to the upper tail mean of token-level divergence between a full-precision teacher and a quantized student. The reconstruction amplitudes are then fine-tuned per row while the pretrained weights remain frozen, preserving the storage format and decoder arithmetic unchanged.

The method is evaluated across the LLaMA and Qwen families under W2A16KV16 and W2A4KV4 configurations, reporting perplexity and downstream task improvements along with GPU decoding latency and throughput. Overall, the work addresses a practically important problem and combines distribution-adaptive scalar reconstruction with prediction-guided calibration, offering meaningful engineering value.

Critical strengths

A practically relevant problem. 2-bit weight quantization is an extreme compression setting, and the paper demonstrates consistent gains over uniform-grid baselines across multiple model families and downstream tasks under a unified ablation protocol.

Attention to both accuracy and execution efficiency. The paper implements a fused decoding kernel, measures throughput, and proposes the dual-signed-basis representation to avoid per-element codebook lookups, showing genuine consideration of deployment constraints.

Critical weaknesses
  1. Insufficient isolation of methodological novelty. Random rotations, minimum-distortion scalar quantization, teacher–student matching, and data selection are all established techniques. The dual-signed-basis representation is essentially a reparameterization of chosen reconstruction levels; while useful for decoding, its independent contribution over the rest of the pipeline is not separately validated. The ablations quantify the combined effect of IAC and KIC but do not cleanly separate each component against a strong matched baseline. In particular, KIC alone yields limited gains, making it difficult to conclude that the design is superior to a simple composition of existing methods.

  2. Deployment claims are stronger than the evidence.Evaluations focus on batch size 1, and the reported kernel latency does not appear to include all transform overhead. Against the uniform-grid baseline, FidelityQ does not show a consistent latency advantage. The W2A4KV4 accuracy results are promising, but no correspondingly optimized execution kernel is provided; the authors acknowledge that fused support for joint quantization remains future work. Thus, end-to-end gains for joint quantization are still anticipatory.

Questions for authors
  1. Relative to a straightforward composition of random rotation, minimum-distortion quantization, and dual-signed-basis reparameterization, what is the genuinely new attribute of IAC? How can this be isolated experimentally?

  2. The theoretical quantizer assumes a Gaussian distribution. How closely do the actual post-transform weight distributions match this assumption across the evaluated models? Could the authors provide empirical distribution or distortion measurements supporting the theory-to-practice connection?

  3. Does the benefit of KIC come primarily from sequence selection, teacher-guided amplitude optimization, or their interaction? What are the results of random selection as a control, with fixed calibration budgets and initialization, across multiple models?

  4. Do the reported linear-layer latencies include the online Hadamard transform? If not, what is the additional overhead? After accounting for total cost, does the latency advantage over the uniform-grid baseline still hold at different batch sizes?

  5. Which of the reported W2A4KV4 benefits are empirically measured and which are expected? How sensitive are the results to calibration-data composition, budget size, and model scale?

  6. There is a numeric inconsistency that requires clarification: the abstract and Section 4.2 report LLaMA-2-7B W2A16 WikiText2 perplexity as 6.62, while Appendix Table 10 reports 6.86. Please clarify which settings correspond to these two numbers.

4 comments

Response to Reviewer yBSp (1/3)

We thank you for the constructive questions. We are encouraged that you found 2-bit weight quantization "a practically relevant problem" in which the paper "demonstrates consistent gains over uniform-grid baselines across multiple model families and downstream tasks", and that you recognized our "attention to both accuracy and execution efficiency" and "genuine consideration of deployment constraints". Following your questions, we isolated each link of IAC (Q1), measured the rotated weights of all five evaluated models against the Gaussian theory (Q2), split KIC into magnitude optimization and sequence selection (Q3), timed the linear stack with the Hadamard transforms included at batch sizes 1 to 128 (Q4), and checked the W2A4KV4 and Table 10 numbers (Q5, Q6). Each analysis supports the paper: IAC comes out ahead of the tested alternative at every link, the rotated rows match the Gaussian optimum, both parts of KIC bring gains, the latency advantage holds with the transforms included, and every reported W2A4KV4 accuracy number is measured. We address each point below.

Q1: What is genuinely new in IAC relative to composing random rotation, minimum-distortion quantization, and dual-signed-basis reparameterization, and how can it be isolated experimentally?

A1: Isolating each link on LLaMA-2-7B at W2A16 shows that IAC comes out ahead of the tested alternative at every link, which identifies its new attribute. IAC targets predictive fidelity under extreme compression, keeping the predictions of the full-precision model at 2.007 bits per weight with per-row scales and lookup-free decoding (Fig. 1a). The three operations were introduced elsewhere for different purposes: random rotation in QuaRot [1] suppresses outliers so that activations can be quantized, minimum-distortion quantization in SqueezeLLM [2] and LeanQuant [3] fits free levels decoded through a lookup table, and the dual-signed basis of binary-coding methods and DB-LLM [4] enables binary arithmetic. Pursuing predictive fidelity, IAC is to our knowledge the first to obtain from them the minimum-distortion member of the binary-coding family, with a structure determined by the source distribution. Table R5 sets each link against its alternative: rotation brings the gap between plain MSE and the layer loss down by two orders of magnitude, and without it 2-bit RTN diverges (Table 10), the zero-free mirrored family improves on the zero-including grid, IAC's levels improve on the clipped member of the same family under identical KIC training, and the {±1} basis improves on DB-LLM's {0, 1} basis. Together these links give the fidelity-efficiency tradeoff of Fig. 1a, where FidelityQ matches the Wiki2 of QuIP# [5] at a linear-stack latency close to that of the uniform grid and well below those of QuIP# and AQLM [6] (Table 8).

Table R5. Each link of IAC against its alternative on LLaMA-2-7B at W2A16; each row uses the setting of the experiment it cites.

LinkAlternativeIAC's choiceMetricAlternativeIACSource
Rotationno rotationHadamard rotationgap between plain MSE and layer loss≈10⁻²≈10⁻⁴App. A, Table 3
Grid familyzero-including uniform grid (Base)zero-free mirrored familyWiki2 / C49.03 / 16.747.92 / 13.00protocol of Table R2
Level placementMSE-clipped grid + KICIAC levels + KICC4 (tied / free)9.74 / 9.769.72Table R2
BasisDB-LLM {0, 1} + per-row offset{±1} sign planesC412.829.74Table 5

[1] QuaRot: Outlier-free 4-bit inference in rotated LLMs, NeurIPS, 2024.

[2] SqueezeLLM: Dense-and-sparse quantization, ICML, 2024.

[3] LeanQuant: Accurate and scalable large language model quantization with loss-error-aware grid, ICLR, 2025.

[4] DB-LLM: Accurate dual-binarization for efficient LLMs, Findings of ACL, 2024.

[5] QuIP#: Even better LLM quantization with Hadamard incoherence and lattice codebooks, ICML, 2024.

[6] Extreme compression of large language models via additive quantization, ICML, 2024.

Q2: How closely do the rotated weights of the evaluated models match the Gaussian assumption behind the quantizer?

A2: The measurements in Table R6 show that, after rotation, the weight rows of all five evaluated models match the Gaussian assumption closely in kurtosis, optimal distortion, and level locations. We measured 3024 rows per model, covering every projection type across nine evenly spaced layers. The measured optimal distortion and levels agree with the Gaussian theory to about the third decimal on every model, which provides the empirical basis for the mirrored IAC levels. We will add Table R6 to App. A.

Table R6. Rotated weight rows against the Gaussian theory, medians over 3024 rows per model.

QuantityL2-7BL2-13BL3-8BQwen3-8B-BaseL-30BGaussian
Excess kurtosis-0.008-0.0070.0060.004-0.0030
MMSE distortion0.11730.11730.11770.11760.11740.1175
m1/σ0.4520.4530.4510.4510.4520.453
m2/σ1.5091.5101.5101.5091.5091.510

Q3: Does KIC's benefit come from sequence selection, teacher-guided amplitude optimization, or their interaction, measured against random selection across models?

A3: KIC's benefit comes from both parts: teacher-guided amplitude optimization provides the large recovery, and sequence selection adds a consistent gain that grows with the damage it selects. Across the four selection rules of Table 6 on Qwen3-8B, as the tail-KL mass of the selected set rises from 1.00× to 1.45× that of a random draw, Wiki2 falls monotonically from 17.05 to 16.69. On IAC's levels at W2A16, amplitude optimization lowers C4 from 12.53 to 9.74 and improves every other metric (Table 2). KIC trains the magnitudes of a mirrored level set, so on Base's zero-including grid it has nothing to train and Base + KIC in Table 2 stays at its starting point by construction; on the MSE-clipped mirrored grid the same KIC lowers C4 from 13.00 to 9.76 (Table R2 in our response to Reviewer tvS3). Selection acts through this optimization, so we measure it with the optimization, calibration, and initialization fixed and only the training sequences changed. In Table R4 on LLaMA-2-7B at W2A4KV4, every selection seed beats every random seed on both Wiki2 and C4, and on Qwen3-8B-Base selection also raises the 7-task average and MMLU. The gain is concentrated where selection aims: on the worst 1% of held-out tokens, tail perplexity falls from 96.7 to 79.8 against a random draw of the same size (Fig. 4b). We will report the two contributions separately in Sec. 4.3.

Table R4. Tail-divergence selection against random selection, three seeds per arm, mean ± standard deviation (also in our response to Reviewer tvS3).

Model, W-A-KV, budgetRandom (Wiki2 / C4)Tail-divergence (Wiki2 / C4)
LLaMA-2-7B, 2-16-16, 10247.15 ± 0.02 / 9.72 ± 0.017.12 ± 0.02 / 9.70 ± 0.02
LLaMA-2-7B, 2-4-4, 10247.53 ± 0.01 / 10.12 ± 0.017.50 ± 0.01 / 10.11 ± 0.01
Qwen3-8B-Base, 2-4-4, 51210.75 ± 0.01 / 16.74 ± 0.0210.73 ± 0.05 / 16.70 ± 0.09

Q4: Do the linear-layer latencies include the online Hadamard transform, and does the advantage over the uniform grid hold across batch sizes?

A4: The measurements in Table R7 show that FidelityQ keeps its latency advantage once the online Hadamard transforms are counted: the transforms take a small share of the total time, and FidelityQ runs at a speed comparable to the uniform 2-bit grid at small batch sizes and becomes faster at large ones, 21% faster at batch 128. Table 8 reports linear kernels without the transforms. On the RTX A6000 the transforms add 1.11 ms per token, and the uniform grid, which uses the same rotation, pays the same cost. Table R7 times the full 32-layer linear stack of LLaMA-2-7B with transforms on an A100-80GB, using kernels written for this response and captured in CUDA graphs; both formats share the same kernel skeleton and differ only in the per-element decode. FidelityQ gains at large batch sizes because each sign plane decodes with one shift and one bitwise operation and the two amplitudes are applied once per row. We will add Table R7 to App. D.

Table R7. Latency (ms) of the 32-layer linear stack of LLaMA-2-7B with online Hadamard transforms, A100-80GB.

Batch sizeHadamardHadamard shareUniform + transformsFidelityQ + transformsFidelityQ / uniform
10.399%4.064.231.04
20.417%5.806.041.04
40.424%9.879.450.96
80.483%17.5718.471.05
160.593%17.6718.611.05
320.754%21.3021.301.00
641.134%32.7628.950.88
1281.905%50.8640.020.79

Q5: Which W2A4KV4 benefits are measured, and how sensitive are the results to calibration-data composition, budget size, and model scale?

A5: The results in Table 1, Tables R1a and R1b, and App. Table 7 show that every reported W2A4KV4 accuracy number is measured, and that FidelityQ's advantage generalizes across calibration data, budget, and model scale. At W2A4KV4 on LLaMA-2-7B, FidelityQ stays far ahead of Base both with the five-way pool and with C4 + FineWeb alone, and removing any single source of the pool barely affects the other domains (Tables R1a and R1b in our response to Reviewer tvS3). Doubling the level-training budget improves every tested corpus (App. Table 7). The W2A4KV4 gain over Base holds on every model of Table 1, from 7B to 30B, and Q2 shows that the rotated rows of all five models follow the same Gaussian theory. The 4-bit activation and KV quantization is simulated in these accuracy measurements, and the fused kernel and latency results are measured at W2A16; we will state this in the paper.

Q6: Which settings correspond to the LLaMA-2-7B W2A16 Wiki2 perplexity of 6.62 in the abstract and 6.86 in App. Table 10?

A6: There is no discrepancy in the FidelityQ result. The 6.86 in Table 10 belongs to the EfficientQAT [1] g64 row, while FidelityQ is 6.62, matching the abstract and Sec. 4.2. We will highlight the FidelityQ row in Table 10 so that the two cannot be confused.

[1] EfficientQAT: Efficient quantization-aware training for large language models, ACL, 2025.

Dear Reviewer yBSp,

We sincerely thank you for your time in reviewing our paper and for your valuable and constructive comments. In our response, we isolated each component of IAC against its alternative, measured how closely the rotated weights of all five models match the Gaussian reference, separated the contributions of sequence selection and amplitude training in KIC, and timed the online Hadamard transforms across batch sizes. The revised PDF adds these results, with all changes marked in red: the Gaussian comparison in App. A.4 (Tables 6 and 7), the paired selection results in Sec. 4.3 (Table 3), and the latency with online transforms in Sec. 4.4 and App. I (Tables 4 and 28). It also states in Sec. 4.1 that the W2A4KV4 accuracy is simulated, and shades the FidelityQ rows of Table 15.

Since the author-reviewer discussion phase will end very soon, we would appreciate it if you could kindly let us know whether these responses address your concerns. We are happy to answer any further questions.

Best regards, Authors

Reviewer_tvS3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies 2-bit scalar weight quantization of LLMs, both weight-only (W2A16) and jointly with 4-bit activations and KV cache (W2A4KV4). FidelityQ has two components. Information-Aligned Analytic Coding (IAC) applies a QuaRot-style randomized Hadamard transform. It then fits, for each output row, a mirrored zero-free four-level set {±m1,±m2}\{\pm m_1,\pm m_2\} by Lloyd–Max alternation, assigns codes with GPTAQ sequential error compensation, and rewrites the level set exactly as two sign planes with row-wise amplitudes a1,2=(m2±m1)/2a_{1,2}=(m_2\pm m_1)/2. A linear layer then becomes two sign-plane reductions with no codebook lookup. Knowledge-Intensive Calibration (KIC) scores candidate sequences by the mean of their top-25% token-wise teacher–student KL divergences and keeps the top-kk. It then trains only the two magnitudes per row under hard-label CE plus 0.1× soft-target CE on the teacher’s top-512 tokens, with signs and assignments frozen. Appendix A gives three results: a centering lemma for the rotation, an OBS-based identity bounding the gap between sensitivity-weighted and unweighted sequential residuals (measured on LLaMA-2-7B in Table 3), and a proof that the sign-amplitude form represents the Gaussian four-level MMSE quantizer exactly. Experiments cover LLaMA-2-7B/13B, LLaMA-3-8B, LLaMA-30B and Qwen3-8B-Base, with WikiText2/C4 perplexity and nine downstream tasks; on LLaMA-2-7B the method reports 6.62 (W2A16) and 6.96 (W2A4KV4) WikiText2 perplexity. Component, level-placement and selection ablations are included, along with a fused batch-1 kernel (5.11 ms linear stack; 47 tok/s eager-mode decode).

Critical strengths
  1. Large gains over per-row scalar baselines in a deployable, lookup-free format. Table 1 runs at 2.007 bits per weight (App. D.3). There, FidelityQ beats every scalar baseline listed for each model (the controlled Base with rotation + GPTAQ + uniform grid, QuaRot, ResComp, and per-row DB-LLM), in both W2A16 and W2A4KV4. For example, LLaMA-2-7B W2A4KV4 reaches 6.96/9.97 Wiki2/C4 against 11.27/21.41 for Base, and a 7-task average of 57.25 against 45.01. LLaMA-30B W2A4KV4 reaches 5.71 Wiki2. Even without any gradient training, Base + IAC (7.78 on LLaMA-2-7B W2A16, Table 2) is below every scalar baseline in Table 1. The format has a working fused kernel: in Table 8 the linear stack takes 5.11 ms at W2A16, against 8.83 ms for QuIP# and 9.93 ms for AQLM 2×8 at the same storage. For deployments that need plain scalar arithmetic, this is a practically useful operating point.

  2. Correct analysis and an informative diagnostic for rotated pipelines. The results in Appendix A are elementary but correct. Prop. 2(i)–(iii) gives an exact identity linking GPTQ-style layer loss to unweighted sequential residuals, and Table 3 measures it on LLaMA-2-7B. After rotation, the coefficient of variation of the sequential sensitivities falls from 0.21–0.79 to 0.007–0.017, and the weighted/unweighted residual gap falls to about 10−410^{-4}. This is a clean, reusable justification for fitting plain-MSE levels after Hadamard rotation. The authors also state the calibration-budget mismatch with the PTQ baselines openly (App. B.1).

Critical weaknesses
  1. The headline accuracy claims are confounded by calibration data and training budget, and FidelityQ trails trained baselines on C4 and downstream tasks.

What: Each Table 1 entry comes from one KD-trained checkpoint, calibrated on 4096 sequences drawn from a “mixed-domain pool” (Sec. 4.1, App. B.2). I could not find the pool’s sources or size in Sec. 4.1, App. B.1–B.2, C.3 or E. The ablations in Tables 2 and 5 calibrate on WikiText2. By contrast, QuaRot/GPTQ use 128 sequences and no gradient training (App. B.1). Evidence of domain-specific fitting: On LLaMA-2-7B, FidelityQ’s Wiki2/C4 ratio is 6.62/9.53 = 0.69. FP16 gives 0.79, and every other trained method in Table 10 sits near 0.80 (QuIP# 6.66/8.35, AQLM 2×8 7.24/8.96, EfficientQAT g64 6.86/8.51). The same holds on LLaMA-2-13B (0.70 vs. 0.75 for FP16) and LLaMA-3-8B (0.64 vs. 0.69). Also suggestive: moving from 512 WikiText2 sequences (Table 2) to the 4096-sequence pool cuts Wiki2 by 0.49 but C4 by only 0.21. The zero-anchored variant in Table 5 shows the extreme case: 6.87 Wiki2 but 12.82 C4. On metrics less tied to the calibration domain, FidelityQ is not the strongest: LLaMA-2-7B: C4 is 9.53 against 8.90/8.51 for the scalar EfficientQAT g128/g64, and the 7-task average is 59.63 against 60.26/60.80 (Table 10). LLaMA-2-13B: C4 is 8.43 and the 7-task average 63.97, against 7.27 and 67.23 for AQLM 1×16 (Table 11). LLaMA-3-8B (appendix only): 9.63/15.08 Wiki2/C4, against 9.40/12.77 for EfficientQAT g64 and 7.76/10.89 for AQLM (Table 12). Table 1 uses AQLM 2×8 rather than 1×16 as the codebook reference, and on 13B, 1×16 (5.59 Wiki2) beats FidelityQ (5.86).

EfficientQAT uses group-wise scales and so more bits, so it is not a matched comparison either. That is exactly why a matched per-row trained baseline is needed before claiming that FidelityQ closes the gap to codebook methods (Fig. 1a; Sec. 6).

W2A4KV4: The abstract’s claim (8.31 → 6.96 against RCP) compares the authors’ run with a quoted number. FidelityQ’s activation and KV quantizer (granularity, symmetry, clipping) is never specified (Sec. 4.1, Alg. 1 line 7, App. B), so there is no way to check that the protocols match. Why it matters: The central claim is preserved predictive fidelity, but the strongest evidence for it (WikiText2) is also the metric most exposed to the unspecified calibration pool. What would resolve it: (i) Specify the pool. (ii) Report the LLaMA-2-7B headline results with a pool that excludes WikiText2 (for example C4/FineWeb, as in Tables 6–7). (iii) Add a per-row trained scalar baseline with the same data and step budget (see Q2). (iv) Give the A4/KV4 settings.

  1. The parts of the method that are new account for little of the measured gain; the core pieces are established.

Representation: a1s1+a2s2a_1 s_1 + a_2 s_2 with si∈{±1}s_i\in\{\pm1\} and row-wise scales, fitted by alternating code/scale updates, is 2-bit binary-coding quantization. This appears in Xu et al. (ICLR 2018, alternating multi-bit quantization) and LQ-Nets (Zhang et al., ECCV 2018), and for LLMs in LUT-GEMM (Park et al., ICLR 2024) and ShiftAddLLM (You et al., NeurIPS 2024), both with lookup-free or multiplication-free kernels for this form. None of these is cited, and Prop. 3(i) restates their representable set. The symmetric uniform 2-bit grid {±Δ/2,±3Δ/2}\{\pm\Delta/2,\pm3\Delta/2\} is itself in the family (m2=3m1m_2=3m_1). The {0,1} basis of DB-LLM differs only by a per-row offset (Remark 5). Level placement: For a Gaussian, the four-level Lloyd–Max quantizer has distortion 0.1175σ20.1175\sigma^2; the MSE-optimally clipped uniform grid has 0.1188σ20.1188\sigma^2 (Max, 1960; I recomputed both). That is a gap of about 1%. Table 5 agrees: clipped uniform gives 7.89/12.87 against 7.78/12.53 for IAC. So nearly all of the Base→IAC gain (9.56→7.78), and the 4.1→9.6 dB SQNR jump in Fig. 3a, comes from choosing the range. It does not come from non-uniform placement or the maximum-entropy argument of Eq. (2). Selection: Fig. 4b shows KIC-selected and random calibration sets within 0.08% of each other on the whole corpus; the gain appears only on the worst-1% slice. Table 6 shows 17.05→16.69 Wiki2 and 24.46→24.37 C4, on Qwen3-8B (not the Base model in Table 1) with a single random seed and no variance. The training half of KIC (scale-only distillation with frozen codes) closely follows PEQA (Kim et al., NeurIPS 2023) and EfficientQAT’s E2E-QP (which the paper cites). Why it matters: The abstract and Sec. 3.2–3.3 credit predictive fidelity to minimum-distortion coding and divergence-targeted selection. The evidence instead points to range fitting, scale-only distillation and an 8× larger calibration set (Table 2 vs. Table 1). What would resolve it: (i) Position the method against binary-coding quantization and scale-only QAT. (ii) Report “IAC + level training on random sequences” and “clipped uniform + the same training” at the Table 1 budget, with several seeds.

Questions for authors
  1. What is the “mixed-domain pool” behind Table 1? Please give its sources and proportions, its total size, and whether it includes WikiText2 train or other Wikipedia text. Please also report LLaMA-2-7B W2A16/W2A4KV4 with a pool that excludes WikiText2. If the Wiki2 advantage holds and the C4 gap to QuIP#/EfficientQAT narrows, I would consider Weakness 1 largely resolved.
  2. Can you add a per-row trained scalar control with identical data and steps? That is: rotation + GPTAQ + an MSE-clipped uniform grid (i.e., m2=3m1m_2=3m_1), followed by the same scale-only KD, with the ratio either tied or free. Alternatively, EfficientQAT/PEQA-style step-size training at per-channel granularity on the same data. If IAC + KIC beats this clearly on C4 and the 7-task average, the level-placement part of Weakness 2 would be resolved.
  3. In Table 2, Base + KIC leaves perplexity unchanged (9.56→9.56 Wiki2, 18.43→18.42 C4). Yet Base’s group-max levels {±max⁡/3,±max⁡}\{\pm\max/3,\pm\max\} (Fig. 3a) lie in the same {±m1,±m2}\{\pm m_1,\pm m_2\} family that KIC trains. Which parameters are trainable for Base, and do the optimizer and step count match? If something is frozen or mis-specified here, Table 2 would read differently.
  4. How large is the seed variance? Please give mean ± std over several random-selection seeds and calibration runs for Table 6. Ideally, also compare full FidelityQ with IAC + random-selection training on LLaMA-2-7B at the Table 1 budget. If tail-damage selection beats random by more than the seed variance on whole-corpus metrics, the selection part of Weakness 2 would be resolved.
  5. How are activations and the KV cache quantized at W2A4KV4: per-token or per-group, symmetric or asymmetric, with what clipping, and at which linear inputs? Do these settings, and the calibration/training budget, match those behind RCP’s 8.31? If they do, the W2A4KV4 headline is a fair comparison.
7 comments

We thank you for the careful and quantitative review. We are encouraged that you found "large gains over per-row scalar baselines in a deployable, lookup-free format" and "a practically useful operating point", and that you found the analysis of Appendix A "a clean, reusable justification for fitting plain-MSE levels after Hadamard rotation". Following your review, we specified the calibration pool and reran the headline results without WikiText2 (Q1, Q3), added two per-row trained scalar baselines with the same data and budget (Q1, Q4), positioned FidelityQ against binary-coding quantization and scale-only QAT (Q2), compared tail-divergence selection with random selection over three seeds (Q2, Q6), explained the Base + KIC row (Q5), and specified the W2A4KV4 protocol (Q7). Each comparison supports the corresponding claim of the paper. Unless otherwise stated, the new experiments are on LLaMA-2-7B at the Table 1 budget, with 128 WikiText2 sequences for GPTAQ [1] and 1024 C4 + FineWeb sequences for level training, so their absolute values differ from Table 1 and from the 512-sequence ablations of Table 2; all rows within each new comparison share one setting. We address each point below.

Q1 (Weakness 1): Are the headline results confounded by calibration data and training budget, and does FidelityQ trail trained baselines on C4 and downstream tasks? We answer the five points of the weakness in turn.

Q1-1: The mixed-domain pool behind Table 1 is not specified, and the ablations of Tables 2 and 5 calibrate on WikiText2.

A1-1: We specify the pool below, and Q1-2 shows that FidelityQ's accuracy holds when WikiText2 is removed from calibration. The pool draws from the training splits of MMLU auxiliary_train (25%) and of SlimPajama, WikiText2, C4, and FineWeb (18.75% each), and no validation or test text of any evaluated benchmark enters calibration (Q3 gives its size). We will add the pool to App. B.

Q1-2: The low Wiki2/C4 ratio of FidelityQ, the larger Wiki2 than C4 gain from 512 to 4096 sequences, and the zero-anchored row of Table 5 suggest domain-specific fitting to WikiText2.

A1-2: The results in Tables R1a and R1b show that FidelityQ's accuracy does not come from fitting WikiText2. With WikiText2 removed from both calibration sets, FidelityQ still leads every scalar baseline of Table 1 on Wiki2, C4, and Avg-7 in both settings, although those baselines are calibrated on WikiText2, and its C4 improves (Table R1a). Removing any other single source leaves the remaining domains nearly unchanged, so each source adds coverage of its own domain (Table R1b). The zero-anchored row of Table 5 shares IAC's calibration, so its higher C4 comes from the zero level of the {0, 1} basis. A larger calibration budget lowers both Wiki2 and C4, in line with the budget trend of App. Table 7. We will add Tables R1a and R1b to App. B.

Table R1a. FidelityQ without WikiText2 in calibration against the scalar baselines of Table 1, which are calibrated on WikiText2 (LLaMA-2-7B).

MethodCalibrationW2A16 Wiki2W2A16 C4W2A16 Avg-7W2A4KV4 Wiki2W2A4KV4 C4W2A4KV4 Avg-7
QuaRot [2]WikiText224.5239.3942.4547.2377.4939.06
ResComp [3]WikiText210.2819.6847.7012.7823.9144.81
DB-LLM [4]WikiText28.8919.8547.80———
BaseWikiText29.5618.4348.0011.2721.4145.01
FidelityQfive-way pool6.629.5359.636.969.9757.25
FidelityQC4 + FineWeb only7.969.4159.418.419.8357.02

Table R1b. Leave-one-source-out on the five-way pool, LLaMA-2-7B, W2A16.

PoolWiki2C4Avg-7
Five-way pool, two runs6.62 / 6.609.53 / 9.5159.63 / 59.76
minus WikiText27.879.5058.49
minus SlimPajama6.579.5559.78
minus FineWeb6.609.5559.65

[1] GPTAQ: Efficient finetuning-free quantization for asymmetric calibration, ICML, 2025.

[2] QuaRot: Outlier-free 4-bit inference in rotated LLMs, NeurIPS, 2024.

[3] Rethinking residual errors in compensation-based LLM quantization, ICLR, 2026.

[4] DB-LLM: Accurate dual-binarization for efficient LLMs, Findings of ACL, 2024.

Q1-3: On metrics less tied to the calibration domain, FidelityQ trails EfficientQAT [1] g128/g64 and AQLM [2], and Table 1 takes AQLM 2×8 as the codebook reference, leaving out 1×16, which is stronger on LLaMA-2-13B.

A1-3: The comparison in Table R3 shows that FidelityQ's gap to EfficientQAT g128/g64 comes from group-wise storage, and that at the same per-row storage FidelityQ leads EfficientQAT on all three metrics. EfficientQAT g128/g64 stores a scale and zero point for every 128 or 64 weights and trains all weights, which also holds for its g64 reference on LLaMA-3-8B; run with one scale per row on the same data, it falls well behind FidelityQ. Table 1 uses AQLM 2×8 because it is the AQLM configuration at FidelityQ's storage (2.006 against 2.007 bits per weight) and the comparison point of Fig. 1a. The 1×16 configuration stores a 65,536-entry codebook per layer, which places it at a larger and slower operating point: 1.73 GB of linear weights and 23.2 tokens/s in decoding, against 1.51 GB and 47.0 tokens/s for FidelityQ (Table 9). Table 11 reports 1×16 alongside 2×8 on LLaMA-2-13B, and we will state these operating points in Sec. 4.2.

Table R3. EfficientQAT rerun on the same five-way pool sequences as FidelityQ, W2A16.

MethodGranularityBitsTrained parametersA100 hoursWiki2C4Avg-7
EfficientQATg1282.14all weights + scales—6.629.4259.73
EfficientQATper row2.0046.7B weights + 1.4M scales7.210.7015.1241.64
FidelityQper row2.0072.7M level magnitudes3.66.629.5359.63

Q1-4: A per-row trained scalar baseline with the same data and step budget is needed before claiming that FidelityQ closes the gap to codebook methods.

A1-4: The results in Tables R2 and R3 show that FidelityQ comes out ahead of both per-row trained scalar baselines under the same data and training budget. The first is the control you propose in Q4, the MSE-clipped uniform grid trained with the same KIC recipe, data, and seeds, with the level ratio tied to m2 = 3m1 or free; IAC + KIC is lower on C4 than both in every seed and stays ahead at W2A4KV4 (Table R2). The second is EfficientQAT at per-row granularity, which FidelityQ leads on all three metrics with half its training time (Table R3). We will add both baselines to the paper.

Table R2. Matched per-row trained controls on LLaMA-2-7B, three seeds, same KIC recipe, data, and seeds for every trained row; C4 range is the minimum to maximum over seeds, and distortion is that of the starting grid on a rotated Gaussian row.

ConfigurationDistortionW2A16 Wiki2W2A16 C4 rangeW2A16 Avg-7W2A4KV4 Wiki2W2A4KV4 C4 rangeW2A4KV4 Avg-7
Base, zero-including0.1519.0316.7449.67———
Clipped uniform + KIC, ratio tied0.11877.14 ± 0.019.734–9.74957.40 ± 0.24———
Clipped uniform + KIC, ratio free0.11877.22 ± 0.059.743–9.79158.01 ± 0.527.56 ± 0.0210.160–10.16755.85 ± 1.40
IAC + KIC0.11737.15 ± 0.029.715–9.73358.00 ± 0.167.53 ± 0.0110.119–10.13055.92 ± 0.66

Q1-5: The W2A4KV4 comparison with RCP's 8.31 lacks the activation and KV quantizer settings needed to check that the protocols match.

A1-5: The A4/KV4 settings below show that FidelityQ's W2A4KV4 results use a standard quantizer whose protocol can be checked against RCP's. FidelityQ quantizes the inputs of all linear layers per token and asymmetrically with clip 0.9, and the KV cache per token across heads, also asymmetrically, with clip 0.95, with per-row weights at 2.007 bits. These are the settings of the W2A4KV4 runs in Table 1. Switching them to RCP's A/KV settings changes Wiki2 and C4 by less than 0.01, so FidelityQ stays ahead of RCP [3] under RCP's own protocol (Q7). We will specify these settings in Sec. 4.1, Alg. 1, and App. B.

[1] EfficientQAT: Efficient quantization-aware training for large language models, ACL, 2025.

[2] Extreme compression of large language models via additive quantization, ICML, 2024.

[3] Rotate, clip, and partition: Towards W2A4KV4 quantization by integrating rotation and learnable non-uniform quantizer, Findings of EMNLP, 2025.

Q2 (Weakness 2): Do the new parts of the method account for little of the measured gain? We answer the two resolutions you proposed in turn.

Q2-1: Position the method against binary-coding quantization (Xu et al. [1], LQ-Nets [2], LUT-GEMM [3], ShiftAddLLM [4]) and scale-only QAT (PEQA [5], the E2E-QP stage of EfficientQAT [6]).

A2-1: FidelityQ is new in both families. To our knowledge, IAC is the first binary-coding quantizer whose basis is derived from the source distribution: rotation makes the minimum-distortion 2-bit code of every row analytic, and Prop. 3 shows that two sign planes store this code exactly, so FidelityQ obtains its levels without search and decodes them without a lookup table. KIC is the scale-only training step that starts from these source-derived levels and trains only their 2.7M amplitudes against the output divergence of the full-precision teacher on the sequences where the quantization error concentrates. We thank you for these references and will cite Xu et al. (2018), LQ-Nets, LUT-GEMM, ShiftAddLLM, and PEQA in Sec. 2 and relate Prop. 3 to binary-coding quantization. These works share the sign basis Σ α_i b_i with FidelityQ and fit it numerically to weights of arbitrary distribution:

  • Xu et al. (2018) solve for the binary codes and scales by alternating minimization at every training step of recurrent language models; FidelityQ makes every row near-Gaussian by rotation (Reviewer yBSp, Q2), takes the mirrored minimum-distortion levels ±0.453σ and ±1.510σ for this source (Sec. 3.2), and fits each row once with the pretrained weights unchanged.
  • LQ-Nets learn the basis jointly with all weights of a CNN in full quantization-aware training; in FidelityQ the optimum is analytic, Prop. 3 stores every mirrored set in two sign planes without approximation error, and training is post hoc and limited to the level magnitudes.
  • LUT-GEMM decodes group-scaled binary-coded weights through lookup tables of partial sums for weight-only inference; FidelityQ decodes with two scaled sign-plane dot products, keeps one pair of amplitudes per row at 2.007 bits, and runs the same code at W2A4KV4.
  • ShiftAddLLM fits a group-scaled power-of-two shift-add form by minimizing weight and output-activation error; FidelityQ targets predictive fidelity, since after rotation plain MSE matches the layer loss to about 10^-4 (App. A) and KIC trains the two amplitudes per row against the output divergence of the full-precision teacher.

Inside our pipeline, IAC comes out ahead of the other members of the family that we tested, the symmetric uniform grid (Table R2) and DB-LLM's [7] zero-including {0, 1} basis (Table 5).

PEQA and the E2E-QP stage of EfficientQAT share KIC's frozen-code, scale-only training:

  • PEQA quantizes the weights to a uniform grid by round-to-nearest and fine-tunes the per-channel scales on downstream task data; KIC starts from IAC's mirrored minimum-distortion levels after GPTAQ [8] and restores agreement with the full-precision teacher on calibration data, so one quantized model serves all tasks.
  • E2E-QP trains the step sizes end to end with the next-token loss after the Block-AP stage of EfficientQAT has trained all weights block by block; KIC keeps the pretrained weights frozen, trains only the level magnitudes against the teacher's output divergence, and draws its sequences by tail-divergence selection (Table R4).

At the same per-row granularity and data, KIC leads EfficientQAT on Wiki2, C4, and Avg-7 with half the training time (Table R3). We will add scale-only QAT to Sec. 2 alongside binary-coding quantization.

[1] Alternating multi-bit quantization for recurrent neural networks, ICLR, 2018.

[2] LQ-Nets: Learned quantization for highly accurate and compact deep neural networks, ECCV, 2018.

[3] LUT-GEMM: Quantized matrix multiplication based on LUTs for efficient inference in large-scale generative language models, ICLR, 2024.

[4] ShiftAddLLM: Accelerating pretrained LLMs via post-training multiplication-less reparameterization, NeurIPS, 2024.

[5] Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization, NeurIPS, 2023.

[6] EfficientQAT: Efficient quantization-aware training for large language models, ACL, 2025.

[7] DB-LLM: Accurate dual-binarization for efficient LLMs, Findings of ACL, 2024.

[8] GPTAQ: Efficient finetuning-free quantization for asymmetric calibration, ICML, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.