acceptodds
Under review as a conference paper at ICLR 2027

LRSR: Hardware-Friendly Approximation of Per-Element Quantization via Low-Rank Scale Reconstruction

Abstract

Post-training quantization is a crucial component for deploying Large Language Models (LLMs) in resource-constrained environments. While fine-grained methods (e.g., per-group) offer high accuracy, they heavily rely on hardware-specific implementations and suffer from memory-bound computations, limiting their applicability on standard matrix multiplication accelerators. Conversely, hardware-friendly quantizations (e.g., per-channel) experience severe accuracy degradation at low bit-widths. To address these limitations, we propose Low-Rank Scale Reconstruction (LRSR), a novel quantization method that approximates per-element scale matrices using an upper-bounding rank-1 decomposition, enabling mathematically correct reconstruction via general matrix multiplication (GEMM). Furthermore, we introduce several synergetic clusterization techniques to significantly boost accuracy and allow optimal hardware load balancing. Experimental evaluations on Qwen3 models demonstrate that at INT4, LRSR reduces KL-Divergence by up to 31% compared to the per-channel baseline, while utilizing 29 less metadata memory compared to per-group (g=128) quantization. This establishes a new Pareto frontier for near-zero overhead, hardware-friendly and calibration-free post-training quantizations.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
large language models, post-training quantizations, model compressions, hardware-friendly computations, low-rank approximations

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_igi6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

To address the memory overhead and quantization accuracy challenges in post-training weight quantization (PTQ) for large language models, this study proposes Low-Rank Scale Reconstruction (LRSR) without requiring calibration data. The core of the method is an upper-bounding rank-1 decomposition that approximates the ideal per-element scale matrix via a low-rank matrix. While guaranteeing no weight clipping, it pre-multiplies the scaling vector into the activation matrix parameters, thereby completing the linear layer computation through a single GEMM operation and lightweight vector scalings, which greatly saves the metadata memory overhead of the scale factors. To mitigate the approximation error introduced by the upper-bounding rank-1 decomposition, the study further introduces 1DOS and K-Means clustering strategies aimed at improving reconstruction accuracy while accounting for hardware load balancing. Evaluations on the Qwen3-4B model (WikiText-2 dataset) demonstrate that at INT4 precision, this method achieves memory footprint and inference latency close to the per-channel baseline while narrowing the generation quality gap (e.g., KL-Divergence, PPL) with the per-group method.

Critical strengths

S1. Practical engineering significance. The paper targets the pain point that fine-grained quantization (per-group) relies on custom low-level kernels in LLM deployment, attempting to balance the storage overhead of scale factor metadata and quantization accuracy under a calibration-free setting, which lowers the barrier to using the quantization pipeline S2. Intuitive formulation and low metadata overhead. Starting from the algebraic properties of the matrix multiplication reduction axis, the method decouples the scale factors via a rank-1 decomposition. The method reduces the metadata storage complexity to O(N+K)O(N+K) without altering the fundamental GEMM operation. S3. Clear theoretical derivation. The derivation of Algorithm 1 is developed under a strict no-clipping premise, and Proposition 1 proves that under round-to-nearest conditions, the elementwise quantization error upper bound of the proposed method is no worse than the per-channel baseline. Figure 1 provides an intuitive visualization of the scale factor envelope, making it easy to understand.

Critical weaknesses

W1. Imprecise positioning of research novelty. For Eq. 6, applying per-channel quantization with scale uiu_i to the input-channel rescaled weights Wi,k/vkW_{i,k}/v_k yields the exact same quantized integer weights (W^i,k\hat{W}_{i,k}), which is algebraically equivalent to bilateral diagonal scaling. Thus, this algebraic property alone is insufficient to serve as one of the core contributions. The actual novelty is limited to the calibration-free upper-bounding scale construction (Algorithm 1) and the subsequently introduced clustering strategies (Section 3.3). However, the paper currently provides insufficient justification for the benefits of these two components compared to existing methods. Moreover, locking row maxima cic_i before taking column maxima vkv_k in Algorithm 1 is overly sensitive to single-point outliers, causing severe degeneration (e.g., Fig. 15 & 17) W2. Operator type mismatch in experimental validation. While Section 1 motivates standard GEMM(W AW\ A in Eq.6) via NPUs lacking custom kernels, Section 4 are conducted entirely under the Weight-INT4, Activation-BF16 setting, applying INT4 quantization only to WW. In practice, any standard GEMM hardware requires both operands to have homogeneous data types. Executing a multiplication between INT4 and BF16 requires either dequantizing INT4 to BF16 beforehand or introducing mixed-precision custom kernels. The former fails to achieve compute acceleration, while the latter violates the premise of "not requiring custom kernels". W3. Thin experimental evaluation. The paper lacks comparisons with other methods. For instance, Section 2.3 discusses methods such as QuaRot, SplitQuant, and AWQ, yet Section 4 and Appendix C only compare against the most basic round-to-nearest per-channel and per-group baselines. The paper neither provides comparisons of quantization accuracy and latency against other calibration-free methods (e.g., QuaRot, SplitQuant), nor includes the quantization accuracy of calibrated methods (e.g., AWQ) as a reference baseline, making it difficult to accurately assess the accuracy cost incurred for being calibration-free. Furthermore, experiments are only conducted on the WikiText-2 dataset and a few small-scale models (1.7B/4B), lacking validation across more datasets and larger model scales.

Questions for authors

Q1. In Transformer architectures, for linear projections immediately preceded by LayerNorm or RMSNorm (e.g., q_proj / k_proj / v_proj), one could in principle follow AWQ's approach to absorb the scaling vector vkv_k offline into the preceding normalization weight parameters, thereby eliminating the runtime activation scaling overhead. Please explain whether the method proposed in this paper can perform a similar offline operator fusion instead of dynamically executing the multiplication of vkAk,jv_k A_{k,j} at runtime. Q2. Regarding W2, please provide empirical accuracy results when WW and AA share the same data types (e.g., W4A4 or W8A8), or otherwise revise the hardware acceleration arguments in the paper. Q3. Regarding W3, please supplement the paper with empirical comparisons in both accuracy and latency against the calibration-free methods mentioned (e.g., QuaRot, SplitQuant), as well as accuracy comparisons against calibration-based methods (e.g., AWQ). Additionally, it is recommended to include evaluations on more standard downstream reasoning tasks (e.g., GSM8K) or larger-scale models (e.g., 7B).

1 comment

We thank the Reviewer igi6 for the detailed technical feedback regarding bilateral diagonal scaling, offline operator fusion, data type homogeneity, and experimental baselines.


Critical Weakness W1: Imprecise positioning of research novelty. For Eq. 6, applying per-channel...

We agree with this assessment and have revised the contribution statements in Section 1 (lines 054–065) so that the algebraic factorability of bilateral scaling is no longer claimed as a standalone contribution. Instead, the contribution is framed around the calibration-free upper-bounding decomposition (Algorithm 1) and the clusterization strategies (Section 3.3).

Regarding the sensitivity of Algorithm 1 to single-point outliers: locking the row maxima uiu_i in Step 1 indeed causes vkv_k to be pinned near 1.0 whenever a column contains an element close to its row maximum in any non-collinear row, which explains the spike at 1.0 in Figure 16 and the limited gain on gate_proj in Figure 15. This property directly motivates our 1DOS and Constrained K-Means clusterization methods in Section 3.3 and Appendix B. By separating high-angular-distance outlier rows (1DOS) or grouping collinear output channels into C=3C=3 balanced clusters (Constrained K-Means), the rank-1 bound is computed over collinear subsets rather than across the entire layer. In our expanded evaluation in Table 3 (lines 1080–1120), LRSR-1DOS and LRSR-KMeans close up an average of 75% of the KLD gap between Per-Channel and Per-Group (g=128g=128) on the Qwen3 family (1.7B, 4B, and 8B).


Critical Weakness W2 & Question Q2: Operator type mismatch in experimental validation... Please provide empirical accuracy results when W^\hat{W} and A~\tilde{A} share the same data types (e.g., W4A4 or W8A8)...

We address this point from both hardware execution and empirical perspectives. First, in weight-only W4A16 execution, unpacking 4-bit weights to 16-bit floats prior to or within the matrix engine is a uniform bitwise conversion independent of scale metadata. In contrast, executing per-group quantization without custom fused kernels requires slicing the reduction axis into K/gK/g separate GEMM calls per layer, which introduces substantial launch and accumulation overhead.

Second, to provide empirical results when weights and activations share homogeneous integer data types (W8A8 and W4A4), we evaluated LRSR on a physical Rockchip RK3588 NPU (executing native INT8xINT8->INT32 and INT4xINT4->INT16 GEMM operations) alongside the RK3576 NPU (W8A16 and W4A16). Please refer to our response to Reviewer SY4q (Question 4) for the complete benchmark table and analysis across Qwen3-4B, Gemma4-5B-E2B, and LFM2.5-2.6B.


Critical Weakness W3 & Question Q3: Thin experimental evaluation... Please supplement the paper with empirical comparisons in both accuracy and latency against the calibration-free methods mentioned...

We expanded Section 4 and Appendix C to include Qwen3-8B, Gemma4-5B-E2B, and LFM2.5-2.6B, added the C4 dataset, evaluated a 4×34 \times 3 composability matrix with Standard, QuaRot, AWQ, and GPTQ (Figures 3–4 and Table 4), and added five downstream benchmarks including GSM8K, MMLU-Pro, and HumanEval (Figure 5 and Table 6). Please see the General Response for an overview of these additions. Finally, we discuss the runtime trade-offs of online Hadamard transforms (O(Klog⁡K)O(K \log K)) versus offline calibration in Appendix C.3 (lines 1197–1204).


Question Q1: In Transformer architectures, for linear projections immediately preceded by LayerNorm or RMSNorm...

Yes, LRSR supports this offline operator fusion for the five linear projections per Transformer block that immediately follow a normalization layer (q_proj, k_proj, v_proj after attention normalization, and gate_proj, up_proj after MLP normalization). For a single projection or when separate rescaled views are used, the input scale vector vv can be multiplied directly into the normalization weight vector offline. Furthermore, when sibling projections sharing the same normalization output (q/k/v or gate/up) consume a single shared activation tensor, one can define a shared input vector as the element-wise maximum of the sibling vv vectors and recompute each sibling projection's output vector uu. Because the shared input vector remains bounded by 1 and upper-bounds each individual sibling vector, Proposition 1 continues to hold for every layer while allowing a single offline fusion into the normalization weights. Explicit runtime pre-scaling is then only required for o_proj and down_proj. We have added this analysis to Section 3.1 (lines 191–195) and Appendix D (lines 1384–1403).

Reviewer_SY4qOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes LRSR, which approximates per-element quantization scale matrices using an upper-bounding rank-1 decomposition. By moving scaling operations outside GEMM, LRSR improves the accuracy of per-channel quantization with low memory overhead. The paper also introduces clustering techniques to further reduce quantization errors.

Critical strengths
  • The paper proposes a novel approach to reconstructing quantization scales using low-rank decomposition, improving quantization accuracy while preserving standard GEMM operations.
  • The method is simple and calibration-free, and the paper provides theoretical analysis and experimental results on accuracy, latency, and memory overhead.
Critical weaknesses
  • Insufficient motivation: Scale metadata accounts for only a small fraction of the total model memory, so the practical value of reducing its memory overhead is unclear.
  • Unclear practical benefits: Compared with widely used per-group quantization, LRSR still has noticeably lower accuracy and does not demonstrate significant inference performance improvements.
  • Limited experimental evaluation: The end-to-end experiments only cover Qwen3-1.7B and Qwen3-4B, lacking evaluations on larger models, downstream tasks, and comparisons with mainstream quantization methods.
  • Insufficient hardware validation: LRSR requires additional input activation scaling, which may affect low-bit activation quantization accuracy, while the experiments mainly focus on weight-only quantization.
Questions for authors
  1. On Qwen3-4B, the additional scale metadata of per-group quantization accounts for only about 2% of the total GPU memory usage. Could the authors clarify whether scale storage or memory access is a practical bottleneck and what benefits can be achieved by reducing this overhead?

  2. Could the authors compare LRSR with per-group quantization using optimized kernels in real inference frameworks, such as vLLM or TensorRT-LLM, in terms of end-to-end latency and throughput? In which hardware or inference scenarios does LRSR provide practical advantages?

  3. Could the authors extend the evaluation to larger models, such as Qwen3-8B and Qwen3-14B, and include downstream tasks such as MMLU-Pro, GSM8K, and HumanEval? Comparisons with mainstream quantization methods, such as GPTQ and AWQ, would also be helpful.

  4. Could the authors evaluate LRSR under W4A8 and W4A4 quantization and analyze how the additional input scaling affects activation quantization accuracy, computational overhead, and actual hardware efficiency?

3 comments

We thank the Reviewer SY4q for the constructive comments and questions regarding practical deployment scenarios, model scaling, downstream benchmarks, and low-bit activation quantization.


Critical Weakness 1 & Question 1: Insufficient motivation: Scale metadata accounts...

On GPUs, the ~2% memory footprint of per-group scale metadata (g=128g=128) is not a storage capacity bottleneck. Our original framing around metadata savings was imprecise, and we have updated the Abstract, Section 1, and Tables 2 and 5 accordingly (see the General Response).

The practical bottleneck addressed by LRSR is operator compatibility on fixed-function hardware rather than memory capacity. In per-group quantization, the scale factor changes along the reduction axis inside the dot product (Eq. 4), which requires custom fused dequantization kernels. On many specialized accelerators such as edge NPUs, custom kernels are unavailable and only GEMM operations are exposed. Executing per-group quantization on such devices requires splitting the reduction axis into K/gK/g separate GEMM calls per layer, which degrades inference speed. Consequently, these accelerators rely on per-channel quantization. LRSR provides a two-dimensional scale approximation whose factors are applied outside the GEMM contraction (Eq. 6), improving accuracy over per-channel scaling without requiring custom kernels or multiple GEMM calls.


Critical Weakness 2 & Question 2: Unclear practical benefits: Compared with widely used per-group...

On GPUs where optimized per-group kernels exist (such as Marlin in vLLM or our CUTLASS kernels in Table 5), LRSR and per-group quantization achieve comparable prefill and KV-cached decode latency (within 1–2% of the per-channel baseline across Qwen3-1.7B, 4B, and 8B in Table 5). Therefore, on general-purpose GPUs with fused kernel support, per-group quantization remains the preferred choice.

LRSR provides practical advantages in deployment scenarios where custom per-group kernels are unavailable, such as edge NPUs and fixed-function matrix engines. We integrated LRSR as a GGML backend in llama.cpp for Rockchip RK3588 and RK3576 NPUs using the vendor's MATMUL API. Because the hardware has limited support for per-group scaling inside its MAC array, per-channel and LRSR are the only single-GEMM granularities available. Overriding every Matrix Multiplication operation in the computing graph to the RK3588 NPU via standard GEMM increases prompt processing throughput by 4–5x compared to ARM CPU execution while significantly reducing active power consumption, and LRSR recovers accuracy over the native per-channel NPU pipeline as shown under Question 4 below.


Critical Weakness 3 & Question 3: Limited experimental evaluation: The end-to-end experiments only cover...

We extended our evaluation to Qwen3-8B (the largest dense model fitting on available hardware), Gemma4-5B-E2B, and LFM2.5-2.6B across WikiText-2 and C4, added comparisons and composability evaluations with QuaRot, AWQ, and GPTQ (Figures 3–4 and Table 4), and evaluated five downstream benchmarks including MMLU-Pro, GSM8K, and HumanEval (Figure 5 and Table 6). Please refer to the General Response and Section 4 / Appendix C of the revised manuscript for the full results.


Critical Weakness 4 & Question 4: Insufficient hardware validation: LRSR requires additional...

As analyzed in Appendix D (lines 1295–1403), multiplying input activations by the reduction-axis vector vv expands the intra-token dynamic range by a factor of ~3.2x. Under 8-bit activation quantization, the INT8 grid has sufficient resolution to accommodate this range expansion directly, whereas 4-bit activation quantization (W4A4) requires pairing LRSR with pre-conditioning to smooth activation outliers before quantization.

To validate this on physical integer GEMM hardware, the table below reports results from our llama.cpp backend on the Rockchip RK3588 NPU (which executes native W8A8: INT8xINT8->INT32 and W4A4: INT4xINT4->INT16 with dynamic per-token activation quantization) and the Rockchip RK3576 NPU (which executes native W8A16: FP16xINT8->FP32 and W4A16: FP16xINT4->FP32) over 32 chunks of 512 tokens on WikiText-2.

The complete benchmark table is provided in a separate reply comment directly below.

Under W8A8, LRSR directly reduces KLD and improves Top-1 agreement compared to standard per-channel quantization across all three models, both with and without Hadamard rotation. Under W4A4, unconditioned 4-bit activations fail to converge for both standard per-channel and LRSR; however, applying the Hadamard transform restores generation, and combining Hadamard with LRSR achieves higher Top-1 token agreement (73.70% vs. 72.72%) than Hadamard with per-channel scaling. Under the W8A16 and W4A16 pipelines on RK3576, LRSR similarly improves KLD and Top-1 agreement across all configurations.

ChipsetPipelineMethodQwen3-4B PPL / ΔPPL / KLD / Top-1 (%)Gemma4-5B-E2B PPL / ΔPPL / KLD / Top-1 (%)LFM2.5-2.6B PPL / ΔPPL / KLD / Top-1 (%)
RK3588W8A8Per-Channel12.15 / +0.37 / 0.0449 / 91.897.33 / +0.22 / 0.0289 / 92.8951.28 / +5.22 / 0.0447 / 89.56
RK3588W8A8LRSR12.24 / +0.47 / 0.0381 / 92.357.28 / +0.17 / 0.0214 / 93.8250.77 / +4.71 / 0.0414 / 89.95
RK3588W8A8QuaRot (Per-Channel)11.94 / +0.16 / 0.0019 / 97.947.13 / +0.02 / 0.0015 / 98.1645.85 / -0.21 / 0.0044 / 96.65
RK3588W8A8QuaRot (LRSR)11.95 / +0.17 / 0.0018 / 98.157.13 / +0.02 / 0.0014 / 98.1645.87 / -0.19 / 0.0042 / 96.85
RK3588W4A4Per-Channel26523.34 / +26511.56 / 8.7540 / 1.97602591.25 / +602584.14 / 11.3235 / 4.88477.43 / +431.37 / 4.0174 / 17.24
RK3588W4A4LRSR20771.10 / +20759.32 / 8.5834 / 1.65446495.66 / +446488.55 / 10.9794 / 3.70442.66 / +396.60 / 3.9295 / 17.82
RK3588W4A4QuaRot (Per-Channel)15.04 / +3.26 / 0.4766 / 72.7214.75 / +7.64 / 0.7285 / 64.9747.15 / +1.09 / 1.0558 / 56.31
RK3588W4A4QuaRot (LRSR)19.31 / +7.54 / 0.4995 / 73.7013.83 / +6.72 / 0.6666 / 67.2263.33 / +17.27 / 1.0027 / 56.46
RK3576W8A16Per-Channel11.98 / +0.20 / 0.0024 / 97.787.14 / +0.03 / 0.0021 / 97.7945.48 / -0.58 / 0.0048 / 96.57
RK3576W8A16LRSR12.01 / +0.23 / 0.0023 / 97.957.13 / +0.02 / 0.0011 / 98.4745.08 / -0.97 / 0.0047 / 96.68
RK3576W8A16QuaRot (Per-Channel)-7.12 / +0.01 / 0.0010 / 98.2845.37 / -0.69 / 0.0030 / 97.32
RK3576W8A16QuaRot (LRSR)-7.12 / +0.01 / 0.0008 / 98.5445.93 / -0.13 / 0.0025 / 97.49
RK3576W4A16Per-Channel20.33 / +8.55 / 0.6126 / 72.0126.14 / +19.03 / 1.2953 / 52.7042.81 / -3.25 / 0.9188 / 57.67
RK3576W4A16LRSR16.65 / +4.87 / 0.4310 / 75.7511.85 / +4.75 / 0.5180 / 69.2337.97 / -8.09 / 0.8372 / 60.11
RK3576W4A16QuaRot (Per-Channel)12.87 / +1.09 / 0.2797 / 79.4610.32 / +3.21 / 0.3610 / 75.7653.51 / +7.45 / 0.5605 / 66.91
RK3576W4A16QuaRot (LRSR)14.30 / +2.52 / 0.2676 / 80.039.33 / +2.22 / 0.2748 / 78.4149.73 / +3.67 / 0.5316 / 68.33

Thank you for the additional NPU results. I appreciate the accuracy gains with W4A16. I still have two concerns.

  1. LRSR’s input scaling widens the activation range, making 4-bit activations harder to quantize. Even with QuaRot, W4A4 perplexity is higher than the per-channel baseline on two of the three models.
  2. The reported 4–5× speedup compares CPU and NPU execution, so it is unclear how much of that gain comes from LRSR itself. I’m keeping my original rating.
Reviewer_VkNZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

LRSR approximates quantization scales with two vectors, reducing metadata while keeping scaling outside the main matrix multiplication.

Critical strengths
  • Simple and deterministic, without calibration data or training.
  • The proposed method requires little additional metadata.
  • Demonstrates an intermediate quality–memory trade-off between per-channel and per-group quantization.
Critical weaknesses
  • OneBit already uses rank-one magnitude factorization and input/output scaling, challenging the broad novelty claim.
  • End-to-end evaluation mainly covers two small Qwen models and WikiText.
  • Decoding without a KV cache provides limited deployment evidence.
  • The reported 29× reduction concerns additional overhead; total measured memory savings are approximately 2%.
  • Algebraic GEMM compatibility does not establish efficient execution on target hardware.

[1] OneBit: Towards Extremely Low-bit Large Language Models, 2024

Questions for authors

Beyond the calibration-free upper-bounding algorithm, what is technically new compared with OneBit’s rank-one magnitude factorization and input/output scaling?

1 comment

We thank the Reviewer VkNZ for the constructive comments and for highlighting the connection to OneBit and the limitations of our initial evaluation.


Critical Weakness 1 & Question 1: OneBit already uses... what is technically new compared with OneBit...

OneBit and LRSR address different problems and operate in different quantization regimes, which we now discuss in Section 2.3 (lines 155–161). OneBit targets 1-bit (±1\pm 1) weight binarization, where the round-to-nearest integer grid degenerates to a sign function. Consequently, OneBit uses rank-1 SVD/NMF of ∣W∣|W| only as a parameter initialization prior to 50 epochs of quantization-aware training (QAT) with knowledge distillation. In contrast, LRSR is a calibration-free post-training quantization (PTQ) granularity for multi-bit integer grids (such as INT4) that requires no training or fine-tuning.

In the multi-bit PTQ setting, applying OneBit's unconstrained Frobenius-norm factorization (such as rank-1 SVD) to the scale matrix underestimates local outlier scales (uvT<S+uv^T \lt S^+), which clips large weights to ±qmax⁡\pm q_{\max} and degrades model outputs without retraining. LRSR enforces an element-wise upper-bounding constraint (uvT≥S+uv^T \ge S^+, Proposition 1) that prevents weight clipping while allowing quantized weights to distribute across the multi-bit integer grid to compensate for the rank-1 approximation error (Appendix A, Figure 7). Beyond the upper-bounding decomposition, our work also introduces outlier-aware scale clusterization (1DOS and Constrained K-Means in Section 3.3 and Appendix B) and demonstrates composability as a scale granularity with QuaRot, AWQ, and GPTQ (Table 4).

To illustrate why standard SVD cannot be used in place of our upper-bounding algorithm in PTQ, the table below compares standard rank-1 SVD against Upper-Bounding Decomposition at INT4 on WikiText-2 using the same metrics as Table 3 in the manuscript:

ModelDecompositionPPLΔPPLΔP (%)KLDTop-1 (%)
Qwen3-1.7BRank-1 SVD3075266.32+3075249.79-42.5313.27440.00
Qwen3-1.7BUpper-Bounding (Ours)20.75+4.23-6.250.508271.83
Qwen3-4BRank-1 SVD3281473.36+3281459.36-45.9613.54140.01
Qwen3-4BUpper-Bounding (Ours)20.29+6.30-4.130.443175.11
Qwen3-8BRank-1 SVD1378588.25+1378578.31-47.0212.58500.08
Qwen3-8BUpper-Bounding (Ours)11.48+1.52-5.950.337476.97
Gemma4-5B-E2BRank-1 SVD9399478.26+9399435.58-31.8813.13380.98
Gemma4-5B-E2BUpper-Bounding (Ours)54.61+11.91-7.541.402950.69
LFM2.5-2.6BRank-1 SVD51.75+6.33-10.571.731645.59
LFM2.5-2.6BUpper-Bounding (Ours)44.76-0.67-0.370.150181.83

As shown in the table, replacing the upper-bounding constraint with standard rank-1 SVD causes weight clipping that collapses generation across Qwen3 and Gemma4 models (KLD >12.5> 12.5 and Top-1 agreement near 0%0\%), whereas the upper-bounding decomposition preserves model accuracy.


Critical Weakness 2: End-to-end evaluation mainly covers two small Qwen models and WikiText.

We have expanded the evaluation to five models across three architectures (up to 8B parameters), two datasets (WikiText-2 and C4), four composable quantization families (Standard, QuaRot, AWQ, and GPTQ), and five downstream tasks. Please refer to the General Response as well as Tables 3, 4, and 6 in the revised manuscript.


Critical Weakness 3: Decoding without a KV cache provides limited deployment evidence.

We updated the runtime benchmark to use a pre-allocated static KV-cache across Qwen3-1.7B, Qwen3-4B, and Qwen3-8B. Please see the General Response and Table 2 (Section 4.2) and Table 5 (Appendix C.3) in the revised manuscript.


Critical Weakness 4: The reported 29× reduction concerns additional overhead; total measured memory savings are approximately 2%.

We agree that emphasizing the 29x metadata reduction overstated the practical benefit of saving ~2% total memory. The objective of LRSR is to improve accuracy over per-channel quantization on accelerators that only support standard GEMM calls, while matching the memory and latency profile of per-channel quantization. As noted in the General Response, we removed the 29x metadata and Pareto frontier claims from the manuscript.


Critical Weakness 5: Algebraic GEMM compatibility does not establish efficient execution on target hardware.

In addition to the KV-cached CUTLASS GPU profiling in Table 5 and the offline normalization fusion in Appendix D (lines 1384–1403), we evaluated LRSR on two physical Neural Processing Units (Rockchip RK3588 and RK3576) using the vendor's fixed GEMM API. Please refer to our response to Reviewer SY4q (Question 4) for the complete NPU hardware results across W8A8, W4A4, W8A16, and W4A16 pipelines.

Comments on the paper

1 comment

We thank the reviewers and the Area Chair for their time and constructive feedback. We have uploaded a revised manuscript that addresses the shared concerns regarding positioning, experimental scope, runtime profiling, and downstream evaluation.

First, we clarified the positioning and motivation of the paper. LRSR is presented as a quantization granularity (an intermediate alternative between per-channel and per-group) designed for fixed-function GEMM accelerators where custom per-group kernels are unavailable, rather than as a method aimed at saving GPU metadata memory. We removed the misleading 29x metadata reduction and Pareto frontier claims from the Abstract, Section 1 (lines 054–065), and Section 5, removed the Pareto plots, and removed the metadata overhead column from the runtime tables (Table 2 and Table 5).

Second, we expanded the generative distribution evaluation in Section 4.1 (Figures 3–4) and Appendix C.2 (Tables 3–4, lines 1026–1187) from two models on one dataset to five models across three architectures (Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Gemma4-5B-E2B, and LFM2.5-2.6B) evaluated on both WikiText-2 and C4. Because LRSR defines a scale granularity, it is orthogonal to weight pre-processing and calibration methods. Table 4 evaluates Per-Channel, LRSR-Naive, and Per-Group (g=128g=128) across four method families: Standard quantization without any preprocessing, QuaRot, AWQ, and GPTQ. Across all 40 evaluated INT4 configurations, replacing per-channel with LRSR-Naive reduces KL-Divergence by up to 34% and increases Top-1 token agreement by up to 7.8 percentage points (for example, on Qwen3-4B averaged over WikiText-2 and C4, KLD drops from 0.583 to 0.400 under STD, from 0.436 to 0.365 under QuaRot, from 0.438 to 0.357 under AWQ, and from 0.318 to 0.263 under GPTQ; with clustered variants closing up to 75% of the KLD gap to per-group on Qwen3-1.7B, 4B, and 8B).

Third, we updated the hardware profiling pipeline in Section 4.2 (Table 2, lines 414–431) and Appendix C.3 (Table 5, lines 1188–1234) to measure prefill (2048 tokens) and decoding (128 tokens) with a static KV-cache across Qwen3-1.7B, Qwen3-4B, and Qwen3-8B. Across all three model scales, the prefill and decode latencies of LRSR-Naive remain within 2% of the per-channel baseline while maintaining the same memory footprint (e.g., 675.8 ms vs. 682.9 ms prefill, 5515.9 ms vs. 5459.9 ms decode, and 2877.0 MB vs. 2876.7 MB on Qwen3-4B; and 1298.6 ms vs. 1311.9 ms prefill, 7060.9 ms vs. 6950.4 ms decode, and 6063.9 MB vs. 6061.3 MB on Qwen3-8B).

Fourth, we added five downstream benchmarks in Section 4.3 (Figure 5, lines 432–461) and Appendix C.4 (Table 6, lines 1235–1294): MMLU-Pro (5-shot CoT), ARC-Challenge (0-shot), WinoGrande (0-shot), GSM8K (8-shot CoT), and HumanEval (pass@1). Across Qwen3-1.7B, Qwen3-4B, and LFM2.5-2.6B at INT4, LRSR-Naive improves prediction agreement with the unquantized BF16 reference on 14 of the 15 model-task pairs and recovers task accuracy on multi-step generation tasks compared to per-channel quantization (for instance, recovering GSM8K from 12.89% to 39.84% and HumanEval from 4.27% to 15.85% on Qwen3-1.7B; and recovering MMLU-Pro from 38.28% to 44.53%, GSM8K from 56.64% to 61.72%, and HumanEval from 26.22% to 43.90% on Qwen3-4B).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.