acceptodds
Under review as a conference paper at ICLR 2027

ShamAN-Q: Shampoo Augmented NanoQuant for Sub-1-bit LLM Weights

Abstract

We introduce ShamAN-Q, a sub-1-bit post-training quantization method that extends NanoQuant (Chong et al., 2026) by replacing its diagonal reconstruction geometry with a dense but tractable curvature metric, using a form and computation paradigm popularized by the Shampoo optimizer (Gupta et al., 2018). For each linear weight, ShamAN-Q fits a Kronecker product to the empirical Fisher Information matrix of a small calibration set by Kullback–Leibler minimization and uses the result to form a Mahalanobis reconstruction loss. The continuous ADMM updates from NanoQuant become solutions to Sylvester equations, while its discrete projection and deployment format remain unchanged. Because the curvature is local to a given set of weights, ShamAN-Q re-measures the input curvature statistic for each layer immediately before layer factorization and periodically refreshes all statistics on the partially quantized model. ShamAN-Q also redistributes the uniform rank from NanoQuant across layers at the same total number of bits. On Qwen3-Base, ShamAN-Q lowers WikiText-2 perplexity at 1 bpw from 27.56 to 22.96 (0.6B), 19.21 to 16.72 (1.7B), and 14.29 to 13.80 (4B) while increasing zero-shot reasoning task performance on the Eleuther LM Evaluation Harness. On 0.6B, ShamAN-Q at 0.8 bpw matches the published perplexity of NanoQuant at 1.0 bpw.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Quantization, Optimization, Geometry, LLM, Hardware

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LSX8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes ShamAN-Q, which is a post-training quantization method that quantizes an LLM to less than 1 bit per weight. ShamAN-Q is an extension to an existing work called NanoQuant. The major contributions of ShamAN-Q are as follows:

  1. ShamAN-Q adapts KL-Shampoo’s curvature estimator to NanoQuant’s LB-ADMM step.
  2. The authors proposes non-uniform rank allocation
  3. The authors propose periodic curvature refresh to NanoQuant’s pipeline
  4. The authors propose to apply spectral tempering to the estimated curvature, which reduces trust in the dominant curvature directions to prevent inaccuracy.

Experimental evidence shows ShamAN-Q improves model performance compared to NanoQuant with a minor increase in wall-clock time. Ablation studies validate that tempering and curvature refresh can improve performance.

Critical strengths

The paper advances the sub-field of sub-1-bit post-training quantization by adapting techniques (Kronecker curvature estimators and allocating different compression budgets to different layers) from other sub-fields (above 1-bit) of PTQ. These novel adaptations (tempering and dense-curvature LB-ADMM) are technically challenging due to the problem structure of sub-1-bit PTQ. Empirical improvements upon the baseline method NanoQuant is significant.

Critical weaknesses
  1. The impacts of key contributions should be verified in isolation. In its current state, the paper reads more like a technical documentation, whose main goal is to present every technical detail about a product, and less like a research paper, whose main goal is to present the few key ideas/insights that enable/contribute to the success of the product.
    a. Spectral tempering, or reducing the trust in the dominant curvature directions, is presented as a useful technique since the empirical Fisher is inaccurate when reconstruction error is large, which often is the case for sub-1-bit quantization. I think this explanation is believable, but an ablation study would make it more convincing. Consider sweeping reconstruction error from small to large (perhaps by varying bits per weight) while comparing the model perplexity of tempering versus no tempering. While Table 4 contains ablation studies involving tempering, it does not show why tempering is useful, only that it does. Sweeping reconstruction error, on the other hand, would reflect the “why”.
    b. The idea of periodic curvature refresh is intuitive, but its impact on model perplexity and end-to-end runtime is unclear. Consider sweeping through the hyperparameter Delta from small to large and show perplexity versus runtime. Are extremely frequent refreshes impractical? When does the performance benefit of increasing refresh frequency saturate?
    c. Why do both allocators in Section 4.2 allocate more bits to later layers? Consider adding more details.
  2. The paper is difficult to read.
    a. Not every reader is an expert in NanoQuant. When presenting a new technical detail about ShamAN-Q (in Section 4), consider reminding the reader why NanoQuant implemented that technical detail the way it does, and why it is technically challenging to apply NanoQuant’s implementation directly to ShamAN-Q. I understand that Section 3.1 already contains all relevant details about NanoQuant, but a gentle reminder can reduce the load on the reader’s short-term memory and help them absorb the information faster.
    b. Consider using the first sentence in a paragraph to motivate the reader to read that particular paragraph. This can be especially helpful in regions of the paper that contain a lot of technical details. First-time readers often get lost in such regions and forget why they are reading. For example, line 362 could summarize what is happening in the paragraph from line 362 to line 377.
Questions for authors

See the weakness section above.

3 comments

We thank the reviewer for the careful read of our work and the direct comments. We are glad that you recognize the challenge of adapting the techniques to this difficult setting.

Weaknesses

  1. We provide an additional LOO ablation that examines the effects of each contribution in isolation at the 1.7B size. They show that the individual effects are significant for the components individually except for the refreshing, suggesting that they interact with some of the other contributions. We can include this result in the main text, along with the additional explanation. Together with the cumulative ablation, it provides a more complete picture of the effects of individual components and the design process. | Ablation | PPL | zero-shot | ΔPPL (paired) | Δzero-shot | |---|---|---|---|---| | None (full recipe) | 15.95 ± 0.30 | 0.455 ± 0.008 | – | – | | NKP instead of KL | 16.97 ± 0.33 | 0.439 ± 0.008 | +1.02 ± 0.07 | −0.016 ± 0.011 | | No tempering (p = 1) | 17.18 ± 0.28 | 0.429 ± 0.008 | +1.23 ± 0.37 | −0.026 ± 0.010 | | Uniform ranks | 16.91 ± 0.22 | 0.446 ± 0.018 | +0.95 ± 0.09 | −0.009 ± 0.021 | | No curvature refresh | 16.03 ± 0.27 | 0.455 ± 0.007 | +0.08 ± 0.10 | +0.000 ± 0.007 | | No fresh input factor | 15.99 ± 0.17 | 0.438 ± 0.026 | +0.04 ± 0.19 | −0.017 ± 0.029 |

We can add this table and explanation in the main text.

  • a) This is a good suggestion. As the error itself is not an independent variable, we sweep bpw to examine the relationships between tempering, error, and PPL. The results show that at every bpw level, tempering helps with PPL. Further, Zero-shot accuracy improves at every bpw level. | bpw | PPL, p = 1 | PPL, p = ½ | ΔPPL | Untempered metric error, Δ | |---|---|---|---|---| | 0.4 | 38.04 ± 0.31 | 36.93 ± 1.17 | −1.10 | +0.0103 | | 0.55 | 29.03 ± 0.83 | 27.00 ± 0.55 | −2.03 | +0.0090 | | 0.7 | 23.13 ± 0.29 | 21.85 ± 0.14 | −1.28 | +0.0075 | | 0.85 | 19.88 ± 0.44 | 18.98 ± 0.40 | −0.90 | +0.0060 | | 1.0 | 17.43 ± 0.01 | 16.84 ± 0.03 | −0.59 | +0.0044 | | 1.3 | 14.70 ± 0.17 | 14.19 ± 0.10 | −0.51 | +0.0023 |

At every bpw level, while tempering does make the proxy error of the untempered metric worse, perplexity actually improves. The spread generally grows as the error grows, but it also generally gets noisier. We can plot this table as a figure with the accompanying text as part of the caption in the main text, and include the table in the appendix for completeness.

  • b) Exactly. The refresh requires forward and backward passes through the entire network, and so are relatively more expensive than layerwise or blockwise optimizations. We swept the refresh interval Δ (in outer ADMM iterations) at 4B and measured the perplexity cost directly on WikiText-2: | Δ (refresh interval) | PPL (WikiText-2) | Wall-clock (4B, 1×A100) | |---|---:|---:| | 0 (every iteration) | 13.996 | 4h 28m | | 3 | 14.426 | 5h 19m | | 9 | 13.989 | 4h 37m | | 18 | 13.788 | 4h 39m | | 20 (paper's default) | 13.749 | 3h 38m |

Runtime grows with refresh frequency as expected, from 3h 38m at the default setting from the paper (Δ=20) up to 5h 19m at Δ=3; Δ=0 is cheaper than Δ=3/9/18 since it does not actually need an extra forward/backward pass for the refresh.

  • c) Good question. We empirically found that the later layers are more sensitive in our paradigm, which includes re-tuning after preceding perturbations. Concretely, we measured the change in PPL resulting from binarizing groups of blocks at different snapshots as the algorithm proceeds along the network, and corresponding weight reconstruction errors. We see that the weight reconstruction is about the same difficulty across the network, as evidenced by the flat error in the uniform bit allocation setting. However, the impact of this error on PPL is heavier at the end. We also examine the same measurements for a simple geometrically ramping bit allocation where block bb gets bit budget ∝exp⁡(ρb/(B−1))\propto \exp(\rho b/(B-1)). We see that despite larger weight reconstruction error in the early part of the network, we do not incur significant additional increase in PPL, and conversely, we reduce both the weight reconstruction error and PPL penalty in the later layers. | depth (blocks) | uniform ranks: ADMM weight error | uniform ranks: PPL after group | uniform ranks: ΔPPL added by group | ramp ρ = 0.6: bit multiplier | ramp ρ = 0.6: ADMM weight error | ramp ρ = 0.6: PPL after group | ramp ρ = 0.6: ΔPPL added by group | |---|---|---|---|---|---|---|---| | 0–6 (first quarter) | 0.364 | 15.83 | +1.60 ¹ | 0.74× → 0.85× | 0.468 | 16.68 | +1.88 | | 7–13 | 0.344 | 17.51 | +1.67 | 0.87× → 1.00× | 0.381 | 18.17 | +1.49 | | 14–20 | 0.336 | 20.35 | +2.84 | 1.02× → 1.17× | 0.311 | 20.69 | +2.53 | | 21–27 (last quarter) | 0.352 | 27.35 | +7.00 | 1.20× → 1.35× | 0.266 | 25.31 | +4.62 |

We can update the text to include this explanation and include this table in the appendix.

Weaknesses, cont'd

  1. We thank the reviewer for this helpful perspective. In trying to balance exposition with the space constraints of the current format, we necessarily had to make a tradeoff between background material and method description. However, we can certainly improve our clarity as outlined below, following your suggestions:
    • a) We can motivate our techniques by more directly tying them back to the baseline NanoQuant method.
    • b) We appreciate this candid assessment. We can provide stronger thesis statements for subsections and paragraphs where they are weak. For example, also addressing the previous point (a), we could start the paragraph you identified in L366-372 with “Now, we provide the continuous U\mathbf{U} and V\mathbf{V} updates that optimize Eq (14), which are closed-form solutions to Sylvester equations instead of the simpler normal equations found in NanoQuant.”

I appreciate the authors for the hard work. My concerns have been addressed.

I intend to keep my already positive score.

Please treat my comments about readability as general suggestions for post-rebuttal edits. I really hope the paper can be accessible to a broader range of audiences (e.g. people who are not experts on PTQ).

It would be great if the addtional results above are included in the paper (main text or appendix) so readers who are interested can find the information.

Reviewer_zpW5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

A - Paper Summary

ShamAN-Q is a drop-in improvement to NanoQuant, a sub-1-bit PTQ method representing each linear layer as two low-rank binary factors plus two FP16 scale vectors. NanoQuant fits these factors by Frobenius reconstruction after diagonal preconditioning (per-channel second moments). ShamAN-Q replaces this diagonal geometry with a dense two-sided Mahalanobis metric estimated by a Kronecker-product fit of the empirical Fisher (H ≈ A ⊗ G), similar to how Shampoo generalizes diagonal AdaGrad. The continuous LB-DMM updates become Mahalanobis Sylvester equations, solvable in a cached joint eigenbasis; SVID projection, scale extraction, packing, and distillation are unchanged.

B - Paper Contribution

(1) Dense Kronecker curvature via a KL-Shampoo fit with scale-matched shrinkage, preferring KL to NKP because its token weights are bounded leverage scores (condition ~40 vs. hundreds);

(2) non-uniform rank allocation at bit parity using a depth prior plus either a hand table or a measured-sensitivity allocator;

(3) spectral tempering (p=12p = \frac{1}{2}) because the 1-bit reconstruction error is far outside the Fisher's quadratic regime;

(4) fresh per-layer input statistics and periodic full refresh on the partially quantized model.

C - Paper Validation

Model: Qwen3-Base at 0.6B, 1.7B, 4B, 8B, 14B at 1.0, 0.8, and 0.55 bpw budgets (bit parity with NanoQuant).

Metrics: WikiText-2 perplexity and zero-shot mean on Eleuther LM Eval Harness.

Baselines: NanoQuant published and reproduced.

Ablation cumulative; wall-clock and peak memory reported.

Critical strengths
  1. The conceptual transplant is sharp, correct, and carried through end-to-end. "ShamAN-Q is to NanoQuant what Shampoo is to diagonal AdaGrad" is right, and the paper follows the analogy through the estimator (KL-Shampoo fixed point), the solve (Sylvester instead of normal equations), and the robustness engineering (tempering). This is consequential because it gives the sub-1-bit PTQ literature a principled, well-understood curvature upgrade rather than an ad hoc heuristic, and it preserves the deployment format so inference kernels are inherited unchanged.

  2. Strong 0.55 bpw results and strict bit parity. At 0.55 bpw, ShamAN-Q at 0.6B reaches 38.80 PPL vs. 62.65 reproduced (and 52.94 published), and at 0.8 bpw ShamAN-Q at 0.6B matches NanoQuant's 1.0 bpw perplexity. Every model spends exactly the same total bits as NanoQuant, and the effective bpw values (0.527–0.545 at "0.55 bpw") are reported honestly. This is consequential because it demonstrates a true compression-efficiency gain in the regime where sub-1-bit methods matter, under the correct control.

Critical weaknesses
  1. Gains vanish at scale, and everything is single-run. The 1.0 bpw improvement is 4.60 PPL at 0.6B but 0.01 PPL at 14B; the 14B zero-shot mean (0.506 vs. 0.498) is within noise of a single run. No seed variance is reported, and the paper concedes that the fresh/refresh contribution is within noise in aggregate. This matters because the method's value proposition weakens where sub-1-bit compression is most needed (large models), and the evidence cannot distinguish the 14B tie from an under-tuned configuration.

  2. The reproduction gap muddies the headline numbers and the overhead is legitimate. NanoQuant reproduced is worse than published at 0.6B (29.21 vs. 27.56) and 4B (14.86 vs. 14.29), yet the abstract quotes published-vs-ShamAN-Q deltas (27.56→22.96), conflating method gain with reproduction gap. Separately, quantization time rises ~1.7–1.8× (0.6B 1h12m→1h59m; 4B 2h30m→4h30m), which for gains that vanish at 14B is a hard trade. This matters because a reader cannot tell from the headline how much of the improvement is method vs. reproduction, and the cost is non-trivial.

Questions for authors
  • Can you report seed variance for the 1.7B/4B/8B cells, and state whether the 14B tie is a fundamental limit or an under-tuned depth-prior/calibration-budget artifact (the paper says tuning "would likely help")?

  • Can you separate "vs. published NanoQuant" and "vs. reproduced NanoQuant" in the abstract and Table 1, so the method gain is not conflated with the reproduction gap?

  • Can you give a leave-one-out ablation (rather than cumulative) so KL-vs-NKP and fresh/refresh can be attributed individually, given that the cumulative ablation only puts tempering and rank allocation outside noise?

2 comments

We thank the reviewer for a careful examination of the numbers we report and their interpretation. We believe that incorporating the answers will improve the presentation of our results.

Weaknesses

  1. We note that not all gains saturate with size. While the gains in WikiText-2 PPL do plateau, the gains in zero-shot mean stay consistently present. These metrics together provide a fuller view on model performance. We address seed variance directly under Questions (Q1) below: results are stable across seeds at 1.7B and 8B, while 4B currently shows a wider spread that we are tightening with additional clean-cache reruns now in progress.
  2. You are correct that the reproduction gap is not separated from the method gap in the headline numbers. However, the effects are mostly in opposite directions, as the reproduction gap is largely negative or small, whereas the method gap is positive. We chose this reporting because it was conservative, favoring the NanoQuant as a strong baseline. This shows that even though we were not able to reproduce fully the published number, ShamAN-Q still improves upon it. We can separate these effects for a clearer presentation.

Questions

  • The seed variances are as follows (PPL on WikiText-2, 3 seeds): | Model | mean PPL | std PPL | |---|---:|---:| | 1.7B | 16.116 | 0.121 | | 4B | 14.599 | 0.556 | | 8B | 11.803 | 0.302 | | 14B | 11.119 | 0.056 |

1.7B-8B show a tight (1.7B) to moderate (4B) spread; 14B is very tight. The similar PPL at 14B is not a fundamental limit. We take as many relevant hyperparameters as is possible from NanoQuant without tuning, e.g. number of calibration sequences, number of tokens per calibration sequence. Further, because the same model is used to evaluate PPL and zero-shot reasoning task average score, it is not necessarily optimally tuned for either individually.

  • As mentioned above in the “Weaknesses” section, we can report these effects separately so it is clearer.
  • We provide an additional LOO ablation that examines the effects of each contribution in isolation at the 1.7B size. They show that the individual effects are significant for the components individually except for the refreshing, suggesting that they interact with some of the other contributions. We can include this result in the main text, along with the additional explanation. Together with the cumulative ablation, it provides a more complete picture of the effects of individual components and the design process. | Ablation | PPL | zero-shot | ΔPPL (paired) | Δzero-shot | |---|---|---|---|---| | None (full recipe) | 15.95 ± 0.30 | 0.455 ± 0.008 | – | – | | NKP instead of KL | 16.97 ± 0.33 | 0.439 ± 0.008 | +1.02 ± 0.07 | −0.016 ± 0.011 | | No tempering (p = 1) | 17.18 ± 0.28 | 0.429 ± 0.008 | +1.23 ± 0.37 | −0.026 ± 0.010 | | Uniform ranks | 16.91 ± 0.22 | 0.446 ± 0.018 | +0.95 ± 0.09 | −0.009 ± 0.021 | | No curvature refresh | 16.03 ± 0.27 | 0.455 ± 0.007 | +0.08 ± 0.10 | +0.000 ± 0.007 | | No fresh input factor | 15.99 ± 0.17 | 0.438 ± 0.026 | +0.04 ± 0.19 | −0.017 ± 0.029 |

We can add this table and explanation in the main text.

I thank the authors for addressing my questions and concerns. I will maintain my positive score.

It would be great if the authors could include the mentioned table and explanation in the main text.

Reviewer_Xhz5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ShamAN-Q extends NanoQuant for sub-1-bit post-training quantization by introducing a dense, two-sided reconstruction metric based on Kronecker curvature statistics. The resulting continuous factor updates are solved through Sylvester equations, while the binary representation and discrete projection are retained. The method additionally uses spectral tempering, refreshed calibration statistics, and nonuniform rank allocation under a fixed total bit budget. Experiments on Qwen3-Base models from 0.6B to 14B report improved WikiText-2 perplexity over NanoQuant, with mixed downstream accuracy results at the lower bit budgets and increased offline quantization cost.

Critical strengths

1.Meaningful perplexity improvements at matched storage budgets. Table 1 reports improvements across multiple model sizes and bit budgets. At approximately 1 bpw, perplexity improves from 29.21 to 22.96 on 0.6B and from 18.76 to 16.72 on 1.7B relative to the reproduced baseline. Reporting both published and reproduced NanoQuant results is useful. The uniform-rank experiment in Table 4 also provides evidence that the gains are not solely due to rank redistribution.

2.A coherent extension of the reconstruction solver. The two-sided quadratic objective leads naturally to the Sylvester updates in Eqs. (15)–(16). Retaining Euclidean consensus penalties preserves the existing discrete projection, and retaining the packed representation offers compatibility with the existing deployment implementation. This is a practical way to incorporate richer reconstruction geometry without introducing a new weight format.

Critical weaknesses

1.The specific contribution of the proposed curvature geometry is not sufficiently isolated or justified. Table 4 is informative, but it does not compare matched diagonal, input-only dense, and two-sided dense metrics while holding the diagonal maps, refresh schedule, and tuning settings fixed. This distinction matters because Section 4.3 replaces the KL-fitted input statistic with an unweighted fresh second moment and subsequently tempers the metric. These may be useful heuristics, but the resulting metric is no longer the original KL fit. Consequently, the experiments do not establish how much benefit specifically requires output-side correlations or the KL estimator. The statement that the square-root metric is “regret-optimal” also lacks a specified uncertainty model, theorem, or supporting derivation.

2.The downstream evaluation is insufficiently specified to support the breadth of the conclusions. The manuscript reports an Eleuther Harness average without identifying the constituent tasks, metric choices, or per-task results. All results are single runs, yet several design choices are described as being within or outside noise without corresponding uncertainty estimates. Moreover, the downstream improvements are not uniform: at approximately 0.55 bpw, the reproduced NanoQuant average is 0.409 versus 0.397 for ShamAN-Q on 4B, and 0.419 versus 0.410 on 8B. These point estimates do not establish significant regressions, but they do require a more qualified conclusion. Evaluation on one model family and against one baseline further limits the scope of the claimed advantage.

Questions for authors

1.Controlled evidence: Can you compare diagonal, input-only dense, and two-sided dense reconstruction metrics using identical ranks, calibration data, diagonal maps, refresh schedules, and tuning budgets? This would help determine whether output-side correlations provide a meaningful benefit beyond a simpler input-aware reconstruction objective.

2.Mathematical clarification: How should the KL-fitting interpretation be understood after replacing the fitted input factor with the fresh unweighted second moment and applying spectral tempering? Please also justify the claim in Section 4.1 that the fitted statistics have the same diagonals as NanoQuant’s moments, despite the token weights in Eq. (9). For Eq. (8), please clarify the likelihood or regularization interpretation when the empirical Fisher is rank deficient.

3.Evaluation clarification: Which exact tasks, harness versions, metrics, and averaging rules produce the reported zero-shot means? Please specify the calibration/evaluation splits and provide per-task results with appropriate uncertainty estimates, particularly for the lower-bit settings where perplexity and downstream accuracy move in different directions.

4.Deployment claim: Section 5.3 states that decoding throughput is unchanged. Was this measured with the proposed nonuniform ranks? The same representation and total storage budget do not by themselves guarantee identical kernel performance when layer dimensions change.

5 comments

We thank the reviewer for the thorough reading and thoughtful comments and questions. They have been tremendously helpful. We are encouraged that you felt you learned something interesting and found the method to demonstrate practical and meaningful improvements over prior art.

Addressing Weaknesses:

  1. We have run an additional experiment to better isolate the effect of the different components. The square-root metric is “regret-optimal” in the original setting of the usage as an optimization preconditioner. To directly isolate the curvature geometry's own contribution (holding rank allocation and spectral tempering fixed), we ran a controlled 4B comparison across three curvature variants, measuring PPL on WikiText-2:
Curvature variantPPL (WikiText-2)
Diagonal13.792
Input-only dense13.955
Two-sided dense (ours)13.700

This shows the benefit depends on output-side correlations, not just on having a dense (non-diagonal) curvature estimate. That is, fitting a fresh input-only dense statistic with a diagonal output statistic is actually slightly worse than the diagonal baseline (13.955 vs. 13.792), while the full two-sided fit that also incorporates dense output-side statistics actually delivers a gain (13.700).

  1. This is valuable feedback. We address this below in item (3) of the “Questions” section.

Answers to Questions:

  1. As stated above in the “Weaknesses” section, we have run additional experiments to address these questions.
  2. We thank the reviewer for the detailed clarifying questions.
    • a) On fresh factors: We adopt an approach with two time scales, because our hypothesis was that the input statistic would be less robust to staleness, as binarization of the preceding layers together with the effects of TuneFP\rm{TuneFP} in the current block would more dramatically affect the input of the current layer than the gradients at the output. However, we also would like to avoid frequent full backward passes at the short time scale. Thus, the fresh unweighted second moments are the K-FAC approximations to the input statistics, which make the assumption that xt\mathbf{x_t} and δt\boldsymbol{\delta_t} are independent (Martens et al., 2015). These are computable without a full backward pass. We see this as a computationally friendly proxy for the KL fit in lieu of fresh gradients when we still have fresh activations from the partial forward pass. The KL fit is used at the longer time scale for output statistics, in which the token weights then allow the input statistic to propagate into the overall curvature of the layer. Taken together, this can be viewed as using K-FAC to approximate a half step of the KL fit. The tempering can then be applied on top of this. In our experiments, we logged the quadratic expansion of the loss at each layer, \rm{tr}(\mathbf{G_\gamma}\Delta \mathbf{W}\mathbf{A_\rm{fresh}}\Delta\mathbf{W}^\top), as a diagnostic metric. It showed that the curvature from using the fresh input statistic with the stale output statistic still predicts well the increase in loss caused by going from the original weights to the ADMM solution of the current layer, consistent with our hypothesis. Note that while this is a decent fit, it ignores cros-layer interactions that compound through the network, so tempering hedges against this mismatch between the objective at each layer and the end-to-end PPL (as opposed to an inaccurate quadratic model). We can include this interpretation in the text.
    • b) On Eq (8)-(9): You are correct, that as written the claim in Section 4.1 is ambiguous. It should instead note that the diagonal of Afresh\mathbf{A_\textrm{fresh}} matches the din2d^2_{\textrm{in}} diagonal from NanoQuant, and this claim should belong in Section 4.3. More generally, the form of the shrinkage from Eq. 3 relates to Eq. 10, not Eq. 8. In terms of the likelihood, you are also right in noticing that it has an interpretation problem when H\mathbf{H} is deficient in rank. The log⁡det⁡\log\det term is technically infinite, corresponding to degenerate multivariate Gaussians with lower effective dimension than the ambient dimension, meaning the distribution is supported on a subset of zero measure. However, since we drop the log⁡det⁡H\log\det \mathbf{H} term in our KL fit as having no dependence on A\mathbf{A} or G\mathbf{G}, the optimization objective stays well-posed for positive definite matrices A\mathbf{A}, G\mathbf{G}. Noting that the statistics accumulate NN rank-1 terms, where NN is the number of calibration tokens, as long as NN exceeds the dimensions of these matrices max⁡(m,n)\max(m,n), positive definiteness is achieved. In our settings, we have N≥128×2048>max⁡(2048,6144)N\ge128\times2048>\max(2048,6144). The regularization can then be seen as a numerical guard for Cholesky decomposition and inversion instead of being required to achieve positive definiteness. We can include this explanation in the main text to provide this intuition.

Answers to Questions, cont'd

  1. The tasks were BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, from lm-eval harness version 0.4.13. The reported zero-shot mean is an unweighted average of the accuracies of the 6 tasks. We report the uncertainties produced by the harness, which are standard deviations computed from a binomial distribution fit. The calibration was performed using 128 windows of 2048 tokens from the WikiText-2 train split, chosen with replacement, except for 8B and 14B, which used 512 windows from the same train split. Perplexity was computed from the WikiText-2 test split using non-overlapping 2048-token windows and averaged per token. Thus, there is no data leakage in calibration. The per-task results are in the table below. We will include this explanation in the main body and full table in the appendix (or the body if space permits).

Zero-shot accuracy, lm-eval 0.4.13, full eval sets. Avg. = unweighted mean of the six task accuracies. PPL = WikiText-2 test, 2048-token non-overlapping windows.

ModelMethodbpwBoolQPIQAHellaSwagWinoGrandeARC-eARC-cAvg.
Qwen3-0.6B-BaseNanoQuant (diag)0.97353.552.827.950.027.320.638.7
Qwen3-0.6B-BaseOurs0.97362.057.929.052.236.219.142.7
Qwen3-1.7B-BaseNanoQuant (diag)0.98661.458.930.750.835.218.942.6
Qwen3-1.7B-BaseOurs (uniform ranks)0.98662.260.731.153.640.820.644.8
Qwen3-4B-BaseNanoQuant (diag)0.98656.061.032.855.745.621.845.5
Qwen3-4B-BaseOurs0.98660.661.633.355.844.821.946.4
Qwen3-8B-BaseNanoQuant (diag)0.99462.262.634.058.647.323.148.0
Qwen3-8B-BaseOurs0.99462.063.535.659.352.924.849.7
Qwen3-14B-BaseNanoQuant (diag)0.99868.263.736.259.746.224.849.8
Qwen3-14B-BaseOurs0.99864.962.936.462.450.626.150.6
Qwen3-32B †NanoQuant (diag)0.99363.666.440.862.865.034.455.5
Qwen3-32B †Ours0.99380.967.544.468.065.837.760.7
max. std. error0.91.20.51.41.01.4

† Qwen3-32B is the post-trained checkpoint (no Base release).

  1. We have measured the runtime directly with the kernel-level decode benchmark (gemv quant kernels, seqlen 2048, batch size 1, 128 generated tokens, 5 warmup + 10 profiled iterations, A100 GPU), comparing the uniform-rank (NanoQuant) and measured-sensitivity non-uniform-rank (ShamAN-Q) checkpoints at matched bit-parity:
ModelUniform tok/sShamAN-Q tok/sΔ
0.6B417.68412.21-1.3%
1.7B346.86345.71-0.3%
4B236.11232.17-1.7%
8B194.92190.09-2.5%
14B135.55136.76+0.9%

Measured throughput differences are small (within ±2.5%) and within the noise band of the uniform baseline (±2.2% is one standard deviation across 5 repeated decode runs of the same uniform checkpoint), consistent with the §5.3 claim that deployment throughput is effectively unchanged; at 14B the sign flips to being slightly positive (+0.9%), but this is still well within the same noise band as the other sizes. We can add this table to the appendix and include this text and a reference in the main body.

Thank you for the per-task breakdown and the A100 decode measurements. The throughput results address my question about the overhead of nonuniform ranks in the tested setting.

My remaining concern is what capabilities survive at these bit budgets. All six downstream tasks ask the model to score predefined answers. At 0.986 bpw, for example, ShamAN-Q improves the six-task average over NanoQuant on Qwen3-4B, but its absolute accuracy is 33.3% on HellaSwag and 55.8% on WinoGrande. I would find a comparison with the unquantized checkpoint particularly informative here. MMLU would broaden the coverage of subject knowledge, and GSM8K with generated answers would test whether the model can still solve multi-step problems, especially around 0.55 bpw.

I would also like to understand the effect on conversation. Since the paper already evaluates a post-trained Qwen3-32B checkpoint, would it be feasible to compare its responses before and after quantization on the same multi-turn prompts? Even a small blinded comparison could show whether it still follows instructions, remembers a constraint introduced in an earlier turn, and gives a coherent answer when the user asks a follow-up.

I appreciate the perplexity gains and the additional experiments. Given the limited evidence on reasoning and open-ended generation at the lowest bit budgets, I maintain my weak-reject recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.