acceptodds
Under review as a conference paper at ICLR 2027

FastKron: Efficient Quantization with Kronecker-Factored Hessians

Abstract

We accelerate a family of algorithms for neural network quantization which utilize a two-sided version of the GPTQ/LDLQ algorithm. Standard GPTQ-style adaptive rounding uses one-sided correlation information derived from input activations. A natural two-sided extension can additionally capture correlations across output channels. It utilizes a general Kronecker-factored approximation of the weight matrix curvature. This approach has been used in BoA and YAQA. But making a concrete algorithmic implementation of this two-sided GPTQ variant is nontrivial. BoA uses a large number of sequential steps, while YAQA improves over the sequential depth but still has quartic total cost. We introduce FastKron, an efficient algorithmic implementation that combines anti-diagonal parallelism with a recursive divide-and-conquer construction. For an weight matrix, FastKron uses sequential steps while reducing the total work from to . Thus, it matches the cubic scaling of GPTQ while exploiting richer curvature information. Moreover, FastKron is modular with respect to both the base quantizer and the Hessian estimator. We also provide practical benchmarks, consider a range of Hessian approximations that FastKron can be used with, and provide an efficient technique to compute these Hessians.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
quantization, gptq, yaqa, kronecker, divide and conquer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_U4cKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

FastKron proposes an efficient implementation of two-sided GPTQ-style adaptive rounding with Kronecker-factored curvature. By combining anti-diagonal parallelism with recursive divide-and-conquer, it reduces computational complexity from quartic to cubic while preserving the original rounding results. The paper also introduces a memory-efficient procedure for computing Kronecker-factored Hessian approximations.

Critical strengths
  1. FastKron reduces the rounding work from quartic to cubic complexity for square matrices, while proving that its recursive and anti-diagonal variants produce the same output as the underlying two-sided procedure.
  2. The benchmarks show that FastKron avoids the rapidly increasing cost of the earlier anti-diagonal implementation and remains within a constant factor of GPTQ’s core rounding time. The paper also presents a recursive Hessian-computation flow intended to reduce fast-memory requirements.
Critical weaknesses
  1. FastKron acknowledges GPTQ-2D (CoRR 2026), which also achieves cubic-time two-sided adaptive rounding, and explains that it uses a different algorithmic technique. However, the paper does not compare the two implementations directly. As a result, the relative runtime, memory use, and practical benefits of FastKron’s recursive construction over GPTQ-2D remain unclear. This limits how confidently I can assess the distinct practical value of the proposed algorithm.
  2. The proposed recursive Hessian-computation flow is not evaluated on an actual model. The paper analyzes its compute and memory complexity, but does not report measured peak memory or end-to-end quantization time for this flow. Those measurements would help establish whether the flow’s theoretical memory trade-off is useful in practice.
Questions for authors
  1. Can the authors clarify the practical and theoretical trade-offs between FastKron and GPTQ-2D (CoRR 2026)? How do they compare in terms of runtime constants, memory requirements, parallel depth, and implementation complexity?
  2. For the recursive Hessian-computation flow, can the authors provide measured peak memory and end-to-end quantization time on representative models, or clarify which parts of the reported memory and compute analysis are theoretical estimates rather than measured results?
3 comments

We thank the reviewer for their feedback. We will address all mentioned weaknesses and questions below.

W1/Q1: Please clarify trade-offs between FastKron and GPTQ-2D.

We have added a new appendix F that compares the two algorithms.

W2/Q2: Please provide end-to-end quantization time and memory.

We have added a new Table 4 at the end of the paper, which provides a realistic model (or at least lower bound, assuming an optimized implementation) of the time and memory requirements of the various flows. (While Table 3 is useful for understanding the theoretical complexity under simplified assumptions, Table 4 is a more realistic version of the same table.) In particular, for Table 4 we plug in values of two actual language models (Llama-3-8B and Qwen3-32B), which shows their time and memory requirements for the flow.

This only compares the flows and therefore demonstrates the advantage of the recursive halving flow. It does not contain the time required for rounding (running one of the FastKron-* algorithms). The rounding time can be extrapolated from the benchmarks. (In the most simple way, plug in an appropriate value for nn and multiply the benchmark time by the number of linear modules.)

For example, FastKron-antidiagonal on 16x16 tiles (used by YAQA) needs 30 minutes on Llama-3-8B and 5 hours on Qwen3-32B, whereas our new FastKron algorithm only needs 30 seconds and 3 minutes, respectively.

We hope that this addresses the reviewer's questions/critique to their satisfaction, so that they can hopefully upgrade their rating to recommend clear acceptance of the paper.

I appreciate the authors' response and the discussion in Appendix F.

While the appendix clarifies the algorithmic differences between FastKron and GPTQ-2D, my main concern about their substantial overlap remains. Both methods achieve the same asymptotic complexity, and the authors acknowledge that GPTQ-2D may have a lower constant factor. Without direct runtime and memory comparisons, the distinct practical advantages of FastKron remain unclear.

Furthermore, the new Hessian-computation results are still based on analytical estimates rather than actual measurements.

Overall, my concerns are not fully resolved. While I am not inclined to champion this paper, I will maintain my original borderline score.

We do not claim that there is a distinct practical advantage over GPTQ-2D, and similarly there is no distinct practical advantage of GPTQ-2D over FastKron. Both achieve the same runtime in theory and practice, developed independently. Also note that we have developed FastKron (and documented it on OpenReview) before GPTQ-2D was made available as a preprint.

In fact, one can see the existence of the independent research effort GPTQ-2D as further evidence of the importance of the problem. Presumably the authors of GPTQ-2D were running into the same computational bottlenecks as we did.

The recursive halving flow is another important ingredient for two-sided GPTQ and is only provided in this paper. While it is true that the numbers in the new table are technically estimates, we want to note that we have verified the model with actual implementations, in particular we ran the recursive halving flow on the mentioned language models and the estimated numbers accurately predict the measured quantization time. We can also provide measurements for the camera ready version.

Reviewer_2Nz7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a mechanism to accelerate a family of post-training quantization methods which utilize a two-sided variant of GPTQ. For context, GPTQ quantizes weights one at a time, while adjusting the other weights to compensate for error in output activations, treating every output channel as being independent and identical. The two-sided variant of GPTQ enables the use of a (approximated curvature) matrix that describes the correlation between output channels. FastKron is a combination of a recursive version and an antidiagonal version of a naive method, and reduces the number of sequential steps from O(mn) to O(m + n), while reducing the total work from O(m^2n^2) to O(mn(m+n)). In other words, FastKron achieves GPTQ-level performance (asymptotically) while enabling the use of curvature information. Empirical evaluation shows FastKron can achieve GPTQ-level performance in terms of wall clock time. The paper additionally discusses FastKron can be applied to multiple algorithms that differ in how they obtain their curvature matrices. The paper proposes a recursive algorithm to reduce the time/memory complexity of the computation of curvature matrices.

Critical strengths

The proposed algorithm, FastKron, can reduce the complexity (in terms of the number of iterations and in terms of the total work) by combining recursion and antidiagonal parallelism. For practitioners who want to use the baseline algorithm but have limited computational resources, FastKron is useful. The proposed recursive halving flow algorithm is novel and interesting.

Critical weaknesses

This is a borderline paper for me. I think the paper can be improved in the following aspects.

  1. The research problem is unclear: The abstract starts with “we accelerate a family of algorithms…”. Why is this family of algorithms interesting to a potential reader? For example, does this family significantly outperform all other non-GPTQ-variant post-training-quantization algorithms? Does this family’s performance advantage (over other non-GPTQ-variant PTQ methods) come with the downside that it is impractical to run for LLMs? How impractical? Does it take days, months, or years to run a naive version? Does it take days, months, or years to run a most recent baseline (e.g. YAQA)? Establishing the research problem (in the abstract/introduction section) can help readers understand the significance of the proposed algorithm.
  2. The runtime bottleneck is unclear: I like the breakdown in section 2.6, but I think this incredibly useful insight should show up at the beginning of the paper, and the metrics used for reporting should be wall clock time because it helps the readers understand what the bottleneck of the full algorithm is. In particular, if the authors can show accumulating the Hessian takes x% of the end-to-end wall clock latency, computing the inverses and Cholesky decomposition takes y%, the core algorithm takes z%, and propagating calibration data takes a% of the time, and z is larger than x, y, and a, then it is immediately clear that the core algorithm should be accelerated and the FastKron algorithm is incredibly useful (by Amdahl’s law). The paper does mention on line 304 that the core algorithm is the bottleneck (without FastKron), but showing profiling results on real hardware would make this more convincing. In the paper’s current state, it is hard to tell which of Section 2 and Section 4 is the primary contribution.
  3. Over-reliance on big-O: Big-O is useful, but profiling results on real hardware (execution time, peak memory usage) are useful in a different way. Consider expanding Section 4/Table 3 to report both metrics. If some baseline methods are impractical, try to estimate how impractical they might be on real hardware. For example, how much memory would it (if you have infinite memory) take to run? How long does it take (assuming you have unlimited computational resources) to run the algorithm?
Questions for authors

See the weakness section above.

4 comments

We thank the reviewer for their feedback. We address all mentioned weaknesses below, in slightly varied order.

W1a: Unclear research problem. Why is this family of algorithms interesting?

This family of algorithms is among the most popular quantization methods: GPTQ has been one of the most popular quantization algorithms for multiple years now, and there are multiple recent papers (e.g. YAQA, BoA, WaterKron, GPTQ-2D) that all use the extended two-sided GPTQ algorithm. In the "standard" GPTQ algorithm, all parts of the pipeline (Hessian accumulation, inverse computation, GPTQ rounding algorithm, forward propagation) have cubic complexity, see section 2.6 of our paper. With the "new" two-sided GPTQ algorithm (used in e.g. YAQA and the other mentioned works), suddenly the rounding algorithm implementation has quartic complexity, which makes it the dominant cost. Thus it is a natural research question if the new algorithm can be implemented with cubic complexity too, matching standard GPTQ.

W3: Can you expand section 4 / table 3 to also report time/memory estimates on real hardware?

We have now added a new Table 4 at the very end of the paper, which uses a more detailed and realistic model to predict flow times (compared to Table 3, which is better for understanding the theoretical complexity with simplified assumptions). We also partially tested that the predictions match real implementations. It assumes a good/optimized implementation though, so it can also be seen as a lower bound. It also assumes a single iteration of Frobenius/FlipFlop/Marginal estimator, multiple iterations would take longer. (But they would hurt the recursive halving flow the least, since the additional computation there only happens in the leaves, so it would make the recursive halving flow look even more favorable.)

In particular, in the table, we plug in the values for the models Llama-3-8B and Qwen3-32B, which shows realistic quantization time and memory usage for quantizing those models.

W1b and W2: Does it take days, months, or years to run a recent baseline implementation, e.g. YAQA? What's the share of the rounding algorithm in percent?

In the terminology of our paper, YAQA uses the FastKron-antidiagonal implementation of the core rounding algorithm, and the "simultaneous Hessian flow" to accumulate the (Frobenius) Hessian approximation. We will only assume a single iteration of the Hessian estimator, and we will assume k=219k = 2^{19} calibration tokens (less than what YAQA does, but matches the WaterKron experiments).

For this setting, we can look at Table 4 to see the time it takes to compute the curvature matrices A,BA,B for e.g. Llama-3-8B, which is about 20min. Then running FastKron-antidiagonal on 16x16 tiles (which is essentially what YAQA does), it costs about another 30 minutes just to run the rounding algorithm. So the rounding algorithm takes up 60% of the total time. This share gets worse the bigger the matrices get, due to the quartic vs cubic growth. For example, for Qwen3-32B, Table 4 lists a flow time of about 2h, but rounding with FastKron-antidiagonal on 16x16 tiles will take about another 5h, occupying more than 70% of the total time. (Theoretically, having models with larger and larger matrices, the share of the rounding algorithm will approach 100%, because of O(n4)O(n^4) versus O(n3)O(n^3) growth.)

In comparison, running FastKron rounding will take less than 30 seconds for Llama-3-8B, about 2% of the total quantization time. For Qwen3-32B, it would be about 3min, again about 2% of total time. (And theoretically, having models with larger and larger matrices, the share of the FastKron rounding algorithm will stay small, because it has O(n3)O(n^3) complexity like the components of the flow.)

Moreover, since YAQA uses the "simultaneous Hessian flow", we can see that it requires a lot of memory to store A,BA,B for every weight matrix. So for Llama-3-8B, one would need a GPU with 80GB of VRAM or more. (Table 4 says 63GB, but there is a little overhead which is not contained in the formula.) Qwen3-32B can't be quantized with this flow on a single GPU. In comparison, with our "recursive halving flow", one could quantize Llama-3-8B on a GPU with 24GB of VRAM and Qwen3-32B on a GPU with 80GB of VRAM.

We thank the reviewer once again for their feedback and we hope that we have addressed the critique to their satisfaction. We would be grateful if the reviewer could re-evaluate their rating, and consider recommending acceptance of the paper.

AuthorsOpenReview ↗

Sidenote on YAQA source code

[Sidenote regarding the share of time that the rounding takes: The YAQA paper itself used multiple iterations of the Hessian estimator (which multiplies the total flow cost by the number of iterations), and uses a lot more calibration tokens (also multiplying the flow time accordingly). So with their hyperparameters, FastKron-antidiagonal will also just make up a small percentage of the total quantization time (due to an unusually large amount of calibration tokens and multiple iterations of the Hessian approximator, they say themselves that less calibration data works without hurting quality, and the WaterKron paper shows that 2 iterations of the Hessian approximator are enough). Also both their flow implementation and FastKron-antidiagonal implementation are suboptimal, which puts additional noise into the ratio if you take YAQA's source code.]

Thank you for your response.

Your response has addressed W1 and W3. W2 has not been fully addressed, especially the following concern: "In particular, if the authors can show accumulating the Hessian takes x% of the end-to-end wall clock latency, computing the inverses and Cholesky decomposition takes y%, the core algorithm takes z%, and propagating calibration data takes a% of the time, and z is larger than x, y, and a, then it is immediately clear that the core algorithm should be accelerated and the FastKron algorithm is incredibly useful (by Amdahl’s law)... In the paper’s current state, it is hard to tell which of Section 2 and Section 4 is the primary contribution."

In the LLaMA-8B example, the authors state that the core algorithm takes 30 minutes, and propagating calibration data plus accumulating the Hessian takes 20 minutes, but it is unclear how long it takes to compute the inverses.

Similarly, in the Qwen3-32B example, the authors state that the core algorithm takes 5 hours, and propagating calibration data plus accumulating the Hessian takes 2 hours, but it is unclear how long it takes to compute the inverses.

Therefore, the values of x, y, z, and a remain unknown. Without them, it is difficult to evaluate the practical contributions of Sections 2 and 4.

I will keep my current rating unchanged at this stage and reconsider it after discussing the paper with fellow reviewers and the AC.

The time it takes to compute the inverses is negligible, similar to FastKron it only makes up a tiny fraction the total time, in fact even less than FastKron because there are highly optimized inversion kernels available.

The only relavant costs are the forward/backward propagation and Hessian accumulation, together called the "flow", and the rounding algorithm. The costs of both have been discussed in detail, including concrete examples of models.

Reviewer_HUtyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FastKron, which proposes adaptive rounding like GPTQ to effectively capture correlations between input activations and across output dimensions unlike GPTQ which only captures correlations along input dimensions. FastKron combines anti-diagonal parallelism with a recursive divide and conquer scheme to reduce arithmetic complexity from O(n^4) to O(n^3).

Critical strengths
  • The paper is clearly written and easy to follow. The authors introduce clear and meaningful algorithmic improvements to GPTQ.
  • The paper reduces the arithmetic complexity for two-sided adaptive rounding from quartic to cubic complexity for square matrices. This is a significant improvement.
  • The acceleration introduced preserves the underlying algorithm as shown in Theorem 2.1.
  • The method is orthogonal to the choice of base quantizer and method to construct the input side and output side geometry. This potentially will be useful for future work.
Critical weaknesses
  • The empirical evaluations of the end to end quantization for downstream tasks is something that is missing in the paper. It is hard to assess any practical benefit of the method compared to existing methods on representative LLMs using the same quantizer.
  • It's not clear where the benefit of richer curvature information becomes more significant. Maybe for low bit precision such as 3-bit or 2-bit?
  • Its not clear which approximation for the curvature estimator is ideal? Is the simple marginal approximation sufficient if FastKron is used?
Questions for authors

Please address the above weaknesses.

1 comment

We thank the reviewer for their feedback and for pointing out the strengths of the submission.

All the mentioned weaknesses concern evaluation of downstream quality of the quantized model, so we would like to clarify that this paper, intentionally, only concerns the compute (and memory) efficiency of the quantization method. This is because we did not invent a new method itself, but we accelerate an existing method. The only empirical experiments we perform are therefore (efficiency) benchmarks. For evaluation on downstream tasks, we refer to the other works that utilize this algorithm (with suboptimal implementations) and perform evaluation on downstream tasks, see the citations in section 3. To say it plainly: If we ran downstream evaluations and chose the same hyperparameters (curvature approximator, scaling factors, damping factor, base quantizer) as these other works, then we would get the exact same tables that are already reported in the other works.

We address all mentioned weaknesses separately below, referring to relevant other works.

W1: Empirical evaluation on downstream tasks is missing.

As we just discussed, the downstream empirical evaluation is done in other papers, concretely:

  • YAQA ("Model-preserving adaptive rounding") evaluates the method with the Frobenius curvature approximator on the Fisher information matrix
  • BoA evaluates the method with a custom local curvature approximator for the attention module
  • WaterKron compares FlipFlop, Frobenius, and Marginal curvature approximators for the Fisher information matrix

W2: It is not clear where the benefit of richer curvature becomes more significant. Maybe for low bit precision such as 3-bit or 2-bit?

Judging from the experimental results of the mentioned other papers, and from our own experience, it improves quantization quality (e.g. KL divergence on the calibration dataset) across all precisions. Or in other words, it allows you to quantize with fewer bits while giving the same output quality.

W3: It is not clear which approximation for the curvature estimator is ideal. Is the simple marginal approximation sufficient if FastKron is used?

The question which curvature estimator is best is the central question of the "WaterKron and FlipFlop Hessian" paper, and is the reason why the FlipFlop estimator has been developed in the first place. This paper gives a theory and empirical results, supporting the argument that FlipFlop seems to be the best curvature estimator, at least in the setting that is studied in that paper (which uses unbounded integers).

We hope that this clarifies both the questions of the reviewer as well as the focus of the paper and why, intentionally, we don't list downstream evaluations. We would be grateful if, in light of these clarifications, the reviewer could reconsider their rating and recommend the paper for clear acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.