acceptodds
Under review as a conference paper at ICLR 2027

Bits Under ZK-LLM: Evaluating Zero-Knowledge-Friendly Quantization for Verifiable Private LLM Inference

Abstract

Zero-knowledge (ZK) proofs are emerging as a promising approach for enabling private, verifiable LLM governance and auditing, where regulators, users, and auditors need to verify claims about training-data usage or LLM inference-time behavior, while model providers must protect proprietary model parameters. However, despite the growing interest in ZK-LLMs, the understanding of ZK-friendly quantization remains limited. This gap matters because in the ZK setting, quantization directly shapes the arithmetic structure, constraint complexity, and proving cost of ZK inference. ZK protocols operate over finite fields and incur costs that depend heavily on the number and type of arithmetic operations, nonlinearities, and lookup constraints. Understanding ZK-friendly quantization is therefore essential for making ZK-LLMs practical. In this work, we present the first systematic study of ZK-friendly quantization for LLMs. We first formalize the definition of ZK-friendly quantization, capturing the properties required for ZK proof generation. We then evaluate nine language models, including Qwen2.5-14B and the mixture-of-experts model Qwen3-30B-A3B, across a broad design space of weight, activation, and nonlinear lookup table precision. Our results show that activation precision is substantially more sensitive than weight precision, while nonlinear lookup approximations can become the dominant source of utility degradation. Also, we identify RMSNorm inverse-square-root lookups as a recurring bottleneck in several large models and recover near-baseline utility by selectively increasing precision only at the bottleneck. Finally, we show that reducing bit-width or lookup-table size does not necessarily yield proportional end-to-end proving savings, showing that conventional low-bit quantization heuristics do not directly translate to ZK proving efficiency and motivating operator-aware precision selection.

Primary area:
datasets and benchmarks
Keywords:
LLM, Quantization, Zero-knowledge proof

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_f8cWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a framework for exploring weight, activation, and nonlinear lookup-table precision in LLMs intended for zero-knowledge inference. It evaluates nine models using a PyTorch quantize–dequantize simulator and measures selected proving configurations using modified DeepProve backends.

The central findings are that activations are more sensitive than weights under the evaluated quantization policy; uniformly discretized inverse-square-root tables can severely degrade RMSNorm-based models; selectively increasing RMSNorm table precision can recover utility; and reducing nominal precision does not necessarily reduce total proving time. An additional case study benchmarks a proof of a quantized Min-K% membership-inference statistic.

Critical strengths
  1. Quantization optimized for conventional inference need not optimize proof generation. Separating numerical precision, arithmetic-circuit structure, lookup-table size, and total prover cost is useful. The paper’s emphasis on measuring realized costs rather than assuming that fewer bits automatically imply faster proofs is appropriate.
  2. The nonlinear-operator diagnosis is informative.
  3. The paper includes KL divergence, top-token overlap, low-probability-position overlap, and full ARC-Easy and MMLU evaluations. This is valuable because preserving perplexity does not guarantee preservation of individual predictions or audit statistics. The released downstream results support the claim that the selected Full16 configuration retains much of the reference performance in the simulator.
Critical weaknesses

The RMSNorm failure is convincingly diagnosed for the chosen implementation: uniformly discretizing a range extending to roughly 65,461 with a B16 table gives spacing close to one, causing many small inputs to map to zero. However, this establishes a weakness of that range-selection and uniform-grid design, not that RMSNorm generally requires a 24-bit input table in ZK inference.

Relevant alternatives include static percentile calibration, finer static scale granularity, outlier smoothing, range reduction, segmented tables, and proving normalization through algebraic constraints on a supplied witness. Several of these remain compatible with a fixed proof circuit. In particular, DeepProve’s documented normalization approach avoids explicitly tabulating inverse square root in the manner evaluated here.

Similarly, the conclusion that activations generally need 12–14 bits is based primarily on static per-tensor max calibration using eight windows. Appendix G already shows that changing activation-scale selection can substantially recover W12A12 utility. Although dynamic calibration may introduce proof overhead, static per-channel or grouped alternatives should be compared before presenting the observed precision threshold as broadly characteristic of ZK-friendly quantization.

The definition also needs care: runtime-dependent values do not require a runtime-dependent circuit. Computing a scale or selecting a branch can be encoded in a fixed circuit, albeit at additional cost. Static calibration is a potentially efficient design choice, not a universal prerequisite for ZK compatibility.

Questions for authors
  • Can the authors provide end-to-end simulator-versus-prover numerical validation, including integer requantization, normalization, attention, and the exact LUT input/output precisions?

  • How do the activation and RMSNorm findings change with stronger static calibration and normalization-aware proof constructions, rather than only uniformly enlarging the LUT?

  • What is the measured proving overhead of selectively promoting RMSNorm precision on a model where that intervention is needed for utility?

  • How stable is the approximately 1200× projected speedup when floating-point proofs are measured at multiple tile widths and fixed overhead is accounted for?

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer f8cW [part 1]

We thank the reviewer for the insightful feedback and valuable comments.

[W1/Q2] RMSNorm’s failure analysis

We agree that increasing to 24 bit-width is not the only fix. The utility can be restored with a normalization-aware construction and RMSNorm at 16 bit-width. Hence, we evaluated three more alternatives: percentile calibration, range reduction, and witness-checked normalization (DeepProve’s normalization approach). Other methods (e.g., finer granularity, outlier smoothing, segmented table) were not evaluated separately. Range reduction is itself a segmented table, and the other two cannot shrink the input range of the RMSNorm LUT.

We used the four large models, keeping every other setting the same as in Table 5 in the paper. Below are the full results of PPL ratio compared to the W16A16 linear-only baseline (WikiText-2). Other LUTs use 16 bits.

Table 1. PPL ratio of RMSNorm constructions

RMSNorm constructionTable sizeLlama-3.1-8BQwen2.5-7BQwen2.5-14BQwen3-30B-A3B
Uniform grid (Full B16)2^161126.81496.932.46131.3
Uniform grid (RMS B24)2^241.00020.99870.99971.0014
Percentile range, P99.992^161059.21386.430.4775.94
Percentile range, P99.92^161034.11552.429.2527.70
Percentile range, P992^16divergeddivergeddivergeddiverged
Range reduction2^161.00050.99881.00081.0017
Witness-checked2^161.12461.10571.21121.0217

As shown in the Table above, range reduction fully recovered utility while keeping the table at 16 bits, whereas percentile range selection was not effective. As shown in the paper (R3, Section 5.1), most inputs fall into the lowest few entries of the RMSNorm LUT, yet the LUT must still cover a few tokens with massive activations. P99.9 and P99.99 calibration cover almost the same range as the uniform B16 grid, so they give no finer resolution for small inputs. P99, on the other hand, resolves small inputs but cannot cover the large ones, so PPL diverges. For range reduction, the table only needs to cover [1, 4), because we rewrite each input as m·4^k with 1 ≤ m < 4. That is, an input outside this range is divided (or multiplied) by 4 until it falls inside the table. The value is then read from the table and scaled back with a simple shift. Thus, no part of the input range is lost, and small and large inputs get the same relative precision. Witness-checked normalization lets the prover supply the inverse square root and the circuit only checks it. This check needs its own table, and at 2^16 entries it rounds the outputs too coarsely.

We confirm that range reduction is an effective alternative to B24. It keeps proving time and memory at the B16 level. It, however, needs more lookups per token. B24 is simple to implement and the added cost is fixed regardless of model size. However, it still uses more memory and proving time than B16. We therefore refine our conclusion slightly.

[W2/Q2] Activation precision under stronger calibration

For alternative static calibration, we repeated the setting of Table 9 (in the paper) at W16A16, W16A10 and W12A12 with four large models. We changed how activation scales are chosen.

Table 2. PPL ratio to the BF16 model under different activation calibrations (WikiText-2)

PPL ratioConfigPer-tensor (paper)8× dataP99.99Per-channelPer-group
Qwen2.5-7BW16A161.0031.0019224.21.6361.004
W12A121.1691.2238765.41.6341.011
W16A101.4941.7428538.31.6331.036
Llama-3.1-8BW16A161.0031.00261.111.2351.008
W12A121.7421.72960.881.2351.013
W16A101.8871.88260.641.2341.027
Qwen2.5-14BW16A161.0011.00116.021.5321.005
W12A121.0681.06015.821.5291.011
W16A101.2031.26315.321.5281.026
Qwen3-30B-A3BW16A160.9971.00127.471.0031.000
W12A121.0311.05226.931.0021.000
W16A101.2821.29427.191.0010.993

Static per-group calibration recovers most of the loss, while 8x more calibration data was not effective. We did not measure the proving cost as group scales are fixed constants, so they remain compatible with a fixed circuit. Hence, we slightly revise the claim that with static per-group calibration, about 10 bits activation successfully recovers the lost utility.

AuthorsOpenReview ↗

Rebuttal to Reviewer f8cW [part 2]

[W3] Definition of static compilability

Yes, we agree with you that a static circuit can compute runtime-dependent values in ZK, which would, yet, incur high prover overhead. For example, computing the scale dynamically would require calculating the maximum and minimum values of the activations in every layer, which are non-linear functions applied to every element of the activations. The cost would be at least as expensive as the non-linear activation functions, which contribute to the majority of the prover time in zkLLMs. Therefore, the cost of dynamic scales would be at least 2x more than the static calibration. This is a very conservative estimation ignoring the additional cost of applying the quantization after computing the scales dynamically. We will refine Definition 3 to clarify this and emphasize our focus on efficiency.

[Q1] Simulator vs. integer execution

For GPT-2 Small, we measured the perplexity of DeepProve’s integer execution and compared it with the simulator’s perplexity, with all LUTs at 16 bits. Each number is relative to its own floating-point model.

Table 3. PPL ratio of DeepProve’s integer execution and of the simulator

PPL ratioWikiText-2 IntegerWikiText-2 Simulator
W16A160.9901.003
W12A160.9971.004
W8A161.3051.315
W16A121.0101.019
W16A101.2541.283
W16A810.00154.412

The results show that integer execution and simulation exhibit consistent overall trends across quantization settings. Both preserve near-baseline utility at W12A16 and W16A12, while reducing weight or activation precision further leads to substantial perplexity degradation. In particular, both pipelines exhibit severe degradation at W16A8, although the simulator predicts a larger loss (54.412 versus 10.001). These results provide empirical support for using the simulator to assess the relative utility impact of quantization configurations.

[Q3] Proving cost of the RMSNorm fix

Table 4. Proving time and peak memory of the RMSNorm fixes

Proving time, peak memoryGPT-2 SmallGPT-2 MediumGPT-2 LargeTinyLlama-1.1BLlama-3.1-8B (1 block)
Baseline80.3 s, 20.3 GiB170.1 s, 34.8 GiB432.4 s, 122.5 GiB407.7 s, 90.4 GiB275.1 s, 85.7 GiB
Selective B24 (RMSNorm only)+17.1 s (+21.3%), 22.2 GiB+17.6 s (+10.4%), 36.6 GiB+15.7 s (+3.6%), 124.4 GiB+18.4 s (+4.5%), 92.3 GiB+16.6 s (+6.0%), 87.1 GiB
All LUTs at B24+177.7 s (+218.6%), 95.0 GiB+333.3 s (+189.6%), 169.5 GiBout of memory (> 245 GB)--
Range reduction (B16)+1.9 s (+2.4%), 20.3 GiB+6.2 s (+3.7%), 35.0 GiB−6.7 s (−1.5%), 122.3 GiB-10.1 s (-2.5%), 90.2 GiB+3.1 s (+1.1%), 85.2 GiB

Selective B24 uses a B24 table only for RMSNorm. Other operations, GeLU (or SiLU) and the exponential, use B16 LUTs. The larger the model, the smaller the relative increase in cost. This is because the added proving cost is proportional to the LUT size and is nearly fixed. However, using B24 tables for all nonlinear operations makes the proving cost very large. This shows that selective bit width increment is still important. Range reduction costs almost the same proving time and memory as the B16 baseline. The additional lookup queries are very few compared with the other lookups in the whole GPT-2 model, so they did not significantly affect the proving cost.

In conclusion, range reduction is efficient. It can achieve B24-level utility at B16 cost and is less sensitive to the input range. Still, B24 remains the simplest alternative, and its fixed cost becomes negligible for large models.

[Q4] Stability of the FP vs. Integer comparison

An FP proof cannot cover a whole projection (e.g., 5,120 output columns) at once (Section 5.2.1). Thus, we proved a tile of 8 output columns and multiplied its proving time by (total columns / 8). We agree that the fixed overhead is not separated, so we proved FP tiles of 4, 8, 16, 32 and 64 output columns. The time grows linearly with the number of columns and the fixed cost is about 11-12% of the 8-column proving time (Table 6). For the seven projections of one Qwen2.5-14B block, the two ways of extrapolating give the following FP/integer ratios.

Table 5. FP/integer proving-time ratio for one Qwen2.5-14B block

FP extrapolationFP / integer
Paper's setting1,179×
Fixed cost separated1,067×

(Both numbers were measured again in one session on the same server)

Table 6. FP proving time of one tile

Input dimension4 cols8 cols16 cols32 cols64 colsFixed cost
5,120 (Q, K, V, O, gate, up)4.859.8419.3837.4772.021.04 s
13,824 (down)14.2023.8645.6688.95–2.91 s

Separating the fixed cost reduces the ratio by about 10%, and the gap still remains three orders of magnitude. We will revise the numbers in the paper.

Reviewer_nDM4OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies quantization for zero-knowledge proofs of LLM inference. It defines two properties of ZK-friendly quantization: ZK efficiency, measured as relation size relative to floating-point inference; fidelity to the original model. It then builds a PyTorch framework that emulates bounded-integer linear layers through quantize-dequantize operations and replaces nonlinear operators (GeLU/SiLU, normalization inverse square root, softmax exponential and reciprocal) with fixed lookup tables (LUTs). On the utility side, the paper evaluates nine models, up to Qwen2.5-14B and the MoE model Qwen3-30B-A3B. On the cost side, proving time is measured with an extended version of DeepProve.

Critical strengths

The question is practically relevant, and the study is broad. Prior ZK-LLM systems validate their numerical choices on one small model. A study that tests which precision choices hold up across scales and architectures, including an MoE model, is useful to people building these systems.

Proving cost is measured with a real prover, not only estimated. Showing that bit-width barely affects proving time within the field range, and that LUT cost is driven by query count until the table itself dominates (Table 12), corrects a naive assumption that “fewer bits means cheaper proofs.” This is a useful message, even if it follows from how the backend works.

Critical weaknesses

There are some minor flaws in the experiments: Proving cost is measured end-to-end only on GPT-2 models, where the RMSNorm bottleneck does not occur, and for 14B only on linear projections. The 1,200× floating-point comparison extrapolates from 8 output columns and uses a different backend.

In addition, the observation that LUT proving cost is often driven by the number of queries rather than the table size has been discussed in recent work (https://eprint.iacr.org/2026/1453.pdf). The authors should cite and discuss this paper, and clarify what they add beyond it.

Questions for authors

Please specify the proof system underlying DeepProve in your experiments: the field, the polynomial commitment scheme, and the lookup argument used for the LUTs. Your cost conclusions depend on these choices.

The cost conclusions depend on these choices. In particular, would the finding that bit-width barely affects proving cost still hold with a smaller field (e.g., a 31-bit field such as BabyBear) or a different proof backend?

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer nDM4 [part 1]

We thank the reviewer for the helpful comments. We address the weaknesses and questions below:

[W1] Robustness of the FP vs. Integer comparison

Proving cost on RMSNorm models. While GPT-2 uses LayerNorm, it also computes an inverse square root. Therefore we applied the same LUT inside each LayerNorm. The proving cost of the LUT, which depends on the table size, the number of lookups, and the number of normalizations, is the same for LayerNorm and RMSNorm. We will clarify that in the paper.

Regarding the proving cost with B24 RMSNorm LUT, Table below shows the results.

Table 1. Proving cost with a selective B24 RMSNorm LUT

ModelProving Time OverheadPeak Memory
GPT-2 Small+17.1 s (+21.3%)22.2 GiB
GPT-2 Medium+17.6 s (+10.4%)36.6 GiB
GPT-2 Large+15.7 s (+3.6%)124.4 GiB
TinyLlama-1.1B+18.4 s (+4.5%)92.3 GiB
Llama-3.1-8B (1 block)+16.6 s (+6.0%)87.1 GiB

The results show that B24 RMSNorm introduces a nearly constant proving-time overhead of approximately 16-18 seconds, regardless of model size. Consequently, its relative overhead generally decreases as the proving workload increases, from 21.3% for GPT-2 Small to 3.6% for GPT-2 Large. The additional peak memory usage is also modest (approximately 1.4-1.9 GiB). These results demonstrate that selectively increasing RMSNorm precision to B24 is practical and incurs limited additional proving cost, particularly for larger models.

Extrapolation from 8 output columns. An FP proof cannot cover a whole projection (e.g., 5,120 output columns) at once (Section 5.2.1). Thus, we proved a tile of 8 output columns and multiplied its proving time by (total columns / 8). We agree that the fixed overhead is not separated, so we proved FP tiles of 4, 8, 16, 32 and 64 output columns. The time grows linearly with the number of columns and the fixed cost is about 11-12% of the 8-column proving time (Table 3). For the seven projections of one Qwen2.5-14B block, the two ways of extrapolating give the following FP/integer ratios.

Table 2. FP/integer proving-time ratio for one Qwen2.5-14B block

FP extrapolationFP / integer
Paper's setting1,179×
Fixed cost separated1,067×

Table 3. FP proving time of one tile

Input dimension4 cols8 cols16 cols32 cols64 colsFixed cost
5,120 (Q, K, V, O, gate, up)4.859.8419.3837.4772.021.04 s
13,824 (down)14.2023.8645.6688.95–2.91 s

Separating the fixed cost reduces the ratio by about 10%, and the gap still remains three orders of magnitude. We will revise the numbers in the paper.

Different backends. The FP proofs use a general-purpose prover (gnark’s Groth16), while the integer proofs use DeepProve. This is because FP arithmetic was implemented as R1CS in Ernstberger et al. (2025), which is not compatible with the backend of DeepProve based on sumcheck and GKR. To provide a fair comparison, we further tested the same integer projections with the same Groth16 prover. It shows that FP proving is about 40 to 50 times slower than integer proving. This is consistent with the overhead reported in Ernstberger et al. (2025), where it takes 64 constraints to implement 1 FP multiplication. However, when implemented in sumcheck-based backends, FP arithmetic does not yield efficient special protocols, such as the sumcheck protocol for matrix multiplications with O(n^2) prover. This is where the additional speedup comes from in DeepProve.

Either way, FP proving is impractical for zkLLM inference, and all existing zkLLM schemes rely on quantization. We do not claim this as a new finding in our paper, but demonstrate this fact to show the necessity of ZK-friendly quantization. We will clarify this in the paper.

AuthorsOpenReview ↗

Rebuttal to Reviewer nDM4 [part 2]

[W2] Celer related work

Thanks for pointing this out. We will cite Celer in the paper. Qu et al. show that when the number of lookups exceeds the table size, the proving cost of existing lookup arguments is largely affected by the lookup queries. We study which table precision an LLM needs for utility and what it costs in a ZK proof. In our study, we test with different sizes of tables and the size itself dominates the proving cost (Table 12 in Appendix I).

[Q1] Settings of the proof system

DeepProve works over a 254 bit BN254 field, proves layers with sumcheck and lookups with LogUP-GKR, and commits with HyperKZG. We will add details to the paper.

[Q2] Smaller fields and other backends

BabyBear, DeepProve and zkAgent. Not in general. In a 254-bit field, the products and sums of a layer can fit so the number of constraints does not change with the bit width. However, in a 31-bit field such as BabyBear, the numbers may overflow after a matrix multiplication. For example, the sum of products of 16-bit values will be larger than the field size, whereas 8-bit values can fit directly without extra constraints. In such cases, lower bit widths can reduce the proving cost significantly. By the time of our submission, there were only two zkLLM backends that produced end-to-end proofs for models with real weights: DeepProve and zkAgent, and both use BN254 with 254-bit field and sumcheck. Earlier works mostly focus on implementing specific layers of LLMs and some were tested on synthetic data. Therefore, we were not able to test our design on other backends, especially those over smaller fields. We will clarify this in the paper.

FlashProve. After our submission, Anthropic released FlashProve [1], which shares some similar techniques on quantization, justifying the necessity of our study. We inspected their implementation, which uses 12-bit activation integer quantization with power-of-two block scales on Llama 4 Maverick, a 400B MoE model. The ZK backend is over the Goldilocks field of 64 bits, avoiding overflows after a matrix multiplication. Notably, FlashProve uses lookup tables of size 2^16 for all non-linear layers except RMSNorm, possibly due to its sensitivity as we discovered in our paper. For RMSNorm (the reciprocal of square-root step), instead of a large lookup table of size 2^24, they use auxiliary witnesses to compute the square-root and reciprocal, together with range proofs realized using lookup tables. The dominating cost is roughly 7 multiplications and 41 lookups from B8 lookup tables per computation. Using our approach, it would cost 1 lookup from the B24 lookup table. We expect that the prover time of our approach will be faster. Unfortunately, the FlashProve implementation does not support larger lookup tables, so we could not directly measure the performance.

We also don’t have enough time to measure the perplexity of FlashProve and our approach of using the B24 lookup table for RMSNorm. This would require importing their quantized weights and scales to our framework, and implementing their RMSNorm computations. We will add an additional comment if any new result becomes available.

[1] R. Rinberg, A. Allain, A. Chiesa, M. Christ, R. Dalal, G. Fenzi, B. Mayer, and G. Weissenberg, “FlashProve: An Open-Source Library for Provable Inference,” Anthropic Technical Report, November 2026.

Reviewer_FD3pOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies quantization for zero-knowledge-verifiable LLM inference, examining the trade-off between model fidelity and proving cost. It develops a PyTorch-based evaluation framework and measures selected configurations using DeepProve. Across nine language models, the authors find that activation precision is more sensitive than weight precision, identify RMSNorm inverse-square-root lookups as a recurring utility bottleneck, and show that lower numerical precision does not necessarily translate into proportional proving savings.

Critical strengths
  • Relevant and useful research question. The paper examines quantization from the perspective of proof efficiency rather than conventional inference efficiency. This distinction is practically important because reducing bit-width does not necessarily simplify the computation represented by a proof system.

  • Informative operator-level diagnosis. The operator-wise ablations provide useful evidence connecting utility degradation to RMSNorm lookup approximation. Selectively increasing its precision recovers near-baseline utility in several models, offering an actionable alternative to uniformly increasing all LUT precisions.

  • Broad and complementary evaluation. The study covers nine models, including an MoE architecture, and evaluates perplexity, output-distribution fidelity, and downstream accuracy. Actual proving measurements complement the utility experiments and reveal trade-offs that would be missed by evaluating quantization accuracy alone.

Critical weaknesses
  • The connection between large-model utility and proving cost remains incomplete. The main utility findings concern modern models with RMSNorm, whereas full-model proving experiments primarily use smaller GPT-2 models. Consequently, the proposed selective B24 RMSNorm configuration is not evaluated end to end on a representative affected model. Moreover, QDQ validation focuses on local projections rather than the complete integer execution pipeline, limiting the strength of the resulting deployment recommendations.

  • The generality of the findings requires clearer qualification. The main utility experiments rely on static per-tensor calibration and uniformly spaced LUTs, while proving measurements primarily use one backend. It remains unclear how much the observed precision requirements depend on calibration ranges, LUT discretization, or backend implementation choices. Comparisons with alternative static calibration and LUT designs would strengthen the broader design recommendations.

Questions for authors
  • How sensitive are the main findings to calibration data and LUT construction? Which proving-cost conclusions are expected to generalize beyond DeepProve?

  • Can you provide a representative block-level or full-model measurement combining utility and proving cost for the selective B24 RMSNorm configuration, together with numerical validation against the actual integer execution?

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer FD3p [part 1]

We greatly appreciate the reviewer's constructive and thoughtful feedback.

[W1/Q2-1] Utility and proving cost measurement for B24 RMSNorm

Utility. Regarding the utility of the RMSNorm fix, we evaluated the impact of selectively increasing the RMSNorm lookup-table precision to B24 while keeping all other nonlinear operations at B16. We report the perplexity (PPL) ratio relative to the W16A16 linear-only baseline on WikiText-2.

Table 1. PPL ratio with a selective B24 RMSNorm LUT

ModelPPL Ratio (B24 RMSNorm)
Llama-3.1-8B1.0002
Qwen2.5-7B0.9987
Qwen2.5-14B0.9997
Qwen3-30B-A3B1.0014

The results demonstrate that B24 RMSNorm preserves model utility nearly perfectly, with PPL ratios ranging from 0.9987 to 1.0014 (within 0.14% of the baseline). In contrast, using B16 uniformly for RMSNorm results in substantial perplexity degradation, with PPL ratios of 1126.8, 1496.9, 32.46, and 131.3 for Llama-3.1-8B, Qwen2.5-7B, Qwen2.5-14B, and Qwen3-30B-A3B, respectively. These findings show that selectively increasing RMSNorm precision to B24 effectively mitigates approximation errors while maintaining baseline-level model utility across different architectures and model sizes.

Proving cost. To quantify proving overhead, we evaluated selective B24 lookup tables for RMSNorm across different model sizes, while keeping other nonlinear operations at B16, as shown in the table below. Note that GPT-2 uses LayerNorm, which also computes an inverse square root, so we applied the same LUT inside each LayerNorm. The proving cost of the LUT, which depends on the table size, the number of lookups, and the number of normalizations, is the same for LayerNorm and RMSNorm. The RMSNorm models, TinyLlama-1.1B and Llama-3.1-8B, confirm that the B24 LUT adds the same fixed cost as in GPT-2, regardless of the model family.

Table 2. Proving cost with a selective B24 RMSNorm LUT

ModelProving Time OverheadPeak Memory
GPT-2 Small+17.1 s (+21.3%)22.2 GiB
GPT-2 Medium+17.6 s (+10.4%)36.6 GiB
GPT-2 Large+15.7 s (+3.6%)124.4 GiB
TinyLlama-1.1B+18.4 s (+4.5%)92.3 GiB
Llama-3.1-8B (1 block)+16.6 s (+6.0%)87.1 GiB

The results show that B24 RMSNorm introduces a nearly constant proving-time overhead of approximately 16-18 seconds, regardless of model size. Consequently, its relative overhead generally decreases as the proving workload increases, from 21.3% for GPT-2 Small to 3.6% for GPT-2 Large. The additional peak memory usage is also modest (approximately 1.4-1.9 GiB). These results demonstrate that selectively increasing RMSNorm precision to B24 is practical and incurs limited additional proving cost, particularly for larger models.

[W1/Q2-2] Numerical validation against the complete integer execution pipeline

For GPT-2 Small, we measured the perplexity of DeepProve’s integer execution and compared it with the simulator’s perplexity, with all LUTs at 16bits. Each number is relative to its own floating-point model.

Table 3. PPL ratio of DeepProve’s integer execution and of the simulator

PPL ratioWikiText-2 IntegerWikiText-2 Simulator
W16A160.9901.003
W12A160.9971.004
W8A161.3051.315
W16A121.0101.019
W16A101.2541.283
W16A810.00154.412

The results show that integer execution and simulation exhibit consistent overall trends across quantization settings. Both preserve near-baseline utility at W12A16 and W16A12, while reducing weight or activation precision further leads to substantial perplexity degradation. In particular, both pipelines exhibit severe degradation at W16A8, although the simulator predicts a larger loss (54.412 versus 10.001). These results provide empirical support for using the simulator to assess the relative utility impact of quantization configurations.

AuthorsOpenReview ↗

Rebuttal to Reviewer FD3p [part 2]

[W2/Q1-1] Alternative static calibration and LUT designs

Static calibration. For alternative static calibration, we repeated the setting of Table 9 (in the paper) at W16A16, W16A10 and W12A12 with four large models. We changed how activation scales are chosen.

Table 4. PPL ratio to the BF16 model under different activation calibrations

PPL ratioConfigPer-tensor (paper)8× dataP99.99Per-channelPer-group
Qwen2.5-7BW16A161.0031.0019224.21.6361.004
W12A121.1691.2238765.41.6341.011
W16A101.4941.7428538.31.6331.036
Llama-3.1-8BW16A161.0031.00261.111.2351.008
W12A121.7421.72960.881.2351.013
W16A101.8871.88260.641.2341.027
Qwen2.5-14BW16A161.0011.00116.021.5321.005
W12A121.0681.06015.821.5291.011
W16A101.2031.26315.321.5281.026
Qwen3-30B-A3BW16A160.9971.00127.471.0031.000
W12A121.0311.05226.931.0021.000
W16A101.2821.29427.191.0010.993

Static per-group calibration recovers most of the loss, while 8x more calibration data was not effective. We did not measure the proving cost as group scales are fixed constants, so they remain compatible with a fixed circuit. Hence, we slightly revise the claim that with static per-group calibration, about 10 bits activation successfully recovers the lost utility.

LUT designs. For alternative LUT designs, we evaluated three additional approaches: (1) percentile-based LUT range selection (P99.99, P99.9, and P99) with B16 precision, (2) applying range reduction while retaining B16 LUTs, and (3) witness-checked normalization (DeepProve’s normalization approach).

We first evaluated the utility on WikiText-2, reporting PPL ratios relative to the W16A16 linear-only baseline, as elaborated below.

Table 5. PPL ratio of RMSNorm constructions

RMSNorm constructionTable sizeLlama-3.1-8BQwen2.5-7BQwen2.5-14BQwen3-30B-A3B
Uniform grid (Full B16)2^161126.81496.932.46131.3
Uniform grid (RMS B24)2^241.00020.99870.99971.0014
Percentile range, P99.992^161059.21386.430.4775.94
Percentile range, P99.92^161034.11552.429.2527.70
Percentile range, P992^16divergeddivergeddivergeddiverged
Range reduction2^161.00050.99881.00081.0017
Witness-checked2^161.12461.10571.21121.0217

The results show that at P99.99, the PPL ratios are 1059.2, 1386.4, 30.47, and 75.94 for Llama-3.1-8B, Qwen2.5-7B, Qwen2.5-14B, and Qwen3-30B-A3B, respectively. At P99.9, the corresponding ratios are 1034.1, 1552.4, 29.25, and 27.70, while P99 causes all four models to diverge. These results indicate that percentile-based range selection alone cannot reliably preserve model utility. In contrast, range reduction (B16) achieves PPL ratios of 1.0005, 0.9988, 1.0008, and 1.0017, closely matching selective B24 RMSNorm (1.0002, 0.9987, 0.9997, and 1.0014).

We further measured the proving-time overhead of each approach.

Table 6. Proving time of the RMSNorm constructions

RMSNorm constructionGPT-2 SmallGPT-2 MediumGPT-2 LargeTinyLlama-1.1BLlama-3.1-8B (1 block)
Witness-checked (baseline)80.3 s170.1 s432.4 s407.7 s275.1 s
Uniform grid (Full B16)81.5 s (+1.5%)172.7 s (+1.5%)432.9 s (+0.1%)401.9 s (−1.4%)277.3 s (+0.8%)
Uniform grid (RMS B24)97.4 s (+21.3%)187.7 s (+10.4%)448.1 s (+3.6%)426.1 s (+4.5%)291.6 s (+6.0%)
Range reduction82.2 s (+2.4%)176.3 s (+3.7%)425.7 s (−1.5%)397.6 s (−2.5%)278.1 s (+1.1%)

The percentile range approach uses the same B16 table as the uniform B16 grid, so the proving cost is the same. Uniform B16 and range reduction stay within 4% of the baseline, because they add only a few lookups per token. Selective B24 adds more overhead, which comes from the 2^24 table itself. The relative overhead becomes smaller as the model size grows.

Overall, range reduction offers the best efficiency-utility trade-off, achieving B24-level utility at approximately B16-level proving cost while being less sensitive to the input range. Selective B24 RMSNorm remains a simple and practical alternative, particularly for larger models, where its nearly fixed overhead becomes relatively small.

[Q1-2] Generality beyond DeepProve

Our proving cost conclusion for LUT holds when the backend uses a lookup argument such as LogUp and plookup, where the prover processes every table entry once and it pays the same amount of costs per lookup. However, for a backend that uses a smaller field than DeepProve (254-bit), the proving can depend on the bit width (please see our response to Reviewer nDM4, [Q2]).

Comments on the paper

2 comments
Area_Chair_AuADOpenReview ↗

Reminder: Reviewer Engagement During the Discussion Period

Dear Reviewers,

As the discussion period is drawing to a close, could you please review the authors' rebuttal and provide your feedback at your earliest convenience?

Active engagement in discussions with the authors is an essential part of the peer-review process and plays an important role in ensuring the success of the conference. Your participation would be greatly appreciated.

Thank you for your time and contributions.

Best regards,

AC

AuthorsOpenReview ↗

Summary of reviewers feedback and authors response

We thank all reviewers for their constructive critiques and thoughtful feedback. We appreciate that the reviewers identified several consistent strengths in our submission. The comments helped us greatly in improving the manuscript.

Strengths: Reviewers found the research question relevant and timely: quantization for ZK-LLMs should be studied from the perspective of proof efficiency rather than conventional inference efficiency. They also recognized the operator-level diagnosis that connects utility degradation to the RMSNorm lookup, the broad evaluation, and the use of a real prover.

In response to common concerns, we have improved the manuscript. We extended the analysis of the RMSNorm bottleneck: beyond the B24 LUT, we now compare the utility and the proving cost of alternative constructions (percentile range, range reduction, and witness-checked normalization). We made the floating-point vs. integer comparison more systematic by measuring FP tiles of several widths, separating the fixed cost, and adding a comparison with the same prover. We also clarified the definition of static compilability, the proof system settings, and the scope of our proving-cost conclusions beyond DeepProve.

In the revised version of the paper, we added the following parts that address common questions and weaknesses from reviewers (changes are marked in blue):

Section 5.2.2: Proving Cost of the RMSNorm Constructions Appendix A: End-to-End Validation against Integer Execution Appendix J: Alternative RMSNorm Constructions and Their Proving Cost Appendix K: Estimating the Floating-Point Proving Cost Appendix L: Activation Precision under Alternative Static Calibration Appendix M: Proof System Settings and Other Backends

Since we add a new Table 3, later table numbers shift by one. The tables cited in our rebuttal as Table 5, 9, and 12 are now Table 6, 10, and 13.

The reviews made our analysis more compelling and more systematic. We hope that our findings will serve as useful support for the design of future ZK-LLM systems.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.