- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a framework for exploring weight, activation, and nonlinear lookup-table precision in LLMs intended for zero-knowledge inference. It evaluates nine models using a PyTorch quantize–dequantize simulator and measures selected proving configurations using modified DeepProve backends.
The central findings are that activations are more sensitive than weights under the evaluated quantization policy; uniformly discretized inverse-square-root tables can severely degrade RMSNorm-based models; selectively increasing RMSNorm table precision can recover utility; and reducing nominal precision does not necessarily reduce total proving time. An additional case study benchmarks a proof of a quantized Min-K% membership-inference statistic.
Critical strengths
- Quantization optimized for conventional inference need not optimize proof generation. Separating numerical precision, arithmetic-circuit structure, lookup-table size, and total prover cost is useful. The paper’s emphasis on measuring realized costs rather than assuming that fewer bits automatically imply faster proofs is appropriate.
- The nonlinear-operator diagnosis is informative.
- The paper includes KL divergence, top-token overlap, low-probability-position overlap, and full ARC-Easy and MMLU evaluations. This is valuable because preserving perplexity does not guarantee preservation of individual predictions or audit statistics. The released downstream results support the claim that the selected Full16 configuration retains much of the reference performance in the simulator.
Critical weaknesses
The RMSNorm failure is convincingly diagnosed for the chosen implementation: uniformly discretizing a range extending to roughly 65,461 with a B16 table gives spacing close to one, causing many small inputs to map to zero. However, this establishes a weakness of that range-selection and uniform-grid design, not that RMSNorm generally requires a 24-bit input table in ZK inference.
Relevant alternatives include static percentile calibration, finer static scale granularity, outlier smoothing, range reduction, segmented tables, and proving normalization through algebraic constraints on a supplied witness. Several of these remain compatible with a fixed proof circuit. In particular, DeepProve’s documented normalization approach avoids explicitly tabulating inverse square root in the manner evaluated here.
Similarly, the conclusion that activations generally need 12–14 bits is based primarily on static per-tensor max calibration using eight windows. Appendix G already shows that changing activation-scale selection can substantially recover W12A12 utility. Although dynamic calibration may introduce proof overhead, static per-channel or grouped alternatives should be compared before presenting the observed precision threshold as broadly characteristic of ZK-friendly quantization.
The definition also needs care: runtime-dependent values do not require a runtime-dependent circuit. Computing a scale or selecting a branch can be encoded in a fixed circuit, albeit at additional cost. Static calibration is a potentially efficient design choice, not a universal prerequisite for ZK compatibility.
Questions for authors
-
Can the authors provide end-to-end simulator-versus-prover numerical validation, including integer requantization, normalization, attention, and the exact LUT input/output precisions?
-
How do the activation and RMSNorm findings change with stronger static calibration and normalization-aware proof constructions, rather than only uniformly enlarging the LUT?
-
What is the measured proving overhead of selectively promoting RMSNorm precision on a model where that intervention is needed for utility?
-
How stable is the approximately 1200× projected speedup when floating-point proofs are measured at multiple tile widths and fixed overhead is accounted for?
2 comments
Rebuttal to Reviewer f8cW [part 2]
[W3] Definition of static compilability
Yes, we agree with you that a static circuit can compute runtime-dependent values in ZK, which would, yet, incur high prover overhead. For example, computing the scale dynamically would require calculating the maximum and minimum values of the activations in every layer, which are non-linear functions applied to every element of the activations. The cost would be at least as expensive as the non-linear activation functions, which contribute to the majority of the prover time in zkLLMs. Therefore, the cost of dynamic scales would be at least 2x more than the static calibration. This is a very conservative estimation ignoring the additional cost of applying the quantization after computing the scales dynamically. We will refine Definition 3 to clarify this and emphasize our focus on efficiency.
[Q1] Simulator vs. integer execution
For GPT-2 Small, we measured the perplexity of DeepProve’s integer execution and compared it with the simulator’s perplexity, with all LUTs at 16 bits. Each number is relative to its own floating-point model.
Table 3. PPL ratio of DeepProve’s integer execution and of the simulator
| PPL ratio | WikiText-2 Integer | WikiText-2 Simulator |
|---|---|---|
| W16A16 | 0.990 | 1.003 |
| W12A16 | 0.997 | 1.004 |
| W8A16 | 1.305 | 1.315 |
| W16A12 | 1.010 | 1.019 |
| W16A10 | 1.254 | 1.283 |
| W16A8 | 10.001 | 54.412 |
The results show that integer execution and simulation exhibit consistent overall trends across quantization settings. Both preserve near-baseline utility at W12A16 and W16A12, while reducing weight or activation precision further leads to substantial perplexity degradation. In particular, both pipelines exhibit severe degradation at W16A8, although the simulator predicts a larger loss (54.412 versus 10.001). These results provide empirical support for using the simulator to assess the relative utility impact of quantization configurations.
[Q3] Proving cost of the RMSNorm fix
Table 4. Proving time and peak memory of the RMSNorm fixes
| Proving time, peak memory | GPT-2 Small | GPT-2 Medium | GPT-2 Large | TinyLlama-1.1B | Llama-3.1-8B (1 block) |
|---|---|---|---|---|---|
| Baseline | 80.3 s, 20.3 GiB | 170.1 s, 34.8 GiB | 432.4 s, 122.5 GiB | 407.7 s, 90.4 GiB | 275.1 s, 85.7 GiB |
| Selective B24 (RMSNorm only) | +17.1 s (+21.3%), 22.2 GiB | +17.6 s (+10.4%), 36.6 GiB | +15.7 s (+3.6%), 124.4 GiB | +18.4 s (+4.5%), 92.3 GiB | +16.6 s (+6.0%), 87.1 GiB |
| All LUTs at B24 | +177.7 s (+218.6%), 95.0 GiB | +333.3 s (+189.6%), 169.5 GiB | out of memory (> 245 GB) | - | - |
| Range reduction (B16) | +1.9 s (+2.4%), 20.3 GiB | +6.2 s (+3.7%), 35.0 GiB | −6.7 s (−1.5%), 122.3 GiB | -10.1 s (-2.5%), 90.2 GiB | +3.1 s (+1.1%), 85.2 GiB |
Selective B24 uses a B24 table only for RMSNorm. Other operations, GeLU (or SiLU) and the exponential, use B16 LUTs. The larger the model, the smaller the relative increase in cost. This is because the added proving cost is proportional to the LUT size and is nearly fixed. However, using B24 tables for all nonlinear operations makes the proving cost very large. This shows that selective bit width increment is still important. Range reduction costs almost the same proving time and memory as the B16 baseline. The additional lookup queries are very few compared with the other lookups in the whole GPT-2 model, so they did not significantly affect the proving cost.
In conclusion, range reduction is efficient. It can achieve B24-level utility at B16 cost and is less sensitive to the input range. Still, B24 remains the simplest alternative, and its fixed cost becomes negligible for large models.
[Q4] Stability of the FP vs. Integer comparison
An FP proof cannot cover a whole projection (e.g., 5,120 output columns) at once (Section 5.2.1). Thus, we proved a tile of 8 output columns and multiplied its proving time by (total columns / 8). We agree that the fixed overhead is not separated, so we proved FP tiles of 4, 8, 16, 32 and 64 output columns. The time grows linearly with the number of columns and the fixed cost is about 11-12% of the 8-column proving time (Table 6). For the seven projections of one Qwen2.5-14B block, the two ways of extrapolating give the following FP/integer ratios.
Table 5. FP/integer proving-time ratio for one Qwen2.5-14B block
| FP extrapolation | FP / integer |
|---|---|
| Paper's setting | 1,179× |
| Fixed cost separated | 1,067× |
(Both numbers were measured again in one session on the same server)
Table 6. FP proving time of one tile
| Input dimension | 4 cols | 8 cols | 16 cols | 32 cols | 64 cols | Fixed cost |
|---|---|---|---|---|---|---|
| 5,120 (Q, K, V, O, gate, up) | 4.85 | 9.84 | 19.38 | 37.47 | 72.02 | 1.04 s |
| 13,824 (down) | 14.20 | 23.86 | 45.66 | 88.95 | – | 2.91 s |
Separating the fixed cost reduces the ratio by about 10%, and the gap still remains three orders of magnitude. We will revise the numbers in the paper.
Rebuttal to Reviewer f8cW [part 1]
We thank the reviewer for the insightful feedback and valuable comments.
[W1/Q2] RMSNorm’s failure analysis
We agree that increasing to 24 bit-width is not the only fix. The utility can be restored with a normalization-aware construction and RMSNorm at 16 bit-width. Hence, we evaluated three more alternatives: percentile calibration, range reduction, and witness-checked normalization (DeepProve’s normalization approach). Other methods (e.g., finer granularity, outlier smoothing, segmented table) were not evaluated separately. Range reduction is itself a segmented table, and the other two cannot shrink the input range of the RMSNorm LUT.
We used the four large models, keeping every other setting the same as in Table 5 in the paper. Below are the full results of PPL ratio compared to the W16A16 linear-only baseline (WikiText-2). Other LUTs use 16 bits.
Table 1. PPL ratio of RMSNorm constructions
As shown in the Table above, range reduction fully recovered utility while keeping the table at 16 bits, whereas percentile range selection was not effective. As shown in the paper (R3, Section 5.1), most inputs fall into the lowest few entries of the RMSNorm LUT, yet the LUT must still cover a few tokens with massive activations. P99.9 and P99.99 calibration cover almost the same range as the uniform B16 grid, so they give no finer resolution for small inputs. P99, on the other hand, resolves small inputs but cannot cover the large ones, so PPL diverges. For range reduction, the table only needs to cover [1, 4), because we rewrite each input as m·4^k with 1 ≤ m < 4. That is, an input outside this range is divided (or multiplied) by 4 until it falls inside the table. The value is then read from the table and scaled back with a simple shift. Thus, no part of the input range is lost, and small and large inputs get the same relative precision. Witness-checked normalization lets the prover supply the inverse square root and the circuit only checks it. This check needs its own table, and at 2^16 entries it rounds the outputs too coarsely.
We confirm that range reduction is an effective alternative to B24. It keeps proving time and memory at the B16 level. It, however, needs more lookups per token. B24 is simple to implement and the added cost is fixed regardless of model size. However, it still uses more memory and proving time than B16. We therefore refine our conclusion slightly.
[W2/Q2] Activation precision under stronger calibration
For alternative static calibration, we repeated the setting of Table 9 (in the paper) at W16A16, W16A10 and W12A12 with four large models. We changed how activation scales are chosen.
Table 2. PPL ratio to the BF16 model under different activation calibrations (WikiText-2)
Static per-group calibration recovers most of the loss, while 8x more calibration data was not effective. We did not measure the proving cost as group scales are fixed constants, so they remain compatible with a fixed circuit. Hence, we slightly revise the claim that with static per-group calibration, about 10 bits activation successfully recovers the lost utility.