acceptodds
Under review as a conference paper at ICLR 2027

QuIL: Quantifying Quantization-induced Information Loss in Neural Networks

Abstract

Floating-point bit-string representations of parameters are ubiquitous building blocks of neural networks, and their quantization into lower precision formats is increasingly used to reduce their computational requirements. The degradation that quantization induces in the model is, however, not well studied, and its detection relies on task-specific heuristics. We propose QuIL: An information-theoretic approach for measuring quantization-induced information loss. By modelling the probability distribution of the underlying floating-point bit-strings of the parameters with a probabilistic circuit, we can measure information loss as the change in entropy across quantization levels. Experiments on neural-network regression, vision transformers, and GPT models show that QuIL, computed from the parameter posterior alone, identifies the bit-width at which quantization begins to degrade accuracy, calibration, and predictive density, offering a principled basis for selecting parameter precision.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
neural networks, probabilistic circuits, quantization, entropy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TqgNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces QuIL, an entropy-based measure of posterior structure removed by quantization, and extends BitVI to floating-point representations. It evaluates whether this measure predicts precision sensitivity in Bayesian neural networks and pretrained transformers.

Critical strengths
  • The mass-preserving coarsening formulation enables exact entropy calculations for the fitted posterior across multiple precisions without refitting.
  • The evaluation separates posterior-approximation errors in a tractable example and includes deterministic quantization comparisons, with strong sensitivity rankings at three bits.
Critical weaknesses
  • Appendix A.1 identifies the loss with the conditional entropy of discarded bits, whereas QuIL measures the differential-entropy increase from within-cell uniformization. For equal-width cells, a conditionally uniform discarded suffix has maximal conditional entropy but zero QuIL. The paper does not reconcile these different information quantities, weakening the central theoretical interpretation.

  • In Table A9, its four-bit attention correlation with accuracy change is +0.57, reversing the expected negative direction, while empirical-Fisher trace reaches -0.78. Its five-bit block correlation is also nearly zero. These results do not support the claim that QuIL is the most consistent sensitivity predictor.

  • At two bits and correlation 0.9, Table A10 reports a mean-field error of 0.278 ± 0.250 nats, substantially above the approximately 0.10 stated in the main text. The error increases with aggressive coarsening and posterior dependence, so the reported cancellation is not reliably established in this critical regime.

  • Mean BoolQ accuracy declines around six bits before the pronounced entropy rise, although seed variability is large. On RESISC, two-bit coarsening improves NLPD and ECE relative to ten bits while reducing accuracy. With the selection threshold remaining hypothetical, these results do not establish a reliable mapping from QuIL to the onset or type of predictive harm.

Questions for authors

See weakness

Reviewer_XDBQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an information theoretic metric that extends bit-string variational inference to floats using deterministic probabilistic circuits to measure the information loss due to quantization in the typical differential entropy formalization of information. They show experiments with toy models, shallow regression, ViT and GPT-2 and present that the posterior entropy change correlates to layer-sensitivity and that identifies the low-bit regime where predictive performance degrades, instead of simply being a downstream task based heuristic.

Critical strengths

Strong idea: viewing quantization from the precision reduction formulation as opposed to a perturbation to the point-estimate that forces the performance degradation to become a downstream heuristic is interesting. Seems to provide clean numerical-uncertainties and is effective both from the performance degradation as well as interpretation of quantization effects standpoints.

Expansion to non-uniform floating point intervals is a strong contribution

Critical weaknesses

While the hypothesis is strong, the results dont seem to show the expected improvements over simpler methods like the unrefined diagonal Laplace or the empirical fischer trace. This points towards the large number of parameters per model weight of a full probabilistic circuit seem unjustified if the performance difference to simpler methods is minimal or even negligible. The added compute for effectively being a curvature proxy makes the current results look negative. Same point applies to Tab. A3 on the Gpt-2 results.

Based on the results, the initial claims seem partly overstated.

Questions for authors

Please address the negligible gain of QuIL (for significantly more compute) over the simple Laplace and EF methods in Tab.1.

Below 4-5 bits, how much of the entropy surge is connected to true performance degradation? It is unclear if this is just an effect of some exponent-bit merging. Also in Fig.7, the ECE seems to jump at 3 bits before falling again and with small uncertainties; but the QuiL score increases smoothly - this doesn’t conform strongly with the claim that Quil tracks ECE atleast in the ViT case.

Why the discrepancy between ECE-Quil in Fig 7. vs Figs.A10, A11? This reversal of trends is not addressed and seems to hurt the narrative.

All the plots showing the different Block level test NLL and Quil plots have lines that are different shades of black/grey are practically unreadable.

Reviewer_TfeAOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates how to quantify the information loss when neural network weights are quantized to lower bit-widths. The authors argue that existing approaches measure this loss only indirectly through task-specific quantities like the loss curvature or the layer reconstruction error. On the other hand, QuIL quantifies the loss as the entropy increase of a weight posterior when it is coarsened from BB to tt bits, so that the bit-width at which degradation begins can be identified from the posterior alone. Since the posterior is represented as a deterministic probabilistic circuit whose sum nodes store conditional bit probabilities, every coarser resolution is obtained by collapsing subtrees into uniform leaves that keep their probability mass, and the entropy at all bit-widths is computed exactly from one fitted circuit. Moreover, they also propose a floating-point version of BitVI.

Critical strengths
  1. The paper is clearly written and the problem is highly relevant
  2. Treating precision reduction as circuit subtrees gives every bit-width from a single fit, with no refitting needed between levels nd without requiring heavy downstream evaluation
  3. The Appendix is detailed, which should support reproducibility once code is released
Critical weaknesses
  1. The claim that QuIL "accurately predicts and is the most consistent metric" seems a bit overstated, since the only levels where it is competitive are at p=3p=3 (shown in Table 1) and partly at p=2p=2. In Table A9, where the other levels are presented, the other metrics beat QuIL on Block, Attention and MLP at p=5p=5 and p=4p=4. The paper should report all levels in the main table and discuss the fact that on those levels QuIL is not superior to the other metrics.

  2. The claims on the predicting capability of QuIL on the quantization damage are not adequately tested, since entropy and RMSE both rise below 4-5 bits (Fig. 6) holds for any monotone function of the bit-width. In the same way, if the correlations r≥0.85r \ge 0.85 in Sec. 4.1 span precision levels, they are mostly driven by both quantities moving with the bit count. Moreover, the whole accuracy range in Fig. 7 is only 0.25 pp and no seed variance is shown, so it is unclear whether the change exceeds noise

  3. The experimental scope and computational cost do not support entirely the claims made in the abstract/introduction, since, for example, the GPT models mentioned in the abstract are evaluated only through their binary classification heads on BoolQ and MRPC, while inner layers are tested only on ViT/CIFAR-10. Moreover, each weight needs 2B+1−12^{B+1}-1 circuit parameters, and each large-scale experiment takes 6–15 hours, while the EF trace that matches QuIL needs one pass over the data, so wall-clock and peak memory against the baselines are needed

Questions for authors
  1. If a threshold τ\tau is fixed on one dataset, how often does the bit-width it selects match the observed degradation on the other datasets?
  2. What are the per-layer wall-clock time and peak memory of QuIL compared to the baselines?
  3. Does the ranking in Table 1 still hold when QuIL is computed on inner layers of a GPT model rather than on its classification head only?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.