acceptodds
Withdrawn from ICLR 2027

The Privacy Side Effect of Cheap Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reducing the numerical precision of model weights is common in deployment because it lowers computational cost. This reduction also selectively lowers a model’s ability to reproduce the training data word-for-word, by several bits, before it substantially affects general capability. This difference is termed the selectivity gap. The gap is measured across a 29-fold range in parameter count and is replicated in a second model family, an error-correcting production quantizer, and text-to-image diffusion, where the gap is more pronounced. Although the gap narrows as models grow, it stays above 24.2 % points of retention across all sizes tested. The content destroyed is not predicted by data duplication, although a magnitude filter, such as quantization, should spare duplicated content. Instead, the sharpness of the loss surface around a memorized sequence predicts its sensitivity to numerical rounding. This extends prior observations linking memorization to sharp loss regions by showing that the same sharpness also predicts which memorized content is lost under quantization. In diffusion models, the selectivity gap persists, but weight-space sharpness is no longer predictive. Instead, the gap is associated with the stability of image generation to perturbations in the starting noise. Finally, decomposing a realistic deployment recipe shows that the 8-bit activation quantizer, rather than the 4-bit weight quantizer, is primarily responsible for the loss of training-image reproduction. Numerical precision is therefore both an efficiency-related deployment choice and a privacy-relevant configuration that can substantially affect a model’s ability to reproduce its training data.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
quantization, memorization, training data extraction, diffusion models

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_R7pCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether quantization reduces reproduction of training data more than general capability. It measures this difference through a selectivity gap across five Pythia sizes, Amber-7B, several quantizers, and Stable Diffusion models. It also examines which memorized examples survive quantization, finding associations with SAM sharpness in language models and initial-noise basin stability in diffusion. An additional diffusion experiment separates the effects of weight and activation quantization.

Critical strengths

S1. The selectivity gap is a useful and interpretable comparison. Measuring extraction and capability against the same full-precision baseline makes their different responses to quantization clear. The positive gap appears across the tested language-model sizes, quantizers, and both model families. A related concurrent preprint [1] reports the same qualitative phenomenon using a different metric; explaining how the present study extends that work would help position its contribution.

S2. The activation decomposition provides useful insight. Table 2 isolates weight and activation quantization and attributes the observed elimination of image copies primarily to the activation component.

References

[1] Akshay Sasi. Bits and Memories: Measuring Verbatim Extraction Across LLM Quantization. arXiv:2607.25451, 2026.

Critical weaknesses

W1. The diffusion mechanism has unclear independent image coverage and an incomplete definition. The primary set R contains 70 captions but only five independent reference images. The authors acknowledge similar caption dependence in the mechanism set S, but do not report its number of independent image groups. Its caption-level intervals and significance test therefore do not establish generalization across independent images (Section 7; Appendix G). The additional image groups validate the quantization intervention, but not the basin predictor. The paper also omits its complete computation: perturbation distribution, radius values, number of perturbations, similarity measure, and aggregation rule. A reproducible definition and analysis accounting for image groups are needed to assess this central result.

W2. The language-model analysis does not isolate curvature from general perturbation sensitivity. SAM sharpness measures loss increase within a finite perturbation radius. Since an individual sequence's gradient need not vanish, this statistic includes first-order sensitivity as well as curvature. The reported association supports local sensitivity to weight perturbations. A curvature-specific interpretation requires separating the second-order contribution or showing that the association is not explained by the gradient term.

W3. The language-model effect is not specific to memorized training text. Table 7 shows that confidence-matched held-out continuations are about as fragile as memorized continuations. This points to a broader sensitivity of exact generation supported by model weights, rather than a vulnerability specific to training-data memorization. Appendix F also shows that the NF4 extraction reduction at 12B falls from 37.3% under greedy decoding to 23.7% under stronger search. The gap remains a useful empirical finding, but the main text should foreground these limits on its privacy interpretation. As the authors acknowledge, failure of exact reproduction also leaves open whether the same information remains accessible in other forms.

W4 (Minor). Model coverage limits transfer to current deployment settings. The diffusion experiments use SD v1.4 and SD2.1-base. The conclusions should remain tied to this coverage. Newer diffusion backbones would be useful extensions where comparable training-data ground truth is available.

W5 (Minor). The five-bit elimination claim needs qualification. Appendix H already explains the threshold dependence and incomplete suppression in SD2.1, while Appendix G.1 shows that the precise six-to-five-bit collapse is not universal across image groups. The abstract and contribution statements should consistently describe suppression with these qualifications.

Questions for authors

Q1. Could you assess whether the GPTQ selectivity gap persists under the stronger extraction procedure in Appendix F?

Q2. How many independent reference-image groups are in the basin-stability regression? Could you provide its full definition and assess the association at the image-group level, including the additional groups in Appendix G.1?

Q3. How does the implemented SAM statistic distinguish curvature from first-order sensitivity?

Q4. How should the privacy benefit be interpreted if the same information remains accessible through non-verbatim outputs?

Reviewer_Kf7hOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Summary: This paper analysis the implications of model quantization on privacy. Specifically, it checks a range of weight and activation quantizations and compares the rate of capability degradation with the rate of memorization degradation (for various notions of memorization). It reports that capability degrades more slowly with quantization than memorization. Thus there is a certain “privacy side effect of cheap inference”. Both LLMs and diffusion model families are explored. It also explores some potential explanations for this phenomena by correlating the degradation with other indicators such as sharpness (loss landscape curvature). This could suggest that memorized sequences are high curvature and those high-curvature places are brittle to quantization.

Critical strengths

S1: The main empirical result is quite intuitive and probably correct. The controlled experiment with paired comparison deserves some credit.

S2: Some reasonable attempts at analysis are provided in terms of the curvature, precision loss, duplication, etc.

S3: Good to have evaluated on both LLM and diffusion models.

Critical weaknesses

W1: It feels like an evaluation report, not an ICLR paper. I didn’t get any new take home message in terms of novel insight, algorithm, or actionable item. It’s known that different capabilities decay at different rates with quantization. (Anyone who shipped a multi-task model to production knows it at least). That exact memory recall is one of the faster decaying capabilities might not have been published previously, but it’s exactly what anyone would have predicted if asked to guess. The correlations to curvature, etc are mildly interesting, but also generally expected. (And the vulnerability of high-curvature to quantization is obvious).

W2: It’s not 100% established from the paper if memorization as such has a special relationship with quantization, or if what’s vulnerable to quantization are tasks that require exactly one specific long sequence to be output. The other tasks compared with memorization might not require exact reproduction of 32 token sequence in the way that the memorization evaluation does. I think Appendix C doesn’t fully confirm this because the context window in the copy experiment is longer than the memory experiment (96 vs 32).

W3: While it’s good for the world that there might be some “free” privacy in practice. It’s oversold as a contribution. It’s not a guarantee, and it may be weaker under stronger attacks. Meanwhile there is not some new actionable insight about how to make the models more private than status quo.

W4: Missing opportunity to experiment on intervention to better establish causality between the observed correlates and degradation. Does intervening on sharpness with SAM or similar have the expected impact?

W5: In the privacy defense community there are papers that used quantization related tools to provide privacy defense. This paper seems not to cite them. Which further undermines novelty. For example: https://ieeexplore.ieee.org/abstract/document/9897681 https://pmc.ncbi.nlm.nih.gov/articles/PMC10538103/ https://arxiv.org/abs/2610.00382 Also I came across this related work https://arxiv.org/abs/2607.25451 which is not cited.

Questions for authors

Future work:

  • Can you intervene on sharpness etc to check for causation.
  • Can you precisely distinguish between memorization vs other exact string output tasks to more precisely link memorisation as such with quantization?
  • How can the apparently different results in diffusion & LLM sections be unified in order to provide a general theory of what's going on here?
Reviewer_5tFVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the impact that quantization has on verbatim training data memorization in LLMs (Pythia series and Amber-7B) and text-to-image diffusion models (StableDiffusion). The authors find that, for the models tested, the more quantization is applied, the more verbatim memorization is reduced. From full precision down to 5 or 4 bits, the models still retain most of their original utility on a few tasks (e.g., LAMBADA, HellaSwag, ARC-Easy) while losing the majority of memorized spans. Below 4-bit quantization, utility drops sharply, closing the gap with memorization. The paper then attempts to correlate certain properties of the models (loss sharpness) and training data (duplication) with the reduction in memorization, and finds differing predictors between LLMs (loss sharpness) and diffusion models (sampling trajectory).

Critical strengths

The effect of quantization on verbatim memorization and its comparison to utility retention is an interesting result. Measuring this in open-data models is a solid choice.

Critical weaknesses

Motivation: While the study of quantization's impact on memorization is an interesting question, I'm not sure how the observed results can be applied. The last two sentences of Section 6 appear to suggest that quantization can be a privacy defense. However, the full precision model is often available along the quantized versions, defeating any privacy benefits. The paper currently does not explicitly describe any practical motivation. A more useful result could come if there is correlation between quantization-aware training and memorization.

Thoroughness: The authors should investigate more open-data LLMs, particularly the Olma family, or the full Pebble series from which Amber-7B comes, to confirm their findings are generalizable. Pythia is a good choice, but is rather out of date. It might even be possible to extend to closed-data LLMs like Qwen, relying on verbatim extraction to establish baseline memorization against a public corpus. The quantization scheme for the LLMs can also be extended: the authors separate weight vs activation quantization in the case of diffusion, but did not do the same for LLMs.

Writing quality: The writing is written almost entirely from a third-person perspective and a passive voice. Certain concepts and metrics are introduced without clear origin or explanation. For example, the paper introduces a concept called "selectivity gap" to describe the gap between memorization and utility. The word "selectivity" appears to imply that models are being selective about what to retain, which I don't think is quite the case. Another example is "distribution-level quality" and "FID", metrics measured for diffusion models but are not explained.

Questions for authors

Do these findings extend to other model families like Olmo (or even Qwen)?

Does weight vs activation quantization make any difference in the case of LLMs?

Comments on the paper

1 comment

We thank the reviewers and the area chair for their time. Reviewers noted that the selectivity gap is interpretable, the paired design is controlled, the use of open-data models is sound, and the weight-and-activation decomposition is informative.

We want to note what the current paper already establishes:

  1. The gap stays positive across a 29-fold range of model sizes, two quantizer families, a second model family, and a stronger search-and-sampling attacker (Appendix F).
  2. Duplication, the best-established driver of memorization, does not predict which memorized sequences survive once the other predictors are controlled, even when it is counted exactly in the model's own corpus.
  3. In diffusion, the activation quantizer removes replication in all 14 independent image groups we tested (Appendix G.1).

The reviews also gave helpful feedback, which we will use to improve the paper. Table 7 shows that confidence-matched held-out text is about as fragile as memorized text, so the claim is best stated as exact recall stored in the weights, with memorized text as one case. SAM sharpness at a finite radius includes a first-order term, and we will separate it from the second-order curvature term. We will also state the threat model on the first page. Our limitations section already states that the effect is not a guarantee.

We are withdrawing to incorporate these extensions properly rather than within a short discussion window.

Note. arXiv 2607.25451 was posted on 28 July 2026, about seven weeks before the abstract deadline. arXiv 2610.00382 was posted on 30 September 2026, after the submission deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.