acceptodds
Under review as a conference paper at ICLR 2027

Output Probabilities Under Homomorphic Encryption: Depth Chosen for Accuracy Is Not Enough

Abstract

Homomorphic encryption (HE) lets a server run a model on data it never sees, and HE inference is judged by task accuracy, approximation error, and multiplicative depth. Many deployments, however, consume the output probabilities themselves, through confidence thresholds, abstention rules and cost-weighted decisions, and none of those three quantities certifies them. Replacing the output softmax by a polynomial, the usual HE-compatible route, leaves the predicted label unchanged in every seed of ten backbone and dataset settings while inflating expected calibration error on the same logits by 1.1x to 30x, and by roughly 10x to 20x over the full vocabularies of two language models. Under the repeated-squaring schedule used for high-precision homomorphic exponentials, accuracy reaches its ceiling one to four multiplicative levels before calibration comes within 0.005 of the exact softmax: the multiplicative depth sufficient to preserve predictions can be systematically insufficient to preserve output probabilities. We trace the effect to a non-affine residual in pairwise log-odds, which yields a label-free diagnostic, computable from the public polynomial and the operating range before deployment, that chooses depth where accuracy cannot. The repair depends on the interface. Where logits may leave the ciphertext, defer the softmax to the client; where an existing pipeline already returns polynomial numerators, invert them in plaintext as a compatibility layer, using a numerator monotone by construction; where the distribution itself must stay encrypted, fold the calibration temperature into the model and let the diagnostic set the depth, which in exact-arithmetic simulation lands within 0.004 ECE of the client-side result on 96 of 99 seeds. Real CKKS runs of the head and numerator reproduce the simulated effect.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
homomorphic encryption, calibration, polynomial approximation, softmax, privacy-preserving inference, uncertainty quantification, CKKS, temperature scaling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_XQr4OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the final softmax when a network is run under homomorphic encryption, where the exponential is replaced by a polynomial. The issue that is discussed: with the logits held fixed, a low-degree polynomial can leave the predicted class unchanged while changing the probabilities substantially. The authors explain this distortion and discuss three remedies. (R1) the server returns logits and the client computes a temperature-scaled softmax. (R2c) the server returns the polynomial numerator and the client inverts it before applying softmax. (R3) when the probabilities must stay encrypted, the calibration temperature is folded into the model and the polynomial depth is chosen by a label-free rule.

Critical strengths
  1. A clear research direction and a convincing demonstration of the effect.

  2. The proposed label-free depth-selection heuristic is potentially useful in selecting an effective depth that brings calibration close to the exact softmax.

Critical weaknesses

1. R1 is trivial, and R2c is close to trivial and does not add much

Returning the logits and applying temperature scaling on the client is standard practice. The paper itself says (Section 5.3) that R1 is "a deployment design rule ... not a new calibration algorithm" and that returning logits "is common in FHE classifiers". R2c inverts a public, deterministic scalar function on the client, which is immediate once the polynomial is known. The paper also notes (Section 5.7) that R2c hides nothing that R1 reveals and exists only for compatibility with pipelines that already return numerators.

2. The effect is expected

Confidence distortion caused by reduced numerical precision has been reported outside HE by Gu et al. (2026 [1]), as the submission acknowledges.

Regardless of previous work, without extra depth, the degree-3 fit to exp has large relative errors (Table 4). Distorted probabilities are the expected outcome of such a fit. On the other hand, ffter the max-shift the predicted class sits at u = 0, so the label survives whenever the floored polynomial is largest at 0 (Prop. 9). Therefore, "same label, different probabilities" is what one would predict.

3. The effect has been reported before

Pathan et al. [2] is cited but not discussed for its most relevant result. They compare a CNN with an exact softmax head against one with an HE-friendly Chebyshev head: accuracy drops slightly while ECE rises more than a hundredfold, which is the submission's main observation in an HE setting. The submission refers to this work but never to this result.

The submission should acknowledge that a large calibration loss from a polynomial softmax head under HE has already been reported.

4. Missing related work

Kuang and Wong [3] show that calibration degrades as quantization becomes coarser, and that temperature scaling improves calibration error for quantized networks, with some caveats. This is the submission's diagnosis and its R1 remedy for a different numerical approximation, although in their setting accuracy slightly drops, so the calibration loss is not hidden by unchanged predictions.

References

[1] Jiawei Gu, Fengyuan Nie, Hao Tang, and Yanpeng Sun, "Precision-Induced Miscalibration: Understanding and Correcting Confidence Distortion in Quantized Neural Networks," in International Conference on Machine Learning (ICML), PMLR 306, pp. 37158–37194, 2026. link.

[2] Shadman Mahmood Khan Pathan, Qianlong Wang, Jonathan Takeshita, Sakan Binte Imran, and Sachin Shetty, "Privacy-Preserving Post Deployment Model Calibration with Fully Homomorphic Encryption," ICNC 2026. link.

[3] Joey Kuang and Alexander Wong, "On Calibration of Modern Quantized Efficient Neural Networks," arXiv:2309.13866, 2023.

Questions for authors

Please reevaluate the novelty of the paper.

Reviewer_7G81OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates probability calibration errors caused by polynomial approximations of the output softmax in homomorphic-encryption-based inference. The authors show that such approximations can preserve classification accuracy while substantially distorting predicted probabilities. They analyze the distortion using pairwise log-odds, propose a depth-selection diagnostic, and introduce several remedies, including deferred client-side softmax, polynomial inversion, and monotone polynomial construction. The methods are evaluated across multiple datasets and architectures using numerical simulations and selected CKKS experiments.

Critical strengths

The paper addresses a relevant issue in HE-based inference by distinguishing prediction accuracy from probability calibration. The observation that accuracy-preserving approximations can affect confidence-based decisions is potentially useful, and the log-odds analysis provides a mathematical characterization of this effect.

Critical weaknesses
  • The practical significance and methodological novelty are insufficiently established. The reported calibration degradation is primarily demonstrated using low-degree polynomial approximations of the exponential function, which can introduce substantial numerical errors over wide logit ranges. However, existing methods such as HETAL (ICML 2023) and Cho et al. (CCS 2024) employ more sophisticated techniques to achieve high-precision homomorphic softmax evaluation. The manuscript does not establish whether the reported calibration failures remain significant under these approaches. Moreover, the proposed remedies largely rely on established techniques, including client-side softmax evaluation, temperature scaling, and polynomial inversion. The more distinctive encrypted-output depth-selection approach is primarily evaluated through exact-arithmetic simulations. Consequently, the practical need for the proposed methods and their advantages over existing HE techniques remain unclear.
  • The presentation quality requires substantial improvement. The figures are poorly designed and difficult to interpret. In particular, Figure 2 presents the central experimental findings using excessively small plots, labels, and annotations, making the results difficult to examine. Figures 1 and 3 also provide limited visual clarity relative to their intended purposes. The writing is unnecessarily dense, with numerous abbreviations, fragmented explanations, and frequent cross-references that obscure the main contributions. The extensive mathematical presentation further complicates the narrative without consistently clarifying the underlying methodological advances. Overall, the presentation falls considerably short of the clarity expected for an ICLR submission.
Questions for authors
  • How significant are the reported calibration errors when using high-precision homomorphic softmax methods, such as HETAL (ICML 2023) and Cho et al. (CCS 2024)? Can the authors demonstrate that the proposed depth-selection strategy provides meaningful benefits over these approaches under comparable precision and computational budgets?
  • Can the authors validate the proposed encrypted-output remedy in a complete CKKS inference pipeline, including the required encrypted operations, and demonstrate its practical efficiency compared with existing high-precision implementations?
Reviewer_HohqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how polynomial softmax approximations in HE inference can preserve classification accuracy while distorting probabilities and calibration. It introduces a log-odds distortion diagnostic for depth selection and interface-dependent remedies involving deferred softmax, polynomial inversion, and temperature scaling. Experiments across multiple datasets and backbones show that the proposed remedies can improve calibration while preserving classification accuracy.

Critical strengths

The experiments and log-odds analysis distinguish prediction preservation from probability fidelity, providing guidance for probability-sensitive HE inference.

Critical weaknesses

Conventional HE classification inference does not generally require an encrypted output softmax. Prior work, including Chex-Mix [R1], explicitly states that the final softmax is typically evaluated on the client. Moreover, top-1 classification requires only the logits. Consequently, the practical relevance of the paper's central problem setting is insufficiently justified.

[R1] Deepika Natarajan et al. “Chex-Mix: Combining homomorphic encryption with trusted execution environments for oblivious inference in the cloud.” 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P), 2023.

Questions for authors

As explicitly stated in Chex-Mix (EuroS&P 2023, Section 3.2.2), "all prior HE works for inference [24], [27], [33], [48] implement any final softmax layers directly on the client device," since homomorphic evaluation of softmax is computationally expensive. Moreover, for conventional classification tasks, the predicted class can be determined directly from the logits without computing softmax.

Given these considerations, could the authors clarify what the novel technical contributions of this paper are beyond identifying calibration degradation in a setting that existing HE inference approaches generally avoid? In particular, it would be helpful to explain how the proposed methods advance the state of the art in those scenarios.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.