- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies the final softmax when a network is run under homomorphic encryption, where the exponential is replaced by a polynomial. The issue that is discussed: with the logits held fixed, a low-degree polynomial can leave the predicted class unchanged while changing the probabilities substantially. The authors explain this distortion and discuss three remedies. (R1) the server returns logits and the client computes a temperature-scaled softmax. (R2c) the server returns the polynomial numerator and the client inverts it before applying softmax. (R3) when the probabilities must stay encrypted, the calibration temperature is folded into the model and the polynomial depth is chosen by a label-free rule.
Critical strengths
-
A clear research direction and a convincing demonstration of the effect.
-
The proposed label-free depth-selection heuristic is potentially useful in selecting an effective depth that brings calibration close to the exact softmax.
Critical weaknesses
1. R1 is trivial, and R2c is close to trivial and does not add much
Returning the logits and applying temperature scaling on the client is standard practice. The paper itself says (Section 5.3) that R1 is "a deployment design rule ... not a new calibration algorithm" and that returning logits "is common in FHE classifiers". R2c inverts a public, deterministic scalar function on the client, which is immediate once the polynomial is known. The paper also notes (Section 5.7) that R2c hides nothing that R1 reveals and exists only for compatibility with pipelines that already return numerators.
2. The effect is expected
Confidence distortion caused by reduced numerical precision has been reported outside HE by Gu et al. (2026 [1]), as the submission acknowledges.
Regardless of previous work, without extra depth, the degree-3 fit to exp has large relative errors (Table 4). Distorted probabilities are the expected outcome of such a fit. On the other hand, ffter the max-shift the predicted class sits at u = 0, so the label survives whenever the floored polynomial is largest at 0 (Prop. 9). Therefore, "same label, different probabilities" is what one would predict.
3. The effect has been reported before
Pathan et al. [2] is cited but not discussed for its most relevant result. They compare a CNN with an exact softmax head against one with an HE-friendly Chebyshev head: accuracy drops slightly while ECE rises more than a hundredfold, which is the submission's main observation in an HE setting. The submission refers to this work but never to this result.
The submission should acknowledge that a large calibration loss from a polynomial softmax head under HE has already been reported.
4. Missing related work
Kuang and Wong [3] show that calibration degrades as quantization becomes coarser, and that temperature scaling improves calibration error for quantized networks, with some caveats. This is the submission's diagnosis and its R1 remedy for a different numerical approximation, although in their setting accuracy slightly drops, so the calibration loss is not hidden by unchanged predictions.
References
[1] Jiawei Gu, Fengyuan Nie, Hao Tang, and Yanpeng Sun, "Precision-Induced Miscalibration: Understanding and Correcting Confidence Distortion in Quantized Neural Networks," in International Conference on Machine Learning (ICML), PMLR 306, pp. 37158–37194, 2026. link.
[2] Shadman Mahmood Khan Pathan, Qianlong Wang, Jonathan Takeshita, Sakan Binte Imran, and Sachin Shetty, "Privacy-Preserving Post Deployment Model Calibration with Fully Homomorphic Encryption," ICNC 2026. link.
[3] Joey Kuang and Alexander Wong, "On Calibration of Modern Quantized Efficient Neural Networks," arXiv:2309.13866, 2023.
Questions for authors
Please reevaluate the novelty of the paper.