acceptodds
Under review as a conference paper at ICLR 2027

HELITE-RESNET: A CKKS-FRIENDLY RESIDUAL NETWORK FOR END-TO-END ENCRYPTED FACIAL EXPRESSION RECOGNITION

Abstract

Since facial expression recognition (FER) processes sensitive biometric data, privacy concerns arise. Secure multiparty computation and trusted execution environments are common but require multiparty interaction and trusted hardware, respectively. Fully homomorphic encryption instead enables non-interactive, end-to-end encrypted inference. Unlike exact-integer FHE schemes, CKKS supports approximate arithmetic on real and complex numbers and SIMD packing. However, CKKS-based residual networks still incur substantial computational and memory overheads. Prior work optimizes ciphertext packing, convolution mapping, and polynomial activations, while global costs from stage topology and cross-resolution residual paths remain underexplored. We propose HELite-ResNet, a CKKS-friendly residual network for end-to-end encrypted FER. The proposed framework is the first CKKS-based residual-network framework for encrypted FER. Guided by encrypted execution cost, HELite-ResNet reconfigures ResNet20 stages and proposes an FHE-friendly HELite Transition to reduce homomorphic computation overhead. Knowledge distillation maintains performance without deployment overhead. Compared with the fastest FIDESlib-ResNet20 baseline, HELite-ResNet reduces average encrypted inference latency by 28.6% across CIFAR-10, FERPlus, and RAF-DB, whilst the encrypted accuracy is even slightly higher with the identical CKKS parameters and peak GPU memory usage.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Fully Homomorphic Encryption, CKKS, Privacy-Preserving Inference, Facial Expression Recognition, Efficient Residual Networks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5yjZOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

A new framework is proposed to work on encrypted face data. The authors propose HELite-ResNet a residual network architecture designed with CKKS in mind. Empirically the work is verified using three datasets, CIFAR-10, FERPlus, and RAF-DB.

Critical strengths

The method is empirically verified on multiple datasets.

The combination of a new model architecture and some form of knowledge distillation seems to be novel. However, this is not my area of expertise so I cannot claim this with certainty.

Critical weaknesses

I will fully admit that as a reviewer, this paper is not in my main area of expertise. Therefore, please consider my critique lightly as I am not intimately familiar with much of the literature in this domain. For example, I could not verify much of the mathematics used in this work.

Issue #1: The threat model is kind of sparsely described. I don’t know if this is normal for this type of literature but I did not have a good understanding of the attacker or the defender from reading what the authors wrote in this subsection. It would be much better if this section was further clarified.

Issue #2: Data alignment. The authors claim their work is targeted for facial expression recognition. However, one of the datasets they use is CIFAR-10. This makes no sense at all. I would like to see the CIFAR-10 dataset removed from the experimental results. In addition, when they do comparisons to other methods, this is done on CIFAR-10 (for example in table 2). I would much prefer that the authors use less method and instead only test on datasets for the given application they are concerned with (faces). Right now, the way the paper reads it is like they say they said they have a new fast horse that is great on land. However, when they run this horse in a race, the race takes place in the ocean. I can kind of see the strength of the “HELite-ResNet” in the results, but it is not abundantly clear (as it should be for an ICLR paper).

Questions for authors
  1. Can you fix the writing of the threat model section?
  2. Can you do a comparison between your architecture and competitors on an actual face dataset?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 5yjZ

Thank you for your constructive comments. Detailed point-by-point responses are provided below.

1. Revision of the Threat Model Description

Thank you for this suggestion. The corresponding revisions to the threat model have already been addressed in our response to Reviewer 1, Comment 2, and incorporated into Sec. 2.3 of the revised manuscript. In addition, we further clarify the scope and limitations of the threat model, including the exclusion of malicious computation, side-channel attacks, and secret-key compromise.

2. Evaluation on Facial Expression Recognition Datasets

Thank you for this suggestion.

(1) Experimental comparison and performance evaluation. The relevant experimental comparisons and performance evaluations have been addressed in our response to Reviewer 1, Comment 3.

(2) Dataset selection. FERPlus and RAF-DB serve as our primary evaluation datasets, while CIFAR-10 is used only as a supplementary encrypted-inference benchmark for comparison with existing homomorphic encryption inference methods. Therefore, our primary experimental conclusions are based on the results on FERPlus and RAF-DB.

Reviewer_Ho61OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HELite-ResNet, a CKKS-friendly residual network for end-to-end encrypted facial expression recognition. The method reduces the number of residual blocks, redesigns cross-resolution transitions through spatial compaction and parameter-free shortcuts, and employs knowledge distillation to preserve classification performance. Experiments on CIFAR-10, FERPlus, and RAF-DB report an average 28.6% reduction in encrypted inference latency compared with the ResNet20 baseline while maintaining comparable accuracy.

Critical strengths

The paper demonstrates measurable reductions in encrypted inference latency using an actual CKKS implementation under matched hardware and cryptographic configurations. The operation-count analysis and transition-level evaluation provide evidence of practical efficiency improvements.

Critical weaknesses
  • The methodological novelty is insufficient for ICLR. The proposed framework primarily combines established techniques, including residual-block reduction, parameter-free downsampling shortcuts, ciphertext compaction, bootstrapping relocation, and knowledge distillation. In particular, the DownsampleA shortcut follows a design already introduced in the original ResNet [1]. More broadly, HE-aware architecture optimization has been investigated in HEMET [2] and AutoFHE [3], while Fhelipe [4] and DaCapo [5] have explored ciphertext-layout optimization and graph-aware bootstrapping management, respectively. Although the specific HELite Transition implementation differs from these methods, the underlying design principles substantially overlap with existing research. The manuscript does not establish a sufficiently distinctive algorithmic or methodological contribution beyond their combination and application to a particular residual-network architecture. The facial expression recognition application alone provides limited additional novelty.
  • The experimental results do not sufficiently establish the benefits of the proposed design over simpler alternatives. The main comparison uses the original ResNet20, whereas HELite-ResNet removes two residual blocks, naturally reducing the number of encrypted operations. Although the paper provides operation-count analyses and plaintext ablations, it does not adequately isolate the end-to-end encrypted latency improvements attributable specifically to the proposed HELite Transition. Comparisons with equally compact residual networks using conventional downsampling shortcuts and existing bootstrapping strategies are needed to establish the benefits of the proposed design. Without such comparisons, the reported efficiency improvements provide limited evidence of a substantial methodological advancement.

[1] K. He et al., "Deep Residual Learning for Image Recognition," CVPR, 2016. [2] Q. Lou and L. Jiang, "HEMET: A Homomorphic-Encryption-Friendly Privacy-Preserving Mobile Neural Network Architecture," ICML, 2021. [3] W. Ao and V. N. Boddeti, "AutoFHE: Automated Adaption of CNNs for Efficient Evaluation over FHE," USENIX Security, 2024. [4] A. Krastev et al., "A Tensor Compiler with Automatic Data Packing for Simple and Efficient Fully Homomorphic Encryption," PLDI, 2024. [5] S. Cheon et al., "DaCapo: Automatic Bootstrapping Management for Efficient Fully Homomorphic Encryption," USENIX Security, 2024.

Questions for authors
  • Can the authors compare HELite-ResNet against an equally compact ResNet using the same number of residual blocks, standard parameter-free downsampling shortcuts, and identical knowledge distillation under the same CKKS implementation? How much additional encrypted inference acceleration is attributable specifically to the proposed HELite Transition?
  • What specific methodological advances distinguish the proposed transition and bootstrapping strategy from existing HE-aware architecture and ciphertext-execution optimizations, beyond the particular implementation choices adopted for ResNet20?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Ho61

Thank you for your constructive comments. Detailed point-by-point responses are provided below.

1. Additional Comparative Experiment

This paper further evaluates the proposed transition main branch on RAF-DB against a Standard-Main-Branch Baseline. Both models share identical residual-block counts, channel widths, DownsampleA shortcuts, distillation settings, CKKS configuration, and hardware.

ModelPlaintext Acc. (%)Mean Encrypted Time (s/image)
Standard-Main-Branch Baseline83.9340.45
HELite-ResNet84.4537.28

Plaintext accuracy is measured on the full RAF-DB test set. To reduce the influence of device-state variations, encrypted inference time is measured on the same 100 images by running the two models sequentially for each image before proceeding to the next. The results show that the proposed transition main branch reduces mean full-network encrypted inference time by 7.84% while improving plaintext accuracy by 0.52 percentage points.

2. Technical Innovations in HELite Transition and Bootstrapping

The detailed structure and ciphertext execution of HELite Transition are illustrated in Fig. 3, while the execution levels and bootstrapping schedules are presented in Appendix C, Fig. C1.

(1) Compact-before-expand. This paper replaces stride-2 convolutional downsampling in the transition main branch with average pooling and ciphertext slot compaction before channel expansion. Subsequent homomorphic convolutions therefore operate on a compact low-resolution layout. Specifically, HELite reorganizes ciphertext slots through parallel data-pair gathering, reducing the multiplicative depth of column compaction while accommodating the additional average-pooling operation (Appendix C, Eq. C5).

(2) Shortcut and ciphertext alignment. The parameter-free DownsampleA shortcut uses spatial sampling, slot compaction, and zero-channel padding to align with the main branch, eliminating the original 1×1 encrypted projection convolution.

(3) Bootstrapping schedule. HELite shares an input bootstrap between the two branches and merges the channel-expanding convolution outputs before refreshing them together. This reduces bootstrapping operations from six to three per transition, as detailed in Appendix C, Fig. C1.

The corresponding changes in rotations and ciphertext–plaintext multiplications are summarized below (Appendix C, Table C2).

OperationLayer2 OriginalLayer2 HELiteLayer3 OriginalLayer3 HELite
Rotations235171389254
Ct–Pt Multiplications71465713681241
Bootstraps6363

Overall, HELite Transition reduces rotations by 27.2% and 34.7% in Layer2 and Layer3, respectively, while reducing bootstrapping operations by 50% in both transitions.

Unlike prior methods that optimize homomorphic encryption architecture, ciphertext layout, or bootstrapping schedules separately, our approach coordinates these operations within cross-resolution residual transitions to reduce their combined ciphertext execution cost.

3. Distinctions from Prior HE Optimization Methods

HEMET demonstrates the importance of HE-aware architecture design through mobile-block selection and coefficient merging, while AutoFHE jointly optimizes polynomial activation degrees and bootstrapping placement. Fhelipe provides automatic ciphertext layout optimization and bootstrapping management, and DaCapo optimizes bootstrapping placement through ciphertext liveness analysis.

**This paper focuses on a different design problem: reducing encrypted inference latency by jointly restructuring residual-network topology and cross-resolution ciphertext execution while preserving recognition accuracy through offline knowledge distillation.

Reviewer_W8yoOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is about using homomorphic encryption scheme to encrypt facial images against unauthorized facial expression recognition (FER). It promotes the use of a CKKS-based residual network for end-to-end encryption of FER.

The authors should define CCKS before using it in the title and abstract and the paper.

From google search, CCKS refers to "a Deep Residual Network (ResNet) architecture tailored for benchmarking tasks provided by the China Conference on Knowledge Graph and Semantic Computing (CCKS)".

The paper is more suitable for a security conference instead of ICLR.

Critical strengths

(+) The problem of protecting facial images against unauthorized facial expression recognition is important.

(+) The use of a homomorphic encryption scheme is a valid and known method for end to end encryption.

(+) The use of teacher Resnet20 model is helpful for offline training.

(+) The paper includes some measurement study.

Critical weaknesses

(-) The paper packed with terminology that is not well recognized by ICLR audiences and readership, such as homomorphic encryption and the China Conference on Knowledge Graph and Semantic Computing (CKKS).

(-) It is unclear what novelty of the proposed CKKS-friendly residual network HELite-ResNet in terms of advancing the state of art in protecting facial images against unauthorized facial expression recognition.

(-) Experimental evaluations are very limited and there is no comparison with the recent solution approaches in the literaturs on protecting facial images against unauthorized facial expression recognition.

(-) The paper is hard to read and it is unconvincing why the use of CKKS-friendly residual network is novel.

(-) The paper is missing discussions on the limitations of using a homomorphic encryption scheme and the limitations of the proposed CKKS-friendly residual network HELite-ResNet.

(-) The evaluation did not include the facial datasets and benchmarks.

Questions for authors

** It is unclear what novelty of the proposed CKKS-friendly residual network HELite-ResNet in terms of advancing the state of art in protecting facial images against unauthorized facial expression recognition.

** Experimental evaluations are very limited and there is no comparison with the recent solution approaches in the literaturs on protecting facial images against unauthorized facial expression recognition.

** Why the use of CKKS-friendly residual network is novel and what threats thiis solution approach tackles and what limitations to the facial expression recognition applications.

** The paper is missing discussions on the limitations of using a homomorphic encryption scheme and the limitations of the proposed CKKS-friendly residual network HELite-ResNet, especially comparing to the STOA facial expression recognition protection solutions in the past 5 years.

** Evaluation benchmarks and datasets are not described. Neither the facial expression recognition threats vs utility tradeoff.

2 comments
AuthorsOpenReview ↗

Response to Reviewer W8yo

Thank you for your constructive comments. Detailed point-by-point responses are provided below.

1. Methodological Novelty and Design Rationale

The motivation for HELite-ResNet is that conventional residual networks incur substantial computational overhead under CKKS encryption, particularly during cross-resolution transitions, channel expansion, and residual-branch alignment. To address these challenges, we develop HELite-ResNet with three coordinated designs:

(1) Ciphertext-cost-aware architecture. We redesign the residual network to reduce expensive encrypted operations while maintaining recognition accuracy.

(2) HELite Transition. We perform spatial downsampling and ciphertext slot compaction before channel expansion, reducing the cost of subsequent homomorphic convolutions.

(3) Coordinated ciphertext execution. We integrate parameter-free shortcuts, input refresh, convolution-output merging, and residual alignment to reduce ciphertext operations and bootstrapping overhead.

Compared with the FIDESlib-ResNet20 baseline under identical CKKS configurations, HELite-ResNet reduces encrypted inference latency by 29.3% on FERPlus and 28.7% on RAF-DB, while improving encrypted accuracy from 84.20% to 84.50% and from 84.10% to 84.30%, respectively (Table 1). These results demonstrate that our design improves encrypted inference efficiency without sacrificing recognition accuracy.

2. Threat Model, Security Guarantees, and Limitations

(1) The threat model considers an honest-but-curious cloud server as the potential adversary, which may infer users' identities or other private information from plaintext facial data. To address this threat, the client encrypts facial images using CKKS before uploading them, enabling the server to perform FER inference without accessing the plaintext images.

(2) The confidentiality of inputs and intermediate data relies on the underlying CKKS cryptographic security assumptions.

(3) The threat model does not cover malicious computation, side-channel attacks, or secret-key compromise. These roles and security boundaries are clarified in revised Sec. 2.3.

3. Comparison with Recent Privacy-Preserving Facial Expression Recognition Methods and Experimental Evaluation

(1) To compare our approach with existing privacy-preserving facial expression recognition methods, we evaluate them in terms of privacy-preserving techniques, recognition accuracy, support for privacy-preserving cloud inference, and inference time. The results are summarized below.

MethodPrivacy-preserving techniqueDataset and accuracyCloud privacy inferenceInference time
Liang et al. (ICME 2023)Low-resolution feature extraction and identity disentanglementRAF-DB: 72.33% → 67.13% (−5.20 percentage points)Yes/
Federated FER (Sensors 2025)FedAvg-based federated learningRAF-DB: 83.6% / 82.8% (centralized/federated)No13.92 ms/image
Xu et al. (WACV 2025)Frequency-based identity suppression and expression-feature compensationCREMA-D: 87.20% → 78.84% (−8.36 percentage points)Yes/
FIDESlib-ResNet20 (Rovida & Leporati, 2024; Agulló-Domingo et al., 2025)CKKS homomorphic encryptionRAF-DB: 84.10% encrypted accuracy; 96.9% plaintext–ciphertext prediction agreementYes49.25 s/image
HELite-ResNetCKKS homomorphic encryptionRAF-DB: 84.30% encrypted accuracy; 97.3% plaintext–ciphertext prediction agreementYes35.11 s/image

(2) Existing methods primarily achieve privacy protection through identity suppression, feature protection, or federated learning. In contrast, HELite-ResNet enables complete encrypted cloud inference using CKKS, protecting facial inputs and intermediate computations.

On FERPlus and RAF-DB, HELite-ResNet achieves encrypted recognition accuracies of 84.50% and 84.30%, respectively, with plaintext–ciphertext prediction agreement rates of 96.2% and 97.3%, respectively, indicating negligible impact of encryption on recognition performance. Compared with FIDESlib-ResNet20, it reduces encrypted inference latency by 29.3% and 28.7% on the two datasets, respectively. Furthermore, the CKKS configuration employed in this study targets 128-bit classical security (Sec. 4.1).

I appreciate the additional experiment and explanation in your rebuttal.

I asked for response to five concerns and the rebuttal above only touches the first three and Even the first three issues raised remain unconvincing. Overall, my concerns about the novelty and contribution of the proposed approach remain. I keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.