- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a practical architecture method for implementing selective homomorphic encryption in privacy-preserving text classification. Its main advantage lies in demonstrating that sensitive tokens can be integrated by means of an encrypted post-fusion module, without fully encrypting the Transformer inference. This research has a solid empirical foundation and the implementation results are encouraging.
Critical strengths
s2. This paper discusses the key trade-offs between privacy, predictive utility, and computational efficiency. Its later fusion architecture separates plaintext context processing from encrypted sensitive token processing, and limits expensive homomorphic operations to a lightweight merging module. This design choice is of great significance, providing a practical alternative for only performing masking inference and fully encrypting the Transformer.
s1. The evaluation covers 13 text classification datasets, two Transformer backbone networks, and multiple controlled benchmark methods, including ZEROZ ablation experiments, which helps to distinguish the contributions of private token information from the additional capacity of the model. It is notable that the authors implement the encrypted merging and full-layer homomorphic encryption benchmarks using GPU-based CKKS technology, rather than relying solely on plaintext simulation.
Critical weaknesses
w1. The speed improvement in the report is impressive compared to the fully encrypted Transformer, but the comparison is not entirely fair. Full-layer homomorphic encryption (HE) protects all input tokens and processes the entire Transformer, while LFT only reveals the masked context and mask positions, encrypts only the selected token embeddings, and updates the classification representation through a single merge operation. Additionally, the full-layer baseline model calculates some output positions that are not needed in classification, and the authors have acknowledged this. Therefore, the significant speed improvement mainly reflects the advantages of adopting different privacy and computational trade-offs, rather than achieving the same performance improvement under the same privacy guarantee.
w2. Since the sensitive tokens are excluded from the main body of the Transformer and only introduced through the final cross-attention module, they cannot participate in the multi-layer context interaction provided by the standard Transformer. This may limit its performance in tasks that require more complex relationships between public and private tokens, which explains the weak recovery ability on the MedNLI dataset and the decrease in accuracy on the MRPC task. Moreover, the experiments on the merge operation indicate that removing the softmax operation usually only leads to a small loss in accuracy, but the paper did not report the corresponding encryption inference cost.
Questions for authors
See weaknesses.
Response to Reviewer ATNs
We thank the reviewer for the positive assessment and the constructive comments. We address both weaknesses in order.
W1. Comparison under different privacy guarantees
We agree that LFT and full-layer HE make different privacy trade-offs (Section 6). The revision adds two things.
W2 (a). Multi-layer interaction between public and private tokens
We tested this explanation with the same experiment.
On the eleven single-text datasets, running more layers under encryption adds little over LFT. MedNLI and MRPC gain more: 7.0 pp on average with Insert-10 and 8.3 pp with Insert-12. These results support your explanation for MedNLI, where Insert-k is significantly more accurate than LFT only with ten or more layers. On MRPC, Insert-1 is already about 3 pp more accurate than LFT. The cost grows roughly linearly with k.
W2 (b). Encrypted cost without the softmax
Appendix E (Table 10) now reports this cost. The implementation and the measurement procedure are the same as for the cross-attention merger.
For a single input, the softmax takes most of the time, so removing it saves a lot. Under batching, one softmax serves 32 inputs, and the saving is smaller. Cross-attention is more accurate than the linear variant on 12 of the 13 datasets (Table 9), so we keep the softmax.
We hope these responses are helpful, and we are glad to discuss any remaining points.