- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RACE-HE, combining low-rank decomposition with a diagonal-sparse correction that reuses ciphertext rotations. It further applies rotation-aware diagonal pruning to reduce HE inference costs. Experiments on ViT-Tiny, GPT-2, and Llama 3.1-8B show improved accuracy rotation tradeoffs, with a 24.2% end-to-end latency reduction on ViT-Tiny.
Critical strengths
a. The rotation-sharing strategy between the low-rank and sparse branches is well motivated by HE execution costs. The ablation studies provide reasonable evidence for its effectiveness. b. The authors provide real end-to-end HE evaluation on ViT-Tiny, including bootstrapping and nonlinear operations, rather than relying entirely on theoretical rotation counts.
Critical weaknesses
Limited improvement over compressed baselines. Table 3 reports a 24.2% end-to-end latency reduction over the dense model, but pure SVD already achieves 20.4%. RACE-HE offers only approximately 4.7% additional improvement over pure SVD. Moreover, the comparison lacks an SVD + pruning baseline. It is therefore unclear how much practical benefit comes specifically from the proposed techniques.
Insufficient HE evaluation on larger models. The GPT-2 and Llama 3.1-8B results mainly rely on structural rotation counts and plaintext/simulated ciphertext quality. Actual HE measurements are limited to selected projection-level experiments. It remains unclear whether the reported rotation reductions translate into meaningful end-to-end latency improvements on these models.
Questions for authors
1.Can the authors compare RACE-HE against SVD + pruning under the same OpenFHE setup, with matched accuracy and recovery budgets? 2.Can the authors provide more representative encrypted runtime measurements for GPT-2, beyond a single projection? 3.For the language model baselines, are the skipped dense blocks selected independently for each method, or inherited from RACE-HE? How would independent selection affect the comparison?
3 comments
Response to Reviewer AKMe (2/3)
Q3.2), W3.2) Can the authors provide more representative encrypted runtime measurements for GPT-2, beyond a single projection?
A3.2) We thank the reviewer for this insightful comment. We agree that a single-projection benchmark offers limited evidence of representativeness. Full end-to-end encrypted GPT-2 inference is beyond our current compute budget, since even the dense ViT-Tiny run in Table 3 takes about 9 hours per inference. To provide broader evidence, we extended the OpenFHE benchmark to all 48 Transformer projections of GPT-2. The benchmark covers the fused QKV, attention output, MLP up, and MLP down projections of all 12 blocks, including rectangular operators and the projections of the blocks kept dense. Each projection is evaluated as an independent PC-MM kernel under both the dense model and RACE-HE, with one warm-up and three timed runs. Both use BSGS, with the baby-step sizes selected automatically for each operator. All 288 timed executions passed verification against a plaintext reference.
The RACE-HE model is derived from the BSGS-aware GPT-C3 variant of Table 9 (Table 8 in the submission) by adjusting two MLP-up ranks. BSGS cost changes stepwise with the padded rank rather than smoothly with the rank itself. We therefore truncated the recovered AB of the MLP-up projections of blocks 3 and 4 from ranks 198 and 195 to 30 by SVD, which reduces their padded rank from 256 to 32. S, the biases, and the other 46 projections are unchanged. This adjustment lowers the total rotations by 1.1% (15,924 to 15,750) and the plaintext multiplications by 2.3% (260,640 to 254,620), and raises validation perplexity by 1.2%. It therefore accounts for only a small part of the measured gain.
Summed over all 48 projections, PC-MM latency decreases from 23,643 ± 98 s to 18,733 ± 30 s, a 1.26× speedup (20.8% reduction). The speedup ranges from 1.16× for MLP up to 1.43× for MLP down (Table R3.2).
Table 9 reports an individual block-1 attention-output projection (block 0 in the submission) rather than a model-wide speedup. In the expanded benchmark, the same projection achieves 1.607×, consistent with the previously reported 1.61×. Summing all 12 attention-output projections yields 1.245×, reflecting differences in selected ranks, padded layouts, sparse supports, and dense blocks across the model. For example, the attention outputs of blocks 3, 4, and 6 have ranks 138, 135, and 129, which pad to 256, and run at about 1.00×, 0.96×, and 0.97×. Blocks 11 and 12 are kept dense. Block 1 is thus a favorable case. We retain Table 9 as an illustrative kernel measurement and use the full projection coverage to characterize performance across the model.
These results are sums of independent PC-MM kernels, not end-to-end latency. They exclude bootstrapping and nonlinear layers, so the end-to-end gain for GPT-2 would be smaller, as Table 3 shows for ViT-Tiny (81.2% lower PC-MM latency but 25.9% lower end-to-end latency). We have added these results to Section 4.3 and Table 4, and revised the caption of Table 9 accordingly.
Table R3.2. Measured OpenFHE PC-MM latency across all 48 GPT-2 projections (Table 4 in the revised manuscript).
| Projection group | Dense (s) | RACE-HE (s) | Speedup |
|---|---|---|---|
| QKV (12) | 5,730 ± 40 | 4,760 ± 2 | 1.20× |
| Attention output (12) | 2,186 ± 21 | 1,756 ± 21 | 1.25× |
| MLP up (12) | 7,492 ± 33 | 6,445 ± 32 | 1.16× |
| MLP down (12) | 8,236 ± 43 | 5,772 ± 34 | 1.43× |
| All 48 | 23,643 ± 98 | 18,733 ± 30 | 1.26× |
Note. Each projection is evaluated as an independent encrypted PC-MM kernel with 256-token inputs. Latencies are the mean ± SD of three repeated sums over the independent kernels. Online encoding is included, and setup and bootstrapping are excluded.
Response to Reviewer AKMe (3/3)
Q3.3) For the language-model baselines, are the skipped dense blocks selected independently for each method or inherited from RACE-HE? How would independent selection affect the comparison?
A3.3) We thank the reviewer for pointing out an important point. The dense blocks are inherited from RACE-HE. For each language-model configuration, the block-level validation of Section 3.1 is run with the DSLR candidate operators, and pure SVD, ASVD, and pure S + AB keep the same blocks dense (Section 4.1, Table 6). Within the remaining blocks, each baseline chooses its own projection-wise ranks, retaining a projection dense where needed, so that its total matches the RACE-HE configuration. This allocation is then held fixed during recovery. We share the dense blocks so that all methods compress the same projections, and the quality difference then reflects the decomposition applied to those projections rather than different skipping decisions.
In an additional experiment in which ASVD selects its own dense blocks instead of inheriting those of DSLR, DSLR retains its perplexity advantage. To check that inheriting the blocks does not favor DSLR, we let ASVD, the strongest GPT-2 baseline, run the same block-level validation at GPT-C2, where its margin to DSLR is among the smallest. The tolerance and validation data are the same as for DSLR, and the ranks of the remaining blocks are re-matched to the same on the same rank grid (Table R3.3). ASVD keeps six blocks dense, as DSLR does, and five of the six coincide with those of DSLR. With its own selection, ASVD reaches 30.51 PPL, compared with 29.73 for DSLR. When ASVD instead allocates ranks over all blocks without dense blocks, it reaches 31.09 PPL. Block selection thus improves ASVD by 0.58 PPL but leaves a 0.78 PPL gap to DSLR. All three runs follow the GPT-2 recovery schedule of Table 7, with progressive block-wise fine-tuning (3, 4, 5, and 7 epochs for blocks 1–3, 4–6, 7–9, and 10–12), followed by 25 epochs of full fine-tuning and 5 epochs of LoRA. We report perplexity on the WikiText-2 test split using 256-token sequences. The DSLR value of 29.73 comes from a separately regenerated GPT-C2 checkpoint. Table 1 reports 29.2 for the original run. Independent selection therefore does not change the comparison at this budget, although this experiment covers one configuration and one seed.
Table R3.3. Independent block selection for ASVD at GPT-C2 (Table 17 in the revised manuscript). All rows have the same (23,610).
| Method | Dense blocks | Test PPL ↓ |
|---|---|---|
| DSLR | 1, 6, 7, 9, 11, 12 | 29.73 |
| ASVD, own block selection | 1, 5, 6, 9, 11, 12 | 30.51 |
| ASVD, no dense blocks | None | 31.09 |
Note. Blocks are numbered from 1, as in Eq. 5.
We have added this experiment to Appendix G.3 and described the protocol in Section 4.1.
We hope that the new SVD + pruning baseline, the full 48-projection GPT-2 benchmark, and the independent block-selection experiment address the reviewer's concerns. We would be happy to discuss any further questions during the discussion period.
Response to Reviewer AKMe (1/3)
We thank the reviewer for carefully reviewing our submission and providing valuable feedback, and for recognizing our rotation-sharing design and end-to-end HE evaluation. Please see below for our responses to the questions and comments.
Q3.1), W3.1) Can the authors compare RACE-HE against SVD + pruning under the same OpenFHE setup, with matched accuracy and recovery budgets? The end-to-end gain over pure SVD in the submitted Table 3 (about 4.7%) also appears limited.
A3.1) We thank the reviewer for suggesting this baseline and for raising this concern. In the submitted Table 3, pure SVD was matched in Rdir to DSLR, the unpruned parent of RACE-HE, but reached lower accuracy than RACE-HE (79.1% vs. 79.9%). Avoidable overhead in the CC-MM and bootstrapping stages, which all configurations share, also diluted the differences among the linear layers. We therefore revised Table 3 in four ways (Table R3.1). First, the baselines are now matched in accuracy, and pure SVD needs a higher rank (Rdir 3,672 instead of 2,824) to reach 79.9%. Second, we added SVD + pruning, which applies the BSGS-aware pruning of RACE-HE, with the same target ρBSGS=0.2 and the same recovery budget, to a pure-SVD parent whose rank is increased in the same way. Third, we added timings with pre-encoded weights and now report the mean ± SD of three runs. Fourth, we reduced this overhead (Section 4.3 and Appendix C.4).
Within each attention block, the rotations of the K and V ciphertexts are now precomputed once and reused across the CC-MM rotations, common wrap-around rotations are reused, and the attention diagonals are evaluated with BSGS. Skipping the first and last refresh also reduces bootstrapping from 49 to 47 operations. These changes apply identically to all columns and leave the linear layers unchanged. With online encoding, as in the submission, the dense end-to-end time falls from 39,410 s to 32,617 s, whereas the PC-MM latencies of Dense, DSLR, and RACE-HE stay within 0.2% of the submitted values (e.g., 2,040 s vs. 2,036 ± 8 s for RACE-HE).
At matched accuracy and with weights pre-encoded, RACE-HE is 5.7% faster end to end than pure SVD and 10.3% faster than SVD + pruning, with 40.7% and 57.7% lower PC-MM latency, respectively. The 1,403 s gap to pure SVD is more than ten times the run-to-run standard deviation, and it remains 5.1% even if none of the remaining non-PC-MM difference is credited to RACE-HE. Pruning does not help pure SVD at this accuracy. To recover the accuracy lost to pruning, SVD + pruning needs a higher-rank parent and ends with more rotations than pure SVD (Rdir 4,872 vs. 3,672), so pure SVD is the stronger SVD-based baseline.
The end-to-end percentage is smaller than the PC-MM reduction because RACE-HE changes only the linear layers. For pure SVD, PC-MM accounts for 12.4% of the end-to-end time, whereas CC-MM and bootstrapping, which weight compression does not affect, account for 86.4%. The same PC-MM saving would therefore become a larger end-to-end fraction in pipelines where CC-MM and bootstrapping are further accelerated.
Table R3.1. Encrypted ViT-Tiny inference in OpenFHE with BSGS scheduling at matched accuracy (Table 3 in the revised manuscript).
Note. Mean ± SD over three runs. Pre-encoded rows exclude online encoding of the plaintext weights. The listed components sum to the end-to-end time, and percentages are relative to Dense.
The table also separates the contribution of each technique. Rotation-aware DSLR decomposition alone is 4.3% faster end to end than pure SVD (30.0% lower PC-MM latency), and rotation-aware pruning on DSLR adds a further 1.4% (15.4% lower PC-MM latency) with a 0.1 pp accuracy change.