acceptodds
Under review as a conference paper at ICLR 2027

RACE-HE: Rotation-Aware Compression for Efficient Private Inference with Homomorphic Encryption

Abstract

Homomorphic encryption (HE) enables privacy-preserving machine learning inference by allowing an untrusted server to compute over encrypted client data. However, its practical deployment remains constrained by the high cost of homomorphic linear layers, particularly the ciphertext rotations required for packed matrix multiplication. This paper presents RACE-HE, a model compression framework for HE-based private inference that restructures weight matrices before encrypted evaluation. RACE-HE approximates selected weight matrices using a rotation-aware diagonal-sparse low-rank (DSLR) decomposition, W ≈ S + AB. RACE-HE exploits the property that S and B can share input rotations, allowing the diagonal term S to recover information lost by the low-rank approximation while limiting additional input rotations. It then applies rotation-aware diagonal pruning, jointly removing packed diagonals that share an input rotation. On ViT-Tiny with DermaMNIST, RACE-HE reduces rotations by 75.4% with a 1.1 percentage point drop in accuracy. On GPT-2 and Llama 3.1-8B, RACE-HE reduces rotations by 29.0% and 34.8%, respectively, with perplexity increases of 2.5 and 1.8, achieving better quality–rotation trade-offs than low-rank and sparseplus-low-rank baselines. When combined with structurally auto-tuned BSGS in OpenFHE, RACE-HE reduces end-to-end ViT-Tiny latency by 24.2%. By restructuring plaintext weight matrices, RACE-HE complements existing HE execution optimizations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Privacy-preserving machine learning, Homomorphic encryption, Model compression, Ciphertext rotation, Private inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AKMeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RACE-HE, combining low-rank decomposition with a diagonal-sparse correction that reuses ciphertext rotations. It further applies rotation-aware diagonal pruning to reduce HE inference costs. Experiments on ViT-Tiny, GPT-2, and Llama 3.1-8B show improved accuracy rotation tradeoffs, with a 24.2% end-to-end latency reduction on ViT-Tiny.

Critical strengths

a. The rotation-sharing strategy between the low-rank and sparse branches is well motivated by HE execution costs. The ablation studies provide reasonable evidence for its effectiveness. b. The authors provide real end-to-end HE evaluation on ViT-Tiny, including bootstrapping and nonlinear operations, rather than relying entirely on theoretical rotation counts.

Critical weaknesses

Limited improvement over compressed baselines. Table 3 reports a 24.2% end-to-end latency reduction over the dense model, but pure SVD already achieves 20.4%. RACE-HE offers only approximately 4.7% additional improvement over pure SVD. Moreover, the comparison lacks an SVD + pruning baseline. It is therefore unclear how much practical benefit comes specifically from the proposed techniques.

Insufficient HE evaluation on larger models. The GPT-2 and Llama 3.1-8B results mainly rely on structural rotation counts and plaintext/simulated ciphertext quality. Actual HE measurements are limited to selected projection-level experiments. It remains unclear whether the reported rotation reductions translate into meaningful end-to-end latency improvements on these models.

Questions for authors

1.Can the authors compare RACE-HE against SVD + pruning under the same OpenFHE setup, with matched accuracy and recovery budgets? 2.Can the authors provide more representative encrypted runtime measurements for GPT-2, beyond a single projection? 3.For the language model baselines, are the skipped dense blocks selected independently for each method, or inherited from RACE-HE? How would independent selection affect the comparison?

3 comments
AuthorsOpenReview ↗

Response to Reviewer AKMe (1/3)

We thank the reviewer for carefully reviewing our submission and providing valuable feedback, and for recognizing our rotation-sharing design and end-to-end HE evaluation. Please see below for our responses to the questions and comments.

 

Q3.1), W3.1) Can the authors compare RACE-HE against SVD + pruning under the same OpenFHE setup, with matched accuracy and recovery budgets? The end-to-end gain over pure SVD in the submitted Table 3 (about 4.7%) also appears limited.

A3.1) We thank the reviewer for suggesting this baseline and for raising this concern. In the submitted Table 3, pure SVD was matched in RdirR_{\mathrm{dir}} to DSLR, the unpruned parent of RACE-HE, but reached lower accuracy than RACE-HE (79.1% vs. 79.9%). Avoidable overhead in the CC-MM and bootstrapping stages, which all configurations share, also diluted the differences among the linear layers. We therefore revised Table 3 in four ways (Table R3.1). First, the baselines are now matched in accuracy, and pure SVD needs a higher rank (RdirR_{\mathrm{dir}} 3,672 instead of 2,824) to reach 79.9%. Second, we added SVD + pruning, which applies the BSGS-aware pruning of RACE-HE, with the same target ρBSGS=0.2\rho_{\mathrm{BSGS}} = 0.2 and the same recovery budget, to a pure-SVD parent whose rank is increased in the same way. Third, we added timings with pre-encoded weights and now report the mean ± SD of three runs. Fourth, we reduced this overhead (Section 4.3 and Appendix C.4).

Within each attention block, the rotations of the K and V ciphertexts are now precomputed once and reused across the CC-MM rotations, common wrap-around rotations are reused, and the attention diagonals are evaluated with BSGS. Skipping the first and last refresh also reduces bootstrapping from 49 to 47 operations. These changes apply identically to all columns and leave the linear layers unchanged. With online encoding, as in the submission, the dense end-to-end time falls from 39,410 s to 32,617 s, whereas the PC-MM latencies of Dense, DSLR, and RACE-HE stay within 0.2% of the submitted values (e.g., 2,040 s vs. 2,036 ± 8 s for RACE-HE).

At matched accuracy and with weights pre-encoded, RACE-HE is 5.7% faster end to end than pure SVD and 10.3% faster than SVD + pruning, with 40.7% and 57.7% lower PC-MM latency, respectively. The 1,403 s gap to pure SVD is more than ten times the run-to-run standard deviation, and it remains 5.1% even if none of the remaining non-PC-MM difference is credited to RACE-HE. Pruning does not help pure SVD at this accuracy. To recover the accuracy lost to pruning, SVD + pruning needs a higher-rank parent and ends with more rotations than pure SVD (RdirR_{\mathrm{dir}} 4,872 vs. 3,672), so pure SVD is the stronger SVD-based baseline.

The end-to-end percentage is smaller than the PC-MM reduction because RACE-HE changes only the linear layers. For pure SVD, PC-MM accounts for 12.4% of the end-to-end time, whereas CC-MM and bootstrapping, which weight compression does not affect, account for 86.4%. The same PC-MM saving would therefore become a larger end-to-end fraction in pipelines where CC-MM and bootstrapping are further accelerated.

Table R3.1. Encrypted ViT-Tiny inference in OpenFHE with BSGS scheduling at matched accuracy (Table 3 in the revised manuscript).

Dense (BSGS)Pure SVDSVD + pruningDSLRRACE-HE
PT accuracy (%)80.179.979.980.079.9
RdirR_{\mathrm{dir}} / CdiagC_{\mathrm{diag}}9,174 / 9,2233,672 / 3,7654,872 / 4,9652,824 / 4,2992,261 / 3,665
Scheduled rotations2,7152,2362,6381,9491,559
PC-MM, pre-encoded (s)9,706 ± 413,081 ± 134,313 ± 132,158 ± 101,826 ± 7
PC-MM, online encoding (s)10,772 ± 473,423 ± 154,792 ± 142,407 ± 112,036 ± 8
CC-MM (s)13,736 ± 6313,701 ± 8913,643 ± 8613,538 ± 6613,503 ± 66
Bootstrapping (s)7,826 ± 327,720 ± 347,820 ± 377,743 ± 347,780 ± 32
Other (s)283 ± 2286 ± 2286 ± 3275 ± 2276 ± 2
End-to-end, pre-encoded (s)31,551 ± 8724,788 ± 107 (−21.4%)26,062 ± 105 (−17.4%)23,714 ± 79 (−24.8%)23,385 ± 81 (−25.9%)
End-to-end, online encoding (s)32,617 ± 9625,130 ± 106 (−23.0%)26,541 ± 100 (−18.6%)23,963 ± 80 (−26.5%)23,595 ± 85 (−27.7%)

Note. Mean ± SD over three runs. Pre-encoded rows exclude online encoding of the plaintext weights. The listed components sum to the end-to-end time, and percentages are relative to Dense.

The table also separates the contribution of each technique. Rotation-aware DSLR decomposition alone is 4.3% faster end to end than pure SVD (30.0% lower PC-MM latency), and rotation-aware pruning on DSLR adds a further 1.4% (15.4% lower PC-MM latency) with a 0.1 pp accuracy change.

AuthorsOpenReview ↗

Response to Reviewer AKMe (2/3)

Q3.2), W3.2) Can the authors provide more representative encrypted runtime measurements for GPT-2, beyond a single projection?

A3.2) We thank the reviewer for this insightful comment. We agree that a single-projection benchmark offers limited evidence of representativeness. Full end-to-end encrypted GPT-2 inference is beyond our current compute budget, since even the dense ViT-Tiny run in Table 3 takes about 9 hours per inference. To provide broader evidence, we extended the OpenFHE benchmark to all 48 Transformer projections of GPT-2. The benchmark covers the fused QKV, attention output, MLP up, and MLP down projections of all 12 blocks, including rectangular operators and the projections of the blocks kept dense. Each projection is evaluated as an independent PC-MM kernel under both the dense model and RACE-HE, with one warm-up and three timed runs. Both use BSGS, with the baby-step sizes selected automatically for each operator. All 288 timed executions passed verification against a plaintext reference.

The RACE-HE model is derived from the BSGS-aware GPT-C3 variant of Table 9 (Table 8 in the submission) by adjusting two MLP-up ranks. BSGS cost changes stepwise with the padded rank rather than smoothly with the rank itself. We therefore truncated the recovered AB of the MLP-up projections of blocks 3 and 4 from ranks 198 and 195 to 30 by SVD, which reduces their padded rank from 256 to 32. S, the biases, and the other 46 projections are unchanged. This adjustment lowers the total rotations by 1.1% (15,924 to 15,750) and the plaintext multiplications by 2.3% (260,640 to 254,620), and raises validation perplexity by 1.2%. It therefore accounts for only a small part of the measured gain.

Summed over all 48 projections, PC-MM latency decreases from 23,643 ± 98 s to 18,733 ± 30 s, a 1.26× speedup (20.8% reduction). The speedup ranges from 1.16× for MLP up to 1.43× for MLP down (Table R3.2).

Table 9 reports an individual block-1 attention-output projection (block 0 in the submission) rather than a model-wide speedup. In the expanded benchmark, the same projection achieves 1.607×, consistent with the previously reported 1.61×. Summing all 12 attention-output projections yields 1.245×, reflecting differences in selected ranks, padded layouts, sparse supports, and dense blocks across the model. For example, the attention outputs of blocks 3, 4, and 6 have ranks 138, 135, and 129, which pad to 256, and run at about 1.00×, 0.96×, and 0.97×. Blocks 11 and 12 are kept dense. Block 1 is thus a favorable case. We retain Table 9 as an illustrative kernel measurement and use the full projection coverage to characterize performance across the model.

These results are sums of independent PC-MM kernels, not end-to-end latency. They exclude bootstrapping and nonlinear layers, so the end-to-end gain for GPT-2 would be smaller, as Table 3 shows for ViT-Tiny (81.2% lower PC-MM latency but 25.9% lower end-to-end latency). We have added these results to Section 4.3 and Table 4, and revised the caption of Table 9 accordingly.

Table R3.2. Measured OpenFHE PC-MM latency across all 48 GPT-2 projections (Table 4 in the revised manuscript).

Projection groupDense (s)RACE-HE (s)Speedup
QKV (12)5,730 ± 404,760 ± 21.20×
Attention output (12)2,186 ± 211,756 ± 211.25×
MLP up (12)7,492 ± 336,445 ± 321.16×
MLP down (12)8,236 ± 435,772 ± 341.43×
All 4823,643 ± 9818,733 ± 301.26×

Note. Each projection is evaluated as an independent encrypted PC-MM kernel with 256-token inputs. Latencies are the mean ± SD of three repeated sums over the independent kernels. Online encoding is included, and setup and bootstrapping are excluded.

AuthorsOpenReview ↗

Response to Reviewer AKMe (3/3)

Q3.3) For the language-model baselines, are the skipped dense blocks selected independently for each method or inherited from RACE-HE? How would independent selection affect the comparison?

A3.3) We thank the reviewer for pointing out an important point. The dense blocks are inherited from RACE-HE. For each language-model configuration, the block-level validation of Section 3.1 is run with the DSLR candidate operators, and pure SVD, ASVD, and pure S + AB keep the same blocks dense (Section 4.1, Table 6). Within the remaining blocks, each baseline chooses its own projection-wise ranks, retaining a projection dense where needed, so that its total RdirR_{\mathrm{dir}} matches the RACE-HE configuration. This allocation is then held fixed during recovery. We share the dense blocks so that all methods compress the same projections, and the quality difference then reflects the decomposition applied to those projections rather than different skipping decisions.

In an additional experiment in which ASVD selects its own dense blocks instead of inheriting those of DSLR, DSLR retains its perplexity advantage. To check that inheriting the blocks does not favor DSLR, we let ASVD, the strongest GPT-2 baseline, run the same block-level validation at GPT-C2, where its margin to DSLR is among the smallest. The tolerance and validation data are the same as for DSLR, and the ranks of the remaining blocks are re-matched to the same RdirR_{\mathrm{dir}} on the same rank grid (Table R3.3). ASVD keeps six blocks dense, as DSLR does, and five of the six coincide with those of DSLR. With its own selection, ASVD reaches 30.51 PPL, compared with 29.73 for DSLR. When ASVD instead allocates ranks over all blocks without dense blocks, it reaches 31.09 PPL. Block selection thus improves ASVD by 0.58 PPL but leaves a 0.78 PPL gap to DSLR. All three runs follow the GPT-2 recovery schedule of Table 7, with progressive block-wise fine-tuning (3, 4, 5, and 7 epochs for blocks 1–3, 4–6, 7–9, and 10–12), followed by 25 epochs of full fine-tuning and 5 epochs of LoRA. We report perplexity on the WikiText-2 test split using 256-token sequences. The DSLR value of 29.73 comes from a separately regenerated GPT-C2 checkpoint. Table 1 reports 29.2 for the original run. Independent selection therefore does not change the comparison at this budget, although this experiment covers one configuration and one seed.

Table R3.3. Independent block selection for ASVD at GPT-C2 (Table 17 in the revised manuscript). All rows have the same RdirR_{\mathrm{dir}} (23,610).

MethodDense blocksTest PPL ↓
DSLR1, 6, 7, 9, 11, 1229.73
ASVD, own block selection1, 5, 6, 9, 11, 1230.51
ASVD, no dense blocksNone31.09

Note. Blocks are numbered from 1, as in Eq. 5.

We have added this experiment to Appendix G.3 and described the protocol in Section 4.1.

 

We hope that the new SVD + pruning baseline, the full 48-projection GPT-2 benchmark, and the independent block-selection experiment address the reviewer's concerns. We would be happy to discuss any further questions during the discussion period.

Reviewer_gfMTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RACE-HE is an offline compression method for CKKS private inference under diagonal packing. It targets the ciphertext rotations in plaintext–ciphertext matrix multiplication (PC-MM).

  • Decomposition. Each projection is approximated as W≈S+ABW \approx S + AB. The sparse wrapped-diagonal correction SS is preferably placed on offsets already used by BB, so the two share rotated inputs.
  • Pruning. Rotation units are pruned by an output-energy score, followed by recovery fine-tuning.
  • Results. The paper reports structural rotation reductions of 75.4% / 29.0% / 34.8% on ViT-Tiny (DermaMNIST), GPT-2 and Llama-3.1-8B (WikiText-2). A full encrypted ViT-Tiny run in OpenFHE shows 81.1% lower PC-MM latency and 24.2% lower end-to-end latency than dense BSGS.
Critical strengths
  1. A simple, HE-specific idea, isolated by a controlled ablation. Placing SS on BB's offsets lets SxSx and BxBx share input rotations. With the schedule, rank grid and recovery protocol held fixed, this lowers RdirR_{\text{dir}} by 18–27% compared with residual-energy-only S+ABS+AB, at similar quality.

  2. An end-to-end encrypted evaluation. Table 3 runs all twelve ViT-Tiny blocks with polynomial non-linearities and 49 bootstraps at 128-bit security. The appendices openly state where the method does not help.

Critical weaknesses
  1. Beyond plain truncated SVD, the measured end-to-end benefit is small, and most of it does not come from the linear layers.

    • Pure SVD at the same RdirR_{\text{dir}} already gives a 20.4% end-to-end reduction, out of RACE-HE's 24.2% (39,410 → 31,369 s), at 0.8 pp lower accuracy (Table 3, l. 382 and 387, p. 8).
    • Against pure SVD, RACE-HE saves 405 s (16.6%) in PC-MM but is only 4.7% faster end-to-end; DSLR alone is 1.5% / 3.9% faster (Table 3, l. 386–387).
    • Non-PC-MM time (l. 387 minus l. 386) comes from an identical 49-bootstrap circuit, yet it varies by ≈1,200 s across columns. It accounts for 97% / 73% of the DSLR / RACE-HE gap to SVD. No variance is reported over the three runs (l. 389–390).
    • Why consequential: the paper's significance rests on encrypted-inference savings, and the evidence does not clearly show a meaningful gain over the simplest baseline.
  2. The quality advantage over low-rank baselines is not robust on ViT-Tiny, the only model evaluated under encryption.

    • Nearly identical structures give different accuracies across tables. Pure SVD scores 77.1% in Table 1 (l. 280, p. 6) but 79.1% in Table 3 (l. 382, p. 8). DSLR scores 79.1% in Tables 1–2 (l. 280, l. 333) but 80.0% in Table 3. The latency configurations "are generated separately from the recovered checkpoints" (l. 517–518, p. 10).
    • These 1–2 pp swings match the ViT margins: DSLR beats SVD by 2.0–3.9 pp and ASVD / S+ABS+AB by 0.7–3.0 pp (Table 1, l. 278–280). All results use a single seed (Table 6 note, l. 1008, p. 19), and no variance is reported in Sec. 4 or App. C–F.
    • The LLM margins (2.0–8.1 PPL, Table 1, l. 282–289) come from a shared recovery protocol (l. 323), and I find them credible. They lack any end-to-end encrypted measurement, however (only single-projection kernels, Tables 4 and 8).
    • Why consequential: only ViT-Tiny supports both the quality claim and the encrypted-latency claim, and on it the quality margin cannot be separated from checkpoint and fine-tuning variance.
Questions for authors
  1. Can you compare RACE-HE with pure SVD at matched accuracy, reporting mean ± std over several timing runs, with weights pre-encoded offline? Sec. 4.3 credits fewer "mask encodings" (l. 454–455, p. 9). Please also explain the non-PC-MM variation. A ≥5% end-to-end gain with non-overlapping variation would raise my rating; a gain within the variation would confirm my concern.

  2. What differs between the ViT checkpoints of Table 1 and Table 3 (l. 517–518, p. 10; App. C.4, l. 963–966, p. 18)? An explanation, plus multi-seed results that preserve DSLR > SVD/ASVD, would make me view the ViT quality claims as sound. Balanced accuracy is preferable, since DermaMNIST is, to my knowledge, class-imbalanced.

  3. Which corpus is used for LLM recovery fine-tuning? Only evaluation "on WikiText-2" is stated (l. 264, p. 5). Was the dense "Pretrained Baseline" (Fig. 3, p. 6) fine-tuned equally? If the PPL deltas largely vanish or grow on a held-out corpus such as C4, my LLM assessment would change accordingly.

  4. Under per-projection key streaming (App. F.1, l. 1202–1209, p. 23), do the requested key subsets reveal each layer's support of the "proprietary" model (App. A, l. 707, p. 14)? Stating which structure is public would resolve this. Otherwise, the claim that RACE-HE "does not expand the adversarial view" (l. 715) should be qualified.

4 comments
AuthorsOpenReview ↗

Response to Reviewer gfMT (1/4)

We sincerely thank the reviewer for the thoughtful review and constructive feedback. We also appreciate the reviewer's recognition of rotation sharing as a simple, HE-specific idea isolated by a controlled ablation. Please see below for our responses to the questions and comments.

 

Q2.1), W2.1) Can the authors compare RACE-HE with pure SVD at matched accuracy, reporting mean ± std over several timing runs with weights pre-encoded offline, and explain the variation in non-PC-MM time?

A2.1) We thank the reviewer for this critical question. Addressing it substantially strengthened the encrypted evaluation of Section 4.3, on which the significance of the paper rests. Table R2.1 shows the revised Table 3, which replaces the submitted one and follows each of the reviewer's requests.

First, the baselines are now matched in accuracy rather than in RdirR_{\mathrm{dir}}. We increase the rank of pure SVD until its PT accuracy reaches that of RACE-HE (79.9%), which raises its RdirR_{\mathrm{dir}} from 2,824 to 3,672. Second, we report latencies with the weights pre-encoded offline, together with the online-encoding setting of the submission. Third, every entry is the mean ± SD of three runs. Fourth, the submitted pipeline left avoidable overhead in the parts shared by all configurations, which diluted the differences among linear operators. Within each attention block, we now precompute the rotations of the K and V ciphertexts once and reuse them across the CC-MM rotations, reuse common wrap-around rotations, and apply BSGS to the attention diagonals. We also skip the first and last refresh, reducing bootstrapping from 49 to 47 operations. These changes apply identically to all five columns and leave every linear operator unchanged. With online encoding, as in the submission, the PC-MM latencies of Dense, DSLR, and RACE-HE stay within 0.2% of the submitted values (e.g., 2,040 s vs. 2,036 ± 8 s for RACE-HE), whereas the dense end-to-end time falls from 39,410 s to 32,617 s.

With weights pre-encoded, RACE-HE is 5.7% faster end to end than pure SVD at matched accuracy (23,385 ± 81 s vs. 24,788 ± 107 s), and the 1,403 s gap is more than ten times either standard deviation. PC-MM latency is 40.7% lower. The end-to-end gap now comes almost entirely from the linear layers. PC-MM accounts for 1,255 s of the 1,403 s gap (89%), whereas in the submitted table it accounted for only 405 s of a 1,485 s gap (27%). Without pre-encoding, the end-to-end gain is 6.1%, so it does not depend on online mask encoding, and we have revised the corresponding sentence of Section 4.3 (l. 454–455 of the submission). DSLR alone is 4.3% faster end to end than pure SVD, and rotation-aware pruning adds a further 1.4%.

We could not isolate a single cause of the earlier variation of about 1,200 s. In the new measurements, each configuration runs on an otherwise idle machine. Across the five columns, the non-PC-MM time now differs by at most 289 s, or 0.9% of the dense end-to-end time. Even if none of this remaining difference is credited to RACE-HE, the PC-MM difference alone gives a 5.1% end-to-end gain over pure SVD.

Table R2.1. Encrypted ViT-Tiny inference in OpenFHE with BSGS scheduling at matched accuracy (Table 3 in the revised manuscript).

Dense (BSGS)Pure SVDSVD + pruningDSLRRACE-HE
PT accuracy (%)80.179.979.980.079.9
RdirR_{\mathrm{dir}} / CdiagC_{\mathrm{diag}}9,174 / 9,2233,672 / 3,7654,872 / 4,9652,824 / 4,2992,261 / 3,665
Scheduled rotations2,7152,2362,6381,9491,559
PC-MM, pre-encoded (s)9,706 ± 413,081 ± 134,313 ± 132,158 ± 101,826 ± 7
PC-MM, online encoding (s)10,772 ± 473,423 ± 154,792 ± 142,407 ± 112,036 ± 8
CC-MM (s)13,736 ± 6313,701 ± 8913,643 ± 8613,538 ± 6613,503 ± 66
Bootstrapping (s)7,826 ± 327,720 ± 347,820 ± 377,743 ± 347,780 ± 32
Other (s)283 ± 2286 ± 2286 ± 3275 ± 2276 ± 2
End-to-end, pre-encoded (s)31,551 ± 8724,788 ± 107 (−21.4%)26,062 ± 105 (−17.4%)23,714 ± 79 (−24.8%)23,385 ± 81 (−25.9%)
End-to-end, online encoding (s)32,617 ± 9625,130 ± 106 (−23.0%)26,541 ± 100 (−18.6%)23,963 ± 80 (−26.5%)23,595 ± 85 (−27.7%)

Note. Mean ± SD over three runs. All columns share the same non-linear pipeline and 47 bootstrapping operations. Baselines are matched to the PT accuracy of RACE-HE. Pre-encoded rows exclude online encoding of the plaintext weights. The listed components sum to the end-to-end time, and percentages are relative to Dense.

We have replaced Table 3 and revised Section 4.3, the abstract, the introduction, and the conclusion accordingly.

AuthorsOpenReview ↗

Response to Reviewer gfMT (2/4)

Q2.2), W2.2) What differs between the ViT checkpoints of Table 1 and Table 3? Do multi-seed results, preferably with balanced accuracy, preserve DSLR > SVD/ASVD?

A2.2) We thank the reviewer for this careful cross-table check and apologize for the confusion.

Table 1 and Table 3 report different compressed operators, not two evaluations of one checkpoint. Table 1 uses DSLR structures selected under the direct-diagonal cost of Section 3.1. Table 3 uses the BSGS-aware variant of Algorithm 1 (Appendix C.4), which keeps the reconstruction constraint against W but minimizes the scheduled cost instead. For DSLR, it keeps the per-projection rank and correction counts of ViT-C3, so RdirR_{\mathrm{dir}} and CdiagC_{\mathrm{diag}} are those of ViT-C3, but it places the correction diagonals on chunk-wise physical diagonals chosen by scheduled BSGS rotations. It also pads the intermediate dimension to a power of two so that S and B share baby-step rotations. The resulting weights therefore differ from the Table 1 checkpoint even though the structural counts are the same. The pure-SVD arm of the submitted Table 3 was selected under the same BSGS cost and matched to the RdirR_{\mathrm{dir}} of the BSGS DSLR operator.

Each Table 3 operator is recovered with the same recipe as Table 1 (Table 7), namely 15 epochs of full-model fine-tuning with seed 42 that keeps the final-epoch checkpoint, so the recovery settings are identical across the two tables. However, matching the aggregate RdirR_{\mathrm{dir}} does not fix how the budget is allocated across projections. For example, the pure-SVD arm of the submitted Table 3 keeps four projections dense, whereas the Table 1 pure SVD factorizes all 48 projections. The cross-table differences therefore reflect different compressed operators and their separate recovery runs, not a change in recovery protocol or BSGS scheduling applied to an unchanged model.

Table 1 evaluates the direct-diagonal cost–quality trade-off, while Table 3 compares execution within one BSGS pipeline. Its PT accuracies certify the timed operators, so quality comparisons should be read within one table. In the revised Table 3, pure SVD is matched in accuracy instead of RdirR_{\mathrm{dir}} (see A2.1).

Multi-seed results. We reran the ViT-Tiny comparison of Table 1 with five recovery seeds (42–46). All runs start from one dense checkpoint with fixed compressed structures and follow the recovery recipe above. Seed 42 reproduces all nine ViT entries of Table 1. Table R2.2 reports the mean and sample SD, and Table R2.3 reports the paired per-seed differences.

Table R2.2. Five-seed recovery results for ViT-Tiny on DermaMNIST at the rotation budgets of Table 1 (Table 13 in the revised manuscript). Entries are mean ± sample SD (%).

Config.MethodRdirR_{\mathrm{dir}}AccuracyBalanced accuracy
ViT-C1DSLR5,00978.9 ± 1.762.7 ± 1.8
SVD5,00876.7 ± 1.255.9 ± 1.3
ASVD5,00979.1 ± 0.760.7 ± 3.1
ViT-C2DSLR3,81478.7 ± 0.759.4 ± 3.5
SVD3,81276.1 ± 1.455.6 ± 2.9
ASVD3,81176.7 ± 0.556.6 ± 2.0
ViT-C3DSLR2,82478.3 ± 0.757.0 ± 2.4
SVD2,82077.2 ± 0.356.6 ± 1.6
ASVD2,82277.4 ± 0.955.7 ± 2.7

Table R2.3. Paired per-seed difference, DSLR − baseline (pp), computed from Table 14 in the revised manuscript. Parentheses give the number of seeds in which DSLR is higher.

Config.Accuracy vs. SVDBalanced accuracy vs. SVDAccuracy vs. ASVDBalanced accuracy vs. ASVD
ViT-C1+2.2 (4/5)+6.8 (5/5)−0.2 (3/5)+2.0 (4/5)
ViT-C2+2.6 (5/5)+3.7 (4/5)+2.0 (5/5)+2.7 (4/5)
ViT-C3+1.1 (5/5)+0.4 (3/5)+0.9 (5/5)+1.3 (3/5)

DSLR has the highest mean accuracy at ViT-C2 and ViT-C3 and the highest mean balanced accuracy at all three budgets. Across the 15 paired runs, DSLR is more accurate than pure SVD in 14 and than ASVD in 13. Paired t-tests (n = 5) give p < 0.05 for ViT-C2 and ViT-C3 accuracy against both baselines and for ViT-C1 balanced accuracy against pure SVD. The other per-configuration differences are not significant at this sample size. In particular, DSLR and ASVD are tied in ViT-C1 accuracy, so the 1.3 pp single-seed margin over ASVD at ViT-C1 in Table 1 is not robust. These results cover recovery-seed variation for one dense checkpoint and do not involve HE noise or latency.

We have revised Section 4.1 to state the ViT-Tiny claim from the five-seed means, under which DSLR exceeds pure SVD at all three budgets and ASVD at ViT-C2 and ViT-C3, and matches ASVD at ViT-C1. The Table 1 note now identifies the ViT entries as seed 42, Appendix G.1 reports the five-seed and per-seed results, and the Table 3 note and Appendix C.4 describe how the timed operators were generated and recovered.

AuthorsOpenReview ↗

Response to Reviewer gfMT (3/4)

Q2.3) Which corpus is used for LLM recovery fine-tuning? Was the dense pretrained baseline fine-tuned equally, and would the perplexity gaps change on a held-out corpus such as C4?

A2.3) We thank the reviewer for this important question. Recovery fine-tuning uses the training split of WikiText-2 (raw, v1). The validation split is used for checkpoint selection and block-level validation, and we report perplexity on the test split. The dense baseline in Table 1 and Fig. 3 is the public pretrained checkpoint and was not fine-tuned on WikiText-2. We agree that its gaps to the compressed models therefore mix the cost of compression with the benefit of adapting to WikiText-2.

Since a model provider often serves a model adapted to its target domain, a dense model fine-tuned on the same data is also a natural reference. We therefore fine-tuned each dense model with the recovery schedule of Table 7 while keeping all projections dense (Dense′), and also applied DSLR with the same recovery to Dense′ at GPT-C1 and Llama-C1. Fine-tuning lowers the dense perplexity from 25.1 to 24.9 for GPT-2 and from 8.5 to 7.8 for Llama 3.1-8B. Against Dense′, the gap of the Table 1 configuration grows from +1.7 to +1.9 perplexity for GPT-C1 and from +1.1 to +1.8 for Llama-C1, so the submitted gaps understated the cost of compression, mainly for Llama 3.1-8B. When compression starts from Dense′, the gaps to Dense′ are +1.6 and +1.7 at rotation reductions of 31.2% and 21.0%.

We also evaluated the Llama 3.1-8B models on C4, which our fine-tuning and checkpoint selection do not use (Table R2.4). Fine-tuning on WikiText-2 alone raises the C4 perplexity of the dense model from 10.8 to 14.3 (+32%), so adapting to the recovery corpus costs quality on other text even without compression. Relative to Dense′, DSLR raises perplexity by 41% on C4 but by 23% on WikiText-2, so recovery restores in-domain quality more than general quality, and the gap to the dense model grows on C4. Even so, at the same RdirR_{\mathrm{dir}}, DSLR keeps the lowest C4 perplexity among the compressed models (20.2 vs. 24.2–27.4), 16–26% below the baselines, similar to its 20–23% margin on WikiText-2.

The matched-budget comparisons of Table 1 do not depend on the dense reference, because all methods start from the same pretrained checkpoint and use the same recovery, and at Llama-C1 they also hold on C4. What changes is the cost relative to the dense model, which we now report against both references. As the C4 results show, the recovery data should reflect the data distribution of the deployment. We have added the corpus and splits to Section 4, marked the dense baseline as the pretrained checkpoint in Table 1 and Fig. 3, reported the gaps against both references in Section 4.1, and added these results and this limitation to Appendix G.4.

Table R2.4. WikiText-2 and C4 perplexity of Llama 3.1-8B at Llama-C1 (Tables 18 and 19 in the revised manuscript).

ModelRdirR_{\mathrm{dir}}WikiText-2 PPL ↓C4 PPL ↓
Dense720,6728.510.8
Dense′720,6727.814.3
Pure SVD570,14212.527.4
ASVD570,14812.024.2
Pure S + AB570,14412.426.9
DSLR570,1509.620.2
Dense′ → DSLR569,5749.521.2

Note. Dense′ is the dense model fine-tuned with the recovery schedule of Table 7, and Dense′ → DSLR applies DSLR and the same recovery to Dense′, so it is fine-tuned twice. The other compressed rows are the Llama-C1 models of Table 1. C4 perplexity is measured on a fixed subset of the C4 validation split.

AuthorsOpenReview ↗

Response to Reviewer gfMT (4/4)

Q2.4) Under per-projection key streaming, do the requested key subsets reveal each layer's support of the proprietary model? If so, the claim that RACE-HE "does not expand the adversarial view" should be qualified.

A2.4) We thank the reviewer for identifying this ambiguity. Yes, when keys are tailored to the compressed model, the key requests reveal structural information to the client, who generates the keys. Under per-projection streaming with direct-diagonal accounting, the client observes Kℓ=(ΩSℓ∪ΩBℓ∪ΩAℓ)∖{0}K_\ell = (\Omega_{S_\ell} \cup \Omega_{B_\ell} \cup \Omega_{A_\ell}) \setminus \{0\} for each projection. From these sets, the client can infer whether each projection is decomposed or dense, its rank and extra correction offsets, and the offsets removed by pruning. The sets do not reveal the weight values or which branch uses a shared offset. In the non-interactive deployment of the main evaluation, the client sees only the union of offsets over all projections, which for GPT-2 equals that of the dense model because the dense output head spans the full offset range (Appendix F.1).

This leakage comes from tailoring keys to the model rather than from the compression itself. RACE-HE adds no messages or rounds, and all rotations are executed on the server. If the client instead generates a key set determined only by public information, namely the layer dimensions and the packing, every RACE-HE rotation is still covered. Under direct-diagonal evaluation, for example, this is the dense model's key set, and the messages the client exchanges then have the same form as in the dense protocol. The rotation and latency results of Section 4 are unchanged in this case. Only the key-transfer savings of Appendix F.1 are lost, since the transfer then equals the dense baseline in Table 12. The same applies to HE-aware diagonal pruning such as Hunter and Artemis whenever keys are tailored to the pruned model.

We have revised Appendix A accordingly. It now states that the architecture, activation degrees, and compression structure are treated as public, while the weight values, the pre-compression weights, and the calibration data remain private. We have also qualified the claim that RACE-HE "does not expand the adversarial view" (l. 715 of the submission). RACE-HE does not expand the client's view when keys are fixed by public information, and tailored keys additionally reveal the compression structure. Appendix F.1 now states that its transfer estimates assume this disclosure.

 

We hope that the matched-accuracy latency measurements, the five-seed results, the fine-tuned dense reference and C4 evaluation, and the clarified key disclosure address the reviewer's concerns. We would be happy to discuss any further questions during the discussion period.

Reviewer_YPBNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a framework (RACE-HE) for homomorphic encryption based efficient inference. Key contributions are: 1. rotation-aware DSLR decomposition where proposed method add a sparse correction S to standard low rank compression approximation. 2. pruning the based on the rotations. The main idea behind contributions is HE property when two linear computations require the same rotation offset on the same input and the server can perform that rotation once and reuse it in the next computation. The paper presents well evaluation on the framework with different models and configurations to support claims made.

Critical strengths
  1. The proposed framework in paper is novel and based on the key property in the HE where two linear computations require the same rotation offset on the same input. papers presents mathematics based backing for the two compression stages of RACE-HE framework. In First stage, it shows how sparse correction S ins added to standard low rank compression (W~ S+AB) and provides detail reasoning about the budget and other details. second stage, it provides the details about the rotation aware diagonal pruning for further compression with reasoning of parameter choices.

The novelty and detailed reasoning support my assessment

  1. The provides results in the evaluation of the framework which useful in privacy preserving ML. It shows the framework improves latency and shows significant reduction in the computation by compression. It shows result ViT (prediction) and two LLMs to back the claims where it shows that 75.4% reduction with 79 % accuracy (ViT) and 29% perplexity (GPT-2) relative to the dense model. Overall evaluation framework is well presented to support the claims.
Critical weaknesses
  1. The results (line 403-408) shows the drop in accuracy (ViT) (<1%) and Increase in perplexity (2.5 to 5) in LLMs case. How the drop would affect if the requirement is very accurate task? maybe the frameworks wound not be helpful in very precise inference. Further, The results are only provided on the relatively small models and it does not provide on recent models (LLMs like kimi, llama, Qwen) with different parameter sizes (16B,70B..etc).
  2. The paper only shows the selected configuration (fig3) and seems like the best ones are shown and there is no explanation about the other configuration (for example 3a at 70% reduction there is point 71 % accuracy). Needs to add the explanation about the other configs.
  3. The fig 4 is not easy to understand and it is the papers main result. It is diffcult to see because of papers method how much improvement is gained and at what cost.
Questions for authors
  1. Can authors show the evaluation on the recent LLM with more parameters to provides support to usefulness in practical settings.
  2. Can authors provide the explanations about the other configurations and why the accuracy drop
6 comments
AuthorsOpenReview ↗

Response to Reviewer YPBN (1/3)

We sincerely thank the reviewer for the thoughtful review and constructive feedback, and for recognizing the novelty of our work. Please see below for our responses to the questions and comments.

 

W1.1a) How would the accuracy drop affect tasks that require very high accuracy? RACE-HE may not be helpful for very precise inference.

A1.1a) We thank the reviewer for raising this point. RACE-HE does not guarantee lossless quality, and whether it helps depends on the accuracy requirement of the target application. However, the quality loss is set by the chosen configuration rather than fixed by the method. The threshold schedule (τstart,α)(\tau_{\mathrm{start}}, \alpha) and the pruning ratio ρ\rho determine the operating point, and Fig. 3 shows the resulting range of quality–rotation trade-offs. Raising the reconstruction threshold moves each projection toward its dense form, and a projection keeps its original weights when no candidate meets the threshold with fewer rotations (Appendix C.2). The dense model is therefore the limiting case of the same procedure.

For tasks with strict accuracy requirements, a milder operating point can be chosen from this trade-off, as we do for the ViT-Tiny configurations in Table 1. We select these configurations by validation accuracy, without using the test set. For example, ViT-C1 reduces rotations by 45.4% while reaching 78.9±1.7%78.9 \pm 1.7\% test accuracy over five recovery seeds (Appendix G.1), compared with 80.1% for the dense model. Encrypted execution adds little further loss, since the estimated CT quality in Table 1 differs from the PT quality by at most 0.1 in every configuration. If no validated configuration meets an application's requirement, the dense model remains the appropriate choice. A joint allocation of ranks and diagonal structures across layers, informed by activation sensitivity, could improve this trade-off and extend RACE-HE to stricter requirements. We have added this discussion of accuracy requirements to Appendix F.2.

 

W1.1b), Q1.1) Can the authors evaluate a recent LLM with more parameters to support the usefulness of RACE-HE in practical settings?

A1.1b) We thank the reviewer for this suggestion. We added an evaluation of Qwen3-32B [1], about four times the parameter count of Llama 3.1-8B (Appendix E.2, Table 11). We apply DSLR decomposition at (τstart,α)=(0.80,0.15)(\tau_{\mathrm{start}}, \alpha) = (0.80, 0.15) to all 448 attention and MLP projections in its 64 Transformer blocks, without block skipping or pruning. Recovery trains rank-32 LoRA adapters on A and B for five epochs on the WikiText-2 training split with S frozen, and we select the checkpoint with the lowest validation perplexity. Relative to the dense model, DSLR reduces direct rotations by 48.2% (1,769,024 to 915,773) and active diagonals by 24.5% (1,769,472 to 1,335,256), while test perplexity increases from 10.7 to 12.2 (Table R1.1).

The decomposition thus carries over to this scale unchanged. It nearly halves the direct rotations, and the diagonal correction is active in 444 of the 448 projections. The rotation reduction exceeds the active-diagonal reduction because most correction diagonals lie on offsets that B already rotates, so they add plaintext multiplications but no rotations. Since the adapters are attached to A and B with S frozen, merging them leaves the supports and both counts unchanged.

Since recovery at this scale is costly, we evaluated a single operating point within our time and compute budget, without the matched baselines of Table 1 or encrypted evaluation. We have added these results to Appendix E.2 and referenced them from Section 4.1.

Table R1.1. Direct-diagonal cost and WikiText-2 test perplexity of Qwen3-32B at one DSLR operating point, (τstart,α)=(0.80,0.15)(\tau_{\mathrm{start}}, \alpha) = (0.80, 0.15) with ρ=0\rho = 0 (Table 11 in the revised manuscript).

ModelRdirR_{\mathrm{dir}}CdiagC_{\mathrm{diag}}Test PPL ↓
Dense1,769,0241,769,47210.7
DSLR915,773 (48.2%↓)1,335,256 (24.5%↓)12.2

Note. Counts cover the 448 attention and MLP projections of the 64 Transformer blocks under direct-diagonal accounting, as in Table 1. The DSLR row reports the checkpoint with the lowest validation perplexity over five LoRA epochs. The dense model is not fine-tuned. Perplexities are not comparable across models with different tokenizers.

 

[1] An Yang, Anfeng Li, Baosong Yang, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.

AuthorsOpenReview ↗

Response to Reviewer YPBN (2/3)

W1.2), Q1.2) Fig. 3 seems to show only the best configurations. Please explain the other configurations and why their accuracy drops.

A1.2) We thank the reviewer for pointing out the accuracy variation among configurations with similar rotation reductions. The resulting analysis helped us clarify this aspect of the results.

Fig. 3 shows every configuration of the decomposition sweep, including the lower-accuracy points. The stars mark the configurations carried to Table 1, selected to cover different reduction levels with good quality at each level, so the other points are part of the reported trade-off rather than excluded results. Near 70% reduction, we believe the reviewer refers to (0.65,0.25)(0.65, 0.25), at 69.5% reduction and 73.0% accuracy.

The variation arises because similar aggregate rotation counts do not imply similar allocations across Transformer blocks. The schedule (τstart,α)(\tau_{\mathrm{start}}, \alpha) sets a reconstruction threshold that decreases linearly with depth (Eq. 5), so different settings can reach the same reduction with different per-block ranks and diagonal structures. Since blocks differ in their sensitivity to approximation error, this can change both the accuracy and how well recovery restores it.

To examine this, we compared (0.65,0.25)(0.65, 0.25) with (0.60,0.15)(0.60, 0.15), which is ViT-C3 (Table R.2). They reach nearly identical rotation reductions (69.5% vs. 69.2%) and average ranks (28.9 vs. 29.0), but their recovered accuracies differ substantially (73.0% vs. 79.1%). A follow-up evaluation gives a similar gap (73.2% vs. 79.0%). The gap also far exceeds the recovery-seed variation of ViT-C3, which reaches 78.3±0.7%78.3 \pm 0.7\% over five seeds (Appendix G.1).

Under Eq. 5, the (0.65,0.25)(0.65, 0.25) schedule is stricter in early blocks and looser in late ones, and the two schedules cross at block 6. Table R.3 shows the resulting allocation. The average rank of (0.65,0.25)(0.65, 0.25) is higher at block 1 (33.8 vs. 30.8), equal in blocks 5–7, and lower at block 12 (23.3 vs. 26.3). Restoring the dense weights of a single block without further fine-tuning lowers validation accuracy in all twelve blocks of (0.65,0.25)(0.65, 0.25) and in ten of (0.60,0.15)(0.60, 0.15), which suggests that the recovered blocks co-adapt. Accuracy thus depends on the interaction between allocation and recovery, although these diagnostics do not attribute the gap to individual blocks.

Table R1.2. Two ViT-Tiny configurations with similar direct-rotation budgets but different recovered accuracy.

(τstart,α)(\tau_{\mathrm{start}}, \alpha)RdirR_{\mathrm{dir}}CdiagC_{\mathrm{diag}}kˉ\bar{k}Rotation reduction (%)Sweep test accuracy (%)Follow-up test accuracy (%)
(0.65, 0.25)2,8024,26228.969.573.073.2
(0.60, 0.15)2,8244,29929.069.279.179.0

Note. Accuracies are measured after recovery fine-tuning. The follow-up evaluates a rerun of (0.65, 0.25) and a saved recovery export of (0.60, 0.15), both produced with the sweep recipe (seed 42, 15 recovery epochs). The same two checkpoints are used in Table R.3.

Table R1.3. Block-wise compression allocation and dense-weight restoration diagnostics. A denotes (0.65, 0.25) and B denotes (0.60, 0.15).

Blockkˉb\bar{k}_b (A)DbpostFTD_b^{\mathrm{postFT}} (A, %)ΔVal. (A, pp)kˉb\bar{k}_b (B)DbpostFTD_b^{\mathrm{postFT}} (B, %)ΔVal. (B, pp)
133.861.2−2.830.864.4−2.2
231.563.0−3.528.565.4−3.0
333.864.7−10.230.866.5−6.6
433.066.3−0.131.567.4−4.1
531.567.8−0.731.568.3−1.5
630.069.1−2.830.069.1−0.3
729.370.5−4.429.370.1−1.2
824.872.0−0.925.570.9+1.2
927.073.6−0.328.571.90.0
1025.574.9−4.028.572.6−0.1
1124.076.2−1.527.073.4−0.9
1223.377.6−1.926.374.3−1.8

Note. Blocks are indexed as in Eq. 5. kˉb\bar{k}_b averages the ranks of the block's four projections. DbpostFTD_b^{\mathrm{postFT}} is the normalized Frobenius distance between the recovered and original dense weights, aggregated over the four projections. ΔVal. is the change in validation accuracy after restoring only that block's dense weights, without additional fine-tuning.

We have added a brief explanation to Section 4.1 and these analyses to Appendix G.2.

AuthorsOpenReview ↗

Response to Reviewer YPBN (3/3)

W1.3) Fig. 4 is hard to read, so it is difficult to see how much improvement RACE-HE provides and at what cost.

A1.3) Thank you for this helpful feedback. We agree that Fig. 4 was difficult to read, especially since it presents one of our main results. We have revised the figure to make it easier to interpret. For each decomposition configuration, the most pruned point (ρ=0.2\rho = 0.2) is now annotated with the quality difference between RACE-HE and dense diagonal pruning at the matched budget, so the gain can be read directly against its cost in rotation reduction on the x-axis. The decomposition configuration and pruning ratio of each point are also indicated through the legend and color coding. The revised figure is included in the updated manuscript.

 

We hope these responses address the reviewer's concerns, and we would be glad to clarify any remaining questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.