acceptodds
Withdrawn from ICLR 2027

BEYOND MATRIX ARITHMETIC: RETHINKING LINEAR PROJECTIONS FOR HOMOMORPHIC TRANSFORMERS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fully homomorphic encryption (FHE) enables Transformer inference on en- crypted data, but its high execution cost limits practical deployment. Existing systems optimize packed computation, GPU kernels, and encoded-weight stor- age, yet weight preparation remains costly when projection layouts require many encoded operands and redundant expansion. Limited GPU memory further leads to repeated encoding when prepared weights cannot be retained across calls. Mo- tivated by profiling that reveals substantial weight-preparation costs, we rethink linear projection execution in GPU CKKS by jointly organizing layouts, prepa- ration, and retention. Phase-specific plans avoid full-slot weight expansion dur- ing prefill and reduce encoded operand counts through rectangular vector–matrix computation during decode, while preserving compatibility with adjacent oper- ators. We separate cross-call retention from current-batch execution: selected complete projection weight sets remain cached, while other weights are prepared in temporary batches without evicting retained sets. Cached and newly prepared weights share a batched accumulation kernel that directly consumes compact en- coded operands. On Llama-3-8B with an A100 GPU, 128-token first-layer prefill latency decreases from 1099.3 to 303.7 s, a 72.4% reduction. Across the first three layers at 129 keys, average decode latency is 16.0 s per layer, compared with 65.8 s for a same-backend Cachemir-style integration, including required in- terface adaptations. Together with component ablations, these results show that reducing weight preparation and preserving encoded-weight reuse are effective complements to optimizing encrypted arithmetic.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Fully Homomorphic Encryption, Privacy-Preserving Transformer Inference, Bootstrapping Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rxAHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a system optimization for reducing the cost of CKKS in transformer-based models. The proposed techniques are evaluated on the Llama3-8B model.

Critical strengths
  • I believe this paper tries to address an important problem because being able to effectively apply CKKS to transformer models would be a kind of milestone of end-to-end privacy-preserving ML applications.
Critical weaknesses
  • I'm not sure I have fully understood the contributions, specifically to ICLR. The meat of methodology, i.e. Section 4, mostly presents something already existing, such as Cerium and fused MAC.

  • Figure 2 does not look very professional

  • The evaluation could be improved. For example, when testing prefill, only the first layer is evaluated, and we have no idea how much performance overhead for the end-to-end execution. Also, only one backbone model is being tested, i.e. Llama3-8B.

  • I also believe some important references were missing.

Questions for authors

I think this paper needs a lot of work before publication. I hope my comments in the "Critical Weaknesses" could help the authors to improve the work. I don't have more specific questions beyond my comments above.

Reviewer_caGPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the cost of preparing and repeatedly encoding plaintext weights for GPU-based homomorphic Transformer inference. The problem is real: weight preparation can take substantial time, and storing encoded weights can require considerable memory. The proposed design combines phase-specific projection layouts, period-sized weight preparation, selective caching of complete projection weight sets, and a shared batched accumulation kernel. The evaluation uses Llama-3-8B on an A100, measuring prefill for the first layer and decoding across the first three layers, supported by component ablations and operator-level numerical checks. The authors report approximately 72% lower first-layer prefill latency than their baseline and roughly 4.1× faster decoding than a local Cachemir-style integration. These measurements cover individual layers rather than full-model generation.

Critical strengths
  1. The paper identifies a useful opportunity to reduce weight-preparation costs. With the blocked layout held fixed, periodic preparation lowers latency without changing the number of encodings or product terms. This supports optimizing operand generation, although the profiling does not fully separate preparation work from synchronization waits or show that the same bottleneck dominates other implementations.

  2. The retention experiments provide evidence that selective caching reduces repeated encoding. Under the same cache limit, retaining complete projection weight sets avoids more encoding work than either entrywise LRU or no caching. Tests using the same key confirm that caching and batched accumulation preserve ciphertext payloads and metadata for identical representations and encoding states. These checks support the reuse mechanism, though its performance benefit is demonstrated only over a short decode sequence.

Critical weaknesses
  1. The comparisons do not clearly distinguish algorithmic improvements from baseline implementation overhead. Most of the reported decode advantage comes from avoiding 53.253 seconds of adapter and conversion work in the local Cachemir-style port, rather than faster matrix execution. The authors acknowledge that this overhead is not intrinsic to Cachemir, leaving unclear how much advantage would remain against a better-integrated baseline. The THOR/MOAI prefill comparisons provide similarly limited evidence because they use extrapolated cost estimates, including an assumed 8× factor for MOAI, rather than complete measured runs.

    The prefill baseline is reported as 1099.3 s in the abstract and Section 5.2, but as 1093.9 s in Table 1. Please reconcile these values and clarify which measurement supports the reported 72.4% reduction.

  2. The methodological novelty is limited, and the benefit of combining the existing techniques is insufficiently demonstrated. Periodic compression builds on Cerium, batched accumulation is adapted from Cheddar, and specialized VMM and encoded-weight caching are established approaches. The main addition is coordinating these techniques through projection-level retention and temporary batching, with a deterministic first-fit admission policy. Yet retention gains are measured only on the next decode call across three layers, and batching yields no observed latency improvement in the reported comparison. Correctness coverage is also incomplete: Section 5.5 states that numerical validation does not cover the blocked-to-rectangular performance comparisons. These results support useful implementation improvements, but do not convincingly establish a substantial methodological advance or a sustained benefit from the proposed coordination.

Questions for authors
  1. How much of the encoding API time in Section 3.1 is attributable to encoding, data transfer, and synchronization waits? How did you account for any overlap between them?

  2. Which adapter and conversion costs in the local Cachemir-style implementation are required by its layout, and which are specific to the port? What performance advantage remains after removing avoidable port-specific overhead?

  3. Were both layouts in the blocked-to-rectangular performance comparisons numerically validated against the same plaintext reference? What evidence shows that retention gains persist over multiple decode steps and a larger set of layers?

Reviewer_uxa1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the cost of preparing and reusing plaintext model weights in GPU-based CKKS Transformer inference with encrypted activations. The authors combine phase-specific projection layouts and period-sized weight preparation with selective retention of complete projection weight sets; retained and temporary operands share an accumulation path. Using a Phantom-based backend on an A100 GPU, they evaluate Llama-3-8B on 128-token first-layer prefill and per-layer decoding averaged over the first three layers at 129 keys, focusing on these execution regions rather than end-to-end generation.

Critical strengths

The paper identifies weight preparation as a major performance bottleneck beyond encrypted matrix arithmetic. Its coordinated design balances encoded-weight reuse with GPU memory constraints, reducing repeated preparation costs. The experiments demonstrate substantial latency improvements in the evaluated execution regions, while the ablations provide useful insights into the effects of projection layouts, weight preparation, and caching strategies.

Critical weaknesses
  1. Table 2(b) compares projection-level retention against no-cache and entrywise LRU, but it bundles complete-projection grouping with the non-evicting first-fit admission policy described in Appendix B.4. The gain cannot therefore be attributed separately to either choice. Although Section 6 distinguishes the system from Cerium's weight compression and memory management and Cachemir's layout-aware decoding, isolating this retention mechanism would clarify its incremental contribution.
  2. Section 5.5 acknowledges that the blocked-to-rectangular Q/gate controls in Table 2(a) lack matching numerical validation. Table 5 checks optimized Llama-3-8B projections against plaintext references, but does not check both timed variants for the same logical inputs and weights. The fixed-operand MAC argument in Appendix B.1 does not establish logical output equivalence across layouts, leaving the reported representation-level speedups without a paired correctness check.
  3. As detailed in Appendix B.3, the THOR/MOAI prefill comparisons are estimates rather than direct target-shape layer measurements. The reported 4.12× decode ratio, meanwhile, compares this method with a local Cachemir-style port. Appendix B.5 reports 53.253 s per layer for the port versus 0.188 s for this method in interface/conversion handling, whereas the native-matrix intervals, including weight preparation, are 5.260 s versus 8.581 s. Because these intervals describe portions of the measured execution rather than independently isolated causal effects, the result supports a latency advantage in this local integration but does not establish the same speedup over the original Cachemir system or faster intrinsic matrix arithmetic.
  4. The Reproducibility Statement reports one observation per projection-level timing configuration, while run-to-run variability for the current main results is insufficiently documented. The larger measured latency reductions remain notable, but their run-to-run stability is not quantified. This uncertainty is particularly important for the smaller retention gains and nearly equal batching times. Separately, Table 2(b) evaluates retention over a 129-to-130-key transition across three layers, using a fixed 24 GiB persistent-cache budget specified in Appendix B.5. The reported evaluation does not establish how the retention gains behave over longer decode sequences or across different persistent-cache budgets.
Questions for authors
  1. How much of the retention gain in Table 2(b) comes from complete-projection grouping versus non-evicting admission? If these effects cannot be isolated, what prevents a controlled comparison?

  2. What numerical evidence establishes the equivalence of the timed blocked and rectangular Q/gate variants in Table 2(a) under matching logical inputs and weights? A comparison against a common plaintext reference, including valid-slot errors, output layouts, scales, and levels, would help address this concern.

  3. Which adaptation costs in the local Cachemir-style port arise from its backend interfaces, and how should the reported 4.12× speedup be interpreted relative to Cachemir's original integrated design?

  4. What run-to-run variability is observed in the evaluated configurations? A limited check across additional decoding steps or cache budgets would also help assess the scope of reuse.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.