- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets on scaling up FHE-based LLM inference to decoder models like Llama-3.1-8B. The paper scales up it in two directions: 1) fully encrypted prompts and 2) a long context is public and only the query is sensitive. The implementation results show the proposed scheme runs the prefill of Llama-3.1-8B in 20 seconds on 8 GPUs. The motivation is good, but I feel the presentations of technical contributions are not very clearly.
Critical strengths
-
The presentation is clear. The figures are well designed and helpful for understanding the scheme.
-
Although the source code is not allowed to redistribute, the paper provided executable binaries and detailed descriptions of proposed algorithms.
Critical weaknesses
- The baselines are limited. For end-to-end evaluation, as far as I know, there are few works also work on FHE-based inference of decoding model, like Cachemir[1]. For the matrix multiplications, authors compared their scheme with JKLS algorithm and Powerformer, but some works contributed in this part are not mentioned and compared, like Tricycle[2]
[1] Cachemir: Fully homomorphic encrypted inference of generative large language model with kv cache. arXiv preprint arXiv:2602.11470, 2026.
[2] Tricycle: Private transformer inference with tricyclic encodings. CCS 2026
-
The comparisons with the baseline are running in different library and different GPU, while the authors don’t provide a benchmark on the hardware and library, so the comparison results are not clear to me.
-
The work builds on existing works and the improvement is incremental, such as the ring switch and the coeff-encoding PCMM are all directly used from existing works. And this work is also not the first one to considering using coeff-encoding PCMM in transformer inference, which is explored in LEAF [3].
[3] LEAF: A low-latency evaluation architecture for feedforward block in privacy-preserving transformer inference. Cryptology ePrint Archive, Paper 2025/956, 2025.
Questions for authors
-
The paper only provides overall run time and a simple analysis of where the speedup comes from, which looks not clear to me. I think the time cost, ciphertext depth, and comparison with existing works of each step in an end-to-end evaluation flow can help to understand the improvement better, like the one in NEXUS and MOAI.
-
N=2^8 degree is used in the coeff-encoding PCMM, which is much smaller than widely adopted parameters like N=2^16. I’m interested in what the plaintext modulus, ciphertext modulus and scale are used in this case and how about the error of this step.
We thank you for your careful reading. We will add more kernel-level comparison and move Table 2 to the main text. As Cerium (SOSP 2026) explains, FHE-based Llama inference raises significant system problems; to our knowledge, Sylph is the first to run the complete chain of prefill, decoding and token feedback (homomorphic token selection and embedding lookup) end to end on real GPUs.
W1. We will add both works as baselines: 'Cachemir' measures individual kernels and derives its end-to-end figure from them: 3.02 s per decoder block on one A100-80GB (customized Phantom library, sequence length 512), i.e. ~100 s per token. It does not cover the token feedback, and it reports 694 GB of encoded weights without describing how a full run fits on an 80 GB GPU. Sylph’s figures are measured on end-to-end encrypted runs. Per token, the 32 layers take 24 s on one B200 or 31 s on one RTX PRO 6000 after a 128-token prompt, and 22 s on eight RTX PRO 6000 with 3968 further public tokens (Table 7); the final RMSNorm, LM head, encrypted token selection and embedding lookup add ≈2.2 s (App. F). The converted weights take 58 GiB (Table 9).
Tricycle. Its tricyclic encoding also gives depth-1 slot-encoded matmuls with O(n) rotations, so it is the right comparison for our depth-1 PCMM (Sec. 4.2); the asymptotics are comparable, and we will add it next to JKLS and Powerformer. Unlike tricyclic encodings, which need their own packing with coprime dimensions, ours keeps the slot layout of the rest of the pipeline.
Tricycle is end-to-end only on BERT; at Llama-2-7B dimensions it reports matmul micro-benchmarks (one layer, 32 tokens, one A100, HEaaN v1): 7.3–12.1 s for QKV and 0.5–1.0 s for QKᵀ, noting that its PCMM cost grows superlinearly with the matrix dimension. In Sylph (BLAS-based PCMM for projections), at 128 tokens on one RTX PRO 6000, QKV takes 326 ms , QK 316 ms, and a complete layer including bootstrapping 3.55 s (Table 11). As GPUs and libraries differ, we will report both works side by side with their settings, not as speedups.
W2. The paper has three controls:
W3. We do not claim coefficient-encoded PCMM or its ring switching. LEAF adopts the same PCMM for BERT-base's feed-forward block in isolation (BFV, on CPU), and notes that using it inside a slot-encoded CKKS pipeline is hard because the conversions offset much of the gain. We will cite it.
Our contribution is the global integration that lets each step use its best algorithm. Given the rigidity of FHE ciphertexts, this requires: layouts and conversions: packing layouts that absorb the bit reversal (App. E) and SinC-preserving ring switching (App. G), both new; circuit design (levels, bootstrapping placement) driven by range calibration, which speeds up RMSNorm by 15× and SoftMax by 2.2×, plus slim sum-of-squares evaluation (a further 1.25×); deployment: GPU memory, parallelism and communication (6.5× per layer on 8 GPUs). Correctness is part of this: Sylph matches FP16 perplexity under FHE, while the released approximations of Cerium and MOAI receive out-of-domain inputs on Llama-3 (App. L).
Q1. Per-layer prefill at 128 tokens on RTX PRO 6000. Step times (Table 11) include the format conversions and bootstrapping attributed to each step. Kernel times (Fig. 4a–b; one GPU, same library) exclude them; "prior" is Cerium's algorithm for projections, JKLS for QK + ScoreV, and prior-work input ranges for non-linear layers.
Levels: PCMM runs at the lowest level, the depth-1 PCMM uses one level, the SoftMax main track 8, and bootstrapping returns 14; we will add a level column. We compare kernels within one library because papers attribute bootstrapping to steps differently.Q2. N = 2⁸ is used only for the PCMM, on Module-LWE ciphertexts of rank k = 16, so the underlying lattice dimension is kN = 2¹². CKKS has no plaintext modulus; the precision is set by the scaling factor, Δ = 2³⁸ on a 53-bit base modulus, with log PQ = 106 at this step (≥128-bit security, lattice estimator). The measured precision of this step is 15 bits; 12 bits suffice to match FP16 perplexity (Table 4).