acceptodds
Under review as a conference paper at ICLR 2027

Taurus: Accelerating Private LLM Inference via Homomorphic Speculative Decoding

Abstract

Cloud-hosted large language model (LLM) inference raises privacy concerns, since sensitive user inputs are exposed to the service provider. Fully homomorphic encryption (FHE) offers a promising solution by enabling inference directly over encrypted inputs. However, its substantial computational overhead remains a major obstacle to practical deployment. This overhead is particularly severe in autoregressive decoding, which generates only one token at a time, while FHE is more efficient when many values are packed and processed together. Speculative decoding instead processes multiple tokens in one model evaluation pass, creating an opportunity to better utilize FHE's packing ability. However, its speculation-specific operations, including sampling, verification, and state updates, are expensive to realize under FHE. In this paper, we propose Taurus, the first framework for fully homomorphic speculative decoding. Taurus packs many speculative candidate tokens together to share expensive FHE computation across multiple token states. To be more efficient and accurate, Taurus develop a homomorphic sampling protocol that avoids scanning the entire vocabulary with encrypted comparisons. Taurus also proposes a private speculative verification protocol to avoid encrypted division or clipping, and performs state updates obliviously without revealing which candidate tokens are accepted. Together, these protocols enable efficient speculative decoding entirely on the server, reducing per-token latency while preserving the privacy for both client and server. We implement Taurus and conduct experiments on six open-weight models with 1.7B to 8B parameters. On eight NVIDIA RTX 4090 GPUs, Taurus achieves a 1.80× to 3.02× speedup over CacheMir in end-to-end decoder latency per committed token.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
homomorphic encryption, private-preserving machine learning, speculative decoding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_K1yBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper attempts to address a problem in FHE-based autoregressive decoding: typically, the model only advances one token at a time, while ciphertext computation is expensive, but can be parallelized. This paper packs different candidate states into ciphertexts for parallel processing, allowing multiple tokens to be processed simultaneously, thus it can accelerate secure inference. The paper designs several algorithms to accelerate FHE-based secure inference. Besides processing multiple states in parallel, it also constructs short CDFs hierarchically for discrete comparisons and keeps selection decisions private using encrypted selectors, fixed scans, and arithmetic combinations. Overall, this paper investigates a noteworthy problem in the field of secure inference and makes substantial contributions.

Critical strengths
  1. Taurus makes a good observation: it finds that FHE-based ciphertext computation can handle multiple potential states simultaneously and can be used to improve efficiency. Experiments also verify this, compared to CacheMir, Taurus significantly reduces the time per token.

  2. Beyond simply packing multiple candidates into ciphertexts, Taurus also implements a complete pipeline including encrypted sampling, candidate verification, and KV cache commit etc.

  3. I also really like the ablation experiments in this paper; they clearly explain the functions of several key mechanisms.

Critical weaknesses
  1. The main end-to-end experiments focus on a context length of 256 tokens, but as far as I know, using longer contexts might yield different results, so experiments with longer context lengths, such as 512 or 1,024 tokens, need to be added.

  2. Although the paper presents meaningful ablation experiments, it does not isolate the contribution of packing different candidate states into ciphertexts within speculative decoding. If speculative decoding is used without Taurus’s packing strategy, how does the performance change?

  3. The paper claims to protect the privacy of both parties, which I understand to mean protecting the user and the server. That is, the server cannot see the user's input and output, and the user cannot obtain model weights or intermediate outputs. However, the paper's security discussion mainly focuses on the semi-honest server, with limited discussion of what the client can learn. For example, does the client know how many tokens are generated in each round?

Questions for authors
  1. Can experiments with longer context lengths, such as 512 or 1,024 tokens, be conducted? And how does it compare to the baseline under longer context lengths?
  2. If speculative decoding is used without Taurus's packing strategy, how does the performance change?
  3. As mentioned in point 3 of the Weakness section, what information is available from the client's perspective?
  4. The paper has tested cryptographic generation for each prompt and observed no inconsistencies in the output tokens. I'm curious if further testing with longer, complete answers is possible, comparing Taurus's accuracy in mathematical problems or its code test pass rate against the baseline. I'd like to know if Taurus can maintain the quality of the final answer while increasing generation speed?
Reviewer_dB8jOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors mainly address the challenges of applying speculative decoding under FHE. The main challenges are the large vocabulary size and the high cost of accurate comparisons under CKKS. Thus, the authors introduce hierarchical sampling with a three-level tree structure to reduce the number of comparisons, and switch from CKKS to TFHE to perform these comparisons efficiently. Also, to compute acceptance quickly, they reformulate division as multiplication. In the experiments, they demonstrate efficiency across various domains.

Critical strengths
  • Hierarchical sampling is novel and well suited to FHE, as it handles large vocabularies through packed ciphertext operations while significantly reducing the number of expensive comparisons. The authors demonstrate its efficiency experimentally.
  • Acceptance is computed efficiently by reformulating divisions as multiplications.
  • The methods are evaluated across various domains, including mathematical reasoning, dialogue, and code generation.
Critical weaknesses
  • The novelty of the acceptance reformulation is somewhat unclear. Reciprocal operations are also expensive in MPC, so this challenge is not specific to FHE. A comparison with the acceptance verification techniques used in prior MPC-based speculative decoding systems, such as POST, would help clarify what is new in the proposed formulation.
Questions for authors

Please see the weakness section.

Reviewer_86aaOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Taurus introduces a novel method to improve the throughput of encrypted inference, which is to use the excess capacity in CKKS ciphertexts to run speculative decoding. In order to efficiently implement speculative decodings under FHE, Taurus introduces three new algorithms. First, HierSample uses scheme switching + coarse comparisons to sample candidate decodings. After a forward pass, VerifyMSS selects the speculative branch to proceed with the inference. Finally, FixedTraceCommit loads the KV cache & other token values into the inference pipeline for the next inference round.

Critical strengths

This is a nice, well-executed improvement to the throughput of encrypted LLM inference for this particular token sampling method. Several ideas come together to make this work.

Critical weaknesses

I found the evaluation of the end-to-end decoding latency to be too sparse. Most of the evaluation focuses on ablation studies for the individual algorithm steps, which is interesting but should not be the focus of this section. I would have liked to see more than one draft model architecture benchmarked during a decoding path. In general, this method heavily relies on the existence of a high-quality draft model, which ultimately limits the generality of this technique.

Questions for authors
  1. What is the breakdown of the timing between the draft forward pass and the draft sampling?
  2. Does the performance of the draft model depend on the plain sampling method? Is this approach compatible with greedy decoding or top-k decoding?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.