acceptodds
Under review as a conference paper at ICLR 2027

You Can't Read My Mind: Non-interactive Secure Autoregressive LLM Inference

Abstract

Large Language Models (LLMs) excel at autoregressive generation but typically require server-side deployment, raising the risk of sensitive-data leakage. Fully Homomorphic Encryption (FHE) enables secure LLM inference directly over ciphertexts. However, autoregressive inference under FHE remains underexplored and faces challenges in both efficiency and generation quality. Token-by-token generation requires expensive vocabulary-wide operations, while approximate token selection over vocabulary introduces semantic errors and unintended mixtures of token embeddings. These operations recur at every decoding step, making their computational costs and approximation errors a bottleneck to a practical inference under FHE. To address these challenges, we propose , a non-interactive framework for secure autoregressive LLM inference. We introduce a novel inference flow: the server performs reasoning over ciphertexts, while a lightweight client model generates the answer locally with the returned reasoning latents. This design keeps vocabulary-dependent operations outside FHE evaluation. Fixed-length reasoning in encrypted latent space further reduces encrypted decoding from an average of 26.7 steps to 5 on GSM8K-Aug. To accelerate the decoding stage under FHE, we introduce a layout-compatible KV-cache decoding algorithm that efficiently reuses prefill KV-cache without packing-format conversion. We implement TELEX as an end-to-end system that achieves near-lossless generation quality and substantial efficiency gains. TELEX achieves an per-step decoding speedup compared to MOAI. On a single H200 GPU, we report an amortized time of 49.9 seconds per query for decode stage and an end-to-end time of 14.8 minutes per query. TELEX is the first open-source system to support end-to-end, non-interactive autoregressive LLM inference under FHE.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
fully homomorphic encryption, secure autoregressive LLM inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yTMfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This research introduces TELEX, a non-interactive framework for privacy-preserving autoregressive LLM inference using Fully Homomorphic Encryption (FHE). To address the computational overhead and numerical errors associated with vocabulary-dependent operations under FHE, the authors introduce an FHE-friendly latent reasoning transfer mechanism. The server performs encrypted reasoning in a continuous latent space, while a lightweight client model generates the final answer using the decrypted reasoning representations. Additionally, the authors develop a layout-compatible KV-cache mechanism to accelerate encrypted autoregressive decoding. Experimental results demonstrate substantial decoding speedups while largely preserving the accuracy of the server model.

Critical strengths
  1. Effective integration of latent reasoning and FHE
  2. Efficient KV-cache design
  3. The presentation and organization are generally strong.
Critical weaknesses
  1. Limited task diversity and uncertain generalization beyond arithmetic reasoning. Although the authors evaluate TELEX on several benchmarks, these datasets evaluate related mathematical reasoning capabilities and may not sufficiently demonstrate the general applicability of the framework.
  2. Insufficient attribution of the performance gains from latent reasoning transfer The proposed framework combines several components. However, the current results do not sufficiently isolate their individual contributions. For example, Table 1 reports that TELEX outperforms SIM-CoT on GSM8K-Aug. However, the comparison involves different inference settings: SIM-CoT uses a 1B model, whereas TELEX additionally receives latent representations from a 3B server. Therefore, the performance difference cannot be attributed solely to the proposed transfer mechanism.
  3. Limited analysis of the fixed latent reasoning budget. One of TELEX's main efficiency advantages comes from reducing the number of encrypted decoding steps through fixed-length latent reasoning. However, the relationship between the latent budget, reasoning complexity, and final answer accuracy is insufficiently explored.
Questions for authors
  1. SIM-CoT comparison: Could the authors clarify the differences between the SIM-CoT baseline and TELEX in Table 1, particularly regarding model capacity, training supervision, and inference configuration?
  2. Contribution of latent transfer: Could the authors provide a more detailed ablation comparing the client with matched server latents, mismatched latents, zero latents, and no latent input? How much of the final improvement is attributable to the transferred representations rather than additional supervision and the dynamic selector?
  3. Latent budget: How sensitive is TELEX to the number of latent reasoning steps? Is there a consistent accuracy–efficiency trade-off when increasing or decreasing the latent budget?
3 comments

W1. Additional Evaluation Beyond Mathematical Reasoning

Thank you for your suggestion. We additionally evaluate TELEX on ProsQA[1], a benchmark for multi-step logical reasoning and planning. We use Coconut-GPT2 as the server model to provide reasoning latents, and Pythia-14M as the client model that consumes these latents. As shown in Table 1, TELEX achieves 95.4% accuracy, retaining 97.3% of the server model's accuracy. We further compare three Pythia-14M configurations trained with different forms of supervision. As shown in Table 2, by leveraging server's reasoning latents, TELEX's 95.4% outperforms No-CoT's 62.4% and SFT-CoT's 69.8% . These results show that TELEX can successfully leverage the server's reasoning latents to produce high quality answers than the small-model baselines do.

Table 1. Recovery of server accuracy on ProsQA

ModelAccuracy (%)Recovery (%)
Server: Coconut-GPT2 (124M)98.0100
Client: Pythia-14M95.497.3

Table 2. Accuracy comparison on ProsQA (Pythia-14M)

MethodAccuracy (%)SupervisionInference setting
No CoT62.4answer-onlyquestion
SFT-CoT69.8CoT+answerquestion
TELEX95.4CoT+answer(L2C)/answer-only(HP)question+reasoning latents

W2. Performance Gains from Server Reasoning Latents

Thank you for raising this concern. We would like to emphasize that TELEX's latent transfer mechanism is designed to enable a lightweight client model to leverage reasoning latents generated by a more capable server model. The results in Tables 1 and 2 of TELEX main paper are intended to demonstrate that server provided reasoning latents enable a lightweight client model to approach the server's accuracy, substantially outperforming the client-only baselines. For example, TELEX enables a 1B client model to approach the accuracy of a 3B server model by leveraging server reasoning latents. Table 3 further quantifies the accuracy gap between the 3B server and different 1B client. As shown in Table 3, TELEX 1B achieves comparable performance with server 3B model by using server's reasoning latents while SFT-CoT 1B and SIM-CoT 1B have larger gaps of −4.1 and −6.2, respectively.

Table 3. Accuracy comparison between the 3B server and 1B client configurations.

ModelGSM8K-Aug (%)OOD Avg. (%)Avg. gap (%)Uses server latent?
Server 3B62.163.20.0✗
TELEX 1B62.561.6-0.6✓
SFT-CoT 1B58.458.8-4.1✗
SIM-CoT 1B56.156.8-6.2✗

W3 & Q3. Latent Budget Sensitivity and Accuracy–Efficiency Trade-off

Thank you for this insightful question. Figure 3 of SIM-CoT [2] shows that model performance is sensitive to the latent reasoning budget, with additional latent steps generally improving accuracy. This observation motivates a corresponding analysis of the accuracy–efficiency trade-off in TELEX. Such an analysis, however, requires end-to-end retraining for every latent-budget setting, and each training run for 3B-4B models takes approximately one week with our current computational resources. We are therefore unable to complete a controlled sweep within the rebuttal period. We commit to including this experiment in the final version of the paper. Specifically, we will evaluate multiple latent-step settings under matched training conditions and report their effects on task accuracy, encrypted inference latency and computational cost. This analysis will clarify how the latent budget should be selected for reasoning tasks of different complexity.

Q1. Clarification of the SIM-CoT Comparison

Thank you for your valuable question. Both the SIM-CoT 1B baseline and the TELEX 1B client model are initialized from the same base model, Llama-3.2-1B-Instruct. SIM-CoT uses step-level CoT supervision and the TELEX client is trained to leverage reasoning latents, with HP using answer-only supervision and L2C using CoT-plus-answer supervision. At inference, SIM-CoT uses the question as input, while TELEX uses the question and server's reasoning latents.

[1] Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., & Tian, Y. (2025). Training Large Language Models to Reason in a Continuous Latent Space. Conference on Language Modeling (COLM).

[2] Wei, Xilin, et al. "Sim-cot: Supervised implicit chain-of-thought." International Conference on Learning Representations. Vol. 2026. 2026.

Q2. Effectiveness of Latent Transfer and Component-wise Ablations

Thank you for raising this question. We provide evaluation results under different latent-input conditions in Table 4. As shown in Table 4, replacing matched latents with mismatched or zero-valued latents, or removing the latent input altogether, substantially reduces accuracy. These results demonstrate TELEX indeed leverage reasoning latents to improve the performance. We also report the ablation results to separate HP/L2C and selector contribution. As shown in Table 5, both HP and L2C receive reasoning latents, with answer-only and CoT+answer supervision, respectively. L2C alone achieves 61.0% on GSM8K-Aug and 61.3% OOD average. Adding the frozen selector increases these to 62.5% and 61.6%, corresponding to gains of 1.5 and 0.3. An oracle selector achieves 67.6% and 63.3% accuracy, suggesting that HP direct answer readout and L2C-based CoT expansion provide complementary predictions that there is room to improved selection.

Table 4. Accuracy under different latent-input conditions.

Latent-input conditionGSM8K-AugGSM-HardMultiArithSVAMPOOD Avg.
Matched62.4716.3098.8969.6761.62
Mismatched3.410.768.3330.0013.03
Zero0.300.080.000.330.14
No latent input6.601.5912.7826.6713.68

Table 5. Ablation results for HP, L2C and the selector

MethodGSM8K-AugOOD Avg.Training Supervision
HP51.054.0answer-only
L2C61.061.3CoT+answer
TELEX62.561.6-
TELEX (oracle)67.663.3-

Dear Reviewer yTMf,

Thank you again for the time and care you have put into reviewing this paper. I sincerely appreciate your thoughtful feedback and your contribution to the review process.

The authors have now responded to your initial comments. When you have a chance, could you please take a look at their responses and engage with them in the discussion? Your follow-up feedback would be very helpful in clarifying any remaining questions and informing our final assessment.

Many thanks for your continued engagement and support!

Best wishes,

Your AC

Reviewer_UpjsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

TELEX proposes a non-interactive private inference pipeline that separates encrypted reasoning from local answer generation. A server evaluates a pretrained latent-reasoning model under FHE and returns encrypted latent states to the client. After decryption, an adapted lightweight client model generates the answer through either Hidden-Prefix or Latent-to-CoT decoding, with a learned selector choosing between their outputs. The system also introduces a layout-compatible KV-cache implementation that combines column-packed prefill states with row-packed decoding states.

The main evaluation uses a SIM-CoT Llama-3.2-3B server and a Llama-3.2-1B client on arithmetic reasoning benchmarks. The paper reports close agreement between plaintext and FHE accuracy, an 85.91× first-step decoding speedup over MOAI, and amortized computation times of 49.9 seconds for decoding and 14.8 minutes per query including prefill.

Critical strengths
  1. The inference decomposition addresses an important FHE bottleneck. Moving vocabulary projection, token selection, and answer generation outside the encrypted computation substantially changes the cost structure of private autoregressive inference. Combining encrypted latent reasoning with a local answer generator is a promising architectural direction, particularly when non-interactivity is a requirement.

  2. The implementation provides evidence that latent transfer can tolerate FHE approximation. For the 1B client, the reported in-domain accuracy remains 62.5% under FHE, while average out-of-domain accuracy changes from 61.6% to 61.5%. Together with the concrete KV-cache design and runtime breakdowns, this supports the feasibility of the proposed pipeline, although its comparative advantage remains insufficiently established.

Critical weaknesses
  1. The incremental methodological contribution beyond SIM-CoT and Interlat is insufficiently isolated and validated. i) The server’s latent reasoning capability comes from SIM-CoT. The cross-model communication interface also has substantial precedent in Interlat, which already transfers hidden states through a learned adapter, supports heterogeneous models, and uses matched-versus-mismatched conditional distribution separation to encourage latent utilization. TELEX explicitly builds on this interface. The remaining learning contributions appear to be the particular HP/L2C combination, preference objectives, modifications to the separation loss, and output selector. However, the paper does not adequately establish their incremental benefit over an Interlat-style receiver trained on the same SIM-CoT trajectories. Comparing the complete pipeline against client-only models does not isolate these contributions. Likewise, reporting an oracle selector establishes potential complementarity between candidates, but does not demonstrate that the proposed training and selection mechanisms are necessary. ii) The mixed prefill/decode cache layout is a more concrete candidate for technical novelty. Its value, however, needs to be distinguished from the already-established benefit of enabling KV reuse. Overall, the paper demonstrates a working combination more convincingly than it demonstrates which new components are responsible for its advantages.

  2. The evaluation does not establish a fair system-level advantage over the closest alternatives, with functionality, output, security assumptions, and cost accounting insufficiently aligned. i) The headline comparison against MOAI is inadequate as the principal evidence of superiority over existing encrypted decoding systems. Table 4 reports an 85.91× first-step improvement, while Table 3 reports an approximately 85.9× decoding improvement from enabling KV caching within the latent pipeline. This suggests that the headline gain largely reflects cache-enabled incremental execution and associated implementation changes. A direct comparison with Cachemir, which already supports encrypted KV-cache maintenance, is therefore essential. The statement that Cachemir sacrifices prefill efficiency should be substantiated through a matched prefill-plus-decode comparison. MPCache is also relevant to the broader efficiency discussion. ii) At the system level, TELEX returns intermediate latent states and relies on an additionally trained client model to produce answers. It therefore evaluates a different inference pipeline from direct encrypted execution of an existing model. A fair comparison must explicitly specify the intended output, permitted disclosures, client computation, and model-access assumptions. One more concern is the unaligned functionality and disclosure results. iii) Finally, the approximately 22× decoding improvement over the cache-enabled w/o Latent variant becomes approximately 2.2× when prefill is included. The batch-amortized figures do not establish individual-request latency, and the complete cost of local generation is not clearly accounted for. Given that the client already runs a 1B model, the modest accuracy gains also warrant comparison against a similarly trained local model, including offline distillation.

Questions for authors
  1. Which components of Section 3.2 are new relative to SIM-CoT and Interlat? Under identical server trajectories and training data, how does TELEX compare with an Interlat-style receiver? What gains are attributable separately to HP/L2C, the preference and separation objectives, and the selector?

  2. How does TELEX compare with Cachemir when both execute the same latent trajectory using matched models, contexts, hardware, security parameters, numerical precision, and batching? Please distinguish gains from KV reuse from gains due specifically to the proposed mixed-layout design. For MPCache, which functionality and deployment assumptions prevent or permit a meaningful comparison?

  3. What is the intended private-inference functionality, and what may each party learn? In particular, are server hidden states explicitly permitted outputs, and is server model privacy part of the claim? What server embedding or model information must the client possess, and how are these assumptions aligned with the baselines?

4 comments

W1(i), Q1. Comparison with Interlat

Thank you for your insightful question. In our design, SIM-CoT/Coconut latent reasoning model are using as TELEX's reasoner and provide reasoning latents to the client model. In section 3.2, we would like to highlight two key differences between TELEX and Interlat:

  1. TELEX trains a lightweight client to leverage reasoning latents while Interlat is designed to enable two comparably sized models to communicate within a latent space. In our asymmetric server–client setting, we observe that the client model may introduce errors when expanding reasoning latents to CoT due to its limited model capacity. We therefore design Hidden prefix (HP) which relies more on server's reasoning latents to read out the answer instead of generating CoT itself. As shown in Table 1, our experiments demonstrate that HP and L2C provide complementary benefits.
  2. To accommodate our asymmetric setting that server usually provides strong reasoning capabilities, TELEX adopts a different adapter architecture from Interlat. Interlat contextualizes the received latents through self-attention, using them as queries, keys, and values. In contrast, TELEX retains a frozen base mapping and introduces a trainable branch in which learned client-prefix (question) cross-attends to the encoded server reasoning latents (reasoning). Thus, TELEX uses learned output queries to selectively extract task-relevant information from the server's reasoning. We further compare TELEX L2C and Interlat using the same training setup. As shown in Table 2, TELEX outperforms Interlat by 32.9 and 39.9 on GSM8K-Aug and OOD Avg., respectively.

Table 1. Ablation results for HP, L2C and selector

MethodGSM8K-AugOOD Avg.Training Supervision
HP51.054.0answer-only
L2C61.061.3CoT+answer
TELEX62.561.6-
TELEX (oracle)67.663.3-

Table 2. Comparison of Interlat and TELEX L2C

MethodGSM8K-AugOOD Avg.
Interlat28.121.4
TELEX L2C61.061.3

W1(ii), W2(i), Q2. Comparison with Cachemir

Thank you for your valuable question. Since Cachemir's complete inference pipeline code hasn't been released, we reproduced Cachemir based on its published algorithms and available artifact in PhantomFHE and evaluated it on the same SIM-CoT-Llama-3.2-3B latent-generation workload as TELEX. Both implementations use an H200 GPU, a prefix length of 128 tokens, a batch size of 256 and matched CKKS parameters (N=216N = 2^{16} and 128-bit security). As shown in Table 1, TELEX gains a 2.54$\times$ speedup over Cachemir + MOAI prefill and 37.59$\times$ over Cachemir with its native prefill execution. These gains come from the following factors:

Prefill and KV-cache layout conversion

  1. Prefill. As acknowledged in section 5.4 of Cachemir[1], their packing method is suboptimal due to additional constraints on the form of K and V. We follow section 5.4 of the Cachemir paper to implement its native KV cache packing for prefill. As shown in Table 3, the prefill time is 32,114.40 s per input with Cachemir’s packing and 838.10 s per input with the MOAI prefill.
  2. KV-cache layout conversion. TELEX doesn't require KV cache layout conversion because its KV-cache layout is compatible with both prefill and decoding. In contrast, integrating Cachemir with MOAI prefill method requires an additional 160.4 seconds per input to convert the layout of the entire encrypted KV cache.

Decoding

  1. Decode. For decoding, we provide a breakdown of the runtime over five decoding steps in Table 4. Both TELEX and Cachemir adopt KV caches, and TELEX achieves a 25.2$\times$ speedup over Cachemir for decoding. This speedup benefits from TELEX’s layout compatibility and efficient batched execution. In particular, TELEX’s compact packing reduces the number of bootstrapping operations per Transformer layer from 1,792 to 50. This contribute a 35.6$\times$ speedup in bootstrapping, which is the most time-consuming component.
  2. KV-cache maintenance. TELEX reduces KV cache maintenance overhead by simply appending references to the newly generated ciphertexts without any additional homomorphic repacking operations such as rotations, masking, and additions. For comparison, Cachemir's KV-cache maintenance accounts for 39.78 seconds per input over five decoding steps.

Table 3: Prefill plus decode time comparison

MethodPrefill (s)KV Cache Layout Conversion (s)Decode 5 latents(s)Total (min)Total Runtime relative to TELEX
TELEX838.100.0049.9514.801.00×
Cachemir + MOAI prefill838.10160.401,257.9337.612.54×
Cachemir32,114.400.001,264.58556.3237.59×

Table 4: Decoding time comparison breakdown

Decode componentTELEX(s/input)Cachemir(s/input)Runtime relative to TELEX
Q/K/V projection3.98211.7753.15×
Attention20.57108.155.26×
KV-cache update<0.00139.78—
Output projection0.76134.58177.46×
Gate/up projection6.36162.8525.62×
SiLU and gated product0.231.667.33×
Down projection1.9819.009.59×
Norm, residual and mask0.0911.08128.97×
Bootstrapping15.99569.0635.60×
Total decoding49.951,257.93≈25.2×

Discussion of MPCache. In this work, we focus on FHE-based secure inference. FHE and MPC are two privacy-enhancing techniques and that are designed for different deployment scenarios. FHE allows a computing server to perform model inference directly on encrypted data uploaded by the client without requiring interaction between the server and the client during inference. In contrast, MPC enables the client and one or more servers to jointly perform secure inference through interactive protocols. MPC-based approaches require the client to remain online throughout inference. Consequently, MPC-based approaches generally rely on a reliable network connection and sufficient bandwidth, whereas FHE-based approaches rely more on the computational resources on the server.

Due to these fundamental differences, directly comparing the performance of MPC-based (like MPCache) and FHE-based approaches is generally inappropriate without first defining the target scenario and settings. Instead, we discuss MPC-based approaches as related works and discuss their advantages and limitations.

[1] Yu, Ye, et al. "Cachemir: Fully homomorphic encrypted inference of generative large language model with kv cache." arXiv preprint arXiv:2602.11470 (2026).

W2(iii). Client-Side Runtime and Memory Analysis

Thank you for raising this point. We report the inference latency and GPU memory consumption of the TELEX 1B client model on a single NVIDIA H200 GPU. As shown in Table 5, the client model takes 1.132 and 0.136 seconds per query at batch sizes of 1 and 16, respectively. This time cost is negligible compared with server reasoning under FHE (14.8 min / query). The TELEX client-side 1B model requires only 6.49 GB and 10.85 GB of GPU memory for batch sizes of 1 and 16, respectively, making it feasible to deploy on consumer-grade GPUs.

Table 5. TELEX 1B inference on client side

SettingTime (s/query)Memory (GB)
Batch_size = 11.1326.49
Batch_size = 160.13610.85

Q3. Clarification of functionality and threat model

Thank you for raising this question. TELEX's intended functionality is to allow the client to use the model's service of server securely:

  • The client sends an encrypted embedding to the server, and the server processes it using FHE without needing access to the plaintext.
  • The server returns encrypted reasoning latents to the client. The client decrypts them and uses the decrypted latents and the question to generate an answer. These latents are permitted outputs and serve as reasoning representations for the client during inference.
  • Regarding model privacy, in our TELEX’s design, the large model of server never leaves server.

We follow the threat model definition and the privacy setting concretely as the research line of secure inference, e.g., [2,3]. More concretely, two parties are involved: client and server (i.e., the service provider). The client uses the services and models provided by the server, by uploading data encrypted using FHE. We consider semi-honest server in the threat model, meaning that server follows the protocol precisely but it may attempt to infer private information from client's data. [3] has discussed that under semi-honest assumption it remains a realistic and reasonable approach. To protect client's privacy, we set the FHE scheme with parameters that provide 128-bit security, following the guideline in [4]. Under this security setting, server can perform the inference on the encrypted data without learning any information about the client's input data.

[2] THOR: Secure transformer inference with homomorphic encryption, CCS 2025.

[3] Powerformer: Efficient and high-accuracy privacy-preserving language model with homomorphic encryption, ACL 2025.

[4] Security guidelines for implementing homomorphic encryption. Cryptology ePrint Archive, 2024.

Dear Reviewer Upjs,

Thank you again for the time and care you have put into reviewing this paper. I sincerely appreciate your thoughtful feedback and your contribution to the review process.

The authors have now responded to your initial comments. When you have a chance, could you please take a look at their responses and engage with them in the discussion? Your follow-up feedback would be very helpful in clarifying any remaining questions and informing our final assessment.

Many thanks for your continued engagement and support!

Best wishes,

Your AC

Reviewer_NMbnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors primarily address inference efficiency under FHE. They introduce latent Transformers to perform reasoning efficiently with fewer output tokens. These outputs are then processed by lightweight models on the client side. The authors experimentally demonstrate that the latent models achieve a 3X reduction in latency on GSM8K.

Critical strengths
  • The attempt to improve output token efficiency by performing reasoning in latent space is novel.
Critical weaknesses
  • The choice of baseline overstates the method’s effectiveness. Since Cachemir also uses FHE and reuses the KV cache, a comparison against Cachemir would be fairer.
  • Latent reasoning alone appears to provide roughly a 3X speedup, whereas speculative decoding typically achieves a 2~3X speedup while also offering performance guarantees. Moreover, the proposed method requires additional training both a latent reasoning Transformer and a lightweight model, making speculative decoding a more attractive alternative.
  • The evaluation focuses on mathematical reasoning benchmarks such as GSM8K. Experiments in other domains are also needed.
Questions for authors

Please see the weakness section.

3 comments

W1. Comparison with Cachemir

Thank you for your valuable question. Since Cachemir's complete inference pipeline code hasn't been released, we reproduced Cachemir based on its published algorithms and available artifact in PhantomFHE and evaluated it on the same SIM-CoT-Llama-3.2-3B latent-generation workload as TELEX. Both implementations use an H200 GPU, a prefix length of 128 tokens, a batch size of 256 and matched CKKS parameters (N=216N = 2^{16} and 128-bit security). As shown in Table 1, TELEX gains a 2.54$\times$ speedup over Cachemir + MOAI prefill and 37.59$\times$ over Cachemir with its native prefill execution. These gains come from the following factors:

Prefill and KV-cache layout conversion

  1. Prefill. As acknowledged in section 5.4 of Cachemir[1], their packing method is suboptimal due to additional constraints on the form of K and V. We follow §5.4 of the Cachemir paper to implement its native KV-cache packing for prefill. As shown in Table 1, the prefill time is 32,114.40 s per input with Cachemir’s packing and 838.10 s per input with the MOAI prefill.
  2. KV-cache layout conversion. TELEX doesn't require KV cache layout conversion because its KV-cache layout is compatible with both prefill and decoding. In contrast, integrating Cachemir with MOAI prefill method requires an additional 160.4 seconds per input to convert the layout of the entire encrypted KV cache.

Decoding

  1. Decode. For decoding, we provide a breakdown of the runtime over five decoding steps in Table 2. Both TELEX and Cachemir adopt KV caches, and TELEX achieves a 25.2$\times$ speedup over Cachemir for decoding. This speedup benefits from TELEX’s layout compatibility and efficient batched execution. In particular, TELEX’s compact packing reduces the number of bootstrapping operations per Transformer layer from 1,792 to 50. This contribute a 35.6$\times$ speedup in bootstrapping, which is the most time-consuming component.
  2. KV-cache maintenance. TELEX reduces KV cache maintenance overhead by simply appending references to the newly generated ciphertexts without any additional homomorphic repacking operations such as rotations, masking, and additions. For comparison, Cachemir's KV-cache maintenance accounts for 39.78 seconds per input over five decoding steps.

Table 1: Prefill plus decode time comparison

MethodPrefill (s)KV Cache Layout Conversion (s)Decode 5 latents(s)Total (min)Total Runtime relative to TELEX
TELEX838.100.0049.9514.801.00×
Cachemir + MOAI prefill838.10160.401,257.9337.612.54×
Cachemir32,114.400.001,264.58556.3237.59×

Table 2: Decoding time comparison breakdown

Decode componentTELEX(s/input)Cachemir(s/input)Runtime relative to TELEX
Q/K/V projection3.98211.7753.15×
Attention20.57108.155.26×
KV-cache update<0.00139.78—
Output projection0.76134.58177.46×
Gate/up projection6.36162.8525.62×
SiLU and gated product0.231.667.33×
Down projection1.9819.009.59×
Norm, residual and mask0.0911.08128.97×
Bootstrapping15.99569.0635.60×
Total decoding49.951,257.93≈25.2×

[1] Yu, Ye, et al. "Cachemir: Fully homomorphic encrypted inference of generative large language model with kv cache." arXiv preprint arXiv:2602.11470 (2026).

W2. Comparison with Speculative Decoding

Thank you for raising this point and suggesting speculative decoding. We agree speculative decoding is a promising direction for improving efficiency. However, we emphasize that TELEX avoids vocabulary-scale homomorphic computation and the failure modes associated with approximate token selection (Appendix A), thereby improving efficiency and the reliability of answer generation in privacy-preserving LLM inference. In addition, we would like to highlight the challenges of applying speculative decoding under privacy-preserving LLM inference: if the draft model is on the client side, the client would need to exchange draft proposals and verification results across multiple rounds. TELEX support non-interactive inference that client only needs to send the query once and then stay offline. And if the draft model is held on the server, the following three issues arise:

  1. The token errors and embedding divergence failure mode in Appendix.A of main paper still exist when speculative decoding generates corrected token or bonus token.
  2. The extra FHE computation must be performed using draft model which will introduce extra FHE computation overhead.
  3. Similar to token selection, draft verification process is also a discrete decisions with non-linear function which have to be approximated and may cause semantically wrong answer and mixture of embedding

Overall, these challenges introduce additional protocol complexity and computational overhead, suggesting that the 2–3× speedups reported in plaintext speculative decoding may not directly translate to non-interactive, privacy-preserving LLM inference.

W3. Additional Evaluation Beyond Mathematical Reasoning

Thank you for your suggestion. We additionally evaluate TELEX on ProsQA[1], a benchmark for multi-step logical reasoning and planning. We use Coconut-GPT2 as the server model to provide reasoning latents, and Pythia-14M as the client model that consumes these latents. As shown in Table 3, TELEX achieves 95.4% accuracy, retaining 97.3% of the server model's accuracy. We further compare three Pythia-14M configurations trained with different forms of supervision. As shown in Table 4, by leveraging server's reasoning latents, TELEX's 95.4% outperforms No-CoT's 62.4% and SFT-CoT's 69.8% . These results show that TELEX can successfully leverage the server's reasoning latents to produce high quality answers than the small-model baselines do.

Table 3. Recovery of server accuracy on ProsQA

ModelAccuracy (%)Recovery (%)
Server: Coconut-GPT2 (124M)98.0100
Client: Pythia-14M95.497.3

Table 4. Accuracy comparison on ProsQA (Pythia-14M)

MethodAccuracy (%)SupervisionInference setting
No CoT62.4answer-onlyquestion
SFT-CoT69.8CoT+answerquestion
TELEX95.4CoT+answer(L2C)/answer-only(HP)question+reasoning latents

[2] Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J., & Tian, Y. (2025). Training Large Language Models to Reason in a Continuous Latent Space. Conference on Language Modeling (COLM).

Dear Reviewer NMbn,

Thank you again for the time and care you have put into reviewing this paper. I sincerely appreciate your thoughtful feedback and your contribution to the review process.

The authors have now responded to your initial comments. When you have a chance, could you please take a look at their responses and engage with them in the discussion? Your follow-up feedback would be very helpful in clarifying any remaining questions and informing our final assessment.

Many thanks for your continued engagement and support!

Best wishes,

Your AC

Comments on the paper

1 comment
AuthorsOpenReview ↗

Sincere Thanks and Rebuttal Summary

Dear Area Chair,

We sincerely thank you and all reviewers for the time and effort devoted to evaluating our work. In summary:

  • All three reviewers recognized the value of integrating latent reasoning with FHE to improve inference efficiency. Reviewer NMbn explicitly described our attempt to improve output token efficiency through latent-space reasoning as “novel.” Reviewer Upjs noted that our inference decomposition “addresses an important FHE bottleneck” and described the combination of encrypted latent reasoning and local answer generation as “a promising architectural direction.” Reviewer yTMf highlighted the “Effective integration of latent reasoning and FHE” as a critical strength.
  • Reviewers highlighted the effectiveness of our design and the evidence supporting its feasibility. Reviewer yTMf identified “Effective integration of latent reasoning and FHE” and “Efficient KV-cache design” as critical strengths. Reviewer Upjs also recognized that our implementation, concrete KV-cache design, and runtime breakdowns support the feasibility of the proposed pipeline.
  • 2/3 reviewers (Upjs and yTMf) currently recommend weak acceptance and reviewer Upjs's concerns have been addressed. During rebuttal, we further strengthened the empirical evidence through matched comparisons with Cachemir and Interlat, additional component and latent-input ablations, and evaluation on ProsQA beyond arithmetic reasoning. Reviewer Upjs stated that their main concerns were addressed and raised their rating.

However, we respectfully ask you to consider that Reviewer NMbn’s remaining concern focuses on a client-side speculative decoding alternative that requires intermediate client participation, whereas TELEX targets non-interactive inference. We have also provided an analytical comparison explaining why fewer target-model decoding passes do not necessarily reduce the total FHE workload. Moreover, TELEX already exploits parallelism similar to that brought by speculative decoding. This is achieved through interleaved batching enabled by our layout-compatible KV-cache algorithm. While we acknowledge the potential value of integrating speculative decoding, this does not diminish TELEX’s contribution to non-interactive privacy-preserving inference.

We sincerely hope that the strengths recognized across the reviews, the additional evidence provided during rebuttal, and Reviewer Upjs’s explicit positive reassessment will be taken into consideration in your final decision.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.