acceptodds
Under review as a conference paper at ICLR 2027

RESAD: Round-Efficient Secure Autoregressive Decoding Scheme for Transformer Inference

Abstract

Secure Transformer inference for generative large language models can provide protection for both user inputs and model parameters. Existing secure transformer inference schemes based on secure multi-party computation suffer from high latency during secure autoregressive decoding. This latency is amplified by sequential token generation: each token requires a secure inference pass, and the communication rounds of secure protocols are accumulated over many decoding steps. In this paper, we propose RESAD, a round-efficient secure autoregressive decoding scheme for Transformer-based models. To alleviate the performance bottlenecks introduced by nonlinear functions in secure decoding, we design low-round secure computation protocols, with a focus on Softmax and GeLU. For Softmax, a novel bucket-based maximum approximation algorithm is proposed to reduces secure maximum computation from logarithmic-round to constant-round. We further introduce a novel interval lookup-table-based approximation protocol which is used not only for secure evaluation on the exponential and reciprocal functions in few rounds, but also for the secure evaluation of GeLU and LayerNorm. These techniques substantially reduce the interaction overhead of nonlinear functions in secure autoregressive decoding. These techniques are integrated into an end-to-end secure Transformer inference scheme optimized for autoregressive decoding. Experiments on Transformer models show that, compared with the state-of-the-art efficient scheme, RESAD reduces communication rounds by 56.4%-63.2%, online communication by 28.9%-51.4% and end-to-end runtime by 24.4%-43.3% in WAN settings and by 14.5%-16.4% when input 1024th tokens under LAN, while maintaining comparable accuracy. We further implement and evaluate RESAD on TinyLlama, demonstrating its applicability to LLaMA-style models.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Privacy-Preserving Machine Learning, Transformers, Autoregressive Generation, Secure Inference, Secure Multi-Party Computation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_PKgoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper focuses on reducing the communication rounds in secure autoaggressive decoding models by proposing a bucket-based maximum approximation for Softmax and an interval look-up-table based approximation protocol for GELU and Layernorm. Then the paper adopts existing protocols and evaluates the proposed scheme.

Critical strengths

The submission includes the details of their implementation and codebase.

The paper includes details of the proposed algorithms.

Critical weaknesses
  1. The technical contribution is limited. The paper adopts existing MIC protocol directly in Section 3 and 4 without improvements. Although the paper proposed bucket-based approximate maximum and interval lookup table, it doesn’t give a comprehensive analysis of choosing the number and size of buckets/intervals. So the proposed techniques are difficult to be applied.

  2. The paper aims to reduce the number of communication rounds in secure transformer inference, but it only considered SMPC-based baselines. Indeed, FHE-based transformer inference is also round-efficient, and I think authors should discuss about why rule out FHE-based baselines.

  3. The improvement is not significant. From evaluation results, although the proposed method reduces the communication rounds and the online communication overhead, but the total communication overhead is even larger than baselines.

Questions for authors
  1. In section B, authors said Boyle et al’s secure multi-interval containment (MIC) protocol is used to securely identify bucket membership and the rightmost non-empty bucket, which also contributes to the overall round complexity. I would like to know more about the protocol to ensure the round complexity of RESAD is O(1).

  2. For the constant-round secure approximate maximum, is there any analysis/instructions about how to choose the number of bucket and the interval of each bucket to ensure that the approximate error won’t affect the model performance? Similarly, for Section 4.1, how to choose p? I think such analysis is necessary especially when the proposed algorithms are used in large and deep models.

3 comments
AuthorsOpenReview ↗

Response to Weaknesses W1–W2

We thank the reviewer for the constructive feedback and valuable suggestions.

W1.

The key insight behind our first contribution is that secure Softmax does not require an exact maximum, but only a numerically safe shifting value. Existing maximum-based secure Softmax protocols typically compute the exact maximum through sequential comparisons, incurring input-length-dependent communication rounds. By exploiting the shift invariance of Softmax, we propose a bucket-based approximate maximum algorithm that replaces sequential comparisons with parallel bucket localization, achieving constant communication rounds with linear communication complexity, while maintaining numerical stability.

Our second contribution introduces a significant-bit-based interval construction that reduces the table-entry complexity for secure nonlinear evaluation from O(2n)O(2^n) to O(2p(n−p))O(2^p(n-p)). Combined with different secure MIC/ILUT protocols, we obtain efficient secure protocols for nonlinear functions. As shown in the ILUT-only ablation experiments (Appendix F.4, Figures 23–25), compared with SIGMA for GPT-2 Small and Medium, ILUT reduces online runtime by 42.4%–44.1%, total runtime by 33.7%–35.5%, online communication by 35.8%–41.1%, and total communication by 22.6%–25.1%, while maintaining comparable model accuracy. These results demonstrate the efficiency of our proposed ILUT construction.

While both contributions build on existing MIC/FSS primitives, our novelty lies in the new approximation constructions, i.e., bucket-based approximate maximum reducing communication rounds from O(log⁡m)O(\log m) to O(1)O(1) and precision-aware interval tables reducing table-entry complexity from O(2n)O(2^n) to O(2p(n−p))O(2^p(n-p)), and the resulting nonlinear protocols that reduce communication rounds from 15 to 7 compared with Grotto, rather than the underlying cryptographic primitives.

We have added rigorous theoretical analyses of parameter selection and the accuracy–cost tradeoff. For ApproxMax, Appendix B.2 derives the numerical-stability condition δ≤0.689005\delta\leq 0.689005 and proves that, under our reference range and bucket budget U=256U=256, selecting δ=0.5\delta=0.5 achieves the minimum refinement depth d=2d=2, requiring only six rounds for the masked-output interface. For ILUT, Appendix C.2 establishes a relative input-approximation bound of 21−p2^{1-p} and a table-size bound of O(2p(n−p))O(2^p(n-p)), showing that p=8p=8 is the smallest admissible precision under the stated BF16-resolution criterion. These analyses provide explicit parameter-selection rules.

See Sections 3–4, Appendix B.2 (pp. 15–18), and Appendix C.2 (pp. 22–26).

W2.

We agree that FHE-based inference is an important alternative for reducing interaction rounds. However, RESAD and FHE-based schemes follow different technical paradigms. While FHE-based schemes can achieve low or even non-interactive inference, they often incur substantial computational overhead from homomorphic operations. In contrast, RESAD focuses on reducing communication rounds and online latency within the FSS-based secure inference paradigm with preprocessing.

To provide a concrete comparison, we have added experiments against Cachemir, a recent state-of-the-art FHE-based generative inference scheme, using its publicly available implementation (https://zenodo.org/records/23189548). Since Cachemir's public CPU artifact provides a benchmark for a single Transformer decoder block, we align the evaluation scope by measuring RESAD on CPUs at the same block level.

Experimental settings. We use the same CPU environment described in Section 6 and follow configuration supported by Cachemir's public artifact. Specifically, we evaluate a synthetic LLaMA-style decoder block with a hidden dimension of 512, an intermediate dimension of 2048, eight attention heads, and a batch size of one. The benchmark processes the 128th token with a preconstructed 127-token KV cache and synthetic weights. The Cachemir artifact uses a reduced CKKS configuration with log⁡N=13\log N=13, level 6, and 16 threads.

Cachemir performs homomorphic evaluation of encrypted inputs on a single server, whereas RESAD uses secret sharing between two non-colluding computing parties with a trusted preprocessing dealer. They rely on different cryptographic and trust assumptions, therefore, this comparison is intended to provide a runtime at the same block. We also acknowledge that RESAD includes interactive network communication, whereas Cachemir performs local homomorphic evaluation, leading to different performance characteristics. As shown in Table 13, Under WAN settings, RESAD achieves a 19.58$\times$ speedup over Cachemir, despite the additional network latency caused by interactive communication. Under LAN settings, where network latency is substantially lower, RESAD achieves a larger speedup of 154.42$\times$.

See Appendix F.6, Table 13, p. 37.

AuthorsOpenReview ↗

Response to Weakness W3

We thank the reviewer for highlighting this important tradeoff and motivating further analysis.

W3.

We agree that the total communication overhead is even larger than baselines. However, the main goal of this paper is to reduce online runtime. As shown in Table 5, RESAD achieves lower online runtime than SIGMA at the 512th and 1024th token positions under LAN settings, and at all evaluated token positions under WAN settings, across all three models.

Although communication overhead affects communication time and consequently overall runtime, it is not the only determining factor. Communication rounds must also be considered, particularly in latency-sensitive environments. Despite incurring higher total communication, RESAD achieves lower total runtime than SIGMA at the 512th and 1024th token positions under LAN settings, and at all evaluated token positions under WAN settings, across all three models. This demonstrates that the reduction in communication rounds can effectively offset the additional communication overhead.

The performance comparison is summarized below:

Token positionLAN (Online)LAN (Total)WAN (Online)WAN (Total)
128th↓ 21.28%–37.50%↓ 21.11%–30.90%↑ 52.34%–56.18%↑ 29.90%–43.29%
512th↑ 15.61%–22.48%↑ 2.88%–4.52%↑ 56.12%–59.44%↑ 20.56%–35.49%
1024th↑ 39.48%–42.86%↑ 12.55%–16.44%↑ 58.96%–61.51%↑ 14.43%–27.70%

↑: Runtime reduction relative to SIGMA (RESAD is faster); ↓: Runtime increase relative to SIGMA (RESAD is slower). The ranges cover GPT-2 Small, GPT-2 Medium, and TinyLlama-1.1B.

To quantify this tradeoff, we establish a latency cost model incorporating computation overhead, communication rounds, and communication volume, and theoretically derive the performance crossover conditions as functions of bandwidth and effective per-round latency (Equations (54)–(56)). We further validate these conditions through experiments on GPT-2 Small and Medium at the 512th and 1024th token positions, varying both bandwidth and RTT.

At a fixed bandwidth of 20 Mbps, the interpolated crossover RTTs are approximately 45.17/59.91 ms for GPT-2 Small and 59.48/83.16 ms for GPT-2 Medium (512th/1024th tokens, respectively). Increasing the bandwidth to 50 Mbps reduces these thresholds to 16.39/19.37 ms and 21.56/25.29 ms, respectively. At 100 Mbps, the identified crossover RTTs further decrease to 6.95–8.38 ms, while GPT-2 Small at the 1024th token already outperforms SIGMA at the lowest tested RTT of 5 ms.

Conversely, at a fixed RTT of 20 ms, RESAD achieves lower total runtime at tested bandwidths of at least 50 Mbps for GPT-2 Small and 100 Mbps for GPT-2 Medium, across both token positions. At 40 ms RTT, 50 Mbps is sufficient for both models. At 100 ms RTT, RESAD outperforms SIGMA even at 20 Mbps across all four evaluated cases.

Regarding preprocessing availability, RESAD achieves lower online runtime in all additionally tested configurations when preprocessing materials are ready. Even when preprocessing costs are included, RESAD maintains lower total runtime in 89 of 120 configurations. Furthermore, at 100 Mbps and 80 ms RTT, RESAD achieves 2.33–2.50× online speedups and 1.34–1.53× total speedups.

These results demonstrate that the reduction in communication rounds can effectively outweigh the additional preprocessing communication, yielding substantial runtime improvements across a broad range of network conditions.

See Section 6.3, Table 5, and Appendix F.3, Equations (54)–(56), Tables 9–10, and Figure 22 (pp. 33–34).

AuthorsOpenReview ↗

Response to Questions Q1–Q2

We thank the reviewer for these valuable questions, which have helped us clarify the protocol complexity and parameter-selection rationale.

Q1.

We thank the reviewer for this question. We have added a detailed description of the MIC protocol and a formal round-complexity analysis in Appendix B.4.

MIC protocol. We adopt the FSS-based Multiple Interval Containment (MIC) protocol proposed by Boyle et al. (Function Secret Sharing for Mixed-Mode and Fixed-Point Secure Computation, EUROCRYPT 2021, Section 4.2 and Theorem 4). Given a secret input xx and uu public intervals Ij=[lj,rj]I_j=[l_j,r_j], MIC securely computes the interval-membership indicators hj=1x∈Ijh_j=\mathbf{1}{x\in I_j}.

Specifically, MIC consists of two phases:

Offline phase: The trusted dealer generates an input mask rinr^{\mathrm{in}}, a single DCF key pair shared across all uu intervals, and interval-specific correction shares. The corresponding FSS keys are distributed to the two computing parties.

Online phase: Given the masked input x^=x+rin\hat{x}=x+r^{\mathrm{in}}, each party performs two local DCF evaluations per interval and combines the results with its correction shares to obtain additive shares [hj][h_j] of the interval-membership indicators. Importantly, all 2u2u DCF evaluations are local and require no interaction. When a masked output is needed, the parties exchange their output shares in one communication round.

Therefore, MIC requires only one online round regardless of u, although the local DCF work and key material scale linearly with u, as established by Boyle et al.

Constant-round ApproxMax. In RESAD, the first MIC invocation evaluates bucket membership for all input elements in parallel. After aggregating bucket occupancy and encoding the nonempty buckets, the second MIC invocation identifies the rightmost nonempty bucket through its corresponding power-of-two interval. Since both MIC invocations rely on local FSS evaluations, the interaction overhead arises primarily from the necessary share-to-mask conversions. As detailed in Appendix B.4, each refinement stage requires at most three online communication rounds, including output conversion.

Furthermore, the refinement depth satisfies

d=max⁡1,⌈log⁡U(W/δ)⌉,d = \max\\{1, \lceil \log_U(W/\delta) \rceil\\},

where WW, UU, and δ\delta are public parameters independent of the input length mm. Consequently, ApproxMax requires at most 3d=O(1)3d=O(1) online communication rounds with respect to mm. Under our configuration W=32768W=32768, U=256U=256, and δ=0.5\delta=0.5, we obtain d=2d=2, requiring at most six rounds.

See Section 3.2 (pp. 4–5) and Appendix B.4 (pp. 19–21) for the MIC construction, protocol costs, and formal round-complexity proof.

Q2.

We thank the reviewer for this valuable suggestion. We have added rigorous theoretical analyses of parameter selection and the accuracy–cost tradeoffs for both ApproxMax and ILUT.

For ApproxMax, Appendix B.2 establishes a one-sided approximation error bound 0≤M~−M≤δ0\le\widetilde M-M\le\delta. To ensure numerical stability in the subsequent Softmax computation, we derive a sufficient condition δ≤0.689005\delta\le0.689005, accounting for the finite-precision exponential approximation and the reciprocal lookup domain [0.5,m][0.5,m]. Under our reference public range W=32768W=32768 and bucket budget U=256U=256, the round-first optimization selects δ=0.5\delta=0.5 and d=2d=2, requiring only six communication rounds. The two refinement stages use 256 buckets each, with widths of 128 and 0.5, respectively. Appendix B.2 also provides a generalized parameter-selection rule for other validated input ranges and bucket budgets.

For ILUT, our choice of p=8p=8 follows the widely adopted BF16 numerical-precision format, which has seven explicitly stored fraction bits and one implicit leading significand bit, giving eight significant bits. Appendix C.2 establishes a relative input-approximation bound of 21−p2^{1-p}. Requiring at least BF16-level significand resolution gives 21−p≤2−72^{1-p}\le2^{-7}, hence p≥8p\ge8. Since the table-size bound O(2p(n−p))O(2^p(n-p)) increases with pp for p<n−2p < n − 2, p=8p=8 is the smallest admissible choice under this precision criterion, balancing accuracy and lookup costs.

We further provide operator-specific error bounds at p=8p=8, including 2.9962×10−32.9962\times10^{-3} for the exponential and 1.454×10−31.454\times10^{-3} for the GeLU correction. For deeper models, Appendix C.2, Equations (52)–(53), also provides a sufficient precision-selection condition based on layer-wise error propagation and model sensitivity. These results justify our selected parameters under the stated numerical assumptions, while providing guidance for adapting them to other model architectures.

See Appendix B.2, Equations (1)–(14) (pp. 15–18), and Appendix C.2, Equations (35)–(53) (pp. 22–26).

Reviewer_bJJ7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

RESAD addresses the communication-round overhead of secure autoregressive Transformer inference in a preprocessed, semi-honest two-party setting with a trusted offline dealer. Its two main contributions are a bucket-based approximate-maximum protocol for Softmax stabilization and a compact, nonuniform interval lookup-table construction for nonlinear functions. The approximate-maximum protocol has interaction depth independent of input length for fixed range and precision parameters. The lookup construction combines significant-bit-based intervals with existing secure lookup primitives.

The authors integrate these techniques into secure decoding with KV caching and evaluate GPT-2 and TinyLlama models. Compared with SIGMA, the reported experiments show substantial reductions in interaction rounds and online communication, translating into lower WAN latency while maintaining broadly comparable benchmark accuracy. The performance evaluation primarily measures individual decoding steps at different context lengths.

Critical strengths
  1. Clear technical contribution. The approximate-maximum protocol exploits Softmax’s shift invariance to replace sequential comparisons with parallel bucket localization. Its interaction depth is independent of input length for fixed range and precision parameters.

  2. Meaningful empirical gains. Evaluations span nonlinear functions, Transformer operators, and full-model decoding steps. Substantial round reductions translate into lower total WAN runtime, supporting the practical value of the approach.

Critical weaknesses
  1. Technical details need clarification. The ILUT contribution should be distinguished more precisely from existing lookup methods. Apparent discrepancies between the pseudocode and evaluated architectures, together with incomplete truncation specifications, limit reproducibility.

  2. Approximate-maximum configuration needs clarification. Although the 3d3d-round bound is stated, the practical choices of input range, bucket budget, and approximation tolerance need clearer justification. How these choices apply across layers and models, and how out-of-range inputs are handled, remains insufficiently explained.

  3. Writing and presentation need improvement. The relationship between the algorithms, pseudocode, and evaluated architectures is sometimes difficult to follow. Terminology should also be more precise: “end-to-end” currently refers to a full-model decoding step and should be clearly distinguished from complete response generation.

Questions for authors
  1. What distinguishes ILUT from SIGMA’s lookup techniques, and how much improvement comes separately from ILUT and approximate maximum computation?

  2. Are output interfaces and timing boundaries aligned across RESAD and all baselines? Specifically, do they expose the same outputs and consistently include or exclude output reconstruction, client-side token selection, and token re-sharing? Please describe the baseline adaptations.

2 comments
AuthorsOpenReview ↗

Response to Weaknesses W1–W3

We thank the reviewer for these thoughtful questions and constructive suggestions.

W1.

We thank the reviewer for this valuable suggestion. We have further clarified the technical contributions and implementation details to improve reproducibility.

Regarding ILUT novelty, our contribution lies in the precision-aware construction of nonuniform intervals and their corresponding lookup-table entries, rather than the underlying secure lookup primitives. Unlike SIGMA's function-specific lookup optimizations, ILUT provides a unified construction applicable to different nonlinear functions. Specifically, it groups inputs according to their top-pp significant bits, evaluates the target function at interval representatives, and merges adjacent intervals with identical quantized outputs, reducing the table-entry complexity from O(2n)O(2^n) to O(2p(n−p))O(2^p(n-p)). We explicitly distinguish this construction from the existing secure lookup backends of Boyle et al. and Grotto.

Regarding architectural consistency, RESAD primarily targets GPT-style Transformer models, while TinyLlama is additionally evaluated to demonstrate the extensibility of our proposed techniques to LLaMA-style architectures. Therefore, our complete prefill and decoding pseudocode is presented based on the GPT-2 architecture. We have revised Figures 14 and 20 to correctly reflect GPT-2's Pre-LayerNorm structure, placing LayerNorm before the attention and feed-forward sublayers. For the TinyLlama extension, we additionally provide secure SiLU and RMSNorm protocols in Figures 18–19 to support its corresponding nonlinear components.

Regarding truncation, we acknowledge that the truncation protocols were inadvertently omitted from the previous manuscript. We thank the reviewer for pointing out this omission. We have now added Appendix D.1 (Figures 10–11), which explicitly presents the two truncation protocols used in our construction: local arithmetic right shift on additive shares following SecureML, and preprocessing-assisted arithmetic right shift on masked shares following ABY3. These additions clarify the truncation interfaces and their use in subsequent secure protocols.

See Sections 4–5, Appendix C.3, Appendix D.1 (Figures 10–11, p. 27), Appendix D.4 (Figure 14, p. 28), and Appendix E (Figures 18–20, pp. 30–31).

W2.

We thank the reviewer for this suggestion. We have added a theoretical parameter-selection analysis in Appendix B.2, explicitly characterizing the tradeoff among approximation tolerance, bucket budget, and communication rounds.

We use the same ApproxMax configuration across all evaluated Transformer layers and models, rather than tuning the parameters separately for each layer. Specifically, after reducing the fractional precision to b=8b=8, we adopt a conservative public input range of [−16384,16384)[-16384,16384), corresponding to a width of W=32768W=32768, and set the bucket budget to U=256U=256.

To ensure numerical stability, our analysis derives a sufficient approximation-tolerance condition δ≤0.689005\delta\leq0.689005, accounting for the exponential approximation error and the reciprocal lookup domain. Under this constraint, the round-first optimization selects δ=0.5\delta=0.5 and two refinement stages, with bucket widths of 128 and 0.5, respectively. The resulting protocol requires only six online rounds for the masked-output interface and guarantees 0≤M~−M≤0.50\leq\widetilde M-M\leq0.5 for in-range inputs.

The public range is selected conservatively to cover the attention-score values encountered in our evaluated models. We further verified that no attention-score inputs exceeded the selected public range across the evaluated models and token positions. Appendix B.2 also provides a general selection rule for adapting the range, bucket budget, and refinement depth to other settings.

See Section 3.2 and Appendix B.2, Lemmas 1–4 and Equations (1)–(14), pp. 15–18.

W3.

We thank the reviewer for this valuable suggestion. We have carefully reviewed the terminology used in related secure inference literature to clarify the precise meanings of relevant concepts and ensure consistent usage throughout the paper.

First, we have clarified the correspondence between the high-level algorithms, secure protocol implementations, and evaluated Transformer architectures. In particular, we have refined the pseudocode to reflect the architecture-specific differences between GPT-2 and TinyLlama, and explicitly linked the proposed secure primitives to their corresponding Transformer components.

Second, we have revised the terminology to distinguish single-step decoding from complete autoregressive response generation.

These revisions improve the consistency between the algorithms, pseudocode, and evaluated architectures, while making the scope of our performance claims explicit.

See Sections 5–6.3, Table 5, and Appendices D–E.

AuthorsOpenReview ↗

Response to Questions Q1–Q2

We thank the reviewer for the constructive questions and valuable feedback, which have helped us further clarify our technical contributions and experimental methodology.

Q1.

We thank the reviewer for this question. We clarify the distinction between ILUT and SIGMA's lookup techniques and have added component-level ablation experiments to quantify their individual contributions.

Regarding ILUT novelty, SIGMA relies on function-specific lookup optimizations, requiring separately tailored table constructions and indexing strategies for different nonlinear functions, such as two 8-bit LUTs for exponential, reduced-precision indexing for reciprocal, and a customized floating-point representation for reciprocal square root. Its fixed-point lookup techniques generally employ uniform quantization within their respective indexed domains. In contrast, ILUT provides a unified, precision-aware construction of nonuniform intervals applicable to different nonlinear functions, without requiring function-specific index decomposition.

Specifically, ILUT groups inputs according to their top-pp significant bits, automatically generating finer intervals for smaller magnitudes and wider intervals for larger magnitudes. This nonuniform partitioning preserves higher absolute precision near zero while efficiently covering a wide dynamic range with bounded relative approximation error. By merging adjacent intervals with identical quantized outputs, ILUT further reduces redundant entries, achieving a table-entry complexity of O(2p(n−p))O(2^p(n-p)) instead of O(2n)O(2^n). Consequently, it reduces secure lookup costs while balancing numerical accuracy and efficiency. Our contribution lies in this unified interval and table-entry construction rather than the underlying secure lookup primitives.

Regarding individual contributions, we have added full-model ablation experiments on GPT-2 Small and Medium at the 512th and 1024th token positions, comparing SIGMA, ApproxMax-only, ILUT-only, and complete RESAD under identical evaluation settings. Compared with SIGMA, ApproxMax-only reduces online runtime by 13.8%–15.2%, while ILUT-only achieves a larger reduction of 42.4%–44.1%. Combining both techniques further improves the reduction to 59.2%–61.7%. The corresponding online communication reductions are 2.0%–2.6%, 35.8%–41.1%, and 43.7%–50.9%, respectively. Meanwhile, complete RESAD maintains comparable model accuracy.

These results demonstrate that ILUT provides the larger individual online performance improvement, while ApproxMax contributes additional reductions in communication rounds and online latency when combined with ILUT.

See Section 4.1 and Appendix F.4, pp. 35–36.

Q2.

Thank you for raising this question. Yes, we align both the semantic output and the measurement boundary across all full-model comparisons. Each decode-step benchmark takes the current token and a precomputed KV cache and returns the protected full-vocabulary logits for that position. RESAD and SIGMA use the same GPT-2/LLaMA wrapper, model dimensions, fixed-point configuration, KV-cache representation, party topology, and benchmark harness; the SIGMA adaptation replaces only the corresponding nonlinear protocols. For SHAFT, we added a logits-only cached-decoding wrapper with the same model dimensions, cache length, and token position.

For the reported per-token latency and communication, timing starts immediately before the secure model forward and ends when the protected logits are produced. Their communication counters follow the same boundary. The reported total cost adds protocol-specific preprocessing—key generation, packing, dispatch, and unpacking for RESAD/SIGMA, and tuple-provider work for SHAFT.

Detailed settings can be found in our code repository https://anonymous.4open.science/r/resad-7C54.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Response to Weaknesses W1–W3

We thank the reviewer for the constructive feedback. We have extended the manuscript with component-level ablations, network crossover experiments, and theoretical analyses of the approximation parameters.

W1

To isolate the contributions, we have added full-model ablations comparing SIGMA, ApproxMax-only, ILUT-only, and complete RESAD.

The results show that ApproxMax-only reduces online runtime by 13.8%–15.2%, ILUT-only by 42.4%–44.1%, and their combination by 59.2%–61.7%, while maintaining comparable task accuracy.

See Appendix F.4, Figures 23–25, pp. 35–36.

W2.

We agree that RESAD trades additional preprocessing overhead for reduced online interaction. As shown in Table 5, RESAD benefits more from higher network latency due to its fewer communication rounds, and its advantages become more pronounced under LAN settings as the context length increases. When preprocessing materials are available in advance, RESAD achieves lower online runtime across all additionally evaluated network configurations. Even when preprocessing costs are included, RESAD maintains lower total runtime under the evaluated WAN settings and at the 512th and 1024th tokens under LAN, demonstrating that its advantages are not limited to WAN settings.

The performance comparison is summarized below:

Token positionLAN (Online)LAN (Total)WAN (Online)WAN (Total)
128th××√√
512th√√√√
1024th√√√√

√: RESAD outperforms SIGMA; ×: RESAD is slower than SIGMA. The results are consistent across GPT-2 Small, GPT-2 Medium, and TinyLlama-1.1B.

To quantify this tradeoff, we establish a latency cost model incorporating computation overhead, communication rounds, and communication volume, and theoretically derive the performance crossover conditions as functions of bandwidth and effective per-round latency (Equations (54)–(56)). We further validate these conditions through experiments on GPT-2 Small and Medium at the 512th and 1024th token positions, varying both bandwidth and RTT, and visualize the crossover boundaries in Figure 22.

At a fixed bandwidth of 20 Mbps, the crossover RTTs are approximately 45.17/59.91 ms for GPT-2 Small and 59.48/83.16 ms for GPT-2 Medium (512th/1024th tokens, respectively). Increasing the bandwidth to 50 Mbps reduces these thresholds to 16.39/19.37 ms and 21.56/25.29 ms, respectively. At 100 Mbps, the identified crossover RTTs further decrease to 6.95–8.38 ms, while GPT-2 Small at the 1024th token already outperforms SIGMA at the lowest tested RTT of 5 ms.

Conversely, at a fixed RTT of 20 ms, RESAD achieves lower total runtime at bandwidths of at least 50 Mbps for GPT-2 Small and 100 Mbps for GPT-2 Medium, across both token positions. At 40 ms RTT, 50 Mbps is sufficient for both models. At 100 ms RTT, RESAD outperforms SIGMA even at 20 Mbps across all four evaluated cases.

Furthermore, at WAN setting, RESAD achieves 2.332.33–2.50×2.50\times online speedups and 1.341.34–1.53×1.53\times total speedups.

When preprocessing materials are generated and stored in advance, online runtime is the relevant response-latency metric. Otherwise, preprocessing costs must also be included. To quantify the associated storage requirements, we additionally evaluate the per-server preprocessing storage. At the 1024th token position, 1 TB of storage per server can accommodate preprocessing materials for approximately 5,223, 1,960, and 1,092 individual decoding steps, respectively, at this fixed context length.

See Section 6.3, Table 5 (p. 9), and Appendix F.3, Equations (54)–(56), Tables 9–11, Figure 22, pp. 33–35.

W3.

We thank the reviewer for this valuable suggestion. To evaluate the cumulative effects of repeated approximation and growing KV caches, we have added a long-context autoregressive rollout experiment in Appendix F.5.

Specifically, we evaluate GPT-2 Small using a 511-token prefix from WikiText-2, followed by 513 consecutive decoding steps as the context grows to 1024 tokens. Compared with SIGMA, RESAD reduces cumulative online runtime by 15.8% under LAN and 63.2% under WAN, and protocol-total runtime by 2.3% and 34.9%, respectively. These results demonstrate that RESAD's efficiency advantages persist across consecutive decoding steps, particularly under WAN conditions.

Regarding generation quality, RESAD only replaces the secure implementations of nonlinear operators without modifying the model architecture or pretrained weights. The introduced approximation errors are small, and the resulting outputs remain closely aligned with plaintext inference, consistent with the accuracy and operator-error evaluations in Tables 6–7.

See Appendix F.5, Table 12 (p. 37), and Tables 6–7 (pp. 10 and 32).

AuthorsOpenReview ↗

Response to Questions Q1–Q2

We thank the reviewer for these thoughtful questions and constructive suggestions.

Q1.

To quantify the individual contributions, we conduct full-model ablation experiments on GPT-2 Small and Medium at the 512th and 1024th token positions, comparing SIGMA, ApproxMax-only, ILUT-only, and complete RESAD under identical network and model settings. The results show that ApproxMax-only reduces online runtime by 13.8%–15.2%, while ILUT-only achieves a larger reduction of 42.4%–44.1%. Combining both techniques further improves the reduction to 59.2%–61.7%, demonstrating their complementary contributions.

Meanwhile, the accuracy remains comparable to SIGMA. For GPT-2 Small and Medium, RESAD achieves accuracies of 32.19% and 42.79%, respectively, compared with 32.43% and 43.06% for SIGMA, corresponding to differences of only 0.24 and 0.27 percentage points. Furthermore, Appendices B.2 and C.2 theoretically establish the relationships between approximation parameters, numerical error, and protocol costs, providing guidance for selecting the bucket resolution and significant-bit precision.

See Appendix F.4, Figures 23–25, pp. 35–36, and Appendices B.2 and C.2.

Q2.

Network delay. As shown in Table 5, RESAD's lower communication-round count provides greater benefits under higher network latency, while increasing context length makes its advantages more pronounced under LAN settings. Regarding preprocessing availability, RESAD achieves lower online runtime across all additionally evaluated network configurations when preprocessing materials are ready. When preprocessing costs are included, RESAD still achieves lower total runtime under the evaluated WAN settings and at longer context lengths under LAN, although the performance advantage depends on bandwidth and RTT.**

The performance comparison is summarized below:

Token positionLAN (Online)LAN (Total)WAN (Online)WAN (Total)
128th××√√
512th√√√√
1024th√√√√

√: RESAD outperforms SIGMA; ×: RESAD is slower than SIGMA. The results are consistent across GPT-2 Small, GPT-2 Medium, and TinyLlama-1.1B.

To characterize these crossover conditions, we establish a latency cost model incorporating computation overhead, communication rounds, and communication volume, and theoretically derive the crossover conditions with respect to bandwidth and effective per-round latency (Equations (54)–(56)). We further validate these conditions through experiments on GPT-2 Small and Medium at the 512th and 1024th token positions, varying both bandwidth and RTT.

At 20 Mbps, the interpolated crossover RTTs are approximately 45.17/59.91 ms for GPT-2 Small and 59.48/83.16 ms for GPT-2 Medium (512th/1024th tokens, respectively). Increasing the bandwidth to 50 Mbps reduces these thresholds to 16.39/19.37 ms and 21.56/25.29 ms, respectively. At 100 Mbps, the identified crossover RTTs further decrease to 6.95–8.38 ms, with GPT-2 Small at the 1024th token already outperforming SIGMA at the lowest tested RTT of 5 ms.

Conversely, at a fixed RTT of 20 ms, RESAD achieves lower protocol-total runtime at bandwidths of at least 50 Mbps for GPT-2 Small and 100 Mbps for GPT-2 Medium, across both token positions. At 100 ms RTT, RESAD outperforms SIGMA even at 20 Mbps in all four evaluated cases.

Context length. As shown in Table 5, RESAD's online runtime advantage becomes more pronounced as the token position increases across all three models under both LAN and WAN settings. RESAD also achieves progressively greater reductions in total runtime, from 2.9%–4.5% at the 512th token to 12.5%–16.4% at the 1024th token. These results demonstrate that RESAD becomes increasingly advantageous for longer-context decoding, particularly under LAN settings.**

Preprocessing availability. Regarding preprocessing availability, RESAD achieves lower online runtime in all tested configurations when preprocessing materials are ready, and maintains lower protocol total runtime in 89 of 120 configurations when preprocessing costs are included. These results demonstrate that the reduction in communication rounds can effectively offset the additional preprocessing overhead across a broad range of network conditions.**

When preprocessing materials are generated and stored in advance. To quantify the associated storage requirements, we additionally evaluate the per-server preprocessing storage. At the 1024th token position, 1 TB of storage per server can accommodate preprocessing materials for approximately 5,223, 1,960, and 1,092 individual decoding steps, respectively, at this fixed context length.

See Section 6.3, Table 5, and Appendix F.3, Equations (54)–(56), Tables 9–11, Figure 22, pp. 33–35.

AuthorsOpenReview ↗

Response to Questions Q3

We thank the reviewer for these thoughtful questions and constructive suggestions.

Q3.

We have added a representative multi-token autoregressive rollout experiment in Appendix F.5 to evaluate cumulative runtime and the effects of repeated approximation.

Experimental setup. We use GPT-2 Small with the first 511 tokens of a WikiText-2 test article as the initial prefix. After prefill constructs the KV cache and produces the 512th token, we perform 513 consecutive decoding steps (token positions 512–1024) under LAN and WAN settings, comparing RESAD against SIGMA.

Cumulative performance. As shown in Table 12, RESAD reduces cumulative online runtime by 15.8% (LAN) and 63.2% (WAN), and total runtime by 2.3% (LAN) and 34.9% (WAN). It also reduces online communication rounds by 66.4%, confirming that the performance advantages persist throughout continuous decoding with a growing KV cache.

Continuation quality. RESAD preserves the original model architecture and pretrained parameters, replacing only the secure operator implementations. The resulting approximation introduces limited numerical error, and the generated outputs closely match the plaintext baseline. This observation is consistent with the comparable task accuracy and operator-level errors reported in Tables 6–7.

See Appendix F.5, Table 12 (p. 37), and Tables 6–7 (pp. 10 and 32).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.