- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper observed that standard multi-head attention still evaluates each head independently, which incurs the cost of secure attention-score computation and Softmax. The paper proposes a framework which computes attention maps for a set of real anchor heads and reconstructs the remaining maps through fixed convex combinations. It achieves 1.47-1.69x speedups in MPC-based secure transformer inference. Although the author states that they used generative AI tools for assisting with translation from Chinese to English, I feel the presentation is still difficult to follow.
Critical strengths
The motivation that reduce the computational overhead in multi-head attention in secure transformer inference makes sense.
The experimental results show improvements over other baselines.
Critical weaknesses
The writing needs a lot of improvements. For example, in the evaluation section, the texts in the main body just copy and paste all numbers in the table/figure again, don’t show any analysis or explanations about the evaluation results.
The work doesn’t provide the codebase for their experiments, and the reproducibility details are not listed.
The paper’s motivation is to reduce the computational overhead in MPC-based transformer inference, but the proposed methods are not focusing on MPC part. The authors don’t mention whether it also works on other secure transformer inference solutions.
Questions for authors
For the MPC-based secure transformer inference results in Table 2, could authors provide some analysis or explanations about the results? I think it can help to understand the significance of the work better, since the numbers in latency and communication overhead look similar with other works.
Could authors provide ablation studies to isolate the benefit of each proposed method?
What’s precision and nonlinear approximations are used in the MPC implementation? Will the numerical errors affect the logits, perplexity or accuracy of the proposed scheme?
4 comments
2. Relationship Between CASH and MPC
Reviewer comment: “The paper’s motivation is to reduce the computational overhead in MPC-based transformer inference, but the proposed methods are not focusing on MPC part.”
CASH is tailored to MPC by reducing the execution scale of expensive secure operators and avoiding input-dependent online selection overhead. Its contribution lies at the level of adapting the model computation graph to the MPC cost structure, rather than redesigning the underlying cryptographic protocol.
CASH exploits redundancy across attention heads: it performs Q/K projections, attention-score computation, and secure Softmax only for real anchor heads; reconstructs the remaining attention maps through fixed convex combinations; and retains the Value branch and output projection for every logical head. The Head-Fisher mechanism allocates the global anchor budget according to task sensitivity and reconstruction error, improving model performance under a fixed total amount of independent secure-attention computation.
This design also avoids the online interaction overhead introduced by head selection. As described in Section 3.4 and Appendix B, anchor selection, coefficient fitting, budget allocation, and head permutation are all completed offline. Online inference therefore uses a fixed computation graph and does not require head comparison, head selection, or dynamic routing that depends on private inputs. The analysis of dynamic CHAI in Figure 1(d) shows that online selection logic introduces additional interaction overhead. During inference, the reconstruction coefficients and attention maps remain private, and the secure matrix-multiplication cost introduced by reconstruction is fully included in the efficiency measurements.
Reducing MPC inference overhead by modifying model structure is an established research direction. MPCFormer (ICLR 2023) combines MPC-friendly operator approximation with knowledge distillation; MPCViT (ICCV 2023) searches attention architectures based on MPC latency; and MPCache (NeurIPS 2025) reduces private-inference overhead through MPC-friendly KV-cache eviction. CASH complements these works: it uses redundancy across attention heads to perform static reconstruction, thereby reducing the computational workload of the underlying secure operators. The actual acceleration depends on the backend's cost characteristics, and model-level modifications can be combined with lower-level secure-operator optimizations.
[1] Li et al., ICLR 2023. MPCFormer: Fast, Performant and Private Transformer Inference with MPC.
[2] Zeng et al., ICCV 2023. MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attention.
[3] Zeng et al., NeurIPS 2025. MPCache: MPC-Friendly KV Cache Eviction for Efficient Private LLM Inference.
3. Other Secure-Inference Backends
Reviewer comment: “The authors don’t mention whether it also works on other secure transformer inference solutions.”
Section 4.3 and Table 3 of the original manuscript report cross-backend experiments with CASH on BumbleBee (2PC) and PUMA (3PC). These experiments use the LLaMA-1 7B attention module and compare secure-execution costs at an input length of 8 and total anchor budgets .
On PUMA, CASH achieves -- latency speedup and -- communication reduction. On BumbleBee, the corresponding ranges are -- for latency speedup and -- for communication reduction. The results show that reducing the number of independently computed attention maps is effective across multiple secure backends, although the magnitude of the benefit depends on the operator and communication costs of each backend. These experiments retain each backend's original security model and preprocessing assumptions, as described in Appendix B of the original manuscript. The results provide direct evidence that CASH applies beyond SHAFT.
4. Ablations and Diagnostics for Individual Components
Reviewer comment: “Could authors provide ablation studies to isolate the benefit of each proposed method?”
The original manuscript already includes ablation experiments and mechanism analyses for the core components, specifically in Section 4.4, Figures 8 and 5, and Appendices C and D. We clarify below how each experiment corresponds to a component of the method.
(1) Budget allocation and task-sensitivity weighting. Section 4.4 and Figure 8 compare Uniform, Recon., and Head-Fisher on LLaMA-1 7B under a fixed total budget of . The three methods share the same candidate anchors and reconstruction coefficients and differ only in their budget-allocation criteria. Head-Fisher achieves a five-task average accuracy of 49.98% and a PPL of 24.54; the corresponding results for Recon. are 31.95% and 2146.26, while Uniform has a PPL of 1047.41. This controlled experiment validates the effects of cross-layer budget allocation and task-sensitivity weighting. Figure 5 further shows that Head-Fisher weighting increases the Spearman correlation between the proxy score and the ranking of actual NLL changes from 0.486 to 0.866, providing a mechanism-level explanation for this improvement.
(2) Convex reconstruction. Appendix C and Table 5 compare hard sharing with convex reconstruction under the same CHAI layer-wise head configuration and a total budget of . Their ARC-500 accuracies are 25.6% and 32.2%, respectively, and their PPL values are 26.91 and 16.89, respectively. Under the same head-count configuration, convex reconstruction achieves better predictive quality, providing evidence complementary to the budget-allocation ablation.
(3) Static execution. Offline head permutation consistently reorders Q/K/V, the output projection, and the reconstruction coefficients, preserving the original computational correspondence and compiling the selected anchors into fixed execution positions. Its role is supported by the execution description in Section 3.4 and Appendix B, together with the MPC cost analysis in Figure 1.
Therefore, the paper provides experimental or mechanism-level evidence for budget allocation, attention reconstruction, and static secure execution. In the revised manuscript, we will further highlight the correspondence between these existing results and the design objective of each component.
5. Reproducibility and Code
Reviewer comment: “The work doesn’t provide the codebase for their experiments, and the reproducibility details are not listed.”
We have provided anonymized code and execution instructions at: https://anonymous.4open.science/r/CASH-368B/
In addition, Section 4.1 and Appendices A, B, and D of the original manuscript describe the main reproducibility settings in detail, including the evaluated models, exact-head budgets, 128 C4 calibration sequences of length 1024, evaluation tasks, hardware and network conditions, anchor search, reconstruction-coefficient fitting, Head-Fisher normalization, budget optimization, and static compilation.
The coefficients are fitted using SLSQP with a tolerance of and a maximum of 2,000 iterations. Appendix D also provides the baseline implementations, metric definitions, and the scope of secure-inference measurements. In the revised manuscript, we will consolidate the relevant configurations so that readers can more easily connect the code with the experimental pipeline described in the paper.
6. MPC Precision, Nonlinear Approximation, and Numerical Error
Reviewer comment: “What’s precision and nonlinear approximations are used in the MPC implementation? Will the numerical errors affect the logits, perplexity or accuracy of the proposed scheme?”
Precision and nonlinear approximation. The main experiments use the SHAFT secure backend and retain SHAFT's numerical settings: a 64-bit arithmetic ring and 16-bit fixed-point fractional precision. Secure Softmax uses an input-clipping interval of and a 16-step ODE approximation, while the remaining nonlinear operations use the secure operators provided by the backend. CASH performs only MPC-friendly adaptation of the model computation graph and does not modify the underlying protocol or numerical precision.
Numerical-error analysis. We added paired experiments on four models, comparing Dense and 50%-budget CASH under native floating-point execution, a matched-operator floating-point reference, and actual MPC execution. Logits RMSE is computed relative to the matched-operator floating-point reference. Changes in PPL and accuracy are computed relative to each method's own native floating-point result, using 8,192 paired token positions and 200 ARC-Easy questions, respectively.
Table R2: Effect of the MPC numerical implementation. Each cell reports Dense / CASH; PPL is reported as a relative change, and accuracy as an absolute percentage-point change.
| Model | Logits RMSE | WikiText-2 PPL change | ARC-Easy accuracy change (percentage points) |
|---|---|---|---|
| OPT-1.3B | 0.038 / 0.057 | +0.7% / +1.8% | 0.0 / |
| OPT-2.7B | 0.047 / 0.065 | % / +1.1% | +0.5 / |
| OPT-6.7B | 0.055 / 0.096 | +1.4% / +4.3% | / |
| LLaMA-1-7B | 0.049 / 0.074 | +0.8% / +2.5% | 0.0 / |
Across the evaluated samples, CASH's PPL increases by 1.1%--4.3% relative to its own floating-point result. When moving from floating-point execution to MPC, its accuracy decreases by an additional 0.5--1.5 percentage points compared with Dense. This result shows that different attention-computation paths respond differently to the same backend numerical settings and quantifies the additional numerical impact of CASH in actual secure execution. CASH's acceleration does not come from reducing numerical precision; it comes from reducing the number of independent secure-attention computations.
Response to Reviewer (hfDY)
We thank the reviewer for the attention to our research motivation and experimental design, as well as the suggestions for improving the readability of the paper. To improve clarity, we will further organize the logical relationship among attention reconstruction, budget allocation, and static execution in the revised manuscript. We will also strengthen the connection between the experimental observations and the method's mechanisms, while reducing repetitive descriptions of the numbers in the tables. Below, we respond point by point based on evidence already included in the original manuscript and additional experiments.
1. Results in Table 2 and Their Significance
The main advantage of CASH is that it preserves higher model quality at a similar secure-computation cost. Under the same exact-attention-head budget, different methods reduce a similar amount of independent attention computation, so their latency and communication costs are close. The key difference is how effectively they preserve the information carried by the replaced attention paths. Taking LLaMA-1 7B with a 50% budget and an input length of 512 as an example, the corresponding results from Tables 1 and 2 of the original manuscript are shown below. Accuracy and PPL are obtained from plaintext quality evaluation, while latency and communication are measured under MPC.
Table R1: Predictive quality and secure-inference cost under the same budget.
Compared with SA, CASH increases latency and communication by only approximately 0.65% and 0.62%, respectively, while improving the five-task average accuracy by 13.43 percentage points. Compared with HM, CASH achieves both higher accuracy and slightly lower secure-execution cost. Averaged across four models, the quality-retention rates are 92.0% for CASH and 73.9% for SA. These results show that similar secure-computation costs do not imply similar predictive quality; the key is how the omitted independent attention maps are represented. Relative to Dense, CASH achieves 1.47--1.69× prefill speedup and reduces communication by 33.6%--41.9%, including the cost of the secure matrix multiplications required for private reconstruction. In the revised manuscript, we will organize the result analysis around this relationship between quality and cost so that the conclusions corresponding to the tables are clearer.