acceptodds
Under review as a conference paper at ICLR 2027

Partition-and-Merge Attention: Sequence-Axis Obfuscation for Private LLM Inference

Abstract

Large language models are widely deployed as cloud services, making the protection of users' sensitive inputs increasingly important. Cryptographic methods provide strong privacy guarantees, but their high cost limits their practicality for large-scale LLM inference. Obfuscation-based methods retain efficient inference but largely focus on feature transformations, leaving the sequence dimension unprotected. We introduce Partition-and-Merge Attention (PMA), which adds sequence-axis obfuscation to feature-transformed inference. During prefill, a Trusted Client computes attention within the current block, while the Server computes attention over key/value rows permuted within preceding blocks. During decode, the Client accumulates newly generated key/value rows and permutes each completed window before uploading it to the Server cache. Exact merging preserves ordinary attention in real arithmetic while keeping historical attention on the Server. Experiments on Qwen1.5-MoE-A2.7B show reduced ordered prompt reconstruction under the evaluated prefill attack. Under the evaluated completed-window reconstruction attack, PMA reduces ROUGE-L F1 from 0.9813–1.0000 for attention-path adaptations of STIP and AloePri to 0.2419, while maintaining WikiText-2 perplexity close to ordinary inference. PMA adds sequence-axis obfuscation while retaining the asymptotic attention complexity of feature-based obfuscation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
privacy-preserving inference, large language models, attention, obfuscation, client-server inference

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_pSXSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors focus on a gap between the two main approaches to private LLM inference: cryptographic methods are secure but slow, while obfuscation-based methods are fast but fail to hide sequence correspondence, i.e., which row corresponds to which token position. To address this, they propose PMA, which splits attention between a Trusted Client, handling the current block in prefill and the pending window in decode, and a Server, handling the historical K/V cache with row-permuted entries, and then exactly merges the two partial softmax states on the Client.

Critical strengths
  • The observation that feature obfuscation leaves the sequence axis unprotected is well motivated, and the causal-mask argument makes it concrete.

  • The design is clean. The exact log-sum-exp merge keeps attention identical to ordinary inference.

Critical weaknesses
  • PMA only hides the row order of the K/V cache inside attention. The FFN and residual stream still run on the Server, which receives hidden states in the ordinary feature basis and gets the merged attention output back in logical token order. I am not a cryptography expert, but even combined with a feature-obfuscation scheme, token order seems to remain exposed along this path, since feature obfuscation changes values rather than row positions. If so, the contribution is considerably weaker than claimed.

  • Tokens are permuted only within each block, and block order stays visible. An attacker therefore already knows every token’s position to within the block size. If they attempt to undo the permutation at all, what remains is reordering 16 (if block_size is 16) words into a coherent sentence, which seems fairly easy for a language model.

For these reasons I am currently leaning toward rejection. Since cryptography is not my primary area, I will weigh the other reviewers’ assessments carefully before settling on a final decision.

Questions for authors

Please see the Weaknesses section.

5 comments
AuthorsOpenReview ↗

Response to Reviewer pSXS — Part 1/1

Response to Reviewer pSXS — Part 1/1

W1 — Order exposure through FFNs and residual paths

PMA focuses on sequence correspondence in attention and the K/V cache, so the paper discusses other modules in less detail. Modules outside attention use composable feature obfuscation and sequence-axis obfuscation mechanisms. Per-token normalization, FFNs, and per-token MoE computation can follow the same row permutation as their inputs. Both residual branches use the same row permutation, and feature transformations are paired with the corresponding weights. The Client permutes inputs before sending them, while the other modules on the Server, including the final normalization and LM head, preserve the permuted physical row order. The Client handles positional encoding and causal dependencies at the PMA boundary. The output transformation is fused into the Server projection offline, so returned representations remain in the corresponding obfuscated feature coordinates. Thus, computation in the other modules need not return per-token representations to the Server in their original logical order.

We have implemented full-model obfuscation in the Qwen1.5-MoE-A2.7B decoder. Feature protection uses an AloePri-inspired expanded feature transformation, combined with vocabulary-row obfuscation, fixed perturbations to embedding/LM-head weights, and sequence-axis obfuscation (Seq). The Server holds obfuscated weights and processes inputs and intermediate representations in physical row order. The Client maintains the feature encoding/decoding maps, vocabulary mapping, and logical-position correspondence. To test whether sequence-axis obfuscation in other modules suppresses order-based associations in Server transcripts, we conduct a prefill comparison using each configuration’s Server transcript. Both configurations enable PMA and keep the other protection transformations and PMA state fixed, toggling only Seq in the other modules. TTRSR is 0.4167% with Seq enabled and 16.2500% with it disabled, a difference of 15.8333 percentage points. All 39 recovered positions in the disabled configuration come from direct correspondences between the source’s own ordered prefix and input aliases. This comparison supports the ability of sequence-axis obfuscation in other modules to suppress the evaluated prefill association attack in combination with PMA’s attention protection.

Table pSXS-1. Effect of sequence-axis obfuscation in other modules on prefill association-based recovery

Protection in other modulesPrefill TTRSR ↓
Feature + sequence-axis obfuscation0.4167% (1/240)
Feature obfuscation (Seq disabled)16.2500% (39/240)

W2 — Language-model reordering with known block order

To test whether known block boundaries help recover the original order, we directly evaluate language-model reordering. O denotes the original attack, and A adds shared row-assignment search; O+LM and A+LM reorder their respective recovered token multisets. LM reordering preserves the token multiset recovered in the first stage, including multiplicities. Conditioned on the public prefix, it uses the public candidate language model to compute the conditional log-probability of each candidate ordering and searches for an ordering with beam search of width 4. This procedure only changes the order of the recovered tokens. All four settings use the same model and frozen targets.

On the same targets, word-level ROUGE-L F1 is 0.175364 for O and 0.164826 for O+LM, and 0.223618 for A and 0.203335 for A+LM. LM reordering with known block boundaries does not improve reconstruction in either comparison.

We believe a possible reason is that the first-stage token multiset already contains errors and omissions: reordering these tokens cannot recover missing content, and an ordering preferred by the language model for fluency need not match the original text. Thus, knowing block boundaries does not lead to further improvement from language-model reordering in this experiment.

Table pSXS-2. Decode reordering with known block boundaries

MetricOAO+LMA+LM
Decode word-level ROUGE-L F10.1753640.2236180.1648260.203335

Q — Questions referring to the weaknesses

W1 describes the principles, implementation, and prefill comparison for protecting modules outside attention. W2 presents language-model reordering with known block boundaries and analyzes the results.

Dear Reviewer pSXS,

Many thanks again for your time and effort in reviewing this submission. I greatly appreciate your thoughtful comments and valuable contribution to the review process.

Now that the authors have responded to your review, could you please read through their responses and join the discussion when you have a chance?

Thank you again for your continued help and support!

Best wishes,

Your AC

AuthorsOpenReview ↗

Response to Reviewer pSXS

Dear Reviewer pSXS,

Thank you again for your thoughtful comments.

In our response, we explained how feature and sequence-axis obfuscation in other modules compose with PMA, and added a prefill comparison using Server transcripts to examine order-based associations. We also evaluated language-model reordering with known block boundaries; it did not improve reconstruction in the tested setting.

We would appreciate any further feedback on whether these explanations and experiments address your concerns about order exposure outside attention and recovery through language-model reordering. We would be happy to clarify any remaining questions.

Thank you for your time and consideration.

Reviewer_HaBSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper points out unresolved problems in obfuscation techniques for inputting sensitive data into servers and proposes solutions. It identifies a method to determine the original word order by utilizing the causal masking of attention in prefill, and a method to identify the word order by observing the update order of the KV cache in decoding, and identifies these as sequence correspondence exposure. As solutions, it proposes having the client handle the parts where masking occurs, blocking the token sequence and inserting row substitutions between blocks, and processing tokens in a window.

Critical strengths

S1. The method of cleverly separating calculations and processing some of them on the client side to prevent leaks that occur during prefill and decode, thereby preventing the transmission of specific information to the server, is novel.
S2. It is possible to perform a strict merge of client and server results without any loss of information.
S3. To address the issue of client calculation speed that arises from having the client handle some of the calculations, end-to-end verification combining the server and client is performed.

Critical weaknesses

W1. The biggest concern is the evaluation method for 5.5 End-to-end Efficiency. While trusted-client full attention is used as a comparison target in Table 4, its speed advantage is obvious, and the lack of comparison with STIP, AloePri, KV-Cloak, etc., is unnatural. Also, regarding the ratio of prefill to decode, there is a possibility that the speed will decrease significantly as the proportion of decode increases, and this point is not clear if the method only considers up to a maximum of 2048 for prefill and decode combined. Two Intel Xeon Platinum 8470Qs are hardly a standard client CPU, and we would like to see evaluations with longer token lengths using consumer CPUs, or evaluations of speed separated into prefill and decode.
W2: Sequence correspondence exposure is a new point, but isn't it an aspect that STIP and AloePri don't try to conceal in the first place and are not emphasized? Taking that into consideration, the design of Tables 1 and 2 is too focused on the PMA's concerns, and the validity of the comparison is questionable. Shouldn't we evaluate multiple more practical attack methods?

Questions for authors

Q1. I'm curious why you're using Qwen1.5-MoE-A2.7B and Qwen2.5-32B. I can't think of any reason not to use the Qwen3 series or OLMoE-1B-7B, so why deliberately use older models?

Q2. Regarding W1, if the situations in which PMA is effective and those in which it is not are clearly defined, even if there are speed issues in certain situations, it won't be fatal. Could you add some consideration of such conditions?

Q3. There is a character encoding issue in Figure 1, so it needs to be corrected.

5 comments
AuthorsOpenReview ↗

Response to Reviewer HaBS — Part 1/2

Response to Reviewer HaBS — Part 1/2

W1 — Efficiency baselines, long generation, and Client CPUs

The additional performance experiments use two AMD EPYC 7542 CPUs and an NVIDIA A100 80GB PCIe GPU to examine performance on another hardware configuration. We use Qwen1.5-MoE-A2.7B with BF16 and SDPA and report median end-to-end request latency.

We extend Table 4’s end-to-end comparison to D256 and add STIP-Attn, AloePri-Attn, and KV-Cloak (Tables HaBS-1 and HaBS-2). Model, text, reference trajectory, and Server are fixed; latency and ratios to Ordinary are reported separately. Full denotes Trusted-Client Full Attention. STIP-Attn and AloePri-Attn cover the attention path; KV-Cloak uses its online-cache implementation.

To simulate consumer-grade CPU performance, we restrict the Client to eight physical CPU cores and compare PMA with Full under the same configuration (Table HaBS-3).

In Tables HaBS-1 and HaBS-2, at fixed T2048, PMA/Ordinary decreases from 4.79×–4.80× to approximately 3.04× as D increases from 16 to 256, indicating that longer generation amortizes the initial request cost. At fixed D256, Full/PMA increases from approximately 1.16× to 2.34×–2.39× as T increases from 256 to 2048, demonstrating the benefit of reducing historical attention computation on the Trusted Client. Table HaBS-3 shows that restricting the Client to eight physical cores increases PMA latency to 1.2060× and 1.4912× its original values at T256 and T2048, respectively. PMA still provides speedups of 2.59× and 6.18× over Full under the same configuration, demonstrating the benefit in the simulated consumer-grade CPU setting.

Table HaBS-1. Efficiency comparison with third-party methods on Text 1

TDOrd. (ms)PMA (ms)PMA/Ord.Full (ms)Full/Ord.STIP-Attn (ms)STIP-Attn/Ord.AloePri-Attn (ms)AloePri-Attn/Ord.KV-Cloak (ms)KV-Cloak/Ord.
25616798.2602746.7293.4409×6826.2088.5514×831.5001.0416×804.9301.0084×893.6801.1195×
204816835.0853997.8704.7874×20895.43525.0219×880.1171.0539×848.0161.0155×1070.7361.2822×
25625611651.13234311.0532.9449×39953.7393.4292×12162.6601.0439×11684.8221.0029×11660.8231.0008×
204825611692.38235600.0433.0447×83212.7547.1168×12131.3201.0375×11786.8061.0081×11808.6871.0099×

Table HaBS-2. Efficiency comparison with third-party methods on Text 2

TDOrd. (ms)PMA (ms)PMA/Ord.Full (ms)Full/Ord.STIP-Attn (ms)STIP-Attn/Ord.AloePri-Attn (ms)AloePri-Attn/Ord.KV-Cloak (ms)KV-Cloak/Ord.
25616800.8542775.4233.4656×6901.3688.6175×834.3221.0418×806.3571.0069×882.9191.1025×
204816837.4634016.5364.7961×21321.50325.4596×876.3171.0464×849.3321.0142×1064.0961.2706×
25625611718.99834540.3472.9474×40149.6213.4260×12107.2921.0331×11722.3171.0003×11575.4170.9877×
204825611774.01535783.5633.0392×85613.4837.2714×12206.7471.0368×11675.9570.9917×11930.2521.0133×

Table HaBS-3. Simulated consumer-grade CPU performance

T/DPMA (ms)PMA/Ord.Full (ms)Full/PMAPMA reduced/original cores
256/163317.5084.1167×8593.5752.59×1.2060×
2048/165969.3887.2035×36867.7206.18×1.4912×

W2 — Comparison objectives and practical attacks

We focus on sequence correspondence left unprotected by existing attention-path adaptations: it can help attackers align obfuscated representations with candidate text. Tables 1 and 2 test whether this remains exploitable and whether sequence protection reduces reconstruction risk. STIP, AloePri, and KV-Cloak differ in objectives and coverage, so we distinguish their adaptation scopes and attacker observations. PMA protects sequence correspondence, retains historical attention on the Server, and composes with protection of other modules.

Table HaBS-4 retains the paper’s attention-path results and adds KV-Cloak’s server-visible fused BF16 online cache. On the same supplementary targets and online-cache observations, word-level ROUGE-L F1 is 0.175364 for PMA, versus 0.994048 for Ordinary and 0.997024 for KV-Cloak. This supports the value of sequence protection against reconstruction in this setting.

AuthorsOpenReview ↗

Response to Reviewer HaBS — Part 2/2

Response to Reviewer HaBS — Part 2/2

To evaluate stronger reconstruction procedures, we compare four attacks: O is the original attack, A adds row-assignment search, and O+LM and A+LM apply language-model reordering to the outputs of O and A, respectively. A jointly searches candidate tokens and their physical-row correspondences, using a shared row assignment across heads during decode. LM reordering preserves the token multiset recovered in the first stage, including multiplicities. Conditioned on the public prefix, it uses the public candidate language model to compute the conditional log-probability of each candidate ordering and searches for an ordering with beam search of width 4. This procedure only changes the order of the recovered tokens.

For a consistent comparison, all four attacks use the same model and frozen targets. Table HaBS-5 reports prefill TTRSR and decode word-level ROUGE-L F1.

A increases decode word-level ROUGE-L F1 from 0.175364 to 0.223618, confirming a benefit from searching row correspondence. O+LM and A+LM yield 0.164826 and 0.203335, respectively. We believe a possible reason why LM reordering does not further improve reconstruction is that the first-stage token multiset already contains errors and omissions: reordering these tokens cannot recover missing content, and an ordering preferred by the language model for fluency need not match the original text. These results show that row-assignment search strengthens the attack, while reconstruction quality against PMA remains low under this stronger attack.

Table HaBS-4. Extension of the original Table 2 and online-cache attacks on KV-Cloak

Methodword-level ROUGE-L F1
Ordinary1.000000
STIP-Attn1.000000
AloePri-Attn0.981349
PMA0.241855
Ordinary0.994048
PMA0.175364
KV-Cloak fused online0.997024

Table HaBS-5. Original and stronger attacks

MetricOAO+LMA+LM
Prefill TTRSR (%)0.00000.41670.00000.0000
Decode word-level ROUGE-L F10.1753640.2236180.1648260.203335

Q1 — Choice of earlier models

Our method performs broadly consistently across models, so we chose the established Qwen1.5-MoE-A2.7B and Qwen2.5-32B for our experiments. We additionally evaluate Qwen3-8B-Base, covering a newer architecture with GQA and learned Q/K normalization. Tables HaBS-6 and HaBS-7 retain the paper’s attack settings, changing only the model and required architecture/tokenizer adaptations.

On Qwen3, prefill recovery decreases from 79.1667% for Ordinary to 1.2500% for PMA, and decode word-level ROUGE-L F1 decreases from 0.993007 to 0.237665. Decode follows the matched-feature controlled evaluation in the original Table 2, with candidate and observed representations in aligned feature coordinates. These additional results show a substantial reduction in reconstruction quality on the evaluated Qwen3 configuration, supporting favorable protection across the tested models.

Table HaBS-6. Original prefill attack on Qwen3-8B-Base

MethodTTRSR
Ordinary79.1667%
PMA1.2500%

Table HaBS-7. Original decode attack on Qwen3-8B-Base

Methodword-level ROUGE-L F1
Ordinary0.993007
PMA0.237665

Q2 — Conditions for a speed advantage

PMA is suitable for private inference where the Trusted Client has basic computational resources. Same-host deployment can combine trusted isolation, such as a CPU TEE, with GPU computation. Remote deployment uses a separate Trusted Client and Server and is suited to high-bandwidth, low-RTT links.

The efficiency advantage is measured relative to Trusted-Client Full Attention: in the D256 experiment in W1, PMA retains a 2.34×–2.39× speedup at T2048; in the eight-core simulation of consumer-grade CPU performance, it retains a 6.18× speedup at T2048/D16. User interactions with LLMs often produce long contexts, making the benefit of avoiding complete historical attention on the Trusted Client more pronounced.

Applicability mainly depends on trusted execution conditions, link quality, and Client resources. Environments lacking the required trusted isolation hardware and a suitable Client–Server link are not well suited to the current deployment. Workloads extremely sensitive to additional latency, or devices with severely limited compute and memory, such as low-compute mobile phones, are also less likely to benefit. Current efficiency experiments simulate the Trusted Client with an ordinary CPU process; actual CPU TEE overhead still needs to be measured on the corresponding hardware.

Q3 — Character encoding in Figure 1

We investigated the character-encoding issue in Figure 1 and found that it mainly affected a small number of viewing environments, such as the Edge browser, while the figure displayed correctly in other viewers. We have fixed this issue.

Dear Reviewer HaBS,

Many thanks again for your time and effort in reviewing this submission. I greatly appreciate your thoughtful comments and valuable contribution to the review process.

Now that the authors have responded to your review, could you please read through their responses and join the discussion when you have a chance?

Thank you again for your continued help and support!

Best wishes,

Your AC

Reviewer_PG7jOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the sequence-correspondence leakage that persists in feature-obfuscation-based LLM inference. The authors observe that the causal attention structure during prefill and the K/V-cache insertion order during decode expose the correspondence between representations and logical token positions. To address this, the paper proposes Partition-and-Merge Attention (PMA): the trusted client computes causal attention within the current input block, or attention over the not-yet-uploaded generation window, while the server computes attention over historical K/V with shuffled row order; exact merging of partial softmax states then preserves the original model's computation. Experiments show that PMA substantially degrades text reconstruction under the evaluated attacks, largely preserves WikiText-2 perplexity, and reduces latency compared with executing full attention on the trusted client. However, the current attack evaluation does not fully exploit the permutations or the complete interaction transcript, and is therefore insufficient to fully support the claimed privacy protection.

Critical strengths
  1. The paper is clearly structured; the derivations of attention partitioning and exact merging are relatively complete, and the computational correctness is well supported.

  2. Combining sequence obfuscation with a division of computation between the trusted client and the server is somewhat inspiring; the experiments largely preserve language quality and reduce latency relative to the trusted-client full-attention scheme.

  3. The authors demonstrate solid theoretical grounding and provide sound derivations.

Critical weaknesses
  1. The discussion of the method's applicable scope and practical deployment conditions is not sufficiently focused. The paper separately mentions the reliance on an underlying feature-obfuscation scheme, the CPU-simulated trusted client, and communication overhead, but lacks a consolidated discussion explaining how these conditions jointly affect practical applicability. A clearer summary is needed of the scenarios for which the scheme is suitable, the extent of current validation, and the remaining limitations.

  2. There is no analysis of key parameters. In the experiments, the block size B and the window size τ are both fixed at 16, yet these two parameters directly determine the computational burden on the trusted client and the scope of sequence obfuscation. The paper does not adequately show how different settings affect reconstruction quality, latency, and communication cost, nor does it discuss how to choose these parameters in practice.

  3. The privacy experiments have limited coverage. The reconstruction experiments are based mainly on 30 source texts from WikiText-2 and focus on Qwen1.5-MoE-A2.7B. Although runtime results on Qwen2.5-32B are provided, no corresponding privacy evaluation is included. It is therefore difficult to judge whether the observed protection generalizes to larger models and to different types of inputs.

  4. The efficiency experiments do not yet adequately cover the full continuous generation process. The formal timing includes only 16 decode steps, while the first window upload occurs at step 17. Although Appendix H verifies the upload functionality, there is no timing measurement spanning multiple windows with repeated uploads. The current results therefore provide insufficient support for the efficiency advantage in long-term generation scenarios.

  5. As part of a complete private inference system, the protection boundary of the method requires a more focused description. The authors note that PMA mainly protects the attention interface, while the remaining hidden states and model computation rely on the underlying feature-obfuscation scheme. However, privacy validation of the fully integrated system is lacking, and further explanation is needed regarding which information remains visible to the server and whether such information affects the protection claimed in this work.

Questions for authors
  1. When the block size B and the window size τ vary, how do the privacy protection, computation, and communication overhead change? What is the main rationale for choosing 16?

  2. When generation spans multiple window-upload cycles, can PMA's average latency still maintain the advantage reported in the paper?

4 comments
AuthorsOpenReview ↗

Response to Reviewer PG7j — Part 1/2

Response to Reviewer PG7j — Part 1/2

W1 — Applicability and deployment conditions

PMA is intended for collaborative private inference in which the Trusted Client has sufficient compute and memory resources. It primarily protects sequence correspondence in attention and the K/V cache and composes with feature and sequence-axis obfuscation in other modules. Same-host deployment can combine trusted isolation, such as a CPU TEE, with GPU computation. Remote deployment uses a separate Trusted Client and Server and is better suited to high-bandwidth, low-RTT links. For requests with a long history, PMA retains historical attention on the Server while the Client mainly processes the current block or window. This makes the computational and latency benefits more pronounced relative to keeping complete attention on the Trusted Client.

Our current experiments simulate the Trusted Client with an ordinary CPU process, validating functionality and performance under this computation partition. Controlled network experiments and experiments simulating consumer-grade CPU performance examine the effects of communication and compute resources. Overall, applicability depends on trusted execution conditions, link quality, and Client resources. Environments that lack the required trusted isolation and cannot provide a suitable Client–Server link, as well as workloads extremely sensitive to added latency or devices with severely limited compute and memory, such as low-compute mobile phones, are less likely to benefit from the current design.

W2 — Analysis of key parameters

To test another hardware configuration, we use dual AMD EPYC 7542 CPUs and an NVIDIA A100 80GB PCIe GPU, with Qwen1.5-MoE-A2.7B, BF16 and SDPA. We report median end-to-end request latency.

We vary B and τ\tau separately, fixing the other at 16 (Tables PG7j-2 and PG7j-3). Timing uses T2048/D16 for B and T256/D256 for τ\tau. Reconstruction for τ\tau uses the first uploaded window, so target length varies with τ\tau. Ord. denotes Ordinary; Full denotes Trusted-Client Full Attention.

Recovery remains low across B values. Larger τ\tau generally reduces decode recovery, with little change for τ\tau≥16. We believe that more target positions make joint search harder.

Latency remains stable over the tested range. B increases Client within-block computation; τ\tau increases pending K/V capacity and computation but lowers upload frequency. Batched transfers keep core bytes and rounds unchanged across B. At D256, τ=4\tau=4, 16, and 64 give 63, 15, and 3 uploads per layer, with protocol-derived decode traffic of 143.625, 141.375, and 132.375 MiB.

Table PG7j-2. Reconstruction and request latency across B values

Timing input: Text 1

BTTRSR ↓Ord. (ms)PMA (ms)PMA/Ord.Full (ms)Full/Ord.
40.4167% (1/240)840.6074098.2674.8754×21392.96025.4494×
80.0000% (0/240)842.0904101.7954.8710×21474.94825.5020×
160.0000% (0/240)835.0853997.8704.7874×20895.43525.0219×
320.8333% (2/240)842.5774076.2824.8379×21583.24425.6158×
640.0000% (0/240)842.5784173.8024.9536×21557.87725.5856×

Timing input: Text 2

BTTRSR ↓Ord. (ms)PMA (ms)PMA/Ord.Full (ms)Full/Ord.
40.4167% (1/240)840.0264164.4224.9575×21437.93525.5206×
80.0000% (0/240)839.5474150.2384.9434×21495.47225.6037×
160.0000% (0/240)837.4634016.5364.7961×21321.50325.4596×
320.8333% (2/240)841.8544072.9144.8380×21539.26225.5855×
640.0000% (0/240)838.6974108.7694.8990×21631.40725.7917×

Table PG7j-3. Reconstruction and request latency across τ\tau values

Timing input: Text 1

τ\tauROUGE-L F1 ↓Ord. (ms)PMA (ms)PMA/Ord.Full (ms)Full/Ord.
40.39781711752.77133384.4092.8406×40369.8553.4349×
80.26029211756.24436838.9153.1336×39986.8333.4013×
160.17536411651.13234311.0532.9449×39953.7393.4292×
320.18430311696.27334039.0412.9102×40203.4233.4373×
640.17765511726.95433592.3562.8645×40507.1803.4542×

Timing input: Text 2

τ\tauROUGE-L F1 ↓Ord. (ms)PMA (ms)PMA/Ord.Full (ms)Full/Ord.
40.39781711706.93933274.4372.8423×40357.0923.4473×
80.26029211720.49436701.2833.1314×40384.3683.4456×
160.17536411718.99834540.3472.9474×40149.6213.4260×
320.18430311658.73733976.4232.9142×40509.9723.4746×
640.17765511695.90833410.6172.8566×40203.5723.4374×
AuthorsOpenReview ↗

Response to Reviewer PG7j — Part 2/2

Response to Reviewer PG7j — Part 2/2

W3 — Model and dataset coverage

We add original-attack tests on Qwen2.5-32B and CNN/DailyMail news, contrasting with WikiText-2 encyclopedic text. On 32B, PMA reduces prefill recovery from 80.8333% to 0% and decode word-level ROUGE-L F1 from 1.000000 to 0.219289 relative to Ordinary. On Qwen1.5-MoE-A2.7B news inputs, PMA achieves 0.4167% and 0.184296, respectively. These results show that our method performs well across different models and datasets.

Table PG7j-4. Original prefill attack on Qwen2.5-32B

MethodTTRSR (%)
Ordinary80.8333
PMA0.0000

Table PG7j-5a. Original decode attack on Qwen2.5-32B

Methodword-level ROUGE-L F1
Ordinary1.000000
PMA0.219289

Table PG7j-5b. Original attacks on PMA with news text

StageMetricResult
PrefillTTRSR0.4167%
Decodeword-level ROUGE-L F118.4296%

W4 — Efficiency during sustained generation

We extend generation to D128 and D256, replaying the same reference trajectory across methods and including actual uploads in request latency. Table PG7j-6 extends the paper’s Table 4 on the new server, reporting median latency and ratios to Ordinary.

At D256, PMA remains approximately 1.16× faster than Full at T256 and 2.34×–2.39× at T2048 despite repeated uploads. The growing advantage over the tested range is consistent with PMA retaining historical attention on the Server while the Client handles only the current block or window.

Table PG7j-6. Long-generation request latency on two inputs

Text 1

TDOrd. (ms)PMA (ms)PMA/Ord.Full (ms)Full/Ord.
25616798.2602746.7293.4409×6826.2088.5514×
2561285881.08517508.1642.9770×21752.4333.6987×
25625611651.13234311.0532.9449×39953.7393.4292×
204816835.0853997.8704.7874×20895.43525.0219×
20481285922.03118887.3203.1893×49426.8748.3463×
204825611692.38235600.0433.0447×83212.7547.1168×

Text 2

TDOrd. (ms)PMA (ms)PMA/Ord.Full (ms)Full/Ord.
25616800.8542775.4233.4656×6901.3688.6175×
2561285888.95017766.4553.0169×21654.3203.6771×
25625611718.99834540.3472.9474×40149.6213.4260×
204816837.4634016.5364.7961×21321.50325.4596×
20481285895.45418902.1313.2062×50312.7608.5342×
204825611774.01535783.5633.0392×85613.4837.2714×

W5 — Protection boundary of the complete system

PMA focuses on sequence correspondence in attention and the K/V cache, so the paper discusses other modules in less detail. Other modules use feature and sequence-axis obfuscation: per-token normalization, FFNs and MoE follow input permutations, with identical permutations on residual branches and transforms paired with weights. The Client permutes inputs and handles positions and causality at the PMA boundary; Server modules, including final normalization and the LM head, preserve physical order. Offline-fused output transforms keep returned representations obfuscated.

We have implemented full-model obfuscation in Qwen1.5-MoE-A2.7B by combining AloePri-inspired expanded feature transforms, vocabulary-row obfuscation, fixed embedding/LM-head weight perturbations and Seq. To test association through Server transcripts, both prefill configurations enable PMA and fix all other transforms and PMA state, toggling only Seq in other modules. TTRSR is 0.4167% with Seq and 16.2500% without it (Table PG7j-7): Seq suppresses the evaluated ordered-prefix attack alongside PMA.

Table PG7j-7. Effect of sequence-axis obfuscation in other modules on prefill association-based recovery

Protection in other modulesPrefill TTRSR ↓
Feature + sequence-axis obfuscation0.4167% (1/240)
Feature obfuscation (Seq disabled)16.2500% (39/240)

Q1 — Parameter variation and the choice of 16

B=τ=16B=\tau=16 is a reference setting balancing Client resources, upload frequency, and reconstruction results, also informed by prior work. PagedAttention (§7.2) defaults to 16; KV-Cloak (§VI-A) tests 16, 32, and 64 and uses 16 in its quality evaluation.

Our sweeps show stable latency; τ=16\tau=16 gives decode word-level ROUGE-L F1 of 0.175364 with bounded Client window capacity. See W2 and Tables PG7j-2 and PG7j-3 for computation, communication and recovery.

Q2 — Average latency across repeated uploads

Yes. With repeated uploads included, D256 retains approximately 1.16× and 2.34×–2.39× speedups over Full at T256 and T2048, respectively (W4, Table PG7j-6).

Dear Reviewer PG7j,

Many thanks again for your time and effort in reviewing this submission. I greatly appreciate your thoughtful comments and valuable contribution to the review process.

Now that the authors have responded to your review, could you please read through their responses and join the discussion when you have a chance?

Thank you again for your continued help and support!

Best wishes,

Your AC

AuthorsOpenReview ↗

Response to Reviewer PG7j

Dear Reviewer PG7j,

Thank you again for your thoughtful comments.

In our response, we consolidated the discussion of deployment conditions and added block/window-size experiments, privacy evaluations on Qwen2.5-32B and CNN/DailyMail, and timing over 256 decode steps with repeated uploads. We also clarified the protection of other modules and included a prefill comparison using Server transcripts.

We would appreciate any further feedback on whether these additions address your concerns about applicability, evaluation coverage, and sustained-generation efficiency. We would be happy to clarify any remaining questions.

Thank you for your time and consideration.

Reviewer_epELOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that feature-obfuscation schemes for private LLM inference (e.g., STIP, AloePri) leave sequence correspondence exposed: causal-mask support counts reveal token order in prefill, and K/V-cache insertion order reveals it in decode. PMA splits attention between two parties. A Trusted Client computes masked attention within the current block (prefill) or pending window (decode). A Server computes unmasked attention over earlier blocks whose rows are independently permuted.

Critical strengths
  1. The paper identifies a real, well-argued gap in feature obfuscation. The causal-support counting argument in Sec. 3.1 is crisp. The fix is simple, exact in real arithmetic, and keeps Client attention cost bounded by the block/window size.

  2. The ablation that re-exposes causal support (TTRSR 0% → 30%) isolates the contribution of the causal partition. The latency results show a clear scaling advantage over Trusted-Client Full Attention.

Critical weaknesses

The full pipeline appears to re-expose the order that the attention interface hides. In the default prototype, the Server sends normalized hidden states in the Ordinary basis to the Client and receives attention outputs back in logical token order. The Server therefore observes per-token states in logical order at every layer, in the plaintext basis at layer 0. The permuted Q/K/V rows are deterministic functions of these states under fixed, request-independent signed feature permutations. Unless the surrounding obfuscation scheme breaks this link, the row permutations can be undone by matching. The attacks examine only attention representations in isolation, so the security claim for the full pipeline is not supported.

Questions for authors
  1. In the evaluated prototype, how is the Server prevented from linking the logically ordered hidden states and returned attention outputs to the permuted Q/K/V rows? Which feature-obfuscation scheme is assumed end to end? Does reconstruction from the Server’s full transcript, not only the attention interface, remain low?

  2. How do the attacks perform when the attacker solves the within-block assignment? Examples include matching rows to candidate tokens at each logical position using RoPE-dependent keys, or recovering each block’s token multiset and reordering it with a language model.

  3. How do leakage and cost change with block/window size (e.g., B = τ ∈ {4, 8, 32, 64})? In particular, how does traffic scale, given 1.16 GiB per request and 12.6 s latency at 1 Gbps for T = 2048?

6 comments
AuthorsOpenReview ↗

Response to Reviewer epEL — Part 1/2

Response to Reviewer epEL — Part 1/2

W1 — Order re-exposure in the complete inference pipeline

PMA focuses on sequence correspondence in attention and the K/V cache, so the paper discusses other modules in less detail. Modules outside attention use composable feature obfuscation and sequence-axis obfuscation mechanisms.

Per-token normalization, FFNs, and MoE computation follow the input row permutation; both residual branches use the same permutation, with feature transforms paired with their weights. The Client permutes inputs, and Server modules, including final normalization and the LM head, preserve physical row order. The Client handles positional encoding and causality at the PMA boundary. Fusing the output transform into the Server projection offline keeps returned representations obfuscated.

We have implemented full-model obfuscation in the Qwen1.5-MoE-A2.7B decoder. Q1 and Table epEL-1 present the prefill comparison.

Q1 — Breaking associations between other modules and permuted rows

We have implemented full-model obfuscation by combining an AloePri-inspired expanded feature transformation with vocabulary-row obfuscation, fixed embedding/LM-head weight perturbations, and sequence-axis obfuscation (Seq). The Server processes obfuscated inputs and intermediate states in physical row order. The Client maintains feature encoding/decoding maps, vocabulary mapping, and logical positions at the PMA boundary.

To test whether Seq suppresses order-based associations, we compare prefill attacks using each configuration’s Server transcript. Both configurations enable PMA and fix all other protection transformations and PMA state, toggling only Seq in other modules.

With Seq, TTRSR is 0.4167%, versus 16.2500% without it (Table epEL-1). Thus, sequence-axis obfuscation in other modules suppresses the evaluated attack’s use of ordered prefixes to recover token positions, complementing PMA’s attention protection.

Table epEL-1. Effect of sequence-axis obfuscation in other modules on prefill association-based recovery

Protection in other modulesPrefill TTRSR ↓
Feature + sequence-axis obfuscation0.4167% (1/240)
Feature obfuscation (Seq disabled)16.2500% (39/240)

Q2 — Within-block assignment and language-model reordering

To test whether explicit row-correspondence search and language-model reordering strengthen reconstruction, we compare four attacks: O is the original attack; A adds row-assignment search; O+LM and A+LM apply language-model reordering to the outputs of O and A, respectively. All four use the same model, data, and targets, changing only row search or reordering.

A compares each candidate’s position-dependent RoPE representation with observed rows, searching unused physical rows within the block or window to select a token and its row correspondence. Decode uses shared row assignment: all heads use the same candidate row index.

LM reordering preserves the recovered token multiset and multiplicities. Conditioned on the public prefix, the public candidate language model scores candidate orderings by conditional log-probability; beam search of width 4 searches their order. Only token order changes. O+LM and A+LM use the multisets recovered by O and A, respectively.

In Table epEL-2, assignment attack A increases decode word-level ROUGE-L F1 from 0.175364 to 0.223618. With LM reordering, O+LM and A+LM achieve 0.164826 and 0.203335, respectively.

Row-assignment search improves recovery, but overlap with the target remains low. We believe PMA’s per-head correspondence perturbation makes simultaneous alignment across heads difficult, potentially limiting shared row assignment’s benefit.

LM reordering does not improve word-level ROUGE-L F1 in this experiment. We believe a possible reason is that the first-stage token multiset already contains errors and omissions: reordering these tokens cannot recover missing content, and an ordering preferred by the language model for fluency need not match the original text.

Table epEL-2. Within-block assignment and LM reordering

MetricOAO+LMA+LM
Prefill TTRSR (%)0.00000.41670.00000.0000
Decode word-level ROUGE-L F10.1753640.2236180.1648260.203335

Q3 — Block/window sizes and communication cost

The additional performance experiments use two AMD EPYC 7542 CPUs and an NVIDIA A100 80GB PCIe GPU to examine performance on another hardware configuration. We use Qwen1.5-MoE-A2.7B with BF16 and SDPA and report median end-to-end request latency.

AuthorsOpenReview ↗

Response to Reviewer epEL — Part 2/2

Response to Reviewer epEL — Part 2/2

To examine how the parameters affect reconstruction and cost, we vary B and τ\tau separately, fixing the other parameter at 16 (Tables epEL-3 and epEL-4). Performance is measured at T2048/D16 for B and T256/D256 for τ\tau. The τ\tau reconstruction evaluation uses the first uploaded window, so target length varies with τ\tau. Ord. denotes Ordinary, and Full denotes Trusted-Client Full Attention.

Tables epEL-3 and epEL-4 show low prefill recovery across the tested B range. Increasing τ\tau generally reduces decode reconstruction quality, with only small changes for τ\tau≥16. We believe that larger windows contain more positions to recover, making joint search more difficult. Request latency also remains relatively stable across both parameter sweeps.

Communication latency is primarily determined by transferred bytes, link bandwidth, and the RTT of serial interactions. B controls the Client’s within-block computation during prefill. The current implementation transfers block tensors in batches: at fixed T2048/D16 and τ=16\tau=16, B=4B=4, 8, 16, 32, and 64 all produce 1161.023 MiB of core protocol tensors and the same number of interaction rounds. Thus, B mainly changes the computation partition and obfuscation granularity. In the paper’s controlled network experiment at 1 Gbps and 1 ms RTT, request latency increases from 1767.318 ms on the same host to 12582.258 ms. This is consistent with the cost of transferring the recorded bytes at the specified bandwidth, and indicates that high-bandwidth, low-RTT links are better suited to realizing the computational benefit.

For the paper’s MHA configuration, the core decode traffic is

Vdecode(D)=DL(4dw+4nh)+⌊D−1τ⌋L(2τdw),V_{\mathrm{decode}}(D)=DL(4dw+4n_h)+\left\lfloor\frac{D-1}{\tau}\right\rfloor L(2\tau dw),

where D is the number of decode steps, L=24L=24 is the number of layers, d=2048d=2048 is the total head width, nhn_h=16 is the number of heads, and w=2w=2 bytes is the BF16 element width; log-normalizers use 4 bytes. This count includes only core protocol tensors, excluding metadata and encapsulation. The first term accounts for per-step interactions, and the second for completed-window uploads. At fixed D, increasing τ\tau combines uploads into fewer, larger batches. When D is much larger than τ\tau, the average upload traffic per token approaches 2Ldw. For D256, τ=4\tau=4, 16, and 64 result in 63, 15, and 3 uploads, respectively, with formula-derived core decode traffic of 143.625, 141.375, and 132.375 MiB. At T2048/D16 and τ=16\tau=16, decode traffic is approximately 6.023 MiB, while the remaining approximately 1155 MiB comes from prefill.

Table epEL-3. Reconstruction and request latency across B values

Timing input: Text 1

BTTRSR ↓Ord. (ms)PMA (ms)PMA/Ord.Full (ms)Full/Ord.
40.4167% (1/240)840.6074098.2674.8754×21392.96025.4494×
80.0000% (0/240)842.0904101.7954.8710×21474.94825.5020×
160.0000% (0/240)835.0853997.8704.7874×20895.43525.0219×
320.8333% (2/240)842.5774076.2824.8379×21583.24425.6158×
640.0000% (0/240)842.5784173.8024.9536×21557.87725.5856×

Timing input: Text 2

BTTRSR ↓Ord. (ms)PMA (ms)PMA/Ord.Full (ms)Full/Ord.
40.4167% (1/240)840.0264164.4224.9575×21437.93525.5206×
80.0000% (0/240)839.5474150.2384.9434×21495.47225.6037×
160.0000% (0/240)837.4634016.5364.7961×21321.50325.4596×
320.8333% (2/240)841.8544072.9144.8380×21539.26225.5855×
640.0000% (0/240)838.6974108.7694.8990×21631.40725.7917×

Table epEL-4. Reconstruction and request latency across τ\tau values

Timing input: Text 1

τ\tauROUGE-L F1 ↓Ord. (ms)PMA (ms)PMA/Ord.Full (ms)Full/Ord.
40.39781711752.77133384.4092.8406×40369.8553.4349×
80.26029211756.24436838.9153.1336×39986.8333.4013×
160.17536411651.13234311.0532.9449×39953.7393.4292×
320.18430311696.27334039.0412.9102×40203.4233.4373×
640.17765511726.95433592.3562.8645×40507.1803.4542×

Timing input: Text 2

τ\tauROUGE-L F1 ↓Ord. (ms)PMA (ms)PMA/Ord.Full (ms)Full/Ord.
40.39781711706.93933274.4372.8423×40357.0923.4473×
80.26029211720.49436701.2833.1314×40384.3683.4456×
160.17536411718.99834540.3472.9474×40149.6213.4260×
320.18430311658.73733976.4232.9142×40509.9723.4746×
640.17765511695.90833410.6172.8566×40203.5723.4374×

Dear Reviewer epEL,

Many thanks again for your time and effort in reviewing this submission. I greatly appreciate your thoughtful comments and valuable contribution to the review process.

Now that the authors have responded to your review, could you please read through their responses and join the discussion when you have a chance?

Thank you again for your continued help and support!

Best wishes,

Your AC

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary to the AC

Dear Area Chair,

Thank you for your time and for overseeing the review and discussion process.

In our rebuttal, we provided additional experiments and technical clarifications in response to the reviewers’ concerns. We summarize the main additions below.

Reviewer epEL — Protection across the inference pipeline and stronger attacks

  • Clarified how feature and sequence-axis obfuscation in other modules compose with PMA, and implemented this combination in Qwen1.5-MoE-A2.7B. In the evaluated prefill association attack using Server transcripts, enabling Seq in other modules reduces TTRSR from 16.2500% to 0.4167%.
  • Added within-block row-assignment search and language-model reordering. Row-assignment search strengthens reconstruction, while recovery against PMA remains low; LM reordering does not further improve recovery in the evaluated setting.
  • Added block/window-size experiments and explained their effects on Client computation, communication volume, and upload frequency.

Reviewer PG7j — Applicability, parameter selection, and evaluation coverage

  • Consolidated the discussion of trusted execution conditions, Client resources, and network requirements.
  • Added parameter sweeps and privacy evaluations on Qwen2.5-32B and CNN/DailyMail news inputs.
  • Extended timing to 256 decode steps, including repeated window uploads.

Reviewer HaBS — Efficiency comparisons and model coverage

  • Added comparisons with STIP-Attn, AloePri-Attn, and KV-Cloak, longer-generation timing, and an eight-core Client setting to simulate consumer-grade CPU performance.
  • Clarified the comparison scopes and added KV-Cloak online-cache reconstruction results and stronger attacks.
  • Added Qwen3-8B-Base privacy experiments and clarified deployment scenarios.

Reviewer pSXS — Order exposure outside attention and LM reordering

  • Explained and evaluated sequence-axis obfuscation in other modules to address associations between ordered inputs and obfuscated representations.
  • Evaluated LM reordering with known block boundaries.

We hope this summary helps with your assessment of the paper. We would welcome any remaining questions. If appropriate, we would also appreciate your help in encouraging further reviewer feedback before the discussion closes.

Thank you for your time and consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.