acceptodds
Under review as a conference paper at ICLR 2027

Traces of a Prompt: Uncovering and Mitigating Privacy Risks of Exposed States in TEE-Partitioned LLM Inference

Abstract

Cloud-based LLM services must protect sensitive prompts while maintaining efficient inference. To balance these goals, partitioned inference keeps selected computations inside a trusted execution environment (TEE) while offloading others to an untrusted GPU. However, the intermediate states exposed by this partitioning may reveal private inputs. We introduce Prompt-State Matching Attack, which constructs direct, invariant, and joint-state fingerprints for vocabulary-based prompt reconstruction. Given observations of selected states and access to chosen-input probing, the attack achieves near-perfect token recovery through exhaustive vocabulary search in controlled FP32 experiments across four LLMs. We experimentally evaluate token-aligned transformations and analyze how relationships among exposed states support reconstruction. To mitigate this leakage, we propose StateVeil, which protects outsourced linear computation using Dual-LPN-based sparse encoding. We provide a conditional reduction for transcript privacy, complemented by concrete attack-cost estimates. Even with the true prefix supplied at each position, the evaluated Euclidean direct matcher achieves token recovery from StateVeil's masked outputs across four models and four protected projections. StateVeil achieves end-to-end latency comparable to the evaluated partitioned-inference baseline, while model quality remains comparable to the original models in most evaluated model–task settings.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large language models, Privacy-preserving inference, Trusted execution environments, Prompt privacy

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EsmkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that in TEE-partitioned LLM inference, where an untrusted GPU runs the linear projections, the exposed intermediate states can leak the prompt. Its Prompt-State Matching Attack lets a model-aware provider probe every vocabulary token on its own copy of the model and match direct, invariant, or joint-state fingerprints against the victim’s exposed states. This recovered 100% of tokens from unprotected Q/K/V/S/A/O at first, middle, and last layers across four LLMs, and 99.81–100% under masking, permutation, and orthogonal encodings. As a defense, StateVeil masks quantized activations with fresh sparse Dual-LPN masks over a finite field, using a shared public matrix. The TEE unmasks the exact integer products the GPU returns.

Critical strengths
  • The attack analysis is systematic and shows that common transformations don’t protect exposed states。

  • The proposed framework is evaluated against its own attack, with carefully claims.

Critical weaknesses
  • The attack is largely a re-application of known vocabulary matching under assumptions that favor the attacker, and parts of it are not measured. The 100% recovery numbers come from an idealized setting: FP32, deterministic exact computation, and a probe model identical to the victim’s. They also assume the adversary can see the exposed states and position-aligned heads. So the paper doesn’t show that these attacks work against the actual deployed systems it names (Talaria, KV-Cloak, STIP) in realistic conditions.

  • StateVeil’s security evidence is thinner than the abstract suggests.The privacy theorem is conditional on an exact-weight Dual-LPN assumption. The cost estimates (about 2^162–2^168) are for classical attacks on a single instance. The usable guarantee depends on a heuristic exposure cap (about 2^24 token positions per master matrix). The paper itself says this is “not a proven security limit.” In practice that is roughly 47k requests of 256+100 tokens before the matrix must be rotated, which is a real operational cost that is barely evaluated. The empirical defense test is a single matcher (Euclidean direct matching) at 0% recovery. That shows only that this specific attack fails, and it has no stronger or learned adversaries.

  • Limited novelty. The attack applies the existing vocabulary-matching recipe [1,2] to more tensors (Q, K, V, S, A, O). The 100% direct-matching result follows from determinism: an unmasked state matched against an identical probe model should match exactly. The “invariant” and “joint” strategies reuse known tricks, such as sorting to remove permutations and paired Q/K encodings from KV-Cloak and STIP. The joint result is derived analytically (Eq. 8), not measured. StateVeil combines Slalom/Talaria-style additive masking, int8 quantization with modular arithmetic, and an off-the-shelf Dual-LPN assumption. Its new elements are a shared public master matrix and precomputed products, which are engineering choices. The privacy theorem is a conditional reduction to an existing assumption. The paper never states precisely what is new relative to these baselines.

[1] Rahul Krishna Thomas, Louai Zahran, Erica Choi, Akilesh Potti, Micah Goldblum, and Arka Pal. Hidden no more: Attacking and defending private third-party LLM inference. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 59434–59469. PMLR, 2025. [2] Zhifan Luo, Shuo Shao, Su Zhang, Lijing Zhou, Yuke Hu, Chenxu Zhao, Zhihao Liu, and Zhan Qin. Shadow in the cache: Unveiling and mitigating privacy risks of KV-cache in LLM inference. In Network and Distributed System Security Symposium. Internet Society, 2026.

Questions for authors

see weakness

3 comments

1. Attack assumptions and additional experimental evidence

We thank Reviewer Esmk for the detailed comments on the attack assumptions and experimental coverage.

Our intended adversary is the model-aware cloud provider, which knows the model weights, tokenizer, and inference implementation. Access to an identical probe model follows from this threat model. The victim’s TEE secrets remain unavailable, and each matching construction requires fingerprints that can be compared without those secrets. Our conclusions are scoped to the states observable under the specified outsourcing configuration.

Additional FP16/BF16 and MoE experiments. We have expanded the evaluation to include FP16/BF16 experiments on Llama-3.2-1B, Llama-3.1-8B, Qwen2.5-7B, Qwen3-8B, and OLMoE-1B-7B-0924, together with FP32 experiments on OLMoE.

Direct matching covers the first, middle, and last layers and all six states, Q,K,V,S,A,OQ,K,V,S,A,O. Invariant matching is evaluated at layer 0 using:

  • Gram: vec(X≤iX≤i⊤)\mathrm{vec}(X_{\le i}X_{\le i}^{\top}).
  • Projection: vec(XiP)\mathrm{vec}(X_iP).
  • Sorting: sort(vec(Xi))\mathrm{sort}(\mathrm{vec}(X_i)).

Gram and Projection are evaluated on Q,K,V,A,OQ,K,V,A,O, with SS not evaluated; Sorting covers all six states.

Under FP16, every evaluated direct-matching and invariant-matching configuration achieves 100% token recovery across all five models. In the additional FP32 OLMoE experiments, direct matching achieves 100% for all six states at layers 0, 8, and 15, and every evaluated invariant fingerprint achieves 100% at layer 0.

BF16 direct-matching results:

ModelLayerQKVSAO
Llama-3.2-1B0, 8, 15100%100%100%100%100%100%
Llama-3.1-8B0, 16, 31100%100%100%100%100%100%
Qwen2.5-7B0100%100%100%100%93.6%100%
Qwen2.5-7B1499.7%97.9%98.0%99.5%88.3%99.2%
Qwen2.5-7B2799.8%99.7%99.8%37.3%7.6%57.6%
Qwen3-8B0100%100%100%100%100%100%
Qwen3-8B18, 35100%100%100%96.7%96.7%100%
OLMoE-1B-7B-09240100%100%100%100%100%100%
OLMoE-1B-7B-09248100%100%100%18.1%96.6%100%
OLMoE-1B-7B-092415100%100%100%59.6%100%100%

Under BF16, direct matching of Q,K,VQ,K,V maintains 97.9%–100% recovery across all evaluated models and layers. However, recovery drops substantially for some S,A,OS,A,O states. For example, the corresponding rates at the last layer of Qwen2.5-7B are 37.3%, 7.6%, and 57.6%. These results demonstrate that recovery depends on numerical precision, model, layer, and exposed state.

BF16 invariant-matching results: The following table lists configurations with recovery below 100%. Every tested state in all remaining evaluated configurations achieves 100%. All experiments use layer 0; “—” denotes a state that was not evaluated.

ModelFingerprintQKVSAO
Llama-3.2-1BGram100%100%100%—100%99.8%
Llama-3.1-8BGram100%100%100%—100%99.8%
Qwen2.5-7BGram100%84.7%100%—90.1%90.6%
Qwen2.5-7BProjection100%100%100%—92.6%100%
Qwen2.5-7BSorting100%100%100%100%77.5%97.0%

Additional joint-state matching experiments. We also test the specified KV-Cloak and STIP Q/K transformations offline.

For KV-Cloak’s operator-fused transformation, Q~=QM−T\widetilde Q=QM^{-T} and K~=KM\widetilde K=KM. Their joint product preserves QK⊤QK^{\top}. For STIP’s evaluated feature-permutation configuration, Q~=QΠ\widetilde Q=Q\Pi and K~=KΠ\widetilde K=K\Pi, giving the same joint-product invariance. We compute these fingerprints from transformed layer-0 state pairs and use them for vocabulary-based reconstruction:

ModelKV-Cloak Q/K transformationSTIP feature permutation
Llama-3.2-1B100%100%
Llama-3.1-8B100%100%
Qwen2.5-7B100%100%
Qwen3-8B100%100%
OLMoE-1B-7B-0924100%100%

These experiments add measured evidence for the specified joint fingerprints. Our named-system conclusions refer to these evaluated transformation configurations and our Talaria reimplementation.

We thank Reviewer Esmk for prompting these additional evaluations and a clearer statement of their scope.

2. Conditional privacy, exposure budget, and stronger attack evaluations

We thank Reviewer Esmk for highlighting the distinction between the formal guarantee, concrete estimates, operational assumptions, and empirical evidence.

Formal guarantee and concrete estimates. StateVeil’s privacy theorem is a conditional reduction under the stated exact-weight Dual-LPN assumption. Its analysis accounts for jointly exposed values and repeated observations under the shared public matrix. The approximately 21622^{162}–21682^{168} estimates concern the analyzed classical attacks on a single instance; they do not establish an equivalent security level for the complete service under arbitrary repeated observations.

Exposure budget and rotation. The approximately 2242^{24}-token-position cap is a heuristic operational budget, not a proven security limit. Exposure accounting must include observations across projections sharing the same matrix.

We measured the preparation time for the master matrix and projection-specific recovery tables using one NVIDIA L40S GPU and 16 CPU threads. Preparation takes 48 seconds for Llama-3.2-1B and 424 seconds for Llama-3.1-8B. Under the illustrative budget of approximately 47k requests per matrix, with 256 input and 100 output tokens per request, the amortized preparation overhead is small, approximately 1.0 ms and 9.0 ms per request, respectively. This budget remains a heuristic operating choice under the stated exposure accounting. Preparation costs are reported separately from online inference latency.

Additional learned and optimization-based adversaries. We agree that failure of Euclidean direct matching alone is insufficient empirical coverage. We have added three complementary attack families:

  • TokenInfer [1]: classification-based attack. A learned classifier maps exposed intermediate representations to token identities.
  • TokenInv [2]: optimization-based attack. The attacker optimizes continuous input embeddings and recovers discrete tokens using activation consistency and semantic information.
  • SentenceInv [3]: generation-based attack. A learned autoregressive decoder reconstructs text from exposed representations, targeting sentence-level content and semantics.

We compare the unprotected model, Talaria, and StateVeil using Llama-3.2-1B on LMSYS-Chat data.

We report Token Reconstruction Accuracy (TRA), the proportion of original tokens correctly recovered, and Cosine Semantic Similarity (CSS), the cosine similarity between embeddings of reconstructed and original text. Lower values indicate less successful reconstruction.

AttackOriginal TRAOriginal CSSTalaria TRATalaria CSSStateVeil TRAStateVeil CSS
State Matching100.00%—100.00%—0.00%—
TokenInfer90.10%84.25%2.89%8.76%2.86%7.08%
TokenInv100.00%100.00%0.58%12.19%0.00%7.56%
SentenceInv29.70%72.79%5.78%16.64%5.59%17.42%

Bold indicates the lower value between the two defenses. “—” indicates an unreported metric.

Both defenses substantially reduce reconstruction under the additional attacks. StateVeil achieves slightly lower TRA, while SentenceInv yields lower CSS against Talaria. These small differences do not establish a consistent advantage across all metrics. The clearest distinction is State Matching: it achieves 100% token recovery against the evaluated Talaria configuration and 0% against StateVeil, identifying a leakage mode not captured effectively by the other attacks.

These results extend the defense evaluation beyond a single Euclidean matcher to include learned classification, optimization, and generative reconstruction. They provide broader empirical evidence under the tested settings, while the general privacy guarantee remains conditional on the stated assumptions and exposure model.

[1] Xinjian Luo, Ting Yu, and Xiaokui Xiao. Prompt Inference Attack on Distributed Large Language Model Inference Frameworks. ACM CCS, 2025.

[2] Wenjie Qu et al. Prompt Inversion Attack Against Collaborative Inference of Large Language Models. IEEE Symposium on Security and Privacy, 2025.

[3] Kang Gu et al. Towards Sentence Level Inference Attack Against Pre-trained Language Models. Proceedings on Privacy Enhancing Technologies, 2023(3), pp. 62–78.

We thank Reviewer Esmk again for the suggestions that motivated this broader evaluation and clearer separation of security claims.

3. Clarifying the contributions relative to prior work

We thank Reviewer Esmk for requesting a more precise novelty statement. We acknowledge the established foundations of vocabulary-based reconstruction, transformation invariants, additive masking, and Dual-LPN. Our contributions concern their application and analysis at specified outsourcing boundaries and the construction of StateVeil for protected LLM linear execution.

Prompt-State Matching Attack: analysis of specified TEE-boundary observations.

Thomas et al. demonstrate vocabulary-based reconstruction from hidden states, including transformed hidden states, and Luo et al. investigate reconstruction from exposed KV caches. Our contribution is to characterize what remains identifiable from the particular values exposed by outsourced operations, including relationships between simultaneously observable states.

This analysis considers three cases: direct fingerprints from exposed attention states, invariant fingerprints from token-aligned encodings, and joint fingerprints derived from paired observations. For example, individual Q/K transformations can obscure their coordinates while preserving a joint score fingerprint that remains comparable across independently instantiated transformations.

Determinism explains why the correct candidate reproduces an unprotected state. Unique token recovery additionally requires sufficient discrimination between candidate fingerprints: deterministic functions can still produce collisions. We treat the direct-matching results as a baseline and examine which states and preserved relationships retain this discrimination. The additional BF16 experiments demonstrate that recovery is not uniformly perfect across states and models.

StateVeil: fresh sparse masking with reusable recovery tables.

For a quantized linear operation, StateVeil uses a public matrix AdA_d and a projection-specific table B=AdWqB=A_dW_q. For each protected input row, it samples fresh exact-weight sparse coefficients ss, sends Cq+sAdC_q+sA_d, and subtracts sBsB from the returned product. The sparse coefficients allow both masking and recovery to use sparse combinations of table rows.

Compared with Slalom. Slalom adds a fresh pseudorandom mask to each input and removes its effect using a precomputed mask–weight product. It provides cryptographic privacy and computation integrity, but each recovery record is used only once, requiring continued preprocessing and loading during sustained inference. StateVeil generates fresh masks and corresponding recovery terms from reusable tables. Within the analyzed exposure budget, this avoids replenishing per-invocation recovery records while keeping online masking and recovery sparse. Its privacy guarantee relies on the exact-weight Dual-LPN assumption and accounts for repeated observations under the shared matrix.

Compared with Talaria. Talaria combines a reusable public low-rank basis with fresh private coefficients and uses precomputed basis–weight products for recovery. This low-rank masking leaves some input features unchanged, which our State Matching attack exploits in the evaluated configuration. StateVeil uses a full-column-rank public matrix and fresh sparse finite-field coefficients, removing that particular public-nullspace leakage. Its broader privacy guarantee is supplied by the conditional reduction.

The specific contribution combines a shared master matrix across projection widths, projection-specific recovery tables, fresh sparse mask generation, exact quantized execution, and an analysis of jointly exposed values and repeated observations. Our TDX prototype demonstrates this construction in an inference system, achieving end-to-end latency comparable to our Talaria reimplementation.

We thank Reviewer Esmk again for helping us make these technical distinctions and the scope of the contribution more explicit.

Reviewer_RDb2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Prompt-State Matching Attack, which reconstructs private prompts in TEE-partitioned LLM inference using vocabulary-based matching of exposed states, transformation-invariant fingerprints, and relationships between jointly exposed states. To mitigate this leakage, the paper proposes StateVeil, which uses fresh sparse masks over a finite field to protect outsourced linear computations while preserving the specified quantized computation.

Critical strengths
  1. Prompt-State Matching Attack achieves 100% token recovery across four LLMs at the first, middle, and last layers, showing reliable prompt reconstruction.
  2. StateVeil provides protection against Prompt-State Matching Attack: the evaluated Euclidean direct matcher achieves 0% token recovery across four models, even with true prefixes supplied at each position.
Critical weaknesses
  1. The FP32 evaluation lacks corresponding FP16/BF16 experiments, leaving effectiveness under common lower-precision deployment settings insufficiently established.

  2. The paper reports latency but provides no breakdown of TEE computation, GPU execution, masking, or communication, nor explicitly quantifies the substantial overhead relative to No-TEE.

  3. Section 4.3 of the paper reuses unprotected score-matching results without collecting protected victim traces, leaving joint-state matching against KV-Cloak and STIP without end-to-end experimental validation.

  4. The evaluation covers only Llama and Qwen models, limiting evidence that the attack and defense generalize to other architectures.

  5. The paper’s threat model assumes that TEEs are secure. Given that commercially available trusted GPUs (e.g., H100) can execute models entirely within a trusted environment, the paper does not explain whether the research remains practically relevant.

Questions for authors
  1. Can you repeat attack and defense evaluations under FP16 and BF16 across more architectures and larger models, preferably including MoE, and explain which architectural properties may affect applicability?

  2. Can you break down latency into TEE computation, GPU execution, masking, and communication, and quantify the throughput reduction relative to No-TEE?

  3. Given commercially available trusted GPUs, can you clarify whether this research remains practically relevant?

  4. Can you evaluate joint-state matching on actual KV-Cloak and STIP traces, verify that the required states are observable and aligned, and report token recovery rates?

  5. Can you explain why Qwen3-8B’s SQuAD EM is substantially lower than those of Qwen2.5-7B and Llama-3.2-1B?

4 comments

Q1. Additional evaluations under FP16/BF16 and across architectures, including MoE.

We thank Reviewer RDb2 for the suggestions regarding numerical precision and architectural coverage. We have added FP16/BF16 attack evaluations on Llama-3.2-1B, Llama-3.1-8B, Qwen2.5-7B, Qwen3-8B, and OLMoE-1B-7B-0924, together with FP32 evaluations on OLMoE. Below, we report these attack results, the completed StateVeil defense evaluations, architectural applicability, and hardware limitations.

Experimental setup. Direct matching covers the first, middle, and last layers of each model and all six states, Q,K,V,S,A,OQ,K,V,S,A,O. Invariant matching is evaluated at layer 0 using three fingerprints:

  • Gram: vec(X≤iX≤i⊤)\mathrm{vec}(X_{\le i}X_{\le i}^{\top}).
  • Projection: vec(XiP)\mathrm{vec}(X_iP).
  • Sorting: sort(vec(Xi))\mathrm{sort}(\mathrm{vec}(X_i)).

Gram and Projection are evaluated on Q,K,V,A,OQ,K,V,A,O, with SS not evaluated; Sorting is evaluated on all six states. The tables report token recovery rates.

FP32 and FP16 results. In the additional FP32 experiments on OLMoE, direct matching achieves 100% recovery for all six states at layers 0, 8, and 15. All evaluated invariant fingerprints also achieve 100% recovery at layer 0. Under FP16, every evaluated direct-matching and invariant-matching configuration achieves 100% recovery across all five models. The layer indices for direct matching are the same as those reported below.

BF16 direct-matching results.

ModelLayerQKVSAO
Llama-3.2-1B0, 8, 15100%100%100%100%100%100%
Llama-3.1-8B0, 16, 31100%100%100%100%100%100%
Qwen2.5-7B0100%100%100%100%93.6%100%
Qwen2.5-7B1499.7%97.9%98.0%99.5%88.3%99.2%
Qwen2.5-7B2799.8%99.7%99.8%37.3%7.6%57.6%
Qwen3-8B0100%100%100%100%100%100%
Qwen3-8B18, 35100%100%100%96.7%96.7%100%
OLMoE-1B-7B-09240100%100%100%100%100%100%
OLMoE-1B-7B-09248100%100%100%18.1%96.6%100%
OLMoE-1B-7B-092415100%100%100%59.6%100%100%

Under BF16, direct matching of Q,K,VQ,K,V maintains 97.9%–100% recovery across all evaluated models and layers. However, recovery drops substantially for some S,A,OS,A,O states. For example, the corresponding rates at the last layer of Qwen2.5-7B are 37.3%, 7.6%, and 57.6%. These results show that sensitivity to numerical precision depends on the model, layer, and exposed state. Complete recovery under FP32 therefore does not directly generalize to every BF16 configuration.

BF16 invariant-matching results. The following table lists configurations with recovery below 100%. Every tested state in all remaining evaluated configurations achieves 100% recovery. All invariant experiments use layer 0; “—” denotes a state that was not evaluated.

ModelFingerprintQKVSAO
Llama-3.2-1BGram100%100%100%—100%99.8%
Llama-3.1-8BGram100%100%100%—100%99.8%
Qwen2.5-7BGram100%84.7%100%—90.1%90.6%
Qwen2.5-7BProjection100%100%100%—92.6%100%
Qwen2.5-7BSorting100%100%100%100%77.5%97.0%

These results show that the evaluated invariant fingerprints can retain token-reconstruction information under lower precision, although recovery also depends on the particular model and state.

StateVeil defense results. In the original submission, StateVeil reduces the evaluated Euclidean direct matcher’s token recovery rate to 0% across four models, even when the true prefix is supplied at each position. We additionally evaluate three attack families on Llama-3.2-1B using LMSYS-Chat data:

  • TokenInfer: classification-based attack. A learned classifier maps exposed intermediate representations to token identities.
  • TokenInv: optimization-based attack. The attacker optimizes continuous input embeddings and then recovers discrete tokens using activation consistency and semantic information.
  • SentenceInv: generation-based attack. A learned autoregressive decoder reconstructs text from exposed representations, targeting sentence-level content and semantics.

TRA denotes Token Reconstruction Accuracy, and CSS denotes Cosine Semantic Similarity between embeddings of the reconstructed and original text. Lower values indicate less successful reconstruction.

AttackTypeStateVeil TRAStateVeil CSS
State MatchingState matching0.00%—
TokenInferClassification2.86%7.08%
TokenInvOptimization0.00%7.56%
SentenceInvGeneration5.59%17.42%

These results broaden the empirical evidence for StateVeil against different attack methods. Its privacy guarantee remains conditional on the cryptographic assumptions and exposure conditions stated in the paper.

MoE and architectural applicability. OLMoE uses dropless token-choice routing, selecting 8 of 64 experts per layer for each token. All evaluated configurations achieve 100% recovery under FP32 and FP16. Under BF16, direct matching of Q,K,V,OQ,K,V,O achieves 100% across all evaluated layers, and all evaluated invariant fingerprints achieve 100% at layer 0. These results extend the attack evidence to the tested MoE architecture.

Vocabulary probing requires reproducible computation from the same prefix and candidate token, together with comparable victim and probe observations in terms of position and representation. With reproducible token-wise routing, the attacker can compute each candidate’s expert selection using the known model, so MoE routing does not inherently prevent matching. Architectures whose expert assignments depend on other tokens in the batch, involve unreproducible randomness, or prevent observation alignment require additional handling and are outside the coverage of these experiments. StateVeil’s protection operates at the level of outsourced linear computations.

Model scale and hardware limitations. Our available GPU hardware prevents us from completing evaluations on models with larger parameter counts. The additional experiments expand precision and MoE coverage, while the evaluated model sizes remain within 1B–8B parameters. Effectiveness at larger scales remains to be evaluated.

We thank Reviewer RDb2 again for the suggestions, which helped us broaden the evaluation and clarify the method’s applicability and experimental scope.

Q2. Latency breakdown and overhead relative to No-TEE.

We thank Reviewer RDb2 for requesting a clearer performance breakdown. We profile Llama-3.2-1B with an input length of 64 tokens.

ComponentPrefill (ms)Decode (ms/token)
TEE computation474.5151.6
Masking2,320.5118.0
Transfers/RPC176.050.2
GPU computation223.545.4
Total3,194.5365.2

TEE computation includes framework overhead. Masking includes encoding and recovery, while Transfers/RPC includes data copies, host protocol processing, and scheduling. Decode values assume that the host-side measurements are per-token averages.

The No-TEE baseline runs the model entirely on the GPU. Our current TEE supports CPU execution only, so operations retained inside the TEE—including masking and recovery—run on the CPU, while outsourced linear operations execute on the untrusted GPU. The profile shows that TEE computation and masking dominate latency. This difference in CPU/GPU placement is a major source of overhead relative to No-TEE.

Q3. Practical relevance given commercially available confidential GPUs.

We thank Reviewer RDb2 for raising this deployment question. Confidential GPUs provide an important option for protecting inference, but their performance and deployment constraints still motivate considering partitioned execution.

The performance impact is workload dependent. Chrapek et al. [1] report throughput penalties of 4–8% for their evaluated H100 workloads. For serving workloads involving repeated model swapping, Martínez Ibarra et al. [2] report that non-confidential execution achieves 45–70% higher throughput and 20–30% lower latency, primarily because confidential execution adds encryption and decryption costs during model loading.

Deployment restrictions also affect resource utilization. NVIDIA’s documented confidential-VM configurations [3] require a supported hardware and software stack, assign an entire GPU to one confidential VM, and do not support MIG, MPS, or GPUDirect RDMA in those configurations. These constraints limit GPU sharing and deployment flexibility.

Together, these findings motivate selective outsourcing as a design option even when confidential GPUs are available. Systems may retain selected operations within the trusted environment and outsource others to improve throughput or resource utilization. Such partitioning exposes intermediate states across the trusted boundary, creating a privacy problem that trusted hardware alone does not resolve. Our work analyzes this exposure, and StateVeil protects the inputs and outputs of outsourced linear computations. Its applicability therefore depends on the states exposed outside the trusted boundary, regardless of the hardware used within it.

[1] Marcin Chrapek, Marcin Copik, Etienne Mettaz, and Torsten Hoefler. Confidential LLM Inference: Performance and Cost Across CPU and GPU TEEs. IEEE International Symposium on Workload Characterization (IISWC), 2025.

[2] Antonio Martínez Ibarra et al. Performance of Confidential Computing GPUs. IEEE International Conference on Distributed Computing Systems (ICDCS), 2025.

[3] NVIDIA. Trust and Threat Model. NVIDIA Confidential Computing Documentation, Confidential Virtual Machines, version 1.0.0. Accessed November 15, 2026.

Q4. Joint-state matching against KV-Cloak and STIP.

We thank Reviewer RDb2 for requesting additional joint-state matching evaluations. We test the specified Q/K transformations offline and perform vocabulary-based matching using joint score fingerprints computed from the transformed states.

KV-Cloak. Its operator-fused query/key transformation can be written as Q~=QM−T\widetilde Q=QM^{-T} and K~=KM\widetilde K=KM, where MM is secret and invertible. In an outsourcing configuration where the untrusted attention worker receives both transformed states, their product preserves the attention-score fingerprint:

Q~K~⊤=QK⊤.\widetilde Q\widetilde K^{\top}=QK^{\top}.

The attacker compares this joint fingerprint with candidate-token probes without recovering MM.

STIP. In the evaluated feature-permutation configuration, the cloud computes Q~=QΠ\widetilde Q=Q\Pi and K~=KΠ\widetilde K=K\Pi using the same secret feature permutation. Consequently,

=Q\Pi\Pi^{\top}K^{\top} =QK^{\top}.$$ The joint product therefore provides a probe-comparable score fingerprint without revealing the permutation. We apply these transformations to **layer-0 Q/K states**, compute the joint products from the transformed pairs, and use the resulting fingerprints for token reconstruction. The results across five models are: | Model | KV-Cloak Q/K transformation | STIP feature permutation | |---|---:|---:| | Llama-3.2-1B | 100% | 100% | | Llama-3.1-8B | 100% | 100% | | Qwen2.5-7B | 100% | 100% | | Qwen3-8B | 100% | 100% | | OLMoE-1B-7B-0924 | 100% | 100% | Joint-state matching achieves **100% token recovery across all five models** under both evaluated transformations, demonstrating that the preserved relationships between transformed states can support prompt reconstruction. **Q5. Qwen3-8B’s SQuAD results and the effect of StateVeil.** We thank Reviewer RDb2 for highlighting this difference. Qwen3-8B’s lower SQuAD EM is already present in the **unprotected Original baseline**, where it achieves 23.60%, compared with 78.06% for Qwen2.5-7B and 27.36% for Llama-3.2-1B. This cross-model difference therefore predates StateVeil. The reported measurements do not isolate its underlying cause. Our quality evaluation separates the effect of quantization from that of the protection mechanism. StateVeil exactly recovers the specified quantized Linear outputs. We now report **Original / Quantized / StateVeil**, including the additional OLMoE results: | Model | MMLU Acc. | SQuAD EM | SQuAD F1 | |---|---|---|---| | Llama-3.2-1B | 37.83 / 37.58 / 37.58 | 27.36 / 26.41 / 26.41 | 42.36 / 41.68 / 41.68 | | Llama-3.1-8B | 63.89 / 63.99 / 63.99 | 57.25 / 56.56 / 56.56 | 72.20 / 71.58 / 71.58 | | Qwen2.5-7B | 71.79 / 71.36 / 71.36 | 78.06 / 74.74 / 74.74 | 86.53 / 83.91 / 83.91 | | Qwen3-8B | 72.85 / 72.61 / 72.61 | 23.60 / 20.31 / 20.31 | 51.18 / 48.36 / 48.36 | | OLMoE-1B-7B-0924 | 53.16 / 52.81 / 52.81 | 34.00 / 35.60 / 35.60 | 51.16 / 52.41 / 52.41 | All values are percentages. For Qwen3-8B, the EM decrease from 23.60% to 20.31% is already present in the quantized reference, and StateVeil preserves the same result. Across all five models and all reported metrics, **StateVeil matches the quantized baseline**, showing no additional quality degradation from the protection mechanism in these evaluations.
Reviewer_Qm4TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates privacy leakage in TEE-partitioned LLM inference and proposes both an attack and a defense. The Prompt-State Matching Attack reconstructs private prompts from exposed intermediate states using direct, invariant, and joint-state matching. To mitigate these risks, the authors introduce StateVeil, a Dual-LPN-based sparse masking framework for protecting outsourced linear operations, together with conditional privacy guarantees. Experiments on four LLMs demonstrate the effectiveness of prompt reconstruction under unprotected settings and show that StateVeil resists the evaluated direct matching attack while maintaining comparable inference latency to Talaria.

Critical strengths
  1. The paper addresses an important privacy concern in TEE-partitioned LLM inference. The attack analysis provides useful insights into potential information leakage from exposed intermediate states.

  2. StateVeil combines a concrete masking mechanism, theoretical privacy analysis, and system implementation. The integration of theoretical and experimental results is a valuable aspect of this work.

Critical weaknesses
  1. While the proposed Dual-LPN-based masking is interesting, its specific advantages over existing approaches, such as Slalom and Talaria, are not entirely clear. In particular, StateVeil achieves comparable end-to-end latency to Talaria but has higher first-token latency. A more detailed comparison of security guarantees and computational overhead would help clarify its incremental contributions.

  2. The defense evaluation mainly considers Euclidean direct matching, leaving its effectiveness against stronger adaptive attacks less clear. Additionally, a direct security comparison with Talaria under the same threat model would be valuable. The quantization-induced performance degradation on SQuAD also deserves further discussion. Overall, more evidence would help establish the practical trade-offs among privacy, efficiency, and model quality.

Questions for authors
  1. Could the authors further clarify the main technical advantages of StateVeil over Slalom and Talaria, particularly regarding security guarantees and computational efficiency?
  2. Could the authors provide additional evaluations against stronger attacks, such as invariant or joint-state matching, and ideally compare StateVeil with Talaria under consistent attack settings?
3 comments

We thank Reviewer Qm4T for the careful evaluation and constructive feedback. We address the two questions below.

Q1. Technical advantages over Slalom and Talaria.

Thank you for requesting a clearer comparison. We clarify each approach’s mechanism, limitation, and the corresponding benefit of StateVeil.

Compared with Slalom. Slalom adds a fresh pseudorandom mask to each input and removes its effect using a precomputed mask–weight product. It provides cryptographic privacy and computation integrity, but each recovery record is used only once, requiring continued preprocessing and loading during sustained inference. StateVeil instead constructs fresh masks and corresponding recovery terms through sparse combinations of reusable tables. Within the analyzed exposure budget, this avoids replenishing per-invocation recovery records while keeping online masking and recovery sparse. Its privacy guarantee relies on the exact-weight Dual-LPN assumption and accounts for repeated observations under the shared matrix. StateVeil’s main advantage over Slalom is therefore reusable preprocessing for sustained inference.

Compared with Talaria. Talaria combines a reusable public low-rank basis with fresh private coefficients to generate masks, and uses precomputed basis–weight products for recovery. However, this low-rank masking leaves some input features unchanged. Our State Matching attack uses these features to recover tokens in the evaluated configuration. StateVeil addresses this leakage using a full-column-rank public matrix and fresh sparse finite-field coefficients, with conditional privacy under the exact-weight Dual-LPN assumption. Its analysis accounts for jointly exposed values and repeated observations across projections. As shown in Q2, StateVeil resists the evaluated State Matching attack. Our TDX prototype achieves end-to-end latency comparable to our Talaria reimplementation.

Q2. Additional attacks and direct security comparison.

Thank you for highlighting the limited empirical attack coverage. We agree that failure of Euclidean direct matching alone does not establish robustness against stronger adversaries. We have therefore expanded the evaluation with three complementary attack families:

  • TokenInfer [1]: classification-based attack. A learned classifier maps observed intermediate representations to token identities.
  • TokenInv [2]: optimization-based attack. The attacker optimizes continuous input embeddings and then recovers discrete tokens using activation consistency and semantic information.
  • SentenceInv [3]: generation-based attack. A learned autoregressive decoder reconstructs text from exposed representations, targeting sentence-level content and semantics.

We compare the unprotected model, Talaria, and StateVeil using Llama-3.2-1B on LMSYS-Chat data.

We report Token Reconstruction Accuracy (TRA), the proportion of original tokens correctly recovered, and Cosine Semantic Similarity (CSS), the cosine similarity between embeddings of the reconstructed and original text. Lower values indicate less successful reconstruction.

AttackOriginal TRAOriginal CSSTalaria TRATalaria CSSStateVeil TRAStateVeil CSS
State Matching100.00%—100.00%—0.00%—
TokenInfer90.10%84.25%2.89%8.76%2.86%7.08%
TokenInv100.00%100.00%0.58%12.19%0.00%7.56%
SentenceInv29.70%72.79%5.78%16.64%5.59%17.42%

Bold indicates the lower value between the two defenses. “—” indicates an unreported metric.

Both defenses substantially reduce reconstruction under the additional attacks. StateVeil achieves slightly lower TRA across these attacks, while SentenceInv yields lower CSS against Talaria. These small differences do not establish a consistent advantage across all metrics. The clearest distinction is State Matching: it achieves 100% token recovery against the evaluated Talaria configuration and 0% against StateVeil, revealing a leakage mode that the other attacks do not capture effectively.

These results extend our evaluation beyond a single Euclidean matcher to include learned classification, optimization, and generative reconstruction. They provide broader empirical evidence under the tested settings, while our general privacy guarantee remains conditional on the stated cryptographic assumptions and exposure model.

[1] Xinjian Luo, Ting Yu, and Xiaokui Xiao. Prompt Inference Attack on Distributed Large Language Model Inference Frameworks. ACM CCS, 2025.

[2] Wenjie Qu et al. Prompt Inversion Attack Against Collaborative Inference of Large Language Models. IEEE Symposium on Security and Privacy, 2025, pp. 1695–1712.

[3] Kang Gu, Ehsanul Kabir, Neha Ramsurrun, Soroush Vosoughi, and Shagufta Mehnaz. Towards Sentence Level Inference Attack Against Pre-trained Language Models. Proceedings on Privacy Enhancing Technologies, 2023(3), pp. 62–78.

We thank Reviewer Qm4T again for the helpful suggestions, which have helped us clarify StateVeil’s technical contributions and strengthen the empirical evaluation.

Thank you for your detailed response and clarifications. Given my limited expertise in this specific area, and after considering the comments and assessments of other reviewers, I believe it would be appropriate to maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.