acceptodds
Under review as a conference paper at ICLR 2027

When Secrets Reveal Themselves: Vulnerabilities of Covariant Transformations in Private LLM Inference and a Low-rank Remedy

Abstract

Covariant transformations provide an efficient approach to privacy-preserving large language model (LLM) inference by applying secret transformations to private inputs and intermediate representations for privacy protection, while correspondingly transforming the model weights to preserve the original plaintext computation. However, preserving the plaintext computation through gated feed-forward networks (FFNs) and residual connections imposes common structural constraints on existing covariant transformation schemes. With the proposed CrossWeight, we show that these constraints make it possible to derive an invariant by composing transformed FFN weights, enabling recovery of the secret transformations and then the reconstruction of the private inputs. Across multiple covariant transformation schemes and model architectures, CrossWeight achieves text-token recovery rates approaching 100%, revealing a critical privacy vulnerability. To mitigate this, we further explore an irreversible transformation mechanism based on trainable low-rank transformations, together with an attack-aware objective to guarantee privacy and teacher–student distillation to preserve model utility. Extensive experiments demonstrate that the proposed method substantially reduces the effectiveness of CrossWeight and existing attacks while achieving a favorable privacy–utility trade-off.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM inference, privacy, model obfuscation, prompt reconstruction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uExmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BG: Users may not want a cloud LLM provider to see their private inputs. To protect privacy, existing covariant transformation methods change the way of user inputs and model weights represented, to hide the privacy inputs and intermediate results from the service provider.

RP: However, to keep the model utility, the covariant transformations should follow certain rules. These rules can give attackers a way to recover the private inputs.

Method: Firstly, the paper introduces CROSSWEIGHT, an attack that recovers private inputs. It cancels out the transformations on user input by multiplying weights from different parts of the model. Then the attacker can use the publicly available original model to work backward the user inputs. Secondly, the paper proposes a defense using low-rank transformations, which discard some information so that the cancellation exploited by CROSSWEIGHT no longer works.

Critical strengths
  1. The paper exposes the structural weakness in covariant transformations. It identifies an exploitable invariant arising from the model’s structural constraints.
  2. The CROSSWEIGHT is solid. Table 2 reports 100% text-token recovery across all three protection schemes on Qwen3-1.7B, Llama3-8B, and the MoE model Qwen3-30B-A3B.
Critical weaknesses

The weakness is mainly on the proposed defense. It is designed to defend against only one specific attacker (the proposed CrossWeight). However, making CrossWeight ineffective doesn’t mean that the private inputs cannot be recovered. In particular, the low-rank transformations are initialized deterministically from the identity and then trained only lightly, so the deployed weights likely remain close to the public plaintext weights, and an attacker who simply compares the two directly (without the cancellation step CrossWeight relies on) may still recover the token permutation. The paper does not evaluate any such adaptive attacker, so the claim of privacy guarantee may be not supported.

Questions for authors

In Table 2, the Llama3-8B / EIA row and ConjFormer Column reports 0.00 TTRSR but 100.00 BLEU-4. Is this a typo? Why BLEU-4 got such different result only on this setting?

2 comments
AuthorsOpenReview ↗

Response to Reviewer uExm (Part 1/2)

We sincerely thank the reviewer for the valuable and thoughtful comments, and for recognizing the structural vulnerabilities in covariant transformations revealed by our work, the soundness of our CROSSWEIGHT attack, and the 100% text-token recovery demonstrated across three protection schemes and multiple dense and MoE model architectures.

Please find the point-by-point response below.

W1 (Part I): Vulnerability of identity initialization.

Response: We thank the reviewer for this insightful observation. We agree that when choosing the identity basis (Pˇ(0)=B=[Ir;0]\check{P}^{(0)} = B = [I_r; 0]) for initialization, light fine-tuning may leave individual factors close to BB, potentially allowing an adaptive attacker to directly compare the deployed weights (e.g., W~e\tilde{W}_e) against the public plaintext weights (WeW_e) without relying on CROSSWEIGHT's cancellation step.

We clarify that this is specific to the identity-aligned initilazation, which is not necessary for our proposed low-rank framework.. Importantly, this issue can be addressed through random orthogonal initialization or an equivalent basis reparameterization of the trained checkpoint, without modifying the underlying defense mechanism.

1. Random Orthogonal Initialization:

In Appendix E, B=[Ir;0]B=[I_r;0] was adopted as a simple and convenient initialization for the low-rank initialization, rather than as an essential component of our defense. We may instead adopt a secret random orthogonal initialization with R∈Rr×rR\in\mathbb{R}^{r\times r}, setting Pˇ(0)=BR\check{P}^{(0)}=BR and Qˇ(0)=R⊤B⊤\check{Q}^{(0)}=R^\top B^\top, where RR⊤=IrRR^\top=I_r. Since Pˇ(0)Qˇ(0)=BRR⊤B⊤=BB⊤\check{P}^{(0)}\check{Q}^{(0)}=BRR^\top B^\top=BB^\top, this modification preserves exactly the same initial low-rank projection while randomly rotating the low-rank coordinates, thereby preventing the direct coordinate alignment exploited by the adaptive attacker. Importantly, this change does not alter the underlying defense mechanism. Retraining Qwen3-0.6/1.7B with this random orthogonal initialization yields the following results:

ModelInitializationPPL ↓MMLU ↑PIQA ↑C-Eval ↑CROSSWEIGHT TTRSR (%) ↓Direct Comparison (Π Recovery %) ↓
Qwen3-1.7BIdentity10.8554.8473.1256.3922.2498.56
Qwen3-1.7BRandom Orthogonal10.7754.7572.8056.4622.610.00
Qwen3-0.6BIdentity20.5441.8566.0539.3824.3999.76
Qwen3-0.6BRandom Orthogonal20.8141.6366.6538.8624.940.00

As shown above, random orthogonal initialization can reduce direct comparison recovery to 0.00% while preserving comparable utility and CROSSWEIGHT resistance. This confirms that the observed vulnerability is specific to the identity-aligned initialization rather than inherent to our low-rank framework.

2. Inherent Basis Equivalence of Existing Results:

Furthermore, the low-rank transformations (Pˇ,Qˇ)(\check{P},\check{Q}) naturally admit an orthogonal change of basis: for any trained pair (Pˇ,Qˇ)(\check{P},\check{Q}) and any secret orthogonal matrix R∈Rr×rR\in\mathbb{R}^{r\times r}, the rotated pair (PˇR,R⊤Qˇ)(\check{P}R,R^\top\check{Q}) preserves exactly the same composition, (PˇR)(R⊤Qˇ)=PˇQˇ(\check{P}R)(R^\top\check{Q})=\check{P}\check{Q}. As a complementary verification, this reparameterization can be applied directly to already-trained adapters without retraining, removing the direct coordinate alignment between the deployed embedding weights W~e=ΠWe(PˇeR)\tilde{W}_e=\Pi W_e(\check{P}_eR) and the truncated plaintext weights. The results obtained by directly reparameterizing the already-trained Qwen3-1.7B adapters without retraining are shown below, with similar findings on Qwen3-0.6B and Llama3-8B.

ConfigurationPPL ↓MMLU ↑PIQA ↑C-Eval ↑CROSSWEIGHT TTRSR (%) ↓Direct Comparison (Π Recovery %) ↓
Identity10.8554.8473.1256.3922.2498.56
Identity Init. + Basis Rotation10.8554.8473.1256.3922.240.00

Post-training rotation reduces direct comparison recovery from 98.56% to 0.00% while leaving all reported utility metrics and recovery results against CROSSWEIGHT, VMA, IA, and IMA unchanged. Together with random orthogonal initialization, this demonstrates that the observed coordinate-alignment leakage can be mitigated without modifying the proposed low-rank defense or its training objectives.

We have added the random orthogonal initialization results to revised Tables 3 and 8, and extended Appendix G.3 with evaluations of the reviewer's suggested direct weight-comparison attack and the extended basis reparameterization.

AuthorsOpenReview ↗

Response to Reviewer uExm (Part 2/2)

W1 (Part II): Scope of Privacy Guarantees

We agree with the reviewer that our defense is designed to counter a specific attack (i.e., the proposed CrossWeight), and that robustness against a particular attack does not imply a universal cryptographic guarantee against all possible adaptive adversaries. Our primary contribution lies in identifying a fundamental structural vulnerability shared by existing lossless covariant transformation schemes through CROSSWEIGHT, and proposing a targeted low-rank remedy that disrupts the PQ=IdPQ=I_d cancellation mechanism, achieving a favorable empirical privacy–utility trade-off against CROSSWEIGHT and commonly studied privacy attacks. In the revised manuscript, we have tempered the broad privacy claims in the Abstract and Section 4 to more accurately position our defense as an empirically validated mitigation rather than a universal privacy guarantee. We have also explicitly acknowledged this limitation in the added Limitation section and identified broader theoretical security analyses, including robustness against potentially stronger adaptive attacks, as an important direction for future work.

Q1: Typo in the Llama3-8B / EIA result in Table 2.

Response: Thank you for catching this typo. The BLEU-4 value was a transcription error and has been corrected in Table 2.

Reviewer_6NZhOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This is a solid piece of research work. The article proposes clear and effective attack methods centered on co-variable transformation as a privacy protection model, and combines theoretical analysis with real large model experiments to verify their effectiveness, further exploring corresponding defense strategies. The overall structure is clear, with natural transitions between derivation and method introductions, making it easy to understand.

Critical strengths
  1. Clear problem, clear attack strategy. The article reveals the structural constraints introduced by covariant transformations to maintain model computation, and uses the relationship between different weights to recover secret transformations, forming a complete input recovery flow.

  2. Theoretical analysis provides a basis for the attack. The authors derive the necessary form of FFN intermediate transformation under specific conditions, as well as the invariants generated across weight combinations, giving the attack mechanism a clear mathematical explanation.

  3. Experimental validation is relatively thorough. The attack was validated on three protection schemes—STIP, CONJFORMER, and AloePri—as well as multiple dense and MoE models, demonstrating strong resilience.

  4. Research forms a comprehensive discussion of offense versus defense. The authors further propose a defense scheme combining low-rank transformation, anti-match training, and distillation, exploring a compromise between privacy and model capability.

Critical weaknesses

Safety conclusions for low-rank defense should still be cautiously limited. Proving PQ≠IPQ\neq I can show that the exact relationship of the original attack dependencies has been broken, but not enough to prove that the input cannot be recovered by other means. In particular, low-rank mappings are irreversible in continuous space, but this does not mean tokens in the finite list are indistinguishable; There is still a certain recovery success rate in the experiment. Therefore, current evidence is better suited to support "reducing the effectiveness of the attack under assessment" rather than general privacy guarantees. It is recommended that the author clarify this boundary and further analyze whether the residual leak involves critical sensitive information.

Questions for authors

What is the scope of CROSSWEIGHT applied to other protective methods such as SnD? In particular, when the protection scheme implements representation projection through neural networks and uses learning-based reconstruction to recover information, rather than satisfying the compatibility linear transformation relationship described herein, is the attack still effective? If it cannot be directly applied, can you explain its key obstacles and the conditions required to scale the attack?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6NZh

We sincerely thank the reviewer for the valuable and thoughtful comments, and for recognizing the clear problem formulation and attack strategy, the theoretical analysis underpinning our attack, the comprehensive experimental validation across multiple protection schemes and model architectures, and the exploration of defense strategies that balance privacy protection and model capability.

Please find the point-by-point response below.

W1: Scope of privacy guarantees and residual sensitive information leakage.

Response: We agree that our proposed defense reducing the effectiveness of the attack under assessment rather than universal privacy guarantees against all possible attacks. Specifically, the primary contribution of this work is uncovering a fundamental structural vulnerability shared by existing covariant-transformation-based LLM inference frameworks, formulating the mathematically grounded CROSSWEIGHT attack, and introducing our first attempt in remedy. While our defense effectively disrupts the exact structural relationships exploited by CROSSWEIGHT, which confirms that the vulnerability of covariant transformation comes from the requirement of PQ=I and sheds lights on possible directions for defense, it does not establish formal privacy guarantees or rule out all possible attacks. We clarify this limitation in the added Limitation section and leave stronger theoretical privacy guarantees for future work.

To assess residual sensitive information leakage, we additionally report the PII Recovery Success Ratio (PIIRSR) of the proposed defense method on PUPA, which measures the percentage of personally identifiable information (e.g., names, addresses, and phone numbers) recovered by the best-performing attack CROSSWEIGHT. As shown below, our defense substantially reduces both TTRSR and PIIRSR compared with AloePri across all evaluated models, although some residual leakage remains.

ModelMethodTTRSR (%) ↓PIIRSR (%) ↓
Qwen3-0.6BAloePri100.0095.58
Ours24.395.06
Qwen3-1.7BAloePri100.0095.58
Ours22.246.05
Llama3-8BAloePri100.0095.58
Ours3.510.43

Q1: Applicability of CROSSWEIGHT beyond covariant transformation-based schemes (e.g., SnD [1]).

Response: CROSSWEIGHT specifically exploits the compatibility relations and gated-FFN constraints of covariant transformation-based schemes, which yield the cross-weight invariant in Lemma 1 and enable recovery of secret transformations. In contrast, SnD protects transmitted embeddings through randomized noise and user-side denoising, without the corresponding model-weight transformations exploited by CROSSWEIGHT. Similarly, learning-based schemes that apply neural-network projections to transmitted representations do not protect subsequent hidden states through layer-wise transformations or satisfy these compatibility relations. Thus, CROSSWEIGHT cannot be directly applied to such schemes.

The key distinction lies in layer-wise protection. Protecting only transmitted representations may leave subsequent server-visible hidden states as potential targets for reconstruction attacks, motivating covariant transformation-based schemes to protect intermediate representations throughout inference, as discussed in the Introduction. However, maintaining such layer-wise protection while preserving the original computation imposes the structural compatibility constraints that CROSSWEIGHT exploits. Extending CROSSWEIGHT to learning-based schemes would therefore require identifying alternative exploitable relationships or developing different reconstruction techniques.

[1] P. Mai, R. Yan, Z. Huang, Y. Yang, and Y. Pang, “Split-and-denoise: Protect large language model inference with local differential privacy,” ICML, 2024.

Reviewer_xaZsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a reconstruction attack on covariant transformation schemes for private LLM inference. It shows that composing transformed FFN down and up projections cancels compatible hidden transformations. An element-wise product with the transpose then eliminates diagonal scaling, leaving a permutation that can be recovered through sorted row signatures. The recovered permutation enables recovery of the input transformation or vocabulary mapping.

Under an exact-reference-checkpoint threat model, the attack achieves complete prompt recovery against STIP, ConjFormer, and AloePri on the evaluated models. The paper also proposes learned low-rank transformations, an attack-aware matching objective, and teacher–student distillation. Experiments on three dense models show reduced reconstruction success and better utility than the selected privacy baselines.

Critical strengths
  1. The paper proposes a structural attack with a convincing mathematical mechanism: The combination of cross-weight cancellation and the scale-canceling invariant converts transformation recovery into a tractable matching problem. This explains the vulnerability directly and gives the contribution value beyond a particular implementation.

  2. The attack produces consequential end-to-end leakage. The experiments recover complete prompts across three schemes and several model architectures, including a 30B-parameter MoE model.

Critical weaknesses
  1. The defense lacks an attack tailored to its new structure. Introducing (K=PˇdQˇu≠I)(K=\check P_d\check Q_u\neq I) breaks the cancellation used by CROSSWEIGHT, but leaves a new estimation problem. The paper does not evaluate an adaptive attack that explicitly accounts for the unknown (K), for example through joint estimation or alternative weight compositions. The tested rank retains about 98.4% of the hidden dimensions, and rank deficiency alone does not establish uncertainty about token identity. The fixed vocabulary permutation also preserves frequency and co-occurrence information, an attack surface left unexamined in this paper.

  2. The claimed privacy–utility advantage is not established across comparable operating points. The DP baselines are each reported at a single parameter setting with severely degraded utility, while the proposed method is evaluated at several ranks. These results support superiority over the selected configurations, but do not establish a better privacy–utility frontier. The ablations also do not separate the benefits of rank reduction, noise removal, distillation, and attack-aware training.

Questions for authors
  1. Clarification of the defense evaluation: Are the reported CROSSWEIGHT results obtained using its original matching procedure? Have you separated Stage-I permutation-recovery failure from Stage-II vocabulary-matching failure by supplying the correct (Z), as used in the training objective in Eq. (11)? This would clarify what the defense is actually preventing.

  2. Targeted validation of the remedy: Do you have results for (r=dr=d) with the same training objectives, and for the selected low-rank setting with (λC=0\lambda_C=0)? These controls would materially affect my assessment of whether rank deficiency and the attack-aware objective each contribute to the claimed improvement.

4 comments
AuthorsOpenReview ↗

Response to Reviewer xaZs (Part 1/2)

We thank the reviewer for the constructive feedback and for recognizing the principled mathematical foundation of our structural attack and its practical effectiveness in recovering complete prompts across multiple protection schemes and model architectures.

Please find the point-by-point response below.

W1: Adaptive Joint-Estimation Attack, Low-Rank Trade-off, and Frequency-Based Attacks

Response: 1. Adaptive Attack via Joint Estimation: To explicitly account for the unknown low-rank matrices, we evaluate a tailored adaptive attack under the known-reference-model setting. Using the transformed weight compositions C:=W~eW~g=ΠWeK1WgZC := \widetilde{W}_e\widetilde{W}_g=\Pi W_e K_1 W_g Z and B:=W~dW~u=Z−1H−1WdK2WuHZB:=\widetilde{W}_d\widetilde{W}_u=Z^{-1}H^{-1}W_d K_2 W_u HZ (where K1=PˇeQˇgK_1=\check{P}_e\check{Q}_g and K2=PˇdQˇuK_2=\check{P}_d\check{Q}_u), we formulate the recovery of Π\Pi as a joint estimation problem: min⁡Π,Z,K1,K2,H ∥C−ΠWeK1WgZ∥F2+λ∥B−Z−1H−1WdK2WuHZ∥F2s.t. Π∈Pn, Z∈Pm, rank⁡(K1),rank⁡(K2)≤r, H∈Dm.\min_{\Pi,Z,K_1,K_2,H}\ \|C-\Pi W_eK_1W_gZ\|_F^2+\lambda\|B-Z^{-1}H^{-1}W_dK_2W_uHZ\|_F^2 \quad \text{s.t.}\ \Pi\in\mathcal{P}_n,\ Z\in\mathcal{P}_m,\ \operatorname{rank}(K_1),\operatorname{rank}(K_2)\leq r,\ H\in\mathcal{D}_m. We enforce rank constraints through low-rank factorization, relax permutations using annealed Sinkhorn operators, and jointly optimize the variables with Adam. Due to dense Sinkhorn's (n2)\mathcal(n^2) memory overhead for vocabulary size nn, we evaluate all layers on 4,096-token subsets, favoring the attacker by restricting the search space to a known candidate pool. Even under this advantageous prior, the joint-estimation attack recovers at most 1 token (0.02%), failing to improve upon CROSSWEIGHT on the proposed defense due to the challenging non-convex optimization problem introduced by the unknown low-rank factors. Full formulations and experimental details are provided in the revised Appendix G.2. However, we would like to further clarify that this is our first attempt regarding adaptive attacks due to constrained time. We appreciate the reviewer for bringing this out and find it an interesting direction that we will further explore in our future work.

ModelCROSSWEIGHT (Full vocab., %)CROSSWEIGHT (4,096 tokens, %)Joint Estimation (4,096 tokens, %)
Qwen3-0.6B15.0316.46 (674)0.00 (0)
Qwen3-1.7B11.0612.13 (497)0.02 (1)
Llama3-8B0.800.20 (8)0.00 (0)

2. Low-Rank Trade-off and Security Boundary: The rank r is adjustable, allowing an empirical defense–utility trade-off, as demonstrated in Table 6. Notably, our defense remains effective against CROSSWEIGHT even when 98.4% of the hidden dimensions are retained. Further reducing the rank improves empirical defense at the cost of model utility. Importantly, our defense combines low-rank transformations with the attack-aware contrastive objective, rather than relying on rank deficiency alone. We have tempered broad privacy claims in the Abstract and Section 4 and added a dedicated Limitations section, clarifying that our defense provides empirically validated mitigation against CROSSWEIGHT rather than a universal privacy guarantee. The primary contribution of this work remains uncovering a fundamental structural vulnerability shared by existing covariant-transformation-based LLM inference frameworks, formulating the mathematically grounded CROSSWEIGHT attack, and introducing our first attempt to remedy this vulnerability. Investigating stronger adaptive attacks and establishing formal privacy guarantees remain future work.

3. Token frequency attacks: Following AloePri, we consider Token Frequency Matching Attack (TFMA) and Substitution Deciphering Attack (SDA) [1]. TFMA exploits token frequencies, while SDA leverages contextual dependencies through a GPT-2-based language model.

We evaluate two attacker knowledge settings: Domain-aware, using MedDialog to recover prompts from Huatuo26M-Lite (both medical-domain datasets), and Distribution-aware, using a separate subset of Huatuo26M-Lite to recover target prompts from the same dataset. Experiments on Qwen3-0.6B/1.7B (sharing the same tokenizer) and Llama3-8B show limited recovery performance for both attacks, supporting the empirical robustness of our defense against frequency-based attacks. Further details are provided in revised Appendix G.1.

Target ModelPublic DatasetTFMA Top-1 (%)TFMA Top-10 (%)SDA BLEU-4
Qwen3-0.6B/1.7BMedDialog0.050.350.94
Qwen3-0.6B/1.7BHuatuo0.472.834.02
Llama-3-8BMedDialog0.171.081.07
Llama-3-8BHuatuo1.548.983.71

[1] N. Kambhatla, et al., “Decipherment as regression: Solving historical substitution ciphers by learning symbol recurrence relations,” Findings of EACL, 2023.

AuthorsOpenReview ↗

Response to Reviewer xaZs (Part 2/2)

W2: Insufficient privacy–utility comparison with DP baselines and lack of ablations isolating the contributions of individual defense components.

Response: We have expanded the evaluation of all DP baselines across a broader range of privacy budgets for Qwen3-0.6B, Qwen3-1.7B, and Llama3-8B. We evaluate the baselines at matched utility (PPL) and privacy (TTRSR) by binary-searching their privacy budgets (lower values are better for both). The table below reports the lowest measured TTRSR at matched PPL and the lowest measured PPL at matched TTRSR across SANTEXT, RANTEXT, and DP-Forward.

ModelProposed PPLBest DP PPL at matched TTRSRProposed TTRSR (%)Best DP TTRSR (%) at matched PPL
Qwen3-0.6B20.5424,408.1524.3999.20
Qwen3-1.7B10.8542,422.9422.2495.88
Llama3-8B8.8427,905.213.5194.82

As shown, at matched utility, the DP baselines suffer nearly 100% privacy leakage. Conversely, at matched privacy, their utility degrades catastrophically. This demonstrates a superior empirical trade-off for our proposed defense. We have added the detailed trade-off curves as Figure 5 in the revised Appendix. Furthermore, we have carefully revised our claims in Section 4 and added a dedicated Limitations section to explicitly state that we demonstrate a stronger empirical trade-off across the measured points, rather than claiming dominance over a continuous theoretical frontier or equivalence to formal DP guarantees.

Regarding the ablation study and the separation of benefits derived from rank reduction and attack-aware training, please refer to our detailed responses to Q1 and Q2 below, which analyze the individual contributions of these components.

Q1: Clarification of the defense evaluation and separation of Stage-I permutation-recovery failure from Stage-II vocabulary-matching failure by supplying the correct ZZ

Response: The reported CROSSWEIGHT results against our defense use the original two-stage attack procedure, where Stage II relies on the permutation ZZ recovered in Stage I. To isolate the effect of Stage-II vocabulary matching, we additionally evaluate CROSSWEIGHT by supplying the ground-truth ZZ on Qwen3-0.6B/1.7B and Llama3-8B.

As shown below, providing the correct ZZ substantially increases the recovery rate of the input permutation Π\Pi, but does not achieve complete recovery. This demonstrates that both imperfect Stage-I permutation recovery and Stage-II vocabulary-matching errors contribute to the empirical defense against CROSSWEIGHT.

Stage-II InputQwen3-0.6BQwen3-1.7BLlama3-8B
Ground-truth ZZ64.25%46.32%31.38%
Recovered ZZ (Stage I)15.03%11.06%0.80%

We have revised Section 5.4 to include above ablation results.

Q2: Targeted validation of rank deficiency and the attack-aware objective through ablations with r=dr=d and λC=0\lambda_C=0.

Response: We conduct ablation experiments on Qwen3-0.6B to assess the contributions of rank reduction and the attack-aware loss. We report the recovery rate of the secret input permutation matrix Π\Pi by CROSSWEIGHT and PPL on WikiText-2. Here, r=1024r=1024 corresponds to r=dr=d (full rank), and w/o LCL_C denotes λC=0\lambda_C=0.

Without either component, CROSSWEIGHT achieves a 98.72% recovery rate. Adding the attack-aware loss alone reduces this to 37.17%, while rank reduction alone (r=1008r=1008) reduces it to 85.80%. Combining both further lowers the rate to 15.03%, demonstrating their complementary contributions to the empirical defense against CROSSWEIGHT. Moreover, as shown in Table 6 of both the original and revised manuscripts, further reducing the rank enhances defense effectiveness at the expense of increased PPL, highlighting the empirical trade-off between privacy and utility.

Metricr=1024r=1024, w/o LCL_Cr=1024r=1024, with LCL_Cr=1008r=1008, w/o LCL_Cr=1008r=1008, with LCL_C
CROSSWEIGHT recovery (%) ↓98.7237.1785.8015.03
PPL ↓19.5020.3620.1220.54

We have revised Section 5.4 to include above ablation results.

We would like to thank the reviewer again for the constructive comments. Please let us know if there are any further questions.

Thank the authors for their response. My questions are addressed. I maintain my positive view of this paper.

I keep my current positive rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank the Area Chair and all reviewers for their valuable time, constructive feedback, and positive assessments of our work. In addition to our detailed point-by-point responses, we would like to summarize the key contributions acknowledged by reviewers, the major concerns raised, and the improvements made in the revised manuscript.

Importantly, the primary contribution of our work is to uncover a fundamental structural vulnerability shared by existing covariant-transformation-based LLM inference frameworks, develop the mathematically grounded CROSSWEIGHT attack, and take an initial step toward addressing this vulnerability through a proposed defense..


Key Contributions Acknowledged by Reviewers

  • Novel Structural Attack: CROSSWEIGHT reveals a fundamental vulnerability in covariant transformation-based privacy-preserving LLM inference. [xaZs, 6NZh, uExm]
  • Solid Theoretical Foundation: The attack is supported by mathematical analysis of structural constraints and cross-weight invariants. [xaZs, 6NZh]
  • Strong Experimental Validation: The attack achieves complete prompt recovery across multiple protection schemes and dense/MoE architectures. [xaZs, 6NZh, uExm]
  • Defense Investigation: The work further explores a low-rank defense as an initial attempt to mitigate the identified vulnerability. [6NZh]

Major Concerns Raised by Reviewers

The reviewers' concerns primarily focus on the proposed defense rather than the CROSSWEIGHT attack itself:

  • Security Boundaries: The scope of the proposed defense and its privacy guarantees requires further clarification. [xaZs, 6NZh, uExm]
  • Adaptive Attacks: The proposed defense requires further evaluation against adaptive reconstruction, direct weight-comparison, and frequency-based attacks. [xaZs, uExm]

Major Updates in the Revised Manuscript

Based on the reviewers' constructive feedback, we have strengthened the revised manuscript through additional experiments and clarifications, with the following key improvements:

  • Clarified Security Boundaries of the Defense [xaZs, 6NZh, uExm]: In response to concerns about general privacy guarantees, we added a dedicated Limitations section, explicitly positioning our defense as an empirically evaluated mitigation against CROSSWEIGHT and other evaluated attacks, rather than a universal privacy guarantee.

  • Expanded Adaptive Attack Evaluation [xaZs, uExm]: In response to xaZs, we implemented an adaptive joint-estimation attack and evaluated token frequency attacks (Appendix G.1–G.2). Addressing uExm's concern, we showed that random orthogonal initialization mitigates direct weight-comparison attacks without changing our low-rank defense mechanism or compromising its effectiveness (Tables 3, 7, and 8; Appendix G.3).

  • Strengthened Privacy–Utility Comparisons and Ablations [xaZs]: We expanded comparisons with DP baselines across a broader range of privacy budgets (Figure 5). We further conducted targeted ablations to isolate the individual contributions of rank reduction and attack-aware training, and analyzed the respective effects of Stage-I permutation recovery and Stage-II vocabulary matching on the defense's effectiveness (Section 5.4).


We sincerely appreciate the reviewers' constructive engagement and are grateful that Reviewer xaZs has explicitly confirmed that their concerns have been addressed. We believe these additional experiments, analyses, and revisions have substantially addressed the reviewers' concerns and strengthened the manuscript. We sincerely thank the Area Chair and all reviewers again for their valuable time, constructive feedback, and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.