acceptodds
Under review as a conference paper at ICLR 2027

Orthogonal Cipher Token Encryption: Equivariant Utility and Invariant Leakage

Abstract

We study orthogonal cipher token encryption, in which a client rotates latent token vectors with a secret orthogonal transform while an equivariant server computes directly on the rotated representations. Because the server is constructed from rotation-equivariant operations, the same server can process arbitrary client keys without key-specific retraining, and the client can decrypt the returned features before ordinary task decoding with little utility loss in the evaluated settings. We show, however, that direct no-key decoder failure is not sufficient evidence of representation privacy. Under closed-reference auxiliary knowledge, orthogonal invariants such as norms, pairwise similarities, and Gram matrices remain observable and can enable strong token-, instance-, and class-level leakage. Client pre-normalization suppresses norm fingerprints but leaves Gram structure intact. In a controlled Food-101 experiment, salted/input-protected rotation substantially reduces Gram retrieval while retaining authorized utility, whereas noise-only perturbation reduces leakage largely by destroying utility. These results position orthogonal ciphering as a key-agnostic equivariant utility interface rather than a privacy mechanism by itself; additional mechanisms are required to suppress invariant leakage.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
orthogonal ciphering, equivariant neural networks, privacy-preserving inference, representation leakage, invariant attacks, Gram matrices, embedding inversion, encrypted inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_M9F6OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors present a work for encrypted inference where latent token representations are encrypted by the client and processed by the server, whose outputs are then processed and passed through a prediction head to recover intended output. The authors also show that this is not sufficient to preserve privacy as gram- structure is still preserved in the latent encrypted space.

Critical strengths

The paper presents an approach that may be taken by others to do "encrypted inference" and show that it doesn't really work and there is more research to be done in this area.

Critical weaknesses

Are there other approaches that are also exposed the same gram-structure weakness? Would be good to have a discussion on them.

Questions for authors

Are there other approaches that are also exposed the same gram-structure weakness? Would be good to have a discussion on them.

2 comments
AuthorsOpenReview ↗

Author Response: CW1 / Q1

This is an important distinction. Any interface that exposes a set of vectors transformed by the same shared orthogonal action satisfies (XR)(XR)⊤=XX⊤,(XR)(XR)^\top=XX^\top, so the within-set Gram matrix is mathematically available without the key. Privacy from Symmetry/ConjFormer therefore lies in the same mathematical vulnerability class when the relevant transformed vectors are exposed under one shared action.

However, the algebraic implication is not by itself evidence that every complete system is empirically vulnerable. We now distinguish three levels of claim: the mathematical invariant, representation-level evidence under a specified interface, and a demonstrated full-system attack. Whether SentinelLMs, Split-and-Denoise, AloePri, or another system exposes the required vectors depends on its mechanism, interface, noise, aggregation, and threat model.

Our ConjFormer-style, Sentinel-style, and Split-and-Denoise-style rows apply representation transformations under a common retrieval protocol. They are not full-system reproductions and are not presented as claims that those complete systems are broken. Only EqCipher is evaluated as a full system in this work. We revised the related-work and discussion sections to make this conditional scope explicit.

Addressed locations:

Main: Related Work, “Scope of comparison”; Discussion, “Broader applicability of Gram leakage.”

Supplement: Section C, “Scope of Comparisons to Related Methods.”

Thank you again for your constructive review. We have revised the manuscript to directly address your question on whether the Gram-structure weakness extends beyond our specific EqCipher implementation.

The revised Related Work, Discussion, and Supplementary Section C now distinguish between: (i) the mathematical implication for representations exposed under a shared orthogonal action, (ii) representation-level evidence, and (iii) full-system empirical vulnerability. We also clarify that our comparator rows should not be interpreted as claims that the complete cited systems are broken.

We hope this addresses your main concern. If you feel the revision resolves the issue you raised, we would be grateful if you could reconsider the overall assessment accordingly.

Reviewer_vHNMOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses orthogonal cipher token encryption. The client rotates latent token vectors using a secret orthogonal matrix RR, and the server processes these rotated tokens using only equivariant operations. Permissible operations include slot mixing, norm-dependent gating, norm-based normalization, and attention mechanisms based on invariant inner products. The client then decrypts the output by applying the transpose of RR before passing it to a task head. The evaluation covers both utility and information leakage.

Several findings are reported:

  1. Rotation defeats coordinate-space retrieval but not norm-based or Gram-based attacks.
  2. Client-side pre-normalization removes norm leakage but not Gram leakage.
  3. Appending random "salt" coordinates before rotation reduces Gram retrieval while retaining accuracy in some cases.

The authors conclude that orthogonal encryption should be viewed not as a privacy mechanism, but rather as an equivariant utility interface.

Critical strengths
  1. The paper repeatedly states that its leakage results are closed-reference and that none of the mechanisms provides a formal privacy guarantee, which honestly limits the scope of its claims.
  2. Because the server is built only from equivariant operations, the same server works for every client key without retraining. This is a clean design.
Critical weaknesses
  1. The novelty is not clear to me. There is not a single citation in the introduction, which makes it very hard to understand where this work stands relative to prior work.

  2. The core results appear to follow directly from known facts. That norms and Gram matrices are invariant under rotation is already stated in Eq. (4, 21). If the reference set contains the correct answer, it is expected that retrieval using these invariants succeeds. EDNN has already shown the same lesson. I would like the authors to clarify what is new compared to these works.

  3. While the authors develop their arguments with equations, many symbols are never defined, which makes the paper very difficult to follow. For example Eq. (12, 15) use queries, keys, and values qi,kj,vjq_i, k_j, v_j but never say how they are computed. This is even more confusing because the server avoids dense projections over the encrypted axis, which is the usual way to compute q,k,vq, k, v. Additionally there are many undefined notations, to name a few, LL, jℓj_\ell, dsingled_{single} dseqd_{seq}, hωh_\omega zz, etc.

  4. The threat model is also unclear. It is not stated whether the attacker is the server itself, or whether the key RR is reused across multiple queries. As a result, it is difficult to properly evaluate the experimental results.

  5. The experiments do not sufficiently support the privacy conclusions. Chance levels are not reported, and many values appear to be at chance level. For example, the Food-101 value of 0.040 in Table 3 equals 10/25,250 (the test-set size). Yet Table 2 highlights small differences among such values in bold. When evaluating the defenses (noise, salting), there are no attacks adapted to those defenses (adaptive attacks). All defenses are evaluated only with the same distance-based retrieval.

Questions for authors
  1. What is the key finding of this paper that cannot be derived directly from the invariance, or that was not already shown by EDNN?

  2. How is attention, in particular q,k,vq, k, v, actually computed in the evaluated models, given that the server cannot use dense projections over the encrypted axis?

  3. Could the authors provide complete definitions of the notation and the model architectures, so that the method and experiments can be reproduced?

  4. What exactly is the threat model? Who is the attacker, and how often is the key RR refreshed? Would the conclusions still hold if RR were reused across queries?

  5. How do the leakage results compare to chance level, and how much do they vary across keys and seeds?

  6. Do the defenses (noise, salting) remain effective against attacks designed specifically for them?

7 comments
AuthorsOpenReview ↗

Author Response: CW1 + Q1 + CW2

Thank you for highlighting this issue. We agree that orthogonal preservation of norms and inner products is classical, and the original submission did not position the work adequately. We do not claim orthogonally equivariant inference itself as new. The revision now explicitly discusses Privacy from Symmetry/ConjFormer as prior work developing this interface for language-model inference.

EDNN addresses a different differential-signature inversion mechanism for obfuscated embedding matrices. Our contribution is instead an empirical characterization of whether orthogonal invariants are sufficiently discriminative to cause leakage in learned representations. We evaluate this boundary across vision and text; NN, norm, Gram, and EDNN-style attacks; pre-normalization, noise, and salting; fresh and reused keys; and defense-aware supervised probes. We also separate two questions that were conflated in the original presentation: whether a coordinate-space decoder fails without the key, and whether the representation is private under invariant auxiliary information.

The empirical distinction is substantial. Pure rotation leaves average vision Gram retrieval at 100%. In the five-seed Food-101 study, salted rotation reduces Gram Top-10 retrieval to 0.202±0.028%0.202\pm0.028\% while retaining 88.52±0.31%88.52\pm0.31\% authorized Top-1 accuracy. We present salting as an empirical trade-off under the evaluated protocols, not as a privacy guarantee.

Addressed location: Main: Introduction, Related Work, and Discussion. Supplement: Section C and Sections G–I covering attack scope and additional evaluations.

AuthorsOpenReview ↗

Author Response: CW3 + Q2 + Q3

We agree that this was unclear in the original submission. The revision now defines the server state as X∈RB×T×C×V,X\in\mathbb{R}^{B\times T\times C\times V}, where BB is batch size, TT is token or sequence position, CC is channel count, and VV is the encrypted coordinate dimension. Query, key, value, and output projections are bias-free channel maps: their learned matrices act on CC, never on VV. Thus no learned dense matrix selects a basis in the encrypted coordinate space.

Attention scores contract over the channel and encrypted-coordinate axes. Because all participating vectors share the same orthogonal action, these inner products are invariant. The resulting softmax weights therefore remain unchanged, while their weighted aggregation of values transforms equivariantly. Multi-head attention partitions CC, not VV. EqText and COCO use causal masking; the vision models use full attention.

We also expanded the reproducibility material. The supplementary now gives an architecture and tensor-shape table, exact q/k/v/o and attention equations, activation and normalization definitions, optimizer and training settings, parameter counts where recorded, candidate-set sizes, and attack protocols. These additions explain both why the implementation is equivariant and how the reported experiments were instantiated.

Addressed location:Main: Methods, “Implemented Query, Key, and Value Construction.”

Supplement: Sections A–B for architectures and exact attention; Section D for training configurations; Section G for attack definitions.

AuthorsOpenReview ↗

Author Response: CW4 + Q4

We appreciate this question. The revision now defines the primary attacker as an honest-but-curious server, or an equivalent observer of the protected client-to-server tensor. The attacker knows the architecture and protection mechanism but not the realized key. Closed-reference retrieval additionally grants the attacker a known token dictionary or plaintext image gallery containing the target candidate. The implementation generates each orthogonal matrix R R by Gaussian sampling followed by QR factorization. By default, it draws a fresh key per example and forward call. That key is shared across interacting tokens and channels within the example, which is necessary for the equivariant attention computation. We added a seed-42 fresh-versus-static sensitivity experiment. Deliberate global static reuse raises the rotation linear-probe accuracy from 67.66 % 67.66% to 90.42 % 90.42%, and the salted linear probe from 1.65 % 1.65% to 85.20 % 85.20%. For rotation, plaintext–protected pairwise-cosine correlation rises from 8.58 % 8.58% to 100.00 % 100.00%. These results show that static key reuse substantially increases leakage under the evaluated protocol. Intermediate or periodic refresh schedules and repeated-query attacks were not evaluated and are now stated as limitations.

Addressed locations:

Main: Methods, “Implementation-Level Threat Model and Key Lifecycle”; Experiments, “Fresh-versus-static key sensitivity.”

Supplement: Sections E and I.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.