Orthogonal Cipher Token Encryption: Equivariant Utility and Invariant Leakage

We study orthogonal cipher token encryption, in which a client rotates latent token vectors with a secret orthogonal transform while an equivariant server computes directly on the rotated representations. Because the server is constructed from rotation-equivariant operations, the same server can process arbitrary client keys without key-specific retraining, and the client can decrypt the returned features before ordinary task decoding with little utility loss in the evaluated settings. We show, however, that direct no-key decoder failure is not sufficient evidence of representation privacy. Under closed-reference auxiliary knowledge, orthogonal invariants such as norms, pairwise similarities, and Gram matrices remain observable and can enable strong token-, instance-, and class-level leakage. Client pre-normalization suppresses norm fingerprints but leaves Gram structure intact. In a controlled Food-101 experiment, salted/input-protected rotation substantially reduces Gram retrieval while retaining authorized utility, whereas noise-only perturbation reduces leakage largely by destroying utility. These results position orthogonal ciphering as a key-agnostic equivariant utility interface rather than a privacy mechanism by itself; additional mechanisms are required to suppress invariant leakage.