My main concern is that the security evaluation is much weaker than the claims. The defense is in effect a monoalphabetic substitution cipher over tokens. The attacker has unlimited ciphertext and white-box access to a model that both writes and reads the cipher. The attacks tested are mostly ones the training loss was built to defeat (nearest neighbour on embeddings and activations, and prompting the model to think in English). The natural adaptive attacks are missing:
Sequence alignment with a public reference model. The encoded model is trained by KL to copy the base model’s distribution token by token under σ. With greedy decoding on the same prompt, its trace should be close to σ applied to the base model’s trace. In these experiments the base models (Qwen3-8B, QwQ-32B) are public, and the activation attack itself already assumes access to base traces. So aligning encoded and base generations on a few hundred prompts and taking the majority vote per encoded token looks like it could recover close to all of the mapping. If the threat model says the base is never released, the attacker still has close relatives: an earlier checkpoint, a sibling model, or any model trained on similar data. A distribution-level alignment against such a model is a much stronger test than cosine nearest neighbour, and the paper’s threat model is not consistent on whether the attacker has a reference model.
Classical decipherment. A 1:1 substitution with K≈8k symbols and unlimited ciphertext is the textbook case for frequency analysis plus n-gram/LM-guided decipherment (EM- or Bayesian-style work by Knight and others). Token frequencies alone pin down many high-frequency function tokens. Nothing in the objective changes the unigram or n-gram statistics of the encoded text; they are kept by design, because the KL loss preserves the base distribution. This attack needs to be evaluated.
Using the model as a translation oracle. The causality loss forces the model to really read its encoded trace, and the answer span must be readable. So an attacker can put an encoded trace in the prompt or a previous turn and ask the model in the answer region to “repeat”, “translate” or “summarize the reasoning above”. Copying single tokens (“repeat after me: r4821”) is the extreme case. The prompting attacks only test whether the model thinks in readable tokens, not whether it will translate encoded content into its readable answer.
Learning the encoding instead of breaking it. The defended model learned the encoded vocabulary from about 8k traces. A student could plausibly do the same with more data, logit-level (KL) distillation from the teacher’s full distribution over encoded tokens, or a two-stage scheme (learn the encoded “language”, then reason in it). Distillation is only tested as SFT on 1k s1K samples with randomly initialized new embeddings. The 0% collapse could simply reflect too little data to learn ~8k new tokens. It doesn’t show the traces are useless as a training signal. A curve over the number of distillation samples is needed.
Cheap repair of the weights. Masking encoded logits gives 30–32% retention without further training. An attacker with the weights would next do a short SFT or RL run on the masked model (e.g., on its own correct, now-readable traces) or train a small translation head. How much compute it takes to recover most of the utility this way is the key measure of the defense’s cost to an attacker, and it isn’t reported. The conclusion itself says the defense can only make distillation “more expensive”, so that cost should be measured.