acceptodds
Withdrawn from ICLR 2027

Hidden Tape: Addressable Communication Beneath Identical Text

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Identical text can conceal different messages in its token representation. Prior work uses this freedom to steer a model’s response through retokenization, or hides pay- loads in generated text whose surface changes with the message. We instead hold one text fixed and ask whether its token order can store a message that a later query reads by address. We introduce Hidden Tape, an addressable communication chan- nel that encodes messages through token order while preserving decoded bytes, to- ken count, and the complete token histogram. A writer commits a token sequence before the query arrives, so one fixed sequence must support every address. We construct matched token cells that realize this channel and demonstrate reliable neural readout and learned writing. Three supervised-calibrated Qwen2.5-0.5B readers recover every bit of all 256 eight-bit messages after being frozen. An MLP writer trained with scalar correctness rewards achieves 99.98% query accuracy in the sixteen-bit setting. Targeted cell interventions verify address-specific readout. Decoding and canonically re-encoding the eight-bit tapes preserves their text while erasing the message, reducing the same readers from 100% to exactly 50% accu- racy. These results establish token order as writable, addressable message state beneath a fixed textual surface and show how the transport interface preserves or removes that state.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tokenization, language models, hidden communication, addressable memory, representation analysis, canonicalization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bhTTOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents Hidden Tape, a procedure for encoding hidden messages in sequences of tokens used by LLMs. It exploits the fact that certain strings can be tokenized in two different ways to construct "bits", which can be concatenated together to convey multi-bit messages, and later decoded by reading off one bit at a time. The paper shows a small model can learn this string-to-bit mapping to encode and decode messages.

Critical strengths

None

Critical weaknesses
  • The significance is very minor. The paper uses an already known phenomenon (that a string can have two tokenizations) to encode and decode messages and trains LLMs to learn this simple, deterministic mapping. It is unclear what this encoding scheme could be practically used for and what scientific insight is gained by showing small LLMs can learn it.

  • The paper is incredibly hard to understand. It is filled with needless jargon that obfuscates the meaning of certain concepts, e.g., "tokenizer census", "experimental transport", "logical code", "canonical erasure", etc. The method lacks a clear progression from the idea (encoding bits of information in token sequences) to the implementation (using models to read and write them?) and the evaluation reads as a list of disconnected, poorly motivated experiments.

Questions for authors

What is the practical significance and scientific merit of this work?

Reviewer_jjLpOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether token sequences with identical decoded text can carry addressable binary messages. It introduces matched token cells that preserve decoded text, token count, and token histogram while encoding information through token order. Experiments demonstrate neural readout, learned writing, and message erasure through canonical re-encoding.

Critical strengths

The paper studies an interesting and clearly defined problem. The matched-cell construction is mathematically well motivated, and the exhaustive eight-bit evaluation provides convincing support for addressable information storage. The targeted interventions and canonicalization experiments are particularly useful in distinguishing token-level information from visible text.

Critical weaknesses

The experimental communication setting relies on deliberately constructed token sequences and calibrated neural readers, leaving some uncertainty about its generality. The contribution of learned communication beyond the predefined encoding structure could also be explained more clearly.

Questions for authors
  1. The experiments mainly use repeated matched cells based on a fixed short string; could the authors clarify how broadly the construction applies to more natural textual inputs?

  2. Since canonical re-encoding removes the hidden message, please clarify which practical communication interfaces can preserve the required noncanonical token sequences.

  3. Eq. (4) specifies a fixed permutation and mask; could the authors clarify which parts of this mapping are provided during reader calibration and what the reward-trained writer must learn independently?

  4. Table 1 reports only one sixteen-bit writer seed; please make this limitation clearer when discussing scalability and reliability at larger message widths.

My main concerns are about the scope and significance of the learned communication results rather than their technical correctness. I would be open to raising my rating to 3, or potentially 4, depending on how convincingly the authors address these concerns and establish the broader contribution.

Reviewer_C9YyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper hides bits in token order beneath fixed text. If a string has two tokenisations p and q, the blocks pq and qp decode identically and use the same tokens, so R such blocks in sequence encode R bits. A LoRA-tuned Qwen2.5-0.5B reader retrieves a requested bit. A REINFORCE-trained writer reaches 99.98% at 16 bits. Canonical re-tokenisation drops accuracy to 50%.

Critical strengths

The controlled setup is clean: identical bytes, length and histogram isolate order. The routing question is genuinely non-trivial: frozen pretrained models encode each cell's bit locally (>95% probe accuracy) but transfer it to the query position poorly (58–72%). The interventions, visible-text control and reproducibility details are appropriate

Critical weaknesses

Clarity. A three-sentence idea is hidden behind undefined jargon ("A writer observes a binary word and commits one token tape"; "token tape" is never defined). The clarifying example only appears at the end of Section 3.3. The formalism is far heavier than needed: Theorem 1 has a two-line proof, Proposition 2 says a constant input gives chance accuracy, and Appendix C uses Toeplitz theory to show that attention is sensitive to order.

Significance. The existence of the channel and the 50% after canonicalisation follow from the construction (pq ≠ qp; Proposition 2). The non-trivial claim is that a calibrated reader learns address-based routing, and that claim lacks a clean test. The writer and the frozen-model probes are evaluated on held-out parity splits, but the calibrated reader never is. At 8 bits, calibration draws 32,000 samples from only 2,048 word–query pairs, so the reported 100% is effectively training accuracy. At 16 bits, there is no seen/unseen split, the evaluation set also drives early stopping, and only one seed is reported, with no count of runs that failed the inclusion criterion.

Motivation. The channel needs three things together:

  • a sender who controls raw token IDs,
  • a receiver fine-tuned on a shared address convention,
  • a transport that passes IDs through without re-tokenising.

Most real pipelines exchange text and re-tokenise it, which the paper itself shows erases the message. In the remaining setting, sender and receiver already cooperate and could send the bits directly. Hiding them only matters against an observer who reads decoded text but not token IDs, and the paper never says who that observer is or what is at stake. Citing steganography and watermarking does not replace a threat model or a use case.

Questions for authors
  1. What is the calibrated reader's accuracy on words unseen during calibration, at each width, measured on a test set not used for early stopping?
  2. Which concrete setting, or which observer, makes this channel matter?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.