acceptodds
Under review as a conference paper at ICLR 2027

WordMark: A Tokenizer-agnostic Word-level Watermark for Large Language Models

Abstract

Large language models are increasingly used by writing services to generate professional documents whose provenance and integrity may later need to be verified. Although multi-bit watermarking is able to embed a signed provenance record directly in the text, existing generation-time methods typically define bits over model-specific token sequences. Thus, verification requires the originating tokenizer, making these methods poorly suited to services that route requests among heterogeneous or evolving backend models. We present WordMark, a tokenizer-agnostic, word-level multi-bit watermark. During generation, a lightweight generator-specific token encoder predicts the word-level bit that each candidate token is likely to produce, enabling bit-guided autoregressive sampling. A shared word decoder then uses representations from an auxiliary language model to recover context-dependent bits directly from the resulting words. WordMark embeds each ECC codeword bit as a parity constraint over a small group of carrier words, reducing forced low-probability choices. For watermark detection, the payload is represented by repeated index-dependent ECC codewords and verified through designated-codeword matching, reducing false positives on non-watermarked text. Experiments across multiple auxiliary language models, generators, payload lengths, and text domains demonstrate reliable payload recovery and watermark detection while preserving generation quality without requiring the detector to know the generator tokenizer.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM watermark, provenance verification, tokenizer-independent watermark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nP66OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The problem is a real deployment issue: generation-time multi-bit watermarks define their bits over the originating model's tokens, so to read the payload you need that model's tokenizer. A writing service that routes between different backends can't guarantee that.

WordMark moves the bit channel to words. There are two learned pieces. A word decoder reads a bit from an auxiliary LM's hidden state at the end of a word, and is shared across generators. To get a usable 0/1 split, the authors train linear probes on several layers, balance the auxiliary model's probability mass across the two bits (Eq. 20), keep enough score spread to avoid a degenerate split (Eq. 21), and pick the layer by median ExcessBLP over 50 probe initialisations. The decoder is then trained on the frozen probe labels and frozen.

The second piece is a per-generator token encoder. Since the generator picks tokens before the word is finished, the target is y_G(b | c, v) = Σ_w p_G(w | c, v) D(b | c, w) (Eq. 1), approximated by a small fusion network over three generator hidden-state layers plus the candidate token embedding (Eq. 4). Word starts and continuations get separate encoders.

Embedding uses a deliberately narrow carrier class: ASCII alphabetic words only, no alphanumerics, URLs, emails or identifiers; for hyphen/apostrophe forms only the part before the connector (Appendix E.1). A carrier state machine keeps generation and detection in agreement (E.2), with a compiled tokenizer lookup for candidate detokenisation (E.3). Each BCH codeword bit is a parity constraint over K carrier words (Eqs. 7–8): the first K−1 take the naturally induced bit, the last is forced so the XOR matches. Word starts use a hard support gate (fall back to natural sampling below τ₀), continuations a soft penalty λ.

Detection skips the first W = 10 carriers, decodes the rest, XORs groups of K into codeword bits, and groups n bits per codeword (Eqs. 15–16). Following BREW, the payload is encoded into index-dependent masked codewords c_u = BCH(p ⊕ z_u); the detector BCH-decodes each recovered sequence, votes for a payload, rebuilds that payload's expected codewords, and scores the fraction within Hamming radius (Eq. 17). A watermark is declared if S(x) ≥ γ.

Experiments: four generators (Llama2-7B, Gemma2-9B, Mixtral-8×7B, Qwen2.5-14B-Instruct) × three auxiliary LMs (Pythia-410M, Qwen2.5-3B, Mistral-7B), on OpenWebText/BillSum/Wikipedia/CC-News and Alpaca. Payloads are 11 and 19 bits on OBWC with BCH(31,11,5) and BCH(51,19,5), and 5/6 bits on Alpaca. Defaults: K = 1, W = 10, τ₀ = 0.2, τ₁ = 0.55, λ = 1, temperature 1.0, top-p 0.99. Baselines are MPAC (bias 2), Qu et al. (bias 6, RS(3,1,1)) and BREW (bias 6, BCH(15,5,3)).

Results: carrier-bit accuracy 0.926–0.971 across the twelve pairs, PRA 0.982–1.000, PBA 0.990–1.000. PPL in the no-detection setting goes up modestly (Gemma2-9B 6.56→7.09, Llama2-7B 8.77→10.68); in the BREW-style detection setting it goes up a lot (Gemma2-9B 9.30→15.91, Qwen2.5-14B/Alpaca 2.09→6.26). Detection in Table 3: AUC 0.9963/0.9997 and TPR@1%FPR 0.9861/0.9900, against BREW's 0.9988/0.9961 and 1.0000/0.9660 — with a lower PPL on Gemma2, but essentially the same PPL on Qwen2.5-14B.

Critical strengths

The word-level channel does what it claims for detection, and the cross-model matrix is the right evidence. One decoder per auxiliary LM, reused across four generators with different tokenizers, reaching PRA ≥ 0.98 in all twelve cells (Table 1) — that's exactly the experiment the "shared decoding interface" claim needs. The carrier state machine and compiled detokenisation lookup also show the authors did the hard engineering rather than hand-waving it.

Payload recovery and watermark presence are treated as different questions. For multi-bit schemes, an ECC decode of non-watermarked text can produce a valid codeword, so "it decoded" is not "it's watermarked". Adopting BREW's repeated index-dependent codewords and reporting AUC/TPR@1%FPR instead of accuracy is the right stance, and the numbers (AUC 0.996–0.9997) are competitive with BREW while being tokenizer-agnostic at detection time.

The evaluation measures the right thing on fallback. CBA is computed over the actual carrier stream, including positions that fell back to natural sampling (Appendix I.3), not just the successes. That's a small choice but it keeps the number honest, and the per-dataset breakdown shows the channel isn't carried by one easy corpus.

Critical weaknesses
  1. One inserted word breaks the payload, and nobody tested it. The detector groups consecutive carrier bits: N_seq = ⌊N_car / (nK)⌋ complete sequences, with codeword index u selecting the mask z_u (Algorithm 2). There's no sync marker and no resynchronisation search. So if you insert or delete one carrier word, every later bit shifts by one, the XOR grouping pairs the wrong bits, and the codeword is gone. Even if BCH could absorb the errors, the index — and therefore the mask — is attached to a frame position, so a shifted frame decodes to a different payload candidate. Cropping the start also changes which words count as the first W eligible words.

The paper's motivation is text that gets copied, exported to PDF, posted or redistributed, and the threat model explicitly includes piggyback edits. Yet there is no experiment with insertion, deletion, paraphrase, reformatting, PDF round-trip, whitespace/Unicode normalisation or cropping. I searched the text for "synchron", "align", "sliding", "insertion", "deletion" — nothing. For a word-level channel, frame alignment is the first thing to establish, and the design has no way to recover it. Either add a synchronisation scheme or say clearly that this only works on verbatim text.

  1. The headline numbers don't exercise the syndrome mechanism. K = 1 is the default, and Tables 1–3 all use it. With K = 1 the parity block is vacuous — the syndrome is supposed to be the solution to forced low-support bits, and it's inactive in every headline number. It only appears in Figure 3 (Gemma2-9B, OpenWebText, Qwen2.5-3B aux, BCH(15,5,3), K = 1..4). So I can't tell whether the distortion reduction actually shows up in the main comparison.

Relatedly, detection and multi-bit are never evaluated together beyond 5 bits. Table 1 has CBA/PRA/PBA at 11 and 19 bits but no AUC/TPR; Table 3 has AUC/TPR at 5 bits only. "Multi-bit with reliable detection" is therefore demonstrated at 5 bits, and "multi-bit recovery" at 19 bits.

  1. The payload is far smaller than the motivating use case. The payload is supposed to be a URL pointing to a signed provenance record — that's 100+ bits. The largest payload here is 19 bits. Each codeword needs nK carrier words (51 for BCH(51,19); roughly 255 for a code that can carry 128 bits). There's no scaling curve and no statement of how many tokens a 64- or 128-bit payload would need.

  2. The method needs white-box access to the generator. Three hidden-state layers plus the token embedding, and a separately trained encoder per generator. The motivating setting is a service routing across backend LLMs, which in practice often means third-party APIs. For those, this doesn't work. The paper also cites publicly-detectable watermarking but doesn't discuss that WordMark verification needs the detector model and the secret κ — it isn't publicly detectable in the cryptographic sense.

  3. The comparison isn't at matched distortion. Baselines keep their recommended settings: MPAC at bias 2, Qu and BREW at bias 6 (Appendix I.2). Those are different points on different quality–detectability curves, so Table 3 compares methods rather than operating points. MPAC's low PPL comes with AUC 0.699 on Qwen2.5-14B, while BREW and Qu pay PPL 6.25/6.35 against 2.09 non-watermarked. On the other side, WordMark's quality advantage is not uniform: it disappears on Qwen2.5-14B/Alpaca. And in Table 1, several watermarked PPLs are below the non-watermarked reference (Qwen2.5-14B-Instruct generator: 3.323/3.520 vs 3.454), which needs an explanation — the reference may not be length- or domain-matched.

  4. γ, sample sizes and false-positive calibration are missing. The detection threshold is never given a value or a calibration procedure, and there's no bound on false positives — "reduces false positives" is an empirical claim with no stated guarantee. The number of documents per cell, the number and source of non-watermarked negatives, and the number of seeds are all unstated, even though TPR@1%FPR is estimated from those negatives. There is no evaluation on human-written, machine-translated or out-of-domain text, and no multilingual evaluation at all. That last one matters because the carrier policy is ASCII A-Z/a-z only, which means WordMark has essentially no carriers in Chinese, Japanese, Arabic or Cyrillic text. The paper cites SAE-Mark, which is explicitly multilingual, and never tests it.

  5. Small reporting gaps. No confidence intervals or significance tests anywhere. PRA/PBA are computed only on examples that complete at least one codeword block (I.3), so short generations are excluded and the reported PRA overstates end-to-end success — the completion rate isn't given. And a document full of numbers, URLs or code has few carriers; there's no sensitivity analysis for that either.

Questions for authors
  1. What happens to payload recovery if I insert one word, delete one word, paraphrase a sentence, round-trip through PDF/markdown, or crop the first paragraph? Please give PRA/AUC against edit rate, and say explicitly whether verbatim text is required. If there's no resynchronisation, what's the intended mitigation?
  2. What are PRA, AUC and PPL for 32/64/128-bit payloads, and how many generated tokens would a URL-sized payload need at K = 1 and K = 3?
  3. Why is detection reported only at 5 bits while 11/19-bit results have no AUC/TPR? Please add detection at 11 and 19 bits.
  4. If syndrome embedding is a contribution, why is K = 1 used in every headline result? Please run Tables 1 and 3 at K = 2 or 3.
  5. Can WordMark work with an API-only backend that exposes no hidden states? If not, please state that as an assumption, and give the onboarding cost for a new backend (encoder training, calibration, auxiliary-LM inference per verification).
  6. What is the block-completion rate per configuration, and what is PRA if examples without a complete block count as failures?
  7. What is γ in each setting, how is it chosen, and what are the false-positive counts on human-written and out-of-domain text? How many negatives were used for the 1% FPR point?
  8. Please show quality–detectability curves by sweeping each baseline's bias, and compare at matched PPL or matched TPR@1%FPR. Why was MPAC run at bias 2 when its detection is near chance on Qwen2.5-14B?
  9. Why is the watermarked PPL lower than the non-watermarked PPL for the Qwen2.5-14B-Instruct generator in Table 1?
  10. What happens on Chinese or Arabic text, given the ASCII-only carrier policy? And how many documents, negatives and seeds were used per cell?
1 comment

W1&Q1 (need to show robustness against text editing)

Please refer to “Common concern of the three reviewers: synchronization robustness under text editing”

W2 (need to show distortion reduction from syndrome size KK)

We thank the reviewer for raising this concern. We used K=1K=1 as the default because it prioritizes recovery accuracy and requires fewer carriers per codeword, but agree that this setting alone does not demonstrate the benefit of syndrome embedding. To examine the quality–recovery trade-off, the table below reports a controlled KK-ablation using Gemma-2-9B on OpenWebText and BCH$(31,11)$. Increasing KK reduces the density of constrained carrier positions within the embedding interval to 1/K1/K, allowing more words to be sampled without a syndrome constraint. WM PPL decreases from 9.846 to 8.887, approaching the matched Non-WM PPL of 8.738, while SBA and payload recovery decrease because one syndrome bit(i.e., codeword bit) requires KK carrier bits to decode.

KKCBA (%)SBA (%)PBA (%)PRA (%)WM PPL
197.097.099.899.89.846
297.494.997.597.39.309
397.292.195.194.39.208
497.189.583.382.38.887

W2&W3&Q2&Q3 (need detection reported for larger payload length)

We thank the reviewer for raising the payload-scaling concern. We add a long-payload BREW-style evaluation with BCH(255,79,27) and K=1K=1, targeting 5 indexed codewords. One codeword requires 255K255K post-warmup carrier words; 5 indexed codewords provide repeated recovery and verification evidence and require 1,275 carriers at K=1K=1. However, even with \texttt{max_new_tokens}=3200, only approximately 43% of generations completed at least one codeword under ordinary EOS stopping. To address early termination, we use EOS continuation: sampled EOS tokens remain in the autoregressive history, and generation continues without inserting text or resetting the watermark state, stopping after five complete indexed BREW blocks, reaching the 3,200-token limit, or sampling EOS 64 times. The results are shown in the following table, where CBA is 0.9710, close to the 0.9702 observed for BCH(31,11) with the same generator--auxiliary pair, suggesting comparable average word-level channel accuracy in these two settings. Payload recovery additionally depends on syndrome-bit accuracy, ECC, and block completion; early termination remains an important practical constraint. We will report carrier budgets explicitly, since token requirements depend on carrier density and stopping behavior.

The original 5-bit experiment provides a controlled comparison on decoding and detection with baselines under a 250-token budget, whereas the 11/19-bit experiments only evaluate decoding performance. The new 79-bit experiment jointly evaluates decoding and detection under a longer continuation protocol; we will make this protocol and the distinct evaluation purposes explicit in the revised manuscript.

PayloadCBABCRAvg. blocksPRAPBAAUCTPR@1% FPRPPL (WM / NWM)
79 bits0.971085.4%4.12699.1%99.6%0.98300.978214.956 / 8.547

W4&Q5 (need to explain whether WordMark works with an API-only backend that exposes no hidden states, and give the onboarding cost for a new backend)

We thank the reviewer for identifying two important assumptions that were not stated clearly enough.

First, WordMark does not claim cryptographic public watermark detection. The issuer's public–private key pair authenticates the provenance record. Comparing the received text's digest with this signed digest then checks text integrity. However, watermark detection and payload recovery require the masking key κ\kappa, which is used to construct indexed codewords during embedding and reconstruct the expected indexed codewords during verification. Releasing κ\kappa would make this computation publicly executable, but would not provide the asymmetric separation between secret embedding and public verification needed in public-key watermarking schemes. We will clarify this distinction in the threat model and related-work discussion.

Second, WordMark does not support an output-only API backend. Embedding bits requires access to the generator distribution and control of autoregressive sampling; our implementation additionally uses prefix hidden states and token embeddings to train generator-specific token encoders. Thus, a text-only API or an API exposing only log-probabilities is insufficient. The intended deployment setting is a service-controlled pool of self-hosted backends, or provider-cooperative backends that deploy the WordMark wrapper.

Reviewer_ESvZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents WordMark, a tokenizer-agnostic, word-level multi-bit watermarking framework for LLMs. By pairing a shared word decoder with generator-specific token encoders, WordMark embeds watermarks during autoregressive generation without requiring the source tokenizer for detection. It integrates syndrome-based embedding and error-correcting codes to preserve text quality and enhance payload recovery, demonstrating strong performance across diverse language models and domains.

Critical strengths
  1. Combining a shared word decoder with generator-specific token encoders effectively bridges token-level generation and word-level watermark decoding, while syndrome-based embedding balances watermark reliability with text quality.
  2. The paper addresses the practical limitation of tokenizer-dependent watermarking, which is particularly relevant for applications involving heterogeneous LLM backends.
Critical weaknesses
  1. The paper lacks experiments on common text perturbations, such as word substitution, insertion, deletion, and paraphrasing. These common attacks could disrupt watermark synchronization and payload recovery, raising concerns about its practical robustness.
  2. The evaluation focuses primarily on token-level multi-bit watermarking methods, omitting tokenizer-agnostic approaches such as SAEMark. Comparing against these baselines would better highlight WordMark's advantages in efficiency, detection performance, and text quality.
Questions for authors
  1. Could the authors evaluate WordMark under common text modifications, such as word substitution, insertion, deletion, and paraphrasing? In particular, how do these perturbations affect word-level carrier synchronization and payload recovery?
  2. Could the authors provide additional comparisons with existing tokenizer-agnostic watermarking approaches, such as SAEMark (NeurIPS 2025)? Such comparisons would help clarify WordMark's practical advantages in terms of generation efficiency, detection performance, and text quality.
Reviewer_F4bjOpenReview ↗
Rating
2(−1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper proposes WordMark, a tokenizer-agnostic, word-level multi-bit watermark for large language models. Existing generation-time watermarks embed signals into model-specific token sequences, requiring the exact original tokenizer for detection. WordMark solves this by learning a context-dependent word-to-bit mapping via a shared auxiliary LLM, allowing a universal decoder to extract bits directly from words. During generation, a lightweight token encoder guides sampling. To preserve text quality, WordMark uses syndrome-based embedding to enforce parity constraints over word blocks, avoiding forced, low-probability token choices. Combined with robust error-correction verification, WordMark ensures reliable, cross-model payload recovery.

Critical strengths
  • Tokenizer-Agnostic Design: A single shared word decoder reliably extracts watermarks across diverse LLMs without needing the originating tokenizer.

  • Preserved Text Quality: Syndrome-based embedding minimizes generation distortion by avoiding forced, low-probability token selections during the watermarking process.

  • Low Overhead: The lightweight token encoder steers autoregressive sampling efficiently without requiring the expensive generation of multiple candidate sentences.

  • Reliable Detection: Integrating repeated index-dependent ECC codewords ensures highly accurate payload recovery and effectively minimizes false positive rates.

Critical weaknesses
  • Fragmented Presentation: Core algorithmic details are heavily buried in appendices, requiring substantial effort to follow the main text.

  • Language Limitations: The carrier policy only uses ASCII alphabetic words, severely limiting its applicability to non-English or highly technical texts.

  • Text Quality Degradation: Despite syndrome embedding, forced carrier positions still noticeably increase perplexity compared to unwatermarked baseline generations.

  • Desynchronization Vulnerability: Word-level embedding remains highly vulnerable to insertion and deletion attacks, which can desynchronize carrier blocks and disrupt decoding.

  • Paraphrasing Susceptibility: Semantic-preserving paraphrasing will naturally alter the specific carrier words, easily destroying the embedded watermark payload.

  • Auxiliary Model Dependency: Decoding relies entirely on a specific auxiliary LLM; payload recovery fails if this exact model becomes unavailable.

  • Computational Overhead: While lightweight, running the token encoder alongside the generator at every step still introduces unavoidable computational latency.

Questions for authors
  • How do you propose mitigating WordMark's critical vulnerability to desynchronization caused by simple insertion/deletion edits or semantic-preserving paraphrasing?

  • Can this framework be practically extended to non-English languages or code generation?

  • Since detection relies entirely on a specific auxiliary LLM, how should deployers manage the risk of that model being deprecated, updated, or made unavailable?

  • What is the actual empirical latency overhead (e.g., in milliseconds per token) introduced by running the token encoder during inference?

  • Perplexity still increases noticeably due to forced carrier positions. Are there theoretical bounds or further optimizations possible to minimize this distortion?

1 comment

Thank you for the additional comparisons and analyses. The SAEMark and matched-PPL evaluations, corrected PPL reference, block-completion results, and latency measurements address several of my earlier questions.

However, the central robustness issue remains unresolved. The authors confirm that insertions, deletions, or paraphrasing can destroy synchronization and leave the verifier unable to attribute the text. A signed-digest check can reject an altered document, but it cannot track its source after the edits expected during redistribution. This substantially narrows WordMark’s utility to integrity verification of unchanged text.

The long-payload result also falls short of the stated provenance use case: the 79-bit experiment uses continuation up to 3,200 tokens, achieves an 85.4% block-completion rate, and raises PPL from 8.547 to 14.956. It does not establish practical recovery of a URL-sized payload under ordinary generation. Moreover, false-positive evaluation uses non-watermarked model outputs rather than human-written text, and the reported experiments use one seed.

The additions improve the empirical evaluation, but the central deployment limitations remain. I would revise my rating from weak accept to weak reject.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.