- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The problem is a real deployment issue: generation-time multi-bit watermarks define their bits over the originating model's tokens, so to read the payload you need that model's tokenizer. A writing service that routes between different backends can't guarantee that.
WordMark moves the bit channel to words. There are two learned pieces. A word decoder reads a bit from an auxiliary LM's hidden state at the end of a word, and is shared across generators. To get a usable 0/1 split, the authors train linear probes on several layers, balance the auxiliary model's probability mass across the two bits (Eq. 20), keep enough score spread to avoid a degenerate split (Eq. 21), and pick the layer by median ExcessBLP over 50 probe initialisations. The decoder is then trained on the frozen probe labels and frozen.
The second piece is a per-generator token encoder. Since the generator picks tokens before the word is finished, the target is y_G(b | c, v) = Σ_w p_G(w | c, v) D(b | c, w) (Eq. 1), approximated by a small fusion network over three generator hidden-state layers plus the candidate token embedding (Eq. 4). Word starts and continuations get separate encoders.
Embedding uses a deliberately narrow carrier class: ASCII alphabetic words only, no alphanumerics, URLs, emails or identifiers; for hyphen/apostrophe forms only the part before the connector (Appendix E.1). A carrier state machine keeps generation and detection in agreement (E.2), with a compiled tokenizer lookup for candidate detokenisation (E.3). Each BCH codeword bit is a parity constraint over K carrier words (Eqs. 7–8): the first K−1 take the naturally induced bit, the last is forced so the XOR matches. Word starts use a hard support gate (fall back to natural sampling below τ₀), continuations a soft penalty λ.
Detection skips the first W = 10 carriers, decodes the rest, XORs groups of K into codeword bits, and groups n bits per codeword (Eqs. 15–16). Following BREW, the payload is encoded into index-dependent masked codewords c_u = BCH(p ⊕ z_u); the detector BCH-decodes each recovered sequence, votes for a payload, rebuilds that payload's expected codewords, and scores the fraction within Hamming radius (Eq. 17). A watermark is declared if S(x) ≥ γ.
Experiments: four generators (Llama2-7B, Gemma2-9B, Mixtral-8×7B, Qwen2.5-14B-Instruct) × three auxiliary LMs (Pythia-410M, Qwen2.5-3B, Mistral-7B), on OpenWebText/BillSum/Wikipedia/CC-News and Alpaca. Payloads are 11 and 19 bits on OBWC with BCH(31,11,5) and BCH(51,19,5), and 5/6 bits on Alpaca. Defaults: K = 1, W = 10, τ₀ = 0.2, τ₁ = 0.55, λ = 1, temperature 1.0, top-p 0.99. Baselines are MPAC (bias 2), Qu et al. (bias 6, RS(3,1,1)) and BREW (bias 6, BCH(15,5,3)).
Results: carrier-bit accuracy 0.926–0.971 across the twelve pairs, PRA 0.982–1.000, PBA 0.990–1.000. PPL in the no-detection setting goes up modestly (Gemma2-9B 6.56→7.09, Llama2-7B 8.77→10.68); in the BREW-style detection setting it goes up a lot (Gemma2-9B 9.30→15.91, Qwen2.5-14B/Alpaca 2.09→6.26). Detection in Table 3: AUC 0.9963/0.9997 and TPR@1%FPR 0.9861/0.9900, against BREW's 0.9988/0.9961 and 1.0000/0.9660 — with a lower PPL on Gemma2, but essentially the same PPL on Qwen2.5-14B.
Critical strengths
The word-level channel does what it claims for detection, and the cross-model matrix is the right evidence. One decoder per auxiliary LM, reused across four generators with different tokenizers, reaching PRA ≥ 0.98 in all twelve cells (Table 1) — that's exactly the experiment the "shared decoding interface" claim needs. The carrier state machine and compiled detokenisation lookup also show the authors did the hard engineering rather than hand-waving it.
Payload recovery and watermark presence are treated as different questions. For multi-bit schemes, an ECC decode of non-watermarked text can produce a valid codeword, so "it decoded" is not "it's watermarked". Adopting BREW's repeated index-dependent codewords and reporting AUC/TPR@1%FPR instead of accuracy is the right stance, and the numbers (AUC 0.996–0.9997) are competitive with BREW while being tokenizer-agnostic at detection time.
The evaluation measures the right thing on fallback. CBA is computed over the actual carrier stream, including positions that fell back to natural sampling (Appendix I.3), not just the successes. That's a small choice but it keeps the number honest, and the per-dataset breakdown shows the channel isn't carried by one easy corpus.
Critical weaknesses
- One inserted word breaks the payload, and nobody tested it. The detector groups consecutive carrier bits: N_seq = ⌊N_car / (nK)⌋ complete sequences, with codeword index u selecting the mask z_u (Algorithm 2). There's no sync marker and no resynchronisation search. So if you insert or delete one carrier word, every later bit shifts by one, the XOR grouping pairs the wrong bits, and the codeword is gone. Even if BCH could absorb the errors, the index — and therefore the mask — is attached to a frame position, so a shifted frame decodes to a different payload candidate. Cropping the start also changes which words count as the first W eligible words.
The paper's motivation is text that gets copied, exported to PDF, posted or redistributed, and the threat model explicitly includes piggyback edits. Yet there is no experiment with insertion, deletion, paraphrase, reformatting, PDF round-trip, whitespace/Unicode normalisation or cropping. I searched the text for "synchron", "align", "sliding", "insertion", "deletion" — nothing. For a word-level channel, frame alignment is the first thing to establish, and the design has no way to recover it. Either add a synchronisation scheme or say clearly that this only works on verbatim text.
- The headline numbers don't exercise the syndrome mechanism. K = 1 is the default, and Tables 1–3 all use it. With K = 1 the parity block is vacuous — the syndrome is supposed to be the solution to forced low-support bits, and it's inactive in every headline number. It only appears in Figure 3 (Gemma2-9B, OpenWebText, Qwen2.5-3B aux, BCH(15,5,3), K = 1..4). So I can't tell whether the distortion reduction actually shows up in the main comparison.
Relatedly, detection and multi-bit are never evaluated together beyond 5 bits. Table 1 has CBA/PRA/PBA at 11 and 19 bits but no AUC/TPR; Table 3 has AUC/TPR at 5 bits only. "Multi-bit with reliable detection" is therefore demonstrated at 5 bits, and "multi-bit recovery" at 19 bits.
-
The payload is far smaller than the motivating use case. The payload is supposed to be a URL pointing to a signed provenance record — that's 100+ bits. The largest payload here is 19 bits. Each codeword needs nK carrier words (51 for BCH(51,19); roughly 255 for a code that can carry 128 bits). There's no scaling curve and no statement of how many tokens a 64- or 128-bit payload would need.
-
The method needs white-box access to the generator. Three hidden-state layers plus the token embedding, and a separately trained encoder per generator. The motivating setting is a service routing across backend LLMs, which in practice often means third-party APIs. For those, this doesn't work. The paper also cites publicly-detectable watermarking but doesn't discuss that WordMark verification needs the detector model and the secret κ — it isn't publicly detectable in the cryptographic sense.
-
The comparison isn't at matched distortion. Baselines keep their recommended settings: MPAC at bias 2, Qu and BREW at bias 6 (Appendix I.2). Those are different points on different quality–detectability curves, so Table 3 compares methods rather than operating points. MPAC's low PPL comes with AUC 0.699 on Qwen2.5-14B, while BREW and Qu pay PPL 6.25/6.35 against 2.09 non-watermarked. On the other side, WordMark's quality advantage is not uniform: it disappears on Qwen2.5-14B/Alpaca. And in Table 1, several watermarked PPLs are below the non-watermarked reference (Qwen2.5-14B-Instruct generator: 3.323/3.520 vs 3.454), which needs an explanation — the reference may not be length- or domain-matched.
-
γ, sample sizes and false-positive calibration are missing. The detection threshold is never given a value or a calibration procedure, and there's no bound on false positives — "reduces false positives" is an empirical claim with no stated guarantee. The number of documents per cell, the number and source of non-watermarked negatives, and the number of seeds are all unstated, even though TPR@1%FPR is estimated from those negatives. There is no evaluation on human-written, machine-translated or out-of-domain text, and no multilingual evaluation at all. That last one matters because the carrier policy is ASCII A-Z/a-z only, which means WordMark has essentially no carriers in Chinese, Japanese, Arabic or Cyrillic text. The paper cites SAE-Mark, which is explicitly multilingual, and never tests it.
-
Small reporting gaps. No confidence intervals or significance tests anywhere. PRA/PBA are computed only on examples that complete at least one codeword block (I.3), so short generations are excluded and the reported PRA overstates end-to-end success — the completion rate isn't given. And a document full of numbers, URLs or code has few carriers; there's no sensitivity analysis for that either.
Questions for authors
- What happens to payload recovery if I insert one word, delete one word, paraphrase a sentence, round-trip through PDF/markdown, or crop the first paragraph? Please give PRA/AUC against edit rate, and say explicitly whether verbatim text is required. If there's no resynchronisation, what's the intended mitigation?
- What are PRA, AUC and PPL for 32/64/128-bit payloads, and how many generated tokens would a URL-sized payload need at K = 1 and K = 3?
- Why is detection reported only at 5 bits while 11/19-bit results have no AUC/TPR? Please add detection at 11 and 19 bits.
- If syndrome embedding is a contribution, why is K = 1 used in every headline result? Please run Tables 1 and 3 at K = 2 or 3.
- Can WordMark work with an API-only backend that exposes no hidden states? If not, please state that as an assumption, and give the onboarding cost for a new backend (encoder training, calibration, auxiliary-LM inference per verification).
- What is the block-completion rate per configuration, and what is PRA if examples without a complete block count as failures?
- What is γ in each setting, how is it chosen, and what are the false-positive counts on human-written and out-of-domain text? How many negatives were used for the 1% FPR point?
- Please show quality–detectability curves by sweeping each baseline's bias, and compare at matched PPL or matched TPR@1%FPR. Why was MPAC run at bias 2 when its detection is near chance on Qwen2.5-14B?
- Why is the watermarked PPL lower than the non-watermarked PPL for the Qwen2.5-14B-Instruct generator in Table 1?
- What happens on Chinese or Arabic text, given the ASCII-only carrier policy? And how many documents, negatives and seeds were used per cell?
W1&Q1 (need to show robustness against text editing)
Please refer to “Common concern of the three reviewers: synchronization robustness under text editing”
W2 (need to show distortion reduction from syndrome size K)
We thank the reviewer for raising this concern. We used K=1 as the default because it prioritizes recovery accuracy and requires fewer carriers per codeword, but agree that this setting alone does not demonstrate the benefit of syndrome embedding. To examine the quality–recovery trade-off, the table below reports a controlled K-ablation using Gemma-2-9B on OpenWebText and BCH$(31,11)$. Increasing K reduces the density of constrained carrier positions within the embedding interval to 1/K, allowing more words to be sampled without a syndrome constraint. WM PPL decreases from 9.846 to 8.887, approaching the matched Non-WM PPL of 8.738, while SBA and payload recovery decrease because one syndrome bit(i.e., codeword bit) requires K carrier bits to decode.
W2&W3&Q2&Q3 (need detection reported for larger payload length)
We thank the reviewer for raising the payload-scaling concern. We add a long-payload BREW-style evaluation with BCH(255,79,27) and K=1, targeting 5 indexed codewords. One codeword requires 255K post-warmup carrier words; 5 indexed codewords provide repeated recovery and verification evidence and require 1,275 carriers at K=1. However, even with \texttt{max_new_tokens}=3200, only approximately 43% of generations completed at least one codeword under ordinary EOS stopping. To address early termination, we use EOS continuation: sampled EOS tokens remain in the autoregressive history, and generation continues without inserting text or resetting the watermark state, stopping after five complete indexed BREW blocks, reaching the 3,200-token limit, or sampling EOS 64 times. The results are shown in the following table, where CBA is 0.9710, close to the 0.9702 observed for BCH(31,11) with the same generator--auxiliary pair, suggesting comparable average word-level channel accuracy in these two settings. Payload recovery additionally depends on syndrome-bit accuracy, ECC, and block completion; early termination remains an important practical constraint. We will report carrier budgets explicitly, since token requirements depend on carrier density and stopping behavior.
The original 5-bit experiment provides a controlled comparison on decoding and detection with baselines under a 250-token budget, whereas the 11/19-bit experiments only evaluate decoding performance. The new 79-bit experiment jointly evaluates decoding and detection under a longer continuation protocol; we will make this protocol and the distinct evaluation purposes explicit in the revised manuscript.
W4&Q5 (need to explain whether WordMark works with an API-only backend that exposes no hidden states, and give the onboarding cost for a new backend)
We thank the reviewer for identifying two important assumptions that were not stated clearly enough.
First, WordMark does not claim cryptographic public watermark detection. The issuer's public–private key pair authenticates the provenance record. Comparing the received text's digest with this signed digest then checks text integrity. However, watermark detection and payload recovery require the masking key κ, which is used to construct indexed codewords during embedding and reconstruct the expected indexed codewords during verification. Releasing κ would make this computation publicly executable, but would not provide the asymmetric separation between secret embedding and public verification needed in public-key watermarking schemes. We will clarify this distinction in the threat model and related-work discussion.
Second, WordMark does not support an output-only API backend. Embedding bits requires access to the generator distribution and control of autoregressive sampling; our implementation additionally uses prefix hidden states and token embeddings to train generator-specific token encoders. Thus, a text-only API or an API exposing only log-probabilities is insufficient. The intended deployment setting is a service-controlled pool of self-hosted backends, or provider-cooperative backends that deploy the WordMark wrapper.