- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ByteMark defines a shared text-watermarking signal on the UTF-8 bytes of published text, not on generator-specific token IDs or semantic embeddings. During generation, candidate tokens are scored through their constituent bytes and high-scoring candidates receive a logit bias. Detection reconstructs the keyed byte scores directly from text, deduplicates repeated context–byte pairs, and applies a standardized statistic with an idealized finite-sample null distribution. Experiments investigate representation replacement, heterogeneous-model relays, rewriting with and without re-embedding, generation overhead, and cross-implementation agreement. The construction provides a clear tokenizer-independent detection interface, but its quality trade-off and incremental contribution relative to closely related prior disclosures require further substantiation.
Critical strengths
-
A concrete and reproducible tokenizer-independent detector.
Defining the signal on published text bytes removes the need to recover the generating tokenizer. The implementation discussion addresses byte fragments and token-to-byte mapping instead of relying on isolated-token decoding. The reported Python, JavaScript, and C++ agreement is directly relevant to the goal of a shared detection specification.
-
Careful treatment of repeated observations, clear robustness boundaries, and promising computational efficiency.
Deduplicating context–byte pairs addresses a genuine dependence problem in keyed-score aggregation, and the fixed-text ideal-PRF analysis gives a transparent statistical reference. The paper also reports non-participating rewriting separately from cooperative re-embedding, avoiding an important ambiguity in interpreting watermark continuity. The method modifies token selection using existing model logits. The reported throughput is close to unwatermarked generation in the tested setting, suggesting a potentially useful efficiency advantage over rejection-based sentence-level methods.
Critical weaknesses
-
The detection and efficiency results lack generation-quality evaluation, and the unknown-source comparisons do not adequately quantify practical alternatives.
ByteMark biases the token distribution, but the paper does not establish the resulting quality cost through perplexity, task performance, or human/model preference evaluations. Using the same bias parameters as KGW does not ensure comparable distortion because the boosted candidate sets are constructed differently. Averaging byte scores within tokens may also interact with token length and vocabulary structure. Consequently, high detection rates and low runtime overhead do not establish a favorable quality–detection trade-off. This is a central missing component for the deployment motivation.
Replacing a detector's native tokenizer or embedding pipeline demonstrates representation dependence, but does not by itself establish superiority over a properly calibrated candidate search. Appendix E discusses Bonferroni and joint-score calibration, whereas Table 4 reports the FPR of an uncorrected search without the corresponding corrected-search TPR and runtime. The relay experiment similarly emphasizes mean local scores instead of document-level detection at controlled FPR. These comparisons are needed to quantify the practical benefit beyond the construction's already clear representation invariance.
-
Closely related prior disclosure narrows the novelty claim, while empirical operating points and sample exclusions require stronger validation.
The public patent document WO2025050084A1, "Improvements to language model watermarking", published on March 6, 2025, discusses watermarking ASCII/UTF-8 text representations instead of token IDs, tokenizer-free detection, and use across models (§§0014–0023 and 0033–0039). Its bitwise selection and token-acceptance procedure differs from ByteMark's score-and-bias construction. Nevertheless, the conceptual overlap is direct, and the submission should explain what its specific algorithm, calibration, and implementation contribute beyond that disclosure. This is a novelty and positioning concern.
The main tables explicitly use same-pool empirical thresholds, while Appendix C excludes invalid or unscoreable outputs without replacement. Such results can support a controlled comparison on the evaluated pool, but do not independently validate a deployment FPR or establish end-to-end detection success. The paper should report separate calibration/test results, confidence intervals, and the numbers of requested, generated, and scored samples for each method. Appendix I.3 provides useful theoretical-threshold diagnostics, but does not replace this validation of the main empirical operating points.
Questions for authors
-
Generation-quality trade-off and comparison with existing methods.
What generation-quality cost accompanies the reported detection rates? At comparable quality degradation and a common FPR, how does ByteMark compare with KGW and SynthID?
-
Tokenizer-search baselines and heterogeneous-model detection.
When the generating tokenizer is included in the candidate set, what TPR and runtime does a properly corrected tokenizer search achieve? What are the corresponding document-level results on the heterogeneous-model relay?
-
Technical novelty relative to prior disclosure.
How does the proposed construction differ technically from WO2025050084A1, "Improvements to language model watermarking", and which advantages beyond the shared tokenizer-independent carrier are supported by the present analysis or experiments?
-
Threshold calibration, sample exclusions, and end-to-end reliability.
Which human texts are used exclusively for threshold calibration and which for independent FPR testing? How many outputs are excluded for each method, and do the conclusions hold on a common valid subset and when generation failures are included in the end-to-end denominator?
9 comments
Reply for uidj 2
Response to Reviewer uidJ, Part 2 of 2.
Q4 and W2, calibration and denominators. We agree that the submission should have separated calibration from testing. We have now fixed 1,000 calibration and 1,000 disjoint test texts in each of C4, Wikipedia and LFQA. Test FPR uses thresholds from the calibration texts only. TPR uses submission or independently calibrated thresholds, as labelled.
| Method | Test FPR (%) | Unscored (%) | TPR, scored, submission (%) | TPR, scored, calibrated (%) | TPR, requests, submission (%) | TPR, common, submission (%) |
|---|---|---|---|---|---|---|
| ByteMark | 0.80 [0.50, 1.13] | 1.29 | 99.46 [99.20, 99.65] | 99.32 [99.04, 99.54] | 98.18 [97.74, 98.55] | 99.48 [99.22, 99.68] |
| KGW | 1.17 [0.96, 1.39] | 1.02 | 99.64 [99.42, 99.79] | 99.53 [99.28, 99.71] | 98.62 [98.24, 98.94] | 99.67 [99.45, 99.82] |
| SynthID | 1.24 [1.01, 1.48] | 1.22 | 99.98 [99.87, 100.00] | 99.98 [99.87, 100.00] | 98.76 [98.39, 99.06] | 99.98 [99.87, 100.00] |
| EXP | 1.08 [0.84, 1.32] | 1.11 | 99.89 [99.74, 99.96] | 99.55 [99.31, 99.73] | 98.78 [98.41, 99.08] | 99.88 [99.73, 99.96] |
| ITS | 1.17 [0.94, 1.40] | 1.00 | 98.00 [97.55, 98.39] | 98.70 [98.32, 99.01] | 97.02 [96.48, 97.50] | 98.08 [97.62, 98.47] |
Brackets give 95% intervals. For the test FPR they come from a text-level bootstrap over the 3,000 test texts. The common subset contains the requests scored for every method. All five methods retain a high TPR. The revision will re-measure all detection results at held-out thresholds.
Q3, relation to WO2025050084A1. We thank the reviewer for this important prior work, which we had not known before submission. Section 3 of the submission lays out how we arrived at this carrier. It starts from the letters of a text, which any verifier can read without a tokenizer, notes that letters exist only in alphabetic scripts, and therefore moves to the UTF-8 byte stream, which covers all text. UTF-8 is also the encoding in which text is most widely stored and exchanged, and this made it our choice. We are glad to see that the patent takes the same view. It precedes us in proposing this carrier with tokenizer-free detection, and we do not claim priority for that choice.
The constructions share a byte-level carrier but differ. As the reviewer notes, the patent's bitwise selection and token-acceptance procedure differs from ByteMark's score-and-bias construction. We address the algorithm, implementation and calibration, then give a direct comparison.
Algorithm, a general source. The source was designed to be general. It deliberately outputs a uniform value on [0, 1), which can be transformed into any distribution, so any mechanism that draws on keyed random numbers can use it. ByteMark's rule, a logit bias on whole tokens, is one such mechanism. Byte-level SynthID, EXP and ITS run on the same source, with quality close to the originals (Section 1) and direct detection of 99.0% to 100.0% (Section 2).
Implementation, no framework dependence. A token-level watermark depends on the tokenizer, and it also carries a less visible dependence on the generator that turns seeds into random numbers. In practice this second dependence ties implementations to a framework. The released code of KGW, EXP and ITS draws its random numbers from PyTorch's generator. The PyTorch documentation states that results "may not be reproducible between CPU and GPU executions, even when using identical seeds", and the steps it offers to limit nondeterminism hold "for a specific platform, device, and PyTorch release" (Reproducibility note). Our random numbers come from AES, a widely adopted standard whose output is the same in every implementation. The detector needs neither PyTorch nor NumPy, and stand-alone Python and C++ versions reproduce the same scores up to floating-point rounding, at 0.001 to 0.006 seconds per text (Section 3).
Calibration. Our detection statistic has a known null distribution for a fixed text under an ideal keyed function, and thresholds are set on human texts separate from the test texts. The held-out test FPR is 0.80% at the nominal 1% level (Q4 above).
Results. For a direct comparison, we implemented the patent's construction as we understand it and ran it on the prompts of Section 1 at a nominal 1% FPR.
| Method | TPR (%) |
|---|---|
| Patent construction, our implementation, = 128 | 85.0 |
| Patent construction, our implementation, = 32 | 90.5 |
| ByteMark | 99.0 |
This is our reconstruction, not the inventors' code. We had to fix details that the patent leaves open, and we regard the numbers as indicative.
The revision will cite and discuss the patent, credit its contributions, and state our own as the byte-level source, ByteMark's rule, the adaptation of established mechanisms and their evaluation.
Thank you for the additional experiments. The independent calibration split, explicit denominators, document-level relay results, and patent acknowledgment address important parts of my initial review, but several consequential issues remain:
1. The tokenizer search omits the standard document-level baseline. Taking maxima at individual bytes or over 50-byte windows alters how original detectors aggregate evidence, inflating null scores differently than document-level maximization. The natural baseline—running each detector on the full text and combining document-level evidence across candidate tokenizers (via multiple-testing-corrected p-values or calibrated document scores)—requires no token alignment. Please report this baseline (at least for the single-generator setting with the true tokenizer) alongside held-out FPR and runtime to separate search cost from local-maximum aggregation.
2. The byte-level adaptations lack algorithmic specifications and theoretical justification. Extending the random source to SynthID, EXP, and ITS requires more than uniform marginal PRF outputs. Because candidate tokens share byte prefixes, they share underlying PRF values, affecting the joint distribution, dependence structure, and distortion-free guarantees. Please specify the sampling and detection procedures—including how random values are indexed across candidates, positions, and layers—and clarify which original properties hold theoretically versus empirically, as detection rates and perplexity alone do not establish distribution preservation.
3. The length-based quantile correction overlooks repeated PRF inputs. The stated variance of a token’s mean byte score, , assumes independent byte scores. Under rolling contexts, repeated context–byte pairs receive identical PRF outputs (e.g., after eight spaces, a two-space candidate queries the same input twice, giving variance instead of ; generally for multiplicities ). Since detector deduplication does not apply during generation, please clarify how candidate scoring and quantile correction handle such repetitions.
4. Quality and deployment claims overreach the evaluated conditions. Identical bias parameters do not ensure matched distortion across different candidate sets, and the PPL shift from +41.0% to +53.3% under quantile ranking is non-negligible. Please specify the checkpoints, sampling settings, and paired uncertainty estimates for the quality experiments. Moreover, since the original protocol uses temperature 1, top-p 1, and EOS suppression, please clarify which new results retain these settings and what evidence supports extrapolating to natural stopping or tighter sampling.
5. Aggregation of FPRs and confidence intervals. With 3,000 test texts, one decision per text changes FPR in increments, indicating that and average across configurations. Please specify the decisions per text, averaging rule, and whether the bootstrap groups all observations from the same text.
Points 1 and 2 are central to my final recommendation. Please distinguish evidence already available from revisions planned for later.
Reply for uidj 1
Response to Reviewer uidJ, Part 1 of 2. We thank the reviewer for the detailed and constructive review. We also appreciate that the review finds the tokenizer-independent detector concrete and reproducible, and that it values the careful treatment of repeated observations, the clear robustness boundaries and the computational efficiency. For the experiments shared across the reviews, we kindly refer the reviewer to Sections 1 to 3 of the general response. Here we focus on the reviewer's own questions, with Q1, W1 and Q2 in this part and Q4 and Q3 in Part 2.
Q1, quality against KGW and SynthID. We agree that this comparison is central, and we first describe how we set it up. A watermark consists of a mechanism that alters sampling and a random source that drives it, and the change to the text comes mainly from the mechanism, which is why KGW and SynthID already differ in quality on token IDs. A direct comparison of ByteMark's default rule with SynthID would therefore mainly reflect the two mechanisms, whereas our contribution is the source. For this reason Section 1 compares each mechanism with itself on the two sources. For the logit bias, ByteMark and KGW have close similarity scores and detection rates, and ByteMark's perplexity change is no larger than KGW's (+41.0% against +55.2% on the full vocabulary, −37.5% against −37.9% on the top 41 candidates). For SynthID, the byte-level version keeps the behaviour of the original (perplexity change −6.1% against +5.1%, where a value nearer zero is better, and TPR 100.0% against 99.5%), and EXP and ITS show the same pattern. We read this as evidence that the byte-level source brings no systematic shift in quality.
W1, token length. We are grateful for this observation. It concerns how the rule treats tokens of different lengths, which we had not examined, and the reviewer is right that the two interact. We regard this as a limitation of the current rule. A token's score is the mean of its byte values, each uniform on [0, 1]. For a token of k bytes this mean is 0.5 for every k, but its variance is 1/(12k). The logit shift δ is the same for every selected token, yet because of this variance tokens with more bytes are selected somewhat less often, and generated text leans slightly toward shorter tokens. Detection does not use tokens and is unaffected.
The dependence can be reduced. The distribution of the mean of k uniform values is known in closed form, so a token's score can be replaced by its quantile within the distribution for its own length. Tokens of common lengths then have nearly the same chance of being selected. To see how much the change matters, we generated with both rules under the conditions of the quality experiment.
Detection and similarity are almost the same under the two rules, so the effect of the dependence on our results is small. The quantile ranking is nonetheless the more principled rule, and we will present it as an alternative in the revision.
Q2, corrected search and relays. Following the reviewer's request, Section 2 now reports a jointly calibrated search under two statistics, on single-generator text and two relays. On single-generator text with the tokenizer known, each method detects 98.5% to 99.0% of the texts under its better statistic. With thirty candidates the better of the two statistics detects at most 30% of the texts for KGW, 42% for SynthID, 11% for ITS and 90% to 95% for EXP, while ByteMark detects 98.5% to 100.0% without a search, and the byte-level versions of SynthID, EXP and ITS detect 99.0% to 100.0% in the same way. Over ten candidates a search takes 0.13 to 48 seconds per text, against 0.006 seconds for ByteMark (Section 3). Our reading is that, under these two statistics, a properly corrected search stays close to ByteMark only for EXP, which is also the slowest of the four detectors, and that for the other methods the rate falls as candidates are added.