acceptodds
Under review as a conference paper at ICLR 2027

RamoMark: Multi-Bit Watermarking via Randomness-Modulated Speculative Sampling

Abstract

Speculative sampling accelerates LLM inference through a draft-then-verify paradigm, where proposed tokens are accepted or rejected by comparing the probability ratio of target and draft models against a random threshold. In this paper, we identify this threshold as an intrinsic and controllable source of randomness, and show that multi-bit watermarks can be embedded into the generated text by modulating it-hence unifying efficient inference and output provenance within a single sampling framework. Specifically, we introduce a control variable that modulates the threshold, and construct a communication channel by treating this variable as the input and the generated token as the output. This construction naturally casts multi-bit watermarking as a channel coding problem. Drawing on coding theory, we propose randomness-modulated speculative sampling with watermark (RamoMark), a keyless and robust watermarking algorithm that provably preserves the output distribution. As the codebook is constructed offline, RamoMark scales naturally to long-bitstream watermarking. Empirical evaluations confirm reliable watermark detection with negligible degradation in generation quality and superior performance over baseline methods.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Watermarking, Multi-bit Watermarking, Speculative Decoding, Steganography, Information Theory, Channel Coding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_caRYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RamoMark, a mutilbit watermarking scheme that leverages speculative decoding acceptance threshold as a channel to embed a multibit watermark. In particular, the authors highlight that a given token distribution is different whether it is accepted (in that case this is the draft model distribution) or rejected (in that case it is the residual distribution). By conditioning the sampling of the acceptance threshold uniform variable on a binary variable encoding the message, RamoMark can therefore encode the message in the generated token sequence.

Specifically, given a generation length nn and a message mm, RamoMark encodes mm in a sequence of nn bits with a ratio θ\theta of zeros (where θ∈[0,1]\theta \in [0,1] is a parameter of the scheme). When sampling the token at position tt, if the tt-th bit is 00 then the acceptance threshold is sampled in [0,θ][0,\theta] and else in [θ,1][\theta,1]. For detection, given a generated text, RamoMark uses an LLM forward pass to compute an approximate maximum likelihood message.

Their results show that RamoMark outperforms prior multibit watermark schemes, with minimal impact on average token acceptance. They also show that their approach can be adapted for zero-bit detection.

Critical strengths
  • The conceptual idea behind the method is clever, easy-to-understand and appears to work relatively well in practice. In particular, unlike many prior multibit watermarking works, it does not rely on position allocation but uses a novel mechanism (i.e. the speculative decoding threshold)
  • The paper is easy to follow
Critical weaknesses
  • The method appears to have several practical limitations not well highlighted in the paper. RamoMark assumes that the number of generated tokens is known in advance. In practice, this is not the case. RamoMark requires speculative decoding to be "turn on" all the times: in practice, depending on requests batch sizes, model providers can dynamically decide when to use speculative decoding. RamoMark detection, unlike most prior works, is expensive as it requires a model forward pass. Lastly, RamoMark has been evaluated only with a simple draft model speculative decoding approach and it remains unclear if it works with newer speculative decoding variants (e.g. MTP, DFlash, DTrees...).
  • The evaluation section is very sparse. In particular, there is no robustness evaluation. The scheme parameter θ\theta is not ablated and the motivation behind its choice in remark 4.2 is not evaluated empirically. The quality evaluation is very sparse (only PPL metrics). The zerobit evaluation is slightly disingenuous as RamoMark in the zero-bit setting requires held-out calibration unlike most baselines, and does not add significant values to the paper (I would move it to the appendix and e.g. instead have more in-depth evaluation of the multibit setting). Lastly, the writing in the evaluation section is poor as it only describes the figures/table values and does not give real interpretation/explanation of the results.
Questions for authors
  • How could RamoMark work in a setting where the generated text length is not known in advance?
  • I was wondering if RamoMark would work in settings where the estimated probability without the prompt differ significantly from the probabilities at generation time? I was thinking of thinking models without the thinking trace, or agentic settings (e.g. a codebase where some of the code/documentation is AI-written)?
5 comments
AuthorsOpenReview ↗

Responses to unknown generation length and omputational cost

We sincerely thank the reviewer for recognizing the novelty and simplicity of using the speculative-sampling acceptance threshold as a watermarking channel, and for the thoughtful questions about deployment. We agree that the original evaluation placed too much emphasis on clean recovery and did not sufficiently explain several practical limitations. We have now conducted additional ablations and quality/robustness evaluations. Below, we distinguish the demonstrated behavior of the submitted RamoMark construction from extensions that would require further algorithm design.

Weakness 1 / Question 1: What if the generation length is unknown in advance?

We agree that the current fixed-block construction chooses its codeword length before generation. To test how sensitive it is to early termination, we decoded prefixes of the same already-generated Qwen/CNN watermarked continuations while retaining the original fixed codebook. The following are message-recovery counts on 32 prompts:

Observed fraction of planned payload50%75%90%100%
4 bits (120-token planned)29/3229/3232/3230/32
8 bits (240-token planned)24/3229/3231/3231/32

Thus, early termination need not always destroy the watermark, and at these tested budgets useful recovery is possible before the full block is observed. For deployment, one potential extension is framed, independently decodable blocks with an explicit synchronization rule; complete blocks could be recovered even when generation stops unexpectedly. This remains a proposed extension, not an implemented or validated rateless solution. We will clarify this limitation and include the partial-block recovery figure.

Weakness 1: Dynamic speculative decoding and newer variants (MTP, DFlash, DTrees)

The current implementation assumes ordinary draft–verify–correct speculative sampling with a well-defined acceptance threshold and residual correction law. If speculative decoding is turned off, the emitted token is not in general generated through the same control-dependent channel. Similarly, MTP, DFlash and tree-based decoding may use different proposal/verification structures; the induced channel and receiver likelihoods must be derived for each variant.

We therefore agree that compatibility with dynamic SD-on/off switching and these modern speculative-decoding variants has not been demonstrated. We will make the assumptions explicit in the limitations section and will not suggest that the present experiments establish drop-in compatibility. Nevertheless, we would like to note that one main novelty our our paper lies in embedding watermark naturally in the specilative sampling pipeline.

Weakness 1: Computational cost of model-dependent detection

We agree that the submitted detector is more expensive than key-based detectors using only inexpensive token statistics: it requires target/draft likelihood evaluation and can dominate a downstream detection workload. The new Qwen/CNN comparison records the following descriptive generation and decoding times, in seconds per continuation:

BitsPayload tokensMethodGeneration (s)Decoding (s)
4120RamoMark9.203.23
4120QM-SD14.880.50
8240RamoMark19.546.51
8240QM-SD29.490.86
8320RamoMark25.1310.60
8320QM-SD44.811.34

These results illustrate both a generation-time advantage in these runs and a substantial detection-time disadvantage. We will explicitly state that “negligible overhead” applies only to the generation-time randomness modulation, not to offline detection.

Weakness 2: Ablation and motivation for choosing θ\theta

We performed a matched five-value θ\theta ablation on 128 Qwen/CNN prompts, using the same gold messages and saved burn-in sequences for each choice. Invalid receiver replays count as failures.

θ\theta4-bit exact (n=120n=120)8-bit exact (n=240n=240)
0.500127/128126/128
0.625124/128127/128
0.750124/128126/128
0.875112/12899/128
0.95056/12839/128

Mean AATPS remains approximately 2.53–2.58 across the tested thresholds, but recovery deteriorates markedly at large θ\theta. These results show that the parameter matters and that the reference choice θ=0.75\theta=0.75 is not uniquely optimal in the measured configurations. We will include the recovery/AATPS ablation figure and describe θ=0.75\theta=0.75 as the selected operating point rather than an empirically proven optimum.

AuthorsOpenReview ↗

Responses to quality evaluation and robustness

Weakness 2: Quality evaluation beyond perplexity

We agree that perplexity alone is not sufficient to characterize generation quality. We have therefore also measured token Distinct-2, repeated 5-gram fraction and reference-token ROUGE-L, in addition to the original target-likelihood diagnostics. Our matched Qwen/CNN multi-bit study compares RamoMark with vanilla speculative sampling:

BitsMetricVanilla SDRamoMarkRamoMark − vanilla
4Distinct-2 (higher is better)0.96000.9540−0.0059
4Repeated 5-grams (lower is better)0.00190.0036+0.0018
4Reference-token ROUGE-L0.09630.0951−0.0012
8Distinct-20.92640.9323+0.0059
8Repeated 5-grams0.00360.0017−0.0019
8Reference-token ROUGE-L0.06860.0699+0.0013

These changes are comparatively small in the tested 4/8-bit settings, although the paired intervals do not establish equivalence. ROUGE-L is an overlap metric against available references, not a validated evaluation of semantic faithfulness for these continuations.

We also evaluated Qwen/CNN at 16 and 32 bits against QuantileMark, XMark and MPAC. This reveals a meaningful limitation at longer payloads:

BitsMethodDistinct-2 ↑Repeated 5-grams ↓
16RamoMark0.80570.0293
16QuantileMark-AR0.86310.0191
16XMark-AR0.89110.0100
16MPAC-AR0.87770.0133
32RamoMark0.74690.0580
32QuantileMark-AR0.80640.0384
32XMark-AR0.83750.0265
32MPAC-AR0.82380.0322

We will include the paired quality table/figure and the long-payload diversity–repetition comparison. The 500-prompt-per-cell one-bit experiments on Qwen and Vicuna additionally include processed LPPL, raw PPL, Distinct-2, repetition and top-1 diagnostics. We will not claim unchanged output quality at every message length.

Weakness 2: Robustness

We thank the reviewer for pointing out that the abstract's unqualified use of “robust” was not supported by edit-based experiments in the submitted paper. Our motivation for the word was information-theoretic: incorporating error-correcting codes into watermarking can improve robustness against text modifications [1, 2]. However, we agree that empirical validation is necessary and we did not inlcude them in the submitted manuscript. Our primary focus in the submitted work was instead to formulate speculative sampling as a communication channel and use channel codes to embed multi-bit watermarks.

To further investigate robustness, we evaluated context-addressed RamoMark with a robust likelihood decoder on Qwen2.5-7B/0.5B and CNN/DailyMail, considering 4-, 8-, 16-, and 32-bit watermarks under token substitution, deletion, insertion, and tail truncation at 1%, 5%, and 10% edit rates.

The following table reports exact message recovery (rather than bit-level recovery) over 32 prompts per setting, due to computational constraints. S/D/I denotes substitution/deletion/insertion.

BitsEdit rateAttackRamoMarkMPACXMark
41%D/I29/2829/2931/31
45%S/D/I19/26/2330/30/2530/31/31
81%D/I30/3123/2131/29
85%S/D/I17/27/2323/23/2427/30/27
161%D/I32/3217/2030/30
165%S/D/I12/18/1715/18/1526/29/30

These results show that RamoMark is not uniformly more robust than existing methods.
In the revised manuscript, we will include more robustness results. We will also revise the abstract to avoid an unqualified claim of robustness for the submitted scheme.

Importantly, robustness is only one of the objectives of our work. In our separate Qwen/CNN comparison with QuantileMark-SD, RamoMark achieves 19.7%–23.6% higher mean AATPS across six 4-/8-bit operating points. These results illustrate the sampling-efficiency benefit of incorporating watermarking directly into speculative sampling. Improving the robustness of our coding-theoretic speculative-sampling-based watermark remains an interesting future work.

[1] Qu, Wenjie, et al. “Provably Robust Multi-bit Watermarking for AI-generated Text.” 34th USENIX Security Symposium (USENIX Security 25), 2025.

[2] Luan, Xiaokun, et al. “Robust and Efficient Watermarking of Large Language Models Using Error Correction Codes.” Proceedings on Privacy Enhancing Technologies, 2025.

AuthorsOpenReview ↗

Responses to calibration, one-bit evaluation, other questions and conclusion

Weakness 2: Calibration and the auxiliary one-bit evaluation

We appreciate the reviewer pointing out that the one-bit experiment requires held-out calibration and is not automatically comparable with key-based detectors. We have now also evaluated a released-code adaptation of He et al.'s Gumbel/pseudorandom-acceptance method with the Ars-τ detector on six Qwen/Vicuna model–dataset settings, using 500 separate calibration and 500 test prompts per setting. He et al.'s observed test FPR ranges from 0.8% to 2.6% even though the operating threshold was selected for nominal 1% FPR.

ModelDatasetRamoMark TPR (%)He et al. TPR (%)RamoMark AATPSHe et al. AATPSHe test FPR (%)
QwenCNN/DailyMail88.055.02.60032.47611.6
QwenELI595.078.62.52662.28982.4
QwenGSM8K53.26.03.48334.11081.6
VicunaCNN/DailyMail19.224.62.06112.19232.6
VicunaELI538.053.21.87501.69512.0
VicunaGSM8K26.415.22.00981.87460.8

RamoMark has higher observed TPR on three Qwen datasets and Vicuna/GSM8. We will make these mixed results explicit, replace the phrase “improved watermark–efficiency frontier” with a narrower operating-point interpretation. We will include a two-model, six-cell TPR–AATPS comparison figure in the revision.

We do agree with the reviewer that we could move it to the appendix and instead have more in-depth evaluation of the multibit setting.

Question 2: Missing original prompt, hidden reasoning and agentic settings

Our detector is “prompt-free” only in the sense that it does not require the original private user prompt. It still relies on an adequate visible prefix to reconstruct the conditional target/draft likelihoods.
Robust watermarking of hidden-reasoning outputs or documents assembled from heterogeneous sources requires further study.

Concluding Remarks

We are grateful for the reviewer's practical perspective, which has helped us identify both stronger empirical evidence and meaningful limitations. We hope that the expanded ablations, quality measurements, cost analysis and more careful account of the supported settings substantially improve the paper. We would greatly appreciate it if the reviewer could revisit the evaluation in light of these new results and clarifications. Thank you again for the constructive feedback!

Reviewer_DkHSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multi-bit watermarking method integrated with speculative sampling. The key idea is to encode information by modulating the random acceptance threshold rather than directly modifying token probabilities. The authors formulate watermark embedding as a communication problem and use constant-composition codes for message encoding and likelihood-based decoding. The paper provides a theoretical result on preserving the target token distribution under a Bernoulli control input. Experiments on Qwen and Vicuna models show substantially higher 4-bit and 8-bit message recovery rates than MPAC and XMark, while maintaining sampling efficiency and output likelihood close to vanilla speculative sampling.

Critical strengths
  1. Using the acceptance threshold in speculative sampling as a communication channel is novel.
  2. RamoMark consistently outperforms MPAC and XMark in message recovery, especially for 8-bit payloads. The additional measurements of sampling efficiency and output quality make the results convincing.
Critical weaknesses
  1. The distribution-preservation guarantee is not fully justified for the proposed coding scheme. Theorem 4.1 assumes that the control bit follows a Bernoulli distribution at each context. However, constant-composition codewords only guarantee a fixed empirical proportion of zeros and ones. This does not establish the required conditional distribution given the generation history, so the claimed preservation of the full output distribution remains unclear.
  2. The practical robustness and scalability are insufficiently evaluated. Experiments only consider 4-bit and 8-bit payloads on unmodified generated text. There are no evaluations under paraphrasing, deletion, or other common text modifications. The claimed scalability to long bitstreams is also not experimentally demonstrated.
Questions for authors
  1. How does the constant-composition constraint guarantee distribution preservation when the control bits are correlated across positions and the token distributions depend on previous outputs?
  2. How sensitive is decoding accuracy to the length of the unwatermarked prefix used for prompt-free detection?
2 comments
AuthorsOpenReview ↗

Responses on distribution preservation and long bitstreams

We sincerely thank the Reviewer DkHS for the encouraging assessment of the acceptance-threshold communication-channel formulation and the reported multi-bit recovery gains over MPAC and XMark. We also appreciate the two important concerns about distribution preservation and the practical scope of our experiments. We have investigated longer payloads, text edits, and detector-side context sensitivity, and we clarify below which properties are established and which require a more restricted statement.

Weakness 1 / Question 1:Preservation of distribution

We thank the reviewer for pointing out this important distinction. We would like to clarify that our notion of unbiasedness refers to approximate rather than strict distribution preservation. Specifically, our constant-composition code ensures that the empirical proportions of zeros and ones in each codeword match the prescribed distribution. However, as the reviewer correctly noted, this does not guarantee exact preservation of the joint output distribution, since the control symbols may be correlated across positions.

Our theoretical result establishes an exact mixture identity at each fixed context, under the corresponding Bernoulli assumption on the control variable. We agree that constant composition alone does not ensure this assumption conditionally on the generation history. We will clarify that our coding construction is intended to achieve approximate unbiasedness, rather than the strict distribution-preservation guarantee commonly considered in watermarking literature, and revise the theoretical statements and related claims accordingly.

Furthermore, our empirical evaluations, including perplexity, token diversity, and repetition rates, provide additional evidence that RamoMark introduces relatively limited degradation in text quality under the tested settings, although these measurements do not establish exact distribution preservation.

We thank the reviewer again for this valuable observation, which will help us improve the precision and presentation of our theoretical results.

Weakness 2: Scalability beyond 4/8 bits

We extended the construction to 16- and 32-bit payloads using an outer Reed–Solomon (RS) code with soft inner decoding. On 64 Qwen2.5-7B/0.5B CNN/DailyMail prompts, we obtained the following clean message recovery results:

BitsWatermarked tokenstreamRamoMarkXMarkMPAC
1676864/6462/6442/64
32115261/6435/648/64

At 32 bits, RamoMark outperforms XMark-AR and MPAC-AR in exact message recovery. The 16/32-bit results support practical feasibility at longer payload lengths, rather than unlimited scalability. Furthermore, our codebooks can be constructed offline and reused, reducing online encoding overhead.

AuthorsOpenReview ↗

Responses on Robustness, Sensitivity to Prefix,

Weakness 2: Robustness

We thank the reviewer for highlighting the lack of robustness experiments in the submitted manuscript. Our use of the term “robust” in abstract was motivated by an information-theoretic perspective, as incorporating error-correcting codes into watermarking can improve robustness against text modifications [1, 2]. However, we agree that this motivation should be supported by empirical evaluations, which were not included in our original submission. Our main focus was instead on formulating speculative sampling as a communication channel and exploiting channel coding to embed multi-bit watermarks.

To further investigate robustness, we evaluated a context-addressed RamoMark with a robust likelihood decoder using Qwen2.5-7B/0.5B on CNN/DailyMail. We considered 4-, 8-, 16-, and 32-bit watermarks under token substitution, deletion, insertion, and tail truncation at edit rates of 1%, 5%, and 10%.

The following table presents exact message recovery (rather than bit-level recovery) over 32 prompts per setting, due to computational constraints. D/I denotes deletion/insertion, while S/D/I denotes substitution/deletion/insertion.

BitsEdit rateAttackRamoMarkMPACXMark
41%D/I29/2829/2931/31
45%S/D/I19/26/2330/30/2530/31/31
81%D/I30/3123/2131/29
85%S/D/I17/27/2323/23/2427/30/27
161%D/I32/3217/2030/30
165%S/D/I12/18/1715/18/1526/29/30

The results show that RamoMark achieves promising robustness under low-rate insertion and deletion attacks, particularly for longer payloads, where it outperforms both MPAC and XMark. Nevertheless, XMark generally performs better under stronger edits, indicating that RamoMark does not uniformly outperform existing methods in robustness. We will include these additional evaluations in the revised manuscript and clarify the scope of our robustness claims in the abstract.

We would also like to emphasize that robustness is not the sole objective of our work. Our main contribution is to integrate multi-bit watermarking into speculative sampling while retaining its sampling efficiency. In a separate Qwen/CNN comparison with QuantileMark-SD, RamoMark achieves 19.7%–23.6% higher mean AATPS across six 4-/8-bit operating points, demonstrating the advantage of embedding watermarks directly into the speculative sampling procedure. We believe that both watermarking performance and sampling efficiency should be considered when evaluating our approach.

Further improving the robustness of our coding-theoretic watermarking framework, particularly against stronger text modifications, remains an interesting direction for future work.

[1] Qu, Wenjie, et al. “Provably Robust Multi-bit Watermarking for AI-generated Text.” 34th USENIX Security Symposium (USENIX Security 25), 2025.

[2] Luan, Xiaokun, et al. “Robust and Efficient Watermarking of Large Language Models Using Error Correction Codes.” Proceedings on Privacy Enhancing Technologies, 2025.

Question 2: Sensitivity to the length of the unwatermarked prefix

We agree that “prompt-free detection” needs clarification. It means that our receiver does not require the original private user prompt; it does not mean that it can reconstruct the token likelihoods without sufficient visible conditioning history.

We performed an N=128N=128 Qwen receiver-side ablation in which the watermarked continuations were generated with their original complete burn-in, but the detector was allowed to observe only a shorter suffix:

BitsReceiver burn-in (tokens)Exact recovery counts (same order)
432, 64, 96, 1202/128, 10/128, 34/128, 124/128
832, 64, 128, 2400/128, 0/128, 3/128, 126/128

The results demonstrate a meaningful dependence on reconstructing the original conditional history. We will include the corresponding prefix-sensitivity figure and restrict our claims to settings with an adequate visible prefix.

We believe the new 16/32-bit experiments and edit/context ablations considerably strengthen the empirical understanding of when RamoMark is effective. Equally importantly, the revised theoretical interpretation will make the guarantees precise. We hope these additions and clarifications help resolve the reviewer's concerns, and we would sincerely appreciate a reconsideration of the assessment in light of the expanded evidence. Thank you for your valuable suggestions.

Reviewer_UvgmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RamoMark, a multi-bit watermark for speculative sampling that modulates the acceptance threshold uu rather than token selection: a control bit AA determines whether u∼Unif[0,θ]u \sim \mathrm{Unif}[0, \theta] or u∼Unif[θ,1]u \sim \mathrm{Unif}[\theta, 1], turning verification into a channel from AA to the output token, over which messages are sent with an offline constant-composition codebook (a fraction θ\theta of zeros, motivated by a fixed-context mixture identity for A∼Bernoulli(1−θ)A \sim \mathrm{Bernoulli}(1-\theta)) and recovered by a keyless likelihood decoder that uses the visible generated prefix in place of the original prompt. On Qwen2.5-7B/0.5B and Vicuna-7B/68M with 4- and 8-bit payloads, it reports substantially higher exact recovery than autoregressive MPAC and XMark (about 95% at n=128n = 128–512512) with AATPS and target likelihood close to vanilla speculative sampling, and a one-bit variant is compared with MSE and MWS (Hu & Huang, 2024).

Critical strengths

The acceptance-threshold formulation is a simple, elegant way to embed a payload within an already widely deployed acceleration step: it admits an exact target-preserving mixture at a fixed context, changes only a small part of the sampling procedure, and keeps AATPS near vanilla, offering a promising route to combining inference acceleration with multi-bit provenance, which is the main source of the paper's value. Empirically, the reported recovery gains over the evaluated autoregressive baselines are large (about 10–32 points over XMark and 29–67 over MPAC at the operating points highlighted in the text), and at the selected operating points in Table 1 the confidence intervals do not resolve any degradation in AATPS or raw target likelihood (though this does not establish equivalence); the margin is wide enough that I consider the method's practical effectiveness plausible despite the reporting concerns below.

Critical weaknesses

The paper is not adequately positioned against its closest prior work: QuantileMark (interval-based embedding) is cited only among multi-bit methods that the introduction argues fail under naive integration, without substantive comparison, and He et al. (2026b), which also injects pseudorandomness into draft-token acceptance, is not evaluated in Section 5.2 even though that section claims to improve the watermark–efficiency frontier; in addition, the abstract's "robust" claim is not supported by any evaluation under substitution, insertion/deletion, or paraphrasing, a natural concern since the decoder aligns observed and codeword positions without synchronization, which together with the issues below leaves the novelty margin and the scope of the claims difficult to assess and weighs most heavily against acceptance.

The theoretical and empirical claims also exceed what is documented: Theorem 4.1 is a correct fixed-context mixture identity, but constant composition does not ensure this mixture conditional on each realized generation history (in a toy example with θ=1/2\theta = 1/2 and codewords {01,10}\{01, 10\}, per-position marginals match the target while the joint distribution does not), so the abstract's "provably preserves" and the promised long-sequence guarantee are not established beyond the authors' own "approximately unbiased" framing; moreover, Figure 3 states N=100N = 100 prompts per method and blocklength, yet many entries in Tables 3–4 (e.g., 98.84%, 87.21%) are incompatible with a simple success count out of 100, the parenthesized RamoMark trial counts promised in the captions are absent, and Table 1 reports no nn or NN, with three of its four values not matching the corresponding series in Tables 3–4.

Questions for authors
  1. Figure 3 states N=100N = 100 per method and blocklength, but many recovery values in Tables 3–4 cannot arise from a simple count out of 100, and the parenthesized RamoMark counts are missing. Please report success counts, evaluated sample sizes, any aggregation rule, and whether methods use matched prompt sets. Given that Appendix B counts invalid replays as failures, please clarify whether any trials are removed before forming the "available self-prompt" cohort (e.g., when a payload block cannot be completed). Please also state the nn, NN, and cohort behind Table 1, and the self-prompt length (total visible token budget) relative to the baselines. If removals are outcome-dependent or materially change the results, my soundness assessment would drop; if they reflect benign filtering or reporting errors, it would improve.

  2. Relation to the closest prior work.** Does the rejection-stage limitation argued in the introduction also apply to an adaptation of quantile-based partitioning that modulates both draft and residual sampling, and what coupling or efficiency constraints distinguish such an adaptation from RamoMark? Since Section 5.2 claims an improved frontier, please compare with He et al. (2026b) in the one-bit setting, or explain precisely why the settings are not comparable. A clear distinction and this comparison would raise my contribution assessment.

  3. What does "robust" refer to in the abstract? Please report basic results under token substitution, insertion/deletion, truncation, or paraphrasing, or restrict the claim accordingly. Either would address this concern; a synchronization-tolerant decoder can be left to future work.

4 comments
AuthorsOpenReview ↗

Response to the Comparison with QuantileMark

We sincerely thank the reviewer Uvgm for the careful assessment and constructive suggestions. We especially appreciate the recognition of our acceptance-threshold formulation, the coding-theoretic perspective, and the promising combination of multi-bit provenance with speculative sampling. We agree that our original submission needed a more substantive comparison with the closest prior work, a clearer evaluation of robustness, and more transparent numerical reporting.

We have conducted substantially additional experiments addressing these points and explain below both what the new evidence establishes and where the original claims should be restricted.

Weakness 1 / Question 2: Comparison with QuantileMark and the rejection-stage limitation

We thank the reviewer for highlighting QuantileMark. We admit that it is an important and closely related multi-bit watermarking baseline. Our argument about the rejection stage concerns a naive integration that embeds information only in the draft proposals while leaving the correction distribution message-independent: rejection can then erase the signal carried by the proposal. It does not imply that quantile-based watermarking cannot be combined with speculative sampling. In particular, if both the draft proposal and the rejection correction are constructed consistently for the same message-dependent target distribution, standard speculative sampling can reproduce that distribution.

To test this stronger alternative as suggested by the reviewer, we implemented QuantileMark-SD, which combines QuantileMark with speculative decoding. Unlike merely comparing against autoregressive QuantileMark, this directly evaluates message recovery and accepted output tokens per target verification step (AATPS).

We did the comparison via 128 paired Qwen2.5-7B/0.5B prompts on CNN/DailyMail at each of six 4/8-bit operating points. We chose only 128 prompts due to the limit of our computation resource and time. We will implement a more comprehensive experiment in the revised version.

BitsPayload tokensRamoMark recoveryQM-SD recoveryRamoMark AATPSQM-SD AATPSAATPS gain
496120/128128/1282.5242.088+20.9%
4120124/128128/1282.5812.089+23.6%
4160127/128128/1282.5552.089+22.3%
8192125/128128/1282.5662.089+22.8%
8240126/128128/1282.5262.110+19.7%
8320126/128128/1282.5582.136+19.7%

The AATPS advantage is 19.7%–23.6% across these six configurations. All six prompt-paired bootstrap 95% intervals for the absolute AATPS difference are positive (differences range from approximately 0.417 to 0.492 accepted tokens per target verification step). Nevertheless, QuantileMark-SD consistently obtains highe watermark recovery rate.
Therefore, unlike our improvements over MPAC and XMark, here we do not claim that RamoMark dominates QuantileMark on both objectives. Rather, the two schemes exhibit different recovery–efficiency operating points, reflecting the distinction between modulating the speculative acceptance randomness and sampling from a QuantileMark-dependent target distribution.

We will add a complete table and a figure showing the six clean recovery/AATPS operating points in the paper. We will also clarify the scope of the rejection-stage argument in the introduction.

AuthorsOpenReview ↗

Response to the Comparison with He et al. (2026)

Weakness 1 / Question 2: Comparison with He et al. (2026) in the one-bit setting

We agree that He et al. is important prior work for our auxiliary one-bit evaluation. We implemented a Gumbel-watermark/pseudorandom-acceptance variant with the Ars-τ detector based on their paper, adapting the released code to our declared top-k=50k=50, temperature-11, four-token lookahead and fixed 128-token observation window. Each model–dataset cell includes 500 held-out test prompts and 500 separate calibration prompts. The resulting six-cell measurements are:

ModelDatasetRamoMark TPR (%)He et al. TPR (%)RamoMark AATPSHe et al. AATPSHe test FPR (%)
QwenCNN/DailyMail88.055.02.60032.47611.6
QwenELI595.078.62.52662.28982.4
QwenGSM8K53.26.03.48334.11081.6
VicunaCNN/DailyMail19.224.62.06112.19232.6
VicunaELI538.053.21.87501.69512.0
VicunaGSM8K26.415.22.00981.87460.8

RamoMark has higher observed TPR on three Qwen datasets and Vicuna/GSM8. We will make these mixed results explicit, replace the phrase “improved watermark–efficiency frontier” with a narrower operating-point interpretation. We will include a two-model, six-cell TPR–AATPS comparison figure in the revision.

Weakness 2 / Question 3: Robustness to text edits and the scope of “robust”

We thank the reviewer for pointing out that the abstract's unqualified use of “robust” was not supported by edit-based experiments in the submitted paper. Our motivation for the word was information-theoretic: incorporating error-correcting codes into watermarking can improve robustness against text modifications [1, 2]. However, we agree that empirical validation is necessary and we did not inlcude them in the submitted manuscript. Our primary focus in the submitted work was instead to formulate speculative sampling as a communication channel and use channel codes to embed multi-bit watermarks.

To further investigate robustness, we evaluated context-addressed RamoMark with a robust likelihood decoder on Qwen2.5-7B/0.5B and CNN/DailyMail, considering 4-, 8-, 16-, and 32-bit watermarks under token substitution, deletion, insertion, and tail truncation at 1%, 5%, and 10% edit rates.

The following table reports exact message recovery (rather than bit-level recovery) over 32 prompts per setting, due to computational constraints. S/D/I denotes substitution/deletion/insertion.

BitsEdit rateAttackRamoMarkMPACXMark
41%D/I29/2829/2931/31
45%S/D/I19/26/2330/30/2530/31/31
81%D/I30/3123/2131/29
85%S/D/I17/27/2323/23/2427/30/27
161%D/I32/3217/2030/30
165%S/D/I12/18/1715/18/1526/29/30

These results show that RamoMark is not uniformly more robust than existing methods.
In the revised manuscript, we will include more robustness results. We will also revise the abstract to avoid an unqualified claim of robustness for the submitted scheme.

Importantly, robustness is only one of the objectives of our work. In our separate Qwen/CNN comparison with QuantileMark-SD, RamoMark achieves 19.7%–23.6% higher mean AATPS across six 4-/8-bit operating points. These results illustrate the sampling-efficiency benefit of incorporating watermarking directly into speculative sampling. Improving the robustness of our coding-theoretic speculative-sampling-based watermark remains an interesting future work.

[1] Qu, Wenjie, et al. “Provably Robust Multi-bit Watermarking for AI-generated Text.” 34th USENIX Security Symposium (USENIX Security 25), 2025.

[2] Luan, Xiaokun, et al. “Robust and Efficient Watermarking of Large Language Models Using Error Correction Codes.” Proceedings on Privacy Enhancing Technologies, 2025.

AuthorsOpenReview ↗

Response to the Figure 3 denominators, self-prompt, and distribution preservation

Weakness 3 / Question 1: Figure 3 denominators, self-prompt selection and Table 1

We appreciate the reviewer identifying these inconsistencies. We agree that entries such as 98.84% cannot be interpreted as integer success counts out of the captioned N=100N=100 trials.
We here report explicit denominators and distinguish the experimental cohorts:

ExperimentModel/datasetPayload lengthReceiver-visible burn-inAttempted promptsRamoMark recovery
Formal 4-bit caseQwen/CNN120120128124/128
Formal 8-bit caseQwen/CNN240240128126/128
RS-coded 16-bit caseQwen/CNN7681926464/64
RS-coded 32-bit caseQwen/CNN11521926461/64

For these configurations, the total receiver-visible lengths (burn-in plus payload) are, respectively, 240, 480, 960 and 1344 tokens. The six-row 128-prompt QuantileMark-SD comparison above likewise uses shared source prompt.

We will replace or explicitly update any original numerical series whose numerator, denominator, self-prompt eligibility rule or exact cohort cannot be verified. We will correct the N=100N=100 caption, add nn, NN, burn-in and total visible length to the relevant tables, and state whether a comparison uses shared prompts or independent generation cohorts.

Weakness 3: Scope of the distribution-preservation guarantee

We thank the reviewer for pointing out this important distinction. We would like to clarify that our notion of unbiasedness refers to an approximate rather than strict distribution preservation. Specifically, our constant-composition code ensures that the proportions of zeros and ones in each codeword match the prescribed distribution over the block. However, as the reviewer correctly noted, this does not guarantee exact preservation of the joint output distribution, which requires a stronger conditional independence property.

Though we have indicated this in the paper, we will surely clarify this distinction in the revised manuscript more precisely and qualify our theoretical claims accordingly. Furthermore, our empirical evaluations, including perplexity, token diversity, repetition rates, and other quality metrics, provide additional evidence that RamoMark introduces relatively limited degradation in text quality under the tested settings, although these measurements do not establish exact distribution preservation.

Conclusion.

We hope that our additional experiments, clarifications, and corrections satisfactorily address the reviewer's main concerns. We remain confident in the value of our channel-coding approach to multi-bit watermarking and would be very grateful if the reviewer could kindly reconsider their evaluation in light of these new results. Thank you again for your thoughtful review!

I thank the authors for the detailed rebuttal and the substantial new experiments. My comments are based on the rebuttal; where the revised PDF already addresses a point, a pointer to the relevant section would suffice.

[Resolved or largely resolved]

  1. QuantileMark (Q2). The QuantileMark-SD comparison and the clarification that the rejection-stage argument concerns only naive integration address my positioning concern. The results show a trade-off rather than dominance: QuantileMark-SD reaches 128/128128/128 recovery in all six settings, while RamoMark has 19.719.7–23.6%23.6\% higher AATPS and shorter generation time (9.209.20 s vs. 14.8814.88 s at 4 bits). I agree the paper should be framed around this operating point.

  2. He et al. (2026b) (Q2). The six-cell comparison is provided, with mixed results (RamoMark has higher TPR in four of six cells), and the "improved frontier" claim will be narrowed. One follow-up remains (C).

  3. Robustness (Q3). The authors provide both basic results and a restricted claim, which meets my stated criterion. RamoMark is competitive at 1%1\% insertion/deletion but weaker than XMark at 5%5\% (e.g., 12/18/1712/18/17 vs. 26/29/3026/29/30 out of 3232 at 16 bits). Please also report the announced 10%10\% and truncation settings and describe the robust decoder, which is not in the submitted manuscript.

  4. Distribution preservation. I welcome the acknowledgment that Theorem 4.1 is a fixed-context identity that constant composition does not guarantee along the generation history. Please revise the abstract, the opening of Section 4, and the conclusion accordingly; the abstract on OpenReview still says "robust" and "provably preserves." The new 16/32-bit results, with roughly 1.51.5–3×3\times the baselines' repeated 5-gram rate, are consistent with this concern; a vanilla reference at these lengths would help. The θ\theta ablation also shows AATPS is insensitive to θ\theta (2.532.53–2.582.58), as identities (12)–(13) imply, and θ=0.5\theta = 0.5 does at least as well as 0.750.75, so the efficiency rationale in Remark 4.2 should be corrected.

[Remaining concerns]

A. Burn-in dependence. Recovery jumps sharply only when the detector sees the complete burn-in: 34/128→124/12834/128 \to 124/128 (4 bits, 96→12096 \to 120 tokens) and 3/128→126/1283/128 \to 126/128 (8 bits, 128→240128 \to 240), whereas 16/32 bits succeed with a 192192-token burn-in. This is hard to reconcile with the gradual context-absorption argument in Section 4.2.

  • Is the watermarked block generated from (user prompt, burn-in) or from the burn-in alone? If the latter, generator and detector contexts match by construction, and the prompt-free setting is not tested. Please report recovery when the payload is generated from prompt plus burn-in and decoded from the burn-in alone.
  • With burn-in, the total visible length is 240240 and 480480 tokens for 4/8 bits. At similar lengths, the original Tables 3–4 report XMark at 99.20%99.20\% (n=256n = 256) and 99.18%99.18\% (n=512n = 512), versus RamoMark's 124/128124/128 and 126/128126/128. Please compare at matched total length, or use the preceding block as context to amortize the burn-in.

B. Original results. The verified Qwen/CNN counts (124/128124/128, 126/128126/128) broadly match the original Qwen levels. However, the rebuttal does not explain the non-integer percentages, any pre-cohort removals, or the nn and NN behind Table 1. Which original series are retained in the revision, and are verified Vicuna results, where the largest gains were reported, included?

C. He et al. comparability. The He et al. variant's test FPR exceeds 1%1\% in five of six cells (0.80.8–2.6%2.6\%), and its AATPS deviates from vanilla by −10%-10\% to +18%+18\% (e.g., 4.11084.1108 vs. 3.48343.4834 on Qwen/GSM8K with L=4L = 4), although no target-preserving rule can exceed the per-step acceptance of vanilla, 1−TV(Q,P)1 - \mathrm{TV}(Q, P). Also, all twelve RamoMark entries are identical to Table 5, where no separate calibration split is described, while the He et al. variant uses 500500 calibration and 500500 test prompts. Were both methods evaluated on the same test prompts and step-counting rule, and what is RamoMark's empirical test FPR under this protocol?

[Updated assessment]

The rebuttal resolves my concerns about positioning and the scope of the robustness and theoretical claims, and clarifies the contribution as a distinct, efficiency-favorable operating point rather than dominance over prior work. However, the required revision is substantial (replacing main results, narrowing the abstract, and noting that decoding is about 6.56.5–8×8\times slower than QuantileMark-SD), and concern A bears directly on the prompt-free claim and the headline gains. I keep my rating at Weak rejection for now.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revision

We sincerely thank the Area Chair and all reviewers for their careful evaluations, constructive comments, and valuable suggestions.

In response, we have provided detailed point-by-point replies to each reviewer and conducted substantial additional experiments to address the main concerns. Our key additions and clarifications include:

  1. Comparison with closely related work: We implemented QuantileMark-SD and conducted a new comparison, demonstrating 19.7%–23.6% higher AATPS for RamoMark, while acknowledging QuantileMark's slightly higher watermark message recovery. We also extended our one-bit evaluation to He et al. (2026) across six model-dataset settings.

  2. Robustness evaluation: We evaluated token substitution, deletion, insertion, and truncation at different edit rates. The results demonstrate promising robustness under low-rate edits, while also identifying limitations under stronger attacks. Improving the watermark robustness while maintaining our strong speculation performance is an interesting future direction.

  3. Scalability to longer watermarks: We extended RamoMark to 16- and 32-bit messages using Reed–Solomon coding, achieving desirable recovery results.

  4. Additional ablations and quality evaluations: We investigated the choice of θ\theta, sensitivity to the unwatermarked prefix, variable-length observations, and decoding overhead. We also extended the quality evaluation beyond perplexity to include token diversity, repetition rates, and ROUGE-L.

  5. Theoretical and experimental clarifications: We clarified the distinction between approximate unbiasedness and strict distribution preservation, the scope of our theoretical guarantees, and the experimental denominators and evaluation protocols. We will incorporate these clarifications and the new results into the revised manuscript.

We believe these additions substantially strengthen the empirical evidence and clarify the practical scope of our contributions. In particular, our central contribution is to formulate speculative sampling as a communication channel and enable multi-bit watermarking through channel coding, while retaining the sampling-efficiency benefits of speculative decoding.

We sincerely hope that our responses and new results satisfactorily address the reviewers' main concerns. We warmly welcome any further questions or discussion from the Area Chair and reviewers, and would be very grateful if the reviewers could kindly reconsider their evaluations, including their ratings, in light of the additional evidence and clarifications.

Thank you again for your time, thoughtful feedback, and consideration!

AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Revisions for the Area Chair

Dear Area Chair,

As the discussion period concludes, we would like to briefly summarize our main contributions and the additional evidence provided during the rebuttal to facilitate your assessment.

1. Main Contributions and Novelty

Our central contribution is a novel communication-theoretic framework that unifies multi-bit watermarking and speculative sampling within a single algorithm.

  • Novel communication channel formulation: We identify the intrinsic randomness of speculative verification—the random acceptance threshold—as a medium for information embedding. To our knowledge, this is the first work to formulate speculative sampling as a communication channel and exploit coding theory for multi-bit watermarking. It is quite interesting to us, and we believe this opens up opportunities for both communities.

  • Coding-theoretic watermarking: We develop RamoMark using offline-constructed, reusable codebooks. More importantly, our channel formulation is not restricted to this particular coding scheme, opening up opportunities to leverage advances in coding theory for watermarking.

2. Main Additions and Clarifications During Rebuttal

We have substantially revised the manuscript, with changes highlighted in blue, and conducted extensive additional experiments.

  • Comparison with closely related work: Implemented QuantileMark-SD and demonstrated 19.7%–23.6% higher AATPS across six multi-bit configurations, with a modest trade-off in watermark recovery. We also conducted additional comparisons with He et al. (2026) across six model–dataset settings.

  • Scalability: Extended RamoMark to 16- and 32-bit watermarks using Reed–Solomon coding, achieving 64/64 and 61/64 exact message recovery, respectively.

  • Output quality: Conducted additional diversity, repetition, and ROUGE-L evaluations. We explicitly acknowledge the limitations under stronger edits and longer payloads.

  • Theoretical and practical clarifications: Distinguished exact fixed-context distribution preservation from the empirical behavior of constant-composition coding, investigated threshold and prefix sensitivity, reported detection costs, and clarified compatibility with newer drafting architectures whenever standard probabilistic verification is retained.

We particularly appreciate Reviewer DkHS's recognition of the novelty of our communication-channel formulation and Reviewer Uvgm's acknowledgment that our rebuttal has resolved their concerns regarding positioning and the scope of our theoretical and robustness claims.

While detection complexity, robustness, and more flexible coding remain important directions for improvement, we believe the principal value of RamoMark lies in introducing a new information- and coding-theoretic perspective on jointly achieving multi-bit watermarking and efficient LLM inference, rather than merely developing another watermarking algorithm.

We sincerely thank the Area Chair and all reviewers for their time, constructive feedback, and consideration.

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.