- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes RamoMark, a mutilbit watermarking scheme that leverages speculative decoding acceptance threshold as a channel to embed a multibit watermark. In particular, the authors highlight that a given token distribution is different whether it is accepted (in that case this is the draft model distribution) or rejected (in that case it is the residual distribution). By conditioning the sampling of the acceptance threshold uniform variable on a binary variable encoding the message, RamoMark can therefore encode the message in the generated token sequence.
Specifically, given a generation length and a message , RamoMark encodes in a sequence of bits with a ratio of zeros (where is a parameter of the scheme). When sampling the token at position , if the -th bit is then the acceptance threshold is sampled in and else in . For detection, given a generated text, RamoMark uses an LLM forward pass to compute an approximate maximum likelihood message.
Their results show that RamoMark outperforms prior multibit watermark schemes, with minimal impact on average token acceptance. They also show that their approach can be adapted for zero-bit detection.
Critical strengths
- The conceptual idea behind the method is clever, easy-to-understand and appears to work relatively well in practice. In particular, unlike many prior multibit watermarking works, it does not rely on position allocation but uses a novel mechanism (i.e. the speculative decoding threshold)
- The paper is easy to follow
Critical weaknesses
- The method appears to have several practical limitations not well highlighted in the paper. RamoMark assumes that the number of generated tokens is known in advance. In practice, this is not the case. RamoMark requires speculative decoding to be "turn on" all the times: in practice, depending on requests batch sizes, model providers can dynamically decide when to use speculative decoding. RamoMark detection, unlike most prior works, is expensive as it requires a model forward pass. Lastly, RamoMark has been evaluated only with a simple draft model speculative decoding approach and it remains unclear if it works with newer speculative decoding variants (e.g. MTP, DFlash, DTrees...).
- The evaluation section is very sparse. In particular, there is no robustness evaluation. The scheme parameter is not ablated and the motivation behind its choice in remark 4.2 is not evaluated empirically. The quality evaluation is very sparse (only PPL metrics). The zerobit evaluation is slightly disingenuous as RamoMark in the zero-bit setting requires held-out calibration unlike most baselines, and does not add significant values to the paper (I would move it to the appendix and e.g. instead have more in-depth evaluation of the multibit setting). Lastly, the writing in the evaluation section is poor as it only describes the figures/table values and does not give real interpretation/explanation of the results.
Questions for authors
- How could RamoMark work in a setting where the generated text length is not known in advance?
- I was wondering if RamoMark would work in settings where the estimated probability without the prompt differ significantly from the probabilities at generation time? I was thinking of thinking models without the thinking trace, or agentic settings (e.g. a codebase where some of the code/documentation is AI-written)?
5 comments
Responses to quality evaluation and robustness
Weakness 2: Quality evaluation beyond perplexity
We agree that perplexity alone is not sufficient to characterize generation quality. We have therefore also measured token Distinct-2, repeated 5-gram fraction and reference-token ROUGE-L, in addition to the original target-likelihood diagnostics. Our matched Qwen/CNN multi-bit study compares RamoMark with vanilla speculative sampling:
| Bits | Metric | Vanilla SD | RamoMark | RamoMark − vanilla |
|---|---|---|---|---|
| 4 | Distinct-2 (higher is better) | 0.9600 | 0.9540 | −0.0059 |
| 4 | Repeated 5-grams (lower is better) | 0.0019 | 0.0036 | +0.0018 |
| 4 | Reference-token ROUGE-L | 0.0963 | 0.0951 | −0.0012 |
| 8 | Distinct-2 | 0.9264 | 0.9323 | +0.0059 |
| 8 | Repeated 5-grams | 0.0036 | 0.0017 | −0.0019 |
| 8 | Reference-token ROUGE-L | 0.0686 | 0.0699 | +0.0013 |
These changes are comparatively small in the tested 4/8-bit settings, although the paired intervals do not establish equivalence. ROUGE-L is an overlap metric against available references, not a validated evaluation of semantic faithfulness for these continuations.
We also evaluated Qwen/CNN at 16 and 32 bits against QuantileMark, XMark and MPAC. This reveals a meaningful limitation at longer payloads:
| Bits | Method | Distinct-2 ↑ | Repeated 5-grams ↓ |
|---|---|---|---|
| 16 | RamoMark | 0.8057 | 0.0293 |
| 16 | QuantileMark-AR | 0.8631 | 0.0191 |
| 16 | XMark-AR | 0.8911 | 0.0100 |
| 16 | MPAC-AR | 0.8777 | 0.0133 |
| 32 | RamoMark | 0.7469 | 0.0580 |
| 32 | QuantileMark-AR | 0.8064 | 0.0384 |
| 32 | XMark-AR | 0.8375 | 0.0265 |
| 32 | MPAC-AR | 0.8238 | 0.0322 |
We will include the paired quality table/figure and the long-payload diversity–repetition comparison. The 500-prompt-per-cell one-bit experiments on Qwen and Vicuna additionally include processed LPPL, raw PPL, Distinct-2, repetition and top-1 diagnostics. We will not claim unchanged output quality at every message length.
Weakness 2: Robustness
We thank the reviewer for pointing out that the abstract's unqualified use of “robust” was not supported by edit-based experiments in the submitted paper. Our motivation for the word was information-theoretic: incorporating error-correcting codes into watermarking can improve robustness against text modifications [1, 2]. However, we agree that empirical validation is necessary and we did not inlcude them in the submitted manuscript. Our primary focus in the submitted work was instead to formulate speculative sampling as a communication channel and use channel codes to embed multi-bit watermarks.
To further investigate robustness, we evaluated context-addressed RamoMark with a robust likelihood decoder on Qwen2.5-7B/0.5B and CNN/DailyMail, considering 4-, 8-, 16-, and 32-bit watermarks under token substitution, deletion, insertion, and tail truncation at 1%, 5%, and 10% edit rates.
The following table reports exact message recovery (rather than bit-level recovery) over 32 prompts per setting, due to computational constraints. S/D/I denotes substitution/deletion/insertion.
| Bits | Edit rate | Attack | RamoMark | MPAC | XMark |
|---|---|---|---|---|---|
| 4 | 1% | D/I | 29/28 | 29/29 | 31/31 |
| 4 | 5% | S/D/I | 19/26/23 | 30/30/25 | 30/31/31 |
| 8 | 1% | D/I | 30/31 | 23/21 | 31/29 |
| 8 | 5% | S/D/I | 17/27/23 | 23/23/24 | 27/30/27 |
| 16 | 1% | D/I | 32/32 | 17/20 | 30/30 |
| 16 | 5% | S/D/I | 12/18/17 | 15/18/15 | 26/29/30 |
These results show that RamoMark is not uniformly more robust than existing methods.
In the revised manuscript, we will include more robustness results.
We will also revise the abstract to avoid an unqualified claim of robustness for the submitted scheme.
Importantly, robustness is only one of the objectives of our work. In our separate Qwen/CNN comparison with QuantileMark-SD, RamoMark achieves 19.7%–23.6% higher mean AATPS across six 4-/8-bit operating points. These results illustrate the sampling-efficiency benefit of incorporating watermarking directly into speculative sampling. Improving the robustness of our coding-theoretic speculative-sampling-based watermark remains an interesting future work.
[1] Qu, Wenjie, et al. “Provably Robust Multi-bit Watermarking for AI-generated Text.” 34th USENIX Security Symposium (USENIX Security 25), 2025.
[2] Luan, Xiaokun, et al. “Robust and Efficient Watermarking of Large Language Models Using Error Correction Codes.” Proceedings on Privacy Enhancing Technologies, 2025.
Responses to calibration, one-bit evaluation, other questions and conclusion
Weakness 2: Calibration and the auxiliary one-bit evaluation
We appreciate the reviewer pointing out that the one-bit experiment requires held-out calibration and is not automatically comparable with key-based detectors. We have now also evaluated a released-code adaptation of He et al.'s Gumbel/pseudorandom-acceptance method with the Ars-τ detector on six Qwen/Vicuna model–dataset settings, using 500 separate calibration and 500 test prompts per setting. He et al.'s observed test FPR ranges from 0.8% to 2.6% even though the operating threshold was selected for nominal 1% FPR.
| Model | Dataset | RamoMark TPR (%) | He et al. TPR (%) | RamoMark AATPS | He et al. AATPS | He test FPR (%) |
|---|---|---|---|---|---|---|
| Qwen | CNN/DailyMail | 88.0 | 55.0 | 2.6003 | 2.4761 | 1.6 |
| Qwen | ELI5 | 95.0 | 78.6 | 2.5266 | 2.2898 | 2.4 |
| Qwen | GSM8K | 53.2 | 6.0 | 3.4833 | 4.1108 | 1.6 |
| Vicuna | CNN/DailyMail | 19.2 | 24.6 | 2.0611 | 2.1923 | 2.6 |
| Vicuna | ELI5 | 38.0 | 53.2 | 1.8750 | 1.6951 | 2.0 |
| Vicuna | GSM8K | 26.4 | 15.2 | 2.0098 | 1.8746 | 0.8 |
RamoMark has higher observed TPR on three Qwen datasets and Vicuna/GSM8. We will make these mixed results explicit, replace the phrase “improved watermark–efficiency frontier” with a narrower operating-point interpretation. We will include a two-model, six-cell TPR–AATPS comparison figure in the revision.
We do agree with the reviewer that we could move it to the appendix and instead have more in-depth evaluation of the multibit setting.
Question 2: Missing original prompt, hidden reasoning and agentic settings
Our detector is “prompt-free” only in the sense that it does not require the original private user prompt. It still relies on an adequate visible prefix to reconstruct the conditional target/draft likelihoods.
Robust watermarking of hidden-reasoning outputs or documents assembled from heterogeneous sources requires further study.
Concluding Remarks
We are grateful for the reviewer's practical perspective, which has helped us identify both stronger empirical evidence and meaningful limitations. We hope that the expanded ablations, quality measurements, cost analysis and more careful account of the supported settings substantially improve the paper. We would greatly appreciate it if the reviewer could revisit the evaluation in light of these new results and clarifications. Thank you again for the constructive feedback!
Responses to unknown generation length and omputational cost
We sincerely thank the reviewer for recognizing the novelty and simplicity of using the speculative-sampling acceptance threshold as a watermarking channel, and for the thoughtful questions about deployment. We agree that the original evaluation placed too much emphasis on clean recovery and did not sufficiently explain several practical limitations. We have now conducted additional ablations and quality/robustness evaluations. Below, we distinguish the demonstrated behavior of the submitted RamoMark construction from extensions that would require further algorithm design.
Weakness 1 / Question 1: What if the generation length is unknown in advance?
We agree that the current fixed-block construction chooses its codeword length before generation. To test how sensitive it is to early termination, we decoded prefixes of the same already-generated Qwen/CNN watermarked continuations while retaining the original fixed codebook. The following are message-recovery counts on 32 prompts:
Thus, early termination need not always destroy the watermark, and at these tested budgets useful recovery is possible before the full block is observed. For deployment, one potential extension is framed, independently decodable blocks with an explicit synchronization rule; complete blocks could be recovered even when generation stops unexpectedly. This remains a proposed extension, not an implemented or validated rateless solution. We will clarify this limitation and include the partial-block recovery figure.
Weakness 1: Dynamic speculative decoding and newer variants (MTP, DFlash, DTrees)
The current implementation assumes ordinary draft–verify–correct speculative sampling with a well-defined acceptance threshold and residual correction law. If speculative decoding is turned off, the emitted token is not in general generated through the same control-dependent channel. Similarly, MTP, DFlash and tree-based decoding may use different proposal/verification structures; the induced channel and receiver likelihoods must be derived for each variant.
We therefore agree that compatibility with dynamic SD-on/off switching and these modern speculative-decoding variants has not been demonstrated. We will make the assumptions explicit in the limitations section and will not suggest that the present experiments establish drop-in compatibility. Nevertheless, we would like to note that one main novelty our our paper lies in embedding watermark naturally in the specilative sampling pipeline.
Weakness 1: Computational cost of model-dependent detection
We agree that the submitted detector is more expensive than key-based detectors using only inexpensive token statistics: it requires target/draft likelihood evaluation and can dominate a downstream detection workload. The new Qwen/CNN comparison records the following descriptive generation and decoding times, in seconds per continuation:
These results illustrate both a generation-time advantage in these runs and a substantial detection-time disadvantage. We will explicitly state that “negligible overhead” applies only to the generation-time randomness modulation, not to offline detection.
Weakness 2: Ablation and motivation for choosing θ
We performed a matched five-value θ ablation on 128 Qwen/CNN prompts, using the same gold messages and saved burn-in sequences for each choice. Invalid receiver replays count as failures.
Mean AATPS remains approximately 2.53–2.58 across the tested thresholds, but recovery deteriorates markedly at large θ. These results show that the parameter matters and that the reference choice θ=0.75 is not uniquely optimal in the measured configurations. We will include the recovery/AATPS ablation figure and describe θ=0.75 as the selected operating point rather than an empirically proven optimum.