- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces COMPASS, a training-free watermarking method for continuous diffusion language models. It steers intermediate latent representations toward key-defined target anchors during denoising. Detection re-encodes the output text and measures its preference for target over non-target anchors. Experiments on Cola and ELF-L show improved detection and robustness to text edits with low computational overhead.
Critical strengths
-
The paper studies a novel setting. To the best of my knowledge, it is the first work on watermarking continuous diffusion language models.
-
Although the method builds on existing watermarking ideas, its design is reasonable and well suited to the continuous denoising process.
Critical weaknesses
-
On page 3, the authors explain that decoding is many-to-one and re-encoding cannot recover the original latents, which weakens the image-watermarking baselines. The same issue also affects COMPASS. Although COMPASS only needs to preserve a bias toward target regions, this advantage needs clearer evidence. Comparing the watermark signal before decoding and after re-encoding would help.
-
The claim of preserving text quality needs stronger support. The absolute log-perplexity gap on Cola is substantial, but its direction is not reported. Also, the ELF-L examples use weaker steering than the main experiments. Raw perplexity values, human evaluation, and examples generated with the main settings would make the quality claims more convincing.
-
Detection thresholds are calibrated on unedited, unwatermarked texts and then fixed across attacks. However, the false-positive rate may change after editing, especially when texts become shorter. The authors should apply the same edits to unwatermarked texts and report the resulting false-positive rates.
-
The ablations mainly vary hyperparameters, but do not clearly justify several design choices. For example, how does random target-anchor selection compare with choosing the nearest target anchor? Does SLERP offer an advantage over normalized linear interpolation?
Questions for authors
See Weaknesses.
6 comments
Quality evaluation: raw perplexity, signed changes, paired multi-judge ratings, and MAUVE
W2: The claim of preserving text quality needs stronger support. The absolute log-perplexity gap on Cola is substantial, but its direction is not reported. Also, the ELF-L examples use weaker steering than the main experiments. Raw perplexity values, human evaluation, and examples generated with the main settings would make the quality claims more convincing.
Raw perplexity and signed change. Perplexity is calculated using the pretrained Llama-3.1-8B base model, with Cola continuation tokens scored conditional on their prompts (Appx. N.1). We note that our method is on the Pareto frontier at any text quality value that a model provider may select (please see Figure 3). We selected the values for Tables 1 and 2 that reflect an average log PPL among all compared methods. For Cola, the reference-model perplexity increases from 26.34 (clean) to 78.06 (COMPASS), and a signed log PPL of +1.0865. We include the additional results in Appx. O.3, Table 36. While perplexity measures the reference model's assigned likelihood, it is not by itself a complete measurement of fluency, coherence, or preservation of meaning. This is why we also reported scores for independent language-model judges and add additional results.
Paired quality assessment and interpretable scores. CompassJudger evaluates fluency, coherence, repetition, naturalness, and meaningfulness on a 0-4 degradation scale (lower is better). The previously reported 9.1% degradation on Cola and 4.2% on ELF-L is a relative increase in the mean rating. We now give the absolute values rather than relying on a percentage of an ordinal rating.
| Quality evaluation | Clean mean ↓ | COMPASS mean ↓ | Mean paired change ↓ |
|---|---|---|---|
| Cola | 2.105 | 2.296 | +0.191 |
| ELF-L | 2.220 | 2.314 | +0.094 |
We added a multi-judge evaluation on 600 Cola continuations (Appx. O.1, Table 34). In addition to CompassJudger-2-7B-Instruct, we evaluate outputs with Llama-3.1-8B-Instruct and Mistral-Small-24B-Instruct. Each judge independently scores the same five linguistic dimensions using the same 0-4 degradation rubric. We report the mean paired increase in degradation rating (watermarked minus clean), with lower values indicating better preservation. All eight baselines are evaluated at their strengthened paraphrase-evaluation configurations.
| Method | CompassJudger-2 ↓ | Llama-3.1-8B ↓ | Mistral-Small-24B ↓ |
|---|---|---|---|
| COMPASS | +0.264 | +0.305 | +0.510 |
| TextSeal | +0.304 | +0.320 | +0.576 |
| KGW | +0.591 | +0.439 | +0.761 |
| Unigram | +0.984 | +0.522 | +0.862 |
| SynthID | +0.620 | +0.434 | +0.795 |
| DiPmark | +1.838 | +0.690 | +0.967 |
| GumbelSoft | +1.810 | +0.662 | +0.962 |
| Tree-Ring | +1.632 | +0.619 | +0.963 |
| Gaussian Shading | +0.672 | +0.609 | +0.921 |
COMPASS has the lowest observed mean degradation under all three judges, with TextSeal second in each case. The remaining baselines are included in Table 34. We emphasize the consistent ordering of the point estimates across independent evaluators, without treating the small gap to TextSeal as proven statistically significant.
We also include an exploratory distributional assessment against human-written C4 continuations using the MAUVE score (Appx. O.2, Table 35). A higher MAUVE score means that the generated text is closer to the human distribution. On the original 600 Cola prompts, COMPASS obtains 0.611, compared with 0.436 for TextSeal. This analysis is restricted to prompt-conditioned Cola: unconditional ELF-L has no corresponding prompt-matched human-reference distribution in our experiment. We present MAUVE as complementary evidence about distributional agreement, not as proof of semantic faithfulness or an individual-text human-quality rating.
We have not conducted a formal blinded human-rating study within the limited time. Instead, we provide reproducible cross-judge ratings, paired uncertainty estimates, reference-model perplexity, and the distributional check above.
To summarize, the additional judges, prompt-paired comparisons, and distributional analysis substantially broaden the original text quality evaluation. The new results further support the conclusion that COMPASS achieves favorable empirical quality–detection trade-off for the studied CDLMs.
W3: Detection thresholds are calibrated on unedited, unwatermarked texts and then fixed across attacks. However, the false-positive rate may change after editing, especially when texts become shorter. The authors should apply the same edits to unwatermarked texts and report the resulting false-positive rates.
In all evaluations, we kept the same fixed threshold (calibrated to 1% FPR on unattacked unwatermarked generations) rather than recalibrating per source or attack. Our goal was to reflect the most realistic deployment scenario: at detection time, the detector does not know in advance whether an arbitrary input text is human-written, unwatermarked model output, edited, truncated, or modified in any adversarial way, and must evaluate any candidate text using that single frozen threshold.
In Tables 1 and 2 we relabelled the attack columns to "TPR at the clean-calibrated threshold" and added the results for both FPRs and recalibrated TPRs to the appendix.
False-positive rate on Cola:
We evaluated all 11 post-generation attacks on unwatermarked outputs from Cola with the same frozen 1%-FPR thresholds from the initial submission. We observe that the resulting FPR increase is very modest (with average around 1.79%), considering that the TPR for COMPASS stays much higher (77.2%-88.3%) than for other methods (e.g., TPR collapsing to 0.0%–5.5% for TreeRing) :
For Cola:
| Method | WD10 | WD30 | WD50 | Span30 | First50 | Last50 | Swap20 | Typo5 | Shuffle | CP25 | CP50 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| COMPASS (n=600) | 1.7% | 2.2% | 3.2% | 1.7% | 2.2% | 3.5% | 0.8% | 1.0% | 1.0% | 1.2% | 1.2% |
| KGW (n=600) | 1.0% | 1.3% | 1.2% | 0.5% | 0.7% | 0.7% | 0.7% | 1.0% | 1.3% | 1.0% | 1.0% |
| Unigram (n=600) | 0.5% | 0.7% | 0.3% | 0.7% | 0.7% | 0.5% | 0.8% | 2.5% | 0.8% | 0.8% | 0.8% |
| GumbelSoft (n=600) | 0.3% | 0.3% | 0.3% | 0.3% | 0.3% | 1.3% | 0.7% | 0.3% | 0.7% | 1.0% | 1.0% |
| TextSeal (n=600) | 1.3% | 0.8% | 1.0% | 0.5% | 0.3% | 0.2% | 1.0% | 0.8% | 0.3% | 1.3% | 1.3% |
| SynthID (n=600) | 1.7% | 0.7% | 0.3% | 0.3% | 1.0% | 0.5% | 0.7% | 1.0% | 1.0% | 1.0% | 1.0% |
| DiPmark (n=600) | 2.0% | 1.7% | 2.2% | 1.3% | 1.5% | 1.2% | 1.0% | 2.2% | 0.8% | 1.0% | 1.0% |
| Tree-Ring (n=600) | 1.0% | 0.2% | 0.0% | 0.0% | 0.0% | 0.0% | 2.0% | 2.3% | 1.0% | 1.0% | 1.0% |
| Gaussian Shading (n=600) | 1.8% | 0.8% | 2.2% | 1.3% | 2.3% | 3.3% | 1.3% | 0.7% | 1.5% | 1.0% | 1.0% |
We also report the corresponding TPRs:
| Method | Deg. (%) ↓ | |Δ log PPL| ↓ | Unattacked ↑ | WD10 ↑ | WD30 ↑ | WD50 ↑ | Span30 ↑ | First50 ↑ | Last50 ↑ | Swap20 ↑ | Typo5 ↑ | Shuffle ↑ | CP25 ↑ | CP50 ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| COMPASS (ours) | 9.1 | 1.09 | 94.2 | 86.8 | 88.3 | 82.3 | 86.2 | 77.2 | 86.0 | 89.7 | 78.7 | 88.2 | 77.3 | 54.3 |
| KGW | 10.9 | 0.76 | 35.2 | 17.8 | 10.2 | 4.7 | 16.3 | 11.7 | 15.8 | 14.7 | 11.8 | 28.8 | 23.0 | 14.7 |
| Unigram | 11.6 | 0.89 | 35.2 | 24.8 | 20.5 | 13.2 | 17.8 | 10.3 | 17.5 | 27.8 | 22.3 | 29.8 | 22.7 | 13.5 |
| GumbelSoft | 22.4 | 1.50 | 49.5 | 21.8 | 4.7 | 1.3 | 25.2 | 16.8 | 21.2 | 14.3 | 9.8 | 39.7 | 32.8 | 23.8 |
| TextSeal | 9.2 | 0.72 | 76.8 | 47.8 | 22.2 | 6.2 | 52.5 | 37.5 | 46.5 | 44.7 | 29.3 | 70.0 | 65.3 | 50.2 |
| SynthID | 15.0 | 1.07 | 72.0 | 26.5 | 7.0 | 1.8 | 35.3 | 25.3 | 29.5 | 17.2 | 12.0 | 55.5 | 51.3 | 30.8 |
| DiPmark | 15.9 | 1.06 | 8.3 | 6.7 | 4.7 | 2.3 | 6.5 | 4.3 | 3.5 | 4.2 | 3.5 | 7.7 | 5.3 | 3.5 |
| Tree-Ring | 25.9 | 2.41 | 89.2 | 2.0 | 0.0 | 0.0 | 5.5 | 0.0 | 0.0 | 79.0 | 5.2 | 14.7 | 38.0 | 20.2 |
| Gaussian Shading | 34.2 | 1.29 | 31.5 | 32.5 | 45.0 | 45.8 | 43.8 | 42.7 | 50.0 | 31.5 | 37.3 | 33.7 | 26.8 | 23.0 |
On surface edits (Swap20, Typo5, Shuffle) and dilution (CP25, CP50), COMPASS remains its FPR close to 1% (0.8%-1.2% FPR). Under deletion and truncation, shortening further reduces the number of averaged latents and increases the variance of the mean margin, raising COMPASS's FPR to 1.7%-3.5%, comparable to Gaussian Shading (up to 3.3%) and DiPmark (up to 2.2%), while COMPASS still retains 77.2%-88.3% TPR on those same edits. By contrast, Tree-Ring's 0.0% FPR under heavy deletion/truncation coincides with its TPR collapsing to 0.0%–5.5%.
Thank you & watermark survival through decoding and re-encoding
We thank the Reviewer for the positive feedback. We appreciate recognizing our watermarking scheme for CDLMs as novel and well designed. We address each concern one by one below:
We compare detection on the final generated latents before decoding with detection after the generated text is re-encoded. For Gaussian Shading and Tree-Ring, the corresponding latent representation is passed through the inversion and keyed detector.
For Cola:
For ELF-L:
The signal for COMPASS remains detectable after re-encoding almost to the same extent as before the decoding. We attribute this robustness of COMPASS to the fact that it relies on a directional bias rather than exact reconstruction of every latent. We provide further details in Appx. J.9 (Figure 18).