acceptodds
Under review as a conference paper at ICLR 2027

When Words Change but Meaning Remains: Robust Semantic-Stylistic Watermarking for LLMs

Abstract

Watermarking can support provenance verification for text generated by large language models (LLMs), but many existing methods encode evidence in token-level statistics that degrade under meaning-preserving rewriting. We introduce SS-WM, a base-model-training-free watermarking framework that distributes key-conditioned evidence across a joint semantic–stylistic carrier. The semantic channel uses a projected representation derived from a frozen sentence encoder, while the stylistic channel uses distributional statistics of textual realization. A secret key selects carrier constraints, and inference-time reranking softly steers generation toward the resulting key-specific region while preserving prompt fidelity. Detection extracts the same carrier feature families from completed text and performs a keyed statistical hypothesis test, supporting candidate-key verification after paraphrasing, translation, summarization, stylistic rewriting, truncation, and partial reuse. SS-WM does not assume that individual semantic coordinates correspond to interpretable discourse structures; instead, robustness relies on empirical transformation stability and complementarity with stylistic evidence. On LLaMA-2-7B across fixed, composed, and adaptive transformations, with additional cross-model evaluation on LLaMA-2-13B, Mistral-7B, and OPT-2.7B, SS-WM improves watermark retention over the evaluated token-level and semantics-aware baselines while maintaining strong clean-text detection and limited quality degradation. Additional analyses characterize prefix-to-sequence surrogate alignment, stylistic feature contributions, wrong-key behavior, prompt compatibility, and the trade-off between logit-access and black-box deployment.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models (LLMs), Text Watermarking, Semantic–Stylistic Watermarking, Transformation Robustness, AI-Generated Text Attribution

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pL6hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SS-WM, a watermark scheme with two channels: in both semantic and stylistic features rather than conventional token-level schemes like KGW and the others. During generation, it periodically reranks candidate next tokens to move the text toward those favored by the secret key selected regions (style and semantics). For detection, it extracts the same features and compares their likelihood under the candidate key with an unwatermarked baseline. The intuition is that paraphrasing may change the words while preserving enough meaning and style to retain the signal. The experiments report promising results of detection rate after paraphrasing.

Critical strengths
  1. Combining semantic and style features for watermarking is well motivated.
  2. The paper evaluates wrong-key and partial-text detection, which matter for false-positive control and broader robustness.
Critical weaknesses
  1. The "attack" folder under the released repository contains only a single readme file. I can't find the actual paraphrasing implementation, which is the crucial part to evaluate the strength of the paper. Many watermarking papers make overclaims about their results on paraphrasing robustness by doing less-intensive attacks than the advertised ones. This limits my confidence in the central robustness claim.

  2. Another important weakness is the limitation on forgery attack: the threat model assumes that a secret key prevents an attacker from imitating its selected constraints, despite allowing access to watermarked outputs. With enough chosen prompts and marked examples, an attacker might infer the key-dependent style feature pattern without recovering the key itself. My suggestion will be to actually collect enough samples, run an agent on style mining, and see whether it can actually recover the secret styling key or no.

Questions for authors
  1. Can you release the full implementation and concrete data log? Many folders are empty.

  2. Given increasing numbers of outputs under one key, how well can an attacker infer its feature pattern and make unrelated text pass that key’s detector?

6 comments

We thank the reviewer for recognizing the motivation for a joint semantic–stylistic watermark and for emphasizing wrong-key and partial-text detection. These are important to the practical interpretation of verification results.

Weakness 1:

The "attack" folder under the released repository contains only a single readme file. I can't find the actual paraphrasing implementation, which is the crucial part to evaluate the strength of the paper. Many watermarking papers make overclaims about their results on paraphrasing robustness by doing less-intensive attacks than the advertised ones. This limits my confidence in the central robustness claim.

We thank the reviewer for pointing out this issue and apologize for the oversight. During preparation of the initial anonymous repository, we inadvertently omitted the executable attack scripts from the uploaded archive, leaving only the README in the attack directory. This was an upload mistake rather than an intended restriction on implementation access. We acknowledge that this omission made it difficult for the reviewer to examine the attack procedures and assess their strength.

We have now corrected this omission and updated the anonymous repository with the attack implementations and supporting evaluation scripts. In particular, the repository includes code for GPT-3.5 paraphrasing, modern-model rewriting, style rewriting, expansion, feature-aware candidate selection, and adaptive attacks. It also provides utilities for admissibility checking, recording candidate-generation attempts, applying fallback and selection rules, and exporting detector scores. Reviewers can therefore inspect the attack prompts and configurations, follow how rewriting candidates are generated and selected, and examine how the selected outputs are passed to the watermark detector.

To further improve transparency, we have clarified the attack configurations in Appendix F.4 (pp. 25–27). The revised description explicitly distinguishes the default GPT-3.5 paraphrasing attack from the intensity-controlled experiments. The default setting uses a general paraphrasing instruction without a specified replacement percentage, whereas the intensity-controlled settings request 20%, 40%, 60%, and 80% content-word replacement. The corresponding measured token-level edit rates are approximately 18%, 37%, 55%, and 72%, respectively. We also document the semantic admissibility criteria, retry budgets, fallback procedures, and candidate-selection rules to clarify how attack strength is evaluated beyond the requested rewriting intensity.

We have additionally strengthened the attack evaluation with contemporary rewriting and more demanding attack settings. Appendix H.3 (Table 20) reports single-pass and three-stage GPT-5.6 Sol rewriting, including measured edit rates, independent semantic similarity, and admissibility. Appendix H.4 (Table 22) includes expansion, transferred-proxy B4 scrubbing, and public-feature-aware attacks. These experiments provide a broader assessment of robustness under different rewriting mechanisms and attacker capabilities.

Question 1:

Can you release the full implementation and concrete data log? Many folders are empty.

We thank the reviewer for pointing out the incomplete contents of the original anonymous archive. As discussed in our response to pL6h.W1, the attack scripts were inadvertently omitted during the initial upload, leaving README files in the relevant directories. We apologize for this mistake. The updated repository includes the SS-WM implementation, supported paraphrasing and rewriting attacks, admissibility filtering, adaptive and feature-aware candidate selection, query accounting, and detector-score export. Appendix F.4 (pp. 25–27) complements the code with the attack prompts, parameter settings, fallback rules, and evaluation definitions. The updated artifact is designed to let reviewers inspect and execute the supported workflows and generate concrete source-level traces. Its logging and export utilities record the source identifier, attack configuration, generated candidates, admissibility decisions, selected rewrite, query counts, and detector scores as applicable to each run. The release is an executable implementation with logging support rather than a static archive of historical API responses. With the documented dependencies and access to the relevant models or APIs, reviewers can follow how attack outputs are produced and how detection statistics are calculated. We appreciate the reviewer for highlighting this omission, which has led to a substantially more inspectable release.

Weakness 2: Forgery through key-dependent profile inference

Another important weakness is the limitation on forgery attack: the threat model assumes that a secret key prevents an attacker from imitating its selected constraints, despite allowing access to watermarked outputs. With enough chosen prompts and marked examples, an attacker might infer the key-dependent style feature pattern without recovering the key itself. My suggestion will be to actually collect enough samples, run an agent on style mining, and see whether it can actually recover the secret styling key or no.

We thank the reviewer for identifying this important security concern. We agree that key secrecy alone may not prevent an adversary from learning and imitating the stylistic patterns induced by a watermark. Our original evaluation examined wrong-key acceptance and rewriting robustness but did not directly investigate this form of chosen-prompt profile inference. To address this limitation, we have added a targeted forgery experiment in Appendix H.8 (p. 38, Table 27) and expanded the security discussion in Section 4.4 (p. 9).

First, we investigate whether an attacker can infer a key-dependent feature profile from observed watermarked outputs. The attacker collects 10, 100, or 1,000 marked outputs for each of ten target keys and uses the public feature extractor to estimate the induced stylistic and semantic shifts. The estimated profile is then compared with the target profile. The stylistic-profile cosine similarity increases from 0.775 with ten observations to 0.96 with 1,000, indicating that additional same-key observations enable increasingly accurate approximation of the target's observable feature pattern.

Second, we evaluate whether the inferred profile can facilitate forgery of unrelated text. Using 500 unrelated C4 passages disjoint from the profile-estimation material, the attacker generates eight candidate rewrites per round with GPT-5.6 Sol. Candidates are checked for admissibility, and the rewrite closest to the inferred stylistic or joint semantic–stylistic profile is selected. This process is repeated over three rounds. Importantly, selection relies on the estimated feature profile, without access to the secret key or target detector scores.

Additional security experiment (Appendix H.8, Table 27): Targeted forgery acceptance (%) under increasing observation budgets.

Observed outputs per keyStylistic-profile forgeryJoint-profile forgery
103.54.5
10014.020.0
1,00024.033.5

The results demonstrate that forgery becomes increasingly effective as the observation budget grows. Stylistic-profile acceptance rises from 3.5% to 24.0%, while joint-profile acceptance increases from 4.5% to 33.5%. In comparison, unmodified human-written text has an acceptance rate of 1.1%, and the different-key-profile control achieves 1.9% at the 1,000-observation setting. These controls suggest that the improvement arises from learning target-specific watermark characteristics rather than generic rewriting alone.

Regarding recovery of the secret styling key, we distinguish exact key recovery from effective profile inference. Our experiment does not reconstruct the cryptographic key or identify the exact keyed codebook selections. Instead, it tests whether an adversary can approximate the observable feature pattern sufficiently to produce text accepted by the target detector. The results show that successful profile imitation and targeted forgery do not require exact key recovery. While the experiment implements feature-guided iterative rewriting, it does not evaluate an unrestricted autonomous style-mining agent; exact key recovery therefore remains an open question.

Finally, we have clarified the security implications. Wrong-key acceptance measures whether a watermark generated under one key is accepted under another, whereas targeted forgery evaluates deliberately modified unrelated text against the intended key. These are different adversarial conditions. Accordingly, the 1% null-FPR calibration does not guarantee resistance to an attacker who learns the target's observable feature pattern. We have clarified in Section 4.4 and Appendix E.6 that SS-WM provides statistical verification under a supplied candidate key, not cryptographic proof of provenance or unconditional forgery resistance.

We appreciate the reviewer for motivating this additional experiment. The findings substantiate the concern that repeated access to marked outputs can expose exploitable key-dependent regularities, while establishing the distinction between effective profile imitation and exact secret-key recovery.

Question 2:

Given increasing numbers of outputs under one key, how well can an attacker infer its feature pattern and make unrelated text pass that key’s detector?

We thank the reviewer for this insightful question. To investigate this threat directly, we have added a chosen-prompt profile-inference and targeted forgery experiment in Appendix H.8 (p. 38, Table 27). The attacker observes 10, 100, or 1,000 watermarked outputs generated under the same target key and uses the public feature extractor to estimate the corresponding stylistic or joint semantic–stylistic profile. The estimated profile is then used to guide rewriting of 500 unrelated C4 passages from a disjoint test set, with eight candidates generated per round and the admissible candidate closest to the inferred profile selected over three rounds. This allows us to evaluate whether information learned from observed watermarked outputs can facilitate forgery without direct access to the secret key.

The reported results show that the attack becomes substantially more effective as the number of observed outputs increases. Stylistic-profile forgery acceptance rises from 3.5% to 14.0% and 24.0% for 10, 100, and 1,000 observations, respectively. The joint semantic–stylistic attack achieves 4.5%, 20.0%, and 33.5% at the same observation budgets. Consistent with this trend, the reported cosine similarity between the inferred and target stylistic profiles increases from 0.775 with ten observations to 0.96 with 1,000 observations. In comparison, unmodified human-written text has an acceptance rate of 1.1%, while the different-key profile control achieves 1.9% in the reported 1,000-observation setting.

These findings directly support the reviewer's concern: an adversary can progressively approximate the observable feature pattern associated with a secret key and exploit that information to increase the probability of making unrelated text pass the corresponding detector. Importantly, the attack does not recover the secret key or reconstruct its exact codebook selections; rather, it demonstrates that effective profile imitation can enable targeted forgery without such recovery. We have therefore clarified in Section 4.4 and Appendix E.6 that SS-WM provides statistical verification under the specified threat model, rather than an unconditional guarantee against forgery by adversaries with extensive access to marked outputs. We appreciate the reviewer for motivating this additional security analysis.

Reviewer_Sub5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SS-WM, a watermarking framework for large language models that shifts the watermark carrier from token-level statistics to a joint representation of semantic and stylistic features, with the goal of keeping the watermark detectable after meaning-preserving transformations such as paraphrasing, translation, and summarization. The method consists of key-conditioned semantic and stylistic target constraints, prefix-level surrogate penalties that steer generation during decoding, and a keyed likelihood-ratio detector that separates false-positive-rate calibration from registry-level attribution. Experiments are conducted mainly on LLaMA-2-7B and compare against baselines including KGW, SIR, and EXP-Edit. The authors report higher detection rates at 1% FPR under a range of rewriting attacks.

Critical strengths

Strength 1: The problem is well motivated, and the carrier design is a meaningful direction. Token-level watermarks are known to degrade sharply under rewriting; in Table 10, KGW-1 drops from 98.4% TPR on clean text to 28.5% under the composed attack. Moving the watermark carrier to a joint semantic and stylistic representation, and explicitly contrasting it with semantic-only designs such as SIR and SemStamp in Section 2, is a useful direction for robust text watermarking.

Strength 2: The evaluation covers a broad range of attack types and includes some checks of the method's assumptions. The appendix evaluates paraphrasing, DIPPER, translation, summarization, style rewriting, composed attacks, and detector-guided adaptive selection (Tables 10-14), and examines detector behavior such as wrong-key acceptance (Tables 6-8). However, this breadth holds mainly across attack types: the main experiments use only LLaMA-2-7B, and the cross-model results are limited to clean-text AUC (Table 15).

Critical weaknesses

Weakness 1: The sources of novelty and of the performance gains are not clearly established. The ablations are limited to two settings: a sweep over the semantic and stylistic weights (Table 17, evaluated only on clean text and a single paraphrase attack), and a leave-one-out analysis of stylistic feature families (each family affects results by at most 3.8 points). With the semantic component alone, the post-paraphrase detection rate already reaches 85.7%, exceeding all semantic baselines (SIR: 74.0%), while adding the stylistic component improves it by only about 6.9 points. This suggests that the gains come mainly from the semantic component and its specific implementation rather than from the joint carrier itself. Key components, including the prefix penalty and reranking, the prompt-fidelity term, the likelihood-ratio detector, the projection network, and the codebook size, are not ablated. The baselines are run with default parameters and their perplexities are not reported, which makes it hard to judge whether the comparison is fair. There is also no quantitative comparison with SimMark, the most closely related method.

Weakness 2: Despite the strong reported results, the validation of real-world robustness and practical usability is insufficient. In terms of attack strength, the paper does not use the strongest current paraphrasing models; human rewriting and expansion are mentioned in the threat model but not evaluated; there are no iterative optimization-based attacks or attacks that exploit knowledge of the feature space; and neither the paraphrase strength nor the admissibility criteria for attacks are defined. In terms of quality, perplexity increases from 5.82 to 6.58 (about 13%), without a baseline comparison or any human evaluation. In terms of applicability, only 200-token texts are tested, and in the exact short-answer setting the detection rate is only 63.5% (Table 8). In terms of deployment, no wall-clock overhead is measured; each key requires generating about 1,000 watermarked texts to fit the detector; and the false-positive rate is calibrated on only 5,000 human-written C4 texts, with the number of candidate keys left unspecified.

Questions for authors
  1. Numerical inconsistencies: The F1 scores for the same setting differ between Table 9 and Table 10 (e.g., 0.991 vs. 0.993 for SS-WM). In Table 10, several rows report a Best F1 lower than the F1 at 10% FPR (e.g., 0.792 vs. 0.815 for KGW-1 under summarization). The SS-WM rows for DIPPER-1 and 1-hop translation are identical. The main GPT-3.5 paraphrase result (92.6%) also does not match the Strong level in Table 11 (89.7%). Could the authors explain how F1 is computed and verify these numbers?

  2. Adaptive attack: In Table 12, why is the TPR under the adaptive attack higher than under the weakest single attack in the candidate pool?

  3. Statistical reliability: What is the sample size per cell? Could the authors report confidence intervals or variance for the main results?

  4. Sources of gain: Could the authors add ablations on the prefix penalty, the likelihood-ratio detector, the projection network, and the codebook size, and report tuned baselines together with their perplexities?

  5. Stronger attacks: How does SS-WM perform against stronger paraphrasing models, human rewriting, iterative optimization-based attacks, and attacks that exploit knowledge of the feature space?

4 comments

We thank the reviewer for recognizing the breadth of our evaluation and for raising important questions about novelty, component contributions, and baseline fairness.

Weakness 1:

The sources of novelty and of the performance gains are not clearly established. The ablations are limited to two settings: a sweep over the semantic and stylistic weights (Table 17, evaluated only on clean text and a single paraphrase attack), and a leave-one-out analysis of stylistic feature families (each family affects results by at most 3.8 points). With the semantic component alone, the post-paraphrase detection rate already reaches 85.7%, exceeding all semantic baselines (SIR: 74.0%), while adding the stylistic component improves it by only about 6.9 points. This suggests that the gains come mainly from the semantic component and its specific implementation rather than from the joint carrier itself. Key components, including the prefix penalty and reranking, the prompt-fidelity term, the likelihood-ratio detector, the projection network, and the codebook size, are not ablated. The baselines are run with default parameters and their perplexities are not reported, which makes it hard to judge whether the comparison is fair. There is also no quantitative comparison with SimMark, the most closely related method.

We agree that the semantic channel accounts for a substantial part of SS-WM's robustness. Our claim is therefore not that stylistic features alone explain the improvement, but that combining semantic and stylistic evidence provides additional robustness over a strong semantic-only implementation. The expanded channel ablation in Section 4.3 (p. 8) and Appendix H.5 (p. 37, Table 23) supports this distinction. Under the main GPT-3.5 paraphrase, semantic-only, stylistic-only, and joint SS-WM achieve 85.7%, 78.5%, and 92.6% TPR, respectively. The incremental benefit of the joint carrier is also observed under other transformations:

Additional channel ablation (Table 23): Joint-carrier gain relative to semantic-only SS-WM, in TPR percentage points.

AttackJoint-carrier gain
GPT-3.5 paraphrase+6.9
Summarization+6.2
Composed attack+8.9
Style rewriting+3.7
Modern rewriting+5.5

We also extended the component ablations (Appendix H.5, Table 24). Reranking every ten steps rather than every five reduces paraphrase/composed TPR from 92.6%/76.9% to 89.5%/73.4%. Removing the prompt-fidelity term raises paraphrase TPR to 93.5% but increases perplexity from 6.58 to 7.00, illustrating its quality–robustness role. Replacing the learned semantic projection with 32-dimensional PCA lowers paraphrase/composed TPR to 87.5%/70.5%; varying the codebook size and the number of selected entries further examines design sensitivity. To isolate verification from generation, we also evaluated a masked carrier-distance z-test on the same texts: it reaches 87.5%/69.0%, compared with 92.6%/76.9% for the keyed likelihood-ratio detector. Because these texts are fixed, the 5.1- and 7.9-point differences more directly reflect the detector choice.

To address baseline fairness, Table 3 (p. 7) now includes SimMark and reports perplexity for each method; Appendix F.3 (pp. 24–25) documents baseline settings. Appendix H.2 (p. 35, Table 19) further compares methods with watermark strength adjusted toward SS-WM's clean-text PPL of 6.58 ± 0.05 where possible:

Strength-adjusted comparison (Table 19): TPR (%) at the 1% null-FPR operating point; modern rewriting is evaluated on admissible outputs, while fixed paraphrase and composed attacks are unfiltered.

MethodPPLParaphraseComposedModern
KGW-1 (adjusted)6.5861.532.023.7
Unigram WM (adjusted)6.5862.533.028.9
SemaMark (adjusted)6.5463.551.552.6
SS-WM (default)6.5892.676.976.5

These adjusted results should not be interpreted as universal quality matching: baselines without a suitable strength control retain their default or closest attainable PPL (e.g., SIR 6.65 and SimMark 6.25). Matching FPR controls the detection operating point; matching PPL addresses one, but not every, aspect of text quality.

Finally, Table 24 makes the runtime trade-off explicit. Reranking at every step increases paraphrase TPR to 93.5% but raises the generation-time ratio from 3.64× (default, every five steps) to 19.0×. Using 20 candidates gives 90.0% paraphrase and 73.5% composed TPR at 2.75× cost. A strictly matched ablation removing all prefix guidance is not included; changing reranking frequency does not substitute for that test. Overall, the results support a substantial semantic contribution, a measurable additional benefit from the joint carrier, and a clearer account of design and baseline-quality trade-offs.

Weakness 2 (part 1):

Despite the strong reported results, the validation of real-world robustness and practical usability is insufficient. In terms of attack strength, the paper does not use the strongest current paraphrasing models; human rewriting and expansion are mentioned in the threat model but not evaluated; there are no iterative optimization-based attacks or attacks that exploit knowledge of the feature space; and neither the paraphrase strength nor the admissibility criteria for attacks are defined.

We agree that the original attack suite alone could not establish robustness against contemporary or adaptive adversaries. We therefore added single-pass and three-stage GPT-5.6 Sol rewriting (Section 4.2, p. 7; Appendix H.3, p. 35, Table 20). At the original 1% null-FPR threshold, SS-WM achieves 76.5% and 68.5% TPR, respectively, on admissible final rewrites. Appendix H.4 (p. 36, Table 22) adds expansion, transferred-proxy B4 scrubbing, and public-feature-aware selection:

AttackSS-WM TPR (%)
Expansion (~1.5×)84.0
B4 scrubbing (transferred proxy)57.5
Feature-aware: stylistic77.0
Feature-aware: joint70.0

These are conditional TPRs on admissible outputs. B4 uses a proxy developed for Unigram watermarking rather than one trained specifically against SS-WM, so it is a transfer stress test, not a fully SS-WM-adapted optimization attack.

Appendix F.4 (pp. 25–27) now defines admissibility using independent MPNet similarity ≥0.75, bidirectional MNLI entailment ≥0.70, and LLaMA-2-13B perplexity ≤30, with explicit retry, fallback, and selection rules. It also separates requested rewriting intensity from observed changes: the GPT-3.5 Strong prompt requests 60% content-word replacement, while measured token edits are approximately 55%. These automatic checks cannot guarantee factual fidelity.

The public-feature-aware attacker generates eight candidates, filters them, and selects the admissible rewrite closest to a human-text reference mean in stylistic or joint feature space, without knowing the secret key or detector score. The joint attack yields 70.0% TPR for SS-WM versus 72.5% for SIR, showing that SS-WM does not lead under every attack. The separate score-aware stress test (Appendix G.9, Table 13) grants detector-score access; SS-WM TPR declines from 84.6% (5 candidates) to 78.3% (20 candidates). These conditional rates may involve different eligible source populations across budgets.

For three-stage rewriting, each stage is checked against the original text, and failed stages revert to the last admissible version; three scheduled stages thus need not mean three successful rewrites. Human-authored rewriting remains untested (Section 4.6, p. 10). We present these additions as broader attack coverage, not exhaustive robustness.

Weakness 2 (part 2):

SS-WM raises perplexity from 5.82 to 6.58 (~13%). Table 3 (p. 7) now reports baseline perplexities, and Appendix H.6 (p. 37, Table 25) adds independent similarity, NLI non-contradiction, factual-QA, and summarization metrics. These show measurable content-quality effects but cannot replace human judgments; no human quality study is claimed.

We also measured detection across output lengths (Appendix H.9, Table 29):

LengthPost-paraphrase TPR (%)
50 tokens63.5
100 tokens81.5
200 tokens92.6

These figures use the global threshold, not length-specific 1% FPR calibration. The 63.5% detection rate for exact short answers in Table 8 (pp. 9–10) comes from a different task, not the 50-token length-sweep condition. Both demonstrate limitations for short or constrained outputs.

Appendix H.9 (pp. 38–39, Table 28) reports A100 wall-clock times: 5.5 s for unwatermarked generation, 20 s for logit-access SS-WM (3.64×), and approximately 60 ms for post-enrollment verification. For strict black-box selection over five completed candidates, batching takes 5.5 s versus 27.5 s sequentially; batching assumes sufficient parallel capacity. The same table reports paraphrase TPR of 88.0%, 90.5%, 91.8%, and 92.6% when fitting each key's detector with 100, 250, 500, and 1,000 marked texts, respectively.

Finally, Section 4.1 (p. 6) specifies 5,000 C4 calibration passages and 51 evaluated keys. Applying the C4-calibrated threshold to held-out corpora gives FPR of 1.0% on C4, 1.7% on Wikipedia, and 1.9% on CNN/DailyMail (Table 29). Thus, 1% is a calibration target rather than a cross-domain guarantee. Our deployment claim remains verification under a supplied candidate key, with acknowledged limitations from corpus shift, enrollment cost, and short outputs.

Question 1: Could the authors clarify the F1 calculations, identical DIPPER-1/translation results, and the GPT-3.5 paraphrase intensity difference?

We thank the reviewer for identifying these inconsistencies and apologize for errors introduced when consolidating results across methods, thresholds, and attacks. Section 4.1 (p. 6) now specifies the balanced-set operating-point formula, F1 = 2TPR/(1 + TPR + FPR) (rates as fractions). Best F1 is the maximum over decision thresholds and therefore cannot be below F1 at an included operating point. We corrected the reporting inconsistencies across Tables 9–10.

Rechecking the evaluation logs revealed that the one-hop translation row had inadvertently reused DIPPER-1 values. The corrected translation TPRs are 88.4% (F1 0.933) at 1% FPR and 93.5% (F1 0.919) at 10% FPR; DIPPER-1 remains 89.4% and 94.5%. The revision propagates the translation correction through Tables 3, 7, 10, 11, 18, 23, and 30 (95% CI [86.4%, 90.3%]). The ranking against k-SemStamp (83.9%) remains unchanged.

SS-WM attackTPR @1% FPRTPR @10% FPR
DIPPER-189.4%94.5%
Translation (1-hop)88.4%93.5%

Finally, the 92.6% main GPT-3.5 paraphrase result uses a general single-pass prompt, whereas the 89.7% Strong result requests 60% content-word replacement (approximately 55% observed token edits). Appendix F.4 (p. 25) now distinguishes these protocols.

Question 2: Why can adaptive-attack TPR exceed that of the weakest single attack?

Response. Table 12 (Appendix G.8, pp. 29–31) compares different candidate sets and denominators. The worst fixed attack can include a composed transformation and evaluates all outputs; constrained adaptive selection chooses the lowest-scoring admissible single transformation and excludes sources without an admissible rewrite. An inadmissible candidate cannot be selected. Thus, 81.5% constrained-adaptive TPR versus 81.2% fixed summarization for SS-WM is not a contradiction. The unconstrained single-transformation result is 78.5%. On identical sources with identical candidates and thresholds, minimum-score selection cannot yield a higher detection indicator than a candidate in its pool; that pointwise guarantee does not extend to differently filtered populations.

Question 3: What are the sample sizes and confidence intervals?

Section 4.1 (p. 6) specifies 1,000 marked texts per fixed-attack cell, pooled over seeds 42, 43, and 44. Appendix H.10 (p. 40, Table 30) reports 95% percentile-bootstrap intervals from 1,000 source-text-level resamples:

SS-WM attackTPR (%)95% CI (%)
GPT-3.5 paraphrase92.6[90.8, 94.1]
Translation (1-hop)88.4[86.4, 90.3]
Composed76.9[74.2, 79.4]

Resampling by source accounts for repeated measurements on the same text. Admissible-only attack TPR uses only sources with a qualifying rewrite, so its denominator varies by method and attack. Wrong-key uncertainty similarly groups the 50 alternative-key decisions for each text rather than counting them as independent texts.

Question 4: Can you ablate the prefix penalty, detector, projection, and codebook, and report tuned baseline perplexities?

Appendix H.5 (pp. 36–37, Tables 23–24) now includes channel ablations, reranking-frequency and candidate-budget sweeps, removal of prompt fidelity, a masked-distance z-test in place of the likelihood-ratio detector, PCA in place of the learned projection, and codebook-size/selection variants. A fully matched ablation removing all prefix guidance is not included; the reranking sweep only partially addresses that request. Table 3 reports baseline PPL and adds SimMark, while Appendix H.2 (p. 35, Table 19) compares strength-adjusted methods near PPL 6.58 ± 0.05 when feasible and identifies unmatched default settings.

Question 5: What happens under modern paraphrasing, human rewriting, iterative attacks, and feature-aware attacks?

Appendix H.3 (p. 35, Table 20) reports 76.5% and 68.5% SS-WM TPR under single-pass and three-stage GPT-5.6 Sol rewriting on admissible final outputs. Appendix H.4 (p. 36, Table 22) reports 57.5% against transferred-proxy B4 scrubbing, and 77.0%/70.0% under stylistic/joint public-feature-aware selection. B4 does not use a proxy specifically trained for SS-WM; the feature-aware attacker knows neither the secret key nor detector score. Human-authored rewriting was not evaluated (Section 4.6, p. 10). These extensions broaden attack coverage without claiming robustness to all adaptive or human adversaries.

Dear Reviewer,

Thank you once again for your thoughtful comments and valuable feedback.

We have made every effort to address your suggestions and concerns, and we would be grateful if you could review our rebuttal and let us know if you have any remaining questions or concerns.

Thank you very much for your time and consideration.

Best regards,

The Authors

Reviewer_7ayQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies robust provenance verification for LLM text after semantic-preserving rewriting. It proposes SS-WM, which reranks generations toward secret-key-selected projected semantic and stylistic carrier targets and verifies a supplied key with a likelihood-ratio detector. Experiments compare robustness, quality, wrong-key behavior, prompt compatibility, and deployment modes across several rewriting attacks and generators.

Critical strengths

A concrete joint-carrier formulation with evidence that both channels matter. Equations 5–8 specify how the secret key determines semantic and stylistic target regions and how these are used during reranking. The semantic-only and stylistic-only results in Table 17, together with the feature-family removals in Table 5, show that the reported robustness is not reducible to a single stylistic heuristic.

The evaluation probes several practical failure modes rather than only reporting clean detection. Tables 3 and 10 show a substantial advantage over the evaluated baselines after paraphrasing, translation, summarization, style rewriting, and composed attacks at a common 1% FPR. Table 7 separates wrong-key acceptance from null false positives, and Table 8 importantly exposes a meaningful limitation under exact short-answer prompts rather than presenting watermarking as uniformly compatible with constrained generation.

Critical weaknesses

The main claim is robustness to rewriting, but every rewriter in the evaluation is from around 2023: GPT-3.5, DIPPER, NLLB-200, and BART. The authors acknowledge this in the limitations section, but it’s a significant gap, because a real adversary would most likely paraphrase with a more recent frontier model. The generators are also dated (LLaMA-2, Mistral-7B, OPT-2.7B).

I’m also concerned that the semantic channel works by steering the content of the generation toward a key-chosen region of embedding space, so the watermark changes what the model says, not only how it says it.

Quality is measured only with automatic metrics, and the semantic similarity score is computed with the same encoder family used for steering. Also, without a human evaluation, it’s hard to know whether responses become less accurate or drift off topic.

The baseline comparison is inconsistent. SemStamp, k-SemStamp, SemaMark, and Unigram Watermark, the closest related methods, appear only in Table 3. All the detailed experiments (per-attack, intensity, adaptive, score-aware, edge-case and cross-model) compare only against KGW, SIR and EXP-Edit, so the strongest claims are made against the weakest baselines.

Finally, I’m not convinced the keys are very distinctive. Each key picks 500 semantic entries over only 32 dimensions, about 15 per dimension, and the targets are averaged. An average of random bin centers in [−4, 4] tends to land near zero, which is the mean of normal text. So the watermark may be pushing text toward “average” text rather than toward something key-specific, and the paper doesn’t look at this.

Questions for authors

How far apart are the target vectors of different keys in practice? It would help to see the distribution of distances between random key pairs compared with the natural spread of non-watermarked text, how far watermarked text actually moves from the null distribution, and whether wrong-key acceptance increases for keys whose targets are closer together.

Also, would be great to hear your thoughts on the issues mentioned in "critical weaknesses" section.

5 comments

We thank the reviewer for recognizing the contribution of the joint semantic–stylistic carrier and the value of evaluating rewriting robustness. We also appreciate the observations about wrong-key acceptance, null false positives, and reduced detection performance on exact short answers.

Weakness 1

The main claim is robustness to rewriting, but every rewriter in the evaluation is from around 2023: GPT-3.5, DIPPER, NLLB-200, and BART. The authors acknowledge this in the limitations section, but it’s a significant gap, because a real adversary would most likely paraphrase with a more recent frontier model. The generators are also dated (LLaMA-2, Mistral-7B, OPT-2.7B).

Our original evaluation included GPT-3.5, DIPPER, NLLB-200, and BART to examine different transformation mechanisms, including paraphrasing, translation, summarization, style rewriting, and composed attacks. These experiments provide a useful basis for comparison with established watermarking methods and help assess whether the watermark survives different types of semantic and stylistic modifications. However, we recognize that diversity of transformations alone does not establish robustness against more capable, recent language models. We have therefore extended the evaluation to address this concern directly, as detailed below. To test a contemporary rewriter directly, we have now added GPT-5.6 Sol at the documented low reasoning-effort setting in Section 4.2 (p. 7) and Appendix H.3 (p. 35, Table 20) of the revised manuscript. We compare one pass with three scheduled rewriting stages using the original threshold calibrated at 1% null FPR. SS-WM achieves 76.5% and 68.5% TPR, respectively, on admissible final rewrites. SIR achieves 52.5% and 42.5%, SimMark achieves 52.6% and 42.8%, while k-SemStamp achieves 50.5% and 41.5%. Token edit rates rise from 60.0% to 72.5%, while reported admissibility falls from 91.0% to 77.5%. In the revised manuscript, Appendix F.4 (pp. 25–26) explains the independent similarity, bidirectional entailment, and perplexity checks and the fallback when a stage fails. Because the admissibility checks may retain different source texts, the two TPRs compare the evaluated protocols rather than the effect of two additional successful rewrites on an identical set of texts.

Additional evaluation — contemporary GPT-5.6 Sol rewriting (Appendix H.3, Table 20). TPR (%) at the original 1% null-FPR threshold, conditional on admissible final rewrites.

WatermarkSingle passThree scheduled stages
SIR52.542.5
SimMark52.642.8
k-SemStamp50.541.5
SS-WM76.568.5

To address the generator concern, we have now evaluated Qwen3.5-9B in Section 4.1 (p. 6) and Appendix H.3 (p. 36, Table 21) of the revised manuscript. After modern rewriting, the reported TPR is 73.0% for SS-WM, 50.5% for SIR, and 47.0% for k-SemStamp. After the composed attack, the rates are 69.0%, 60.0%, and 57.5%. The Qwen evaluation uses greedy decoding and a generator-specific detector. This evaluates the method on a newer generator after generator-specific enrollment, whereas original Table 15 examined clean-text transfer of a detector fitted on LLaMA-2-7B. These results support different deployment claims.

Additional evaluation — Qwen3.5-9B with generator-specific enrollment (Appendix H.3, Table 21). TPR (%) under the reported attacks; Qwen uses greedy decoding.

WatermarkModern rewritingComposed attack
SIR50.560.0
k-SemStamp47.057.5
SS-WM73.069.0

Weakness 2

I’m also concerned that the semantic channel works by steering the content of the generation toward a key-chosen region of embedding space, so the watermark changes what the model says, not only how it says it.

SS-WM reranks candidate continuations using the base-model score, a prompt-fidelity penalty, and the two watermark penalties in Equation (8). The semantic term acts on candidate continuations, so it can affect which claims, details, or emphasis appear. The prompt-fidelity term discourages divergence under its scoring rule. It does not constrain all candidate continuations to express the same propositions. We agree that this is a genuine design trade-off.

The original submission already tested a practical consequence of that trade-off. Original Section 4.5 and Table 8 compared watermarked and unwatermarked task performance in factual QA, summarization, and exact short-answer generation. Factual QA exact match and summarization ROUGE-L each fell by 0.01, while exact short-answer exact match fell by 0.04 and watermark detection dropped to 63.5%. These results show that the effect depends on the task and how much freedom the model has in producing a valid answer. Aggregate task scores leave open how often an open-ended response changes a claim or omits a detail.

To assess this effect more directly, we have now added a same-prompt comparison in Section 4.6 (p. 10) and Appendix H.6 (p. 37, Table 25) of the revised manuscript. The independent similarity and non-contradiction scores are 0.81 and 78.5% for a marked–unmarked pair, compared with 0.84 and 83.5% for two unmarked samples. This control accounts for some variation between completions from the same prompt even without watermarking. The gap suggests a measurable content-fidelity cost, although it does not establish factual errors in individual outputs. The fidelity-term ablation in Appendix H.5 (p. 37, Table 24) complements this comparison by showing that removing the term increases paraphrase detection from 92.6% to 93.5% while increasing perplexity from 6.58 to 7.00. That ablation characterizes the detection–quality trade-off of the term. The same-prompt and task comparisons are the more direct evidence about content.

Additional evaluation — same-prompt content fidelity (Appendix H.6, Table 25). These automatic measurements are not human judgments or direct factuality guarantees.

Same-prompt comparisonMPNet similarity ↑NLI non-contradiction ↑
Watermarked vs. unwatermarked0.8178.5%
Unwatermarked vs. unwatermarked0.8483.5%

Weakness 3

Quality is measured only with automatic metrics, and the semantic similarity score is computed with the same encoder family used for steering. Also, without a human evaluation, it’s hard to know whether responses become less accurate or drift off topic.

Original Section 4.6 and Appendix G.10 already treated Sentence-BERT similarity as descriptive because its encoder family also participates in the carrier. The original evaluation also included reference-based QA exact match and F1, and summarization ROUGE-L. These reference-based scores provide task-level evidence independent of the carrier encoder. They cover specific reference-based outcomes and leave factual completeness, topical drift, and naturalness in open-ended responses less directly assessed.

In Appendix H.6 (p. 37, Table 25), we have now added all-mpnet-base-v2 similarity and a DeBERTa MNLI non-contradiction measure in the revised manuscript. The former assesses broad semantic alignment between same-prompt completions. The latter estimates contradiction under a separate classifier; it is not a direct test of factual correctness. We retain QA and summarization scores alongside these measures because the measures can disagree. A response can remain on topic while omitting a critical fact, and two responses can agree while both are incorrect. We also distinguish this output comparison from the bidirectional entailment filter for attack candidates in Appendix F.4 (p. 25). The two checks compare different pairs of texts for different purposes. Section 4.6 (p. 10) retains the absence of human assessment as a limitation. These additional measures reduce dependence on the carrier encoder, while human judgments of naturalness and factual completeness remain outside the evaluation.

Weakness 4

The baseline comparison is inconsistent. SemStamp, k-SemStamp, SemaMark, and Unigram Watermark, the closest related methods, appear only in Table 3. All the detailed experiments (per-attack, intensity, adaptive, score-aware, edge-case and cross-model) compare only against KGW, SIR and EXP-Edit, so the strongest claims are made against the weakest baselines.

The original comparison included SemStamp, k-SemStamp, SemaMark, and Unigram Watermark at the same operating FPR, but it did not extend these methods to every detailed attack. In the revised Table 3, SS-WM achieves 88.4% versus 83.9% for k-SemStamp after one-hop translation and 81.2% versus 79.1% after summarization. Those results support a comparative claim for the specific settings in the main table. However, that summary could not establish how these methods rank across the detailed attacks that were not evaluated for them. We have therefore now added SimMark to Table 3 (p. 7), documented the baseline configurations in Appendix F.3 (pp. 24–25) in the revised manuscript, and extended the per-attack comparison to the closer methods in Appendix H.1 (p. 34, Table 18), including DIPPER-1, DIPPER-2, and style rewriting. Tables 11–13 (pp. 31–32) add k-SemStamp and SemaMark to the intensity, adaptive, and score-aware comparisons. Appendix H.3 (pp. 35–36, Tables 20–21) reports closer baselines for modern rewriting and the newer-generator experiment. Appendix H.2 (p. 35, Table 19) separately provides a target-perplexity comparison where watermark strength can be tuned. Matching FPR controls the detector operating point. Matching perplexity controls one aspect of generation quality.

The expanded comparison also shows where SS-WM does not lead. The summarization gap over k-SemStamp is only 2.1 percentage points. In Appendix H.4 (p. 36, Table 22), joint public-feature-aware selection reverses the comparison with SIR, with conditional TPRs of 70.0% for SS-WM and 72.5% for SIR. The edge-case and clean cross-generator transfer tables still cover fewer methods. Section 5 (p. 10) therefore ties the comparative claim to the methods and attacks actually evaluated.

Weakness 5

Finally, I’m not convinced the keys are very distinctive. Each key picks 500 semantic entries over only 32 dimensions, about 15 per dimension, and the targets are averaged. An average of random bin centers in [−4, 4] tends to land near zero, which is the mean of normal text. So the watermark may be pushing text toward “average” text rather than toward something key-specific, and the paper doesn’t look at this.

The original construction in Section 3.4 explicitly averages selected target values that share a coordinate. The reviewer is right to raise the possibility of concentration. With 500 selections across 32 semantic coordinates, the average is approximately 15.6 selections per coordinate under uniform allocation. Distinct entry sets can yield similar averages. The original wrong-key experiment in Section 4.4 and Table 7 supplied an operational check of whether another key accepts a marked text, but its aggregate rates did not show how key confusion depends on target proximity or how target separation compares with natural variation. We have now added that analysis in Section 4.4 (p. 9) and Appendix H.7 (p. 38, Table 26) of the revised manuscript. The median semantic distance between key targets is 4.8, compared with 7.5 between unwatermarked semantic feature vectors. Under this metric, semantic key targets are closer to one another than a typical pair of unwatermarked text feature vectors. For style, the corresponding medians are 21.0 and 14.5. Thus, the empirical separation differs by channel, and the stylistic coordinates contribute separation in the joint space. Pairwise key distances describe how close targets are to one another. Establishing how close their common center is to the unwatermarked mean would require a separate center-to-null comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.