- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how to constrain the source evidence provided by watermarks in individual texts while simultaneously estimating the proportion of watermarked text within a population. The authors propose a "Sequence-level Red-Green Watermark" (RGS), which injects signals via controlled reweighting or rejection sampling, and demonstrate that it satisfies local differential privacy (LDP) relative to the base model. Furthermore, the paper develops a method for proportion estimation, analyzes the interplay between fixed keys, generation diversity, and estimation error, and evaluates the approach’s feasibility across tasks involving C4 dataset continuation, essay writing, and code generation.
Critical strengths
-
Valuable Problem Formulation. The paper shifts the focus of watermarking toward population-level proportion estimation, effectively balancing statistical utility with constraints on source evidence for individual texts. The mechanism is straightforward and offers provable probabilistic bounds on the estimated proportions.
-
Coherent Analysis and Experimentation. The study distinguishes between sampling fluctuations and persistent biases—the latter stemming from insufficient generation diversity—and demonstrates the impact of adjusting the "green" baseline. On the C4 dataset, with and , the mean absolute error (MAE) is 0.85 percentage points.
Critical weaknesses
-
Sensitivity to Common Text Editing. RGS relies on sequence-level hashing; even minor modifications to a text segment can alter its red-green labels. While the authors acknowledge this limitation, they do not evaluate the impact of editing on the accuracy of proportion estimation, leaving the method's effectiveness for text circulating in real-world scenarios unclear.
-
Insufficient Empirical Evidence Regarding Generation Quality. Although the paper claims that the method preserves generation quality, the experiments focus primarily on estimation error. There is a lack of comparative results—such as essay quality or code correctness—before and after the application of the watermark, making the claim that "generation quality is maintained" insufficiently supported.
Questions for authors
-
In the experiments, the non-watermarked (unwatermarked) texts are drawn from the same base model. In real-world settings, however, the background corpus would likely include human-authored content and outputs from other models. Since the estimation formula assumes that the "green" proportion in unwatermarked text remains close to the theoretical baseline, does this approximation still hold when the data originates from diverse sources? If the proportion deviates from the baseline, does this introduce additional bias or increased variance in the estimation?
-
Because RGS requires rejecting certain candidates and regenerating text, the actual computational cost depends on the number of resampling iterations performed. Compared to direct generation using the base model, how much do the average number of generation attempts and the actual elapsed time increase? Do these overheads vary significantly depending on the task type or the level of privacy protection?
Thank you for taking the time to review our submission and offer valuable feedback. We address your questions and concerns in our response below.
Diverse data sources
Although our experiments use unwatermarked text from the same base model, our method allows for arbitrary text distributions pi. We have updated the paper to highlight this. Under Assumption 1, we have that Ek[gi(k)]=π0 for arbitrary distributions pi.
For a fixed key k, the unwatermarked green probability gi(k) can deviate from π0. This leads to estimator bias, which we denote as U(k)=I01∑i∈I0gi(k)−π0 in Section 5.2. We note that Lemma 1 directly shows that greater diversity among unwatermarked texts makes gi(k) concentrate more tightly around π0, thus reducing the bias.
For a fixed key, under Assumption 1, the green indicator Hi=1hk(Yi)=G is a Bernoulli random variable with success probability gi, whose variance is gi(1−gi). Therefore deviations of gi from π0 changes the variance, but the overall effect on variance depends on the value of π0 and the direction of the deviation. More importantly, we note that the variance term of estimator error vanishes like O(1/n) for fixed π0,π1 (see Proposition 2). Therefore, bias is the main concern at larger sample sizes.
Text editing
We agree that editing can have nontrivial impacts on estimation, and that our experiments do not test RGS's robustness to editing. However, we would like to highlight that the strong privacy guarantees of RGS are intended to reduce user incentives for editing. Also, we could compose RGS with various standard techniques for making watermarks robust, like locality sensitive hashing, or hashing with some error correcting codes.
Quality preservation
Please see our discussion in response to all reviewers.
Rejection Sampler efficiency
Please see our discussion in response to all reviewers.