- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how to identify passages whose watermark proportions exceed a chosen threshold. FissionGuard uses data fission to separate the evidence for locating passages from the evidence for testing them. It then uses partial-conjunction tests to account for the watermarked tokens allowed under the null, followed by online multiple testing. The paper proves FDR control at fixed finite horizons under stated assumptions and gives power bounds. Simulations and experiments on Mistral and Qwen show improved token coverage over sample splitting.
Critical strengths
-
The paper addresses a useful problem. Detecting watermark presence does not establish that a substantial fraction of a passage is watermarked. Testing a chosen proportion threshold gives a more informative claim about mixed-source text.
-
The method handles selection and testing carefully. Data fission separates passage localization from validation, while partial-conjunction testing accounts for the strong signals allowed under the null. The FDR argument clearly connects these steps under the stated assumptions.
-
The experiments support the main idea. The paper checks calibration near the proportion-null boundary and shows coverage gains over sample splitting. For example, on Mistral text with random marking, coverage increases from 87.7% to 95.3%, with empirical FDR below the target.
Critical weaknesses
-
The text experiments need independent validation. Appendix E states that the model-generated texts were used during method development and that the main temperatures were chosen based on watermark evidence strength. This limits how confidently the reported gains can be extended to new texts and generation settings.
-
The practical requirements need clearer guidance. The experiments use position-dependent seeds, with source identifiers added for internally mixed texts. The paper should clarify what position and request information a detector needs when applying the method across multiple texts.
-
Sensitivity to the detection settings is not fully studied. The experiments fix the fission allocation and the alarm and localization windows. More guidance is needed on choosing these settings and on how they affect coverage and purity, especially for short passages or weak watermark signals.
Questions for authors
-
Independent evaluation. Do the coverage gains remain on held-out prompts after all method settings are fixed? In particular, how well does the method perform on internally mixed text at lower generation temperatures?
-
Requirements for practical use. What position and request information must be available to reproduce the experimental seeds? How should randomness reuse be prevented when the same key is used across multiple requests?
-
Choice of the detection settings. How should users choose the fission allocation and scanning windows for a new application? Are the coverage gains over sample splitting stable across reasonable choices of these parameters?
4 comments
Response to Reviewer n8mk (Part II)
Q2. (a) Requirements for practical use. What position and request information must be available to reproduce the experimental seeds?
Response. Thank you for these questions. Given the watermark key, the detector needs the original generation request ID, the token’s position in that request and the preceding four token IDs. Using this information, we reproduced every token-level p-value exactly after splitting each internally mixed text into multiple monitoring requests.
(b) How should randomness reuse be prevented when the same key is used across multiple requests?
Response. Assign a unique ID to each original generation request and use it when computing the seeds [1]. Different request IDs keep the hash inputs distinct even when token positions and contexts repeat under the same key.
Q3. (a) Choice of the detection settings. How should users choose the fission allocation and scanning windows for a new application?
Response. We appreciate your comments. We recommend starting with ρ=0.5, a minimum alarm spacing of 64 tokens and a minimum localization length of 32 tokens. Our response to your W3 explains these choices, and the comparisons below show how coverage and purity change when they are varied.
(b) Are the coverage gains over sample splitting stable across reasonable choices of these parameters?
Response. FissionGuard had higher token coverage on both models across the allocations and window settings compared below. On the original Table 2 texts, we matched FissionGuard’s ρ to Splitting’s alarm-side token fraction and also reversed the original odd/even assignment at 50/50. Raising ρ from 0.25 to 0.75 increased purity by about four percentage points on both models and reduced coverage by less than one point.
Token coverage and purity by allocation on the original Table 2 texts (%)
| Method | FissionGuard ρ | Splitting alarm-side token fraction | Mistral (T=1.5) Cov. / Purity | Qwen (T=2.0) Cov. / Purity |
|---|---|---|---|---|
| FissionGuard | 0.25 | — | 96.13 / 80.27 | 96.61 / 80.70 |
| Splitting | — | 25% | 94.51 / 82.77 | 95.21 / 83.17 |
| FissionGuard | 0.5 | — | 95.88 / 82.74 | 96.59 / 82.98 |
| Original Splitting | — | 50% | 92.39 / 85.56 | 93.68 / 85.67 |
| Swapped parity | — | 50% | 90.89 / 85.74 | 93.06 / 85.43 |
| FissionGuard | 0.75 | — | 95.52 / 84.37 | 96.23 / 84.46 |
| Splitting | — | 75% | 0.02 / 100.00 | 0.05 / 100.00 |
Original Splitting uses odd positions for alarms, and swapped parity uses even positions. At 75%, Splitting made only one report on Mistral and three on Qwen, so its 100% purity reflects very few reports.
To assess window sensitivity, we varied the minimum alarm spacing and minimum localization length on the original Table 2 texts, with the other settings unchanged. Each remaining row changes one.
Token coverage by minimum alarm spacing and localization length (FissionGuard / Splitting, %)
| Minimum alarm spacing (tokens) | Minimum localization length (tokens) | Mistral (T=1.5) | Qwen (T=2.0) |
|---|---|---|---|
| 64 (default) | 32 | 95.88 / 92.39 | 96.59 / 93.68 |
| 32 | 16 | 96.93 / 90.34 | 97.48 / 91.64 |
| 64 | 16 | 97.85 / 93.25 | 98.33 / 94.19 |
| 128 | 32 | 96.24 / 91.62 | 96.63 / 93.24 |
| 256 | 32 | 94.21 / 78.64 | 95.89 / 78.97 |
Reducing only the minimum localization length improved coverage with little change in purity. The longest minimum alarm spacing reduced both token coverage and purity. Empirical passage-level FDR remained below the target.
We generated 1,024- and 2,048-token texts at the Table 2 temperatures, scaling the watermarked regions and gaps in proportion to input length. For each model and input length, we generated 100 texts per layout. We compared the Table 2 defaults with the 32/16 setting (alarm spacing/localization length) above.
Token coverage pooled across the four layouts (%, Table 2 defaults → minimum alarm spacing 32 and minimum localization length 16)
| Input length (tokens) | Mistral (T=1.5) FissionGuard | Mistral (T=1.5) Splitting | Qwen (T=2.0) FissionGuard | Qwen (T=2.0) Splitting |
|---|---|---|---|---|
| 2,048 | 92.73 → 95.12 | 84.83 → 87.66 | 94.04 → 95.78 | 88.30 → 90.08 |
| 1,024 | 83.81 → 90.14 | 58.83 → 71.64 | 85.04 → 91.68 | 66.75 → 79.12 |
The 32/16 setting increased coverage and purity for both methods, with larger coverage gains on the 1,024-token texts. FissionGuard continued to cover more watermarked tokens. Empirical passage-level FDR remained below the target.
Our response to your W3 discusses the paper’s weak-signal results. Our response to your Q1 reports lower-temperature text results.
References
[1] Christ, M., Gunn, S., & Zamir, O. (2024). Undetectable watermarks for language models. In S. Agrawal & A. Roth (Eds.), Proceedings of Thirty Seventh Conference on Learning Theory (Vol. 247, pp. 1125–1139). PMLR.
Thanks for the detailed response. I have maintained my rating .
Thank you for your careful review and taking time to read our response. We appreciate your helpful suggestions.
Response to Reviewer n8mk (Part I)
Response. Thank you for your comments. The Table 2 temperatures were fixed before generating the original internally mixed texts. The original corpora were used in development, so we evaluated new held-out texts with fixed detector settings. Our response to your Q1 reports the results.
We apologize for the ambiguity in Appendix E. Earlier detector comparisons informed the Table 2 temperatures. We chose each lower temperature before generating its held-out texts. We will clarify this description in Appendix E.
Response. The coverage gains persisted on new held-out texts at the additional lower temperatures. Our response to your Q1 reports the results, including the lower temperatures for each model.
Response. We appreciate your comments. The detector needs the original generation request ID and token positions, even when the monitoring request boundaries change. We will make this explicit in Appendix E. Our response to your Q2 gives the full requirements and verifies that the token-level p-values are reproduced exactly.
Response. Thank you for your careful reading. We use ρ=0.5, a minimum alarm spacing of 64 tokens and a minimum localization length of 32 tokens. The 64-token spacing lets the alarm rule accumulate evidence after each restart and matches its shortest scanning window. The 32-token minimum allows localization to select a shorter passage within that window. FissionGuard had higher token coverage on both models across the allocation and window choices reported in our response to your Q3.
At ρ=0.5, fission allocates equal information to alarms and validation. Larger ρ gives more information to alarms and localization and less to validation.
For shorter expected passages, the smaller alarm spacing and localization length tested in our response to your Q3 can improve recovery. These choices preserve the FDR guarantee in Theorem 3.3.
The paper’s controlled simulations show that weaker watermark signals reduce recovery with the detector settings fixed, consistent with the dependence on signal strength in the power bounds of Section 3.3. Our response to your Q1 reports lower-temperature text results.
Response. Thank you for this helpful suggestion. With both detectors fixed before evaluation, FissionGuard had higher token coverage than 50/50 Splitting on held-out prompts for every internally mixed layout and temperature in the table below.
Response. The coverage gains also persisted at the lower temperatures shown below. We generated new texts at each temperature using the same held-out prompts and evaluated them with the detector settings from Table 2.
Token coverage on held-out Mistral-7B-Instruct-v0.3 and Qwen3-8B texts at lower temperatures (FissionGuard / Splitting, %)
Each cell contains 100 texts. FissionGuard’s empirical passage-level FDR ranged from 0.00% to 0.42% across the table.