- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
In this paper, a watermarking method, HiddenMark is studied. This method perturbs generation along a direction in the vocabulary space, derived from the LM head, so that its effect on every next-token distribution can be written in a closed form. The watermark embedder sets a budget with a total cross-entropy allowance. When watermarking is applied, it uses bisection to choose the largest value of Δ_t in [0, Δ_max] whose expected overhead stays within the step’s allotment.
The authors provide theoretical arguments and conduct experiments on AceReason-Nemotron-7B, Llama-3.1-Nemotron-Nano-8B, and Llama-3.1-8B-Instruct to study the effectiveness of the method.
Critical strengths
-
The idea is clear and the algorithm with a closed form is easy to implement.
-
I think using a cross-entropy budget to adjust the watermark strength is useful for obtaining a better trade-off between detection performance and text quality.
Critical weaknesses
-
In the Axiom 2 of the paper, a low false positive rate is claimed to be applied for "a single key held fixed across all texts", and "a fixed-key-valid null" is proposed as one of the contributions in page 3. However, the theorem 2 in this paper seems to provide the bound over the randomness of the key and is not suitable for a fixed key. It seems the theory does not fully support the claim of the goal.
-
Some components of the proposed method in this paper seem to be closely related to previous work that is not cited in the paper.
For example, the paper "A Unified Framework for LLM Watermarks", which is available in https://arxiv.org/pdf/2602.06754 has already proposed watermarking method under a token-level cross-entropy budget. See Equation (14) in that paper, which is equivalent to the expected overhead used in Section 4.2 of the present manuscript. This paper should be cited, and the authors should compare their method with the method proposed in this paper.
Also, there are only 2 baselines in this paper. The method should also be compared with the following baseline: https://proceedings.neurips.cc/paper_files/paper/2024/file/21b5883bc8fec922fdbbb06675388164-Paper-Conference.pdf, which also study the trade-off between detectability and text quality.
Questions for authors
Please see the weakness section. Could the authors add more relevant related works and do a more comprehensive comparision with more baselines?