acceptodds
Under review as a conference paper at ICLR 2027

HiddenMark: Analytic Hidden-State Watermarking with Closed-Form Guarantees

Abstract

A provider deploying an LLM watermark has to commit to two numbers before the first token is generated: how much task accuracy the watermark will cost, and how much output must accumulate before detection is reliable. Existing watermarks answer both only after the fact, by running the watermark and measuring. We formalize the requirement as a Predictability Standard, give a certification procedure that applies to any watermark, and give a watermark designed to meet the full standard. HiddenMark perturbs generation along a single known direction in vocabulary space, derived from the LM head, so that its effect on every next-token distribution is available in closed form. This lets a deployer specify a budget that the controller enforces, and lets HiddenMark certify the outcome before generation. From one 30-prompt probe the procedure issues two certificates with finite-sample coverage guarantees, validated on HiddenMark, KGW and SynthID deployments: the output length needed to reach 95% detection (88% realized coverage at 90% nominal, within a factor of 2.0 over a 996-fold range) and a lower bound on the change in task accuracy. The detection certificate is as tight as the outcome can be measured: its error matches the disagreement between two independent 100-prompt evaluations of the same deployment. Under the single key a provider actually deploys, the usual closed-form threshold inflates the false positive rate by up to ; one precomputed scalar brings the inflation to –. Building the certificates exposed why watermarks destroy task accuracy while perplexity stays flat, and fixed the design: a spread-spectrum direction that bounds how far any token can be promoted. On GSM8K with a model that passes a qualification statistic computed from plain generation, HiddenMark matches KGW and SynthID on the accuracy–detection trade-off, and unlike them exposes a budget that is enforced during generation and a detector whose null holds under a deployed key; on a model the statistic rejects, it falls behind KGW.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM watermarking, Predictability Standard, hidden-state perturbation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_TzHhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

In this paper, a watermarking method, HiddenMark is studied. This method perturbs generation along a direction in the vocabulary space, derived from the LM head, so that its effect on every next-token distribution can be written in a closed form. The watermark embedder sets a budget with a total cross-entropy allowance. When watermarking is applied, it uses bisection to choose the largest value of Δ_t in [0, Δ_max] whose expected overhead stays within the step’s allotment.

The authors provide theoretical arguments and conduct experiments on AceReason-Nemotron-7B, Llama-3.1-Nemotron-Nano-8B, and Llama-3.1-8B-Instruct to study the effectiveness of the method.

Critical strengths
  1. The idea is clear and the algorithm with a closed form is easy to implement.

  2. I think using a cross-entropy budget to adjust the watermark strength is useful for obtaining a better trade-off between detection performance and text quality.

Critical weaknesses
  1. In the Axiom 2 of the paper, a low false positive rate is claimed to be applied for "a single key held fixed across all texts", and "a fixed-key-valid null" is proposed as one of the contributions in page 3. However, the theorem 2 in this paper seems to provide the bound over the randomness of the key and is not suitable for a fixed key. It seems the theory does not fully support the claim of the goal.

  2. Some components of the proposed method in this paper seem to be closely related to previous work that is not cited in the paper.

For example, the paper "A Unified Framework for LLM Watermarks", which is available in https://arxiv.org/pdf/2602.06754 has already proposed watermarking method under a token-level cross-entropy budget. See Equation (14) in that paper, which is equivalent to the expected overhead used in Section 4.2 of the present manuscript. This paper should be cited, and the authors should compare their method with the method proposed in this paper.

Also, there are only 2 baselines in this paper. The method should also be compared with the following baseline: https://proceedings.neurips.cc/paper_files/paper/2024/file/21b5883bc8fec922fdbbb06675388164-Paper-Conference.pdf, which also study the trade-off between detectability and text quality.

Questions for authors

Please see the weakness section. Could the authors add more relevant related works and do a more comprehensive comparision with more baselines?

Reviewer_u8ziOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes HiddenMark, a logit-perturbation watermark built from a fixed direction in vocabulary space derived from the LM head, together with a "Predictability Standard" consisting of three axioms: the deployer specifies a per-token quality budget that the system enforces during generation and certifies a bound on task-accuracy change before deployment; the detection threshold comes from a closed-form null valid under a single fixed key; and a statistic computed from plain generation decides whether a model can carry the certificates at all. The direction is binarized (w_v ∈ {±1}) to bound per-token probability amplification, addressing what the authors identify as the mechanism by which watermarking destroys task accuracy while leaving perplexity intact. From a 30-prompt probe, split-conformal procedures issue two pre-deployment certificates (output length for 95% detection; a lower bound on accuracy change), validated on HiddenMark, KGW, and SynthID. The paper further shows that the standard closed-form null drifts under a fixed deployed key (inflating false positives by up to 778×) and corrects it with one scalar. Experiments are on GSM8K, IFEval, MMLU-Pro, and HumanEval with three models.

Critical strengths
  1. The amplification analysis gives a testable mechanism for the perplexity–accuracy gap, not just another quality metric.

  2. The fixed-key drift result exposes a real gap between evaluation and deployment.

  3. The certificates are the right framing, and they are validated rather than asserted.

Critical weaknesses
  1. The paper is very hard to follow. I want to be precise about why, because the paper is not disorganized so much as written for a reader who already knows the intended argument. The core structure — three axioms, a probe, two conformal certificates, a qualification statistic, a fixed-key correction, and a spread-spectrum direction — is introduced piecemeal across Sections 1, 3, 4, and 5, with each component defined only partially before it is used.

  2. The technical contribution is difficult to isolate from prior work, and several components appear to be repackaged known ideas. As best I can reconstruct it, the method is: (a) a fixed sign-vector direction added to logits, which the paper itself notes "is structurally a fixed green list, as in Unigram (Zhao et al., 2023), whose bias flips sign with the keyed position"; (b) an entropy gate and per-token bisection controller, which resembles entropy-gated and adaptive-strength KGW variants (Lee et al., 2024; Liu & Bu, 2024); (c) a centered z-statistic, which is a standard variance-reduction step; and (d) split-conformal prediction, which is a standard tool. The genuinely new pieces appear to be the combination — specifically, the observation that amplification A_t (Prop. 1) is the mechanism linking per-token distortion to task accuracy, the fixed-key drift analysis (Prop. 2), and the qualification statistic ρ_W. These are interesting, but the paper does not clearly delineate them from the standard components, and a reader is left unsure what would be lost if, say, the direction were not binarized but the controller and certificates were retained.

  3. The fixed-key drift analysis, while interesting, is presented with a mismatch between the theory and the deployment protocol. Proposition 2 shows that under a fixed key the mean of the uncentered statistic is O(1), not shrinking in n. This is correct and worth pointing out. But the paper's own detection procedure (§4.3) and the deployed protocol (§6.4) center at c estimated from the first 30 plain texts, and the empirical check uses 64–96 keys with the worst key reported. The theorem is about the key distribution; the deployment is about one key. The claim that centering "reduces the inflation to 5–7×" is an empirical observation on two models, not a consequence of the theory — the residual term Σ_t s_t ε_t in the proof sketch is not bounded by the centering argument. The paper would be stronger if it stated clearly what the centering does and does not guarantee, and whether the residual 5–7× inflation is acceptable for a deployed false-positive level.

  4. The evaluation is narrow, and the "parity" claim is weaker than it sounds. The main comparison (Table 2) is on GSM8K with one qualified model (AceReason-7B), where HiddenMark's accuracy changes are within noise of KGW and its detection length (2,015–4,631 tokens) is comparable to KGW δ=4 (2,420) and far better than SynthID (286,750). But: the accuracy certificate's tightest bound (−9.0 pp) comes from 12 calibration cells drawn from three (model, task) pairs, which the paper itself says is "too few to test transfer to an unseen pair." The detection certificate is validated on 48 cells but the realized coverage at 90% nominal is 88%, within one standard error — fine, but the bound is a factor of ~2, which is not a tight prediction. And the qualification statistic ρ_W ≥ 0.985 is calibrated on three models. So the "parity" claim is really "comparable on one task with one qualified model, with certificates whose coverage is validated on a small and non-diverse calibration set." That is a reasonable empirical result, but the framing in the abstract ("matches existing watermarks on a qualified model and adds a budget enforced during generation and a null that holds under the deployed key") overstates the strength of the evidence.

  5. The relationship between the quality budget and task accuracy is asserted more than demonstrated. Axiom 1 asks for a bound on task-accuracy change. The controller enforces a per-token cross-entropy allowance, and §6.1 shows that this allowance does not bound accuracy (a run can meet a small divergence budget and lose 28 points). The paper's resolution is to certify accuracy via conformal prediction from a probe, not from the budget. So the budget is not actually the quantity that yields the accuracy certificate — the probe is. This is a coherent design, but it means the "quality predictability" axiom is satisfied by a statistical procedure that is independent of the mechanism the controller enforces. The paper should either (a) show that the budget constrains the probe's inflation ι in a way that makes the certificate tighter, or (b) be explicit that the budget is a heuristic input and the certificate is the actual guarantee. As written, the connection between the two is unclear.

Questions for authors
  1. Can you provide a single end-to-end worked example in the main text? 
  2. What exactly is new relative to Unigram (Zhao et al., 2023) and entropy-gated KGW? 
  3. Does the fixed-key correction have a theoretical guarantee, or only an empirical one? 
  4. How does the probe-based accuracy certificate relate to the enforced budget?
  5. How should the parity claim be interpreted given the calibration-set limitations? 
  6. Why largely changed the title from “HiddenMark: Analytic Hidden-State Watermarking with Closed-Form Guarantees” to “WATERMARKS THAT CERTIFY THEIR COST BEFORE DEPLOYMENT”
Reviewer_tp4qOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper claims to theoretically model the detection-quality tradeoff in LLM watermarking, and propose a watermarking method operating on the LLM latent space (LM head) that makes this tradeoff tractable.

Critical strengths

The paper's premise is motivated: indeed most watermarking schemes have a strength hyperparameter (e.g. gamma and delta for KGW, the depth in SynthID) that controls the detectability-distortion/diversity tradeoff, but the measurement is empirical. The authors claim to be able to certify this tradeoff beforehand.

Critical weaknesses

The paper is inconsistent (evaluation on "30 samples", so it's indeed an empirical measurement), densely written, and very difficult to understand for no reason. The experimental section does not resemble a typical paper from the LLM watermarking literature. The results seem questionable.

Questions for authors

Can you present a simple detectability vs quality tradeoff of your method vs baselines? Simple quantification: TPR@FPR or pvalue, versus a text quality metric. Competing methods (e.g. KGW) should trace a curve based on hyperparameter choices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.