acceptodds
Under review as a conference paper at ICLR 2027

Functional Subspace Watermarking for Large Language Models

Abstract

Model-side watermarking protects the ownership of large language models (LLMs) by leveraging model weights or internal representations. However, these internal signals often undergo complex distortions during practical model modifications, such as fine-tuning, quantization, pruning, and knowledge distillation, making reliable watermark extraction extremely challenging. Although model-side watermarking has been widely studied, existing methods still lack sufficient robustness against parameter-level and representation-space perturbations. To address this issue, we propose Functional Subspace Watermarking (FSW), which embeds multi-bit watermark information into a low-dimensional functional backbone of LLM internal representations. Specifically, FSW solves a generalized eigenvalue problem to jointly model functional sensitivity and compression invariance, thereby extracting watermark directions that are both task-critical and stable under common model modifications. We further introduce an adaptive spectral truncation strategy to select an appropriate spectral band that balances robustness and model utility, while incorporating a vector consistency constraint to mitigate representation drift during watermark injection. Experiments show that FSW achieves 100% bit recovery accuracy across diverse LLM backbones and, while maintaining competitive utility on most evaluated backbones, preserves watermark detectability under LoRA fine-tuning, noise perturbation, pruning, quantization, and backbone-preserving distillation. In addition, we further evaluate FSW under untargeted representation perturbation and homogeneous output-only distillation. The results show that, under the tested budgets, FSW retains statistically detectable watermark signals across multiple challenging configurations, demonstrating its robustness to common model modifications while characterizing its practical robustness boundary.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language model watermarking, model ownership verification, functional subspace, robust watermarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aYeNOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

LLMs are valuable IP for their developers. To protect the IP, this paper proposes FSW, which aims to embed watermark into LLMs and detect it with white-box access to the suspected model. The basic idea of FSW is to fine-tune the model by selecting internal representation directions, which are important to maintain functionality of LLMs and robust to weight modification. As a result, such directions could be verified with private vectors as watermark signal. Further experiments demonstrated the effectiveness of FSW. Despite this paper is overall well-written and the method design is concrete and sound, I have some concerns on its design choices and lack of analysis experiments (see questions). Therefore, I will give it score of 2. If all the questions are addressed during rebuttal, I would be open to improve my score.

Critical strengths
  1. the motivation of LLM IP protection is concrete
  2. the proposed method is sound and easy to understand.
  3. the writing is good and the paper is well structured.
  4. the experiments are fairly strong in breath
Critical weaknesses
  1. the design choices are not well explained(see questions)
  2. some analysis experiments about important hypers are missing(see questions)
Questions for authors
  1. The generalized eigenspace is optimal for the proposed trace surrogate, but the paper does not establish why this surrogate is the appropriate objective for watermark robustness. Besides, the author did not explain why the resulting FSW design is preferable to alternative subspace constructions. Could the authors provide stronger theoretical justification for these design choices?
  2. how to pick τlower\tau_{lower} and τupper\tau_{upper}? Since they decide what eigenvalues would be chosen, I believe they are vital for FSW. However, I did not see any analysis experiments over the choice on them.
  3. It seems that evaluation about adaptive attack is absent (e.g. the attackers have known FSW), which I think is worthy to explore for a model watermarking algorithm. Could the authors provide the settings of adaptive attack and how to defend it?
9 comments
AuthorsOpenReview ↗

Response to Reviewer aYeN (1/3)

1. Response to Question 1: Why is the surrogate objective appropriate for watermark robustness, and what advantages does FSW’s subspace selection offer?

We thank the reviewer for asking about the design rationale. FSW jointly considers task relevance and stability under model modifications, and uses spectral-band selection to balance watermark robustness and model utility.

Design rationale. In §4.2, u⊤Fuu^\top Fu in Eq. (2) measures the local sensitivity of the language modeling loss to direction uu, while u⊤CAuu^\top C_Au in Eq. (3) measures the expected squared perturbation along that direction under the modeled transformations. The former alone does not ensure stability after modification, while the latter alone does not ensure task relevance. Equation (4) therefore uses their ratio to jointly evaluate candidate directions. This surrogate has an explicit connection to watermark detection. Proposition 2 in Appendix B.2 establishes

\|U^\top\Delta r(x,a)\|_2^2 = \operatorname{tr}(U^\top C_AU).$$ Corollary 1 in Appendix B.3 further shows that the change in the detection score is bounded by the drift in projected coordinates. Thus, among directions with comparable task relevance, smaller projected perturbations help control detection-score drift. This provides a rationale for jointly considering task relevance and modification stability that is directly connected to the detection statistic. **Advantages over alternative selection strategies.** Proposition 1 in Appendix B.1 establishes the optimality of generalized eigenvectors for the stated trace surrogate. The final construction must also account for the effect of watermark embedding on model utility. FSW therefore applies the spectral-band selection in Eq. (6) to this ordering. The original manuscript evaluates the contributions of both joint modeling and spectral-band selection: - **Joint modeling.** Table 5 in §5.3 shows that removing compression-invariance modeling reduces the detection score after the attack from **5.84** for full FSW to **0.96**. This supports explicitly accounting for modification-induced perturbations alongside functional sensitivity. - **Spectral-band selection.** §5.3, Figure 3, and Table 11 in Appendix H show that, under backbone-preserving distillation on LLaMA-2-7B, FSW achieves **95.90%** detection-score retention, compared with **80.77%, 74.04%, 65.48%, and 78.11%** for Top-k, Mid-50%, Low-75%, and Last-k, respectively. FSW also achieves the highest detection score after distillation, **5.8438**, and the lowest PPL after distillation, **5.9139**, among these strategies, demonstrating a better balance between detection-signal preservation and model utility. **Manuscript clarification.** In the revised manuscript, we will clarify the connection between task relevance, projected perturbations, and detection-score stability in §4.2, with explicit references to Table 5, Figure 3, and Table 11. We will also distinguish theoretical optimality under the trace surrogate from the empirical evidence supporting the final spectral-band selection. Together, these theoretical results connect FSW’s surrogate objective to detection stability, while the experiments in the original manuscript support the practical advantages of joint modeling and spectral-band selection.
AuthorsOpenReview ↗

Response to Reviewer aYeN (2/3)

2. Response to Question 2: How are the spectral thresholds selected, and how do threshold changes affect direction selection?

We thank the reviewer for asking about spectral threshold selection and sensitivity. We determined the thresholds through experimental comparisons, balancing watermark performance and model utility. As stated in §5.1 of the original manuscript, the subspace dimension and spectral thresholds are selected under validation-set perplexity constraints. Appendix D.3 reports the final settings:

(τlower,τupper)=(10−4,0.6).(\tau_{\rm lower},\tau_{\rm upper})=(10^{-4},0.6).

Selection rationale and procedure. Following Eq. (6) in §4.2, we first retain directions satisfying

\leq \frac{\lambda_i}{\lambda_1} \leq \tau_{\rm upper},$$ and then select the top $k$ eligible directions in descending eigenvalue order. The lower threshold filters out directions with low functional sensitivity relative to the modeled perturbations. The upper threshold limits direct selection from the leading end of the spectrum to account for the effect of watermark embedding on model utility. Once determined, these thresholds are kept fixed. §5.3, Figure 3, and Table 11 in Appendix H provide experimental support for the final spectral-band selection. Under backbone-preserving distillation on LLaMA-2-7B, this selection achieves **95.90%** detection-score retention, compared with **80.77%** for direct Top-k selection. Its PPL after distillation is also lower, at **5.9139**, compared with **6.6026** for Top-k. These results support spectral-band selection as a way to balance detection-signal preservation and model utility. **Additional threshold sensitivity analysis.** To directly examine how the thresholds affect the selected directions, we performed fixed-spectrum scans on Qwen2.5-7B using four construction datasets: 125, 250, and 500 WikiText-2 samples, and 500 C4 samples. We fixed $k=32$ throughout. For each dataset, we kept the computed eigenvalues and eigenvectors unchanged and varied only the thresholds. | Threshold setting | Results across the four construction datasets | | --- | --- | | Fix the upper threshold at 0.6 and set the lower threshold to $10^{-5}$, $10^{-4}$, $10^{-3}$, or $10^{-2}$ | Each dataset yields exactly the same 32 directions as its corresponding default setting | | Fix the lower threshold at $10^{-4}$ and change the upper threshold to 0.3 | **25–31 directions** are shared with the corresponding default selection | | Fix the lower threshold at $10^{-4}$ and change the upper threshold to 0.9 | **30–32 directions** are shared with the corresponding default selection | Under the default thresholds, the four datasets provide **110, 216, 395, and 409** eligible directions, respectively. All provide enough candidates to select 32 directions without invoking the fallback procedure. At the four tested lower thresholds from $10^{-5}$ to $10^{-2}$, the final selection remains unchanged for each dataset because the top 32 eligible directions continue to satisfy the respective lower thresholds. Changing the upper threshold affects part of the selection: setting it to 0.3 replaces **1–7 directions**, while setting it to 0.9 replaces **0–2 directions**. Thus, the upper threshold has a more direct effect on the final selection in these settings. We also examined a stricter lower threshold of 0.1. This leaves fewer than 32 eligible directions for all four datasets and triggers the implementation’s fallback procedure, which skips the top 1% of directions in the eigenvalue ranking and selects the next 32. None of the four default configurations triggers this procedure. These results complement the performance comparisons in the original manuscript by directly showing how the thresholds affect the number of eligible candidates and the final selected directions. **Manuscript clarification.** In the revised manuscript, we will consolidate the experimental basis for threshold selection, their fixed use, the sensitivity results above, and the handling of insufficient candidates in §4.2 and Appendix D.3. Together, the thresholds are determined through experimental comparisons that balance watermark performance and model utility, the original experiments support the effectiveness of the final spectral-band selection, and the additional scans clarify how threshold changes affect the selected directions.
AuthorsOpenReview ↗

Response to Reviewer aYeN (3/3)

3. Response to Question 3: How is FSW evaluated and protected against adaptive attacks when the attacker knows the method?

We thank the reviewer for raising the question of adaptive attacks. We have evaluated an attack that directly exploits FSW’s public construction procedure: the attacker reconstructs a candidate subspace using independent public data and then suppresses that subspace to attempt watermark removal. Section 3 and Appendix F of the original manuscript describe this setting, and Table 9 reports the results at the maximum tested suppression strength. Across two models, three seeds, and four attack strengths, all 24 attacked checkpoints pass verification and retain 100% bit accuracy.

Attack setting. The attacker has white-box access to the released model and knows the watermark layer, subspace dimension kk, spectral thresholds, and public generalized eigenvalue procedure. Using independent public data, the attacker re-estimates F^\hat F and C^A\hat C_A, constructs a candidate subspace U^\hat U, and suppresses it. This attack targets the entire estimated subspace without first recovering the individual watermark keys, directly testing whether the attacker can exploit FSW’s public design to locate and remove the watermark. The evaluation covers LLaMA-2-7B and Qwen2.5-7B, with three seeds per model and suppression strengths

ρ∈{0.25,0.5,0.75,1.0}.\rho\in\{0.25,0.5,0.75,1.0\}.

Pre- and post-attack evaluations are paired using the same underlying model checkpoints and held-out split, yielding 24 attacked checkpoints.

Experimental results. Across the four strengths, each model passes verification on 12/12 attacked checkpoints, giving 24/24 overall, all with 100% bit accuracy. Appendix F, Table 9 reports the results at the maximum tested strength, ρ=1.0\rho=1.0. Scores below are means and standard deviations across three seeds:

ModelPre-attack scorePost-attack scoreDetection thresholdPost-attack bit accuracyPost-attack passes
LLaMA-2-7B5.766 ± 1.7525.786 ± 1.7121.954100%3/3
Qwen2.5-7B4.948 ± 0.3344.958 ± 0.3431.024100%3/3

Neither model shows a notable decrease in detection score at the maximum tested strength. Table 9 also reports small perplexity changes, while HellaSwag and ARC-Easy performance remains essentially unchanged in the full attack evaluation.

Explanation and verification procedure. Appendix F further examines whether the attacker accurately reconstructs the watermark-bearing subspace. The overlap between the reconstructed and owner subspaces is 0.00397 ± 0.00010 for LLaMA-2-7B and 0.00952 ± 0.00031 for Qwen2.5-7B, close to the respective random-overlap levels of 0.00391 and 0.00893. This explains the observed attack outcome: under the tested settings, reconstruction from independent data does not accurately locate the actual watermark-bearing directions, so suppressing the estimated subspace does not effectively remove the detection signal. As specified in Section 3, FSW’s algorithm and hyperparameters may be public, while the keys, target-bit mapping, and challenge set remain private. The experiment further tests whether the attacker can bypass individual-key recovery by reconstructing and suppressing the entire subspace. After the attack, the owner continues to use the original verifier without re-embedding the watermark or changing the detection rule in response to the attack.

Manuscript clarification. In the revised manuscript, we will highlight this adaptive attack more prominently in the main text, consolidating the attacker’s knowledge, the reconstruction and suppression procedure, and verification results across all four strengths, with explicit references to Appendix F and Table 9. Together, these evaluations directly support FSW’s continued detectability under the tested public-information subspace-reconstruction and suppression attack.

Reviewer_BayHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the limitations of model watermarking in adversarial settings, the paper proposes Functional Subspace Watermarking (FSW). The method constructs a low-dimensional functional subspace by solving a generalized eigenvalue problem using the Fisher information matrix and a perturbation covariance matrix derived from predefined representation perturbations. Watermark embedding is realized via lightweight fine-tuning with a combined loss of language modeling, representation consistency and hinge loss, embedding the watermark as geometric statistical constraints within this subspace. The authors conduct experiments to demonstrate the effectiveness of the method.

Critical strengths

1.The paper is well written and easy to follow. 2.The overall design of the method is self-consistent: the Fisher matrix measures functional sensitivity, the compression-invariance matrix measures the variance of representation perturbations, and the generalized Rayleigh quotient selects candidate directions. The watermark embeds via optimizing the joint loss while maintaining stable content quality.

Critical weaknesses

1.The GEVP computation for subspace construction appears heavily dependent on the quantity, quality and distribution of samples in the calibration set. In addition, I am concerned whether the method may fail under out-of-distribution (OOD) test samples. I encourage the authors to provide more theoretical or experimental evidence to demonstrate how variations of samples in calibration set affect the final results. 2.The τ thresholds used for generalized eigenvalue selection rely heavily on empirical tuning. Although the authors conduct experiments to verify the effectiveness of this set of parameters, I argue that such thresholds are strongly dependent on the distribution of training data.

Typos 1.Both the number of layers and the loss use the letter of capital L. The notation of loss is correct in Figure 2 but wrong in the main content. 2.The font size in Figure 3 is too small.

Questions for authors

1.Only three types of attacks are considered. While they are indeed representative attack types, this raises the concern of excessive subjectivity during method design. Would FSW require retraining when new types of attacks emerge? 2.I’m wondering if the experiments include performance of the original watermark-free model for comparison? I believe this comparison would make the evaluation of generation quality more convincing.

5 comments
AuthorsOpenReview ↗

Response to Reviewer BayH (1/3)

1. Response to Weakness 1: How do the quantity, quality, and distribution of calibration samples affect subspace construction? Can OOD inputs cause the method to fail?

We thank the reviewer for raising the questions of calibration-data selection and out-of-distribution generalization. Calibration samples affect subspace construction through matrix estimation. In FSW’s intended setting, however, the model owner selects the calibration data and performs verification using designated private challenge inputs. We examine construction results across different sample sizes and data sources to clarify the practical effect of this dependence.

Role and selection of calibration samples. According to Eqs. (2) and (3) in §4.2, the calibration set is used to estimate the functional sensitivity matrix FF and perturbation covariance matrix CAC_A, which jointly determine the generalized eigenvectors in Eq. (5). Sample size and representativeness therefore affect matrix estimation and direction selection. In practice, the owner can select representative calibration data relevant to the model’s intended use. Appendix D.2 specifies 500 training samples for estimating each matrix and separates the data used for subspace estimation, watermark embedding, and evaluation. Appendix A.3 also describes solving on a numerically stable subspace and applying regularization when necessary to mitigate numerical instability caused by near-singular directions.

Construction results under different calibration settings. We examined four calibration settings on Qwen2.5-7B: 125, 250, and 500 samples from WikiText-2, and 500 samples from C4. Using the default spectral thresholds in Appendix D.3 and fixing the subspace dimension at k=32k=32, we obtained the following results:

Calibration dataSample sizeCandidate directions within the default spectral bandSelected directions
WikiText-212511032
WikiText-225021632
WikiText-250039532
C450040932

All four settings provide enough admissible directions to construct the target-dimensional subspace using the default spectral-band rule in Eq. (6), without triggering the fallback for insufficient candidates. These results show that construction succeeds across the examined sample sizes and data sources, rather than being restricted to one fixed sample size or a single corpus. This comparison assesses construction feasibility under calibration-data variations; it does not require different calibration sets to produce identical directions.

OOD inputs and the verification protocol. Section 4.1 distinguishes the calibration set DD used for subspace estimation from the challenge set C\mathcal C used for watermark embedding and verification. Under the protocol in §4.3, the owner verifies a candidate model using designated private challenge inputs and stored verification information. Verification does not require re-estimating the subspace from ordinary user inputs. Consequently, a distribution shift in the model’s everyday inputs does not require the owner to change the verification inputs. The protocol therefore does not depend on watermark recovery from arbitrary OOD inputs.

In summary, calibration-data representativeness is a construction consideration that the owner can actively control. The reported comparisons show that the target subspace can be constructed under the default rule across different sample sizes and corpus sources. Verification using private challenge inputs also clarifies the scope of the OOD concern: watermark recovery from arbitrary new inputs is not a prerequisite for this protocol.

AuthorsOpenReview ↗

Response to Reviewer BayH (2/3)

2. Response to Weakness 2: Rationale for Spectral Threshold Selection and Data Dependence

We thank the reviewer for raising the question of spectral threshold selection and its dependence on the data. FSW applies the thresholds to normalized eigenvalues and selects them under validation-set perplexity constraints. We further examined direction selection across different sample sizes and corpus sources: the same default thresholds can be used in all four construction configurations without configuration-specific retuning.

Threshold definition and selection rationale. According to Eq. (6) in §4.2, we retain directions satisfying

\leq \frac{\lambda_i}{\lambda_1} \leq \tau_{\mathrm{upper}},$$ and select the top $k$ admissible directions in descending eigenvalue order. The lower bound filters directions with low functional sensitivity relative to the modeled perturbations. The upper bound limits the direct selection of leading spectral directions to account for the effect of watermark embedding on model utility. **Performance evidence in the original manuscript.** Section 5.3, Figure 3, and Table 11 in Appendix H compare different spectral selection strategies. Under backbone-preserving distillation on LLaMA-2-7B, FSW achieves **95.90%** detection-score retention, compared with **80.77%** for direct Top-k selection. FSW also achieves a lower post-distillation PPL of **5.9139**, compared with **6.6026** for Top-k. These results support spectral-band selection as a strategy that balances detection-signal preservation and model utility. **Additional direction-selection sensitivity analysis.** To directly examine the data dependence of the thresholds, we evaluated four construction configurations on Qwen2.5-7B: 125, 250, and 500 samples from WikiText-2, and 500 samples from C4, with $k=32$ throughout. The same default thresholds yield **110, 216, 395, and 409 candidate directions**, respectively. All configurations therefore provide enough candidates to select 32 directions under the default rule, without retuning the thresholds or triggering the fallback. Within each configuration, we held the computed eigenvalues and eigenvectors fixed and varied only the thresholds: | Threshold variation | Direction-selection results across the four construction configurations | |---|---| | Fix the upper bound at 0.6 and set the lower bound to $10^{-5}$, $10^{-4}$, $10^{-3}$, or $10^{-2}$ | Each configuration selects exactly the same 32 directions as under its own default setting | | Fix the lower bound at $10^{-4}$ and change the upper bound to 0.3 | **25–31 directions** overlap with the respective default selection | | Fix the lower bound at $10^{-4}$ and change the upper bound to 0.9 | **30–32 directions** overlap with the respective default selection | Thus, across the tested construction data and threshold values, varying the lower bound leaves the final selection unchanged, while varying the upper bound changes some selected directions but retains most default directions. This scan directly characterizes how thresholds affect direction selection and complements the performance comparison of spectral selection strategies in Table 11. In summary, the thresholds are selected empirically under validation constraints, but this does not imply that they must be retuned for every construction dataset. The original performance comparison supports spectral-band selection, while the additional scan shows that the same default thresholds can be used across the tested sample sizes and corpus sources and quantifies how nearby threshold changes affect the selected directions. In the revised manuscript, we will clarify the normalized threshold definition, validation-based selection rationale, and use of fixed defaults in §4.2 and Appendix D.3, and add the direction-selection sensitivity results to Appendix H. **3. Response on Notation and Figure Readability** We thank the reviewer for pointing out the inconsistent notation and figure readability issues. We will make the following changes in the revised manuscript: 1. **Standardize the loss notation.** We will retain $L$ for the number of Transformer layers and consistently use $\mathcal{L}$ for the loss, with individual terms denoted by $\mathcal{L}_{\mathrm{LM}}$, $\mathcal{L}_{\mathrm{wm}}$, and $\mathcal{L}_{\mathrm{con}}$. This will align the main text with Figure 2 and avoid ambiguity. 2. **Increase the font sizes in Figure 3.** We will enlarge the axis labels, tick labels, legend, and numerical annotations to ensure readability at the figure’s normal size in the paper.
AuthorsOpenReview ↗

Response to Reviewer BayH (3/3)

Response to Question 1: Representativeness of the Perturbation Operators and Applicability to New Attacks

We thank the reviewer for asking about the representativeness of the perturbation design. We select these three operators to capture representative mechanisms of representation change under common model modifications: linear projection simulates capacity reduction, quantization noise simulates precision loss, and structural dropout simulates feature sparsification, rather than designing separate defenses against three specific attacks. FSW uses these perturbations to estimate the matrix in Eq. (3), then combines it with functional sensitivity to select stable watermark directions. Appendices B.2–B.3 further connect projected representation changes to verification-score drift. The design therefore focuses on representation-level stability.

Table 2 and Table 12 in Appendix H of the original manuscript evaluate this design under LoRA fine-tuning, noise perturbation, pruning, quantization, and backbone-preserving distillation. Distillation involves model training and knowledge transfer, rather than directly applying the three construction operators. For example, after backbone-preserving distillation on LLaMA-2, the detection score changes from 6.0938 to 5.8438, retaining 95.90% of the original score. This shows that the stability of the selected directions can extend to actual modifications beyond the specific implementations of the construction operators.

Regarding retraining, these modifications are applied after watermark embedding, and the modified models are assessed using the original verifier, without attack-specific watermark re-embedding. Thus, new attack types that modify the model through these representation-change mechanisms do not automatically require retraining. The existing verifier can be used for detection, while robustness depends on the modification’s actual effect on the watermark-bearing directions.

In the revised manuscript, we will clarify the representation-change mechanisms associated with the three operators in Appendix A.2 and distinguish construction operators from actual attack evaluations in §5.1. We will also explain how the distillation results support applicability across modifications and clarify that the evaluations do not involve attack-specific watermark re-embedding.

Response to Question 2: Comparison with the Original Unwatermarked Model

We thank the reviewer for suggesting a performance comparison with the original unwatermarked model. Table 1 in §5.2 already compares the unwatermarked fine-tuning baseline, Clean FT, with FSW. To address the reviewer’s question about the original model, we additionally provide a three-way comparison for Mistral-7B-v0.3 from the main experimental setting. Here, Original denotes the model before any additional fine-tuning or watermark embedding and is distinct from Clean FT.

Model statusPPL ↓HellaSwag ↑ARC-Easy ↑
Original unwatermarked model17.2368.72%76.23%
Unwatermarked fine-tuning: Clean FT16.6868.60%77.00%
Watermark fine-tuning: FSW17.2569.00%77.00%

Compared with the original model, FSW increases PPL by only 0.02, corresponding to a relative change of approximately 0.12%. HellaSwag and ARC-Easy accuracies increase by 0.28 and 0.77 percentage points, respectively. Thus, for this model and experimental setting, FSW maintains language modeling performance close to that of the original model without reducing downstream task accuracy, supporting a limited impact of watermark embedding on model utility. The Original comparison captures quality changes across the complete watermark embedding process, while Clean FT provides an unwatermarked fine-tuning reference.

In the revised manuscript, we will clarify the distinction between Original and Clean FT in §5.1 and add this three-way comparison to Appendix H, with consistent model names and evaluation-metric descriptions.

Reviewer_PfZAOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a novel model watermarking method for LLMs under the white-box setting. The approach is based on two "sensitivy matrices". The first one is meant to approximate the Fisher Information matrix of a given token representation with respect to the model parameters. The second is based on what I would call a "finite difference" approach, it models the expected sensitivity of the token representation against a set of model transforms . In this paper, the authors focus solely on "compression" transformation, approximated with three functions: linear projection, quantization noise (strangely modeled as Gaussian instead of uniform) and dropout. Leveraging these two matrices, the authors solve a generalized eigenvalue problem to find a subspace in the model parameters that is both invariant to the chosen model transformations and the least impactful on the model behavior. The authors then propose to embed the watermark by fine-tuning only this subspace. To do so, embedding is enforced through a specififc watermark loss based on communicating a MM-bit binary message through MM secret carriers over a fixed set of images (termed the calibration set). During the detection phase, the message is extracted by aggregating the the watermark score, given by the correlation between the input representation and the secret carrier. The approach is validated under five small open weight models, under different types of model modifications (not necessarily "compression" type modifications ). A comparison wit the SEAL algoritm is also performed as well as an ablation study on the size of the secret subspace and the selection strategy for the directions serving as the basis of the subspace.

Critical strengths

The overall idea and methodology is sound in my opinion and good way to address the problem of model watermarking in the white-box setting proposed by authors. Of course the use of the Fisher Information matrix is already quite standard in the model quantization literature, as such the novelty is still somewhat limited.

Critical weaknesses

I apologize in advance to the authors for what might seem like pedantism and/or nitpicking. I have tried to be as precise as possible in my critique and provide them with as many information as possible for actually improving the paper.

Non-rigorous formalism, missing notation, mistakes: Section 4.2 is marred with inaccuracies and mistakes which makes me question the overall rigor the paper. The authors don't distinguish between the notion of a set and of distribution: $\mathcal{C}, \mathcal{D} and \mathcal{A} are defined as set but used as distributions in Eq 2,3 and 10. Are we to understand that the expectation is computed assuming a uniform distribution on these sets? This should be stated clearly. Some notation is not defined in a timely manner: LLML_{LM} is used on p4, not defined in the notation in Appendix A and finally said to be "the standard next-token language modeling objective on the training data" in the next section (which is not by itself a definition). This is problematic given that LL is defined before that as a constant, making Eq 2. look completely meaningless to a first-time reader. Less important, FF is said to be a "Fisher-style matrix", yet equation 2 is not necessarily an expectation on the score except if LLML_{LM} is defined to be the log likelihood of the input under θ0\theta_0 (which could be said, but is currently not). More seriously, the authors approximate the detection score SS as a Gaussian in Eq. 13 without any justification (and no, Appendix C.1 is not a rigorous justification, just handwaving). At the the very least one would require the CLT to be invoked and the score under H0\mathcal{H}_0 to indeed have mean 00 which is definitely not obvious apriori since zz is centered using the mean on D\mathcal{D} not C\mathcal{C}. Also note that if I fix yjy_j, my expectation of SS is never guaranteed to be zero. Finally, on line 229, the authors say they "sample MM approximately orthogonal secret key vectors". Though this is quite standard in classical watermarking (the subspace approach is the one followed by Broken-Arrows [1]), the way the author state it is meaningless. There are two possible methods. One can either sample high-dimensional vectors uniformly in the subspace, which are then approximately orthogonal specifically because of the high dimension of the space. Another more rigorous way, is to actually sample an orthonormal matrix (or more generally a Haar measure), for which there are non-trivial efficient algorithms -- see the pedagogical technical report in [2] which discusses the traps of a naive implementation.

The evaluation protocol is not sound: The authors rely on AUC and bit-accuracy to evaluate the performance of their watermark. I will critique each of these choices in turn:

  • The fact that the algorithm is evaluated both in a zero-bit and multi-bit settings is already bizarre in itself; these are two different problems (detection vs communication over an unreliable channel) and I highly suspect that at least on of them is dealt with quite non-optimally. The trouble is, the detection score and AUC are meaningless in this context. The threshold of the detection score is computed empirically for a nominal PFA of 10−610^{-6} which is good practice...but it is estimated on only 42 trials! I shouldn't have to explain that such a low number of observations is insufficient to guarantee a low PFA in this context. This is even more true given the fact that the authors never provide a plot of the empirical PFA against the theoretical one -- i.e. they never construct nor validate a sound p-value from their score. I refer the authors to [3] or [4] [Appendix C] for an example of this kind of methodology done correctly for watermarking. At the very least, a log-log plot to see if the tail of the score matches a Gaussian tail should be provided. The use of AUC is another longstanding problem in the way the machine learning community evaluates watermarking -- at one point this might require somebody to produce a technical note on why this is such a bad practice. In the meantime, the authors should note that a high AUC does not guarantee a good watermarking scheme. For instance, for an AUC = 0.98 and a FPR = 10−310^{-3}, we have 10−3≤TPR≤0.97910^{-3} \leq TPR \leq 0.979. For a FPR = 10−210^{-2} and a TPR = 0.5, we have 0.62≤AUC≤0.990.62 \leq AUC \leq 0.99.
  • Finally, bit-accuracy. By itself, a high bit-accuracy is meaningless: even if it is close to 100100%, one should understand that a multi-bit watermark succeeds if and only if the message is perfectly decoded. Now the authors report low bit-accuracy under perturbations (far lower than 9999%) which means that their 8 or 64-bit payload will almost never be communicated correctly. Interestingly, the authors claim that error-correcting codes could be integrated to improve on this problem, which is indeed the correct point of view. The authors refer to Appendix C.2 for more details...which are absent from said appendix. In my humble opinion, its is fine to not use ECC during experiments. One can invoke the Shannon capacity of a binary symmetric channel as a meaningful limit on the maximum achievable rate for the authors problem. AS a bonus, bit-accuracy would be meaningful in this setting: it is (one minus) the flip probability of the communication channel. I enjoin the authors to perform this analysis (and I am ready to explain in it more details if I am unclear) as it would make their evaluation protocol far more rigorous.

Undefined and unclear terms in the evaluation protocol: The authors report "score retention" in Table 2 which is never defined (is it a ratio, a relative error, an absolute error). What is even more bizarre is that "score retention", which I suppose is better when higher, gets over 100100% for quantization -- this looks to be either a mistake in the protocol (contamination with the training set ?) or a bizarre artifact. Yet the authors don't question it, which I believe to be a problem. Same problem with "mean-level verification" which is never defined.

[1] Furon, Teddy, and Patrick Bas. “Broken Arrows.” EURASIP Journal on Information Security 2008 (October 2008): ID 597040. https://doi.org/10.1155/2008/597040.

[2] Mezzadri, Francesco. "How to generate random matrices from the classical compact groups." arXiv preprint math-ph/0609050 (2006).

[3] Fernandez, Pierre, Antoine Chaffin, Karim Tit, Vivien Chappelier, and Teddy Furon. “Three Bricks to Consolidate Watermarks for Large Language Models.” 2023 IEEE International Workshop on Information Forensics and Security (WIFS), December 2023, 1–6. https://doi.org/10.1109/WIFS58808.2023.10374576.

[4] Gesny, Enoal, Eva Giboulot, Teddy Furon, and Vivien Chappelier. "Guidance watermarking for diffusion models." In International Conference on Learning Representations, vol. 2026, pp. 113957-113988. 2026.

Questions for authors

I am quite frustrated by this paper. I find the main idea to be methodologically enticing, and I believe the approach to be one of the correct way to solve the watermarking problem in the white-box setting chosen by the authors. On the other hand, the idea is executed and presented without any of the rigor expected of an A* conference. Consequently, in its current state, I believe this paper should be rejected due to: 1) a non-rigorous evaluation protocol, making the assessment of the method impossible and 2) a presentation of a method and its formalism full of mistakes, innacuracies and handwaving.

As a show of good faith, I have provided a thorough breakdown of my argument in the Weakness section. I am ready to substantially improve my score provided the authors do the necessary work to address my concerns:

  • Please, go over the list of problems in the first part of the weakness section and address them.
  • Notably, the Gaussian approximation should be theoretically justified (with actual mathematical arguments) and empirically evaluated (again I provided examples for doing so in the Weakness section) -- this should make Appendix C.1 actually useful.
  • For the zero-bit evaluation, drop the AUC and report TPR@FPR=10−610^{-6}
  • Instead of bit-accuracy, cast the multi-bit problem as a transmission problem over a noisy binary symetric channel and report the Shannon capacity. Since the payload is actually small, you can also use more recent recent results such as Polyanskiy's bounds [5] at a given probability of decoding error. The analysis can be performed in Appendix C.2.
  • Define non-standard terms such as score retention and mean-level verification
  • Question the fact that you obtain a "score rentention" of 102102% for quantization in Table 2, which means that a perturbation improves your score...there is either an problem in the evaluation or the cause needs to be explained clearly
  • There is a strong degradation in both PPL and utility for Deepseek, but the authors brush it off. Please, again, explain the cause of this degradation clearly.

[5] Polyanskiy, Yury, H. Vincent Poor, and Sergio Verdu. “Channel Coding Rate in the Finite Blocklength Regime.” IEEE Transactions on Information Theory 56, no. 5 (2010): 2307–59. https://doi.org/10.1109/TIT.2010.2043769.

8 comments
AuthorsOpenReview ↗

Response to Reviewer PfZA (1/6)

We sincerely thank the reviewer for the careful reading and detailed, specific comments, particularly the references and suggestions for improvement. These comments are valuable for strengthening the rigor and clarity of the paper. We take these concerns seriously and respond point by point below, explaining the corresponding revisions.

1. Clarifications on Mathematical Formulation, Notation, and Method Details

  • Distinguishing sets from sampling distributions. Equations (2), (3), and (10) use uniform empirical averages over D\mathcal D and C\mathcal C. For A\mathcal A, the three perturbation types are sampled with equal probability, while randomness within each type follows its respective generating distribution. We will explicitly state these conventions in §4.1–4.2 and denote the sampling distributions by PDP_{\mathcal D}, PCP_{\mathcal C}, and PAP_{\mathcal A}, respectively, to distinguish sets from distributions.

  • Loss definition and notation. LLM\mathcal L_{\mathrm{LM}} denotes the average negative log-likelihood for next-token prediction:

    =-\frac{1}{T-1}\sum_{t=2}^{T} \log p_\theta(x_t\mid x_{<t}).$$ Its role is described in §4.3. We will place this mathematical definition before Eq. (2) and include it in Table 6 of Appendix A. We will consistently use $\mathcal L_{\mathrm{LM}}$ for the loss and $L$ for the number of model layers to eliminate ambiguity.
  • Meaning of the “Fisher-style” matrix. In Eq. (2), FF is the second-moment matrix of loss gradients with respect to the selected-layer representation, evaluated on the calibration data. “Fisher-style” refers to its gradient outer-product structure and does not imply equivalence to the population Fisher information matrix in its strict statistical definition. Its role is to measure local first-order loss sensitivity along direction uu through u⊤Fu=E[(u⊤g)2]u^\top Fu=\mathbb E[(u^\top g)^2]. We will clarify this interpretation and its distinction from the Fisher information matrix in §4.2.

  • Gaussian approximation and the zero-mean assumption. The aggregation structure in Eq. (12) supports a Gaussian approximation under explicit conditions, but centering by the reference mean does not automatically ensure a zero-mean detection score. Equation (13) should therefore use S≈N(μ0,σS2)S\approx\mathcal N(\mu_0,\sigma_S^2), with the background mean and score variability estimated from independent clean data. Let AgA_g denote the average score obtained by aggregating inputs and bit projections within source document gg. If the document-level scores are independent and identically distributed with variance 0<τ2<∞0<\tau^2<\infty, the central limit theorem gives

    \frac{\sqrt G(S-\mu_0)}{\tau} \xrightarrow{d}\mathcal N(0,1).$$ This treatment accommodates within-document dependence and does not require independence among bit projections. For Qwen, 2,000 clean inputs from 80 calibration source documents yield $\widehat\mu_0=0.8355$. Under a normal model for document-level scores, accounting additionally for parameter-estimation uncertainty gives a predictive $t$ threshold of **4.6184** at nominal $\alpha=10^{-6}$.A separate batch of 2,000 inputs forms four independent evaluation groups. Each group contains 20 source documents with 25 inputs per document, totaling 500 inputs. The clean model exceeds the threshold in **0/4 groups**, while the watermarked model exceeds it in **4/4 groups**, showing that a detectable watermark signal remains after recalibration. The corresponding revisions allow a nonzero background mean, specify the conditions for the Gaussian approximation, and add resampling distribution plots, QQ plots, and tail-probability comparisons over the observable range. These diagnostics and independent evaluations assess the approximation’s behavior; they do not establish that the actual false positive rate reaches $10^{-6}$. This explicitly distinguishes the model-based nominal level from the empirical false positive rate.
  • Generation of key vectors. We use explicit QR orthogonalization rather than relying on the approximate orthogonality of high-dimensional random vectors. Specifically, we sample a matrix G∈Rk×MG\in\mathbb R^{k\times M} with independent N(0,1)\mathcal N(0,1) entries, compute its thin QR decomposition G=QRG=QR when M≤kM\leq k, and take the columns of QQ as the keys bjb_j. This construction satisfies Q⊤Q≈IMQ^\top Q\approx I_M up to numerical precision, with deviations arising from floating-point computation and precision conversion. We will therefore revise line 229 to state explicitly that “MM orthonormal secret key vectors are generated by QR orthogonalization of a Gaussian random matrix,” and specify M≤kM\leq k. Our use of this construction relies on orthogonality, not on an assumption of Haar-uniform sampling.

AuthorsOpenReview ↗

Response to Reviewer PfZA (2/6)

2.Evaluation Protocol: Statistical Detection and Full-Message Recovery

  • Distinguishing ownership detection from multi-bit recovery. We thank the reviewer for emphasizing the distinction between ownership detection and message recovery. We evaluate both tasks to assess whether the watermark remains detectable and whether the embedded payload can be fully recovered. They use the same underlying projection statistics but have different decision rules and success criteria.

    Equation (12) in §4.3 defines the aggregated detection score, with ownership detection determined by whether this score exceeds the threshold. Equation (15) defines bitwise decoding, whereas successful full-message recovery requires all target bits to be correct. Algorithm 1 presents these as separate steps, “Statistical Ownership Verification” and “Multi-bit Decoding.” Section 5.2 also explicitly describes ownership detection and full-payload recovery as complementary aspects of watermark reliability. Thus, the original manuscript distinguishes these tasks: successful detection does not imply successful full-message recovery. To align the experimental presentation with this distinction, the revised manuscript will separately report “detections / detection groups” and “exact message recoveries / detection groups,” while retaining bit accuracy to describe bitwise recovery quality.

  • Purpose and evidential scope of the 42 negative controls. We thank the reviewer for raising concerns about statistical reliability at low false-positive levels. We clarify that the 42 trials in Table 8 evaluate whether an already determined detection threshold produces false positives; they are not used to estimate that threshold. Section 4.3 explicitly states that calibration trials are independent of the trials used to evaluate false positives, and Step 19 of Algorithm 1 separately specifies null-distribution calibration and threshold determination. Table 8 covers clean models, modified unwatermarked models, and incorrect keys, with 0/42 observed false positives, providing preliminary support for specificity under these tested conditions. We agree that this result does not establish an actual false-positive rate of 10−610^{-6}; Equation (14) provides a nominal tail probability under the Gaussian model. The revised Table 8 and Appendix E will make this distinction explicit and report nominal levels separately from observed false-positive results.

  • Reliability of detection scores, p-values, and tail probabilities. We thank the reviewer for suggesting theoretical–empirical comparisons to examine tail-probability calibration. Using the calibration rule described in the first part of our response, we fixed the model, key, and calibration parameters and evaluated additional independent clean inputs. We also generated score-distribution plots, QQ plots, and a log–log comparison of nominal and empirical false-positive rates.

    The additional evaluation comprises 60 documents and 1,500 inputs, organized into three detection groups, each containing 20 documents and 500 inputs. There is no document overlap between these groups or with the original calibration and evaluation data. The results are:

    Independent clean detection groupDetection scoreOne-sided p-value under the predictive t model
    1−0.84890.9875
    20.48500.6822
    3−0.32200.9399

    We observed 0/3 false positives at each of 11 prespecified nominal levels ranging from 0.5 to 10−610^{-6}. The log–log plot reports the one-sided 95% confidence upper bound associated with this zero-false-positive result. With only three independent detection groups, these results provide an independent check of the fixed calibration rule but cannot establish agreement in the extreme tail or demonstrate an actual false-positive rate of 10−610^{-6}. The theoretical basis and assumptions underlying the p-values are provided in the first part of our response; here, we report the outcomes observed under that rule.

    Manuscript revisions. Appendix C.1 will include these distributional and tail-probability diagnostic plots, explicitly identifying triangles as confidence upper bounds associated with zero observed false positives, rather than observed false-positive rates. Appendix E will report the individual group scores, p-values, and false-positive counts. These revisions will connect the calibration rule, statistical assumptions, and empirical checks while clearly distinguishing model-based nominal significance levels from observed false-positive results.

AuthorsOpenReview ↗

Response to Reviewer PfZA (3/6)

  • Detection performance at the specified operating point and the role of AUC. We thank the reviewer for emphasizing detection performance at very low false-positive levels. We agree that a high AUC alone does not establish detection effectiveness at the target operating point and should not replace evaluation at a specified threshold. Because AUC describes overall discrimination across thresholds, we will retain it as a supplementary metric and use false-positive and detection results at the specified operating point as the primary evidence for ownership detection.

    Section 5.1 already includes threshold exceedance as an evaluation criterion, and Tables 3 and 4 report detection decisions. Our additional evaluation reports group-level outcomes under the frozen calibration rule described in the first part of our response. At the threshold corresponding to the nominal significance level α=10−6\alpha=10^{-6}, the clean model produced 0/3 false positives, and the watermarked model was detected in 3/3 groups. Each group contains 20 documents and 500 inputs. These results directly report performance under the specified detection rule without inferring it from AUC.

    Regarding TPR@FPR=10−610^{-6}, we distinguish the model-based nominal significance level from the actual false-positive rate. The available independent evaluation is insufficient to establish an actual FPR of 10−610^{-6}. We will therefore label these outcomes as “detection results at the threshold corresponding to the nominal significance level,” rather than empirically validated TPR@FPR=10−610^{-6}.

    Manuscript revisions. Section 5.1 will clarify the roles of these metrics, and the main detection results will prioritize false-positive and detection outcomes at the specified operating point. Appendix E will report the number of independent detection groups, false-positive counts, detection counts, and confidence intervals. AUC will be retained only as supplementary information and will no longer support claims about detection performance at an extremely low false-positive operating point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.