- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LLMs are valuable IP for their developers. To protect the IP, this paper proposes FSW, which aims to embed watermark into LLMs and detect it with white-box access to the suspected model. The basic idea of FSW is to fine-tune the model by selecting internal representation directions, which are important to maintain functionality of LLMs and robust to weight modification. As a result, such directions could be verified with private vectors as watermark signal. Further experiments demonstrated the effectiveness of FSW. Despite this paper is overall well-written and the method design is concrete and sound, I have some concerns on its design choices and lack of analysis experiments (see questions). Therefore, I will give it score of 2. If all the questions are addressed during rebuttal, I would be open to improve my score.
Critical strengths
- the motivation of LLM IP protection is concrete
- the proposed method is sound and easy to understand.
- the writing is good and the paper is well structured.
- the experiments are fairly strong in breath
Critical weaknesses
- the design choices are not well explained(see questions)
- some analysis experiments about important hypers are missing(see questions)
Questions for authors
- The generalized eigenspace is optimal for the proposed trace surrogate, but the paper does not establish why this surrogate is the appropriate objective for watermark robustness. Besides, the author did not explain why the resulting FSW design is preferable to alternative subspace constructions. Could the authors provide stronger theoretical justification for these design choices?
- how to pick and ? Since they decide what eigenvalues would be chosen, I believe they are vital for FSW. However, I did not see any analysis experiments over the choice on them.
- It seems that evaluation about adaptive attack is absent (e.g. the attackers have known FSW), which I think is worthy to explore for a model watermarking algorithm. Could the authors provide the settings of adaptive attack and how to defend it?
9 comments
Response to Reviewer aYeN (2/3)
2. Response to Question 2: How are the spectral thresholds selected, and how do threshold changes affect direction selection?
We thank the reviewer for asking about spectral threshold selection and sensitivity. We determined the thresholds through experimental comparisons, balancing watermark performance and model utility. As stated in §5.1 of the original manuscript, the subspace dimension and spectral thresholds are selected under validation-set perplexity constraints. Appendix D.3 reports the final settings:
Selection rationale and procedure. Following Eq. (6) in §4.2, we first retain directions satisfying
\leq \frac{\lambda_i}{\lambda_1} \leq \tau_{\rm upper},$$ and then select the top $k$ eligible directions in descending eigenvalue order. The lower threshold filters out directions with low functional sensitivity relative to the modeled perturbations. The upper threshold limits direct selection from the leading end of the spectrum to account for the effect of watermark embedding on model utility. Once determined, these thresholds are kept fixed. §5.3, Figure 3, and Table 11 in Appendix H provide experimental support for the final spectral-band selection. Under backbone-preserving distillation on LLaMA-2-7B, this selection achieves **95.90%** detection-score retention, compared with **80.77%** for direct Top-k selection. Its PPL after distillation is also lower, at **5.9139**, compared with **6.6026** for Top-k. These results support spectral-band selection as a way to balance detection-signal preservation and model utility. **Additional threshold sensitivity analysis.** To directly examine how the thresholds affect the selected directions, we performed fixed-spectrum scans on Qwen2.5-7B using four construction datasets: 125, 250, and 500 WikiText-2 samples, and 500 C4 samples. We fixed $k=32$ throughout. For each dataset, we kept the computed eigenvalues and eigenvectors unchanged and varied only the thresholds. | Threshold setting | Results across the four construction datasets | | --- | --- | | Fix the upper threshold at 0.6 and set the lower threshold to $10^{-5}$, $10^{-4}$, $10^{-3}$, or $10^{-2}$ | Each dataset yields exactly the same 32 directions as its corresponding default setting | | Fix the lower threshold at $10^{-4}$ and change the upper threshold to 0.3 | **25–31 directions** are shared with the corresponding default selection | | Fix the lower threshold at $10^{-4}$ and change the upper threshold to 0.9 | **30–32 directions** are shared with the corresponding default selection | Under the default thresholds, the four datasets provide **110, 216, 395, and 409** eligible directions, respectively. All provide enough candidates to select 32 directions without invoking the fallback procedure. At the four tested lower thresholds from $10^{-5}$ to $10^{-2}$, the final selection remains unchanged for each dataset because the top 32 eligible directions continue to satisfy the respective lower thresholds. Changing the upper threshold affects part of the selection: setting it to 0.3 replaces **1–7 directions**, while setting it to 0.9 replaces **0–2 directions**. Thus, the upper threshold has a more direct effect on the final selection in these settings. We also examined a stricter lower threshold of 0.1. This leaves fewer than 32 eligible directions for all four datasets and triggers the implementation’s fallback procedure, which skips the top 1% of directions in the eigenvalue ranking and selects the next 32. None of the four default configurations triggers this procedure. These results complement the performance comparisons in the original manuscript by directly showing how the thresholds affect the number of eligible candidates and the final selected directions. **Manuscript clarification.** In the revised manuscript, we will consolidate the experimental basis for threshold selection, their fixed use, the sensitivity results above, and the handling of insufficient candidates in §4.2 and Appendix D.3. Together, the thresholds are determined through experimental comparisons that balance watermark performance and model utility, the original experiments support the effectiveness of the final spectral-band selection, and the additional scans clarify how threshold changes affect the selected directions.Response to Reviewer aYeN (3/3)
3. Response to Question 3: How is FSW evaluated and protected against adaptive attacks when the attacker knows the method?
We thank the reviewer for raising the question of adaptive attacks. We have evaluated an attack that directly exploits FSW’s public construction procedure: the attacker reconstructs a candidate subspace using independent public data and then suppresses that subspace to attempt watermark removal. Section 3 and Appendix F of the original manuscript describe this setting, and Table 9 reports the results at the maximum tested suppression strength. Across two models, three seeds, and four attack strengths, all 24 attacked checkpoints pass verification and retain 100% bit accuracy.
Attack setting. The attacker has white-box access to the released model and knows the watermark layer, subspace dimension , spectral thresholds, and public generalized eigenvalue procedure. Using independent public data, the attacker re-estimates and , constructs a candidate subspace , and suppresses it. This attack targets the entire estimated subspace without first recovering the individual watermark keys, directly testing whether the attacker can exploit FSW’s public design to locate and remove the watermark. The evaluation covers LLaMA-2-7B and Qwen2.5-7B, with three seeds per model and suppression strengths
Pre- and post-attack evaluations are paired using the same underlying model checkpoints and held-out split, yielding 24 attacked checkpoints.
Experimental results. Across the four strengths, each model passes verification on 12/12 attacked checkpoints, giving 24/24 overall, all with 100% bit accuracy. Appendix F, Table 9 reports the results at the maximum tested strength, . Scores below are means and standard deviations across three seeds:
| Model | Pre-attack score | Post-attack score | Detection threshold | Post-attack bit accuracy | Post-attack passes |
|---|---|---|---|---|---|
| LLaMA-2-7B | 5.766 ± 1.752 | 5.786 ± 1.712 | 1.954 | 100% | 3/3 |
| Qwen2.5-7B | 4.948 ± 0.334 | 4.958 ± 0.343 | 1.024 | 100% | 3/3 |
Neither model shows a notable decrease in detection score at the maximum tested strength. Table 9 also reports small perplexity changes, while HellaSwag and ARC-Easy performance remains essentially unchanged in the full attack evaluation.
Explanation and verification procedure. Appendix F further examines whether the attacker accurately reconstructs the watermark-bearing subspace. The overlap between the reconstructed and owner subspaces is 0.00397 ± 0.00010 for LLaMA-2-7B and 0.00952 ± 0.00031 for Qwen2.5-7B, close to the respective random-overlap levels of 0.00391 and 0.00893. This explains the observed attack outcome: under the tested settings, reconstruction from independent data does not accurately locate the actual watermark-bearing directions, so suppressing the estimated subspace does not effectively remove the detection signal. As specified in Section 3, FSW’s algorithm and hyperparameters may be public, while the keys, target-bit mapping, and challenge set remain private. The experiment further tests whether the attacker can bypass individual-key recovery by reconstructing and suppressing the entire subspace. After the attack, the owner continues to use the original verifier without re-embedding the watermark or changing the detection rule in response to the attack.
Manuscript clarification. In the revised manuscript, we will highlight this adaptive attack more prominently in the main text, consolidating the attacker’s knowledge, the reconstruction and suppression procedure, and verification results across all four strengths, with explicit references to Appendix F and Table 9. Together, these evaluations directly support FSW’s continued detectability under the tested public-information subspace-reconstruction and suppression attack.
Response to Reviewer aYeN (1/3)
1. Response to Question 1: Why is the surrogate objective appropriate for watermark robustness, and what advantages does FSW’s subspace selection offer?
We thank the reviewer for asking about the design rationale. FSW jointly considers task relevance and stability under model modifications, and uses spectral-band selection to balance watermark robustness and model utility.
Design rationale. In §4.2, u⊤Fu in Eq. (2) measures the local sensitivity of the language modeling loss to direction u, while u⊤CAu in Eq. (3) measures the expected squared perturbation along that direction under the modeled transformations. The former alone does not ensure stability after modification, while the latter alone does not ensure task relevance. Equation (4) therefore uses their ratio to jointly evaluate candidate directions. This surrogate has an explicit connection to watermark detection. Proposition 2 in Appendix B.2 establishes
\|U^\top\Delta r(x,a)\|_2^2 = \operatorname{tr}(U^\top C_AU).$$ Corollary 1 in Appendix B.3 further shows that the change in the detection score is bounded by the drift in projected coordinates. Thus, among directions with comparable task relevance, smaller projected perturbations help control detection-score drift. This provides a rationale for jointly considering task relevance and modification stability that is directly connected to the detection statistic. **Advantages over alternative selection strategies.** Proposition 1 in Appendix B.1 establishes the optimality of generalized eigenvectors for the stated trace surrogate. The final construction must also account for the effect of watermark embedding on model utility. FSW therefore applies the spectral-band selection in Eq. (6) to this ordering. The original manuscript evaluates the contributions of both joint modeling and spectral-band selection: - **Joint modeling.** Table 5 in §5.3 shows that removing compression-invariance modeling reduces the detection score after the attack from **5.84** for full FSW to **0.96**. This supports explicitly accounting for modification-induced perturbations alongside functional sensitivity. - **Spectral-band selection.** §5.3, Figure 3, and Table 11 in Appendix H show that, under backbone-preserving distillation on LLaMA-2-7B, FSW achieves **95.90%** detection-score retention, compared with **80.77%, 74.04%, 65.48%, and 78.11%** for Top-k, Mid-50%, Low-75%, and Last-k, respectively. FSW also achieves the highest detection score after distillation, **5.8438**, and the lowest PPL after distillation, **5.9139**, among these strategies, demonstrating a better balance between detection-signal preservation and model utility. **Manuscript clarification.** In the revised manuscript, we will clarify the connection between task relevance, projected perturbations, and detection-score stability in §4.2, with explicit references to Table 5, Figure 3, and Table 11. We will also distinguish theoretical optimality under the trace surrogate from the empirical evidence supporting the final spectral-band selection. Together, these theoretical results connect FSW’s surrogate objective to detection stability, while the experiments in the original manuscript support the practical advantages of joint modeling and spectral-band selection.