- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a watermark for model ownership carried by which valid solution a model returns to a constraint problem, rather than by how the answer is worded.
Critical strengths
(1) The carrier resists correctness-checking, correctness-driven fine-tuning, and answer-only extraction.
Because the designated reply must be a correct answer chosen among verified alternatives, correctness-checking does not expose the mark and correctness-driven fine-tuning does not overwrite it, with 98.96% to 100% of owner matches retained. Because the verifier reads the parsed solution rather than the wording, answer-only extraction leaves the mark intact, and under that attack this method keeps 99.8% balanced accuracy while all six text watermarks fall to chance at 50.0%.
(2) The paper defines mark removal precisely, as finding another valid solution.
Resistance to removal is usually argued informally, by showing that a rewriter or a paraphrase model fails to strip the mark, which does not say what an attacker would have to do instead. Requiring another valid solution does say it, and turns an open-ended claim into one that a single experiment can settle. That is a contribution to how subsequent work can argue about removal.
Critical weaknesses
(1) The security claim is not established, because no exact solver is ever run on the protected problems.
A. The five commercial reasoning models of Table 1 stand in for the solver that removal route (i) requires, since Section 3.1 defines that route as re-solving "with a key-independent solver", yet no exact solver for X3C, Kakuro or anchored graph coloring is run anywhere in the paper. Those five models reach at most 35/96 validity, so the experiment measures whether an LLM can solve the problems from scratch, not whether an exact solver can find another solution.
B.The screening was performed by two models that the paper does not identify. Appendix A kept later candidates "only if neither of two external reasoning models found a different valid solution", but neither their identity, their settings nor their budget is reported, and the fixed screening budget of Section 3.2 is also unreported. The hardness of the catalog is defined relative to those models, so the filter cannot be reproduced or checked.
(2) The paper's own results contradict its stated conclusions.
A. Table 4 reports owner match, validity and MMLU under model modification but not the acceptance probability. "SVA, related checkpoints" leaves 56/96 owner matches, which by Equation 7 gives acceptance probabilities of 3.61%, 4.50% and 3.09%, so the owner would accept this model with only 3% to 4.5% probability, while it stays valid on 93.75% of the protected problems with MMLU unchanged at 77.27. That meets the attacker-success definition of Section 3.1, yet Section 4.6 counts every row as a failed removal.
B. The abstract's invariance claim is stated without its scope. It reads "rewriting that preserves the solution preserves the mark", yet Table 5 gives an owner match of 100.00% under the canonical trained prompt, 95.83% under four new prefixes, 40.63% under full Chinese, 4.17% under structured JSON, and 0 of 192 valid outputs under two isomorphic views per problem. In the last three conditions validity falls by exactly the same amount as the owner match, so the model stops producing valid answers rather than returning a different solution, and the claim should be scoped to rewriting of the answer rather than of the problem statement or the output format.
Questions for authors
-
Related to Weakness 1. Was an exact solver for X3C, Kakuro or anchored graph coloring ever run on the protected problems with the stored solution blocked, and if so, what time and node budget does it need?
-
Related to Weakness 1. Which two models screened the protected candidates, with what settings and budget, and what is the fixed screening budget of Section 3.2?
-
Related to Weakness 2. "SVA, related checkpoints" fails verification at 3.61% to 4.50% acceptance while staying 93.75% valid, so does the paper consider this a successful removal under its own Section 3.1 definition, and does the Section 4.6 conclusion still hold?
-
Related to Weakness 2. To which class of rewriting does the invariance claim apply, given that owner match falls to 40.63% under full Chinese and 4.17% under structured JSON while validity falls by exactly the same amount?