- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CAST is a model-ownership watermarking framework designed for text-only verification. It fine-tunes an LLM so that replacing individual tokens in a secret trigger produces signed changes in the probability that the first generated token belongs to a secret vocabulary subset. These signs encode a multi-bit watermark. Verification estimates the relevant probabilities through repeated text-only queries, while sequential token-wise optimization reduces the number of simultaneously retained computation graphs during embedding. Experiments across several LLMs evaluate bit recovery, utility, distinctiveness, and robustness to model modifications. The approach is practically motivated, but the complete-recovery analysis and the claimed structural memory advantage over EaaW need stronger justification.
Critical strengths
-
A concrete text-only verification mechanism.
The vocabulary-subset event can be estimated directly from sampled outputs, without requiring logits from the suspicious API. This is a useful adaptation of attribution-based watermarking to a realistic access restriction, and the method does not require unusually long target responses or a separately trained detector.
-
Broad empirical coverage and a useful memory–computation trade-off.
The evaluation covers multiple model families, several watermark lengths, and modifications including fine-tuning, quantization, and merging. The supplementary generative-task evaluation is also useful. Sequential backpropagation provides a genuine reduction in simultaneously retained model activations relative to the tested EaaW implementation, which may make watermark embedding more accessible under limited GPU memory.
Critical weaknesses
-
The theoretical recovery guarantee and empirical ownership reliability are insufficiently established.
The complete-recovery proof assumes independence that the extraction algorithm does not provide. Algorithm 2 estimates the original-trigger probability once and reuses it for all bit estimates. Writing these estimates as , independent masked-prompt sampling still gives for distinct and . Appendix A nevertheless derives the complete-watermark recovery expression by assuming independent bit extraction. The theorem is therefore not established for the stated algorithm, although this does not imply that the method fails or that the numerical expression is necessarily incorrect. A dependence-aware analysis, a valid union bound, or independently re-estimated baselines would be needed, with the latter increasing the query budget. More broadly, WSR measures bit-matching accuracy rather than ownership-level reliability. The paper does not clearly establish ownership-level TPR/FPR or complete-message recovery across independently sampled triggers, keys, watermarks, and training runs. While the binomial null calculation is valid when the target watermark is independently uniform and independent of null extraction, the corresponding assumptions and uncertainty estimates for the actual ownership decision procedure require clarification.
-
The practical advantages and distinction from prior attribution-based watermarking require stronger controlled comparisons.
The memory comparison with EaaW overstates an implementation-dependent limitation. A global regression objective does not inherently require retaining all LLM computation graphs simultaneously: for fixed masks, EaaW's attribution can be expressed as , allowing the loss derivatives with respect to to be computed before recomputing individual outputs and accumulating model gradients. Although this introduces computation and replay constraints, Table 13 establishes an advantage over the evaluated implementation rather than an unavoidable asymptotic limitation. A recomputation/checkpointing baseline is needed. Moreover, EaaW already investigates label-only verification in Appendix D, albeit for image classification rather than text-only LLM APIs. CAST should therefore distinguish its vocabulary-subset events, token-wise differences, and repeated text sampling from prior attribution-based verification, ideally through a controlled baseline using the same observation mechanism. Practical verification costs also remain substantial: at , a 128-bit watermark requires 12,900 API requests, and single-token responses do not eliminate prompt-processing costs. End-to-end latency and performance under restricted query budgets are not established. Finally, the prefix-attack evaluation does not distinguish model-generated prefixes from prefixes prepended at the API output layer. The latter could disrupt first-token verification without substantially affecting answer utility, so the reported MMLU degradation does not establish an inherent watermark-removal–utility trade-off.
Questions for authors
-
Theoretical recovery guarantee and empirical calibration.
How should Theorem 1 be modified to account for the shared original-trigger estimate in Algorithm 2? Under the corrected analysis, what recovery guarantee or empirical calibration is supported at the default sampling budget?
-
Memory efficiency and attribution-based baselines.
Does CAST retain a substantial memory–time advantage over EaaW with recomputation or checkpointing? Can the effect of token-wise attribution be separated from that of the vocabulary-subset observation mechanism?
-
Ownership-level verification reliability.
How many independent trigger/key/watermark configurations and training seeds underlie the reported WSRs? What are the ownership-level TPR/FPR and complete-watermark recovery rates, with uncertainty estimates, at a prespecified threshold?
-
Query efficiency and robustness to output-layer manipulation.
Under a fixed query budget, what verification performance is achieved? For prefix manipulation, does the utility degradation remain when the prefix is added only at the output layer and downstream answers are extracted appropriately?
5 comments
Response to Reviewer Wyc5 (2/3)
W2-1&Q2 Comparison with Memory-optimized EaaW
Thank you for suggesting this comparison!
CAST retains higher computation efficiency at comparable peak memory to optimized EaaW. Following the reviewer's suggestion, we implemented two-pass recomputation for EaaW: first compute the regression objective, then recompute model outputs to accumulate gradients without retaining all activation graphs. We added this comparison on 128-bit Qwen3-4B-Instruct-2507 in Appendix D.6 (Table 15) with a brief pointer in Section 3.5:
| Method | Peak GPU memory (GiB) | Time per pass (s) | Gradient error vs. retained EaaW |
|---|---|---|---|
| EaaW, retained graphs | 84.89 | Reference | |
| EaaW, recomputation | 17.33 | ||
| CAST, sequential objectives | 18.05 | - |
Recomputation brings EaaW's memory usage close to CAST's, but requires additional forward passes. CAST processes its local objectives sequentially without this recomputation, using 34% less time at comparable peak memory. CAST therefore retains its computational advantage against memory-optimized EaaW.
W2-2&Q2 Distinction from EaaW and Controlled Comparison
Thank you for asking us to clarify the distinction from EaaW!
1. CAST introduces token-wise attribution and vocabulary-subset observations to address two limitations of EaaW. EaaW also studies label-only verification for image classification, while CAST addresses text-only LLM APIs. We compare these two components below. Token-wise attribution replaces global regression with direct local differences, improving embedding efficiency. Vocabulary-subset observations turn generated tokens into a statistically estimable watermark signal for text-only verification.
2. Token-wise attribution improves the memory–time trade-off. CAST processes local objectives sequentially, whereas memory-optimized EaaW recomputes model outputs for its global regression objective. Appendix D.6 (Table 15) reports:
| Method | Peak GPU memory (GiB) | Time per pass (s) |
|---|---|---|
| EaaW, retained graphs | 84.89 | |
| EaaW, recomputation | 17.33 | |
| CAST, sequential objectives | 18.05 |
CAST uses 34% less time at comparable peak memory to recomputation-enabled EaaW.
3. Vocabulary-subset observations enable accurate text-only watermark extraction. We equip EaaW's global regression with the same vocabulary-subset observations as CAST. The comparison below shows over 99% WSR for both subset-based methods, compared with 50% for the paper's original EaaW baseline:
| Method | Text-only WSR (%) |
|---|---|
| CAST | |
| Original EaaW | 50.00 |
| EaaW with subset observations |
Together, these results support the complementary roles of CAST's two designs: token-wise attribution improves embedding efficiency, while vocabulary-subset observations support accurate text-only verification.
W2-3&Q4 API Query Cost
Thank you for highlighting verification costs!
1. Total token costs of CAST remain manageable, and requests can be spread out to meet API rate limits. CAST requests only one generated token per query. For 128 bits and , this gives approximately 12.9K generated tokens, plus 1.65M input tokens for 128-token prompts. These explicit token counts allow the cost to be budgeted, while requests can be scheduled over a longer period to stay within queries-per-minute (QPM) limits.
2. Fewer queries can still support successful ownership verification. We added a reduced-query evaluation on the same reference checkpoint in Appendix D.7 (Table 16):
| Total queries | Ref. WSR (%) | Ref. ownership acceptance (%) | Local time (s) | |
|---|---|---|---|---|
| 10 | 1.3K | 100.00 | ||
| 25 | 3.2K | 100.00 | ||
| 100 | 12.9K | 100.00 |
A 90% reduction in queries still supports successful ownership verification on the reference model.
3. Multi-token extraction could also reduce queries at the same generated-token budget. The multi-token extension introduced in our response to Reviewer C3c9’s W1&Q1 extracts multiple token scores per query, reducing queries at the same generated-token budget (Appendix D.7, Table 17):
| Generated tokens per query | Queries per prompt | Total queries | Generated tokens | Input tokens |
|---|---|---|---|---|
| 1 (budget reference) | 96 | 12.4K | 12.4K | 1.6M |
| 2 | 48 | 6.2K | 12.4K | 793K |
| 4 | 24 | 3.1K | 12.4K | 396K |
| 6 | 16 | 2.1K | 12.4K | 264K |
At the same generated-token budget, four-token extraction reduces queries by 75% and also lowers input-token costs.
Response to Reviewer Wyc5 (3/3)
W2-4&Q4 Response Templates and Output Post-processing
Thank you for raising this practical concern! We describe the prefix-manipulation experiment in Appendix D.3 and report additional multi-token results in Appendix D.3.1 (Table 11).
1. CAST can recover the watermark under fixed, reversible API templates. Removing the known template restores watermark extraction when the original text and tokenization are preserved. Table 10 instead tests prefixes supplied before generation, including the simple prefix Sure.:
| Prefix setting | WSR (%) | MMLU accuracy (%) |
|---|---|---|
| No attack | 100.00 | 69.74 |
Short context prefix: Sure. | 59.00 | 56.80 |
Long context prefix: OK, there is the answer. | 55.00 | 56.16 |
Here, MMLU choices are scored immediately after the supplied prefix. A fixed prefix added after generation can preserve answer content and be removed before verification. Table 10 does not measure this case.
2. Templates that prevent reliable answer extraction also reduce API usability. If an attacker varies the template so that the verifier cannot reliably separate it from the generated answer, ordinary applications using the same parsing procedure face the same difficulty.
3. CAST can also be extended to multi-token embedding and verification, overcoming reliance on the first token alone. We extended CAST to embed and extract the watermark across 2, 4, or 6 output tokens, then tested whether it remains recoverable after deleting the first token:
| Generated tokens per query | Queries per prompt | Full-text WSR (%) | WSR after first-token deletion (%) |
|---|---|---|---|
| 2 | 48 | 100.00 | 53.91 |
| 4 | 24 | 100.00 | 100.00 |
| 6 | 16 | 97.66 | 97.66 |
The four- and six-token versions still pass ownership verification after deletion (threshold: 82/128). This demonstrates a feasible way to mitigate first-token dependence.
Response to Reviewer Wyc5 (1/3)
Dear Reviewer Wyc5,
Thank you for your careful analysis and constructive suggestions! We appreciate your recognition of CAST's concrete text-only verification mechanism, broad empirical evaluation, and practical memory–computation trade-off. We address your concerns below.
W1-1&Q1 Recovery Analysis and Calibration
Thank you for pointing out the dependence between bit estimates!
1. Bit recovery is independent once the shared original-prompt estimate is fixed. Given this estimate, independently sampled masked-prompt responses yield independent bit decisions. We can therefore multiply their conditional recovery probabilities and then average over the shared estimate, without changing CAST or adding queries.
2. We can also bound recovery without assuming bit independence. Keeping Theorem 1's Gaussian approximation and single-bit analysis, a union bound gives
Pr(w^=w)≥max(0,1−k[1−Φ(ϵ1−ϵ22T)]).
This replaces the product of marginal success probabilities and allows dependent bit-recovery events. We revised Theorem 1 in Section 3.5 and its proof in Appendix A.
3. Removing the independence assumption gives a similar watermark-recovery lower bound. With the same k=128, T=100, and ϵ=0.25:
The two bounds differ by only 0.014 percentage points, retaining a similar recovery bound without the independence assumption.
W1-2&Q3 Ownership-level Reliability
Thank you for raising this question about verification reliability!
1. CAST maintains stable repeated extraction across different configurations. We vary the trigger, vocabulary key, watermark, and training seed on Qwen3-4B-Instruct-2507. We evaluated eight successfully watermarked models with twenty extractions per model (Appendix D.8, Table 18):
With the prespecified threshold of 82/128 matching bits, all 160 extractions pass ownership verification, including those with some incorrectly recovered bits.
2. CAST shows low false-positive rates in the tested ownership-verification settings. We test whether an unwatermarked model, or a model carrying a different watermark, is mistakenly accepted as the owner's model. Each setting uses 10,000 tests (Appendix D.8, Table 19):
Due to sampling randomness, we cannot guarantee a 0% FPR. The theoretical FPR of 0.09312% applies to a uniformly random watermark independent of the extracted result. The empirical results are close to the theoretical bound. The low FPRs complement the high ownership acceptance above, supporting reliable verification in these settings.
W2-1&Q2 Comparison with Memory-optimized EaaW
Thank you for suggesting this comparison!
CAST retains higher computation efficiency at comparable peak memory to optimized EaaW. Following the reviewer's suggestion, we implemented two-pass recomputation for EaaW: first compute the regression objective, then recompute model outputs to accumulate gradients without retaining all activation graphs. We added this comparison on 128-bit Qwen3-4B-Instruct-2507 in Appendix D.6 (Table 15) with a brief pointer in Section 3.5:
Recomputation brings EaaW's memory usage close to CAST's, but requires additional forward passes. CAST processes its local objectives sequentially without this recomputation, using 34% less time at comparable peak memory. CAST therefore retains its computational advantage against memory-optimized EaaW.