- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SAME, a framework for protecting the IP of open-source LLMs even after the LLMs' fine-tuning. The basic idea of SAME is to prepare a randomly-chosen private word embedding matrix, which is used to fine-tune the LLMs. As a result, the LLMs with such private word embedding matrix could behave exactly the same to the original LLMs with same input. This alignment could be used as a evidence for LLM IP verification. Further experiments demonstrate the watermark of SAME could survive further fine-tuning on downstream tasks.
Critical strengths
1.SAME is simple and clear, easy to understand.
Critical weaknesses
-
How the motivation of SAME induces its design is not well explained
-
Despite the effectiveness of SAME reported in the paper, the mechanic of SAME and how it works are not discussed: why LLMs could be fine-tuned with randomly chosen word embedding matrix?
-
For experiments, all baseline methods adopt different watermark dataset from SAME, which makes a unfair comparison. Since the construction of dataset is vital to this backdoor-like watermark injection, using same dataset for fair comparison is necessary.
-
Why SAME could survive fine-tuning and removal attack are not explained, making it unconvincing.
Questions for authors
- What is the social consequence if the LLMs owner choose not to embed watermark into their models? Since open-source model owners all assume that their models might be fine-tuned, why they still want to verify the model's IP after its fine-tuning?
- Since word embedding carries important semantic information for NLP tasks, it is hard to believe that LLMs can be trained with randomly chosen word embedding matrix. Why SAME could do that?
- What nature of SAME makes it robust to fine-tuning attack and watermark removal attack?
2 comments
Response to Reviewer zaWJ (Part 2/2)
What makes SAME robust to downstream fine-tuning and watermark removal? (W4Q3)
SAME persists because its native and private pathways are trained through shared semantic computation. During watermark injection, both pathways receive the same semantic supervision, linking the private codebook to the model’s semantic capabilities. During downstream adaptation, native-only fine-tuning updates the model’s semantic computation while the private codebook remains fixed. If the model retains its semantic capabilities, the private pathway can follow newly learned knowledge and task capabilities.
Our experiments support this functional coupling. Fig. 3 shows that the learned private vocabulary remains stronger than random controls after adaptation. Fig. 4 shows persistent native–private alignment, Fig. 5 evaluates repeated heterogeneous fine-tuning, and Fig. 6 shows coupled degradation and recovery under MEraser. Prior work on entangled watermarking, semantic sharing, multi-view representation learning, and watermark learnability provides related motivation [1–5]. These works support the intuition behind our design, but do not establish a theory or guarantee for SAME.
Existing watermark removal methods mainly target specific trigger–response patterns. SAME is coupled to the model’s shared semantic computation, so removing its private pathway may require damaging the model’s original semantic output capability. We evaluate this trade-off under the removal methods considered in the paper. Designing attacks that selectively remove SAME while preserving native utility remains an open problem, as discussed in Appendix F.
[1] Jia et al., “Entangled Watermarks as a Defense against Model Extraction,” USENIX Security Symposium 2021.
[2] Yang et al., “Rethinking Label-Wise Cross-Modal Retrieval from A Semantic Sharing Perspective,” IJCAI 2021.
[3] Hwang et al., “Multi-View Representation Learning via Total Correlation Objective,” NeurIPS 2021.
[4] Masrani et al., “Task-Agnostic Language Model Watermarking via High Entropy Passthrough Layers,” AAAI 2025.
[5] Gu et al., “On the Learnability of Watermarks for Language Models,” ICLR 2024.
Why embed a watermark and verify ownership after fine-tuning? (Q4)
Hugging Face’s Model Card specification provides a base_model field for fine-tuned, adapter, merged, and quantized models. The upstream license may also require attribution or modification notices. Documenting model provenance supports reproducibility and transparent release. SAME adds a functional ownership signal that remains verifiable after downstream fine-tuning.
Open-weight models can be fine-tuned, distilled, or merged and redistributed under new names, which can obscure the original developer’s contribution. Recent cases involving Cursor’s Composer 2 and Rio 3.5 show the practical importance of identifying upstream models [1, 2].
Fine-tuning can substantially change model parameters and behavior, so model names and external metadata may not be enough to verify provenance. SAME embeds a private watermark into the model’s learned computation, enabling ownership verification after downstream adaptation while preserving legitimate model reuse.
References
[1] TechCrunch, “Cursor Admits Its New Coding Model Was Built on Top of Moonshot AI’s Kimi,” 2026; Cursor, “A Technical Report on Composer 2,” 2026.
[2] Decrypt, “Rio de Janeiro Built an AI Model That Beat DeepSeek—But Was Based on Someone Else’s Work,” 2026.
Response to Reviewer zaWJ (Part 1/2)
We thank the reviewer for raising these important questions.
Fixed trigger–response watermarks are tied to a limited set of input patterns and may be overwritten by later adaptation. SAME addresses these issues by assigning every native token a corresponding private embedding. Private queries can then be composed across tasks. During training, native and private inputs pass through the same Transformer and receive the same target responses. This shared supervision links the private codebook to the model’s semantic computation, allowing it to follow knowledge and capabilities acquired during downstream fine-tuning.
The private vectors serve as stable token identities, while the Transformer learns their task-relevant semantic and contextual roles. Prior work supports the learnability of such representations: Transformers can operate with random Gaussian embeddings [1], frozen random input representations [2], hash-based or LSH-generated representations [3, 4], and random non-orthogonal embeddings [5]. Arpit et al. [6] further showed that deep networks tend to learn simple patterns shared across examples before memorizing individual samples. These findings support the use of training to align initially random private vectors with shared semantic computation. The method does not assume that an arbitrary random matrix works without training.
Our experiments provide direct evidence in the SAME setting. Fig. 2 shows improved private-path loss and held-out response quality. Fig. 3 shows that learned private vectors outperform random controls, confirming that the semantic behavior comes from the learned private–model pairing. Fig. 4 shows persistent alignment between native and private hidden representations across layers and adaptation. Together, these results support the claim that the private codebook learns a reusable semantic pathway. The owner can independently sample and keep this codebook private, making watermark creation reproducible while keeping independently sampled codebooks highly unlikely to collide.
[1] Kumar et al., “From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables,” AAAI 2022.
[2] Bochkov, “Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations,” TMLR 2025.
[3] Xue et al., “HashFormers: Towards Vocabulary-independent Pre-trained Transformers,” EMNLP 2022.
[4] Sankar et al., “ProFormer: Towards On-Device LSH Projection Based Transformers,” EACL 2021.
[5] Vural et al., “Learning to Recall with Transformers Beyond Orthogonal Embeddings,” ICLR 2026.
[6] Arpit et al., “A Closer Look at Memorization in Deep Networks,” ICML 2017.
The comparison is fair with respect to the downstream evaluation. As described in Section 4.1, all methods use the same Qwen3-8B base model and are implemented in a unified codebase. For each benchmark, Vanilla, SAME, and all baselines are evaluated on the same downstream dataset, using the same downstream fine-tuning protocol and the same utility and verification metrics. The results are therefore directly comparable under the downstream adaptation setting.
The watermark-injection data are intentionally different because the comparison targets a specific practical property. SAME is injected once on general Alpaca data and then reused across all downstream tasks. Each baseline is injected using the corresponding task-specific dataset. This is not an uncontrolled change in the downstream evaluation. It is the experimental distinction that tests whether SAME can support one-time injection and cross-task reuse, while the baselines require task-specific watermark injection. The tables therefore compare all methods under the same base model, downstream data, adaptation procedure, and evaluation protocol.