- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces Matryoshka fingerprinting, a fingerprinting scheme to simultaneously protect authorship on multiple models. The scheme achieves this by optimizing a latent fingerprint in the representation space of multiple models simultaneously, with the larger model's representation encapsulating the smaller model's latent a la Matryoshka embeddings. Once the optimized latent is obtained, an unused token is assigned this embedding for each model. The paper evaluates this scheme on multiple model families, as well as multiple deployment settings, demonstrating near perfect fingerprinting rates.
Critical strengths
- The black box nature of the fingerprinting scheme is nice.
- Thorough investigation on FPR, robustness, stealthiness etc. Also the adaptive attacks presented are nice
- Most baselines are covered well.
Critical weaknesses
- The experimental results seem to be quite misleading to me. For Matryoshka FP the fingerprint is explicitly optimized for all models, which is not the case for REEF or PlugAE, which only optimize the FP on a single model and are testing its "generalization". In the case of the reported evals, REEF should also not really trigger on other models, since it has a high specificity. A fairer comparison would be to check Matryoshka FP on other related models which were not included for optimizing the FP, or to extend the baselines to use multiple models. A better baseline would be to modify, e.g. PlugAE to account for different models in searching the fingerprint. For example Table 12 reports that using 2 models gives some generalization for Matryoshka FP, and I believe that similar generalization gains would occur for PlugAE as well. In fact this idea of using multiple models for optimizing the fingerprint is not particularly novel, with MergePrint, or RAFP both using this and reporting some gains.
1.1 Further, the metrics of similarity and TPR are not comparable. The paper acknowledges this but still presents them together. I would argue that with similarity one needs to use a threshold to have a low false positive rate and then use that threshold to compute the TPR.
- The threat model is unclear. What is the motivation to protect all models in a family with the same fingerprint? Indeed, this is bad from a security perspective since leaking one fingerprint destroys fingerprints across the whole model family (Nasery et al 2025). I get the generalization arguments but these seem to be insufficiently explored in the paper empirically, e.g. the empirical results from from Table 12 cast some doubt on the efficacy of MF, especially under varying hidden sizes.
Questions for authors
- Could the authors provide intuition on the differences between their method and PlugAE, which lead to much better performance, e.g. on the Mistral models in Table 1.
- What is the motivation of using the same fingerprint to protect all models in a family?
The following are minor comments and questions
- How is the vocab manipulation attack done?
- What is the threshold for detection and the number of fingerprints?
- What is in Prop 1? It would be good to have some actual numbers for this, e.g. it seems that for and vocab size of 32k.
- In terms of adaptive attacks, the attacks (suppress-top-k, suppress-top-word) from Nasery et al 2026 seem to be directly applicable here. Do the authors have a sense of how these would work?
- All experiments seem to be for invisible tokens, so why can an attacker not just strip those out?
- In Table 8 what does the perplexity value denote and how was it computed?
refs Are robust llm fingerprints adversarially robust?, Nasery et al, SaTML'26
Thank you Reviewer HdF1. We answer each question below.
R1 (Why MF performs better than PlugAE). Both MF and PlugAE optimise token embeddings to encourage models to produce a target response. However, the key difference is how the fingerprints are constructed. PlugAE generates fingerprints for individual models, while MF jointly optimises a shared fingerprint across multiple models. This allows MF to account for differences between models during optimisation, rather than relying on a fingerprint designed for one model to transfer to others. This helps explain MF's better performance in Table 1, particularly on Mistral models. Moreover, MF supports models with different hidden sizes, whereas PlugAE's fingerprint embeddings have a fixed dimensionality. Another important advantage of MF is its stronger robustness against model modifications. To verify this, we have conducted additional experiments comparing MF with PlugAE under fine-tuning and pruning on four models: Llama-2-7B, Mistral-7B-v0.1, Qwen2.5-7B, and Phi-3-mini-4k-instruct. The results are summarised below.
R2 (Why use the same fingerprint for multiple models). Model providers often release many versions and sizes of their models. If every model needs a different fingerprint, the provider must create, store, and verify each one separately. MF makes this easier by allowing multiple models to share the same fingerprint. This reduces the work needed to manage fingerprints and makes ownership verification simpler across different models.
R3 (How is the vocab manipulation attack done). We have described how this attack is performed in Appendix E.1.2 on Page 12 (Paragraph 3).
R4 (Detection threshold and number of fingerprints). In our experiments, we measure TRR directly by checking whether each fingerprint query produces the expected response. Therefore, no detection threshold is needed to determine whether an individual query succeeds. In reality, when verifying model ownership using multiple fingerprint queries, the detection threshold can be set based on the application's requirements. Moreover, the number of fingerprint queries used in each experiment is provided in Appendix E.1.
R5 (Clarification of Proposition 1). In Proposition 1, Δm is the minimum logit gap between the target token and its strongest competing token across the considered prompt templates and generation positions. Proposition 1 shows that when this gap is large enough, the target response remains stable across different templates, even when the decoding temperature changes within the specified range. This explains why MF can consistently trigger the same response under different prompt templates. We will clarify the definition of Δm and add a numerical example to make the result easier to understand.
R6 (More adaptive attacks). Following your suggestion, we have conducted additional experiments to evaluate MF against three adaptive attacks based on Nasery et al.: SuppressNeighbor, SuppressTop-k, and SuppressLookahead. We have tested each attack using 100 fingerprint queries on two models, Llama-2-7B and Mistral-7B-v0.1. The results are summarised below.
R7 (Why not simply remove invisible tokens). As discussed in Appendix G, MF supports both invisible and visible tokens, each with its own trade-offs. Invisible tokens are harder to notice but can be removed by text sanitization. Visible tokens are more noticeable but cannot be removed by simply stripping invisible characters. Users can choose the token type that best suits their needs, as explained in Appendix G.
R8 (What does perplexity mean in Table 8). Perplexity (PPL) is a well-known metric that is used to measure how likely a piece of text is under a language model. Lower PPL generally means that the text looks more natural to that model. As described in Appendix E.1.2, we use 100 clean prompts and four reference models (OPT-6.7B, Mistral-7B-v0.1, Qwen2.5-7B, and Phi-3-mini). We calculate PPL using the average negative log probability of the tokens. We compare clean prompts with prompts containing fingerprints, both before and after word swapping, random deletion, and random addition. Table 8 reports the average PPL. We will explain this calculation more clearly in the paper.