- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes FamilyTrace, a black-box fingerprinting method for LLM family-level provenance identification. The key idea is that models from the same family share similar word association preferences even after model modifications. The method extracts fingerprints by asking models to rank candidate words under progressively informative contexts, and compares them with reference family fingerprints for closed-set and open-set identification. Experiments on 105 LLMs across six families demonstrate its effectiveness against various model modifications.
Critical strengths
-
The paper is well-written, with a clear motivation and problem formulation.
-
The fingerprinting design is simple and intuitive but effective. The experiments are comprehensive, covering various model modifications and comparisons with seven baselines.
Critical weaknesses
-
The paper assumes that FFN layers store much of the pre-training knowledge and are only slightly modified during post-training, leading to similar word association preferences within the same model family. However, these assumptions are not directly validated.
-
It is unclear whether FamilyTrace identifies model lineage or similarity in pre-training data. Independently trained LLMs sharing similar training corpora may exhibit similar word associations, potentially causing false attribution. Can increasing the number and diversity of anchors reduce this risk?
Questions for authors
-
Could targeted fine-tuning on anchor words alter or remove the fingerprint?
-
Could the authors further validate the assumptions about FFN layers and the effects of post-training?
-
Could similar pre-training corpora lead to false attribution across different LLM families? Would more diverse anchors help?
-
Missing reference on LLM fingerprinting: iSeal: Encrypted Fingerprinting for Reliable LLM Ownership Verification (Xiong et al., AAAI 2026).
3 comments
Response to Questions 3-4 and Weakness 2
Weakness 2: It is unclear whether FamilyTrace identifies model lineage or similarity in pre-training data. Independently trained LLMs sharing similar training corpora may exhibit similar word associations, potentially causing false attribution. Can increasing the number and diversity of anchors reduce this risk?
Question 3: Could similar pre-training corpora lead to false attribution across different LLM families? Would more diverse anchors help?
Response: In our setting, a family is one pre-trained base checkpoint and the models derived from it (Section 3). If the fingerprint reflected the pre-training corpus instead of the checkpoint, models of the same series would be hard to separate. Qwen-2.5-7B and Qwen-2.5-14B belong to a series pre-trained on one corpus of 18 trillion tokens (Qwen2.5 technical report, arXiv:2412.15115). They are separate families in the main benchmarks, and models of one are attributed to the other in at most 2.9% of the cases. This is a harder case than independently trained models with similar corpora, since the two models also come from one developer and share a unified vocabulary. Across the five Qwen-2.5 sizes from 0.5B to 32B in Appendix B.11, CS Acc. is 0.940. The fingerprint therefore follows the checkpoint rather than the corpus alone.
The risk the reviewer describes appears in the open-set setting. When Qwen-2.5-32B is held out as unknown, 20% of its models are accepted as another Qwen-2.5 size (TPR 0.800, Appendix B.11). These errors stay within one developer and series. We will state this boundary in the limitations, and the results below show that more anchors reduce it.
More anchors also improve attribution. In Appendix B.8, increasing the number of anchors from 3 to 7 raises CS Acc. from 0.882 to 0.939 on the non-adversarial benchmark and from 0.836 to 0.901 on the anti-provenance benchmark. To examine the risk directly, during the rebuttal we extended the default query list on the five Qwen-2.5 sizes of Appendix B.11 with query lists of Appendix B.12 that change the anchor words, which adds anchors from new semantic groups.
Table R6. Results on the five Qwen-2.5 sizes when the default query list is extended with lists of new anchors, averaged over 20 random draws of the added lists.
| Query lists | Queries | CS Acc. | OS Acc. | TPR | FPR | AUROC |
|---|---|---|---|---|---|---|
| 2 | 48 | 0.954 | 0.875 | 0.976 | 0.151 | 0.968 |
| 4 | 96 | 0.987 | 0.886 | 0.993 | 0.141 | 0.994 |
| 8 | 192 | 1.000 | 0.883 | 1.000 | 0.146 | 1.000 |
As shown in Table R6, TPR, the proportion of unknown models correctly rejected, and AUROC increase with the number of anchors and both reach 1.000 with 192 queries. More and more diverse anchors therefore reduce the risk that a family of the same series is accepted as a known one.
Question 4: Missing reference on LLM fingerprinting: iSeal: Encrypted Fingerprinting for Reliable LLM Ownership Verification (Xiong et al., AAAI 2026).
Response: Thank you for the pointer. iSeal (Xiong et al., AAAI 2026) embeds an encrypted fingerprint into a model before release and uses it to verify the ownership of that specific model. FamilyTrace addresses a complementary setting. It requires no change to the model and attributes already released models at the family level, including models into which no fingerprint was embedded. We will add iSeal to the related work in the revision.
Follow-up to Reviewer DYyb
Dear Reviewer DYyb,
Thank you again for your review. We have posted our responses to your comments, which discuss whether targeted fine-tuning on anchor words can alter the fingerprint (Q1), validate the assumptions about FFN layers and the effects of post-training (W1 and Q2), examine model lineage versus similarity in pre-training data and the effect of more diverse anchors (W2 and Q3), and add the missing reference to iSeal (Q4).
We would be very happy to discuss any remaining questions. We sincerely appreciate your time and effort, and we hope that our responses are helpful for your evaluation of our work.
Response to Questions 1-2 and Weakness 1
We thank the reviewer for the careful assessment and the insightful questions.
Response: Targeted fine-tuning can alter or remove the fingerprint if the attacker knows the anchor words or the query list. Section 5 simulates such an attack by tampering with the rankings of some anchors. Since the similarity to a family is the median over the six anchors, CS Acc. on the non-adversarial benchmark stays at 0.826 or above when two anchors are tampered with, but attribution fails with three or more (Table 11). This simulation corresponds to the case where fine-tuning fully controls the rankings of the targeted anchors. Such an attack is outside our threat model, which assumes that the provider does not adapt its responses to fingerprinting queries (Section 3.2), and the same risk applies to fingerprints that rely on fixed queries. The auditor can reduce this risk by keeping query lists private and generating a new list for each audit, so that the attacker does not know which anchors to target.
Response: The two observations about FFN layers and post-training are drawn from prior work (Section 1). They motivate our hypothesis that the models of a family largely share their word association preferences, and Section 5 tests this hypothesis through its predictions. In a controlled experiment with three base models and three fine-tuning datasets (Table 15), models fine-tuned from the same base model on different data are more similar to each other than models fine-tuned on the same data from different base models (mean pairwise similarity of 0.241 versus -0.005, with a p-value below 10−4 in a permutation test). Queries with common-sense orderings also carry less family information.
During the rebuttal, we also measured on the models of our benchmarks how large FFN layers are and how much post-training changes the weights. FFN layers hold 63.4% to 77.8% of the parameters of the six base models. For 74 fine-tuned models of our benchmarks, all 50 anti-provenance models and 24 non-adversarial models, we compared the weights with those of the parent model, using the Frobenius norm of the update relative to that of the parent weights, computed for each weight matrix and averaged in proportion to the matrix sizes. Table R5 reports the change for each family.
Table R5. Relative weight change of 74 fine-tuned models, compared with their parent models.
As shown in Table R5, post-training changes the weights of every family only slightly, by at most 2.12% for the FFN weights. Of the 74 models, 23 do not change FFN weights at all, since they are LoRA adapters that only update attention weights.
The size of the change also relates to the fingerprint. On the anti-provenance benchmark, a larger change goes with a lower own-family similarity (Spearman correlation of -0.827 over 50 models). On the non-adversarial benchmark, all 24 models keep an own-family similarity between 0.913 and 0.997. These results support the observation about post-training and indicate that larger changes of the weights preserve less of the fingerprint.