- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The author proposed a framework to resolve model ownership or lineage. The framework uses palimpsestic memorization characteristics of the model as the fingerprint; combining the fingerprint with cryptographic commitments and append-only public ledgers, the model owner can create a proof (and a timestamp). The model owner can challenge the release of a later-released model to check if this new model is finetuned from previous models without declaring the model lineage. Empirical evaluation is conducted with three families of large language models.
Critical strengths
The structure is clear, and leveraging palimpesestic memorization as a model fingerprint to resolve model ownership is novel. Resolving model ownership is an important field and can lead to practical solutions to protect the properties of the model trainer.
Critical weaknesses
About the structure of the paper:
- Palimpsestic memorization is a key concept used in the proposed framework. It shall be defined in the manuscript.
About the guarantee of the proposed mechanism:
- What’s the theoretical guarantee of using palimpsestic memorization phenomenon as a fingerprint (i.e., is the fingerprint a 1-to-1 mapping to the model)? For example, if an adversary steals the model, how easy is it to forge a dataset with similar pattern? Also, how likely are two independently trained models to share similar fingerprints? What are the TP, FP, TN, FN for using palimpsestic memorization as a model fingerprint in the contest/challenge?
- How is the audit panel selected? How does different selection of audit panels affect the effectiveness of the proof?
Others:
- Why focus on LLMs? Does palimpsestic memorization work for other ML models?
Questions for authors
Additional questions:
- How is the model initialization verified? Is it randomly initialized or a pretrained model checkpoint from other runs and how to verify them? Proof of Learning-based works also test whether the initial state of the model follows a claimed initialization process (e.g., random init, or a model weights released from another Proof of Learning certificate). Does the new method rely on similar mechanisms to check the model init?
4 comments
Details for W2.1 (false positives and false negatives)
Our fingerprint is not one-to-one, and this is by design. A fine-tuned copy has different weights from the original yet can retain the same training signal. That persistence is precisely what lets the original claimant challenge a later release that omits her claim. Conversely, one model can carry several fingerprints: a model Bob fine-tunes from Alice's can show both his trace and hers (Fig. 1). The fingerprint is evidence of provenance because a genuine descendant can retain the signal, while a model whose losses are independent of the claimant's random schedule matches only with bounded probability.
Can an adversary inspect the model and forge a matching dataset? We assume open weights. Once the honest model is released, the adversary may freely inspect the weights and choose examples by their losses. It must, however, cryptographically commit to the ordered dataset before the public beacon issues the fresh key that determines which entries are used for training and in what order. It can choose the examples, but it cannot then move low-loss examples into favorable positions under that commitment; a new ordering needs a new registration and a fresh key. Forging therefore reduces to passing the test by chance, which is the false-positive event bounded below. Release order is another safeguard: in a formal contest the judge (Def. 1 in App. B.1; Protocol 1) requires the challenger's release record to precede the target's, so a claim registered after the honest model was released cannot be used to contest it.
How likely are two independently trained models to share a fingerprint? Here "independent" means that the target's losses on the registered dataset are independent of the claimant's key. This covers a model trained on the same or overlapping data; what matters is that its training did not depend on which entries the key selected or on their order. The key assigns the training-position labels to the dataset entries uniformly at random, up to negligible cryptographic terms (beacon unpredictability, commitment binding, collisions in the keyed shuffle).
Detector used in our experiments (one panel, permutation threshold; Protocol 2). The audit compares the observed statistic with those from random reshufflings of the panel's labels and accepts when (App. B.3; and in our experiments). Fix the panel. Under independence, the registered labelling of the panel is itself just a random shuffle of the labels, so it is statistically indistinguishable from the reshuffled labellings. The observed statistic is therefore equally likely to hold any rank among the values, and requires it to rank among the largest. Hence the false-positive rate is at most per audit and the true-negative rate at least : 1% and 99% at the level used throughout.
Empirical check. We tested whether Pythia's training order is detected in the Pythia-deduped models. These were trained independently, on a deduplicated version of the same corpus (the Pile) and in a different order: nearly the same data, unrelated order.
| Model | Permutation p-value |
|---|---|
| Pythia-1.4B | 0.00947 |
| Pythia-1.4B-deduped | 0.14677 |
| Pythia-6.9B | 0.00031 |
| Pythia-6.9B-deduped | 0.20655 |
At the 1% level the original models are detected (true positives) and the independently trained deduped models are not (true negatives). All four models are scored on a shared audit panel of 10,000 Pile sequences, with 99,999 permutations of the Pearson statistic. Two independent models are a sanity check consistent with the bound. Kuditipudi et al. (2025) make the same comparison.
Details for W2.1, part 2: majority detector and false positive
A second detector with a provable false-negative bound. The permutation threshold depends on the data, which makes false negatives harder to bound. App. B.3 announces a second detector with an analytic threshold and a majority vote. App. B.3 now states its learning conditions; we give the definition and analysis below and will add them to the revision. The detector is defined as follows. After the target weights are fixed, the auditor uses fresh public randomness to draw an odd number of independent random subsets (panels) of examples from the registered dataset. The number of panels is a public parameter; we use 149 throughout, as in Fig. 6. Each example has a position label : zero if unused, otherwise its training position divided by the number of used examples, so . Let be its loss under the target model. On a panel, let be the covariance of and and the variance of . A panel passes if , where is a public threshold; a panel with fails. Since , this is the squared Pearson correlation of Protocol 2, scaled by and compared with a fixed threshold instead of a permutation threshold. The audit declares a match if at least 75 of the 149 panels pass.
False positives and true negatives (majority detector). Whatever the label values on a panel are, each way of matching them to the panel's examples is equally likely: the matching is a uniform shuffle . Write for the label that example receives. Standardize the panel's losses as , where is their mean on the panel, so that and . Then has mean zero and satisfies , so the panel passes only if .
Reveal the shuffled labels one at a time. Revealing position changes the expected final sum by , where is the mean of the later losses and is the mean of the labels not yet revealed: a larger label here leaves a smaller average for the remaining positions. Since labels lie in , this change ranges over an interval of length at most . Moreover : removing from the losses not yet passed lowers their sum of squared deviations by , and the initial sum is . The Azuma–Hoeffding inequality for this martingale gives At , the threshold used in Fig. 6, this is 2% per panel, so the expected number of passing panels is at most . A match needs 75, so by Markov's inequality its probability is at most . Markov uses only the per-panel probability, so the panels may share one key. Thus, for one audit of an independent target, the false-positive rate is at most 4% and the true-negative rate at least 96%.
This bound is conservative. Because , a panel passes only if , while a shuffled correlation has standard deviation . For general the same argument gives .
Details for W2.1, part 3: false negatives and repeated attempts
False negatives and true positives (majority detector). Fix a genuine target, either the released model or a modified descendant. Assume (i) used examples have lower mean loss than unused ones by at least , with variance at most in each group, and (ii) among used examples, later training position does not correlate positively with loss. With the fraction of examples used, the covariance over the dataset splits into a within-used term, nonpositive by (ii), and a between-group term of magnitude at least , since used labels average at least . Dividing by the standard deviation of , the normalized covariance has magnitude at least because this ratio only grows if the true gap exceeds . If , a random panel passes with probability at least . A panel can miss the signal in two ways. Its covariance can fall below half the dataset covariance in magnitude: the panel covariance has mean squared error at most because , so Chebyshev's inequality bounds this by . Or its loss variance can exceed four times the dataset variance: its expectation is at most the dataset variance, so Markov bounds this by . Outside both events , and the panel passes. For a fixed model and key the 149 panels are independent, so Hoeffding's inequality bounds majority rejection by ; 149 is the smallest odd panel count for which this is below 1%. Thus the false-negative rate is at most 0.95% and the true-positive rate at least 99.05% for genuine targets that satisfy (i)–(ii) with . This condition is sufficient, not necessary.
Measured false negatives (new Fig. 6). Fig. 6 reports the majority detector (149 panels, ) on the tested checkpoints with their true training order: 1,000 audits per cell, each resampling panels from a fixed pool of scored examples. The false-negative rate moves from 100% to 0% as crosses a model-dependent size. It is 0% at n = 3K for Pythia-6.9B and 12B, at 30K for CrystalCoder (100K pool) and for Pythia-1.4B on a 1M pool; for Pythia-1.4B and 2.8B on 10K pools the n = 10K cell is a single decision, which accepts. For OLMo-7B it is 9.8% at 60K and 0 of 1,000 at each of 70K, 80K, 90K and 100K (Wilson 95% upper bound 0.38%). At every smaller measured panel size all audits reject. These repetitions resample a fixed pool; they are not independent training runs.
In the contest: repeated attempts. The bounds above are for one audit. Let be the per-audit false-positive bound: for the permutation detector, for the majority detector. With registrations and audits per registration, the probability that any false claim succeeds is at most .
Both counts are limited in practice. Every registration and audit request is an authenticated, permanent ledger record (Def. 3), so attempts are public and countable. In a contest only claims released before the target are admissible, which fixes once the target is released. A deployment can then cap the number of audits per claim and target, charge a fee per request, or fix a cap in advance and require significance at level for every audit, which keeps that claim's overall false-positive rate at most . It can also lower or raise outright; for the majority detector the false-negative argument then needs , so the panel size grows linearly in .
We sincerely thank you for your thoughtful review and insightful feedback. We are glad that you find the structure “clear”, and resolving model ownership “an important field”, and our use of palimpsestic memorization to construct model ownership “novel”. We address your concerns and questions below.
Q1 (Model Initialization and Proof of Learning).
We would like to clarify that our protocol does not rely on a Proof-of-Learning initialization certificate: our guarantee concerns subsequent model lineage relative to a base model, so it does not need to verify how that base was trained. The framework accepts both random initialization and pretrained checkpoints as the base model.
W1 (definition and formalization)
We have added an explicit definition of both palimpsestic memorization and memorization in the revised paper Appendix B.3:
W2.1 (Theoretical guarantee)
Thank you for raising this important question. We agree that the formal guarantees of our protocol should be presented more clearly. We’d like to clarify that the fingerprint does not need to be one-to-one. Alice commits to her data before public randomness sets the training order, so Bob cannot forge a matching dataset and order afterwards. Following your suggestion, we give the false-positive and false-negative analysis in the follow-up comment below; we will add it to the revision. We show that a model generated independently of the claimed training schedule is matched to that schedule with bounded small probability. For a second detector that takes a majority vote over panels, we also bound the probability of missing a true match by 0.95% when the target model retains a sufficiently strong loss gap between used and unused data.
W2.2 (audit panel selection)
We agree that the selection of the audit panel is important. The audit panel is sampled pseudorandomly with a cryptographic hash seeded by fresh public randomness. Our guarantees rely on this random sampling. Since the panel is drawn at random from the population, a large enough panel gives a statistic close to the population’s, so different random panels give similar results.
W3 (Why LLMs?)
Thank you for this question. We focus on LLMs because this is where ownership disputes matter most today. Open-weight LLMs are widely shared, fine-tuned and re-released on platforms such as Hugging Face, often without clear lineage declarations. LLMs also give us the experimental setting we need, since open LLMs such as Pythia, OLMo and CrystalCoder publish their training order and intermediate checkpoints. The fingerprint could also work for other model families if training order leaves a detectable signal in their per-example losses.
We hope our clarifications and the added definitions, proofs and experiments address your concerns. If so, we would greatly appreciate it if you would consider raising your score. We are happy to answer any further questions.