acceptodds
Under review as a conference paper at ICLR 2027

TokenPrint: A Calibrated Token-Space Fingerprint for Language-Model Provenance

Abstract

Establishing the provenance of a language model – including its base checkpoint and possible overlap in training distributions – s a governance challenge that metadata alone cannot resolve. We introduce a training-free fingerprint based on the top- vocabulary projections of late hidden states elicited by 250 fixed knowledge probes, compared using Jaccard overlap over decoded token strings. We evaluate the method on 32 open-weight models from nine families (0.6B–32B) with documented relationships. (1) A similarity ladder}broadly follows model relatedness: independently trained models on identical data score 0.48 raw (0.35 vocabulary-corrected), followed by shared-base fine-tunes (0.39/0.33), same-developer relatives (0.38/0.28), and models with no documented relationship (0.22/0.17). This identical-data signal persists across three organizations, two tokenizer families, and two architecture classes, and emerges within the first 1% of training before measurable task competence, suggesting a contribution from shared training data beyond capability convergence. (2) As a nearest-neighbor lineage-retrieval method, the fingerprint ranks the exact documented base among the top two candidates for all five R1 distillations (mean rank 1.8, MRR 0.60), including a math-specialized base not identifiable from coarse metadata. (3) A depth ablation shows that lineage group discrimination strengthens toward the output distribution, with AUC increasing from 0.72 at quarter depth to 0.90 at the output; using only the top 5 output tokens retains AUC 0.87. (4) The fingerprint remains stable under quantization, with Jaccard similarity of 0.92 under int8 and 0.82–0.85 under int4, compared with a maximum cross-model similarity of 0.81 in the calibration pool. We release the probes, code, and fingerprints.

Primary area:
interpretability and explainable AI
Keywords:
LLM, Provenance, Alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZwEMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free fingerprinting method for identifying model lineage groups. The method selects the top-kk vocabulary tokens by projecting hidden states at selected depths onto the vocabulary space and compares fingerprints from different models using Jaccard similarity across 250 probes. The experimental results demonstrate the method's effectiveness and robustness.

Critical strengths
  1. The proposed fingerprinting method is training-free, making it potentially more cost-effective than existing methods. The experiments also demonstrate its robustness under various conditions.
Critical weaknesses
  1. Although the experimental results show that fingerprint similarity is consistent with documented model lineage relationships, the comparative evaluation is limited. In particular, the paper does not experimentally compare the proposed method with most of the related methods discussed in the related-work section, making it difficult to assess its performance relative to existing approaches.

  2. The paper's presentation becomes less clear starting from Section 3 (Method). The notation (e.g., PP and FF in Equation (1)), the explanation of excess similarity in Section 3.3, the rationale for selecting the probe suite, and the explanation of the evaluation metrics (e.g., AUC for lineage-group identification) could be presented more clearly. This makes the methodological foundations of the work difficult to follow and weakens the presentation of its contributions.

Questions for authors
  1. Why do you not compare the proposed method with other related methods? Are such comparisons unnecessary, or are direct comparisons difficult because of differences in experimental settings?

  2. Have you tested alternative probe-suite selections to assess how sensitive the results are to the choice of probes?

Reviewer_T8KyOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TokenPrinter a new method to fingerprint LLM Lineage. Using a Jaccard distance computed from a top-k of unembeded activations throughout the transformer stack, TokenPrinter successfully retrieves LLM lineage, under the form of fine-tuning, training on the same data or model from the same provider. The paper proposes different ways to test the algorithm to take into account the effect of the tokeniser, LLM architecture etc... It validates its method using more than 32 open LLMs.

Critical strengths
  • the paper proposes a new method for an increasingly important question. Indeed since the rise of LLMs a couple years ago many models are published everyday, yet little is known about their training methods or relationship to previous models and the field is becoming more and more opaque at this level.
  • the method is new and seems to yield relevant results.
Critical weaknesses
  • the most important weakness of the paper is that despite citing relevant literature like PhyloLM it does not efficiently compare itself to such algorithms. This growing body of literature published a couple of very related algorithms each with their own upsides and limitations and it would really be important that TokenPrinter highlights what it brings to this scientific line of research even more so because some of these methods do not present the same tokeniser issues than TokenPrinter. For example comparing accuracy of methods to the cited PhyloLM [1] but also possibly to LLMDNA [2] both published in ICLR the two previous years would be particularly important to situate the paper in its field. Without this comparison of upside and downsides (accuracy but also compute cost of the algorithm) this work is orphaned and very hard to estimate its contribution yet to an already existing field of research.
  • Then the paper only tests one set of probes, it would be important to test one or two other sets of probes to test consistency of the results across inputs. Also most claims of similarity are based on 3 to 6 models (cf upper side of Table 1). It would be interesting to have more models to test the robustness of these results. It can be hard to find multiple models trained on the same dataset but it is very easy to find multiple models fine-tuned from the same base (cf [1] and [2]).
  • Finally some minor comments: line 153-154 isn't it supposed to be S(a,b)−S~(a,b)S(a,b)-\tilde S(a,b) (first S missing), Figure 4 the legend is hiding an important part of the figure. Putting the exact equation of S^\tilde could be helpful. It is not clear to me if most figures use the average of the SlS_l or just one of them? This aggregation of the SlS_l for various l could be made clearer. Also it could be more transparent to say how many glitch tokens were removed just to make sure it is discarding only a very low number of them and the process is benign.

[1] Yax, N., Oudeyer, P.-Y., & Palminteri, S. (2025). PhyloLM: Inferring the phylogeny of large language models and predicting their performances in benchmarks. In International Conference on Learning Representations (ICLR 2025). https://doi.org/10.48550/arXiv.2404.04671 [2] Wu, Z., Zhao, H., Wang, Z., Guo, J., Wang, Q., & He, B. (2026). LLM DNA: Tracing model evolution via functional representations. In International Conference on Learning Representations (ICLR 2026). https://doi.org/10.48550/arXiv.2509.24496

Questions for authors
  • see my minor comment points in weaknesses - essentially some notation clarifications.
Reviewer_LZDLOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

TokenPrint fingerprints the language model by the tokens its later layers like on a fixed set of probe questions, comparing the models by overlapping corrections with a common protocol. In the open weight model pool, the similarity decreases from pairs of identical data through recorded fine-tuning and same-lineage models to pairs with no recorded relationship. Even in different organizations and architectures, models trained on the same public corpus are similar to each other, and the signal is already present early in training. For lineage retrieval, the recorded base ranked among the top two candidates in every refined inference model tested but once was the top candidate.

Critical strengths

The same corpus experiment tested what drives similarity: models from multiple organizations (including non-Transformers) are much more similar to each other than to pairs of unrelated models, and the signal is present before the model performs the task. This separates shared data from capability fusion and is the main contribution of this paper. The authors also demonstrate the limitations of the approach (a pair of independently trained old models scores as high as a fine-tuned pair) and report retrieval failures alongside the success of a single best candidate.

Critical weaknesses

One of the main arguments, stability under quantification, lacks experimental support. Quantitative checkpoints appear only in one table, and the title of the table references an analysis that does not exist and claims a very slim advantage over unrelated pairs. This argument, which is not supported by evidence, is the main reason why I lowered my rating. The reliability of the search results has also not been confirmed. In most queries, the database listed in the document is very close to the best candidate, no uncertainty is provided for each pair of databases, and the only robustness check (removing the probe field) is a simple yes/no decision.

Questions for authors
  1. Please report the specific details of the quantization experiment: model, method, and comparison of full-precision score and quantized score.

  2. A "no document association" baseline refers to a situation where document records are missing. How can we rule out baseline differences caused by corpus overlap?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.