acceptodds
Under review as a conference paper at ICLR 2027

Weights Have Traits: Estimating Fine-Tuning Lineage in LLMs

Abstract

Over a million deep learning models are hosted online, and most are fine-tuned descendants of a few ancestors. Their lineage is rarely fully documented, even though provenance, auditing, and safe deployment depend on it. We hypothesize that fine-tuning acts as an evolutionary process, so that weights have traits inherited from their ancestors. We show that a model phylogeny (evolutionary tree) can be estimated from the weights of the observed leaf models alone, with all ancestors latent. On controlled LLM fine-tuning trees whose true structure is known, our estimates contain every true clade (a group of models descended from one ancestor) in 85–100% of trees, compared with 3% for a random tree. Treating each weight layer as a gene-like estimator of lineage, we find that a single weight matrix, such as one attention key matrix (0.1–0.2% of the weights), estimates the phylogeny as well as all the weights combined. Moreover, estimation accuracy is positively associated with two measures from phylogenetics, the Atteson margin and a tree-likeness score based on four-point additivity. The latter needs no true tree. We also report two secondary findings. Models closer in weight space often behave more similarly on held-out tasks, and weight-based estimation matches behavior-based estimation (PhyloLM) on two documented real families while outperforming it on the controlled Llama trees (100% of clades versus 35–56%).

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
LLM, phylogeny, LLM behavior, fine-tuning, model lineage

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_FgwBOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a new method to build the phylogenetic trees of LLMs combining various ideas in the field: using weight based methods with NJ algorithm to not only focus on the relationship between two models but to build entire phylogenetic trees (something that was only done using behavioral methods so far). They show this new method outperforms behavioral methods exploiting this information stored in the weights of the model.

Critical strengths
  • the paper adresses a very timely and important topic and the authors very accurately explained why in the introduction
  • it is very well explained (except for the first weakness I gave below)
  • the authors tested their algorithm across various experiments (shared lineage, phylogenies, behavior prediction and in controlled and "in the wild" conditions)
  • this paper could well deserve being highlighted at the conference.
Critical weaknesses

These weaknesses are minor and only to help future readers better understand the paper:

  • The process of constructing the general tree from layer trees is not explained in the paper (its refers to Billera–Holmes–Vogtmann (BHV) Frechet mean which many readers will not be familiar with). Adding a formula and a scheme would greatly help understand the core of the algorithm - this is a crucial component of the method and should not be left as a reference. Readers will want to know how it works without having to read other related math papers.
  • It could also be interesting to report the compute time difference between PhyloLM and this new algorithm (theoretical complexity + actual compute time).
Questions for authors
  • how does the algorithm behave for model merging? Can it retrieve this very specific form of "phylogenetics" that operates on weights directly?
  • How does the algorithm behave if the common ancestors are included in the models tested? NJ is not tuned for this but this could still be a good ablation to add as, in practice, it can sometimes be hard to know where a model comes from.

Suggestion for future work: the NJ algorithm tries to rebuild the ancestors at each step. Maybe it could be adapted to find the ancestors among the set of models rather than building them. Not sure if this is going to work but it could be a way to improve the algorithm and which could be reused both in machine learning but also possibly in biological genetics directly.

2 comments
AuthorsOpenReview ↗

Response and thank you for review

Thank you for the thorough assessment and for suggestions that have improved the paper.

BHV Fréchet mean. We agree that this deserved more explanation. Section 3 now defines the mean as P̂ = argmin_T Σ_ℓ d_BHV(T, T_ℓ)², the tree that minimizes the summed squared distance to the layer trees T_ℓ. App. G explains the geodesic distance (Owen and Provan, 2011) and the algorithm we use to compute the mean, and includes a four-leaf schematic. One useful property is that a clade on which the layers disagree is pulled toward zero length and can be left as a polytomy rather than a forced resolution.

Compute time. On one hand, weight-based estimation needs no forward passes. Distances cost O(n²P) for n models and P selected parameters, and NJ costs O(n³) per layer, whereas PhyloLM generates 128 × 32 four-token continuations per model. We also measured both on the same cluster. For a controlled Llama-3.2-1B tree, weight distances take 9.3 s per model on CPU, and PhyloLM takes 10.1 s per model on one GPU. For the two documented 7B families, weight distances for all 20 models take 17 minutes on 8 CPU cores, and PhyloLM takes about 5.5 minutes per family and sampling seed on one GPU. The wall times are therefore similar, but weight distances need no GPU or prompts, and they do not vary with a sampling seed as PhyloLM's tree-likeness does. The real advantage in our method is that a model never has to be fully loaded into memory at any time.

Model merging. This is an interesting question that we have not yet tested. Since a tree cannot represent a model with two parents, a merged model would need to be flagged rather than placed in the tree, and we see detecting merges as a natural extension. Ancestors among the observed models. Thank you for suggesting this ablation, which we ran. With all or half of the ancestors included as observed models, NJ still estimates 100% of clades on the paper's Flan and Llama trees, and every included ancestor gets a shorter terminal edge than every leaf in its tree. When every model is observed, as in MoTHer's setting, a minimum spanning tree on MoTHer's distance also gives the correct undirected tree in every condition.

Finding ancestors among observed models. Thank you for the suggestion. We like this idea, and the ablation included in the above suggests it can work. When ancestors are included among the observed models, NJ gives every included ancestor a shorter terminal edge than every leaf in its tree (median 0.00012 versus 0.00277 for Flan), so the ancestor already sits almost exactly at the internal node NJ creates. NJ could therefore check, at each join, whether one of the joined models lies at the new node and use that model as the ancestor instead of building a new one. We have added this direction to the future work in the Limitations paragraph.

Reviewer_FgwBOpenReview ↗

Rebuttal answer

I thank the authors for their answers. I maintain my score and my very positive recommandation.

Reviewer_qfLPOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes that fine-tuning large language models acts as an evolutionary process where weights inherit ancestral traits. The reconstruction of a model's phylogeny (family tree) can use only the weights of observed leaf models. The authors treat each weight layer as a gene-like estimator by computing pairwise cosine distances between leaf models, applying the neighbor-joining algorithm to build per-layer trees, combining them via the BHV Fréchet mean, and introducing a label-free "tree-likeness" score to evaluate reliability without ground truth. Experiments on 46 controlled fine-tuning trees built from Flan-T5-base and Llama-3.2-1B demonstrate that this approach correctly identifies all true clades in 85–100% of trees, achieves full accuracy using just a single attention key matrix (0.1–0.2% of total weights), and significantly outperforms the behavior-based PhyloLM method on controlled benchmarks.

Critical strengths
  1. This is a very interesting topic and has widely applications.
  2. The proposed weight-based method successfully estimates every true clade in 85–100% of controlled fine-tuning trees, significantly outperforming a random baseline (3%) and behavior-only methods like PhyloLM (35–56%) on controlled Llama models. Furthermore, a single weight matrix—such as one attention key matrix representing only 0.1–0.2% of total weights—is sufficient to estimate the phylogeny as accurately as using all weights combined, making the approach highly computationally efficient.
  3. This method proves effective across different architectures (encoder-decoder Flan-T5 and decoder-only Llama-3.2-1B) and remains highly accurate under both full fine-tuning and low-rank adaptation (LoRA) regimes, even at very low ranks (e.g., r=8r=8).
Critical weaknesses
  1. The weight-based phylogeny estimation requires the underlying model architectures to be identical so that the weight spaces are compatible.
  2. Although the authors validate their method on two real-world Hugging Face families (Mistral-7B and Llama-2-7B), the primary controlled training trees were generated using a relatively simple Poisson branching process. These synthetic trees may not fully capture the complex, messy, and undocumented fine-tuning histories found in real-world public model hubs.
Questions for authors

Do strict architectural constraints often imply that the models themselves share a certain lineage, making the current algorithm seem redundant? Please explain for me, I will consider the score.

1 comment
AuthorsOpenReview ↗

Response and thank you for review

Thank you for the review and for the direct question, which we answer first.

Q. Does an identical architecture already imply shared lineage? We do not think so, for two reasons. First, within a family, architecture cannot distinguish between possible trees. All ten Mistral-7B models in App. D share one architecture, yet there are 34,459,425 rooted binary trees on ten models, and the question is which models descend from which. A random tree contains every true clade in only 3% of our controlled trees, whereas the weights do so in 83–100%.

Second, a shared architecture does not imply a shared ancestor, and we tested this directly on Pythia-160M (App. J). We used fifteen pretraining runs with an identical architecture and the same training data. Runs with independent initializations have nearly orthogonal weights, with a median cosine distance of 1.00 for the embedding, attention and MLP matrices. In contrast, the four runs that share an initialization remain close after 143,000 steps on different data orders, and NJ groups them together in all 148 layers, while runs that share only the data order are not grouped. The weights therefore reflect shared ancestry rather than shared architecture or data. However, we find your idea compelling and can imagine a scenario where architecture and data information are also fed into the model distances which can be used to create more comprehensive tree estimation.

W1. Identical architectures required. We agree, and the Limitations paragraph states this. The same requirement applies to other weight-based methods. Across architectures, behavior-based estimation such as PhyloLM can be used instead, and the paper compares against it directly.

W2. Poisson trees may be unrealistic. We agree that the generator is simple. The revision adds a realistic complication, relatives that reuse the same dataset, and shows that it is the main way estimation fails and that the fitted margin flags it (Table 3, App. I). The documented families in App. D also reflect real histories from Hugging Face.

Reviewer_tWEnOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates fine-tuning lineage estimation for LLMs assuming that only leaf models are observed while all intermediate ancestors remain latent. Treating each weight matrix as an independent "gene", the authors compute pairwise cosine distances, reconstruct layer-wise trees via NJ, and aggregate them using the BHV. Evaluated on various controlled fine-tuning trees and real-world Hugging Face families, the method recovers 92%–100% of true clades. The authors also show that a label-free tree-likeness score reliably predicts reconstruction accuracy, that weight proximity correlates with behavioral similarity on held-out tasks, and that weight-based estimation significantly outperforms the behavior-only PhyloLM on controlled trees.

Critical strengths
  1. The paper introduces classical phylogenetic tools to LLM weight spaces, which is novel and interesting.

  2. The experiments design are rigorous, and the analysis process is also thorough.

  3. The insights are meaningful. For example, it shows one attention key matrix (0.1–0.2% of the weights) suffices for exact lineage recovery, which has high engineering application value.

Critical weaknesses
  1. Nearly all weight-based variants achieve 100% clade recovery on Llama (Table 2), which may suggest the setting is somewhat too easy.

  2. The scope of the experiment is limited. The parameters of the selected models are small, containing only 250M and 1B. The FT process is also short (2000 steps). It is unclear whether the conclusion holds under larger models and continual learning.

  3. Although Table 1 lists prior weight-based methods, the experiments only compare the proposed method against a behavior-only baseline. Why not compare against simpler weight-based baselines?

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response. Thank you for the review!

Thank you for the thorough assessment! We address your comments below.

W1. The Llama setting may be too easy. Although the weights reach 100% on Llama, the same trees are not easy for other methods. On Llama, hierarchical clustering of our distances estimates 71–88% of clades, MoTHer's tree 72–81%, PhyloLM 35–56%, and a random tree 9% (Table 2, App. H). To push our method toward failure, we also retrained the 20 hardest topologies with narrower task pools (Table 3). Weights then estimate 65–100% of clades depending on the task pool, and 77% overall for Llama. The errors come from leaves in different lineages that were fine-tuned on the same dataset, and removing those leaves restores 100%.

W2. Model size, short fine-tuning, continual learning. Each fine-tuning runs for 2,000 steps, but leaves sit one to four sequential fine-tunings deep, each on a different task, so the deepest leaves have gone through four stages of training. The documented families are 7B models whose forks went through multi-stage pipelines, such as supervised fine-tuning followed by DPO or RL from AI feedback. We have also started retraining the 20 hardest topologies with every fine-tuning shortened to 500 steps, and with each fine-tuning in a tree assigned 500, 1,000, 2,000 or 4,000 steps at random. In the 11 trees finished so far at 500 steps and the 8 with mixed lengths, the weights estimate every clade, as they do at 2,000 steps. We will post the full results here before the discussion closes.

W3. Simpler weight-based baselines. Thank you for this suggestion. App. H now compares our method with five hierarchical clustering methods on our distances, which estimate 58–93% of clades, and with MoTHer, which estimates 63–82% when ancestors are latent. NJ applied to MoTHer's own distance estimates 98–100%, close to our 91–100%, so most of the gain comes from using a phylogenetic estimator rather than from the particular distance. MoTHer's rank distance does slightly better than cosine distance on Flan LoRA, while cosine distance applies to every training regime and to single weight matrices.

AuthorsOpenReview ↗

Update for added experiments

The fine-tuning length results for W2 are now in our reply to the general comment above. With every fine-tuning shortened to 500 steps, or with 500 to 4,000 steps assigned at random, the weights estimate every clade of all 20 harder topologies (App. I, Table 17). We also added the two more families.

I appreciate the authors' efforts to address my concerns. I will maintain my positive recommendation, but I now support the acceptance more than during the initial review.

Reviewer_S8sVOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper claims that we can reconstruct the evolutionary history of fine-tuned language models by examining their weights, even when their parent models and intermediate checkpoints are unavailable.

Critical strengths

Surprising finding that a single weight matrix is sufficient: This suggests that genealogical information is highly distributed or redundant across model parameters --> which is interesting.

Critical weaknesses

The important caveat is that their strongest results come from relatively small models (Flan-T5-base and Llama-3.2-1B), synthetically generated training trees, and compatible architectures. Their real-world validation involves only two documented model families.

Generally I like the idea but there are not enough experiments the authors should investigate bigger models.

Look at fine-tuning in terms of instruction tuning, also using RL, and even model distillation. The paper has potential but the experiments are not broad enough. (there are many models open sourced online based on which the authors could extend their analysis further without the need for much more compute).

Questions for authors

How much of the reconstruction performance comes from the phylogenetic framework itself, rather than the inherent clustering structure of inherited parameters? -->Could the authors compare against standard hierarchical clustering methods

How robust is reconstruction to different fine-tuning intensities? real-world fine-tuning trajectories can vary substantially in duration, learning rates, and dataset sizes

Why is a single weight matrix sufficient? The finding that one attention key matrix can recover the complete ancestry is intriguing, but its underlying explanation remains unclear --> does this also happens at all fine tuning intensities of is it only for small fine-tunes?

3 comments
AuthorsOpenReview ↗

Response. Thank you for the helpful review!

Thank you for the careful review and for the concrete suggestions! We have revised accordingly.

Larger models; instruction tuning, RL, distillation. The controlled trees use 250M and 1B models because every node of every lineage has to be trained. At 7B, the documented families in App. D include supervised fine-tuning, DPO, and RL from AI feedback, and the weights estimate every documented clade in both families. Because documented families need no training, we are adding two more that include distillation. The first is Llama-3.1-8B, with the Tulu 3 chain (SFT, then DPO, then RL with verifiable rewards), DeepSeek-R1-Distill-Llama-8B, and Hermes 3. The second is Qwen2.5-7B, with its Math and Coder continued-pretraining lines, DeepSeek-R1-Distill-Qwen-7B, and OpenThinker-7B. We will post the results here before the discussion closes.

Q1. Phylogenetic estimator vs. clustering. Thank you for this question, which we tested directly. The phylogenetic estimator does matter. On the same cosine distances, hierarchical clustering estimates 58–93% of clades, compared with 91–100% for our method (App. H). The difference is even clearer for single key matrices. Under UPGMA, none of them estimates every clade in any of the four conditions we checked, while under NJ all 12 Flan encoder keys and 9 of the 16 Llama keys do. The reason is that hierarchical clustering assumes every leaf has diverged from the root by the same amount. Our leaves are one to four fine-tunings deep, so that assumption fails, whereas NJ does not require it.

Q2. Fine-tuning intensity. The harder trees in Table 3 vary the fine-tuning data rather than its length, and there the weights estimate 65–100% of clades depending on the task pool. The submitted experiments used a fixed 2,000 steps per fine-tuning, and the revised Limitations paragraph now gives the preliminary results described below. To test fine-tuning length directly, we are retraining the same 20 hard topologies for Llama full fine-tuning in two ways, keeping the paper's datasets and learning rate. In the first, every fine-tuning runs for 500 steps. In the second, each fine-tuning in a tree is assigned 500, 1,000, 2,000 or 4,000 steps at random, so the edges of one tree differ in training length by up to a factor of eight. So far 11 trees have finished with 500 steps and 8 with mixed lengths, and the weights estimate every clade in all of them, as they do at 2,000 steps. On the 8 trees finished in both settings, the tree-likeness score is 0.855 at 500 steps and 0.843 with mixed lengths, compared with 0.854 at 2,000 steps, while MoTHer's tree estimates 96% (500 steps) and 67% (mixed length) of clades and PhyloLM 19% and 29%. We will post the full results here before the discussion closes.

Q3. Why one matrix suffices. This is a good question, and we have a partial answer. A leaf’s weight matrix equals the base matrix plus the updates on the edges from the root to that leaf, so the difference between two leaves is the sum of the updates on the path between them. If updates on different edges are nearly orthogonal, the squared distance between two leaves is a sum of per-edge terms, a path length on the tree, which is what NJ needs, and this holds for each matrix separately. On the paper’s trees, the median cosine between updates on two edges lies between −0.03 and 0.01. In the harder trees it rises to 0.26 (Flan) and 0.47 (Llama) for edges trained on the same dataset, which is where estimation fails (App. I). Layer norms, whose updates are far from orthogonal, are also among the weakest single estimators. Orthogonality may therefore be part of the explanation, but it does not show why Flan’s decoder matrices do less well than its encoder matrices, whose updates are about as orthogonal. In the layer trace (Fig. 3), errors in the decoder-only Llama are small and spread across layers, while in the encoder–decoder Flan they are concentrated in the decoder, so architecture also plays a role.

AuthorsOpenReview ↗

Update for promised experiments

We have posted the two results we promised, on fine-tuning length (Q2) and on two documented 7–8B families with distillation from DeepSeek-R1, as a reply to our general comment above. In brief, the weights estimate every clade of all 20 harder topologies at 500 steps and with mixed lengths from 500 to 4,000 steps. Across the four documented families, the weights and PhyloLM both estimate every documented clade in three, and App. D explains the one clade the weights miss.

Reviewer_S8sVOpenReview ↗

Comment

I will keep my current score.

The authos have expanded the experiments, but they have not demonstrated that their method scales to larger models in the controlled lineage-reconstruction setting. The real-world experiments are useful, but they do not establish that the reconstruction performance observed in controlled 1B trees generalizes to 7B+ models with complex, branching fine-tuning histories. I think is an interesting idea, but needs more experiments at scale.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.