- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a new method to build the phylogenetic trees of LLMs combining various ideas in the field: using weight based methods with NJ algorithm to not only focus on the relationship between two models but to build entire phylogenetic trees (something that was only done using behavioral methods so far). They show this new method outperforms behavioral methods exploiting this information stored in the weights of the model.
Critical strengths
- the paper adresses a very timely and important topic and the authors very accurately explained why in the introduction
- it is very well explained (except for the first weakness I gave below)
- the authors tested their algorithm across various experiments (shared lineage, phylogenies, behavior prediction and in controlled and "in the wild" conditions)
- this paper could well deserve being highlighted at the conference.
Critical weaknesses
These weaknesses are minor and only to help future readers better understand the paper:
- The process of constructing the general tree from layer trees is not explained in the paper (its refers to Billera–Holmes–Vogtmann (BHV) Frechet mean which many readers will not be familiar with). Adding a formula and a scheme would greatly help understand the core of the algorithm - this is a crucial component of the method and should not be left as a reference. Readers will want to know how it works without having to read other related math papers.
- It could also be interesting to report the compute time difference between PhyloLM and this new algorithm (theoretical complexity + actual compute time).
Questions for authors
- how does the algorithm behave for model merging? Can it retrieve this very specific form of "phylogenetics" that operates on weights directly?
- How does the algorithm behave if the common ancestors are included in the models tested? NJ is not tuned for this but this could still be a good ablation to add as, in practice, it can sometimes be hard to know where a model comes from.
Suggestion for future work: the NJ algorithm tries to rebuild the ancestors at each step. Maybe it could be adapted to find the ancestors among the set of models rather than building them. Not sure if this is going to work but it could be a way to improve the algorithm and which could be reused both in machine learning but also possibly in biological genetics directly.
2 comments
Rebuttal answer
I thank the authors for their answers. I maintain my score and my very positive recommandation.
Response and thank you for review
Thank you for the thorough assessment and for suggestions that have improved the paper.
BHV Fréchet mean. We agree that this deserved more explanation. Section 3 now defines the mean as P̂ = argmin_T Σ_ℓ d_BHV(T, T_ℓ)², the tree that minimizes the summed squared distance to the layer trees T_ℓ. App. G explains the geodesic distance (Owen and Provan, 2011) and the algorithm we use to compute the mean, and includes a four-leaf schematic. One useful property is that a clade on which the layers disagree is pulled toward zero length and can be left as a polytomy rather than a forced resolution.
Compute time. On one hand, weight-based estimation needs no forward passes. Distances cost O(n²P) for n models and P selected parameters, and NJ costs O(n³) per layer, whereas PhyloLM generates 128 × 32 four-token continuations per model. We also measured both on the same cluster. For a controlled Llama-3.2-1B tree, weight distances take 9.3 s per model on CPU, and PhyloLM takes 10.1 s per model on one GPU. For the two documented 7B families, weight distances for all 20 models take 17 minutes on 8 CPU cores, and PhyloLM takes about 5.5 minutes per family and sampling seed on one GPU. The wall times are therefore similar, but weight distances need no GPU or prompts, and they do not vary with a sampling seed as PhyloLM's tree-likeness does. The real advantage in our method is that a model never has to be fully loaded into memory at any time.
Model merging. This is an interesting question that we have not yet tested. Since a tree cannot represent a model with two parents, a merged model would need to be flagged rather than placed in the tree, and we see detecting merges as a natural extension. Ancestors among the observed models. Thank you for suggesting this ablation, which we ran. With all or half of the ancestors included as observed models, NJ still estimates 100% of clades on the paper's Flan and Llama trees, and every included ancestor gets a shorter terminal edge than every leaf in its tree. When every model is observed, as in MoTHer's setting, a minimum spanning tree on MoTHer's distance also gives the correct undirected tree in every condition.
Finding ancestors among observed models. Thank you for the suggestion. We like this idea, and the ablation included in the above suggests it can work. When ancestors are included among the observed models, NJ gives every included ancestor a shorter terminal edge than every leaf in its tree (median 0.00012 versus 0.00277 for Flan), so the ancestor already sits almost exactly at the internal node NJ creates. NJ could therefore check, at each join, whether one of the joined models lies at the new node and use that model as the ancestor instead of building a new one. We have added this direction to the future work in the Limitations paragraph.