acceptodds
Withdrawn from ICLR 2027

Reconstructing LLM Lineage Trees via Fine-Tuning Path Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-tuned (FT) derivatives of large language models (LLMs) are now released in large numbers, yet their provenance relies on self-reports and cannot be verified. Misattributed lineage leaves a verifier unable to audit outsourced fine-tuning, confirm license compliance of derived models, or trace vulnerabilities that a parent passes to its children. All three require the direction of derivation, not merely its presence. Existing output-only methods are limited to pairwise derivation tests or undirected lineage estimation, while recovering a directed tree requires white-box access to model weights. We formalize the problem of reconstructing a directed model tree, including internal nodes, under gray-box access, where only teacher-forced log-likelihoods are observed and weights are never inspected. Fine-tuning raises the log-likelihood of the training data, and a child inherits this increase from its parent. Building on this property, we define an asymmetric per-task log-likelihood deviation between any two models as a trace of fine-tuning history, and derive from it an edge weight for each ordered pair of models. We propose TRACE-Edge, which exactly solves for the maximum spanning directed tree under these edge weights, with neither hyperparameters nor tree-structure constraints. We evaluate it on five configurations totaling 25 model trees built from three base models and seven tasks. TRACE-Edge achieves a parent-match accuracy of 0.74 when probed with the fine-tuning examples, and on average outperforms an output-similarity baseline even with held-out probes drawn from the same distribution as the training data.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
model provenance, model lineage, model tree recovery, large language models, fine-tuning, gray-box access, log-likelihood

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_u58KOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses the problem of model lineage tracing. Output similarity typically only determines whether models are related or reconstructs undirected structures; reconstructing directed model lineages usually relies on weighted access. The authors aim to recover internal nodes and parent-child directions using only teacher-forced log-likelihood under gray-box conditions. TRACE-Edge designs directed edge scores based on the likelihood differences across tasks, and then uses the Chu–Liu/Edmonds algorithm to find the maximum weighted directed spanning tree.

Critical strengths

The problem is meaningful, and the distinction between "model similarity" and "direct parent-child relationship" is worth investigating. The authors also clearly demonstrate that gray-box access is superior to black-box access that only observes the output. The experiment included five configurations, five seeds for each, totaling 25 trees. In Table 2, when using training samples as probes, TRACE-Edge's parent-match accuracy was 0.740, better than MST's 0.620; TRACE-Set was even higher at 0.837. The paper does not conceal the fact that another method yielded better results in this setting.

Figures 3 and 4 are helpful in explaining the reasons for the failure. Under the training sample probe, most of the ancestor task signals were preserved; after switching to held-out samples, these signals were significantly weakened. The positive ancestor task recognition rate given in the text decreased from 0.917 to 0.458, which is consistent with the decline in tree recovery performance.

Critical weaknesses

W1: The advantage under realistic probe conditions is small and unstable. The held-out accuracy in Table 2 is 0.558, and the MST is 0.507; in the HET-8B configuration, the TRACE-Edge is only 0.233, lower than the MST of 0.567. Therefore, the method cannot yet be considered to stably recover the lineage of different models.

W2: Full parameter fine-tuning results significantly weaken the generalization claim. In Appendix D.4 and Table 6, under the held-out condition, TRACE-Edge is 0.500, but the MST reaches 0.867. This is not a "minor degradation" in the general sense, but a reversal of the main advantage.

W3: All nodes are observed, the root node is usually known, the task set is fully known, and nodes share the fine-tuning configuration. Real-world model source tracing often lacks intermediate checkpoints and includes unknown tasks, mixed data, and different hyperparameters.

W4: 1. For Case (4), the fraction in the table is written as �0��0�, but Equation (7) should actually take the maximum value of �1��1�. For example, when �2��2�, the fraction in the formula is 1, but the table shows 9. This row needs to be corrected unless additional conditions are added.

Questions for authors

Q1 Why does it significantly lose to MST in a Full-FT held-out setup?

Q2 How does Simple Distance MST perform under the same log-likelihood access conditions?

Q3 Please correct Table 1 Case (4) and clarify when edge scores can actually distinguish between parent nodes and more distant ancestors.

1 comment

Thank you very much for your detailed review and for recognizing the problem setting. We ran additional experiments following the suggestions from you and the other reviewers. Their results require substantial revisions to both the content and the claims of the paper. We have therefore decided to withdraw the submission and resubmit it after revision. Thank you again for your time and effort.

Reviewer_iWJZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes TRACE, a gray-box framework for reconstructing directed lineage trees of fine-tuned large language models (LLMs) without accessing model parameters. The authors exploit the observation that fine-tuning increases the log-likelihood of training examples and that these changes may be inherited by subsequent fine-tuned models. Experiments on 25 model trees across five configurations, constructed from three pre-trained models and seven fine-tuning tasks, show that TRACE achieves a parent-match accuracy of 0.740 when using actual fine-tuning examples and 0.558 when using held-out samples.

Critical strengths
  1. The paper addresses the problem of reconstructing directed lineage relationships among fine-tuned LLMs, which is an interesting research direction for model provenance verification, intellectual property protection, and license compliance.
  2. The paper provides experimental results across different base models, fine-tuning tasks, and tree configurations. The evaluation considers both training examples and held-out samples, providing insights into how the availability of fine-tuning data affects lineage reconstruction.
Critical weaknesses
  1. The related work section overlooks closely related studies on fine-tuned LLM attribution, such as [1]. This prior work investigates how to attribute fine-tuned LLMs to their original pre-trained models, sharing similar motivations concerning model provenance. Although the proposed TRACE framework addresses the more general problem of reconstructing complete directed model lineage trees, the authors do not sufficiently discuss how their approach differs from or extends this line of research. The paper should provide a more comprehensive review of existing model attribution methods and clearly establish its contributions relative to these studies.
  2. The experimental evaluation primarily compares the proposed methods against a trivial star baseline and an MST baseline adapted from Model Provenance Testing (MPT). Although closely related model lineage reconstruction methods, including PhyloLM and MoTHer, are discussed, their potential adaptations as baselines or direct comparisons are not sufficiently explored.
  3. The proposed method appears to rely heavily on access to the exact fine-tuning examples, raising serious concerns about its applicability to realistic model provenance verification scenarios. Specifically, the parent-match accuracy of TRACE-Edge drops from 0.740 to 0.558 when the probe set changes from training examples to held-out samples, while that of TRACE-Set drops from 0.837 to 0.500. More importantly, the accuracy of identifying ancestor fine-tuning tasks decreases substantially from 0.917 to 0.458, suggesting that the proposed likelihood-based signals may not generalize reliably to unseen samples. Since the actual fine-tuning datasets are typically unavailable when auditing third-party models, the framework's effectiveness under the more realistic held-out setting is particularly important. However, TRACE-Edge only marginally outperforms the MST baseline in this setting, and its performance varies considerably across model configurations. These results raise concerns about whether the proposed method captures reliable model lineage information or primarily exploits model-specific fitting to the original fine-tuning examples. The authors should further investigate how to reconstruct model lineage when the exact fine-tuning datasets are unavailable.
  4. The experimental evaluation considers only five tree configurations, four of which have a maximum depth of two, while the remaining configuration has a depth of three. Such shallow trees may not adequately represent complex model development pipelines involving many successive fine-tuning stages. In particular, the proposed method relies on the assumption that fine-tuning traces can be inherited across multiple generations. The authors should evaluate significantly deeper and larger lineage trees, to investigate whether the proposed method can reliably recover long fine-tuning histories.
  5. The experiments are restricted to three relatively small pre-trained models: Llama-3.2-3B, Llama-3.1-8B, and Qwen2.5-3B, with the largest containing only 8B parameters. These models may not adequately represent the diversity and complexity of modern LLMs. Since the proposed method relies on changes in log-likelihood induced by fine-tuning, its effectiveness may vary substantially with model capacity, pre-training capabilities, and knowledge retention behavior. Notably, TRACE-Edge achieves a parent-match accuracy of only 0.233 on HET-8B under the held-out setting, compared with 0.667 on HET-3B, suggesting considerable sensitivity to the underlying base model. The authors should consider larger models and investigate whether the proposed likelihood-based signals remain reliable across more capable models.
  6. Although the paper introduces an asymmetric edge-scoring function as a central component of TRACE-Edge, the experimental evaluation does not sufficiently isolate the contributions of its individual components. Specifically, the proposed score combines the largest positive log-likelihood deviation with penalties based on the second-largest positive deviation and the most negative deviation. However, the paper does not demonstrate whether these penalty terms are necessary or how much each contributes to lineage reconstruction accuracy. Furthermore, the paper provides limited sensitivity analysis regarding important experimental factors, such as probe set size, fine-tuning epochs, learning rates, and LoRA configurations. More comprehensive ablation and sensitivity studies are needed to justify the proposed scoring design and demonstrate its robustness.
  7. The proposed framework assumes that the verifier has gray-box access to all models, including the ability to obtain log-likelihoods for arbitrary input-output pairs. However, such information may not be available through many deployed LLM APIs, limiting the method's applicability to real-world model auditing scenarios. The authors should clarify the practical feasibility of these assumptions and evaluate the framework's robustness under more restrictive API access conditions.

[1] Foley, Myles, et al. "Matching pairs: Attributing fine-tuned models to their pre-trained large language models." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Questions for authors

See Weakness above.

1 comment

Thank you very much for your detailed review and for recognizing the problem setting. We ran additional experiments following the suggestions from you and the other reviewers. Their results require substantial revisions to both the content and the claims of the paper. We have therefore decided to withdraw the submission and resubmit it after revision. Thank you again for your time and effort.

Reviewer_PWD3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reconstruction of directed LLM lineage trees using teacher-forced log-likelihoods without inspecting model weights. The verifier observes all models in a tree, knows the root and a task set covering the fine-tuning history, and obtains task-specific probe examples.

The proposed methods use differences in task-level log-likelihoods as evidence of fine-tuning history. TRACE-Set infers parent relationships from inclusion among estimated ancestral task sets. TRACE-Edge constructs asymmetric edge scores and uses the Chu–Liu/Edmonds algorithm to find a maximum-weight rooted directed spanning tree.

Evaluation covers five synthetic tree configurations with five training seeds each, using three base models and seven tasks. TRACE-Edge achieves 0.740 parent-match accuracy with training probes and 0.558 with held-out probes, compared with 0.620 and 0.507 for an output-disagreement MST baseline. Additional experiments investigate root estimation and full-parameter fine-tuning.

Critical strengths
  1. The paper addresses a useful intermediate-access provenance problem. Reconstructing directed parent-child relationships, including internal nodes, from likelihood observations is a meaningful extension beyond pairwise similarity testing. The explicit formulation of the verifier’s access and prior knowledge helps clarify when such reconstruction may be possible.

  2. The proposed reconstruction pipeline is simple and globally consistent. TRACE-Edge converts task-level likelihood differences into a rooted directed spanning tree without requiring known depths, branching factors, or a tuned threshold. The experiments also examine important failure conditions, including held-out probes, similar tasks, deeper trees, unknown roots, and full-parameter fine-tuning. The candid analysis of weakened ancestral signals is a useful aspect of the work.

Critical weaknesses
  1. The edge score can prefer an ancestor over the direct parent even when the assumed inheritance pattern holds perfectly. Consider a chain r → p → m with three probe tasks. Suppose the task-likelihood increments are (1.0, 0, 0) from r to p and (0, 0.1, 0) from p to m. Each update affects exactly one task, and previous improvements are perfectly retained. Nevertheless, Eq. 7 gives s(p → m) = 0.1 but s(r → m) = 0.9. Together with s(r → p) = 1.0, the maximum-weight tree favors attaching both descendants directly to r rather than recovering the true chain. Thus, the stated sparsity and inheritance intuition alone does not justify the score’s preference for immediate parents. This is consequential because cross-task likelihood scales can differ substantially, as the paper itself notes for gsm8k. The method needs either additional assumptions or stronger empirical evidence that this failure mode is controlled.

  2. The empirical advantage is not established against strong baselines with the same access, and generalization evidence remains limited. The main external baseline uses top-1 output disagreement, whereas TRACE uses richer teacher-forced likelihood observations. A symmetric distance between task-likelihood profiles, followed by MST construction and orientation from the known root, would help determine whether asymmetry is actually responsible for the improvement. With held-out probes, TRACE-Edge’s average advantage is only 0.051, and it performs substantially worse than MST on HET-8B (0.233 versus 0.567). In the additional full-FT configuration, held-out accuracy is 0.500 versus 0.867 for MST. Moreover, the 25 main trees represent five seeds of five configurations with mostly the same small topology, rather than broad structural diversity. These results leave the method’s reliability beyond favorable training-probe conditions uncertain.

Questions for authors

The following clarification and validation requests could materially affect my assessment:

  1. Under what additional conditions does Eq. 7 prefer the direct parent over a more distant ancestor? How frequently does the unequal-task-increment failure described above occur in the evaluated trees, and would task-wise normalization change the results?

  2. How does TRACE-Edge compare with an MST based on distances between teacher-forced task-likelihood profiles, using the same probes and known root? Please provide a paired uncertainty estimate for the held-out accuracy difference across the evaluated trees.

  3. How sensitive is reconstruction to varying fine-tuning strength across nodes, repeating the same task along a lineage, or training each node on a task mixture? These settings directly challenge the assumption that one additional edge produces a distinctive single-task increment.

  4. Can you clarify the construction of the training and held-out probe sets? Appendix D.2 describes drawing probes from the training split, while the held-out condition requires exclusion from the actual fine-tuning examples. Please specify the fine-tuning subset sizes and the exclusion procedure.

1 comment

Thank you very much for your detailed review and for recognizing the problem setting. We ran additional experiments following the suggestions from you and the other reviewers. Their results require substantial revisions to both the content and the claims of the paper. We have therefore decided to withdraw the submission and resubmit it after revision. Thank you again for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.