acceptodds
Under review as a conference paper at ICLR 2027

Which Models Are Our Models Built On? Auditing Invisible Dependencies in Modern LLMs

12 views

Abstract

Modern LLM training pipelines increasingly rely on other models to generate data, filter corpora, judge outputs, and guide development decisions. These dependencies are *recursive*: a model may depend on an upstream artifact whose own dependencies are documented only in separate releases and artifacts. As a result, the full dependency structure is fragmented across heterogeneous public artifacts, with complexity and recursive depth far outpacing humans' ability to trace. We introduce **ModSleuth**, an agentic system that recursively reconstructs LLM dependency graphs from public artifacts with source-grounded evidence. We find that the primary challenge is no longer information extraction, but defining what constitutes a dependency and reconciling artifact references across inconsistent documentation. We address these challenges through a formalization that distinguishes direct and indirect dependencies, represents heterogeneous pipeline roles through operation-centered relationships, and resolves artifact identities across names, versions, and repositories. Applying ModSleuth to four public-artifact-rich LLM releases, we recover 1,060 source-verified dependencies and construct large-scale dependency graphs of modern LLM development. These graphs reveal multi-hop license obligations, train–evaluation coupling, discrepancies between released and training-time artifacts, and documentation inconsistencies that would otherwise be difficult to uncover. We release ModSleuth and the resulting dependency graphs to support transparent analysis of the increasingly complex ecosystems underlying modern LLMs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, dependency graphs, model provenance, data provenance, LLM transparency, artifact auditing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_E2M5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ModSleuth, an agentic system for reconstructing documented dependencies among model and dataset artifacts. It distinguishes training-related dependencies from indirect development influences, represents relationships through operations, and reconciles artifact references at different levels of specificity using an identity lattice. The system recursively processes public documentation to construct evidence-linked dependency graphs. Across four well-documented LLM releases, it recovers 1,060 verified relationships under the main attribution scope, exceeding the reported general-purpose baselines. Human and cross-model audits assess accepted-edge reliability, while case studies illustrate hidden upstream models, training–evaluation coupling, and license-relevant dependency paths.

Critical strengths
  1. The system captures dependencies beyond weight inheritance, including data generators, filters, rewriters, and judges. The evidence-linked case studies demonstrate the value of recursive, cross-source provenance tracing.
  2. The resource is supported by meaningful reliability checks: human annotators jointly support 284/300 sampled accepted edges, and a different model family upholds 483/500. The paper appropriately acknowledges that these checks do not measure missed dependencies.
Critical weaknesses
  1. The comparison does not isolate methodological gains from additional resources. ModSleuth performs staged, recursive investigations, whereas the main agent baseline uses a single-prompt configuration. Missing resource accounting, matched-budget comparisons, and component ablations prevent attribution of the gains to identity resolution and reconciliation.
  2. Edge-level verification does not establish the validity of composed dependency paths. The paper does not sufficiently specify how version uncertainty and mixed direct/indirect relationships propagate through multi-hop queries. For example, a training dependency of an evaluation-only component is not necessarily a training dependency of the target model.
Questions for authors
  1. Can you report the resource budgets and human involvement for ModSleuth and CC-single, and provide a matched-budget comparison or component ablation?
  2. How are direct and indirect dependencies composed along paths? In Table 2, can a training dependency reached through an evaluation-only edge be counted as a direct dependency of the target?
1 comment
AuthorsOpenReview ↗

Response to Reviewer E2M5

We thank the reviewer for their thoughtful review. We are encouraged that the reviewer finds that our work can "demonstrate the value of recursive, cross-source provenance tracing" and "is supported by meaningful reliability checks." We respond to each question below and would be glad to discuss further during the discussion period.

"The comparison does not isolate methodological gains from additional resources. ModSleuth performs staged, recursive investigations, whereas the main agent baseline uses a single-prompt configuration. Missing resource accounting, matched-budget comparisons, and component ablations prevent attribution of the gains to identity resolution and reconciliation."

"Can you report the resource budgets and human involvement for ModSleuth and CC-single, and provide a matched-budget comparison or component ablation?"

The reviewer is correct that the paper’s results do not tell whether ModSleuth outperforms other agents when budget matched. Our central claim is that, while no other agent is able to perform the task meaningfully, ModSleuth is the first to make it practically feasible---a contribution independent of cost efficiency.

Nonetheless, we ran additional experiments to compare ModSleuth and CC under the same budget, by repeatedly prompting Claude Code until its token use matches ModSleuth. When evaluated with Olmo 3 as a seed, Claude Code finds 134 verified edges (after 12 repetitions), compared with 182 for ModSleuth.

"Edge-level verification does not establish the validity of composed dependency paths. The paper does not sufficiently specify how version uncertainty and mixed direct/indirect relationships propagate through multi-hop queries. For example, a training dependency of an evaluation-only component is not necessarily a training dependency of the target model."

"How are direct and indirect dependencies composed along paths? In Table 2, can a training dependency reached through an evaluation-only edge be counted as a direct dependency of the target?"

Thank you for catching the lack of clarification of this point in the paper. Direct and indirect labels apply to individual edges, and a direct edge reached via an indirect edge should not establish a direct dependency of the target. Our original submitted Table 2 mistakenly counted a direct dependency reached via an indirect dependency as a direct dependency. We have corrected this and will clarify that an edge is a direct dependency only if it can be reached via direct edges alone, otherwise it is an indirect dependency. We have recomputed Table 2 with this rule and corrected it in the revised PDF.

On version uncertainty, each dependency claim is attached to the most specific identity that the evidence supports, so a claim about a family (e.g., "Olmo 3 32B") is stored on the family-level node rather than copied to each checkpoint (Section 3.1). On path validity, the audits in Section 4 are edge-level, while every multi-hop path that we report as a finding in Section 5.2 and Appendix E was manually verified against primary sources.

Reviewer_ktECOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the largely invisible dependency relationships among open-source foundation models and introduces ModSleuth, an agentic system for reconstructing these dependencies from public documentation. ModSleuth combines multi-stage evidence extraction, identity resolution, and recursive dependency discovery to construct source-grounded dependency graphs. The resulting analysis reveals substantially more model relationships than existing documentation and highlights potential issues such as multi-hop licensing obligations and train–evaluation coupling.

Critical strengths
  1. The paper highlights an underexplored but practically important issue: model dependencies are often incomplete or difficult to trace from existing documentation, which can complicate provenance analysis, licensing, and auditing.

  2. ModSleuth provides an end-to-end workflow for recursively discovering, resolving, and verifying model dependencies.

  3. The reconstructed dependency graphs uncover substantially more relationships and several non-trivial patterns, including multi-hop dependencies and train–evaluation coupling, demonstrating the practical value of more systematic dependency reconstruction.

Critical weaknesses
  1. ModSleuth recovers more verified dependencies than generic agents, but the comparison does not control for inference and search effort, nor isolate the contributions of its key components. Without a budget-matched baseline and component ablations, it is unclear whether the gains come from the proposed design or simply greater search and execution effort.
  2. While the recovered dependency graphs reveal several audit-relevant findings, the main quantitative evaluation focuses on the number and correctness of recovered dependencies. It remains unclear how well these additional dependencies cover consequential risks or improve downstream auditing decisions.
Questions for authors
  1. How can the authors disentangle the gains from ModSleuth's structured design from those due to additional inference and search effort?
  2. How do the additional dependencies recovered by ModSleuth translate into practical audit value?
1 comment
AuthorsOpenReview ↗

Response to Reviewer ktEC

We thank the reviewer for their thoughtful review. We are encouraged that the reviewer finds that our work can "uncover substantially more relationships and several non-trivial patterns." We respond to each question below and would be glad to discuss further during the discussion period.

"ModSleuth recovers more verified dependencies than generic agents, but the comparison does not control for inference and search effort, nor isolate the contributions of its key components. Without a budget-matched baseline and component ablations, it is unclear whether the gains come from the proposed design or simply greater search and execution effort."

"How can the authors disentangle the gains from ModSleuth's structured design from those due to additional inference and search effort?"

The reviewer is correct that the paper’s results do not tell whether ModSleuth outperforms other agents when budget matched. Our central claim is that, while no other agent is able to perform the task meaningfully, ModSleuth is the first to make it practically feasible---a contribution independent of cost efficiency.

Nonetheless, we ran additional experiments to compare ModSleuth and CC under the same budget, by repeatedly prompting Claude Code until its token use matches ModSleuth. When evaluated with Olmo 3 as a seed, Claude Code finds 134 verified edges (after 12 repetitions), compared with 182 for ModSleuth.

"While the recovered dependency graphs reveal several audit-relevant findings, the main quantitative evaluation focuses on the number and correctness of recovered dependencies. It remains unclear how well these additional dependencies cover consequential risks or improve downstream auditing decisions."

"How do the additional dependencies recovered by ModSleuth translate into practical audit value?"

First, in addition to the quantitative results, Section 5.2 also presents qualitative findings related to concrete, consequential risks. For example, SmolLM3's pretraining data FineMath is filtered by a classifier trained on Llama-3-70B-Instruct annotations, which has possible license implications even though the SmolLM3 card does not mention Llama. Nemotron 3 Super uses RL data derived from SWE-Bench-Verified, while reporting SWE-Bench-Verified during evaluation. These dependencies span multiple artifacts and hops, and are difficult to find from any single source.

Second, the issues we found are widespread, as we find the same patterns across releases. For every evaluated model, over 75% of verified dependencies come from external organizations, and benchmarks such as GSM8K, MATH, and IFEval are used in both training and evaluation.

Third, we believe ModSleuth does address an impactful and consequential problem: tracing which upstream models contribute to a released model. These dependencies often have legal, regulatory, commercial, and competitive implications. Their importance is already reflected in real-world’s policy, e.g., the European Commission’s mandatory template for public summaries of AI training content [1] requires general-purpose AI providers to report their synthetic data use.

ModSleuth directly supports two groups of users. For external auditors, it turns scattered, difficult-to-analyze evidence into a source-grounded dependency graph, significantly reducing the time and efforts for them. Model developers can also use the same workflow for pre-release self-auditing: identifying potential license or policy concerns, improving disclosures, and modifying the training pipeline when needed. Section 5.2 illustrates how this can affect development decisions, e.g., filtering Llama-Nemotron data to exclude samples touched by Llama models to comply with Llama's license restrictions.

In summary, although ModSleuth is not intended to make automated legal or compliance judgements, it can make practical auditing significantly more efficient and systematic---an increasingly important capability as model-development pipelines become highly complex and interconnected.

[1] https://digital-strategy.ec.europa.eu/en/faqs/template-general-purpose-ai-model-providers-summarise-their-training-content

Reviewer_z7PZOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors present an agentic harness tool for constructing a dependency graph for an LLM based on its documentation and the documentation of its recursive dependencies. The dependency graph is designed to capture diverse forms of dependencies present in training pipelines of current models. Performance of the tool is compared to directly using frontier agents. A dependency graph generated using four models as seeds is analyzed.

Critical strengths

The presented tool is carefully designed and may be useful, and the paper is beautifully clear and well-written.

Critical weaknesses

My main concern is that the significance of ModSleuth, the main presented contribution, is unclear. Sadly contributions which may have been significant in past years now seem less so given current coding agents.

The presented dataset contains dependencies of only 4 seed models. The dataset's size is minimal compared eg to "Navigating Dataset Documentations in AI" (cited in the paper), and the presented analysis is shallow. While some issues were uncovered by the presented tool it is unclear how widespread or impactful they are.

Questions for authors

It would help if the authors elaborated on their scientific contributions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer z7PZ

We thank the reviewer for their careful and constructive review. We are encouraged that the reviewer finds our work "carefully designed" and considers our paper "beautifully clear and well-written." We respond to each point below and would be glad to discuss further during the discussion period.

"My main concern is that the significance of ModSleuth, the main presented contribution, is unclear. Sadly contributions which may have been significant in past years now seem less so given current coding agents."

We agree that modern coding agents are highly capable, and that a contribution centered only on building another agent would be less significant today. However, ModSleuth’s contributions go beyond another implementation of agents.

First, we formulate the problem of recursive model-dependency tracing, an increasingly important problem as LLM pipelines use upstream models in various different ways. We define direct and indirect dependencies, address ambiguous model/dataset identities, evaluate several general-purpose agents, and present quantitative and qualitative findings. These contributions are orthogonal to the particular agent used.

Second, today’s general-purpose coding agents are still not sufficient for this task yet. As discussed in the paper, the challenges are not information gathering and extraction---exactly because the coding agents perfectly solve them. Rather, the challenges are: deciding what constitutes a dependency, and resolving ambiguity in model and dataset names, versions, and relationships. ModSleuth addresses these challenges in a targeted way. As additional evidence, we evaluated the current version of Claude Code (see our response to Reviewer ktEC). Even when we ran it iteratively to make it even more competitive, Claude Code finds 134 verified edges for Olmo 3, compared to 182 for ModSleuth.

To summarize, improvements in coding agents are complementary to our work. Stronger agents can improve ModSleuth’s underlying extraction engine, but we think orthogonal contributions, such as task definition, dependency semantics, identity resolution, and audit findings remain valuable.

"The presented dataset contains dependencies of only 4 seed models. The dataset's size is minimal compared eg to "Navigating Dataset Documentations in AI" (cited in the paper), and the presented analysis is shallow."

While there are only 4 seed models, ModSleuth does recursive expansion, so our full graph contains 2,526 model and dataset artifacts and 9,112 dependency edges, with 1,060 source-verified relationships attributed to the four seeds. This includes upstream artifacts such as Olmo 2, Tulu 3, Nemotron 3 Nano, Nemotron-Cascade, the FineMath classifier, and UltraRM. On the other hand, "Navigating Dataset Documentations in AI" analyzes individual dataset cards, and does not analyze models or the relationships between artifacts.

"While some issues were uncovered by the presented tool it is unclear how widespread or impactful they are."

First, these issues are widespread, as we find the same patterns across releases. For every evaluated model, over 75% of verified dependencies come from external organizations, and benchmarks such as GSM8K, MATH, and IFEval are used in both training and evaluation.

Second, we believe ModSleuth does address an impactful and consequential problem: tracing which upstream models contribute to a released model. These dependencies often have legal, regulatory, commercial, and competitive implications. Their importance is already reflected in real-world’s policy, e.g., the European Commission’s mandatory template for public summaries of AI training content [1] requires general-purpose AI providers to report their synthetic data use.

ModSleuth directly supports two groups of users. For external auditors, it turns scattered, difficult-to-analyze evidence into a source-grounded dependency graph, significantly reducing the time and efforts for them. Model developers can also use the same workflow for pre-release self-auditing: identifying potential license or policy concerns, improving disclosures, and modifying the training pipeline when needed. Section 5.2 illustrates how this can affect development decisions, e.g., filtering Llama-Nemotron data to exclude samples touched by Llama models to comply with Llama's license restrictions.

In summary, although ModSleuth is not intended to make automated legal or compliance judgements, it can make practical auditing significantly more efficient and systematic---an increasingly important capability as model-development pipelines become highly complex and interconnected.

[1] https://digital-strategy.ec.europa.eu/en/faqs/template-general-purpose-ai-model-providers-summarise-their-training-content

Thank you for your response. My concerns were not addressed, and I maintain my rating.

Could you please elaborate on which concerns you feel were not addressed in our response? We would be happy to provide further clarification and engage in a constructive discussion of any outstanding issues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.