- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ModSleuth, an agentic system for reconstructing documented dependencies among model and dataset artifacts. It distinguishes training-related dependencies from indirect development influences, represents relationships through operations, and reconciles artifact references at different levels of specificity using an identity lattice. The system recursively processes public documentation to construct evidence-linked dependency graphs. Across four well-documented LLM releases, it recovers 1,060 verified relationships under the main attribution scope, exceeding the reported general-purpose baselines. Human and cross-model audits assess accepted-edge reliability, while case studies illustrate hidden upstream models, training–evaluation coupling, and license-relevant dependency paths.
Critical strengths
- The system captures dependencies beyond weight inheritance, including data generators, filters, rewriters, and judges. The evidence-linked case studies demonstrate the value of recursive, cross-source provenance tracing.
- The resource is supported by meaningful reliability checks: human annotators jointly support 284/300 sampled accepted edges, and a different model family upholds 483/500. The paper appropriately acknowledges that these checks do not measure missed dependencies.
Critical weaknesses
- The comparison does not isolate methodological gains from additional resources. ModSleuth performs staged, recursive investigations, whereas the main agent baseline uses a single-prompt configuration. Missing resource accounting, matched-budget comparisons, and component ablations prevent attribution of the gains to identity resolution and reconciliation.
- Edge-level verification does not establish the validity of composed dependency paths. The paper does not sufficiently specify how version uncertainty and mixed direct/indirect relationships propagate through multi-hop queries. For example, a training dependency of an evaluation-only component is not necessarily a training dependency of the target model.
Questions for authors
- Can you report the resource budgets and human involvement for ModSleuth and CC-single, and provide a matched-budget comparison or component ablation?
- How are direct and indirect dependencies composed along paths? In Table 2, can a training dependency reached through an evaluation-only edge be counted as a direct dependency of the target?
Response to Reviewer E2M5
We thank the reviewer for their thoughtful review. We are encouraged that the reviewer finds that our work can "demonstrate the value of recursive, cross-source provenance tracing" and "is supported by meaningful reliability checks." We respond to each question below and would be glad to discuss further during the discussion period.
The reviewer is correct that the paper’s results do not tell whether ModSleuth outperforms other agents when budget matched. Our central claim is that, while no other agent is able to perform the task meaningfully, ModSleuth is the first to make it practically feasible---a contribution independent of cost efficiency.
Nonetheless, we ran additional experiments to compare ModSleuth and CC under the same budget, by repeatedly prompting Claude Code until its token use matches ModSleuth. When evaluated with Olmo 3 as a seed, Claude Code finds 134 verified edges (after 12 repetitions), compared with 182 for ModSleuth.
Thank you for catching the lack of clarification of this point in the paper. Direct and indirect labels apply to individual edges, and a direct edge reached via an indirect edge should not establish a direct dependency of the target. Our original submitted Table 2 mistakenly counted a direct dependency reached via an indirect dependency as a direct dependency. We have corrected this and will clarify that an edge is a direct dependency only if it can be reached via direct edges alone, otherwise it is an indirect dependency. We have recomputed Table 2 with this rule and corrected it in the revised PDF.
On version uncertainty, each dependency claim is attached to the most specific identity that the evidence supports, so a claim about a family (e.g., "Olmo 3 32B") is stored on the family-level node rather than copied to each checkpoint (Section 3.1). On path validity, the audits in Section 4 are edge-level, while every multi-hop path that we report as a finding in Section 5.2 and Appendix E was manually verified against primary sources.