- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposed a method MPS for model provenance. Given a target model and a candidate models set, it tries to find a small source set that contains all source models of the target model. MPS computes a distance for each prompt between each candidate and the target model. Then it centers the distances per prompt, computes a studentized t-statistic for each candidate (Eq. 6), takes T_min, and gets a p-value by permuting model labels within each prompt (Eq. 9). It then iteratively moves the argmin candidate into the source set while the null "all candidates are equally close" is rejected. The overall process is kinda like Model Confidence Set but in reverse [1]. Overall, the paper have a good theoretical grounding. The statistics machinery might not be new but the formulation is novel and applies to the problem well.
[1] Hansen, Peter R., et al. “The Model Confidence Set.” Econometrica, vol. 79, .no 2, Econometric Society, 2011, pp. 453-497, https://doi.org/10.3982/ECTA5771
Critical strengths
-
The problem is important and timely. Treating multi-source provenance as set-valued inference, including the "no source" case, is a useful framing that many existing tests do not provide in a measurable way.
-
The method needs only black-box access, works with any distance, is simple, and is cheap once outputs are cached.
-
There is a sizeable benchmark built from real Hugging Face lineage metadata (562 models).
-
The sensitivity analyses are broad (α, R, T, |M|). The same-family study (G.3) honestly reports failure modes.
-
Code and data are released. The paper is generally clearly written.
Critical weaknesses
-
The null and the assumptions can be unrealistic. The authors' own G.3 shows the assumption breaks with same-family candidates, and this can limit the effectiveness of the proposed method in cases the models in the candidate sets are all derived from the same ancestor model or shared large amount of overlapping training data.
-
The benchmark sidesteps the hard cases. Negatives in the experiments are drawn from different families, however real audits face siblings, same-family models and descendants. In addition, since similarity is symmetric, MPS cannot tell ancestors from descendants or siblings.
-
The guarantee is framed incorrectly. A rejection adds a model to source set . So the type-I error governs false inclusion, not missed sources. Fig. 3(a) shows exactly what you would expect if controls inclusions: coverage stays flat as grows, while set size grows.
-
The paper is internally inconsistent in several places.
a. Fig. 10 contradicts Fig. 2. In Fig. 10, Semantic-MPS with |M*| = 1 has mean first-stage p ≈ 0.22-0.32. Since p ≤ 1, P(p ≤ α) ≤ (1−E[p])/(1−α), which caps first-stage rejection at about 0.72-0.82. A source is included only if the first stage rejects, so coverage cannot exceed this. Yet Fig. 2 reports coverage of about 0.95 in the same settings.
b. |M| = 0 "coverage" is always 1.0, yet set sizes are above 0.* Fig. 3(a) shows set size ≈ 2 at α = 0.3 and |M| = 100; the bars in Fig. 2 are about 1. Under the paper's definition this coverage is trivially 1. Under the code's definition (mcs_evaluation), the output must be empty. Table 4 uses yet another, non-trivial value (0.97).
c. Fig. 10's null mean p is about 0.85–0.98, whereas valid p-values under an exact null average about 0.5. The test is strongly conservative or uses a mismatched null. Reading a large p as "evidence of non-infringement" (G.5) confuses absence of evidence with evidence of absence.
d. The code in the anonymous repo is inconsistent with the paper.
-
There's label noise: unsloth/* models (34 of the 462 depth-1 chains) are mostly re-uploads of the identical model, about half of them have byte-identical weight files comparing to the upstream model and the other half are mostly bf16 converted versions. They're not derived child and detecting them is trivial. Shouldn't weaken the main conclusion of the experiments, though.
Questions for authors
-
Please restate exactly what α controls. Report coverage separately from the false-inclusion rate, preferably including for |M*| = 0.
-
How do you reconcile Fig. 10 with Fig. 2?
-
How are Bench-A pools of size 50 built? Does the FWER extension correct for multiplicity across candidates? How was REEF's threshold tuned?