- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper looks for a provenance signal in how a model reacts under stress. They propose two methods: The whitebox method PAD compares the change in layer-to-layer representation alignment between ordinary inputs and adversarial ones. The blackbox method OQT asks underdetermined open questions, subtracts the answer by some standard reference answers, and compares what is left.
Critical strengths
The black-box method does improve the ranking scores. The authors show that a cutoff chosen on a few vendor lines does not travel.
Critical weaknesses
The black-box result is a ordering signal, not a reliable test of top-1. However I think in applications, people will care more about blackbox setting & top-1 result.
The white-box claim is not supported by a full comparison. Logits, clean representation matching, and the same matching on adversarial inputs appear for two victim models. The numbers that cover every family are PAD alone. The paper’s own text says the single-victim table does not show that PAD is generally better. A reader therefore cannot tell whether the clean-versus-adversarial difference is doing the work, or whether ordinary representation matching would have ranked these families just as well.
Questions for authors
Is the black-box cutoff still useful on a family that contains a real derivative, when that family was not used to choose the cutoff?
Full comparisons and ablations, clean-only vs PAD to strengthen PAD for PAD, and cutoff transfer for OQT
Q1. Full-cohort comparisons and ablations
We added full-cohort comparisons with REEF and Adv-REEF variants and a separate matched component ablation. All comparisons use the same 34 checkpoints, nine lineages, 561 pairs and 200 probe pairs. To include the six models with fewer than 18 layers, we use two layer-selection rules fixed before scoring: relative depth and the final layer. REEF uses clean inputs, whereas Adv-REEF uses adversarial inputs.
Table R4.1 reports PAD with SSIM matcher. At empirical FPR≤1%, PAD recovers 69/79 positives versus at most 52/79 for the four REEF/Adv-REEF variants; PAD and the highest-recall baseline each incur four false positives among 482 negatives. The Frobenius matcher gives TP/FP counts of 46/4 at the 1% FPR cap and 58/22 at the 5% cap.
Table R4.1. Full-cohort comparison with REEF and Adv-REEF variants.
Q2. Clean-only versus differential fingerprints
The matched ablation shows that subtraction improves low-false-positive detection. We compared the clean–clean alignment matrix C with the differential fingerprint D=C−A, where A is the cross-condition alignment matrix, using the same 34 models, 200 probe pairs, and model-pair labels. At empirical FPR≤5%, Frobenius yields numerically similar TP/FP counts: 59/79 true positives and 23/482 false positives for C. With SSIM, subtraction increases true positives from 70/79 to 74/79 while reducing false positives from 21/482 to 17/482, giving a net 5.06-percentage-point recall gain. Thus, under the same false-positive-rate cap, the differential fingerprint achieves both higher recall and fewer false positives than clean-only with SSIM.
Table R4.2. Clean-only versus differential fingerprints with SSIM on all 561 pairs.
Q3. Cutoff transfer
Thank you for the focused question; we have made the cutoff a separate experiment. The cutoff does transfer to a family with a real derivative that was not used to choose it. One family is held out at a time: the threshold is set on the unrelated pairs of the other four, and the held-out pairs are scored unchanged:
Table R4.3. Counts on the held-out family, with the threshold fixed on the other four.
Counts are pair evaluations summed over the five folds, not distinct pairs: 53 distinct co-release and 299 distinct unrelated pairs are ever scored.
Choice of rule. What it recovers then follows from the rate the verifier will accept: 1.5% gives half of the derivatives, and 5% or 10% gives the same half while accusing 20 and 28 unrelated pairs instead of 6. We report a tolerance because it is the quantity a verifier sets, and because recall stops increasing at 1.5%, so a looser setting buys accusations rather than derivatives. The full argument is in our answer to Reviewer PE1F.
Top-1. We do not claim a single-name verdict, and 12 positives could not support one. The resolution the score does deliver is a shortlist: the true ancestor lands in the top three for seven of the twelve pairs, and its mean rank is 5.7 of the 34 candidates where an arbitrary order gives 17.5. A verifier works through a handful of candidates instead of the whole pool, which is what the top-3 count and the mean rank report.
Planned revision: the counts above appear beside the ablation, and the score is read as a screening and ranking signal rather than a decision rule.