- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks whether a black-box suspect LLM comes from a given source. From MMLU, the authors keep questions where the source is stably wrong. The suspect is asked the same questions, and the score is how often it picks that same wrong option. The evaluation shows advantage compared with several baselines.
Critical strengths
Stemma is clearly ahead of LLMmap, reduced-budget LLMPrint, MPT, and ZeroPrint. The threat model is clear, and the method is easy to adopt.
Critical weaknesses
The closest prior work is ErrorTrace (NeurIPS 2025), not those four baselines. ErrorTrace already uses characteristic mistakes as a provenance signal. The two papers do have differences: Stemma is more fine-grained, but the paper mentions ErrorTrace in one sentence and never compares with it. As written, it is unclear which difference with ErrorTrace leads to the performance gains.
Questions for authors
I am not asking for a full comparison with ErrorTrace on all the datasets, but it would be helpful to do some ablation study against ErrorTrace on some datasets.
Response to Reviewer dnue
We thank the reviewer for highlighting the connection to ErrorTrace and suggesting an empirical comparison. We agree that ErrorTrace and Stemma share a common starting point in leveraging errors on multiple-choice questions for fingerprint construction. The two methods, however, differ in their detection objectives, provenance signals, and information requirements. We have conducted a systematic methodological and experimental comparison. We summarise the key findings below.
(1) Methodological differences. ErrorTrace primarily targets family-level attribution, whereas Stemma verifies provenance against a specific source checkpoint. Beyond these different objectives, although both methods leverage errors on multiple-choice questions, they use them in fundamentally different ways. ErrorTrace constructs fingerprints from characteristic error spaces (question sets identified through reference-model error patterns) and scores suspects by tracking changes in the gap between their observed and expected error rates as these sets shrink, relying on aggregate error patterns across the sets. In contrast, Stemma assesses induced decision region inheritance through source–suspect agreement on specific answer choices, prioritising erroneous source decisions for their greater specificity and focusing on individual-question-level decision outcomes. Regarding information requirements, ErrorTrace needs multiple reference models from related and unrelated groups for fingerprint construction, demanding greater prior knowledge of and access to reference models, whereas Stemma requires only the source model and one unrelated background model.
(2) Experimental comparison. We adapt ErrorTrace to source-specific verification by treating the seven provenance groups as distinct families and replacing its original datasets with four MMLU supercategories, while preserving its core error-space construction and contraction mechanisms. For efficiency, we precompute answer-choice predictions for all 56 checkpoints using local white-box inference (approximating greedy decoding) and reuse them for both methods.
We evaluate both methods on the same source–suspect pairs, with 4 held-out positive and 24 negative suspects per source (196 pairs across seven sources), across 18 reference-size configurations (k=2,…,7 groups; n=2,…,4 models per group) and five random seeds. Evaluation pairs remain fixed across configurations within each seed. We adapt ErrorTrace's gain-based scoring to produce continuous verification scores and report AUROC, standardised pAUC@5% FPR, TPR@1% FPR, and d′ . Key source-level macro results are summarised below.
Best, Median, and Worst are selected by macro AUROC among the 17 configurations with successful construction across all targets and seeds, excluding the configuration with ErrorTrace construction failures.
ErrorTrace achieves competitive performance under favourable configurations, marginally exceeding Stemma's near-saturated macro AUROC in only two of the 18 configurations (0.990–0.991 vs. 0.987). However, Stemma consistently achieves stronger low-FPR discrimination across all evaluated configurations, while requiring substantially fewer reference models. Notably, ErrorTrace's best-AUROC configuration requires 20 reference models, involving over 10× as many model–question evaluations for fingerprint construction and approximately 5.8× as many queries for initial verification.
Beyond reference set size, we also found that ErrorTrace's performance is sensitive to the specific reference models selected. To investigate this, we conducted a separate experiment using instruction-tuned sources, fixing k=3, n=3, and the evaluation pairs while varying reference composition. In one Qwen2.5-14B case, selecting less similar rather than highly similar references improved ErrorTrace's AUROC from 0.426 to 1.000. In another Qwen2.5-7B case, AUROC increased from 0.611 to 0.944. In comparison, Stemma achieved AUROCs of 1.000 and 0.989, respectively, on the same evaluation pairs without requiring such reference selection. These examples highlight the practical challenge of selecting suitable references for ErrorTrace fingerprinting construction, which Stemma avoids.
The complete experimental protocols and results are provided in Appendix O, with a corresponding cross-reference added to the Related Work section.