acceptodds
Under review as a conference paper at ICLR 2027

Stemma: Induced Decision Regions Reveal LLM Provenance

Abstract

LLM provenance testing asks whether a suspect LLM belongs to the same lineage as a source. Existing black-box methods largely rely on evidence tied to particular elicitation interfaces or response-level characteristics, but these signals may shift under prompting, adaptation, or deployment without changes in model lineage, weakening provenance reliability. To address this limitation, we introduce induced decision regions by mapping open-ended outputs into a shared finite decision space, abstracting away surface-form variation and reframing provenance testing around induced decision region inheritance. Empirical analysis shows stronger inheritance in related models and reveals that the provenance signal is unevenly distributed across probes, with discriminative power concentrated in a small subset whose informativeness transfers across model pairs. Building on this formulation, we propose Stemma, a black-box LLM fingerprinting method that estimates induced decision region inheritance using probes selected for stability, robustness, and specificity. Across 770 source-suspect pairs from 56 public checkpoints spanning diverse model transformations, Stemma achieves 0.967 AUC and 87.1% TPR at 1% FPR. It further achieves 0.995 AUC and 92.4% TPR at 1% FPR on 1,260 pairs covering 91 deployment instances, demonstrating robustness to deployment variations. Across both benchmarks, Stemma substantially outperforms four representative baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM provenance, model fingerprinting, model lineage, intellectual property protection, induced decision regions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dnueOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether a black-box suspect LLM comes from a given source. From MMLU, the authors keep questions where the source is stably wrong. The suspect is asked the same questions, and the score is how often it picks that same wrong option. The evaluation shows advantage compared with several baselines.

Critical strengths

Stemma is clearly ahead of LLMmap, reduced-budget LLMPrint, MPT, and ZeroPrint. The threat model is clear, and the method is easy to adopt.

Critical weaknesses

The closest prior work is ErrorTrace (NeurIPS 2025), not those four baselines. ErrorTrace already uses characteristic mistakes as a provenance signal. The two papers do have differences: Stemma is more fine-grained, but the paper mentions ErrorTrace in one sentence and never compares with it. As written, it is unclear which difference with ErrorTrace leads to the performance gains.

Questions for authors

I am not asking for a full comparison with ErrorTrace on all the datasets, but it would be helpful to do some ablation study against ErrorTrace on some datasets.

1 comment
AuthorsOpenReview ↗

Response to Reviewer dnue

We thank the reviewer for highlighting the connection to ErrorTrace and suggesting an empirical comparison. We agree that ErrorTrace and Stemma share a common starting point in leveraging errors on multiple-choice questions for fingerprint construction. The two methods, however, differ in their detection objectives, provenance signals, and information requirements. We have conducted a systematic methodological and experimental comparison. We summarise the key findings below.

(1) Methodological differences. ErrorTrace primarily targets family-level attribution, whereas Stemma verifies provenance against a specific source checkpoint. Beyond these different objectives, although both methods leverage errors on multiple-choice questions, they use them in fundamentally different ways. ErrorTrace constructs fingerprints from characteristic error spaces (question sets identified through reference-model error patterns) and scores suspects by tracking changes in the gap between their observed and expected error rates as these sets shrink, relying on aggregate error patterns across the sets. In contrast, Stemma assesses induced decision region inheritance through source–suspect agreement on specific answer choices, prioritising erroneous source decisions for their greater specificity and focusing on individual-question-level decision outcomes. Regarding information requirements, ErrorTrace needs multiple reference models from related and unrelated groups for fingerprint construction, demanding greater prior knowledge of and access to reference models, whereas Stemma requires only the source model and one unrelated background model.

(2) Experimental comparison. We adapt ErrorTrace to source-specific verification by treating the seven provenance groups as distinct families and replacing its original datasets with four MMLU supercategories, while preserving its core error-space construction and contraction mechanisms. For efficiency, we precompute answer-choice predictions for all 56 checkpoints using local white-box inference (approximating greedy decoding) and reuse them for both methods.

We evaluate both methods on the same source–suspect pairs, with 4 held-out positive and 24 negative suspects per source (196 pairs across seven sources), across 18 reference-size configurations (k=2,…,7k=2,\ldots,7 groups; n=2,…,4n=2,\ldots,4 models per group) and five random seeds. Evaluation pairs remain fixed across configurations within each seed. We adapt ErrorTrace's gain-based scoring to produce continuous verification scores and report AUROC, standardised pAUC@5% FPR, TPR@1% FPR, and d′d' . Key source-level macro results are summarised below.

MethodknAUROC ↑pAUC@5% FPR ↑TPR@1% FPR ↑d' ↑
Stemma––0.9870.9820.9644.13
ErrorTrace (Best)540.9910.9540.9004.08
ErrorTrace (Median)430.9710.9150.8213.33
ErrorTrace (Worst)220.7340.7200.4432.15

Best, Median, and Worst are selected by macro AUROC among the 17 configurations with successful construction across all targets and seeds, excluding the configuration with ErrorTrace construction failures.

ErrorTrace achieves competitive performance under favourable configurations, marginally exceeding Stemma's near-saturated macro AUROC in only two of the 18 configurations (0.990–0.991 vs. 0.987). However, Stemma consistently achieves stronger low-FPR discrimination across all evaluated configurations, while requiring substantially fewer reference models. Notably, ErrorTrace's best-AUROC configuration requires 20 reference models, involving over 10× as many model–question evaluations for fingerprint construction and approximately 5.8× as many queries for initial verification.

Beyond reference set size, we also found that ErrorTrace's performance is sensitive to the specific reference models selected. To investigate this, we conducted a separate experiment using instruction-tuned sources, fixing k=3k=3, n=3n=3, and the evaluation pairs while varying reference composition. In one Qwen2.5-14B case, selecting less similar rather than highly similar references improved ErrorTrace's AUROC from 0.426 to 1.000. In another Qwen2.5-7B case, AUROC increased from 0.611 to 0.944. In comparison, Stemma achieved AUROCs of 1.000 and 0.989, respectively, on the same evaluation pairs without requiring such reference selection. These examples highlight the practical challenge of selecting suitable references for ErrorTrace fingerprinting construction, which Stemma avoids.

The complete experimental protocols and results are provided in Appendix O, with a corresponding cross-reference added to the Related Work section.

Reviewer_onKMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Stemma, a method for testing whether a language model shares a common origin with a known source model. It selects multiple-choice questions on which the source repeatedly chooses the same wrong answer, strongly prefers that answer over alternatives, and disagrees with an unrelated comparison model. These distinctive mistakes serve as a fingerprint. Stemma then measures how often the model being tested reproduces those answers when the answer options are rearranged.

Critical strengths

I am generally favorable to this paper. The method has a clear motivation. Distinctive mistakes are a sensible source of evidence, as agreement on a commonly known correct answer provides less information about shared origins than repeatedly making the same unique mistake. Although constructing the fingerprint requires access to the source model’s answer scores, testing another model requires only its generated responses, not its internal parameters or scores.

The experiments provide compelling evidence in my view that selecting the right questions helps. Table 15 compares different question-selection procedures while keeping the number of questions and the testing procedure fixed. At the reported benchmark false-positive limit of 1%, Stemma detects 87.1% of related pairs, compared with 12.9% when questions are selected randomly. This supports the usefulness of the selection procedure, rather than attributing the improvement merely to asking multiple-choice questions.

Critical weaknesses

The 1% false-positive rate does not establish how to maintain that rate on new models. Appendix F reports the best detection rate achievable without exceeding 1% false positives on the evaluated pairs. This is a useful benchmark comparison, but it does not show how a user should choose a decision cutoff before testing an unknown model.

Questions for authors

None in particular

2 comments
AuthorsOpenReview ↗

Response to Reviewer onKM

We thank the reviewer for the positive assessment and for highlighting this important practical consideration.

Our work primarily focuses on identifying and constructing robust and discriminative fingerprints for LLM provenance testing. Following common evaluation practices in prior fingerprinting work, we originally evaluated Stemma using ROC-based metrics to assess its discriminative power. While the resulting alignment scores can be combined with different threshold calibration approaches, we recognise that providing explicit binary decision rules would make Stemma more straightforward to use in practice. To address this concern, we have extended Section 4.3 with two binary verification strategies, presented their experimental results in Section 5.4, and provided detailed procedures and sensitivity analyses in Appendix E.

(1) Fixed Threshold. We consider a predefined alignment threshold of 0.5, the midpoint between the idealised extremes of no alignment and perfect alignment. A source–suspect pair is classified as related if its alignment score is at least 0.5, and unrelated otherwise. This provides a lightweight verification option for preliminary screening or when unrelated control models are unavailable for threshold calibration. Although this heuristic does not offer formal statistical guarantees, it achieves high recall: 91.8% with 4.97% FPR on the main benchmark and 97.3% with 5.18% FPR under deployment variations. These results suggest that it can help identify potential provenance relationships without requiring additional calibration overhead, albeit at the cost of a higher false-positive rate.

(2) Statistical Verification. For applications requiring more stringent verification, we introduce a control-based statistical testing procedure inspired by Model Provenance Testing (MPT; Nikolic et al., 2025). Specifically, we compare the source–suspect alignment score against the source–control alignment scores of unrelated control models using one-sided pooled two-proportion Z-tests with Holm–Bonferroni correction. A pair is classified as related only when its alignment is significantly higher than that of every control model. While the nominal significance level does not provide a strict guarantee on provenance FPR due to statistical approximations and dependence on control pool composition, our experiments demonstrate substantially lower false-positive rates. In one evaluated control-selection setting, with 10 controls and a significance level of 0.05, the procedure achieves 84.9% recall with 0.30% FPR on the main benchmark and 90.7% recall with 0.27% FPR under deployment variations. Excluding both the source and suspect provenance groups from the control pool yields FPRs of 1.25% and 1.32%, respectively, highlighting the sensitivity of verification performance to control selection. We further examine sensitivity to control selection, control set size, and significance level in Appendix E.

I thank the authors for providing a fixed threshold analysis in response. It does address my point about practical cutoffs. It does not necessarily guarantee false positive rates on new models, but in my view the paper is still interesting, and I maintain my positive score.

Reviewer_iPgvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies pairwise provenance testing for LLMs: deciding whether a suspect model T is derived from, or shares ancestry with, a source model S. The fingerprint is built using S’s next-token logits, and T is accessed only through generated responses. The authors map open-ended outputs onto a finite decision space (here, multiple-choice answer options), define the resulting partition of the query space as induced decision regions, and argue that related models inherit this region. Stemma selects 40 MMLU probes for stability, robustness and specificity, using a background model. It then scores T by how often T’s decisions agree with S’s across cyclic permutations of the answer options. The experimental result are very impressive on 770 pairs, outperforming prior work.

Critical strengths
  1. Provenance testing is an important and timely problem.

  2. Comparing models at the level of decisions rather than raw outputs is an intuitive idea.

  3. The open-source evaluations are a useful contribution to the community.

Critical weaknesses
  1. Access model and practical setting. The method needs logit access to the source S, while T is accessed only through responses. This is “black-box” only with respect to T. In third-party auditing, logit access to S may not be available. Prior work such as Nikolic et al. (MPT) needs only next-token outputs from both models. The intended deployment setting is not made clear.

  2. Novelty. The core claim, that decision regions are inherited and measurable, amounts to measuring how often S and T give the same answers on a probe set. Agreement-based similarity between the responses of S and T is close in spirit to prior output-agreement methods. The paper should state more precisely what is new beyond the multiple-choice instantiation and the probe-selection heuristics.

  3. This approach seems unlikely to admit FPR guarantees. Because Nikolic et al. sample prompts at random, it is easier to argue a one-sided statistical error bound. The Stemma approach is closer to rejection sampling (maybe can be phrased that way). AUC describes what the score returns across all thresholds, but it is not a test with a bounded FPR.

  4. The validation in Fig. 1 still relies on control models, which seem necessary to establish a threshold. The set used for this analysis should be disjoint from the evaluation set. The paper assumes this set is representative.

  5. The reported baseline AUCs are very different from what prior work reports. For example, a baseline TPR of 0.016. This discrepancy needs to be explained - maybe related to my next point.

  6. Benchmark construction. It is unclear how the target models T were derived and selected, which raises the possibility of benchmark hacking. The paper also does not evaluate on benchmarks from prior work.

Writing Nits

  • "Induced decision regions": "induced" suggests that active changes are made to the model. "Inferred decision regions" may be clearer.
  • sim(·,·) between soft decision vectors is used in Eq. 3 before it is defined.
  • "Transformation-matched pairs", "background model" and "background alignment" are used before they are defined.
  • Fig. 1(a): please define CCDF in the caption.
  • Fig. 2: it is unclear what the (Gold) in Mercury (Gold) means, and the caption does not explain the figure.
Questions for authors
  1. What auditing or deployment setup do you have in mind? Does it realistically provide logit access to S?

  2. Have you considered any measure of statistical significance for your results?

  3. How are the reported FPRs obtained, and how would a threshold be set in practice?

  4. What set was the Fig. 1 analysis run on, and is it disjoint from the evaluation set?

  5. Why do the baseline results differ so much from those reported in the original papers?

  6. How were the target models T derived and selected?

  7. Why not evaluate on prior benchmarks?

  8. What do “transformation-matched pairs”, “background model”, “background alignment” and “(Gold)” in Fig. 2 mean?

4 comments
AuthorsOpenReview ↗

Response to Reviewer iPgv (1/3)

We thank the reviewer for the detailed feedback and constructive suggestions. To address the concerns and questions, we group our responses by topic below, referring to the reviewer's weaknesses as W and questions as Q.

(1) Access Setting (W1, Q1). Although Stemma requires stronger source-side access than MPT (Nikolic et al., 2025), similar asymmetric access settings have been used in prior model fingerprinting and auditing work, including LLMPrint (Hu et al., 2026), RUT (Zhu et al., 2026), TRAP (Gubri et al., 2024), ADV-TRA (Xu et al., 2024), and IPGuard (Cao et al., 2021). A representative scenario involves a model owner or an auditor with access to the source model's logits, either through a publicly available checkpoint or with the model owner's authorisation, constructing a fingerprint to investigate whether a model deployed through a third-party black-box API shares provenance with the source. This setting supports practical applications such as IP protection and licence compliance, where model owners have direct incentives to investigate potential unauthorised reuse, and model supply-chain auditing, where downstream deployers collaborate with model owners to obtain precomputed fingerprints and assess the provenance of models served through third-party black-box APIs. We have further clarified the intended auditing scenario in Section 2.2.

(2) Novelty (W2). We agree that decision agreement itself is not new. In our work, it serves as an observable measure of decision-region preservation, rather than defining the underlying decision-region structure. Our central contribution is to extend the decision-region perspective from deep classifiers, where decision boundaries capture model-specific and lineage-dependent characteristics (Cao et al., 2021; Tang et al., 2025; Xu et al., 2024), to open-ended LLMs, where comparable regions are not naturally defined. By inducing a finite decision space, we enable the study of decision-region inheritance in LLMs, empirically characterise its discriminative properties and transferability (Section 3), and demonstrate its effectiveness for provenance testing through Stemma. Notably, Stemma's multiple-choice formulation and probe-selection strategy represent one practical instantiation of this perspective, identifying stable, robust, and specific source decisions to capture provenance-discriminative region differences.

More broadly, this perspective provides a conceptual bridge between LLM fingerprinting and decision-geometry-based fingerprinting in deep classifiers. This connection highlights the potential for new LLM fingerprint construction strategies that draw on established methods, such as actively constructing queries near induced decision boundaries, inspired by IPGuard (Cao et al., 2021); developing query trajectories that remain informative under boundary shifts, inspired by ADV-TRA (Xu et al., 2024); and exploring queries near intersections of multiple decision boundaries, inspired by IBSF (Bai et al., 2024). We have further highlighted this broader research perspective and its potential directions in Section 6.

(3) Terminology and Presentation (Writing Nits, Q8). We thank the reviewer for these detailed suggestions. We retain induced decision regions because the term reflects our deliberate use of a decision interface to elicit discrete decisions from open-ended LLM behaviour, thereby defining decision regions over the query space (Eq. 2) and enabling us to assess their inheritance without modifying the underlying model.

Following the reviewer's suggestions, we have clarified the remaining terms at their first occurrence. “Transformation-matched pairs” refer to related and unrelated models sharing the same broad transformation type (Section 3.2). The “background model” is an unrelated model used to assess specificity, while “background alignment” measures how often it reproduces the source decision (Section 4.2). We have also defined the similarity function before Eq. 3, expanded CCDFs as complementary cumulative distribution functions in the Figure 1 caption, and clarified that “Gold” denotes the ground-truth answer (Mercury), while “Decision” denotes the source model's selected answer (Earth) in Figure 2.

AuthorsOpenReview ↗

Response to Reviewer iPgv (2/3)

(4) Statistical Evaluation and Verification (W3, Q2–Q3). Our work primarily focuses on constructing robust and discriminative fingerprints for LLM provenance testing. Following common practice in prior fingerprinting work, we originally evaluated Stemma using ROC-based metrics, reporting mean ± standard deviation across five runs. Across both benchmarks, Stemma's performance gains over the baselines are substantially larger than the corresponding across-run standard deviations, providing evidence of consistent performance advantages. In particular, the reported TPR@1% FPR evaluates discriminative performance at a predefined stringent FPR level of 1% on the empirical ROC curve, rather than demonstrating that a fixed verification rule achieves 1% FPR or providing a deployment-time FPR guarantee.

While Stemma's alignment scores can be combined with different threshold calibration approaches, we recognise that explicit binary decision rules would make Stemma more straightforward to use in practice. To address this concern, we have extended Section 4.3 with two binary verification strategies, presented their experimental results in Section 5.4, and provided detailed procedures and sensitivity analyses in Appendix E.

  • Fixed Threshold. We consider a predefined alignment threshold of 0.5, the midpoint between the idealised extremes of no alignment and perfect alignment. A source–suspect pair is classified as related if its alignment score is at least 0.5. This provides a lightweight verification option for preliminary screening or when unrelated control models are unavailable. Although this heuristic does not offer formal statistical guarantees, it achieves 91.8% recall with 4.97% FPR on the main benchmark and 97.3% recall with 5.18% FPR under deployment variations.
  • Statistical Verification. For applications requiring more stringent verification, we introduce a control-based statistical testing procedure inspired by Model Provenance Testing (MPT; Nikolic et al., 2025). Specifically, we compare the source–suspect alignment score against the source–control alignment scores of models unrelated to the source using one-sided pooled two-proportion Z-tests with Holm–Bonferroni correction. A pair is classified as related only when its alignment is significantly higher than that of every control model. In one evaluated control-selection setting, with 10 controls and a significance level of 0.05, the procedure achieves 84.9% recall with 0.30% FPR on the main benchmark and 90.7% recall with 0.27% FPR under deployment variations. Excluding both the source and suspect provenance groups from the control pool yields FPRs of 1.25% and 1.32%, respectively, highlighting the sensitivity of verification performance to control selection. While the nominal significance level does not provide a formal guarantee on provenance FPR due to potential dependence among probe observations, normal-approximation limitations, and sensitivity to control pool composition and coverage, our experiments demonstrate substantially reduced empirical false-positive rates.

(5) Figure 1 Analysis (W4, Q4). Figure 1 is intended as an exploratory analysis to characterise induced decision region inheritance in LLMs and demonstrate the potential of this perspective for provenance testing. We analyse MMLU test questions using three transformation-matched related–unrelated pairs from the Qwen2.5-7B and Qwen3-1.7B lineages, covering instruction tuning, quantisation, and distillation, with the exact checkpoint identities and analysis protocol provided in Appendix B.2. These pairs are used to provide empirical intuition and preliminary evidence for the proposed perspective, rather than being assumed to be statistically representative of all LLM lineages. Here, the unrelated models serve as analytical comparisons, not as controls for calibrating a verification threshold.

Importantly, the relevant distinction is whether information from Figure 1 is used to construct or optimise Stemma, rather than whether the analysed models overlap with the evaluation benchmark. Although models used in Figure 1 are also included in the main benchmark, neither the provenance labels nor the informativeness measurements from Figure 1 are used in Stemma's fingerprint construction or hyperparameter selection. As explained in Section 3.2, informativeness is an analytical measure rather than a construction objective, and Stemma instead selects probes based on source-decision stability, robustness, and specificity. Furthermore, Stemma's ranking hyperparameters were selected using three separate development lineages disjoint from the main benchmark, and the resulting configuration was fixed across all evaluations (Appendix C). Therefore, this overlap does not compromise the independence of Stemma's fingerprint construction, hyperparameter selection, or performance evaluation.

AuthorsOpenReview ↗

Response to Reviewer iPgv (3/3)

(6) Baseline Evaluation Differences (W5, Q5). We believe the differences from previously reported baseline results mainly reflect variations in benchmark composition, evaluation metrics, query interfaces, and, in some cases, implementation configurations. Baseline implementation details and computational adaptations are documented in Appendix H.

Regarding benchmarks, we use our own systematically constructed benchmark, whose design and rationale are detailed in (7).

Regarding metrics, prior studies adopt different evaluation metrics, including binary verification precision and recall (e.g., MPT; Nikolic et al., 2025) and ROC-based measures (e.g., ZeroPrint; Shao et al., 2026). As our focus is on the discriminative power of provenance signals, we evaluate all methods using continuous pairwise provenance scores and ROC-based metrics, avoiding dependence on method-specific threshold calibration and control-set selection. In particular, the cited TPR of 0.016 refers to ZeroPrint's TPR at 1% FPR on the empirical ROC curve for pretrained sources, rather than the recall or measured FPR of a fixed binary verification rule.

Regarding query interfaces, unlike some prior evaluations, our main benchmark uses model-appropriate query interfaces: pretrained sources use raw completion prompts, whereas many downstream suspects use their recommended chat templates, reflecting realistic deployment conditions. As discussed in Section 5.2, this heterogeneity substantially affects baseline performance. To isolate its impact, we additionally evaluated all methods under a uniform raw interface (Appendix J). All baselines achieve higher AUCs in this setting. For example, ZeroPrint improves from 0.623 to 0.797, even exceeding the AUC of 0.720 reported in its original study. More notably, MPT improves from 0.555 to 0.989, surpassing Stemma's AUC of 0.967. Nevertheless, Stemma retains stronger low-FPR discrimination, while demonstrating greater robustness to query-interface heterogeneity in our main evaluation.

(7) Benchmarks (W6, Q6–Q7). Our benchmark was deliberately designed for source-specific LLM provenance testing, emphasising diversity across model families and transformation types, balanced composition across provenance groups and source types, and comparability across source types and evaluation conditions, while maintaining computational efficiency. Prior studies have largely relied on study-specific benchmarks that do not directly align with our evaluation requirements, with no universally adopted benchmark or evaluation protocol in this field. We therefore developed a benchmark tailored to these evaluation objectives, with its model selection criteria, construction protocol, and complete checkpoint list detailed in Section 5.1 and Appendix F.

We adopted a structured, design-driven approach to benchmark construction, defining the target composition before selecting individual checkpoints. Specifically, we established seven provenance groups of eight checkpoints each, comprising one pretrained model, one instruction-tuned model, typically three fine-tuned or adapter-based variants, and one each for merging, quantisation, and distillation where available. This standardised allocation ensures balanced group sizes and comparable transformation coverage, reducing the risk that uneven representation skews aggregate results. We then selected widely used model families represented in prior work, spanning 1.7B–14B parameters, deliberately including three Qwen-based groups across different model sizes and generations to introduce challenging negative pairs. Finally, we populated each predefined transformation category with publicly available Hugging Face checkpoints, prioritising popularity while considering provenance clarity and documentation quality. During evaluation, we systematically evaluated all source–suspect pairs under a uniform protocol, reporting aggregate results in Section 5 and detailed performance breakdowns by provenance group, transformation type, and deployment condition in Appendices I and K.

I appreciate the authors clarifications and thoughtful rebuttal.

I believe the paper requires more careful explanation and examination of the differences in benchmarks, the application scenario (with concrete examples to API access to logits) and the addition of the statistical verification. While I generally liked the paper despite these weaknesses, I believe it is not yet ready so I encourage the authors to improve and maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

We thank the reviewers and Area Chair for their constructive feedback. To facilitate the final assessment, we briefly summarise the key additions and clarifications made during the rebuttal.

  1. Binary Verification Methods (iPgv, onKM). While Stemma primarily focuses on developing and validating discriminative provenance signals that can support different verification strategies, we further introduced two practical binary decision procedures to facilitate deployment: fixed-threshold and control-based statistical verification. The latter achieves 84.9% recall at 0.30% empirical FPR on the main benchmark and 90.7% recall at 0.27% FPR under deployment variations, maintaining strong detection performance (Sections 4.3, 5.4; Appendix E).
  2. Comparison with ErrorTrace (dnue). We clarified the fundamental differences between Stemma and ErrorTrace in their objectives, underlying methodology, and information requirements: Stemma performs source-specific provenance verification through individual decision agreement, whereas ErrorTrace focuses on family-level attribution using aggregate error patterns and requires multiple reference models. Nevertheless, we conducted a systematic empirical comparison across 18 reference configurations and five seeds. In our experiments, Stemma outperforms ErrorTrace in AUROC across most evaluated configurations and consistently achieves stronger low-FPR discrimination, even against ErrorTrace's best-AUROC configuration (TPR@1% FPR: 96.4% vs. 90.0%), while requiring substantially fewer reference models (Appendix O).
  3. Evaluation and Practical Setting (iPgv). We clarified the asymmetric-access auditing scenario, systematic benchmark construction, baseline performance discrepancies, and query-interface effects. We also explained why the exploratory analysis does not compromise the independence of fingerprint construction or evaluation.
  4. Novelty and Research Contributions (iPgv). We clarified that our contribution goes beyond answer agreement by extending decision-region inheritance from conventional classifiers to open-ended LLMs through an induced finite decision space. In Stemma, decision agreement serves as an observable measure of inheritance, while Stemma itself represents just one possible practical instantiation of this broader perspective. This broader perspective provides a foundation for exploring new fingerprinting strategies for LLM provenance.

We hope these additional results and clarifications will be considered in the final assessment. We appreciate the reviewers' and Area Chair's time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.