acceptodds
Under review as a conference paper at ICLR 2027

Reliable LLM Provenance via Panic-Pattern Fingerprinting

Abstract

Training competitive large language models (LLMs) is expensive, making LLMs high-value intellectual property and motivating reliable provenance tests for both open-weight and API-served models. In this paper, we propose a unified provenance framework based on *panic patterns*—model-specific fingerprints elicited by stressful inputs. In the white-box setting, we introduce **P**anic **A**lignment **D**ifference (PAD), which fingerprints models by measuring stress-induced shifts in layerwise representation alignment via a clean-versus-adversarial differential. In the black-box setting, we introduce **O**pen-ended **Q**uestion **T**est (OQT), which uses open-ended questions as stressors and compares residual response embeddings after removing the consensus answer estimated from a reference pool. Across a range of open and proprietary LLMs, our stress-response fingerprints reduce spurious similarity, remain informative under the evaluated model variations, and provide complementary provenance evidence across access levels with bounded verification cost.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Provenance, Fingerprinting

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_NXbeOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks for a provenance signal in how a model reacts under stress. They propose two methods: The whitebox method PAD compares the change in layer-to-layer representation alignment between ordinary inputs and adversarial ones. The blackbox method OQT asks underdetermined open questions, subtracts the answer by some standard reference answers, and compares what is left.

Critical strengths

The black-box method does improve the ranking scores. The authors show that a cutoff chosen on a few vendor lines does not travel.

Critical weaknesses

The black-box result is a ordering signal, not a reliable test of top-1. However I think in applications, people will care more about blackbox setting & top-1 result.

The white-box claim is not supported by a full comparison. Logits, clean representation matching, and the same matching on adversarial inputs appear for two victim models. The numbers that cover every family are PAD alone. The paper’s own text says the single-victim table does not show that PAD is generally better. A reader therefore cannot tell whether the clean-versus-adversarial difference is doing the work, or whether ordinary representation matching would have ranked these families just as well.

Questions for authors

Is the black-box cutoff still useful on a family that contains a real derivative, when that family was not used to choose the cutoff?

1 comment
AuthorsOpenReview ↗

Full comparisons and ablations, clean-only vs PAD to strengthen PAD for PAD, and cutoff transfer for OQT

Q1. Full-cohort comparisons and ablations

We added full-cohort comparisons with REEF and Adv-REEF variants and a separate matched component ablation. All comparisons use the same 34 checkpoints, nine lineages, 561 pairs and 200 probe pairs. To include the six models with fewer than 18 layers, we use two layer-selection rules fixed before scoring: relative depth and the final layer. REEF uses clean inputs, whereas Adv-REEF uses adversarial inputs.

Table R4.1 reports PAD with SSIM matcher. At empirical FPR≤1%, PAD recovers 69/79 positives versus at most 52/79 for the four REEF/Adv-REEF variants; PAD and the highest-recall baseline each incur four false positives among 482 negatives. The Frobenius matcher gives TP/FP counts of 46/4 at the 1% FPR cap and 58/22 at the 5% cap.

Table R4.1. Full-cohort comparison with REEF and Adv-REEF variants.

MethodAUROCAPTP at FPR≤1%FP at FPR≤1%TP at FPR≤5%FP at FPR≤5%
REEF (relative depth)0.97730.90684337224
Adv-REEF (relative depth)0.98890.94395247524
REEF (final layer)0.92070.78864735321
Adv-REEF (final layer)0.96150.87064836223
PAD0.97040.95056947417

Q2. Clean-only versus differential fingerprints

The matched ablation shows that subtraction improves low-false-positive detection. We compared the clean–clean alignment matrix CC with the differential fingerprint D=C−AD=C-A, where AA is the cross-condition alignment matrix, using the same 34 models, 200 probe pairs, and model-pair labels. At empirical FPR≤5%, Frobenius yields numerically similar TP/FP counts: 59/79 true positives and 23/482 false positives for CC. With SSIM, subtraction increases true positives from 70/79 to 74/79 while reducing false positives from 21/482 to 17/482, giving a net 5.06-percentage-point recall gain. Thus, under the same false-positive-rate cap, the differential fingerprint achieves both higher recall and fewer false positives than clean-only with SSIM.

Table R4.2. Clean-only versus differential fingerprints with SSIM on all 561 pairs.

MatrixAUROCAPTP at FPR≤5%FP at FPR≤5%
C: clean–clean0.97650.928370/7921/482
PAD0.97040.950574/7917/482

Q3. Cutoff transfer

Thank you for the focused question; we have made the cutoff a separate experiment. The cutoff does transfer to a family with a real derivative that was not used to choose it. One family is held out at a time: the threshold is set on the unrelated pairs of the other four, and the held-out pairs are scored unchanged:

Table R4.3. Counts on the held-out family, with the threshold fixed on the other four.

Held-out familyPositivesThresholdFoundFP unrelatedFP co-release
llama-3.1-8b60.48431/1003/24
llama-3.1-70b30.49020/756/21
deepseek-v310.45414/620/4
qwen3-30b-a3b10.48101/561/10
qwen3-next-80b-a3b10.48600/560/10
Pooled12—6 (recall 0.500)6/349 (1.72%)10/69 (14.5%)

Counts are pair evaluations summed over the five folds, not distinct pairs: 53 distinct co-release and 299 distinct unrelated pairs are ever scored.

Choice of rule. What it recovers then follows from the rate the verifier will accept: 1.5% gives half of the derivatives, and 5% or 10% gives the same half while accusing 20 and 28 unrelated pairs instead of 6. We report a tolerance because it is the quantity a verifier sets, and because recall stops increasing at 1.5%, so a looser setting buys accusations rather than derivatives. The full argument is in our answer to Reviewer PE1F.

Top-1. We do not claim a single-name verdict, and 12 positives could not support one. The resolution the score does deliver is a shortlist: the true ancestor lands in the top three for seven of the twelve pairs, and its mean rank is 5.7 of the 34 candidates where an arbitrary order gives 17.5. A verifier works through a handful of candidates instead of the whole pool, which is what the top-3 count and the mean rank report.

Planned revision: the counts above appear beside the ablation, and the score is read as a screening and ranking signal rather than a decision rule.

Reviewer_TWmWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes "panic patterns" as a provenance fingerprint for deciding whether a suspect LLM was derived from a reference model. The idea is that a model's reaction to stressful or underdetermined inputs is shared within a model family but differs across unrelated families. The authors give one method for each access level. In the white-box setting, PAD measures how a model's internal layer-to-layer representation structure changes between clean and adversarially perturbed inputs, and compares that change across models. In the black-box setting, OQT asks open-ended questions, removes the answer that most models give (estimated from a reference pool), and compares what remains of two models' responses. PAD separates related from unrelated checkpoints well on a 34-model cohort (AUROC 0.94) and holds up under fine-tuning, pruning and merging. OQT improves over comparing raw responses directly but falls below the existing black-box method LLMmap. The authors present the two methods as complementary evidence across access levels.

Critical strengths

The idea and motivation are strong. Provenance for derived and API-served models is an important problem, and using a model's characteristic reaction to stressful or underdetermined inputs as a family signature is intuitive. The observation that open-ended answers are dominated by a shared consensus, and that removing it exposes family-specific signal, is a useful insight.

Critical weaknesses
  • Missing baselines: The paper motivates provenance for API-served models, so the black-box method carries its practical claim. There, OQT underperforms LLMmap on the paper's own comparison while using more queries, and the authors acknowledge this (Table 4). Closely related probe-based fingerprints cited in Sec 2 are not compared. The white-box method cannot be used on API models and is compared only against REEF variants, not the weight-based fingerprints the paper itself cites (HuRef, SLIP, ZEST).
  • Not rigorous enough to support "reliable": The black-box lineage result rests on a small number of positive pairs with no confidence intervals, and the calibrated threshold does not transfer to held-out vendor lines (Table 12). The adaptive attack on PAD uses a single model and seed. PAD's perturbations act at the tokenizer, and related models share tokenizers while unrelated ones do not, so without a tokenizer-matched control the results do not separate weight provenance from tokenizer identity.
Questions for authors

No further questions until the baselines are meaningfully filled. See weaknesses.

2 comments
AuthorsOpenReview ↗

Baseline explanation

Q1. Baseline comparisons

We extend the comparison on both access levels. On the weight side, the original manuscript includes the output-space Logits baseline alongside REEF and Adv-REEF in the MPT and Llama-2 reference evaluations, and we add two independent weight fingerprints, HuRef-ICS and GhostSpec-MSE, with comparisons on fixed common subsets. Each column of Table R3.1 uses identical checkpoints and ancestry labels and includes both PAD matchers. On the 25-model, 300-pair subset, PAD-SSIM obtains AUROC/AP of 0.9651/0.9600 versus GhostSpec-MSE's 0.9433/0.9110. On the eight-model HuRef subset, HuRef-ICS obtains 1.0000/1.0000 and PAD-SSIM 0.9896/0.9931. These results extend the comparison beyond representation-based REEF.

Table R3.1. Independent fingerprint baselines and controlled REEF comparisons. Each cell is AUROC / AP.

Method28 models / 378 pairs25 models / 300 pairs24 models / 276 pairs8 models / 28 pairs
PAD-F / fixed sigmoid0.8799 / 0.81650.8575 / 0.82890.8547 / 0.83080.8281 / 0.9103
PAD-SSIM0.9709 / 0.95970.9651 / 0.96000.9652 / 0.96120.9896 / 0.9931
REEF-clean0.9888 / 0.9623—0.9887 / 0.97081.0000 / 1.0000
Adv-REEF0.9981 / 0.9929—0.9991 / 0.99761.0000 / 1.0000
GhostSpec-MSE—0.9433 / 0.91100.9494 / 0.92491.0000 / 1.0000
HuRef-ICS———1.0000 / 1.0000

On the behavioral side, the two probe-based fingerprints cited in Section 2 are reproduced from their authors' released code: our API client replaces theirs, QueRE reads the yes/no answers from text, and Hide and Seek bounds the token lengths. Neither separates our lineage pairs. Table R3.2 places both on the same low-false-positive scale as the rest of the comparison, and Table R3.3 compares QueRE with OQT on the 435 pairs they share.

Table R3.2. Low false-positive comparison on the shared collection. Each method is run on all the pairs it covers.

MethodCoverageAUROCTPR at 1% FPRTPR at 5% FPRTPR at 10% FPRpAUC below 10%
OQT (ours)35 models, 595 pairs0.7220.0900.2180.3330.218
LLMmap35 models, 595 pairs0.8550.0130.0900.1670.098
Nikolić et al.35 models, 595 pairs0.7090.0770.1790.3330.209
QueRE32 models, 496 pairs, 9 lineage0.5370.0000.0000.1110.029

Table R3.3. QueRE and OQT on the 435 pairs both cover.

QueRE (32 models, 9 of 12 lineage pairs)QueREOQT, same pairs
AUROC, lineage (435 pairs both cover)0.540 [0.399, 0.680]0.867 [0.753, 0.965]
AUROC, same-vendor-line positives0.488 [0.416, 0.561]0.734 [0.649, 0.813]
Discriminator accuracy, lineage vs unrelated0.844 vs 0.847—

Brackets are 95% intervals from a bootstrap over the pairs, resampling lineage positives and negatives separately.

Q2. Baseline selection and coverage

We prioritized complementary baselines with checkpoint-pair scores that can be evaluated directly under our provenance labels, adding HuRef-ICS and GhostSpec-MSE to the existing REEF, Adv-REEF and Logits comparisons.

Among the weight-based methods, the remaining cited ones differ in their evaluation requirements. SLIP is a weight-partitioning protocol for protecting model IP during hybrid inference, and it does not define a retrospective provenance score between two released checkpoints. Zest provides LIME-based behavioral distances and reference-based normalization, and its paper includes text classification. Applying it to our generative LLM cohort requires specifying and validating a common text-perturbation and output-scoring protocol.

REEF already evaluates PCS, ICS and Logits in Section 5.2.1 and Table 1. We retain REEF as an established representation baseline and prioritize complementary weight fingerprints over repeating every earlier comparator, so some baselines (pcs etc.) are not separately rerun here. We nevertheless add HuRef-ICS directly, alongside GhostSpec-MSE, to address the requested weight-based comparisons.

The same criterion applies to the black-box side. QueRE and Hide and Seek both ship code and run on stored responses, so both are reproduced above. QueRE's discriminator yields a pair score and enters the AUC table. Its preferred read-out needs token log-probabilities, which our providers do not return for the lineage models, so we read it from its text branch. That is a limitation of the method in the API-served setting rather than of the substitution.

Hide and Seek's released loop is given five models, two of them related, and must name those two. It named them in 5 of the 39 answers it produced, where a guess succeeds once in ten. To check that the setup can work at all, one group contained the same model twice, and it named that pair in all four attempts; the misses are the method's, not the setup's.

AuthorsOpenReview ↗

Tokenzier, depth, adaptive attack for PAD, and Intervals, budget, threshold for OQT

Q3. Tokenizer and depth controls

We tested eight models in two groups:

  • MPT group: MPT-7B (reference), MPT-7B-Instruct and MPT-7B-Chat (derivatives), and independently trained Pythia-6.9B (control).
  • Llama group: Llama-2-7B (reference), Vicuna-7B-v1.5 and WizardMath-7B-v1.0 (derivatives), and independently trained Llama-1-7B (control).

All eight models have 32 layers and hidden width 4,096. Within each group, the same 400 clean/perturbed inputs produce identical token IDs, including special tokens. We use the primary Frobenius matcher with the same extraction and interpolation, followed by the same fixed sigmoid mapping; higher scores indicate greater similarity. For each reference, both derivatives score higher than the independent control (Table R3.4). Thus, shared tokenization and layer count alone cannot explain the observed PAD rankings.

Table R3.4. Tokenizer- and depth-matched independent controls.

ReferenceCandidateRelationshipPAD score ↑
MPT-7BPythia-6.9BIndependent0.250974
MPT-7BMPT-InstructDerived0.955746
MPT-7BMPT-ChatDerived0.926360
Llama-2-7BLlama-1-7BIndependent0.010636
Llama-2-7BVicuna-1.5Derived0.871485
Llama-2-7BWizardMath-1.0Derived0.925924

Q4. Adaptive attacks across models and seeds

We evaluated a PAD-targeted fine-tuning attack on Llama-2-7B, Qwen2.5-0.5B and Llama-3.2-1B-Instruct, with seeds 42, 2026 and 7. Under the configuration specified below, both PAD-F and PAD-SSIM detect all nine attacked checkpoints after 200 updates, giving a utility-constrained evasion rate of 0/9 for each detector. All nine also satisfy the same capability-preservation criterion. Table R3.5 reports the full score range across all three seeds.

We fine-tuned suspect models with a PAD-targeted objective, combined with language-modeling and KL regularization to preserve utility. Evasion requires initial detectability, a post-attack score below the fixed detection threshold, and no more than a two-percentage-point accuracy drop from the unmodified base on each benchmark. Every endpoint uses the same 1,140 MMLU and 1,172 ARC-Challenge questions under our custom zero-shot letter-likelihood protocol; utility preservation is measured after training.

Threshold selection. We select the most permissive cutoff satisfying a 5% empirical FPR cap using only 141 negative pairs from six non-attacked calibration lineages; all three attacked lineages are excluded from threshold fitting. Both cutoffs yield 7/141 calibration false positives (4.96%). The resulting detection rules are *PAD-F sigmoid score ≥0.23 and PAD-SSIM score ≥0.93.

Table R3.5. Adaptive-attack results at 200 updates, three seeds per model.

ModelPAD-F sigmoid score range ↑PAD-SSIM score range ↑Utility criterion metEvasion ASR: F / S ↓
Llama-2-7B0.7052–0.81920.9918–0.9954Yes (3/3)0/3 / 0/3
Qwen2.5-0.5B0.9356–0.94950.9849–0.9898Yes (3/3)0/3 / 0/3
Llama-3.2-1B-Instruct0.3819–0.89200.9727–0.9911Yes (3/3)0/3 / 0/3

Q5. Intervals, budget, threshold

You are right that the black-box result has to survive the three checks under "Not rigorous enough": the size of the positive set, the query budget, and the transfer of the cutoff. None of the three is answered by collecting more positives: each is settled on the responses we already have.

Twelve positives. We agree that they are few, and we rest no claim on any one of them. The line-held-out AUC of 0.850 has a 95% bootstrap interval from 0.750 to 0.934, so the cohort bounds the estimate without settling it. Dropping each lineage pair in turn also leaves the residual AUC between 0.827 and 0.868, against 0.613 to 0.686 for absolute similarity — the gap of 0.20 is nearly three times the wider of the two spreads.

Query budget. The paper states the three budgets explicitly: OQT collects 750 responses per model, LLMmap's released protocol spends 88 short completions per model, and QueRE's protocol needs twelve requests per question. OQT therefore costs more than LLMmap and less than QueRE, and what its budget buys is reported as a curve: 50 responses per model already reach AUROC 0.806 ± 0.054, and the full set reaches 0.840.

Threshold transfer. We hold out a whole checkpoint-sharing family, choose the threshold that a 1.5% false-accusation rate implies on the other four, and apply it to the held-out pairs unchanged. The five folds recover 6 of the 12 derivatives, flagging 1.72% of the held-out unrelated pairs and 14.5% of the co-releases. The cutoff therefore travels; what it recovers then follows from the rate a verifier accepts.

Planned revision: the two reproductions, the query-budget curve and the transfer statistics sit next to the results they support, and the baseline discussion says which methods were run and why.

Reviewer_PE1FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies LLM provenance, aiming to identify whether models share a common weight origin. It proposes two fingerprinting methods based on model responses to stressful or underdetermined inputs. In the white-box setting, Panic Alignment Difference (PAD) measures changes in layerwise representation alignment between clean and adversarial conditions. In the black-box setting, Open-ended Question Test (OQT) compares response embeddings after removing the consensus estimated from a reference model pool. Experiments cover 34 checkpoints from nine families for PAD and 35 models from 12 vendor lines for OQT. The results show that PAD retains family-level signals after several model modifications, while OQT improves lineage ranking over direct response similarity.

Critical strengths

The paper addresses an important practical problem and explores model provenance under both white-box and black-box access. The methods are intuitive and extract fingerprints from existing models without requiring watermarks to be inserted before release. A particular strength is OQT’s consensus-removal step, which improves lineage AUROC from 0.644 to 0.840 using the same collected responses. PAD also shows promising stability under fine-tuning, pruning, and merging. The authors clearly report several limitations, including difficulties with threshold transfer, which helps readers understand the scope of the results.

Critical weaknesses

The PAD results suggest that the fingerprints contain information about model lineage. However, the paper should better explain why the clean–adversarial difference is effective. It remains unclear how much of the improvement comes from subtraction itself, and whether the fingerprints reflect model lineage or differences in tokenization and overall sensitivity to perturbations.

Although OQT improves AUROC from 0.644 to 0.840, Top-1 retrieval remains unchanged at 3/12. Threshold transfer also achieves only 5.7% precision on the co-release task. These results support candidate ranking, but do not establish reliable provenance decisions for unseen model families.

The controlled PAD attack experiment covers only one base model, one random seed, and 200 training steps with rank-8 LoRA. Model utility is evaluated using a small diagnostic set of 64 questions. The attack protocol also lacks an independently calibrated decision threshold, so changes in similarity alone cannot show how often attacks successfully evade detection. This limits conclusions about robustness against stronger attacks and across different model families.

Questions for authors

Q1. Could the authors compare the differential fingerprint with its individual alignment matrices, REEF, and Adv-REEF on the complete cohort under matched evaluation conditions? What controls would distinguish lineage information from tokenizer effects or general perturbation sensitivity?

Q2. Can the authors evaluate fixed-reference OQT on an independently held-out cohort containing checkpoint-sharing positive pairs from unseen families, using a threshold selected exclusively on calibration families, and report precision, recall, and false-positive rates?

Q3. How does PAD’s evasion success rate change across additional base models, seeds, and adaptation budgets when success is defined using an independently calibrated threshold and attacks are compared under matched utility constraints?

2 comments
AuthorsOpenReview ↗

Q1. Full-cohort comparisons, tokenizer and sensitivity for PAD

Q1. Full-cohort comparisons, tokenizer and sensitivity

Q1.1. Full-cohort comparisons and ablations

We added full-cohort comparisons with REEF and Adv-REEF variants and a separate matched component ablation. All comparisons use the same 34 checkpoints, nine lineages, 561 pairs and 200 probe pairs. To include the six models with fewer than 18 layers, we use two layer-selection rules fixed before scoring: relative depth and the final layer. REEF uses clean inputs, whereas Adv-REEF uses adversarial inputs.

Table R2.1 reports PAD with SSIM matcher. At empirical FPR≤1%, PAD recovers 69/79 positives versus at most 52/79 for the four REEF/Adv-REEF variants; PAD and the highest-recall baseline each incur four false positives among 482 negatives. The Frobenius matcher gives TP/FP counts of 46/4 at the 1% FPR cap and 58/22 at the 5% cap.

(Optional, details please refer to our response to reviewer ndF7 Q3) The component ablation in Table R2.2 further isolates subtraction: with SSIM at FPR≤5%, PAD gives 74/79 TP and 17/482 FP, versus 70/79 and 21/482 for CC, showing higher recall with fewer false positives at this operating point.

Table R2.1. Full-cohort comparison with REEF and Adv-REEF variants.

MethodAUROCAPTP at FPR≤1%FP at FPR≤1%TP at FPR≤5%FP at FPR≤5%
REEF (relative depth)0.97730.90684337224
Adv-REEF (relative depth)0.98890.94395247524
REEF (final layer)0.92070.78864735321
Adv-REEF (final layer)0.96150.87064836223
PAD0.97040.95056947417

(Optional) Table R2.2. Clean-only versus differential fingerprints with SSIM on all 561 pairs.

MatrixAUROCAPTP at FPR≤5%FP at FPR≤5%
C: clean–clean0.97650.928370/7921/482
PAD0.97040.950574/7917/482

Q1.2. Tokenizer and sensitivity

We tested both explanations directly.

Tokenizer:

  • MPT group: MPT-7B (reference), MPT-7B-Instruct and MPT-7B-Chat (derivatives), and independently trained Pythia-6.9B (control).
  • Llama group: Llama-2-7B (reference), Vicuna-7B-v1.5 and WizardMath-7B-v1.0 (derivatives), and independently trained Llama-1-7B (control).

All eight models have 32 layers and hidden width 4,096. Within each group, the same 400 clean/perturbed inputs produce identical token IDs, including special tokens. We use the primary Frobenius matcher with the same extraction and interpolation, followed by the fixed sigmoid mapping of Reviewer ndF7 Q1; higher scores indicate greater similarity. For each reference, both derivatives score higher than the independent control (Table R2.3). Thus, shared tokenization and layer count alone cannot explain the observed PAD rankings.

Table R2.3. Matched independent controls.

ReferenceCandidateRelationshipPAD score (sigmoid) ↑
MPT-7BPythia-6.9BIndependent0.250974
MPT-7BMPT-InstructDerived0.955746
MPT-7BMPT-ChatDerived0.926360
Llama-2-7BLlama-1-7BIndependent0.010636
Llama-2-7BVicuna-1.5Derived0.871485
Llama-2-7BWizardMath-1.0Derived0.925924

Sensitivity:

We test whether a single measure of perturbation response explains the lineage signal. Mean-only keeps its signed average; RMS-only keeps its overall magnitude, avoiding cancellation between positive and negative entries. Both replace the matrix with one number and score model pairs by the negative absolute difference of those numbers, discarding which layer pairs change. The unit-norm diagnostic divides each DD by its Frobenius norm and compares the normalized matrices with negative Frobenius distance. It removes overall scale while retaining the relative layerwise pattern.

Table R2.4. Aggregate sensitivity versus differential shape, 34 models / 561 pairs.

DiagnosticInformation retainedAUROCAP
Mean of D onlySigned average change0.68210.3448
RMS of D onlyOverall change magnitude0.72230.3793
Unit-norm differential shapeRelative layerwise pattern without overall scale0.96920.9007

Mean-only and RMS-only yield AUROCs of 0.6821 and 0.7223, whereas the unit-norm comparison yields 0.9692. Thus, the relative layerwise pattern retains lineage-ranking information that these two scalar summaries do not capture.

AuthorsOpenReview ↗

Q2.held-out-family threshold for OQT and Q3. adaptive attack for PAD

Q2. Held-out-family threshold

Thank you for this question. We ran exactly the protocol you describe: a whole checkpoint-sharing family is held out at a time, the threshold is the 1.5% quantile of the calibration unrelated pairs from the other four families, and the held-out family's pairs are then scored unchanged. We report the two halves separately, because they answer different quantities: the ordering, which should not need the held-out family, and the count, which should. Table R2.5. Ordering: the ablation, with and without the held-out family.

ScoreLineage AUCTop-1Top-3Median rank
Absolute response similarity0.6443/126/128.5 of 34
Residual, fixed reference0.8403/127/122.5 of 34
Residual, line held out0.8504/125/125.5 of 34
Subtracting a reference is what makes the score work: the lineage AUC rises from 0.644 to 0.840. The two ways of subtracting it are a tie, and we do not rank them: their AUCs differ by 0.010, and each is better on a different per-suspect number. The fixed reference puts 7 of 12 partners in the top three at a median rank of 2.5; the line-held-out diagnostic puts 5 of 12 at 5.5. We use the fixed reference as the main score and the line-held-out variant where a verifier must keep the suspect's own vendor line out of the reference.
Table R2.6. Counts on the held-out family, with the threshold fixed on the other four.
Held-out familyPositivesThresholdFoundFP unrelated
---------------
llama-3.1-8b60.48431/100
llama-3.1-70b30.49020/75
deepseek-v310.45414/62
qwen3-30b-a3b10.48101/56
qwen3-next-80b-a3b10.48600/56
Pooled12—6 (recall 0.500)6/349 (1.72%)

Counts are pair evaluations summed over the five folds, not distinct pairs: 53 distinct co-release and 299 distinct unrelated pairs are ever scored. At a 1.5% false-accusation tolerance the five folds recover 6 of the 12 derivatives, accusing 1.72% of the held-out unrelated pairs and 14.5% of the co-releases. We report that rate rather than a threshold rule, because recall stops paying above it: 5% and 10% recover the same six while accusing 20 and 28 unrelated pairs instead of 6.

The 5.7% precision you quote belongs to that threshold rather than to the score: it was fitted on pairs from three vendor lines and then applied to co-releases, whose scores overlap with unrelated pairs. The operating points above are the same comparison on the full pool.

The revised manuscript will report these operating points in full, with the transfer and budget statistics.

Planned revision: the score is presented as screening and ranking evidence rather than a decision rule, the ablation and the leave-one-family-out table are reported next to it, and the decision boundary is discussed in terms of the tolerance a verifier sets.

Q3. Adaptive attacks across models and seeds

We evaluated a PAD-targeted fine-tuning attack on Llama-2-7B, Qwen2.5-0.5B and Llama-3.2-1B-Instruct, with seeds 42, 2026 and 7. Under the configuration specified below, both PAD-F and PAD-SSIM detect all nine attacked checkpoints after 200 updates, giving a utility-constrained evasion rate of 0/9 for each detector. All nine also satisfy the same capability-preservation criterion. Table R2.7 reports the full score range across all three seeds.

We fine-tuned suspect models with a PAD-targeted objective, combined with language-modeling and KL regularization to preserve utility. We use rank-8 q/v LoRA and learning rate 2×10−52\times10^{-5}. Evasion requires initial detectability, a post-attack score below the fixed detection threshold, and no more than a two-percentage-point accuracy drop from the unmodified base on each benchmark. Every endpoint uses the same 1,140 MMLU and 1,172 ARC-Challenge questions under our custom zero-shot letter-likelihood protocol; utility preservation is measured after training.

Threshold selection. We select the most permissive cutoff satisfying a 5% empirical FPR cap using only 141 negative pairs from six non-attacked calibration lineages; all three attacked lineages are excluded from threshold fitting. Both cutoffs yield 7/141 calibration false positives (4.96%). The resulting detection rules are PAD-F sigmoid score ≥0.231766 and PAD-SSIM score ≥0.931962.

Table R2.7. Adaptive-attack results at 200 updates, three seeds per model.

ModelPAD-F sigmoid score range ↑PAD-SSIM score range ↑Utility criterion metEvasion ASR: F / S ↓
Llama-2-7B0.7052–0.81920.9918–0.9954Yes (3/3)0/3 / 0/3
Qwen2.5-0.5B0.9356–0.94950.9849–0.9898Yes (3/3)0/3 / 0/3
Llama-3.2-1B-Instruct0.3819–0.89200.9727–0.9911Yes (3/3)0/3 / 0/3
Reviewer_ndF7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies LLM provenance, meaning whether a suspect model was derived from a reference model or trained independently. It proposes two fingerprints based on how a model reacts to stressful inputs. In the white-box setting, Panic Alignment Difference (PAD) uses CKA to compare every pair of layers, once on clean prompts and once between perturbed and clean prompts. The fingerprint is the difference of the two matrices, meant to remove structure all models share. In the black-box setting, the Open-ended Question Test (OQT) embeds answers to open-ended questions and subtracts the reference pool's average answer embedding (the consensus) before comparing models.

On 34 checkpoints from nine families, the paper reports an AUROC of 0.94 for PAD. For OQT on 35 models, it reports a rise in lineage AUROC from 0.64 to 0.84 when the consensus is removed.

Critical strengths

The paper studies an important problem. Open-weight models are routinely fine-tuned, merged and re-released, so checking shared origin without a pre-release watermark matters in practice.

The strongest part is the OQT consensus-removal result, because it is cleanly isolated. Similarity with and without consensus removal comes from the same responses, so only the scoring can explain the gain. The appendix shows that the score after removal is well above chance. I also appreciate the paper's honesty about its limits, such as the poor transfer of the OQT threshold and LLMmap's higher AUROC. Still, Figure 4 suggests OQT leads LLMmap at low false-positive rates, which would be worth reporting directly.

Critical weaknesses

However, I do not find the evidence sufficient for the paper's claims about PAD. Since the paper itself presents OQT as a ranking signal, any reliability claim has to rest on PAD.

My main concern is that PAD's AUROC and AP in Table 6 do not match Figure 5. The appendix says Figures 5 and 6 show the score matrices underlying Table 6. Reading Figure 6's scores from its colorbar, I get Table 6's SSIM AUROC of 0.9704. The same reading of Figure 5 gives an AUROC of about 0.88 and an AP of about 0.78 for the primary Frobenius matcher, not the reported 0.94 and 0.92. These metrics depend only on how pairs are ordered, so a strictly monotone mapping to the bounded score cannot explain the gap. The paper never defines that mapping. Both figures still show clear family structure, so PAD does separate the families in this cohort. But the reported cohort metrics are the paper's main evidence for it, and they may overstate its strength.

My second concern is that the cohort cannot tell shared weights apart from a shared tokenizer and layer count. Every within-family pair shares a tokenizer, apart from added tokens. Only 20 of the 482 cross-family pairs do, and none of those 20 matches in layer count. So a rule that ranks pairs by a shared tokenizer, with a shared layer count as a tie-break, already reaches an AUROC above 0.99 on this cohort, higher than PAD's reported 0.94. This matters because PAD's stressor is a character-level perturbation, so checkpoints with the same tokenizer see the same change in input tokens. As written, the paper shows that PAD separates models that differ in tokenizer or layer count, but does not convincingly demonstrate that it detects shared weights. Independently trained models with a reference's tokenizer and layer count, such as Llama-1-7B for Llama-2-7B [1] or Pythia-6.9B for MPT-7B, would test this directly, and PAD may well pass.

Overall, I think the idea is interesting, and the consensus-removal result is useful. But the current evidence shows scores that rank related models higher, not reliable provenance. I would raise my score if the released distances reproduce the reported metrics, PAD ranks independent models with a reference's tokenizer and layer count below that reference's derivatives, and PAD beats its clean matrices alone under the same matcher. The conclusion should also narrow its adaptive-attack claim, unless the model stays detectable after the longer run I ask for below.

References:

[1] Zhu et al. (2025). Independence Tests for Language Models.

Questions for authors
  1. Can the authors release the distance matrix behind Figure 5 and define the mapping to the bounded score? Showing that the released distances reproduce Table 6 with the stated matcher would resolve my main concern.

  2. Could the authors add a few independently trained models with a reference’s tokenizer and layer count, and report their scores against that reference next to the scores of its derivatives? If PAD keeps them clearly below the derivatives, my second concern would be resolved.

  3. Could the authors compare PAD with its clean matrices alone, under the same interpolation and matcher? The paper’s REEF and Adv-REEF baselines compare hidden representations across the two models, so they change more than the clean-adversarial subtraction they are meant to isolate. A clear margin over the clean matrices would support the paper’s claim about the subtraction.

2 comments
AuthorsOpenReview ↗

Main Questions for PAD (correct a mistake, tokenizer, depth and clean-only vs PAD)

Q1. Numerical consistency and score mapping

Thank you for identifying this inconsistency. We independently recomputed all 561 model pairs and confirmed that the Frobenius AUROC, AP and Top-1 entries were reported incorrectly. The corrected AUROC, AP and Top-1 are 0.8764, 0.7840 and 20/25; macro AUROC remains 0.9256. The raw scores, original computation records and Figures 5–6 agree with these values; the original SSIM row is also reproduced. We apologize for the reporting error and will correct the corresponding main-text statement together with Table 6. The bounded Frobenius score is defined below; its monotonicity preserves the ranking of negative distances.

d &= \|\widetilde{\Delta}_f-\widetilde{\Delta}_g\|_F,\\ s_F &= \frac{1}{1+\exp(3(d-1.5))}=\sigma(4.5-3d). \end{aligned}$$ **This fixed sigmoid expresses distance on a bounded similarity scale for easier reporting and comparison. PAD's core contribution is the differential fingerprint constructed from clean and perturbed representation alignments.** If you are still instersted in distance matrix behind figure 5, we can open source the values immediately. However, the table is too large and needs to be sent separately for comments. **Table R1.1. Corrected complete-cohort results.** | Matcher | AUROC | AP | Macro AUROC | Top-1 | | --- | --- | --- | --- | --- | | Frobenius | 0.8764 | 0.7840 | 0.9256 | 20/25 | | SSIM | 0.9704 | 0.9505 | 0.9342 | 24/25 | # Q2. Tokenizer and depth controls We tested eight models in two groups: - **MPT group:** MPT-7B (reference), MPT-7B-Instruct and MPT-7B-Chat (derivatives), and independently trained Pythia-6.9B (control). - **Llama group:** Llama-2-7B (reference), Vicuna-7B-v1.5 and WizardMath-7B-v1.0 (derivatives), and independently trained Llama-1-7B (control). All eight models have 32 layers and hidden width 4,096. Within each group, the same 400 clean/perturbed inputs produce identical token IDs, including special tokens. We use the primary Frobenius matcher with the same extraction and interpolation, followed by the fixed sigmoid mapping of Q1; higher scores indicate greater similarity. **For each reference, both derivatives score higher than the independent control** (Table R1.2). Thus, shared tokenization and layer count alone cannot explain the observed PAD rankings. **Table R1.2. Tokenizer- and depth-matched independent controls.** | Reference | Candidate | Relationship | PAD score ↑ | | --- | --- | --- | --- | | MPT-7B | Pythia-6.9B | Independent | 0.250974 | | MPT-7B | MPT-Instruct | Derived | 0.955746 | | MPT-7B | MPT-Chat | Derived | 0.926360 | | Llama-2-7B | Llama-1-7B | Independent | 0.010636 | | Llama-2-7B | Vicuna-1.5 | Derived | 0.871485 | | Llama-2-7B | WizardMath-1.0 | Derived | 0.925924 | # Q3. Clean-only versus differential fingerprints The matched ablation shows that subtraction improves low-false-positive detection. We compared the clean–clean alignment matrix $C$ with the differential fingerprint $D=C-A$, where $A$ is the cross-condition alignment matrix, using the same 34 models, 200 probe pairs, and model-pair labels. At empirical FPR≤5%, Frobenius yields numerically similar TP/FP counts: 59/79 true positives and 23/482 false positives for $C$. With SSIM, subtraction increases true positives from 70/79 to 74/79 while reducing false positives from 21/482 to 17/482, giving a net 5.06-percentage-point recall gain. Thus, under the same false-positive-rate cap, the differential fingerprint achieves both higher recall and fewer false positives than clean-only with SSIM. **Table R1.3. Clean-only versus differential fingerprints with SSIM on all 561 pairs.** | Matrix | AUROC | AP | TP at FPR≤5% | FP at FPR≤5% | | --- | --- | --- | --- | --- | | C: clean–clean | 0.9765 | 0.9283 | 70/79 | 21/482 | | PAD | 0.9704 | 0.9505 | 74/79 | 17/482 |
AuthorsOpenReview ↗

Low false-positive rates for OQT

Q4. Low false-positive rates

Thank you for reading Figure 4 this closely, and for suggesting that the comparison be reported directly. We will add the comparison to the revised manuscript as a table and keep the curve for the same pairs, so the two readings can be checked against each other. On the shared collection of 595 pairs, OQT leads at 1% and 5% false-positive rates and ties hypothesis testing at 10%, while LLMmap leads on overall AUC; the revised text will attribute that difference to the labels LLMmap fits for every pool model, which a label-free score cannot use.

Table R1.4. Low false-positive comparison on the shared collection. Each method is run on all the pairs it covers.

MethodCoverageAUROCTPR at 1% FPRTPR at 5% FPRTPR at 10% FPRpAUC below 10%
OQT (ours)35 models, 595 pairs0.7220.0900.2180.3330.218
LLMmap35 models, 595 pairs0.8550.0130.0900.1670.098
Nikolić et al.35 models, 595 pairs0.7090.0770.1790.3330.209
QueRE32 models, 496 pairs, 9 lineage0.5370.0000.0000.1110.029

Hide and Seek has no score to put in this table. Each of its nine groups holds five models, two of them related, and it must name those two: it did so in 5 of the 39 answers it produced, where guessing succeeds 1 time in 10.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes in the Revised Paper

Summary of Changes in the Revised Paper

We thank the reviewers for their constructive feedback. The revised manuscript strengthens the evaluation of both PAD and OQT and clarifies the presentation. The main changes are:

  1. PAD: false-positive reduction and the role of differencing. We added low-FPR comparisons on the existing 34-model, 561-pair cohort and a clean-only ablation, highlighting the benefit of differencing with the SSIM matcher. Key cross-family examples and low-FPR results now appear in the main text. (Section 5.1.1; Tables 1–4.)

  2. PAD: broader baselines and controlled comparisons. We expanded REEF/Adv-REEF comparisons across layer-selection rules, added HuRef-ICS and GhostSpec-MSE on compatible checkpoint subsets, and included matched independent-model controls and component/normalization analyses. (Appendix A.4; Tables 10–12.)

  3. PAD: adaptive-attack evaluation. We added a 200-update attack study across three models with three seeds each, using fixed detection thresholds and explicit utility-preservation and evasion criteria. The results and evaluation protocol are provided in the main text and appendix. (Section 5.1.2; Table 5; Appendix A.5.)

  4. OQT: retrieval metrics and additional baselines. We added Top-3 and median-rank results, model/pair coverage and low-FPR detection metrics, QueRE and Hide and Seek reproductions, and three controls using the same responses. Query protocols and budgets are also clarified. (Sections 5.2.1–5.2.2; Tables 6–7; Appendix N; Table 25.)

  5. OQT: calibration, robustness, and query budget. We added leave-one-family-out threshold transfer and single-pair sensitivity analyses, expanded the query-budget protocol description, and updated embedding and response post-processing results. (Sections 5.2.3–5.2.4; Appendices G, J, and O.)

  6. Presentation and numerical corrections. We redrew Figure 1, organized PAD results around effectiveness and robustness, and streamlined the appendix. We also corrected PAD summary metrics to match the reported score matrices and clarified model names and numerical precision. (Figure 1; Section 5.1; Appendix A.3; Table 9.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.