acceptodds
Under review as a conference paper at ICLR 2027

DistDNA: Language Model Fingerprinting under Stochastic Decoding

10 views

Abstract

Behavioral fingerprints identify a large language model (LLM) from its responses to a shared set of prompts, enabling applications such as verifying whether a deployed model corresponds to the checkpoint it claims to be. Deployed models, however, are commonly served with stochastic decoding, under which the same model may produce different responses to an identical prompt, depending on an unknown temperature and top- setting. Existing representations such as LLM DNA provide limited treatment of this variation. A natural remedy is to characterize models by their response distributions, which, however, raises two challenges. First, each distribution must be estimated from only a small number of sampled responses. Second, a change in decoding setting can alter responses to the same extent as a change of checkpoint. Consequently, fingerprint distances alone provide an incomplete basis for model identity, and identification typically relies on a classifier trained jointly on fingerprints from many checkpoints. We propose DistDNA, which represents a model by its prompt-conditional response distributions and compares models by prompt-wise maximum mean discrepancy (MMD). The model under examination is assigned to the known checkpoint whose reference fingerprints, collected at several decoding settings, lie nearest to its own, without training a classifier. We show that finite sampling adds variance-dependent terms to the expected squared fingerprint distance, with these terms decreasing as for equal response counts, where is the number of responses per prompt. Experiments across 115 model checkpoints demonstrate that DistDNA improves identification accuracy over the strongest baseline, LLM DNA, by 13.25%.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, stochastic decoding, model fingerprinting, maximum mean discrepancy, radial basis function

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZwfkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DistDNA is a black-box method for detecting which model responds to a request. DistDNA represents a model by the distributions of sentence-embedded responses that it samples repeatedly for a fixed prompt panel, and compares models by prompt-wise MMD against references enrolled at several decoding settings, using either nearest-reference matching or an SVM. The results show promising progress toward being able to identify models behind APIs or request router middleware.

Critical strengths

I appreciate the authors for taking LLM lineage and authenticity tracking a step forward. The main strength of the paper is that it shifts the perspective away from "How closely related are models?" (e.g., LLM DNA) towards "Am I actually talking to the model I was requesting?". In times where request routers gain popularity, I believe, this is an important question to answer, especially when aiming to integrate generative models into (business) processes that require reliability. Also, incorporating serving parameters into the detection framework is a meaningful step forward and helpful in practice.

Critical weaknesses

The paper uses a very specific dataset as a basis for evaluation (Dolly). Hence, I am wondering how the panel of reference prompts could be varied (or better said should be varied) to reduce the risk of closed-source model providers or router services to simply try and detect the panel used in the paper and provide answers generated by the expected model? Put simply, are there any other datasets that would work with the DistDNA approach?

As a second point: The paper claims to be able to identify which model is responding. What is the confidence of that? Based on the claims in the introduction, I think, it would be of great value to the readers of the paper to understand how "reliable" DistDNA is. Given generic verification is an open set problem, a potentially suitable starting point is to compare different panels, provide performance estimates, and compute the confidence levels in the approach (e.g., via p-values for each candidate across panels).

The third aspect that I think deserves separate attention are reasoning and thinking models. The panel used in the study uses prompts designed for non-thinking models and a generation of models that came up over 3 years ago. The paper would benefit from exploring the reliability of DistDNA in context of models that can represent a much greater data distribution and cater to more complex query than what the Dolly panel offers.

As a minor remark: To me is not fully clear what the term "calibration" means throughout the paper. It would be helpful to introduce its meaning once with an explicit statement.

Questions for authors

Please see the weakness section for both context and questions.

3 comments

Thank you for the constructive questions on prompt generality, panel refreshability, decision confidence, and modern reasoning services.

Concern 1.1: "are there any other datasets that would work with the DistDNA approach?"

Response. Yes—beyond procedural and random-CFG domains (Table 9), DistDNA transfers directly to BIG-Bench Hard (BBH) and MMLU-Pro. Evaluating 24 services (144 queries/panel) separately on 20 Dolly, 20 BBH, and 20 MMLU-Pro prompts under matched routes, decoding settings, and budgets shows strong identification across all three:

Prompt panelLinear nearestRBF nearestLinear SVMRBF SVM
Dolly84.03%84.03%90.97%90.97%
BBH88.89%88.19%90.28%88.89%
MMLU-Pro84.72%85.42%90.28%87.50%

Both structured benchmarks outperform Dolly on nearest-reference (+4.86 / +4.17 pp on BBH; +0.69 / +1.39 pp on MMLU-Pro), while SVM stays consistently high (87.50%–90.97%; §5.7, App. F.12).

Concern 1.2: How stable is identification when the prompt panel changes?

Response. Over ten disjoint 10-prompt A/B splits, mean absolute accuracy gaps stay within 1.61–2.66 points. Split 1 illustrates this consistency:

ConfigurationPanel A (%)Panel B (%)
DistDNA-Linear / nearest85.3283.14
DistDNA-Linear / SVM93.2092.87
DistDNA-RBF / nearest85.7183.61
DistDNA-RBF / SVM93.1293.65

Each panel uses ten disjoint prompts with aligned references and fresh query draws, confirming DistDNA captures intrinsic behavior rather than specific Dolly questions (App. F.7).

Concern 2: A provider could “detect the panel used in the paper”.

Response. Private, periodically refreshed panels prevent panel detection, as DistDNA requires prompt alignment between reference and query rather than any fixed public list. During partial refreshes, prior responses remain reusable while distances update incrementally only for newly added prompts (App. F.7). Passive identification from unaligned traffic is promising future work.

Concern 3.1: How reliable are individual identifications?

Response. Post-hoc logistic calibration cuts held-out Brier error by ~19% (p<0.05p<0.05) while keeping 5-bin ECE within 3.5%–4.1%:

ConstructionBaseline Brier↓Fitted Brier↓Paired 95% Brier gain5-bin ECE↓
DistDNA-Linear0.09150.0738[0.0061, 0.0329]0.0410
DistDNA-RBF0.08430.0680[0.0055, 0.0306]0.0353

Paired 95% intervals strictly exclude zero. Margin abstention retains 85.63% of RBF queries at 3.14% error, or 72.08% at 0.58% error (App. F.9).

Concern 3.2: How does DistDNA handle models outside the gallery?

Response. Pairing our classifier-free matcher with a lightweight learned rejection gate more than doubles accepted-and-correct identifications over LLMmap's official detector at comparable Unknown FAR:

MethodUnknown FAR (%)Known FRR (%)Correct + accepted (%)
LLMmap (official detector)53.0047.5843.00
DistDNA-Linear (learned gate)57.833.7586.44
DistDNA-RBF (learned gate)52.173.7986.35

At the 95% target, DistDNA-RBF cuts Known FRR from 47.58% to 3.79% while lifting accepted-and-correct queries to 86.35% (and lowers Unknown FAR to 18.25% at 15.50% FRR under an 80% target; §5.7, App. F.8).

Concern 4.1: Is the gallery restricted to older models?

Response. No. Beyond Falcon-H1R-7B (Jan 2026), our 86-service expansion spans DeepSeek V4 Flash/Pro, Qwen3.8 Flash/Max, GLM5.3/Flash, Kimi K3/K3.1, Gemma 4, and MiniMax M3/M3.1 (App. E.1), reaching 86.43% / 86.82% nearest-reference and 88.37% / 88.18% SVM accuracy (§5.7, App. F.13).

Concern 4.2: Does DistDNA work with reasoning/thinking models?

Response. Yes. Across 64 verified reasoning-enabled services against the full 86-service gallery under matched modes, DistDNA-Linear / DistDNA-RBF reach 82.81% / 83.33% nearest-reference and 85.42% / 85.42% SVM accuracy (§5.7, App. F.13).

Concern 5: What does “calibration” mean?

Response. Revised §4.1 and App. F.8–F.9 separate kernel scale calibration (fixing bandwidths), probability calibration (per-decision confidence), and rejection thresholding (open-set filtering).

Thank you for the detailed rebuttal. It addresses some of my concerns. However, there are still a couple remaining:

(1) Regarding robustness to panel detection: You mention that private (rotating) panels prevent detection, but there is no evaluation. My concern is broader as in that the method sends the same prompt multiple times and maybe even at several different decoding settings. Hence, these panels and prompt patterns could be detected as verification traffic. To me, this is a key limitation that is worth experimental evaluation (incl. the cost to maintain the panel with state-of-the-art models, given that new open and closed weight models are being released at high frequency recently).

(2) Generality of datasets beyond Dolly: Thanks for the additional experiments, they definitely help to understand the dynamics of distdna better. However, the sample size is relatively small and the differences appear to be within sampling noise. This leads me to think that the new panels show the same effect as dolly rather than generalizing dynamics.

(3) Reasoning models: The performance drop between non-reasoning and reasoning models is quite significant. I would have appreciated a discussion what causes this drop. Also, it might be worth discussing what kind of panel is useful for reasoning models. Does the panel have to be more complex compared to prompts drawn from datasets meant for non-reasoning models?

With these points still open, I maintain my original score.

Thank you for taking the time to follow up. Your three questions prompted us to run several additional analyses across our 86-service banks, which helped clarify how the method behaves in each setting:

1. Verification Traffic Patterns and Gallery Maintenance Costs

  • Clarifying single-setting, single-draw verification (Rq=1R_q=1): We appreciate the chance to clarify how verification queries operate. Multi-setting collection is only performed offline on trusted references during enrollment, whereas every online verification query runs at one single serving setting (§3.1, §4). Furthermore, because Theorem 3.2 separates reference and query sampling variance (E∥Γ^q−Γ^r∥2=∥Γq−Γr∥2+Vˉq/Rq+Vˉr/Rr\mathbb{E}\|\widehat\Gamma_q-\widehat\Gamma_r\|^2=\|\Gamma_q-\Gamma_r\|^2+\bar V_q/R_q+\bar V_r/R_r), keeping Rr=4R_r=4 at enrollment allows online verification with a single response per prompt (Rq=1R_q=1, zero repeated prompts) and random 20-of-60 prompt subpanels (which reduce average prompt overlap between sessions from 20 to 6.67 prompts):
Gallery and query mode (1 setting)CallsRBF nearest
20 prompts × 4 responses8086.82%
20 prompts × 1 response (0 repeats)2083.29%
random 20-of-60 × 4 responses8095.66%
random 20-of-60 × 1 response (0 repeats)2091.63%

Single-draw querying cuts online calls by 75%, avoids repeating prompts or sweeping settings within a verification, and retains 83.29% and 91.63% accuracy. We agree that evaluating broader undetectability against semantic, timing, or account-history detectors remains an important direction for future work.

Measured SOTA maintenance costs: Because nearest-reference matching is classifier-free (Eq. 7), adding a new model requires 640 responses (independent of gallery size) and zero retraining. Recorded ledger charges have a median of $0.258/service across the 86-service Dolly bank ($0.011–$5.487).

2. Dataset Generality and Complementary Error Dynamics Beyond Dolly

Your point that the marginal accuracies of Dolly (84.03%), BBH (88.19%), and MMLU-Pro (85.42%) look close on the 24-service cohort motivated us to inspect their per-query error overlap across all 144 queries. Interestingly, the panels resolve largely complementary errors: between BBH (17 RBF errors) and MMLU-Pro (21 RBF errors), only 1 query is wrong under both—meaning 16 of 17 BBH errors (94.1%) are resolved by MMLU-Pro and 20 of 21 MMLU-Pro errors (95.2%) are resolved by BBH. Because they capture complementary signals, combining them into a balanced 20-prompt mixture (7 Dolly + 7 BBH + 6 MMLU-Pro) at the exact same 80-call budget improves accuracy over any single panel.

20-prompt panelLinear nearestRBF nearestLinear SVMRBF SVM
Dolly (homogeneous)84.03%84.03%90.97%90.97%
BBH (homogeneous)88.89%88.19%90.28%88.89%
MMLU-Pro (homogeneous)84.72%85.42%90.28%87.50%
7 Dolly + 7 BBH + 6 MMLU-Pro (same budget)96.94%97.22%97.22%97.64%

3. Understanding the Reasoning-Cohort Gap and Prompt Complexity

  1. Sibling-version ambiguity and dispersion: On the 64 reasoning-requested services (83.33% Exact Top-1 vs. 96.97% on the 22 other services), Top-3 accuracy is 99.22% (381/384), family accuracy is 98.70%, and 59/64 errors (92.19%) are confusions between sibling versions within the true family. Because the reasoning cohort concentrates dense sibling series (e.g., 17 GLM and 38 Qwen variants), equal-weighting shared families narrows the Top-1 difference from 13.64 to 1.86 pp (95% CI spanning zero). At the same time, within-prompt RBF variance Vˉg\bar V_g (Theorem 3.2) is 35.6% higher (+0.01949, 95% CI [0.01330, 0.02603]), causing adjacent siblings to occasionally swap Rank 1 and 2. Answer-head (83.33%) vs. tail (83.07%) encoding is unchanged, while exploratory answer + reasoning-trace fusion lifts RBF accuracy to 85.16% nearest / 88.28% SVM.
  2. **How harder panels help reasoning models:
ServicesRBF nearestLinear nearestQuery variance↓Rival margin↑
Dolly82.58%82.58%0.070130.01313
MMLU-Pro84.09%83.33%0.068210.03429 (+161%)
BBH87.12%87.88%0.06269 (−10.6%)0.02170 (+65%)

Both harder panels significantly increase separation margins from the nearest rival (positive paired CIs), while BBH reduces query variance by 10.6% and lifts nearest-reference accuracy by +4.54 to +5.30 pp.

We hope this resolves your concerns, and we warmly welcome your reassessment.

Reviewer_KuwPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies closed-set identification of enrolled LLM checkpoints under unknown stochastic decoding settings. DISTDNA represents checkpoint–decoding pairs using prompt-conditional distributions of semantic response embeddings and compares them through linear or RBF-kernel MMD. Identification uses either nearest-reference matching across multiple enrolled decoding settings or a supervised SVM. The paper derives finite-sampling variance identities and evaluates the method on 115 checkpoint entries across 36 query decoding settings, supplemented by smaller-scale representation, sampling, and robustness studies.

Critical strengths
  1. The paper addresses a practical difficulty in behavioral model identification: stochastic decoding makes individual responses unstable indicators of checkpoint identity. Its distinction between checkpoint identity and checkpoint–decoding behavior is useful, and the population representation and finite-sampling identities are clearly specified under explicit assumptions. This provides a coherent foundation for studying repeated-response fingerprints.
  2. The 115-entry evaluation provides meaningful evidence of improvement over the evaluated external baselines. Reporting both nearest-reference and supervised identification helps distinguish fingerprint-distance quality from classifier performance, while shared response banks and matched response budgets improve comparability. The sampling-allocation studies provide useful practical insights, and the manuscript appropriately acknowledges its closed-set scope and the limits of behavioral attribution.
Critical weaknesses
  1. In Table 2, RBF improves over Linear by only 0.92 percentage points in nearest-reference Top-1 and 0.51 points in supervised Top-1, while Linear achieves slightly higher supervised Top-3. The dimensionality difference from the LLM-DNA adaptation also complicates attribution of the gains. Table 2 reports descriptive variability rather than paired uncertainty, and the supplementary controls concern smaller experiments. Thus, the main results do not clearly isolate the benefit of the nonlinear kernel. Figure 3 adds limited evidence because of overlapping labels, crowded regions, and the absence of the Linear control.
  2. The directly relevant identification method LLMmap is discussed but not evaluated; a budget-aware comparison or a clear explanation of incompatibilities is needed. Although Appendix C.7 includes domain and native-chat experiments, these conditions are separately enrolled and calibrated, so they do not establish transfer from fixed references to an unseen deployment interface. Open-set rejection is also unevaluated. These gaps limit the practical conclusions beyond the demonstrated closed-set setting.
Questions for authors
  1. On the 115-entry protocol, what are the paired uncertainty intervals for RBF versus Linear and for each construction versus LLM-DNA? Considering the dimensionality controls, which gains can be attributed specifically to nonlinear distributional matching rather than repeated sampling or retaining uncompressed means?
3 comments

Thank you for the constructive feedback on baselines, dimensionality attribution, and deployment scope.

Concern 1.1: Why use RBF when its overall improvement over Linear is small?

Response. Aggregate averages look close because first-order mean shifts separate most checkpoints, but RBF gains 29–33 points when models share equal or nearby means with distinct covariance or dispersion:

CasesLinear (%)RBF (%)
Equal means, orthogonal variation58.3091.60
Equal means, random noise58.4091.05
Nearby means, distinct variations61.2790.22

At equal population means, Linear distance is zero whereas characteristic RBF separates higher-order structure (§3), holding under noise and nearby means. Linear serves as the fast first-order default; RBF provides higher-order resolution when candidate means converge (revised §5.3, App. F.6).

Concern 1.2: Does the dimensionality difference explain gains over LLM-DNA?

Response. No. Controlling width in both directions—compressing DistDNA to 512 coordinates or expanding LLM-DNA to 15,360—confirms both DistDNA variants preserve their full lead at equal size:

ConstructionDim.Nearest (%)SVM (%)
LLM-DNA51273.6682.22
DistDNA-Linear51280.3186.79
DistDNA-RBF51282.8787.47
LLM-DNA15,36073.9784.17
DistDNA-Linear15,36080.3789.42
DistDNA-RBF15,36081.2989.93

Expanding LLM-DNA to 15,360 coordinates adds just 0.31 points on nearest, whereas 512-d DistDNA beats 512-d LLM-DNA by +6.65 (Linear) and +9.21 (RBF), matching the full-width margins (revised §5.3, App. F.3).

Concern 1.3: Do paired intervals support RBF over Linear and both constructions over LLM-DNA?

Response. Yes—both variants hold positive intervals over LLM-DNA across readouts, and RBF holds one over Linear on nearest. “Pointwise 95%” is a 95% confidence interval computed separately per contrast:

Readout / contrastGainPointwise 95%
Nearest: RBF − Linear0.92[0.06, 1.98]
Nearest: RBF − LLM-DNA7.63[5.17, 10.23]
Nearest: Linear − LLM-DNA6.71[4.50, 8.99]
SVM: RBF − Linear0.51[−0.04, 1.17]
SVM: RBF − LLM-DNA5.72[3.64, 8.12]
SVM: Linear − LLM-DNA5.22[3.13, 7.56]

Because Linear and RBF target complementary regimes within DistDNA rather than competing as rivals, their aggregate gap is modest (SVM: [−0.04, 1.17]), whereas both beat LLM-DNA by 5.22–7.63 points across readouts (revised §5.2, App. F.1).

Concern 2: "LLMmap is discussed but not evaluated".

Response. In a full 115-identity evaluation at matched query and enrollment response counts, both DistDNA variants outperform LLMmap by over 16 points:

MethodSize (KiB)Top-1 (%)
LLMmap363.55
PhyloLM-4char0.3161.28
LLM-DNA473.66
DistDNA-Linear12080.37
DistDNA-RBF48081.29

Both gains hold positive 95% pointwise intervals, and 4 KiB DistDNA-Linear (80.31%, Concern 1.2) preserves this >16-point lead at comparable storage. LLMmap uses released probes and mapping with signature averaging to match response budgets (revised Table 2, App. F.5).

Concern 3: Do the results establish transfer to an unseen deployment interface?

Response. Yes across heterogeneous and opaque chat interfaces when endpoints are enrolled and queried in their deployed mode. Embedding only returned text without template or logit access, DistDNA handles model-specific chat templates and unseen provider wrappers directly—reaching 100% on our 13-model local chat gallery and 86.43%–86.82% nearest accuracy across 86 live API services (revised §5.7, App. F.14). Enrolling through each deployed endpoint aligns interface conditions without cross-template adaptation.

Concern 4: “Open-set rejection is also unevaluated.”

Response. Pairing DistDNA's classifier-free matcher with a lightweight learned gate more than doubles correct-and-accepted identifications over LLMmap's official detector:

MethodUnknown FAR (%)Known FRR (%)Correct + accepted (%)
LLMmap (official)53.0047.5843.00
DistDNA-Linear (gate)57.833.7586.44
DistDNA-RBF (gate)52.173.7986.35

At a 95% known-acceptance target (vs. LLMmap's Boolean rule), DistDNA-RBF lifts correct-and-accepted queries from 43.00% to 86.35% and cuts Known FRR over 12× (47.58% to 3.79%) at slightly lower Unknown FAR (52.17% vs. 53.00%). An 80% target lowers RBF Unknown FAR to 18.25% at 15.50% Known FRR (revised §5.7, App. F.9).

Concern 5: Figure 3 is crowded and omits Linear.

Response. Revised Figure 3 now places DistDNA-Linear and DistDNA-RBF side by side in separate, de-cluttered panels with non-overlapping labels for direct visual comparison (revised §5.4).

Thank you for the rebuttal. The paired analysis and the new LLMmap comparison strengthen the evidence for closed-set identification. My assessment remains centered on two points: what accounts for the performance gap between the two 512-dimensional constructions based on promptwise means, and how often RBF offers a meaningful advantage over Linear on real queries. As these points remain unclear, I am retaining my original rating of 2.

Thank you for promptly narrowing the discussion to these two exact points:

(1) at 512-d, the gap comes from data-oblivious random sketching (LLM-DNA) vs. unlabeled enrollment-span projection (DistDNA, Corollary A.2)

(2) on real queries, RBF’s gain concentrates on the 20% crowded-rival boundary queries (+3.86 pp) once well-separated pairs saturate (revised §5.3, App. F.3, and App. F.6).

1. Why the two 512-d constructions differ

Equal coordinate count (m=512m = 512, 4 KiB) fixes container size:

RepresentationWidthCompression OperatorNearest (%)SVM (%)
LLM-DNA512Gallery-blind Gaussian sketch G∈R512×15,360G \in \mathbb{R}^{512 \times 15,360} (App. B.2)73.6682.22
LLM-DNA (control for higher d)15,36030× wider gallery-blind Gaussian sketch73.9784.17
DistDNA-Linear512Unlabeled enrollment-span PCA (Corollary A.2, App. A.1)80.3186.79
DistDNA-RBF512Unlabeled enrollment-kernel Nyström (Corollary A.2)82.8787.47

Because N=920N{=}920 enrolled references span rank ≤919≪15,360\le 919 \ll 15,360, LLM-DNA's blind Gaussian matrix discards 96.7%96.7\% (1−512/15,3601 - 512/15,360) of the enrollment subspace while keeping O(1/R)\mathcal{O}(1/R) sampling noise across all 512 axes—and expanding it 30×30\times to 15,360-d adds just +0.31 pp (73.97%). Conversely, projecting onto the unlabeled enrollment span (requiring zero labels, zero training, and zero extra queries) preserves DistDNA-Linear's full MMD geometry (80.31% vs. 80.37% uncompressed) while zeroing out >14,800>14,800 pure-noise axes. Replacing enrollment-span projection with a blind Gaussian sketch simply reverts to the 73.66% baseline.

2. How often—and where—RBF beats Linear on real queries

Expanding kσ(z,w)=e−1/σ2exp⁡(⟨z,w⟩/σ2)k_\sigma(z,w) = e^{-1/\sigma^2}\exp(\langle z,w\rangle/\sigma^2) decomposes squared RBF-MMD into Linear MMD plus higher-order dispersion. When checkpoint means are already well separated (~80% of queries, App. A.5) or T∈{0.1,0.3}T \in \{0.1, 0.3\}, both kernels agree by construction (+0.92 pp full-gallery average). Slicing the exact same queries shows where RBF acts:

Benchmark Scenario (115 Checkpoints)Real-Query SliceLinear (%)RBF (%)RBF Gain
Overall galleryAll queries across 36 (T,p)(T,p) cells (nearest)80.3781.29+0.92 pp
Crowded-mean boundary20% queries (nearest)37.1441.00+3.86 pp
Zero-mean-gap residual (all TT)All queries, mean-invariant residuals (SVM)0.8727.28+26.41 pp
Zero-mean-gap residual (high TT)High-temperature slice, mean-invariant residuals (SVM)0.8742.68+41.81 pp

Across the 20% crowded-mean queries (n=828n{=}828), RBF lifts accuracy by +3.86 pp (>4×>4\times the gallery average), and on mean-invariant residuals where Linear sits at chance (0.87%) at any width, RBF recovers 27.28%–42.68% purely from real response dispersion.

We hope this resolves both of your concerns, and we warmly welcome your reassessment.

Reviewer_FULtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies exact identification of enrolled language-model checkpoints when the queried model uses unknown temperature and top-p settings. DistDNA collects repeated responses to shared prompts, embeds them with a frozen semantic encoder, and compares prompt-conditional response distributions using kernel mean embeddings and maximum mean discrepancy. Identification uses either nearest-reference matching across multiple enrolled decoding settings or an SVM.

Critical strengths

The 115-entry gallery and 36-setting query grid test identification under changes beyond the enrollment range. Shared response banks and readout policies make the representation comparisons useful, while the additional fresh-response study strengthens the sampling-allocation findings.

The linear representation, variance-corrected estimator, projection-width comparisons, and fixed-budget repetition experiments help identify which components matter. In particular, retaining the strong linear control and reporting mixed effects from variance correction make the empirical account more informative.

Critical weaknesses

DistDNA-Linear already achieves 80.37% nearest-reference and 89.42% supervised accuracy, compared with 81.29% and 89.93% for DistDNA-RBF. Most of the improvement over LLM-DNA therefore survives without the nonlinear distributional representation. Since LLM-DNA compresses the concatenated means to 512 coordinates while DistDNA-Linear retains 15,360, the headline comparison does not isolate the benefit of distributional matching from retaining a larger representation. The variance theorem is a standard sample-mean variance decomposition and does not establish that the proposed representation improves identification under decoding shifts.

LLMmap explicitly targets model identification under unknown sampling settings, yet appears only in the related-work discussion. The evaluated external baselines primarily originate from broader behavioral-representation or model-relationship tasks.

Questions for authors

Is the RBF improvement over DistDNA-Linear reliable under paired uncertainty analysis and fresh response draws?

How do the methods compare at matched fingerprint storage and computational cost, in addition to matched response counts?

3 comments

Thank you for the questions on attribution, uncertainty, and cost.

Concern 1.1: Most gains survive in Linear; why introduce RBF?

Response. RBF gains 29–33 points over Linear when models share equal or nearby means with distinct covariance or dispersion:

CasesLinear (%)RBF (%)
Equal means, orthogonal variation58.3091.60
Equal means, random noise58.4091.05
Nearby means, distinct variations61.2790.22

At equal means, Linear distance is zero while characteristic RBF separates distinct distributions (§3), holding under noise and nearby means. Linear resolves first-order shifts; RBF adds higher-order sensitivity when means converge (§5.3, App. F.6).

Concern 1.2: Do extra coordinates explain the gain attributed to distributional matching?

Response. No. Both DistDNA variants retain a substantial advantage over LLM-DNA at matched dimensionality, indicating that their gains cannot be explained by representation dimensionality alone:

ConstructionDim.Nearest (%)SVM (%)
LLM-DNA51273.6682.22
DistDNA-Linear51280.3186.79
DistDNA-RBF51282.8787.47
LLM-DNA15,36073.9784.17
DistDNA-Linear15,36080.3789.42
DistDNA-RBF15,36081.2989.93

Expanding LLM-DNA to 15,360-d adds just 0.31 points on nearest (1.95 SVM). Compressing DistDNA to 512-d narrows SVM gains relative to 512-d LLM-DNA (+7.20/+7.71 to +4.57/+5.25), yet still preserves +6.65/+9.21 nearest gains at 512-d and +5.25/+5.76 SVM gains at matched 15,360-d (§5.3, App. F.3).

Concern 1.3: How do performance and cost compare at matched storage?

Response. At matched 4 KiB (512-d), DistDNA-Linear beats LLM-DNA by +6.65 nearest and +4.57 SVM. For original designs (P=20P{=}20 prompts, R=4R{=}4 draws, d=768d{=}768, m=512m{=}512, N=920N{=}920 references), storage, per-fingerprint build and per-query matching complexity, and Xeon Gold 6346 (4 BLAS threads) timings across all 4,140 queries (BGE / MPNet, from encoded responses) are:

DesignFp / galleryBuildMatchBuild (s)Match+rank (s)
LLM-DNA (512-d)4 KiB / 3.59 MiBO(PRd+Pdm)O(Nm)1.54 / 1.660.18 / 0.19
DistDNA-Linear120 KiB / 107.81 MiBO(PRd)O(NPd)1.29 / 1.2735.08 / 2.33
DistDNA-RBF480 KiB / 431.25 MiBO(PRd)O(NPR²d)0.78 / 0.7746.40 / 39.41

LLM-DNA also stores a shared 60 MiB projection matrix. RBF uses R=4×R{=}4\times Linear's payload while pairwise comparisons add an R2R^2 factor in matching complexity, completing all 4,140 queries in 39.41–46.40 s (App. F.4).

Concern 2.1: Is the RBF–Linear improvement reliable under paired uncertainty?

Response. Yes for nearest. “Pointwise 95%” is a 95% confidence interval computed separately per contrast:

Readout / contrastGainPointwise 95%
Nearest: RBF − Linear0.92[0.06, 1.98]
Nearest: RBF − LLM-DNA7.63[5.17, 10.23]
Nearest: Linear − LLM-DNA6.71[4.50, 8.99]
SVM: RBF − Linear0.51[−0.04, 1.17]
SVM: RBF − LLM-DNA5.72[3.64, 8.12]
SVM: Linear − LLM-DNA5.22[3.13, 7.56]

RBF holds a positive interval over Linear on nearest ([0.06, 1.98]), and both beat LLM-DNA by 5.22–7.63 points. Since Linear and RBF target complementary regimes in DistDNA, their overall gap is modest (SVM: [−0.04, 1.17]), while both reliably beat LLM-DNA (§5.2, App. F.1).

Concern 2.2: Does the RBF–Linear comparison persist under fresh response draws?

Response. No. Against frozen references and kernel scales, DistDNA-Linear overtakes DistDNA-RBF on fresh draws:

Fresh Draws ReadoutDistDNA-Linear (%)DistDNA-RBF (%)Delta
Nearest-reference76.6772.06−4.61
SVM83.4480.17−3.27

Mean aggregation smooths cross-draw variation against fixed references, whereas RBF's dispersion sensitivity registers run-to-run drift unless references are refreshed. Linear is thus the drift-tolerant default across draws; RBF targets overlapping means (App. F.2).

Concern 3: How does DistDNA compare with LLMmap?

Response. Both variants beat LLMmap by over 16 points across all 115 identities at matched response budgets:

MethodSize (KiB)Top-1 (%)
LLMmap363.55
PhyloLM-4char0.3161.28
LLM-DNA473.66
DistDNA-Linear12080.37
DistDNA-RBF48081.29

Both gains hold positive 95% pointwise intervals, and 4 KiB DistDNA-Linear (80.31%, Concern 1.2) keeps this over 16-point lead at matched storage. LLMmap uses native probes and mapping with signature averaging to match response counts (Table 2, App. F.5).

Concern 4: What does the variance theorem contribute?

Response. It is a core pillar for finite-sample identification, decomposing empirical distances into population separation plus 1/R1/R query- and candidate-dependent variance offsets. Paired with a positive population margin (exceeding decoding drift) and debiased variance correction or sufficient repetition RR, it establishes reliable detection under finite budgets and decoding shifts (§3, §5, App. A).

The matched-width results and LLMmap comparison address my baseline concerns. I’m keeping my score because the benefit of nonlinear matching remains unclear. RBF improves accuracy on crowded cases in the original evaluation, but its behind Linear on newly generated answers. I’d be more persuaded if the gain on those hard cases also held up there.

Thank you for confirming that the matched-width controls and LLMmap comparison address your baseline concerns.

There is a straightforward reason why RBF trails Linear in Appendix F.2 while helping on crowded cases elsewhere, and it comes down to cross-run drift versus matched collection:

Why RBF trails in App. F.2: Only the queries were newly generated while the reference gallery stayed frozen from an earlier run. Because RBF compares response spread (higher-order dispersion) in addition to the mean, any small run-to-run shift between frozen references and fresh queries adds mismatch to the spread terms. Linear averages across responses, smoothing out that cross-run drift.

What happens on newly generated answers without cross-run mismatch: When newly generated responses are evaluated under a consistent protocol—both in our fresh-seed benchmark (App. C.7) and across our 86 live APIs (App. F.13)—the hard-case pattern holds up. On the 86 live APIs, Linear and RBF make identical predictions across the 80% well-separated queries, and every difference occurs on the hardest 10% crowded boundary, where RBF lifts accuracy from 35.3% to 39.2% (and from 58.3% to 60.4% on the hardest 10% of the fresh-seed bank).

In short, DistDNA’s primary advantage over prior baselines comes from prompt-aligned, multi-setting distributional matching itself, while the kernel choice inside DistDNA reflects a standard bias–variance trade-off (§5.3): Linear is naturally steadier when queries drift relative to older frozen references, whereas RBF provides fine-grained tie-breaking among crowded rivals under matched collection conditions. We hope this resolves your concerns, and we warmly welcome your reassessment.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you to Reviewers FULt, KuwP and Zwfk for the constructive feedback. We have updated the manuscript (§5.3, §5.7, App. A.1, E.1, and F.1–F.13) with new experiments covering matched-width controls, real-query distributional slices, open-set rejection and confidence calibration, and evaluations on structured benchmarks and live reasoning APIs:

1. Subspace Alignment at Matched Width & Operating Regimes of Linear vs. RBF (§5.3, App. A.1, F.2–F.6) 512-d vs. 15,360-d controls (App. A.1, F.3): On all 115 checkpoints (4,140 queries), expanding LLM-DNA’s gallery-blind Gaussian sketch from 512-d to 15,360-d moves nearest-reference accuracy from 73.66% to 73.97% (+0.31 pp; 82.22% to 84.17% SVM). In contrast, compressing DistDNA to 512-d (4 KiB) via unlabeled enrollment-span projection (Corollary A.2: PCA for Linear, Nyström for RBF—using zero labels, classifier training, or extra queries) reaches 80.31% (Linear) and 82.87% (RBF) nearest-reference accuracy (86.79% and 87.47% SVM). This shows that the accuracy gain comes from preserving the rank-≤919\le 919 reference subspace while filtering >14,800>14,800 orthogonal sampling-noise axes (Theorem 3.2), rather than coordinate count, and holds at a 4 KiB footprint.

When RBF beats Linear vs. cross-draw drift (App. F.2, F.6): A spherical Taylor expansion (Eq. 13) shows that when checkpoint means are well separated (~80% of gallery queries), both kernels agree (+0.92 pp overall, 95% CI [0.06, 1.98]). Slicing the same 4,140 real queries shows that on the hardest 20% crowded-rival boundary queries (n=828n=828), RBF lifts nearest-reference accuracy from 37.14% to 41.00% (+3.86 pp), and on mean-invariant residuals—where Linear drops to 0.87% chance at any width—RBF recovers 27.28% overall and 42.68% at high temperatures. Conversely, on fresh query draws against frozen references (App. F.2), Linear is less sensitive to run-to-run sampling drift (76.67% vs. 72.06% nearest). In practice, Linear serves as a fast, drift-tolerant default, while RBF resolves closely clustered rivals with overlapping means.

2. Open-Set Rejection, Calibrated Confidence, & LLMmap Baseline (§5.7, App. F.4–F.5, F.8–F.9) Closed- and open-set comparisons (App. F.4–F.5, F.8): At matched budgets across the 115 checkpoints, DistDNA improves over LLMmap’s matcher (63.55%) by +16.82 pp (Linear) and +17.74 pp (RBF), running all 4,140 queries in 2.3–46.4 s on CPU. For unseen out-of-gallery models, pairing nearest-reference matching with a lightweight rejection gate trained only on pseudo-unknown enrollment splits cuts Known FRR from 47.58% (LLMmap’s detector) to 3.79% and raises accepted-and-correct queries from 43.00% to 86.35% at comparable Unknown FAR (52.17% vs. 53.00%) at a 95% target, or lowers Unknown FAR to 18.25% (17.75% Linear) at an 80% target.

Confidence calibration (App. F.9): Logistic calibration on distance margins cuts held-out Brier error by ~19% (p<0.05p<0.05, 5-bin ECE 3.5%–4.1%), retaining 85.63% of RBF queries at 3.14% error or 72.08% at 0.58% error, supporting open-set deployment with calibrated per-query uncertainty and no external unknown training models.

3. Structured Benchmarks, Panel Rotation, & Modern Reasoning APIs (§5.7, App. E.1, F.7, F.10–F.13) BBH, MMLU-Pro, and panel refresh (App. F.7, F.12): Across 24 services (144 queries/panel), structured BBH (88.89% / 88.19% nearest; 90.28% / 88.89% SVM) and MMLU-Pro (84.72% / 85.42% nearest; 90.28% / 87.50% SVM) match or improve upon Dolly (84.03% nearest; 90.97% SVM) by +4.86 / +4.17 pp and +0.69 / +1.39 pp. Ten disjoint 10-prompt A/B splits show 1.61–2.66 pp mean gap, and incremental distance updates allow private panel rotation without re-collecting prior responses.

Live API and reasoning galleries (App. E.1, F.10–F.11, F.13): DistDNA reaches 97.64% nearest / 98.11% SVM across 106 live API-chat endpoints, 86.43% / 86.82% nearest (88.37% / 88.18% SVM) across 86 modern native endpoints (DeepSeek V4, Qwen3.8, GLM-5.3, Kimi K3/K3.1, Gemma 4, MiniMax M3/M3.1), and 82.81% / 83.33% nearest (85.42% SVM) on the 64-service cohort with verified reasoning traces.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.