- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DistDNA is a black-box method for detecting which model responds to a request. DistDNA represents a model by the distributions of sentence-embedded responses that it samples repeatedly for a fixed prompt panel, and compares models by prompt-wise MMD against references enrolled at several decoding settings, using either nearest-reference matching or an SVM. The results show promising progress toward being able to identify models behind APIs or request router middleware.
Critical strengths
I appreciate the authors for taking LLM lineage and authenticity tracking a step forward. The main strength of the paper is that it shifts the perspective away from "How closely related are models?" (e.g., LLM DNA) towards "Am I actually talking to the model I was requesting?". In times where request routers gain popularity, I believe, this is an important question to answer, especially when aiming to integrate generative models into (business) processes that require reliability. Also, incorporating serving parameters into the detection framework is a meaningful step forward and helpful in practice.
Critical weaknesses
The paper uses a very specific dataset as a basis for evaluation (Dolly). Hence, I am wondering how the panel of reference prompts could be varied (or better said should be varied) to reduce the risk of closed-source model providers or router services to simply try and detect the panel used in the paper and provide answers generated by the expected model? Put simply, are there any other datasets that would work with the DistDNA approach?
As a second point: The paper claims to be able to identify which model is responding. What is the confidence of that? Based on the claims in the introduction, I think, it would be of great value to the readers of the paper to understand how "reliable" DistDNA is. Given generic verification is an open set problem, a potentially suitable starting point is to compare different panels, provide performance estimates, and compute the confidence levels in the approach (e.g., via p-values for each candidate across panels).
The third aspect that I think deserves separate attention are reasoning and thinking models. The panel used in the study uses prompts designed for non-thinking models and a generation of models that came up over 3 years ago. The paper would benefit from exploring the reliability of DistDNA in context of models that can represent a much greater data distribution and cater to more complex query than what the Dolly panel offers.
As a minor remark: To me is not fully clear what the term "calibration" means throughout the paper. It would be helpful to introduce its meaning once with an explicit statement.
Questions for authors
Please see the weakness section for both context and questions.
3 comments
Thank you for the detailed rebuttal. It addresses some of my concerns. However, there are still a couple remaining:
(1) Regarding robustness to panel detection: You mention that private (rotating) panels prevent detection, but there is no evaluation. My concern is broader as in that the method sends the same prompt multiple times and maybe even at several different decoding settings. Hence, these panels and prompt patterns could be detected as verification traffic. To me, this is a key limitation that is worth experimental evaluation (incl. the cost to maintain the panel with state-of-the-art models, given that new open and closed weight models are being released at high frequency recently).
(2) Generality of datasets beyond Dolly: Thanks for the additional experiments, they definitely help to understand the dynamics of distdna better. However, the sample size is relatively small and the differences appear to be within sampling noise. This leads me to think that the new panels show the same effect as dolly rather than generalizing dynamics.
(3) Reasoning models: The performance drop between non-reasoning and reasoning models is quite significant. I would have appreciated a discussion what causes this drop. Also, it might be worth discussing what kind of panel is useful for reasoning models. Does the panel have to be more complex compared to prompts drawn from datasets meant for non-reasoning models?
With these points still open, I maintain my original score.
Thank you for taking the time to follow up. Your three questions prompted us to run several additional analyses across our 86-service banks, which helped clarify how the method behaves in each setting:
1. Verification Traffic Patterns and Gallery Maintenance Costs
- Clarifying single-setting, single-draw verification (): We appreciate the chance to clarify how verification queries operate. Multi-setting collection is only performed offline on trusted references during enrollment, whereas every online verification query runs at one single serving setting (§3.1, §4). Furthermore, because Theorem 3.2 separates reference and query sampling variance (), keeping at enrollment allows online verification with a single response per prompt (, zero repeated prompts) and random 20-of-60 prompt subpanels (which reduce average prompt overlap between sessions from 20 to 6.67 prompts):
| Gallery and query mode (1 setting) | Calls | RBF nearest |
|---|---|---|
| 20 prompts × 4 responses | 80 | 86.82% |
| 20 prompts × 1 response (0 repeats) | 20 | 83.29% |
| random 20-of-60 × 4 responses | 80 | 95.66% |
| random 20-of-60 × 1 response (0 repeats) | 20 | 91.63% |
Single-draw querying cuts online calls by 75%, avoids repeating prompts or sweeping settings within a verification, and retains 83.29% and 91.63% accuracy. We agree that evaluating broader undetectability against semantic, timing, or account-history detectors remains an important direction for future work.
Measured SOTA maintenance costs: Because nearest-reference matching is classifier-free (Eq. 7), adding a new model requires 640 responses (independent of gallery size) and zero retraining. Recorded ledger charges have a median of $0.258/service across the 86-service Dolly bank ($0.011–$5.487).
2. Dataset Generality and Complementary Error Dynamics Beyond Dolly
Your point that the marginal accuracies of Dolly (84.03%), BBH (88.19%), and MMLU-Pro (85.42%) look close on the 24-service cohort motivated us to inspect their per-query error overlap across all 144 queries. Interestingly, the panels resolve largely complementary errors: between BBH (17 RBF errors) and MMLU-Pro (21 RBF errors), only 1 query is wrong under both—meaning 16 of 17 BBH errors (94.1%) are resolved by MMLU-Pro and 20 of 21 MMLU-Pro errors (95.2%) are resolved by BBH. Because they capture complementary signals, combining them into a balanced 20-prompt mixture (7 Dolly + 7 BBH + 6 MMLU-Pro) at the exact same 80-call budget improves accuracy over any single panel.
| 20-prompt panel | Linear nearest | RBF nearest | Linear SVM | RBF SVM |
|---|---|---|---|---|
| Dolly (homogeneous) | 84.03% | 84.03% | 90.97% | 90.97% |
| BBH (homogeneous) | 88.89% | 88.19% | 90.28% | 88.89% |
| MMLU-Pro (homogeneous) | 84.72% | 85.42% | 90.28% | 87.50% |
| 7 Dolly + 7 BBH + 6 MMLU-Pro (same budget) | 96.94% | 97.22% | 97.22% | 97.64% |
3. Understanding the Reasoning-Cohort Gap and Prompt Complexity
- Sibling-version ambiguity and dispersion: On the 64 reasoning-requested services (83.33% Exact Top-1 vs. 96.97% on the 22 other services), Top-3 accuracy is 99.22% (381/384), family accuracy is 98.70%, and 59/64 errors (92.19%) are confusions between sibling versions within the true family. Because the reasoning cohort concentrates dense sibling series (e.g., 17 GLM and 38 Qwen variants), equal-weighting shared families narrows the Top-1 difference from 13.64 to 1.86 pp (95% CI spanning zero). At the same time, within-prompt RBF variance (Theorem 3.2) is 35.6% higher (+0.01949, 95% CI [0.01330, 0.02603]), causing adjacent siblings to occasionally swap Rank 1 and 2. Answer-head (83.33%) vs. tail (83.07%) encoding is unchanged, while exploratory answer + reasoning-trace fusion lifts RBF accuracy to 85.16% nearest / 88.28% SVM.
- **How harder panels help reasoning models:
| Services | RBF nearest | Linear nearest | Query variance↓ | Rival margin↑ |
|---|---|---|---|---|
| Dolly | 82.58% | 82.58% | 0.07013 | 0.01313 |
| MMLU-Pro | 84.09% | 83.33% | 0.06821 | 0.03429 (+161%) |
| BBH | 87.12% | 87.88% | 0.06269 (−10.6%) | 0.02170 (+65%) |
Both harder panels significantly increase separation margins from the nearest rival (positive paired CIs), while BBH reduces query variance by 10.6% and lifts nearest-reference accuracy by +4.54 to +5.30 pp.
We hope this resolves your concerns, and we warmly welcome your reassessment.
Thank you for the constructive questions on prompt generality, panel refreshability, decision confidence, and modern reasoning services.
Concern 1.1: "are there any other datasets that would work with the DistDNA approach?"
Response. Yes—beyond procedural and random-CFG domains (Table 9), DistDNA transfers directly to BIG-Bench Hard (BBH) and MMLU-Pro. Evaluating 24 services (144 queries/panel) separately on 20 Dolly, 20 BBH, and 20 MMLU-Pro prompts under matched routes, decoding settings, and budgets shows strong identification across all three:
Both structured benchmarks outperform Dolly on nearest-reference (+4.86 / +4.17 pp on BBH; +0.69 / +1.39 pp on MMLU-Pro), while SVM stays consistently high (87.50%–90.97%; §5.7, App. F.12).
Concern 1.2: How stable is identification when the prompt panel changes?
Response. Over ten disjoint 10-prompt A/B splits, mean absolute accuracy gaps stay within 1.61–2.66 points. Split 1 illustrates this consistency:
Each panel uses ten disjoint prompts with aligned references and fresh query draws, confirming DistDNA captures intrinsic behavior rather than specific Dolly questions (App. F.7).
Concern 2: A provider could “detect the panel used in the paper”.
Response. Private, periodically refreshed panels prevent panel detection, as DistDNA requires prompt alignment between reference and query rather than any fixed public list. During partial refreshes, prior responses remain reusable while distances update incrementally only for newly added prompts (App. F.7). Passive identification from unaligned traffic is promising future work.
Concern 3.1: How reliable are individual identifications?
Response. Post-hoc logistic calibration cuts held-out Brier error by ~19% (p<0.05) while keeping 5-bin ECE within 3.5%–4.1%:
Paired 95% intervals strictly exclude zero. Margin abstention retains 85.63% of RBF queries at 3.14% error, or 72.08% at 0.58% error (App. F.9).
Concern 3.2: How does DistDNA handle models outside the gallery?
Response. Pairing our classifier-free matcher with a lightweight learned rejection gate more than doubles accepted-and-correct identifications over LLMmap's official detector at comparable Unknown FAR:
At the 95% target, DistDNA-RBF cuts Known FRR from 47.58% to 3.79% while lifting accepted-and-correct queries to 86.35% (and lowers Unknown FAR to 18.25% at 15.50% FRR under an 80% target; §5.7, App. F.8).
Concern 4.1: Is the gallery restricted to older models?
Response. No. Beyond Falcon-H1R-7B (Jan 2026), our 86-service expansion spans DeepSeek V4 Flash/Pro, Qwen3.8 Flash/Max, GLM5.3/Flash, Kimi K3/K3.1, Gemma 4, and MiniMax M3/M3.1 (App. E.1), reaching 86.43% / 86.82% nearest-reference and 88.37% / 88.18% SVM accuracy (§5.7, App. F.13).
Concern 4.2: Does DistDNA work with reasoning/thinking models?
Response. Yes. Across 64 verified reasoning-enabled services against the full 86-service gallery under matched modes, DistDNA-Linear / DistDNA-RBF reach 82.81% / 83.33% nearest-reference and 85.42% / 85.42% SVM accuracy (§5.7, App. F.13).
Concern 5: What does “calibration” mean?
Response. Revised §4.1 and App. F.8–F.9 separate kernel scale calibration (fixing bandwidths), probability calibration (per-decision confidence), and rejection thresholding (open-set filtering).