acceptodds
Under review as a conference paper at ICLR 2027

RMF: Runtime Model Fingerprinting for Detecting API Model Substitution

Abstract

Third-party LLM aggregation platforms provide convenient access to multiple models through unified APIs, yet their opaque backend makes it difficult for users to verify which model actually serves a request, enabling API fraud through model substitution. Existing black-box fingerprinting methods primarily perform pre-deployment verification and therefore cannot reliably authenticate the model during actual task execution. To address this limitation, we propose a runtime model fingerprinting method (RMF) that embeds lightweight fingerprint probes directly into real-world task prompts. RMF constructs compact behavioral fingerprints from model-specific preferences over underdetermined ranking tasks and employs an adaptive probe search framework to select discriminative fingerprints for different task contexts. We evaluate RMF on 23 frontier LLMs across four real-world application domains. Compared with state-of-the-art baselines, RMF substantially improves model identification in both binary verification and multiclass identification settings, while introducing no statistically significant degradation in task performance and negligible additional token overhead.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Model Fingerprinting, Model Substitution Detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HWEWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper suggests to embedd fingerprint queries into task prompts to detect which LLM an API provider is serving. This could enable detecting fraudulent API providers. The prompts work by having LLMs rank different options as part of the task, which the authos show to uncover LLM specific characterstics. Multiple ranking tasks are combined to a probe bundle for increase detection efficiency, with selection done by via optimizting a second-order approximation of a probe bundle's utility. The method is reported to consistently outperform baselines in terms of both model detection and token overhead.

Critical strengths
  • The proposed method overall performs well
  • The paper performs multiple ablations on the method
Critical weaknesses
  • Very little information about the implemented baselines is given, and it is not really clear, how much of the performance gap can be explained by the method being tuned to the datasets at hand, why the baselines weren't. The choice of baselines is also a bit unclear, with very recent methods cited in the related work like Hu et al 2026 being omitted.
  • It seems very easy for an attacker to circumvent the proposed probes, as they are embedded in the task prompt, but are functionally completely separate.
  • Minor: Presentation could still be improved at some parts. For example, figure 2 is hard to read without a lot of zooming, especially part b). Some claims like "Our method achieves the optimal performance." also seem a bit strong
Questions for authors
  • How were the baselines selected and tuned?
8 comments

We thank the reviewer for the constructive feedback on baseline fairness, runtime security, and presentation. Our responses and additional results follow.

R1 (W1, Q1)

Baseline selection. We reviewed 2025–2026 methods for single-request, black-box model verification. We excluded RUT (Zhu et al., 2026) and LLMPrint (Hu et al., 2026) due to their white-box reference/base-model requirements, and MET (Gao et al., 2025) due to repeated distributional testing. We selected LLMmap (Pasquini et al., 2025), FLIPS (Richardeau et al., 2026), and GateScope (Lin et al., 2026) as compatible baselines.

Baseline implementation and fairness. We followed the original configurations: LLMmap uses seven probes (excluding one harmful probe), E5 embeddings, and a Transformer; FLIPS uses the 500-symbol {Open,mart} probe, Native/Fixed100 features, and XGBoost; GateScope uses four DCC probes, contrastive features, and One-vs-Rest XGBoost. For fairness, we recollected responses with probes and tasks in single requests and retrained all classifiers on probe responses, consistent with RMF's runtime setting.

Additional experiments. To assess dataset-specific adaptation, we applied RMF's filtering, search, and enrollment procedures to the baselines on PolicyQA. As shown in Table 1, these procedures yield limited and inconsistent gains. RMF achieves 70.17% average accuracy, outperforming the strongest adapted baseline (53.13%) by 17.04 percentage points, with substantially higher open-set unknown recognition (57.98% vs. 18.19%). Thus, dataset-specific adaptation alone cannot explain RMF's advantage.

Table 1. Impact of Dataset-Specific Adaptation on Baseline Performance

MethodClosed-set Acc.Open-set CCROpen-set URROpen-set Acc.Average
LLMmap68.07±4.9471.57±1.5517.97±6.5660.85±1.6354.62±3.67
LLMmap + Adaptation67.56±5.3772.24±3.2512.44±6.2160.28±3.1253.13±2.06
FLIPS45.57±1.5847.59±1.429.62±2.0340.00±1.1335.70±1.54
FLIPS + Adaptation47.08±3.4250.19±2.866.62±2.3441.48±2.1436.34±1.88
GateScope59.55±2.4862.00±1.2312.50±2.7252.10±1.3246.54±1.94
GateScope + Adaptation60.57±2.7857.33±3.3818.19±4.6849.50±2.2346.40±1.47
RMF79.75±2.4872.97±2.8957.98±4.5969.97±1.8270.17±2.95

R2 (W2)

Why Runtime Fingerprinting? Existing methods verify model identity and execute actual tasks in separate API requests. Even without recognizing probes, a provider could route simple tasks to the advertised model and complex ones to cheaper substitutes. Thus, successful fingerprint verification cannot guarantee the model used for subsequent tasks. Therefore, runtime fingerprinting is essential to link model verification with task execution.

Adversarial Probe Detection and Separation. We acknowledge that providers may attempt to separate probes from tasks. We consider a strong adversary that knows our ranking-based probe design and has fingerprint instances for supervised training. We evaluate Generic rule, Representation model, and LLM detector using probe segmentation accuracy (percentage of spans with IoU≥0.5\mathrm{IoU}\geq 0.5), fingerprint verification pass rate, and main-task performance change relative to the 87.4% baseline (Table 2).

Table 2. Evaluation of Adversarial Probe Detection and Separation

MethodProbe Segmentation AccuracyFingerprint Verification Pass RateMain-task Performance Change
Generic rule42.8%0.0%−82.7 pp
Representation model50.3%6.0%−51.2 pp
LLM detector80.8%100.0%−0.5 pp

The results show that (1) lightweight detectors struggle with separation: Generic rule and Representation model achieve only 42.8% and 50.3% segmentation accuracy, with 0% and 6% fingerprint verification rates. Inaccurate segmentation also damages task performance. (2) LLM-based detection is effective but costly: It achieves 80.8% segmentation accuracy and 100% fingerprint verification with minimal task degradation, but increases first-token latency by 2.06× (2.72s to 5.59s) and cost by 2.86× (2.70 to 7.71, USD × 10⁻⁴), excluding fingerprint verification. Thus, task concatenation increases the difficulty and cost of adversarial separation.

Impact of Task Concatenation. We tested a representation-based detector on LLMmap probes, finding that classification accuracy drops from 66.7% to 40.5% after task concatenation, confirming increased detection difficulty.

In practice, RMF does not assume probes are undetectable. Instead, it provides a runtime fingerprinting framework that binds verification to task execution. Continuously updating the fingerprint library can further hinder targeted probe detection and separation.

R3 (W3)

We will improve the readability of Figure 2, especially panel (b), and revise overly strong claims to ensure more precise wording.

Which LLM are you using? It seems like running sufficiently small LLM detectors would have negligible cost to providers.

We used GPT-5.6-Luna for probe segmentation, which incurs relatively high costs. We will further evaluate smaller LLM detectors and report their accuracy, latency, and costs during this discussion. We also highlight that, although cheaper models may reduce attack costs, the additional detection step still introduces latency overhead.

Reviewer_MvVEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses API model substitution, where a gateway serves a cheaper model than the one the client pays for. Pre-deployment fingerprinting sends dedicated queries before the real task, so a gateway could route only those queries to the advertised model. RMF avoids this by appending a few fingerprint probes to the real task prompt, so that the probes and the task go through the same routing decision in one request. Each probe asks the model to rank a small set of candidates with no correct answer (e.g., colors). The returned ranking is converted to pairwise binary features, and the per-model preference probabilities form a reference fingerprint. A factorization machine selects a probe bundle for each task domain from a pool of 20 probes. At runtime, the observed fingerprint is matched to the reference fingerprints by a cross-entropy-style cost (Eq. 3). On four benchmarks (PolicyQA, MBPP+, MATH-500, PubMedQA) and 23 LLMs, RMF improves over LLMmap, FLIPS, and GateScope in closed-set, open-set, and binary identification, with about 408 extra tokens per request.

Critical strengths
  • The problem is practical, and the argument against pre-deployment verification (Sec. 1, 2) is clear. Verifying the model inside the task request is a sensible direction.
  • RMF needs only text input and output. Table 2 shows that it adds fewer tokens than the baselines.
  • The evaluation covers fine-grained (20 models) and coarse-grained (8 families) multiclass settings, open-set rejection, and binary verification. The mean binary AUROC is 0.949 against 0.865 for the strongest baseline (Sec. 4.3).
  • The substitution experiment in Fig. 7(b) is close to the intended use case, and an anonymous code package is provided.
Critical weaknesses
  1. No evaluation against a provider that tries to defeat the method. The threat model is a malicious provider, yet no experiment tests one. The main argument for RMF over pre-deployment verification (Sec. 2: "identity evidence is generated under the same routing decision as the task answer") holds only if the provider cannot tell the probe from the task. The probes are fixed-format ranking questions from a pool of 20, and the selected bundle is fixed for all enrollment and online evaluation (Sec. 3.3, 3.4). Such probes seem easy to detect. A provider that detects them could send the probe part to the advertised model and the task part to a cheaper one, or strip the probes. The authors should at least test whether a simple detector (rules, a classifier trained on the probe pool, or an LLM asked to find injected ranking questions) can find the probes in task prompts. They should also report identification performance when the provider answers the probes with the advertised model and routes the rest elsewhere. If the probes are detectable, the consequences should be discussed.

  2. The source of the improvement is unclear. RMF introduces four things at once: ranking probes with no unique answer, probe filtering by consistency and separability, FM-guided bundle search per domain, and matching with probability vectors. There is no ablation. The baselines do not get any domain adaptation (probe search, filtering, or enrollment on a same-domain task). The gain in Table 1 may therefore come from domain adaptation rather than from ranking probes. I would like to see (a) RMF with random or fixed bundles instead of FM search, and (b) the baselines' probes with the same filtering, search, and enrollment procedure.

  3. It is unclear whether the identification accuracy is sufficient for per-request verification. The method targets verification of each request, but at k=1 the accuracy in Fig. 7(a) is about 76% (closed-set), 66% (open-set), and 85% (binary). A detector that errs on 15% to 34% of single requests is hard to use as evidence of fraud. Higher accuracy requires averaging over several requests (k up to 5), which raises the number of requests and the cost. The binary results are reported mainly as AUROC, and no operating point is given. Please report the detection rate at fixed low false-positive rates for each k, and the number of requests needed to reach a usable operating point.

  4. The claim of no task degradation rests on a single p-value. The abstract and Sec. 4.4 state that there is no statistically significant degradation, based on p = 0.4307. A non-significant result does not show that the difference is small. The test, the sample size, and whether the 16 task-model pairs were pooled are not stated. If they were pooled, gains and losses can cancel. In Fig. 4, Gemini on MATH-500 drops from 90.8 to 86.0, and GPT on PubMedQA from 70.0 to 67.3. Please report the per-pair paired differences with confidence intervals, and an equivalence test (e.g., TOST) with a margin chosen in advance.

Questions for authors
  1. Which models were used as "representative models" for probe filtering (Sec. 3.2) and FM training (Sec. 3.3)? Are the unknown models in the open-set evaluation excluded from both?
  2. In Fig. 7(b), the confidence for GPT-5.6-Luna seems to fall much less than for DeepSeek-V4 or GPT-5.2. With the calibrated threshold, what fraction of the Luna requests are flagged as substitution?
  3. Are the probes placed at the end of the prompt, or can they appear elsewhere? How often does a model ignore the probe or return a ranking that cannot be parsed, and how are such cases counted?
11 comments
AuthorsOpenReview ↗

Response to Reviewer MvVE, part 1/4

We sincerely thank the reviewer for the thoughtful and constructive feedback on the security, effectiveness, and evaluation of RMF. These comments have helped us strengthen our analysis and conduct additional experiments to better validate our method. Our detailed responses follow.

R1 (W1)

Why Runtime Fingerprinting? Existing methods verify model identity and execute actual tasks in separate API requests. Even without recognizing probes, a provider could route simple tasks to the advertised model and complex ones to cheaper substitutes. Thus, successful fingerprint verification cannot guarantee the model used for subsequent tasks. Therefore, runtime fingerprinting is essential to link model verification with task execution.

Adversarial Probe Detection and Separation. We acknowledge that providers may attempt to separate probes from tasks. We consider a strong adversary that knows our ranking-based probe design and has fingerprint instances for supervised training. We evaluate Generic rule, Representation model, and LLM detector using probe segmentation accuracy (percentage of spans with IoU≥0.5\mathrm{IoU}\geq 0.5), fingerprint verification pass rate, and main-task performance change relative to the 87.4% baseline (Table 1).

Table 1. Evaluation of Adversarial Probe Detection and Separation

MethodProbe Segmentation AccuracyFingerprint Verification Pass RateMain-task Performance ChangeFirst-token LatencyCost
Generic rule42.8%0.0%−82.7 pp——
Representation model50.3%6.0%−51.2 pp——
GPT-5.6-Luna80.8%100.0%−0.5 pp2.06×2.86×
GPT-6-Luna76.5%98.0%−0.5 pp2.67×1.49×
Qwen3.7-Flash74.5%84.0%−2.5 pp1.52×1.22×

The results show that (1) lightweight detectors struggle with separation: Generic rule and Representation model achieve only 42.8% and 50.3% segmentation accuracy, with 0% and 6% fingerprint verification rates. Inaccurate segmentation also damages task performance. (2) LLM-based detection is effective but introduces latency and cost overhead: It achieves 80.8% segmentation accuracy and 100% fingerprint verification with minimal task degradation, but increases first-token latency by 2.06× (2.72s to 5.59s) and cost by 2.86× (2.70 to 7.71, USD × 10⁻⁴), excluding fingerprint verification. Thus, task concatenation increases the difficulty and cost of adversarial separation.

Impact of Task Concatenation. We tested a representation-based detector on LLMmap probes, finding that classification accuracy drops from 66.7% to 40.5% after task concatenation, confirming increased detection difficulty.

In practice, RMF does not assume probes are undetectable. Instead, it provides a runtime fingerprinting framework that binds verification to task execution. Continuously updating the fingerprint library can further hinder targeted probe detection and separation.

AuthorsOpenReview ↗

Response to Reviewer MvVE, part 2/4

R2(W2)

Table 2. Ablation of Probe Bundle Search

MethodClosed-set Acc.Open-set CCROpen-set URROpen-set Acc.Average
RMF (Random Search)73.50±1.5972.89±2.6118.89±2.9462.09±2.3656.84±2.05
RMF (FM Search)79.75±2.4872.97±2.8957.98±4.5969.97±1.8270.17±2.95

Table 3. Ablation of Baseline Probe Selection and Enrollment

MethodClosed-set Acc.Open-set CCROpen-set URROpen-set Acc.Average
LLMmap68.07±4.9471.57±1.5517.97±6.5660.85±1.6354.62±3.67
LLMmap + RMF Filtering, Search & Enrollment67.56±5.3772.24±3.2512.44±6.2160.28±3.1253.13±2.06
FLIPS45.57±1.5847.59±1.429.62±2.0340.00±1.1335.70±1.54
FLIPS + RMF Filtering, Search & Enrollment47.08±3.4250.19±2.866.62±2.3441.48±2.1436.34±1.88
GateScope59.55±2.4862.00±1.2312.50±2.7252.10±1.3246.54±1.94
GateScope + RMF Filtering, Search & Enrollment60.57±2.7857.33±3.3818.19±4.6849.50±2.2346.40±1.47

Table 4. Ablation of Probe Task Design

MethodClosed-set Acc.Open-set CCROpen-set URROpen-set Acc.Average
RMF (Short QA)65.50±5.7669.54±3.2112.22±6.2658.08±2.6051.34±2.35
RMF (Ranking)79.75±2.4872.97±2.8957.98±4.5969.97±1.8270.17±2.95

We agree that further ablation studies would help clarify the sources of RMF's performance gains. We therefore conducted three additional experiments on PolicyQA.

(a) Ablation of Probe Bundle Search. We replaced FM-guided search with random search while keeping the remaining pipeline unchanged. As shown in Table 2, FM-guided search improves average accuracy from 56.84% to 70.17% (+13.33 pp), confirming the contribution of our search strategy.

(b) Ablation of Baseline Probe Selection and Enrollment. We applied RMF's filtering, search, and enrollment procedures to the baseline probes. As shown in Table 3, these procedures improve some metrics but yield limited overall gains. RMF achieves 70.17% average accuracy, outperforming the strongest baseline under the same procedures (53.13%) by 17.04 pp. This shows that the performance gap cannot be explained solely by differences in probe selection and enrollment.

(c) Ablation of Probe Task Design. We further replaced ranking probes with short question-answering probes, retaining the remaining pipeline where applicable. As shown in Table 4, the ranking-based design improves average accuracy from 51.34% to 70.17% (+18.83 pp), demonstrating the effectiveness of our probe design.

Together, these results demonstrate the contributions of FM-guided search and ranking-based probe design, while confirming that RMF maintains its advantage when baselines receive the same probe selection and enrollment procedures, further validating the effectiveness of our method.

AuthorsOpenReview ↗

Response to Reviewer MvVE, part 3/4

R3(W3)

Identifying model substitution from a single API request is inherently challenging, and we acknowledge that our single-request performance can be further improved. In practice, binary verification (whether the API serves the advertised model) is often more important. We would like to highlight that RMF achieves 84.90% balanced accuracy (BA) in single-request binary verification, which could provide useful evidence of model identity and substantially outperforms the strongest baseline (73.49%).

In practical API usage, multiple requests often occur within the same workflow or session, allowing identification evidence to be accumulated across requests. To simulate this scenario, we treated evaluation queries as a request stream and aggregated fingerprint evidence from consecutive requests for a joint verification decision. Unlike Fig. 7(a), which examines repeated executions of the same query, this experiment uses distinct queries to better reflect practical usage. The table reports the average substitution detection rate (recall) across four models at a fixed FPR of at most 1%.

The results show that aggregating four distinct requests achieves 92.48% recall at FPR ≤ 1%, increasing to 95.02% with five requests and 98.02% with nine. These results demonstrate that aggregating evidence across a small number of requests can substantially improve verification reliability. We acknowledge that verification remains more challenging for occasional users with very few requests and leave further improvements in such cold-start scenarios for future work.

Table 5. Substitution Detection Rate at FPR ≤ 1% Across Distinct Requests

Number of distinct requests (kk)23456789
Average Recall @ FPR ≤ 1%70.38%84.80%92.48%95.02%95.73%97.45%97.80%98.02%

R4(W4)

The performance decreases in individual model–task pairs may arise from random fluctuations rather than systematic degradation, as some pairs also show improvements (e.g., Gemini on PubMedQA increases from 74.7% to 78.0%). In our experiments, each model was evaluated on 378 MBPP+, 500 MATH-500, 300 PubMedQA, and 300 PolicyQA questions, with responses paired on the same questions. We computed the performance difference for each of the 16 model–task pairs and averaged them with equal weight, obtaining −0.43 percentage points (pp). The reported p = 0.4307 comes from a two-sided exact sign-permutation test over these 16 differences, rather than pooling individual questions. The overall 95% CI [−1.46, +0.54] pp was obtained by bootstrapping the 16 differences 10,000 times.

Per-Pair Differences and Equivalence Test. We report the paired performance differences and 95% CIs for all 16 model–task pairs in Table X. All intervals include zero, suggesting that the observed changes may be influenced by random fluctuations. We further conducted a two one-sided tests (TOST) equivalence test with a ±2 pp margin on the equally weighted average difference. The resulting 90% CI [−1.37, +0.50] pp lies entirely within this margin (p = 0.0050), supporting equivalence in overall average task performance.

Additional Repeated Experiments. We repeated each request 10 times for Gemini–MATH-500 and GPT–PubMedQA. Their average performance changes were −0.32 pp (95% CI: [−6.83, +6.19]) and −0.37 pp (95% CI: [−3.81, +4.32]), respectively, suggesting that the original larger decreases may have been influenced by random fluctuations.

Table 6. Per-Pair Task Performance Differences with 95% Confidence Intervals

ModelTaskDifference (pp)Paired 95% CI (pp)
GeminiMBPP++0.79[−2.65, +4.23]
GeminiMATH-500−4.80[−10.40, +0.82]
GeminiPubMedQA+3.33[−1.00, +7.67]
GeminiPolicyQA+1.42[−1.25, +4.10]
DeepSeekMBPP++1.06[−2.65, +4.76]
DeepSeekMATH-500−1.20[−3.40, +1.00]
DeepSeekPubMedQA+0.67[−4.00, +5.33]
DeepSeekPolicyQA−2.10[−5.41, +1.19]
GPTMBPP++1.32[−1.59, +4.23]
GPTMATH-500−0.60[−3.00, +1.80]
GPTPubMedQA−2.67[−7.00, +1.33]
GPTPolicyQA−1.61[−3.68, +0.49]
QwenMBPP+−0.53[−3.44, +2.65]
QwenMATH-500+2.20[−0.20, +4.80]
QwenPubMedQA−1.33[−5.33, +2.67]
QwenPolicyQA−2.86[−7.39, +1.67]
Reviewer_JANWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a new fingerprinting setting where the fingerprinting probes are concatenated with real task and submitted in a single API request. The underlying goal is to operate in real-time. The authors suggest that it avoids the backend model to be changed after verification or selectively substituted when prompted non-fingerprint requests. They develop a method that effectively implements such task-coupled protocol and show that it minimizes the overhead while keeping strong identification performance.

Critical strengths

The authors do a very good job posing the problem of designing fingerprinting schemes that work well at runtime to avoid model replacement between identification and real work usage. The idea of designing fingerprinting schemes that are embedded in task requests is also very interesting and promising. Moreover, the identification scheme relies on a small number parameters. It is not exactly training free because of the FM-guided part and threshold calibration part, but does not rely on training a classifier for identification, which makes the method particularly interpretable. The metrics of performance and token usage are well evaluated and compared across several state of the art methods. The compact representation of the probes, the adaptive scheme and pairwise preference representation are very effective design choices.

Critical weaknesses
  1. If I understand correctly, the fingerprinting probe and the task are concatenated and the main motivation for that is that at runtime the provider cannot replace the advertised model by another cheaper model. This is supposed to be a guarantee that the fingerprinted model is also the model that is executed for the task. However, if a provider can make the difference between an non-fingerprinting and fingerprinting request, what makes them enable to separate an API request in two parts (resp. fingerprinting and non fingerprinting) and route each one to separate models (the advertised model and a cheaper one for the main task)? What guarantee do we have that the model answering the main task is the fingerprinted model ? 2. The paper gives a method to find a good probe set for a given task domain, but it does not say how a given real-world request is assigned to a task domain ? 3. Less critical : Fairness in the comparison with other fingerprinting models (in terms of performance) : the authors compare to other fingerprinting methods using the same fingerprinting+task concatenated setting. However, some of the methods, e.g. LLMmap, rely on training a classifier for model identification : shouldn’t one retrain the baselines in this setting for a fair comparison?

minor remark: Minor : I think the acronym FM (factorisation machine) is never explicitely defined

Questions for authors

Questions For Authors* • If a provider can make the difference between an non-fingerprinting and fingerpringing request, what makes them enable to separate an API request in two parts (resp. fingerprinting and non fingerprinting) and route each one to separate models (the advertised model and a cheaper one for the main task)? • In a real work deployment scenario, how would a given real-world request is assigned to a task domain ? • Could you include the following paper in your related work section, as it seems to me that it also proposes runtime fingerprinting : LLMs Have Rhythm: Fingerprinting Large Language Models Using Inter-Token Times and Network Traffic Analysis. Saeif Alhazbi*, Ahmed Hussain, Gabriele Oligeri, Panos Papadimitratos

6 comments

We thank the reviewer for the feedback on runtime security, task domains, baseline fairness, and related work. Our responses follow.

R1 (W1, Q1)

Why Runtime Fingerprinting? Existing methods verify model identity and execute actual tasks in separate API requests. Even without recognizing probes, a provider could route simple tasks to the advertised model and complex ones to cheaper substitutes. Thus, successful fingerprint verification cannot guarantee the model used for subsequent tasks. Therefore, runtime fingerprinting is essential to link model verification with task execution.

Adversarial Probe Detection and Separation. We acknowledge that providers may attempt to separate probes from tasks. We consider a strong adversary that knows our ranking-based probe design and has fingerprint instances for supervised training. We evaluate Generic rule, Representation model, and LLM detector using probe segmentation accuracy (percentage of spans with IoU≥0.5\mathrm{IoU}\geq 0.5), fingerprint verification pass rate, and main-task performance change relative to the 87.4% baseline (Table 2).

Table 2. Evaluation of Adversarial Probe Detection and Separation

MethodProbe Segmentation AccuracyFingerprint Verification Pass RateMain-task Performance Change
Generic rule42.8%0.0%−82.7 pp
Representation model50.3%6.0%−51.2 pp
LLM detector80.8%100.0%−0.5 pp

The results show that (1) lightweight detectors struggle with separation: Generic rule and Representation model achieve only 42.8% and 50.3% segmentation accuracy, with 0% and 6% fingerprint verification rates. Inaccurate segmentation also damages task performance. (2) LLM-based detection is effective but costly: It achieves 80.8% segmentation accuracy and 100% fingerprint verification with minimal task degradation, but increases first-token latency by 2.06× (2.72s to 5.59s) and cost by 2.86× (2.70 to 7.71, USD × 10⁻⁴), excluding fingerprint verification. Thus, task concatenation increases the difficulty and cost of adversarial separation.

Impact of Task Concatenation. We tested a representation-based detector on LLMmap probes, finding that classification accuracy drops from 66.7% to 40.5% after task concatenation, confirming increased detection difficulty.

In practice, RMF does not assume probes are undetectable. Instead, it provides a runtime fingerprinting framework that binds verification to task execution. Continuously updating the fingerprint library can further hinder targeted probe detection and separation.

R2 (W2, Q2)

In our work, we consider coarse-grained task domains, such as mathematics, coding, medicine, and public policy, making it easier to categorize user requests. Our experiments have demonstrated that such coarse-grained domain classification is sufficient for effective model identification.

In practice, requests from the same source often have relatively predictable task distributions, making it easy to define appropriate domains. Furthermore, even when a request belongs to multiple domains, our method remains robust to domain assignment. For example, using mathematics-specific and coding-specific fingerprints on the same math-related coding tasks yields comparable closed-set classification accuracies of 76.18% and 75.29%, respectively, both outperforming random probes (64.71%). These results show that either relevant domain can achieve effective model identification.

R3 (W3)

We agree that a fair comparison requires all methods to be trained and evaluated under the same fingerprinting-plus-task setting. In fact, in our experiments, we recollected model responses for each baseline by combining its probes with real tasks in a single request, following the same setting as RMF. We then used the probe responses to retrain the corresponding classifiers. Thus, all baselines were trained and evaluated under identical conditions, further demonstrating the effectiveness of RMF.

R4 (W4)

We will explicitly define FM as "factorization machine" at its first occurrence in the revised manuscript.

R5 (Q3)

We will include this work in the Related Work section. It proposes a passive fingerprinting method that identifies LLMs by analyzing inter-token arrival times and network traffic patterns, such as packet sizes. However, its applicability to third-party API platforms has two limitations. First, intermediate gateways may introduce variations in traffic patterns through proxying, buffering, and response transmission mechanisms, interfering with the original timing characteristics. Second, the approach is vulnerable to simple evasion: an intermediary can introduce artificial delays to distort inter-token timing features, potentially rendering fingerprinting ineffective. We are conducting additional experiments to investigate these limitations.

R5 (Q3)

We further evaluated this limitation by training a traffic-based classifier on 100 DeepSeek-V4-Flash responses and 100 negative samples from official APIs, then testing it on 60 responses from each class via a third-party platform. Accuracy was only 56.7%, approaching random guessing, suggesting that third-party gateways can disrupt traffic-based fingerprints and undermine passive fingerprinting reliability.

We sincerely thank Reviewer JANW once again for the constructive feedback. We would appreciate it if you could let us know whether our responses and additional experiments have addressed your concerns. We would be happy to provide further clarification if needed.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Initial Reviews and author initial response, part 1/3

We sincerely thank the Area Chair and all three reviewers for their careful evaluation and constructive feedback, which have helped us identify important limitations and strengthen our work. We have carefully addressed all reviewer comments and conducted extensive additional experiments. Below, we summarize three major concerns and our corresponding responses.

### 1. Necessity of Runtime Fingerprinting and Robustness to Probe Separation (JANW-W1, MvVE-W1, HWEW-W2)

Motivation for Runtime Fingerprinting. Existing fingerprinting methods verify model identity and execute actual tasks in separate API requests. Even without recognizing fingerprinting probes, a provider could route simple requests to the advertised model and complex requests to cheaper substitutes. Therefore, successful pre-deployment verification cannot guarantee the model used for subsequent tasks. RMF addresses this limitation by binding fingerprinting probes and actual tasks within a single API request, directly associating model verification with task execution.

Adversarial Probe Detection and Separation. We acknowledge that a malicious provider may attempt to separate probes from actual tasks. To evaluate this risk, we considered a strong adversary that knows our ranking-based probe design and has access to fingerprint instances for supervised training. We evaluated rule-based, representation-based, and LLM-based detectors, measuring probe segmentation accuracy (percentage of spans with IoU ≥ 0.5), fingerprint verification pass rate after segmentation, and main-task performance change relative to the 87.4% baseline.

Table 1. Evaluation of Adversarial Probe Detection and Separation

MethodSegmentation Acc.Verification Pass RateTask ChangeFirst-token LatencyCost
Generic rule42.8%0.0%−82.7 pp——
Representation model50.3%6.0%−51.2 pp——
GPT-5.6-Luna80.8%100.0%−0.5 pp2.06×2.86×
GPT-6-Luna76.5%98.0%−0.5 pp2.67×1.49×
Qwen3.7-Flash74.5%84.0%−2.5 pp1.52×1.22×

The results show that (1) Lightweight detectors struggle with separation: Generic rule and Representation model achieve only 42.8% and 50.3% segmentation accuracy, with 0% and 6% verification pass rates, substantially degrading task performance. Additionally, concatenating probes with real tasks reduces representation-based detection accuracy from 66.7% to 40.5%. (2) LLM-based detection is effective but introduces latency and cost overhead: Although more effective, it increases first-token latency by 1.52×–2.67× and cost by 1.22×–2.86×. Thus, task concatenation increases the difficulty and cost of adversarial separation.

In practice, RMF does not assume probes are undetectable. Instead, it provides a runtime fingerprinting framework that binds verification to task execution. Continuously updating the fingerprint library can further hinder targeted probe detection and separation.

AuthorsOpenReview ↗

Summary of reviews and author response, part 2/3

2. Component Contributions and Baseline Fairness (MvVE-W2, HWEW-W1, JANW-W3)

To clarify the sources of RMF's performance gains and ensure fair comparisons, we conducted three additional experiments on PolicyQA.

(a) Ablation of Probe Bundle Search. We replaced FM-guided search with random search while keeping the remaining pipeline unchanged. As shown in Table 2, FM-guided search improves average accuracy from 56.84% to 70.17% (+13.33 pp), confirming the contribution of our search strategy.

Table 2. Ablation of Probe Bundle Search

MethodClosed-set Acc.Open-set CCROpen-set URROpen-set Acc.Average
RMF (Random Search)73.50±1.5972.89±2.6118.89±2.9462.09±2.3656.84±2.05
RMF (FM Search)79.75±2.4872.97±2.8957.98±4.5969.97±1.8270.17±2.95

(b) Ablation of Baseline Probe Selection and Enrollment. We applied RMF's filtering, search, and enrollment procedures to the baseline probes. As shown in Table 3, these procedures improve certain metrics but yield limited overall gains. RMF still achieves 70.17% average accuracy, outperforming the strongest baseline under the same procedures (53.13%) by 17.04 pp. This indicates that differences in probe selection and enrollment alone cannot explain RMF's performance advantage.

Table 3. Ablation of Baseline Probe Selection and Enrollment

MethodClosed-set Acc.Open-set CCROpen-set URROpen-set Acc.Average
LLMmap68.07±4.9471.57±1.5517.97±6.5660.85±1.6354.62±3.67
LLMmap + RMF Filtering, Search & Enrollment67.56±5.3772.24±3.2512.44±6.2160.28±3.1253.13±2.06
FLIPS45.57±1.5847.59±1.429.62±2.0340.00±1.1335.70±1.54
FLIPS + RMF Filtering, Search & Enrollment47.08±3.4250.19±2.866.62±2.3441.48±2.1436.34±1.88
GateScope59.55±2.4862.00±1.2312.50±2.7252.10±1.3246.54±1.94
GateScope + RMF Filtering, Search & Enrollment60.57±2.7857.33±3.3818.19±4.6849.50±2.2346.40±1.47

(c) Ablation of Probe Task Design. We further replaced ranking probes with short question-answering probes, retaining the remaining pipeline where applicable. As shown in Table 4, the ranking-based fingerprint design improves average accuracy from 51.34% to 70.17% (+18.83 pp), demonstrating its effectiveness in capturing distinctive model behaviors.

Table 4. Ablation of Probe Task Design

MethodClosed-set Acc.Open-set CCROpen-set URROpen-set Acc.Average
RMF (Short QA)65.50±5.7669.54±3.2112.22±6.2658.08±2.6051.34±2.35
RMF (Ranking)79.75±2.4872.97±2.8957.98±4.5969.97±1.8270.17±2.95

Furthermore, all baseline classifiers were retrained using model responses collected with their probes and real tasks combined in single requests, matching RMF's runtime evaluation setting. Together, these results clarify the contributions of RMF's key designs and demonstrate that its performance advantage persists under fairer baseline comparisons.

AuthorsOpenReview ↗

Summary of reviews and author response, part 3/3

### 3. Practical Verification Reliability (MvVE-W3, Q2)

We acknowledge that identifying model substitution from a single API request is challenging and that our single-request performance can be further improved. In practical applications, binary verification determines whether the API serves the advertised model. RMF achieves 84.90% balanced accuracy in this setting, compared with 73.49% for the strongest baseline, providing potentially useful evidence of model identity.

In practical API usage, multiple requests often occur within the same workflow or session. To simulate this scenario, we treated evaluation queries as a request stream and aggregated fingerprint evidence from consecutive requests for joint verification. Unlike Fig. 7(a), which uses repeated executions of the same query, this experiment uses distinct queries. We report the average substitution detection recall across four models at a fixed false-positive rate (FPR) of at most 1%.

Table 5. Substitution Detection Rate at FPR ≤ 1% Across Distinct Requests

Number of distinct requests ((k))23456789
Average Recall @ FPR ≤ 1%70.38%84.80%92.48%95.02%95.73%97.45%97.80%98.02%

The results show that aggregating four distinct requests achieves 92.48% recall at FPR ≤ 1%, increasing to 95.02% with five requests and 98.02% with nine requests. Thus, accumulating evidence across a small number of requests substantially improves verification reliability, although single-request detection at a strict FPR remains challenging.

We sincerely appreciate the reviewers' detailed feedback, which has enabled us to substantially strengthen the experimental evidence and clarify the capabilities and limitations of RMF. We hope these additional experiments and explanations provide the Area Chair with a more complete basis for evaluating our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.