- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper suggests to embedd fingerprint queries into task prompts to detect which LLM an API provider is serving. This could enable detecting fraudulent API providers. The prompts work by having LLMs rank different options as part of the task, which the authos show to uncover LLM specific characterstics. Multiple ranking tasks are combined to a probe bundle for increase detection efficiency, with selection done by via optimizting a second-order approximation of a probe bundle's utility. The method is reported to consistently outperform baselines in terms of both model detection and token overhead.
Critical strengths
- The proposed method overall performs well
- The paper performs multiple ablations on the method
Critical weaknesses
- Very little information about the implemented baselines is given, and it is not really clear, how much of the performance gap can be explained by the method being tuned to the datasets at hand, why the baselines weren't. The choice of baselines is also a bit unclear, with very recent methods cited in the related work like Hu et al 2026 being omitted.
- It seems very easy for an attacker to circumvent the proposed probes, as they are embedded in the task prompt, but are functionally completely separate.
- Minor: Presentation could still be improved at some parts. For example, figure 2 is hard to read without a lot of zooming, especially part b). Some claims like "Our method achieves the optimal performance." also seem a bit strong
Questions for authors
- How were the baselines selected and tuned?
8 comments
Which LLM are you using? It seems like running sufficiently small LLM detectors would have negligible cost to providers.
We used GPT-5.6-Luna for probe segmentation, which incurs relatively high costs. We will further evaluate smaller LLM detectors and report their accuracy, latency, and costs during this discussion. We also highlight that, although cheaper models may reduce attack costs, the additional detection step still introduces latency overhead.
We thank the reviewer for the constructive feedback on baseline fairness, runtime security, and presentation. Our responses and additional results follow.
R1 (W1, Q1)
Baseline selection. We reviewed 2025–2026 methods for single-request, black-box model verification. We excluded RUT (Zhu et al., 2026) and LLMPrint (Hu et al., 2026) due to their white-box reference/base-model requirements, and MET (Gao et al., 2025) due to repeated distributional testing. We selected LLMmap (Pasquini et al., 2025), FLIPS (Richardeau et al., 2026), and GateScope (Lin et al., 2026) as compatible baselines.
Baseline implementation and fairness. We followed the original configurations: LLMmap uses seven probes (excluding one harmful probe), E5 embeddings, and a Transformer; FLIPS uses the 500-symbol
{Open,mart}probe, Native/Fixed100 features, and XGBoost; GateScope uses four DCC probes, contrastive features, and One-vs-Rest XGBoost. For fairness, we recollected responses with probes and tasks in single requests and retrained all classifiers on probe responses, consistent with RMF's runtime setting.Additional experiments. To assess dataset-specific adaptation, we applied RMF's filtering, search, and enrollment procedures to the baselines on PolicyQA. As shown in Table 1, these procedures yield limited and inconsistent gains. RMF achieves 70.17% average accuracy, outperforming the strongest adapted baseline (53.13%) by 17.04 percentage points, with substantially higher open-set unknown recognition (57.98% vs. 18.19%). Thus, dataset-specific adaptation alone cannot explain RMF's advantage.
Table 1. Impact of Dataset-Specific Adaptation on Baseline Performance
R2 (W2)
Why Runtime Fingerprinting? Existing methods verify model identity and execute actual tasks in separate API requests. Even without recognizing probes, a provider could route simple tasks to the advertised model and complex ones to cheaper substitutes. Thus, successful fingerprint verification cannot guarantee the model used for subsequent tasks. Therefore, runtime fingerprinting is essential to link model verification with task execution.
Adversarial Probe Detection and Separation. We acknowledge that providers may attempt to separate probes from tasks. We consider a strong adversary that knows our ranking-based probe design and has fingerprint instances for supervised training. We evaluate Generic rule, Representation model, and LLM detector using probe segmentation accuracy (percentage of spans with IoU≥0.5), fingerprint verification pass rate, and main-task performance change relative to the 87.4% baseline (Table 2).
Table 2. Evaluation of Adversarial Probe Detection and Separation
The results show that (1) lightweight detectors struggle with separation: Generic rule and Representation model achieve only 42.8% and 50.3% segmentation accuracy, with 0% and 6% fingerprint verification rates. Inaccurate segmentation also damages task performance. (2) LLM-based detection is effective but costly: It achieves 80.8% segmentation accuracy and 100% fingerprint verification with minimal task degradation, but increases first-token latency by 2.06× (2.72s to 5.59s) and cost by 2.86× (2.70 to 7.71, USD × 10⁻⁴), excluding fingerprint verification. Thus, task concatenation increases the difficulty and cost of adversarial separation.
Impact of Task Concatenation. We tested a representation-based detector on LLMmap probes, finding that classification accuracy drops from 66.7% to 40.5% after task concatenation, confirming increased detection difficulty.
In practice, RMF does not assume probes are undetectable. Instead, it provides a runtime fingerprinting framework that binds verification to task execution. Continuously updating the fingerprint library can further hinder targeted probe detection and separation.
R3 (W3)
We will improve the readability of Figure 2, especially panel (b), and revise overly strong claims to ensure more precise wording.