- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a fingeprinting method to identify LLMs being served behind an API. The proposed method is non-invasive, and aims to produce unique strings which trigger a particular response for an LLM and its derivative LLMs. These unique strings are from "low probabilty" prompt regions, which are unlikely to be contained in fine-tuning datasets. As a result, the behaviour of LLMs on these fingerprints is fairly stable even after fine-tuning. These fingerprints are constructed using discrete optimization techniques like GCG, with an optional perplexity regularizer. The technique is evaluated on Llama models, and the fine-tuning persistence as well as robustness to prompt templates, decoding time changes and quantization are also reported.
Critical strengths
- The theoretical results saying that behaviour on rare prompts would not be disrupted under standard fine-tuning is nice. The gradient analysis is non-trivial in my view and adds to the paper. However, I am unsure if the theory connects much to practice (see below).
- The evaluations seem thorough, though the adversarial evaluations (esp for the robustness are lacking, see below).
- The paper is very well written and easy to follow.
Critical weaknesses
My main concern is with the claimed robustness of the presented scheme
- Fig 8 seems to imply that the scheme has limited robustness, since ab adversary could easily filter out RAFP fingerprints which look like outliers on the pplx length curves? I am unsure if this takes into account the regularizers to generate more realistic fingerprints.
- It would be nice if the paper could clarify why RAFP fingerprint prompts have (a). I believe there is some kind of an impossiblity here-- if (a) is true, an adversary can easily filter out prompts which look unnatural based on perplexity (e.g. the attack on TRAP from Nasery et al 2026). To alleviate this, the present paper incorporates a perplexity loss in section 6. However, this means that the resulting fingerprints are then closer to fine-tuning distributions that an adversary might pick, which means that fine-tuning might lead to forgetting. It would strengthen the paper to characterize the tradeoffs introduced by the perplexity loss.
- I also do not believe that is a good assumption for adversarial fine-tuning. Indeed, Xu et al 2025 (MEraser) explicitly construct fine-tuning distributions which seem to overlap with certain fingeprinting distributions. In my view it should be fairly straightforward for an adaptive adversary to construct query distributions for fine-tuning to disrupt RAFP fingerprints.
- Minor issue --- Missing prior work -- Gubri et al (TRAP)
Questions for authors
- Were the results in Fig 8 generated with the pplx loss? What are the tradeoffs in robustness that this loss brings?
4 comments
I thank the authors for their rebuttal. The results with the PPL aware optimization of fingerprints have addressed some of my concerns and I am raising my score. I would also appreciate if the authors could provide more context around the newly conducted fine-tuning experiments --- how were the rare-region samples constructed? Was any GCG done on them?
Clarification on Adaptive Rare-Region Fine-Tuning and Summary of Manuscript Revisions
Dear Reviewer A8e2,
We thank the reviewer for the follow-up question. In our adaptive fine-tuning experiment, we construct rare-region training pairs using the initialization procedure of RAFP (Section 4.1). Specifically, we first generate rare prompts using bottom-k sampling (k=2000) and then sample corresponding target responses conditioned on those prompts using the same bottom-k strategy. We do not perform GCG optimization on these injected pairs.
Intuitively, these samples are intended to increase the downstream training distribution's coverage of rare regions, rather than serve as optimized fingerprints themselves. They are generated independently of the secret verification fingerprints. This experiment therefore evaluates whether greater rare-region exposure alone can weaken fingerprint persistence.
Additional manuscript revisions
We have also incorporated the following updates into the revised paper, with all changes highlighted in blue:
-
Adaptive fine-tuning evaluation (Section 6 Ablation, Table 4): We add the new experiments in the revised paper
-
Perplexity-filtering evaluation (Section 6, Table 3a): We extend the comparison to include random triggers and GCG-QA with and without PPL regularization. The revised table reports fingerprint matching rates after applying the PPL filter, providing a clearer evaluation of the tradeoff between filter acceptance and fingerprint persistence.
-
Additional related work (Section 2): We cite and mention TRAP (Gubri et al., 2024) and MEraser (Zhang et al., 2025) in our related work. We thank the reviewer for pointing out these two meaningful related works.
We sincerely thank the reviewer for the feedback and for reconsidering the score. Your suggestion helped us strengthen the paper.
Thanks for the clarification. In this case, for a more thorough adaptive attack I believe the attacker should match the fingerprint distribution, i.e. perform some steps of GCG on top of the rare region init. However I do not expect this to change the results much.
Addressing PPL-Filtering Vulnerability and Adaptive Fingerprint Erasure with New Robustness Evaluations
W1, W2, Q1: Perplexity filtering and tradeoff of fingerprint persistence
We thank the reviewer for highlighting the tradeoff between filtering resistance and fingerprint persistence.
We clarify that Figure 8 was generated using RAFP fingerprints without PPL regularization (λ=0). The figure illustrates that unregularized rare-region fingerprints can be distinguished from natural prompts using perplexity, motivating our ablation PPL-aware optimization shown in Section 6.
We clarify that low probability under the downstream fine-tuning distribution does not necessarily imply rejection by a practical PPL filter. Our PPL-aware optimization aims to balance these competing objectives rather than eliminate the tradeoff entirely.
To further investigate this tradeoff, we extend our evaluation to conventional GCG-QA fingerprints, with and without PPL regularization, using the same filtering conditions. As shown in our additional results (Table 1), we report both matching accuracy among fingerprints surviving the filter on both base model and SFT model.
Importantly, we discover this tradeoff is not unique to RAFP only, since conventional GCG-based methods may also produce high-perplexity trigger strings. Our additional comparison evaluates whether RAFP-PPL (λ=0.3) offers a more favorable balance between filter acceptance and persistence than GCG-QA under equivalent PPL constraints.
Table 1: Fingerprint robustness under perplexity filtering (%).
- Values are percentages (counts). PPL rejection is rejected/total; Base/SFT matching is successful/surviving total.
We will clarify that RAFP does not guarantee resistance to arbitrary input filtering; rather, RAFP seeks persistent fingerprint behaviors under practical filtering constraints.
W3: Robustness against adaptive fine-tuning.
We thank the reviewer for pointing out the paper Xu et al. 2025 (MEraser). We agree that an adaptive adversary could intentionally manipulate the fine-tuning distribution to increase exposure to rare-region prompts, further violating our assumption 1. However, even if an adversary knows RAFP's construction strategy, it is still hard for he/she to guess or access the full secret fingerprint pairs. In our appendix A.8, Unforgeable property, we show the probability that the adversary model erases all fingerprint prompts is very small. Even to achieve a 1% probability of erasing the fingerprints, the adversary would need to generate N = 2.71e105 query-response pairs.
Following the reviewer's suggestion, we evaluate this threat under a realistic downstream adaptive finetuning setting, we use the full ShareGPT fine-tuning dataset (50k samples) and inject rare-region samples at ratios of 1%, 2%, 5%, and 10%. These samples follow RAFP's rare-token initialization strategy but do not contain the secret verification fingerprints. In Table 2, we compare GCG-QA, RAFP, and RAFP-PPL on the resulting fine-tuned checkpoints. This experiment shows that injecting a small fraction of rare-region samples only slightly disrupts RAFP fingerprints.
Table 2: Fingerprint persistence under adaptive fine-tuning w/ rare-data injection (%).
- Fingerprint matching rates use 10 fingerprints per method. Utility retention is normalized to the standard-SFT checkpoint. Rare-data injection ratio refers to the final training dataset size (50k).
W4: Missing related work
We thank the reviewer for identifying the missing citation to Gubri et al.'s TRAP. We will incorporate TRAP into the related work discussion and further discuss MEraser as an adaptive fingerprint erasure method. These works provide important context for distinguishing ordinary downstream fingerprint persistence from robustness against deliberate removal.