- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a new setting that we can use dynamic black-box provenance to verify which large language model generates the response, rather than the existing white-box (need access to the LLM parameters, gradients or logits) or static black-box (auditors must send a fixed shared prompt panel to the models, which is really expensive in time and moeny cost, and missing the interactiction between users and the LLMs) settings. The author proposes to firstly use a frozen proxy LLM to do the DCT extraction, then use the extracted features to train a simple linear classifier, then accumulate the log probes to calculate the average score. The experimental results shows that the performance is both fast and accurate.
Critical strengths
-
The problem setting is quite straightforward to follow, previous work focus on static panel, which is not realistic in the real work deployment, then move to a more realistic scenario, that interact with the user logging.
-
The method design is elegant and lightweight to build, when using the DCT to extract DC and AC, the computational complexity is only , which is realy fast.
-
Extensive experiments that covers around 100 models, with 8 different proxy LLMs, and the results hold across various task, domain, and token length.
Critical weaknesses
-
From Table 2, I find that the domain shift really breaks the performance in this field, the performance drops to 33% from 96.2% when transfering to the Math100, even though it is enough better than other baselines. However, in the real world application (since the primary area allocated as societal consideration), whether it means that if the collected prompt distribution does not cover the real domain of the user query, the auditing will always fail, is ther any way can aviod or mitigate this situation.
-
The most critical one is that in the Eq.8, it assums that model responses are conditionally independent given the source LLM and the observed prompts, which may sounds reasonable in the single-turn queries, but it may not hold in realistic multi-turn or agent based settings. It is worthy to discuss what is the scope of the assumption, and what if the real case is adaptive, multi-turn interactions.
Questions for authors
- I find that in the model families, there are lots of reasoning models (Qwen3-4B-Thinking, DeepSeek-R1-Distill), but I cannot find whether the contents contains the reasoning trace when feding in the proxy LLM to extract fingerprints. Moreover, if yes, whether the performance comes from the features extracting from the reasoning traces, rather than the real LLM fingerprints.
Thanks for the rebuttal. I am raising the score to 3.