- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper addresses black-box model ownership verification and proposes TOPF to overcome the unreliability of textual fingerprints under stochastic sampling. TOPF extracts features from the token-level log-probabilities assigned by the source and reference models to responses generated by a suspicious model. It combines relative similarity to these models with similarity to the source model to determine model provenance. Experiments show that TOPF outperforms existing black-box methods in many settings and achieves performance comparable to white-box methods.
Critical strengths
- The paper addresses an important limitation of existing methods in handling output variability caused by stochastic sampling. Addressing this issue is practically relevant, particularly for reasoning models, whose long reasoning trajectories may amplify such variability.
- The proposed method enables ownership verification without modifying the source model. It does not require embedding additional fingerprints into the source model, offering the practical benefit of avoiding potential performance degradation.
Critical weaknesses
- The risk of false positives for similar models is insufficiently evaluated. Independently fine-tuned models sharing a pre-trained model represent a realistic and challenging scenario for ownership verification. However, the paper does not cover this scenario. This is particularly concerning for TOPF, which uses relative similarity to the source and reference models for verification. Such fine-tuned models may be more similar to the source model than reference models from different model families, potentially leading to false positives. Evaluating false positives on such negatives is therefore necessary to substantiate TOPF’s effectiveness.
- The substantial performance degradation under model merging is insufficiently analyzed. Table 12 reports an AUC of 0.114 for TOPF under model merging while MET achieves a high AUC of 0.957. The explanation that merging weakens source-specific probability patterns does not adequately explain MET’s strong performance or TOPF’s performance substantially below random guessing. Given that an existing method maintains reasonable performance across the evaluated strategies, including model merging, it is unclear what practical advantage TOPF offers despite its severe failure in this setting. A more thorough analysis is therefore needed to support the claimed benefits of TOPF.
Questions for authors
- Could the authors clarify which models are used as negatives in the ownership verification evaluation? Do these include models from the same family as the source model? If the negatives come only from different model families, the evaluation may underestimate the risk of false positives.
- Could the authors also explain how the metrics in Table 1 are aggregated across the different post-development strategies?
8 comments
A Kind Reminder Regarding Our Rebuttal
Dear Reviewer EA4j,
Thank you again for your valuable comments and feedback on our submission. With three days remaining in the discussion period, we would like to kindly follow up and ask whether our rebuttal has sufficiently addressed your concerns.
Should any questions or concerns remain, we would be grateful for the opportunity to provide further clarification. If you find that our responses have resolved the issues you raised, we would sincerely appreciate your consideration of updating your evaluation scores as appropriate.
Thank you very much for your time and consideration.
Best regards,
The Authors
Thank you very much for your detailed response and additional experiments. These results address some of my concerns. However, I would appreciate further clarification regarding the potential for false positives.
My main concern is that negative models independently fine-tuned from the same base model as the protected source may receive high verification scores due to their similarity to the source model, potentially leading to false positives in practical ownership verification.
Although the additional experiments achieve an AUC of 1.000, this does not necessarily rule out false positives under the original decision threshold. In practical ownership verification, the ground-truth labels of suspicious models are unknown, making it infeasible to recalibrate the threshold using the models being verified. Therefore, it is important to evaluate whether the related negative models can be correctly rejected using the threshold calibrated on the original validation set.
I would therefore appreciate clarification on the following points:
- Were the same reference models as those in the original manuscript used in the additional experiments? Also, were any of the negative models in the additional experiments used during encoder training?
- Could the authors provide the verification scores of the positive and negative models in the additional experiments?
- When applying the verification thresholds calibrated on the original validation sets, without recalibrating them for the additional experiments, what are the resulting TPR, FPR, and accuracy?
If the negative models sharing the same base model receive sufficiently low verification scores and the original decision thresholds yield few or no false positives, my concern would be substantially alleviated.
Response 1 to Reviewer `EA4j`
In the original evaluation, negatives are models associated with the other source lineages, including their post-development variants. For example, DS-R1-8B is evaluated against Qwen3-8B and GPT-OSS-20B variants. Appendix F lists the checkpoints.
We additionally evaluate two derived models and five related negatives per source, all sharing Llama-3.1-8B Base ancestry. Related negatives originate from other branches of the shared ancestor. DS-R1-8B and Llama-3.1-8B-Instruct belong to separate branches and serve as negatives for each other. TOPF achieves AUC and pAUC of 1.000 on both sources, separating the tested derivatives from models with shared pretraining ancestry.
Table 1. Verification with two derived models and five related negatives per source, all sharing Llama-3.1-8B Base ancestry.
The metrics in manuscript Table 1 are computed from pooled model-level scores for each protected source. Each evaluated checkpoint or deployment variant contributes one verification score and one label.
We combine entries from all evaluated strategies and construct a single ROC curve per source. AUC, pAUC, and TPR@3% are computed from this curve. MD is computed from the corresponding positive and negative score distributions.
Each model entry receives equal weight, so strategies with more evaluated variants contribute more entries to the pooled results. Manuscript Table 12 provides the complementary breakdown by strategy.
Merging can weaken source-specific probability patterns. In the original DS merging evaluation, TOPF achieves AUC 0.114 on DeepLlama-Chimera-NSFW-8B, while MET achieves 0.957. Its subsequent unavailability prevents further diagnosis of the difference. Results for other protected sources show that this failure does not extend to every evaluated merge.
Table 2. Merge detection for other protected sources, using their respective evaluation pools.
We additionally construct 12 DS merges using linear merging, TIES, and DARE-TIES. Each method combines DS with Llama-Instruct at DS coefficients of 0.25, 0.50, and 0.75, and with Hermes-3 at 0.50. TOPF achieves AUC 0.981 and pAUC 0.915, supporting effective detection across these controlled merges. These results demonstrate TOPF's effectiveness with a small query budget across the tested merges.
Table 3. DS merging experiment with 12 positives and nine negatives. TOPF uses 20 responses; MET uses 25 prompts with five generations each, totaling 125 responses.
Thank you again for the helpful feedback and thoughtful guidance. We hope our detailed responses successfully resolve your concerns. If there are any remaining uncertainties, we would be happy to clarify them.