acceptodds
Under review as a conference paper at ICLR 2027

VLM DNA: Tracing Vision-Language Models via Distance-Preserving Representations

Abstract

Identifying vision-language models (VLMs) and tracing their relationships across fine-tuning trajectories requires compact model representations that can be ex- tracted without access to model internals. We study training-free VLM identity representation extraction using only image-question queries and generated re- sponses, without modifying model parameters or requiring shared architectures. Existing text-only response representations do not explicitly account for the rela- tionship between generated answers and their visual inputs, potentially overlook- ing useful signals for distinguishing VLMs. We introduce VLM DNA, a compact representation of model behavior, and DNAExtract, a training-free framework for extracting it. DNAExtract probes each model with a shared set of image-question pairs and encodes the images and generated responses using frozen CLIP image and text encoders. It then constructs outer-product interaction matrices to repre- sent image-response associations and applies sparse random projection to obtain compact DNA vectors designed to approximately preserve pairwise distances be- tween the original cross-modal representations. Experiments on 39 open-source VLMs from seven families demonstrate that VLM DNA achieves an AUC of 0.929 for relation detection, an AUC of 0.923 for one-to-one verification, and a rank-1 accuracy of 89.38% with a mean average precision of 0.936 for one-to-many identification. These results outperform text-only response embedding baselines and support the utility of cross-modal representations for VLM identification and trac- ing. Code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM Lineage Tracing, Model Identification, Training-Free Representation, Random Projection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_orjmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method to classify VLMs into the same model family based on their responses to the same VQA questions. Specifically, the CLIP visual and text embeddings for each question's images and responses are extracted and then projected to compute the Euclidean distance between projected vectors. The proposed method is evaluated on diverse base models and achieves high accuracy.

Critical strengths
  • The proposed method is simple and straightforward, and achieves high accuracy on diverse models, validating the effectiveness of the proposed method.
  • The proposed method proposes a unified representation for multiple tasks, i.e., relation detection, one-to-one verification, and one-to-many base-model identification can be unified within a common representation.
  • The paper is well written and the method is easy to follow.
Critical weaknesses
  • Novelty limited. The clip based model identification methods have already been explored in previous works. Also the proposed method uses CLIP to construct VQA pair to characterize VLM behavior, the idea shares similarity with previous works.
  • The method relies on distances between the behavior differences to the same VQA questions, and assumes models in the same family exhibits similar behavior. However, the data the model finetuned on may also changes the behavior of the model. Does the author studied the influence of the behaviour of models in different family finetuning under the same data.
Questions for authors

Can the author examines the method's effectiveness when models in dfiferent family is finetuned with the same data? And the implact of data aginst the impact of base model.

1 comment

Dear Reviewer orjm,

Thank you again for your time and effort in reviewing our paper.

As the discussion period draws to a close, we would greatly appreciate any further feedback on our rebuttal and additional experimental results. We remain happy to clarify any outstanding questions or concerns.

Thank you for your consideration.

Best regards,

The Authors

Reviewer_cUyQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VLM DNA, a black-box vector that summarizes how a vision-language model behaves, and uses it to trace which models are related. Every model answers the same 600 image-question probes. Each answer is encoded with a CLIP text encoder and combined with the CLIP embedding of its image through an outer product. The outer products for all 600 probes are stacked into one long vector, and a shared sparse random projection reduces it to 512 dimensions. This vector is used in three tasks. An SVM trained on the difference between two models' vectors decides whether they are from the same family, a distance threshold makes the same decision without training, and the nearest of seven base models is taken as a model's base.

On 39 open-weight VLMs from seven families, the paper reports an AUC of 0.929 for relation detection against 0.918 for an MPNet text baseline, a verification AUC of 0.923, and 89.38% rank-1 accuracy for base model identification. It also reports that the projection preserves pairwise distances, that distances between models computed on two disjoint probe sets are strongly correlated, and that a fine-tuned model stays within the verification threshold of its base.

Critical strengths
  • The problem matters. The paper cites an analysis of Hugging Face lineage records finding that about 60% of the VLM releases it studied have no recorded parent, and a black-box way to trace lineage would help with auditing and license checks.

  • The method is simple and needs no training to extract the vectors. Once the probes, encoders and projection are fixed, adding a new model takes only one round of queries.

  • Relation detection is tested by holding out whole families, which is stricter than the 80/20 split of model pairs used in LLM DNA, and each task is reported with several metrics and plots, such as EER, ROC curves, CMC curves and distance histograms. Some statements are also carefully worded, for example that a cross-family label does not mean the models were developed independently.

Critical weaknesses
  • The overall setup is close to LLM DNA, with the same number of probes, decoding settings, SVM and Mantel test, so the comparison with text-only methods is the main way to see what the paper adds. The abstract says the method beats text-only baselines on all three tasks, but verification and identification are compared only with random guessing, and the original LLM DNA pipeline and PhyloLM are not run on the same answers. Adding these would make the claim much more convincing.

  • The experiments cover a narrower setting than the motivation. Identification always picks one of seven known base models and cannot report that a model's parent is not in the list, while the motivation is that many models have no recorded parent. There are no tests with quantization, model merging, a different system prompt, paraphrased outputs or deliberate attempts to avoid detection, and only open-weight models are used, although the introduction mentions models available only through APIs. At the moment the results mainly show that the method separates seven families of open-weight models on Hugging Face fairly well.

  • It would be good to know how much a model's vector changes when its answers are sampled again. Answers are sampled at temperature 0.7, and each model answers each probe only once. This seems important because the drift after fine-tuning on only 10 samples is already about as large as the distance between the closest same-family pair in Fig. 4a, and the verification threshold works best if two runs of one model are clearly closer to each other than two models from the same family are.

Questions for authors
  • How far apart are two sampled runs of the same model, compared with same-family distances and the verification threshold?

  • Could you add text-only baselines for verification and identification, or adjust the claim in the abstract?

  • What happens in identification when the true parent is not among the candidates? And how much does the vector change under quantization or a different system prompt?

1 comment

Dear Reviewer cUyQ,

Thank you again for your time and effort in reviewing our paper.

As the discussion period draws to a close, we would greatly appreciate any further feedback on our rebuttal and additional experimental results. We remain happy to clarify any outstanding questions or concerns.

Thank you for your consideration.

Best regards,

The Authors

Reviewer_fioMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to identify VLMs and trace their relationships using only image–question queries and generated responses. Building on LLM DNA, it proposes a training-free pipeline that combines frozen CLIP image and response embeddings through outer products and compresses them into compact fingerprints using sparse random projection. Experiments on 39 VLMs from seven families evaluate relationship detection, verification, and base-model identification, alongside analyses of distance preservation and stability under different probes and fine-tuning.

Critical strengths
  1. This paper is clearly written and well-presented.

  2. The problem being studied also appears to be important and relatively under-studied, tracing relationships between current available VLMs, thus having strong pratical values in area such as copyright protection.

  3. The methodlogy is generally sound and intuitively make sense, while also simple and elegant, but arguably less novel comparing to LLM DNA.

Critical weaknesses
  1. The methodological novelty over LLM DNA appears limited: the proposed approach retains its shared-probing, response-encoding, and random-projection pipeline, primarily adding multimodal probes, CLIP features, and outer-product fusion. Moreover, because image embeddings are shared across models, the pre-projection outer-product distance reduces to an image-norm-weighted text-embedding distance, becoming identical to the text-only distance when image embeddings are normalized.

  2. I am also not fully convinced about the experimental settings, the authors assumed that the actual parent model always exist in the search set, however, in reality, when the target parent model is unknown, if the parent model is not present in the search set, will VLM DNA successfully conculde there's no possible match?

  3. The experiments primarily distinguish same-family from cross-family models. Shared architectures, language backbones, chat templates, or response styles could explain this separation; stronger controls are needed, such as specific checkpoint identification within the same model family.

  4. The controlled fine-tuning experiment covers only one model and up to 10,000 examples, if we consider more realistic training setting, where the number of examples, training iteration could be significantly larger, will VLM DNA still be effective?

  5. This paper does not discuss or compare against established representation-similarity approaches such as CKA[1], RSA[2], or Gromov–Wasserstein[3]. While activation-based implementations would require additional access, these approaches could be applied to response embeddings under the same black-box protocol. Such comparisons would help establish whether the proposed DNA construction offers advantages over standard measures of behavioral similarity.

[1] Similarity of Neural Network Representations Revisited.

[2] Representational Similarity Analysis—Connecting the Branches of Systems Neuroscience.

[3] Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance.

Questions for authors

Given that LLM DNA already can achieve similar objective on pure text level, does adding visual information in this process actually improved upon LLM DNA? For example, is it possible that only based on the text output of VLM, LLM DNA can already suffice?

1 comment

Dear Reviewer fioM,

Thank you again for your time and effort in reviewing our paper.

As the discussion period draws to a close, we would greatly appreciate any further feedback on our rebuttal and additional experimental results. We remain happy to clarify any outstanding questions or concerns.

Thank you for your consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.