acceptodds
Under review as a conference paper at ICLR 2027

Delta Activations: A Representation for Finetuned Large Language Models

Abstract

The success of powerful open source Large Language Models (LLMs) has enabled the community to create a vast collection of post-trained models adapted to specific tasks and domains. However, navigating and understanding these models remains challenging due to inconsistent metadata and unstructured repositories. We introduce Delta Activations, a method to represent finetuned models as vector embeddings by measuring shifts in their internal activations relative to a base model. Clustering analysis shows that Delta Activations achieve strong separation of finetuned domains, significantly outperforming baselines such as flattened weights, salient parameter masks, and output embeddings, while being more lightweight and computationally efficient. Delta Activations also demonstrate desirable properties: it is robust across finetuning settings and remains useful even in dense local perturbation banks where many specialists lie close to the same pretrained weights. The method extends naturally to task embedding via few-shot finetuning for reliable model retrieval, and to model selection for merging by quantifying similarity. Delta Activations provide a practical foundation for organizing and reusing the growing ecosystem of post-trained models.

Primary area:
interpretability and explainable AI
Keywords:
deep learning, finetuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_a4JgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a method called delta activation to identify whether (or how) a foundation model is fine-tuned by datasets on specific domains. Delta activation uses the difference of hidden states of models before and after fine-tuning to identify the changes during fine-tuning, and uses the delta as the fingerprint of the fine-tuning process. For the delta activation, the authors perform clustering experiments to show the effectiveness of the fingerprint. Users can cluster the fine-tuned model to specific domain and know how the model is fine-tuned.

Critical strengths

The paper falls in an interesting scope that use the parameters or hidden states rather than model outputs as the "fingerprint" or "watermark" of LLMs. This method might be helpful as an supplementary way to identify the authorship of LLMs or fine-tuning.

The paper is easy to follow and has clear statements.

Critical weaknesses

The motivation of the paper is the main concern. I personally do not see any use case of the propose scenario: selecting model by clustering the activations/hidden states. For LLM using, people either use API or fine-tuned model now. For APIs, there is no need and not possible to have the hidden states. For open-sourced fine-tuned models, why there is a need to know what dataset is used to fine-tune the model? Why we do not know what dataset or task the model is fine-tuned on? If not, why we are using the model?

The paper seems not ready to be published. Most of the tables do not have the bottom line, and the captions are to the bottom of the tables (I think table captions should be above the table itself, for ICLR papers). The spaces are not well organized, some tables and figures and texts are too close. All the experiment figures look the same: x, y axles with colored points. The presentation quality is far from an ICLR paper.

Questions for authors

What is the use case of the proposed method? Who will use it, in which scenatio?

Why the users do not know which task(s) is the LLM fine-tuned on?

If we can use hidden states to cluster the models, why we are not using the model parameters (or maybe some layers' parameters)?

How does the method scale when we have more tasks (e.g., 100 datasets)? How does the method scale when the fine-tuning hyperparameters are different, or even the fine-tuning method is different? For example, LoRA fine-tuning, quantized model?

Reviewer_szLfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Delta Activations, which represent finetuned LLMs through activation differences from a shared base model. It evaluates domain clustering and explores few-shot task representation, model retrieval, and LoRA selection for merging.

Critical strengths
  1. Simple representation with minimal information requirements. Delta Activations represent existing finetuned models without their training data, metadata, or an additional learned encoder. This makes the approach useful for organizing model collections with incomplete documentation and provides a concrete methodological contribution.
Critical weaknesses
  1. Insufficient downstream baselines leave the practical gains unclear.

In Section 3.4, EmbedLLM [1] is absent from task retrieval comparisons, while the merging experiment omits both the retrieved Top-1 LoRA alone and alternatives such as LoRA-Flow [2] under matched candidate pools and data budgets. These missing controls prevent a clear assessment of whether Delta Activations improve model selection and whether subsequent merging adds value. Repeated trials are also needed to establish the stability of the reported merging improvement.

  1. Missing comparison with an existing activation-based model similarity method.

The baselines of this paper omit REEF [3], which also characterizes models through hidden representations on shared inputs. Although REEF targets model provenance, its CKA-based similarity provides a relevant activation-based baseline for domain separation. Without this comparison, the advantage of the proposed delta-and-average representation over an existing activation-based approach remains unclear.

[1] EmbedLLM: Learning Compact Representations of Large Language Models. In ICLR 2025.

[2] LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks. In ACL 2024.

[3] REEF: Representation Encoding Fingerprints for Large Language Models. In ICLR 2025.

Questions for authors
  1. In Section 3.4 (“Task embedding”), is there evidence that Delta Activations select models with better target-task performance than EmbedLLM, beyond identifying the correct domain?

  2. In the merging experiment, how does the retrieved Top-1 LoRA alone compare with the reported 36.3%? This comparison would clarify whether merging provides additional value beyond retrieval.

  3. On the model pools evaluated in Table 2, how does Delta Activations compare with REEF’s CKA-based similarity in separating fine-tuning domains?

Reviewer_uUXpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Delta Activations, a lightweight method to represent fine-tuned models as vector embeddings by comparing activations between the fine-tuned and base models. This is done with a single forward pass (per prompt) on a generic probe dataset. The authors show clustering results across 5 domains and in the neural thicket setting, as well as in a number of other studies. These include training settings, preference optimization, and full finetuning, in addition to extensions to few-shot task embedding and model selection for merging.

Critical strengths

1- The method is simple and does not rely on metadata or evaluation datasets. Furthermore, it is cheap to compute, and new models can be embedded without recomputing the existing ones. 2- The main clustering results beat baselines, and this paper further includes a broad set of ablations (probe number, length, content, layer and token position, training settings).

Critical weaknesses

1-The paper can be hard to apply to the real world. The main method requires access to activations, which is not possible for closed-source models. The paper’s variants (Delta Logits, Delta Meaning) perform worse. Additionally, it requires the same base model, which limits applicability across model families 2- The scope of the experiments is too narrow. The main empirical evidence is silhouette score and clustering on the authors’ own pools of 15 models with distinct domains. The task embedding and cross-checkpoint experiments have no baselines. The same holds for the DPO and full finetuning results, and the model selection experiment compares only to random selection.

Questions for authors

1- Is the CountSketch baseline a fair comparison? It performs below the random baseline in Table 4 (Hit@10 .333 vs .463). Could a stronger weight-space baseline be used? 2- In “model selection and similarity measurement,” why choose only the single most similar model as an anchor and sample the remaining 19 randomly? 3- Was the Alpaca probe tested on the neural thicket setting, and how much does it degrade compared with the chat-template probe? The generality of the probe is one of the paper’s strengths. 4- How do you see Delta Activations being applied in practice, for example on a hub with thousands of models from mixed bases? 5- Can the method be tested on real-world fine-tuned models? 6- In “cross-checkpoint clustering,” why is a silhouette score of 0.39 considered “cleanly recovering” the clusters? 7- In Section 2.4 (line 161), the definition of Δ_f(x) is repeated, is this intentional?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.