acceptodds
Under review as a conference paper at ICLR 2027

MAGE: Matrix-weighted Attention Graph Energies for LLM Family Fingerprinting

Abstract

Limited transparency in large language model (LLM) development motivates provenance analysis to distinguish independently developed models from derivatives. We introduce Matrix-weighted Attention Graph Energies (MAGE), a framework for extracting attention metrics that can be used as model fingerprints. MAGE represents self-attention as a matrix-weighted graph, with tokens as nodes and attention-weighted value–output projection matrices as edge weights. Inspired by this representation and the Laplacian quadratic energy form, we define three attention energy metrics. Each metric's layerwise profiles show high within-model Pearson correlations across the evaluated prompts. This consistency despite input-dependent attention supports their use as model fingerprints. To evaluate the utility of these profiles, we construct a model family classifier that compares the three metrics' z-score-normalized layerwise profiles using Dynamic Time Warping (DTW). With a fixed bank of 23 reference models from six families, the classifier achieves 25/30 accuracy in a development evaluation on models outside the bank. Separately, predictions for three of four DeepSeek distilled models match their documented Qwen or Llama base lineages rather than their DeepSeek product identity. We also evaluate robustness under LoRA-based supervised fine-tuning: the fixed classifier retains correct family predictions for all 32 post-training checkpoints from 8 source models. These results suggest that MAGE captures consistent attention characteristics across prompts useful for model family identification and base-lineage analysis. Our code will be available at: https://github.com/piberius0421/mage_llm_fingerprint after acceptance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLMs, matrix-weighted graph, LLM fingerprint

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_hY1XOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

Summary This paper introduce Matrix-weighted Attention Graph Energies (MAGE), a framework for extracting attention metrics that can be used as model fingerprints. It represents self-attention as a matrix-weighted graph incorporating attention probabilities and value–output projections. It combines token representations and attention probabilities obtained from a model’s forward pass with its value–output projections, resulting in a single matrix-weighted attention graph captures both the aggregation weights between tokens and the transformations of the transmitted vector information. For model family identification, authors use reference bank comprising 23 models: four Llama, five Qwen, four Gemma, four SmolLM2, four Pythia, and two DeepSeek models. Comparison is done by expressing each prompt’s metric profiles in terms of relative depth in [0, 1] and linearly interpolate them onto 64 uniformly spaced grid points, followed by preprocessing and Dynamic Time Warp matching. Results for within-model profile consistency across prompts shown the consistent layerwise profile shapes recurrence within each model across different prompts. For between model and within family profile structure, authors compute pairwise DTW costs between these model profiles and compare each with the six family representatives used by the classifier, showing that all 23 models had the lowest cost for their own family.For unsee mode family classification, authors classified 30 unseen models while keeping the 23 reference models and six family representative profiles, with results supporting the utility of the three metrics for family identification beyond the reference bank. By applying LoRA-based supervised fine-tuning to eight source models from the Llama and Qwen families using 4,000 instruction–response examples from No Robots and two seeds results , authors demonstrate robustness of family identification under the evaluated LoRA-SFT conditions. Authors analysis points to that cross-prompt consistency of the three metrics and the similarity of profiles within families suggest that MAGE may capture recurring internal model characteristics despite input variation.

Contributions

  • paper define three metrics inspired by a matrix-weighted graph representation of self-attention and the Laplacian quadratic energy form.
  • paper analyze within-model profile consistency across prompts and profile relationships across models and families
  • paper assess the identifying information and scope of the proposed fingerprint through model family classification andbase-lineage attribution for distilled models

Validations

  • For within-model cross-prompt consistency, authors evaluate 23 reference models across 6 families (Llama, Qwen, Gemma, SmolLM2, Pythia, DeepSeek) on 1,000 diverse prompts (499,500 prompt pairs per metric).
  • For within-family clustering & unseen family classification, authors showed lower median DTW cost within families (0.21810.2181 over 35 pairs) than between families (1.52071.5207 over 218 pairs) on the 23 reference models (23/23 self-classification).
  • For Post-training robustness (LoRA-SFT), authors fine-tuned 8 Llama and Qwen models with LoRA (targeting attention and MLP projections including WV,WOW_V, W_O) on 4,000 No Robots examples across 2 seeds and 2 epochs (epochs 1 and 3). Achieved 32/32 (100%) accuracy, including when excluding the source checkpoint from the reference bank.
  • For Base-lineage attribution on distilled models, authors evaluated 4 DeepSeek-R1-Distill models against their documented Qwen or Llama base families. Correctly matched 3/4 models (Qwen-1.5B, Qwen-7B, Llama-8B), while misclassifying DeepSeek-R1-Distill-Qwen-14B as DeepSeek
Critical strengths
  1. The paper introduces a clean, practical way to fingerprint LLMs by incorporating the learned value–output projection matrices (WOWVW_O W_V) directly into attention-graph analysis, rather than relying only on scalar attention probabilities (αij\alpha_{ij}). By defining three layerwise energy metrics (Sl,Dl,D~lS_l, D_l, \tilde{D}_l) that handle causal masking and asymmetric weights, MAGE compresses complex forward-pass activations into a compact three-dimensional trajectory across layers. This is consequential because it makes it possible to directly compare models with different hidden dimensions and layer counts using relative-depth interpolation and Dynamic Time Warping (DTW), while capturing identifying signal from the projection weights that improves family classification over scalar attention graphs alone (13/15 vs. 10/15).
  2. Equally important is the strong empirical evidence that these layerwise profiles are both prompt-invariant and resilient to post-training. Even though attention weights change with every input, testing across 23 reference models and 1,000 diverse prompts shows that a model's layerwise energy curve stays nearly identical regardless of the text fed into it, with median prompt-pair Pearson correlations exceeding 0.949 across all six family representatives in Table 1. For a fingerprint to be useful in practice, it cannot depend on a fragile set of probing queries and must survive downstream adaptation. Showing that forward-pass activations reliably collapse onto a stable, family-specific curve—capable of classifying 25/30 unseen models, holding up across 32/32 LoRA fine-tuned checkpoints, and recovering the base Qwen or Llama lineage for 3 out of 4 distilled DeepSeek models—provides a compelling empirical foundation for white-box model auditing.
Critical weaknesses
  1. The main weakness of the paper is that it never convincingly shows we actually need dynamic forward-pass attention graphs rather than just simple static weight norms across layers. Because the value-output weights (WOWVW_O W_V) and normalization layers (LayerNorm or RMSNorm) apply fixed scaling that spikes sharply in the final layers—for example, DlD_l stays near zero early on before jumping to ~75 in Figure 2b and 2×1032 \times 10^3 in Figure 1c—this fixed layer-to-layer jump alone will naturally force any two prompts to have near-perfect Pearson correlations (>0.949> 0.949). While the authors compare MAGE against a scalar attention baseline without WOWVW_O W_V, they only test it on 15 of the 30 unseen models using 100 prompts (where 13/15 vs. 10/15 is too small a gap to be convincing), and they completely skip a basic weight-only baseline that doesn't use forward passes. Since MAGE already requires white-box access to the weights and runs up to 1,000 forward passes per model, the core premise of building input-dependent attention graphs is hard to justify if simple weight norms do the same job.
  2. As defined in paper, distillation evaluation suffers from both an unchecked reference-bank overlap and a misleading motivation. In the LoRA test, the authors carefully verify that removing the exact source model from the 23-model reference bank still gives 32/32 accuracy, yet for the four DeepSeek distilled models (where 3 of 4 match their Qwen or Llama base families), they never mention if the exact base checkpoints (like Qwen2.5-1.5B in Table 1) were removed from the bank. Without that check, it is impossible to tell whether MAGE is picking up a broader family signature or just matching a fine-tuned model back to its exact starting checkpoint. Furthermore, the Introduction motivates the paper with real-world API distillation disputes (like OpenAI/DeepSeek and Anthropic/Moonshot AI), where the goal is catching the teacher model that generated the training data. However, predicting Qwen instead of DeepSeek for DeepSeek-R1-Distill-Qwen shows that MAGE only tracks the student's starting weights and is blind to the teacher—contradicting the opening pitch and leaving the lineage claim resting on just four models.
Questions for authors
  1. Since MAGE assumes white-box weight access and runs up to 1,000 forward passes per model, how does a purely static, weight-only baseline—such as matching layerwise profiles of ∥M(l,h)∥F\Vert{}M^{(l,h)}\Vert{}_F or LayerNorm/RMSNorm scale parameters (γ(l)\gamma^{(l)}) without running any forward passes or computing attention probabilities αij(l,h)\alpha_{ij}^{(l,h)}—perform on the 30 unseen models?
  2. In Section 5.4, why was the scalar-weighted Dirichlet energy baseline (setting M(l,h)=IM^{(l,h)} = I) evaluated on only 15 of the 30 unseen models using 100 prompts (yielding 13/15 vs. 10/15) rather than the full 30-model test set? What is the scalar baseline's accuracy across all 30 unseen models and the 4 distilled models?
  3. In the LoRA-SFT experiment (Section 5.6), authors explicitly verify that excluding the source checkpoint from the 23-model reference bank and recomputing family centroids preserves 32/32 accuracy. For the four DeepSeek distilled models (Table 2), are the exact base checkpoints of the three correctly attributed models (Qwen2.5-1.5B, Qwen2.5-7B, and Llama-3.1-8B) present in the 23-model reference bank while Qwen2.5-14B is absent?
  4. What are the classification results and DTW costs for the four distilled models when their exact base checkpoints are excluded from the reference bank?
  5. Because taking a pointwise arithmetic mean across interpolated curves can smear shifted peaks (conflicting with the motivation for DTW in Section 4.2), how does your classifier compare against direct 1-Nearest Neighbor (1-NN) DTW over the 23 reference models, or against simple Euclidean (L2L_2) distance (DTW window = 0) on the 64-point interpolated grid?
  6. Section 5.5 refers to the 30 unseen models as a "development evaluation". Were pipeline hyperparameters—such as the 64-point interpolation grid, the DTW warping window of 3, and the metric combination (Sl+Dl+D~lS_l + D_l + \tilde{D}_l)—selected by maximizing accuracy on these 30 unseen models, or tuned strictly on the 23 reference models?

Suggested revisions:

  1. The opening paragraph motivates model provenance using disputes over unauthorized black-box API distillation (citing OpenAI/DeepSeek and Anthropic/Moonshot AI), where an auditor wants to identify the teacher model that generated synthetic training data. However, classifying DeepSeek-R1-Distill-Qwen as Qwen rather than DeepSeek shows that MAGE recovers the student's starting base weights and is largely invariant to the distillation teacher. We strongly suggest revising the Introduction so it does not conflate black-box teacher data attribution with white-box base-checkpoint lineage.
  2. extending the controlled experiments proposed in Section 5.7 to include full-parameter fine-tuning, continual pre-training, and controlled pre-training runs (varying training corpora under a fixed architecture and normalization scheme) would help isolate how much of the MAGE signature stems from learned data representations versus architectural hyperparameters.
8 comments
AuthorsOpenReview ↗

Response to Reviewer hY1X (Part I)

Thank you for your careful review.

We revised the manuscript to make the progression clearer. MAGE represents token interactions in self-attention as a directed matrix-weighted graph. We observe recurring layerwise energy profiles across diverse prompts and similarities within model families, then evaluate their identifying information through model-family classification.

  • Sections 1–3 make this connection explicit. Proposition 1 in Section 3.2 and Appendix C.5 describe how projections and attention-weighted token state differences jointly determine the energy.
  • Appendix I.5, Tables 30–31, adds static projection controls. Appendix F.1, Table 13, adds raw-energy variation among equal-length prompts. Section 5.4 and Appendices I.3–I.4 also provide existing-method comparisons.
  • Section 5.6 and Appendix G.3 explain the documented starting checkpoints and base-family evaluation target.
  • We revised “development evaluation” in the Abstract and Section 5.5, and “development panel” in Appendix H, because these terms could unintentionally suggest that we selected hyperparameters based on the 30-model evaluation results. We set the default configuration using the 23-model reference bank, evaluated the 30 unseen models, and then conducted sensitivity analyses. The revised wording and the explanation in Appendix H clarify this sequence.

The tail-layer, query-specific exact-source, and direct-reference analyses are reported directly below.

Responses to Weaknesses

W1: We performed the requested weight-only comparison and explain its results and relation to our research perspective in Q1. Here we address how much the final-layer peaks affect the high Pearson correlations.

Using the existing 1,000-prompt profiles of all 23 reference models, we removed either the last layer or the last ⌈L/10⌉\lceil L/10\rceil layers and recomputed Pearson correlations. We used the original layer values before interpolation and compared all 499,500 prompt pairs for each model and metric. The table reports the median across the 23 models of their prompt-pair Pearson medians.

MetricAll layersLast layer removedLast ⌈L/10⌉\lceil L/10\rceil layers removed
SlS_l0.97810.97650.9741
DlD_l0.99700.99740.9966
D~l\widetilde D_l0.98770.98900.9864

For Llama-3.2-3B in Figure 2, the DlD_l median is 0.9968 with all layers, 0.9967 after removing the last layer, and 0.9974 after removing the last three. High consistency therefore persists in the remaining profile in this example. Some models show decreases: Pythia-410M's SlS_l falls from 0.9429 to 0.8839 after removing the final 10%. Overall correlations remain high while exhibiting these model-specific changes.

We used 1,000 prompts to examine profile consistency across diverse inputs. Appendix H.1, Table 21, reports classification with fewer prompts. Across 20 trials using the same prompt subset for the bank and queries, 10 prompts yielded an average of 24.00/30 correct predictions, and 100 prompts yielded 24.65/30.

W2: We checked which starting checkpoints are present in the bank and evaluated each query with its own exact source excluded. All four predictions and the 3/4 accuracy were retained. Q3–Q4 provide the checkpoint list, exclusion conditions, and all family DTW costs.

The examples in the Introduction motivate examining how models were developed and where they originated. The revised text connects this motivation to studying internal self-attention characteristics when weights and attention are accessible. The DeepSeek experiment asks whether the original base-family characteristics remain in models initialized from Qwen/Llama checkpoints and trained on data generated by DeepSeek-R1. Section 5.6 and Appendix G.3 identify the documented starting checkpoints as the basis for the target labels and report base-lineage attribution for these four models.

The suggested full-parameter fine-tuning, continual pre-training, and controlled pre-training experiments would help examine how training affects the energy profiles. However, varying training data while fixing architecture and normalization requires models with matched initialization, training procedures, and other conditions. If suitable public checkpoints cannot be obtained, the comparison models need to be trained specifically for this purpose. Constructing and training such a controlled model set was beyond what we could complete during this revision. Section 5.7 and the Conclusion describe the corresponding research direction: separating the effects of training and model design and tracking intermediate training checkpoints.

AuthorsOpenReview ↗

Response to Reviewer hY1X (Part II)

Responses to Questions

Q1: Our study starts from interpreting token interactions in self-attention as a directed matrix-weighted graph. Our main finding is that energy metrics defined on this representation exhibit recurring layerwise shapes across diverse prompts and similarities among models within the same family. Model-family classification is an application used to evaluate the identifying information and utility of these characteristics. The name MAGE (Matrix-weighted Attention Graph Energies) reflects this perspective.

The edge weight represents both the aggregation weights between tokens, given by attention probabilities, and the vector transformation through the value–output projection. We therefore use αijWOWV\alpha_{ij}W_OW_V to include both components; WOWVW_OW_V alone does not represent the input-dependent aggregation weights between tokens.

We also agree that it is useful to test how much weight-only statistics explain profile shapes and family-identification results. We conducted OV-, WV-, and WO-only comparisons using the same 23-model reference bank, 30 unseen models, and comparison procedure. The proposed OV-norm feature averages the headwise Frobenius norms within each layer:

ulOV=1Hl∑h∣M(l,h)∣F.u_l^{\mathrm{OV}}=\frac{1}{H_l}\sum_h\\|M^{(l,h)}\\|_F .

We applied the existing 64-point interpolation and z-score normalization to each profile, averaged model profiles within each family, and compared them using window-3 DTW.

MethodInputCorrect
OV normProjection weights25/30
OV squared norm (secondary control)Projection weights23/30
WV normProjection weights22/30
WO normProjection weights20/30
MAGE-100100 prompts24/30
MAGE-10001,000 prompts25/30

OV norm and MAGE-1000 each correctly classified 25 of the 30 models. However, MAGE correctly classified all five models that OV norm misclassified, and OV norm correctly classified all five that MAGE misclassified. Thus, the same overall accuracy was accompanied by different model-level family predictions. We added the feature definitions, comparison procedure, and predictions for all models to Appendix I.5, Tables 30–31.

Q2: We designed the scalar comparison as a supporting controlled experiment to examine the contribution of the value–output transformation in the matrix-weighted graph. Profiles with this transformation replaced by the identity matrix II require separate extraction, so extending the evaluation to all 30 queries entails additional computation time. Given this experimental purpose and the available time and computing resources, we retained the existing 23-model reference bank and conducted the comparison on 15 unseen queries using 100 prompts.

Both methods used the same selected models, prompts, preprocessing, and DTW procedure. Under these conditions, the scalar baseline obtained 10/15 and MAGE obtained 13/15. Appendices I.1–I.2, Tables 26–27, report the model-level results and experimental details. Scalar accuracies for all 30 queries and the four distilled models have not yet been computed.

Q3: The exact starting checkpoints, checked against the official model card, are:

Distilled queryDocumented exact sourceIn the original 23-model bank?
DeepSeek-R1-Distill-Qwen-1.5BQwen2.5-Math-1.5BYes
DeepSeek-R1-Distill-Qwen-7BQwen2.5-Math-7BNo
DeepSeek-R1-Distill-Llama-8BLlama-3.1-8BNo
DeepSeek-R1-Distill-Qwen-14BQwen2.5-14BNo

Only the Qwen-1.5B query's exact source is present. Qwen-7B and Llama-8B were correctly classified as Qwen and Llama even though their starting checkpoints were absent from the bank. The absence of the source therefore applies to these two correctly classified models as well as Qwen-14B. Appendix B.1, Table 5, lists the references; Appendix G.3, Table 17, lists the documented starting checkpoints.

AuthorsOpenReview ↗

Response to Reviewer hY1X (Part III)

Q4: For each query, we removed its own exact source wherever present and recomputed the affected family representative, keeping the other references, preprocessing, and DTW settings fixed. The Qwen-1.5B query consequently uses 22 references; the other three use the original 23, which already exclude their respective sources.

The full family DTW costs under this query-specific exclusion are below. The common “DeepSeek-R1-Distill-” prefix is omitted. The smallest cost determines the prediction.

QueryBank sizePredictionLlamaQwenGemmaSmolLM2PythiaDeepSeek
Qwen-1.5B22Qwen1.0379120.1501012.1295580.8174321.9229251.002349
Qwen-7B23Qwen1.6134850.8550262.1293571.6840061.9705480.924934
Llama-8B23Llama0.1638071.0074281.9329400.5571341.1924900.431525
Qwen-14B23DeepSeek0.9804760.7908782.1134790.8656431.8303090.477076

For Qwen-1.5B, the Qwen cost increases from 0.104649 to 0.150101 while the prediction is retained. The other queries' banks and costs are unchanged. All four predictions and the 3/4 correct count therefore remain the same under query-specific exact-source exclusion.

Q5: Using the existing 1,000-prompt profiles of the same 23-model bank and 30 unseen models on a 64-point grid, we obtained the following results.

RepresentativesDistanceCorrect
Six family-average profilesDTW, w=3w=325/30
23 individual references, 1-NNDTW, w=3w=322/30
Six family-average profilesSquared Euclidean, w=0w=019/30

The 1-NN result comes from reanalyzing the existing profiles; the family-average DTW and Euclidean results are reported in Appendix H.3, Table 23.

Q6: The 23-model bank, G=64G=64, w=3w=3, and default Sl+Dl+D~lS_l+D_l+\widetilde D_l combination were specified before observing the classification results for the 30 unseen models. We constructed the classifier using reference-bank profiles and applied it to the 30 external checkpoints with those choices fixed.

We subsequently reused the collected profiles to analyze sensitivity to the grid, window, metric combination, and number of prompts, as reported in Appendix H. We retained the original configuration after these analyses; the main-text result of 25/30 uses that configuration.

The terms “development evaluation” and “development panel” could unintentionally suggest that we selected the settings based on the 30-model results. We therefore revised them in the Abstract, Section 5.5, and Appendix H to distinguish evaluation with a fixed configuration from the subsequent sensitivity analyses.

Reviewer_VRDHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MAGE constructs layerwise attention-energy profiles incorporating value–output projections and uses dynamic time warping to identify model families. The paper reports cross-prompt stability and robustness under the evaluated LoRA fine-tuning conditions.

Critical strengths

The representation is compact, and implementation details are supported by useful attention-output reconstruction checks.

Sensitivity analyses and explicit reporting of misclassifications help clarify the method’s empirical scope.

Critical weaknesses

Although a scalar attention-energy baseline is provided, comparisons with simple architecture/tokenizer features, layerwise weight statistics, and existing weight-based fingerprints are missing. Given the white-box setting, it remains unclear whether MAGE provides identifying information beyond simpler structural or parameter-based features.

The 30-model evaluation uses checkpoints outside the reference bank but remains a development panel, without a separate evaluation after method and hyperparameter choices are frozen. Correlated Base/Instruct checkpoints and limited controls for shared architecture versus training lineage further weaken the provenance claims.

Questions for authors

Can MAGE be compared against architecture/tokenizer and weight-based baselines, ideally including independently trained models with matched architectures and tokenizers?

Were the reference bank and hyperparameters fixed before evaluating the 30-model panel? Does performance generalize to new checkpoints selected after these choices are frozen?

5 comments
AuthorsOpenReview ↗

Response to Reviewer VRDH (Part I)

Thank you for your careful review.

We have added the requested comparisons with static weight statistics and existing fingerprinting methods. We also revised the manuscript to make the progression clearer: defining energy metrics from a graph representation, observing recurring profile shapes across prompts and similarities within families, and evaluating their use in model-family classification.

  • Sections 1–3 connect the research question, graph representation, and metric definitions. Proposition 1 in Section 3.2 and its proof in Appendix C.5 explain how projections and attention-weighted token state differences jointly determine the energy.
  • Section 5.4 and Table 2 now include REEF and HuRef-ICS comparisons; Appendix I.5, Tables 30–31, reports OV-, WV-, and WO-norm controls. Table 3 in Section 5.5 brings together family-wise accuracy and misclassification costs.
  • We revised “development evaluation” in the Abstract and Section 5.5, and “development panel” in Appendix H, because these terms could unintentionally suggest that we selected hyperparameters based on the 30-model evaluation results. We fixed the reference bank and default settings before evaluating the 30 unseen models, then conducted sensitivity analyses afterward. The revised wording and the explanation in Appendix H clarify this sequence.

Responses to Weaknesses

W1: We added OV-, WV-, and WO-norm controls based on static projection statistics, together with REEF and HuRef-ICS comparisons. These evaluations use the same 23-model reference bank and 30 unseen models. Q1 gives the results and procedures.

We also expanded the explanation of how weight statistics relate to the energy. Proposition 1 expresses how projections and attention-weighted token state differences jointly determine the energy. Appendix F.1, Table 13, adds a comparison of raw energy values among equal-length prompts for all 23 reference models. Energy values varied with the input even when projection weights and token count were fixed. We present this variation alongside the consistency of layerwise profile shapes.

W2: We evaluated the 30 unseen models after fixing the reference bank and default settings. The main evaluation retained those settings throughout the subsequent sensitivity analyses. Q2 explains this sequence and the scope of the additional evaluations.

Base/Instruct checkpoints from the same series are related, so their classification outcomes may also be correlated. Appendix H notes this dependence, and main-text Table 3 reports family-wise accuracy and misclassification cases. Section 5.7 discusses hypotheses about the contributions of architecture and training, together with controlled studies needed to examine them.

Responses to Questions

Q1: Our starting point is to interpret token interactions in self-attention as a directed matrix-weighted graph. Our main finding is that the energy metrics defined from this representation exhibit recurring layerwise shapes across diverse prompts and similarities between models within the same family. Model-family classification is an application used to evaluate the identifying information and utility of these characteristics. The name MAGE (Matrix-weighted Attention Graph Energies) reflects this perspective.

An architecture/tokenizer-based baseline evaluates how well model architecture and tokenizer information distinguish family labels. This is relevant to the classification application, but serves a different purpose from directly examining the recurring structures and characteristics of input-dependent attention energies studied in our work. We therefore did not extend the comparison scope to classification using only architecture and tokenizer information.

In MAGE's graph representation, an edge weight represents both the aggregation weights between tokens, given by attention probabilities, and the vector transformation given by the value–output projection. Since WOWVW_OW_V alone does not express the input-dependent aggregation weights between tokens, we include both components through αijWOWV\alpha_{ij}W_OW_V.

AuthorsOpenReview ↗

Response to Reviewer VRDH (Part II)

Q1 (continued): Beyond this rationale for the graph representation, we agree that weight-only controls are useful for examining how much of the profile shapes and family-identification results can be explained by static weight statistics. We therefore evaluated OV-, WV-, and WO-only controls using the same 23-model reference bank, 30 unseen models, and comparison procedure. We also evaluated REEF and HuRef-ICS on these models.

MethodInputCorrect
OV normProjection weights25/30
WV normProjection weights22/30
WO normProjection weights20/30
HuRef-ICSFull-Wikipedia token frequencies and model weights21/30
REEFThe same 100 prompts24/30
MAGE-100100 prompts24/30
MAGE-10001,000 prompts25/30

The OV-norm feature averages the headwise Frobenius norms within each layer: ulOV=Hl−1∑h∣M(l,h)∣Fu_l^{\mathrm{OV}}=H_l^{-1}\sum_h\\|M^{(l,h)}\\|_F. Each static profile follows the existing interpolation, z-score normalization, family averaging, and DTW procedure.

OV norm and MAGE-1000 both correctly classified 25 of the 30 models. However, MAGE correctly classified all five models that OV norm misclassified, and OV norm correctly classified all five that MAGE misclassified. Thus, the two representations produced different family predictions despite having the same overall accuracy. Appendix I.5, Tables 30–31, provides the feature definitions and predictions for every model.

REEF computes CKA between representations obtained from the same 100 prompts, and we average query–reference similarities within each family. HuRef-ICS uses embeddings constructed with full-Wikipedia token frequencies and QK, VO, and MLP invariant terms from the final two layers; we apply the same family-mean aggregation. REEF and MAGE each achieved 24/30 with the same 100 prompts, while HuRef-ICS achieved 21/30. Section 5.4, Table 2, and Appendices I.3–I.4 describe these adaptations to family classification and their results.

Comparing independently trained models with matched architectures and tokenizers would help us examine how training affects MAGE's energy metrics and layerwise profiles. This follows the direction outlined in Section 5.7 for distinguishing the effects of training and model design. During this revision, however, we have been unable to obtain a model collection satisfying these controls with verifiable independent training and training conditions. Without suitable public checkpoints, we would need to construct models with matched architectures and tokenizers and train them independently, which we cannot complete within the revision period.

Q2: Yes. The 23-model reference bank, G=64G=64, DTW window w=3w=3, and default combination Sl+Dl+D~lS_l+D_l+\widetilde D_l were specified before observing the classification results for the 30 unseen models. We constructed the classifier using reference-bank profiles and then evaluated the 30 external checkpoints with that configuration fixed.

The sensitivity analyses in Appendix H were conducted afterward. We varied prompt budget, grid, window, and metric combination using the collected profiles to examine their effects on performance. The main-text 25/30 result uses the original configuration; we did not reselect the default settings based on the sensitivity results.

The 30 queries share no checkpoint IDs with the 23 references and extend to different sizes and generations. For example, the Llama references are 3.2-series 1B/3B models, while the queries include 3- and 3.1-series 8B models. The Qwen references are 0.5B/1.5B models, while the queries span 3B–32B. Tables 5 and 11 provide the lists.

The additional comparisons in this revision also use the same 30 models so that the methods are evaluated on a common set.

Thank you for the additional baseline comparisons and for clarifying that the evaluation settings were fixed in advance. These address several of my concerns.

However, my central concern remains. The static OV-norm baseline achieves the same 25/30 accuracy as MAGE, while architecture/tokenizer controls and independently trained models with matched architectures remain unevaluated. Although the authors now emphasize the stability of layerwise energy profiles, it remains unclear what the proposed representation reveals beyond simpler structural or weight-based features.

I therefore maintain my original rating.

Reviewer_rqoJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MAGE, a training-free fingerprint for open-weight LLMs. Self-attention is treated as a graph with edge weight a_ij W_O W_V, and three per-layer scalars (S_I, DI, D. _I) are computed from it. The layerwise profiles of these scalars barely change across prompts (median Pearson > 0.95), so the authors use them to classify model families with DTW against 6 family centroids. Results: 25/30 on unseen models, 3/4 correct base lineage for DeepSeek-R1-Distill models, 32/32 after LoRA SFT.

Critical strengths
  1. Simple, training-free, no modification to the model.
  2. Reporting is careful: lots of sensitivity analysis in the appendix, failure cases listed with their DTW costs.

3.The LoRA and distilled-model tests are the right kind of tests for a fingerprint.

Critical weaknesses
  1. The key ablation is missing. D_l has cross-prompt correlation 0.997, i.e. it hardly depends on the input, so it may just be reading the per-layer norm of W_O W_V. That quantity needs no forward pass. Sec 5.4 removes Mand keeps attention; I need the opposite: drop attention/hidden states, _OW_V, same DTW pipeline. If that also gets ~25/30, the graph/energy keep per-layer stats of W framing adds nothing. This decides my score.

  2. Family classification is partly architecture detection. Gemma and Pythia are separable from config. json alone. The hard cases are the Llama-block families (Llama/SmolLM2/DeepSeek- LLM/Qwen2.5) and that is where the errors are (both Qwen2.5-14B -> DeepSeek, Table 10). The DeepSeek "family" is two near-identical checkpoints (cost 0.01, Fig 4) with no unseen positives, so it is basically a sink. Please add a config-metadata baseline and report accuracy on the Llamablock subset separately.

  3. No comparison to existing training-free fingerprints. HuRef is cited but not run; REEF (Zhang et al., ICLR 2025) is not cited at all and is the obvious baseline. The only competitor is the authors' own scalar baseline, on 15 models / 100 prompts.

4: the intro motivates with output-distillation allegations, but the method is white-box and only tested on weight-inheriting fine-tunes. Results are sensitive to grid size (G=48 -> 21/30, 128 -> 22/30) and D_lalone already gives 24/30, so the 3-metric combination is not clearly needed.

Questions for authors
  1. Weight-only baseline from W1: how much of 25/30 survives?

  2. What does a config.json baseline get on the 30 models?

  3. Can REEF be run on the same 30 models?

3 comments
AuthorsOpenReview ↗

Response to Reviewer rqoJ (Part I)

Thank you for your careful review. We have completed the requested OV-norm and REEF evaluations on the same 30 unseen models and added a HuRef-ICS comparison.

We also revised the organization and explanations to make the connection between our research question, empirical observations, and application clearer. MAGE represents token interactions in self-attention as a directed matrix-weighted graph. The main empirical finding is recurring layerwise structure across diverse prompts, together with similarities within model families. Model-family classification evaluates the identifying information and utility of these characteristics.

  • Sections 1–3 clarify this progression. Proposition 1 in Section 3.2 and its proof in Appendix C.5 express the relationship between projections and attention-weighted token state differences.
  • Section 5.4 and Table 2 now include REEF and HuRef-ICS. Appendices I.3–I.4 give their evaluation protocols and full predictions; Appendix I.5, Tables 30–31, reports the static projection controls.
  • Appendix F.1, Table 13, adds raw-energy variation among equal-length prompts for all 23 reference models.

Responses to Weaknesses

W1: We further analyzed the raw energy values to examine how high cross-prompt Pearson correlations relate to input-dependent variation. For Llama-3.2-1B, the median prompt-pair Pearson correlation of the DlD_l profile was 0.9990 across 1,000 prompts. Yet among 64 prompts with exactly 29 tokens, the raw value at layer 9 ranged from 2.049 to 2.406 at the 5th–95th percentiles. Energy values varied across prompts even with projection weights and token count fixed. Thus, DlD_l is not simply a readout of the layerwise norm of WOWVW_OW_V. The observed characteristic is that energy values vary with the input under fixed projection weights, while their layerwise shapes remain consistent. We report the analysis for all 23 reference models in Appendix F.1, Tables 12–13.

To explain this relationship, we added Proposition 1 in Section 3.2:

\operatorname{tr}\\!\left((M^{(l,h)})^\top M^{(l,h)}\Sigma^{(l,h)}\right), \qquad \Sigma^{(l,h)}=\frac{1}{T}\sum_{i,j}\alpha_{ij}^{(l,h)} d_{ij}^{(l)}(d_{ij}^{(l)})^\top .$$ Here, $\Sigma^{(l,h)}$ is computed from each prompt's attention probabilities and token state differences. This expression shows how projection magnitude and direction, together with input-dependent token state differences, determine the energy. The proof appears in Appendix C.5. We also tested how well static weight statistics identify model families. OV norm and MAGE-1000 both obtained 25/30, but their errors occurred on different models. Q1 explains how this comparison relates to our research perspective and presents the detailed results. **W2**: Separately counting the existing predictions for the Llama-block families gives Llama 4/4, Qwen 8/10, and SmolLM2 2/2, or 14/16 overall. We retained the original 23-model bank, all six candidate families, and the 1,000-prompt setting. Family-identifying information was therefore also observed among families with related block structures. The five errors in the full evaluation comprise two Qwen, one Gemma, and two Pythia models. Family-wise counts and misclassification costs now appear in main-text Table 3. The DeepSeek reference consists of the two checkpoints noted in the review, and the ordinary unseen queries include no native DeepSeek model. DeepSeek remains a candidate in the 14/16 result above. Thus, the available results cover Llama, Qwen, and SmolLM2 queries; identifying native DeepSeek queries requires a separate evaluation. We also examined why the two Qwen2.5-14B models were classified as DeepSeek. Decomposing the costs along the original shared DTW paths shows that $D_l$ and $\widetilde{D}_l$ were closer to Qwen for both models. In contrast, $S_l$ was closer to DeepSeek, and its cost difference exceeded the combined difference in the other two components, producing the DeepSeek predictions. Section 5.7 presents hypotheses about how the observed profiles arise and explains the need to distinguish the contributions of architecture and training to test these hypotheses. Q2 also explains our decision regarding the config-metadata comparison. **W3**: We added a citation to REEF and comparisons with both REEF and HuRef-ICS using the same 23-model bank and 30 unseen models. The results appear in Section 5.4, Table 2, with the adaptation procedures and all model-level predictions in Appendices I.3–I.4. Q3 describes the comparison settings and results.
AuthorsOpenReview ↗

Response to Reviewer rqoJ (Part II)

W4: The distillation examples in the Introduction motivate the difficulty of establishing how models were developed and where they originated. We revised the text to connect this motivation to our question about internal self-attention characteristics and make clearer that we study models with accessible weights and attention. The DeepSeek evaluation in Section 5.6 examines whether base-family characteristics are retained in models initialized from Qwen/Llama checkpoints and trained on data generated by DeepSeek-R1. The ground-truth family is that of the starting checkpoint documented in the model card.

We combined the three metrics to capture different aspects of token interactions. Subsequent evaluations of individual metrics and their combinations showed differences both in the number of correct predictions and in which models were misclassified. For example, DlD_l alone and the three-metric combination made different predictions for seven of the 30 models. The combination correctly classified four models that DlD_l missed, while DlD_l correctly classified three models that the combination missed. These differences suggest that the metrics may summarize different aspects of models' internal characteristics.

Our central finding is that the three energy metrics defined from a matrix-weighted graph representation exhibit recurring layerwise shapes across diverse prompts and similarities among models within the same family. Model-family classification is an application used to evaluate the utility of these characteristics. We also view the high classification accuracy of DlD_l alone as evidence that these energy characteristics carry useful family-identifying information.

Responses to Questions

Q1: Our starting point is to interpret token interactions in self-attention as a directed matrix-weighted graph. Our main finding is that the energy metrics defined from this representation exhibit recurring layerwise shapes across diverse prompts and similarities between models within the same family. Model-family classification is an application used to evaluate the identifying information and utility of these characteristics. The name MAGE (Matrix-weighted Attention Graph Energies) reflects this perspective.

An edge weight in this graph represents both the aggregation weights between tokens, given by attention probabilities, and the vector transformation given by the value–output projection. Since WOWVW_OW_V alone does not express the input-dependent aggregation weights between tokens, we include both components through αijWOWV\alpha_{ij}W_OW_V.

We also agree that weight-only controls are useful for examining how much of the profile shapes and family-identification results can be explained by static weight statistics. We therefore evaluated OV-, WV-, and WO-only controls using the same 23-model reference bank, 30 unseen models, and comparison procedure. The requested OV-norm feature averages the headwise Frobenius norms within each layer:

ulOV=1Hl∑h∣M(l,h)∣F.u_l^{\mathrm{OV}}=\frac{1}{H_l}\sum_h\\|M^{(l,h)}\\|_F.

We apply the existing 64-point interpolation and z-score normalization to each profile, average the model profiles within each family to form its representative, and compare them using window-3 DTW.

MethodEvaluation inputCorrect / Evaluated
OV normProjection weights25/30
OV squared norm (secondary control)Projection weights23/30
WV normProjection weights22/30
WO normProjection weights20/30
MAGE-100100 prompts24/30
MAGE-10001,000 prompts25/30

OV norm and MAGE-1000 both correctly classified 25 of the 30 models. However, MAGE correctly classified all five models that OV norm missed, and OV norm correctly classified all five that MAGE missed. For example, OV norm classified Gemma-2-9B and 9B-it as DeepSeek, whereas MAGE correctly identified both as Gemma. Conversely, OV norm correctly identified Qwen2.5-14B and 14B-Instruct as Qwen, whereas MAGE classified both as DeepSeek.

Thus, the two representations led to different model-level family predictions despite having the same overall accuracy. The raw-energy analysis in W1 and Proposition 1 show how MAGE energies reflect both input-dependent token state differences and projections. We revised the manuscript to explain the recurring layerwise structures observed in these energies together with their use in classification. Feature definitions, the comparison procedure, and predictions for every model are included in Appendix I.5, Tables 30–31.

AuthorsOpenReview ↗

Response to Reviewer rqoJ (Part III)

Q2: We would like to clarify the purpose of model-family classification in our study. Our starting point is to interpret token interactions in self-attention as a directed matrix-weighted graph. Our main finding is that the three energy metrics defined from this representation exhibit recurring layerwise shapes across diverse prompts and similarities among models within the same family. Model-family classification is an application used to evaluate the identifying information and utility of these characteristics.

A config-only classifier evaluates how well architecture metadata distinguishes family labels. This is a relevant question for the classification application, but it serves a different purpose from directly examining the recurring structures and characteristics of input-dependent attention energies studied in our work. We therefore did not extend the scope of our comparisons to classification using configuration metadata. Instead, we evaluated weight-only controls to examine how much of the classification results can be explained by static statistics of the projections included in the energy definition, and compared with REEF and HuRef-ICS, which construct fingerprints from internal model characteristics.

In follow-up work, we aim to examine how model architecture and tokenizer choice affect MAGE's energy metrics and layerwise profiles. Section 5.7 presents hypotheses about how the profiles arise and outlines controlled experiments to distinguish the effects of training and model design.

Q3: We completed the REEF comparison using the same 23-model reference bank and 30 unseen models. REEF and MAGE used the same 100 prompts.

MethodEvaluation inputCorrect / Evaluated
MAGESame 100 prompts24/30
REEFSame 100 prompts24/30
HuRef-ICSFull Wikipedia token frequencies and model weights21/30

Following the official REEF implementation, we extracted decoder representations at each prompt's final input token, standardized the features, and computed linear CKA between models. To compare models with different layer counts, we matched the relative depth of the reference layer in the official implementation. For family classification, we averaged query–reference CKA scores within each family and selected the family with the highest average. Appendix I.3 specifies this adaptation procedure.

REEF and MAGE both obtained 24/30 with the same 100 prompts. REEF correctly classified 10/10 Qwen and 5/10 Gemma models, while MAGE obtained 7/10 and 9/10, respectively. The methods had the same overall correct count but different family-wise strengths. Table 28 reports predictions for every model.

In addition to the requested REEF comparison, we added HuRef-ICS in Appendix I.4, Table 29. It uses embeddings selected through full-Wikipedia token frequencies and QK, VO, and MLP invariant terms from the final two layers. Averaging query–reference similarities within each family gives 21/30. Both comparisons also appear in Section 5.4, Table 2.

Reviewer_o9wCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MAGE, a white-box fingerprinting framework tailored for LLM family attribution. The core insight is to represent the self-attention mechanism of each transformer layer as a matrix-weighted graph, where edge weights are defined as the product of post-softmax attention probabilities and the OV (output-value) transformation matrix M^((l,h))=W_O^((l,h)) W_v^((l,h)). Building on this representation, three per-layer energy metrics—Signed Attention Energy S_l, Attention Energy D_l, and Normalized Attention Energy (D_l ) ̃—are derived and aggregated across layers to form a depth-interpolated, z-scored profile. Classification is performed via DTW-based comparison against family prototypes constructed from 23 reference models spanning 6 families. The framework is evaluated on 30 out-of-library models (achieving 25/30 correct identifications), DeepSeek-distilled models (3/4 lineage accuracy), and LoRA-SFT adapted models (32/32 retained lineage).

Critical strengths
  1. Novel formulation connecting attention mechanics to model-level fingerprinting. Representing attention edges as α_ij M rather than scalar α_ij is a principled step beyond prior attention-graph baselines (e.g., Dirichlet energy). The use of W_o W_v as the edge transformation operator aligns with mechanistic interpretability research (Transformer Circuits), which identifies the OV matrix as the primary operator responsible for writing information into the residual stream.
  2. Solid engineering execution. Key implementation choices include hook placement at the post-Norm, pre-Attention stage, factorized computation that avoids explicit materialization of the C×C matrix, and attention-reconstruction validation (achieving ~1e-3 relative Frobenius error). The overall pipeline is reproducible and numerically stable.
  3. Empirically promising results on practical threat models. The framework effectively addresses three scenarios of critical concern in LLM provenance tracking and IP protection: (a) out-of-library models, (b) distilled variants, and (c) LoRA-SFT adapted checkpoints.
  4. Cross-prompt stability is well demonstrated. Pearson correlations exceeding 0.95 across 1000 prompts support the claim that energy profiles represent model-level statistical invariants, rather than artifacts dependent on input prompts.
Critical weaknesses
  1. Incomplete Ablation of the Core Design Choice. The paper’s central claim is that matrix-weighted edges (αij×WOWV) outperform scalar edge weights. While the authors report a scalar-only baseline (achieving 10/15 vs. 13/15 on a subset of the test set), three critical control variants are missing from the ablation study.
  2. Closed-Set Classifier with Unclear Open-Set Behavior. The DTW-based 1-NN classifier operates over a fixed set of 6 known families. No explicit open-set rejection threshold or confidence calibration method is provided. In real-world deployment scenarios (e.g., detecting unauthorized distillation), the system must be capable of handling models from unseen families without forcing a classification into one of the known categories. While the 30 out-of-library test results are promising, they are evaluated under the assumption that the ground-truth family is always among the 6 reference families.
  3. Small Reference Set and Baseline Gap • 23 reference models across 6 families is a relatively limited dataset for a method claiming general-purpose fingerprinting capabilities. • Baseline comparisons are restricted to scalar attention energy and simple statistical features. Direct comparison against recent white-box fingerprinting works (e.g., weight-based, activation-based, or spectral methods proposed in 2024–2025) is absent. • No comparison is provided against frequency-domain or weight-subspace methods that also claim to exhibit family-level invariance.
  4. Robustness Boundaries Are Unexplored • No evaluation is conducted under quantization (INT8/INT4), which is a standard deployment practice for most production LLMs. • No evaluation covers architectural mismatch scenarios (e.g., cross-family distillation where the student model’s architecture differs from the teacher’s). • No stress testing is performed on very short prompts (token count < 32), where energy estimates typically exhibit higher variance.
Questions for authors
  1. Ablation: Can you add OV-only, random-matrix, and WV/WO-only ablation variants on the same 30-model benchmark?
  2. Open-set:How does the method perform when the query model belongs to a family not included in the reference set? Is there a natural rejection threshold derived from DTW distance values?
  3. Quantization: Have you tested whether INT8/INT4 quantization alters energy profiles enough to disrupt classification performance?
  4. Failure analysis:For the 5/30 out-of-library misclassifications and 1/4 distillation error, which layers contributed most to the incorrect predictions?
4 comments
AuthorsOpenReview ↗

Response to Reviewer o9wC (Part I)

Thank you for your careful review. Your suggestions on controls and practical evaluation conditions helped us strengthen the comparisons and robustness analysis.

We revised the manuscript to explain more clearly how the research perspective and main observations lead to the classification experiments. We represent token interactions in self-attention as a directed matrix-weighted graph and observe recurring layerwise shapes across prompts and within-family similarities in the resulting attention energy metrics. We then use these characteristics in a model-family classifier to evaluate their identifying information and practical utility. The revision follows this progression:

  • Sections 1–3 connect the motivation for the graph representation and metrics to the properties we investigate. Proposition 1 in Section 3.2 describes how projections and attention-weighted token state differences jointly contribute to the energies. Its proof appears in Appendix C.5, and general graph-theoretic background has moved to Appendix C.1.
  • Section 5.4 and Table 2 add comparisons with REEF and HuRef-ICS. Experimental settings and model-level predictions appear in Appendices I.3–I.4.
  • Section 5.6 and Appendix G.4 add INT8/INT4 robustness results, and Appendix J reports exploratory open-set analysis.

Responses to weaknesses

W1: We used the suggested controls to examine the role of each component and expanded the theoretical explanation. The existing scalar control replaces the value–output transformation with the identity matrix. We have now also evaluated OV-, WV-, and WO-norm controls on 30 unseen models. Q1 explains how this comparison relates to our research perspective and presents the results, with experimental details added to Appendix I.5.

We added Proposition 2 and its proof in Appendix C.6 to help interpret the random-matrix control. With token representations and attention probabilities fixed, an input-independent random projection satisfying E[M(l,h)]=0\mathbb{E}[M^{(l,h)}]=0 and E[(M(l,h))⊤M(l,h)]=γˉ(l,h)I\mathbb{E}[(M^{(l,h)})^\top M^{(l,h)}]=\bar\gamma^{(l,h)}I gives

\mathbb{E}[D_l]=\frac{1}{H}\sum_h\bar\gamma^{(l,h)}\operatorname{tr}(\Sigma^{(l,h)}).$$ Here, $\Sigma^{(l,h)}$ is the attention-weighted second moment of token state differences. The expectation is over random projection draws for a fixed input. The first identity shows that the positive and negative contributions to $S_l$ cancel in expectation. The second shows how the expected $D_l$ depends on projection scale and the attention-weighted magnitude of token state differences. We report the full classification results of the random-matrix control motivated by this theoretical reference in Q1 and have added them to Appendix C.6, Table 6. **W2**: To explore an open-set extension, we removed one family from the bank at a time and evaluated whether queries from that family could be identified as unknown. We compared raw DTW costs with a score that accounts for family-dependent distance scales. Q2 and Appendix J, Table 32 present the results and their implications for a rejection threshold. **W3**: We compared representation-based REEF and HuRef's weight-based invariant-term cosine similarity (HuRef-ICS) on the same 23-model bank and 30 unseen models. Both methods compute similarity between a query and each reference. To apply them to family classification, we averaged these similarities over references in each family and selected the family with the highest average. REEF uses CKA between decoder representations at each prompt's final input token. We selected the comparison layer based on the official implementation, adjusting for model depth. HuRef-ICS uses embeddings selected by token frequencies in the full Wikipedia corpus, together with QK, VO, and MLP invariant terms from the final two layers. | Method | Evaluation input | Correct / Evaluated | |---|---|---:| | MAGE | Same 100 prompts | 24/30 | | REEF | Same 100 prompts | 24/30 | | HuRef-ICS | Full Wikipedia token frequencies and model weights | 21/30 | We added the results to Section 5.4, Table 2, with detailed comparison procedures and model-level predictions in Appendices I.3–I.4.
AuthorsOpenReview ↗

Response to Reviewer o9wC (Part II)

W4: Given the widespread use of weight quantization in LLM deployment, we added HQQ INT8/INT4 results for six models to Section 5.6 and Appendix G.4. Both INT8 and INT4 achieved 6/6 correct predictions with the unquantized bank fixed, and these results were retained when each query's original checkpoint was excluded from the bank. We discuss profile changes and metric-extraction checks in Q3.

A systematic evaluation of teacher–student architecture mismatch requires a set of models with comparable training data, distillation procedures, and other key conditions. If public checkpoints meeting these conditions are unavailable, the comparison requires training and evaluating models under controlled conditions. Constructing such a model set and conducting the controlled training would require a larger study than we could undertake during this revision. We view teacher–student architecture mismatch as an important evaluation condition for the follow-up work discussed in Section 5.7, which aims to distinguish the contributions of architecture and training.

For short prompts, we would like to highlight the input lengths and profile-consistency results from our existing evaluation. Our 1,000-prompt set includes inputs shorter than 32 tokens. For example, the input lengths for Llama-3.2-3B range from 14 to 160 tokens, with a median of 31 tokens (Appendix D.4). Across the full prompt set, which includes these varied input lengths, the layerwise profiles of all three metrics generally showed high prompt-pair Pearson correlations. We report the results for all 23 reference models in Appendix F.1, Table 12.

Responses to Questions

Q1: Our starting point is to interpret token interactions in self-attention as a directed matrix-weighted graph. Our main finding is that the energy metrics defined from this representation exhibit recurring layerwise shapes across diverse prompts and similarities between models within the same family. Model-family classification is an application used to evaluate the identifying information and utility of these characteristics. The name MAGE (Matrix-weighted Attention Graph Energies) reflects this perspective.

An edge weight in this graph represents both the aggregation weights between tokens, given by attention probabilities, and the vector transformation given by the value–output projection. Since WOWVW_OW_V alone does not express the input-dependent aggregation weights between tokens, we include both components through αijWOWV\alpha_{ij}W_OW_V.

We also agree that weight-only controls are useful for examining how much of the profile shapes and family-identification results can be explained by static weight statistics. We therefore evaluated OV-, WV-, and WO-only controls using the same 23-model reference bank, 30 unseen models, and comparison procedure. We constructed the following layerwise features from the Frobenius norms of the projections:

u_l^{\mathrm{WV}}=\frac{1}{H_l}\sum_h\\|W_V^{(l,g(h))}\\|_F,\qquad u_l^{\mathrm{WO}}=\frac{1}{H_l}\sum_h\\|W_O^{(l,h)}\\|_F.$$ For GQA, each query head uses its corresponding value head. We apply the existing 64-point interpolation and z-score normalization to each profile, average the model profiles within each family to form its representative, and compare them using window-3 DTW. | Method | Evaluation input | Correct / Evaluated | |---|---|---:| | OV norm | Projection weights | 25/30 | | OV squared norm (secondary control) | Projection weights | 23/30 | | WV norm | Projection weights | 22/30 | | WO norm | Projection weights | 20/30 | | MAGE-100 | 100 prompts | 24/30 | | MAGE-1000 | 1,000 prompts | 25/30 | We added the feature definitions, evaluation procedure, and predictions for every model to Appendix I.5, Tables 30–31. OV norm and MAGE-1000 both correctly classified 25 of the 30 models. However, they misclassified different models: OV norm correctly classified all five models that MAGE-1000 missed, and MAGE-1000 correctly classified all five that OV norm missed. Static projection statistics were useful for family identification, and the two methods made different model-level predictions despite having the same overall accuracy.
AuthorsOpenReview ↗

Response to Reviewer o9wC (Part III)

Q1 (continued): Random-matrix control

We also completed the random-matrix control. We retained the original model's attention probabilities and token representations, replacing only the projection used in the energy calculation with a random matrix. We constructed this matrix as a product of Gaussian factors and matched its Frobenius norm to the original projection for each head. Each random matrix was fixed across prompts. Using the same 100 prompts, we constructed profiles for all 23 references and 30 queries separately for each of three seeds and applied the existing MAGE preprocessing and DTW classification procedure.

ConditionCorrect / EvaluatedAccuracy
Random matrix — seed 4218/3060.0%
Random matrix — seed 12321/3070.0%
Random matrix — seed 202617/3056.7%
Random matrix mean18.67/3062.2%
MAGE — existing 100-prompt result24/3080.0%

Randomizing the matrix structure of the value–output projections while preserving each head's projection norm reduced family classification accuracy for all three seeds. This supports the contribution of the original projection structure to the family-identifying performance of the energy profiles. We added the experimental settings and results after the discussion of Proposition 2 in Appendix C.6, together with Table 6.

Q2: We conducted leave-one-family-out analysis using the existing 1,000-prompt profiles of the 23 reference and 30 unseen models. In each fold, we removed all references from one of Llama, Qwen, Gemma, SmolLM2, or Pythia and treated queries from that family as unknown. The other queries remained known, and we evaluated the same 30 queries in each of the five folds. DeepSeek has reference models but no ordinary unseen queries, so it was not used as a held-out family.

Let f^\hat f denote the retained family with the lowest DTW cost to the query, and let df^(Q)d_{\hat f}(Q) denote that cost. We compared the raw cost with a score divided by the within-family reference distance:

s_{\mathrm{scaled}}(Q)=\frac{d_{\hat f}(Q)}{r_{\hat f}}.$$ To calculate $r_f$, we left out each retained reference in turn and computed its DTW cost to the mean profile of the remaining references in the same family. We used the median of these costs as $r_f$. This measures the scale of profile differences within a family using only the references. Both methods retain the family prediction obtained from the lowest DTW cost and reject the query as unknown only when its score exceeds the threshold. We evaluated performance across the full threshold range. The AUROC values below measure how well known and unknown queries are distinguished, with unknown as the positive class. | Held-out family | Unknown queries | Raw DTW AUROC | Family-scaled AUROC | |---|---:|---:|---:| | Llama | 4 | 0.3269 | 0.9519 | | Qwen | 10 | 0.5350 | 0.9900 | | Gemma | 10 | 0.9900 | 0.4850 | | SmolLM2 | 2 | 0.4464 | 1.0000 | | Pythia | 4 | 0.7788 | 0.4423 | | Macro average | — | 0.6154 | 0.7738 | Macro AUOSCR, which evaluates correct known-query classification together with unknown rejection, also increased from 0.5402 to 0.6793. Accounting for within-family distances improved discrimination for Llama, Qwen, and SmolLM2, but reduced it for Gemma and Pythia. With these substantial differences across families, the current results make it difficult to identify a natural common rejection threshold from raw DTW costs alone. An open-set extension would need to account for both family-dependent distance scales and overlap between known and unknown scores. We report the settings and full results in Appendix J, Table 32, and added open-set extension as a future direction in Section 6.
AuthorsOpenReview ↗

Response to Reviewer o9wC (Part IV)

Q3: We evaluated six models in FP16, HQQ INT8, and HQQ INT4: Llama-3.2-1B, Llama-3.2-1B-Instruct, Qwen2.5-1.5B, Qwen2.5-1.5B-Instruct, SmolLM2-360M, and SmolLM2-1.7B. We kept the bank fixed using the existing 1,000-prompt profiles of the 23 unquantized models. We also retained the classifier settings and used the same 100 prompts for every query condition.

Query conditionFixed bankQuery-specific source checkpoint excluded
FP166/66/6
HQQ INT86/66/6
HQQ INT46/66/6

For source-checkpoint exclusion, we removed each query's original checkpoint and recomputed family representatives from the remaining 22 references.

We quantized the attention and MLP projection weights in every decoder layer. MAGE used the dequantized WVW_V and WOW_O, so its metric computation reflected the same projections as the forward pass.

We assessed profile changes by comparing FP16 and quantized profiles obtained from the same prompt. We computed Pearson correlations on the native layers and took the median over 100 prompts for each model and metric. The lowest of these medians across the six models and three metrics was 0.999989 for INT8 and 0.988633 for INT4.

The raw energy values also changed. Under INT4, the relative Frobenius change in the energy array reached 9.24% compared with FP16, while profile shape and correct family predictions were retained. We report the settings, checks, and full results in Appendix G.4, Tables 19–20, and summarize them in Section 5.6.

Q4: Using the existing 1,000-prompt profiles and unchanged classifier, we decomposed the DTW costs by depth for the five unseen-model errors and the one distilled-model error. The depths and metrics that contributed most to the incorrect predictions are summarized below.

The cost difference is the true-family cost minus the incorrectly predicted-family cost. We divided the 64-point grid into four equal-sized blocks and report the block with the largest difference. The last column identifies the metric with the largest difference over the full paths. The layer ranges in parentheses approximately map the interpolation coordinates back to the original layers.

ModelMain depth block (approx. native layers)Block cost differenceMain metric over full paths
Pythia-70M25–50% (2–4)0.1843D~l\widetilde{D}_l
Pythia-12B75–100% (27–36)0.5815D~l\widetilde{D}_l
Qwen2.5-14B75–100% (36–48)0.3580SlS_l
Qwen2.5-14B-Instruct75–100% (36–48)0.3259SlS_l
Gemma-2-27B0–25% (1–12)0.2285D~l\widetilde{D}_l
DeepSeek-R1-Distill-Qwen-14B75–100% (36–48)0.3682SlS_l

For the three Qwen-based 14B models, the largest cost difference favoring the incorrect family occurred at the final grid point, corresponding to layer 48. Over the full paths, DlD_l and D~l\widetilde{D}_l had lower costs to Qwen. However, the cost advantage for DeepSeek in SlS_l exceeded the combined advantage for Qwen in the other two metrics, leading to the DeepSeek predictions.

The largest contribution favoring the incorrect family came from the middle 25–50% depth block for Pythia-70M and the final 25% for Pythia-12B. For both models, D~l\widetilde{D}_l made the largest contribution over the full paths. For Gemma-2-27B, the largest difference occurred at the first-layer coordinate. The early-depth contribution favoring Pythia was substantially offset at intermediate depths, leaving a final cost difference of 0.0432.

We obtained these layerwise contributions by retaining each family's original DTW path shared by the three metrics, using the family-level costs reported in revised Table 3 and Appendix G.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.