acceptodds
Under review as a conference paper at ICLR 2027

Gradient Entropy Dynamics Characterize Model Adaptability

Abstract

Gradients guide how a neural network learns, but can they also reveal how well a pretrained model will adapt to a downstream task? We uncover a systematic rise in the Rényi entropy of gradients across neuron units as task learning begins. This rise occurs when dominant gradient components diminish faster in relative terms, distributing gradient mass more evenly across units. Beyond this observation, we establish a theoretical connection between gradient Rényi entropy and adaptation dynamics, showing that higher gradient entropy yields a smaller upper bound on the remaining loss gap. This connection provides a theoretical basis for assessing adaptability through gradient entropy and motivates a Rényi-1/2 gradient-entropy score for ranking pretrained models, as well as a valley-restricted variant that limits the influence of high-gradient-energy regions. Experiments with CNN- and ViT-based models across ten downstream datasets show that the proposed score achieves stronger average ranking correlations than the compared transferability estimators. Even without labels, the method remains competitive with, and can outperform, most label-based alternatives. These findings highlight gradient entropy as an informative measure of pretrained-model adaptability to downstream tasks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
model ranking, transfer learning, gradient entropy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PsquOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates Rényi entropy of normalized squared gradients as a signal of pretrained-model adaptability. It derives a conditional characterization of entropy growth and convergence bounds involving the effective dimension eS1/2=∥g∥12/∥g∥22e^{S_{1/2}}=\|g\|_1^2/\|g\|_2^2. It then proposes initial gradient entropy, optionally restricted to low-gradient-energy regions, as a supervised or label-free transferability score. Validation includes toy experiments, ResNet training trajectories, ranking 11 CNNs and six self-supervised ViTs across ten datasets, and hyperparameter ablations. Results are promising, particularly for weighted Kendall correlation, although gains are not consistent across metrics.

Critical strengths
  1. On ViTs, supervised weighted Kendall correlation improves from the best baseline’s 0.405 to 0.527; the label-free variant reaches 0.503. This suggests practical value when downstream labels are unavailable.

  2. The entropy-evolution identity explains when gradient mass becomes more dispersed, while the effective-dimension formulation motivates the choice of Rényi order 1/21/2. This gives the proposed score a useful mathematical interpretation.

Critical weaknesses
  1. The convergence results concern a particular sign-gradient update and depend on smoothness, PL constants, initial loss gaps, and an entropy lower bound along the trajectory. These quantities can differ across models, and a smaller upper bound does not establish an ordering of actual losses or downstream test accuracies. Figure 2 also contains early entropy decreases, leaving the initial-entropy assumption in Corollary 4.6 insufficiently supported. The theoretical positioning should additionally address prior gradient-density analyses in Bernstein et al., signSGD: Compressed Optimisation for Non-Convex Problems, ICML 2018, which study a closely related normalized ℓ1/ℓ2\ell_1/\ell_2 quantity.

  2. The score depends on the number of gradient units, and Figure 4 already shows strong CNN correlations at α=0\alpha=0, where it counts active coordinates. This raises an unresolved model-size/support confound. Moreover, the CNN weighted-Kendall gain over NCTI is only 0.011, without uncertainty estimates, and CIFAR-10 is used for both hyperparameter selection and evaluation. On ViTs, supervised Spearman correlation is 0.537 versus LogME’s 0.634, so the claimed superiority needs qualification.

Questions for authors
  1. What exactly is gig_i: a parameter derivative, a signed neuron-wise average, or another aggregation? Which parameters are differentiated, how is the supervised head initialized or trained, and how are gradients aggregated across batches? Also specify the contrastive objective, augmentations, and projection head.

  2. What supports applying the initial-entropy argument to the benchmark fine-tuning procedures, particularly given the early entropy decreases in Figure 2 and the model-dependent constants in the bounds?

  3. How do the conclusions change with model-size/active-support controls, a normalized effective-dimension score, repeated scoring runs, and evaluation excluding the hyperparameter-selection dataset?

2 comments
AuthorsOpenReview ↗

Response [1/2]

We sincerely thank the reviewer for the thoughtful comments and constructive feedback. We have carefully considered these concerns and revised our manuscript to address them.

W1 & Q2: Theoretical validity and positioning of gradient entropy & applicability of initial entropy to practical fine-tuning.

In Section 3, we show that gradient entropy can increase during learning. We interpret this trend as follows: although individual optimization steps may not uniformly improve all model components, the overall learning process progressively captures task-relevant patterns from the training data. At the early stages of training, different components of the model may respond differently to various task factors, depending on both the input data's sensitivity to these factors and the information the model has already learned. This leads to heterogeneous gradient responses across neuron units. As training progresses, the model gradually adapts to these factors, and the gradient mass becomes more evenly distributed across different units, thereby increasing gradient entropy. Motivated by this observation, we hypothesize that, for a given downstream task, a higher initial gradient entropy, indicating a more evenly distributed gradient mass across neuron units, reflects a broader task-responsive representation and can serve as a useful signal for predicting downstream performance. This intuition is further supported by our theoretical analysis in Section 4.

Specifically, we introduce a latent-factor model in which each pretrained model captures a subset of task-relevant factors. For model mm, let Tm=Sm∩TT_m=S_m\cap T and km=∣Tm∣k_m=|T_m| denote its task-relevant coverage. Proposition 4.2 shows that the initial task-responsive gradients are determined by the factors captured by the model, and hence its effective dimension is

{\|Q_m\beta_{T_m}\|_2^2}.$$ Under the isotropic task-coefficient assumption, Theorem 4.3 further shows that $$\mathbb E_\beta[d_m] =1+\frac{2}{\pi}(k_m-1),$$ while the expected Bayes-optimal downstream accuracy also increases strictly with $k_m$. Thus, we do not assume that entropy is related to performance; rather, both quantities are derived from the same task-relevant coverage $k_m$, which establishes their consistent expected ranking across candidate models. This analysis is independent of the subsequent entropy growth during fine-tuning. The role of Section 3 is to motivate gradient entropy from learning dynamics, while the revised Section 4 provides the direct theoretical basis for using initial gradient entropy to compare pretrained models. **Q1: Clarification of gradient computation and contrastive learning setup.** $g_i$ is the neuron-wise gradient. The gradient is computed by aggregating the gradients over the entire dataset, i.e., by taking the mean of the per-sample gradients across all data points. For supervised scoring, we use a linear classifier head initialized with Kaiming-uniform initialization to produce the logits. The classifier head is kept fixed and is not updated during gradient computation. For unsupervised scoring, we do not use an additional projection head; instead, we directly use the penultimate features. We compute the self-supervised loss using the same data augmentation strategy as MoCoV2. We have added the details in the revised appendix.
AuthorsOpenReview ↗

Response [2/2]

W2 & Q3: Robustness, model-Size confounding, and generalizability of the proposed score.

Rényi entropy sensitivity is driven more by the variability of nonzero gradient magnitudes. We find that transformer models are more sensitive to small α\alpha due to their greater nonzero gradient heterogeneity. Compared with CNNs, ViT has a broader gradient magnitude distribution, containing both extremely small and relatively large nonzero gradients. We demonstrate the comparison of gradient magnitude between ResNet-18 and ViT-16/B on CIFAR in the following.

|g| / max|g|ResNet-18 ratio(%)ViT-B/16 ratio %
= 018.3160.000
(0, 1e-12]0.0020.000
(1e-12, 1e-10]0.0040.005
(1e-10, 1e-9]0.0080.044
(1e-9, 1e-8]0.0490.271
(1e-8, 1e-7]0.2770.952
(1e-7, 1e-6]1.9523.413
(1e-6, 1e-5]11.2566.469
(1e-5, 1e-4]37.44313.537
(1e-4, 1e-3]28.88438.879
(1e-3, 1e-2]1.80134.983
(1e-2, 1e-1]0.0061.433
(1e-1, 1]0.0000.013

This explains why α=0\alpha=0 has similar results to α=0.5\alpha=0.5 in the CNN benchmark but different results in the ViT benchmark.

Regarding model size as an indicator of model rank, we agree that performance can be correlated with model size, consistent with the scaling laws. Accordingly, we report the performance τw\tau_w/ρ\rho across model ranks, using model size DD and normalized effective dimension deff/Dd_{eff}/D as the ranking criterion, below:

indicatorCNN supCNN unsupViT supViT unsup
#params0.602/0.6610.602/0.6610.395/0.3280.395/0.328
Normalized eff dim-0.358/-0.475-0.320/-4370.301/0.4970.163/0.280
entropy(sup)0.706/0.8070.695/0.7800.565/0.5750.482/0.632

Although model size exhibits a positive correlation with performance, its correlation is substantially weaker than that of entropy. Moreover, for CNNs, the results based on the normalized effective dimension show that deffd_{\mathrm{eff}} is not always aligned with model size, suggesting that the proposed score captures information beyond model scale alone. These results further support entropy as a more suitable indicator for ranking models.

Regarding the repeated runs, we report the results (τw\tau_w/ρ\rho) on other 2 seeds (seed =1,2) in the following. Note that we do not require model updates, so the seeds mostly affect the data sampling and augmentation. We present the results as follows:

SettingRunCaltech101CarsDTDAvg
CNN unsup00.758/0.9270.826/0.8550.787/0.8560.791/0.879
10.758/0.9270.849/0.8730.819/0.8880.809/0.896
20.758/0.9270.876/0.8820.840/0.8970.825/0.902
CNN sup00.646/0.8820.757/0.7730.854/0.9200.752/0.858
10.706/0.9090.757/0.7730.766/0.8430.743/0.842
20.520/0.7820.778/0.7910.850/0.9160.716/0.829
ViT unsup00.682/0.8860.254/0.4290.682/0.8860.539/0.733
10.682/0.8860.254/0.4290.682/0.8860.539/0.733
20.755/0.9430.254/0.4290.682/0.8860.564/0.752
ViT supe00.755/0.9430.484/0.3710.682/0.8860.640/0.733
10.755/0.9430.484/0.3710.682/0.8860.640/0.733
20.554/0.7140.484/0.3710.620/0.7710.553/0.619

We also report the dataset-wise results (τw/ρ\tau_w/\rho) over 3 seeds with the unsupervised entropy method in ViT benchmark as follows

SeedCal101CarsCIFAR100DTDAircraftFlowersFoodPetsSUN397Avg
00.682/0.8860.254/0.4290.267/0.6000.682/0.8860.676/0.6570.414/0.6000.188/0.2000.414/0.6000.762/0.8290.482/0.632
10.682/0.8860.254/0.4290.267/0.6000.682/0.8860.676/0.6570.414/0.6000.188/0.2000.414/0.6000.676/0.6570.472/0.613
20.755/0.9430.254/0.4290.267/0.6000.682/0.8860.676/0.6570.414/0.6000.188/0.2000.414/0.6000.762/0.8290.490/0.638

The ranking is mostly seed-invariant except at a near-tie. For instance, in ViT unsupervised setting, on every dataset the top three and the bottom model are unchanged across all three runs, and the only movement is one adjacent swap between dino_vits8 and mae_vitb16, whose scores differ by less than the run-to-run drift.

Note that all the reported experiments in the rebuttal and revised paper are averaged across 9 datasets, excluding CIFAR-10.

Reviewer_c21SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript proposes gradient-entropy statistics for estimating how well pretrained networks transfer to downstream image-classification tasks without fully fine-tuning. It defines a distribution from squared gradient components, explores its Rényi entropy during learning, and derives an entropy derivative with a sufficient condition for increase. Under coordinate-wise smoothness, a particular scaled-sign update, and a trajectory-wise Polyak–Łojasiewicz condition, it obtains loss-gap bounds involving the order-1/21/2 effective dimension; an additional entropy-floor assumption permits an initial-checkpoint bound. The practical method ranks models using exponentiated entropy, optionally restricted to low-gradient-energy regions, with supervised or contrastive objectives. Experiments cover eleven CNNs and six self-supervised ViTs across ten datasets, reporting favorable weighted Kendall correlations and competitive label-free results. These findings support an empirical association with transfer performance.

Critical strengths

S1. The score proposed in this manuscript has an intuitive interpretation as an effective number of gradient-bearing units: it approaches one when a single component dominates and equals the number of active units when their magnitudes are equal.

S2. Choosing a pretrained model without thoroughly fine-tuning all candidates offers clear practical benefits. Section 5 assesses both supervised ImageNet CNNs and self-supervised ViTs across 10 target datasets and 10 established transferability estimators.

S3. The ViT ablation provides positive evidence that the distribution of gradient magnitudes contributes to the ranking signal in that setting.

Critical weaknesses

W1. The theory does not establish that initial gradient entropy can rank different pretrained models. Section 4 shows how gradient entropy enters a bound on loss reduction for a particular optimization update. However, the bound also depends on gradient magnitude, curvature, the initial loss gap, and other model-specific constants. These quantities can differ substantially across pretrained models. Therefore, a model with higher initial gradient entropy does not necessarily converge faster or achieve a lower final loss. Moreover, a tighter upper bound does not establish a better actual outcome, even when the other constants are equal.

W2. As the authors themselves show in Lemma A.2, the proposed order-(1/2) score is [ e^{S_{1/2}(p)}=\frac{|g|_1^2}{|g|_2^2}. ] This quantity is already known as numerical sparsity, and a normalized version has been used to study gradient density and its relationship with optimization. There is a close predecessor to the convergence analysis. Karimi, Nutini, and Schmidt (2016) analyze scaled-sign updates under smoothness and PL conditions; the update-and-descent inequality in Theorem 4.3 follows as a special case of their analysis. A substantial part of Section 4, therefore, expresses established optimization results in terms of entropy. The application to pretrained-model ranking may still be original and useful. However, the paper should distinguish these contributions from existing mathematical foundations and explain what new insights or practical benefits the entropy formulation provides.

W3. Theorem 3.3 shows that entropy increases when larger gradient components decrease faster in relative terms. It does not show that this condition generally holds during learning. Observing that large gradients become smaller is insufficient: their relative rate of decrease must also be compared with that of smaller gradients. More importantly, Corollary 4.6 assumes that gradient entropy never falls below its initial value during adaptation. However, the curves in Figure 2b appear inconsistent with the assumption used to justify that score, which limits its applicability to the experiments.

Questions for authors

Why should higher initial gradient entropy predict better test accuracy across different models, when the theory only bounds training loss for a specific optimization method?

1 comment

We thank the reviewer for raising these important concerns. We have substantially revised the theoretical framework and clarified these aspects in the revised manuscript. Our detailed responses are provided below.

W1 & Q1: The previous theory does not explain why initial entropy can rank different pretrained models.

We sincerely thank the reviewer for identifying this limitation in our theoretical analysis. In the revised Section 4, we have substantially revised the theoretical framework to directly establish the relationship between initial gradient entropy and cross-model transferability, moving beyond the previous reliance on optimization-based bounds for loss reduction.

Specifically, we introduce a latent-factor model to characterize how pretrained representations encode task-relevant information. For each candidate model mm, we define Tm=Sm∩TT_m = S_m \cap T as the set of task-relevant factors captured by the model with km=∣Tm∣k_m = |T_m|. Proposition 4.2 shows that the initial task-responsive gradients are determined by the factors captured by the model, yielding

{\left\|Q_m \beta_{T_m}\right\|_2^2}$$ Building on this result, Theorem 4.3 further establishes that, under the assumptions of orthogonal mixing and isotropic task coefficients, $$\mathbb E_\beta[d_m]=1+\frac{2}{\pi}(k_m-1).$$ Moreover, the expected Bayes-optimal downstream accuracy also increases strictly with $k_m$. Thus, both expected quantities are governed by the same task-relevant coverage measure, establishing a consistent ranking of candidate models based on gradient entropy and downstream performance. This analysis provides a theoretical justification for using initial gradient entropy as an indicator of cross-model transferability. **W2: the score is numerical, and related optimization analyses already exist.** We thank the reviewer for pointing out this connection. In the revised manuscript, we develop our theoretical framework independently of the numerical sparsity lemma. Moreover, our contributions extend beyond theoretical analysis, encompassing an empirical finding, its theoretical explanation, and demonstrated performance improvements on existing transferability benchmarks. **W3: Entropy growth is conditional.** We agree that gradient entropy does not necessarily increase at every training step, as local decreases may occur due to complex optimization dynamics. Nevertheless, our visualizations reveal an overall increasing trend throughout training (Figure 1, 5, and 6). We interpret this trend as follows: although individual optimization steps may not uniformly improve all model components, the overall learning process progressively captures task-relevant patterns from the training data. At the early stages of training, different components of the model may respond differently to various task factors, depending on both the input data's sensitivity to these factors and the information the model has already learned. This leads to highly heterogeneous gradient responses across neuron units. As training progresses, the model gradually acquires information about these factors, making its gradient responses across different units more balanced and less heterogeneous, thereby increasing gradient entropy. This motivates our use of gradient entropy as an indicator of how broadly the model's units respond to the given data, rather than assuming that the model can progressively homogenizes its responses to different factors at every successive checkpoint. Importantly, our revised theoretical analysis in Section 4 does not rely on the monotonic increase of entropy or any trajectory-level entropy-floor assumption. Instead, it directly establishes a theoretical connection between the initial gradient effective dimension and expected downstream performance through task-relevant factor coverage. Therefore, the cross-model ranking result is independent of whether gradient entropy increases monotonically during fine-tuning.
Reviewer_P4HQOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies transferability estimation. The authors observe that, once a model starts learning a task, an entropy measure of the gradients across neuron units goes up, and show that the entropy does not decrease when units carrying more gradient mass decay faster in relative terms. They link the upper bound on the remaining loss gap to this entropy and propose a scoring signal based on it. Experiments on 11 CNNs, 6 self-supervised ViTs and 10 downstream datasets show a reasonable level of effectiveness.

Critical strengths

The theory is self-consistent, with simple methods and decent results. The label-free version loses little (τw: CNN 0.727→0.701, ViT 0.527→0.503). I checked part of the proofs and found no errors so far. Some of the hyperparameter choices have a theoretical justification. The paper is generally informative.

Critical weaknesses
  1. For a pre-trained model, the entropy already rises sharply at the very start of fine-tuning, i.e. S(p_0) is the value before the rise. The authors could explain further why this value can distinguish different models.
  2. The advantage in the main results is less solid than the abstract suggests, and the best configuration may depend on tuning. On CNNs, τw is 0.727 vs. 0.716 for NCTI, a margin of only 0.011. K and q are selected on CIFAR-10, which is itself one of the ten evaluation datasets, while ViTs use whole-model gradients.
Questions for authors
  1. I want to ask which parameters are the gradients computed over (the whole backbone, or a probe on top of the features)?
  2. Does the assumption of Corollary 4.6 (non-decreasing entropy) hold across the evaluated pre-trained models? Does the size of the entropy increase during fine-tuning correlate with final accuracy?
1 comment

We sincerely thank the reviewer for this insightful comment. We further clarify the theoretical foundation of our approach and provide a more direct justification for using initial gradient entropy to assess cross-model transferability.

W1: Why can initial entropy distinguish pretrained models if entropy rises only after fine-tuning begins?

In Section 3, we show that gradient entropy can increase during learning. We attribute this trend to the model's progressive adaptation to task-relevant factors. Initially, different neuron units exhibit heterogeneous gradient responses due to variations in input sensitivity and pretrained knowledge. As training progresses, gradient mass becomes more evenly distributed across units, leading to higher gradient entropy. Motivated by this observation, we hypothesize that, for a given downstream task, a higher initial gradient entropy, indicating a more evenly distributed gradient mass across neuron units, reflects a broader task-responsive representation and can serve as a useful signal for predicting downstream performance. This intuition is further supported by our theoretical analysis in Section 4.

Specifically, we introduce a latent-factor model in which each pretrained model captures a subset of task-relevant factors. For model mm, let Tm=Sm∩TT_m=S_m\cap T and km=∣Tm∣k_m=|T_m| denote its task-relevant coverage. Proposition 4.2 shows that the initial task-responsive gradients are determined by the factors captured by the model, and hence its effective dimension is

=\frac{\|Q_m\beta_{T_m}\|_1^2} {\|Q_m\beta_{T_m}\|_2^2}.$$ Under the isotropic task-coefficient assumption, Theorem 4.3 further shows that $$\mathbb E_\beta[d_m] =1+\frac{2}{\pi}(k_m-1),$$ while the expected Bayes-optimal downstream accuracy also increases strictly with $k_m$. Thus, we do not assume that entropy is related to performance; rather, both quantities are derived from the same task-relevant coverage $k_m$, which establishes their consistent expected ranking across candidate models. **W2: Robustness of performance gains and hyperparameter selection.** Regarding hyperparameter selection, we clarify that CIFAR-10 is used only for selecting the hyperparameters $K$ and $q$, and is excluded from the main evaluation results. Specifically, Tables 1 and 2 report average ranking correlations over the remaining nine datasets, ensuring that the dataset used for hyperparameter selection does not overlap with the evaluation datasets. The selected configurations are then fixed across all nine evaluation datasets, with $(K,q)=(4,3)$ for CNNs and full-model gradients for ViTs. These architecture-specific choices are determined through the same validation protocol on CIFAR-10, rather than being tuned separately on individual test datasets. Importantly, our method maintains the strongest average performance in several key comparisons. On CNNs, the supervised Gradient Entropy score achieves $\tau_w=0.706$ and $\rho=0.807$, outperforming the strongest baseline, NCTI (0.702 and 0.772, respectively). On ViTs, our supervised method achieves the highest weighted Kendall correlation 0.565, while the label-free variant achieves the best results on both metrics $\tau_w=0.482$, $\rho=0.632$. We acknowledge that the improvement in CNN weighted Kendall correlation is modest, but the results demonstrate that our method remains competitive with or outperforms existing state-of-the-art estimators across the evaluated settings. **Q1: Which parameters are the gradients computed over?** Gradients are computed with respect to the entire model, excluding the classifier. The hyperparameters $K$ and $q$ are used to identify and select model regions with relatively small gradient magnitudes. We hypothesize that these regions preserve much of the meaningful information acquired during pretraining. **Q2: Validity of the Entropy Monotonicity Assumption and Its Correlation with Downstream Accuracy.** We visualize the gradient entropy trajectories of three representative models in the Appendix Figure 6, all of which exhibit an overall increasing trend. However, this trend does not imply a strictly monotonic increase at every training step, as local fluctuations may arise from the complex optimization dynamics during training. Due to computational constraints, we cannot track the entropy trajectories of all models in the benchmark. Nevertheless, the results for ResNet-18 and ResNet-50 in Figure 6 suggest that a larger increase in gradient entropy does not necessarily translate into better downstream performance. In fact, between these two models, the one exhibiting a larger entropy increase achieves lower downstream performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.