- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates Rényi entropy of normalized squared gradients as a signal of pretrained-model adaptability. It derives a conditional characterization of entropy growth and convergence bounds involving the effective dimension . It then proposes initial gradient entropy, optionally restricted to low-gradient-energy regions, as a supervised or label-free transferability score. Validation includes toy experiments, ResNet training trajectories, ranking 11 CNNs and six self-supervised ViTs across ten datasets, and hyperparameter ablations. Results are promising, particularly for weighted Kendall correlation, although gains are not consistent across metrics.
Critical strengths
-
On ViTs, supervised weighted Kendall correlation improves from the best baseline’s 0.405 to 0.527; the label-free variant reaches 0.503. This suggests practical value when downstream labels are unavailable.
-
The entropy-evolution identity explains when gradient mass becomes more dispersed, while the effective-dimension formulation motivates the choice of Rényi order . This gives the proposed score a useful mathematical interpretation.
Critical weaknesses
-
The convergence results concern a particular sign-gradient update and depend on smoothness, PL constants, initial loss gaps, and an entropy lower bound along the trajectory. These quantities can differ across models, and a smaller upper bound does not establish an ordering of actual losses or downstream test accuracies. Figure 2 also contains early entropy decreases, leaving the initial-entropy assumption in Corollary 4.6 insufficiently supported. The theoretical positioning should additionally address prior gradient-density analyses in Bernstein et al., signSGD: Compressed Optimisation for Non-Convex Problems, ICML 2018, which study a closely related normalized quantity.
-
The score depends on the number of gradient units, and Figure 4 already shows strong CNN correlations at , where it counts active coordinates. This raises an unresolved model-size/support confound. Moreover, the CNN weighted-Kendall gain over NCTI is only 0.011, without uncertainty estimates, and CIFAR-10 is used for both hyperparameter selection and evaluation. On ViTs, supervised Spearman correlation is 0.537 versus LogME’s 0.634, so the claimed superiority needs qualification.
Questions for authors
-
What exactly is : a parameter derivative, a signed neuron-wise average, or another aggregation? Which parameters are differentiated, how is the supervised head initialized or trained, and how are gradients aggregated across batches? Also specify the contrastive objective, augmentations, and projection head.
-
What supports applying the initial-entropy argument to the benchmark fine-tuning procedures, particularly given the early entropy decreases in Figure 2 and the model-dependent constants in the bounds?
-
How do the conclusions change with model-size/active-support controls, a normalized effective-dimension score, repeated scoring runs, and evaluation excluding the hyperparameter-selection dataset?
2 comments
Response [2/2]
W2 & Q3: Robustness, model-Size confounding, and generalizability of the proposed score.
Rényi entropy sensitivity is driven more by the variability of nonzero gradient magnitudes. We find that transformer models are more sensitive to small due to their greater nonzero gradient heterogeneity. Compared with CNNs, ViT has a broader gradient magnitude distribution, containing both extremely small and relatively large nonzero gradients. We demonstrate the comparison of gradient magnitude between ResNet-18 and ViT-16/B on CIFAR in the following.
| |g| / max|g| | ResNet-18 ratio(%) | ViT-B/16 ratio % |
|---|---|---|
| = 0 | 18.316 | 0.000 |
| (0, 1e-12] | 0.002 | 0.000 |
| (1e-12, 1e-10] | 0.004 | 0.005 |
| (1e-10, 1e-9] | 0.008 | 0.044 |
| (1e-9, 1e-8] | 0.049 | 0.271 |
| (1e-8, 1e-7] | 0.277 | 0.952 |
| (1e-7, 1e-6] | 1.952 | 3.413 |
| (1e-6, 1e-5] | 11.256 | 6.469 |
| (1e-5, 1e-4] | 37.443 | 13.537 |
| (1e-4, 1e-3] | 28.884 | 38.879 |
| (1e-3, 1e-2] | 1.801 | 34.983 |
| (1e-2, 1e-1] | 0.006 | 1.433 |
| (1e-1, 1] | 0.000 | 0.013 |
This explains why has similar results to in the CNN benchmark but different results in the ViT benchmark.
Regarding model size as an indicator of model rank, we agree that performance can be correlated with model size, consistent with the scaling laws. Accordingly, we report the performance / across model ranks, using model size and normalized effective dimension as the ranking criterion, below:
| indicator | CNN sup | CNN unsup | ViT sup | ViT unsup |
|---|---|---|---|---|
| #params | 0.602/0.661 | 0.602/0.661 | 0.395/0.328 | 0.395/0.328 |
| Normalized eff dim | -0.358/-0.475 | -0.320/-437 | 0.301/0.497 | 0.163/0.280 |
| entropy(sup) | 0.706/0.807 | 0.695/0.780 | 0.565/0.575 | 0.482/0.632 |
Although model size exhibits a positive correlation with performance, its correlation is substantially weaker than that of entropy. Moreover, for CNNs, the results based on the normalized effective dimension show that is not always aligned with model size, suggesting that the proposed score captures information beyond model scale alone. These results further support entropy as a more suitable indicator for ranking models.
Regarding the repeated runs, we report the results (/) on other 2 seeds (seed =1,2) in the following. Note that we do not require model updates, so the seeds mostly affect the data sampling and augmentation. We present the results as follows:
| Setting | Run | Caltech101 | Cars | DTD | Avg |
|---|---|---|---|---|---|
| CNN unsup | 0 | 0.758/0.927 | 0.826/0.855 | 0.787/0.856 | 0.791/0.879 |
| 1 | 0.758/0.927 | 0.849/0.873 | 0.819/0.888 | 0.809/0.896 | |
| 2 | 0.758/0.927 | 0.876/0.882 | 0.840/0.897 | 0.825/0.902 | |
| CNN sup | 0 | 0.646/0.882 | 0.757/0.773 | 0.854/0.920 | 0.752/0.858 |
| 1 | 0.706/0.909 | 0.757/0.773 | 0.766/0.843 | 0.743/0.842 | |
| 2 | 0.520/0.782 | 0.778/0.791 | 0.850/0.916 | 0.716/0.829 | |
| ViT unsup | 0 | 0.682/0.886 | 0.254/0.429 | 0.682/0.886 | 0.539/0.733 |
| 1 | 0.682/0.886 | 0.254/0.429 | 0.682/0.886 | 0.539/0.733 | |
| 2 | 0.755/0.943 | 0.254/0.429 | 0.682/0.886 | 0.564/0.752 | |
| ViT supe | 0 | 0.755/0.943 | 0.484/0.371 | 0.682/0.886 | 0.640/0.733 |
| 1 | 0.755/0.943 | 0.484/0.371 | 0.682/0.886 | 0.640/0.733 | |
| 2 | 0.554/0.714 | 0.484/0.371 | 0.620/0.771 | 0.553/0.619 |
We also report the dataset-wise results () over 3 seeds with the unsupervised entropy method in ViT benchmark as follows
| Seed | Cal101 | Cars | CIFAR100 | DTD | Aircraft | Flowers | Food | Pets | SUN397 | Avg |
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0.682/0.886 | 0.254/0.429 | 0.267/0.600 | 0.682/0.886 | 0.676/0.657 | 0.414/0.600 | 0.188/0.200 | 0.414/0.600 | 0.762/0.829 | 0.482/0.632 |
| 1 | 0.682/0.886 | 0.254/0.429 | 0.267/0.600 | 0.682/0.886 | 0.676/0.657 | 0.414/0.600 | 0.188/0.200 | 0.414/0.600 | 0.676/0.657 | 0.472/0.613 |
| 2 | 0.755/0.943 | 0.254/0.429 | 0.267/0.600 | 0.682/0.886 | 0.676/0.657 | 0.414/0.600 | 0.188/0.200 | 0.414/0.600 | 0.762/0.829 | 0.490/0.638 |
The ranking is mostly seed-invariant except at a near-tie. For instance, in ViT unsupervised setting, on every dataset the top three and the bottom model are unchanged across all three runs, and the only movement is one adjacent swap between dino_vits8 and mae_vitb16, whose scores differ by less than the run-to-run drift.
Note that all the reported experiments in the rebuttal and revised paper are averaged across 9 datasets, excluding CIFAR-10.
Response [1/2]
We sincerely thank the reviewer for the thoughtful comments and constructive feedback. We have carefully considered these concerns and revised our manuscript to address them.
W1 & Q2: Theoretical validity and positioning of gradient entropy & applicability of initial entropy to practical fine-tuning.
In Section 3, we show that gradient entropy can increase during learning. We interpret this trend as follows: although individual optimization steps may not uniformly improve all model components, the overall learning process progressively captures task-relevant patterns from the training data. At the early stages of training, different components of the model may respond differently to various task factors, depending on both the input data's sensitivity to these factors and the information the model has already learned. This leads to heterogeneous gradient responses across neuron units. As training progresses, the model gradually adapts to these factors, and the gradient mass becomes more evenly distributed across different units, thereby increasing gradient entropy. Motivated by this observation, we hypothesize that, for a given downstream task, a higher initial gradient entropy, indicating a more evenly distributed gradient mass across neuron units, reflects a broader task-responsive representation and can serve as a useful signal for predicting downstream performance. This intuition is further supported by our theoretical analysis in Section 4.
Specifically, we introduce a latent-factor model in which each pretrained model captures a subset of task-relevant factors. For model m, let Tm=Sm∩T and km=∣Tm∣ denote its task-relevant coverage. Proposition 4.2 shows that the initial task-responsive gradients are determined by the factors captured by the model, and hence its effective dimension is
{\|Q_m\beta_{T_m}\|_2^2}.$$ Under the isotropic task-coefficient assumption, Theorem 4.3 further shows that $$\mathbb E_\beta[d_m] =1+\frac{2}{\pi}(k_m-1),$$ while the expected Bayes-optimal downstream accuracy also increases strictly with $k_m$. Thus, we do not assume that entropy is related to performance; rather, both quantities are derived from the same task-relevant coverage $k_m$, which establishes their consistent expected ranking across candidate models. This analysis is independent of the subsequent entropy growth during fine-tuning. The role of Section 3 is to motivate gradient entropy from learning dynamics, while the revised Section 4 provides the direct theoretical basis for using initial gradient entropy to compare pretrained models. **Q1: Clarification of gradient computation and contrastive learning setup.** $g_i$ is the neuron-wise gradient. The gradient is computed by aggregating the gradients over the entire dataset, i.e., by taking the mean of the per-sample gradients across all data points. For supervised scoring, we use a linear classifier head initialized with Kaiming-uniform initialization to produce the logits. The classifier head is kept fixed and is not updated during gradient computation. For unsupervised scoring, we do not use an additional projection head; instead, we directly use the penultimate features. We compute the self-supervised loss using the same data augmentation strategy as MoCoV2. We have added the details in the revised appendix.