- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a label-free and fine-tuning-free method for estimating dataset learning utility before training. The method uses manifold volume, computed as the regularized log-determinant of the feature covariance matrix extracted from a frozen pretrained model, to measure the effective information and diversity of a dataset. Theoretical analysis establishes several properties related to data redundancy and diminishing returns. Experiments across nine datasets and three modalities show a strong positive correlation between manifold volume and downstream learning performance, achieving an average Pearson correlation of 0.816.
Critical strengths
The paper considers a practically important problem by proposing a simple and computationally efficient method for estimating dataset learning utility without requiring labels or model fine-tuning.
Critical weaknesses
Severe Shortcomings:
Low novelty and incomplete review of the related works.
The proposed manifold volume metric is also related to classic log-determinant based feature diversity and information measure, e.g., coding rate (Yu et al., NeurIPS 2020; Huang et al., ICML 2022). More significantly, Xu et al. (NeurIPS 2021) have previously examined data valuation based on volume, its correlation to learning performance, and sensitivity to data replication. Such related works are not well reviewed or evaluated.
References
[1] Yu, Y., Chan, K. H. R., You, C., Song, C., & Ma, Y. (2020). Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction. Advances in Neural Information Processing Systems (NeurIPS 2020).
[2] Huang, L.-K., Huang, J., Rong, Y., Yang, Q., & Wei, Y. (2022). Frustratingly Easy Transferability Estimation. Proceedings of the 39th International Conference on Machine Learning (ICML 2022), PMLR 162, 9201–9225.
[3] Xu, X., Wu, Z., Foo, C. S., & Low, B. K. H. (2021). Validation Free and Replication Robust Volume-based Data Valuation. Advances in Neural Information Processing Systems (NeurIPS 2021).
Questions for authors
Would the authors please directly compare your proposed metric with existing volume or coding-rate based data valuation methods to demonstrate its superiority in downstream learning utility prediction?
We thank the reviewer for pointing out Xu et al. (NeurIPS 2021), Yu et al. (NeurIPS 2020), and Huang et al. (ICML 2022), and for suggesting a direct comparison. Our work differs from these studies in problem formulation and target setting: we study label-free, fine-tuning-free dataset valuation before training under controlled within-class redundancy (fixed nominal size and class composition) as well as cross-model utility prediction, where neither sample counts nor labels are available.
We compared Vol against Xu et al.'s Robust Volume (RV) and the unsupervised overall coding-rate terms derived from Yu et al. and Huang et al. on the same 78 candidates across 13 dataset-backbone pairs using identical downstream accuracies.
Comparison with label-free volume-based data valuation (Xu et al., 2021): Among the cited works, Xu et al. directly addresses label-free, replication-robust data valuation. Across all six replacement rates, raw RV and Vol achieved mean Spearman correlations of 0.982 and 0.978, respectively, whereas mean Pearson correlation was only 0.298 for raw RV compared to 0.816 for Vol. When retaining the original feature dimensions, RV assigned zero to 13 of the 78 candidates because the grid-compressed feature matrix was rank-deficient (for example, on ESC-50 with AST, RV was zero at replacement rates of 60%, 80%, and 100%, despite downstream accuracies of 88.50%, 86.50%, and 65.25%, respectively). By using sample-normalized centered covariance with identity regularization, Vol retains the contributions of nonzero feature directions and distinguishes these three conditions with scores of 173.02, 145.26, and 98.37, without requiring grid-discretization hyperparameters.
Comparison with coding-rate criteria (Yu et al., 2020; Huang et al., 2022): Both Yu et al. (representation learning via coding-rate reduction) and Huang et al. (TransRate for supervised model transferability estimation) rely on the coding-rate difference R(Z) - R(Z|Y), which requires ground-truth labels Y to compute the class-conditional term R(Z|Y). When labels are unavailable, only their unsupervised overall coding-rate term R(Z, ε) can be computed. Note that Huang et al. adopt the coding-rate formulation of Yu et al. combined with feature normalization for ranking models on a fixed labeled task rather than proposing a standalone label-free dataset metric. Under our label-free protocol, Yu et al.'s overall coding-rate term (which uses uncentered features scaled by m / (n * ε²)) is highly sensitive to the hyperparameter ε: across architectures with different feature dimensions m, both its mean Spearman and Pearson correlations fluctuate substantially with ε, and without feature centering, constant feature shifts affect the score. More generally, overall coding-rate formulations depend on tuning the scale/distortion parameter ε and feature preprocessing. In contrast, Vol uses raw centered frozen representations with a fixed identity regularizer, requiring neither extra feature preprocessing nor tuning of ε (nor grid hyperparameters as in RV), while achieving strong rank and linear correlations (0.978 Spearman and 0.816 Pearson) and theoretical invariance to uniform replication.
In summary, the direct comparison demonstrates that Vol substantially outperforms the existing label-free volume baseline of Xu et al. in linear correlation with downstream accuracy (0.816 vs. 0.298) and avoids rank-deficiency collapse on low-rank candidates, while providing a hyperparameter-free, centered geometric criterion that avoids the strong hyperparameter sensitivity of Yu et al.'s overall coding rate.