acceptodds
Under review as a conference paper at ICLR 2027

Estimating Dataset Learning Utility Before Training: A Manifold Volume Perspective

Abstract

Assessing the quality of a candidate dataset before model training is increasingly important for data acquisition and emerging data marketplaces. However, dataset size can be easily inflated by duplicate or highly redundant samples, while existing intrinsic quality metrics do not directly reflect downstream learning utility and model-aware valuation methods often require training on the candidate data itself. In this work, we study label-free and fine-tuning-free dataset-level quality assessment, aiming to estimate the future learning utility of an entire candidate dataset before training on it. We propose a simple geometric criterion based on the manifold volume of frozen pretrained representations. Our key hypothesis is that informative and non-redundant data span richer and more diverse feature directions, whereas redundant data concentrate in a more restricted subspace. Theoretically, we show that manifold volume is invariant to trivial sample replication, becomes dimensionally constrained under severe redundancy, and exhibits diminishing returns along already well-covered feature directions. We evaluate the proposed criterion through controlled redundancy experiments on nine datasets spanning three modalities, four architecture families, and 13 dataset-backbone pairs. Manifold volume consistently exhibits a strong positive linear association with downstream learning utility, achieving the highest average Pearson correlation of 0.816.

Primary area:
datasets and benchmarks
Keywords:
Dataset Quality Assessment, Dataset Valuation, Manifold Volume, Learning Utility

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_eLCTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a label-free and fine-tuning-free method for estimating dataset learning utility before training. The method uses manifold volume, computed as the regularized log-determinant of the feature covariance matrix extracted from a frozen pretrained model, to measure the effective information and diversity of a dataset. Theoretical analysis establishes several properties related to data redundancy and diminishing returns. Experiments across nine datasets and three modalities show a strong positive correlation between manifold volume and downstream learning performance, achieving an average Pearson correlation of 0.816.

Critical strengths

The paper considers a practically important problem by proposing a simple and computationally efficient method for estimating dataset learning utility without requiring labels or model fine-tuning.

Critical weaknesses

Severe Shortcomings:

Low novelty and incomplete review of the related works.

The proposed manifold volume metric is also related to classic log-determinant based feature diversity and information measure, e.g., coding rate (Yu et al., NeurIPS 2020; Huang et al., ICML 2022). More significantly, Xu et al. (NeurIPS 2021) have previously examined data valuation based on volume, its correlation to learning performance, and sensitivity to data replication. Such related works are not well reviewed or evaluated.

References

[1] Yu, Y., Chan, K. H. R., You, C., Song, C., & Ma, Y. (2020). Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction. Advances in Neural Information Processing Systems (NeurIPS 2020).

[2] Huang, L.-K., Huang, J., Rong, Y., Yang, Q., & Wei, Y. (2022). Frustratingly Easy Transferability Estimation. Proceedings of the 39th International Conference on Machine Learning (ICML 2022), PMLR 162, 9201–9225.

[3] Xu, X., Wu, Z., Foo, C. S., & Low, B. K. H. (2021). Validation Free and Replication Robust Volume-based Data Valuation. Advances in Neural Information Processing Systems (NeurIPS 2021).

Questions for authors

Would the authors please directly compare your proposed metric with existing volume or coding-rate based data valuation methods to demonstrate its superiority in downstream learning utility prediction?

1 comment

We thank the reviewer for pointing out Xu et al. (NeurIPS 2021), Yu et al. (NeurIPS 2020), and Huang et al. (ICML 2022), and for suggesting a direct comparison. Our work differs from these studies in problem formulation and target setting: we study label-free, fine-tuning-free dataset valuation before training under controlled within-class redundancy (fixed nominal size and class composition) as well as cross-model utility prediction, where neither sample counts nor labels are available.

We compared Vol against Xu et al.'s Robust Volume (RV) and the unsupervised overall coding-rate terms derived from Yu et al. and Huang et al. on the same 78 candidates across 13 dataset-backbone pairs using identical downstream accuracies.

  1. Comparison with label-free volume-based data valuation (Xu et al., 2021): Among the cited works, Xu et al. directly addresses label-free, replication-robust data valuation. Across all six replacement rates, raw RV and Vol achieved mean Spearman correlations of 0.982 and 0.978, respectively, whereas mean Pearson correlation was only 0.298 for raw RV compared to 0.816 for Vol. When retaining the original feature dimensions, RV assigned zero to 13 of the 78 candidates because the grid-compressed feature matrix was rank-deficient (for example, on ESC-50 with AST, RV was zero at replacement rates of 60%, 80%, and 100%, despite downstream accuracies of 88.50%, 86.50%, and 65.25%, respectively). By using sample-normalized centered covariance with identity regularization, Vol retains the contributions of nonzero feature directions and distinguishes these three conditions with scores of 173.02, 145.26, and 98.37, without requiring grid-discretization hyperparameters.

  2. Comparison with coding-rate criteria (Yu et al., 2020; Huang et al., 2022): Both Yu et al. (representation learning via coding-rate reduction) and Huang et al. (TransRate for supervised model transferability estimation) rely on the coding-rate difference R(Z) - R(Z|Y), which requires ground-truth labels Y to compute the class-conditional term R(Z|Y). When labels are unavailable, only their unsupervised overall coding-rate term R(Z, ε) can be computed. Note that Huang et al. adopt the coding-rate formulation of Yu et al. combined with feature normalization for ranking models on a fixed labeled task rather than proposing a standalone label-free dataset metric. Under our label-free protocol, Yu et al.'s overall coding-rate term (which uses uncentered features scaled by m / (n * ε²)) is highly sensitive to the hyperparameter ε: across architectures with different feature dimensions m, both its mean Spearman and Pearson correlations fluctuate substantially with ε, and without feature centering, constant feature shifts affect the score. More generally, overall coding-rate formulations depend on tuning the scale/distortion parameter ε and feature preprocessing. In contrast, Vol uses raw centered frozen representations with a fixed identity regularizer, requiring neither extra feature preprocessing nor tuning of ε (nor grid hyperparameters as in RV), while achieving strong rank and linear correlations (0.978 Spearman and 0.816 Pearson) and theoretical invariance to uniform replication.

In summary, the direct comparison demonstrates that Vol substantially outperforms the existing label-free volume baseline of Xu et al. in linear correlation with downstream accuracy (0.816 vs. 0.298) and avoids rank-deficiency collapse on low-rank candidates, while providing a hyperparameter-free, centered geometric criterion that avoids the strong hyperparameter sensitivity of Yu et al.'s overall coding rate.

Reviewer_7s93OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a score that ranks candidate training datasets, e.g. datasets offered in a data marketplace, by their expected downstream utility. The score uses no labels and no fine-tuning. It is the "manifold volume" Vol=12log⁡2det⁡(Im+CD)\mathrm{Vol}=\tfrac12\log_2\det(I_m + C_{\mathcal D}), where CDC_{\mathcal D} is the centered empirical covariance of frozen pretrained features of the candidate set (Eqs. 3–6). Three properties are shown: invariance when every sample is replicated (Prop. 1), a rank bound when only a few samples are distinct (Prop. 2), and diminishing returns in each eigenvalue (Eq. 12). The experiments cover 13 dataset–backbone pairs in vision, text and audio. In each pair, the authors replace a fraction r∈{0,…,100}%r \in \{0,\dots,100\}\% of every class with copies of one seed sample at fixed dataset size, fine-tune on each variant, and correlate the scores with test accuracy. Vol reaches a mean Spearman correlation of 0.978 and the highest mean Pearson correlation (0.816) among Vendi, Div and Rel. A single cross-model experiment (ResNet-18 → ResNet-50 on CIFAR-100) is presented as initial evidence that the score transfers across models.

Critical strengths
  1. A practically relevant problem, clearly formalized. Sec. 2 states the label-free and fine-tuning-free constraints together with an ordering criterion, and the marketplace motivation, where labels stay undisclosed before purchase (l. 66–69), is concrete.
  2. A simple, cheap and well-defined score. It needs one forward pass and an m×mm\times m log-determinant, and the identity term keeps it finite under rank deficiency. Invariance to duplication (Prop. 1) is a desirable property for valuation, because padding a dataset with copies does not inflate its score.
  3. Broad coverage with checkable results. The study covers three modalities, ImageNet scale and four architecture families, and Table 2 reports every per-level value. I recomputed the correlations from Table 2 and they match the reported values up to rounding.
Critical weaknesses
  1. The benchmark cannot separate Vol from trivial statistics, and the headline result uses the secondary metric.
    • The only corruption studied is exact intra-class duplication at fixed size (l. 299–303). The true ordering is therefore known by construction: it is simply −r-r.
    • Spearman correlation is declared the primary metric (l. 320–323). On it, Vol has the lowest mean of the four scores: 0.978, against 0.982 (Vendi), 0.987 (Rel) and ∣−0.987∣|{-0.987}| (Div). The abstract and conclusion instead headline the mean Pearson correlation of 0.816.
    • Recomputing from Table 2, the trivial score −r-r reaches a mean Spearman of 0.987. It is equivalent to the label-free count of distinct inputs.
    • Vol's Pearson advantage over −r-r (0.816 vs. 0.756) depends entirely on the r=100%r=100\% level. Restricted to r≤80%r\le 80\%, the mean Pearson values are 0.939 (Vol), 0.943 (Vendi) and 0.951 (−r-r).
    • Each correlation uses 6 points, and no seeds or variances are reported. For a single pair, a Pearson correlation of 0.8 from 6 points has a 95% confidence interval of roughly [−0.03,0.98][-0.03, 0.98], so the gap of 0.816 vs. 0.801 cannot be interpreted.
    • I suggest the following:
      • (a) repeated runs or kk data splits, to put error bars on Acc and on the correlations;
      • (b) naturally varying candidates, e.g. disjoint folds or subsets of different composition, source and size, instead of synthetic duplication only;
      • (c) near-duplicates (augmentations), label noise and off-task samples;
      • (d) a distinct-count baseline.
  2. Vol barely responds to dataset size and ignores task relevance, so it cannot meet the paper's own criterion in general.
    • Invariance to duplicated data is desirable. But the 1/n1/n normalization also makes Vol nearly blind to genuinely new i.i.d. samples once n≫mn\gg m, because CDC_{\mathcal D} has already converged to the population covariance. A random 10% subset would therefore score about the same as the full set, although fine-tuning on it would yield lower accuracy. This conflicts with the ordering criterion of Sec. 2, and the experiments cannot detect it because all candidates have the same size.
    • The score has no reference to the buyer's existing data and no "zero point". At r=80%r=80\%, the fine-tuned ResNet-18 on CIFAR-100 reaches 61.51% (Table 2), essentially the 61.52% of the initial model (Table 3). Vol for that candidate is 50.22, far from zero.
    • No feature normalization is applied (l. 308), so the identity term fixes an arbitrary scale.
      • If all eigenvalues are much smaller than 1, Vol ≈ tr(CD)/(2ln⁡2)\mathrm{tr}(C_{\mathcal D})/(2\ln2), i.e. total variance rather than a volume.
      • Rankings can flip under global rescaling. The spectra {10}\{10\} and {1,1,1,1,1}\{1,1,1,1,1\} give 1.73 < 2.50 bits; after multiplying all eigenvalues by 0.1 they give 0.50 > 0.34.
    • I suggest candidates of varying size and source, and an ablation over the regularizer scale (ε2Im\varepsilon^2 I_m) and feature standardization.
  3. The paper's novelty is not positioned against the closest prior work, and the theory is elementary.
    • The formula is known. Up to a factor dd inside the determinant, Eq. 4 is the "feature volume" / semantic scale of Ma et al. (ICLR 2023, Delving into Semantic Scale Imbalance). That paper introduces the quantity as "the volume of manifold" and already reports that it saturates with sample count. Both are instances of the Gaussian coding rate (Ma et al., 2007; Yu et al., NeurIPS 2020).
    • The problem is not new. Xu et al. (NeurIPS 2021, Validation Free and Replication Robust Volume-based Data Valuation) propose validation-free, replication-robust, volume-based data valuation, with data marketplaces as a motivating use case.
    • Related lines are missing. Transferability estimation (LEEP, LogME) and training-free data valuation (DAVINZ) are not discussed either. None of these works is cited, so the claim of "a new problem" (contribution 1) needs to be qualified.
    • The theory is elementary. Props. 1–2 and Eq. 12 are correct, but they are short facts shared by other spectral scores; Vendi, for instance, is also invariant under uniform replication. None of them links Vol to downstream utility, which the paper acknowledges (Sec. 3.3).
    • The theory does not match the experiments. Prop. 1 concerns uniform replication of every sample, not the corruption used in the experiments. Prop. 2 is vacuous for ImageNet with ViT-B/16 at r=100%r=100\% (q−1=999>m=768q-1=999 > m=768), where Vol barely changes (31.78 → 31.51) while accuracy drops from 79.56% to 19.34%.
Questions for authors
  1. Which reference data define Div and Rel (l. 316–319)? Rel = 1.00 and Div = 0.00 at r=0r=0 in all rows, which suggests the clean candidate itself is the reference. Please also explain the sign convention used for Div.
  2. Which kernel, features and normalization were used for Vendi?
  3. How many runs or seeds underlie Tables 2–3, and what are the variances? Is fine-tuning full or partial, and for how many epochs?
  4. Which ViT-L/14 checkpoints were used, and does their pretraining data overlap with the ImageNet-1K training split that serves as the candidate pool?
  5. How does Vol rank random clean subsets of size nn, n/2n/2 and n/10n/10, and candidates from different sources, compared with Acc?
  6. How does Vol respond to augmented near-duplicates, off-task samples and label noise?
  7. How sensitive are the rankings to replacing ImI_m with ε2Im\varepsilon^2 I_m, and to feature standardization?
  8. Could you report the correlations for r≤80%r\le 80\%, add a distinct-count baseline, and give the correlation, baselines and seeds for Table 3?
  9. How does the method relate to Xu et al. (2021) and Ma et al. (2023)?
  10. To quantify the cost of being label-free, could you add a fine-tuning-free baseline that does use labels, such as LogME?
2 comments
  1. Reference set and sign convention for Div and Rel. For each dataset-backbone pair, we use its unmodified candidate set (r = 0) as the reference. We subtract the reference mean from the candidate features, compare second-order statistics along the principal directions of the reference covariance, and aggregate them using an unweighted geometric mean. Under this convention (Amiri et al., 2023), Rel measures second-order alignment with the reference (Rel = 1.00 at r = 0 and decreases as redundancy grows), whereas Div >= 0 measures normalized statistical divergence from the reference (Div = 0.00 at r = 0 and increases monotonically as redundancy grows, yielding a negative correlation with accuracy).

  2. Vendi kernel, features, and normalization. We use a cosine kernel for all pairs and the same types of frozen representations used for Vol: pre-classifier features for ResNet, projected encode_image outputs for CLIP, and the final-layer representation of the first token for BERT and AST. We L2-normalize each feature vector. Given the normalized feature matrix X, we compute the spectrum in feature space because XᵀX/n and XXᵀ/n share the same nonzero eigenvalues. We normalize the eigenvalues to sum to one and report the exponential of their Shannon entropy.

  3. Runs, seeds, and fine-tuning. In Table 2, each condition is evaluated with a single run using seed 42 for candidate construction; cross-seed variances are therefore not reported. We fine-tune for 15 epochs on CIFAR-10 with ResNet-18 and ESC-50 with AST. We use 10 epochs for CIFAR-100 with ResNet-18 and for ImageNet and Places365 with ResNet-50. We use 5 epochs for AG News, TNEWS, and THUCNews with BERT, and for Speech Commands with AST. All of these settings update the full backbone and classification head. For ViT-B/16 and ViT-L/14, we update the full visual encoder and classification head for 5 epochs per replacement rate. In Table 3, we train ResNet-50 from random initialization on the internal dataset D₀ alone and on each of the six unions D₀ U D. Each of the seven conditions has one 90-epoch run with training seed 42.

  4. ViT-L/14 checkpoint and pretraining overlap. We load OpenAI's official 224-pixel ViT-L/14 checkpoint using clip.load("ViT-L/14"). The CLIP paper reports zero-shot evaluation on ImageNet and separately analyzes overlap between the pretraining data and 35 downstream evaluation sets. The median overlap was 2.2% and the mean was 3.2%, with no overlap detected in nine evaluation sets. We can therefore consider any overlap with the ImageNet-1K candidate training pool to be negligible.

  5. Clean subsets of different sizes and candidates from different sources. Using seed 42, we sampled nested subsets from the clean CIFAR-100 candidate pool and scored them with ResNet-18. Vol was 57.592 for the full set (25,000 samples), 57.561 for the half-size subset (12,500), and 57.365 for the one-tenth subset (2,500). Vol and downstream accuracy ranked the three subsets in the same order.

  1. Near-duplicates, off-task samples, and label noise. Because our valuation setting is strictly label-free and does not access annotations at evaluation time, Vol does not respond to pure label noise, which falls outside the scope of our problem setting. Regarding input-level effects, augmented near-duplicates still concentrate in a local subspace of pretrained representations and thus reduce Vol (whereas distinct-input counts treat every perturbed copy as unique). Our current dataset construction method is not applicable to off-task samples. We appreciate this suggestion and will consider designing appropriate protocols for these settings.

  2. Regularization scale and feature standardization. We varied one factor at a time across the same 78 candidates and 13 dataset-backbone pairs. We computed correlations with downstream accuracy within each pair and weighted all pairs equally. First, we retained feature centering, applied no per-dimension standardization, and replaced I with ε²I. We tested 13 values of ε from 0.01 to 100. Across all six replacement rates, mean Spearman and Pearson correlations were 0.987 and 0.968 at ε = 0.01. For the default setting (ε = 1), Table 2 reports 0.978 and 0.816. At ε = 10, the correlations fell to 0.398 and 0.178, and the candidate ranking changed in 11 of the 13 pairs.

In the second ablation, we fixed ε = 1, estimated a standard deviation for each feature dimension from the unmodified candidate set of each pair, and applied the same scales at every replacement rate. With standardization, the mean Pearson correlation across all six rates was 0.864, and the ranking changed in only one pair. A large regularization scale can therefore substantially affect the rankings. At the default scale, per-dimension standardization mainly affected the spacing between scores.

  1. Correlations over 0-80%, distinct-input counts, and Table 3 baselines. We excluded the 100% replacement condition from Table 2, computed correlations within each pair, and averaged them equally over the 13 pairs. The mean correlations (Spearman, Pearson) were Vol (0.969, 0.939), Vendi (0.977, 0.943), Rel (0.977, 0.794), Div (-0.977, -0.832), -r (0.977, 0.951), and Xu's RV (0.975, 0.629).

We also counted distinct inputs directly. Across the six TNEWS conditions, the number of distinct input texts fell from 50,299 to 15. Its Spearman and Pearson correlations with accuracy were 0.943 and 0.695 (compared to 0.777 Pearson for Vol). For Table 3, deduplication by file content gave distinct-image counts ranging from 24,999 to 100. Vol, -r, and the distinct-image count each had a Spearman correlation of 0.943 with downstream accuracy gain (Gain). Their Pearson correlations were 0.986, 0.939, and 0.938, respectively. Table 3 used seed 42 for the data split and training, and seed 42 + r for replacement at rate r.

  1. Relation to Xu et al. and Ma et al. Xu et al. use grid-based compression and a density correction to limit score inflation under replication. With the original feature dimensions retained, RV was zero for 13 of our 78 candidates because the number of occupied cells was smaller than the feature dimension. Vol retains the contributions of directions with positive eigenvalues even when the covariance is rank-deficient. Across all six rates, mean Spearman correlation was 0.982 for raw RV and 0.978 for Vol. Mean Pearson correlation was 0.298 and 0.816, respectively. The observed differences concern discrimination among low-rank candidates and the linear association of the raw scores with accuracy. Ma et al. quantify semantic scale by class, study its dependence on sample count, and use it to guide training. Their class-wise procedure requires class labels, which are unavailable in our setting, where the entire candidate set receives a single score.

  2. LogME as a label-aware baseline. We study pre-acquisition valuation when labels are unavailable to the evaluator, including datasets that have yet to be annotated and datasets whose owners withhold valuable labels until after acquisition. Because LogME requires ground-truth labels to compute label-conditional evidence, it cannot operate under our label-free constraint and thus represents a separate, label-available setting rather than a directly comparable baseline.

Reviewer_KVyeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a label-free, fine-tuning-free score for estimating a dataset's utility before training. Using a frozen pretrained model, it computes the regularized log-determinant "manifold volume" of the centered feature covariance, which captures both the number of effective feature directions a dataset spans and the variation along them. The authors show this score is invariant to uniform replication, rank-bounded by the number of distinct samples, and has diminishing returns along already-covered directions.

Critical strengths

Understanding the data utility before training is clearly a very important topic as we further scale up. The question is well-motivated.

Critical weaknesses

The proposed manifold volume is closely related to the volume-based data valuation of Xu et al. (NeurIPS 2021, "Validation Free and Replication Robust Volume-based Data Valuation"), which is not cited. That work also values data by the log-determinant volume of the feature matrix, det⁡(X⊤X)\sqrt{\det(X^\top X)}, also uses only inputs (no labels or validation set), and also explicitly addresses replication robustness, proposing a Robust Volume (RV) with provable inflation bounds. Could the authors clarify what they consider the contribution beyond Xu et al.?

Why is label-free considered an advantage here? Label quality itself is also an important aspect for data quality. Can we incorporate label information through gradient embedding?

Questions for authors

See above.

1 comment

We thank the reviewer for pointing us to Xu et al. (NeurIPS 2021). Their Robust Volume (RV) combines grid-based compression with density weighting and provides an upper bound on the inflation caused by replication. Beyond avoiding grid-discretization hyperparameters, Vol achieves exact invariance to uniform replication via sample-count normalization and remains well-defined when high-dimensional representations become rank-deficient. We evaluated RV on the same 78 candidates across 13 dataset-backbone pairs, using ω = 0.1 and β = 10. For 13 out of 78 candidates, the number of occupied grid cells was smaller than the feature dimension, yielding an RV of zero. For example, on ESC-50 with AST, RV was zero at replacement rates of 60%, 80%, and 100%. Downstream accuracy at these rates was 88.50%, 86.50%, and 65.25%, respectively, and Vol distinguished the three candidates with scores of 173.02, 145.26, and 98.37.

Across all six replacement rates, RV and Vol had similar mean Spearman correlations with downstream accuracy (0.982 and 0.978, respectively). Because Spearman correlations are already near saturation for all compared methods under monotonically increasing redundancy (all around 0.98), Pearson correlation better distinguishes how closely a metric tracks downstream accuracy changes. Mean Pearson correlation was 0.298 for raw RV and 0.816 for Vol. Thus, the advantages of Vol lie in maintaining discrimination on all 13 low-rank candidates where RV collapsed to zero and exhibiting a substantially stronger linear association between its raw scores and downstream accuracy.

We agree that label accuracy and consistency are important aspects of data quality. We focus on settings where labels are unavailable at valuation time. This includes both unannotated data and data whose owners withhold valuable annotations until after purchase. For example, a hospital might allow a prospective buyer to inspect de-identified pathology slides while withholding rare tumor subtype labels reviewed by several pathologists until a procurement agreement is signed. The buyer could still use a frozen encoder to extract image features and compute Vol as an initial check for extensive duplication or broader coverage of image features. In this setting, the score reflects the information available from the inputs themselves.

Regarding incorporating label information via gradient embeddings: while computing manifold volume over per-sample loss gradient embeddings is a reasonable option when annotations are available, it does not apply to the label-free valuation setting studied in Section 2. Because labels are either absent or withheld by the data owner at valuation time, supervised loss gradients cannot be computed prior to acquisition; requiring extra labels would violate our core label-free constraint and turn the task into a different, label-dependent problem regime.

Reviewer_Z29bOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a label-free, fine-tuning-free framework for estimating dataset learning utility by computing the log-determinant of the feature covariance matrix extracted from a frozen pre-trained model. The proposed log-determinant criterion is closely related to established spectral diversity and information-based criteria, including those used in D-optimal experimental design. The paper should more clearly distinguish its methodological contribution from these existing approaches. The proposed metric is sensitive to feature scaling, as uniformly scaling the representations increases the log-determinant score. However, the paper does not investigate feature normalization, regularization choices, or the robustness of the metric across different representation extractors. Although the method explicitly evaluates the absolute manifold volume of a candidate dataset, the practical data acquisition scenario concerns its marginal utility relative to an existing training set. The current metric does not explicitly account for redundancy or complementarity between the candidate dataset and previously available data. Although the paper reports both Spearman and Pearson correlations, the proposed method does not outperform the baselines in average Spearman correlation, despite dataset ranking being its primary objective. Its improvement in average Pearson correlation over Vendi Score is also marginal (0.816 vs. 0.801). Additional statistical analysis is needed to establish whether this advantage is robust, particularly given that each correlation is computed over only six redundancy levels. The experimental evaluation is largely restricted to artificially constructed redundancy levels, where class-wise samples are replaced by exact duplicates. Such controlled perturbations naturally reduce representation diversity and may favor diversity-based metrics. It remains unclear whether the proposed method can reliably rank independently sampled candidate datasets with varying task relevance, noise levels, and distribution shifts.

Critical strengths

The paper includes detailed tables and informative figures that effectively illustrate the experimental results and facilitate comparisons across datasets, models, and baseline methods.

Critical weaknesses

The paper lacks sufficient details regarding the experimental setup, implementation, and evaluation protocols, making it difficult to assess the reproducibility and reliability of the reported results.

A more detailed theoretical discussion would be helpful in further clarifying the relationship between the proposed metric and downstream learning utility.

The experimental design is not sufficiently comprehensive or convincing to support the paper's central claims. Additional experiments under more diverse and realistic settings are needed to demonstrate the effectiveness and generalizability of the proposed approach.

Questions for authors

Could the authors elaborate on the key methodological innovations of the proposed approach, particularly in comparison with existing log-determinant-based methods such as D-optimal experimental design and Determinantal Point Processes (DPPs)? Additionally, what are the novel aspects of the experimental design compared with existing dataset valuation and diversity evaluation studies?

Could the authors provide more comprehensive details regarding the experimental setup, including dataset construction, feature extraction, model training configurations, hyperparameter selection? Such information would help improve the reproducibility of the experiments and allow a more thorough assessment of the reported results.

2 comments

Classical D-optimal design aims to improve the precision of parameter estimates in a specified statistical model. If a dataset is scored directly by the determinant of its unnormalized information matrix, replicating every observation the same number of times inflates the determinant whenever the matrix has full rank, even though the copies add no new observations. By contrast, Vol Vol operates on the centered feature covariance normalized by the sample count, and therefore remains invariant under uniform replication. A fixed-size determinantal point process (DPP) assigns selection weights to subsets of a given size drawn from a common pool. We study a distinct task at the dataset level: given an entire candidate dataset, can a score computed before fine-tuning and without access to labels distinguish its downstream utility from that of other candidates sharing the same size and class composition but differing in within-class redundancy?

Prior volume-based studies address different settings: Xu et al. (NeurIPS 2021) examined how appending copies of existing samples inflates data valuation, whereas Ma et al. (ICLR 2023) used class-wise feature volume (requiring ground-truth labels) to characterize semantic scale imbalance across classes. Our experimental contribution is a controlled within-class replacement protocol that varies redundancy while holding both the total sample count and the per-class sample counts fixed. This models a data procurement scenario in which a supplier meets delivery and class quotas by padding classes with repeated examples, enabling us to test whether Vol Vol tracks differences in downstream utility when neither nominal dataset size nor class proportions distinguish the candidates.

Taking CLIP ViT-L/14 and ResNet-50 on ImageNet-1K as examples, both experiments evaluate six within-class replacement rates 𝑟 ∈ { 0 , 20 , 40 , 60 , 80 , 100 } % r∈{0,20,40,60,80,100}%. Within each class, we randomly select one image as a template and replace 𝑟 % r% of randomly selected images in that class one-for-one with copies of the template (using random seed 42). Ground-truth labels are used solely to construct the controlled candidates and for downstream supervised fine-tuning; Vol Vol itself uses only image features extracted by the frozen encoder.

CLIP ViT-L/14: We use OpenAI's official 224-pixel checkpoint and construct candidates from the full ImageNet-1K training split. Applying the official preprocessing pipeline, we extract the 768-dimensional projected features returned by encode_image. We fine-tune the full visual encoder together with a newly initialized 1,000-class linear head for 5 epochs while keeping the text encoder frozen, using AdamW with cross-entropy loss, a batch size of 64, learning rates of 10 − 5 10 −5 for the backbone and 10 − 3 10 −3 for the head, and a weight decay of 10 − 4 10 −4 . ResNet-50: We split each class in the ImageNet-1K training set into two halves, A and B. We pretrain from random initialization on A for 90 epochs using SGD (batch size 256, initial learning rate 0.1 decayed by 0.1 × 0.1× every 30 epochs, momentum 0.9, weight decay 10 − 4 10 −4 ). We construct candidates from B and score them using the 2,048-dimensional global-average-pooled features immediately preceding the classification head. For each replacement rate, we reload the initial ResNet-50 checkpoint and fine-tune both the backbone and its existing classification head for 10 epochs using SGD (batch size 64, initial learning rate 10 − 3 10 −3 , momentum 0.9, weight decay 10 − 4 10 −4 , cosine annealing).

For both architectures, we evaluate top-1 accuracy on the 50,000 images of the official ImageNet-1K validation set.

References:

Xu, X., Wu, Z., Foo, C. S., & Low, B. K. H. (2021). Validation Free and Replication Robust Volume-based Data Valuation. NeurIPS 2021. Ma, Y., Jiao, L., Liu, F., Yang, S., Liu, X., & Li, L. (2023). Delving into Semantic Scale Imbalance. ICLR 2023.

We would like to supplement our response above with additional details addressing feature normalization, regularization choices, Spearman vs. Pearson correlations, marginal utility, and the complete training configurations across all 13 dataset-backbone pairs:

  1. Feature standardization and regularization scale: Across the 78 candidates (13 dataset-backbone pairs, weighted equally), we conducted one-factor ablations on regularization scale and feature standardization. First, retaining feature centering without standardization and replacing I with ε²I across 13 values of ε in [0.01, 100], mean (Spearman, Pearson) correlations across 0-100% are (0.987, 0.968) at ε = 0.01, (0.978, 0.816) at default ε = 1 (Table 2), and (0.398, 0.178) at ε = 10 (rankings changed in 11 of 13 pairs). Second, fixing ε = 1 with per-dimension standardization estimated from each pair's unmodified set (r = 0), mean Pearson correlation increases from 0.816 to 0.864, with ranking changed in only 1 of 13 pairs. Standardization at ε = 1 thus mitigates scaling sensitivity and improves linear correlation while largely preserving within-pair rankings.

  2. Spearman saturation, Pearson correlation, and comparison with Xu et al. (2021): Under monotonically increasing intra-class redundancy, Spearman correlations are near saturation across all compared metrics (all around 0.98), so Pearson correlation better distinguishes how closely a score's magnitude tracks the nonlinear degradation in downstream accuracy. On the same 78 candidates, we also evaluated Xu et al.'s Robust Volume (RV, ω = 0.1, β = 10): while RV and Vol achieve similar mean Spearman correlations (0.982 vs. 0.978), raw RV achieves a mean Pearson correlation of only 0.298 compared to 0.816 for Vol, and RV collapses to zero on 13 low-rank candidates where occupied grid cells are fewer than the feature dimension.

  3. Marginal utility relative to an existing dataset D₀: When a buyer already possesses an internal training set D₀ (as in Section 4.3, Table 3), the framework naturally yields a marginal volume score Vol(D₀ U D) - Vol(D₀). By the diminishing-returns property (Eqs. 12-13), candidates overlapping with directions already well-covered by D₀ contribute little marginal volume, whereas complementary directions yield larger gains.

  4. Complete experimental configurations for all 13 pairs: For Vol, we extract pre-classifier features for ResNet, projected encode_image outputs for CLIP ViT, and final-layer first-token representations for BERT and AST. All fine-tuning runs update the full backbone and classification head:

  • CIFAR-10 / ResNet-18: Pretrained from scratch on half A (resnet18_modified, 32x32, 90 epochs, batch 128, SGD, lr 0.1, momentum 0.9, weight decay 5e-4, StepLR x0.1 every 30 epochs). Fine-tuned per level from the reloaded initial checkpoint (including head): 15 epochs, batch 128, SGD, lr 1e-3, momentum 0.9, weight decay 5e-4, MultiStepLR x0.1 after epochs 8 and 12, RandomCrop(32, padding=4), horizontal flip, and CIFAR-10 normalization (best test Top-1 over 15 epochs).
  • CIFAR-100 / ResNet-18: Same 90-epoch pretraining on half A except weight decay 1e-4. Fine-tuned per level from the full initial checkpoint: 10 epochs, batch 128, SGD, lr 0.005, momentum 0.9, weight decay 1e-4, cosine annealing, RandomCrop(32, padding=4), and horizontal flip.
  • ImageNet-1K & Places365 / ResNet-50: Pretrained from scratch on each half A (224x224 crop, 90 epochs, batch 256/GPU, SGD, lr 0.1, momentum 0.9, weight decay 1e-4, StepLR x0.1 every 30 epochs, AMP). Fine-tuned per level from the full initial checkpoint: 10 epochs, batch 64, SGD, lr 0.001, momentum 0.9, weight decay 1e-4, cosine annealing, RandomResizedCrop(224) + horizontal flip (eval: resize 256, center-crop 224).
  • ImageNet-1K & Places365 / CLIP ViT-B/16 & ViT-L/14 (4 pairs): Initialized from OpenAI ViT-B-16.pt / ViT-L-14.pt with CLIP preprocessing; full visual encoder and new linear head fine-tuned for 5 epochs per level (text encoder frozen), batch 64/GPU, AdamW, lr 1e-5 (backbone) and 1e-3 (head), weight decay 1e-4 (best eval accuracy over 5 epochs).
  • AG News, TNEWS, THUCNews / BERT-base (3 pairs): Initialized from Google bert-base-uncased (AG News: 4 classes, max length 128, batch 64/GPU) and bert-base-chinese (TNEWS: 15 classes, max length 64, batch 128/GPU; THUCNews: 14 classes, max length 128, batch 64/GPU); new head, full fine-tuning for 5 epochs, initial lr 2e-5, weight decay 0.01, FP16.
  • ESC-50 & Google Speech Commands / AST (2 pairs): Initialized from MIT/ast-finetuned-audioset-10-10-0.4593 (16 kHz resampling, 1,024 frames); new head (ESC-50: 50 classes, 15 epochs; GSC: 35 classes, 5 epochs), batch 16/GPU, initial lr 5e-5, weight decay 0.01, FP16.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.