acceptodds
Under review as a conference paper at ICLR 2027

Fidelity-Diversity Metrics for Text Datasets

Abstract

As language modeling technology matures, there is an increasing research focus on the composition and curation of datasets used to train these models. For instance, practitioners commonly seek to augment high-quality datasets with additional text to enhance the performance of models trained on that data. However, informed decisions about data augmentation require more nuanced assessments about data quality. We build on work measuring the precision and recall of generative models to develop a pair of metrics that quantify (1) fidelity, capturing how closely candidate text resembles reference data, and (2) diversity, capturing how well it covers the modes of the reference dataset. Our metrics are based on optimal transport divergence functionals between discrete text summaries. In experiments on M2D2 text datasets, we show that these metrics are able to disentangle a lack of fidelity from a lack of diversity in deficient candidate text. In further experiments, our metrics detect diversity deficits in synthetic GSM8K-style math datasets, which correlate with degradations in downstream accuracy of language models finetuned on this synthetic data.

Primary area:
generative models
Keywords:
Diversity, Synthetic Data, Optimal Transport, Generative Modeling, Large Language Models, Wasserstein Distance, Nearest Neighbor, Clustering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SMdEOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This paper argues that when you compare a reference dataset to a synthetically generated dataset, auditing that dataset for fidelity and diversity is important (because they mirror precision and recall). Fidelity is whether the dataset measures the same properties that the reference dataset measures. Diversity is ensuring the synthetically dataset (or the augmented dataset) is representative. They formulate this as an optimal transport problem: representing the samples with a Gaussian or an embedding and measuring how much mass to move from the synthetic/augmented set to the reference set. Their experiments show that their fidelity metric is able to capture different concepts/topics and their diversity metric is able to truly measure diversity (they do this with a synthetic setting in which they increase/decrease the diversity).

Critical strengths

The idea is well formulated, and the writing is well structured/reads well. Their experiments are also quite interesting, as they seem to be working as intended.

They also back their empirical claims with significance tests.

Critical weaknesses

I think this paper has much more potential than what is presented. Currently, the empirical contribution is weak.

While reading the paper, I thought of two potential directions that this can be applied to make this paper much stronger and impactful: (1) using these metrics to select a meaningful set of training data for fine-tuned or post-trained models, and (2) measuring whether popular benchmarks do indeed have high fidelity and diversity compared to a particular training task. To clarify, if the community used DtrD_{\text{tr}} for training and Dte,iD_{\text{te},i} for i∈0,1,...,ki \in 0,1,...,k as the set of testing sets, you can use these metrics to test whether (1) DtrD_{\text{tr}} is a good training set compared to the test sets, or (2) the testing sets truly cover DtrD_{\text{tr}}.

Questions for authors

See Weaknesses.

2 comments

Thank you for the positive comments and the thoughtful suggestions. We are encouraged that you see potential in this method, and we find both of your proposed applications exciting.

(1) Training data selection. Our GSM8K experiment (Section 4.2) is a first case study of exactly this direction. There, we score a family of synthetic finetuning datasets against a reference before any training, and find that our diversity score strongly correlates with the downstream accuracy of the finetuned models (r=+0.973r = +0.973, p=0.0002p = 0.0002), while fidelity does not. This finding holds after removing the human-written seed problems as a potential confounder (r=+0.968r = +0.968, p=0.0003p = 0.0003; see our response to reviewer MbcM). Since our metrics are inexpensive relative to finetuning (embedding cost is linear in corpus size, and the transport cost depends only on the number of atoms), this suggests a practical workflow in which practitioners may cheaply screen candidate training sets with our metrics. We will make this framing explicit in the revision.

(2) Benchmark auditing. Our framework also applies to this setting. Taking the target task data as the reference and the benchmark as the candidate, diversity measures how well the benchmark covers the task distribution, while fidelity measures whether benchmark is on-distribution. Because the two metrics are separate, they distinguish two distinct failures: whether the benchmark is overly concentrated on a few modes of the task (low diversity), or contains too much out-of-distribution material (low fidelity). Our new controlled experiments on mode collapse and outlier injection (see our response to reviewer MbcM) validate precisely these two cases. We will add a discussion of this application to the paper.

On the empirical contribution. Our primary goal was to establish that the metrics behave as intended, which we view as a prerequisite for trusting them in downstream applications such as the ones you propose. During the rebuttal, we have substantially extended this validation with experiments on candidate sample size, duplication, mode collapse in both the reference and the candidate, outliers, randomly sampled (non-nested) subdomains, and a comparison against DCScore. In each case, the metrics respond as expected. We believe these results, together with the GSM8K study, strengthen the empirical case considerably, and we will include them in the revision.

We hope these clarifications address your concerns, and would be grateful if you would consider them in your assessment.

While I appreciate your answers to my questions (especially the correction to my misunderstanding in "Training data selection") and your general efforts in the rebuttal, I think there are a significant number of changes to be made from the original submitted work. I believe it would be beneficial to submit to another cycle and would result in a stronger submission. I can raise my assessment scores, but I will keep my rating the same.

Reviewer_MbcMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes fidelity and diversity scores for comparing a candidate text dataset with a reference dataset. Both datasets are embedded and represented either by fitted mixture models or by empirical distributions over their embeddings. Fidelity is defined using a nearest-neighbor transport cost and measures whether candidate examples are close to the reference dataset. Diversity is defined using the difference between the full Wasserstein distance and the nearest-neighbor cost and measures the additional cost required to match the full reference distribution. The product of the two scores equals an exponentiated and normalized Wasserstein distance. The paper evaluates the scores on controlled M2D2 topic subsets and synthetic GSM8K datasets. The M2D2 experiments generally separate topic mismatch from topic coverage, while the GSM8K diversity score correlates with the accuracy of models fine-tuned on the synthetic datasets.

Critical strengths
  1. Meaningful and interpretable contribution. The paper provides a simple directional decomposition of dataset mismatch into fidelity and diversity. This is useful because a single symmetric dataset distance cannot indicate whether poor agreement results from unrealistic candidate examples or insufficient coverage of the reference dataset.
  2. Promising initial empirical evidence. The scores generally behave as expected in the controlled M2D2 experiments, and the diversity score is associated with downstream performance in the synthetic GSM8K experiment. The paper also tests multiple embedding models and two fine-tuned model sizes.
Critical weaknesses
  1. The experimental validation is not yet sufficient for a general-purpose dataset metric. The evaluation is limited to two constructed settings, and the paper does not test important failure conditions such as changes in sample size, duplication, outliers, mode collapse, incomplete references, or sensitivity to the metric’s main design choices. Existing metrics also recover much of the controlled GSM8K ordering, so the practical advantage of the proposed metric beyond its interpretability remains unclear.
  2. The main experiments do not fully isolate diversity. In M2D2, the number of topics, dataset size, and fitted GMM complexity change together, and the evaluated datasets are nested. In GSM8K, the experiment uses one nested sequence of the first kk seed problems without repeated seed selection, data generation, or fine-tuning. Increasing kk also increases the number of original GSM8K examples included in training. These factors make it difficult to determine how much of the reported trend is specifically attributable to diversity. The bootstrap intervals do not capture these sources of variation.
Questions for authors
  1. Can the M2D2 experiment be repeated with a fixed number of examples and a controlled representation, such as a fixed number of GMM components? Can the authors also sample multiple independent combinations of topics instead of using one nested sequence?
  2. Can the GSM8K experiment be repeated across randomly selected seed sets, independent synthetic generations, and multiple fine-tuning seeds? Can the original seed problems be excluded from the fine-tuning data so that their changing proportion does not confound the comparison?
  3. The diversity score can be perfect even when the candidate does not cover the reference support. Consider a one-dimensional example with G=δ0G=\delta_0, H=12δ−1+12δ1H=\frac{1}{2}\delta_{-1}+\frac{1}{2}\delta_1, and d(x,y)=∣x−y∣d(x,y)=|x-y|. The nearest-neighbor cost is CNN(G,H)=1C_{\mathrm{NN}}(G,H)=1. Optimal transport splits the mass at 00 equally between −1-1 and 11, so W1(G,H)=12∣0−(−1)∣+12∣0−1∣=1W_1(G,H)=\frac{1}{2}|0-(-1)|+\frac{1}{2}|0-1|=1. The normalization is also κ(H)=Eη,η′∼H[∣η−η′∣]=1\kappa(H)=\mathbb{E}_{\eta,\eta'\sim H}[|\eta-\eta'|]=1. Therefore,

D(G,H)=exp⁡(−W1(G,H)−CNN(G,H)κ(H))=exp⁡(0)=1.D(G,H)=\exp\left(-\frac{W_1(G,H)-C_{\mathrm{NN}}(G,H)}{\kappa(H)}\right)=\exp(0)=1.

However, GG has only one support point and places no mass on either reference point. The total mismatch is assigned entirely to fidelity, with F(G,H)=exp⁡(−1)F(G,H)=\exp(-1), while diversity is perfect. This is mathematically consistent with the proposed decomposition, but it shows that DD measures the additional transport cost after removing the nearest-neighbor cost, rather than reference-support coverage in general. Is this behavior intended, and how should practitioners interpret a perfect diversity score in such cases?

7 comments
AuthorsOpenReview ↗

re: "test important failure conditions such as changes in sample size, duplication, outliers, mode collapse, incomplete references"

We thank the reviewer for raising these important questions. To address these concerns, we conduct new experiments probing the behavior of our metrics under each of these failure modes, using M2D2 Wikipedia data (the Nature subdomain, with results replicated on the Biology subdomain, or the 10 pooled subdomains from Collection 0). In each setting, the behavior of our metrics coincide with our intuitive understanding of fidelity and diversity.

(a) Sample size: We explore the effect of varying the sample size of the reference in our response to reviewer MpRZ's first question; we direct the reviewer to the discussion there. We also ran a similar experiment varying the candidate sample size instead of the reference sample size, on the Nature and Biology subdomains. More specifically, we compute fidelity-diversity scores for $ \lvert G \rvert \in \{200, 500, 1000, 1500\}$ against the full reference split, using the same GMM + BIC method as in Section 4.1. For each choice of $\lvert G \rvert $, we subsampled uniformly without replacement from the full candidate split, with results reported as average ±\pm standard deviation over 8 random seeds for ∣G∣∈200,500,1000,1500\lvert G \rvert \in \\{200, 500, 1000, 1500\\}.

PoolnnFidelityDiversity
Nature15000.5024±0.00210.5024 \pm 0.00210.9660±0.00200.9660 \pm 0.0020
Nature10000.5028±0.00250.5028 \pm 0.00250.9529±0.00180.9529 \pm 0.0018
Nature5000.5010±0.00290.5010 \pm 0.00290.9320±0.00270.9320 \pm 0.0027
Nature2000.5023±0.00500.5023 \pm 0.00500.8989±0.00680.8989 \pm 0.0068
Biology15000.5007±0.00140.5007 \pm 0.00140.9668±0.00120.9668 \pm 0.0012
Biology10000.5007±0.00150.5007 \pm 0.00150.9542±0.00150.9542 \pm 0.0015
Biology5000.4991±0.00280.4991 \pm 0.00280.9348±0.00290.9348 \pm 0.0029
Biology2000.5024±0.00310.5024 \pm 0.00310.9006±0.00400.9006 \pm 0.0040

We observe degradations in measured diversity as ∣G∣|G| shrinks from 15001500 to 200200, with fidelity unaffected. This agrees with the expected behavior since we expect sparse candidate datasets to have worse coverage of the reference, which should degrade diversity and leave fidelity unchanged, in expectation over the points omitted.

(b) Duplication: We explore the effects of duplicates in the candidate dataset using data pooled over all 10 subdomains in the first M2D2 Wikipedia collection. We begin by drawing N=10000N = 10000 reference and candidate points. We then subsample DD distinct points from the candidate uniformly without replacement for each D∈100,150,250,500,750,1000,2500,5000,10000D \in \\{100, 150, 250, 500, 750, 1000, 2500, 5000, 10000\\}. Finally, for all DD, we sample the remaining N−DN-D points from the subset of size DD uniformly with replacement. This gives candidates with the same total size NN, where smaller DD correspond to candidates with less distinct content and more duplicates. We compute fidelity-diversity scores using the unquantized empirical measures (for more discussion on this point, see our response to reviewer MpRZ, question 1) between the candidates and the fixed reference, listed in the table below.

DDFidelityDiversity
100000.4772±0.00030.4772 \pm 0.00030.9731±0.00020.9731 \pm 0.0002
50000.4772±0.00050.4772 \pm 0.00050.9623±0.00050.9623 \pm 0.0005
25000.4772±0.00050.4772 \pm 0.00050.9522±0.00090.9522 \pm 0.0009
10000.4772±0.00140.4772 \pm 0.00140.9375±0.00190.9375 \pm 0.0019
7500.4769±0.00140.4769 \pm 0.00140.9327±0.00180.9327 \pm 0.0018
5000.4761±0.00240.4761 \pm 0.00240.9262±0.00210.9262 \pm 0.0021
2500.4764±0.00190.4764 \pm 0.00190.9137±0.00180.9137 \pm 0.0018
1500.4778±0.00230.4778 \pm 0.00230.9017±0.00430.9017 \pm 0.0043
1000.4774±0.00440.4774 \pm 0.00440.8930±0.00610.8930 \pm 0.0061

Observe that diversity falls as the number of duplicates increases, while fidelity remains stable. Note that even for the sparsest D=100D=100 candidate, our measured diversity is quite high, since uniformly sampling the DD distinct points roughly preserves the coverage of the original candidate dataset (D=ND=N). In contrast, the experiments from Section 4.1 witness steep diversity drops, since they use nested candidate datasets to examine the behavior of our metrics when the candidate (nonuniformly) omits modes of the data distribution.

AuthorsOpenReview ↗

re: "test important failure conditions such as changes in sample size, duplication, outliers, mode collapse, incomplete references" (cont.)

(c) Mode collapse: We study the effect of mode collapse in both the reference (resulting in incomplete references) and candidate datasets. To illustrate the effect of incomplete references on our metric, we simulate selecting such references by first fitting kk-means (with k=20k=20) on the reference dataset, and then dropping data from all but the top mm clusters. To isolate the effect of missing modes from the possibly confounding effect of using a smaller reference, we always resample uniformly with replacement to 1500 total points. We compute fidelity and diversity scores using the same candidate dataset across all choices of mm, and repeat this procedure across 8 random seeds, and also for two pools of M2D2 data.

PoolmmFidelityDiversity
Nature200.4919±0.00200.4919 \pm 0.00200.9677±0.00210.9677 \pm 0.0021
150.4838±0.00340.4838 \pm 0.00340.9702±0.00110.9702 \pm 0.0011
100.4722±0.00610.4722 \pm 0.00610.9720±0.00210.9720 \pm 0.0021
50.4454±0.01040.4454 \pm 0.01040.9751±0.00300.9751 \pm 0.0030
Biology200.4903±0.00220.4903 \pm 0.00220.9688±0.00150.9688 \pm 0.0015
150.4833±0.00100.4833 \pm 0.00100.9703±0.00150.9703 \pm 0.0015
100.4689±0.00440.4689 \pm 0.00440.9731±0.00160.9731 \pm 0.0016
50.4424±0.00580.4424 \pm 0.00580.9743±0.00340.9743 \pm 0.0034

Observe that removing more modes of the reference (smaller mm) consistently results in lower fidelity scores, while diversity scores witness mild improvements. This agrees with the expected behavior, since selecting a reference with less coverage means more of the candidate data lies further from some mode in the reference. Simultaneously, it also becomes easier for a given candidate dataset to cover the reference.

We run the same experiment, but now excise modes of the candidate instead of the reference. Once again, the results follow our intuition that diversity should drop as more modes are dropped, while fidelity remains flat.

PoolmmFidelityDiversity
Nature200.5024±0.00210.5024 \pm 0.00210.9660±0.00200.9660 \pm 0.0020
150.5036±0.00290.5036 \pm 0.00290.9401±0.00460.9401 \pm 0.0046
100.5031±0.00350.5031 \pm 0.00350.9282±0.00360.9282 \pm 0.0036
50.5065±0.00500.5065 \pm 0.00500.9025±0.00920.9025 \pm 0.0092
20.5018±0.01320.5018 \pm 0.01320.8727±0.01230.8727 \pm 0.0123
10.5011±0.02080.5011 \pm 0.02080.8438±0.01450.8438 \pm 0.0145
Biology200.5007±0.00140.5007 \pm 0.00140.9668±0.00120.9668 \pm 0.0012
150.5002±0.00280.5002 \pm 0.00280.9442±0.00250.9442 \pm 0.0025
100.5048±0.00250.5048 \pm 0.00250.9268±0.00460.9268 \pm 0.0046
50.5091±0.00340.5091 \pm 0.00340.8976±0.00520.8976 \pm 0.0052
20.5104±0.01180.5104 \pm 0.01180.8565±0.01090.8565 \pm 0.0109
10.5136±0.01650.5136 \pm 0.01650.8262±0.00970.8262 \pm 0.0097

(d) Outliers: We model the presence of outliers by selecting a source subdomain for the reference and candidate data, and injecting points uniformly at random from a distant subdomain into the candidate. We consider four settings by composing two sources, Nature and Biology, with two distant subdomains, Sports and math.AP. We report scores for f∈0.01,0.05,0.10,0.25f \in \\{0.01, 0.05, 0.10, 0.25\\} the fraction of points replaced by outliers, over 8 random seeds. Observe that fidelity consistently trends downward as a larger fraction of the candidate is replaced by outliers, while diversity remains stable. These trends hold both with quantization (GMM + BIC) and unquantized (uq).

(Source, Outlier)ffuq fiduq divGMM+BIC fidGMM+BIC div
(Nature, Sports)0.010.501±0.0020.501 \pm 0.0020.967±0.0020.967 \pm 0.0020.627±0.0300.627 \pm 0.0300.945±0.0240.945 \pm 0.024
0.050.496±0.0020.496 \pm 0.0020.969±0.0020.969 \pm 0.0020.618±0.0360.618 \pm 0.0360.947±0.0170.947 \pm 0.017
0.100.490±0.0020.490 \pm 0.0020.970±0.0020.970 \pm 0.0020.613±0.0140.613 \pm 0.0140.935±0.0190.935 \pm 0.019
0.250.473±0.0030.473 \pm 0.0030.973±0.0020.973 \pm 0.0020.554±0.0190.554 \pm 0.0190.926±0.0210.926 \pm 0.021
(Nature, math.AP)0.010.502±0.0020.502 \pm 0.0020.966±0.0020.966 \pm 0.0020.633±0.0280.633 \pm 0.0280.950±0.0190.950 \pm 0.019
0.050.499±0.0020.499 \pm 0.0020.967±0.0020.967 \pm 0.0020.637±0.0230.637 \pm 0.0230.946±0.0150.946 \pm 0.015
0.100.496±0.0020.496 \pm 0.0020.967±0.0020.967 \pm 0.0020.611±0.0200.611 \pm 0.0200.942±0.0160.942 \pm 0.016
0.250.486±0.0020.486 \pm 0.0020.965±0.0020.965 \pm 0.0020.560±0.0170.560 \pm 0.0170.943±0.0110.943 \pm 0.011
(Biology, Sports)0.010.500±0.0010.500 \pm 0.0010.967±0.0010.967 \pm 0.0010.647±0.0170.647 \pm 0.0170.953±0.0180.953 \pm 0.018
0.050.495±0.0010.495 \pm 0.0010.969±0.0010.969 \pm 0.0010.636±0.0240.636 \pm 0.0240.947±0.0130.947 \pm 0.013
0.100.491±0.0010.491 \pm 0.0010.971±0.0010.971 \pm 0.0010.619±0.0170.619 \pm 0.0170.948±0.0100.948 \pm 0.010
0.250.476±0.0010.476 \pm 0.0010.972±0.0010.972 \pm 0.0010.587±0.0100.587 \pm 0.0100.935±0.0150.935 \pm 0.015
(Biology, math.AP)0.010.500±0.0010.500 \pm 0.0010.967±0.0010.967 \pm 0.0010.646±0.0200.646 \pm 0.0200.950±0.0170.950 \pm 0.017
0.050.498±0.0020.498 \pm 0.0020.967±0.0010.967 \pm 0.0010.631±0.0370.631 \pm 0.0370.957±0.0150.957 \pm 0.015
0.100.496±0.0020.496 \pm 0.0020.966±0.0010.966 \pm 0.0010.617±0.0270.617 \pm 0.0270.961±0.0130.961 \pm 0.013
0.250.488±0.0020.488 \pm 0.0020.961±0.0010.961 \pm 0.0010.570±0.0200.570 \pm 0.0200.935±0.0200.935 \pm 0.020
AuthorsOpenReview ↗

re: "Can the authors also sample multiple independent combinations of topics instead of using one nested sequence?"

To address this question, we provide some results from using randomly sampled subdomains, instead of nested sequences. However, we first clarify that the original nested design was selected not to make the measurement task easier, but to give a setting in which the clear expectation was to measure an increase in diversity with the number of subdomains. Randomly sampling subdomains introduces ambiguity, since different subdomains can have varying internal diversity.

Nonetheless, we generally expect diversity to increase as more subdomains are included in the candidate, and we verify that this is the case in two Wikipedia and two S2ORC collections. Indeed, we find that over 10 random seeds, the diversity scores computed on GMM+BIC quantized data exhibit significant positive OLS coefficients with the number of subdomains, while fidelity coefficients are not significant.

Collection (C)Diversity OLS β±\beta \pm sd (pp)Fidelity OLS β±\beta \pm sd (pp)
S2 C0+0.01349±0.00532+0.01349 \pm 0.00532 (<0.0001<0.0001)−0.00194±0.00635-0.00194 \pm 0.00635 (0.3600.360)
S2 C1+0.01030±0.00609+0.01030 \pm 0.00609 (0.00050.0005)−0.00147±0.00602-0.00147 \pm 0.00602 (0.4600.460)
wiki C0+0.01470±0.00393+0.01470 \pm 0.00393 (<0.0001<0.0001)−0.00057±0.00673-0.00057 \pm 0.00673 (0.7960.796)
wiki C1+0.01204±0.00577+0.01204 \pm 0.00577 (0.00010.0001)−0.00253±0.00704-0.00253 \pm 0.00704 (0.2860.286)
Reviewer_MpRZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies the quality evaluation of text datasets and proposes a pair of metrics that quantify fidelity and diversity, respectively. In the paper, fidelity indicates the similarity of the evaluated data with the reference data, and diversity represents the coverage of the evaluated data over the reference data. To evaluate these two metrics, the paper first transforms texts into embedding vectors and then represents them as weighted points. Based on this, fidelity is defined as the distance between the candidate points and their nearest reference point, and diversity are defined as the cost difference between the optimal transport and the nearest neighbor matching. The final scores offer a multiplicative decomposition of the Wasserstein distance. The authors offer experiments on the M2D2 subset and GSM8K dataset, reporting that the diversity metric can capture data variance.

Critical strengths
  1. The paper studies the quality evaluation of LLM-generated datasets, which is a timely topic. With the development of large language models and the increasing number of LLM-generated datasets, this problem becomes an increasingly important topic.

  2. The proposed metric separates fidelity from diversity. This setting can avoid evaluation error when a dataset is close to the reference data while covering a limited portion of its distribution.

  3. The paper provides extensive experiments. The evaluation consists of controlled topic-based experiments and downstream training applications.

Critical weaknesses
  1. The construction of the reference data and its sensitivity remain unclear. The proposed metrics rely on the reference data, which determines whether the evaluation results accurately reflect the actual characteristics of the evaluated data. However, the paper provides limited guidance on general rules of reference construction.

  2. The computational scalability of the proposed metrics is insufficiently characterized. Although the paper employs GMM quantization and a debiased Sinkhorn approximation, the scalability of the proposed metrics on larger datasets is still a concern. The authors should provide a complexity analysis or runtime empirical results to clarify this.

  3. The paper lacks comparisons with a directly related baseline. The paper focuses on the quality evaluation of LLM-generated datasets, which is highly related to DCScore [1].  Although the paper compares the proposed metrics with several existing metrics (FID, MAUVE, VENDI score), it does not include a comparison with DCScore.

  4. The rationale for the generated GSM8K reference needs further clarification. The authors use fully synthetic GSM8K data as the reference data, but whether the synthetic dataset preserves the structural coverage of the original dataset is not validated. Furthermore, previous works have shown shifted linguistic distributions [2] and diversity reduction [3] in generated text. It is also necessary to clarify whether these phenomena have an impact on the reference data.

References:

[1] Measuring Diversity in Synthetic Datasets

[2] Domain Regeneration: How well do LLMs match syntactic properties of text domains?

[3] Benchmarking Linguistic Diversity of Large Language Models

Questions for authors

Refer to Critical Weaknesses for more details.

7 comments
AuthorsOpenReview ↗

re: "1. The construction of the reference data and its sensitivity remain unclear..."

Selecting the reference is crucially important for our method to work well, and we thank the reviewer for highlighting this point. In general, we recommend the following heuristics for selecting a reference, some of which were implicit in our paper but are worth stating more explicitly.

An obvious requirement is that the reference dataset HH should adequately cover the target distribution, since our method cannot detect a candidate's failure (of fidelity) to cover a mode that is also missing in the reference. We explore the effects of reducing coverage of the reference in section B.3 of the appendix, as well as in our response to question 1(c) from Reviewer MbcM. A more subtle requirement is that ∣H∣|H| should be large enough to produce stable quantization fits (if we choose to quantize) and fidelity-diversity scores. The number of samples required thus depends on the choice of quantization, whether the data exhibits concentration around modes, and the desired level of stability.

On a related note, the choice of quantization method and level of resolution (call this parameter kk) matter significantly for efficiently producing meaningful fidelity-diversity scores. As usual, the right choice of quantization and kk depends on the reference / candidate samples and data distribution. For instance, we recommend performing quantization if the data is clustered, with reference and candidate kks selected to be at least as large as the number of clusters we expect in the corresponding datasets. The M2D2 validation experiments provide one such example of this, where a criterion optimizing for density fit (BIC) selects reasonable kk in clustered data. However, if the data exhibits no clear clusters or contains many duplicates, quantization via likelihood-based mixtures no longer produce stable discrete summaries of the data and should not be used. Instead, we recommend using the unquantized empirical measure, as demonstrated in the GSM8K experiments.

To address these concerns about sensitivity to the choice of reference, we conduct an additional experiment on Collection 0 of the M2D2 Wikipedia data to explore the effect of sample size ∣H∣|H|. We compute fidelity-diversity scores for ∣H∣∈200,500,1000,2000,5000,10000|H| \in \\{200, 500, 1000, 2000, 5000, 10000\\} against the full candidate dataset held fixed at ∣G∣=14556|G| = 14556, using the same GMM + BIC method as in Section 4.1. For each choice of ∣H∣|H|, we subsampled uniformly without replacement from the full reference (which contained 14550 points). We report average ±\pm standard deviation over 8 random seeds for ∣H∣∈200,500,1000,2000|H| \in \\{200, 500, 1000, 2000\\} and 3 random seeds for ∣H∣∈5000,10000|H| \in \\{5000, 10000\\}.

∣H∣\lvert H \rvertBIC kkFidelityDiversity
2005.00.333±0.0160.333 \pm 0.0160.970±0.0130.970 \pm 0.013
5005.00.331±0.0110.331 \pm 0.0110.978±0.0060.978 \pm 0.006
10006.20.352±0.0520.352 \pm 0.0520.978±0.0060.978 \pm 0.006
200015.00.478±0.0070.478 \pm 0.0070.978±0.0040.978 \pm 0.004
500030.00.545±0.0020.545 \pm 0.0020.978±0.0050.978 \pm 0.005
1000066.70.608±0.0150.608 \pm 0.0150.966±0.0080.966 \pm 0.008
1455090.00.6310.6310.9530.953
AuthorsOpenReview ↗

re: "2. The computational scalability of the proposed metrics is insufficiently characterized."

The cost of our metrics is dominated by embedding and quantization, both linear in corpus size; the transport computation itself is independent of corpus size. Let NN be the number of text units, dd the embedding dimension, kk the number of atoms per measure, TT the number of kk-means iterations, RR the number of kk-means restarts, and CmodelC_{\text{model}} the cost of embedding a single unit. The table below breaks down the cost by stage.

Computational cost of each stage.

StageCost
EmbeddingO(N⋅Cmodel)O(N \cdot C_{\text{model}})
Quantization (kk-means)O(R,T,N,k,d)O(R\\,T\\,N\\,k\\,d)
Quantization (GMM, BIC over grid K\mathcal{K})O(R,T,N,d∑k∈Kk)O\big(R\\,T\\,N\\,d \sum_{k \in \mathcal{K}} k\big)
Cost matrix between atomsO(kG,kH,d)O(k_G\\, k_H\\, d)
Nearest-neighbour projection (fidelity)O(kG,kH)O(k_G\\, k_H)
Exact OT (network simplex)O(k3log⁡k)O(k^3 \log k) worst case
Debiased SinkhornO(kG,kH)O(k_G\\, k_H) per iteration
Scaling factor κ(H)\kappa(H)O(kH2,d)O(k_H^2\\, d)

The key property is that quantization decouples corpus size from transport cost: once each corpus is summarized by kk atoms, every subsequent step is a function of kk alone. With quantization, the method scales to corpora far larger than the ones we study, bounded in practice by the cost of embedding.

AuthorsOpenReview ↗

re: "3. The paper lacks comparisons with a directly related baseline [DCScore]."

We thank the reviewer for pointing us to this interesting and relevant work. Like the Vendi score, DCScore is a reference-free score and thus does not allow the user to encode a desired target distribution through the specified reference.

We provide a summary comparison of our diversity score and DCScore on the first M2D2 Wikipedia and S2ORC collections and the GSM8K data. As in [1], we report Spearman's ρ\rho in each setting. We find that DCScore with the best choice of τ∈1.0,0.5,0.2,0.1,0.05,0.02,0.01\tau \in \\{1.0, 0.5, 0.2, 0.1, 0.05, 0.02, 0.01\\} serves as a strong baseline for computing a reference-free notion of diversity. If accepted, we will update our paper to include DCScore in our discussion of existing methods and comparison studies.

SettingDCScore, τ=1\tau = 1 (default)DCScore, best τ=0.2\tau = 0.2Diversity (ours)
GSM8Kρ=+0.893\rho = +0.893ρ=+1.000\rho = +1.000ρ=+1.000\rho = +1.000
M2D2 Wikiρ=+0.976\rho = +0.976ρ=+1.000\rho = +1.000ρ=+0.988\rho = +0.988
M2D2 S2ρ=+0.195\rho = +0.195ρ=+1.000\rho = +1.000ρ=+1.000\rho = +1.000

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.