acceptodds
Under review as a conference paper at ICLR 2027

SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics

13 views

Abstract

Evaluating text generation requires measuring how well the generated distribution matches the data distribution. For autoregressive models, this is done by the per- plexity. Diffusion and flow-based language models can only provide a likelihood bound, whose tightness differs between model families. Sample-based substitutes such as generative perplexity with entropy do not consider the distribution fit. We propose SOL, a distance between text distributions. Each sequence is represented by the empirical measure of its hidden states under a fixed transformer and the distributions of these measures are compared by the double sliced Wasserstein dis- tance. We prove that SOL is a metric if the transformer is injective. Experiments show that SOL detects distributional failures, recovers expected model trends, and provides stable sample-based estimates. We put forward SOL to fill the gap in the current evaluation protocol used for non auto-regressive models. As a first step we use SOL to re-evaluate a variety of models trained on OpenWebText.

Primary area:
datasets and benchmarks
Keywords:
Diffusion Language Models, Evaluation, Language Metric, Optimal Transport, Flow Based Model, Sliced Optimal Transport

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xEaiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper mainly proposes SOL. Its core idea is that existing likelihood-based and sample-based evaluation metrics on text generation are relatively poor, so instead, it compared how close the distribution of model-generated text is to the distribution of real text. To this end, the paper proposes a new metric for evaluating text quality. Since computing this metric using Wasserstein-on-Wasserstein is relatively difficult, it uses Double-Sliced Wasserstein to compute it.

The paper then conducts several experiments. The first examines the mode collapse problem of generation perplexity. The second reuses the human judgments from the MAUVE paper to compare how well automatic metrics agree with human rankings of eight GPT-2 generation configurations. Finally, it reevaluates DLMs against AR models and finds that the ordering of Gen. PPL does not always agree with that of SOL.

Critical strengths

Generative perplexity is indeed a poor metric, although this is probably already a consensus in the field[1][2]. However, there is still no widely accepted, better alternative to the existing automatic evaluation metrics. Therefore, I think this is a good problem to address, and it is indeed worth solving.

[1] Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics. arXiv 2606.08417

[2] On the Relation between Quality-Diversity Evaluation and Distribution-Fitting Goal in Text Generation. ICML 2020

Critical weaknesses
  1. My first concern is that, if this metric is intended to be widely adopted, I think its many complicated design choices may hinder its adoption. For example, many widely used metrics, such as entropy, perplexity, have relatively simple designs, and it is fairly intuitive why they should work. Although some of them were later shown to be vulnerable to hacking, they were still relatively reliable metrics for a long time. With such a complicated design, I suspect there may be many potential vulnerabilities to exploit(like [3][4]). In other words, we do not know how robust SOL is against hacking. This also leads to a related question: SOL introduces a substantial number of design choices, but have the authors sufficiently demonstrated the necessity of these choices and the robustness of the resulting model rankings to them?
  2. My next concern is that the evidence for alignment between SOL and human judgments is insufficient. The paper reuses human judgments from the MAUVE paper, but ultimately, I think a new metric still needs to be validated through human evaluation[5]. Without new human evaluations, it is difficult to determine how useful this score actually is. Even if SOL is indeed a good metric at the distributional level, it would still be of little use if the generated text looks very poor to humans. The fact that SOL does not agree with Gen. PPL does not necessarily mean that SOL itself is a good metric.
  3. Furthermore, the results in Appendix D.9 do not seem to demonstrate that SOL is particularly stable.(correct me if i am wrong with that) Simply changing the encoding from [x] to [x, x] changes the optimal sampling temperature from 0.9 to 1.3, which raises further concerns about the robustness of the metric.

[3] On the Blind Spots of Model-Based Evaluation Metrics for Text Generation. ACL 2023

[4] MENLI: Robust Evaluation Metrics from Natural Language Inference. TACL 2023

[5] SummEval: Re-evaluating Summarization Evaluation. TACL 2021

Questions for authors
  1. The first question is whether the model rankings produced by SOL have any practical significance for evaluation. Since the paper proposes SOL as a practical language model evaluation metric, what sufficient evidence do the authors provide to demonstrate that a better SOL score leads to a more meaningful comparison of model quality? also, is there any evidence that SOL in Table 3/22 also align with human judgments across these models?

  2. The second question is related to the weakness mentioned above. The authors' own experiments have already shown that one representation choice can change which model is selected as the best. What about the other design choices? How robust are the resulting model rankings to these choices?

3 comments

We thank the reviewer for the careful reading and the detailed comments.

“If this metric is intended to be widely adopted, I think its many complicated design choices may hinder its adoption.”

We agree that the practical setup should be simple. A user chooses only the encoder and the reference set. Parameters have fixed defaults: 5,000 samples, Lξ=Lg=8192L_\xi=L_g=8192 slices and the bidirectional Dream encoder. Our implementation uses these defaults.

“We do not know how robust SOL is against hacking.”

Gen. PPL can be improved without matching the data because its optimum is not the data distribution. It is minimized by a single sequence, the most likely one under the reference model. SOL is different. For an injective encoder, SOL is a metric, and its only minimizer is the data distribution. Lowering SOL therefore means moving closer to the data in the encoder's representation. Both examples in G2 reach very good Gen. PPL and entropy, and SOL does not reward them. We do not claim robustness against adversarial attacks on a specific encoder.

“SOL introduces a substantial number of design choices, but have the authors sufficiently demonstrated the necessity of these choices and the robustness of the resulting model rankings to them?”

The design choices fall into two groups.

The first group consists of modeling choices, and we show why each is needed. A single last-token state misses changes at the beginning of a text (Figure 3). A mean-pooled vector is less sensitive than the full set of token states in nine of ten structural perturbations. Causal encoders react mainly to changes early in the text, while the bidirectional Dream encoder reacts evenly. The model rankings, however, barely depend on the encoder. Across GPT-2 Large, OLMo, and Dream, the rank correlations over 36 model configurations are 0.990 to 0.996.

The second group consists of numerical choices, which only control how accurately SOL is computed. Going from 1,024 to 8,192 slices changes SOL by at most 1.5%, and the quantile resolution changes it by less than 0.1%. At the defaults, these approximations have converged.

“The paper reuses human judgments from the MAUVE paper, but ultimately, I think a new metric still needs to be validated through human evaluation.”

We agree. We ran a new human study directly on the OpenWebText models from our experiments (G1, Section 5.4, Appendix E.2). The human ranking and SOL have a Spearman correlation of 0.970.97.

“Simply changing the encoding from [x][x] to [x,x][x,x] changes the optimal sampling temperature from 0.9 to 1.3, which raises further concerns about the robustness of the metric.”

The [x,x][x,x] representation of [1] is not part of SOL. It is an alternative that we tested and rejected. SOL uses [x][x].

Appendix D.8 explains the shift. In the second copy, the encoder predicts each token from the first copy with 99% top-1 accuracy. High-temperature samples are then no longer surprising, and the score mainly reflects word choice. With a separator, [x,sep,x][x,\mathrm{sep},x] again prefers temperature 0.9. We will rename this appendix section to make clear that [x,x][x,x] is not used by SOL.

“What sufficient evidence do the authors provide to demonstrate that a better SOL score leads to a more meaningful comparison of model quality? Also, is there any evidence that SOL in Table 3/22 also align with human judgments across these models?”

The study in G1 tests this directly on the OpenWebText benchmark. It covers real OpenWebText, Soft-Masked, AR, LangFlow, DUO, MDLM, and FMLM. The human annotators and SOL clearly agree on the rankings of the tested models.

Furthermore, the new baselines in G3 show a case that Gen. PPL and entropy cannot resolve. MDLM-MMD [2] samples in 16 steps, while MDLM with ReMDM [3] uses 1,024 steps with remasking. We therefore expect a clear quality gap. However, both have almost the same Gen. PPL (28.628.6 and 28.928.9), and MDLM-MMD also matches the data entropy. SOL separates them clearly, with 56.856.8 against 23.623.6 for Dream.

“What about the other design choices? How robust are the resulting model rankings to these choices?”

For the encoder, the model rankings are almost unchanged: across GPT-2 Large, OLMo, and Dream, the rank correlations over 36 model configurations are 0.990 to 0.996. The numerical parameters such as number of slices have converged at the defaults we suggest.

We are happy to answer any further questions. If our responses resolve the reviewer's concerns, we kindly ask the reviewer to raise the score.

[1] Repetition Improves Language Model Embeddings, Springer et al., 2025.

[2] Representation-Space MMD for Diffusion Language Models, Drobyshevskiy et al., 2026.

[3] Remasking Discrete Diffusion Models with Inference-Time Scaling, Wang et al., 2025.

Thank you for the clarifications and additional experiments. However, the new human study uses only 163 judgments (77 from one annotator), and Opus 5.5 reverses SOL on MDLM versus LangFlow and DUO. Rankings of close models also flip across encoders (such as LDDM-U versus AR in Table 26), leaving my robustness concerns unresolved. Therefore, I will maintain my current score.

Thank you for the reply. We see the remaining concerns differently. Below we lay out our reasoning step by step:

 

  1. In your original review, you agree that Gen. PPL is a poor metric and that a better alternative is worth developing. A main reason is that Gen. PPL is not a distributional metric. It scores single sequences and is minimized by a single sequence. A better alternative therefore has to compare distributions.

     

  2. Token sequences carry no meaningful geometry, so in practice this comparison needs an encoder. Every distributional metric for text therefore depends on an encoder, e.g. MAUVE. FID depends on an encoder in the same way in the image setting. Although FID has well known problems, we would argue that it is still useful for the image community.

     

  3. The existing encoder-based metrics for text show pathological behavior. We highlight this throughout the paper and the revision.

     

  4. Opus 5.5 reverses the ranking of SOL only within the middle group of LangFlow, DUO, and MDLM. In the human study, these three models are close, so some disagreement between judges is to be expected. Opus agrees with SOL on all other pairs. We do not think that these reversals invalidate the method.

     

  5. SOL avoids the pathological behavior of Gen. PPL and MAUVE described in steps 1 and 3, and its ranking agrees with human judgment (ρ=0.97\rho = 0.97 on OpenWebText). Close pairs such as LDDM-U and AR can change order across encoders, as for every encoder-based metric (step 2). Still, 96.7% to 98.3% of model pairs keep their order. As for FID, the encoder is fixed and reported together with the score.

     

Taken together, the current choice is between Gen. PPL, which does not compare distributions, existing encoder-based metrics with failures such as those described above, and SOL. SOL is the only option that compares distributions while avoiding these failures. In particular SOL is theoretically justified and is an actual distance under light assumptions on the encoder. We believe it is a clear improvement over the metrics in current use.

If you disagree with one of the steps above, we would be grateful to know which one, so that we can address it directly. Otherwise, we kindly ask you to reconsider your assessment of our work.

Reviewer_MdMGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose SOL, a new evaluation metric for quantifying the quality of generated test by leveraging the double sliced wasserstein distance between text distributions. In practice, the authors employ a learned causal transformer to map the text to a compact representation to get empirical meta-measures on Rd\mathbb{R}^d. The authors provide statistical gurantees for their estimator and validate SOL on expected behaviour and correlation with human judgement.

Critical strengths
  • The paper is well motivated and tackles reall issues in evaluating generated text: the mode collapse scenario is a practical testament to why users need to use SOL instead of just measuring gen. ppl and entropy.

  • The authors show that the method is practical and lightweight, providing an ideal configuration for practicioners.

Critical weaknesses
  • The proposed estimator appears to have an irreducible error as per equation (9), which depends on the choice of reference measure γ\gamma and variance of the estimator remain unclear. While this error can be mitigated with increasing the number of slices, which the authors show empirically in figure 6, the factors that determine bias and variance of the estimators are unexplored.

  • It is difficult to position the work in the current literature. It is not clear why one should prefer SOL over the gradient moment metric in [1] or the distance-based metrics proposed in [2], and more in general it is not clear to me why the author chose the wasserstein distance and not another distance.

[1] Hogeboom et al. ; Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD [2] Franca et al. ; Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics

Questions for authors
  • Can the authors show what is the bias and variance of the proposed estimator for SOL? What are the factors behind the two? For example, the authors mention in section 5 that for DREAM they had to increase the number of slices. I am interested if there are other factors into play and how they influence bias and variance.
  • Why did the authors choose the Wasserstein distance to build SOL? More in general, why do the authors think that a distance and not, for example, a divergence is desirable as an evaluation metric?
4 comments

We thank the reviewer for the detailed comments and the additional references.

“The proposed estimator appears to have an irreducible error as per equation (9), which depends on the choice of reference measure γ\gamma and variance of the estimator remain unclear.”

To be clear, there is no irreducible term in (9). The bound is

E∣DSWLξ,Lg(μn,νn∣γ)−DSW(μ,ν∣γ)∣≤8Rm2(γ)n1/4+2R(m2(γ)2Lξ+m4(γ)Lg)1/4\mathbb{E}\left| \mathrm{DSW}_ {L_\xi,L_g}(\mu_n,\nu_n\mid\gamma) - \mathrm{DSW}(\mu,\nu\mid\gamma) \right| \leq \frac{\sqrt{8}R\sqrt{m_2(\gamma)}}{n^{1/4}} + 2R\left( \frac{m_2(\gamma)^2}{L_\xi} + \frac{m_4(\gamma)}{L_g} \right)^{1/4}

The first term is the error from using nn samples. The second is the error from using finitely many slices. Both vanish as n,Lξ,Lg→∞n, L_\xi, L_g \to \infty. The reference measure γ\gamma enters only through the constants m2(γ)m_2(\gamma) and m4(γ)m_4(\gamma). It determines which distance is computed, but not whether it is estimated consistently.

“Can the authors show what is the bias and variance of the proposed estimator for SOL? What are the factors behind the two? For example, the authors mention in section 5 that for DREAM they had to increase the number of slices.”

We thank the reviewer for this question. The error has two sources.

First, the slices are random. For fixed texts, this only adds noise around the value for these texts. The noise has mean zero for SOL2\mathrm{SOL}^2, and its variance is of order 1/Lξ+1/Lg1/L_\xi + 1/L_g.

Second, using nn samples adds a finite-sample error. It is bounded by the first term in (9) and vanishes at rate n−1/4n^{-1/4}.

Dream has a larger hidden dimension than GPT-2 Large (35843584 vs. 12801280) and therefore larger variance over slices. Going from 1,024 to 8,192 slices lowers the relative standard deviation from 2.8% to 0.5%, while the mean changes by less than 1%.

“It is not clear why one should prefer SOL over the gradient moment metric in [1] or the distance-based metrics proposed in [2] [...]”

We thank the reviewer for the references. We implemented the gradient moment (GM) metric of [1] (see G4). Larger GPT-2 models should be closer to WebText. SOL recovers this order at 256 and 512 tokens, while GM ranks GPT-2 Medium best at both. GM compares only mean gradients of the reference model, and different text distributions can share the same mean gradient.

The metrics in [2] represent each text by a single mean-pooled vector. SOL represents each text by the distribution of its contextual token states. In our ablation, this makes SOL more sensitive than mean pooling in nine of ten structural perturbations, such as word swaps and sentence permutations.

“Why did the authors choose the Wasserstein distance to build SOL?”

The Wasserstein distance is a well-studied metric between probability distributions. Its main drawback is the computational cost in high dimensions. Slicing alleviates this, since the one-dimensional problems reduce to sorting. For 5,000 texts, SOL takes about one second once the hidden states are computed. Other distances would be possible, but this combination made Wasserstein a natural choice.

“More in general, why do the authors think that a distance and not, for example, a divergence is desirable as an evaluation metric?”

For evaluation, we want the metric properties. A metric is zero exactly when the generated and reference distributions agree. By the triangle inequality, the scores of two models differ by at most the distance between their distributions. Practically divergences such as KL can become infinite under support mismatch and require densities or a quantization as in MAUVE.

We hope the added experiments and clarifications address the concerns behind the rating and we are happy to answer any follow-up.

[1] Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD, Hoogeboom et al., 2026.

[2] Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics, Franca and Tong, 2026.

I thank the authors for their rebuttal. I am satisfied with the comparison with GM and the following discussion in D.11 and the comparison with [2]. Likewise, the motivations behind modeling choices I raised in my questions are now clear.

I have follow-up questions regarding the statistical properties of the estimator used for SOL: the authors note that increasing the dimensionality increases the variance of the estimate, which can also be seen when comparing figure 7 with figure 9. Does the constant RR in equation 9 absorb the dimensionality scaling that emerges from the experiments?

My question is motivated by the fact that the authors in proposition 4.2 use an approach similar to [3], where they assume a fixed constant RR to include the support in a ball of radius RR in Rd\mathbb{R}^d, while other approaches that study the statistical properties of sliced wasserstein distances, like [4], prefer to avoid this by using other set of assumptions to showcase a more transparent relation to dimensionality and the properties of the underlying data distribution. This is particularly interesting because the authors use the double sliced Wasserstein distance for which this kind of analysis remains, to my knowledge, underexplored, and because in the proposed work dimensionality and properties of the distributions used to compute SOL depend not only on the generated text but also on the pretrained model used to map to Rd\mathbb{R}^d. Can the authors elaborate on how these elements affect their convergence analysis? What practical implications this has for selecting a pretrained transformer and ensuring the reliability and comparability of SOL across diverse datasets and language models?

[2] Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics, Franca and Tong, 2026.

[3] Nadjahi et al. Statistical and Topological Properties of Sliced Probability Divergences

[4] Nietert et al. Statistical, Robustness, and Computational Guarantees for Sliced Wasserstein Distances

AuthorsOpenReview ↗

Properties of Sliced Estimators

Bounds on the Monte-Carlo Error

In our revised version, we added a clarifying remark after Prop 4.1 (lines 337-339):

The bound is uniform over the distributions, while variance-based Monte Carlo bounds (Nadjahi et al., 2020, Theorem 6) illustrate why the slicing estimators may scale poorly with dimension (Nietert et al., 2022).

Our bound in Eq. (9) is uniform over distributions supported in a ball of radius RR. Dimensional dependence enters implicitly through RR and the moments of the reference measure γ\gamma. Alternatively, as in Nadjahi et al. [3, Theorem 6], distribution-dependent Monte Carlo bounds can be expressed in terms of the variance of projected transport costs. The difference between these two approaches is nicely described in the introduction of Nietert et al. [4]. We aimed for a uniform statement, since such a statement applies across meta-measures (i.e., across language models). Nevertheless, the same principle applies to DSW, where both levels of slicing contribute to the variability.

Generally speaking, the issue illustrated by variance-based bounds is not necessarily dimensionality, but rather the anisotropy of the distributions, which can become more pronounced in higher dimensions. For example, consider the standard sliced Wasserstein distance with two concentric spherical distributions μ=U(rSd−1)\mu=U(r S^{d-1}) and ν=U(sSd−1)\nu=U(s S^{d-1}), with r,s>0r,s>0. Their projected cost W22(eξ,μ,eξ,ν)=(r−s)2/dW_2^2(e_\xi,\mu, e_\xi,\nu)=(r-s)^2/d is constant across directions, leading to zero Monte Carlo slicing error in any dimension. Applying an anisotropic linear map AA to both distributions changes the projected cost to (r−s)2∣ATξ∣2/d(r-s)^2|A^T\xi|^2/d, introducing projection variance and hence Monte Carlo error.

In most practical applications, the number of required projections for slicing estimators often grows with the dimension. Our best explanation is that the primary driver is not dimensionality itself, but the tendency of higher-dimensional real-world distributions to become more anisotropic.

Practical Implications

Our compact-support assumption is natural for language evaluation, since a fixed transformer maps a finite vocabulary with bounded sequence length into a bounded space.

In practice, we keep the reference encoder fixed when comparing generative models, ensuring a common underlying metric. Across datasets, the convergence guarantee remains valid, although the actual slicing variance may differ. For new encoders, particularly higher-dimensional ones, we recommend checking numerical convergence as in Figures 6 and 7. However, extensive experiments already establish Dream as a reliable reference backbone, with suitable numbers of samples and slices.

[3] Nadjahi et al. Statistical and Topological Properties of Sliced Probability Divergences

[4] Nietert et al. Statistical, Robustness, and Computational Guarantees for Sliced Wasserstein Distances

I thank the authors for the discussion, I still think that this direction requires more investigation, as the approach by [4] provides insights that the current uniform approach doesn't provide. Nevertheless most of my concerns have been solved and I will increase my score to weak accept, I encourage the authors to include this discussion in the paper and warn practicioners of what they might miss with Eq. (9) as is now, together with an intuition of what they might expect with a different approach.

Reviewer_XmwcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a sample-based metric for comparing generated text distributions with real data distributions. It represents each text sequence as an empirical distribution of hidden states extracted from a fixed transformer, and then uses double-sliced Wasserstein distance to compare the resulting distributions. A theoretical result is provided to show that SOL satisfies the metric property under an injectivity assumption. The experiments suggest that SOL captures some distributional differences that existing metrics may overlook.

Critical strengths
  1. The motivation is convincing. SOL offers a common evaluation approach that only requires generated samples and reference texts. This is useful for evaluating diffusion language models with different generation and sampling procedures.
  2. The experiments provide useful evidence for the proposed metric. Generative perplexity improves even when the generated distribution collapses, while SOL reflects the increasing distributional mismatch.
Critical weaknesses
  1. SOL relies heavily on the representations from a fixed transformer. Although the authors evaluate several embedding models, the choice of backbone affects the resulting scores.
  2. The correlation with human judgments is encouraging, but the evaluation is based on eight GPT-2 model and decoding configurations. The comparison seems to be less direct. Stronger human evaluation would make the conclusions more convincing.
Questions for authors
  1. How should the reference embedding model be selected in practice? Since different encoders can produce different model rankings, it would be helpful to understand whether there is a recommended choice.
  2. How confident are the human rankings from the original GPT-2 study remain representative of the newly generated samples? Is there any additional evidence that the correlation holds for diffusion language models?
1 comment

We thank the reviewer for the positive assessment and the detailed comments.

“SOL relies heavily on the representations from a fixed transformer. Although the authors evaluate several embedding models, the choice of backbone affects the resulting scores.”

We agree that the absolute SOL value depends on the encoder. Scores should therefore be compared with the same encoder and reference set. The rankings, however, change very little. Across the 36 configurations we evaluated with all three encoders, the rank correlations are 0.9900.990--0.9960.996, and 96.796.7%--98.398.3% of model pairs are ordered identically.

“How should the reference embedding model be selected in practice?”

Our recommendation is Dream for final evaluation. It weights positions more evenly than causal encoders and agrees best with the human judgments. GPT-2 Large is about 5×5\times cheaper and gives almost the same ranking, so it is useful for cheaper evaluation during development.

“The correlation with human judgments is encouraging, but the evaluation is based on eight GPT-2 model and decoding configurations. The comparison seems to be less direct. Stronger human evaluation would make the conclusions more convincing.”

We agree and added the direct human study in G1 (Section 5.4, Appendix E.2) on the OpenWebText models from our experiments. It includes five diffusion/flow models and the AR baseline. The human ranking agrees with SOL at ρ=0.97\rho=0.97, see also our general answer.

“How confident are the human rankings from the original GPT-2 study remain representative of the newly generated samples? Is there any additional evidence that the correlation holds for diffusion language models?”

For the original comparison, we use the same public checkpoints, decoding parameters, and prompting protocol as MAUVE. We also check five generation seeds and the 256-token setting used in their human study. More importantly, the new study in G1 (Section 5.4, Appendix E.2) no longer relies on transferring the MAUVE judgments: it directly evaluates the diffusion/flow models considered here.

We hope these additions address the questions and concerns. If so, we would appreciate reconsideration of the score.

Comments on the paper

1 comment

General response

We thank the reviewers for the careful reading and the detailed comments. We summarize here the main additions in the revised paper, marked in blue. We address each reviewer's individual comments and questions below.

G1. Human study on OpenWebText models (Section 5.4, Appendix E.2). We added a blind pairwise study on real OpenWebText and six models from our experiments, five of them diffusion or flow models. Annotators compare two full 1,024-token samples.

We collected 163 judgments from 8 annotators. The human ranking and SOL have a Spearman correlation of ρ=0.97\rho=0.97. In 20 of the 21 model pairs, the annotators prefer the model with the better SOL. The remaining pair is tied.

We also collected 1,510 judgments from Claude Opus 5.5. Its ranking has correlation ρ=0.89\rho=0.89 with SOL. We find that human annotators and Opus 5.5 agree with SOL on characterizing model quality.

G2. Gen. PPL and degenerate samples (Section 3 and Section 5.6). We added two examples with very good Gen. PPL and entropy while the samples are clearly flawed.

Using a model with a significantly modified sampler lowers Gen. PPL from 4747 to 3838 at entropy 5.425.42. At the same time, 6565% of the 4-grams within a sample are repeats, compared with 77% for the original sampler and 55% for real text.

We also added a released-model example. CoBit-M [1] has Gen. PPL 19.719.7 and entropy 5.385.38. However, 5252% of its samples contain the same two web footers, compared with 0.20.2% of real text. SOL is 38.038.0 for CoBit-M and 22.222.2 for the AR baseline.

G3. New baselines (Tables 4 and 26, Appendix C.5). We added MDLM with ReMDM sampling [2], MDLM-MMD [3], CoBit-S [1], and CoBit-M [1], using the authors' released code. MDLM-MMD and MDLM with ReMDM have almost the same Gen. PPL (28.628.6 and 28.928.9), and MDLM-MMD matches the data entropy. Their SOL values are 56.856.8 and 23.623.6.

G4. Gradient moments (Appendix D.10). We added the gradient moment metric of Hoogeboom et al. [4]. In the GPT-2 model-size experiment, larger GPT-2 models should be progressively closer to WebText. SOL recovers this ordering at both 256 and 512 tokens: Small, Medium, Large, XL. Gradient moments do not and instead rank GPT-2 Medium best at both lengths.

[1] CoBit: Language Modeling with Bitstream Diffusion, Batzolis et al., 2026.

[2] Remasking Discrete Diffusion Models with Inference-Time Scaling, Wang et al., 2025.

[3] Representation-Space MMD for Diffusion Language Models, Drobyshevskiy et al., 2026.

[4] Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMD, Hoogeboom et al., 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.