- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper describes a new alpha cross-entropy loss for training diffusion language models. They show that this improves performance on math and coding benchmarks.
Critical strengths
- Empirical results show improvement over baseline
- The method seems to remove incorrect parallel predictions in diffusion language models, especially at low NFE, due to their sequence-level alpha loss.
Critical weaknesses
- I think the learned distribution is different than the actual ground truth joint data distribution . It is not obvious to me why equation (12) (or any of the other equations, for that matter), would support learning to decode the joint data distribution. Fundamentally, it still makes mean-field token-wise predictions whose log-probs are simply added together. That is, the actual predictor model still does not seem to have interdependence. I can believe that this learns something different than the naive token-wise marginal distributions, but I still don't think that actually gives the ground truth joint data distribution. What distribution is actually being learned here? Can you characterize it in terms of the joint data distribution? Is it some tilted version of the ground truth joint data distribution? This is a serious problem because if our model isn't trained to learn the underlying ground truth distribution, then we can't really rely on it to make valid predictions in general.
- Proposition 1 seems to have an unrealistic assumption that the different completions will be disjoint token-wise ("different groups use disjoint token sets at every position"). For instance, with the prefix "
Hi!Mynameis", we could have valid completions "JohnSmithWang" or "JuliaSmithPatel". The middle token of both of those completions is stillSmith. What would your loss do under this scenario?
Questions for authors
See weaknesses. To address them, could you try the following experiment: Construct a synthetic low-dimensional dataset for which the joint probabilities are known (e.g., sequences of length 8, with vocabulary of A, B, C, D). Report how the learned probabilities differ from the ground truth probabilities. I also want more rigorous theoretical analysis of your loss's learned distributions.
3 comments
Rebuttal 2
Q.1
... could you try the following experiment: Construct a synthetic low-dimensional dataset for which the joint probabilities are known (e.g., sequences of length 8, with vocabulary of A, B, C, D). ... I also want more rigorous theoretical analysis of your loss's learned distributions.
We ran this experiment (new Appendix G). Sequences have length 8 over the vocabulary A, B, C, D, and all 65,536 probabilities are enumerated exactly, so there is no sampling error. The data distribution is a Markov chain in which adjacent letters must differ: the first letter has probabilities (0.4, 0.3, 0.2, 0.1) for A, B, C, D, and each next letter is drawn from the same weights restricted to letters different from the previous one. A sequence is valid if no two adjacent letters are equal, which holds for 8,748 sequences. It does not satisfy the assumptions of Proposition 1.
One parallel step. All eight tokens are predicted at once, independently. We use the exact cross-entropy optimum and numerically optimize the alpha loss over factorized distributions. We enumerate all 65,536 sequences to calculate validity and total variation distance to the data.
We also calculate how many different valid sequences we expect to see in 1,000 independent generations. Repeated answers count only once, and invalid answers do not count. The column "Distinct valid / 1,000" therefore measures both correctness and variety. We compute this expectation directly from the probabilities, without sampling.
| Predictor | Valid (%) | Distinct valid / 1,000 | TV to data |
|---|---|---|---|
| MDLM (cross-entropy) | 10.9 | 108 | 0.891 |
| MDLM, token temperature 0.5 | 7.2 | 70 | 0.928 |
| MDLM, greedy tokens | 0.0 | 0 | 1.000 |
| AlphaDLM, | 67.4 | 201 | 0.806 |
| AlphaDLM, | 88.0 | 143 | 0.851 |
| AlphaDLM, | 98.6 | 52 | 0.923 |
| AlphaDLM, | 100 | 1 | 0.995 |
In one step, changing the loss raises validity from 10.9% to 88.0% at , with 143 different valid answers expected in 1,000 generations. Neither predictor reproduces the data distribution. Sharpening the tokens of MDLM does not help: greedy selection returns AAAAAAAA, which is invalid. For and , AlphaDLM is also closer to the data in total variation.
Several steps, trained denoisers. We train two neural denoisers on sequences from the same distribution above, masking tokens and predicting the missing ones. Both use the same architecture and training setup: MDLM uses cross-entropy, and AlphaDLM uses sequence-level alpha loss with for masked tokens.
Generation starts with all eight positions masked and takes 1, 2, 4, or 8 calls. "MDLM, exact" uses the true conditional token probabilities at each call and requires no training. With one call, trained AlphaDLM reaches 92.1% validity. All eight positions are masked, so its loss uses , as in the one-step calculation above.
| Valid (%) / calls | 1 | 2 | 4 | 8 |
|---|---|---|---|---|
| MDLM, exact | 10.9 | 16.7 | 34.1 | 100 |
| MDLM, trained | 10.8 | 15.5 | 30.4 | 95.5 |
| AlphaDLM, trained () | 92.1 | 96.0 | 98.2 | 99.8 |
The exact MDLM recovers the data distribution with eight calls. AlphaDLM already produces mostly valid outputs with one call, but its outputs become less diverse with more calls. With eight calls, MDLM is the better choice: it is valid and recovers the data distribution. The loss is useful when few calls are available and a valid output matters more than covering all valid outputs.
Concluding remarks. We would be grateful to know whether our answers address your concerns. If so, we kindly ask you to consider increasing your rating. We are happy to discuss any remaining questions.
Thanks for the thorough rebuttal. I've increased my rating.
Rebuttal 1
W.1
We agree: AlphaDLM does not learn the joint data distribution q, and we do not claim that it does. But neither does a perfectly trained MDLM once several tokens are revealed in one step. Let q be the data distribution of the tokens revealed together and S its support (the valid completions). For a distribution π, let pπ=∏iπi be the product of its marginals πi: the same token frequencies, but independent tokens. The total correlation TC(π)=KL(π∥pπ) measures the dependence between tokens under π and is zero exactly when they are independent.
MDLM. The cross-entropy optimum is pq, so its divergence from the data is KL(q∥pq)=TC(q). It equals q only if the tokens are independent, and it generates invalid outputs whenever S is not a Cartesian product.
AlphaDLM. By the characterization in Appendix A (Eq. 16), for 0<α<1 the optimal factorized predictor is pα∗=pπ∗, where
π∗∈argminπ KL(π∥q)+αH(pπ),
with π supported on S and H the entropy. The loss keeps the data but charges α for the entropy of the generated outputs. Two consequences follow (new Proposition 2 in Section 3.2, proved in Appendix G).
The entropy that bounds invalid outputs can only decrease as α grows. In the experiment of Q.1 the first bound is close to the measured value (0.82 against 0.88 at α=0.5).
So α trades diversity for validity, and k selects the point. Without factorization the optimum is the tilted distribution q1/(1−α) that the reviewer mentions (Eq. 8, due to [1]); the display above is its factorized counterpart (equivalent to Eq. 9, following [2]). We do not try to recover q in few steps, which no factorized model can do. We aim at valid outputs in few steps by changing only the loss, and the revised Discussion states the price in diversity.
W.2
The shared token is not an obstacle. The loss maximizes ∑yq(y)∏ipi(yi)α. Since every valid completion has
Smithin the middle, its factor is common to all terms: the objective equals p2(Smith)α times the same objective for the first and last positions. Every optimum therefore predictsSmithwith probability one, and the other two positions are solved as if the middle one were absent. Proposition 1 applies to them, with the two groups (John,Wang) and (Julia,Patel), which share no token at either position. If the names have probabilities 0.65 and 0.35, cross-entropy assigns 2⋅0.65⋅0.35=45.5 to the mixed names. For α>1/2 the optimum of our loss is one complete name, soJohn Smith Patelhas probability zero. It cannot keep both names: any factorized distribution that does so must also produce the mixed names.We agree that general overlapping supports are outside Proposition 1, and this is a real limitation. For q(AA)=0.4 and q(AB)=q(BA)=0.3, the optimum at α=0.5 still assigns 4.96% to the invalid
BB, compared with 9% for cross-entropy (Appendix A.1). In this general case exact validity is not guaranteed, but the new Proposition 2 holds for any support and bounds the probability of invalid outputs. Our answer to Q.1 measures it on a distribution with an overlapping support.References
[1] Sypherd et al. A Tunable Loss Function for Robust Classification. 2022.
[2] Lapidoth, Pfister. Two Measures of Dependence. Entropy, 2019.