- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces exponential residual flows (ERF), which augment discrete states with continuous residual variables to construct deterministic trajectories with the prescribed time marginals. It then uses these trajectories for consistency distillation into a residual-conditioned student.
Critical strengths
The deterministic construction provides an interesting basis for trajectory-based distillation in discrete diffusion. The distilled model shows promising few-step results, particularly for uniform diffusion.
Critical weaknesses
My main concern is whether the student actually learns the deterministic endpoint associated with its residual input. Changes in residuals can alter the order of token transitions and subsequent predictions, making this map potentially difficult to learn. Generation quality alone does not establish that the student reproduces this dependence.
The residual-conditioning ablation in Appendix H.10 makes this question particularly relevant. Removing residual inputs preserves similar performance in several settings and improves the reported TinyGSM MDM results. It therefore remains unclear whether the gains come from learning the deterministic coupling or primarily from soft-target self-distillation.
The smaller gains in masked diffusion also leave the practical benefit of the proposed construction less clear in that setting.
Questions for authors
-
With a fixed ERF teacher, how accurately does the student predict the endpoint associated with the same initial state and residual input?
-
Does matching the residual input to the teacher trajectory improve endpoint prediction compared with shuffled residual inputs?
-
How do the authors interpret the settings where removing residual conditioning maintains or improves performance?
3 comments
Thanks for rebuttal.
Yes, actually I already mentioned strong stochastic flowmap at section that is only visible to AC.
Actually I personally don't believe strong stochastic flowmap can really learn SDE map as well, since it is also be extremely hard to be trained, so again, lipshitz non-smooth. but since CTMC is much more chaotic compared to SDE, since we can't assume any local-smoothness of score function in CTMC compared to continuous space SDE.
I think binary MNIST example partially resolve my concern, especially large t regime. But still learnability of CTMC is exponential w.r.t. vocab size ^ sequence length, so considering simplicity of MNIST, not really sure I should believe this example as concrete evidence. Maybe this is my personal bias, but want to hear author's opinions on this.
To sum up, I want to see more evidence that (1) your model really learned deterministic mapping or (2) what mechanism that student model is using, if they are not learned exact mapping. Current score is kind of 2.5, but slightly lean to rejection.
We understand your point of view and agree that it is not obvious that a network could learn the ideal map because of the smoothness and generalization issue. However, we want to emphasize that even if the method is not able to recover this map, it is still learning some dependance on the residuals and not ignoring them as shown in the new experiment summed up in the additional figure that we put in Appendix H.11. Therefore, this mapping can be used as an approximation to a stochastic generator for .
Furthermore, just to clarify, we refer to Stochastic Flow Maps as relevant prior work illustrating the challenges associated with learning the flow of the ODE considered in our paper, but our framework is not exactly similar to theirs. Indeed, a genuine analogue of Stochastic Flow Maps in our setting would require learning a map that depends jointly on the initial condition and the realizations of Poisson processes, which is considerably more challenging than learning a map involving a finite number of exponential random variables.
In addition, we conducted extensive additional experiments on Fashion-MNIST, which presents a substantially more challenging setting than MNIST (with for Fashion-MNIST, compared to for MNIST). As shown in the table below, explicitly providing the exponential random variables as inputs significantly improves performance. Interestingly, we found that using Gumbel random variables instead of exponentials, obtained by simply applying a logarithmic transformation to the inputs, further improves the model's ability to learn the flow map. A similar observation was reported by Zhang et al. in Gumbel Distillation. Fashion-FD ↓
| Input | NFE 1 | NFE 2 | NFE 4 | NFE 8 | NFE 16 |
|---|---|---|---|---|---|
| Gumbel | 155.21 ± 4.65 | 119.06 ± 7.46 | 97.33 ± 7.97 | 96.10 ± 8.49 | 126.96 ± 12.50 |
| No noise input | 365.70 ± 4.54 | 294.98 ± 4.31 | 223.80 ± 5.31 | 162.38 ± 5.12 | 170.58 ± 5.62 |
KID ×1000 ↓
| Input | NFE 1 | NFE 2 | NFE 4 | NFE 8 | NFE 16 |
|---|---|---|---|---|---|
| Gumbel | 463,677 ± 19,294 | 341,718 ± 24,205 | 256,276 ± 20,608 | 241,563 ± 23,912 | 325,890 ± 45,642 |
| No noise input | 946,048 ± 5,792 | 766,118 ± 11,491 | 579,445 ± 12,474 | 399,909 ± 14,857 | 425,965 ± 17,005 |
It is worth noting that all the other distillation methods we tested (i.e., all baselines except ours) either collapsed or failed to produce satisfactory results on this dataset. In particular, despite extensive hyperparameter tuning, we were unable to obtain convincing results with these methods using only a few sampling steps. We find our latest results encouraging and believe that further methodological and algorithmic developments could make learning the flow map underlying ERC a viable and effective approach.
Similarly, we believe that our work opens the door to new methodological and algorithmic developments for discrete generative models. As was the case with early work on probability flow ODEs, we expect that further theoretical and practical investigations will be valuable in fully exploring the potential of this approach. Making it work better at scale is still an open problem but we hope this paper inspires some new works in the community that can figure out better ways.
We thank the reviewer for these interesting questions and reviewing our paper carefully.
We agree that the actual learning of the theoretical map is not automatic and we do not claim that we learn the ideal mapping. However, we see two advantages to this formulation:
It is true that the map is a priori hard to learn but we do not believe it is impossible. It might be useful to notice that conceptually, this is very close to Strong Stochastic Flow Maps (McCallum et al) who learn a similar dependency in the continuous setting.
We ran an additional experiment on MNIST that shows that the network is able to approximately match the residual.
Pixel agreement (%) between one-step ERC predictions (argmax) and the 128-step teacher on binary MNIST.
Results average over 16 held-out images and four exponential-clock draws per image and starting time. Shuffling changes E while holding X fixed.
The table, also shows that this agreement is higher when matching the input residual with the one the teacher starts with.
We want to emphasize that it is only possible to test this exact feature in the one-step setting, which explains why the results are not perfect. Indeed, when doing multiple steps with the student, we refresh the exponentials at each step so the final output does not correspond to the teacher trajectory started at that residual.
As mentioned by the reviewer, learning the map requires substantial computation inside the network. We hypothesize that it is sometimes easier (although suboptimal) for the network to just ignore it which may explain those results. However, by using the residuals it can theoretically reach better performance (and learn a useful object), hence the motivation of our method. The updated Appendix H.11 in the updated paper shows that changing the residuals while holding the tokens and noise time fixed changes ERC’s predictions. The student therefore learns a mapping that depends on the residuals. In principle, this gives it access to information about the endpoint of the teacher’s specific trajectory that is unavailable from the tokens and noise time alone.
An open question is how to process the exponential residuals so that the student more faithfully learns how the teacher’s endpoint depends on them. This could involve better flow-map losses that supervise the student along the teacher’s trajectories, or architectural changes that give the network a more effective way to use the residuals.