- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper develops a flow–score formulation connecting GFlowNets with discrete diffusion and transfers score-entropy objectives to data-based GFlowNet training. A transition-sampled variant avoids the need for analytic backward visitation probabilities. This enables pre-training from terminal samples without an explicit reward model, followed by reward-based post-training of the same model. Experiments evaluate the full pipeline on peptide generation and data-only modeling on Ising configurations and binarized MNIST.
Critical strengths
-
The flow–score formulation provides a useful connection between data-based and reward-based training. The transition-sampled objective requires only local backward transition probabilities, allowing pre-training without analytic visitation probabilities and subsequent reward-based training of the same model.
-
The peptide study uses identical Replay-TB post-training and ten seeds. Learned- and oracle fixed- baselines help assess reward-scale estimation.
Critical weaknesses
-
Score entropy [1] and reward-free GFlowNet pre-training [2] are established, so the methodological advance relative to these foundations could be explained more explicitly. Fitting a terminal data distribution differs from the outcome-conditioned pre-training of [2], and the transition-sampled formulation appears useful. However, the paper could distinguish more clearly which aspects follow from the score-entropy framework of [1] and which require new derivations or enable capabilities unavailable in earlier approaches. A comparison of assumptions, objectives, and adaptation procedures would make the contribution easier to assess.
-
A matched behavior-cloning baseline on backward-sampled trajectories is missing. The offline-TB comparisons are informative, but they do not isolate whether the gains come from the proposed score-based objective or from obtaining a useful initialization through data-based training. Using the same architecture, pre-training data, backward trajectories, and post-training budget would help answer this question. Validation also remains limited to peptides of length at most ten, energy and magnetization statistics for Ising models, and qualitative MNIST samples, leaving the broader empirical applicability less clear.
[1] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. In International Conference on Machine Learning, 2024.
[2] Ling Pan, Moksh Jain, Kanika Madan, and Yoshua Bengio. Pre-Training and Fine-Tuning Generative Flow Networks. In International Conference on Learning Representations, 2024.
Questions for authors
See weaknesses.
We thank the reviewer for the constructive feedback and for pointing us to related work. We address below the novelty of our contribution relative to SEDD and Pan et al., and clarify its relationship to behavior cloning.
1. Novelty relative to SEDD and Pan et al.
We build on score entropy from Lou et al. rather than claiming score entropy itself as a new principle. Our contribution is its mapping to GFlowNet flow ratios and construction DAGs, including objectives that can be evaluated using backward trajectories.
We thank the reviewer for pointing to the work on pre-training GFlowNets; however, we want to clarify that what is considered pretraining in our setting and theirs is significantly different. Pan et al. address outcome-conditioned pretraining, designed to support adaptation to potentially many downstream reward functions. Their method involves additional outcome-conditioning and marginalization machinery.
Our method directly fits a given data distribution, and subsequently continues training the same model with conventional reward-based objectives. This approach requires neither outcome-conditioned pretraining, nor an auxiliary predictor to an intractable adaptation quantity used in Pan et al.
The distinction concerns what is learned during pretraining and how the resulting model is adapted downstream.
2. Relationship to behaviour cloning We thank the reviewer for their recommendation and add the requested baseline. We agree that our approach belongs to the broad class of imitation-learning methods. It is not intended to establish that imitation learning itself is novel. The contribution concerns the specific score-based formulation and its connection to diffusion.
In the unrestricted tabular setting, score-entropy learning and maximum-likelihood training of the forward policy recover the same population-optimal forward policy under matched supervision. We have formalised this relationship in Appendix D.
Our analytic DSE objective marginalises over next-action uncertainty and is therefore a Rao–Blackwellization of its sampled counterpart. The same marginalization principle can also be applied to policy behavior cloning (BC). Our new experiments show that this marginalization is practically beneficial for BC, while analytic BC and analytic DSE achieve similar downstream performance in the evaluated settings.
The objective-specific contribution is therefore the score-based formulation, its theoretical connection to diffusion, and the associated estimators, not a claim of universal empirical superiority over maximum-likelihood training.