- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
GFlowNets are generative models that learn to sample from a distribution , where , over a discrete space . They do so by learning a flow on a directed acyclic graph with a source node supplying , terminal nodes absorbing , and intermediate nodes satisfying flow conservation. Different ways of parameterizing and learning such flow give rise to three methods: Flow Matching (FM), Detailed Balance (DB), and Trajectory Balance (TB). The paper provides a unified theoretical analysis of the training of GFlowNets under these three paramterizations. In particular, it establishes three main results:
-
Result 1 (Thm 3.2): A small expected loss implies that the learned distribution is close to the target distribution, provided the training samples sufficiently cover the relevant parts of the graph.
-
Result 2 (Thm 4.6): A small gradient of the expected loss also implies that the learned distribution is close to the target, provided the model can adequately correct its balance errors through gradient parameter updates and is sufficiently well-conditioned.
-
Result 3 (Theorems 5.4, 5.9, and 5.12): Under additional assumptions, SGD is shown to converge toward a small expected loss at a rate they quantify.
Critical strengths
- The paper provides a unified theoretical framework for understanding the training of GFlowNets across FM, DB, and TB, connecting balance loss, stationarity, and SGD convergence to sampling accuracy.
Critical weaknesses
- The theoretical guarantees rely on several strong assumptions, e.g. bounded sampling coverage ( in the paper), a small parameterization residual ( in the paper), and good conditioning of the effective Fisher matrix ( in the paper). The paper does not establish whether these assumptions hold for general neural-network GFlowNets.
Questions for authors
- Could the authors comment on how reasonable the assumptions on sampling coverage, the parameterization residual, and effective Fisher conditioning are for practical neural-network GFlowNets? In particular, can these assumptions be theoretically justified for non-tabular models, or did the authors observe that they are empirically verified during training?
3 comments
Thank you for your response. I will maintain my score, as I value the theoretical unification the paper proposes.
That said, the experiments do not fully resolve my concern about whether the assumptions hold for practical neural GFlowNets. The experiments use a single graph with 25 states, whereas the applications listed in your introduction involve much larger state spaces. Moreover, your appendix states that “Each state has a trainable 16-dimensional embedding.” It is unclear whether the observed behavior extends to models that compute representations from state features over such large state spaces, where maintaining a separate trainable embedding for every possible state may be impractical...
Thank you for clarifying this concern. In the revision, we replaced the per-state trainable embedding with a non-tabular model that computes representations from fixed 8-dimensional random state features using a two-layer message-passing GNN. All GNN and edge-scoring parameters are shared across states, so the number of trainable parameters is independent of the state-space size. We reran all TB, DB, and FM experiments with this model, and observed the same qualitative loss–discrepancy and gradient–loss relationships. This addresses the specific concern that the previous evidence depended on maintaining a separate trainable embedding for every state, and also hints the behaviors for larger state spaces. We have updated the experimental description, figures, and appendix accordingly.
Responses to Reviewer eQ78 (1/1)
Thank you for your time and effort in reviewing our paper! We are grateful for your constructive suggestions, which have significantly guided our improvements. Before addressing your specific questions, we would like to highlight our major revisions:
We have improved the loss-to-distribution bound to a linear bound given by KL(pF⊤∣∣pR)+KL(pR∣∣pF⊤)≤2κ2∥ν/μ∥∞LO(θ) through carefully analyzing the occupancy mass cν to demonstrate that cν∼KL. Consequently, the stationary-to-optimality bound and the finite-time convergence rate can be enhanced accordingly.
We have added a new section for numerical experiments to verify the theories discussed in Sections 3 and 4. We trained TB, DB, and FM GFlowNets on a small DAG and evaluated the expected loss, gradient norm, and terminal symmetric KL during training. By plotting the data points on logarithmic axes, we observed that the symmetric KL-loss points align along a line with a slope of approximately 1, whereas the loss-gradient norm points align along a line with a slope of approximately 2. These results are consistent with Theorem 3.4 and Lemma 4.4, respectively, and together illustrate Theorem 4.5.
Below, we briefly respond to your comments and questions.
Sampling Coverage
On a finite object space, if μt=(1−α)pt+αq,α>0,qmin:=minoq(o)>0, then ∥ν/μ∥∞≤1/(αqmin) whenever the discrepancy is nonzero.
For TB with exponentially many trajectories, the coverage factor Cν=∥ν/μ∥∞ can be sharpened by restricting the comparison to the function space containing the TB imbalances, rather than requiring coverage of individual trajectories. To be precise, define ϕ(τ):=(1,(1{e∈τ})e∈E,(1{sn(τ)=x})x∈X)⊤,HTB:=span{ϕj(⋅)}j, and for any distribution ρ, denote the covariance matirx as Σρ:=Eρ[(ϕ−Eρϕ)(ϕ−Eρϕ)⊤], the coverage factor is CTB(ν,μ):=supf∈HTBVarμ(f)Varν(f)=λmax(Σμ†/2ΣνΣμ†/2), which can be controlled to the order of dim(HTB) under careful design.
We have included the this improved coverage factor and corresponding discussions in Appendix B.1 of our revision.
Parameterization Geometry
The assumptions regarding bounded residuals and the eigenvalues of the Fisher matrix are commonly referenced in the theoretical analysis of the natural policy gradient and natural actor-critic methods in reinforcement learning (e.g., [1][2][3]), where the residuals parellels the compatiable function approximation error. We acknowledge that establishing a uniform bound for ϵμ and λ throughout the training process is difficult to justify theoretically. However, numerical experiments indicate that PL condition, which directly arises from these assumptions, is well-preserved during training. This provides empirical evidence for the validity of these assumptions.
We hope our response addresses your concerns. If so, we wonder if you could kindly consider raising your score? We will also be happy to answer any further questions you may have. Thank you very much!
[1] Yanli Liu, Kaiqing Zhang, Tamer Basar, and Wotao Yin. An improved analysis of (variance-reduced) policy gradient and natural policy gradient methods. Neurips 2020.
[2] Yuhao Ding, Junzi Zhang, and Javad Lavaei. On the global optimum convergence of momentum-based policy gradient. AISTATS 2022.
[3] Ganesh, Swetha, Jiayu Chen, Washim Uddin Mondal, and Vaneet Aggarwal. Order-optimal global convergence for actor-critic with general policy and neural critic parametrization. UAI 2025.