acceptodds
Under review as a conference paper at ICLR 2027

Rethinking GFlowNet Training: Negative Reinforce Dynamics

10 views

Abstract

Generative Flow Networks (GFlowNets) are models that learn to sample discrete objects proportional to a target reward. They are trained by minimizing balance errors over sampled trajectories, which inherently ties correctness to exploration: if a region is not visited, balance cannot be enforced there. Adaptive exploration is therefore crucial, yet standard training relies on a semi-gradient update that treats the sampling distribution as fixed at each optimization step. On the other hand, the full gradient reveals a second mechanism that reduces the observed errors in an anti-exploratory manner. In this paper, we establish a connection between the full gradient and min-min dynamics between the GFlowNet and a decoupled sampler, showing that the sampler reduces observed violations by avoiding high-error regions. This insight motivates a bridge between two-player games and single-policy dynamics, which underpins the design of our proposed method, Negative Reinforce Dynamics (NRD). By tying the parameters of an adversarial game together, we recover a single-policy dynamic that extends exploration while maintaining correctness. In standard benchmarks, our method outperforms existing approaches in both convergence speed and exploration efficiency, without training a separate teacher or auxiliary exploration policy.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
GFlowNet, amortized sampler

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_yJuHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on the gradient used to train GFlowNets. Differentiating an expected trajectory-balance loss through the sampling distribution introduces a score-function term that can reduce the observed loss by avoiding trajectories with large errors. The authotrs considers Negative Reinforce Dynamics (NRD) that adds a score correction only for trajectories with negative balance residuals, encouraging exploration of underrepresented trajectories without a separate teacher policy. The paper establishes fixed-point and contraction properties in restricted population-dynamics settings, and evaluates distribution matching, structured-object search, and peptide generation. Experiments cover small grid distributions, structured objects, and peptide generation, with particularly favorable grid and peptide-oracle results and mixed outcomes on other structured tasks.

Critical strengths
  1. Pretty well-demonstrated motivational example and example demonstration in Figure 1.
  2. This work gives a useful and reasonably distinctive connection between loss optimization and exploration.
Critical weaknesses
  1. The main distribution-matching evidence uses an approximate metric where exact validation seems feasible (Section 6). Terminal probabilities on the small grids are estimated using backward trajectory samples and normalized before computing TV. Exact marginal propagation on these finite graphs, or a convincing estimator-error study, would strengthen the most direct evidence that NRD improves the target distribution rather than the proxy.
Questions for authors
  1. Can exact terminal probabilities be computed for the reported grid environments, and is the method ranking stable to replacing the estimated TV metric with those probabilities? If an existing estimator-validation result is available, please identify it.
Reviewer_ZPxoOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Negative Reinforce Dynamics (NRD) -- an adaptive exploration method for training GFlowNets. It augments the standard trajectory balance (TB) loss with an additional gradient term that pushes the current policy towards sampling trajectories that are under-allocated, i.e., where the mass assigned by the forward policy is smaller than the one assigned by the reward and the backward policy. Authors theoretically analyse the proposed training procedure under a specific continuous-time dynamics setup, showing that it has the same equilibrium as the standard TB training, as well as converges at least as fast as TB, or faster under some conditions. Experiments demonstrate improved convergence speed in comparison to the on-policy variation of the trajectory balance loss, as well as in comparison to various prior adaptive exploration methods that train a second exploratory sampling policy.

Critical strengths

The paper is generally well-written and easy to follow. The idea for the method is presented in a very nice way, and, in my opinion, is especially interesting since adding a gradient term aimed at increasing the loss can be very counterintuitive at first, but everything falls together well in the end. Removing the need for training an additional exploratory policy that appeared in prior works while still carrying the same exploration benefits is genuinely an impressive strength, especially taking in the account how simple to implement the final method is.

The empirical results presented in the main text are impressive as well, especially the improved converegnce speed in comparison to the methods that require learning the second exploratory policy (though I would suggest keeping the standard TB as a baseline on each plot to better understand the whole picture).

I did not check theoretical proofs in details, however, the presented theoretical results are a sound motivation on why the method should converge to the same optimum, introducing no bias, and offer convergence speed benefits in comparison to TB.

Critical weaknesses

I have a number of concerns regarding the experimental evaluation.

First, from my understanding, experiments on Figure 6 and Figure 7 are on extremely small toy tasks (31x31 or 15x15 grids), so one should expect the GFlowNet to eventually converge towards the optimal perfect sampler. However, one can see that different training curves do not converge towards the same TV metric. This is puzzling to me. Is it happening due to very short training runs? If so, it would be more informative to increase the number of training iterations and verify that all methods converge to the optimum, while the proposed one converges faster than others. In addition, it is not even clear whether NRD converges to the optimum: the TV value of 0.25 on 8 Gaussians is rather large. I do understand that, as written in Appendix, the TV is evaluated using MC estimates of GFlowNet sampling probabilities. Then, to obtain a reference metric value to track converence to the optimum, one could compute the optimal forward policy using dynamic programming (since the environment is very small), and check what TV value it will produce. I would also suggest running the same experiment on the classic hypergrid benchmark from GFlowNet literature of larger size, like 20x20x20x20, and verify that the models converge to the optimum using the protocol from prior literature (see, e.g., [1]).

The results for some baselines on toy environments also look strange, e.g., all adaptive exploration baselines from previous literature performs worse than the plain on-policy TB on Moons and Adaptive Teacher performs worse on GridWorld. Do the authors have an explanation for this?

Second, all experiments are performed with trajectory balance loss (and its variations as baselines). However, other GFlowNet training losses have been proposed in the literature. Subtrajectory balance (SubTB) [1] is a well-established one that has shown better performance in comparison to TB across a veriety of domains and benchmarks. To better quantify the empirical efficiency of NRD, I would suggest adding it as a baseline. In addition, I would welcome a discussion on whether a similar gradient term can be added to SubTB loss to guide exploration, further improving the empirical performance.

Third, the main text presents the experiments on very small synthetic environments, plus one biological task that is still rather small-scale compared to real-world domains (191019^{10} candidates), so other larger environments would be good to back up the performance of the method. I do note that Appendix contains additional experiments on other environments (Table 2). However, it only compares the methods using average reward of the final model as a metric, and many methods obtain very similar results across it. Judging from the main text results, I would expect NRD to improve upon the convergence speed and/or final distribution approximation quality in comparison to the baselines. I would suggest presenting training convergence plots for these environments too, as well as tracking some distribution aprroximation metric, e.g. reward correlation presented in [2] or flow consistency in sub-graphs (FCS) presented in [3].

Finally, it is stated in Appendix that the backward policy is learned in some experiments, however, I did not find a specific discussion on how exactly it is learned. Is it simply updated through the TB loss, while the forward policy is updated through TB loss plus NRD gradient component?

The paper presents a very interesting method that is simple yet potentially strong, however, my concerns prevent me from recommending acceptance for the current manuscript. If they are sufficiently addressed, I am willing to increase my score.

References:
[1] Madan et al. Learning GFlowNets from partial episodes for improved convergence and stability. ICML 2023.
[2] Whittamer et al. Trajectory balance: Improved credit assignment in GFlowNets. NeurIPS 2022.
[3] Silva et al. When do GFlowNets learn the right distribution? ICLR 2025.

Questions for authors

Main questions that can affect my assessment are given in detail in Weaknesses.

I also have an additional question. If one looks at min-max formulation in Equation (9), the inner maximization problem should in theory collapse to the model that always samples one single trajectory that has the highest base-model loss value, since it is in essence a policy gradient procedure that uses the loss as the reward to maximize. If we restrict the loss only to under-allocated trajectories, it will still collapse to the single under-allocated trajectory with the highest loss. However, when applying the gradient updates to the same model, as done in NRD, we obtain a theoretically sound procedure that has the same optimum as trajectory balance optimization. Can the authors give any intuition on why this happens? And what if the inner maximization problem included not only the loss, but also some regularization term, like entropy of the sampling policy QϕQ_\phi to avoid the collapse, akin to the existing theoretical connections between entropy-regularized RL and GFlowNet samplers [4]?

References:
[4] Tiapkin et al. Generative Flow Networks as Entropy-Regularized RL. AISTATS 2024.

Reviewer_6rsFOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper revisits trajectory-balance (TB) GFlowNets training through the score-function decomposition of the expected-loss gradient. It proposes Negative Reinforce Dynamics (NRD), which adds a sign-selective score correction to the standard TB update to encourage exploration while preserving the target sampling distribution. The paper then analyzes the resulting dynamics in simplified settings and evaluates the method on distribution-learning and structured-generation tasks.

Critical strengths
  1. Motivated by the insights from the score-function identity, the NRD is a reasonable approach for improving GFlowNets exploration. The theoretical analysis provides useful support, including on-policy convergence in the free-trajectory setting and hitting-time improvements for one-dimensional exponential families.
  2. The numerical experiments cover both distributional learning with illustrative gradient-field and more practical high-reward discovery tasks.
  3. The paper is generally clearly written and easy to follow.
Critical weaknesses
  1. Many theoretical statements appear to be flawed under the currently stated assumptions:
  • In Thoerem 3, the Epη(δη)=0\mathbb{E}{p_\eta}(\delta_\eta)=0 at every time, equivalently log⁡(Z/Z⋆)=−DKL(pη∣pη⋆)\log(Z/Z^\star)=-D_{\mathrm{KL}}(p_\eta|p_{\eta^\star}). This condition is not generally maintained by jointly updating the normalizer with a finite learning rate.
  • In Corollary 1, D˙NRD≤D˙TB<0\dot D_{\mathrm{NRD}}\leq\dot D_{\mathrm{TB}}<0 holds at every interior state “away from balance” is too strong. If p=p⋆p=p^\star but Z≠Z⋆Z\neq Z^\star, trajectory balance is violated, yet D=0D=0 and both discrepancy derivatives are zero.
  • A merely differentiable parameterization does not guarantee the existence of the vector-field Jacobians asserted in Eq. (15) since a differentiable function can have a discontinuous derivative. Suitable local regularity, such as continuously differentiable residuals and sampling probabilities on a fixed finite support, needs to be there.
  • Full support sampler QQ and positive rewards RR do not rule out the case where the backward policy assigns zero probability to some trajectory, causing pi⋆=R(xi)PB(τi∣xi)/Z⋆=0p_i^\star=R(x_i)P_B(\tau_i\mid x_i)/Z^\star=0. The full-support policy assumption needs to be there.
  • The "if and only if" statement in the introduction is too strong. A forward policy can produce the correct terminal marginal while allocating probability across trajectories differently from that backward policy.
  1. The baseline implementation and hyperparameter-selection procedures need more explanation. In the experiments, the vanilla TB performs competitively with, and sometimes better than, several exploration baselines on the structured tasks. This does not by itself indicate an implementation problem, but it makes the implementation and tuning procedures important for interpreting the results. I saw that Appendix A lists configurations, yet it is unclear how they were selected and whether the methods received comparable tuning effort. Under-tuned baselines would undermine the fairness of the comparisons and could warrant rejection.
Questions for authors
  1. The score-function analysis identifies a potentially undesirable mechanism in the full gradient: the expected loss can decrease by avoiding trajectories with large balance errors. However, standard TB already omits this score term. What specific limitation of standard TB does NRD address? Statements such as “fundamentally misaligned with the true goal” should distinguish the behavior of the full-gradient dynamics from the properties of the semi-gradient TB update.

  2. Can the authors address the theoretical issues above in the weaknesses, particularly the meaning of "fast normalization" in Theorem 3?

  3. How were the baseline hyperparameters selected, and did all methods receive comparable tuning budgets? Were the implementations checked against official code or reference results? Please also clarify which score correction was used for NRD in the main experiments.

  4. Do the theoretical results extend to a fixed positive coefficient multiplying the NRD score correction and how sensitive are the results to its value? This would help justify whether the strength of the additional correction term in NRD is not sufficiently examined.

  5. In Figure 2, does the illustration represent a decoupled sampler or one that shares parameters with the forward policy? The figure changes the sampling distribution while keeping the residual landscape fixed, whereas shared parameters generally cause both to change. Please clarify how this illustration relates to NRD.

  6. What additional computation does NRD require relative to TB under matched experimental settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.