acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Cooperative GFlowNets

38 views

Abstract

Generative Flow Networks (GFlowNets) are trained to sample objects from compositional spaces with probability proportional to a reward function, but ineffective exploration can cause learned samplers to concentrate on a few high-reward regions. An intuitive remedy is to train multiple GFlowNets independently in parallel and combine their samples. However, this does not prevent multiple models from collapsing onto the same high-reward areas, leaving large regions of the state space unexplored. With this in mind, we propose Hierarchical Cooperative GFlowNets (HCG), an approach that dynamically partitions the search space to coordinate discovery without relying on hand-crafted boundaries. HCG imposes an ordering in which the highest-ranked model may explore the entire space, while lower-ranked models are progressively pushed toward regions not covered by higher-ranked ones. We provide a theoretical characterization of how specific learning objectives can shift probability mass away from designated areas, instantiating this mechanism within our architecture. A shared classifier coordinates the hierarchy by partitioning the search space, keeping the GFlowNets independently parameterized and enabling asynchronous training. Furthermore, we derive a normalizer-weighted aggregation rule for the ensemble and prove that it recovers the target reward-proportional distribution under the stated assumptions. Finally, experiments across multimodal distributions, sequence generation, combinatorial optimization, and biological sequence design show that HCG improves mode coverage and target approximation, translating increases in the number of coordinated samplers into broader, complementary exploration.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
GFlowNet, sampling, amortized sampler

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ktiaOpenReview ↗
Rating
4(+3)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In order to improve exploration, and therefore sampling, in GFlowNets, the authors propose an ensembling scheme whereby a state space is partitioned into regions where separate GFlowNets learn to sample from that region's reward function. To achieve this sampling method, the authors propose a theoretical framework where the state space can be separated by hierarchically allocating regions of the space. This means that each GFlowNet will be given (or rather will take) a subset of the state space, necessarily disjoint from those partitions of higher-rank GFlowNets. Learning using the trajectory balance loss in such a setting is difficult since a non-zero reward function would ordinarily be required over the entire space, thereby allocating probability mass everywhere. The authors alleviate this difficulty by proposing a new loss function that allows for learning the TB objective in each GFlowNet's subspace, but allowing for zero-probability forward policies in the rest of the space. In order to actually learn these partitioned networks, however, a classifier is needed to specify whose subspace a given final state belongs to. This classifier is trained using a cross-entropy loss. The proposed algorithm is well-supported, and is strikingly similar to Q-learning algorithms with a delayed critic update, but GFlowNet-style. Overall, great idea and great results to support the idea! :)

Critical strengths

This paper shines in the experiments section, where in four tasks, hierarchical cooperative GFlowNets are shown to mostly outperform or otherwise match standard baselines and exploration-based baselines! The theoretical results are impressive because they show such large gaps in performance between the methods, clearly demonstrating the HCG is effective at exploration, and learning to sample.

Critical weaknesses

Firstly, I feel unfortunately compelled to reject this paper because of the lack of a related works section, and a lack of citations throughout the text. This made it a lot harder to compare this work with prior work: it took me a substantial amount of effort to find related works, to place this work into context. Additionally, a general lack of comparison with prior GFlowNet-related literature, including work in regularised RL like trajectory general mellowmax, and searching after sampling like local search GFlowNets made the results less interesting.

Secondly, I have very strong concerns with Section 3.1, specifically in the definitions of the supports SiS_i, and the ownership sets OiO_i. In particular, I do not believe that for a reasonable definition of "support," that anything other than Si=XS_i = \mathcal{X} is possible for any of the GFlowNets (because remember, we are not yet dealing with HCGs). This yields a substantial simplification: since GFlowNets are necessarily full-support over the state space, all the GFlowNets GiG_i must also be full support, thus the ownership set is empty for all i>0, collapsing to a single active GFlowNet over the state space. This is seemingly a useless formalism. However, I fully understand why you have defined the sets in this way, since in Section 4 you will replace the notion of "support" with one of "empirical support" (which is still undefined mind you), which will instantly fix this issue. However, my concern with this "empirical support" fix is that the regions are inevitably going to be "swiss cheese," with many gaps, individual samples of one GFlowNet in a sea of another. If there is a simple fix, I could imagine it being with the hull of the sampled points defining the observed regions, but I didn't put much thought into it. In any case, I understand that the authors were going for a theoretical justification of the method, I appreciate it, I think its cool, just please attempt a fix (and if I'm wrong, then please correct me, and add these details to the paper so that others do not make the same mistake).

Do not despair, I really want to see this paper published! Address these two points (and my concerns in the questions below) and I will be more than happy to increase the score!

Questions for authors

First, some basics that can be cleared up somewhat quickly:

  • Classic ensembling question incoming: how does a K-times larger GFlowNet perform, with the same parameters as the ensemble, in the tasks? How do different network sizes affect performance (give me a nice performance curve).
  • On the topic of parameters of the neural networks, no details are given in the paper or the appendix about the networks, please provide said details. Which architecture, what sorts of hyperparameters, any tricks used to get it to work, optimisers, learning rates, temperatures, etc.?
  • How do the exploration methods you list and the methods I stated in the weaknesses compare in terms of performance?

Second, some more involved questions:

  • You introduce a new hyperparameter, K being the number of GFNs to train with. Given that you don’t know the number of modes ahead of time, what is the effect of specifying too few modes in a many-mode setting, and conversely, the effect of specifying too many modes in a few-mode setting? For instance, what happens when K=100 when there are only 3 Gaussians, and conversely?
  • Is Figure 1 an actual GFN example or is it just hypothetical? If it’s the latter, please state this explicitly, if the former, please provide experiment details somewhere for reproducibility. Further, explicitly demonstrating the coverage problem is not done experimentally, and there is no citation to work that explicitly states this coverage problem, thus I am left doubting whether the issue is real, despite the success of your later experiments. In total, I want an explanation of why the method works, supporting your explanation for why it works and was needed, with experiments. Further, I think the figure is deeply flawed, and will lead to misunderstandings of how GFNs actually work. I would further state that two GFNs allocating their entire distributions on the separate modes makes no sense, since they should still sample proportionally to the NON-ZERO rewards between and outside of the rings, so some coverage must be placed here as well. The figure seems misleading. If you could produce a similar figure to Figure 1 with actually learned GFlowNet samples versus the true distribution (like in the 8-Gaussians) this might be more convincing.
  • I believe it is common in GFlowNet literature to provide existence and uniqueness results to any new formulation. I would consider adding these to the text. Existence is trivial given that Equation 9 going to zero given a mix between the GFlowNet and zero-mass policy is already basically shown. Uniqueness is also somewhat trivial given a particular backward policy. Just a thought, not necessary by any means, I also can't remember if every paper I've seen does this.
  • Figure 3: Why do you think the single GFN collapsed to a single mode? Could it be that the credit assignment at such long trajectory lengths was causing difficulties in finding other modes, so as soon as a single mode was found, it was given almost all the probability mass? I can see that this is a difficult exploration problem, but I’d like some additional context for why this is the case. Also, why not use another loss from the GFlowNet playbook? Like SubTB or something of the sort?

Third, of utmost importance:

  • In the AMP task, I have mostly seen top-K diverse rewards reported as a standard metric. I would strongly consider restating the results in this way. In particular, use the Levenshtein distance to measure uniqueness of a given mode according to a task-specific distance threshold, then take the mean of these samples' rewards. Also, I think the idea of capturing all the samples throughout training is interesting, and more closely resembles what a true scientific process would be, I don't think this is standard in the literature. Take the top-100 samples over 1000 samples generated with the final model after training. This will be much more interesting than the combined 1000s of samples from the course of training. See here for examples: https://proceedings.iclr.cc/paper_files/paper/2026/hash/7688bc1bfa9e1f0c540bca03974df304-Abstract-Conference.html.

In my mind, this last question is of the utmost importance. Your results must be placed in context of the field, meaning that you should report results in a standard manner. You may, of course, argue that this is not standard :)

Final notes on writing:

  • There are a number of anomolies in the writing. I don't understand the last 2 paragraphs of Section 3.2. While the paper is more or less well-written, some sections can be convoluted, with writing that is rather difficult to understand, even after multiple passes.
  • Finally, I don't like the "cooperative" in the name. This elicits the idea of game theory or of multi-agent systems, not an ensemble of GFlowNets defined over disjoint subsets... I would highly consider a quick swap with "coordinated" or "combinations."
Reviewer_9FDaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Hierarchical Cooperative GFlowNets (HCG), a framework for coordinating multiple GFlowNets to reduce redundant exploration of the same high-reward regions. HCG assigns priorities to independently parameterized models and uses a shared classifier to infer ownership of terminal states. A modified Trajectory Balance (TB) objective discourages lower-priority models from sampling regions already covered by higher-priority models. The resulting models are combined through a mixture weighted by their learned normalizing constants.

Critical strengths

1.The problem is well motivated, and the coordination mechanism is intuitive. Hierarchical ownership encourages complementary exploration, while the shared classifier supports parallel training without direct cross-model queries. Appendix E supports its effectiveness: under matched architectures and trajectory budgets, HCG achieves a TV distance of 0.0816±0.02430.0816\pm0.0243, versus 0.79960.7996 for an independent ensemble on Rings.

2.The aggregation rule has a clear theoretical justification. Proposition 3.1 establishes target recovery through normalizer-weighted aggregation under disjoint coverage and local calibration. The paper appropriately distinguishes this conditional result from guarantees about actual training dynamics.

Critical weaknesses

1.The distinction from Boosted GFlowNets needs clarification. Both approaches direct additional models toward regions insufficiently covered by earlier components. HCG introduces dynamic ownership and concurrent training, but their advantages over sequential residual learning are not clearly established through methodological discussion or direct comparison.

2.The main DiffGrid comparisons confound coordination with model capacity and sampling budget. HCG uses eight components with two-layer, 128-unit forward policies, whereas ACE and SA use one-layer, 64-unit policies. Appendices D and E partially address this concern through wall-clock and matched-ensemble comparisons, but a controlled comparison at the main experimental scale is still needed to isolate the benefit of coordination.

Questions for authors

1.What are the main advantages of hierarchical ownership over Boosted GFlowNets’ residual-reward approach? Can the authors provide a comparison under matched component counts and computational budgets?

2.Does HCG retain its advantage in the main DiffGrid settings with identical component architectures and matched total trajectory budgets?

Reviewer_cagWOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes hierarchical cooperative GFlowNets as a training paradigm to improve mode coverage (exploration) of GFlowNets. More specifically, the training of a single GFlowNet can collapse such that it learns to approximate the target distribution only over a subset of its true support SS.

To adress this issue, the paper proposes to train multiple GFNs GiG_i which learn to approximate the target distribution over disjoint subsets Si⊂SS_i \subset S. A model hierarchy is proposed that induces ownerships oio_i over terminal states. GFlowNet GiG_i is only allowed to approximate the target distribution over terminal states that belong to its ownership region. Regions owned by higher ranked models are not allowed to be owned by lower ranked models and are actively unlearned. Hence, lower ranked models are actively forced to assign probability mass to under explored regions of the target distribution.

Critical strengths
  • Generative models for scientific discovery heavily rely on good mode coverage / mixing to be able to generate candidates of high diversity and fidelity. Naively training multiple GFN from different seeds might increase diversity, but can lead to the same collapse and to computational inefficiency, effectively modeling the same modes with multiple models. The proposed approach to train a hierarchy of GFN with disjoint supports is well motivated and interesting.
  • The paper proposes a novel method to unlearn specific terminal states of a GFlowNet using a modified trajectory balance (TB) criterion. This is an interesting contribution on its own, because learning a GFlowNet with some R(x)=0R(x)=0 is not trivial. Instead of modifying the terminal reward R(x)R(x), the paper proposes to add an indicator function 1{x∈D}\mathbf{1} \{ x \in D\} to the flow ratio inside the logarithm. This flips the training dynamics and forces PF=0P_F=0 if the terminal state xx is in the forbidden region DD.
Critical weaknesses

Theoretical motivation vs actual implementation

A central claim of the paper is that the proposed model hierarchy improves exploratory coverage of GFlowNets. The paper provides a strict mathematical definition of the model hierarchy based on the support sets SiS_i of the individual GFlowNets. However, although the paper statest that it is straight forward to determine SiS_i the paper only provides experimental results for the case where the model hierarchy is inferred by a learned coordinator network. The coordinator learns to predict a very different (probabilistic) assignment of the ownership regions oio_i.

It is unclear if the strict mathematical definition based on SiS_i works at all. In fact, it could have very undesirable training dynamics as ownership regions could change rapidly, requiring rapid change os learning and unlearning. Inferring the ownership regions oio_i through a trained coordinator cannot cause such rapid shifts because a GFlowNet from the model hierarchy is required to sample a specific terminal state xx more often then a lower ranked model in order to assume its ownership. Further, a trained coordinator network will introduce a lot of structure to the ownership regions (e.g. more or less simple boundaries between ownership regions, dependent on training setup and model choice). While the paper introduces the coordinator based ownership inference as a cheaper, practical approximation it is very likely that it is an actual hard requirement for good performance. Experiments to clear this up are missing (questions Q1+Q3).

Central elements of the method not well characterized

A central part of hirarchical cooperative GFlowNets is the modified TB criterion to unlearn terminal states that already learned to have a terminal reward R(x)>0R(x) > 0. The paper proposes to add an indicator function 1{x∈D}\mathbf{1} \{ x \in D\} to the flow ratio inside the logarithm. While this forces PF→0P_F \rightarrow 0 for forbidden terminal states, it also pushes PFP_F and PBP_B out of balance, i.e. while PF→0P_F \rightarrow 0, PBP_B must be as large as possible to make the flow ratio as small as possible. It is not discussed and, hence unclear if PFP_F, PBP_B and ZZ of the resulting GFlowNet still share all the properties and interpretations with a GFlowNet trained with standard TB (question Q4).

Further, the dynamics of unlearning seem to be understudied by the authors although this is a central part of the method (question Q5). The power of a GFlowNet is that terminal states with high R(x)R(x) share compositional rules across modes. Rules learned for mode A transfer to (probably unseen) mode B, yielding good exploration. Conversely, unlearning a terminal state might break a GFlowNet if not done carefully because it requires to push PF(τf)=0P_F(\tau_f)=0, where τf\tau_f is a trajectory ending in a forbidden terminal state. If another allowed trajectory τa\tau_a shares a lot of trajectory steps with τf\tau_f it is hard to unlearn without forgetting important compositional rules.

Interestingly, the proposed method seems to be able to solve this problem via its auxiliary reward r(x)r(x). Inside the forbidden region the modified TB loss yields $\mathcal{L} = \left [ \log \left (\frac{Z P_F(\tau)}{r(x) P_B(\tau|x)} + 1 \right) \right ] $. If τ\tau is an unwanted trajectory, we force the fraction inside the logarithm to 00. This requires PB(τ∣x)P_B(\tau|x) to be as large as possible, i.e. we learn to assign backward probability along the trajectories and to those actions that are required to enter the forbidden regions.

Using r(x)=const.r(x)= const. as the authors propose, causes all compositional rules along a forbidden trajectory to be unlearned equally, because we need to reduce PF(τ)P_F(\tau) across all actions along this sequence in the same way. A better choice for r(x)r(x) would be to base it on the similarity of τ\tau to allowed trajectories. High similarity means it shares a lot of composition rules required in other regions. This yields a large r(x)r(x), effectively reducing the gradient magnitude over this forbidden trajectory (question Q6).

Experiments not well designed

The provided experiments are not useful to understand the basic strengths and limitations of the proposed method. Important experimental questions to understand the dynamics are for example:

  • How much the model hirarchy and ownership regions change during training. How frequently is unlearning actually happening? Does this contribute significantly to the failure cases?
  • How good good is the proposed TB loss for unlearning without forgetting important compositional rules?
  • Is coordinator based ownership inference close to support based assignment?

In particular, the provided grid example is a degenerate setup that cannot really adress any of the important questions. It seems to be sufficient if the GFN learns to terminate correctly based on the distance to the center while assigning uniform probability for all other actions (question Q7). Hence, unlearning is not really an issue here, because we only need to reduce the probability of terminating early to escape the mode in the center, which is certainly easy. Infact, a single GFN where we simply reject sampled terminal actions during sampling from time to time might show fast coverage across both modes as well.

Presentation

Most of the paper is clearly written, with consistent notation. However, some core information is missing or comes very late:

  • The considered training scenario for the GFN is unclear for a long time. It only becomes clear in section 4.3, the paper states the online training scenario with: 1) sampling batch from GFN, 2) evaluating reward on samples, 3) updating GFN and the ownership predictor on sampled batch.
  • The support sets SiS_i of the GFNs are introduced in line 152. A definition based on flow, forward or backward probability would help for following sections.
  • Specifically, section 4.2 states that A GFlowNet GiG_i owns a terminal xx if it is the highest-ranked component whose support contains xx. Determining the ownership is straight forward through evaluation of forward and backward policy. However, the paper does not provide how and I actually have a lot of questions there: 1) with the log-flow parametrization typically used for GFNs, they do not express exact zero-probability. So Si=S ∀iS_i = S \: \forall i unless there are singularities in the flow. Are action masks assumed? Do we truncate?
  • Equation 8 introduces 1(x∈Oi\mathbf{1}( x \in O_i but there is no definition. Is it the indicator function that is 1 if x∈Oix \in O_i?.
Questions for authors
  • Q1: How similar are the ownership regions oio_i for the two cases: 1) Definition based on support SiS_i and 2) Inferred by trained coordinator. How much / rapidly do they change during training? Suited experiments could be defined based on the grid example from Fig. 1.
  • Q2: How to calculate the support set sis_i from $P_F and PBP_B? Doesn't a GFN have full support by design? Is some thresholding applied in practice to remove regions with very low probability mass?
  • Q3: How well do hirarchical cooperative GFlowNets perform with ownership definition based on SiS_i?
  • Q4: What implications does the modified TB loss have for PBP_B and ZZ? Naturally, PBP_B is increased for trajectories leading to forbidden states.
  • Q5: Can you provide experiments that show the dynamics of the unlearning. For example a well designed grid experiment with compositional rulse shared across modes. 1) Train a GFN until all modes are captured. 2) Unlearn one of the modes and report modeling performance on the remaining modes.
  • Q6: Is forgetting an issue with the proposed TB loss? Can you comment if an auxiliary reward r(x)r(x) based on trajectory similarity (as described above in limitations section) could help to guide unlearning?
  • Q7: What is the root state for the grid example from Fig. 1? Is it the center of the circles?
1 comment
Reviewer_cagWOpenReview ↗

Kind reminder

As the discussion period is already halfway through, I would appreciate any answer to my questions and concerns.

Comments on the paper

1 comment

To comment 1. This explanation makes sense to me. I appreciate the more detailed outline of how and why GFlowNets practically experience mode collapse, or systematic under-coverage of the space. This exlanation should also be added elsewhere in the paper to ensure that other readers can gain the same understanding.

Please include commentary on this in the introduction and Section 3 so that future readers can understand the problem. (not absolutely necessary, but it will be a better paper for it)

To comment 2. If generable sets are the valid formalism then please say so in the main body. I think this change is of utmost importance if your theoretical results are going to be broadly understood. The animation is great, and I think it should replace Figure 1, but that is your choice of course.

To comment 3. I am very surprised by the TGM results given that in my own experiments comparing against it, TGM seems to perform remarkably well compared to standard GFlowNets. Your comment that it produced shorter sequences and thus they appear as lower diversity is interesting as well. From my AMP experiments I recall that shorter sequences were the ones that often generated the highest rewards, often clustering around length 16. Please provide details about the sequence lenghts you observed for all tested methods. Perhaps this is a problem of the learned proxy reward model? Perhaps TGM is really bad, but I would like more validation before this claim can be supported. Although, I do believe that this is the problem 'we independently select up to 100 high-reward candidates with pairwise Levenshtein ≥3'... This is not the threshold used by literally any other paper I've read. Usng a threshold of 3 seems almost laughable. Please re-state these experiments with more standard thresholds for AMP, which you can find in some of the papers I suggested previously.

Finally, thank you for including the related works section, it will be of great use to future readers. I also don't think you need to cite and talk about TGM or local search GFN necessarily, I just wanted comparisons against prior work that seemed to perform well on these tasks.

Please address my above comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.