- Rating
- 4(+3)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In order to improve exploration, and therefore sampling, in GFlowNets, the authors propose an ensembling scheme whereby a state space is partitioned into regions where separate GFlowNets learn to sample from that region's reward function. To achieve this sampling method, the authors propose a theoretical framework where the state space can be separated by hierarchically allocating regions of the space. This means that each GFlowNet will be given (or rather will take) a subset of the state space, necessarily disjoint from those partitions of higher-rank GFlowNets. Learning using the trajectory balance loss in such a setting is difficult since a non-zero reward function would ordinarily be required over the entire space, thereby allocating probability mass everywhere. The authors alleviate this difficulty by proposing a new loss function that allows for learning the TB objective in each GFlowNet's subspace, but allowing for zero-probability forward policies in the rest of the space. In order to actually learn these partitioned networks, however, a classifier is needed to specify whose subspace a given final state belongs to. This classifier is trained using a cross-entropy loss. The proposed algorithm is well-supported, and is strikingly similar to Q-learning algorithms with a delayed critic update, but GFlowNet-style. Overall, great idea and great results to support the idea! :)
Critical strengths
This paper shines in the experiments section, where in four tasks, hierarchical cooperative GFlowNets are shown to mostly outperform or otherwise match standard baselines and exploration-based baselines! The theoretical results are impressive because they show such large gaps in performance between the methods, clearly demonstrating the HCG is effective at exploration, and learning to sample.
Critical weaknesses
Firstly, I feel unfortunately compelled to reject this paper because of the lack of a related works section, and a lack of citations throughout the text. This made it a lot harder to compare this work with prior work: it took me a substantial amount of effort to find related works, to place this work into context. Additionally, a general lack of comparison with prior GFlowNet-related literature, including work in regularised RL like trajectory general mellowmax, and searching after sampling like local search GFlowNets made the results less interesting.
Secondly, I have very strong concerns with Section 3.1, specifically in the definitions of the supports , and the ownership sets . In particular, I do not believe that for a reasonable definition of "support," that anything other than is possible for any of the GFlowNets (because remember, we are not yet dealing with HCGs). This yields a substantial simplification: since GFlowNets are necessarily full-support over the state space, all the GFlowNets must also be full support, thus the ownership set is empty for all i>0, collapsing to a single active GFlowNet over the state space. This is seemingly a useless formalism. However, I fully understand why you have defined the sets in this way, since in Section 4 you will replace the notion of "support" with one of "empirical support" (which is still undefined mind you), which will instantly fix this issue. However, my concern with this "empirical support" fix is that the regions are inevitably going to be "swiss cheese," with many gaps, individual samples of one GFlowNet in a sea of another. If there is a simple fix, I could imagine it being with the hull of the sampled points defining the observed regions, but I didn't put much thought into it. In any case, I understand that the authors were going for a theoretical justification of the method, I appreciate it, I think its cool, just please attempt a fix (and if I'm wrong, then please correct me, and add these details to the paper so that others do not make the same mistake).
Do not despair, I really want to see this paper published! Address these two points (and my concerns in the questions below) and I will be more than happy to increase the score!
Questions for authors
First, some basics that can be cleared up somewhat quickly:
- Classic ensembling question incoming: how does a K-times larger GFlowNet perform, with the same parameters as the ensemble, in the tasks? How do different network sizes affect performance (give me a nice performance curve).
- On the topic of parameters of the neural networks, no details are given in the paper or the appendix about the networks, please provide said details. Which architecture, what sorts of hyperparameters, any tricks used to get it to work, optimisers, learning rates, temperatures, etc.?
- How do the exploration methods you list and the methods I stated in the weaknesses compare in terms of performance?
Second, some more involved questions:
- You introduce a new hyperparameter, K being the number of GFNs to train with. Given that you don’t know the number of modes ahead of time, what is the effect of specifying too few modes in a many-mode setting, and conversely, the effect of specifying too many modes in a few-mode setting? For instance, what happens when K=100 when there are only 3 Gaussians, and conversely?
- Is Figure 1 an actual GFN example or is it just hypothetical? If it’s the latter, please state this explicitly, if the former, please provide experiment details somewhere for reproducibility. Further, explicitly demonstrating the coverage problem is not done experimentally, and there is no citation to work that explicitly states this coverage problem, thus I am left doubting whether the issue is real, despite the success of your later experiments. In total, I want an explanation of why the method works, supporting your explanation for why it works and was needed, with experiments. Further, I think the figure is deeply flawed, and will lead to misunderstandings of how GFNs actually work. I would further state that two GFNs allocating their entire distributions on the separate modes makes no sense, since they should still sample proportionally to the NON-ZERO rewards between and outside of the rings, so some coverage must be placed here as well. The figure seems misleading. If you could produce a similar figure to Figure 1 with actually learned GFlowNet samples versus the true distribution (like in the 8-Gaussians) this might be more convincing.
- I believe it is common in GFlowNet literature to provide existence and uniqueness results to any new formulation. I would consider adding these to the text. Existence is trivial given that Equation 9 going to zero given a mix between the GFlowNet and zero-mass policy is already basically shown. Uniqueness is also somewhat trivial given a particular backward policy. Just a thought, not necessary by any means, I also can't remember if every paper I've seen does this.
- Figure 3: Why do you think the single GFN collapsed to a single mode? Could it be that the credit assignment at such long trajectory lengths was causing difficulties in finding other modes, so as soon as a single mode was found, it was given almost all the probability mass? I can see that this is a difficult exploration problem, but I’d like some additional context for why this is the case. Also, why not use another loss from the GFlowNet playbook? Like SubTB or something of the sort?
Third, of utmost importance:
- In the AMP task, I have mostly seen top-K diverse rewards reported as a standard metric. I would strongly consider restating the results in this way. In particular, use the Levenshtein distance to measure uniqueness of a given mode according to a task-specific distance threshold, then take the mean of these samples' rewards. Also, I think the idea of capturing all the samples throughout training is interesting, and more closely resembles what a true scientific process would be, I don't think this is standard in the literature. Take the top-100 samples over 1000 samples generated with the final model after training. This will be much more interesting than the combined 1000s of samples from the course of training. See here for examples: https://proceedings.iclr.cc/paper_files/paper/2026/hash/7688bc1bfa9e1f0c540bca03974df304-Abstract-Conference.html.
In my mind, this last question is of the utmost importance. Your results must be placed in context of the field, meaning that you should report results in a standard manner. You may, of course, argue that this is not standard :)
Final notes on writing:
- There are a number of anomolies in the writing. I don't understand the last 2 paragraphs of Section 3.2. While the paper is more or less well-written, some sections can be convoluted, with writing that is rather difficult to understand, even after multiple passes.
- Finally, I don't like the "cooperative" in the name. This elicits the idea of game theory or of multi-agent systems, not an ensemble of GFlowNets defined over disjoint subsets... I would highly consider a quick swap with "coordinated" or "combinations."
Kind reminder
As the discussion period is already halfway through, I would appreciate any answer to my questions and concerns.