acceptodds
Under review as a conference paper at ICLR 2027

GFlowNets for Strategic Planning in Hierarchical and Cyclic Environments

Abstract

Generative Flow Networks (GFN) sample terminal states in proportion to reward and have shown promising results in molecular and biological sequence design. Their capacity to explore diverse high-reward solutions may extend naturally to strategic decision-making in tasks with hierarchies, prerequisite chains, and alternative routes, yet they have been underexplored due to departure from standard formulations such as cyclicity in transitions. We designed and evaluated a sub-trajectory-balance (SubTB) GFN with replay on HierarchyCraft , which is a discrete hierarchical environment whose cyclic state space and recursive tool dependencies allow the freedom of diverse valid solution routes. We handle the cyclic state space using the theory of and tested it empirically with the SubTB GFN. Compared against an entropy-regularized SoftDQN agent, which shares the optimal policy of the GFN, the GFN attains higher success rates on four of the five deep tasks and more strategic modes on most of them. At similar success rate, the GFN finds about two to four times as many strategic modes as SoftDQN and PPO. We further note that the gap persists across nearly all setups we tried for SoftDQN, suggesting that objectives with a shared optimum can learn at different rates and that flow-based objectives may retain an advantage, especially in long-horizon and sparse-signal tasks.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Probabilistic Method, GFlowNets, Reinforcement Learning, Hierarchical learning, Path Planning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wdzhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the application of Generative Flow Networks (GFlowNets) to hierarchical planning environments with cyclic state transitions, complex dependencies, and multiple valid solution paths. Unlike conventional GFlowNets, which typically operate on directed acyclic graphs (DAGs), practical planning environments often involve reversible actions and repeated state visits. Building on the non-acyclic GFlowNet framework of Morozov et al. (2025), the authors implement a Sub-Trajectory Balance (SubTB) GFlowNet with experience replay in the HierarchyCraft environment. Experimental results demonstrate advantages over several reinforcement learning baselines, particularly in long-horizon tasks with sparse rewards.

Critical strengths
  1. Meaningful problem setting. Extending GFlowNets to hierarchical planning with cyclic transitions is a worthwhile research direction, although the underlying non-acyclic formulation is largely inherited from prior work.

  2. Promising experimental results. The proposed method achieves strong performance on several challenging tasks. In Table 2, GFN achieves a 90.6% success rate on Gold, compared with 4.6% for SoftDQN. In the tuned experiments (Table 8), GFN achieves 54.4% success and 37.7 strategic modes on Ender-eye, compared with 28.3% and 7.7 for SoftDQN. The additional ablations on demonstrations, backward policies, and n-step backups further strengthen the empirical analysis.

Critical weaknesses
  1. Limited methodological novelty. The main components are largely adapted from existing work, including the non-acyclic GFlowNet formulation of Morozov et al. (2025), SubTB, and replay-based training. The contribution primarily lies in their integration and empirical evaluation on HierarchyCraft. The paper introduces neither a new flow-matching objective nor a novel learning algorithm or convergence analysis for cyclic environments.

  2. Insufficient explanation of Equation (5). Although Equation (5) defines edge flow through the expected number of edge visits, its role in addressing repeated state visits is not sufficiently explained. The authors should clarify the intuition behind this formulation, the interpretation of the expectation over trajectories, and how the resulting expected-visit flow handles cycles.

  3. Unclear practical implementation of Equation (5). It remains unclear how the edge flow defined in Equation (5) is implemented or computed in practice. Both TB and SubTB are formulated using state flows rather than explicit edge-flow estimates. The authors should clarify whether Equation (5) is used directly in training or serves only as a theoretical justification for the state-flow parameterization.

  4. Insufficient justification for combining TB and SubTB. Equation (6) introduces an additional TB loss on top of SubTB. However, SubTB already incorporates constraints over different trajectory lengths, including full-trajectory balance, with their relative weighting controlled by λ_sub. Although the authors state that the additional TB term anchors the global flow scale, its necessity remain unclear. In my opinion, this is very likely unnecessary.

  5. Limited comparison with recent GFlowNet methods. More advanced GFlowNet training approaches, such as Adaptive Teacher, could potentially be incorporated into the proposed framework and yield stronger performance. Comparisons with such methods would provide a more comprehensive evaluation of the proposed approach.

Questions for authors

see weaknesses

Reviewer_s2MJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the application of GFlowNets to RL-centric cyclic environments. The authors combine sub-trajectory balance (SubTB), a non-acyclic GFlowNet formulation, and a retrospective replay mechanism to evaluate GFlowNets in these environments. The authors mainly consider HierarchyCraft benchmark. The results show that GFlowNets achieve competitive or higher performance compared to RL baselines, including SoftDQN, PPO, and SAC.

Critical strengths

None

Critical weaknesses
  • Limited novelty and contribution. The methodological novelty is limited, and the experimental scope and insights are also insufficient. The finding is incremental (see below), while the main method combines existing SubTB, non-acyclic GFlowNets, and retrospective replay. The evaluation is also limited to a single HierarchyCraft benchmark.

  • Insufficient evidence to support the claim. The authors argue that "GFN’s credit assignment mechanism contributes to its higher performance", but the experimental result is insufficient to support this. It remains unclear whether the improvements stem from GFlowNet-specific properties or from the multi-step credit assignment in TB. If former, the authors do not consider counterpart RL methods (PCL, TD(�0��0�)) which also leverage subtrajectories with multiple lengths in training. If latter, this advantage is already well established in the literature and does not carry a novel finding.

Questions for authors
  • Can the authors compare SubTB with its RL counterparts (PCL, TD(λ\lambda))?

  • The authors mention that "GFN’s credit assignment mechanism contributes to its higher performance." Can the authors explain why GFlowNets can provide better credit assignment than RL methods (including PCL, TD(λ\lambda))? I was also wondering whether this claim is limited to the general GFlowNet formulation (DB, TB, SubTB) or the specific formulation of SubTB.

  • How well does the proposed approach generalize beyond HierarchyCraft? Can similar improvements be observed in other environments?

Reviewer_NgSaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the application of non-acyclic GFlowNets, specifically following the formulation of Morozov et al. (2025), to HierarchyCraft, a symbolic planning environment with recursive crafting dependencies.

The authors use SubTB with an additional TB term and demonstration-based replay to train the non-acyclic GFlowNet. They obtain substantially better results compared to SoftDQN, particularly on the gold and ender-eye tasks. The authors attribute these improvements mainly to more effective credit assignment provided by SubTB.

Critical strengths
  1. Strong empirical results. The paper shows that using a combination of TB and SubTB is very advantageous compared to SoftDQN on multiple tasks in the evaluated environment. In particular, the improvement on the gold task is substantial, with the success rate increasing from 4.6% to 90.6%. The main comparisons also use 10 random seeds, which provides more statistical reliability to the results.

  2. HierarchyCraft as a new environment for GFlowNets. HierarchyCraft has not been previously evaluated with GFlowNets and seems to be a good environment for evaluating GFlowNets in long-horizon planning tasks, especially in environments with cycles.

Critical weaknesses
  1. Limited novelty. The main method is basically a direct application of what was already established by Morozov et al. (2025), but using SubTB and TB. In fact, Corollary 3.11 of the original paper by Morozov et al. already states that, under a fixed backward policy PBP_B, any GFlowNet loss can be used to learn the corresponding forward policy PFP_F in the non-acyclic setting.

  2. Missing Detailed Balance comparison. Although Morozov et al. already showed that their framework could be used with different GFlowNet losses, I think a comparison between the proposed method and Detailed Balance (and stable DB) is missing, especially since Detailed Balance was the objective tested in the original paper

  3. Difficulty isolating the effect of SubTB. The paper presents SubTB as a way to improve credit assignment in this cyclic environment. However, in practice, SubTB is combined with an additional TB term, making it difficult to isolate the effect of SubTB. It would be useful to include an ablation with different weights for the TB term, ranging from its full weight to zero, where only the SubTB term remains.

Questions for authors
  1. Comparison with other GFlowNet losses. It would be good to see a comparison between your proposed combination of SubTB and TB, Detailed Balance, and pure TB.

  2. Time-augmented DAG. Although adding time to the states to obtain a time-augmented DAG could change the sampling distribution due to trajectory multiplicity, have you compared your method against this alternative, instead of keeping the environment cyclic?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.