GFlowNets for Strategic Planning in Hierarchical and Cyclic Environments

Generative Flow Networks (GFN) sample terminal states in proportion to reward and have shown promising results in molecular and biological sequence design. Their capacity to explore diverse high-reward solutions may extend naturally to strategic decision-making in tasks with hierarchies, prerequisite chains, and alternative routes, yet they have been underexplored due to departure from standard formulations such as cyclicity in transitions. We designed and evaluated a sub-trajectory-balance (SubTB) GFN with replay on HierarchyCraft , which is a discrete hierarchical environment whose cyclic state space and recursive tool dependencies allow the freedom of diverse valid solution routes. We handle the cyclic state space using the theory of and tested it empirically with the SubTB GFN. Compared against an entropy-regularized SoftDQN agent, which shares the optimal policy of the GFN, the GFN attains higher success rates on four of the five deep tasks and more strategic modes on most of them. At similar success rate, the GFN finds about two to four times as many strategic modes as SoftDQN and PPO. We further note that the gap persists across nearly all setups we tried for SoftDQN, suggesting that objectives with a shared optimum can learn at different rates and that flow-based objectives may retain an advantage, especially in long-horizon and sparse-signal tasks.