I have a number of concerns regarding the experimental evaluation.
First, from my understanding, experiments on Figure 6 and Figure 7 are on extremely small toy tasks (31x31 or 15x15 grids), so one should expect the GFlowNet to eventually converge towards the optimal perfect sampler. However, one can see that different training curves do not converge towards the same TV metric. This is puzzling to me. Is it happening due to very short training runs? If so, it would be more informative to increase the number of training iterations and verify that all methods converge to the optimum, while the proposed one converges faster than others. In addition, it is not even clear whether NRD converges to the optimum: the TV value of 0.25 on 8 Gaussians is rather large. I do understand that, as written in Appendix, the TV is evaluated using MC estimates of GFlowNet sampling probabilities. Then, to obtain a reference metric value to track converence to the optimum, one could compute the optimal forward policy using dynamic programming (since the environment is very small), and check what TV value it will produce. I would also suggest running the same experiment on the classic hypergrid benchmark from GFlowNet literature of larger size, like 20x20x20x20, and verify that the models converge to the optimum using the protocol from prior literature (see, e.g., [1]).
The results for some baselines on toy environments also look strange, e.g., all adaptive exploration baselines from previous literature performs worse than the plain on-policy TB on Moons and Adaptive Teacher performs worse on GridWorld. Do the authors have an explanation for this?
Second, all experiments are performed with trajectory balance loss (and its variations as baselines). However, other GFlowNet training losses have been proposed in the literature. Subtrajectory balance (SubTB) [1] is a well-established one that has shown better performance in comparison to TB across a veriety of domains and benchmarks. To better quantify the empirical efficiency of NRD, I would suggest adding it as a baseline. In addition, I would welcome a discussion on whether a similar gradient term can be added to SubTB loss to guide exploration, further improving the empirical performance.
Third, the main text presents the experiments on very small synthetic environments, plus one biological task that is still rather small-scale compared to real-world domains (1910 candidates), so other larger environments would be good to back up the performance of the method. I do note that Appendix contains additional experiments on other environments (Table 2). However, it only compares the methods using average reward of the final model as a metric, and many methods obtain very similar results across it. Judging from the main text results, I would expect NRD to improve upon the convergence speed and/or final distribution approximation quality in comparison to the baselines. I would suggest presenting training convergence plots for these environments too, as well as tracking some distribution aprroximation metric, e.g. reward correlation presented in [2] or flow consistency in sub-graphs (FCS) presented in [3].
Finally, it is stated in Appendix that the backward policy is learned in some experiments, however, I did not find a specific discussion on how exactly it is learned. Is it simply updated through the TB loss, while the forward policy is updated through TB loss plus NRD gradient component?
The paper presents a very interesting method that is simple yet potentially strong, however, my concerns prevent me from recommending acceptance for the current manuscript. If they are sufficiently addressed, I am willing to increase my score.
References:
[1] Madan et al. Learning GFlowNets from partial episodes for improved convergence and stability. ICML 2023.
[2] Whittamer et al. Trajectory balance: Improved credit assignment in GFlowNets. NeurIPS 2022.
[3] Silva et al. When do GFlowNets learn the right distribution? ICLR 2025.