- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CBFlowNet, a method for improving exploration in GFlowNets under limited sampling budgets. It uses a learned controller to periodically select subsets of action types, based on set-level reward predictions and a bandit-style exploration strategy. The selected subsets guide trajectory collection toward different subspaces during training. The paper also provides theoretical analyses of trajectory-support preservation, zero-loss target consistency, and subspace-selection errors. Experiments on molecular design, RNA design, and bit-sequence generation show improvements in finding high-reward candidates.
Critical strengths
-
Strong empirical results in molecular mode discovery. CBFlowNet discovers approximately 41% more high-reward modes than QGFN under the matched total reward-evaluation budget reported in Table 5. This is a useful comparison, as it suggests that the improvement is not simply due to additional reward evaluations.
-
A reasonable approach to learning action-subset utilities. Rather than relying only on individual action scores, CBFlowNet uses set-level reward feedback to learn which action combinations are worth exploring. The ablations in Figure 6 show that Deep Sets-based selection consistently outperforms Top-K and random selection across the tested arm-scoring variants, suggesting that learned subset selection is more effective than these simpler alternatives.
Critical weaknesses
-
Unclear generalization from complete to partial sets. The Deep Sets scorer is trained on complete sets but used to score partial sets during greedy subset construction (Appendix F). Good predictions on complete sets do not necessarily imply reliable rankings of partial sets, which are what the selection procedure actually depends on. The authors acknowledge this issue, but there is no direct evaluation of these intermediate rankings. The theoretical analysis accounts for the effects of partial-set queries in the selection residual $e_t$, without establishing that this residual is small. The main concern is therefore whether the scorer can reliably guide subset construction in practice.
-
The ablations do not fully isolate the benefit of interaction-aware and adaptive selection. Deep Sets is trained with set-level reward labels, while Top-K relies on individual arm scores. It is therefore unclear whether modeling non-additive interactions provides a benefit beyond learning from set-level supervision. Also, Appendix E.1 shows that a fixed four-arm subspace already contains all target modes in the bit-sequence task, making it difficult to assess the benefit of switching subspaces in this benchmark. The matched-budget molecular results support the overall method’s effectiveness, but do not by themselves resolve these mechanism-level attribution questions.
Questions for authors
-
Given the complete-to-partial-set mismatch discussed in Appendix F, is there empirical evidence that scores on partial sets reliably guide greedy construction toward high-quality complete sets? For example, at frozen policy checkpoints, how does the sampling yield of the greedily constructed set compare with that of the best feasible complete set under the same cardinality constraints? Could you report this comparison both with and without the dropout exploration bonus?
-
Would Deep Sets still outperform an additive predictor trained with the same set-level supervision protocol and using the same action identities and auxiliary features, with exploration settings and reward-evaluation budgets matched? The current Top-K comparison differs in both the use of set-level supervision and the ability to model action interactions, leaving unclear whether non-additive interaction modeling provides a benefit beyond learning from set-level feedback.
-
How is the fixed subset chosen in the Hard Pruning baseline? Are the subset size, gating and exploration settings, and total reward-evaluation budget matched to CBFlowNet? In particular, does Hard Pruning freeze a subset selected by the same learned procedure after a comparable warm-up, or does it use a different selection rule?
10 comments
Response to Reviewer Kpum(part 2/2)
W2 part 2. A fixed four-arm subspace already contains all target modes in the bit-sequence task, making it difficult to assess the benefit of switching subspaces in this benchmark.
Thank you for pointing this out. We would like to clarify two points.
First, this complete coverage results from the deliberate construction of the toy environment(bitseq), which serves as an illustrative demonstration. In molecular generation, there is no corresponding guarantee built into the task that a small, fixed action subspace contains all high-reward modes. The coverage property of this toy example therefore does not necessarily generalize to the molecular setting.
Second, complete coverage does not imply efficient discovery. Within a given subspace, some modes may be considerably easier for the GFlowNet to discover than others. After the more accessible modes have been found, discovering additional high-reward modes may become increasingly difficult. A mode that is difficult to discover in one subspace may be more accessible in another, where different available actions provide alternative generation trajectories. Switching subspaces can therefore offer complementary exploration opportunities and potentially improve mode discovery under a fixed reward-evaluation budget, even when a single subspace contains all target modes.
Q3. How is the fixed subset chosen in the Hard Pruning baseline? Are the subset size, gating and exploration settings, and total reward-evaluation budget matched to CBFlowNet? In particular, does Hard Pruning freeze a subset selected by the same learned procedure after a comparable warm-up, or does it use a different selection rule?
Thank you for raising this point. In the original Hard Pruning baseline, we uniformly sample 30 actions from the full action set and keep this subset fixed. Thus, the subset is selected randomly, rather than by CBFlowNet’s learned selection procedure.
To directly assess the benefit of continued adaptive subspace selection, we introduce a controlled Warm-up Hard Pruning baseline. We first train CBFlowNet for 10,000 steps, during which the Deep Sets scorer is also trained. We then fork the same trained checkpoint into two branches:
CBFlowNet (continued adaptive selection): continues training with adaptive subspace selection.
Warm-up Hard Pruning: freezes the (K=30) subset selected by the learned Deep Sets scorer at the branching point and continues training within this fixed subset.
The two branches therefore share exactly the same warm-up training history and learned subset-selection procedure. We match the subset size (K=30), hard-gating strategy, exploration parameter (), and total reward-evaluation budget across methods, with all other training settings held constant. The 10,000-step warm-up, including the reward evaluations used to train the scorer, is included in each branch’s total budget. Both branches consequently receive the same remaining reward-evaluation budget after the split.
| Method | Modes with R > 7.5 | Modes with R > 8 | Top-1000 reward |
|---|---|---|---|
| Deep Sets | 14,833 | 3,012 | 8.432 |
| Warm-up Hard-pruning | 5016 | 789 | 8.379 |
| Hard Pruning | 2467 | 315 | 8.262 |
Warm-up Hard-pruning outperforms random Hard Pruning, confirming the benefit of reward-informed subset selection. Nevertheless, Deep Sets discovers almost 3 times and 4 times as many modes at R>7.5 and R>8, respectively, and achieves a higher top-1000 reward. These results support the benefit of continued adaptive subspace selection over one-time pruning(W2 part2). They also complement our response to W2: subspace coverage alone does not guarantee efficient mode discovery, and switching subspaces can provide complementary exploration opportunities.
Response to the Revision
Thank you for the clarifications, which address my original concerns. I have two further questions about the experimental settings and results.
-
Table 7 and Appendix H specify for the RNA experiments. However, the CBFlowNet mode-discovery curves for RNA1, RNA2, and RNA3 in Figure 4 appear to match the curves labeled in Figures 9, 10, and 11 at every plotted point. The RNA1 and RNA2 average-reward curves also appear to match their corresponding curves.
Since changes the training target , could you clarify:
- Which was used for each method in Figure 4, and whether any table entries, figure labels, or descriptions need correction?
- Which configurations, seeds, and runs produced the CBFlowNet curves in Figure 4 and Figures 9 to 11?
- Whether all methods used the same or were tuned separately under comparable protocols? If tuned separately, please describe the protocol.
-
Algorithm 1 places full-vocabulary initialization before the training steps. If Table 4's 100,000 steps refer to this subsequent loop, the batch size of 4 would already account for Table 5's 400k training evaluations. Does the 500k total include both training and unrestricted-evaluation calls during initialization, and what stopping rule enforces this budget?
Please also clarify how Appendix J's "100 evaluations per 400 sampling steps" corresponds to the 100k bandit evaluations in Table 5. Please specify what "sampling steps" means, the evaluation batch size , and the evaluation frequency, along with which samples contribute to the reported mode counts and Top-1000 reward.
For QGFN, the strongest baseline in Table 5, please report the variant, , -step setting, any relevant schedule, reward exponent, and tuning protocol.
The additional experiments have strengthened the paper. Clarifying these points would help me decide whether to recommend acceptance.
Response to Reviewer Kpum(part 1/2)
Dear Reviewer Kpum Thank you for your valuable comments, which are crucial to improve of our paper. We first summarize the proposed weaknesses and questions(we noticed that some weaknesses and questions are alike, but some weaknesses contain several parts to clarify).
Thank you for highlighting this issue. We directly evaluated the rankings of candidate additions at partial sets encountered during greedy construction. With the GFlowNet, Deep Sets scorer, and arm features frozen(from iteration 50000), we examined prefixes of sizes 5, 10, 20, and 29. For each prefix, we evaluated ten candidate additions, including the greedy choice. Each resulting partial set was completed to 30 blocks with the same suffixes, and each completion was evaluated using 400 samples under the same sampling settings. We report the within-prefix Spearman correlation between prediction means:
ρ=1−10(102−1)6∑i=110(r^i−ri)2.
Higher values indicate better agreement between the two rankings.
The positive correlations across all evaluated prefix sizes indicate that higher predicted scores tend to correspond to higher completion rewards, supporting their usefulness for guiding greedy construction. Ranking agreement increases from 0.58 at prefix size 5 to 0.81 at size 29, suggesting more accurate rankings as the partial sets approach the target size.
According to your suggestion, we compare the constructed complete set with the best found reference set in molecule generation task. At a frozen policy checkpoint, we perform multi-start swap search over complete 30-block sets, initialized from random, Top-30, and greedy selections. We evaluate 2,000 candidate sets using GFN sampled candidates for each set, which provides a high-budget best-found reference. We compare the best set with greedy selections with and without the dropout bonus, the results is shown below. We report the mean transformed training reward, mean raw proxy score, and high-score yield, defined as the fraction of evaluated molecules whose raw proxy score exceeds 7.5.
Greedy selection without the exploration bonus achieves a mean reward close to the best-found reference (0.5357 vs. 0.5416), with comparable proxy scores and high-score yield (50.25% vs. 49.75%). Both greedy variants substantially outperform random selection.
Thank you for highlighting this confounding factor. To address this concern, we conducted an additional experiment on molecular generation with an additive predictor trained using the same set-level supervision, action identities, and auxiliary features as Deep Sets, with matched exploration settings and reward-evaluation budgets. All other experimental settings were kept unchanged. This baseline learns from set-level feedback while restricting its predictions to additive contributions from individual actions.
The additive predictor slightly improves over Top-K. Deep Sets retains an advantage over the additive predictor under the matched protocol: it discovers 13.1% more modes with R > 7.5 and 17.8% more modes with R > 8, with a smaller improvement in Top-1000 reward.
These results provide preliminary evidence that non-additive interaction modeling offers benefits beyond set-level supervision alone, particularly for discovering high-reward modes in this task.