- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper propose RC-GFN that uses frozen reference rewards and enable to concentration parameter an interpretable meaning. It derives population calibration and finite-reference error bounds, with experiments on HyperGrid and four molecular/sequence benchmarks.
Critical strengths
The motivation is clear, the method is simple, and the separation of reference error from policy-fitting error is useful. The mode-discovery improvements, particularly on sEH and RNA, are encouraging.
Critical weaknesses
Practical calibration remains uncertain. Table 5 shows a TFBind10 top-set mass of 0.292 under the learned policy versus 0.673 under the frozen target. Can the authors identify the main cause and test whether an alternative training objective or replay strategy reduces this gap? Related approaches such as SubTB (Madan et al., 2023) and RapTB (Wang et al., 2026) should be discussed as complementary training methods. Beyond current experiments, a controlled, enumerable setting such as, a bounded Game of 24. It can help assess how reference-set size and coverage affect calibration under sparse rewards. In particular, what happens when the reference set misses all/most of correct solutions, and how do reward ties affect the resulting concentration?
The empirical comparison could better isolate the benefit of rank calibration. Please clarify whether tuning budgets were comparable. Since GR-GFN also uses rank-derived targets, can the authors clarify why its comparison is limited to HyperGrid? Another downstream comparison will help support the method. If underperform on some benchmarks, diagnosis and analysis on such cases will also be helpful.
Questions for authors
See weakness part. I'm willing to raise my score if the authors solve my concerns.
2 comments
Response to Reviewer zbof (Part 2/2)
Respond to Critical Weaknesses 3: Comparability of tuning budgets across methods.
To make the tuning of RC-GFN and the temperature-controlled baselines comparable, we use the concentration experiment introduced above. Since and are both solved from the same promised mass before training, the two methods receive the same tuning budget. We then compare the mass each method delivers on , how uniformly its policy spreads this mass within and how many modes it discovers during training. Except on TFBind10, both RC-GFN and the baselines largely fulfill the promised masses. However, the policy of RC-GFN is more uniform within , and RC-GFN discovers modes faster or discovers more modes on most tasks and promised masses (Section 4.3 “CONCENTRATION EXPERIMENT”).
Respond to Critical Weaknesses 4: Downstream comparison with GR-GFN and diagnosis of underperformance.
GR-GFN was designed for multi-objective optimization, which differs substantially from our single-objective tasks, and its code is not publicly available, so faithfully reproducing its details on our downstream tasks was not straightforward, and we therefore reproduced GR-GFN in a minimal form and tested it only on HyperGrid. We have now added GR-GFN as a baseline to Section 4.2 “MODE DISCOVERY EXPERIMENT”, and its implementation details are given in Appendix H.1.2 “IMPLEMENTATION DETAILS”. RC-GFN discovers more modes than GR-GFN on sEH, TFBind10 and RNA, whereas GR-GFN performs best on QM9. To diagnose this case, we computed the exact sampling mass that each trained policy places on the top set of each task, where is the quantile that defines the mode set.
| Task | Target mass of RC-GFN | RC-GFN | GR-GFN |
|---|---|---|---|
| QM9 | 0.181 | 0.188 ± 0.017 | 0.454 ± 0.009 |
| sEH | 0.139 | 0.267 ± 0.026 | 0.697 ± 0.038 |
| TFBind10 | 0.131 | 0.034 ± 0.001 | 0.023 ± 0.000 |
| RNA | 0.885 | 0.764 ± 0.008 | 0.182 ± 0.006 |
On QM9, the mass that RC-GFN places on is close to its target mass. GR-GFN fits a cross-entropy toward the best candidate of each batch, which has no finite optimum, so its concentration keeps increasing during training and is set by the training length rather than by a parameter. Within this budget, GR-GFN places much more mass on than RC-GFN on QM9, where the mode set covers a large part of , so the higher concentration yields more modes. The same mechanism overshoots on sEH, where GR-GFN places most of its mass on and discovers substantially fewer modes than RC-GFN. On TFBind10 and RNA, GR-GFN places less mass on than RC-GFN and discovers fewer modes. Note that the mode discovery experiment does not require RC-GFN to deliver its target mass, and we designed a separate experiment in Section 4.3 "CONCENTRATION EXPERIMENT" to evaluate whether the promised concentration is delivered.
Response to Reviewer zbof (Part 1/2)
We thank the reviewer for the careful reading and constructive feedback. Our response rests on three additions to the revised manuscript. First, a concentration experiment calibrates the concentration parameter λ of RC-GFN and the inverse temperature β of the temperature-controlled baselines before training to the same promised mass on the top set Tq (Section 4.3 "CONCENTRATION EXPERIMENT"). Second, a failure case analysis of TFBind10 tests interventions on the training objective, the replay strategy and other factors of the original setting (Appendix J "FAILURE CASE ANALYSIS"). Third, a sparse reward experiment in a bounded Game of 24 examines how the reference size, the coverage of the reference and reward ties affect calibration (Appendix K "SPARSE REWARD EXPERIMENT"). We address your concerns point by point below.
Respond to Critical Weaknesses 1: Large fitting error on TFBind10 and complementary training objectives and replay strategies.
We agree that RC-GFN shows a large fitting error on TFBind10 in the mode discovery experiment, and the above concentration experiment confirms this observation. This difficulty is shared by all tested methods. In the concentration experiment, the temperature-controlled baselines also fall far short of the promised mass on TFBind10, while RC-GFN comes closest to it at every promised mass (Table 2). We therefore analyze TFBind10 as a failure case in Appendix J “FAILURE CASE ANALYSIS”, with a series of interventions that each change one factor of the original setting (including the SubTB objective and a change of replay strategy suggested by the reviewer). Some of these interventions slightly reduce the fitting error, but none of them brings the mass of the policy on Tq close to that of its target. We then compare TFBind10 with TFBind8 [1,2], which is built in the same way and differs only in the sequence length and the transcription factor. The two reward landscapes are similarly rugged, but their promised masses are fulfilled very differently. A visualization of the two landscapes shows that the modes of TFBind10 are far more scattered across the design space, which helps explain its persistent policy-fitting gap. RC-GFN uses ranks rather than numerical reward gaps to specify its target weights. In the concentration experiment, its learned policy is generally more uniform within the top set Tq than the temperature-controlled baselines. Moreover, on TFBind10, RC-GFN places more mass on the mode set than the temperature-controlled baselines at all three promised masses and turns it into more discovered modes, as shown in Section 4.3 "CONCENTRATION EXPERIMENT".
In addition, we now discuss SubTB and RapTB as complementary training methods in Section 2 "RELATED WORK". RapTB is designed for language-model policies in which every prefix can terminate and receives its own reward, which does not hold in our tasks, where only complete molecules and sequences are evaluated. The original RC-GFN setting already trains with the prioritized replay training (PRT) of Shen et al. [3], and the replay intervention replaces it with uniform replay.
[1] Barrera, Luis A., et al. "Survey of Variation in Human Transcription Factors Reveals Prevalent DNA Binding Changes." Science, vol. 351, no. 6280, Mar. 2016, pp. 1450–54.
[2] Trabucco, Brandon, et al. "Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization." Proceedings of the 39th International Conference on Machine Learning, PMLR, June 2022, pp. 21658–76.
[3] Shen, Max W., et al. "Towards Understanding and Improving GFlowNet Training." Proceedings of the 40th International Conference on Machine Learning, PMLR, July 2023, pp. 30956–75.
Respond to Critical Weaknesses 2: Effect of reference size, coverage and ties under sparse rewards in a bounded Game of 24.
To assess how reference size and coverage affect the calibration of RC-GFN under sparse rewards, we designed a bounded Game of 24 based on the Expr24 task of RapTB (Appendix K "SPARSE REWARD EXPERIMENT"). We found that a larger reference lowers the reference error and brings the target closer to the promised mass. Excluding all correct expressions from the reference leaves the frozen target mass on the correct set nearly unchanged. A reference that misses most of them lies between full coverage and this case. Excluding also the incorrect expressions closest to 24 lowers this mass markedly, because these expressions then share the highest empirical rank with the correct set. Reward ties make the λ given by the no-tie formula (Section 3.4, Proposition 4) over-concentrate the target, while a λ solved from the tie-group sizes restores the promised mass in the target, and the learned policy follows it, although a fitting gap remains.