acceptodds
Under review as a conference paper at ICLR 2027

Data first, Rewards Later: GFlowNets Across Pre-Training and Post-Training

13 views

Abstract

Discrete diffusion models and Generative Flow Networks (GFlowNets) are two seemingly alternative approaches to modelling discrete data. They differ in terminology and parametrisation, but most critically in how they are trained. While discrete diffusion models are primarily trained from data, GFlowNets are primarily trained from rewards. We show that this separation is not fundamental: both model classes admit a common flow-score formulation that allows training objectives to transfer between them. Building on this connection, we propose a pre-training/post-training regime analogous to the typical LLM training pipeline. A GFlowNet can first be pre-trained directly from data, without an explicit reward model, and subsequently post-trained with standard reward-based GFlowNet objectives without changing the underlying generative model. This combines the flexible construction spaces and structured generation native to GFlowNets with efficient learning from existing data characteristic of discrete diffusion models. On peptide generation, data-based pretraining substantially improves subsequent reward-query efficiency under identical post-training, reaching 88.6% high-reward samples compared with 73.5% when training from untrained initialization.

Primary area:
generative models
Keywords:
Generative Flow Networks, Discrete Diffusion, Data-Based Pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_rMoLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper develops a flow–score formulation connecting GFlowNets with discrete diffusion and transfers score-entropy objectives to data-based GFlowNet training. A transition-sampled variant avoids the need for analytic backward visitation probabilities. This enables pre-training from terminal samples without an explicit reward model, followed by reward-based post-training of the same model. Experiments evaluate the full pipeline on peptide generation and data-only modeling on Ising configurations and binarized MNIST.

Critical strengths
  • The flow–score formulation provides a useful connection between data-based and reward-based training. The transition-sampled objective requires only local backward transition probabilities, allowing pre-training without analytic visitation probabilities and subsequent reward-based training of the same model.

  • The peptide study uses identical Replay-TB post-training and ten seeds. Learned-ZZ and oracle fixed-ZZ baselines help assess reward-scale estimation.

Critical weaknesses
  • Score entropy [1] and reward-free GFlowNet pre-training [2] are established, so the methodological advance relative to these foundations could be explained more explicitly. Fitting a terminal data distribution differs from the outcome-conditioned pre-training of [2], and the transition-sampled formulation appears useful. However, the paper could distinguish more clearly which aspects follow from the score-entropy framework of [1] and which require new derivations or enable capabilities unavailable in earlier approaches. A comparison of assumptions, objectives, and adaptation procedures would make the contribution easier to assess.

  • A matched behavior-cloning baseline on backward-sampled trajectories is missing. The offline-TB comparisons are informative, but they do not isolate whether the gains come from the proposed score-based objective or from obtaining a useful initialization through data-based training. Using the same architecture, pre-training data, backward trajectories, and post-training budget would help answer this question. Validation also remains limited to peptides of length at most ten, energy and magnetization statistics for Ising models, and qualitative MNIST samples, leaving the broader empirical applicability less clear.

[1] Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution. In International Conference on Machine Learning, 2024.

[2] Ling Pan, Moksh Jain, Kanika Madan, and Yoshua Bengio. Pre-Training and Fine-Tuning Generative Flow Networks. In International Conference on Learning Representations, 2024.

Questions for authors

See weaknesses.

1 comment

We thank the reviewer for the constructive feedback and for pointing us to related work. We address below the novelty of our contribution relative to SEDD and Pan et al., and clarify its relationship to behavior cloning.

1. Novelty relative to SEDD and Pan et al.

We build on score entropy from Lou et al. rather than claiming score entropy itself as a new principle. Our contribution is its mapping to GFlowNet flow ratios and construction DAGs, including objectives that can be evaluated using backward trajectories.

We thank the reviewer for pointing to the work on pre-training GFlowNets; however, we want to clarify that what is considered pretraining in our setting and theirs is significantly different. Pan et al. address outcome-conditioned pretraining, designed to support adaptation to potentially many downstream reward functions. Their method involves additional outcome-conditioning and marginalization machinery.

Our method directly fits a given data distribution, and subsequently continues training the same model with conventional reward-based objectives. This approach requires neither outcome-conditioned pretraining, nor an auxiliary predictor to an intractable adaptation quantity used in Pan et al.

The distinction concerns what is learned during pretraining and how the resulting model is adapted downstream.

2. Relationship to behaviour cloning We thank the reviewer for their recommendation and add the requested baseline. We agree that our approach belongs to the broad class of imitation-learning methods. It is not intended to establish that imitation learning itself is novel. The contribution concerns the specific score-based formulation and its connection to diffusion.

In the unrestricted tabular setting, score-entropy learning and maximum-likelihood training of the forward policy recover the same population-optimal forward policy under matched supervision. We have formalised this relationship in Appendix D.

Our analytic DSE objective marginalises over next-action uncertainty and is therefore a Rao–Blackwellization of its sampled counterpart. The same marginalization principle can also be applied to policy behavior cloning (BC). Our new experiments show that this marginalization is practically beneficial for BC, while analytic BC and analytic DSE achieve similar downstream performance in the evaluated settings.

The objective-specific contribution is therefore the score-based formulation, its theoretical connection to diffusion, and the associated estimators, not a claim of universal empirical superiority over maximum-likelihood training.

Reviewer_LkC5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates whether GFlowNets, traditionally trained using reward functions, can instead be pretrained directly from data and subsequently fine-tuned using rewards, analogous to the pretraining/post-training paradigm in language models. The main contribution is the introduction of a GFlowNet score, that "interpolates" between GFlowNet flow ratios to discrete score functions. Experiments on antimicrobial-peptide generation demonstrate an improvement in high-reward sample generation from approximately 73.5% without pretraining to 88.6% with data pretraining, after reward-based post-training.

Critical strengths

The relationship between GFlowNets and discrete diffusion is developed through an explicit flow-score correspondence.

The score-entropy loss has a mathematically characterized population minimizer--it allows GFlowNets to be trained without direct access to terminal reward values or the global normalization constant.

The peptide experiments are reasonably convincing, especially because the authors compare against reward-based trajectory-balance pretraining rather than only an untrained initialization.

Critical weaknesses

Previous literature has already established connections between GFlowNets and discrete diffusion, including interpreting discrete diffusion as a special case of GFlowNets.

The theory assumes a fixed backward policy and sufficient support, which is not justified.

The experimental results are modest, and are only limited to simple tasks.

Questions for authors

See weakness.

1 comment

We thank the reviewer for their constructive feedback. We address the main concerns below.

1. Novelty of the GFlowNet-diffusion correspondence

We agree that connections between GFlowNets and discrete diffusion have been explored previously. We do not claim the general observation that these models are related is new.
Our contribution is a constructive transfer of score-based learning objectives to GFlowNets. Specifically, we identify the relevant flow ratios, characterize the population minimizer of the corresponding objectives, and obtain analytic and transition-sampled score-entropy objectives for general construction DAGs under our stated graph and support assumptions.

The pointwise consistency and denoising arguments adapt those of Lou et al. (2024). We do not claim the underlying score-entropy minimization or marginalization identities as new. The GFlowNet-specific work is identifying the appropriate flow ratios and backward conditional marginals, and establishing their relationship to the construction graph and detailed-balance conditions.

The transition-sampled objective requires a separate estimator derivation. We rewrite the conditional-marginal term as an expectation over backward-sampled edges, weighted by the inverse one-step backward probability 1/PB(s∣s′)1/P_B(s\mid s'). This yields an unbiased estimator that requires sampled backward trajectories and local backward transition probabilities, without evaluating the multistep backward marginals required by the analytic formulation. Conversely, analytic DSE marginalizes the next-transition randomness and is the Rao–Blackwellized counterpart of this sampled formulation.

The state-flow identifiability argument is closely related to the completeness property of concrete scores established by Meng et al. (2022, Theorem 1). Their result identifies a normalized discrete probability distribution from neighboring ratios. Our statement concerns unnormalized GFlowNet state flows, which are identified up to a common multiplicative constant, and additionally reconstructs edge flows using a compatible backward policy.

We have revised the manuscript to distinguish these contributions from earlier unifying perspectives and explicitly attribute the inherited proof arguments, while clarifying the GFlowNet-specific identifications and estimator derivation.

2. Fixed backward policy and support

A fixed backward policy is a deliberate design choice, analogous to specifying the corruption process in diffusion models, rather than a fundamental restriction to the GFlowNet formalism. Fixing the backward policy does not restrict the set of representable terminal distributions, provided the backward policy has sufficient support as discussed in GFlowNet Foundations, Section 2.6 [1].

The backward policy determines how terminal reward is distributed among construction trajectories, and its choice can affect learning efficiency, particularly in sparse-reward settings. These optimization considerations are distinct from terminal-distribution expressivity.

A simple fixed backward policy makes sampling backward trajectories inexpensive and avoids training or evaluating an additional backward-policy network. A tractable backward policy enables analytic marginalization over next transitions, reducing conditional gradient variance. These computational advantages motivate our choice to keep the backward policy fixed. Extending the data-based formulation to jointly learn the backward policy is a possible direction, but is not investigated by our present results.

The support assumptions ensure that the local flow ratios and the sampling estimators used in our objectives are well defined. We choose a backward policy with positive probability on all valid backward transitions and formulate the score ratios on states with positive induced visitation probability.

3. Experimental scope

Our evaluation spans peptide generation, Ising distributions, and MNIST generation and inpainting, examining different aspects of the transferred objectives. We acknowledge that the complete data-pretraining/reward-post-training pipeline is demonstrated on the peptide benchmark, whereas the Ising and MNIST experiments evaluate data-based generative learning.

In regard to the earlier considerations regarding the backward policy, our uniform backward policy also provides broad coverage of partial observation patterns during training, supporting the inpainting capabilities demonstrated in our MNIST experiments.

Given your positive assessment of the technical soundness and presentation, as well as the strengths noted in the theoretical and experimental contributions, we hope that our clarifications address the main concerns underlying your overall recommendation. We would appreciate your reconsideration of the overall assessment in light of these clarifications and the revised manuscript.

[1] Bengio et al. GFlowNet Foundations. JMLR, 2023, Section 2.6.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

General Response

We thank the reviewers for their constructive feedback and for recognising the flow–score correspondence, clarity of presentation, and empirical benefits of data-based GFlowNet pretraining. We address concerns about novelty (LkC5, rMoL), behaviour cloning (uWxG, rMoL), fixed backward policies (LkC5), and the peptide benchmark (uWxG, rMoL). We also provide new derivations and two additional BC baselines.

1. Score Entropy vs. Policy-Based Behaviour Cloning (uWxG, rMoL)

Conventional behaviour cloning (BC) maximises the likelihood of demonstrated state-action transitions under a forward policy. Our score-entropy objectives address the same data-learning problem using a score-based parametrisation and a different objective. Under matched supervision, they recover the same empirical forward policy in the unrestricted tabular setting; we derive this relationship in the revised appendix.

Importantly, analytic DSE (Eq. 8) Rao–Blackwellizes sampled DSE′ (Eq. 9) by marginalising over compatible next transitions, reducing conditional gradient variance. This is a standard variance-reduction principle, not unique to score entropy. We show that conventional BC can be marginalised analogously to obtain analytic BC, and provide the objective and gradient relationships in the appendix.

2. Why We Keep the Backward Policy Fixed (LkC5)

As discussed in Sec. 3.1, a fixed backward policy corresponds to the prescribed noising process in discrete diffusion. Fixing it does not restrict terminal-distribution expressivity: with sufficient backward support, any target distribution admits a consistent forward policy. Our uniform backward policy satisfies this condition.

The choice also has practical advantages. Uniform backward sampling requires no additional learned network, and its tractable marginals enable analytic DSE and Rao–Blackwellization, removing variance due to sampling the next transition. Learned backward policies may offer optimisation benefits, but fixing a tractable one is a deliberate trade-off, not a restriction on the target distributions we can model. We will clarify this in Secs. 3.1 and 4.1.

3. Novelty and Relation to Prior Work (LkC5, rMoL)

We agree that connections between GFlowNets and diffusion predate our work (Zhang et al., 2023; Carter et al., 2026). We do not claim to discover this connection. Zhang et al. (2023) also introduce MLE-GFN for data-based learning; our contribution differs in deriving local score-entropy objectives and their analytic and transition-sampled estimators for construction DAGs. Our contribution is an explicit local flow-ratio formulation, a characterisation of GFlowNet training objectives in terms of these scores, and the transfer of data-based score entropy to GFlowNet construction DAGs.

Score entropy itself originates with Lou et al. (2024) for discrete diffusion. Our work derives the corresponding GFlowNet objectives and uses them for direct pretraining from terminal samples, without requiring reward evaluations or a proxy reward model. Pan et al. (2024) instead study outcome-conditioned pretraining and reward adaptation. We learn one specified terminal data distribution and subsequently post-train the same GFlowNet with standard reward-based objectives, without a separate adaptation model. We will sharpen these distinctions in the related-work section.

4. Additional Experiments and Setup Clarification (uWxG, rMoL)

We add conventional BC and analytic BC as baselines alongside analytic DSE, using the same data, architecture, and reward-based post-training protocol. Across 10 paired seeds, analytic DSE improves pretraining high-reward fraction over conventional BC by 5.39 percentage points, while analytic BC improves it by 8.08 ± 1.00 percentage points (paired mean ± SD). Analytic BC also exceeds DSE on this metric. The differences are substantially smaller after post-training. These results support the practical benefit of analytic marginalisation, including for conventional BC, rather than claiming DSE universally outperforms BC.

Finally, we clarify that our peptide experiments adapt the length-10 setting of Dall'Antonia et al. (2026), building on the broader sequence-design setting of Jain et al. (2022); they do not reproduce Jain et al.'s benchmark exactly. We use a fixed reward oracle and pretrain on 67 high-reward examples selected from 212 candidates.

MethodHR% preNovel preHR% postNovel post
DSE67.34±1.55154±1888.58±0.941802±19
Sampled BC61.95±2.10188±2687.62±0.961785±20
Analytic BC70.03±2.32160±2489.00±0.991808±18

We thank the reviewer for the constructive feedback on the peptide benchmark and the need for a behavior-cloning baseline. We clarify the experimental protocol below and report the additional baseline results.

1. Peptide benchmark and length restriction

Our peptide experiments build on the antimicrobial-peptide generation setting of Jain et al. (2022), adopting the shorter sequence length (up to 10 amino acids) and random-forest-based reward formulation used by Dall'Antonia et al. (2026). We further adapt the construction graph to permit arbitrary-position residue insertion. We have updated the manuscript's attribution to clarify these differences from the original Jain et al. benchmark.

The experiment uses a frozen surrogate reward oracle consisting of five pathogen-specific random-forest classifiers. For a peptide xx, we define its antimicrobial activity score as

pAMP(x)=max⁡jpj(x).p_{\mathrm{AMP}}(x)=\max_j p_j(x).

We classify a peptide as high-reward if pAMP(x)≥0.95p_{\mathrm{AMP}}(x)\geq 0.95, corresponding to the maximum reward R(x)=1R(x)=1. Our pretraining dataset comprises 67 high-reward peptides selected from 212 training candidates. All 212 oracle evaluations used for this selection are included in our reward-query accounting. The DSE pretraining objective itself does not require reward evaluations.

Unlike autoregressive sequence construction, our insertion-based construction DAG permits multiple construction trajectories to reach the same peptide. At length 10, a terminal peptide admits up to 10!10! action-labelled construction trajectories, providing a nontrivial construction space even for short sequences. The length restriction follows the chosen surrogate-oracle benchmark rather than a limitation of our method, which applies to arbitrary finite construction DAGs. Extending the experiments would primarily require a suitable reward oracle for longer sequences.

2. Behavior-cloning baselines

We have added both the requested conventional behavior-cloning (BC) baseline and a stronger Rao–Blackwellized BC baseline. The comparisons use matched data, architectures, optimization budgets, backward trajectories, and Replay-TB continuation.

Conventional BC supervises a single sampled next action. Analytic BC instead uses its exact conditional distribution under the backward policy. Both optimize the same expected likelihood objective.

Analytic BC improves over conventional BC by 7.78 ± 1.84 percentage points after pretraining and 8.08 ± 1.00 percentage points after the log⁡Z\log Z warmup. These results demonstrate that analytic marginalization provides a substantial practical benefit under the tested optimization budget.

After reward-based post-training, all three methods attain similar performance. Analytic BC exceeds analytic DSE by 0.42 ± 1.16 percentage points, indicating a small observed difference in this setting and consistent with their close mathematical relationship.

We have included these comparisons in Appendix E, alongside a derivation in Appendix D showing that BC and DSE recover the same population-optimal forward policy in the unrestricted tabular setting. Their objectives and parametrisations nevertheless differ, particularly in how forward-policy normalisation is enforced.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.