acceptodds
Under review as a conference paper at ICLR 2027

Mixture of GFlowNets

16 views

Abstract

Generative Flow Networks (GFlowNets) learn a policy that samples compositional objects in proportion to an unnormalized reward . This makes them useful wherever many diverse, high-reward candidates are needed and oracle queries are expensive, as in AI for Science. In practice, a single GFlowNet often fails to cover all modes of a complex distribution, and the obvious ways to scale compute saturate: larger batches and data-parallel gradient averaging draw every trajectory from one, possibly biased, policy, so extra rollouts rediscover known modes instead of finding new ones. We explore whether this mode-bias problem (mode collapse in the extreme) can be addressed with a population of asynchronously trained policies which divide the state space between them. **Mixture of GFlowNets (MoG)** replaces the single forward policy with a _community_ of independent GFlowNets coupled through one jointly trained soft classifier, which assigns each member its own region of the state space through reward shaping. Mixing the members under their learned partition functions provably recovers the target . **MoG exchange (MoGX)** also lets the classifier route sampled trajectories: a trajectory the classifier assigns to another member is sent to that member and trained on there, with no importance sampling. We evaluate both methods by keeping the environment-interaction budget fixed while scaling the community size, turning parallel _compute_ into mode _coverage_ without extra oracle calls. On HyperGrid, across six reward landscapes and communities of up to , MoG and MoGX keep finding new modes as grows, while data-parallel training saturates. On a genomic ideotype-design task, MoGX covers % of the optima at , against % for independent agents and % for data-parallel training. MoG also has a lower per-iteration synchronization cost than data-parallel training.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Generative Flow Networks, Population Based Training, Asynchronous Amortized Sampling

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_mfpVOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes Mixture of GFlowNets (MoG), which uses a learned classifier to partition the reward distribution across multiple GFlowNets. The authors propose "Mode Bias", a postive feedback loop that leads to mode collapse. It's a very insightful and important diagnosis for GFlownet. Further, the MoGX introduces classifier-guided trajectory exchange to improve exploration. Experiments demonstrate improved mode coverage under a fixed sampling budget.

Critical strengths

The paper identifies and clearly formulates mode bias as an bottleneck in GFlowNet exploration, highlighting why simply increasing parallel sampling may fail to discover new modes. This is a valuable insight with practical application for Science. The proposed framework is simple and well-motivated. The experiments cover multiple reward landscapes and population sizes, and the improvements of MoGX are encouraging.

Critical weaknesses

First, the paper should better discuss recent approaches to mode collapse, for example, RapTB (Wang et al., ICML 2026), which explores credit assignment and diversity-aware replay, and ACE (Dall'Antonia et al., ICML 2026), which studies complementary exploration. Replay buffer can be a vital factor to mitigate mode collapse, such disscussion or experiments will better clarify the contribution of MoGX, especially given the reduced gains with SubTB in appendix.

The paper lacks direct evidence that learned partitioning and classifier-guided exchange are responsible for the improvements. Comparisons with fixed partitions or random exchange, along with an analysis of mode overlap between policies, would strengthen the claims. The mixture-correctness result is valid under exact component convergence, but the joint classifier-policy training dynamics are not discussed. In particular, Appendix C derives classifier self-consistency using partition function priors, whereas the classifier seems to be trained using equally sampled components. The authors should clarify this discrepancy and discuss how imperfect component convergence and partition-function estimation affect mixture correctness.

Most experiments use HyperGrid-style environments. Additional experiments on structured or variable-length generation tasks, such as molecular generation (SMILES), would better demonstrate the generality and practical relevance of the method. Moreover, the task like mouelces generation will also provide additional metrics for evaluating the diversity (fingerprint similarity). I suggest such experiments for it will better support your claims.

Questions for authors

See weakness part. I'm wiling to raise my score if the authors address my concerns above.

5 comments

Thank you for the review, and for your openness to a higher score. You raised three concerns; we answer them in the order of your review. Several answers use new analyses of the paper runs. We ran the two ablations that you asked for, and we give the results below. We also implemented a molecule task. Its runs are waiting in the cluster queue, and we will post them here when they are complete.

W1. Related work and replay. Thank you for the references, which we now discuss in Sec. 2. RapTB (Wang et al., ICML 2026) works inside one learner. A rooted-prefix balance term sends credit to early prefixes. Its submodular replay refreshes the buffer for reward and diversity. ACE (Dall'Antonia et al., ICML 2026) trains an explorer GFlowNet together with the main GFlowNet. ACE uses divergent trajectory balance to push the explorer toward high-reward states that the main sampler under-represents.

The main GFlowNet of ACE trains on samples from the two GFlowNets. ACE mixes the two policies by their partition functions, but only to collect training data; the main GFlowNet alone must match the target. In both papers, then, a single sampler is responsible for the full target, while better credit assignment, better replay or one dedicated explorer helps it. MoG instead splits the target itself across KK peers that train in parallel, and the ZZ-weighted mixture is itself the sampler. The ideas are compatible, because each MoG component is an ordinary GFlowNet with its own shaped reward. A component could train with RapTB and refresh its buffer with SubM on autoregressive tasks, or pair with its own ACE explorer that searches the regions it under-samples.

About replay: the paper reports every result in two regimes, with a replay buffer (ρ=0.5\rho=0.5) and without one (ρ=0\rho=0). The methods have the same rank order in the two regimes. Thus, the gain of MoGX does not depend on replay. For a direct test, we added a diversity-aware replay baseline to the comparison. These runs are in the compute cluster queue with the others.

Our new step analysis identifies a specific cause for the smaller gains under SubTB. On Bitwise XOR, SubTB and DB reach a mode hit rate in the range of 0.55–0.82 within 600 steps, but TB needs 2–5k steps, depending on the reward. The gains of MoG are largest when the base objective is slow to take off, as TB is here. We have added this analysis to the appendix.

W2. Evidence that the learned partition and exchange cause the gains. We ran the two ablations that you suggested. They run 4 seeds of MoGX at K=2,…,64K = 2, \dots, 64 on the three compositional rewards and ideotype.

  • Frozen partition. fψf_\psi is a fixed random partition for both shaping and routing. We calibrated it so that every component owns a region. Its normalized entropy (0.5) matches the entropy of the learned partition.
  • Random destination. fψf_\psi is learned and still selects which trajectories leave a rank. But it sends them to a random other rank.

The frozen partition tests whether the partition must be learned. The random destination tests whether fψf_\psi must route the trajectories. We ran 384 new experiments: the two ablations, an unmodified MoGX control, and NoComm, with 4 seeds at K=2,…,64K = 2, \dots, 64 on 4 tasks. The control reproduces the MoGX of the paper. The table gives the modes at a common budget of 1.39×1081.39 \times 10^{8} trajectories for K=64K=64, and for K=32K=32 on ideotype (mean ± s.d. over 4 seeds).

Task (budget)NoCommMoGXFrozen partitionRandom destination
Bitwise XOR, K=64 (M modes)11.4 ± 0.4111.4 ± 0.6121.1 ± 2.7 (1.09×)121.1 ± 1.0 (1.09×)
Mult. Coprime, K=64 (M modes)21.3 ± 0.7101.5 ± 0.7101.2 ± 0.2 (1.00×)105.0 ± 0.3 (1.04×)
Cond. Multiscale, K=64 (M modes)18.5 ± 0.272.0 ± 2.070.5 ± 0.7 (0.98×)76.5 ± 0.6 (1.06×)
Ideotype, K=32 (% of optima)20.7 ± 0.729.1 ± 0.833.7 ± 2.3 (1.16×)23.7 ± 2.3 (0.81×)
Ideotype, K=64 (% of optima)28.9 ± 0.436.5 ± 1.441.2 ± 0.6 (1.13×)32.3 ± 0.2 (0.89×)

Partition plus exchange drives the gain. At K=64K=64, all three MoGX variants find 10–11× the modes of NoComm on Bitwise XOR, about 5× on Mult. Coprime, and about 4× on Cond. Multiscale. This holds even when the partition is random and fixed. So the gain comes from the mixture structure of MoGX, and it does not depend on a carefully trained classifier.

A fixed partition is a simpler variant that also works. On 3 of 4 tasks (considering the full KK sweep), the fixed random partition matches the learned one or does better. However, the fixed partition also has a cost: a random classifier leaves more than half of the 64 components with no states, so we had to calibrate it to a target entropy first. Instead, the learned partition is balanced with no tuning. We added the fixed partition to the paper as a simpler option; thank you for suggesting this.

The learned classifier matters for routing. On HyperGrid, the policies rarely find the same mode (ρ≤1.2\rho \le 1.2 for every MoGX variant). There, routing has no overlap to remove, and random routing is as good as MoGX (+4% to +9% at K=64K=64). Ideotype is different, because policies can find the same optima. There, guided routing finds 23% more optima than random routing at K=32K=32 and 13% more at K=64K=64. With random routing, the policies overlap more than in any other arm (see below). We added this analysis in the appendix.

Mode overlap. We measured the redundancy ρ=∑k∣Sk∣/∣⋃kSk∣\rho = \sum_k |S_k| / |\bigcup_k S_k| from the per-rank logs of every K=64K=64 run. This value is the mean number of agents that found each mode (1 = disjoint, 64 = identical). On ideotype (1,024 optima, where some overlap is unavoidable), it is 7.7 for NoComm, 5.0 for MoG, 3.3 for MoGX, and 12 for DDP, without replay. For MoGX, we count each optimum for the agent that sampled it, before the exchange. Because MoG has no exchange, its decrease from 7.4 to 5.0 is direct evidence that the learned partition sends agents to different optima. Moreover, the ablations you suggested show that guided routing also keeps the agents apart. The pairwise Jaccard index between the optima of two agents is 0.11 for MoGX, 0.43 for NoComm, and 0.53 for the random destination (ρ=12.2\rho = 12.2). The frozen partition is in between (Jaccard 0.21). On Bitwise XOR (2322^{32} modes), even independent agents rarely find the same mode (ρ=1.3–1.4\rho = 1.3\text{–}1.4), so overlap cannot explain the gap. However, the difference is due to learning speed. Each agent samples the same number of trajectories, and its mode hit rate jumps once it learns the parity rule. MoG agents learn it sooner: they reach a hit rate of 0.5 after 2,900 steps at K=64K=64, against 4,600 for independent agents. MoGX components also train on the high-reward trajectories that other components route to them, so they reach a hit rate of 0.5 after just 675 steps.

Appendix C and the classifier prior. Thank you for the careful reading; you are correct. Each rank averages its classifier loss over its own batch, and the ranks then average their gradients. So every rank has the same weight, and the class prior is uniform. For β=0\beta = 0, the Bayes-optimal classifier is fk∗(x)=πk(x)/∑jπj(x)f^{\ast}_k(x) = \pi_k(x) / \sum_j \pi_j(x). In MoGX, fψf_\psi trains on the post-exchange batch, with labels from the receiving rank, so πk\pi_k is replaced by that batch's distribution. Mixture correctness does not depend on priors or labels. The proposition holds for any pointwise assignment fψ(x)∈ΔK−1f_\psi(x) \in \Delta^{K-1}, because ∑kZkR~k/Zk=R\sum_k Z_k \tilde{R}_k / Z_k = R.

The self-consistency argument in App. C changes in light of this. With the old prior, every classifier was a fixed point, so the argument did not predict where training goes. With the uniform prior, a fixed point needs one of two conditions. Either each state belongs to one component, or all the components that share a state have equal ZkZ_k. If they do not, the classifier moves the shared mass to the component with the smaller ZkZ_k. So at exact convergence, shared states move toward the components with less reward mass. We have updated App. C with this result; thanks for pointing it out.

Imperfect convergence and partition-function estimation. We have measured both.

During training, ∑kZk∗=Z\sum_k Z^{\ast}_k = Z for any classifier state. Thus, Δ(t)=log⁡∑kZk(t)−log⁡Z\Delta(t) = \log \sum_k Z_k(t) - \log Z tests whether the learned partition functions keep up with the moving classifier. Across all paper MoG/MoGX runs, from 5k steps on, MoG has Δ∈[−0.04,0]\Delta \in [-0.04, 0] nats at every KK. This is the same range as a single GFlowNet. MoGX has a constant +0.09–0.10 nats.

We also computed each component's exact target by passing all 2.8×1082.8 \times 10^{8} states through the classifier. This is too expensive to repeat along a run, so we did it on 16 final checkpoints (original, K=4,8,16K = 4, 8, 16, and 6464, 2 seeds). For MoG, each learned log⁡Zk\log Z_k matches its target to 0.005 nats RMS. For MoGX, every log⁡Zk\log Z_k is about 0.1 nats high, which is the same offset as above. Because all components share this shift, it cancels in the mixture weights wk=Zk/∑jZjw_k = Z_k / \sum_j Z_j, and the weights stay within 0.006 of exact.

The mixture is also robust to imperfect convergence. In every MoGX checkpoint, the median component's KL divergence to its own target is 1.6–22× the value of sampling noise. The mixture's KL to the true target is within 1.35× of that value, because the component errors cancel in the ZZ-weighted sum.

We also queued in the compute cluster a stress test: the mixture KL along training, with lrψ/lrZ\mathrm{lr}_\psi / \mathrm{lr}_Z from 10−310^{-3} to 11 (the paper uses 10−210^{-2}). We will post the result here.

W3. Structured, variable-length tasks. We agree, and we built such a task. The environment assembles molecules from 72 fragments with ordered attachment instructions (up to six fragments). The reward is RDKit QED. The environment has 201 action IDs, up to 129 legal actions per state, and trajectories of variable length. The runs of MoG, MoGX, NoComm, and DDP are in the compute cluster queue; we will post the results and analysis here as soon as they complete.


To summarize, the ablations show that partition plus exchange drives the gain, even with a fixed random partition. The learned classifier matters for routing: on ideotype, guided routing finds up to 23% more optima than random routing and keeps the agents apart. The results show that the mixture is correct during training and under imperfect convergence. The replay baseline and the molecule task are in the compute cluster queue. We will post each result in this thread when it is complete.

Reviewer_PtYtOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Mixture of GFlowNets (MoG), which trains several GFlowNets to cover different parts of the same reward distribution. A classifier learns together with the GFlowNets and assigns soft weights to terminal states. These weights shape each model’s reward and help it focus on different parts of the state space. The paper shows that mixing the models using their partition functions recovers the original target distribution if each model learns its shaped target distribution exactly. MoGX also uses the classifier to send sampled trajectories to other models for training. The methods are tested on six HyperGrid reward landscapes and a genomic ideotype-design task, with up to 64 models and a fixed total trajectory budget. The results show better mode coverage than independent and data-parallel training in several settings. The paper also tests the effects of replay and training objectives, and measures mixture accuracy and communication costs.

Critical strengths
  • The method offers a clear way for several GFlowNets to learn different parts of the same reward distribution. The classifier shapes each model’s reward, while the partition-function-weighted mixture recovers the original target distribution if each model learns its shaped target exactly. This is a useful design because it allows models to focus on different regions while keeping the same overall sampling target.
Critical weaknesses

None

Questions for authors
  • Please clarify how the KL divergence in Figure 4 is computed. Does it use probabilities normalized over the 1,024 modes, include an additional off-mode category, or cover all terminal states?
  • Are there any constraints in the genomic task that prevent uniform sampling of terminal states? If no further constraints apply, could you provide this baseline under the same oracle budget and explain what the comparison means for the claimed gain in sample efficiency?
4 comments

Thank you for the positive assessment. Both of your questions made the paper stronger.

Q1. KL in Fig. 4. The KL is over 1,025 categories: the 1,024 modes, plus one off-mode category for all other terminal states. We now state this in the caption.

We also computed the KL that a perfect mixture would give with the same sample sizes: 1.30×10−41.30 \times 10^{-4}. The reported KL (1.51×10−41.51 \times 10^{-4}) is within 1.2× of this value. Thus, the mixture is as close to the target as this sample size can measure. We will add this reference value to the paper.

Q2. Uniform sampling on ideotype. No constraint prevents it: every vector in {0,1}20\lbrace 0,1 \rbrace^{20} is a valid terminal state. At B=10B=10, a uniform sampler hits an optimum with probability 2−102^{-10}. It finds all 1,024 optima within our 2.62×1082.62 \times 10^{8}-call budget. It matches MoGX at K=64K=64 (402 optima) after 5.2×1055.2 \times 10^{5} calls. A flat-policy rollout of the same DAG (an untrained GFlowNet) needs 1.9×1061.9 \times 10^{6} calls.

This means that at B=10B=10, the result supports the ranking of coordination schemes under a shared budget (MoGX > MoG ≈ NoComm > DDP), but it does not support a sample-efficiency claim. We will add the two baselines to the figure. We will present B=10B=10 as a controlled testbed with a known number of optima.

To test sample efficiency, we ran B=20B=20 (2402^{40} states, 2202^{20} optima) with 4 seeds. At the same budget, MoGX at K=64K=64 finds 1,364 ± 47 optima. This is 5.5× more than uniform sampling (250) and 28× more than the flat policy (49). The order MoGX > MoG > NoComm ≫ DDP holds. We will add B=20B=20 and the two baselines in the camera-ready version.

All my questions have been addressed. I have observed that existing GFlowNet methods can suffer from mode collapse. This paper identifies a cause of this problem, and I think this paper addresses that problem reasonably well. I support acceptance of this paper. As a minor suggestion that does not affect my recommendation, some figures are inconvenient positions. I hope this will be addressed in the revised version.

In the revision, each figure now sits next to the text that refers to it.

Thank you for your support; if our answers have resolved your questions, we would be grateful if you would consider updating your score to reflect your support for acceptance. If any concern remains, we are glad to answer it before the discussion ends.

Thank you for your response. All of my concerns have been addressed, and I am raising my score. As a minor issue, my earlier comment about figure placement referred to the figure between paragraphs in the Conclusion section. This is only a presentation suggestion and does not affect my recommendation.

Reviewer_evY6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Mixture of GFlowNets (MoG) and MoG exchange (MoGX) to address mode bias and incomplete mode coverage in GFlowNets. A community of KK independent GFlowNets is trained in parallel and coupled through a single jointly trained soft classifier fψf_\psi that partitions the terminal state space, so each component kk is trained on a shaped reward R~k(x)=R(x)⋅fψ(x)k\tilde{R}_k(x) = R(x) \cdot f_\psi(x)_k. The authors show that the ZZ-weighted mixture of the components recovers the target p(x)∝R(x)p(x) \propto R(x) exactly, assuming each component converges to its shaped target. MoGX additionally routes a fraction of on-policy trajectories to the component preferred by the classifier, which needs no importance sampling correction since GFlowNet objectives are off-policy. With a fixed total environment-interaction budget and KK up to 64, MoG and especially MoGX keep discovering new modes on six HyperGrid landscapes (three of them new) and a genomic ideotype design task based on a real wheat genetic map, while independent agents (NoComm) and data-parallel training (DDP) saturate or degrade. MoG also has a lower synchronization cost than DDP.

Critical strengths

Clear motivation with a simple and correct theoretical result: The paper clearly explains why increasing the batch size or data-parallel gradient averaging is not enough for mode coverage, since all trajectories are still drawn from essentially one reachable set. Learning a soft partition so that each GFlowNet specializes in a subregion is a novel idea, and the mixture-correctness proposition is clean and correctly proved. Fig. 4 also nicely shows that biased, sparsely supported components recombine into a near-uniform mixture.

Well-designed scaling experiments: Fixing the total trajectory budget while scaling KK is the right comparison when oracle calls are expensive. The authors report both the replay and pure on-policy regimes, include communication cost tables, and introduce three new compositional HyperGrid rewards (bitwise XOR, multiplicative coprime, conditional multiscale) that expose the limits of single-policy and independent-population methods better than the classic corner/sparse rewards. The MoGX vs MoG ablation is also informative.

Critical weaknesses

Limited environment diversity: Both HyperGrid and the ideotype task use fixed-length trajectories on a product space with simple local actions, and the authors themselves note that graph or sequence generation with large action spaces and variable lengths is not tested. This is the main limitation for me, since the paper is motivated by AI for Science, where molecule or biological sequence design would be much more convincing. It is unclear whether the classifier can still learn a useful partition in these settings.

The scaling results mix several effects: Since the total budget is fixed, larger communities get far fewer gradient steps per agent (4k steps at K=64K=64 vs 128k at K=2K=2), so the scaling curves mix the benefit of specialization with the under-training of each specialist. On the sparse reward, no method benefits from larger KK, which weakens the general 'more parallel compute gives more mode coverage" claim. In addition, KK is chosen a priori with no mechanism to retire stalled components or grow the population, and nothing guarantees that the classifier gives every component a meaningful region. On sparse landscapes, a few components that find modes early could dominate the classifier loss while the rest end up on empty regions. On the theory side, the mixture proposition assumes each component exactly reaches its shaped target, but the shaped reward changes every time the classifier updates. If log⁡Zk\log Z_k adapts more slowly than fψf_\psi shifts, the TB objective is temporarily biased, and thepaper does not discuss the relative learning dynamics of ψ\psi and log⁡Zk\log Z_k. The empirical KL check is reassuring, but it does not replace this analysis.

Wall-clock cost of MoGX: At K=64K=64 the exchange step alone takes about 760ms of the 1477 ms per iteration, so MoGX is slower per step than both NoComm (713 ms) and DDP (1249 ms). MoGX is better per oracle call, but it is unclear whether it is still better under a fixed wall-clock budget, which matter for the "scaling compute" narrative.

Minor: EGFN is run with N(0,1)\mathcal{N}(0, 1) mutation noise over entire weight tensors and a much larger batch size than its original design, which collapses its population to nearly deterministic policies. Smaller perturbation scales (e.g. σ∈{10−2,10−3}\sigma \in \{10^{-2}, 10^{-3}\}) should be tested for a fair comparison. Some algorithmic details (exchange selection rule, clipping of the shaping term, interaction of shaping with replay) are only clear in the appendix, and the clipping of log⁡fψ\log f_\psi to [−20,0][-20, 0] technically breaks the exact recovery result, which should be mentioned even if the error is negligible.

Questions for authors

Q1 Experiments on molecule or sequence design. Have the authors tried the framework on a molecule or sequence task, even a modest one like QM9 or a short peptide design problem? I would like to know whether the classifier can still learn a useful soft partition when trajectories have variable length and the action space is large. This is the question that would most affect my assessment.

Q2 Wall-clock comparison. Under a fixed wall-clock budget instead of a fixed trajectory count, how does MoGX compare to DDP and NoComm in mode discovery, given the current exchange overhead?

Q3 Mode allocation across components. In the K=64K=64 experiments on Bitwise XOR or ideotype design, how evenly are the discovered modes distributed among the 64 agents? I would like to see the component utilization, and how sensitive it is to the entropy coefficient β\beta on the classifier and to the exchange fraction N/∣B∣N/|B|.

Q4 MoG vs MoGX. On the ideotype environment, MoG alone basically matches NoComm while MoGX pulls ahead. Is this because the linkage structure makes reward shaping alone insufficient without trajectory routing? Similarly, is the larger MoGX-MoG gap in the pure on-policy regime mainly because early trajectories are more valuable when there is no replay?

Q5 Adaptive community size (suggestion for future work). Can KK be made adaptive, for example by monitoring the per-component discovery rate or the entropy of the classifier?

4 comments

Thank you for a detailed and constructive review. Each of your questions led to a new analysis. Together, these analyses make the central claim of the paper stronger: the learned partition makes each agent learn faster.

W1 / Q1. Molecule or sequence tasks. We agree that this question is the most important, so we built a new environment for this. The environment assembles molecules from 72 fragments with ordered attachment instructions (up to six fragments), with RDKit QED reward. The environment has 201 action IDs, up to 129 legal actions at a state, and trajectories of variable length. We have requested compute resources to run MoG, MoGX, NoComm, and DDP in our cluster.

When the runs are complete, we will post them here with the answer to your exact question. Does fψf_\psi learn a non-trivial partition (normalized entropy, share per component)? How does coverage compare across methods? Several assembly histories can give the same molecule, so we will count distinct molecules (canonical SMILES). As mfpV suggests, we will also report fingerprint diversity (mean pairwise Tanimoto similarity of Morgan fingerprints).

W2. Specialization vs. under-training. This is a sharp observation, indeed. Our data separates the two effects clearly: every run trained well past the paper budget, so we can compare all methods at matched optimizer steps per agent.

Under-training explains the decline of the baselines. At matched steps (14,700–19,400 per agent, depending on the reward), KK has almost no effect on the cumulative hit rate (distinct modes found per trajectory sampled) of NoComm, DDP, MoG, and MoGX. This is true for the three compositional rewards: the ratio K=64/K=2K{=}64/K{=}2 of the hit rate is in the range 0.86–1.11 in all cases. At K=64K=64 and the paper budget, we read NoComm and DDP before TB takes off. Thus, their ratio at the paper budget is in the range 0.008–0.14.

Specialization explains the lead of MoG and MoGX. MoGX reaches a hit rate of 0.5 in 550–2,825 steps at every KK, and NoComm needs 4,375–10,075 steps, depending on the reward. MoG gets much faster at large KK, while NoComm stays flat. For example, on Bitwise XOR (replay), MoG needs 5,000 steps to reach a hit rate of 0.5 at K=2K=2, and 2,925 steps at K=64K=64. NoComm needs 4,375 steps at K=2K=2 and 4,550 at K=64K=64. This dependence on KK is the specialization effect that you asked us to isolate. After all methods take off, MoG and MoGX stay 4–10% ahead at K=64K=64. When oracle calls are expensive, the important property is that an agent learns in a few hundred steps, not in several thousand. We will state the result in these terms.

Ideotype is the task closest to the AI-for-Science motivation. There, we trained every run about 30× past the paper budget, and the 1.24–1.46× lead of MoGX over NoComm holds at every step count. At the paper budget, K=64K=64 already has 73–97% of the optima that it finds by the end of the run.

Sparse. We agree, and we will state the claim more precisely. The number of trajectories limits Sparse: the hit rate is 6–9×10−76\text{–}9 \times 10^{-7} per trajectory at every KK and every step. Partitioning helps when policy bias is the bottleneck. It cannot help a policy to find a first mode.

Empty components. At K=64K=64, all 64 agents find modes in every run (see Q3 answer). The largest MoGX component holds 13% of the ideotype optima found. For NoComm, this value is 22%.

W2. Learning dynamics of ψ\psi and log⁡Zk\log Z_k. The structure of MoG gives a direct test. Because ∑kfψ(x)k=1\sum_k f_\psi(x)_k = 1, the shaped targets satisfy ∑kZk∗(t)=Z\sum_k Z^{\ast}_k(t) = Z for any classifier state ψt\psi_t. Thus, Δ(t)=log⁡∑kZk(t)−log⁡Z\Delta(t) = \log \sum_k Z_k(t) - \log Z is zero if and only if the learned partition functions, in aggregate, are up to date with the classifier. We calculated Δ(t)\Delta(t) against the exact log⁡Z\log Z for every paper MoG/MoGX run (12,348 rank runs).

From 5k steps on, MoG has Δ∈[−0.04,0]\Delta \in [-0.04, 0] nats at every KK and reward, which is the same range as a single GFlowNet. By the end of the runs, ∣Δ∣≤0.013|\Delta| \le 0.013. (On Sparse, no method puts mass on the targets, NoComm included.) This result holds while fψf_\psi still changes a lot. At K=64K=64 in MoG, its normalized entropy increases from about 0.1 to 0.6–0.7, and Δ\Delta stays flat. With our learning rates (lrψ=10−3\mathrm{lr}_\psi = 10^{-3}, lrZ=0.1\mathrm{lr}_Z = 0.1), log⁡Zk\log Z_k does not lag behind fψf_\psi.

Per component, we enumerated all 2.8×1082.8 \times 10^{8} states of the saved checkpoints at K=4,8,16K = 4, 8, 16, and 6464. This gives each exact moving target log⁡∑xR(x)fψ(x)k\log \sum_x R(x) f_\psi(x)_k. The learned log⁡Zk\log Z_k of MoG matches this target with an RMS error of 0.005 nats. MoGX has a common offset of +0.09–0.10 nats in every component. The offset cancels in the softmax, so the mixture weights stay within 0.006 of exact.

We also queued in the cluster a stress test of this point: the mixture KL along training, with lrψ/lrZ\mathrm{lr}_\psi / \mathrm{lr}_Z from 10−310^{-3} to 11 (the paper uses 10−210^{-2}). We will post the result here.

W3 / Q2. Wall-clock. We plotted every run again against wall-clock time. All methods ran on identical nodes. End to end, MoGX is 1.1–1.5× slower per iteration than NoComm on HyperGrid (1.8–2.0× on ideotype).

At K=64K=64, MoGX keeps its advantage. On ideotype, MoGX leads for the whole run. At 11 h, when NoComm reaches its trajectory budget, MoGX has about 1.25× the optima of NoComm and 1.65× those of DDP. On the HyperGrid compositional rewards, MoGX leads NoComm and DDP for the first 1.5–4 h, depending on the reward.

MoGX needs a certain wall-clock time to reach the trajectory budget of the paper. If the baselines get the same time, MoGX finds 1.8–7.4× more modes than the best baseline. For much longer HyperGrid runs (24 h), MoG is the best method. It keeps the benefit of the partition without the cost of the exchange. In these long runs, MoGX is at 0.82–0.95× NoComm.

We have added the two wall-clock figures to the appendix.

Q3. Mode allocation at K=64K=64. We analyzed all 64 per-rank histories of each run.

No component is empty in any run. On ideotype, the weakest MoGX agent still holds 6–7 of about 400 optima, and no component dominates.

On Bitwise XOR at the paper budget, 2–5 NoComm or MoG agents hold 10–40× the median (Gini 0.50–0.56). The full histories show that staggered take-off, not a takeover by a few components, drives this. The Gini peaks at 0.7–0.8 at about 1–2k steps and decreases to 0.02–0.10 by the end. Thus, no agent stalls.

MoGX agents take off together (Gini 0.08–0.10 at the paper budget). On ideotype, the allocation of MoGX is less even (Gini 0.23–0.24, against 0.12–0.14 for NoComm and MoG), because some agents find few optima. But no agent dominates: the largest holds 13% of the optima found, about the same as MoG and less than NoComm (16–19%).

The overlap results show the effect of the partition. The mean number of agents that found each ideotype optimum is 7.4 for NoComm, 5.0 for MoG, and 3.3 for MoGX. MoG has no exchange, so its reduction comes from the learned partition alone.

About sensitivity: the paper runs use β=0\beta=0 and N=128N=128 of ∣B∣=1024|B|=1024. We swept both at K=64K=64 on Bitwise XOR (new runs, on-policy, 4 seeds per setting): N∈{16,32,64,256,512}N \in \lbrace 16, 32, 64, 256, 512 \rbrace (1.6–50% of ∣B∣|B|) at β=0\beta=0, and β∈{±0.01,±0.05,±0.1}\beta \in \lbrace \pm 0.01, \pm 0.05, \pm 0.1 \rbrace at N=128N=128. β\beta changes the classifier, but not discovery. The normalized entropy of the classifier peaks at 0.35 for β=−0.1\beta=-0.1 and at 0.15 for β=+0.1\beta=+0.1, against 0.20 for β=0\beta=0. At the paper budget, every β\beta finds 0.95–1.01× the modes of the paper MoGX runs, with the same allocation (Gini 0.10–0.12).

NN matters more, but the best range is wide. N=64N=64 is within 5% of the paper result. N=32N=32 and N=256N=256 lose 12–14%. N=16N=16 and N=512N=512 lose 27–28%. Even so, N=16N=16 finds 10× the modes of MoG (N=0N=0). No agent is empty in any of the 48 runs. No agent holds more than 3.9% of the modes (1/K=1.6%1/K = 1.6\%), and the overlap stays low (ρ=1.09–1.13\rho = 1.09\text{–}1.13).

Q4. MoG vs MoGX. On ideotype, MoG divides the space: each optimum is found by 5.0 agents on average, against 7.4 for NoComm. But each MoG agent finds fewer optima (25, against 37 for NoComm), so the union stays the same (320 vs 318 for NoComm). The exchange turns this division into coverage: the union of MoGX increases to 402. Our runs cannot isolate the role of linkage.

About the two regimes: at K=64K=64, the MoGX/MoG ratio is similar with and without replay. The values are Bitwise XOR 12.9× vs 13.4×, Cond. Multiscale 2.30× vs 1.96×, and ideotype 1.25× vs 1.24× (with replay vs without). We will make the caption precise. Your intuition about early trajectories agrees with the step analysis: the main effect of MoGX is a much earlier take-off.

Q5. Adaptive KK. Thanks, this is indeed a promising direction. Our logs suggest which signal to use. With NoComm and MoG, agents take off one by one, between about 600 and 5,000 steps, and every agent takes off by the end (Q3). Thus, a low discovery rate early in training does not mean that a component has stalled, and a rule that retires slow components must allow for this delay. MoGX agents take off together, so the exchange already removes most of this delay.

Minor: EGFN mutation scale. We ran the official mutation operator on our architecture at σ∈{1,10−1,10−2,10−3}\sigma \in \lbrace 1, 10^{-1}, 10^{-2}, 10^{-3} \rbrace. A smaller σ\sigma delays the collapse, but it does not prevent it. Collapse occurs after about 1/σ21/\sigma^2 generations: 10–20 at 10−110^{-1}, 1,000–2,000 at 10−210^{-2}, and about 10510^5 at 10−310^{-3}. With selection, σ=10−2\sigma=10^{-2} collapses at about generation 100, and a run lasts 600–107,000 generations. σ=10−3\sigma=10^{-3} stays diverse because the population barely moves from its random initialization. Thus, it sends to the replay buffer as many distinct states and modes as a population with no mutation.

On Original and Sparse, the population finds at most 6 modes in 14M samples at any σ\sigma. The full sweep at K=64K=64 on Bitwise XOR is queued on our cluster. We will post the modes that the full population finds, and the modes that the star alone finds.

Minor: clipping. The e−20e^{-20} floor contributes at most 6×10−56 \times 10^{-5} of the target of any component (2×10−62 \times 10^{-6} in every MoGX checkpoint). We will state this next to the proposition. The exchange selection rule and the interaction of shaping with replay move into the main text.


To summarize: specialization makes each agent learn faster, and MoGX keeps its lead on ideotype under every budget that we measured. Also, the mixture stays correct throughout training. Regarding the molecule task, we will post the results here as soon as possible.

AuthorsOpenReview ↗

More experiments

Here are the results of the experiments we promised.

Q1. Molecule task. The environment builds a molecule from 72 fragments, with up to 6 fragments and at most 27 ordered instructions. It has 201 action IDs and up to 129 legal actions per state. Trajectories have 14–15 steps on average and 28 at most. The reward is QED; a mode is a distinct canonical SMILES with QED≥0.8\mathrm{QED} \geq 0.8. We ran MoG, MoGX, NoComm and DDP at K=2,…,64K = 2, \dots, 64, on-policy and with replay, with 4 seeds, for 16.8M sampled molecules each.

Does fψf_\psi learn a useful partition here? Yes. We measure how sure the MoGX classifier is about each agent's own samples. We use the mean assignment entropy divided by its maximum, log⁡K\log K, so 0 means certain and 1 means uniform. This value is below 0.03 up to K=16K=16, 0.11 at K=32K=32 and 0.23 at K=64K=64. The partition is also balanced, which answers your concern about empty components. At K=64K=64, every MoG or MoGX agent finds between 0.66% and 2.6% of the distinct high-QED molecules of its community (1/K=1.6%1/K = 1.6\%). The learned log⁡Zk\log Z_k also divide the reward mass evenly: their mean over agents is within 0.31 nats of log⁡Z−log⁡K\log Z - \log K, at every KK. Thus, variable length and a large action space do not hurt the classifier.

Discovery. At K=64K=64, MoG and MoGX both find more distinct high-QED molecules than NoComm, paired by seed:

K=64K=64, ratio to NoComm (over 4 seeds)on-policy (mean [min, max])replay (mean [min, max])
MoG1.037 [1.034, 1.042]1.025 [1.022, 1.027]
MoGX1.046 [1.012, 1.074]1.028 [1.015, 1.048]
DDP1.023 [1.016, 1.038]0.997 [0.994, 1.000]

The margins are smaller than on HyperGrid because the set of molecules with QED≥0.8\mathrm{QED} \geq 0.8 is very large; there is little redundancy for the partition to remove. However, MoG and MoGX still lead at K=64K=64, so the specialist policies themselves are slightly better samplers.

W2. Learning dynamics of ψ\psi and log⁡Zk\log Z_k. We ran the promised stress test:

  • the HyperGrid original reward (1,024 modes), where we can enumerate all states,
  • MoG and MoGX at K=4K = 4, 16 and 64, with 2 seeds,
  • six (lrψ,lrZ)(\mathrm{lr}_\psi, \mathrm{lr}_Z) settings, with lrψ/lrZ\mathrm{lr}_\psi / \mathrm{lr}_Z from 10−310^{-3} to 10−110^{-1}.

In MoGX, every setting with a working classifier converges in the same way. Each log⁡Zk\log Z_k is within 0.5 nats of its moving target by iteration 600–2,666. This warm-up is longer when lrZ\mathrm{lr}_Z is smaller, and does not depend on the ratio. After that, each log⁡Zk\log Z_k stays within 0.26 nats in all 30 runs, and within 0.13 nats in 25 of them, even while the classifier is still sharpening. After convergence, the typical mixture weight is within 1.5% of its exact value at every KK.

Minor: EGFN mutation scale. We ran EGFN again with σ=10−2\sigma = 10^{-2} and σ=10−3\sigma = 10^{-3}. We report two counts: the modes of the community (as in the paper), and the modes in the samples of the gradient-trained star.

  • Bitwise XOR and Mult. Coprime: a smaller σ\sigma raises the EGFN count by at most 1.7×1.7\times. All of the gain comes from the near-random samples of the population. The count of the star does not change. MoGX finds 3.7×3.7\times more modes than the best EGFN at K=4K=4, and 1919–23×23\times more at K=32K=32.
  • Original: σ≤10−2\sigma \leq 10^{-2} roughly doubles EGFN at K=8K = 8–1616. Independent agents still find 2.62.6–6.8×6.8\times more.
  • Cosine: at K=64K=64, σ=10−2\sigma = 10^{-2} gives a community count near 1,100, but the star finds at most 1 mode. Those modes come from 63 mutating policies that sweep the space instead of a learned sampler.
  • Sparse: with σ=10−3\sigma = 10^{-3}, EGFN finds all 1,023 modes up to K=32K=32 (σ=1\sigma = 1: 52 at K=16K=16), while NoComm and MoGX find about 180. This is the point we discussed in the previous reply about the Sparse reward: the bottleneck is the first mode, and a partition cannot help with that. Our best guess is that a population near its random initialization fills the replay buffer of the star with broad exploratory samples, but we did not test this yet.

We added these results to the appendix. The paper now says that EGFN with a small σ\sigma is the strongest method on Sparse at every KK we tested.

The molecule task shows a sharp, balanced partition on a variable-length task with a large action space, and a lead for MoG and MoGX at K=64K=64. The stress test shows that log⁡Zk\log Z_k keeps up with fψf_\psi across a 100×100\times range of learning-rate ratios. We hope that these answers resolve your remaining concerns, and we are glad to answer more questions.

Comments on the paper

2 comments

We thank the reviewers for their feedback. They found the mode-bias diagnosis "very insightful and important" (mfpV) and the mixture-correctness result "clean and correctly proved" (evY6). They also found the fixed-budget scaling experiments "the right comparison when oracle calls are expensive" (evY6). Their questions asked for evidence on why MoG and MoGX work; the new analyses give this evidence. Every paper run trained for 24h and logged per-rank histories, so most of these analyses use the paper runs. We mark new runs as new, and the end of this note lists the runs that still wait for compute.

  • Specialization makes agents learn faster (evY6). At matched optimizer steps per agent, KK barely affects how many modes each trajectory finds. For NoComm, DDP, MoG and MoGX on the three compositional rewards, the mode hit rate at K=64K=64 is 0.86–1.11 times the mode hit rate at K=2K=2, depending on the reward. So the drop with KK in the paper's plots comes from fewer steps per agent, not from larger communities. MoGX reaches a hit rate of 0.5 in 550–2,825 steps at every KK, while NoComm needs 4,375–10,075 steps. The take-off of MoG gets faster as KK increases. After all methods take off, MoG and MoGX stay 4–10% ahead on HyperGrid. On ideotype, at every train step, MoGX finds 1.24–1.46× as many modes as NoComm.

  • Wall-clock (evY6). On ideotype, MoGX leads for the whole run (about 1.25× NoComm at 11 h). On HyperGrid, MoGX leads for the first 1.5–4 h (depending on the reward), which covers our trajectory budget. For longer runs, MoG is the best method.

  • The partition divides the space (evY6, mfpV). At K=64K=64, no component is empty. On ideotype, the mean number of agents that found each optimum decreases from 7.4 (NoComm) to 5.0 (MoG) and 3.3 (MoGX).

  • Mixture correctness during training and at larger KK (evY6, mfpV, PtYt). The learned log⁡Zk\log Z_k keeps up with the moving classifier. Moreover, each learned log⁡Zk\log Z_k matches its exact shaped target. As we estimate the KL from samples, even a perfect mixture gives a KL above zero; we call this value the sampling floor. At K=4,…,64K = 4, \dots, 64, the mixture KL stays within 1.35× of this floor, even if some components are not converged.

  • Harder ideotype instance (PtYt). At B=20B=20 (2402^{40} states) and the same budget, MoGX finds 1.17× the optima of NoComm, 5.5× the modes of uniform sampling and 28× the modes of an untrained GFlowNet. At B=10B=10, uniform sampling covers the space, so we now present it as a controlled test of coordination schemes.

  • Ablations (mfpV). We ran the two ablations: a frozen random partition and exchange to a random destination. They confirm that partition plus exchange drives the gain. At K=64K=64, every variant finds 10–11× the modes of NoComm on Bitwise XOR and about 4× on Conditional Multiscale, even with a fixed random partition. On 3 of 4 tasks, the fixed partition does as well as the learned one; thus, MoGX does not need a well-trained classifier to divide the space. However, the classifier matters for the routing. On ideotype, guided routing finds 23% more optima than random routing at K=32K=32, and the policies overlap much less (pairwise Jaccard 0.12 vs 0.54). The learned partition is also balanced with no tuning. A random partition leaves more than half of the 64 components with no states until we calibrate it. We have added these new results to the paper.

  • Variable-length molecule task (evY6, mfpV). We built a fragment-based molecule environment with a QED reward. It has 201 action IDs, up to 129 legal actions per state, and trajectories of variable length. The runs are in our compute cluster queue; we will report the results when they complete.

The three sets of results that we promised are complete.

Molecule task (evY6 Q1, mfpV W3). We ran MoG, MoGX, NoComm and DDP at K=2,…,64K = 2, \dots, 64, with and without replay, 4 seeds each. Every run trained to 16.8M molecules; the trajectories have variable length, and each state has up to 129 legal actions. On this task, fψf_\psi still learns a sharp and balanced partition. At K=64K=64, MoG and MoGX find more high-QED molecules than independent agents in every seed.

The mixture during training (evY6 W2, mfpV W2). We trained MoG and MoGX on the HyperGrid original reward of Fig. 4 (1,024 modes), where we can compute every state exactly. We tried three ratios lrψ/lrZ\mathrm{lr}_\psi / \mathrm{lr}_Z: 10−310^{-3}, 10−210^{-2} and 10−110^{-1}. During training, we compared each learned log⁡Zk\log Z_k with the exact target. After the warm-up, each learned log⁡Zk\log Z_k stays within 0.26 nats of its exact moving target, while fψf_\psi still changes. The typical mixture weight is within 1.5% of its exact value.

EGFN with smaller mutation noise (evY6). We ran the full sweep at σ=10−2\sigma = 10^{-2} and σ=10−3\sigma = 10^{-3}. On four of the five HyperGrid rewards, the ranking in the paper holds. On the sparse reward, a small σ\sigma makes EGFN the best method. The paper now states this.

We also fixed the figure placement that PtYt pointed out.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.