acceptodds
Under review as a conference paper at ICLR 2027

TreeRef-BFN: Equivariance-Free De Novo Molecule Generation based on 2D Topology and Internal 3D Geometry

Abstract

De novo 3D molecular generation jointly models molecular size, topology, and geometry. Most methods pre-sample molecular size and generate Cartesian coordinates, limiting variable-size conditional tasks such as fragment completion and scaffold decoration while often relying on equivariant architectures. Internal-coordinate methods avoid rigid-body redundancy but typically require a known molecular graph or autoregressive construction, which may accumulate errors. We propose TreeRef, a tree-based molecular representation that assigns molecular topology and topology-dependent local 3D geometry to a naturally variable-size tree. RingRef nodes encode ring closures while preserving the tree structure, while Null nodes allow molecular size to emerge directly from node occupancy. Based on TreeRef, we develop TreeRef-BFN, a Bayesian Flow Network with a standard Transformer backbone that globally couples these locally defined variables and jointly generates discrete molecular variables and continuous local geometry. A single pretrained TreeRef-BFN supports unconditional generation and variable-size structure-conditioned 3D generation through masking alone, without retraining. Empirical studies demonstrate strong chemical validity, molecular stability, and diversity, accurate local geometric distributions, fast sampling, and competitive property-conditioned generation, establishing TreeRef-BFN as an efficient and flexible framework for 3D molecular generation.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
De novo molecular generation, 3D molecular generation, Bayesian Flow Networks, Molecular representation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2cDwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed TreeRef-BFN, which use a tree slot representation to generate molecule size, chemical topology and 3D geometry at the same time. In this representation, Real node means a real atom and Null state means the slot is empty, so the model can decide the atom number. RingRef node is used to encode the ring closure which can not be expressed by tree. Each slot also contains element, bond type, bond length, bond angle, dihedral angle and local chirality.

The authors use a normal Transformer with BFN to generate these discrete and continuous variables together, then decode them to chemical graph and 3D coordinates. Because internal coordinates are used, the network do not need equivariant architecture.

By fixing some variables and generating others, one pretrained model can be used for 2D to 3D generation, scaffold decoration and fragment completion. But for property conditional generation, a separate model still need to be trained. In main table, the validity is 96.56% on QM9 and 82.4% on GEOM-Drugs, and the angle and dihedral distributions are very close to reference.

Critical strengths

Strengths

  1. I think the unified representation is a good idea. Atom number, topology, ring and local geometry are all in one representation, so joint modeling become possible. The Null and RingRef design is simple but useful for variable size and ring structure.

  2. Bond length, angle and dihedral are generated directly, and the local geometry results are good. When pairwise angle is removed, the performance drop, which also support this design.

  3. Different conditional tasks can share one pretrained model only by masking variables. No need to train a special model for each task, this is convenient.

  4. The ablation study is quite complete, including simplified angle representation, post hoc ring closure, model size and sampling steps. The authors also analyzed the dependence on tree serialization. The multi-view consistency training is useful for conditional generation.

Critical weaknesses

Weaknesses

  1. The decoding process is complicated. Although variables are generated jointly, coordinates are still rebuilt by tree order. Incompatible angles need to be clipped or treated as degenerate, and RingRef need constrained matching. Ring closure matching do not adjust real atom positions, so it seems some closure error may remain. How large is this error in practice?

  2. The conditional generation of default model is not stable. In 2D to 3D task, small model get 93.00% validity, but default large model only get 64.65%, and medium model with multi-view consistency training get 94.55%. So better unconditional generation not always means better conditional generation. Which model the authors suggest to use for these tasks?

  3. Local geometry metrics are not enough. Bond length, angle and dihedral distributions close to reference can not guarantee each molecule has no clash, low energy or near equilibrium. I think more energy, force and global structure evaluation are needed.

Questions for authors

See weaknesses. Besides, the stability results are confusing to me. In main table, molecule stability of 100 step QM9 model is 92.52%. But in appendix Table 7, L model in same setting has 89.08% EDM stability and 96.19% MiDi stability. I know different metrics have different definition, but please explain which procedure or result version the main table number come from.

5 comments
AuthorsOpenReview ↗

Response regarding decoding error

Thank you for the questions on decoding, 2D-to-3D generation, and the evaluation metrics. We have added these points to the paper and the appendix.

Ring closure and angles. Ring closure only adds the ring bonds. It does not move the Real atoms already built. The residual is the distance from each virtual point to its matched atom, reported as the Closure column of Table 7 in Appendix C.1 and averaged over successfully decoded molecules. For the main model at 100 steps the mean is 0.089 Å. At the same budget, S and M are 0.305 Å and 0.126 Å. Raising the main model from 10 to 2,000 steps lowers this error from 0.164 Å to about 0.08 Å, with little further change after 100 steps. Relative to a bond length, the error is small.

When an atom is placed, the predicted angles sometimes cannot be realized together. The decoder applies three adjustments in a fixed order. Appendix C.2, Table 8 counts them.

The first drops the sibling angle. On a non-root atom, if a later child's angle to the parent bond is already within about 10−610^{-6} degrees of 0∘0^\circ or 180∘180^\circ, the two bonds coincide and no rotation about the parent remains. The sibling angle is then meaningless, and the decoder places the atom along the parent bond. An ordinary near-linear bond, such as 179∘179^\circ, is kept. This rate is 0% for the main model at every budget we tried.

The second pushes the sibling angle. From the second child of a non-root atom onward, the two angles to the parent bond fix an interval of sibling angles that can still be built. A prediction outside that interval is moved to the nearer endpoint, and both angles to the parent bond stay as predicted. The sibling angle cannot be realized in three dimensions without changing those parent angles. At 100 steps the main model pushes 4.11% of the eligible children, by 0.07∘0.07^\circ on average and 1.49∘1.49^\circ at most. The error on the parent angle is 0.00∘0.00^\circ.

The third flattens a child into a plane. It applies only at a root with at least three children. The first two children define a plane. If no height above that plane can meet both angles from the third child to the first two, the required squared height is negative. The decoder sets the height to zero, so the third child lies in the plane and each of those two angles changes slightly. At 100 steps this happens in 17.76% of the main model's molecules, and each angle moves by about 0.03∘0.03^\circ.

More sampling steps also reduce how often these adjustments are needed. From 10 to 2,000 steps, the main model's push rate falls from 9.29% to 3.55%, and its flattening rate falls from 24.98% to 16.27%. Each individual move remains a fraction of a degree. Relative to a bond angle, the adjustments are small.

Changes:

  • The end of “RingRef closure” in Section 3.3, pointing to the Closure column of Table 7 in Appendix C.1.
  • Appendix C.2, Table 8: the decode-time angle adjustments and their size.
AuthorsOpenReview ↗

Response regarding the model choice for 2D-to-3D conditional generation

2D-to-3D. The values 93.00%, 64.65%, and 94.55% are xyz2mol validity. Table 3 also reports 3D validity, with RDKit applied directly to the coordinates: 93.50% for S, 79.20% for L, and 95.30% for M-inv2. The two columns move together, and xyz2mol is the stricter one. Pretraining generates topology and geometry together. The model never sees an input in which the topology is already given and the geometry channels are still empty.

The channel-deletion experiment in the appendix asks which channel the model actually reads. We noise every channel of a validation molecule together, replace one channel by its prior, and take a single denoising step. At t=0.1t=0.1, removing the element or the bond barely moves the distance error: the change is below 0.004 Å for S, M, L, and M-inv2. Removing geometry raises the distance error from about 0.09 Å to about 0.16 Å, and element accuracy falls from about 0.67 to 0.21–0.26. At t=0.5t=0.5 the split is sharper. With geometry left in place the distance error is about 0.001 Å; with geometry removed, element accuracy drops below 0.04. A second check is ordinary unconditional sampling, 32 molecules and 20 steps, with one channel reset to its prior before every network call. Blanking geometry hurts M, L, and M-inv2. Blanking the element or the bond does not. The network uses geometry to decide the discrete structure. This matches the usual relation: one geometry generally determines one topology, while one topology can take several geometries.

2D-to-3D hands the model a finished graph while the geometry still starts from the prior, the reverse of training, where geometry becomes informative first and the discrete labels follow it. In Figure 12b, blanking the geometry input throughout sampling degrades M, L, and M-inv2, while blanking the element or the bond does not, and S does not drop in the same way. During joint generation the larger models read the geometry channel when deciding the discrete structure. In 2D-to-3D that channel starts from the prior, so under Frozen conditioning validity falls from S to L. This task is pattern matching. M-inv2 is trained on several trees of the same molecule and aligns those serializations with the consistency term, so it has seen more patterns. Its xyz2mol validity is 94.55%, already above S. We therefore recommend the view-augmented model for this task. The more fundamental remedy is pretraining adapted to incomplete channels or incomplete TreeRefs. We point to this direction in the conclusion.

Changes:

  • “2D-to-3D generation” in Section 4.3 reports the drop from S to L under Frozen conditioning and that M-inv2 recovers most of the loss. Table 3 reports both xyz2mol and RDKit 3D validity. The channel-deletion experiment in the appendix shows that the model uses geometry to decide the discrete structure.
  • The conclusion states that future work will design pretraining better suited to incomplete channels or incomplete TreeRefs.
AuthorsOpenReview ↗

Response regarding the evaluations

Energy, force, and global structure. We plot the six checks in one figure in Appendix C.3, so they can be read together. The first four are pass rates. The last two are the force RMS and the energy released by relaxation. TreeRef-BFN has the highest pass rate on steric clash, aromatic-ring flatness, and double-bond flatness, and a 99.19% pass rate on the UFF energy ratio. The force RMS is the force before optimization: 21.44 kcal/mol/Å, fourth place and close to the best value of 20.39 kcal/mol/Å. After 200 steps of MMFF94 minimization, the median energy released is ΔE=5.00\Delta E=5.00 kcal/mol, the smallest among the compared models. The force at the start is already small, and the energy lost over those 200 steps is the smallest, so the coordinates are already near a force-field minimum. Together with the pass rates above, the generated molecules rarely place atoms on top of one another, and aromatic rings and double bonds keep the planarity they require.

Changes:

  • Appendix C.3: definitions of the six checks, and the figure.
  • The QM9 paragraph in Section 4.2: the summary of these results.

Stability and validity. The 92.52% in the main table is Exact molecule stability. It applies the QM9 valence table to the generated topology, so it reads the 2D bond table and not the coordinates. MiDi at 96.19%, in the same setting, also reads that bond table, with the MiDi valence table. EDM at 89.08% is the 3D measure: bonds are inferred from distances on the decoded coordinates. Validity is completed in the same way. 2D validity applies RDKit to the generated bond table. 3D validity applies RDKit to the decoded coordinates. xyz2mol infers bonds from the coordinates and then applies RDKit. In Appendix Table 7, the L model at 100 steps matches the main table on these numbers. The previous caption did not separate them. It now does.

Changes:

  • The caption of Table 1 states that Exact and MiDi score the generated topology, while EDM, 3D validity, and xyz2mol score the decoded coordinates.
  • Appendix C.1, Table 7 uses the same definitions and gives the L-model numbers that correspond to the main table.
Reviewer_dTVMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of 3D molecular generation. The authors propose TreeRef-BFN, a BFN that globally couples these locally defined variables and jointly generates discrete molecular variables and continuous local geometry. TreeRef-BFN achieves state-of-the-art performance on diverse benchmarks.

Critical strengths
  • This paper is easy to follow.

  • TreeRef does not directly generate Cartesian coordinates, but represents a molecule as a rooted full k-ary tree whose slots jointly encode molecular occupancy, topology, and local 3D geometry.

  • TreeRef-BFN is evaluated on diverse benchmarks.

Critical weaknesses
  • BFNs have been widely used for molecular generation. The difference to existing methods(e.g., GeoBFN) should be discussed.

  • The performance improvement is not significant. The best performance should be highlighted in the table.

  • The ablation studies should be included. It is important to show which design choice contributes mostly to the performance.

  • Ablation studies should be included. It would be important to identify which design choices contribute most significantly to the overall performance improvements.

Questions for authors

See weaknesses

3 comments
AuthorsOpenReview ↗

Response regarding the distinction from GeoBFN, highlighting the best results

Difference from GeoBFN. GeoBFN and UniGEM generate Cartesian coordinates with equivariant networks, and the atom count is fixed in advance. UniGEM's Bayesian-flow variant only replaces the coordinate update. MolFLAE decodes the same variables from an equivariant latent, and the atom count is likewise given before decoding. TreeRef does not learn the molecule in Cartesian coordinates. It uses a tree. Because a molecule is not of maximum degree, the tree necessarily contains empty nodes. The model learns whether each node is empty. An empty child slot has no edge to its parent, and the branch ends there. When both the parent and the child slot are occupied, the two nodes are connected. The topology and the size of the molecule therefore arise during generation. They are not an input fixed in advance. A connected parent–child pair learns its local geometry directly. These local quantities are invariant to global translation and rotation, so no equivariant architecture is required.

Changes:

  • The end of Section 2.2 compares GeoBFN, UniGEM, and MolFLAE with TreeRef.

Highlighting the best results. Table 1 now reports results under several evaluation protocols, with the best value in each column in bold and the second best underlined. We have also added the full set of metrics. When a model directly generates a topology, 2D validity runs RDKit on that graph, and Exact and MiDi stability score the generated 2D structure with the QM9 and MiDi valence tables. On the decoded 3D coordinates of every model, 3D validity runs RDKit again, EDM stability infers bonds from distances, and xyz2mol reconstructs a molecule from the coordinates before RDKit. To assess the geometry of the generated molecules, we compare their bond-length, bond-angle, and dihedral distributions with the training set by W1W_1 distance. Cells those papers do not report are left blank. Under these side-by-side metrics, TreeRef-BFN is competitive in validity and stability. Its geometry is the closest to the reference data: the bond, angle, and dihedral distances are the lowest among the models we compare. The appendix adds six further metrics of energy, force, and global structure: steric clash, aromatic-ring flatness, double-bond flatness, the UFF energy ratio, the RMS MMFF94 atomic force, and the energy released by 200 steps of MMFF94 relaxation. TreeRef-BFN has the highest pass rate on clash, aromatic-ring flatness, and double-bond flatness, and a 99.19% pass rate on the UFF energy ratio. The atomic-force RMS is the push on the atoms before any optimization. TreeRef-BFN is at 21.44 kcal/mol/Å, fourth place and close to the best value of 20.39 kcal/mol/Å, so the forces on the generated coordinates are already small. ΔE\Delta E is the energy lost after 200 steps of MMFF94 relaxation. Its median is 5.00 kcal/mol, the smallest among the compared models. The starting force is already near the minimum, and walking 200 steps releases the least energy, so these coordinates are already near an MMFF94 minimum.

Changes:

  • Table 1 bolds the best value in each column and underlines the second best. The caption states that starred rows are taken from the original papers.
AuthorsOpenReview ↗

Response regarding the ablation studies

Ablations. We expanded the ablation analysis in both the main text and the appendix, adding a third generative ablation on canonical ordering (w/o canonical) alongside the existing two ablations, as well as a systematic study of root/traversal choice, view count, and alignment loss in Appendix C.5.

Table 1 reports three retrained ablations:

  • Replacing pairwise angles with a single parent-relative angle drops 2D validity from 96.56% to 90.12% and Exact molecule stability from 92.52% to 88.27%, showing that pairwise angular redundancy helps the model learn local geometry.
  • Replacing RingRef with post-hoc ring closure (Post-hoc ring closure in Table 1) increases sampling time from 12.3 to 22.4 minutes and lowers 2D validity to 92.40% and Exact molecule stability to 85.4%, because ring connectivity is no longer learned jointly.
  • Training on one random tree per molecule instead of the canonical tree (w/o canonical) lowers 2D validity to 91.47% and Exact molecule stability to 83.11%, and raises bond, angle, and dihedral W1W_1 to 0.0101 Å, 0.680°, and 1.720°, confirming that the canonical tree supplies a strong inductive bias at a modest trade-off in uniqueness (99.9% vs. 96.11%).
  • The experimental setup and detailed results for Appendix C.5 are in our response to Reviewer p6MU, Q1. The conclusion is that the canonical tree is an inductive bias: regressors that keep it perform best on the canonical validation set. Adding training views and turning on the alignment loss further lowers the error. With more views the model already aligns different TreeRefs of the same molecule, so the extra gain from the loss shrinks, but the loss still helps. The canonical tree is therefore the default serialization, and using the other valid trees as data augmentation is justified.
  • On conditional generation (Section 4.3 and Appendix C.9), at equal model width (M vs. M-inv2), multi-view consistency boosts 2D-to-3D 3D validity from 83.95% to 95.30% (xyz2mol from 78.40% to 94.55%) and scaffold-replacement success from 54.9% to 63.8% (vs. 33.6% for L), making invariance training the largest contributor on incomplete-TreeRef tasks.

Changes:

  • The Ablations paragraph in Section 4.2 reports the three retrained models and points to the model-size and invariance ablations in Section 4.3.
  • Ablations on QM9 in the appendix gives the numbers for these three.
  • Appendix C.5 covers the root, the traversal, and the consistency term.
  • The QM9 paragraph in Section 4.2 reports the step sweep.
  • Section 4.3 reports the conditional results of M and M-inv2.

Thanks for the response. I keep my positive score.

Reviewer_p6MUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce a de novo molecular generation method that builds upon their proposed, tree-based molecular representation to simultaneously sample atomic coordinates and the number of nodes. The approach is motivated by wanting to achieve better local topology modelling, which seems supported by the experimental results.

Critical strengths

S1: The central motivation of the paper is solid, and the proposed approach to solving it is novel and interesting.

S2: The paper, in general, is well executed: it’s easy to follow, has properly defined experiments that follow established standard, and the results are convincing enough.

Critical weaknesses

W1: My main objection / question would be regarding the novelty of the outcome. I don’t understand how it is, on a high level, different from simply having a more standard representation, with a corresponding mask capturing which nodes are being excluded. Especially given the fact that authors already introduce the number of maximum atoms. Many existing papers seem to use such mechanism, e.g. [1-4] to name just a few; yet the authors seem to frame it as a substantial novelty / contribution.

[1] DiffDec: Structure-Aware Scaffold Decoration with an End-to-End Diffusion Model [2] Multi-domain Distribution Learning for De Novo Drug Design [3] SynCoGen: Synthesizable 3D Molecule Generation via Joint Reaction and Coordinate Modeling [4] FlowMol3: Flow Matching for 3D De Novo Small-Molecule Generation

Questions for authors

Q1: Can authors clarify if the proposed canonical ordering also affects the root node, and if not, how exactly it’s being chosen? And secondly, did the authors attempt to quantify how the potential ambiguity of the tree structure (if canonical ordering was not used) affects the results; e.g., would it make sense to utilize these different potential representations as a data augmentation?

2 comments
AuthorsOpenReview ↗

Response regarding the distinction from mask-based variable-size representations and canonical tree ordering

W1. Thank you for the comment on our contribution. We first want to state what that contribution is. Internal geometry describes molecular geometry accurately. However, existing internal-geometry methods usually rely on a known molecular topology. TreeRef jointly generates global topology and local geometry from scratch, using the occupancy of each position on the tree together with the bond length, bond angle, dihedral, and handedness of each node relative to its parent and siblings. These local geometric quantities are invariant, so TreeRef does not rely on an equivariant architecture. In our experiments, the generated bond-length, bond-angle, and dihedral distributions are the closest to the reference data among the de novo models we compare.

The maximum atom count is not a device we added in order to generate molecules of variable size. Atoms in a molecule are not of maximum degree, so a tree representation contains empty positions. An empty position is marked Null on the element channel: it is unoccupied and is not expanded, and the branch ends there. The size of the molecule is the number of positions the model marks as occupied during generation. A molecule generated by TreeRef is therefore smaller than the preset upper bound. This follows from the tree itself. It is not produced by inserting fake atoms to obtain variable length. Setting the upper bound to 1+kN1+kN is then a natural extension.

We have rewritten the first contribution so that this is stated directly.

Changes:

  • The TreeRef comparison at the end of Section 2.2.
  • The first contribution in Section 1.

Q1. Root selection is the first step of the canonical ordering. Among atoms whose degree is at most kk, we take the one with the smallest eccentricity, and break ties by Weisfeiler–Lehman rank and then by input index (Appendix A.1).

We measured the effect of other valid trees in two experiments.

The first is a generation ablation (Table 1). Training on one random tree per molecule, with the canonical tree left out, lowers 2D validity from 96.56% to 91.47% and molecule stability from 92.52% to 83.11%, while uniqueness rises to 99.9%. Random trees increase diversity and cost some chemical quality. The canonical tree is an inductive bias: the model learns it more readily and gives up some diversity.

The second experiment compares traversal and augmentation on a small TreeRef regressor (Appendix C.5). We train three regressors: (1) the canonical tree plus views with a random legal root and a shuffled BFS order; (2) the canonical tree, plus views that keep the canonical root and only shuffle the BFS order; (3) fully random trees only, with no canonical tree in training. We evaluate them on the canonical tree, on trees that keep the canonical root and shuffle BFS, and on trees that draw a new root and shuffle the order. As the number of training views grows, the error of all three regressors falls on both the canonical and the random validation sets. The model is learning more than one layout, including how different TreeRefs of the same molecule correspond. The two regressors that keep the canonical tree do best on the canonical validation set and beat the regressor trained without it. Each is second best on the random validation set that matches its own training, and worst on the other random validation set. The regressor trained only on random trees performs similarly on all three validation sets. The canonical tree is not a special case of a random tree on which performance would look the same. It is rarely seen in that training run, and is therefore a harder serialization to learn.

The alignment loss is tested on regressor (1), which keeps the canonical tree and fills the remaining views at random. Turning the loss on lowers the error on all three validation sets. From 2 views to 8, the gap between the loss on and the loss off shrinks from about 7–9% to about 3–4.5%. With more views the model already aligns different TreeRefs of the same molecule, so the extra gain gets smaller, but the loss still helps.

The canonical tree is therefore the default serialization. When compute allows, we add more tree views and keep the alignment loss on. On 2D-to-3D generation and molecular editing, M-inv2, which uses this augmentation, outperforms single-view models of the same size, most clearly on scaffold replacement, where a large part of the molecule is missing. Using the other valid trees as data augmentation is justified.

Changes:

  • The opening of “Tree construction and packing” in Section 3.1.
  • The end of the Ablations paragraph in Section 4.2, which points to Appendix C.5.

I appreciate the clarification on Q1. I remain somehow confused / unconvinced about the answer to W1, as it seems tangential to my original question. Still, revising the contributions addresses my main qualm, and I remain supportive of the acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.