- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper targets synthesis-aware molecular generation for ionizable lipids, where existing methods either assemble from a fixed reactant catalogue or generate whole molecules freely. FORGE conditions a discrete flow-matching graph generator on a structured "reaction program" that specifies the assembly chemistry, depth, per-atom precursor roles, reaction-core positions, and coarse per-role morphology counts, but not precursor identities. Molecules are generated at atom-and-bond resolution, then verified by reverse decomposition followed by exact forward replay.
One shared model is evaluated on Ugi 3-CR, repeated aza-Michael addition, and repeated reductive amination across three seeds, against an unconditioned baseline, a cyclically mislabelled control, a finite-catalogue oracle, and three external generators. The paper reports higher exact-L1 yields for the conditioned model than for the unconditioned baseline on all three reactions, and recovery of precursors outside the training catalogues. Three generated lipids were synthesized and compared to DLIN-MC3-DMA in a mouse FLuc reporter study.
Critical strengths
-
The verification protocol is careful and reusable. The paper distinguishes the recovered witness set Ê_c(G) from the true preimage E*_c(G) and claims only the inclusion that bounded reverse search licenses (Eq. 25). It defines strong catalogue escape (Def. C.5) and then states plainly that its component-novelty metric does not establish it. Decomposition coverage, the replay invariant, abstention, and exact ambiguity are reported separately rather than collapsed into one "synthesizability" number. This protocol is worth adopting independently of the generative model, and it is a substantial part of why my recommendation is not lower.
-
The representation is a sensible fit to the domain. Encoding lipids as a spanning tree plus at most three closure edges supports the full 194-atom range without an O(N²) edge state, which matters for molecules of this size and near-acyclic topology. Combined with program-fixed coordinates held constant throughout sampling, the construction yields zero fixed-coordinate violations and zero support overflows across 27,648 attempts, so the conditioning mechanism itself is demonstrably doing what it claims.
Critical weaknesses
1. The headline gain is a validity effect, and no experiment isolates the contribution of reaction chemistry.
Table 3 gives both validity and exact-L1, which permits the conditional quantity the paper never reports. For Ugi, FORGE is 96.5% valid / 96.4% exact-L1 and the unconditioned model is 35.9% / 34.8%, so exact-L1 given validity is 99.9% vs 96.9%. For reductive amination it is 86.6% vs 88.3%, favoring the unconditioned model. The 61.6-point headline gap is therefore almost entirely about whether a valid connected graph is emitted at all, not about transform consistency, which is near-free in both cases. The abstract's claim that conditioning increased "exact transform-consistent generation" does not hold in that form.
The cyclic control cannot repair this. Since the program already fixes per-atom roles, core positions, and morphology counts, and Π_P pins the core throughout sampling (Prop. C.6), the reaction-identity token is largely redundant. Its null effect is equally consistent with "chemistry matters but is already encoded elsewhere in P" and with "chemistry does not matter." The ablation that separates these, conditioning on chemistry and depth alone without roles, core, and morphology, is absent, and it is exactly the claim in the title.
2. The generalization claim is contradicted by the paper's own held-component result.
The stated advantage over catalogue methods is reaching precursors outside the inventory. On the component-disjoint fold FORGE recovers 1.4 ± 1.7 held components per 1,000 attempts (per-seed 3.3 / 1.0 / 0.0), while unconditional DeFoG recovers 15.5 ± 2.4, roughly eleven-fold more. The contribution bullet rests instead on component novelty, a much weaker predicate that any atom-level perturbation satisfies; by rate, DeFoG is higher there too (97% of verified products vs FORGE's 87%).
Table 10 suggests the expansion runs away from realistic lipids rather than toward them: descriptor-manifold precision 29.5 for FORGE vs 54.6 for the catalogue oracle, fingerprint precision 0.0, and diversity 0.643, the lowest of any method in Table 2. The parsimonious reading is local perturbation around training templates, not catalogue escape.
Questions for authors
Answers to 1–3 could change my assessment.
-
Tables 1/3 and 2/7 disagree by ~3× on the same configuration. Ugi "Shared null" is 35.9% valid / 34.8% verified in Table 3, but "Shared-null + post-hoc" is 194.7 ± 15.6 valid / 114.4 ± 10.9 verified per 1,000 in Table 2. Same models and same fold? If so, one is in error; if not, which supports the 34.8% in the abstract?
-
Can you report exact-L1 conditional on validity for every row in Table 3, and is a (chemistry + depth only) ablation feasible during discussion? If the contrast largely vanishes under conditioning, how should the transform-consistency claim be restated?
-
Reproducibility. A.7 states that no source fingerprint match was recovered for seeds 1 and 2 and that no complete training manifest was recovered. Since every headline number is a mean over those three seeds, can they be re-run under a pinned implementation before camera-ready?
-
In vivo. How many animals and injection sites produced the four ROI values, and which limb is which formulation? RLU is a luminometer unit, so were these IVIS acquisitions, and can calibrated radiance or total flux be reported with a color scale and ROI definition? Why a single 4-hour timepoint? How were FORGE-1/2/3 selected from ~2,900 verified products, and was the D.2 role-resolved SMILES screen used? Was an animal-use protocol approved, and by which body?
-
Architecture and theory. Given the negative conditional-dependence diagnostic, FACT outperforming the full model on reductive amination (520.3 vs 318.8 per 1,000), and PCGrad removal improving all three means, what justifies joint whole-molecule denoising? Relatedly, with training times confined to [0.02, 0.98] and PCGrad not a gradient of any scalar loss, what does Theorem 3.1 establish about the system actually trained?
8 comments
Q3. Reproducibility of the main results
Since every headline number is a mean over those three seeds, can they be re-run under a pinned implementation before camera-ready?
We completed a new five-seed study comprising 35 training runs across seven configurations, recording the source code, training splits, configurations, random seeds, and checkpoint hashes.
| Reaction | Exact-L1 success (%) |
|---|---|
| Ugi | 93.68 ± 3.00 |
| Aza-Michael | 92.66 ± 0.80 |
| Reductive amination | 85.84 ± 1.68 |
We evaluated five independently trained models over 8,192 attempts per reaction per seed. We retained the submitted architecture, expanded the lipid data for aza-Michael and reductive amination, and ran additional updates. The original three-seed means were 96.4%, 72.2%, and 52.3%, respectively. Through this new study, we documented the training procedure, obtained high reaction-compatible yields across all three families, and completed the controlled ablations in the first comment. We report the new and archived studies separately and specify the data and training differences.
W2. Structural novelty and generalization beyond the catalog
The generalization claim is contradicted by the paper's own held-component result.
In the designated held-out experiment, FORGE recovered 1.4 excluded precursors per 1,000 attempts and DeFoG recovered 15.5. To examine the broader question of generation beyond the training catalog, we added precursor-distance, novelty-denominator, and exhaustive catalog-reachability analyses.
For the original 87% versus 97% comparison, we divided distinct component-novel products by all verified attempts, which made the rates sensitive to duplicate generation. To compare novelty among unique products, we recalculated the rates using distinct verified products as the denominator:
| Ugi novelty metric | FORGE | DeFoG |
|---|---|---|
| Novel precursor rate among distinct verified products (%) | 97.27 | 96.98 |
| Similarity to nearest training product (lower is more novel) | 0.660 | 0.773 |
Across three Ugi seeds, we measured ECFP4 Tanimoto similarity to the nearest training product. FORGE's lower similarity indicates greater average structural separation. We further compared precursors within each reaction role and found greater distances for FORGE isocyanides (0.464 versus 0.215) and aldehydes (0.213 versus 0.173), with comparable amine distances (0.316 versus 0.312).
To test catalog escape directly, we exhaustively enumerated 485,625 Ugi precursor combinations under the registered reaction, yielding 488,250 distinct reachable products. We counted a generated product as an escape when it passed exact verification and structural screening and fell outside this complete set (Definition C.5).
| Qualified Ugi catalog escapes / 1,000 attempts | FORGE | DeFoG |
|---|---|---|
| Mean ± SD | 833.3 ± 23.2 | 258.4 ± 22.9 |
In this separate three-seed Ugi evaluation, FORGE produces 3.2 times as many qualified escapes per attempt. Because we enumerated the full reachable set, these results directly establish generation beyond every product obtainable from the specified finite inventory.
In Table 10, we compared the generated structures with the published-lipid distribution. FORGE achieved descriptor-manifold precision of 29.5 versus 54.6 for the catalog oracle, fingerprint precision of 0.0 under the reported metric, and diversity of 0.643 versus 0.727 for DeFoG. We then used exhaustive reachability analysis to establish expansion of the accessible product space. To assess biological function, we also synthesized three selected lipids and measured reporter expression in vivo, as detailed in the next comment. Together, these evaluations measure structural novelty, catalog expansion, and experimental performance.
Q4. In vivo experimental design and candidate selection
In vivo. How many animals and injection sites produced the four ROI values, and which limb is which formulation?
How were FORGE-1/2/3 selected from ~2,900 verified products, and was the D.2 role-resolved SMILES screen used?
To evaluate the generated designs experimentally, we synthesized three FORGE lipids, formulated each as FLuc mRNA-LNPs, and measured reporter expression in vivo (Figure 2 and Appendix B.5). We tested the formulations in two mice across four intramuscular sites, with one formulation per hindlimb and one site per formulation. At 0.05 mg mRNA/kg, we measured bioluminescence with Xenogen IVIS four hours after administration to assess early reporter expression. We administered D-luciferin approximately nine minutes before imaging.
We display calibrated radiance in photons/s/cm²/sr, with color ranges of 1.00 × 10⁷ to 1.00 × 10⁸ and 2.00 × 10⁷ to 2.00 × 10⁸. To quantify expression, we drew circular ROIs around each injection site and normalized the signals to nonluminescent background ROIs in the corresponding images. We report these background-normalized relative signals below and will use this precise label in the revised figure and text.
| Mouse | Injection site | Formulation | Background-normalized signal |
|---|---|---|---|
| 1 | Right hindlimb, ROI 1 | FORGE-1 | 5.997 × 10⁵ |
| 1 | Left hindlimb, ROI 2 | FORGE-2 | 2.036 × 10⁶ |
| 2 | Right hindlimb, ROI 3 | FORGE-3 | 2.881 × 10⁶ |
| 2 | Left hindlimb, ROI 4 | MC3 | 2.195 × 10⁵ |
We identify the limbs anatomically from the supine, head-up image orientation. FORGE-3 produced an observed 13.1-fold higher signal than MC3 at separate injection sites in the same mouse. By synthesizing, formulating, and testing the generated lipids, we demonstrated reporter expression in vivo at the four measured sites.
To select candidates for these experiments, we ranked the generated products and screened the leading designs in the following sequence:
| Stage | Selection procedure |
|---|---|
| Generation | Approximately 2,900 reaction-verified FORGE products generated using reaction programs with defined precursor roles |
| Activity prediction | Candidates ranked using the off-the-shelf LANTERN transfection predictor, with the top 20 prioritized |
| Precursor analysis | Appendix D.2 role-resolved SMILES analysis applied to the top 20 candidates to examine the associated precursor structures |
| Final selection | Three candidates selected for experimental synthesis based on synthetic accessibility |
We specified precursor roles during generation and then applied the Appendix D.2 analysis to the top-20 shortlist. From this screen, we selected three candidates using computational predictions and chemical accessibility before experimental testing. We conducted all animal procedures under an approved institutional animal-use protocol and will provide the identifying institution and protocol details after double-blind review.
Q5. Joint molecular generation and theoretical guarantees
Given the negative conditional-dependence diagnostic, FACT outperforming the full model on reductive amination (520.3 vs 318.8 per 1,000), and PCGrad removal improving all three means, what justifies joint whole-molecule denoising?
We use joint denoising to exchange information across the headgroup, linkers, and hydrophobic regions during generation. Alongside the conditional-dependence diagnostic, we directly tested this architecture against FACT. Joint generation yields 856.9 versus 56.9 exact-L1 products per 1,000 attempts for Ugi and 767.7 versus 396.4 for aza-Michael, and also exceeds the larger FACT model on both reactions. Reductive-amination yields are 318.8 versus 520.3. These comparisons identify the reaction families that benefit from joint generation. We further removed PCGrad and improved mean performance across all three families.
In Theorem 3.1, we prove recovery of the reaction-conditioned distribution at the population optimum of the reference objective under exact continuous-time sampling. Proposition C.6 further guarantees preservation of fixed coordinates during finite numerical sampling. To evaluate our implementation, which uses training times , auxiliary objectives, PCGrad, and a capped Euler sampler, we conducted the new five-seed study and matched ablations.
We appreciate the opportunity to resolve the reporting discrepancy and provide the requested evidence. We apologize for the length of the response and thank the reviewer for the careful consideration. In light of the new chemistry ablations, exhaustive catalog-escape results, and clarified in vivo findings, we would be grateful if the reviewer would consider a higher score and renewed confidence in the submission.
We thank the reviewer for the careful analysis and for recognizing the verification protocol and molecular representation. To address the concerns, we reverified the final checkpoint, completed 35 new training runs under a pinned implementation, and added the requested chemistry and catalog-escape experiments. We also clarify how we selected, synthesized, and tested the three generated lipids in the in vivo study. Comments 2 and 3 cover reproducibility, novelty, and the animal experiments.
Q1. Resolving the checkpoint discrepancy
We traced the reporting error to the intermediate shared-null checkpoint that we had used in Tables 2/7. For Tables 1/3, we had used the final checkpoint.
Mean ± SD across three seeds, 3,072 attempts each. We independently reverified all 9,216 final-checkpoint attempts, confirming the abstract's 35.9% validity and 34.8% exact-L1. We will correct Tables 2, 7, and 8 and the associated text. Against this final baseline, FORGE achieves 96.5% validity and 96.4% exact-L1, a 61.6-point increase in verified-generation yield.
W1 + Q2. Isolating learned reaction chemistry
We obtained 100% replay precision for every configuration in Table 3. Exact-L1 among valid molecules therefore equals decomposition coverage:
We report the per-seed aggregated values from Table 3. In the original Ugi comparison, we supplied both models with the same reaction core and structural constraints, and conditioning primarily increased the yield of valid molecules. To isolate the contribution of learned chemistry, we then trained matched models receiving chemistry and depth only, or neither input. We disabled core pinning and optional chemistry-specific decoder corrections in both models.
Mean ± SD across five paired seeds, 8,192 attempts per reaction per seed. By adding chemistry and depth, we increased reaction compatibility among valid molecules by 18.77, 39.91, and 6.64 points. These gains directly demonstrate a learned chemical contribution, with the largest per-attempt improvements for Ugi and aza-Michael.
We also found that we had permuted labels consistently during training and evaluation in the original cyclic control. To test reaction identity directly, we instead changed only the label at inference, holding the trained model and other inputs fixed and disabling core pinning. Switching Ugi to aza-Michael changed Ugi compatibility from 44.10% to 0.00% and aza-Michael compatibility from 0.04% to 62.73%. Switching aza-Michael to reductive amination changed the corresponding rates from 57.85% to 0.82% and 11.76% to 31.45%. Switching reductive amination to Ugi increased Ugi compatibility from 0.00% to 22.62%, with reductive-amination compatibility at 43.55% (five paired seeds, 512 attempts per reaction per seed).
We further compared full and reduced neural-program conditioning while holding production decoding and structural constraints fixed:
Five seeds, 512 matched attempts each. We qualified products by exact replay, structural screening, absence from the training-product union, and a novel recovered precursor. Full conditioning increases qualified novelty yield, with slightly lower overall validity, and reduces undesirable O–O-containing outputs. Reduced conditioning yields higher ECFP4 diversity. Together, these experiments isolate learned chemistry and show how the complete program improves qualified discovery under fixed constraints. We will clarify both contributions in the abstract and manuscript.