- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper extends Discretized Boltzmann Generators (which is a new term coined by the authors) by replacing the atom-wise graph encoder with a text prefix.
Existing work is limited by how the conditioning is done, i.e., hand-designed features of a chemical graph are used as embeddings. This comes with three challenges: First, conditioning length grows with the number of atoms, second, a variable number of atoms is not permitted and third, extensions potentially require new hand-crafted features. The text prefix extends these graph conditionings by specifying the molecular graph, the solvent environment, the potential energy model, and temperature.
The authors train models between 8M and 85M parameters on Many Peptides, and then scale to 640M parameters, where they report the best rESS on 2AA/4AA/8AA peptides so far. They further train a joint model on peptides, MDQM9-nc and FreeSolv.
Additionally they apply speculative decoding, estimate solvation free energies. Four new MD datasets are also announced.
Critical strengths
- The paper is written well and gives a nice and honest history of existing methods.
- The scaling study is large and carefully documented, and the Many Peptides numbers of the 640M model are a big step forward.
- The discretization analysis is nice! The finding that the resolution has an optimum (in contrast to Rehman et al.) and the derivation of the quantization energy error are nice additions.
- The limitations section is honest about the scaling law and the small-molecule results, and the datasets will be useful if released.
Critical weaknesses
- As far as I could see, GPU hours for the experiments were not reported
- The actual method change, prefix instead of graph encoder, is never ablated. There is no run with the same transformer, data, Muon optimizer and token budget but Robin-style conditioning. Table 2 compares against a reproduced Robin whose parameter count and training tokens are not reported, so I cannot tell whether the 13× rESS gap between Robin PBG comes from the conditioning, the optimizer or just far more tokens.
- The state-of-the-art comparison is not like-for-like. The hardest 8AA system was removed from the test set because the model fails on it (Appendix E), while the literature rows are presumably on all 30 systems (?), and the paper still bolds its own numbers.
- There are no seeds or error bars anywhere.
- There is no comparison against MD/REMD throughput, which is the paper's own motivation.
Questions for authors
- What are Robin's parameter count and token budget in Table 2, and does your reproduction match the published numbers? Can you add the controlled ablation (same transformer, data, optimizer, tokens) with graph-encoder vs. prefix conditioning?
- Would be great if the authors could report Table 2 on all 30 8AA systems including YFPHAGYT, and state whether the TarFlow and Prose rows are on 30 systems.
- Why are the 160M and 320M models worse than the 85M model on 2AA/4AA despite more compute?
- Are any rows of Table 2 repeated with different seeds?
- What are the GPU-hours of the 640M run, and how many effective samples per second does MD or REMD produce on the same octapeptides and hardware?
In general, I like the paper and am willing to raise my score if the authors address my concerns.
3 comments
# Response to 4igz part 2
Why are the 160M and 320M models worse than the 85M model on 2AA/4AA despite more compute?
Because the Table 2 rows were trained to different budgets with different settings. At a matched budget, rESS increases with model size at every peptide length.
Table 2 reports the best available checkpoint of each size. The 85M model was trained to , the 160M model to and the 320M model to (Table 2), with different hardware and batch sizes (Table 26). To isolate model size, we trained every PBG size to with several training seeds (new Appendix D.4, Table 11). At this matched budget, rESS increases with size at every peptide length and for every seed: on 2AA 0.27–0.29 (85M), 0.43 (160M), 0.47–0.48 (320M) and 0.66 (640M); on 4AA 0.09, 0.18, 0.22–0.24 and 0.40. The main text now points to this comparison.
Are any rows of Table 2 repeated with different seeds?
At the time of submission, no. Now every measured row of Table 2 is the mean of three sampling seeds, and new training replicates are in Appendix D.4.
- Sampling seeds. Every measured row of Table 2 is evaluated three times on the same checkpoint, samples per system each. The table reports the mean and the largest deviation of a seed mean. Tables 4 and 15 also report means over three sampling seeds.
- Training seeds (Table 11). We retrained every PBG size to with three seeds for 30M–85M and two for 160M and 320M. Across training runs, rESS varies about 7 times as much as across sampling seeds, E-W2 and T-W2 about 2 times, and TICA-W2 about as much.
While preparing these replicates, we corrected the log-likelihood of reflected conformers in some rows of the submitted Table 2. Please see the global response for details.
What are the GPU-hours of the 640M run, and how many effective samples per second does MD or REMD produce on the same octapeptides and hardware?
The 640M model took 1,138 GPU-hours to train on 16 AMD MI250X GPUs. On the same H100 GPUs, PBG 640M converges at about 0.12 H100-hours per system, while MD needs 30 to 35 H100-hours to reach the same torsion and TICA errors and about one H100-hour for the energy error.
Table 26 now gives the GPU-hours of every Table 2 model, from 17 (49M) to 1,138 (640M). The 130 checkpoints of the scaling model took 1,259 GPU-hours on RTX PRO 6000 and H100 NVL GPUs (Appendix G).
For sampling, we do not report a ratio of effective sample sizes, because the importance-sampling ESS of a BG and the autocorrelation ESS of MD measure different quantities. Instead, we plot energy and torsion metrics against energy evaluations and H100-hours per system, for PBG 640M and for the MD that generated Many Peptides, both timed on the same H100 GPUs (new Figure 5, Appendix D.9). PBG 640M uses speculative decoding with the 15M drafter, which keeps the exact likelihood of the 640M model. We did not run REMD.
Concluding remarks
We thank the reviewer again for their time, the constructive review and the offer to raise the score. The conditioning ablation, the 30-system table, the seed replicates, the GPU-hours and the comparison with MD answer each question. The YFPHAGYT investigation also found an artifact in the Many Peptides data that affects previous transferable Boltzmann generators. We kindly ask the reviewer to reconsider their review in light of these new experiments and clarifications. We are happy to continue the discussion and provide any further clarifications.
Dear reviewer 4igz, We are very appreciative of your time and constructive feedback on our submission. We hope that our rebuttal responses have addressed your questions.
As the end of the rebuttal period is quickly approaching, we would be happy to answer any remaining or additional questions. If our responses and additional experiments address your concerns, we would be grateful if you would consider updating your assessment.
# Response to 4igz part 1
We thank reviewer 4igz for their encouraging and precise review. We are glad that the reviewer finds the paper "written well" with "a nice and honest history of existing methods", the scaling study "large and carefully documented", the 640M results "a big step forward", and the discretization analysis a nice addition. We addressed every question with new experiments. Below we answer them in turn. All new and changed text in the revised PDF is marked in green.
ROBIN has 132M parameters and was trained to D/N≈1100 loss tokens per parameter (correspondence with the authors; Appendix D.1). Our evaluation ("ROBIN (T=1.0)" in Table 2, three sampling seeds) agrees with the published values (T=0.9) on rESS and on the torsion and TICA metrics. The 8AA energy W2 is higher, which we attribute mainly to the different logit temperature. We added the controlled ablation: at matched transformer, data, optimizer and tokens, the prefix is as good as the atom-wise encoder at 30M and better at 85M, and most of the gap to ROBIN comes from the optimizer settings and random translation.
For the ablation, we trained a new ladder of nine models at 30M and at 85M (Appendix D.6). All rungs share the PBG transformer, the data and 48N loss tokens. Rung L0 uses the ROBIN protocol and ROBIN's atom features, and each rung switches one difference to the PBG setting (Table 14). The last step changes only the conditioning.
Table R.1. Ladder from the ROBIN protocol to PBG at 85M, unclipped metrics, means over test systems and three sampling seeds (Table 15 of the revised PDF).
Three steps account for at least 90% of the L0-to-L8 change of each metric in log scale: L1, L7 and, at 85M, L8. All rungs see the same loss tokens, so the gap does not come from more training tokens.
The new Table 13 reports Table 2 on all 30 octapeptides. In Table 2, the TarFlow and Prose rows are on the same 29 octapeptides as our rows, after correspondence with the authors, so the comparison and the bold values are like-for-like (Appendices D.5 and E).
While answering this question, we found that the reference MD of YFPHAGYT contains a stereochemistry artifact of Many Peptides, which explains the poor performance (new Appendix D.5). Its published starting structure has L-Pro3, but a Phe2 side-chain atom lies within 1 Å of the Pro3 ring. The default OpenMM energy minimization removes this overlap and inverts the Pro3 Cα, so the reference trajectory has D-Pro3. The same artifact is in 276 training peptides, and all models learned it: 90.9% to 99.6% of PBG samples and 98.7% to 99.0% of ROBIN samples of YFPHAGYT have D-Pro3. Our filter compares samples with the sequence and rejected them. The public evaluation code of Tan et al. compares samples with one reference MD frame and cannot detect an artifact that the reference shares.
Table R.2. 8AA results on all 30 octapeptides with a filter based on the reference structure (Table 13 of the revised PDF). Literature rows are the published 30-system values.