- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper presents three different contributions. 1) They train SAE on BioEmu representations and show that biophysical, thermodynamic and structural ensemble properties can be predicted from those. In particular they claim to predict DG and DDG values without sampling (as opposed to BioEmu. 2) They train a representation adapter to steer BioEmu towards specific radiuses of gyration. Then, they apply it as a steering force at a well-chosen diffusion timestep and show it effectively steers. 3) They train other small MLP adapters with an RMSD minimization objective to steer the generation towards specific protein conformations, which they evaluate on benchmark datasets with good results.
Using BioEmu is the main connection between the three parts of the paper (at least between 1 and 2-3). Line 139, "In this work, we steer with an SAE feature which correlates strongly with the protein radius of gyration": this sentence is misleading at the very least, since the steering and MLP adapters don't rely on the SAE features. From the code provided, I suspect it might have been the initial intention to use the SAE features, but it did not work.
Critical strengths
S1. The steering towards a given state ("open the potential cryptic pockets of this protein") is an interesting idea
Critical weaknesses
W1. Overall, the paper is underspecified and the claims are not precised. Reviewers (and readers) should not be expected to read the Appendix to understand the method presented. A few examples:
- Line 139: "In this work, we steer with an SAE feature which correlates strongly with the protein radius of gyration": that’s not what is described in the paper
- Line 160: " To this end, we correlate many different biophysical properties with feature activations in our trained SAEs." No description of the SAE training in the main, no pointer to the relevant appendix section.
- Line 302: "To ensure the train and test splits were sufficiently different, we performed clustering on our full dataset and made the train/validation/test split based on those clusters." The datasets are only mentioned in the related work section, and it is unclear how the splitting is made in relation to the benchmark used.
- Line 376: "In (Fig. 3d), the residual fraction at all timesteps for all cases is above 0.5". That's not what the figure shows. The domain motion exception is correctly mentioned but not the kinases, which remain below 0.5 for all timesteps.
W2. I don't think part 1 is novel, nor particularly interesting. I am not surprised to learn that the latent space of BioEmu incorporates those features. Your reported figures for DG prediction (around .56 spearman) do not match the ones reported in BioEmu (.67, fig 4a).
W3. For part 2, steering has only been studied on Rg (as stated in the limitations). It has not been compared to the many established steering methods, so the benefit of the strategy remains to be proven. Rg is differentiable wrt to coordinates, so existing steering methods are applicable.
W4. Part 3 relies on prior knowledge of the objective at hands (e.g. sampling DFG-in), contrary to the baselines. On domain motions, this results on an arguably arbitrary objective. On DFG-in/out, this corresponds to existing established collective variables that are not compared. I think the opening of cryptic pocket represent your most meaningful contribution. Despite this, the proposed approach does not result on sota performance.
W5. The experimental setup of part3 is not clean enough. As mentioned above, the splitting strategy wrt to the benchmark is not clear enough to me. Moreover, the results on cryptic pockets and domain motions are not really comparable to ConforNets, since the systems used are not the same. Considering the small number of systems per dataset, even a single system can have a non negligible impact. I would suggest presenting the results on the exact same systems as ConforNets for a fair comparison (or to run ConforNets on the same systems as the authors). As a side note, the results are presented with too many digits compared to your stds.
Questions for authors
Q1 In the Rg steering part, did you try updating the embeddings at multiple diffusion timesteps instead of a single one, similarly to the conformational sampling part ? Steering too early and too late is detrimental to the structure's validity as you showed, but applying steering at multiple intermediate steps could potentially refine the guidance towards the target property.
4 comments
(4) We originally chose the benchmark and success@B metric because it is used in the original BioEmu and ConforMix papers, but not only is it small, it also doesn’t capture what we’re truly trying to test (if you sample the targeted conformer more than once for a given sequence, the score remains the same). As such, we’ve moved the success@B tables to the appendix and switched to a separate pure percentage metric (originally presented in Fig. 3f, but not well highlighted), which scores the hit percentage of samples that are within the tolerance range. This tests targeted steering rather than broadening the sampling distribution, and is what our steering method is intended to do. The success@B metric measures whether at least one sample hits the target, but doesn’t distinguish between methods that hit the target multiple times, which doesn’t test targeted steering very well. Our relative performance on this metric is much clearer. The updated table is (Table 1, shown below), in which we match or outperform BioEmu on all targets, and perform better than ConforMix on most targets as well. We exclude ConforNets domain motions and cryptic pockets here, as you mention that they aren’t comparable due to ConforNets requiring per-protein training, which we agree with. We kept in the ConforNets kinase results as those aren’t trained per protein. The full ConforNets results are still present in the success@B appendix results. Most of the values on the BioEmu benchmark are within error due to the low number of systems, so we’ve also added our own larger test set to corroborate (Table 1, shown below), which shows our steering outperforming BioEmu and ConforMix. On domain motions, we agree that this is not an ideal metric, but it does provide some distinction, with the rest inferred from the input representations. For DFG-in/out, we compared to BioEmu’s FK steering [1] with the Kincore [2] DFG dihedral distance as the collective variable, in which we match on DFG-in and outperform on DFG-out (Table 2). We agree that the cryptic pocket result is the most consequential result, as it allows for steering without a target or collective variable at inference time, and does better than existing methods on our larger test set.
Benchmark panels
| Benchmark | N | BioEmu | ConforMix | ConforNets | Ours |
|---|---|---|---|---|---|
| CP (apo) | 32 | 0.08 ± 0.03 | 0.14 ± 0.05 | – | 0.11 ± 0.04 |
| CP (holo) | 32 | 0.15 ± 0.05 | 0.20 ± 0.05 | – | 0.26 ± 0.06 |
| DM (nearer) | 20 | 0.45 ± 0.08 | 0.16 ± 0.03 | – | 0.59 ± 0.09 |
| DM (further) | 20 | 0.12 ± 0.04 | 0.16 ± 0.03 | – | 0.16 ± 0.06 |
| Kin. (DFG-in) | 20 | 0.55 ± 0.09 | 0.42 ± 0.08 | 0.65 ± 0.09 | 0.55 ± 0.09 |
| Kin. (DFG-out) | 20 | 0.22 ± 0.06 | 0.30 ± 0.08 | 0.37 ± 0.09 | 0.30 ± 0.08 |
Held-out test sets
| Benchmark | N | BioEmu | ConforMix | Ours |
|---|---|---|---|---|
| CP (apo) | 134 | 0.11 ± 0.02 | 0.087 ± 0.016 | 0.17 ± 0.03 |
| CP (holo) | 134 | 0.078 ± 0.017 | 0.12 ± 0.02 | 0.16 ± 0.03 |
| DM (nearer) | 69 | 0.34 ± 0.05 | 0.114 ± 0.016 | 0.42 ± 0.05 |
| DM (further) | 69 | 0.09 ± 0.02 | 0.038 ± 0.008 | 0.14 ± 0.04 |
(5) We addressed the splitting strategy above in (1.3). We agree that the results are not comparable to ConforNets as ConforNets is trained on each individual protein, and is also not tested on the exact same set. As such, we’ve removed ConforNets from our evaluated metric for cryptic pockets and domain motions, mentioned in (4), but have the full success@B with consistent systems in the appendix (Table A2-A5) and retain the kinases since those aren’t per protein. We’ve also fixed the digits.
Q1: We originally did not try updating the embeddings at multiple diffusion timesteps. We’ve now added this to (Fig. 2b). We find that performance is very close to the best single site, which makes sense as none of the other timesteps had much of an effect.
[1] Lewis et al. Scalable emulation of protein equilibrium ensembles with generative deep learning, 2025. [2] Modi et al. Kincore: a web resource for structural classification of protein kinases and their inhibitors, 2021.
thanks for your anwser
Thanks for your answer.
1/2. Thanks for fixing the specifics, but I still think the paper should focus on the steering method and its validation.
- Thanks for comparing to FK, but you could also include conformix, conforflux, confornets, etc..
4/5. You state "We exclude ConforNets domain motions and cryptic pockets here, as you mention that they aren’t comparable due to ConforNets requiring per-protein training, which we agree with." But my statement was "not really comparable to ConforNets, since the systems used are not the same." since you had 30 and not 32 systems.
You should include confornets results for all datasets, since they seem to outperform your method. To make the computational aspects clear, you can include a column with the overall time. A possibility would be to draw metric curves with time and num_samples as the x-axis.
Overall, this does not really address my concerns that:
- the first parts about R_g and representation analysis are not interesting enough for ICLR. I think a deeper rewrite is needed that directly investigate the steering aspects of the project
- the part about steering is unsufficiently benchmarked and compared to SOTA, especially considering its limited applicability (DFG-in/out) and intuition (for the domain motion).
I choose to maintain my rating
Thank you for your quick response.
(1/2) We have now expanded the steering analysis, including a distance-stratified analysis and compute cost comparison.
(3) ConforMix, ConforFlux, and ConforNets are all diversification methods. In their diversification configurations, there isn’t a way to specify a level (e.g. sample R_g one standard deviation from the mean), so they would each sample the same mean R_g regardless of what R_g we request from steering. Our method, as well as BioEmu’s FK steering, targets a specific R_g, which is why we chose to compare to FK and why it doesn’t make much sense to compare to diversification methods for this.
(4/5) Thank you for the comment. We have now added back ConforNets to the BioEmu benchmark, evaluated on the same systems as all of the other methods (we dropped one domain motion system that didn’t fit in H200 RAM for ConforNets). We are unable to run ConforNets on our own test set, as it would take too long to run within the remaining rebuttal period (1000+ GPU-hours). In fact, this is one of the main drawbacks of ConforNets compared to our method. At your suggestion, we have added a plot (Fig. 3b) showing the compute difference between the different benchmarked methods, with ConforNets coming out 10-20x more expensive in GPU-hours for cryptic pockets and domain motions than our method, ConforMix, and BioEmu. This is potentially very important for applications like high throughput screening, where our method would be an order of magnitude faster. We have also now emphasized in the main text that our method works comparatively better on systems with targets close to the unsteered ensemble, whereas diversification methods tend to work comparatively better on systems with targets far from the unsteered ensemble that BioEmu very rarely samples, which is a more specific claim. In (Table 1, below), we show that at shifts within 0-2 units of the success threshold, we outperform BioEmu and ConforMix (matching at 1-2 for kinases), and either match or outperform ConforNets at a tenth of the compute cost, with the exception of kinases at 1-2. Meanwhile, at large shifts, we perform competitively with BioEmu and ConforMix, and worse than ConforNets. As such, for systems that target structures close to the unsteered ensemble, our method is better than the other benchmarked methods either in compute cost or in pure performance, with the kinase exception.
BioEmu benchmark set
| Benchmark | d₀/τ | N | BioEmu | ConforMix | ConforNets | Ours |
|---|---|---|---|---|---|---|
| CP | < 1 | 4 | 0.78 ± 0.07 | 0.85 ± 0.09 | 0.981 ± 0.019 | 0.985 ± 0.015 |
| 1–2 | 13 | 0.27 ± 0.07 | 0.36 ± 0.10 | 0.45 ± 0.09 | 0.43 ± 0.09 | |
| ≥ 2 | 47 | 0.011 ± 0.004 | 0.059 ± 0.019 | 0.14 ± 0.03 | 0.050 ± 0.016 | |
| DM | < 1 | 7 | 0.75 ± 0.08 | 0.17 ± 0.03 | 0.58 ± 0.06 | 0.951 ± 0.016 |
| 1–2 | 14 | 0.30 ± 0.06 | 0.20 ± 0.03 | 0.40 ± 0.06 | 0.44 ± 0.10 | |
| ≥ 2 | 17 | 0.10 ± 0.05 | 0.12 ± 0.04 | 0.25 ± 0.06 | 0.14 ± 0.07 | |
| Kin. | < 1 | 14 | 0.83 ± 0.03 | 0.63 ± 0.07 | 0.87 ± 0.07 | 0.86 ± 0.02 |
| 1–2 | 18 | 0.20 ± 0.05 | 0.31 ± 0.08 | 0.42 ± 0.10 | 0.27 ± 0.06 | |
| ≥ 2 | 8 | 0.00 ± 0 | 0.00 ± 0 | 0.08 ± 0.08 | 0.00 ± 0 |
Held-out test sets
| Benchmark | d₀/τ | N | BioEmu | ConforMix | Ours |
|---|---|---|---|---|---|
| CP | < 1 | 16 | 0.78 ± 0.04 | 0.41 ± 0.08 | 0.96 ± 0.02 |
| 1–2 | 64 | 0.14 ± 0.03 | 0.23 ± 0.04 | 0.33 ± 0.05 | |
| ≥ 2 | 188 | 0.018 ± 0.006 | 0.032 ± 0.006 | 0.040 ± 0.012 | |
| DM | < 1 | 24 | 0.78 ± 0.05 | 0.23 ± 0.02 | 0.93 ± 0.04 |
| 1–2 | 38 | 0.23 ± 0.04 | 0.106 ± 0.017 | 0.37 ± 0.06 | |
| ≥ 2 | 76 | 0.020 ± 0.009 | 0.013 ± 0.005 | 0.029 ± 0.013 |
We thank the reviewer for their helpful feedback. Our response will be split up into multiple comments due to the character limit. Based on their comments, we have substantially revised our claims, fixed incorrect lines, added supporting tests, and strengthened steering results. We hope our responses below address their concerns:
(1.1) Line 139: this is a mistake from a previous version, which has now been corrected. In the main text we only steer with the learned MLP. We also included the attempt to steer with SAE features in the appendix (Fig. A1), but it is less central to the main claims of the paper.
(1.2) Line 160: we have moved all of the work relating to SAEs to the appendix, as they are not central to the main claims of the paper. SAE details are in appendix sections (A.1-A.4). Our main contributions are extracting ensemble and free energy in a single pass, showing BioEmu’s activations can be directly manipulated to steer R_g at a single timestep, and showing that this is generalizable and can be extended to targeting specific conformation generation without a target structure or collective variable at inference time.
(1.3) Line 302: The datasets are listed in the Methods section, 3.2, lines 143 and 144 in the original submission. However, you are correct that we did not clarify how we split with respect to the benchmark. We have added this to the methods section, and clarified that we removed proteins which have over 30% sequence identity and over 50% coverage, which is the same sequence identity metric we used for making the train/validation/test split.
(1.4) line 376: this is a typo from before adding the kinase. The timestep analysis here is not central to our claim, and has been removed from the main text to improve focus.
(2) Thank you for the comment. Part 1 helps motivate steering the hidden representations directly, and we show that ensemble and free energies can be predicted with a single pass. Our results are calculated from the most updated bioemu-benchmarks github (https://github.com/microsoft/bioemu-benchmarks), which is closer to the Science publication of BioEmu [1], in which they say “BioEmu achieved a mean absolute error of less than 0.9 kcal/mol and a Spearman correlation coefficient of approximately 0.6 for the MEGAscale test proteins,” which is fairly close to our predictions. I believe the (0.67, Fig. 4a) you cite is from the older bioArxiv submission from 2024. To clear up any confusion, we recalculate the Delta G correlation from the bioemu-benchmarks github pipeline in (Fig. 1e).
(3) This is a good comparison to add, so we’ve added a comparison to BioEmu’s own Feynman-Kac (FK) steering method [1] with R_g as the collective variable in (Fig. 2f). We find that our steering outperforms BioEmu’s FK steering in the median for reaching the requested target (0.20 vs. 0.24, where perfect is 0), and also has fewer large errors.