- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Review
This was a challenging read and I apologize if my review sounds overly negative. It will also be quite long since I have a lot of questions.
The main text appears (to me) to be heavily LLM generated. I think the introduction does a fine job of setting the stage and motivating what prior research and results influenced this paper. It is however also immediately unclear to me what the actual contributions are. Clearly, REM Harness (RH) is proposed by the authors. But then the benchmark ViroHub is referred to as “a newly curated benchmark” without a reference; looking through the appendix, I believe it is also a contribution, but is never clearly stated. The same applies to VenusREM2. In the introduction, is says “VenusREM2 combines the harness with 6 variants of ProSST” but it is also not clearly stated that this model is introduced by the authors. Perhaps this is obvious to other reviewers, but I think a crucial job of the introduction is to clearly frame the paper and its contributions and I don’t think this is the case here. This results in a wobbly introduction, where I don’t know what to expect from the paper.
My main issue with the paper and reason for rejection is Section 2. Section 2 is overly complex and simultaneously verbose and overly concise: the introduction to the section is essentially a bullet point list later sentence become unreadable: “To allocate the complementary homolog evidence used by TRANCEPTION and VENUSREM (Notin et al., 2022a; Tan et al., 2025b), we distinguish a diagnostic-based target share from its scale-adjusted coefficient.”
A flood of new terms and quantities are constantly introduced without motivation whatsoever. I struggled to understand what the authors are actually proposing. Let’s take the Section 2 subsection titles as examples “ADAPTIVE RETRIEVAL AS SCALE ALLOCATION” and “COHERENCE-GATED CALIBRATION OF THE NATIVE CHANNEL”; I really don’t know what any of this means, even after reading the paper several times. What is “scale allocation” and what is “adaptive retrieval”? It isn’t clearly defined. What is the “native channel”? Solely based on the presentation, I cannot recommend acceptance of this paper. It needs a thorough revision by humans. If all text is human written then I apologize, but my points about verbosity, complexity, and general confusion still stand.
Paper summary
Now for the actual summary. The authors make three contributions: (1) a REM Harness method, which is a training-free method of adding MSA-derived information, structural plausibility via RSA/pLDDT, (2) VenusREM2 which is an application of the harness by which an ensemble of the existing ProSST models are wrapped in the REM Harness, and (3), I think, a ViroHub benchmark. As the ViroHub benchmark is only described in the appendix, I will not include in the review.
The REM Harness, which is the main contribution of the paper, builds on existing ideas and seeks to modulate existing variant effect predictions:
- MSA-derived per-site retrieval, akin to the retrieval mechanism in Tranception. A parameter is defined in terms of site uncertainties.
- Model scores are calibrated using to account for the model’s bias.
- Model scores are further calibrated given the exposure (via RSA) and structural uncertainty (via pLDDT).
While I think these components make sense, their motivation is lacking and they are mostly applied and ablated. Additionally, the ablations in table 1 show that each contribution as a marginal effect across benchmarks. This table does not contain any measure of uncertainty and the net effects on aggregated Spearman correlations range from 0.02 to 0.028. I would very much like to see statistical test for significance. Figure 13 in the appendix shows a 95 % assay-bootstrap interval and the overlap between e.g. the full RH pipeline and the effect of including MSA retrieval only does not appear significant.
Critical strengths
- State of the art scores on known benchmarks like ProteinGym is certainly a strength.
- Comprehensive experiments
Critical weaknesses
- Poor presentation. See above summary for detailed description. It is very challenging to read Section 2 and get an understanding of the exact mechanism the authors propose. While numerous equations define many quantities, few are adequately introduced, motivated, and described. To me, a lot of the text appears to be heavily LLM generated. For a concrete example, see L179: "The preference perspective of Gordon et al. (2025) motivates this direction; gated and ungated variants test the rule (Section 3.4)." This sounds like its directly taken from Claude. The over-the-top complexity obfuscates the REM Harness method. I think that simplicity is positive. If a simple, well-motivated component makes a positive contribution, I would prefer to let the simplicity speak for itself.
- Unclear contributions. It is clear that the REM Harness is the main contribution of the paper. VenusREM2 appears as a secondary contribution which is the harness wrapped around an ensemble of the existing ProSST suite. ViroHub is described as a novel benchmark but is not detailed in the text. Reading the introduction, it is not clear what the main contributions are.
- Lack of significance. Table 1 does not include measures of uncertainty and all reported scores are highly similar. This warrants statistical tests to identify whether the improvements from the RH components are statistically significant.
- Lack of novelty. This point is more a matter of personal taste. I find new contributions that keep hill climbing on the ProteinGym benchmark (and related benchmarks) uninspiring. While there are some neat ideas, continuous hill climbing will not solve any real problems.
Questions for authors
This might seem like a lot and I don't expect the reviewers to answer all of my questions during the rebuttal. I hope that the extent of my questions will underline the points I raised in the summary and weaknesses.
- L073: What does “scale-aware” mean? What is a “scale-aware mixing weight”? What "scale" are we referring to?
- L076: “ reversing mutation contrasts.”? What is a mutation contrast?
- L098: Using the term “backbone” in this context is a bit confusing to me since backbones are often used to refer to the structural backbone of the protein in question.
- L099: Equation 1 is confusing and I had to read this section a couple of times to understand the notation. The
tfshorthand is not explained - only in the appendix. I now see that equation aims to establish three types of “score fields”; one in the encoder-only setting, one in the decoder-only setting, and one for the wild-type sequence. This can be explained better. - L104: What is meant by the following: “Other native fields retain interface-specific scores, including unnormalized or window-aggregated outputs”? What is a “native field”? What is a “Raw-RH pair”?
- L107: What is a “query-mapped site”? Maybe the query is the wild-type sequence?
- L107: Here a tokenizer is explicitly mentioned; so am I right in assuming that we’re firmly in a language modeling paradigm?
- L107: What does “Equal row weights and no pseudocounts define aligned frequencies and a log-softmax family field; either field X uses wild-type-relative contrasts” mean? What are wild-type relative contrasts? If I had just finished reading the EVE/EVcouplings papers, this might make more sense, but a more thorough explanation of the terms you introduce would greatly help the reader.
- L145: What is the role of the mixing weight? What exactly is it mixing?
- L146: “To allocate the complementary homolog evidence used by TRANCEPTION and VENUSREM (Notin et al., 2022a; Tan et al., 2025b), we distinguish a diagnostic-based target share from its scale-adjusted coefficient.” What does this even mean?
- L151: What is z, and why are we clipping it?
- What is model–family Pearson alignment? None of the quantities or how they’re defined is motivated whatsoever.
- L179: “The preference perspective of Gordon et al. (2025) motivates this direction; gated and ungated variants test the rule (Section 3.4).” Perhaps I’m wrong, but this sounds so much like Claude.
- L205: What is RSA? What is pLDDT? I am well aware of the answers, but these terms need to be defined before being used. Which structure prediction model is used to get the structures and these quantities? Is it the wild-type structure? How reliable is that for the assays with very high mutation numbers?
- L240: It has not been explicitly stated that VenusREM2 is a contribution of this paper. Is that the case?
- L240: since ProSST is an integral part of VenusREM2, it warrants a more thorough introduction. What does it do differently? Why did you choose this model? What do the different vocabularies signify?
- What is ViroHub? Is it also a contribution by the authors in this paper? It is not clear from the main text.
- L305: “Across 148 ProteinGym proteins scored with ESM-2 wild-type readouts, the fraction of globally–locally separated sites correlates with retrieval gain” - what is meant by wild-type readout? What are globally-locally separate sites?
- L320: What is adaptive integration?
- L438: What is v_48_020? Not all readers know that this refers to a specific ProteinMPNN weight checkpoint name.
- Why wasn’t the existing ViroBench and ViroGym benchmarks used?
4 comments
Questions
Q1. "Scale" is the spread of substitution scores, measured by standard deviations and .
Thank you for asking us to define it more clearly. We have replaced "scale-aware mixing weight" with a direct explanation in Section 2.2(iv):
L183–186: The same mixing weight gives more influence to scores that vary more. We therefore adjust from Equation 6 for the variation in PFM and MSA scores. Mutation scores compare each replacement with the WT residue. Across all possible substitutions, the sample standard deviation of these score differences is for the PFM and for the MSA.
Equation (7) gives the resulting mixing coefficient (L187–190).
Q2. We replaced "mutation contrast" with the explicit difference between a replacement score and the WT score.
Thank you for pointing out the unclear wording. A structural weight between zero and one leaves the substitution score unchanged or shrinks it toward zero without reversing its sign (Section 2.4, L246–258; Appendix E.7, L2156–2159).
Q3. We meant the frozen PFM and have replaced the ambiguous use of "backbone."
Thank you for pointing out the possible confusion with protein structure. The text now uses "PFM" or "original model," and we updated the labels in Figure 5b.
Q4. Section 2.1 now explains the three scoring methods and defines teacher-forced scoring (tf).
We appreciate your suggestions for clarifying the notation. The revised definitions are:
- L100–101: "Masked scoring replaces each WT residue in turn with a mask token to form and predicts amino-acid probabilities at that position."
- L101–102: "WT scoring predicts probabilities at every position from the full unmasked WT sequence in one pass."
- L103–104: "Teacher-forced scoring (tf) predicts them from preceding WT residues in the decoding order."
Equation (1) (L104–107) defines how these probabilities give the log-probability scores.
Q5. We replaced "native field" with "PFM scores" and now describe the before-and-after comparison directly.
We appreciate your request for clearer descriptions. Section 2.1 (L098–099) now defines the PFM score matrix. Section 3.1 (L300–303) compares predictions before and after applying the harness to the same configuration. Appendix E.4 (L1887–1889) explains that ProtSSN returns logits and long ProSST sequences use log probabilities averaged over overlapping windows.
Q6. We replaced "query-mapped sites" with a direct description of how the MSA aligns to the WT sequence.
Thank you for pointing out the ambiguity. Yes, the query is the WT sequence. A "query-mapped site" is an MSA position aligned to a WT residue. Section 2.1 (L136–140) now identifies the WT sequence as the alignment reference. Appendix E.5 (L2000–2003) defines the positions used when MSA coverage is partial.
Q7. The tokenizer maps amino-acid symbols to columns of the PFM score matrix.
Thank you for raising this question. The MSA calculation itself is frequency counting. We now describe this step directly using amino-acid counts and frequencies in Section 2.1 (L136–142), with vocabulary and gap handling in Appendix E.4.
Q8. We separated MSA frequency counting from substitution scoring and removed the compressed sentence.
Thank you for the careful reading. We count each aligned sequence once, divide amino-acid counts by the number of sequences, and apply log-softmax (L136–142, Section 2.1, Equation (2)). "WT-relative contrast" meant the replacement amino-acid score minus the WT score at the same position. We now state this directly when introducing mutation scoring in Section 2.4 (L253–258).
Q9. controls the balance between PFM scores and MSA scores .
We appreciate your request to clarify what is being mixed. Equation (3) (L147–150) states
A larger gives more weight to the MSA scores. Section 2.2 introduces this relationship before explaining how we calculate for each protein and PFM.
Q10. We removed the unclear sentence and now explain the MSA weight calculation step by step.
Thank you for highlighting how difficult the sentence was to follow. We intended to describe two steps:
-
We first set the MSA weight from WT-residue probabilities and the correlation between PFM and MSA scores, using prediction entropy to combine the two measures.
-
We then adjust for the different standard deviations of their substitution scores to obtain .
Section 2.2 now introduces the weighted average and explains these steps in Equations (3)–(7) (L147–193).
Q11. Clipping keeps the WT-probability term in the interval [0, 1].
Thank you for noticing that was undefined. It denotes the input to
Very low WT probabilities can make the mean negative probability exceed one, so clipping caps the resulting weight. Section 2.2 (L156–162) and Appendix E.5 (L2007–2010) give the definition.
Q12. We compare the PFM and MSA vectors of 20 amino-acid scores at each position.
We appreciate your request for a definition and motivation. We calculate their Pearson correlation, then average across positions. Lower correlation suggests that the MSA may supply amino-acid preferences not reflected in the PFM scores. Section 2.2(ii) and Equation (5) (L164–171) explain this directly.
Q13. We removed the unclear sentence and explain the background correction directly.
Thank you for the feedback. Section 2.3 (L196–215) now describes how we estimate amino-acid preferences repeated across PFM positions and determine how much to subtract.
Q14. We now define RSA and pLDDT before using them and identify the structure source.
Thank you for pointing out these omissions. We should have introduced the abbreviations before using them.
-
Definitions and inputs (L241–245, Section 2.4). The revised text states:
After fusion and background correction, we use two measurements from the WT structure predicted by AlphaFold2 (Jumper et al., 2021): (i) pLDDT (predicted local distance difference test score) estimates confidence in the local structure. Low pLDDT indicates less reliable structural information (Zhang et al., 2024). (ii) RSA (relative solvent accessibility) measures residue exposure to solvent. Exposed residues often tolerate substitutions better than buried residues (Tsishyn et al., 2025).
-
Other abbreviations. We also spell out SARS-CoV-2, RBD, ZIKV, and E in the immune-escape analysis (L428–431, Section 3.6).
-
Variants with many substitutions. The scores use the WT sequence and structure and sum the individual substitution scores (L253–258). They therefore cannot fully capture mutation-induced structural changes or interactions. Figure 3 reports performance by mutation count, including five or more substitutions.
Q15. We now explicitly identify VenusREM2 as a contribution of this paper.
We are grateful that you flagged this omission. VenusREM2 combines six ProSST models with the harness and is now listed as the third contribution in the Introduction (L081–085). Section 2.5 (L263–287) explains the ensemble, and Section 3.3 reports its results.
Q16. We added an introduction to ProSST, our reason for choosing it, and the meaning of its vocabulary sizes.
We appreciate your request for a fuller introduction.
- ProSST [1] combines amino-acid tokens with quantized local-structure tokens through disentangled attention.
- We chose it for its mutation prediction performance and to build directly on VenusREM.
- The vocabulary sizes (20, 128, 512, 1024, 2048, and 4096) specify the number of structure-token categories.
The Introduction now introduces ProSST alongside VenusREM:
L053–055: Similarly, VenusREM combines MSA amino-acid frequencies with scores from ProSST, a sequence–structure Transformer with disentangled attention (Li et al., 2024a; Tan et al., 2025a).
Section 2.5 (L263–267) lists the six structure vocabularies, with inference details in Appendix E.4.
[1] Mingchen Li et al. ProSST: Protein Language Modeling with Quantized Structure and Disentangled Attention. NeurIPS 2024.
Q17. We now explicitly present ViroHub as a dataset contributed by this work.
Thank you for pointing out that its role was unclear. It contains 616,690 single-amino-acid substitutions from 89 assays, covering 18 virus types, 52 WT sequences, and seven phenotypes. The Introduction lists it as the second contribution, and Section 3.2 (L306–319) describes it. Appendix A.3 documents the sources and processing.
Q18. This analysis compares site-specific MSA scores with protein-average PFM preferences and log UniProt amino-acid frequencies.
We appreciate your careful reading. "WT readout" meant PFM scores from the unmasked WT sequence. A separated site has Pearson between its MSA profile and at least one of two references: PFM scores averaged across positions in the same protein or log UniProt amino-acid frequencies. This differs from the site-specific PFM–MSA correlation used to calculate . Section 3.4 and Appendix C.1 define the analysis.
Q19. We now use "adaptive MSA fusion" consistently and refer to its definition in Section 2.2.
Thank you for flagging the inconsistent terminology. This is the weighted average of PFM and MSA scores, with calculated for each protein and PFM (L152–193). Section 3.4 evaluates its effect on prediction.
Q20. We replaced the unexplained checkpoint name with "ProteinMPNN" in the main text.
Thank you for flagging this. The checkpoint details for ProteinMPNN [2] are now explained in Appendix G.3:
L2287–2289: In the identifier
v_48_020, 48 specifies the graph neighborhood size and 020 denotes the standard deviation (0.20 Å) of Gaussian noise added to backbone coordinates during training (Dauparas et al., 2022).
[2] Dauparas et al. Robust deep learning-based protein sequence design using ProteinMPNN. Science, 2022.
Q21. We removed the ViroBench citation and clarified how ViroHub expands on ViroGym.
We appreciate your question about these benchmarks.
- ViroBench [3] concerns nucleic acids. We removed its citation from the paper to avoid confusion.
- Compared with ViroGym [4], ViroHub provides broader viral protein mutation coverage: 89 assays across 18 virus types, including 21 immune-escape and 18 cell-entry assays, versus 14 and 11 in ViroGym. Section 3.2 (L309–312) states this comparison, and Appendix A.3 gives the sources and processing details.
[3] Dongxin Ye et al. ViroBench: Benchmarking Nucleotide Foundation Models on Viral Genomics Tasks. KDD 2026. [4] Yichen Zhou et al. ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins. arXiv, 2026.
We sincerely thank the reviewer for the careful reading, detailed suggestions, and recognition of our comprehensive experiments and predictive performance. We address the presentation and technical concerns below.
Weaknesses
W1. Poor presentation.
The Method now explains each step's purpose before introducing its calculation.
Thank you for your candid feedback. We agree that a simple method should be explained plainly. We simplified the subsection titles, clarified the definitions, and removed the sentence you quoted.
For example, Section 2.3 now explains the purpose of background correction before introducing its equation:
W2. Unclear contributions.
The Introduction now presents our method, benchmark, and ensemble model as three distinct contributions (L73–85).
We appreciate your suggestion to make their roles explicit:
REM-Harness is a general, training-free framework for frozen PFMs. Across three benchmarks, all 71 configurations improve Spearman correlation, with an average gain of 0.073.
ViroHub is a community benchmark containing 616,690 substitutions from 89 assays across 18 virus types, supporting studies of immune escape and other viral phenotypes. Section 3.2 (L306–319) describes its scope and data curation, and Table 1 compares it with ProteinGym and VenusMutHub.
VenusREM2 combines six ProSST checkpoints with the harness, reaching 0.556 Average Spearman in the official ProteinGym comparison. Section 2.5 explains the ensemble. Compared with VenusREM's loops, MSA counting and mutation scoring are 62.3× and 20.7× faster, respectively, after model inference.
W3. Lack of significance.
For ProSST-2048, the full harness significantly improves on adaptive MSA fusion on all three full benchmarks.
Thank you for requesting this analysis. We added 95% confidence intervals and two-sided p-values for paired differences between component settings. We use 20,000 paired bootstrap samples for the intervals and 20,000 paired group-wise permutations for the p-values, retaining each benchmark's aggregation.
The table below summarizes the full benchmarks. The added SaProt ablation also shows a gain from the full harness beyond adaptive fusion on full ProteinGym (0.01257, p < 0.0001; Appendix Table 29).
Each entry gives the two-sided p-value (Yes if p < 0.05; No otherwise). Score differences and 95% confidence intervals are reported in Appendix Table 27 for ProSST and Table 29 for SaProt.
W4. Lack of novelty.
The work contributes a reusable prediction framework, a viral dataset, and biological analysis alongside benchmark gains.
Thank you for raising the question of scientific value beyond benchmark scores.