- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposes MERLE, a framework for sequential many-objective biological sequence optimization, specifically addressing settings where the initial observation dataset consists of prior, often biased, objective function evaluations (e.g., prior wet lab experiments). The method proposes a statistical framework to take into account both the effect of a positional mutation and the effect of the specific amino acid mutation. Through extensive benchmarking over 20 datasets against 5 multi-objective Bayesian optimization (BO) methods and 3 active learning methods, the authors demonstrate the utility of their method, which outperforms the evaluated baselines. Ablation studies further show the importance of each aspect of the developed method, with site-effect Thompson draw and position-specific response modeling having the largest effect on the model's performance.
Critical strengths
- The authors have performed comprehensive benchmarking against multiple baselines (of different families), as well as 20 biological datasets, solidifying their empirical results.
- The proposed statistical modeling introduces a clever inductive bias for the task of biological sequence optimization, taking into account both the effect of site mutation and individual amino acid mutations within each site.
- Extensive ablation studies were performed to isolate the effect of individual components of the proposed framework.
Critical weaknesses
- The phrase "representation distillation" could be potentially misleading. It is often used to describe the process of training a smaller, so-called student network to preserve the internal representation of a larger, more powerful model, often called the teacher [1]. Additionally, the formulation referred to as "representation distillation" in this work is inherited from [2] (as cited by the authors), which called this formulation "log odds ratio at the mutated position" to score mutations. Thus, for consistency with the literature, and to avoid confusion, I strongly recommend that the authors consider renaming the phrase "representation distillation" to a more appropriate term.
- Throughout the results section, in particular the tables, the standard error (SE) values aren't sufficiently taken into account when interpreting the results. For example, Table 2 is used to show that the strongest baseline is dependent on the initial observation dataset. However, the reported values show substantial overlap in their error ranges. In Table 1, MESMO seems to be the baseline according to the dHV metric (although it should be noted that there is still substantial overlap between the baselines). While the results of Table 2 clearly support that different initialization of the initial observation dataset affects the results, I believe the overlap of the SE values makes it difficult to single out a strongest baseline and subsequently argue that it depends on initialization. In Table 2, MESMO remains quite close to the underlined "strongest baseline" in the first two columns with substantial SE overlap.
- is used throughout the paper as the comparison metric. While this metric makes intuitive sense at first sight, it does lead to some relatively counterintuitive results. As an example, the "uncensored" initialization in Table 2 should provide a more informative starting dataset , as it samples from the entire library, as opposed to the default initialization used in Table 1 (and elsewhere), which samples from the bottom 75% of the library. However, the values of the "uncensored" column in Table 2 are substantially lower than the values in Table 1. I suspect that this is due to and, thus, . This is also the likely explanation behind the drop in performance of the strongest baseline in Figure 3, despite increasing the number of initial observations. To see the full picture, and values should be reported alongside , whether in the main text or the appendix. Additionally, as a sanity check, one would expect (and possibly ) to increase with increasing the offline repertoire . I recommend inlcuding both and alongside in Figure 3.
- In Table 5, the benchmark would benefit from comparison to other hierarchical optimization frameworks such as BoTier [3] or Chimera [4]. The current results might favor MERLE, as it is the only method where targeted demand optimization is possible, whereas all the baselines were not informed of the demand. Including baselines capable of taking into account demand hierarchy would make the results more meaningful.
- The results in Table 8 point to the position-specific response model being the biggest source of improvement. This raises questions about whether the proposed acquisition and allocation are necessary. Further experimentation with standard acquisition functions (e.g., UCB and standard Thompson sampling) could help clarify the contribution of the surrogate model further.
- The paper is missing a Discussion/Conclusion section.
[1] Tian, Y., Krishnan, D., & Isola, P. (2020). Contrastive Representation Distillation. International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=SkgpBJrtvS. [2] Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., & Rives, A. (2021). Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in Neural Information Processing Systems, 34, 29287–29303. https://proceedings.neurips.cc/paper/2021/hash/f51338d736f95dd42427296047067694-Abstract.html [3] Haddadnia, M., Grashoff, L., & Strieth-Kalthoff, F. (2025). BoTier: multi-objective Bayesian optimization with tiered objective structures. Digital Discovery, 4(6), 1417–1422. https://doi.org/10.1039/d5dd00039d [4] Häse, F., Roch, L. M., & Aspuru-Guzik, A. (2018). Chimera: enabling hierarchy based multi-objective optimization for self-driving laboratories. Chemical Science, 9(39), 7642–7655. https://doi.org/10.1039/c8sc02239a
Questions for authors
- Can the authors further clarify what they mean by "extrapolative search" and how that's different from the standard exploration-exploitation trade-off in standard Bayesian Optimization (BO)? One could argue that, in many BO applications, the exploration part of the algorithm is also extrapolative. Is "extrapolative" here referring to the initialization being focused on the bottom 75%?
- Can the authors please report values in addition to values (refer to point 3 in Critical Weaknesses)? Do the trends stay the same?
- In Section 5.3 (line 423), it is stated that "removing the site-effect Thompson draw ...". Can the authors please clarify what this was replaced with? Does this mean in Eq. 8?
- In Table 2, in the first column ("uncensored"), qEHVI has the exact same performance as STAGE-BO, but is not underlined. Can the authors please clarify whether this is due to rounding or a simple typo?
5 comments
Thank you again for the thoughtful questions. They helped us see where the paper needs to be clearer, and we answer them in order below.
Q1: what "extrapolative" means
- In our setting the initial data is censored at the 75% quantile of the library, so the high-performing region we want to expand into has no observation in at all (Sec. 3).
- Exploration in standard BO can also leave the observed range, but there this is a by-product of the acquisition rule. Here it is part of the task definition, since the target lies beyond and can only be reached by querying. This is why the teacher prior and the position model matter: they carry information to variants and conditions that never covers.
Q2: next to , and the trends
- The table with both levels is in our first reply (W3). The trends stay the same: MERLE has the higher under all five rules, and the uncensored rule is lower in only because is already 0.82.
- For the sanity check on the offline size (Fig. 3), grows steadily with , as you expected (averaged over the 19 tables that hold every size):
| 75 | 150 | 300 | 600 | 1200 | |
|---|---|---|---|---|---|
| 0.36 | 0.43 | 0.49 | 0.55 | 0.59 |
- This growth explains why of the strongest baseline falls as the repertoire grows. We will add and to Fig. 3.
Q3: what replaces the site-effect Thompson draw (Sec. 5.3)
- Your reading is correct. Without the draw we set in Eq. 8, so the site effect enters through its posterior mean and no noise term is added.
- Everything else in the model is left unchanged, so the ablation isolates the draw alone.
Q4: qEHVI and STAGE-BO in Tab. 2 ("uncensored")
- This is due to rounding: the two values agree at the printed precision.
You also found our benchmarking comprehensive, the inductive bias clever, and the ablations extensive, and we hope our answers have resolved your concerns. If so, we would be grateful if you would reconsider your rating, and if any question remains, we are happy to discuss it further.
Dear Reviewer r6c7,
One more point on W4 (hierarchical baselines such as BoTier and Chimera for the demand of Tab. 5). We did not run them in our first reply, and we have now.
- The demand of Tab. 5 has two tiers: three conditions the campaign is for, and a floor on the other three. We fixed the tiers from it before any run. For BoTier, tier 1 is the floor and tier 2 is the objective on the three conditions, counted only where the floor holds. For Chimera, tier 1 is that objective with a 10% tolerance and tier 2 is the other three.
- The demand gives no weights inside the three conditions, so we ran two readings, equal weights and a fresh random weight vector each step, and report both.
- Like every method in Tab. 5, both run on the shared surrogate, with the same libraries, seeds, budget and target evaluator. MERLE told the demand (dHV 0.527) leads all four, paired by library:
| Method | Weights | dHV | MERLE − method |
|---|---|---|---|
| BoTier | equal | 0.385 | +0.142* |
| BoTier | random | 0.403 | +0.123* |
| Chimera | equal | 0.390 | +0.136* |
| Chimera | random | 0.383 | +0.143* |
Note: * paired 95% interval over 20 libraries excludes zero.
We hope this resolves W4. Thank you again for your review.
Thank you for your careful review.
W1: the term "representation distillation"
W2: overlapping standard errors
W3: dHV versus HV
W4: hierarchical baselines (Tab. 5)
W5: acquisition versus surrogate
W6: We will add a Conclusion & Discussion section in next revised version.
[1] Hinton et al., 2015. Distilling the Knowledge in a Neural Network. [2] Tian et al., 2020. Contrastive Representation Distillation. [3] Meier et al., 2021. Language models enable zero-shot prediction of the effects of mutations on protein function. [4] Haddadnia et al., 2025. BoTier: multi-objective Bayesian optimization with tiered objective structures. [5] Häse et al., 2018. Chimera: enabling hierarchy based multi-objective optimization for self-driving laboratories.