acceptodds
Under review as a conference paper at ICLR 2027

MERLE: Many-Objective Extrapolative Search via Representation Distillation for Biological Landscape Explainability

Abstract

Biological landscape search balances many objectives with limited measurements. Restricted initial observations demand extrapolation under substantial predictive uncertainty. We study sequential many-objective search where each query reveals a complete response profile, aiming to expand the observed Pareto frontier under a fixed measurement budget. We introduce MERLE, which distills pretrained mutation preferences into interpretable, objective-specific priors and combines them with an adaptive position-level response model. Together they guide frontier search and reveal which positions matter to the campaign. At each position, the model pools substitutions to estimate assay effects and uncertainty. MERLE scores candidates against unmet frontier requirements and allocates queries across teacher-guided, position-only, and embedding-coverage proposals based on observed frontier gains. Across diverse landscapes and altered campaign settings, MERLE improves frontier discovery over competitive baselines. Ablations identify when the teacher and position field help; site deletion and independent-assay measurements link frontier gains to consequential, reproducible choices of mutation positions across laboratories and distinct antibody and serum panels studied.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
multi-objective optimization, representation learning, explainable machine learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_r6c7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes MERLE, a framework for sequential many-objective biological sequence optimization, specifically addressing settings where the initial observation dataset consists of prior, often biased, objective function evaluations (e.g., prior wet lab experiments). The method proposes a statistical framework to take into account both the effect of a positional mutation and the effect of the specific amino acid mutation. Through extensive benchmarking over 20 datasets against 5 multi-objective Bayesian optimization (BO) methods and 3 active learning methods, the authors demonstrate the utility of their method, which outperforms the evaluated baselines. Ablation studies further show the importance of each aspect of the developed method, with site-effect Thompson draw and position-specific response modeling having the largest effect on the model's performance.

Critical strengths
  1. The authors have performed comprehensive benchmarking against multiple baselines (of different families), as well as 20 biological datasets, solidifying their empirical results.
  2. The proposed statistical modeling introduces a clever inductive bias for the task of biological sequence optimization, taking into account both the effect of site mutation and individual amino acid mutations within each site.
  3. Extensive ablation studies were performed to isolate the effect of individual components of the proposed framework.
Critical weaknesses
  1. The phrase "representation distillation" could be potentially misleading. It is often used to describe the process of training a smaller, so-called student network to preserve the internal representation of a larger, more powerful model, often called the teacher [1]. Additionally, the formulation referred to as "representation distillation" in this work is inherited from [2] (as cited by the authors), which called this formulation "log odds ratio at the mutated position" to score mutations. Thus, for consistency with the literature, and to avoid confusion, I strongly recommend that the authors consider renaming the phrase "representation distillation" to a more appropriate term.
  2. Throughout the results section, in particular the tables, the standard error (SE) values aren't sufficiently taken into account when interpreting the results. For example, Table 2 is used to show that the strongest baseline is dependent on the initial observation dataset. However, the reported dHV\textrm{dHV} values show substantial overlap in their error ranges. In Table 1, MESMO seems to be the baseline according to the dHV metric (although it should be noted that there is still substantial overlap between the baselines). While the results of Table 2 clearly support that different initialization of the initial observation dataset D0D_0 affects the results, I believe the overlap of the SE values makes it difficult to single out a strongest baseline and subsequently argue that it depends on D0D_0 initialization. In Table 2, MESMO remains quite close to the underlined "strongest baseline" in the first two columns with substantial SE overlap.
  3. dHV=HV(YB)−HV(Y0)\textrm{dHV} = \textrm{HV}(Y_B) - \textrm{HV}(Y_0) is used throughout the paper as the comparison metric. While this metric makes intuitive sense at first sight, it does lead to some relatively counterintuitive results. As an example, the "uncensored" initialization in Table 2 should provide a more informative starting dataset D0D_0, as it samples from the entire library, as opposed to the default initialization used in Table 1 (and elsewhere), which samples from the bottom 75% of the library. However, the values of the "uncensored" column in Table 2 are substantially lower than the dHV\textrm{dHV} values in Table 1. I suspect that this is due to HV(Y0uncensored)>HV(Y0default)\textrm{HV}(Y^{\textrm{uncensored}}_0) > \textrm{HV}(Y^{\textrm{default}}_0) and, thus, dHVuncensored<dHVdefault\textrm{dHV}^{\textrm{uncensored}} < \textrm{dHV}^{\textrm{default}}. This is also the likely explanation behind the drop in performance of the strongest baseline in Figure 3, despite increasing the number of initial observations. To see the full picture, HV(YB)\textrm{HV}(Y_B) and HV(Y0)\textrm{HV}(Y_0) values should be reported alongside dHV\textrm{dHV}, whether in the main text or the appendix. Additionally, as a sanity check, one would expect HV(Y0)\textrm{HV}(Y_0) (and possibly HV(YB)\textrm{HV}(Y_B)) to increase with increasing the offline repertoire NN. I recommend inlcuding both HV(YB)\textrm{HV}(Y_B) and HV(Y0)\textrm{HV}(Y_0) alongside dHV\textrm{dHV} in Figure 3.
  4. In Table 5, the benchmark would benefit from comparison to other hierarchical optimization frameworks such as BoTier [3] or Chimera [4]. The current results might favor MERLE, as it is the only method where targeted demand optimization is possible, whereas all the baselines were not informed of the demand. Including baselines capable of taking into account demand hierarchy would make the results more meaningful.
  5. The results in Table 8 point to the position-specific response model being the biggest source of improvement. This raises questions about whether the proposed acquisition and allocation are necessary. Further experimentation with standard acquisition functions (e.g., UCB and standard Thompson sampling) could help clarify the contribution of the surrogate model further.
  6. The paper is missing a Discussion/Conclusion section.

[1] Tian, Y., Krishnan, D., & Isola, P. (2020). Contrastive Representation Distillation. International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=SkgpBJrtvS. [2] Meier, J., Rao, R., Verkuil, R., Liu, J., Sercu, T., & Rives, A. (2021). Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in Neural Information Processing Systems, 34, 29287–29303. https://proceedings.neurips.cc/paper/2021/hash/f51338d736f95dd42427296047067694-Abstract.html [3] Haddadnia, M., Grashoff, L., & Strieth-Kalthoff, F. (2025). BoTier: multi-objective Bayesian optimization with tiered objective structures. Digital Discovery, 4(6), 1417–1422. https://doi.org/10.1039/d5dd00039d [4] Häse, F., Roch, L. M., & Aspuru-Guzik, A. (2018). Chimera: enabling hierarchy based multi-objective optimization for self-driving laboratories. Chemical Science, 9(39), 7642–7655. https://doi.org/10.1039/c8sc02239a

Questions for authors
  1. Can the authors further clarify what they mean by "extrapolative search" and how that's different from the standard exploration-exploitation trade-off in standard Bayesian Optimization (BO)? One could argue that, in many BO applications, the exploration part of the algorithm is also extrapolative. Is "extrapolative" here referring to the initialization being focused on the bottom 75%?
  2. Can the authors please report HV(YB)\textrm{HV}(Y_B) values in addition to dHV\textrm{dHV} values (refer to point 3 in Critical Weaknesses)? Do the trends stay the same?
  3. In Section 5.3 (line 423), it is stated that "removing the site-effect Thompson draw ...". Can the authors please clarify what this was replaced with? Does this mean θp,c=μp,c\theta_{p,c} = \mu_{p,c} in Eq. 8?
  4. In Table 2, in the first column ("uncensored"), qEHVI has the exact same performance as STAGE-BO, but is not underlined. Can the authors please clarify whether this is due to rounding or a simple typo?
5 comments

Thank you for your careful review.

W1: the term "representation distillation"

  • We use "distillation" in its broader sense, where a teacher is compressed into a compact artifact for a small learner. This includes output-level distillation [1], not only hidden-state matching [2] (Sec. 4.1).
  • The log-odds score comes from [3]; we add its signed transfer to each condition (Tab. 7).

W2: overlapping standard errors

  • The ±\pm in Tab. 1 and Tab. 2 is the spread across tables (protein difficulty), so we compare methods paired within each table, where MERLE's margin over the strongest baseline excludes zero in four of the five rules.

W3: dHV\mathrm{dHV} versus HV\mathrm{HV}

  • As you suspected, a larger HV(Y0)\mathrm{HV}(Y_0) leaves less room for dHV\mathrm{dHV}, hence our headroom shares. Both levels are below (default: Tab. 1, rest: Tab. 2); MERLE ends higher under every rule. We will add them to Tab. 2, Fig. 3:
D0D_0 ruleHV(Y0)\mathrm{HV}(Y_0)HV(YB)\mathrm{HV}(Y_B): MERLE / best baselineMargin
default0.490.86 / 0.80+0.06
uncensored0.820.92 / 0.88+0.04
band0.500.85 / 0.80+0.04
sites0.490.85 / 0.82+0.03
single0.310.87 / 0.81+0.06

W4: hierarchical baselines (Tab. 5)

  • That is by design: Tab. 5 asks what a demand input adds, and the panel, BOtied included, has none.
  • BoTier [4] and Chimera [5] address a different problem: they need a tier order and thresholds, while our demand is a distribution over all six conditions.
  • It is not only the position model: with MESMO and STAGE-BO on that model, scored under the demand of Tab. 5, MERLE configured for it still leads by 0.04 to 0.05 (intervals exclude zero).

W5: acquisition versus surrogate

  • We mounted the position model under four standard rules (new, Tab. 8). MERLE stays ahead of each, and its lead is larger early in the budget and under a demand:
RuleMERLE - rule: full budgetwhole budgetdemand
RHVS-UCB+0.04*+0.04*+0.08*
RHVS-TS+0.03*+0.04*+0.05*
STAGE-BO+0.02+0.03*+0.05*
MESMO+0.01+0.03*+0.04*
  • Here, * paired 95% interval excludes zero. "Whole budget": dHV averaged over all queries; "demand": MERLE configured for Tab. 5's demand.
  • The lead over MESMO and STAGE-BO is larger for a small D0D_0 too: with 75 profiles it is 0.03 and 0.04 (intervals exclude zero).

W6: We will add a Conclusion & Discussion section in next revised version.

[1] Hinton et al., 2015. Distilling the Knowledge in a Neural Network. [2] Tian et al., 2020. Contrastive Representation Distillation. [3] Meier et al., 2021. Language models enable zero-shot prediction of the effects of mutations on protein function. [4] Haddadnia et al., 2025. BoTier: multi-objective Bayesian optimization with tiered objective structures. [5] Häse et al., 2018. Chimera: enabling hierarchy based multi-objective optimization for self-driving laboratories.

Thank you again for the thoughtful questions. They helped us see where the paper needs to be clearer, and we answer them in order below.

Q1: what "extrapolative" means

  • In our setting the initial data D0D_0 is censored at the 75% quantile of the library, so the high-performing region we want to expand into has no observation in D0D_0 at all (Sec. 3).
  • Exploration in standard BO can also leave the observed range, but there this is a by-product of the acquisition rule. Here it is part of the task definition, since the target lies beyond D0D_0 and can only be reached by querying. This is why the teacher prior and the position model matter: they carry information to variants and conditions that D0D_0 never covers.

Q2: HV(YB)\mathrm{HV}(Y_B) next to dHV\mathrm{dHV}, and the trends

  • The table with both levels is in our first reply (W3). The trends stay the same: MERLE has the higher HV(YB)\mathrm{HV}(Y_B) under all five D0D_0 rules, and the uncensored rule is lower in dHV\mathrm{dHV} only because HV(Y0)\mathrm{HV}(Y_0) is already 0.82.
  • For the sanity check on the offline size NN (Fig. 3), HV(Y0)\mathrm{HV}(Y_0) grows steadily with NN, as you expected (averaged over the 19 tables that hold every size):
NN751503006001200
HV(Y0)\mathrm{HV}(Y_0)0.360.430.490.550.59
  • This growth explains why dHV\mathrm{dHV} of the strongest baseline falls as the repertoire grows. We will add HV(YB)\mathrm{HV}(Y_B) and HV(Y0)\mathrm{HV}(Y_0) to Fig. 3.

Q3: what replaces the site-effect Thompson draw (Sec. 5.3)

  • Your reading is correct. Without the draw we set θp,c=μp,c\theta_{p,c}=\mu_{p,c} in Eq. 8, so the site effect enters through its posterior mean and no noise term is added.
  • Everything else in the model is left unchanged, so the ablation isolates the draw alone.

Q4: qEHVI and STAGE-BO in Tab. 2 ("uncensored")

  • This is due to rounding: the two values agree at the printed precision.

You also found our benchmarking comprehensive, the inductive bias clever, and the ablations extensive, and we hope our answers have resolved your concerns. If so, we would be grateful if you would reconsider your rating, and if any question remains, we are happy to discuss it further.

Dear Reviewer r6c7,

One more point on W4 (hierarchical baselines such as BoTier and Chimera for the demand of Tab. 5). We did not run them in our first reply, and we have now.

  • The demand of Tab. 5 has two tiers: three conditions the campaign is for, and a floor on the other three. We fixed the tiers from it before any run. For BoTier, tier 1 is the floor and tier 2 is the objective on the three conditions, counted only where the floor holds. For Chimera, tier 1 is that objective with a 10% tolerance and tier 2 is the other three.
  • The demand gives no weights inside the three conditions, so we ran two readings, equal weights and a fresh random weight vector each step, and report both.
  • Like every method in Tab. 5, both run on the shared surrogate, with the same libraries, seeds, budget and target evaluator. MERLE told the demand (dHV 0.527) leads all four, paired by library:
MethodWeightsdHVMERLE − method
BoTierequal0.385+0.142*
BoTierrandom0.403+0.123*
Chimeraequal0.390+0.136*
Chimerarandom0.383+0.143*

Note: * paired 95% interval over 20 libraries excludes zero.

We hope this resolves W4. Thank you again for your review.

Reviewer_z59qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MERLE addresses many-objective protein variant selection under a limited measurement budget. It compresses pretrained mutation preferences into scalar priors and combines objective-specific transfer coefficients, a position-level response model, and adaptive query allocation to expand the observed Pareto frontier. The experiments demonstrate improvements across several campaign settings. However, the distinct methodological contribution, biological interpretability, and scope of applicability are not sufficiently established.

Critical strengths

Broad experimental coverage. The evaluation includes multiple datasets and random seeds, competitive optimization and active-learning baselines, and variations in initialization, partial feedback, measurement noise, and batch size.

Reasonable handling of pretrained priors. Objective-specific signed coefficients and held-out shrinkage avoid assuming that pretrained mutation preferences are uniformly beneficial across assays.

Critical weaknesses

The methodological novelty requires clearer positioning. Representation distillation primarily consists of caching scalar pretrained mutation scores and learning their objective-specific contributions through regression, rather than distilling hidden representations or training a student network. While useful, its substantive advance over pretrained-score calibration, position-effect modeling, and adaptive sampling needs clearer articulation.

The benefit of the teacher prior is strongly conditional. Ablations indicate that its contribution is concentrated in the seven largest candidate libraries, with no clear improvement established for the other thirteen. Paired confidence intervals for cross-campaign prior-transfer gains also include zero. These findings limit broad attribution of the performance gains to representation distillation and call for a sharper characterization of when it adds value beyond the position model.

Biological interpretability needs stronger substantiation. Position effects and transfer coefficients provide readable statistical explanations, but do not directly establish biological mechanisms or causality. The reported site-deletion and independent-assay analyses should clarify whether they control for confounders such as sampling density, available substitutions, and response magnitude, and whether they reveal insights beyond previously known sensitive positions.

The task scope and biological significance need clearer presentation. Fixed-library replay experiments do not directly establish applicability to open-ended sequence design or combinatorial multi-site optimization. The main text should also identify the proteins, assay responses, and their biological meaning more explicitly, so that the practical significance of frontier expansion can be assessed without relying primarily on the appendix.

Questions for authors

Above

1 comment

Thank you for the review.

W1: the term "distillation"

  • We use the word in its broader sense: a large model is compressed into a compact artifact a small learner can use. This includes output-level distillation [1], not only hidden-state matching [2], so our usage fits it (Sec. 4.1).
  • In our method, each position's ESM-2 distribution becomes one cached score per variant [3], and the response model learns its signed transfer to each condition from held-out responses.
  • Each part is ablated separately: the position field raises dHV from 0.31 to 0.36 at a fixed acquisition (Tab. 8), the site-effect Thompson draw helps across the panel, and the prior adds most in large libraries (Tab. 7).
  • Adaptive sampling adds to this: on the same position model, MERLE leads MESMO, STAGE-BO, UCB and Thompson by 0.03 to 0.04 in dHV averaged over the whole budget (new, paired intervals exclude zero), and it also leads MESMO and STAGE-BO with a small D0D_0 of 75 profiles.

W2: the teacher

  • The teacher is built for budgets that cover few variants, and that is where it helps: removing it costs about 0.04 dHV on the 7 largest libraries, and the loss grows with library size (Tab. 7, Fig. 5).
  • This is its gain on top of the position field, as we remove it from the full model.
  • Tab. 6 asks a different question, whether a fitted prior carries over to new conditions, and all four policies with a surrogate gain from carrying it.

W3: interpretability

  • We test by intervention (Tab. 10) whether the readout shows what the search relies on: we forbid k positions and rerun the campaign, and random positions matched on candidates removed cost almost nothing.
  • We added two stricter controls for Tab. 10, matched on visit density and response magnitude; their paired intervals against the top positions exclude zero.
Change in dHVk=5k=40
top-ranked positions-0.049-0.038
density-matched-0.001+0.005
magnitude-matched-0.005-0.009
  • Separate assays of one protein revisit the same sites far more than chance (Tab. 11), and our readout beats a sequence-only ranking as a fixed rule (Tab. 23).

W4: scope

  • Fixed-library replay lets us know the true Pareto frontier, so frontier expansion is measured exactly.
  • Each table's protein and assay are listed in App. D.1 (Tab. 14), and we will add a short version to Sec. 5. Seventeen of the 20 are viral escape scans (conditions are antibodies or sera), plus TEM-1 and VIM-2 lactamases under antibiotics and Hsp90 across environments.

You also noted the broad coverage and careful handling of pretrained priors. We hope our answers resolve your concerns; if so, please consider revising your rating, and we welcome further questions.

[1] Hinton et al., 2015. Distilling the Knowledge in a Neural Network. [2] Tian et al., 2020. Contrastive Representation Distillation. [3] Meier et al., 2021. Language models enable zero-shot prediction of the effects of mutations on protein function.

Reviewer_EDdBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies sequential many-objective selection from a fixed protein-variant library under a limited measurement budget. MERLE combines pretrained mutation preferences, signed condition-specific prior coefficients, and a position-level response model that shares evidence across substitutions. It uses frontier-oriented acquisition and embedding coverage to select further measurements, with each query revealing a complete response profile. The evaluation includes 20 response tables with five seeds per table, altered campaign settings, component ablations, and position-readout analyses.

Critical strengths
  1. The combination of condition-specific prior coefficients and a position-sharing response model is well motivated. A substitution favored by a pretrained model need not help every measured condition, so learning the direction and strength of transfer is preferable to treating the pretrained score as a universal fitness function. Table 8 shows a substantial gain from the position model even when the MESMO acquisition rule is held fixed, providing direct support for the central modeling choice.
  2. The evaluation is relatively broad. It includes 20 response tables, five seeds per table, and changes to the initial repertoire, noise, partial feedback, and batching. The position-deletion tests match random controls by the number of candidate variants removed and rerun the campaign. This gives the readout analysis stronger evidence than a qualitative visualization alone.
Critical weaknesses
  1. The main comparison does not directly establish superiority over the original full implementations of all named baselines. Appendix D.3 uses a shared Bayesian linear response model for multiple baselines, while Table 8 shows that replacing this model with the position field accounts for much of the gain. This is a reasonable controlled design, but the conclusion needs to be restricted accordingly. A comparison of the position model with another strong acquisition rule would help establish the additional benefit of the full method. Since the teacher and coverage components mainly help in larger libraries, their practical cost should also be considered when explaining where the full construction is worthwhile.
  2. The additional decision value of the numerical explanation weights remains limited. The deletion tests show that the readout reflects positions on which the search depends, but Table 22 does not show an improvement over a rule using only the previously visited positions. The cross-assay results mainly support recurring site selection rather than stable rankings of biological effects. The paper acknowledges these limitations. I therefore credit the explanation of search behavior, but do not regard improved downstream decisions from the numerical explanation weights as an established contribution.
Questions for authors
  1. What is the paired uncertainty in the gain of full MERLE over the position model with MESMO acquisition in Table 8? A further combination with another strong acquisition rule would help determine how much of the advantage specifically requires the proposed acquisition construction.
  2. Given the visit-log control in Table 22, which decision benefits in the paper specifically require the numerical explanation weights, rather than the history of visited positions? Please clarify the intended scope of the explanation claim in the main contribution statement.
1 comment

Thank you for the careful read.

W1 and Q1: baselines and the position model

  • Every baseline gets the same linear model (App. D.3) so that model and acquisition are not confounded; the position field is part of MERLE.
  • To test the step you suggest, we put the field under four literature rules (new, next to Tab. 8). It helps all of them, and MERLE (0.38) stays ahead of each on the same model. Its lead is larger earlier in the budget and under a demand:
RuledHV lineardHV with fieldMERLE - rule: full budgetwhole budgetdemand
MESMO0.310.36+0.01+0.03*+0.04*
STAGE-BO0.310.36+0.02+0.03*+0.05*
RHVS-TS0.290.35+0.03*+0.04*+0.05*
RHVS-UCB0.310.33+0.04*+0.04*+0.08*
  • Note: * paired 95% interval excludes zero. "Whole budget" is dHV averaged over all queries; "demand" is MERLE configured for the demand of Tab. 5.
  • The lead is also larger for a small D0D_0: with 75 profiles, MERLE beats MESMO and STAGE-BO on the same model by 0.03 and 0.04 (intervals exclude zero).

W1: cost of the full construction

  • The teacher takes one cached ESM-2 pass per position, shared across conditions and campaigns. A MERLE campaign then runs in under a CPU-second, against about seven for MESMO or STAGE-BO (App. C.6).
  • The teacher pays off most on the 7 largest libraries, where removing it costs about 0.04 dHV, and it does not resolvably hurt on the other 13 (Tab. 7). The pass is paid once per protein and returns most where libraries are large.

W2 and Q2: numerical weights

  • We claim that the weights rank the positions the search depends on. Deleting the top 5 costs far more than deleting the bottom 5 (Tab. 10), and a control matched on visit count costs almost nothing (new, for Tab. 10).
  • When we commit the readout as a rule before seeing any new response, it does at least as well as adaptive MERLE run without a head start (Tab. 22). A visit log reaches the same level because it records the same search, so our claim is that the readout explains what the search depends on, which we will state in Sec. 1.
  • Among the rankings we compared, ours gives the best fixed rule, ahead of the PLM alone and the shared surrogate (Tab. 23).

Ethics flag

  • This is a retrospective replay on published mutational-scanning tables. The policy only picks among variants already measured, generates no sequences and runs no assay.
  • The output is a ranking inside a measured table, not a new sequence. App. F discusses dual use, and we will add release safeguards there, such as stating that the policy should not be used to propose unmeasured variants without expert review.

You also found the position-sharing model well motivated and the evaluation broad. We hope our answers resolve your concerns; if so, please consider revising your rating, and we welcome further questions.

Comments on the paper

5 comments

Dear reviewers and AC,

Thanks for reading so closely. We've replied in each thread; this note is for all.

What the paper adds, and what you liked

  • Where we think the novelty is:
    • The problem: many-objective search past censored initial data, on a small query budget (Sec. 3).
    • The method: a cached PLM score with a learned per-condition transfer, plus a response field shared across a position (Sec. 4).
    • The experiments: fixed-library replay with a known Pareto frontier, covering unusual scenarios, like a demand concentrated on three conditions (Tab. 5), partial profiles and noisy measurements (Tab. 4), batches of 8 or 32 (Fig. 4), and a second campaign on new conditions (Tab. 6).
  • You also noted close strengths:
    • (EDdB) a "well motivated" prior-plus-position model, and deletion tests with matched controls.
    • (z59q) "broad experimental coverage" and a "reasonable handling of pretrained priors".
    • (r6c7) "comprehensive benchmarking", "a clever inductive bias" and "extensive ablation studies".

On "representation distillation"

  • We use the word the way it started out. Bucilă et al. [1] squeezed a large model into a small one trained on its outputs, and Hinton et al. [2] showed that the teacher's softened outputs carry the knowledge. Even in protein language model work the word gets stretched: [3] uses it to split large-model embeddings into subspaces, with no student trained. Matching hidden states [4] is just one flavour.
  • In MERLE we call the teacher once per position and cache one standardised score per variant (Sec. 4.1, Eq. 4; the masked log-odds of [5]). The search relies on what we learn on top: a signed coefficient γc\gamma_c per condition, shrunk with held-out data (Eq. 5, 6), so a preference counts only where an assay backs it.

On "extrapolative"

  • By default the initial data comes from the bottom 75% of the library by best response, so nothing from the top quarter is in it.
  • dHV\mathrm{dHV} counts the gain beyond that data's frontier, so every gain has to come from variants outside its range. That takes signal that reaches past the data: the teacher score and the position field.
  • BO exploration can wander outside the observed range as a side effect; here, getting out there is the task (Sec. 3).

On the evidence

  • The headline: MERLE ends ahead of the strongest baseline under every way we build the initial data, and the paired comparison clears zero by default (Tab. 1, 2).
  • We ran 20 response tables, five seeds each, against eight baselines (five multi-objective BO, three active learning), plus random and a white-box oracle.
  • All methods share features and evaluator (App. D.3).
  • We also varied the initial data: five ways to build it (Tab. 2), other sizes (Fig. 3).
  • Differences are paired by table and bootstrapped over tables.

We hope this helps. If it addresses your concerns, we'd be grateful if you could reflect that in your overall assessment, and we're happy to discuss anything unclear.

[1] Bucilă et al., 2006. Model Compression. KDD. [2] Hinton et al., 2015. Distilling the Knowledge in a Neural Network. NIPS 2014 Deep Learning Workshop. [3] Catrina et al., 2026. Reverse Distillation: Consistently Scaling Protein Language Model Representations. ICLR. [4] Tian et al., 2020. Contrastive Representation Distillation. ICLR. [5] Meier et al., 2021. Language models enable zero-shot prediction of the effects of mutations on protein function. NeurIPS.

Dear reviewers,

Thank you for your comments. After some thought, we decided to change the title slightly, following the concern about "representation distillation" raised by r6c7 and z59q. The new title is "MERLE: Many-Objective Extrapolative Search via Reversible Priors for Biological Landscape Exploration", for three reasons:

  • "Distillation" can be confused with the related work on distillation, where a student model is trained to match a teacher.
  • "Reversible" names what is new: a signed coefficient lets each assay follow, flip, or ignore the pretrained mutation score.
  • "Exploration" fits a search problem better, since the task is to explore a landscape that has not been measured yet.

We have uploaded a revised version, edited carefully according to your comments (please view it on a desktop browser, since the mobile page may still show the old version). The main changed text is in blue. Some of the changes:

  • Terminology: "representation distillation" => "reversible prior"; "teacher" => "prior".
  • New results: the position field under four literature rules (EDdB, r6c7); hypervolume levels next to dHV (r6c7); deletion controls matched on visit count and response size (z59q); the sign of the transfer coefficient across libraries.
  • Clarity: key definitions as equations, and a new Conclusion and Discussion section (r6c7).

We hope our earlier answers and the revised version address your concerns. We would be glad to hear your feedback, and we hope you will find the contributions of this paper worth reconsidering your ratings.

Best regards.

Dear reviewers,

There is about three days left in the discussion period. We know you are busy with your own submissions and the other papers you reviewed, and we would simply like to invite you back to this discussion. Your time and thoughts are truly valuable to us.

We have put our best effort into this work. We hope you will weigh its novelty, its soundness, and the quality of its experiments. Have your concerns been resolved? And if some remain, are they serious enough to justify rejecting the paper, or are they minor and fixable when weighed against the scientific value it offers? We would appreciate it if you could take this into account in your overall rating.

With many thanks, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.