acceptodds
Withdrawn from ICLR 2027

Four Points or Nothing: What a Batch Can and Cannot Measure at Its Own Leads

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Campaigns in protein engineering and molecular design repeatedly ask a local question at the current best candidate: do these two changes interact here? We show that a batch can be full rank for the second-order interaction model and still contain no assumption-free measurement of a single one of the local interactions at its own leads: global identifiability does not imply local measurability. The estimand is the four-point contrast of a double-mutant cycle, so a model-free linear estimator of it, unbiased under every response function, exists if and only if all four points have been observed, and is then unique in that class. Auditing twelve batch rules on four real near-complete landscapes shows this is not an artefact of classical criteria: determinantal, gradient-embedding and redundancy-aware information batches reach the same near-zero certification as Bayesian -optimality by three different routes, while naive exploitation certifies the most by accident. Three structural causes explain it, two computable from the proposed batch alone: separation to minimum Hamming distance two, occupancy below a classical extremal threshold, and rectangles away from the leads. An exact bias identity then gives a parameter-free boundary between direct measurement and the exact second-order projection, which a fitted model tracks but does not obey. It holds on 93 of 93 coordinate pairs, and a factorial sweep crossing it in both arguments puts the oracle crossing where the identity does.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
batch experimental design, batch active learning, Bayesian optimal design, identifiability, local interactions, epistasis, combinatorial design, protein fitness landscapes, functional ANOVA

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TWE7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I read this paper as an audit of what automated batch selection allows a campaign to measure at its leads. The scientific question is whether two changes interact at a specified background, expressed through the four-point contrast of a double-mutant cycle. The paper establishes that a linear estimator unbiased under every response function exists exactly when all four corners have been observed, even though a rectangle-free batch can identify a full second-order response model. Its geometric analysis explains missing certificates through Hamming separation, per-background occupancy, and the placement of rectangles relative to the leads. A further result gives an exact identity between the mean squared error of the second-order functional-ANOVA projection over uniformly sampled rectangles and the higher-order variance involving the two coordinates. The experiments audit twelve batch rules on four biological landscapes using a shared random-forest surrogate, second-order ridge refits, and RECTIFY as a constructive rectangle-completion control. The study demonstrates that global identifiability can coexist with zero local certification and derives an oracle boundary between projection error and direct-measurement variance; the fitted-model and decision comparisons depend on the evaluation setting.

I recommend a borderline rejection because the exact certification result and the measure-or-model interpretation are not yet connected convincingly. The structural audit is relevant to scientific experimental design, and I find its conclusions persuasive. The paper explains the distinction between global identifiability and local measurement clearly and gives its negative results serious attention. Beyond the classical support and estimability facts, the projection-bias identity is the theoretical contribution I find most promising. My rating rests mainly on the explanation of the fitted-model crossing and the incomplete specification of the projection and evaluation targets. Until those points are resolved, I cannot judge how much the oracle benchmark explains the local comparisons. A rebuttal that reconciles the risk distributions, clarifies the existing calculations, and brings the empirical claims into line with them could raise my rating.

Critical strengths
  • I find the distinction between model-based identifiability and direct local measurement useful for experimental design. It clarifies which scientific questions a batch can answer without assumptions about the response surface.
  • The connection between the estimand and the geometry of the measurements gives the audit a clear mathematical basis. I particularly appreciate the combination of structural reasoning and the projection-bias calculation.
  • I value the broad retrospective evaluation and the candid treatment of negative results. The paper recognises that certification alone does not guarantee better predictions or better optimisation decisions.
Critical weaknesses

My main concern is the connection between the structural certificate and the choice between direct measurement and a fitted model. The quantitative comparisons do not consistently use the local evaluation target that motivates the paper.

  • I cannot reconcile the explanation of the fitted-model crossing with the projection identity. Corollary 1 and the surrounding discussion in Section 4 (pp. 5-6) call Equation (3) necessary, but not sufficient, for a fitted second-order model to win. Under the theorem's uniform rectangle distribution, orthogonality gives a stronger statement: for a fixed coordinate pair and any second-order function gg,
E[(κij(f)−κij(g))2]=4cicjV≥3(ij)+E[(κij(P2f)−κij(g))2].\mathbb{E}\bigl[(\kappa_{ij}(f)-\kappa_{ij}(g))^2\bigr] =4c_i c_j V_{\geq 3}^{(ij)} +\mathbb{E}\bigl[(\kappa_{ij}(P_2f)-\kappa_{ij}(g))^2\bigr].

This decomposition also holds conditionally on the training data when the evaluation rectangle is independently uniform. A fitted model therefore cannot improve on the exact projection's population risk under that distribution. Yet Figure 2 and Section 5.2 (p. 8) attribute the fitted ridge's later crossing to a shrinkage advantage over the oracle projection. Table 19 reports a fitted-model win on Poelwijk at σ=0.1284\sigma=0.1284, despite V/thr.=2.62V/\mathrm{thr.}=2.62: the model's RMSE is 0.233 and the contrast's is 0.262. Evaluation at selected leads can permit this reversal, as the paper's discussion of distribution shift recognises. In that case, the uniform-risk necessity statement no longer governs the comparison. I would like the evaluation distribution in Figure 2 to be specified and the effects of target selection, finite-sample variation, and shrinkage to be distinguished. This is a substantive but repairable problem with the interpretation. Proposition 4 itself remains valid.

  • The real-landscape verification leaves me unsure which measure defines the projection. Proposition 4 assumes a uniform product measure, while Appendix N (p. 30) says the decomposition for incomplete landscapes is computed from the cells that exist. That description does not specify the full-space P2fP_2f and V≥3(ij)V_{\geq 3}^{(ij)}. A uniform distribution on observed cells generally loses the product structure used in the orthogonality proof; extending the response to missing cells requires a completion convention. I appreciate the complete-subgrid and deletion checks in Table 20, although stability of the predicted winner does not establish the original full-space identity. The projection and weighting procedure should be stated explicitly, along with the subgrid level counts. Numerical identity checks on complete product spaces should also be distinguished from approximate comparisons on incomplete support. This clarification affects what the reported boundary means.

  • I find the statistical interpretation of the 93-pair check too strong. Section 5.2 and Table 18 report 93/93 correct sides of the boundary with p=10−28p=10^{-28}, but 78 pairs come from one landscape and different pairs share higher-order ANOVA components. The prediction and measured projection error also come from the same response surface and are linked by the identity being checked. No null model is specified that would justify the significance claim. Agreement between these calculations is useful evidence of numerical correctness. Reporting the discrepancies and uncertainty due to sampling or approximation would convey that evidence without suggesting 93 independent confirmations of practical predictive validity. This reporting issue should be straightforward to fix.

  • The comparisons at the leads leave the decision value of certification unresolved. Table 17 evaluates the two estimators on the subset certified by RECTIFY, and Table 10 usefully repeats this for Top-N. The estimators share the same interactions within each comparison, but the batch rule selects the subset, which need not represent the predeclared estimand set QQ. Table 8 changes the target again by excluding the combinations each rule measured, leaving different rules with different evaluation sets. I also could not locate a mathematical definition of this regret. Section 3 already provides practical motivation and a concrete decision interpretation, and the paper acknowledges the negative decision results. I still need a clearer account of how the certificate informs that scientific task. Defining regret precisely and comparing rules on a common lead-conditioned target set would make the evidence easier to interpret. A worked example from an existing run could show when certification changes a local inference or decision. My concern is the evidence connecting certification to its use; RECTIFY need not become a competitive optimiser for this audit to be worthwhile.

Questions for authors
  1. In Figure 2, are the oracle and fitted-model errors evaluated on independent uniform rectangles, on the fixed lead set, or on certified subsets? How does the shrinkage explanation reconcile with the orthogonal risk decomposition above?
  2. How are P2fP_2f and V≥3(ij)V_{\geq 3}^{(ij)} computed on incomplete landscapes? Please specify the measure, missing-cell treatment, and whether the complete-subgrid comparisons recalculate the threshold using the subgrid level counts.
  3. What statistical test and null hypothesis produce p=10−28p=10^{-28}? Are the residual errors in Table 18 due to rectangle sampling, incomplete support, or another approximation?
  4. What is the exact definition of held-out decision regret, and which candidate combinations are compared? Can an example from the existing results clarify what certification contributes when the lead and evaluation target are held fixed?
1 comment

Since the paper has issues on presentations and readability, the authors did not try to address any of the points, I keep the original rating for the paper.

Reviewer_n556OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The submission examines an apparent conflict between two common assumptions about the form of a response function over discrete variables in terms of sampling strategies. The first approximates the response function as an additive model up to second-order interactions and the second allows the function to freely vary. Under the second, estimating interaction effects between two "mutations" requires evaluating all four points of a rectangle with the individual mutations and their combination. Such closed rectangles are not necessary for the first as the interactions can be read off the fitted additive model.

Empirically, the authors investigate the frequency of selecting closed rectangles (the only way to "certify" a local interaction in the assumption-free picture) under different rules for sample selection.

Critical strengths

Fundamental questions around the selection of query points in response landscapes, such as the tradeoff between relying on an assumed approximation vs probing in an assumption-free manner, are interesting and impactful. Paired with an adaptive multi-batch strategy, the work might be suitable for publication in a future revision.

Critical weaknesses

The paper revolves around a manufactured conflict that can be technically addressed with little substance, but sold as something grand ("a measurable paradox" L77). The conflict disappears as soon as the difference in assumptions (additive model vs model free) are stated. The empirical investigations count rectangles, predominantly, focusing on a hypothetical need to "certify" interactions under the model-free assumption rather than evaluating the quality of fit.

In addition to the trope of adeptly tackling a manufactured conflict, the paper has other unpleasant characteristics of AI writing:

  • certify appears 98 times in the manuscript, buy appears 19 times.
  • Slick pitch language that makes parsing annoying: "The extremal problem is Zarankiewicz’s, the bound Reiman’s, the asymptotics Kov˝ ari et al." (L221), "Conclusion. Three things. Experimental design ..." (L479), "Both matter; only κij answers a question asked at a named background." (L166). "Buying the geometry deliberately does not buy more of it" (L465)

"Proposition 3(i) is the most useful line in the paper, because dmind_\text{min} is computable from a proposed batch before a single measurement: it turns the audit into a prediction" (L238) --> 3(i) provides a quick necessary (but not sufficient) condition for 2-coordinate rectangles. It says nothing about k-coordinate rectangles for k>=4, which are important (as the paper points out). More importantly, it's not motivated why screening directly for rectangles is expensive; at best 3(i) provides an explanation for the frequency of 2-coord rectangles under different batch specs. It's odd (or oddly telling) to call it the most useful line in the paper.

Questions for authors

Can the authors clearly state what nontrivial insight about experimental design this work establishes beyond the well-known fact that a design optimized for one estimand need not be appropriate for another?

One other direction that might benefit the paper: how can any of these insights play into an adaptive, multi-batch strategy?

Reviewer_LqGXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper distinguishes global identifiability under a second-order model from model-free measurability of local interactions. It characterizes when a four-point interaction contrast admits a linear estimator unbiased for every response function, and audits twelve batch-selection rules on four biological landscapes. It also derives a functional-ANOVA identity relating the local interaction error of the exact second-order projection to higher-order variance, yielding an oracle boundary between projection and direct measurement.

Critical strengths
  1. Clear distinction between identification and measurement. The paper demonstrates that identifying a second-order model does not guarantee direct measurement of a specified local interaction. The four-point characterization and geometric diagnostics provide a useful perspective on experimental design.
  2. Interpretable bias identity and transparent limitations. Proposition 4 precisely identifies which higher-order components contribute to local interaction error. The paper also reports that deliberate rectangle completion does not consistently improve certification or downstream decisions, appropriately limiting its practical claims.
Critical weaknesses
  1. The explanation of fitted-model performance beyond the oracle boundary is insufficiently established. Under the uniform rectangle distribution of Proposition 4, the exact second-order projection minimizes interaction MSE over all second-order functions; shrinkage cannot improve upon this population optimum. A fitted model can outperform that projection on selected leads, but this involves a different evaluation distribution, where the orthogonality argument no longer applies. Although the paper acknowledges this distinction, Figure 2 attributes the later ridge crossover to shrinkage without clearly specifying the evaluation distribution for each curve or isolating this mechanism. Please compare both models on the same evaluation set and report the error decomposition, including the cross term.
  2. The Dₛ-optimal criterion is inconsistent with the stated priors. Section 3 removes additive effects through an unregularized projection, whereas Appendix H specifies unit Gaussian priors on both the additive and interaction parameters. Integrating out additive parameters under these priors generally produces a different information criterion, which can change the ranking of candidate batches. Please clarify which criterion was implemented, explain how historical observations enter the objective, and verify whether resolving this discrepancy affects the reported certification results.
  3. The ANOVA computation on incomplete landscapes needs a precise definition. Proposition 4 assumes a uniform product measure, but Appendix N computes decompositions from observed cells of incomplete landscapes without sufficiently specifying missing-cell handling or preservation of orthogonality. For ParD3, which has three coordinates, the total higher-order variance and each pair-specific higher-order variance should coincide under the stated decomposition, yet Table 17 reports 0.046 and 0.0437. Please explain this discrepancy and report orthogonality and identity checks. The complete-subgrid sensitivity analysis is useful, but stability of the predicted winner does not establish correctness of the original decomposition.
Questions for authors

Please address the concerns raised in the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.