- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
I read this paper as an audit of what automated batch selection allows a campaign to measure at its leads. The scientific question is whether two changes interact at a specified background, expressed through the four-point contrast of a double-mutant cycle. The paper establishes that a linear estimator unbiased under every response function exists exactly when all four corners have been observed, even though a rectangle-free batch can identify a full second-order response model. Its geometric analysis explains missing certificates through Hamming separation, per-background occupancy, and the placement of rectangles relative to the leads. A further result gives an exact identity between the mean squared error of the second-order functional-ANOVA projection over uniformly sampled rectangles and the higher-order variance involving the two coordinates. The experiments audit twelve batch rules on four biological landscapes using a shared random-forest surrogate, second-order ridge refits, and RECTIFY as a constructive rectangle-completion control. The study demonstrates that global identifiability can coexist with zero local certification and derives an oracle boundary between projection error and direct-measurement variance; the fitted-model and decision comparisons depend on the evaluation setting.
I recommend a borderline rejection because the exact certification result and the measure-or-model interpretation are not yet connected convincingly. The structural audit is relevant to scientific experimental design, and I find its conclusions persuasive. The paper explains the distinction between global identifiability and local measurement clearly and gives its negative results serious attention. Beyond the classical support and estimability facts, the projection-bias identity is the theoretical contribution I find most promising. My rating rests mainly on the explanation of the fitted-model crossing and the incomplete specification of the projection and evaluation targets. Until those points are resolved, I cannot judge how much the oracle benchmark explains the local comparisons. A rebuttal that reconciles the risk distributions, clarifies the existing calculations, and brings the empirical claims into line with them could raise my rating.
Critical strengths
- I find the distinction between model-based identifiability and direct local measurement useful for experimental design. It clarifies which scientific questions a batch can answer without assumptions about the response surface.
- The connection between the estimand and the geometry of the measurements gives the audit a clear mathematical basis. I particularly appreciate the combination of structural reasoning and the projection-bias calculation.
- I value the broad retrospective evaluation and the candid treatment of negative results. The paper recognises that certification alone does not guarantee better predictions or better optimisation decisions.
Critical weaknesses
My main concern is the connection between the structural certificate and the choice between direct measurement and a fitted model. The quantitative comparisons do not consistently use the local evaluation target that motivates the paper.
- I cannot reconcile the explanation of the fitted-model crossing with the projection identity. Corollary 1 and the surrounding discussion in Section 4 (pp. 5-6) call Equation (3) necessary, but not sufficient, for a fitted second-order model to win. Under the theorem's uniform rectangle distribution, orthogonality gives a stronger statement: for a fixed coordinate pair and any second-order function ,
This decomposition also holds conditionally on the training data when the evaluation rectangle is independently uniform. A fitted model therefore cannot improve on the exact projection's population risk under that distribution. Yet Figure 2 and Section 5.2 (p. 8) attribute the fitted ridge's later crossing to a shrinkage advantage over the oracle projection. Table 19 reports a fitted-model win on Poelwijk at , despite : the model's RMSE is 0.233 and the contrast's is 0.262. Evaluation at selected leads can permit this reversal, as the paper's discussion of distribution shift recognises. In that case, the uniform-risk necessity statement no longer governs the comparison. I would like the evaluation distribution in Figure 2 to be specified and the effects of target selection, finite-sample variation, and shrinkage to be distinguished. This is a substantive but repairable problem with the interpretation. Proposition 4 itself remains valid.
-
The real-landscape verification leaves me unsure which measure defines the projection. Proposition 4 assumes a uniform product measure, while Appendix N (p. 30) says the decomposition for incomplete landscapes is computed from the cells that exist. That description does not specify the full-space and . A uniform distribution on observed cells generally loses the product structure used in the orthogonality proof; extending the response to missing cells requires a completion convention. I appreciate the complete-subgrid and deletion checks in Table 20, although stability of the predicted winner does not establish the original full-space identity. The projection and weighting procedure should be stated explicitly, along with the subgrid level counts. Numerical identity checks on complete product spaces should also be distinguished from approximate comparisons on incomplete support. This clarification affects what the reported boundary means.
-
I find the statistical interpretation of the 93-pair check too strong. Section 5.2 and Table 18 report 93/93 correct sides of the boundary with , but 78 pairs come from one landscape and different pairs share higher-order ANOVA components. The prediction and measured projection error also come from the same response surface and are linked by the identity being checked. No null model is specified that would justify the significance claim. Agreement between these calculations is useful evidence of numerical correctness. Reporting the discrepancies and uncertainty due to sampling or approximation would convey that evidence without suggesting 93 independent confirmations of practical predictive validity. This reporting issue should be straightforward to fix.
-
The comparisons at the leads leave the decision value of certification unresolved. Table 17 evaluates the two estimators on the subset certified by RECTIFY, and Table 10 usefully repeats this for Top-N. The estimators share the same interactions within each comparison, but the batch rule selects the subset, which need not represent the predeclared estimand set . Table 8 changes the target again by excluding the combinations each rule measured, leaving different rules with different evaluation sets. I also could not locate a mathematical definition of this regret. Section 3 already provides practical motivation and a concrete decision interpretation, and the paper acknowledges the negative decision results. I still need a clearer account of how the certificate informs that scientific task. Defining regret precisely and comparing rules on a common lead-conditioned target set would make the evidence easier to interpret. A worked example from an existing run could show when certification changes a local inference or decision. My concern is the evidence connecting certification to its use; RECTIFY need not become a competitive optimiser for this audit to be worthwhile.
Questions for authors
- In Figure 2, are the oracle and fitted-model errors evaluated on independent uniform rectangles, on the fixed lead set, or on certified subsets? How does the shrinkage explanation reconcile with the orthogonal risk decomposition above?
- How are and computed on incomplete landscapes? Please specify the measure, missing-cell treatment, and whether the complete-subgrid comparisons recalculate the threshold using the subgrid level counts.
- What statistical test and null hypothesis produce ? Are the residual errors in Table 18 due to rectangle sampling, incomplete support, or another approximation?
- What is the exact definition of held-out decision regret, and which candidate combinations are compared? Can an example from the existing results clarify what certification contributes when the lead and evaluation target are held fixed?
Since the paper has issues on presentations and readability, the authors did not try to address any of the points, I keep the original rating for the paper.