- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This manuscript argues that pointwise prediction error is insufficient for evaluating surrogate models on combinatorial fitness landscapes. Given a true fitness function , prediction , and residue , it distinguishes pointwise error from errors in first-order mutational effects , and second-order mutation-square interactions . The key math observation is that fixing , and hence pointwise risk, doesn't determine quantities such as ; therefore residual fields with equal magnitude can have very different geometry relative to the mutational graph. Further, the manuscript derive margin-based bounds connecting and to mutation/inter-action sign errors and a corresponding bound on changes in high-margin local-optimum status.
Critical strengths
The concept is useful and the math formulation is clean. In the science surrogate modelling, we often care about derivatives, orderings, interactions, optimization trajectories rather than only scalar prediction RMSE. The paper turns the intuition into a precise graph-based formulation. The argument that two predictors with comparable fitness RMSE may induce different mutational gradients is very insightful. The experiments are well-controlled and convincing as mechanism demo. Plus, the metrics are made to be actionable. Relational-aware training is important because otherwise the contribution risks being only diagnostic. The intervention shows that the relational quantities can be changed by the learning objective, with order-specific effects rather than a trivial uniform smoothing effect.
Critical weaknesses
- The strongest empirical phenomenon is counterfactual rather than naturally occurring. The predefined natural matched- exp is negative, where the hypothesized large separation doesn't occur. The strongest results instead come from taking already-trained residuals and using GT labels to construct so that test is exactly equal. This means the exp demonstrates possibility conditional on residual direction, not that realistic independently trained models with comparable generalization error routinely exhibit significant relational divergence. Relation-aware training improves , but pointwise equivalence fails under the prespecified criterion. So I think the result doesn't yet demonstrate that "better relational fidelity at the same predictive accuracy"; a pointwise/relational trade-off is still possible.
- Measurement noise for second differences. A second-order finite difference combines 4 experimental measurements and can amplify assay noise. The manuscript reports that about 62.5% of Domingo mutation squares are assay-ambiguous, although the model ordering survives when restricting sign recovery to assay-confident squares. This highlights the problem that noise-aware evaluation is unavailable across most datasets and is not built directly into or the training loss.
Questions for authors
- Can you demonstrate a substantial relational difference between independently trained models that naturally achieve equivalent , without any post-hoc residual rescaling?
- What is the intended practical use of the learned-direction calibration?
- Can relation-aware training improve while enforcing actual pointwise equivalence rather than only a validation admissibility threshold?
- What are the exact split geometries? In particular, how do the conclusions change under Hamming-distance extrapolation, held-out mutation/site splits, or other splits (which may prevent near-neighbor leakage between training/testing genotypes)?