- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies multi-fidelity learning under a budget on changes to an existing predictor. It introduces the Scale-Separation Index (SSI) to characterize worst-case normalized updates under fixed-kernel assumptions and proposes spectral filtering to suppress high-gain residual directions. The paper establishes theoretical properties of the filter and a population bound under independent audit sampling. Experiments on QM7b and MF-PCBA show modest improvements over matched controls, while revealing limitations in predictive competitiveness and the transfer of update-budget guarantees.
Critical strengths
- Careful and transparent experimental evaluation. The paper distinguishes predictive error, update magnitude, and low-fidelity agreement, and reports both positive and negative findings. Its detailed protocols and uncertainty estimates make the empirical claims relatively easy to assess.
- Controlled evaluation of directional filtering. The equal-spectrum scrambled control isolates directional alignment from shrinkage strength. The direct filter outperforms scrambled directions in 21 of 22 sensitivity configurations, providing evidence for a directional effect.
Critical weaknesses
- Limited practical gains and insufficiently isolated benefits. Improvements over matched controls are small, and the proposed method uses more of the update budget. Comparisons at matched realized update magnitudes are missing. Stronger feasible predictors perform substantially better in several settings, while isotropic regularization also outperforms SSI regularization in a reported QM7b experiment.
- Limited validation of preservation and generalization. The experiments do not explicitly separate retention and correction regions, leaving the motivating preservation objective insufficiently tested. Moreover, test-budget violations in the pooled experiment and the weak correlation between SSI and neural-network interference raise concerns about practical transfer beyond the fixed-kernel setting.
Questions for authors
-
Contribution of directional control beyond regularization strength. Could the authors report direct-filter comparisons against trace-matched isotropic and training-covariance controls at \tau=0.20 and across additional budget levels? In particular, how should the substantially better isotropic-control result in the reported \tau=0.50 QM7b experiment be interpreted? Does directional filtering retain an advantage when regularization strength is appropriately matched?
-
Accuracy gains at matched realized update magnitudes. Could the authors provide paired accuracy-update frontiers, with confidence intervals, comparing direct filtering with \gamma=0 scalar blending and strong baselines such as NARGP and AR1 at matched realized update magnitudes? This would help determine how much of the reported improvement is attributable specifically to directional filtering rather than greater budget utilization.
-
Validity of budget transfer and the preservation use case. Could the authors clarify whether the pooled-gate protocol uses audit samples independent of candidate fitting, as required by Theorem 2, and explain the discrepancy between anchor feasibility and the 133/160 test-feasibility result? In addition, can they evaluate a setting with distinct retention and correction regions, reporting the update on the retained region and predictive error on the correction region?