- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to trade off the cost of acquiring new measurements against the benefit of improved predictions when reusing existing predictors in a new environment. The proposed framework combines source-model selection, kernel-based residual correction, active measurement selection, and a two-step lookahead stopping rule. The method is evaluated on ChEMBL and HELP, with the main focus on the trade-off between measurement cost and prediction error.
Critical strengths
- The problem is practically relevant, and explicitly accounting for measurement cost is well motivated.
- The method and evaluation protocol are described in considerable detail, and the paper is fairly transparent about cases where some components do not show clear gains.
- Reusing each acquired label for both model selection and prediction correction is computationally appealing.
Critical weaknesses
- The theoretical contribution is limited
Most of the derivations are based on standard kernel-regression updates, Gaussian conditioning, and simple identities for the lookahead objective. They help explain how the algorithm is computed, but provide relatively little new understanding of when or why the different components should work together. The paper does not establish conditions under which the proposed acquisition or stopping rules are reliable.
- The methodological novelty seems to lie mainly in combining existing ideas
Valuing a measurement through the prediction rule that will ultimately use it is a reasonable design choice. However, the main ingredients—model selection, residual correction, variance-reduction-based acquisition, and cost-sensitive stopping—are all closely related to existing methods. The paper explains well how these components are connected, but less clearly what new methodological principle emerges from this particular combination.
- The experiments do not fully establish the need for the more complex design
On ChEMBL, adaptive stopping does not show a general advantage over a fixed measurement count selected on development data. Similarly, the comparison between -based valuation and posterior-median valuation does not establish an overall advantage for the former. The local analysis of an optional second measurement is also not equivalent to directly comparing complete one-step and two-step policies. Overall, it remains unclear how much of the benefit comes from the full valuation/stopping mechanism rather than from the simpler residual-correction component.
- The numerical reliability of the acquisition-value calculation deserves further checking
The current risk is computed analytically, while future risks are approximated using only three Gauss–Hermite nodes per mixture component. The paper itself notes that the resulting integrand can be nonlinear and nonsmooth, so this is only a numerical approximation.
Using the equations in the paper, one can construct a simple case where a measurement does not change the final prediction at all, yet the three-node approximation assigns it a positive expected benefit. This suggests that quadrature error can in principle affect the acquire/stop decision. I would therefore like to see a basic consistency check—e.g., zero estimated benefit when the prediction is unchanged—and a sensitivity analysis with more accurate integration.
- The stopping rule may miss other worthwhile measurements
The algorithm first selects a single candidate using posterior geometry, then performs detailed valuation only for that proposal. If that candidate is rejected, the procedure stops. However, posterior geometry measures within-component variance reduction, while the final objective depends on the error of the selected and corrected predictor. These two rankings need not agree. Therefore, rejecting the geometry-selected candidate does not imply that all other candidates have negative value. The paper acknowledges that posterior geometry is only a proposal mechanism rather than the exact downstream acquisition value, but does not quantify how often this approximation affects stopping.
Questions for authors
- Which results provide the clearest evidence that the valuation and two-step lookahead offer benefits beyond residual correction with a simpler fixed-budget strategy?
- How sensitive are the stopping decisions and final cost-inclusive losses to the number of quadrature nodes?
- When the geometry-selected proposal is rejected, how often is there another candidate with positive expected net value?
Second Response to Reviewer 2UuG (1/3): Method and Expected Loss
Thank you for your follow-up and for pointing out where our previous response was unclear. We appreciate the opportunity to explain our method and results more clearly.
How does the method work? & Does it have access to the true labels before selection?
The method does not access the true labels of unmeasured cases. Therefore, we cannot know the actual prediction error in advance. When we previously said that we "compute the loss," we meant that we compute the expected error using the information currently available to us.
1. Preparation: Tools for Estimating Prediction Error
We use two types of past data to make these estimates. First, we use source data to prepare several predictors. Next, we check how well these predictors perform on development data from other environments. Since the true labels are already known in the development data, we can compare the predicted values with the actual values. We use these differences to build a probability model that describes how actual values may differ from the predictors’ outputs. In our paper, we use a Gaussian mixture model. This model describes what values the unknown labels in a new environment might take, and how likely those values are.
2. What should we measure first?
In a new environment, we first decide what to measure. For example, suppose the model expects that if one candidate's measured value turns out to be higher than predicted, the values of several other cases are also likely to be higher. Measuring that candidate would then help us learn not only about that candidate, but also about the other cases. Posterior geometry uses these relationships to score how much measuring each candidate is expected to tell us about the cases we want to predict. It recommends the candidate with the highest score.
3. How can we tell whether a measurement will help before taking it?
Next, we consider whether that candidate is actually worth measuring. We do not know the actual measurement result yet, but the probability model we prepared tells us what values might occur and how likely they are. Using this distribution and a numerical integration method (Gauss–Hermite rule), we choose hypothetical measurement values to use in the calculation and assign a weight to each. For each hypothetical measurement value, we ask: “If this were the actual result, which predictor would we select, and how would we correct the predictions for the other cases?” We apply the same selection and correction procedure that we would use after obtaining the actual measurement result, but we try it out in advance.
4. How do we calculate the expected loss without knowing the true labels?
For a given hypothetical measurement result, we calculate the expected error as follows. First, we use the hypothetical measurement result to select a predictor. We then calculate the difference between that hypothetical result and the selected predictor’s prediction for the candidate. Our correction rule uses this difference to adjust the predictions for the other cases. Second, we use the probability model to obtain the conditional distribution that answers: “If this measurement result occurred, what could the labels of the cases we want to predict be?” For each prediction target, we calculate the absolute difference between its corrected prediction from the first step and each possible value under this conditional distribution. We weight these differences by how likely the corresponding values are and combine them to obtain the expected error for that target. We then average these target-specific expected errors over all prediction targets to obtain the overall expected error under this hypothetical measurement result. We repeat the same calculation for every other hypothetical measurement result. By averaging the resulting expected errors using the weights assigned to those hypothetical results, we can estimate how much error would remain, on average, if we measured this candidate and used its result.