acceptodds
Under review as a conference paper at ICLR 2027

When Reuse Requires New Data: Deciding What to Measure and When to Stop

Abstract

Reusing past data can reduce the need for new data collection, but discovering how to use it in a new environment can itself require costly measurements. We study this as a decision problem whose objective is to minimize the combined burden of prediction error and measurement cost, rather than pursue accuracy regardless of the cost of obtaining it. We develop a procedure that uses selected measurements from the new environment to choose among predictors trained on past data and correct their predictions for other, unmeasured cases. Known relationships between cases guide the choice of what to measure. Before collecting more, the procedure considers possible results and calculates which predictor it would choose and how it would correct the predictions, without retraining the source models. It continues only when the expected improvement justifies the cost, then updates the choice and corrections using the actual result. The decision to acquire more is thus based on the prediction improvement expected from using the results, rather than uncertainty reduction alone. We evaluate using existing records from two sources: the ChEMBL bioactivity database, recording how strongly molecules bind to proteins, and the Hardware-adaptive Efficient Latency Predictor (HELP) study, recording how long neural networks run on devices. In one molecular example, the procedure uses 28.6% fewer measurements than a reference that uses all candidate measurements for the same predictor selection and correction, with an observed increase of about 0.7% in log-scale mean absolute error. In one hardware example, it uses 89.1% fewer measurements than the corresponding reference, with an observed increase of about 26% in that error measure. Mechanism analyses examine how three elements interact within the procedure: relationships that let measurements inform other cases, corrections that match the errors that remain, and judgments of whether further measurement is worth its cost.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
data reuse, source selection, prediction correction, active learning, adaptive stopping, cost-sensitive learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GcawOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to trade off the cost of acquiring new measurements against the benefit of improved predictions when reusing existing predictors in a new environment. The proposed framework combines source-model selection, kernel-based residual correction, active measurement selection, and a two-step lookahead stopping rule. The method is evaluated on ChEMBL and HELP, with the main focus on the trade-off between measurement cost and prediction error.

Critical strengths
  • The problem is practically relevant, and explicitly accounting for measurement cost is well motivated.
  • The method and evaluation protocol are described in considerable detail, and the paper is fairly transparent about cases where some components do not show clear gains.
  • Reusing each acquired label for both model selection and prediction correction is computationally appealing.
Critical weaknesses
  1. The theoretical contribution is limited

Most of the derivations are based on standard kernel-regression updates, Gaussian conditioning, and simple identities for the lookahead objective. They help explain how the algorithm is computed, but provide relatively little new understanding of when or why the different components should work together. The paper does not establish conditions under which the proposed acquisition or stopping rules are reliable.

  1. The methodological novelty seems to lie mainly in combining existing ideas

Valuing a measurement through the prediction rule that will ultimately use it is a reasonable design choice. However, the main ingredients—model selection, residual correction, variance-reduction-based acquisition, and cost-sensitive stopping—are all closely related to existing methods. The paper explains well how these components are connected, but less clearly what new methodological principle emerges from this particular combination.

  1. The experiments do not fully establish the need for the more complex design

On ChEMBL, adaptive stopping does not show a general advantage over a fixed measurement count selected on development data. Similarly, the comparison between FF-based valuation and posterior-median valuation does not establish an overall advantage for the former. The local analysis of an optional second measurement is also not equivalent to directly comparing complete one-step and two-step policies. Overall, it remains unclear how much of the benefit comes from the full valuation/stopping mechanism rather than from the simpler residual-correction component.

  1. The numerical reliability of the acquisition-value calculation deserves further checking

The current risk is computed analytically, while future risks are approximated using only three Gauss–Hermite nodes per mixture component. The paper itself notes that the resulting integrand can be nonlinear and nonsmooth, so this is only a numerical approximation.

Using the equations in the paper, one can construct a simple case where a measurement does not change the final prediction at all, yet the three-node approximation assigns it a positive expected benefit. This suggests that quadrature error can in principle affect the acquire/stop decision. I would therefore like to see a basic consistency check—e.g., zero estimated benefit when the prediction is unchanged—and a sensitivity analysis with more accurate integration.

  1. The stopping rule may miss other worthwhile measurements

The algorithm first selects a single candidate using posterior geometry, then performs detailed valuation only for that proposal. If that candidate is rejected, the procedure stops. However, posterior geometry measures within-component variance reduction, while the final objective depends on the error of the selected and corrected predictor. These two rankings need not agree. Therefore, rejecting the geometry-selected candidate does not imply that all other candidates have negative value. The paper acknowledges that posterior geometry is only a proposal mechanism rather than the exact downstream acquisition value, but does not quantify how often this approximation affects stopping.

Questions for authors
  1. Which results provide the clearest evidence that the valuation and two-step lookahead offer benefits beyond residual correction with a simpler fixed-budget strategy?
  2. How sensitive are the stopping decisions and final cost-inclusive losses to the number of quadrature nodes?
  3. When the geometry-selected proposal is rejected, how often is there another candidate with positive expected net value?
Reviewer_WZfDOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a method for predicting responses in a new environment by deciding which measurements to acquire while reusing predictors trained on data from previous environments. The authors formalize the problem using an acquisition pool P\mathcal{P} and a prediction set Q\mathcal{Q}, with the goal of minimizing prediction error on Q\mathcal{Q} and measurement cost. At each step, the method selects one of several source predictors based on initial error estimates and observed errors, and then corrects the predictions using kernel ridge regression. A Gaussian mixture model over the responses is used to choose which sample in P\mathcal{P} is most informative about Q\mathcal{Q}. The adaptive stopping criterion uses a two-step lookahead that compares the expected loss with the measurement cost. Experiments on ChEMBL and HELP show that the method uses fewer measurements than acquiring the full pool, with small-to-moderate increase in error.

Critical strengths

Extensive and transparent empirical evaluation: The experiments are extensive and carefully designed, and the experimental protocol and statistical analysis appear sound. The authors also report negative results transparently, although these do not seem to support the proposed method (see Weaknesses).

Critical weaknesses
  1. Poorly motivated and inconsistent method design. The proposed method has several moving components, many of which are not properly motivated or analyzed. For example,
  • the selected source predictor is corrected using kernel ridge regression with a domain-specific relationship kernel KK that may not always be easily attainable and is not compared with alternatives;
  • the Gaussian mixture model is not checked for whether it fits the data well, which is a concern given the small development set relative to the number of components;
  • the adaptive stopping rule uses a two-step lookahead, but the choice of horizon is neither motivated nor tested; the same applies to how the hypothetical responses are generated and how many are used. More fundamentally, the conceptual link between the selected source predictor and the outcome model is missing. In particular, it is unclear to me why the method plans with a model (GMM) that is misaligned with the prediction model (selected source predictor), when the mixture model itself can also be used for prediction.
  1. Empirical results do not support the proposed method. The experiments are extensive, but they largely do not support the proposed method design. The reported measurement savings are relative to “acquire-all,” so they mainly reflect the choice of this pessimistic baseline. There are no comparisons with existing acquisition or stopping methods, and the one external predictor (HELP10) achieves lower MAE. The measurement-selection comparison only tests against random selection, which is a weak baseline, and even then the simultaneous confidence band shows an improvement only at a few measurement counts on ChEMBL, while on HELP it is worse than random on the first measurement. The adaptive stopping rule also does not improve over a fixed measurement count on ChEMBL.
  2. Unconvincing practical motivation. The paper is framed around deciding which data to acquire and when to stop in a new environment while reusing past predictors. While this is a conceptually interesting problem, the experiments do little to show that it is practically useful. Both case studies are small in scale, and for HELP the cost of measurement is low. The measurement cost is also in terms of prediction error units, with no obvious choice on how it would be set in practice.
Questions for authors
  1. Can the authors more broadly motivate the components of their methodology and establish why each step is beneficial?
  2. The practicality of the algorithm is not clear. What are some larger-scale settings where the method might be useful? How would the cost λ\lambda be chosen in practice? Could the method also be used for settings where each measurement has a different cost?
  3. Can the authors compare against the related work discussed in Section 2 (particularly the active transfer learning works), or explain why these methods cannot be applied to this setting?
Reviewer_2UuGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a method to decide when and whether new data should be acquired and considers the cost of additional experiments. It attempts to introduce an adaptive stop criterion, but it doesn't show statistically significant impact. The method is assessed for two task sets: ChEMBL protein-ligand binding affinity prediction, across multiple protein targets and HELP, a dataset of neural network execution times across multiple different devices.

Critical strengths
  1. The subject of the paper is really interesting from the point of view of ML applications to different fields, mainly how they should integrate with experimental work. Consideration of the cost as principal component of the method is realistic and moves forward from traditional active learning approaches where this is not common.
  2. Results show a notable improvement of the method over naive approaches (sampling all available points or not sampling any additional point).
Critical weaknesses
  1. Missing baselines. The paper does not compare the method to other traditional active learning acquisition functions (BALD, EPIG, entropy) and therefore it is impossible to tell how much it contributes to the overall field. Critically, it would be interesting to see how a method that incorporates experimental cost would fare against these methods that ignore that component.
  2. Limited evaluation. Evaluation is only performed in two settings: 1) protein-ligand binding and 2) neural network latency. They are admittedly diverse scenarios. But the number of available points to acquire is quite limited 15 and 45 respectively and they are somewhat obscure benchmarks. More general benchmarks from active learning like UCI or MNIST would be useful to better compare the method to the state-of-the-art.
  3. Statistical significance [Not a major problem, but makes reading confusing. I do not understand what the authors are trying to say and that may be a personal problem]. The way the results describe improvements with a 90% CI spanning 0 is confusing. They claim "improvement" and then seems like the conclusion is that it was not a significant improvement. If the CI spans 0, there just isn't any difference, so there is no point in describing the average (or the center of the CI) as an improvement if it's positive, it just is not significant.
Questions for authors
  1. Why results are described with 90% CI instead of the more traditional 95%, what is the rationale behind this decision?
  2. Am I missing some important component from the method that inherently distinguishes from other active learning acquisition functions (apart from acknowledging the cost)?
3 comments
AuthorsOpenReview ↗

Second Response to Reviewer 2UuG (1/3): Method and Expected Loss

Thank you for your follow-up and for pointing out where our previous response was unclear. We appreciate the opportunity to explain our method and results more clearly.

How does the method work? & Does it have access to the true labels before selection?

The method does not access the true labels of unmeasured cases. Therefore, we cannot know the actual prediction error in advance. When we previously said that we "compute the loss," we meant that we compute the expected error using the information currently available to us.

1. Preparation: Tools for Estimating Prediction Error

We use two types of past data to make these estimates. First, we use source data to prepare several predictors. Next, we check how well these predictors perform on development data from other environments. Since the true labels are already known in the development data, we can compare the predicted values with the actual values. We use these differences to build a probability model that describes how actual values may differ from the predictors’ outputs. In our paper, we use a Gaussian mixture model. This model describes what values the unknown labels in a new environment might take, and how likely those values are.

2. What should we measure first?

In a new environment, we first decide what to measure. For example, suppose the model expects that if one candidate's measured value turns out to be higher than predicted, the values of several other cases are also likely to be higher. Measuring that candidate would then help us learn not only about that candidate, but also about the other cases. Posterior geometry uses these relationships to score how much measuring each candidate is expected to tell us about the cases we want to predict. It recommends the candidate with the highest score.

3. How can we tell whether a measurement will help before taking it?

Next, we consider whether that candidate is actually worth measuring. We do not know the actual measurement result yet, but the probability model we prepared tells us what values might occur and how likely they are. Using this distribution and a numerical integration method (Gauss–Hermite rule), we choose hypothetical measurement values to use in the calculation and assign a weight to each. For each hypothetical measurement value, we ask: “If this were the actual result, which predictor would we select, and how would we correct the predictions for the other cases?” We apply the same selection and correction procedure that we would use after obtaining the actual measurement result, but we try it out in advance.

4. How do we calculate the expected loss without knowing the true labels?

For a given hypothetical measurement result, we calculate the expected error as follows. First, we use the hypothetical measurement result to select a predictor. We then calculate the difference between that hypothetical result and the selected predictor’s prediction for the candidate. Our correction rule uses this difference to adjust the predictions for the other cases. Second, we use the probability model to obtain the conditional distribution that answers: “If this measurement result occurred, what could the labels of the cases we want to predict be?” For each prediction target, we calculate the absolute difference between its corrected prediction from the first step and each possible value under this conditional distribution. We weight these differences by how likely the corresponding values are and combine them to obtain the expected error for that target. We then average these target-specific expected errors over all prediction targets to obtain the overall expected error under this hypothetical measurement result. We repeat the same calculation for every other hypothetical measurement result. By averaging the resulting expected errors using the weights assigned to those hypothetical results, we can estimate how much error would remain, on average, if we measured this candidate and used its result.

AuthorsOpenReview ↗

Second Response to Reviewer 2UuG (2/3): Measurement Decisions and Extended Candidate Selection

5. Why do we also consider taking one more measurement?

Even if the first measurement does not improve predictions very much, an additional measurement chosen using that result might lead to a larger improvement. Therefore, for each hypothetical result of the first measurement, our method also considers what would happen if we took a second measurement. We choose the candidate for this second measurement based on the first hypothetical result. We then consider hypothetical results for that candidate and calculate the expected error after using each result, following the same procedure described above.

6. How do we decide when to stop?

For one hypothetical first measurement result, we obtain two values:

  • a. The expected prediction error after using the first hypothetical measurement result.
  • b. The expected prediction error after taking one more measurement and using its result.

We then compare (a) with (b + the cost of the second measurement) and keep the smaller value. We repeat this comparison for the other hypothetical first measurement results. We average the values kept using the weights of the hypothetical first results, then add the cost of the first measurement. This gives us the total expected cost of measuring the candidate recommended by posterior geometry and responding to its result. We compare this total expected cost with the expected error of simply using our current predictions without taking any more measurements. If measuring leads to a lower total expected cost, we actually measure the recommended candidate, update our predictions using the observed result, and repeat the same decision process. Otherwise, we stop measuring and use our current predictions.

7. Additional experiments: Can we find a more useful measurement candidate?

In our additional experiments, we extended the way measurement candidates are selected. The original method considered only the candidate recommended by posterior geometry. In the extended method, we also consider the other unmeasured candidates. For each one, we use the hypothetical calculations described above to estimate the expected error. At this stage, we do not yet consider taking a second measurement. We retain the posterior geometry candidate and add the other candidate with the lowest expected error. We then apply the two-step value calculation described above to both candidates. Taking measurement costs into account, we decide whether to measure anything further and, if so, which candidate to measure. This allows us to consider not only which candidate is expected to provide information about the prediction targets, but also which candidate is expected to reduce error when its result is used by our selection and correction procedure.

AuthorsOpenReview ↗

Second Response to Reviewer 2UuG (3/3): Empirical Results and Confidence Intervals

What is the empirical advantage of the method?

The practical advantage is that our method makes better use of the available measurements while keeping the computational cost of deciding what to measure low.

Better predictions from the same number of measurements

Even with the same number of measurements, the final predictions can differ depending on what we choose to measure. In ChEMBL, when we compared methods at equal measurement counts and averaged the results, posterior geometry achieved lower prediction error than entropy or BALD. This improvement came from obtaining information that was more useful for prediction using the same number of measurements.

Less computation for measurement decisions

We should consider not only prediction accuracy, but also how many resources are spent on measurements and computation to achieve that accuracy. In our implementations, the complete decision-making process was much faster with posterior geometry than with the other information-based criteria. Therefore, even when differences in prediction error are small, a practical difference remains. In HELP, EPIG achieved lower prediction error, but the procedure using posterior geometry required much less computation time.

Lower combined error and measurement cost than a fixed count

We also found an advantage over methods that fix the number of measurements in advance. In HELP, adapting the number of measurements to the results actually observed reduced the average combined prediction error and measurement cost compared with a fixed count chosen on development data. Both approaches used the same predictors, correction procedure, and posterior-geometry selection rule. In other words, the benefit came not only from being able to correct predictions, but also from deciding how much additional information was worth buying to make use of those corrections.

Additional gains from extending candidate selection

In our follow-up experiments on HELP, we compared our extension with one that adds an EPIG-selected alternative to the posterior geometry recommendation. Using the same prediction procedure and matched measurement counts, our extension achieved lower average prediction error.

Considering whether a measurement would help the current predictor selection and correction procedure led to better predictions.

Quantitative comparisons

Advantage demonstratedDatasetEvaluation metricReduction [95% CI]
Posterior geometry over entropyChEMBLMAE at equal measurement counts0.01686 [0.00285, 0.03216]
Posterior geometry over BALDChEMBLMAE at equal measurement counts0.01511 [0.00146, 0.03028]
Adaptive stopping over development-selected fixed countHELPPrediction error plus measurement cost (JcJ_c)0.02112 [0.00892, 0.02966]
Expected-error-based candidate extension over PG+EPIGHELPlog-MAE at equal measurement counts0.01313 [0.00142, 0.03407]

Measurement-selection comparisons are averaged over the full range of measurement counts, while the stopping comparison is averaged over the full range of evaluation costs. The candidate-extension comparison matches measurement counts along the paths for each cost setting before averaging. Confidence intervals were calculated using paired bootstrap resampling at the protein or device level.

Online computation time comparison

Measurement-selection criterionChEMBL online computation timeHELP online computation time
Posterior geometry0.061 s0.022 s
Entropy1.835 s0.760 s
BALD1.830 s0.764 s
EPIG70.493 s89.223 s

These implementations combine each measurement-selection criterion with the original two-step stopping rule, using the same hardware and a single thread. The values are the mean online computation times until the procedure actually stops, measured on 15 ChEMBL episodes and 4 HELP episodes at one specified measurement-cost setting for each dataset. Physical measurement time is excluded. The actual number of measurements differs across methods.

Why did we use 90% confidence intervals?

We used 90% confidence intervals consistently across the original submission, but we did not have a study-specific statistical reason to prefer 90% over 95%. Our previous reply should have stated this directly instead of simply providing 95% intervals.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.