- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether external data can improve target prediction and whether it is worth acquiring. It studies this question through high-dimensional weighted ridge regression with heterogeneous data sources and privacy noise. The main contributions are a deterministic equivalent of prediction risk, optimal source weights and regularization parameters under a Gaussian prior, and experiments using the predicted risks to select auxiliary datasets. The analysis examines how sample size, distribution shifts, and privacy noise affect the value of external data.
Critical strengths
- The opening question is interesting and practically meaningful. Understanding whether external data are useful before committing to acquiring them is an important problem.
- The theoretical risk characterization provides a concrete way to study the effects of data heterogeneity and privacy noise. The simulations and real-data experiments provide supporting evidence for risk prediction and dataset selection.
Critical weaknesses
-
My main concern is the connection between risk prediction and the final acquisition decision. Predicting a change in risk or ranking candidate datasets does not fully answer whether one should buy the data. The paper does not sufficiently explain how acquisition costs, the required improvement, and uncertainty in the predicted benefit enter this decision. Consequently, the decision support promised by the opening motivation remains incomplete.
-
The presentation brings together data acquisition, transfer learning, differential privacy, and connections to multi-task learning, but their roles in answering the central question are not always clear. I would prefer either a narrower formulation or a more explicit framework connecting these components to the final decision. In particular, the additional privacy analysis should lead to clearer guidance about when external data should or should not be acquired.
Questions for authors
-
How would a user turn the predicted risk into an explicit “buy or do not buy” decision? What determines the minimum worthwhile improvement, and when should the sequential selection procedure stop?
-
Which population quantities can realistically be obtained before purchasing the data? How reliable are the resulting decisions when the non-private reference data used to estimate these quantities are limited or unrepresentative?
-
How can users assess uncertainty in the predicted benefit, particularly when it is close to zero? The experimental confidence intervals describe average performance across repetitions, but how would the method support a reliable decision in a single application?
7 comments
Response to Question 3 - Uncertainty of predicted benefit.
This is a good point and, to address it, in the revision we additionally establish a non-asymptotic bound for the prediction risk which scales as , assuming ridge penalties bounded away from zero and bounded eighth moment for . We refer the reviewer to the paragraph immediately following the statement of Theorem 4.1, as well as Proposition B.2 in Appendix B.4 for details. In particular, Eq. (B.74) gives a confidence interval for the prediction risk which can be used for uncertainty quantification, thus supporting a reliable decision in a single application even when the predicted benefit is close to 0. The argument of Proposition B.2 follows rather closely the one we used to prove Theorem 4.1, the main addition being the quantitative resolvent estimates taken from (Louart, 2026).
Thank you for the response and the additional analysis addressing uncertainty. I continue to find the motivating problem meaningful. However, my main concern about the significance and depth of the contribution remains.
The response explains the roles of privacy, transfer learning, and data acquisition, but the central insight connecting them is still insufficiently clear to me. The clarification that the framework predicts gains while leaving the acquisition criterion to the user also narrows the practical contribution relative to the motivating question.
Could you clarify the key new insight obtained by considering these elements jointly? In particular, what do your results reveal about when privacy changes the value of external data, and how does this inform an actual acquisition decision? Please connect your answer to specific theoretical or empirical results and explain the substantive advance over existing risk analyses.
I will retain my current rating for now to allow for further clarification. If this concern about the significance of the contribution remains unresolved, I intend to revise my rating downward.
We thank the reviewer for the positive evaluation and the insightful comments. We address below the concerns raised in the review and have uploaded a revision marking the changes in red.
Response to Weakness 1 and Question 1 - From risk to buying decision.
We agree that ranking datasets leaves open whether any of them is worth acquiring. Two hospitals that are offered the same records at the same price may require different improvements in prediction accuracy to justify the expense. The final decision is thus not purely a technical machine learning problem. Our paper nevertheless provides a practical tool for making a more informed decision.
In fact, our results allow each hospital to estimate its expected gain, given the data it already has and the privacy level at which possible additional datasets are shared. The hospital can then decide what improvement is worthwhile. For the sequential selection procedure illustrated in Figure 3, the user can set this minimum improvement and stop when no remaining dataset is predicted to meet it. If the predicted gain is close to the threshold, uncertainty in that estimate may affect the decision, and we discuss uncertainty quantification in our answer to Question 3 below.
We can reword the abstract and introduction to clarify further how our predictions inform the buyer’s decision without unintentionally suggesting a one-size-fits-all criterion.
Response to Weakness 2 - Unclear role of components.
The question we study lies at the intersection of data acquisition, transfer learning, and differential privacy, and each field contributes one piece of the answer. In particular, the role of each field to the central question can be described as follows.
Suppose a hospital is offered data from a second hospital at a given price and privacy level. Work on private data acquisition asks whether the gain in accuracy justifies the cost of obtaining the data. For the first hospital, the relevant gain is the reduction in prediction error on its own patients. To predict that gain, we need to understand how the information in the two datasets would work together. Transfer learning studies show that simply pooling them can hurt, e.g., a feature that predicts an outcome in one population may have a different effect in the other, due to distribution and/or model shift (Yang et al., 2025). Therefore, results on multi-task regression (e.g., Ilbert et al., 2024) play an important role by letting us write the second hospital’s coefficients as the target coefficients plus a source-specific deviation (see Eq. (3.3) in our paper). This allows the two hospitals to share information while accounting for differences between their patients. Finally, privacy enters in the calculation because the second hospital can share only the privatized observations.
Our contribution is to predict, from population-level quantities, how much the joint fit at the given privacy level would improve over training on the first hospital’s data alone. Figure 1(c) shows how that gain changes for the same source as noise is added to its labels. If the remaining gain is too small to justify the price, the first hospital should decline the offer.
Moreover, our analysis can guide the decision on private data acquisition based on other factors, such as the abundance of reference non-private data that can be itself used for training (Figure 6). In particular, under a restricted amount of clean auxiliary reference data, there can be a substantial relative improvement resulting from the addition of private data over the non-private baseline (Figure 6(a)). Thus, our framework can estimate the worth of data and give quantifiable comparison to when, e.g., reference data is more abundant, as in this case the improvement from additional private data is expected to be more modest (Figure 6(c)).
Response to Question 2 - Obtainable population quantities, reliability.
We kindly refer the reviewer to our replies to Reviewer TBCT and Reviewer LAdA (Weakness 1 and Question 1), where we discuss in detail which population quantities can realistically be obtained before purchasing the data, in the context of the two motivating examples involving the UK Biobank and the US Census.
Concerning reliability, we note that the predictions of Figure 2 remain reliable using as little as 50 samples to estimate population-level statistics, see Figure 5 in Appendix C.3 of the revision. Furthermore, even if the public data is then used for prediction, adding private data can still improve performance, see our response to Reviewer LAdA (Question 3) and Figure 6 in Appendix C.3 of the revision.