acceptodds
Under review as a conference paper at ICLR 2027

High-Dimensional Asymptotics and Dataset Selection for Private Transfer Learning

Abstract

To commit to buying external data or participating in a collaborative learning process, one must decide whether the additional data will improve prediction enough to justify the cost. This comes with several challenges: (i) in many settings, e.g. in healthcare, the decision relies only on aggregated statistics available publicly, rather than individual-level data; (ii) covariate and model shifts can induce negative transfer, so the additional data deteriorates rather than improves performance; (iii) if the data is sensitive, its privatization requires the injection of noise, which can also offset the benefit of a larger sample size. In this paper, we model the problem of dataset selection through high-dimensional linear regression with multiple heterogeneous sources and a weighted ridge estimator. Our approach uses only summary statistics and it gives privacy guarantees either on labels only or jointly on features and labels, in terms of -zero-concentrated differential privacy. The main technical contribution is a deterministic equivalent of the test error, which captures the interactions between sample size, covariance structure, model shift, regularization and privacy noise. Our theory allows to optimize hyper-parameters (weights and ridge regularizers) and, more broadly, to decide when private external datasets are useful. Overall, our work provides a theoretically tractable foundation for private transfer learning, and we support this foundation via experiments on both synthetic and real-world datasets.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
high dimensional regression, empirical risk minimization, privacy, dataset selection, transfer learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_coYNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether external data can improve target prediction and whether it is worth acquiring. It studies this question through high-dimensional weighted ridge regression with heterogeneous data sources and privacy noise. The main contributions are a deterministic equivalent of prediction risk, optimal source weights and regularization parameters under a Gaussian prior, and experiments using the predicted risks to select auxiliary datasets. The analysis examines how sample size, distribution shifts, and privacy noise affect the value of external data.

Critical strengths
  1. The opening question is interesting and practically meaningful. Understanding whether external data are useful before committing to acquiring them is an important problem.
  2. The theoretical risk characterization provides a concrete way to study the effects of data heterogeneity and privacy noise. The simulations and real-data experiments provide supporting evidence for risk prediction and dataset selection.
Critical weaknesses
  1. My main concern is the connection between risk prediction and the final acquisition decision. Predicting a change in risk or ranking candidate datasets does not fully answer whether one should buy the data. The paper does not sufficiently explain how acquisition costs, the required improvement, and uncertainty in the predicted benefit enter this decision. Consequently, the decision support promised by the opening motivation remains incomplete.

  2. The presentation brings together data acquisition, transfer learning, differential privacy, and connections to multi-task learning, but their roles in answering the central question are not always clear. I would prefer either a narrower formulation or a more explicit framework connecting these components to the final decision. In particular, the additional privacy analysis should lead to clearer guidance about when external data should or should not be acquired.

Questions for authors
  1. How would a user turn the predicted risk into an explicit “buy or do not buy” decision? What determines the minimum worthwhile improvement, and when should the sequential selection procedure stop?

  2. Which population quantities can realistically be obtained before purchasing the data? How reliable are the resulting decisions when the non-private reference data used to estimate these quantities are limited or unrepresentative?

  3. How can users assess uncertainty in the predicted benefit, particularly when it is close to zero? The experimental confidence intervals describe average performance across repetitions, but how would the method support a reliable decision in a single application?

7 comments

We thank the reviewer for the positive evaluation and the insightful comments. We address below the concerns raised in the review and have uploaded a revision marking the changes in red.

Response to Weakness 1 and Question 1 - From risk to buying decision.

We agree that ranking datasets leaves open whether any of them is worth acquiring. Two hospitals that are offered the same records at the same price may require different improvements in prediction accuracy to justify the expense. The final decision is thus not purely a technical machine learning problem. Our paper nevertheless provides a practical tool for making a more informed decision.

In fact, our results allow each hospital to estimate its expected gain, given the data it already has and the privacy level at which possible additional datasets are shared. The hospital can then decide what improvement is worthwhile. For the sequential selection procedure illustrated in Figure 3, the user can set this minimum improvement and stop when no remaining dataset is predicted to meet it. If the predicted gain is close to the threshold, uncertainty in that estimate may affect the decision, and we discuss uncertainty quantification in our answer to Question 3 below.

We can reword the abstract and introduction to clarify further how our predictions inform the buyer’s decision without unintentionally suggesting a one-size-fits-all criterion.

Response to Weakness 2 - Unclear role of components.

The question we study lies at the intersection of data acquisition, transfer learning, and differential privacy, and each field contributes one piece of the answer. In particular, the role of each field to the central question can be described as follows.

Suppose a hospital is offered data from a second hospital at a given price and privacy level. Work on private data acquisition asks whether the gain in accuracy justifies the cost of obtaining the data. For the first hospital, the relevant gain is the reduction in prediction error on its own patients. To predict that gain, we need to understand how the information in the two datasets would work together. Transfer learning studies show that simply pooling them can hurt, e.g., a feature that predicts an outcome in one population may have a different effect in the other, due to distribution and/or model shift (Yang et al., 2025). Therefore, results on multi-task regression (e.g., Ilbert et al., 2024) play an important role by letting us write the second hospital’s coefficients as the target coefficients plus a source-specific deviation (see Eq. (3.3) in our paper). This allows the two hospitals to share information while accounting for differences between their patients. Finally, privacy enters in the calculation because the second hospital can share only the privatized observations.

Our contribution is to predict, from population-level quantities, how much the joint fit at the given privacy level would improve over training on the first hospital’s data alone. Figure 1(c) shows how that gain changes for the same source as noise is added to its labels. If the remaining gain is too small to justify the price, the first hospital should decline the offer.

Moreover, our analysis can guide the decision on private data acquisition based on other factors, such as the abundance of reference non-private data that can be itself used for training (Figure 6). In particular, under a restricted amount of clean auxiliary reference data, there can be a substantial relative improvement resulting from the addition of private data over the non-private baseline (Figure 6(a)). Thus, our framework can estimate the worth of data and give quantifiable comparison to when, e.g., reference data is more abundant, as in this case the improvement from additional private data is expected to be more modest (Figure 6(c)).

Response to Question 2 - Obtainable population quantities, reliability.

We kindly refer the reviewer to our replies to Reviewer TBCT and Reviewer LAdA (Weakness 1 and Question 1), where we discuss in detail which population quantities can realistically be obtained before purchasing the data, in the context of the two motivating examples involving the UK Biobank and the US Census.

Concerning reliability, we note that the predictions of Figure 2 remain reliable using as little as 50 samples to estimate population-level statistics, see Figure 5 in Appendix C.3 of the revision. Furthermore, even if the public data is then used for prediction, adding private data can still improve performance, see our response to Reviewer LAdA (Question 3) and Figure 6 in Appendix C.3 of the revision.

Response to Question 3 - Uncertainty of predicted benefit.

This is a good point and, to address it, in the revision we additionally establish a non-asymptotic bound for the prediction risk which scales as 1/n1/\sqrt{n}, assuming ridge penalties bounded away from zero and bounded eighth moment for ∣∣θ∣∣||\theta||. We refer the reviewer to the paragraph immediately following the statement of Theorem 4.1, as well as Proposition B.2 in Appendix B.4 for details. In particular, Eq. (B.74) gives a confidence interval for the prediction risk which can be used for uncertainty quantification, thus supporting a reliable decision in a single application even when the predicted benefit is close to 0. The argument of Proposition B.2 follows rather closely the one we used to prove Theorem 4.1, the main addition being the quantitative resolvent estimates taken from (Louart, 2026).

Thank you for the response and the additional analysis addressing uncertainty. I continue to find the motivating problem meaningful. However, my main concern about the significance and depth of the contribution remains.

The response explains the roles of privacy, transfer learning, and data acquisition, but the central insight connecting them is still insufficiently clear to me. The clarification that the framework predicts gains while leaving the acquisition criterion to the user also narrows the practical contribution relative to the motivating question.

Could you clarify the key new insight obtained by considering these elements jointly? In particular, what do your results reveal about when privacy changes the value of external data, and how does this inform an actual acquisition decision? Please connect your answer to specific theoretical or empirical results and explain the substantive advance over existing risk analyses.

I will retain my current rating for now to allow for further clarification. If this concern about the significance of the contribution remains unresolved, I intend to revise my rating downward.

Reviewer_LAdAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether private auxiliary datasets can improve prediction on a target task. It analyzes weighted joint ridge regression with source-specific model deviations, heterogeneous feature covariances, and privacy noise. The main result is a deterministic equivalent of target prediction risk in a proportional, under-parameterized regime. The analysis covers label privatization and extends to joint feature and label privatization under Gaussian assumptions. The paper also derives Bayes-optimal tuning for Gaussian priors and uses predicted risk to rank candidate datasets. Synthetic, real-data, and semi-synthetic experiments evaluate the risk predictions and the resulting selection procedure.

Critical strengths

1.An informative and interpretable risk characterization. Theorem 4.1 brings covariance shift, model shift, sample allocation, regularization, and label privacy noise into a unified target-risk expression. The general bias formula (4.7) preserves signal alignment, while the isotropic coefficient assumption is needed only for the simplified formula (4.8). This provides a useful theoretical basis for understanding when auxiliary data improve or harm target prediction.

2.Useful empirical validation with clear experimental controls. The experiments examine risk prediction and dataset selection on synthetic, real-world, and semi-synthetic data. The independent reference partition in Figure 3 separates parameter estimation from model fitting and evaluation, while repeated trials and confidence intervals document empirical variability. These controls strengthen the empirical support within the evaluated settings.

Critical weaknesses

1.The information requirements weaken the central pre-acquisition application. The selector requires source-specific covariances, noise levels, and response-dependent signal or model-shift quantities. In Figure 3, these are estimated from a non-private labeled reference partition containing roughly half of each dataset (Appendix C.1). Although this partition is independent of evaluation data, the paper does not establish that the public summaries cited in its motivation provide comparable information. I therefore find the claimed applicability to acquisition decisions under restricted data access insufficiently supported. This is consequential because deciding whether to acquire data before accessing it is the paper’s central motivation.

2.The practical implications of the privacy analysis remain limited. Assumptions A4–A5 fix privacy-noise variances and require eventually inactive clipping. Although bounded labels permit a constant label-privacy budget, the sufficient sub-Gaussian constructions in Appendix A.1 use budgets growing as log d for labels and d log d for features. The joint-privacy result does not characterize utility under a fixed total record-level budget as dimension grows, and the real-data results report noise variances without corresponding numerical privacy budgets. These limitations leave the connection between the reported gains and a specified level of privacy protection unclear, weakening a central component of the paper’s practical contribution.

Questions for authors
  1. Which publicly available summaries in the motivating applications provide the response-dependent quantities required by Eqs. (4.7)-(4.8), without access to a large non-private labeled reference dataset?

  2. What clipping thresholds and privacy budgets correspond to the reported noise settings? Could the authors include a finite-dimensional check with an explicitly specified budget and report whether clipping is active?

  3. How does the method compare with simple reference-based target-risk or model-similarity selectors? If reference samples can be used for training, how much improvement comes from acquiring private data?

  4. How do rankings and harmful-acquisition frequency change with reference-data size? How do the selections change when ranking and evaluation use matched hyperparameters?

2 comments

We thank the reviewer for the useful comments. We address below all points raised in the review, and we have uploaded a revision marking the changes in red.

Response to Weakness 1 and Question 1 - Publicly available summaries estimating population-level quantities in (4.7)-(4.8).

As mentioned by the reviewer, in order to estimate the population-level statistics needed in (4.7)-(4.8), in the original version of the paper we used a non-private labeled reference partition containing roughly half of each dataset. However, we note that the predictions of Figure 2 remain reliable using as little as 50 samples to estimate population-level statistics, see Figure 5 in Appendix C.3 of the revision. Thus, the sample size needed to compute the quantities in (4.7)-(4.8) does not have to be that large. Furthermore, even if the public data is then used for prediction, adding private data still improves performance, see also our response to Question 3 below and Figure 6 in Appendix C.3 of the revision.

In our response to Reviewer TBCT, we have discussed why using public data to estimate population-level statistics on private data sources is realistic for UK Biobank and US Census data, which are our two motivating applications mentioned in the introduction. We have also incorporated this discussion at the end of Section 5 of the revision.

Since the question of the reviewer concerns the case in which a large non-private reference is not available, from this discussion, let us highlight that, in genetics, some population-level quantities are known, so that one does not need to estimate them from public data. For example, this is the case of the variance of the label noise σc2\sigma_c^2, which corresponds to the heritability of the trait in genetics. As there are reliable estimates of the heritability in the literature, one does not need to estimate σc2\sigma_c^2 from the data. This is also the case of the model bias, which is often adjusted for in genomic studies. In fact, virtually all medical data has a selection bias, which is instead not present in census data, see e.g. [R1, R2]. Finally, for both UK Biobank and US Census, the publicly available data may simply not include some of the features or observables of interest for the problem at hand. In these cases, computing population-level statistics from publicly available data is feasible, but additional private data is still crucially needed to reliably estimate a given outcome not well represented in the public data.

[R1] Schoeler et al., "Participation bias in the UK Biobank distorts genetic associations and downstream analyses", Nature Human Behavior, 2023.

[R2] Benonisdottir and Kong, "Studying the genetics of participation using footprints left on the ascertained genotypes", Nature Genetics, 2023.

Response to Weakness 2 and Question 2 - Clipping thresholds and privacy budgets.

In our real data experiments (Figures 2-3), the clipping threshold is set to the maximum possible value the labels can take in the dataset, after normalization in each of the repetitions. More precisely, we set Ky=2max⁡(∣(a−μ)/s∣,∣(b−μ)/s∣)K_y = 2 \max (|(a - \mu)/s|, |(b - \mu)/s|), where each possible label in the dataset belongs to the interval (a,b)(a,b), and μ\mu and ss are the repetition's sample mean and standard deviation, used for normalizing the data. In this way, the clipping function is effectively the identity, i.e., clipping is inactive.

To address the concern on the overall privacy budget spent, as well as on the choice of the clipping constant, we have added Table 1 in Appendix C.3 of the revision (table below), where we report the mean and standard deviation throughout the 100 experimental runs of the privacy budget ρ\rho and clipping constant KyK_y, corresponding to findings in Figures 2-3. In summary, this conversion shows that performance gains from adding privatized data can be obtained while enforcing ρ∼1\rho \sim 1 (as typically required in practice) and no clipping.

DatasetNoise σDP2\sigma_{DP}^2ρ\rho (mean ± SD)KyK_y (mean ± SD)
Student43.59 ± 0.515.72 ± 0.48
Student160.90 ± 0.135.72 ± 0.48
LSAC45.83 ± 0.787.54 ± 0.54
LSAC161.46 ± 0.207.54 ± 0.54
Lianjia467.83 ± 19.5131.19 ± 5.26
Lianjia1616.96 ± 4.8831.19 ± 5.26

Response to Question 3 - Comparison with simple selectors.

We thank the reviewer for the suggestion to compare with simple reference-based target-risk or model-similarity selectors. We have done that and our method keeps performing better or on par compared with all other baselines.

More precisely, we have updated Figure 3 including a comparison with methods that do selection based on reference data, specifically the ones factoring in target risk and model similarity. To keep the comparison fair and simple, we have not changed the setting, i.e., we have kept the same number of training samples, same noise levels, same availability of target, same auxiliary and reference data points. From the updated plot of Figure 3, we can see that, for the Lianjia dataset, our method outperforms the two new baselines based on reference data, and for the LSAC dataset our method either matches or remains close to such baselines. We have additionally updated the tables in Appendix C.4 containing the exact changes in the target excess risk.

Following the other suggestion of the reviewer, we have also performed an additional experiment in which the additional reference data is added to training, and we report the findings in Figure 6 of Appendix C.3 of the revision. In our experiments, we find that under a restricted amount of clean auxiliary reference data, the percentage improvements can be of the same order as observed in Figure 3, with the deterministic equivalent prediction being still predictive of the actual performance. When reference data is more abundant, the improvement from additional private data becomes smaller.

Response to Question 4 - Sensitivity to reference-data size.

We thank the reviewer for these questions, and we start by discussing how rankings and harmful-acquisition frequency change with reference-data size.

We note that we were able to run our approach with as little as 25 reference samples, while still obtaining an improvement in performance. However, as expected, more reference data does help with selection and lower harmful client acquisition. To make this observation quantitative, we have run the experiments of Figure 3 reducing the number of reference data points. In particular, we report in Appendix C.3 the change when the number of reference samples passes from 25 to 400 in (i) best client selection, which refers to the percentage of runs where our method first selects the client leading to the optimal test error, (ii) harmful client acquisition, which refers to the percentage of runs where our method acquires a client that actually lowers test performance, and (iii) target risk change, which refers to the relative improvement after acquiring the maximum of 3 clients considered in Figure 3. As in Figure 3, we consider the LSAC and Lianjia datasets. Note that, as the amount of reference data is rather small, we use a joint ridge estimator, as defined in equation (3.3) of our paper, on reference data to get initial accurate estimates of model signals. We report our findings in the table below (Table 2 in the revision), which shows a moderate improvement in all three metrics passing from 25 to 400 reference samples.

DatasetBest client selectionHarmful acquisitionsTarget risk change
LSAC16% → 28%12.7% → 11.3%−2.20% → −2.71%
Lianjia12% → 16%6.0% → 6.0%−2.89% → −3.43%

As for the question about using matched hyperparameters, we note that using hyperparameters based on reference data estimates, instead of selecting them via a validation set, reflects the setting in which the expected benefit of auxiliary data must be predicted before accessing the observations used for model fitting or evaluation. This is due to the fact that in the setting we are considering, when deciding whether or not to acquire the data, we do not have access to the private data that we can later run validation on.

That being said, we have rerun the previous experiment (reported in the table just above), using the same validation hyperparameters of each run for ranking and for evaluation, and we report the findings in the table below. We can see that the performance in terms of best client selection does not change noticeably, while the metrics capturing harmful client acquisition and target risk worsen. This can reflect the fact that the hyperparameters selected via validation depend on the small sample of observed data, as they are selected differently in each run. This additional sample variability can interfere with the population prediction the deterministic equivalent is trying to capture, thus resulting in deteriorated performance.

DatasetBest client selectionHarmful acquisitionsTarget risk change
LSAC16% → 26%18.9% → 20.3%−0.72% → −1.10%
Lianjia11% → 17%7.8% → 8.0%−1.65% → −2.42%
Reviewer_TBCTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers the problem of valuing additional datasets for private learning. The paper considers ridge regression as a canonical example, and considers a problem setup where there are C clients, each with their own dataset, with potentially a different ground truth βi\beta_i and also a different feature distribution XiX_i. The authors consider two setups, one where only the label in each labelled example is private (i.e., label DP), and one where both the features and labels are private (standard DP). The authors give an estimator and prove under appropriate assumptions the estimator is Bayes-optimal (for label privacy) and also give expressions for the bias and variance (i.e., the overall risk) of the estimator which are correct in the limit as the number of examples goes to infinity (for both privacy notions). This gives a way to determine the risk for any set of clients being included using the population-level statistics for each client, and thus evaluate the value of including each client. Experiments show that for regression benchmarks, the asymptotic risk expression is a good predictor of empirical performance, and that when used as a utility function for choosing what clients to include, the client choice beats natural baselines and gets close to an oracle lower bound.

Critical strengths
  • The theoretical results are robust. The authors' result is Bayes-optimal (for label privacy), and accounts for a number of factors including ground truth shifts and covariances of individual client features, DP noise levels, non-uniform weighting of different clients, and different ridge regularization factors. This makes it hard to imagine further extensions of the authors' result for the private ridge regression problem, and makes it more likely others could re-apply these results, i.e. the paper is unlikely to have limitations that exclude its use in more general settings of interest.
  • The experiments provide convincing evidence that the authors' risk predictor is accurate in practice, and also allows one to effectively solve the meta-problem of deciding which datasets to include in the first place. Between the above strength and this one, despite being a largely theoretical paper, it suggests this paper avoids the common pitfalls of theory papers of (i) restricting the problem setting too much to be applicable in practice and (ii) having results which don't predict real-world behavior meaningfully.
Critical weaknesses
  • In the context of deciding which datasets to include without directly evaluating the data, the paper's claim is to only use population-level statistics which the authors argue can be estimated from public data. Presumably we would need population-level statistics for each client, and it seems like (i) if private data sources are much more abundant than public data sources, there will not be enough public data sources which provide accurate estimates of population-level statistics for private data, (ii) if public data sources are as abundant as private data sources and have representative population-level statistics, the marginal utility from including noisy private data would be small. And of course, if we try to get the population-level statistics from the private data directly, we need to use noisy estimates to preserve privacy. This aspect of the problem setup is a bit unconvincing to me.
Questions for authors
  • I would be happy to hear further arguments for why using public data to estimate population-level statistics on private data sources is a realistic setting.
2 comments

We thank the reviewer for the encouraging and insightful comments. We address below the concern about using public data to estimate population-level statistics on private data sources. We have uploaded a revision marking the changes in red.

Response to Weakness and Question - Estimation and access to population-level statistics.

First of all, we would like to note that the public data source may allow reliable estimation of population-level statistics and, at the same time, may not be abundant enough, in the sense that including noisy private data helps in a significant way. In fact, the predictions of Figure 2 remain reliable using as little as 50 samples to estimate population-level statistics, see Figure 5 in Appendix C.3 of the revision. Furthermore, even if the public data is also used for prediction, adding private data still improves performance, see also our response to Question 3 of Reviewer LAdA and Figure 6 in Appendix C.3 of the revision.

As in the introduction we mentioned the UK Biobank and the US Census as examples to motivate our setting, we will now discuss why using public data to estimate population-level statistics on private data sources is realistic in these two contexts.

As for UK Biobank data, one interesting case covered by our theory is that of a multi-population study, where the different clients correspond to hospitals recording data of people from different ethnicities. While the majority of UK Biobank samples are from Europeans, the data also includes samples from a variety of ethnicities (Central/South Asian, African, East Asian and Middle Eastern), each with a sample size of several thousands which would suffice for reliable estimation of population-level quantities. Now, there are still reasons to look for additional data from different hospital clients, such as (1) one is interested in filtering people with a certain feature (e.g., cancer patients), or (2) while the genetic Gram matrix XTXX^T X is contained in the UK Biobank, the particular observable yy of interest was not at all contained in the data or, if it was contained in the dataset, the corresponding sample size was too small for reliable estimation. Finally, some population-level quantities may simply be known, so that one does not need to estimate them from public data. For example, this is the case of the variance of the label noise σc2\sigma_c^2, which corresponds to the heritability of the trait in genetics. As there are reliable estimates of the heritability in the literature, one does not need to estimate σc2\sigma_c^2 from the data. This is also the case of the model bias, which is often adjusted for in genomic studies. In fact, virtually all medical data has a selection bias, which is instead not present in census data, see e.g. [R1, R2].

As for the US Census, this is an example of a large database from which covariances can be estimated from public data: the data can be downloaded directly from https://data.census.gov/table/ or used through the preprocessed version provided by the Folktables dataset [R4], and covariance estimates can then be extracted, as done for instance in Figure 2 (page 13, reported in the supplementary material) of [R3]. However, the publicly available data (1) does not contain data at the level of single individuals, and (2) does not include some features that may be of interest for the task at hand. In the US, data brokers sell features such as precise geolocation, health information, and political beliefs about the same population (see, for instance, the California data broker registry: https://cppa.ca.gov/data_broker_registry/). Estimating the covariance from Census data can thus help in deciding whether or not to purchase such data. Providing aggregated information to potential buyers, through a small sample of the data or public data visualizations, is also a common practice.

We have added a discussion on this point at the end of Section 5 of the revision.

[R1] Schoeler et al., "Participation bias in the UK Biobank distorts genetic associations and downstream analyses", Nature Human Behavior, 2023.

[R2] Benonisdottir and Kong, "Studying the genetics of participation using footprints left on the ascertained genotypes", Nature Genetics, 2023.

[R3] Cheng, Cheng and Duchi, "Collaboratively Learning Linear Models with Structured Missing Data", NeurIPS 2023.

[R4] Ding, Hardt, Miller and Schmidt, "Retiring Adult: New Datasets for Fair Machine Learning", NeurIPS 2021.

Thanks to the authors for their response. The fact that tens of public samples suffice, as well as some of the arguments made using real examples for why the problem setting is realistic, have addressed my concern. I remain in support of acceptance and will raise the sub-scores accordingly.

Comments on the paper

1 comment

We thank the Reviewers and the AC for their work.

Here is a recap of the main points discussed in this phase:

Reviewers TBCT, LAdA and coYN asked which population quantities can be estimated or obtained before buying the private data and how much public data is needed. In Section 5, we explained which quantities can be obtained from public data in the UK Biobank and US Census examples. With as few as 50 reference samples, the risk predictions in Figure 2 remain reliable (Figure 5). Even when we use the reference data for training, adding private data can still improve prediction (Figure 6). Reviewer TBCT confirmed that these additions addressed their concern.

Reviewer LAdA asked how our method compares with selecting datasets by target risk or model similarity on the reference data. We added both baselines to Figure 3. On LSAC, the three methods give similar results. On Lianjia, our method outperforms them.

Reviewer LAdA asked what privacy budgets correspond to our experiments, and Reviewer coYN asked when privacy changes the decision to buy a dataset. We reported the privacy budgets in Table 1 and clarified how Theorem 4.1 relates privacy noise to target error. The same amount of label noise can have a different effect depending on the auxiliary and target covariances and sample sizes. We clarify this point explicitly with an example in Section 5: Figure 2(c) predicts that about 75 private samples suffice to reduce target risk by 0.2 at ρ≈3.5\rho \approx 3.5; with a stronger privacy guarantee (ρ≈1\rho \approx 1), the predicted reduction stays below 0.2 for all sample sizes shown. Furthermore we added a Figure 7 which, for each privacy level and target sample size, shows how many private samples we predict are needed to reduce target error by at least 1% compared with training on non-private data alone, thus informing potential acquisition decisions.

Reviewer coYN also asked how reliable the decision is when the predicted gain is close to the buyer’s threshold. We additionally established a non-asymptotic bound for the prediction risk which scales as n−1/2n^{-1/2} in Proposition B.2. This can be used for uncertainty quantification, via the calculation of a confidence interval from Eq. (B.74).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.