acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Multi-Source Data Collection under Distribution Shifts

Abstract

When labeled data from a target population are scarce, one can acquire additional training observations from related sites. These sites often differ in their covariate distributions and conditional outcome distributions , making their value for target prediction unequal and initially unknown. We present an adaptive framework for planning batched data collection across source sites under a fixed budget. We formulate the problem as a Markov decision process, in which states are posterior beliefs about source response functions and actions are site selection for the next batch. Our objective is to reduce disagreement among predictors fitted to the available target labels, measured through their prediction variance over the target population. We focus on one-step lookahead, which selects the next batch to minimize the expected remaining predictor variance. In a tractable Gaussian model, we show that exact one-step lookahead is asymptotically optimal for this objective under regularity and sufficient-sampling conditions. Our framework supports a range of posterior modules (e.g., BNNs and TabPFN) for simulating source labels, as well as different prediction models. Experiments with linear and neural synthetic data and U.S. census income records show improved target prediction over several data collection baselines.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Adaptive data collection, Distribution shift, Bayesian decision-making, Sequential experimental design, Uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UrtZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an adaptive MDP framework for planning batched data collection across source sites under a fixed budget.

Critical strengths
  1. Figure 1 is excellent. It is clear, helps to summarize the method flow in great details and improves the readability.

  2. The perspective of viewing the problem as a Markov Decision Process (MDP) is quite interesting.

  3. Experimental results are quite convincing compared to the existing baselines.

Critical weaknesses

Major comments:

  1. For Assumption 2, it would be helpful to provide an immediate explanation afterwards, what does rtkr_t^k mean there? I didn't see a definition of rr before Assumption 2 (or it is not a too frequent or explicit notation used before). And ntkn_t^k is the sample size of site kk, right? So does this mean the sample size is changing over updates. Is there any monotonicity of information gain (sample size increasing) over tt? Is there a requirement for the source sample size must be larger than the target sample size? In addition, some small experiments on the violation of Assumption 2 (limited source sites) would be beneficial. It can be helpful to see whether the method performs no worse than the target-only method (or any baselines compared) under limited source data. If it cannot be better than some baselines, it is also understandable because the sample size requirement is an assumption made, but it is helpful to see the sensitivity empirically.

  2. The experiment does not assess a natural target-only benchmark? Is it not applicable here, or could the author(s) add it?

Minor comments:

  1. Connection to federated learning that does not rely on conditional homogeneity can also be mentioned:

Han, Larry, et al. "Federated adaptive causal estimation (face) of target treatment effects." Journal of the American Statistical Association 120.551 (2025): 1503-1516.

Liu, Y., Levis, A., Zhu, K., Yang, S., Gilbert, P. and Han, L., 2026, April. Privacy-Protected Causal Survival Analysis Under Distribution Shift. In International Conference on Learning Representations (Vol. 2026, pp. 94685-94722).

  1. The author(s) can also specify more explicitly what are summaries are shared across sites, whether there is a leading analysis center to aggregate information, whether individual-level summaries are needed or just site-level averages (such as a meta-analysis-type framework), and whether the estimation precision is expected to be increased in the order of source sample sizes such as o(n−1/p)o(n^{-1/p}) for some p>0p>0... Or, if the paper does not focus on privacy issue, i.e., information can be pooled across sites.

  2. If there is a chance before the camera-ready version, I would strongly recommend extend the Conclusion section, at least to be 1/4 page length. Discussion on 1-2 limitations and direct extensions would be helpful. Discussion with nonparametric frequentist and meta-analysis methods are also helpful.

  3. Page 3, line 151: "subscripts index acquisition rounds" is confusing, does this mean distribution of specified variables?

  4. Page 3, line 152 afterwards: The sample size statement for target is a little bit confusing, do you mean there are possibly more covariate data than the outcome in the target? For example, if the target site have 1000 observations, it can have only 100 observed outcomes and 900 missing outcomes? Maybe re-organize this part a little bit. Does every source site need to have sufficiently larger sample size to the target under the framework? Is there any sample size order requirement, e.g., ntgt=o(nk)n_{tgt} = o(n_k) for every kk?

  5. Is there any main advantage for using squared loss? Any practical recommendations for the loss choice?

  6. If there is no prior for the target conditional mean function ff, is there any way to estimate it, such as maximum likelihood estimation (MLE), or using any learning method, such as ensemble learning?

  7. For in-text formula presentation, consider using a/ba/b instead of ab\frac{a}{b} and M−1M^{-1} instead of 1M\frac{1}{M}; see lines 287 and 345.

Questions for authors

See above.

2 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for the careful review and positive assessment, and we are glad that the reviewer finds the MDP perspective interesting and the experimental results convincing. The reviewer's comments helped us clarify our definitions and sample sizes, compare our method with a target-only model, connect our work to federated methods, and expand the conclusion. We have updated the manuscript to reflect these changes, and we address each point below.

M1 and M2: Assumption 2, sample sizes, limited source data, and a target-only benchmark

We thank the reviewer for these suggestions. Section 4 now defines these counts before the sufficient-sampling assumption.

Definitions. Yes, the sample size of site kk changes over rounds. In our notation, it is rtkr_t^k, the number of labeled observations from source kk through round tt, including initialization; nt=∑krtkn_t = \sum_k r_t^k, without a superscript, is the total over all sources. Buying bb labels from source kk increases rtkr_t^k by bb and leaves the other counts unchanged. The source distributions themselves remain fixed.

Monotonicity. The counts are nondecreasing. Under coherent Bayesian updating, the predictor variance also decreases in conditional expectation, E[C(μt+1)∣Dt,at]≤C(μt)\mathbb E[C(\mu_{t+1}) \mid \mathcal D_t, a_t] \le C(\mu_t). Neither the realized variance nor the target MSE must decrease after every batch.

Source and target sample sizes. No, there is no such requirement: the method applies for any source and target sample sizes. Our asymptotic analysis keeps the target sample fixed while the source samples grow.

Limited source data and a target-only benchmark. In our setting, the labeled target sample is fixed, and only source data are collected: at each round, the policy decides which source to buy labels from. A target-only model is fitted to the target labels alone, so it uses none of the collected source data, and its error stays constant as collection proceeds. It is therefore a fixed reference rather than a competing collection policy. Our experiments show how the method compares with it under limited data. Appendix G.7 (Figures 11–12) tracks allocation and prediction from initialization through 100 rounds: target-only BLR fitted to the 100 target labels has a test MSE of 2.728 at every round, while one-step lookahead falls below it after the first purchase and reaches about 2.52 at round 100 (2.54 for BNN, compared with 2.645 for target-only). Appendix G.10 (Figure 18) starts with only 10 labels per source. A target-only model can do better when the sources are less informative about the target: under strong covariate shift (Appendix G.9), and when the target lies outside the class of the source mixture (Appendix G.10, where target-only least squares gives 2.81, compared with 2.87 for one-step lookahead).

Minor 1: Connection to federated learning without conditional homogeneity

We thank the reviewer for these helpful references. Section 2 now discusses Han et al. (2025), Federated Adaptive Causal Estimation (FACE) of Target Treatment Effects, and Liu et al. (2026), Privacy-Protected Causal Survival Analysis Under Distribution Shift. FACE combines density-ratio adjustment, adaptive source weighting, and the communication of summary statistics. It relies on transportability for an informative subset of sources, rather than conditional homogeneity of every source. Liu et al. study federated causal survival effects at a target site under their identification and exchangeability assumptions.

These methods adapt how information is borrowed from heterogeneous sites for causal estimation. Our question is complementary: which new source labels to buy under a collection budget for target prediction. The revised manuscript makes this distinction and discusses distributed summaries as an extension. We do not claim that our allocation theorem provides causal identification or privacy guarantees.

Minor 2: What information is shared across sites

The formulation permits centralized access to source data and imposes no privacy or communication constraint; this is now explicit in Section 5. A coordinator can retain the collected source observations and the fixed target data, update the source beliefs, and score candidate purchases. With known-noise BLR, sites can instead communicate the sufficient statistics Xk⊤XkX_k^\top X_k and Xk⊤ykX_k^\top y_k rather than individual observations. Inference with unknown noise needs additional information, such as yk⊤yky_k^\top y_k, and general predictive modules may require individual-level data.

Under the regularity and sufficient-sampling conditions of the Gaussian theorem, the predictor-variance objective has leading order C(μt)∼(∑kck)2/ntC(\mu_t) \sim \big(\sum_k \sqrt{c_k}\big)^2 / n_t under the Oracle allocation. This is a statement about variance; it is not a general meta-analytic precision law, a rate for target MSE, or a privacy guarantee for arbitrary shared summaries.

AuthorsOpenReview ↗

Response Part 2

Minor 3: Expanding the conclusion

We thank the reviewer for this suggestion and have expanded the conclusion. It now discusses two limitations: the prediction error that remains despite reduced source uncertainty, and the accuracy and computational cost of posterior-predictive simulation. It also discusses direct extensions, including target-only components, nonparametric bootstrap or ensemble uncertainty, and meta-analysis using site-specific estimates and uncertainty summaries, which would require suitable calibration, update rules, and communication assumptions.

Minor 4: Notation on page 3, line 151

We have clarified this in Section 3. The subscript tt in Dtk\mathcal D_t^k and μt\mu_t indexes acquisition rounds: the datasets and posteriors change as observations are collected, while the population distributions PkP^k remain fixed. The subscripts XX and Y∣XY\mid X, by contrast, identify the covariate marginal and the conditional outcome law.

Minor 5: Target sample sizes

Yes. In the reviewer's example, the 100 labeled pairs form the target-fitting sample, while all 1,000 observed covariate vectors can be used to approximate the target expectation in the acquisition criterion; the other 900 outcomes are unobserved. Section 3 now distinguishes the mm labeled target pairs from the NUN_U available target covariates and includes this example. There is no finite-sample requirement that each source exceed the target sample. The Gaussian theorem fixes the target sample and lets the source counts grow, with a separate full-rank condition on the target design.

Minor 6: The choice of squared loss

Squared loss targets the conditional mean and gives a transparent acquisition criterion. For hˉ(x)=Eμ[hθ(x)]\bar h(x) = \mathbb E_\mu[h_\theta(x)],

C(μ)=EXVarμ[hθ(X)]=EXEμ[(hθ(X)−hˉ(X))2].C(\mu) = \mathbb{E}_X \mathrm{Var}_\mu [h_\theta(X)] = \mathbb{E}_X \mathbb{E}_\mu [(h_\theta(X) - \bar{h}(X))^2].

It measures predictor uncertainty in squared prediction units, admits the total-variance identity for uncertainty reduction, and yields an explicit Gaussian allocation analysis. It does not, however, imply optimal population target MSE.

In practice, the fitting loss should match the task: squared loss for conditional means, absolute or quantile loss for medians or other quantiles, and log or Brier loss for class probabilities. The fitting loss and the acquisition discrepancy are separate choices. Appendix C.3, “Classification and other losses,” explains how other fitting rules can retain variance-based acquisition or use a compatible measure of disagreement. The Gaussian allocation theorem does not automatically extend to these alternatives.

Minor 7: Estimating the target function without a prior

No prior on the target response is required. Given each draw from the source posterior, the fixed labeled target sample is used to fit the target prediction rule. The Gaussian analysis uses least squares over the source predictions, and Section 5 and the appendix also permit nonlinear combiners and other fitting rules. Maximum likelihood estimation or ensemble learning can therefore be used, provided the construction rule is specified consistently across draws and rounds. A model fitted only to the target labels is also possible and is now included as a reference (M1 and M2), but source acquisition cannot change it unless source information enters its fitting rule.

Minor 8: In-text formulas

We thank the reviewer for this suggestion. We have changed the indicated inline fractions to slash notation and written the Monte Carlo averaging factor as M−1M^{-1}; the changes are marked in blue in the manuscript.

Reviewer_bC1ROpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper uses a Markov decision process to make decision in sequentially choosing source data with the goal of minimizing the prediction variability in the target domain. It takes a Bayesian approach to avoid complex optimization issues. In a very special scenario when everything is Gaussian and the models are all linear, the paper establishes some theoretical properties of the proposed method.

While the proposed work is useful and is an improvement compared to non-adaptive decision making procedures, the method lacks novelty. Markov decision process is a familiar and widely used method, and using it in the multi-source domain adaption problem here is only an application in a different context. The theoretical results are also not very impressive, with theorem 1 more or less a given, while theorem 2 establishes optimality only relative to other one-step procedures.

Indeed, reinforcement learning seems a very natural option to handle this type of problem. It will be very helpful to explore the difference, connection, feasibility and comparison with reinforcement learning approach to this problem.

Critical strengths

The paper applies MDP to the multi-source single target domain shift context and allows arbitrary model structure, model combination scheme and loss function, hence is flexible. Under special setting, the paper provides theoretical results.

Critical weaknesses

The paper lacks innovation in terms of methodology. It is a flexible application of MDP, a very traditional method. The paper also fails to consider the more modern approach to the same problem, which could be a very natural option to solve the problem.

Questions for authors
  1. Will it be possible to provide a clear algorithm to help make the implementation details transparent?
  2. From the current writing, it is not clear how the step t+1 operation benefits from the computation/results already conducted in the step t operation, or does it benefit from it at all? To this end, it may help to describe the special linear/Gaussian case in terms of step t to t+1, instead of a general one-step description.
  3. It is very hard for a reader not to think of the reinforcement learning approach to this problem. I understand that a detailed comparison will be too big a project, but some discussion seems unavoidable.
  4. The asymptotic optimality result is established under the Gaussian linear model, while the general framework allows for much richer models. Can the authors discuss whether the theoretical analysis can be extended to other parametric settings beyond the Gaussian setting?
  5. Theorem 3 establishes asymptotic convergence to the oracle allocation. Can the authors characterize the convergence rate of rtk/ntr_t^k/n_t, C(μt)C(\mu_t)?
  6. Could the authors provide Gaussian experiments with a longer TT showing the evolution of rtk/ntr_t^k/n_t, C(μt)C(\mu_t), and target MSE? This would provide a more direct empirical validation of the asymptotic oracle result.
  7. The experiments use relatively short horizons (T=17,29T=17, 29) and a small number of sources KK, while the main theoretical result is asymptotic. Could the authors provide experiments with other settings (larger KK and/or larger TT) and examine whether the MSE of OSL is still close to oracle? Such experiments would also help identify regimes in which one-step lookahead deviates from the oracle or loses its advantage over the baselines.
  8. Although the motivating setup allows both PXP_X and PY∣XP_{Y|X} to differ across sources, all synthetic covariates appear to follow the same N(0,I10)N(0,I_{10}) distribution. Could the authors provide systematic experiments with covariate shift?
  9. Since the empirical result of the proposed method is close to oracle, I wonder if there is any computational challenge when implementing the method. Could the authors provide a computation time comparison across the methods used in the experiments?
3 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for the careful review and detailed questions, and we are glad that the reviewer finds the work useful and an improvement over non-adaptive procedures, and the framework flexible in allowing arbitrary model structures, combination schemes, and loss functions. The reviewer's questions helped us clarify our contributions and strengthen the paper. We now explain our contributions more precisely and discuss the connection to reinforcement learning (W1 and Q3). We clarify the strength of our theoretical results, add a finite-horizon guarantee in an affine Gaussian model, and discuss extensions beyond the Gaussian setting and the convergence rate (W2, Q4 and Q5). We make the algorithm and its round-to-round updates easy to find (Q1 and Q2), and we add new experiments with longer horizons, more sources, and systematic covariate shift (Q6, Q7 and Q8), together with a comparison of computation time (Q9). We have updated the manuscript to reflect these changes, and we address each point below.

W1 and Q3: Methodological novelty and the connection to reinforcement learning

We thank the reviewer for this comment. We would like to use this opportunity to clarify our contribution and to discuss the connection to reinforcement learning.

A belief-MDP framework for adaptive multi-source data collection. As the reviewer rightly points out, MDPs are a well-established tool, and our contribution is not a new generic MDP method. Our contribution is framing the problem of multi-source data collection under distribution shift as a belief MDP, in which new labels are valued by how much they reduce uncertainty in the predictor fitted to the target's labels (Section 3). This framing is what gives the flexibility the reviewer notes: the source models, the combination rule, and the loss can all be chosen freely.

Theoretical insight: one-step lookahead achieves the oracle allocation. In a Gaussian setting, we characterize the oracle allocation of labels across sources and prove that one-step lookahead achieves it (Theorem 3). It does so without estimating the unknown quantities on which the oracle allocation depends, and no policy that samples every source sufficiently often can do better asymptotically.

Model-agnostic implementation of the belief-MDP. Our third contribution is a way to implement one-step lookahead that is not tied to the Gaussian model or to any particular model class. The challenge is where the beliefs come from. A belief state usually requires a model for each source, such as Bayesian linear regression with a prior, and committing to such a model would restrict the method to that class. Instead, our implementation only needs a predictive distribution for the labels a source would return, given the data collected so far. Tabular foundation models such as TabPFN provide exactly this: they perform posterior inference implicitly, so we can simulate source labels without specifying a parametric model (Section 5; Appendix F). Classical Bayesian models, such as Bayesian linear regression and Bayesian neural networks, fit into the same framework. Given these predictive distributions, we implement one-step lookahead by simulation (Section 5; Appendix E.1, Algorithm 1).

Connection to reinforcement learning. We thank the reviewer for this suggestion and agree that the connection deserves discussion. Our belief MDP is a Bayesian model-based reinforcement learning problem: the state is the current belief about the sources and the data collected so far, an action buys labels from one source, and the objective is the uncertainty remaining in the target predictor at the end of collection. Reinforcement learning is therefore not an alternative to our formulation but a way to solve it. One-step lookahead is one such solution; another is a policy learned over the full horizon, as in deep adaptive design (Foster et al., 2021). We use one-step lookahead because it needs no training on simulated collection problems, and because looking one step ahead already suffices in the Gaussian setting, where it attains the asymptotically optimal allocation (Theorem 3). Learned policies are a promising direction when many similar collection problems are available. We have added this discussion to Section 2 and the Conclusion.

AuthorsOpenReview ↗

Response Part 2

W2, Q4 and Q5: Strength of the theory, extensions beyond the Gaussian setting, and convergence rate

We thank the reviewer for these questions. We would like to use this opportunity to clarify the contribution of our theoretical results.

Strength of the results. We agree that Theorem 1 follows from the law of total variance. We include it because it gives the exact one-step gain, tr⁡(ΣtgtCov⁡[ψˉt+1])\operatorname{tr}\big(\Sigma^{\mathrm{tgt}}\operatorname{Cov}[\bar\psi_{t+1}]\big), on which the allocation analysis builds. Our main theoretical insight is that one-step lookahead achieves the oracle allocation (Theorem 3). In the Gaussian setting, the asymptotically optimal allocation has the square-root form of Neyman allocation, αk∝ck\alpha_k\propto\sqrt{c_k}, and we use it only as an oracle benchmark (Lemma 2). Importantly, one-step lookahead achieves this allocation without computing it. The oracle allocation depends on the unknown source parameters through the coefficients ckc_k and the source sensitivities JkJ^k. One-step lookahead never estimates these quantities; it only compares the expected reduction in predictor variance from each source, and it reaches the oracle allocation automatically. Theorem 3 is also stronger than optimality among one-step procedures: its lower bound (Eq. 9) applies to every non-anticipating policy that samples each source sufficiently often (Assumption 2), so no such policy can do better asymptotically. We now also add a finite-horizon guarantee: in an affine Gaussian model, one-step lookahead is exactly optimal at every finite horizon among all non-anticipating policies, and it samples every source sufficiently often without forced exploration (Proposition 11, Appendix D.5).

Beyond the Gaussian setting (Q4). We restrict our theory to the Gaussian setting to provide clean insight into the behavior of one-step lookahead. This insight motivates our general empirical methodology, which applies well beyond the Gaussian setting. We expect the analysis to extend to regular, well-specified parametric source models. If the posterior covariance of source kk is approximately Ik(θk⋆)−1/rtkI_k(\theta_k^\star)^{-1}/r_t^k and the fitted target predictor is differentiable in the source parameters, a delta-method expansion gives C(μt)≈∑kckrtk,ck=EX∼PXtgt[gk(X)⊤Ik(θk⋆)−1gk(X)],C(\mu_t) \approx \sum_k \frac{c_k}{r_t^k}, \qquad c_k = \mathbb E_{X\sim P_X^{\mathrm{tgt}}}\big[g_k(X)^\top I_k(\theta_k^\star)^{-1} g_k(X)\big], where gk(x)g_k(x) is the derivative of the target predictor with respect to θk\theta_k. The oracle allocation then has the same form, αk∝ck\alpha_k \propto \sqrt{c_k}. Proving that one-step lookahead attains it would also require control of the posterior under adaptive sampling, which is an interesting direction for future work. We now include this discussion in the Conclusion.

Convergence rate (Q5). Theorem 3 shows that ntC(μt)→c⋆=(∑jcj)2n_t C(\mu_t) \to c^\star = \big(\sum_j \sqrt{c_j}\big)^2 almost surely, so C(μt)=c⋆/nt+o(1/nt)C(\mu_t) = c^\star/n_t + o(1/n_t): the objective decreases at rate 1/nt1/n_t with the optimal constant. A quantitative rate for the allocation rtk/ntr_t^k/n_t is harder to obtain. One-step lookahead reaches the oracle allocation only through the expected gains it compares, so a rate for the allocation requires a rate for how fast these gains, and hence the posterior, concentrate (Theorem 7, Lemma 8). Because the data are collected adaptively, standard arguments for independent samples do not apply, which makes this an interesting direction for future work. Empirically, in our new 100-round experiments, the allocation of one-step lookahead closely tracks the oracle allocation (Appendix G.7, Figures 11–12; see Q6).

AuthorsOpenReview ↗

Response Part 3

Q1 and Q2: A clear algorithm, and how each round builds on the previous one

We thank the reviewer for these suggestions. We apologize that the algorithm was not easy to find in the appendix. The full procedure is given in Appendix E.1, Algorithm 1 (“One-step site selection”), which specifies the joint simulation of each batch, updates from copies of the current posterior, Monte Carlo scoring, source selection, and the update after the real data arrive. We now refer to Algorithm 1 directly in the main text (Section 5), so that it is easy to find.

Each round builds on the previous one implicitly, through the posterior: information from every real acquisition is retained in the next posterior, and the procedure never restarts from the prior. The lookahead itself, however, is recomputed from scratch at every round: hypothetical batches are simulated from copies of the current posterior and discarded after scoring, so the scores of the candidate sources are computed afresh from the updated posterior. For Bayesian linear regression with known noise, the state of source kk can be stored as its precision Λtk=(Vtk)−1\Lambda_t^k = (V_t^k)^{-1} and information vector ηtk=Λtkmtk\eta_t^k = \Lambda_t^k m_t^k. A new batch (Xb,yb)(X_b, y_b) from source kk updates them as Λt+1k=Λtk+(σk)−2Xb⊤Xb,ηt+1k=ηtk+(σk)−2Xb⊤yb,\Lambda_{t+1}^k = \Lambda_t^k + (\sigma^k)^{-2} X_b^\top X_b, \qquad \eta_{t+1}^k = \eta_t^k + (\sigma^k)^{-2} X_b^\top y_b, and then Vt+1k=(Λt+1k)−1V_{t+1}^k = (\Lambda_{t+1}^k)^{-1} and mt+1k=Vt+1kηt+1km_{t+1}^k = V_{t+1}^k \eta_{t+1}^k. Only the selected source's state changes, and with unknown noise each conditional Gaussian on the noise grid is updated in the same way, together with its grid weight. As the reviewer suggests, this round-to-round recursion is now explicit in Appendix E.1.

Q6, Q7 and Q8: Longer horizons, more sources, and covariate shift

We thank the reviewer for these suggestions, which led to three new sets of experiments.

Longer horizons (Q6). We added BLR and BNN experiments with 100 rounds, which show rtk/ntr_t^k/n_t, C(μt)C(\mu_t), and target MSE over rounds (Appendix G.7, Figures 11–12). One-step lookahead closely tracks the Oracle's allocation and objective, and its allocation approaches the limit predicted by our theory. FG, uniform collection, and DAD allocate more labels away from the Oracle proportions.

More sources (Q7). We added experiments with K=5K = 5, 1010, and 2020 sources over 100 rounds, in which either 40% or all of the sources contribute to the target (Appendix G.8, Figure 13). On average over these settings, the epistemic objective of one-step lookahead after 100 rounds is 18% lower than that of FG, 59% lower than that of uniform collection, and 96% lower than that of DAD.

Covariate shift (Q8). Our synthetic experiments now also include systematic covariate shifts, in addition to conditional outcome shifts. We added BLR and BNN experiments with shifts in the source covariate means, scales, and covariance shapes, while retaining heterogeneous source response functions and hidden noise (Appendix G.9, Figures 14–17); the target covariates remain N(0,I10)\mathcal N(0, I_{10}). Across both noise configurations, 30 rounds, and 10 seeds, one-step lookahead follows the Oracle allocation as the shifts change it. On average over all shift settings, it recovers 94% (BLR) and 91% (BNN) of the Oracle's improvement over a static allocation on the epistemic objective, compared with at most 14% for the other policies. In 47 of the 48 settings, it also has the highest recovery and the lowest (or tied-lowest) test error among the acquisition policies.

Q9: Computation time

We thank the reviewer for this question. We now report the computation time of every method (Appendix G.7.1, Table 10). One-step lookahead costs more than the baselines because it simulates future data: on one CPU core, a decision takes 39 seconds for BLR and 46 seconds for BNN, compared with under a second for FG, DAD, and uniform collection. When labels are costly to collect, as when a hospital obtains labeled patient records from partner hospitals or survey responses are collected in a new region, this time is small relative to the cost of a batch of labels. It can also be reduced: the simulations for different candidate sources are independent and can run in parallel, and with fewer posterior draws a decision takes about 3–6 seconds in our covariate-shift experiments.

Reviewer_z9uJOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies adaptive data collection from multiple source populations to improve prediction on a target population with limited labeled data. The authors formulate the problem as a belief MDP and propose a one-step lookahead policy that minimizes posterior variance of the target predictor. They establish asymptotic optimality of the source allocation in a Gaussian setting and show empirical improvements on synthetic and real-world data.

Critical strengths
  • Relevant and practically motivated problem of data collection under distribution shifts.

  • Flexible framework supporting different posterior models (e.g., BNNs, TabPFN) and prediction models.

  • Theoretical analysis establishing asymptotic oracle allocation, complemented by empirical evaluations.

Critical weaknesses
  • Limited novelty. The belief-MDP formulation and one-step Bayesian lookahead are established techniques in experimental design and adaptive labeling. The optimal allocation follows the classical Neyman allocation structure and can be derived through influence functions and the delta method. Related adaptive oracle-allocation guarantees already exist (e.g., Gupta et al., 2024).

  • Relatedly, questionable necessity of the MDP formulation. The asymptotic allocation could potentially be achieved using a simpler adaptive plug-in estimator of source-specific variance contributions, without posterior-predictive rollouts.

  • Mismatch between objective and prediction performance. Minimizing predictor variance does not necessarily minimize target prediction error, especially under conditional outcome shifts and model misspecification.

  • Restrictive theoretical guarantees. The optimality result applies to Gaussian linear models, requires sufficient sampling of every source, and does not establish finite-budget optimality.

  • Limited empirical validation. Synthetic experiments consider only conditional outcome shifts, use four seeds, and select the more favorable noise configuration for the main figures. The real-world experiment has only one run and does not achieve the best final performance.

Questions for authors
  • Necessity of sequential planning: Could the same asymptotic optimality be achieved by estimating the coefficients ckc_k and adaptively selecting sources according to ck/nk2c_k/n_k^2? What advantages does the proposed MDP formulation offer over this simpler alternative?

  • How does the method perform under simultaneous covariate and conditional outcome shifts, and when the target response cannot be adequately represented by the source models?

3 comments
AuthorsOpenReview ↗

Response Part 1

We thank the reviewer for the careful and detailed review, and we are glad that the reviewer finds the problem relevant and practically motivated, the framework flexible, and the theoretical analysis well complemented by empirical evaluation. The reviewer's comments helped us clarify our contributions and strengthen the paper. We now explain our contributions more precisely (W1), the advantages of one-step lookahead over plug-in rules (W2 and Q1), and when reducing predictor variance also reduces prediction error (W3). We add a setting in which one-step lookahead needs no forced exploration and is exactly optimal at every finite horizon (W4), and new experiments with simultaneous covariate and conditional outcome shifts and with targets the sources cannot represent, together with a re-conducted ACS experiment with 30 evaluation seeds (W5 and Q2). We have updated the manuscript to reflect these changes, and we address each point below.

W1: Novelty relative to belief MDPs, Bayesian lookahead, and Neyman allocation

We thank the reviewer for this comment. We would like to use this opportunity to clarify our contribution.

A belief-MDP framework for adaptive multi-source data collection. As the reviewer rightly points out, belief MDPs and one-step lookahead are well-known tools. However, our contribution is framing the problem of multi-source data collection under distribution shift as a belief MDP, in which new labels are valued by how much they reduce uncertainty in the predictor fitted to the target's labels (Section 3).

Theoretical insight: one-step lookahead achieves the oracle allocation. To understand one-step lookahead, we analyze it in a Gaussian setting (Section 4). We first derive the asymptotically optimal allocation, which has the square-root form of Neyman allocation, αk∝ck\alpha_k\propto\sqrt{c_k} (Lemma 2); as the reviewer points out, this allocation is well known and can be derived with the delta method. Our key contribution is to prove that one-step lookahead achieves it (Theorem 3).

Notably, one-step lookahead reaches this allocation without ever computing it. The oracle allocation depends on the unknown source parameters through the coefficients ckc_k and the source sensitivities JkJ^k. Existing adaptive allocation methods estimate these unknown quantities from the data and then compute the allocation (e.g., Chernoff, 1953; Ford and Silvey, 1980; Hahn et al., 2011; Dai et al., 2023; Gupta et al., 2024). One-step lookahead never estimates ckc_k or JkJ^k, and it never computes or optimizes the oracle allocation. It only compares the expected reduction in predictor variance from each source, and it reaches the oracle allocation automatically. This is why the same method applies beyond the Gaussian setting. With neural networks or TabPFN, ckc_k and JkJ^k have no closed form, so plug-in rules are not directly available, but one-step lookahead still applies. Theorem 3 also shows that no policy that samples every source sufficiently often can do better asymptotically. Further, we now add a special case in which this sufficient-sampling condition is not needed: in an affine Gaussian model, one-step lookahead samples every source sufficiently often on its own and is exactly optimal at every finite horizon (Proposition 11, Appendix D.5).

Model-agnostic implementation of the belief-MDP. Our third contribution is a way to implement one-step lookahead that is not tied to the Gaussian model or to any particular model class. The challenge is where the beliefs come from. A belief state usually requires a model for each source, such as Bayesian linear regression with a prior, and committing to such a model would restrict the method to that class. Instead, our implementation only needs a predictive distribution for the labels a source would return, given the data collected so far. Tabular foundation models such as TabPFN provide exactly this: they perform posterior inference implicitly, so we can simulate source labels without specifying a parametric model (Section 5; Appendix F). Classical Bayesian models, such as Bayesian linear regression and Bayesian neural networks, fit into the same framework. Given these predictive distributions, we implement one-step lookahead by simulation (Section 5; Appendix E.1, Algorithm 1).

Gupta et al. (2024). We thank the reviewer for pointing us to Gupta et al. (2024), Online Data Collection for Efficient Semiparametric Inference. They choose sequentially among data sources to estimate a parameter defined by moment conditions. The optimal allocation depends on unknown parameters, so they estimate it from the data collected so far and allocate accordingly (explore-then-commit and explore-then-greedy). They show that this plug-in approach achieves the oracle's asymptotic MSE. We compare such a plug-in rule with one-step lookahead in our response to W2 and Q1 below.

AuthorsOpenReview ↗

Response Part 2

W2 and Q1: Necessity of the MDP formulation compared with a plug-in rule

We thank the reviewer for this question, which helped us clarify the role of the belief-MDP formulation. As the reviewer suggests, in the Gaussian setting a plug-in rule is a natural alternative that targets the same asymptotic allocation. However, one-step lookahead, the policy derived from our belief-MDP formulation, has two advantages over such rules.

Generality: one-step lookahead (belief MDP) needs only a simulator, while plug-in rules need a closed form for ckc_k. The plug-in rule requires a closed form for ckc_k, the amount by which each source's labels reduce the variance of the target predictor. Such a closed form exists for tractable models such as our Gaussian model, which we use only for theoretical motivation. For many other models, no closed form exists, including models that often work best on real-world tabular data: nonsmooth predictors, such as random forests, and models without explicit parameters, such as TabPFN or Kumo (Grinsztajn et al., 2022; Hollmann et al., 2025). Even where a closed form exists, it can be highly complex, for example when the sources share structure so that labels from one source also inform the others, and it has to be derived and estimated separately for each model. One-step lookahead is a single procedure for all of these cases: it only needs to simulate labels from the current beliefs.

Finite budgets: one-step lookahead (belief MDP) performs well with small budgets, while plug-in rules require many labels. Our asymptotic result is only for theoretical motivation; in general, one-step lookahead is designed for any finite budget. The plug-in rule, by contrast, is justified only as the number of labels grows, while data collection runs for a finite, often small, number of rounds. With few labels, the estimates c^k\hat c_k can be poor, and an allocation built on wrong estimates can be arbitrarily far from optimal. One-step lookahead instead averages over the current posterior rather than plugging in point estimates, so its choices account for what is still unknown about each source.

Experiments. We added a small Gaussian experiment with two sources, which shows this advantage in a low-data regime and at finite horizons (Appendix G.12): each source starts with 6 labels, and 30 labels are bought one at a time. One-step lookahead uses prior knowledge about the sources through its posterior, whereas the plug-in rule does not. As a result, the epistemic variance of the plug-in rule is 28–52% higher at every budget, and the test error of one-step lookahead is up to 32% lower.

W3: Objective versus prediction performance

We thank the reviewer for this comment. We agree that reducing predictor variance does not in general minimize target prediction error. This holds for any acquisition rule that reduces uncertainty, because the prediction error itself is unknown while the data are being collected. In general, it is almost impossible to guarantee that the two agree, or to fully remedy the mismatch.

However, the gap can be small when the belief model is well specified, so that it covers the true source response functions, and the prediction model is rich enough to represent the target response function; reducing C(μt)C(\mu_t) then reduces target prediction error. Modern models have high capacity: tabular foundation models such as TabPFN and Kumo give accurate predictive distributions on real-world tabular data (Hollmann et al., 2025), and flexible predictors such as neural networks and random forests perform well on tabular data (Grinsztajn et al., 2022). Our framework accommodates both (Section 5).

Our experiments showcase this to some extent. With neural sources, a neural combination of them as the target, and a random-forest predictor, one-step lookahead has lower target error than FG, uniform collection, and DAD throughout collection, ending at 4.31 versus 5.22 for FG and 5.05 for uniform (Section 6, Figure 6). With neural sources and a linear target, FG needs 22 rounds to match the error that one-step lookahead reaches after 10, and with a random-forest predictor one-step lookahead has lower mean error than the baselines in every round (Figure 5). On real ACS data, one-step lookahead has the lowest loss among the acquisition policies (Appendix G.11, Table 15).

This is in no way to suggest that our approach covers every case or that the mismatch never arises. How well reducing predictor variance reduces prediction error is limited by the quality of the belief model and the prediction model. We show a case where the two diverge: a target outside the class of the prediction model leaves an error that no allocation can remove (Appendix G.10). We also state this limitation explicitly in the Conclusion.

AuthorsOpenReview ↗

Response Part 3

W4: Restrictive theoretical guarantees

We thank the reviewer for this question. We would like to emphasize that we restrict our theory to the Gaussian setting to provide clean insight into the behavior of one-step lookahead. This insight motivates our general empirical methodology, which applies well beyond the Gaussian setting. Extending our guarantees beyond Gaussian models, to finite budgets, and without the sufficient-sampling assumption are interesting directions for future work.

To go beyond the sufficient-sampling assumption, we now add a setting in which it is not needed. In a simpler affine Gaussian model, pure one-step lookahead samples every source sufficiently often on its own, without forced exploration, and it is exactly optimal at every finite horizon (Proposition 11, Appendix D.5, referenced in Section 4). In general, sufficient sampling is difficult to prove, and we enforce it by reserving a vanishing fraction of rounds for forced exploration, which does not change the oracle limits (Corollary 10).

W5 and Q2: Empirical validation, simultaneous shifts, and targets the sources cannot represent

We are grateful for these comments, which led us to run a more rigorous set of experiments, ground our claims more firmly, and state their limitations explicitly. We address each point in turn.

Simultaneous covariate and conditional outcome shifts. Our synthetic experiments now also include systematic covariate shifts, in addition to conditional outcome shifts. We added BLR and BNN experiments with shifts in the source covariate means, scales, and covariance shapes, while retaining heterogeneous source response functions and hidden noise (Appendix G.9, Figures 14–17); the target covariates remain N(0,I10)\mathcal N(0, I_{10}). Across both noise configurations, 30 rounds, and 10 seeds, OSL follows the Oracle allocation as the shifts change it. On average over all shift settings, it recovers 94% (BLR) and 91% (BNN) of the Oracle's improvement over a static allocation on the epistemic objective, compared with at most 14% for the other policies. In 47 of the 48 settings, it also has the highest recovery and the lowest (or tied-lowest) test error among the acquisition policies.

Number of seeds and noise configurations. All new experiments use 10 or more seeds, and the redesigned ACS experiment uses 30. The new covariate-shift and larger-KK experiments report both noise configurations (Appendices G.8 and G.9). For the original comparisons, the appendix reports the other noise configuration for each synthetic figure in the main text (Figures 4–6; Appendix G.6, Figures 8–10), and one-step lookahead has the lowest error in all three.

The ACS experiment. We have now re-conducted the real-data (ACS) experiment with 30 evaluation seeds (Figure 7; Appendix G.11, Table 15). With Bayesian linear regression as the belief model, one-step lookahead has the lowest average test loss among all acquisition policies: 9% lower than the best baseline (FG) and 18% lower than uniform collection.

Targets the sources cannot represent. To study this case, we added shared-subspace experiments. The coefficients of all sources lie in a common low-dimensional subspace: each is a random combination of the same three directions. The target's coefficients are a new random draw from this subspace, rather than a weighted average of the sources (Appendix G.10, Figure 18; Table R1). Such a target can lie outside the set of all mixtures of the sources, so the mixture predictor cannot represent it exactly. We consider two settings: linear sources, and neural sources whose last-layer weights share such a subspace. In both, one-step lookahead has the lowest loss among the acquisition policies: 2.87 versus 2.98–3.33 for linear sources and 3.11 versus 3.31–3.64 for neural sources. For linear sources, the gap is larger when the target lies outside all mixtures of the sources (Appendix G.10).

SourcesOSLFGUniformDAD
Linear (Figure 18, left)2.872.983.083.33
Neural (Figure 18, middle)3.113.313.353.64

Table R1: Targets that are not a mixture of the sources: loss averaged over the 20 acquisition rounds, mean over 30 random draws of the target (K=10K=10 sources, 10 initial labels per source, 20 labels per purchase, 100 target labels). In both settings, the target is a new random draw from the subspace shared by the sources, rather than a weighted average of the sources. For neural sources, FG fits random-forest source models (3.31) or OLS source models (3.33); the better of the two is shown. Values are averages over rounds 1–20 of the curves in Figure 18. Source: revised manuscript, Appendix G.10 (Figure 18).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback on our paper. We are glad to see that the reviewers find the problem relevant and practically motivated (z9uJ), appreciate the flexibility of the framework in supporting different posterior modules, prediction models, and loss functions (z9uJ, bC1R), find the MDP perspective interesting (UrtZ), and recognize the theoretical analysis, complemented by empirical evaluation (z9uJ, bC1R), and the experimental results compared with existing baselines (UrtZ). We summarize our key contributions below.

We study which source to collect labels from when several sources differ from the target population, in their covariates, their outcome mechanisms, or both. Our contributions are a belief-MDP framework for this problem, a theoretical insight into how one-step lookahead solves it, and an implementation that works with any belief and predictive model; we summarize each below.

A belief-MDP framework for adaptive multi-source data collection. We formulate this problem as a belief MDP, in which a source is valuable when its data reduce uncertainty in the predictor fitted to the target's labels (Section 3). This single criterion accounts for a source's relevance to the target, its noise level, how well its covariates cover the target, and how much is still unknown about it. As Reviewers z9uJ and bC1R point out, belief MDPs and one-step lookahead are well-known tools; our contribution is to bring them to multi-source data collection under distribution shift, where they give a principled and flexible way to value new data.

Theoretical insight: one-step lookahead achieves the oracle allocation. To understand one-step lookahead, we analyze it in a Gaussian setting (Section 4). We first derive the asymptotically optimal allocation, which has the square-root form of the well-known Neyman allocation (Lemma 2). Our key contribution is to prove that one-step lookahead achieves this allocation (Theorem 3). Notably, one-step lookahead reaches this allocation without ever computing it: it does not estimate the unknown quantities on which the allocation depends, and only compares the expected reduction in predictor variance from each source. This also distinguishes it from plug-in rules, which estimate these quantities and therefore need a closed form for them: one-step lookahead needs only a simulator, and it uses everything in its current beliefs, which matters at small budgets (Appendix G.12).

Model-agnostic implementation of the belief-MDP. Our third contribution is a way to implement one-step lookahead that is not tied to the Gaussian model or to any particular model class. The challenge is where the beliefs come from. A belief state usually requires a model for each source, such as Bayesian linear regression with a prior, and committing to such a model would restrict the method to that class. Instead, our implementation only needs a predictive distribution for the labels a source would return, given the data collected so far. Tabular foundation models such as TabPFN provide exactly this: they perform posterior inference implicitly, so we can simulate source labels without specifying a parametric model (Section 5; Appendix F). Classical Bayesian models, such as Bayesian linear regression and Bayesian neural networks, fit into the same framework. Given these predictive distributions, we implement one-step lookahead by simulation (Section 5; Appendix E.1, Algorithm 1). Empirically, one-step lookahead improves target prediction over existing collection policies on synthetic and real data (Section 6).

AuthorsOpenReview ↗

Summary of Revisions

In response to the reviewers, we have made the following revisions, which are marked in blue in the revised paper:

New experiments. We conducted additional experiments:

  • Longer horizons and more sources (bC1R; Appendices G.7 and G.8). We run the linear and neural experiments over 100 rounds, tracking the allocation, the remaining predictor uncertainty, and the target error. We also increase the number of sources to 10 and 20, with either 40% or all of them contributing to the target.
  • Simultaneous covariate and conditional outcome shifts (z9uJ, bC1R; Appendix G.9). In the linear and neural settings, whose sources already differ in their response functions, we also shift the sources' covariate distributions away from the target's: their means, their spread, and the shape of their covariances.
  • Targets that are not a mixture of the sources (z9uJ; Appendix G.10). Our predictor combines the sources as a weighted mixture, so it can represent the target exactly only if the target is such a mixture. Here the target is related to the sources but is not a mixture of them: all coefficients lie in a common low-dimensional subspace, and the target's coefficients are a separate draw from this subspace rather than a weighted average of the sources' coefficients. We run this with linear sources and with neural sources.
  • ACS with 30 evaluation seeds (z9uJ; Appendix G.11). We redesigned the real-data experiment on ACS income: five sources defined by usual weekly hours worked, the whole workforce as the target, and 30 evaluation seeds instead of a single run.

Across these experiments, one-step lookahead performs best among the acquisition policies on average; the individual responses give the details. We now also highlight the limitations of this approach. Under strong covariate shift, or when the target lies outside the class of the source mixture, a model fitted to the target labels alone can do better (Appendices G.9 and G.10).

Theory. We add a setting in which one-step lookahead needs no forced exploration and is exactly optimal at every finite horizon (z9uJ, bC1R; Proposition 11, Appendix D.5), and the Conclusion now discusses extensions of the theory beyond the Gaussian setting (bC1R).

Presentation. We expand the related work to connect our setting to federated methods (UrtZ; Section 2) and expand the Conclusion with its limitations, including when reducing predictor variance also reduces prediction error, and directions for future work (z9uJ, UrtZ). We also refer to Algorithm 1 directly in the main text and make its round-to-round updates explicit (bC1R; Appendix E.1).

We are grateful to the reviewers for helping us strengthen both the evaluation and the presentation of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.