- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes an adaptive MDP framework for planning batched data collection across source sites under a fixed budget.
Critical strengths
-
Figure 1 is excellent. It is clear, helps to summarize the method flow in great details and improves the readability.
-
The perspective of viewing the problem as a Markov Decision Process (MDP) is quite interesting.
-
Experimental results are quite convincing compared to the existing baselines.
Critical weaknesses
Major comments:
-
For Assumption 2, it would be helpful to provide an immediate explanation afterwards, what does mean there? I didn't see a definition of before Assumption 2 (or it is not a too frequent or explicit notation used before). And is the sample size of site , right? So does this mean the sample size is changing over updates. Is there any monotonicity of information gain (sample size increasing) over ? Is there a requirement for the source sample size must be larger than the target sample size? In addition, some small experiments on the violation of Assumption 2 (limited source sites) would be beneficial. It can be helpful to see whether the method performs no worse than the target-only method (or any baselines compared) under limited source data. If it cannot be better than some baselines, it is also understandable because the sample size requirement is an assumption made, but it is helpful to see the sensitivity empirically.
-
The experiment does not assess a natural target-only benchmark? Is it not applicable here, or could the author(s) add it?
Minor comments:
- Connection to federated learning that does not rely on conditional homogeneity can also be mentioned:
Han, Larry, et al. "Federated adaptive causal estimation (face) of target treatment effects." Journal of the American Statistical Association 120.551 (2025): 1503-1516.
Liu, Y., Levis, A., Zhu, K., Yang, S., Gilbert, P. and Han, L., 2026, April. Privacy-Protected Causal Survival Analysis Under Distribution Shift. In International Conference on Learning Representations (Vol. 2026, pp. 94685-94722).
-
The author(s) can also specify more explicitly what are summaries are shared across sites, whether there is a leading analysis center to aggregate information, whether individual-level summaries are needed or just site-level averages (such as a meta-analysis-type framework), and whether the estimation precision is expected to be increased in the order of source sample sizes such as for some ... Or, if the paper does not focus on privacy issue, i.e., information can be pooled across sites.
-
If there is a chance before the camera-ready version, I would strongly recommend extend the Conclusion section, at least to be 1/4 page length. Discussion on 1-2 limitations and direct extensions would be helpful. Discussion with nonparametric frequentist and meta-analysis methods are also helpful.
-
Page 3, line 151: "subscripts index acquisition rounds" is confusing, does this mean distribution of specified variables?
-
Page 3, line 152 afterwards: The sample size statement for target is a little bit confusing, do you mean there are possibly more covariate data than the outcome in the target? For example, if the target site have 1000 observations, it can have only 100 observed outcomes and 900 missing outcomes? Maybe re-organize this part a little bit. Does every source site need to have sufficiently larger sample size to the target under the framework? Is there any sample size order requirement, e.g., for every ?
-
Is there any main advantage for using squared loss? Any practical recommendations for the loss choice?
-
If there is no prior for the target conditional mean function , is there any way to estimate it, such as maximum likelihood estimation (MLE), or using any learning method, such as ensemble learning?
-
For in-text formula presentation, consider using instead of and instead of ; see lines 287 and 345.
Questions for authors
See above.
2 comments
Response Part 2
Minor 3: Expanding the conclusion
We thank the reviewer for this suggestion and have expanded the conclusion. It now discusses two limitations: the prediction error that remains despite reduced source uncertainty, and the accuracy and computational cost of posterior-predictive simulation. It also discusses direct extensions, including target-only components, nonparametric bootstrap or ensemble uncertainty, and meta-analysis using site-specific estimates and uncertainty summaries, which would require suitable calibration, update rules, and communication assumptions.
Minor 4: Notation on page 3, line 151
We have clarified this in Section 3. The subscript in and indexes acquisition rounds: the datasets and posteriors change as observations are collected, while the population distributions remain fixed. The subscripts and , by contrast, identify the covariate marginal and the conditional outcome law.
Minor 5: Target sample sizes
Yes. In the reviewer's example, the 100 labeled pairs form the target-fitting sample, while all 1,000 observed covariate vectors can be used to approximate the target expectation in the acquisition criterion; the other 900 outcomes are unobserved. Section 3 now distinguishes the labeled target pairs from the available target covariates and includes this example. There is no finite-sample requirement that each source exceed the target sample. The Gaussian theorem fixes the target sample and lets the source counts grow, with a separate full-rank condition on the target design.
Minor 6: The choice of squared loss
Squared loss targets the conditional mean and gives a transparent acquisition criterion. For ,
It measures predictor uncertainty in squared prediction units, admits the total-variance identity for uncertainty reduction, and yields an explicit Gaussian allocation analysis. It does not, however, imply optimal population target MSE.
In practice, the fitting loss should match the task: squared loss for conditional means, absolute or quantile loss for medians or other quantiles, and log or Brier loss for class probabilities. The fitting loss and the acquisition discrepancy are separate choices. Appendix C.3, “Classification and other losses,” explains how other fitting rules can retain variance-based acquisition or use a compatible measure of disagreement. The Gaussian allocation theorem does not automatically extend to these alternatives.
Minor 7: Estimating the target function without a prior
No prior on the target response is required. Given each draw from the source posterior, the fixed labeled target sample is used to fit the target prediction rule. The Gaussian analysis uses least squares over the source predictions, and Section 5 and the appendix also permit nonlinear combiners and other fitting rules. Maximum likelihood estimation or ensemble learning can therefore be used, provided the construction rule is specified consistently across draws and rounds. A model fitted only to the target labels is also possible and is now included as a reference (M1 and M2), but source acquisition cannot change it unless source information enters its fitting rule.
Minor 8: In-text formulas
We thank the reviewer for this suggestion. We have changed the indicated inline fractions to slash notation and written the Monte Carlo averaging factor as ; the changes are marked in blue in the manuscript.
Response Part 1
We thank the reviewer for the careful review and positive assessment, and we are glad that the reviewer finds the MDP perspective interesting and the experimental results convincing. The reviewer's comments helped us clarify our definitions and sample sizes, compare our method with a target-only model, connect our work to federated methods, and expand the conclusion. We have updated the manuscript to reflect these changes, and we address each point below.
We thank the reviewer for these suggestions. Section 4 now defines these counts before the sufficient-sampling assumption.
Definitions. Yes, the sample size of site k changes over rounds. In our notation, it is rtk, the number of labeled observations from source k through round t, including initialization; nt=∑krtk, without a superscript, is the total over all sources. Buying b labels from source k increases rtk by b and leaves the other counts unchanged. The source distributions themselves remain fixed.
Monotonicity. The counts are nondecreasing. Under coherent Bayesian updating, the predictor variance also decreases in conditional expectation, E[C(μt+1)∣Dt,at]≤C(μt). Neither the realized variance nor the target MSE must decrease after every batch.
Source and target sample sizes. No, there is no such requirement: the method applies for any source and target sample sizes. Our asymptotic analysis keeps the target sample fixed while the source samples grow.
Limited source data and a target-only benchmark. In our setting, the labeled target sample is fixed, and only source data are collected: at each round, the policy decides which source to buy labels from. A target-only model is fitted to the target labels alone, so it uses none of the collected source data, and its error stays constant as collection proceeds. It is therefore a fixed reference rather than a competing collection policy. Our experiments show how the method compares with it under limited data. Appendix G.7 (Figures 11–12) tracks allocation and prediction from initialization through 100 rounds: target-only BLR fitted to the 100 target labels has a test MSE of 2.728 at every round, while one-step lookahead falls below it after the first purchase and reaches about 2.52 at round 100 (2.54 for BNN, compared with 2.645 for target-only). Appendix G.10 (Figure 18) starts with only 10 labels per source. A target-only model can do better when the sources are less informative about the target: under strong covariate shift (Appendix G.9), and when the target lies outside the class of the source mixture (Appendix G.10, where target-only least squares gives 2.81, compared with 2.87 for one-step lookahead).
We thank the reviewer for these helpful references. Section 2 now discusses Han et al. (2025), Federated Adaptive Causal Estimation (FACE) of Target Treatment Effects, and Liu et al. (2026), Privacy-Protected Causal Survival Analysis Under Distribution Shift. FACE combines density-ratio adjustment, adaptive source weighting, and the communication of summary statistics. It relies on transportability for an informative subset of sources, rather than conditional homogeneity of every source. Liu et al. study federated causal survival effects at a target site under their identification and exchangeability assumptions.
These methods adapt how information is borrowed from heterogeneous sites for causal estimation. Our question is complementary: which new source labels to buy under a collection budget for target prediction. The revised manuscript makes this distinction and discusses distributed summaries as an extension. We do not claim that our allocation theorem provides causal identification or privacy guarantees.
The formulation permits centralized access to source data and imposes no privacy or communication constraint; this is now explicit in Section 5. A coordinator can retain the collected source observations and the fixed target data, update the source beliefs, and score candidate purchases. With known-noise BLR, sites can instead communicate the sufficient statistics Xk⊤Xk and Xk⊤yk rather than individual observations. Inference with unknown noise needs additional information, such as yk⊤yk, and general predictive modules may require individual-level data.
Under the regularity and sufficient-sampling conditions of the Gaussian theorem, the predictor-variance objective has leading order C(μt)∼(∑kck)2/nt under the Oracle allocation. This is a statement about variance; it is not a general meta-analytic precision law, a rate for target MSE, or a privacy guarantee for arbitrary shared summaries.