- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies conditional independence testing with limited target data and additional data from different source domains. It proposes DA-Diff, which trains a conditional diffusion model by weighting target and source data. The analysis separates estimation error from the bias caused by differences between domains. The learned model is then used in a conditional randomization test, called DA-CIT. Experiments cover synthetic data and the Sachs dataset.
Critical strengths
-
The paper addresses a useful problem. Extra data can improve conditional distribution estimation, but differences between domains can also make the resulting test less reliable.
-
The theory describes the trade-off between using more source data and introducing more bias. This is a useful way to understand multi-source diffusion models.
-
The experimental results are promising. DA-Diff improves over target-only training in Table 1. DA-CIT generally keeps the observed Type I error near the nominal level while maintaining competitive power. The paper also includes several baselines and sensitivity studies.
Critical weaknesses
-
It is unclear whether the theory covers the actual training procedure. Theorems 1–3 analyze an estimator with a given weight vector. In practice, the method repeatedly estimates source differences, selects weights using validation data, and updates the model. It also uses different weights at different diffusion times. Appendix K.8 analyzes the estimated discrepancy matrix, but assumes that the validation data are independent of the fitted models. In the actual procedure, validation data affect the weights, which then affect later models. I could not find a proof that handles this dependence. Showing that the discrepancy estimate is consistent does not, by itself, establish the stated guarantee for the full algorithm.
-
The conditions for asymptotic Type I error control are strong and differ from the experimental settings. Equation (32) requires source domains to become increasingly similar to the target as the sample size grows. It also requires the effective sample size to grow faster than the square of the target sample size. The main experiments instead use ten source domains, similar sample sizes across domains, and fixed shift strengths. Scaling this setup does not satisfy Equation (32). Some nonlinear transformations in Appendix D.2 also produce distributions outside Assumption 2. These are sufficient conditions, so failing to meet them does not mean the method fails. However, the current theory does not directly explain why the method works in the experiments. The paper should make this limitation clearer.
-
The testing experiments do not clearly show how much the proposed weighting helps. A useful comparison would use pooled or finetuned diffusion models inside the same CRT, with the same source data, target split, and test statistic. This would show whether the proposed weights improve testing beyond simpler ways of using extra data. The Cat-all experiments combine domains before testing, so they do not provide this controlled comparison. Also, each setting uses only 100 repetitions. At a true rejection rate of 0.05, the standard error is about 0.022. The results are encouraging, but they cannot precisely distinguish good calibration from a small increase in Type I error.
Questions for authors
-
Does the theory cover the full algorithm, including repeated use of validation data and weights that vary across diffusion time? If not, please state which version of the method is covered.
-
What guarantee remains when source–target differences stay fixed and source sample sizes grow at the same rate as the target sample size?
-
How does DA-CIT compare with the same target-domain CRT using pooled or finetuned diffusion models? Please also report uncertainty intervals for the estimated Type I error.