- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether two interventions induce the same outcome distribution when outcomes are structured and high-dimensional, such as images, text, and graphs. The proposed method, DR-KTE-Deep, combines doubly robust estimation of interventional mean embeddings with deep kernel learning. The kernel is trained to maximize an estimated signal-to-noise criterion related to test power. A characteristic kernel component is retained so that the final kernel remains sensitive to general distributional differences.
The paper provides a doubly robust representation of the distributional treatment effect, asymptotic validity of a held-out test after kernel learning and nuisance estimation, and uniform finite-sample control of the kernel-learning criterion. Experiments on image, text, and graph outcomes generally show higher power than fixed Gaussian and bandwidth-optimized DR-KTE baselines while maintaining Type-I error near the nominal level.
Critical strengths
-
The paper addresses an important problem. Many modern causal outcomes are images, text, or graphs, for which a scalar average treatment effect can miss important changes in the outcome distribution. The paper gives a general way to test changes in the full marginal interventional distribution without first reducing the outcome to a single scalar feature.
-
The theoretical construction is careful. The paper separates kernel learning, validation, and final testing, which allows the final test to remain valid after data-dependent kernel selection. The characteristic component also preserves the omnibus interpretation of the test. The experiments cover several different types of structured outcomes and show large power improvements in several settings.
Critical weaknesses
-
I am not yet convinced that the experiments establish the unique value of the proposed deep-kernel learning procedure relative to strong current alternatives. The main direct comparisons are DR-KTE with a fixed Gaussian kernel and DR-KTE-Opt with an optimized Gaussian bandwidth. These are natural predecessors, but they can be weak representations for images, text, and graphs. This is especially visible in the text ablation: once the text is represented with a strong fixed embedding, all three methods have almost identical power. The scalar experiments also contain settings where the optimized Gaussian kernel performs as well as or slightly better than the deep kernel. Therefore, part of the reported gain may come from replacing a weak fixed representation rather than from the proposed causal power-learning objective itself. The paper itself cites recent methods that learn causal outcome representations or informative witness locations, but does not compare against them empirically. These methods do not all test exactly the same omnibus null, so a direct comparison may require adaptation. Still, a stronger comparison or a clearer explanation of why these approaches are not comparable would make the contribution easier to assess. Durand et al. is a UAI 2025 paper on learning causal response representations, while related approaches from Wibisono and Wang and Zenati and Gretton appeared in 2026.
-
There remains a gap between the theoretical robustness claims and the empirical validation. The structured-outcome experiments are synthetic or semi-synthetic, which is useful for measuring power but gives limited evidence about performance with realistic nuisance estimation, weak overlap, or complex observational confounding. The robustness experiment studies a misspecified outcome model while assuming the logging policy is known. It does not show the complementary case of a misspecified propensity model with a well-specified outcome model. This seems important for a method whose causal construction is based on double robustness.
The dimension-scaling experiment also shows that power eventually collapses and Type-I error rises above the nominal level for all methods as irrelevant outcome dimensions are added. I would like a clearer discussion of what this implies for genuinely high-dimensional applications.
Questions for authors
-
How much of the empirical gain comes from the proposed power-learning objective rather than simply learning a better outcome representation? Could the authors compare DR-KTE against stronger fixed representations for each modality, such as pretrained image or text features? The text-embedding ablation is particularly important because the advantage of DR-KTE-Deep almost disappears there.
-
Can the authors provide the complementary double-robustness experiment? Section D.5 studies outcome-model misspecification while using the true logging policy. What happens when the propensity model is misspecified but the conditional outcome model is well estimated? I would also be interested in the case where both nuisance models are imperfect.
-
How realistic are the nuisance-rate assumptions for the estimators used in the experiments? The appendix states that the required learner rates are assumed rather than proved. Can the authors provide empirical evidence on calibration under weaker overlap or harder nuisance estimation, or explain when these conditions should be expected to hold in practice?
5 comments
Additional baselines
We add DR-ME (Zenati and Gretton, 2026) and adapt DEA (Durand et al., 2025) and MCF (Wibisono and Wang, 2026). Neither provides a drop-in test of our marginal interventional null: DEA learns outcome projections using a conditional-dependence regression criterion, which can miss mean-preserving shifts, whereas MCF learns a scalar causal-contrast feature without our variance normalisation. To compare them under the same null and testing protocol, we evaluate their learned features in our doubly robust held-out cross-U test, yielding DR-DEA and DR-MCF. These are testing adaptations, not the original procedures.
| / | DR-KTE | DR-ME | DR-MCF | DR-DEA | DR-KTE-Deep |
|---|---|---|---|---|---|
| OCT-Drusen | 0.051 0.003 / 0.058 0.004 | 0.044 0.003 / 0.045 0.004 | 0.318 0.049 / 0.053 0.003 | 0.053 0.010 / 0.059 0.008 | 0.371 0.080 / 0.058 0.005 |
| Derma-Melanoma | 0.171 0.009 / 0.051 0.002 | 0.400 0.029 / 0.045 0.004 | 0.795 0.041 / 0.049 0.004 | 0.048 0.004 / 0.053 0.004 | 0.898 0.040 / 0.053 0.004 |
| Chest-Cardiomegaly | 0.156 0.007 / 0.050 0.003 | 0.178 0.009 / 0.046 0.002 | 0.368 0.048 / 0.048 0.004 | 0.043 0.004 / 0.044 0.005 | 0.928 0.046 / 0.051 0.004 |
DR-KTE-Deep has higher estimated power than DR-MCF on all three scenarios. DR-DEA remains near nominal rejection under the alternatives, consistent with the limited regression signal in our mean-preserving image constructions. Our criterion instead targets the studentised distributional discrepancy (Proposition 4.1).
Robustness to misspecification
We test complementary nuisance misspecification at (500 learning, 500 validation and 500 test observations). The misspecified propensity omits nonlinear treatment-assignment features; the misspecified CME uses a linear input kernel instead of features capturing the nonlinear outcome mechanism:
| / | Both well-specified | Misspecified | Misspecified | Both misspecified |
|---|---|---|---|---|
| Causal-Blob | 0.267 0.033 / 0.058 0.011 | 0.211 0.030 / 0.055 0.009 | 0.599 0.053 / 0.292 0.059 | 0.935 0.029 / 0.735 0.054 |
| Scenario II | 0.860 0.032 / 0.046 0.004 | 0.510 0.043 / 0.045 0.006 | 0.980 0.008 / 0.155 0.039 | 0.995 0.003 / 0.865 0.033 |
| Scenario III | 0.268 0.024 / 0.046 0.004 | 0.192 0.015 / 0.045 0.006 | 0.702 0.023 / 0.155 0.039 | 0.833 0.022 / 0.865 0.033 |
| Scenario IV | 0.893 0.017 / 0.046 0.004 | 0.782 0.025 / 0.045 0.006 | 0.994 0.002 / 0.155 0.039 | 0.993 0.002 / 0.865 0.033 |
| Fake-MNIST | 0.330 0.044 / 0.048 0.005 | 0.309 0.043 / 0.055 0.004 | 0.604 0.054 / 0.146 0.019 | 0.812 0.029 / 0.454 0.016 |
With a richer propensity model, misspecifying the CME leaves Type-I error near nominal. A richer CME substantially reduces the inflation from propensity misspecification, but does not fully restore calibration at this sample size; misspecifying both nuisances causes much greater inflation.
Proposition 3.3 gives double robustness of the population contrast, not automatically of test calibration. Theorem 4.3 requires sufficiently small null bias, ensured by a nuisance-error product ; this need not hold when propensity misspecification persists and the CME is estimated nonparametrically (Benkeser et al., 2017; Dukes et al., 2024).
To test whether a better outcome model reduces this inflation, we keep the misspecified propensity, learned kernel and test set fixed, and refit the CME on and observations (dividing its ridge penalty by 2 and 4):
| / | |||
|---|---|---|---|
| Causal-Blob | 0.599 0.053 / 0.292 0.059 | 0.602 0.027 / 0.094 0.010 | 0.575 0.038 / 0.047 0.004 |
| Scenario III | 0.702 0.023 / 0.155 0.039 | 0.744 0.030 / 0.071 0.005 | 0.752 0.026 / 0.066 0.005 |
Improving the CME markedly reduces Type-I inflation: calibration becomes close to nominal on Causal-Blob, though some inflation remains in Scenario III, with broadly similar estimated power.
With an oracle propensity, fitting the CME also improves Scenario III power (0.616 versus 0.349; Appendix D.5), with similar Type-I error.
Nuisance-rate assumptions
For , a correctly specified root- propensity estimator and any consistent CME suffice for the product-rate condition of Theorem 4.3 (Chernozhukov et al., 2018). Our experiments fall within this favorable regime: treatment assignment follows the logistic family used for propensity estimation, which admits a parametric rate under standard regularity conditions. The remaining requirement is CME consistency, for which KRR has established sufficient conditions. More generally, two nonparametric estimators with root-mean-square rates suffice when . Hilbert-valued KRR can attain such rates under suitable smoothness and regularisation conditions, without an exponent depending directly on outcome dimension (Li et al., 2024).
We stress-test overlap by scaling the treatment-assignment coefficients and widening the true propensity clipping bounds (, with 500 learning, 500 validation and 500 test observations):
| Propensity bounds | Propensities at the bounds | Propensity error | Effective sample size | / | |
|---|---|---|---|---|---|
| Causal-Blob, | 4% | 0.024 | 59% | 0.382 0.024 / 0.050 0.004 | |
| Causal-Blob, | 21% | 0.030 | 30% | 0.130 0.012 / 0.044 0.004 | |
| Causal-Blob, | 36% | 0.044 | 16% | 0.072 0.007 / 0.048 0.005 | |
| Causal-Blob, | 8% | 0.022 | 25% | 0.259 0.022 / 0.122 0.014 | |
| Causal-Blob, | 8% | 0.022 | 23% | 0.467 0.018 / 0.238 0.033 | |
| OCT-Drusen, | 7% | 0.034 | 50% | 0.371 0.080 / 0.058 0.005 | |
| OCT-Drusen, | 22% | 0.040 | 20% | 0.198 0.048 / 0.042 0.003 | |
| OCT-Drusen, | 36% | 0.052 | 10% | 0.045 0.005 / 0.034 0.003 |
With bounds , weaker overlap reduces effective sample size and power but leaves null rejection near nominal. Widening the bounds produces finite-sample Type-I inflation, which Theorem 4.3 does not rule out.
Dimension scaling experiment
Appendix D.11 adds independent noise coordinates while keeping the treatment-related difference in the first two dimensions. All three kernel tests lose power, but DR-KTE-Deep retains an advantage through .
We traced the previously reported Type-I inflation to an implementation error in this experiment. After correction, all three displayed kernel tests have null rejection rates near 5% across dimensions.
| / | DR-KTE | DR-KTE-Opt | DR-KTE-Deep |
|---|---|---|---|
| 0.849 0.013 / 0.046 0.003 | 0.910 0.011 / 0.048 0.003 | 0.973 0.009 / 0.043 0.004 | |
| 0.528 0.019 / 0.048 0.004 | 0.731 0.019 / 0.052 0.004 | 0.935 0.025 / 0.050 0.003 | |
| 0.292 0.012 / 0.053 0.004 | 0.536 0.022 / 0.048 0.004 | 0.920 0.028 / 0.042 0.003 | |
| 0.091 0.005 / 0.051 0.004 | 0.155 0.016 / 0.048 0.003 | 0.507 0.073 / 0.050 0.004 | |
| 0.058 0.004 / 0.049 0.004 | 0.059 0.004 / 0.054 0.004 | 0.119 0.021 / 0.052 0.004 | |
| 0.050 0.004 / 0.047 0.003 | 0.054 0.004 / 0.050 0.004 | 0.050 0.003 / 0.051 0.004 |
We will update Appendix D.11 accordingly.
We thank you for your thoughtful feedback. Below, we provide responses to your concerns and would be happy to offer further clarification if needed. Should you find our responses satisfactory, we would be grateful if you would consider increasing your score.
On representation learning
We provide additional experiments to assess the contribution of our power-learning criterion beyond fixed and unsupervised outcome representations.
Image outcomes
Appendix D.6 isolates the training objective: on OCT-Drusen, with the same encoder architecture, initial weights and data, DR-KTE-Deep achieves power 0.371 ± 0.080, versus 0.057 ± 0.004 (random features) and 0.052 ± 0.004 (pixel autoencoder).
We also test stronger fixed representations. We embed each image using a frozen ResNet-18 pretrained on ImageNet, then apply DR-KTE, DR-KTE-Opt and DR-KTE-Deep to the same embeddings.
Compared with Table 1 of the submission, pretrained ResNet features improve the estimated power of both Gaussian-kernel baselines over raw pixels in all ten scenarios. Yet on the same pretrained representation, DR-KTE-Deep has higher estimated power in every scenario, with similar Type-I error. Thus, its gains are not solely due to replacing raw pixels with informative pretrained features.
Text outcomes
In Appendix D.7, pretrained sentence embeddings already make the original sentiment and clinical-outcome shifts easy to detect, leaving little room for kernel learning. We therefore introduce Clinical Notes (Style), where treatment shifts writing toward telegraphic or longer, flowing prose rather than ordinary prose, while prompting the generator to preserve clinical content. This is a less directly encoded difference for semantic embeddings.
On this style-shift scenario, the fixed-kernel tests have substantially lower estimated power than in the original. This supports the value of learning the kernel when the treatment-related difference is less well represented by the pretrained embedding. Moreover, DR-KTE-Deep trained directly on raw tokens achieves power comparable to DR-KTE on pretrained embeddings in Oncology Notes (0.809 ± 0.023 versus 0.828 ± 0.035) and Clinical Notes (0.939 ± 0.013 versus 0.894 ± 0.027).
Scalar outcomes
On scalar outcomes, which are not the focus of the method, DR-KTE-Opt uses the same power-learning criterion as DR-KTE-Deep but optimises only the Gaussian bandwidth. Its advantage in Scenario III is about one standard error (0.653 ± 0.027 against 0.614 ± 0.028).