- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper looks into challenges when evaluating using benchmarks for causal effect estimation. A popular approach is to fit a generator to real-world covariates that can then be used to generate more realistic synthetic data. However, the usefulness of this approach as a real-world data proxy depends on how well the generated data approximates a real-world distribution, and some methods for assessing this match can produce different results. These differences can have downstream effects on causal estimator accuracy. Focusing on a single dataset (CATSS), the authors study these issues. They fit 8 generators to the set of 49 covariates and compare estimator performance for 6 estimators. They find significant variability in verification results depending on experimental settings (such as continuous vs binary outcome).
Critical strengths
The problem the authors tackle here is nuanced and interesting, and not one I've heard discussed much. While it's logical to use some method to ensure generated data matches the source dataset, an exploration of how results can differ depending on what verification method is used is novel and has the potential to be very practically valuable. In general, I'm a big fan of evaluation methodology papers, and I think the authors have found a promising direction to explore.
Critical weaknesses
Unfortunately, the focus of this paper is too narrow to be a substantial enough contribution. A general exploration across multiple datasets, with different features, of the effect of verification method could produce some interesting conclusions that could have practical value to practitioners. This paper, though, analyzes a single data set. Because of this, the results are only able to go as far as saying "there are substantial differences, so be careful", without any additional guidance on how a generator should be chosen in practice.
This paper also is missing some key details, making it hard to properly assess. The biggest issue is that I can't find what the authors actually did to introduce generator errors. The core experimental results focus on verification power under severe generator errors and under milder errors. However, all I can find in the paper are references to adding generator errors, not what the procedure for adding them actually is. On page 5, you mention "generator-error severity" as one of the dimensions of the stress test, with levels of "none", "mild", and "severe". In the previous paragraph (lines 229-239), you talk about creating verification controls, which I at first thought was the generator errors. However, the method listed there is just to remove the dependence of treatment and outcome on covariates, which is a binary (with or without dependence), not something that can scale from none to mild to severe. If the methodology for inducing generator errors is explained in the paper, it needs to be called out much more explicitly.
Ultimately, this paper is an evaluation paper. It's not proposing a new methodology or a new data set - rather, it's highlighting an often-overlooked element of causal evaluation and showing important adjustments to existing data generation and evaluation procedures. Since the main contribution, then, is raising awareness of potential evaluation pitfalls, the demonstration of those pitfalls, and narrative about how to avoid them, needs to be strong. Unfortunately, in this paper, that demonstration amounts a couple of experimental results on a single data set with minimal generalization.
Questions for authors
What did you do to introduce generator errors?
We thank the reviewer for the comments. We agree that the original paper did not explain the generator-error mechanisms clearly and that evaluating only CATSS limited the conclusions.
We have revised the experimental design to address these concerns.
In the revised experiments, we have introduced two different types of generator error.
In the first method, we mix the patient’s treatment-outcome distribution with the average distribution obtained across patients. This reduces the dependence of treatment and outcome on patient characteristics.
In the second method, we retain the original treatment model but change the outcome distribution within each treatment group.
Both methods allow us to introduce errors while preserving the population-level marginal treatment-outcome distribution.
We measure the errors using total-variation distance rather than simply classifying them as mild or severe. We have tested multiple error levels, ranging from 0 to 0.20, where achievable.
The equations and steps used to generate these errors are explained in Section 3 and Appendix D.
We have expanded the study by adding the NHEFS dataset and simulations with known treatment effects.
The revised evaluation now includes 11,187 patients from CATSS, 1,566 participants from NHEFS, and controlled simulations with 10 covariates and binary or continuous outcomes.
We have used these datasets to examine how verification performance changes with the type of outcome and the amount of generator error.
For example, at an error level of 0.10 in CATSS, C2ST detected 61% of errors, compared with 26% for MMD. In NHEFS, for the binary death outcome, C2ST detected 86–99% of errors, while MMD detected 24–29%, depending on the error mechanism.
For continuous weight change, the detection rates were lower.
The NHEFS experiments use fitted models as reference distributions, not the unknown true causal effects in the original dataset. We have explained this limitation in the paper.
We have also found that a verification result alone is not enough to determine whether a benchmark is suitable for causal estimation.
Our experiments suggest that we should first check whether the statistical test is correctly calibrated. We should then examine whether the test can detect controlled errors and how much error can remain undetected.
We should also examine whether these errors change the treatment effect or the ranking of causal estimators.
For example, we observed that some errors changed the benchmark treatment effect substantially while estimator rankings remained relatively stable. This means that the reliability of a benchmark also depends on what we want to evaluate.
We do not conclude that C2ST is always better than MMD or that one generator should be used in every setting. The results depend on the outcome, type of error, and data representation.
Further, we have also added detailed definitions of generator errors, then we have also tested additional data settings, and included confidence intervals and detection rates.
We have also reorganized the paper to focus on verification process and therefore we have moved the earlier clinical and estimator experiments to the appendix.
These changes address the main concerns about the experiments and help explain when verification methods can be useful for causal benchmarking.