acceptodds
Under review as a conference paper at ICLR 2027

Validating Causal Benchmarking in Clinical Registries: Antimicrobial Timing in Septic Shock

Abstract

Generative causal benchmarks offer a way to compare treatment-effect estimators when the true effects in observational data are unknown. Their appeal is to combine known, model-defined effects with data that resemble the original study. However, a benchmark can pass verification simply because the tests cannot detect its errors. We investigate this problem through controlled experiments and the Cooperative Antimicrobial Therapy of Septic Shock (CATSS) registry of 11,187 patients from 32 hospitals, where we study antimicrobial timing and mortality. Septic shock is a time-critical condition with substantial mortality, and effective antimicrobial therapy is often initiated under urgent clinical uncertainty. This makes treatment timing a consequential setting in which reliable causal evaluation is particularly important. We find that outcome type sharply changes how added covariates affect verification power. Under severe generator error at , adding just five covariates reduces the energy test's detection rate from to for binary outcomes. In the corresponding continuous-outcome setting, detection remains through 50 covariates. A similar failure appears in CATSS: energy and maximum mean discrepancy detect a deliberately broken generator using treatment and outcome alone, but miss it once patient covariates are included. A classifier two-sample test detects the failure in both representations. On benchmarks generated by two model families, estimator comparisons further reveal robustness differences obscured by full-model evaluations: the doubly robust estimators remain comparatively stable when either the treatment or outcome model alone is restricted. These comparisons inform our clinical analysis, where effective antimicrobial therapy within one hour of shock onset is associated with a 14.4-percentage-point lower adjusted mortality risk. These findings show that apparent agreement between generated and observed data is not enough. Before using a benchmark to judge causal estimators, we should establish which generator failures its verification procedure can actually detect.

Primary area:
datasets and benchmarks
Keywords:
Causal Inference, Causal Benchmarking, Benchmark Verification, Two-Sample Testing, Treatment Effect Estimation, Doubly Robust Estimation, Tabular Foundation Models, Septic Shock

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kRPTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks into challenges when evaluating using benchmarks for causal effect estimation. A popular approach is to fit a generator to real-world covariates that can then be used to generate more realistic synthetic data. However, the usefulness of this approach as a real-world data proxy depends on how well the generated data approximates a real-world distribution, and some methods for assessing this match can produce different results. These differences can have downstream effects on causal estimator accuracy. Focusing on a single dataset (CATSS), the authors study these issues. They fit 8 generators to the set of 49 covariates and compare estimator performance for 6 estimators. They find significant variability in verification results depending on experimental settings (such as continuous vs binary outcome).

Critical strengths

The problem the authors tackle here is nuanced and interesting, and not one I've heard discussed much. While it's logical to use some method to ensure generated data matches the source dataset, an exploration of how results can differ depending on what verification method is used is novel and has the potential to be very practically valuable. In general, I'm a big fan of evaluation methodology papers, and I think the authors have found a promising direction to explore.

Critical weaknesses

Unfortunately, the focus of this paper is too narrow to be a substantial enough contribution. A general exploration across multiple datasets, with different features, of the effect of verification method could produce some interesting conclusions that could have practical value to practitioners. This paper, though, analyzes a single data set. Because of this, the results are only able to go as far as saying "there are substantial differences, so be careful", without any additional guidance on how a generator should be chosen in practice.

This paper also is missing some key details, making it hard to properly assess. The biggest issue is that I can't find what the authors actually did to introduce generator errors. The core experimental results focus on verification power under severe generator errors and under milder errors. However, all I can find in the paper are references to adding generator errors, not what the procedure for adding them actually is. On page 5, you mention "generator-error severity" as one of the dimensions of the stress test, with levels of "none", "mild", and "severe". In the previous paragraph (lines 229-239), you talk about creating verification controls, which I at first thought was the generator errors. However, the method listed there is just to remove the dependence of treatment and outcome on covariates, which is a binary (with or without dependence), not something that can scale from none to mild to severe. If the methodology for inducing generator errors is explained in the paper, it needs to be called out much more explicitly.

Ultimately, this paper is an evaluation paper. It's not proposing a new methodology or a new data set - rather, it's highlighting an often-overlooked element of causal evaluation and showing important adjustments to existing data generation and evaluation procedures. Since the main contribution, then, is raising awareness of potential evaluation pitfalls, the demonstration of those pitfalls, and narrative about how to avoid them, needs to be strong. Unfortunately, in this paper, that demonstration amounts a couple of experimental results on a single data set with minimal generalization.

Questions for authors

What did you do to introduce generator errors?

1 comment

We thank the reviewer for the comments. We agree that the original paper did not explain the generator-error mechanisms clearly and that evaluating only CATSS limited the conclusions.

We have revised the experimental design to address these concerns.

  1. How did we introduce errors into the generator?

In the revised experiments, we have introduced two different types of generator error.

In the first method, we mix the patient’s treatment-outcome distribution with the average distribution obtained across patients. This reduces the dependence of treatment and outcome on patient characteristics.

In the second method, we retain the original treatment model but change the outcome distribution within each treatment group.

Both methods allow us to introduce errors while preserving the population-level marginal treatment-outcome distribution.

We measure the errors using total-variation distance rather than simply classifying them as mild or severe. We have tested multiple error levels, ranging from 0 to 0.20, where achievable.

The equations and steps used to generate these errors are explained in Section 3 and Appendix D.

  1. How have we addressed the limitation of using only one dataset?

We have expanded the study by adding the NHEFS dataset and simulations with known treatment effects.

The revised evaluation now includes 11,187 patients from CATSS, 1,566 participants from NHEFS, and controlled simulations with 10 covariates and binary or continuous outcomes.

We have used these datasets to examine how verification performance changes with the type of outcome and the amount of generator error.

For example, at an error level of 0.10 in CATSS, C2ST detected 61% of errors, compared with 26% for MMD. In NHEFS, for the binary death outcome, C2ST detected 86–99% of errors, while MMD detected 24–29%, depending on the error mechanism.

For continuous weight change, the detection rates were lower.

The NHEFS experiments use fitted models as reference distributions, not the unknown true causal effects in the original dataset. We have explained this limitation in the paper.

  1. What guidance do the results provide for selecting a causal benchmark?

We have also found that a verification result alone is not enough to determine whether a benchmark is suitable for causal estimation.

Our experiments suggest that we should first check whether the statistical test is correctly calibrated. We should then examine whether the test can detect controlled errors and how much error can remain undetected.

We should also examine whether these errors change the treatment effect or the ranking of causal estimators.

For example, we observed that some errors changed the benchmark treatment effect substantially while estimator rankings remained relatively stable. This means that the reliability of a benchmark also depends on what we want to evaluate.

We do not conclude that C2ST is always better than MMD or that one generator should be used in every setting. The results depend on the outcome, type of error, and data representation.

  1. Changes made to the paper

Further, we have also added detailed definitions of generator errors, then we have also tested additional data settings, and included confidence intervals and detection rates.

We have also reorganized the paper to focus on verification process and therefore we have moved the earlier clinical and estimator experiments to the appendix.

These changes address the main concerns about the experiments and help explain when verification methods can be useful for causal benchmarking.

Reviewer_qzieOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses verification of semi-synthetic data (with real-world covariates, but synthetic treatment and outcome models) for benchmarking treatment effect estimators. The authors ask: when can such verification be trusted, as we vary the number of covariates, number of samples and the outcome type (discrete/continuous)? The authors create known generator failures as positive controls, and sample two real datasets as a null control. A good verification method should flag all generator failures, while failing to distinguish the null controls. The authors find that the outcome type changes whether the number of covariates affects the power of energy-based verification in controlled experiments. On CATSS data, energy/MMD fail to detect broken generators as covariates are added. Further, the authors study how deliberately misspecifying one or both nuisance models of treatment effect estimators increases their RMSE on a selected benchmark. Finally, the authors apply two of the estimators (AIPW and TMLE) to estimate the treatment effect of antimicrobial therapy at different windows for septic shock.

Critical strengths

The paper attempts to address an interesting question which is interesting and highly relevant in practice: if we use semi-synthetic data to benchmark causal estimators, when can we trust it? When can we trust a verification of a generator of such data? Can we design a set of controlled experiments which finally tell us if we can trust a verification, and thus the generator, and thus its benchmarking results?

Critical weaknesses

I do not think that the paper has substantial novelty. In its controlled experiments, it discovers some failures of discrepancy-based verification, but I find the analysis to be not systematic enough. The obtained results w.r.t. discrete vs. continuous outcomes and energy vs C2ST could be artifacts of testing on paired data with the same covariates. The paper further illustrates known robustness of estimators (here, I am even less convinced that this is a contribution: Double robustness or the absence of such is known for all listed treatment effect estimation methods, and has been observed empirically many times. It also follows directly from the data-generating process that the authors impose on the generated semi-synthetic data. )

To me, a much more interesting failure of semi-synthetic data generators for causal purposes is that no matter how complex the generator is, it imposes the same fairly simple data-generating process from a causal perspective, where ignorability based on the covariates is satisfied. This is almost never true in real data, and is also where treatment effect estimation fails. It would be more interesting and more novel to look at verifications of generators that differ not only by used models, but also by the complexity of the imposed data-generating process. Combined with the absence of theory, I do not find the contribution of the paper novel and systematic enough at this point.

There are multiple statements similar to "a large verification p-value indicates failure to detect a discrepancy, not proof of benchmark realism" in the paper. Is the paper's message a critique of the incorrect interpretation of p-values, which, while very valid, is definitely not novel?

Further, the writing of the paper is hard to follow, not well structured and heavy on paragraphs. It does not result in a compelling story, but rather reads as a long list of first (existing) generators, then (existing) verification methods, then (existing) causal effect estimators. All of those take a lot of space, but do not give the reader a new, interesting understanding.

Questions for authors

Could you summarize again your most important finding, and outline to which extent it is unknown/surprising in the field?

  • How do you define "generator-error severity" formally and how do you classify it into mild and severe?
  • Why is TMLE a "robustness check" for AIPW and what conclusions does this imply?
  • How is 6 (application) conceptually connected to the rest of the paper, since it seems to be ATE estimation on a real-world dataset?
2 comments

We thank the reviewer for the detailed comments. We have reviewed the concerns and revised the experiments and presentation of our contributions.

  1. Main contribution and difference from existing research

We agree that a large p-value does not prove that two distributions are the same. This is a known limitation and is not our main contribution.

Our study examines how generator errors missed by verification methods affect treatment effects and causal estimator rankings.

We introduced controlled errors, measured them using total-variation distance, and examined their effect on verification and causal estimation.

At an error level of 0.10, the benchmark treatment effect increased to 1.3–2.0 times its original value, while MMD and C2ST still missed many errors. Estimator rankings remained relatively stable in several settings despite these changes.

Our main contribution is to examine how undetected generator errors affect the reliability of causal benchmarks.

  1. Effect of paired data on verification

We found an error in our original permutation procedure, which did not preserve the pairing between real and generated observations.

We corrected the procedure and repeated the experiments. The earlier finding that verification power decreased sharply with additional covariates was no longer supported.

We have removed this conclusion and reported the corrected results in Section 4 and Appendix B. We also added controlled error levels to compare binary and continuous outcomes.

  1. Measuring generator-error severity

We now use total-variation distance to measure the difference between reference and generated conditional distributions, averaged across patients.

We tested error levels of 0, 0.02, 0.05, 0.10, 0.15, and 0.20, where supported by the error mechanism.

This replaces the earlier mild and severe categories. The methods are described in Section 3 and Appendix D.

  1. What about unmeasured confounding in semi-synthetic causal benchmarks?

We agree that this is an important limitation. A generator may reproduce the observed data but still fail to represent the true causal relationship when some confounding factors are not recorded.

To examine this, we have added an experiment where an unobserved variable affects both treatment and outcome.

We varied the strength of this confounding while keeping the true causal treatment effect fixed at 0.798. However, the benchmark-defined treatment effect increased from 0.798 to 2.428.

During these experiments, we found that both verification methods continued to give rejection rates close to the expected 5% level because the observed distributions were matched.

We also evaluated AIPW using stronger nuisance models. At the highest confounding level, the average estimate was 2.401, which was close to the benchmark effect of 2.428 but different from the true causal effect of 0.798.

These results show that even when we improve the estimator, it cannot recover the causal effect if the required confounding information is missing. We have presented this analysis in Section 6.

  1. TMLE as a robustness check

We originally used TMLE to compare results with AIPW because both methods are doubly robust.

However, agreement between these methods does not establish that the benchmark is correct.

As these experiments mainly confirm known estimator properties, we moved them to Appendix J and removed them from the main contributions.

  1. How is the clinical application connected to the main research problem?

The earlier clinical analysis studied the relationship between timing of antimicrobial treatment and mortality in septic shock.

However, the revised paper focuses on the reliability of causal benchmarking rather than estimating the clinical effect of antimicrobial timing on the outcome of the patient.

Therefore, we have moved the clinical analysis to Appendix J. We also make clear that these results are observational associations and may be affected by unmeasured confounding and treatment-timing bias.

  1. Changes to the organization of the paper

We have reorganized the paper to make the research question and experiments easier to follow.

The revised study first examines the corrected verification procedure, after that it introduces controlled errors, then measures their detection rates, and finally examines how these errors affect treatment effects and estimator rankings.

We have also included the unmeasured-confounding experiment to show the limitations of verification when the observed data do not contain all the information required for causal inference.

We recognize that the experiments do not provide a complete theoretical explanation of verification power. Our results are limited to the tested datasets, error mechanisms, and verification methods. However, they provide a more systematic evaluation of the errors that verification can miss and their consequences for causal estimation.

I thank the authors for their transparent response. Considering the bugs in the initial manuscript, and the very substantial changes in the revised version (which would require a new review process), I choose to maintain my score.

Reviewer_VRrpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the reliability of statistical verification procedures for generative causal benchmarks and their implications for causal estimator selection. Using the CATSS registry of 11,187 patients, the authors evaluate eight generator families and five two-sample testing procedures under different data representations. Controlled experiments reveal a substantial loss of verification power for binary outcomes when covariates are added, while continuous-outcome errors remain detectable; C2ST also detects a deliberately misspecified generator that several distance- and kernel-based tests fail to identify. The paper further benchmarks six causal estimators under nuisance-model restrictions using TARNet and TabPFN-3, motivating the use of AIPW and TMLE in a clinical antimicrobial-timing analysis.

Critical strengths
  1. The paper addresses a fundamental limitation of generative causal benchmarking by demonstrating that failure to reject a two-sample test does not necessarily establish benchmark reliability.

  2. The combination of controlled stress tests, eight generator families, five verification procedures, and alternative data representations provides a useful empirical investigation of verification failures.

Critical weaknesses
  1. Shared covariates may cause overly conservative permutation tests, potentially explaining the reported power loss.

  2. The estimator experiments mainly reproduce known double-robustness patterns without establishing the benefit of verification.

  3. The broken generator changes the marginal treatment-outcome distribution, leaving conditional-error detection untested.

Questions for authors
  1. Were the verification tests calibrated using pair-preserving permutations?

  2. Does the binary-versus-continuous contrast persist under matched error strengths and feature scales?

  3. Can C2ST detect conditional errors that preserve the marginal treatment-outcome distribution?

  4. Does verification quality meaningfully affect causal estimator rankings?

1 comment

We thank the reviewer for the comments and for identifying an important issue in our verification experiments. We have corrected the problem and performed additional experiments to address the questions.

  1. Were the verification tests calibrated using pair-preserving permutations?

In our original experiments, we used pooled permutations, where the labels of real and generated records were randomly exchanged across all observations. However, both records shared the same patient covariates, and this pairing was not preserved during permutation.

We found that this produced an incorrect null distribution and made several tests overly conservative. We have now corrected the procedure by exchanging treatment and outcome values only within each patient pair.

After making this correction, the rejection rates under the null were much closer to the expected 5% level. We also observed that some of our original conclusions changed. For example, we no longer find that adding more covariates generally causes verification power to collapse.

We have explained the corrected procedure in Section 3 and provided the validity conditions and additional results in Appendix B.

  1. Does the difference between binary and continuous outcomes remain when error severity and feature scales are considered?

In our earlier experiments, we classified generator errors as mild or severe. However, these settings did not represent the same amount of error for binary and continuous outcomes.

To address this, we have used total-variation distance to measure the difference between the reference and generated distributions. This allows us to compare detection rates at the same error level.

At an error level of 0.15, C2ST identified 98–99% of errors for binary outcomes, compared with 71–72% for continuous outcomes. After correcting the experiments, we found that errors in binary outcomes were actually easier to detect than those in continuous outcomes. This was different from our earlier findings.

We also examined the effect of feature scaling. In one of our experiments, rescaling the covariates reduced MMD detection power from 100% to 13%. This shows that the representation of the data can also affect verification results. These findings are discussed in Section 4 and Appendix I.

  1. Can C2ST detect conditional errors while preserving the marginal treatment-outcome distribution?

We have introduced two new error mechanisms for this purpose.

The first mechanism mixes the patient’s original treatment-outcome distribution with the average distribution across patients. The second mechanism changes the outcome distribution within each treatment group while retaining the treatment model.

Both mechanisms preserve the population-level marginal treatment-outcome distribution while changing the relationship with patient characteristics.

In the CATSS experiments, at an error level of 0.10, C2ST detected 61% of the introduced errors, compared with 26% for MMD. At an error level of 0.15, these values increased to 96% and 89%, respectively.

These results show that C2ST can detect conditional errors in the settings we tested. However, we also found that both methods can miss errors, particularly when the error level is moderate.

  1. Does verification quality affect causal estimator rankings?

We have added a separate experiment to study this question. We introduced controlled errors into the benchmark and examined their effect on the average treatment effect and the ranking of nine causal estimators.

At an error level of 0.10, the benchmark treatment effect had already changed to 1.3–2.0 times its original value in the tested settings. However, several of these errors were still not detected by MMD or C2ST.

We found that the rankings of the estimators remained mostly the same in several experiments, even when the benchmark treatment effect changed considerably. However, when the error level increased, the rankings also changed in some experiments.

These results show that generator errors can affect the benchmark in different ways. Therefore, in case we want to measure how accurately an estimator estimates the treatment effect, we need to check whether the benchmark treatment effect is correct. If we want to compare different estimators, we also need to check whether their rankings change because of these errors.

We have added these experiments to Section 5 and Figure 2.

We have also moved the earlier double-robustness experiments to the appendix, as they mainly confirm already known properties of the estimators and do not directly show how reliable the verification methods are.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We would like to thank all the reviewers for their comments and suggestions. We have reviewed all concerns raised and made substantial changes to the paper.

One important issue identified during this process was with the permutation test used in our original experiments. Since the real and generated records shared the same patient information, randomly changing the labels across all records was not correct. We have corrected this by performing permutations within each patient pair and repeated the experiments. Some of our earlier findings changed after this correction, and we have removed those conclusions from the revised paper.

In addition, we have performed new experiments to understand what type of errors the verification tests can detect. We now measure generator errors using total-variation distance and compare the results at different error levels. We have also included another healthcare dataset, NHEFS, along with CATSS and simulations where the true treatment effect is known.

We further analyzed that how errors that are not detected by verification can affect the treatment effect and the ranking of causal estimators. We also studied unmeasured confounding to understand whether a generator that matches the observed data can still produce an incorrect causal effect.

Based on these results, we have reorganized the paper and moved the earlier estimator robustness and clinical analyses to the appendix. The main paper now focuses on the reliability of benchmark verification and its effect on causal estimation.

We have also explained which results from the original submission were corrected or withdrawn in Appendix K.

We thank the reviewer for the comments and for identifying an important issue in our verification experiments. We have corrected the problem and performed additional experiments to address the questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.