- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how to estimate Tian–Pearl bounds on PNS when subpopulations have insufficient experimental and observational data. The authors use the existing formulas to compute bounds for well-supported subpopulations, then train regression models to predict bounds for other subpopulations. Mask augmentation extends the query representation to include subpopulations with unspecified covariate values. Evaluations on synthetic data and the Hillstrom randomized email experiment show that the framework improves bound prediction accuracy.
Critical strengths
1. The paper addresses a clear and practically relevant problem: fine-grained subpopulations often have too few samples to reliably estimate bounds on PNS.
2. The mask-augmentation strategy is straightforward and easy to implement.
3. The experiments are reasonably comprehensive, covering five regressors, four SCMs, and multiple data budgets.
Critical weaknesses
1. The experiments lack comparisons with recent relevant baselines.
2. The framework mainly uses the existing Tian–Pearl formulas to generate training labels and regression models to predict PNS bounds. The main addition is mask augmentation, but its methodological novelty relative to existing approaches remains unclear.
3. The paper aims to address data scarcity, but the Hillstrom test queries all contain relatively large numbers of customers and therefore mainly evaluate well-supported subpopulations. The authors acknowledge this limitation, but it still limits the evidence supporting their claims about data-scarce subpopulations.
4. Algorithm 1 reports errors over the full query sets without explicitly stating whether training and validation queries are excluded from evaluation.
Questions for authors
1. Why should patterns learned from well-supported subpopulations generalize to data-scarce subpopulations?
2. How are queries split into training, validation, and test sets in the synthetic experiments? Please clarify this in the experimental setup.
3. Please include additional relevant baselines. If they are not directly comparable, please explain the differences in assumptions or estimation targets.
4. Please provide the appendix referenced in the paper, including the SCM equations, model configurations, and other implementation details.
Thank you very much for your thoughtful review and conductive advices for our paper. We appreciate the opportunity to clarify several points raised in your comments.
Regarding your specific questions and concerns:
W1&Q3 We also tried to find a suitable baseline. However, as discussed in our Related Work, existing PNS research mainly focuses on theoretically deriving tighter bounds, usually assuming access to oracle distributions, or on using PNS to support machine learning. Closely related work uses ML to estimate conditional causal effects such as ATE. However, ATE and PNS belong to different levels of Pearl’s causal hierarchy, so their estimates cannot be directly compared, as also discussed in our Related Work. We have not found a directly applicable method for estimating PNS bounds for subgroups with no data. Therefore, we use plug-in estimation and other ML models as baselines.
W2 We agree that the ML models themselves are standard. Our primary goal is to investigate the learnability of PNS bounds, a counterfactual defined in Structural Causal Models that is rarely served as a target function in ML. Our main contributions are therefore the formulation of the ML pipeline for learning PNS bounds and the proposed mask augmentation.
W3 We agree that this is a limitation, as we mentioned in the paper. The Hillstrom experiment provides an initial real-world evaluation but still suffers from selection bias. To address this issue, as mentioned in the conclusion, an external audit could be conducted, although it would be expensive. Conducting such an audit is beyond the scope of the current study and remains an important direction for future work.
W4 Thank you for the question about training and validation queries exclusion. We keep all queries for evaluation because the labels for evaluation (oracle) and training are different. Therefore, we still need to check whether the model can make accurate predictions on those seen queries. In addition, even if we exclude all seen queries, the MAE increases by about 15%, as shown in the table below, while our conclusion remains unchanged.
Q1 At a high-level, individuals within the same SCM share the same structural equations. These shared mechanisms can introduce learnable structure into the mapping from covariate configurations to PNS bound endpoints. By learning this mapping from well-supported subpopulations, a neural network may also predict bounds for data-scarce subpopulations.
Q2&Q4 Thank you for raising this point. We apologize that we did not make this clear enough in the main paper. The appendix is included in the supplementary material, and Section B provides detailed information on the train-validation-test split, SCM equations, and model configurations.