- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the selection of a population subgroup and a subset of treatment variables to intervene on, with the aim of maximizing the subgroup’s post-intervention success probability. The proposed SCAT method learns and prunes a classification tree using the minimum description length (MDL) criterion. Covariate conditions along a path define the subgroup, while treatment conditions define the intervention. A validation set is then used to select the candidate with the highest estimated success probability. Under an approximate partition assumption and the stated causal assumptions, the authors show that a near-optimal intervention can be obtained without recovering the causal graph among treatments. They also establish a finite-sample bound that relates action regret to prediction error.
Critical strengths
The paper is well motivated and brings subgroup selection, intervention-variable selection, and the choice of treatment values into a common framework. Its explicit treatment of partial in- terventions under causal dependence among multiple treatment variables has practical relevance. The method is straightforward, using the tree’s partition structure to extract subgroup conditions and intervention rules directly from its paths. These rules are intuitive and interpretable. The theoretical results also have value. Under the stated causal assumptions and partition approxi- mation conditions, they show that near-optimal subgroup interventions can be identified from an observational conditional response model without recovering the causal graph among treatments.
Critical weaknesses
W1. The separate contributions of the tree structure and MDL lack experimental validation The authors explain their reasons for using a tree structure and MDL, but direct experimental comparisons are missing. An ordinary predictive tree also allows subgroups and interventions to be extracted from its paths. A top-1 mixture-of-experts model with X as the gate input and Lasso penalties on parameters associated with T within each expert could also support adaptive grouping and treatment-variable selection within groups. The experiments do not include these structurally related alternatives, making it difficult to assess the respective benefits of the tree structure and MDL. Comparisons under the same action-selection and evaluation procedures would help establish their empirical contributions.
W2. The link between the optimization objective and practical intervention goals is insufficiently justified The introduction motivates the work through intervention decisions in healthcare, public policy, and related applications. Such applications often focus on the improvement an in- tervention provides relative to no intervention or an existing policy. The paper instead maximizes the absolute post-intervention success probability in the selected subgroup. This measures the outcome level after intervention, but does not directly quantify the benefit attributable to the intervention itself. This distinction is particularly relevant because the method jointly selects the subgroup and intervention. For a fixed population and reference policy, maximizing post-intervention success probability can be equivalent to maximizing improvement. This equivalence generally no longer holds when the population is also selected, because baseline success probabilities differ across subgroups. A subgroup with a high baseline success probability and only a small intervention benefit may receive a higher value than one with a lower baseline but a substantial intervention benefit. The reported high success probabilities therefore do not establish that the method identifies populations that benefit substantially from intervention. The authors should explain why this objective is appropriate for the motivating applications and report success probabilities under both the reference condition and the recommended intervention for the same selected subgroup. This would help distinguish a favorable baseline from an improvement caused by the intervention.
W3. The evaluation does not clearly separate subgroup selection from response esti- mation Each method is evaluated on its own selected subgroup and intervention using success prob- ability and its estimation error. These metrics characterize the final recommendation, but do not establish the accuracy of subgroup identification. Because the evaluation targets dif- fer across methods, a lower estimation error may also reflect the selection of a population whose response is easier to estimate, rather than stronger estimation performance. I suggest adding simulations with clearly defined ground-truth treatment-effect subgroups. The eval- uation should separately compare subgroup identification accuracy and response estimation error within the same ground-truth subgroup under identical intervention conditions. This would distinguish selection performance from estimation performance. It does not require a proof of consistent recovery of the true partition. Rather, simulations with known ground truth would provide direct evidence for the validity and interpretation of the learned groups.
W4. Practical relevance and interpretability lack concrete empirical support The introduction repeatedly emphasizes practical applications, but the experiments are lim- ited to semi-synthetic and fully synthetic data. There is no application with real treatments and real outcomes, which limits the evidence for the method’s practical value. The paper also does not present specific learned subgroup rules and intervention recommendations, leav- ing their simplicity and practical meaning unclear. The authors should show representative subgroup rules and interventions, examine whether they agree with the known mechanisms in simulations, and provide a domain-grounded example illustrating their interpretability and feasibility. Without such evidence, the claims of practical relevance and interpretability remain insufficiently supported.
Questions for authors
Why is a decision tree chosen to estimate the response function? Beyond the ease of extract- ing subgroup and intervention rules, is there theoretical or empirical evidence for its relative advantages? When the tree is deep and has many leaves, can MDL pruning and validation-set selection effectively control the number of candidates and maintain stable recommendations?
5 comments
W2. The objective: absolute success probability versus improvement
We thank the reviewer for this point. We agree that the improvement over a reference is an important objective in many applications, and that a high success probability alone does not show how much of it is due to the action. Following the suggestion, we made two additions to the paper.
- We report, for the selected subgroup, the success probability under the reference and under the recommended action (Appendix G.7).
- If the goal is to optimize the improvement, our framework still applies, but it needs a different setup: the partition-based model must describe the improvement and not the success probability. We give preliminary theoretical results for this in the new Appendix A.2.
Both probabilities for the selected subgroup. The reference is the success probability of the same subgroup under the treatments its individuals actually received:
| family | datasets | reference | recommended action | difference |
|---|---|---|---|---|
| MT (paper) | 13 | 0.632 | 0.698 | +0.066 |
| ACIC 2016 (paper) | 30 | 0.818 | 0.975 | +0.156 |
| MTAC | 20 | 0.611 | 0.667 | +0.057 |
| MTH | 20 | 0.454 | 0.510 | +0.056 |
| MTHAC | 20 | 0.428 | 0.491 | +0.063 |
| MTD (n = 20000) | 20 | 0.474 | 0.773 | +0.299 |
| MTDS | 20 | 0.434 | 0.583 | +0.149 |
| MTDAC | 20 | 0.427 | 0.548 | +0.121 |
The difference is positive in every family and ranges from 0.06 to 0.30. So the reviewer is right that both cases occur: sometimes the selected subgroup has a favorable baseline, and sometimes the action carries most of the gain. This is expected, since SCAT maximizes the success probability and not the improvement.
Optimizing the improvement. For this goal, the partition-based model must approximate $P(Y\mid X,T) - P(Y\mid X)$ instead of , i.e., the learner models the difference directly. We show in Appendix A.2 (Proposition A.3) that the population result carries over under this assumption: the best cell of such a model gives the action with the largest improvement, up to the approximation error, and again without the causal graph among the treatments. The finite-sample result and the experiments are left as future work.
Why the paper studies the absolute success probability. We see the two objectives as complementary, as they answer different questions.
- The absolute success probability answers a question of its own. A clinician may ask whether there is a group of patients who, under a certain treatment plan, reach a very high survival probability (or, with the outcome reversed, remain at very high risk). The answer to this question is a level and not a difference, as it tells where a consistent plan is enough and where it is not.
- An improvement is relative to a reference, and it should be read together with it. In observational data the individuals already receive some existing treatment practice, so "no intervention" means keeping this practice and not the absence of treatment. A small difference from that practice then does not necessarily mean that the recommended action is unimportant, since it can also mean that the existing practice in that subgroup is already close to the recommended action.
W3. Separating subgroup selection from response estimation
Estimation error within the same subgroup. In our evaluation this error is identical for all methods by construction. A method only proposes candidate subgroup-level actions, and the estimate of any candidate is computed by one procedure that is shared by all methods: the Laplace estimate on the held-out instances that are in the subgroup and whose observed treatments match the action. For a fixed ground-truth subgroup and intervention, every method would therefore report the same estimate and the same error.
Hence the differences in estimation error between the methods come only from which subgroup-level action is selected, and never from a better or worse estimator. An action that is supported by more held-out instances has a more reliable estimate, and we regard selecting such an action as a merit of the method. We state this explicitly in the experimental setup.
Identification accuracy. We added a family with a known ground truth as suggested (MTD, 20 datasets, n = 20000): a subgroup defined by three covariate conditions is planted, in which two treatments set jointly are the best action. For the action selected by each method, we check whether its subgroup uses the planted covariates and whether its intervention contains the planted treatment values.
Share of runs (5 splits per dataset), average rank over the datasets in parentheses:
| method | subgroup and action recovered | action recovered | true success probability |
|---|---|---|---|
| SCAT | 0.47 (2.20) | 0.55 (2.88) | 0.773 |
| PT-Exact | 0.00 (4.08) | 0.32 (3.65) | 0.586 |
| PT-Hybrid | 0.04 (3.65) | 0.33 (3.52) | 0.591 |
| MB-IAMB | 0.04 (3.65) | 0.34 (3.45) | 0.594 |
| MB-Lasso | 0.03 (3.78) | 0.26 (4.05) | 0.562 |
| MB-DRScreen | 0.04 (3.65) | 0.34 (3.45) | 0.597 |
SCAT recovers both the subgroup and the action on more datasets than every competitor (better on 11 to 12 of the 20 datasets and worse on at most 1, p <= 0.002).
We thank the reviewer for the encouraging and very sharp reviews which help us improve the paper.
W1. Separate contributions of the tree structure and MDL
We added both suggested alternatives, under the same action-selection and evaluation procedure as SCAT. For the predictive tree, CART (with cross-validation for pruning) replaces the MDL tree and nothing else changes. The mixture of experts follows the reviewer's description: a gate on X with 2, 4, or 8 experts, and L1-penalized logistic experts on T, both tuned by cross-validation.
True success probability and absolute estimation error of the selected action on the two benchmarks of the paper, average rank over the three methods in parentheses:
First, our main contribution is not a specific algorithm, but the reduction of the causal problem to learning a partition-based predictive model, and this reduction does not restrict which partition-based learner is used: the population result (Theorem 3.3) holds for any partition of the covariates and the treatments. SCAT is one instantiation, with which we validate the framework empirically. The comparison with CART confirms this: CART performs on par with the MDL tree on MT (SCAT is better on 7 datasets and worse on 6, p = 0.79), and the MDL tree is slightly ahead on ACIC (better on 17, tied on 4, worse on 9, p = 0.024). We chose the MDL tree for a practical reason: it has no hyperparameter and is deterministic, while the action selected with CART changes with the seed of the cross-validation folds on 5 of the 8 MT and ACIC datasets we checked (10 seeds on the same data split). Which partition learner suits which data is left as future work.
Second, the partition structure matters. The mixture of experts is clearly worse: SCAT has the higher success probability on 11 of the 13 MT datasets (p = 0.003) and on all 30 ACIC datasets (p = 2e-9). We think the reason is that each expert is additive in the treatments, so it cannot represent that a combination of treatment values is good while each value alone is not, which a tree on (X,T) can. Further, the subgroups of a mixture of experts are regions of a linear gate and not rules, so they are also harder to read.
So the benefit comes from the partition structure and much less from the specific criterion, which fits our claim that the contribution is the reduction to a partition-based predictive model.