- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies heterogeneous treatment effects in cases where the relevant effect modifiers are not directly observed but may be encoded in complex pre-treatment measurements such as satellite imagery. The key idea is to identify a small set of features that contains all information about treatment-effect heterogeneity. The method is validated and on two real-world anti-poverty programs.
Critical strengths
- The problem is very relevant and the core idea is very good and ambitious.
- The combination of learned representations with conditional effect-modifier selection is conceptually interesting.
- The real-world applications are very strong and make the work more interesting than a purely synthetic study. They are discussed rigorously.
- Figure 2 is useful.
- The semi-synthetic CelebA experiment is a neat way to test if the method can recover effect modifiers.
- Compared with approaches that treat a satellite embedding directly as an effect modifier, conditional feature selection is a more principled way to deal with correlated representation features.
Critical weaknesses
Presentation
- The introduction does not clearly position the method relative to causal inference, causal discovery, and conditional feature selection. The paper does not explain the core idea simply enough at the start.
- The paper moves between a general method and very specific poverty examples, which makes the intended scope unclear.
- A running example would help. The cash-transfer example from Page 1 could be used throughout.
- The full pipeline should be shown in one simple diagram with more detail and explanation than the given the “From Tokens to Policy” overview.
- The main assumptions are spread across the paper. They should be summarized in one place together with their role and failure modes.
Method
- Overall, the causal interpretation seems stronger than what can be established in practice. NEXIS selects features in a learned representation, but treating these features as the true causal effect modifiers requires strong assumptions that are hard to follow and to verify.
- The method assumes that the pre-treatment data capture all relevant treatment-effect heterogeneity. This seems very strong in realistic policy settings.
- The method also assumes that the learned representation exposes the relevant latent modifiers in a useful way. To me, it is unclear how realistic this is.
- The representation step can itself create problems, for example when several concepts are mixed in one coordinate or one concept is spread across several coordinates.
- The paper should separate the randomized and non-randomized settings more clearly.
- It is unclear how stable the selected modifiers are across different spatial resolutions, feature definitions, or learned representations.
- The jump from CelebA to the real-world applications is large. CelebA uses known modifiers in a controlled setting, while the policy data contain unknown, correlated, spatially aggregated, and, most importantly, potentially unmeasured modifiers.
Questions for authors
- How general is the method beyond the satellite-imagery applications?
- How common are randomized experiments in the policy settings NEXIS is meant to address? As many real policy interventions are not randomized.
- What happens if intervention placement depends on geographic features (specifically if these are visible in the satellite images, such as roads or infrastructure)?
- How should selected variables be interpreted when they are proxies for earlier socioeconomic processes rather than directly manipulable causes?
- How sensitive are the results to the choice of spatial resolution and aggregation level?
- The treatment may be randomized within the study, but the study population and geographic setting are not randomly selected. What does this imply for the causal interpretation?
- Can the choice of spatial unit or resolution change the detected heterogeneity and create ecological-fallacy-type interpretations?
2 comments
Thank you for your response and the additional effort. I will need some more time to carefully review the revisions before deciding whether to raise my score.
We thank Reviewer @xVg4 for the detailed feedback, concerning (i) the assessment of the pipeline requirements, including the gap between CelebA and the applications and the stability of the results (shared concerns with all reviewers), (ii) presentation, and (iii) further minor questions.
i. Requirements assessment and results interpretation
As for any causal identification result, and as acknowledged in the paper's limitations, the pipeline requirements cannot be verified from data alone. NEXIS discoveries are therefore, to start with, hypotheses of causal and interpretable treatment interactions. Our belief in these requirements can still be strengthened empirically, and this revision does so in two ways that directly bridge the gap from CelebA to the applications:
ii. Presentation
iii. Further questions
We hope these new results address the reviewer's concerns and reinforce the assessment of the paper's contribution.