- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes Self-Controlled Intervention (SCI) to study primarily how image occlusion affects vision models. It measures changes in predictions and internal representations, focusing on the distribution of effects. Experiments cover several classifiers and object detectors.
Critical strengths
The distributional analysis provides useful insights that average performance metrics can miss. E.g., some images become more vulnerable than others, and changes in internal representations do not always follow changes in predictions.
Critical weaknesses
-
Limited Novelty. I feel the proposed framework has components including paired image comparisons, quantile analysis, MMD and HSIC, are well established techniques. But it remains unclear to me what new insights this framework provides scientifically.
-
Limited causal interpretation. I appreciate the authors' aim to study causal effects through this technique, but I feel the main focus is on model robustness analysis rather than establishing any underlying causal mechanisms. E.g, the foreground/background comparison attempts to separate information loss from artificial effects caused by masking. However, this requires assumptions that are not experimentally established. The authors acknowledge these limitations, which weaken the causal interpretation.
Questions for authors
- What important findings could SCI reveal that simpler paired occlusion analysis using per-image effect distributions could not?
- How can the authors support the assumptions needed to separate information loss from masking artifacts?
- Where is the code you stated in the reproducibility secion?