- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1(−1)
Read review
Paper summary
The paper introduces Steerable Lens, a framework using multiscale image edits to study classifier sensitivity. Through experiments on MNIST and CelebA, it examines how classifier confidence, transfer to other models, image structure, and human judgments differ. The study highlights that increasing classifier confidence does not necessarily produce a recognizable target-class change.
Critical strengths
Clear separation of evidence. The paper distinguishes confidence, transfer, reconstruction, and human perception, avoiding unsupported claims of semantic change.
Transparent evaluation. It matches rendered-image budgets, reports non-attainment, and separates selected illustrations from aggregate results.
Useful multiscale probes. Signed band interventions and random controls reveal scale-dependent sensitivity, supported by within-sample robustness checks.
Critical weaknesses
Limited novelty. The framework combines established representations and gradient-based editing, with limited new insight beyond the known confidence–recognizability gap.
Limited perceptual validation. Two complete rating sets across 24 source–attribute cases provide insufficient evidence for general perceptual conclusions.
Limited independent confirmation. Reused test images, a test-selected judge, and absent driver-seed replication weaken generalizability. Some comparisons retain only 7 of 25 sources.
Confounded representation effects. Matched L2 budgets do not isolate geometry from optimization, while reconstruction-family dependence and unmatched controls leave smoothing or preconditioning as alternative explanations.
Questions for authors
i. How do you distinguish greater smooth-reconstruction compatibility from weaker target-directed progress? Does CSP’s structural advantage persist when both image-change budget and achieved target margin are matched?
ii. How do you separate classifier sensitivity from smoothing and preconditioning induced by CSP coordinates? Do the conclusions persist with spatially correlated controls and comparable image-space optimization, rather than matched L2 distance alone?
iii. Do the rankings persist on untouched images with independently trained drivers and validation-selected judges? How sensitive are the conclusions to filtering for common budget attainment?
iv. Can independent raters confirm the perceptual advantage on fully paired source cases? On those same edits, do automated diagnostics predict recognizable target change separately from corruption and unrelated-content changes?