acceptodds
Withdrawn from ICLR 2027

CONFIDENCE IS NOT EVIDENCE: PROBING CLASSIFIER SENSITIVITY WITH STEERABLE LENS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Moving an image along a classifier's gradient is a common way to probe what change would carry it across a decision boundary, but crossing that boundary does not by itself establish a recognizable change toward the target class. We introduce Steerable Lens, a framework for investigating classifier sensitivity through complex steerable-pyramid (CSP) interventions, with pixel and Fourier-phase edits as comparators. At matched image-change budgets, we measure three quantities separately: the optimized classifier's confidence, transfer to independent classifiers, and compatibility with a smooth image transformation. These measurements yield different method rankings. A pixel perturbation can drive the optimized classifier to near-certain target confidence, above 96% on MNIST digits, yet independent classifiers rarely agree on the target, and the edit is poorly captured by a smooth deformation. CSP edits align closely with a smooth warp-plus-gain model while classifier confidence stays low. On CelebA the structural ordering between representations depends on the reconstruction model. In a pilot human study, CSP edits receive more ratings of clear attribute change and no visible corruption than pixel or Fourier-phase edits. Frozen band probes further reveal fine-scale sensitivity in the examined driver classifiers, while larger budgets reduce common attainment across bands and controls. These results position CSP as a controlled diagnostic representation and motivate evaluating optimized confidence, classifier transfer, and smooth reconstruction separately, without treating any one as certification of recognizable target-class change.

Primary area:
interpretability and explainable AI
Keywords:
complex steerable pyramids, local-phase edit, interpretability, image classifiers, cnn, convolution neural networks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HgdaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

The paper introduces Steerable Lens, a framework using multiscale image edits to study classifier sensitivity. Through experiments on MNIST and CelebA, it examines how classifier confidence, transfer to other models, image structure, and human judgments differ. The study highlights that increasing classifier confidence does not necessarily produce a recognizable target-class change.

Critical strengths

Clear separation of evidence. The paper distinguishes confidence, transfer, reconstruction, and human perception, avoiding unsupported claims of semantic change.

Transparent evaluation. It matches rendered-image budgets, reports non-attainment, and separates selected illustrations from aggregate results.

Useful multiscale probes. Signed band interventions and random controls reveal scale-dependent sensitivity, supported by within-sample robustness checks.

Critical weaknesses

Limited novelty. The framework combines established representations and gradient-based editing, with limited new insight beyond the known confidence–recognizability gap.

Limited perceptual validation. Two complete rating sets across 24 source–attribute cases provide insufficient evidence for general perceptual conclusions.

Limited independent confirmation. Reused test images, a test-selected judge, and absent driver-seed replication weaken generalizability. Some comparisons retain only 7 of 25 sources.

Confounded representation effects. Matched L2 budgets do not isolate geometry from optimization, while reconstruction-family dependence and unmatched controls leave smoothing or preconditioning as alternative explanations.

Questions for authors

i. How do you distinguish greater smooth-reconstruction compatibility from weaker target-directed progress? Does CSP’s structural advantage persist when both image-change budget and achieved target margin are matched?

ii. How do you separate classifier sensitivity from smoothing and preconditioning induced by CSP coordinates? Do the conclusions persist with spatially correlated controls and comparable image-space optimization, rather than matched L2 distance alone?

iii. Do the rankings persist on untouched images with independently trained drivers and validation-selected judges? How sensitive are the conclusions to filtering for common budget attainment?

iv. Can independent raters confirm the perceptual advantage on fully paired source cases? On those same edits, do automated diagnostics predict recognizable target change separately from corruption and unrelated-content changes?

Reviewer_9eT3OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper compares pixel, Fourier-phase and steerable-pyramid-phase gradient edits at equal image change. It shows that driver confidence, transfer to other classifiers, and smooth-deformation fit rank the methods differently. It argues these should be reported separately, with a very small human pilot favoring the pyramid-based edits.

Critical strengths
  • S1: The paper reports its evidence transparently. Qualitative figures state how they were selected, and the reported numbers are consistent.
  • S2: For MNIST all 4 methods are evaluated on the same 300 sources at matched image distance. The protocol was fixed before the run. Differences are reported as paired within-source medians with bootstrap intervals (Sec. 5.1).
Critical weaknesses
  • W1: The core issue with this paper is its presentation. As stated in the “AI use statement” generative AI assisted in the editing of the manuscript. I am not suggesting that this is an understatement but for a fact this paper reads tailored to machines rather than humans indeed. A multitude of sentences spread over more than 3 lines with colons, semicolons and several commas. Often sentences state what is not done before they say what is done. This issue is so sever that I will open multiple sub-points of this weakness.
    • W1-1: Most importantly the proposed method is unclear. From what I have read, I can not clearly say what exactly the steerable lens framework is they propose. At least the high-level idea must be clear from the paper.
    • W1-2: Sec. 3 overwhelms the reader with math without any motivation. Even lacking definition of symbols: What is a analysis transform, synthesis operator, edited coefficients, band, scale and orientation, low- and high-pass residuals and the log-magnitude increment? These are examples from the first 5-line paragraph alone.
    • W1-3: The paper does not introduce most of the technical terms (examples see below) assuming the reader knows (I do not). More so, many terms appear with synonyms without clarification. To confuse even more, often terms mean something which is established in the related literature but the author use another name for it without stating that either. Here are some examples (the list is much much longer):
      • Sec. 4 defines “optimized classifier G” but never says “driver” = G. The term first appears in the abstract undefined. This is the same as the optimized classifier and the driver classifier as far as I can tell. This is essentially a source model or surrogate model (transfer attacks), or classifier under explanation (counterfactuals).
      • Then there are the independent classifiers, which (so I assume) are related to the judge / two-judge agreement (Sec. 1, Sec. 4) but this is never made clear.
      • In Abstract, Sec. 5.1–5.2 the authors talk about the structural axis / structural ordering / structural comparison but its not clear what is meant here. Is this the the smooth-reconstruction score E?
  • W2: The authors write in their ethics statement: “Human ratings were provided voluntarily by a convenience sample of friends and family.” This raises the question about bias in the human evaluation study. More so, they only have 2 complete human rating (Sec 6., Limits) because the other supplied ratings were incomplete or duplicate (App. D). Hence, a method effect cannot be separated from rater idiosyncrasy. Not even inferential tests or chance-corrected agreement would resolve this, since rater-level variance cannot be estimated from two ratings. This is critical because the abstract and conclusion present this as the main evidence in favor of CSP on CelebA.
  • W3: The results cannot be reproduced from the submission. It contains no supplementary material. The reproducibility statement (p. 10) refers to a figure manifest with checksums, which is not provided. In detail:
    • Model: App. A.1 states that "CelebA uses historical convolutional, residual-network, and MLP checkpoints".What is historical here? The robust ResNet-50 serves as judge in the frozen probes and as driver in the step-local runs (App. F). Its threat model, ϵ\epsilon and source are not stated. The steerable pyramid lists a few hyperparameters, which are however not explained.
    • Data: The CelebA development split is not described, although it calibrates the optimizer (p. 3) and supplies the band-probe faces (Sec. 5.3).
    • Training: Beyond seeds and validation accuracies for MNIST, no training details are given for any classifier or for the CLIP linear head. The edit optimizer also lacks the coordinate bounds and the block weights.
    • Evaluation: The warp and gain parametrization behind the reconstruction score EE is not specified. The budget-matching search is described only as a search over a scalar amplification or step length (Sec. 4). Its algorithm, search range and stopping rule are missing. Consequently, no reported number can be verified independently.
  • W4: The natural-image experiments rely on 3 custom cat photos (Sec. 5.4, App. F) of unstated origin (two are shown in Fig 13). 3 selected images cannot support any claim about natural images. As illustrations this is fine (as claimed in Sec. 1) but these few samples occupy all of Sec. 5.4, App. F and App. H. Unlike Sec 5.1 and 5.2, they report no reconstruction score E, no transfer rate and no comparison with pixel or Fourier edits at matched budgets. Each target is a neighboring cat class (Egyptian cat, tiger cat), so the steering never leaves the cat category. Even so, the driver reaches only 0.606 and 0.534 target probability at L2 roughly around 28 (App. F). App. G adds four dog close-ups of unstated origin. This does not make it better.
  • W5: The central message that classifier confidence does not establish recognizable change is well known. Just to name the cited work of Ilyas et al. NeurIPS 2019. They attribute adv. examples to non-robust features that are highly predictive yet incomprehensible to humans, which are exactly such confident but unrecognizable edits. A quick check of existing works would have improve the currently very weak baselines: There are spatially transformed and deformation attacks, which would likely score E=100% (Xiao et al., ICLR 2018). Further, based on Tsipras et al., ICLR 2019, without pixel edits from a robust driver, the paper cannot tell whether the smoother CSP edits stem from the steerable-pyramid coordinates or merely from filtering the noisy gradient of a presumably standard MNIST driver. These are highly cited works (known in the field) and they are already 8 years old (so not even sota).
  • W6: The evaluation is weak and thin itself. I am not restate the sample size issue or the thin baselines again. On l. 280/81 its even mentioned that the driver G, so the classifier the edit is optimized against (eg. editing 3 to 8 until L2=2), assigns the target a median probability of only 0.35%. The edit fails even on its own target classifier. This raises questions about the faithfulness of the models overall.
  • W7: E asks how much of an edit can be explained by bending and stretching the original image. Then, its absolute expected that a methods that builds its edits from warp-like pieces, scores high here. This is exactly what CSP is doing (it constructs using many small local displacements). Thats not a finding.
  • W8: This gets worse: Contribution 1 is never tested and contribution 2 is very weak:
    • In contribution 1 they claim to separate frozen sensitivity, persistent coordinate optimization and step-local steering. But there is no experiment on this as far as i can tell.
    • In contribution 2 they claim a reversal across MNIST and CelebA. But as the setups are so different, the reversal could come from the objective, the optimizer or the classifiers rather than the dataset.
Questions for authors
  • Q1 (W1): Can you describe the Steerable Lens framework in two or three sentences?
  • Q2 (W1-3): Could you add a glossary of all terms?
  • Q3 (W3): Which threat model and ϵ\epsilon does the robust ResNet-50 use, and where does it come from?
  • Q4 (W3): Which steerable-pyramid implementation do you use?
  • Q5 (W3): How are the warp and gain parametrized?
  • Q6 (W3): How exactly is the budget-matching search performed?
  • Q7 (W4): Can you report EE, transfer rates and pixel and Fourier comparators at matched budgets on a public set such as the ImageNet validation split?
  • Q8 (W5): What does the paper add beyond Ilyas et al. (2019)?
  • Q9 (W5): How do a spatially transformed attack and pixel edits with a robust driver perform under your protocol?
  • Q10 (W6): Why do the headline results use the direct-coordinate update when the Wirtinger update reaches far higher driver confidence?
  • Q11 (W6): How does EE compare across methods when edits are matched in driver confidence rather than only in L2L_2?
  • Q12 (W7): Why should EE be a neutral criterion across representations, given that phase edits are built from local displacements?
  • Q13 (W8): Which experiment isolates the effect of refreshing the gradient?
  • Q14 (W8): Does the reversal persist when MNIST and CelebA use the same objective, optimizer and classifier family?
Reviewer_LnvCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes using complex steerable pyramids (CSP) as an image representation when introducing perturbations to assess a vision model’s sensitivity. Perturbations are generated based on the gradients of the model’s output score with respect to the amplitudes and phases in CSP bands. The experiments compare CSP-based perturbations (only phase/phase and magnitude) with the pixel/Fourier domain perturbations, showing that CSP-based perturbations are superior on some criteria.

Critical strengths

None

Critical weaknesses

I think introducing CSP-based perturbations may inspire the community. This paper can provide more tools for models’ sensitivity analysis. However, this simple idea can be reached easily when necessary. When CSP-based perturbations are beneficial for models’ analysis, it depends on the task or the purpose of the analysis, and if there is a need to see how phases and amplitudes in certain bands affect a model’s output, one can easily implement CSP-based perturbations. That said, I’m not sure if the experimental results given in the paper provide additional useful insights into using CSP.

It would be nice if the paper could provide some new insights about shortcuts or something in well-known tasks with CSP; however, in the current version of the paper, it seems to just introduce the CSP-based perturbations, and it’s hard for me to see a strong contribution to the community.

Questions for authors

What are the tasks that CSP-based perturbations help find new insights, if any? (perhaps, this is more like a suggestion for future work)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.