acceptodds
Withdrawn from ICLR 2027

SwordBench: Evaluating Orthogonality of Steering Image Representations

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Steering or intervening on model representations at inference time to correct predictions is essential for AI interpretability and safety, yet existing evaluation protocols are limited to ambiguous language modeling tasks. To address this gap, we introduce SwordBench, a benchmark for steering image representations of vision models across multiple backbones and concept removal tasks. Beyond a unified benchmarking suite, we propose new evaluation notions that uncover the second-order effects of orthogonalization among concept activation vectors for pragmatic steering. Specifically, cross-concept robustness measures the stability of concept detection performance across inputs orthogonalized against alternative concepts, and collateral damage quantifies whether steering inadvertently affects model performance on a downstream task for inputs lacking the bias. We find that although a linear support vector machine exhibits superior separability and orthogonality, it fails to achieve zero collateral damage, often trailing sparse autoencoders. In controlled synthetic settings, both standard baselines and optimization-based methods fail to achieve perfect steering.

Primary area:
datasets and benchmarks
Keywords:
Explainable AI, XAI, Mechanistic interpretability, Concept-based explanations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pTfDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Steering, or intervening on a model's representation at test time, is becoming a strategy in self-supervised learning to mitigate shortcut learning by vision models. The authors argue that the well-studied "steering" in the language space mirrors "concept removal" in the vision space. While there are existing benchmarks to assess steering, this paper introduces SWORDBench, a benchmark for steering image representations at test time. The authors evaluate SWORDBench across a range of backbones and concept removal tasks. In addition to providing a benchmark, the authors provide novel metrics to assess steering performance and use those metrics to discover insights.

Critical strengths
  • Thorough evaluation on 5 popular datasets (and investigation into two more)
  • Thorough evaluation metrics and steering metrics that are well discussed
  • Well-identified gap in the literature of steering behaviors on image spaces.
  • Interesting novel metrics. I found the added metrics of cross concept robustness and collateral damage to be really interesting contributions of this work. I think nice extensions of these metrics could contribute to characterizing bias-accuracy tradeoffs that are rampant in the field.
Critical weaknesses
  • I am not entirely convinced that "steering" in language models is the same as "concept removal" in the vision space. The authors seem to take this analogy for granted. While steering can be used to mitigate or remove an effect in language models, it can also be used to add an effect [1, 2]. Thus, I have an issue with the overall framing of the paper that this is steering for vision models. I believe that either the claims should be adjusted to follow concept removal or that additional experiments need to be included for additive effects of steering.

  • Limited novelty in experimentation. Orthogonalization is used for concept erasure which is a well-established approach in the debiasing space, even for CLIP vision models (as used in this paper) [3]. To me, it is not clear that this experiments sufficiently expand on Chuang et al.'s work [3]. Chuang et al. also used orthogonalization to improve performance of classifiers by removing concepts on a similar set of datasets and backbones. The primary difference seems to be this submission's framing of steering instead of debiasing. However, as mentioned in my first weakness, I'm not sure that this steering at all (seems like debiasing called something else).

  • There is a lack of tie to real-world use (although a little in the Appendix). While broad claims are made about the findings on the particular datasets studied here, it is not clear to me what a user would do after running their dataset on SWORDBench. I think this is an important note for a benchmark paper.

  • (Minor). Lack of explanatory figure. Given the complicated pipeline and evaluation, I would have found this helpful for comprehension but not necessary.

[1] https://arxiv.org/abs/2402.01618 [2] https://arxiv.org/html/2602.00333v1#bib.bib3 [3] https://arxiv.org/pdf/2302.00070

Questions for authors

Why do you believe steering in language is equivalent to concept removal in images?

How would you expect results from SWORDBench to be used in the real-world? I.e. how are these insights actionable for downstream users?

Reviewer_YdEsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to provide a benchmark, called SwordBench, for steering image representations of vision models across multiple backbones and concept removal tasks. To create the benchmark, authors have combined 2 synthetic and 4 real datasets for evaluation. Furthermore, they define a set of metrics for evaluation of method on this benchmark.

Critical strengths

The problem identified by the authors seems interesting and the paper looks neat with good figures.

Critical weaknesses
  1. I am not too confident about this work, because as I understand authors have combined existing datasets and tasks to evaluate the methods. There seems to be no actual dataset/benchmark contribution as claimed by the authors. All actual contribution I see is defining some evaluation metrics but other than that, the dataset and methods being evaluated seems to be borrowed from existing research.

  2. Furthermore, the motivation seems to be lacking. The shortcut learning and spurious correlations highlighted as motivation in the introduction seems to be based on relatively old research. I am not completely convinced if they hold true in newer generation of models.

  3. I am not too convinced on use/applicability of some metrics like collateral damage (CD). As per the definition, it is used to measure how well the model behaves against concept-removal method after that concept is removed. And as authors have guessed, ideally it should not have any affect. But most of the time these concepts are not 100% separable and the accuracy of concept-removal method can also affect the results. How does authors handle both of these situtations?

Questions for authors

See Weakness.

Reviewer_nfeCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Classifiers built on frozen vision encoders often rely on spurious shortcuts, and steering, that is, editing the representations of these encoders at inference time (withtout any re-training), offers a way of correcting these biases and enhance interpretability. Unlike in language modeling, vision lacks a protocol for evaluating steering. This paper introduces SwordBench, a benchmark for evaluating concept activation vectors (CAVs) used to steer the final-layer image representations of frozen vision encoders (CLIP, SigLIP, DINOv2). It compares 16 CAV extraction methods, including linear probes, mean-difference variants, PCA/LAT, AurA, and several sparse-autoencoder-based approaches, with nonlinear probes as detection upper bounds. Here "steering" is limited to concept erasure using orthogonal projection. Evaluation uses real, biased datasets (CelebA, ISIC, with Waterbirds and Counteranimal in the appendix) and synthetic counterfactual datasets (ImageNet-W watermarks, ImageNet-C corruptions). The authors propose two new metrics: Cross-Concept Robustness (CCR), the AUC a CAV retains after other concepts are erased from the representation; and Collateral Damage (CD), the downstream accuracy drop on concept-absent samples. Their main conclusions are that linear SVM and logistic regression match nonlinear probes in detection; that optimization-based methods win on disentanglement and CD on real data while statistical methods steer better on synthetic data; that only LAT and PCA approach optimal steering disparity; that AUC and F1 are poor proxies for steering quality; and that SAE-based CAVs need neuron filtering to be competitive.

Critical strengths
  • Tackles an important problem: removing bias from model representations, improving interpretability, introducing a benchmark.
  • Timely and useful resource: Concept-erasure methods are proliferating with no common evaluation protocol in vision. A unified suite covering 16 methods, 3 backbones and 6 datasets, with code released, would be valuable to the community. • Asking whether erasure harms concept-absent inputs, and whether a CAV is entangled with other concepts, is an interesting direction.
  • The paper shows convincingly that high AUC does not imply good erasure, e.g., SVM reaches AUC ≈ 1.0 yet leaves most of the watermark gap open.
Critical weaknesses
  • The benchmark evaluates concept removal only, not steering. Steering covers a much broader family of interventions: adding or amplifying concepts, tuning intervention strength, conditional and multi-concept steering, and intervening at intermediate layers or in generative outputs. SwordBench tests only one of these, orthogonal projection at full strength (α = 1), with α varied for just two methods in an appendix. The title and framing ("steering image representations," the analogy to LLM steering benchmarks such as AxBench) therefore promise more than what is presented in the paper. The authors should rename the benchmark or narrow their claims to linear concept erasure, or add at least additive steering.
  • SwordBench is framed as evaluating CAVs for bias removal, and its findings will be read as guidance on which method to use for that purpose. However, it does not report any bias removal performance. On CelebA and ISIC, it measures only concept detection (F1) and side effects (CD), with no measure of whether removal succeeded. The authors pair CD with an efficacy measure (SD) on synthetic data but drop it on real data, where their method-family conclusions are drawn. Without it, a good CAV and an ineffective one can score the same.
  • The CelebA task does not match the stated motivation. Concept removal is motivated as removing biases and shortcuts, but most of the 39 CelebA attributes are not biases with respect to the Male target. Attributes such as Smiling, Wavy Hair or Attractive have no clear reason to be erased, and others (Mustache, Goatee) are near-perfect correlates of the label. For most CelebA concepts, it is unclear whether removal is desirable at all, so collateral damage is hard to interpret. Yet CelebA, with 39 of the benchmark's real-data concepts, drives most of the real-data conclusions. The settings with genuine spurious concepts (ISIC, Waterbirds, ImageNet-W) are those with few concepts or very small test sets.
  • Several headline findings are not supported by the paper's own results. ◦ Finding 1 states there is no significant difference between linear and nonlinear probes. Table 33 shows significant differences for SVM and LR on CelebA. On ISIC, with only 4 concepts, the Wilcoxon test cannot reach p < 0.125, so the comparison is uninformative by design. ◦ Finding 3 states that only LAT (ImageNet-W) and PCA (ImageNet-C) reach optimal steering disparity. The same section lists other methods that do, and PCA is the worst method on DINOv2 ImageNet-C. On ImageNet-C, nearly all SD values are near or above 1, so no method steers successfully there. ◦ Finding 4 rests on PCA biplots rather than correlation statistics. ◦ The backbone conclusion (SigLIP superior) conflicts with the critical-difference plots.
  • The synthetic results rely on very little data. The test sets are small, so differences between methods often come down to a handful of images. The error bars also look too small, since the seeds don't seem to change the test data. So, I'm not convinced the method rankings on these datasets are reliable.
  • Some important baselines and design choices are missing or not well justified. The paper leaves out well-known concept erasure methods like LEACE and INLP, and the reasons given for this are not convincing. The way the projection is done (without centering) might itself cause part of the collateral damage. Also, the three groups of methods are never clearly defined, and three of the 16 methods turn out to be the same method.
Questions for authors
  1. Can you report a removal-efficacy measure on real data? Examples include post-projection concept decodability by a freshly trained probe, or accuracy on the correlation-reversed ISIC subsets and Waterbirds minority groups. Do the method rankings from CD and CCR hold under such a measure?
  2. How do you rule out that low CD reflects an ineffective CAV rather than a precise one? Specifically, how do you explain SAE-AurA's best-in-class CD given MS = 1.0 and the lowest CCR?
  3. What is the measured correlation between CCR and CD, per dataset and backbone? If CCR does not predict CD, what is its intended use?
  4. How do the results change with a centered projection, h − vᵀ(h − μ)v, and with LEACE or INLP as reference erasure methods? Do the conclusions about optimization-based versus statistical methods survive?
  5. For ImageNet-W/C, what exactly varies across the five seeds? Can you report confidence intervals that include test-set sampling (e.g., bootstrap over test images), ideally with a larger test set and more confounding pairs? Which Finding 3 rankings remain significant?
  6. Given Table 33 and the n = 4 limit on ISIC, will you restate Finding 1 in terms of effect sizes or an equivalence test? Please also reconcile Finding 3 and the SigLIP backbone claim with Tables 2, 27–31 and Figures 18–19.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.