acceptodds
Under review as a conference paper at ICLR 2027

Probing and Steering Vision Model Decisions along Concept Axes in Diffusion Space

Abstract

We introduce Mask-supervised Orthonormal Concept Axes (MOCA), a diffusion-based framework for concept-wise probing and steering of vision model decisions. Through independent manipulation of individual visual concepts in a given image and observing the resulting model response, MOCA enables concept-wise probing underlying a vision model's decisions. By directly scaling concept strengths, MOCA formulates visual explanation as concept-wise interventions, probing feature sensitivity to reveal whether specific visual elements explain the decision or act as spurious correlations. Experiments across multiple datasets and vision models demonstrate that MOCA produces spatially localized and semantically meaningful interventions, enables identification of spurious cues, and supports downstream mitigation. We extend MOCA for visual counterfactual explanation, providing a broader range of explanations for the same prediction through interventions on different visual concepts within the image. Experiment results show that MOCA improves the overall accuracy in the mitigation task by 44.9% on HardImageNet and 3.0% on COCO relative to the corresponding baselines, achieves comparable performance in counterfactual explanation tasks, and consistently support additional explanations through non-ground-truth concepts.

Primary area:
interpretability and explainable AI
Keywords:
Spurious Correlation Mitigation, Visual Counterfactual Explanation, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Jj2JOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MOCA, a diffusion-based framework that decomposes classifier-free guidance as a sum of per-concept contributions. which allows concept-wise probing and helps with visual counterfactual explanations.

Critical strengths

The method provides an appealing and simple way to get per-concept control over image content without training a dedicated model.

Critical weaknesses
  1. Experiments. Important baselines like DVCE or RCSB are missing (while cited in the related work). The included baselines were re-run, and their results fall far below published numbers on the same class pairs. Against published baselines results, MOCA's flip rates for example, are below ACE, RCSB and MaskDiME on all three parts. And even in the paper's own Table 4, MOCA does not clearly outperform ACE.
  2. From presented visual examples, the counterfactuals look more like adversarial examples. I don't find changes produced by MOCA in neither figure 4 or 5 to be meaningful. For example in Figure 4 sorrel -> zebra, MOCA only makes the color of the horse a bit more bleak not making it an actual zebra (while TIME at least puts line pattern on it).
Questions for authors
  1. Why were the baselines re-implemented on your generation trajectory instead of being evaluated in their original settings? Can you compare MOCA against the published numbers under the ACE protocol, and add DVCE and RCSB? Which classifier is explained?
  2. How is the flip rate defined for VCE? Fig. 5 marks flips as successful with a target probability of 0.01. Is success top-1 over all 1000 classes, or a comparison within the class pair?
Reviewer_H6apOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Work introduces MOCA - steering system based on masking mechanism. It is meant for diffusion models. It splits cls-free guidance into concept directions into orthonormal subspaces. It uses SAM to change only exact part of the iamge. Scaling one axis suppresses or amplifies that concept. Applied for steering and counterfactuals.

Critical strengths

first of all, it shows how to mitigate spurious correlation through steering method, nice!

It is model agnostic.

Good FID results, but only 70% of edits is in mask region. I would expect higher.

Interventation are continous and can be better controlled.

Critical weaknesses

I am not sure how ERM is combined with cues, my intuition says that MOCA is not used when ERM is applied. That would be a false claim.

Moreover, the only method that is conditioned on cues, is worse. I am a bit confused with those results.

Some cues are weird such as Bat and baseball player, or people and sunglasses. they are kind of important for those classes and not spurious. As far as I am aware HardImageNet comes with known spurious cues, why not to use them directly?

Isn't the steering just adversarial attack? Fig. 7: almost invisible change flip seat belt to a limousine and volleyball to a knee pad.

Loss uses similarity from CLIP and VIT-H, while it is also used to measure difference. It is obvious you'll get better when you measure what you optimize.

Missing baseline: simple SD inpainting.

Questions for authors

What is MOCA role in ERM? how does it differs from standard ERM?

Why aren't you using known cues from HardImageNet? What is your definition of spurious, I have an impression that it is different than community?

How do you distinguish yourself from adversarial attacks? Can you make an experiment that would act as one, so that we can see if this is an atttack ratehr than steering?

How does MOCA compare to baselines?

Reviewer_qaVfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MOCA, which decomposes diffusion guidance into image-specific concept directions and learns an orthonormal basis using object masks and semantic supervision. Scaling these directions probes how individual objects affect a classifier or VLM. A second optimization steers predictions through a selected concept region. Experiments cover concept localization, sensitivity probing, spurious-cue mitigation, and counterfactual generation.

Critical strengths

The method helps test how different objects affect a prediction. The reported fine-tuning results also show improved classification accuracy.

Critical weaknesses
  1. The edits are not shown to isolate the named concept. Table 8 places only 70.6% and 59.0% of the total change inside the target mask on HardImageNet and COCO. Steering also drops the semantic and alignment losses, with no quantitative semantic check afterward. A prediction change alone therefore does not establish the effect of the named concept.

  2. The scaling protocol is inconsistent. Appendix says the axes are learned once per image at a coefficient of 8, then says they are optimized separately for every coefficient. If the axes change, Figure 2 is not measuring different strengths along a fixed direction.

  3. The mitigation gains are hard to attribute to cue discovery. Table 3 lacks a clearly identified control using the same fine-tuning procedure without MOCA-based cue selection. The paper also does not explain how the selected cues enter ERM training. Some of the improvement could come from fine-tuning itself.

Questions for authors
  1. Are the flip rates and image-quality metrics computed on raw diffusion outputs or mask-composited images? Algorithm 2 uses the composite for its decision loss.
  2. Were the axes fixed across scaling coefficients in Figure 2, or reoptimized for each coefficient?
  3. How do the discovered cues enter ERM training, and what improvement remains over the same fine-tuning procedure without MOCA-based cue selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.