acceptodds
Under review as a conference paper at ICLR 2027

Context or Face? Locating Emotion Representations in CLIP with Sparse Autoencoders

Abstract

Vision-language models such as CLIP can associate emotions with both facial expressions and contextual cues, but their predictions do not reveal which visual evidence contributes to these judgments. We investigate how joy and sadness are represented across facial and contextual evidence in CLIP using sparse autoencoders (SAEs), causal feature ablation, spatial localization, and cross-carrier linear probing. Using FindingEmo decision-factor annotations and cross-fitted feature discovery, we identify context-associated SAE features for both emotions with distinct evidential profiles. Sadness feature 9960 shows reproducible held-out context selectivity, with 95% bootstrap confidence intervals excluding zero across all three folds. Joy feature 7406 shows consistent contextual enrichment and positive heldout selectivity, although its confidence intervals include zero. Both features rank above all 200 randomly sampled SAE control features in context selectivity. At the dense-representation level, a linear probe trained to distinguish joy from sadness using only face-driven images transfers strongly to context-driven images, achieving 0.851 ± 0.011 balanced accuracy and 0.930 ± 0.006 AUROC. These results show that context-associated behavior at the level of individual sparse features can coexist with emotion-discriminative information that remains linearly accessible across facial and contextual evidence.

Primary area:
interpretability and explainable AI
Keywords:
Vision-Language Models, CLIP, Mechanistic Interpretability, Sparse Autoencoders, Emotion Representation, Causal Feature Ablation, Multimodal Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LZUeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This study investigates whether emotion-related outputs in CLIP are supported by facial expressions or contextual cues. Using sparse autoencoder features, feature ablation, and linear probing, the authors analyze joy and sadness representations. The results suggest that some features contribute preferentially to context-based outputs, while emotion information remains accessible across visual sources.

Critical strengths
  1. The research question is interesting: distinguishing facial and contextual evidence may help identify scene-based shortcuts underlying emotion predictions.

  2. The analysis combines feature ablation with cross-group classification, providing evidence of functional contributions beyond qualitative activation maps.

Critical weaknesses
  1. The novelty is limited, and generalizability remains unverified: the analysis covers only a few features, two emotions, one dataset, and one CLIP architecture.

  2. The practical value of the findings remains unclear. The paper does not demonstrate how these insights could improve emotion recognition, robustness, bias mitigation, or any other application scenario.

Questions for authors

See the weaknesses.

Reviewer_esFHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how facial and contextual evidence contribute to emotion-related representations in CLIP. Using FindingEmo decision-factor annotations, the authors examine joy and sadness through sparse autoencoder features, spatial localization, feature ablation, and linear probing. The reported results provide stronger evidence for a context-associated sadness feature, while the joy feature remains statistically uncertain. A linear probe trained on face-driven images also transfers well to context-driven images. The paper interprets these findings as the coexistence of context-associated sparse features and emotion information that remains accessible across different visual sources.

Critical strengths
  1. The paper carefully distinguishes activation, localization, and intervention effects. Ablations are compared against the unablated SAE reconstruction, which controls for the change introduced by reconstruction itself. The interpretation of uncertainty is also appropriately restrained: Table 1 distinguishes the stronger sadness result from the joy result, whose confidence intervals include zero in every fold. The authors do not use the random-feature comparison to dismiss that uncertainty.

  2. The cross-carrier probing experiment provides useful evidence beyond the individual feature examples. In Table 2, a probe trained exclusively on face-driven images achieves 0.851 balanced accuracy and 0.930 AUROC on context-driven images without adaptation. Training-only standardization and results across ten seeds make this a reasonably clear test of information accessibility. The paper also correctly avoids interpreting successful transfer as proof of identical mechanisms.

Critical weaknesses
  1. The independence of feature selection and validation is unclear. Section 3.3 states that primary candidates are retained based partly on their recovery across discovery folds. However, images held out in one fold participate in discovery in the other two folds. If features 7406 and 9960 were selected jointly after comparing all discovery folds, each evaluation fold could indirectly influence the final feature choice. Completing selection before inspecting held-out selectivity scores would not remove this dependence. The bootstrap intervals in Table 1 would then omit uncertainty introduced by selection. This matters because the held-out sadness result is the strongest evidence for the paper’s main conclusion. The selection sequence needs to be specified, with an evaluation that remains independent of the final candidate choice.

  2. The main ablation results measure changes in similarity to one emotion prompt per category. Removing a funeral-associated feature may reduce similarity to “a photo of a sad scene,” but this alone does not distinguish an emotion-related effect from a broader change in scene–text matching. Appendix F examines emotion and cue margins, but only on two generated illustrations. Quantitative margin results or prompt-robustness checks on the natural images would make the interpretation more convincing. Without these checks, the causal findings are most directly evidence about the selected image–text similarities.

Questions for authors
  1. Were features 7406 and 9960 selected after jointly inspecting their recovery or activation patterns across all three discovery folds? Please describe the exact selection sequence and clarify whether each evaluation fold was excluded from every step determining its candidate feature.

  2. Are emotion-margin results or results with alternative prompt wording available for the FindingEmo populations? If so, do the selected features retain their context selectivity? This would help determine whether the main ablation findings extend beyond the two prompts used for discovery and evaluation.

1 comment

Thank you for the additional analyses. The nested evaluation addresses my selection-independence concern, and the natural-image prompt tests strengthen the evidence for feature 9960.

The remaining question concerns emotion specificity. As you acknowledge, the scene-cue comparison remains inconclusive. The current evidence therefore supports a context-associated contribution to sadness-related similarity, while its independence from broader scene–text matching remains open. Could you provide the exact definitions and comparison prompts for the emotion and scene-cue margins, together with their numerical effect estimates and confidence intervals, including the inconclusive comparisons?

Reviewer_UND1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how the vision-language model CLIP represents emotion (joy and sadness) across different visual evidence, specifically distinguishing between facial expressions and background contextual cues.

The authors isolated face and context images using the FindingEmo dataset to analyze both sparse features (via SAE causal ablation and spatial localization) and dense representations (via cross-carrier linear probing). To ensure statistical rigor, feature discovery was separated from validation through three-fold cross-fitting, 95% bootstrap confidence intervals, and 200 random SAE control comparisons

Critical strengths

This paper investigates whether the CLIP relies on facial expressions or contextual cues for emotion recognition. It employs a statistically framework for mechanistic interpretability.

Critical weaknesses
  • W1. Poor organization and readability; lacks a global methodological perspective: The structure of Chapter 3 (Methodology) lacks coherence. Sections 3.1 to 3.7 appear as isolated components, making it difficult for readers to grasp how these modules connect to form a complete pipeline. The paper would benefit from a clear overview of the general methodology at the beginning.

  • W2. Vague description of methodology and implementation details: The description of how facial and contextual visual evidence is extracted and localized is too general. For instance, the reference features are briefly summarized as being "characterized through activation, localization, and causal ablation", without explaining the specific operational principles. Furthermore, the selection criteria for retaining 300 candidate features from the 10,000 largest activations are not explicitly clarified.

  • W3. Extremely limited experimental scope and singular setting: Real-world emotions and relevant datasets typically contain a rich variety of emotion categories, whereas this paper restricts itself solely to "Joy" and "Sadness," which severely simplifies the complexity of emotion recognition. Additionally, all conclusions are based on a single model (CLIP ViT-L/14-336), a single network layer (Layer 22), and a single MSAE checkpoint, raising doubts about generalizability.

  • W4. Lack of methodological innovation and weak statistical significance: The analytical method primarily relies on a combination of existing interpretability tools, offering limited methodological novelty. More importantly, some core conclusions lack statistical significance (e.g., the confidence intervals cross zero). The final core conclusion—that the model utilizes both specific contexts and cross-carrier generalization—reads more like empirical common sense, lacking clear, substantive inspiration for the field.

Questions for authors
  • Q1: Please provide an Overview section at the beginning of Chapter 3 to clearly explain the overall methodological framework. How do Sections 3.1 to 3.7 (e.g., feature discovery, causal ablation, linear probing) connect and serve the core idea of the paper?

  • Q2: Please elaborate on the extraction and localization principles for facial and contextual visual evidence. Specifically, the text mentions "Facial-expression-associated features are first characterized through activation, localization, and causal ablation". What are the specific metrics and step-by-step procedures used to obtain these features?

  • Q3: During the context candidate feature selection phase, "For each context image, we retain 300 unique candidate features from the 10,000 largest positive patch–feature activations". What are the specific principles and logic for selecting these 300 candidates out of the 10,000 activations?

  • Q4: In what stage of the overall framework do the "causal ablation" and "context selectivity" described in Section 3.4 fit? Additionally, what exactly does the "feature ablation" mentioned in line 190 refer to at the operational level?

  • Q5: Please clarify the notations in Equation (2). What do the superscript CC and the bar (ˉ\bar{}) in ΔˉjC\bar{\Delta}^C_j represent? What is its specific mathematical or logical relationship with Δj(x)\Delta_j(x) defined in Equation (1)?

  • Q6: Please explain in detail the specific meanings of the notations in Equation (3), as well as the step-by-step calculation process of this equation in the Control Analyses.

  • Q7: Existing emotion-related datasets and studies are rarely limited to just two emotions. Why did the authors choose only "Joy" and "Sadness"? Are there plans to expand the analysis to more complex emotions to prevent the conclusions from being overly simplified?

  • Q8: The core conclusion states: "These results show that context-associated behavior at the level of individual sparse features can coexist with emotion-discriminative information that remains linearly accessible across facial and contextual evidence". This seems to validate empirical common sense. What specific, substantive inspirations (take-home message) can this conclusion provide for future multimodal model design, bias mitigation, or internal alignment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.