acceptodds
Under review as a conference paper at ICLR 2027

Decodable but Misrouted: Sparse Features Uncover a Readout Gap in Vision-Language Models for Harmful Meme Detection

Abstract

When large vision-language models misclassify harmful memes, the failure may reflect missing internal evidence or an inability to route represented evidence to their outputs. We distinguish these cases in Gemma-3 and Qwen3.5 using sparse autoencoders, role-conditioned probes, causal interventions, and recovery experiments across six harmful content benchmarks, with additional Spanish and Hindi-English code-mixed evaluations. Sparse readouts outperform native prediction on all six primary binary tasks: Qwen averages versus for native macro-F1, residual reconstruction reaches , and Gemma improves from to . These gains measure how accessible the label is to a supervised readout; they do not show that the model's native generation already applies such a decision rule. The most informative token role also varies by task. Under the evaluated scales, Qwen silent-feature ablation is times more probe-sensitive, whereas routed-feature patching on literal yes/no tasks is times more output-sensitive. Native-only threshold calibration explains much, but not all of the gap: on five tasks with matched probe scores, it recovers % of the raw native-to-probe difference, while direct routing adds mean macro-F1 beyond calibrated native scoring. Joint gold-label, probe-KL, and pairwise LoRA supervision improves dedicated FHM prediction, but a gold-only adapter performs better on the shared seven-task mean. A case study of Gemma-3-12B on the Facebook Hateful Memes dataset finds a distributed rank-32 image-prompt interaction, reaching versus native macro-F1. Robustness controls show that the signal is not explained solely by accompanying OCR and depends on paired visual evidence, and that it extends beyond English. In many of the errors we study, the evidence is represented but does not reach the answer; therefore, routing rather than representation alone is a common bottleneck in harmful meme classification.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, vision-language models, sparse autoencoders, harmful meme detection, representation probing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RYE5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates why large vision-language models misclassify harmful memes, distinguishing between failures to represent relevant evidence and failures to route represented evidence into the final prediction. The authors study Gemma-3 and Qwen3.5 across six harmful-content benchmarks, with additional Spanish and Hindi-English code-mixed evaluations. Using sparse autoencoders, supervised probes, and causal interventions, they compare native predictions with sparse-feature readouts, residual reconstruction, and output-routed representations. The authors report that sparse readouts outperform native predictions across all six primary binary tasks. Their intervention experiments suggest that features predictive of harmful content are substantially separated from those controlling the native output, particularly in Qwen.

Critical strengths
  1. The paper goes beyond evaluating classification accuracy by asking whether relevant harmful-content information is internally accessible but fails to influence the model's native prediction.
  2. The authors recognize that decodability does not establish that the native model uses the decoded information. Their comparison of probe-discriminative features with output-routed features, together with feature ablations and patching interventions, attempts to establish a more direct connection between internal representations and model behavior.
  3. The experiments span Gemma-3 and Qwen3.5 and six primary binary classification tasks involving different forms of harmful content.
Critical weaknesses
  1. Direct routing improves performance beyond calibrated native scoring, but the abstract does not establish that this improvement is consistent across all tasks. Similarly, joint gold-label, probe-KL, and pairwise LoRA supervision improves dedicated FHM prediction, while a gold-only adapter performs better on the shared seven-task mean. These results suggest that the proposed mechanistic insights do not yet translate into a consistently superior intervention across settings.
  2. The low-rank image-prompt interaction result is presented as a case study on Gemma-3-12B and the Facebook Hateful Memes dataset. The multilingual evaluations use internal holdouts, and the authors acknowledge that fine-grained classes remain difficult. Therefore, the extent to which the identified routing phenomenon generalizes to larger LVLMs, other cultural contexts, and more complex harmful-content taxonomies remains uncertain.
  3. The authors should clarify whether the routing explanation remains equally compelling across tasks and models after accounting for calibration.
Questions for authors
  1. After matching native and probe-based decision thresholds and calibration procedures across all tasks, how much of the remaining performance gap can be attributed to routing?
  2. Do the observed separation between probe-discriminative and output-routed features and the gains from direct routing persist across more layers, model scales, and harmful-content benchmarks? Which conclusions are supported by both model families, and which rely on the Qwen or Gemma case studies?
  3. Given that gold-only LoRA performs better than joint supervision on the shared seven-task mean, under what conditions does probe-based supervision offer a reliable advantage?
4 comments
AuthorsOpenReview ↗

Response to Reviewer RYE5

We thank the reviewer for all the questions.

CW1, CW3 & Q1. After matching calibration for the native and the probe, are the gap and the routing gain consistent across tasks and models?

Yes. We gave the native head and the probe exactly the same threshold calibration on the same split. In all three model families, the probe ranks examples better on every task, and routing beats calibrated native scoring on every task. The size of the remaining gap differs by task, and we report it per task rather than as a single mean score.

Macro-F1 after the same calibration protocol (30% calibration / 70% reporting); the last column is AUROC, which uses no threshold.

ModelTaskNative (calibrated)Probe (calibrated)Routed, Eq. (6)AUROC native / probe
Gemma-3-4BCrisisHateMM0.7060.8110.8090.78 / 0.89
Gemma-3-4BFHM0.6470.7120.6980.71 / 0.80
Gemma-3-4BMAMI0.7230.7430.7510.82 / 0.84
Gemma-3-4BHarMeme0.7030.8060.8250.80 / 0.90
Gemma-3-4BMMHS150K0.5530.6460.5630.57 / 0.69
Gemma-3-4BMultiOFF0.6150.6460.6630.69 / 0.72
Gemma-3-12BFHM0.6870.7400.7340.74 / 0.83
LLaVA-NeXT-8BFHM0.6330.6920.7140.68 / 0.78
LLaVA-NeXT-8BMAMI0.6820.7940.7910.77 / 0.88
  • Qwen already has this comparison for its five matched tasks (Section 5, Table 8): routing beats calibrated native on all five, and the probe has higher AUROC on all five.
  • Calibration explains a similar share in both families: about 70% of the raw gap in Qwen and about 65% in Gemma. The larger raw gap in Qwen mainly reflects its large zero-shot threshold offset as a base model, not a different mechanism.
  • What remains after calibration is about +0.10 macro-F1 for Qwen, +0.07 for Gemma, and +0.06 to +0.11 for LLaVA. The gap is small where native ranking is already good (e.g., MAMI) and large where it is poor (e.g., CrisisHateMM, HarMeme, MMHS150K). The second case is a ranking failure that no threshold can fix.

In the revision, the abstract will state the number of tasks on which routing beats calibrated native scoring for each model (5/5 Qwen, 6/6 Gemma, 2/2 LLaVA), not only the Qwen mean, and the appendix will include the table above.

CW2 & Q2. Does the effect hold across layers, scales and benchmarks? Which conclusions need both families?

The central findings hold in three model families (Gemma-3, Qwen3.5, LLaVA-NeXT) and four models from 4B to 12B, at several layers, on six binary and two fine-grained benchmarks. Two results are model-specific, and we label them as case studies.

  • A third model family. LLaVA-NeXT-8B (Llama-3 language model, CLIP vision encoder) with its public 131k-feature SAE at layer 24 shows the same pattern (table above). Its native output says "hateful" for 98% of FHM memes and "misogynous" for all MAMI memes. After calibration, the probe still ranks better (AUROC +0.10 and +0.11), and the 20 output-routed features predict the label worse than the 20 discriminative features.
  • Layers. In Gemma-3-4B, probe features are outside the output's Jacobian workspace at layers 22 and 29 and in a crosscoder spanning layers 9--29. In Gemma-3-12B, the same holds at layers 31 and 41. Section 4.1 reports the layer-22 result; we will add the other layers to Appendix G.
  • Fine-grained taxonomies (Appendix E). With the same calibration for both, the probe still ranks classes better: HarMeme 3-class AUROC 0.71 → 0.81; MMHS150K 6-class 0.79 → 0.83. The low absolute scores come from rare classes, not from a missing gap.
  • Languages (Appendix J, Tables 12--13). Qwen shows a large gap on Spanish EXIST and on Hindi-English MultiBully. For Gemma, MultiBully favors the probe on all five holdout splits. Gemma's Spanish EXIST does not: after calibration, Gemma's native output ranks EXIST memes as well as its probe does. We will state this as a limitation rather than claim a multilingual gain for Gemma.

Which conclusions rest on both families, and which on case studies:

  • Both families (and LLaVA): a readout gap that survives matched calibration; routing improves over calibrated native scoring; probe features are separated from the output pathway.
  • Qwen only: clean two-way causal mediation on the native output. In Gemma, the probe side is clean, but the output side is clean only on MMHS150K (Appendix G).
  • Gemma-3-12B FHM only: the rank-32 image–prompt interaction (Section 4.2), already labeled as a case study.
AuthorsOpenReview ↗

Response to Reviewer RYE5 continued...

Q3. Gold-only LoRA beats joint supervision on the shared seven-task mean. When does probe supervision help?

We compared all three on the same validation examples. Probe supervision helps on exactly the tasks that have matched confounder pairs (FHM, MAMI, HarMeme). Where there are no pairs, the probe-KL term pulls the model toward a frozen probe that is weaker than gold-only LoRA, which costs accuracy. One task (MMHS150K 6-class) causes most of the shared-mean drop.

Macro-F1 on identical validation examples (adapter results as in Table 10):

TaskFrozen probe (teacher)Gold-only LoRAJoint LoRAMatched pairs
FHM0.6620.6370.6554,330
MAMI0.7880.8460.86223
HarMeme (3-class)0.4620.4780.50851
CrisisHateMM-A0.8170.8510.8180
CrisisHateMM-B0.2910.3120.3110
MultiOFF0.6000.6540.6371
MMHS150K (6-class)0.2300.2690.189binary only
  • With matched pairs, probe supervision helps. The pairwise loss teaches the model to separate memes that differ only in image or only in text, which is where the gold label alone is weakest. On the dedicated FHM adapter, the effect is largest: 0.714 versus 0.646 for gold-only, and above the frozen teacher itself (0.676).
  • Without pairs, performance regresses. On CrisisHateMM and MultiOFF, gold-only LoRA already beats the frozen probe, so the KL term pulls the model back toward a weaker teacher.
  • MMHS150K has a weak teacher, receives only 0.034 epochs under round-robin sampling (Table 11), and its auxiliary losses are binary while the target has six classes. Without this one task, joint supervision is slightly better than gold-only on the remaining six tasks (0.632 vs. 0.630).

Practical consideration. We suggest using probe supervision for task-dedicated adapters when matched pairs exist, or when the probe beats gold-only LoRA on validation data. Otherwise, use gold-only LoRA, or the inference-time router of Eq. (6), which improves over calibrated native scoring in every model we tested. Our main claim is diagnostic (the evidence is represented but not read out); we do not claim a single intervention that is best everywhere, and Section 5 and Appendix I.2 already report this task dependence.

In the revision, Appendix I.2 will add the table above and this rule, and Section 5 will state it in one sentence.

We hope the answers above resolve all the points raised. We are happy to clarify any other questions that the reviewer may have. We request that the reviewer increase the scores as they deem fit.

Reviewer_RYE5OpenReview ↗

Thank you for the additional experiments

Thank you for the additional experiments and rebuttal. Although these address several specific points, my main concern regarding the performance gap remains. I prefer to maintain my original score.

Thank you for engaging with our rebuttal. To answer Q1 directly: when the native score and the probe receive identical calibration, threshold error accounts for about two-thirds of the raw gap (69.8% Qwen, 65.5% Gemma). The remaining third (+0.10 macro-F1 for Qwen, +0.07 for Gemma) persists. It cannot be removed by any threshold, because the probe ranks examples better (higher AUROC) on every task of Qwen, Gemma-3-4B/12B and LLaVA-NeXT-8B. It is also not explained by extra supervision: when every readout is trained only on the native calibration data, the probe stays +0.10 ahead of calibrated native scoring (Qwen, mean of five tasks). We attribute this residual to the readout. We have added this decomposition to Section 5 (highlighted in the revised PDF).

On interventions, the paper's claim indicates that the evidence is represented but not read out by the native head. We do not claim one intervention is best everywhere. The calibration-only routing beats calibrated native scoring on every task of every model. Probe-distilled LoRA helps on exactly the tasks with matched confounder pairs. The revised Section 5 states these arguments.

Reviewer_6yfUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether errors in visual language models for detecting harmful memes stem from a lack of task-relevant information within the model or from the failure of encoded information to effectively contribute to the final prediction. Combining experiments with sparse autoencoders, supervised probes, feature-level causal interventions, and output routing, the authors analyze the relationship between the decodability of internal label information and the model’s native predictions across multiple datasets. The experiments demonstrate that sparse probes outperform native predictions on all six binary classification tasks, and that probe-based output routing provides additional benefits beyond native threshold calibration.

Critical strengths
  1. Examining whether the internal information contains task-relevant details and whether the model actually uses this information separately has clear research value.
  2. The experiments cover six harmful content detection datasets and two LVLM families, which lends a certain degree of persuasiveness to the findings.
Critical weaknesses
  1. There are still some technical details that require further clarification by the author.
  2. The quality of the figures in the manuscript could be further improved.
Questions for authors
  1. Is the currently observed separation between discriminative features and routed features a true structural property of the model itself, or is it to some extent pre-determined by the feature selection criteria? In a 65536-dimensional SAE space, the inner products of a large number of feature directions with a fixed output direction may naturally be small. If we define silent features as those with large $ |w_j|$ and small $ |a_j|$, and routed features as those with large ∣aj∣|a_j|, then the clear separation between the two sets is not entirely unexpected.
  2. Regarding Eq. 6 in Section 5. Does the current improvement demonstrate that the internal routing bottleneck in LVLM has been corrected, or does it primarily show that incorporating the results of a supervised external classifier into the native logits improves accuracy? While this does not negate the method’s practical effectiveness, I believe there may still be a conceptual distinction between the External Probe Ensemble and the Correction of Internal Native Routing. I suggest the authors provide a more detailed explanation of this aspect.
  3. The authors define the readout gap in Sec. 2.1, Eq. (1). In Sec. 3, Table 2, Qwen’s average macro-F1 improves from 0.432 to 0.740. I would like further clarification: of the current performance difference, how much is attributable to native routing failure, and how much is solely due to the probe receiving additional task-specific supervision? Based on the metrics reported in lines 1696–1697, approximately 70% of the original gap can be recovered through simple native threshold calibration. I hope the authors will further compare other experimental scenarios under equivalent training data conditions.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 6yfU

We thank the reviewer for these questions. If there are any particular figures or sections the reviewer feels we should improve, we would be happy to revise them. The answers to the "questions for authors" are below.

Q1. Is the split between discriminative and routed features a real property of the model, or a result of how the feature sets are selected?

The selection rule does not create the split. The |a_j| < τ filter in Eq. (4) removes no feature, so the two sets are chosen by two independent criteria. We agree that small inner products are expected in high dimensions, so we do not rest the claim on them. The evidence is causal (Fig. 2) and predictive (converse test below), and both show that the two sets do different jobs.

1. The threshold τ does no work. On every Qwen task, none of the top-20 (or top-100) probe features reaches |a_j| ≥ 0.10; only 41 of all 65,536 dictionary features do. So the silent set is simply the top-20 features by probe weight, and the routed set is the top-20 by output alignment. Neither set is defined using the other's criterion.

2. We agree that static geometry is not enough, and we checked this directly. As a positive control, we trained a probe to predict the model's own yes/no answer from the same layer-20 features. This probe is also nearly perpendicular to the yes/no output direction (87°--92° on all tasks). In other words, even the information the model does use is not readable through a direct layer-20 inner product, because 12 more layers sit between layer 20 and the output. The reviewer is therefore right that near-zero a_j is expected. This is why Section 4.1 already states that a_j "is not a full causal effect", and why our claim rests on the next two tests. We will make this explicit in Section 4.1.

3. Causal test (Fig. 2, Table 6). Removing the 20 discriminative features changes the probe far more than the model's answer, and patching the 20 routed features changes the model's answer far more than the probe. This double dissociation is measured through the full forward pass, so it is not affected by dimensionality. It also holds after both scores are rescaled to unit standard deviation: knockout still moves the probe 3--12.3x more than the output, and patching routed features moves the output 47--151x more than the probe.

4. Converse test: do the routed features predict the label? With the same data and the same classifier, a model that sees only the 20 routed features is clearly worse than one that sees only the 20 discriminative features (AUROC):

Qwen FHMQwen CrisisHateMMQwen MAMIQwen HarMemeLLaVA-NeXT FHMLLaVA-NeXT MAMI
20 routed features0.630.750.710.660.610.75
20 discriminative features0.740.860.870.930.680.82

The features that steer the answer carry much less label information than the features the probe uses. This is the separation we report, and it replicates in a third model family (LLaVA-NeXT-8B).

In the revision, Section 4.1 will state that static alignment is only a screening statistic, report that τ removes no feature, and add the positive control and the converse test.

Q2. Is Eq. (6) a fix of internal routing, or an external ensemble?

We agree with the distinction. Eq. (6) is an output-level bypass (an ensemble of the native score with an external probe), instead of a repair of internal routing. Its purpose is diagnostic. The internal fix is probe-distilled LoRA, which uses no probe or SAE at test time.

  • What Eq. (6) shows. The probe's information is usable by the model's decision boundary with a single weight fitted on calibration data. This improves over calibrated native scoring (Section 5, Table 8). It does not show that the model's own layers have been fixed, and we will no longer describe it that way.
  • Fixing the routing inside the forward pass is hard. Re-inserting sparse features at layer 20 (the hook) recovers only 17.6% of the gap (Table 2), and steering the top probe features one at a time or jointly gave no gain (we will add these results to Appendix I). Information placed inside layer 20 does not reach the output through the remaining layers.
  • The internal fix: probe-distilled LoRA (Section 5, Appendix I.2). It changes the model's own weights and needs no SAE or probe at inference. On FHM, it reaches 0.714 macro-F1, versus 0.648 for the base model and 0.646 for a LoRA trained on the same data with gold labels only. The distilled model also beats its own frozen probe teacher (0.676 on the same test set), so it is not simply copying an external classifier.

In the revision, we will rename Eq. (6) to "output-level probe-logit fusion (external bypass)" and separate it clearly from the internal fix.

AuthorsOpenReview ↗

Response to Reviewer 6yfU continued...

Q3. How much of the gap is native routing failure, and how much is extra supervision for the probe?

We separated the two with controls in both directions. Threshold calibration explains about 70% of the raw Qwen gap, as the paper reports. Even with the same calibration and the same small training set as the native model, the probe still leaves a clear gap on three of the four tasks with enough data. Giving the native model full supervision (gold-label LoRA) does not close the gap on the task where the probe's edge depended on data (FHM).

Qwen3.5-9B, macro-F1 on the same 70% reporting split:

TaskNative, calibrated thresholdProbe, same calibrationProbe trained only on the 30% calibration splitAUROC native / probe
FHM0.7090.7330.7030.76 / 0.81
CrisisHateMM0.6190.8610.8320.67 / 0.94
MAMI0.7600.8040.8070.84 / 0.89
HarMeme0.6500.8240.8440.73 / 0.91
  1. Same calibration for both. The calibrated probe beats the calibrated native model on all five tasks. It also ranks examples better on all five tasks (AUROC, which uses no threshold).
  2. Same training data for both. When the probe sees only the small calibration split used to tune the native threshold, it keeps a large lead on CrisisHateMM, MAMI and HarMeme. On FHM, the lead disappears, so we agree that on FHM the probe's advantage depends on its training data. A supervised logistic regression on the native model's full label-score vector, which gives the native output the same supervision, does not close the gap either. (MultiOFF has only 45 calibration examples, too few for any probe.)
  3. Full supervision for the native model (Section 5, Appendix I.2). Fine-tuning the model's language layers on the full FHM training set with gold labels gives 0.646, below the base model and below the frozen-model probe (0.703). So the FHM gap is not closed by giving the native pathway more data, which points back to how that pathway reads out the information.

In the revision, Section 5 will include this decomposition (calibration, equal data, gold-only LoRA) alongside Table 8.

We hope the answers above resolve all the points raised. We are happy to clarify any other questions that the reviewer may have. We request the reviewer to increase the scores as they deem fit.

Reviewer_LcTfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies failure modes in VLMs employed as harmful meme classifiers, using mechanistic interpretability techniques to distinguish between missing internal evidence and information routing failures. SAE feature vectors and reconstructions are used to develop four interventions, namely, baseline VLM scores, sparse probes, residual reconstruction hooks, and direct logit routing. Experiments demonstrate that VLM classification failure in this setting is primarily driven by routing failures rather than missing internal evidence, with probing outperforming baseline VLM classification and causal validations supporting this finding.

Critical strengths
  • The paper addresses an important problem (VLM-based classification failure analysis) in an important domain (harmful content moderation)
Critical weaknesses
  • Many of the paper's claims and experiments rest on the finding that classification can be better performed using sparse intermediate feature vectors than with the VLM's output itself. However, because the probes trained on these vectors are trained directly on splits of the assessed benchmarks and the VLMs are not, this is not a fair comparison. The authors note this limitation but do not make sufficient effort to correct it (e.g. fine-tuning the VLM as well).
  • Many of the paper's findings do not seem to generalize across the two tested models, and, in general, evaluation on more than two models would be preferred to confirm the robustness of the results.
  • The paper does not cite a highly relevant line of work on VQA image-grounding [1][2]. These prior works should be discussed to clarify the intended contribution of this work. The methods employed by these works may also be helpful to resolving the first concern.
  • The presentation is generally unclear and difficult to parse

[1] Fu, S., Bonnen, T., Guillory, D., & Darrell, T. (2025). Hidden in plain sight: Vlms overlook their visual representations. arXiv preprint arXiv:2506.08008.

[2] Liu, Z., Chen, Z., Liu, H., Luo, C., Tang, X., Wang, S., ... & Tong, H. (2026, April). Seeing but not believing: Probing the disconnect between visual attention and answer correctness in vlms. In International Conference on Learning Representations (Vol. 2026, pp. 72849-72876).

Questions for authors

See the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer LcTf

We thank the reviewer for the feedback and the pointers to related work.

W1. Comparing a trained probe with an untrained VLM is not fair. Did you try fine-tuning the VLM?

Yes. In the paper, we already fine-tuned the VLM on the same training data (gold-label LoRA on all 34 language layers; Section 5, Appendix I.2, Table 10) and calibrated the native threshold (Table 8). We have also added the two remaining controls: identical calibration for both, and identical (small) training data for both. The gap survives on most tasks under each control.

  1. Fine-tuning the VLM on the same labels (Table 10). On FHM, gold-label LoRA reaches 0.646, below the base model (0.648) and below the frozen-model probe (0.703). On easier tasks (MAMI, CrisisHateMM, MultiOFF), gold-only LoRA does catch up. This is consistent with our claim: the vision encoder is never tuned, so when tuning the language layers fixes the output, the evidence was already in the frozen representation and only the readout was missing.
  2. Identical calibration for both. With the same threshold calibration on the same split, the probe still beats the native output on every task of Qwen (5/5), Gemma-3-4B (6/6), Gemma-3-12B (FHM) and a third model, LLaVA-NeXT-8B (2/2). It also ranks examples better (AUROC) in every case. The remaining gap is about +0.10 macro-F1 for Qwen and +0.07 for Gemma.
  3. Identical training data for both. When the Qwen probe sees only the small calibration split used to tune the native threshold, it still clearly beats the calibrated native model on CrisisHateMM (0.83 vs. 0.62), HarMeme (0.84 vs. 0.65) and MAMI (0.81 vs. 0.76). On FHM, it does not. There, the probe's edge needs its training data, but giving the native model the same data through LoRA does not help either.

We also note that the paper never presents the raw gap as a fair classifier comparison: Sections 2.1 and 3 state that it measures supervised accessibility, and the controls above are the like-for-like tests. In the revision, Section 3 will show these controls next to Table 2.

W2. Do the findings generalize beyond two models?

We evaluated a third model family, and the core findings replicate. Two narrower findings are model-specific, and the paper presents them as case studies.

  • LLaVA-NeXT-8B (Llama-3 language model, CLIP ViT-L vision encoder, public 131k SAE at layer 24) shows the same pattern. The native output is almost constant: "hateful" for 98% of FHM memes and "misogynous" for all MAMI memes. After identical calibration, the probe beats the native model (FHM 0.69 vs. 0.63; MAMI 0.79 vs. 0.68) and ranks better (AUROC +0.10 and +0.11). The output-routed features predict the label worse than the discriminative ones. The evidence now covers three families and four models (4B, 8B, 9B, 12B).
  • What holds everywhere: a readout gap that survives matched calibration, routing gains over calibrated native scoring, and the separation of probe features from output-controlling features.
  • What is model-specific: the two-way causal mediation on the native output is clean in Qwen but mixed in Gemma (Appendix G), and the rank-32 image-prompt interaction is a Gemma-3-12B FHM case study (Section 4.2). We label both as such.
AuthorsOpenReview ↗

Response to Reviewer LcTf continued...

W3. Missing related work: Fu et al. (2025) and Liu et al. (2026)

Both papers are relevant, and we will cite and discuss them in Section 2.1 and Appendix A. They support our framing, but they study a different question, and neither explains our results.

Fu et al. (2025) show that VLMs perform worse than a supervised readout of their vision encoder on vision-only tasks (depth, correspondence), that VLM answers resemble their "blind" (no-image) answers, and that tuning the language model helps most.

  • Same diagnosis, different location. Harmful memes cannot be solved from the image alone; the meaning comes from image and text together. Our signal is therefore read from the language model's residual stream after the two are combined, not from the vision encoder. We also go one step further than "the LLM is the bottleneck": we identify which sparse features carry the label and show causally that they do not drive the answer (Section 4).
  • Their remedy does not fully work in our setting. Tuning the language model with gold labels, their main fix, is our gold-only LoRA control. On FHM it does not help (0.646 vs. base 0.648), whereas distilling our probe into the same layers does (0.714, Section 5).
  • The same "blind" behavior appears in our data. Our blank-image audit (Appendix J, Tables 14--15) shows it: on MAMI, Gemma's native answer is unchanged for 96% of memes when the image is replaced by a grey image.
  • On fairness. Fu et al. use the same design raised in W1: a supervised readout compared with a zero-shot VLM. That design is standard in this line of work, and we add the controls described under W1.

Liu et al. (2026) show that deeper VLM layers attend to the right image regions even when the answer is wrong, and they improve answers by masking attention at inference time. We tested whether attention explains our errors. In Gemma-3-4B (FHM, MAMI, HarMeme), the answer token gives the same attention to image and meme-text tokens whether the native answer is right or wrong (difference ≤ 0.001). So in our setting the failure happens after the evidence is attended to: the attended evidence is written into features that the output does not read (Section 4). This agrees with their "seeing but not believing", and explains why attention masking alone would not be enough here.

In the revision, Section 2.1 and Appendix A will add both papers with the points above, and Appendix J will add the attention comparison.

W4. Presentation

We will (i) add a short notation box in Section 2, (ii) move the fair-comparison controls next to Table 2, and (iii) replace dense symbols in the running text with plain names. If there are particular sections the reviewer found hardest to follow, we would be grateful to know, and we will rewrite those first.

We hope the answers above resolve all the points raised. We are happy to clarify any other questions that the reviewer may have. We request the reviewer to increase the scores as they deem fit.

I thank the authors for their detailed response. However, my main concerns regarding the soundness of many of the paper's claims have not been adequately addressed. Accordingly, I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.