- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates why large vision-language models misclassify harmful memes, distinguishing between failures to represent relevant evidence and failures to route represented evidence into the final prediction. The authors study Gemma-3 and Qwen3.5 across six harmful-content benchmarks, with additional Spanish and Hindi-English code-mixed evaluations. Using sparse autoencoders, supervised probes, and causal interventions, they compare native predictions with sparse-feature readouts, residual reconstruction, and output-routed representations. The authors report that sparse readouts outperform native predictions across all six primary binary tasks. Their intervention experiments suggest that features predictive of harmful content are substantially separated from those controlling the native output, particularly in Qwen.
Critical strengths
- The paper goes beyond evaluating classification accuracy by asking whether relevant harmful-content information is internally accessible but fails to influence the model's native prediction.
- The authors recognize that decodability does not establish that the native model uses the decoded information. Their comparison of probe-discriminative features with output-routed features, together with feature ablations and patching interventions, attempts to establish a more direct connection between internal representations and model behavior.
- The experiments span Gemma-3 and Qwen3.5 and six primary binary classification tasks involving different forms of harmful content.
Critical weaknesses
- Direct routing improves performance beyond calibrated native scoring, but the abstract does not establish that this improvement is consistent across all tasks. Similarly, joint gold-label, probe-KL, and pairwise LoRA supervision improves dedicated FHM prediction, while a gold-only adapter performs better on the shared seven-task mean. These results suggest that the proposed mechanistic insights do not yet translate into a consistently superior intervention across settings.
- The low-rank image-prompt interaction result is presented as a case study on Gemma-3-12B and the Facebook Hateful Memes dataset. The multilingual evaluations use internal holdouts, and the authors acknowledge that fine-grained classes remain difficult. Therefore, the extent to which the identified routing phenomenon generalizes to larger LVLMs, other cultural contexts, and more complex harmful-content taxonomies remains uncertain.
- The authors should clarify whether the routing explanation remains equally compelling across tasks and models after accounting for calibration.
Questions for authors
- After matching native and probe-based decision thresholds and calibration procedures across all tasks, how much of the remaining performance gap can be attributed to routing?
- Do the observed separation between probe-discriminative and output-routed features and the gains from direct routing persist across more layers, model scales, and harmful-content benchmarks? Which conclusions are supported by both model families, and which rely on the Qwen or Gemma case studies?
- Given that gold-only LoRA performs better than joint supervision on the shared seven-task mean, under what conditions does probe-based supervision offer a reliable advantage?
4 comments
Response to Reviewer RYE5 continued...
Q3. Gold-only LoRA beats joint supervision on the shared seven-task mean. When does probe supervision help?
We compared all three on the same validation examples. Probe supervision helps on exactly the tasks that have matched confounder pairs (FHM, MAMI, HarMeme). Where there are no pairs, the probe-KL term pulls the model toward a frozen probe that is weaker than gold-only LoRA, which costs accuracy. One task (MMHS150K 6-class) causes most of the shared-mean drop.
Macro-F1 on identical validation examples (adapter results as in Table 10):
| Task | Frozen probe (teacher) | Gold-only LoRA | Joint LoRA | Matched pairs |
|---|---|---|---|---|
| FHM | 0.662 | 0.637 | 0.655 | 4,330 |
| MAMI | 0.788 | 0.846 | 0.862 | 23 |
| HarMeme (3-class) | 0.462 | 0.478 | 0.508 | 51 |
| CrisisHateMM-A | 0.817 | 0.851 | 0.818 | 0 |
| CrisisHateMM-B | 0.291 | 0.312 | 0.311 | 0 |
| MultiOFF | 0.600 | 0.654 | 0.637 | 1 |
| MMHS150K (6-class) | 0.230 | 0.269 | 0.189 | binary only |
- With matched pairs, probe supervision helps. The pairwise loss teaches the model to separate memes that differ only in image or only in text, which is where the gold label alone is weakest. On the dedicated FHM adapter, the effect is largest: 0.714 versus 0.646 for gold-only, and above the frozen teacher itself (0.676).
- Without pairs, performance regresses. On CrisisHateMM and MultiOFF, gold-only LoRA already beats the frozen probe, so the KL term pulls the model back toward a weaker teacher.
- MMHS150K has a weak teacher, receives only 0.034 epochs under round-robin sampling (Table 11), and its auxiliary losses are binary while the target has six classes. Without this one task, joint supervision is slightly better than gold-only on the remaining six tasks (0.632 vs. 0.630).
Practical consideration. We suggest using probe supervision for task-dedicated adapters when matched pairs exist, or when the probe beats gold-only LoRA on validation data. Otherwise, use gold-only LoRA, or the inference-time router of Eq. (6), which improves over calibrated native scoring in every model we tested. Our main claim is diagnostic (the evidence is represented but not read out); we do not claim a single intervention that is best everywhere, and Section 5 and Appendix I.2 already report this task dependence.
In the revision, Appendix I.2 will add the table above and this rule, and Section 5 will state it in one sentence.
We hope the answers above resolve all the points raised. We are happy to clarify any other questions that the reviewer may have. We request that the reviewer increase the scores as they deem fit.
Thank you for the additional experiments
Thank you for the additional experiments and rebuttal. Although these address several specific points, my main concern regarding the performance gap remains. I prefer to maintain my original score.
Thank you for engaging with our rebuttal. To answer Q1 directly: when the native score and the probe receive identical calibration, threshold error accounts for about two-thirds of the raw gap (69.8% Qwen, 65.5% Gemma). The remaining third (+0.10 macro-F1 for Qwen, +0.07 for Gemma) persists. It cannot be removed by any threshold, because the probe ranks examples better (higher AUROC) on every task of Qwen, Gemma-3-4B/12B and LLaVA-NeXT-8B. It is also not explained by extra supervision: when every readout is trained only on the native calibration data, the probe stays +0.10 ahead of calibrated native scoring (Qwen, mean of five tasks). We attribute this residual to the readout. We have added this decomposition to Section 5 (highlighted in the revised PDF).
On interventions, the paper's claim indicates that the evidence is represented but not read out by the native head. We do not claim one intervention is best everywhere. The calibration-only routing beats calibrated native scoring on every task of every model. Probe-distilled LoRA helps on exactly the tasks with matched confounder pairs. The revised Section 5 states these arguments.
Response to Reviewer RYE5
We thank the reviewer for all the questions.
CW1, CW3 & Q1. After matching calibration for the native and the probe, are the gap and the routing gain consistent across tasks and models?
Yes. We gave the native head and the probe exactly the same threshold calibration on the same split. In all three model families, the probe ranks examples better on every task, and routing beats calibrated native scoring on every task. The size of the remaining gap differs by task, and we report it per task rather than as a single mean score.
Macro-F1 after the same calibration protocol (30% calibration / 70% reporting); the last column is AUROC, which uses no threshold.
In the revision, the abstract will state the number of tasks on which routing beats calibrated native scoring for each model (5/5 Qwen, 6/6 Gemma, 2/2 LLaVA), not only the Qwen mean, and the appendix will include the table above.
CW2 & Q2. Does the effect hold across layers, scales and benchmarks? Which conclusions need both families?
The central findings hold in three model families (Gemma-3, Qwen3.5, LLaVA-NeXT) and four models from 4B to 12B, at several layers, on six binary and two fine-grained benchmarks. Two results are model-specific, and we label them as case studies.
Which conclusions rest on both families, and which on case studies: