- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SEFIR, a safety detector for frozen MLLMs. It selects sparse features from the visual encoder, projector, and language model, then combines them using held-out F1 scores. Experiments report gains across several backbones and datasets.
Critical strengths
- Good motivation: using internal representations avoids updating the full backbone for safety detection.
- The experiments cover several models and datasets. SEFIR improves over the supervised probes in Table 6, and the ablations test the main design choices.
Critical weaknesses
- Unclear safety labels: SarcNet (Yue et al., 2024) has sarcasm labels, but Appendix A describes its labels as harmfulness. The paper does not explain how safety labels were obtained, including image and text labels. This makes the SarcNet safety results hard to interpret.
- Unclear gain over prior work: SIREN (Jiao et al., 2026) already uses sparse feature selection and F1-weighted fusion. SEFIR changes pooling and supervision and adds visual and projector features, but there is no direct comparison with SIREN. In Table 7, adding visual and projector features gives only a 1.31-point average gain over language-only detection.
- Limited transfer evidence: Table 3 only tests transfer from Falcon. Without simpler probes trained on the same source data, it is unclear whether SEFIR transfers better than those methods.
- Incomplete efficiency evaluation: Figure 6 reports detector-head FLOPs, excluding pooling and hidden-state access. The paper does not report measured latency or peak memory, so the actual inference cost is unclear.
Questions for authors
- Can you provide the label mappings and explain whether SarcNet (Yue et al., 2024) received new safety annotations?
- Can you compare with SIREN (Jiao et al., 2026) using the same MLLM language states, training data, and prediction head?
- What are the latency and peak memory costs, including hidden-state collection and pooling? The FLOPs in Figure 6 cover only the detector head.
3 comments
Thanks for the rebuttal and response, which addresses most of my concerns. I will update my score accordingly.
Thank you for your response.
Thank you for your feedback.
Q1. ... label mappings ... safety annotations ...
Answer: Sarcasm can convey derogation or abuse, making such uses relevant to safety detection. For example, sample 2506.jpg in the released SARCNet dataset pairs a news screenshot with commentary including "真该死的媒体" ("the media really deserve to die"), an explicit abusive statement within ironic commentary. We include SARCNet as a complementary test of context-dependent image-text understanding. Its separate modality and pair annotations also support SEFIR's modality-specific diagnostic supervision. We retain the original sarcasm labels, mapping image_label, text_label, and multi_label to yv,yt,y, respectively; no new safety annotations were added. We have clarified the SARCNet labels in Appendix A.1 and A.4. Across the other six datasets in Table 1, the mean Macro-F1 scores are 90.25, 90.06, 90.57, and 87.05 for Qwen3-VL-4B, Qwen2-VL-7B, Qwen3-VL-8B, and Llama-3.2-11B, respectively: 14.18, 30.44, 21.21, and 25.54 points above their corresponding Table 1 baselines. SEFIR scores higher than the corresponding Table 1 baseline in each of these 24 backbone-dataset combinations.
Q2. ... SIREN ... language states ...
Answer: SEFIR builds an internal safety-evidence bank from the visual, projector, and language components of frozen MLLMs, combining sparse diagnostic selection with held-out-utility fusion for lightweight detection without backbone realignment. SIREN uses mean pooling, global-label probes, and min-max F1 weighting across language layers. SEFIR instead uses activation-magnitude pooling, image-side targets for visual probes, and text-side targets for projector/language probes; it fuses component evidence with softmax weights derived from truncated probes' held-out Macro-F1 against global labels.
SIREN (Jiao et al., 2026) was developed for text-only LLMs. Its original text-input pipeline omits the image; using image-conditioned MLLM states requires adapting representation extraction and retraining. It was therefore not included as an off-the-shelf multimodal baseline.
We adapted SIREN to frozen Qwen3-VL-4B and Qwen3-VL-8B, retaining its pooling, probing, and weighting choices and using SEFIR's training data and MLP architecture. Training uses fixed SEFIR settings rather than SIREN's hyperparameter search, with validation-based early stopping and calibration. SEFIR scores higher in all eight settings below, averaging a 15.14-point advantage. Entries are Macro-F1 (%); differences are SEFIR minus adaptation.
Q3. ... latency ... peak memory ...
Answer: Figure 6 reports detector-head FLOPs. Appendix A.3, Efficiency Accounting, includes pooling in the total additional arithmetic cost and explains its dependence on image resolution and sequence length. It also identifies hidden-state access and coordinate gathering as memory operations. The distinction between arithmetic cost and runtime efficiency is discussed by Dehghani et al. (2021), cited in the appendix. The additional experiment directly measures the complete detection pipeline on Falcon, comparing latency and peak memory with and without SEFIR under identical conditions (Qwen3-VL-4B, RTX 5090, BF16, batch size 1):
Synchronized timing covers image decoding and preprocessing, backbone prefill, hidden-state collection, pooling, coordinate gathering, fusion, and classification. No representation disk cache is used. Model loading and autoregressive generation are outside the timed path. We have added these measurements to Table 6.
Q4. ... transfer ... source data ...
Answer: We evaluated Falcon-trained SEFIR, Final-Layer Linear Probe, and Naive Hidden-State MLP on four target datasets using Qwen3-VL-4B. All three methods use the same source training data. Target datasets are reserved for evaluation and do not enter model selection or tuning:
With Falcon as the training source, SEFIR's average Macro-F1 exceeds the linear probe by 11.07 points and the MLP by 9.55 points. We have added this comparison to Table 5.