acceptodds
Withdrawn from ICLR 2027

What Output Scores Miss: Auditing Joint Safety in Frozen Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language safety depends on interpreting images and text in context, as their combination can introduce risks absent from either modality alone. Recent benchmarks reveal that models can miss these compositional risks even when their unimodal judgments are correct. Evaluations based on final outputs identify such failures, but leave unresolved whether useful safety evidence remains recoverable from frozen internal states beyond the model’s output scores. We find that joint states and their contrasts with unimodal states provide additional label information, allowing some joint errors to be corrected without updating the backbone. Building on this observation, we introduce Joint Safety Calibration (JSC), a supervised offline framework for auditing this residual joint-safety evidence. JSC combines the same model’s output scores with frozen-state features to assess their incremental predictive value under a controlled comparison. Cross-model experiments on native three-class safety assessment and controlled analyses show improved joint prediction and more effective repair of joint errors. The resulting audit clarifies what information remains recoverable after joint prediction failures and identifies opportunities for improving joint safety judgments through frozen-state readouts.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, multimodal safety, representation probing, hidden-state interpretability, model auditing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mUjLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper

  • Investigates the failure pattern in safety classification from VLMs that the model may correctly classify each individual modality but miss the joint context.
  • It extracts activation-based features from the individual and joint forward pass and then fits a multinomial classifier on the normalized features
    • The features include the final-layer logits, probabilities and confidence (7d) extracted for each view
    • as well as features from dot products with linear directions in intermediate layer activations that correspond to average activations on missed Unsafe and undetected Unsafe errors in the feature space.
  • It shows classification results substantially better than the base model and slightly better than with just the 21-d logits features.
  • It compares to extensive baselines and controls that make a reasonable case for the value of the learned features from the hidden layers.
Critical strengths
  • The paper demonstrates substantial performance improvements in joint classification performance compared to the base-model’s joint classification. It also shows that using information from the hidden layers provided a small but robust additional gain.
Critical weaknesses
  • There are two important baselines missing:
    • a) how much of the gain is explained by just re-calibrating the logit features from the joint view? Could the performance improvement be just due to calibrating the zero-shot LLM classification?
    • b) for the hidden-layer features one important baseline seems missing: Just a strongly regularized linear model from all the hidden features for the 3 views directly towards predicting the label, trained on train_a and train_b. This is how probe-based safety classification is typically done in practice.
  • No deeper analysis in terms of where the gains come from in the features. The majority of the gains come from the logits across the 3 views and are read-out by a linear model. It would be straightforward to look into the mechanism that explains which features exactly lead to the gain in performance
  • No deeper analysis of what kind of datapoints the model fixes and on what datapoints the model regresses. For safety classification not all errors have the same cost, so looking at what the new model trades-off would be important
Questions for authors
  • The cross-benchmark binary results seems lacking critical details. Where is the training data coming from for the non-VLSU results? And what do you do on benchmarks that don’t have image-only and text-only annotations? How do you fit the hidden-layer features on those?
Reviewer_2Z2bOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The question is whether a frozen VLM’s hidden states know more about joint safety than its own output scores do. The authors take VLSU, run the model three times per example (image only, text only, both), and read off the Safe/Borderline/Unsafe token logits. On top of that they add a handful of features from cached hidden states, mostly projections onto directions learned from where the base model got things wrong, and train a linear classifier to predict the joint label. Against a classifier that only sees the logits, this gives about 2.6 Macro-F1 on three-class VLSU across four open models, less on binary benchmarks. Nothing in the paper involves a generated response; it is label prediction from logits throughout.

Critical strengths

The controls are done properly: same output anchor for every readout, held-out splits, random directions and shuffled features as references. Within the question it asks, the comparison is fair.

Critical weaknesses
  • The paper calls itself a joint-safety audit, but it never measures safety. The way the field measures safety is simple: give the model inputs it should refuse, let it answer, and have a judge score the answers. This paper does not generate a single response. It predicts a dataset label from label-token logits and treats a higher Macro-F1 on that as safety evidence. The ethics statement admits the two are not the same thing, and then the paper proceeds as if they were.
  • The right baseline is missing even for the label task. If I want to know whether an image-text pair is unsafe, I would ask GPT-4o, Gemini, or Llama Guard. The paper never runs any of them on its own splits; Table 11 copies numbers from a different protocol and says itself they are not comparable. So I cannot tell whether JSC is useful or just better than a weak internal reference.
  • The obvious probe is missing too. The simplest thing to try is to feed the raw last-layer state into the same classifier. The authors say in Appendix G that they did not run it. Without that number the direction-estimation machinery in Section 3.3 has no demonstrated value. What is shown is that random directions already give about a third of the gain, and that JSC gets extra unimodal labels the baselines do not, so what is left cannot be attributed to the hidden states.
  • The setup is hard to see as practical. JSC needs white-box access, three forward passes per input, cached states, and labeled data for both unimodal and joint views. Anyone with all of that would fine-tune the model or train a dedicated guard classifier. The paper compares to neither, and does not explain why one would accept these costs for a 2.6-point gain in label prediction with no demonstrated effect on what the model actually outputs.
Questions for authors
  • Run a raw hidden-state probe (last-layer state into the same classifier, no direction estimation) under your protocol. Does JSC beat it?
  • Run at least one external judge (GPT-4o, Gemini, or Llama Guard) on your five test splits and report it next to JSC.
  • Is there any result on generated responses, e.g. attack success rate or judge-rated harmfulness, showing that better label prediction leads to safer outputs? If not, the title and abstract should be narrowed to label prediction.
Reviewer_7wCxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The author formulates an output-anchored evaluation of joint safety introduces JSC, which combines compact supervised state features with a shared output anchor and separates coordinate estimation, readout fitting, configuration selection, and held-out evaluation. Across the evaluated backbones, JSC improves held-out joint-label prediction over the tested readout baselines. Experiments on four VLM backbones show consistent improvements

Critical strengths

1、An interesting research angle that examine whether safety-relevant information remains recoverable from frozen hidden representations when output-level safety judgments fail. This is an interesting question for understanding multimodal safety. 2、The comprehensive design of the experiments. The author separates coordinate estimation, classifier fitting, development selection, and held-out evaluation. The results from different aspects strengthens the empirical analysis. 3、JSC get significant improvments on all evaluated backbones. The sufficient comparison and analysis provide useful evidence that the proposed supervised representations contribute to prediction.

Critical weaknesses

1、 Limited methodological novelty. The paper does not introduce a new representation-learning mechanism, optimization framework, or theoretical insight into multimodal safety. The author should provide clear technical contribution of JSC from existing linear probing, representation engineering, and supervised safety classification approaches. 2、Missing Comparisons with stronger hidden-state probing baselines. Although the author carefully design the experiments, without these comparisons, it is difficult to assess whether the proposed method provides a meaningful methodological advantage. 3、Generalization analysis and Interpretation studies could be improved. The paper shows that supervised hidden-state features is effective but the rational or learned feature behind the method that how they relate to multimodal safety reasoning is still needed to be further improved.

Questions for authors

What are the costs associated with JSC, please provide detailed infos? How does JSC compare with standard linear probes trained on raw or PCA-reduced hidden states under matched supervision and parameter budgets? Can the author provide more results regarding to generalization capablity across different safety distribution, model configurations ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.