acceptodds
Withdrawn from ICLR 2027

Non-Intrusive Safety Alignment for Vision-Language Models via Automated Output Rewriting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) remain vulnerable to multimodal jailbreaks. The usual remedy, safety fine-tuning of the model's own parameters, is unavailable for black-box models and can degrade general capability; post-hoc classifiers avoid both problems but answer every violation with the same fixed refusal. We propose NIRA (Non-Intrusive Rewriting Alignment), which places a safety rewriter at the output terminal of a frozen target VLM. The rewriter reads the image, the query and the target's reply; it returns a safe reply unchanged and replaces a harmful one with a refusal written for that request. Its training data come from a fully automated pipeline. We probe the target with multimodal jailbreaks, judge the replies with a guard model, and expand every successful attack with a modality-aware, two-track augmentation engine that re-instantiates the attack's evasion principle as new text and as newly synthesized images. The target's own safe replies, kept as identity pairs, teach the rewriter when not to intervene; trained without them, it refuses nearly every benign query. On JailBreakV-28K against a 30B target, NIRA lowers the attack success rate from 33.2% to 0.7%, and its answer is judged no worse than the undefended one on 88.7% of benign queries. The same rewriter remains effective on four held-out attack benchmarks, and on all five attack sets its only failures are misses: it never turns a reply the target already answered safely into a harmful one, and none of its interventions on a harmful reply leaves it harmful. Applied unchanged to a second target from a different model family, the pipeline yields a rewriter with the same properties.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Safety, Vision-Language Model, Multimodal Safety, Jailbreak Defense, Data Augmentation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yB7LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NIRA, a framework for improving the safety of black-box VLMs without modifying their parameters. It uses an additional VLM to rewrite harmful responses while preserving safe ones. The authors build an automated training pipeline using jailbreak detection, text and image augmentation, and a 235B teacher model to generate safe responses. An 8B rewriter is then trained using LoRA. Experiments on several benchmarks show that NIRA can reduce attack success rates while retaining much of the model's usefulness on benign queries.

Critical strengths
  1. The paper addresses a practical problem. The method does not require access to the target VLM's weights, making it potentially useful for closed-source models.

  2. The experiments cover several attack benchmarks and two target models. The ablation on Safe-to-Safe pairs is useful, as it shows their importance in reducing over-refusal.

Critical weaknesses
  1. The methodological novelty is limited. The approach mainly combines existing techniques, such as data augmentation, teacher-generated supervision, and LoRA fine-tuning. The intervention gate is also learned from Safe-to-Safe training pairs rather than designed as a separate mechanism. I am not sure what the main technical contribution is beyond combining these components.

  2. The contribution of the proposed augmentation is not entirely clear. The method uses a large 235B teacher to generate safe training responses, but there is no comparison with smaller teachers. Also, Table 3 shows that the model without augmentation already performs quite well. It is difficult to tell how much of the improvement comes from the proposed augmentation rather than the teacher-generated training data.

  3. I have some concerns about the evaluation and practical cost. The safety results rely on a single automated evaluator, and the zero-regression claim is not independently verified. The gate recall on SIUO is only 40%, and NIRA also increases over-refusal on MOSSBench. In addition, the method requires several large models for data generation and an extra 8B model during inference, but the overall cost is not fully evaluated.

Questions for authors
  1. What is the main technical difference between NIRA and existing safety rewriting methods? Could the authors better explain the novelty of the proposed framework?

  2. How much does the performance depend on the 235B teacher? Could the authors compare different teacher sizes and further clarify the benefit of the two-track augmentation?

  3. Could the authors evaluate NIRA using another safety evaluator or human annotations to verify whether the zero-regression results still hold?

Reviewer_NcAAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose NIRA, a non-intrusive alignment method for MLLMs. The method first probes a black-box target model with jailbreak attacks and judges whether its responses are harmful. It then expands the successful attacks along two tracks, text and image, and probes the target once more to enlarge the set of harmful samples. A teacher model then writes a tailored safe response for each harmful response, while the target model’s originally safe responses are kept as identity samples to be output unchanged. Together, these are used to train a lightweight rewriter placed at the output end, which lets safe outputs pass through and rewrites harmful ones, achieving non-intrusive alignment.

Critical strengths
  1. The problem is clearly positioned. The setting is black-box, does not modify the target model’s parameters, and is plug-and-play, which is genuinely valuable for deployers of closed-source APIs.

  2. The experiments are extensive, with four held-out sets, two target models, and a white-box SFT reference.

  3. The paper is well-written and easy to follow.

Critical weaknesses
  1. The same model serves as both the evaluator and the training data filter, which may lead to overfitting. Llama Guard 4 plays three roles: labeling the training data, filtering the responses generated by the teacher, and computing ASR at test time. The paper does not validate the results with a second independent judge, nor does it use human annotation to check the judge’s accuracy. Harmful content that Llama Guard misses is invisible in all metrics. In addition, the authors should use a different guard, such as LlavaGuard or ShieldGemma, as a filtering baseline. The absence of this comparison is critical.

  2. NIRA lacks evaluation against adaptive attacks. The rewriter is itself a VLM that reads attacker-controllable text, so it should at least be tested against adaptive attacks, for example, prompt injections targeting the rewriter embedded in the response, or encoded, obfuscated, or low-resource-language outputs.

  3. The benign evaluation is too narrow. It uses only 300 samples from MOSSBench, which are edge cases specifically designed to trigger over-refusal. The false-trigger rate is not measured on standard benign benchmarks such as MMBench or MM-Vet.

  4. The novelty is relatively incremental. The architecture of detection plus rewriting at the output end has precedents. The main differences are threefold: the rewriter sees the image, the data come from the target model’s own failures, and the gate and the rewriting are learned jointly. However, the paper does not include controls that separate these contributions, for example, training the MLLM-Protector architecture on NIRA’s data, or letting NIRA see only the text without the image.

  5. Deployment cost is not reported. The rewriter must regenerate the entire response token by token, which may double latency and prevent streaming output. Moreover, every new target model requires re-probing and calling the 235B teacher model before training. This cost is not compared with white-box SFT.

Questions for authors
  1. The training data labels and the test-time evaluation rely on the same Llama Guard. If different judge models were used for training and evaluation, would the results still hold?

  2. Can the rewriter withstand adaptive attacks? For example, having the target model embed prompt injections targeting the rewriter in its response, or produce encoded, obfuscated, or low-resource-language outputs.

  3. The rewriter does not keep the useful content and remove only the harmful part. Compared with an ordinary safety classifier plus a fixed refusal, where exactly is it stronger in terms of safety?

  4. What is the rewriter’s false-trigger rate in benign settings on MMBench or MM-Vet?

  5. On FigStep and SIUO, the trained rewriter actually performs worse than the untrained one. Why is that?

  6. The rewriter has to regenerate the whole response. How much slower does this make the response? What happens when the response is very long or needs to be streamed? Since responses are truncated at 512 tokens, could harmful content later in the response be missed?

Reviewer_SZbYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles the problem of multimodal jailbreak attacks on vision-language models without modifying the target model itself. It proposes NIRA, a non-intrusive safety rewriter that post-processes VLM responses, preserving benign answers while rewriting harmful ones into request-specific refusals. NIRA is trained through an automated pipeline that combines jailbreak probing, safety judging, and modality-aware attack augmentation.

Critical strengths

None

Critical weaknesses

W1. Limited generality of the trained rewriter.

  • A key motivation of the paper is to avoid the high retraining cost of traditional intrinsic alignment methods, as explicitly highlighted in Figure 1. However, the paper does not demonstrate whether a trained rewriter can generalize across different target VLMs. For PIXTRAL-12B, the authors again construct target-specific training data and train a separate specialized rewriter. If a new rewriter must be trained for each target model, this substantially weakens the claimed advantage over retraining-based alignment methods and should be discussed more carefully.

  • The proposed pipeline also depends heavily on several auxiliary models, particularly Llama-Guard-4 for safe/unsafe labeling. Since the quality of these labels directly determines the training data and, consequently, the resulting rewriter, it is unclear how robust the method is to the choice or quality of the guard model. Similar concerns apply to other components, such as the teacher model used to generate rewriting targets and the image generation/editing models used for augmentation. Ablations or sensitivity analyses across these components would be important to establish the generality and robustness of the proposed pipeline.

Questions for authors

I encourage the authors to carefully address the critical concerns raised above during the rebuttal phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.
Non-Intrusive Safety Alignment for Vision-Language Models via Automated Output Rewriting | acceptodds