- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a process reward model framework for multimodal safety, comprising six safety label categories, an automated step-by-step annotation pipeline, and a human-annotated benchmark, ProSafeBench. Its training method, ASPIRE, combines uncertainty-guided step selection with HSIC regularization and uses step-level rewards to guide downstream policy optimization.
Critical strengths
- The six label categories account for harmful content, over-refusal, and logical issues, providing more fine-grained supervision than binary classification.
- ProSafeBench supports step-level safety assessment and identification of the first unsafe step, making it a valuable evaluation resource.
- The experiments evaluate not only the reward model but also demonstrate a reduction in downstream jailbreak attack success rates while maintaining comparable general multimodal capabilities.
Critical weaknesses
- Why are 16 classification heads used? Given the same inputs and training procedure, how do differences between the classification heads arise and manifest?
- Why is the presence of harmful content in intermediate response steps considered unacceptable? Allowing the model to explore an incorrect direction, recognize the problem, and backtrack may itself help it arrive at a safe answer.
- Please compare performance with other safety alignment methods, such as SPA-VL, mm-rlhf, and safeguard. In addition, please report results on more datasets, such as mm-safetybench, vlguard, and mssbench, and include the false refusal rate.
Questions for authors
See weaknesses.
5 comments
Part 2
Thank you for suggesting that we compare different alignment policies on additional safety benchmarks and examine whether models incorrectly refuse safe requests.
weakness 3: How Does ASPIRE Compare with Other Alignment Policies in Terms of Safety and False Refusal Rates on Additional Benchmarks?
We added comparisons on MM-SafetyBench, VLGuard, and MSSBench among the base policy, policies aligned using SPA-VL [1] and MM-RLHF [2], and the policy trained with ASPIRE process rewards. We also report the false refusal rate on safe VLGuard instructions. Section 5.3 of the revision refers to the complete results in Appendix E.3, Table 27.
These experiments allow direct comparison of the safety of responses generated by different policies, but they cannot establish a ranking of the methods themselves when those methods operate at different levels. ASPIRE trains a step-level safety evaluator, SPA-VL and MM-RLHF use preference supervision to improve generation policies, and inference-time safeguards inspect requests or responses during deployment. These methods can be combined, for example, by training a policy with ASPIRE's step-wise rewards and then using an inference-time safeguard to check the responses generated by that policy. Differences in backbone models, training data, and optimization procedures across policies also prevent these results from separately quantifying the contribution of process rewards.
Section 4.1 of both the original and revised manuscripts describes how ASPIRE trains the Process Reward Model (PRM), while Section 4.2 explains how the PRM is frozen and its rewards are used to train the policy.
What Problems Do the Three Types of Related Work Address?
| Work | Level | Main Output | Target |
|---|---|---|---|
| SPA-VL [1] | Policy level | 100,788 safety preference samples, each containing an image, question, preferred response, and rejected response. The original work trains policies using DPO and PPO separately. | Which responses the policy generates given an image and question. |
| MM-RLHF [2] | Reward modeling and policy alignment | 120,000 pairs of human preference comparisons, a reward model that first generates a critique and then assigns a score, and dynamic weighting based on reward differences. Safety is one of its ten evaluation dimensions. | Evaluation of complete responses and the policies trained using these preferences. |
| Safeguard-style methods | Inference level | Detection of content violations or intervention in generation at inference time. For example, Llama Guard 3 Vision [3] is an independent safety classifier. | Multimodal requests, generated content, or the generation process during deployment. |
ASPIRE's Position in the Safety Alignment Pipeline
| Comparison Dimension | SPA-VL / MM-RLHF | Inference-Time Safeguards (Independent Classifier as an Example) | ASPIRE |
|---|---|---|---|
| Optimization target | Policy model; MM-RLHF also trains a response-level reward model | Safety classifier | Process reward model |
| Supervision or judgment granularity | Preferences between complete responses | Whether a request or response violates safety requirements | Six-class safety labels for each visible response prefix |
| Whether policy weights are updated | Updated during preference alignment | The generation policy's weights remain fixed when deploying the classifier | The policy is not updated during PRM training. During downstream GRPO, the PRM is frozen and the policy is updated. |
| Main evaluation | Safety and helpfulness of policy outputs | Violation detection accuracy, false positives, and false negatives | Step-level classification accuracy and exact first-Unsafe-step localization |
Part 3
Downstream Policy Comparison
At the final-policy level, responses generated by different models can be compared using the same metrics. To this end, we evaluated the base policy, SPA-VL policy, MM-RLHF policy, and the policy trained with ASPIRE rewards on MM-SafetyBench, VLGuard, and MSSBench. The VLGuard false refusal rate is the proportion of safe instructions that are refused. Section 5.3 of the revision refers to the complete results in Appendix E.3, Table 27. All values below are percentages. SafeGuard was not evaluated in this round. ASR denotes attack success rate. FRR is the percentage of safe VLGuard instructions that the policy refused.
| Policy | MM-SafetyBench | VLGuard | VLGuard | MSSBench |
|---|---|---|---|---|
| Metric (%) | ASR ↓ | ASR ↓ | FRR ↓ | Safety rate ↑ |
| Base | 48.75 | 53.00 | 2.80 | 36.53 |
| SPA-VL | 48.24 | 3.00 | 0.00 | 36.36 |
| MM-RLHF | 35.62 | 21.60 | 6.60 | 43.09 |
| ASPIRE | 32.25 | 22.20 | 0.72 | 55.89 |
On MM-SafetyBench, ASPIRE achieves an attack success rate of 32.25%, which is 16.50 percentage points lower than the base policy and 3.37 percentage points lower than MM-RLHF. On MSSBench, ASPIRE achieves the highest safety rate, at 55.89%. SPA-VL performs best on VLGuard, with an attack success rate of 3.00% and a false refusal rate of 0.00% on safe instructions. On the same benchmark, ASPIRE reduces the base policy's attack success rate from 53.00% to 22.20% and its false refusal rate from 2.80% to 0.72%.
On MSSBench, ASPIRE's safety rate is 12.80 percentage points higher than that of the second-highest method, MM-RLHF. Appendix B.12 of the original submission also compares policies trained using the six-class SFT reward model and the ASPIRE reward model on another set of 200 benign requests. These policies refuse 48 requests (24%) and 44 requests (22%), respectively. This set of requests differs from the safe VLGuard instructions above.
The method designs directly evaluated in this paper are process labels and reward strategies. Table 15 in Appendix B.9 of both the original and revised manuscripts compares three supervision granularities. On the same first-Unsafe localization task, binary labels, three reward groups, and six-class labels achieve exact localization rates of 74.00%, 73.67%, and 77.33%, respectively. Table 5 in Section 5.3 of both versions holds the same six-class PRM fixed. When rewards are assigned only at the final prefix, the mean attack success rate across five jailbreak benchmarks is 14.88%, compared with 7.21% when rewards are assigned at every prefix.
References
[1] Y. Zhang et al., “SPA-VL: A comprehensive safety preference alignment dataset for vision language models,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025, pp. 19867–19878.
[2] Y. Zhang et al., “MM-RLHF: The next step forward in multimodal LLM alignment,” in Proc. 42nd Int. Conf. Mach. Learn. (ICML), vol. 267, 2025, pp. 76625–76654.
[3] J. Chi et al., “Llama Guard 3 Vision: Safeguarding human-AI image understanding conversations,” arXiv:2411.10414, 2024.
Part 1
Thank you for recognizing the value of the six label categories and first-unsafe-step localization. Your question about self-correction also reminded us of the need to clarify at which step the model discloses harmful information visible to the user. We clarified this in the revision and, following your suggestions, added statistics on the heads' predictions during training and policy evaluations on three benchmarks.
weakness 1: Why use 16 heads? Why do the heads make different predictions for the same input?
We use 16 lightweight classification heads to obtain multiple six-class probability estimates for the same prefix while performing only one forward pass through the shared backbone. We have not compared different numbers of heads, so the most suitable number remains to be examined. All heads receive the backbone representation of the same prefix; their differences arise from initialization and training. Each head is independently initialized using Kaiming initialization and independently retains each sample with probability 0.8 for its own update during training. Thus, even within the same batch of data, the samples actually used to update each head may differ. The anchoring term also constrains how far each head deviates from its own initial parameters. Section 4.1 and Appendix B.6 of the original submission describe this architecture and the anchoring objective. To answer your question about actual disagreement, we added statistics for all 7,872 candidate steps during one training pass in Appendix B.6 of the revision. The table below takes the class with the highest probability from each head as its prediction.
On 67.98% of the steps, all 16 heads select the same class. On another 12.07% of the steps, at least five different class predictions occur. Actual selection uses the full six-class probabilities from each head, so even when the heads select the same class, their degrees of confidence in that class may still differ.
weakness 2: How is it scored if the model recognizes a risk during exploration and corrects itself in time?
We agree that the model should have the opportunity to identify risks and adjust its response in time. Our evaluation concerns what the user has already seen at the end of each paragraph; the model's private reasoning is not scored. For example, if the model recognizes a risk before providing actionable harmful instructions and instead gives an appropriate warning, this prefix can be labeled Safe with warning, with a corresponding reward of +1. If paragraph 2 has already provided harmful instructions and the model refuses only in paragraph 3, the user has already received these instructions at step 2 [4]. Therefore, the first Unsafe position is step 2. When evaluating step 3, the cumulative response still contains the previously disclosed instructions, so this prefix remains labeled Unsafe. During policy training, when the frozen PRM predicts this class, the reward is −1.
Existing work [1] identifies a related "unintended elicitation" issue, where LRMs may reveal harmful knowledge during extended reasoning even when responding to sensitive yet benign prompts. Although their work studies reasoning traces rather than our user-visible response prefixes, it similarly highlights the importance of identifying harmful content at the point where it emerges.
Appendix A of the original submission defines cumulative-prefix labels, and Section 4.2 specifies the class-to-reward mapping. Section 3 of the revision further clarifies the boundaries of user-visible paragraphs and adds an example in which the model gives a timely warning before disclosure.
References
[1] Q. Ren et al., “More Sail than Ballast: Addressing Harmful Knowledge Leakage in the Expansive Reasoning Space of LRMs,” in Proc. 43rd Int. Conf. Mach. Learn. (ICML), ser. Proc. Mach. Learn. Res. (PMLR), vol. 306, pp. 104700–104724, 2026.