acceptodds
Under review as a conference paper at ICLR 2027

ASPIRE: Active Supervision for Process Information-theoretic Reward Estimation

Abstract

Outcome-based Reward Models (ORMs) have advanced safety alignment for Multimodal Large Language Models (MLLMs), but their coarse feedback only indicates whether a response is safe and provides limited information about where unsafe behaviors emerge. Process Reward Models (PRMs) offer step-level supervision for safety reasoning, yet their development is constrained by the lack of fine-grained datasets, standardized evaluation protocols, and efficient training strategies. In this work, we propose a comprehensive framework for Safety PRMs. We introduce a six-class safety taxonomy, an automated step-wise annotation pipeline for generating process-level supervision, and **ProSafeBench**, a human-annotated benchmark for evaluating safety risk localization. To improve training efficiency under limited supervision, we further propose **ASPIRE** (**A**ctive **S**upervision for **P**rocess **I**nformation-theoretic **R**eward **E**stimation), which combines an information-bottleneck-inspired regularizer with uncertainty-guided example selection to identify informative training steps. Experiments show that ASPIRE achieves stronger Safety PRM performance with 25.3% fewer labeled training steps. It reaches 81.13% Global Step Accuracy on ProSafeBench and improves downstream policy optimization by reducing safety attack success rates while maintaining general multimodal capabilities.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Large Language Models, Safety Alignment, Process Reward Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_p38ROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a process reward model framework for multimodal safety, comprising six safety label categories, an automated step-by-step annotation pipeline, and a human-annotated benchmark, ProSafeBench. Its training method, ASPIRE, combines uncertainty-guided step selection with HSIC regularization and uses step-level rewards to guide downstream policy optimization.

Critical strengths
  1. The six label categories account for harmful content, over-refusal, and logical issues, providing more fine-grained supervision than binary classification.
  2. ProSafeBench supports step-level safety assessment and identification of the first unsafe step, making it a valuable evaluation resource.
  3. The experiments evaluate not only the reward model but also demonstrate a reduction in downstream jailbreak attack success rates while maintaining comparable general multimodal capabilities.
Critical weaknesses
  1. Why are 16 classification heads used? Given the same inputs and training procedure, how do differences between the classification heads arise and manifest?
  2. Why is the presence of harmful content in intermediate response steps considered unacceptable? Allowing the model to explore an incorrect direction, recognize the problem, and backtrack may itself help it arrive at a safe answer.
  3. Please compare performance with other safety alignment methods, such as SPA-VL, mm-rlhf, and safeguard. In addition, please report results on more datasets, such as mm-safetybench, vlguard, and mssbench, and include the false refusal rate.
Questions for authors

See weaknesses.

5 comments

Thank you for recognizing the value of the six label categories and first-unsafe-step localization. Your question about self-correction also reminded us of the need to clarify at which step the model discloses harmful information visible to the user. We clarified this in the revision and, following your suggestions, added statistics on the heads' predictions during training and policy evaluations on three benchmarks.

weakness 1: Why use 16 heads? Why do the heads make different predictions for the same input?

We use 16 lightweight classification heads to obtain multiple six-class probability estimates for the same prefix while performing only one forward pass through the shared backbone. We have not compared different numbers of heads, so the most suitable number remains to be examined. All heads receive the backbone representation of the same prefix; their differences arise from initialization and training. Each head is independently initialized using Kaiming initialization and independently retains each sample with probability 0.8 for its own update during training. Thus, even within the same batch of data, the samples actually used to update each head may differ. The anchoring term also constrains how far each head deviates from its own initial parameters. Section 4.1 and Appendix B.6 of the original submission describe this architecture and the anchoring objective. To answer your question about actual disagreement, we added statistics for all 7,872 candidate steps during one training pass in Appendix B.6 of the revision. The table below takes the class with the highest probability from each head as its prediction.

Head predictions on 7,872 candidate stepsProportion (%)
Mean pairwise class disagreement rate16.30
All 16 heads predict the same class67.98
At least five distinct classes are predicted12.07

On 67.98% of the steps, all 16 heads select the same class. On another 12.07% of the steps, at least five different class predictions occur. Actual selection uses the full six-class probabilities from each head, so even when the heads select the same class, their degrees of confidence in that class may still differ.

weakness 2: How is it scored if the model recognizes a risk during exploration and corrects itself in time?

We agree that the model should have the opportunity to identify risks and adjust its response in time. Our evaluation concerns what the user has already seen at the end of each paragraph; the model's private reasoning is not scored. For example, if the model recognizes a risk before providing actionable harmful instructions and instead gives an appropriate warning, this prefix can be labeled Safe with warning, with a corresponding reward of +1. If paragraph 2 has already provided harmful instructions and the model refuses only in paragraph 3, the user has already received these instructions at step 2 [4]. Therefore, the first Unsafe position is step 2. When evaluating step 3, the cumulative response still contains the previously disclosed instructions, so this prefix remains labeled Unsafe. During policy training, when the frozen PRM predicts this class, the reward is −1.

Existing work [1] identifies a related "unintended elicitation" issue, where LRMs may reveal harmful knowledge during extended reasoning even when responding to sensitive yet benign prompts. Although their work studies reasoning traces rather than our user-visible response prefixes, it similarly highlights the importance of identifying harmful content at the point where it emerges.

Appendix A of the original submission defines cumulative-prefix labels, and Section 4.2 specifies the class-to-reward mapping. Section 3 of the revision further clarifies the boundaries of user-visible paragraphs and adds an example in which the model gives a timely warning before disclosure.

References

[1] Q. Ren et al., “More Sail than Ballast: Addressing Harmful Knowledge Leakage in the Expansive Reasoning Space of LRMs,” in Proc. 43rd Int. Conf. Mach. Learn. (ICML), ser. Proc. Mach. Learn. Res. (PMLR), vol. 306, pp. 104700–104724, 2026.

Thank you for suggesting that we compare different alignment policies on additional safety benchmarks and examine whether models incorrectly refuse safe requests.

weakness 3: How Does ASPIRE Compare with Other Alignment Policies in Terms of Safety and False Refusal Rates on Additional Benchmarks?

We added comparisons on MM-SafetyBench, VLGuard, and MSSBench among the base policy, policies aligned using SPA-VL [1] and MM-RLHF [2], and the policy trained with ASPIRE process rewards. We also report the false refusal rate on safe VLGuard instructions. Section 5.3 of the revision refers to the complete results in Appendix E.3, Table 27.

These experiments allow direct comparison of the safety of responses generated by different policies, but they cannot establish a ranking of the methods themselves when those methods operate at different levels. ASPIRE trains a step-level safety evaluator, SPA-VL and MM-RLHF use preference supervision to improve generation policies, and inference-time safeguards inspect requests or responses during deployment. These methods can be combined, for example, by training a policy with ASPIRE's step-wise rewards and then using an inference-time safeguard to check the responses generated by that policy. Differences in backbone models, training data, and optimization procedures across policies also prevent these results from separately quantifying the contribution of process rewards.

Section 4.1 of both the original and revised manuscripts describes how ASPIRE trains the Process Reward Model (PRM), while Section 4.2 explains how the PRM is frozen and its rewards are used to train the policy.

What Problems Do the Three Types of Related Work Address?

WorkLevelMain OutputTarget
SPA-VL [1]Policy level100,788 safety preference samples, each containing an image, question, preferred response, and rejected response. The original work trains policies using DPO and PPO separately.Which responses the policy generates given an image and question.
MM-RLHF [2]Reward modeling and policy alignment120,000 pairs of human preference comparisons, a reward model that first generates a critique and then assigns a score, and dynamic weighting based on reward differences. Safety is one of its ten evaluation dimensions.Evaluation of complete responses and the policies trained using these preferences.
Safeguard-style methodsInference levelDetection of content violations or intervention in generation at inference time. For example, Llama Guard 3 Vision [3] is an independent safety classifier.Multimodal requests, generated content, or the generation process during deployment.

ASPIRE's Position in the Safety Alignment Pipeline

Comparison DimensionSPA-VL / MM-RLHFInference-Time Safeguards (Independent Classifier as an Example)ASPIRE
Optimization targetPolicy model; MM-RLHF also trains a response-level reward modelSafety classifierProcess reward model
Supervision or judgment granularityPreferences between complete responsesWhether a request or response violates safety requirementsSix-class safety labels for each visible response prefix
Whether policy weights are updatedUpdated during preference alignmentThe generation policy's weights remain fixed when deploying the classifierThe policy is not updated during PRM training. During downstream GRPO, the PRM is frozen and the policy is updated.
Main evaluationSafety and helpfulness of policy outputsViolation detection accuracy, false positives, and false negativesStep-level classification accuracy and exact first-Unsafe-step localization

Downstream Policy Comparison

At the final-policy level, responses generated by different models can be compared using the same metrics. To this end, we evaluated the base policy, SPA-VL policy, MM-RLHF policy, and the policy trained with ASPIRE rewards on MM-SafetyBench, VLGuard, and MSSBench. The VLGuard false refusal rate is the proportion of safe instructions that are refused. Section 5.3 of the revision refers to the complete results in Appendix E.3, Table 27. All values below are percentages. SafeGuard was not evaluated in this round. ASR denotes attack success rate. FRR is the percentage of safe VLGuard instructions that the policy refused.

PolicyMM-SafetyBenchVLGuardVLGuardMSSBench
Metric (%)ASR ↓ASR ↓FRR ↓Safety rate ↑
Base48.7553.002.8036.53
SPA-VL48.243.000.0036.36
MM-RLHF35.6221.606.6043.09
ASPIRE32.2522.200.7255.89

On MM-SafetyBench, ASPIRE achieves an attack success rate of 32.25%, which is 16.50 percentage points lower than the base policy and 3.37 percentage points lower than MM-RLHF. On MSSBench, ASPIRE achieves the highest safety rate, at 55.89%. SPA-VL performs best on VLGuard, with an attack success rate of 3.00% and a false refusal rate of 0.00% on safe instructions. On the same benchmark, ASPIRE reduces the base policy's attack success rate from 53.00% to 22.20% and its false refusal rate from 2.80% to 0.72%.

On MSSBench, ASPIRE's safety rate is 12.80 percentage points higher than that of the second-highest method, MM-RLHF. Appendix B.12 of the original submission also compares policies trained using the six-class SFT reward model and the ASPIRE reward model on another set of 200 benign requests. These policies refuse 48 requests (24%) and 44 requests (22%), respectively. This set of requests differs from the safe VLGuard instructions above.

The method designs directly evaluated in this paper are process labels and reward strategies. Table 15 in Appendix B.9 of both the original and revised manuscripts compares three supervision granularities. On the same first-Unsafe localization task, binary labels, three reward groups, and six-class labels achieve exact localization rates of 74.00%, 73.67%, and 77.33%, respectively. Table 5 in Section 5.3 of both versions holds the same six-class PRM fixed. When rewards are assigned only at the final prefix, the mean attack success rate across five jailbreak benchmarks is 14.88%, compared with 7.21% when rewards are assigned at every prefix.

References

[1] Y. Zhang et al., “SPA-VL: A comprehensive safety preference alignment dataset for vision language models,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2025, pp. 19867–19878.

[2] Y. Zhang et al., “MM-RLHF: The next step forward in multimodal LLM alignment,” in Proc. 42nd Int. Conf. Mach. Learn. (ICML), vol. 267, 2025, pp. 76625–76654.

[3] J. Chi et al., “Llama Guard 3 Vision: Safeguarding human-AI image understanding conversations,” arXiv:2411.10414, 2024.

Reviewer_rKWSOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies process safety reward modeling for multimodal LLMs. They propose presents a framework that contains a six-class taxonomy for cumulative response prefixes, an automated annotation pipeline, and ProSafeBench with 600 human-annotated trajectories and 3,163 steps. ASPIRE combines multi-head uncertainty-based step selection with HSIC dependence regularization. The resulting PRM supplies category-aware rewards for step-level policy optimization. The authors report improved six-class classification, unsafe-step localization, and jailbreak robustnes.

Critical strengths
  1. Results improve over six-class SFT: external six-class accuracy increases from 35.19% to 46.96%, and ProSafeBench exact localization from 74.17% to 77.33%.
  2. The taxonomy distinguishes harmful content from appropriate refusals, over-refusals, and illogical completions. Human evaluation and the 300-instance training-label audit are useful assets.
Critical weaknesses
  1. The anchored ensemble heads and the two-threshold selection on mean confidence and inter-head standard deviation closely follow ActPRM. The paper cites ActPRM and notes that it selects from an already annotated pool, but this does not change the selection mechanism itself, and ActPRM is not used as a baseline. Prior work that uses HSIC as an information-bottleneck regularizer is also not cited (e.g., The HSIC Bottleneck: Deep Learning without Back-Propagation). The framework therefore reads as a combination of existing components, and the paper does not explain what makes this combination non-trivial.
  2. The main results report no seeds, standard deviations, or significance tests, and the gains are small: +0.32 pp on the binary average and +3.16 pp on exact localization. It is unclear where these gains come from.
  3. The "process" signal is thin. Whether a refusal is appropriate or unnecessary is determined by the query, not by the step, and the steps are "\n\n"-separated paragraphs of ordinary responses rather than reasoning chains.
  4. The annotation prompts in Tables 18–20 contain only {query} and {response}. This seems inconsistent with the claim that each label refers to the target step in its image, query, and reasoning context.
Questions for authors

See Weakness.

6 comments

weakness 1: Relationship with ActPRM and Existing HSIC Methods

We agree with your assessment of the selection mechanism. ASPIRE adopts ActPRM's anchored ensemble heads and the two-threshold rule based on confidence and inter-head disagreement. The fact that the candidate pool is already annotated is merely a setting of our experiments. We report that 5,881 of the 7,872 annotated steps are used to clarify how many steps of supervision are actually used for language model training.

The difference between the two methods lies in what content is used for training once a step meets the threshold. ActPRM examines the steps up to the first predicted error in a mathematical solution and selects the entire solution if any of these steps triggers selection. ASPIRE instead evaluates each prefix of a response independently. Specifically, HSIC regularizes the representations of the selection heads, and the probabilities assigned by the heads to the six safety behaviors are used to calculate the confidence and inter-head disagreement of the current prefix. If the prefix is selected, preceding paragraphs remain as context, and the language model learns to predict the corresponding safety label at this position. Other prefixes within the same response are filtered independently. This difference in the selection unit remains even when HSIC is disabled. Section 2 of the original submission already cites ActPRM, and Section 4.1 of the revision adds a citation to the work by Ma et al. on the HSIC bottleneck.

Following your suggestion, we added both binary and six-class safety adaptations of ActPRM to the SFT and ASPIRE comparisons in Section 5.2 of the original submission. Tables 1, 2, and 4 in Section 5.2 of the revision report the results on the corresponding tasks. The following table summarizes these comparisons. Adaptation details and complete results are provided in Appendix E.1 and Table 25 of the revision. The first two columns report equally weighted average accuracy across five external benchmarks, while the last three columns report binary process metrics on ProSafeBench. All values are percentages.

Safety PRMExternal BinaryExternal Six-ClassGlobal Step AccuracyExact First-Unsafe LocalizationTrajectory-Averaged Accuracy
Binary ActPRM92.57—80.6275.8386.67
Six-class ActPRM89.5041.5280.7874.3383.33
Six-class SFT92.7435.1979.6774.1786.00
Six-class ASPIRE93.0646.9681.1377.3387.00

Binary ActPRM performs very similarly to ASPIRE on the external binary task, with a difference of only 0.49 percentage points. When both methods use six-class supervision, ASPIRE achieves 5.44 percentage points higher external six-class accuracy and 3.00 percentage points higher exact first-unsafe-step localization than ActPRM.

weakness 2: Where Are the Gains Mainly Observed, and Are They Stable Across Training Seeds?

You pointed out that the binary improvement reported in Section 5.2 of the original submission is only 0.32 percentage points, and that a single run makes it difficult to determine whether the 3.16-percentage-point improvement in first-unsafe-step localization is stable. Both numbers come from seed 42. To address this, we compared six-class SFT and ASPIRE on the same test sets, pairing runs by training seeds 42, 43, and 44. Section 5.2 of the revision refers to the complete results in Appendix E.2, Table 26. External accuracy is averaged with equal weight across the five benchmarks, and first-unsafe-step localization is evaluated on 600 trajectories annotated as unsafe.

MetricSFT Mean ± SD (%)ASPIRE Mean ± SD (%)Paired Gain [95% CI] (pp)p
External Six-Class Accuracy35.82 ± 0.6247.95 ± 0.89+12.13 [10.56, 13.70]0.0009
External Binary Accuracy92.33 ± 0.3793.22 ± 0.42+0.88 [−0.54, 2.30]0.1170
Exact First-Unsafe-Step Localization74.56 ± 0.3577.39 ± 0.10+2.83 [2.12, 3.55]0.0034

Across the three runs, ASPIRE improves external six-class accuracy by an average of 12.13 percentage points and first-unsafe-step localization by an average of 2.83 percentage points. The 95% confidence intervals for both gains are above zero. The average binary improvement is 0.88 percentage points, but its confidence interval includes zero. Therefore, these three runs are not yet sufficient to confirm a stable binary improvement.

The magnitude of the gains also depends on which behaviors the evaluation distinguishes. For example, in the VLGuard results in Table 2 of Section 5.2 in both the original and revised manuscripts, six-class accuracy increases from 2.35% for SFT to 32.74% for ASPIRE under seed 42. Appropriate refusal and unnecessary refusal are both mapped to Safe in binary evaluation. Therefore, even if the model corrects misclassifications between these two categories, the binary score remains unchanged.

weakness 3: What Process Signals Can Paragraphs Provide?

Thank you for pointing out that whether a refusal is appropriate depends on both the image and the user’s request. Examining the response paragraph by paragraph is necessary to determine when the model actually refuses the request and whether it has already disclosed harmful content before refusing. Even if two responses both end with a refusal, one may explicitly refuse in the first paragraph, while the other may first provide harmful instructions. This distinction can be determined directly from the response visible to the user. Section 3.2 of the original submission introduces the segmentation method, and Section 3 of the revision further clarifies that each step corresponds to a paragraph boundary in the user-visible response.

Table 5 in Section 5.3 of both the original and revised manuscripts also examines whether these intermediate prefixes are useful for policy training. Using the same six-class PRM, we compare assigning rewards only at the final prefix with assigning rewards at every prefix. The following table reports attack success rates (%) on five jailbreak benchmarks.

Reward PlacementFigStepJoodQRMML-WRVoTA-p200Mean ASR ↓
Final Prefix Only6.577.5813.209.5537.5014.88
Every Prefix0.510.001.525.0329.007.21

When rewards are assigned at every prefix, attack success rates decrease across all five benchmarks, with the mean decreasing from 14.88% to 7.21%. This provides direct evidence for the usefulness of intermediate feedback from user-visible responses in policy training.

weakness 4: What Inputs Does the Automatic Annotator Receive?

Thank you for pointing out the ambiguity caused by the prompt templates. Section 3.2 of the original submission describes the annotation pipeline, while Tables 18–20 in Appendix C.1 only show the textual parts of the prompts. When annotating step t, the annotator actually receives both the original image and the text. The original image is provided as visual input, {query} contains the user question, and {response} contains the cumulative response from paragraph 1 through paragraph t. Therefore, the annotator can see the target paragraph and its preceding context. We have explicitly clarified these three inputs before Tables 18–20 in Appendix C.1 of the revision.

Reviewer_MonZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ASPIRE, a framework for process-level safety assessment in MLLMs. It introduces a six-class safety taxonomy, an automated step-level annotation pipeline, and the human-annotated ProSafeBench benchmark. ASPIRE combines HSIC regularization with uncertainty-guided sample selection to train Safety PRMs, which are further used for GRPO-based policy alignment. Experiments demonstrate improvements in safety classification, unsafe-step localization, and jailbreak robustness with fewer labeled training steps.

Critical strengths
  1. The paper addresses an important and meaningful research problem in multimodal model safety.
  2. The paper presents a broad range of experiments to evaluate the effectiveness of the proposed approach.
Critical weaknesses
  1. Limited clarity of technical novelty.
    ASPIRE combines existing techniques, including HSIC regularization, ensemble-based uncertainty estimation, and uncertainty-guided sample selection. However, the paper does not clearly explain the methodological novelty of this combination or how these components interact to improve process-level safety assessment.

  2. Insufficient justification of the information-bottleneck design. Although the paper introduces an information-bottleneck objective, its implementation relies on HSIC regularization. The connection between the HSIC objective and the intended preservation of safety-relevant information is not sufficiently explained or empirically validated.

  3. Incomplete ablation studies. The current ablations do not fully isolate the contributions of HSIC regularization and uncertainty-guided sample selection. In particular, removing HSIC may change the selected training samples, making it difficult to determine the independent effects and potential interactions of these components.

  4. Unclear presentation and organization. The paper lacks a clear connection between its claimed contributions, methodological components, and experimental evidence. Figure 2 combines several stages without clearly distinguishing the core contributions. Furthermore, the uncertainty measures in Figure 2 and Section 4.1 are not consistently described, as the former uses entropy and variance while the latter uses predictive confidence and cross-head standard deviation.

Questions for authors
  1. What is the core technical novelty of ASPIRE beyond combining HSIC regularization and uncertainty-guided sample selection, and how do these components complement each other?

  2. Can the authors clarify the inconsistent uncertainty measures between Figure 2 and Section 4.1, as well as the discrepancy between step-level and cumulative-prefix safety labels in Figure 1 and Appendix A?

  3. How does the six-class safety taxonomy relate to the core ASPIRE framework? Furthermore, can the authors clarify how the proposed method specifically supports the concepts of "Active Supervision" and "Information-theoretic Reward Estimation" emphasized in the title?

3 comments

weakness 1, question 1 and question 3: ASPIRE's Specific Contribution and Its Relationship to Six-Class Supervision

We thank the reviewer for pointing out that our contribution was not sufficiently clear. Our methodological contribution is to integrate fine-grained safety supervision, per-prefix selection, and dependence regularization of the selection heads for multimodal safety assessment.

These three designs determine the prediction space in which uncertainty is measured, which prefixes are selected for supervision, and how the representations generating selection scores are trained. The six-class taxonomy determines which disagreements can trigger supervision. For example, all heads may predict binary Safe while disagreeing on whether a response is an appropriate refusal or an unnecessary refusal. A prefix can be selected when confidence is low or disagreement is high. ASPIRE applies the selection criterion independently to each prefix, retaining preceding paragraphs as context without requiring all prefixes of a trajectory to be selected. HSIC regularizes the head representations that produce these scores. At inference, the final class label is generated by the language model.

Sections 3.1 and 4.1 of the original submission describe the taxonomy and selection rules. Section 4.1 and Appendix E.1 of the revision clarify their interaction and the distinction from trajectory-level selection. In the newly added six-class safety adaptation of ActPRM, external six-class accuracy and exact first-unsafe-step localization are 41.52% and 74.33%, versus 46.96% and 77.33% for ASPIRE (revised Appendix E.1, Table 25).

Supervision granularityProSafeBench exact first-unsafe-step localization ↑ (%)
Binary safety labels74.00
Three reward-aligned groups73.67
Six safety classes77.33

Active Supervision refers to selecting 5,881 training steps from a pool of 7,872 previously labeled steps. Information-theoretic Reward Estimation refers to the use of an information-bottleneck-inspired regularizer during PRM training. During policy training, the frozen PRM maps its predicted classes to rewards: +1 for the three appropriate safety behaviors, −0.5 for unnecessary refusal and illogical completion, and −1 for Unsafe. Section 4.2 of both versions describes this mapping; Appendix B.9, Table 15, compares the three supervision granularities on the same localization task.

weakness 1: Why Use HSIC?

Section 4.1 of both versions presents the information-bottleneck motivation. Six-class cross-entropy encourages the selection heads to preserve information needed to distinguish safety labels, while HSIC penalizes statistical dependence between backbone and head representations. The two objectives are complementary: a constant representation could reduce dependence but would yield the same class distribution for every prefix and therefore fail to distinguish the six classes. The classification objective constrains this degeneracy.

Revised Section 4.1 clarifies that gradients from both the classification loss and HSIC stop at the backbone, which is updated by the language-modeling loss. Thus, HSIC can affect the final generative PRM through sample selection: it regularizes the selection heads, whose changed predictions may alter which prefixes subsequently contribute to LM training.

weakness 3: What Can the Ablations Distinguish?

For all uncertainty-guided selection configurations below, the retention rule is C < 0.70 or S > 0.05. The corresponding results are reported in Section 5.4, Table 6, and Appendix B.10, Tables 13–14, of both versions.

Training configurationMean binary accuracy across five external benchmarks ↑ (%)Global step accuracy ↑ (%)Exact first-unsafe-step localization ↑ (%)
Random selection, 5,881 steps87.5772.1563.67
Uncertainty selection, γ = 091.4878.1973.00
Uncertainty selection, γ = 0.0591.0679.8372.83
Full ASPIRE, γ = 0.10, 5,881 steps93.0681.1377.33
Uncertainty selection, γ = 0.2091.8280.5274.00

At the same budget of 5,881 steps, uncertainty-guided selection improves exact localization over random selection by 13.66%. With the selection thresholds fixed, changing HSIC from γ = 0 to γ = 0.10 improves localization by 4.33%. Both smaller and larger positive weights yield lower scores.

We agree with the your attribution concern: fixed thresholds do not imply identical selected examples, and retention counts may also differ. Revised Section 5.4 clarifies that HSIC may change which prefixes are selected later. The latter difference captures the overall effect of enabling HSIC in online training.

weakness 4 and question 2: Figures and Label Definitions

The entropy and variance labels in Figure 2 were incorrect. The actual retention criterion in Eqs. (3)–(5) of §4.1 in both the original and revised manuscripts is C < 0.70 or S > 0.05. C is the maximum class probability obtained after first averaging the probabilities across heads for each class, and S is the square root of the average cross-head variance over the six classes. The caption of Figure 2 in the revised manuscript now distinguishes the annotation process in §3.2, ASPIRE selection and training in §4.1, and the use of the frozen PRM in §4.2. The labels inside the figure will be aligned with the caption and equations.

Regarding the meanings of step-level labels and cumulative-prefix labels, §3 of the revised manuscript clarifies their relationship: "step-level" specifies when to evaluate, namely at the end of each paragraph; "cumulative prefix" specifies what to evaluate, namely the entire response the user has seen up to that point. For example, if paragraph 2 discloses actionable harmful instructions and paragraph 3 then provides a refusal, the individual paragraphs may be characterized as Safe → Unsafe → Safe, whereas the prefix labels actually used are Safe → Unsafe → Unsafe. The third prefix still contains the instructions disclosed in paragraph 2, so the first Unsafe occurs at step 2. Appendix A defines the labels actually used for training and evaluation. Each step has only one label: the label for step tt is assigned at the end of paragraph tt and evaluates the cumulative response consisting of paragraphs 1 through tt that the user has already seen. Section 3 of the revised manuscript further clarifies the paragraph boundaries at which evaluation takes place.

Thank you again for your thoughtful and constructive feedback. We have carefully addressed your concerns in our rebuttal and revised manuscript, and we sincerely appreciate your time and consideration. We would be grateful to hear whether our revisions have adequately addressed your concerns or if there are any remaining questions that we could further clarify. Thank you once again for your valuable feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Thank you to all reviewers for the thoughtful evaluation and constructive feedback. We would like to clarify ASPIRE's central motivation and contributions, summarize the main issues addressed during rebuttal, and highlight the additional evidence for the final assessment.

1. Core Motivation and Contribution

Our goal is to supervise how multimodal responses unfold, rather than relying only on response-level safety labels. An overall judgment can assess the complete answer, but does not explicitly identify where harmful content first appears or provide localized supervision for appropriate safety behavior and utility failures. We therefore study cumulative prefixes of user-visible responses, not private reasoning. Recognizing a risk and correcting the response before harmful disclosure can be appropriate; a later refusal does not remove harmful instructions already included in the visible response.

ASPIRE contributes an integrated framework for this task: a six-class taxonomy distinguishing appropriate safety behaviors, harmful content, unnecessary refusals, and illogical completions; an automated supervision pipeline with a 300-instance human audit; ProSafeBench, with 600 human-annotated trajectories and 3,163 steps; and a safety-specific PRM training method connecting fine-grained labels, prefix selection, and dependence-regularized selection heads. The contribution is not the invention of HSIC or ensemble uncertainty individually, but the design and evaluation of how contextualized process-safety supervision is allocated for PRM training and how the resulting PRM supports downstream policy alignment.

2. Technical Scope and Key Clarifications

The taxonomy is not merely an output format: it also defines the behavioral distinctions over which selection uncertainty is computed. Heads can agree on binary Safe while disagreeing between an appropriate and an unnecessary refusal. ASPIRE selects individual prefix endpoints while preserving the image, query, and preceding paragraphs as context, rather than automatically selecting the entire trajectory. HSIC regularizes the heads making these decisions; classification and HSIC gradients stop at the backbone, while the language-modeling loss trains the generative PRM. Thus, the regularizer can affect the final model through the selection of subsequent supervision. The revision clarifies this training path, the annotation inputs, the uncertainty definitions, and the sources and observed extent of head disagreement.

3. Additional Evidence and Practical Value

We added binary and six-class safety adaptations of ActPRM. Under the same six-class label space, ASPIRE achieves 46.96% external six-class accuracy and 77.33% exact localization, compared with 41.52% and 74.33% for ActPRM (Table 25). Three paired training seeds further show mean improvements over six-class SFT of 12.13 percentage points in six-class accuracy and 2.83 points in localization. These additions directly address concerns about the closest baseline and single-run evidence.

The experiments also provide insight into the relationship between supervision and downstream rewards. Six-class supervision yields 77.33% exact localization, compared with 73.67% for three reward-aligned groups and 74.00% for binary labels (Table 15), suggesting that distinctions merged by the reward mapping can still be useful for learning the evaluator. With the same six-class PRM, prefix-level feedback yields 7.21% mean jailbreak ASR, versus 14.88% with terminal feedback (Table 5). We additionally compared aligned policies with SPA-VL and MM-RLHF on MM-SafetyBench, VLGuard, and MSSBench, including false refusals (Table 27). Among the evaluated policies, ASPIRE has the lowest MM-SafetyBench ASR and highest MSSBench safety rate, while SPA-VL performs best on VLGuard. ASPIRE reduces the base policy's VLGuard false refusal rate from 2.80% to 0.72%.

We believe ASPIRE offers a promising technical path toward more effective training and evaluation of multimodal safety PRMs, advancing multimodal safety alignment through fine-grained process supervision, more precise localization of unsafe behavior, and targeted process-level rewards for downstream policy optimization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.