acceptodds
Under review as a conference paper at ICLR 2027

Striking the Harmonization: Personalized Safety via the Consistency Constraints

Abstract

Recent research on personalized safety in large language models has separately investigated two phenomena—personalized jailbreaking and personalized maladaptation—yet these lines of work remain disjointed, offering no unified account of the fundamental tension between general safety constraints and individual-specific requirements. We propose a Consistency-based framework that reconceptualizes both safety and personalization as two forms of contextual consistency: the former internalized as an intrinsic model property, the latter grounded in input context. Under this unified consistency perspective, we formalize personalized safety as a Constrained Reinforcement Learning problem. Specifically, we minimize safety consistency to strictly enforce general safety boundaries, while maximizing the personalized reward (the negation of personalized consistency) to drive individual preference satisfaction. This formulation recasts the apparent tension not as a trade-off, but as a jointly satisfiable condition under explicit consistency constraints on safety-personalization feasibility.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Personalized Safety, Personalized Jailbreaking, Personalized Maladaptation, LLM Consistency, Constrained Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UyBaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to balance general safety requirements and personalized needs in large language models. It focuses on two problems, personalized jailbreaking and personalized maladaptation, and proposes to unify them from a consistency perspective. Based on this formulation, the paper treats general safety as a constraint and personalized satisfaction as the optimization objective, and trains the model with constrained reinforcement learning. Experiments on the PENGUIN benchmark show that the proposed method improves personalized satisfaction while maintaining relatively strong safety performance.

Critical strengths

1. Clear and novel problem setting. This paper studies personalized jailbreaking and personalized maladaptation in a unified framework. It aims to improve personalized satisfaction while maintaining general safety constraints.

2. Simple but effective constrained optimization design. This paper formulates personalized safety alignment as a constrained reinforcement learning problem. Personalized adaptation is used as the reward, while general safety consistency is used as the cost.

3. Multi-model empirical validation. The proposed method is evaluated on four models from the Llama and Qwen families. This paper also evaluates risk adaptation, empathy, and personalization. A human evaluation is also provided to check the reliability of the LLM-based scores.

Critical weaknesses

1. Limited comparison with specialized personalized safety methods. The paper mainly compares with general post-training methods, including SFT, GRPO, and PPO. However, recent methods specifically designed for personalized safety are not included in the main experiments. Comparisons with such methods would provide stronger evidence for the effectiveness of the proposed framework.

2. The proposed safety consistency metric is not sufficiently justified. The paper measures general safety consistency using the difference between the actual output change and a Jacobian-based approximation. Although the authors provide some consistency verification, it is still unclear why this difference specifically reflects safety rather than general representation change or model sensitivity.

3. Limited evaluation on personalized safety datasets. The main experiments are conducted only on the PENGUIN dataset. It is therefore unclear whether the reported improvements generalize to other personalized safety settings. Evaluation on additional benchmarks would provide stronger evidence for the generalizability of the proposed method.

Questions for authors
  1. The authors should consider comparing with recent personalized safety methods, rather than only general post-training methods, e.g., CoSA [1], to further validate the reported improvements under a fair comparison setting.

  2. The authors should evaluate the proposed general safety consistency Gz\mathcal{G}_z on inputs with basic persona information but without additional safety persona information. This would help verify whether Gz\mathcal{G}_z specifically captures safety-related changes rather than general changes caused by individual information.

  3. The authors should further evaluate the proposed method on additional personalized safety benchmarks, such as CASTLE [2], to verify whether the reported improvements generalize beyond PENGUIN.

[1] Jingyu Zhang et al., Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirement, ICLR 2025.

[2] Rui Jia et al., CASTLE: A Comprehensive Benchmark for Evaluating Student-Tailored Personalized Safety in Large Language Model, EMNLP 2026.

Reviewer_NnSEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a consistency-based framework for personalized safety alignment, aiming to address both personalized jailbreaking and personalized maladaptation. The authors formulate personalized safety as a constrained reinforcement learning problem, where a rubric-based reward encourages personalized responses and a Jacobian-based consistency cost is introduced to preserve general safety. The framework is optimized using PPO-Lagrange and evaluated on four language models using personalized safety satisfaction (PSS), jailbreak counts (JC), and the proposed consistency-cost metric (GSCC). The results show improvements in PSS and GSCC, while improvements in jailbreak robustness are less consistent.

Critical strengths

The paper provides an interesting perspective on personalized safety through the lens of LLM consistency. The proposed Jacobian-based consistency cost is also an interesting attempt to quantify how personalized information affects model behavior.

Critical weaknesses

1. Unclear motivation and advantages over existing safety alignment methods

The paper does not clearly establish the necessity or advantages of the proposed consistency-based formulation over existing safety alignment methods. For example, the three responses in Figure 1 can be summarized as follows: (1) Safe but less helpful, (2) Unsafe but helpful, and (3) Safe and helpful. Since existing safety alignment methods, such as Safe RLHF, already aim to maximize helpfulness while satisfying safety constraints, they are also designed to favor the third type of response. Therefore, Figure 1 does not convincingly demonstrate a limitation of existing safety alignment methods or motivate the need for the proposed formulation.

2. Insufficient justification and empirical evidence that the proposed method addresses the claimed problems

The paper aims to address both personalized jailbreaking and maladaptation, but it is unclear whether the proposed optimization framework effectively addresses these issues. First, the Jacobian-based consistency cost is used as a proxy for violations of general safety principles without sufficient theoretical justification. Furthermore, the experimental results do not consistently support the claimed improvements in safety. Although the proposed method reduces the consistency cost, GRPO achieves lower jailbreak counts across all evaluated models. These results raise concerns about whether minimizing the proposed cost actually improves general safety. The paper aims to address both personalized jailbreaking and maladaptation, but it is unclear whether the proposed optimization framework effectively addresses these issues. First, the Jacobian-based consistency cost is used as a proxy for violations of general safety principles without sufficient theoretical justification. Furthermore, the experimental results do not consistently support the claimed improvements in safety. Although the proposed method reduces the consistency cost, GRPO achieves lower jailbreak counts across all evaluated models. These results raise concerns about whether minimizing the proposed cost actually improves general safety.

Questions for authors
  • Is there any direct evidence supporting the claim that GRPO shows over-refusal behavior?
  • What reward functions are used for PPO, GRPO, and PPO-Lagrange? My understanding is that the PSS rubric is used to define the reward function. If all methods use the same reward, why does PPO-Lagrange achieve higher PSS than GRPO and PPO, even though it additionally considers a cost constraint?
  • The motivation for using constrained RL is not sufficiently clear. First, if the proposed cost also helps improve PSS, why should it be treated as a separate constraint rather than incorporated into the reward function? Second, how is the cost threshold determined, and how sensitive are the results to this choice? Furthermore, the PSS reward simply combines three components without explicitly considering their relative importance. What motivates the use of a Lagrangian constraint for the cost while using a simple aggregation for the reward components?
  • Minor comment: Many citations are incorrectly formatted (e.g., line 29). Please use parenthetical citations where appropriate and check the citation formatting throughout the paper.
Reviewer_DSMTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper frames personalized jailbreaking and personalized maladaptation as two forms of inconsistency, and proposes optimizing personalized response quality subject to a general-safety consistency constraint. The safety cost compares the change in output probabilities induced by persona information with a Jacobian-based estimate of that change; personalized reward comes from a rubric covering risk adaptation, empathy, and personalization. A PPO-Lagrange procedure trains the policy on a multi-turn extension of PENGUIN. Experiments across four Llama/Qwen models compare against the original instruction models, SFT, GRPO, and PPO, reporting personalized satisfaction, jailbreak counts, the proposed consistency cost, and a human check of judge scores.

Critical strengths
  • The paper addresses an important tension: a response can respect generic safety rules while failing to accommodate a particular user's vulnerabilities, and personalization can itself create opportunities for jailbreaks. Separating personalized utility from a safety constraint is a useful way to formulate this problem.

  • The evaluation covers two model families and two size ranges, and reports both personalization and jailbreak outcomes rather than only the optimized cost. The most encouraging results are on the two 3B models: relative to PPO, jailbreak counts decrease from 49 to 24 and from 30 to 20, while PSS changes from 0.73 to 0.74 and remains 0.72, respectively. The dimension-level scores and disclosed generation/evaluation prompts also help interpret these results.

Critical weaknesses
  • The central safety cost lacks the mathematical and semantic justification needed for the paper's hard-safety claims. In Eq. (5), the directional first-order change ||J delta-h||_2 is replaced by the upper bound ||J||_F ||delta-h||_2; without additional assumptions, their difference in Eq. (6) is not a Taylor approximation error. More fundamentally, even a valid linearity residual need not measure safety: for the scalar map o(h)=(1-h,h), changing h from 0 to 1 gives zero Eq. (6) cost while completely switching the output distribution, including from a safe token to an unsafe one. Eq. (8) and Appendix A constrain average proxy cost, which does not establish the claimed inviolable safety boundary or guaranteed safety feasibility. The experiments do not bridge this gap. Section 4.2 changes both persona information and query semantics, without validating the cost against harmful-output labels. In Tables 2 and 5, Llama-3B GRPO has cost 0.7794 and 3 jailbreaks, whereas the proposed method has the lower cost 0.3601 but 24 jailbreaks. This does not negate the useful PPO comparisons, but it prevents treating lower GSCC as independent evidence of greater safety.

  • The experimental protocol is insufficiently specified to establish the robustness or source of the gains. The paper and appendices do not report train/test sizes and splitting rules for the generated dialogues, the denominator for the main jailbreak counts, exact model checkpoints, the chosen cost threshold, or sufficient training/Jacobian implementation settings. There are no uncertainty estimates for the small PSS gains over PPO (0.00-0.02), and the 90% human confirmation result lacks a sample size and a clear independent annotation protocol. Crucially, no matched PPO-Lagrange comparison using a conventional safety cost, or threshold sweep at matched safety levels, isolates the contribution of Eq. (6). GRPO has fewer jailbreaks on all four models, so the current evidence supports particular operating-point trade-offs rather than the claimed general resolution of the safety-personalization tension.

Questions for authors
  1. What assumptions justify replacing the directional Jacobian change with its Frobenius-norm bound in Eq. (5), and what connects Eq. (6) to unsafe outputs? Please clarify the hidden state, token/sequence aggregation, and Jacobian computation actually used. Can you provide held-out calibration against harmful-output labels, including harmless persona/query changes as controls?

  2. What cost threshold was used, and what proportion of individual responses violates it? Can you compare against PPO-Lagrange with a conventional safety cost under the same data, reward, and compute, or provide a threshold sweep showing personalization at matched jailbreak rates? These results would directly affect my assessment of the proposed cost's contribution.

  3. Please provide the data split and test denominator, exact checkpoints and training settings, uncertainty for the reported gains, and the sample size and procedure behind the human evaluation. Were source scenarios/personas kept separate across training and evaluation?

Comments on the paper

1 comment
AuthorsOpenReview ↗

rebuttal summary

We would like to provide a complete summary of our rebuttal as the final response. We sincerely thank the reviewers and chairs for their comments and efforts.

Theoretical part

  1. Reviewers DSMT, NnSE, and UyBa questioned the justification of the Jacobian-based consistency cost (GSCC). We clarified in the rebuttal that replacing the directional first-order change ∣JΔh∣2|J \Delta h|_2 with ∣J∣F∣Δh∣2|J|_F|\Delta h|_2 provides a rigorous conservative upper bound: it characterizes the direction-independent amplification capability of the current layer, can be computed once in closed form from the softmax Jacobian and reused across perturbations, and is therefore suitable for relative safety-consistency comparison across conditions.

  2. Regarding whether GSCC captures safety-related change rather than generic model sensitivity (Reviewers NnSE and UyBa), we added the requested controlled-variable verification: when the additional safety persona information is removed and only basic persona information varies, the perturbation-cost distributions are largely consistent across conditions (only minor perturbation-induced changes), whereas the distribution based on the additional safety persona information shows clear differences. This indicates that the Jacobian-based design extracts safety-related consistency information rather than general representation change.

  3. Regarding why general safety is formulated as a constraint rather than a reward component (Reviewer NnSE): the Risk sub-score already overlaps with the cost direction, and the two act in opposite directions, so one is suitable as reward and the other as cost. General safety is a feasibility requirement rather than another utility component: a fixed weighted aggregation would allow sufficiently large personalization rewards to compensate for increased safety violations.

Experimental issues

  1. Regarding experiment details (Reviewer DSMT): data splits (5,665 train / 2,433 test; 26,124 / 11,387 turn-level samples); the JC denominator (a raw count of unsafe responses among the same 11,387 test turns, judged by GLM-4.7); exact checkpoints (all RL methods initialized from the same SFT checkpoint; final actor checkpoints of a 5-epoch run); the cost threshold (c = 0, λ\lambda initialized at 0.25); the Jacobian implementation (two prompts differing only in sensitive persona attributes).

  2. Regarding the small PSS margins (Reviewer DSMT): PSS remains at parity with PPO across all four backbones, while GSCC decreases on every backbone and the 3B unsafe counts drop by one-third to one-half. GSCC is a deterministic function of logits and hidden states and is not subject to judge variance; the human evaluation covers 800 randomly sampled examples with four researchers providing binary judgments with rationales.

  3. Regarding the GRPO comparison (Reviewers DSMT and NnSE): We additionally attached the identical consistency cost to GRPO: PSS is unchanged (0.70→0.70 on Qwen-7B, 0.69→0.69 on Qwen-3B) while JC drops by 33% (6→4) and 27% (15→11). At a fixed operating point the cost reduces violations with zero utility loss — precisely the behavior of a constraint — showing the effect is algorithm-agnostic rather than specific to PPO.

  4. Regarding specialized baselines and generalization (Reviewer UyBa): we aligned our data construction with CoSAlign and evaluated on 1,000 sampled examples: our method remains the best (PSS 0.5470 vs. SFT 0.5238, GRPO 0.5257, PPO 0.5399, with the highest risk, empathy, and personalization scores). We further evaluated all methods on the aligned CASTLE_en: the method ordering is preserved on all five metrics (our method achieves the best PSS 0.6996, risk, and empathy scores, and is on par with PPO on personalization), and all methods reach JC = 0 on the jailbreak subset, so the reported differences reflect subtler, context-dependent risk rather than explicit jailbreak compliance.

  5. Regarding comparison with conventional constrained safety alignment (Reviewers NnSE and DSMT): under the same experimental setting, SafeRLHF yields PSS 0.6145 with 230 jailbreaks, while our method reaches PSS 0.75 with 8 jailbreaks and higher risk and empathy scores.

Finally, we would like to state that our primary motivation is to specify what "safe and helpful" means when helpfulness is user-specific and the safety boundary is persona-sensitive. Our formulation makes two contributions. First, it turns an under-specified objective into an explicit constrained-optimization problem in which general safety is a feasibility requirement rather than a tradable utility component. Second, and most importantly, the safety signal is analytic, deterministic, and computable before generation — two forward passes and a closed-form softmax Jacobian — which keeps the policy's next-token distribution inside the locally linear neighborhood of safe base behavior, while leaving absolute safety assessment to direct evaluation outside the optimization loop.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.