- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to balance general safety requirements and personalized needs in large language models. It focuses on two problems, personalized jailbreaking and personalized maladaptation, and proposes to unify them from a consistency perspective. Based on this formulation, the paper treats general safety as a constraint and personalized satisfaction as the optimization objective, and trains the model with constrained reinforcement learning. Experiments on the PENGUIN benchmark show that the proposed method improves personalized satisfaction while maintaining relatively strong safety performance.
Critical strengths
1. Clear and novel problem setting. This paper studies personalized jailbreaking and personalized maladaptation in a unified framework. It aims to improve personalized satisfaction while maintaining general safety constraints.
2. Simple but effective constrained optimization design. This paper formulates personalized safety alignment as a constrained reinforcement learning problem. Personalized adaptation is used as the reward, while general safety consistency is used as the cost.
3. Multi-model empirical validation. The proposed method is evaluated on four models from the Llama and Qwen families. This paper also evaluates risk adaptation, empathy, and personalization. A human evaluation is also provided to check the reliability of the LLM-based scores.
Critical weaknesses
1. Limited comparison with specialized personalized safety methods. The paper mainly compares with general post-training methods, including SFT, GRPO, and PPO. However, recent methods specifically designed for personalized safety are not included in the main experiments. Comparisons with such methods would provide stronger evidence for the effectiveness of the proposed framework.
2. The proposed safety consistency metric is not sufficiently justified. The paper measures general safety consistency using the difference between the actual output change and a Jacobian-based approximation. Although the authors provide some consistency verification, it is still unclear why this difference specifically reflects safety rather than general representation change or model sensitivity.
3. Limited evaluation on personalized safety datasets. The main experiments are conducted only on the PENGUIN dataset. It is therefore unclear whether the reported improvements generalize to other personalized safety settings. Evaluation on additional benchmarks would provide stronger evidence for the generalizability of the proposed method.
Questions for authors
-
The authors should consider comparing with recent personalized safety methods, rather than only general post-training methods, e.g., CoSA [1], to further validate the reported improvements under a fair comparison setting.
-
The authors should evaluate the proposed general safety consistency on inputs with basic persona information but without additional safety persona information. This would help verify whether specifically captures safety-related changes rather than general changes caused by individual information.
-
The authors should further evaluate the proposed method on additional personalized safety benchmarks, such as CASTLE [2], to verify whether the reported improvements generalize beyond PENGUIN.
[1] Jingyu Zhang et al., Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirement, ICLR 2025.
[2] Rui Jia et al., CASTLE: A Comprehensive Benchmark for Evaluating Student-Tailored Personalized Safety in Large Language Model, EMNLP 2026.
rebuttal summary
We would like to provide a complete summary of our rebuttal as the final response. We sincerely thank the reviewers and chairs for their comments and efforts.
Theoretical part
Reviewers DSMT, NnSE, and UyBa questioned the justification of the Jacobian-based consistency cost (GSCC). We clarified in the rebuttal that replacing the directional first-order change ∣JΔh∣2 with ∣J∣F∣Δh∣2 provides a rigorous conservative upper bound: it characterizes the direction-independent amplification capability of the current layer, can be computed once in closed form from the softmax Jacobian and reused across perturbations, and is therefore suitable for relative safety-consistency comparison across conditions.
Regarding whether GSCC captures safety-related change rather than generic model sensitivity (Reviewers NnSE and UyBa), we added the requested controlled-variable verification: when the additional safety persona information is removed and only basic persona information varies, the perturbation-cost distributions are largely consistent across conditions (only minor perturbation-induced changes), whereas the distribution based on the additional safety persona information shows clear differences. This indicates that the Jacobian-based design extracts safety-related consistency information rather than general representation change.
Regarding why general safety is formulated as a constraint rather than a reward component (Reviewer NnSE): the Risk sub-score already overlaps with the cost direction, and the two act in opposite directions, so one is suitable as reward and the other as cost. General safety is a feasibility requirement rather than another utility component: a fixed weighted aggregation would allow sufficiently large personalization rewards to compensate for increased safety violations.
Experimental issues
Regarding experiment details (Reviewer DSMT): data splits (5,665 train / 2,433 test; 26,124 / 11,387 turn-level samples); the JC denominator (a raw count of unsafe responses among the same 11,387 test turns, judged by GLM-4.7); exact checkpoints (all RL methods initialized from the same SFT checkpoint; final actor checkpoints of a 5-epoch run); the cost threshold (c = 0, λ initialized at 0.25); the Jacobian implementation (two prompts differing only in sensitive persona attributes).
Regarding the small PSS margins (Reviewer DSMT): PSS remains at parity with PPO across all four backbones, while GSCC decreases on every backbone and the 3B unsafe counts drop by one-third to one-half. GSCC is a deterministic function of logits and hidden states and is not subject to judge variance; the human evaluation covers 800 randomly sampled examples with four researchers providing binary judgments with rationales.
Regarding the GRPO comparison (Reviewers DSMT and NnSE): We additionally attached the identical consistency cost to GRPO: PSS is unchanged (0.70→0.70 on Qwen-7B, 0.69→0.69 on Qwen-3B) while JC drops by 33% (6→4) and 27% (15→11). At a fixed operating point the cost reduces violations with zero utility loss — precisely the behavior of a constraint — showing the effect is algorithm-agnostic rather than specific to PPO.
Regarding specialized baselines and generalization (Reviewer UyBa): we aligned our data construction with CoSAlign and evaluated on 1,000 sampled examples: our method remains the best (PSS 0.5470 vs. SFT 0.5238, GRPO 0.5257, PPO 0.5399, with the highest risk, empathy, and personalization scores). We further evaluated all methods on the aligned CASTLE_en: the method ordering is preserved on all five metrics (our method achieves the best PSS 0.6996, risk, and empathy scores, and is on par with PPO on personalization), and all methods reach JC = 0 on the jailbreak subset, so the reported differences reflect subtler, context-dependent risk rather than explicit jailbreak compliance.
Regarding comparison with conventional constrained safety alignment (Reviewers NnSE and DSMT): under the same experimental setting, SafeRLHF yields PSS 0.6145 with 230 jailbreaks, while our method reaches PSS 0.75 with 8 jailbreaks and higher risk and empathy scores.
Finally, we would like to state that our primary motivation is to specify what "safe and helpful" means when helpfulness is user-specific and the safety boundary is persona-sensitive. Our formulation makes two contributions. First, it turns an under-specified objective into an explicit constrained-optimization problem in which general safety is a feasibility requirement rather than a tradable utility component. Second, and most importantly, the safety signal is analytic, deterministic, and computable before generation — two forward passes and a closed-form softmax Jacobian — which keeps the policy's next-token distribution inside the locally linear neighborhood of safe base behavior, while leaving absolute safety assessment to direct evaluation outside the optimization loop.