acceptodds
Withdrawn from ICLR 2027

HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Safety alignment in large language models can be fragile under fine-tuning, as even benign task adaptation may increase harmful compliance. Existing defenses either retrain or modify weights during or after fine-tuning, which requires control over training or trades safety against utility, or use model-agnostic guard classifiers, which are not tailored to a given checkpoint and run a separate large model on every prompt. We propose HyperSafe, a post-hoc, model-specific, and non-invasive framework that restores safety by generating a Safe Side Network (SSN) for each fine-tuned checkpoint. A hypernetwork maps layer-wise activation fingerprints, computed from a small set of calibration prompts, to the SSN parameters in a single forward pass. The SSN runs alongside the frozen fine-tuned model as a prompt-level safety classifier: harmful prompts are routed to refusal, while safe prompts are answered by the original model. HyperSafe targets safety degradation from benign fine-tuning rather than adversarial jailbreaks. The hypernetwork is trained once per backbone family; each new checkpoint then requires no gradient updates, no safety data, and no weight modification. On Qwen2-7B and LLaMA-3- 8B, HyperSafe reduces harmful response rates from 19–31% to below 1% on every held-out checkpoint (confirmed on LLaMA-3-8B by a second, independent judge), loses less than 1% task accuracy on average, and adds only 3.4% inference FLOPs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, safety alignment, fine-tuning, hypernetworks, inference-time safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JVjROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HyperSafe is a hypernetwork that reads layer-wise activation fingerprints from 50 calibration prompts of a fine-tuned checkpoint and generates, in one forward pass, the weights of a small Safe Side Network that routes harmful prompts to refusal and leaves safe prompts to the fine-tuned model untouched. It is meta-trained once per backbone across 18 fine-tuning domains and tested on held-out domains.

Critical strengths

The held-out-domain protocol (Appendix C, three splits) is a real zero-shot test, and the Shared-SSN ablation (Table 3) is the right control to check whether fingerprint-conditioning itself is doing anything.

The LoRA-to-full-fine-tuning transfer in Table 15 is a nice generalization check, since the generated SSN is reused unchanged and still reduces harmful outputs.

Critical weaknesses

Section 4 states every evaluated checkpoint is "an independent LoRA adapter for each dataset..." adapters are never merged with one another,'' so all checkpoints in Table 2 and every appendix are single-domain fine-tunes. Real deployed checkpoints are routinely fine-tuned on mixtures (math + instruction + code at once), which is the harder case for an activation-fingerprint method, would the fingerprint average out and look safer than it is when two domains' activation shifts partially cancel?

Section 5 says HyperSafe is not intended as a general jailbreak defense, but Table 6 reports 92.2% detection on WildJailbreak. The paper should clarify whether jailbreak robustness is within scope.

Held-out domain generalization is really held-out dataset-name generalization within an already-covered category. Table 9's splits each still leave 5-6 other instruction datasets (Dolly, OpenOrca, FLAN, OpenHermes, NoRobots, WizardLM) in training whenever one instruction dataset (Alpaca/OpenHermes/Dolly) is held out, and Table 13 shows that removing the entire instruction \emph{category} from training spikes harmful rate from 0.05%0.05\% to 4.88%4.88\% i.e.\ same-category coverage in training is exactly what is propping up the "zero-shot". Split A's held-out QA set (BoolQ) and Split C's held-out QA set (NQ-Open) likewise leave the same three QA in training. Is this domain generalization, or just name-level leave-one-out within a covered category?

Questions for authors

Can authors give per-checkpoint (not pooled) harmful rates and a significance test for the Shared SSN baseline?

Was OneShot re-validated against its originally reported numbers before being used as the main accuracy comparator?

Is HyperSafe being positioned as jailbreak-robust or not? Section 5 and Table 6 disagree.

Why is over-refusal only measured on one split of one model?

Reviewer_TsHzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

HyperSafe is proposed as a defense pipeline against safety degradation caused by benign fine-tuning of language models. Rather than modifying the fine-tuned model’s weights, HyperSafe attaches a Safe Side Network (SSN) that classifies prompts using the target model’s hidden states. The key idea is to amortize checkpoint-specific SSN training through a hypernetwork. During training, the authors collect pairs of fine-tuned checkpoints and independently trained SSNs. The hypernetwork then learns to generate SSN parameters from layer-wise activation fingerprints. Experiments report HyperSafe reduces harmful response rates from 19–31% to below 1% on every held-out checkpoint.

Critical strengths

two critical strengths:

  1. Safety degradation after benign fine-tuning is a common problem. many previous works try to solve it by modifying model weights. but hypersafe proposes a novel way, which amortizes checkpoint-specific SSN training through a hypernetwork. This method may serve as a useful reference for others.
  2. the experiment results show that the method reports consistently low harmful response rates.
Critical weaknesses
  1. I would suggest the paper provide more information about the evaluation details, such as the scoring procedure for open-ended tasks, calibration/probe-set composition, etc.
  2. the target responses, language-modeling head, and relationship to the final classification architecture are insufficiently presented.
Questions for authors

do you have a consistent end-to-end comparison with guard models? In particular, how about evaluating llama guard and your method on the same target checkpoints with the same output judge

Reviewer_UamPOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses safety degradation caused by benign fine-tuning of aligned language models. HyperSafe uses a hypernetwork to generate a checkpoint-specific Safe Side Network (SSN) from activation fingerprints extracted using a small set of domain-specific calibration prompts. The generated SSN reads the frozen model's hidden states and performs prompt-level safety classification, routing flagged requests to refusal while allowing other requests to be answered by the original fine-tuned model. The hypernetwork is trained once per backbone using fine-tuned checkpoints and their separately trained SSNs. Deployment on a new checkpoint requires calibration forward passes and weight generation, without additional optimization or safety data. Experiments on Qwen2-7B and LLaMA-3-8B report harmful response rates below 1% across the held-out checkpoints, with less than one percentage point of average task-accuracy loss.

Critical strengths
  1. HyperSafe offers a potentially useful way to protect fine-tuned checkpoints without modifying their weights or performing additional safety training at deployment. 2.Experiments across two backbones report substantial safety improvements with limited average utility loss, supported by over-refusal evaluation, a second judge, and inference-cost measurements.
Critical weaknesses

1.Insufficient evidence for the necessity of weight generation. The Shared SSN does not explicitly receive the checkpoint fingerprint, so its comparison with HyperSafe conflates the benefits of conditioning and parameter generation. Missing alternatives include a fingerprint-conditioned shared classifier and generating only a classification head or gates. Without these controls, the added value of the central design remains unclear. 2. Missing comparisons with closely related safety methods. SafetyLock, CAST, and SafeSwitch provide relevant alternatives involving transferable safety directions or selective intervention based on internal activations. Although their formulations differ, the paper neither evaluates applicable configurations nor adequately explains their exclusion, leaving HyperSafe’s practical advantage insufficiently established.

Questions for authors

1.How does HyperSafe compare with a fingerprint-conditioned shared classifier and variants that generate only the classification head or gates, under matched training conditions? 2.Why were SafetyLock, CAST, and SafeSwitch excluded, and how would their applicable configurations compare on the same checkpoints at matched benign-refusal rates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.