acceptodds
Under review as a conference paper at ICLR 2027

Beyond Alignment: Enhancing LLM Safety Robustness via Zeroth-Order Optimization

Abstract

Safety alignment for large language models (LLMs) aims to reduce harmful or unsafe behavior while preserving general utility. However, recent findings reveal that alignment effects can be fragile: lightweight post-alignment manipulations, such as parameter noise, activation noise, or quantization, can easily weaken the intended safety behavior. Prior efforts to improve robustness have primarily focused on data curation, modified alignment objectives, and safety-critical parameter identification, leaving the role of the optimizer itself largely unexplored. In this paper, we are the first to study the robustness of safety alignment from the perspective of the base optimizer. This optimizer-centric view naturally points to zeroth-order optimization, which provides a robustness-oriented signal by evaluating safety alignment under perturbations. Based on this insight, we propose a hybrid framework that first performs standard first-order safety alignment and then applies zeroth-order refinement to improve robustness. Both theoretically and empirically, we show that only a few zeroth-order refinement steps can enhance robustness while preserving safety alignment. We further improve the efficiency of zeroth-order refinement by exploiting its inherent perturbation-based evaluations to estimate layer-wise robustness sensitivity, enabling the refinement process to concentrate updates on robustness-critical layers with modest training overhead.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety Alignment, Robustness of Safety Alignment, Zeroth-order Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZAJKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors investigate how to maintain the safe behavior of a language model following quantization or the addition of noise to its weights or activations. They propose an additional "zeroth-order optimization" stage (ZO stage), performed after standard FT on safe responses. ZO stage evaluates updates based on changes in the error resulting from random perturbations to the weights. To reduce the computational cost of these updates, the process is restricted to selected layers.

Experiments were conducted using Llama-3-8B-Instruct and Qwen2-7B-Instruct. This additional ZO based training generally reduces the Attack Success Rate following quantization. The authors also provide a theoretical justification for the method and compare it against continued standard fine-tuning and alternative strategies for layer selection.

Critical strengths
  1. The results show a notable improvement in the context of quantization. For instance, with the Llama model the application of ZO reduces the average ASR from 5.45% to 4.25%, while the models performance metrics remain nearly unchanged.

  2. The authors include several valuable baseline comparisons, such as continuing standard training, selecting layers at random, employing alternative selection criteria, and applying ZO on top of existing methods.

Critical weaknesses
  1. I have a question regarding the variance estimate in Lemma B.4. For the stated Gaussian estimator, the variance does not, in general, tend to zero as β\beta decreases. This can be easily seen in an one dimensional example where f(x) = x. The estimate is v2v^2, which has a variance of 2, when the right side of the lemma gives us 64β264\beta^2. This specific example does not satisfy the PL condition required by the theorems, so it is essential to explicitly state the additional assumptions that allow for the derived bound. Since this bound is used in the proofs of both main theorems, the theoretical guarantees are not established as written. Additionally, the dimensionality factors used in Eq. 1, Algorithm 1, and the appendix need to be made consistent.

  2. The experimental findings are difficult to verify due to inconsistencies between the numerical data and the accompanying descriptions. For instance, some values ​​in Figure 6 correspond to the average of two evaluators, even though the figure is labeled "HarmBench ASR" (not "HarmBench+LLamaGuard ASR"). Table 3 and the surrounding text provide conflicting figures for the FO duration 4587.2 seconds VS 1146.8 seconds. Additionally, the claim that the layer-selection approach consistently yields the best results is contradicted by Table 2, where in 5 out of 9 SGD settings this approach underperforms compared to full-model ZO. Consistent results and comparisons based on an identical computational budget are required to reliably assess the contribution of the proposed method.

Questions for authors
  1. Can you correct the variance estimate in Lemma B.4 and indicate which theoretical results remain valid following this correction? Additionally, please clarify whether the theoretical guarantees extend to the actual algorithm, which updates only a selected subset of layers.

  2. Could you explain the discrepancies between Figures 6-7 and Tables 13-16, and provide consistent, original results? Specifically, what metric is displayed in Figure 6, and why do the evaluations for the same baseline model differ between Tables 13 and 14?

  3. What is the total computational cost of the additional ZO training, including the layer-selection process? Furthermore, why does the FO computation time differ between Table 3 and the main text? Given 64 directions per step, an equal number of FO and ZO updates does not imply equivalent computational costs.

  4. How do you account for the significant divergence between evaluators under high-noise conditions? For instance, in Table 1, with σ=0.30\sigma=0.30, HarmBench reports an ASR of 3.93%, whereas LlamaGuard reports 89.77%.

  5. Is there a comparison against standard gradient-based training, applied with noise added to the weights of the same selected layers, at a comparable computational cost?

3 comments

We thank the reviewer for the detailed feedback on theoretical validity, layer selection, experimental consistency, and computational cost. We address the experimental concerns below.

W1 & Q1: Estimator normalization and theoretical guarantees

Thank you for pointing out the inconsistent dimension factors and the counterexample to Lemma B.4. We will unify the Gaussian estimator’s normalization and replace the original variance statement with a bound that explicitly accounts for directional randomness:

\le 4(d+4)\|\nabla f_\beta\|^2 +3\beta^2h^2(d+4)^3,$$ where $h$ is the gradient-Lipschitz constant. This bound follows from Nesterov and Spokoiny (2017, Lemma 5) through Gaussian symmetrization; Appendix B will include a direct proof for our two-sided estimator. The **gradient-dependent term remains as $\beta\to0$** and must be retained in the convergence analysis. For an average of $q$ independent directions, let $$A_q=1+\frac{4d+15}{q}, \qquad C_q=A_q\nu^2+\frac{3\beta^2h^2(d+4)^3}{q},$$ where $\nu^2$ bounds shared-batch smoothed-gradient variance. Under the revised smoothness assumptions and PL condition on $f_\beta$, we obtain $$\Delta_T \le e^{-\mu\eta T}\Delta_0+\frac{h\eta C_q}{2\mu},$$ $$0<\eta\leq\min\lbrace \frac{1}{hA_q},\frac{1}{2\mu}\rbrace,$$ with $\Delta_t=\mathbb E[f_\beta(\theta_t)-f_\beta^\star]$. Thus, the additional directional contribution changes the admissible stepsize while permitting **linear contraction to an explicit error neighborhood**. We will update Theorem 4.2 accordingly and derive Theorem 4.4’s expected-loss improvement condition from this corrected bound. Both revised statements concern full-parameter refinement. --- # **W2 & Q2: Figure labels and baseline consistency** Sorry for the inaccurate labeling and reporting errors. **Figure 6&7 averages the HarmBench and LlamaGuard-3 evaluator ASRs on the HarmBench subset.** Its axis now reads **"Average ASR ↓"**, and its caption specifies **"HarmBench + LlamaGuard3"**. We also corrected the SGD baseline entries in **Table 14** to match the corresponding reference results in Table 13. **Layer-selection claim.** Thank you for highlighting this overstatement. Selection yields **mixed results after SGD**, but improves ASR in **all evaluated Adam settings**. Adam’s coordinate-wise adaptive scaling may **balance optimization** across layers, making targeted refinement sufficient while avoiding unnecessary updates elsewhere. SGD’s shared stepsize may leave more **uneven refinement** needs, so restricting updates can exclude useful adjustments in some settings. We will revise the discussion to present this as a possible explanation for the optimizer-dependent results. --- # **W2 & Q3: Runtime and direction counts** Sorry for the inconsistent timing reports and unclear cost breakdown. **Timing correction.** Table 3 and Section 5.6 now consistently report **4587.2 s for FO**, replacing the incorrect 1146.8 s denominator. Full-parameter ZO takes **152.11 s**, approximately **3.3%** of FO time. **Direction counts.** Here, $n$ denotes the **number of random perturbation directions per ZO update**. The main robustness experiments use $n=64$; the efficiency experiments use $n=1$. Across eight common settings, the results are: | Evaluator | SGD-400 ASR (%) | ZO, $n=64$ | ZO, $n=1$ | |---|---:|---:|---:| | LlamaGuard3 | 9.9056 | 9.0804 | 9.6805 | | HarmBench | 3.7478 | 3.0414 | 3.5446 | Relative to the same FO checkpoint, $n=1$ retains **27.3% and 28.8%** of the respective $n=64$ ASR reductions. It produces similar average gains across the six continuous-noise settings, but worsens quantization ASR. Thus, $n=1$ represents a lower-cost option with partial aggregate gains, while $n=64$ provides broader robustness improvements. **Layer-selection cost.** At $n=1$, full-parameter and top-4-layer ZO take **152.11 s and 56.96 s**, respectively; these are alternative configurations. Offline scoring takes **3398.93 s** and is excluded from the refinement time. Scores are computed once on the original Llama-3-8B-Instruct and are largely consistent across the datasets examined, supporting reuse of the ranking across tasks sharing the base model and scoring configuration. Selected-layer ZO therefore adds **3455.89 s** on first use, or an average of **3398.93/$R$ + 56.96 s per task** when scoring is shared across $R$ tasks with the same refinement budget. We will report these costs separately. These timings describe $n=1$, not $n=64$; matching update counts does not match computational cost.

Q4: Evaluator disagreement under severe noise

Thank you for highlighting the evaluator disagreement. The disagreement occurs under severe generation degradation: perplexity rises from 8.2841 to 257.6617, while lm-eval falls from 0.6776 to 0.3313. Incoherent or corrupted outputs may be treated differently: HarmBench may be more permissive toward outputs lacking recognizable harmful assistance, whereas LlamaGuard may be more conservative toward suspicious or corrupted content. We use multiple judges to expose such differences and will clarify that their results should be interpreted alongside generation quality and utility.


Q5: Perturbation-aware FO baseline

Thank you for suggesting this comparison. Vaccine provides a representative perturbation-aware FO baseline: it explicitly trains under adversarial activation perturbations using first-order gradients. Figure 4 and Table 11 show that subsequent ZO refinement further improves robustness. These results demonstrate that ZO provides additional benefits even after perturbation-aware FO training, supporting the value of the proposed ZO stage.

AuthorsOpenReview ↗

Thank you & Looking forward to your feedback

Thank you again for your careful review and constructive feedback. We would appreciate your thoughts on our rebuttal. For ease of reference, we summarize our responses below:

  • W1 & Q1, theoretical guarantees. We provided a corrected estimator bound that retains directional randomness and outlined the corresponding convergence analysis for full-parameter ZO, with linear contraction to an explicit error neighborhood.
  • W2 & Q2, reporting and layer selection. We clarified the evaluator averages in Figures 6–7 and corrected the baseline inconsistency. We also acknowledged the mixed layer-selection results after SGD and distinguished them from the improvements across the evaluated Adam settings.
  • W2 & Q3, computational cost. We corrected the FO runtime, separated layer-scoring and ZO training costs, and provided the n=1n=1 comparison, which retains approximately 27%–29% of the aggregate ASR reduction achieved by n=64n=64. We explicitly distinguished these efficiency measurements from the n=64n=64 robustness experiments.
  • Q4, evaluator disagreement. We discussed how severe generation degradation may contribute to divergent judgments and explained why ASR should be interpreted alongside generation quality and utility.
  • Q5, perturbation-aware FO baseline. We highlighted the Vaccine experiments, where subsequent full-parameter ZO further improves robustness after perturbation-aware FO training.

We hope these clarifications are helpful and would welcome your feedback on whether they address your concerns. Please let us know if any points require further explanation during the discussion period.

Reviewer_7GPYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes refining a safety-aligned language model with a short zeroth-order (ZO) optimization stage. Layers are selected using sensitivity to parameter noise and quantization, and finite-difference updates are restricted to the selected layers. Experiments on Llama-3-8B-Instruct and Qwen2-7B-Instruct evaluate harmful-response rates under quantization, weight noise, and activation noise, alongside perplexity and general capability. The paper also presents convergence and robustness guarantees for a smoothed objective and compares refinement after SGD, AdamW, Vaccine, and ASFT.

Critical strengths
  • The empirical question is useful, and the experiments distinguish clean safety alignment from robustness after deployment perturbations. Evaluating three perturbation families, three safety judges, and utility helps expose tradeoffs that a clean refusal score alone would miss. The paper includes continued-FO controls, several starting alignment procedures, and random or alternative layer-selection controls. For example, Table 6 reports W4A4 HarmBench ASR decreasing from 0.0428 after SGD to 0.0315 after ZO while clean ASR remains 0.0113. These results support a potentially useful refinement procedure, independently of whether the theoretical justification holds. Reporting the modest or mixed gains in several appendix settings is also valuable.
Critical weaknesses
  1. The theoretical guarantees rely on an invalid estimator lemma as written. Definition 3.1 uses the usual Gaussian two-sided estimator without a dimension multiplier, but Algorithm 1 introduces d_S and Appendix B.2 introduces d for the same unnormalized Gaussian directions. The latter estimator has expectation d times the gradient of the Gaussian-smoothed objective, contrary to Lemma B.4. More seriously, its variance bound cannot generally vanish as beta tends to zero. In one dimension, the Lipschitz, smooth function f(theta)=theta gives estimator v^2 for v~N(0,1), with mean 1 and variance 2 for every beta, contradicting the claimed bound 64*beta^2 when beta<1/sqrt(32). This is a counterexample to the stated estimator lemma, not directly to the full PL theorem. Since the proofs of Theorems 4.2 and 4.4 explicitly use that variance bound, their guarantees are unsupported as presented; removing the dimension multiplier alone does not repair them.

  2. The efficiency comparison does not yet establish the claimed advantage over continued FO at a comparable computational budget. Table 4 uses 64 directions per ZO update, so matching optimizer-step counts does not match function evaluations or elapsed time. The existing continued-FO control is useful, but Figure 5 labels 400/410 steps whereas its caption and Section 5.7 state 100/110. Similarly, Table 3 and Figure 3 report 4587.2 seconds for FO, while Section 5.6 uses 1146.8 seconds; the same 152.11-second ZO stage therefore yields approximately 3.3% or 13.3% overhead. This is consequential for a contribution advertised as an efficient alternative to further FO training. The reported memory benefit and robustness improvements remain evidence, but the computation/performance tradeoff requires a consistent accounting.

Questions for authors
  1. Which sampling distribution and normalization were used in the implementation? Can the estimator lemma and resulting guarantees be corrected to include the nonvanishing directional variance, and do the revised results still justify the masked refinement procedure?

  2. Which FO checkpoints and timing setup underlie Figure 5 and Table 3? Please reconcile the step counts and runtime denominators, state whether layer scoring is included, and clarify whether available measurements compare continued FO and ZO at matched elapsed time or function-evaluation cost.

3 comments

We thank the reviewer for the constructive comments and valuable suggestions. We address the concerns as follows.

W1: Estimator normalization and theoretical guarantees

Thank you for identifying the normalization inconsistency and the missing directional-variance contribution. We will standardize the Gaussian estimator throughout the paper without an additional dimension multiplier, and replace Lemma B.4 with a finite-smoothing second-moment bound:

\le 4(d+4)\|\nabla f_\beta\|^2 +3\beta^2h^2(d+4)^3,$$ where $h$ is the gradient-Lipschitz constant. This follows from standard Gaussian smoothing analysis (Nesterov and Spokoiny, 2017, Lemma 5); we will include a direct derivation for the symmetric estimator in Appendix B. Importantly, the corrected bound retains the gradient-dependent directional contribution, which does not vanish as $\beta\to0$. This contribution can nevertheless be absorbed into the descent term through a revised stepsize restriction. Specifically, for $q$ independent directions, define $$A_q=1+\frac{4d+15}{q}, \qquad C_q=A_q\nu^2+\frac{3\beta^2h^2(d+4)^3}{q},$$ where $\nu^2$ bounds shared-batch smoothed-gradient variance and is zero for deterministic evaluations. Under explicit smoothness assumptions and a PL condition on $f_\beta$, the corrected full-parameter analysis gives $$\Delta_T \le e^{-\mu\eta T}\Delta_0 + \frac{h\eta C_q}{2\mu} \qquad 0\le\eta\leq\min \lbrace \frac{1}{hA_q},\frac{1}{2\mu}\rbrace .$$ We will revise Theorems 4.2 and 4.4 and their appendix proofs accordingly, including the stepsize and improvement conditions. The revised analysis establishes full-parameter convergence and expected perturbed-loss improvement with corrected assumptions and constants, rather than relying on a variance bound that vanishes with $\beta$. --- # **W2 & Q2: Checkpoints and computational cost** Sorry for the inconsistent step counts and unclear computational-cost reporting. - **Checkpoint correction.** **Section 5.7 and Figure 5** now specify **400-step FO, 410-step FO, and 400-step FO + 10-step ZO**. Both continuations start from SGD checkpoint 400 of a run configured for 500 steps. Figure 5 matches optimizer-update counts, not elapsed time or function-evaluation cost. - **Timing correction.** The correct FO runtime is **4587.2 s**, now consistent across Table 3, Figure 3, and Section 5.6. The previous 1146.8 s denominator was incorrect. - **Direction counts and trade-off.** Here, $n$ denotes the **number of random perturbation directions per ZO update**. Main robustness experiments use $n=64$; efficiency experiments use $n=1$. The following table reports average ASRs across **eight common perturbation settings**, relative to the same SGD-400 checkpoint: | Evaluator | SGD-400 ASR (%) ↓ | ZO, $n=64$ ASR (%) ↓ | ZO, $n=1$ ASR (%) ↓ | |---|---:|---:|---:| | LlamaGuard3 | 9.9056 | 9.0804 | 9.6805 | | HarmBench | 3.7478 | 3.0414 | 3.5446 | Across eight common settings, $n=1$ retains **27.3% of the LlamaGuard3 ASR reduction and 28.8% of the HarmBench reduction** achieved by $n=64$, relative to the same SGD-400 checkpoint. It produces similar average gains across the six continuous-noise settings, but worsens quantization ASR. Thus, $n=1$ represents a lower-cost option with partial aggregate gains, while $n=64$ provides broader robustness improvements. - **Cost breakdown.** At $n=1$, full-parameter ZO takes **152.11 s (3.3% of FO time)**, while top-4-layer ZO takes **56.96 s**. These are alternative configurations. Offline layer scoring takes **3398.93 s** and is excluded from the refinement time. Scores are computed once on the original Llama-3-8B-Instruct and are largely consistent across the datasets examined, supporting reuse across tasks sharing the base model and scoring configuration. Thus, selected-layer ZO adds **3455.89 s** on first use, or **3398.93/$R$ + 56.96 s per task** when scoring is shared across $R$ tasks with the same refinement budget. We will distinguish these preprocessing and refinement costs explicitly. --- # **Q1: Sampling, variance, and masked refinement** Thank you for highlighting the estimator normalization, directional variance, and applicability to masked updates. We will clarify the implemented sampling and normalization and consistently express the Gaussian estimator **without an additional dimension multiplier**. The corrected **second-moment bound** retains gradient-dependent directional variance, which can be absorbed through a stricter stepsize restriction, yielding revised convergence and expected perturbed-loss improvement guarantees. For **masked refinement**, we will add a conditional extension for a fixed selected subspace, replacing $d$ with $d_S$ and explicitly assuming **PL on the restricted smoothed objective**. This assumption does not follow from full-space PL. The resulting guarantee concerns expected perturbed-loss improvement within the selected subspace.

thanks for acknowledging the estimator issue and providing a concrete route to correction. retaining the gradient-dependent directional contribution addresses the specific problem with the original vanishing-variance claim. however, the proposed replacement bound and convergence statements still need the complete derivation, with consistent normalization and assumptions, before i can regard the theoretical concern as resolved. please also state exactly which sampling and normalization the reported implementation used; the response describes how the equations will be standardised but does not unambiguously identify the original implementation.

the distinction between full-space and restricted-subspace PL is appropriate. please make clear in the revised theorem statements that the masked result is conditional on restricted PL, which is an additional assumption, and does not itself justify the layer-selection criterion.

the runtime clarification is useful, but it also materially narrows the efficiency claim. the 152.11 s and 56.96 s figures describe n=1, whereas the main robustness results use n=64. the n=1 configuration retains only about 27-29% of the aggregate ASR reduction and worsens quantization ASR. moreover, selected-layer refinement costs 3455.89 s on first use when scoring is included. these should be presented as distinct operating points, with preprocessing amortisation explicit.

the remaining empirical question is the comparison to continued FO at matched elapsed time for the configuration supporting the main robustness claim. Figure 5 still matches update counts only. if such measurements are unavailable, please state that the current evidence establishes robustness gains and memory advantages, while the relative robustness-versus-runtime advantage remains unestablished. these clarifications improve the paper, but the theory and comparable-budget efficiency concerns are not yet fully resolved.

Follow-up: Matched-time comparison and efficiency claims

Thank you for emphasizing the need for an elapsed-time-matched comparison. Sorry that our previous update-count comparison did not directly address this concern. We have now conducted an additional comparison using approximately 3560 s of additional training time, starting from the same SGD-400 checkpoint. We compare 5 ZO updates with n=64n=64 against 311–312 continued FO updates at three learning rates.

MethodAdditional updatesTraining time (s)Clean ASR ↓Weight ASR ↓Activation ASR ↓Quantization ASR ↓Macro-average ASR ↓
SGD-400 + ZO, n=64n=6453558.370.9004.12610.1843.5825.964
SGD-400 + FO, lr=10−410^{-4}3113562.910.8634.48010.4093.6956.195
SGD-400 + FO, lr=3×10−43\times10^{-4}3123560.050.7505.49712.3754.1647.345
SGD-400 + FO, lr=7.5×10−47.5\times10^{-4}3113559.010.8635.17611.4124.1826.924

All ASRs are reported as percentages. The macro-average assigns equal weight to the weight-noise, activation-noise, and quantization ASRs; clean ASR is reported separately.

At comparable additional training time, ZO achieves the lowest ASR in all three perturbation families among the tested configurations. Compared with the best FO setting by macro-average ASR (lr=10−410^{-4}), ZO reduces the average from 6.195% to 5.964%, an absolute reduction of 0.231 percentage points and a relative reduction of approximately 3.73%. Clean ASR remains close, at 0.900% versus 0.863%, although continued FO achieves the lower value.

These results provide evidence of a robustness advantage for n=64n=64 ZO over continued FO at a matched additional-training-time budget in this setting. This five-update ZO experiment directly supplements the original update-count comparison. We will include these results and distinguish this operating point from the previously reported n=1n=1 configurations.

Preprocessing and refinement costs will be reported separately. The 3398.93 s layer-scoring cost must be included when assessing first-use costs for selected-layer refinement; amortisation applies only when the same ranking is reused across tasks. The 152.11 s and 56.96 s measurements remain specific to n=1n=1 and will not be used to characterize the runtime of n=64n=64.

Reviewer_Tfm8OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper recognizes the challenge of achieving robust safety alignment in supervised fine-tuning of LLMs; specifically, relying solely on first-order (FO) optimizers tends to produce sharp or overfitted boundaries in parameter or activation space. To address this, the authors propose using zeroth-order (ZO) optimization to replace late-stage first-order optimization when fine-tuning a small subset of vulnerable layers in LLMs. The proposed method is evaluated using Attack Success Rate (ASR), where attacks correspond to post-alignment perturbations such as weight noise, activation noise, or quantization that can induce harmful outputs. Ablation results show that using only 10 steps of zeroth-order optimization following first-order optimization reduces ASR by 0.37% to 1.2%. Furthermore, the proposed vulnerable-layer selection technique reduces the ASR by 0.7% to 1.8% compared to fine-tuning all layers.

Critical strengths
  1. The introduction is well-written and motivating. Both the abstract and introduction tell a clear story and make a positive initial impression. In particular, using zeroth-order optimization to reduce first-order optimization's sharp and overfitted output is a sensible design choice.

  2. The results, while marginal, are encouraging and evaluated across a comprehensive benchmark suite.

Critical weaknesses

Major

  1. The method is not fully zeroth-order optimization as the title and abstract suggest. At the end of the introduction, the authors state: "Although ZO can improve alignment robustness, its slower and less stable convergence limits its effectiveness as a standalone alignment optimizer. We therefore combine FO and ZO in a hybrid framework, using FO to establish safety alignment and ZO to further enhance robustness." Later, in Sec. 4, the authors explicitly introduce it as an FO-to-ZO framework. Similarly, Stage I of the proposed algorithm (Alg. 1) is standard first-order fine-tuning, which makes the framing of the paper confusing. The authors should revise the title, abstract, and algorithm description to be more consistent and avoid overclaiming.

  2. The proposed fine-tuning algorithm (Alg. 1) introduces many hyperparameters, such as step sizes. In addition, the layer-wise robustness sensitivity score introduces a new parameter to trade off between different perturbation effects. However, the paper does not explain the guiding principles for choosing these values.

  3. It is unclear how the ASR is computed given continuous input specifications such as model weight perturbation and activation perturbation (e.g., how the perturbations are sampled, how many samples taken, and how evaluation variance is handled).

Minor

  1. The paper is not reproducible, as no code was shared despite claims in the Reproducibility Statement.

  2. There is an inconsistency in the experimental results. While the caption and main text in Sec. 5.7 state that first-order optimization ran for 100 or 110 steps, the legend of Fig. 5 indicates that it ran for 400 or 410 steps.

Questions for authors
  1. The paper lists parameter values but does not explain how they were chosen. Could you provide a parameter sensitivity study and explain the guiding principles for choosing these values?

  2. Could you explain how the ASR is computed across the model weight perturbation space? Since empirical sampling does not provide formal guarantees, could you provide a sound bound on the attack success rate, using ideas similar to those in [*]?

Reference [*] Suresh, Tarun, et al. "BEAVER: An Efficient Deterministic LLM Verifier." arXiv preprint arXiv:2512.05439 (2025).

5 comments

We thank the reviewer for the constructive comments and for recognizing the motivation and benchmark coverage of our work. We address the concerns below.


W1: Clarity of the hybrid framework

Thank you for raising this concern. We agree that the title could more clearly convey the post-alignment refinement setting and will revise it accordingly. The abstract frames our study as investigating robustness “from the perspective of the base optimizer” and explicitly states that we propose “a hybrid framework that first performs standard first-order safety alignment and then applies zeroth-order refinement to improve robustness.” This hybrid design is consistently described in the introduction, Section 4, and Algorithm 1.


W2 & Q1: Hyperparameter selection

Sorry for not clearly explaining the hyperparameter selection procedure. We have added a local grid-search table and selection criteria to Appendix A.1, covering FO learning rates and momentum, ZO perturbation scales, direction counts, refinement steps, and normalization choices. We tuned FO first, followed by ZO from the selected checkpoints, weighting HarmBench and LlamaGuard-3 equally and checking clean safety and general utility.

For layer selection, we sum the equally weighted percentile ranks of weight, activation, and quantization sensitivity. Ranking makes the scales comparable; equal weighting avoids additional tunable aggregation weights without assuming a preference among perturbation families.


W3 & Q2: ASR evaluation and robustness guarantees

Sorry for the unclear description of the evaluation protocol. We clarify the sampling and aggregation procedure below.

Sampling. Weight noise is relative to each tensor's RMS:

W~j=Wj+σRMS⁡(Wj)Zj,Zj∼N(0,I).\widetilde W_j=W_j+\sigma\operatorname{RMS}(W_j)Z_j,\qquad Z_j\sim\mathcal N(0,I).

We perturb floating-point tensors with more than one dimension, excluding one-dimensional parameters such as biases and normalization scales. Each seed defines one jointly perturbed model, with deterministically derived random streams for different tensors. Activation noise is injected at every Transformer block input:

H~l=Hl+αStd⁡d(Hl)⊙Zl,Zl∼N(0,I),\widetilde H_l=H_l+\alpha\operatorname{Std}_{d}(H_l)\odot Z_l,\qquad Z_l\sim\mathcal N(0,I),

where the standard deviation is computed along the hidden dimension. Generation uses greedy decoding (do_sample=False) with at most 512 new tokens.

ASR and variance. For each perturbation strength, we evaluate three independent seeds (42, 43, 44). For each seed, ASR is the fraction of evaluated completions labeled unsafe by the specified judge; we report the mean across seeds. Combined dataset results are weighted by sample count.

BEAVER. Thank you for suggesting this reference. BEAVER bounds output-sequence probabilities for a fixed model and prompt. Our evaluation measures unsafe greedy outputs under random weight or activation perturbations. Applying BEAVER separately to sampled models would certify only those instances, not the unobserved continuous perturbation space. An extension requires sound treatment of continuous perturbations alongside token-space exploration. We therefore distinguish our statistical robustness assessment from deterministic certification, while recognizing BEAVER as a promising foundation for such an extension.


W4: Code availability

Sorry for the code-availability issue. Our code has now been uploaded to the anonymous repository, and the link is included in the revised Reproducibility Statement.


W5: Training-step inconsistency

Sorry for the inconsistent step counts. We have corrected Section 5.7 and the Figure 5 caption to match the actual comparison: 400-step FO, 410-step FO, and 400-step FO followed by 10-step ZO. The original references to 100/110 steps were incorrect.

W1: Clarity of the hybrid framework

We sincerely thank the reviewer again for this valuable suggestion. Following the reviewer's recommendation, we have revised the title from "Beyond Alignment: Enhancing LLM Safety Robustness via Zeroth-Order Optimization" to "Enhancing LLM Safety Robustness via Hybrid First- and Zeroth-Order Optimization" to more accurately reflect the hybrid nature of our approach. We have also renamed Algorithm 1 from "Sensitivity-Guided ZO Robustness Refinement" to "Sensitivity-Guided Hybrid FO-to-ZO Robustness Refinement" to better reflect the complete optimization procedure. We appreciate the reviewer's careful reading and constructive feedback, which have helped us improve the accuracy and consistency of the manuscript.


W2 & Q1:Clearer explanation on hyperparameter selection

We would like to provide a clearer explanation of the rationale behind our hyperparameter choices. We have added the search ranges and selection criteria to Appendix A.1. Our guiding principle is to improve perturbation robustness while preserving clean safety and general utility. We tune FO first to establish safety alignment, then tune ZO from the selected checkpoint, weighting HarmBench and LlamaGuard-3 equally. We further examine two key parameters below.

Perturbation scale μ\mu. This parameter controls the neighborhood explored by ZO. An excessively small scale can make finite-difference loss changes difficult to resolve numerically, whereas an excessively large scale may produce updates less suited to local refinement. The following results illustrate this trade-off; the average assigns equal weight to the three perturbation families.

μ\muWeight ASR ↓Activation ASR ↓Quantization ASR ↓Average ASR ↓
0.0084.80710.8493.2886.315
0.0204.77911.0053.0886.291
0.0504.80711.0363.9886.610

Among the tested values, μ=0.020\mu=0.020 achieves the lowest average ASR, although μ=0.008\mu=0.008 performs slightly better under activation noise. Increasing μ\mu to 0.050 worsens the average ASR, particularly under quantization. These results support a moderate perturbation scale for the evaluated configuration.

Directions per update nn. Here, nn denotes the number of random perturbation directions per ZO update. Increasing nn reduces directional sampling variance but increases function-evaluation cost. Across eight common perturbation settings, we obtain:

EvaluatorSGD-400 ASR (%) ↓ZO, n=1n=1 ASR (%) ↓ZO, n=64n=64 ASR (%) ↓
LlamaGuard39.90569.68059.0804
HarmBench3.74783.54463.0414

Relative to the same FO checkpoint, n=1n=1 retains 27.3% and 28.8% of the respective ASR reductions achieved by n=64n=64. Compared with n=64n=64, n=1n=1 achieves similar average ASR across the six continuous-noise settings but higher ASR under quantization. This motivates using n=64n=64 for the main robustness experiments and considering n=1n=1 as a lower-cost option with partial aggregate gains.

Layer-selection weights. Directly weighting the raw sensitivity scores would require balancing their different scales and searching over additional weight combinations. We therefore convert the sensitivities to percentile ranks and sum them with equal weights, making the scales comparable and giving equal importance to weight noise, activation noise, and quantization. This provides a simple default without expanding the hyperparameter search space. In practical applications, the ranks can instead receive different weights to reflect the perturbations most relevant to the deployment setting.

The authors acknowledged the initial submission overclaims, inconsistent, and released the code late.

My key concerns remain:

  • Renaming the paper to "Hybrid First- and Zeroth-Order Optimization" still overclaims as it isn't genuinely hybrid. The approach is overwhelmingly standard first-order fine-tuning (400 steps) with only a brief ZO refinement (10 steps) on a few layers.
  • The proposed method fundamentally introduces many hyperparameters, which is trivial. A.1 only lists candidates hyperparameters without evidence about parameter sensitivity and choices are optimal.
  • Compute ASR score with only 3 weight perturbations samples (3 random seeds) is insufficient to capture variance across an LLM's massive parameter space, leaving the reported robustness gains statistically flaw.

Overall, this revised version still does not meet the standard expectation for ICLR, so I maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.