- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors investigate how to maintain the safe behavior of a language model following quantization or the addition of noise to its weights or activations. They propose an additional "zeroth-order optimization" stage (ZO stage), performed after standard FT on safe responses. ZO stage evaluates updates based on changes in the error resulting from random perturbations to the weights. To reduce the computational cost of these updates, the process is restricted to selected layers.
Experiments were conducted using Llama-3-8B-Instruct and Qwen2-7B-Instruct. This additional ZO based training generally reduces the Attack Success Rate following quantization. The authors also provide a theoretical justification for the method and compare it against continued standard fine-tuning and alternative strategies for layer selection.
Critical strengths
-
The results show a notable improvement in the context of quantization. For instance, with the Llama model the application of ZO reduces the average ASR from 5.45% to 4.25%, while the models performance metrics remain nearly unchanged.
-
The authors include several valuable baseline comparisons, such as continuing standard training, selecting layers at random, employing alternative selection criteria, and applying ZO on top of existing methods.
Critical weaknesses
-
I have a question regarding the variance estimate in Lemma B.4. For the stated Gaussian estimator, the variance does not, in general, tend to zero as decreases. This can be easily seen in an one dimensional example where f(x) = x. The estimate is , which has a variance of 2, when the right side of the lemma gives us . This specific example does not satisfy the PL condition required by the theorems, so it is essential to explicitly state the additional assumptions that allow for the derived bound. Since this bound is used in the proofs of both main theorems, the theoretical guarantees are not established as written. Additionally, the dimensionality factors used in Eq. 1, Algorithm 1, and the appendix need to be made consistent.
-
The experimental findings are difficult to verify due to inconsistencies between the numerical data and the accompanying descriptions. For instance, some values in Figure 6 correspond to the average of two evaluators, even though the figure is labeled "HarmBench ASR" (not "HarmBench+LLamaGuard ASR"). Table 3 and the surrounding text provide conflicting figures for the FO duration 4587.2 seconds VS 1146.8 seconds. Additionally, the claim that the layer-selection approach consistently yields the best results is contradicted by Table 2, where in 5 out of 9 SGD settings this approach underperforms compared to full-model ZO. Consistent results and comparisons based on an identical computational budget are required to reliably assess the contribution of the proposed method.
Questions for authors
-
Can you correct the variance estimate in Lemma B.4 and indicate which theoretical results remain valid following this correction? Additionally, please clarify whether the theoretical guarantees extend to the actual algorithm, which updates only a selected subset of layers.
-
Could you explain the discrepancies between Figures 6-7 and Tables 13-16, and provide consistent, original results? Specifically, what metric is displayed in Figure 6, and why do the evaluations for the same baseline model differ between Tables 13 and 14?
-
What is the total computational cost of the additional ZO training, including the layer-selection process? Furthermore, why does the FO computation time differ between Table 3 and the main text? Given 64 directions per step, an equal number of FO and ZO updates does not imply equivalent computational costs.
-
How do you account for the significant divergence between evaluators under high-noise conditions? For instance, in Table 1, with , HarmBench reports an ASR of 3.93%, whereas LlamaGuard reports 89.77%.
-
Is there a comparison against standard gradient-based training, applied with noise added to the weights of the same selected layers, at a comparable computational cost?
3 comments
Q4: Evaluator disagreement under severe noise
Thank you for highlighting the evaluator disagreement. The disagreement occurs under severe generation degradation: perplexity rises from 8.2841 to 257.6617, while lm-eval falls from 0.6776 to 0.3313. Incoherent or corrupted outputs may be treated differently: HarmBench may be more permissive toward outputs lacking recognizable harmful assistance, whereas LlamaGuard may be more conservative toward suspicious or corrupted content. We use multiple judges to expose such differences and will clarify that their results should be interpreted alongside generation quality and utility.
Q5: Perturbation-aware FO baseline
Thank you for suggesting this comparison. Vaccine provides a representative perturbation-aware FO baseline: it explicitly trains under adversarial activation perturbations using first-order gradients. Figure 4 and Table 11 show that subsequent ZO refinement further improves robustness. These results demonstrate that ZO provides additional benefits even after perturbation-aware FO training, supporting the value of the proposed ZO stage.
Thank you & Looking forward to your feedback
Thank you again for your careful review and constructive feedback. We would appreciate your thoughts on our rebuttal. For ease of reference, we summarize our responses below:
- W1 & Q1, theoretical guarantees. We provided a corrected estimator bound that retains directional randomness and outlined the corresponding convergence analysis for full-parameter ZO, with linear contraction to an explicit error neighborhood.
- W2 & Q2, reporting and layer selection. We clarified the evaluator averages in Figures 6–7 and corrected the baseline inconsistency. We also acknowledged the mixed layer-selection results after SGD and distinguished them from the improvements across the evaluated Adam settings.
- W2 & Q3, computational cost. We corrected the FO runtime, separated layer-scoring and ZO training costs, and provided the comparison, which retains approximately 27%–29% of the aggregate ASR reduction achieved by . We explicitly distinguished these efficiency measurements from the robustness experiments.
- Q4, evaluator disagreement. We discussed how severe generation degradation may contribute to divergent judgments and explained why ASR should be interpreted alongside generation quality and utility.
- Q5, perturbation-aware FO baseline. We highlighted the Vaccine experiments, where subsequent full-parameter ZO further improves robustness after perturbation-aware FO training.
We hope these clarifications are helpful and would welcome your feedback on whether they address your concerns. Please let us know if any points require further explanation during the discussion period.
We thank the reviewer for the detailed feedback on theoretical validity, layer selection, experimental consistency, and computational cost. We address the experimental concerns below.
W1 & Q1: Estimator normalization and theoretical guarantees
Thank you for pointing out the inconsistent dimension factors and the counterexample to Lemma B.4. We will unify the Gaussian estimator’s normalization and replace the original variance statement with a bound that explicitly accounts for directional randomness:
\le 4(d+4)\|\nabla f_\beta\|^2 +3\beta^2h^2(d+4)^3,$$ where $h$ is the gradient-Lipschitz constant. This bound follows from Nesterov and Spokoiny (2017, Lemma 5) through Gaussian symmetrization; Appendix B will include a direct proof for our two-sided estimator. The **gradient-dependent term remains as $\beta\to0$** and must be retained in the convergence analysis. For an average of $q$ independent directions, let $$A_q=1+\frac{4d+15}{q}, \qquad C_q=A_q\nu^2+\frac{3\beta^2h^2(d+4)^3}{q},$$ where $\nu^2$ bounds shared-batch smoothed-gradient variance. Under the revised smoothness assumptions and PL condition on $f_\beta$, we obtain $$\Delta_T \le e^{-\mu\eta T}\Delta_0+\frac{h\eta C_q}{2\mu},$$ $$0<\eta\leq\min\lbrace \frac{1}{hA_q},\frac{1}{2\mu}\rbrace,$$ with $\Delta_t=\mathbb E[f_\beta(\theta_t)-f_\beta^\star]$. Thus, the additional directional contribution changes the admissible stepsize while permitting **linear contraction to an explicit error neighborhood**. We will update Theorem 4.2 accordingly and derive Theorem 4.4’s expected-loss improvement condition from this corrected bound. Both revised statements concern full-parameter refinement. --- # **W2 & Q2: Figure labels and baseline consistency** Sorry for the inaccurate labeling and reporting errors. **Figure 6&7 averages the HarmBench and LlamaGuard-3 evaluator ASRs on the HarmBench subset.** Its axis now reads **"Average ASR ↓"**, and its caption specifies **"HarmBench + LlamaGuard3"**. We also corrected the SGD baseline entries in **Table 14** to match the corresponding reference results in Table 13. **Layer-selection claim.** Thank you for highlighting this overstatement. Selection yields **mixed results after SGD**, but improves ASR in **all evaluated Adam settings**. Adam’s coordinate-wise adaptive scaling may **balance optimization** across layers, making targeted refinement sufficient while avoiding unnecessary updates elsewhere. SGD’s shared stepsize may leave more **uneven refinement** needs, so restricting updates can exclude useful adjustments in some settings. We will revise the discussion to present this as a possible explanation for the optimizer-dependent results. --- # **W2 & Q3: Runtime and direction counts** Sorry for the inconsistent timing reports and unclear cost breakdown. **Timing correction.** Table 3 and Section 5.6 now consistently report **4587.2 s for FO**, replacing the incorrect 1146.8 s denominator. Full-parameter ZO takes **152.11 s**, approximately **3.3%** of FO time. **Direction counts.** Here, $n$ denotes the **number of random perturbation directions per ZO update**. The main robustness experiments use $n=64$; the efficiency experiments use $n=1$. Across eight common settings, the results are: | Evaluator | SGD-400 ASR (%) | ZO, $n=64$ | ZO, $n=1$ | |---|---:|---:|---:| | LlamaGuard3 | 9.9056 | 9.0804 | 9.6805 | | HarmBench | 3.7478 | 3.0414 | 3.5446 | Relative to the same FO checkpoint, $n=1$ retains **27.3% and 28.8%** of the respective $n=64$ ASR reductions. It produces similar average gains across the six continuous-noise settings, but worsens quantization ASR. Thus, $n=1$ represents a lower-cost option with partial aggregate gains, while $n=64$ provides broader robustness improvements. **Layer-selection cost.** At $n=1$, full-parameter and top-4-layer ZO take **152.11 s and 56.96 s**, respectively; these are alternative configurations. Offline scoring takes **3398.93 s** and is excluded from the refinement time. Scores are computed once on the original Llama-3-8B-Instruct and are largely consistent across the datasets examined, supporting reuse of the ranking across tasks sharing the base model and scoring configuration. Selected-layer ZO therefore adds **3455.89 s** on first use, or an average of **3398.93/$R$ + 56.96 s per task** when scoring is shared across $R$ tasks with the same refinement budget. We will report these costs separately. These timings describe $n=1$, not $n=64$; matching update counts does not match computational cost.