- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper evaluates safety in reasoning-tuned LLMs by examining both visible reasoning traces and final answers. It compares Qwen3 thinking modes at fixed weights and OLMo 3 training branches. It also proposes ESPG, combining evolution strategies, anchored weight decay, and a training-time safety prompt. On VibeThinker-3B, ESPG reduces the reported Full unsafe rate from 15.4% to 9.9% without the safety prompt at test time, while MATH-200 accuracy changes from 93.0% to 92.5%.
Critical strengths
-
The fixed-weight Qwen3 comparison provides useful evidence about the effect of enabling thinking. Examining visible reasoning traces also captures harmful content that final-answer-only evaluation may miss.
-
ESPG shows a statistically supported reduction in the reported unsafe rate, with little change in MATH-200 accuracy. This suggests some promise within the tested setting.
Critical weaknesses
-
The contribution and need for non-gradient optimization are not established. The safety observations largely extend issues already discussed in prior work. ESPG combines existing techniques without showing a clear advantage over well-tuned gradient-based alternatives, including parameter-efficient tuning. The GRPO comparison changes data size, data balance, and generation budget, so it does not isolate the optimizer. Eight A100 GPUs and many judge calls were used; lower training memory alone does not establish a practical cost advantage.
-
The evaluation does not provide sufficiently independent or broad evidence of safety improvement. Restoration uses one 3B model and one training seed per method. The same judge and closely related criteria are used for rewards and testing. Appendix A.6 also reports reuse of the test set during development. Without a fresh test set, independent judging, and over-refusal evaluation, the reported reduction does not convincingly establish reliable safety gains while preserving usefulness.
Questions for authors
-
Under matched data, rewards, and compute budgets, what advantage does ESPG offer over gradient-based safety tuning with comparable weight regularization? Please include a practical comparison of runtime, memory, and judge costs.
-
Can the safety improvement be confirmed on a fresh test set with an independent judge or human evaluation? Does it persist across training seeds and avoid increased refusal of benign requests?
-
Why were ES and ESPG evaluated at iteration 200 when their validation rewards peaked at iteration 150, while GRPO used its validation-selected checkpoint at 150? What happens under a consistent checkpoint-selection rule?
2 comments
(Cont'd) Response to Reviewer ZUV3
2 and Q2-Ans. Baseline comparisons and practical computational cost.
The GRPO comparison does not isolate the optimizer, as the reported configurations differ in training-set size, harmful/benign balance, and rollout budget. As explained in the response to Reviewer pZis (W2, point 4), smaller-data GRPO configurations did not achieve meaningful learning in the exploratory experiments, motivating the use of 1,000 training prompts instead of the 30 used by ESPG.
Revision: §5.1, GRPO experimental differences; Appendix B.3, training configurations; Appendix F.6, GRPO unique-prompt sweep.
It is observed that the reduced training-memory requirements do not necessarily imply lower overall computational cost. Appendix B.7 (compute table) already provides a quantitative comparison: ESPG requires approximately 3.7 × 10¹⁷ training FLOPs, compared with 4.4 × 10¹⁷ for GRPO, representing approximately 16% fewer estimated training FLOPs in the reported configurations. However, ESPG uses more generated responses (90,000 versus 19,200), and the experiments do not provide sufficient runtime or judge-cost measurements to establish a lower end-to-end cost.
Revision: §6, compute-trade-off paragraph; Appendix B.7, computational-cost table.
The revised manuscript clarifies these trade-offs and distinguishes data efficiency, training-memory requirements, computational cost, and overall runtime. Furthermore, the revised version's future directions include matched comparisons against fine-tuned gradient-based and parameter-efficient approaches to strengthen the evaluation.
Revision: §6, compute paragraph and concluding future-work paragraph.
W2. Evaluation independence and generalization
The present study establishes a proof of concept supported by 200 held-out harmful prompts, eight generations per prompt, two safety benchmarks, paired bootstrap analysis, and preserved mathematical reasoning performance (Section 5 and Appendix G). Further experiments can broaden the scope, but the current results are sufficient to establish the validity of the method.
Revision: §5.1, design and evaluation; §5.2, restoration-results table and capability paragraph; Appendices G and H.1.
Regarding judge dependence, please refer to the response to Reviewer pZis (W1, point 5), where the response provides additional cross-judge agreement results. These show high within-judge consistency and strong checkpoint-level rank correlations (0.95–0.98), while also revealing differences in absolute safety judgments.
Revision: Appendix I.1, agreement table, checkpoint-level comparison table, and judge-coverage limitations.
Note that the original 200 test prompts were not used for training, although earlier development experiments were evaluated on them, as disclosed in Appendix B.8. The revised version clearly stated the limitation and the distinction between development and independent confirmation.
Revision: Appendix B.2, prompt separation; Appendix B.8, run history; §6, limitations and future-work paragraphs.
Q3. Why were different checkpoints selected?
Q3-Ans. We have clarified this distinction in the response to Reviewer pZis (W1, point 3); briefly, the selected checkpoints are all at 150 iterations.
Revision: §5.1, design and evaluation; Appendix B.6, training curves; Appendix B.8, all-method iteration-150 checkpoint clarification.
We appreciate these suggestions. While the original submission already provides meaningful contributions through its controlled safety analysis, trace-aware evaluation, low-data restoration approach, and statistically supported anchored-ES results, the additional comparisons and independent validation resulting from these suggestions made the revision significantly stronger.
Response to Reviewer ZUV3
We thank the reviewer for the constructive feedback and for recognizing the value of the controlled Qwen3 experiments and the statistically supported safety restoration results. We also appreciate the concerns and have answered them below.
We thank the reviewer for raising these important concerns regarding the motivation, novelty, and practical advantages of ESPG. We would like to clarify two points.
1 and Q1-Ans. Motivation and contribution of gradient-free safety restoration. The primary contribution of the paper is not to claim that ESPG is the only approach to safety restoration, but to demonstrate that safety degradation in reasoning-tuned models can be systematically evaluated and substantially restored through a simple, low-data, gradient-free intervention.
While prior work has identified reasoning-related safety failures, the study extends these observations through systematic evaluations across training stages, reasoning modes, and model families (Section 4). ESPG then addresses the complementary challenge of restoring safety after reasoning-oriented adaptation.
Revision: §1, motivation/contributions; §2, training objectives and constraints; §3.2, ESPG update and reward; §5, restoration setting.Indeed, the gradient-based methods, including SFT, LoRA, and GRPO, are viable alternatives. However, gradient-free optimization offers four promising advantages:
Reduced training-memory requirements: ES avoids backpropagation and gradient optimizer states, allowing weight optimization through forward passes according to Malladi et al. [1] and Qiu et al. [2].
Direct optimization of safety rewards: ES operates on scalar rewards without requiring differentiable objectives or token-level policy gradients [2, 3]. The advantage provides a straightforward way to optimize the reward function of ESPG, which evaluates both reasoning traces and final answers while encouraging helpfulness on benign requests.
Low unique-data requirements: Prior work demonstrates the potential of ES for data-efficient LLM fine-tuning [2]. The advantage is particularly relevant to safety restoration, where carefully curated and annotated training examples may be limited.
Preservation of existing capabilities: Anchored weight decay constrains deviation from the initial model weights, reducing performance drift during adaptation, as demonstrated by Schweighofer et al. [3]. The advantage is especially relevant when restoring safety without sacrificing previously acquired reasoning capabilities.
Revision: §2, “Training objectives and constraints to preserve safety”; §3.2, reward and update equations; §6, compute-trade-off paragraph.[1] Malladi, S., Gao, T., Nichani, E., Damian, A., Lee, J. D., Chen, D., & Arora, S. (2023). Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36, 53038-53075.
[2] Qiu, X., Gan, Y., Hayes, C.F., Liang, Q., Xu, Y., Dailey, R., Meyerson, E., Hodjat, B. & Miikkulainen, R. (2026). Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning. Proceedings of the 43rd International Conference on Machine Learning.
[3] Salimans, T., Ho, J., Chen, X., Sidor, S., & Sutskever, I. (2017). Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.03864.
[4] Schweighofer, K., Hayes, C. F., Dailey, R., Miikkulainen, R., & Qiu, X. (2026). Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies. ArXiv. https://arxiv.org/abs/2605.30148
Our experimental results support these motivations. ESPG achieves a statistically significant reduction in the Full unsafe rate from 15.4% to 9.9% using only 30 unique training prompts, while largely preserving MATH-200 accuracy (93.0% → 92.5%). Furthermore, the matched ES–ESPG ablation demonstrates an additional 3.81 percentage-point reduction attributable to anchoring (95% CI [−6.13, −1.63]; Table 1).
The revised version makes these motivations explicit and clarifies that the paper’s novelty lies in combining gradient-free optimization, anchored weight decay, a trace-aware safety reward, and a training-time guardrail for safety restoration in reasoning-tuned models, rather than introducing ES or AWD as new optimization techniques.
Revision: §1, ESPG contribution paragraphs; §2, gradient-free training discussion; §6, interpretation of AWD contribution.