- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses safety degradation ("misevolution") in self-evolving LLM training, where a proposer generates tasks and a solver is trained on them with RL (AZR, R-Zero). It argues that the degradation has a geometric cause. Although computed on benign tasks, the solver's gradient has a component in a "safety subspace", and the direction of this component changes from round to round, leaving no stable target for data- or objective-level fixes. The proposed method, SEGO, builds the subspace from MLP-layer representations of 1,024 PKU-SafeRLHF samples (SVD of the non-central covariance HH^T, keeping directions with singular value above τ), removes the subspace component from the solver's gradient at every step, and recomputes the subspace when the accumulated displacement exceeds a threshold γ. The proposer is left unconstrained. Experiments cover AZR (Qwen2.5-7B and Qwen2.5-Coder-3B/7B/14B) and R-Zero (Qwen3-4B/8B-Base), with six ASR benchmarks, three over-refusal benchmarks and eleven code/math benchmarks. SEGO has lower ASR than unconstrained evolution on all 36 model–benchmark pairs, keeps over-refusal on OR-Bench and XSTest-Safe at or below the pre-evolution level, and roughly matches capability.
Critical strengths
-
An important, under-studied problem, addressed with a simple and cheap intervention. Safety erosion inside self-evolving RL loops matters for anyone using these pipelines, and curating the self-generated curriculum is not an obvious option. SEGO adds a periodic SVD and a per-step projection, keeps the host frameworks' hyperparameters and leaves the curriculum untouched, which makes it easy to adopt.
-
A consistent safety benefit across a broad evaluation. SEGO lowers ASR relative to unconstrained evolution on 36/36 pairs over six separately trained configurations and two frameworks. Averaged over the six safety benchmarks, the reduction is 3.6–6.8 points under AZR and 1.6–2.5 under R-Zero, bringing every configuration back to within about one point of the pre-evolution model. Over-refusal on OR-Bench and XSTest-Safe does not increase. If the results hold up, this is a useful practical finding.
Critical weaknesses
1. The mechanistic claims (geometric origin, decoupling from the proposer, a safety-specific subspace) are not supported. None of this affects the practical safety gain above, but these claims make up most of the stated contributions.
- Decoupling. The argument of Sec. 2.3.2 assumes separate proposer and solver parameters (Sec. 2.1, Fig. 1). In AZR, which accounts for 4 of the 6 main configurations, one model with shared parameters plays both roles and receives a single joint update, and the paper does not say what SEGO projects there. If only the solve-role term is projected, the propose-role term reaches the shared weights unprojected and U_S^T g̃ = 0 does not hold for the actual update. If the whole gradient is projected (App. C.1: "projects the gradient before it is passed to the optimizer"), the proposer is constrained, contrary to the abstract and Sec. 2.3.2. In R-Zero, where the models are separate, Sec. 4.5 presents Table 3 as confirming the decoupling. Yet additionally projecting the proposer raises solver ASR on all six benchmarks (six-benchmark mean 0.283 → 0.304, vs 0.308 without SEGO; PKU and JBB end up worse than unconstrained R-Zero). If this difference is noise, then SEGO's own R-Zero gains, which are of similar size relative to the reported deviations, are not established by single comparisons either. This does not affect the larger AZR gains.
- Safety specificity. No experiment separates the safety content of the corpus from generic high-energy directions of the activations. The only control not built from safety data is a random subspace of the same dimension (Table 7), which in expectation removes only about k/d of the gradient energy, and neither k nor that control's removed energy is reported. The generate variant shows that the corpus matters, but it is also built from harmful prompts. The informative control is the same procedure applied to a benign corpus, with matched k or matched removed energy. Without it, the statements that the subspace is "genuinely safety-relevant" (App. D.1) and that the paper gives the first evidence of separability (abstract, Sec. 4.2) are not supported.
- Motivating pilot. Since the in-subspace and orthogonal parts of the gradient are orthogonal, intrusion² + retention² = 1, and averaging over layers or steps can only lower the implied retention. The intrusion ratio of 0.90–0.95 in Fig. 2(a) therefore implies a retention of at most about 0.31–0.44, whereas Fig. 4 reports 0.86–0.99. Neither figure states the model, τ or the set of parameters over which the norms are taken (e.g., whether the unprojected attention parameters are included). Either the two figures measure different things, in which case the pilot does not describe the subspace SEGO protects, or one of them is wrong. Fixed-data RL also shows an intrusion of 0.90–0.94 in Fig. 2(a), so the contrast with self-evolution is about 0.01.
2. Parts of the empirical evidence need to be checked or strengthened. In Table 8, I could not reconcile several non-default entries with the rest of the paper. For example, the ASR reported for τ = 5e-4 (0.3781) coincides to four decimals with the mean ASR of AZR-Step300+DPO in Table 10, and the code average for τ = 1e-5 (0.4578) with the mean code score of AZR-KL in Table 11. Neither baseline involves τ (App. C.2). This may be a table-assembly issue, but the affected entries include the capability values for τ ≤ 1e-5, which are the evidence for the capability drop that motivates the default τ (App. D.2). Separately, App. G reports non-zero three-seed standard deviations for the untrained pre-evolution models without saying what varies across seeds. Inference nondeterminism or the GPT-4o judge could explain some spread, but for the two 100-prompt benchmarks scored 0/1 (JBB, JBB-Benign), almost none of the reported deviations can arise from three such runs, so it is unclear how Tables 13–14 were computed. This matters because 14 of the 36 SEGO–baseline safety gaps are below three points. Finally, SEGO is ahead of KL, SPF and post-hoc DPO in the comparison of Sec. 3.3, which supports the claim that it avoids their failure modes. However, that comparison covers only one of the six configurations (AZR, Qwen2.5-7B), uses a single setting per baseline with no tuning reported (App. F), and does not include the natural in-loop alternative of mixing the same 1,024 safety samples into training.
Weakness 1 is what drives my rating. My soundness score refers to the mechanistic claims and the supporting evidence, not to the practical safety gain, which looks robust under AZR across four separately trained models. This is why I rate the paper a weak rather than a clear rejection. Rescoping the claims and clarifying the AZR implementation can be done in a revision. The benign-corpus control, a corrected Table 8, per-seed scores and the broader comparison need new runs or data. If these are provided and the claims are rescoped, I would raise my score.
Questions for authors
- How was each entry of Table 8 produced (which runs and seeds)? I tried to reproduce it by averaging the per-benchmark scores in each row of Tables 10–12. That gives the No-SEGO and τ = 5e-5 rows exactly (AZR-Step300 and SEGO), but five other entries then match the means of different rows to four decimals:
- τ = 5e-4: ASR 0.3781 = AZR-Step300+DPO (Table 10), i.e. (0.3070 + 0.3419 + 0.5150 + 0.4500 + 0.2580 + 0.3967) / 6
- τ = 1e-5: code 0.4578 = AZR-KL (Table 11); math 0.4331 = AZR-Step500+DPO (Table 12)
- τ = 5e-6: code 0.4490 = AZR-Step500+DPO (Table 11); math 0.4228 = AZR-Step400+DPO (Table 12)
This could easily be a mix-up when the table was put together. If so, a corrected table with per-benchmark results is all I need.
-
What do the three seeds vary for an untrained model, and how were the standard deviations in Tables 13–14 computed? I also could not tell whether Tables 1–2 report a single seed or a mean. Per-seed correct/total counts would settle it (e.g., SEGO-Coder-14B's 0.3630 on HEx-PHI is not a multiple of 1/300).
-
In AZR the proposer and solver share parameters, so which gradient is projected? The decoupling claim should be narrowed accordingly. Seeds for the proposer+solver variant in Table 3 would also help.
-
Do Fig. 2(a) and Fig. 4 use the same model, τ and set of parameters? Which model and task source did the pilot use?
-
Some implementation details are missing: which activations and MLP matrices are used (and on which side the projection is applied), how C is normalized before thresholding, and the resulting k per layer and value of γ.
-
Some capability scores are far below the original AZR paper (e.g., MBPP+ 34.4 vs 65.3 for Qwen2.5-7B). Which harness and prompts did you use, and were the Coder models base or Instruct (Sec. 3.1 says Instruct; AZR used base)? Also, Fig. 5 suggests SEGO was trained to 500 AZR steps; what are the results there?
Experiments that could change my assessment
- A control subspace built the same way from a benign corpus, matched on k or on removed gradient energy.
- In-loop mixing or replay of the same 1,024 PKU-SafeRLHF samples, which is the obvious alternative since SEGO already uses this corpus.
- The comparison with KL, SPF and DPO (plus the replay baseline) on at least one R-Zero configuration, with more than one setting per baseline. The appendix is fine for this.
Other comments (secondary to my assessment)
- Positioning. The operator is essentially that of GPM (ICLR 2021), Adam-NSCL (CVPR 2021) and AlphaEdit (ICLR 2025), and close to NSPO. Safety-specific projection methods (Safe LoRA, SaLoRA, GuardSpace) are not discussed. SPF is described as a single-direction method in Sec. 3.3 and App. C.2, although App. A describes it as low-rank, and it does worse than no intervention.
- "Capability is preserved throughout" should be qualified: on Qwen2.5-Coder-7B, SEGO is 6.7 (HumanEval), 6.1 (HumanEval+), 7.4 (OlympiadBench) and 3.4 (MATH-500) points below AZR.
- The starting models are weakly aligned (pre-evolution six-benchmark mean ASR 0.29–0.56, except 0.14 for Qwen3-8B). A strongly safety-aligned instruct model and jailbreak attacks would make the safety results more convincing.
- Fig. 2(c) is labeled "Safety Score" but rises over training, so it presumably shows ASR. Some summary statements do not match the tables: evolution raises ASR on 33 pairs, not 34; the largest increase is 14.0 points, not 9.0; SEGO is at or below the pre-evolution model on 18 pairs, not 15; Table 4 has nine benchmarks, not eleven. And the DPO "without exception" claim fails on HumanEval+ at Step 300.
7 comments
Response for Part I: Additional experiments and corrected results (2/2)
Part I: Additional experiments and corrected results (2/2)
W2 / Q1 / Q2: Corrected Table 8 and multi-seed results
We thank the reviewer for checking these tables so carefully.
Table 8 (Q1). The reviewer is right that several non-default entries in Table 8 were mixed up when the table was assembled. We regenerated the table from the raw evaluation logs; the corrected table is below, followed by the per-benchmark results.
| τ | ASR (avg) ↓ | Code (avg) ↑ | Math (avg) ↑ |
|---|---|---|---|
| No SEGO | 0.3985 | 0.4778 | 0.4474 |
| 5e-4 | 0.3697 | 0.5122 | 0.4634 |
| 1e-4 | 0.3674 | 0.5023 | 0.4575 |
| 5e-5 | 0.3410 | 0.5074 | 0.4536 |
| 1e-5 | 0.3406 | 0.4814 | 0.4359 |
| 5e-6 | 0.3386 | 0.4751 | 0.4212 |
| τ | PKU | AdvB | HB | JBB | SB | HEx | Avg ↓ |
|---|---|---|---|---|---|---|---|
| No SEGO | 0.3180 | 0.3538 | 0.5350 | 0.4700 | 0.2910 | 0.4233 | 0.3985 |
| 5e-4 | 0.2910 | 0.2712 | 0.5200 | 0.4500 | 0.2890 | 0.3967 | 0.3697 |
| 1e-4 | 0.2980 | 0.2596 | 0.5125 | 0.4600 | 0.2710 | 0.4033 | 0.3674 |
| 5e-5 | 0.2920 | 0.2288 | 0.4725 | 0.4300 | 0.2525 | 0.3700 | 0.3410 |
| 1e-5 | 0.2880 | 0.2558 | 0.4650 | 0.4200 | 0.2545 | 0.3600 | 0.3406 |
| 5e-6 | 0.2890 | 0.2462 | 0.4675 | 0.4200 | 0.2555 | 0.3533 | 0.3386 |
| τ | LCB | CRUX-I | CRUX-O | MBPP | HE | HE+ | MBPP+ | Avg ↑ |
|---|---|---|---|---|---|---|---|---|
| No SEGO | 0.1228 | 0.4275 | 0.3863 | 0.4960 | 0.7744 | 0.7178 | 0.4196 | 0.4778 |
| 5e-4 | 0.1257 | 0.4325 | 0.4188 | 0.5560 | 0.7744 | 0.7423 | 0.5357 | 0.5122 |
| 1e-4 | 0.1287 | 0.4275 | 0.3950 | 0.5500 | 0.7561 | 0.7362 | 0.5223 | 0.5023 |
| 5e-5 | 0.1228 | 0.4225 | 0.4188 | 0.5440 | 0.7866 | 0.7485 | 0.5089 | 0.5074 |
| 1e-5 | 0.1111 | 0.4338 | 0.4163 | 0.4660 | 0.7866 | 0.7362 | 0.4196 | 0.4814 |
| 5e-6 | 0.1199 | 0.4313 | 0.4063 | 0.4980 | 0.7439 | 0.6933 | 0.4330 | 0.4751 |
| τ | MATH-500 | GSM8K | Olymp. | Minerva | Avg ↑ |
|---|---|---|---|---|---|
| No SEGO | 0.6880 | 0.5618 | 0.3561 | 0.1838 | 0.4474 |
| 5e-4 | 0.7140 | 0.5368 | 0.4006 | 0.2022 | 0.4634 |
| 1e-4 | 0.6920 | 0.5929 | 0.3798 | 0.1654 | 0.4575 |
| 5e-5 | 0.6740 | 0.5967 | 0.3635 | 0.1801 | 0.4536 |
| 1e-5 | 0.6320 | 0.6187 | 0.3457 | 0.1471 | 0.4359 |
| 5e-6 | 0.6160 | 0.5656 | 0.3635 | 0.1397 | 0.4212 |
The trend described in App. D.2 is unchanged. ASR decreases from τ = 5e-4 to 5e-5 (0.3697 to 0.3410) and then saturates (0.3406 at 1e-5 and 0.3386 at 5e-6), while capability is stable down to 5e-5 and drops at 1e-5 (code 0.4814 and math 0.4359, compared with 0.5074 and 0.4536 at 5e-5), so the default τ is unchanged.
Multi-seed results (Q2). Tables 1–2 report a single run, which is one of the three seeds. In checking the reviewer's observation, we found that the previous multi-seed results were not computed consistently: some runs were scored over the valid samples after excluding failed judgments or execution errors, rather than over the full benchmark. We therefore re-evaluated all reported results on a single server, with the same counting over the full benchmark and the same rounding, and updated the affected entries. Every entry now corresponds to an integer count over the full benchmark; for example, SEGO-Coder-14B on HEx-PHI is now 0.3633 (109/300). The multi-seed tables report the sample standard deviation across the three runs. For each pre-evolution model, the three runs are repeated evaluations, whose small deviations reflect nondeterminism in batched inference and in the model-based judges. With the revised results, the gap between SEGO and unconstrained evolution in the three-seed means exceeds the larger of the two standard deviations in 33 of the 36 model–benchmark pairs.
We have updated the revised version accordingly: it includes the corrected Table 8 (App. D.2) with per-benchmark results (App. D.5, Tables 11–13 in the revised version), the revised Tables 1–2, and the revised multi-seed results in App. G, reported as mean ± standard deviation. We thank the reviewer again for these careful checks, which strengthen the reliability of our reported results.
Response for Part II: Clarifications and other comments (1/4)
Part II: Clarifications and other comments (1/4)
W1 (decoupling) / Q3: Decoupling in AZR
We thank the reviewer for raising this point, which gives us the opportunity to clarify the implementation details. In AZR, the proposer and solver share parameters, and the joint gradient decomposes into a propose-role term and a solve-role term. SEGO projects only the solve-role term and leaves the propose-role term unprojected; the two are then summed and passed to the optimizer: . This keeps task generation unconstrained, as intended.
We agree that, with shared parameters, the decoupling argument of Sec. 2.3.2 applies to the solve-role component rather than to the full update, and we will narrow the claim accordingly. Within this narrower scope, the decoupling remains meaningful in the shared-parameter setting. Sharing parameters does not imply sharing update directions: the gradient a layer receives lies in the span of the activations of the inputs that produce it, so the propose-role and solve-role terms, computed on different prompts (generating tasks vs. solving them) and toward different objectives, occupy directions determined by their own inputs. Projecting the solve-role term therefore constrains the update driven by task solving, while the update driven by task generation follows its own directions. Consistent with this, AZR, where the propose-role term is left unprojected, is also a setting in which SEGO yields substantial safety gains (Table 1).
In the revision, we will state the AZR implementation explicitly (Sec. 2.1, Sec. 2.3.2, App. C.1) and describe the decoupling at this scope: the projection applies to the solving update; in frameworks with separate proposer and solver (R-Zero) this is the solver's entire update, while in shared-parameter frameworks (AZR) the propose-role term is left unprojected by design. We thank the reviewer again for helping us make this point precise.
W1 (decoupling) / Q3: Proposer ablation on R-Zero with three seeds
We thank the reviewer for this suggestion. We have rerun R-Zero, SEGO (solver-only), and SEGO (proposer + solver) on Qwen3-4B-Base with three seeds. The table below reports ASR on each safety benchmark (mean ± std across three seeds).
| Setting | PKU ↓ | AdvB ↓ | HB ↓ | JBB ↓ | SB ↓ | HEx ↓ |
|---|---|---|---|---|---|---|
| R-Zero | 0.3763 ± 0.0242 | 0.1269 ± 0.0069 | 0.3433 ± 0.0113 | 0.2633 ± 0.0351 | 0.3843 ± 0.0025 | 0.3622 ± 0.0168 |
| SEGO (solver-only) | 0.3710 ± 0.0233 | 0.0923 ± 0.0051 | 0.3033 ± 0.0118 | 0.2333 ± 0.0252 | 0.3747 ± 0.0055 | 0.3244 ± 0.0135 |
| SEGO (proposer + solver) | 0.3743 ± 0.0309 | 0.0968 ± 0.0087 | 0.3025 ± 0.0132 | 0.2200 ± 0.0436 | 0.3767 ± 0.0050 | 0.3300 ± 0.0167 |
Across seeds, projecting the proposer in addition to the solver yields essentially the same safety as solver-only projection, with the differences on all six benchmarks within the seed-to-seed variation. This is consistent with Sec. 2.3.2: once the solver's update is projected, additionally constraining the proposer provides no further safety benefit. SEGO's improvement over R-Zero holds across seeds on most benchmarks. We will update Sec. 4.5 and Table 3 with the multi-seed results and revise the discussion accordingly. We thank the reviewer again for this suggestion, which makes the R-Zero results and the proposer ablation more reliable.
Q5: Implementation details
We thank the reviewer for pointing out these missing details, which we will add to App. C.
- Activations and matrices. The projection is applied to all three MLP matrices. For each matrix, the subspace is computed from that matrix's own input: the MLP input ( for Qwen2.5-7B) for the gate and up projections, and the intermediate activation after SwiGLU () for the down projection.
- Side of the projection. The projection is applied on the input side of each matrix, i.e., along the dimension of the representation from which the subspace is computed.
- Normalization of . The threshold is relative: we retain the singular vectors of whose singular values satisfy , which is equivalent to normalizing by its largest singular value.
- Resulting . With , on Qwen2.5-7B, ranges from 15 to 823 across layers for the gate and up projections (mean 498), and from 18 to 2384 for the down projection (mean 1233). The full per-layer values will be reported in App. C.
- . We use , which triggers recomputation every 50 steps on average, with intervals ranging from 42 to 63 steps over training.
We thank the reviewer again for these questions, which help make the implementation of SEGO more precise and reproducible.
Response for Part I: Additional experiments and corrected results (1/2)
We thank the reviewer for the thorough and careful review. We appreciate that the reviewer sees the safety of self-evolving training as an important and under-studied problem, finds SEGO simple to adopt, and observes a consistent safety benefit across the configurations we evaluated. We first address the points that the reviewer identified as requiring new runs or data, and then respond to the remaining questions and comments.
Part I: Additional experiments and corrected results (1/2)
W1 (safety specificity): Benign-corpus control
We thank the reviewer for this comment and agree that the informative control is the same procedure applied to a benign corpus under matched removed energy. We have run this control.
We built the control subspace by exactly the same procedure as SEGO's (Sec. 2.3.1), from a benign corpus of 1,024 general instructions (Dolly-15k) in the same format as Dsafe, containing no harmful requests or refusals. We set its threshold so that it removes a share of gradient energy comparable to that of SEGO's subspace. Results on Qwen2.5-Coder-7B (AZR) are below.
The benign subspace provides limited protection, whereas the safety subspace reduces ASR to 0.3020. Since the two subspaces remove a comparable share of gradient energy, this difference points to the content of the corpus, and together with the generate variant (App. D.1), suggests that the subspace SEGO protects carries safety-relevant content.
In the revision, we will add the benign-corpus control, together with its removed energy, to App. D.4 (Table 10 in the revised version), and revise the corresponding statements in App. D.1, the abstract and Sec. 4.2 accordingly: the protected subspace is described as safety-relevant on the basis of this control, and the separability is described in a practical sense, namely that projecting out this subspace preserves safety while most of the evolutionary signal, and most of the capability gains, are kept. We thank the reviewer again for this suggestion, which helps us state more precisely what SEGO's subspace captures.
W2: Broader baseline comparison on R-Zero, including replay
We thank the reviewer for this suggestion. We extended the comparison of Sec. 3.3 to R-Zero (Qwen3-4B-Base), with two settings for each baseline, and added the in-loop replay baseline, in which the same 1,024 PKU-SafeRLHF samples used to build SEGO's subspace are mixed into training at each iteration. R-Zero's default configuration already includes a KL term with coefficient 0.01, so that row corresponds to the R-Zero baseline in Tables 1–2; we additionally report a stronger coefficient of 0.1. DPO is applied after the final R-Zero iteration, using the same preference pairs as in Sec. 3.3. The table reports the average over the six safety benchmarks and over the coding and mathematics benchmarks.
SEGO attains the lowest average ASR, while its capability remains close to that of R-Zero. Strengthening the KL term changes ASR only slightly. SPF and DPO lower ASR relative to R-Zero, and for both methods, the setting that brings ASR closest to SEGO is also the one with the larger capability drop. Replay lowers ASR on PKU-SafeRLHF, the corpus it replays, but does not lower the average ASR across the six benchmarks, and it reduces capability. One possible reason is that replay mainly fits the replayed distribution, while its additional training signal competes with the evolutionary signal.
We will add this comparison, with per-benchmark results and implementation details, to App. F. We thank the reviewer again for this suggestion, which helps place SEGO's results in a broader context.