acceptodds
Under review as a conference paper at ICLR 2027

Toward Trustworthy Self-Evolution: Preserving Safety via Geometric Orthogonalization

Abstract

Self-evolving agents generate their own tasks and derive training signal from their own interaction, enabling capability to grow beyond what human-curated data can specify. This autonomy incurs a cost: safety degrades across successive rounds even as task accuracy improves, a failure mode termed misevolution. We show that this degradation is geometric in origin. Although computed entirely from benign tasks, the evolutionary gradient intrudes into a safety subspace of the solver's parameter space; moreover, as the curriculum co-evolves with the model, the direction of this intrusion shifts unpredictably across rounds, affording no stable target for data-level or objective-level correction. We therefore reframe safety preservation as a constraint on optimization geometry and propose SEGO, which orthogonalizes the evolutionary gradient against a safety subspace identified from model representations and updated as the model drifts, permitting unconstrained capability growth while the update carries no component within that subspace. Since the proposer influences the solver solely through its gradient, constraining the solver alone is sufficient, leaving the co-evolutionary dynamic intact. Across multiple self-evolving frameworks and models, SEGO preserves the model's safety throughout evolution while matching the capability growth of unconstrained evolution. Our analysis further establishes the first empirical evidence that safety-critical and capability-driving directions remain separable under self-evolution.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, self-evolving agents, AI safety

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PtMrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses safety degradation ("misevolution") in self-evolving LLM training, where a proposer generates tasks and a solver is trained on them with RL (AZR, R-Zero). It argues that the degradation has a geometric cause. Although computed on benign tasks, the solver's gradient has a component in a "safety subspace", and the direction of this component changes from round to round, leaving no stable target for data- or objective-level fixes. The proposed method, SEGO, builds the subspace from MLP-layer representations of 1,024 PKU-SafeRLHF samples (SVD of the non-central covariance HH^T, keeping directions with singular value above τ), removes the subspace component from the solver's gradient at every step, and recomputes the subspace when the accumulated displacement exceeds a threshold γ. The proposer is left unconstrained. Experiments cover AZR (Qwen2.5-7B and Qwen2.5-Coder-3B/7B/14B) and R-Zero (Qwen3-4B/8B-Base), with six ASR benchmarks, three over-refusal benchmarks and eleven code/math benchmarks. SEGO has lower ASR than unconstrained evolution on all 36 model–benchmark pairs, keeps over-refusal on OR-Bench and XSTest-Safe at or below the pre-evolution level, and roughly matches capability.

Critical strengths
  1. An important, under-studied problem, addressed with a simple and cheap intervention. Safety erosion inside self-evolving RL loops matters for anyone using these pipelines, and curating the self-generated curriculum is not an obvious option. SEGO adds a periodic SVD and a per-step projection, keeps the host frameworks' hyperparameters and leaves the curriculum untouched, which makes it easy to adopt.

  2. A consistent safety benefit across a broad evaluation. SEGO lowers ASR relative to unconstrained evolution on 36/36 pairs over six separately trained configurations and two frameworks. Averaged over the six safety benchmarks, the reduction is 3.6–6.8 points under AZR and 1.6–2.5 under R-Zero, bringing every configuration back to within about one point of the pre-evolution model. Over-refusal on OR-Bench and XSTest-Safe does not increase. If the results hold up, this is a useful practical finding.

Critical weaknesses

1. The mechanistic claims (geometric origin, decoupling from the proposer, a safety-specific subspace) are not supported. None of this affects the practical safety gain above, but these claims make up most of the stated contributions.

  • Decoupling. The argument of Sec. 2.3.2 assumes separate proposer and solver parameters (Sec. 2.1, Fig. 1). In AZR, which accounts for 4 of the 6 main configurations, one model with shared parameters plays both roles and receives a single joint update, and the paper does not say what SEGO projects there. If only the solve-role term is projected, the propose-role term reaches the shared weights unprojected and U_S^T g̃ = 0 does not hold for the actual update. If the whole gradient is projected (App. C.1: "projects the gradient before it is passed to the optimizer"), the proposer is constrained, contrary to the abstract and Sec. 2.3.2. In R-Zero, where the models are separate, Sec. 4.5 presents Table 3 as confirming the decoupling. Yet additionally projecting the proposer raises solver ASR on all six benchmarks (six-benchmark mean 0.283 → 0.304, vs 0.308 without SEGO; PKU and JBB end up worse than unconstrained R-Zero). If this difference is noise, then SEGO's own R-Zero gains, which are of similar size relative to the reported deviations, are not established by single comparisons either. This does not affect the larger AZR gains.
  • Safety specificity. No experiment separates the safety content of the corpus from generic high-energy directions of the activations. The only control not built from safety data is a random subspace of the same dimension (Table 7), which in expectation removes only about k/d of the gradient energy, and neither k nor that control's removed energy is reported. The generate variant shows that the corpus matters, but it is also built from harmful prompts. The informative control is the same procedure applied to a benign corpus, with matched k or matched removed energy. Without it, the statements that the subspace is "genuinely safety-relevant" (App. D.1) and that the paper gives the first evidence of separability (abstract, Sec. 4.2) are not supported.
  • Motivating pilot. Since the in-subspace and orthogonal parts of the gradient are orthogonal, intrusion² + retention² = 1, and averaging over layers or steps can only lower the implied retention. The intrusion ratio of 0.90–0.95 in Fig. 2(a) therefore implies a retention of at most about 0.31–0.44, whereas Fig. 4 reports 0.86–0.99. Neither figure states the model, τ or the set of parameters over which the norms are taken (e.g., whether the unprojected attention parameters are included). Either the two figures measure different things, in which case the pilot does not describe the subspace SEGO protects, or one of them is wrong. Fixed-data RL also shows an intrusion of 0.90–0.94 in Fig. 2(a), so the contrast with self-evolution is about 0.01.

2. Parts of the empirical evidence need to be checked or strengthened. In Table 8, I could not reconcile several non-default entries with the rest of the paper. For example, the ASR reported for τ = 5e-4 (0.3781) coincides to four decimals with the mean ASR of AZR-Step300+DPO in Table 10, and the code average for τ = 1e-5 (0.4578) with the mean code score of AZR-KL in Table 11. Neither baseline involves τ (App. C.2). This may be a table-assembly issue, but the affected entries include the capability values for τ ≤ 1e-5, which are the evidence for the capability drop that motivates the default τ (App. D.2). Separately, App. G reports non-zero three-seed standard deviations for the untrained pre-evolution models without saying what varies across seeds. Inference nondeterminism or the GPT-4o judge could explain some spread, but for the two 100-prompt benchmarks scored 0/1 (JBB, JBB-Benign), almost none of the reported deviations can arise from three such runs, so it is unclear how Tables 13–14 were computed. This matters because 14 of the 36 SEGO–baseline safety gaps are below three points. Finally, SEGO is ahead of KL, SPF and post-hoc DPO in the comparison of Sec. 3.3, which supports the claim that it avoids their failure modes. However, that comparison covers only one of the six configurations (AZR, Qwen2.5-7B), uses a single setting per baseline with no tuning reported (App. F), and does not include the natural in-loop alternative of mixing the same 1,024 safety samples into training.

Weakness 1 is what drives my rating. My soundness score refers to the mechanistic claims and the supporting evidence, not to the practical safety gain, which looks robust under AZR across four separately trained models. This is why I rate the paper a weak rather than a clear rejection. Rescoping the claims and clarifying the AZR implementation can be done in a revision. The benign-corpus control, a corrected Table 8, per-seed scores and the broader comparison need new runs or data. If these are provided and the claims are rescoped, I would raise my score.

Questions for authors
  1. How was each entry of Table 8 produced (which runs and seeds)? I tried to reproduce it by averaging the per-benchmark scores in each row of Tables 10–12. That gives the No-SEGO and τ = 5e-5 rows exactly (AZR-Step300 and SEGO), but five other entries then match the means of different rows to four decimals:
    • τ = 5e-4: ASR 0.3781 = AZR-Step300+DPO (Table 10), i.e. (0.3070 + 0.3419 + 0.5150 + 0.4500 + 0.2580 + 0.3967) / 6
    • τ = 1e-5: code 0.4578 = AZR-KL (Table 11); math 0.4331 = AZR-Step500+DPO (Table 12)
    • τ = 5e-6: code 0.4490 = AZR-Step500+DPO (Table 11); math 0.4228 = AZR-Step400+DPO (Table 12)

This could easily be a mix-up when the table was put together. If so, a corrected table with per-benchmark results is all I need.

  1. What do the three seeds vary for an untrained model, and how were the standard deviations in Tables 13–14 computed? I also could not tell whether Tables 1–2 report a single seed or a mean. Per-seed correct/total counts would settle it (e.g., SEGO-Coder-14B's 0.3630 on HEx-PHI is not a multiple of 1/300).

  2. In AZR the proposer and solver share parameters, so which gradient is projected? The decoupling claim should be narrowed accordingly. Seeds for the proposer+solver variant in Table 3 would also help.

  3. Do Fig. 2(a) and Fig. 4 use the same model, τ and set of parameters? Which model and task source did the pilot use?

  4. Some implementation details are missing: which activations and MLP matrices are used (and on which side the projection is applied), how C is normalized before thresholding, and the resulting k per layer and value of γ.

  5. Some capability scores are far below the original AZR paper (e.g., MBPP+ 34.4 vs 65.3 for Qwen2.5-7B). Which harness and prompts did you use, and were the Coder models base or Instruct (Sec. 3.1 says Instruct; AZR used base)? Also, Fig. 5 suggests SEGO was trained to 500 AZR steps; what are the results there?

Experiments that could change my assessment

  • A control subspace built the same way from a benign corpus, matched on k or on removed gradient energy.
  • In-loop mixing or replay of the same 1,024 PKU-SafeRLHF samples, which is the obvious alternative since SEGO already uses this corpus.
  • The comparison with KL, SPF and DPO (plus the replay baseline) on at least one R-Zero configuration, with more than one setting per baseline. The appendix is fine for this.

Other comments (secondary to my assessment)

  • Positioning. The operator is essentially that of GPM (ICLR 2021), Adam-NSCL (CVPR 2021) and AlphaEdit (ICLR 2025), and close to NSPO. Safety-specific projection methods (Safe LoRA, SaLoRA, GuardSpace) are not discussed. SPF is described as a single-direction method in Sec. 3.3 and App. C.2, although App. A describes it as low-rank, and it does worse than no intervention.
  • "Capability is preserved throughout" should be qualified: on Qwen2.5-Coder-7B, SEGO is 6.7 (HumanEval), 6.1 (HumanEval+), 7.4 (OlympiadBench) and 3.4 (MATH-500) points below AZR.
  • The starting models are weakly aligned (pre-evolution six-benchmark mean ASR 0.29–0.56, except 0.14 for Qwen3-8B). A strongly safety-aligned instruct model and jailbreak attacks would make the safety results more convincing.
  • Fig. 2(c) is labeled "Safety Score" but rises over training, so it presumably shows ASR. Some summary statements do not match the tables: evolution raises ASR on 33 pairs, not 34; the largest increase is 14.0 points, not 9.0; SEGO is at or below the pre-evolution model on 18 pairs, not 15; Table 4 has nine benchmarks, not eleven. And the DPO "without exception" claim fails on HumanEval+ at Step 300.
7 comments
AuthorsOpenReview ↗

Response for Part I: Additional experiments and corrected results (1/2)

We thank the reviewer for the thorough and careful review. We appreciate that the reviewer sees the safety of self-evolving training as an important and under-studied problem, finds SEGO simple to adopt, and observes a consistent safety benefit across the configurations we evaluated. We first address the points that the reviewer identified as requiring new runs or data, and then respond to the remaining questions and comments.

Part I: Additional experiments and corrected results (1/2)

W1 (safety specificity): Benign-corpus control

We thank the reviewer for this comment and agree that the informative control is the same procedure applied to a benign corpus under matched removed energy. We have run this control.

We built the control subspace by exactly the same procedure as SEGO's (Sec. 2.3.1), from a benign corpus of 1,024 general instructions (Dolly-15k) in the same format as DsafeD_{\text{safe}}, containing no harmful requests or refusals. We set its threshold so that it removes a share of gradient energy comparable to that of SEGO's subspace. Results on Qwen2.5-Coder-7B (AZR) are below.

Subspace sourceRemoved energyMean ASR ↓
Unconstrained AZR00.3705
Benign corpus (energy-matched)0.13580.3504
Safety corpus (SEGO)0.13960.3020

The benign subspace provides limited protection, whereas the safety subspace reduces ASR to 0.3020. Since the two subspaces remove a comparable share of gradient energy, this difference points to the content of the corpus, and together with the generate variant (App. D.1), suggests that the subspace SEGO protects carries safety-relevant content.

In the revision, we will add the benign-corpus control, together with its removed energy, to App. D.4 (Table 10 in the revised version), and revise the corresponding statements in App. D.1, the abstract and Sec. 4.2 accordingly: the protected subspace is described as safety-relevant on the basis of this control, and the separability is described in a practical sense, namely that projecting out this subspace preserves safety while most of the evolutionary signal, and most of the capability gains, are kept. We thank the reviewer again for this suggestion, which helps us state more precisely what SEGO's subspace captures.

W2: Broader baseline comparison on R-Zero, including replay

We thank the reviewer for this suggestion. We extended the comparison of Sec. 3.3 to R-Zero (Qwen3-4B-Base), with two settings for each baseline, and added the in-loop replay baseline, in which the same 1,024 PKU-SafeRLHF samples used to build SEGO's subspace are mixed into training at each iteration. R-Zero's default configuration already includes a KL term with coefficient 0.01, so that row corresponds to the R-Zero baseline in Tables 1–2; we additionally report a stronger coefficient of 0.1. DPO is applied after the final R-Zero iteration, using the same preference pairs as in Sec. 3.3. The table reports the average over the six safety benchmarks and over the coding and mathematics benchmarks.

MethodASR (avg) ↓Code (avg) ↑Math (avg) ↑
R-Zero + KL (0.01, default)0.30780.56650.6034
R-Zero + KL (0.1)0.30560.55990.6044
R-Zero + SPF (k = 20)0.29690.54830.6031
R-Zero + SPF (k = 40)0.29170.54310.5980
R-Zero + DPO (β = 0.01)0.29830.52830.5899
R-Zero + DPO (β = 0.001)0.29180.51710.5760
R-Zero + replay0.31800.48540.5509
SEGO0.28270.56110.6153

SEGO attains the lowest average ASR, while its capability remains close to that of R-Zero. Strengthening the KL term changes ASR only slightly. SPF and DPO lower ASR relative to R-Zero, and for both methods, the setting that brings ASR closest to SEGO is also the one with the larger capability drop. Replay lowers ASR on PKU-SafeRLHF, the corpus it replays, but does not lower the average ASR across the six benchmarks, and it reduces capability. One possible reason is that replay mainly fits the replayed distribution, while its additional training signal competes with the evolutionary signal.

We will add this comparison, with per-benchmark results and implementation details, to App. F. We thank the reviewer again for this suggestion, which helps place SEGO's results in a broader context.

AuthorsOpenReview ↗

Response for Part I: Additional experiments and corrected results (2/2)

Part I: Additional experiments and corrected results (2/2)

W2 / Q1 / Q2: Corrected Table 8 and multi-seed results

We thank the reviewer for checking these tables so carefully.

Table 8 (Q1). The reviewer is right that several non-default entries in Table 8 were mixed up when the table was assembled. We regenerated the table from the raw evaluation logs; the corrected table is below, followed by the per-benchmark results.

τASR (avg) ↓Code (avg) ↑Math (avg) ↑
No SEGO0.39850.47780.4474
5e-40.36970.51220.4634
1e-40.36740.50230.4575
5e-50.34100.50740.4536
1e-50.34060.48140.4359
5e-60.33860.47510.4212
τPKUAdvBHBJBBSBHExAvg ↓
No SEGO0.31800.35380.53500.47000.29100.42330.3985
5e-40.29100.27120.52000.45000.28900.39670.3697
1e-40.29800.25960.51250.46000.27100.40330.3674
5e-50.29200.22880.47250.43000.25250.37000.3410
1e-50.28800.25580.46500.42000.25450.36000.3406
5e-60.28900.24620.46750.42000.25550.35330.3386
τLCBCRUX-ICRUX-OMBPPHEHE+MBPP+Avg ↑
No SEGO0.12280.42750.38630.49600.77440.71780.41960.4778
5e-40.12570.43250.41880.55600.77440.74230.53570.5122
1e-40.12870.42750.39500.55000.75610.73620.52230.5023
5e-50.12280.42250.41880.54400.78660.74850.50890.5074
1e-50.11110.43380.41630.46600.78660.73620.41960.4814
5e-60.11990.43130.40630.49800.74390.69330.43300.4751
τMATH-500GSM8KOlymp.MinervaAvg ↑
No SEGO0.68800.56180.35610.18380.4474
5e-40.71400.53680.40060.20220.4634
1e-40.69200.59290.37980.16540.4575
5e-50.67400.59670.36350.18010.4536
1e-50.63200.61870.34570.14710.4359
5e-60.61600.56560.36350.13970.4212

The trend described in App. D.2 is unchanged. ASR decreases from τ = 5e-4 to 5e-5 (0.3697 to 0.3410) and then saturates (0.3406 at 1e-5 and 0.3386 at 5e-6), while capability is stable down to 5e-5 and drops at 1e-5 (code 0.4814 and math 0.4359, compared with 0.5074 and 0.4536 at 5e-5), so the default τ is unchanged.

Multi-seed results (Q2). Tables 1–2 report a single run, which is one of the three seeds. In checking the reviewer's observation, we found that the previous multi-seed results were not computed consistently: some runs were scored over the valid samples after excluding failed judgments or execution errors, rather than over the full benchmark. We therefore re-evaluated all reported results on a single server, with the same counting over the full benchmark and the same rounding, and updated the affected entries. Every entry now corresponds to an integer count over the full benchmark; for example, SEGO-Coder-14B on HEx-PHI is now 0.3633 (109/300). The multi-seed tables report the sample standard deviation across the three runs. For each pre-evolution model, the three runs are repeated evaluations, whose small deviations reflect nondeterminism in batched inference and in the model-based judges. With the revised results, the gap between SEGO and unconstrained evolution in the three-seed means exceeds the larger of the two standard deviations in 33 of the 36 model–benchmark pairs.

We have updated the revised version accordingly: it includes the corrected Table 8 (App. D.2) with per-benchmark results (App. D.5, Tables 11–13 in the revised version), the revised Tables 1–2, and the revised multi-seed results in App. G, reported as mean ± standard deviation. We thank the reviewer again for these careful checks, which strengthen the reliability of our reported results.

AuthorsOpenReview ↗

Response for Part II: Clarifications and other comments (1/4)

Part II: Clarifications and other comments (1/4)

W1 (decoupling) / Q3: Decoupling in AZR

We thank the reviewer for raising this point, which gives us the opportunity to clarify the implementation details. In AZR, the proposer and solver share parameters, and the joint gradient decomposes into a propose-role term and a solve-role term. SEGO projects only the solve-role term and leaves the propose-role term unprojected; the two are then summed and passed to the optimizer: g=gpropose+(I−USUS⊤) gsolveg = g_{\text{propose}} + (I - U_S U_S^\top)\,g_{\text{solve}}. This keeps task generation unconstrained, as intended.

We agree that, with shared parameters, the decoupling argument of Sec. 2.3.2 applies to the solve-role component rather than to the full update, and we will narrow the claim accordingly. Within this narrower scope, the decoupling remains meaningful in the shared-parameter setting. Sharing parameters does not imply sharing update directions: the gradient a layer receives lies in the span of the activations of the inputs that produce it, so the propose-role and solve-role terms, computed on different prompts (generating tasks vs. solving them) and toward different objectives, occupy directions determined by their own inputs. Projecting the solve-role term therefore constrains the update driven by task solving, while the update driven by task generation follows its own directions. Consistent with this, AZR, where the propose-role term is left unprojected, is also a setting in which SEGO yields substantial safety gains (Table 1).

In the revision, we will state the AZR implementation explicitly (Sec. 2.1, Sec. 2.3.2, App. C.1) and describe the decoupling at this scope: the projection applies to the solving update; in frameworks with separate proposer and solver (R-Zero) this is the solver's entire update, while in shared-parameter frameworks (AZR) the propose-role term is left unprojected by design. We thank the reviewer again for helping us make this point precise.

W1 (decoupling) / Q3: Proposer ablation on R-Zero with three seeds

We thank the reviewer for this suggestion. We have rerun R-Zero, SEGO (solver-only), and SEGO (proposer + solver) on Qwen3-4B-Base with three seeds. The table below reports ASR on each safety benchmark (mean ± std across three seeds).

SettingPKU ↓AdvB ↓HB ↓JBB ↓SB ↓HEx ↓
R-Zero0.3763 ± 0.02420.1269 ± 0.00690.3433 ± 0.01130.2633 ± 0.03510.3843 ± 0.00250.3622 ± 0.0168
SEGO (solver-only)0.3710 ± 0.02330.0923 ± 0.00510.3033 ± 0.01180.2333 ± 0.02520.3747 ± 0.00550.3244 ± 0.0135
SEGO (proposer + solver)0.3743 ± 0.03090.0968 ± 0.00870.3025 ± 0.01320.2200 ± 0.04360.3767 ± 0.00500.3300 ± 0.0167

Across seeds, projecting the proposer in addition to the solver yields essentially the same safety as solver-only projection, with the differences on all six benchmarks within the seed-to-seed variation. This is consistent with Sec. 2.3.2: once the solver's update is projected, additionally constraining the proposer provides no further safety benefit. SEGO's improvement over R-Zero holds across seeds on most benchmarks. We will update Sec. 4.5 and Table 3 with the multi-seed results and revise the discussion accordingly. We thank the reviewer again for this suggestion, which makes the R-Zero results and the proposer ablation more reliable.

Q5: Implementation details

We thank the reviewer for pointing out these missing details, which we will add to App. C.

  • Activations and matrices. The projection is applied to all three MLP matrices. For each matrix, the subspace is computed from that matrix's own input: the MLP input (d=3584d = 3584 for Qwen2.5-7B) for the gate and up projections, and the intermediate activation after SwiGLU (d=18944d = 18944) for the down projection.
  • Side of the projection. The projection is applied on the input side of each matrix, i.e., along the dimension of the representation from which the subspace is computed.
  • Normalization of CC. The threshold is relative: we retain the singular vectors of CC whose singular values satisfy σi≥τσmax⁡\sigma_i \geq \tau \sigma_{\max}, which is equivalent to normalizing CC by its largest singular value.
  • Resulting kk. With τ=5×10−5\tau = 5 \times 10^{-5}, on Qwen2.5-7B, kk ranges from 15 to 823 across layers for the gate and up projections (mean 498), and from 18 to 2384 for the down projection (mean 1233). The full per-layer values will be reported in App. C.
  • γ\gamma. We use γ=8.8×10−6\gamma = 8.8 \times 10^{-6}, which triggers recomputation every 50 steps on average, with intervals ranging from 42 to 63 steps over training.

We thank the reviewer again for these questions, which help make the implementation of SEGO more precise and reproducible.

Reviewer_LhpEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SEGO, a method to reduce safety degradation during self-evolution. It estimates a safety-related subspace from activations on safety examples and projects the solver’s training gradients away from that subspace, while leaving task generation unrestricted. The subspace is refreshed when model parameters change sufficiently.

The authors evaluate SEGO within AZR and R-Zero across six Qwen models, using safety, coding, and mathematics benchmarks. SEGO reduces attack success rates in all reported comparisons against unconstrained self-evolution, while maintaining or improving capability scores in most cases.

Critical strengths
  1. The method addresses safety preservation during self-evolution. It adjusts the solver’s updates to reduce safety loss while leaving task generation unrestricted. This helps models continue learning while retaining their safety behavior.

  2. The results are consistent across models and training methods. SEGO improves safety in all reported comparisons with ordinary self-evolution, while maintaining or improving most capability scores. This gives confidence that the approach is useful beyond a single setup.

Critical weaknesses
  1. The paper omits SaLoRA (ICLR 2025) and GuardSpace (ICLR 2026), which already use subspace constraints to preserve safety. Comparing against these methods would clarify what SEGO adds.

  2. It also omits Ponkshe et al. (ICLR 2026), who find that safety and utility representations overlap. This matters because the gradient directions SEGO removes to protect safety may also support useful learning.

Questions for authors

Clarification requests:

  1. Which parts of SEGO are methodologically new compared with SaLoRA (ICLR 2025) and GuardSpace (ICLR 2026)?
  2. Ponkshe et al. (ICLR 2026) find overlapping safety and utility representations. What evidence supports interpreting SEGO’s selected principal components as safety-specific? Could blocking these directions explain the capability losses in some settings?
  3. Table 9 compares adaptive refresh, averaging roughly 50 steps, with fixed intervals of 10 and 200 steps. Do you have a comparison with fixed refresh every 50 steps? This would help determine whether the benefit comes from the adaptive trigger or the refresh frequency.
6 comments
AuthorsOpenReview ↗

Response to Reviewer LhpE (1/3)

We thank the reviewer for the constructive review. We are glad that the reviewer sees value in constraining only the solver's updates while keeping task generation free, and that the reviewer finds the safety improvements consistent across the models and frameworks we evaluated. We respond to each point below.

W1 / Q1: Relation to SaLoRA and GuardSpace, and what SEGO adds

We thank the reviewer for raising this question, which gives us the opportunity to clarify how SEGO relates to SaLoRA and GuardSpace. We agree that SaLoRA and GuardSpace share a common mathematical form with SEGO: an update is constrained by an orthogonal projector of the form (I−UU⊤)(I - UU^\top) derived from the model's representations. This operator is a shared building block of these methods. The methodological novelty of SEGO operates at two levels. First, SEGO preserves safety at the level of the update itself, bringing this operator into the evolutionary loop as an intrinsic safeguard that requires no change to the self-generated data, the proposer or the verifier, and no post-hoc realignment. Second, SEGO adapts the operator to a setting where the premises of prior methods no longer hold: rather than keeping a static target invariant under a fixed data source, it must protect safety while the model continually rewrites its own training signal, which leads SEGO to protect only the dominant safety directions selected by τ\tau to balance safety protection against room for capability growth, track them as the model drifts, and constrain only the solving update.

Problem setting. SaLoRA and GuardSpace preserve the safety of an aligned chat model during LoRA fine-tuning on a fixed dataset. The safety constraint is built into the adapter structure and computed once from the aligned model. GuardSpace restricts adapter updates to the null space of harmful inputs, so that the outputs on these inputs remain unchanged; SaLoRA projects adapter outputs orthogonally to the original safety features, so as to maintain those features. With the base weights frozen and the adaptation low-rank and short, a subspace computed once remains valid throughout training. Self-evolution differs in kind. The model generates its own training signal in a closed loop, there is no fixed dataset to inspect in advance, and full-parameter RL can move the model considerably further than adapter fine-tuning.

Protecting dominant directions rather than the full space. In this setting, the evolutionary gradient lies largely within the span of safety representations (Fig. 2a). A full null-space constraint in the style of GuardSpace would therefore remove most of the evolutionary signal. SEGO instead protects only the dominant directions of this space, with τ\tau governing the trade-off, which leaves room for capability growth: the projected gradient retains most of its magnitude throughout training (Fig. 4).

Tracking the subspace as the model drifts. Because the protection is partial, the protected representations are no longer held fixed, and they drift as the model evolves (Fig. 5). A subspace computed once becomes stale and gradually loses its protection (Table 9, "Fixed"). SEGO therefore treats the subspace as a function of the current parameters and re-estimates it as the model moves. Tracking the subspace is thus a direct consequence of the decision to protect only dominant directions, rather than an addition to an existing method.

Constraining the solving update. Self-evolution has a co-evolutionary structure that fine-tuning does not: one component generates the curriculum and another learns from it. SEGO constrains only the solving update and leaves task generation unconstrained, so that safety is protected without restricting the curriculum that drives capability growth.

The partial protection also motivates the question the reviewer raises in Q2, of what evidence supports interpreting the protected directions as safety-relevant; we address it with a benign-corpus control in our response to Q2.

In the revised version, we will reorganize the related work into three parts: self-evolving agents; safety degradation and safety interventions under self-evolution; and a new part on subspace-constrained optimization that covers SaLoRA and GuardSpace, together with related projection methods from continual learning and model editing. This new part discusses SEGO alongside methods that apply subspace constraints in other settings, and clarifies how self-evolution changes the problem such constraints must solve, a problem that SEGO is designed to address. We thank the reviewer again for this suggestion, which helps us position SEGO more clearly.

AuthorsOpenReview ↗

Response to Reviewer LhpE (2/3)

W2 / Q2: Ponkshe et al., safety relevance of the protected directions, and capability losses

We thank the reviewer for pointing us to Ponkshe et al. (ICLR 2026), which is directly relevant to SEGO, and we will discuss it in the revised related work. We agree that safety and utility representations overlap, and SEGO is designed with this overlap in mind. Ponkshe et al. show that projecting fine-tuning updates against subspaces derived from alignment and safety weight updates cannot selectively remove harmful behavior without a comparable loss in utility. SEGO pursues a complementary goal: limiting the incidental safety drift that arises during benign capability training, at a capability cost controlled by τ\tau. This is also why SEGO protects only the dominant directions selected by τ\tau rather than the full span of safety representations, since protecting the full span would also block much of the signal that drives capability growth. Relatedly, Wei et al. (2024) observe that this overlap varies across layers and is smaller in MLP layers than in attention layers, and SEGO applies its projection to the MLP layers (Sec. 4.1).

Whether the protected directions are safety-relevant. We examine this question through two comparisons. First, the generate variant in App. D.1 builds the subspace by the same procedure from the same harmful prompts, using the model's own responses instead of the curated corpus. Its protection is markedly weaker, close to a random subspace at 3B (Table 7), which shows that the protective effect depends on the content of the corpus rather than on the projection alone. Second, to test the role of safety content directly, we built a control subspace by the same procedure from a benign corpus of 1,024 general instructions (Dolly-15k), with no harmful requests or refusals, and calibrated it to remove a share of gradient energy comparable to that of SEGO. Results on Qwen2.5-Coder-7B (AZR) are below.

Subspace sourceRemoved energyMean ASR ↓
Unconstrained AZR00.3705
Benign corpus (energy-matched)0.13580.3504
Safety corpus (SEGO)0.13960.3020

Under matched energy, the benign subspace provides limited protection, whereas the safety subspace reduces ASR to 0.3020. Since the two subspaces remove a comparable share of gradient energy, this difference points to the content of the corpus, suggesting that the subspace SEGO protects carries safety-relevant content. This is compatible with the overlap reported by Ponkshe et al.: overlapping subspaces can still differ in how much protection they provide.

Capability losses. We agree that, because of this overlap, the directions SEGO removes can carry some signal that is useful for learning, and this can account for the capability losses in some settings, such as Qwen2.5-Coder-7B. This is the trade-off that τ\tau governs: a larger protected subspace strengthens safety but removes more of the evolutionary signal (App. D.2). Under the default τ\tau, the projected gradient retains most of its magnitude throughout training (Fig. 4). Part of this cost can, however, be recovered with longer training: on Qwen2.5-7B, the capability gap that appears early in training narrows as training continues (App. E).

In the revision, we will discuss Ponkshe et al. and Wei et al. in the related work, add the benign-corpus control to App. D.4 (Table 10 in the revised version), describe the relation between SEGO's subspace and capability as a trade-off governed by τ\tau, and qualify the statement that capability is preserved throughout by noting the settings with capability losses. We thank the reviewer again for this question, which helps us state more precisely what SEGO's subspace captures.

AuthorsOpenReview ↗

Response to Reviewer LhpE (3/3)

Q3: Fixed refresh every 50 steps

We thank the reviewer for this suggestion, which helps separate the effect of the adaptive trigger from that of the refresh frequency. We ran SEGO with a fixed refresh every 50 steps under the same setting as Table 9 (AZR, Qwen2.5-7B).

RecomputationASR (avg) ↓Code (avg) ↑Math (avg) ↑
Every 50 steps (fixed)0.34610.50560.4539
Every ~50 steps (adaptive)0.34100.50740.4536

With a comparable average refresh frequency, the fixed schedule yields slightly higher ASR (0.3461 vs. 0.3410), while capability remains similar. One plausible explanation is that the difference comes from when the subspace is refreshed rather than how often. The adaptive trigger accumulates the magnitude of the projected updates, so recomputation comes earlier in phases where the model is updated more strongly and later in phases where the updates are smaller. Larger updates tend to move the safety-relevant representations further, although not in strict proportion, so the adaptive trigger refreshes the subspace roughly when it is likely to have drifted. A fixed interval cannot respond to such changes: in phases with larger updates, the subspace can become stale before the next scheduled refresh, leaving part of the drift unprotected. We will add this comparison to App. D.3. We thank the reviewer again for this suggestion, which helps clarify the role of the adaptive trigger.

Reviewer_PXJwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on safety mitigations against self-evolution. They propose SEGO, which allows for safe, constrained self-evolution. The paper shows that self-evolution may improve mathematical reasoning/coding, but it degrades safety. The authors evaluate their approach by comparing pre-evolution models, models evolved using existing frameworks such as AZR and R-Zero, and models evolved using SEGO.

Critical strengths
  • The paper is very topical and timely; there has been significant progress in self-evolution with LLMs, but much less focus on safety.

  • Uses multiple safety benchmarks and capability benchmarks to assess the method.

Critical weaknesses
  • I think the claim regarding geormetry might be very strong, for eg in L477/478, the authors say that misevolution occurs due to geometry, evidence does not establish that geometry is the underlying cause of safety degradation. The results show that SEGO is able to recover ASR in ~15 out of 36 scenarios, this probably means that gradient geometry matters but not that is the main reason.

  • Maybe i lack enough background, but some of the inferences esp w the graphs were hard to digest or less well justified. In sec 4.2 retention ratio is introduced. A high ratio shows that the projection removes a small fraction of gradient magnitude. It does not establish that the retained gradient is capability-driving, that the removed component is safety-critical, or that the two types of behavior are independently controllable. I would appreciate some explanation around the this please.

  • Claim in L315. However, when checking overrefusal, SEGO tends to over-refuse in some cases.

Questions for authors
  • Is it possible to have the multi-seed sd into 1 table, i feel that there are overlaps between some of the results were are hard to imagine if s and avgs are in different tables, esp with so many numbers.

  • Section 4.3 shows that SEGO stabilizes the safety subspace throughout training. However, does this representational stability 1-1 translate into better behavioral safety at each evolution iteration? How does ASR change with each iteration? I assume it might be a bit noisy?

  • Does gradient retention measure the magnitude of the final optimizer update?

4 comments
AuthorsOpenReview ↗

Response to Reviewer PXJw (1/3)

We thank the reviewer for the thoughtful review. We appreciate the reviewer's view that safety in self-evolution is a timely question that has so far received limited attention, and the reviewer's recognition of the range of safety and capability benchmarks in our evaluation. We respond to each point below.

W1: Strength of the claim that misevolution is due to geometry

We thank the reviewer for this comment and agree with the reading that our evidence shows geometry to be an important factor in safety degradation rather than establishing it as the sole underlying cause; we will narrow the claim accordingly.

Our geometric view originates from an observation that we found puzzling. In self-evolution, the tasks that drive capability growth are benign, such as code and mathematical reasoning, and contain no unsafe content, yet safety degrades as capability improves. This led us to examine where the evolutionary update goes. The pilot in Sec. 2.2 shows that the evolutionary gradient enters the space spanned by safety representations, and that under self-evolution the direction of this component shifts from round to round (Fig. 2b). This suggested that safety can be affected as a side effect of capability updates, without any unsafe signal in the data, and motivated us to constrain the update geometry directly.

The results support that geometry matters: SEGO lowers ASR relative to unconstrained evolution on all 36 model–benchmark pairs, and in about half of them brings ASR back to or below the pre-evolution level. In the remaining pairs, the recovery is partial. One possible explanation is that safety- and capability-relevant directions are partly entangled, so SEGO protects only the dominant safety directions selected by τ\tau, and this selective protection leaves part of the drift unaddressed in exchange for capability growth (Fig. 4). Factors outside the protected subspace may also contribute (Sec. 5). We therefore read these results, as the reviewer does, as showing that update geometry is an important factor in misevolution and a practical lever for mitigating it, rather than its only cause.

In the revision, we will rephrase the claim in the abstract, the introduction and the conclusion (L477–478), replacing "geometric in origin" with the statement that misevolution admits a geometric account and can be mitigated geometrically. We thank the reviewer again for helping us state this claim more precisely.

AuthorsOpenReview ↗

Response to Reviewer PXJw (2/3):

W2: The retention ratio and its relation to capability, safety and separability

We thank the reviewer for these questions, which give us the opportunity to clarify what the retention ratio in Sec. 4.2 measures and how it relates to the other evidence in the paper.

The retention ratio was introduced to address a practical concern. Because SEGO removes part of every evolutionary update, the projection could in principle remove enough of the signal to hold back capability growth. A high retention ratio indicates that this does not happen: most of the gradient survives the projection, so the evolutionary signal remains largely intact. Whether this retained signal actually drives capability can then be read from the outcomes. If the projection had removed much of what drives capability, SEGO would fall behind unconstrained evolution on the capability benchmarks. Instead, it largely keeps pace on most of them (Table 2), and on Qwen2.5-7B, SEGO starts more slowly but the gap narrows as training continues (App. E).

To examine whether the removed component is related to safety, we compare SEGO's subspace with control subspaces built in different ways (App. D.1, Table 7). A random subspace of the same dimension provides almost no protection, so the protection does not come from projecting out an arbitrary subspace of that size. The generate variant, built by the same procedure from the same harmful prompts but with the model's own responses instead of the curated corpus, provides markedly weaker protection, which shows that the protection depends on the corpus used to build the subspace. To isolate the role of the corpus content more directly, we additionally ran a benign-corpus control. We built a subspace by exactly the same procedure from a benign corpus of 1,024 general instructions (Dolly-15k) that contains no harmful requests or refusals, and set its threshold so that it removes a share of gradient energy comparable to that of SEGO's subspace. Results on Qwen2.5-Coder-7B (AZR) are below.

Subspace sourceRemoved energyMean ASR ↓
Unconstrained AZR00.3705
Benign corpus (energy-matched)0.13580.3504
Safety corpus (SEGO)0.13960.3020

The benign subspace provides limited protection, whereas the safety subspace reduces ASR to 0.3020. Since both subspaces remove a comparable share of gradient energy, this difference points to the content of the corpus, indicating that the component SEGO removes carries safety-relevant content.

On the third point, prior analysis has found that safety and utility are difficult to separate linearly (Ponkshe et al., 2026), so the two behaviors cannot be controlled fully independently. This is a difficulty that SEGO is designed to address. Rather than seeking a clean separation, SEGO protects the dominant safety directions selected by τ\tau, which sets how much of the safety-related space is protected and thus balances safety against room for capability growth. The separability we refer to in Sec. 4.2 is meant in this practical sense: projecting out the subspace preserves safety while most of the evolutionary signal, and most of the capability gains, are kept.

In the revision, we will rewrite Sec. 4.2 to explain what the retention ratio measures and how it relates to the behavioral results, describe the separability in this practical sense, and add the benign-corpus control to App. D.4 (Table 10 in the revised version). We thank the reviewer again for these questions, which help us state this part of the paper more precisely.

AuthorsOpenReview ↗

Response to Reviewer PXJw (3/3)

W3: Over-refusal in some cases (L315)

We thank the reviewer for checking the over-refusal results. Relative to the pre-evolution model, SEGO's refusal rate on OR-Bench is lower in all six configurations, and on XSTest-Safe it is at or below that level in all six. Relative to unconstrained evolution, SEGO refuses more benign prompts in two cases, XSTest-Safe on Qwen3-4B and OR-Bench on Qwen2.5-Coder-3B (Table 1). In both cases, unconstrained evolution lowers the refusal rate below the pre-evolution level, and SEGO retains refusal behavior closer to that of the original model. On JBB-Benign, SEGO's refusal rate is lower than that of unconstrained evolution in all six configurations, while on Qwen3-4B it remains above the pre-evolution model.

In the revision, we will describe the over-refusal results at this level of detail: SEGO does not raise over-refusal above the pre-evolution level on OR-Bench and XSTest-Safe, and in a few cases it refuses more benign prompts than unconstrained evolution. We thank the reviewer for helping us make this description more precise.

Q1: Reporting multi-seed means and standard deviations in one table

We thank the reviewer for this suggestion. We agree that reporting means and standard deviations in the same table makes it easier to judge which differences exceed seed-to-seed variation. In the revised version, App. G reports the main results as mean ± standard deviation over three seeds in the same table.

Q2: Subspace stability and behavioral safety over training

We thank the reviewer for this question, which gives us the opportunity to clarify the role of subspace stability in SEGO and how it relates to behavioral safety. The stability of the safety subspace keeps the projection accurate between recomputations: when the subspace stays close to where the safety-relevant directions currently lie, the projection continues to remove the right component of the update. Its effect on behavior therefore passes through the projection. Since the projection protects only the dominant safety directions, factors outside the protected subspace can also affect safety (Sec. 5), and ASR at a given iteration is subject to evaluation noise, we would not expect the two to correspond point by point. What the stability should support is a consistent safety advantage over the course of training.

To examine this, we evaluated the intermediate checkpoints of SEGO and unconstrained evolution on the mean ASR across the six safety benchmarks, every 50 steps under AZR (Qwen2.5-Coder-3B).

CheckpointStep 0Step 50Step 100Step 150Step 200Step 250Step 300
Unconstrained evolution (AZR)0.55790.56220.57180.57090.58020.58980.5970
SEGO (AZR)0.55790.55990.55620.56080.56670.56640.5613

As the reviewer anticipates, the curves fluctuate between neighboring checkpoints. Over the course of training, SEGO's ASR generally stays below that of unconstrained evolution.

In the revision, we will add these trajectories to the appendix alongside Sec. 4.3. We thank the reviewer again for this suggestion, which connects the representational analysis more directly to the behavioral results.

Q3: The retention ratio and the final optimizer update

We thank the reviewer for this careful question, which gives us the opportunity to clarify what the retention ratio is meant to capture. The retention ratio is computed on the gradient before it is passed to the optimizer (App. C.1). It therefore measures how much of the evolutionary signal passes through the projection, rather than the magnitude of the final parameter update. We will state this explicitly in Sec. 4.2 in the revision, and we thank the reviewer again for helping us make this point more precise.

Dear Reviewer PXJw,

Thank you again for your thoughtful review and valuable comments. We hope our response has clarified the points you raised, and we would be happy to answer any further questions. We sincerely appreciate your time and effort in reviewing our paper.

Best regards,
Submission 15733 Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion and the revision

We thank the Area Chair and the reviewers. For each reviewer, we summarize the main points, how we addressed them, and where they are reflected in the revised PDF.

Contribution. In self-evolution, safety degrades even though the self-generated tasks are benign, so the degradation is difficult to address at the data level. SEGO acts on the solving update: it protects the dominant directions of the model's safety representations, re-estimates them as the model drifts, and leaves task generation unconstrained. Each of these choices is designed for what self-evolution specifically requires: partial protection that leaves room for capability growth, tracking of a drifting subspace, and a constraint on the solving update only, so that the co-evolutionary curriculum is not restricted. Across two frameworks and six models, SEGO lowers ASR relative to unconstrained evolution on all 36 model–benchmark pairs and compares favorably with KL, SPF, post-hoc DPO and replay.

Reviewer LhpE

  • Relation to SaLoRA, GuardSpace and projection methods (W1/Q1): new related-work part on subspace-constrained optimization (App. A); AZR solve-role projection (Sec. 2.1, 2.3.2, App. C.1); three-seed proposer ablation (Sec. 4.5, Table 3).
  • Ponkshe et al. and safety relevance (W2/Q2): energy-matched benign-corpus control (App. D.4, Table 10); capability losses described as a trade-off governed by τ (Sec. 3.2, 4.4, 5).
  • Fixed 50-step refresh (Q3): App. D.3, Table 9, with the reviewer's reading of the result.

Reviewer PXJw

  • Strength of the geometric claim (W1): narrowed to "admits a geometric account" (Abstract, Sec. 1, 2.2, 5).
  • Retention ratio and separability (W2, Q3): Sec. 4.2 now states what the ratio measures and how it relates to the behavioral results; benign-corpus control (App. D.4).
  • Over-refusal (W3): described per benchmark, including the two cases where SEGO refuses more benign prompts than unconstrained evolution (Sec. 3.2).
  • Multi-seed results in one table (Q1): App. G. ASR over training (Q2): App. E, Table 14.

Reviewer PtMr

The review listed the points that could change the assessment. We have addressed each of them as follows:

  • Benign-corpus control matched on removed energy: App. D.4, Table 10 (also discussed with Reviewer LhpE).
  • Corrected Table 8 with per-benchmark results: App. D.2, D.5.
  • Per-seed scores: App. G, with per-seed correct/total counts; the gap between SEGO and unconstrained evolution exceeds the larger of the two standard deviations in 33 of 36 pairs.
  • Broader comparison on R-Zero with two settings per baseline and replay: App. F, Table 18.
  • Rescoped claims: "first evidence of separability" removed; the geometric, decoupling and capability claims narrowed (Abstract, Sec. 1, 4.2, 4.4, 5).
  • AZR implementation: Sec. 2.1, 2.3.2, App. C.1 (also discussed with Reviewer LhpE).

The remaining questions are addressed in the revision as well: the pilot setup and nested subspaces (Sec. 2.2, Fig. 2), implementation details (App. C.1), evaluation settings (App. C.3), results at 500 steps and under jailbreak attacks (App. F), and the corrected summary statements (Sec. 3.2).

We believe these revisions have made the paper more rigorous and its contributions clearer. We thank the Area Chair and the reviewers again for their time and careful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.