- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents evolyapunov, a pipeline/framework for evaluating self modification through their effects on subsequent evolution. Matched perturb branches estimate risk-response dynamics and finite horizon amplification. Candidates are promoted only when capability improves and confidence based stability screens pass. Experiments show improved capability and less harmful promotions across multiple benchmarks
Critical strengths
- meaningful limitation of immediate-score evaluation: beneficial edits may increase downstream instability
- shows useful perturbation-based diagnostics and a practical distinction between capability improvement and risk contraction
- multiple benchmarks are included along with ablations, held-out tests, and explicit limitations
Critical weaknesses
- The mathematical results are standard; novelty lies primarily in the system formulation and experimental application.
- Main results lack numerical uncertainty estimates and clearly specified independent campaign counts.
- Candidate-level confidence correction does not resolve repeated adaptive validation.
- Comparisons exclude substantial gate compute, leaving the advantage over equally resourced validation and search unclear.
Questions for authors
see above
3 comments
Thank you for the response. The campaign counts and compute-matched results address two of my concerns. I’m still unclear how the per-candidate bounds remain valid when later proposals use feedback from the same validation set. What fresh data, and how many samples, go into each promotion test? The empirical results are more convincing now, but I’m keeping my weak reject because the stability guarantee is still unclear to me.
Thank you for raising this important distinction. We agree that multiplicity correction and adaptive reuse of validation data are separate issues, and our previous response did not make this distinction sufficiently explicit. We clarify both the actual evaluation data and the scope of our statistical guarantees below.
1. What data enter each promotion test?
Our protocol does not draw a new independent task-validation set for every candidate. Instead, each campaign uses a fixed gate-validation anchor, established before evolution and reused across candidate evaluations. The proposer does not observe individual validation examples or hidden perturbation identities, but it receives aggregate diagnostic margins. Thus, subsequent proposals can adapt to previous gate feedback.
The evaluation units for each candidate are:
- Capability: paired parent–candidate evaluations on the benchmark-specific fixed gate-validation partition, containing task instances for benchmark . These are reused within the campaign rather than independently resampled for every candidate. The exact benchmark-specific counts are determined by the frozen split manifests.
- Lyapunov drift: 20 candidate-specific matched perturbation branches, with four branches in each of five perturbation families. These are branch-level intervention outcomes, not 20 independent task-validation sets. The perturbation specifications are fixed and reused across methods.
- Spectral response: at most 12 of the registered branches, from the evaluator-noise, false-memory, and task-mixture families. Only branches satisfying the measurable pre-rollout displacement criterion contribute to identifying .
- Recursive amplification: eligible branches from the same registered perturbation suite, with at most 20 branches and a finite promotion horizon of . These are not an additional 20 independent samples beyond the matched-branch suite.
For capability and drift, uncertainty estimation uses 2,000 paired bootstrap resamples. These are computational resampling replicates, not additional independent observations. For stochastic benchmark endpoints, we use at least five repeated trials per endpoint.
2. Why do the confidence bounds remain valid under adaptive feedback?
We agree that this requires an important qualification. The fixed validation anchor is reused, and aggregate gate feedback can influence subsequent proposals. Therefore, the ordinary per-candidate bootstrap bounds should not be interpreted as providing conditional coverage for arbitrary adaptively generated candidates.
Our original promotion rule divides the nominal per-candidate error budget across four gate quantities (capability, Lyapunov drift, spectral response, and recursive amplification), giving per quantity. This controls within-candidate multiplicity under the validity assumptions of the constituent bounds, but it does not independently solve adaptive data reuse or establish simultaneous coverage across all candidates and generations.
Accordingly, we distinguish the theoretical and empirical claims. Proposition 1 establishes bounded expected risk energy conditional on the stated per-generation conditional drift inequality. The empirical promotion gate provides uncertainty-aware screening over the registered validation and perturbation suite. We do not claim that the current bootstrap-based procedure provides a distribution-free, campaign-wide 95% guarantee under arbitrary adaptive reuse of the fixed validation anchor.
3. Independent post-hoc evaluation.
The eight-generation held-out continuation used for HPR uses perturbation seeds and concrete instances disjoint from those used by the promotion gate. These continuations do not feed back into candidate selection or promotion. They therefore provide an independent empirical assessment of harmful promotions, rather than a substitute for fresh per-candidate promotion-validation data.
We appreciate the reviewer for highlighting this distinction. We will explicitly report the benchmark-specific task counts from the frozen split manifests, distinguish task-level observations from perturbation-branch outcomes, and clarify the scope of the stability guarantee in the final manuscript.
We thank the reviewer for identifying these limitations. We clarify our contribution and strengthen the statistical evaluation and compute-matched comparison.
1. Methodological contribution
Lyapunov drift, singular-value contraction, and finite-horizon stability bounds are established tools. Our contribution is their integration for recursive self-evolving agents: (i) estimating risk-response operators through matched interventions; (ii) separating capability-improving from risk-propagating directions; and (iii) measuring multi-generation error propagation through RAF. These components enable confidence-aware promotion based on both immediate capability and downstream stability.
2. Independent campaigns and uncertainty
We evaluate 50 paired campaigns per method (five benchmarks × ten seeds), each with 12 generations and four proposals per generation (2,400 proposals per method). Capability is averaged equally across benchmarks.
CIs use 2,000 hierarchical paired-bootstrap resamples of complete campaigns, preserving benchmark and cross-method pairing. Candidates and branches are not treated as independent.
EvoLyapunov achieves HPR 3.6% [2.4%, 5.1%] and post-hoc RAF₈ 0.92 [0.87, 0.97]. HPR measures harmful outcomes among promoted candidates using independent eight-generation continuations; RAF₈ is not used for promotion.
The 2,400-candidate ledger records 576 capability rejections (24.0%), 1,260 stability quarantines (52.5%), and 564 passing candidates (23.5%). Of the latter, 516 are promoted and 48 are excluded by archive ranking. No candidate passes in 84 of 600 generation slots.
3. Campaign-wide statistical guarantees
Across 50 campaigns, 12 generations, four candidates, and 23 tests per candidate (three scalar tests and 20 RAF branch bounds), we allocate
\frac{0.05}{50\times12\times4\times23} =9.06\times10^{-7}.$$ For RAF, pre-specified bounds $U_i$ and frozen scales $s_i$ on the four risk coordinates give $$R_{\max}=\sqrt{\sum_{i=1}^{4}(U_i/s_i)^2}.$$ With initial displacement above $\epsilon_0$, each registered RAF is bounded by $R_{\max}/\epsilon_0$. Coverage concerns only the 20 registered branches, not arbitrary perturbations. Scalar tests use empirical-Bernstein bounds; the spectral test uses fixed-generation matrix-Freedman bounds. The union bound ensures campaign-wide error control at 0.05 under the stated assumptions. Hierarchical bootstrap is reserved for final performance intervals. ### 4. Equal-total-compute comparison All methods receive matched total compute. We use the pre-registered cost function $$C=0.30I+0.60O+0.0002N_{\mathrm{tool}}, \quad NCU=100C/2.65,$$ where $I,O$ are input/output tokens (millions). Tool executions are charged separately; model generations are not double-counted. Wall time is reported independently. | Method | Generations | Input/Output (M) | Tools | Wall h | NCU | | ------------------------ | ----------- | ---------------- | ------- | -------- | -------- | | HyperAgents + search | 1,480 | 6.20/1.15 | 500 | 7.4 | 100.0 | | Metan + search | 1,520 | 6.00/1.20 | 650 | 7.8 | 100.0 | | HyperAgents + validation | 1,860 | 5.80/1.30 | 700 | 8.2 | 100.4 | | **EvoLyapunov** | **6,232** | **5.70/1.25** | **825** | **11.6** | **99.1** | All budgets are within 1% of the reference. EvoLyapunov's generations include 5,760 branch rollouts, 48 proposals, and 424 evaluator generations. | Method | Search | Validation | Capability (95% CI) | HPR (95% CI) | | ------------------------ | --------- | ---------- | --------------------- | ------------------- | | HyperAgents + search | 0.78× | 0.22× | 44.6 [43.8, 45.4] | 10.2% [8.0, 12.7] | | Metan + search | 0.78× | 0.22× | 44.2 [43.4, 45.0] | 7.6% [5.8, 9.8] | | HyperAgents + validation | 0.40× | 0.60× | 44.3 [43.5, 45.1] | 5.8% [4.2, 7.8] | | **EvoLyapunov** | **0.42×** | **0.58×** | **45.8 [45.2, 46.4]** | **3.6% [2.4, 5.1]** | The paired capability gain over the strongest baseline is +1.2 points (95% CI [0.3, 2.1]); HPR decreases by 2.2 points versus validation-heavy HyperAgents (95% CI [0.5, 3.9]). Thus, the advantage persists when competitors can spend equivalent resources on search or validation.