- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies two decisions in self-evolving LLM systems: when to stop optimizing persistent artifacts such as prompts or skills, and which historical artifact to return. The proposed method constructs restart-based sequential tests from paired candidate–incumbent evaluation outcomes and uses the strongest restart process to estimate a change point for output selection. Experiments cover two self-evolving frameworks, three model families, and five benchmarks. The main results show 48.4%–91.6% token savings relative to full-budget runs, with relatively small observed differences in unseen-test accuracy.
Critical strengths
-
A useful problem formulation with a practical interface. Separating stopping time from output selection is well motivated: additional rounds may be needed to establish that progress has stalled, while an earlier artifact may be preferable for deployment. The method reuses existing paired evaluation outcomes and can be integrated without changing candidate generation or acceptance, making the proposed approach practically relevant.
-
Promising computational savings supported by a statistical formulation. Table 1 demonstrates substantial reductions in token consumption while retaining much of the improvement obtained by the underlying evolution process. The explicit assumptions and false-alarm run-length guarantee provide a principled foundation, while the experiments with shared default parameters across models and frameworks offer encouraging initial evidence of applicability.
Critical weaknesses
-
The comparisons do not establish an advantage over simpler stopping strategies. The main baseline is running the underlying system to a fixed budget. There are no direct comparisons with patience-based stopping, development-set-calibrated short budgets, or simple gain-threshold rules. SkillOpt and GEPA serve as host frameworks rather than competing stopping methods, and aGRAPA versus the fixed mixture is an internal ablation. Consequently, the reported savings do not isolate the benefit of the proposed statistical construction over ordinary early stopping. Moreover, the paper does not compare change-point-based output selection with returning the incumbent at the same alarm time. These comparisons are essential to substantiate the contributions of both components.
-
Evidence for robust performance preservation and transfer remains limited. Cross-model and cross-framework experiments concentrate on SearchQA and GSM8K; the more complex tool-oriented tasks receive substantially narrower coverage. Table 7 reports stopping times and change-point estimates, but not the corresponding accuracy–cost trade-offs across parameter settings, and the procedure used to choose the shared defaults is insufficiently documented. Multiple independent evolution runs are also not reported. The paired confidence intervals quantify uncertainty across test examples, not variation across evolution trajectories, and intervals containing zero do not establish non-inferiority—particularly given the wide OfficeQA interval. These limitations weaken the evidence that the method reliably preserves performance across tasks and search dynamics.
Questions for authors
-
Comparative evidence: How does the method compare with a tuned patience rule and a fixed short-budget baseline under a matched development and evaluation protocol? Does it retain an advantage at comparable test accuracy or computational cost?
-
Component attribution: At the same alarm time, how does change-point-based output selection compare with returning the current incumbent? What proportion of the benefit can be attributed specifically to retrospective selection?
-
Clarification and robustness: How were the default parameters selected, and were they fixed before evaluating the reported tasks? Can repeated runs, particularly on OfficeQA and SpreadsheetBench, establish the variability of token savings and test-performance changes?
5 comments
Response to Reviewer wfm2 (2/2)
Q3. Clarification and robustness
How were the default parameters selected, and were they fixed before evaluating the reported tasks?
Each default has a fixed meaning rather than being tuned for performance. sets the smallest worthwhile gain to one percentage point of validation accuracy per round. is the conventional 5% level and gives an average run length to false alarm of at least 20. The mixture grid and the aGRAPA initial fraction are moderate betting fractions within the valid range . We set these values on SkillOpt/SearchQA, our first task, and kept them unchanged for every later task, model and framework.
Can repeated runs, particularly on OfficeQA and SpreadsheetBench, establish the variability of token savings and test-performance changes?
We repeated SpreadsheetBench with a new seed and the same defaults. The two runs compare as follows.
| Original run | New seed | |
|---|---|---|
| Validation accuracy, round 0 → 16 | 37.5% → 70.0% | 32.5% → 67.5% |
| Rounds with accepted updates | 3, 5, 9, 11 | 1, 5, 8 |
| Alarm within 16 rounds | No | No |
| Token saving | 0% | 0% |
Both runs keep improving, and neither reaches the alarm threshold. And the returned artifact is the full-run artifact. Two runs show that this outcome is reproducible. We did not repeat OfficeQA, since the full run costs more than 200M tokens and more than 40 hours, which exceeds our rebuttal budget.
W1. The comparisons do not establish an advantage over simpler stopping strategies
Q1 compares our rule with patience stopping and fixed short budgets, and Q2 compares change-point output selection with returning the incumbent at the same alarm time. For gain-threshold rules, we also replayed a patience variant that resets only on a validation gain of at least ; it gives the same stopping rounds as the patience rules in Q1.
W2. Evidence for robust performance preservation and transfer remains limited
Accuracy–cost trade-off in Table 7. Changing changes when the detector stops, but the returned artifact remains unchanged. Across all settings that stop, each benchmark returns the same artifact, so unseen-test accuracy is unchanged. The table gives the tokens saved relative to the full run for each setting in Table 7.
| SearchQA 0.01 | 0.05 | 0.10 | GSM8K 0.01 | 0.05 | 0.10 | LiveMath 0.01 | 0.05 | 0.10 | OfficeQA 0.01 | 0.05 | 0.10 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.002 | 77% | 92% | 92% | 82% | 82% | 87% | 0% | 15% | 38% | 41% | 48% | 56% |
| 0.005 | 89% | 92% | 92% | 82% | 87% | 87% | 0% | 15% | 45% | 41% | 48% | 56% |
| 0.01 | 92% | 92% | 94% | 87% | 90% | 90% | 0% | 38% | 45% | 41% | 48% | 56% |
| 0.02 | 92% | 94% | 94% | 90% | 93% | 93% | 15% | 45% | 54% | 41% | 48% | 56% |
| 0.05 | 94% | 94% | 94% | 93% | 93% | 93% | 38% | 54% | 61% | 48% | 64% | 64% |
A 0% entry means no alarm within the budget; the full-run artifact is then returned. We will add these values to Table 7.
Default parameters and repeated runs. The defaults were set on our first task and kept unchanged for all later tasks, models and frameworks. A repeated SpreadsheetBench run with a new seed reproduces the original outcome (no alarm, same full-run return). Details are in Q3.
Confidence intervals. The paired intervals compare two fixed artifacts, the returned one and the full-run one, on the same test examples. They therefore measure test-sampling uncertainty, not variation across evolution runs. The OfficeQA interval is wide because its test set has only 172 examples. We agree that an interval containing zero does not establish non-inferiority, and will replace "comparable performance" with "no significant difference on the test set" in the revision.
Additional Results on Comparative Evidence and Component Attribution
Thank you again for your review. To further address your concerns, we ran additional experiments related to your Q1 and Q2 and share the results below, together with a small correction to our previous rebuttal.
Comparative evidence
We replayed AutoGluon's adaptive patience rule (AdaptiveES) on all our saved SkillOpt trajectories. Its parameters are fixed by AutoGluon, independently of our runs. It sets patience (class default), where is the round of the last improvement. Our rule uses the paper's defaults and on every run.
In the table, all paired entries are AdaptiveES / our rule. Stop is the stopping round; "none" means no stop within the budget. Returns is the round of the returned artifact. Saved is the share of full-budget tokens not spent. "Same" means both rules return the same artifact, and marks the round-16 reward-hacking artifact.
| Run | Stop | Returns | Saved | Test accuracy |
|---|---|---|---|---|
| SearchQA / DeepSeek | 11 / 4 | 1 / 1 | 75.1% / 91.6% | Same |
| GSM8K / DeepSeek | 11 / 4 | 1 / 1 | 73.2% / 89.8% | Same |
| LiveMath / DeepSeek | none / 8 | 16$^\ddagger$ / 1 | 0% / 37.8% | |
| OfficeQA / DeepSeek | 14 / 8 | 3 / 3 | 12.6% / 48.4% | Same |
| SpreadsheetBench / DeepSeek | none / none | 11 / 11 | 0% / 0% | Same |
| SearchQA / Qwen3-32B | none / 12 | 33 / 8 | 0% / 75.3% | 73.79% / 72.43% |
| GSM8K / Qwen3-32B | 17 / 8 | 5 / 1 | 62.9% / 84.5% | 95.20% / 95.20% |
| GSM8K () / DeepSeek | 11 / 5 | 1 / 1 | 72.7% / 87.3% | Same |
| SpreadsheetBench, new seed | none / none | 8 / 8 | 0% / 0% | Same |
AdaptiveES never stops earlier than our rule, and our rule saves at least as many tokens on every run. On six runs, both rules return the same artifact. Of the remaining three, AdaptiveES returns the reward-hacking artifact on LiveMath; on GSM8K/Qwen3-32B, both artifacts score 95.20% on unseen test data. On SearchQA/Qwen3-32B, AdaptiveES runs the full budget and its round-33 artifact scores 73.79% on unseen test data, compared with 72.43% for ours, while spending more than four times as many tokens.
Component attribution
The change-point output differs from the alarm-time incumbent in two settings: GEPA/GSM8K with DeepSeek V4 Flash and SkillOpt/GSM8K with Qwen3-32B (see Correction below). In both settings, the change-point output achieves the same unseen-test accuracy as the alarm-time incumbent while using fewer prompt tokens. Selecting this output can therefore reduce inference cost.
The table reports prompt tokens summed over the 500 unseen test queries. Savings are relative to the incumbent at the alarm.
| Setting | Artifact | Round | Test accuracy | Prompt tokens | Saved |
|---|---|---|---|---|---|
| SkillOpt/GSM8K/Qwen3-32B (alarm: 8) | Incumbent at alarm | 5 | 95.20% | 436,828 | — |
| SkillOpt/GSM8K/Qwen3-32B (alarm: 8) | Change-point output | 1 | 95.20% | 203,828 | 53.3% |
| GEPA/GSM8K/DeepSeek V4 Flash (alarm: 6) | Incumbent at alarm | 5 | 97.20% | 478,136 | — |
| GEPA/GSM8K/DeepSeek V4 Flash (alarm: 6) | Change-point output | 2 | 97.20% | 357,636 | 25.2% |
In these two settings, output selection reduces prompt-token usage without reducing observed unseen-test accuracy.
Correction
We previously stated that the change-point output equals the alarm-time incumbent in every default setting except GEPA/GSM8K with DeepSeek V4 Flash. SkillOpt/GSM8K with Qwen3-32B (Table 5) is a second exception.
In this setting, the change-point estimate returns the round-1 artifact, whereas the incumbent at the alarm (round 8) is the round-5 artifact, after two accepted updates raised validation accuracy from 0.94 to 0.96. On the 500 unseen test examples, both artifacts score 95.20%. Each update gained only two net validation items out of 200, and the combined validation gain did not translate into higher unseen-test accuracy.
Response to Reviewer wfm2 (1/2)
Thank you for the careful review, insightful questions and suggestions. We address your comments and questions below.
Q1. Comparative evidence
How does the method compare with a tuned patience rule and a fixed short-budget baseline under a matched development and evaluation protocol?
Both baselines require a number fixed before the run: the patience rule needs the number of rounds without improvement to tolerate (p), and a fixed budget needs the total number of rounds (N′). To compare them with our method, we check which value works on each benchmark.
For the patience rule, the loop stops once the incumbent's validation accuracy has not increased for p consecutive rounds, and the incumbent is returned. We replayed this rule for several p on the saved trajectories. In the table below, entries are stopping rounds; † misses a later validation gain; ‡ reaches the LiveMath exploit; "no stop" runs the full budget.
The table shows that small p stops too early on SpreadsheetBench and OfficeQA, while large p reaches the LiveMath exploit. Tuning would therefore select p=5, the only value that avoids every failure. This choice, however, can be made only after the full trajectories are observed. The future trajectory is unknown, so there is no online evidence for which p to use.
A fixed budget N′ faces the same problem. To keep the gain at round 11 on SpreadsheetBench, N′ must be at least 11; to avoid the exploit at round 16 on LiveMath, N′ must be at most 15. Only a budget between 11 and 15 avoids both failures. N′=11 on SearchQA and GSM8K still runs ten rounds past their last gain at round 1, while our rule stops at round 4. As the same case as patience rule, the choice of N′ is known only after the full trajectories are observed.
Although this replay is not a matched development and evaluation protocol, it favors the baselines: their p and N′ are tuned on the very runs they are evaluated on, while our configuration is fixed in advance. Even with this advantage, neither baseline outperforms our rule. Our rule adapts its stopping round to each run, using one fixed configuration on all five benchmarks without failure, and it does not need the full trajectory to do so.
Does it retain an advantage at comparable test accuracy or computational cost?
At equal test accuracy, yes. Our rule returns the same artifacts as the tuned p=5. Our cost is lower on SearchQA (2.18M vs 3.40M tokens) and GSM8K (1.21M vs 1.84M), and higher on LiveMath (7.38M vs 5.48M). However, p=5 is not known unless we do not stop.
Q2. Component attribution
At the same alarm time, how does change-point-based output selection compare with returning the current incumbent?
In every default setting except GEPA/GSM8K, the two are the same artifact. GEPA/GSM8K is the only exception:
The two artifacts differ because an update accepted after the estimated change point raised validation accuracy by 0.005, a net gain of one item out of 200. Their equal unseen-test accuracy may be partly coincidental, but it is what our formulation predicts: a gain below ε=0.01 is treated as not worthwhile, and a one-item difference is within evaluation noise.
What proportion of the benefit can be attributed specifically to retrospective selection?
Retrospective selection does not aim to raise accuracy or save tokens. The savings come from stopping, and in our runs the selected artifact matches the incumbent at the alarm in unseen-test accuracy. Its purpose is to guard against validation overfitting. Between the change point and the alarm, the loop is already in the non-improvement regime. For updates accepted by the gate in this regime, the validation gain could come from overfitting the validation set. Retrospective selection returns the incumbent before the change point and thus avoids the risk of overfitting.