- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper asks whether self-evolving agents still need a prescribed optimization pipeline when the optimizer is a frontier model. It proposes Open-Ended Optimization (OEO): the framework only fixes the objective, budget, data boundary and evaluator, and the optimizer (GPT-5.5) uses five tools (inspect data, read artifact, run target, write skill, finish) to improve a natural-language skill on its own. OEO is compared with SkillOpt and GEPA on SearchQA, SpreadsheetBench, OfficeQA and LiveMath with two target models. OEO has higher point estimates in 12 of 16 comparisons and uses less target-side tokens. The paper also has a capability ladder, a one-shot rewrite control, and an analysis of edit paths.
Critical strengths
- The question is clear and timely. Separating the fixed "contract" from the optimization procedure is a useful way to frame it.
- The statistics are done carefully, and the one-shot control is a good sanity check.
Critical weaknesses
-
Large seed variance on LiveMath. The headline LiveMath / GPT-5.5 result uses seed 42, which is the best of three runs (0.71 vs 0.55 and 0.50 for seeds 43/44). The SD of OEO is 0.11, about 3-4x of SkillOpt and GEPA. The two largest significant gains in the paper come from this one setting with n=124. Table 2 also keeps the "historical" 85/124 while Table 7 uses 88/124 for the same artifact. With this variance, I am not sure the LiveMath gain is a property of the method or a lucky run.
-
Baselines look very weak. GEPA improves LiveMath / GPT-5.5 only from 0.387 to 0.403, and SkillOpt on SpreadsheetBench / Qwen is even below the initial skill (0.196 vs 0.211) despite having a validation gate. The paper does not say whether official implementations and recommended hyperparameters were used, or why the baselines fail in these settings.
-
Not enough examples of what OEO actually does. The method is very simple, a loop with five tools, so the interesting part is the behavior. But the main paper shows no trajectory, no example of the final skill, no tool-call statistics, and no stopping behavior. The system prompt and action schema are also not in the paper.
-
The gain may come from edit freedom, not from composing the procedure. Figure 4 shows OEO makes much larger and broader edits than SkillOpt, which is restricted to bounded patches. So the gain over SkillOpt may simply come from allowing full rewrites. An ablation is needed to support the claim that the procedure itself is what matters.
Questions for authors
Please see weaknesses.