- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SCOPE co-evolves reusable agent skills and executable challenges using discriminativeness, probe-based learning progress, failure novelty, solvability validation, and hall-of-fame replay. Model parameters remain frozen. Experiments with Qwen2.5-72B-Instruct cover four professional domains and several distribution shifts. The reported average pass rate improves from 52.8% with fixed-task optimization to 62.1%, with ablations supporting learnability estimation, validation, replay, and mutation diversity.
Critical strengths
-
A useful curriculum design principle. Combining measured learning progress with executable solvability checks addresses a concrete limitation of failure-maximizing generation: difficult challenges need not provide productive training feedback. The learnability and validation ablations support this distinction.
-
Meaningful reported generalization gains. SCOPE improves over VALIDATED-ADV by 3.1 percentage points on average and 8.1 points on compositional shifts (Table 1). Freezing skills during evaluation makes these results relevant to persistent procedural improvement rather than continued adaptation at deployment.
Critical weaknesses
-
Inconsistencies in method specification and experimental reporting. Equation (1) minimizes distance over the current task population and archive. Since the evaluated task itself belongs to the current population, its self-distance makes novelty zero, disabling its contribution to task fitness and archive admission through the novelty criterion. The supplementary demo instead compares against archive descriptors only. This may be a notation error, but leaves a central component of the implemented method unclear. Additionally, Appendix B.2 specifies random seeds {13, 42, 2026}, whereas Table 9 reports results for seeds {17, 42, 73}. These inconsistencies raise concerns about the precision of the method specification and experimental reproducibility.
-
The main experiments are insufficiently reproducible. Appendix B.1 describes task counts and categories without identifying the source artifacts or adequately specifying seed-task construction. The construction and contents of the eight initial skills are also missing. The supplement explicitly provides a synthetic simulator rather than the four-domain experimental pipeline and data. Consequently, it does not resolve these omissions or allow independent assessment of the custom benchmark and reported improvements.
Questions for authors
-
Which novelty reference set was used in the main experiments: the population excluding the current task, the archive alone, or something else? Does correcting Equation (1) require any experimental results to change?
-
How were the eight initial skills constructed, and what procedures and source artifacts generated the benchmark? Can the actual initial skill packages and task-construction/evaluation artifacts be provided?
-
What incremental benefit does discriminativeness provide beyond measured learnability? Does an ablation removing both its task-fitness term and its archive-admission criterion preserve the gains, with other components held fixed?
-
What absolute execution limit and total GPU hours apply to each baseline under the accounting in Appendix B.3? Table 2 reports costs only for ablations, while Table 5 varies generations; neither establishes the comparison at matched total inference cost.