- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
In this paper, the author proposes a framework, Automatic Skill Discovery (ASD), to explore letting LLMs compose the skill bank by self-decision with no human intervention. Within the framework, the method only defines some basic data structures for skill storages and some actions for the LLMs to take leaving all other decisions to LLMs for deciding. To validate the effectiveness of the method, the paper apply the ASD with 3 task open sourced models as task execution model on 5 different task datasets, uniformlly use the Claude Opus 4.8 as the optimizer. From the comparison with the other 2 baseline prompt optimization methods, the paper claims the advanatges of the method and additional analysis is conducted to inteprete the feature of the optimized skills.
Critical strengths
-
The idea of the paper is well-presented and easy to follow.
-
The experiment was conducted with a 3 backbone models, providing a view of the generalization of the method.
-
Additional analysis of the optimized process and features of the generated skill bank helps to understand the features of the method.
Critical weaknesses
-
The proposed framework is largely heuristic, with no theoretical guarantees regarding its optimization outcomes. Since nearly all decisions are made autonomously by the LLM, the effectiveness of the framework heavily depends on the capabilities of the optimizer model. Consequently, the observed performance improvements may primarily stem from the optimizer's inherent capabilities, making it a potential bottleneck for the overall framework. This raises a fundamental question: Why not directly use the optimizer model to perform the downstream tasks?
-
Since the optimization process relies entirely on LLM-driven decisions, the stability and reproducibility of the resulting performance remain uncertain. Although the authors state that experiments are repeated three times and average results are reported, it is unclear whether the entire optimization pipeline is independently executed three times or whether the same optimized skill bank is evaluated three times. To assess the robustness of the proposed method, not only the mean value but also the variance should be reported.
-
The reported performance value may be problematic. The reported baseline performance raises concerns about the validity of the experimental comparisons. Specifically, the paper includes GEPA as a baseline with Qwen3-8B as the backbone model. However, the performance reported under this setting appears substantially lower than the corresponding results presented in the original GEPA paper. This discrepancy raises questions about whether the baselines have been properly implemented and evaluated under comparable experimental settings.
-
The paper claims that all methods are compared under the same optimization budget. However, the budget calculation appears to account only for rollouts on the training and validation datasets, excluding additional LLM calls involved in intermediate decision-making. Given that the proposed framework relies heavily on LLM-driven decisions throughout the optimization process, these additional calls may introduce substantial computational and financial overhead, which may be unfair to the other baseline method.
-
The paper only chooses two prompt optimization methods for comparison; more advanced skill discovery and construction methods should be included as baselines.
Questions for authors
-
What's the variance of the independent runs of the method? Can the performance gain pass the statistical significance test?
-
Why is the performance not matching the results reported in other papers? Please detail the settings and explain the cause of those contradictions
-
Can you calculate the budget based on the api calls or the token consumption, and then do the experiment and compare the performance?
-
Can you add more baseline methods for comparison?