- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a method for agentic skill evolution through program synthesis. Exisiting methods mostly need to gather feedback to to optimize the skills. The proposed method can observe the evidence and directly optimize the skills without costly feedback gathering. The method organizes initial skills and task evidence into a program-like representation that make the task requirement and skill procedure directly comparable, and audits for missing requirements. Extensive experimental results show that the proposed method can optimize the skills without rollout-based optimization.
Critical strengths
- The presentation of this paper is clear.
- The idea of optimizing the skills without rollout is interesting and might be useful for resource constrained settings.
- The experimental results show the effectiveness of the proposed method.
Critical weaknesses
- While the idea of optimizing the skills without actual rollouts seems interesting, more types of mistakes require actually running the skills to discover. Simply auditing the static skills may be sufficient to discover some easy cases, but may not be sufficient for more complex cases.
- The types of information to be auditted seem fixed and hand crafted. It is unknown if this information is sufficient to detect the errors in the skills, or if the framework can scale to more complex settings, where more aspects of information needs to be considered in order to audit the skills.
- Simply auditing the skills and improve it, without actual feedback and stronger optimization algorithms (e.g., feedback with reinforcement learning) may limits how far the skill can improve. In some cases, its improvement upper bound may be shallow compared to more principled skill optimization approaches. Also if the audit is not accurate, the errors may accumulate over optimization iterations.
Questions for authors
Please refer to the weakness sections and try to answer the questions.