- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how to identify and retain useful optimizer instructions. TriEvo separates task execution, component editing, and optimizer-instruction revision into three roles. It evaluates revisions through same-parent branch comparisons, fresh gate tasks, and reuse after resetting Policy components.
Critical strengths
- The authors carefully control for gains from simply selecting a better Skill. Same-parent comparisons, duplicate-instruction branches, and reset-and-reuse tests make the evidence more convincing. Negative results and inconclusive comparisons with strong baselines are reported candidly.
Critical weaknesses
1.The novelty is mainly in combining existing mechanisms, and the meaning of optimizer improvement remains unclear. Reset gains could still reflect domain knowledge or better output validity rather than a better optimization strategy. GEPA performs similarly under the same selector, leaving the added value of the three-role structure unclear. 2. The organization makes the paper difficult to follow. The main text has nine sections, but the relationships among the experimental panels are unclear. The single figure does not sufficiently clarify these relationships. Repeated caveats and defensive wording interrupt the argument. I think more space should explain the core procedure and evidence.
Questions for authors
Please see the questions raised in the weaknesses above.
2 comments
Thank you for your response. It partially addresses my concerns, but I believe the quality of the paper still requires further improvement. I have raised my score to 2 as encouragement; however, I do not think the paper is yet suitable for publication at ICLR.
Response to Reviewer qtAg
Our contribution is the procedure for evaluating and retaining a change to the optimizer. The original introduction did not bring this question into focus clearly enough. We have revised the explanation and organized the experiments around it.
Relation to earlier work. Promptbreeder, SkillOpt, and MetaSkill-Evolve already make parts of the update process adaptable. TriEvo evaluates a proposed instruction by comparing what the Policy learns under old and new guides from the same initial state, freezing their descendants before a fresh gate, and testing reuse after the learned Policy components are removed. The original selection and reset panels address these questions; Sections 1–2 now present them together. The reported similarity to GEPA leaves the choice of instruction author open.
Actual edit and its interpretation. The s02 revision mixes strategy and domain-related content. It broadens bootstrap support, adds an evidence check before
no_change, removes a diagnostic requirement, and retains AppWorld guidance. Reset removes inherited Policy state but leaves that guidance in place. The validity counts separately record outputs before and after repair. Isolating the effects of strategy, domain knowledge, and formatting would require further interventions.The changed guide ties old at 3/18 on the gate and at 25/45 after reset, with six successes unique to each reset condition. The case shows how the procedure handles a revision that gives no net gain. Two other sources return no-change and share endpoints; two final tasks remain missing (178/180 physical evaluations). We have not added native-method or direct-Policy comparisons.
Organization. The main text now has six sections. Method and protocol are together, as are selection and reset. Figure 2 links the original tables to their questions, populations, and editable state, and places the new case alongside them. Section 5 summarizes the case; Appendix M contains the instruction excerpt, outcomes, and execution details. The repeated scope qualifications are now consolidated.