acceptodds
Under review as a conference paper at ICLR 2027

TriEvo: Three-Agent Evolution of Policies and Optimizers

Abstract

An adaptive agent can appear to improve because search keeps a lucky component, even if its optimizer does not improve. We ask whether an optimizer revision is worth retaining: old and revised instructions learn from the same parent, their descendants freeze before a fresh gate opens, and both branches are charged. Across nine Skill source runs, gated TriEvo has a +2.53-percentage-point mean difference from a duplicate-incumbent tournament (source-mean 95% interval, 1.83–3.23). After all Policy components are reset, retaining its gated optimizer state yields a +3.02-point difference from accept-all state (1.98–4.07); a GEPA-on-Evolver adapter has the same direction. The original Prompt/Skill/Memory panel has a +7.53-point macro difference, but Memory is negative. These endpoint comparisons motivate a prospective selection-and-reuse procedure. They do not establish superiority to the adapted GEPA author, universal transfer, or a cost advantage. Task-population generalization remains outside the reported aggregate analysis.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tool-using agents, agent self-improvement, prompt optimization, skill evolution, memory evolution, validation-guided optimization, bilevel optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qtAgOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how to identify and retain useful optimizer instructions. TriEvo separates task execution, component editing, and optimizer-instruction revision into three roles. It evaluates revisions through same-parent branch comparisons, fresh gate tasks, and reuse after resetting Policy components.

Critical strengths
  1. The authors carefully control for gains from simply selecting a better Skill. Same-parent comparisons, duplicate-instruction branches, and reset-and-reuse tests make the evidence more convincing. Negative results and inconclusive comparisons with strong baselines are reported candidly.
Critical weaknesses

1.The novelty is mainly in combining existing mechanisms, and the meaning of optimizer improvement remains unclear. Reset gains could still reflect domain knowledge or better output validity rather than a better optimization strategy. GEPA performs similarly under the same selector, leaving the added value of the three-role structure unclear. 2. The organization makes the paper difficult to follow. The main text has nine sections, but the relationships among the experimental panels are unclear. The single figure does not sufficiently clarify these relationships. Repeated caveats and defensive wording interrupt the argument. I think more space should explain the core procedure and evidence.

Questions for authors

Please see the questions raised in the weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer qtAg

Our contribution is the procedure for evaluating and retaining a change to the optimizer. The original introduction did not bring this question into focus clearly enough. We have revised the explanation and organized the experiments around it.

Relation to earlier work. Promptbreeder, SkillOpt, and MetaSkill-Evolve already make parts of the update process adaptable. TriEvo evaluates a proposed instruction by comparing what the Policy learns under old and new guides from the same initial state, freezing their descendants before a fresh gate, and testing reuse after the learned Policy components are removed. The original selection and reset panels address these questions; Sections 1–2 now present them together. The reported similarity to GEPA leaves the choice of instruction author open.

Actual edit and its interpretation. The s02 revision mixes strategy and domain-related content. It broadens bootstrap support, adds an evidence check before no_change, removes a diagnostic requirement, and retains AppWorld guidance. Reset removes inherited Policy state but leaves that guidance in place. The validity counts separately record outputs before and after repair. Isolating the effects of strategy, domain knowledge, and formatting would require further interventions.

The changed guide ties old at 3/18 on the gate and at 25/45 after reset, with six successes unique to each reset condition. The case shows how the procedure handles a revision that gives no net gain. Two other sources return no-change and share endpoints; two final tasks remain missing (178/180 physical evaluations). We have not added native-method or direct-Policy comparisons.

Organization. The main text now has six sections. Method and protocol are together, as are selection and reset. Figure 2 links the original tables to their questions, populations, and editable state, and places the new case alongside them. Section 5 summarizes the case; Appendix M contains the instruction excerpt, outcomes, and execution details. The repeated scope qualifications are now consolidated.

Thank you for your response. It partially addresses my concerns, but I believe the quality of the paper still requires further improvement. I have raised my score to 2 as encouragement; however, I do not think the paper is yet suitable for publication at ICLR.

Reviewer_fNFAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces TriEvo, a evol system with a Policy that solves tasks, an Evolver that updates the Policy, and a Supervisor that revises the Evolver instruction. The main contribution is a procedure for choosing between the old and revised optimizer instructions. Both start from the same parent Policy and are tested on the same next batch of adaptation data. Fresh gate tasks are then used to decide which instruction to keep. Across nine Skill source runs, gated TriEvo beats accept-all by 3.02 pp and the duplicate-incumbent tournament by 2.53 pp. After resetting all Policy components and keeping only the selected optimizer instruction, it still beats accept-all by 3.02 pp.

Critical strengths
  1. They have a good optimizer revision test design. The old and revised Evolver instructions start from the same Policy parent, use the same adaptation batch, and are evaluated on fresh gate tasks only after both descendants are frozen.
  2. When they reset prompt, skill, memory and the rest of the policy state and keep only the optimizer instruction, this controlled experimental setup shows that whether the retained optimizer state is still useful.
Critical weaknesses
  1. The presentation is difficult to follow. I think the paper does not clearly explain what TriEvo is in abstract and introduction. I'm unclear on what "gated TriEvo" refers to and what you mean by "three-agent evolution". My understanding is the three-role evolution.
  2. The overall designed pipeline (three-agent evolution) seems very complex, but from the results, the improve is modest. Also, it's unclear how much of the improved % might be due to the randomness of the models.
Questions for authors
  1. I'm curious what does the Supervisor actually change in the Evolver instruction? More specifically, what does the Supervisor focus on when revising the Evolver’s instructions?
  2. Is there a reason why the Evolver can only modify one Policy component?
  3. In Table 3, why does Memory show almost no improvement under Supervisor revision, while Prompt and Skill both improve by more than 11 pp?
3 comments
AuthorsOpenReview ↗

Response to Reviewer fNFA

We agree that the paper needed a concrete example of what the Supervisor changes. Appendix M now provides one, and Section 2 explains the three roles through the objects they can edit.

Roles and a concrete edit. The Evolver is the optimizer in our system: it proposes edits to an external Policy component using task traces. Its guide specifies how to make those proposals. The Supervisor edits that guide using development evidence, and a fixed evaluation rule decides retention. The Policy solves tasks with its current components. Model weights stay fixed.

In s02, the Supervisor changes the support requirement for a bootstrap edit from “at least two sealed failures” to “at least two related training traces.” It adds a check of recurring problems and counterexamples before no_change, removes a candidate-read diagnostic requirement, and retains AppWorld execution guidance. The full old/new guides and diff are included in the supplementary material.

Outcome and variability. Both guides pass 3/18 on the gate, so old is retained. After reset, both pass 25/45; six tasks pass only with old and six only with new. The task outcomes differ even though the totals are equal. The other two sources return no-change and share endpoints. Two final tasks remain missing, giving 178/180 physical evaluations; Appendix M contains the full source table. Source seeds control ordering, not backend sampling, so D does not establish robustness to model randomness.

Why one component at a time? Holding the other components fixed keeps the Policy edit target fixed while we compare the optimizer guides. For reset, we restore the initial Policy state and retain only the assigned guide. The composition panel then assembles learned component endpoints without jointly optimizing them.

The original tables report gains of 11.85 pp for Prompt and 11.30 pp for Skill, compared with −0.56 pp for Memory. They also report 71.39% for PS and 68.80% for PSM. Section 4 retains these differences: the reported effects vary by component, and adding Memory did not improve PS. We cannot determine the cause of Memory's behavior from these aggregates alone.

Thanks authors taking the time to answer my questions. Not all of my concerns were addressed, especially for my 3rd question, the reason still a bit unclear to me. So I'll keep my original score.

Thank you for raising this. In Table 3, revising the Evolver improved Prompt and Skill by 11.85 and 11.30 percentage points, respectively, while Memory changed by −0.56. We checked whether the two memory banks differed in usefulness once their learned entries reached the Policy.

We tested the frozen seed-101 banks on 24 paired AppWorld tasks, one per task family. Both banks supplied learned entries on every task. Each solved 20 of 24: they solved 18 in common, and each solved two that the other missed. On this panel, the revised memory bank offered no net advantage even when its entries were retrieved.

Reviewer_wDymOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether self-evolving agents learn better optimization instructions, beyond retaining successful task-solving components. TriEvo separates Policy execution, component editing by an Evolver, and revision of the Evolver’s instructions by a fixed Supervisor. Proposed optimizer revisions are evaluated prospectively: old and revised instructions adapt from the same parent, and their frozen descendants are compared on fresh gate tasks. A reset experiment then removes learned Policy components and retains only optimizer instructions. Across nine Skill source runs, gated TriEvo improves over a duplicate-incumbent tournament by 2.53 percentage points, while gated optimizer state improves reset adaptation over accept-all state by 3.02 points. A GEPA-based author exhibits a similar retention benefit.

Critical strengths
  1. The paper addresses an important distinction in self-improvement evaluation. Better final task performance does not necessarily imply a better optimizer. The same-parent comparison, duplicate-incumbent tournament, and gate-blind accept-all control provide a thoughtful way to examine optimizer revision separately from ordinary branch selection.

  2. The reset-and-reuse experiment strengthens the central evidence. Discarding learned Policy components and retaining only optimizer instructions directly addresses inheritance of a successful task-solving artifact. The nine-source experiment, with three reset episodes averaged within each source, provides more informative evidence than evaluating only the final evolved agent.

  3. The reporting is unusually careful about interpretation and accounting. The paper distinguishes source-level uncertainty from task-population uncertainty, charges rejected branches, and reports negative Memory and composition results. It also explicitly avoids claiming superiority over the GEPA author or an efficiency advantage. The primary source-level contrasts can be reproduced from the supplied endpoint counts.

Critical weaknesses
  1. The nature of the retained improvement remains unresolved. Policy reset shows that useful information survives in the optimizer instruction, but does not establish whether that information is a reusable optimization strategy, domain knowledge, or guidance for producing valid outputs. The proposal diagnostic shows validity increasing from 57/71 to 67/71, which could contribute substantially to the improvement. The authors acknowledge these alternatives, but stronger controls and concrete learned-instruction examples are needed to distinguish them.

  2. Generalization evidence remains narrow. The primary confidence intervals condition on one shared, newly authored AppWorld-style final population and one Policy–author model configuration. They therefore do not quantify uncertainty over new task populations or model choices. The reset study establishes reuse within the evaluated setting, while the cross-domain experiments primarily transfer Policy components. Direct evidence that learned optimizer instructions improve adaptation in a different domain or with a different Policy model is still missing.

  3. The incremental benefit over existing meta-optimization needs sharper characterization. TriEvo and the GEPA-on-Evolver adapter have similar results, and the reset contrast against MetaSkill-Evolve remains unresolved. These findings support the retention procedure more clearly than a distinctive revision-authoring mechanism. Applying the same prospective gate to an otherwise unchanged meta-skill optimizer would help identify what the proposed procedure adds. A direct Policy editor with matched access to adaptive-Dev evidence would also test whether the additional optimization layer is necessary.

  4. The gate’s reliability and data requirements are insufficiently characterized. Acceptance requires only one additional success on a 30-task gate containing ten families. The paper appropriately calls this a heuristic, but does not show sensitivity to gate size, acceptance margin, or the number of revision opportunities. The main schedule also consumes 270 fresh gate tasks for 96 adaptation tasks. Understanding this trade-off matters when fresh, independently evaluated task families are scarce, as illustrated by the single outer opportunity in ALFWorld.

  5. Aggregate reporting limits independent verification of the central protocol. The endpoint tables and initial prompts are detailed, but they do not provide the joint task-by-source outcomes, learned optimizer instructions, or complete branch histories needed to examine the main mechanism. These artifacts would also enable verification of family separation, revision timing, information-access boundaries, and realized resource matching. This is particularly important because the contribution depends heavily on how evaluation and state retention are implemented.

Questions for authors
  1. Could you release the learned optimizer instructions, revision histories, task-family assignments, and joint task-by-source outcomes? Can the main contrasts also be evaluated with uncertainty over both sources and task families?

  2. How much of the reset benefit remains under a common prospective validity treatment? Could you separately ablate domain-specific content and general optimization guidance within retained instructions?

  3. What happens when the same gate is added to the native meta-skill update procedure? How does TriEvo compare with a direct Policy editor given the same adaptive-Dev evidence and resource budget?

  4. How sensitive are final and reset performance to gate size, acceptance margin, and revision frequency? Can fewer fresh gate tasks retain most of the benefit?

  5. Do retained Evolver instructions improve fresh adaptation under a different Policy model or task domain, after resetting all Policy components and matching target adaptation resources?

  6. What explains the negative Memory result and the degradation from PS to PSM? Can targeted ablations distinguish retrieval failures, conflicting content, and additional context cost?

2 comments
AuthorsOpenReview ↗

Response to Reviewer wDym

Thank you for asking us to show what the learned optimizer actually contains. We have added a concrete instruction revision and followed it through learning, selection, and reset. This connects the original gate and reuse questions to a change proposed by the Supervisor.

The instruction change. The s02 guide changes the evidence required for a bootstrap Skill edit: “at least two sealed failures” becomes “at least two related training traces.” It also adds an evidence check before no_change, removes a candidate-read diagnostic requirement, and keeps AppWorld execution guidance. This is a mixed edit, so reset alone cannot separate update strategy from domain knowledge. Appendix M gives an excerpt; the full old/new guides and diff are included in the supplementary material. These records cover D, not the original study's complete history.

Reset results and validity. Both guides pass 25/45, with six old-only and six new-only successes. The total score is unchanged. The other two sources return no-change and share endpoints. Appendix M retains all three sources and the two missing final tasks: coverage is 178/180 physical evaluations.

The branches start from the same Policy parent, use the same six adaptation tasks and search limits, and freeze before the 18-task gate opens. Both pass 3/18, retaining old. The proposed guide also enters reset as registered. Reset restores the initial Skill, clears history and behavioral state, and uses fresh sessions without Supervisor updates. Both conditions use the same output checks and repair limit. Evolver outputs meet the schema in 36/54 stages before repair and 54/54 after 18 repairs, including 49 no-change decisions. These counts describe formatting failures; they do not measure how much formatting affects performance.

The original source intervals condition on a shared final panel. We have not added joint source/family uncertainty estimates, and reused endpoints in D do not add independent replications.

Controls and gate allocation. The reported GEPA-on-Evolver differences are small and uncertain, leaving instruction-author superiority open. The suggested direct-Policy and matched native-method controls have not been added. The original protocol allocates 270 tasks to nine fresh 30-task gates, separately from the 96 adaptation tasks; D uses one 18-task gate per source. Gate-size, margin, and frequency sensitivity remain untested. We have not added optimizer transfer with matched adaptation on a new model or domain; frozen-component transfer answers a different question.

Memory. The original tables report −0.56 pp for supervised versus fixed Memory and −2.59 pp for PSM versus PS. Section 4 retains both. Retrieval, capacity, and conflicting instructions are possible explanations, but these aggregate comparisons cannot tell us which caused the decline.

Thank you for your response, it met my expectations and I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.