acceptodds
Under review as a conference paper at ICLR 2027

The Right Lesson at the Right Step: Deriving Control Updates for Self-Evolving Agents

Abstract

Self-evolving agents improve future behavior by reusing past experience, typically as global prompts, memories, or reflections. Yet these mechanisms rarely control where experience takes effect. In long tool-use workflows, the same lesson may correct one decision but distract another, making experience reuse a problem of localized control rather than memory alone. We introduce EvoCUE (Evolution through Control Updates from Evidence), a framework for learning reusable control-program updates from completed agent executions. EvoCUE represents the agent as an explicit state-machine controller, whose nodes perform model or tool calls and whose edges define where control passes next. This makes the workflow editable at precise locations, so each learned update can specify what to add, where it acts, and when it applies. From completed trajectories, EvoCUE uses residual goals and observed execution traces to propose localized instruction or skill edits. Each candidate is evaluated at the point where it would act by resuming the parent and edited controllers from the same checkpoint and comparing their final outcomes. Accepted edits are compiled with applicability rules, confirmed on held-out tasks, and inherited by later executions. We evaluate EvoCUE on long tool-use environments where learned conventions must reach the right execution step. From a minimal AppWorld controller without benchmark-specific onboarding instructions, EvoCUE learns the missing task-completion convention and substantially improves success on Test-Normal and Test-Challenge. On PAST-Bench office workflows, EvoCUE transfers organizational requirements from prior episodes to later tasks, improving task-execution quality. These results show that self-evolving agents should place experience inside the control flow, rather than only store it as text.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, self-improving agents, controller learning, agentic workflows, skill learning, continual learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JYumOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EvoCUE, a framework that improves agents by learning reusable edits to an explicit state-machine controller while keeping the model and tools fixed. It proposes edits from execution traces and residual goals, compares parent and edited continuations from shared checkpoints, and confirms updates on separate training tasks before inheritance. Its principal AppWorld result recovers an omitted task-completion convention, increasing Test-Normal/Test-Challenge success from 13.7%/6.7% to 79.2%/70.0%. A PAST-Bench pilot reports improved transfer of organizational requirements, with task-execution scores increasing from 0.608 to 0.772.

Critical strengths
  1. It designs a coherent and inspectable procedure for persistent agent improvement. Matching the environment state, execution history, controller state, and remaining budget before comparing continuations is a sensible way to remove variation originating before an edit acts. Separate whole-program confirmation and records of rejected candidates provide a useful audit trail (§§3.3–3.4).

  2. It shows substantial transfer within the AppWorld setting. The learned instruction improves hundreds of frozen test executions, with 110 improvements and no regressions on Test-Normal (§4.2). Additional runs without the historical hypotheses recover the same convention (Appendix E.2). This supports a reproducible ability to acquire and reuse missing procedural knowledge from public execution evidence.

Critical weaknesses
  1. The experiments insufficiently isolate the distinctive contribution of localized control and checkpoint testing. In 4.4, the first-entry instruction is absent from subsequent prompts by construction; the comparison therefore primarily establishes the importance of continued instruction availability. Providing the same learned text at every step also solves 14/16 AppWorld tasks, and achieves 0.767 versus 0.772 on PAST-Bench. Both successful controllers use unconditional instruction attachments without new skills. A controlled comparison against task-start evaluation is also missing. This matters because AutoGuide already selects guidance according to the current state, while Self-Harness already validates persistent harness edits and reports learning the AppWorld action-versus-question completion distinction. The paper cites these predecessors, but the additional benefit of its particular mechanism needs stronger evidence. arxiv.org

  2. Evidence for the complete learning procedure beyond the restricted AppWorld setup is weak. Recovering intentionally omitted onboarding information is a valid experiment, but the independent supporting evaluation is limited. The 16-task PAST pilot was previously inspected during development). On unseen supported families, the complete EvoCUE procedure accepts no update: 0.305->0.332 reflects rerunning the unchanged controller. The stronger 0.345>0.729 result comes from installing all distilled knowledge despite rejection by the confirmation stage (Table 2). None of the six additional domains yields an accepted update. These disclosed results leave broader generalization of the proposed selection-and-inheritance procedure insufficiently established.

Questions for authors
  1. GEPA supports updates to individual module prompts. Which controller locations could the adapted GEPA baseline modify, and were its learned instructions available at subsequent action-generation calls? This would help determine whether the comparison isolates the learning algorithm or also changes the available editing interface. arxiv.org

  2. With the proposer, public evidence, candidate space, and learning budget fixed, how much do checkpoint trials improve learning outcomes or cost over task-start trials and a learner that maintains instructions at every step? Does the advantage persist when the official AppWorld onboarding instructions are included?

  3. Why does rule fitting exclude scored, applied trials without an observed behavior change (Appendix A.3)? For the accepted edit, this changes the estimated effect from 5/8 to 5/7. Are fitted rules and acceptance decisions stable when all scored, reached trials are retained?

3 comments
AuthorsOpenReview ↗

Response to Reviewer JYum (1/2): the requested learner-fixed comparison, with and without the official onboarding

We thank the reviewer for the balanced assessment and the concrete questions. Q2 describes exactly the design we ran. All new results are in the revised PDF (new Table 2, Sections 4.2–4.6, new Appendix G).

Protocol of all new runs.

  • Settings and decision rules were registered before launch.
  • One learner is frozen for all runs.
  • Test sets are read in aggregate.
  • Every comparison is paired on one deployment of the same model, with scenario-clustered bootstrap 95% CIs.

W1 / Q2. Checkpoint vs. task-start trials, and learned locations vs. every-step instructions, with proposer, evidence, candidate space and budget fixed.

From C0C_0, one learner varies only the location or only the trial type (Table 2 f, i):

ComparisonTest-NormalTest-Challenge
Learned locations vs. every step+9.9 [4.8, 15.3]+13.8 [10.7, 16.9]
Checkpoint vs. task-start trials+6.2 [1.6, 11.3]+2.3
  • Estimator quality and cost (Appendix G.2). The test uses 24 edits whose effects were measured by full evaluation.
    • Checkpoint trials make 22/24 correct decisions and accept 0 of the 18 neutral or harmful edits.
    • Task-start trials make 20/24 correct decisions and accept 3, including a harmful one.
    • Checkpoint trials cost 85.6k tokens per pair, against 383.0k.
  • With the official onboarding included (C0′C_0').
    • Trials: only the checkpoint learner yields a controller that passes the development gate, and that controller gains +7.3 [3.1, 11.9] on Test-Challenge.
    • Locations: with ACE's playbook text fixed, learned locations beat every-step availability by up to +11.3 [5.7, 16.7].
    • Giving all 305 learner-proposed texts at every step scores 13.5 points [5.3, 21.6] below the learned controller and 10.4 points below C0′C_0' itself.
  • Beyond unconditional instructions.
    • A learner restricted to structural edits (new draft-revising nodes, and a conditional syntax-check subgraph) improves all six random training subsets on both splits (Table 2 d–e). The pooled gain over C0C_0 is +47.3 [39.2, 55.7] / +46.8 [41.5, 51.7], and the gain from C0′C_0' on Test-Challenge is +5.4 [1.3, 9.4].
    • On unseen PAST families, accepted edits carry learned non-empty rules such as "the last action succeeded" and "multiple goals remain".

Relation to AutoGuide and Self-Harness.

  • AutoGuide selects guidelines by context at inference time, and the selected guideline enters the prompt of the current call.
  • Self-Harness lets an agent edit its own harness and accepts candidate edits after regression testing.
  • EvoCUE learns where an edit acts and when it applies, as part of a persistent program. It tests each edit from the checkpoint where it acts and confirms the edit before inheritance.
  • The new results quantify both pieces. Learned locations gain +9.9 / +13.8 over every-step availability. Checkpoint trials reject the harmful edit that task-start trials accept, at 4.5× lower cost.
  • Beyond learning the completion distinction, EvoCUE exceeds the full official onboarding by 6.5 / 14.2 points and keeps improving from C0′C_0'. Section 2 positions EvoCUE against both works.
AuthorsOpenReview ↗

Response to Reviewer JYum (2/2): generalization, GEPA's editing interface, and rule fitting with all trials

W2. Evidence beyond the restricted AppWorld setup.

  • PAST, unseen families, complete procedure (Section 4.5, Table 3, Appendix G.3).
    • Every learning run now accepts updates.
    • On held-out official episodes never run during learning, the task-execution score rises by up to +0.446 [0.247, 0.633], from 0.282 to 0.727, with six tasks improved and none declined.
    • The gated procedure now reaches the level of installing all distilled knowledge (0.729).
  • τ²-bench (Section 4.6, Appendix G.4). We reran it with the official 200-step limit, with GEPA and ACE alongside. In telecom, EvoCUE accepts two updates and is the only method that improves, reaching pass¹ = pass² = 1.000.
  • AppWorld with the official instructions.
    • EvoCUE learned from C0C_0 exceeds C0C_0 with all 17 official onboarding instructions by 6.5 / 14.2 points, and ACE by +18.8 / +24.5.
    • Learning from C0′C_0' gives +7.3 [3.1, 11.9] on Test-Challenge.

Q1. GEPA's editing interface.

  • What GEPA edits. In our adaptation, GEPA optimizes the agent instruction. That instruction is included in every action-generation call, so its learned content is available at every subsequent step. AWM's memory enters the same way (Appendix D). GEPA therefore edits one instruction that acts at all locations.
  • Isolating the interface. The new experiments hold the learner fixed and vary only the interface.
    • The location interface wins by +9.9 / +13.8 (Table 2 f).
    • With ACE's text fixed, it wins by up to +11.3 (Table 2 g–h).
    • The interface is therefore a measured part of EvoCUE's advantage, in addition to the learning algorithm.

Q3. Rule fitting and trials without a behavior change.

  • Why such trials are excluded. A trial in which the edit is applied but behavior does not change carries no information about the edit's effect at that state. Including it dilutes the estimate toward zero.
  • Refit with every scored, reached trial (Appendix G.5, referenced from A.3). We refit on all 22 learning records: 164 rounds, 486 candidates and 19,619 trials, of which 1,358 are newly included.
    • The submission's accepted edit. Its estimate moves from 5/7 to 5/8, its rule stays empty, and the learned controller is unchanged.
    • Confirmation decisions. None of the 164 rounds changes which candidate reaches confirmation.
    • Accepted edits. 31 of 33 keep an identical or runtime-equivalent rule.
  • Why only checkpoint trials are affected. All newly included trials are checkpoint trials, because an edit tested from the task start almost always changes behavior.

We hope these results resolve the reviewer's questions, and we would be grateful if the reviewer would take them into account in the final assessment.

AuthorsOpenReview ↗

Follow-up on the new experiments

We thank the reviewer again for the constructive review. Our replies above report the learner-fixed experiments requested in Q2: learned locations are 9.9 / 13.8 points better than every-step availability, and checkpoint trials accept no neutral or harmful edit. We would be glad to hear whether these results address the reviewer's questions, and we are happy to answer anything that remains unclear.

Reviewer_x5kJOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that a self-evolving agent needs to learn not just what lesson to keep from a past run but where in its control flow that lesson should fire. It models the agent as a state machine, proposes edits attached to specific edges, and tests each one by rewinding a past episode to just before the edit would act and running the old and new controllers forward from that shared prefix. Surviving edits get an applicability rule, a confirmation pass on held-out tasks, and then become part of the controller. The main result is AppWorld going from 13.7% to 79.2% while four adapted baselines move by less than 5 points.

Critical strengths
  • Figure 3 is the best thing in the paper. Showing every candidate the learner ever tried, including the four that went nowhere, with the actual paired outcomes, is the kind of thing missing from most papers. The whole learning history fits in six rows and the reader can see exactly what happened.
Critical weaknesses
  • The paper's own ablation contradicts its thesis. Section 4.4 is supposed to show that location matters. Initial controller solves 4 of 16, the instruction at the first entry solves 5, at the recurring entry solves 14. However, putting the same text into the agent's prompt at every step also solves 14. On PAST the same thing happens, i.e., the learned text at every step gets 0.767 against the learned program's 0.772, and on unseen families 0.725 against 0.729. So across both benchmarks, the only configuration it beats is first-step-only, and that's a straw man as no memory or context method I know of injects a lesson once and then drops it. The honest reading of Section 4.4 is that the lesson needs to be present at the completion decision, which "every step" also achieves, and the paper's contribution isn't needed to get there. That said, targeting just the crucial step might save tokens but the authors do not quantify it.

Even worse, the gap it does beat is an artifact of the authors own controller. The paper states plainly that their controller assembles a fresh prompt per node call, so a first-entry instruction never reaches the later decision. In a conventional agent with a persistent system prompt, the instruction is there at every step by construction. So the 5-versus-14 motivation in Figure 1b is a property of this architecture, not a limitation of the methods being compared against.

  • I have significant concerns about the main result. C0 is the AppWorld controller with the benchmark's onboarding instructions deliberately stripped out. The thing EvoCUE learns is that onboarding convention, almost verbatim, e.g., the question/action distinction and leaving the answer as None. So the 65.5-point gain is the size of the removal, recovered. That's a legitimate experiment, but it has one obvious missing baseline, what does C0 plus the official onboarding instructions score? Without it I can't tell whether 79.2 means the learner recovered what was taken away, fell short of it, or somehow exceeded it. Given that GEPA with official prompts reportedly sits at 54.2 on the same model, 79.2 on Test-Normal and 70.0 on Test-Challenge are high enough that I want them placed against the published leaderboard before I believe what they imply.

  • This point is also proven by the fact that the six further domains in Table 4 are a null result, that the paper unfortunately treats as a footnote. On τ2-bench, EvoClawBench, MBPP, HumanEval, MATH, and HotpotQA, EvoCUE accepts no update at all and for the rest the change is minimal (even negative!). Section 4.6 explains this away in two lines as the initial programs already scoring high or the tasks being short, but that explanation doesn’t cover τ2-bench, where the initial controller solves 13 of 16 and the four tasks it misses are exactly the sort of multi-step tool interaction the method is pitched at. What Table 4 actually shows is that outside the one benchmark the authors handicapped on purpose, the method has produced zero accepted edits anywhere. Right now the paper picks the most flattering artificial result, and the rest of the results are buried in an appendix table rather than discussed as a limitation.

Questions for authors

n/a

4 comments
AuthorsOpenReview ↗

Response to Reviewer x5kJ (1/2): learned locations beat every-step availability, including a persistent system prompt

We thank the reviewer for the sharp critique. We ran exactly the comparisons it calls for. All new results are in the revised PDF (new Table 2, Sections 4.2–4.6, new Appendix G).

Protocol of all new runs.

  • Settings and decision rules were registered before launch.
  • One learner is frozen for all runs.
  • Test sets are read in aggregate.
  • Every comparison is paired on one deployment of the same model, with scenario-clustered bootstrap 95% CIs.

"Every step also gets 14/16; the only configuration beaten is first-step-only."

At test scale, with the learner fixed, learned locations clearly beat every-step availability:

ComparisonTest-NormalTest-Challenge
Learned locations vs. every accepted edit forced to every step (Table 2 f)+9.9 [4.8, 15.3]+13.8 [10.7, 16.9]
Same ACE playbook text: learned locations vs. every step (Table 2 g–h)up to +11.3 [5.7, 16.7]up to +10.9 [6.6, 15.3]
  • A persistent system prompt, tested directly.
    • Two runs put content into every step, which is exactly the persistent-system-prompt setting: ACE's playbook, and all 305 instruction texts our learner ever proposed.
    • Learned locations win in all four ACE comparisons.
    • The learned controllers beat the 305-text prompt by +11.7 [5.8, 18.0] and +13.5 [5.3, 21.6].
    • From C0′C_0', the persistent prompt falls 10.4 points below the starting controller itself.
  • Mechanism. Every-step availability doubles execution failures: 153.5 vs. 77.5 per Test-Challenge run with the same ACE text, and 24.5 vs. 7.5 with the 305 texts. Content that helps the completion decision disrupts code generation at other steps; learned locations keep the help without the disruption.
  • Tokens, as requested. The learned controllers use 176k and 172k tokens per task. Every-step availability uses 290k and 355k, so the learned controllers need 0.48–0.61× the tokens, and the cost of every-step availability grows with each lesson added.
  • Architecture. The advantage holds against a persistent prompt, so it is a property of placement, not of our prompt-per-call design. The 14/16 tie in the submission concerned a single lesson on 16 development tasks. As learned content accumulates, every-step availability is where accuracy is lost.
AuthorsOpenReview ↗

Response to Reviewer x5kJ (2/2): EvoCUE exceeds the full official onboarding; τ²-bench and unseen PAST families

"C0 + official onboarding is missing; place 79.2 against the leaderboard."

Controller (same deployment)Test-NormalTest-Challenge
C0C_0 + all 17 official onboarding instructions68.557.9
ACE, trained on all 90 training tasks56.247.6
EvoCUE learned from C0C_0 (the submission's controller)75.072.1
  • The learner exceeds what was removed. It recovers the removed onboarding and exceeds the full official onboarding by 6.5 and 14.2 points (Table 2 a).
  • Learning continues beyond the omitted rule. C0′C_0' already contains the official completion instructions.
    • From C0′C_0', EvoCUE raises Test-Challenge from 64.3 to 71.6 (+7.3 [3.1, 11.9]) and cuts execution failures by 61%.
    • ACE from the same start changes Test-Challenge by −9.2 [−14.6, −4.0].
    • A learner restricted to structural edits adds +5.4 [1.3, 9.4] from C0′C_0' by inserting new nodes (Table 2 c, e).
  • Leaderboard context.
    • DARC reports GEPA at 54.2 / 33.6 and ACE at 54.8 / 35.7 with this model. Our ACE reaches 56.2 / 47.6, and C0C_0 with the full onboarding reaches 68.5 / 57.9.
    • With only the official completion instructions, C0′C_0' already scores 73.2 / 70.0 (Table 1). These levels therefore reflect this model with a correct completion protocol under our controller.
    • EvoCUE reaches these levels by learning, and it goes further when started from C0′C_0'.
  • Data-efficient, reliable learning. With a third of the training data, any random 32-task subset learns the convention without historical hypotheses (up to 80.4 / 77.7) and beats ACE trained on all 90 tasks. With all 90 tasks, EvoCUE leads ACE by +17.0 / +12.8.

"Six further domains are a null result; τ²-bench in particular."

  • τ²-bench rerun.
    • Setup: the official 200-step limit (the pilot used 100, at which telecom tasks often reached the limit), the strengthened proposer, and GEPA and ACE alongside. Each test task runs twice.
    • In telecom, EvoCUE accepts two updates and is the only method that improves, reaching pass¹ = pass² = 1.000 from 0.975. GEPA and ACE each change it by −0.013.
    • In airline and retail, C0C_0 already passes 85–91% of the tasks.
  • Unseen PAST families, complete procedure.
    • Every learning run now accepts updates.
    • On held-out official episodes, the task-execution score rises by up to +0.446 [0.247, 0.633], from 0.282 to 0.727, with six tasks improved and none declined.
    • The accepted updates carry each family's standard operating procedure.
  • Revision. Section 4.6 now discusses the τ² rerun in the main text, and Appendix G.4 reports all three domains with GEPA and ACE.

We hope the new evidence answers the reviewer's concerns, and we would be grateful if the reviewer would reconsider the assessment in light of it.

Thank you for your reply. After reviewing your reply I am raising my score to 2. I still have many doubts about the usefulness and general applicability of your approach but you partially addresses some of my concerns by showing that you improve somehow even when the onboarding instructions are included.

AuthorsOpenReview ↗

Thank you for the updated assessment

We thank the reviewer for reading our responses and for raising the score. The comparison with the full official onboarding and the tests against persistent prompts have been included in the revision. We would also like to briefly restate the main contribution of the paper.

Existing self-evolving agents mainly decide what to keep from past experience, and the kept content is then given to the agent in a fixed way, usually through the system prompt. EvoCUE also learns where in the control flow each lesson is used and when it applies. Each candidate edit is tested from the checkpoint where it takes effect, before later tasks inherit it. With the same learner, learned locations are 9.9 and 13.8 points better than giving every learned edit at every step. Checkpoint trials accepted none of the neutral or harmful edits, while task-start trials accepted three.

As the reviewer mentioned, EvoCUE still improves when the onboarding instructions are included. The controller learned from C0C_0 is better than C0C_0 with all 17 official instructions (75.0 / 72.1 vs. 68.5 / 57.9). Starting from C0′C_0', which already has the official completion instructions, EvoCUE further improves Test-Challenge by 7.3 points, while ACE from the same start drops by 9.2 points. So EvoCUE can still improve an agent that is already well set up.

The same learner also improves PAST-Bench office tasks, including task families not used during development, and the telecom domain of τ²-bench. Besides instructions, it also learns new graph nodes and conditional rules. Since a standard tool-use agent loop already has a first and a repeated model call, it can be represented by our controller directly.

We are eager to address the remaining doubts. If the reviewer could share which aspects of usefulness or applicability remain most uncertain, we will answer with the evidence we have and incorporate every clarification into the final version. Thank you again for the constructive exchange.

Reviewer_3pxeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces EvoCUE, a framework for learning reusable updates to an agent's control program from completed executions while keeping the model and tools fixed. It represents the agent as a state-machine controller and proposes instruction or skill edits that specify content, location, and scope. Candidates are evaluated by resuming the parent and edited controllers from the same checkpoint, followed by confirmation on separate training tasks before inheritance. Starting from an AppWorld controller without official onboarding instructions, EvoCUE learns a task-completion convention and improves Test-Normal and Test-Challenge success from 13.7% and 6.7% to 79.2% and 70.0%, respectively. On the PAST-Bench pilot, task-execution score increases from 0.608 to 0.772. The paper also studies instruction placement and reports additional domains in which no update was accepted.

Critical strengths

The paper poses a meaningful research question: where should past experience enter an agent's execution process? Distinguishing between storing a lesson and making it available at a relevant decision provides a useful perspective on experience reuse in multi-step tasks.

Critical weaknesses

W1. The empirical validation is limited: successful adaptation is concentrated in narrow settings, with weak evidence of generalization.

  • AppWorld gains mainly recover a completion convention already provided in the omitted official onboarding: action tasks should leave the answer unset. Evidence of broader behavioral improvements is limited.
  • The positive PAST-Bench pilot uses only 16 evaluation tasks, which were inspected during development, limiting evidence of generalization to independent tasks.
  • On unseen PAST families, the full procedure accepts no update. Gains from directly installing distilled knowledge do not demonstrate successful adaptation by the complete method.
  • No update is accepted in any of the six additional domains, leaving cross-domain adaptation unvalidated.

W2. The experimental design does not sufficiently isolate the core mechanisms, and the results fall short of supporting the paper's main claims about learned placement and control-program self-evolution.

  • Recurring-entry and every-step injection both solve 14/16 AppWorld tasks and perform similarly on PAST-Bench (0.772 vs. 0.767). The additional value of selective placement remains unclear.
  • There is no matched-data, matched-budget comparison between checkpoint and task-start evaluation, leaving the contribution of paired trials unestablished.
  • EvoCUE uses 32 training tasks while baselines use 90. Subset selection is insufficiently justified, and matched-pool comparisons are missing.
  • Successful updates add instructions with empty rules, without new skills or graph changes, and evaluation freezes the learned programs. This supports offline instruction adaptation more directly than broader control-program self-evolution.
Questions for authors
  1. Given that providing the same learned text at every step matches or nearly matches the learned program, what evidence shows an additional benefit from learning where the text acts? Does any controlled comparison show that a lesson harms unrelated decisions and that selective placement avoids this harm?

  2. Why was AppWorld chosen as the main evaluation environment, and why does the initial controller omit the official onboarding instructions? Does EvoCUE still improve a controller initialized with the full official instructions, and how much of the reported gain reflects improvements beyond completion-protocol repair?

  3. How were the original 16 diagnostic tasks and the additional 16 tasks selected, and did selection depend on observed execution outcomes? Why does EvoCUE use this 32-task pool while the baselines use all 90 training tasks, and do prior trajectories or historical hypotheses involve tasks outside that pool? Are there comparisons using the same training pool or multiple independently sampled subsets?

  4. How does the proposed form of “self-evolution” differ from offline workflow or prompt optimization, given that the same update procedure appears applicable to an ordinary agent workflow? Which mechanisms specifically address interactive agents, and what evidence supports adaptation beyond instruction and entry-point updates?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3pxe (1/3): learned locations and checkpoint trials, isolated with the learner fixed

We thank the reviewer for the precise questions. Each one maps to a new controlled experiment, now in the revised PDF (new Table 2, Sections 4.2–4.6, new Appendix G).

Protocol of all new runs.

  • Settings, predictions and decision rules were registered before launch.
  • One learner is frozen for all runs and uses no warm-start hypotheses.
  • Selection uses training and development data only; AppWorld test sets are read in aggregate.
  • Every comparison is paired on one deployment of the same model (deepseek-v4-flash), with scenario-clustered bootstrap 95% CIs.

Q1 / W2. What does learning where a lesson acts add over every-step availability? Does a lesson harm unrelated decisions?

Yes. Every-step availability harms unrelated decisions, and learned locations avoid this harm. Three controlled comparisons support this (Table 2 f–h, Section 4.4):

Same learner, data and budgetTest-NormalTest-Challenge
(f) Learned locations vs. every accepted edit forced to every step75.6 vs. 65.7, +9.9 [4.8, 15.3]70.4 vs. 56.6, +13.8 [10.7, 16.9]
(g–h) ACE's playbook text kept verbatim; EvoCUE learns only where each block acts, vs. ACE at every stepup to +11.3 [5.7, 16.7]up to +10.9 [6.6, 15.3]
  • Identical text, placement only. Learned placement wins in all four line × split comparisons. It also halves execution failures, from 153.5 to 77.5 per Test-Challenge run.
  • Everything at every step. We gave all 305 instruction texts our learner ever proposed at every step, on the 115 training and development tasks.
    • The learned controllers score +11.7 [5.8, 18.0] and +13.5 [5.3, 21.6] higher, at 48–61% of the tokens.
    • From C0′C_0', every-step availability falls 10.4 points below the starting controller itself. The added failures are execution failures (24.5 vs. 7.5 per run).

Content that helps one decision disrupts others when it is present everywhere; learned locations keep the help without the disruption. The 14/16 tie in the submission concerned a single lesson on 16 development tasks. With the full stream of learned edits at test scale, the difference is large and significant.

W2. Checkpoint vs. task-start trials, with matched data and budget (Table 2 i, Appendix G.2).

On 24 candidate edits whose effects were measured by full evaluation, each estimator receives the same 16 pairs per edit:

Checkpoint trialsTask-start trials
Correct accept/reject decisions22/2420/24
Neutral or harmful edits accepted0/183/18, incl. a harmful one
Tokens per pair85.6k383.0k
  • End to end from C0C_0. Checkpoint trials yield better controllers: +6.2 [1.6, 11.3] on Test-Normal and +2.3 on Test-Challenge.
  • From C0′C_0', with the official completion instructions included. Only checkpoint trials produce a controller that passes the development gate, and that controller gains +7.3 [3.1, 11.9] on Test-Challenge. Task-start trials with the same seeds and budget produce none.
AuthorsOpenReview ↗

Response to Reviewer 3pxe (2/3): beyond the official onboarding; task selection and independent subsets

Q2 / W1. Why AppWorld, and does EvoCUE still improve a controller that has the official instructions?

  • Why AppWorld. AppWorld is a long-horizon tool-use benchmark with an official checker and 585 hidden test tasks, so test gains are measured at scale.
  • Why omit the onboarding. The omitted onboarding gives a known learning target, which lets us verify what the learner recovers.
  • Full official onboarding as a baseline (Table 2 a–b). We added all 17 official onboarding instructions to C0C_0:
Controller (same deployment)Test-NormalTest-Challenge
C0C_0 + all 17 official instructions68.557.9
ACE, trained on all 90 training tasks56.247.6
EvoCUE learned from C0C_0 (the submission's controller)75.072.1

EvoCUE exceeds the full hand-written onboarding by 6.5 and 14.2 points. It exceeds ACE by +18.8 [10.4, 26.2] and +24.5 [20.4, 28.7].

  • Gains beyond completion-protocol repair (Table 2 c, e). C0′C_0' already contains the official completion instructions.
    • Starting from C0′C_0', EvoCUE raises Test-Challenge from 64.3 to 71.6 (+7.3 [3.1, 11.9]) and Test-Normal from 72.0 to 75.3.
    • It cuts execution failures by 61% (86.5 → 34 per run).
    • ACE started from the same C0′C_0' changes Test-Challenge by −9.2 [−14.6, −4.0].
    • A learner restricted to structural edits adds +5.4 [1.3, 9.4] on Test-Challenge from C0′C_0' (Part 3).

Q3. The 32-task pool, its selection, and independent subsets.

Learning from 32 tasks shows EvoCUE's data efficiency: with about a third of the training data, it outperforms every baseline trained on all 90 tasks. New runs show that this holds for any random 32 tasks and for the full pool. None of these runs uses historical hypotheses (Section 4.3, Appendix G.2):

Training dataTest-NormalTest-Challenge
Random 32-task subset 167.964.5
Random 32-task subset 280.477.7
Random 32-task subset 378.669.1
All 90 training tasks73.260.4
ACE, all 90 training tasks56.247.6
  • Any 32 tasks suffice. Every random subset learns the completion convention and beats ACE trained on all 90 tasks on both splits, by up to +24.2 and +30.1 points. The six structural-edit lines (Part 3) also learn from random 32-task subsets, and all six improve.
  • Matched pool. With all 90 training tasks, EvoCUE leads ACE by +17.0 and +12.8 points.
  • Original selection. All 32 tasks come from the training tasks. No development or test outcome is involved.
    • First 16 (diagnostic subset). Drawn by a seeded shuffle within C0C_0's success and failure strata on training tasks, among tasks that C0C_0 finishes within 8 minutes. Each 8-task half has two successes and six failures, so that discovery and confirmation both contain failures to learn from.
    • Further 16. Added by the same seeded shuffle without looking at any score.
  • Prior trajectories and historical hypotheses. No trajectory or hypothesis involves a task outside the 32-task pool.
    • The warm-start trajectories are C0C_0 runs on tasks of the pool.
    • The eight hypotheses come from earlier runs on eight of the diagnostic tasks.
AuthorsOpenReview ↗

Response to Reviewer 3pxe (3/3): graph edits, interactive mechanisms, and generalization beyond AppWorld

Q4 / W2. How does EvoCUE differ from offline prompt or workflow optimization, and does it adapt beyond instructions?

  • The edit object. An EvoCUE edit specifies its content, a location in the control graph, and an applicability rule. It is tested where it acts and confirmed before inheritance.
  • Mechanisms specific to interactive agents. Both mechanisms produce measured gains: +9.9 / +13.8 for locations, and 0 vs. 3 false accepts for checkpoint trials.
    • Checkpoint trials restore the environment state, execution history, controller state and remaining budget at the decision where an edit acts. They then run both controllers forward in the live environment. An offline workflow has no intermediate interactive state to resume from.
    • Locations in a recurring control loop let an edit act each time the loop returns to a decision, such as before committing.
  • Prompt optimizers use the every-step interface. In our adaptation, GEPA's learned instruction and AWM's memory enter every model call (Appendix D). The placement experiments show that this interface loses accuracy.
  • Graph edits: a new experiment. A skill edit inserts operator nodes, a guarded entry edge and a return edge. We disabled instruction edits, so every accepted edit must insert nodes, and ran six random 32-task subsets from C0C_0 (Table 2 d, Appendix G.2):
Test-NormalTest-Challenge
C0C_010.76.7
Structural edits only, pooled gain+47.3 [39.2, 55.7]+46.8 [41.5, 51.7]
Best line67.963.3
  • All six lines improve both splits, and every CI is above zero.
  • The accepted edits insert nodes between pre-commit and commit that revise the drafted code before it executes. One line learns a conditional subgraph that checks the draft's syntax and rewrites the draft only when the check fails.
  • From C0′C_0', structural edits raise Test-Challenge from 65.7 to 71.1 (+5.4 [1.3, 9.4]) and Test-Normal from 75.3 to 80.4. They cut execution failures by about 40%, through nodes that pass the access token to later API calls, read every file of a directory, and look up existing resources before creating new ones.
  • Learned applicability rules. On unseen PAST families, accepted edits carry non-empty rules such as "the last action succeeded" and "multiple goals remain".
  • Frozen evaluation. Learning happens across tasks. Freezing the program at test time measures what was learned, which is the standard protocol for persistent improvement.

W1. Generalization beyond AppWorld.

  • PAST, unseen families, complete procedure (Section 4.5, Table 3, Appendix G.3).
    • Every learning run now accepts updates.
    • On held-out official episodes never run during learning, the task-execution score rises by up to +0.446 [0.247, 0.633], from 0.282 to 0.727, with six tasks improved and none declined.
    • The gated procedure now reaches the level of installing all distilled knowledge (0.729).
    • The accepted updates carry each family's standard operating procedure and rules such as finding the exact prior record before reusing it.
  • τ²-bench (Section 4.6, Appendix G.4). We reran it with the official 200-step limit, with GEPA and ACE alongside. In telecom, EvoCUE accepts two updates and is the only method that improves, reaching pass¹ = pass² = 1.000 from 0.975.
  • PAST pilot. The pilot uses all 16 evaluation episodes that the official generator provides for those families. The unseen-family result above confirms the transfer on families never used during development.

We hope these results resolve the reviewer's concerns, and we would be grateful if the reviewer would take them into account in the final assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: new controlled experiments and where each concern is resolved in the revised PDF

We thank all reviewers for detailed reviews. The reviews converge on four questions, and new controlled experiments answer each of them. All results are in the revised PDF: a new Table 2 collects the controlled comparisons, Sections 4.2–4.6 discuss them, and a new Appendix G gives the protocol, intervals and details. The settings and decision rules of all new runs were fixed before launch. One fixed learner is used throughout, test sets are read only in aggregate, and every comparison is paired on one deployment of the same model, with 95% scenario-bootstrap intervals.

  1. Where a lesson acts matters. With the learner fixed, learned locations outperform every-step availability of the same edits by +9.9 [4.8, 15.3] and +13.8 [10.7, 16.9] points on Test-Normal and Test-Challenge, at 0.48–0.61× the tokens.
  2. Checkpoint trials improve learning. Checkpoint trials accept no neutral or harmful edit, while task-start trials accept three at 4.5× the cost per pair. End to end, they yield better controllers (+6.2 [1.6, 11.3] on Test-Normal).
  3. EvoCUE goes beyond the omitted onboarding. It exceeds C0C_0 with all 17 official onboarding instructions by 6.5 and 14.2 points. From the official completion instructions it adds +7.3 [3.1, 11.9] on Test-Challenge. It learns from a third of the baselines' training data, and any random third suffices.
  4. The complete procedure adapts beyond AppWorld. On unseen PAST-Bench families, every learning run accepts updates, raising the task-execution score up to 0.282 → 0.727. On τ²-bench telecom, EvoCUE is the only method that improves, reaching pass¹ = pass² = 1.000.
Concern (reviewers)New evidenceRevised PDF
Location adds nothing over every-step availability (3pxe, x5kJ, JYum)+9.9 / +13.8 with the learner fixed; with ACE's text fixed, learned locations win all four comparisons (up to +11.3); all 305 proposed texts at every step score 11.7 / 13.5 points lower§4.4, Table 2 f–h, App. G.2
First entry is a straw man; persistent system prompt (x5kJ)Every-step availability is the persistent-prompt setting, and it loses in every comparison above§4.4, Table 2 f–h
Token cost (x5kJ)Learned locations need 0.48–0.61× the tokens§4.4, App. G.2
Checkpoint vs. task-start trials (3pxe, JYum)False accepts 0 vs. 3 at 4.5× lower cost; end to end +6.2 / +2.3; from C0′C_0' only checkpoint trials pass the development gate§4.4, Table 2 i, App. G.2
C0C_0 + official onboarding (x5kJ, 3pxe, JYum)68.5 / 57.9, against 75.0 / 72.1 for EvoCUE learned from C0C_0§4.2, Table 2 a
Gain is the omitted convention (3pxe, x5kJ)From C0′C_0': +7.3 on Test-Challenge and 61% fewer execution failures; ACE from C0′C_0': −9.2§4.2, Table 2 c
32 vs. 90 training tasks (3pxe)Every random 32-task subset beats ACE trained on all 90 tasks; with all 90 tasks EvoCUE leads ACE by +17.0 / +12.8§4.3, App. A.1, G.2
Only unconditional instructions, no graph changes (3pxe, JYum)Structural edits alone improve all six random subsets, +47.3 / +46.8 pooled, and add +5.4 from C0′C_0'; learned non-empty rules on PAST§4.3, Table 2 d–e
Unseen PAST families accept nothing (3pxe, JYum)Every run accepts updates; 0.282 → 0.727, six tasks improved, none declined§4.5, Table 3, App. G.3
Six further domains (3pxe, x5kJ, JYum)τ²-bench telecom: EvoCUE is the only method that improves, to 1.000§4.6, App. G.4
GEPA's editing interface (JYum)GEPA's instruction enters every model call; the learner-fixed comparisons isolate the interfaceApp. D, Table 2 f–h
Rule fitting excludes no-change trials (JYum)Refit with all trials: no confirmation decision changes, and the learned controller is unchangedApp. A.3, G.5

Each review receives a detailed point-by-point response below it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.