acceptodds
Under review as a conference paper at ICLR 2027

AutoPEER: Issue-Guided Prompt Evolution for Long-Horizon Agents

Abstract

Large language models (LLMs) increasingly power agents that solve complex tasks through multi-step tool interactions. Existing prompt optimization frameworks learn from execution feedback, but give limited attention to credit assignment in long-horizon tasks. Effective prompt evolution requires identifying which decisions in a trajectory should be improved and updating the prompt accordingly. We introduce AutoPEER, a framework that combines Issue-level credit assignment with progressive behavioral verification. AutoPEER locates decisions that need better guidance and records them as Issues to guide targeted prompt patches. Progressive verification tests whether patches change the intended actions, improve complete-task performance, and generalize across tasks after merging. A multidimensional reward rubric selects prompts based on task success and execution quality. Experiments on AppWorld show that AutoPEER improves both task completion and interaction efficiency across models, achieving the highest completion on each evaluated split. These improvements are observed with and without training reference solutions. Notably, prompts learned with one model improve another without further optimization. Additional evaluation on the telecom domain of τ²-Bench demonstrates gains in a different tool environment. Ablations show that targeted repair and progressive verification help agents solve more tasks, while multidimensional rewards encourage more efficient execution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Agents, Prompt Optimization, Self-Improving Agents, Long-Horizon Decision Making, Credit Assignment, Textual Gradients

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_ciySOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AutoPEER, an automatic prompt optimization framework designed for LLM agents operating in long-horizon, stateful tool environments. Rather than rewriting entire prompts or endlessly accumulating experience in a sprawling playbook, AutoPEER couples Issue-level temporal credit assignment with three-stage progressive behavioral verification:

Critical strengths

A central bottleneck in agentic prompt evolution is that trajectory-level feedback is noisy and full-episode validation is computationally expensive, while unconstrained playbook accumulation (like ACE) leads to severe context bloat. AutoPEER addresses both challenges through a cohesive pipeline. By forcing the optimizer to ground prompt edits in specific trajectory timestamps (replay targets) and verifying them via Merge and Rolling across recent batches, the framework effectively filters out spurious local edits (with Full-Episode Validation rejecting ~42% of candidates that pass Fast Replay, and Rolling blocking destructive patch merges) while adding 75.7%–94.5% fewer characters than ACE. This provides a highly practical, traceable paradigm for updating agent instructions without weight updates.

Critical weaknesses
  1. Although Fast Replay is presented as a principal component of the three-stage verification pipeline, the ablation results in Tables 4 and 6 indicate that excluding this module (w/o Fast Replay) achieves identical final TGC on AppWorld Normal (86.31%) and Challenge (76.26%). Furthermore, its removal decreases the number of candidates forwarded to Full-Episode Validation by only 11.8% (from 127 to 112), corresponding to a savings of merely 45 validation episodes over 30 generations. Notably, Fast Replay resets the environment and reruns recorded code or tool prefixes up to turn tt, adding simulator overhead and requiring deterministic, replayable states. The acceptance rule (“a patch passes if the action changes and executes successfully”) may also reject valid patches whose effects appear later (e.g., at turn t+kt+k) or that prevent later errors without changing the action at turn tt.

  2. The initial baseline prompts were overly detailed and lacked analysis of the stability of the training order and ground truth feedback. As seen in Appendix E.1, the initial system prompts were not simple general instructions, but were manually divided into eight clear sections (PE-01 to PE-08), with very specific AppWorld rules pre-written (such as checking the password in apis.supervisor, using page_index to navigate to the bottom of the pagination interface, treating the file system as an application rather than a system file, and specifying when to pass the answer parameter to complete_task). This is equivalent to pre-setting a well-structured framework for local patching. Whether this local patching method would still be effective with a very rudimentary initial prompt remains to be verified. In addition, the training tasks were only run once in a fixed order, without testing the variance caused by shuffling the batch order. Moreover, the effect of providing or not providing a reference ground truth (GT) was unstable. For example, on the DeepSeek Challenge testset, the task completion rate without a ground truth (78.18%) was actually higher than that with a ground truth (76.26%).

Questions for authors
  1. Since AutoPEER visits each training scenario once in a fixed sequential order and only checks regressions on the latest K=3K=3 batches (up to 9 tasks), how sensitive is the final evolved prompt to the ordering of the training batches? Have you observed cases where an update accepted at generation gg causes regressions on tasks from generations ≤g−3\le g - 3, and how does performance/cost scale if KK is increased or sampled via stratified replay across earlier scenarios?
  2. In Table 6, Fast Replay filters out only 16 out of 128 valid candidates (~12.5%) before Full-Episode Validation, and w/o Fast Replay achieves identical TGC on both Normal and Challenge (Table 4) (though SGC on Challenge is 56.83% vs. 61.87%). What is the exact wall-clock time and token/execution cost of running Fast Replay (3 paired trials ×\times up to 3 replay targets per Issue) compared to the 45 full-validation episodes it saves?
  3. On AppWorld Challenge with DeepSeek-V4-Flash, AutoPEER without GT achieves higher TGC (78.18%) than AutoPEER+GT (76.26%), despite AutoPEER+GT achieving higher SGC (61.87% vs. 59.71%) and higher Normal TGC/SGC. Could you elaborate on what types of patches GT induced that lowered task completion on Challenge? Does exposure to reference solution code cause the Patch Generator to propose overly specific rules that overfit to training scenario structures?
1 comment

Thank you for your feedback. We have responded to your questions above and would appreciate hearing your thoughts soon.

Reviewer_tsu3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies prompt optimization for long-horizon tool-using agents. The method extracts issues, supporting evidence, and associated decision points from complete execution trajectories, then generates local prompt patches. Candidates are retained through local action replay, full-task validation, and rolling validation after merging. Candidate selection considers both task success and execution efficiency. The authors evaluate two Workers on AppWorld's Normal and Challenge test sets and on Telecom Solo, with additional cross-Worker prompt transfer and component ablations. The main finding is that targeted diagnosis and layered validation can improve task completion on the evaluated benchmarks while limiting prompt growth.

Critical strengths
  1. The method addresses a practical difficulty in long-trajectory optimization through an inspectable workflow. Issues are linked to trajectory evidence and specific decision points before local patches are proposed and assessed through replay, full-task validation, and rolling validation. This connects the rationale for a modification to its execution outcome and has practical value for prompt maintenance.

  2. The overall improvements have reasonably clear experimental support. Section 5 reports that DeepSeek's Challenge TGC increases from 51.32% to 78.18% without GT, while Qwen improves from 50.0% to 82.5% on Telecom Solo. Evaluation across two Workers and cross-Worker transfer makes the evidence broader than a single model or task set.

Critical weaknesses
  1. The end-to-end efficiency benefit might remain unclear. Table 6 reduces candidate-validation episodes from 381 to 336, but excludes discovery, Parent refresh, and Rolling; replay and prefix reconstruction also incur costs. Although Appendix D.3 adds resource statistics, total tokens, tool calls, and wall-clock time on identical or paired patch sets are still needed to determine whether Fast Replay provides a net saving.

  2. I think stability across independent optimization runs has not been sufficiently established. The main results do not adequately show variation in the final prompts obtained from repeated optimization. The three Telecom evaluations of a fixed prompt measure execution variability only. Please report the number of independent optimization runs and their dispersion. Otherwise, it remains difficult to assess whether the large gains are a consistent property of the optimization method.

Questions for authors

1&2 See weaknesses. 3. Does Fast Replay automatically verify that the proposed repair actually resolves the diagnosed issue, or does it only check whether the modified action is valid? If the latter, how can the method support causal diagnosis and credit assignment?

Reviewer_ng4fOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a prompt optimization framework for long-horizon agents, including credit assignment, patch generation, behavior verification, and rubric-based evaluation. It first identifies problems in execution trajectories, and then generates targeted prompt patches. The generated patches are then validated through local replay, full-episode evaluation, and rolling tests. Experiments on 2 benchmarks demonstrate improvements in task success and execution efficiency over existing prompt optimization methods.

Critical strengths

AutoPEER achieves notable improvements over baselines across two benchmarks and multiple agent models.

Critical weaknesses
  • The proposed framework largely combines established ideas from trajectory-based reflection, localized prompt editing, and execution-based validation, many of which have been explored in prior work. While the specific integration seems effective, the individual components appear relatively straightforward, making the methodological contribution somewhat incremental.
  • Fast Replay does not improve final task completion in the ablation, and only reduces the number of candidates entering full-episode validation from 127 to 112. While this supports its role as a screening mechanism, the reduction is relatively modest, and the paper does not isolate whether the savings from avoided full episodes outweigh the additional replay overhead.
Questions for authors

Could the authors compare the proposed method against GEPA or ACE augmented with the same multidimensional reward and validation protocol? Such comparisons would help clarify whether the improvements primarily arise from the proposed issue-guided optimization or from differences in the overall optimization setup.

Reviewer_tpMrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AutoPEER, a prompt optimization framework for long-horizon LLM agents. The method uses an LLM to identify problems in execution trajectories and generate targeted prompt revisions. These revisions are then evaluated through a three-stage verification process that examines local behavioral changes, overall task performance, and effectiveness across different training tasks. A multidimensional reward function guides the selection of prompt revisions. Experimental results demonstrate improvements in task completion and execution efficiency compared to existing prompt optimization methods.

Critical strengths
  1. The method achieves reasonably strong experimental results, showing improvements in task completion and execution efficiency over existing baselines.

  2. The appendix is comprehensive, providing extensive implementation details, additional experimental analyses, and concrete examples of prompt evolution.

Critical weaknesses
  1. The proposed method lacks substantial novelty. Nearly all core components rely on LLM prompting, using LLMs to analyze execution trajectories, identify issues, and generate prompt revisions. The overall approach is rather straightforward, with limited methodological innovation.

  2. The method heavily relies on LLMs to identify issues, locate problematic decisions, and generate prompt revisions. However, the paper does not directly evaluate the accuracy or reliability of these LLM-generated decisions. It remains unclear whether the identified issues and their attributed causes are actually correct.

  3. The Issue Generator identifies at most three issues, but the paper does not clearly explain how the most important issues are selected when more than three exist. The paper also does not justify or experimentally analyze this fixed limit. This may cause important issues to be overlooked.

  4. The Fast Replay verification criterion is questionable. It only checks whether the generated action changes and executes successfully, without verifying whether the identified issue is actually resolved.

  5. The ablation study does not sufficiently isolate the contributions of individual components. In particular, removing the Issue Generator also removes Fast Replay, making it difficult to assess the independent contribution of the LLM-based issue identification process.

  6. The experiments lack a fair, budget-matched comparison with a simple LLM-based prompt optimization baseline. Although the paper includes Reflection-GT, it uses substantially fewer computational resources than AutoPEER. It remains unclear whether the improvements come from the proposed framework or simply from additional LLM calls and validation.

  7. Although the paper reports token usage and estimated costs, it lacks a detailed analysis of the end-to-end runtime and computational overhead introduced by the multiple LLM-based optimization and verification stages.

Questions for authors
  1. The paper claims to address credit assignment, but I do not see any actual credit assignment mechanism. The method simply uses an LLM to diagnose errors in execution trajectories. How is this different from standard LLM-based error diagnosis, and how does it justify the credit assignment claim?

  2. How do the authors verify the accuracy and reliability of the issues identified by the LLM and their attributed causes?

  3. Why does Patch Generation use constrained editing operations (Addition, Replacement, and Deletion) instead of allowing the LLM to directly rewrite the prompt? Is there any empirical evidence supporting this design choice?

1 comment

Thank you for your feedback. We have responded to your questions above and would appreciate hearing your thoughts soon.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their feedback and discussion. In our rebuttal, we added experiments and analyses of training settings, computational budget, and verification. We randomly reordered the 30 training batches, and the resulting prompt achieved an average TGC of 84.82% on Normal. We also found that 85% of training batches generated fewer than three Issues. Increasing the Issue cap to 5 yielded a Normal TGC of 83.93%. These experiments further examined performance under changes in training order and the Issue cap.

We also increased the training budget for Reflection-GT to test whether more training could further improve task completion:

MethodTraining tokens (M)Normal TGCChallenge TGC
Reflection-GT (main table)16.3285.12%70.50%
Reflection-GT (increased training budget)51.1985.71%64.75%

With more training, Normal performance improved slightly, while Challenge performance declined. This shows that more training does not necessarily lead to better task completion on Challenge.

Regarding validation cost, AutoPEER used 10.35% fewer Worker tokens for complete task execution than the variant without Fast Replay. We also provided an example of verification preventing a regression. In generation 26, a patch reduced total steps on the current batch from 30 to 27, but reduced the number of completed recent tasks from nine to eight. Rolling detected the regression that the current batch did not reveal, rejected the update, and retained the Parent.

We believe this additional evidence deserves full consideration, and we disagree with characterizing AutoPEER as a minor incremental contribution. Its overall optimization process differs clearly from GEPA and ACE, as we have explained in our responses. The assessment of incremental novelty still does not specify which existing work already contains the same overall mechanisms or which of our differences are merely minor extensions. This makes it difficult for us to provide further experiments or explanations targeted at that judgment.

In particular, we have also seen criticisms similar to Reviewer tpMr’s concern about the ablations in AI-generated reviews. We have explained that comparing the full method with the variant without Fast Replay examines the role of early filtering, while comparing the variant that retains Issues but removes Replay with the variant that removes both provides evidence for Issue guidance. We regret that Reviewer tpMr did not participate in the subsequent discussion. Based on these comments and the lack of further discussion, we do not believe the reviewer fully understood our work or the two baseline methods, GEPA and ACE.

Our supplementary materials also provide the complete code and some training trajectories. Even if our work is not particularly good, we do not believe it is meaningless or deserves only a score of 1. We are especially grateful to Reviewer tsu3 for recognizing the value of our work. Frankly, this recognition gave us the motivation to continue the discussion; without that support, we might not have continued this far.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.