acceptodds
Under review as a conference paper at ICLR 2027

GEPA-Curator: Curated Learning for Self Improving Agents

Abstract

Large Language Models (LLMs) are increasingly used in multi-turn agentic workloads that require sustained interaction with complex environments. Existing reflective prompt optimizers such as GEPA learn from failed rollouts, but agentic failures can also come from prompt weaknesses, harness limitations, stochastic execution, or noisy judges. We argue that such failures may not provide useful learning signal, and may even be harmful; and thus it is essential to curate which tasks are provided to the prompt optimizer for better performance. We introduce GEPA-Curator, a framework that augments the prompt optimizer to select which agent experiences should shape prompt optimization. In particular, GEPA-Curator consists of two components: task curation, which filters out tasks that are likely to provide noisy or unreliable learning signals, and prior success retrieval, which selectively retrieves past successful trajectories for a task and provides them as additional context to the prompt optimizer. We evaluate GEPA-Curator on GoBrowse, ALFWorld, AppWorld, and τ -bench, using both Gemini 3.1 Pro and Claude Opus 5 as optimizer models. GEPA-Curator matches or exceeds the mean performance of both baselines, with gains of up to 20.7% over GEPA and 21.2% over ACE.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llms, agents, gepa, data, curation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_K6SdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GEPA-Curator, a framework designed to improve reflective prompt optimization by curating the experiences used for learning. The authors argue that failures in agentic environments do not always indicate prompt deficiencies, as they may instead arise from environment limitations, faulty tasks, or noisy evaluations. Learning from such failures can lead to unnecessary or harmful prompt mutations.

To address this issue, GEPA-Curator introduces two components. Task Curation selects tasks that are likely to provide useful optimization signals based on their execution histories, while Prior Success Retrieval supplies reliable historical successful trajectories to the Reflector as additional context for diagnosing current failures.

Experiments on GoBrowse, ALFWorld, AppWorld, and τ-bench demonstrate consistent improvements over GEPA and ACE across two optimizer models. The authors further extend the approach to Meta-Harness optimization and report additional improvements.

Critical strengths
  1. The paper identifies an important and practical problem in reflective prompt optimization. The observation that agent failures may originate from environment limitations or evaluation noise rather than prompt weaknesses is well motivated.

  2. Both task curation and prior success retrieval can be implemented as an additional layer without fundamentally changing the underlying optimization process. The use of historical successful trajectories is particularly intuitive and potentially useful for improving reflection quality.

  3. The experimental results are promising across multiple agentic benchmarks. The ablation study also suggests that the two proposed components are complementary.

Critical weaknesses
  1. The reliance on strong LLMs raises concerns about cost and the actual source of improvement. The paper uses powerful models like Gemini and Claude for task filtering and prior success retrieval. This introduces additional inference costs and makes it unclear whether the gains come from the proposed curation strategy or simply from the strong models’ reasoning abilities. More detailed ablations, including weaker Curator models and cost-matched baselines, would help clarify this.

  2. The quality of task-selection decisions is not sufficiently evaluated. The curator aims to identify tasks that provide useful learning signals, but the paper mainly reports improvements in final performance. It remains unclear how often useful tasks are skipped or noisy tasks are retained. More importantly, the paper does not show whether the selected tasks actually lead to better prompt updates.

  3. The ablation studies are limited. The main ablation is conducted only on GitLab, making it unclear whether both components are equally effective across other benchmarks. Moreover, the Meta-Harness experiments use only one optimization run per setting, which makes the reported improvements less convincing.

Questions for authors
  1. How much additional inference cost does the Curator introduce? Could the authors report the additional inference costs? How would GEPA-Curator compare with GEPA under the same total inference budget, rather than just the same rollout budget?

  2. Why filter tasks instead of filtering unreliable failures? Since the main concern is that the Reflector may learn from misleading failures, why not let it identify and ignore environment-related or evaluation-related failures directly after execution? How would such a failure-aware approach compare with the proposed task-level curation?

5 comments
AuthorsOpenReview ↗

Response to Review

We appreciate the reviewer for taking the time to review our work and providing us with their valuable feedback.

1. Weak curator model experiment

The reliance on strong LLMs raises concerns about cost and the actual source of improvement. More detailed ablations, including weaker Curator models…

Thank you for suggesting this experiment. We have run experiments using Gemini 3.5 Flash-Lite for the Actor, Reflector, and Curator. We chose this smaller model to demonstrate Curation is not restricted to strong optimizer models, and to ensure affordability.

AppWorld:

MethodTest-N (%)Test-C (%)
GEPA57.3 ± 1.747.6 ± 2.5
GEPA-Curator65.0 ± 3.758.6 ± 1.9

WebArena:

MethodGitLab (%)Shopping (%)Shopping Admin (%)Reddit (%)Average (%)
GEPA66.1 ± 3.279.2 ± 2.664.0 ± 2.582.9 ± 0.673.0
GEPA-Curator69.9 ± 0.882.0 ± 1.565.6 ± 1.781.4 ± 0.474.7

On AppWorld, GEPA-Curator improves over GEPA by 7.7% on Test-N and 11.0% on Test-C. On WebArena, GEPA-Curator improves the average success rate by 1.7%. Thus, even with smaller optimizer models, GEPA-Curator provides considerable gains over GEPA. This demonstrates how the benefit persists when the actor and optimizer use the same underlying model and does not require a stronger teacher model.

AuthorsOpenReview ↗

Response to Review

2. Cost Analysis

How much additional inference cost does the Curator introduce? Could the authors report the additional inference costs?

The motivation of this work is to establish whether an autonomous agent can curate its own learning experiences. We therefore prioritized demonstrating the effectiveness of autonomous curation, and did not initially optimize the frequency of Curator calls or the amount of historical context processed by each call. These implementation choices leave room to improve efficiency.

We introduced three optimization methods that can be applied to GEPA-Curator without score degradation:

  1. Decide the skip interval once. The Curator specifies how many epochs to skip a task, avoiding repeated Curator calls during that interval.
  2. Compact older histories. Keep complete trajectories for recent execution units and summarize older entries using their epochs, outcomes, and previous Curator assessments. We retain the latest four execution units per task on WebArena and the latest one on the non-WebArena benchmarks.
  3. Reusing successful reference decisions. Preserve an optional full successful trajectory across history compaction and supply it to the Reflector when a fresh execution fails. The Curator can replace this reference with a better success from recent executions.
MethodGitLabShoppingShopping AdminReddit
GEPA-Curator84.6 ± 1.585.8 ± 3.573.1 ± 1.985.0 ± 1.8
+ Optimizations83.8 ± 2.485.1 ± 1.473.3 ± 1.686.8 ± 2.8
MethodALFWorldAppWorld Test-NAppWorld Test-Cτ-bench
GEPA-Curator91.8 ± 1.458.7 ± 1.946.2 ± 2.184.0 ± 4.8
+ Optimizations90.0 ± 1.067.1 ± 3.956.3 ± 1.584.0 ± 3.3

As we can see, the performance of GEPA-Curator using these optimizations is on-par with the previous results, so for the final paper, we will use make these part of the default method.

Below, we compare the costs between GEPA, GEPA-Curator, and ACE averaged across Webarena and non-Webarena benchmarks. We report the estimated monetary inference costs accounting for all of the computational expense for the input and output tokens consumed by all of the actor rollouts, reflection, and curation. This also captures the additional inference overhead from the Curator calls and potential longer reflection context lengths.

Benchmark groupMethodEstimated optimization cost (USD)Average test success (%)
WebArenaGEPA594.0578.7
WebArenaGEPA-Curator776.8882.3
Non-WebArenaGEPA132.6555.9
Non-WebArenaACE768.6965.2
Non-WebArenaGEPA-Curator173.9874.4

Compared with GEPA, GEPA-Curator improves average success by 3.6% on WebArena and 18.5% on the non-WebArena settings, at approximately 1.31× the cost in each group. Compared with ACE, it achieves 9.2% higher average success on the non-WebArena settings with 4.42x less cost than ACE (\173.98versus173.98 versus \\768.69).

We separate rollout, reflection, and curation costs below:

Benchmark groupMethodRollouts (USD)Reflection (USD)Curation (USD)Total (USD)
WebArenaGEPA552.7741.27—594.05
WebArenaGEPA-Curator578.4798.8799.54776.88
Non-WebArenaGEPA115.9016.75—132.65
Non-WebArenaACE175.43593.25—768.69
Non-WebArenaGEPA-Curator97.8131.5544.62173.98

We can see based on this breakdown that while the cost of Reflection and Curation is higher for GEPA-Curator, the majority of the cost is associated with the actor rollout cost. In these experiments, we have used gpt-oss-120b, which has extremely cheap API costs compared to normal frontier models. We expect that the relative overhead of GEPA-Curator to decrease when we use more expensive actor models, since rollout costs would begin to dominate.

AuthorsOpenReview ↗

Response to Review

3. Matched Costs Experiments

How would GEPA-Curator compare with GEPA under the same total inference budget, rather than just the same rollout budget?

We also run experiments with matched costs, where we stop the GEPA-Curator training strictly before cost exceeds GEPA. The experiments uses gpt-oss-120b as the actor model and Gemini-3.1-pro as all optimizer models. The results are shown in the table below:

BenchmarkGEPA-Curator test (%)GEPA test (%)GEPA-Curator cost ($)GEPA cost ($)
GitLab87.382.1141.23144.01
Shopping86.083.8155.22156.10
Reddit85.877.6141.01141.89
Shopping Admin70.771.5150.38152.05
ALFWorld94.382.454.9655.66
AppWorld, Test-N / Test-C65.5 / 51.838.0 / 27.860.1461.78

When matching total cost rather than rollout budgets, GEPA-Curator outperforms GEPA by an average of 11.2% in success rate over the six benchmarks.

4. Curation quality and whether selected tasks yield better updates

The quality of task-selection decisions is not sufficiently evaluated. The curator aims to identify tasks that provide useful learning signals, but the paper mainly reports improvements in final performance. It remains unclear how often useful tasks are skipped or noisy tasks are retained. More importantly, the paper does not show whether the selected tasks actually lead to better prompt updates.

We independently audit the Curator's run/skip decisions using Codex to examine execution histories and assess whether failures are prompt-addressable or uninformative. We report the accuracy of the Curator's decisions under these conditions below.

MetricOpus 5Gemini 3.1 Pro
Correctly identifying prompt-addressable failure92.57%86.13%
Correctly identifying uninformative failure95.31%83.82%
Overall accuracy93.94%84.98%

Both models generally select the appropriate action, with Opus 5 being better at task curation than Gemini 3.1 Pro. Opus 5 achieves an overall accuracy of 93.94% and Gemini 3.1 Pro achieved 84.98%. Details of how this metric is constructed can be found in Appendix H.

As shown in Section 5.1, retaining already mastered tasks does not affect the optimization on optimizable tasks. In the mastered + optimizable task composition, performance on optimizable tasks consistently improves and approaches saturation. Instead, the inclusion of impossible or flaky tasks is what harms optimization. These findings suggest that the primary benefit of task curation comes from filtering out impossible and flaky tasks rather than removing already-mastered tasks.

In the table below, we run an ablation using just the task curation component of GEPA-Curator and compare it to GEPA on AppWorld and Gitlab. We can see that using task curation matches or improves over GEPA on both datasets, which shows us that task curation can lead us to better prompt updates.

MethodGitLab (%)AppWorld Test-N (%)AppWorld Test-C (%)
GEPA82.8 ± 1.638.0 ± 2.327.8 ± 2.1
Task selection only84.2 ± 2.946.9 ± 2.127.7 ± 2.1
Reviewer_ormsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper extends GEPA, additionally allowing the "optimizer model" to choose which questions to give to the agent during prompt optimization. For the main experiments they use gpt-oss-120b as the agent model and gemini-3.1-pro and opus-5 for the optimizer model, as well as a different set of models for their meta-harness experiments. Their method does consistently better than the GEPA baseline and they have lots of interesting analysis of dataset composition etc.

Critical strengths
  • Relevant work to the future of agentic AI systems for things we should consider integrating into harnesses
  • Well-written and explained clearly
  • Sensible and interesting analysis
Critical weaknesses
  • The results are not that surprising - they are getting a much much stronger model to prompt-optimize a much weaker model and essentially show that giving the strong model more agency (ie it gets to pick the tasks) then it can do a better job of the prompt-optimization.
  • Related to above, I think this only really becomes relevant to frontier agentic systems if the agent and the optimizer model are the same - which would be self-improvement - but as far as I can see they only have experiments where the optimizer agent is much stronger. These would be interesting and should be fairly easy to run.
Questions for authors
  • Did you try the self-improvement version of this where the agent and the optimizer model are the same? If so what are the results?
  • In Figure 2 right - the "flaky" line seems to go up just as much as the "optimizable" line? Is the trend unexpected, and if so do you have a sense of why?
  • It would be interesting to plot the prompt length over iterations, and also to see some examples of the prompt before and after optimization and the types of edits that are made.
2 comments
AuthorsOpenReview ↗

Response to Review

We appreciate the reviewer for taking the time to review our work and providing us with their valuable feedback.

1. Self-improvement with the same actor and optimizer, using a weak model

The results are not that surprising - they are getting a much much stronger model to prompt-optimize a much weaker model and essentially show that giving the strong model more agency (ie it gets to pick the tasks) then it can do a better job of the prompt-optimization.

…

Related to above, I think this only really becomes relevant to frontier agentic systems if the agent and the optimizer model are the same - which would be self-improvement - but as far as I can see they only have experiments where the optimizer agent is much stronger. These would be interesting and should be fairly easy to run.

…

Did you try the self-improvement version of this where the agent and the optimizer model are the same? If so what are the results?

Thank you for suggesting this experiment. We have run experiments using Gemini 3.5 Flash-Lite for the Actor, Reflector, and Curator. We chose this smaller model to demonstrate Curation is not restricted to strong optimizer models, and to ensure affordability.

AppWorld:

MethodTest-N (%)Test-C (%)
GEPA57.3 ± 1.747.6 ± 2.5
GEPA-Curator65.0 ± 3.758.6 ± 1.9

WebArena:

MethodGitLab (%)Shopping (%)Shopping Admin (%)Reddit (%)Average (%)
GEPA66.1 ± 3.279.2 ± 2.664.0 ± 2.582.9 ± 0.673.0
GEPA-Curator69.9 ± 0.882.0 ± 1.565.6 ± 1.781.4 ± 0.474.7

On AppWorld, GEPA-Curator improves over GEPA by 7.7% on Test-N and 11.0% on Test-C. On WebArena, GEPA-Curator improves the average success rate by 1.7%. Thus, even with smaller optimizer models, GEPA-Curator provides considerable gains over GEPA. This demonstrates how the benefit persists when the actor and optimizer use the same underlying model and does not require a stronger teacher model. Furthermore, we have used the same LLM for all of the components, which shows us that GEPA-Curator can indeed elicit self-improvement.

2. The increasing success rate of the flaky tasks in Figure 2

In Figure 2 right, the “flaky” line seems to go up just as much as the “optimizable” line. Is the trend unexpected, and if so do you have a sense of why?

Thank you for pointing this out. Because we conducted this case study on a small-scale dataset, the optimizer found ways to reward hack on some of the tasks to improve the flaky tasks success rate. As a result, optimizable tasks improve less when mixed with flaky tasks than when mixed with mastered tasks, which is coherent with our motivation.

One example is at iteration 10: the optimizer added an instruction to “perform a plausible generic action to fulfill the mechanical requirement of the task” when the target was unspecified, rather than asking for clarification. This encourages apparent compliance despite an unresolved task requirement. We then observed evaluator leniency on a repository-cloning task: the final agent claimed to have cloned the repository after merely retrieving a clone command, and the evaluator accepted this substitution because the agent lacked terminal access. This particular gain therefore demonstrates flaky evaluation, rather than a newly learned ability to clone the repository.

This observation shows why increased success rate may not reflect improved actor behavior, as they can also be reward hacked.

3. Prompt length and examples of edits

It would be interesting to plot the prompt length over iterations, and also to see some examples of the prompt before and after optimization and the types of edits that are made.

Thank you for this suggestion. We have added prompt length plots, examples of optimized prompts, and examples of types of edits in Appendix J. Prompts before optimization (seed prompts) can be seen at Appendix B.3. The types of edits introduce more explicit navigation steps, stopping conditions, and instructions for handling environment failures.

Thank you for your response and new results. I still feel a weak reject is suitable due to my main concern of it being an incremental change over existing methods.

Reviewer_mhjyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GEPA-Curator augments reflective prompt optimization with an LLM that selects training tasks from their execution histories and retrieves reliable prior successes to contextualize current failures. The authors evaluate the approach on GoBrowse/WebArena, ALFWorld, AppWorld, and a modified τ-bench setting, with additional experiments on QA and harness-code optimization. The main results show improved held-out success rates over GEPA and ACE under matched actor-rollout budgets.

Critical strengths
  1. Consistent and sometimes substantial empirical improvements. The main evaluation covers several agent environments and two optimizer models, with five repeated runs. The AppWorld gains are particularly substantial; for example, with Gemini, Test-N improves from 38.0 to 58.7 over GEPA. This is meaningful evidence that the complete system can be useful beyond one noisy browser dataset.

  2. A practical failure mode and an inspectable intervention. Distinguishing prompt-correctable failures from environment or evaluator failures is well motivated. Maintaining task histories, allowing skipped tasks to return, and supplying corroborated successes are sensible design choices. The explicit prompts and diagnosis examples help readers understand the implementation.

Critical weaknesses
  1. The contribution beyond existing experience-based reflection and simpler selection rules is insufficiently established. The related work omits directly relevant approaches such as ExpeL, which already extracts insights from successful/failed experience pairs and reuses past successes [1]. This does not subsume the proposed noise-aware curation, but makes isolating that additional capability important. The paper does not compare semantic success selection with simply supplying a recent or randomly selected successful trace. Moreover, the numerical curriculum in Appendix A.3 permanently removes repeatedly failing tasks, whereas GEPA-Curator revisits skipped tasks. Its weaker result therefore does not isolate the value of semantic reasoning over trajectories. Appendix C provides useful examples and a manual categorization of flagged cases, but not an independent estimate of whether the Curator correctly distinguishes uninformative tasks from difficult, learnable ones. The current evidence supports the combined system more strongly than the claimed necessity of its particular curation mechanism.

  2. The ablation and budget protocols limit the interpretation of the gains. Section 4.3 reports component ablations on the GitLab validation set, while Section 4.1 uses validation performance to select the final prompt. These are not independent test estimates of component benefit. The same single-domain ablation is also used to justify removing GEPA's Pareto-based search across the other benchmarks. In addition, equal actor-rollout counts do not account for the extra Curator calls, history processing, or longer reflection contexts. No corresponding token, monetary-cost, or wall-clock comparison is reported. Consequently, the results do not yet establish whether the proposed machinery is preferable to simpler history-aware baselines at comparable total cost, or whether the component conclusions generalize to the benchmarks with the largest gains.

[1] Zhao et al. ExpeL: LLM Agents Are Experiential Learners. AAAI 2024. Paper.

Questions for authors
  1. Contribution isolation: How does GEPA-Curator compare with a baseline that retrieves a recent/random successful trace and uses a numerical curriculum with the same temporary skipping and revisit policy? Please specify the numerical curriculum's thresholds and tuning procedure. This comparison would directly test the benefit of semantic curation.

  2. Evaluation clarification: Were the Table 3 validation tasks also used to select each reported checkpoint? Can the main component ablations be evaluated on an independent test split, preferably including AppWorld where the gains are largest? Please also clarify which GEPA acceptance gates remain in the single-lineage variant and whether every proposed mutation is accepted.

  3. Robustness analysis of curation pipeline: On an independently checked sample of both selected and skipped tasks, how often does the Curator mistake a difficult but prompt-correctable failure for an uninformative one? Can the authors distinguish the benefit of removing noisy tasks from the benefit of avoiding already-mastered tasks?

  4. Cost accounting: What are the total optimizer/Curator input and output tokens, call counts, and runtime at the reported rollout budgets? Does the advantage persist at a matched total inference budget?

6 comments
AuthorsOpenReview ↗

Response to Review

We appreciate the reviewer for taking the time to review our work and providing us with their valuable feedback. Below, we provide responses to the questions and concerns raised in the review.

1. ExpeL Comparison

The related work omits directly relevant approaches such as ExpeL, which already extracts insights from successful/failed experience pairs and reuses past successes [1]. This does not subsume the proposed noise-aware curation, but makes isolating that additional capability important.

Thank you for pointing out this related work. We will add ExpeL to our related work and clarify its relationship to GEPA-Curator. The main difference is that ExpeL extracts insights from past trajectories and provides rules and examples to the agent at inference time, whereas GEPA-Curator leverages past experiences to curate the tasks and trajectories used by a prompt optimizer. As such, ExpeL is more directly comparable to GEPA itself, while GEPA-Curator operates on top of existing prompt optimizers. The two methods therefore operate at different levels and are potentially complementary.

Below we run ExpeL on AlfWorld and AppWorld and compare them to GEPA-Curator. Both runs use Gemini-3.1-pro as the optimizer and gpt-oss-120b as the actor with the same rollout budget.

MethodALFWorld (%)AppWorld Test-N (%)AppWorld Test-C (%)
ExpeL90.614.812.6
GEPA-Curator91.858.746.2

ExpeL performs close to GEPA-Curator on ALFWorld (90.6% versus 91.8%). We hypothesize that the way ExpeL extracts rules and provides in-context examples to the actor is good for memorization, which is good for ALFWorld where task setups have common patterns that can be memorized.

However, Expel only achieves 14.8%/12.6% on AppWorld compared to 58.7%/46.2% for GEPA-Curator. We think this is because AppWorld tasks involve more diverse and complex workflows, which are harder for ExpeL’s optimization strategy to memorize. GEPA-Curator, by curating the optimizer’s experience and providing prior successful trajectories, generalizes better to AppWorld.

[1] Zhao et al. ExpeL: LLM Agents Are Experiential Learners. AAAI 2024.

AuthorsOpenReview ↗

Response to Review

2. Contribution Isolation

The paper does not compare semantic success selection with simply supplying a recent or randomly selected successful trace. Moreover, the numerical curriculum in Appendix A.3 permanently removes repeatedly failing tasks, whereas GEPA-Curator revisits skipped tasks. Its weaker result therefore does not isolate the value of semantic reasoning over trajectories.

…

How does GEPA-Curator compare with a baseline that retrieves a recent/random successful trace and/or uses a numerical curriculum with the same temporary skipping and revisit policy? Please specify the numerical curriculum's thresholds and tuning procedure. This comparison would directly test the benefit of semantic curation.

Following the reviewer’s suggestion, we have completed a controlled previous success retrieval and numerical curriculum ablation. Below, we run 2 new experiments:

  • Latest Success Retrieval Only: where we only provide the latest success as context to the reflector
  • Numerical Curriculum with temporary skipping and revisiting + Latest Success Retrieval: When a task passes 3 times in a row, the task is now only skipped for 1 epoch and brought back. When a task fails 3 times in a row, the task is now skipped for 3 epochs and brought back. We also use the latest success retrieval mechanism from above.

These thresholds were selected through manual experimentation, with shorter skip intervals for mastered tasks to periodically detect regressions and longer intervals for difficult tasks to reduce potentially harmful prompt updates.

We run the ablations below on Appworld and Gitlab using the same models and budgets in the paper.

MethodTest-N (%)Test-C (%)GitLab (%)
Success selection only46.8 ± 2.135.9 ± 1.884.7 ± 3.1
Latest success only42.4 ± 3.127.1 ± 0.484.1 ± 0.8

In the table above, we ablate using just the latest success for success retrieval compared to just using the agentic success retrieval. “Success selection only” consistently performs better than “latest success only” on AppWorld and GitLab, since it can provide the more efficient/clean execution trace to reflector.

MethodTest-N (%)Test-C (%)GitLab (%)
GEPA-Curator58.7 ± 1.946.2 ± 2.188.6 ± 2.0
Numerical Curriculum + Latest Success Retrieval52.0 ± 2.836.4 ± 2.285.1 ± 1.4

In the table above, we compare GEPA-Curator to using Numerical Curriculum with latest success retrieval. GEPA-Curator consistently performs better than Numerical Curriculum + Latest Success Retrieval on AppWorld and GitLab.

3. Robust analysis of curation pipeline

Appendix C provides useful examples and a manual categorization of flagged cases, but not an independent estimate of whether the Curator correctly distinguishes uninformative tasks from difficult, learnable ones. … On an independently checked sample of both selected and skipped tasks, how often does the Curator mistake a difficult but prompt-correctable failure for an uninformative one? Can the authors distinguish the benefit of removing noisy tasks from the benefit of avoiding already-mastered tasks?

We independently audit the Curator's run/skip decisions using Codex to examine execution histories and assess whether failures are prompt-addressable or uninformative. We report the accuracy of the Curator's decisions under these conditions below.

MetricOpus 5Gemini 3.1 Pro
Correctly identifying prompt-addressable failure92.57%86.13%
Correctly identifying uninformative failure95.31%83.82%
Overall accuracy93.94%84.98%

Both models generally select the appropriate action, with Opus 5 being better at task curation than Gemini 3.1 Pro. Opus 5 achieves an overall accuracy of 93.94% and Gemini 3.1 Pro achieved 84.98%. Details of how this metric is constructed can be found in Appendix H.

As shown in Section 5.1, retaining already mastered tasks does not affect the optimization on optimizable tasks. In the mastered + optimizable task composition, performance on optimizable tasks consistently improves and approaches saturation. Instead, the inclusion of impossible or flaky tasks is what harms optimization. These findings suggest that the primary benefit of task curation comes from filtering out impossible and flaky tasks rather than removing already-mastered tasks.

AuthorsOpenReview ↗

Response to Review

4. Evaluation clarification

Section 4.3 reports component ablations on the GitLab validation set, while Section 4.1 uses validation performance to select the final prompt. These are not independent test estimates of component benefit. The same single-domain ablation is also used to justify removing GEPA's Pareto-based search across the other benchmarks.

…

Were the Table 3 validation tasks also used to select each reported checkpoint? Can the main component ablations be evaluated on an independent test split, preferably including AppWorld? Please also clarify which GEPA acceptance gates remain in the single-lineage variant and whether every proposed mutation is accepted.

Using Separate Test Sets. We agree that validation-set ablations should be distinguished from independent test estimates. Table 3 reports performance on the GitLab validation set, and we will make that scope explicit. These validation tasks were used for checkpoint selection. The new AppWorld ablations above extend the analysis to Test-N and Test-C, which is the benchmark the original paper reports its largest gains.

In the table below, we compare GEPA with the single-lineage greedy GEPA on both GitLab (validation set) and AppWorld (independent test sets).

MethodTest-N (%)Test-C (%)GitLab Val. (%)
GEPA38.0 ± 2.327.8 ± 2.182.8 ± 1.6
Greedy GEPA37.4 ± 1.623.9 ± 2.282.5 ± 1.9

As we can see, Greedy GEPA performs at or slightly below GEPA on both benchmarks, so our use of the single-lineage variant does not artificially favor GEPA-Curator in our study.

Single Lineage Details. Greedy GEPA (i.e., the single-lineage variant) still performs acceptance gating. It performs gating on the local mini-batch (and therefore will only be accepted when the mini-batch score increases) rather than on the validation data. Thus, not every proposed mutation is accepted. Greedy GEPA removes Pareto-based candidate selection while retaining mini-batch acceptance. Validation performance is used for checkpoint selection and not mutation acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.