acceptodds
Under review as a conference paper at ICLR 2027

EEVEE: Post-Deployment Prompt Learning across Heterogeneous Tasks

Abstract

Prompt learning has become an important approach to adapting large language models to downstream tasks. However, some existing methods require access to model internals, while others are largely designed for single-dataset settings, limiting their practical applicability. In real-world deployments, models are often accessible only as black boxes, while incoming queries form heterogeneous streams drawn from multiple datasets, domains, and task distributions. We therefore propose EEVEE, a multi-dataset post-deployment prompt learning framework for LLM agents, enabling prompt learning under heterogeneous task streams. To mitigate cross-dataset interference, EEVEE introduces a router that partitions incoming inputs into task clusters and assigns them to suitable prompt configurations. This design is optimized via a router-prompt co-evolution strategy, which employs interleaved router and prompt learning phases to address their mutual dependency. Experiments across multiple datasets demonstrate that the framework improves robustness under heterogeneous data streams while maintaining single-benchmark learning capability and efficiency. Specifically, EEVEE improves average multi-benchmark scores by 10.38 and 24.32 points over Qwen3-4B-Instruct and DeepSeek-V3.2, surpassing SOTA methods GEPA and ACE by up to 37.2% and 48.2%.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Post-deployment prompt learning, Heterogeneous tasks, LLM agents, Prompt optimization, Router-prompt co-evolution

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kQyXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

EEVEE adapts black-box language models to heterogeneous tasks without changing their weights. It maintains several specialized prompts and optimizes a prompt-based router to select the appropriate prompt for each input. The router and prompts are co-evolved using correctness feedback. Experiments on four benchmarks and two models show improvements over GEPA and ACE, with additional analyses of transfer, online adaptation, and computational cost.

Critical strengths
  1. Co-evolution addresses the dependency between routing and prompt learning. Alternating updates allow input assignments and specialized prompt capabilities to develop together.

  2. Coverage-based initialization and Pareto-front selection preserve prompts with different strengths. This accommodates heterogeneous tasks while reducing the need to combine potentially conflicting instructions in one prompt.

  3. EEVEE improves average performance over GEPA and ACE across four benchmarks and two models. Routing and co-evolution ablations further support the contributions of its core components.

Critical weaknesses
  1. Requires task-specific data and feedback. Adapting to new tasks depends on representative examples and reliable correctness feedback, which may be costly or unavailable, especially for open-ended tasks.

  2. Fixed prompt capacity. EEVEE refines a fixed set of prompts without automatically adding specialists for new task types, potentially limiting its flexibility as task diversity grows.

  3. Early routing errors may reinforce poor specialization. Prompts learn from their currently assigned examples, so unsuitable initial assignments can shape subsequent capabilities and reinforce the same partition.

  4. Online updates remain challenging. This paper does not clearly explain when to update and how to maintain data during online learning.

Questions for authors

Please refer to the Weaknesses section.

5 comments
AuthorsOpenReview ↗

Response to Reviewer kQyX (1/2)

We thank reviewer kQyX for recognizing our co-evolution design, the diverse prompt selection, and the consistent improvements over GEPA and ACE.

W1. Requirement of task-specific data and feedback

Thanks for the comment. Learning from task examples with correctness feedback is adopted by most prior work in prompt optimization and context adaptation, such as OPRO (Yang et al., 2024), TextGrad (Yuksekgonul et al., 2024), GEPA (Agrawal et al., 2026) and ACE (Zhang et al., 2026). EEVEE follows this setting and does not require additional supervision such as dataset labels. It also requires only a small amount of data: with 200 adaptation examples (50 per benchmark), EEVEE still reaches 48.59, compared with 41.37 for the unadapted model (Appendix B.2).

We also study the case where feedback is sparse or noisy (Appendix B.3), where Sparse-k keeps feedback on only k% of the examples and Noisy-k flips k% of the feedback.

Feedback conditionSparse-50Sparse-25Noisy-10Noisy-20
Retained performance88.4%89.7%89.9%85.7%

EEVEE retains more than 85% of its clean-feedback performance in all conditions and still improves over the unadapted model.

Moreover, many tasks, such as common agent tasks, do not have reference answers but provide a verifier of task success. We additionally evaluate EEVEE in this setting on ALFWorld (Shridhar et al., 2021), a text-based household environment where the agent completes a goal by issuing text actions (e.g., "go to countertop 1", "take apple 1") for up to 50 steps, and the only feedback is whether the goal is completed. We use Qwen3-4B-Instruct as in our main experiments, with an adaptation pool of 40 episodes per task type and 4 prompt slots for four task types: Look (find an object and examine it under a desk lamp), Clean (clean an object in the sink and place it), Heat (heat an object with the microwave and place it) and Cool (cool an object with the fridge and place it).

MethodLookCleanHeatCoolAvg.
No adaptation9.2611.8323.1917.4615.43
EEVEE42.5946.2442.0330.1640.25

EEVEE improves the average success rate from 15.43 to 40.25, showing that it also works with verifier-based feedback. We will clarify the feedback requirements and add these results to the revised paper.

W2. Fixed prompt capacity

Thanks for the comment. EEVEE is not limited by a fixed number of slots as the number of tasks grows, since the number of slots bounds the number of specialized behaviors rather than task types, and a new task can reuse an existing slot when it shares behaviors with other tasks. To verify this, we additionally scale the mixture to 8 tasks by adding IFBench (Pyatkin et al., 2025), CRUXEval (Gu et al., 2024), FinQA (Chen et al., 2021), and BBH (Suzgun et al., 2023) or LegalBench (Guha et al., 2023), while keeping only 4 slots.

MixtureMethodGPQAFormulaTheoremQAHumanEvalIFBenchCRUXEvalFinQABBHLegalBenchAvg.
8-task (BBH)No adaptation55.2245.1715.5850.4131.9766.2467.8383.71–52.02
EEVEE, 4 slots50.8475.6721.3350.8135.6085.7070.2188.38–59.82
8-task (LegalBench)No adaptation52.1945.0016.0049.1932.8867.3367.65–81.6751.49
EEVEE, 4 slots52.5355.6719.5255.6935.3778.4271.29–84.4056.61

With only 4 slots, EEVEE improves the 8-task average by 7.80 and 5.12 points, with large gains on Formula and CRUXEval. We also compare with 8 slots, one per task, on the same mixtures.

MixtureSlotsNo adaptationEEVEEGain
8-task (BBH)452.0259.82+7.80
852.1859.62+7.44
8-task (LegalBench)451.4956.61+5.12
852.0959.99+7.90

The gains are similar with 4 and 8 slots, showing that EEVEE maintains strong performance over a wide range of task numbers without a clear drop as the number of tasks grows. We believe that 8 tasks cover a substantial range, since prompt optimization typically targets a specific deployment workload rather than a general-purpose model. We will add these results to the revised paper.

AuthorsOpenReview ↗

Response to Reviewer kQyX (2/2)

W3. Early routing errors

Thanks for the insightful comment. EEVEE does not lock in early assignments: each co-evolution cycle re-routes all adaptation data, and router evolution learns from cases where the assigned slot fails but another slot succeeds. To verify this, we additionally start EEVEE from a deliberately bad partition, where each slot receives an equal share of all four datasets and its prompt is first evolved on this partition until convergence. All other settings were kept unchanged, and we report the average over the same 3 runs as the main experiments.

SettingGPQAFormulaTheoremQAHumanEvalAvg.
No adaptation56.0045.2214.7949.4641.37
No Co-evolution39.8447.3924.8159.4942.88
EEVEE, bad partition55.5653.1722.8170.3350.47 ± 2.06
EEVEE, original54.5554.5525.2772.6351.75 ± 1.62

Starting from a bad partition, EEVEE reaches 50.47, close to the original 51.75 and clearly above No Co-evolution (42.88). This shows that co-evolution remains effective even with poor initial assignments. We will add these results to the revised paper.

W4. Online update strategy

Thanks for the comment. In our online setting (Appendix B.4), the stream is processed window by window: EEVEE answers all queries in a window with its current router and prompts, and then updates once before the next window. For data maintenance, we compare two strategies: cumulative, which updates on all observed examples, and new-window, which updates only on the latest window. Both continue from the previous router and prompts, and a new prompt is accepted only if it outperforms the empty prompt on validation data. We additionally evaluate the unadapted model on the same stream to verify the actual improvement of EEVEE.

MethodGPQAFormulaTheoremQAHumanEvalAvg.
No adaptation57.7443.0715.9242.4839.80
ACE online29.1242.2316.9228.8629.28
EEVEE cumulative51.0150.6724.7147.9743.59
EEVEE new-window50.0052.9523.4449.4943.97

Both strategies improve over the unadapted model by 3.79 and 4.17 points, respectively, while ACE online falls 10.52 points below it. We will add a detailed description of the online procedure to the revised paper.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer kQyX,

Thank you again for your time and constructive feedback. In our responses, we added experiments on 8-task mixtures with only 4 prompt slots, on starting EEVEE from a deliberately bad partition, and on the ALFWorld agent benchmark with verifier-based feedback. As the discussion period is coming to a close, we would like to check whether our responses have addressed your concerns. We would be happy to provide any further clarification.

Best regards,
The Authors

Reviewer_JNj9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

EEVEE addresses a practical gap in prompt learning: existing methods either require model-internal access or are designed for single-dataset adaptation, while real deployments expose frozen black-box models to heterogeneous streams of queries from multiple datasets, domains, and capability types. EEVEE maintains a set of K specialized text prompts plus a learned router that assigns each incoming query to a prompt slot. Because routing quality depends on prompt quality and vice versa, the paper proposes router-prompt co-evolution: interleaved phases in which router evolution and per-slot prompt evolution alternate, organized into three stages. All learning uses only correctness feedback from a black-box model; no weights are trained and no internal representations are accessed.

Critical strengths

-The paper demonstrates cross-dataset interference concretely and frames the black-box, heterogeneous-stream setting convincingly.

-The paper systematically rules out the most plausible alternative explanations: equal-budget routing controls show that neither random routing nor a mixed single prompt nor even a 100%-accurate dataset-ID classifier or dataset-ID oracle reaches EEVEE, supporting the claim that the router captures behavior-level structure beyond source identity.

Critical weaknesses

-Mutation/reflection-based prompt search (GEPA), evolving contexts (ACE), query routing among expert prompts (MoP), Pareto/memory-based retention (memory-extraction and agent-memory works) are each established. EEVEE's contribution is the joint router-prompt co-evolution schedule and its budgeted three-stage organization — a valuable engineering contribution, but the paper does not quantify how much of the gain comes from (a) the co-evolution schedule vs. (b) simply having more, better-scored prompt slots under a fixed router.

-All experiments are static single-turn benchmarks (QA, math, code). There is no evaluation on multi-step agent tasks (e.g., SWE-bench, WebArena, ALFWorld), tool use, or long-horizon interactive streams — precisely the settings where a "deployed agent" and a heterogeneous query stream would matter most.

-Appendix B.5 (Table 16): EEVEE consumes 104,460 adaptation calls / 154.71M tokens / 7.86h — roughly 2.7× ACE's adaptation tokens (58.36M) and only 32% fewer than GEPA (226.81M). The paper's token advantage (4.32k vs 21.30k per test example) is at inference; the one-off adaptation cost is comparable to the strongest baseline and should be weighed when claiming "efficiency."

Questions for authors

-The paper does not quantify how much of the gain comes from the router-prompt co-evolution schedule versus simply having more, better-scored prompt slots under a fixed router — please compare EEVEE against prompt-only evolution under the learned (or oracle) router on matched budget.

-All experiments are static single-turn benchmarks (QA, math, code); there is no evaluation on multi-step agent tasks (e.g., SWE-bench, WebArena, ALFWorld), tool use, or long-horizon interactive streams — precisely the settings where a "deployed agent" and heterogeneous query streams matter most.

-EEVEE consumes 104,460 adaptation calls / 154.71M tokens / 7.86h — about 2.7× ACE's adaptation tokens (58.36M) and only 32% fewer than GEPA (226.81M); its token advantage (4.32k vs. 21.30k/test example) is at inference only, so the one-off adaptation cost comparable to the strongest baseline should be weighed when claiming efficiency.

5 comments
AuthorsOpenReview ↗

Response to Reviewer JNj9 (1/2): Weaknesses

We thank reviewer JNj9 for recognizing our practical problem setting, the concrete evidence of cross-dataset interference, and the systematic routing controls.

W1. Contribution of co-evolution versus more prompt slots

Thanks for raising this point. We agree it is important to separate the gain of co-evolution from that of more prompt slots. Our results show that co-evolution brings a clear gain beyond simply having more prompt slots under a fixed router. To isolate this, we compare EEVEE with variants that use the same budget and the same number of prompt slots, but with a fixed router (i.e., without co-evolution), as reported in Tables 2 and 5:

  • Oracle dataset ID: each input is routed to the slot of its source dataset using the ground-truth dataset label.
  • Learned classifier: inputs are routed by a classifier trained to predict the source dataset.
  • No Co-evolution: the router is first learned by EEVEE and then fixed.
SettingRouterSlotsAvg.
No adaptationnone141.37
Mixed single promptnone142.12
No Co-evolutionlearned, then fixed442.88
Oracle dataset IDfixed447.15
Learned classifierfixed447.98
EEVEEco-evolved451.75

With a fixed router, more prompt slots reach at most 47.98, even when inputs are routed perfectly by dataset. Co-evolution brings a further 3.77 points and outperforms the oracle on all four benchmarks, since the co-evolved router groups inputs by shared behaviors rather than by datasets (Table 7). We will move this comparison to the main text.

W2. Multi-step agent tasks

Thanks for the suggestion. We agree that multi-step agent tasks are an important setting. We additionally evaluate EEVEE on ALFWorld (Shridhar et al., 2021), as suggested by the reviewer. Following its standard text-based setting, the agent completes household goals by issuing text actions (e.g., "go to countertop 1", "take apple 1") for up to 50 steps, and the only feedback is whether the goal is completed. We use Qwen3-4B-Instruct as in our main experiments, with an adaptation pool of 40 episodes per task type and 4 prompt slots for four task types: Look (find an object and examine it under a desk lamp), Clean (clean an object in the sink and place it), Heat (heat an object with the microwave and place it) and Cool (cool an object with the fridge and place it). We report success rates on the unseen validation split.

MethodLookCleanHeatCoolAvg.
No adaptation9.2611.8323.1917.4615.43
EEVEE42.5946.2442.0330.1640.25

EEVEE improves the average success rate from 15.43 to 40.25, with gains on all four task types, showing that it also applies to multi-step agent tasks. We will add these results to the revised paper.

W3. Adaptation cost

Thanks for pointing this out. We agree that the one-off adaptation cost should be weighed, and we will clarify that our efficiency claim mainly refers to inference cost. Even when adaptation cost is included, EEVEE remains cost-effective under both small and large adaptation budgets.

BudgetMethodAdaptation tokensAvg.
SmallACE58.36M34.92
EEVEE, 200 examples37.90M–59.46M48.59
LargeACE, 3 epochs162.0M40.45
GEPA226.81M37.73
EEVEE154.71M51.75

With a small budget, EEVEE uses adaptation tokens comparable to ACE and achieves 13.67 points higher. With a large budget, EEVEE uses fewer adaptation tokens than both ACE with 3 epochs and GEPA, and achieves 11.30 and 14.02 points higher, respectively. We will add this analysis to the revised paper.

AuthorsOpenReview ↗

Response to Reviewer JNj9 (2/2): Questions

Q1. Co-evolution versus prompt-only evolution under a fixed router

Thanks for the question. As shown in W1, under the same budget and the same number of prompt slots, prompt-only evolution with a fixed oracle or learned router reaches at most 47.98, while EEVEE reaches 51.75 and outperforms the oracle on all four benchmarks. The gains are significant under a paired bootstrap over test questions, with 95% CIs of [2.40, 6.79] over the oracle router and [1.29, 6.30] over the learned classifier.

Q2. Multi-step agent tasks

Thanks for the question. As shown in W2, on the multi-step agent benchmark ALFWorld, where the only feedback is task success, EEVEE improves the average success rate from 15.43 to 40.25, with gains on all four task types and the largest gains on Clean (+34.41) and Look (+33.33).

Q3. Adaptation cost

Thanks for the question. As shown in W3, EEVEE uses comparable or fewer adaptation tokens than ACE and GEPA under both small and large budgets, while achieving 11.30–14.02 points higher accuracy. For example, with a comparable adaptation budget, EEVEE (154.71M tokens) outperforms ACE with 3 epochs (162.0M tokens) by 11.30 points.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer JNj9,

Thank you again for your time and constructive feedback. In our responses, we clarified the gain of co-evolution over prompt-only evolution with a fixed router, added an evaluation on ALFWorld as suggested, and analyzed the adaptation cost under matched budgets. As the discussion period is coming to a close, we would like to check whether our responses have addressed your concerns. We would be happy to provide any further clarification.

Best regards,
The Authors

Reviewer_vBwGOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

EEVEE alternates between evolving a router and evolving the prompts assigned to its slots. It first builds complementary prompts, explores router–prompt combinations, then fixes the router for further prompt optimization. On four mixed benchmarks, it improves the average score over the two base models by 10.38 and 24.32 points. The appendices examine routing alternatives, transfer, costs, and online use.

Critical strengths
  1. The dataset-routing control is important. A classifier recovers dataset identity perfectly, but combining it with independently optimized GEPA prompts scores 48.41, below EEVEE's 51.75. The no-co-evolution result is also much worse. This gives some evidence that the learned partition is doing more than recognizing benchmark formats.
  2. The authors have covered many obvious concerns in the appendix: run variability, adaptation tokens, a higher-budget ACE run, imperfect feedback, and test-then-adapt evaluation. They also show where adaptation hurts, particularly GPQA, rather than reporting only the large code-generation gains.
Critical weaknesses
  1. The comparison I care about most is the simpler classifier-plus-GEPA system, and that comparison uses different repetition protocols. Table 6 reuses two optimization runs with one test evaluation each; EEVEE follows the main three-run/repeated-evaluation protocol. Its margin here is 3.34 points, much smaller than against the mixed single-context baselines. A matched comparison is needed to judge whether the extra router optimization pays off reliably.
  2. The deployment claim should remain fairly narrow. The main results use a prepared adaptation split from four benchmarks with correctness feedback. The online appendix is useful, but gives only ACE as a reference, so it does not show the net improvement over the unadapted model on that stream. On held-out MBPP/MMLU-Pro, the average drops from 70.01 to 69.67. Together with the GPQA regressions, this leaves adaptation to new task types less well supported than specialization within the tested mixture.
Questions for authors
  1. Can the existing runs give a matched-seed comparison with classifier plus single-dataset GEPA, including uncertainty and adaptation tokens? Please distinguish optimization runs from repeated answer sampling.
  2. What does the unadapted model score on the exact first-encounter stream in Table 14, overall and per task?
4 comments
AuthorsOpenReview ↗

Response to Reviewer vBwG (1/2): Weaknesses

We thank reviewer vBwG for recognizing our routing controls and the comprehensive analyses in our appendix.

W1. Matched comparison with classifier + single-dataset GEPA

Thanks for raising this point. Since this is a supplementary comparison rather than a main experiment, we originally ran it twice and evaluated each prompt once to save cost. Here we additionally run a third optimization for each dataset and evaluate every prompt three times, matching the protocol of EEVEE (3 runs × 3 evaluations). All other settings were kept unchanged.

MethodGPQAFormulaTheoremQAHumanEvalAvg.
Classifier + single-dataset GEPA55.11 ± 1.3645.61 ± 2.6925.98 ± 6.5666.80 ± 7.3248.37 ± 1.40
EEVEE54.55 ± 2.0554.55 ± 1.1125.27 ± 4.0972.63 ± 11.1951.75 ± 1.62

Under the matched protocol, EEVEE still outperforms this baseline by 3.38 points, consistent with the 3.34-point gap in the paper. The gains come from Formula (+8.94) and HumanEval (+5.83), while the two methods are comparable on GPQA and TheoremQA. We will update Table 6 and the corresponding text with these results.

W2. Scope of the deployment claim

Thanks for the comment. Adapting from a prepared set of task examples with correctness feedback is the setting adopted by most prior work, such as OPRO (Yang et al., 2024), TextGrad (Yuksekgonul et al., 2024), GEPA (Agrawal et al., 2026) and ACE (Zhang et al., 2026), and these works make similar adaptation claims under this setting. Our deployment claim follows the same convention, and we will also refine its wording in the abstract and introduction. For the online stream, we add the unadapted model on the same stream in Q2, where EEVEE also improves over it by about 4 points.

To show that EEVEE is not limited to the tested benchmarks, we additionally scale the mixture to 8 tasks by adding IFBench (Pyatkin et al., 2025), CRUXEval (Gu et al., 2024), FinQA (Chen et al., 2021), and BBH (Suzgun et al., 2023) or LegalBench (Guha et al., 2023), with 8 slots and a budget scaled with the number of tasks. We further reduce the number of slots from 8 to 4 to examine whether the performance depends on the number of slots.

MixtureSlotsNo adaptationEEVEEGain
8-task (BBH)852.1859.62+7.44
452.0259.82+7.80
8-task (LegalBench)852.0959.99+7.90
451.4956.61+5.12

EEVEE improves the 8-task average by 7.44 and 7.90 points with 8 slots, so its gains are not specific to the original task mixture. The gains remain similar with 4 slots, showing that when scaling up the number of tasks, the number of slots has little effect on EEVEE.

In other adaptation dimensions:

  • (a) Unseen model families. Prompts learned on Qwen3 transfer to DeepSeek-V3.2 and improve it from 39.75 to 54.10 (Table 3).

  • (b) Multi-step agent tasks. Beyond single-turn benchmarks, we additionally evaluate EEVEE on ALFWorld (Shridhar et al., 2021), a text-based household environment where the agent completes a goal by issuing text actions (e.g., "go to countertop 1", "take apple 1") for up to 50 steps, and the only feedback is task success. We use Qwen3-4B-Instruct as in our main experiments, with an adaptation pool of 40 episodes per task type and 4 prompt slots for four task types: Look (find an object and examine it under a desk lamp), Clean (clean an object in the sink and place it), Heat (heat an object with the microwave and place it) and Cool (cool an object with the fridge and place it). We report success rates on the unseen validation split. EEVEE improves the average success rate from 15.43 to 40.25, with gains on all four task types.

MethodLookCleanHeatCoolAvg.
No adaptation9.2611.8323.1917.4615.43
EEVEE42.5946.2442.0330.1640.25
  • (c) Unseen tasks. EEVEE improves on MBPP (69.29 → 70.42), where GEPA and ACE both drop, while the drop on MMLU-Pro (70.74 → 68.92) is consistent with the knowledge-intensive GPQA results analyzed in the paper (Table 3). Such regressions can also be largely avoided at inference: applying our harm-aware gate (Appendix B.4) to the main benchmarks recovers GPQA to 55.78, close to the unadapted 56.00, while keeping the average at 51.85. Moreover, prompt adaptation methods such as OPRO, GEPA and ACE also optimize and evaluate on the target tasks rather than on unseen tasks, and we believe that in practice adaptation is run on the full target workload instead of producing a general-purpose model.

We will clarify this scope in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer vBwG (2/2): Questions

Q1. Uncertainty and adaptation tokens

Thanks for the question. Under the matched protocol (see W1), EEVEE outperforms classifier + single-dataset GEPA by 3.38 points (51.75 ± 1.62 vs. 48.37 ± 1.40). The table below separates the variance across optimization runs from that across repeated answer sampling, and reports the adaptation tokens.

Classifier + single-dataset GEPAEEVEE
Std across 3 optimization runs1.401.62
Std across 3 repeated samplings0.530.76
Adaptation tokens192.43M154.71M

For the 3.38-point gap, a paired bootstrap over test questions gives a 95% CI of [1.21, 5.55]. At the run level, all three EEVEE runs (50.08–53.31) outperform all three runs of classifier + single-dataset GEPA (47.47–49.99). Classifier + single-dataset GEPA also uses 24% more adaptation tokens, since it optimizes a separate prompt for each dataset.

Q2. Unadapted model on the online stream

Thanks for the question. We additionally evaluate the unadapted model on the exact first-encounter stream in Table 14, with the same 2,162 questions in the same order and three samples per question.

MethodGPQAFormulaTheoremQAHumanEvalAvg.
No adaptation57.7443.0715.9242.4839.80
ACE online29.1242.2316.9228.8629.28
EEVEE cumulative51.0150.6724.7147.9743.59
EEVEE new-window50.0052.9523.4449.4943.97

EEVEE also achieves a clear improvement over the unadapted model in the online setting by 3.79 (cumulative) and 4.17 (new-window) points, with consistent gains on Formula, TheoremQA and HumanEval. The GPQA drop is common among prompt learning methods on knowledge-intensive tasks, as analyzed in the paper (Tables 1 and 4), and EEVEE still clearly outperforms ACE online on GPQA (51.01 and 50.00 vs. 29.12). We will add these results to Table 14.

Thank you for the matched classifier-plus-GEPA comparison and the unadapted baseline on the same online stream. These address my main concerns. The GPQA regressions and mixed transfer results remain limitations, but the evidence supports the method in the evaluated setting. I maintain my rating of 3.

We sincerely thank you for your careful review and for acknowledging that our responses address your main concerns. We agree that the GPQA regressions and the mixed transfer results remain a limitation. As discussed in our paper, this is a common challenge for existing prompt learning methods on knowledge-intensive tasks, and EEVEE already shows clearly smaller regressions than GEPA and ACE (Tables 1 and 3). Moreover, the simple inference-time gate in our paper (Appendix B.4) can further mitigate such regressions, recovering GPQA to 55.78, close to the unadapted 56.00, while keeping the gains on the other benchmarks. We will make this limitation and its mitigation clearer in the revised paper. Thank you again for your valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.