acceptodds
Under review as a conference paper at ICLR 2027

ReHarness: Learning Reusable Harness Evolution Across Tasks

Abstract

Large Language Model (LLM) agents increasingly rely on programmable harnesses to coordinate reasoning, tool use, and execution. Existing harness evolution methods typically optimize a task-specific harness from scratch for each new task, requiring substantial task-specific data and evolution steps. In this work, we identify that harness evolution exhibits reusable structure across heterogeneous tasks: independently evolved harnesses repeatedly discover similar improvement strategies, although the resulting harnesses remain task-specific. Building on this insight, we propose ReHarness, a reusable harness evolution framework that learns how to evolve across tasks. ReHarness learns a transferable evolution strategy from multiple offline-given tasks and applies it to rapidly adapt task-specific worker harnesses to new tasks using only a few examples. Experiments across four LLMs and six benchmarks demonstrate that ReHarness consistently improves agent performance on four new tasks not used during the offline phase, while outperforming the task-specific evolution method Self-Harness on two tasks from the offline suite under the same data budget. Code and models will be made publicly available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, harness self-evolving, agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xVdfOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

ReHarness learns a reusable harness adaptation strategy through cross-task optimization. A learned adapter modifies task-specific worker harnesses using five support examples. The paper reports consistent improvements across four models, but insufficient runtime disclosure and unusually weak baselines make the significance of these gains difficult to assess.

Critical strengths
  • Separating reusable adaptation strategies from task-specific harnesses is well motivated, and the support/query formulation encourages generalization.
  • Improvements are consistent across the reported model-benchmark combinations.
  • The five-shot, initial-adapter, and history ablations provide useful evidence for the contribution of offline optimization.
Critical weaknesses
  • My primary concern is the unexplained baseline underperformance. DeepSeek-V4-Flash scores 26.7% initially and 33.3% after ReHarness on SWE-bench Verified, compared with 73.7–79.0% in its official model card; similar things happen to other models, such as GLM 5.3 Flash and GPT 5.6 Sol. In fact, most of the frontier models score around 80% on Swe-Bench verified. Although protocols may differ, the paper does not disclose evaluation settings to reconcile this gap. The gains may reflect recovery from an inadequate initial harness; no specific misconfiguration can be established from the available information.
  • The initialization ablation only considers harnesses constrained to remain simple and general. The method’s advantage over competitive agent systems is unclear.
  • The diversity ablation increases source-task count without matching total data and computation. So it does not isolate the benefit of task diversity. An adapter trained with single task under matched resources would also help establish whether cross-task learning is necessary.
  • The paper does not clearly show what the adapter learns. Several changes in the case study involve answer extraction and formatting, but their contribution to the gains is not measured. The authors should separate these changes from improvements in reasoning and task execution.
Questions for authors
  • Can you provide the complete initial and evolved harnesses, prompts, tools, reasoning settings, token budgets, execution limits?
  • What explains the baseline performance gap?
  • Does cross-task diversity still help when total data and computation are fixed, including a single-task trained adapter control?
7 comments
AuthorsOpenReview ↗

Response to Weakness 1 & Question 1

To Weakness 1 & Question 1 (Configuration Details):

Thank you for your comments. We would like to provide our detailed experimental configuration on SWE-bench Verified evaluation. And we have incorporated all these details into Appendix D.9 in our rebuttal revision, with changes highlighted in $ \textcolor{blue}{blue}$.

Initial and evolved harnesses. The initial worker harness is a minimal and general worker scaffold with six functions: repository inspection, shell command execution, observation parsing and message construction, file modification through shell commands, focused test execution, and final patch extraction from the workspace. The patch is collected from git diff against the initial repository HEAD. We use this minimal and general harness to measure the adapter agent's ability to modify the worker harness more directly. The evolved harness preserves the same interface and execution protocol, but changes the worker agent’s behavior in four ways. It adds an explicit workflow of inspection, reproduction, editing, and testing, instructs the worker agent to make the smallest repository change and rerun focused tests, requires the final response to contain an applicable unified diff, and normalizes and extracts the diff before submission.

Prompts. The worker harness prompt is built from our general worker prompt, with the official SWE-bench prompt content appended. The general part describes the basic worker agent’s role and the interaction protocol. The appended official content specifies the SWE-bench task statement, the testbed working directory, the expected patch output, and the prohibition on committing or resetting the repository. The detailed prompts have been provided in Appendix D.9.

Tools. The worker agent receives a shell interface inside the E2B sandbox. Repository reading, searching, editing, and test execution are all performed through this interface.

Reasoning settings. We use a medium reasoning effort for GPT-5.6-Sol and Qwen3.8-Flash, and a high reasoning effort for DeepSeek-V4-Flash and GLM-5.3-Flash. All four settings are within each model’s supported range. The model endpoint and exact model revision are fixed by the evaluation configuration and recorded in the run artifacts. For sampling, each model runs with its API-native configuration. GPT-5.6 Sol uses its API-default sampling behavior because it doesn’t accept temperature or top-p. DeepSeek-V4-Flash runs in thinking mode with temperature ignored and top-p at its default of 1.0. Qwen3.8-Flash and GLM-5.3-Flash run in thinking mode with temperature=1.0 and top-p=0.95.

Token budget. Each model call has a maximum generation budget of 98304 tokens. The budget includes the complete response generated for one worker agent’s step.

Execution limits. Each instance runs in an isolated E2B sandbox with 8 CPUs and 8 GB memory. The worker agent’s loop allows at most 250 interaction steps. Individual shell commands are limited to 120 seconds, the worker sandbox wall-clock timeout is 3600 seconds, and official patch grading uses a separate 600-second timeout, mainly following the recommended configuration in AgentCompass’s SWE-bench evaluation.

If you would like any further details, please let us know and we will be glad to provide them.

AuthorsOpenReview ↗

Response to Weakness 1 & Weakness 2 & Question 2

To Weakness 1 & Weakness 2 & Question 2 (Reasons for the Gap and Additional Experiments):

Thank you for your comments. In the following, we would like to discuss the reasons for the performance gap and provide experiments under the mainstream SWE-bench evaluation setting to verify the method's advantage over competitive agent systems. Specifically, we have revised three parts of the main text in our rebuttal revision with changes highlighted in $ \textcolor{blue}{blue}$: (1) the first paragraph of Section 4, (2) a new paragraph in Section 4.1 (Main Results), and (3) Table 3. The detailed experimental configuration is provided in the newly added Appendix D.10.

Reasons for the performance gap. As discussed in our response to Weakness 1 & Question 1, we use a general worker harness with the necessary adaptation interfaces, following the design of Meta-Harness, rather than the mini-SWE-agent configuration used in the official SWE-bench evaluation, to measure the adapter agent's ability to modify the worker harness more directly. Our setting therefore differs from the model-card setting in agent implementation. This is the main source of the performance gap.

Evaluation under the official mini-SWE-agent configuration. To test whether the method remains effective on a stronger and widely used agent system, we additionally evaluate it on the mini-SWE-agent protocol. This setting uses the upstream DefaultAgent, LiteLLM model, and Swe-bench agent prompt, one fresh E2B sandbox per instance, the same SWE-bench Verified data and official E2B grader, 250 maximum interaction steps, 98304 maximum generated tokens, a 60-second per-command timeout, a 2400-second E2B sandbox timeout, and patch extraction from git diff. To adapt our method to this protocol, we retain the tool loop, prompt, and execution environment unchanged, and attach the general worker harness as an additive layer to the agent's task and context construction. Then we re-evaluate all four models used in our experiments under this official mini-SWE-agent configuration. The results are shown in the table below.

MethodDeepSeek-V4-FlashGLM-5.3-FlashQwen3.8-FlashGPT-5.6 SolAvg.
Baseline72.783.077.084.979.4
ReHarness78.2 (+5.5)85.5 (+2.5)80.6 (+3.6)88.5 (+3.6)83.2 (+3.8)

Under this official protocol, the baseline reaches 72.7%, 83.0%, 77.0%, and 84.9% on the four models, which recovers to almost the same range as the scores reported in the model cards and public leaderboards. This confirms that the earlier performance gap is mainly due to different evaluation protocols. As shown in the table above, ReHarness improves over this stronger baseline by an average gain of 3.8%. These results show that ReHarness remains effective on a strong and widely used agent system.

AuthorsOpenReview ↗

Response to Weakness 3 & Question 3

To Weakness 3 & Question 3 (Task Diversity Experiments under Fixed Data Budget):

Thank you for your comments. To isolate the effect of task diversity from that of the data budget and computation, we add an ablation in which both the total amount of offline data and the total number of task-level adaptations are fixed. All the analyses and experiments have been added in a new paragraph following Appendix C.4 in our rebuttal revision, with changes highlighted in $ \textcolor{blue}{blue}$. We keep the support and query configuration within each batch unchanged and keep P=3P=3 sequential proposals per batch, and vary only the number of given tasks and the number of batches per task. In this ablation, the multi-task settings use the same task types as our earlier diversity analysis, while the single-task control is trained separately on SpreadsheetBench and BFCL V4.

Given tasksBatchesSize of support setSize of query setTotal data
11251512 ×\times 1 ×\times (5 + 15) = 240
265156 ×\times 2 ×\times (5 + 15) = 240
345154 ×\times 3 ×\times (5 + 15) = 240
435153 ×\times 4 ×\times (5 + 15) = 240

This keeps the total offline data constant. Under this setting, we evaluate them on three new tasks not used in the offline phase, GAIA, τ2\tau^2-Bench, and OfficeQA-Pro. The results are shown in the table below.

SettingGAIAτ2\tau^2-BenchOfficeQA-ProAvg.
Baseline28.779.536.748.3
1 task (SpreadsheetBench)21.676.211.236.3
1 task (BFCL V4)29.876.832.146.2
2 tasks30.685.537.351.1
3 tasks33.683.541.452.8
4 tasks34.186.240.453.6

With the total data budget fixed, increasing the number of given tasks still improves the average performance on new tasks. GAIA improves monotonically across all settings, while τ2\tau^2-Bench and OfficeQA-Pro show non-monotonic variations, consistent with the observation in Appendix C.4. The gain from 2 to 4 tasks is smaller than in the original Appendix C.4, indicating that part of the gain reported in Appendix C.4 came from the larger data budget. However, the direction of the task diversity effect remains the same, as more given tasks still lead to better average performance when the total data budget is held constant.

More importantly, an adapter harness evolved on a single task under the same budget performs worse than both the baseline and all multi-task settings. A single task provides too narrow a signal to capture a reusable evolution strategy, so the resulting adapter harness overfits to task-specific behavior and transfers poorly. This negative transfer is highly dependent on the nature of the source task. The adapter harness optimized on SpreadsheetBench reaches only 11.2% on OfficeQA-Pro, as it learns spreadsheet-specific calculation rules that mislead on document-based numerical QA. In contrast, the adapter harness optimized on BFCL V4 reaches 29.8% on GAIA, slightly above the baseline, likely because BFCL V4 involves tool-calling behaviors that remain useful for GAIA's search-tool use. These disparities indicate that a single-task adapter captures task-specific characteristics rather than a general reusable strategy, further supporting the necessity of cross-task evolution.

Reviewer_2MFhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work first has an interesting observation that heterogeneous tasks could share some common basic harness. Based on this observation, it proposes a framework to learn a transferable method for improving large language model (LLM) agent harnesses. The proposed framework has three agents: a worker agent that solves downstream tasks using harness, an adapter agent that modifies the worker’s harness, and an optimizer agent that improves the adapter’s harness offline. Experimental results across multiple datasets and backbones show the proposed method can effective learn reusable harness.

Critical strengths

This submission has the following Critical Strengths:

  • Its observation that reusable harness exists across different tasks is interesting. This makes the motivation of this work very clear.
  • The proposed solution framework is reasonable. It optimizes the adapter across multiple tasks (offline stage) and then freezes it during deployment (online stage). This provides a coherent formulation of reusable harness evolution and this work is easy to understand.
  • The comprehensive experimental results validate the effectiveness in learning reusable harness.
Critical weaknesses

This submission has the following Critical Weaknesses:

  • This work is interesting. But the current discussion in Reusable Evolution Strategies in Appendix B2 is not enough. For example, Hyperagents [1] already demonstrates transferable improvements. A more detailed comparison should be included to clarify the novelty of this work. Furthermore, the discussion of related work is entirely in the appendix, with little discussion of relevant prior studies in the main text. This might make readers wrongly understand its novelty and contributions.
  • Self-Harness is evaluated only on offline benchmarks. Comparisons with established adaptation methods on unseen tasks under matched data and execution budgets are needed.

Reference
[1] Zhang, Jenny, et al. "Hyperagents." arXiv preprint arXiv:2603.19461 (2026).

Questions for authors

I have the following questions and suggestions:

  • It would be much better to move the introduction and discussion of some closely relevant works to the main text.
  • Would it be possible for the authors to conduct some experiments based on the weakness 2?
  • Could the authors clarify how the averages in Table 6 are computed? They appear inconsistent with the task-wise averaging defined.
  • There are some typos:
    • In ~line 1163, 'an fair comparison' -> 'a fair comparison'.
    • The Table 4's caption is not very coherent: 'with and without H_{opt} on four new tasks without using in the offline'
5 comments
AuthorsOpenReview ↗

Response to Weakness 1 & Question 1

To Weakness 1 & Question 1 (Related Work Additions and Hyperagents Comparison):

Thank you for your helpful suggestions. We will discuss this related work in detail and provide corresponding experiments below. All the additional experiments and discussions have been added into our updated submission.

Related Work Additions. In the Introduction, we primarily distinguish our method from prior harness evolution methods (Meta-Harness and Self-Harness), since they are the most directly related work to us. Hyperagents is related along a different dimension and it also recognizes the need to improve the improvement mechanism itself. Following your suggestion, we have added a new Related Work section (Section 2.1) in the main text, where we discuss the most relevant prior work, including Hyperagents, and clarify how it differs from ReHarness and where our novelty lies. The detailed analysis and the matched budget experiments are provided in Appendix C.5 in our updated submission, and we elaborate on these differences and experiments below.

Differences between Hyperagents and ReHarness. Hyperagents extends the Darwin Gödel Machine (DGM) by integrating a task agent and a meta agent into one editable program, so the meta-level modification procedure itself can be modified. In cross-domain transfer experiments, it freezes the evolved meta agent and runs improvement@k (k = 50) on the target domain, showing cross-domain transfer capability. However, several aspects of Hyperagents place it at a disadvantage relative to ReHarness for few-shot adaptation:

  • Model configuration. Hyperagents uses different models for meta-level modification and task execution (e.g., meta = Claude 4.5 Sonnet, task = GPT-4o), whereas ReHarness uses the same model across the three agents. ReHarness is therefore model-specific and leverages the model's own capability, while Hyperagents' performance may be affected by the choice of the meta model.
  • Online budget. Hyperagents' transfer uses improvement@50 (50 iterations) instead of ReHarness's 5-shot budget. Its online adaptation therefore needs far more samples and steps and is less efficient than ReHarness.
  • Online adaptation mechanism. Hyperagents' transfer is a validation-based search: it generates 50 task agents and selects the best using target-domain validation. ReHarness instead directly modifies the worker harness from 5 support examples through its fixed adapter agent, without any candidate pool or target-domain validation, which makes it more suitable for few-shot adaptation.

Comparison between Hyperagents and ReHarness under a matched budget. To match ReHarness's budget, we make the following modifications to Hyperagents. We divide the offline phase into 10 stages, each corresponding to one ReHarness batch. In each stage, Hyperagents generates and evaluates exactly 3 candidate hyperagents, and each candidate is evaluated on the same 4 offline tasks with 5 support examples and 15 query examples used only for parent selection. This gives 3 $ \times $ 4 = 12 task-level adaptations per stage, matching ReHarness's 3 proposals ×\times 4 tasks, and 50 support and 150 query examples in total across 10 stages, matching ReHarness. Parent selection follows Hyperagents' original mechanism, which samples parents from the archive based on their performance and the number of their compiled children. All experiments, for both ReHarness and Hyperagents, use DeepSeek-V4-Flash as the underlying LLM, with the meta and task agents sharing the same model, so that model type doesn’t affect the comparison. After the offline phase, we hold the learned meta agent fixed and use it to generate new task agents on each unseen task from 5 support examples, with at most Kmax=5K_\text{max}=5 generation steps, and report the best generated task agent.

MethodGAIAτ2\tau^2-BenchOfficeQA-ProAvg.
Baseline28.779.536.748.3
ReHarness33.6 (+4.9)84.5 (+5.0)45.3 (+8.6)54.5 (+6.2)
Hyperagents28.7 (+0.0)80.2 (+0.7)40.4 (+3.7)49.8 (+1.5)

As shown in the table, Hyperagents improves over the baseline by 1.5% on average, clearly less than the 6.2% for ReHarness. This gap indicates that Hyperagents' transfer capability relies on a relatively large amount of online data and search. When we constrain it to the same 5-example budget as ReHarness, its transfer to new tasks largely degrades, and on most tasks it shows little to no transfer.

AuthorsOpenReview ↗

Response to Weakness 2 & Question 2

To Weakness 2 & Question 2 (Matched Budget Self-Harness Comparison on Unseen Tasks):

Thank you for your comments. We would like to clarify that, as discussed in Appendix D.7, Self-Harness is designed for task-specific evolution using substantial data, and it has no mechanism designed for cross-task transfer. It therefore doesn’t directly support evaluation on unseen tasks under a budget matched to ReHarness. For this reason, as in Table 2, we compare Self-Harness with ReHarness only in the offline phase under matched budgets, where Self-Harness has sufficient task-specific data to evolve. The fact that Self-Harness cannot directly operate on unseen tasks actually exposes the inherent limitation of such task-specific harness evolution paradigms. Self-Harness spends a large amount of task data to obtain a harness that is highly specialized and doesn’t transfer. In contrast, after the offline phase, ReHarness can be effectively applied to unseen tasks with only a few examples.

Cross-task experiments using Self-Harness. To assess the cross-task transferability, we run Self-Harness on 4-task offline setting and adapt the resulting harness to unseen tasks. Specifically, we alternate Self-Harness across BFCL V4, MuSR, SpreadsheetBench, and SearchQA. For each task, we run 10 evolution batches. In each batch, Self-Harness uses 5 held-in examples to identify failure patterns and propose candidate harness modifications, and validates each candidate against 15 held-out examples as a regression gate, following its default configuration protocol. The harness obtained from one task is carried as the initial harness for the next, under the same data and evolution budget as ReHarness.

After this alternating evolution, we apply the resulting harness to each unseen task with only 5 examples. Since Self-Harness requires separate held-in and held-out examples, we use the same 5 examples for both, run 5 evolution iterations, and take the final harness as the adapted one, matching ReHarness's online budget Kmax⁡=5K_{\max}=5. Results are as follows.

MethodGAIAτ2\tau^2-BenchOfficeQA-ProAvg.
Baseline28.779.536.748.3
ReHarness33.6 (+4.9)84.5 (+5.0)45.3 (+8.6)54.5 (+6.2)
Self-Harness28.4 (−0.3)81.3 (+1.8)37.8 (+1.1)49.2 (+0.9)

The gains from Self-Harness are small and inconsistent. It improves τ2\tau^2-Bench by 1.8% and OfficeQA-Pro by 1.1%, but slightly decreases GAIA by 0.3%, resulting in an average gain of only 0.9%, compared with 6.2% for ReHarness. These results further demonstrate the limitation of task-specific harness evolution methods such as Self-Harness. They don’t accumulate general experience about how to evolve a harness, and thus require substantial task-specific data to evolve from scratch for each new task. In contrast, ReHarness extracts a reusable harness evolution strategy from offline cross-task evolution and directly applies it to new tasks, leading to good generalization across unseen tasks.

All of the above discussions and experiments have been supplemented in Appendix D.8 in the updated submission and hope these can address your concerns.

AuthorsOpenReview ↗

Response to Weakness 3 & Weakness 4

To Weakness 3 (Clarification on Table 6 Averages):

Thank you for pointing this out. We indeed compute the averages using task-wise averaging. The correct value is 0.60575, not 0.5185,and we apologize for this typo. Since the average score of Δ\Delta is computed as selected average score − parent average score, it is affected as well. The correct value of Δ\Delta is 0.09425, not 0.1815. We have updated Table 6 in our updated submission accordingly.

To Weakness 4 (Typo Fixes and Caption Revision):

Thank you for pointing out these typos. In our updated submission, we have corrected “an fair comparison” to “a fair comparison”. We have also revised the Table 4 caption from “We compare ReHarness with and without $ H_\text{opt} $ on four new tasks without using in the offline phase” to “We compare ReHarness with and without $ H_\text{opt} $ on four new tasks not used during the offline phase.”

If you have more questions or suggestions, please let us know, and we would be happy to have further discussions.

Reviewer_5pcGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

ReHarness learns an editable adapter harness that improves task-specific worker harnesses, using an optimizer to refine the adapter across offline task batches while keeping the underlying language model frozen. At deployment, the learned adapter is fixed and adapts a new worker harness from five support examples, with reported gains across four unseen benchmarks and four models and additional comparisons with Self-Harness on two offline-suite benchmarks. Ablations support a role for cross-task adapter learning and retained optimization history, but strong adaptation baselines on the unseen tasks and a complete account of offline and online cost are still needed.

Critical strengths
  • The paper addresses the repeated cost of rediscovering harness modifications for every new task. Reusing a learned adaptation procedure could make agent customization more practical when a new task supplies only a few examples, rather than requiring a fresh full search. The distinction between transferring a finished worker harness and transferring the procedure that modifies it is a clear, useful research question.

  • An adapter harness changes the worker harness using feedback from a small support set. This makes the transferable object an executable procedure for adapting an agent, rather than a fixed collection of task-specific edits.

  • The paper varies support/query sizes, worker initialization, adaptation attempts, cross-batch history, and offline task diversity. Especially informative are the DeepSeek averages. These controls support a contribution from learning the adapter beyond merely adding examples or running an unlearned edit loop.

Critical weaknesses
  • A pseudocode defect occurs in Algorithms 2 and 3. The adapter produces a stop flag indexed by the current iteration, then the iteration counter is incremented before that indexed flag is tested. As written, the test refers to the next index rather than the flag just produced. Please move the stop test before the increment or rename the returned flag.

  • More baselines are expected. It would be better to make a cross-task comparison with Hyperagents[1] or MCE[2] and a matched-budget reflective prompt optimizer on the unseen benchmarks.

  • The unseen-benchmark evaluation would be stronger with a competitive few-shot adaptation baseline. Table 1 primarily compares with the standard worker, while the Self-Harness comparison uses two offline-suite benchmarks under a different evolution protocol. The initial-adapter ablation supports the benefit of cross-task training, but leaves unclear whether the learned adapter outperforms a strong generic reflective strategy.

  • Appendix C.4 increases the offline task set from two to four while keeping the other settings fixed. Because every task supplies support and query examples, this can increase the amount of feedback and computation as well as diversity, so the observed gains do not isolate diversity itself.

[1]. https://arxiv.org/abs/2603.19461 [2]. https://arxiv.org/abs/2601.21557

Questions for authors

NA

2 comments
AuthorsOpenReview ↗

Response to Weakness 1 & Weakness 2

To Weakness 1 (Pseudocode Correction):

Thank you for pointing this out. In our updated submission, we have removed the iteration index from the stop flag and renamed it to the variable ss in both Algorithm 2 and Algorithm 3.

To Weakness 2 (Cross-Task Baseline Analysis and Matched-Budget Experiments)

Thank you for your helpful suggestions. We have added a detailed discussion of three representative baselines in Appendix C.5 in our rebuttal revision with changes highlighted in $ \textcolor{blue}{blue}$: Hyperagents, MCE, and GEPA (a representative reflective prompt optimization method accepted as ICLR 2026 Oral). Because they differ from ReHarness in their cross-task reusability and online adaptation protocols, we first summarize their key characteristics in the table below. We then provide an analysis of their applicability and conduct the corresponding experiments across these methods.

MethodTypeCross-task reusable strategyUnderlying LLMs across agentsOnline new task transfer cost
ReHarnessHarness evolvingYesSameSmall (5 support, no validation)
HyperagentsAgent evolvingYesDifferentLarge (50 generations and target-domain validation)
MCEContext-engineering skills evolvingNoDifferentNot directly applicable
GEPAPrompts evolvingNoSameNot directly applicable

Hyperagents. Hyperagents makes meta-level modification editable and transferable by freezing the meta agent and running improvement@50 on the target domain. It differs from ReHarness in three settings. (1) Model: Hyperagents uses different models for meta modification and task execution, while ReHarness uses the same model across agents. (2) Online budget: Hyperagents uses 50 iterations for online adaptation and ReHarness uses a 5-shot budget. (3) Adaptation mechanism: Hyperagents performs validation-based search over 50 generated task agents while ReHarness directly modifies the worker harness from 5 support examples without a candidate pool or target-domain validation.

Comparison between Hyperagents and ReHarness under matched budget. We divide Hyperagents' offline phase into 10 stages, each matching one ReHarness batch. Each stage generates and evaluates 3 candidate hyperagents on the same 4 offline tasks with 5 support and 15 query examples used only for parent selection, giving 3 × 4 = 12 task-level adaptations per stage and 50 support and 150 query examples in total, matching ReHarness. All experiments use DeepSeek-V4-Flash for both ReHarness and Hyperagents, with meta and task agents sharing the same model. After offline, we hold the learned meta agent fixed and generate new task agents on each unseen task from 5 support examples, with at most Kmax=5K_\text{max}=5 generation steps, and report the best generated task agent.

MethodGAIAτ2\tau^2-BenchOfficeQA-ProAvg.
Baseline28.779.536.748.3
ReHarness33.6 (+4.9)84.5 (+5.0)45.3 (+8.6)54.5 (+6.2)
Hyperagents28.7 (+0.0)80.2 (+0.7)40.4 (+3.7)49.8 (+1.5)

As shown in the table, Hyperagents improves over the baseline by 1.5% on average, clearly less than the 6.2% for ReHarness. This gap indicates that Hyperagents' transfer capability relies on a relatively large amount of online data and search. When we constrain it to the same 5-example budget as ReHarness, its transfer to new tasks largely degrades, and on most tasks it shows little to no transfer.

MCE cannot be generalized to unseen tasks. MCE is a bi-level framework with a meta-level agent evolving context-engineering skills and a base-level agent executing them to optimize context as files and code. However, it evolves skills separately for each task: its skill database is per-task, and its crossover recombines skills only within the current task's history. Even run across multiple tasks in sequence, MCE has no shared skill pool and no cross-task selection signal, so learned skills remain tied to the task they were evolved on. Moreover, it optimizes context as files and code rather than the full agent harness, giving it a narrower editable surface and more limited generalization to unseen tasks. For these reasons, MCE is not directly comparable in our online evaluation on new tasks.

GEPA cannot be generalized to unseen tasks. GEPA is a reflective prompt optimizer that evolves prompts via generic optimization, natural language reflection on execution traces, and Pareto-based candidate selection. Like MCE, GEPA targets single-task prompt optimization and doesn’t support cross-task transfer. Other representative reflective prompt optimizers, such as APE and EvoPrompt, are also task-specific: they optimize a prompt on a given task using that task's feedback, and don’t learn a reusable adaptation strategy that transfers to unseen tasks. For this reason, we cannot fairly apply these methods to unseen tasks under a matched budget.

AuthorsOpenReview ↗

Response to Weakness 3 & Weakness 4

To Weakness 3 (Competitive Few-Shot Adaptation Experiments on Unseen Tasks):

Thank you for your comments. We use GEPA as a competitive few-shot adaptation baseline and provide the results in Appendix C.5 in our rebuttal revision with changes highlighted in $ \textcolor{blue}{blue}$. We set up a comparison that is more favorable to GEPA. On unseen tasks, we give GEPA 5 training and 15 validation examples to optimize its prompt, while ReHarness adapts from the default setting of 5 support examples. To keep the evaluation fair, we remove these 20 samples from the unseen benchmarks and re-evaluate the baseline, ReHarness, and GEPA on the same remaining samples. This directly compares whether the learned adapter outperforms a strong generic reflective strategy.

MethodGAIAτ2\tau^2-BenchOfficeQA-ProAvg.
Baseline26.978.935.447.1
GEPA31.8 (+4.9)80.0 (+1.1)34.5 (−0.9)48.8 (+1.7)
ReHarness34.9 (+8.0)83.7 (+4.8)46.9 (+11.5)55.2 (+8.1)

As shown in the table, GEPA improves the average by only 1.7%, compared with 8.1% for ReHarness. On OfficeQA-Pro, GEPA is slightly below the baseline, likely because prompt-level reflection cannot effectively optimize output-format and numerical-reasoning constraints that this task requires. These results indicate that the learned adapter harness outperforms a strong generic reflective strategy.

To Weakness 4 (Task Diversity Experiments under Fixed Data Budget):

Thank you for your comments. To isolate the effect of task diversity from that of the data budget, we add an ablation in which the total amount of offline data is fixed. All the analyses and experiments have been added in a new paragraph following Appendix C.4 in our rebuttal revision with changes highlighted in $ \textcolor{blue}{blue}$. We keep the support and query configuration within each batch unchanged and keep P=3P=3 sequential proposals per batch, and vary only the number of given tasks and the number of batches per task:

Given tasksBatchesSize of support setSize of query setTotal data
265156 ×\times 2 ×\times (5 + 15) = 240
345154 ×\times 3 ×\times (5 + 15) = 240
435153 ×\times 4 ×\times (5 + 15) = 240

This keeps the total offline data constant. Under this setting, we evaluate them on three new tasks not used in the offline phase, GAIA, τ2\tau^2-Bench, and OfficeQA-Pro. The results are shown in the table below.

SettingGAIAτ2\tau^2-BenchOfficeQA-ProAvg.
Baseline28.779.536.748.3
2 tasks30.685.537.351.1
3 tasks33.683.541.452.8
4 tasks34.186.240.453.6

With the total data budget fixed, increasing the number of given tasks still improves the average performance on new tasks. GAIA improves monotonically across all settings, while τ2\tau^2-Bench and OfficeQA-Pro show non-monotonic variations. The gain from 2 to 4 tasks is smaller than in the original Appendix C.4, indicating that part of the previous gain came from the larger data budget. However, the direction of the task diversity effect remains the same, as more given tasks still lead to better average performance when the total data budget is held constant.

If you have more questions or suggestions, please let us know, and we would be happy to have further discussions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.