acceptodds
Under review as a conference paper at ICLR 2027

HarnessWeaver: Continual Self-Improvement through Task-Conditioned Harness Composition and Model Routing

Abstract

Continual self-improvement requires agents to use accumulated experience on subsequent tasks, yet failures may arise from limited executor capability or from inadequate planning, tool use, and execution control in the harness. We introduce HarnessWeaver, a two-timescale framework for continual self-improvement through task-conditioned model routing and harness composition without training model parameters. At task time, the Weaver uses accumulated evidence to select an executor and compatible components from a typed Living Pool, favoring low-cost configurations. Between rounds, it proposes model, harness, or joint revisions according to failure mechanisms and uses execution-based validation to admit persistent updates. On ALFWorld and Omni-MATH, frozen pre-adaptation references distinguish first-encounter transfer to new batches from recovery on repeated tasks. HarnessWeaver raises the equal-weight macro-average first-encounter success from 31.5% for the frozen reference to 56.8% (+25.3 points), outperforming the strongest external baseline, ACE at 39.1%, by 17.7 points. In independently rerun ALFWorld controls, Full reaches 58.6% first-encounter success, compared with 54.3% for Harness-only and 48.6% for Model-only, and also achieves the highest final-round success. These results show that reusable model–harness assemblies can serve as units of adaptation that connect accumulated experience, selective revision, and subsequent execution into a continual self-improvement process.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, continual self-improvement, harness composition, model routing, cost-aware adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6oWHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HarnessWeaver, a continual agent-improvement framework that selects an executor and composes task-conditioned harness components, then validates and retains model-routing or harness revisions based on execution feedback. It evaluates first-encounter transfer and repeated-task recovery on ALFWorld and Omni-MATH, reporting improvements over a frozen reference and two external baselines.

Critical strengths

1.Separating first-encounter performance from repeated-task recovery makes the continual-improvement claims more informative than reporting final-round performance alone. 2. The paper includes reasonably clear visualizations, ablations, and cost accounting, which help assess the system’s behavior.

Critical weaknesses
  1. The main ingredients, model routing, compositional harnesses, failure-driven revision, and validation before deployment, are closely related to prior work. The paper does not make sufficiently clear what distinctive mechanism or insight their combination establishes.
  2. The full framework improves results, but Omni-MATH lacks harness-only and model-only controls, while its reported gains appear largely routing-driven. This makes it difficult to assess how much task-conditioned harness composition contributes beyond selecting a stronger executor.
Questions for authors
  1. What is the specific methodological contribution that would remain if model routing and harness adaptation were considered separately?
  2. Can the authors provide Omni-MATH adaptation-surface controls, or a matched comparison that isolates the benefit of harness composition from executor routing? This evidence could materially affect my assessment.
4 comments
AuthorsOpenReview ↗

W1 & Q1: Distinctive methodological contribution

What is the specific methodological contribution that would remain if model routing and harness adaptation were considered separately?

Thank you for distinguishing the evaluation protocol from the question of methodological contribution.

HarnessWeaver takes the perspective that both model selection and harness design can be learned continually from execution experience. Considered separately, the model side accumulates experience about which executor suits which tasks, allowing model choices to be updated from observed outcomes. The harness side accumulates experience about how tasks should be executed, retaining procedural support for search, tool use, and verification as selectable, reusable components. Both turn execution experience into support for subsequent tasks while keeping model weights fixed.

Combining the two introduces a central decision: for the same failure, should the system change the model, change the harness, or change both? HarnessWeaver proposes and compares these three types of candidates through execution on the same failure scope, retaining effective combinations with their applicable task conditions. Experience therefore guides both the selection of existing configurations and decisions about how the current configuration should change. On ALFWorld, Full achieves 58.6%/81.4% first-encounter/final-round success, exceeding Model-only at 48.6%/75.7% and Harness-only at 54.3%/62.9%. Subsequent reuse records show that one retained combination is selected 30 times with 24 successes. These results support the complementary value of the two adaptation capabilities and establish that retained experience is used in later execution.

Our contribution is thus to connect task-configuration selection, intervention after failure, and experience reuse into a continual-improvement process: the model side learns who should execute, the harness side learns how to execute, and their combination learns what should change after failure. Experience-guided search explores candidates, and execution validation determines which combinations to retain. The evaluation approach scores first encounters separately from repeated-task performance to examine whether accumulated experience improves subsequent execution.

AuthorsOpenReview ↗

W2 & Q2: Omni-MATH harness-only and model-only controls

Can the authors provide Omni-MATH adaptation-surface controls, or a matched comparison that isolates the benefit of harness composition from executor routing? This evidence could materially affect my assessment.

Thank you for suggesting these adaptation-surface controls. We have added and completed the Omni-MATH Harness-only and Model-only control experiments. They distinguish changing the executor from changing its execution mechanisms on the same 100-task manifest, with results shown in Section 4.3, Figure 5b.

Adaptation surfaceFirst encounterFinal
Harness-only, fixed 9B11%9%
Model-only, frozen initial components47%56%
Full55%65%

Full obtains higher first-encounter and final-round success than both single-surface controls, exceeding Model-only by 8 and 9 percentage points, respectively. These completed-run comparisons support retaining both adaptation capabilities, with execution evidence determining which intervention to adopt for a given failure scope. Together with the ALFWorld controls, these results support our experimental conclusion that retaining both model selection and harness adaptation enables stronger continual improvement than restricting adaptation to either side in the evaluated runs.

I appreciate the authors’ response and their careful answers to my questions. I believe my initial score reflects my assessment of the paper, and therefore I would like to keep it.

Thank you for taking the time to review our responses and additional experiments. We appreciate your careful consideration. If there are any remaining concerns that we could help clarify, we would welcome the opportunity to address them. Thank you again for your time.

Reviewer_Dh8FOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes HarnessWeaver, a non-parametric continual self-improvement framework for LLM agents. For each task, an LLM "Weaver" picks an executor model and a small set of typed harness components from a Living Pool. After each round, failures are attributed and clustered by mechanism, and an Evolver proposes model-only, harness-only, or joint revisions. Candidates are replayed on the source failures and on held-out "native peer" tasks, and a tiered gate decides what is kept. The paper also proposes an evaluation protocol that separates first-encounter transfer on new batches from recovery on repeated batches, against a frozen pre-adaptation reference. On ALFWorld and Omni-MATH, macro first-encounter success goes from 31.5% (frozen) to 56.8%, vs 39.1% for ACE.

Critical strengths
  1. The question is real and relevant. When an agent fails, it is not obvious whether to change the model or the harness, and most prior work only touches one side. Treating the model–harness pair as the unit of adaptation is a reasonable framing.
  2. The evaluation protocol is the most useful part of the paper. Separating first-encounter success on new batches from final-round success on repeated batches, with a frozen H0 reference, is cleaner than how many self-improvement papers report results, and I think others could adopt it.
Critical weaknesses
  1. Most of the gain seems to come from using a stronger model, not from the proposed mechanism. On Omni-MATH, 446 of 500 online executions are routed to Qwen3.7-Flash, while the frozen reference, DC and ACE all use Qwen3.5-9B. The authors themselves say the Omni-MATH adaptation is "primarily routing-driven" and none of the seed math components change. The paper never reports the frozen score of Qwen3.7-Flash or Qwen3.8-27B with the initial harness, so we cannot tell how much of 13% → 55% is learned routing and how much is simply a better executor.
  2. Baselines are not compared under the same model budget. DC and ACE only get the 9B executor, and ACE is reduced to a single-pass protocol. Even so, ACE beats Full on ALFWorld first-encounter (67.1 vs 58.6), although Full can call the 27B model for most tasks (226 of 350 executions). A fair comparison needs ACE/DC with the same model pool, or at least with the strongest executor HarnessWeaver ends up using.
  3. The ablations do not support the core mechanisms. Removing native-peer validation gives higher macro first-encounter success (58.9 vs 56.8; 72.9 vs 58.6 on ALFWorld). Accept-All and w/o Lifecycle are also higher than Full on this metric (57.7). The paper frames these as "cost-oriented variants", but on the primary metric they suggest the peer validation and gate add cost without adding performance.
  4. Key prompts are missing. The Weaver, attributor, and Evolver prompts are not given in the paper, and the seed components are hand-written. Given the number of LLM-driven roles, it is hard to know how much of the result depends on prompt engineering.
  5. Missing comparisons with the closest work. The related work discusses Self-Harness, AutoSaddler, HarnessX, Adaptive Auto-Harness, ProgRouter, and Better Harnesses / Smaller Models, but none of them is compared experimentally. The contribution reads mostly as a combination of existing harness evolution and model routing ideas.
  6. Presentation is heavy. The paper introduces many concepts (Living Pool, playbooks, routing rules, four lifecycle states, evidence tiers, source/peer sets), and most of the main text is protocol definition.
Questions for authors

See weakness

7 comments
AuthorsOpenReview ↗

W1: Stronger executors and the Omni-MATH gain

Most of the gain seems to come from using a stronger model, not from the proposed mechanism. On Omni-MATH, 446 of 500 online executions are routed to Qwen3.7-Flash, while the frozen reference, DC and ACE all use Qwen3.5-9B. The authors themselves say the Omni-MATH adaptation is "primarily routing-driven" and none of the seed math components change. The paper never reports the frozen score of Qwen3.7-Flash or Qwen3.8-27B with the initial harness, so we cannot tell how much of 13% → 55% is learned routing and how much is simply a better executor.

Thank you for the detailed review and for recognizing the relevance of choosing between model and harness changes, the value of treating their combination as the unit of adaptation, and the first-encounter/recovery evaluation approach.

We fix Qwen3.8-27B throughout the same 100 Omni-MATH tasks. Each task uses its archived pre-adaptation harness assembly, with online configuration and memory updates disabled. The table compares this fixed-executor result with the frozen initial 9B system and Full.

ConfigurationFirst encounter / single passFinal round per batch
Frozen initial system (9B)13%—
Full55%65%
Fixed 27B + pre-adaptation assemblies57%—

Starting from the default 9B system, Full reaches 55% first-encounter success, close to the 57% obtained by selecting 27B in advance. Further within-batch adaptation raises Full’s final-round success to 65%. These results support improvement from the initial configuration through accumulated execution experience. Throughout Full’s online run on Omni-MATH, harness assembly also varies by task: 435 of 500 executions use all four mathematical components, while 65 use the three prompt components without the repair skill. Adaptation in this run therefore includes harness component selection and composition as well as model routing.

AuthorsOpenReview ↗

W2: Matched executor access and the ACE single-pass protocol

Baselines are not compared under the same model budget. DC and ACE only get the 9B executor, and ACE is reduced to a single-pass protocol. Even so, ACE beats Full on ALFWorld first-encounter (67.1 vs 58.6), although Full can call the 27B model for most tasks (226 of 350 executions). A fair comparison needs ACE/DC with the same model pool, or at least with the strongest executor HarnessWeaver ends up using.

We added ACE and Dynamic Cheatsheet experiments using Qwen3.8-27B on the same 70-task ALFWorld stream. Apart from upgrading the task executor, we kept their evaluation settings and memory-update schedules unchanged: ACE holds memory fixed within each batch and updates it afterward, while DC updates memory after each task. Each baseline executes each task once. The table compares their 9B and 27B results with Full.

MethodExecutorFirst encounter / single passFinal round
Full HarnessWeaverAdaptive pool58.6%81.4%
ACE, original9B67.1%—
Dynamic Cheatsheet, original9B55.7%—
ACE, new27B81.4%—
Dynamic Cheatsheet, new27B70.0%—

With 27B, ACE improves from 67.1% to 81.4% and DC from 55.7% to 70.0%, showing the importance of executor capability in baseline comparisons. Full reaches 81.4% final-round success, and a cost-oriented variant reaches 88.6%. These results support the value of continued experience accumulation and adaptation for task recovery.

AuthorsOpenReview ↗

W3: Do the ablations support native peers, the Gate, and lifecycle?

The ablations do not support the core mechanisms. Removing native-peer validation gives higher macro first-encounter success (58.9 vs 56.8; 72.9 vs 58.6 on ALFWorld). Accept-All and w/o Lifecycle are also higher than Full on this metric (57.7). The paper frames these as "cost-oriented variants", but on the primary metric they suggest the peer validation and gate add cost without adding performance.

These variants are cost-oriented control policies that offer different choices of validation expenditure, asset retention, selection feedback, and candidate admission. Full retains all controls as a reference. To avoid confusion with a conventional component-ablation presentation, we now name each variant by its operating policy. The configurations and reported results are unchanged.

Source-Only Validation (formerly w/o Native Peer). On ALFWorld, this policy reduces total online tokens by 14.4% and reaches 72.9% first-encounter success, versus 58.6% for Full. On Omni-MATH, Full scores 55% versus 45%. Checking only the repaired tasks saves calls but leaves less evidence of generalization; a change that fits those failures may not work reliably on new tasks.

Persistent Retention (formerly w/o Lifecycle). This policy reduces ALFWorld total tokens by 10.5% and reaches 88.6% final-round success. On Omni-MATH, its first/final success is 54%/64%, compared with Full’s 55%/65%, and total tokens increase by 6.4%. Lifecycle management determines whether skills, tools, and saved model–component combinations remain eligible for execution as they accumulate. Persistent retention leaves more learned assets available, which can help on the evaluated stream. Over longer runs, however, unused or unreliable skills, tools, and saved model–component combinations can accumulate, increasing selection complexity and context-budget pressure. Lifecycle management provides a way to remove such assets and the saved model–component combinations containing them.

Outcome-Agnostic Selection (formerly w/o KPI Feedback). Withholding historical success and resource-use statistics for models and harness assets (components, routing rules, and saved model–component combinations) gives macro first/final success of 52.4%/68.4%, versus Full’s 56.8%/73.2%. On Omni-MATH, first/final success drops to 42%/54% from Full’s 55%/65%, while total tokens decrease by 10.2%. Feedback grounds model and component selection in observed success and resource use. Without this feedback, the selector still sees task descriptions and applicability conditions, but cannot use the recorded success rates and resource costs to distinguish reliable choices from weak ones. A configuration may score well on a small set of tasks yet be selected inconsistently as alternatives accumulate. Historical feedback is intended to reduce that uncertainty and the risk of performance fluctuations.

Unconditional Admission (formerly Accept-All). Bypassing performance validation reduces ALFWorld total tokens by 34.0%, with 71.4% first-encounter and 67.1% final-round success; Full reaches 58.6% and 81.4%. On Omni-MATH, this policy reaches 44%/60% first/final success versus Full’s 55%/65%, with a 6.2% reduction in total tokens. The admission check within the validation module determines which locally useful changes can be used on subsequent tasks: one candidate improves its source score by 0.5 but requires 1.76 times the tokens and 1.92 times the latency, exceeding the recorded 1.25/1.5 limits, and is therefore withheld. Admitting every engineering-valid change saves validation effort, but can also retain changes that fail to help later tasks or consume excessive resources. The lower ALFWorld final-round score shows why faster admission and sustained recovery should be considered together. These policies expose different performance–cost choices rather than a uniform accuracy ranking.

Reviewer_CfyFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces a continual self-improvement framework that does not update model parameters. During task execution, it selects an execution model and combines harness components from a Living Pool under type and compatibility constraints. Between rounds, it analyzes failures and proposes changes to the model choice, the harness, or both. It then uses results on the source task and similar validation tasks to decide whether to keep each update. The paper evaluates the framework on ALFWorld and Omni-MATH. It separates performance on the first encounter with new tasks from recovery on repeated tasks. The full method reaches first-encounter success rates of 58.6% and 55.0%. The macro average across the two benchmarks is 56.8%, compared with 31.5% for the frozen reference system. The paper also includes ablations on adaptation strategies, an analysis of component reuse, and token cost statistics.

Critical strengths

The most interesting part is using execution evidence to decide between model and harness changes. The same failure may come from limited model capability or problems with planning, tool use, or execution control. Failure analysis alone may not clearly distinguish these causes. The paper separates diagnosing the failure mechanism from choosing a specific intervention. It proposes model-only, harness-only, and joint candidates for the same failure scope, then selects updates based on actual execution results. This offers a useful view of agent adaptation. What gets adapted is the combination of the model and its execution mechanisms. The dependencies between them should also inform the adaptation decision. The current experiments do not yet fully isolate the gains from joint adaptation. Still, this problem framing and the concrete way of addressing it are the main reasons I find the paper interesting.

Critical weaknesses

My main concern is that the paper treats model-harness composition as the core unit of adaptation. But it does not yet clearly show how model selection and harness changes complement each other. The system uses execution feedback to propose and screen candidates, then saves successful configurations. This shows that experience helps the system find better ways to execute tasks. It does not fully explain why joint adaptation is needed. For example, gains on some tasks may mostly come from finding a more suitable off-the-shelf model. Updating the routing while reusing an existing harness could then explain most of the improvement. I would like the paper to answer these questions more clearly: When does changing the model change what is needed from the harness? When does changing the harness change which model works best? Can these dependencies be learned from past experience and transferred to new tasks? The framework allows both to be modified. But it seems there is still a gap between supporting both kinds of changes and showing that they need to adapt together. This matters for how the paper positions its core contribution. Does it offer new insight into how models and execution mechanisms adapt together? Or does it mainly provide an effective configuration search process guided by experience?

Questions for authors
  1. Can the authors provide concrete examples where the appropriate harness depends on the executor, or where a harness change alters which executor works best? How does HarnessWeaver identify and retain these dependencies?

  2. What evidence shows that these model–harness dependencies transfer to new tasks and provide benefits beyond selecting a suitable executor while retaining the initial harness?

5 comments
AuthorsOpenReview ↗

W: Model–Harness Complementarity and Methodological Insight

“But it does not yet clearly show how model selection and harness changes complement each other.” “It does not fully explain why joint adaptation is needed.” “When does changing the model change what is needed from the harness? When does changing the harness change which model works best? Can these dependencies be learned from past experience and transferred to new tasks? The framework allows both to be modified. But it seems there is still a gap between supporting both kinds of changes and showing that they need to adapt together. This matters for how the paper positions its core contribution. Does it offer new insight into how models and execution mechanisms adapt together? Or does it mainly provide an effective configuration search process guided by experience?”

Thank you for the thoughtful review and for recognizing the value of execution-based selection between model and harness interventions. We address your concern about their complementarity below.

Our central insight is that adaptation should operate on a task-scoped executor–harness configuration: after a failure, the system must determine which side to change and which side to preserve. The executor supplies reasoning and instruction-following capabilities, while the harness organizes search, tool interaction, verification, and recovery. Their suitability must therefore be assessed together. HarnessWeaver makes this relationship operational: for a shared failure scope, it proposes model-only, harness-only, and joint interventions, compares their execution evidence, and retains the selected configuration with its applicability.

Executor replacement explains some improvements, but it does not explain all of the observed gains. The ALFWorld controls directly test this explanation: Full achieves 58.6% first-encounter and 81.4% final-round success, compared with 48.6% and 75.7% for Model-only. Harness-only obtains 54.3% and 62.9%, showing the complementary value of keeping executor adaptation available. We have also completed the Omni-MATH single-surface controls on the same 100-task manifest. The table below combines the original ALFWorld results with these additional controls.

BenchmarkConfigurationFirst encounterFinal round
ALFWorldFull58.6%81.4%
ALFWorldModel-only48.6%75.7%
ALFWorldHarness-only54.3%62.9%
Omni-MATHFull55%65%
Omni-MATHModel-only47%56%
Omni-MATHHarness-only11%9%

Across both benchmarks, Full achieves higher first-encounter and final-round success than either restricted configuration. These completed-run comparisons support retaining both adaptation capabilities rather than limiting the framework to executor replacement alone.

A real execution example shows why the pairing matters. In the archived ALFWorld task records, the pre-modification configuration is Qwen3.8-27B with the original harness; the evolved, retained configuration is Qwen3.7-Flash with the revised harness containing the systematic-search skill. To examine their pairing, we additionally executed the same ALFWorld task with Qwen3.8-27B plus the revised harness and Qwen3.7-Flash plus the original harness, alongside those two configurations.

Task: Find a tomato, cool it in a refrigerator, and place it on a countertop.

ExecutorOriginal harnessRevised harness
Qwen3.8-27BSuccessFailure
Qwen3.7-FlashFailureSuccess

With the original harness, Qwen3.7-Flash exhausts its action budget searching cabinets and drawers. The revision redirects its search toward countertops and enables successful completion. For Qwen3.8-27B, the original execution succeeds, but the revised execution places the tomato without cooling it and fails. The same guidance thus produces executor-dependent behavior. Dependencies are retained as executable configurations with applicability evidence. Later selection can therefore reuse an executed pairing rather than treating either the model preference or the harness revision as universally applicable.

Our methodological contribution is to treat a validated, task-scoped model–harness combination as a unit of reusable experience. The system learns which executor and procedural support work together, on which tasks, and with what outcomes. This experience guides both subsequent execution and the decision of what to change or preserve after a failure. HarnessWeaver compares model-only, harness-only, and joint candidates for the same failure scope, then retains the selected combination with its applicability and observed results. Experience-guided search generates alternatives; execution validation and task-scoped retention turn the selected alternative into experience that can guide later tasks. This connects intervention selection with experience accumulation and reuse, beyond searching the two configuration dimensions independently.

AuthorsOpenReview ↗

Q1: Executor-dependent harness effects and their identification

Can the authors provide concrete examples where the appropriate harness depends on the executor, or where a harness change alters which executor works best? How does HarnessWeaver identify and retain these dependencies?

We performed an additional experiment to examine the effects of changing the model and changing the harness separately. Before the ALFWorld round-12 change, the configuration was Qwen3.8-27B with the original harness; the retained configuration was Qwen3.7-Flash with a harness containing an additional search skill. We evaluated each harness with both Qwen3.8-27B and Qwen3.7-Flash, giving four pairings, on the same 25 tasks not encountered in the original run. This lets us compare harnesses with the model held fixed and compare models with the harness held fixed.

ExecutorOriginalRevisedChange
Qwen3.8-27B17/25 (68%)18/25 (72%)+4 points
Qwen3.7-Flash13/25 (52%)15/25 (60%)+8 points

In the 25-task evaluation, the revised harness increases success from 17/25 to 18/25 for Qwen3.8-27B and from 13/25 to 15/25 for Qwen3.7-Flash, with each executor held fixed. All 25 tasks require finding a specified object, cooling it using a refrigerator, and placing it at a specified destination; the object, destination, or scene differs across tasks. The revised harness adds a systematic-search skill that provides search guidance and reminders about required preparation steps. For example, on the task of cooling a tomato and placing it on a countertop, the added search skill guides Qwen3.7-Flash toward the countertops and enables successful completion, while revised-harness Qwen3.8-27B finds and places the tomato but omits cooling; Qwen3.8-27B succeeds with the original harness on that task. On the task of cooling a plate and placing it in a cabinet, the added search skill enables Qwen3.8-27B to complete the cooling step missing in its original execution, while revised-harness Qwen3.7-Flash cools the plate but fails through invalid placement commands; Qwen3.7-Flash succeeds with the original harness.

These paired executions show that the effect of the same harness change depends on the executor. HarnessWeaver identifies this dependency by testing candidate combinations of models and harnesses on failed tasks and separate same-family validation tasks. It retains the tested pairing together with the applicable task conditions and observed outcomes. Later selection can retrieve the combination for matching tasks, using its recorded results to inform the choice.

AuthorsOpenReview ↗

Q2: Retaining dependencies and transferring beyond executor selection

What evidence shows that these model–harness dependencies transfer to new tasks and provide benefits beyond selecting a suitable executor while retaining the initial harness?

Our original experiments already provide evidence that accumulated experience improves performance on previously unencountered tasks. After adaptation on Batch 1, first-encounter success on Batch 2 rises from 40% to 45% on ALFWorld and from 35% to 95% on Omni-MATH, compared with the frozen initial system on the same tasks. Each Batch-2 comparison contains 20 tasks and is scored before the system learns from that batch. The later batches show the same positive first-encounter pattern: gains over their own frozen references are +15/+20 points for ALFWorld Batches 3–4 and +75/+30/+45 points for Omni-MATH Batches 3–5. The table below shows subsequent reuse counts and success rates for two saved model–component configurations. Selection counts include repeated-task rounds. Thus the original stream results already provide evidence that previously accumulated model–harness configuration updates benefit subsequent tasks.

Retained configurationSubsequent selectionsSuccessesSuccess rate
ALFWorld light inspection: 9B + observe-first, plan-first, verify-before-finish302480.0%
Omni-MATH combinatorics: Flash + four seed mathematical components14010977.9%

To further test cross-task generalization and identify a harness contribution beyond executor selection, we added the 2×2 replay experiment described in Q1, holding each executor fixed. On 25 tasks not encountered in the original run, the retained harness revision raises the point estimate from 68% to 72% for 27B and from 52% to 60% for Flash. Thus, the cross-batch experiments show that previously accumulated experience improves performance on subsequent new tasks. The additional crossed replay tests the harness change with each model held fixed, while task-level executions show that its effects depend on the model pairing. Together, these results demonstrate HarnessWeaver’s cross-task transfer performance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive suggestions. We have clarified the contribution, improved the presentation, and completed additional experiments. Changes to the manuscript are marked in blue. The main updates are:

  1. Clarified the contribution and simplified the method (Sections 1–3; Appendix H). We explain how task-scoped model and harness combinations serve as reusable experience, guiding both subsequent execution and decisions about what to change after failure. We shortened repeated descriptions and added a terminology table. (CfyF, Dh8F, 6oWH)
  2. Added Omni-MATH adaptation controls (Section 4.3, Figure 5b). On the same 100 tasks, Full achieves 55%/65% first-encounter/final-round success, compared with 47%/56% for Model-only and 11%/9% for Harness-only. (CfyF, 6oWH)
  3. Added strong-executor comparisons. ACE and Dynamic Cheatsheet achieve 81.4% and 70.0%, respectively, using Qwen3.8-27B on the same 70-task ALFWorld stream with their existing memory-update schedules (Section 4.2, Table 2). Our response to Dh8F’s W1 also reports the additional fixed-Qwen3.8-27B comparison on Omni-MATH. (Dh8F)
  4. Clarified cost-oriented control policies (Section 4.4, Table 3; Appendix G.3). We renamed the four variants Source-Only Validation, Persistent Retention, Outcome-Agnostic Selection, and Unconditional Admission, separated them from the main baseline table, and explained their performance, resource costs, and control responsibilities. The configurations and results are unchanged. (Dh8F)
  5. Expanded reuse and transfer evidence and execution examples (Sections 4.6–4.7, Tables 4–5, Figure 7). We report subsequent reuse of retained configurations and a crossed evaluation of two harness versions with Qwen3.8-27B and Qwen3.7-Flash on 25 previously unencountered tasks. We also expanded the salt-shaker case and added a flow diagram to explain intervention selection and configuration retention. (CfyF, 6oWH)
  6. Expanded reproducibility materials (Appendix I). We provide the Weaver, attributor, and Evolver prompts, their runtime inputs and outputs, retry feedback, and the allowed modification types for Full, Model-only, and Harness-only. We also explain the role of hand-written initial components in establishing basic execution capability. (Dh8F)

Our individual replies provide the experimental results and explanations for each concern.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.