acceptodds
Under review as a conference paper at ICLR 2027

MemoHarness: Agent Harnesses That Learn from Experience

Abstract

An agent harness is the external control layer that turns a base LLM into an executable agent by managing context, tools, orchestration, memory, decoding, and output handling. While harness design strongly affects agent behavior, most automatic improvement methods optimize narrower artifacts such as prompts, pipelines, or workflows, and deployed agents usually reuse a single global harness for all cases. We introduce MemoHarness, an adaptive harness optimization framework that learns from its own executions. MemoHarness decomposes the harness into six editable control dimensions, stores per-case diagnoses and distilled global patterns in a dual-layer experience bank, and adapts the learned harness to each test case using retrieved experience without test-time labels, feedback, or additional search. Across shell-agent, code-generation, and analytical-reasoning benchmarks, MemoHarness consistently improves over fixed harness baselines, transfers across unseen suites and base models, and remains cost-effective through reusable cached context. These results suggest that execution experience is a practical substrate for building agent harnesses that are more adaptive than static hand-engineered configurations.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Harness, Agent, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GdH2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how execution experience can support agent harness optimization and per-task adaptation. MemoHarness uses feedback from training tasks to revise a global harness and build a bank of case records and distilled patterns. For each test task, a controller uses historical experience to propose a one-time configuration edit before execution, without feedback-driven search on that task. The paper evaluates the system on terminal tasks, code generation, and financial reasoning, and reports selective transfer across datasets and models. Ablations examine the full method against a shared trained harness and variants without parts of the experience bank.

Critical strengths
  1. The learned harness improves performance on all evaluated additional base models without re-running the search, and also improves results on some unseen datasets. These findings provide evidence that the learned changes have value beyond the original search setting.

  2. The full method improves performance on terminal tasks, code generation, and financial analysis. These tasks differ in their requirements for tool interaction, execution steps, and intermediate state, showing that the overall gains extend across different types of agent workloads.

Critical weaknesses
  1. The paper’s main contribution is test-time adaptation for each individual task, but I have substantial concerns about the practicality of modifying the harness for every task. In practice, this requires a task-specific harness modification before each execution, adding token and time costs. The cost analysis mainly discusses cache hits and the cost of reading experience context, without reporting the additional cost of generating these modifications or its scale relative to task execution costs and runtime before and after adaptation. It is therefore unclear whether the gains from per-task harness modification justify these additional costs.

  2. The paper does not clearly distinguish the gains from experience accumulation and harness modification. During training, it both accumulates case-level experience and distills global patterns while modifying the harness based on feedback. At test time, it likewise retrieves relevant experience as context and modifies the harness for the current task. These two components remain intertwined. Although the ablations compare the full method with a uniformly applied trained harness and variants that remove parts of the experience, they do not establish how much additional benefit harness modification provides when the same experience context is available. The paper also does not clearly report how much training improves the initial harness before per-task adaptation is introduced. Consequently, the gains from the two stages remain unclear, as does whether they mainly come from accumulating and reusing experience or from modifying the harness itself.

  3. The paper does not clearly show which parts of the harness are modified during training and test-time adaptation, particularly in the per-task test-time edits. The authors divide the modification space into six dimensions and emphasize that harness evolution can change a broader control layer than prompt optimization, but do not specify what actually changes in these dimensions. In particular, it is unclear whether the controller’s test-time edits change mechanisms such as tool use, execution orchestration, and memory management, or mainly update prompts and experience context. It is therefore difficult to assess whether the actual scope of evolution supports the claimed harness-level optimization.

  4. The paper lacks analysis and comparison with closely related work on harness evolution. DGM [1] and HGM [2] study modifying agents based on task execution feedback and selecting versions for further evolution, making them closely related to the first stage. MOSS [3] also explores system source-code modification driven by execution failures. Most importantly, TTHE [4] explicitly studies test-time harness evolution, yet it is absent from the comparison despite test-time adaptation being a main contribution of this paper. The paper needs to explain its specific innovations and advantages in relation to these existing methods.

[1] Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents

[2] Huxley-Gödel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine

[3] MOSS: Self-Evolution through Source-Level Rewriting in Autonomous Agent Systems

[4] TTHE: Test-Time Harness Evolution

Questions for authors

See Weaknesses.

Reviewer_ahJ9OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MemoHarness, a framework for automatically optimizing agent harnesses. It decomposes the harness into six dimensions, learns from past execution experiences during training, and adapts the harness to individual tasks at test time. Experiments across several benchmarks show improvements over existing baselines and some cross-task and cross-model transferability.

Critical strengths
  • Adapting agent harnesses to individual tasks rather than using a fixed configuration is an interesting and worthwhile direction.

  • Well-motivated problem. Existing harness optimization methods often focus on individual components or produce a single global configuration, while different tasks may require different harnesses. Addressing this gap is a meaningful and worthwhile direction.

  • The improvements across multiple benchmarks and the cross-model transfer results suggest that the approach has practical potential.

Critical weaknesses
  • The main ideas of training-time search and test-time adaptation through experience retrieval are not particularly novel, and the proposed framework largely combines existing techniques.

  • The paper does not clearly report the total cost of harness optimization, including token usage, API expenses, and wall-clock time, making it difficult to assess its overall efficiency.

  • The evaluation does not sufficiently demonstrate the method's effectiveness on more challenging long-horizon tasks, leaving its generalizability to complex agentic settings unclear.

Questions for authors

see Weaknesses

Reviewer_LhPVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose MemoHarness, an adaptive harness optimization framework that learns from its own executions. MemoHarness decomposes the harness space into six dimensions. During training-time harness optimization, the system modifies the harness using task-level scoring feedback and organizes execution results into two layers of experience. During test-time case adaptation, it retrieves historical experience for each new case and generates a case-specific configuration. According to the reported results, MemoHarness shows improved task performance.

Critical strengths
  • The paper addresses a well-motivated problem. The overall framework design is reasonable: the decomposition of the harness space into six dimensions makes sense, and the distinction between training-time optimization and test-time case adaptation is clear.
  • The experiments cover different tasks and base models, and the reported results show performance improvements.
Critical weaknesses
  • Despite the sensible framework design and clear presentation, some key components appear to build on existing ideas. The paper could more clearly articulate what new insights arise from their integration beyond the reported performance improvements.
  • In RQ1, MemoHarness is compared with Meta-Harness and AHE. However, line 779 states that the model and tool configurations are not fully matched. Could the authors clarify the specific configurations used for these comparisons? This would improve the transparency of the evaluation and strengthen confidence in the experimental conclusions.
Questions for authors

See Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.