- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how execution experience can support agent harness optimization and per-task adaptation. MemoHarness uses feedback from training tasks to revise a global harness and build a bank of case records and distilled patterns. For each test task, a controller uses historical experience to propose a one-time configuration edit before execution, without feedback-driven search on that task. The paper evaluates the system on terminal tasks, code generation, and financial reasoning, and reports selective transfer across datasets and models. Ablations examine the full method against a shared trained harness and variants without parts of the experience bank.
Critical strengths
-
The learned harness improves performance on all evaluated additional base models without re-running the search, and also improves results on some unseen datasets. These findings provide evidence that the learned changes have value beyond the original search setting.
-
The full method improves performance on terminal tasks, code generation, and financial analysis. These tasks differ in their requirements for tool interaction, execution steps, and intermediate state, showing that the overall gains extend across different types of agent workloads.
Critical weaknesses
-
The paper’s main contribution is test-time adaptation for each individual task, but I have substantial concerns about the practicality of modifying the harness for every task. In practice, this requires a task-specific harness modification before each execution, adding token and time costs. The cost analysis mainly discusses cache hits and the cost of reading experience context, without reporting the additional cost of generating these modifications or its scale relative to task execution costs and runtime before and after adaptation. It is therefore unclear whether the gains from per-task harness modification justify these additional costs.
-
The paper does not clearly distinguish the gains from experience accumulation and harness modification. During training, it both accumulates case-level experience and distills global patterns while modifying the harness based on feedback. At test time, it likewise retrieves relevant experience as context and modifies the harness for the current task. These two components remain intertwined. Although the ablations compare the full method with a uniformly applied trained harness and variants that remove parts of the experience, they do not establish how much additional benefit harness modification provides when the same experience context is available. The paper also does not clearly report how much training improves the initial harness before per-task adaptation is introduced. Consequently, the gains from the two stages remain unclear, as does whether they mainly come from accumulating and reusing experience or from modifying the harness itself.
-
The paper does not clearly show which parts of the harness are modified during training and test-time adaptation, particularly in the per-task test-time edits. The authors divide the modification space into six dimensions and emphasize that harness evolution can change a broader control layer than prompt optimization, but do not specify what actually changes in these dimensions. In particular, it is unclear whether the controller’s test-time edits change mechanisms such as tool use, execution orchestration, and memory management, or mainly update prompts and experience context. It is therefore difficult to assess whether the actual scope of evolution supports the claimed harness-level optimization.
-
The paper lacks analysis and comparison with closely related work on harness evolution. DGM [1] and HGM [2] study modifying agents based on task execution feedback and selecting versions for further evolution, making them closely related to the first stage. MOSS [3] also explores system source-code modification driven by execution failures. Most importantly, TTHE [4] explicitly studies test-time harness evolution, yet it is absent from the comparison despite test-time adaptation being a main contribution of this paper. The paper needs to explain its specific innovations and advantages in relation to these existing methods.
[1] Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents
[2] Huxley-Gödel Machine: Human-Level Coding Agent Development by an Approximation of the Optimal Self-Improving Machine
[3] MOSS: Self-Evolution through Source-Level Rewriting in Autonomous Agent Systems
[4] TTHE: Test-Time Harness Evolution
Questions for authors
See Weaknesses.