acceptodds
Under review as a conference paper at ICLR 2027

miniFlux: Exploring Agent-Directed Harness Evolution for Online Adaptation

Abstract

Agent harnesses have become a crucial factor in agent performance, shaping how models process context, take actions, and receive feedback. However, the support an agent needs can vary across tasks and evolve during execution. Existing approaches to automated harness evolution typically operate offline, relying on fixed task executions that may not capture diverse, emerging needs during deployment. We introduce miniFlux, an experimental framework for agent-directed online harness evolution within each task execution. Starting from a minimal harness (mini), the same model alternates between a main executor that advances the task and an evolver that adapts the active harness based on recent execution evidence. Triggered by periodic checks or executor requests, the evolver uses a bounded tool-use loop to revise runtime prompt guidance, tools, middleware, and context management. Component updates that pass interface and loading checks take effect through hot reload, preserving the progress without resetting the task. Evaluations on ARC-AGI-3, Tetris, and GDPval tasks show that miniFlux achieves higher mean task scores and improved action-level efficiency than the fixed mini baseline across evaluated models, and outperforms Codex in some evaluated settings. Cost analysis shows that online evolution adds relatively small overhead, representing less than 30% of the main execution cost in most evaluated settings. Evolution analyses illustrate that agents create and adapt task-specific components, such as analysis tools and validation middleware, to support information processing and task delivery. The ablation of evolution triggers shows that combining scheduled and agent-requested updates achieves higher mean performance than either mode alone. These findings suggest that adapting the harness online to emerging needs is a promising direction for improving agent performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
harness, agent, self-improvement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_43VaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces an online self-evolving method for AI-Agent harness, with mini-swe-agent(mini) as the baseline. Within each task attempt(without final test outcome), the same frozen base model alternates between two roles: a main executor working on the task, and an online harness evolver that regularly observes the growing trajectory and updates the harness by addressing issues identified from the ongoing execution process. Experiments of DeepSeek-V4-Flash and GLM-5.2 on three benchmarks show that miniFlux surpasses the baseline mini-swe-agent in all six conditions and outperforms Codex in certain settings. While the online evolver adds some cost, miniFlux manages to reduce the execution tokens under certain settings.

Critical strengths

The method is cleanly designed, the experiments are carefully conducted and the results are supportive. The paper evaluates not only final task performance but also cost efficiency, showing that online harness evolution can improve task outcomes, while maintaining resource efficiency in certain settings like GDPval. These results make the approach potential for building more capable and efficient agents.

Critical weaknesses
  1. The current evaluation does not sufficiently disentangle the benefit of online evolution from the value of adding missing capabilities to a minimal harness like mini-swe-agent. Many evolved components (e.g. web tools and validation middleware) are already standard capabilities in modern agent systems. A stronger experiment would compare online evolution against a fixed harness equipped with the same capabilities, or investigate whether online evolution can further improve a mature harness, such as Codex, rather than mainly demonstrating that a minimal agent benefits from acquiring additional tools.
  2. It appears that the effectiveness of online evolution relies largely on long-horizon task trajectories. For tasks that naturally require few steps, the evolver may not have sufficient opportunities to observe and propose any meaningful updates.
Questions for authors
  1. Can the authors provide additional analysis or ablations to disentangle the benefit of online harness evolution from the benefit of adding new capabilities?
  2. How does miniFlux behave on short-horizon tasks where only a few execution steps are available? Do the authors have evidence that the approach remains useful when the evolver has limited opportunities to observe execution trajectories and perform updates? More generally, how does miniFlux performance scale with available trajectory length?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 43Va

We deeply thank Reviewer 43Va for the positive and constructive feedback, particularly the recognition of our method’s clean design and the carefully conducted experiments supporting its effectiveness. Below, we address the remaining concerns and questions in detail.

W1 & Q1: Disentangling the benefit of online harness evolution from that of adding capabilities

We agree that additional analysis and ablations are needed to disentangle the benefit of online harness evolution from that of adding new capabilities. We have therefore implemented two additional baselines:

  • All-installed static harness: All capabilities in our seed repository are pre-installed, and the agent carries out the task using this fixed harness.
  • Reflection-only harness: The evolver is replaced with a dedicated reflection agent, which is prompted to reflect on past execution traces, replan, and provide guidance to the main agent without modifying the harness. We retain miniFlux’s periodic and proactive triggering mechanisms and use the same triggering frequency to control the budget.
ModelHarnessARC-AGI-3 (Score)ARC-AGI-3 (Levels)GDPval
DeepSeek-V4-FlashAll-installed static harness2.69 ± 1.6317.00 ± 2.6577.27 ± 1.59
DeepSeek-V4-FlashReflection-only harness2.70 ± 1.6623.00 ± 7.9477.90 ± 1.07
DeepSeek-V4-FlashminiFlux (online adaptation)3.32 ± 1.7224.33 ± 7.3778.97 ± 0.42

miniFlux achieves higher mean performance than both baselines across the reported metrics. These results support the value of online harness adaptation beyond pre-installing the existing capabilities or providing online reflections alone.

W2 & Q2: Behavior on short-horizon tasks and scaling with available trajectory length

We agree that miniFlux’s effectiveness on short-horizon tasks is also important. In the current approach, the evolution interval, which determines the amount of execution history accumulated between periodic evolution triggers, remains a hyperparameter that can be set empirically based on the expected task length. Despite this flexibility, we acknowledge that tasks requiring only one or a few steps offer limited opportunities for online adaptation. However, our method is not restricted to long-horizon tasks.

To investigate this, we conducted additional experiments on relatively short-horizon tasks from the WorkBuddy Web split (a 20 task subset), where GLM-5.2 takes approximately 20 execution steps per task on average.

ModelHarnessScore mean ± SDAvg. evolutions/taskMean Steps
GLM-5.2mini54.83 ± 9.700.0017.48
GLM-5.2miniFlux (evolution_interval=10)62.00 ± 4.443.0819.28
GLM-5.2miniFlux (evolution_interval=20)64.50 ± 5.631.1718.43

These results suggest that miniFlux remains useful in this relatively short-horizon setting. The longer evolution interval yields a higher mean score despite fewer evolutions per task, suggesting that allowing more execution history to accumulate before adaptation can be beneficial in this setting.

Thank you again for these insightful questions. We plan to expand the analysis with these further related experiments in the revised version of the paper.

Dear Reviewer 43Va,

Thank you again for your thoughtful and constructive comments. As the discussion phase is approaching its end, we would appreciate any further feedback you may have. We have kept our responses concise and focused on addressing your concerns to make them easy to review. We would be grateful to hear whether our responses have addressed your concerns or if any points need further clarification.

Thank you for your time and consideration.

Appreciate your response. I prefer to maintain my rating.

Reviewer_rkZGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper introduces miniFlux, a framework for online agent harness evolution motivated by the observation that the support an agent needs may vary across tasks and emerge only during execution. MiniFlux alternates the same base model between an executor and an evolver: the evolver periodically inspects recent execution traces and can modify runtime prompts, tools, middleware, and context-management components, with accepted updates hot-reloaded into the ongoing task. The method is evaluated on ARC-AGI-3, Tetris, and GDPval, where it generally improves over the static mini baseline. Qualitative examples include creating an arc3_analyze tool that converts raw ARC grids into structured geometric summaries, and repairing an evolved ARC parser after the agent detects malformed coordinates during execution; on GDPval, the evolver additionally introduces retrieval, validation, context-compression, and budget-management components in response to observed difficulties.

Critical strengths

The paper is clearly written and the proposed framework is simple, modular, and easy to follow. The executor–evolver separation provides a clean formulation of online harness adaptation, and the editable component space is broad enough to cover prompts, tools, middleware, and context management. The experiments are accompanied by useful qualitative analyses rather than only performance numbers. Some case studies are illustrative—for example, the agent creates a task-specific arc3_analyze tool that converts the cached grid into structured geometric summaries, reducing manual cell-by-cell parsing. These examples suggest that the framework can, at least in some cases, produce targeted and useful harness modifications tailored to needs that arise during execution.

Critical weaknesses
  1. A major concern is that the baseline may be too under-equipped to support the paper’s main comparison. The mini baseline starts essentially with only Bash, while miniFlux can later add standard capabilities such as file/search tools, web retrieval, validation, and context management. Why not simply provide these common utilities to a static agent from the beginning and use it as a baseline? Without this comparison, it is unclear whether miniFlux improves performance because of online adaptation, or simply because it gives the agent capabilities that the baseline initially lacks. The paper also does not show whether the initial mini harness is sufficiently capable of completing the evaluated tasks in the first place. If some failures are mainly caused by missing basic task-relevant capabilities, fixing them during execution is weak evidence for the benefit of online harness evolution. A capability-matched static baseline is therefore necessary.

  2. A second major concern is that the paper does not establish the specific value of performing harness evolution online. The main motivation is that useful harness adaptations may only become apparent during execution, and that repeated executions used by offline evolution may be impractical. However, the experiments do not compare within-task evolution against either task-conditioned pre-task adaptation or conventional post-task/offline evolution. This is particularly important because the evaluated settings permit repeated executions. The current results therefore show that an online-evolving harness can outperform a fixed minimal harness, but do not show whether performing the adaptation during execution provides an advantage over adapting the harness before or after execution. A direct comparison under matched adaptation and inference budgets would help establish the specific benefit of within-task online adaptation.

Questions for authors
  1. Can the authors provide a compute-matched baseline? miniFlux receives additional Evolver calls beyond the main-agent execution budget. How would a static-harness agent perform if given the same additional inference budget for reflection, replanning, or verification, without being allowed to modify the harness?

  2. Why are there no evaluations on more open-ended, tool-rich, long-horizon agentic tasks? The current benchmarks cover interactive games and bounded professional artifact generation. Since the claimed motivation is to adapt to needs that emerge during execution, tasks involving complex tool orchestration, long-term planning, failure recovery, or exploration—e.g., software engineering, terminal tasks, cybersecurity, or open-ended web tasks—seem particularly relevant.

  3. Could the authors provide more qualitative case studies showing non-trivial harness adaptations that diagnose and resolve complex agent failures? The arc3_analyze tool is an useful example, but many of the other highlighted gains may come from relatively standard utilities such as web retrieval, validation, or context compression. It would strengthen the paper to show whether the Evolver can also produce more substantive task-specific changes to planning, recovery, memory, or tool orchestration in response to complex execution failures.

6 comments
AuthorsOpenReview ↗

Response to Reviewer rkZG #1

We thank Reviewer rkZG for recognizing the clarity of our writing, the clean formulation of our framework, and the usefulness of our analyses. We appreciate the constructive feedback and address the concerns below.

W1 & Q1: Pre-installed static and reflection-only baselines

We agree that comparisons with a static harness equipped with all available components and a reflection-only baseline would help clarify the benefits of online harness evolution. However, including all capabilities may not always be beneficial, since some capabilities are not useful under certain conditions (e.g., web_search() in the network-isolated ARC3 gaming environment), and providing them could confuse the agent and consume unnecessary tokens. Meanwhile, the mini agent can implement task-relevant capabilities through code scripting and execution. The motivation for within-task adaptation is to accurately identify and curate specific capabilities in response to task needs.

To examine these effects, we implemented the two baselines and evaluated them on ARC-AGI-3 and GDPval using the same evaluation configurations as miniFlux. The results are shown below, with each experiment repeated 3 times:

ModelHarnessARC-AGI-3 (Score)ARC-AGI-3 (Levels)GDPval
DeepSeek-V4-FlashAll-installed static harness2.69 ± 1.6317.00 ± 2.6577.27 ± 1.59
DeepSeek-V4-FlashReflection-only static harness2.70 ± 1.6623.00 ± 7.9477.90 ± 1.07
DeepSeek-V4-FlashminiFlux (ours)3.32 ± 1.7224.33 ± 7.3778.97 ± 0.42

For the reflection-only harness, to make inference budgets comparable, we retain the same triggering mechanisms and frequency as miniFlux. The only difference is that the evolver agent is replaced with a reflection agent that reviews past execution traces, reflects or replans, and provides guidance to the main agent without the capability to modify the harness.

On DeepSeek-V4-Flash, miniFlux achieves higher mean performance than both baselines across the reported metrics. These results support the benefits of dynamically adapting the execution framework.

Q2: Additional evaluations on open-ended, tool-rich, long-horizon benchmarks

Thank you for this constructive suggestion. We originally chose interactive games and bounded professional tasks for their long-running and open-ended aspects, while recognizing that other scenarios are also relevant to our method. We have conducted additional experiments on open-ended web development using a 20-task subset from the WorkBuddy Bench Web split. All tasks can be approached through code scripting and execution with the mini-swe-harness agent. The results are shown below, with each experiment repeated 3 times.

ModelHarnessWorkBuddy Web Subset
DeepSeek V4 Flashmini baseline60.33 ± 4.86
DeepSeek V4 FlashminiFlux62.83 ± 3.69
GLM-5.2mini baseline54.83 ± 9.70
GLM-5.2miniFlux64.50 ± 5.63

Our approach achieves higher mean performance than the mini baseline across both models on this new task category. We will include a broader set of experiments to provide further evidence.

AuthorsOpenReview ↗

Response to Reviewer rkZG #2

Q3: More qualitative case studies of useful harness adaptations

We agree that additional qualitative cases showing non-trivial harness adaptations that diagnose and resolve complex agent failures would help demonstrate the usefulness of online adaptation. The table below provides further examples.

BenchmarkComponentEvolutionFailure Resolution
ARC3tools.plan_blocksCreated and installed an offline block-movement planner, then corrected its movement model and target-argument handling using execution feedback.Addressed overly restrictive movement predictions and ignored target requests, enabling replanning from observed game states.
GDPvaltools.pdf_layout_auditCreated and installed a PDF layout-audit tool, then added interpreter discovery to locate an environment with the required dependencies.Recovered from missing-dependency errors and identified sparse pages, guiding pagination revisions that reduced the document from 26 to 21 pages.
Tetrismiddleware.tetris_compactCreated and installed middleware to remove duplicated rotation-preview boards while preserving the current board and game statistics.Reduced repetitive observation content and its context overhead while retaining the information needed for subsequent decisions.
WorkBuddy Webtools.render_checkCreated and installed a browser inspection tool to check element opacity, position, visibility, and console errors at desktop and mobile viewport sizes.Detected in-viewport content hidden by zero opacity and verified that subsequent CSS corrections restored visibility at both viewport sizes.

We will expand the case studies in the revised manuscript to cover a broader range of adaptations.

Finally, regarding W2, we view online and offline harness evolution as complementary approaches that offer benefits under different assumptions and scenarios. Our main focus in this study is to examine the benefits of online evolution. Given the additional time and effort required for these evaluations, we will clarify this scope in the revised manuscript and include comparisons with existing offline evolution methods, as well as evaluations of their combination with online evolution, in the future version.

Thank you for the additional experiments and analyses. I appreciate the authors' efforts to address my concerns. However, two major concerns remain.

1. The motivation remains insufficiently justified. Given the increasingly strong implementation capabilities of modern AI systems, I believe that a clear and compelling research motivation is particularly important. While the authors demonstrate that online harness evolution can improve performance, they have not sufficiently established why within-task harness evolution is an important problem to study, or under what circumstances it offers meaningful advantages over alternative approaches.

2. The qualitative cases are still not sufficiently compelling. Although the additional examples demonstrate that the framework can produce useful harness modifications, most involve relatively straightforward adaptations. They do not convincingly illustrate challenging scenarios in which online harness evolution plays a particularly important role, nor do they provide strong qualitative support for the paper's central motivation.

Overall, I appreciate the additional experimental effort, but these concerns about the research motivation and the significance of the demonstrated adaptations remain. Therefore, I am inclined to maintain my original assessment.

Reviewer_eqH4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes miniFlux, a framework that allows agents to modify their own harness during task execution. The approach is evaluated on ARC-AGI-3, Tetris, and GDPval, showing improvements over a static harness across two models.

Critical strengths
  • The idea of adapting the harness during task execution makes sense and can be useful.

  • The framework is somehow well designed, supporting modifications to tools, prompts, middleware, and context management without restarting the task.

  • The experiments cover different tasks.

Critical weaknesses
  • Missing baselines. The improvements may come from access to additional tools or reflection rather than harness evolution itself. It's better to add the comparisons with a static harness equipped with all available tools and reflection-only baselines.

  • Unclear efficiency gains. The reported efficiency metrics largely focus on the main agent, while evolution requires additional model calls. It's more convincing to have a comparison under matched inference budgets.

  • Limited novelty. Recent work has already explored online harness adaptation. The paper would benefit from a clearer distinction and stronger evidence for the advantages of the method it proposes.

Questions for authors

see weaknesses.

8 comments
AuthorsOpenReview ↗

Response to Reviewer eqH4 #1

We thank Reviewer eqH4 for recognizing the motivation for within-task adaptation and the design of our miniFlux framework. We appreciate the constructive feedback and address the concerns below.

W1: Missing all-installed and reflection-only harness baselines.

We agree that comparisons with a static harness equipped with all available components and a reflection-only baseline would help demonstrate the benefits of online harness evolution more clearly. To this end, we have implemented these two baselines and evaluated them on ARC-AGI-3 and GDPval using the same configurations as miniFlux. The results are shown below, with each experiment repeated 3 times:

ModelHarnessARC-AGI-3 (Score)ARC-AGI-3 (Levels)GDPval
DeepSeek-V4-FlashAll-installed static harness2.69 ± 1.6317.00 ± 2.6577.27 ± 1.59
DeepSeek-V4-FlashReflection-only harness2.70 ± 1.6623.00 ± 7.9477.90 ± 1.07
DeepSeek-V4-FlashminiFlux (online adaptation)3.32 ± 1.7224.33 ± 7.3778.97 ± 0.42

For the reflection-only harness, to make inference budgets more comparable, we retain the same triggering mechanisms and frequency as miniFlux. The only difference is that the evolver agent is replaced with a reflection agent that reviews past execution traces, reflects or replans, and provides guidance to the main agent without the capability to modify the harness.

The results with DeepSeek-V4-Flash show that online adaptation outperforms both baselines, supporting the effectiveness of dynamically adapting the execution framework.

W2: Unclear evolution efficiency gains.

We agree that online evolution requires additional model calls. Our original rationale for reporting main-agent efficiency metrics, such as steps/task, was to quantify the effect of evolution on task execution: whether a continuously evolving harness enables the main agent to complete tasks more efficiently. To quantify the overhead of the evolution process itself, we provide additional analysis in Section 4.3. This analysis shows that the evolution token cost is below 30% of the main execution token cost in most settings. In some cases, improvements to the harness can offset this additional cost. For example, on GDPval, DeepSeek with the mini harness requires 338.63 M tokens, while miniFlux requires 267.87 M tokens, including the evolution cost.

We also report main-agent efficiency in a more budget-comparable setting with the reflection-only baseline. The results show that miniFlux achieves better efficiency in this comparison, highlighting the importance of harness component and workflow updates.

ModelHarnessARC3 Act/LevelGDPval Step/Task
DeepSeek-V4-FlashReflection-only (online reflection)312.1336.22
DeepSeek-V4-FlashminiFlux (online evolution)225.7034.61
AuthorsOpenReview ↗

Response to Reviewer eqH4 #2

W3: Limited novelty and comparison with recent work.

We agree that recent parallel work has explored online harness evolution, and we discuss these approaches in Section 2. Here, we highlight the key distinctions of our approach and provide additional comparison results with a closely related method.

The main distinctions concern (1) the evolution surface, i.e., what can be evolved, and (2) the evolution cycle, i.e., how evolution is performed. We summarize closely related work below. In short, our method focuses on within-task adaptation, supports a wide range of evolvable components, and enables a flexible agentic evolution cycle.

  • TTHE [1] (July 2026) evolves task-harness source code through a separate agentic loop after candidate batch executions.
  • Self-Healing Harness [2] (Sept. 2026) evolves natural-language behavioral rules within a task, with evolution interleaved with the task execution loop.
  • Continual Harness [3] (May 2026) evolves the system prompt, sub-agent definitions, skills, and memory through separate fixed-pass LLM refinement within a game episode.
  • miniFlux (ours, Sept. 2026) evolves the runtime prompt, tools, middleware, and context management through separate agentic loops with dedicated harness-management tools within a task.

We have conducted comparison experiments with Continual Harness, implemented within Prime Agent, using GLM-5.3. The results are shown below, with each experiment repeated 3 times:

ModelHarnessARC-AGI-3 (Score)ARC-AGI-3 (Levels)TetrisGDPval
GLM-5.3mini baseline31.53 ± 7.0785.33 ± 8.50150.44 ± 22.9277.24 ± 0.37
GLM-5.3Prime Agent (w/ Continual Harness)43.50 ± 2.78104.00 ± 3.0096.11 ± 61.9573.23 ± 1.68
GLM-5.3miniFlux46.71 ± 6.53112.00 ± 10.15214.56 ± 47.4383.58 ± 2.76

miniFlux outperforms the Continual Harness baseline on all three benchmarks, supporting the advantages of our design.

We will incorporate the above analyses and results, along with further experiments, into the revised version of our paper.

[1] Nie, Jun, et al. "TTHE: Test-Time Harness Evolution." arXiv preprint arXiv:2607.08124 (2026).

[2] Tayebati, Sina, et al. "Self-healing harness for runtime oversight of agent self-modification." arXiv preprint arXiv:2609.24130 (2026).

[3] Karten, Seth, et al. "Continual harness: Online adaptation for self-improving foundation agents." arXiv preprint arXiv:2605.09998 (2026).

Dear Reviewer eqH4,

Thank you again for your thoughtful and constructive comments. As the discussion phase is approaching its end, we would appreciate any further feedback you may have. We have kept our responses concise and focused on addressing your concerns to make them easy to review. We would be grateful to hear whether our responses have addressed your concerns or if any points need further clarification.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.