- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper introduces an online self-evolving method for AI-Agent harness, with mini-swe-agent(mini) as the baseline. Within each task attempt(without final test outcome), the same frozen base model alternates between two roles: a main executor working on the task, and an online harness evolver that regularly observes the growing trajectory and updates the harness by addressing issues identified from the ongoing execution process. Experiments of DeepSeek-V4-Flash and GLM-5.2 on three benchmarks show that miniFlux surpasses the baseline mini-swe-agent in all six conditions and outperforms Codex in certain settings. While the online evolver adds some cost, miniFlux manages to reduce the execution tokens under certain settings.
Critical strengths
The method is cleanly designed, the experiments are carefully conducted and the results are supportive. The paper evaluates not only final task performance but also cost efficiency, showing that online harness evolution can improve task outcomes, while maintaining resource efficiency in certain settings like GDPval. These results make the approach potential for building more capable and efficient agents.
Critical weaknesses
- The current evaluation does not sufficiently disentangle the benefit of online evolution from the value of adding missing capabilities to a minimal harness like mini-swe-agent. Many evolved components (e.g. web tools and validation middleware) are already standard capabilities in modern agent systems. A stronger experiment would compare online evolution against a fixed harness equipped with the same capabilities, or investigate whether online evolution can further improve a mature harness, such as Codex, rather than mainly demonstrating that a minimal agent benefits from acquiring additional tools.
- It appears that the effectiveness of online evolution relies largely on long-horizon task trajectories. For tasks that naturally require few steps, the evolver may not have sufficient opportunities to observe and propose any meaningful updates.
Questions for authors
- Can the authors provide additional analysis or ablations to disentangle the benefit of online harness evolution from the benefit of adding new capabilities?
- How does miniFlux behave on short-horizon tasks where only a few execution steps are available? Do the authors have evidence that the approach remains useful when the evolver has limited opportunities to observe execution trajectories and perform updates? More generally, how does miniFlux performance scale with available trajectory length?
3 comments
Dear Reviewer 43Va,
Thank you again for your thoughtful and constructive comments. As the discussion phase is approaching its end, we would appreciate any further feedback you may have. We have kept our responses concise and focused on addressing your concerns to make them easy to review. We would be grateful to hear whether our responses have addressed your concerns or if any points need further clarification.
Thank you for your time and consideration.
Appreciate your response. I prefer to maintain my rating.
Response to Reviewer 43Va
We deeply thank Reviewer 43Va for the positive and constructive feedback, particularly the recognition of our method’s clean design and the carefully conducted experiments supporting its effectiveness. Below, we address the remaining concerns and questions in detail.
We agree that additional analysis and ablations are needed to disentangle the benefit of online harness evolution from that of adding new capabilities. We have therefore implemented two additional baselines:
miniFlux achieves higher mean performance than both baselines across the reported metrics. These results support the value of online harness adaptation beyond pre-installing the existing capabilities or providing online reflections alone.
We agree that miniFlux’s effectiveness on short-horizon tasks is also important. In the current approach, the evolution interval, which determines the amount of execution history accumulated between periodic evolution triggers, remains a hyperparameter that can be set empirically based on the expected task length. Despite this flexibility, we acknowledge that tasks requiring only one or a few steps offer limited opportunities for online adaptation. However, our method is not restricted to long-horizon tasks.
To investigate this, we conducted additional experiments on relatively short-horizon tasks from the WorkBuddy Web split (a 20 task subset), where GLM-5.2 takes approximately 20 execution steps per task on average.
These results suggest that miniFlux remains useful in this relatively short-horizon setting. The longer evolution interval yields a higher mean score despite fewer evolutions per task, suggesting that allowing more execution history to accumulate before adaptation can be beneficial in this setting.
Thank you again for these insightful questions. We plan to expand the analysis with these further related experiments in the revised version of the paper.