acceptodds
Withdrawn from ICLR 2027

Co-Harness: Collective Harness Evolution for Agentic Multimodal Large Language Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing multimodal large language model (MLLM) agents generally rely on predefined contexts, fixed tool-use policies, and static orchestration to support long-horizon reasoning and tool use, but these rigid execution mechanisms limit their ability to adapt to new and diverse multimodal agentic tasks. In this work, we aim to improve MLLM agents through harness evolution, which abstracts generalizable experience from successful and failed execution trajectories to refine model harness for related but previously unseen tasks. % To this end, we propose Co-Harness, a new framework that introduces collective learning into multimodal harness evolution to enhance the long-horizon reasoning and tool-use capabilities of multimodal agents. The core idea of Co-Harness is to jointly improve a set of specialized and complementary harnesses through shared experience and reciprocal feedback. Specifically, Co-Harness introduces two collective harness evolving methods: Collective Mixture-of-Harness Evolution (CoMixHE) and Memory-guided Collective Harness Co-Evolution (MemCoHE). CoMixHE collectively evolves specialized and complementary harnesses across and within task domains, expanding harness capacity and broadening the search space of harness optimization. MemCoHE jointly evolves the task model's harnesses and the optimization model's memory, enabling the optimizer to reuse past refinement experiences and adapt more effectively to changing failure patterns. % Extensive experiments on multiple multimodal agentic benchmarks demonstrate the effectiveness of Co-Harness and its transferability across tasks and models.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multimodal Large Language Model;Multimodal Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rY9EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Co-Harness, which evolves several multimodal agent harnesses through shared task trajectories and maintains a memory of earlier harness edits. On four benchmarks, it reports higher Average@4 and Pass@4 than the strongest harness baseline, Meta-Harness, for all three tested task models. The comparison is promising: with Gemini-3-Flash, average Average@4 rises from 43.22 to 47.53.

Critical strengths

The work addresses a practical question: how to improve tool-using multimodal agents while keeping model weights fixed. It compares against both context-level methods and a harness-level method, includes component ablations, and tests transfer to another model and an unseen benchmark.

Critical weaknesses
  1. The concrete harness implementation is insufficiently specified. Section 3 defines a harness as a stateful program managing context, tool use, and the agent loop, but Appendix A.1 mainly describes tool interfaces and Appendix A.3 provides initial prompts. It remains unclear what the actual editable harness contains, which components the optimizer is permitted to modify, and what a representative successful edit looks like. These are central details for assessing and reproducing a method whose main contribution is harness evolution. The inference procedure also needs clarification: the paper describes possible composition of multiple harnesses (§1; Figure 2), while the provided router prompt selects exactly one (Appendix A.3).
  2. The experiments do not isolate the benefit of collective learning from additional capacity and optimization effort. Co-Harness starts with three domains and two experts per domain, whereas Meta-Harness evolves one shared harness (§4.1; Appendix B.1). Table 2 shows that adding a mixture already accounts for a large part of the reported gain. However, the paper does not provide a comparison with multiple independently evolved harnesses under a matched evolution budget, or report enough aggregate optimization cost to assess this alternative explanation. Such a control is important for attributing the improvement to experience sharing.
  3. The evidence for the proposed mechanisms, and consequently their algorithmic novelty, is limited. The method combines multiple specialized harnesses, shared trajectories, and optimizer memory. These are reasonable design choices, but the paper does not clearly demonstrate that experts develop complementary capabilities. In particular, the proposed EHD “loss” is described as an implicit textual judgment by the optimizer (§3.2, Eq. 4), and its separate contribution is not isolated in Table 2. A direct EHD ablation and an analysis of expert specialization would help establish what the proposed mechanism adds beyond a larger harness search space.
Questions for authors

The empirical improvements are also encouraging. My main concern is that the paper does not yet specify its central engineering artifact clearly enough, while the current comparisons leave open whether the gains arise from collective evolution or from greater harness capacity and optimization effort. A concrete implementation description and a budget-matched independent-harness control could substantially my assessment.

Reviewer_yWCLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Co-Harness, which evolves an external harness using execution trajectories and outcome feedback while keeping the model parameters frozen. The authors compare the approach on four major multimodal agent benchmarks. The results show that the overall system has a performance advantage, but key implementation details and the attribution of the gains still require further clarification.

Critical strengths
  1. The paper focuses on improving execution programs with a frozen model. By combining expert specialization, shared execution experience, and optimizer meta-memory, it offers a valuable approach to system design with practical relevance for improving execution on multimodal tool-use tasks.

  2. For Gemini-3-Flash, the mean Avg@4 across the four benchmarks increases from 43.22% with Meta-Harness to 47.53%; after direct transfer to GPT-5-mini, Co-Harness still outperforms the comparison methods. Cross-benchmark transfer also yields positive results, providing empirical support for the potential value of the method.

Critical weaknesses
  1. The descriptions of expert construction and the routing procedure are incomplete. Regarding the domain-specific harnesses, the manuscript states “all of which share the same base initialization,” but does not explain how domains are defined, how experts within each domain are initialized, or how training tasks are assigned.
  2. Validation of EHD and expert complementarity is insufficient. EHD relies on the optimizer to “implicitly assess these differences,” but no specific criteria or prompts for assessing the differences are provided. There is also no ablation that removes EHD alone or any measurement of expert complementarity. The existing results do not sufficiently establish the independent contribution of this mechanism.
  3. The contribution of evolving the complete harness has not been isolated. The authors claim to modify “task prompts, tool-policy and agent loop,” but the manuscript does not present actual examples of changes before and after evolution. I am unclear about how the prompts change through optimization and how the distribution of tool-calling behavior changes.
  4. Disclosure of the optimization process and total cost is incomplete. Although the authors report eight rounds of evolution, the candidate evaluation procedure, and execution limits, the number of candidates per round, the scope of trajectories used by the optimizer, and the scheduling of expert updates remain unclear. The total number of calls, token consumption, and elapsed time are also not reported.
Questions for authors
  1. On page 5, lines 257–258, “each harness’s own execution trajectories” is followed by Tt(¬d)\mathcal T_t^{(\neg d)}. Should this instead be Tt(d)\mathcal T_t^{(d)}? Please clarify the correspondence between a harness’s own experience and cross-domain experience so that the text is consistent with Equation (3).
  2. Editorial correction: the subsection title on page 4, line 209, is written as “Co-Harnesss.” Please confirm and correct it to “Co-Harness.”
Reviewer_Q26oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Co-Harness, a framework that evolves the harness of a multimodal LLM agent without updating model weights. CoMixHE maintains a mixture of harnesses: several domain-specific harnesses with a router that assigns each task to one of them. The optimization agent refines each harness using its own trajectories together with those of the other experts and domains, and it is prompted to keep the experts complementary. MemCoHE gives the optimization agent a persistent meta-memory that records successful edits, failed edits and plans for the next round. Experiments with Gemini-3-Flash and Gemini-2.5-Pro show consistent gains over context-level methods such as XSkill and over Meta-Harness. Harnesses evolved also transfer to other models and datasets

Critical strengths
  1. Co-Harness beats the strongest context-level and harness-level baselines on all four benchmarks and with all three task models. The gains hold up when the harness is transferred to a different model and to an unseen benchmark. Because only the harness evolves and the weights stay fixed, the approach is easy to apply to closed models.
  2. The round-by-round curves show that Meta-Harness regresses after reaching its peak while Co-Harness keeps improving, which supports the motivation for an optimization memory. The ablations cover each component as well as the initial number of domains and experts.
Critical weaknesses
  1. Co-Harness evolves several harnesses in parallel, with multiple experts per domain. Therefore, it likely uses more rollouts and optimizer calls per round than Meta-Harness, so part of the gain may come from more search rather than a better method. The inference cost of routing is not reported either.
  2. On VisualToolBench, the mixture of harnesses alone performs roughly on par with Meta-Harness.
  3. The method combines known ideas: Meta-Harness-style code-level harness search, a mixture-of-experts-style router, and a memory file for the optimizer. And the multimodal aspect comes only from the benchmarks.
  4. The Expert Harness Discrimination Loss is a prompt that asks the optimizer to keep experts different, not a loss that is computed or optimized. The paper never measures whether the experts actually become complementary.
Questions for authors
  1. What is the total evolution and inference cost of each method? How does Co-Harness compare with Meta-Harness under a matched budget, for example Meta-Harness given more rounds or several independent single harnesses?
  2. Can you report results over several independent evolution runs, at least for the ablation, to show that the gain from each component exceeds run-to-run variation?
  3. How are domains and experts defined and routed in practice? Can you show evidence that the experts specialize, for example routing statistics or per-expert accuracy on different task types?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.