- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Co-Harness, which evolves several multimodal agent harnesses through shared task trajectories and maintains a memory of earlier harness edits. On four benchmarks, it reports higher Average@4 and Pass@4 than the strongest harness baseline, Meta-Harness, for all three tested task models. The comparison is promising: with Gemini-3-Flash, average Average@4 rises from 43.22 to 47.53.
Critical strengths
The work addresses a practical question: how to improve tool-using multimodal agents while keeping model weights fixed. It compares against both context-level methods and a harness-level method, includes component ablations, and tests transfer to another model and an unseen benchmark.
Critical weaknesses
- The concrete harness implementation is insufficiently specified. Section 3 defines a harness as a stateful program managing context, tool use, and the agent loop, but Appendix A.1 mainly describes tool interfaces and Appendix A.3 provides initial prompts. It remains unclear what the actual editable harness contains, which components the optimizer is permitted to modify, and what a representative successful edit looks like. These are central details for assessing and reproducing a method whose main contribution is harness evolution. The inference procedure also needs clarification: the paper describes possible composition of multiple harnesses (§1; Figure 2), while the provided router prompt selects exactly one (Appendix A.3).
- The experiments do not isolate the benefit of collective learning from additional capacity and optimization effort. Co-Harness starts with three domains and two experts per domain, whereas Meta-Harness evolves one shared harness (§4.1; Appendix B.1). Table 2 shows that adding a mixture already accounts for a large part of the reported gain. However, the paper does not provide a comparison with multiple independently evolved harnesses under a matched evolution budget, or report enough aggregate optimization cost to assess this alternative explanation. Such a control is important for attributing the improvement to experience sharing.
- The evidence for the proposed mechanisms, and consequently their algorithmic novelty, is limited. The method combines multiple specialized harnesses, shared trajectories, and optimizer memory. These are reasonable design choices, but the paper does not clearly demonstrate that experts develop complementary capabilities. In particular, the proposed EHD “loss” is described as an implicit textual judgment by the optimizer (§3.2, Eq. 4), and its separate contribution is not isolated in Table 2. A direct EHD ablation and an analysis of expert specialization would help establish what the proposed mechanism adds beyond a larger harness search space.
Questions for authors
The empirical improvements are also encouraging. My main concern is that the paper does not yet specify its central engineering artifact clearly enough, while the current comparisons leave open whether the gains arise from collective evolution or from greater harness capacity and optimization effort. A concrete implementation description and a budget-matched independent-harness control could substantially my assessment.