acceptodds
Under review as a conference paper at ICLR 2027

CyberGraft: Evolving the Harness Design Space through High-Leverage Mechanism Mining

Abstract

Automatic harness evolution improves agents without model retraining through trajectory analysis and candidate evaluation. Yet even with accurate root-cause diagnosis, finding effective revisions in a vast design space requires extensive trial and error. Fixed seed harnesses adapt poorly to heterogeneous tasks, unconstrained edits or rigid module recombination can produce low-value candidates, and useful external mechanisms remain difficult to discover and consolidate for reuse. We propose CyberGraft, a framework that continually evolves the harness design space by constructing, reusing, and curating high-leverage mechanism assets. Each asset combines a carefully engineered, reusable mechanism package with a scoped, quantitatively grounded benefit–risk profile. Diagnosis-guided discovery seeks implementations addressing observed needs, while performance-guided discovery investigates strong external agents for alternatives beyond the current diagnosis. Contribution analysis identifies promising core mechanisms, and dependency-guided reconstruction refines their extraction boundaries through validation. Separate comparisons assess source contribution, reconstruction fidelity, and conditional transfer utility. Evidence-guided retrieval supports task-conditioned seed assembly and subsequent candidate generation. Accepted and rejected trials refine packages and profiles, while unmet needs drive further discovery, accumulating both implementations and evidence for future optimization. On Evo-Bench across Search, Office, and General tasks, CyberGraft achieves an Overall score of 53.6 and an AnytimeVal of 54.8, exceeding native evolution by 7.3 and 4.7 points under matched online resource limits with a prebuilt library. Ablations show that continual discovery and adaptive boundaries improve final performance, contribution screening reduces online cost by 27.6% at the same reported performance, and profile-guided reuse raises gate acceptance by 19.8 percentage points.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLM agents, harness evolution

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WmzoOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CyberGraft, a framework for improving agent harnesses while keeping the task-execution model fixed. The framework acquires mechanisms from external agent implementations, reconstructs the dependencies needed to preserve their behavior, and stores each package together with scoped evidence about utility, execution cost, and failure modes. These assets support task-conditioned seed assembly and subsequent harness evolution, while evaluation feedback updates the library.

Critical strengths
  1. The paper addresses a meaningful bottleneck in automatic harness optimization.

  2. The proposed validation structure is conceptually careful.

  3. The reported improvements cover all three evaluated domains.

Critical weaknesses
  1. The comparison does not fully isolate the benefit of the proposed framework from the additional resources invested in its library. Sections 5.1 and 5.2 explicitly give CyberGraft a prebuilt library while matching online limits. This supports an advantage conditional on having that library, but its construction cost and human effort are not quantified in the provided manuscript. Comparisons against simpler retrieval or modular-reuse approaches with access to the same donor pool would help establish the value of the proposed asset representation and acquisition process. Reporting total costs and their amortization across repeated optimization runs would also clarify the practical efficiency claim.

  2. The experiments do not sufficiently separate initialization benefits from subsequent evolution or demonstrate cumulative reuse across runs. As Section 5.2 acknowledges, AnytimeVal reflects both seed quality and later revisions. The current ablations do not establish how much improvement comes from assembling a stronger initial harness versus improving the subsequent search process. Seed-only evaluations and comparisons starting from a common seed would help resolve this ambiguity. A sequence of optimization tasks comparing a carried-over library with a reset library would directly test the broader claim that accumulated assets make future optimization more effective.

  3. The starting harness and the contributions of initialization and subsequent evolution are insufficiently specified. Section 4.3 introduces a feasible base harness, hbaseh_{\mathrm{base}}, but does not identify its implementation or establish whether CyberGraft and the evolution baseline start from the same harness. Moreover, the reported results combine asset-based seed assembly with subsequent evolution, without separately quantifying their contributions. It is therefore unclear how much of the improvement comes from the underlying starting harness, initialization through asset assembly, or later evolutionary updates. The paper should specify the starting implementations and report performance at the base-harness, assembled-seed, and final-evolved-harness stages. A comparison using the same starting harness would help isolate the benefit of the proposed optimization process.

  4. The human-designed harness baselines lack sufficient design and configuration details. Table 1 identifies MiroFlow, Stirrup, and Claw-Eval as the baselines for Search, Office, and General, respectively, but does not adequately describe the concrete workflows and components used in the reported experiments. The manuscript should explain their prompting and planning strategies, tool interfaces, memory and context management, recovery mechanisms, and domain-specific components, including any benchmark-specific adaptations. Since these results are taken from Evo-Bench, the authors should identify the exact versions and configurations underlying those results. These details are needed to understand the capabilities against which CyberGraft is compared and to reproduce the comparison.

Questions for authors
  1. What exact implementation and version are used for hbaseh_{\mathrm{base}}?
  2. What concrete methods, workflows, and components constitute the MiroFlow, Stirrup, and Claw-Eval harnesses used in Table 1?
Reviewer_tZuuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

CyberGraft improves agent harnesses by extracting common mechanisms (e.g., code snippets) from existing implementations and storing them as reusable assets. With each asset we maintain a benefit-risk profile tracking its historical performance. To verify that extracted mechanisms work correctly in a target harness, the framework uses a three-part validation approach. On Evo-Bench, CyberGraft scored 53.6 across Search, Office, and General tasks, vs 46.3 for native evolution, by drawing on a prebuilt library of 45 assets while operating under the same online resource limits.

Critical strengths
  • Evaluating agents is computationally expensive. Archiving verified code alongside test evidence to avoid re-evaluating known components seems like a practical approach.
  • The 3-way validation procedure (evaluating source contribution, fidelity, and transfer utility) is designed to filter out low-quality or non-functional code.
  • The proposed method outperforms baselines, and the ablation study analyzes how some of the components worked well or not.
  • The paper reports promising results across domains. In particular, it reports gains over native evolution cover Search (+7.3), Office (+9.3), and General (+3.1). Removing continual discovery lowers Overall by 4.5 points (Table 2), supporting its contribution beyond the initial library.
Critical weaknesses
  • While the paper emphasizes downstream efficiency gains, it does not provide the initial cost (i.e., compute, time, or API spend) required to construct the asset library. While CyberGraft starts with 45 validated assets, the native evolution does not. There is no baseline using the same assembled seed or asset library. This makes the comparisons hard to interpret.
  • The mechanisms described in Appendix A are already used in MiroFlow, Stirrup, and Claw-Eval, which are the same agents used as baselines. Outperforming baselines by using their best mechanisms does not fully demonstrate the framework’s generality.
  • The paper does not compare the predicted scores with actual performance. In addition, Equation 4 omits trial counts, which could allow assets with small sample sizes to receive disproportionately high rankings.
  • The submission does not discuss some of the literature on automated software transplantation (e.g., Barr et al., ISSTA 2015; CodePhage, Sidiroglou-Douskos et al., PLDI 2015) which could be relevant.
Questions for authors
  • What was the total resource cost (API costs, compute time, human effort) to construct the initial library?
  • What was the held-out performance of the seed harness (h0) prior to evolution?
  • Did you use Evo-Bench dev tasks used to validate library assets? If so, how do you mitigate risk of overfitting to the dev set?
  • How closely did profile-predicted score improvements correlate with actual performance after integration? Also, could you provide standard deviations for the benchmark results in Tables 1 and 2?
  • Please provide the validation results for a complete reconstructed asset, including the actual retrieval/search/gate settings, and explicitly describe the manual intervention.
  • In general, please report per-run results and confidence intervales for the native evolution under the same execution conditions. Were libraries and profiles isolated across independent optimization runs?
Reviewer_2j4GOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper tries to introduce more structure in the harness optimization loop. From what I understand, the idea is to use existing parts from external harnesses and use them to guide the optimization loop of a harness. The authors propose CyberGraft which is a meta-harness that takes into account individual nuances of each part, and integrates external harness components in their place. The main contribution of this work is the idea of utilizing specific components from external harnesses. The authors evaluate CyberGraft on Evo-Bench reporting improvements over the meta-harness that was used as a baseline in the original benchmark. Overall, the paper reads more like software documentation than a research paper: it focuses heavily on describing different components of the framework but does not give intuition or explain concretely why they are necessary and what they actually do.

Critical strengths

(i) The paper introduces more structure in the harness optimization loop: it provides groupings and abstractions (e.g., assets and profiles, packages and mechanisms, etc.) in order to find how to optimize individual parts of the harness in the correct way. As the authors demonstrate, this improves end-to-end harness optimization (due to more targeted modifications) rather than hoping that a model can provide useful feedback for everything at once.

(ii) The problem of how to use parts from external harnesses in order to guide the harness optimization loop is timely and interesting.

Critical weaknesses

(i) I found the paper particularly difficult to follow. There are a lot of moving parts inside the harness optimization loop, and the paper has tried to go over most of them but it was difficult for me to keep up (e.g, mechanisms, packages, assets, dependency graphs, profiles, donors etc.). I also had difficulty distinguishing between which components of CyberGraft are novel, which build on prior work, and the motivation behind them. The paper focuses heavily on documenting the different components of the framework (almost like a technical documentation for a software package), but doesn’t provide enough intuition for why each component is necessary, and what it does in a concrete way. I believe that providing concrete examples and explanations would significantly improve the paper.

(ii) The experimental setup is limited. Although CyberGraft is proposed as a harness optimization method, its evaluation only compares against a meta-harness created as a baseline for the Evo-Bench benchmark. It would be more convincing to showcase improvements over native meta-harnesses (e.g., DGM, Meta-harness, etc.), as well as approaches such as ModularRSI and HarnessBank which are quite similar to CyberGraft. The authors appear to be aware of all of them as they are included in their related work.

Questions for authors

(i) In the evaluation you sourced 23 external harnesses. What criteria were used for the selection?

(ii) In 3.1, what are “external dependencies”, and what are “host bindings”? In 4.1, what are “profiles”? What is the “Graph” in Figure 2? In 4.2, what are “mechanism semantics”? In 4.3, what is the “domain-subdomain index”?

(iii) In the experiments, CyberGraft uses GPT-5.6 Sol, Qwen3.7-Plus, and DeepSeek-V4-Flash, but the Evo-Bench baseline only mentions GPT-5.6 Sol. Did you use the same model setup for both? If not, why?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.