- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CyberGraft, a framework for improving agent harnesses while keeping the task-execution model fixed. The framework acquires mechanisms from external agent implementations, reconstructs the dependencies needed to preserve their behavior, and stores each package together with scoped evidence about utility, execution cost, and failure modes. These assets support task-conditioned seed assembly and subsequent harness evolution, while evaluation feedback updates the library.
Critical strengths
-
The paper addresses a meaningful bottleneck in automatic harness optimization.
-
The proposed validation structure is conceptually careful.
-
The reported improvements cover all three evaluated domains.
Critical weaknesses
-
The comparison does not fully isolate the benefit of the proposed framework from the additional resources invested in its library. Sections 5.1 and 5.2 explicitly give CyberGraft a prebuilt library while matching online limits. This supports an advantage conditional on having that library, but its construction cost and human effort are not quantified in the provided manuscript. Comparisons against simpler retrieval or modular-reuse approaches with access to the same donor pool would help establish the value of the proposed asset representation and acquisition process. Reporting total costs and their amortization across repeated optimization runs would also clarify the practical efficiency claim.
-
The experiments do not sufficiently separate initialization benefits from subsequent evolution or demonstrate cumulative reuse across runs. As Section 5.2 acknowledges, AnytimeVal reflects both seed quality and later revisions. The current ablations do not establish how much improvement comes from assembling a stronger initial harness versus improving the subsequent search process. Seed-only evaluations and comparisons starting from a common seed would help resolve this ambiguity. A sequence of optimization tasks comparing a carried-over library with a reset library would directly test the broader claim that accumulated assets make future optimization more effective.
-
The starting harness and the contributions of initialization and subsequent evolution are insufficiently specified. Section 4.3 introduces a feasible base harness, , but does not identify its implementation or establish whether CyberGraft and the evolution baseline start from the same harness. Moreover, the reported results combine asset-based seed assembly with subsequent evolution, without separately quantifying their contributions. It is therefore unclear how much of the improvement comes from the underlying starting harness, initialization through asset assembly, or later evolutionary updates. The paper should specify the starting implementations and report performance at the base-harness, assembled-seed, and final-evolved-harness stages. A comparison using the same starting harness would help isolate the benefit of the proposed optimization process.
-
The human-designed harness baselines lack sufficient design and configuration details. Table 1 identifies MiroFlow, Stirrup, and Claw-Eval as the baselines for Search, Office, and General, respectively, but does not adequately describe the concrete workflows and components used in the reported experiments. The manuscript should explain their prompting and planning strategies, tool interfaces, memory and context management, recovery mechanisms, and domain-specific components, including any benchmark-specific adaptations. Since these results are taken from Evo-Bench, the authors should identify the exact versions and configurations underlying those results. These details are needed to understand the capabilities against which CyberGraft is compared and to reproduce the comparison.
Questions for authors
- What exact implementation and version are used for ?
- What concrete methods, workflows, and components constitute the MiroFlow, Stirrup, and Claw-Eval harnesses used in Table 1?