- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes CMRF, an evolutionary framework for reusable agent skills across heterogeneous model–harness configurations. Source hosts maintain separate populations, and selection combines mean skill gain, cross-host variance, weak-host gain, worst-host gain, and probe-host transferability. Migrants receive reproductive credit for improving their destination relative to no-skill and local-population references. Skills are separated into a platform-independent core, a model adapter, and a harness invocation layer. Evaluation uses a 48-task verifier-backed suite with a 24/8/16 evolution/development/test split, four Qwen2.5 source configurations, and four additional model–harness configurations. Under a 768 candidate-host evaluation budget and five seeds, the paper reports higher source and transfer pass rates than single-host, pooled-average, and worst-case selection methods, together with ablations of migration, fitness terms, and layered representation.
Critical strengths
- The motivation is consequential and receives experimental support. Table 1 shows that single-host optimization reaches a 43.7% source mean but only 30.9% on the worst host; pooled-average selection improves the mean to 45.8% while the worst host remains at 35.8%. These comparisons make the heterogeneity concern concrete. CMRF's explicit weak/worst-host objectives and separation of core procedures from invocation details respond directly to the problem, with the full system improving the reported worst-source score to 40.7%.
- The experiment suite includes meaningful baselines, outcomes, and feasibility controls. Single-host, pooled-average, worst-case, and Pareto selection are relevant alternatives to the proposed objective. The four-domain task suite uses functional verifiers, so success reflects completed requirements rather than stylistic preference. Migration, fitness-term, representation, frequency, and budget studies test several parts of the idea beyond the main result. These provide useful evidence for robustness across the evaluated configurations, although they do not establish every claimed mechanism or unseen-host generalization.
Critical weaknesses
- The transfer evaluation does not isolate the generalization problem stated in the motivation. The paper seeks skills that remain useful beyond the hosts under which they are optimized. However, Equations 6–7 put probe-host performance U(s) into reproductive fitness, with coefficient 0.25, and Algorithm 1 evaluates candidates on those hosts before selection. Appendix B.1 describes the four transfer configurations as those probe configurations. Their feedback therefore influences which cores survive even though they generate no mutations or parents. The task split and frozen-core transfer are useful controls, but cannot establish generalization to a host absent from the selection objective. Functional verifiers make the reported performance credible on the measured configurations; the methodological issue is that the experiment addresses new tasks on selection-exposed hosts rather than the claimed host-independent transfer. Code inspection confirms this interpretation:
src/core_method.pyuses probe gain in both fitness and nondominated archive construction, andsrc/pipeline.pyincludes these same probe hosts in final evaluation. The code therefore clarifies the protocol rather than resolving host-independent generalization. - The necessary ablation for the named reproductive-credit idea is absent from the otherwise substantial experiment suite. Table 3 tests migration, weak/worst-host terms, variance, and layered skills, but no-migration removes both exchange and destination credit. Removing the weak-host term does not isolate the destination reproductive credit R_h defined in Equations 8–9. Thus, these controls cannot determine whether reproduction history improves over ordinary island migration using the same multi-host fitness. This is consequential because the monolithic ablation alone lowers the reported transfer worst-host score from 38.4% to 33.8%, showing that representation is a major source of benefit. The relevant baseline is a layered, multi-host island optimizer with identical host feedback, migration schedule, and evaluation budget, but zero R_h. Without that comparison, the system-level gains do not establish the distinct mechanism emphasized in the title and motivation. The supplied demo explicitly computes destination credit and feeds accumulated credit back into later parent selection, so the mechanism is implemented. However, simulator success probabilities already encode benefits of core quality, adapters, and invocation compatibility. Its outputs cannot replace a matched zero-credit control on the paper's agent tasks, and its pooled-baseline search budget is also different from CMRF's multi-island budget. These observations concern what the demo establishes; they do not invalidate the separately reported benchmark numbers.
- Writing quality: repeated defensive self-validation and excessive emphasis on protocol boundaries. The introduction asserts that the experiments separate adaptation from sampling and post hoc transfer; Section 4.1 claims the protocol “isolates zero-shot portability” and excludes “hidden evaluation feedback.” Sections 3.4 and the appendices repeatedly emphasize that probe hosts do not generate mutations or parents and that transfer involves no task-specific refinement. The conclusion again claims the ablations “verified the contributions” of the proposed components. These repeated assurances and exclusions give the manuscript an overly defensive, preemptive-rebuttal tone and can create the impression of protesting too much. They also obscure the distinction between the intended controls and the fact that probe scores influence selection. This is the consequence of the author's excessive reliance on GPT writing, and the author should optimize these writing sections to make their paper statements more persuasive, thereby improving the quality of their writing.
Questions for authors
- The demo confirms that probe gain enters selection and archive construction. In the reported benchmark, which final hosts are entirely excluded from these feedback routes? Does the advantage persist on such hosts when every baseline receives the same selection-host access? A task split on selection-exposed hosts does not answer this question.
- With layered skills, multi-host fitness, probe feedback, migration, and evaluation budget fixed, how does CMRF compare with setting destination credit R_h to zero on the actual agent tasks? Does cumulative credit improve over using only current destination gain? The demo implements credit, but does not isolate its contribution under the benchmark conditions.