- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether evolved routing and communication skills transfer to multi-agent teams with different sizes or model compositions. Evo2Team combines source skill evolution with target skill selection, configuration search, task-level composition, and repeated confirmation. Experiments on Count-Frequency and AgentsNet examine whether transfer changes execution, improves quality and cost, and remains effective on new instances. The results show that benefits depend on the transfer direction, sometimes cover only a small subset of tasks, and may not persist on new instances despite repeated confirmation.
Critical strengths
S1. The paper addresses a practical question: whether existing skills remain useful when the team executing them changes.
S2. The authors distinguish selected policies from diagnostic outcomes, report negative transfer and fallback usage, and examine both adaptation and deployment costs.
Critical weaknesses
C1. The distinct contribution relative to existing work remains unclear. SkillLens studies how skill utility varies across consuming models and documents negative transfer. SkillOrchestra uses explicit skill and cost modeling for routing, while SkillOpt studies skill evolution through feedback and validation. This paper extends the setting to changes in team size and model composition, but the resulting methodological advance or new empirical insight is not sufficiently established.
C2. The sources of the observed gains remain difficult to isolate. Evo2Team combines source skills with target configuration search, task-level composition, and fallback. The feedback-guided KNN/CORAL controls are useful, but their available operations differ from those of Evo2Team. It therefore remains unclear how much improvement comes from source knowledge versus target optimization.
C3. The evidence for reliability is limited. The new-instance experiment shows that repeated confirmation on fixed instances does not guarantee subsequent success, but graph instances and model responses change simultaneously, preventing attribution of the failures. The small number of independent instances and limited repeated evaluations also make it difficult to determine whether the observed differences reflect stable transfer patterns.
Questions for authors
q1. What is the central new finding about multi-agent skill transfer beyond existing work on skill consumption, routing, and evolution? Which experimental results most directly support this contribution?
Q2. How much of the improvement can be attributed to transferred source knowledge? Can the existing controls or further analysis distinguish these contributions more clearly?
Q3. How stable are the main findings across independent instances and repeated runs? Can the available evidence distinguish sensitivity to instance changes from variability in model responses?
2 comments
C2 / Q2: attribution; C3 / Q3: reliability
C2 / Q2. How much of the gain comes from source knowledge? Within Evo2Team, we split each test gain into a source part (the source policy deployed as is vs. all-large) and an adaptation part (Evo2Team vs. that source policy). Each part has a paired bootstrap 95% CI.
| CF | Source share of the summed gain | Source significant alone | Adaptation significant |
|---|---|---|---|
| Original ladders (14 directions) | 46% | 7/14 | 8/14 |
| New ladder (8 arms)* | 17% | 2/8 | 6/8 |
*Three original tier models have been retired since submission, so the new runs transfer the paper's banks to a new GPT ladder (gpt-5-nano / gpt-4.1-mini / gpt-4.1).
| Example (CF, RMSE gain over all-large) | Total | From source | From adaptation | Role |
|---|---|---|---|---|
| Q16→Q32 | +6.11 | +5.88 | +0.23 | Source dominates |
| G16→Q16 | +1.74 | +1.62 | +0.12 | Source dominates |
| G4→G8 | +0.07 | −0.30 | +0.37 | Adaptation repairs |
| 4→8 agents, seeds 101 / 202 / 303 | +0.23 / −0.01 / +0.23 | −1.13 / −1.03 / −0.05 | +1.36 / +1.02 / +0.28 | Adaptation repairs |
On the new ladder, the source-derived seeds are acceptable on the adaptation split from the start in 7/8 arms. In CF, the final recipes keep almost all deployed source rules, and adaptation mainly recalibrates thresholds and integer parameters.
Matched-operation control. The reviewer rightly notes that the KNN/CORAL controls differ in their available operations. We therefore added full-nosource: the same edit operators, screening, composition, confirmation and gate, with the source bank replaced by a neutral generic vocabulary.
| Evo2Team | full-nosource | |
|---|---|---|
| CF: acceptable at initialization | 7/8 | 5/8 |
| CF: ahead after search (adaptation split) | 4/4 cells, ≈0.06 RMSE | — |
| AgentsNet: acceptable at initialization | 1/4 | 0/4 |
| CF: held-out difference | +0.019 RMSE | — |
| AgentsNet cost saving | From source | From adaptation |
|---|---|---|
| Original ladders | 3.7% | 2.8% |
| New ladder | 4.0% | 14.7% |
Summary. Source knowledge delivers roughly half the gain on the original ladders and a reliable start on the new ladder. Target adaptation delivers the rest and repairs failed transfers. The pre-deployment check (Part 1) measures which case applies.
C3 / Q3. Instance changes vs. response variability. We re-executed every tested frozen policy, and all-large, on the identical test instances. Only the model responses vary.
| Count-Frequency | AgentsNet | |
|---|---|---|
| Effect of response variability | 21% of task-level variance (median; range 11–50%) | 12/57 transferred task runs flip (≈21%) |
| All-large itself | — | Flips 2/15 and 3/15 tasks |
| Outcome labels unchanged | 20/22 arms (both flips at the gate boundary) | 7/10 arms |
The paper reports that 4 of 22 confirmed task records (18%) fail on new graphs, which matches the same-instance flip rate. Response variability alone therefore suffices to explain these failures. Together with R1-Q2, the CF findings are stable, and single-execution AgentsNet results carry a variability floor of about 20%.
In the revision we will:
- Reword the new-instance claim.
- Report AgentsNet success as a rate over repeated executions, with CIs.
- Add this decomposition as a recommendation for evaluating multi-agent transfer.
C1 / q1: contribution
We thank the reviewer for the precise summary and for recognizing that we report negative transfer, fallback usage, and both adaptation and deployment cost.
C1 / q1. Contribution relative to SkillLens, SkillOrchestra and SkillOpt. The main merit of our paper is not a new optimizer. It is a study of what happens to executable multi-agent rules when the team itself changes, in size or in model ladder. We will discuss each cited work in Related Work:
None of these works asks what happens when the team executing the rules changes. That raises four questions: Does a selected rule change what the new team executes? How many tasks does it reach? How much of the source gain survives? Can we tell in advance?
Central findings, and the results that most directly support them:
We will rewrite the contribution statement around these four findings and move their supporting results into the main text.