- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
EvoMoSkill maintains a shared skill, task-family experts, and a router for frozen LLM agents. CreAGD diagnoses failures, proposes skill or routing edits, compares candidate updates, and validates them before changing the library. The paper reports improvements over single-skill baselines on five benchmarks and five student models.
Critical strengths
The paper looks at a practical problem that is easy to understand: a shared update can help one type of task while making another worse. Testing five benchmarks and five models gives the study useful breadth. I also found the acceptance-check ablations helpful.
Critical weaknesses
The main concern is that the experiments do not directly test the motivating claim. There is no controlled study of conflicting updates, and no clear measure of how often an update helps one family while hurting another. The full method also adds expert capacity, task-family structure, routing, several candidate drafts, and guarded acceptance at the same time. Because these factors are bundled together, it is hard to tell whether mixture-based joint evolution is the key reason for the improvement.
Most parts of the method are familiar on their own, and the paper does not include a closely matched dynamic skill-library or routing baseline. In some cases, static partitions or admission-time scopes perform as well as, or better than, dynamic routing. The final results also use only seed 42. Overall, the current experiments do not yet show enough novelty or robustness beyond a careful combination of existing ideas.
Questions for authors
Can the authors test the interference claim directly? For example, report how often a single-skill update improves one family but hurts another, then show whether EvoMoSkill prevents these regressions when skill capacity and optimization calls are matched.
Please compare against a modular skill-library baseline with the same family information, skill-token budget, candidate budget, and validation budget. It would also help to test noisy or inferred family labels, run independent seeds, and report total teacher calls, rollout cost, token use, and wall-clock time.
7 comments
Table R11. Cross-family interference in candidate and deployed updates. Drops and conflicts exceed family-specific noise thresholds. Mean worst-family change covers regressing updates only.
| Benchmark | Model | Method | Update type | Updates | Drops | Conflicts | Mean worst-family change (pp) |
|---|---|---|---|---|---|---|---|
| SearchQA | DeepSeek-V4-Flash | Single | Shared-skill candidate | 39 | 17 | 2 | −4.4 |
| Single | Deployed full step | 19 | 2 | 0 | −2.6 | ||
| EvoMoSkill | Shared-skill candidate | 9 | 3 | 2 | −3.4 | ||
| EvoMoSkill | Expert-text candidate | 39 | 5 | 0 | −2.6 | ||
| EvoMoSkill | Accepted new expert | 9 | 0 | 0 | — | ||
| EvoMoSkill | Deployed full step | 12 | 1 | 1 | −4.8 | ||
| Kimi-K2.6 | Single | Shared-skill candidate | 35 | 9 | 0 | −6.8 | |
| Single | Deployed full step | 19 | 3 | 0 | −6.7 | ||
| EvoMoSkill | Shared-skill candidate | 7 | 2 | 2 | −7.7 | ||
| EvoMoSkill | Expert-text candidate | 24 | 2 | 1 | −4.8 | ||
| EvoMoSkill | Accepted new expert | 4 | 1 | 1 | −6.0 | ||
| EvoMoSkill | Deployed full step | 10 | 4 | 4 | −6.7 | ||
| ALFWorld | Qwen3-32B | Single | Shared-skill candidate | 4 | 1 | 0 | −15.7 |
| Single | Deployed full step | 1 | 0 | 0 | — | ||
| EvoMoSkill | Shared-skill candidate | 4 | 1 | 0 | −12.5 | ||
| EvoMoSkill | Expert-text candidate | 18 | 5 | 4 | −11.6 | ||
| EvoMoSkill | Accepted new expert | 4 | 3 | 3 | −21.2 | ||
| EvoMoSkill | Deployed full step | 4 | 1 | 0 | −11.5 | ||
| Combo | Shared-skill candidate | 4 | 2 | 0 | −11.2 | ||
| Combo | Expert-text candidate | 16 | 6 | 2 | −19.5 | ||
| Combo | Accepted new expert | 3 | 1 | 0 | −12.5 | ||
| Combo | Deployed full step | 2 | 0 | 0 | — |
Table R11 shows three findings.
- Shared-skill revisions can cause family-level regressions, including conflicts that help one family while harming another. On SearchQA with DeepSeek-V4-Flash, 17 of Single's 39 shared-skill candidates cause a family-level regression, or 43.6%. Two also improve one family while harming another. With Kimi-K2.6, 9 of 35 candidates regress, or 25.7%. This directly shows that a local repair to a shared skill can disrupt procedures that already work for other tasks.
- Expert-text candidates have lower family-level regression rates in these two SearchQA settings. With DeepSeek-V4-Flash, 5 of EvoMoSkill's 39 expert-text candidates regress, or 12.8%. With Kimi-K2.6, 2 of 24 regress, or 8.3%. Compared with Single's corresponding rates of 43.6% and 25.7%, expert revisions less frequently harm other families. This supports retaining separate procedures for different tasks and placing family-specific repairs in the corresponding expert.
- Whole-library acceptance further filters regressions in combined updates. On ALFWorld, 1 of the original EvoMoSkill's 4 deployed updates causes a family-level regression. Neither of the 2 deployed updates retained by Combo regresses. Combo still generates candidates that reduce family performance, but its accepted complete updates preserve family performance. Local candidate screening and whole-library execution checks can therefore work together to control the effects of updates.
2. Ablating credit assignment and scope updates (Table R1)
To distinguish the contribution of the expert structure from that of the update mechanism, we retain the same shared-skill and family-expert structure and vary only credit assignment and scope updates. All variants use the same teacher, student model, and data split, together with the same family information, expert-count limit, candidates per step, acceptance procedure, and training schedule. This ablation uses seed 42 throughout. Each variant is trained independently from scratch, selects its best checkpoint on validation data, and is evaluated on the same test tasks.
Here, credit assignment identifies whether a failure calls for a change to the shared skill, an expert's text, or a routing scope, and provides the teacher with the corresponding revision feedback. When it is disabled, the teacher still receives raw failure trajectories and can propose revisions. Scope updates allow an expert's applicable task range to change after its creation. Fixing scopes disables only this adjustment. Skill text continues to be optimized.
- M0: Fixed scopes, no credit assignment. This retains the modular library and measures the joint benefit of the two mechanisms when compared with the full method.
- M1: Editable scopes, no credit assignment. This removes only attribution feedback from the full method to test the contribution of credit assignment.
- M2: Fixed scopes with credit assignment. This removes only scope adjustment from the full method to test the contribution of dynamic scopes.
Full EvoMoSkill retains both mechanisms. We compare all four settings on ALFWorld with Qwen3-32B and compare the full method with M0 on SearchQA using Kimi-K2.6. In Table R1, the 95% CI estimates the full method's success rate minus that of the comparator in the same row. The p-values use paired McNemar tests.
Table R1. Modular ablations with the same task-family information. Bold marks the highest success rate within each setting. CIs estimate EvoMoSkill minus the comparator.
| Benchmark | Model | Method | Scope updates | Credit | Success (%) | 95% CI (pp) | p |
|---|---|---|---|---|---|---|---|
| ALFWorld | Qwen3-32B | EvoMoSkill | Enabled | Enabled | 86.57 | — | — |
| M0 | Fixed | Disabled | 85.07 | [−2.08,+5.07] | 0.6875 | ||
| M1 | Enabled | Disabled | 77.61 | [+2.59,+15.32] | 0.0118 | ||
| M2 | Fixed | Enabled | 84.33 | [−1.61,+6.09] | 0.4531 | ||
| EvoSkill (task-wise selection) | — | — | 75.37 | — | — | ||
| SearchQA | Kimi-K2.6 | EvoMoSkill | Enabled | Enabled | 84.69 | — | — |
| M0 | Fixed | Disabled | 83.98 | [+0.19,+1.24] | 0.0129 |
Credit assignment provides a significant gain with the same expert capacity and family information. On ALFWorld, the full method reaches 86.57%, while M1 without attribution reaches 77.61%, a decrease of 8.96 percentage points, with p = 0.0118. Both allow scope updates and use the same teacher and acceptance procedure. Identifying the update target associated with a failure is therefore more effective than asking the teacher to process unassigned failure trajectories directly.
The full mechanism also outperforms fixed specialization on question answering. On SearchQA, the full method and M0 reach 84.69% and 83.98%, respectively, with p = 0.0129. Targeted content and scope revisions provide an additional gain with the same family information and expert capacity. Together with the replay results in Table R11, these findings show that local experts provide separate places to revise task-specific procedures, while credit assignment makes those revisions more effective. We will include these settings and results in the revision. Thank you again for your suggestions.
Responses to Reviewer jmVC (1)
Responses to Weakness 1 and Question 1
Thank you for your valuable comments!
We appreciate your suggestion to measure how updates affect different task families directly. We added update replay experiments to test whether a change helps one family while harming another, together with modular ablations to distinguish the roles of credit assignment and scope updates.
1. Replaying updates to measure cross-family interference (Table R11)
To measure each update's effects on different families, we re-evaluate the saved libraries and candidates after training. This procedure assesses changes already generated during training without further skill optimization. It has three steps.
ALFWorld uses 140 tasks not involved in candidate acceptance, with two executions per task, Qwen3-32B, and seed 42. SearchQA uses a fixed set of 300 test questions, with three executions per question, covering three training seeds each for DeepSeek-V4-Flash and Kimi-K2.6. We compare Single, which optimizes one shared skill, with EvoMoSkill, which uses a shared skill and family experts. On ALFWorld, we also include Combo, the whole-library acceptance variant whose rollback rules are given in Table R5. All replay evaluations use gpt-5.6-sol for routing.
To avoid counting random fluctuations as regressions, we independently re-evaluate the unchanged old library. For each benchmark and student model, we pool the family-level differences between repeated evaluations across steps. Each family's noise threshold is the 95th percentile of the absolute differences. Only improvements or declines exceeding the corresponding threshold are counted.
In Table R11, Updates counts evaluated changes, and Drops counts changes that cause at least one family to regress. Conflicts requires the same change to improve one family and harm another, with both effects exceeding their respective thresholds. Mean worst-family change is computed only over regressing updates. For each such update, we take the change in the most adversely affected family, then average these changes in percentage points.