- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
EcoSkill jointly evolves skills, routing policies, and library structure. It estimates skill contributions through sampled coalitions and evaluates performance under library growth, skill corruption, and task-distribution shifts.
Critical strengths
Evaluating skills through their interactions addresses a practical weakness of independent skill ranking. The ablations examine several relevant components.
Critical weaknesses
-
Some averages in Table 1 do not match the scores for the three task groups. This makes the reported gains hard to check.
-
It is unclear how skill scores are computed when a group of skills exceeds the context limit. This could affect which skills are kept or removed.
-
The tests use only one base model and libraries of up to 500 skills. It is unclear whether the gains hold for other models or larger libraries
Questions for authors
-
How are the averages in Table 1 calculated? Could you explain why they differ from the individual results?
-
What happens when a sampled group of skills exceeds the context limit? Are skills removed or shortened, or is the group skipped? How does this affect their scores?
-
Do the gains hold with another base model or a larger library? How does the cost of updating the library grow as more skills are added?