- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Counterfactual Evolution Credit (CEC), a framework for selecting skills in self-evolving LLM agents. It uses counterfactual comparisons to estimate each skill’s contribution to task performance and guide skill selection. Experiments on three benchmarks show improvements over existing baselines.
Critical strengths
- The paper addresses an important problem in skill selection: whether a skill actually contributes to task success, rather than merely being invoked during a successful execution.
- The reported results show modest improvements over the baselines.
Critical weaknesses
- The method compares executions after removing a skill, replacing it with its parent, blocking its invocation, or modifying an individual module. The technical novelty appears limited, as the method mainly combines several straightforward interventions.
- TI and FS are defined as the proportions of selected skills with positive and non-positive test effects, respectively. By these definitions, they should sum to 100%. However, the values reported for CEC in Table 1 are 74.8% + 8.9% = 83.7%. The authors should clarify this difference.
- The three-round evolution setting provides limited evidence for the claimed evolutionary stability. Longer-term evaluations tracking performance improvements and regressions across rounds would strengthen this claim.
- The experimental evaluation is limited to three benchmarks. Additional evaluations on established agent benchmarks, such as WebArena and AppWorld, would strengthen the evidence for the generalizability of CEC.
- Figure 2 is very difficult to read. The text is too small to be clearly legible, which makes the framework difficult to follow.
Questions for authors
See weakness 1-4.