- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Causal Promotion Gates (CPG), a framework for deciding whether agent-generated skill updates should be accepted or not. The method compares current and candidate skills using paired evaluations, confidence bounds on utility and safety, and provenance-based validation splits. Based on these, it decides whether to promote, reject, or defer the updates. The authors evaluate CPG on two self-evolving agent pipelines with several task families and constructed candidate updates.
I overall find the novelty of the work quite limited and the evaluation rather misleading. The method seems to mostly combine existing statistical evaluation techniques into a skill promotion procedure with some hyped up naming. Regarding the evaluations, I have concerns regarding the constructed benchmarks and relatively weak baselines, which make it hard to know how much the proposed method actually improves over existing approaches.
Critical strengths
-
The problem of deciding which agent-generated skill updates to keep is relevant (esp. for agents with persistent skills).
-
The experimental results includes multiple task families, model scales, baselines, and ablations. The authors also report validation costs and some robustness results.
-
I appreciate the use of executable verifiers rather than relying entirely on LLM judges. The explicit DEFER decision also makes sense.
Critical weaknesses
-
I struggle to see what is actually new here. The main method seems to be held-out A/B testing with confidence bounds and safety constraints (plus a defer option). All fairly standard ideas. Existing work on high-confidence policy improvement and Seldonian algorithms already studies similar acceptance rules. Also, SkillOpt (Yang et al., 2026) directly studies validation-gated skill updates, but doesn't seem to be cited here (why???).
-
The evaluation baselines seem weak to me. For instance, Held-Out A/B explicitly excludes paired randomness, provenance isolation, and worst-group constraints (which are exactly what CPG is designed around). So the improvements seem somewhat expected. Why not compare against a properly configured statistical A/B test with these safeguards also in place?
-
Also not particularly convinced by the constructed benchmark. The candidate classes seem tailored to the exact failure modes CPG targets (e.g., provenance-specific proxies and subgroup harms). Again, how much does this tell about real agent-generated updates in the real world?
-
I don't see much evaluation of more complicated real-world settings, such as interacting skill updates, repeated promotions, or safety failures outside the predefined verifiers. Everything is a bit too much controlled.
Questions for authors
-
See my questions in my weaknesses section.
-
Why have the authors not included direct comparisons with related work like SkillOpt and GRASP (Moll, et al.)? Can you provide a thorough review and direct comparison of/with the related work?
-
What is the main methodological novelty beyond existing high-confidence policy improvement and statistical A/B testing methods?
-
Again, why not include a stronger A/B baseline (and even other non-trivial baselines) with paired evaluation, confidence bounds, and safety constraints?
-
What happens with naturally generated skill updates or multiple interacting updates? Any reason these weren't evaluated? Can you provide results for them?