acceptodds
Withdrawn from ICLR 2027

Causal Promotion Gates for Secure Self-Evolving Agent Skills

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Persistent skills enable large language model agents to reuse procedures distilled from prior executions. Promoting candidate skill updates to a shared repository requires determining whether persistent encoding improves performance on independent tasks while avoiding safety and subgroup regressions. Existing promotion mechanisms rely on recurrence, model judgments, source-trajectory attribution, or aggregate validation, but do not isolate the effect of persistently encoding a candidate rule. We introduce the Causal Promotion Gate (CPG), which formulates skill promotion as a distribution-level paired causal decision over persistent policy updates. Given a frozen agent policy, a current skill, and an atomic candidate update, CPG treats the skill version as the intervention and compares matched executions with identical task instances and initial states. The protocol excludes validation tasks that share provenance with the evidence used to generate the update, controls reusable randomness, and randomizes treatment order to reduce execution artifacts. The paired outcomes estimate utility, safety, and pre-registered group-specific effects under a fixed validation budget. One-sided confidence bounds, minimum-effect thresholds, and worst-group constraints yield an auditable three-way decision: promote updates with reproducible benefits and acceptable risk, reject clearly harmful updates, and defer updates when the evidence or protocol validity is insufficient. We evaluate promotion accuracy, calibration, deployment effects, heterogeneous harm, robustness, and validation cost on planted-causal benchmarks and through integrations with trajectory-to-skill pipelines. We compare CPG with common promotion and attribution strategies under matched validation budgets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Causal Skill Promotion, Paired Validation, Safety-Aware Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_T5qVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Causal Promotion Gates (CPG), a framework for deciding whether agent-generated skill updates should be accepted or not. The method compares current and candidate skills using paired evaluations, confidence bounds on utility and safety, and provenance-based validation splits. Based on these, it decides whether to promote, reject, or defer the updates. The authors evaluate CPG on two self-evolving agent pipelines with several task families and constructed candidate updates.

I overall find the novelty of the work quite limited and the evaluation rather misleading. The method seems to mostly combine existing statistical evaluation techniques into a skill promotion procedure with some hyped up naming. Regarding the evaluations, I have concerns regarding the constructed benchmarks and relatively weak baselines, which make it hard to know how much the proposed method actually improves over existing approaches.

Critical strengths
  • The problem of deciding which agent-generated skill updates to keep is relevant (esp. for agents with persistent skills).

  • The experimental results includes multiple task families, model scales, baselines, and ablations. The authors also report validation costs and some robustness results.

  • I appreciate the use of executable verifiers rather than relying entirely on LLM judges. The explicit DEFER decision also makes sense.

Critical weaknesses
  • I struggle to see what is actually new here. The main method seems to be held-out A/B testing with confidence bounds and safety constraints (plus a defer option). All fairly standard ideas. Existing work on high-confidence policy improvement and Seldonian algorithms already studies similar acceptance rules. Also, SkillOpt (Yang et al., 2026) directly studies validation-gated skill updates, but doesn't seem to be cited here (why???).

  • The evaluation baselines seem weak to me. For instance, Held-Out A/B explicitly excludes paired randomness, provenance isolation, and worst-group constraints (which are exactly what CPG is designed around). So the improvements seem somewhat expected. Why not compare against a properly configured statistical A/B test with these safeguards also in place?

  • Also not particularly convinced by the constructed benchmark. The candidate classes seem tailored to the exact failure modes CPG targets (e.g., provenance-specific proxies and subgroup harms). Again, how much does this tell about real agent-generated updates in the real world?

  • I don't see much evaluation of more complicated real-world settings, such as interacting skill updates, repeated promotions, or safety failures outside the predefined verifiers. Everything is a bit too much controlled.

Questions for authors
  • See my questions in my weaknesses section.

  • Why have the authors not included direct comparisons with related work like SkillOpt and GRASP (Moll, et al.)? Can you provide a thorough review and direct comparison of/with the related work?

  • What is the main methodological novelty beyond existing high-confidence policy improvement and statistical A/B testing methods?

  • Again, why not include a stronger A/B baseline (and even other non-trivial baselines) with paired evaluation, confidence bounds, and safety constraints?

  • What happens with naturally generated skill updates or multiple interacting updates? Any reason these weren't evaluated? Can you provide results for them?

Reviewer_k8BVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Causal Promotion Gates, CPG, for deciding whether to promote updates to persistent agent skills. The procedure includes comparison of current and target skills through side by side execution on provenance separated tasks across different evaluation strata. It then applies confidence based utility, safety, and subgroup criteria to decide whether the change should be promoted, rejected, or deferred. The authors report lower false promotion rates and improved utility and safety compared to several baselines.

Critical strengths
  1. The paper addresses a real problem where successful trajectories don’t justify conversion into skill additions.
  2. Treating defer as an explicit state that handles statistical ambiguity and protocol invalidity.
  3. Providing informative case studies and ablations, showing how proxy and boundary harm updates fail.
Critical weaknesses
  1. Limited method novelty. CPG is a well designed A/B test, each of provenance-grouped splits, paired seeds, per-stratum checks, Holm correction, etc. is standard experimental practice. The “causal” part is not justified and such framing adds nothing.
  2. Weakened baselines don’t demonstrate the contribution. The main A/B test baseline has provenance exclusion, pairing, stratum checks all removed. The author’s own falsification criterion identifies this but was never tested against a competent design. Ablations showed that removing those brought CPG down to the same level (17.3% and 16.4% vs A/B’s 16.8%).
  3. The procedure assumes strong prerequisites whose failure was never tested. CPG requires provenance groups, evaluation strata, controllable environments, and executable verifiers all known in advance. The limitations section acknowledges this, but no experiments discuss what happens if they fail. The boundary case in D.3 was caught because it was registered in the stratum, what would happen if it’s not is unknown.
  4. Various inconsistencies across the text and presentation issues. To name a few, forced binary decisions should maintain number of promoted updates per its description, yet Table 5 reported FPR change; Appendix A.2 states δ_U=0.05 while Table 10 use 0.02; the strata lists in section 3 and 4.1 both has 5 items but don’t match, appendix A.2 lists 6; results on controlled trajectory poisoning was never demonstrated. There are also citation issues (Gadbail et al., 2024) and a ?? reference. Moreover, the CPG procedure was never given as a clear algorithm, its various assumptions were scattered across section 3 and the appendix, with heavy use of undefined jargon while describing the method. Section 4 recites literal table values without interpreting them.
Questions for authors
  1. How does CPG compare to a standard sequential A/B baseline that uses provenance grouping, per-stratum checks, and shared seeds?
  2. Why does FPR differ in the forced binary case if the promotions are identical?
  3. How does a truly information-matched judge with CPG’s per-stratum paired results and confidence bounds perform? Does it change when run on a stronger model?
  4. How sensitive is CPG to the setup prerequisites? Can you produce evaluation results where provenance groups are noisy, and registered stratums don’t cover all cases perfectly?
  5. Were seeds reproducible under your vLLM inference settings, and how was this verified? What was the total compute used, including the oracle pool generation?
Reviewer_rPyMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes the Causal Promotion Gate (CPG) for deciding whether candidate updates should enter an agent’s persistent skill repository. CPG compares current and updated skills through provenance-isolated, paired executions, using confidence bounds on utility, safety loss, and subgroup effects to promote, reject, or defer updates. Evaluation covers planted candidate classes, four task domains, and SkillClaw-style and Trace2Skill-style pipelines. The authors report lower false promotion and improved deployment outcomes. The problem is well motivated, but inconsistencies between the stated procedure and reported results undermine these conclusions.

Critical strengths
  1. The paper correctly distinguishes successful source behavior from evidence that a persistent update improves future tasks. This matters because admission errors can propagate across subsequent agent executions.
  2. Provenance isolation, paired execution, subgroup constraints, and explicit deferral address relevant failure modes beyond average performance. Executable verifiers and a separate final-test split provide a promising foundation for evaluating admission decisions.
Critical weaknesses

The methodological novelty beyond established controlled validation remains limited. CPG combines paired A/B evaluation, provenance-based holdouts, subgroup constraints, confidence bounds, and sequential stopping. The principal held-out A/B baseline omits several of these safeguards simultaneously, including provenance exclusion and worst-group constraints. Consequently, the comparison establishes the value of a more comprehensive validation protocol, but does not clearly isolate an additional contribution from the proposed causal formulation. The planted benchmark explicitly contains provenance-specific proxies and subgroup-harming updates, making these safeguards especially advantageous. A stronger controlled-validation baseline and clearer evidence from naturally generated updates would better establish the contribution’s general significance. The reported decisions and statistics are not sufficiently reconciled with the specified protocol. Appendix A.2 sets the default utility threshold to 0.05, whereas Table 10 identifies 0.02 as the default while reproducing the headline results. Table 3 reports rejection of 70.4% of redundant candidates, although the stated rejection rule requires evidence of negative utility, increased safety loss, or subgroup harm; genuinely zero-effect candidates should ordinarily be deferred. The class-level decisions also do not straightforwardly reproduce the overall defer rate and promotion F1 under the documented equal-sized candidate construction. These discrepancies may have explanations involving oracle relabeling or metric conventions, but those explanations are necessary to assess the results. Additionally, the confidence-bound construction and error allocation across sequential looks and endpoints need greater detail to substantiate the claimed calibration.

Questions for authors
  1. Which utility threshold generated the primary results: 0.05 or 0.02? Please reconcile Appendix A.2 with Table 10 and explain whether the correction changes oracle labels or reported performance.
  2. Why are 70.4% of redundant candidates rejected under the stated harm-based rejection rule? Do these candidates incur safety or subgroup regressions, or does the implementation also reject updates for insufficient benefit?
  3. Please provide candidate counts and confusion matrices, including oracle-indeterminate candidates and deferred decisions. How are FPR, FRR, and F1 defined, and how do Table 3’s class-level results aggregate to Table 1?
  4. How does CPG compare with paired held-out validation using the same provenance exclusions, safety and subgroup constraints, thresholds, and stopping budget? Please also specify the confidence-bound construction and error allocation, and report empirical coverage under sequential stopping.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.