acceptodds
Withdrawn from ICLR 2027

When Small Evaluator Biases Become Permanent: Phase Transitions in Self-Evolving Agent Skill Libraries

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Persistent skill libraries allow large language model agents to reuse and refine executable artifacts. Recursive selection and retrieval, however, can amplify small evaluator preferences into lasting shifts in library composition. Identifying this amplification requires separating it from utility differences, repair asymmetries, retrieval preferences, and finite-population drift while modeling artifact evolution and endogenous exposure. We introduce the Causal Replicator–Retrieval model (CRR), a stochastic framework that tracks the quality-weighted frequency of a controlled, task-irrelevant attribute and the resulting cross-group exposure imbalance. CRR integrates quality-conditioned evaluator selection, directional transformation through mutation or repair, and retrieval reinforcement driven by accumulated exposure. Its mean-field dynamics characterize low-bias and biased equilibria, and its local Jacobian determines whether perturbations contract or amplify. We identify the underlying mechanisms using counterfactually matched skill variants, complete lineage and retrieval logs, and randomized channel-level interventions on evaluation, promotion, and repair. A hierarchical Bayesian state-space formulation estimates mechanism parameters and predicts held-out long-run regimes. The posterior informs a minimum stability-margin controller that selects evaluator calibration and exposure de-reinforcement while preserving task utility. We evaluate CRR in synthetic populations and executable-skill sandboxes through parameter recovery, unseen-phase prediction, channel-specific counterfactuals, and bias–utility–cost trade-offs. These evaluations provide a falsifiable framework for diagnosing and controlling evaluator bias in recursively updated skill libraries.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Evaluator Bias, Recursive Skill Libraries, Retrieval Reinforcement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bSkJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors argue that in a self-evolving skill library, an LLM evaluator's preference for task-irrelevant attributes (e.g., formatting) can be amplified by recursive loops into a permanent shift in library composition. To address this, the paper proposes CRR (Causal Replicator–Retrieval), which models three mechanisms separately and uses them to mitigate the shift on three synthetic tasks.

Critical strengths

None

Critical weaknesses

The paper does not demonstrate that the problem CRR aims to solve arises in practice, and simulated dynamics and highly synthetic tasks alone can hardly establish it. Specifically, the main evidence (e.g., Table 1) comes from a simulator that implements CRR's own dynamics, and the bias in the sandbox is injected in a very naive way (e.g., a "VERIFIED" header in the code comments). For these reasons, I lean toward clear rejection.

The method relies on many design choices (e.g., which preference bias the evaluator has) and hyperparameters (e.g., ϵ_q, the stability margin m, and the fixation threshold) that are unknown in advance, even during deployment. In addition, many design choices are not well motivated (e.g., why a two-dimensional mean-field state with logistic and softmax forms is sufficient). This leaves me doubtful about the method's applicability to real skill libraries.

Note: The authors do not need to respond to the weaknesses above separately; answers to the questions below are sufficient.

Questions for authors
  1. Can the authors provide a less synthetic example that supports their main claim, namely that an LLM evaluator's preference for task-irrelevant attributes (e.g., formatting) can be amplified by recursive loops into a permanent shift?
  2. Could the authors describe in detail how the design choices (including hyperparameters) would be applied to a real skill library? For example, when z is unknown or twin matching is unavailable, how can the mechanism parameters be identified?
  3. As noted in the Weaknesses, how were these complex design choices motivated? This concern extends beyond the example given there; other choices also appear arbitrary (e.g., the δ + ηh_t interaction form, the exposure update based on p_{t+1}, and updating the controller every five generations). Were any alternatives compared?
Reviewer_Pp5xOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper provides a detailed analysis of a specific proposal for how skill libraries for agents might evolve. The model is given a theoretical analysis and then evaluated in synthetic populations.

Critical strengths

The main strength of this paper is the proposal of a creative approach to evolving skill libraries that draws on ideas from multiple disciplines, including statistics and evolutionary biology. This proposal is supported by both theoretical and empirical results, with proofs of relevant theoretical properties (such as low bias) and a demonstration in a simple agent architecture that the approach is effective.

Critical weaknesses

The paper is extremely hard to read. I found the introduction of the approach in Section 3.1 confusing, as it referred to many variables that were not properly defined until they were used much later. Section 3.2 is also hard to read and an important part of the paper's contribution that is not explained clearly. Figure 2 is a good illustration of the complexity of the approach and the paper -- the figure itself is pretty confusing and shows how many elements are being combined together. I think this is interesting work but it could be presented in a fashion that would be much more accessible to readers.

The empirical evaluation is fairly limited. It involves running a synthetic analysis using a simplified agent defined by the authors. The actual evaluation isn't apparent without reading the Appendix, and could be described more clearly in the text. Extending the results to a wider range of settings and models would increase the impact of this work.

The baselines that were used were not clearly connected back to related work, which makes it hard to determine whether they represent the state of the art as opposed to reasonable ablations of the current approach.

Questions for authors

N/A

Reviewer_96hUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how small, task-irrelevant preferences of an LLM evaluator become persistent, library-level bias in a recursively self-improving skill system. The core claim is a mechanism: one-step evaluator bias (δ) interacts with exposure reinforcement (ρ) — accepted skills are retrieved more, and retrieval history further increases selection — so that composition perturbations either contract or amplify, with a regime boundary. The paper formalizes this as the Causal Replicator–Retrieval model (CRR): a two-type mean-field dynamical system over quality-weighted composition p_t and log exposure imbalance h_t, with separate channels for quality-conditioned selection, directional mutation/repair (μ₊₋, μ₋₊), and exposure-driven retrieval. Identification uses counterfactually matched skill twins (identical behavior, opposite nuisance attributes), complete lineage/retrieval logging, and randomized channel-level encouragements (evaluator offsets, top-k retrieval promotions, attribute-preserving/swapping repair), estimated in a hierarchical Bayesian state-space model. A minimum stability-margin controller then selects the cheapest combination of evaluator calibration and exposure de-reinforcement that keeps the low-bias fixed point locally stable with high posterior probability. Evaluations span a ground-truth simulator (parameter recovery 94.1% coverage, unseen-cell phase prediction macro-F1 0.842) and a network-isolated executable-skill sandbox with a 7B generator and 72B evaluator (prospective, frozen-policy control: 82.4% low-bias retention vs. 38.6% no-intervention, at 0.7pp utility cost).

Critical strengths
  1. The mechanism, not just the phenomenon, is the contribution — and it is correctly identified as such. "Static debiasing does not capture how residual preferences reshape future candidate pools" is the right framing, and the paper operationalizes it: amplification requires the interaction of one-step bias with endogenous exposure (Table 1: δ=0.08 alone gives 6% fixation; ρ=0.6 alone gives 7.5%; together 71.5%). The case study (Appendix E) is worth the space it takes: the label-exchange intervention where the judge's "robustness" rationale physically moves with the assertive header, followed by retrieval that no longer mentions the attribute at all ("retrieved because it had been retrieved"), is the clearest published demonstration I have seen of how a presentation cue acquires causal force through a feedback loop rather than through continued recognition.

  2. The causal identification strategy is genuinely careful and honestly scoped. Utility-matched twins with a preregistered tolerance, descendants retaining base-pair identifiers, propensity logging with overlap conditions, and — critically — the paper reports the observational-vs-randomized contrast (83.6% vs. 94.1% coverage; mechanism correlation 0.62 → 0.87 when retrieval encouragement is removed) and states plainly that "identifiability from ordinary deployment traces remains outside this evaluation setting." This is the right way to handle an identification problem that most applied papers would have swept under regression.

  3. The prospective protocol deserves emulation. Estimate on the first 30% of each trajectory, select control over the next 20%, freeze for the sealed 50% confirmatory horizon; controllers compared under matched retrieval/verifier/token budgets; intention-to-run handling of failed executions; trajectory-level aggregation with lineage-clustered bootstrap and Holm-corrected families. The ablations (Table 7: each controller component contributes; the stability-margin constraint is what costs money) and the falsifiability framing in the introduction ("the evidence rejects the framework if…") show the authors trying to fail their own method.

  4. The theory is serviceable and the derivations are clean. The closed-form controlled Jacobian (Appendix A.1), posterior-rollout that retains per-draw parameters to preserve temporal dependence, and the explicit disclaimer that r_spec < 1 guarantees only local stability with no finite-library guarantees — this is a paper that knows exactly what its theorem does and does not say.

  5. The cost accounting is unusually candid. Table 12 admits CRR is not the cheapest policy (0.35 vs. oracle 0.28); Table 7 shows what the utility constraint buys; uncertainty peaks exactly at the phase boundary, which is what a calibrated model should do.

Critical weaknesses
  1. The title overclaims: what is shown is smooth amplification, not a phase transition. Table 1 shows monotone escalation of late deviation and fixation probability with δρ — there is no demonstrated bistability, no bifurcation diagram, no basin-of-attraction structure, no escape-rate analysis. "Phase transitions" implies coexisting stable regimes with sharp boundaries; here fixation reaches 71.5% rather than ~100%, and whether runs switch back, or whether the low-bias and biased equilibria genuinely coexist at intermediate parameters, is never analyzed. The "boundary error" metric presumes a threshold. Either provide the bifurcation analysis (the mean-field model makes this cheap — show fixed-point count and stability as a function of δ and ρ) or retitle.

  2. The measured bias is experimenter-injected and partially de-fanged by design; the natural bias is not. The nuisance attribute is a formatting/provenance label, and the evaluator is shown the hidden-test summary (Appendix D templates include {HIDDEN_TEST_SUMMARY} — "utility information available to the evaluator"). Real evaluator biases — self-preference, family preference, verbosity — operate on content the judge must assess without ground truth, and plausibly correlate with quality. The twin design deliberately strips this correlation, which is fine for identification, but then δ is the bias of a near-oracle judge over presentation metadata, and the paper never tests whether dynamics change when the judge sees only rubric + execution outcomes (where δ may become utility-dependent and the quality-adjusted contrast misspecified). This is the single most important external-validity gap.

  3. The endogenous-bias analogue is missing. The most discussed real-world failure is evaluator–generator family overlap (self-preference). The framework could test this directly — provenance labels claiming lineage, with the generator and evaluator swapped across families — and one evaluator-family shift experiment (Llama-3.1-70B) is mentioned only in a single sentence ("retention decreases to 73.0%"). Given the motivation leans heavily on preference leakage, the absence of a self-preference twin experiment is conspicuous.

  4. The state definition requires hidden utility q, which deployments do not have. p_t is quality-weighted composition; the controller's utility constraint uses hidden-test success. The paper never states what the deployable proxy (\hat{q} from executable verifiers, pass rates, rubric scores) does to state estimation and control — a noisy q̂ both contaminates the state and the constraint. A noisy-measurement ablation belongs in the paper, not just the limitations.

  5. The controller's "minimum-cost" claim is relative to its own mechanism grid. The oracle is both better (88.7 vs 82.4) and cheaper (0.28 vs 0.35) than CRR — understandable, but then the honest statement is that estimation overhead is the price of not knowing the parameters, and the paper should quantify how that overhead amortizes with horizon length and whether cheaper two-stage policies (calibrate once, then fixed de-reinforcement) dominate in long deployments. Table 7's "without periodic updates" row (75.8% at 0.29) starts this analysis; it should be extended.

  6. The 2-type mean field is doing a lot of quiet work. The simulator implements multi-type dynamics but no multi-type results are reported; the limitation is acknowledged but the interaction structure of multiple nuisance attributes (e.g., style × provenance × verbosity jointly) is where real libraries live, and 2×2 Jacobian analysis may not extend (sign patterns across attributes could cancel or compound). At minimum, one multi-type cell showing the framework still separates channels would substantiate the generality implied by the framing.

  7. Statistical power and effect-size calibration for the headline interaction. The δ×ρ interaction posterior (0.991, standardized effect 0.43 [0.25, 0.61]) is reported without stating how many independent trajectory cells inform it, and the phase-prediction macro-F1 of 0.842 — while beating the matched black-box (0.771) — is modest in absolute terms, with the improvement concentrated, one suspects, near the boundary. Reporting where in the δ-ρ plane CRR's advantage over the black-box state-space model vanishes would sharpen the contribution.

Questions for authors
  1. Can you produce the bifurcation diagram (number and stability of fixed points vs. δ, ρ, with η and κ varied)? If the model is not genuinely bistable anywhere in the tested region, the "phase transition" language should go.
  2. What happens to measured δ, and to dynamics, when the evaluator is denied the hidden-test summary and must judge from rubric + execution evidence? Does the quality-adjusted contrast survive, or does attribute preference become utility-confounded?
  3. Can you run the self-preference twin (provenance labels asserting shared/different lineage with the evaluator, plus an actual generator–evaluator family swap) as a first-class experiment rather than a robustness footnote?
  4. With q̂ (noisy verifier-based utility) substituted for q in the state and the constraint, how much does identification and control degrade? At what measurement noise does the controller lose to Fixed Debiasing?
  5. Over what horizon does CRR's estimation overhead amortize below the cost of the best fixed policy? Is there a crossover in (horizon × distance-from-boundary) space?
  6. Is η (exposure-dependence of evaluator bias) identifiable from observational data at all, or does it strictly require retrieval encouragement? The 0.87 posterior correlation suggests near-failure; what breaks first as encouragements are throttled toward deployment-feasible levels?
  7. For multi-type attributes: does channel separation survive, and do cross-attribute interactions produce cancellation or compounding in the Jacobian?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.