acceptodds
Under review as a conference paper at ICLR 2027

When Better Gets Worse: Improvement Fidelity for Self-Improving Agents in Adaptive Worlds

Abstract

Self-improving agents increasingly rely on proxy verifiers to choose policy updates, yet deployment can change the world in which those updates are evaluated. An update that looks better to the verifier can therefore become worse after deployment even when the verifier ranks policies well overall. We formalize this gap as Improvement Fidelity, which asks whether proxy improvements preserve the sign and ordering of deployment improvements over the updates an improvement process actually proposes. We show that global policy accuracy need not guarantee update fidelity: operator shift and deployment response can create update-level errors, while candidate margins determine whether those errors change the replacement decision. We introduce PIVOT-KG, a paired, decision-aware validator that allocates scarce high-fidelity evaluation according to the expected reduction in selection regret per unit cost. Across 90 held-out roots in Leduc, Kuhn, and Melting Pot, proxy and deployment optimal sets are disjoint in 51 cases. In an eight-candidate HighwayEnv stress test, PIVOT-KG reduces mean improvement-selection regret from 0.0435 under the exact Uniform validation rule to 0.0055 at the primary budget. Together, these results show why reliable self-improvement should evaluate proposed improvements in the worlds they induce, while providing a practical rule for allocating scarce deployment evidence when it can affect the replacement decision.

Primary area:
reinforcement learning
Keywords:
self-improving agents, reinforcement learning, adaptive environments, policy evaluation, multi-agent reinforcement learning, active validation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2o9XOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether improvements measured by a cheap proxy verifier remain improvements after deployment, when the deployed policy can change the environment or other agents’ responses. It introduces operator-relative Improvement Fidelity, distinguishing aggregate policy-evaluation accuracy from the reliability of the particular updates an improvement process proposes. The theoretical results relate update errors to uniform value accuracy, operator shift, and candidate margins. PIVOT-KG models proxy-to-deployment corrections with a calibrated Gaussian belief and allocates paired high-fidelity evaluations according to expected selection-regret reduction per unit cost. Experiments cover controlled response environments, poker, Melting Pot, and driving simulators. The reported results demonstrate substantial proxy–deployment disagreement and benefits over Uniform in several settings, particularly Leduc and redesigned HighwayEnv. The supplementary artifact provides executable selectors, saved observations, and analysis records that support offline verification of the principal reported results.

Critical strengths
  1. The paper articulates a useful evaluation target for policy improvement. Measuring fidelity over the transitions actually proposed by an improvement operator makes explicit why strong aggregate evaluation metrics may fail to protect replacement decisions. Connecting this issue to deployment response and candidate margins gives the formulation practical relevance: it distinguishes evaluation errors that merely affect predicted values from errors that change the selected update.

  2. The principal allocation comparisons are controlled, and the supplementary artifact substantially improves their verifiability. The Table 1 experiments and redesigned HighwayEnv comparison share posterior updating and terminal selection between PIVOT-KG and Uniform, supporting attribution of their differences to query allocation. The supplementary source and saved observations permit replay of the primary decisions, while redesigned HighwayEnv uses an exact expectation over Uniform query subsets. These features strengthen the positive Leduc and HighwayEnv results. The manuscript also reports several inconclusive and negative comparisons, making the limits of the evidence more transparent.

Critical weaknesses
  1. The methodological advance beyond established knowledge-gradient and simulation-selection methods remains limited. Under the stated posterior-mean terminal decision, Eq. (6) reduces to the standard one-step knowledge-gradient objective, with cost normalization. The supplied implementation explicitly uses correlated Gaussian KG, and correlated beliefs together with paired simulation have direct precedents in Frazier et al. (2009) and Xie et al. (2016). The paper acknowledges these connections, but it remains unclear what additional methodological capability is required by, or developed specifically for, the adaptive-world setting. The theoretical results provide useful organization and counterexamples, but largely follow elementary error and distribution-shift arguments.

  2. The existing stronger-baseline comparisons do not yet establish a compelling incremental benefit of the KG-specific allocation rule. In the first-cohort Leduc primary setting, PIVOT-KG, IVR, and LUCB select the same policy on all 30 test roots and attain the same mean ISR of approximately 0.00927. Thus, this positive result against Uniform does not distinguish the proposed acquisition objective from simpler targeted allocation rules. In the second Leduc cohort, the primary comparison is positive against expected Uniform, but the secondary comparisons against LUCB and top-proxy do not establish a clear advantage; the IVR-menu contrast is also based on few differing selections. The strongest redesigned HighwayEnv result emphasizes Uniform rather than these matched targeted alternatives.

Questions for authors
  1. Clarification of contribution: Which part of PIVOT-KG constitutes a methodological advance beyond applying established correlated Gaussian knowledge gradient to proxy-to-deployment corrections? In particular, does the adaptive-world formulation introduce additional estimation or decision structure beyond the calibrated correction prior, or should the contribution primarily be understood as the evaluation formulation and empirical study?

  2. Evidence that could change the assessment: The supplementary first-cohort Leduc results show identical final selections for PIVOT-KG, IVR, and LUCB on all 30 primary test roots. Are matched IVR, LUCB, and top-proxy results available for the eight-candidate HighwayEnv redesign, using the same prior, observation model, query budget, and terminal selection rule?

  3. Calibration robustness: The exact-versus-noisy calibration comparison is useful, but it does not quantify variability from drawing a different calibration cohort. How stable are the principal contrasts when the correction model is refitted across calibration-root resamples or independent calibration sets? For the first poker cohorts, how sensitive are the conclusions to the observation-variance floor and covariance regularization?

2 comments
AuthorsOpenReview ↗

RESPONSE TO REVIEWER 2o9X

  1. Advance beyond KG. Equation 6 represents an established one-step correlated-Gaussian KG with cost normalization. We introduce Improvement Fidelity as an operator-relative target for the sign and ordering of proxy versus deployment gains for proposed updates and link differential errors to decision margins. PIVOT is the paired validation framework. PIVOT-KG applies existing KG to this structure. It allows evaluation of whether calibration helps, whether new HF observations help, and whether allocating the same HF budget differently helps.

  2. HighwayEnv baselines. The saved-truth replay favors PIVOT-KG over exact Uniform. At K=8 and B=4, mean ISR falls from 0.0435 to 0.0055. This results in a difference of 0.0380 with a paired 95 percent CI of [0.0300, 0.0474]. This replay uses saved deployment values and simulated observations. Sec. 6.5 reports fresh simulator runs where PIVOT-KG, integrated variance reduction (IVR), and Top-Proxy share the panel, prior, observation model, posterior update, query budget, and terminal rule.

Baseline minus PIVOT-KG ISR contrasts are shown with paired 95 percent CIs. Positive values favor PIVOT-KG.

  • Development v2. The IVR result is -0.0030 [-0.0073, 0.0008] and the Top-Proxy result is 0.0135 [0.0024, 0.0307].
  • Independent calibration and test cohort. The IVR result is 0.0016 [-0.0042, 0.0071] and the Top-Proxy result is -0.0001 [-0.0050, 0.0045].

Both IVR intervals include zero. The PIVOT-KG advantage over Top-Proxy in development was not established on the independent cohort. Standard LUCB cannot complete its eight initial pulls at B=4. Strong baselines can make the same decisions. In the first Leduc cohort, KG, IVR, and LUCB chose the same action at all 30 roots. In the exploratory MetaDrive PGMA, KG and IVR agree in all 24 roots and all three disagreements vs Top-Proxy favor Top-Proxy. Across all 24 roots the KG-minus-Top-Proxy utility contrast is -8.396448. The +26.8376 gain for Calibrated-No-HF versus Proxy-Only measures calibration value (Appendix C).

  1. Calibration sensitivity. We varied four observation-variance floors and five covariance-shrinkage settings in Leduc. At the most conservative setting, Uniform minus PIVOT-KG was 0.02246 [-0.00018, 0.05480]. The interval includes zero. Across 500 bootstrap refits of the calibration roots, the Uniform contrast stayed positive with a median of 0.03649 and refit percentiles [0.02278, 0.11266]. IVR minus PIVOT-KG was zero in every refit. This measures prior sensitivity on the 30 test roots. The separate HighwayEnv cohort changes both calibration and test roots. It does not establish an advantage over IVR or Top-Proxy. These checks support a scoped use of decision-aware validation. They provide no evidence of universal KG superiority.

Thank you for the clarifications and additional experiments. They address most of my previous concerns about the baseline comparisons and calibration sensitivity, and they strengthen the empirical evidence. However, the methodological novelty remains limited, and neither of the two new HighwayEnv cohorts shows a clear advantage for PIVOT-KG over IVR. These additions do not substantially change my assessment of the paper’s overall contribution. I therefore maintain my original rating of 2 (Weak Rejection).

Reviewer_BGmCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Policy improvement when measured against a verifier might not transfer exactly as gain after policy deployment. This work formalizes this gap as Improvement Fidelity, which judges the proposed updates instead of the global accuracy. The main contribution of this work is PIVOT-KG, a validator that spends a small budget of expensive evals where they could change the decision. The method is evaluated on poker, Melting Pot and driving simulators, mainly against Uniform allocation of the budget.

Critical strengths

None

Critical weaknesses

W1: Framing against what is tested: Title, abstract and intro frame the paper as self-improving agents in adaptive worlds, yet there are no definitions or formalism of "agents", "adaptive worlds" or "improvement", although agents can have different meanings, as seen in related work: policy optimization and gradient ascent (Mandal et al., 2023) [1] vs LLM agents and agentic RL (Wang et al., 2026) [2]. No training algorithms, no LLM and no proposer generating its own candidates in a loop seam to appear in any experiment, making experiments not matching any of these definitions, since they are mostly single decisions over fixed panels. These candidate panels are action schedules, mixtures and profiles. Not having an in loop proposer, this looks more like budgeted policy selection, an existing setting (Konyushkova et al., 2021) [3], and a known allocation rule with adaptation (Xie et al., 2016) [4]

W2: The setup cannot be clearly assessed and results are not supporting the method: The verifier per environment is not specified, candidate generation outside HighwayEnv is undefined, as are the meaning of "long/short" and the "allocation interaction" for instance. Numbers in Tables 1 and 2 therefore cannot be interpreted clearly, including those for the allocation rule named as the main contribution. Even taking these at face value, in the primary comparisons, the only comparator is Uniform, with 5 out of the 8 (Table 1, Fig. 7, Table 3, Fig. 10) comparisons having intervals including 0, meaning advantage over Uniform cannot be distinguished from noise.


[1] Debmalya Mandal, Stelios Triantafyllou, and Goran Radanovic. Performative reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning (2023)

[2] Zhilin Wang, Han Song, Runzhe Zhan, Jusen Du, Jiacheng Chen, Tianle Li, Qingyu Yin, Yulun Wu, Zhennan Shen, Tong Zhu, Yanshu Li, Guanjie Chen, Derek F. Wong, Yafu Li, Yu Cheng, and Yang Yang. EvoPolicyGym: Evaluating autonomous policy evolution in interactive environments.

[3] Konyushova, Ksenia, et al. "Active offline policy selection." Advances in Neural Information Processing Systems 34 (2021)

[4] Jing Xie, Peter I. Frazier, and Stephen E. Chick. Bayesian optimization via simulation with pairwise sampling and correlated prior beliefs (2016)

Questions for authors

Will PIVOT-KG lower selection regret inside an improvement loop as claimed, where in this loop a learner proposes candidates and the selected update is deployed over several rounds ? Fig. 8 is the only multi-round result and do not seem to be using Algorithm 1

2 comments
AuthorsOpenReview ↗

RESPONSE TO REVIEWER BGmC

The controlled PerformativeWorld extension in Sec. 6.6 and Appendix B (Table 4) demonstrates that PIVOT-KG-Loop-v3c lowers cumulative improvement-selection regret (CISR) relative to integrated variance reduction (IVR) and Top-Proxy. This extension uses a fixed proposer and does not perform in-loop training. Figure 8 illustrates the legacy diagnostic and does not utilize Algorithm 1. In v3c, the proposer generates four candidates from the incumbent per round. The KG branch scores the proxy, initializes the calibrated correction model, and makes two sequential paired HF queries using KG with noisy Gaussian conditioning. It selects the action with the highest posterior mean gain, deploys it, and carries the resulting incumbent and world state into the next round. All methods share the proposer, response protocol, seed schedule, terminal rule, and query budget. They start with the same candidate panel and their states and later panels can diverge after deployment. The supplement maps these steps to the runner. v3c implements Algorithm 1's within-round decision logic in a state-carrying extension with a contextual prior.

Across 120 fresh roots per regime, mean CISR for PIVOT-KG, IVR, and Top-Proxy was 0.00888, 0.05574, and 0.07095 in low regimes. Results were 0.00141, 0.44046, and 0.23241 in medium regimes and 0.08886, 0.98532, and 0.33346 in high regimes. IVR-minus-PIVOT differences and paired 95% root-bootstrap intervals are 0.0469 [0.0385, 0.0565], 0.4391 [0.3706, 0.5168], and 0.8965 [0.6449, 1.1897]. Top-Proxy differences are 0.0621 [0.0491, 0.0755], 0.2310 [0.1805, 0.2823], and 0.2446 [0.2019, 0.2917]. All six prespecified sign-flip tests pass. We chose and froze the v3c configuration after evaluating exploratory variants, and the Holm correction applies only to the six within-v3c tests rather than the preceding search process. Paired 95% root-bootstrap intervals are unadjusted. We fitted the model once and it resets each round. The calibration feature round_fraction is zero in calibration and spans [0,1] in test, with the effect unidentified. In the multivariate audit, 80.3% and 80.0% of test candidate vectors exceed the calibration 95th and 99th distance percentiles. These shifts and diverging trajectories restrict the result to the tested protocol.

The agent holds an incumbent while the operator proposes candidates and the verifier chooses a replacement or retains the incumbent. Improvement Fidelity checks whether a proxy gain survives the response induced by deployment. PIVOT provides paired validation, and PIVOT-KG applies standard KG. Table 1 specifies the protocols. Leduc and Kuhn use nine CFR+ mixtures, including the incumbent copy, against a root-specific perturbed proxy opponent. The proxy is the exact game value and HF uses 4,096 paired hands per query. Melting Pot mixes specialist policies in repeated stag-hunt and compares a frozen-responder proxy with paired native rollouts. The short and long response horizons are 1 and 8 steps in poker and 4 and 32 adaptation episodes in Melting Pot. HighwayEnv scores eight action-by-window schedules in observer mode and uses saved-truth replay or paired fresh actor rollouts. The exploratory MetaDrive PGMA uses four updates of a frozen focal policy, paired response evaluations, and scripted IDM opponents. The allocation interaction quantifies how the Uniform-minus-PIVOT ISR contrast changes between short and long horizons as defined by the formula (ISR_U - ISR_PIVOT)_long - (ISR_U - ISR_PIVOT)_short. We flag uncertain or negative comparisons in the paper. We cited Active Offline Policy Selection (Konyushova et al., 2021) because both methods allocate costly evidence to policy selection. PIVOT evaluates the gain from replacing an incumbent in a candidate-dependent deployment world.

Thank you for your detailed response. Taking your answer into consideration, I still believe that the setup cannot be clearly assessed and results are not supporting the main method. Contribution remains unclear and the framing, presentation and clarity need revisions, as mentioned also by other reviewers. Therefore in its current form I cannot recommend acceptance.

Reviewer_9PKaOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper discusses the problem of agent's improving their policies in adaptive environments. When the environment can change, a proposed change to the agent's policy may initially look good but perform worse when deployed. A notion of fidelity is introduced to capture this notion and some theory is developed. An algorithm is proposed that is better suited to this setting and experiments on a variety of benchmark tasks are conducted demonstrating its utility.

Critical strengths

The paper explains the motivates the problem well and how the proposed could address it. The experiments seem to be comprehensive.

Critical weaknesses

It's unclear to me how novel the contributions are. If the main point is that policy updates can result in worse performance when the environemnt can change in response, then it would seem like this is already well understood in the game theory literature. For example, in two player zero-sum games, choosing a myopic best response to an opponent's current strategy could leave an agent open to being exploited later on as the opponent adapts. Many notions of equilibria in game theory explicitly take this into account.

Additionally, additional clarity could help in the explanations of the algorithm. See questions.

Questions for authors

It seems like the paper has additional context to frame the contributions. Could you elaborate on the background work and how this paper advances the area, particularly with respect to the point brought up in the weaknesses section? It's also unclear what the vanilla PIVOT algorithm is as I did not see a citation. It does not seem to be evaluated in this paper.

Concerning the algorithm, it's unclear what a "High-fidelity" observation is or what a "paired" observation. Could you explain this in more detail and generally how the algorithm works? It seems like it's similar to a model-based RL method where a model needs to be estimated of how the policies will affect the value.

1 comment
AuthorsOpenReview ↗

RESPONSE TO REVIEWER 9PKa

Strategic and performative research describes how policies change evaluation environments. We focus on specific operator updates. We evaluate if a candidate policy improves upon an incumbent policy after each policy induces its own response. Improvement Fidelity measures the alignment between proxy gains and deployment gains for the proposed updates as defined in Section 3 and Equation 2. Section 4 relates these errors to the margin required to maintain a replacement decision. We treat the response protocol as a fixed part of the evaluation process. We do not attempt to solve for an equilibrium.

PIVOT stands for Paired Interventional Validation of Optimization Transitions. This framework provides the basis for our evaluation. We evaluate PIVOT-KG as an instance of Algorithm 1. This application utilizes the one-step correlated-Gaussian knowledge gradient. We provide no separate vanilla PIVOT algorithm or new KG acquisition rule.

A high-fidelity query estimates the return difference between a candidate and an incumbent. The evaluation for each policy occurs under its corresponding deployment response. The paired arms start from a shared root and matched initialization. We use the declared seed schedule. Poker uses common random numbers. Driving and PerformativeWorld use matched starts and common seeds. High-fidelity results are relative to the proxy and remain subject to noise.

We fit a joint Gaussian prior for the candidate-wise correction G_j = Delta_*j - Delta_Vj on separate calibration roots. We condition this prior on paired high-fidelity results. Correlations allow a single query to inform unqueried candidates. We use Equation 6 to select the affordable query. This selection provides the largest expected reduction in terminal selection regret per unit cost. We query each candidate at most once. Algorithm 1 selects the action with the highest posterior mean gain. Retention functions as a zero-gain option. This process estimates a correction for the proposed panel. The algorithm does not learn the policy-dependent response map. It does not plan through a model as in model-based reinforcement learning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.