acceptodds
Withdrawn from ICLR 2027

When Agents Change Their Own Test: Prospective Target Preservation under Candidate-Induced Task Expansion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Open-ended and self-improving (autotelic) agents increasingly help construct the tasks, tests, or subproblems on which progress is later judged. This creates a same-candidate dilemma: if candidate update expands the evaluation population, may those candidate-caused additions change the objective used to decide whether itself is accepted? We formalize candidate-induced target endogeneity and distinguish it from adaptive sampling, missing evaluation coverage (support failure), and intentionally performative objectives. Under prospective semantics, the current target distribution is fixed using only pre-candidate information; evidence may still be sampled adaptively and reweighted to that target, while candidate-caused tasks affect only later targets. We prove that any nontrivial same-round target change can reverse some bounded threshold decision even under exact, noiseless evaluation, so greater statistical precision cannot repair a change in the quantity being judged. Exact replay on 384 recorded LLM update-effect vectors yields false-promotion rates (acceptance under the post-candidate target despite rejection under the prospective target) of 73.4% and 54.1% in the primary and independently seeded confirmation phases when only the target semantics change. More importantly, a pre-specified capability-conditioned task-admission analysis on naturally trained Prioritized Level Replay policies in the Procgen benchmark finds 13/48 false promotions across 11/16 StarPilot seeds and, in an independently seeded replication, 36/192 across 29/64 Heist seeds. Eleven Heist reversals have prospective and post-target signs jointly separated from zero under our 95% certificate. Mean disagreement under same-size candidate-independent expansion is 3.2% and 2.5%, respectively. The Procgen CoinRun environment has no primary false promotions, providing a negative boundary. These results identify target timing as a distinct evaluation contract for self-expanding agents.

Primary area:
reinforcement learning
Keywords:
open-ended learning, agent evaluation, reinforcement learning, autotelic agents, adaptive curricula, target endogeneity, continual learning, policy improvement, off-policy evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sHxGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Self improving agents may unlock new tasks during inference time, which may change the decision that whether we should adopt the new model. This paper propose PTP to fix current evaluation set and reweighing adaptively. Through experiments, the paper proves that changing the target is unstable.

Critical strengths
  1. This paper distinguishes wrong measurement when the evaluation set changes.
Critical weaknesses
  1. The most prominent issue is that the motivation of this paper is impractical. I do not think this new evaluation protocol is beneficial to anyone's research.
  2. There are too many theoretical proofs in the main body, which bring limited information to the paper.
  3. The experiment setting is designed deliberately to cater the paper setting, which may be different from real world distribution significantly.
  4. There is no real agent system results, that demonstrate the disadvantage of a simple evaluation process.
Questions for authors

Actually, I do not think this paper bring anything useful to the research community. I think the authors should at least clarify the position of the proposed method, otherwise I think the basic motivation of this method is confused.

Reviewer_3zxrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the so-called “candidate-induced target endogeneity”: an agent update changes the task distribution used to evaluate that same update. The proposed “Prospective Target Preservation” (PTP) protocol fixes the current evaluation target and postpones newly introduced tasks until subsequent rounds. Standard importance weighting handles nonuniform evaluation sampling. The authors prove that changing the target distribution can reverse some acceptance decisions and demonstrate this effect through controlled LLM experiments and Procgen evaluations. A separate closed-loop simulation shows improved agreement with the fixed target.

Critical strengths

None

Critical weaknesses

W1. The practical motivation is insufficiently established.

The paper does not demonstrate a realistic end-to-end agent system in which candidate-generated tasks alter the current evaluation distribution, violate the intended evaluation rule, and cause meaningful harm. Its central experiments use manually constructed reweighting or task-admission rules. These demonstrate that decision reversals are possible, but provide limited evidence that the problem is sufficiently common or consequential in practical self-improving systems.

W2. The definition of “false promotion” relies on an insufficiently justified reference objective.

The so-called “false promotion” refers to accepting an update under the modified target when the original target would reject it. However, newly unlocked tasks and expanded capabilities may themselves represent useful progress. Although the authors acknowledge alternative evaluation objectives, they do not sufficiently justify why the original task distribution should determine correctness in the motivating applications. Decision disagreement alone does not establish that accepting the update is undesirable.

W3. The theoretical and methodological contributions are limited.

The central theorem establishes that two different distributions can produce opposite threshold decisions for some bounded effect function. This is a basic property of expectations under different distributions. The result does not exploit the specific relationship between model updates and task generation, or establish that actual model updates can produce the constructed effects. The proposed method mainly combines a fixed evaluation target, standard importance weighting, and delayed benchmark revisions, offering limited technical novelty beyond evaluation version management.

W4. The LLM experiments are restricted to small models and contextual-example updates.

The solvers are limited to Qwen2.5-Coder-0.5B and 1.5B, both from the same model family. Learner updates modify an in-context memory containing at most 20 demonstrations, rather than model weights. There is no validation with larger solvers, other model families, or parameter updates through SFT or RL. Consequently, the applicability of these results to more capable, continuously trained LLM agents remains uncertain.

W5. The experiments do not demonstrate practical downstream benefits.

The long-horizon experiment uses a controlled simulator with 200 trajectories of 120 rounds and an external panel of 160 tasks. PTP improves agreement with the fixed evaluation target but slightly reduces external-panel AUC, with the reported confidence interval excluding zero. This does not contradict the target-preservation guarantee, but leaves the practical benefit unresolved. No real-agent closed-loop experiment demonstrates improved downstream capabilities.

W6. The presentation is overly defensive and repetitive.

The manuscript devotes substantial space to anticipating objections and repeatedly qualifying its contribution. Examples include:

“The estimator is inherited importance sampling, not a claimed novelty.” “We therefore avoid broader firstness claims about dynamic evaluation, decision-dependent distributions, benchmark correction, or self-evolving benchmarks.” “This is an analogy and design implication, not a claim that the cited system currently exhibits the failure measured in our experiments.” Similar limitations are reiterated multiple times. For example, “We therefore claim recurrence within tested adaptive-learning regimes, not a universal prevalence rate for open-ended agents” is later restated as “the result supports recurrence in the tested Heist/CentralPLR regime, not a claim that 45% of open-ended agents in general exhibit target endogeneity.”.

The extensive repetition makes the presentation unnecessarily lengthy and obscures the central contribution.

Questions for authors

See above.

Reviewer_WaPZOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper analyzes when we have agents which help construct new tests and tasks which are then used to evaluate the performance of the agent itself. The paper formalizes this concept around the idea of target endogeneity, which broadly covers when the model is allowed to change its own evaluation criteria and how to quantify the change in capabilities. They show that when you allow the agent to change the evaluation criteria, it will always be possible to induce a false promotion/rejection in the process. They run experiments and rest this on some adaptive RL methods and show that we can see these effects while training, and suggest that one ought to freeze the target before each decision before adapting the target dataset.

Critical strengths

None

Critical weaknesses

The first weakness is around the significance of the finding itself. The paper seems to boil down to proving that when an agent can change its own evaluation set, we will have false acceptances/rejections. Thus, we should keep the eval set fixed until we make a decision on whether to change the model. This is fairly intuitive, but the main theoretical result seems to be fairly elementary, that changes to the distributions under which you evaluate a decision threshold changes the threshold itself. The experiments are very small scale and do not reveal any non-trivial scientific or engineering insight into the nature of self-improving agents.

The second critical weakness is the writing at large. The paper uses a lot of mathematical jargon and terminology which make the paper difficult to read. Many of these concepts upon a closer read are common evaluation terminology, but they are obscured by esoteric terminology. There is no clear motivating example and no clear experimental narrative to motivate the work and improve readability.

Questions for authors

In 5.1, why do we change the mass of each candidate for which the recorded effect is positive? Isn't this setup extra conducive to false acceptances?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.