acceptodds
Under review as a conference paper at ICLR 2027

Repeatable but Not Replicable: Evaluator State as a Hidden Facet in Self-Evolving Agents

Abstract

Self-evolving agents improve by keeping new versions of their prompts or code that score higher on test tasks. Exact reruns often check such comparisons, but evaluators also write incidental strings, such as run directories and identifiers, into model inputs, and a rerun repeats them, hiding their effect. We treat these strings, the evaluator state, as a measurement facet: sharing a state cancels only the changes two versions share. In an unmodified SQL-writing agent, a new equal-length identifier changes 48 of 240 first queries but only 4 grades. Decisions on 100 tasks hold within a study, but on 10 tasks two evaluations of the same pair disagree on a median of 13 percent of decisions, and sharing a state did not measurably help. When a GEPA prompt-optimization loop shows the model its run directory, a rerun repeats every step with a database clock fixed, but other directories keep different prompts. On 12 validation tasks, the added lines lower the bar a new version must beat, and 14 of 18 loops keep a version worse on held-out tasks. Without the lines, these losses persist but mostly vanish at 100 validation tasks. Evaluations should rerun comparisons under fresh evaluator states.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-evolving agents, LLM agents, agent evaluation, causal inference, exchangeability, evaluation harnesses

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2o2FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Agent harnesses write incidental strings, such as hashed directories, identifiers and latencies, into model inputs. Exact reruns repeat these strings, so they hide the strings' effect on keep/reject decisions in self-evolving loops. The paper frames this evaluator state as a generalizability-theory facet and shows that a shared state cancels only the shifts both candidates share. On BIRD with Qwen3-8B, swapping identifiers changes many queries but few grades. At 10 tasks, decisions disagree on a median of 13% of cases. With an injected path block, the directory name alone changes which prompt GEPA and OPRO keep.

Critical strengths

A clean causal demonstration that exact reruns hide evaluator-written context that changes which version a self-evolving loop keeps, backed by rigorous pre-registered controls.

Critical weaknesses

The evidence that this causes harm rests on an injected block, a 12-task validation set and a single benchmark. The effect weakens in realistic settings, and the paper tests no comparison against existing selection methods or simple mitigations.

Questions for authors
  1. Does the effect hold with published GEPA settings, on a harness that natively exposes these strings, such as Terminal-Bench?
  2. How large is V_state compared with V_task, with prompt-format perturbations, and with sampling variance at T=0.6?
  3. Does a stable-alias arm, or racing- or PACE-style selection with fresh states, remove the state-induced instability?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 2o2F

Thank you for the review, and for recognizing the causal demonstration. Your summary states our argument more clearly than our own introduction did, so we rewrote the paper to follow its order. Numbers refer to the revised PDF.

Short answers

  • Response to the weakness. We agree: the harm evidence is narrow, and it weakens at 100 validation tasks. The causal result does not weaken. The revised abstract separates the two.
  • Answer to Question 1. The result holds in our larger setting: GEPA's reflection minibatch of three, 100 validation tasks and a nominal 800 calls. That setting still differs from published GEPA in decoding and in the added block. We have not run Terminal-Bench.
  • Answer to Question 2. At the median, the state variance is about 0.4 times the task variance. At temperature 0.6 it adds nothing measurable beyond sampling. We did not measure format changes.
  • Answer to Question 3. We tested none of these in a live loop. Section 6 now states what each covers.

Response to the weakness, in detail

  • Does not weaken. At 100 validation tasks, a clock-fixed rerun repeats all 813 calls and two directories keep different children (Section 5.2).
  • Weakens. Rescored without the block, the 100-task children fall between −1 and +3 points (Section 5.3).
  • Few-task decisions are the realistic case for admission. GEPA admits a child on 3 tasks. With the harness's own strings, a fresh identifier changes a median of 10% of 3-task decisions and 13% of 10-task decisions (Appendix H).

Answer to Question 1, in detail

  • Greedy decoding is what lets a same-state rerun repeat, which attribution needs.
  • At temperature 0.6 the practice is to cache calls. A cached rerun returns stored responses, while a new directory changes every request and redraws every sample. The directory then acts like a sampling seed that the rerun does not vary. This is an argument, not a measurement, and Section 6 says so.
  • Terminal-Bench. GEPA's adapter there shows a container identifier. We lack call-exact reruns for shell episodes in containers, so we could not attribute a difference. It is the main open step (Limitations).

Answer to Question 2, in detail

  • Tasks. The state accounts for 24–54% of the variance of a child-minus-parent difference (median 29%). Two 12-task evaluations agree on 90% of decisions at a gap of 10.7 points if only the state is redrawn, and 19.4 if tasks are too (Figure 5a).
  • Sampling. For two hosted models at 0.6, identical requests differ in 67–73% of first queries, and a directory change adds nothing measurable. At temperature 0 it adds 9.75 points for DeepSeek and nothing for GLM (Appendix D, Table 11).
  • Format. Not measured. Our perturbation is small: an equal-length swap of a 20-character hash.

Answer to Question 3, in detail

  • Stable alias. It fixes the state without averaging over it. Each of our directories is such a fixed string, and with the clock fixed four of them keep four different prompts (Table 2).
  • Racing and PACE. Their guarantees cover what evaluations resample. A state fixed through a run is not resampled. A fresh state per evaluation would in principle bring it under the same guarantees.
  • Margin rule. Registered before reading 15 further loops, a two-task margin prevents 7 of 9 regressions at 12 validation tasks. At 100 tasks a four-task margin prevents 1 of 4 and loses the only gain (Figure 5b).

We would be grateful if you would reconsider your scores in light of the rewritten paper.

Thank you for clarifying the 100-task result. Repeating all 813 calls in the same directory while a different directory led GEPA to keep a different prompt convincingly shows what an exact rerun can miss. The comparison with task variance also addresses my second question.

I remain at weak reject. The live-loop result still relies on an added path block, and rescoring without it gives changes ranging from −1 to +3 points. This leaves open how often identifiers exposed by the harness itself lead to worse prompt selection, and whether resampling those identifiers during selection would help.

Reviewer_MDMzOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines how evaluator-generated context, such as directory names and tool timings, affects self-evolving agents, showing that exact reruns can conceal sensitivity to these incidental inputs.

Critical strengths
  1. It identifies a relevant source of variation in agent evaluation.
  2. Controlled interventions distinguish effects on responses, scores, candidate retention, and held-out performance.
Critical weaknesses
  1. The contribution is insufficiently distinguished from known prompt sensitivity and selection noise. Standard statistical analyses and findings from different settings are not organized into a clear argument.
  2. The closed-loop evidence mainly concerns BIRD and Qwen3-8B under modified GEPA/OPRO configurations. Validation in a system that natively exposes such context under its standard configuration is needed.
  3. The strongest regressions rely on a fixed 12-task validation set, leaving evaluator-state effects entangled with selection overfitting. Independent validation splits and equal-budget, live-loop comparisons are needed.
Questions for authors
  1. Could the authors clarify how the proposed findings differ from established prompt sensitivity and selection noise? What evidence demonstrates that evaluator-state effects represent a distinct phenomenon rather than a manifestation of these known issues?

  2. The closed-loop experiments mainly focus on BIRD and Qwen3-8B using modified GEPA/OPRO configurations. Can the authors provide evidence that similar effects occur in systems that natively expose evaluator context under their standard configurations?

  3. The strongest regressions are observed using a fixed 12-task validation set. How do the authors distinguish evaluator-state effects from selection overfitting? Can independent validation splits or equal-budget, live-loop comparisons substantiate the main conclusions?

2 comments
AuthorsOpenReview ↗

Response to Reviewer MDMz

Thank you for the review. You found the paper difficult to follow despite careful reading, and you named the cause: findings from different settings were not organized into a clear argument. You were right, and we rewrote the paper around that diagnosis. Numbers refer to the revised PDF.

Short answers

  • Answer to Question 1 (difference from known issues). Exact reruns repeat perfectly while a fresh identifier changes a median of 13% of ten-task decisions on the same tasks. So this is neither rerun noise nor task-sampling noise, and pairing the versions did not remove it.
  • Answer to Question 2 (native exposure). Sections 3–4 add no text: the identifier is the harness's own. Only the loops add a block. We have not run a loop on a system with native exposure, and we say so.
  • Answer to Question 3 (state versus selection overfitting). All arms use the same validation tasks, budget and optimizer seeds, so overfitting is equal across them. Only the directory name differs. The causal result also holds at 100 validation tasks.

Response to Weakness 1 (organization), in detail

  • The paper now follows three questions, one per section: does the state change the model's outputs (Section 3), which version is kept (Section 4), and what a loop keeps (Section 5).
  • Table 1 lists, for each question, the setting, what we change, and the rerun control.
  • Each statistical tool appears where it is used, with the question it answers.

Answer to Question 1, in detail: three controls

  • Not rerun or decoding noise. Reruns repeat exactly: 240 of 240 first responses, 59 of 59 episodes, 813 of 813 loop calls.
  • Not task-sampling noise. The 13% uses the same tasks in both evaluations. The state accounts for 24–54% of the variance of a child-minus-parent difference (median 29%; Section 6).
  • Not a change to the prompt under evaluation. Both versions are byte-identical across the two evaluations. Two identical copies are even ranked against each other (Section 3).

Answer to Question 2, in detail

  • From Section 4 on we withhold the harness's elapsed-time field from the model, because it breaks exact reruns (Section 3). We add nothing.
  • Our attribution rule needs same-state reruns that repeat every call. We have these only on BIRD. A loop on GEPA's Terminal-Bench adapter is the main open step (Limitations).

Answer to Question 3, in detail

  • Equal-budget live loops. Without the block, GEPA keeps the same child at four seeds, 1 point better. With it, 13 of 14 directories with a lowered bar keep a child, each 3–16 points worse (Section 5.3).
  • Larger validation set. At 100 tasks, a clock-fixed rerun repeats all 813 calls and two directories keep different children. Rescored without the block, one stays 7 points worse and the other gains 2.
  • Independent 12-task splits (post hoc). Over random subsets, the block lowers the initial prompt's correct answers from 5.4 to 4.8–5.2 of 12. Our fixed set amplifies the drop, and Section 5.3 now says so.
  • Our reading. The directory decides which child is kept. Overfitting explains why a child that clears a low bar is worse. We do not attribute the harm to the state alone.

We would be grateful if you would reconsider your scores in light of the rewritten paper.

Thank you for the detailed response. The revision improves the presentation and clarifies the limitations, but my main concerns remain. The GEPA/OPRO loops rely on an added path block, and the subset analysis does not replace independent-split closed-loop runs. Moreover, using the same validation tasks and budget does not guarantee equal overfitting across optimization trajectories. I therefore maintain my current rating.

Reviewer_mVs8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how evaluator state, such as working directory names and random identifiers, affects the reliability of self-evolving LLM agents. Experiments show that these seemingly irrelevant factors can alter model outputs, evaluation results, and prompt selection decisions, potentially leading to worse-performing agents. The authors highlight the limitations of exact reruns and recommend evaluating candidates across fresh evaluator states to improve reliability.

Critical strengths
  1. The paper highlights an overlooked source of instability in self-evolving agents
  2. Easy to follow
  3. The paper provides useful insights into evaluation reliability and motivates more robust protocols for self-evolving agents.
Critical weaknesses
  1. The core observation that LLMs are sensitive to irrelevant contextual variations is already well established. The paper mainly extends this observation to self-evolving agents.
  2. The experiments primarily focus on SQL-writing agents using the BIRD benchmark, making it unclear whether the findings generalize to other agentic tasks.
  3. The GEPA and OPRO experiments explicitly introduce directory information into their prompts, which is not part of their standard benchmark configurations. This limits the practical significance of the findings.
Questions for authors

please see weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer mVs8

Thank you for the review. The paper took more effort to read than it should have, and it has been rewritten (see the general response). Numbers refer to the revised PDF.

Short answers

  • Response to Weakness 1 (known sensitivity). Agreed, and not our claim. New is that exact reruns cannot see this variation, that giving both versions the same string did not remove it, and that in a loop it decides which prompt is kept.
  • Response to Weakness 2 (only BIRD). Correct for the benchmark. We cover six models, two optimizers and a 15-system source audit, and we do not claim that effect sizes transfer.
  • Response to Weakness 3 (added directory information). True for Section 5 only. Sections 3–4 add no text.

Response to Weakness 1, in detail

  • Reruns certify results that do not replicate. All 240 exact reruns match, while a new identifier of equal length changes 48 of 240 first queries (Section 3). Reruns reproduce every ten-task decision, while fresh identifiers change a median of 13% (Section 4.3).
  • Identical versions get ranked. Two byte-identical copies of one prompt, each under its own identifier, are ranked in all five task groups when ties are broken by cost (Section 3).
  • Pairing did not help. Of 199 grade changes, only 7 involve both versions. Sharing an identifier changes the disagreement by +0.07 points on average (Section 4.3).

Response to Weakness 3, in detail

  • In Sections 3–4 the identifier is the harness's own hashed folder name. We replace only its value.
  • The block added in Section 5 is the one this harness renders, printed verbatim in Section 5.1.
  • Optimizers meet such strings when they run on agents: GEPA's Terminal-Bench adapter shows a container identifier, and SICA and ADAS show identifiers or timestamps (new Figure 2). HSI passes a timestamped run path into its selection inputs (Appendix E).
  • A full loop on a system with native exposure is the main open step (Limitations).

Response to Weakness 2, in detail

  • Models: Qwen3-8B and 32B, Mistral-7B and Mistral-Small-24B, and two hosted models. One hosted model shows no effect, and we report that.
  • The mechanism needs only evaluator strings in the input and decisions on few tasks. Published loops decide on few tasks: GEPA admits a child on 3, DGM screens on 10, ADAS selects on 20 (Table 12).
  • We chose BIRD because our attribution rule needs reruns that repeat call for call.

We would be grateful if you would reconsider your rating in light of the rewritten paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.