- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Agent harnesses write incidental strings, such as hashed directories, identifiers and latencies, into model inputs. Exact reruns repeat these strings, so they hide the strings' effect on keep/reject decisions in self-evolving loops. The paper frames this evaluator state as a generalizability-theory facet and shows that a shared state cancels only the shifts both candidates share. On BIRD with Qwen3-8B, swapping identifiers changes many queries but few grades. At 10 tasks, decisions disagree on a median of 13% of cases. With an injected path block, the directory name alone changes which prompt GEPA and OPRO keep.
Critical strengths
A clean causal demonstration that exact reruns hide evaluator-written context that changes which version a self-evolving loop keeps, backed by rigorous pre-registered controls.
Critical weaknesses
The evidence that this causes harm rests on an injected block, a 12-task validation set and a single benchmark. The effect weakens in realistic settings, and the paper tests no comparison against existing selection methods or simple mitigations.
Questions for authors
- Does the effect hold with published GEPA settings, on a harness that natively exposes these strings, such as Terminal-Bench?
- How large is V_state compared with V_task, with prompt-format perturbations, and with sampling variance at T=0.6?
- Does a stable-alias arm, or racing- or PACE-style selection with fresh states, remove the state-induced instability?
2 comments
Thank you for clarifying the 100-task result. Repeating all 813 calls in the same directory while a different directory led GEPA to keep a different prompt convincingly shows what an exact rerun can miss. The comparison with task variance also addresses my second question.
I remain at weak reject. The live-loop result still relies on an added path block, and rescoring without it gives changes ranging from −1 to +3 points. This leaves open how often identifiers exposed by the harness itself lead to worse prompt selection, and whether resampling those identifiers during selection would help.
Response to Reviewer 2o2F
Thank you for the review, and for recognizing the causal demonstration. Your summary states our argument more clearly than our own introduction did, so we rewrote the paper to follow its order. Numbers refer to the revised PDF.
Short answers
Response to the weakness, in detail
Answer to Question 1, in detail
Answer to Question 2, in detail
Answer to Question 3, in detail
We would be grateful if you would reconsider your scores in light of the rewritten paper.