- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses component-level credit assignment for self-evolving LLM agents on long-horizon tasks, where a single task outcome cannot reveal which edits actually produced their intended behavior. The method registers each mutation intent before editing, then uses paired replay from a shared parent checkpoint to test whether each candidate expresses the intended behavior. The resulting expression signals are used to prune candidates before full evaluation and to weight outcome-based credit. These records accumulate in an Evolution Context that guides later mutations. The method is validated on SEAL-0 and EnterpriseOps-Gym against an adapted GEPA baseline, with additional pruning and context ablations.
Critical strengths
S1: Meaningful problem formulation. The paper separates "an edit that was not executed" from "an edit that was executed but ineffective," which addresses a real and underexplored source of noise in outcome-only credit assignment for long-horizon agent evolution.
S2: Exceptional transparency. The appendices disclose full evolution archives (including negative entries), complete agent configurations, lineage trees, and per-task justifications for benchmark exclusions, and even report details unfavorable to the authors.
Critical weaknesses
W1: Lack of direct validation for the core contribution. The paper's central claim is that credit grounded in behavioral expression is superior to credit based solely on task outcomes. However, no experiment isolates this comparison while holding all other factors fixed: Section 4.5 removes the entire Evolution Context, while Section 4.4 simultaneously changes both candidate generation and pruning. Moreover, 12 of the 13 entries in the final EnterpriseOps archive carry negative importance, including components actually used by the best-performing agent. This mismatch between credit sign and the components' actual contribution undermines the credibility of the claimed "component-level credit assignment."
W2: Severely insufficient baselines and unfair comparisons. The experiments include almost no comparison with related work. On SEAL-0, the only baseline is GEPA, adapted by the authors themselves, which stagnates after its second iteration; on EnterpriseOps, no external method is evaluated at all. The most relevant prior works, such as EVOTOOL, ADAS, DGM, and AFlow, appear only in the related-work section. The most natural baseline is also missing: low-fidelity screening of all candidates on a task subset under a matched compute budget. Section 4.4 justifies fairness by matching the number of full task evaluations, yet the replay overhead alone exceeds the task-evaluation cost, so the pruning condition consumes more than twice the total compute of its counterpart. Consequently, the current results cannot disentangle the contribution of the method itself from the gains brought by additional compute.
W3: The applicability boundary of the core mechanism remains unclear. Distinguishing "an edit that was not executed" from "an edit that was executed but ineffective" is a reasonable motivation, but its effectiveness rests on several premises that the paper does not test, and these premises break down in different ways across model scales and task environments. First, short-horizon continuation from a parent checkpoint implicitly assumes that edits affect only local decisions; the more an edit restructures the overall workflow, or the longer, more stateful, and more nondeterministic the task, the larger the gap between replay and actual execution, and the less reliable the check becomes. Second, "expression" measures compliance with the new instruction rather than whether the edit is effective, and the relationship between the two is never validated. The discriminative power of this signal also depends on model scale: strong models almost always comply with instructions, so expression saturates and offers little additional information, whereas weak models behave stochastically, making the expression signal noisy and requiring many replay repetitions to stabilize. The regime in which the mechanism is genuinely effective may therefore be limited to mid-capability models and a narrow class of tasks where evaluation is expensive and local replay is feasible. Finally, the sign of credit is determined entirely by task outcomes; after normalization and LLM-based adjudication, the actual contribution of expression weighting to final decisions is unclear.
W4: The paper as a whole exhibits the immature characteristics of an auto-research output. The method overview (Figure 1) is a raster image with dense elements and very small text, and its layout resembles the output of a text-to-image model, making it difficult to serve as an accurate description of the method; the remaining figures contain annotations that do not correspond to their axes and residual internal run identifiers. The pseudocode consists mainly of opaque function calls and steps such as "update using an LLM." The writing relies heavily on metaphors, and terms are often used before being defined; the main-text method section is under two pages, while the appendix consists largely of verbatim dumps of prompts and configurations. The experimental setup is simplistic and insufficiently rigorous: the development sets contain only 5–8 tasks, each condition is run with a single evolution run, and only one model family is used. Overall, the paper reads more like an intermediate product of an automated research pipeline than a mature work refined under a unified experimental protocol.
Questions for authors
Q1 : Under a matched compute budget, does paired-replay pruning outperform simply screening all candidates on a task subset, and if not, what does the replay mechanism contribute beyond additional compute?
Q2 : With all other components held fixed, how does expression-weighted credit compare against outcome-only credit, and how do you reconcile the predominantly negative final archive with the fact that the best agent uses those very components?
Q3 : The headline Qwen3.5 SEAL-0 run was restarted from the R4 parent after a bug, and the final agent was generated in R6. Do the results hold across multiple independent, uninterrupted evolution runs?
Overall, I like this paper. If the authors can address my questions during the discussion period, I would consider raising my score.