acceptodds
Under review as a conference paper at ICLR 2027

Measured Behavior, Recorded Zero: Measurement Failures in Self-Improving Coding Agents

Abstract

Self-improving coding agents evolve a scaffold around a frozen language model and select variants using benchmark performance. We study whether that selection signal reliably reflects what the agent and the search actually did. On the 59-task Polyglot subset used by the Huxley–Gödel Machine (HGM), the default self-hosted setup records zero tool use at 7B, 14B, and 32B, yet the 14B transcripts contain dispatchable tool calls on 30/59 tasks. Recovering those calls changes tool use from 0/59 to 32/59 while resolve rate remains 0/59. In two evolutionary runs, all 14 proposed children are invalid Python and are archived with the same utility (0.0) that a valid child solving no tasks would receive. Replacing the proposal model with a frontier hosted model changes validity from 0/14 to 14/14 while preserving the same destructive edit pattern. Across ten configurations, resolve rates remain between 0/59 and 4/59. These results do not show that scaffold evolution fails. They show that, in this deployment regime, the recorded objective can collapse distinct behavioral states and can therefore obscure whether a search step acted, failed, or produced an executable candidate. We provide a network-level measurement layer and a set of inexpensive controls for evaluating self-improving agents.

Primary area:
datasets and benchmarks
Keywords:
self-improving agents, meta-agents, agent evaluation, measurement validity, coding agents, tool use, LLM agents, benchmark design

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_x4cuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates measurement failures in self-improving coding agents, focusing on a self-hosted deployment of HGM. It identifies two failure modes: tool calls emitted in message text are not dispatched by the harness, and syntactically invalid candidate programs receive the same zero utility as valid candidates that solve no tasks. Recovering tool calls increases tool participation from 0/59 to 32/59 tasks without improving task success, while two evolutionary runs produce 14 invalid children. The paper recommends explicit validity checks, separate failure states, and trajectory-level logging.

Critical strengths
  • The analysis examines trajectories and reconstructed patches, providing concrete explanations for failures that aggregate success rates alone cannot diagnose.
  • The parser intervention directly demonstrates the tool-interface mismatch, and the implementation audit usefully distinguishes HGM’s behavior from safeguards already present in DGM and Hyperagents.
Critical weaknesses
  1. The contribution is too limited for a main-conference research paper. The central observations are a tool-call serialization mismatch in a particular self-hosted deployment and a missing validity check in one implementation. The proposed remedies—syntax checks, explicit failure states, and trajectory logging—are standard engineering practices. Appendix J further establishes that two of the three audited systems already distinguish invalid candidates from valid zero-scoring candidates. The paper therefore documents a narrow implementation defect without establishing a substantially new evaluation methodology or a broader empirical finding.

  2. The empirical study never reaches a functioning self-improvement process. Neither evolutionary run produces a single syntactically valid child. Consequently, the paper cannot evaluate whether the proposed measurements improve candidate selection, reduce wasted search budget, or enable better final performance. The ten configurations largely characterize near-zero performance under related deployment conditions; they do not substitute for an experiment demonstrating consequences for actual scaffold evolution. The frontier-model proposal control establishes syntactic validity, but the paper does not adequately document the execution and evaluation of those children.

  3. The central argument conflates diagnostic information with the optimization objective. If two agents solve no tasks, assigning both a success rate of zero is not itself a measurement failure. Recovering tool calls changes behavior, but the recovered actions still fail to solve tasks and frequently target the wrong artifacts. The paper does not establish that distinguishing these behaviors would improve selection. Similarly, a statistically insignificant difference between 0/59 and 4/59 does not imply that a selector cannot distinguish their numerical scores. The stronger claims about an uninformative selection objective are therefore not supported by the reported interventions.

  4. A consequential modification to parent selection undermines attribution to the released system. Appendix G states that the authors modified selection to fall back to all evaluated nodes when the original implementation would otherwise fail. This directly affects whether invalid zero-scoring children can participate in subsequent selection. However, the main text presents their continued eligibility as a property of the released system without adequately separating the original behavior from the patched behavior. The paper must establish which claimed consequences occur without this intervention.

  5. The compute-matched baseline is insufficiently specified and confounded. The main text describes a matched-budget comparison, whereas Appendix F discusses tripling the sampling budget and Appendix G states that compute is reported rather than matched. The precise matching relationship is unclear. Moreover, best-of-3 uses the required tool policy while the default baseline uses auto, so the comparison changes both sampling and protocol enforcement. The corresponding single-trajectory required baseline already solves 4/59 tasks, compared with 3/59 for best-of-3. These results do not establish the claimed attribution to repeated sampling.

Questions for authors

See Weaknesses.

Reviewer_Pr1YOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper analyzes the evaluation protocol of three recursive self-improvement (RSI) algorithms: Darwin-Goedel Machine, Huxley-Goedel Machine (HGM), and HyperAgents. The main takeaway of the paper is that, in the self-hosted deployment regime the authors study, the recorded objective can collapse distinct behavioral states and therefore obscure whether a search step acted, failed, or produced an executable candidate. The audit shows that, for example, in HGM with some self-hosted models, the model calls the right tools but in the wrong format – in the message and not in the structured tool-call field. However, fixing this failure mode did not improve the results. Moreover, the authors show that the HGM archive does not distinguish between a broken child and a child with zero performance. Experiments performed on frontier models reveal that the percentage of structured tool calls and the validity of children increase from 0% to 100%.

Critical strengths
  1. The paper is clearly written.
  2. The reporting is rigorous, both statistically (Fisher’s exact test p-values, Clopper–Pearson intervals) and in its honesty about the research process: the authors report which of their initial hypotheses were falsified (Appendix D).
Critical weaknesses
  1. The paper shows that in methods designed for large models, the recorded objective can collapse distinct behavioral states when used with weaker, self-hosted, quantized models. This is a very interesting observation. However, it appears to hold only in a particular regime, and the authors do not demonstrate that accounting for it would alter the utility of the RSI algorithm. Therefore, I do not think this contribution is sufficient for a main-track paper at a top-tier conference.
  2. As a concrete example, the authors state, “A useful archive should therefore retain a distinction between (i)… (ii)… (iii)”, which I agree with. But useful to whom? The selection method? A human designer? Where in the paper is this usefulness demonstrated?
Questions for authors
  1. What is the consequence of Contribution (3)? The authors write that the HGM does not perform a validity check and recommend removing children without a valid score from parent selection. Moreover, they state, “HGM retains the same predicate and its import, but the released selection path never calls it. [...] Thus the HGM observation is a concrete wiring defect in a released implementation”. Does the HGM paper state that they utilize that gate? Appendix G says, “HGM samples archive nodes of strictly positive utility before expanding one”, so a child with utility 0.0 won’t be selected. Therefore, why do we need this gate, aside from improving the interpretability of the results? Which HGM decision would it change?
  2. In the paper, the authors state: “mean_utility: 0.0—arithmetically identical to a working child that ran and solved nothing. Those nodes then compete for the selection, consume evaluation budget, and become eligible parents.” This echoes the quoted sentence from the previous point. Do I understand correctly that this is the result of your patch to the HGM algorithm?
  3. [Minor] The observation from Appendix D about Qwen2.5-Coder’s problems with parsing <tool_call> seems to be documented in issue (vLLM #32926). Can you elaborate on how this observation relates to that issue?
Reviewer_P7mUOpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether the scalar benchmark scores used to select variants in self-improving coding agents preserve enough information about what the agent and evolutionary search actually did. Using the 59-task Polyglot subset associated with HGM, the authors instrument model traffic at the network level and compare recorded tool use, transcript-level attempted calls, executed actions, candidate validity, and final task resolution.

The paper reports two main failure modes. First, self-hosted Qwen models can emit dispatchable tool calls in the message body while the harness records zero structured calls, so repairing the parser changes observed tool participation without changing the final resolve count. Second, evolutionary proposals that are invalid Python can be inserted into the archive with the same zero utility as executable candidates that solve no tasks. The study evaluates ten configurations, audits related implementations, and proposes additional validity, trajectory, and matched-compute controls for future evaluations of self-improving agents.

Critical strengths
  1. The paper identifies an important measurement problem that is easy to miss when self-improving systems are evaluated only through a final scalar score. The separation among attempted actions, harness-consumed actions, executable candidates, and benchmark outcomes is conceptually useful and has direct implications for the design of agent evaluation pipelines.

  2. The network-level instrumentation and implementation audit provide concrete evidence rather than relying only on abstract arguments. The paper also examines multiple model scales, another model family, a second benchmark, parser and policy interventions, and candidate validity. Its repeated emphasis that the findings do not establish the failure of scaffold evolution is appropriately cautious.

Critical weaknesses
  1. The experimental configurations do not support a clean interpretation of several central comparisons. The main text describes repeated sampling as a matched model-call-budget baseline, but the best-of-3 arm also changes the tool policy to required, triples sampling, and is compared in different passages with both the default auto arm and a single required-policy arm. The evolutionary runs introduce a more serious provenance issue: the released positive-utility parent selector would crash after the first all-zero generation, so the reported search proceeds only after adding a fallback that selects from all evaluated nodes. The paper does not present the archive transition, seed retention, and parent eligibility rules with enough precision to separate observations about the released system from behavior created by the experimental modification. These confounds weaken the causal and methodological conclusions drawn from the controls.

  2. Several reported categories and statistical comparisons are not aligned with the experimental design. Page 3 calls 47 of 59 outputs syntactically valid JSON even though only 38 parse, whereas Table 7 distinguishes emitted, parsed, and runnable calls. More importantly, configurations are evaluated on the same tasks, but the paper reports pairwise Fisher exact tests as if the samples were independent rather than using the paired task outcomes. With only zero to four successes per arm, 45 pairwise comparisons, and no paired contingency tables, the claims about indistinguishable resolve rates and the interpretation of the success floor are not sufficiently supported. The prose also sometimes treats a non-significant change from zero to two or four solved tasks as behaviorally invisible, conflating an observed numerical difference with a failure to reject the null.

Questions for authors
  1. For every comparison involving auto, required, best-of-3, and the evolutionary seed, what were the exact tool policy, number of trajectories per task, model-call budget, token budget, and selection rule? Which specific pair is intended to be compute matched?

  2. In the evolutionary runs, does the 4/59 seed remain in the archive after the first generation, and which nodes are eligible for parent selection under the released code versus the patched fallback? Please provide a short generation-level archive trace for one run.

  3. What are the exact paired task-level contingency tables for the main configuration comparisons, and how do the conclusions change under exact McNemar tests with an appropriate treatment of multiple comparisons? Please also reconcile the emitted, parsed, and runnable JSON counts and distinguish numerical changes in resolved tasks from statements about statistical significance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.