- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates measurement failures in self-improving coding agents, focusing on a self-hosted deployment of HGM. It identifies two failure modes: tool calls emitted in message text are not dispatched by the harness, and syntactically invalid candidate programs receive the same zero utility as valid candidates that solve no tasks. Recovering tool calls increases tool participation from 0/59 to 32/59 tasks without improving task success, while two evolutionary runs produce 14 invalid children. The paper recommends explicit validity checks, separate failure states, and trajectory-level logging.
Critical strengths
- The analysis examines trajectories and reconstructed patches, providing concrete explanations for failures that aggregate success rates alone cannot diagnose.
- The parser intervention directly demonstrates the tool-interface mismatch, and the implementation audit usefully distinguishes HGM’s behavior from safeguards already present in DGM and Hyperagents.
Critical weaknesses
-
The contribution is too limited for a main-conference research paper. The central observations are a tool-call serialization mismatch in a particular self-hosted deployment and a missing validity check in one implementation. The proposed remedies—syntax checks, explicit failure states, and trajectory logging—are standard engineering practices. Appendix J further establishes that two of the three audited systems already distinguish invalid candidates from valid zero-scoring candidates. The paper therefore documents a narrow implementation defect without establishing a substantially new evaluation methodology or a broader empirical finding.
-
The empirical study never reaches a functioning self-improvement process. Neither evolutionary run produces a single syntactically valid child. Consequently, the paper cannot evaluate whether the proposed measurements improve candidate selection, reduce wasted search budget, or enable better final performance. The ten configurations largely characterize near-zero performance under related deployment conditions; they do not substitute for an experiment demonstrating consequences for actual scaffold evolution. The frontier-model proposal control establishes syntactic validity, but the paper does not adequately document the execution and evaluation of those children.
-
The central argument conflates diagnostic information with the optimization objective. If two agents solve no tasks, assigning both a success rate of zero is not itself a measurement failure. Recovering tool calls changes behavior, but the recovered actions still fail to solve tasks and frequently target the wrong artifacts. The paper does not establish that distinguishing these behaviors would improve selection. Similarly, a statistically insignificant difference between 0/59 and 4/59 does not imply that a selector cannot distinguish their numerical scores. The stronger claims about an uninformative selection objective are therefore not supported by the reported interventions.
-
A consequential modification to parent selection undermines attribution to the released system. Appendix G states that the authors modified selection to fall back to all evaluated nodes when the original implementation would otherwise fail. This directly affects whether invalid zero-scoring children can participate in subsequent selection. However, the main text presents their continued eligibility as a property of the released system without adequately separating the original behavior from the patched behavior. The paper must establish which claimed consequences occur without this intervention.
-
The compute-matched baseline is insufficiently specified and confounded. The main text describes a matched-budget comparison, whereas Appendix F discusses tripling the sampling budget and Appendix G states that compute is reported rather than matched. The precise matching relationship is unclear. Moreover, best-of-3 uses the required tool policy while the default baseline uses auto, so the comparison changes both sampling and protocol enforcement. The corresponding single-trajectory required baseline already solves 4/59 tasks, compared with 3/59 for best-of-3. These results do not establish the claimed attribution to repeated sampling.
Questions for authors
See Weaknesses.