- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose HGM-Dual, an extension of the Huxley-Gödel Machine for optimizing the prompts, tools, and workflow of a frozen LLM agent. It keeps HGM's search loop but gives the editor a summary of edits and outcomes along the parent's lineage. When pre-search stability checks pass, it generates one open-ended child and two children targeting frequent error categories, retaining only the best. On DeepPlanning's travel and shopping tasks, HGM-Dual outperforms the HGM baselines at 4k evaluations. They also report matching the HGM variants' final performance with 2-4× fewer task-agent evaluations.
Critical strengths
-
Harness search is expensive (Table 6 estimates hundreds of dollars per 1k task-agent evaluations), and making better use of previous edits is a good way to improve it. Both components fit into HGM without changing parent selection.
-
At 4k evaluations, HGM-Dual improves both composite score and case accuracy over HGM-CF in both domains, and the held-out error breakdown helps explain the those gains. Shopping failures involving unmet features fall from 24.2% for the seed to 10.8%, compared with 20.8% for HGM-CF. The analyses also provide some evidence for the mechanisms. Table 3 shows that evolution context increases the fraction of edits improving on their parent from 31.3% to 41.0%, and the error-conditioning ablation keeps the number of children and best-of-three selection fixed, which is a useful control.
Critical weaknesses
- The evidence for the paper's main claims is incomplete. My main concern is the eval split. The headline results combine the 60 optimization instances with 60 unseen instances. Figure 4c,d provides encouraging held-out failure analysis, including the overall rate of tasks with any failure. However, I would like to see the composite score, case accuracy, and budget curves reported separately on the unseen instances. Otherwise it's difficult to tell how much of the headline improvement generalizes beyond the cases used during search.
The uncertainty and cost reporting also need clarification. Appendix A.3 mentions independent runs, but I can't determine how many contributed to each result or what the ± values represent. Variation across searches and variation when re-evaluating a fixed harness should be distinguished. This is important for both the performance gains and the claimed 2–4× budget savings. The authors correctly count evaluations of rejected children, but the budget measures generated plans instead of total cost, and editor costs (and summarization costs, if "Summarize" is an LLM call) are not reported.
The error-conditioning ablation is useful but is limited to shopping at 1k evaluations. It reports 90.35 with conditioning versus 87.68 without it, but the unclear uncertainty reporting makes this difference hard to assess. Evolution context alone already raises HGM-CF from 84.01 to 88.21, so the additional benefit of targeted generation needs a closer look. The ablation removes the error category from the prompt. It's not clear whether category-specific traces remain. If they do, the ablated editor might still receive information about the targeted error. Results on travel and across independent searches would make this component more convincing.
Looking at all of this, the gains might be real, but without held-out results and run-to-run variance I can't tell whether they exceed the noise, which keeps me below the acceptance threshold.
- The novelty is incremental and the scope is narrow. The contribution builds on existing approaches that use previous candidates or execution feedback, including Meta-Harness, ADAS, and GEPA. The specific addition is the structured lineage summary combined with targeted child generation. A comparison with an unstructured history under the same context budget would help isolate the summary's value. The scope is also limited to one benchmark, one editor model, and domain-specific error categories. The Mistral experiment broadens the evidence a bit, but case accuracy stays at the seed's 1.6% despite a higher composite score. The results support the method on DeepPlanning, but its broader usefulness is less clear.
Related work to look into: Trace (Cheng et al., 2024) uses execution traces and feedback to update workflows, close to the role of the lineage summary. STOP (Zelikman et al., COLM 2024) and Gödel Agent (Yin et al., ACL 2025) are earlier self-improving agents in the same line as DGM and HGM. VeRO (Ursekar et al., ICML 2026) and HarnessOpt-Bench (Ursekar et al., 2026) study how to evaluate harness optimization itself, including held-out scoring and how gains vary across tasks.
Questions for authors
-
Could you report Tables 1–2 and Figure 4a,b separately on the 60 unseen instances? This would really address my main concern about generalization.
-
How many independent searches and repeated evaluations of each selected harness contribute to the results? What do the ± values and plotted error bars represent? Please report variation across searches separately from re-evaluation noise.
-
Does the error-conditioning ablation remove category-specific traces as well as the category label? Could you report the ablation on travel and clarify the number of independent searches behind Table 4?
-
How many tasks is each child evaluated on before selection, and what are the editor costs per method (and summarization costs, if Summarize is an LLM call)? Please also confirm the editor models used and whether HGM-CF and HGM-Dual receive otherwise matched inputs and instructions.
-
How are κ and ρ_B calculated, how were their thresholds chosen, and did every experimental setting pass the pre-search stability checks?
2 comments
Does the error-conditioning ablation remove category-specific traces as well as the category label? Could you report the ablation on travel and clarify the number of independent searches behind Table 4?
Response: The error conditioning ablation changes only what the error-conditioned children see. There is no error-conditioned child agent generation. As a result, it loses:
-
its assigned category label,
-
the traces for that category (two failing instances, each with query, plan and failure).
Instead, each child is asked to come up with edits aimed at whatever most limits this agent’s score. The general edit, the shared batch, the number of children and best-of-pool selection are unchanged. The ablated children still receive the generic failure report similar to HGM-Dual and HGM-CF. The ablation therefore tests assigning each child its own category and traces, not access to error information.
On travel, the pre-search test disables error conditioning (κ = 0.27 < 0.4, ρ₁₆ = 0.36 < 0.7). The travel HGM-Dual is therefore HGM-CF plus evolution context, and it has no error conditioning to ablate.
How many tasks is each child evaluated on before selection, and what are the editor costs per method (and summarization costs, if Summarize is an LLM call)? Please also confirm the editor models used and whether HGM-CF and HGM-Dual receive otherwise matched inputs and instructions.
Response: Each child is evaluated on 16 optimization instances before any selection.
We follow the protocol established by previous Recursive Self-improvement works, DGM (Zhang et al., 2025) and HGM (Wang et al., 2025) to use number of task agent evaluation as the estimate of cost/budget for independent search process. This is because the task agent evaluation is the major source of cost. Editor and summarizer together use less than 0.5% of tokens in every arm for travel. While the total cost of one independent search of HGM-Dual for 4k budget evaluation is ~ 2500$, only editor + summarizer cost is around ~60$. Over HGM-CF, HGM-Dual adds only $34 per independent search on travel, which is negligible compared to the total task agent cost.
In every baseline, HGM-Dual and ablation search, the editor is GPT-5.4 (reasoning effort medium). The summarizer is GPT-5.4 (medium), and the task agent is GPT-5.4-mini (high). HGM-CF and HGM-Dual receives matched inputs and instruction. The only different is the two components: agent evolution as interpretable context, and error-conditioned child agent generation.
How are κ and ρ_B calculated, how were their thresholds chosen, and did every experimental setting pass the pre-search stability checks?
Response:
How κ and ρ_B are computed.
κ and ρ_B are computed once per task, from two independent evaluations of the seed harness on the 60 optimization instances.
κ is Cohen’s kappa between the two evaluations:
κ = (p_o − p_e) / (1 − p_e), p_e = f₁f₂ + (1 − f₁)(1 − f₂)
Here p_o is the fraction of instances with the same outcome in both evaluations, and p_e is the agreement expected from the failure rates f₁ and f₂ alone. f₁ and f₂ are failure rates from two independent runs. Κ measures task-level agreement.
ρ_B is the probability that a random batch of B instances has the same top error category in both evaluations:
ρ_B = Pr_b [ top(b, 1) = top(b, 2) ], b ⊂ {1,…,N}, |b| = B
top(b, r) is the highest-ranked non-generic error category on batch b in evaluation r, using HGM-Dual’s own categorizer and tie-break. ρ_B measures dominant-category stability. We use B = 16, the batch on which HGM-Dual ranks categories during an expansion and estimate ρ₁₆ from 5,000 random batches.
How the thresholds were chosen.
κ ≥ 0.4 is the conventional lower bound of moderate agreement (Landis & Koch, 1977). Below it, a re-evaluation substantially changes which instances fail.
An error-conditioned child is built around a single category, the top error category on the 16-instance batch. This helps only if that category is a reproducible weakness of the harness. If re-evaluating the same harness on the same batch would rank a different category first, the child specializes on sampling noise, and its evaluation still consumes budget. ρ_B measures this directly. A threshold of 0.7 guarantees that at least 70% of error-conditioned children target it. We require a clear majority rather than 0.5 because each child costs a full batch evaluation.
Did every setting pass?
Travel HGM-Dual result uses the general path with evolution context and no error-conditioned children. Every shopping HGM-Dual result uses the dual path.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.
We thank the reviewer for the careful review and constructive feedback.
Response: We thank the reviewer for this suggestion. Below we report Tables 1 and Fig. 4(a,b) recomputed on the 60 unseen instances only.
Table: Performance on the 60 unseen instances (budget 4000)
Performance for different budget scenarios on the 60 unseen instances (Fig 4(a,b))
HGM-Dual's improvements carry over to unseen instances. It is clearly better than the seed in both tasks at every budget, and it keeps most of its gain: 9.6 of 13.1 points in travel and 6.6 of 8.1 in shopping. The baseline methods transfer less to the unseen instances. HGM-CF's performance drops from 85.5 on the optimization instances to 82.7 on the unseen ones at 4k evaluation budget. HGM-Dual also maintains similar performance trend at different budget scenarios for unseen samples.
Previous Recursive Self-improvement works, DGM (Zhang et al., 2025) and HGM (Wang et al., 2025) use the same set of samples for optimization and evaluation. We have followed the same protocol and reported the performance of HGM-Dual on all samples. We will add the performance of unseen cases in the revised version.
Response: For HGM-Dual, we ran two independent searches per task. The harness each search selects at a given budget was evaluated twice on all 120 instances, giving four evaluations per reported point. Previous Recursive Self-improvement works, DGM (Zhang et al., 2025) and HGM (Wang et al., 2025) report result through one search per task. We have increased it to two searches and multiple evaluations for better stability understanding. The ± values in Tables 1–2 and the error bars in Fig. 4(a,b) are the sample standard deviation over all evaluations behind an entry. For HGM-Dual it pools from two independent searches and two evaluations per search. The intervals in Fig. 4(c,d) are 95% paired bootstrap intervals over the 60 held-out instances. We resample instances with replacement (10,000 times, with the same resample for both methods) and recompute the difference in failure rate. They reflect uncertainty from which instances are in the benchmark, not variation across searches.
Separating the two independent searches, at the 4k budget:
Travel: the two HGM-Dual searches reach 88.28 ± 0.15 and 87.27 ± 2.03 composite score, against 84.11 for second best HGM-CF.
Shopping: HGM-Dual searches reach 88.24 ± 0.83 and 90.23 ± 0.82 composite score, against 86.67 for second best HGM-CF.
Each HGM-Dual search scores above HGM-CF on both tasks, by 3.2–4.2 points on travel and 1.6–3.6 points on shopping. The averaged HGM-Dual results reported in the paper (87.77 ± 1.31 and 89.23 ± 1.33) therefore do not depend on one favorable search. We will report per-search values and both sources of variation separately in the revision.