acceptodds
Under review as a conference paper at ICLR 2027

HGM-Dual: Lineage-Aware Guided Agent Harness Optimization for Long-Horizon Planning

Abstract

Harness optimization, which improves an agent's prompts, tools, and workflows, is an increasingly effective way to enhance agents built on frozen or closed-source LLMs. Leading approaches such as the Darwin–G\"odel Machine and the Huxley–G\"odel Machine (HGM) rely on recursive self-improvement and were developed mainly for coding, where the competence needed to improve the agent coincides with the competence being improved. Planning tasks, in which an agent must produce a sequence of tool calls that jointly satisfies interacting constraints, break this alignment. There, open-ended self-modification repeatedly explores ineffective changes and consumes substantial evaluation budgets. We introduce HGM-Dual, a diagnostic-guided framework for budget-efficient harness evolution for planning agents. First, HGM-Dual represents agent evolution as lineage-aware context. This context records what each ancestor changed, which behaviors improved or regressed, and which modifications were ineffective, giving the editor explicit edit–outcome attribution instead of an unstructured archive of past agents. Second, HGM-Dual introduces error-conditioned dual-path child generation. For each selected parent, it generates a general child that preserves open-ended exploration and specialized children that target the harness components most likely responsible for the parent's highest-ranked constraint violations. HGM-Dual outperforms the strongest baseline by up to 14.45% relative improvement in case accuracy while matching HGM variants' final performance with 2X–4X fewer evaluation budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Recursive Self Improvement, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pztgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose HGM-Dual, an extension of the Huxley-Gödel Machine for optimizing the prompts, tools, and workflow of a frozen LLM agent. It keeps HGM's search loop but gives the editor a summary of edits and outcomes along the parent's lineage. When pre-search stability checks pass, it generates one open-ended child and two children targeting frequent error categories, retaining only the best. On DeepPlanning's travel and shopping tasks, HGM-Dual outperforms the HGM baselines at 4k evaluations. They also report matching the HGM variants' final performance with 2-4× fewer task-agent evaluations.

Critical strengths
  1. Harness search is expensive (Table 6 estimates hundreds of dollars per 1k task-agent evaluations), and making better use of previous edits is a good way to improve it. Both components fit into HGM without changing parent selection.

  2. At 4k evaluations, HGM-Dual improves both composite score and case accuracy over HGM-CF in both domains, and the held-out error breakdown helps explain the those gains. Shopping failures involving unmet features fall from 24.2% for the seed to 10.8%, compared with 20.8% for HGM-CF. The analyses also provide some evidence for the mechanisms. Table 3 shows that evolution context increases the fraction of edits improving on their parent from 31.3% to 41.0%, and the error-conditioning ablation keeps the number of children and best-of-three selection fixed, which is a useful control.

Critical weaknesses
  1. The evidence for the paper's main claims is incomplete. My main concern is the eval split. The headline results combine the 60 optimization instances with 60 unseen instances. Figure 4c,d provides encouraging held-out failure analysis, including the overall rate of tasks with any failure. However, I would like to see the composite score, case accuracy, and budget curves reported separately on the unseen instances. Otherwise it's difficult to tell how much of the headline improvement generalizes beyond the cases used during search.

The uncertainty and cost reporting also need clarification. Appendix A.3 mentions independent runs, but I can't determine how many contributed to each result or what the ± values represent. Variation across searches and variation when re-evaluating a fixed harness should be distinguished. This is important for both the performance gains and the claimed 2–4× budget savings. The authors correctly count evaluations of rejected children, but the budget measures generated plans instead of total cost, and editor costs (and summarization costs, if "Summarize" is an LLM call) are not reported.

The error-conditioning ablation is useful but is limited to shopping at 1k evaluations. It reports 90.35 with conditioning versus 87.68 without it, but the unclear uncertainty reporting makes this difference hard to assess. Evolution context alone already raises HGM-CF from 84.01 to 88.21, so the additional benefit of targeted generation needs a closer look. The ablation removes the error category from the prompt. It's not clear whether category-specific traces remain. If they do, the ablated editor might still receive information about the targeted error. Results on travel and across independent searches would make this component more convincing.

Looking at all of this, the gains might be real, but without held-out results and run-to-run variance I can't tell whether they exceed the noise, which keeps me below the acceptance threshold.

  1. The novelty is incremental and the scope is narrow. The contribution builds on existing approaches that use previous candidates or execution feedback, including Meta-Harness, ADAS, and GEPA. The specific addition is the structured lineage summary combined with targeted child generation. A comparison with an unstructured history under the same context budget would help isolate the summary's value. The scope is also limited to one benchmark, one editor model, and domain-specific error categories. The Mistral experiment broadens the evidence a bit, but case accuracy stays at the seed's 1.6% despite a higher composite score. The results support the method on DeepPlanning, but its broader usefulness is less clear.

Related work to look into: Trace (Cheng et al., 2024) uses execution traces and feedback to update workflows, close to the role of the lineage summary. STOP (Zelikman et al., COLM 2024) and Gödel Agent (Yin et al., ACL 2025) are earlier self-improving agents in the same line as DGM and HGM. VeRO (Ursekar et al., ICML 2026) and HarnessOpt-Bench (Ursekar et al., 2026) study how to evaluate harness optimization itself, including held-out scoring and how gains vary across tasks.

Questions for authors
  1. Could you report Tables 1–2 and Figure 4a,b separately on the 60 unseen instances? This would really address my main concern about generalization.

  2. How many independent searches and repeated evaluations of each selected harness contribute to the results? What do the ± values and plotted error bars represent? Please report variation across searches separately from re-evaluation noise.

  3. Does the error-conditioning ablation remove category-specific traces as well as the category label? Could you report the ablation on travel and clarify the number of independent searches behind Table 4?

  4. How many tasks is each child evaluated on before selection, and what are the editor costs per method (and summarization costs, if Summarize is an LLM call)? Please also confirm the editor models used and whether HGM-CF and HGM-Dual receive otherwise matched inputs and instructions.

  5. How are κ and ρ_B calculated, how were their thresholds chosen, and did every experimental setting pass the pre-search stability checks?

2 comments

We thank the reviewer for the careful review and constructive feedback.

Could you report Tables 1–2 and Figure 4a,b separately on the 60 unseen instances? This would really address my main concern about generalization.

Response: We thank the reviewer for this suggestion. Below we report Tables 1 and Fig. 4(a,b) recomputed on the 60 unseen instances only.

Table: Performance on the 60 unseen instances (budget 4000)

MethodTravel compositeTravel case acc. (%)Shopping compositeShopping case acc. (%)
Seed Agent78.49 ± 3.0226.6783.74 ± 0.0250.83
HGM-CA†82.81 ± 3.9852.5086.07 ± 0.2256.67
HGM-CF82.71 ± 1.3352.5087.42 ± 1.8561.67
HGM-Dual88.05 ± 1.3665.4290.31 ± 0.5867.50

Performance for different budget scenarios on the 60 unseen instances (Fig 4(a,b))

TaskMethodSeed1k2k3k4k
TravelHGM-CA78.4979.95 ± 1.5576.09 ± 5.0879.79 ± 3.8382.81 ± 3.98
TravelHGM-CF78.4985.16 ± 1.8485.16 ± 1.8485.16 ± 1.8482.71 ± 1.33
TravelHGM-Dual78.4984.32 ± 2.2186.39 ± 1.3486.80 ± 2.1888.05 ± 1.36
---------------------
ShoppingHGM-CA83.7487.25 ± 0.1285.35 ± 1.4386.07 ± 0.2286.07 ± 0.22
ShoppingHGM-CF83.7484.89 ± 4.2888.51 ± 0.0286.51 ± 1.0887.42 ± 1.85
ShoppingHGM-Dual83.7490.18 ± 2.8890.24 ± 2.2289.52 ± 2.7990.31 ± 0.58

HGM-Dual's improvements carry over to unseen instances. It is clearly better than the seed in both tasks at every budget, and it keeps most of its gain: 9.6 of 13.1 points in travel and 6.6 of 8.1 in shopping. The baseline methods transfer less to the unseen instances. HGM-CF's performance drops from 85.5 on the optimization instances to 82.7 on the unseen ones at 4k evaluation budget. HGM-Dual also maintains similar performance trend at different budget scenarios for unseen samples.

Previous Recursive Self-improvement works, DGM (Zhang et al., 2025) and HGM (Wang et al., 2025) use the same set of samples for optimization and evaluation. We have followed the same protocol and reported the performance of HGM-Dual on all samples. We will add the performance of unseen cases in the revised version.

How many independent searches and repeated evaluations of each selected harness contribute to the results? What do the ± values and plotted error bars represent? Please report variation across searches separately from re-evaluation noise.

Response: For HGM-Dual, we ran two independent searches per task. The harness each search selects at a given budget was evaluated twice on all 120 instances, giving four evaluations per reported point. Previous Recursive Self-improvement works, DGM (Zhang et al., 2025) and HGM (Wang et al., 2025) report result through one search per task. We have increased it to two searches and multiple evaluations for better stability understanding. The ± values in Tables 1–2 and the error bars in Fig. 4(a,b) are the sample standard deviation over all evaluations behind an entry. For HGM-Dual it pools from two independent searches and two evaluations per search. The intervals in Fig. 4(c,d) are 95% paired bootstrap intervals over the 60 held-out instances. We resample instances with replacement (10,000 times, with the same resample for both methods) and recompute the difference in failure rate. They reflect uncertainty from which instances are in the benchmark, not variation across searches.

Separating the two independent searches, at the 4k budget:

  • Travel: the two HGM-Dual searches reach 88.28 ± 0.15 and 87.27 ± 2.03 composite score, against 84.11 for second best HGM-CF.

  • Shopping: HGM-Dual searches reach 88.24 ± 0.83 and 90.23 ± 0.82 composite score, against 86.67 for second best HGM-CF.

Each HGM-Dual search scores above HGM-CF on both tasks, by 3.2–4.2 points on travel and 1.6–3.6 points on shopping. The averaged HGM-Dual results reported in the paper (87.77 ± 1.31 and 89.23 ± 1.33) therefore do not depend on one favorable search. We will report per-search values and both sources of variation separately in the revision.

Does the error-conditioning ablation remove category-specific traces as well as the category label? Could you report the ablation on travel and clarify the number of independent searches behind Table 4?

Response: The error conditioning ablation changes only what the error-conditioned children see. There is no error-conditioned child agent generation. As a result, it loses:

  • its assigned category label,

  • the traces for that category (two failing instances, each with query, plan and failure).

Instead, each child is asked to come up with edits aimed at whatever most limits this agent’s score. The general edit, the shared batch, the number of children and best-of-pool selection are unchanged. The ablated children still receive the generic failure report similar to HGM-Dual and HGM-CF. The ablation therefore tests assigning each child its own category and traces, not access to error information.

On travel, the pre-search test disables error conditioning (κ = 0.27 < 0.4, ρ₁₆ = 0.36 < 0.7). The travel HGM-Dual is therefore HGM-CF plus evolution context, and it has no error conditioning to ablate.

How many tasks is each child evaluated on before selection, and what are the editor costs per method (and summarization costs, if Summarize is an LLM call)? Please also confirm the editor models used and whether HGM-CF and HGM-Dual receive otherwise matched inputs and instructions.

Response: Each child is evaluated on 16 optimization instances before any selection.

We follow the protocol established by previous Recursive Self-improvement works, DGM (Zhang et al., 2025) and HGM (Wang et al., 2025) to use number of task agent evaluation as the estimate of cost/budget for independent search process. This is because the task agent evaluation is the major source of cost. Editor and summarizer together use less than 0.5% of tokens in every arm for travel. While the total cost of one independent search of HGM-Dual for 4k budget evaluation is ~ 2500$, only editor + summarizer cost is around ~60$. Over HGM-CF, HGM-Dual adds only $34 per independent search on travel, which is negligible compared to the total task agent cost.

In every baseline, HGM-Dual and ablation search, the editor is GPT-5.4 (reasoning effort medium). The summarizer is GPT-5.4 (medium), and the task agent is GPT-5.4-mini (high). HGM-CF and HGM-Dual receives matched inputs and instruction. The only different is the two components: agent evolution as interpretable context, and error-conditioned child agent generation.

How are κ and ρ_B calculated, how were their thresholds chosen, and did every experimental setting pass the pre-search stability checks?

Response:

How κ and ρ_B are computed.

κ and ρ_B are computed once per task, from two independent evaluations of the seed harness on the 60 optimization instances.

κ is Cohen’s kappa between the two evaluations:

κ = (p_o − p_e) / (1 − p_e), p_e = f₁f₂ + (1 − f₁)(1 − f₂)

Here p_o is the fraction of instances with the same outcome in both evaluations, and p_e is the agreement expected from the failure rates f₁ and f₂ alone. f₁ and f₂ are failure rates from two independent runs. Κ measures task-level agreement.

ρ_B is the probability that a random batch of B instances has the same top error category in both evaluations:

ρ_B = Pr_b [ top(b, 1) = top(b, 2) ], b ⊂ {1,…,N}, |b| = B

top(b, r) is the highest-ranked non-generic error category on batch b in evaluation r, using HGM-Dual’s own categorizer and tie-break. ρ_B measures dominant-category stability. We use B = 16, the batch on which HGM-Dual ranks categories during an expansion and estimate ρ₁₆ from 5,000 random batches.

How the thresholds were chosen.

κ ≥ 0.4 is the conventional lower bound of moderate agreement (Landis & Koch, 1977). Below it, a re-evaluation substantially changes which instances fail.

An error-conditioned child is built around a single category, the top error category on the 16-instance batch. This helps only if that category is a reproducible weakness of the harness. If re-evaluating the same harness on the same batch would rank a different category first, the child specializes on sampling noise, and its evaluation still consumes budget. ρ_B measures this directly. A threshold of 0.7 guarantees that at least 70% of error-conditioned children target it. We require a clear majority rather than 0.5 because each child costs a full batch evaluation.

Did every setting pass?

Travel HGM-Dual result uses the general path with evolution context and no error-conditioned children. Every shopping HGM-Dual result uses the dual path.

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.

Reviewer_jUP9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the challenge of efficiently optimizing an LLM agent’s framework for long-horizon planning tasks. Existing approaches, such as the Darwin–Gödel Machine (DGM) and the Huxley–Gödel Machine (HGM), rely on an evolutionary search process in which an agent iteratively modifies its own implementation. However, in the context of planning tasks, as opposed to pure coding, this approach can lead to the consumption of a significant evaluation budget, as the agent may perform unnecessary modifications or alter components unrelated to the actual causes of failure.

Hence the authors propose "HGM-Dual." Its first component, the "agent evolution context," structures information regarding modifications made in previous generations, the resulting performance shifts, and both resolved and persistent errors, providing this data to inform subsequent iterations. The second component, "error-conditioned dual-path generation," creates a "general child" aimed at broad improvements, alongside two "specialized children" designed to address the most frequent error types currently affecting the agent. Following a standardized evaluation procedure, the best-performing child is selected and added to an evolutionary archive.

Critical strengths

1. The paper identifies practical inefficiencies in existing evolutionary harness optimization. One strength of this paper is its concrete analysis of the limitations of applying HGM directly to planning tasks. In particular, Figure 1 shows that similar edits recur and that the dominant failure categories often remain unresolved after modification. I found this analysis to be a convincing motivation for the proposed method, rather than simply introducing another optimization approach.

2. The paper demonstrates meaningful performance improvements on planning tasks and includes comparisons with adapted HGM baselines. Another strength is that the evaluation goes beyond the original HGM and includes HGM-CA and HGM-CF, which are adapted to the planning setting. In particular, HGM-CF uses a similar editing interface to HGM-Dual, making the comparison more meaningful.

Critical weaknesses

1. Insufficient experimentation to isolate the core contributions of the proposed method

This paper identifies "lineage-aware evolution context" and "error-conditioned dual-path generation" as its primary contributions. However, the dual-path generation process involves creating one "general" child and two "specialized" children at each expansion step, subsequently selecting the best-performing candidate. It is therefore crucial to distinguish whether performance gains stem from the error-conditioned guidance itself or from the act of generating multiple candidates and selecting the best one.

2. Limited scope of evaluation and unclear basis for generalization

Although the core claim of HGM-Dual centers on efficient harness optimization for long-horizon planning, the experiments are confined to only two tasks—travel and shopping—from the DeepPlanning benchmark. While both tasks involve complex constraints, they rely heavily on the structure and stability of the error categories provided by the evaluator. In practice, "specialized editing" leverages predefined error categories and their frequencies, activating only when these signals are sufficiently stable. Consequently, it remains uncertain whether this method would be effective in planning environments where clear error classification is difficult or where only incomplete feedback is available.

Questions for authors

Q1. Contribution of Error-Conditioned Generation : Table 4 evaluates error conditioning only on shopping at a budget of 1,000. Could the authors provide similar results for travel planning across different budgets? This would help clarify whether the benefits of error conditioning are consistent across tasks.

Q2. Robustness to Error Categories : How sensitive is HGM-Dual to noisy or incomplete error categories? Could the authors examine its robustness to inaccurate error labels and the choice of stability thresholds (κ ≥ 0.4, ρB ≥ 0.7)?

Q3. Overall Optimization Cost : The reported efficiency focuses on evaluation budgets, but HGM-Dual also requires additional editing and summarization. Could the authors compare total token usage and API costs with HGM-CF?

Q4. Generalization to Unseen Instances : The evaluation includes 60 optimization instances and 60 unseen instances. Could the authors report performance separately for these two groups? This would help assess whether the improvements generalize beyond the optimization set.

2 comments

We thank the reviewer for the careful review and constructive feedback.

Are the gains due to error conditioning or simply best-of-three selection?

Response: We agree that the benefit of error-conditioned guidance should be distinguished from the benefit of generating multiple candidates and selecting the best. Table 4 directly addresses this concern through the “w/o error cond.” ablation. This variant retains the lineage-aware evolution context, the same number of children per expansion (three), the generation sequence, and the best-of-three selection procedure. It removes the assigned error categories and their corresponding category-specific traces from the two targeted children, which instead receive open-ended editing instructions while retaining generic failure feedback. On Shopping at the same 1,000-evaluation budget, HGM-Dual achieves 90.35 ± 2.27, compared with 87.68 ± 0.29 without error conditioning. The 2.67-point improvement therefore supports an additional benefit from error-conditioned guidance when candidate count and selection are held fixed. Both variants already benefit from generating and selecting among multiple candidates, so this difference cannot be explained by that mechanism alone.

Could the error-conditioning ablation be reported on Travel and at other budgets?

Response: κ measures chance-corrected pass/fail agreement across repeated evaluations; ρ₁₆ measures how often a 16-instance batch retains the same top error category. Both metrics are explained in detail in our response to Reviewer pztg, and we will expand their explanation in the revision. On Travel, the pre-search checks disable error conditioning (κ = 0.27 < 0.4; ρ₁₆ = 0.36 < 0.7). Travel HGM-Dual is therefore HGM-CF plus evolution context, with no error-conditioning component to ablate.

Specialized editing relies on stable, predefined error categories, leaving its effectiveness unclear when error classification is difficult or feedback is incomplete.

Response: A sufficiently informative feedback signal is a prerequisite for most self-improvement frameworks, including DGM/HGM: without it, search cannot reliably distinguish improvements from regressions. This is a shared requirement of feedback-driven improvement, not specific to our method. Crucially, HGM-Dual does not require reliable error classification to operate: when categories are unstable, it disables error-conditioned generation and retains lineage-aware evolution, as demonstrated on Travel. Instead, a core contribution of HGM-Dual is to add targeted edits guided by error categories when those categories are reliable. Error categorization is also not inherently restricted to the travel and shopping tasks, and its effectiveness in other domains is possible.

How robust is the method to noisy categories and the choice of stability thresholds?

Response: We do not expect moderate changes to the thresholds to substantially alter the algorithm's behavior: κ ≥ 0.4 follows the conventional lower bound of moderate agreement (Landis and Koch, 1977), while ρ₁₆ ≥ 0.7 requires a clear majority agreement on the dominant error category. Both serve as reliability checks for enabling specialized generation. At approximately $2,500 per search per task, repeated searches across threshold settings are prohibitively costly. We therefore leave a systematic evaluation of threshold sensitivity to future work.

Could total token usage and API costs be compared with HGM-CF?

Response: Task-agent evaluation dominates optimization cost. On Travel, editing and summarization use less than 0.5% of total tokens in every arm. A 4,000-evaluation HGM-Dual search costs approximately $2,500, including $60 (2.4%) for editing and summarization—only $34 more than HGM-CF. We will include this breakdown in the revision.

Could performance be reported separately on the optimization and unseen instances?

Response: Below are the results at 4,000 evaluations.

60 unseen instances

MethodTravel compositeTravel case accuracy (%)Shopping compositeShopping case accuracy (%)
Seed Agent78.49 ± 3.0226.6783.74 ± 0.0250.83
HGM-CA82.81 ± 3.9852.5086.07 ± 0.2256.67
HGM-CF82.71 ± 1.3352.5087.42 ± 1.8561.67
HGM-Dual88.05 ± 1.3665.4290.31 ± 0.5867.50

60 optimization instances

MethodTravel compositeTravel case accuracy (%)Shopping compositeShopping case accuracy (%)
Seed Agent74.43 ± 3.4623.3380.08 ± 2.3250.83
HGM-CA83.65 ± 1.3355.8384.57 ± 1.5960.00
HGM-CF85.52 ± 1.1860.0085.93 ± 2.1460.00
HGM-Dual87.50 ± 1.4163.3388.16 ± 2.6265.83

On unseen instances, HGM-Dual improves over HGM-CF by 5.34 score points and 12.92 percentage points in case accuracy on Travel, and 2.89 points and 5.83 percentage points on Shopping. The gains therefore extend beyond the optimization instances. We will add these separate results in the revision.

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.

Reviewer_xRrcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HGM-Dual, a diagnostic-guided framework for optimizing agent harnesses for long-horizon planning. The method extends the Huxley-Gödel Machine (HGM) with two main components.

First, HGM-Dual constructs a lineage-aware evolution context for each harness. This context records the functionality introduced by ancestral edits, changes in performance and error profiles, and observed behavioral consequences. The summarized context is provided to the editor to help it avoid ineffective or redundant modifications. Second, HGM-Dual introduces error-conditioned dual-path generation. For each selected parent, it generates one general child through open-ended editing and two specialized children targeting the parent’s two most frequent error categories. The best child is selected under the same evaluation protocol.

The method is evaluated on the Travel and Shopping tasks from the DeepPlanning benchmark. The task agent is based on GPT-5.4-mini and the editor uses GPT-5.4. HGM-Dual is compared against HGM, HGM-CA, HGM-CF, ADAS, and GEPA. At a 4000-evaluation budget, HGM-Dual achieves the best composite score and case accuracy on both tasks. The gains are particularly strong in the low-budget Shopping setting. Ablations suggest that both the evolution context and error-conditioned generation contribute to the improvement. The paper also reports a preliminary transfer experiment using Mistral-Medium-3.5-128B as the task-agent backbone.

Overall, the paper presents a clear and practically motivated modification to evolutionary harness optimization. However, the empirical scope, statistical validation, budget-matched baseline comparison, and reproducibility details remain insufficient for strong conclusions.

Critical strengths

Strength 1: Clear diagnosis of an important failure mode The paper identifies a concrete weakness of existing open-ended harness evolution: edits may be redundant, unrelated to the parent’s actual failures, or unable to resolve persistent constraint violations. The analysis in Figure 1 gives a useful empirical motivation for the proposed design. Strength 2: Interpretable and modular method Both lineage-aware context and dual-path generation are relatively simple and interpretable. The framework can associate an edit with its targeted error, behavioral consequence, and performance outcome, which is useful for debugging and analyzing agent optimization. Strength 3: Consistent gains on the reported benchmark HGM-Dual improves both composite score and case accuracy over the main HGM variants on Travel and Shopping. Its advantage is especially pronounced in the low-budget Shopping setting, suggesting that the proposed mechanism may allocate limited evaluation resources more effectively.

Critical weaknesses

Weakness 1: Limited empirical scope and insufficient statistical validation The main claims are supported by two tasks from one benchmark, a limited number of task-agent backbones, and relatively small evaluation sets. The paper does not clearly report the number of independent evolution runs, random seeds, confidence intervals, or statistical tests. Since evolutionary search can be highly stochastic, a small number of runs may not be sufficient to support strong claims about robustness and efficiency.

The authors should report multiple independent searches and provide statistical tests or confidence intervals for final performance and budget-performance curves. Weakness 2: Baseline comparison is not fully budget-matched ADAS and GEPA are evaluated only with 1000 evaluations, while the main HGM-Dual comparison uses 4000 evaluations. Although the authors explain that API cost limited the experiments, the absence of complete budget-matched curves makes it difficult to compare the methods fairly.

Moreover, HGM-Dual generates three children per expansion and retains only one. Although the rejected children are counted in the evaluation budget, the number of editor calls, prompt tokens, and total LLM inference cost should be reported separately. Weakness 3: Dependence on hand-designed diagnostics The error-conditioned path depends on manually designed error categories, thresholds, and deterministic rule-based classifiers. The current experiments do not establish whether the method is a general principle for harness optimization or a benchmark-specific system tailored to the error structure of DeepPlanning.

The paper should examine threshold sensitivity, noisy or incomplete error categorization, automatic error diagnosis, and behavior when the diagnostic categories are incorrect.

Questions for authors

How many independent evolution runs and random seeds were used for each method? What sources of randomness contribute to the means and standard deviations in Tables 1–5? Besides lineage context and error-conditioned generation, do HGM-Dual and HGM-CF differ in editor prompts, context length, number of candidate generations, or number of LLM calls? What model and prompt are used for Summarize? What is the maximum token length of the lineage context, and how is information loss controlled for long lineages? Were the thresholds κ ≥ 0.4 and ρB ≥ 0.7 fixed before evaluation? Were sensitivity analyses performed?

2 comments

We thank the reviewer for the careful review and constructive feedback.

How many independent searches and random seeds were used?

Response: HGM-Dual uses two independent searches per task, both with seed 42. Each selected harness is evaluated twice on all 120 instances (60 optimization and 60 unseen), giving four evaluations per reported point.

At 4,000 evaluations, the per-search results are:

TaskHGM-Dual search 1HGM-Dual search 2HGM-CF
Travel88.28 ± 0.1587.27 ± 2.0384.11
Shopping88.24 ± 0.8390.23 ± 0.8286.67

Each ± above reflects two re-evaluations of one selected harness. Both searches outperform HGM-CF on both tasks, so the averaged gains do not depend on one favorable search. More searches are costly: each costs approximately $2,500 per task.

Are the final performance gains statistically supported?

Response: We compared HGM-Dual with HGM-CF using per-instance scores averaged over evaluations, a two-sided paired permutation test (100,000 sign flips), and 95% bootstrap confidence intervals over the 120 instances:

TaskScore improvement95% CIp-value
Travel+3.66[+0.79, +6.59]0.014
Shopping+2.56[+0.19, +5.06]0.038

HGM-Dual significantly outperforms HGM-CF on both tasks in this paired analysis of all 120 instances.

What sources of randomness contribute to the reported standard deviations?

Response: The standard deviations in Tables 1–2 and Fig. 4(a,b) are computed over two independent searches and two evaluations per selected harness on all 120 instances (seen and unseen). Independent searches reflect editor, summarizer, and task-agent sampling during search, affecting which harness is selected. Repeated evaluations reflect task-agent sampling when the same selected harness is run again.

ADAS and GEPA use 1,000 evaluations, whereas HGM-Dual uses 4,000.

Response: We extended GEPA to 4,000 evaluations:

TaskGEPA 1kGEPA 4kHGM-Dual 1kHGM-Dual 4k
Travel81.6780.9483.83 ± 0.3787.77 ± 1.31
Shopping81.1882.4090.35 ± 2.2789.23 ± 1.33

Longer optimization does not consistently improve GEPA, and HGM-Dual outperforms it at both matched budgets on both tasks. ADAS was not rerun at 4,000 evaluations due to computational cost.

Please report editor, summarizer, and total inference costs, including rejected children.

Response: Task-agent evaluation dominates optimization cost. On Travel, editing and summarization use less than 0.5% of total tokens in every arm. A 4,000-evaluation HGM-Dual search costs approximately $2,500, including $60 (2.4%) for editing and summarization—only $34 more than HGM-CF. We will include this breakdown in the revision. Evaluations of rejected children are included in the budget.

Do HGM-Dual and HGM-CF otherwise differ in prompts, context length, candidate generation, or LLM calls?

Response: The editor and summarizer use GPT-5.4 (medium reasoning effort); the task agent uses GPT-5.4-mini (high). HGM-CF and HGM-Dual share the base editing instructions, allowed context length, and maximum LLM-call limit. Their differences are evolution context and error-conditioned generation, with candidate evaluations charged to the same budget.

What model and prompt are used for Summarize?

Response: Summarize uses GPT-5.4 (medium reasoning effort) after each evaluation batch of a non-root harness. Instruction prompt of the summarizer has been added in the revised version of the paper (Appendix 7, marked in blue).

What is the maximum token length of the lineage context?

Response: The lineage context has a budget of 15,000 tokens. Context from each node is instructed to be under 300 words.

Were κ ≥ 0.4 and ρ_B ≥ 0.7 fixed before evaluation?

Response: κ measures chance-corrected pass/fail agreement across repeated evaluations; ρ₁₆ measures how often a 16-instance batch retains the same top error category. Both metrics are explained in detail in our response to Reviewer pztg, and we will expand their explanation in the revision. The thresholds were fixed before evaluation. κ ≥ 0.4 follows the conventional lower bound of moderate agreement (Landis and Koch, 1977). ρ₁₆ ≥ 0.7 requires a clear majority of batches to retain the same dominant error category across re-evaluations, since each specialized child costs a full batch evaluation.

Were threshold sensitivity and robustness to noisy or incomplete categories evaluated?

Response: We do not expect moderate changes to the thresholds to substantially alter the algorithm's behavior: κ ≥ 0.4 follows the conventional lower bound of moderate agreement (Landis and Koch, 1977), while ρ₁₆ ≥ 0.7 requires a clear majority agreement on the dominant error category. Both serve as reliability checks for enabling specialized generation. However, this expectation requires empirical validation. At approximately $2,500 per search per task, repeated searches across threshold settings are prohibitively costly. We therefore leave a systematic evaluation of threshold sensitivity to future work.

Dependence on hand-designed diagnostics The error-conditioned path depends on manually designed error categories, thresholds, and deterministic rule-based classifiers.

Response: A sufficiently informative feedback signal is a prerequisite for feedback-driven self-improvement, including DGM/HGM. Our contribution is to organize this feedback into dominant error categories that guide the editor toward targeted edits, improving the efficiency of limited evaluation budgets. Our hand-designed diagnostics instantiate this principle for travel/shopping tasks, while the principles are applicable to other tasks, such as coding. Therefore, the categories may vary by domain, while the role in guiding targeted edits remains the same.

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading and constructive feedback. We have uploaded a revised paper with all changes marked in blue. Details are in the individual responses.

Strengths noted by reviewers

  • A clear, well-motivated diagnosis of why open-ended harness evolution is inefficient for planning (xRrc, jUP9).
  • A simple, interpretable, and modular method that fits into HGM without changing parent selection (xRrc, pztg).
  • Consistent gains in composite score and case accuracy over adapted HGM baselines on both tasks (xRrc, jUP9, pztg). The analyses and controlled ablation support the proposed mechanisms (pztg).

Common concerns and how we addressed them

  1. Generalization to unseen instances (jUP9, pztg). We added results on the 60 unseen instances. HGM-Dual remains best on every metric. It improves over HGM-CF by 5.34 composite points and 12.92 case-accuracy points on travel, and by 2.89 and 5.83 points on shopping.
  2. Independent searches and statistical support (xRrc, pztg). We have compared independent search results with baselines. Each HGM-Dual search individually outperforms HGM-CF. It proves that the averaged/pooled results in do not depend on a single favorable search.
  3. Total inference cost. We added a cost breakdown. Editing and summarization use less than 0.5% of total tokens. A 4,000-evaluation HGM-Dual search costs about 2500 USD, of which about 60 USD goes to editing and summarization. That is only 34 USD more than HGM-CF.
  4. Stability thresholds. We added a formal description of the pre-search stability test (κ\kappa, ρB\rho_B) and the rationale for its thresholds, which were fixed before any search.
  5. Budget-matched baselines and matched settings (xRrc, pztg).
    • Evaluated GEPA baseline for 4,000 evaluation budget. HGM-Dual outperforms GEPA at both 1k and 4k budgets.
    • HGM-CF and HGM-Dual share the same models, editing instructions, context length, and LLM-call limit. Only the two proposed components differ.

Additions to the main paper (in blue)

  • Pre-search stability test (Sec. 3.4)
  • Cost breakdown (Sec. 4)
  • Generalization to unseen instances (Sec. 4.1)
  • Consistency across independent searches (Sec. 4.1)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.