acceptodds
Under review as a conference paper at ICLR 2027

ConSeqHarness: Memory-Conditioned Sequential Optimization of Agent Harnesses

Abstract

As large language models (LLMs) increasingly rely on tools, routing, persistent state, and verification for long-horizon execution, agent performance depends not only on the model itself but also on the external harness that organizes its execution. Existing optimizers use prior code, scores, and traces to guide new proposals, but typically treat such experience as proposal context rather than reusable, validated optimization state. We introduce the verified harness transition as a reusable optimization primitive that binds a motivating failure to a typed and scoped edit, its measured gain, validation evidence, and non-target effects. This turns sequential harness optimization into a memory-augmented process in which previously validated changes can explicitly condition future decisions. We formalize this view as the Harness-Reflected Decision Process and instantiate it with Memory-***Con***ditioned ***Seq***uential Optimization of Agent ***Harness***es (ConSeqHarness), a constrained sequential optimizer that retrieves compatible verified transitions, proposes bounded structural edits, and atomically updates both the harness and transition memory only after structural, runtime, improvement, and regression checks. Across three target models and five public agent benchmarks, ConSeqHarness ranks first in all 15 settings and improves macro-average accuracy over the strongest baseline by 2.73–3.43 points under the same candidate-attempt budget. Controlled ablations further show that retaining accepted experience and representing it as structured transitions both contribute to the gains, while unverified memory and unconstrained editing substantially increase regressions and runtime failures. *The source code is available at https://anonymous.4open.science/r/ConSeqHarness-579E.*

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM Agents, Agent Harnesses

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_BcVWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ConSeqHarness improves the software that controls how a language-model agent uses tools, passes information, and produces answers. It records successful changes, including what failed, what was changed, and which checks passed. It then uses relevant records to guide later edits. A change is kept only after checks for valid structure, successful execution, improvement, and regressions. The paper reports the best results in all 15 main model–benchmark comparisons under an 18-attempt budget, with average gains of 2.73–3.43 percentage points over Meta-Harness. It also compares different memory formats and examines whether earlier improvements persist. Its formal guarantees concern the checked training examples.

Critical strengths

The memory comparison addresses the main idea directly. Flat Accepted History keeps the same kinds of accepted evidence and the same editing and checking machinery, but stores the records as ordinary text. Structured records score 1.67–1.94 points higher (Table 4). This is useful evidence about how to represent experience, beyond showing that having any memory helps.

The reported improvements cover several models and tasks. The main results are consistently positive, and the average advantage remains positive across three data samples for one model. Table 3 reports longest-path LLM-node counts and output-token caps. That supports improving the deployed workflow, although it does not show that actual token use, runtime, or training cost stays the same.

Critical weaknesses

Some results and evaluation settings need clarification. Figure 2b marks the GPT5.4-Pro OfficeQA run as stopping at attempt 16, but Figure 3b reports 18 accepted changes. Those cannot describe the same run: App. C.3 defines accepted candidates as a subset of charged attempts (A ⊆ C), so a run that stops at attempt 16 has at most 16 accepted changes, yet Fig. 3b's legend reads 'Retained at t=18'. The text also says LiveMath solves its training cases early, while Figure 2 stops near 48–50% accuracy.

The paper does not give the actual train/test set sizes or repeated-search uncertainty for the main memory comparison. The ALFWorld example is also unclear: Figure 5b gives the agent a formal goal and an initial state, then produces a command list. The authors need to explain how this matches the described task of acting from partial observations. These details matter for interpreting the reported gains.

The Random Retrieval experiment appears to change record order, not relevance. Appendix B.3 says the editor receives all compatible records. Appendix C.3 says this experiment shuffles those same records. Its accuracy drop therefore does not establish that irrelevant memory hurts performance, as Section 5.7 claims. It may instead show sensitivity to ordering. This distinction matters because selecting relevant past edits is a central part of the method.

Questions for authors
  1. Do Figures 2b and 3b describe the same OfficeQA run? How do 18 accepted changes fit with stopping at attempt 16? Why does LiveMath stop near 48–50% when the text says its training cases are solved?

  2. What are the train/test sizes? For ALFWorld, what state information can the agent see, and how is its command list executed and scored?

  3. Does Random Retrieval change the records supplied to the editor, or only their order? If records are sampled or truncated, please explain how.

  4. Do repeated searches on the same data preserve the gains over Meta-Harness and Flat Accepted History? Please include uncertainty and the actual editing and evaluation costs.

7 comments
AuthorsOpenReview ↗

Response to Reviewer BcVW (Part 1/3)

We appreciate your review and your recognition of the Flat Accepted History control and deployment audit. We clarify the reported results and provide further evidence on evaluation reliability and optimization cost.


W1. Figure 2b marks GPT5.4-Pro/OfficeQA as stopping at attempt 16, but Figure 3b reports 18 accepted changes, although accepted candidates are a subset of charged attempts. The text says LiveMath solves its training cases early, while Figure 2 stops near 48–50% accuracy.

A1. Figure 2 reports test-set scores from post-hoc evaluation of frozen checkpoints, so the test-score plateau and the training-side search stop should be read separately.

Figure 2(b) and Figure 3(b) refer to the same GPT5.4-Pro/OfficeQA run, which completes 18 attempts and accepts all 18 candidates. The test score first plateaus at attempt 16, while the final two attempts further improve training performance before budget exhaustion. Figure 3(b)’s “Retained at t=18t=18” audit corresponds to this final checkpoint. We have updated Figure 2(b) to distinguish the plateau start from the actual stopping point.

Both LiveMath runs stop after the fourth candidate attempt because all training-side examples have been solved. The 50.00% Seed2.0-Pro and 47.61% GPT5.4-Pro scores in the figure are obtained by evaluating the final frozen harnesses on the test set. We have clarified this distinction in Section 5.5.1 and the Figure 2 caption, ensuring that the stopping conditions and reported results are consistent across Figures 2 and 3.


W2. The paper does not give actual train/test sizes or repeated-search uncertainty for the main memory comparison. The authors should explain how ALFWorld’s formal goal, initial state, and complete command list match the described task of acting from partial observations.

A2. The main-experiment training/test sizes are OfficeQA 50/196, TableEval 200/775, SearchQA 400/200, ALFWorld 64/3,489, and LiveMath 35/59. The small files included in the anonymous repository support code execution and evaluation checks, while the main experiments use the fixed complete evaluation manifests.

Fixed-split search variability

We completed three independent full searches for both Full and Flat across all benchmark configurations on GPT-5.4-Pro and Qwen3.6-Max. All searches follow the experimental setup in Table 4, start from empty memory, and allow up to 18 candidate attempts. Each final harness is evaluated once after being frozen.

Table 1. Full–Flat independent-search statistics

Target modelFull, mean ± SDFlat, mean ± SDMean Full–Flat gain95% paired interval
GPT5.4-Pro80.14 ± 0.2978.19 ± 0.27+1.94[0.56, 3.32]
Qwen3.6-Max80.44 ± 0.5078.91 ± 0.42+1.53[−0.75, 3.81]

For each repetition, we first compute the equal-weight macro-average over the five benchmarks and then compute the Full–Flat paired difference using fixed repetition indices. Full has the higher mean score on both target models, with mean gains of 1.94 and 1.53 points, in the same direction as the gain reported in Table 4 of the paper.

ALFWorld inputs and execution protocol

For ALFWorld, ReAct, Meta-Harness, and ConSeqHarness share the same planning-and-replay protocol. Each method generates a complete action sequence from the given task and initial state, which is then replayed unchanged in the environment under identical evaluation conditions.

For the primary experiment, 64 tasks from the official pool of 3,553 are used for optimization, with the remaining 3,489 held out. Each task is scored by the maximum fraction of satisfied goal conditions along its trajectory, and held-out scores are averaged equally. All methods follow the same ALFWorld evaluation protocol, ensuring a consistent comparison of planning performance. Further implementation details are provided in Section 5.1 and Appendices C.1 and C.4.

AuthorsOpenReview ↗

Response to Reviewer BcVW (Part 2/3)

W3. The Random Retrieval experiment appears to change record order, and Appendices B.3 and C.3 supply and shuffle the same compatible records. Its accuracy drop may show sensitivity to ordering, so it does not establish Section 5.7’s claim that irrelevant memory hurts performance.

A3. Random Retrieval uniformly shuffles records within the compatibility-filtered pool while keeping all other retrieval and editing conditions unchanged. This control examines how the ordering of compatible history influences subsequent editing decisions.

Under the same search budget and validation protocol, the five-benchmark macro-average decreases by 6.68 points for GPT-5.4-Pro and 6.16 points for Qwen3.6-Max relative to Full. These results indicate that the ordering of accepted evidence affects how the editor reuses historical repairs.

We revised Section 5.7 and the note for Table 4 to clarify that this control permutes compatible records. Further details on record selection and context truncation are provided in Q3.


Q1. Do Figures 2b and 3b describe the same OfficeQA run? How do 18 accepted changes fit with stopping at attempt 16? Why does LiveMath stop near 48–50% when the text says its training cases are solved?

A4. Yes. Both figures correspond to the same GPT-5.4-Pro run on OfficeQA, where the test score plateaus at attempt 16 and the search exhausts its budget at attempt 18, with all 18 candidates accepted. In contrast, LiveMath stops after attempt 4 because all training examples are solved. The reported test scores of 50.00% and 47.61% are obtained from the respective frozen harnesses. W1 provides the detailed audit supporting these observations.


Q2. What are the train/test sizes? For ALFWorld, what state information can the agent see, and how is its command list executed and scored?

A5. The main-experiment sizes and metrics are listed below. All methods share the data manifest and evaluator for each benchmark.

Table 2. Main-experiment training/test sizes

BenchmarkTraining sizeTest sizeReported metric
OfficeQA50196Equal-weight aggregation over difficulty groups and numerical tolerances
TableEval200775SEAT F1
SearchQA400200Mean answer-token F1
ALFWorld643,489Mean goal-condition completion
LiveMath3559Equal-weight accuracy over four monthly groups

OfficeQA averages scores across difficulty groups and four numerical tolerances, while LiveMath averages scores across four monthly groups. Both use their respective frozen outputs, with all evaluation settings fixed in advance.

For ALFWorld, all methods receive the same task specification, including the goal PDDL, initial state, and action grammar. Each method generates a complete action sequence, which is then replayed in the same environment. We measure the maximum goal-condition completion along each trajectory and average it over 3,489 held-out tasks. The remaining 64 tasks are used for optimization. Further details are provided in Section 5.1 and Appendices C.1 and C.4.


Q3. Does Random Retrieval change the records supplied to the editor, or only their order? If records are sampled or truncated, please explain how.

A6. It changes only the order. Full and Random Retrieval first apply compatibility filtering to the accepted memory. Random Retrieval then uniformly permutes all records that pass the filter, preserving the record set and count.

We also verified that all compatible records fit within the shared context budget and reach the editor without truncation. The editor token cap limits only output length. Thus, the observed differences reflect record ordering rather than changes in the supplied evidence.

AuthorsOpenReview ↗

Response to Reviewer BcVW (Part 3/3)

Q4. Do repeated searches on the same data preserve the gains over Meta-Harness and Flat Accepted History? Please include uncertainty and the actual editing and evaluation costs.

A7. To assess search variability, we conducted three independent full searches for Meta-Harness, Full, and Flat on the same primary data split. All searches start from empty memory under identical experimental settings and an 18-attempt budget. Each final harness is evaluated once after being frozen.

Table 3. Repeated searches for Meta-Harness, Full, and Flat on the same fixed split

Target modelMeta-Harness, mean ± SDFull, mean ± SDFlat, mean ± SDMean Full–Meta gain95% paired intervalMean Full–Flat gain95% paired interval
GPT5.4-Pro76.72 ± 0.1380.14 ± 0.2978.19 ± 0.27+3.42[2.47, 4.37]+1.94[0.56, 3.32]
Qwen3.6-Max77.32 ± 0.2480.44 ± 0.5078.91 ± 0.42+3.12[2.51, 3.73]+1.53[−0.75, 3.81]

For each repetition, we compute the equal-weight macro-average across five benchmarks and pair results by repetition index. Across three repetitions, Full consistently outperforms both Meta-Harness and Flat on GPT-5.4-Pro and Qwen3.6-Max, confirming the relative ordering of the main experiment despite search variability.

Paper Table 2 further examines data-sampling variability, showing macro-average gains of +2.73, +2.02, and +2.14 points across three Seed2.0-Pro manifests. Together, these results support the consistent advantage of ConSeqHarness across repeated searches and data samples. We also controlled for validation cost by matching target-model requests between Meta-Harness and ConSeqHarness on GPT-5.4-Pro. Under identical experimental settings and an 18-attempt budget, both methods used exactly 12,518 target-model requests.

Table 4. Target-model-request-matched comparison

MethodOfficeQATableEvalSearchQAALFWorldLiveMathFive-benchmark macro-averageTarget-model requests
Meta-Harness68.4291.7686.7892.4443.8476.6512,518
ConSeqHarness73.0495.6689.2894.7447.5780.0612,518
Difference+4.62+3.90+2.50+2.30+3.73+3.410

At the same target-model request budget, ConSeqHarness has a 3.41-point higher five-benchmark macro-average than Meta-Harness. The fixed-split repeated searches and request-matched control give the same relative ordering.

Table 5. Representative optimization cost under GPT5.4-Pro

For a fair comparison, Table 5 reports the complete optimization cost of seven methods across five benchmarks using GPT-5.4-Pro as the target model. Editor requests are counted per call, with token usage reported in units of 10,000. Target-model requests include all evaluations performed throughout the search.

MethodEditor requestsEditor input, 10K tokensEditor output, 10K tokensTarget-model requests
ReAct0000
GEPA90729.013,058
Trace2Skill4,5382,7003403,775
SkillOpt5173905013,712
AFlow901102712,518
Meta-Harness1802203612,518
ConSeqHarness76611518,455

Table 4 compares performance under equal target-model request budgets, while Table 5 reports the full offline optimization cost using GPT-5.4-Pro. Detailed accounting rules are provided in Section 5.7, the note to Table 4, and Appendix C.3.


These clarifications and additional experiments directly address the concerns about evaluation validity, comparison fairness, and computational cost. We respectfully ask you to reconsider the rating in light of the new evidence.

Reviewer_RZBQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed ConSeqHarness, a method for improving the executable harness around a frozen LLM. It records accepted edits together with their motivating failures, applicability conditions, measured gains and validation outcomes. Each round retrieves compatible records, proposes a bounded edit, and updates the harness and memory after structural, runtime, improvement and regression checks. Only the final harness runs at deployment. Across three models and five tasks, the method reports the best accuracy in all 15 primary settings, improving macro-average accuracy over Meta-Harness.

Critical strengths

The problem is relevant and decently motivated. Scoped edits, regression checks and accepted-transition memory form a sensible approach to accumulating the harness improvements (Section 4).

The evaluation covers multiple models and task types. Flat Accepted History is a good control as it retains accepted evidence and the optimization machinery while changing the memory format, giving a direct test of the representation contribution (Table 4).

Critical weaknesses

The training curves and transition counts need clarification. Figure 2 is labeled as training accuracy, but its initial and final values match Table 1’s corresponding ReAct and ConSeqHarness test accuracies across all ten curves. Figure 2(b) also places the GPT5.4-Pro/OfficeQA stop marker at attempt 16, while Figure 3(b) reports 18 accepted transitions. The authors should clarify which splits and runs produced these figures and reconcile the counts.

The uncertainty in the structured-memory gain is not reported. Table 4 shows improvements over Flat Accepted History on GPT5.4-Pro and Qwen3.6-Max, without uncertainty estimates.

The comparison with closely related work can be stronger. ConSeqHarness identifies Self-Harness as a related method using tested, failure-driven local modifications, but excludes it from the experimental baselines (Section 2; Section 5.2 and Table 1)

Questions for authors

Could you clarify why Figure 2’s training-curve endpoints match the test scores in Table 1? For GPT5.4-Pro on OfficeQA, could you also explain how the stop marker at attempt 16 in Figure 2(b) relates to the 18 accepted transitions in Figure 3(b)?

Could you provide results across independent optimization runs, with uncertainty estimates, for Full versus Flat Accepted History in Table 4?

Could you clarify why Self-Harness was not included as a baseline and whether a comparison under matched conditions is feasible?

Can you clarify which aspects of the structured transition records drive the improvement over Flat Accepted History in Table 4? In particular, what evidence distinguishes the contributions of the applicability fields, typed edit contracts, and recorded validation effects?

8 comments
AuthorsOpenReview ↗

Response to Reviewer RZBQ (Part 1/3)

Thank you for highlighting the ambiguities in Figure 2 and requesting stronger comparative evidence. We clarify the figure and provide additional experiments to address these concerns.


W1. Figure 2’s training-curve endpoints match Table 1’s test accuracies, while its GPT5.4-Pro/OfficeQA stop marker at attempt 16 contrasts with Figure 3(b)’s 18 accepted transitions. The authors should clarify which splits and runs produced these figures and reconcile the counts.

A1. We clarify the evaluation source of Figure 2 and the meaning of attempts 16 and 18 for OfficeQA.

Data splits and run correspondence in Figure 2

Figure 2(a) corresponds to Seed2.0-Pro and Figure 2(b) to GPT5.4-Pro. Both use the fixed primary-seed-40 manifests. Each panel contains one curve for OfficeQA, TableEval, SearchQA, ALFWorld, and LiveMath. Thus, the ten curves come from the corresponding model–benchmark runs and their frozen checkpoints. Figure 2 reports post-hoc evaluation on the fixed test split after the search is complete. We evaluate only after freezing the final harness and search decisions, ensuring that test information does not influence any stage of optimization.

The first and last points of all ten curves reuse the same per-example predictions and correctness records as Table 1 for the corresponding model, benchmark, and run, which explains the exact agreement. In the figure setup, W0W_0 represents the fixed ReAct execution behavior as an editable graph, and the initial evaluation uses the same records. Section 4.1 specifies this correspondence between W0W_0 and fixed ReAct execution.

We relabeled “training ACC” as “post-hoc test score” in Figure 2 and Section 5.5.1, and clarified the endpoint provenance and evaluation timing in Appendix D.

OfficeQA attempt and acceptance counts

Figure 2(b) and Figure 3(b) use the same GPT5.4-Pro/OfficeQA optimization run. The run completes 18 charged attempts, and all 18 candidates pass the complete acceptance gate, so tstop=n=18t_{\mathrm{stop}}=n=18. In Figure 2(b), t=16t=16 is the first point at which the final test score reaches its plateau. Attempts 17 and 18 continue to improve the training-side score while the test score remains on that plateau.

The structural-retention and persistent-effectiveness curves in Figure 3(b) use these 18 accepted transitions as their statistical population. To distinguish the test-score plateau from budget exhaustion, we revised the marker at t=16t=16 to indicate the plateau start, draw a solid line from t=16t=16 to t=18t=18, and mark t=18t=18 as budget exhaustion. We also clarified in Figure 3 and Appendix C.3 that both panels use the same run and satisfy n≤t≤18n\leq t\leq18. Thus, the Figure 2 endpoints agree because they reuse the same frozen outputs, while OfficeQA attempts 16 and 18 denote the test-score plateau start and budget exhaustion, respectively, rather than conflicting stop and acceptance counts.


W2. The uncertainty in the structured-memory gain is not reported. Table 4 shows improvements over Flat Accepted History on GPT5.4-Pro and Qwen3.6-Max, without uncertainty estimates.

A2. To answer the uncertainty concern directly, we ran three independent full searches for Full and Flat for every benchmark configuration on GPT5.4-Pro and Qwen3.6-Max.

The additional searches follow the experimental setup in Table 4, with a maximum of 18 candidate attempts per run. Each search starts from empty memory and evaluates the test set after the final harness is frozen. We first compute the equal-weight macro-average over the five benchmarks for each repetition, then compute paired Full–Flat differences using fixed repetition indices.

Table 1. Independent searches on the fixed data split

Target modelFull, mean ± SDFlat, mean ± SDMean Full–Flat gain95% paired tt interval
GPT5.4-Pro80.14 ± 0.2978.19 ± 0.27+1.94[0.56, 3.32]
Qwen3.6-Max80.44 ± 0.5078.91 ± 0.42+1.53[−0.75, 3.81]

The table reports the mean, sample standard deviation, and the 95% paired tt interval for the mean paired gain, with two degrees of freedom. Full has the higher mean score on both target models, with gains of 1.94 and 1.53 points, respectively, in the same direction as the gain reported in Table 4 of the paper. These results provide the requested estimate of full-search variability.

AuthorsOpenReview ↗

Response to Reviewer RZBQ (Part 2/3)

W3. The comparison with closely related work can be stronger. ConSeqHarness identifies Self-Harness as a related method using tested, failure-driven local modifications, but excludes it from the experimental baselines (Section 2, Section 5.2 and Table 1)

A3. We added a five-benchmark comparison with Self-Harness using the authors’ public code to assess the benefit of structured transition memory. Since Self-Harness was developed concurrently with our main experiments, it was discussed as related work in Section 2, while our original harness-level baselines were AFlow and Meta-Harness.

For a fair comparison, both methods use GPT-5.4-Pro and GPT-5.5 as the target and editor, respectively, under identical experimental conditions and an 18-attempt budget. Since Self-Harness originally uses the target model as the proposer, we label this a unified-editor adaptation. The adaptation preserves Self-Harness's original validation and merging rules. All proposals and repairs count toward the 18-attempt budget, and testing occurs only after the harness is frozen.

Table 2. Self-Harness comparison with a unified editor

MethodOfficeQATableEvalSearchQAALFWorldLiveMathFive-benchmark macro-average
Self-Harness (unified-editor adaptation)71.6794.4488.2495.3845.2478.99
ConSeqHarness73.3395.5589.4294.8647.6180.15

ConSeqHarness has a 1.16-point higher five-benchmark macro-average and scores higher on four of the five benchmarks. Together with the Full–Flat results in Table 4, this provides further evidence for the benefit of structured transition memory.


Q1. Could you clarify why Figure 2’s training-curve endpoints match the test scores in Table 1? For GPT5.4-Pro on OfficeQA, could you also explain how the stop marker at attempt 16 in Figure 2(b) relates to the 18 accepted transitions in Figure 3(b)?

A4. Figure 2 reports post-hoc test scores for frozen checkpoints, so its endpoints reuse the same predictions and correctness records as Table 1. For OfficeQA, attempt 16 is the start of the final test-score plateau, while attempt 18 is budget exhaustion and the run accepts 18 candidates. The evaluation source, timing, and figure revisions are detailed in W1 above.


Q2. Could you provide results across independent optimization runs, with uncertainty estimates, for Full versus Flat Accepted History in Table 4?

A5. We completed three independent full searches for each Full and Flat configuration on the fixed data split. Table 1 in W2 reports gains of 1.94 and 1.53 points, alongside their uncertainty estimates. Each full-search repetition is the statistical unit.


Q3. Could you clarify why Self-Harness was not included as a baseline and whether a comparison under matched conditions is feasible?

A6. Self-Harness was developed concurrently with our algorithm and primary experiments, so the submission discussed it in related work and used AFlow and Meta-Harness as the experimental baselines. We implemented the matched comparison through a benchmark adapter and completed the five-benchmark comparison under a shared target model, editor, and candidate budget. ConSeqHarness has a 1.16-point higher macro-average. The adaptation details and full results are given in W3 above.

AuthorsOpenReview ↗

Response to Reviewer RZBQ (Part 3/3)

Q4. Can you clarify which aspects of the structured transition records drive the improvement over Flat Accepted History in Table 4? In particular, what evidence distinguishes the contributions of the applicability fields, typed edit contracts, and recorded validation effects?

A7. The three field groups have distinct roles. Applicability fields specify when a historical repair can be reused. Applicability fields determine when repairs can be reused, typed edit contracts constrain how edits are applied, and validation effects guide retrieval based on observed outcomes. The definitions are given in section 4.3 and Appendix B.3.

To compare their roles in the editor input, we added three single-field natural-language controls. Each variant replaces one field group with semantically equivalent natural language while retaining typed structure for the other two groups. All variants share the same underlying data and optimization procedure as the full method.

Using GPT5.4-Pro as the target and GPT-5.5 as the editor, we evaluate all five benchmarks under the protocol of Table 4. Each configuration has three independent full searches, with Full reusing those in W2. The table reports the equal-weight five-benchmark macro-average for each repetition.

Table 3. Single-field natural-language controls

ConfigurationFive-benchmark macro-average, mean ± SD
Full80.14 ± 0.29
Natural-language applicability fields79.81 ± 0.38
Natural-language edit contract79.10 ± 0.25
Natural-language validation effects79.31 ± 0.57

Replacing the three field groups with natural language lowers the macro-average by approximately 0.33, 1.04, and 0.83 points, respectively. The edit-contract variant shows the largest mean decrease. These results suggest that typed representations benefit editor decisions, with edit contracts showing the largest effect. The findings complement the Full–Flat comparison in W2.


Together, these additional analyses clarify the reported results, strengthen the evidence for structured transition memory, and address concerns about evaluation reliability. We hope these clarifications facilitate your assessment and respectfully ask you to reconsider your rating.

Comments on the paper

9 comments
AuthorsOpenReview ↗

Response to Reviewer XzQB (Part 1/4)

We appreciate your careful scrutiny of the reported results. We now provide explicit scoring formulas and per-benchmark counts, supported by unified extraction checks and independent searches. We also clarify the evaluation protocols for ALFWorld and Meta-Harness.


W1(a). Under Eq. 23’s single-pass binary accuracy, OfficeQA scores 73.33 and 68.50 cannot both be k/N with the same N unless N ≥ 600, larger than App. C.2’s one-third hold-out suggests. All 15 Meta-Harness entries for seed 40 are multiples of 0.1, whereas other methods and seeds 80 and 2027 have two-decimal values.

A1(a). The original manuscript used ACC as a common label and gave the basic per-example binary-accuracy form in Eq. 23. The reported scores also use benchmark-specific aggregation. For OfficeQA, 50 of 246 questions are used for optimization and 196 for held-out evaluation, with 108 hard and 88 easy questions. The same frozen outputs are scored at tolerances 0%, 0.1%, 1%, and 5%, averaging hard/easy groups equally at each tolerance and then averaging the four tolerances.

SOfficeQA=1004∑τ∈{0,0.001,0.01,0.05}ah,τ+ae,τ2.S_{\mathrm{OfficeQA}}=\frac{100}{4}\sum_{\tau\in\{0,0.001,0.01,0.05\}}\frac{a_{h,\tau}+a_{e,\tau}}{2}.

The per-cell counts are:

Table 1. GPT5.4-Pro/OfficeQA group counts

MethodHard correct counts, denominator 108Easy correct counts, denominator 88Aggregate score
ConSeqHarness33, 46, 83, 8562, 81, 86, 8673.33%
Meta-Harness16, 31, 67, 9064, 76, 88, 8868.50%

These counts yield 73.3323% and 68.4975%, displayed as 73.33% and 68.50%. Thus, a single shared k/Nk/N denominator is not the applicable reporting unit. TableEval uses SEAT F1, SearchQA mean answer-token F1, and ALFWorld goal-condition completion over 64 optimization and 3,489 held-out tasks. LiveMath uses 59 held-out questions grouped into four fixed monthly sets and averages within-group accuracy equally. For seed 40, Meta-Harness task scores are exported to one decimal place and displayed with a trailing zero, giving at most 0.05 percentage points of rounding error. The benchmark splits, aggregation formulas, and export rule are now stated in section 5.1, the table note, Appendices C.1, C.2, C.4.1, and D.


W1(b). In Table 4, the four weaker variants have almost the same reliability metrics on two target models, despite macro-averages over at most 18 attempts per benchmark. This contrasts with the Full rows’ Eff. of 62.49 vs 80.70 and requires the per-benchmark numerators and denominators.

A1(b). The apparent similarity arises from different counting units. Eff. and Accpt. measure candidate-level effectiveness and acceptance, respectively, while Reg. and Fail track regressions and runtime errors at the sample-execution level. Specifically, Eff. measures the fraction of accepted candidates achieving positive diagnostic gain without guard regression, whereas Accpt. measures the fraction of charged candidates passing the complete gate. Because validation may execute multiple samples per candidate, the denominators for Reg. and Fail can exceed 18.

The arrays below follow the order OfficeQA, TableEval, SearchQA, ALFWorld, and LiveMath. CC, AA, and EE denote charged candidate attempts, accepted candidates, and effective accepted candidates. G/RG/R denotes guard evaluations and regressions, while X/FX/F denotes sample executions and runtime-error events.

Table 2a. GPT5.4-Pro per-benchmark counts for paper Table 4

ConfigurationCCAAEEG/RG/RX/FX/F
GPT Full15,17,18,10,1715,16,8,7,42,11,7,3,44276/0, 8973/3, 7129/49, 7903/35, 13861/299474/89, 10043/87, 13714/142, 6695/79, 8881/126
GPT Flat10,12,12,17,137,1,4,13,133,1,4,1,77507/101, 9430/118, 11489/178, 6928/109, 9271/15512596/267, 7724/148, 4877/107, 10744/264, 9253/218
GPT GraphEdit w/o TM13,15,18,15,1811,6,1,7,92,1,1,6,715279/472, 5214/155, 6369/186, 7898/214, 9981/29315192/550, 11424/417, 4383/147, 13622/493, 9137/340
GPT Random Retrieval16,15,15,15,162,8,9,11,12,8,5,2,016562/644, 9639/355, 9874/385, 6595/244, 8077/30311218/527, 11570/512, 16916/731, 13045/566, 8430/375
GPT w/o Verified Write16,10,17,11,1110,3,8,2,32,1,5,2,17314/546, 8387/570, 13836/993, 14861/1054, 8328/56416373/1078, 7440/506, 13015/856, 14116/909, 13217/875
GPT Unconstrained Edit17,11,15,15,1611,5,3,1,48,2,2,0,217710/1893, 12875/1391, 14827/1604, 10485/1116, 7991/84011576/1315, 10587/1170, 11137/1213, 10441/1132, 8757/1004
AuthorsOpenReview ↗

Response to Reviewer XzQB (Part 2/4)

Table 2b. Qwen3.6-Max per-benchmark counts for paper Table 4

ConfigurationCCAAEEG/RG/RX/FX/F
Qwen Full12,11,13,12,128,8,11,3,108,6,9,2,87887/126, 4380/37, 15153/168, 8503/103, 12362/17913564/134, 6006/67, 16340/180, 12320/122, 10294/140
Qwen Flat11,17,18,15,108,6,15,14,18,3,3,13,117977/377, 13242/235, 8456/195, 5829/101, 14009/26517439/366, 15976/326, 4595/79, 15948/312, 5068/131
Qwen GraphEdit w/o TM15,12,15,16,123,9,5,1,112,1,2,1,95185/164, 10600/301, 4895/143, 6251/194, 14308/4056240/245, 5691/189, 15100/567, 11259/378, 15258/561
Qwen Random Retrieval10,12,12,12,123,4,4,8,51,4,1,3,417044/664, 14781/546, 11597/438, 14319/518, 12935/5206946/303, 12051/565, 14169/624, 14625/633, 11174/529
Qwen w/o Verified Write17,18,15,12,179,1,7,6,55,0,6,3,34338/307, 16494/1192, 14654/1050, 16321/1174, 7418/4955351/341, 7828/502, 16338/1113, 6283/425, 9049/628
Qwen Unconstrained Edit12,18,11,10,113,4,5,5,20,4,0,4,116042/1708, 7345/798, 6987/721, 17539/1872, 12438/13454194/498, 11766/1323, 14766/1628, 10471/1192, 4066/429

The four metrics are computed as 100E/A100E/A, 100A/C100A/C, 100R/G100R/G, and 100F/X100F/X, respectively, and macro-averaged equally across valid benchmarks. These counts explain why similar averages can arise from different denominators. The calculation details are provided in Section 5.7, the Table 4 note, and Appendix C.3.


W1(c). Seed2.0-Pro has 13 retained OfficeQA transitions in Fig. 3a but one retained L1 edit in Table 3, with the same mismatch for SearchQA (8 vs 2) and LiveMath (2 vs 1). GPT5.4-Pro/OfficeQA stops at t=16 in Fig. 2b yet has n=18 accepted transitions in Fig. 3b, although every acceptance uses a charged attempt.

A1(c). Figure 3a counts retained transitions, whereas Table 3 deduplicates L1 edits by their final target node. The 13 of 14 retained OfficeQA transitions, along with 8 for SearchQA and 2 for LiveMath, correspond to 1, 2, and 1 distinct L1 nodes, respectively. An earlier modification is retained only if it survives subsequent edits. Figures 2b and 3b refer to the same GPT-5.4-Pro/OfficeQA run, where the test score plateaus at attempt 16 and the budget is exhausted at attempt 18, with all 18 candidates accepted. We have clarified these counting differences in the figures and Appendix C.3.


W2. With one LLM node and “boxed output repair,” gains may come from normalizing outputs and fixing parse failures, so please split format/parse errors from reasoning errors and clarify whether ALFWorld baselines received the same fully observable state. The statistical evidence is limited to a single-seed typed-vs-flat comparison without intervals and data-sampling checks on one model, with a negative TableEval advantage at seed 80.

A2. We conducted three additional checks. First, we rescored the frozen Seed2.0-Pro/LiveMath outputs from all three methods using a unified extraction rule:

Table 3. LiveMath unified extraction and error breakdown

MethodOriginal ACCUnified extractionExecution error or missingNon-unique extractionParseable but wrong
ReAct29.5131.511.004.0063.49
Meta-Harness44.8045.800.501.5052.20
ConSeqHarness50.0050.000.500.5049.00

Under unified extraction, ConSeqHarness still outperforms ReAct and Meta-Harness by 18.49 and 4.20 points, respectively, with the same single-node structure and output-token cap. Second, all ALFWorld methods receive the same fully observable task state and action interface, with complete action sequences replayed and scored by the same GC evaluator on 3,489 held-out tasks. Third, independent Full–Flat searches on the fixed split yield:

Table 4. Full–Flat independent-search statistics

Target modelFull, mean ± SDFlat, mean ± SDMean paired gain95% paired interval
GPT5.4-Pro80.14 ± 0.2978.19 ± 0.27+1.94[0.56, 3.32]
Qwen3.6-Max80.44 ± 0.5078.91 ± 0.42+1.53[−0.75, 3.81]

The repeated searches and the three Seed2.0-Pro manifest results in paper Table 2 jointly support the macro-average gain under fixed and resampled splits.

AuthorsOpenReview ↗

Response to Reviewer XzQB (Part 3/4)

Q1. What is the test-set size for each benchmark? Is ACC ever averaged over more than one evaluation pass?

A3. The test sets contain 196 OfficeQA, 775 TableEval, 200 SearchQA, 3,489 ALFWorld, and 59 LiveMath tasks. Each final harness is evaluated once. OfficeQA and LiveMath apply their fixed aggregation rules to that pass, while repeated-search means average the results of independently frozen harnesses.


Q2. What is the total optimization cost of each method? This should include editor tokens, the target-model runs on the diagnostic, guard and smoke sets, and the full training-set rerun after each acceptance. Only proposals are counted against T=18.

A4. We report the full offline optimization cost under GPT-5.4-Pro, covering editor usage and target-model evaluations throughout the search, including validation and post-acceptance reruns:

Table 5. Editor usage | Method | Candidate attempts | Accepted updates | Editor requests | Input, 10K tokens | Output, 10K tokens | | --- | ---: | ---: | ---: | ---: | ---: | | ReAct | 0 | 0 | 0 | 0 | 0 | | GEPA | 90 | 15 | 90 | 72 | 9.0 | | Trace2Skill | 15 | 5 | 4,538 | 2,700 | 340 | | SkillOpt | 65 | 9 | 517 | 390 | 50 | | AFlow | 90 | 15 | 90 | 110 | 27 | | Meta-Harness | 90 | 15 | 180 | 220 | 36 | | ConSeqHarness | 76 | 34 | 76 | 61 | 15 |

Table 6. Target-model call breakdown

MethodInitial training evaluationSmokeDiagnosticGuardFull training reruns after acceptanceInternal evaluationTotal requests
ReAct0000000
GEPA7491805,2024,1402,24754013,058
Trace2Skill749309001,34774903,775
SkillOpt7491304,7693,7191,9492,39613,712
AFlow7491805,2024,1402,247012,518
Meta-Harness7491805,2024,1402,247012,518
ConSeqHarness7491524,7048,2884,562018,455

Together, the tables account for the requested optimization costs. ConSeqHarness uses 18,455 target-model requests, compared with 12,518 each for AFlow and Meta-Harness. Editor usage is reported separately.


Q3. Meta-Harness was designed for long searches with filesystem access to every earlier candidate. How did you adapt it to 18 attempts and the typed graph, and does the ranking hold with a larger budget?

A5. Meta-Harness retains its native executable-code search and filesystem access to all previous candidates, including rejected ones. Both methods start from the same smoke-checked W0W_0 and follow identical evaluation settings, with each parallel proposal or repair counting toward the 18-attempt budget. We further extend both methods to T=22T=22 under the same protocol:

Table 7. Five-benchmark macro-average under the extended budget

Target modelMeta-Harness, T=18T=18ConSeqHarness, T=18T=18Meta-Harness, T=22T=22ConSeqHarness, T=22T=22Gain at T=22T=22
GPT5.4-Pro76.7280.1576.9480.09+3.15
Seed2.0-Pro79.9882.7179.8582.82+2.97
Qwen3.6-Max77.3280.5377.4980.45+2.96

ConSeqHarness remains higher on all three target models, with gains of 3.15, 2.97, and 2.96 points, so the relative ranking persists under the extended budget.


Q4. Fig. 1 has a panel titled "Regression Control: Held-out Metrics (mean ± std)". This seems to contradict the statement that the test set is never used during optimization. Is the panel only illustrative?

A6. The Figure 1 panel is illustrative. Its schematic success rate, Answer F1, and latency are not test-set measurements. Operational gating uses training-side DguardtD_{\mathrm{guard}}^t, and DtestD_{\mathrm{test}} is evaluated only after the final harness and search decisions are frozen.


Q5. Props. 1–3 follow directly from the definitions and the acceptance rule. Would it be more accurate to call them properties of the protocol rather than theoretical results?

A7. We renamed Appendices A.2–A.4 to “Protocol-Induced Algorithmic Properties” while retaining the Proposition format. The three properties characterize state closure, the feasibility of verified edits, and conditional training-side progress under the protocol's assumptions. We have clarified these assumptions and the scope of each result in the revised appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.