- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ConSeqHarness improves the software that controls how a language-model agent uses tools, passes information, and produces answers. It records successful changes, including what failed, what was changed, and which checks passed. It then uses relevant records to guide later edits. A change is kept only after checks for valid structure, successful execution, improvement, and regressions. The paper reports the best results in all 15 main model–benchmark comparisons under an 18-attempt budget, with average gains of 2.73–3.43 percentage points over Meta-Harness. It also compares different memory formats and examines whether earlier improvements persist. Its formal guarantees concern the checked training examples.
Critical strengths
The memory comparison addresses the main idea directly. Flat Accepted History keeps the same kinds of accepted evidence and the same editing and checking machinery, but stores the records as ordinary text. Structured records score 1.67–1.94 points higher (Table 4). This is useful evidence about how to represent experience, beyond showing that having any memory helps.
The reported improvements cover several models and tasks. The main results are consistently positive, and the average advantage remains positive across three data samples for one model. Table 3 reports longest-path LLM-node counts and output-token caps. That supports improving the deployed workflow, although it does not show that actual token use, runtime, or training cost stays the same.
Critical weaknesses
Some results and evaluation settings need clarification. Figure 2b marks the GPT5.4-Pro OfficeQA run as stopping at attempt 16, but Figure 3b reports 18 accepted changes. Those cannot describe the same run: App. C.3 defines accepted candidates as a subset of charged attempts (A ⊆ C), so a run that stops at attempt 16 has at most 16 accepted changes, yet Fig. 3b's legend reads 'Retained at t=18'. The text also says LiveMath solves its training cases early, while Figure 2 stops near 48–50% accuracy.
The paper does not give the actual train/test set sizes or repeated-search uncertainty for the main memory comparison. The ALFWorld example is also unclear: Figure 5b gives the agent a formal goal and an initial state, then produces a command list. The authors need to explain how this matches the described task of acting from partial observations. These details matter for interpreting the reported gains.
The Random Retrieval experiment appears to change record order, not relevance. Appendix B.3 says the editor receives all compatible records. Appendix C.3 says this experiment shuffles those same records. Its accuracy drop therefore does not establish that irrelevant memory hurts performance, as Section 5.7 claims. It may instead show sensitivity to ordering. This distinction matters because selecting relevant past edits is a central part of the method.
Questions for authors
-
Do Figures 2b and 3b describe the same OfficeQA run? How do 18 accepted changes fit with stopping at attempt 16? Why does LiveMath stop near 48–50% when the text says its training cases are solved?
-
What are the train/test sizes? For ALFWorld, what state information can the agent see, and how is its command list executed and scored?
-
Does Random Retrieval change the records supplied to the editor, or only their order? If records are sampled or truncated, please explain how.
-
Do repeated searches on the same data preserve the gains over Meta-Harness and Flat Accepted History? Please include uncertainty and the actual editing and evaluation costs.
7 comments
Response to Reviewer BcVW (Part 2/3)
W3. The Random Retrieval experiment appears to change record order, and Appendices B.3 and C.3 supply and shuffle the same compatible records. Its accuracy drop may show sensitivity to ordering, so it does not establish Section 5.7’s claim that irrelevant memory hurts performance.
A3. Random Retrieval uniformly shuffles records within the compatibility-filtered pool while keeping all other retrieval and editing conditions unchanged. This control examines how the ordering of compatible history influences subsequent editing decisions.
Under the same search budget and validation protocol, the five-benchmark macro-average decreases by 6.68 points for GPT-5.4-Pro and 6.16 points for Qwen3.6-Max relative to Full. These results indicate that the ordering of accepted evidence affects how the editor reuses historical repairs.
We revised Section 5.7 and the note for Table 4 to clarify that this control permutes compatible records. Further details on record selection and context truncation are provided in Q3.
Q1. Do Figures 2b and 3b describe the same OfficeQA run? How do 18 accepted changes fit with stopping at attempt 16? Why does LiveMath stop near 48–50% when the text says its training cases are solved?
A4. Yes. Both figures correspond to the same GPT-5.4-Pro run on OfficeQA, where the test score plateaus at attempt 16 and the search exhausts its budget at attempt 18, with all 18 candidates accepted. In contrast, LiveMath stops after attempt 4 because all training examples are solved. The reported test scores of 50.00% and 47.61% are obtained from the respective frozen harnesses. W1 provides the detailed audit supporting these observations.
Q2. What are the train/test sizes? For ALFWorld, what state information can the agent see, and how is its command list executed and scored?
A5. The main-experiment sizes and metrics are listed below. All methods share the data manifest and evaluator for each benchmark.
Table 2. Main-experiment training/test sizes
| Benchmark | Training size | Test size | Reported metric |
|---|---|---|---|
| OfficeQA | 50 | 196 | Equal-weight aggregation over difficulty groups and numerical tolerances |
| TableEval | 200 | 775 | SEAT F1 |
| SearchQA | 400 | 200 | Mean answer-token F1 |
| ALFWorld | 64 | 3,489 | Mean goal-condition completion |
| LiveMath | 35 | 59 | Equal-weight accuracy over four monthly groups |
OfficeQA averages scores across difficulty groups and four numerical tolerances, while LiveMath averages scores across four monthly groups. Both use their respective frozen outputs, with all evaluation settings fixed in advance.
For ALFWorld, all methods receive the same task specification, including the goal PDDL, initial state, and action grammar. Each method generates a complete action sequence, which is then replayed in the same environment. We measure the maximum goal-condition completion along each trajectory and average it over 3,489 held-out tasks. The remaining 64 tasks are used for optimization. Further details are provided in Section 5.1 and Appendices C.1 and C.4.
Q3. Does Random Retrieval change the records supplied to the editor, or only their order? If records are sampled or truncated, please explain how.
A6. It changes only the order. Full and Random Retrieval first apply compatibility filtering to the accepted memory. Random Retrieval then uniformly permutes all records that pass the filter, preserving the record set and count.
We also verified that all compatible records fit within the shared context budget and reach the editor without truncation. The editor token cap limits only output length. Thus, the observed differences reflect record ordering rather than changes in the supplied evidence.
Response to Reviewer BcVW (Part 3/3)
Q4. Do repeated searches on the same data preserve the gains over Meta-Harness and Flat Accepted History? Please include uncertainty and the actual editing and evaluation costs.
A7. To assess search variability, we conducted three independent full searches for Meta-Harness, Full, and Flat on the same primary data split. All searches start from empty memory under identical experimental settings and an 18-attempt budget. Each final harness is evaluated once after being frozen.
Table 3. Repeated searches for Meta-Harness, Full, and Flat on the same fixed split
| Target model | Meta-Harness, mean ± SD | Full, mean ± SD | Flat, mean ± SD | Mean Full–Meta gain | 95% paired interval | Mean Full–Flat gain | 95% paired interval |
|---|---|---|---|---|---|---|---|
| GPT5.4-Pro | 76.72 ± 0.13 | 80.14 ± 0.29 | 78.19 ± 0.27 | +3.42 | [2.47, 4.37] | +1.94 | [0.56, 3.32] |
| Qwen3.6-Max | 77.32 ± 0.24 | 80.44 ± 0.50 | 78.91 ± 0.42 | +3.12 | [2.51, 3.73] | +1.53 | [−0.75, 3.81] |
For each repetition, we compute the equal-weight macro-average across five benchmarks and pair results by repetition index. Across three repetitions, Full consistently outperforms both Meta-Harness and Flat on GPT-5.4-Pro and Qwen3.6-Max, confirming the relative ordering of the main experiment despite search variability.
Paper Table 2 further examines data-sampling variability, showing macro-average gains of +2.73, +2.02, and +2.14 points across three Seed2.0-Pro manifests. Together, these results support the consistent advantage of ConSeqHarness across repeated searches and data samples. We also controlled for validation cost by matching target-model requests between Meta-Harness and ConSeqHarness on GPT-5.4-Pro. Under identical experimental settings and an 18-attempt budget, both methods used exactly 12,518 target-model requests.
Table 4. Target-model-request-matched comparison
| Method | OfficeQA | TableEval | SearchQA | ALFWorld | LiveMath | Five-benchmark macro-average | Target-model requests |
|---|---|---|---|---|---|---|---|
| Meta-Harness | 68.42 | 91.76 | 86.78 | 92.44 | 43.84 | 76.65 | 12,518 |
| ConSeqHarness | 73.04 | 95.66 | 89.28 | 94.74 | 47.57 | 80.06 | 12,518 |
| Difference | +4.62 | +3.90 | +2.50 | +2.30 | +3.73 | +3.41 | 0 |
At the same target-model request budget, ConSeqHarness has a 3.41-point higher five-benchmark macro-average than Meta-Harness. The fixed-split repeated searches and request-matched control give the same relative ordering.
Table 5. Representative optimization cost under GPT5.4-Pro
For a fair comparison, Table 5 reports the complete optimization cost of seven methods across five benchmarks using GPT-5.4-Pro as the target model. Editor requests are counted per call, with token usage reported in units of 10,000. Target-model requests include all evaluations performed throughout the search.
| Method | Editor requests | Editor input, 10K tokens | Editor output, 10K tokens | Target-model requests |
|---|---|---|---|---|
| ReAct | 0 | 0 | 0 | 0 |
| GEPA | 90 | 72 | 9.0 | 13,058 |
| Trace2Skill | 4,538 | 2,700 | 340 | 3,775 |
| SkillOpt | 517 | 390 | 50 | 13,712 |
| AFlow | 90 | 110 | 27 | 12,518 |
| Meta-Harness | 180 | 220 | 36 | 12,518 |
| ConSeqHarness | 76 | 61 | 15 | 18,455 |
Table 4 compares performance under equal target-model request budgets, while Table 5 reports the full offline optimization cost using GPT-5.4-Pro. Detailed accounting rules are provided in Section 5.7, the note to Table 4, and Appendix C.3.
These clarifications and additional experiments directly address the concerns about evaluation validity, comparison fairness, and computational cost. We respectfully ask you to reconsider the rating in light of the new evidence.
Response to Reviewer BcVW (Part 1/3)
We appreciate your review and your recognition of the Flat Accepted History control and deployment audit. We clarify the reported results and provide further evidence on evaluation reliability and optimization cost.
W1. Figure 2b marks GPT5.4-Pro/OfficeQA as stopping at attempt 16, but Figure 3b reports 18 accepted changes, although accepted candidates are a subset of charged attempts. The text says LiveMath solves its training cases early, while Figure 2 stops near 48–50% accuracy.
A1. Figure 2 reports test-set scores from post-hoc evaluation of frozen checkpoints, so the test-score plateau and the training-side search stop should be read separately.
Figure 2(b) and Figure 3(b) refer to the same GPT5.4-Pro/OfficeQA run, which completes 18 attempts and accepts all 18 candidates. The test score first plateaus at attempt 16, while the final two attempts further improve training performance before budget exhaustion. Figure 3(b)’s “Retained at t=18” audit corresponds to this final checkpoint. We have updated Figure 2(b) to distinguish the plateau start from the actual stopping point.
Both LiveMath runs stop after the fourth candidate attempt because all training-side examples have been solved. The 50.00% Seed2.0-Pro and 47.61% GPT5.4-Pro scores in the figure are obtained by evaluating the final frozen harnesses on the test set. We have clarified this distinction in Section 5.5.1 and the Figure 2 caption, ensuring that the stopping conditions and reported results are consistent across Figures 2 and 3.
W2. The paper does not give actual train/test sizes or repeated-search uncertainty for the main memory comparison. The authors should explain how ALFWorld’s formal goal, initial state, and complete command list match the described task of acting from partial observations.
A2. The main-experiment training/test sizes are OfficeQA 50/196, TableEval 200/775, SearchQA 400/200, ALFWorld 64/3,489, and LiveMath 35/59. The small files included in the anonymous repository support code execution and evaluation checks, while the main experiments use the fixed complete evaluation manifests.
Fixed-split search variability
We completed three independent full searches for both Full and Flat across all benchmark configurations on GPT-5.4-Pro and Qwen3.6-Max. All searches follow the experimental setup in Table 4, start from empty memory, and allow up to 18 candidate attempts. Each final harness is evaluated once after being frozen.
For each repetition, we first compute the equal-weight macro-average over the five benchmarks and then compute the Full–Flat paired difference using fixed repetition indices. Full has the higher mean score on both target models, with mean gains of 1.94 and 1.53 points, in the same direction as the gain reported in Table 4 of the paper.
ALFWorld inputs and execution protocol
For ALFWorld, ReAct, Meta-Harness, and ConSeqHarness share the same planning-and-replay protocol. Each method generates a complete action sequence from the given task and initial state, which is then replayed unchanged in the environment under identical evaluation conditions.
For the primary experiment, 64 tasks from the official pool of 3,553 are used for optimization, with the remaining 3,489 held out. Each task is scored by the maximum fraction of satisfied goal conditions along its trajectory, and held-out scores are averaged equally. All methods follow the same ALFWorld evaluation protocol, ensuring a consistent comparison of planning performance. Further implementation details are provided in Section 5.1 and Appendices C.1 and C.4.