- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces EvoRule Harness, a plug-in component designed to equip existing Harness optimizers with a continuous, adaptive memory system. Rather than representing rule-based experience as a monolithic entity, this approach encodes each rule as a combination of applicability conditions and intervention actions, while tracking the feedback—the outcomes—resulting from their execution. The system maintains these rules through five types of atomic operations: addition, deletion, refinement, splitting, and merging. The system employs a dual-layer memory architecture consisting of volatile and stable memory. New rules are initially stored in volatile memory; they are promoted to stable memory only after undergoing execution-based validation, confidence score updates, and a promotion-gating check. During the inference phase, the system selects applicable rules through semantic retrieval and conflict arbitration, enabling the persistent reuse of historical experience. This work evaluated EvoRule Harness by integrating it into Meta-Harness, TTHE, and Continual Harness. Its effectiveness was validated through comparisons with a range of baselines, including prompting-based methods, memory-augmented approaches, and offline retrieval techniques. Experimental results demonstrate performance gains on benchmarks such as Tau-Bench, Terminal-Bench-2, and SWE-bench Verified. Furthermore, continuous adaptation experiments, ablation studies, and case analyses provide additional evidence for the system's efficacy.
Critical strengths
-
The research addresses a problem of practical significance. This paper investigates the challenges of accumulating experience and maintaining reliability during continuous harness optimization. Balancing the incorporation of new experience with the preservation of existing, effective knowledge is a valuable area of study within the field of harness optimization.
-
The methodology is systematic and comprehensive. The work establishes a complete memory management framework. Leveraging a dual-layer memory structure (Stable/Volatile), atomic rule editing, confidence-based updates, and a rule-promotion mechanism, the framework supports the continuous updating, verification, and rollback of rules.
-
Strong modularity and extensibility. EvoRule Harness can be integrated as an independent component into various existing harness optimizers. Performance improvements have been observed across Meta-Harness, TTHE, and Continual Harness, providing preliminary evidence that the component is reusable across different systems.
-
Extensive experimental analysis and case studies. The paper includes ablation studies, statistics on rule operations, continuous adaptation experiments, and multiple case analyses. The appendix also provides detailed prompt templates, helping readers understand the system’s operational logic and implementation details.
Critical weaknesses
-
Limited methodological novelty. The core mechanism draws heavily from existing memory management concepts—including conditional retrieval, structured rule editing, dual-layer memory, confidence updates, and rule promotion. While this combined design is practically useful, the specific technical contributions that distinguish it from existing agent-based memory frameworks remain unclear.
-
Lack of fair comparative experiments against memory baselines. Although Table 1 includes methods like MemoHarness and Reflexion, the study lacks controlled experiments—specifically, tests where alternative memory mechanisms are integrated into the same underlying "Harness" optimizer. Consequently, it is difficult to discern whether the observed performance gains stem from EvoRule’s unique design or simply from the general benefits of incorporating memory enhancement.
-
Absence of sensitivity analysis for key hyperparameters. Several core hyperparameters—such as the rule promotion threshold (0.8), the minimum number of validation rounds (3), and the influence gate threshold (0.6)—are fixed throughout the experiments. The paper does not conduct sensitivity tests for these parameters. Given that such parameters directly govern the stability and plasticity of the memory system, its overall robustness has not been sufficiently validated.
-
Insufficient description of memory initialization and online evaluation protocols. The paper fails to clarify whether the initial memory store begins empty, nor does it explicitly detail the logical flow connecting task execution, feedback collection, rule updates, and final scoring across different benchmarks. Such details significantly influence the performance of continuous adaptive methods and are critical for ensuring fair comparative evaluation.
-
Limited validation of lifelong adaptive capabilities. The current ten-round experiments primarily report aggregated Pass@1 metrics, lacking a direct analysis of performance retention on historical tasks. Furthermore, since rule promotion and conflict resolution are sensitive to task ordering, the paper provides little to no assessment of the system’s stability under varying task sequences or scenarios involving distribution shifts.
Questions for authors
-
Memory Baselines. Can the authors conduct controlled comparative experiments—using the same underlying Harness optimizer, consistent feedback signals, and identical computational budgets—against simple empirical memory mechanisms or existing memory-based methods? This would help isolate the incremental gains specifically attributable to the unique design of the EvoRule Harness.
-
Rule Validation and Hyperparameters. How were the promotion threshold (0.8), minimum validation rounds (3), and influence gate threshold (0.6) determined? Could the authors provide a sensitivity analysis for these parameters, and clarify the calculation and verification processes for rule confidence scores as well as the individual contribution of each patch?
-
Online Evaluation Protocol. Is the initial memory bank empty, or is it pre-loaded with rules? At what stage does the memory update occur within the various benchmarks?
-
Memory Generalization and Lifelong Adaptation. Could the authors include a dedicated experiment on memory generalization? This would involve freezing the memory after accumulating rules on source tasks, then transferring it directly to unseen tasks or environments—comparing this against settings with an empty memory or one that undergoes online updates during the target task. Furthermore, could the authors analyze how variations in task sequence affect rule reuse, the retention of performance on historical tasks, and the final experimental outcomes?
7 comments
response for Reviewer nv8b (2/7)
Memory augmentation helps: CLIN-, ExpeL-, and ACE-style substrates raise Pass@1 from 76.4 to 77.8-78.5. But memory alone does not explain the gain. Full EvoRule reaches 80.3 Pass@1 and 78.6 R10 cumulative, with regression down to 3.1% and edit precision up to 0.79, versus 6.4-8.6% regression and 0.60-0.66 precision for the three substrate variants. The ablations locate the gain: removing provenance raises regression to 5.2%; removing rollback to 7.4%; removing the precedent gate to 6.2%; and removing Split/Merge together lowers Pass@1 to 79.2 and raises regression to 5.7%. Each piece contributes, but the full combination gives the lowest regression and highest edit precision. This substantiates EvoRule’s contribution to lifelong harness adaptation: the baselines can store and update experience, but they cannot attribute, reverse, and gate a single patch without disturbing the others. We have included these results in the revision (Table R1 and Appendix A.4).
response for Reviewer nv8b (3/7)
W2: Lack of fair controlled comparisons with memory baselines. Although Table 1 includes MemoHarness and Reflexion, it does not integrate alternative memory mechanisms into the same “Harness” optimizer, so it remains unclear whether gains stem from EvoRule’s unique design or memory enhancement generally. Q1: Can authors compare under the same Harness optimizer, consistent feedback, and equal compute against simple empirical or existing memory methods to isolate incremental gains attributable specifically to EvoRule’s design?
A: We thank you for suggesting this controlled comparison. To directly assess whether EvoRule provides benefits beyond memory augmentation, we conducted additional experiments integrating alternative memory mechanisms into the same Meta-Harness optimizer.
We compare Reflexion-style episodic memory, MemoHarness-style experience retrieval, ExpeL-style insight editing, and ACE-style delta memory. All variants use the same frozen Claude Opus 4.6 backbone, feedback signal, data split, and matched token and LLM-call budgets, with results averaged over five seeds. The Meta-Harness-only baseline retains the optimizer’s standard search history and has no added memory module. The other baselines adapt the respective memory mechanisms to this common framework; they are not evaluations of the original systems in their entirety. Table R7 reports the results below.
Table R7. Controlled memory baselines on the same Meta-Harness optimizer (frozen Claude Opus 4.6; same split and budget; mean over 5 seeds).
| Memory mechanism inside Meta-Harness | Terminal-Bench-2 Pass@1 ↑ | R10 Cum. ↑ | Regression Rate ↓ | Edit Precision ↑ |
|---|---|---|---|---|
| Meta-Harness (no memory) | 76.4 | 76.4 | 8.6% | N/A |
| Reflexion-style episodic memory | 77.6 | 77.0 | 7.8% | 0.58 |
| MemoHarness-style experience retrieval | 78.2 | 77.4 | 6.5% | 0.64 |
| ExpeL-style insight editing | 78.1 | 77.3 | 6.7% | 0.63 |
| ACE-style delta memory | 78.5 | 77.5 | 6.4% | 0.66 |
| EvoRule (ours) | 80.3 | 78.6 | 3.1% | 0.79 |
Memory augmentation helps in general, but EvoRule adds a further 1.8 points over the strongest memory baseline and lowers the regression rate from 6.4% to 3.1%. This indicates that the gain is not merely from adding memory. It comes from the harness optimization design: volatile-to-stable promotion, evidence-gated validation, stable-prioritized conflict resolution, and traceable atomic edits. The ablation results in Table 5 point in the same direction: removing the stable tier, the volatile tier, or the promotion gate degrades performance by 1.4 to 3.7 points.
W3: No sensitivity analysis for key hyperparameters. Core settings such as the rule promotion threshold 0.8, minimum validation rounds 3, and influence gate threshold 0.6 are fixed across experiments, with no sensitivity tests. Since these parameters govern memory stability and plasticity, overall robustness is insufficiently validated. Q2: How were the promotion threshold 0.8, minimum validation rounds 3, and influence gate threshold 0.6 chosen? Can the authors provide sensitivity analysis for these parameters and clarify how rule confidence scores are calculated and verified, as well as each patch’s individual contribution?
A: Thanks for these comments. We agree that , , and were fixed without sensitivity analysis.were fixed without sensitivity analysis. We now report a sweep on Terminal-Bench-2-Stream. The thresholds were chosen on a held-out validation split before final test evaluation, not tuned on the test set. Table R8 shows that performance is stable in a broad region around the chosen values, and that stricter thresholds trade $ Pass@1 $ for lower regression and higher promotion precision.
Table R8. Hyperparameter sensitivity (Meta-Harness base; frozen Claude Opus 4.6; mean over 5 seeds).
| Setting | Pass@1 ↑ | Regression Rate ↓ | Promotion Precision ↑ | #Promoted |
|---|---|---|---|---|
| $ c=0.75 $ | 80.1 | 3.6% | 0.84 | 15 |
| $ c=0.80 $ (default) | 80.3 | 3.1% | 0.86 | 13 |
| $ c=0.85 $ | 80.0 | 3.0% | 0.88 | 11 |
| $ c=0.90 $ | 79.6 | 2.9% | 0.90 | 9 |
| $ \mathrm{age}=2 $ | 80.0 | 3.5% | 0.84 | 15 |
| $ \mathrm{age}=3 $ (default) | 80.3 | 3.1% | 0.86 | 13 |
| $ \mathrm{age}=4 $ | 80.1 | 3.0% | 0.87 | 11 |
| $ \mathrm{age}=5 $ | 79.7 | 2.9% | 0.89 | 9 |
| $ \epsilon=0.4 $ | 79.8 | 4.0% | 0.83 | 16 |
| $ \epsilon=0.6 $ (default) | 80.3 | 3.1% | 0.86 | 13 |
| $ \epsilon=0.8 $ | 79.9 | 2.9% | 0.88 | 10 |
Rule confidence is computed as in Eq. (6), with the prior strength fixed at throughout the sensitivity analysis. These results show that EvoRule maintains consistent performance across a range of hyperparameter settings. Stricter thresholds reduce regression and improve promotion precision at the cost of slightly lower Pass@1, demonstrating a controllable tradeoff between stability and adaptability.
response for Reviewer nv8b (1/7)
For long-horizon reliability, existing methods alone are insufficient. CLIN evolves task-level causal abstractions without a patch unit, so it cannot attribute or reverse a single update. ExpeL edits an insight pool through aggregate votes without per-patch rollback, so it cannot undo one insight without disturbing others. ACE applies batch-level deltas without individual undo or a tier, so it cannot gate a single patch or protect stable memory. No existing work combines all these components in this way, and the combination is necessary rather than incidental: a patch-level unit with provenance and undo_ref makes an update attributable and reversible; the precedent gate operates on that unit; disjoint tiers separate admission from consolidation; one-way atomic promotion keeps volatile edits from reaching stable memory directly; and per-patch undo makes a bad promotion reversible without touching other rules.
Their benefits are synergistic, because each mechanism presupposes the others and the empirical gains depend on the whole protocol rather than on any single component. This also matches the NeurIPS 2026 reviewing guidelines (https://neurips.cc/Conferences/2026/ReviewerGuidelines), which recognize a novel combination of existing techniques as a valid form of originality.
For the specific differences from prior systems, we now put them in a separate paragraph. CLIN has no patch unit. ExpeL has no per-patch rollback. ACE has no individual undo or tier. EvoRule treats each edit as a patch with provenance and undo_ref, conditions REFINE/SPLIT on the applicability context, gates each patch before it enters volatile memory, and promotes it one-way and atomically into a disjoint stable tier. The difference is the ability to attribute, reverse, and gate a single update without touching others, which none of the three provides inside the same Meta-Harness loop.
The controlled comparisons and targeted ablations below ask what the protocol adds beyond the related approaches and which mechanisms produce that gain. All variants share the same Meta-Harness base, frozen Claude Opus 4.6 backbone, split, and budget, averaged over five seeds. The -style variants reuse only the memory-handling approach of each prior system inside our framework; they are not the original systems run unchanged. Holding base, backbone, split, and budget fixed isolates the effect of the memory substrate and operation set, and the results show that the full patch lifecycle is what delivers long-horizon reliability.
Table R8. Memory substrate and atomic-operation ablations (Meta-Harness base; frozen Claude Opus 4.6; same split and budget; mean over 5 seeds).
undo_refremoved)