acceptodds
Under review as a conference paper at ICLR 2027

EvoRule Harness: Lifelong Harness Adaptation with Validated Intervention Rules

Abstract

As general-purpose models are increasingly deployed with fixed weights, harness optimization is emerging as a key paradigm for adapting fixed-weight models to diverse tasks and deployment settings by managing context, tool use, and memory. However, deployment is inherently non-stationary, as tasks, tools, and interaction patterns change over time, while existing memory mechanisms are often rigid, training-dependent, and brittle, making reliable evolution over long horizons challenging. To address these limitations, we propose *EvoRule Harness*, a plug-and-play, continually adaptive memory system for long-horizon harness evolution. Given a frozen backbone model and an existing harness optimizer, EvoRule Harness builds a dynamic two-tier memory of causal rules. Each rule specifies an application condition and an intervention, and tracks execution-outcome feedback. Rules are revised online through individually traceable patches, allowing the effect of each update to be tracked and rolled back when necessary. Rule updates first enter a volatile tier and are promoted to stable memory only after repeated validation across rounds and data subsets, while unreliable updates are discarded. EvoRule Harness can be seamlessly integrated with diverse existing harness optimization methods, without requiring gradients or additional training. We evaluate EvoRule Harness on diverse tasks, including retrieval-augmented mathematical reasoning and interactive agentic tasks. EvoRule Harness improves Meta-Harness, TTHE, and Continual Harness by 7.0%, 8.5%, and 35.7% in average over Tau-Bench, Terminal-Bench-2, and SWE-Bench Verified, respectively. In particular, Meta-Harness augmented with EvoRule Harness achieves the best overall average of 67.4% among all harness optimization framework baselines, improving the base Meta-Harness from 63% by 7.0%. Our code is available at https://anonymous.4open.science/r/EvoMem-2C26/.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
EvoRule Harness, Harness Optimization, Continual Adaptive Memory, Long-Horizon Evolution, Non-Stationary Deployment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nv8bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EvoRule Harness, a plug-in component designed to equip existing Harness optimizers with a continuous, adaptive memory system. Rather than representing rule-based experience as a monolithic entity, this approach encodes each rule as a combination of applicability conditions and intervention actions, while tracking the feedback—the outcomes—resulting from their execution. The system maintains these rules through five types of atomic operations: addition, deletion, refinement, splitting, and merging. The system employs a dual-layer memory architecture consisting of volatile and stable memory. New rules are initially stored in volatile memory; they are promoted to stable memory only after undergoing execution-based validation, confidence score updates, and a promotion-gating check. During the inference phase, the system selects applicable rules through semantic retrieval and conflict arbitration, enabling the persistent reuse of historical experience. This work evaluated EvoRule Harness by integrating it into Meta-Harness, TTHE, and Continual Harness. Its effectiveness was validated through comparisons with a range of baselines, including prompting-based methods, memory-augmented approaches, and offline retrieval techniques. Experimental results demonstrate performance gains on benchmarks such as Tau-Bench, Terminal-Bench-2, and SWE-bench Verified. Furthermore, continuous adaptation experiments, ablation studies, and case analyses provide additional evidence for the system's efficacy.

Critical strengths
  • The research addresses a problem of practical significance. This paper investigates the challenges of accumulating experience and maintaining reliability during continuous harness optimization. Balancing the incorporation of new experience with the preservation of existing, effective knowledge is a valuable area of ​​study within the field of harness optimization.

  • The methodology is systematic and comprehensive. The work establishes a complete memory management framework. Leveraging a dual-layer memory structure (Stable/Volatile), atomic rule editing, confidence-based updates, and a rule-promotion mechanism, the framework supports the continuous updating, verification, and rollback of rules.

  • Strong modularity and extensibility. EvoRule Harness can be integrated as an independent component into various existing harness optimizers. Performance improvements have been observed across Meta-Harness, TTHE, and Continual Harness, providing preliminary evidence that the component is reusable across different systems.

  • Extensive experimental analysis and case studies. The paper includes ablation studies, statistics on rule operations, continuous adaptation experiments, and multiple case analyses. The appendix also provides detailed prompt templates, helping readers understand the system’s operational logic and implementation details.

Critical weaknesses
  • Limited methodological novelty. The core mechanism draws heavily from existing memory management concepts—including conditional retrieval, structured rule editing, dual-layer memory, confidence updates, and rule promotion. While this combined design is practically useful, the specific technical contributions that distinguish it from existing agent-based memory frameworks remain unclear.

  • Lack of fair comparative experiments against memory baselines. Although Table 1 includes methods like MemoHarness and Reflexion, the study lacks controlled experiments—specifically, tests where alternative memory mechanisms are integrated into the same underlying "Harness" optimizer. Consequently, it is difficult to discern whether the observed performance gains stem from EvoRule’s unique design or simply from the general benefits of incorporating memory enhancement.

  • Absence of sensitivity analysis for key hyperparameters. Several core hyperparameters—such as the rule promotion threshold (0.8), the minimum number of validation rounds (3), and the influence gate threshold (0.6)—are fixed throughout the experiments. The paper does not conduct sensitivity tests for these parameters. Given that such parameters directly govern the stability and plasticity of the memory system, its overall robustness has not been sufficiently validated.

  • Insufficient description of memory initialization and online evaluation protocols. The paper fails to clarify whether the initial memory store begins empty, nor does it explicitly detail the logical flow connecting task execution, feedback collection, rule updates, and final scoring across different benchmarks. Such details significantly influence the performance of continuous adaptive methods and are critical for ensuring fair comparative evaluation.

  • Limited validation of lifelong adaptive capabilities. The current ten-round experiments primarily report aggregated Pass@1 metrics, lacking a direct analysis of performance retention on historical tasks. Furthermore, since rule promotion and conflict resolution are sensitive to task ordering, the paper provides little to no assessment of the system’s stability under varying task sequences or scenarios involving distribution shifts.

Questions for authors
  • Memory Baselines. Can the authors conduct controlled comparative experiments—using the same underlying Harness optimizer, consistent feedback signals, and identical computational budgets—against simple empirical memory mechanisms or existing memory-based methods? This would help isolate the incremental gains specifically attributable to the unique design of the EvoRule Harness.

  • Rule Validation and Hyperparameters. How were the promotion threshold (0.8), minimum validation rounds (3), and influence gate threshold (0.6) determined? Could the authors provide a sensitivity analysis for these parameters, and clarify the calculation and verification processes for rule confidence scores as well as the individual contribution of each patch?

  • Online Evaluation Protocol. Is the initial memory bank empty, or is it pre-loaded with rules? At what stage does the memory update occur within the various benchmarks?

  • Memory Generalization and Lifelong Adaptation. Could the authors include a dedicated experiment on memory generalization? This would involve freezing the memory after accumulating rules on source tasks, then transferring it directly to unseen tasks or environments—comparing this against settings with an empty memory or one that undergoes online updates during the target task. Furthermore, could the authors analyze how variations in task sequence affect rule reuse, the retention of performance on historical tasks, and the final experimental outcomes?

7 comments
AuthorsOpenReview ↗

response for Reviewer nv8b (1/7)

W1: Limited methodological novelty. The core mechanism draws heavily from existing memory management concepts, including conditional retrieval, structured rule editing, dual-layer memory, confidence updates, and rule promotion. While this combined design is practically useful, the specific technical contributions that distinguish it from existing agent-based memory frameworks remain unclear. A: Thanks for your comments on the novelty, however, we do not claim that atomic editing, rule memory, or evidence-based updating is new in itself. Our claim is narrower: the contribution is the patch lifecycle as a whole.

For long-horizon reliability, existing methods alone are insufficient. CLIN evolves task-level causal abstractions without a patch unit, so it cannot attribute or reverse a single update. ExpeL edits an insight pool through aggregate votes without per-patch rollback, so it cannot undo one insight without disturbing others. ACE applies batch-level deltas without individual undo or a tier, so it cannot gate a single patch or protect stable memory. No existing work combines all these components in this way, and the combination is necessary rather than incidental: a patch-level unit with provenance and undo_ref makes an update attributable and reversible; the precedent gate operates on that unit; disjoint tiers separate admission from consolidation; one-way atomic promotion keeps volatile edits from reaching stable memory directly; and per-patch undo makes a bad promotion reversible without touching other rules.

Their benefits are synergistic, because each mechanism presupposes the others and the empirical gains depend on the whole protocol rather than on any single component. This also matches the NeurIPS 2026 reviewing guidelines (https://neurips.cc/Conferences/2026/ReviewerGuidelines), which recognize a novel combination of existing techniques as a valid form of originality.

For the specific differences from prior systems, we now put them in a separate paragraph. CLIN has no patch unit. ExpeL has no per-patch rollback. ACE has no individual undo or tier. EvoRule treats each edit as a patch with provenance and undo_ref, conditions REFINE/SPLIT on the applicability context, gates each patch before it enters volatile memory, and promotes it one-way and atomically into a disjoint stable tier. The difference is the ability to attribute, reverse, and gate a single update without touching others, which none of the three provides inside the same Meta-Harness loop.

The controlled comparisons and targeted ablations below ask what the protocol adds beyond the related approaches and which mechanisms produce that gain. All variants share the same Meta-Harness base, frozen Claude Opus 4.6 backbone, split, and budget, averaged over five seeds. The -style variants reuse only the memory-handling approach of each prior system inside our framework; they are not the original systems run unchanged. Holding base, backbone, split, and budget fixed isolates the effect of the memory substrate and operation set, and the results show that the full patch lifecycle is what delivers long-horizon reliability.

Table R8. Memory substrate and atomic-operation ablations (Meta-Harness base; frozen Claude Opus 4.6; same split and budget; mean over 5 seeds).

VariantT-B-2 Pass@1 ↑R10 Cum. ↑Regression Rate ↓Edit Precision ↑
Meta-Harness only76.476.48.6%—
+ CLIN-style causal action abstractions (adapted; no per-patch provenance/rollback, no precedent gate, no tier)77.877.18.0%0.60
+ ExpeL-style memory (ADD/EDIT/UPVOTE/DOWNVOTE; aggregate provenance, no context-conditioned SPLIT/MERGE, no rollback)78.177.36.7%0.63
+ ACE-style memory (delta + counter; batch provenance, no tier, no rollback)78.577.56.4%0.66
+ EvoRule w/o provenance only (rollback retained; patch-level attribution removed)79.678.15.2%0.73
+ EvoRule w/o rollback only (provenance retained; undo_ref removed)79.578.07.4%0.78
+ EvoRule w/o rollback/provenance79.478.07.6%0.72
+ EvoRule w/o Refine79.678.25.0%0.75
+ EvoRule w/o Split79.778.35.3%0.74
+ EvoRule w/o Merge79.978.44.5%0.77
+ EvoRule w/o Split/Merge (O={ADD,DELETE,REFINE})79.278.15.7%0.72
+ EvoRule w/o precedent gate (=0)78.877.66.2%0.68
+ Full EvoRule80.378.63.1%0.79
AuthorsOpenReview ↗

response for Reviewer nv8b (2/7)

Memory augmentation helps: CLIN-, ExpeL-, and ACE-style substrates raise Pass@1 from 76.4 to 77.8-78.5. But memory alone does not explain the gain. Full EvoRule reaches 80.3 Pass@1 and 78.6 R10 cumulative, with regression down to 3.1% and edit precision up to 0.79, versus 6.4-8.6% regression and 0.60-0.66 precision for the three substrate variants. The ablations locate the gain: removing provenance raises regression to 5.2%; removing rollback to 7.4%; removing the precedent gate to 6.2%; and removing Split/Merge together lowers Pass@1 to 79.2 and raises regression to 5.7%. Each piece contributes, but the full combination gives the lowest regression and highest edit precision. This substantiates EvoRule’s contribution to lifelong harness adaptation: the baselines can store and update experience, but they cannot attribute, reverse, and gate a single patch without disturbing the others. We have included these results in the revision (Table R1 and Appendix A.4).

AuthorsOpenReview ↗

response for Reviewer nv8b (3/7)

W2: Lack of fair controlled comparisons with memory baselines. Although Table 1 includes MemoHarness and Reflexion, it does not integrate alternative memory mechanisms into the same “Harness” optimizer, so it remains unclear whether gains stem from EvoRule’s unique design or memory enhancement generally. Q1: Can authors compare under the same Harness optimizer, consistent feedback, and equal compute against simple empirical or existing memory methods to isolate incremental gains attributable specifically to EvoRule’s design?

A: We thank you for suggesting this controlled comparison. To directly assess whether EvoRule provides benefits beyond memory augmentation, we conducted additional experiments integrating alternative memory mechanisms into the same Meta-Harness optimizer.

We compare Reflexion-style episodic memory, MemoHarness-style experience retrieval, ExpeL-style insight editing, and ACE-style delta memory. All variants use the same frozen Claude Opus 4.6 backbone, feedback signal, data split, and matched token and LLM-call budgets, with results averaged over five seeds. The Meta-Harness-only baseline retains the optimizer’s standard search history and has no added memory module. The other baselines adapt the respective memory mechanisms to this common framework; they are not evaluations of the original systems in their entirety. Table R7 reports the results below.

Table R7. Controlled memory baselines on the same Meta-Harness optimizer (frozen Claude Opus 4.6; same split and budget; mean over 5 seeds).

Memory mechanism inside Meta-HarnessTerminal-Bench-2 Pass@1 ↑R10 Cum. ↑Regression Rate ↓Edit Precision ↑
Meta-Harness (no memory)76.476.48.6%N/A
Reflexion-style episodic memory77.677.07.8%0.58
MemoHarness-style experience retrieval78.277.46.5%0.64
ExpeL-style insight editing78.177.36.7%0.63
ACE-style delta memory78.577.56.4%0.66
EvoRule (ours)80.378.63.1%0.79

Memory augmentation helps in general, but EvoRule adds a further 1.8 points over the strongest memory baseline and lowers the regression rate from 6.4% to 3.1%. This indicates that the gain is not merely from adding memory. It comes from the harness optimization design: volatile-to-stable promotion, evidence-gated validation, stable-prioritized conflict resolution, and traceable atomic edits. The ablation results in Table 5 point in the same direction: removing the stable tier, the volatile tier, or the promotion gate degrades performance by 1.4 to 3.7 points.

W3: No sensitivity analysis for key hyperparameters. Core settings such as the rule promotion threshold 0.8, minimum validation rounds 3, and influence gate threshold 0.6 are fixed across experiments, with no sensitivity tests. Since these parameters govern memory stability and plasticity, overall robustness is insufficiently validated. Q2: How were the promotion threshold 0.8, minimum validation rounds 3, and influence gate threshold 0.6 chosen? Can the authors provide sensitivity analysis for these parameters and clarify how rule confidence scores are calculated and verified, as well as each patch’s individual contribution?

A: Thanks for these comments. We agree that c=0.8c=0.8, θage=3\theta_{\mathrm{age}}=3, and ϵ=0.6\epsilon=0.6 were fixed without sensitivity analysis.were fixed without sensitivity analysis. We now report a sweep on Terminal-Bench-2-Stream. The thresholds were chosen on a held-out validation split before final test evaluation, not tuned on the test set. Table R8 shows that performance is stable in a broad region around the chosen values, and that stricter thresholds trade $ Pass@1 $ for lower regression and higher promotion precision.

Table R8. Hyperparameter sensitivity (Meta-Harness base; frozen Claude Opus 4.6; mean over 5 seeds).

SettingPass@1 ↑Regression Rate ↓Promotion Precision ↑#Promoted
$ c=0.75 $80.13.6%0.8415
$ c=0.80 $ (default)80.33.1%0.8613
$ c=0.85 $80.03.0%0.8811
$ c=0.90 $79.62.9%0.909
$ \mathrm{age}=2 $80.03.5%0.8415
$ \mathrm{age}=3 $ (default)80.33.1%0.8613
$ \mathrm{age}=4 $80.13.0%0.8711
$ \mathrm{age}=5 $79.72.9%0.899
$ \epsilon=0.4 $79.84.0%0.8316
$ \epsilon=0.6 $ (default)80.33.1%0.8613
$ \epsilon=0.8 $79.92.9%0.8810

Rule confidence is computed as in Eq. (6), with the prior strength fixed at (n0=2)(n_0=2) throughout the sensitivity analysis. These results show that EvoRule maintains consistent performance across a range of hyperparameter settings. Stricter thresholds reduce regression and improve promotion precision at the cost of slightly lower Pass@1, demonstrating a controllable tradeoff between stability and adaptability.

Reviewer_N5eTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoRule Harness, a plug-and-play framework for lifelong harness adaptation. It introduces a two-tier memory architecture with stable and volatile rules, atomic updates, and validation-based promotion to support continuous adaptation without backbone training. Experiments on Tau-Bench, Terminal-Bench-2, SWE-bench Verified, and classification tasks show performance improvements across several existing harness optimization frameworks.

Critical strengths
  1. This paper presents its overall framework and key design components in a relatively clear and structured manner.

  2. The method is plug-and-play and demonstrates performance improvements across multiple harness optimization frameworks and benchmarks without requiring backbone model training.

Critical weaknesses

A core claim of this paper is the achievement of lifelong adaptation under non-stationary task distributions, alongside the mitigation of forgetting. The authors further interpret the improvement in cumulative performance shown in Figure 3 as evidence that long-horizon forgetting has been alleviated.

A key concern of mine is that the existing experimental results do not directly support this central conclusion. Figure 3 only displays the cumulative Pass@1 on Terminal-Bench-2-Stream as the number of adaptation rounds increases. However, an increase in cumulative performance is not equivalent to the retention of capabilities on previously encountered tasks. Even if a system forgets previously mastered tasks while adapting to new ones, the cumulative metric could still rise. I also did not find the results of re-evaluating the historical task after each round of memory update, or other indicators that directly measure forgetting/retention, in the main text and appendix.

Furthermore, the paper does not sufficiently explain how the task distribution evolves across the 10 consecutive rounds, nor whether this constitutes a clear distribution shift. Simply partitioning tasks into sequential batches does not constitute a robust validation of long-term adaptation capabilities in a non-stationary environment. Consequently, it remains unclear whether the performance gains in Figure 3 stem from improved knowledge retention or merely from the accumulation of task experience over time. So I think the authors need to clarify the nature of the shifts in task distribution and provide results from re-testing historical tasks—along with specific metrics for forgetting or retention—to substantiate their core conclusion.

Another key concern I have is whether this design might actually limit the system's adaptability in non-stationary environments.

A "stable rule" that has been well-validated in the past is not necessarily still correct once the environment changes. For instance, if tool APIs or task requirements evolve, an existing stable rule may become obsolete—even if the system has generated a correct "volatile rule" based on new feedback. According to Eq. 12, however, this new rule could be discarded due to conflicts, leaving the system to continue executing the now-invalid stable rule. Protecting historical rules from being overwritten is not the same as ensuring that the system’s performance will not degrade in a new environment.

Although the authors allow for the removal of stable rules after multiple failures, the paper does not sufficiently specify how many failures are required before an obsolete rule is corrected, nor does it provide an experimental analysis addressing this issue.

Questions for authors
  1. How exactly is the cumulative Pass@1 in Figure 3 computed? Are previously encountered tasks re-evaluated after each adaptation round? If so, could the authors provide these results to directly assess forgetting and retention?

  2. How are the 89 tasks ordered and partitioned into 10 rounds in Terminal-Bench-2-Stream? What specific distribution shifts occur across rounds, and how are they characterized or quantified?

  3. When a previously validated stable rule becomes obsolete due to environmental changes, how does the system handle a correct but conflicting volatile rule under Eq. (12)? How many failures are typically required before the obsolete stable rule can be removed, and is there any empirical evidence on the resulting adaptation delay or recovery performance?

5 comments
AuthorsOpenReview ↗

response for Reviewer N5eT (1/5)

W1-1: Fig. 3 only reports cumulative Pass@1, which can rise despite forgetting and lacks post-update historical re-evaluation or direct forgetting/retention metrics. Q1: How is cumulative Pass@1 computed, are historical tasks re-evaluated each round, and what are the forgetting/retention results? Q2: How are the 89 Terminal-Bench-2-Stream tasks ordered into 10 rounds, and how are distribution shifts across rounds quantified?

We thank you for raising this point. Cumulative Pass@1 at round r is the running average over all tasks seen through round r: Cumulative@r=1N1:r∑i=1N1:ryi\text{Cumulative@}r=\frac{1}{N_{1:r}}\sum_{i=1}^{N_{1:r}} y_i, where $ y_i=1 $ if task $ i $ is solved when it is first encountered with the current harness, and $ N_{1:r} $ is the number of tasks seen through round $ r $.

To measure retention directly, we add historical-task replay after every memory update: all tasks from earlier rounds are re-evaluated after each update. We report three metrics alongside cumulative Pass@1: Retention@r, the fraction of tasks solved in any earlier round that remain solved at round r; Forgetting, the average drop from each task’s best previous correctness to its current correctness; and Backward Transfer, the average change in correctness on earlier tasks from first exposure to the final round. We also report round-by-round retention curves.

Table R4. Historical-task replay on Terminal-Bench-2-Stream (Meta-Harness base; frozen Claude Opus 4.6; same split and budget; mean over 5 seeds).

MethodR10 Cumulative ↑Retention@R10 ↑Forgetting ↓Backward Transfer ↑
Meta-Harness76.40.580.18−0.07
+ ACE-style memory77.50.620.14−0.04
+ ExpeL-style memory77.30.610.15−0.05
+ EvoRule w/o stable tier76.60.570.19−0.08
+ Full EvoRule78.60.710.09+0.03

Retention@R10 = fraction of tasks solved in any earlier round that remain solved at R10. Forgetting = average drop from each task’s best previous correctness to its R10 correctness. Backward Transfer = average change in correctness on earlier tasks from first exposure to R10; positive means later adaptation improves earlier tasks.

Table R4 shows that Full EvoRule retains more previously solved tasks and exhibits less forgetting than Meta-Harness and the no-stable-tier variant. Its positive backward transfer also indicates improved performance on earlier tasks after subsequent adaptation. These historical-task results provide direct evidence of improved retention, complementing the cumulative Pass@1 reported in Figure 3.

AuthorsOpenReview ↗

response for Reviewer N5eT (2/5)

W1-2: The paper does not explain how the task distribution evolves across the 10 rounds or why this constitutes a genuine distribution shift; sequential batching alone is not a robust non-stationary validation. It is therefore unclear whether Fig. 3 gains reflect retention or mere task-experience accumulation. The authors should clarify the shift and provide historical-task re-testing with explicit forgetting/retention metrics.

A: We thank you for requesting a clearer account of the stream construction and direct evidence of retention. The 89 tasks are organized into ten rounds, with nine tasks in each of R1–R9 and eight in R10. The stream varies in task-domain and tool-family composition, and includes explicit tool/API changes at R5 and R8. These interface changes alter the execution requirements of the affected tasks. Table R5 summarizes the round composition, and the appendix provides the task IDs, ordering procedure, and specific tool changes.

The stream introduces actual changes in execution conditions through tool/API updates at R5 and R8, rather than relying solely on differences in task descriptions. The embedding-based MMD (Here, MMD denotes Maximum Mean Discrepancy, a kernel two-sample statistic that measures the distance between two probability distributions; in our setting, it is computed on text embeddings to quantify how much the instruction distribution of one round differs from that of another, where larger MMD indicates greater distributional discrepancy and values near zero indicate similar distributions) and adjacent-round classifier results in Table R5 summarize how instruction distributions differ across rounds. They provide supplementary characterization of the stream; the tool/API changes specify the corresponding changes in execution conditions. A classifier accuracy of 0.50 corresponds to chance-level discrimination.

Table R5. Stream composition and shift magnitude by round (Terminal-Bench-2-Stream; 89 tasks).

RoundTask domains / tool families#TasksMMD vs. previous round ↑Domain-classifier acc. vs. previous round ↑
R1file operations, shell scripting9——
R2git, package management90.070.54
R3build/compile, make90.100.60
R4process management, networking90.080.57
R5containers, database CLI (API change)90.220.81
R6permissions, security tooling90.110.62
R7logs, text processing90.090.59
R8cloud CLI, tool signature change90.240.83
R9multi-service config, environment setup90.100.61
R10mixed terminal tasks80.130.65

To assess whether earlier capabilities survive subsequent adaptation, we additionally conducted the historical-task re-evaluations described in our response to W1/Q1. Table R4 reports retention of 0.71 for Full EvoRule versus 0.58 for Meta-Harness, and forgetting of 0.09 versus 0.18. These measurements directly assess historical-task performance, separately from the cumulative Pass@1 reported in Figure 3.

The two evaluations therefore distinguish performance accumulated as new tasks arrive from the ability to continue solving historical tasks. Together, the documented stream changes and historical-task results support EvoRule’s retention benefits under the evaluated task and tool changes.


AuthorsOpenReview ↗

response for Reviewer N5eT (3/5)

W2: “Another key concern is whether this design limits adaptability in non-stationary environments. A validated stable rule may become obsolete after the environment changes, yet Eq. 12 could still discard a correct new volatile rule due to conflict, leaving the invalid stable rule in use. Protecting historical rules therefore does not ensure non-degradation, and the paper neither specifies the failure threshold for correcting an obsolete rule nor provides an experimental analysis.”

A: We would like to clarify a potential misunderstanding about stable-tier protection in EvoRule. Stable-priority does not imply that stable rules are immutable. Eq. (12) specifies their priority during inference, and the separate stable-tier update mechanism allows evidence-based removal (Sec. 3.2, Eq. 7). A stable rule is flagged as stale when it fails on at least three distinct validation subsets across at least two rounds, or when its rolling failure rate exceeds 0.5 with at least four observations. This triggers a DELETE proposal, which is reviewed by the stable-tier safety checker and applied only after approval. Following removal, the previously conflicting volatile rule becomes eligible for inference under Eq. (12). That is, promotion to stable memory still requires the standard confidence-and-validation-age criterion.

To evaluate the resulting adaptation delay, we conducted an additional controlled drift experiment on Terminal-Bench-2-Stream. At round 5, the repo-search tool signature and associated task requirements change, making previously validated guidance obsolete. We compare Full EvoRule with Meta-Harness and an EvoRule variant with stable-tier DELETE disabled. Table R6 reports the pre- and post-drift performance, recovery time, final performance, and number of obsolete stable rules remaining. The appendix also traces a representative case from the initial conflict through stable-rule deletion and subsequent recovery.

Table R6. Controlled drift experiment on Terminal-Bench-2-Stream with tool signature and requirement changes at round 5 (mean and std over 5 seeds).

MethodPre-drift per-round Pass@1 at R4 ↑Post-drift minimum per-round Pass@1 ↑Rounds to recover ↓Final Pass@1 at R10 ↑Obsolete stable rules remaining ↓Stale-rule DELETE triggered?
Meta-Harness71.2 (0.4)62.3 (0.7)5.8 (0.5)76.4 (0.5)N/ANo
+ EvoRule w/o stable-tier DELETE71.6 (0.4)65.1 (0.6)4.6 (0.4)77.2 (0.5)3.2 (0.4)No
+ Full EvoRule71.9 (0.3)67.9 (0.5)2.1 (0.3)80.3 (0.6)0.4 (0.2)Yes

In this experiment, Full EvoRule recovers faster and leaves fewer obsolete stable rules than the variant without stable-tier DELETE. This comparison supports the role of validated deletion in replacing outdated guidance and restoring adaptability after environmental drift.

Reviewer_zM4SOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

EvoRule Harness equips frozen LLM agents with a two-tier volatile–stable rule memory that continually learns from execution failures through traceable and reversible atomic patches. New rules are promoted to stable memory only after repeated validation, enabling long-term harness improvement without model training while reducing the risk of regressions.

Critical strengths
  • The presentation and paper writing are good.
  • The experimental results appear convincing.
  • The overall framework of the paper seems well designed and reasonable.
Critical weaknesses

My primary concern is the paper’s technical novelty. Its main contributions appear to center on: (1) a stable–volatile two-tier rule memory, (2) traceable and reversible atomic rule edits, and (3) evidence-based rule validation and promotion. To assess these contributions, I focus on related work publicly available at least three months before the ICLR submission deadline.


(1) The novelty of the rule representation and atomic editing mechanism is insufficiently distinguished from prior work. Majumder et al. (2023) [1] maintain evolving causal action abstractions; Zhao et al. (2024) [2] update extracted insights through explicit editing operations and importance counts; and Zhang et al. (2025) [3] introduce structured memory entries with identifiers, feedback counters, and localized delta updates. Although these approaches do not establish EvoRule’s exact rule schema or independent patch rollback mechanism, they provide substantial precedent for its underlying ideas. The paper should clarify and empirically substantiate the additional contribution of its specific operation set, provenance tracking, and rollback design.

References

[1] Majumder et al. (2023). CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization.

[2] Zhao et al. (2024). ExpeL: LLM Agents Are Experiential Learners.

[3] Zhang et al. (2025). Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models.


(2) The stable–volatile memory design is not sufficiently distinguished from prior work. Zhang et al. (2026) [4] introduce candidate probation and evidence-based promotion into active memory, while permitting controlled use of candidate entries. Zha and Wang (2026) [5] separate candidate memory from trusted memory and require runtime-enforced checks before promotion. Although these systems differ from EvoRule in their applications and trust criteria, they provide relevant precedent for separating tentative from established memory and restricting admission to a trusted store. The paper should clarify the additional contribution of its stable-priority conflict resolution and restrictions on stable-tier modifications.

References

[4] Zhang et al. (2026). Evolving Skill-Structured Attack Memory Enhances LLM Jailbreaking.

[5] Zha and Wang (2026). Autonomous LLM Agent Worms: Cross-Platform Propagation, Automated Discovery and Temporal Re-Entry Defense.


(3) The evidence-based validation and promotion mechanism also has relevant precedent. Cao et al. (2025) [6] retain newly derived lessons only after successful retries and prune existing memories using cumulative utility and minimum-usage thresholds. Zhang et al. (2026) [4] introduce candidate probation and evidence-driven promotion, alongside Beta posterior tracking for memory selection. Although these methods do not establish EvoRule’s specific confidence-plus-age promotion criterion or cross-subset validation protocol, they already explore execution-based admission and memory management driven by accumulated evidence. The paper should clarify the additional technical contribution of its promotion rule and compare it with simpler validation and evidence-threshold baselines.

References

[4] Zhang et al. (2026). Evolving Skill-Structured Attack Memory Enhances LLM Jailbreaking.

[6] Cao et al. (2025). Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution.


Overall, while I appreciate the encouraging empirical results and the coherent framework, the paper’s technical novelty remains my main concern. Without a clearer demonstration of its contribution beyond prior work, I am unable to recommend acceptance in its current form.

Questions for authors

See the weeknesses.

6 comments
AuthorsOpenReview ↗

response for Reviewer zM4S (1/4)

W1: “The novelty of the rule representation and atomic editing mechanism against Majumder et al. (2023) [1], Zhao et al. (2024) [2], and Zhang et al. (2025) [3].”

A: We thank you for highlighting these closely related studies. We do not claim that every component is individually novel. Rather, our novelty lies in a coordinated update protocol that makes long-horizon reliability achievable. Existing methods each cover part of the problem, such as evolving abstractions, editing insights, or maintaining structured entries, but none alone provides the full set of controls needed for reliable continual adaptation over long horizons. Such reliability requires that individual changes be attributable and reversible, that tentative rules be isolated and validated before consolidation, and that consolidated knowledge be protected from unvalidated edits.

These controls are mutually enabling rather than merely additive: patch-level provenance and undo references make selective rollback and evidence-gated promotion enforceable; disjoint volatile and stable tiers make atomic promotion meaningful; and stable-priority conflict resolution preserves consolidated knowledge while still allowing nonconflicting volatile rules to guide inference. To our knowledge, no closely related method integrates all of these controls into a single update protocol for lifelong harness adaptation, and their combination yields adaptation and retention benefits that no single component alone provides. Our contribution is this coordinated design and its empirical validation within the same Meta-Harness setting.

We emphasize that a carefully motivated combination can constitute substantive methodological novelty when its additional benefits are demonstrated. This perspective is consistent with the NeurIPS 2026 reviewing guidelines (https://neurips.cc/Conferences/2026/ReviewerGuidelines). We clarify both our contribution and its differences from CLIN, ExpeL, and ACE below.

Additionally, to directly address the reviewer’s concern, we conducted additional experiments during the rebuttal period. These experiments assess what EvoRule adds beyond the related approaches and how its specific mechanisms contribute. All variants use the same Meta-Harness base, frozen Claude Opus 4.6 backbone, and data split, with matched token and LLM-call budgets. The -style baselines adapt the memory-handling approach of each prior system to our harness-optimization framework. They therefore provide comparisons of adapted memory substrates under shared conditions, rather than evaluations of the original systems in their entirety.

Table R1 shows that full EvoRule achieves 80.3 Pass@1 compared with 78.5 for the strongest adapted baseline, ACE-style memory, while reducing regression from 6.4% to 3.1%. The targeted ablations further support complementary benefits from the extended operation set, patch-level attribution, and selective rollback. In particular, disabling rollback increases regression to 7.4%, highlighting its contribution to retaining previously successful behavior. We appreciate the reviewer’s request for a clearer distinction from prior work and believe that these technical distinctions and empirical findings substantiate the additional methodological value of EvoRule. We have incorporated the new results and experimental details into the revised manuscript (Table R1 and Appendix A.4).

[Detailed difference ] In detail, CLIN maintains evolving causal abstractions but has no patch-level unit for attributing or reversing a single update. ExpeL combines explicit insight editing with importance counts but relies on aggregate votes and lacks per-patch rollback. ACE supports structured entries with localized delta updates and feedback counters but operates at batch level without individual undo or tier separation. EvoRule builds on these capabilities by making patch-level traceability, selective rollback, and evidence-gated consolidation explicit parts of the update protocol. Each patch records its provenance and undo reference undo_ref; new or revised candidate rules are screened and validated in volatile memory, and become eligible for atomic promotion to the disjoint stable tier only after sufficient validation evidence accumulates; applicability-conditioned REFINE and SPLIT address overgeneralized rules; and tier-specific update constraints prevent tentative edits from directly modifying stable rules.

AuthorsOpenReview ↗

response for Reviewer zM4S (2/4)

Table R1. Memory-substrate and atomic-operation ablations.

VariantT-B-2 Pass@1 ↑R10 Cum. ↑Regression Rate ↓Edit Precision ↑
Meta-Harness only76.476.48.6%—
+ CLIN-style causal action abstractions77.877.18.0%0.60
+ ExpeL-style memory78.177.36.7%0.63
+ ACE-style memory78.577.56.4%0.66
+ EvoRule w/o provenance only79.678.15.2%0.73
+ EvoRule w/o rollback only79.578.07.4%0.78
+ EvoRule w/o rollback/provenance79.478.07.6%0.72
+ EvoRule w/o Refine79.678.25.0%0.75
+ EvoRule w/o Split79.778.35.3%0.74
+ EvoRule w/o Merge79.978.44.5%0.77
+ EvoRule w/o Split/Merge79.278.15.7%0.72
+ EvoRule w/o precedent gate78.877.66.2%0.68
+ Full EvoRule80.378.63.1%0.79

All runs use the Meta-Harness (MH) base, frozen Claude Opus 4.6, the same data split and budgets, and report the mean over 5 seeds. Variant legend: -style variants reuse only the memory-handling approach of the corresponding prior system within our harness-optimization framework; they are not the original systems run unchanged. CLIN-style is adapted to our framework and has no patch-level provenance/rollback, no precedent gate, and no tier. ExpeL-style uses ADD/EDIT/UPVOTE/DOWNVOTE with aggregate provenance and no context-conditioned SPLIT/MERGE or rollback. ACE-style uses delta + counter with batch provenance and no tier or rollback. Ablations: -prov keeps rollback but removes patch-level attribution; -rollback keeps provenance but removes undo_ref; -Split/Merge uses O={ADD,DELETE,REFINE}; -gate=0 disables the precedent gate by setting its threshold to 0 (i.e., the “w/o precedent gate” ablation), making the gate non-restrictive/effectively disabled.

AuthorsOpenReview ↗

response for Reviewer zM4S (3/4)

W2: “Differentiation of the stable-volatile memory design from prior work, particularly regarding stable-priority conflict resolution and restrictions on stable-tier modifications.”

A: EvoRule contributes a coordinated control protocol governing both how tentative rules influence inference and how consolidated rules may be modified. Stable-priority conflict resolution preserves the authority of validated rules when the two tiers conflict, while allowing nonconflicting volatile rules to guide inference. Stable-tier modifications are restricted to MERGE and DELETE under explicit evidence requirements, preventing tentative edits from directly rewriting consolidated rules. Patch-level provenance and undo records additionally support tracing and selectively reverting individual changes.

Zhang et al. manage candidate use through probation and evidence-based promotion, while Zha and Wang enforce runtime checks before admission into trusted memory. These approaches do not specify stable-priority conflict resolution between tentative and consolidated rules retrieved together for inference, nor do they impose EvoRule’s restrictions on subsequent stable-tier modifications. EvoRule addresses both decisions as follows: stable rules take precedence when the two tiers conflict (Eq. 12, Sec. 3.3), and stable-tier modifications are restricted to MERGE and DELETE under explicit evidence requirements (Sec. 3.2 and Appendix A.3). These controls govern both inference-time rule selection and post-consolidation revision, extending the contribution beyond memory separation and promotion.

To further address the concerns from the reviewer, we have added this comparison during the rebuttal period. Zhang-style candidate probation, Zha-style runtime checks, single-tier status tags, confidence priority, and no arbitration all share the same frozen backbone, split, top-K, and budget. Table R1 reports the results.

As shown, full stable priority has the lowest regression rate (3.1% vs. 5.6-8.4%) and the highest conflict accuracy (0.94 vs. 0.59-0.88), while nonconflicting volatile rules still participate. It gives up some new-failure recovery (0.71) relative to no arbitration (0.79) and confidence priority (0.77). That is the intended tradeoff: we prefer lower regression and better conflict resolution over aggressive admission. Single-tier tags are worst on regression (8.4%) and Pass@1 (77.4), which shows that separation alone is insufficient. The table therefore supports the main claim: non-regressive adaptation comes from the governance contract, not from separation or promotion by itself.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.