acceptodds
Under review as a conference paper at ICLR 2027

MetaRule: Transactional Procedure Learning for Cross-Domain Harness Evolution

Abstract

Language-model agents rely on harnesses to organize context, memory, tools, and output handling, yet repeated execution does not explain when a local trace should become durable behavior. Existing model-centric approaches can store memories, revise prompts, or search harness configurations, but they lack a transaction for deciding when execution evidence is closed enough to persist, limiting its scope, and keeping learning separate from evaluation. We present , a transactional procedure-learning mechanism for self-evolving harnesses. It converts answer-free execution traces into typed, scoped, lifecycle-managed programs: replayable checkpoints expose telemetry to an isolated diagnoser; bounded same-thread revision and typed closure establish whether a procedure completed; and paired admission, provenance, and delayed publication control what a successor harness may execute. The compiler dispatches only trigger- and capability-matched operations in the next frozen state, while official verification and read-only held-out evaluation preserve the measurement boundary. Across scientific and competition-mathematics benchmarks, the complete treatment improves held-out performance over native harness evolution, with the clearest gains in registered cross-domain and cross-type transfer and in recovery from native negative transfer. Matched frozen-state and trajectory-shadow analyses place these gains at the complete harness-treatment level and quantify uncertainty around persistent procedure transfer. provides a replayable, auditable substrate for studying how execution evidence changes an agent's future behavior.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language agents, test-time adaptation, procedural retrieval, cross-domain transfer, meta-cognitive reasoning, agent memory, harness optimization, failure diagnosis, in-task revision, meta-rule distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_UpErOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper presents MetaRule, a transactional framework for adapting language-model agent harnesses across tasks. The framework combines answer-free execution evidence, isolated diagnosis, bounded same-thread revision, typed procedural closure, paired admission tests, and delayed publication into a frozen successor state. Procedures are assembled from a finite registry, and rejected positive candidates may produce precautionary reverse rules. The principal contribution is an integrated, auditable lifecycle governing how execution evidence becomes persistent agent behavior.

The evaluation compares MetaRule with an unextended MemoHarness baseline (B0) and native harness evolution (T1) on GPQA-Diamond, AIME2025, and HMMT, supplemented by execution audits and mechanism ablations. Relative to T1, the complete system improves cross-domain GPQA Avg@5 from 72.63% to 82.11% and cross-type HMMT Avg@5 from 35.00% to 55.00%. The strongest evidence concerns complete-system improvements and recovery from negative transfer; the independent contribution of persistent procedures remains inconclusive in the reported frozen-dispatch and trajectory-shadow analyses (Sections 4–6).

Critical strengths
  1. Frozen within-batch states, scoped rule dispatch, delayed publication, and explicit separation of diagnosis from private answers provide a concrete framework for inspecting adaptive agents. Separating procedural closure from official correctness verification is particularly useful because it avoids treating a completed repair as proof of a correct answer. This protocol is a meaningful systems contribution that makes the source and timing of behavioral changes easier to examine (Section 3; Appendices C and E).

  2. On HMMT, MetaRule improves overall Avg@5 from 35.83% to 57.50%, with an exact task-level sign-flip p-value of 0.0007. On the cross-type subset, it improves eight tasks, ties four, and degrades none relative to T1 (Table 4). These results provide meaningful evidence that the integrated system can mitigate deterioration introduced by native evolution. Reporting paired task-level statistics and explicitly limiting causal interpretations strengthens this finding.

Critical weaknesses
  1. Each primary treatment has only one training trajectory, and held-out sets are small. The HMMT improvement over B0 is not statistically conclusive (+5.83 points; p = 0.2585), limiting confidence in a stable advantage over the original harness.
Questions for authors
  1. Does MetaRule outperform a control with the same diagnosis–revision–closure pipeline and matched computational budget, but fixed trigger-conditioned counterchecks instead of learned persistent rules? Do the principal gains, especially the HMMT recovery and improvement over B0, persist across independent training trajectories?
  2. What behavior distinguishes a reverse rule from its rejected positive candidate, and why must every batch publish an artifact?
  3. Which closure conditions are independently verified, and which rely on the task model’s self-report?
6 comments
AuthorsOpenReview ↗

Response to Reviewer UpEr (part 1/4) W1.1/ Q1.2, W1.2

Thank you for recognizing the frozen-state publication protocol and independently verified correctness. You ask whether the gains recur across training runs and whether fixed counterchecks explain them. Across the paper HMMT run and two supplementary campaigns, MetaRule reaches 62.50% ± 4.17% Cons@5, alongside higher Avg@5 than both paper references. We address repeatability first, then compare fixed controls and describe the publication and closure contracts.

Metrics and campaigns. Avg@5 is per-attempt verifier accuracy; HMMT Cons@5 is plurality-answer accuracy; binary Pass@5 and Best@5 coincide. All HMMT task execution uses Haiku 4.5. The paper trajectory used GPT-5.5 for isolated diagnosis; its unavailability through our rebuttal access led us to use Haiku 4.5 for diagnosis in the two added campaigns. Individual results and mean ± sample SD describe these three campaigns, including this configuration change. B0/T1 are paper references.

W1.1 / Q1.2: Independent Trajectories

The three-state evaluation answers the repeatability part descriptively: all three MetaRule trajectories exceed the paper B0 and T1 Avg@5 references. The original trajectory's exact task-paired sign-flip tests give p=0.0007p=0.0007 against T1 and p=0.2585p=0.2585 against B0; added MetaRule trajectories quantify variation across trained states but do not change the original task-paired B0 test.

We evaluate three training states. Each trains for five iterations on the same 39 HMMT tasks and freezes its iteration-5 state for the same 24 held-out tasks × five generations. All use Claude Haiku 4.5 at medium effort, zero tools, and per-task caps of 20 calls and 1,000,000 output tokens. The paper run and supplementary campaigns are completed evaluations of separately trained states. Held-out execution is read-only. Mean ± sample SD describes the three completed campaigns, with the held-out set fixed.

HMMT treatmentAvg@5Pass@5Cons@5Best@5
T1 (paper)35.83%58.33%41.67%58.33%
B0 (paper K=5 reference)51.67%79.17%50.00%79.17%
MetaRule seed 42 (paper trajectory)57.50%87.50%66.67%87.50%
MetaRule seed 2026111352.50%75.00%62.50%75.00%
MetaRule seed 2026111453.33%75.00%58.33%75.00%
MetaRule (three trajectories, mean ± SD)54.44% ± 2.68%79.17% ± 7.22%62.50% ± 4.17%79.17% ± 7.22%

Each Full campaign in the preceding table uses the same 24 held-out tasks × five generations, including every generated attempt in its 120-outcome aggregate.

Your question specifically asks whether HMMT recovery persists across training trajectories. We fix the 11 T1-harmed tasks from the original paper comparison and compare per-task five-generation accuracy. The paper trajectory improves all 11 and restores eight to at least B0. The added seed20261113/20261114 trajectories improve 10/11 and 8/11 and restore 8/11 and 7/11 to at least B0; all remaining tasks tie T1. Thus the recovery pattern recurs at the task level, with no further decline relative to T1 on this fixed subset. In the original trajectory, "7/18" illustrates reliable plurality selection and "279" illustrates coverage beyond both references.

The paired analysis treats each held-out task as one unit, averaging its five generations before the sign-flip test. The supplementary campaigns reuse the same 24 tasks, providing evidence across trained states. Sample SD describes the completed campaigns; task-level tests quantify the original paired contrasts.

W1.2: Held-out Precision

Thank you for raising the concern that “held-out sets are small.” Compact, high-difficulty benchmarks are established evaluation instruments in reasoning research. For example, s1 (EMNLP 2025) evaluates on AIME24 with 30 problems and GPQA Diamond with 198 questions [1]. AceReason-Nemotron (NeurIPS 2025) likewise evaluates on AIME2024/2025 and HMMT2025 February [2]. These published evaluations illustrate the value of carefully constructed competition-mathematics and expert-level science benchmarks for identifying reasoning improvements.

For MetaRule, we strengthen this benchmark evidence through paired task-level evaluation and repeated training campaigns. Across the paper trajectory and both added campaigns, recovery recurs on the original T1-harmed tasks, and every campaign exceeds the paper B0 Avg@5 reference. The central finding is recurring recovery across separately trained states: the improvement is visible in the behavior of the same held-out problems.

[1] Muennighoff et al. s1: Simple test-time scaling. EMNLP 2025. https://aclanthology.org/2025.emnlp-main.1025/

[2] Chen et al. AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning. NeurIPS 2025. https://neurips.cc/virtual/2025/poster/119111

AuthorsOpenReview ↗

Response to Reviewer UpEr (part 2/4) Q1.1 part1

Q1.1: Fixed Trigger-Conditioned Counterchecks

You ask whether MetaRule outperforms “the same diagnosis–revision–closure pipeline and matched computational budget, but fixed trigger-conditioned counterchecks instead of learned persistent rules.” F2 implements the fixed-trigger alternative in that same pipeline; Generic-reasoning is a second control for extra reasoning. Both use the seed-42 frozen iteration-5 state, the same 24 held-out tasks × five generations, Haiku 4.5 at medium effort, zero tools, official verifier, task order, retry policy, and configured caps of 20 calls and 1M output tokens per task. The cost table below reports actual consumption so that the budget part of your question can be assessed directly.

The fixed-check arm uses this predeclared operation set and trigger map:

Registered operationConcrete action
Check givensRe-read the task and check use of stated facts, constraints, and requested format.
Independent derivationDerive a fresh solution from the task evidence.
Separate criteriaEvaluate every required criterion independently and check joint satisfaction.
Resolve conflictsLocate inconsistent claims or steps and reconcile them against task evidence.
Preserve supported workKeep supported conclusions and revise only unsupported or conflicting parts.
Failure triggerFixed checks selected
Choice mapping or missing derivationCheck givens; independent derivation
Conflated criteriaCheck givens; separate criteria
Format errorCheck givens; preserve supported work
Reasoning contradictionCheck givens; resolve conflicts

F2 Fixed-check applies the mapped countercheck after diagnosis, followed by one revision opportunity and typed closure. F2 Generic-reasoning uses a general re-derive-and-inspect instruction in the same pipeline, testing whether extra reasoning alone reproduces the fixed-check effect.

HMMT controlAvg@5Pass@5Cons@5Best@5
F2 Fixed-check (seed 42)39.17%70.83%41.67%70.83%
F2 Generic-reasoning (seed 42)50.00%79.17%50.00%79.17%
B0 (paper reference)51.67%79.17%50.00%79.17%
T1 (paper r3)35.83%58.33%41.67%58.33%
MetaRule seed 42 (paper trajectory)57.50%87.50%66.67%87.50%
MetaRule (three trajectories, mean ± SD)54.44% ± 2.68%79.17% ± 7.22%62.50% ± 4.17%79.17% ± 7.22%
Held-out cost armInput + output tokens (M)
F2 Fixed-check6.271
F2 Generic-reasoning5.760*
B0 (paper)3.688
T1 (paper r3)3.462
MetaRule seed 42 (paper trajectory)8.807
MetaRule (three trajectories, mean ± SD)6.383 ± 2.103

*The Generic-reasoning input-token meter is missing for one call; its total sums all recorded input and output usage.

On the seed-42 paper trajectory, Full has higher observed Avg@5 and Cons@5 than both F2 controls: the advantage concerns correct individual generations and correct plurality answers within the same repair pipeline. Full consumes 8.807M held-out tokens, compared with 6.271M for Fixed-check. This answers the same-pipeline part of your question through a measured performance–cost comparison under shared configured caps. Earlier frozen states provide lower-consumption operating points: iter1 uses 4.063 ± 1.045M tokens (1.10× B0; 1.17× T1), with 54.72% ± 4.88% Avg@5 and 61.11% ± 6.36% Cons@5. These are measured checkpoint comparisons against the paper references.

A/B/C three-arm replay. Each arm uses a frozen iteration-5 parent, read-only evaluation, Haiku 4.5 at medium effort, zero tools, the official verifier, and the same task set, under shared configured caps of 20 calls and 1M output tokens per task.

ArmChecking policyComparison role
A: Published-rule dispatchMatch a retained rule's trigger and scope, execute its required checks, and verify the closure contract.A–B compares learned retained contracts with a predefined map; A–C compares their deployment with general repair.
B: Fixed trigger checkSelect operations from a predefined failure-to-check map; then revise and close.B–C tests adding predefined checks to general repair.
C: Rule-free pipelineKeep diagnosis, revision, and closure, with no rule lookup or rule-specific required checks.General-repair reference; the model can still inspect its answer.
AuthorsOpenReview ↗

Response to Reviewer UpEr (part3/4) Q1.1 part2, Q2.1

The reconstructed five-replicate records cover 23 complete common tasks across three states; one task lacks four A generations in seed 20261113. The 24-task check uses the complete states, seeds 42 and 20261114. Entries are mean ± sample SD across states.

Evaluation scopeArmAvg@5Pass@5Cons@5Best@5
23 common tasks; three statesA: Published rule59.71% ± 1.81%85.51% ± 2.51%69.57% ± 4.35%85.51% ± 2.51%
23 common tasks; three statesB: Fixed check58.26% ± 3.14%81.16% ± 2.51%62.32% ± 10.94%81.16% ± 2.51%
23 common tasks; three statesC: Rule-free57.68% ± 3.05%82.61% ± 4.35%63.77% ± 6.64%82.61% ± 4.35%
24 tasks; two complete statesA: Published rule56.67% ± 1.18%83.33% ± 5.89%66.67% ± 5.89%83.33% ± 5.89%
24 tasks; two complete statesB: Fixed check54.17% ± 1.18%77.08% ± 2.95%54.17% ± 5.89%77.08% ± 2.95%
24 tasks; two complete statesC: Rule-free57.50% ± 2.36%85.42% ± 2.95%60.42% ± 8.84%85.42% ± 2.95%

For three states, descriptive post-hoc exact two-sided task-paired sign-flip tests give A–B/A–C p=.6719/.4927 for Avg@5 and .2344/.2188 for Cons@5. Testing averages generations and states within each task. No contrast reaches .05 after Holm correction across three contrasts per metric. The two scopes agree on A's higher plurality-answer accuracy; the full-24-task comparison places C higher on per-attempt accuracy and coverage.

For your fixed-countercheck alternative, A–B is the closest contrast: both preserve repair and closure, while the source and applicability of the checks change. The recurring benefit is plurality-answer accuracy in both evaluation scopes. A–C also favors correct answers forming the plurality under the same aggregation rule. The paired tests quantify the uncertainty of these differences; the primary Full campaign remains a separate system-level evaluation.

For the geometry problem with answer "7/18", A matches a retained missing-derivation rule, requires checking the givens and independent derivation, and returns a verifier-correct revision after closure. This is the operational role of the retained contract.

Q2.1: Reverse-Rule Semantics

You ask “What behavior distinguishes a reverse rule from its rejected positive candidate?” A positive candidate proposes reusable behavior and must pass paired efficacy admission. A reverse rule is a preventive constraint: it identifies the rejected behavior and blocks blind reuse under a defined trigger, with provenance, required checks, capability scope, and expiry. In the audited GPQA run, five rejected positive candidates followed this precautionary path.

F3 is the offline candidate-to-guard trace audit. Its input is an archived candidate, its paired-gate decision, the published reverse artifact, and the later execution receipt. Appendix D.1 provides one documented GPQA case with the following findings:

Audit stepRecorded finding
Positive proposalOn missing derivation, check givens, derive independently, and preserve supported work.
Admission decisionPositive admission is rejected for insufficient execution coverage; the parent and source gate remain linked to the reverse artifact.
Reverse programForbid blind reuse of that parent. On the missing-derivation trigger, require checking givens and independent derivation within the artifact’s scope.
Later executionA held-out Biology trace executes the two required checks; closure records completion, and the public answer changes B→A and passes the official verifier.

The compiled reverse instruction states: “Use only the public givens and execute every countercheck before changing the supported conclusion.” It also instructs against shortcut reuse of the rejected parent. Rejecting the parent controls positive-program inheritance; the reverse actively binds two checks to later matching tasks, with closure requiring their declared completion. This is the operational difference between withholding a positive program and deploying a precautionary checking contract. F3 establishes the recorded candidate→guard→action path; the accuracy comparisons evaluate the resulting policies.

Reviewer_y17jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MetaRule is a framework for learning and transferring reusable reasoning procedures across tasks through persistent updates to an LLM agent's harness. They introduce a transactional lifecycle consisting of isolated failure diagnosis using answer-free execution telemetry, procedural closure checks, and delayed publication into subsequent frozen harness states. Rejected procedures are used to produce precautionary reverse rules intended to prevent potentially harmful reuse.

The authors evaluate MetaRule on three benchmarks against a native MemoHarness baseline and an evolving harness baseline. MetaRule improves Avg@5 over T1 from 83.27% to 88.73% on GPQA-Diamond and from 35.83% to 57.50% on HMMT. The largest gains are seen on cross-domain and cross-type tasks, particularly in recovering performance degraded by native harness evolution.

Critical strengths
  1. The paper proposes a transactional mechanism for managing persistent procedural updates, including isolated diagnosis, typed closure, paired admission, and frozen-state publication. The distinction between procedural completion and answer correctness is appreciated.

  2. The HMMT experiments show native harness evolution can degrade performance, while the complete MetaRule treatment recovers much of the loss. The task-level analysis shows that MetaRule improves all 11 tasks harmed by eveolving harness. The paper also supports with detailed execution traces and matched comparisons.

Critical weaknesses
  1. The experiments do not convincingly establish the value of persistent procedures.

The learned procedures are seem generic, consisting of operations like verifying givens, deriving independently, and preserving supported work. These could just be incorporated directly into a static system prompt or selected using simple predefined triggers. None of the five positive procedure candidates pass admission in the audited GPQA run, and the isolated persistence ablations do not show statistically reliable improvements.

  1. The evaluation does not separate the proposed mechanism from baseline degradation and additional inference.

The strongest improvement is on HMMT, where the evolving T1 baseline performs considerably worse than the original B0 baseline (35.83% versus 51.67% Avg@5). MetaRule recovers this degradation but improves over B0 by only 5.83 percentage points, a difference that is not statistically significant (p = 0.2585). One comparison could be against a harness that selectively rejects harmful updates(?). Additionally, MetaRule introduces diagnosis and revision steps that provide additional opportunities for reasoning. Without an inference-budget-matched self-verification baseline, we cannot tell whether the improvements are from procedural learning or from additional verification and reasoning. This isn't entirely a fundamental problem but clarity here will help.

  1. Limited generalization across agentic tasks.

The paper motivates MetaRule as a general framework for evolving LLM-agent harnesses but the evaluation is restricted to multiple-choice science questions and competition mathematics problems. These tasks have relatively simple execution structures and well-defined answers, unlike long-horizon agentic tasks involving tool calls, changing environments, and multi-step workflows. I believe the current benchmarks are not enough especially given how capable even the open source small models have become at these.

Questions for authors
  1. How would MetaRule compare against a fixed harness containing the same five operation primitives, either applied unconditionally or selected through the existing failure triggers?

  2. Could the authors compare MetaRule against T1 with an equivalent inference budget allocated to additional reasoning or self-verification?

  3. Why are rejected positive procedures allowed to produce persistent reverse rules without satisfying the same efficacy requirements?

7 comments
AuthorsOpenReview ↗

Response to Reviewer y17j (part 1/6) W1.1 / Q1 part1

Thank you for proposing concrete fixed-check, self-verification, and selective-update controls. You also ask whether the method extends to repository tasks. On HMMT, MetaRule improves plurality-answer reliability across three trajectories; on SWE-bench Verified Mini, it produces the highest attempt-level test-verified repair rate among the compared arms. We address the two settings separately below, since they measure different outcomes.

Metrics and campaigns. Avg@5 is per-attempt verifier accuracy; HMMT Cons@5 is plurality-answer accuracy; binary Pass@5 and Best@5 coincide. All HMMT task execution uses Haiku 4.5. The paper trajectory used GPT-5.5 for isolated diagnosis; its unavailability through our rebuttal access led us to use Haiku 4.5 for diagnosis in the two added campaigns. Individual results and mean ± sample SD describe these three campaigns, including this configuration change. B0/T1 are paper references.

W1 / Q1: Persistent Procedures

W1.1 / Q1: Fixed-Policy Controls

You proposed static prompts, predefined triggers, fixed checks, and extra reasoning as alternatives. We turn each into an executable control below; the main comparison is whether learned procedure selection improves the reliability of the answer selected from repeated attempts.

The five registered primitives implement your “verifying givens, deriving independently, and preserving supported work” suggestions, together with criterion separation and conflict resolution:

Registered operationConcrete action in a check
Check the givensRe-read the task statement and check that the candidate uses the stated facts and constraints and follows the requested output format.
Derive independentlyDerive a fresh solution directly from the task evidence.
Check criteria separatelyEvaluate every required criterion independently and check joint satisfaction.
Resolve conflictsLocate inconsistent claims or steps, then reconcile them against task evidence before closure.
Preserve supported workKeep conclusions supported by the evidence and revise only unsupported or conflicting parts.

The predefined trigger map is fixed before evaluation:

Fixed failure triggerAssigned operations
Choice mapping or missing derivationCheck the givens; independent derivation
Conflated criteriaCheck the givens; check criteria separately
Format errorCheck the givens; preserve supported work
Reasoning contradictionCheck the givens; resolve conflicts

The F-family labels distinguish the controls below. Their performance branches use the seed-42 frozen iteration-5 state and 24 HMMT held-out tasks × five generations, with Haiku 4.5 at medium effort, zero tools, the official verifier, and shared configured caps. The table reports four-metric outcomes.

For your two specific alternatives, checks applied “unconditionally” and checks selected by “predefined triggers,” the F-family uses these settings:

Family / armSettingTask-stage sequenceContrast being tested
F1 Fixed-unconditionalApply all five operations at the checking stage, without failure-specific selection.Initial response → diagnosis → all five checks → revision → typed closure.Whether a uniform checklist suffices within the repair pipeline.
F1 Fixed-triggeredAfter diagnosis, use the frozen mapping above to select two operations for the reported failure type.Initial response → diagnosis → two assigned checks → revision → typed closure.Performance of predefined trigger-conditioned checks.
F2 Fixed-checkPreserve the frozen parent state and diagnosis–revision–closure pipeline, but replace persistent procedure lookup with the predefined trigger-to-check mapping.Diagnosis → selected countercheck → one revision opportunity → typed closure.Learned persistent procedures versus fixed checks under the shared task pipeline.
F2 Generic-reasoningUse the same frozen state and pipeline as F2 Fixed-check, but replace the failure-specific check with a general re-derive-and-inspect instruction.Diagnosis → generic re-derivation and answer inspection → one revision opportunity → typed closure.Failure-specific checks versus generic extra reasoning in the pipeline.
F4 Static-promptPlace all five operations in every initial prompt; disable post-diagnosis revision.Initial prompt with five operations → response; record diagnosis telemetry.Whether the primitives suffice as an initial static prompt.

F1 compares trigger-selected and unconditional operations within the repair pipeline. F2 is an earlier, separately sampled fixed-check/generic-reasoning campaign from the same parent state; its fixed-check treatment uses the same mapping as F1 Fixed-triggered. F4 tests initial-prompt injection without post-diagnosis repair. F3 is the trace audit described under Q3 below.

AuthorsOpenReview ↗

Response to Reviewer y17j (part 2/6) W1.1/ Q1 part2, W1.2/ Q3, W1.3/ W2.2 part1

The performance outcomes are:

Frozen-state branchAvg@5Pass@5Cons@5Best@5
F1 Fixed-triggered40.00%70.83%37.50%70.83%
F1 Fixed-unconditional44.17%75.00%41.67%75.00%
F2 Fixed-check39.17%70.83%41.67%70.83%
F2 Generic-reasoning50.00%79.17%50.00%79.17%
F4 Static-prompt45.83%83.33%45.83%83.33%
B0 (paper K=5 reference)51.67%79.17%50.00%79.17%
MetaRule (three trajectories, mean ± SD)54.44% ± 2.68%79.17% ± 7.22%62.50% ± 4.17%79.17% ± 7.22%

Across the reported F-family campaigns, Full has higher Avg@5 and Cons@5 than these fixed policies. This advantage covers correct individual attempts and the correctness of the plurality answer. F1/F2 test predefined checks and F4 tests static-prompt injection; the later A/B/C replay compares deployed checking policies within each frozen parent.

W1.2 / Q3: Reverse-Rule Case

You note that “None of the five positive procedure candidates pass admission in the audited GPQA run.” Section 3.4 describes how rejected candidates can still inform later behavior through a distinct precautionary artifact. A positive procedure proposes behavior for reuse and must pass paired efficacy admission; a reverse rule instead blocks blind reuse of a failed parent procedure under a defined trigger. The artifact stores its source gate, parent candidate, trigger, permitted operations, capability scope, and expiry.

F3 is an offline trace audit of candidate → gate → reverse artifact → execution. In Appendix D.1, the parent specifies checking givens, independent derivation, and preserving supported work; positive admission fails for insufficient execution coverage. The reverse requires the first two operations under missing derivation. Its compiled instruction states: “Use only the public givens and execute every countercheck before changing the supported conclusion.” It also instructs against shortcut reuse of the rejected parent. Rejecting the parent controls positive-program inheritance; publishing the reverse creates a trigger-conditioned checking obligation, enforced through required-check closure. The later Biology trace executes those checks, changes B→A, and passes the official verifier. For your “same efficacy requirements” question, positive procedures receive paired efficacy admission; reverse artifacts receive precautionary admission based on provenance, scope, expiry, and executable-contract validity. F3 establishes execution, while dispatch and persistence comparisons evaluate utility.

W1.3 / W2.2: Replication and Persistence

We additionally examine repeatability across training trajectories. Each trajectory exceeds the paper B0 Avg@5 and Cons@5 references; the table reports the run-level spread. The original task-paired sign-flip tests are p=0.0007 against T1 and p=0.2585 against B0.

HMMT treatmentAvg@5Pass@5Cons@5Best@5
T1 (paper)35.83%58.33%41.67%58.33%
B0 (paper K=5 reference)51.67%79.17%50.00%79.17%
MetaRule seed 42 (paper trajectory)57.50%87.50%66.67%87.50%
MetaRule seed 2026111352.50%75.00%62.50%75.00%
MetaRule seed 2026111453.33%75.00%58.33%75.00%
MetaRule (three trajectories, mean ± SD)54.44% ± 2.68%79.17% ± 7.22%62.50% ± 4.17%79.17% ± 7.22%

All rows use the same 24-task × five-generation denominator, with every generation included in the reported aggregate. The three-run pattern is higher than B0 on both per-attempt success and plurality-answer accuracy, showing that correctness appears more often and is more likely to survive aggregation. The original task-paired B0 test remains p=0.2585p=0.2585; the additional states strengthen the repeatability description but are not additional paired B0 runs.

Two earlier controls add mechanism-level context. With the learned state fixed, dispatching a retained rule on the same 24 HMMT held-out tasks with three generations raises Avg@3 from 43.06% to 51.39% (+8.33 points; paired p=.3335). Appendix-G Full–Shadow varies downstream publication and dispatch while preserving validation and artifact history: pair 2 yields +4.17 points across all tasks (p=.4575), and cross-type differences across three pairs are +1.67, +6.67, and −1.67 points.

AuthorsOpenReview ↗

Response to Reviewer y17j (part 3/6) W1.3/ W2.2 part2, W2.1

A/B/C three-arm replay. Each arm uses a frozen iteration-5 parent, read-only evaluation, Haiku 4.5 at medium effort, zero tools, the official verifier, and the same task set, under shared configured caps of 20 calls and 1M output tokens per task.

ArmChecking policyComparison role
A: Published-rule dispatchMatch a retained rule's trigger and scope, execute its required checks, and verify the closure contract.A–B compares learned retained contracts with a predefined map; A–C compares their deployment with general repair.
B: Fixed trigger checkSelect operations from a predefined failure-to-check map; then revise and close.B–C tests adding predefined checks to general repair.
C: Rule-free pipelineKeep diagnosis, revision, and closure, with no rule lookup or rule-specific required checks.General-repair reference; the model can still inspect its answer.

The reconstructed five-replicate records cover 23 complete common tasks across three states; one task lacks four A generations in seed 20261113. The 24-task check uses the complete states, seeds 42 and 20261114. Entries are mean ± sample SD across states.

Evaluation scopeArmAvg@5Pass@5Cons@5Best@5
23 common tasks; three statesA: Published rule59.71% ± 1.81%85.51% ± 2.51%69.57% ± 4.35%85.51% ± 2.51%
23 common tasks; three statesB: Fixed check58.26% ± 3.14%81.16% ± 2.51%62.32% ± 10.94%81.16% ± 2.51%
23 common tasks; three statesC: Rule-free57.68% ± 3.05%82.61% ± 4.35%63.77% ± 6.64%82.61% ± 4.35%
24 tasks; two complete statesA: Published rule56.67% ± 1.18%83.33% ± 5.89%66.67% ± 5.89%83.33% ± 5.89%
24 tasks; two complete statesB: Fixed check54.17% ± 1.18%77.08% ± 2.95%54.17% ± 5.89%77.08% ± 2.95%
24 tasks; two complete statesC: Rule-free57.50% ± 2.36%85.42% ± 2.95%60.42% ± 8.84%85.42% ± 2.95%

For three states, descriptive post-hoc exact two-sided task-paired sign-flip tests give A–B/A–C p=.6719/.4927 for Avg@5 and .2344/.2188 for Cons@5. Testing averages generations and states within each task. No contrast reaches .05 after Holm correction across three contrasts per metric. The two scopes agree on A's higher plurality-answer accuracy; the full-24-task comparison places C higher on per-attempt accuracy and coverage.

This addresses your question about putting the five primitives in a static prompt or selecting them with predefined triggers: A retains an explicit trigger, scope, and execution contract, while B fixes the mapping and C retains general repair. A's higher observed plurality accuracy in both scopes is the most consistent outcome of this comparison. The paired tests report its statistical strength.

In the geometry problem with answer 7/18, a retained missing-derivation rule in A requires checking the givens and independent derivation; closure records those checks and the official verifier accepts the revision. This trace shows how a learned contract governs execution using the existing primitive vocabulary.

W2: Baseline and Inference

W2.1: Task-Level Recovery

We additionally examine recovery on the 11 tasks harmed by T1 in the original paper comparison, using per-task five-generation accuracy. The paper trajectory improves all 11 and restores eight to at least B0. The added seed20261113/20261114 trajectories improve 10/11 and 8/11, with the remaining tasks tying T1, and restore 8/11 and 7/11 to at least B0. Every trajectory therefore recovers most affected tasks to the original harness level, with no further decline relative to T1 on this fixed subset.

Two task traces make the benefit concrete. For the answer "7/18", MetaRule produces four correct generations out of five, compared with B0's one and T1's zero: the correct solution becomes the plurality answer. For the cross-type number-theory answer "279", MetaRule solves the task twice, extending coverage beyond both references' zero successes in five attempts. Together, these examples show more reproducible correct solutions and additional solved tasks.

These task transitions locate the recovery: correct generations recur more often and can form the plurality under the shared aggregation rule. The repeatability response above reports the corresponding task-paired statistics.

Reviewer_UZZGOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes MetaRule, a framework for cross-domain harness evolution that converts execution traces into persistent, executable procedural rules. The authors evaluate the framework on GPQA-Diamond, AIME 2025, and HMMT, reporting improvements over native harness evolution, particularly on cross-domain and cross-type evaluation splits.

Critical strengths

None.

Critical weaknesses
  • Unclear methodological novelty relative to existing harness evolution and procedural-memory approaches.

The paper motivates a transactional mechanism for transferring useful procedural knowledge across tasks and domains. However, it does not convincingly establish why such a mechanism is necessary or how it differs substantially from existing approaches.

Several closely related studies are missing from the bibliography:

Skill-Pro [1] learns reusable, executable procedural skills with activation, execution, and termination conditions, together with verification gates for reliable reuse.

HarnessEvolve [2] performs evidence-driven harness evolution with separate optimization and gating mechanisms intended to prevent harmful updates and catastrophic forgetting.

Agentic Harness Engineering (AHE) [3] uses execution evidence to evolve harness components and investigates the transferability and attribution of improvements.

The paper also lacks direct comparisons with these recent harness-evolution approaches. It remains unclear whether the proposed transactional architecture provides meaningful advantages over simpler extensions of existing methods, such as incorporating cross-domain examples into an evolving procedure bank.

[1] Mi et al. Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents. ICML 2026. https://arxiv.org/abs/2602.01869

[2] Jiang et al. HarnessEvolve: Learning from Reference Trajectories for Reliable Agent Self-Evolution. 2026. https://arxiv.org/abs/2609.00829

[3] Lin et al. Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses. 2026. https://arxiv.org/abs/2604.25850

  • Excessive architectural complexity without sufficient component-level attribution.

Figure 1 introduces numerous interacting mechanisms, including candidate distillation, paired admission gates, failure analysis, reverse-rule generation, and successor-state publication.

It remains unclear which components are necessary for the reported improvements and which merely introduce additional engineering complexity. In particular, the paper's mechanism-specific analyses do not establish a clear causal benefit from persistent procedural transfer itself.

Consequently, it is difficult to distinguish improvements attributable to the proposed mechanism from those arising from additional refinement steps, heuristic rules, or increased computational expenditure.

  • Performance improvements are confounded by substantially greater computational expenditure.

The comparison is not budget-matched. Appendix G reports 7,027, 12,189, and 14,593 tokens per selected GPQA-Diamond outcome for B0, T1, and MetaRule, respectively.

Thus, MetaRule uses approximately twice as many tokens as B0 and approximately 20% more than T1. These measurements also do not capture the complete cost of harness evolution and evaluation.

The improvements could therefore arise partly from additional inference, revision, or computation rather than the proposed procedural-learning mechanism.

Without compute-matched comparisons, it is difficult to determine whether the additional architectural complexity and computational cost are justified.

Questions for authors
  • Novelty and related work: How does MetaRule fundamentally differ from existing persistent procedural-memory and harness-evolution approaches, particularly Skill-Pro [1], HarnessEvolve [2], and AHE [3]? What limitations of these approaches necessitate the proposed transactional architecture? Could a simpler harness-evolution approach incorporating cross-domain procedural examples achieve similar improvements?

  • Compute-matched comparison: How does MetaRule perform when B0 and T1 receive equivalent inference-token budgets, revision opportunities, and training/evolution compute? What are the complete costs of diagnosis, candidate generation, and iterative harness updates?

8 comments
AuthorsOpenReview ↗

Response to Reviewer UZZG (part 1/6) W1.1/Q1.1 , W1.2/Q1.2

Thank you for pointing us to recent harness-evolution work and asking for direct comparisons with simpler procedure-bank extensions. MetaRule connects execution evidence to future behavior through explicit admission and publication rules. Across the paper run and two supplementary HMMT campaigns, it reaches 54.44% ± 2.68% Avg@5 and 62.50% ± 4.17% Cons@5. We first distinguish the designs, then compare official-code-based HMMT adaptations and report component and compute evidence.

Metrics and campaigns. Avg@5 is per-attempt verifier accuracy; HMMT Cons@5 is plurality-answer accuracy; binary Pass@5 and Best@5 coincide. All HMMT task execution uses Haiku 4.5. The paper trajectory used GPT-5.5 for isolated diagnosis; its unavailability through our rebuttal access led us to use Haiku 4.5 for diagnosis in the two added campaigns. Individual results and mean ± sample SD describe these three campaigns, including this configuration change. B0/T1 are paper references.

W1 / Q1: Methodological Positioning

W1.1 / Q1.1: Design Differentiation and Component Necessity

You ask what is methodologically distinct about MetaRule relative to recent harness-evolution methods. The design specifies three linked decisions for every reusable procedure: what evidence may inform it, how it is admitted, and when it becomes active. Diagnosis is isolated from the current answer and private gold; bounded revision stays in the original task thread; and admitted procedures or typed precautionary artifacts are published only at the batch barrier. This makes each change's evidence, scope, and activation time inspectable.

MethodMethodological focusAuthor-stated limitation or future directionMetaRule evidence relevant to that direction
Skill-Pro [1]Converts experience into reusable executable skills with activation and verification conditions.Its conclusion identifies integration of implicit execution modules as future work.MetaRule specifies an answer-isolated diagnosis, typed closure, paired admission, and batch-boundary publication. These are complementary lifecycle mechanisms; the HMMT adaptation executes its native skill selection, semantic evolution, and clipped-surrogate acceptance on a shared task interface.
HarnessEvolve [2]Uses reference trajectories, quality/performance gates, and held-out selection to address credit assignment, shortcut learning, and forgetting.The paper presents these as challenges addressed by its method; it does not state them as unresolved limitations. We could not locate a public implementation for a code-based reproduction.MetaRule makes evidence source, artifact scope, and activation time explicit through isolated diagnosis, paired admission, and frozen successor-state publication. The comparison here is paper-level, not an executed HarnessEvolve baseline.
Agentic Harness Engineering [3]Evolves coding-agent harness components from execution evidence.AHE discusses broader repository/language evaluation, operating-point coupling, and self-modification guardrails as open directions.Typed provenance, scope, expiry, and closure contracts make each reusable artifact’s execution boundary inspectable. Checkpoint measurements separately expose performance–cost operating points.

MetaRule's methodological contribution is the transaction boundary linking answer-free evidence to a typed reusable artifact: diagnosis isolates evidence, admission evaluates a proposed change, and publication controls later activation. Component probes test replacement policies; evolving-bank controls test the simpler alternative in your review. The single-trajectory probes measure sensitivity to those replacements, rather than establish every component's necessity.

W1.2 / Q1.2: Related Work Update

We have updated the related work and references to include Skill-Pro, HarnessEvolve, and Agentic Harness Engineering, clarifying their relationship to MetaRule's transactional lifecycle. Additions are marked in red and revised text in blue in the updated PDF. We also added their design comparison above and integrated the public Skill-Pro and AHE implementations with HMMT. Both have completed training and frozen K=5 evaluation (120 outcomes each). Both adaptations use the same HMMT split, Haiku task model, official verifier, and held-out protocol. Skill-Pro additionally uses Qwen3-8B for semantic evolution and conditional-likelihood scoring. We could not locate public HarnessEvolve code as of November 14, 2026, so HarnessEvolve is covered by the design comparison above.

[1] Mi et al. Skill-Pro. https://arxiv.org/abs/2602.01869

[2] Jiang et al. HarnessEvolve. https://arxiv.org/abs/2609.00829

[3] Lin et al. Agentic Harness Engineering. https://arxiv.org/abs/2604.25850

AuthorsOpenReview ↗

Response to Reviewer UZZG (part 2/6) W1.3/Q1.3, W1.4/Q1.4

W1.3 / Q1.3: Evolving-Bank Control

Your proposed “simpler extensions” are “cross-domain examples ... in an evolving procedure bank.” T1-lite implements that alternative: it retains train-only cross-task retrieval, bank updates, and the task-side case-controller, while disabling global controller evolution, controller canaries, paired admission gates, and distillation. After five iterations on 39 HMMT training tasks, its frozen bank has 219 entries (24 train-only bootstrap and 195 formal training exposures). On the same 24 held-out tasks, it obtains 50.00% Acc@1, including 6/12 same-type and 6/12 cross-type tasks. Each task is evaluated once; the next comparison reports five-generation metrics.

  • T1: 35.83% Avg@5; 41.67% Cons@5.
  • B0: 51.67% Avg@5; 50.00% Cons@5.
  • T1-lite: 50.00% Acc@1;
  • MetaRule: 54.44% ± 2.68% Avg@5; 62.50% ± 4.17% Cons@5 across three training trajectories.

T1-lite tests example transfer through a growing bank while keeping the global harness policy fixed. Its task-side case-controller selects retrieved procedural guidance for the current task. Same-type tasks come from the registered training types; the cross-type set contains Number Theory tasks. The single-attempt result shows usable transfer in both strata; the official-code-based evaluation below provides the repeated-generation comparison.

W1.4 / Q1.4: Official-Code-Based HMMT Adaptations

To address your request for “direct comparisons with these recent harness-evolution approaches,” we downloaded the public Skill-Pro and Agentic Harness Engineering implementations and connected their native mechanisms to our HMMT environment. Both use the ordered 39-train/24-held-out split, five training iterations, Claude Haiku 4.5 for task execution, zero task-side tools, and the official verifier. Each complete solution is one terminal environment action, with a 32,000-output-token cap including a 10,000-token thinking budget. The terminal iteration-5 state is frozen before five read-only generations per held-out task. The adaptation settings follow.

BaselineNative mechanisms executedHMMT adaptation and execution evidence
Skill-ProSix initial skills with initiation/policy/termination definitions; native LLM skill selection with exploration and a NONE option; semantic-gradient evolution; Best-of-3 candidate generation; PPO clipped-surrogate acceptance; native pool maintenance.Haiku solves tasks and selects skills, including the native abstention option; Qwen3-8B generates updates and scores conditional action likelihoods. Five rounds use decaying exploration, a six-experience evolution threshold, at most two parents per round, PPO clipping 0.2, and acceptance margin zero. Four updates are accepted; the final pool contains ten skills.
AHEOriginal NexAU task agent, evolution prompt and guide, evaluation-query builder, evolution agent, and file-editing tools. The optimizer reads training traces and edits the harness consumed by subsequent iterations.Haiku performs task execution and evolution. The optimizer reads training evidence and edits the evolving workspace, with task tools and model/budget fixed. Each of five optimizer rounds reaches its configured 20-step limit. Updates affect the system prompt and long-term memory; upstream stability and attribution inputs are unpopulated.
HMMT armAvg@5Pass@5Cons@5Best@5
Skill-Pro, official-code-based HMMT adaptation45.83%83.33%58.33%83.33%
AHE, 20-step HMMT adaptation52.50%75.00%62.50%75.00%
T1 (paper K=5)35.83%58.33%41.67%58.33%
B0 (paper K=5)51.67%79.17%50.00%79.17%
MetaRule Full (three trajectories)54.44% ± 2.68%79.17% ± 7.22%62.50% ± 4.17%79.17% ± 7.22%

MetaRule combines reliable answer production with task coverage. Its higher Avg@5 than both adaptations means correct solutions occur more often per generation. Against Skill-Pro, higher Cons@5 extends this advantage to the answer returned by plurality voting; Skill-Pro has higher five-attempt coverage. Against AHE, MetaRule maintains the same mean plurality accuracy while covering more tasks within five attempts. These complementary measures distinguish finding a successful solution from consistently returning one. This descriptively compares three MetaRule campaigns with one trajectory per baseline under the stated adaptations.

One accepted Skill-Pro update checks derived values against the original constraints before finalization. AHE updates add problem-solving strategies and a verification checklist to the prompt and long-term memory. These adaptations evaluate native skill selection/evolution and harness editing; T1-lite separately tests the simpler cross-domain example bank.

AuthorsOpenReview ↗

Response to Reviewer UZZG (part 3/6) W2.1, W2.2 part1

W2: Component Attribution

W2.1: Component Replacement

We updated Fig. 1 in the revised PDF to make the diagnosis, revision, closure, admission, and publication mechanisms easier to follow visually. Typed closure checks the required operation/evidence contract before accepting a revision; paired admission compares a candidate with its off arm on reserved training tasks; publication activates the artifact in the next frozen state. To connect that overview to the component experiments, we compare three targeted replacements under a shared HMMT setup: five training iterations, the 39/24 split, task and gate schedules, Haiku 4.5 at medium effort for execution and isolated diagnosis, the same verifier and retry rules, zero tools, and per-task caps of 20 calls and 1M output tokens. Each intervention changes one lifecycle decision, including the inputs it supplies downstream; every held-out evaluation is read-only, with five generations per task:

ArmExecuted settingLifecycle boundary
C-DISTILLChoose the first candidate in lexicographic candidate-ID order from the same finite candidate set, replacing evidence-conditioned candidate selection.Candidate distillation.
C-GATEAccept iff the paired mean task delta is non-negative and there is no invalid regression.Full admission contract versus a fixed utility/regression rule.
C-FAILPass an empty diagnosis downstream while retaining the original diagnosis for analysis.Failure analysis.
MetaRule FullRun the complete diagnosis–revision–closure–admission–publication lifecycle.Integrated treatment.

Full’s admission also checks execution/evidence coverage, repeated task harm, and the configured uncertainty bound. C-FAIL supplies an empty diagnosis downstream to repair and learning; the original diagnosis is retained in the audit record. Each completed component probe uses one training trajectory. The Full row reports the paper trajectory together with the two added training trajectories, summarized as mean ± sample SD; the probe rows are individual run results. The component outcomes are:

HMMT component comparisonAvg@5Pass@5Cons@5Best@5
C-DISTILL54.17%79.17%66.67%79.17%
C-GATE52.50%75.00%54.17%75.00%
C-FAIL55.83%83.33%62.50%83.33%
T1 (paper)35.83%58.33%41.67%58.33%
B0 (paper K=5 reference)51.67%79.17%50.00%79.17%
MetaRule (three trajectories, mean ± SD)54.44% ± 2.68%79.17% ± 7.22%62.50% ± 4.17%79.17% ± 7.22%

The component profiles make candidate selection, admission, and failure analysis separately inspectable. Full combines explicit control of answer replacement, candidate admission, and future activation in one lifecycle, with higher observed Avg@5 and Cons@5 than the paper B0 reference. The single-campaign probes map sensitivity to alternative policies; the next controls examine checking and rule dispatch.

W2.2: Persistence Attribution

Your persistence question calls for alternatives that retain repair while changing the source of the checks. F2 Fixed-check replaces learned rule lookup with a predefined map: choice/missing derivation → check givens and derive independently; conflated criteria → check givens and separate criteria; contradiction → check givens and resolve conflicts; format error → check givens and preserve supported work. F2 Generic-reasoning instead requests general re-derivation and answer inspection. Both retain diagnosis, one revision opportunity, and typed closure, using the seed-42 frozen state, 24 tasks × five generations, Haiku 4.5 for execution and diagnosis, zero tools, and the official verifier.

Same-pipeline controlAvg@5Pass@5Cons@5Best@5
F2 Fixed-check39.17%70.83%41.67%70.83%
F2 Generic-reasoning50.00%79.17%50.00%79.17%
MetaRule seed 42 (paper trajectory)57.50%87.50%66.67%87.50%
B0 (paper reference)51.67%79.17%50.00%79.17%
MetaRule Full (three trajectories)54.44% ± 2.68%79.17% ± 7.22%62.50% ± 4.17%79.17% ± 7.22%

Full’s advantage over these fixed and generic controls appears in both producing correct attempts and selecting a correct plurality answer. Configured caps are shared; seed-42 Full uses 8.807M held-out tokens, versus 6.271M for Fixed-check and 5.760M recorded for Generic-reasoning, whose input meter is missing for one call. This is a measured performance–cost comparison. A/B/C below extends the rule-policy comparison to three frozen states.

Reviewer_NaxUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MetaRule proposes a controlled process for turning an agent’s experience into reusable procedural rules. It separates diagnosis, answer revision, validation, and publication of rules for later tasks. The safeguards against exposing answers during diagnosis are thoughtful, and the task-level analysis is more informative than reporting aggregate accuracy alone.

Critical strengths

The idea of converting exec traces into structured programs is novel and may be useful, which enables multi-turn RL to benefit from richer environment feedback. The framework could be useful while working with open source frameworks like https://github.com/harbor-framework/harbor and propose some meaningful engineering insights.

Critical weaknesses

The experiments support the complete system more clearly than the claimed benefit of learning persistent procedures. The comparisons designed to isolate rule execution and persistence remain inconclusive. All five positive candidates in the audited GPQA run were rejected, while the transferred rules were precautionary checks assembled from a small predefined set of operations. It is therefore unclear how much the learned history contributes beyond a fixed checking policy.

The strongest HMMT improvement is largely a recovery from damage introduced by native evolution. MetaRule improves average accuracy by 21.67 points over that degraded baseline, but only 5.83 points over the original system, with the latter difference not statistically clear. Each primary treatment also has only one training trajectory, and the GPQA comparison is not compute-matched. These limitations make it difficult to separate a repeatable learning benefit from extra checking and additional computation.

Questions for authors

Would the result hold across independent training runs and against a fixed checking policy with the same inference budget? What evidence shows that retaining learned rules helps beyond applying the same validation steps without persistence?

6 comments
AuthorsOpenReview ↗

Response to Reviewer NaxU (Part 1/5) W1.1

Thank you for highlighting executable procedures and task-level analysis. Your central question is whether learned procedures add value beyond the complete system and its additional computation. Across the paper HMMT run and two supplementary campaigns, MetaRule reaches 54.44% ± 2.68% Avg@5 and 62.50% ± 4.17% Cons@5. We answer that question with component replacements, same-pipeline fixed-check controls, frozen-state rule-policy replays, and checkpoint-level token accounting.

Metrics and campaigns. Avg@5 is per-attempt verifier accuracy; HMMT Cons@5 is plurality-answer accuracy; binary Pass@5 and Best@5 coincide. All HMMT task execution uses Haiku 4.5. The paper trajectory used GPT-5.5 for isolated diagnosis; its unavailability through our rebuttal access led us to use Haiku 4.5 for diagnosis in the two added campaigns. Individual results and mean ± sample SD describe these three campaigns, including this configuration change. B0/T1 are paper references.

W1: Persistent Procedure Value

W1.1: Component Probes

You asked whether the evidence supports the complete system more clearly than the benefit of learning persistent procedures. We address that question by replacing one lifecycle decision at a time under a shared component-probe configuration: five training iterations, the 39/24 train/held-out split, task and gate schedules, Claude Haiku 4.5 at medium effort, provider/verifier/retry rules, zero tools, and the same configured per-task caps. Each arm freezes its iteration-5 state before five read-only generations per held-out task.

ArmExecuted settingDecision tested
B0Native MemoHarness without procedure-bank or controller evolutionOriginal harness reference.
T1Native procedure-bank and controller evolutionNative evolution reference.
C-DISTILLChoose the first candidate in lexicographic candidate-ID order from the same finite candidate set, replacing evidence-conditioned candidate selection.Candidate selection and distillation.
C-GATEAccept iff the paired mean task delta is non-negative and there is no invalid regression.Full admission contract versus a fixed utility/regression rule.
C-FAILReplace the diagnosis passed to later stages with an empty diagnosis, while keeping the original diagnosis available for analysis.Failure-analysis contribution.
MetaRule FullExecute the complete diagnosis–revision–closure–admission–publication lifecycle.Integrated treatment.

Full’s admission also checks execution/evidence coverage, repeated task harm, and the configured uncertainty bound. C-FAIL supplies an empty diagnosis downstream to repair and learning; the original diagnosis is retained in the audit record. Each completed component probe uses one training trajectory. MetaRule Full reports the paper trajectory together with the two added trajectories, summarized as mean ± sample SD; the probe rows are single-run observations. The component profile is:

HMMT component comparisonAvg@5Pass@5Cons@5Best@5
C-DISTILL54.17%79.17%66.67%79.17%
C-GATE52.50%75.00%54.17%75.00%
C-FAIL55.83%83.33%62.50%83.33%
T1 (paper)35.83%58.33%41.67%58.33%
B0 (paper K=5 reference)51.67%79.17%50.00%79.17%
MetaRule Full (three trajectories, mean ± SD)54.44% ± 2.68%79.17% ± 7.22%62.50% ± 4.17%79.17% ± 7.22%

The component profiles test candidate selection, admission, and downstream diagnosis as distinct lifecycle decisions. Full combines those decisions with higher observed per-attempt and plurality-selected accuracy than the paper B0 reference. The probes map sensitivity to the replacement policies; the retained-rule interventions below address execution and publication.

AuthorsOpenReview ↗

Response to Reviewer NaxU (Part2/5) W1.2&Q2,W1.3

W1.2&Q2: Persistence Attribution

You also ask whether the benefit comes from “learning persistent procedures”. The earlier frozen-state dispatch test toggles a retained rule in the same state, raising Avg@3 from 43.06% to 51.39% (p=.3335p=.3335). Appendix-G Full–Shadow preserves validation and artifact history while disabling executable transfer to successor states; Pair 2 gives +4.17 points overall (p=.4575p=.4575). The A/B/C replay below adds predefined-check and general-repair alternatives to examine the deployed checking policy.

A/B/C three-arm replay. Each arm uses a frozen iteration-5 parent, read-only evaluation, Haiku 4.5 at medium effort, zero tools, the official verifier, and the same task set, under shared configured caps of 20 calls and 1M output tokens per task.

ArmChecking policyComparison role
A: Published-rule dispatchMatch a retained rule's trigger and scope, execute its required checks, and verify the closure contract.A–B compares learned retained contracts with a predefined map; A–C compares their deployment with general repair.
B: Fixed trigger checkSelect operations from a predefined failure-to-check map; then revise and close.B–C tests adding predefined checks to general repair.
C: Rule-free pipelineKeep diagnosis, revision, and closure, with no rule lookup or rule-specific required checks.General-repair reference; the model can still inspect its answer.

The reconstructed five-replicate records cover 23 complete common tasks across three states; one task lacks four A generations in one seed. The 24-task check uses the complete states in other two seeds. Entries are mean ± sample SD across states.

Evaluation scopeArmAvg@5Pass@5Cons@5Best@5
23 common tasks; three statesA: Published rule59.71% ± 1.81%85.51% ± 2.51%69.57% ± 4.35%85.51% ± 2.51%
23 common tasks; three statesB: Fixed check58.26% ± 3.14%81.16% ± 2.51%62.32% ± 10.94%81.16% ± 2.51%
23 common tasks; three statesC: Rule-free57.68% ± 3.05%82.61% ± 4.35%63.77% ± 6.64%82.61% ± 4.35%
24 tasks; two complete statesA: Published rule56.67% ± 1.18%83.33% ± 5.89%66.67% ± 5.89%83.33% ± 5.89%
24 tasks; two complete statesB: Fixed check54.17% ± 1.18%77.08% ± 2.95%54.17% ± 5.89%77.08% ± 2.95%
24 tasks; two complete statesC: Rule-free57.50% ± 2.36%85.42% ± 2.95%60.42% ± 8.84%85.42% ± 2.95%

For three states, descriptive post-hoc exact two-sided task-paired sign-flip tests give A–B/A–C p=.6719/.4927p=.6719/.4927 for Avg@5 and .2344/.2188 for Cons@5. Testing averages generations and states within each task. No contrast reaches .05 after Holm correction across three contrasts per metric. The two scopes agree on A's higher plurality-answer accuracy.

The clearest retained-rule signal is correct answers recurring often enough to form the plurality: A has the highest observed Cons@5 in both scopes. The plurality rule itself is unchanged across arms. This addresses your distinction between a capable repair pipeline and useful retained procedures, while the paired tests quantify the strength of the observed differences. A is the published-rule replay arm; the earlier primary Full results describe a separate evaluation campaign.

A concrete execution makes “retaining learned rules” explicit. On the geometry problem with answer "7/18", A matches a retained missing-derivation rule, executes checking the givens and independent derivation, and passes closure and the official verifier. The retained object binds existing primitives to a learned trigger, scope, provenance, and required-check contract.

W1.3: Reverse-Rule Case

You note that “All five positive candidates in the audited GPQA run were rejected.” Section 3.4 describes the corresponding precautionary branch: paired efficacy admission governs positive procedures, while a rejected candidate can yield a scoped reverse rule. Appendix D.1 makes the execution difference concrete: the positive candidate specifies checking givens, independent derivation, and preserving supported work; the gate rejects its admission for insufficient execution coverage. The published reverse artifact requires the first two operations under the missing-derivation trigger. Its compiled instruction states: “Use only the public givens and execute every countercheck before changing the supported conclusion.” It also instructs the agent against shortcut reuse of the rejected parent. Rejecting the parent controls positive-program inheritance; publishing the reverse creates a trigger-conditioned checking obligation for later tasks, enforced through the required-check closure contract. Parent, source gate, scope, and expiry remain attached to that obligation. Frozen dispatch and Full–Shadow separately probe its downstream use and transfer.

AuthorsOpenReview ↗

Response to Reviewer NaxU (part 3/5) W1.4&Q1

W1.4 & Q1: Fixed Operations and Fixed-Policy Controls

You describe the transferred rules as “precautionary checks assembled from a small predefined set of operations” and ask how much learned history contributes beyond a fixed checking policy. We therefore make our five registered operations explicit and test them under fixed policies, then compare fixed checks and generic reasoning inside the shared diagnosis–revision–closure pipeline. The operation labels below are our implementation's primitive names:

Registered operationConcrete action in a check
Check the givensRe-read the task statement and check that the candidate uses the stated facts and constraints and follows the requested output format.
Independent derivationDerive a fresh solution directly from the task evidence.
Separate criteriaEvaluate every required criterion independently and check joint satisfaction.
Resolve conflictsLocate inconsistent claims or steps, then reconcile them against task evidence before closure.
Preserve supported workKeep conclusions supported by the evidence and revise only unsupported or conflicting parts.

The predefined trigger map is fixed before evaluation:

Fixed failure triggerAssigned operations
Choice or missing derivationCheck the givens, independent derivation
Conflated criteriaCheck the givens, separate criteria
Format errorCheck the givens, preserve supported work
Reasoning contradictionCheck the givens, resolve conflicts

To test whether the operation set alone accounts for the outcome, we execute three fixed-policy variants on the HMMT held-out protocol:

Fixed-policy armExecuted setting
F1 Fixed-triggeredAfter diagnosis, use the frozen failure-to-check map above to choose two operations, then revise and close.
F1 Fixed-unconditionalApply all five operations at the checking stage, without failure-specific selection, within diagnosis–revision–closure.
F4 Static-promptPut all five operations in every initial prompt; record diagnosis telemetry and disable post-diagnosis revision.

F1 compares operation selection at the checking stage; F4 moves the operations into the initial prompt and omits post-diagnosis repair. F2 below is an earlier, separately sampled campaign from the same seed-42 parent; F1 Fixed-triggered and F2 Fixed-check use the same mapping. The four-metric outcomes are:

HMMT fixed-policy comparisonAvg@5Pass@5Cons@5Best@5
F1 Fixed-triggered40.00%70.83%37.50%70.83%
F1 Fixed-unconditional44.17%75.00%41.67%75.00%
F4 Static-prompt45.83%83.33%45.83%83.33%
B0 (paper K=5 reference)51.67%79.17%50.00%79.17%
MetaRule Full (three trajectories, mean ± SD)54.44% ± 2.68%79.17% ± 7.22%62.50% ± 4.17%79.17% ± 7.22%

Across the reported fixed-operation campaigns, Full has higher Avg@5 and Cons@5 than these controls. The advantage concerns both correct individual generations and the correctness of the plurality-selected answer. These results compare fixed-trigger and initial-prompt uses of the primitives; F2 below tests fixed checks inside diagnosis–revision–closure, and the later A/B/C replay compares deployed policies within each frozen parent.

Same-pipeline F2 comparison. To address whether the fixed policy or extra reasoning within diagnosis–revision–closure explains the result, F2 evaluates two controls from the seed-42 frozen iteration-5 state. Fixed-check applies the frozen failure-to-check map; Generic-reasoning uses a general re-derive-and-inspect instruction. Both use the same 24 tasks × five generations, Haiku 4.5, verifier, task order, zero tools, and configured caps of 20 calls and 1M output tokens per held-out task. Provider receipts record realized token use.

Each F2 arm and each Full trajectory covers 120 held-out task runs. F2 reports the seed-42 controls; the Full cost is mean ± SD across three training trajectories.

Frozen-state controlAvg@5Pass@5Cons@5Best@5Held-out tokens (M)
F2 Fixed-check39.17%70.83%41.67%70.83%6.271
F2 Generic-reasoning50.00%79.17%50.00%79.17%5.760*
B0 (paper K=5 reference)51.67%79.17%50.00%79.17%3.688
T1 (paper K=5 reference)35.83%58.33%41.67%58.33%3.462
MetaRule seed 42 (paper trajectory)57.50%87.50%66.67%87.50%8.807
MetaRule (three trajectories, mean ± SD)54.44% ± 2.68%79.17% ± 7.22%62.50% ± 4.17%79.17% ± 7.22%6.383 ± 2.103

*The Generic-reasoning input-token meter is missing for one call; the total sums all recorded input and output usage. The fixed-check arm's output-token total reconciles all 510 calls.

Comments on the paper

1 comment

Summary of Reviews and Discussion

We thank all reviewers for their feedback. Below is a summary of reviews and discussion.

ReviewerScoreStrengths highlightedWeaknesses / QuestionsAuthor responseFurther reply?
NaxU032Recognized trace-to-program learning and task-level analysis.Persistent learning versus fixed checks/extra compute; repeatability.Added two HMMT trajectories, recovery analysis, fixed-policy controls, A/B/C replay, and cost accounting. Added runs improve/tie T1 on all 11 harmed tasks, restoring 8/11 and 7/11 to at least B0.No update reported.
UZZG011No critical strengths listed.Novelty/necessity, recent baselines, component attribution, compute.Updated related work; added public-code-based Skill-Pro/AHE HMMT adaptations (45.83%/52.50% Avg@5), T1-lite (50.00% Acc@1), component probes, and cost accounting. MetaRule Avg@5 is 54.44% ± 2.68%.Retained assessment; novelty and benefit over baselines remain concerns.
y17j222Praised the lifecycle, closure/correctness separation, and HMMT recovery.Fixed operations, selective rollback, and tool-using agentic tasks.Added F-family controls, H-Rollback (51.67% Avg@5; 66.67% Cons@5), and SWE-bench Verified Mini.No update reported.
UpEr213Recognized frozen states, scoped dispatch, delayed publication, diagnosis/answer separation, and HMMT recovery.Repeatability, fixed counterchecks, reverse-rule behavior, and closure verification.Added repeated campaigns and policy controls; clarified reverse-rule execution and structural versus semantic closure.Soundness raised to 3; no further questions; overall unchanged.

Overall Discussion

UpEr21 raised soundness to 3 (overall 3); UZZG01 retained 1. NaxU03 and y17j22 have not updated. Current score total: 8.

Key strengths highlighted by reviewers

  • Inspectable procedure learning: reviewers valued frozen states, scoped dispatch, delayed publication, and separating procedural completion from correctness.
  • HMMT recovery: the paper trajectory improves all 11 tasks harmed by T1 and restores 8/11 to at least B0. The two added trajectories improve 10/11 and 8/11, with all remaining tasks tying T1, and restore 8/11 and 7/11 to at least B0.
  • Repeated performance: three campaigns yield 54.44% ± 2.68% Avg@5, 79.17% ± 7.22% Pass@5, 62.50% ± 4.17% Cons@5, and 79.17% ± 7.22% Best@5. The paper used GPT-5.5 for diagnosis; added campaigns used Haiku 4.5.

Suggestions Focused On

  • Repeatability and held-out precision.
  • Persistent-rule value beyond fixed primitives/triggers, static prompts, generic reasoning, and rollback.
  • Recent harness-evolution methods and simpler evolving banks.
  • Inference and training/evolution costs.
  • Tool use, changing environments, and multi-step workflows.

How we addressed these suggestions

1. Added repository-repair evaluation. On a fixed 17-issue SWE-bench Verified Mini split, each arm receives five repair attempts per issue, scored by executable tests.

ArmAvg@5Pass@5Cons@5*Best@5
B029.41%41.18%29.41%41.18%
T138.82%58.82%47.06%58.82%
MetaRule42.35%52.94%47.06%52.94%

*Cons@5 is the fraction of issues with at least three test-passing attempts. MetaRule achieves the highest per-attempt repair accuracy and exceeds B0 on all four metrics. Across 85 attempts, repository-tool use is 24–147 calls (median 62), with file editing and test commands in every attempt. Contextual same-slice open-model records include Gemma-3-4B, Qwen3-1.7B, SWE-Llama-7B, and SWE-Lego-Qwen3-8B at 0/17, and Lingma variants at 0/17 and 1/17. Different model/agent configurations make these context, not controlled, comparisons. We cite Agentic CLEAR, SWEnergy, and HiSentinel on Mini’s use for multi-step agent evaluation.

2. Added mechanism controls and comparisons. F-family tests fixed operations/triggers/static prompts; H-Rollback tests selective rejection; probes test candidate selection, admission, and diagnosis. A/B/C compares published-rule dispatch, fixed checks, and rule-free repair. Published-rule dispatch has the highest Cons@5 in both scopes (69.57% ± 4.35% on 23 tasks; 66.67% ± 5.89% on 24 tasks). We added related work, Skill-Pro/AHE adaptations, and T1-lite’s evolving-bank comparison.

3. Reported costs and repeated HMMT evidence. Held-out token costs and cumulative training/evolution costs are reported separately. Iter1 and iter2 use 1.10× and 1.07× B0 held-out tokens, respectively, while their observed Avg@5 and Cons@5 exceed B0. Across trajectories, the original 11-task recovery pattern recurs; campaign means and sample SD are reported separately from task-paired tests.

Thank you for your time and guidance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.