- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper presents MetaRule, a transactional framework for adapting language-model agent harnesses across tasks. The framework combines answer-free execution evidence, isolated diagnosis, bounded same-thread revision, typed procedural closure, paired admission tests, and delayed publication into a frozen successor state. Procedures are assembled from a finite registry, and rejected positive candidates may produce precautionary reverse rules. The principal contribution is an integrated, auditable lifecycle governing how execution evidence becomes persistent agent behavior.
The evaluation compares MetaRule with an unextended MemoHarness baseline (B0) and native harness evolution (T1) on GPQA-Diamond, AIME2025, and HMMT, supplemented by execution audits and mechanism ablations. Relative to T1, the complete system improves cross-domain GPQA Avg@5 from 72.63% to 82.11% and cross-type HMMT Avg@5 from 35.00% to 55.00%. The strongest evidence concerns complete-system improvements and recovery from negative transfer; the independent contribution of persistent procedures remains inconclusive in the reported frozen-dispatch and trajectory-shadow analyses (Sections 4–6).
Critical strengths
-
Frozen within-batch states, scoped rule dispatch, delayed publication, and explicit separation of diagnosis from private answers provide a concrete framework for inspecting adaptive agents. Separating procedural closure from official correctness verification is particularly useful because it avoids treating a completed repair as proof of a correct answer. This protocol is a meaningful systems contribution that makes the source and timing of behavioral changes easier to examine (Section 3; Appendices C and E).
-
On HMMT, MetaRule improves overall Avg@5 from 35.83% to 57.50%, with an exact task-level sign-flip p-value of 0.0007. On the cross-type subset, it improves eight tasks, ties four, and degrades none relative to T1 (Table 4). These results provide meaningful evidence that the integrated system can mitigate deterioration introduced by native evolution. Reporting paired task-level statistics and explicitly limiting causal interpretations strengthens this finding.
Critical weaknesses
- Each primary treatment has only one training trajectory, and held-out sets are small. The HMMT improvement over B0 is not statistically conclusive (+5.83 points; p = 0.2585), limiting confidence in a stable advantage over the original harness.
Questions for authors
- Does MetaRule outperform a control with the same diagnosis–revision–closure pipeline and matched computational budget, but fixed trigger-conditioned counterchecks instead of learned persistent rules? Do the principal gains, especially the HMMT recovery and improvement over B0, persist across independent training trajectories?
- What behavior distinguishes a reverse rule from its rejected positive candidate, and why must every batch publish an artifact?
- Which closure conditions are independently verified, and which rely on the task model’s self-report?
6 comments
Response to Reviewer UpEr (part 2/4) Q1.1 part1
Q1.1: Fixed Trigger-Conditioned Counterchecks
You ask whether MetaRule outperforms “the same diagnosis–revision–closure pipeline and matched computational budget, but fixed trigger-conditioned counterchecks instead of learned persistent rules.” F2 implements the fixed-trigger alternative in that same pipeline; Generic-reasoning is a second control for extra reasoning. Both use the seed-42 frozen iteration-5 state, the same 24 held-out tasks × five generations, Haiku 4.5 at medium effort, zero tools, official verifier, task order, retry policy, and configured caps of 20 calls and 1M output tokens per task. The cost table below reports actual consumption so that the budget part of your question can be assessed directly.
The fixed-check arm uses this predeclared operation set and trigger map:
| Registered operation | Concrete action |
|---|---|
| Check givens | Re-read the task and check use of stated facts, constraints, and requested format. |
| Independent derivation | Derive a fresh solution from the task evidence. |
| Separate criteria | Evaluate every required criterion independently and check joint satisfaction. |
| Resolve conflicts | Locate inconsistent claims or steps and reconcile them against task evidence. |
| Preserve supported work | Keep supported conclusions and revise only unsupported or conflicting parts. |
| Failure trigger | Fixed checks selected |
|---|---|
| Choice mapping or missing derivation | Check givens; independent derivation |
| Conflated criteria | Check givens; separate criteria |
| Format error | Check givens; preserve supported work |
| Reasoning contradiction | Check givens; resolve conflicts |
F2 Fixed-check applies the mapped countercheck after diagnosis, followed by one revision opportunity and typed closure. F2 Generic-reasoning uses a general re-derive-and-inspect instruction in the same pipeline, testing whether extra reasoning alone reproduces the fixed-check effect.
| HMMT control | Avg@5 | Pass@5 | Cons@5 | Best@5 |
|---|---|---|---|---|
| F2 Fixed-check (seed 42) | 39.17% | 70.83% | 41.67% | 70.83% |
| F2 Generic-reasoning (seed 42) | 50.00% | 79.17% | 50.00% | 79.17% |
| B0 (paper reference) | 51.67% | 79.17% | 50.00% | 79.17% |
| T1 (paper r3) | 35.83% | 58.33% | 41.67% | 58.33% |
| MetaRule seed 42 (paper trajectory) | 57.50% | 87.50% | 66.67% | 87.50% |
| MetaRule (three trajectories, mean ± SD) | 54.44% ± 2.68% | 79.17% ± 7.22% | 62.50% ± 4.17% | 79.17% ± 7.22% |
| Held-out cost arm | Input + output tokens (M) |
|---|---|
| F2 Fixed-check | 6.271 |
| F2 Generic-reasoning | 5.760* |
| B0 (paper) | 3.688 |
| T1 (paper r3) | 3.462 |
| MetaRule seed 42 (paper trajectory) | 8.807 |
| MetaRule (three trajectories, mean ± SD) | 6.383 ± 2.103 |
*The Generic-reasoning input-token meter is missing for one call; its total sums all recorded input and output usage.
On the seed-42 paper trajectory, Full has higher observed Avg@5 and Cons@5 than both F2 controls: the advantage concerns correct individual generations and correct plurality answers within the same repair pipeline. Full consumes 8.807M held-out tokens, compared with 6.271M for Fixed-check. This answers the same-pipeline part of your question through a measured performance–cost comparison under shared configured caps. Earlier frozen states provide lower-consumption operating points: iter1 uses 4.063 ± 1.045M tokens (1.10× B0; 1.17× T1), with 54.72% ± 4.88% Avg@5 and 61.11% ± 6.36% Cons@5. These are measured checkpoint comparisons against the paper references.
A/B/C three-arm replay. Each arm uses a frozen iteration-5 parent, read-only evaluation, Haiku 4.5 at medium effort, zero tools, the official verifier, and the same task set, under shared configured caps of 20 calls and 1M output tokens per task.
| Arm | Checking policy | Comparison role |
|---|---|---|
| A: Published-rule dispatch | Match a retained rule's trigger and scope, execute its required checks, and verify the closure contract. | A–B compares learned retained contracts with a predefined map; A–C compares their deployment with general repair. |
| B: Fixed trigger check | Select operations from a predefined failure-to-check map; then revise and close. | B–C tests adding predefined checks to general repair. |
| C: Rule-free pipeline | Keep diagnosis, revision, and closure, with no rule lookup or rule-specific required checks. | General-repair reference; the model can still inspect its answer. |
Response to Reviewer UpEr (part3/4) Q1.1 part2, Q2.1
The reconstructed five-replicate records cover 23 complete common tasks across three states; one task lacks four A generations in seed 20261113. The 24-task check uses the complete states, seeds 42 and 20261114. Entries are mean ± sample SD across states.
| Evaluation scope | Arm | Avg@5 | Pass@5 | Cons@5 | Best@5 |
|---|---|---|---|---|---|
| 23 common tasks; three states | A: Published rule | 59.71% ± 1.81% | 85.51% ± 2.51% | 69.57% ± 4.35% | 85.51% ± 2.51% |
| 23 common tasks; three states | B: Fixed check | 58.26% ± 3.14% | 81.16% ± 2.51% | 62.32% ± 10.94% | 81.16% ± 2.51% |
| 23 common tasks; three states | C: Rule-free | 57.68% ± 3.05% | 82.61% ± 4.35% | 63.77% ± 6.64% | 82.61% ± 4.35% |
| 24 tasks; two complete states | A: Published rule | 56.67% ± 1.18% | 83.33% ± 5.89% | 66.67% ± 5.89% | 83.33% ± 5.89% |
| 24 tasks; two complete states | B: Fixed check | 54.17% ± 1.18% | 77.08% ± 2.95% | 54.17% ± 5.89% | 77.08% ± 2.95% |
| 24 tasks; two complete states | C: Rule-free | 57.50% ± 2.36% | 85.42% ± 2.95% | 60.42% ± 8.84% | 85.42% ± 2.95% |
For three states, descriptive post-hoc exact two-sided task-paired sign-flip tests give A–B/A–C p=.6719/.4927 for Avg@5 and .2344/.2188 for Cons@5. Testing averages generations and states within each task. No contrast reaches .05 after Holm correction across three contrasts per metric. The two scopes agree on A's higher plurality-answer accuracy; the full-24-task comparison places C higher on per-attempt accuracy and coverage.
For your fixed-countercheck alternative, A–B is the closest contrast: both preserve repair and closure, while the source and applicability of the checks change. The recurring benefit is plurality-answer accuracy in both evaluation scopes. A–C also favors correct answers forming the plurality under the same aggregation rule. The paired tests quantify the uncertainty of these differences; the primary Full campaign remains a separate system-level evaluation.
For the geometry problem with answer "7/18", A matches a retained missing-derivation rule, requires checking the givens and independent derivation, and returns a verifier-correct revision after closure. This is the operational role of the retained contract.
Q2.1: Reverse-Rule Semantics
You ask “What behavior distinguishes a reverse rule from its rejected positive candidate?” A positive candidate proposes reusable behavior and must pass paired efficacy admission. A reverse rule is a preventive constraint: it identifies the rejected behavior and blocks blind reuse under a defined trigger, with provenance, required checks, capability scope, and expiry. In the audited GPQA run, five rejected positive candidates followed this precautionary path.
F3 is the offline candidate-to-guard trace audit. Its input is an archived candidate, its paired-gate decision, the published reverse artifact, and the later execution receipt. Appendix D.1 provides one documented GPQA case with the following findings:
| Audit step | Recorded finding |
|---|---|
| Positive proposal | On missing derivation, check givens, derive independently, and preserve supported work. |
| Admission decision | Positive admission is rejected for insufficient execution coverage; the parent and source gate remain linked to the reverse artifact. |
| Reverse program | Forbid blind reuse of that parent. On the missing-derivation trigger, require checking givens and independent derivation within the artifact’s scope. |
| Later execution | A held-out Biology trace executes the two required checks; closure records completion, and the public answer changes B→A and passes the official verifier. |
The compiled reverse instruction states: “Use only the public givens and execute every countercheck before changing the supported conclusion.” It also instructs against shortcut reuse of the rejected parent. Rejecting the parent controls positive-program inheritance; the reverse actively binds two checks to later matching tasks, with closure requiring their declared completion. This is the operational difference between withholding a positive program and deploying a precautionary checking contract. F3 establishes the recorded candidate→guard→action path; the accuracy comparisons evaluate the resulting policies.
Response to Reviewer UpEr (part 1/4) W1.1/ Q1.2, W1.2
Thank you for recognizing the frozen-state publication protocol and independently verified correctness. You ask whether the gains recur across training runs and whether fixed counterchecks explain them. Across the paper HMMT run and two supplementary campaigns, MetaRule reaches 62.50% ± 4.17% Cons@5, alongside higher Avg@5 than both paper references. We address repeatability first, then compare fixed controls and describe the publication and closure contracts.
Metrics and campaigns. Avg@5 is per-attempt verifier accuracy; HMMT Cons@5 is plurality-answer accuracy; binary Pass@5 and Best@5 coincide. All HMMT task execution uses Haiku 4.5. The paper trajectory used GPT-5.5 for isolated diagnosis; its unavailability through our rebuttal access led us to use Haiku 4.5 for diagnosis in the two added campaigns. Individual results and mean ± sample SD describe these three campaigns, including this configuration change. B0/T1 are paper references.
W1.1 / Q1.2: Independent Trajectories
The three-state evaluation answers the repeatability part descriptively: all three MetaRule trajectories exceed the paper B0 and T1 Avg@5 references. The original trajectory's exact task-paired sign-flip tests give p=0.0007 against T1 and p=0.2585 against B0; added MetaRule trajectories quantify variation across trained states but do not change the original task-paired B0 test.
We evaluate three training states. Each trains for five iterations on the same 39 HMMT tasks and freezes its iteration-5 state for the same 24 held-out tasks × five generations. All use Claude Haiku 4.5 at medium effort, zero tools, and per-task caps of 20 calls and 1,000,000 output tokens. The paper run and supplementary campaigns are completed evaluations of separately trained states. Held-out execution is read-only. Mean ± sample SD describes the three completed campaigns, with the held-out set fixed.
Each Full campaign in the preceding table uses the same 24 held-out tasks × five generations, including every generated attempt in its 120-outcome aggregate.
Your question specifically asks whether HMMT recovery persists across training trajectories. We fix the 11 T1-harmed tasks from the original paper comparison and compare per-task five-generation accuracy. The paper trajectory improves all 11 and restores eight to at least B0. The added seed20261113/20261114 trajectories improve 10/11 and 8/11 and restore 8/11 and 7/11 to at least B0; all remaining tasks tie T1. Thus the recovery pattern recurs at the task level, with no further decline relative to T1 on this fixed subset. In the original trajectory, "7/18" illustrates reliable plurality selection and "279" illustrates coverage beyond both references.
The paired analysis treats each held-out task as one unit, averaging its five generations before the sign-flip test. The supplementary campaigns reuse the same 24 tasks, providing evidence across trained states. Sample SD describes the completed campaigns; task-level tests quantify the original paired contrasts.
W1.2: Held-out Precision
Thank you for raising the concern that “held-out sets are small.” Compact, high-difficulty benchmarks are established evaluation instruments in reasoning research. For example, s1 (EMNLP 2025) evaluates on AIME24 with 30 problems and GPQA Diamond with 198 questions [1]. AceReason-Nemotron (NeurIPS 2025) likewise evaluates on AIME2024/2025 and HMMT2025 February [2]. These published evaluations illustrate the value of carefully constructed competition-mathematics and expert-level science benchmarks for identifying reasoning improvements.
For MetaRule, we strengthen this benchmark evidence through paired task-level evaluation and repeated training campaigns. Across the paper trajectory and both added campaigns, recovery recurs on the original T1-harmed tasks, and every campaign exceeds the paper B0 Avg@5 reference. The central finding is recurring recovery across separately trained states: the improvement is visible in the behavior of the same held-out problems.
[1] Muennighoff et al. s1: Simple test-time scaling. EMNLP 2025. https://aclanthology.org/2025.emnlp-main.1025/
[2] Chen et al. AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning. NeurIPS 2025. https://neurips.cc/virtual/2025/poster/119111