acceptodds
Under review as a conference paper at ICLR 2027

Meta²: Precomputing Reusable Knowledge for Online Harness Evolution in Long-Horizon Agents

33 views

Abstract

Harness optimization improves long-horizon agents by refining the prompts, tools, memory, and hooks around a frozen model. Offline methods can learn from completed trajectories, but deploy one shared harness that cannot react to evidence revealed during a new run. Online evolution can react within the run, but may have to reason about every intervention from scratch. We introduce Meta², which bridges these regimes by precomputing reusable knowledge about how to evolve a harness, rather than precomputing one universal harness. Offline, Meta² compares three rollout conditions: the task agent alone, the current Meta-Skill-guided system, and a training-only evolver with privileged verifier information. These contrasts provide evidence that an evolver should act, hold, or avoid an intervention, which is aggregated across tasks into a textual Meta-Skill M. Online, M is frozen. At checkpoints within an uninterrupted held-out run, the evolver combines it with active-run evidence and chooses Keep or a task-local Evolve edit; the task agent then continues from the same environment state. Neither held-out verifier feedback nor task-specific edits propagate across runs. Meta² achieves the highest point estimates among the evaluated methods on both Terminal-Bench 2.1 and DeepSWE 1.1 with GPT-5.6-Terra, reaching 81.00% and 44.44%, respectively. These scores are 6.33 and 5.33 percentage points above the strongest comparators. With DeepSeek-V4-Flash-0731, Meta² achieves 77.67% on Terminal-Bench 2.1, 3.00 percentage points above the strongest comparator. Cumulative ablations associate the largest additional gains with contrastive Meta-Skill construction, while checkpoint sensitivity exposes a responsiveness–cost tradeoff. These results support precomputing reusable evolution knowledge while keeping each realized harness edit local to the active task.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM Agents, Harness Optimization, Self-Improvement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M3wnOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Meta², a framework that integrates offline knowledge learning with online harness evolution to improve the performance of long-horizon agents. Rather than optimizing a fixed harness for all tasks, Meta² learns reusable evolution knowledge from contrastive execution trajectories, capturing when interventions are beneficial, unnecessary, or potentially harmful. During deployment, a dedicated evolver leverages this knowledge and observations from the ongoing task to selectively modify the agent's harness without interrupting execution. By separating globally reusable knowledge from task-specific adaptations, the framework enables agents to respond dynamically to emerging challenges while avoiding undesirable cross-task interference. Experiments across long-horizon agent benchmarks demonstrate the effectiveness of this approach, highlighting the value of contrastive knowledge construction and adaptive harness evolution.

Critical strengths
  1. Novel and well-motivated framework for adaptive harness evolution. The paper presents a compelling framework that integrates offline knowledge learning with online harness evolution. Rather than optimizing a static harness, Meta² learns reusable guidance to inform harness adaptation during task execution. This design enables task-specific adaptation while facilitating knowledge transfer across tasks, effectively addressing key limitations of existing offline and online approaches.
  2. Well-designed contrastive learning strategy and comprehensive empirical evaluation. The paper introduces a thoughtful contrastive learning strategy that distills actionable guidance from diverse execution outcomes, explicitly capturing when to intervene, when to refrain from intervention, and which interventions to avoid. Extensive experiments demonstrate consistent improvements across multiple benchmarks and model backbones, while ablation studies and cost analyses further substantiate the effectiveness and practical value of the proposed framework.
Critical weaknesses
  1. Insufficient causal attribution in contrastive Meta-Skill learning. The proposed method derives intervention guidance from outcome differences across independently executed A/B/C rollouts. However, these trajectories may diverge due to stochastic variations rather than the interventions themselves, making it difficult to determine whether a particular harness modification actually contributed to success or failure. Although the authors acknowledge this limitation, the lack of checkpoint-level counterfactual analysis weakens the justification for the paper's central contrastive learning mechanism. More controlled experiments would be necessary to establish whether the extracted guidance reliably captures effective intervention strategies rather than spurious correlations.
  2. Limited evidence for the generalizability of learned Meta-Skills. A central motivation of the paper is to learn reusable knowledge that can guide harness evolution across tasks. However, the experiments construct separate Meta-Skills for each benchmark and model backbone, without evaluating whether the learned knowledge transfers across different task domains or underlying models. Consequently, the results primarily demonstrate improvements within the same benchmark and backbone settings, rather than broader transferability. This limitation weakens the paper's central claim of reusable evolution knowledge and raises concerns about the scalability and practical applicability of the proposed framework beyond the evaluated settings.
Questions for authors
  1. Given that the A/B/C comparisons rely on independently sampled trajectories, to what extent can the learned intervention guidance be reliably attributed to harness modifications rather than stochastic variations in agent execution? Although the paper acknowledges that these comparisons do not establish checkpoint-level causality, could the authors provide controlled counterfactual experiments that compare intervention decisions from identical intermediate agent and environment states to further validate the effectiveness of the extracted guidance?

  2. The experiments demonstrate the effectiveness of Meta-Skills learned and evaluated within the same benchmark and model backbone, but their transferability across different task distributions or backbones remains unclear. Could the authors clarify the intended scope of knowledge reusability and investigate whether a Meta-Skill learned in one setting can effectively guide harness evolution in another without additional offline optimization?

  3. The proposed contrastive learning framework relies on both the privileged verifier-informed evolver (condition C) and the targeted solicitation procedure, yet their individual contributions are not explicitly isolated in the current ablation studies. Could the authors provide additional controlled ablations to disentangle the effects of these components, ideally under comparable offline computational budgets? In particular, how much of the observed improvement can be attributed to verifier-informed contrasts versus the additional informative trajectories generated through targeted solicitation?

5 comments

Thank you for your positive and constructive review, and for recognizing the novel and well-motivated framework, the well-designed contrastive learning strategy, and the comprehensive empirical evaluation. We appreciate your suggestions on checkpoint-level causal attribution, the transferability of the Meta-Skill, and isolating the contribution of condition C under comparable budgets. We have conducted additional experiments on a checkpoint-level counterfactual from identical intermediate states, cross-backbone and cross-benchmark transfer, and a budget-controlled ablation of condition C. We address each point below.


Q1: Given that the A/B/C comparisons rely on independently sampled trajectories, to what extent can the learned guidance be attributed to harness modifications rather than stochastic variations? Could the authors provide controlled counterfactual experiments that compare intervention decisions from identical intermediate states?

We thank the reviewer for this question and for suggesting a controlled counterfactual. The learned guidance can be attributed to the evolver's decisions: in the suggested checkpoint-level counterfactual, executing a Meta-Skill-guided decision from an identical intermediate state raises the pass rate by 5.41 pts over withholding it. Below, we (1) describe the offline mechanisms that keep stochastic variation out of the Meta-Skill, (2) report this counterfactual, and (3) add detailed case studies.

(1) Offline mechanisms against spurious attribution. Three rules in the prompts (Appendix H) separate the evolver's effect from stochastic variation:

  • Decision-level judgment. Reflect is prompted to judge each evolver decision by what was visible at that checkpoint, independently of the final outcome, and writes an entry only when it can point to one concrete intervention and to what the other group did at the corresponding point.
  • Explicit null verdict. When the same failure appears on both sides, or the difference is explained by the task agent's own variation, Reflect returns no_supervisor_effect and extracts no entry.
  • Cross-task support and re-testing. Aggregate keeps only patterns supported by at least two training tasks, since a pattern produced by stochastic variation on one task rarely recurs on another; Update then refines or deletes entries that mislead the evolver in later batches.

(2) Checkpoint-level counterfactual. To test whether the Meta-Skill-guided decisions themselves improve outcomes, we execute or withhold each decision from the same intermediate state.

  • Fork points: in one Meta$^2$ run on each of the 60 Terminal-Bench 2.1 test tasks, the earliest checkpoint of each type (first, mid-run, terminal) at which the evolver cited a Meta-Skill entry and chose EVOLVE (74 in total).
  • Branches: the full sandbox (memory, workspace, background processes, task agent context) is forked into 3 branches that execute the decision and 3 that withhold it (terminal: send back vs. accept), 444 in total.
  • Isolation: the evolver is disabled after the fork, so the two groups differ only in this decision.
Checkpoint typeFork pointsExecuteWithholdΔ\Delta (pts)
First checkpoint (start)4071.6766.67+5.00
Mid-run (during)2152.3849.21+3.17
Terminal (end)1356.4146.15+10.26
All7463.5258.11+5.41

Results:

  • The gain is attributable to the guided decision. Both groups start from the same state and are repeated three times, so post-fork stochastic variation affects them alike; executing the guided decision still raises the pass rate by 5.41 pts, with gains at all three checkpoint types and the largest at terminal send-back.
  • The learned guidance applies to new tasks. All fork points are on held-out test tasks, so entries learned from training contrasts lead to beneficial decisions beyond the tasks they were extracted from.

(3) Detailed case studies. Appendix D compares runs with and without the Meta-Skill on the same task, marking the KEEP / EVOLVE decision and the cited entry at every checkpoint; besides configure-git-webserver, we added three cases covering a mid-run act entry, an avoid entry, and a hold entry.

Paper changes: In the revision, we have added (1) to Sec. 3.2 and Appendix B.1, (2) to Appendix E.4,, and (3) to Appendix D.


Q2: The transferability of Meta-Skills across different task distributions or backbones remains unclear. Could the authors clarify the intended scope of knowledge reusability and investigate whether a Meta-Skill learned in one setting can effectively guide harness evolution in another without additional offline optimization?

We thank the reviewer for this question. Meta$^2$ is designed to reuse one frozen Meta-Skill across unseen tasks within a deployment setting (a given backbone and benchmark), and all test tasks in our evaluation are held out from Meta-Skill construction. To evaluate transfer beyond this setting, we directly apply a Meta-Skill constructed in one setting to another, without any additional offline optimization. The task agent harness, checkpoint schedule, and evaluation protocol are the same as in the main experiments. The results are as follows:

Cross-backbone transfer (Terminal-Bench 2.1):

Meta-Skill source → target backboneTask agent aloneEvolver without Meta-SkillTransferred Meta-SkillMeta$^2$ (built on target)
DeepSeek-V4-Flash → GPT-5.6-Terra66.3374.6774.6781.00
GPT-5.6-Terra → DeepSeek-V4-Flash70.0072.3373.0077.67

Cross-benchmark transfer (GPT-5.6-Terra):

Meta-Skill source → target benchmarkTask agent aloneEvolver without Meta-SkillTransferred Meta-SkillMeta$^2$ (built on target)
DeepSWE 1.1 → Terminal-Bench 2.166.3374.6773.3381.00
Terminal-Bench 2.1 → DeepSWE 1.134.6736.4438.6744.44
  • Cross-backbone: The Meta-Skill built with DeepSeek-V4-Flash does not exceed the evolver without Meta-Skill on GPT-5.6-Terra, and the one built with GPT-5.6-Terra brings a slight gain on DeepSeek-V4-Flash. Since different models have different behavioral tendencies and error patterns, the interventions that pay off differ across backbones. Therefore, a separate offline learning process for each backbone yields better performance.
  • Cross-benchmark: The Meta-Skill built on DeepSWE 1.1 slightly lowers performance on Terminal-Bench 2.1, while the one built on Terminal-Bench 2.1 improves DeepSWE 1.1 over the evolver without Meta-Skill. DeepSWE 1.1 consists of relatively specialized software engineering tasks, whereas Terminal-Bench 2.1 covers a wide range of tasks with software engineering as one category, so its Meta-Skill contains guidance that also applies to DeepSWE 1.1.

Intended scope of reuse. We recommend learning a separate Meta-Skill for each backbone and benchmark, since different backbones and tasks exhibit different error types. The intended reuse unit is a deployment setting: one frozen Meta-Skill guides multiple unseen tasks with a given backbone and benchmark, without task-specific offline relearning. The cross-backbone and cross-benchmark experiments additionally probe transfer beyond this setting and show that target-specific learning delivers the strongest performance in the evaluated pairs.

Paper changes: We have added these results to Appendix E.9 (referenced in Sec. 4.2).


Q3: The individual contributions of the verifier-informed evolver (condition C) and the targeted solicitation procedure are not isolated. Could the authors provide controlled ablations under comparable offline budgets, and how much of the improvement comes from verifier-informed contrasts versus the additional trajectories generated through targeted solicitation?

We thank the reviewer for this suggestion. Under a fixed offline budget, the improvement comes from the verifier-informed contrasts of condition C: adding C improves over A + B by 4.00 / 7.11 pts on TB2.1 / DeepSWE 1.1. In Meta$^2$, targeted solicitation is realized through condition C on every training task: the evolver receives the task's privileged verifier information, is instructed to make only general harness edits and never to pass this information to the task agent, and a post-hoc audit removes trajectories with suspected leakage. The contribution of solicitation is therefore the contribution of condition C, which we isolate below.

Budget-controlled ablation of condition C. Full Meta$^2$ uses 17 rollouts per training task (5 / 5 / 7 under A / B / C). We keep this total fixed, so that every variant has the same number of trajectories, and vary only the conditions:

  • B only (the self-distilled setting in Fig. 4): 17 rollouts under B; pass/fail pairs within each task are given to Reflect.
  • A + B: 8 rollouts under A and 9 under B, so only the A–B contrast directions are available (B-pass/A-fail, B-fail/A-pass, A-pass/B-pass).
  • A + B + C (full Meta$^2$): 5 / 5 / 7 rollouts under A / B / C.

All other components are identical, so the difference between A + B and A + B + C isolates the effect of verifier-informed contrasts at a fixed number of trajectories.

Offline rollouts per taskTB2.1DeepSWE 1.1
Task agent alone66.3334.67
Evolver without Meta-Skill74.6736.44
Evolver with Meta-Skill distilled from B only77.0035.56
Evolver with Meta-Skill distilled from A + B (new)77.0037.33
Evolver with Meta-Skill distilled from A + B + C (Meta$^2$)81.0044.44
  • More trajectories alone do not explain the gain. B only and A + B use the same number of rollouts as the full pipeline, yet A + B is only on par with or slightly above B only, and both stay below Meta$^2$.
  • Verifier-informed contrasts account for the remaining improvement. At the same budget, adding C raises performance over A + B by 4.00 pts on TB2.1 and 7.11 pts on DeepSWE 1.1.
  • C supplies a signal that A and B lack. A–B contrasts mostly reveal when the current evolver's interventions are unnecessary or harmful, and thus yield mainly hold and avoid guidance; on tasks where both A and B fail, they contain no successful recovery. C-pass/B-fail contrasts provide most act candidates in the full pipeline, including all act candidates for the terminal checkpoint, and on some training tasks they are the only informative contrast. These are the decision types that the counterfactual in Q1 identifies as beneficial, especially terminal send-back (+10.26 pts).

Paper changes: We have added this ablation to Sec. 4.5 and Appendix E, and clarified the implementation of condition C in Sec. 3.2 and Appendix B.


Thank you again for taking the time to review our paper and for the helpful feedback. We hope these clarifications and new results address your questions. If you have any further questions or suggestions, please do not hesitate to let us know.

Reviewer_HKuQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work introduces a framework designed for improving long-horizon agents by optimizing their operational harness through a hybrid approach. The framework enjoys merits for both offline and online harness methods, being both adaptive and reusable across tasks, which is achieved by computing reusable knowledge offline using contrastive trajectories and a meta-skill structure. Improved performance was achieved in the experimental study on public benchmarks.

Critical strengths
  • The authors formulate agent harness as a hybrid process with offline learned reusable knowledge and online task adaptation locally. This approach makes the proposed method enjoy both advantages from offline and online methods.
  • A checkpoint-based interface is proposed to achieve environment status and an offline contrastive procedure that turns rollout comparison into reusable knowledge.
  • Experimental evaluation is conducted to achieve improved performance over benchmarks, which supports the claims of the paper.
Critical weaknesses
  • The authors should make it clear about the check and mitigation of possible misjudgment of the supervisor evolver. Any mistake that occurred would adversely impact the evolving trajectory of agents and hurt task execution.
  • The offline reusable knowledge learning process is not clear enough. It seems that the model evaluates end-to-end success which may or may not capture the true causality that leads to the success or failure. This therefore imposes uncertainty when re-using the knowledge for online guidance.
  • While the checkpoint-based interface brings benefits, there is also some cost associated. The authors should make it clear about the complexity and token cost for achieving such benefits, both theoretically and experimentally, which can help assess the ROI when applying the proposed techniques.
  • The meta skills and reusable knowledge should be assessed and quantified/characterized, so that it is more confident to re-use such knowledge for guiding the online tasks and/or decide if any learned knowledge should be updated or retired.
  • The types of benchmarks used in this study is limited, making it not easy to assess the generality of the proposed techniques.
Questions for authors

n/a

7 comments

Thank you for your thoughtful review and for recognizing the hybrid formulation of Meta$^2$, which is both adaptive and reusable across tasks, the checkpoint-based interface with the offline contrastive procedure, and the experimental support for our claims. We appreciate your suggestions on checking evolver misjudgment, clarifying the causality of offline learning, quantifying cost and ROI, characterizing the Meta-Skill, and broadening the benchmarks. We have clarified the offline learning process and added analyses of evolver decisions and failures, a checkpoint-level counterfactual, a cost and ROI analysis, a quantitative characterization of the Meta-Skill, and results on two conversational benchmarks. We address each point below.

Q1: The authors should make it clear about the check and mitigation of possible misjudgment of the supervisor evolver. Any mistake that occurred would adversely impact the evolving trajectory of agents and hurt task execution.

We thank the reviewer for raising this point. Meta$^2$ checks and mitigates evolver misjudgment by design, through (1) system-level constraints and (2) the hold and avoid entries of the Meta-Skill; (3) a new audit further quantifies which failed trials are caused by evolver decisions. All analyses below use Terminal-Bench 2.1 with and without the Meta-Skill (300 trials each).

(1) System-level constraints. The following constraints apply to the evolver in both settings:

  • Bounded notices and send-backs: at most one notice per checkpoint and two per run while the task agent is working, which prevents frequent interruption of the task agent, and at most two send-backs per run at the terminal checkpoint.
  • Validated edits: each harness edit is compiled and executed once at registration; if the rebuilt harness fails to load, the edit is rolled back automatically and the evolver receives the error with one more attempt.
Trigger countWithout Meta-SkillMeta$^2$
Trials that exhaust the notice budget13634
Trials that reach the send-back limit911
Harness edits rejected by validation00

With the Meta-Skill, 4× fewer trials exhaust the notice budget, and neither setting produces an invalid harness edit. Send-backs are used slightly more often under Meta$^2$; terminal send-back is the decision type that the counterfactual in Q2 shows to be most beneficial (+10.26 pts).

(2) Hold and avoid entries regulate EVOLVE. Besides act entries, which support EVOLVE, the Meta-Skill contains hold entries, which support KEEP, and avoid entries, which rule out a specific EVOLVE. We group all first and mid-run checkpoints by the entry types cited in the evolver's reasoning:

Cited entry typesCheckpointsEVOLVE rate
Hold / avoid only30410.5%
Act only18261.5%
Act + hold / avoid41244.2%
None3834.2%
Meta$^2$ (total)93636.2%
Without Meta-Skill88468.9%

With the Meta-Skill, the evolver chooses EVOLVE about half as often, and its decisions follow the cited entry types: it chooses KEEP at 89.5% of checkpoints that cite only hold / avoid entries. The reduction is selective: checkpoints citing only act entries still choose EVOLVE at 61.5%, and the pass rate rises from 74.67 to 81.00.

(3) Attribution of failures to evolver decisions. To check whether failures come from low-quality KEEP / EVOLVE decisions, we audit every failed trial in both settings (57 with Meta$^2$, 76 without) with Claude Opus 5.5 under a fixed rubric. For each trial, the annotator identifies the failing test and its cause, checks whether the defect was visible to the evolver at some checkpoint, and assigns one primary label. Trials from the two settings are anonymized and randomly interleaved, and every cited piece of evidence is verified against the raw trajectory.

Primary labelWithout Meta-SkillMeta$^2$
M1: misleading EVOLVE (wrong content, or diverting a viable path)150
M2: faulty check, probe, or tool00
M3: accepting a visibly defective artifact at the terminal checkpoint1814
M4: missing a needed EVOLVE00
M5: harmful send-back00
Failures caused by evolver decisions (M1–M5)3314
Failures unrelated to evolver decisions4343
  • Fewer evolver-caused failures. The Meta-Skill reduces failures caused by evolver decisions from 33 to 14, while failures unrelated to the evolver stay at 43.
  • No misleading EVOLVE. All 15 M1 failures occur without the Meta-Skill, mostly when the evolver supplies a concrete answer or fact before the evidence supports it, the situation that avoid entries describe.
  • Fewer defective acceptances. The remaining evolver-caused failures are terminal acceptances of defective artifacts, which Meta$^2$ also reduces (14 vs. 18).

Together, the system-level constraints and the hold and avoid entries reduce both over-intervention and evolver-caused failures.

Paper changes: We have added these analyses to Appendix E.6, and the values of all limits to Appendix C.3.


Q2: The offline reusable knowledge learning process is not clear enough. It seems that the model evaluates end-to-end success which may or may not capture the true causality that leads to the success or failure. This therefore imposes uncertainty when re-using the knowledge for online guidance.

We thank the reviewer for this question. Although the verifier scores only a whole run, Meta$^2$ extracts decision-level guidance from run-level contrasts: Reflect judges each evolver decision by what was visible at its checkpoint and writes an entry only when an outcome difference can be attributed to that decision. Below, we (1) clarify this process, (2) verify the resulting guidance with a new checkpoint-level counterfactual, and (3) add detailed case studies.

(1) Clarification of the offline learning process. Reflect, Aggregate, and Update are LLM calls with the same backbone as the task agent and the evolver. For each batch rr of training tasks:

  • Reflect (per task and contrast direction): given the task statement, its verifier information (used only to explain outcomes), and records of the two groups of runs, it returns a verdict and candidate entries.
  • Aggregate (per type ×\times time point): merges the batch's candidates, tagged with source tasks, into candidate entries without task-specific details.
  • Update (per batch): given M(r−1)\mathcal{M}^{(r-1)} with each candidate entry's supporting tasks, it applies ADD / MERGE / REFINE / DELETE to obtain M(r)\mathcal{M}^{(r)}.

The next batch collects its B rollouts with M(r)\mathcal{M}^{(r)}, so its contrasts test the updated entries again; after the last batch, M\mathcal{M} is frozen.

Attributing lessons to evolver decisions. Three rules in the prompts (Appendix H) separate the evolver's effect from other factors, such as the task agent's own randomness:

  • Decision-level judgment. Reflect is prompted to judge each evolver decision by what was visible at that checkpoint, independently of the final outcome, and writes an entry only when it can point to one concrete intervention and to what the other group did at the corresponding point.
  • Explicit null verdict. When the same failure appears on both sides, or the difference is explained by the task agent's own variation, Reflect will return no_supervisor_effect and extracts no entry.
  • Cross-task support and re-testing. Aggregate keeps patterns supported by multiple training tasks, and Update refines or deletes entries that mislead the evolver in later batches.

Why not checkpoint-level control in training. Forking every checkpoint into execute / withhold branches, recursively and with repeats for stochasticity, has a cost that grows exponentially with the number of checkpoints. We therefore uses run-level A/B/C contrasts as a scalable training signal.

(2) Checkpoint-level counterfactual. To test whether guidance learned from run-level contrasts improves individual decisions, we execute or withhold each guided decision from the same intermediate state.

  • Fork points: in one Meta$^2$ run on each of the 60 TB2.1 test tasks, the earliest checkpoint of each type (first, mid-run, terminal) at which the evolver cited a Meta-Skill entry and chose EVOLVE (74 in total).
  • Branches: the full sandbox and task agent context are forked into 3 branches that execute the decision and 3 that withhold it (terminal: send back vs. accept), 444 in total.
  • Isolation: the evolver is disabled after the fork, so the branches differ only in this decision.
Checkpoint typeFork pointsExecuteWithholdΔ\Delta (pts)
First checkpoint (start)4071.6766.67+5.00
Mid-run (during)2152.3849.21+3.17
Terminal (end)1356.4146.15+10.26
All7463.5258.11+5.41

Results:

  • From identical states, executing the guided decision raises the pass rate by 5.41 pts, with gains at the first, mid-run, and terminal checkpoints. The gain is therefore attributable to the decision itself rather than to a run-level correlation.
  • The guidance transfers to unseen tasks. All fork points are on held-out test tasks, so guidance learned from training contrasts improves decisions on new tasks.

(3) Detailed case studies. To show how a cited entry changes the subsequent trajectory, Appendix D compares runs with and without the Meta-Skill on the same task, marking the KEEP / EVOLVE decision and the cited entry at every checkpoint. In addition to configure-git-webserver case, we add three cases covering a mid-run act entry, an avoid entry, and a hold entry.

Paper changes: In the revision, we have (1) rewritten the offline part of Sec. 3 and added its implementation details to Appendix C, (2) added the counterfactual experiment to Appendix E.3, and (3) added the case studies to Appendix D.


Reviewer_wvfeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

META² constructs a textual Meta-Skill offline from training rollout contrasts. During held-out tasks, an online evolver uses this frozen knowledge to make KEEP/EVOLVE decisions and apply task-local harness edits.

Critical strengths

The paper is well organized. The figures are clearly presented.

Critical weaknesses

The methodological contribution is limited. The core approach combines reusable textual guidance with an online evolver. Although Figure 4 includes a self-distilled Meta-Skill, it remains unclear how much the proposed construction adds over the same evolver supplied with simple general instructions. Separate ablations of privileged verifier information and targeted solicitation would better establish their contributions.

Questions for authors

see above

5 comments

Dear Reviewer wvfe,

I appreciate the concerns raised regarding the methodological novelty and the need for more fine-grained ablations. These are valid points, and additional experiments would certainly strengthen the paper. However, I have a somewhat more positive overall assessment. In particular, I find the contrastive A/B/C rollout design for extracting reusable evolution knowledge reasonably interesting, and Figure 4 provides some evidence of its benefits over self-distilled guidance. Together with the empirical improvements reported on both benchmarks and the additional results with a second backbone, I think the paper presents meaningful contributions to this research direction. While I agree that some aspects could be better substantiated, I wonder whether a Clear Rejection might be somewhat too strong an assessment. I would encourage reconsidering the overall rating.

Best regards, Reviewer M3wn

Thank you for your careful review, for the accurate summary of Meta$^2$, and for finding the paper well organized, the figures clear, and its soundness free of substantive concerns. We appreciate your suggestions on comparing against general instructions and on separating the contributions of privileged verifier information and targeted solicitation. We have clarified the methodological contribution of Meta$^2$ and conducted additional experiments on a general-instruction control, a checkpoint-level counterfactual, and a budget-controlled ablation of condition C. We address each point below.


Q1: The methodological contribution is limited. The core approach combines reusable textual guidance with an online evolver.

We thank the reviewer for this comment, which helps us state the contribution more precisely. However we respectfully disagree that the methodological contribution is limited. Reusable textual guidance and an online evolver are components of Meta$^2$; its contribution lies in what is optimized offline and in how this knowledge is learned and reused. We explain this from three perspectives.

(1) Optimization target. Existing harness optimization follows two paradigms, each with a limitation:

  • Offline harness optimization (e.g., AHE, Meta-Harness, ACE, TF-GRPO) learns from verified training trajectories but deploys one shared harness or context for all tasks, which may overfit the training tasks and cannot respond to evidence revealed during a new run.
  • Online harness evolution (e.g., Continual Harness) can respond within a run, but its evolver either reasons about each intervention from scratch or relies on verifier feedback, which is unavailable at test time.

Motivated by meta-learning, Meta$^2$ applies offline optimization to the evolver instead of the task agent: it optimizes the evolver's knowledge M\mathcal{M} instead of a harness. During execution, the evolver combines M\mathcal{M} with evidence from the ongoing run to decide whether and how to edit the task agent's harness.

(2) Technical challenges. Realizing this design raises three challenges, addressed in Sec. 3.1–3.3:

  1. Intervene without interrupting (Sec. 3.1). The evolver must edit the harness before the run ends, without resetting the environment or taking over the task agent's actions. Design: every L steps and at the completion claim, the evolver chooses KEEP or EVOLVE and edits through bounded channels; the task agent then continues from the same environment state.
  2. Learn checkpoint-level guidance from run-level outcomes (Sec. 3.2). The verifier scores only a whole run, and on hard tasks the natural rollouts often all fail, leaving no successful recovery to learn from. Design: in A/B/C rollouts, condition C gives a training-only evolver privileged verifier information to obtain such recoveries, under a leak prohibition and audit. Five contrast directions turn outcome differences into act, hold, and avoid evidence. Batched Reflect / Aggregate / Update then distill this evidence into M\mathcal{M}, removing task-specific details and outcome differences caused by the task agent's own randomness.
  3. Reuse across tasks without contamination (Sec. 3.3). At test time there is no verifier, and an edit useful for one task may harm another. Design: M\mathcal{M} is frozen and each task starts from the seed harness h(0)h(0). Carrying edits across tasks instead lowers performance from 81.00 to 78.33 on Terminal-Bench 2.1 and from 44.44 to 39.11 on DeepSWE 1.1 (Fig. 4).

(3) Empirical evidence. Holding the task agent and evolver fixed, replacing the learned M with other textual guidance does not reach Meta$^2$ (81.00 / 44.44 on TB2.1 / DeepSWE 1.1): general instructions score 73.00 / 31.11 (Q2), a self-distilled Meta-Skill 77.00 / 35.56, and a Meta-Skill distilled from A + B only 77.00 / 37.33 (Q3). The gain therefore depends on how M is constructed.

Paper changes: We revised Sec. 1 and 3 to state these challenges and designs explicitly.


Q2: How much does the proposed construction add over the same evolver supplied with simple general instructions?

We thank the reviewer for suggesting this comparison. The proposed construction adds a clear gain: under matched format and length, general instructions do not improve the evolver, and Meta$^2$ exceeds them by 8.00 / 13.33 pts. We support this with three pieces of new evidence: (1) a general-instruction control, (2) a checkpoint-level counterfactual, and (3) detailed case studies.

(1) General-instruction control. To test whether the gain requires learning from rollouts, we replace the Meta-Skill with general instructions and keep all other components fixed.

Setting:

  • Generator: the same LLM as the evolver (GPT-5.6-Terra, medium), given only the evolver's system prompt and a task-independent benchmark description, without seeing any rollout, score, or task content.
  • Matched format: the instructions follow the Meta-Skill schema (type act/hold/avoid ×\times time point start/during/end; title, when, guidance, why), match its number of entries and its length within ±15\pm 15 %, and are placed at the same position in the evolver's prompt.
Evolver knowledgeTB2.1DeepSWE 1.1
None (task agent alone)66.3334.67
Evolver, no knowledge74.6736.44
Evolver + general instructions (new)73.0031.11
Evolver + self-distilled Meta-Skill (B-only)77.0035.56
Evolver + contrastive Meta-Skill (A+B+C, Meta$^2$)81.0044.44

Results:

  • General instructions do not improve the evolver. They score 1.67 / 5.33 pts below the evolver without knowledge.
  • The cited entries differ. With general instructions, the evolver mostly cites hold or avoid entries and almost never acts at the first checkpoint. With the Meta-Skill, it applies concrete, setting-specific edits, such as installing a standing instruction at the start of a Terminal-Bench run.

Why offline optimization is needed. We believe that the evolver in this scenario is a specialized role: whether an edit rescues a run or interrupts productive work depends on the task agent, the benchmark, and the edit channels, which a capable LLM cannot reliably infer from its prior without observing outcomes.

(2) Checkpoint-level counterfactual. To test whether individual Meta-Skill-guided decisions improve task outcomes, we execute or withhold each decision from the same intermediate state. Setting:

  • Fork points: in one Meta$^2$ run on the 60 Terminal-Bench 2.1 test tasks, the earliest checkpoint of each type (first, mid-run, terminal) at which the evolver cited a Meta-Skill entry and chose EVOLVE.
  • Branches: the full sandbox (memory, workspace, background processes, task agent context) is forked into 3 branches that execute the decision and 3 that withhold it (harness restored, reply "Continue the task."; at the terminal checkpoint, send back vs. accept).
  • Isolation: the evolver is disabled after the fork, so the two groups differ only in this decision.
  • Scale: 74 fork points and 444 branches, all scored by the benchmark verifier.
Checkpoint typeFork pointsExecuteWithholdΔ\Delta (pts)
First checkpoint (start)4071.6766.67+5.00
Mid-run (during)2152.3849.21+3.17
Terminal (end)1356.4146.15+10.26
All7463.5258.11+5.41

Results:

  • From identical states, executing the guided decision raises the pass rate by 5.41 pts, with gains at all three checkpoint types and the largest at terminal send-back (+10.26). This indicates that the Meta-Skill-guided decisions themselves improve task outcomes.
  • Absolute pass rates are below 81.00 because fork points concentrate on harder tasks and the evolver is disabled after the fork; the relevant comparison is execute the evolve decision vs. withhold at the same fork point.

(3) Detailed case studies. To show how the Meta-Skill changes individual decisions, we compare trajectories on the same task with and without the Meta-Skill, using the same task agent harness, model, and checkpoint schedule.

  • Appendix D gives one such comparison on configure-git-webserver: without the Meta-Skill, the evolver asks for end-to-end validation only "where feasible", and the task agent delivers an installer it never runs; with the Meta-Skill, the evolver requires a push-and-curl check at the first checkpoint, and the run passes.
  • We add three further cases: a mid-run act entry on DeepSWE 1.1 (obsidian-linter-link-format-conversion), an avoid entry against premature redirection (winning-avg-corewars), and a hold entry that preserves the notice budget (path-tracing-reverse). Each case marks the KEEP / EVOLVE decision and the cited entry at every checkpoint.

Paper changes: We added (1) and (2) to Sec. 4.5 and Appendix E, and the new case studies to Appendix D.


Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response

We thank all reviewers (wvfe, HkuQ, M3wn) for their thoughtful feedback and constructive suggestions. We appreciate the positive feedback on our framework, contrastive procedure for learning reusable evolution knowledge, and comprehensive empirical evaluation.

Our responses clarify Meta$^2$'s central contribution: learning how to evolve an agent's harness, so that experience acquired offline guides when and how to adapt to new evidence during execution. We further support this contribution with new controlled experiments and analyses. Below, we summarize these clarifications and findings.

1. We have clarified Meta²’s central methodological contribution and how Meta$^2$ addresses key limitations of existing harness optimization (HkuQ, M3wn). Offline optimization produces a shared harness that remains fixed during deployment, while existing online approaches either derive interventions afresh or rely on verifier feedback. Meta$^2$ shifts offline optimization from the task agent's harness to the evolver's knowledge, enabling task-specific harness adaptation without verifier feedback at test time.

2. We have clarified Meta$^2$’s technical contribution: a contrastive procedure learns when and how to evolve the harness.

  • Learning when to act, when to hold, and which interventions to avoid (HkuQ, M3wn). Outcome differences across A/B/C rollouts are distilled into the Meta-Skill; condition C provides privileged verifier information only during training to obtain successful recoveries on hard tasks (Sec. 3.2).
  • Selective adaptation during execution (HkuQ, M3wn). At periodic checkpoints and completion claims, the evolver chooses KEEP or EVOLVE and edits the harness through bounded channels, and the task agent continues from the same environment state (Sec. 3.1).
  • Reusable knowledge with task-specific edits (M3wn). The Meta-Skill is frozen at deployment and each task begins with the seed harness; carrying edits across tasks lowers performance from 81.00 to 78.33 on Terminal-Bench 2.1 and from 44.44 to 39.11 on DeepSWE 1.1 (Sec. 3.3, Fig. 4).

3. The experiments in our paper demonstrate consistent improvements.

  • Improvements across benchmarks and backbones (HkuQ, M3wn). With GPT-5.6-Terra, Meta$^2$ reaches 81.00 / 44.44 on Terminal-Bench 2.1 / DeepSWE 1.1, vs. 66.33 / 34.67 for the task agent alone and 74.67 / 36.44 for the evolver without Meta-Skill, and exceeds the strongest baseline on Terminal-Bench 2.1 by 6.33 pts (Sec. 4.2).
  • Favorable cost-performance trade-off (M3wn). Meta$^2$ lies on the performance–full-cost Pareto frontier on both benchmarks (Sec. 4.3).

4. Additional rebuttal experiments further strengthen our findings.

  • Learned knowledge outperforms matched general instructions (wvfe): general instructions score 73.00 / 31.11, below the evolver without knowledge and 8.00 / 13.33 pts below Meta$^2$ (Sec. 4.5, App. E.3).
  • Meta-Skill-guided decisions improve success from identical states (M3wn, HkuQ, wvfe): +5.41 pts over withholding across 74 fork points (444 branches), and +10.26 pts at terminal send-back (App. E.4).
  • Condition C improves performance at a matched rollout budget (wvfe, M3wn): with 17 rollouts per training task, adding condition C improves over A + B by 4.00 / 7.11 pts (Sec. 4.5, Appendix E.5).
  • The Meta-Skill reduces failures caused by evolver misjudgments (HkuQ): with the Meta-Skill, failures caused by evolver decisions drop from 33 to 14, and none comes from a misleading EVOLVE (Appendix E.6).
  • Meta$^2$ outperforms reflection alone with a smaller Meta-Skill (HkuQ): its 14 entries are all cited and reach 81.00, while a Reflect-only variant with 57 entries leaves 15 uncited and reaches 75.67 (App. E.7).
  • Meta$^2$ lowers cost per solved task under repeated deployment and ROI (HkuQ): both its cost per solved task and its incremental cost-effectiveness ratio are below those of both online methods (App. E.8).
  • Knowledge learned on Terminal-Bench improves DeepSWE (M3wn): 36.44 → 38.67 without further offline optimization; Meta-Skills built on the target setting perform best (App. E.9).
  • Meta$^2$ leads on two conversational benchmarks (HkuQ): 95.00 on τ2\tau^2-bench telecom (seed 85.26) and 24.17 on τ3\tau^3-bench banking knowledge (seed 20.30) (App. E.10).
  • Case studies show how act, hold, and avoid guidance changes individual KEEP/EVOLVE decisions (wvfe, M3wn, HkuQ): three additional cases show how act, hold, and avoid entries change individual KEEP / EVOLVE decisions (App. D).

We believe Meta$^2$ makes a timely contribution, supported by broad empirical evaluation and new evidence on decision-level effects, the role of condition C, cost, and generality.

We thank the reviewers, the Area Chair, the Senior Area Chair, and the Program Chairs again for their consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.