- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Meta², a framework that integrates offline knowledge learning with online harness evolution to improve the performance of long-horizon agents. Rather than optimizing a fixed harness for all tasks, Meta² learns reusable evolution knowledge from contrastive execution trajectories, capturing when interventions are beneficial, unnecessary, or potentially harmful. During deployment, a dedicated evolver leverages this knowledge and observations from the ongoing task to selectively modify the agent's harness without interrupting execution. By separating globally reusable knowledge from task-specific adaptations, the framework enables agents to respond dynamically to emerging challenges while avoiding undesirable cross-task interference. Experiments across long-horizon agent benchmarks demonstrate the effectiveness of this approach, highlighting the value of contrastive knowledge construction and adaptive harness evolution.
Critical strengths
- Novel and well-motivated framework for adaptive harness evolution. The paper presents a compelling framework that integrates offline knowledge learning with online harness evolution. Rather than optimizing a static harness, Meta² learns reusable guidance to inform harness adaptation during task execution. This design enables task-specific adaptation while facilitating knowledge transfer across tasks, effectively addressing key limitations of existing offline and online approaches.
- Well-designed contrastive learning strategy and comprehensive empirical evaluation. The paper introduces a thoughtful contrastive learning strategy that distills actionable guidance from diverse execution outcomes, explicitly capturing when to intervene, when to refrain from intervention, and which interventions to avoid. Extensive experiments demonstrate consistent improvements across multiple benchmarks and model backbones, while ablation studies and cost analyses further substantiate the effectiveness and practical value of the proposed framework.
Critical weaknesses
- Insufficient causal attribution in contrastive Meta-Skill learning. The proposed method derives intervention guidance from outcome differences across independently executed A/B/C rollouts. However, these trajectories may diverge due to stochastic variations rather than the interventions themselves, making it difficult to determine whether a particular harness modification actually contributed to success or failure. Although the authors acknowledge this limitation, the lack of checkpoint-level counterfactual analysis weakens the justification for the paper's central contrastive learning mechanism. More controlled experiments would be necessary to establish whether the extracted guidance reliably captures effective intervention strategies rather than spurious correlations.
- Limited evidence for the generalizability of learned Meta-Skills. A central motivation of the paper is to learn reusable knowledge that can guide harness evolution across tasks. However, the experiments construct separate Meta-Skills for each benchmark and model backbone, without evaluating whether the learned knowledge transfers across different task domains or underlying models. Consequently, the results primarily demonstrate improvements within the same benchmark and backbone settings, rather than broader transferability. This limitation weakens the paper's central claim of reusable evolution knowledge and raises concerns about the scalability and practical applicability of the proposed framework beyond the evaluated settings.
Questions for authors
-
Given that the A/B/C comparisons rely on independently sampled trajectories, to what extent can the learned intervention guidance be reliably attributed to harness modifications rather than stochastic variations in agent execution? Although the paper acknowledges that these comparisons do not establish checkpoint-level causality, could the authors provide controlled counterfactual experiments that compare intervention decisions from identical intermediate agent and environment states to further validate the effectiveness of the extracted guidance?
-
The experiments demonstrate the effectiveness of Meta-Skills learned and evaluated within the same benchmark and model backbone, but their transferability across different task distributions or backbones remains unclear. Could the authors clarify the intended scope of knowledge reusability and investigate whether a Meta-Skill learned in one setting can effectively guide harness evolution in another without additional offline optimization?
-
The proposed contrastive learning framework relies on both the privileged verifier-informed evolver (condition C) and the targeted solicitation procedure, yet their individual contributions are not explicitly isolated in the current ablation studies. Could the authors provide additional controlled ablations to disentangle the effects of these components, ideally under comparable offline computational budgets? In particular, how much of the observed improvement can be attributed to verifier-informed contrasts versus the additional informative trajectories generated through targeted solicitation?
5 comments
Q2: The transferability of Meta-Skills across different task distributions or backbones remains unclear. Could the authors clarify the intended scope of knowledge reusability and investigate whether a Meta-Skill learned in one setting can effectively guide harness evolution in another without additional offline optimization?
We thank the reviewer for this question. Meta$^2$ is designed to reuse one frozen Meta-Skill across unseen tasks within a deployment setting (a given backbone and benchmark), and all test tasks in our evaluation are held out from Meta-Skill construction. To evaluate transfer beyond this setting, we directly apply a Meta-Skill constructed in one setting to another, without any additional offline optimization. The task agent harness, checkpoint schedule, and evaluation protocol are the same as in the main experiments. The results are as follows:
Cross-backbone transfer (Terminal-Bench 2.1):
| Meta-Skill source → target backbone | Task agent alone | Evolver without Meta-Skill | Transferred Meta-Skill | Meta$^2$ (built on target) |
|---|---|---|---|---|
| DeepSeek-V4-Flash → GPT-5.6-Terra | 66.33 | 74.67 | 74.67 | 81.00 |
| GPT-5.6-Terra → DeepSeek-V4-Flash | 70.00 | 72.33 | 73.00 | 77.67 |
Cross-benchmark transfer (GPT-5.6-Terra):
| Meta-Skill source → target benchmark | Task agent alone | Evolver without Meta-Skill | Transferred Meta-Skill | Meta$^2$ (built on target) |
|---|---|---|---|---|
| DeepSWE 1.1 → Terminal-Bench 2.1 | 66.33 | 74.67 | 73.33 | 81.00 |
| Terminal-Bench 2.1 → DeepSWE 1.1 | 34.67 | 36.44 | 38.67 | 44.44 |
- Cross-backbone: The Meta-Skill built with DeepSeek-V4-Flash does not exceed the evolver without Meta-Skill on GPT-5.6-Terra, and the one built with GPT-5.6-Terra brings a slight gain on DeepSeek-V4-Flash. Since different models have different behavioral tendencies and error patterns, the interventions that pay off differ across backbones. Therefore, a separate offline learning process for each backbone yields better performance.
- Cross-benchmark: The Meta-Skill built on DeepSWE 1.1 slightly lowers performance on Terminal-Bench 2.1, while the one built on Terminal-Bench 2.1 improves DeepSWE 1.1 over the evolver without Meta-Skill. DeepSWE 1.1 consists of relatively specialized software engineering tasks, whereas Terminal-Bench 2.1 covers a wide range of tasks with software engineering as one category, so its Meta-Skill contains guidance that also applies to DeepSWE 1.1.
Intended scope of reuse. We recommend learning a separate Meta-Skill for each backbone and benchmark, since different backbones and tasks exhibit different error types. The intended reuse unit is a deployment setting: one frozen Meta-Skill guides multiple unseen tasks with a given backbone and benchmark, without task-specific offline relearning. The cross-backbone and cross-benchmark experiments additionally probe transfer beyond this setting and show that target-specific learning delivers the strongest performance in the evaluated pairs.
Paper changes: We have added these results to Appendix E.9 (referenced in Sec. 4.2).
Q3: The individual contributions of the verifier-informed evolver (condition C) and the targeted solicitation procedure are not isolated. Could the authors provide controlled ablations under comparable offline budgets, and how much of the improvement comes from verifier-informed contrasts versus the additional trajectories generated through targeted solicitation?
We thank the reviewer for this suggestion. Under a fixed offline budget, the improvement comes from the verifier-informed contrasts of condition C: adding C improves over A + B by 4.00 / 7.11 pts on TB2.1 / DeepSWE 1.1. In Meta$^2$, targeted solicitation is realized through condition C on every training task: the evolver receives the task's privileged verifier information, is instructed to make only general harness edits and never to pass this information to the task agent, and a post-hoc audit removes trajectories with suspected leakage. The contribution of solicitation is therefore the contribution of condition C, which we isolate below.
Budget-controlled ablation of condition C. Full Meta$^2$ uses 17 rollouts per training task (5 / 5 / 7 under A / B / C). We keep this total fixed, so that every variant has the same number of trajectories, and vary only the conditions:
- B only (the self-distilled setting in Fig. 4): 17 rollouts under B; pass/fail pairs within each task are given to Reflect.
- A + B: 8 rollouts under A and 9 under B, so only the A–B contrast directions are available (B-pass/A-fail, B-fail/A-pass, A-pass/B-pass).
- A + B + C (full Meta$^2$): 5 / 5 / 7 rollouts under A / B / C.
All other components are identical, so the difference between A + B and A + B + C isolates the effect of verifier-informed contrasts at a fixed number of trajectories.
| Offline rollouts per task | TB2.1 | DeepSWE 1.1 |
|---|---|---|
| Task agent alone | 66.33 | 34.67 |
| Evolver without Meta-Skill | 74.67 | 36.44 |
| Evolver with Meta-Skill distilled from B only | 77.00 | 35.56 |
| Evolver with Meta-Skill distilled from A + B (new) | 77.00 | 37.33 |
| Evolver with Meta-Skill distilled from A + B + C (Meta$^2$) | 81.00 | 44.44 |
- More trajectories alone do not explain the gain. B only and A + B use the same number of rollouts as the full pipeline, yet A + B is only on par with or slightly above B only, and both stay below Meta$^2$.
- Verifier-informed contrasts account for the remaining improvement. At the same budget, adding C raises performance over A + B by 4.00 pts on TB2.1 and 7.11 pts on DeepSWE 1.1.
- C supplies a signal that A and B lack. A–B contrasts mostly reveal when the current evolver's interventions are unnecessary or harmful, and thus yield mainly hold and avoid guidance; on tasks where both A and B fail, they contain no successful recovery. C-pass/B-fail contrasts provide most act candidates in the full pipeline, including all act candidates for the terminal checkpoint, and on some training tasks they are the only informative contrast. These are the decision types that the counterfactual in Q1 identifies as beneficial, especially terminal send-back (+10.26 pts).
Paper changes: We have added this ablation to Sec. 4.5 and Appendix E, and clarified the implementation of condition C in Sec. 3.2 and Appendix B.
Thank you again for taking the time to review our paper and for the helpful feedback. We hope these clarifications and new results address your questions. If you have any further questions or suggestions, please do not hesitate to let us know.
Thank you for your positive and constructive review, and for recognizing the novel and well-motivated framework, the well-designed contrastive learning strategy, and the comprehensive empirical evaluation. We appreciate your suggestions on checkpoint-level causal attribution, the transferability of the Meta-Skill, and isolating the contribution of condition C under comparable budgets. We have conducted additional experiments on a checkpoint-level counterfactual from identical intermediate states, cross-backbone and cross-benchmark transfer, and a budget-controlled ablation of condition C. We address each point below.
We thank the reviewer for this question and for suggesting a controlled counterfactual. The learned guidance can be attributed to the evolver's decisions: in the suggested checkpoint-level counterfactual, executing a Meta-Skill-guided decision from an identical intermediate state raises the pass rate by 5.41 pts over withholding it. Below, we (1) describe the offline mechanisms that keep stochastic variation out of the Meta-Skill, (2) report this counterfactual, and (3) add detailed case studies.
(1) Offline mechanisms against spurious attribution. Three rules in the prompts (Appendix H) separate the evolver's effect from stochastic variation:
no_supervisor_effectand extracts no entry.(2) Checkpoint-level counterfactual. To test whether the Meta-Skill-guided decisions themselves improve outcomes, we execute or withhold each decision from the same intermediate state.
Results:
(3) Detailed case studies. Appendix D compares runs with and without the Meta-Skill on the same task, marking the KEEP / EVOLVE decision and the cited entry at every checkpoint; besides
configure-git-webserver, we added three cases covering a mid-run act entry, an avoid entry, and a hold entry.Paper changes: In the revision, we have added (1) to Sec. 3.2 and Appendix B.1, (2) to Appendix E.4,, and (3) to Appendix D.