acceptodds
Under review as a conference paper at ICLR 2027

Learning Meta-Skills for Agent Harness Design in Test-Time AI4AI

Abstract

Agent performance depends on both reasoning ability and the environment in which it acts. We study test-time AI-for-AI, asking how a **Builder** can learn to construct better execution environments for a **Target** while both models’ weights remain fixed. To make the Builder's experience reusable, we introduce **meta-skills**: principles specifying when support is needed and what resources to provide. The Builder learns these principles from Target's execution feedback on the development set, then uses the frozen skill bank to construct harnesses for unseen tasks. Across Harness-Bench and NewtonBench, full-bank meta-skills improve macro-average performance by 8.95 percentage points over no-skill construction, and 12.02 points over direct delivery of the same bank to the Target. These results highlight the value of translating experience into executable support. Gains when the same model serves both roles further suggest a path to self-improvement through learning to build better environments.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agent, Meta-Skill, AI4AI

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_F7WLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how a Builder can learn meta-skills from a Target’s execution feedback and use them to construct harnesses for new tasks. With both models’ weights fixed, the Builder translates learned support principles into tools, memory, and execution controls. Comparisons with no-skill construction, direct skill delivery to the Target, and independently learned Target skills support the value of learning and implementing external support.

Critical strengths
  1. The research question is clear and valuable. Learning how to support an agent provides a meaningful complement to learning task-solving skills directly.
  2. The experimental design is solid. The baselines closely match the central research questions. Experiments across models, component ablations, and outcome-transition analyses provide complementary evidence for the main findings.
Critical weaknesses

The case studies leave some ambiguity about how support design produces the gains. In the NewtonBench example, the Builder provides not only memory and failure-recovery mechanisms, but also specific scientific guidance, such as jointly inferring energy loss and using cross-mass probes. Success may therefore reflect both execution support and task-solving guidance, whose contributions are not clearly separated. By comparison, the checker and submission controls in the Harness-Bench example more directly illustrate translating workflow requirements into executable support. These cases may demonstrate different sources of improvement, and their relationship to the paper’s interpretation of meta-skills could be clearer. This does not undermine the overall performance gains, but it limits our understanding of what the method has learned.

Questions for authors
  1. In the NewtonBench case, does the specific scientific guidance come from the learned meta-skills, or does the Builder derive it from the current task? How can the contributions of task-solving guidance and execution support be distinguished?
  2. Could the authors show the relevant meta-skills verbatim and map them to the generated instructions, tools, or control logic, explaining what changed relative to the no-skill Builder? This would clarify the connection between learned experience, support design, and improved execution.
2 comments
AuthorsOpenReview ↗

Follow-Up on Our Rebuttal Responses

Dear Reviewer F7WL,

As the discussion period is coming to a close, we would like to kindly follow up and ask whether our responses have adequately addressed your concerns, or if you have any further questions.

If your major concerns have been resolved, we would greatly appreciate it if you could reflect this in your final score and confidence updates. Thank you very much for your time and thoughtful feedback!

Best regards, Authors of Paper 27157

Thanks. All Concerns are solved. I think the paper meets the bar for ICLR. So I keep my score and support the authors.

Reviewer_TuQnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a framework that enables a model to build the harness (builder) for a model that solves specific tasks (target) with meta-skills it learned from the target's performance and feedback from the development environment. Then, the meta skill is paired with the builder model on the test set to generate a fresh harness from scratch on test tasks. The paper shows the benefits of using the meta skills learned on the development set of test tasks, and ablations show that the meta skill is indeed transferable across models and task types when building the harness and thus is a kind of meta skill. The paper shows the gains on HarnessBench and NewtonBench and shows that with GPT 5.6 Sol as the Builder, it improves consistently across 3 models with full meta skill bank enabled to the builder, as usually the best setting compared to different baselines (retrieving meta skills, no meta skills enabled but harness update enabled, plain answer with the target model, some other baselines of target skills while the target tries to solve the problem). Additional ablation study reuses of meta-skills across target model types, although the transfer experiments do not show generalization to unseen task family types.

Critical strengths

The concept of meta-skill learning is relatively novel, though it may have some conceptual overlaps with concurrent works, as discussed in the paper, and it is quite useful in practice, as demonstrated in experiments, though cross-benchmark generalization is not established. The pipeline is mostly inference time and without model parameter updates. With a strong harness revision model (builder), it can learn what useful edits and harnesses are.

Critical weaknesses

The demonstrated generalization is limited to held-out instances within familiar task categories or physical mechanisms, with separate skill banks learned for each benchmark. Transfer across benchmarks or unseen task families remains untested, while transfer across model configurations yields mixed results. Consequently, the evidence establishes the usefulness of learned harness-design principles within the evaluated settings, but provides limited support for their broader transferability.

Questions for authors
  1. Can you demonstrate that the "meta-skills" learned from one benchmark can be transferred to other benchmarks?
  2. I have some general concerns about the hyperparameters used in experiments: for instance, why is temperature 0 used for models? I remember 0 is not a good temperature to use for multiple sampling, reasoning mode, and answer diversity. I wonder if this may lead to changing of the evaluation results. Can you clarify if 192 tokens is enough for the skill recording for both the target's own skills and the builder's skills?
  3. What evidence shows the additional value of the proposed when/provide/use format of skills or meta-skills? Have you evaluated a Builder receiving unstructured skills extracted from the same development set?
  4. Although there is a related works section discussing the difference to MCE [1] (ICML 2026), which also learns skills that guide an agent for a task-solving model. Could the authors discuss more on the methodology differences? Also, can you clarify the difference between your method and Skill-MAS [3] (EMNLP 2026)? One arXiv paper, though not a peer-reviewed one, HiSME [2], also explores a similar meta-skill learning setting. If it is possible, can the authors discuss this (not required by the policy but just want to have a better understanding).

[1] Ye, Haoran, et al. "Meta context engineering via agentic skill evolution." arXiv preprint arXiv:2601.21557 (2026). [2] Li, Xujun, et al. "You Live More Than Once: Towards Hierarchical Skill Meta-Evolving." arXiv preprint arXiv:2605.28390 (2026). [3] Lin, Hehai, Qi Yang, and Chengwei Qin. "Skill-MAS: Evolving Meta-Skill for Automatic Multi-Agent Systems." arXiv preprint arXiv:2606.18837 (2026).

Reviewer_2CHyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks: in an AI4AI setup (a Builder designs a harness/environment for a Target agent, both weights frozen), can the Builder learn reusable meta-skills—principles of when / provide / use support—from Target execution feedback on development tasks, and then reuse them on held-out tasks? They freeze the meta-skill bank at test time, retrieve top-2 via BM25, and show +8.95 points on Harness-Bench / NewtonBench over no-skill construction, and that meta-skills help more when given to the Builder than when directly injected into the Target.

Critical strengths
  • The conceptual move is interesting: instead of teaching the Target better reasoning, teach the environment designer what support to lay down. The Builder-vs-Target transfer comparison (Sec 5.2, same model self-improvement +18.71 points) is the most novel part.
  • The ablation design is thoughtful: native, no-skill, direct Builder skills, independent structured Target skills, mined free-form Target skills, retrieved vs full bank. That's a lot of controls and they mostly point the same way.
  • The cross-Builder/cross-Target transfer experiment (Sec 5.3) is a genuine stress test: Sol-to-Qwen transfer works, Sol-to-Gemini-Pro doesn't, which is honest rather than cherry-picked.
Critical weaknesses
  • The loop is hard to follow. Builder observes Target → writes a principle → constructs a harness → Target executes → feedback → revise skill. The reader has to infer from Table 1 and Sec 3.1 what a "meta-skill" actually looks like in text; only one illustrative example is given. A concrete 10-line dump of the skill bank after one development task would have helped a lot.
  • Numbers move a lot between settings. +8.95 aggregate sounds nice, but on NewtonBench the no-skill Builder is already at ~52 and the full bank at ~65; on Harness-Bench the gain is uneven and the Qwen curve in Fig. 2 actually regresses after the second pass. The paper acknowledges this but doesn't explain when refinement helps vs hurts.
  • Baseline "independent structured Target skills" and "mined free-form Target skills" are built from the same GPT-5.6-Sol critic—so the comparison partly measures "Builder-readable vs Target-readable skill representation" rather than the meta-skill idea itself.
  • No error analysis of why a skill fails. The paper reports gains and regressions but doesn't show a case where a retrieved meta-skill actively misleads the Builder (presumably they exist, given BM25 retrieval on ~20 skills).
Questions for authors

Q1. Retrieval is BM25 over ~20 skills. With a pool this small, why not just stuff all skills into the context? The "retrieved vs full bank" comparison in Table 3 shows full bank wins by 7.19 points on NewtonBench—isn't that the more important finding, and BM25 just a compromise? Q2. The cross-Builder transfer results (Sec 5.3) are honest but confusing: Sol→Qwen works, Sol→Gemini-Pro fails. Is this because skills encode Sol-specific phrasing (e.g., "call the file-versioning tool") that doesn't map to Gemini's toolset? A skill-inspection ablation would clarify.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.