acceptodds
Under review as a conference paper at ICLR 2027

MetaExp: Learning to Self-Improve through Experience via Verifiable Cross-Task Environment Scaling

Abstract

Language-model agents can accumulate interaction feedback, but reliably turning that feedback into reusable experience remains challenging. Existing methods either optimize external experience scaffolds or train experience-related behavior on specific task distributions, leaving open whether experience learning itself can become a transferable model capability. Training such a capability calls for supervision beyond isolated tasks, which mainly support in-episode problem solving rather than revealing what knowledge remains useful across episodes. In this work, we introduce MetaExp, a scalable framework that trains this capability through verifiable task families with controlled cross-task structure. Each family preserves a reusable factor while varying task-specific realizations, enabling transfer without direct solution reuse. We propose two complementary designs: Shared-Tool families keep black-box tools fixed across tasks to elicit operational experience, while Shared-Workflow families preserve latent procedures across changing realizations to elicit procedural experience. We scale environments under both designs, collect and filter sequential interaction trajectories, and fine-tune Qwen3.5-122B-A10B on 4,956 trajectories from 1,000 task families. With parameters frozen during downstream evolution, MetaExp improves by points on SpreadsheetBench, peaks at on APEX-Agents, and improves by on AutomationBench. The gains persist under domain-excluded training, supporting experience learning as a transferable capability for continued non-parametric improvement after deployment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM agent, experience learning, self-evolution, data synthesis

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_4am6OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MetaExp, a framework for training LLM agents to acquire, update, and reuse transferable experience through verifiable cross-task environments. It constructs Shared-Tool and Shared-Workflow task families to generate sequential execution and reflection trajectories for supervised fine-tuning. Experiments demonstrate improved experience-driven adaptation on three held-out agent benchmarks without further parameter updates.

Critical strengths
  • The paper addresses an important problem by treating experience learning as a transferable model capability rather than task-specific adaptation.

  • The complementary Shared-Tool and Shared-Workflow designs provide a scalable approach to generating structured cross-task supervision.

  • The evaluation covers heterogeneous benchmarks, domain-excluded training, and behavioral analyses, providing useful evidence of improved experience utilization.

Critical weaknesses
  • The training objective is not directly aligned with the claimed cross-task experience learning capability. Section 4.3 filters trajectories based on current-task success, while Eq. (8) merely imitates teacher behavior without evaluating whether experience updates benefit subsequent tasks. More importantly, the absence of independent-task SFT and frozen-bank controls makes it difficult to distinguish genuine experience learning from improved general task-solving or skill-utilization abilities.
  • The ablation studies do not convincingly establish the complementarity of the two task-family designs. In Table 4, the full model combines training data from both designs, whereas each single-design variant uses only one subset.
Questions for authors

see wekaness.

2 comments
AuthorsOpenReview ↗

Official Response from MetaExp Authors

Dear Reviewer 4am6,

Thank you for your constructive comments. We clarify the training design and its supporting evidence below.

W1(a). Alignment between the training objective and cross-task experience learning

Although Eq. (8) uses standard SFT, the supervised behaviors are shown within task chains: each execution is conditioned on experience accumulated from earlier family members, while each reflection uses execution feedback to create, revise, or retain skills for subsequent tasks. Cross-family interleaving further encourages selective reuse in the presence of irrelevant experience. Thus, training supervises experience-conditioned execution and feedback-driven updating rather than isolated task solving.

Importantly, filtering does not require task success. We retain trajectories with complete execution, evaluation, and reflection stages and valid skill artifacts, including those from failed tasks. These failures provide supervision for learning from negative feedback. We will clarify the ambiguous wording "verified task completion" in Section 4.3.

As discussed in Appendix D, explicitly optimizing downstream experience utility through utility-based filtering or cross-episode reinforcement learning is a promising extension of our current SFT approach.

W1(b). Distinguishing experience learning from general task solving and skill utilization

Our existing analyses provide complementary evidence for improvements in both experience utilization and updating.

  • Experience utilization. On SkillsBench, MetaExp performs better with identical curated skills (Table 2). Under SeqEvo, its gains are concentrated on tasks invoking accumulated skills (+9.5+9.5 points versus −4.2-4.2 for the backbone; Table 6), linking improvement to actual experience use.
  • Experience updating. MetaExp produces fewer, longer skills emphasizing failure diagnosis and verification, with fewer instance-specific details (Tables 7–8). On SpreadsheetBench, its bank grows from only 2.0 skills at k=5k=5 to 4.7 at k=40k=40, while total content increases from approximately 1.6K to 10.4K tokens and performance improves from 69.5 to 73.5. These patterns suggest continued consolidation and refinement of reusable experience rather than simple accumulation of task-specific records.

Moreover, experience-driven gains are not coupled with improvements in initial task-solving ability. On SpreadsheetBench, MetaExp's S(0)S(0) decreases from 68.9 to 65.0 (62.7 for Strict-OOD), yet Δ40\Delta_{40} improves from −2.2-2.2 for the backbone to +8.5+8.5 (+9.3+9.3 for Strict-OOD). Together, these findings support transferable experience-learning behavior beyond general task-solving improvements.

W2. Complementarity of the two task-family designs

The two designs exhibit distinct performance and efficiency patterns. Shared-Tool achieves stronger improvement on SpreadsheetBench (+9.7+9.7 versus +6.1+6.1 for Shared-Workflow), consistent with operational experience over stable tools. Shared-Workflow, meanwhile, reduces execution turns on APEX-Agents (−2.6-2.6) and AutomationBench (−2.4-2.4), consistent with learning reusable procedures.

Combining both designs yields broader benefits. The full model achieves positive gains across all three benchmarks, including +8.5+8.5 on SpreadsheetBench with only +0.5+0.5 additional execution turns, +1.8+1.8 on APEX-Agents (versus −0.5-0.5 and +0.4+0.4 for the single-design variants), and +14.8+14.8 on AutomationBench (versus +7.6+7.6 and +7.1+7.1), alongside a 7.07.0-turn reduction.

These differences are not simply monotonic with training-data volume: Shared-Tool alone (approximately 2.5K trajectories) achieves a larger SpreadsheetBench gain than the full model (approximately 5K), while Strict-OOD MetaExp reaches +9.3+9.3 with 3,634 trajectories. The observed patterns support the complementary roles of operational and procedural experience supervision.

Thank you to the authors for their detailed response, which has addressed all of my concerns. Taking into account the other reviewers' comments and the authors' rebuttal, I have decided to keep my original positive score.

Reviewer_HQASOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MetaExp, a framework for training LLMs to acquire, update, and reuse experience across unseen task distributions. It constructs verifiable cross-task families using Shared-Tool and Shared-Workflow designs to generate sequential execution and reflection trajectories for supervised fine-tuning. The proposed cold-start evaluation protocol assesses experience accumulation while keeping model parameters frozen.

Critical strengths
  1. The paper is clearly written, with strong motivation and a well-designed experimental evaluation.
  2. The paper investigates whether experience learning can become a transferable capability, addressing an important challenge in self-improving LLMs.
  3. Data Construction is interesting. The proposed Shared-Tool and Shared-Workflow task families provide structured supervision for learning reusable operational and procedural experience.
Critical weaknesses
  1. While the proposed cross-task family construction is well-motivated, the training trajectories are collected using a stronger teacher model, GLM-5.2. It would be valuable to investigate whether comparable improvements can be achieved when the training trajectories are collected using the same Qwen3.5 backbone.
  2. Please also conduct the Domain-Excluded OOM Evaluation on other benchmarks to further verify the generalization.
Questions for authors

Please see the weaknesses section.

2 comments
AuthorsOpenReview ↗

Official Response from MetaExp Authors

Dear Reviewer HQAS,

Thank you for your positive assessment and constructive suggestions. We address both concerns below.

W1. Can comparable gains be obtained with data collected by Qwen3.5 itself

Teacher supervision helps bootstrap effective experience-learning behavior. Our training requires demonstrations of experience acquisition, updating, and reuse, which the original Qwen3.5 backbone exhibits only weakly. On SpreadsheetBench, it invokes accumulated skills on just 14.1% of tasks, loses 4.2 points relative to matched empty-bank executions when doing so (Table 6), and produces more fragmented, instance-specific skills (Tables 7–8). Self-collected trajectories would therefore provide less reliable supervision without additional selection mechanisms.

As discussed in Appendix D, downstream-utility-based filtering and reinforcement learning with cross-episode rewards offer potential approaches to selecting or optimizing experience updates based on their benefits to subsequent tasks. These could also enable more effective self-bootstrapping, though at additional computational cost.

Importantly, the teacher is used only for trajectory collection. During evaluation, MetaExp operates with frozen parameters and an initially empty skill bank on unseen task distributions. Thus, the observed gains reflect transferable experience-learning behavior rather than the reuse of teacher-provided experience.

W2. Domain-excluded evaluation on additional benchmarks

We chose SpreadsheetBench for a targeted Strict-OOD evaluation, excluding three closely related training domains and approximately 27% of trajectories. Despite this reduction, MetaExp retains Δ40=+9.3\Delta_{40}=+9.3, with consistent gains across three adaptation orders (+8.0+8.0, +10.6+10.6, +9.4+9.4). Extending this evaluation to APEX-Agents and AutomationBench would require excluding a much broader range of overlapping domains, substantially changing both training volume and diversity. This would make it difficult to isolate the effect of domain exclusion from changes in the training distribution. We agree that broader domain-excluded evaluation would provide valuable additional evidence of generalization.

The paper is of high quality. Thank you for the rebuttal. I will keep my score.

Reviewer_9crMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether a model, after a single general training stage, can get better at acquiring, updating and reusing external experience on unseen task distributions, with frozen parameters and an initially empty experience store. Following the principle that experience should transfer without allowing direct reuse of solutions, the authors build two kinds of verifiable task families, Shared-Tool and Shared-Workflow, collect trajectories of a teacher model solving and reflecting on tasks in sequence, and fine-tune Qwen3.5-122B-A10B on them. Under a cold-start sequential evolution protocol, the trained model improves as experience accumulates on SpreadsheetBench, APEX-Agents and AutomationBench, whereas the same base model gains little or even degrades. The effect persists when related domains are removed from the training data.

Critical strengths

The problem setup and evaluation protocol are a contribution in themselves. SeqEvo measures zero-experience performance, the change from accumulated experience, and the ability to use externally provided experience separately. This gives learning from experience a measurement that is operational and comparable, and no longer mixed up with target-domain fine-tuning or a pre-built skill library. Details such as the cap on bank size, the data splits, three random orders and reported standard deviations are handled properly.

Several lines of evidence support each other. After removing three related domains and 27% of the training data, the Strict-OOD variant still reaches a Δ40 of 9.3, so the gain does not depend on nearby domains. The paired analysis in Table 6 shows that the base model does worse when it invokes its own skills, while MetaExp's gains are concentrated in trajectories where a skill was actually used, which partly separates the value of the experience content from other changes in behavior. Tables 7 and 8 show that MetaExp writes fewer and longer skills, with more content about failures and verification and no instance-specific IDs, in line with the training objective. Taken together, these are more convincing than a single Δ curve.

Critical weaknesses

A confound in the core claim has not been ruled out. The training data include assistant tokens from both the execution and reflection stages. From the domain tables in Figure 2, at k=0 the base model scores about 5.5 on AutomationBench against about 14.5 for MetaExp, and about 7.8 against 12.2 on APEX. So before any experience is collected, SFT has already raised the model's general agent ability under the Claude Code harness by a large margin, which has little to do with learning to learn from experience. Part of the larger Δ may therefore come from simply using the Skill tool in the harness more fluently. An SFT control trained on execution-stage trajectories only is needed. If its Δ is clearly lower, the paper's claim holds; if it is close to MetaExp, the contribution may mainly be harness adaptation. Table 4 currently ablates only the family type and not the composition of the training data. SFT also affects zero-experience performance in both directions, with k=0 on SpreadsheetBench dropping from 68.9 to 65.0, and the main text does not discuss this.

The comparison with system-level methods is made under different conditions. The EvoSkill and SkillGrad numbers in Table 1 are taken from Wang et al., with a different base model, protocol and number of verified trajectories, and the authors acknowledge that they are not directly comparable. An important argument of the paper is that system-level methods are limited by the frozen model's ability to diagnose and abstract, yet no system-level method is run with the same base model under SeqEvo. Without that, there is no way to tell whether a well-designed external reflection and skill management pipeline could give the untrained base model a similar Δ. Given that the gain on APEX is 1.8 with standard deviations of 1.4 to 1.7, this comparison is necessary to judge the size of the contribution.

Questions for authors
  1. Could you add a control model fine-tuned only on execution-stage trajectories, with all other data and settings unchanged, and report its S(0) and Δ(k) on the three benchmarks? Relatedly, why does SFT lower S(0) on SpreadsheetBench but raise it substantially on the other two benchmarks? Please list the aggregate k=0 results of the base model and MetaExp on all three benchmarks in the main text and discuss them.
  2. Can you run at least one system-level method, such as SkillGrad or Trace2Skill, with the same base model under the SeqEvo protocol?
  3. Table 6 shows that the base model also drops by 1.2 points on tasks where it does not invoke any skill. Why does it degrade as k grows on SpreadsheetBench? Is it because the skill bank takes up context, or because the skill descriptions interfere?
  4. The final 4,956 trajectories come from 1,000 families of five tasks each plus interleaved tasks from other families. How many trajectories were there before filtering, how many did each of the three filters remove, and what was the pass rate of the task and verifier synthesis stage?
  5. The student reaches a Δ of 8.5, while the teacher GLM-5.2 only gets 1.3 under SeqEvo. Is this because the teacher's S(0) is already near the ceiling, or because filtering kept only the reflection trajectories in which the teacher happened to succeed, creating a selection effect?
  6. The adaptation sets are small, 40 tasks for SpreadsheetBench and 20 per domain for AutomationBench. Can you report Δ separately for each of the three orders to show how sensitive it is to the adaptation order?
2 comments
AuthorsOpenReview ↗

Official Response from MetaExp Authors (1/2)

Dear Reviewer 9crM,

Thank you for your constructive feedback and for recognizing the strengths of our work. We address your concerns below.

W1 / Q1. Distinguishing experience learning from general agentic ability

We agree that distinguishing experience learning from general task-solving improvements is important. Our existing results provide complementary evidence that MetaExp improves how models use and organize accumulated experience.

1. Evidence of improved experience learning. On SkillsBench, where all models receive identical expert-curated skills, MetaExp improves the average score from 33.3 to 41.4 and pass rate from 47.1% to 62.1%, demonstrating stronger utilization of fixed external experience (Table 2). Under SeqEvo, the paired analysis (Table 6) further shows that, relative to matched empty-bank executions, the backbone loses 4.2 points on tasks where it invokes accumulated skills, whereas MetaExp gains 9.5 points. Thus, invocation alone does not guarantee improvement; the accumulated experience must also be useful and appropriately applied. Moreover, Tables 7–8 show that MetaExp consolidates feedback into fewer, more structured skills emphasizing failure diagnosis and verification. Together, these findings support improvements in both experience utilization and feedback-driven experience organization.

2. Understanding differences in zero-experience performance. Our training families emphasize long-horizon, multi-tool workflows, which more closely resemble APEX-Agents and AutomationBench than localized spreadsheet editing. This may explain why SFT improves S(0)S(0) on the former benchmarks but slightly reduces it on SpreadsheetBench, where execution turns at k=0k=0 also increase modestly (7.6 to 8.1). We will explicitly report the aggregate S(0)S(0) values:

ModelSpreadsheetBenchAPEX-AgentsAutomationBench
Qwen3.5-122B-A10B68.97.85.5
MetaExp-Qwen3.5-122B-A10B65.012.214.5

Importantly, despite its lower S(0)S(0) on SpreadsheetBench, MetaExp achieves Δ40=+8.5\Delta_{40}=+8.5, while the backbone declines by 2.2 points. The Strict-OOD variant exhibits the same pattern: it starts even lower (62.7) but reaches Δ40=+9.3\Delta_{40}=+9.3, exceeding the backbone's initial performance at k=20k=20 and k=40k=40 under all three adaptation orders (Q6). These patterns cannot be explained solely by improved zero-experience task-solving ability.

Additionally, we agree that execution-only SFT would help isolate reflection-stage supervision. However, since execution trajectories are themselves conditioned on accumulated experience, such a control would retain supervision of experience use while removing direct supervision of experience updating. Together with our fixed-skill evaluation, paired invocation analysis, and skill-bank analysis, the existing evidence supports transferable improvements in how MetaExp acquires, organizes, and uses experience, beyond general task-solving gains.

W2 / Q2. Comparison with system-level methods

1. Evidence under a fixed experience-management mechanism. MetaExp aims to improve the model's underlying experience-learning capability rather than redesign external experience-management pipelines. Under SeqEvo, all models share the same Claude Code harness, reflection prompt, and skill-creation mechanism, with frozen parameters and an evolving skill bank. On SpreadsheetBench, MetaExp gains 8.5 points, while its matched Qwen3.5 backbone declines by 2.2; even stronger models, Qwen3.6-Plus and GLM-5.2, achieve only −1.7-1.7 and +1.3+1.3, respectively. These results highlight the importance of the model's experience-learning behavior under a fixed external mechanism.

2. Complementarity with system-level approaches. Trace2Skill (Ni et al., 2026, Appendix G) evaluates several system-level methods with the same Qwen3.5-122B-A10B backbone on SPR-Verified, demonstrating that optimized external pipelines can also yield substantial improvements. However, its strongest setting relies on offline consolidation of 200 labeled target-domain trajectories, ground-truth-assisted analysis, and an expert-written initial skill, unlike our cold-start SeqEvo setting. While these results are not directly comparable, they illustrate the complementary focus of the two approaches: system-level methods improve how experience is externally managed, whereas MetaExp strengthens the model's ability to acquire, update, and reuse experience.

Regarding APEX-Agents, we acknowledge the modest aggregate gain and domain heterogeneity. At k=40k=40, MetaExp improves by 2.8 points in law and 5.0 in management consulting, compared with 0.8 and 0.9 for the backbone, while investment banking remains an exception (Section 5.2).

AuthorsOpenReview ↗

Official Response from MetaExp Authors (2/2)

Q3. Why does the backbone degrade without skill invocation

In Claude Code, non-invoked skills appear only as names and short descriptions in the skill inventory; their full contents are loaded only upon invocation. Thus, the small declines in the non-invoked subset (−1.2-1.2 for the backbone and −1.9-1.9 for MetaExp) may reflect stochastic variation, metadata interference, or selection effects, as skill invocation is model-dependent. This subset is particularly small for MetaExp (less than 20% of tasks), with substantial variability at k=40k=40 (−4.7±6.7-4.7\pm6.7). While these effects cannot be fully disentangled, the main behavioral difference lies in the invoked subset (−4.2-4.2 vs. +9.5+9.5), where accumulated experience is actually used.

Q4. Data construction statistics

Task-and-verifier synthesis required 5,812 attempts to obtain 5,000 validated tasks (approximately 86.0% acceptance). Cross-family interleaving reuses these tasks without introducing additional unique instances. Subsequent filtering removed 44 trajectories with incomplete execution, evaluation, or reflection, or invalid skill artifacts, yielding 4,956 training trajectories.

Importantly, filtering is not conditioned on task success: failed-task trajectories are retained as long as all stages complete and valid skill artifacts are produced. We will clarify the ambiguous phrase "verified task completion" in Section 4.3.

Q5. Why does the teacher gain less on SpreadsheetBench

GLM-5.2 starts at 88.3 on SpreadsheetBench, leaving considerably less headroom than MetaExp-Qwen3.5 (65.0). Moreover, our filtering does not require task success (Q4), so valid reflections from both successful and failed tasks are retained. The training data are therefore not restricted to successful teacher trajectories.

Q6. Sensitivity to adaptation order

All three runs use the same adaptation/target split with independently randomized adaptation orders. The per-order SpreadsheetBench results are:

ModelOrder 1 (Δ20\Delta_{20} / Δ40\Delta_{40})Order 2 (Δ20\Delta_{20} / Δ40\Delta_{40})Order 3 (Δ20\Delta_{20} / Δ40\Delta_{40})
Qwen3.5-122B-A10B−2.0-2.0 / −5.4-5.4−3.7-3.7 / −2.0-2.00.00.0 / +0.8+0.8
MetaExp+4.8+4.8 / +4.0+4.0+15.1+15.1 / +16.3+16.3+4.3+4.3 / +5.1+5.1
Strict-OOD MetaExp+10.9+10.9 / +8.0+8.0+11.7+11.7 / +10.6+10.6+6.9+6.9 / +9.4+9.4

Despite variation in magnitude, MetaExp consistently achieves positive gains exceeding those of the backbone under every order, while Strict-OOD MetaExp also maintains positive gains throughout. AutomationBench's aggregate results and standard deviations across three orders are reported in Figure 2. We will include these SpreadsheetBench per-order results in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.