acceptodds
Under review as a conference paper at ICLR 2027

Learning from Revision Consequences: Hindsight Meta-Experience Distillation for Self-Improving Agents

Abstract

As agents continuously improve by generating and revising Skills, the process that discovers and refines those Skills becomes a learnable object in its own right. Task-Skills directly act on task execution, whereas Meta-Skills govern how agents discover and improve future Skills; their value therefore emerges through the subsequent search processes they induce. Existing approaches improve Meta-Skills from observed raw Skill-search trajectories and branch outcomes. However, branch performance entangles the effects of the initial discovery state and the Meta-Skill revision that generated the search process, making it difficult to characterize what a particular revision actually changed, and potentially favoring revisions evaluated from advantageous starting states. We introduce HMED (Hindsight Meta-Experience Distillation), a mechanism for constructing Meta-Experience for self-improving agents. HMED restores the entry state of the completed search event cited by the revision and re-executes the incumbent and revised Meta-Skills from that state, so that the changes associated with the revision can be observed under a shared condition. Each comparison is distilled into a Meta-Experience, a structured record that can be reused by future updates, so that even revisions that are not ultimately retained still contribute a learning signal. Across three interactive agent benchmarks and two representative large language models with agentic capabilities, HMED consistently improves Skill discovery performance over strong baselines, shifting Meta-Skill learning beyond branch outcomes toward the consequences of changing the improvement process.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Self-Improving Agent, Meta-Skill

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2KDqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a hierarchical skill self-improving method for LLM, where Meta-Skills guide the revision of lower-level Task-Skills. The paper further introduce Hindsight Meta-Experience Distillation, which re-evaluates Skills through state replay and stores the comparison results as historical memory. The method is evaluated on models including Qwen3-32B and DeepSeek-v4-Flash, achieving better performance than methods such as MetaSkill-Evolve.

Critical strengths

Major Strengths

The idea of the paper is interesting, and the proposed method brings performance improvements over existing methods.

Minor Strengths

The paper is clearly written.

Critical weaknesses

1.The proposed method relies on state replay to re-evaluate Skills. However, accurately reproducing historical states can be difficult in many real-world scenarios, such as Agent tasks involving external APIs or robotic environments. This may limit the applicability of the proposed method.

  1. The contribution is largely incremental compared with MetaSkill-Evolve. In particular, the performance improvements in Table 4 are very limited, while the proposed method incurs an additional 27% computational cost. Without sufficient statistical evidence and comparisons under matched computational budgets, the effectiveness of the proposed method remains unclear.
Questions for authors
  1. How can the proposed method be applied to environments where historical states cannot be accurately reproduced?

  2. Can the authors report the standard errors of the results in Table 4 to further demonstrate the performance improvements?

  3. Can the authors increase the number of search iterations and candidate evaluations for MetaSkill-Evolve to match the computational cost of the proposed method?

4 comments

We thank the reviewer for the careful and constructive review. The questions about replayability, framework scope, and additional computation helped us clarify the interface required by HMED and the role of its matched continuations.

1. Replayability and environment assumptions

HMED does not require bitwise restoration of an external model provider or of every hidden state of a physical environment. It requires a serializable discovery state sufficient to execute a continuation, including the current Task-Skill, origin tasks, learning frontier, directive archive, visible search history, task configuration, and a verifiable entry snapshot.

For DeepSeek-v4-Flash, the model is accessed through an external API. The replay interface restores the discovery state relevant to continuation; it does not restore the provider’s hidden sampler state or guarantee bitwise-identical responses.

HMED therefore applies directly when a benchmark or system provides a replay or memoization interface for the relevant discovery state. Environments without such an interface require an adapter or an approximate replay protocol. The requirement is an interface, not a fully reversible environment.

2. Framework scope

HMED is implemented within a two-timescale Skill-evolution framework, but its mechanism requires an editable improvement policy, a provenance-linked discovery event with a restorable entry state, and an executable continuation for comparing incumbent and candidate policies.

The method is defined at this interface level rather than by MetaSkill-Evolve’s particular prompt format. GEPA and PromptBreeder do not directly expose all of these replay interfaces, so applying HMED to them would require an adapter. We have not evaluated HMED on other self-improvement frameworks.

3. Additional evolution cost

HMED incurs approximately 1.27×1.27\times the evolution cost of the original MetaSkill-Evolve protocol because revisions are evaluated through matched continuations.

We also tested a budget-augmented MetaSkill-Evolve baseline in the Qwen3-32B/ALFWorld setting. The augmented baseline used 1.21×1.21\times the original baseline cost and achieved 49.25% on the reuse protocol, whereas HMED achieved 50.75%. Relative to the same original reference, the residual evolution-cost difference is approximately 1.27/1.21−1≈5%1.27/1.21-1\approx5\%.

In the measured ScienceWorld downstream workload, the resulting policy reduced model calls by 6.9% and environment steps by 6.5%. The corresponding break-even estimate is approximately 6.2 equivalent execution rounds under that workload.

4. Is the gain only due to extra search budget?

The budget-sensitivity check shows that increasing ordinary proposal and discovery budget alone did not reproduce the reuse result in the tested setting. Additional search can increase coverage, but it does not directly construct paired evidence about the consequence of a particular Meta-Skill revision.

The two methods allocate their additional budget differently, so this check does not establish exact equality for every call, token, and environment component. It tests the simpler explanation that ordinary additional search alone explains the observed reuse gain.

5. What is specific to HMED

HMED treats the consequence of a particular Meta-Skill revision as the unit returned to future Meta-Skill updates. Matched continuation constructs this evidence; the Meta-Experience record serializes and reuses it.

This differs from retaining only a successful branch outcome. Valid non-adopted revisions also leave records that remain available to later updates. The contribution is the revision-centered, provenance-linked comparison record and its reuse in subsequent Meta-Skill updates.

We hope the replay clarification, budget check, and reuse results help clarify HMED's practical scope and contribution.

The authors have not addressed my question regarding standard errors. My concern is that the performance differences between the methods are very small, and the observed improvements may simply result from randomness across multiple runs. Reporting standard errors would help address this concern. Furthermore, the applicability of the proposed method remains limited in tasks where complete replay is not possible, such as robotics. In addition, the poor clarity of the main figure negatively affected my impression of the paper. Therefore, I have decided to maintain my rating.

Thank you again for the follow-up clarification. We sincerely apologize for overlooking the request for standard errors in our previous response. The reviewer’s concern is well taken, and we should have reported these statistics directly.

We retrieved and checked the original reuse experiment records and provide the run-level results and corresponding standard errors below:

ModelMethodSuccessful tasks in each runMean ± SE (%)
Qwen3-32BMetaSkill-Evolve64/134, 68/13449.25 ± 1.49
Qwen3-32BHMED67/134, 69/13450.75 ± 0.75
DeepSeek-v4-FlashMetaSkill-Evolve98/134, 106/13476.12 ± 2.99
DeepSeek-v4-FlashHMED102/134, 104/13476.87 ± 0.75

HMED achieves a higher mean in both reuse settings, and the reported standard errors quantify the observed variation across runs. These results support the reuse of the learned Meta-Skill after resetting the Task-Skill library, consistent with the broader improvements in the main experiments. Table 4 is intended to evaluate reuse across fresh discovery origins, while the main experiments provide the primary performance comparison.

Regarding robotics, we agree with the reviewer that irreversible physical interactions create a genuine applicability constraint. HMED requires a restorable discovery state and the ability to execute comparable continuations. Access to an external LLM API does not imply that a physical environment can be fully replayed. The evaluated protocol therefore applies to replayable settings; extending it to irreversible real-world interactions would require an appropriate replay mechanism or a validated approximation.

We also appreciate the reviewer’s observation about the main figure. This is a formatting and layout issue rather than a change to the method itself. We will enlarge the labels, reduce visual crowding, and distinguish the adoption and feedback paths more clearly so that the transition from the cited discovery event to matched continuation and subsequent update is easier to follow.

Thank you again for the careful reading and for identifying these specific points. We hope that the corrected statistics, the additional clarification of the replay scope, and the planned figure improvements address the remaining concerns and provide a stronger basis for reconsidering the rating.

Thank you to the authors for providing the standard errors. These results further reinforce my original concerns. The extremely small differences in mean performance, together with the large standard errors, make the experimental results highly unconvincing. Therefore, I have decided to maintain my rating.

Reviewer_eMYWOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how a self-improving agent can learn the policy that governs its own skill discovery and refinement, which the authors call a Meta Skill. The core observation is that existing methods update this policy from branch level search outcomes, and such outcomes confound the starting discovery state with the effect of a particular revision, so a successful branch does not reveal whether the revision itself helped. The proposed method, Hindsight Meta Experience Distillation, grounds each proposed revision in a completed search event, restores the entry state of that event, re-runs the incumbent and revised Meta Skills from the same starting points, and distills the paired comparison into a structured record. This record serves two roles: it decides whether the revision is adopted, and it is fed back into how future revisions are proposed. Experiments across three interactive agent benchmarks and two language models show consistent improvements over skill evolution baselines, and additional studies separate the value of comparison based selection from the value of feeding revision consequences back into later updates, including a reuse protocol that freezes the learned Meta Skill and resets the skill library.

Critical strengths
  1. The problem identification is precise and consequential. The confound between the starting discovery state and the effect of a Meta Skill revision is a real flaw in branch level learning signals, and the paper articulates clearly why a favorable branch can mask a weak revision while a difficult starting point can bury a useful one. Framing the needed experience unit around the revision itself, rather than the branch, is a genuine conceptual contribution to self-improving agent research.

  2. The mechanism directly addresses the stated confound and is thoughtfully designed. Restoring the entry state of the cited event and running matched continuations isolates the revision as the intended experimental difference, and the decision to retain records of valid but non-adopted comparisons means even rejected revisions still contribute a learning signal to future updates. The distillation step is deterministic and requires no additional model call, which keeps the feedback loop simple and reproducible.

  3. The evaluation pursues attribution rather than only headline numbers. The ablation that removes consequence feedback while keeping matched selection cleanly separates selection value from learning value, and the analysis of disagreement between branch outcomes and matched contrasts supports the central claim that the two signals are not interchangeable. The reuse protocol, which freezes the learned Meta Skill and resets the skill library at new discovery origins, tests whether the learned improvement policy transfers beyond the stored skills themselves.

Critical weaknesses
  1. The adoption signal rests on a small and noisy evidence base. Each comparison aggregates contrast over only two restored states, and each arm at each state is a single sampled execution subject to model and environment randomness, so the paired result is only a local observed contrast. Reported results are means over very few runs with no variance or significance analysis, making it hard to judge how reliable the matched signal is in individual windows, or how stable the final gains over the strongest baseline are across runs.

  2. The method is tightly coupled to one specific two-timescale evolution framework, and its scalability beyond the studied setting is underexamined. The extra matched continuations make evolution noticeably more expensive, the amortization argument relies on assumptions that are only checked in one supplementary workload, and the paper does not discuss behavior when the skill library, the visible history, or the number of update windows grows, nor whether the approach transfers to other self-improvement frameworks.

Questions for authors
  1. Is the improvement over the strongest baseline stable across individual runs, and what is the run-to-run variance? This materially affects how much weight the main table can bear.

  2. How does the method behave as the skill library and history grow, and does the bounded serialization of past records become a bottleneck for later update windows?

1 comment

We thank the reviewer for the careful and constructive assessment. The review highlighted the importance of run-to-run reliability, evolution cost, and longer-horizon resource growth. We provide updated three-run results and clarify the scope of the implementation.

1. Run-to-run results

The updated main results are means over three independent runs with sample standard deviations:

ModelBenchmarkMetaSkill-EvolveHMED
Qwen3-32BALFWorld51.74 ± 1.5554.23 ± 0.43
Qwen3-32BScienceWorld20.67 ± 0.2522.54 ± 0.44
Qwen3-32BBiomniBench-DA18.09 ± 0.6420.95 ± 1.03
DeepSeek-v4-FlashALFWorld77.36 ± 1.8881.09 ± 1.14
DeepSeek-v4-FlashScienceWorld28.79 ± 0.7732.66 ± 0.47
DeepSeek-v4-FlashBiomniBench-DA37.29 ± 4.7842.29 ± 2.36

HMED achieves the highest mean in all six configurations and is higher in all 18 corresponding run-level comparisons.

For the reuse protocol, the two runs were:

ModelMetaSkill-EvolveHMED
Qwen3-32B47.76, 50.7550.00, 51.49
DeepSeek-v4-Flash73.13, 79.1076.12, 77.61

The corresponding means are 49.25 versus 50.75 and 76.12 versus 76.87. The corresponding standard errors are 1.49 and 0.75 for Qwen3-32B, and 2.98 and 0.75 for DeepSeek-v4-Flash.

2. Framework scope

HMED is implemented in a two-timescale Skill-evolution framework, but its mechanism depends on four interfaces: an editable improvement policy, a provenance-linked discovery event with a verifiable entry state, and an executable continuation for comparing incumbent and candidate policies.

GEPA and PromptBreeder optimize prompts but do not directly expose this complete provenance-linked replay interface. Applying HMED to those systems would require an adapter. We have not evaluated HMED on other self-improvement frameworks.

3. Evolution cost and downstream reuse

HMED incurs approximately 1.27×1.27\times the evolution cost of the original MetaSkill-Evolve protocol. In a Qwen3-32B/ALFWorld budget-sensitivity check, the augmented MetaSkill-Evolve baseline used 1.21×1.21\times the original cost and achieved 49.25% on reuse, while HMED achieved 50.75%. The residual cost difference relative to the same original reference is approximately 1.27/1.21−1≈5%1.27/1.21-1\approx5\%.

In the measured ScienceWorld downstream workload, the resulting policy reduced model calls by 6.9% and environment steps by 6.5%, corresponding to a workload-specific break-even estimate of approximately 6.2 equivalent execution rounds.

4. Bounded feedback and growth

HMED stores complete comparison records, while each subsequent updater receives a bounded serialization containing at most the two most recent comparison records. The source-event lookup menu is a separate provenance mechanism and is bounded independently.

This controls the number of comparison records exposed to each updater. The underlying Task-Skill archive, restorable discovery-state storage, and total execution history may continue to grow with search. Thus bounded serialization controls updater context without implying constant total memory or computation.

We hope the updated run-level results, cost analysis, and protocol clarification make the reliability and practical scope of HMED clearer.

Reviewer_vNybOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Hindsight Meta-Experience Distillation (HMED) for improving textual policies governing agent skill discovery. Each proposed Meta-Skill revision is compared with the incumbent through continuations from matched starting states. The comparison informs revision selection and produces structured feedback for subsequent updates, including feedback from rejected revisions. Experiments cover three interactive benchmarks and two language models, supplemented by ablations, a reuse evaluation, and computational-cost analysis.

Critical strengths
  1. The method addresses a meaningful attribution problem. Branch outcomes reflect both the initial discovery state and the improvement policy. Comparing revisions from restored shared states is a sensible way to obtain more interpretable evidence about a proposed change.

  2. The feedback ablation is relevant to the central design. Retaining matched selection while removing consequence feedback helps distinguish learning from revision comparisons from merely filtering revisions. The reuse experiment also attempts to separate the contribution of learned Meta-Skills from retained Task-Skills.

Critical weaknesses
  1. The methodological novelty is limited, and the distinctive value of the experience representation is not sufficiently established. HMED combines established ideas in textual policy evolution, paired evaluation, and experience reuse within an existing Meta-Skill evolution framework. The integration is coherent, but the scientific contribution appears primarily incremental. The comparison-only ablation supports the usefulness of returning additional feedback, but does not establish that the proposed revision-centered representation is preferable to simpler alternatives. Comparisons against ordinary history summaries, raw comparison traces, or unpaired revision feedback under comparable context budgets would better isolate the contribution of Meta-Experience itself. Without these controls, it remains unclear whether the improvement comes from the specific experience construction or from providing the updater with more relevant historical information.
  2. The empirical evidence is not sufficiently strong to justify the additional computation and complexity. The main results report means over only two runs, without variability or confidence intervals. Several improvements are small, particularly the reuse gains of 1.50 and 0.75 percentage points, making their reliability difficult to assess. The principal feedback ablation is also restricted to one model–benchmark setting. Matched starting states control initial conditions but not execution randomness. In ALFWorld, adoption relies on short continuations from two states and a strictly positive observed contrast. Repeated comparisons or sensitivity analysis are needed to establish whether these decisions reliably identify better revisions. Finally, HMED incurs approximately 27% additional evolution cost, but the paper does not provide a budget-matched comparison in which the baseline spends the same resources on additional search or evaluation. Consequently, the reported gains do not yet establish that the proposed mechanism is a better use of computation.
Questions for authors
  1. Can the authors provide per-run results or uncertainty estimates, especially for the feedback ablation and reuse experiment? How stable are revision decisions under repeated matched continuations?

  2. Does HMED retain its advantage when baselines receive the same total evolution budget?

  3. How does structured Meta-Experience compare with simpler history summaries or unpaired revision feedback under matched context budgets?

2 comments

We thank the reviewer for the careful and constructive assessment. The review identified the central attribution question: whether HMED's gain comes from revision-centered experience construction or simply from exposing the updater to more relevant historical information. We address this question with the ablations, a raw-history comparison, and run-level results.

1. What the ablations isolate

The ablation conditions were evaluated under the original two-run protocol. The full HMED condition scores 54.11 in this set; the three-run re-evaluation of the same configuration in the main table scores 54.23 ± 0.43. The ablation differences below are computed within the two-run set.

Feedback returned to the updaterALFWorld
Matched comparison only48.88
Outcome-level record51.49
Outcome + process-level record (HMED)54.11
No event grounding47.76
MetaSkill-Evolve51.74

Adding outcome-level feedback raises the score by 2.61 points, and adding process-level consequences yields a further 2.62-point increase. The two increments are nearly equal. These results support the claim that revision-consequence records improve later Meta-Skill updates beyond matched comparison alone or outcome-level feedback alone.

A valid revision that is not adopted still produces a comparison record and can contribute to later updates. This is the feedback channel isolated by the comparison-only condition.

We also evaluated a raw-history variant in the same ALFWorld setting. It supplies the same two most recent comparison records as raw trajectory text rather than as distilled fields. The three runs scored 60, 62, and 68 of 134 tasks, corresponding to 47.26 ± 3.11%, compared with 54.23 ± 0.43% for HMED. The raw-history variant does not exceed MetaSkill-Evolve (51.74%) and lands close to the no-grounding condition (47.26% versus 47.76%).

The two feedback conditions were not token-matched, so this comparison varies both content organization and serialized length. We use it to show that raw trajectory text does not reproduce the observed gain; whether a raw paired trace serialized under exactly the same token budget would perform similarly remains a separate format question.

2. Run-level consistency

The updated main results are means over three independent runs, reported with sample standard deviations. The added run uses a new random seed while keeping the evaluation protocol fixed across methods; corresponding runs are paired.

ModelBenchmarkMetaSkill-EvolveHMED
Qwen3-32BALFWorld51.74 ± 1.5554.23 ± 0.43
Qwen3-32BScienceWorld20.67 ± 0.2522.54 ± 0.44
Qwen3-32BBiomniBench-DA18.09 ± 0.6420.95 ± 1.03
DeepSeek-v4-FlashALFWorld77.36 ± 1.8881.09 ± 1.14
DeepSeek-v4-FlashScienceWorld28.79 ± 0.7732.66 ± 0.47
DeepSeek-v4-FlashBiomniBench-DA37.29 ± 4.7842.29 ± 2.36

HMED has the higher mean in all six configurations and is higher in all 18 corresponding run-level comparisons.

For the reuse protocol, the two runs were:

ModelMetaSkill-EvolveHMED
Qwen3-32B47.76, 50.7550.00, 51.49
DeepSeek-v4-Flash73.13, 79.1076.12, 77.61

The corresponding means are 49.25 versus 50.75 for Qwen3-32B and 76.12 versus 76.87 for DeepSeek-v4-Flash. The corresponding standard errors are 1.49 and 0.75 for Qwen3-32B, and 2.98 and 0.75 for DeepSeek-v4-Flash.

These results characterize run-to-run variation; they do not provide a per-decision stability estimate for every individual matched comparison.

3. What matched continuation controls

The reviewer is right that matching the discovery state does not remove model or environment randomness. Matched continuation controls the systematic starting-state difference: incumbent and candidate are executed from the same restored source state and an additional pre-registered coverage state.

Each continuation runs for at most two discovery steps. The number of restored states is fixed at K=2K=2 in the audited segments. Invalid comparisons are not treated as negative evidence; the incumbent is retained, while the record is preserved for traceability.

The branch/matched-direction disagreement rates are 45.5% on ALFWorld and 75.0% on ScienceWorld. These results show that branch outcome and revision-centered consequence are not redundant signals, although they do not by themselves separate state confounding from execution randomness.

4. Evolution cost and budget sensitivity

HMED incurs approximately 1.27×1.27\times the evolution cost of the original MetaSkill-Evolve protocol because each revision is evaluated through matched continuations.

We also tested a budget-augmented MetaSkill-Evolve configuration in the Qwen3-32B/ALFWorld setting. The augmented baseline used 1.21×1.21\times the original baseline cost and achieved 49.25% on the reuse protocol, whereas HMED achieved 50.75%. Relative to the same original reference, the residual cost difference is approximately 1.27/1.21−1≈5%1.27/1.21-1\approx5\%.

This check shows that increasing ordinary search budget alone did not reproduce the observed reuse result in this setting. The additional budget is allocated differently by the two methods, so this is not a claim of strict equality for every compute component.

5. Scope of the claim

The evidence supports three conclusions:

  1. revision-consequence feedback improves later Meta-Skill updates beyond matched comparison alone and outcome-level feedback;
  2. event grounding contributes to the observed gain;
  3. the resulting Meta-Skill remains effective across fresh discovery origins under the reuse protocol.

We hope these updated ablations, the raw-history comparison, and the run-level results provide a stronger basis for assessing HMED's attribution and empirical support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.