acceptodds
Under review as a conference paper at ICLR 2027

OpenSpace: Skill Ecosystems for Self-Evolving Agent Harness

Abstract

Agent skills encapsulate reusable procedures imported from communities or abstracted from past executions. However, as libraries expand, overlapping scopes and uneven quality complicate retrieval, while static instructions fail to generalize across varying task contexts. We introduce OpenSpace, a self-evolving agent harness that transforms static skill collections into a dynamic ecosystem continually refined by execution evidence. OpenSpace operates through three coupled mechanisms: evidence-grounded acquisition (abstracting verified methods from raw trajectories), quality-driven maintenance (structurally integrating repairs and derivations while tracking empirical utility), and context-aware activation (dynamically binding optimal skills to the runtime environment). On EvoAgentBench, SkillFlow, and LiveMCPBench, OpenSpace delivers substantial gains in procedural transfer and continual task completion. Strictly without updating the foundation model's weights, it outperforms the strongest non-oracle baselines across all twelve backbone-domain settings on EvoAgentBench (by 5.4–34.5 percentage points) and boosts task completion on SkillFlow from 36.14% to 42.77%. Because OpenSpace evolves the external ecosystem rather than the internal model, it provides a highly effective, training-free paradigm to scale agent capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Skills, Self-Evolving Agents, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HmkUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OpenSpace, a novel harness that studies how agents can acquire, maintain, and reuse a skill library, transforming static skill collections into a dynamic ecosystem. It combines evidence-grounded skill acquisition, maintenance, and context-aware skill retrieval. Experiments show improvements on unseen tasks in EvoAgentBench and sequential tasks in SkillFlow and LiveMCPBench.

Critical strengths
  1. The framework provides a practical design for maintaining reusable skills, including creating new skills, repairing faulty instructions, and extending existing procedures, while tracking evidence for each version. It also reviews proposed skill updates and uses quality-aware retrieval to select skills based on their past performance.

  2. Experimental results show that OpenSpace outperforms the baselines in all EvoAgentBench settings, while enabling evolution improves its own performance on SkillFlow and LiveMCPBench.

Critical weaknesses
  1. It is unclear how much of OpenSpace’s advantage comes from its skill evolution method. Table 1 lacks an OpenSpace baseline with evolution disabled, while Table 2 shows that evolution gives both OpenSpace and Claude Code the same 6.63-point improvement.

  2. There is no ablation study that isolates the benefits of the review process, trust calibration, or the distinction between FIX and DERIVE.

Questions for authors

Please see the weaknesses above.

3 comments

Thank you for the careful review and constructive comments. We are glad you found the framework practical, in particular its version-tracked skill maintenance and quality-aware retrieval. We address each point below and will incorporate the new results into the revision.


Contribution of the evolution mechanism (W1).

Thank you for this question. To separate the contributions of the harness and of evolution in table 1, we ran two additional configurations on Gemma4 31B with the OpenSpace harness on EvoAgentBench: (i) evolution disabled, and (ii) our evolution module replaced by ReasoningBank[1], a standard experience-memory method.

Benchmark (Gemma4 31B)OpenClaw VanillaOpenSpace w/o evolutionOpenSpace + ReasoningBankOpenSpace (full)
BrowseComp-Plus13.3%21.5%27.7%27.7%
LiveCodeBench72.1%74.4%76.7%81.4%
SWE-bench Verified25.0%67.9%67.9%76.8%
GDPval44.1%56.7%50.0%58.3%
Macro Average38.6%55.1%55.6%61.1%
  • Harness. Without evolution, OpenSpace already outperforms every method in Table 1 on this backbone, including Anchor (GT) (44.5 on average). The harness is itself a core contribution of OpenSpace, and the revision will report its gain separately.
  • Evolution. With the harness fixed, evolution raises the average from 55.1% to 61.1% and improves all four benchmarks. The largest gains are on SWE-bench Verified (+8.9) and LiveCodeBench (+7.0). ReasoningBank[1] in the same harness reaches only 55.6%. It matches our method on BrowseComp-Plus, brings no gain on SWE-bench Verified, and lowers GDPval. The improvement therefore comes from the design of our evolution mechanism, not from adding experience memory in general.

For Table 2, the Claude Code results use SkillFlow's own skill-evolution procedure. Each system solves 11 more tasks with evolution, which gives the same +6.63 gain. OpenSpace starts from a much stronger baseline (36.14% vs. 28.31%), so the tasks left for it are harder. Its evolution still solves just as many additional tasks, which is a larger share of its remaining failures. Overall, OpenSpace reaches 42.77% versus 34.94%. This matches the EvoAgentBench ablation above: the harness and evolution each contribute substantial gains. We will add these configurations to the revision.

References

[1] S. Ouyang et al., "ReasoningBank: Scaling agent self-evolving with reasoning memory," in Proc. 14th Int. Conf. Learning Representations (ICLR), 2026.

Fine-grained component ablations (W2).

Thank you for this suggestion. Below we clarify how FIX and DERIVE differ, then report ablations on the evolution operations, the review process, and trust calibration. We choose SkillFlow to conduct evolution operations and review process since it can be run in parallel, while we run trust calibration on LiveMCPBench since it has a bigger skill library.

FIX vs. DERIVE. FIX applies when an existing skill is defective. It repairs the skill and creates a new version, and the old version is retired. DERIVE applies when an existing skill is reused on a new task. It builds a new skill that extends the original to new functionality, and the original stays available. FIX improves the quality of the library, while DERIVE expands its coverage.

FIX/DERIVE and review ablations (SkillFlow). On SkillFlow (MiniMax M2.5), we disable each operation in turn, as well as the review step:

ConfigurationCAPTUREFIXDERIVEReviewPass rateΔ vs. Full (pp)
Full✓✓✓✓42.77%—
w/o FIX✓—✓✓40.36%−2.41
w/o DERIVE✓✓—✓39.16%−3.61
CAPTURE-only✓——✓37.35%−5.42
w/o review✓✓✓—38.55%−4.22

Every component contributes. Removing FIX or DERIVE alone lowers performance by 2.41 and 3.61 points. Removing both drops it by 5.42 points, close to the sum of the two. This shows that the two operations provide complementary gains rather than overlapping ones. Disabling review costs 4.22 points. Review likely keeps captured skills from overclaiming capabilities they do not have, which would otherwise mislead skill selection on later tasks.

Trust calibration (LiveMCPBench). To test whether recorded skill quality improves later skill selection, we ran LiveMCPBench (MiniMax M2.5) with skill evolution on and turned off only the quality information used at retrieval time. This covers the ranking bonus, the filtering of candidates with a history of fallbacks, and the quality hints shown to the selector. The pass rate drops from 67.37% to 63.16%.

We also replayed the candidate lists of MiniMax and DeepSeek on the 95 LiveMCPBench tasks to examine the success threshold. By default, a skill must succeed on two distinct tasks before the selector receives a positive hint about it. With a threshold of one, 68.42% (MiniMax) and 67.37% (DeepSeek) of tasks would endorse at least one candidate after a single success. With a threshold of three, only 11.6% and 14.7% of tasks would be affected. The two-success threshold therefore prevents premature endorsement, and a stricter threshold changes little.

We will add these ablations and the threshold analysis to the revision, together with the clarification of FIX and DERIVE.

Thank you for your response and additional experiments. All of my concerns have been addressed.

Reviewer_YesyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OpenSpace is an agent harness that maintains an evolving library of reusable skills while keeping foundation model weights frozen. After each task execution, the framework extracts evidence from execution traces, categorizes updates into new skill creation via CAPTURE, defect repair via FIX, and specialized extension via DERIVE, tracks usage and empirical trust for each version, and uses agentic search to select and bind skills for subsequent tasks. The empirical evaluation covers procedural transfer on EvoAgentBench and continual online evolution on SkillFlow and LiveMCPBench. Across these benchmarks, the paper reports substantial gains over published baselines on EvoAgentBench and over OpenSpace variants with evolution disabled on SkillFlow and LiveMCPBench.

Critical strengths
  1. Sections 3.1 and 3.2 separate creating a new root skill via CAPTURE, repairing a defective skill in place via FIX, and deriving a specialized variant while preserving its parent via DERIVE. Because these three operations affect downstream reuse differently, assigning each operation explicit versioning rules and evidence requirements provides a structured foundation for maintaining a persistent library. In addition, distinguishing passive skill availability from active application enables accurate credit assignment when an agent succeeds only after abandoning a flawed procedure.

  2. Table 5 links specific library updates to downstream execution behavior, tracing a captured HWPX document cleanup procedure, a repaired Wikipedia title lookup workflow, and a derived direct API fallback. Showing the exact instruction changes and their reuse on later tasks makes the CAPTURE, FIX, and DERIVE operations concrete in practice. These qualitative cases confirm that the framework produces reusable procedural guidance that directly shapes future task execution.

Critical weaknesses
  1. The main EvoAgentBench comparison in Table 1 evaluates OpenSpace against published results from OpenClaw and Nanobot without testing OpenSpace with evolution disabled, which confounds the effect of skill evolution with differences in the underlying agent harness. Table 2 illustrates this ambiguity: enabling evolution adds 6.63 percentage points to both Claude Code and OpenSpace, yet OpenSpace holds a 7.83-point advantage even when evolution is turned off. While this confirms that evolution helps within OpenSpace, it fails to prove that the proposed mechanism accounts for the performance gap against other frameworks.

  2. In Section 4.3, the reported 14.28% token reduction on SkillFlow counts only solver tokens during task execution and excludes the tokens consumed by the analyzer, author, reviewer, and selector modules. Omitting these four LLM modules makes it impossible to tell whether OpenSpace reduces total token cost or actually spends more tokens overall to maintain and search the library. Moreover, because SkillFlow and LiveMCPBench update the skill library sequentially across tasks, the skills generated from early tasks determine what guidance exists for later tasks. Reporting only a single run under one fixed task order in Tables 2 and 3 therefore fails to show whether the online evolution gains hold up under different task arrival orders or repeated runs.

  3. The experiments disable evolution entirely without verifying whether FIX/DERIVE or attribution-sensitive trust outperform simpler baselines. The count of accepted updates in Appendix B does not resolve this issue, because acceptance does not guarantee that a proposed correction functions correctly or preserves prior performance.

Questions for authors
  1. On a representative EvoAgentBench setting, how do no evolution, a standard experience-memory approach, and the full method compare when using the identical OpenSpace harness, tools, backbone, and total inference budget? Please clarify what this comparison establishes beyond the pre-existing harness gap shown in Table 2.

  2. What occurs when FIX and DERIVE are substituted by capture-only updates, or when attribution-sensitive trust is replaced by basic usage and success logs? For updates that pass review, please report how often they produce the intended improvement in later runs. How often do they cause the agent to fail tasks it could solve before the update?

5 comments

We thank the reviewer for the thoughtful feedback and for recognizing the structured versioning rules behind CAPTURE, FIX and DERIVE, and the qualitative execution traces in Table 5. Your comments helped us better isolate the contribution of skill evolution and improve the clarity of our presentation. Below, we address each point with new controlled experiments.


On separating harness quality from skill evolution (W1 & Q1).

Thank you for this helpful comment. We agree that a comparison under the same harness is needed to separate the effect of skill evolution from the effect of the base agent. We ran the comparison asked for in Q1. We used Gemma4 31B as the backbone and covered all four EvoAgentBench domains. The OpenSpace harness, tools, backbone model, and total inference step were exactly the same in all three settings:

  • OpenSpace (w/o Evolution): The original OpenSpace harness with skill evolution turned off.
  • OpenSpace + ReasoningBank (Experience Memory): Our skill creation, revision, and review pipeline is replaced with ReasoningBank [1], a standard trajectory-level memory. After each training task, Gemma4 31B pulls up to three lessons from the trajectory, whether the task succeeded or failed. Before each new task, we find the most similar past training task in the same domain and give its lessons to the solver.
  • OpenSpace (Full): Our full pipeline for creating, revising, and reviewing skills.
Benchmarkw/o evolution+ ReasoningBankFull
BrowseComp-Plus21.54%27.69%27.69%
LiveCodeBench74.42%76.74%81.40%
SWE-bench Verified67.86%67.86%76.79%
GDPval56.67%50.00%58.33%
Macro Average55.12%55.57%61.05%

A strong base harness, and evolution improves it further. We agree with the reviewer that the OpenSpace harness is strong on its own. With evolution turned off, it already reaches a 55.12% macro average under Gemma4 31B, above vanilla OpenClaw (38.63%) and vanilla Nanobot (34.68%). This is consistent with the harness gap shown in Table 2. Our evolution mechanism still brings a clear gain on top of this strong base. It raises the macro average from 55.12% to 61.05% (+5.93 points), including +8.93 points on SWE-bench Verified and +6.98 points on LiveCodeBench. Gains are usually harder to get when the base agent is already strong, because many of the easier failures have already been fixed. So we see this improvement as strong evidence for the mechanism.

What this comparison adds beyond Table 2. Table 2 shows that evolution helps. The new comparison shows that the gain comes from our specific mechanism, not from reusing past experience in general.

We use ReasoningBank as a representative standard experience-memory method. It runs in the same harness and learns from the same past trajectories as our method, so the only difference is how that experience is used. On this strong base, ReasoningBank adds only +0.45 points on average and even drops on GDPval (50.00% vs. 56.67%), whereas Full Evolution improves the macro average by +5.93 points and outperforms ReasoningBank by +5.48 points.

We will add these results and this discussion to the revised paper. We hope they answer your question.

References

[1] S. Ouyang et al., "ReasoningBank: Scaling agent self-evolving with reasoning memory," in Proc. 14th Int. Conf. Learning Representations (ICLR), 2026.

Total Token Cost and Task Order (W2)

Thank you for these two suggestions. We ran additional experiments on both points and summarize the results below.

Total token cost. The 14.28% figure in Section 4.3 measures execution tokens only. Following your suggestion, we measured all five modules on all three benchmarks (SkillFlow and LiveMCPBench with MiniMax M2.5, EvoAgentBench with Gemma4 31B). The breakdown is below, and we will add the corresponding figure to Appendix.

Benchmarkw/o evolutionSolverSelectorExecution ΔAnalyzerAuthorReviewerWith evolution (total)Per solved task
SkillFlow54.7746.950.00−14.28%4.002.641.3554.93 (+0.29%)−15.25%
LiveMCPBench41.0934.081.54−13.33%2.453.101.9443.11 (+4.91%)−8.20%
EvoAgentBench107.8195.880.92−10.21%7.013.834.25111.89 (+3.79%)−6.21%

Execution = Solver + Selector. Token counts are in millions (M).

Execution tokens fall by 10.2–14.3% on all three benchmarks, and the analyzer, author, and reviewer add a modest overhead. With all modules included, total usage is within +0.3% to +4.9% of the no-evolution baseline. On SkillFlow, the execution savings nearly offset the evolution overhead. Because OpenSpace also solves more tasks, the cost per solved task falls by 6.2–15.2% on all three benchmarks.

Most of the evolution overhead goes into writing and reviewing skills, and a skill can be reused on later tasks once it exists. We therefore expect the overhead per task to shrink as the library matures.

In the revision, we will report the execution, total, and per-solved-task numbers in Section 4.3 and put the full breakdown in the appendix.

Task order and repeated runs. SkillFlow's task order is fixed by the official benchmark protocol, so we ran this study on LiveMCPBench (95 tasks, MiniMax M2.5). We repeated the full online-evolution run under two random permutations of the task order. Each permutation is an independent run from the same basic library, so the spread below reflects order effects and run-to-run variance together.

SettingPass rateΔ vs. w/o evolution
w/o evolution baseline (MiniMax M2.5)58.95%—
Original order (Table 3)67.37%+8.42
Shuffled exp 165.26%+6.31
Shuffled exp 268.42%+9.47
Mean ± std (3 orders)67.02 ± 1.61%+8.07

Task order does change the results, since earlier tasks shape which skills are available later. Across the three runs, pass rates vary within 3.2 points, and every run stays 6.3–9.5 points above the no-evolution baseline. In the revision, we will report LiveMCPBench as mean ± std across orders and add this table to the appendix.

On component ablation and the effect of accepted updates (W3 & Q2)

We thank the reviewer for this question. We agree that a count of accepted updates does not show whether an update works as intended or preserves earlier behavior. We ran additional experiments to address each part of the question.

FIX/DERIVE vs. capture-only updates. We ran this ablation on SkillFlow (MiniMax M2.5), where the full system reaches 42.77%. We removed each update type and kept the rest of the pipeline unchanged. Every variant scores below the full system, and the capture-only variant drops the most.

SettingCAPTUREFIXDERIVEPass rateΔ vs. Full (pp)
Full✓✓✓42.77%—
w/o FIX✓–✓40.36%−2.41
w/o DERIVE✓✓–39.16%−3.61
CAPTURE-only✓––37.35%−5.42

Attribution-sensitive trust versus basic logs. (EvoAgentBench, Gemma4 31B.) To isolate the trust mechanism, we kept the Analyzer, Author, and Reviewer, along with FIX and DERIVE updates. We changed only how skill trust is computed. In the basic-log variant, a skill version counts as used in a task whenever it is delivered to the agent or explicitly invoked, and it receives the task's overall outcome. If a task uses three skills, all three get the same success or failure record. The full system instead tracks how much each skill contributed to the outcome. Both variants see the same task-level success signal. They differ only in how that signal is assigned to individual skills.

BenchmarkFullBasic logsΔ
BrowseComp-Plus27.69%23.08%−4.61 pp
LiveCodeBench81.40%75.58%−5.82 pp
SWE-bench Verified76.79%69.64%−7.15 pp
GDPval58.33%58.33%0.00 pp
Macro Average61.05%56.66%−4.39 pp

Attribution helps on three of the four benchmarks, most of all on SWE-bench Verified. Overall, assigning credit to each skill raises the pass rate by 4.39 pp compared with giving every skill the same usage-and-outcome record.

Accepted updates: improvements and regressions in later runs. We selected the tasks whose execution reuses a skill updated in an earlier run and compared each task's outcome before and after the whole evolution. On both LiveMCPBench and SkillFlow, our method brings clear improvements in later runs. A small number of regressions also occur, which underscores that credit attribution and performance guarantees remain difficult problems in this area.

BenchmarkReused-task subset (N)ImprovedRegressedNet change
LiveMCPBench336/33 = 18.18%1/33 = 3.03%+15.15 pp
SkillFlow6210/62 = 16.13%0/62 = 0.00%+16.13 pp

Split by the outcome before the update:

QuestionLiveMCPBenchSkillFlow
Of the tasks that previously failed, how many are now solved?6/17 = 35.29%10/26 = 38.46%
Of the tasks that previously passed, how many now fail?1/16 = 6.25%0/36 = 0.00%

On both benchmarks, accepted updates rescue roughly a third of the previously failed tasks, while regressions on previously solved tasks are rare.

This reflects a deliberate cost trade-off. Directly verifying an update would require replaying previously solved tasks with the updated skill before accepting it, which adds extra task executions for every accepted update. We do not perform such replays. Instead, the only reward is each task's final success or failure, which the Analyzer attributes to the skills involved. The Reviewer screens each proposed update before it enters the library. Any regression that passes review surfaces in later runs, where attribution assigns the failure back to the updated skill and it becomes a candidate for another FIX. The results above show that this loop yields clear net gains with few regressions, without the cost of replay.

Reviewer_qMW7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OpenSpace, a training-free agent harness that maintains an evolving ecosystem of reusable skills. It combines evidence-grounded acquisition through CAPTURE, FIX, DERIVE, and NOOP operations; version-aware maintenance with usage-based trust tracking; and context-aware skill retrieval and activation. Experiments on EvoAgentBench, SkillFlow, and LiveMCPBench report improvements in procedural transfer and sequential task completion. The central contribution is a system design that connects skill evolution, explicit evidence attribution, and version-specific quality records while keeping foundation-model weights frozen.

Critical strengths
  1. Separating skill availability, actual use, content approval, and execution success is a useful design choice. Version records make updates easier to inspect.

  2. CAPTURE, FIX, and DERIVE provide a clear structure for skill maintenance. The combination with evidence tracking is interesting, and reported gains span multiple domains and backbones.

Critical weaknesses
  1. Table 1 does not establish matched tools, budgets, retries, or auxiliary models across harnesses. The released code also differs from historical snapshots and documents dependency conflicts. Results need links to exact configurations and artifacts, including test-time updates and feedback access. Unfairness or leakage has not been established.

  2. The experiments do not isolate review, FIX/DERIVE, or quality tracking. Both harnesses gain 6.63 points from evolution in Table 2. A same-budget skill-accumulation baseline and comparisons with related systems are needed to establish the specific contribution. Table 4 also changes both selection and instruction-delivery timing.

  3. Attribution accuracy is not measured. The Wikipedia diagnosis needs fuller evidence. Trust thresholds lack sensitivity tests, while capped counts depend on exposure and cancel when both reach the cap. Unexecuted updates may replace active skills, so first-use checks and recovery need explanation. These are unresolved risks, not demonstrated failures.

Questions for authors
  1. Could author add matched-budget baselines and component ablations, including fixed instruction-delivery timing?

  2. Could author provide exact run artifacts, feedback boundaries, repeated results, and judge settings?

  3. Can author audit attribution, test trust thresholds, explain update recovery, and report total costs and library-growth behavior?

4 comments

Thank you for your thoughtful and detailed review of our work. We appreciate your recognition of the separation of availability, use, approval, and success signals, as well as your positive feedback on the CAPTURE/FIX/DERIVE structure of the paper. Below, we address the weaknesses and questions you raised:


On experimental fairness and reproducibility (W1 & Q2).

We agree that exact configurations are essential for verifying fairness, and clarify them below.

Configuration alignment. For nanobot and OpenClaw, we use the scores reported by EvoAgentBench, which were obtained under the same budget, tools, and retry settings as our runs. For our own runs, we follow the EvoAgentBench configuration where one exists. Otherwise, we use the benchmark's original configuration with the official default settings of nanobot and OpenClaw.

Run settings. Each task runs for at most 100 agent steps, with at most 2 tool-call-level retries. The available tools are read, write, edit, and shell. All auxiliary components (analysis, skill generation, review, and retrieval) use the same backbone as the main agent in each experiment. Judges follow EvoAgentBench (GPT-4o for GDPval, Qwen3-32B for BrowseComp-Plus).

Feedback boundaries. No test-set feedback is used to update any method in EvoAgentBench. We follow the EvoAgentBench protocol, which its official self-evolving baselines also use. During training, OpenSpace uses the training tasks, its own trajectories (including tool returns), and official-scorer feedback on training tasks. At test time, the learned skill library is frozen and only retrieved and used.

Repeated results. We repeated the full online-evolution run on LiveMCPBench (MiniMax M2.5) under three task orders: the original order and two random permutations. Each run starts from the same basic library, so the spread reflects order effects and run-to-run variance together.

SettingPass rateΔ vs. w/o evolution (pp)
w/o evolution58.95%—
Original order67.37%+8.42
Shuffled order 165.26%+6.31
Shuffled order 268.42%+9.47
Mean ± std (3 orders)67.02 ± 1.61%+8.07

Pass rates vary within 3.2 points across orders, and every run stays 6.3–9.5 points above the no-evolution baseline. We will report LiveMCPBench as mean ± std in the revision and add this table to the appendix.

Code and artifacts. All experiments in the paper were run with the code released at https://anonymous.4open.science/r/OpenSpace-ICLR2027-2F8C/, which includes the dependency fixes and supersedes the earlier snapshots.

Ablation on each components and matched-budget baselines (W2 & Q1)

We thank the reviewer for this suggestion. We agree that the contribution of each component should be isolated, and we have run the additional experiments below. All results are pass rates, and the SkillFlow and LiveMCPBench experiments use MiniMax M2.5.

For the ablation on review and FIX/DERIVE, we choose SkillFlow since it can be run in parallel across different task families. For quality tracking, we select LiveMCPBench because skill quality issues are more likely to appear there. To align Table 1 (OpenSpace w/o evolve) with Table 4 (retrieval), we run the experiment on EvoAgentBench.

w/o FIX/DERIVE (SkillFlow). Removing any component lowers the pass rate, and the full system performs best. Capture-only, which is the skill accumulation baseline, is the lowest among the ablation results.

SettingCAPTUREFIXDERIVEPass rateΔ vs. Full (pp)
Full✓✓✓42.77%—
w/o FIX✓–✓40.36%−2.41
w/o DERIVE✓✓–39.16%−3.61
CAPTURE-only✓––37.35%−5.42

Every component contributes. Removing FIX or DERIVE alone lowers performance by 2.41 and 3.61 points. Removing both drops it by 5.42 points, close to the sum of the two. This shows that the two operations provide complementary gains rather than overlapping ones.

w/o review (SkillFlow). Without the review step, the pass rate falls from 42.77% to 38.55% (−4.22 pp). Disabling review costs 4.22 points. Review likely keeps captured skills from overclaiming capabilities they do not have, which would otherwise mislead skill selection on later tasks.

w/o quality tracking (LiveMCPBench). To test whether quality records improve later skill selection, we keep skill evolution on and turn off only the historical quality information used at retrieval. This covers the ranking bonus, the candidate filter based on past fallbacks, and the quality hints shown to the selector. On the 95 MiniMax tasks, the full system passes 67.37%. Without quality information, it passes 63.16%, a drop of 4.21 pp.

Evolve vs No Evolve OpenSpace (EvoAgentBench). We conduct the experiment to measure the evolution contribution beyond the SkillFlow bench. The backbone is Gemma4 31B.

BenchmarkOpenSpace (Full)w/o evolution+ ReasoningBank [1]
BrowseComp-Plus27.7%21.5%27.7%
LiveCodeBench81.4%74.4%76.7%
SWE-bench Verified76.8%67.9%67.9%
GDPval58.3%56.7%50.0%
Macro Average61.1%55.1%55.6%
  • Harness. Without evolution, OpenSpace already outperforms every method in Table 1 on this backbone, including Anchor (GT) (44.5 on average). The harness is itself a core contribution of OpenSpace, and the revision will report its gain separately.
  • Evolution. With the harness fixed, evolution raises the average from 55.1% to 61.1% and improves all four benchmarks. The largest gains are on SWE-bench Verified (+8.9) and LiveCodeBench (+7.0). The improvement therefore comes from the design of our evolution mechanism. ReasoningBank [1], a standard trajectory-level experience-memory method, is run in the same harness and learns from the same training trajectories as our method. It reaches only 55.6% on average, matches our method on BrowseComp-Plus, brings no gain on SWE-bench Verified, and lowers GDPval. The gain therefore does not come from reusing past experience in general.
  • Table 2. Both harnesses gain 6.63 points from evolution. OpenSpace starts from a stronger baseline (36.14% vs. 28.31% for Claude Code), so its remaining failures are harder, yet evolution still recovers the same number of tasks.

Selection vs. instruction-delivery timing (SWE-bench Verified in EvoAgentBench). To separate the two factors, we fix the injection timing to early injection for both methods and vary only the retrieval method.

MethodPass rateΔ vs. Full (pp)
Full agentic search + early injection76.8%—
Ranked skill retrieval + early injection71.4%−5.4

With timing held constant, the gap comes from the selection method. The results show that agentic search gains 5.4 pp over ranked skill retrieval. These results suggest that agentic search helps the agent make better use of the skill library.

References

[1] S. Ouyang et al., "ReasoningBank: Scaling agent self-evolving with reasoning memory," in Proc. 14th Int. Conf. Learning Representations (ICLR), 2026.

Attribution, Trust, and Cost of Skill Updates (W3 & Q3)

Thank you for this helpful comment. We will address your concerns step by step. We conducted further analysis and statistics on the data to present the results more clearly. We chose LiveMCPBench because its task sequence is longer than that of SkillFlow, which makes it easier to show skill-library growth behavior. LiveMCPBench experiments use MiniMax M2.5.

1. On attribution. We agree that attribution is central to skill evolution. We provide three pieces of downstream evidence related to the attribution.

The wikipedia diagnosis: In one run, the agent loaded a Wikipedia skill and requested the page NIO_(company). This title does not exist, so the call failed with a missing extract field. In the same run, the agent searched Wikipedia, found the correct title Nio Inc., and retrieved the content. FIX traced the failure to the title-lookup step and added one step to the skill: search for the correct title before fetching content. Later, an unrelated task loaded the revised skill to write a Python example on convolutional neural networks. The agent ran the new search step first, retrieved the matching article, saved it to /tmp/cnn_content.txt, and then wrote the requested file. A recovery that happened once became a reusable step for a different task.

Paired outcomes on reused skills. To see whether this pattern extends beyond a single case, we compared runs with and without evolution on the tasks that loaded an evolved skill. We counted only tasks with valid results in both runs. On LiveMCPBench, 18.2% of these tasks went from failure to success and 3.0% went from success to failure, for a net pass-rate gain of 15.2 percentage points. On SkillFlow, 16.1% went from failure to success and none regressed, for a net gain of 16.1 points. If outcomes were often credited to the wrong skill or the wrong step, evolved skills would carry those errors into later tasks, and regressions would be common. Instead, they are rare (3.0% and 0%), which is consistent with attribution landing on the right components.

Attribution-sensitive trust vs. basic logs (EvoAgentBench, Gemma4 31B). We keep the Analyzer, Author, and Reviewer with FIX and DERIVE, and change only how skill trust is computed. In the basic-log variant, every skill delivered to or invoked by the agent receives the task's overall outcome; the full system instead tracks how much each skill contributed.

BenchmarkFullBasic logsΔ
BrowseComp-Plus27.69%23.08%−4.61 pp
LiveCodeBench81.40%75.58%−5.82 pp
SWE-bench Verified76.79%69.64%−7.15 pp
GDPval58.33%58.33%0.00 pp
Macro Average61.05%56.66%−4.39 pp

Per-skill credit assignment helps on three of the four benchmarks, most of all on SWE-bench Verified.

2. On trust thresholds and capped counts.

Trust threshold (LiveMCPBench). To see how quality records shape retrieval, we replayed the candidate lists for 95 MiniMax tasks and 95 DeepSeek tasks. By default, a skill must succeed on two different tasks before the selector gets a positive hint about it. If one success were enough, 68.42% (MiniMax) and 67.37% (DeepSeek) of tasks would endorse at least one candidate early. If three successes were required, only 11.6% and 14.7% of tasks would be affected. So the two-success threshold often prevents us from recommending a skill based on a single experience, while a third success would change the hints for only a few tasks.

3. On unexecuted updates and recovery. Safe maintenance of a skill library is an open question for self-evolving agents in general, and our design takes a conservative approach. First, an update is committed only after a failure and a successful recovery have both been observed, so every revision is backed by a path that has already worked once. Second, if a revised skill fails when it is next used, that failure enters the same FIX loop and is corrected. This requires only the final success/failure signal and no additional judge cost. In practice, revisions to existing skills are sparse. On LiveMCPBench, only 11 of 44 accepted library actions are FIX commits, spread across 10 of 95 tasks. This limits how often an untested update can affect an active skill. We will describe this mechanism more explicitly in the revision and note finer-grained safeguards, such as version rollback, as a natural extension.

4. On total cost. We follow the EvoAgentBench protocol: skills are evolved on the training split and evaluated on held-out tasks. The table reports total token usage, including tokens spent on failed tasks and on skill updates. Cost per solved task is total tokens divided by the number of solved tasks. Arrows denote without → with evolution.

BenchmarkExecution tokens (M)Total incl. updates (M)Tokens per solved task (K)
EvoAgentBench107.81 → 96.80 (−10.2%)107.81 → 111.89 (+3.8%)718.7 → 674.1 (−6.2%)
SkillFlow54.77 → 46.95 (−14.3%)54.77 → 54.93 (+0.3%)912.9 → 773.7 (−15.2%)
LiveMCPBench41.09 → 35.62 (−13.3%)41.09 → 43.11 (+4.9%)733.8 → 673.6 (−8.2%)

Evolved skills reduce execution tokens by 10.2–14.3% and increase the number of solved tasks by 10.7–18.3%. As a result, the cost per solved task falls by 6.2–15.2%. Even with every update token included, total usage stays within 5% of the baseline. This update cost is a one-time investment: the evolved skills are reused on unseen tasks, so the cost is spread further as more tasks of similar types arrive.

5. On library growth. We will add library-growth curves to the Appendix. On LiveMCPBench (95-task online evolution), the library grows steadily from 69 to 102 entries. That is 33 new entries, about one for every three tasks, with no sudden jumps. New entries come from CAPTURE (31) and DERIVE (2). The 11 FIX commits revise existing entries without increasing the library size. The library is thus maintained dynamically: it grows in step with the arrival of new task types, while recurring patterns are absorbed by updating existing skills rather than adding new ones.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chair

We thank the AC and reviewers for their time and constructive feedback. We are glad the reviewers recognized our practical, versioned approach to skill maintenance, the examples tracing specific updates to their reuse on later tasks, and the gains observed across domains and backbones. Below we summarize our contributions and the new results from the rebuttal.

Contributions. OpenSpace is a training-free, self-evolving agent harness. It evolves a skill ecosystem while model weights stay frozen. It has three parts: (1) evidence-grounded acquisition (CAPTURE / FIX / DERIVE / NOOP), (2) versioned, reviewed maintenance with per-skill trust, and (3) quality-aware agentic skill retrieval.

OpenSpace beats the strongest published non-oracle baseline in all 12 backbone-domain settings on EvoAgentBench, by 5.4–34.5 pp. On SkillFlow (MiniMax M2.5), evolution lifts completion from 36.14% to 42.77%. Claude Code with evolution reaches 34.94%. On LiveMCPBench, evolution lifts MiniMax M2.5 from 58.95% to 67.37% and DeepSeek V4 Flash 0731 from 82.11% to 88.42%.

New results during the rebuttal. The reviewers asked four main questions. Do the gains come from evolution or from the harness? Which components matter? Is the method robust? What does it cost? The new results answer each one.

Evolution improves an already strong harness. On EvoAgentBench (Gemma-4-31B), we keep the harness, tools, and inference-step budget the same. Without evolution, the harness scores 55.1% (macro avg.). Adding ReasoningBank gives 55.6% (+0.5 pp). Full OpenSpace gives 61.1% (+5.9 pp). A standard experience-memory method adds little to this harness. Our skill evolution adds 5.9 pp.

Removing any tested component lowers completion. On SkillFlow (MiniMax M2.5), removing FIX, DERIVE, both (CAPTURE-only), or review costs 2.4, 3.6, 5.4, and 4.2 pp. Without quality records at retrieval, LiveMCPBench drops by 4.2 pp (MiniMax M2.5). Replacing attribution-sensitive trust with basic usage logs drops EvoAgentBench by 4.4 pp (Gemma-4-31B).

Agentic search selects better skills. On SWE-bench Verified (Gemma-4-31B), we fix early injection for both methods. Agentic search still beats ranked retrieval by 5.4 pp (76.8% vs. 71.4%).

Gains are robust to task order. Over 3 task orders on LiveMCPBench (MiniMax M2.5), completion is 67.02% ± 1.61 pp. Every order is 6.3–9.5 pp above the no-evolution run.

Evolution lowers the cost per solved task. We count the solver and all four auxiliary modules. Total tokens rise by 0.3–4.9%. Tokens per solved task fall by 6.2–15.2% across the three benchmarks.

Evolved skills help more than they hurt. We study the tasks that loaded an evolved skill (MiniMax M2.5). These are 33 of 95 tasks on LiveMCPBench and 62 of 166 on SkillFlow. We compare each task with its no-evolution run. On LiveMCPBench, 6 tasks (18.2%) flip from fail to pass and 1 (3.0%) regresses. On SkillFlow, 10 (16.1%) flip to pass and none regress.

Further analyses. We add a trust-threshold analysis and library-growth statistics (69→102 library entries over 95 LiveMCPBench tasks). We also clarify the run settings, judges, and train/test feedback boundaries.

Reviewer Yesy raised the rating from 2 to 3 and has no further questions. Reviewer HmkU confirmed that all concerns were addressed. Reviewer qMW7 asked about comparison fairness, related systems, and the trust/attribution design. In response, we added the same-harness comparison, the ReasoningBank baseline, the trust ablation, and the threshold analysis. All new results will be included in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.