- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces OpenSpace, a novel harness that studies how agents can acquire, maintain, and reuse a skill library, transforming static skill collections into a dynamic ecosystem. It combines evidence-grounded skill acquisition, maintenance, and context-aware skill retrieval. Experiments show improvements on unseen tasks in EvoAgentBench and sequential tasks in SkillFlow and LiveMCPBench.
Critical strengths
-
The framework provides a practical design for maintaining reusable skills, including creating new skills, repairing faulty instructions, and extending existing procedures, while tracking evidence for each version. It also reviews proposed skill updates and uses quality-aware retrieval to select skills based on their past performance.
-
Experimental results show that OpenSpace outperforms the baselines in all EvoAgentBench settings, while enabling evolution improves its own performance on SkillFlow and LiveMCPBench.
Critical weaknesses
-
It is unclear how much of OpenSpace’s advantage comes from its skill evolution method. Table 1 lacks an OpenSpace baseline with evolution disabled, while Table 2 shows that evolution gives both OpenSpace and Claude Code the same 6.63-point improvement.
-
There is no ablation study that isolates the benefits of the review process, trust calibration, or the distinction between FIX and DERIVE.
Questions for authors
Please see the weaknesses above.
3 comments
Fine-grained component ablations (W2).
Thank you for this suggestion. Below we clarify how FIX and DERIVE differ, then report ablations on the evolution operations, the review process, and trust calibration. We choose SkillFlow to conduct evolution operations and review process since it can be run in parallel, while we run trust calibration on LiveMCPBench since it has a bigger skill library.
FIX vs. DERIVE. FIX applies when an existing skill is defective. It repairs the skill and creates a new version, and the old version is retired. DERIVE applies when an existing skill is reused on a new task. It builds a new skill that extends the original to new functionality, and the original stays available. FIX improves the quality of the library, while DERIVE expands its coverage.
FIX/DERIVE and review ablations (SkillFlow). On SkillFlow (MiniMax M2.5), we disable each operation in turn, as well as the review step:
| Configuration | CAPTURE | FIX | DERIVE | Review | Pass rate | Δ vs. Full (pp) |
|---|---|---|---|---|---|---|
| Full | ✓ | ✓ | ✓ | ✓ | 42.77% | — |
| w/o FIX | ✓ | — | ✓ | ✓ | 40.36% | −2.41 |
| w/o DERIVE | ✓ | ✓ | — | ✓ | 39.16% | −3.61 |
| CAPTURE-only | ✓ | — | — | ✓ | 37.35% | −5.42 |
| w/o review | ✓ | ✓ | ✓ | — | 38.55% | −4.22 |
Every component contributes. Removing FIX or DERIVE alone lowers performance by 2.41 and 3.61 points. Removing both drops it by 5.42 points, close to the sum of the two. This shows that the two operations provide complementary gains rather than overlapping ones. Disabling review costs 4.22 points. Review likely keeps captured skills from overclaiming capabilities they do not have, which would otherwise mislead skill selection on later tasks.
Trust calibration (LiveMCPBench). To test whether recorded skill quality improves later skill selection, we ran LiveMCPBench (MiniMax M2.5) with skill evolution on and turned off only the quality information used at retrieval time. This covers the ranking bonus, the filtering of candidates with a history of fallbacks, and the quality hints shown to the selector. The pass rate drops from 67.37% to 63.16%.
We also replayed the candidate lists of MiniMax and DeepSeek on the 95 LiveMCPBench tasks to examine the success threshold. By default, a skill must succeed on two distinct tasks before the selector receives a positive hint about it. With a threshold of one, 68.42% (MiniMax) and 67.37% (DeepSeek) of tasks would endorse at least one candidate after a single success. With a threshold of three, only 11.6% and 14.7% of tasks would be affected. The two-success threshold therefore prevents premature endorsement, and a stricter threshold changes little.
We will add these ablations and the threshold analysis to the revision, together with the clarification of FIX and DERIVE.
Thank you for your response and additional experiments. All of my concerns have been addressed.
Thank you for the careful review and constructive comments. We are glad you found the framework practical, in particular its version-tracked skill maintenance and quality-aware retrieval. We address each point below and will incorporate the new results into the revision.
Contribution of the evolution mechanism (W1).
Thank you for this question. To separate the contributions of the harness and of evolution in table 1, we ran two additional configurations on Gemma4 31B with the OpenSpace harness on EvoAgentBench: (i) evolution disabled, and (ii) our evolution module replaced by ReasoningBank[1], a standard experience-memory method.
For Table 2, the Claude Code results use SkillFlow's own skill-evolution procedure. Each system solves 11 more tasks with evolution, which gives the same +6.63 gain. OpenSpace starts from a much stronger baseline (36.14% vs. 28.31%), so the tasks left for it are harder. Its evolution still solves just as many additional tasks, which is a larger share of its remaining failures. Overall, OpenSpace reaches 42.77% versus 34.94%. This matches the EvoAgentBench ablation above: the harness and evolution each contribute substantial gains. We will add these configurations to the revision.
References
[1] S. Ouyang et al., "ReasoningBank: Scaling agent self-evolving with reasoning memory," in Proc. 14th Int. Conf. Learning Representations (ICLR), 2026.