acceptodds
Under review as a conference paper at ICLR 2027

OpenClaw-Skill: Collective Skill Tree Search for Agentic Large Language Models

Abstract

Equipping Large Language Model (LLM) agents with effective skills is crucial for solving complex tasks in real-world systems such as OpenClaw. In this work, we aim to develop a framework that automatically constructs reusable skills to enhance LLM agents in tool use, multi-step reasoning, and dynamic environment interaction. To this end, we propose Collective Skill Tree Search (CSTS), a novel tree-search-based skill construction framework that constructs structured, diverse, and generalizable trees of skills. The core idea of CSTS is to leverage collective intelligence to jointly search, identify, and compose effective skills through two iterative phases: Collective Skill Node Generation (CSN-Gen) and Collective Skill Node Assessment (CSN-Assess). CSN-Gen exploits collective knowledge from multiple models to explore diverse candidate skills for each subtask, enabling comprehensive skill discovery. CSN-Assess employs multiple models as judges to evaluate and select skill nodes using two scoring mechanisms: (1) collective quality scoring, which aggregates independent evaluations to obtain a robust estimate of skill effectiveness, and (2) collective transferability scoring, which explicitly verifies whether a skill generalizes across different models. With CSTS, we construct comprehensive trees of skills together with skill-augmented training data, enabling models to effectively learn and utilize procedural knowledge. Furthermore, we introduce Collective Skill Reinforcement Learning (CSRL), which actively selects multiple relevant skills from the tree to broaden solution-space exploration and avoid being trapped by a single skill or homogeneous suboptimal solutions. As a result, our trained model, OpenClaw-Skill, exhibits strong agentic capabilities in long-horizon planning, tool use, and generalization on challenging benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Skill, Tree Search, Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kyfCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OpenClaw-Skill, a pipeline for building skill-augmented training data and training small agent models for OpenClaw-style tasks. The core component, Collective Skill Tree Search (CSTS), decomposes a task into ordered subtasks. For each subtask, several frontier models produce trajectories, and a separate synthesizer distills these into candidate skill documents. Candidates are scored by a panel of judges (quality) and by whether other models can succeed using them (transferability). The selected skill path and teacher executions are used for SFT. The authors then introduce Collective Skill RL (CSRL), which samples rollouts conditioned on the top-K skills per subtask, optimizes a GRPO-style objective, and feeds the cross-skill advantage back into the node scores to revise the skill path. Experiments on QwenClawBench, PinchBench and ALFWorld show improvements over the base Qwen models and several skill-based RL baselines.

Critical strengths
  1. Using cross-model execution as a filter for skills is a sensible idea. It grounds skill selection in whether a skill actually helps a different model, rather than relying only on judge opinions, and the ablation in Appendix E.2 suggests it contributes to the final result.
  2. The evaluation targets realistic OpenClaw-style environments rather than only toy benchmarks. The gains are consistent across several backbones, including Gemma and MiniCPM in the appendix.
Critical weaknesses
  1. The paper frames itself as a skill construction method, but every reported result comes from a model that was also fine-tuned on GPT-5.4 teacher trajectories. As a result, I can’t tell how much of the gain comes from the skills and how much from distilling a strong teacher. At minimum I’d expect three controls: SFT on teacher trajectories without skills, the frozen skill bank injected into the base model with no training, and the trained model evaluated without skill injection. Without these, the central claim that CSTS produces high-quality, transferable skills isn’t really tested.
  2. The novelty is more limited than presented. The overall recipe (multiple models jointly expand and evaluate a tree, the tree becomes SFT data, the model is trained on it) closely follows Mulberry’s Collective MCTS, which is cited only in passing and never discussed. The search itself is greedy layer-wise selection with no rollout-based value estimation or backpropagation, so calling it “tree search” overstates it. On the RL side, the authors themselves note that the within-skill and cross-skill advantages sum to (r − μ)/(σ + δ). That means the policy update is standard GRPO over the pooled group, and the only new part is the EMA score feedback.
  3. The three limitations used to motivate the work (fragmentation, single-model bias, poor cross-backbone transfer) are asserted but never demonstrated, either in this paper or for the baselines. Some of them sit uneasily with prior work. Trace2Skill, which is cited, already targets fragmented skills and reports that its skills transfer across model scales and families. Yet neither Trace2Skill nor CoEvoSkills nor EvoSkill is included as a baseline. It’s also unclear whether Skill-GRPO, Skill-SD and SkillRL start from the same SFT data and use the same skills as the proposed method.
  4. The evaluation is small and reported without any measure of variance. QwenClawBench has 100 tasks, several categories have fewer than ten, and much of the 9B model’s gain comes from two such small categories (CS and SVM). Some categories actually regress against the base model or baselines, e.g., FQT for the 4B model, and this isn’t discussed. Also, SkillRL was originally evaluated on ALFWorld, WebShop and search QA, but only ALFWorld is used here. The authors should explain why the other settings were left out.
Questions for authors
  1. During construction, skills are scored conditioned on a prefix context. At test time those scores are averaged into a single context-free value and skills are retrieved by embedding similarity. How much of the tree structure actually matters at inference? Could you report how often the retrieved skills are relevant and actually followed? In the Appendix H example, the retrieved skill says secondary sources shouldn’t be the final evidence, but the model ends up using values from a third-party summary page.
  2. What is the total compute and API cost of building CSTS-SFT-2K? This involves three generator models, three judges, cross-model execution for every candidate, and a separate synthesizer. How would the method compare against baselines given a matched budget, for example SFT on an equal amount of plain teacher trajectories?
  3. The training tasks are reverse-synthesized from public agent trajectories (ClawGym, Nex, SWE-agent), and the training and test examples in the appendix are structurally very similar (financial ratio from a filing, write to file, verify). Did you check for task-level overlap or near-duplicates between the synthesized training tasks and the evaluation benchmarks?
9 comments

Dear reviewer kyfC,

Thank you for your detailed and constructive review. We appreciate your recognition of our cross-model skill assessment and our evaluation across realistic agent tasks. We are going to try our best to address your questions one by one as follows.


W1: Contributions of Skill Construction

... At minimum I’d expect three controls: SFT on teacher trajectories without skills, the frozen skill bank injected into the base model with no training, and the trained model evaluated without skill injection ...

Thank you for raising this concern. We conducted additional controls using Qwen3.5-9B. Teacher-only SFT uses the exact same teacher trajectories as CSTS-SFT, with only the skill documents removed from the student's training inputs. The base-model control retrieves and injects skills from the skill bank frozen after CSTS construction. We also evaluate teacher-only SFT with CSTS skills at test time and compare the same CSTS-SFT + CSRL checkpoint with and without test-time skill injection.

With the same teacher demonstrations and the same CSTS skills provided at test time, CSTS-SFT outperforms teacher-only SFT by 3.6 points (42.8 vs. 39.2), supporting the benefit of skill-conditioned training. Injecting frozen CSTS skills improves the base model by 3.8 points (34.5 → 38.3) without updating its parameters. For the same CSTS-SFT + CSRL checkpoint, removing test-time skill injection reduces performance by 4.7 points (44.9 → 40.2). Together, these controls show that CSTS produces reusable skills that benefit the base model without additional training, while skill-conditioned training provides further gains beyond matched teacher supervision alone.

MethodTest-time skill injectionQwenClawBench
Qwen3.5-9BNo34.5
Qwen3.5-9BYes38.3
Teacher-only SFTNo36.9
Teacher-only SFTYes39.2
CSTS-SFTYes42.8
CSTS-SFT + CSRLNo40.2
CSTS-SFT + CSRLYes44.9

W2: Methodological Differences from Mulberry

... closely follows Mulberry’s Collective MCTS, which is cited only in passing and never discussed ...

Thank you for this comment. Mulberry is cited as Yao et al. (2024) in Section 2.3, and we provide a more detailed comparison below to clarify the methodological differences. Our work differs from Mulberry in research topic (reasoning vs. agent skills), training framework (SFT vs. SFT&RL), and search strategy (MCTS vs. CSRL-guided search). We clarify these differences below.

1. Different research topics. Mulberry focuses on multimodal reasoning, searching reasoning paths to construct SFT data for single-turn reasoning and reflection. Our work focuses on searching, selecting, and composing reusable skills into skill trees, and training models to effectively use these hierarchical skills to solve OpenClaw-style multi-turn agent tasks.

2. Different methodological designs. These different research topics lead to two main differences in our method.

  • 2.1 SFT data construction. Mulberry uses Collective MCTS to construct reasoning demonstrations, whereas CSTS performs layer-wise skill-tree search. Compared with reasoning-only settings, MCTS-style exploration in multi-turn agentic tasks can incur substantially higher costs, as switching to a different skill at an earlier layer requires regenerating the subsequent execution trajectory. Our design therefore prioritizes stability and efficiency in the costly process of generating teacher demonstrations.

  • 2.2 CSRL with cross-skill advantage feedback. Mulberry does not include an RL stage, whereas we introduce CSRL to jointly support policy learning and skill-tree adaptation. CSRL explores multiple candidate skills within each rollout group and uses cross-skill advantage feedback to update skill-node scores and revise skill paths.

3. Comparison with Mulberry. We additionally construct SFT data using Mulberry’s CoMCTS with the same search-iteration limit and fine-tune the same base model. CSTS-SFT outperforms CoMCTS-SFT by 3.5 points on QwenClawBench, supporting the effectiveness of our skill-tree construction for long-horizon agent tasks. The results also suggest that CoMCTS may require more search iterations to achieve comparable performance on long-horizon agent tasks, but doing so would substantially increase search costs.

MethodQwenClawBench
Qwen3.5-9B34.5
CoMCTS-SFT39.3
CSTS-SFT42.8

W3: Evidence for the Three Limitations and Additional Baseline Comparisons

W3(a): Clarification and Evidence for the Three Limitations

The three limitations used to motivate the work (fragmentation, single-model bias, poor cross-backbone transfer) are asserted but never demonstrated, either in this paper or for the baselines.

Thank you for this suggestion. We clarify the meaning of these limitations and the supporting empirical evidence below.

1. Skill fragmentation. We would like to clarify that our work focuses on skill coordination and execution dependencies across subtasks in long-horizon tasks. As discussed in the Introduction, local skills need to be organized into coherent execution sequences. Table 3 in the main text provides relevant evidence: both settings use multi-model skill generation and SFT, while CSN-Gen decomposes tasks into ordered subtasks and generates candidate skills conditioned on preceding skills and their execution history, improving performance by 4.2 points. This result supports our motivation to model cross-subtask dependencies through structured skill construction.

SettingQwenClawBench
SFT with vanilla multi-model skill generation35.6
SFT with CSN-Gen39.8

2. Single-model bias. Skills constructed from trajectories generated by one model may inherit its problem-solving preferences, restricting the diversity of strategies they capture. CSN-Gen therefore uses heterogeneous models to provide complementary execution experience. Our additional ablation shows that increasing the number of generation models from one to three improves performance by 2.3 points, supporting the effectiveness of incorporating multiple models into skill construction. Consistent with this finding, MultiEvoSkill [1] reports performance drops across all three ablation benchmarks when any proposer model is removed.

Number of generation models in CSN-GenQwenClawBench
N=137.5
N=238.2
N=339.8

[1] Du, K., Liu, J., Shan, R., Zhang, Y., Li, K., Sun, X., & Zhang, W. (2026). MultiEvoSkill: Structure-Aware Multi-Agent Skill Evolution for Generalizable Skill Artifacts. arXiv preprint arXiv:2610.34260.

3. Cross-backbone transfer. CoEvoSkills [2] reports that GPT-5.2 achieves 65.0% with Opus-transferred skills, compared with 69.8% with its own evolved skills, revealing a 4.8-point gap relative to model-matched skills. This illustrates that skill utility can remain backbone-dependent despite successful transfer. In our paper, Table 8 shows that adding transferability scoring improves performance by 1.8 points (43.1 → 44.9). These findings support explicitly evaluating cross-model utility during skill construction.

Scoring variantQwenClawBench
Without transferability scoring43.1
With transferability scoring44.9

[2] Hanrong Zhang et al. CoEvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verification. arXiv preprint arXiv:2604.01687, 2026.

W3(b): Comparison with Trace2Skill

Some of them sit uneasily with prior work. Trace2Skill, which is cited, already targets fragmented skills and reports that its skills transfer across model scales and families. Yet neither Trace2Skill nor CoEvoSkills nor EvoSkill is included as a baseline.

Thank you for the suggestion. We respectfully clarify that Trace2Skill and CSTS organize skills at different levels. Trace2Skill analyzes successful and failed trajectories to propose local patches to the same initial skill, then merges these patches into a single skill. CSTS decomposes long-horizon tasks into ordered subtasks, builds a candidate skill tree with one layer per subtask, and constructs coherent skill paths conditioned on preceding skills and their execution history, explicitly modeling cross-subtask execution dependencies.

Moreover, Trace2Skill, CoEvoSkills, and EvoSkill focus on training-free skill construction and optimization, whereas our full framework also includes skill-augmented SFT and RL.

To directly compare skill-construction methods, we additionally reproduced these three methods and evaluated their generated skills on QwenClawBench using the same frozen Qwen3.5-9B backbone. CSTS skills improve the base model’s score from 34.5 to 38.3, outperforming all three baselines and supporting the effectiveness of CSTS-generated skills.

SettingQwenClawBench
Qwen3.5-9B34.5
+ Trace2Skill36.3
+ CoEvoSkills35.7
+ EvoSkill35.9
+ CSTS skills38.3

W3(c): SFT Initialization and Skill Usage across Baselines

It’s also unclear whether Skill-GRPO, Skill-SD and SkillRL start from the same SFT data and use the same skills as the proposed method.

Thank you for raising this concern. We kindly clarify that constructing a tree of skills through CSTS is one of our key contributions. Skill-SD and SkillRL have their own skill-construction designs, which we preserve in the main comparisons. Meanwhile, Skill-GRPO, as our skill-conditioned counterpart to vanilla GRPO, uses the skills generated by CSTS.

Nevertheless, to control for differences in SFT initialization and skill construction, we additionally evaluated Skill-GRPO, Skill-SD, SkillRL, and CSRL starting from the same CSTS-SFT checkpoint and using the same CSTS-generated skills, as shown below. CSRL still outperforms all baselines under this setting, although the performance gaps narrow because these baselines also benefit from CSTS-generated skills and CSTS-SFT initialization.

MethodQwenClawBench
CSTS-SFT42.8
Skill-GRPO38.7
Skill-SD40.3
SkillRL41.5
CSTS-SFT + Skill-GRPO43.2
CSTS-SFT + Skill-SD43.1
CSTS-SFT + SkillRL43.4
CSTS-SFT + CSRL44.9
Reviewer_YgpyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OpenClaw-Skill, a framework for constructing skills and training agents to use them. The authors propose Collective Skill Tree Search, which generates candidate skills from multiple models, evaluates their quality and cross-model execution performance, and selects the best skill path for supervised fine-tuning. Collective Skill Reinforcement Learning then compares rollouts conditioned on different skills and uses execution feedback to update both the policy and skill selection. Experiments show improvements on QwenClawBench, PinchBench, and ALFWorld.

Critical strengths
  1. The proposed framework jointly addresses skill generation and training agents to use those skills, connecting skill construction, execution feedback, and policy optimization.
  2. The authors train models across multiple sizes and demonstrate improvements over backbone models and skill-based training baselines on several agent benchmarks.
Critical weaknesses
  1. The comparisons do not establish benefits under matched total compute. Although the ablation matches the SFT initialization and policy rollout counts, the full costs of skill-tree construction and updates, including teacher executions, candidate generation, and assessment, are not reported. Comparisons against GRPO-based training under a matched total computational cost are needed to determine whether the gains justify the additional complexity and computation.
  2. The reliance on subtask-level verifiers raises scalability concerns. The method requires verification for individual subtasks to assess skills, filter trajectories, and provide RL rewards, introducing supervision requirements beyond task-level outcomes. However, the paper does not adequately explain how the criteria and verifiers for subtasks are generated, implemented, or validated. Without demonstrating a scalable way to obtain reliable subtask-level feedback, applicability to broader and more diverse tasks remains uncertain.
Questions for authors

Please see the weakness.

4 comments

Dear Reviewer Ygpy,

Thank you for your thoughtful review. We appreciate your positive assessment of our framework and its performance across model sizes and agent benchmarks. We will do our best to address each of your concerns in detail below.

W1: Performance under matched total compute

W1(a): API costs of skill construction

Although the ablation matches the SFT initialization and policy rollout counts, the full costs of skill-tree construction and updates, including teacher executions, candidate generation, and assessment, are not reported.

Thank you for the suggestion. We agree that providing more details on token usage and API costs during skill construction helps clarify the overall computational overhead. We summarize the token usage and estimated API costs for the CSTS skill construction stages below.

StageAPI tokensAPI cost
Candidate trajectory generation93M$180.1
Skill synthesis24M$205.7
Collective quality assessment62M$114.7
Cross-model execution187M$354.3
Teacher trajectory generation68M$279.0
Total434M$1,133.8

W1(b): Comparison with GRPO

Comparisons against GRPO-based training under a matched total computational cost are needed to determine whether the gains justify the additional complexity and computation.

Thank you for the suggestion. Following your recommendation, we account for skill construction in our token-usage analysis and evaluate GRPO with an expanded training budget. Standard GRPO uses approximately 326M tokens, while CSRL uses approximately 358M tokens with skill guidance, or about 1.1× as many. Skill construction additionally consumes approximately 434M API tokens. We use aggregate token usage across skill construction and RL for an approximate budget comparison.

Both methods start from the Qwen3.5-9B checkpoint. CSRL retains the CSTS skill tree but omits SFT warm-up, with an aggregate token count of 792M tokens (434M + 358M). We extend GRPO training to 978M tokens, three times its original budget and higher than CSRL’s aggregate count. Extended GRPO achieves 36.8, while CSRL without CSTS-SFT achieves 38.9, a 2.1-point improvement. These results show that CSRL retains an advantage over standard GRPO even when GRPO is allocated a larger aggregate token budget.

MethodAggregate token countRelative token budgetQwenClawBench
Qwen3.5-9B——34.5
+ GRPO326M1.0×36.2
+ GRPO (extended training)978M3.0×36.8
+ CSRL792M2.4×38.9

W2: Scalability and Reliability of Subtask Verifiers

The reliance on subtask-level verifiers raises scalability concerns.

W2(a): Construction and scalability of subtask verifiers

Thank you for raising this concern. We provide additional details of our automated data synthesis process to explain how subtask verifiers can be constructed at scale. We first extract key steps with verifiable outcomes from open-source agent trajectories, including tool calls, numerical calculations, file creation, etc. We then use an LLM to analyze their context and execution results to generate atomic rubrics, each specifying the completion conditions for an individual operation. Through the reuse and recombination of compatible atomic rubrics, we can formulate coherent subtask instructions in a scalable manner while constructing each subtask’s verifier from the same combined rubrics.

W2(b): Reliability of subtask-level feedback

Thank you for raising this concern. We conducted a human audit of 200 subtask trajectories sampled from the CSRL stage, including 100 accepted and 100 rejected by the verifier. Each trajectory was manually reviewed against the subtask requirements and execution evidence. The disagreement rate between the verifier and human judgments on the audited sample is 2.5%, with the confusion matrix reported below.

Verifier judgmentHuman: SuccessHuman: FailureTotal
SuccessTP = 98FP = 2100
FailureFN = 3TN = 97100

Dear Reviewer Ygpy,

Thank you again for your careful review and constructive feedback. With the discussion period nearing its end, we would like to kindly follow up on whether our rebuttal has resolved your concerns. We have provided additional experiments and explanations in response to your comments and would greatly appreciate your assessment of whether these sufficiently address the points you raised. We are happy to clarify anything further before the discussion closes.

Thank you again for your careful consideration and valuable feedback!

Thanks the authors for the detailed response. The added information about API cost partially addresses my concern about the overhead of skill construction. However, the subtask verifier remains insufficiently specified, and its scalability is not yet established, so I'll maintain my original score.

Dear Reviewer Ygpy,

Thank you for your follow-up and for your positive feedback on the cost analysis. We would like to further clarify how we construct subtask verifiers at scale and how we assess the reliability of their judgments.

Scalability of verifier construction. As outlined in W2(a), we generate each subtask and its verifier together from a shared pool of reusable atomic rubrics. The process has three steps.

  1. Build a shared rubric pool. We identify operations with verifiable outcomes in public agent trajectories. For each operation, an LLM uses its context and execution result to derive an atomic rubric specifying what counts as success. The resulting pool covers capabilities such as field extraction, numerical calculation, and file writing.

  2. Sample rubrics to generate a subtask and its verifier. We sample atomic rubrics from the pool and combine the operations they describe into a subtask instruction. The completion criteria in those same rubrics form the verifier. For example, a calculation rubric and a file-writing rubric yield the instruction “Compute the total and save it to the specified file.” The verifier checks whether the total is correct and whether the specified file contains it. We can then pair the same file-writing rubric with a field-extraction rubric to create another subtask without writing a new verifier by hand.

  3. Compose long-horizon tasks. We further compose these subtasks into long-horizon agent tasks while retaining each subtask’s verifier, so intermediate steps can still be checked individually.

Verification reliability. As reported in W2(b), human reviewers checked 200 CSRL subtask trajectories against the subtask requirements and execution evidence. They agreed with 98 of the verifier’s 100 acceptances and 97 of its 100 rejections. This 2.5% disagreement rate (5/200) supports the reliability of subtask-level verifier feedback during training.

Thank you again for your time and consideration.

Reviewer_wX8VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OpenClaw-Skill, a pipeline for constructing and internalizing reusable procedural skills for language-model agents. Its first component, Collective Skill Tree Search (CSTS), decomposes a task into ordered subtasks, collects candidate executions from heterogeneous frontier models, distills each execution into a skill node, and scores nodes using both multi-model quality judgments and cross-model execution-based transferability. A greedy, prefix-conditioned path is selected for verified teacher demonstrations and supervised fine-tuning, while alternative nodes are retained. Its second component, Collective Skill Reinforcement Learning (CSRL), samples target-policy rollouts under multiple retained skills, applies a GRPO-style update, uses a cross-skill reward signal to recalibrate node scores, and may revise a path and regenerate its suffix. At test time, subtask descriptions retrieve skills from a frozen bank by embedding similarity, after which a cached score selects the injected skill.

Critical strengths
  • A coherent approach to constructing executable and compositional agent skills. CSTS does more than summarize successful trajectories: it conditions each candidate on the selected prefix and environment history, tests transfer by having other models execute the skill, retains alternatives, and verifies complete teacher trajectories.

  • Broad empirical coverage with useful component analyses. The paper evaluates multiple model scales on two OpenClaw-style benchmarks, transfers the framework to ALFWorld, and adds Gemma/MiniCPM results. The improvements over untrained backbones are large, and the comparisons to Skill-GRPO, Skill-SD, and SkillRL are favorable.

Critical weaknesses
  • The experiment does not isolate the claimed CSRL feedback mechanism. In Equation 7, the within-skill and cross-skill terms sum exactly to �0��0�, so the policy-gradient advantage is ordinary normalization over the joint KGK G rollout group. The distinctive remaining mechanism is therefore multi-skill sampling plus use of the cross-skill term to update node utilities, rerank nodes, and revise paths. However, Table 7 compares CSRL at K=2,G=6K=2,G=6 against a “GRPO” control at K=1,G=12K=1,G=12, simultaneously changing the sampling distribution and disabling cross-skill comparison. It does not compare against K=2,G=6K=2,G=6 with the same skill-conditioned rollouts but frozen node scores and paths, nor separately ablate utility calibration and suffix regeneration. This makes the 1.6-point gain over the reported GRPO control insufficient evidence for the paper’s central bidirectional tree-policy optimization claim.

  • The evaluation does not yet cleanly separate the skill/tree contribution from teacher-data quality, and uncertainty is missing. CSTS-SFT-2K is produced using several frontier models, GPT-5.4 teacher executions, verification, and filtering. The paper compares different skill-generation variants, but it does not report a matched control trained on the same verified teacher action tokens with the skill documents removed, or a flat/independent-skill control with the same demonstrations and token budget.

Questions for authors
  1. With exactly the same CSTS-SFT checkpoint, K=2,G=6K=2,G=6 rollouts, sampled skills, rollout budget, and seeds, what are the results for (a) joint multi-skill GRPO with frozen scores and a frozen path, (b) adding score calibration but no path/suffix revision, and (c) full CSRL?

  2. Can the authors provide a control trained on the exact same verified GPT-5.4 demonstrations with skill text removed (and ideally a flat-skill version with matched context/token budget), along with uncertainty for the principal comparisons? Please also explain how source trajectories were screened against QwenClawBench and both PinchBench task sets for exact and semantic overlap. If the gains remain robust under these controls and the task pools are demonstrably non-overlapping, my assessment would become more favorable.

5 comments

Dear Reviewer wX8V,

Many thanks for your valuable comments and questions. We are glad that you found our skill construction approach coherent and the paper easy to follow. We also appreciate your recognition of the breadth of our evaluation. Below, we address your concerns and questions individually.

W1 & Q1: Distinguishing CSRL from multi-skill GRPO and isolating the feedback mechanism

W1(a): Difference from multi-skill GRPO.

... so the policy-gradient advantage is ordinary normalization over the joint (K G) rollout group. The distinctive remaining mechanism is therefore multi-skill sampling plus use of the cross-skill term to update node utilities ...

Thank you for raising this concern. Unlike GRPO with skill augmentation, which expands the set of skills used for training and treats skill-conditioned inputs as independent GRPO groups, CSRL shares reward information across skills for the same subtask prefix, enabling joint comparison of diverse trajectories and providing informative advantages, thereby improving policy learning. Furthermore, compared with multi-skill GRPO, which samples rollouts under multiple skills within a single group, CSRL uses cross-skill feedback to calibrate node scores and revise skill paths, adapting subsequent skill selection and training contexts to the evolving policy and improving task performance. We compare these variants with the results reported below.

MethodSkills per subtaskGRPO groupsGroup compositionTree feedbackQwenClawBench
CSTS-SFT————42.8
+ GRPO with skill augmentation221 × 6No43.2
+ Single-skill GRPO111 × 12No43.3
+ Multi-skill GRPO212 × 6No43.7
+ CSRL212 × 6Yes44.9

W1(b) & Q1: Ablations of the feedback mechanism.

However, Table 7 compares CSRL at (K=2,G=6) against a “GRPO” control at (K=1,G=12), simultaneously changing the sampling distribution and disabling cross-skill comparison.

… what are the results for (a) joint multi-skill GRPO with frozen scores and a frozen path, (b) adding score calibration but no path/suffix revision, and (c) full CSRL?

Thank you for your suggestion! We agree that further ablations of the CSRL feedback mechanism are valuable for analyzing the contributions of its different components. Therefore, following your suggestion, we have added comparative experiments to further clarify these contributions. The results of these comparative experiments are presented below.

SettingK × GScore calibrationPath revisionQwenClawBench
CSTS-SFT---42.8
+ single-skill GRPO1 × 12--43.3
+ (a) Multi-skill GRPO, frozen scores and path2 × 6NoNo43.7
+ (b) Multi-skill GRPO + score calibration2 × 6YesNo43.4
+ (c) Full CSRL2 × 6YesYes44.9

These results highlight the importance of coupling score calibration with path revision. Score calibration updates skill-value estimates using policy feedback, while path revision dynamically switches to higher-scoring candidates and regenerates the suffix, keeping subsequent training contexts consistent with the updated skill choices.

W2 & Q2: Skill contributions and evaluation reliability

W2 & Q2(a): Separating skill contributions from teacher supervision.

The evaluation does not yet cleanly separate the skill/tree contribution from teacher-data quality ...

Can the authors provide a control trained on the exact same verified GPT-5.4 demonstrations with skill text removed (and ideally a flat-skill version with matched context/token budget)?

Thank you for raising this concern. We agree with your point and have added two controls to distinguish the contributions of teacher supervision and the skills constructed by CSTS. Our teacher-only SFT control uses the exact same verified teacher action targets as CSTS-SFT, with only the skill documents removed from the student’s training inputs. For flat-skill SFT, we use the same skill synthesizer as CSTS to directly generate one task-level skill for each training task. We pair these flat skills with the same teacher trajectories to construct the baseline. CSTS-SFT outperforms teacher-only SFT by 5.9 points (42.8 vs. 36.9) and flat-skill SFT by 4.7 points (42.8 vs. 38.1). These results support the benefit of using skills constructed by CSTS during SFT, beyond matched teacher supervision alone and directly generated task-level flat skills.

SettingQwenClawBench
Qwen3.5-9B34.5
Teacher-only SFT36.9
Flat-skill SFT38.1
CSTS-SFT42.8

Q2(b): Uncertainty of the principal comparisons.

… along with uncertainty for the principal comparisons?

Thank you for the suggestion. We report uncertainty for the principal comparisons on QwenClawBench, where our main controlled experiments are conducted, and on ALFWorld, which provides a distinct embodied-task setting. Results are reported as mean ± sample standard deviation across repeated evaluations of fixed checkpoints.

MethodQwenClawBench (5 runs)
Qwen3.5-9B34.5 ± 0.4
GRPO36.2 ± 0.3
Skill-GRPO38.7 ± 0.6
Skill-SD40.3 ± 0.5
SkillRL41.5 ± 0.7
OpenClaw-Skill 9B44.9 ± 0.4
MethodALFWorld (3 runs)
Qwen2.5-7B-Instruct14.5 ± 0.7
GRPO72.6 ± 0.8
Skill-GRPO80.8 ± 0.6
Skill-SD85.1 ± 0.9
SkillRL89.9 ± 0.5
OpenClaw-Skill-7B92.5 ± 0.6

Q2(c): Potential exact and semantic overlap between synthetic training and evaluation tasks.

Please also explain how source trajectories were screened against QwenClawBench and both PinchBench task sets for exact and semantic overlap.

Thank you for this suggestion. For ALFWorld, we used the official training split. To address your concern, we conducted the following two analyses of the synthesized OpenClaw-style training tasks.

1. Overlap screening. We screened the 200 synthesized OpenClaw-style training tasks against the evaluation sets using two complementary checks.

For exact overlap, n-gram matching has been used to screen for overlap between training and evaluation data [1–3]. We used exact 13-word-gram matching, following GPT-3’s initial training-data filtering approach [1].

For semantic overlap, following prior work on LLM-based overlap detection [4–6], we first used Qwen3-Embedding-8B to retrieve the five most similar evaluation queries for each training task. We then used DeepSeek V4 Flash to assess whether the paired tasks were semantic duplicates. As an additional check, we manually reviewed a random 10% sample of the embedding-retrieved candidate pairs and found high agreement with the LLM judgments (97%). The screening results are reported below, with calculated relative to the 200 training tasks.

Evaluation setExact overlap (13-gram)Semantic overlap (DeepSeek V4 Flash)
QwenClawBench0.0%0.5%
PinchBench (123 tasks)0.0%1.5%

2. SFT after potential semantic overlap removal. To further assess the influence of these potentially overlapping samples, we removed the four tasks and their derived demonstrations from the CSTS-SFT training set, repeated SFT from the initial Qwen3.5-9B checkpoint, and evaluated the resulting model. Removing the flagged tasks changed performance by only −0.1 points on QwenClawBench and −0.2 points on PinchBench. These small changes support the robustness of our CSTS-SFT results.

Training dataNumber of training tasksQwenClawBenchPinchBench (123 tasks)
Original200 (100%)42.851.4
Overlap-filtered196 (98%)42.751.2

[1] Tom B. Brown et al. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.

[2] Jeffrey Li et al. DataComp-LM: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems (NeurIPS), 37, Datasets and Benchmarks Track, 2024.

[3] Aakanksha Chowdhery et al. PaLM: Scaling Language Modeling with Pathways. Journal of Machine Learning Research, 24(240):1–113, 2023.

[4] Shubham Toshniwal et al. OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data. International Conference on Learning Representations (ICLR), 2025.

[5] Qihao Zhao et al. MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark. ACL, 2025.

[6] Chunyuan Deng et al. Investigating Data Contamination in Modern Benchmarks for Large Language Models. NAACL, 2024.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Kind Follow-up on the Review Discussion for Submission 54629

Dear Area Chair,

Thank you for your time and effort in handling our submission. We appreciate the reviewers’ thoughtful suggestions, which helped us design more informative controls and strengthen the evaluation.

Our paper introduces OpenClaw-Skill, a framework that organizes reusable skills into skill trees and trains LLM agents to apply them in multi-step tasks. The reviewers highlighted the breadth of our evaluation (wX8V), the connection between skill construction and policy optimization (Ygpy), and the use of realistic OpenClaw-style environments (kyfC).

Two questions motivated our main rebuttal experiments: how much the skills contribute beyond teacher supervision, and what additional benefit comes from updating the skill tree during training. On QwenClawBench, with the teacher demonstrations and test-time skills held fixed, including skills during supervised fine-tuning (SFT) raised the score from 39.2 to 42.8. At the same rollout budget, multi-skill GRPO with a frozen tree scored 43.7, while CSRL achieved 44.9 by using cross-skill feedback to update skill scores and revise paths.

We also added direct comparisons with Mulberry’s CoMCTS, Trace2Skill, CoEvoSkills, and EvoSkill. To address the cost concern, we reported the API cost of skill construction and showed that our approach retained its advantage when GRPO and teacher-only SFT were each given larger token budgets. We also checked subtask verifier judgments against human assessments, screened the synthesized training tasks for potential overlap with the evaluation tasks, and extended the evaluation to WebShop and HotpotQA.

We posted our point-by-point responses on November 15 and followed up with all three reviewers on November 18. OpenReview records an edit to Reviewer kyfC’s original review later that day. We have yet to receive a follow-up comment on our rebuttal in any of the three discussion threads. With less than 12 hours remaining, we would be grateful if you could invite the reviewers to share their assessment of the additional results. We would welcome the opportunity to answer any remaining questions before the discussion closes.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.