- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces UniSkill, a framework that jointly trains an actor and a skill proposer using a shared language model. The proposer creates and revises a skillbank, while the actor retrieves skills to solve tasks. Proposed skills receive contrastive feedback based on how they change the actor's likelihood of previously collected successful and failed trajectories, avoiding additional evaluation rollouts for each proposal. An external critic checks proposal validity and grounding, and a support regularizer maintains exploration over skill-edit operations. The authors evaluate UniSkill on ALFWorld and WebShop, reporting task performance, computational costs, ablations, and analyses of the relationship between proposal scores and actual utility.
Critical strengths
-
UniSkill evaluates proposed skills using successful and failed trajectories already collected during actor training, avoiding additional environment rollouts for each proposal. The timing comparison in Figure 3 shows that this substantially reduces skill-evaluation costs. Using the current actor to compute the score also allows the feedback to reflect changes in the actor's capabilities during training.
-
The paper includes ablation studies on joint training, the alignment reward, and skill-edit support regularization. These experiments help explain the contribution of the main components. The authors also evaluate proposed skills through fresh rollouts with a fixed actor and discuss cases where positive scores do not lead to better performance, making the limitations of the proposed feedback clearer.
-
UniSkill achieves substantial improvements over RL-only baselines on both ALFWorld and WebShop, with the main results reported over three independent training runs. The authors also conduct experiments with a smaller backbone and alternative skill critics. These additional results suggest that the method's benefits are not limited to the default model configuration.
Critical weaknesses
-
The experiments are conducted only on ALFWorld and WebShop, and the ALFWorld results are already close to saturation. The main experiments also use the relatively old Qwen2.5-7B-Instruct model, while the additional 3B experiments remain within the same model family. It is unclear whether the method would provide similar benefits with newer and stronger backbone models. Evaluating more challenging tasks, such as repository-level coding on SWE-bench, together with more recent models, would help demonstrate the broader applicability of the proposed approach.
-
The method uses top-1 skill retrieval, but the paper does not systematically evaluate how increasing the skillbank size affects retrieval quality and downstream task performance. Potential issues such as redundant, conflicting, or outdated skills are also not sufficiently examined. The reported skillbank growth curves and qualitative examples are insufficient to establish the reliability of the method as the skillbank becomes larger, limiting the evidence for its effectiveness in long-term skill accumulation scenarios.
-
Compared with Skill1, UniSkill improves the success rates on ALFWorld and WebShop by only 0.9 and 1.8 percentage points, respectively, but the paper does not provide comparable uncertainty estimates for Skill1 across multiple runs. Figure 3 also reports timing for selected training stages, leaving the complete training cost unclear. Repeating the strongest baseline under matched conditions and reporting complete training costs would help clarify whether the improvements are consistent and whether the method maintains its advantage under comparable computational budgets.
Questions for authors
-
The current OOD evaluation mainly considers distribution shifts within a single task environment. If tasks from multiple datasets or different environments are mixed during training with a shared actor, proposer, and skillbank, would UniSkill retain its benefits? Could the authors provide results or discussion on skill transfer and interference across tasks, as well as generalization to unseen task types or task combinations?
-
UniSkill relies on an external LLM to evaluate the appropriateness and quality of skill proposals. How necessary is this external Skill Critic for the framework? Could the shared actor-proposer model evaluate its own skill proposals instead, and would the method still achieve effective skill learning without relying on an external model?
-
The case studies show that some skill proposals passing the admission criteria can still reduce actual task performance. When these skills are added to the skillbank and repeatedly reused in subsequent tasks, how does UniSkill identify and correct their negative effects? Is the current skill-update mechanism sufficient to address such cases?
7 comments
Response to Reviewer jhrc
Q2 (Corresponding to Weakness 2)
A2: Thank you for raising the concern about retrieval reliability as the skillbank continues to accumulate skills. We provide further analysis of changes in skillbank size, redundancy, conflicts, and outdated content.
1. Skillbank size and task performance.
We fix the actor at Step 75 and evaluate it with skillbank snapshots from Steps 25, 75, 100, and 200. For each snapshot, we perform one rollout on each of the same 128 evaluation tasks, keeping the retrieval, decoding, and environment settings unchanged:
| Skillbank snapshot | Number of skills | Task success rate (%) |
|---|---|---|
| Step 25 | 29 | 62.50 |
| Step 75 | 47 | 62.50 |
| Step 100 | 51 | 64.84 |
| Step 200 | 58 | 69.53 |
Across the evaluated snapshots, the success rate increases from 62.50% to 69.53%, showing that continued skill accumulation and revision provide further task benefits. Holding the actor fixed distinguishes this change from improvements in the policy’s own capabilities.
2. Redundancy and its effect on top-1 retrieval.
UniSkill introduces new skills through ADD, revises existing content through UPDATE, and allows the skillbank to remain unchanged through NO_EDIT. The support regularizer encourages continued exploration of all three editing operations, helping prevent collapse to a single operation and allowing skill accumulation to incorporate both additions and revisions. Figure 6 in the original submission illustrates this effect.
We further compare the skillbanks produced by standard UniSkill and a run exhibiting ADD collapse at the same training step. For the same 30 randomly sampled ALFWorld tasks, we retrieve the top-5 candidates from each skillbank and manually audit them. Within each candidate set, entries that provide the same guidance under the same applicability conditions are grouped together; one entry per group is counted as the representative, and the remaining entries are counted as redundant. We also check whether the top-1 skill is applicable to the task.
| Skillbank | Number of skills | Mean redundant entries per top-5 set | Tasks with an applicable top-1 skill |
|---|---|---|---|
| Standard UniSkill | 62 | 0.2 | 30/30 (100%) |
| ADD-collapse run | 1,431 | 3.6 | 24/30 (80%) |
The top-5 candidates are examined only for the redundancy analysis; task execution still uses top-1 retrieval. Across these 30 tasks, the ADD-collapse run produces a larger skillbank but more repetitive candidates and lower top-1 applicability. Standard UniSkill maintains lower redundancy and higher retrieval applicability with a more compact skillbank. These results highlight the importance of revising existing content alongside adding new entries for effective skill accumulation.
3. Conflicting guidance.
We also check for conflicts among the top-5 candidates for these 30 tasks. We classify guidance as conflicting only when it gives incompatible instructions under the same applicability conditions. Such conflicts occur in the candidate sets for 0/30 tasks with standard UniSkill and 4/30 tasks with the ADD-collapse run. Thus, in this audit, standard UniSkill’s retrieved candidates exhibit both less redundancy and more consistent guidance.
4. Outdated content and effectiveness under continued accumulation.
Whether a skill remains useful depends on the actual benefit it provides to the current actor. After an existing skill is used in new interactions, the proposer can suggest an UPDATE based on the resulting trajectories, and the revision is evaluated using Contrastive Action Feedback from the current actor.
For skills previously observed to reduce execution success, we conduct a controlled continuation of training and execution comparisons. All eight skills that were reused were subsequently updated. Holding the actor at each skill’s first UPDATE checkpoint fixed, all eight updated versions achieve higher success rates than their original versions, with an average improvement of 15.6 percentage points. The detailed experimental setup and tracking results are provided in our response to Q6.
Together with the 250-step training dynamics in Figure 1 of the original submission, these results provide quantitative evidence on continued skill accumulation through changes in skillbank size, candidate redundancy and conflicts, retrieval applicability, and execution gains after updates. Thank you for the suggestion. We will include these results in the revised manuscript to clarify UniSkill’s skill accumulation and maintenance within the evaluated range of skillbank sizes.
Response to Reviewer jhrc
Q3 (Corresponding to Weakness 3)
A3: Thank you for suggesting a more complete comparison of performance variability and computational cost. Regarding Skill1’s statistics across multiple runs, we checked the original paper and confirmed that its main results on both ALFWorld and WebShop are averages over three runs. Its appendix D.2 also reports the standard deviation of its ALFWorld success rate. We include this information alongside UniSkill’s results over three runs:
| Method | ALFWorld success rate (%) | WebShop task score | WebShop success rate (%) |
|---|---|---|---|
| Skill1 | 97.5 ± 0.6 | 89.7 | 82.9 |
| UniSkill | 98.4 ± 0.8 | 90.5 ± 1.4 | 84.7 ± 0.5 |
Skill1 does not report the corresponding standard deviations for WebShop, so we retain its three-run averages for these metrics. These statistics provide a more complete account of the mean performance and available variability estimates for both methods.
Regarding computational cost under matched conditions, Skill1’s results are taken from its original paper. We also attempted to reproduce Skill1 using its official repository, but could not complete the reproduction because files required for execution were missing, an issue also reported by other users in the repository’s issue tracker. We therefore provide a computational cost comparison with Evolving-RL under the same training framework, ALFWorld setup, and eight-GPU hardware configuration. Evolving-RL also jointly optimizes the policy and skills, but evaluates skills through additional environment rollouts. Using 90% validation success rate as a common target, we estimate the cumulative cost of first reaching this target from the average per-step stage timings reported in Figure 3:
| Method | Target success rate | Steps to first reach target | Cumulative time for timed stages | GPU-hours |
|---|---|---|---|---|
| Evolving-RL | 90% | 45 | 19.49 h | 155.92 |
| UniSkill | 90% | 95 | 13.37 h | 106.96 |
Although UniSkill requires more training steps, its lower per-step skill evaluation cost reduces the cumulative time for the timed stages by approximately 31.4% when reaching the same target success rate. We further compare cumulative costs over 250 training steps. For UniSkill, we report the recorded time; because Evolving-RL did not complete 250 steps, we extrapolate its time from the average per-step time of the completed training:
| Method | Training steps | Timing basis | Cumulative time for timed stages | GPU-hours |
|---|---|---|---|---|
| Evolving-RL | 250 | Extrapolated estimate | 108.28 h | 866.22 |
| UniSkill | 250 | Measured | 35.18 h | 281.47 |
In addition, UniSkill’s total skill critic API cost for the 250-step run is 4.2 USD.
Thank you for the suggestion. We will include these run statistics and cost analyses in the revised manuscript to provide a more complete comparison of performance and efficiency.
Response to Reviewer jhrc
Q1 (Corresponding to Weakness 1)
A1: Thank you for pointing out the limitations in model and task coverage. We use Qwen2.5-7B-Instruct in the main experiments to match the backbone used by the compared baselines, reducing the influence of differences in base-model capabilities on the methodological comparison.
Regarding applicability to newer models, we conducted supplementary ALFWorld experiments with Qwen3-4B-Instruct-2507, comparing the initial model, GRPO, SkillRL, and UniSkill. Regarding more challenging tasks, we further extended UniSkill to repository-level code repair and evaluated it on SWE-bench Verified. The results are shown below:
† For SWE-bench Verified, we follow the evaluation setup used in the DAgger study. The initial-model and GRPO scores are taken from Revisiting DAgger in the Era of LLM-Agents [1].
On ALFWorld, all methods use the same backbone, tool interface, interaction budget, and evaluation settings. UniSkill achieves an 89.1% success rate, outperforming GRPO and SkillRL by 35.2 and 21.9 percentage points, respectively. These results provide further evidence of UniSkill’s effectiveness with a newer-generation backbone.
For repository-level code repair on SWE-bench, given the time constraints of the rebuttal period, we randomly sampled 100 of the 293 SkyRL-v0 training tasks and trained UniSkill using OpenHands. At evaluation time, we froze the model parameters and skillbank. UniSkill resolved 99 of 466 tasks in the separate SWE-bench Verified test set, achieving a 21.2% resolution rate, exceeding the reported initial-model and GRPO results of 11.2% and 11.6%, respectively.
These experiments extend the evaluation of UniSkill to a newer backbone and repository-level code repair beyond the original environments, providing initial evidence of its broader applicability. We are also considering extending training to the full set of 293 SkyRL-v0 tasks.