acceptodds
Under review as a conference paper at ICLR 2027

UniSkill: Learning Actor-Aligned Skill Proposals for an Evolving Policy

Abstract

Large language model agents can improve across tasks by retaining reusable skills distilled from prior interactions. Recent work jointly optimizes task execution and skill extraction, enabling the policy and skillbank to co-evolve. However, as the actor continues learning, rewarding skill proposals through their reuse in subsequent training steps may conflate skill benefits with actor improvement, while directly testing each proposed skill requires costly additional actor rollouts. In this paper, we introduce UniSkill, which uses a shared policy to interact with the environment and propose skillbank edits (Add, Update, or No Edit) from the resulting trajectories. Specifically, the actor learns from environment rewards, while contrastive action feedback guides skill proposal learning. This feedback provides an actor-alignment signal by measuring how replacing the retrieved skill with a proposed skill changes the current actor's action log-likelihood gap between previously collected successful and failed trajectories from the same task, thereby avoiding new rollouts for each proposal. Since proposal-level feedback may suppress an otherwise appropriate edit operation when the proposed skill content scores poorly, we further apply skill-edit support regularization to preserve exploration. Empirically, UniSkill achieves strong performance, reaching 98.4% success on ALFWorld and 84.7% on WebShop while maintaining stable joint training. Further ALFWorld experiments show that UniSkill remains effective when the shared policy uses a smaller backbone.

Primary area:
reinforcement learning
Keywords:
LLM agents, reinforcement learning, skill learning, skill-augmented agents, policy-skill co-evolution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jhrcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces UniSkill, a framework that jointly trains an actor and a skill proposer using a shared language model. The proposer creates and revises a skillbank, while the actor retrieves skills to solve tasks. Proposed skills receive contrastive feedback based on how they change the actor's likelihood of previously collected successful and failed trajectories, avoiding additional evaluation rollouts for each proposal. An external critic checks proposal validity and grounding, and a support regularizer maintains exploration over skill-edit operations. The authors evaluate UniSkill on ALFWorld and WebShop, reporting task performance, computational costs, ablations, and analyses of the relationship between proposal scores and actual utility.

Critical strengths
  1. UniSkill evaluates proposed skills using successful and failed trajectories already collected during actor training, avoiding additional environment rollouts for each proposal. The timing comparison in Figure 3 shows that this substantially reduces skill-evaluation costs. Using the current actor to compute the score also allows the feedback to reflect changes in the actor's capabilities during training.

  2. The paper includes ablation studies on joint training, the alignment reward, and skill-edit support regularization. These experiments help explain the contribution of the main components. The authors also evaluate proposed skills through fresh rollouts with a fixed actor and discuss cases where positive scores do not lead to better performance, making the limitations of the proposed feedback clearer.

  3. UniSkill achieves substantial improvements over RL-only baselines on both ALFWorld and WebShop, with the main results reported over three independent training runs. The authors also conduct experiments with a smaller backbone and alternative skill critics. These additional results suggest that the method's benefits are not limited to the default model configuration.

Critical weaknesses
  1. The experiments are conducted only on ALFWorld and WebShop, and the ALFWorld results are already close to saturation. The main experiments also use the relatively old Qwen2.5-7B-Instruct model, while the additional 3B experiments remain within the same model family. It is unclear whether the method would provide similar benefits with newer and stronger backbone models. Evaluating more challenging tasks, such as repository-level coding on SWE-bench, together with more recent models, would help demonstrate the broader applicability of the proposed approach.

  2. The method uses top-1 skill retrieval, but the paper does not systematically evaluate how increasing the skillbank size affects retrieval quality and downstream task performance. Potential issues such as redundant, conflicting, or outdated skills are also not sufficiently examined. The reported skillbank growth curves and qualitative examples are insufficient to establish the reliability of the method as the skillbank becomes larger, limiting the evidence for its effectiveness in long-term skill accumulation scenarios.

  3. Compared with Skill1, UniSkill improves the success rates on ALFWorld and WebShop by only 0.9 and 1.8 percentage points, respectively, but the paper does not provide comparable uncertainty estimates for Skill1 across multiple runs. Figure 3 also reports timing for selected training stages, leaving the complete training cost unclear. Repeating the strongest baseline under matched conditions and reporting complete training costs would help clarify whether the improvements are consistent and whether the method maintains its advantage under comparable computational budgets.

Questions for authors
  1. The current OOD evaluation mainly considers distribution shifts within a single task environment. If tasks from multiple datasets or different environments are mixed during training with a shared actor, proposer, and skillbank, would UniSkill retain its benefits? Could the authors provide results or discussion on skill transfer and interference across tasks, as well as generalization to unseen task types or task combinations?

  2. UniSkill relies on an external LLM to evaluate the appropriateness and quality of skill proposals. How necessary is this external Skill Critic for the framework? Could the shared actor-proposer model evaluate its own skill proposals instead, and would the method still achieve effective skill learning without relying on an external model?

  3. The case studies show that some skill proposals passing the admission criteria can still reduce actual task performance. When these skills are added to the skillbank and repeatedly reused in subsequent tasks, how does UniSkill identify and correct their negative effects? Is the current skill-update mechanism sufficient to address such cases?

7 comments
AuthorsOpenReview ↗

Response to Reviewer jhrc

Q1 (Corresponding to Weakness 1)
A1: Thank you for pointing out the limitations in model and task coverage. We use Qwen2.5-7B-Instruct in the main experiments to match the backbone used by the compared baselines, reducing the influence of differences in base-model capabilities on the methodological comparison.

Regarding applicability to newer models, we conducted supplementary ALFWorld experiments with Qwen3-4B-Instruct-2507, comparing the initial model, GRPO, SkillRL, and UniSkill. Regarding more challenging tasks, we further extended UniSkill to repository-level code repair and evaluated it on SWE-bench Verified. The results are shown below:

MethodALFWorld success rate (%)SWE-bench Verified resolved (%)
Initial model17.211.2†
GRPO53.911.6†
SkillRL67.2—
UniSkill89.121.2

† For SWE-bench Verified, we follow the evaluation setup used in the DAgger study. The initial-model and GRPO scores are taken from Revisiting DAgger in the Era of LLM-Agents [1].

On ALFWorld, all methods use the same backbone, tool interface, interaction budget, and evaluation settings. UniSkill achieves an 89.1% success rate, outperforming GRPO and SkillRL by 35.2 and 21.9 percentage points, respectively. These results provide further evidence of UniSkill’s effectiveness with a newer-generation backbone.

For repository-level code repair on SWE-bench, given the time constraints of the rebuttal period, we randomly sampled 100 of the 293 SkyRL-v0 training tasks and trained UniSkill using OpenHands. At evaluation time, we froze the model parameters and skillbank. UniSkill resolved 99 of 466 tasks in the separate SWE-bench Verified test set, achieving a 21.2% resolution rate, exceeding the reported initial-model and GRPO results of 11.2% and 11.6%, respectively.

These experiments extend the evaluation of UniSkill to a newer backbone and repository-level code repair beyond the original environments, providing initial evidence of its broader applicability. We are also considering extending training to the full set of 293 SkyRL-v0 tasks.

[1] Revisiting DAgger in the Era of LLM-Agents.

AuthorsOpenReview ↗

Response to Reviewer jhrc

Q2 (Corresponding to Weakness 2)
A2: Thank you for raising the concern about retrieval reliability as the skillbank continues to accumulate skills. We provide further analysis of changes in skillbank size, redundancy, conflicts, and outdated content.

1. Skillbank size and task performance.

We fix the actor at Step 75 and evaluate it with skillbank snapshots from Steps 25, 75, 100, and 200. For each snapshot, we perform one rollout on each of the same 128 evaluation tasks, keeping the retrieval, decoding, and environment settings unchanged:

Skillbank snapshotNumber of skillsTask success rate (%)
Step 252962.50
Step 754762.50
Step 1005164.84
Step 2005869.53

Across the evaluated snapshots, the success rate increases from 62.50% to 69.53%, showing that continued skill accumulation and revision provide further task benefits. Holding the actor fixed distinguishes this change from improvements in the policy’s own capabilities.

2. Redundancy and its effect on top-1 retrieval.

UniSkill introduces new skills through ADD, revises existing content through UPDATE, and allows the skillbank to remain unchanged through NO_EDIT. The support regularizer encourages continued exploration of all three editing operations, helping prevent collapse to a single operation and allowing skill accumulation to incorporate both additions and revisions. Figure 6 in the original submission illustrates this effect.

We further compare the skillbanks produced by standard UniSkill and a run exhibiting ADD collapse at the same training step. For the same 30 randomly sampled ALFWorld tasks, we retrieve the top-5 candidates from each skillbank and manually audit them. Within each candidate set, entries that provide the same guidance under the same applicability conditions are grouped together; one entry per group is counted as the representative, and the remaining entries are counted as redundant. We also check whether the top-1 skill is applicable to the task.

SkillbankNumber of skillsMean redundant entries per top-5 setTasks with an applicable top-1 skill
Standard UniSkill620.230/30 (100%)
ADD-collapse run1,4313.624/30 (80%)

The top-5 candidates are examined only for the redundancy analysis; task execution still uses top-1 retrieval. Across these 30 tasks, the ADD-collapse run produces a larger skillbank but more repetitive candidates and lower top-1 applicability. Standard UniSkill maintains lower redundancy and higher retrieval applicability with a more compact skillbank. These results highlight the importance of revising existing content alongside adding new entries for effective skill accumulation.

3. Conflicting guidance.

We also check for conflicts among the top-5 candidates for these 30 tasks. We classify guidance as conflicting only when it gives incompatible instructions under the same applicability conditions. Such conflicts occur in the candidate sets for 0/30 tasks with standard UniSkill and 4/30 tasks with the ADD-collapse run. Thus, in this audit, standard UniSkill’s retrieved candidates exhibit both less redundancy and more consistent guidance.

4. Outdated content and effectiveness under continued accumulation.

Whether a skill remains useful depends on the actual benefit it provides to the current actor. After an existing skill is used in new interactions, the proposer can suggest an UPDATE based on the resulting trajectories, and the revision is evaluated using Contrastive Action Feedback from the current actor.

For skills previously observed to reduce execution success, we conduct a controlled continuation of training and execution comparisons. All eight skills that were reused were subsequently updated. Holding the actor at each skill’s first UPDATE checkpoint fixed, all eight updated versions achieve higher success rates than their original versions, with an average improvement of 15.6 percentage points. The detailed experimental setup and tracking results are provided in our response to Q6.

Together with the 250-step training dynamics in Figure 1 of the original submission, these results provide quantitative evidence on continued skill accumulation through changes in skillbank size, candidate redundancy and conflicts, retrieval applicability, and execution gains after updates. Thank you for the suggestion. We will include these results in the revised manuscript to clarify UniSkill’s skill accumulation and maintenance within the evaluated range of skillbank sizes.

AuthorsOpenReview ↗

Response to Reviewer jhrc

Q3 (Corresponding to Weakness 3)
A3: Thank you for suggesting a more complete comparison of performance variability and computational cost. Regarding Skill1’s statistics across multiple runs, we checked the original paper and confirmed that its main results on both ALFWorld and WebShop are averages over three runs. Its appendix D.2 also reports the standard deviation of its ALFWorld success rate. We include this information alongside UniSkill’s results over three runs:

MethodALFWorld success rate (%)WebShop task scoreWebShop success rate (%)
Skill197.5 ± 0.689.782.9
UniSkill98.4 ± 0.890.5 ± 1.484.7 ± 0.5

Skill1 does not report the corresponding standard deviations for WebShop, so we retain its three-run averages for these metrics. These statistics provide a more complete account of the mean performance and available variability estimates for both methods.

Regarding computational cost under matched conditions, Skill1’s results are taken from its original paper. We also attempted to reproduce Skill1 using its official repository, but could not complete the reproduction because files required for execution were missing, an issue also reported by other users in the repository’s issue tracker. We therefore provide a computational cost comparison with Evolving-RL under the same training framework, ALFWorld setup, and eight-GPU hardware configuration. Evolving-RL also jointly optimizes the policy and skills, but evaluates skills through additional environment rollouts. Using 90% validation success rate as a common target, we estimate the cumulative cost of first reaching this target from the average per-step stage timings reported in Figure 3:

MethodTarget success rateSteps to first reach targetCumulative time for timed stagesGPU-hours
Evolving-RL90%4519.49 h155.92
UniSkill90%9513.37 h106.96

Although UniSkill requires more training steps, its lower per-step skill evaluation cost reduces the cumulative time for the timed stages by approximately 31.4% when reaching the same target success rate. We further compare cumulative costs over 250 training steps. For UniSkill, we report the recorded time; because Evolving-RL did not complete 250 steps, we extrapolate its time from the average per-step time of the completed training:

MethodTraining stepsTiming basisCumulative time for timed stagesGPU-hours
Evolving-RL250Extrapolated estimate108.28 h866.22
UniSkill250Measured35.18 h281.47

In addition, UniSkill’s total skill critic API cost for the 250-step run is 4.2 USD.

Thank you for the suggestion. We will include these run statistics and cost analyses in the revised manuscript to provide a more complete comparison of performance and efficiency.

Reviewer_DmZ9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to give useful learning feedback to skill proposals when an LLM agent policy and its skill memory are trained jointly. Prior feedback either rewards skills through later reuse, which conflates skill utility with policy improvement, or evaluates each candidate skill through extra environment rollouts, which is costly. UniSkill lets a single shared policy both act with retrieved skills and propose skillbank edits, choosing among adding a new skill, updating the retrieved one, or leaving the bank unchanged. Its key idea is contrastive action feedback: with the current policy and previously collected successful and failed trajectories of the same task held fixed, the method measures how swapping the retrieved skill for the proposed skill changes the gap between action log likelihoods on those trajectories, yielding an actor alignment signal that requires no new rollouts. Proposal format is checked directly, while a prompted external model serves as a skill critic that judges edit appropriateness and content grounding, and a support regularizer keeps the skill edit operations from collapsing to a single choice during training. The actor is trained with group relative policy optimization and the proposer with a REINFORCE style objective, and edits that pass both critic and alignment checks are committed to the skillbank so the policy and skills evolve together. Experiments on two interactive agent benchmarks show top performance among the compared methods, stable joint training where a closely related baseline that evaluates skills through extra rollouts collapses, reduced evaluation cost, and preserved effectiveness with a smaller backbone.

Critical strengths
  1. The problem is well identified and the proposed feedback fits it cleanly. The paper clearly articulates why skill utility must be judged relative to the actor that will use it, and why evaluating every proposal with additional rollouts is expensive. The contrastive likelihood gap offers an actor specific and rollout free signal that directly addresses this credit assignment gap in prior joint training schemes.

  2. The alignment signal is validated with unusual honesty. The authors measure how well the proxy agrees with actual rollout gains under a fixed policy, openly report a nontrivial set of disagreement cases, dissect both agreeing and disagreeing cases, and derive an additional admission safeguard from the observed failure modes. This level of self scrutiny is uncommon and strengthens trust in the method.

  3. The overall algorithmic story is coherent and empirically supported. Ablations isolate the respective roles of joint learning, alignment feedback, and the support regularizer; training remains stable over an extended schedule where the closest joint training baseline degrades; the measured per step cost is lower; and the approach retains its benefit with a smaller backbone, giving a fairly complete picture for the studied settings.

Critical weaknesses
  1. The framework leans on a large frozen external model as the skill critic. Rewards for edit appropriateness and content grounding are binary verdicts produced by a prompted external model that is kept frozen and separate from the trained policy, so part of the claimed self evolution is effectively outsourced. Critic errors propagate into both the proposer reward and the skillbank admission decision, and the reported robustness to alternative critics rests on single runs.

  2. The alignment proxy remains noisy and only moderately related to true utility. Positively scored proposals sometimes reduce actual performance, and some of these still pass every admission criterion and enter the skillbank, meaning harmful guidance can be committed and later retrieved. Meanwhile the reward magnitude and the operation probability floor are chosen heuristically from preliminary runs, without a systematic sensitivity analysis across environments, leaving the robustness of these choices unclear.

Questions for authors
  1. What is the long run fate of committed skills: how often are harmful entries later pruned or overwritten as training proceeds, and does skillbank quality keep improving or plateau?
  2. Could the contrastive feedback reward trajectory specific content rather than transferable guidance, and is there quantitative evidence of skill reuse beyond tasks that share an obvious subproblem?
4 comments
AuthorsOpenReview ↗

Response to Reviewer DmZ9

Q1 (Corresponding to Weakness 1)
A1: Thank you for your suggestions. Regarding your concern that critic errors may affect both rewards and skillbank admission decisions, we had all four critics evaluate the same randomly sampled set of 50 skill proposals. Human evaluators then examined the corresponding trajectories and skillbank context to separately verify whether each critic’s operation-appropriateness and content-grounding judgments were reasonable.

As reported in Appendix D.2, Table 6 of the original submission, we compared three external critics on WebShop. Following your suggestion to strengthen the robustness evidence beyond single runs, we conducted two additional independent training runs for each alternative critic and now report means and standard deviations across three runs for all three external critics. To further examine whether an external critic is necessary, we also conducted an experiment in which the current Qwen2.5-7B-Instruct shared policy serves as the actor, proposer, and skill critic. For the critic role, we use reinforcement learning with rewards based solely on whether its output follows the prescribed format, without additional task-specific SFT or external critic API calls. Task performance and human audit results are summarized below:

Skill criticTask scoreSuccess rate (%)Operation judgments passing human auditContent judgments passing human audit
DeepSeek-V4-Pro90.5 ± 1.484.7 ± 0.550/5050/50
DeepSeek-V4-Flash89.6 ± 1.282.8 ± 0.849/5049/50
Qwen3.5-35B-A3B91.4 ± 1.085.7 ± 1.650/5049/50
Qwen2.5-7B-Instruct (shared policy)89.883.650/5048/50

The three external critics achieve mean success rates of 82.8%–85.7%, with 98%–100% of judgments passing human audit for both criteria. The Qwen2.5-7B-Instruct shared-policy setting achieves an 83.6% success rate, with 100% and 96% of its operation and content judgments passing human audit, respectively. These results strengthen the evidence from repeated runs and the assessment of critic reliability. They also show that the shared policy can perform the auxiliary auditing role without external critic API calls while achieving similar task performance. Thank you again for your suggestions. We will include these results in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer DmZ9

Q2 (Corresponding to Weakness 2)
A2: Thank you for your careful analysis of alignment feedback reliability and the rationale for our parameter choices. Regarding your concern about the effects of admitted skills when subsequently retrieved, we conducted a controlled training-continuation experiment to track skills that satisfy all admission criteria but reduce execution success. We examined their subsequent use, updates, and changes in utility. Since this is closely related to the long-term evolution question you raised in Question for Authors 1, we report the tracking results together in A3.

Regarding the choice of reward magnitude η\eta, we selected η=0.05\eta=0.05 based on the magnitude distributions of alignment feedback on both benchmarks. Appendix B, Table 4 of the original submission reports the ALFWorld statistics: this threshold leaves 93.3% of the raw scores unclipped while limiting a small fraction of large-magnitude feedback. The corresponding proportion on WebShop is 98.4%, supporting our use of the same setting in both environments. The supplementary statistics are shown below:

BenchmarkSample sizeRalign<−0.05R_{\mathrm{align}}<-0.05∣Ralign∣≤0.05\lvert R_{\mathrm{align}}\rvert\leq0.05Ralign>0.05R_{\mathrm{align}}>0.05
ALFWorld5663.0%93.3%3.7%
WebShop (supplementary)5001.2%98.4%0.4%

The statistics from both environments show that η=0.05\eta=0.05 preserves the vast majority of raw alignment feedback while limiting a small fraction of large-magnitude values, providing a basis for using a common setting.

Regarding sensitivity to the operation probability floor pmin⁡p_{\min}, its purpose is to preserve exploration opportunities for each edit operation and prevent operations from being suppressed prematurely during training. As described in Appendix B, under an independent-sampling approximation with group size G=8G=8, an operation selected with probability pp appears at least once in a group with probability:

Pr⁡(Nop≥1)=1−(1−p)8.\Pr(N_{\mathrm{op}}\geq1)=1-(1-p)^8.

Using 50% as the reference target gives p≥0.083p\geq0.083. We therefore adopt the slightly higher value of pmin⁡=0.1p_{\min}=0.1 as the exploration-probability reference for support regularization. To further examine sensitivity, we adjusted the reference target to 25% and 75%, yielding probabilities of approximately 0.035 and 0.159, respectively. We used pmin⁡=0.04p_{\min}=0.04 and 0.160.16 and compared them with the default setting on WebShop, keeping all other training settings unchanged:

pmin⁡p_{\min}Task scoreSuccess rate (%)
0.0488.882.0
0.10 (default)90.584.7
0.1690.485.2

Across these settings, task scores range from 88.8 to 90.5, and success rates range from 82.0% to 85.2%. The weaker exploration constraint yields a lower success rate, while the stronger constraint achieves performance close to the default setting, indicating that effective training is not restricted to the single value pmin⁡=0.1p_{\min}=0.1. Thank you again for your suggestions. We will include the WebShop distribution statistics and the sensitivity experiments in the revised manuscript to more fully present the rationale for our parameter choices and performance under different operation probability floors.

AuthorsOpenReview ↗

Response to Reviewer DmZ9

Q3 (Corresponding to Question for Authors 1)
A3: In Weakness 2, you noted that skills with positive alignment scores may still reduce task performance even after satisfying all admission criteria, raising the concern that such “harmful guidance” could enter the skillbank and be retrieved again. This question further asks whether these skills are subsequently revised and whether the overall skillbank continues to improve. We conducted the following tracking and controlled evaluation experiments to examine both aspects.

As shown in the Q2 analysis of Section 5, Figure 5, and Appendix E, RalignR_{\mathrm{align}} distinguishes skill utility on average but does not guarantee positive execution gains for every admitted proposal. Skill utility depends on the task performance of the actor using it and cannot be determined solely by text quality or rule-based checks. Identifying whether a skill reduces performance therefore requires controlled rollouts with the actor and task held fixed. Following this principle, we examined both individual skill entries and the utility of the skillbank as a whole.

1. Identifying skills with negative rollout gains and tracking their subsequent use and updates.

Following the protocol in Appendix D.3, we fixed the actor at Step 25 and performed 32 rollouts each using the original skill and the candidate skill on the same task. Among 100 proposals, 53 received positive alignment scores, and nine both satisfied all admission criteria and exhibited negative rollout gains.

These candidate proposals were generated during a separate evaluation outside the training process and were not necessarily the entries admitted during the original training run. To track their subsequent evolution, we restored the Step 25 model and skillbank, inserted these skills according to the original ADD/UPDATE rules, and resumed training through Step 250. We tracked the nine admitted skill IDs and their text versions, without increasing their retrieval probabilities or forcing the proposer to modify them.

Tracking outcomeCount and proportion
Original version was retrieved for use8/9 (88.9%)
Entry was updated after its original version was used8/9 (88.9%)
Entry remained unchanged at Step 2501/9 (11.1%)

All proportions use the nine tracked entries as the denominator; retrieval and update events may overlap. UniSkill currently has no explicit deletion operation; existing content is replaced through UPDATE. The tracking results show that eight original versions with negative rollout gains were replaced with new content after being used, demonstrating that these entries can be revised through subsequent interactions. To further examine whether overall skillbank quality improves, we conducted the fixed-actor comparison below.

2. Evaluating skillbank utility over training with a fixed actor.

To assess changes in skillbank utility during normal training and distinguish them from improvements in the actor itself, we used skillbank snapshots saved during the original training run. We fixed the Step 75 actor, the same set of 128 validation tasks, and the retrieval settings, varying only the skillbank snapshot and measuring task success through environment rollouts:

Skillbank snapshotSuccessful tasksValidation success (%)
Initial empty bank74/12857.81
Bank 2580/12862.50
Bank 7580/12862.50
Bank 10083/12864.84
Bank 20089/12869.53

With the actor held fixed, success increases from 62.50% to 69.53% as the skillbank evolves from Bank 25 to Bank 200, showing an overall upward trend with further improvement at later stages. Bank 200 improves success by 11.72 and 7.03 percentage points over the empty bank and Bank 75, respectively, indicating that continued skillbank evolution provides further benefits to the same actor.

The entry-level tracking examines whether skills with negative rollout gains are subsequently updated and replaced, while the fixed-actor comparison evaluates whether skillbank evolution yields actual task gains. Thank you again for your thoughtful suggestion, which has helped us extend the analysis beyond initial skill admission to subsequent use, updates, and changes in utility.

AuthorsOpenReview ↗

Response to Reviewer DmZ9

Q4 (Corresponding to Question for Authors 2)
A4: Whether contrastive feedback might reward trajectory-specific content rather than transferable guidance was a key consideration in our design. As discussed in the “Skill Proposal Rewards” section, restating a particular trajectory may increase action likelihood without producing a reusable skill. Therefore, proposals first undergo skill critic checks, and RalignR_{\mathrm{align}} is computed only after they pass these checks. The critic prompt in Appendix A.3 explicitly instructs the critic to reject proposals that merely restate the task or outcome, or copy instance-specific identifiers, locations, or entire trajectories without reusable abstraction.

To examine the effectiveness of this screening, we randomly sampled 50 proposals that passed the critic checks and received Ralign>0R_{\mathrm{align}}>0 during WebShop training, and manually audited them against their source trajectories:

Provide reusable guidanceMerely restate trajectory details
49/50 (98%)1/50 (2%)

Regarding quantitative evidence of cross-task reuse, we further analyzed retrieval logs from WebShop training. Among the 54 entries in the final skillbank, 38 (70.4%) were retrieved for use in at least three distinct tasks.

For example, skill 3cc567493c originated from a task involving the purchase of women’s shorts and distilled the general principle of explicitly incorporating all purchase constraints into the search query. This principle accommodates the attribute requirements of different products. The skill was subsequently retrieved for tasks involving blankets, men’s loafers, and women’s sweaters, covering different combinations of constraints such as environmental friendliness and dimensions, slip resistance and shoe size, and sleeve length and fit. This example illustrates how guidance abstracted from a specific purchasing task can be reused across tasks.

The human audit indicates that positive-scoring proposals predominantly provide reusable guidance, while the retrieval statistics and example demonstrate reuse across products, attribute values, and constraint combinations. Thank you for your suggestion. We will include these audit results, reuse statistics, and the illustrative example in the revised manuscript.

Reviewer_hkMmOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Large Language Model (LLM) agents can improve performance by storingReusable textual skills in a skillbank. While jointly optimizing the agent (Actor) and skill extractor (Proposer) allows both to co-evolve, traditional approaches struggle with evaluation delays, credit conflation, and high computation costs from needing additional environment rollouts for every proposed skill. To address these challenges, UniSkill introduces a shared-policy framework where a single model acts as both the task executor and skill proposer. Instead of costly new rollouts, UniSkill employs Contrastive Action Feedback—evaluating proposed skills by measuring how they alter the current actor's action log-likelihood gap between previously collected successful and failed trajectories from the same task. Coupled with Skill-Edit Support Regularization to prevent premature suppression of edit operations, UniSkill achieves state-of-the-art performance (98.4% success on ALFWorld and 84.7% on WebShop) with significantly greater training efficiency and stability.

Critical strengths
  1. Rollout-Free Actor-Aligned Feedback: Addresses the critical challenges of high skill-evaluation costs and credit assignment conflation. Instead of performing new environment rollouts, it uses the current Actor to compute the log-likelihood difference of recorded actions (RalignR_{\text{align}}) between original and proposed skills on existing successful/failed trajectory pairs, dramatically reducing computational overhead while ensuring instant, accurate alignment.

  2. Support Regularization to Prevent Exploration Collapse: Solves the severe training instability typically seen in joint training. To prevent poor skill content from inadvertently suppressing valid skill-edit operations (ADD/UPDATE), the paper enforces a probability floor (pmin⁡p_{\min}) across edit choices. This prevents the model from collapsing into a single operation during late-stage training, guaranteeing sustained stability throughout long-term policy-skill co-evolution.

Critical weaknesses
  1. Reliance on an External High-Capability LLM as Skill Critic: Although the Shared-Policy design (sharing weights between Actor and Proposer) lowers training overhead, the framework still requires a frozen high-capability external model (DeepSeek-V4-Pro in the paper) to serve as a Skill Critic for format validation, action validity checks, and trajectory grounding verification. It does not achieve complete "single-model self-contained co-evolution," remaining dependent on calls to a high-quality external API during training and deployment.

  2. Residual Mismatch Between Contrastive Feedback: As highlighted in the Q2 analysis, RalignR_{\text{align}} serves only as an offline proxy signal and cannot 100% guarantee performance gains when a proposed skill is executed in the actual environment (e.g., at Step 25, ~24.5% of positive-scoring proposals yielded negative actual rollout gains). If the Actor fails to execute multi-step logic in the new skill correctly, or if the skill text leaves implicit constraints unstated, offline action log-likelihood alignment signals can lead to false positives.

  3. High Dependency on the Quality and Diversity of Historical Trajectories: Contrastive Action Feedback relies on having paired successful and failed reference trajectories from the same task. In early training stages or on extremely difficult tasks where the agent fails to discover successful trajectories (or succeeds on all attempts, lacking failure cases), the contrastive feedback signal cannot be constructed, potentially hindering the learning of the Skill Proposer.

Questions for authors

see Weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer hkMm

Q3 (Corresponding to Weakness 3)
A3: Thank you for highlighting the availability of reference trajectories at different training stages. As described in Appendix B of the original submission, training begins with a warm-up phase: the actor learns from task and format rewards, while the proposer receives only format rewards. This helps the model adapt to environment interaction and the skill-editing format, preparing it for subsequent joint learning.

We further measured the number of proposals generated after the reference-trajectory requirements were met, alongside validation performance at each training stage:

EnvironmentTraining stepsMean proposals generated per stepValidation success at stage end (%)
ALFWorld1–5052.0980.5
ALFWorld51–20018.7297.7
ALFWorld201–2507.20100.0
WebShop1–5049.6069.5
WebShop51–20045.8775.0
WebShop201–25022.7285.2

Early in training, both environments yield approximately 50 proposals per step on average, indicating that online exploration produces successful and failed reference trajectories that support proposer learning. Later in training, ALFWorld validation success reaches 100%, while the mean proposal count decreases to 7.20 per step. On WebShop, validation success reaches 85.2%, leaving room for further improvement, and an average of 22.72 proposals per step continues to provide data for skill learning.

The actor’s task proficiency is closely linked to both the opportunities for proposer learning and the need for further skill improvement. As the actor becomes able to complete tasks consistently, failed trajectories become less frequent, and the need for further skill refinement also diminishes. When task performance still leaves room for improvement, successful and failed trajectories continue to provide learning opportunities for the proposer. Proposal counts should therefore be interpreted alongside the actor’s current performance; a decrease in proposal count alone does not indicate that learning is being hindered. This changing demand for skill improvement is a natural part of actor–proposer co-evolution.

These results show that actor–proposer joint learning remains effective as trajectory availability evolves over training in both environments. Thank you again for your thoughtful suggestion, which has helped us explain more clearly how trajectory availability relates to actor–proposer joint learning.

AuthorsOpenReview ↗

Response to Reviewer hkMm

Q2 (Corresponding to Weakness 2)
A2: Thank you for your careful analysis of the discrepancy between contrastive feedback and actual execution gains. This is precisely why we conducted the rollout-based validation in the Q2 analysis of Section 5. Our goal is to provide a timely learning signal based on the current actor while reducing feedback delays and avoiding additional evaluation rollouts for each candidate skill. To assess its effectiveness, we examine both its agreement with actual execution gains and whether it improves final task performance through training ablations.

Regarding the false positives you highlighted, the evaluation in Figure 5 holds the actor and task fixed and compares success rates over 32 rollouts each using the original skill and the candidate skill. Following the same protocol in Appendix D.3, we extended this evaluation to Step 100:

CheckpointMean gain for positive-scoring proposals (pp)Mean gain for negative-scoring proposals (pp)Positive-scoring proposals with negative rollout gains
Step 25+8.4−3.424.5%
Step 75+7.2−0.515.6%
Step 100 (additional)+4.9−0.95.1%

Positive and negative scores refer to Ralign>0R_{\mathrm{align}}>0 and Ralign<0R_{\mathrm{align}}<0, respectively. Across these checkpoints, the proportion of false positives decreases from 24.5% to 15.6%, and then to 5.1%. Meanwhile, positive-scoring proposals consistently yield positive mean rollout gains, whereas negative-scoring proposals yield negative mean gains. These results indicate that contrastive feedback distinguishes skill utility on average, with a decreasing proportion of false positives as training progresses across the evaluated checkpoints.

The training ablation in Table 2 further supports the learning value of this feedback. With the critic checks and skillbank update rules kept unchanged, including the proposer’s alignment reward increases ALFWorld success from 89.1% to 98.4%. Even with skill retrieval disabled at evaluation time, success increases from 84.4% to 96.1%. Thus, without requiring additional evaluation rollouts for each candidate skill, this feedback provides a learning signal that improves task performance through joint training.

Thank you again for the suggestion. We will include the Step 100 rollout evaluation in the revised manuscript to more fully document both the prediction errors of contrastive feedback and its benefits for learning.

AuthorsOpenReview ↗

Response to Reviewer hkMm

Q1 (Corresponding to Weakness 1)
A1: Thank you for raising this question and giving us the opportunity to clarify the responsibilities of the skill critic and when it is used. Skill proposal formatting is checked programmatically. The skill critic assesses whether the skill-edit operation is appropriate and whether the content is grounded in the trajectories and provides reusable guidance, while Contrastive Action Feedback evaluates how well the proposed skill aligns with the current actor. In the main experiments, we use an independent, frozen skill critic so that, with the critic held fixed, we can focus on studying how actor-specific alignment feedback supports actor–proposer joint learning within UniSkill. Moreover, inference with the trained policy and skillbank does not require any external critic API calls.

Regarding the reliance on a high-capability external model, Appendix D.2, Table 6 of the original submission compares three external critics, with similar performance across these choices. To further explore the feasibility of “single-model self-contained co-evolution” in response to your concern, we conducted an additional experiment in which the current Qwen2.5-7B-Instruct shared policy serves as the actor, proposer, and critic. For the critic role, we apply reinforcement learning using only a format reward based on whether its output follows the prescribed format, without additional task-specific SFT or external critic API calls. We also manually audited the judgments of all four critics on the same 50 skill proposals, separately assessing their operation-appropriateness and content-grounding judgments. The WebShop performance and audit results are shown below:

Skill criticTask scoreSuccess rate (%)Operation judgments passing human auditContent judgments passing human audit
DeepSeek-V4-Pro90.584.750/5050/50
DeepSeek-V4-Flash88.983.649/5049/50
Qwen3.5-35B-A3B90.485.250/5049/50
Qwen2.5-7B-Instruct (shared policy)89.883.650/5048/50

The shared-policy setting achieves an 83.6% success rate, with 100% and 96% of its operation and content judgments passing human audit, respectively. These results provide evidence that the shared policy can also perform the auxiliary auditing role, achieving similar WebShop performance without relying on an external critic API. Thank you again for the suggestion. We will include these experiments in the revised manuscript and clarify the critic’s responsibilities and usage stages.

AuthorsOpenReview ↗

A Kind Reminder

Dear Reviewer hkMm,

Thanks for your time and effort in reviewing this paper. Your comments are valuable for improving the quality of this paper. As the author–reviewer discussion period will end in about 2 days, we would appreciate it if you could review our responses and let us know whether they address your concerns.

Thanks again for your time and help in improving the quality of this paper.

Warm regards,
The Authors

Comments on the paper

1 comment

Dear Reviewer DmZ9,

Thank you for taking the time to carefully review our responses and supplementary experiments. We are glad that they have addressed your main concerns. Your thoughtful feedback has helped us strengthen both the experimental evidence and the presentation of our work. We will incorporate these results and clarifications into the revised manuscript.

Thank you again for your constructive comments and consideration.

Warm regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.