- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a skill internalization framework, which involves two key modules. The environment harness module provides in-trajectory guidance using generated code. The skill internalization procedure applies SFT on assisted and successful trajectories weighted by assistance benefit, and applies RL on unassisted trajectories. Experiments are conducted on the proposed HARD-ALFWORLD and Search-QA, showing the effectiveness of the proposed method. Further analysis of the learned skill shows that the proposed method can better internalize the skill bank.
Critical strengths
- The core method is clearly described using mathematical formulations, which is easy to read.
- The proposed method is thoroughly evaluated in two domains. Ablations, analyse on the training efficiency and further analysis on the constructed skill bank are also useful.
Critical weaknesses
- Many important components of skill learning are omitted from the main text. (1) The task q, the skill, the skill set Kq, and the skill bank Ms should be formally defined in Preliminaries, as these are core concepts of this paper. (2) The skill learning process and the skill bank maintenance method (currently Appendix B.5) should be placed in the method section, as the proposed framework is incomplete without them.
- The authors implicitly assume that the rules can be expressed through multiple conditional branches (page 5, around line 216), which limit the expressiveness of the skills.
Questions for authors
- How to ensure the code generated by the environment harness are correct and faithfully provide the expected guidance of the selected skill? If the code contains some runtime error or the branch condition is incorrect, will the proposed method still perform well?
- Could you please provide an example of one complex skill that can not be expressed through multiple conditional branches, and show how the environment harness behaves in such scenarios?
- The authors propose a new benchmark which is claimed to be "more challenging long-horizon interactions". Could you also report the performance on the original ALFWorld benchmark, which is widely used and can help readers better compare your algorithm with others?
- As no citations exist in Appendix B.5, I assume the skill bank maintenance process is also a new module the authors propose, i.e., it is different from the baseline. Could you please briefly introduce the difference of this process with baselines? I wonder whether this is a key module that makes the proposed method outperform the baseline. How will the proposed method perform if it directly uses the skill learning process from the baselines?
- In Table 3, why do SkillDuet w/o environment harness and SkillDuet w/o agent harness cost more tokens than SkillDuet?
- (minor) The definition of sequence concatenation should be put on line 139, not line 148, as this operator is already used there.
4 comments
Response to Reviewer jekY (2/2)
[Q1] Correctness and robustness of generated rule code.
Thanks for the question. (1) To ensure feedback quality, we address this at three stages: (i) rule generation, requiring observation-grounded triggers and avoiding redundant feedback; (ii) rule execution, returning empty feedback for invalid code or unmet conditions rather than random feedback; and (iii) skill-bank maintenance, removing consistently harmful skills to prevent them from generating misleading feedback.
(2) We additionally use an LLM judge (DeepSeek-V4.1-Flash) to evaluate 17,047 sampled feedback messages, finding that 97.69% are consistent with their source skills and 82.92% are relevant to the current decision state, consistent with our manual review of 100 sampled messages.
(3) Despite occasional rule errors, SkillDuet outperforms the variant without the environment harness (Tab. 3 of the original manuscript), suggesting that feedback benefits outweigh error costs.
[Q4] Unclear novelty and contribution of skill bank maintenance compared with baselines.
Thanks for the feedback. (1) Our skill-bank maintenance differs from the baselines in three aspects. (i) Skill retrieval: Following EnvHarness, we use embedding similarity filtering followed by top-K MMR selection, whereas Skill0 uses category matching, Skill0.5 uses top-K cosine similarity, and D2Skill combines similarity, historical utility, and UCB exploration. (ii) Skill discovery: Following SkillRL, we extract category-specific strategies from successes and common mistakes from failures, but use the current policy on online unassisted trajectories without further general-skill summarization. In contrast, Skill0/0.5 use fixed skill banks, D2Skill triggers discovery when assisted success falls below 0.5 using an external LLM, and EnvHarness generates skills offline from reshaped environments. (iii) Skill removal: We remove skills after consecutive retrieval steps with negative assistance benefit, whereas D2Skill removes low-utility skills only when capacity is exceeded, and Skill0 progressively withdraws guidance through a curriculum. We have added these details to Apps. B.2 and B.5 of the revised manuscript.
(2) Is skill-bank maintenance the key to the gains? We attribute the main gains to dual-harness guidance. In Tab. 3 of the original manuscript, removing either harness while retaining identical skill-bank maintenance reduces Avg. Succ. from 38.3 to 32.4/33.8. Thus, skill-bank maintenance alone cannot explain the gains.
(3) As suggested, we additionally supplement SkillDuet w/ fixed skill bank, replacing online maintenance with the fixed SkillRL-based bank used by Skill0/0.5, while preserving dual-harness guidance and PAIR. As shown in Tab. R2, the fixed-bank variant achieves 36.4 ± 1.5 / 29.6 ± 2.1 success on valid-seen / valid-unseen, 1.9 / 0.7 percentage points below SkillDuet. However, it still outperforms Skill0.5 (35.7 / 22.6) and Skill0 (33.8 / 27.8), indicating that online maintenance contributes additional gains but does not fully account for SkillDuet’s advantage. We have added this comparison to App. B.5.3 (Tab. 9) of the revised manuscript.
Table R2: SkillDuet with online skill-bank maintenance versus the fixed SkillRL-based bank used by Skill0 and Skill0.5, using Qwen2.5-7B-Instruct on Hard-ALFWorld. Toks. are reported in thousands; Avg. Succ. reports mean ± standard deviation over three evaluation runs.
| Split | Method | Direct | Single | Dual | Triple | Quad | Toks. | Turns | Avg. Succ. |
|---|---|---|---|---|---|---|---|---|---|
| Valid-seen | SkillDuet (Ours) | 64.3 | 60.7 | 41.7 | 17.9 | 7.1 | 101.5 | 43.0 | 38.3 ± 0.9 |
| SkillDuet w/ fixed skill bank | 63.1 | 58.3 | 40.5 | 14.3 | 6.0 | 82.7 | 38.0 | 36.4 ± 1.5 | |
| Valid-unseen | SkillDuet (Ours) | 53.1 | 55.6 | 21.0 | 17.3 | 3.8 | 80.8 | 34.3 | 30.3 ± 2.1 |
| SkillDuet w/ fixed skill bank | 55.6 | 48.1 | 23.5 | 13.6 | 6.4 | 86.2 | 39.3 | 29.6 ± 2.1 |
[Q5] Higher token costs of single-harness variants than SkillDuet.
Thanks for the question.
Each decision follows the format <think>{reasoning}</think><action>{action}</action>. w/o environment harness and w/o agent harness generate an average of 63.7 and 33.5 reasoning tokens per decision, respectively, compared with 19.7 for SkillDuet. This suggests that dual-harness guidance, combining start-of-trajectory skills with in-trajectory feedback, may help the agent make more informed decisions with less reflection. Moreover, the lower reasoning-token count of w/o agent harness may reflect the more frequent guidance provided by in-trajectory feedback compared with start-of-trajectory skills alone.
[Q6] Misplaced definition of sequence concatenation.
Thanks for the feedback. We have moved the definition of to its first use in Sec. 3.1 and removed the duplicate in Sec. 3.2.
Thanks for the detailed response. The authors have addressed my concerns. Experiments on the original ALFWorld benchmark justify the necessity of the new benchmark. The explanation of the skill-learning process and the rule generation process enhances the completeness of this paper. Therefore, I maintain my positive rating.
We greatly appreciate your positive support
We are glad that you appreciate our efforts, and sincerely thank you for your positive score! Your constructive feedback has greatly helped us improve our paper. Have a wonderful day! :)
Response to Reviewer jekY (1/2)
Dear jekY,
Thank you for your thoughtful review! We are glad that you find our paper easy to read and appreciate the ablations, training efficiency analysis, and skill bank analysis. Due to space limitations, we split our response into two comments: this comment addresses W1–W2 and Q2–Q3, while the second addresses Q1 and Q4–Q6. We apologize for any inconvenience and appreciate your understanding.
[W1] Key definitions and skill learning details are missing from the main text.
Thanks for the feedback. (1) The definitions of task, skill, skill set, and skill bank, previously spread across Secs. 1, 3.1, 4.1, and App. B.2 of the original manuscript, have now been consolidated in Sec. 3.1 of the revised manuscript. A skill m is a reusable strategy distilled from prior interactions and stored as structured text. The bank Ms starts empty and is updated online. For task q, Kq contains up to K skills retrieved from Ms by embedding-based maximal marginal relevance, or is empty when none meets the relevance threshold.
(2) As suggested, we have added Sec. 4.3 in the revised manuscript, summarizing the procedure previously in App. B.5 of the original manuscript. After each training step, the current policy discovers strategies and common mistakes from representative unassisted trajectories, followed by deduplication. Skills are removed after C consecutive retrieval steps with negative average assistance benefit.
[W2] Conditional rule-based compilation limits skill expressiveness.
Thanks for the comment. (1) To preserve skill expressiveness, we supplement rather than replace the agent harness with the environment harness. The agent harness provides complete natural-language skills at the start of each trajectory, including those difficult to compile into rules, while the environment harness adds in-trajectory feedback.
(2) To avoid redundant guidance, conditional rules provide only feedback relevant to the current state rather than complete skills. In Tab. 3 of the original manuscript, Triggered full-skill guidance uses the same triggers but returns complete skills. It perform worse (32.8 vs. 38.3) and uses more tokens (122.5k vs. 101.5k). This demonstrates the benefit of targeted feedback.
[Q2] Handling of complex skills that cannot be expressed through conditional branches.
Thanks for the feedback. (1) During rule generation, skills that cannot be expressed as conditional branches are skipped. If no skills can be compiled, the environment harness returns no feedback. However, these skills can still guide the agent through the complete skills provided by the agent harness at the start of each trajectory.
(2) The following Hard-ALFWorld skill was not compiled because minimizing redundant travel is a global planning principle rather than a step-specific rule with observable triggers.
[Q3] Missing evaluation on the original ALFWorld benchmark.
Thanks for the suggestion. As suggested, we additionally evaluate ReAct, GRPO, and SkillDuet on the original ALFWorld using Qwen3-4B-Instruct-2507 and Qwen2.5-7B-Instruct under our main settings. Both trained methods approach ceiling performance, leaving limited room for improvement. This motivates Hard-ALFWorld's instance-bound goals, same-type distractors, and initial-scene-grounded references, which require longer-term information tracking (Sec. 4.4 of the revised manuscript). On Hard-ALFWorld, GRPO achieves only 33.3% and 24.9% on valid-seen and valid-unseen (Tabs. 1 and 2 of the original manuscript). We have added these results to App. B.1.7 (Tab. 8) of the revised manuscript.
Table R1: Success rates (%) on the original ALFWorld. Toks. are reported in thousands.
Qwen3-4B-Instruct-2507
Qwen2.5-7B-Instruct