acceptodds
Under review as a conference paper at ICLR 2027

SkillDuet: Internalizing Skills through Agent–Environment Dual Harnesses

Abstract

Skills are reusable strategies distilled from prior interactions that guide agents in task execution. Although skills improve inference-time performance, they do not become intrinsic agent capabilities, motivating skill internalization for autonomous execution without external guidance. However, existing methods typically rely on start-of-trajectory skill guidance, whose influence may gradually diminish over long interaction. We propose SkillDuet, a novel skill-internalization framework that can deliver in-trajectory guidance, providing persistent supervision at relevant decision points. To realize such supervision, the core of SkillDuet is agent–environment dual harnesses: beyond the start-of-trajectory guidance provided by an agent harness, an environment harness augments observations with skill-derived feedback to provide in-trajectory guidance. We further design Paired Trajectory Internalization procedure, which collects paired assisted and unassisted trajectories, then converts successful assisted trajectories into skill-free supervision by removing external guidance. It internalizes skill-guided behavior through SFT with task-benefit weighting on converted trajectories, while using RL on unassisted trajectories to reinforce autonomous execution. Furthermore, to evaluate skill internalization in challenging long-horizon tasks, we introduce Hard-ALFWorld, an ALFWorld variant with multiple goals, same-type distractors, and scene-grounded object references. Experiments on Hard-ALFWorld and Search-QA with two backbones demonstrate that SkillDuet achieves leading performance in most settings and remains competitive across all evaluations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Skill Internalization, LLM Agents, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jekYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a skill internalization framework, which involves two key modules. The environment harness module provides in-trajectory guidance using generated code. The skill internalization procedure applies SFT on assisted and successful trajectories weighted by assistance benefit, and applies RL on unassisted trajectories. Experiments are conducted on the proposed HARD-ALFWORLD and Search-QA, showing the effectiveness of the proposed method. Further analysis of the learned skill shows that the proposed method can better internalize the skill bank.

Critical strengths
  1. The core method is clearly described using mathematical formulations, which is easy to read.
  2. The proposed method is thoroughly evaluated in two domains. Ablations, analyse on the training efficiency and further analysis on the constructed skill bank are also useful.
Critical weaknesses
  1. Many important components of skill learning are omitted from the main text. (1) The task q, the skill, the skill set Kq, and the skill bank Ms should be formally defined in Preliminaries, as these are core concepts of this paper. (2) The skill learning process and the skill bank maintenance method (currently Appendix B.5) should be placed in the method section, as the proposed framework is incomplete without them.
  2. The authors implicitly assume that the rules can be expressed through multiple conditional branches (page 5, around line 216), which limit the expressiveness of the skills.
Questions for authors
  1. How to ensure the code generated by the environment harness are correct and faithfully provide the expected guidance of the selected skill? If the code contains some runtime error or the branch condition is incorrect, will the proposed method still perform well?
  2. Could you please provide an example of one complex skill that can not be expressed through multiple conditional branches, and show how the environment harness behaves in such scenarios?
  3. The authors propose a new benchmark which is claimed to be "more challenging long-horizon interactions". Could you also report the performance on the original ALFWorld benchmark, which is widely used and can help readers better compare your algorithm with others?
  4. As no citations exist in Appendix B.5, I assume the skill bank maintenance process is also a new module the authors propose, i.e., it is different from the baseline. Could you please briefly introduce the difference of this process with baselines? I wonder whether this is a key module that makes the proposed method outperform the baseline. How will the proposed method perform if it directly uses the skill learning process from the baselines?
  5. In Table 3, why do SkillDuet w/o environment harness and SkillDuet w/o agent harness cost more tokens than SkillDuet?
  6. (minor) The definition of sequence concatenation should be put on line 139, not line 148, as this operator is already used there.
4 comments
AuthorsOpenReview ↗

Response to Reviewer jekY (1/2)

Dear jekY,

Thank you for your thoughtful review! We are glad that you find our paper easy to read and appreciate the ablations, training efficiency analysis, and skill bank analysis. Due to space limitations, we split our response into two comments: this comment addresses W1–W2 and Q2–Q3, while the second addresses Q1 and Q4–Q6. We apologize for any inconvenience and appreciate your understanding.

[W1] Key definitions and skill learning details are missing from the main text.

Thanks for the feedback. (1) The definitions of task, skill, skill set, and skill bank, previously spread across Secs. 1, 3.1, 4.1, and App. B.2 of the original manuscript, have now been consolidated in Sec. 3.1 of the revised manuscript. A skill mm is a reusable strategy distilled from prior interactions and stored as structured text. The bank MsM_s starts empty and is updated online. For task qq, KqK_q contains up to KK skills retrieved from MsM_s by embedding-based maximal marginal relevance, or is empty when none meets the relevance threshold.

(2) As suggested, we have added Sec. 4.3 in the revised manuscript, summarizing the procedure previously in App. B.5 of the original manuscript. After each training step, the current policy discovers strategies and common mistakes from representative unassisted trajectories, followed by deduplication. Skills are removed after CC consecutive retrieval steps with negative average assistance benefit.


[W2] Conditional rule-based compilation limits skill expressiveness.

Thanks for the comment. (1) To preserve skill expressiveness, we supplement rather than replace the agent harness with the environment harness. The agent harness provides complete natural-language skills at the start of each trajectory, including those difficult to compile into rules, while the environment harness adds in-trajectory feedback.

(2) To avoid redundant guidance, conditional rules provide only feedback relevant to the current state rather than complete skills. In Tab. 3 of the original manuscript, Triggered full-skill guidance uses the same triggers but returns complete skills. It perform worse (32.8 vs. 38.3) and uses more tokens (122.5k vs. 101.5k). This demonstrates the benefit of targeted feedback.


[Q2] Handling of complex skills that cannot be expressed through conditional branches.

Thanks for the feedback. (1) During rule generation, skills that cannot be expressed as conditional branches are skipped. If no skills can be compiled, the environment harness returns no feedback. However, these skills can still guide the agent through the complete skills provided by the agent harness at the start of each trajectory.

(2) The following Hard-ALFWorld skill was not compiled because minimizing redundant travel is a global planning principle rather than a step-specific rule with observable triggers.

When multiple objects must be moved to different destinations, group actions by destination or by object to reduce redundant travel.


[Q3] Missing evaluation on the original ALFWorld benchmark.

Thanks for the suggestion. As suggested, we additionally evaluate ReAct, GRPO, and SkillDuet on the original ALFWorld using Qwen3-4B-Instruct-2507 and Qwen2.5-7B-Instruct under our main settings. Both trained methods approach ceiling performance, leaving limited room for improvement. This motivates Hard-ALFWorld's instance-bound goals, same-type distractors, and initial-scene-grounded references, which require longer-term information tracking (Sec. 4.4 of the revised manuscript). On Hard-ALFWorld, GRPO achieves only 33.3% and 24.9% on valid-seen and valid-unseen (Tabs. 1 and 2 of the original manuscript). We have added these results to App. B.1.7 (Tab. 8) of the revised manuscript.

Table R1: Success rates (%) on the original ALFWorld. Toks. are reported in thousands.

Qwen3-4B-Instruct-2507

SplitMethodPickLookCleanHeatCoolPick2Toks.TurnsAvg. Succ.
Valid-seenReAct3.80.03.72.11.31.4131.449.32.4 ± 0.9
GRPO100.0100.098.889.6100.098.634.019.798.3 ± 0.7
SkillDuet (Ours)100.0100.098.893.8100.0100.029.317.799.0 ± 0.3
Valid-unseenReAct11.10.06.50.04.83.9130.948.54.7 ± 1.3
GRPO100.096.398.9100.0100.096.127.117.198.8 ± 1.3
SkillDuet (Ours)100.096.3100.097.1100.098.024.815.898.8 ± 1.3

Qwen2.5-7B-Instruct

SplitMethodPickLookCleanHeatCoolPick2Toks.TurnsAvg. Succ.
Valid-seenReAct1.00.03.70.04.01.49.55.21.9 ± 1.4
GRPO99.0100.0100.097.9100.0100.09.89.099.5 ± 0.3
SkillDuet (Ours)99.0100.0100.0100.0100.0100.012.310.299.8 ± 0.3
Valid-unseenReAct2.80.01.14.41.60.06.14.21.7 ± 1.3
GRPO98.698.1100.098.6100.0100.012.910.499.3 ± 0.6
SkillDuet (Ours)98.6100.0100.098.6100.0100.013.610.899.5 ± 0.4
AuthorsOpenReview ↗

Response to Reviewer jekY (2/2)

[Q1] Correctness and robustness of generated rule code.

Thanks for the question. (1) To ensure feedback quality, we address this at three stages: (i) rule generation, requiring observation-grounded triggers and avoiding redundant feedback; (ii) rule execution, returning empty feedback for invalid code or unmet conditions rather than random feedback; and (iii) skill-bank maintenance, removing consistently harmful skills to prevent them from generating misleading feedback.

(2) We additionally use an LLM judge (DeepSeek-V4.1-Flash) to evaluate 17,047 sampled feedback messages, finding that 97.69% are consistent with their source skills and 82.92% are relevant to the current decision state, consistent with our manual review of 100 sampled messages.

(3) Despite occasional rule errors, SkillDuet outperforms the variant without the environment harness (Tab. 3 of the original manuscript), suggesting that feedback benefits outweigh error costs.


[Q4] Unclear novelty and contribution of skill bank maintenance compared with baselines.

Thanks for the feedback. (1) Our skill-bank maintenance differs from the baselines in three aspects. (i) Skill retrieval: Following EnvHarness, we use embedding similarity filtering followed by top-K MMR selection, whereas Skill0 uses category matching, Skill0.5 uses top-K cosine similarity, and D2Skill combines similarity, historical utility, and UCB exploration. (ii) Skill discovery: Following SkillRL, we extract category-specific strategies from successes and common mistakes from failures, but use the current policy on online unassisted trajectories without further general-skill summarization. In contrast, Skill0/0.5 use fixed skill banks, D2Skill triggers discovery when assisted success falls below 0.5 using an external LLM, and EnvHarness generates skills offline from reshaped environments. (iii) Skill removal: We remove skills after CC consecutive retrieval steps with negative assistance benefit, whereas D2Skill removes low-utility skills only when capacity is exceeded, and Skill0 progressively withdraws guidance through a curriculum. We have added these details to Apps. B.2 and B.5 of the revised manuscript.

(2) Is skill-bank maintenance the key to the gains? We attribute the main gains to dual-harness guidance. In Tab. 3 of the original manuscript, removing either harness while retaining identical skill-bank maintenance reduces Avg. Succ. from 38.3 to 32.4/33.8. Thus, skill-bank maintenance alone cannot explain the gains.

(3) As suggested, we additionally supplement SkillDuet w/ fixed skill bank, replacing online maintenance with the fixed SkillRL-based bank used by Skill0/0.5, while preserving dual-harness guidance and PAIR. As shown in Tab. R2, the fixed-bank variant achieves 36.4 ± 1.5 / 29.6 ± 2.1 success on valid-seen / valid-unseen, 1.9 / 0.7 percentage points below SkillDuet. However, it still outperforms Skill0.5 (35.7 / 22.6) and Skill0 (33.8 / 27.8), indicating that online maintenance contributes additional gains but does not fully account for SkillDuet’s advantage. We have added this comparison to App. B.5.3 (Tab. 9) of the revised manuscript.

Table R2: SkillDuet with online skill-bank maintenance versus the fixed SkillRL-based bank used by Skill0 and Skill0.5, using Qwen2.5-7B-Instruct on Hard-ALFWorld. Toks. are reported in thousands; Avg. Succ. reports mean ± standard deviation over three evaluation runs.

SplitMethodDirectSingleDualTripleQuadToks.TurnsAvg. Succ.
Valid-seenSkillDuet (Ours)64.360.741.717.97.1101.543.038.3 ± 0.9
SkillDuet w/ fixed skill bank63.158.340.514.36.082.738.036.4 ± 1.5
Valid-unseenSkillDuet (Ours)53.155.621.017.33.880.834.330.3 ± 2.1
SkillDuet w/ fixed skill bank55.648.123.513.66.486.239.329.6 ± 2.1

[Q5] Higher token costs of single-harness variants than SkillDuet.

Thanks for the question. Each decision follows the format <think>{reasoning}</think><action>{action}</action>. w/o environment harness and w/o agent harness generate an average of 63.7 and 33.5 reasoning tokens per decision, respectively, compared with 19.7 for SkillDuet. This suggests that dual-harness guidance, combining start-of-trajectory skills with in-trajectory feedback, may help the agent make more informed decisions with less reflection. Moreover, the lower reasoning-token count of w/o agent harness may reflect the more frequent guidance provided by in-trajectory feedback compared with start-of-trajectory skills alone.


[Q6] Misplaced definition of sequence concatenation.

Thanks for the feedback. We have moved the definition of ⊕\oplus to its first use in Sec. 3.1 and removed the duplicate in Sec. 3.2.

Thanks for the detailed response. The authors have addressed my concerns. Experiments on the original ALFWorld benchmark justify the necessity of the new benchmark. The explanation of the skill-learning process and the rule generation process enhances the completeness of this paper. Therefore, I maintain my positive rating.

AuthorsOpenReview ↗

We greatly appreciate your positive support

We are glad that you appreciate our efforts, and sincerely thank you for your positive score! Your constructive feedback has greatly helped us improve our paper. Have a wonderful day! :)

Reviewer_SsVDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SkillDuet, a framework for internalizing external skills into LLM agents so that they can perform tasks without relying on skill retrieval at inference time. The main motivation is that skills provided only at the beginning of a trajectory may become less effective during long-horizon interactions.

To address this, SkillDuet introduces two complementary harnesses. The agent harness provides retrieved skills at the beginning of a task, while the environment harness converts the same skills into conditional rules that provide feedback during execution. The framework then collects assisted and unassisted trajectories. Successful assisted trajectories are converted into skill-free training examples by removing the external guidance, and the proposed PAIR objective combines weighted SFT on these trajectories with RL on unassisted trajectories.

The authors also introduce Hard-ALFWorld, an extended version of ALFWorld involving multiple goals, distractors, and grounded object references. Experiments on Hard-ALFWorld and Search-QA using two Qwen backbones show that SkillDuet achieves the highest aggregate performance in seven out of eight evaluation settings, without requiring external skill guidance at inference time.

Critical strengths

S1. A practical approach to learning from external skills without requiring them at inference time.

I find the separation between training-time assistance and skill-free inference to be one of the more appealing aspects of the paper. Rather than simply retrieving skills at each decision step, SkillDuet uses them to produce successful trajectories and then trains the policy to reproduce the behavior without the additional guidance.

The trajectory conversion step is particularly important here. The ablation in Table 9 shows that removing external guidance during SFT improves performance compared with retaining it (38.3 vs. 31.4 on Hard-ALFWorld valid-seen). This provides useful evidence that the conversion step contributes to autonomous execution.

S2. Meaningful ablations and evaluation on more challenging tasks.

The experiments cover two backbones and both in-distribution and out-of-distribution settings. The introduction of Hard-ALFWorld is also valuable, as its multiple goals and instance-level references require more involved decision-making than standard ALFWorld tasks.

I also appreciate that the authors examine the individual harnesses rather than relying entirely on final benchmark results. In Table 3, the full method performs better than using either harness alone, which supports the motivation for combining the two forms of guidance.

Critical weaknesses

W1. The contribution of skill internalization is not fully isolated from the effect of successful trajectory supervision.

My main concern is whether the performance gains can be attributed specifically to internalizing skill-guided behavior, rather than simply learning from a different distribution of successful trajectories.

In Section 5.2 and Figure 3(a), the authors compare their method against an unassisted SFT variant. However, as the paper itself acknowledges, the two settings may differ in the number of successful SFT trajectories, their lengths, and their effective weights. These differences are especially relevant because assistance is likely to produce more successful trajectories on difficult tasks.

The ablations in Tables 3 and 9 provide evidence that the proposed components are useful, but they do not completely resolve this issue. A comparison that controls the number and token length of successful SFT examples, as well as the effective SFT contribution, would make the internalization claim more convincing.

W2. The evidence for persistent guidance in long-horizon tasks is still somewhat indirect, and the improvements over strong baselines are often modest.

A central motivation of the paper is that start-of-trajectory guidance becomes less effective as interactions become longer. However, the analysis in Figure 2 does not directly measure this phenomenon. It uses reference trajectory length as a proxy for task horizon and reports each group's contribution to the overall success-rate difference rather than its within-group success-rate improvement. Since the number of tasks varies substantially across length groups (Appendix B.1.6), this makes the horizon-dependent results difficult to interpret.

It would also be helpful to know when the environment harness actually provides feedback and whether late-stage interventions account for a substantial portion of the improvement.

Additionally, although SkillDuet achieves the best average performance in most settings, the margins over the strongest baseline are relatively small in several cases. For example, on Hard-ALFWorld valid-unseen with Qwen3-4B, the difference is 55.0 vs. 53.5, and on Search-QA OOD it is 35.9 vs. 35.2. The main tables report variation across evaluation runs, while the additional training-seed analysis is limited to Qwen3-4B on Search-QA. More evidence of training stability would help establish how reliable these improvements are.

Questions for authors

Q1. Could the authors compare SkillDuet with a baseline using the same number of successful SFT trajectories, with comparable trajectory lengths and effective SFT weights, but collected without skill guidance? This would help clarify whether the main benefit comes from skill internalization or from having more successful training demonstrations.

Q2. How frequently does the environment harness provide feedback at different stages of a trajectory? In particular, does feedback delivered in later stages contribute more to task success than early-stage feedback? An early-only versus late-only feedback comparison would help validate the paper's central motivation.

Q3. Could the authors report success-rate improvements within each trajectory-length group in Figure 2, ideally with uncertainty estimates? The current contribution-based metric is difficult to interpret because the groups contain different numbers of tasks.

Q4. How stable are the improvements over the strongest baselines across different training seeds on Hard-ALFWorld? Given the relatively small margins in several settings, this information could materially affect the assessment.

4 comments
AuthorsOpenReview ↗

Response to Reviewer SsVD

Dear Reviewer SsVD,

Thanks for your professional review! We appreciate your recognition of SkillDuet’s trajectory conversion for skill-free inference, supported by meaningful ablations and evaluation on challenging tasks.

[W1&Q1] Isolating skill internalization.

Thanks. (1) It's a fair comparison for SkillDuet and Unassisted SFT (Fig. 3) to use online rejection sampling with equal trajectory sampling budgets. Skill-guided sampling is designed to produce more successes on challenging tasks; the higher success count is therefore expected (clarified in revised Sec. 5.2).

(2) As suggested, we additionally compare offline SFT with policy-token budgets matched within 0.38% (Tab. R1). Sampling 16 trajectories per task per setting across 500 tasks and matching per-task success counts yields 1,432 trajectories per setting from 211 shared tasks. Subsampling converted assisted data to 1,280 trajectories gives 1,857,795 vs. 1,850,732 loss-bearing policy tokens. Under identical SFT settings with guidance removed, converted assisted data perform better despite fewer samples, supporting skill internalization. These results are included in App. E of the revised manuscript.

Table R1: Offline SFT results, Qwen2.5-7B-Instruct on Hard-ALFWorld. Toks. in thousands.

SplitMethodDirectSingleDualTripleQuadToks.TurnsAvg. Succ.
Valid-seenSFT (Unassisted)39.322.617.98.34.8117.543.518.6 ± 0.6
SFT (Converted assisted)41.729.825.07.17.1117.844.022.1 ± 1.7
Valid-unseenSFT (Unassisted)29.617.38.64.90.0128.246.212.2 ± 0.9
SFT (Converted assisted)33.323.512.33.72.6124.145.615.2 ± 0.9

[W2&Q2] Feedback frequency and timing

Thanks. As suggested, we additionally add a feedback-timing analysis to App. F, Fig. 6 of the revised manuscript, covering 12,509 assisted trajectories from 1,681 Hard-ALFWorld training tasks using Qwen2.5-7B-Instruct, each receiving feedback at least once (198K feedback messages).

(1) To examine the environment harness’s feedback frequency across trajectory stages, we divide each trajectory into ten relative-position bins (t/Tt/T) and measure the fraction of steps receiving feedback. Feedback occurs throughout the trajectory; 50.2% of feedback fall in the second half (Fig. 6).

(2) For early-only (t≤T/2t \leq T/2) versus late-only (t>T/2t > T/2) feedback, we select 94 tasks with both types and match trajectory counts per task, yielding 147 per group (Tab. R2). Late-only feedback is associated with higher task success, supporting our motivation for providing guidance throughout interaction.

Table R2: Success rates by feedback timing.

Feedback timingSucc. (%)95% CI
Early-only20.0[12.8, 28.3]
Late-only44.0[33.2, 54.6]
Δ (Late − Early)+23.9[14.0, 34.9]

[W2&Q3] Horizon-dependent gains.

Thanks. As suggested, revised Fig. 2 reports within-group success-rate gains over each baseline with uncertainty estimates, using five approximately equal-count bins of reference trajectory length. SkillDuet generally improves long-horizon performance (≥16 steps), with gains tending to grow with reference length.

[W2&Q4] Training-seed stability.

Thanks. As suggested, we additionally add training seeds 43 and 44 to seed 42 for SkillDuet and the strongest baselines within 2 points (Tabs. R3–R4), and report the mean ± std across training seeds. Toks. are in thousands. We have added these results to App. G of the revision.

Table R3: Qwen3-4B-Instruct-2507 on Hard-ALFWorld, initialized from shared SFT warmup weights.

SplitMethodDirectSingleDualTripleQuadToks.TurnsAvg. Succ.
Valid-seenGRPO75.873.058.340.935.378.036.156.7 ± 1.6
SkillDuet (Ours)77.481.362.744.838.183.137.760.9 ± 0.2
Valid-unseenGRPO73.770.851.952.722.277.636.254.5 ± 0.9
SkillDuet (Ours)75.373.353.953.123.585.338.556.1 ± 0.8

Table R4: Search-QA. Toks. and Turns are ID / OOD.

ModelMethodNQTrivAvg. (ID)PopHotp2WkMuSBamAvg. (OOD)Toks. (ID / OOD)Turns (ID / OOD)
Qwen3-4B-Instruct-2507GRPO36.561.548.9 ± 0.541.540.037.215.241.035.0 ± 0.73.1 / 3.22.8 / 3.1
D2Skill38.759.449.0 ± 0.433.937.531.418.041.532.4 ± 0.61.8 / 2.62.9 / 3.2
Skill037.959.748.8 ± 0.343.542.836.417.534.334.9 ± 0.64.4 / 5.22.6 / 3.1
SkillDuet (Ours)38.262.050.1 ± 0.745.240.735.916.240.035.6 ± 0.52.6 / 3.62.5 / 3.0
Qwen2.5-7B-InstructSkill044.458.751.5 ± 0.746.144.041.122.536.538.0 ± 0.54.0 / 4.72.6 / 3.0
SkillDuet (Ours)46.562.654.5 ± 0.546.245.343.022.042.839.9 ± 1.82.7 / 3.42.9 / 3.5
AuthorsOpenReview ↗

Looking forward to your response

Dear Reviewer SsVD,

We sincerely thank you for your careful review and constructive suggestions, which have greatly helped us improve our paper. We hope our responses and revisions have adequately addressed your concerns. Should you have any further questions or concerns, please do not hesitate to let us know. We would be more than happy to provide further clarification before the discussion period ends.

Thank you again for your time and consideration. Have a wonderful day!

Best regards,

The Authors of SkillDuet

Thank you for the detailed response and the additional experiments. The matched-token SFT comparison addresses my concern about the amount of supervision. The revised Figure 2 and the additional training seeds also strengthen the evidence for the method’s gains.

My main remaining concern is whether feedback delivered later in a trajectory accounts for the dual harness’s advantage on longer tasks. The revised Figure 2 supports the value of the environment harness on longer tasks. The early-only and late-only comparison, however, groups rollouts retrospectively by when feedback was triggered. Even within the same task, the groups may differ in agent progress and in the states that trigger feedback. A comparison that assigns feedback availability to the early or late part of the rollout under a comparable feedback budget would help isolate the effect of timing.

The rebuttal has improved my assessment of the empirical results. Given the importance of persistent guidance to the paper’s motivation, the remaining uncertainty about its role on longer tasks leads me to retain my rating of 2 (Weak rejection).

AuthorsOpenReview ↗

Thanks for your feedback!

Thanks for your feedback!

(1) Since the final trajectory length is unknown before rollout, our earlier analysis grouped completed rollouts by when feedback actually occurred, rather than controlling feedback timing in advance.

(2) To address this, we are testing three settings on Hard-ALFWorld valid-seen, using the Qwen2.5-7B-Instruct GRPO checkpoint at training step 100. All settings retain the start-of-trajectory skill guidance for the same task. We use each task's known reference length TqrefT_q^{\mathrm{ref}} to define feedback windows before rollout, where mq=⌊Tqref/2⌋m_q=\lfloor T_q^{\mathrm{ref}}/2\rfloor and tt is the interaction step. Early-only allows feedback at 1≤t≤mq1\le t\le m_q, late-only at mq<t≤Tqrefm_q<t\le T_q^{\mathrm{ref}}, and no in-trajectory feedback disables it entirely. Both feedback conditions use the same task-specific rules for each task, with no feedback after TqrefT_q^{\mathrm{ref}}.

(3) The success rates are 22.9% (early-only), 26.4% (late-only), and 20.0% (no feedback). These results suggest that both early and late feedback are beneficial, with late feedback providing a larger improvement.

We sincerely appreciate your time and constructive feedback. We hope our clarifications and new experimental evidence have helped resolve your concerns, and would be grateful if you could reconsider your score. Thank you once again, and have a wonderful day! :)

Reviewer_XKLLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

SkillDuet provides retrieved skills through both initial prompts and conditional feedback during execution, then removes this guidance from successful trajectories for weighted SFT alongside RL on unassisted trajectories. Experiments cover HARD-ALFWORLD, Search-QA, and two backbones, with the highest mean aggregate score in seven of eight settings.

Critical strengths
  1. The dual-harness design has direct ablation support. The full method outperforms initial guidance alone, in-trajectory feedback alone, and repeated full-skill text, supporting the value of conditional feedback.
  2. HARD-ALFWORLD introduces instance discrimination, multiple goals, and identity tracking. Reference uniqueness checks and successful trajectory replay provide useful safeguards for task validity.
Critical weaknesses
  1. Positioning against close prior work is weak. Ibrahim et al. (ICLR 2026, arXiv 2510.01375) already run hint-guided rollouts, strip the hints and fine-tune on the result, which is close to conversion plus SFT. Siri and Skill-SD also internalize skills into a skill-free policy, but they only appear in an appendix citation list. SkillC and OPID aren't cited. The new piece seems to be compiling the same skills into rule code and hiding that feedback. The paper should say this plainly and compare against at least one of these methods.
  2. Ablation conclusions rest on one small group. All ablations use Qwen2.5-7B on valid-seen from a single training run. Single accounts for most of each aggregate gap: 4.5 of 5.9 points vs. w/o environment harness, and 5.0 of 4.5 vs. w/o agent harness. That group has only 28 tasks, and on valid-unseen the Single gap mostly disappears (55.6 vs. 51.8 for Skill0).
  3. The core motivation isn't tested directly. The claim that start-of-trajectory guidance fades over long interactions is only backed by citations. Fig. 2 compares the full method with other baselines, not dual vs. single harness. Each bin is divided by all 274 tasks, and about 74% of tasks fall in 17–25 steps, which is exactly where the gains show up. The authors' own sampling-budget explanation fits the pattern equally well.
  4. A key design detail is missing from the method section. Fig. 9 shows the rule feedback tells the policy "Do not explicitly mention, reference, or reveal the rule feedback... as if you had noticed the relevant information on your own." This is what makes converted trajectories look self-generated, but Sec. 4.2 and B.4 describe Ψ as plain deletion. The skill block carries no such instruction, and nobody checks whether converted reasoning still points to removed content.
  5. Many margins are within noise, yet the claims are strong. Std only covers evaluation seeds. In the one training-seed study, seed 43 has OOD 34.9 (below Skill0's 35.2) and seed 42 has ID 49.2 (below D2Skill's 49.5). Several headline gaps are about 1 point, such as 49.2 vs. 48.5 for GRPO and 55.0 vs. 53.5. Even so, the paper says experiments "consistently validate the effectiveness ... across diverse evaluation settings" (L79) and "demonstrate the strong effectiveness" (L368).
  6. Baselines are underspecified. The paper points to Appendix C.3 for implementation details, but only Skill0 and Skill0.5 get them there. EnvHarness on Qwen2.5-7B scores 3.3 and 3.0, close to ReAct, which suggests it had no comparable policy training. Skill0 runs text-only without its compression rewards. With margins this small, these choices matter.
  7. The rule compiler hides engineering cost. The G.4 prompt is about eight pages of hand-written ALFWorld mechanics. There are no numbers on how often a 4B/7B policy produces working code, abstains, crashes or actually fires feedback. The case studies show generic "Nothing happens, check location/inventory" messages, so it's unclear how much task-specific compilation adds over a fixed rule.
Questions for authors
  1. In Table 3, "Triggered full-skill guidance" and "w/o agent harness" have identical Direct/Single/Dual scores (65.5/35.7/39.3). Is this a copy error?
  2. The skill bank starts empty and retrieval needs similarity above 0.7. What fraction of training tasks actually had a non-empty K_q, i.e., how often did the dual harnesses run?
  3. The unnormalized full method (38.3) beats normalized weights (34.1). Since w_q ≤ 1, the full method also has a smaller effective λ_SFT. What is the mean w_q during training? Does uniform weighting at a matched effective λ close the gap?
  4. For the Unassisted SFT ablation, how many successful trajectories and SFT tokens per step does it get compared with the full method?
  5. How large is the final skill bank, and how many skills were added and removed during training?
  6. In Fig. 5(a), does "w/ skill" evaluation use only the agent harness, or rule feedback as well? If rules are used, which model compiles them for the GRPO policy?
8 comments
AuthorsOpenReview ↗

Response to Reviewer XKLL (1/4)

Dear XKLL,

Thanks for your professional review and recognition of SkillDuet’s dual-harness design, supported by ablation studies. Due to space limits, we respond in four comments: (1) W1–W2; (2) W3–W5; (3) W6–W7 and Q1–Q2; and (4) Q3–Q6. We apologize for any inconvenience and appreciate your understanding.

[W1] Weak positioning against close prior work.

Thanks for the feedback. (1) As clarified in our Abstract (p. 1, lines 14–16) and Introduction (p. 2, lines 54–57) of the original manuscript, our work addresses the diminishing influence of start-of-trajectory guidance over long interactions. To this end, we propose an agent–environment dual harness, where the environment harness augments observations with skill-derived feedback to provide in-trajectory guidance.

(2) Our approach differs from these prior methods in how skill guidance is delivered. Ibrahim et al. [1] and SkillC [2] provide skills only once, before the first decision. SIRI [3] places skills after the interaction history at each decision, while Skill0 and Skill0.5 (already compared) inject skills alongside compressed histories at every step. In contrast, our dual harness provides complete skills once at the start of each trajectory via the agent harness, while the environment harness conditionally triggers skill-derived feedback only at relevant states, rather than repeatedly injecting full skills. The effectiveness of this design is validated by the ablation in Tab. 3 of the original manuscript. Skill-SD [4] and OPID [5] use skills as privileged information to construct training signals, rather than to guide trajectory sampling. We have added this discussion to Sec. 2.2 of the revised manuscript.

(3) As suggested, we additionally supplement the results of OPID. SkillDuet outperforms OPID on Hard-ALFWorld (Tab. R1), suggesting that skill-guided sampling may better support internalization in long-horizon tasks. Implementation details and results are provided in App. C.3 of the revised manuscript.

Table R1: Comparison of SkillDuet and OPID using Qwen2.5-7B-Instruct on Hard-ALFWorld. Toks. in thousands.

SplitMethodDirectSingleDualTripleQuadToks.TurnsAvg. Succ.
Valid-seenOPID50.034.520.27.12.4100.039.922.9 ± 1.2
SkillDuet (Ours)64.360.741.717.97.1101.543.038.3 ± 0.9
Valid-unseenOPID38.328.416.04.93.8104.641.018.4 ± 0.7
SkillDuet (Ours)53.155.621.017.33.880.834.330.3 ± 2.1

[W2] Ablation conclusions rest on one small group

Thanks. (1) As suggested, we additionally supplement the dual-harness ablation with Qwen3-4B-Instruct-2507 under three training seeds (42, 43, 44) and evaluate on Hard-ALFWorld. Each trained policy is evaluated with three generation seeds, and we report mean ± standard deviation across the evaluation-averaged training runs in Tab. R2. The gains extend beyond Single: on valid-seen, Dual improves by 7.5 / 10.7 points over the two ablations, respectively; on valid-unseen, Triple improves by 9.1 / 10.7 points and Quad by 9.0 / 10.0 points. We have added these results to App. D of the revised manuscript.

(2) Regarding the large Single gains in the original Qwen2.5-7B-Instruct ablation: Most Single tasks (25/28 valid-seen; 24/27 valid-unseen, Tab. 4) involve co-location, requiring the agent to identify an object based on its initial spatial relation, a constraint that can be forgotten over long interactions. SkillDuet addresses this through feedback at relevant steps, thus outperforming w/o environment harness. However, feedback rules derived from retrieved skills have a finite set of trigger conditions and thus cannot cover every situation requiring guidance, especially when task-level constraints must shape the initial plan. This may also explain the low Single score of w/o agent harness. Thus, in-trajectory guidance complements rather than replaces start-of-trajectory guidance.

(3) Skill0 injects skills alongside compact histories at every step, mitigating guidance decay and explaining the small gap on Single tasks. However, repeated full-skill injection introduces redundancy, while history compression loses reasoning context, resulting in worse overall performance than SkillDuet.

Table R2: Dual-harness ablation on Hard-ALFWorld with Qwen3-4B-Instruct-2507 initialized from shared SFT warmup weights. Toks. in thousands.

SplitMethodDirectSingleDualTripleQuadToks.TurnsAvg. Succ.
Valid-seenSkillDuet (Ours)77.481.362.744.838.183.137.760.9 ± 0.2
w/o environment harness75.470.255.232.130.293.139.252.6 ± 1.9
w/o agent harness75.073.452.031.025.497.440.551.3 ± 2.1
Valid-unseenSkillDuet (Ours)75.373.353.953.123.585.338.556.1 ± 0.8
w/o environment harness74.164.644.444.014.598.640.848.6 ± 0.1
w/o agent harness70.862.142.842.413.5103.042.246.6 ± 0.6
AuthorsOpenReview ↗

Response to Reviewer XKLL (2/4)

[W3] The core motivation lacks empirical validation.

Thanks. (1) To examine whether start-of-trajectory guidance weakens over long interactions, as suggested, we additionally supplement both single-harness variants in Fig. 2 of the revised manuscript. Tasks are grouped into five approximately balanced bins by reference trajectory length, and we report SkillDuet's success gain over each method per bin. SkillDuet outperforms both variants on long-horizon tasks (≥16 steps), with the largest gains in this range over w/o environment harness, which receives guidance only at the start. This suggests that start-of-trajectory guidance may become less effective over longer interactions.

(2) Our case studies (App. F of the original manuscript, and now App. H of the revision) also provide evidence by comparing agent-harness-only (start-of-trajectory guidance only) and our dual harnesses on the same tasks with identical retrieved skills. In cases 1352 and 1990, start-of-trajectory guidance alone fails to correct errors, while in-trajectory feedback enables recovery and task completion.


[W4] Missing design details and validation of guidance removal.

Thanks. (1) Each rule feedback instructs the policy not to explicitly reference it but to incorporate it into its reasoning. We have revised Secs. 4.1–4.2 and Apps. B.3–B.4 to explicitly clarify this design.

(2) We add this instruction only to rule feedback. Rule feedback appears at a specific step in response to the preceding action, so the next reasoning step tends to cite it directly. In contrast, retrieved skills are provided upfront as general strategies and typically integrated into the agent’s plan without explicit references.

(3) We further use DeepSeek-V4.1-Flash to check for references to removed guidance in 1,610 converted trajectories randomly sampled from SkillDuet training with Qwen2.5-7B-Instruct. Each trajectory received non-empty feedback at least once before conversion. These trajectories contain 50,004 decision steps, including 17,047 that originally received non-empty feedback. The check finds no references to skills or rule feedback at any step, consistent with our manual review of 100 sampled trajectories. We have added this check to App. B.4 of the revised manuscript.


[W5] Many performance margins are within noise.

Thanks. (1) We use multiple training seeds to account for training variability and assess stability. We therefore focus on mean performance across seeds (Tab. R4), where our method has higher aggregate means than the strongest compared baselines on both ID (50.1 vs. 49.0 for D2Skill) and OOD (35.6 vs. 34.9 for Skill0).

(2) To further address noise concerns, we additionally supplement multi-seed training (seeds 42, 43, 44) for SkillDuet and the strongest baselines within 2 points of SkillDuet in each setting, and report the mean ± std across training seeds in Tabs. R3 and R4. We have added these results to App. G of the revised manuscript. We also tone down the claims at L79 and L368 of the original manuscript (now p. 2, lines 79–80 and p. 8, lines 420–423 of the revised manuscript).

Table R3: Results of Qwen3-4B-Instruct-2507 on Hard-ALFWorld, initialized from the shared SFT warmup weights and averaged over training seeds 42, 43, and 44. Avg. Succ. reports mean ± standard deviation across training seeds after averaging the three evaluations within each training run. Toks. in thousands.

SplitMethodDirectSingleDualTripleQuadToks.TurnsAvg. Succ.
Valid-seenGRPO75.873.058.340.935.378.036.156.7 ± 1.6
SkillDuet (Ours)77.481.362.744.838.183.137.760.9 ± 0.2
Valid-unseenGRPO73.770.851.952.722.277.636.254.5 ± 0.9
SkillDuet (Ours)75.373.353.953.123.585.338.556.1 ± 0.8

Table R4: Search-QA results averaged over training seeds 42, 43, and 44. Aggregate scores report mean ± standard deviation across training seeds after averaging the three evaluations within each training run. Toks. (in thousands) and Turns are reported as ID / OOD.

ModelMethodNQTrivAvg. (ID)PopHotp2WkMuSBamAvg. (OOD)Toks. (ID / OOD)Turns (ID / OOD)
Qwen3-4B-Instruct-2507GRPO36.561.548.9 ± 0.541.540.037.215.241.035.0 ± 0.73.1 / 3.22.8 / 3.1
D2Skill38.759.449.0 ± 0.433.937.531.418.041.532.4 ± 0.61.8 / 2.62.9 / 3.2
Skill037.959.748.8 ± 0.343.542.836.417.534.334.9 ± 0.64.4 / 5.22.6 / 3.1
SkillDuet (Ours)38.262.050.1 ± 0.745.240.735.916.240.035.6 ± 0.52.6 / 3.62.5 / 3.0
Qwen2.5-7B-InstructSkill044.458.751.5 ± 0.746.144.041.122.536.538.0 ± 0.54.0 / 4.72.6 / 3.0
SkillDuet (Ours)46.562.654.5 ± 0.546.245.343.022.042.839.9 ± 1.82.7 / 3.42.9 / 3.5
AuthorsOpenReview ↗

Response to Reviewer XKLL (3/4)

[W6] Baselines are underspecified and potentially unfairly compared.

Thanks for the feedback. (1) Regarding Skill0, we use the text-only mode supported by its official codebase to match the input modality of all methods. We omit the visual-context compression reward, which is inapplicable to text inputs, while preserving all other components. We have clarified this configuration in App. C.3 of the revised manuscript.

(2) As suggested, we additionally evaluate EnvHarness with GRPO training. As shown in Tab. R5, SkillDuet outperforms EnvHarness on both valid-seen and valid-unseen tasks. In the revised manuscript, we have updated the EnvHarness description (Sec. 5.1) and its GRPO training details (App. C.3), and replaced the EnvHarness results with the GRPO-trained model (Tabs. 1 and 2).

Table R5: Comparison with GRPO-trained EnvHarness using Qwen2.5-7B-Instruct on Hard-ALFWorld. Toks. are reported in thousands.

SplitMethodDirectSingleDualTripleQuadToks.TurnsAvg. Succ.
Valid-seenEnvHarness (GRPO)61.959.540.513.11.2103.340.735.2 ± 0.9
SkillDuet (Ours)64.360.741.717.97.1101.543.038.3 ± 0.9
Valid-unseenEnvHarness (GRPO)48.144.414.813.62.686.036.324.9 ± 0.9
SkillDuet (Ours)53.155.621.017.33.880.834.330.3 ± 2.1

[W7.1] Rule compilation relies on substantial hand-written mechanics, with unclear benefits over fixed rules.

Thanks for the question. (1) Our prompt contains: (i) Environment-agnostic instructions: The rule-generation task, input/output formats, and runtime input schema for generated rules. (ii) Knowledge shared with the standard ALFWorld prompt: Valid actions, single-object holding, placement requirements, simultaneous goal satisfaction, and initial-state references. (iii) Additional environment mechanics: Since rule generation uses the policy itself rather than an external stronger model (e.g., o3 in D2Skill), we specify mechanics to reduce hallucinated rules: accessibility and container-opening requirements, co-located receptacles, device usage for cleaning/heating/cooling without prior placement or manual opening, and property inspection after pickup.

(2) Skills discovered during training contain knowledge beyond these mechanics, including using fixed lamps at their locations rather than picking them up, holding the target object when using a lamp for an original light-examination task, and placing objects on destination surfaces that already contain other objects.

(3) Tab. 3 of the original manuscript includes a Rule generation without skills ablation, which relies solely on the fixed environment knowledge in the rule-generation prompt. Performance drops from 38.3 to 32.1, demonstrating that discovered skills provide additional environment knowledge that improves performance.


[W7.2] Rule compilation and feedback statistics.

Thanks for the question. As suggested, we additionally supplement training dynamics from a full SkillDuet run (Qwen2.5-7B-Instruct, Hard-ALFWorld) in Fig. 7 and Tab. 16 (App. F) of the revised manuscript. On average, 74.3% of compilations yield executable code, 1.3% abstain by design, and 24.4% yield invalid code; among rule invocations, 98.7% complete without runtime errors (Fig. 7(b)). Feedback is emitted at 35.2% of non-terminal assisted steps and in at least one step of 62.4% of assisted trajectories (Fig. 7(c)).


[Q1] Potential copy error in Table 3.

Thanks for the feedback. The matching results are coincidental. The reported results come from three independent evaluations. We provide the results for each generation seed in Tab. R8.

Table R8: Results across three evaluation seeds for Triggered full-skill guidance and w/o agent harness from Tab. 3 of the original manuscript.

MethodEvaluation SeedDirectSingleDualTripleQuadToks.Turns.Avg. Succ.
Triggered full-skill guidance064.332.132.121.410.7122.045.132.1
100067.935.750.07.110.7120.544.334.3
200064.339.335.710.710.7125.044.732.1
Avg.65.535.739.313.110.7122.544.732.8
w/o agent harness060.739.339.321.410.7127.344.234.3
100064.335.739.314.314.3126.744.033.6
200071.432.139.314.310.7127.744.233.6
Avg.65.535.739.316.611.9127.244.133.8

[Q2] Harness coverage.

Thanks for question. Starting from an empty bank, retrieval first reaches full coverage at training step 2. Averaged over training, 99.5% of tasks retrieve a non-empty skill set, with 2.97 retrieved skills per task, and 74.0% of tasks have executable rule code (Fig. 7(a) of the revised manuscript).

Comments on the paper

1 comment

Dear AC,

We sincerely thank the AC for your hard work and all the reviewers for their constructive feedback! SkillDuet addresses the potentially diminishing influence of start-of-trajectory skill guidance over long interactions through agent–environment dual harnesses. The agent harness provides retrieved skills at the start, while the environment harness compiles the same skills into task-specific rules for conditional in-trajectory feedback. PAIR combines task-benefit-weighted SFT on converted skill-free trajectories and RL on unassisted trajectories to internalize skill-guided behavior for autonomous execution.

Below we summarize positive comments and manuscript revisions.

(1) Positive comments

Agent–environment dual harnesses

Dual harnesses instantiate the same retrieved skills as start-of-trajectory guidance and conditional in-trajectory feedback.

  • [XKLL]: "The dual-harness design has direct ablation support. ... supporting the value of conditional feedback."
  • [SsVD]: "... the full method performs better than using either harness alone, which supports the motivation for combining the two forms of guidance."

Skill internalization

PAIR internalizes skill-guided behavior for autonomous execution without external guidance.

  • [SsVD]: "I find the separation between training-time assistance and skill-free inference to be one of the more appealing aspects of the paper."
  • [jekY]: "... the proposed method can better internalize the skill bank."

Hard-ALFWorld: a more challenging benchmark

We introduce Hard-ALFWorld with multiple goals, same-type distractors, and scene-grounded object references for challenging long-horizon tasks.

  • [XKLL]: "Reference uniqueness checks and successful trajectory replay provide useful safeguards for task validity."
  • [SsVD]: "The introduction of Hard-ALFWorld is also valuable ..."
  • [jekY, follow-up]: "Experiments on the original ALFWorld benchmark justify the necessity of the new benchmark."

Empirical evaluation and analysis

We evaluate two backbones on Hard-ALFWorld and Search-QA in ID/OOD settings.

  • [SsVD]: "Meaningful ablations and evaluation on more challenging tasks."
  • [jekY]: "The proposed method is thoroughly evaluated in two domains."

(2) Revisions of the manuscript

Main Paper

  • [XKLL]: Sec. 2.2. Expanded related-work comparisons to clarify SkillDuet's contribution.
  • [XKLL, jekY]: Secs. 3.1 and 4.1–4.3. Clarified definitions, feedback instructions, and conversion, and summarized skill-bank maintenance from App. B.5 in the main text.
  • [XKLL]: Sec. 5.1; Tabs. 1–2. Added GRPO training to the EnvHarness baseline to improve comparability.
  • [XKLL, SsVD]: Sec. 5.2; Fig. 2. Added within-bin gains and single-harness variants to make horizon-dependent results easier to interpret.
  • [XKLL, SsVD]: Secs. 5.2–5.3; Figs. 3(a) and 5(a). Clarified Unassisted SFT rollout budgets and the “w/ skill” evaluation setting.

Appendix

  • [jekY]: App. B.1.7; Tab. 8. Added original ALFWorld evaluation to assess the need for a more challenging benchmark.
  • [XKLL]: App. B.4. Audited converted trajectories to check for explicit references to removed guidance.
  • [jekY]: App. B.5.3; Tab. 9. Added a fixed-skill-bank control to assess the contribution of online maintenance.
  • [XKLL]: App. C.3. Expanded baseline implementation details to clarify comparison settings.
  • [XKLL]: App. C.3; Tab. 11. Added OPID baseline to strengthen empirical positioning.
  • [XKLL]: App. D; Tab. 12. Expanded harness ablations to assess generality beyond the original setting.
  • [XKLL]: Apps. E and F; Tab. 14 and Fig. 7(d). Added a matched uniform-weight control to test whether gains arise from a lower effective SFT coefficient.
  • [XKLL, SsVD]: Apps. E and F; Tab. 15 and Fig. 7(e). Added supervision statistics and a matched-token SFT control to examine the effect of supervision quantity.
  • [XKLL, jekY]: App. F; Fig. 7(a), (b), and (f); Tab. 16. Added training-dynamics statistics to clarify harness coverage, generated-code reliability, and skill-bank evolution.
  • [XKLL, jekY]: App. F (Feedback quality). Added LLM-based and manual feedback audits to assess consistency with source skills and relevance to the decision state.
  • [XKLL, SsVD]: App. F; Figs. 6 and 7(c). Analyzed feedback frequency and added observational timing and agent-only comparisons to examine associations with task success.
  • [XKLL, SsVD]: App. G; Tabs. 17–18. Added multi-seed comparisons against the strongest baselines with small original gaps to SkillDuet to assess stability.
  • [XKLL]: App. H; Fig. 8 and Tab. 19. Added fixed-rule comparisons to assess the full framework's benefit.

We have made every effort to address all reviewers' concerns, and sincerely hope our responses have resolved them. We sincerely thank the AC for your time and dedication.

Best regards,

Authors of SkillDuet

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.