acceptodds
Under review as a conference paper at ICLR 2027

SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents

Abstract

Agent frameworks increasingly package procedural knowledge as skills: instruction files an agent reads on demand, while public libraries now hold thousands of them. Which skill to read has thus become a decision the policy itself makes in the middle of an episode, yet no existing signal trains it. We show that the default remedy, outcome-rewarded RL over the candidate slate, cannot teach it, for a structural reason we identify and name selector credit starvation: under a broadcast, sequence-level advantage, the few tokens that name the chosen skill carry a vanishing share of the loss, and the credit they inherit is increasingly wrong-signed as trajectories lengthen. A correct choice is punished whenever the execution after it fails, even though the choice itself is among the most valuable decisions in the trajectory. Auditing a completed run's own training artifacts confirms all three properties, each worsening monotonically with horizon. SkillGate removes the failure by construction: it partitions the token support into two disjoint credit channels, outcome credit reaching only execution tokens, and a separate action-local advantage reaching exactly the skill-naming tokens, positive only when a trajectory's single read is the correct one. On five agentic benchmarks under a 16-candidate slate, SkillGate lifts a 9B policy from 40.8% to 53.2% trial success, well ahead of the identical budget spent on outcome reward alone, while cutting exposure to misleading candidates by two thirds and reading fewer skills.

Primary area:
reinforcement learning
Keywords:
large language model agents, skill selection, reinforcement learning, credit assignment, long-horizon tasks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wnsvOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SkillGate, which trains in-policy skill selection through separate credit assignment for skill-selection and execution tokens with length-independent weighting, and evaluates its effectiveness on five agent benchmarks.

Critical strengths
  • The paper identifies weak and potentially conflicting learning signals for skill-selection tokens in long agent trajectories. Its trajectory analysis provides concrete motivation for treating selection and execution separately.

  • SkillGate separates selection and execution credit within a single policy and improves aggregate task success over outcome-only RL across five benchmarks.

Critical weaknesses
  • Table 3 pairs external selectors with a frozen SFT executor, whereas SkillGate benefits from RL training of both selection and execution. This confounds the selection architecture with executor training. Despite discussing related approaches such as daVinci-kernel and ToolOmni, the paper lacks controlled comparisons with their joint-training or credit-separation strategies. Consequently, the evidence is insufficient to establish the proposed design’s advantage over relevant alternatives.

  • SkillGate assigns zero selection utility to every read in a trajectory containing multiple reads, including a correct reselection after an initial mistake. This conflates unnecessary reading with potentially useful correction. Although the constraint helps in Table 2, removing it yields higher success in Table 5 (Action credit: 65.3%; SkillGate: 61.2%). Without recovery-specific analysis or repeated training runs, the paper does not establish when this restriction improves task performance or whether the reported differences are robust.

Questions for authors
  1. Have SkillGate and the main baselines been independently trained with multiple training seeds? If so, please report the mean and variability of final performance, along with the consistency of the rankings across methods. If not, what additional evidence helps distinguish a reproducible benefit of the proposed design from differences specific to individual training runs, and how does this limitation constrain the paper’s conclusions?

  2. Among the cited related work, daVinci-kernel jointly trains skill selection and execution using execution outcomes, while ToolOmni separates the rewards and advantages for retrieval and execution. Thus, explicitly training selection or separating selection and execution learning signals has already been explored. What additional benefits do SkillGate’s identity-token-level credit assignment and length-independent weighting provide over these approaches? In particular, are there comparative results demonstrating their advantages over existing credit-separation strategies, independently of the single-read utility? If the current ablations establish this distinction, please identify which comparisons do so.

  3. In Table 3, external selectors are paired with a frozen SFT executor, whereas SkillGate receives additional RL training for both selection and execution. It is therefore difficult to determine whether the performance difference reflects an advantage of in-policy selection or the effect of additional executor training. Do you have comparisons pairing an external selector with an RL-trained executor under matched training budgets and supervision? Have you also evaluated a baseline trained with oracle skills provided during training, using RL solely to learn execution? Without these controls, what supports interpreting Table 3 as evidence of limitations in separate-selector approaches, beyond the benefit of additional executor training?

  4. What criterion motivated adopting the single-read constraint, given that Action credit achieves higher success than SkillGate on the held-out Claw-Eval subset (65.3% vs. 61.2%)? Have you analyzed the existing trajectories to distinguish unnecessary additional reads from reselection that successfully corrects an initially wrong skill choice?

9 comments
AuthorsOpenReview ↗

Author Response (1/5)

Thank you for dedicating your time and effort to our work.

1

Table 3 pairs external selectors with a frozen SFT executor while SkillGate is RL-trained, so selection architecture is confounded with executor training. Controlled comparisons with the credit-separation strategies of daVinci-kernel and ToolOmni are also missing.

To address your concerns: (1) with new-added the executor matched experiment, the conclusion is unchanged. External routers paired with the SkillGate executor remain largely below SkillGate. (2) The paper's ablations already compare, in the same setting, the credit strategies that daVinci-kernel and ToolOmni represent.

(1) Matched-executor analysis. We compare two executors on eval70: the frozen SFT model of Table 3 and the SkillGate checkpoint (identical RL training). The Oracle-only rows hand the oracle directly to the executor, which is equivalent to a perfect router. The first eight rows form four pairs: the same source of selection, with the SFT executor and the SkillGate executor comparison.

Configuration (eval70, 280 trials)SelectionExecutorSuccess
SFT-9B router → SFTExternalSFT40.7
SFT-9B router → SkillGate executorExternalSkillGate43.6
Qwen3.5-27B router → SFTExternalSFT36.8
Qwen3.5-27B router → SkillGate executorExternalSkillGate43.2
Reranker top-1 → SFTExternalSFT31.8
Reranker top-1 → SkillGate executorExternalSkillGate38.2
Oracle-only → SFT (perfect-router ceiling)GivenSFT48.2
Oracle-only → SkillGate (perfect-router ceiling)GivenSkillGate52.9
SkillGate (in-policy)Policy's ownSkillGate50.0
  • External routers stay below SkillGate even with the SkillGate executor, which reflects the selection ability gained through SkillGate training.
  • SkillGate is close to the perfect-router ceiling: selecting by itself among 16 candidates, it reaches 94.6% of that ceiling.
  • Executor training contributes (the two Oracle-only rows).

The RL-trained controls without identity-token credit (Outcome-only, Task-mask only, Skill-free RL) receive the same executor training as SkillGate and stay at 46.0%–47.0% on eval385, against 53.2% for SkillGate.

(2) Controlled comparison with the credit strategies of daVinci-kernel and ToolOmni. The two systems cannot be applied as-is to read calls inside the execution trajectory, so we compare the credit designs they represent, under the same initialisation, data and budget:

Credit strategyRepresentative methodCounterpart in the papereval385 successeval70 oracle readeval70 misleading read
Selection signal from the task outcome, joint trainingdaVinci-kernelOutcome-only47.054.369.6
Skill1 (joint selection-and-use)44.753.545.5
Separate selection signal, placed on the trajectory or groupToolOmniTrajectory bonus48.647.955.4
Group-level regret46.533.647.5
Separate signal on identity tokens, N/MN/M weightOursSkillGate-MR (Action credit in the paper)51.980.032.5
SkillGate53.283.921.8

Both existing strategies stay within 44.7%–48.6% in our setting, while identity-token credit reaches 51.9%–53.2%; SkillGate-MR does not use single-read, so the gap is unrelated to it. Neither method performs in-policy selection: they select and execute in two separate stages, so they do not encounter selector credit starvation.

Revision. We will add the matched-executor analysis to Sec. 4.4 and mark in Sec. 4.3 how each ablation row corresponds to an existing credit strategy.

AuthorsOpenReview ↗

Author Response (2/5)

2 (Weakness 2 and Question 4)

Single-read gives zero utility to every read in a multi-read trajectory, including a correct reselection after a wrong first read, and removing it gives higher success in Table 5 (Action credit 65.3% vs SkillGate 61.2%). What criterion motivated single-read, and can a recovery analysis separate unnecessary extra reads from useful correction?

Action credit in the paper (the version that allows multiple reads) is a variant of SkillGate. Our naming may have caused a misunderstanding, so we rename it SkillGate-MR. To address your concern about the single-read constraint, we ran a temporally ordered recovery analysis on the existing trajectories: single-read does not suppress correction after a wrong first read; it reduces wrong first reads and extra reads after the oracle, both of which lower task success. The gap on Claw147 is not significant and does not come from recovery. Overall, the two versions trade wins on task success, while SkillGate selects more cleanly and costs less at inference.

(1) SkillGate-MR (Action credit in the paper) is a variant of SkillGate. It applies the same credit-separation method and only drops the single-read requirement when assigning selection credit:

Identity-token creditRead call removed from the outcome lossN/MN/M weightWhich selection receives credit
SkillGateYesYesYesThe only read in the trajectory, when it reads the oracle
SkillGate-MR (Action credit in the paper)YesYesYesThe first read that reaches the oracle; other reads are unrestricted

SkillGate-MR is therefore an experimental variant, not a different method, and its strength is itself evidence that SkillGate's core method works. We adopt SkillGate as the default because it reads more cleanly at lower cost.

(2) Criteria for adopting single-read.

  1. Credit cannot be earned by reading more. If each read is scored separately, a policy can collect selection credit simply by reading several skills.
  2. Cost and consistency. Without the constraint the model reads more skills and uses more tool calls and tokens, without a consistent gain in success:
MetricSkillGateSkillGate-MR
Clean oracle read, eval70 / Claw14775.4 / 76.964.6 / 73.5
Misleading read, eval70 / Claw14721.8 / 15.032.5 / 20.4
Turns / tool calls / input tokens per trial, eval38517.3 / 8.8 / 521k18.3 / 10.4 / 555k

(3) Recovery analysis: unnecessary extra reads vs. successful correction. In the 280 eval70 trajectories of each model, skill reads are labelled in temporal order: whether the first read is wrong, whether a wrong first read is later followed by the oracle (correction), and whether other skills are read after the oracle (extra reads).

First read wrongCorrection rateExtra reads after the oracleExactly one read, of the oracle
SkillGate52 (18.6%)23.1%14 (5.0%)183 (65.4%)
SkillGate-MR73 (26.1%)23.3%33 (11.8%)161 (57.5%)
Task-mask only (no selection credit)180 (64.3%)22.2%60 (21.4%)48 (17.1%)
  • Correction is not suppressed in SkillGate. After a wrong first read, the three models reach the oracle at the same rate, although only SkillGate-MR gives credit to a corrective read.
  • Single-read reduces the other two behaviours: wrong first reads by 7.5 pp [−13.9, −1.4] and extra reads after the oracle by 6.8 pp [−12.1, −2.1].

(4) Effect of these behaviours on task success. Within each prompt group of the training rollouts, we compare the success of trajectories with different reading patterns:

ComparisonSkillGate training rolloutsOutcome-only training rollouts
Extra reads after the oracle vs. oracle only−4.9 pp [−9.0, −0.8]−5.2 pp [−10.7, +0.1]
Wrong and uncorrected vs. oracle only−19.9 pp [−28.6, −11.2]−19.0 pp [−25.5, −12.6]
Wrong then oracle vs. wrong and uncorrected+12.9 pp [+2.1, +23.9]+8.4 pp [+1.9, +14.9]

This answers when the constraint helps: extra reads and wrong first reads both lower success, and these are the two behaviours single-read targets. Correction is beneficial, and the single-read setting does not reduce it.

(5) The Claw147 gap does not come from recovery. The 147 trajectories of each model, sorted by the order of their reads:

Claw147 (147 tasks, 1 trial each)Oracle onlyOracle, then othersWrong, then oracleRead, never the oracleNo readpass@1
SkillGate11322201090 (61.2%)
SkillGate-MR1086027696 (65.3%)
  • SkillGate-MR has no wrong-then-oracle trajectory on Claw147, so its additional successes cannot come from correction.

Revision. We will add the recovery analysis to Appendix C.1, and rename Action credit to SkillGate-MR throughout the paper.

AuthorsOpenReview ↗

Author Response (3/5)

3

Were SkillGate and the main baselines trained with multiple seeds? If not, what evidence separates a reproducible benefit of the design from run-specific differences, and how does this limit the conclusions?

Response:

Each configuration is trained once: one run takes 100 steps of on-policy GRPO on 16 H800 GPUs and collects 12,800 long-horizon trajectories online, so repeating all eight RL configurations over multiple seeds is prohibitively expensive. Evaluation-side randomness is controlled by four trials per task, SkillGate is best or tied for best among the 9B methods on all five benchmarks, and three further pieces of evidence show that the gain comes from the credit design rather than from the chance of a single run.

(1) Reproducibility.

All configurations share the same SFT checkpoint, the same 491 training tasks and the same 100-step budget (Sec. 4.1, Appendix A). The three pieces of evidence are:

#EvidenceResult
1Independent runs split consistentlyTwo runs with identity-token selection credit (SkillGate, SkillGate-MR): 53.2% and 51.9% on eval385. Six runs without it (Outcome-only, Task-mask only, Skill-free RL, Skill1, Group-level regret, Trajectory bonus): 44.7%–48.6%
2Large effect sizes in selection behaviour, which transfer to unseen tasks and unseen skillsvs Outcome-only on eval70: clean oracle read +53.9 pp [+44.3, +63.2], misleading read −47.9 pp [−57.5, −37.9]. On Claw147: clean oracle read 76.9% vs 34.7% (+42.2 pp [+33.3, +51.0]), misleading read 15.0% vs 53.7% (−38.8 pp [−47.6, −29.9])
3Training curves show a sustained trendSkillGate's clean oracle read rises 27.8 → 55.9 → 69.1 → 78.2 → 76.7%; under Outcome-only it falls from 13.9% to 9.1%

Evidence 1 is the most direct: the eight RL runs are independent and fall into two non-overlapping groups according to whether identity-token selection credit is present. The effect is thus reproduced by two independent runs, and the six runs without it give a reference for run-to-run variation. Differences of 40–58 pp in selection behaviour, which persist on a benchmark never seen in training, are not explained by such variation.

(2) Mean, variability and ranking consistency. With one run per configuration there is no cross-seed mean or variance to report. The ranking is consistent: the two runs with identity-token selection credit rank above all six runs without it, and the two groups do not overlap.

(3) Scope of the conclusions. The paper's two conclusions rest on large effects: identity-token selection credit changes selection behaviour by 40–58 pp, and task success improves over Outcome-only by 6.2 pp on eval385 and by 7.9 pp on eval70 (p=0.020).

Reviewer_v5VHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses credit assignment for skill selection in long-horizon agents. SkillGate separates training into two token-level channels: task-outcome credit for execution and oracle-supervised credit for skill selection. Experiments on five benchmarks show improved task success and more accurate selection with fewer skill reads.

Critical strengths
  1. Separating selection from execution credit is reasonable and well motivated.
  2. The experiments are thorough, covering multiple benchmarks, baselines, and ablations.
  3. The results demonstrate that SkillGate improves task success from 47.0% to 53.2% over outcome-only RL, substantially reduces misleading-skill exposure, and reads fewer skills.
Critical weaknesses
  1. The related work section appears near the end of the paper and is brief, difficult to follow, and insufficiently organized. Its compressed discussion does not clearly explain the limitations of existing methods, how the proposed method differs from and improves upon them, or the significance of the contribution.
  2. The paper proposes reinforcement learning for selecting a single skill. The evaluation does not address multiple complementary skills, no applicable skill, or changes in candidate construction. ·
  3. This work is a preliminary exploration of post-training for skill selection, but its generalizability to diverse skill-selection settings, reward signal design, and skill construction remains unclear.
Questions for authors
  1. I think this paper is not the first to argue that “in-policy skill selection should be optimised on its own.” What specific advantage does SkillGate offer over the closest methods with separate selection objectives?
  2. Can the method handle tasks requiring cooperation among multiple useful skills, or cases where no suitable skill is available? Or are these extensions left for future work?
  3. Could the authors automate skill-set construction and evaluate whether SkillGate generalizes to automatically constructed skill sets, rather than relying on manually designed oracle skills and synthetic hard negatives?
5 comments
AuthorsOpenReview ↗

Author Response (1/4)

Thank you for dedicating your time and effort to our work.

1

Related work comes late in the paper and is brief and loosely organised. It does not make clear the limitations of existing methods, how SkillGate differs from them, or why the contribution matters.

To address this concern, the revision moves Related Work to Sec. 2, rewrites it along 2 threads, and adds content stating what existing methods do, what they leave open and where SkillGate differs.

(1) The new organisation.

  1. Selecting skills or tools from a large library (retrievers, rerankers and routers such as Skill1, AutoTool and SelSkill). A separate module is optimised for selection accuracy, which does not guarantee task success: in Table 3 the 27B router selects the oracle more often than the 9B router (68.6% vs 60.0%) and yet succeeds less (36.8% vs 40.7%).
  2. Joint training of selection and execution (daVinci-kernel, ToolOmni). Selection has its own reward or advantage, but it is a separate stage before execution, and its signal comes from downstream execution results or is averaged over the response length.
  3. Credit assignment for long-horizon LLM agents (local and segment-level credit such as IG-Search and CARL). SkillGate applies this idea to skill reads inside the execution trajectory and handles the two problems specific to them: outcome noise and dilution by trajectory length.

(2) The contribution, stated more explicitly.

  • Diagnosis. Under outcome-only RL, skill-name tokens take only 0.14% of the loss weight and 39.3% of oracle reads receive a negative advantage, although reading the oracle raises success by 11.2 pp.
  • Method. A minimal change that targets this diagnosis: two channels that share no token, one forward pass and one update, with no extra selector model or rollouts.
  • Result. Under the same initialisation and budget, eval385 success rises from 47.0% to 53.2% and the misleading read falls from 69.6% to 21.8%.
AuthorsOpenReview ↗

Author Response (2/4)

2 (Weakness 2 and Question 2)

The evaluation does not cover multiple complementary skills, the absence of any applicable skill, or changes in candidate construction. Can the method handle these cases, or are they left for future work?

Response:

Single-oracle, single-read is the cleanest setting in which to isolate, measure and fix selector credit starvation, and we chose it deliberately. SkillGate's two channels do not depend on it: extending to multiple skills only requires replacing the selection utility u(a)u(a), and "no applicable skill" is handled in the same way. Changes in candidate construction already have direct experimental results.

(1) Why we study this setting first.

  • It matches the most common use of a skill library: reading the single most suitable guide on demand during execution.
  • With a unique oracle, a selection can be judged independently of the execution outcome, which all of the paper's starvation measurements rely on.
  • Even this setting is far from solved: the SFT model reads a misleading skill on 61.8% of trials (Table 3), outcome-only RL makes this worse during training, and none of the six frontier models in Table 1 reads the oracle on half of the trials.

(2) How the framework extends. Whether a task has a single oracle skill, several valid skills or a combination of complementary skills, it meets the same core problem, selector credit starvation, and the same solution applies: only the computation of the utility changes. SkillGate has three parts: (i) read calls are removed from the outcome loss; (ii) each selection has a fixed total weight N/MN/M, independent of trajectory length; (iii) the selection utility u(a)u(a). Only (iii) uses the single-oracle, single-read assumption.

SettingSelection utilityStatus
Single oracle, single read (paper default)u(a)=1u(a)=1 iff aa is the only read in the trajectory and it reads s⋆s^\starTrained and evaluated
Multiple reads allowedThe selection that first reads the oracle receives credit; other reads are unrestricted (SkillGate-MR, Action credit in the paper)Trained and evaluated: eval385 51.9%, oracle read 80.0%, misleading read 32.5%
Multiple valid skills (any one suffices)Replace s⋆s^\star with a valid set VxV_x; reading any member earns creditDrop-in replacement
Multiple complementary skillsu(a)=1u(a)=1 iff the set of skills read in the trajectory satisfies S(τa)∈FxS(\tau_a)\in\mathcal{F}_x, where Fx\mathcal{F}_x is the family of minimal sufficient sets for task xxReduces to the paper's definition when Fx={{s⋆}}\mathcal{F}_x=\{\{s^\star\}\}
Step-wise marginal creditu(aj)=V(Sj−1∪{sj})−V(Sj−1)u(a_j)=V(S_{j-1}\cup\{s_j\})-V(S_{j-1}), with VV a coverage functionAlso gives positive credit to a corrective read after a wrong first read
No applicable skillAbstention (not reading) becomes an explicit selection with an identity span that can carry credit; its utility is 1 when the slate contains no valid skillMasking and weighting unchanged

SkillGate-MR shows that identity-token credit remains effective once the single-read constraint is lifted. This is the first step towards multi-skill settings, and it is already implemented; a full evaluation of multi-skill cooperation and abstention is our next step.

For "no applicable skill", the agent prompt already says "If none applies, do not read a skill", so not reading is part of the action space. The selector channel does not reward the act of reading itself (its advantages sum to zero within a group, Sec. 3.1), and after training SkillGate reads fewer skills, not more: 1.11 per trial, against 1.88 for Outcome-only.

(3) Changes in candidate construction: direct results. The same SkillGate checkpoint is compared with Outcome-only under four changes in how the candidates are constructed (clean oracle read: the oracle is the only skill read in the trajectory):

Change in the candidatesSkillGateOutcome-only
Unseen tasks + unseen skills (eval70; training and evaluation oracles are disjoint)Clean oracle read 75.4%, misleading read 21.8%21.4%, 69.6%
Held-out benchmark (Claw147; no Claw-Eval task in the training set)Clean oracle read 76.9%, misleading read 15.0%34.7%, 53.7%
Slates with regenerated misleading descriptions (same 70 tasks)Success 51.1%, clean oracle read 83.9%, misleading read 13.9%45.4%, 23.6%, 66.8%
Description-length cue reversed (15 tasks in which a misleading description is shorter than the oracle's, Appendix C.4)Oracle read 80.0%, clean oracle read 63.3%46.7%, 18.3%

Under all four changes SkillGate's clean oracle read is more than 40 pp above Outcome-only. It learns to judge from the task content whether a skill fits, not to memorise names or exploit surface features of the descriptions.

Revision. We will add a "Scope and extensions" paragraph to Sec. 3 with this scope and the extensions in the table of part (2).

AuthorsOpenReview ↗

Author Response (3/4)

3

This is a preliminary exploration, and its generalisability across skill-selection settings, reward-signal designs and skill construction remains unclear.

The paper addresses selector credit starvation, which arises whether a task has a single oracle skill or needs a combination of skills. All three dimensions you raise are covered by controlled experiments in the paper, and they agree: once the selection signal lands on the identity tokens and is independent of trajectory length, the gain is stable.

(1) Skill-selection settings.

  • Five agentic benchmarks from different sources: SkillGate is best or tied for best among the 9B methods on every one (Table 1).
  • Training and evaluation tasks are disjoint, and so are their oracle skills.
  • Claw-Eval is entirely absent from training: on its 147 held-out tasks the clean oracle read is 76.9%, against 34.7% for Outcome-only.

(2) Reward-signal design. The paper compares eight designs of the selection signal under the same initialisation and budget:

Design of the selection signalMethodeval385 successeval70 oracle readeval70 misleading read
None (task outcome only)Outcome-only47.054.369.6
None, with outcome credit removed from read callsTask-mask only46.048.973.6
Supervised: teacher forcingSelection BC44.771.431.4
Supervised: preferenceSelSkill-DPO46.266.151.8
Oracle signal on the prompt groupGroup-level regret46.533.647.5
Oracle signal on the whole trajectoryTrajectory bonus48.647.955.4
Oracle signal on identity tokens (any number of reads)SkillGate-MR51.980.032.5
Oracle signal on identity tokens (single-read)SkillGate53.283.921.8

The six designs without identity-token credit stay within 44.7%–48.6%, while the two with it reach 51.9% and 53.2%. What decides the result is where the signal lands, not one particular definition of the utility.

(3) Skill construction. The slates are built by an automated pipeline, and SkillGate's selection advantage holds with regenerated misleading descriptions, a reversed description-length cue and a held-out benchmark.

4

The paper is not the first to argue that in-policy skill selection should be optimised on its own. What specific advantage does SkillGate offer over the closest methods with separate selection objectives?

Response:

Prior work such as daVinci-kernel and ToolOmni does train selection separately, but not as in-policy selection of skill reads inside the execution trajectory, which is the case SkillGate addresses.

(1) Three concrete differences.

Existing separate selection objectivesSkillGateBenefit
Signal sourcedaVinci-kernel: leave-one-out over downstream execution scores, still outcome-derivedOutcome-independent oracle utilityThe sign is unaffected by execution noise. On our training batches the outcome signal pushes down 35.2% of oracle selections; with SkillGate the share is 0%
Where credit landsThe whole selection response, or all tokens of a stageSkill-identity tokens only; the read call is removed from the outcome lossThe signal acts directly on which skill is chosen and is not shared with other tokens
WeightVaries with response length, or is averaged over itFixed at N/MN/M for every selectionIndependent of trajectory length and skill-name length, so long-horizon tasks are not diluted

SkillGate also needs no separate stage: daVinci-kernel requires extra rollouts for each selection plan and ToolOmni updates its two stages in sequence, whereas SkillGate updates both channels in one forward pass.

(2) Controlled comparison in the same setting. The two systems cannot be applied as-is to read calls inside the execution trajectory, so we compare the credit designs they represent, under the same initialisation, data and budget:

Credit designCounterpart in the papereval385 successeval70 oracle readeval70 misleading read
Selection signal from the task outcomeOutcome-only47.054.369.6
Skill1 (joint selection-and-use)44.753.545.5
Separate selection signal, placed on the trajectory or groupTrajectory bonus48.647.955.4
Group-level regret46.533.647.5
Label-based supervised selectionSelection BC44.771.431.4
SelSkill-DPO46.266.151.8
Separate selection signal on identity tokens, N/MN/M weightSkillGate-MR51.980.032.5
SkillGate53.283.921.8

With the same oracle information, moving the signal from the trajectory level to the identity tokens sharply improves reading behaviour and raises success rate.

Revision. We will state the contribution in this more specific form in the Introduction and Related Work.

Reviewer_y1NKOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses an important credit-assignment challenge in skill-augmented language-model agents: learning which skill to select while also learning to execute the task. It proposes SkillGate, a simple approach that separates outcome-based execution credit from oracle-based selection credit at the token level, with normalization that prevents skill-selection signals from being diluted by long trajectories. Experiments across five benchmarks show promising improvements over matched outcome-only RL and supervised selection baselines, supported by informative credit-placement ablations. The work offers a practical contribution to improving in-policy skill selection.

Critical strengths
  1. A clear and practically relevant problem. The distinction between choosing an appropriate procedural guide and successfully executing it is meaningful. The trajectory audit motivates why these decisions may benefit from different training signals.

  2. A simple solution aligned with the problem. Token-level masking separates selection from execution credit, while action-balanced weighting addresses the small number of skill-identity tokens. The method integrates into the existing policy update without requiring a separate selection model.

  3. Promising results with useful controlled comparisons. SkillGate improves overall trial success from 47.0% for outcome-only SkillRL to 53.2%, using the same initialization and training budget. Comparisons with selection BC, DPO, task-mask-only training, and alternative credit placements strengthen the evidence. Disjoint training and evaluation skill identities also provide useful evidence of generalization.

Critical weaknesses
  1. The explanation of selector credit starvation needs stronger validation. Negative outcome advantages on correct selections do not alone establish a structural failure of policy gradients. Gradient measurements and a reweighting-only control would help clarify the mechanism.

  2. The selection objective assumes one oracle skill and a single read. This limits its applicability to tasks requiring multiple valid or complementary skills. The scope and possible extensions should be discussed more explicitly.

  3. The results lack independent training-seed replication. Task-level bootstrap intervals are useful, but additional runs would strengthen confidence in training stability. External-router comparisons would also benefit from a matched executor.

Questions for authors
  1. Can the authors separate the effects of selector-token reweighting, oracle-based local credit, and removal of outcome credit from read calls? Do gradient measurements support the proposed starvation mechanism?

  2. How are oracle skills validated, and how would the method handle multiple valid skills or tasks requiring several complementary skill reads?

  3. Are the main gains reproducible across independent training seeds? Can external routing be evaluated with an executor matched to SkillGate’s execution capability?

6 comments
AuthorsOpenReview ↗

Author Response (1/3)

Thank you for dedicating your time and effort to our work.

1 (Weakness 1 and Question 1)

The starvation mechanism needs stronger validation. Can the effects of selector-token reweighting, oracle-based local credit and removal of outcome credit from read calls be separated, with gradient measurements and a reweighting-only control?

To address your concerns, we ran two analyses, the gradient measurements and the reweighting-only control, and extended them into a component-wise decomposition.

(1) Gradient measurements on real training batches. On all on-policy batches of the outcome-only run (12,800 trajectories, 15,102 skill selections), we computed the gradient coefficient each selection actually receives (advantage × token weight) under six credit configurations:

ConfigurationOracle selections pushed downMisleading selections reinforcedWrong-signed gradient massWithin-group SNRStep-level SNRSelection / task gradient mass
(a) Outcome-only35.2%49.4%46.1%0.301.090.0009
(b) Reweighting only35.2%49.4%46.3%0.290.980.195
(c) Local credit + weight, outcome credit kept on read calls2.1%28.2%0.3%0.963.430.069
(d) As (c), with the residual outcome credit also reweighted9.0%45.8%29.7%0.863.010.221
(e) Local credit + outcome credit removed, no weight0%0%0%1.772.31<0.0001
(f) SkillGate (all three)0%0%0%1.643.430.069

Columns 1–3 give the share of clean oracle selections with a negative coefficient, the share of misleading selections with a positive one, and the share of coefficient mass these two carry. SNR is that of the gradient projected onto the oracle logit. The last column is the coefficient mass on selection tokens relative to execution tokens.

  • Outcome-only credit is diluted and wrong-signed, (a). Selection receives 0.09% of the gradient mass, and nearly half of it points the wrong way. This is structural: the outcome advantage is one constant per trajectory, so a failed execution after a correct read penalises the read, and the share of oracle selections pushed down grows with length, from 29.3% (≤3k tokens) to 62.2% (>16k tokens).
  • Reweighting repairs only the magnitude, (b). It amplifies the selection gradient by about 215×, yet both sign metrics are unchanged.
  • Each SkillGate component fixes one problem. Local credit corrects the sign, (b)→(c); removing outcome credit from read calls clears the residual wrong signal, (c)→(f); the length-independent weight makes the correct signal large enough, (e)→(f).

We do not claim that policy gradients stop working altogether in this setting. Our claim is that this large share of wrong-signed signal substantially hinders the policy from learning to read the correct skill, which the results below confirm.

(2) How these differences turn into learning outcomes.

Simulation. A selector simulation calibrated on the audit statistics (softmax selector, no LLM, 20 seeds, the same coefficient λ=0.2\lambda=0.2 for all designs):

Credit designP(oracle), step 10P(oracle), step 100Step 100, long trajectories only (>16k tokens)
Outcome-only37.4 ± 0.938.5 ± 0.937.5 ± 0.9
Reweighting-only60.3 ± 3.487.9 ± 1.169.4 ± 7.5
SkillGate81.8 ± 1.094.1 ± 0.894.1 ± 0.8

Reweighting-only does learn at mixed trajectory lengths, but it starts slowly and falls far behind SkillGate on long trajectories. Amplifying the signal is not enough; the signal itself has to be corrected.

Real training. Reading behaviour over all on-policy rollouts of the two runs (same SFT checkpoint, 100 steps, 12,800 trajectories each), in 20-step windows:

Training steps0–1920–3940–5960–7980–99
SkillGate: clean oracle read27.8%55.9%69.1%78.2%76.7%
SkillGate: misleading read35.5%28.0%15.9%5.4%7.0%
Outcome-only: clean oracle read13.9%16.0%13.5%11.5%9.1%
Outcome-only: misleading read38.1%41.1%48.2%50.1%53.5%

Under Outcome-only the clean oracle read falls and the misleading read keeps rising, as the wrong-signed gradient predicts. Under SkillGate selection improves steadily.

AuthorsOpenReview ↗

Author Response (2/3)

2 (Weakness 2 and Question 2)

The objective assumes one oracle skill and a single read; scope and extensions to multiple valid or complementary skills should be discussed. How are oracle skills validated?

Response:

Single-oracle, single-read is the cleanest setting in which to isolate, measure and fix selector credit starvation, and we chose it deliberately. SkillGate's two channels do not depend on it: extending to multiple skills only requires replacing the selection utility u(a)u(a). Every oracle skill passed execution verification in the task environment.

(1) Why we study this setting first.

  • It matches the most common use of a skill library: reading the single most suitable guide on demand during execution.
  • With a unique oracle, a selection can be judged independently of the execution outcome, which all of the paper's starvation measurements rely on.
  • Even this setting is far from solved: the SFT model reads a misleading skill on 61.8% of trials (Table 3), outcome-only RL makes this worse during training, and none of the six frontier models in Table 1 reads the oracle on half of the trials.

(2) Oracle validation. Training and eval70 oracles are execution-verified: following the skill must solve the task in its environment, and failures are revised and verified again. Claw147 oracles are audited against the task materials.

DataOraclesVerificationResult
Training set491Execution verification247 passed on the first attempt; 219 failed, then passed after revision and re-verification; 25 passed after regeneration by a stronger model
eval7070Execution verification (the 4 SWE oracles with the gold patch)33 passed on the first attempt; 29 after revision; 8 after regeneration by a stronger model
Claw147147Item-by-item audit against the task, mock server, fixtures and grader56 confirmed correct; 91 had incorrect instructions in the body fixed. Names and descriptions are unchanged, so the selection input is unchanged
  • The oracle helps. Giving the SFT executor only the oracle raises success from 37.1% to 48.2% (Table 3).

(3) How the framework extends. Whether a task has a single oracle skill, several valid skills or a combination of complementary skills, it meets the same core problem, selector credit starvation, and the same solution applies: only the computation of the utility changes. SkillGate has three parts: (i) read calls are removed from the outcome loss; (ii) each selection has a fixed total weight N/MN/M, independent of trajectory length; (iii) the selection utility u(a)u(a). Only (iii) uses the single-oracle, single-read assumption.

SettingSelection utilityStatus
Single oracle, single read (paper default)u(a)=1u(a)=1 iff aa is the only read in the trajectory and it reads s⋆s^\starTrained and evaluated
Multiple reads allowedThe selection that first reads the oracle receives credit; other reads are unrestricted (SkillGate-MR, Action credit in the paper)Trained and evaluated: eval385 51.9%, oracle read 80.0%, misleading read 32.5%
Multiple valid skills (any one suffices)Replace s⋆s^\star with a valid set VxV_x; reading any member earns creditDrop-in replacement
Multiple complementary skillsu(a)=1u(a)=1 iff the set of skills read in the trajectory satisfies S(τa)∈FxS(\tau_a)\in\mathcal{F}_x, where Fx\mathcal{F}_x is the family of minimal sufficient sets for task xxReduces to the paper's definition when Fx={{s⋆}}\mathcal{F}_x=\{\{s^\star\}\}
Step-wise marginal creditu(aj)=V(Sj−1∪{sj})−V(Sj−1)u(a_j)=V(S_{j-1}\cup\{s_j\})-V(S_{j-1}), with VV a coverage functionAlso gives positive credit to a corrective read after a wrong first read

SkillGate-MR shows that identity-token credit remains effective once the single-read constraint is lifted. This is the first step towards multi-skill settings, and it is already implemented.

Revision. We will add a "Scope and extensions" paragraph to Sec. 3 and the verification statistics to Appendix A.

AuthorsOpenReview ↗

Author Response (3/3)

3 (Weakness 3 and Question 3)

There is no independent training-seed replication: are the main gains reproducible? Can external routing be evaluated with an executor matched to SkillGate?

Response:

(1) Each configuration is trained once: one run takes 100 steps of on-policy GRPO on 16 H800 GPUs and collects 12,800 long-horizon trajectories online, so repeating all eight RL configurations over multiple seeds is prohibitively expensive. Evaluation-side randomness is controlled by four trials per task, SkillGate is best or tied for best among the 9B methods on all five benchmarks, and three further pieces of evidence show that the gain comes from the credit design rather than from the chance of a single run. (2) With an executor matched to SkillGate, external routers remain below in-policy selection, which is already close to the perfect-router ceiling.

(1) Reproducibility.

All configurations share the same SFT checkpoint, the same 491 training tasks and the same 100-step budget (Sec. 4.1, Appendix A). The three pieces of evidence are:

#EvidenceResult
1Independent runs split consistentlyTwo runs with identity-token selection credit (SkillGate, SkillGate-MR): 53.2% and 51.9% on eval385. Six runs without it (Outcome-only, Task-mask only, Skill-free RL, Skill1, Group-level regret, Trajectory bonus): 44.7%–48.6%
2Large effect sizes in selection behaviour, which transfer to unseen tasks and unseen skillsvs Outcome-only on eval70: clean oracle read +53.9 pp [+44.3, +63.2], misleading read −47.9 pp [−57.5, −37.9]. On Claw147: clean oracle read 76.9% vs 34.7% (+42.2 pp [+33.3, +51.0]), misleading read 15.0% vs 53.7% (−38.8 pp [−47.6, −29.9])
3Training curves show a sustained trendSkillGate's clean oracle read rises 27.8 → 55.9 → 69.1 → 78.2 → 76.7%; under Outcome-only it falls from 13.9% to 9.1%

Evidence 1 is the most direct: the eight RL runs are independent and fall into two non-overlapping groups according to whether identity-token selection credit is present. The effect is thus reproduced by two independent runs, and the six runs without it give a reference for run-to-run variation. Differences of 40–58 pp in selection behaviour, which persist on a benchmark never seen in training, are not explained by such variation.

(2) Matched executor. In an external-router pipeline, the router picks one skill from the 16 candidates and only that skill is handed to the executor. We compare two executors on eval70: the frozen SFT model of Table 3 and the SkillGate checkpoint (identical RL training). The Oracle-only rows hand the oracle directly to the executor, which is equivalent to a perfect router; in the last row SkillGate selects by itself among the 16 candidates. The first eight rows form four pairs: the same source of selection, with the SFT executor above and the SkillGate executor below.

Configuration (eval70, 280 trials)SelectionExecutorSuccess
SFT-9B router → SFTExternalSFT40.7
SFT-9B router → SkillGate executorExternalSkillGate43.6
Qwen3.5-27B router → SFTExternalSFT36.8
Qwen3.5-27B router → SkillGate executorExternalSkillGate43.2
Reranker top-1 → SFTExternalSFT31.8
Reranker top-1 → SkillGate executorExternalSkillGate38.2
Oracle-only → SFT (perfect-router ceiling)GivenSFT48.2
Oracle-only → SkillGate (perfect-router ceiling)GivenSkillGate52.9
SkillGate (in-policy)Policy's ownSkillGate50.0
  • External routers stay below SkillGate even with the SkillGate executor, by 6.4–11.8 pp, which reflects the selection ability gained through SkillGate training.
  • SkillGate is close to the perfect-router ceiling: selecting by itself among 16 candidates, it reaches 94.6% of that ceiling (gap 2.9 pp [−2.5, +8.6]).
  • Executor training contributes 4.7 pp (the two Oracle-only rows).

Revision. We will collect the reproducibility evidence in a robustness subsection and add the matched-executor table to Sec. 4.4.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (2/2)

G3. Baselines with a separate selection stage and objective still read the oracle less often (v5VH-4, wnsv-1/4, wnsv follow-up 1). Neither daVinci-kernel nor ToolOmni addresses in-policy selection inside the execution trajectory. Skill1 keeps a separate selection stage with an outcome-derived objective, as in daVinci-kernel, and remains well below identity-token credit. A selector trained separately on gold labels (Selection BC), as in ToolOmni's retrieval stage, does help, but even paired with the SkillGate executor it reads the oracle in 65.7% of trials, against 83.9% for SkillGate (Table A).

G4. Single-read reduces wrong first reads and redundant reads without suppressing correction (wnsv-2). In a temporally ordered recovery analysis, SkillGate and SkillGate-MR correct a wrong first read at the same rate (23.1% vs 23.3%), while SkillGate makes fewer wrong first reads and fewer extra reads after the oracle, two behaviours that lower task success. The gap on Claw147 is not significant and does not come from correction.

G5. Each configuration is trained once; the effect is consistent across independent runs, throughout training and across evaluations (y1NK-3, wnsv-3, wnsv follow-up 2). The cost of each run rules out seed replications, and single training runs are the common protocol in this line of work.

  • Across runs. The eight independent RL runs fall into two non-overlapping groups (Table A): the two with identity-token credit reach 51.9%–53.2% on eval385 and the six without it 44.7%–48.6%. Oracle read and misleading read split in the same way.
  • Throughout training. SkillGate's clean oracle read rises steadily while that of Outcome-only declines; over the last 60 steps the per-step values are 54.7%–86.7% and 3.9%–22.7%, with no overlap.
  • Across evaluations. With four evaluation trials per task, SkillGate is best or tied for best among the 9B methods on all five benchmarks, and the change in selection behaviour transfers to the held-out Claw147 (clean oracle read 76.9%, against 34.7% for Outcome-only).

G6. The skill sets are built automatically, and the framework extends directly (y1NK-2, v5VH-2/3/5). Oracles are distilled by an LLM from execution trajectories and execution-verified; misleading skills are synthesised from them by error injection. Multiple valid or complementary skills and "no applicable skill" only replace the selection utility u(a)u(a). Under four changes in candidate construction, SkillGate's clean oracle read stays more than 40 pp above Outcome-only.

Revision plan.

  1. Sec. 4.6 and Appendix D: credit decomposition with the reweighting-only control, simulation and training curves.
  2. Sec. 4.3: how the ablation rows correspond to existing credit strategies; Task-mask only as the trained counterpart of identity-token credit without the weight.
  3. Sec. 4.4: matched executor, Oracle-skill RL and the separately trained selector.
  4. Related Work moved to Sec. 2, with a comparison table against daVinci-kernel and ToolOmni.
  5. Sec. 3 and Appendices A and C.1: scope and extensions, slate-construction pipeline, recovery analysis.
  6. New robustness and generalisation subsections; Action credit renamed SkillGate-MR.
AuthorsOpenReview ↗

General Response (1/2)

We thank the three reviewers for their careful reading and the discussion. The main conclusions of our responses, including the experiments added in the rebuttal, are summarized here.

Notation. eval385: the 385-trial protocol of Table 1. eval70: 70 tasks × 4 trials (Tables 2–3). Claw147: the 147 held-out Claw-Eval tasks (Table 5). Outcome-only: SkillRL (outcome only) in the paper. SkillGate-MR: Action credit in the paper, SkillGate without the single-read constraint. Clean oracle read: the oracle is the only skill read.

G1. Under outcome credit the selection signal is diluted and wrong-signed; SkillGate corrects both (y1NK-1, wnsv-4, wnsv follow-up 3). On the real batches of the outcome-only run, the selection tokens receive a negligible share of the gradient, 35.2% of oracle selections are pushed down and 49.4% of misleading selections are reinforced. Reweighting amplifies the magnitude by about 215× but leaves every wrong sign in place; only oracle-based local credit corrects the sign. Actual training confirms this (Table A): only the two runs with oracle credit on the identity tokens learn to select, and Task-mask only, which is SkillGate trained without the weighted selection term (λ=0\lambda=0), stays at the level of Outcome-only.

Table A. Eight RL runs with the same initialization, tasks and 100-step budget, and one staged pipeline (%):

MethodSelection crediteval385 successOracle readMisleading read
Skill1Dedicated selection turn, outcome-derived objective44.753.545.5
Skill-free RLNo skills46.0––
Task-mask onlyNone (λ=0\lambda=0)46.048.973.6
Group-level regretOracle signal on the prompt group46.533.647.5
Outcome-onlyTask outcome only47.054.369.6
Trajectory bonusOracle signal on the whole trajectory48.647.955.4
SkillGate-MROracle signal on identity tokens51.980.032.5
SkillGateThe same, with single-read53.283.921.8
Selection BC selector → SkillGate executor*Separate selector trained on oracle labels51.965.727.3

*New evaluation under the same eval385 protocol; a staged pipeline, not one of the eight RL runs.

G2. The gain comes from identity-token selection credit, not from single-read or executor training (wnsv-1/4/5, y1NK-3).

  • Not single-read. SkillGate-MR drops the constraint and still stands above every run without identity-token credit (Table A).
  • Not executor training. All eight runs receive RL training under the same budget, yet only the two with identity-token credit improve selection (Table A). Oracle-skill RL, trained with only the oracle provided, learns execution but not selection (Table B).
  • Unchanged with a matched executor. Even with the SkillGate executor, external routers stay below SkillGate's own selection, which reaches 94.6% of the perfect-router ceiling (Table B).

Table B. Matched executor and Oracle-skill RL, both added in the rebuttal (eval70 success, %):

Source of selectionSFT executorOracle-skill RLSkillGate executor
SFT-9B router40.7–43.6
Qwen3.5-27B router36.8–43.2
Reranker top-131.8–38.2
Oracle given (perfect router)48.250.452.9
The model's own selection among 16 candidates37.141.150.0

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.