acceptodds
Under review as a conference paper at ICLR 2027

SIAR: Skill Influence Attribution for Skill-Augmented Agentic Reinforcement Learning

Abstract

Large language model (LLM) agents have shown strong potential in long-horizon interactive tasks, and recent skill-augmented agentic reinforcement learning (RL) methods further improve them by distilling past experience into reusable natural-language skills. However, existing methods remain coarse-grained in both skill updating and policy optimization: skills are updated according to weakly diagnostic outcome signals or fixed schedules, and all actions in a trajectory receive a uniform advantage regardless of the retrieved skill's influence. To address these limitations, we propose SIAR, a skill-augmented agentic RL framework centered on skill influence attribution. SIAR measures how strongly the retrieved skill drives the agent toward each executed action, producing a step-level attribution signal that constitutes the missing fine-grained evidence. Building on this signal, attribution-guided skill updating rewrites a skill only when accumulated attribution evidence suggests a mismatch between its influence and task outcomes, and attribution-calibrated advantage assignment augments the uniform trajectory-level advantage with additional credit or penalty for strongly skill-influenced actions. The two components allow skills and the agent policy to co-evolve throughout training. Experiments on three challenging agentic tasks (ALFWorld, WebShop, and search-augmented QA) show that SIAR outperforms existing skill-augmented agentic RL methods.

Primary area:
reinforcement learning
Keywords:
LLM Agents, Reinforcement Learning, Skills

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iBGLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SIAR, a skill-augmented agentic reinforcement learning framework that leverages step-level skill influence attribution to enable the co-evolution of skills and agent policies. By quantifying how strongly retrieved skills drive individual actions through log probability comparisons, SIAR guides skill updates only when accumulated evidence indicates a mismatch with task outcomes and calibrates advantage assignment to provide targeted credit or penalties for highly influenced steps. Experiments on ALFWorld, WebShop, and search-augmented QA benchmarks demonstrate that this approach significantly outperforms existing methods by effectively aligning policy optimization with fine-grained skill influence signals.

Critical strengths
  1. The writing is well-presented.
  2. The motivation is well-justified.
Critical weaknesses
  1. Please explain how the current implementation controls for positional embedding shifts when computing Δt\Delta_t. Did you use padding or other alignment techniques? I suggest adding an additional set of experiments to demonstrate that this does not affect the final results.

  2. I suggest discussing the sensitivity of the performance to the threshold N−=5N^-=5. Specifically, address how the method handles scenarios where failed trajectories are scarce (leading to skill stagnation). In later training stages where failures are rare, does this lead to skill stagnation?

Questions for authors

Please refer to "Critical Weaknesses".

Reviewer_SdwGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper targets skill-augmented agentic RL, where natural-language skills are injected into an LLM agent’s prompt and refined jointly with the policy. It argues that current methods are too coarse in two places: they rewrite skills based on raw outcomes or fixed schedules, and they give every step in a trajectory the same advantage. SIAR addresses both with a step-level “skill influence” score, defined as the token-averaged log-probability difference of the executed action with versus without the skill in context (stop-gradient). It uses this score in two ways: Skill updating: a skill is rewritten only when an influence–outcome consistency score is negative and enough successes and failures have been collected. Advantage calibration: steps with above-group-mean influence receive an extra outcome-signed advantage, added to the GRPO advantage. On ALFWorld, WebShop, and seven search-QA benchmarks, it improves over SkillRL, Skill0, D2Skill and SkillC. LLM baselines are included: GPT-4o, Gemini-2.5-Pro, and the Qwen2.5-7B-Instruct backbone, plus prompting, memory, and RL baselines. However, only a single 7B backbone is trained.

Critical strengths

Firstly, this paper targets a well-motivated problem. Separating “the skill is bad” from “the rollout failed for other reasons” is a real gap in skill-evolving RL. Evidence-gated updating is a sensible alternative to schedule-driven rewriting, which can overwrite reliable guidance. Besides, this paper demonstrates a consistent empirical gain with supporting analyses. SIAR achieves the best overall numbers on all three task families (93.4 ALFWorld, 82.4 WebShop success, 48.9 QA average), including out-of-domain QA gains.

Critical weaknesses

First of all, the skill influence score Δ\Delta in Eq. (3) is not a reliable measure of how much an action is driven by the skill. Since ata_t is sampled from the skill-conditioned policy, the expectation of each per-token term in Δ\Delta is a KL divergence:

Eaℓ∼πθ(⋅∣x,ot,s,a<ℓ)[log⁡πθ(aℓ∣x,ot,s,a<ℓ)−log⁡πθ(aℓ∣x,ot,a<ℓ)]=DKL(πθ(⋅∣x,ot,s,a<ℓ),∥,πθ(⋅∣x,ot,a<ℓ))≥0.\mathbb{E}_{a_\ell \sim \pi_\theta(\cdot \mid x,o_t,s,a_{\lt \ell})}\left[\log \pi_\theta(a_\ell \mid x,o_t,s,a_{\lt \ell})-\log \pi_\theta(a_\ell \mid x,o_t,a_{\lt \ell})\right]=D_{\mathrm{KL}}\left(\pi_\theta(\cdot \mid x,o_t,s,a_{\lt \ell})\\,\Vert\\,\pi_\theta(\cdot \mid x,o_t,a_{\lt \ell})\right)\ge 0.

Δ\Delta is therefore structurally biased to be positive, and indeed every reported Δˉ\bar{\Delta} is positive. It measures how strongly the skill shifts the output distribution, not whether that shift is beneficial. Equivalently, Δ\Delta is the pointwise mutual information between the skill and the action, and dependence can arise from lexical copying as easily as from genuine guidance: inserting text into the prompt raises the probability of tokens that appear in it, regardless of decision quality. The paper's own example fits this reading. In the failed trajectory of Fig. 2, the highest-influence step is the invalid action "examine countertop 1", which yields "Nothing happens." In addition, the policy is never trained without skills, so the skill-free counterfactual is partly out of distribution. If the action tokens include the <think> reasoning, Δ\Delta largely measures how closely the reasoning paraphrases the skill.

Besides, it is unclear where the gain comes from. Static skills alone lift GRPO from 77.6 to 82.8 (Table 3). These skills are written by GPT-5.4 from trajectories of a GiGPO-trained agent, and GPT-5.4 keeps rewriting them during training. Because the SIAR policy is never evaluated without skills or with swapped skills, the improvement cannot be separated into a better policy, better skill text, and knowledge inherited from the stronger GiGPO agent. On search-augmented QA, the entire pipeline adds only 1.8 points over static skills.

Last but not least, neither use of Δ\Delta is properly controlled. Since Δ\Delta is positive in expectation, the update criterion κ(sq)<0\kappa(s_q) < 0 behaves much like an influence-weighted failure rate. Yet the paper never compares it with the outcome-based trigger it criticizes, and the scheduled baseline performs twice as many updates. The step advantage in Eq. (7) is not zero-mean and can dominate the standardized GRPO advantage at ω=1.5–2.0\omega = 1.5–2.0. ALFWorld results range from 84.4 to 93.4 across ω\omega (Fig. 4), and no shuffled-Δ\Delta or constant-bonus control is reported.

Questions for authors

First of all, given that E[Δ]≥0\mathbb{E}[\Delta] \ge 0 by construction, how do you separate genuine skill guidance from the mere distribution shift the skill induces? Do the action tokens in Eq. (3) include the <think> reasoning? How does Δ\Delta correlate with lexical overlap between the action and the skill text, and how much of Δ\Delta survives a controlled counterfactual, such as a length-matched or shuffled skill, instead of an empty skill slot?

Besides, could you evaluate the SIAR policy without skills and with the initial skills, and the GRPO + static-skill policy with SIAR's final skills? Could you also report results when the initial skills are not derived from GiGPO trajectories?

Moreover, how does SIAR compare with an outcome-only update trigger and with a schedule matched to the same number of updates? How does the step advantage compare with a version using shuffled Δ\Delta or a constant bonus, and what is the typical magnitude of ω⋅AS\omega \cdot A^S relative to AEA^E?

Finally, why does the re-evaluated SkillRL checkpoint fall below the number in Table 1? Did all skill-based baselines start from the same initial skills?

Reviewer_KFbWOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SIAR, a skill-augmented agentic RL framework that uses step-level skill influence attribution to guide both skill updating and policy optimization. The authors measure skill influence by comparing action log probabilities with and without the retrieved skill. This signal is then used to decide when a skill should be updated and to assign additional step-level advantages on top of the original GRPO advantage. Experiments on ALFWorld, WebShop, and search-augmented QA show improvements over existing baselines.

Critical strengths

Good experimental results, especially on ALFWorld. I also like the motivation of using a more fine-grained learning signal for skill-augmented RL. The supplementary material is good and anticipates many of the questions I had while reading the paper.

Critical weaknesses

I think my main concern is the methodology validation. The proposed skill influence score measures how much a skill affects the probability of an action, but this does not necessarily reflect the action's contribution to task success. I am not fully convinced that the current experiments sufficiently validate the proposed step-level credit assignment.

Another concern is the trajectory selection for skill updating. In Appendix A.5, the authors keep the shortest successful trajectory and the failed trajectory with the highest mean skill influence. However, according to Eq. 4, a failed trajectory with high positive skill influence would contribute negatively to the consistency score. I think this selection strategy could introduce bias toward negative consistency scores, making skill updates more likely.

I also have some concerns about the fairness of the baseline comparisons. The initial skills are generated by GPT-5.4 using trajectories from a GiGPO-trained agent and the backbone model, while most baseline results are directly taken from previous papers. I am not sure whether the baselines have comparable skill initialization and training budgets.

I also think the claims about "co-evolution" and the "mutually reinforcing loop" are not sufficiently supported. While the ablation results show that both components improve performance, the paper does not really show how the policy and skills improve each other throughout training. I would like to see some training curves or analysis of skill quality and policy performance before and after skill updates to better support these claims.

Questions for authors
  1. The authors use an LLM judge to evaluate the selected steps and compare them against random selection. Does it really show that the influence score reflects an action's contribution to task success? I would like to see a more direct validation.

  2. Why should a highly skill-influenced action receive an additional negative advantage simply because the final trajectory fails? For long-horizon tasks, some earlier actions may be correct or even necessary, while the failure is caused by mistakes at later steps. I am not convinced that the current method can distinguish these cases. Also, I would like to see a comparison with simpler advantage-reweighting methods to better understand the benefit of the proposed attribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.