- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SkillScafold, a new method in Agentic RL which aims to improve skill generation and skill-free inference, by providing a novel training pipeline.
The authors highlight significant motivating observations, such as instability and high cost of rollout utility estimation, query and policy dependence of the rollout skill utility. Based on them, they provide a better way to estimate utility with group-based approach, based on probabilities of preferable trajectories rather than the rollout based utility. By jointly training a single model on both skill generation and skill-conditioned inference, and using proposed utility estimation method to select skill candidates, they achieve promising results on various benchmarks such as WebShop, ALFWorld, ScienceWorld and r^2-bench. In particular, they measure the internalization of skills by the model by providing skill-free benchmarks, which shows the positive effect of joint train.
Critical strengths
- The paper practically addresses when and which self-generated skill to inject in agentic RL. Section 3 empirically proves that skills can hurt performance depending on the current policy's strength. This idea matters because it is simple, cheap, and easily applicable to other hint-guided RL exploration methods.
- The method is thoroughly evaluated on four benchmarks, including the long-horizon r^2 benchmark, outperforming standard RL and deep variety of baselines. Significant work has been done in theoretical analysis, which makes deeper understanding on components of the method. This extensive evaluation makes the reported gains highly plausible and clearly isolates where the improvements come from.
- The paper is well-written, the motivating experiments section in general makes sense, despite the existence of weaknesses. Overall, the text makes deeper understanding of the problem even for people whose research is not much related to the field
Critical weaknesses
- The "motivating questions" experiments are model and dataset-specific. The concern is that specified Qwen3-8B, which was used across all experiments, may bring a model-specific bias in motivating questions formulation, because skill-guided and skill-free inference performance may vary across models. Alongside, the "non-saturated pilot subset" is of an undetermined nature: there is no evidence as to how it was formed. In the end, certain evaluations, such as disagreement dependence on K, may vary across different subsets, which make the pilot study biased.
- There is no ablations on effectiveness of group-based utility estimation in comparison with rollout based approach.
- The methodology of scouts-to-skills pipeline raises questions. There is no evidence on choosing 4 scout trajectories instead of another number and why are we passing only positive candidate instead of all (or one positive-one negative) candidates.
- Minor issues: N is not large enough for some benchmark on figure 4, WebShop is considered to be a relatively small benchmark in figure 4, which makes unreliable eval. In general, multiseed is conducted for comparison in WebShop and ALFWorld, because of small validation groups
- The GRPO and "no executor injection" setups (Table 1 and Table 3) differ only in existence of auxiliary loss; surprisingly, this yields a substantial performance boost (90.3 vs 85.6 ALFWorld, 41.5 vs 38.2 WebShop), which raises questions about how much of an improvement the skill injection mechanism contributes. In particular, that may be caused by divergence of configs the models were trained with.
Questions for authors
- Are the paper findings transferrable on different models? Can authors rerun some benchmarks (table 1, motivating expreriments) with different models, like qwen3-4b?
- Why do authors choose K=4 to execute skillscafold? How do performance and inference ablates on different K? Can authors provide
- It's not clear whether we exclude all-negative scenarios from r^2 benchmark or not. In general, the appendix say that we exclude all-success, all-failure tasks from r^2, but experiments like figure 12 considered to have them in dataset.
- Are the evaluation pipelines differs in Table 1 and Table 3? If so, are the configs of GRPO and "no executor injection" differs, and how much improvement the skill injection mechanism gives?
4 comments
Reply to Reviewer v4om(2/4)
(2) W2: Comparison with Rollout-Based Utility Estimation
W2: There is no ablations on effectiveness of group-based utility estimation in comparison with rollout based approach.
Response: Thanks for the suggestion. We address this through a method-level comparison and an additional utility-estimation experiment.
1. Method-level comparison. Table 1 includes Evolving-RL, which evaluates skills through downstream environment rollouts. With Qwen3-8B, the results are:
| Method | ALFWorld | WebShop |
|---|---|---|
| Evolving-RL† | 74.8 | 41.8 |
| Evolving-RL* | 77.0 | 27.7 |
| SkillScaffold* | 94.8 | 45.1 |
| (* denotes skill-free evaluation; † denotes evaluation with skills.) |
These method-level results show that SkillScaffold achieves higher success rates than the rollout-based baseline on both benchmarks without requiring candidate-specific environment evaluation.
2. Direct utility-estimation comparison. We extend the evaluation in Section 5.4, Figure 5(a), with rollout-based utility estimation. For each candidate skill, we sample trajectories on the corresponding task and use its empirical pass rate as the utility estimate. The highest-scoring candidate is selected and evaluated using separate downstream rollouts. With four candidates per query, this requires additional candidate-evaluation rollouts.
| Utility estimation | Extra candidate-evaluation rollouts/query | Downstream success rate (%) |
|---|---|---|
| Likelihood utility + gated top-1 | 0 | 67.6 |
| Rollout utility, | 4 | 60.5 |
| Rollout utility, | 8 | 60.0 |
| Rollout utility, | 16 | 68.4 |
| Rollout utility, | 32 | 67.8 |
Our likelihood-based utility exceeds rollout-based selection by 7.1/7.6 percentage points at . At , rollout-based selection achieves slightly higher success rates by 0.8/0.2 points, but requires 16/32 additional environment rollouts per query. Our method therefore provides competitive downstream performance without candidate-specific environment evaluation. We have added this direct utility-estimation comparison to Appendix B.7 of the revised manuscript.
(3) W3 & Q2: Scout Budget Hyperparameter Experiments
W3: The methodology of scouts-to-skills pipeline raises questions. There is no evidence on choosing 4 scout trajectories instead of another number and why are we passing only positive candidate instead of all (or one positive-one negative) candidates.
Q2: Why do authors choose K=4 to execute skillscafold? How do performance and inference ablates on different K? Can authors provide
Response: Thanks for the question and suggestion. We have added a Scout-budget ablation on WebShop. To keep the total number of Executor training rollouts per query fixed at eight, we vary the allocation between Scouts and Post-Scouts while fixing the candidate count at .
| Scouts | Post-Scouts | WebShop success rate (%) |
|---|---|---|
| 2 | 6 | 39.33 |
| 4 | 4 | 45.10 |
| 6 | 2 | 37.82 |
With , fewer Scout trajectories may limit the availability of reliable evidence for skill generation and utility estimation. With , only two Post-Scout rollouts remain, reducing the proportion of potentially skill-guided trajectories in the Executor training group. The 4/4 allocation balances these two needs and performs best among the settings. We have added this Scout-budget ablation to Appendix B.6 of the revised manuscript.
Reply to Reviewer v4om(3/4)
(4) W4: Evaluation Reliability
W4: Minor issues: N is not large enough for some benchmark on figure 4, WebShop is considered to be a relatively small benchmark in figure 4, which makes unreliable eval. In general, multiseed is conducted for comparison in WebShop and ALFWorld, because of small validation groups
Response: Thanks for highlighting the importance of evaluation reliability. We would like to gently clarify that Figure 4 presents the architectural overview of SkillScaffold. To directly address the evaluation reliability on ALFWorld and WebShop, we report additional multi-seed comparisons below.
Tables 1–3 originally report single training runs, with Pass@1 estimated by averaging over rollouts per task. To assess training-run variability, we reran SkillScaffold and the strongest skill-based RL baselines—SKILL1 on ALFWorld and Evolving-RL on WebShop—across three independent random seeds. The mean ± standard deviation results are:
| Benchmark | Method | Success rate (%) |
|---|---|---|
| ALFWorld | SkillScaffold* | 93.94 ± 0.74 |
| ALFWorld | SKILL1† | 90.30 ± 1.37 |
| WebShop | SkillScaffold* | 45.04 ± 1.30 |
| WebShop | Evolving-RL† | 40.60 ± 1.17 |
(* denotes skill-free evaluation; † denotes evaluation with skills.)
SkillScaffold exceeds the strongest baselines by 3.64/4.44 percentage points on ALFWorld/WebShop in mean performance. These repeated-run results strengthen the evidence for the consistency of the reported gains.
(5) Q3: Clarification of -Bench Filtering and Online Scout Outcomes
Q3: It's not clear whether we exclude all-negative scenarios from r^2 benchmark or not. In general, the appendix say that we exclude all-success, all-failure tasks from r^2, but experiments like figure 12 considered to have them in dataset.
Response: Thanks for the question. We would first clarify that the benchmark is -Bench.
We evaluate the post-SFT model on all training queries using eight rollouts per query, then exclude queries with 0/8 or 8/8 successes to retain queries with mixed observed outcomes for group-relative learning.
This initial filtering does not guarantee mixed outcomes in subsequent Scout groups. During training, we sample four fresh Scout trajectories per query at a nonzero temperature. Queries with an initial success rate of 1/8 or 2/8 can therefore yield all-wrong Scout groups, while those with 6/8 or 7/8 can yield all-correct groups. Thus, the all-wrong and all-correct groups in Figure 12 are consistent with our filtering protocol.
Reply to Reviewer v4om(4/4)
(6) W5 & Q4: Contribution of Skill Injection and Configuration Consistency
W5: The GRPO and "no executor injection" setups (Table 1 and Table 3) differ only in existence of auxiliary loss; surprisingly, this yields a substantial performance boost (90.3 vs 85.6 ALFWorld, 41.5 vs 38.2 WebShop), which raises questions about how much of an improvement the skill injection mechanism contributes. In particular, that may be caused by divergence of configs the models were trained with.
Q4: Are the evaluation pipelines differs in Table 1 and Table 3? If so, are the configs of GRPO and "no executor injection" differs, and how much improvement the skill injection mechanism gives?
Response: Thanks for raising this question. We clarify the configurations and distinguish the contributions of Generator learning and Executor skill injection below.
1. Evaluation pipeline and training configurations. Tables 1 and 3 use the same evaluation pipeline, test splits, decoding settings, and Pass@1 estimation with eight rollouts per task.
For training, GRPO uses a clipping parameter of 0.20 with a small KL penalty. SkillScaffold uses lower/upper clipping parameters of 0.20/0.28 and no KL penalty. These settings were chosen to accommodate joint Generator–Executor optimization with shared model parameters. Importantly, Full SkillScaffold and No Executor injection use identical training configurations and Executor rollout budgets.
2. Contribution of Generator learning. The "No Executor injection" variant is not merely standard GRPO with an auxiliary loss. Instead, it maintains the complete SkillScaffold training architecture: the policy is still actively trained to diagnose Scout trajectories, synthesize high-utility skills, and optimize skill-generation tokens via RL loss.
Through this active training loop, the model continuously encounters, generates, and refines skills. Because the Generator and Executor share underlying parameters, training the model to master skill synthesis directly instills robust task-abstraction and reasoning capabilities into the shared backbone. This explains why the variant outperforms standard GRPO even when prompt-level skill injection is withheld during Executor execution.
3. Controlled comparison of skill injection. Comparing Full SkillScaffold with No Executor injection retains Generator learning and strictly controls the training configuration:
| Training setup | ALFWorld AVG | WebShop AVG |
|---|---|---|
| No Executor injection | 90.3 | 41.5 |
| Full SkillScaffold | 94.8 | 45.1 |
| Additional gain | +4.5 pp | +3.6 pp |
These results demonstrate an additional benefit from training-time Executor skill injection, beyond retaining Generator learning. Both variants are evaluated without skills.
For clarity, the quoted WebShop GRPO score of 38.2 is its ID score; its overall AVG is 36.5.
Thank you again for your constructive feedback, which helped us strengthen the evidence for cross-model generalization and evaluation reliability, add direct utility-estimation comparisons and hyperparameter experiments on Scout allocation.
Reply to Reviewer v4om(1/4)
Thank you for your careful review and constructive suggestions!
Summary. Your main concerns cover cross-model generalization, utility estimation, Scout-budget choices, evaluation reliability, and experimental protocols. We provide additional cross-model experiments, direct utility-estimation comparisons, Scout-budget ablations, and three-seed evaluations, alongside clarifications of pilot sampling, τ²-Bench filtering, and the controlled comparison of skill injection.
(1) W1 & Q1: Cross-Model Generalization and Pilot-Study Protocol
Response: Thanks for the question and suggestion.
1. Cross-model evaluation. To test transferability, we repeat the Table 1 comparisons using Qwen2.5-7B-Instruct, including GRPO, SkillRL, and the strongest skill-based RL baseline on each benchmark.
(* denotes skill-free evaluation; † denotes evaluation with skills.)
SkillScaffold exceeds the strongest competing results by 4.95/3.00 percentage points on ALFWorld/WebShop, supporting its effectiveness beyond Qwen3-8B. Full category-level results are reported in Appendix B.2 of the revised manuscript.
2. Cross-model pilot validation. We have added two pilot experiments with Qwen2.5-7B-Instruct: rollout-based utility-estimation stability and within-query candidate-effect evaluation. At K=8, independent rollout-based utility estimates disagree in sign for 29.9% of candidate–query pairs whose two estimates are both nonzero (72/241). Moreover, 40 of 100 queries contain candidates with both positive and negative measured downstream effects. These results support the motivating observations beyond Qwen3-8B. The additional results are reported in Appendix B.2 of the revised manuscript.
3. Pilot-subset construction. We clarify that the original non-saturated subset was randomly sampled from WebShop training queries whose skill-free success rate under the base Qwen3-8B model satisfies 20%≤SR≤80%, estimated using K=8 rollouts per query. This criterion uses skill-free performance rather than candidate skill effects. We have added this sampling protocol to Appendix A.5 of the revised manuscript.