acceptodds
Under review as a conference paper at ICLR 2027

Shaping the Group via Skill-Probed Policy Optimization for Language Agents

Abstract

Improving language agents on long-horizon tasks hinges on exploration: an agent can only reinforce the behaviors it manages to discover through interaction. In complex environments with delayed rewards, however, breakthrough behaviors stay buried in the low-probability tail of a combinatorial action space, so passive sampling keeps rediscovering the same failures. This bottleneck is especially acute for group relative reinforcement learning such as Group Relative Policy Optimization (GRPO), which forgoes memory-intensive value networks and instead computes advantages from performance differences within a group of rollouts. When the rollouts fail identically, the advantage vanishes and training suffers gradient starvation. To surface the buried behaviors, we propose SPPO, which turns exploration from passive sampling into active group shaping. For each task, an auxiliary generator produces a current skill that approaches the reachable performance ceiling and a lagged skill from a historical snapshot that provides an intermediate coaching floor. Paired with unguided baseline rollouts, these skills prune the search space and shape a trajectory group with a wide reward spread, restoring non-zero, discriminative advantages. The generator itself is trained from environment feedback, with the current and lagged rollouts placed in one group. Crucially, the actor is optimized via a skill-stripped objective with asymmetric credit attribution: positive-advantage trajectories are double-scored to internalize breakthroughs into the autonomous policy, while negative-advantage trajectories are attributed to the skill to protect the standalone policy from probe-induced errors. On a challenging long-horizon agent benchmark, group shaping lowers the fraction of gradient-starved all-fail groups by about points on average over training relative to unguided sampling, and the resulting agent operates completely autonomously at inference time. Our code is available at https://anonymous.4open.science/r/SPPO-shaping-the-group.

Primary area:
reinforcement learning
Keywords:
LLM agents, skill-guided exploration, co-evolution agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RnEnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Skill-Probed Policy Optimization (SPPO) to alleviate gradient starvation in group-relative reinforcement learning for long-horizon language agents. It introduces current and lagged skills to shape rollout groups and employs asymmetric credit attribution to internalize skill-guided successful behaviors into a skill-free actor. Experiments on ALFWorld show improved task success rates over conventional agent RL baselines.

Critical strengths
  1. Meaningful motivation and a coherent technical framework

    The paper addresses a practical challenge in agent reinforcement learning: sparse rewards and unsuccessful rollouts can limit exploration and effective policy optimization. Using natural-language skills to guide exploration and subsequently internalizing successful behaviors into a standalone policy is a reasonable approach. The reported improvements over GRPO and GiGPO are encouraging.

Critical weaknesses
  1. Limited novelty and insufficient differentiation from closely related work

    The core ideas of skill-guided exploration, group-level rollout shaping, and skill internalization overlap substantially with existing research. In particular, GOLF [1] already uses natural-language feedback to introduce successful trajectories into low-reward groups and improve exploration. SKILL0 [2] and Skill0.5 [3] also investigate skill internalization in agentic RL. Although SPPO introduces temporal skill contrast and asymmetric credit attribution, its distinct methodological contributions over existing guided exploration and internalization approaches remain insufficiently established.

  2. Missing strong baselines and inadequate component comparisons

    The main experiments compare SPPO primarily against conventional RL methods, while omitting closely related skill-based approaches such as SKILL0 and Skill0.5, as well as a GOLF-style guided exploration baseline. Moreover, it omits essential controlled baselines, including a frozen skill generator, successful-trajectory SFT/distillation, and standard guided GRPO with matched rollout data. Without these comparisons, it is difficult to determine whether the improvements arise from the proposed optimization objective or simpler forms of skill-guided training.

  3. Limited evaluation benchmarks and insufficient generalization evidence

    All main experiments are conducted on a single benchmark, ALFWorld, using only 128 games from its in-distribution valid_seen split. The available valid_unseen split is not evaluated, and no experiments are reported on other agent environments such as WebShop or ScienceWorld. Given the paper's broader motivation concerning long-horizon language agents, the current evaluation is insufficient to establish whether SPPO generalizes across unseen environments, task structures, and interaction patterns. Furthermore, using a single training seed limits the reliability of the reported improvements.

  4. Unclear performance attribution due to additional expert supervision and computational overhead

    The skill generator is initialized using GPT-5.5-generated skills distilled from reference walkthroughs. This introduces privileged demonstration-derived information that is not explicitly matched in the main RL baselines. Additionally, SPPO requires an auxiliary trainable generator and additional skill-generation and optimization overhead. Without comparisons under matched supervision and total training compute, the reported improvements cannot be confidently attributed to the proposed algorithmic mechanism.

  5. Insufficient validation of the proposed optimization mechanism

    The existing ablations demonstrate the importance of lagged skills and double-scoring within the proposed framework, but do not establish their necessity over simpler alternatives. In particular, the skill-stripped objective updates the unguided policy using trajectories sampled under skill-conditioned distributions, introducing a distribution mismatch that deserves further justification. The paper also lacks comparisons with conventional off-policy distillation and simpler auxiliary skill-conditioned losses. These limitations make it difficult to isolate the benefits of asymmetric credit attribution from those of guided exploration and successful-trajectory reuse.

[1] Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning

[2] SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization

[3] Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learning

Questions for authors

See weaknesses

2 comments

Thank you for the careful and constructive review. Below we respond to each point in turn.

W1 (novelty). We will add a detailed comparison to the paper. The main differences are:

  • GOLF is not a skill method. It turns failed rollouts plus an external critique into a refinement prompt, and injects the improved answers into low-reward groups as off-policy guidance. SPPO instead trains a skill generator with a temporal contrast between current and lagged skills, and trains the actor with asymmetric credit attribution.
  • SKILL0 removes skills from the context with a fixed curriculum until the actor runs with none. In SPPO, a trainable generator writes the skill for each task.
  • Skill0.5 internalizes only general skills and keeps task-specific skills in the context at test time, so it is not evaluated on a bare actor as ours is (see W4).

The SKILL0 paper reports 87.9% (3B) and 89.8% (7B) on ALFWorld for SKILL0 and 82.4% and 89.9% for SkillRL, which at 7B do not exceed GiGPO or SPPO. GOLF and SKILL0 also use different settings (math, code and open tasks for GOLF, Qwen2.5-VL for SKILL0). A direct comparison needs reruns under a matched protocol, which we could not finish within the rebuttal period.

W2 (baselines and component comparisons). For the skill-based baselines, see W1. For the controlled baselines (Qwen2.5-1.5B, test set valid_seen with 128 games, versus 90.6% for SPPO):

  • Frozen skill generator with SPPO's actor objective: 71.9%.
  • Frozen skill generator with standard guided GRPO and the same 8+4+4 rollouts per task: 64.1%.
  • SFT of the actor on the GPT-5.5/walkthrough-derived data, then GRPO: 82.8% at the final step.
  • GRPO with 16 unguided rollouts per task: 72.7%.

W3 (benchmarks). Following GiGPO, we evaluate on valid_seen, since the baselines are only evaluated there. The 3-run experiments and the additional WebShop experiments are still running, and we will report them as soon as they finish.

W4 (attribution of extra supervision and compute). The GPT-5.5/walkthrough-derived initialization of the skill generator is part of our method. Other methods also carry components that we do not use, such as extra credit assignment and reward terms, and these contribute to their results. To separate the effect of the extra supervision and compute, we ran the controls listed in W2 (also in our reply to Reviewer hUh3, W1 and W2).

W5 (validation of the optimization mechanism).

  • Distribution mismatch: the bias is by design, and the argument is below.
  • Necessity of double-scoring: without it (w=1), the success of guided rollouts falls to 0 and the policy entropy rises from 1.0 to 6.3 (details in our reply to Reviewer Un7v, Supplementary analysis).
  • Necessity of lagged skills: on hard tasks, the current skill rescues 15.5% of all-fail groups and the lagged skill 12.4% (Fig. 4).
  • Separating asymmetric credit from guided exploration: with the same frozen generator and rollouts, SPPO's actor objective reaches 71.9% and standard guided GRPO 64.1% (W2).
  • Simpler auxiliary skill-conditioned losses: we tried a KL term that constrains how the actor follows the skill, but it had little effect. Double-scoring solved this problem much better.

Off-policy argument. SPPO reshapes the rollout group with skill guidance, which changes the sampling distribution. We argue that the resulting update is still an ascent direction. Let p=πθ(⋅∣x,H)p=\pi_\theta(\cdot|x,H) be the skill-free policy and q=πθ(⋅∣x,H,s)q=\pi_\theta(\cdot|x,H,s) the skill-conditioned behavior policy.

  1. Skill-free scoring of a guided trajectory with A>0A>0 gives Ea∼q[A∇log⁡p(a)]=Ea∼p[ρA∇log⁡p(a)]E_{a\sim q}[A\nabla\log p(a)]=E_{a\sim p}[\rho A\nabla\log p(a)] with ρ=q/p>0\rho=q/p>0. This is the policy gradient of pp with each term reweighted by a positive factor, so every successful trajectory is pushed up. The skill is a short hint, so qq approximates pp. If ∣ρ−1∣≤ϵ|\rho-1|\le\epsilon, the update differs from the exact policy gradient of these terms by at most ϵ Ep[∣A∣ ∥∇log⁡p∥]\epsilon\,E_p[|A|\,\|\nabla\log p\|], so it remains an ascent direction for small ϵ\epsilon. We have not measured ϵ\epsilon. The skill raises ρ\rho on high-reward trajectories in the low-probability tail of pp, which sampling from pp alone rarely reaches.
  2. The second score of double-scoring and the negative-advantage terms are computed in the skill context where the trajectory was sampled. Their ratio is the standard on-policy ratio, so they are the exact policy gradient of qq and carry no mismatch. The weights are shared, so these terms can still move pp, but they do so as an ordinary on-policy update. Without them (w=1), the skill context only receives negative updates and the performance drops to 63.3%.

We will add this analysis and a discussion of related work on RL from off-policy guidance and hints to the paper.

Dear Reviewer,

Thank you for your constructive feedback. We have submitted a revised version of the paper in line with your comments. Every requested change is included except the 3-run experiments, which we did not have time to complete. We will run the main experiments with three seeds and add the mean and standard deviation in the next revision.

Best regards, The Authors

Reviewer_Un7vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a simple and straightforward approach to address the lack of successful rollouts in long-horizon agent RL. By introducing different skills during exploration, the method increases the chance of discovering successful trajectories while creating more diverse rollout groups for policy optimization.

Critical strengths
  1. Clear and meaningful motivation. The motivation is straightforward and addresses an important problem in long-horizon agent RL.
  2. Promising experimental results. The experiments demonstrate clear performance improvements over existing baselines.
Critical weaknesses
  1. Lack of comparison with cold-start SFT. If skill-guided exploration can produce successful trajectories, a simpler alternative would be to collect these trajectories for cold-start SFT, followed by standard GRPO. It is unclear whether the proposed framework provides additional benefits over this straightforward approach.

  2. Lack of comparison with larger rollout groups. If skills mainly help discover successful trajectories and increase reward diversity, increasing the number of unguided rollouts might achieve similar effects. The paper should compare against GRPO with larger rollout groups under comparable computational or interaction budgets.

Questions for authors

I personally like this paper and find its motivation and approach appealing. However, I believe it is important to demonstrate that the proposed method outperforms the simpler alternatives mentioned above. In particular, comparisons with cold-start SFT + GRPO and GRPO with larger rollout groups would significantly strengthen the paper. If the authors can provide convincing experimental evidence showing clear advantages over these simple baselines, I would be happy to increase my rating.

2 comments

Thank you for the careful and constructive review. Below we respond to each point in turn.

W1 (cold-start SFT). We ran this control. We used the GPT-5.5/walkthrough-derived data that pre-trains the skill generator to SFT the actor, then ran standard GRPO (Qwen2.5-1.5B, 8 rollouts per task, 150 steps). Success on the test set (valid_seen, 128 games) is 68.8% after SFT and 82.8% at the final step, approaching SPPO's 90.6% (see our reply to Reviewer hUh3, W1). This is a single run. SFT+RL was our original design. We did not adopt it because all RL baselines start from the base model, and we kept the starting point matched. In SPPO the skills are also generated online and co-trained with the actor, not collected once as static data.

W2 (larger rollout groups). We ran GRPO with 16 unguided rollouts per task, matching the 16 rollouts per task of SPPO. It reaches 72.7% on the test set, versus 90.6% for SPPO (see hUh3 W2). The effect of skills is also not just more rollouts. The drop in all-fail groups appears only on hard tasks and not on easy or medium ones, which more unguided rollouts would not explain.

Q1. See W1 (cold-start SFT + GRPO) and W2 (GRPO with larger rollout groups).

Supplementary analysis: the w=1 ablation (referenced from hUh3 W5). Reviewer hUh3 asked why removing double-scoring (w=1) collapses training. Comparing the w=1 run with full SPPO (1.5B, means over steps 126-150, with steps 101-125 in parentheses):

  • Guided rollouts: success is 0.000 (0.064) vs. 0.796 (0.690), and their valid-action ratio falls to 0.002 (0.239) while unguided rollouts and full SPPO's guided rollouts stay above 0.99. Guided success is 0 from step 107 on. Unguided success is 0.507 vs. 0.829 and still rising.
  • Group shaping stops: the rescue rate falls to 0 and the all-fail fraction returns to the unguided level (about 0.21), vs. 0.03.
  • Validation success peaks at 0.617 (step 105) and then drops to 0.297.
  • Policy entropy rises from 1.0 (steps 1-25) to 6.3 (steps 126-150), while full SPPO stays at 1.0-1.2.

With w=1, guided samples get only negative-advantage updates in the skill context, so their successes are never reinforced there, which the (1-w) term prevents. This is a mismatch between the sampling and training distributions, not a failure of internalization. The guided trajectories are ones the skill-free policy would almost never sample, which causes output collapse. We do not log the probability of following skills directly, so we reword "the actor gradually stops acting on the skills" as "the policy in the skill context degrades until it no longer outputs valid actions". We will add these observations to the paper.

Supplementary analysis 2: group shaping and lagged skills (referenced from hUh3 W4 and W6). In an all-fail group every advantage is zero, so the task gives no gradient. Skill guidance turns it into an any-pass group, and one successful trajectory on a hard task shows the actor how to solve it. At step 0 the all-fail rate is 0.75 for the unguided rollouts and 0.50 for the full SPPO group, and we believe these rescues accumulate over training. This is not visible in the average reward of Fig. 2(b), since it comes from hard tasks that previously gave no signal. GRPO with 16 unguided rollouts reaches only 72.7% (see W2), so it is not just more samples. On the lagged-skill ablation, Fig. 4 shows that on hard tasks the current skill rescues 15.5% of all-fail groups and the lagged skill 12.4%, about 25% fewer, and the gap widens during training. This difference is the self-contrast signal for the generator, so the two are not homogeneous prompt perturbations.

Dear Reviewer,

Thank you for your constructive feedback. We have submitted a revised version of the paper in line with your comments. Every requested change is included except the 3-run experiments, which we did not have time to complete. We will run the main experiments with three seeds and add the mean and standard deviation in the next revision.

Best regards, The Authors

Reviewer_hUh3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SPPO, a GRPO-style method for long-horizon language agents that targets "gradient starvation", where all rollouts in a group fail identically and advantages vanish. For each task, a separately trained skill generator produces a short natural-language strategy hint from its current weights (s+s^+) and a hint from a KK-step-lagged snapshot (s−s^-). The actor collects 4 rollouts under each hint plus 8 unguided rollouts. Before the policy update, the hints are stripped. Skill-guided actions with non-negative advantage are scored under both the skill-free and skill-conditioned contexts (weight w=0.5w=0.5 each). Skill-guided actions with negative advantage are scored only under the skill-conditioned context, so the failure is attributed to the hint. The generator is first SFT'd on GPT-5.5-written skills distilled from reference walkthroughs, then trained with GRPO on the outcomes of its hints, with lagged-skill rollouts used only in the baseline. On ALFWorld (valid_seen), with no skills at inference, SPPO reaches 90.6% (Qwen2.5-1.5B) and 94.5% (Qwen2.5-7B), above the GiGPO numbers reported by Feng et al. (2025). Group shaping lowers the all-fail group fraction by about 5 points. Ablations show drops without lagged skills (85.9%) and without double-scoring (63.3%).

Critical strengths
  1. All-fail groups really do stall GRPO in long-horizon agent RL. Using hints only at rollout time, then stripping them so the deployed agent runs without any skill, is a clean way to inject guidance with no inference-time cost or dependency.

  2. Scoring positive and negative skill-guided actions under different contexts directly addresses the train/deploy mismatch that arises when guided data is used to train an unguided policy. The ablation shows the design matters a great deal (90.6% vs. 63.3% without double-scoring).

  3. SPPO beats the reported GiGPO results at both scales. The paper goes beyond final accuracy with diagnostics: the all-fail group fraction over training, guided vs. unguided reward, and per-difficulty any-win rates for current vs. lagged skills. The code is released.

Critical weaknesses
  1. The gains are confounded with privileged, external supervision that the baselines do not receive. The generator is SFT'd on skills written by GPT-5.5 while it was viewing the reference walkthroughs of 708 training games (App. B). The prompt asks for the receptacle "the walkthrough actually uses" to be listed first. The generator therefore learns ALFWorld-specific priors (where objects tend to be, which appliance performs each transformation) distilled from a frontier model with access to solutions. GRPO and GiGPO get none of this. The improvement may reflect injected domain knowledge rather than group shaping. This could be tested with GRPO plus frozen SFT-generator hints, or with an actor first SFT'd on the same GPT-5.5 data.

  2. The rollout and compute budgets do not match the baselines. SPPO uses 16 actor rollouts per task (8 unguided + 4 + 4), plus generator inference and training. No baseline is run with 16 unguided rollouts per task, even though the all-fail fraction depends directly on group size. Fig. 2(a) points to a setup mismatch: at step 0 the unguided SPPO rollouts and GRPO come from the same initial actor, yet their all-fail rates differ (~65% vs. ~80%). The generator's size and cost are not reported.

  3. The evaluation protocol cannot support differences of about 4 points.

    • All baseline numbers in Table 1 are copied from Feng et al. (2025) rather than rerun.
    • SPPO uses a single seed and the checkpoint with the best validation score; the baselines' selection protocol is unknown.
    • The test set has 128 games, so one game is 0.78 points. The margin over GiGPO (+3.9 at 1.5B, +3.7 at 7B) is about 5 games.
    • The per-category percentages imply different category compositions in the 1.5B and 7B evaluations (e.g., Look 11/12 vs. 9/10, Pick2 16/21 vs. 23/28; Cool 21/24 vs. 24/25). This contradicts the claim that all methods are scored on the same matched games.
    • Fig. 3 shows GiGPO and GRPO training curves, which implies reruns exist, but Table 1 does not use them.
  4. The mechanistic evidence is weaker than the claims. The contributions state that SPPO "eliminates gradient starvation", but the all-fail fraction only falls from about 65% to about 55% (about 5 points on average). Skill-guided rollouts earn about the same or slightly less reward than unguided ones (Fig. 2b). The current skill rescues 15.5% of hard tasks vs. 12.4% for the lagged skill. These modest effects make it unclear how they produce the large final gains. The claim that gains come from group shaping "rather than skills serving as better test-time inputs" is never tested: there is no evaluation of the actor with skills at test time, and no frozen-generator variant.

  5. The double-scoring ablation exposes an unexplained failure mode. Without double-scoring (w=1w=1), SPPO drops to 63.3%, about 9.5 points below plain GRPO (72.8%) in the same table. So scoring skill-guided successes only under the skill-free context actively harms the policy, which undercuts the "internalization" story. The explanation offered ("the actor gradually stops acting on the skills") is not supported by any measurement.

  6. Key ablations are missing. There is no GRPO with 16 unguided rollouts, no frozen (non-RL) generator, no alternative to attributing negatives to the skill, no sweep of ww, KK, or Ms/M0M_s/M_0, and no comparison to the cited concurrent skill-RL methods (SkillRL, Skill0, Skill1). The lagged-skill ablation (−4.7, single seed) is hard to interpret, because Fig. 4 shows lagged and current skills performing almost identically except on hard tasks.

  7. The off-policy treatment is not justified. Skill-guided actions are sampled from πθold(⋅∣x,H,s)\pi_{\theta_{old}}(\cdot\mid x,H,s) but scored with the skill-free ratio πθ(a∣x,H)/πθold(a∣x,H)\pi_\theta(a\mid x,H)/\pi_{\theta_{old}}(a\mid x,H). This is not an importance ratio with respect to the behavior policy, so the resulting gradient is biased in a way the paper does not analyze. The claim that negative-advantage terms "protect" the skill-free policy is also not demonstrated: those terms still update the shared weights under the skill-conditioned context. Related work on RL from off-policy guidance and hints (e.g., mixed-policy or prefix/hint-guided GRPO) is not discussed.

  8. The evaluation scope is narrow. Only ALFWorld valid_seen (in-distribution) is evaluated, and baselines on it are already near saturation (GiGPO 86–91%). There are no results on valid_unseen (out-of-distribution) or on WebShop or other agent benchmarks used by GiGPO. Generality beyond one simulated household environment is untested.

Questions for authors
  1. Can you add controls that separate group shaping from the extra supervision and compute? Specifically: (a) GRPO/GiGPO with 16 unguided rollouts per task; (b) GRPO whose rollouts receive hints from the frozen SFT generator, without generator RL; and (c) an actor first SFT'd on the same GPT-5.5/walkthrough-derived data, then trained with GRPO. If SPPO's advantage largely disappears under (b) or (c), the contribution would need reframing.

  2. Can you rerun all baselines under your exact protocol (same 128 games, same checkpoint-selection rule) with at least 3 seeds and report mean ± std? Please also explain why the per-category success rates imply different category compositions in the 1.5B and 7B evaluations, and whether the GiGPO/GRPO curves in Fig. 3 come from your own reruns.

  3. Why does removing double-scoring drop performance to 63.3%, well below plain GRPO (72.8%)? Can you show, for that variant, the skill-guided success rate and the actor's likelihood of following skills over training, to support the stated explanation?

  4. Do the gains hold on ALFWorld valid_unseen and on a second environment such as WebShop? What is the generator's model size, and what is the total training cost relative to GiGPO?

  5. How do you justify scoring skill-sampled actions with the skill-free ratio, given that the behavior policy was skill-conditioned? Can you provide evidence, for example the skill-free log-probability of negative-advantage skill-guided actions before and after updates, that negative-advantage attribution actually protects the standalone policy?

2 comments

Thank you for the careful and constructive review.

W1 (privileged supervision). We ran both controls you suggested (Qwen2.5-1.5B, ALFWorld, 150 steps). Results are on the test set (valid_seen, 128 games), versus 90.6% for full SPPO. (1) Frozen SFT generator, only the actor trained, same hints and 16 rollouts per task, evaluated without skills: 71.9% with SPPO's actor objective and 64.1% with standard guided GRPO. (2) Actor SFT on the same GPT-5.5/walkthrough data, then GRPO: 68.8% after SFT and 82.8% at the final step. SFT+RL was our original design, but we kept the base-model starting point of all RL baselines. So hints from the SFT generator alone do not account for the gain (19-27 points lower when frozen). Walkthrough-derived supervision is itself a strong signal, which we will state in the paper.

W2 (sample and compute budget). GRPO with 16 unguided rollouts per task (the same budget as SPPO's 8+4+4) reaches a best test-set result of 72.7%, versus 90.6% for SPPO, so extra rollouts alone do not close the gap. The all-fail reduction is not a group-size effect, since it appears only on hard tasks, not on easy or medium ones. On Fig. 2(a), the curves are 9-step moving averages, so the plotted starting point is not the raw value. At step 0 the true all-fail rates are 0.75 for SPPO's unguided rollouts, 0.75 for GRPO, and 0.50 for the full SPPO group, so the two setups start identically. The generator uses the same base model as the actor, and we will state this in the paper.

W3 (evaluation protocol).

  1. Seed: fixed to 0 for all main experiments and ablations, and not selected.
  2. The additional runs for the 3-run mean are still running. We will report the mean as soon as they finish.
  3. Baselines: we reran GiGPO and GRPO at 1.5B (83.6% and 75.0%), one run each. Our GiGPO rerun does not reach the published number, so Table 1 uses the published numbers.
  4. Category composition: you are right that the 1.5B and 7B evaluations do not use the same games. The test set is 128 draws from valid_seen's 140 games, and the game-list order before the draw depends on the machine. The 7B model was trained on two nodes and the 1.5B on one, so different games were drawn. The draw code is identical to GiGPO's. We will report 3-run results on a fixed, identical test set.
  5. Checkpoint selection: GiGPO does not state a rule, and its released script evaluates on the test set every 5 steps without saving checkpoints. We compare against its reported numbers, i.e. the baseline's best test-set result.

W4 (mechanism evidence). The gain depends on training the generator together with the actor (see W1). The mechanism of group shaping is in our reply to Reviewer Un7v (Supplementary analysis 2). We do not add skills at test time because every baseline ends up as a pure actor, and because our objective strengthens the actor's own ability, not its ability after seeing a skill.

W5 (collapse without double-scoring). The full analysis is in our reply to Reviewer Un7v (Supplementary analysis). With w=1, guided success falls to 0 and policy entropy rises from 1.0 to 6.3, while full SPPO stays stable.

W6 (missing ablations). GRPO with 16 unguided rollouts and the frozen generator are done (see W2 and W1). The sweeps over w, K and Ms/M0 and the alternatives for attributing negatives could not be finished in the rebuttal period. We will add them in the revision. SkillRL and Skill1 use skills at test time, so their tasks are not aligned with ours (see W4). Published SKILL0 and SkillRL numbers are in our reply to Reviewer RnEn (W1). The lagged-skill ablation is discussed in Supplementary analysis 2 of our reply to Reviewer Un7v. We will re-report the -4.7 ablation as a 3-run mean (see W3).

W7 (off-policy treatment). The derivation is in our reply to Reviewer RnEn (W5). In short, skill-free scoring of a successful guided trajectory is the skill-free policy gradient with each term reweighted by a positive likelihood ratio, and the remaining terms are on-policy in the skill context.

W8 (evaluation scope). The baselines are evaluated only on ALFWorld valid_seen, so we followed this setting. To test generality beyond ALFWorld, we are running 3-run WebShop experiments. They are still running, and we will report the results as soon as they finish.

Q1. We ran all three controls (W1, W2): GRPO with 16 unguided rollouts 72.7%, frozen generator 71.9% (SPPO objective) and 64.1% (guided GRPO), cold-start SFT then GRPO 82.8% at the final step (single run). Full SPPO: 90.6%.

Q2. See W3. Q3. See W5.

Q4. See W8 (valid_unseen, WebShop) and W2 (generator size).

Q5. The score comes from the environment reward, not from the skill. The skill only changes the sampling distribution, so that high-reward trajectories in the low-probability tail of the skill-free policy get sampled. The bias is by design (see W7). We do not log the skill-free log-probability of negative-advantage guided actions, so the w=1 measurements (W5) are indirect evidence.

Dear Reviewer,

Thank you for your constructive feedback. We have submitted a revised version of the paper in line with your comments. Every requested change is included except the 3-run experiments, which we did not have time to complete. We will run the main experiments with three seeds and add the mean and standard deviation in the next revision.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.