- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PHS extracts reusable WHEN/DO/PREVENTS procedures by contrasting successful reference rollouts (Kimi-K2.6) with failed learner rollouts. During training, selected procedures are added only as privileged context to re-score the learner's own unhinted rollouts. The token-level signal is combined with the GRPO advantage , either multiplicatively or additively (, ). Policies are evaluated without hints on AppWorld, EnterpriseOps, cross-environment transfer, BFCL and .
Critical strengths
- The idea is simple and cheap: re-scoring costs one extra teacher-forced pass, requires no hinted sampling, and uses no hints at test time.
- The paper includes useful controls: hint-only (OPSD), reassigned and generic procedures, and annealed vs. fixed .
- The evaluation is broad: in-domain, cross-environment, and two external benchmarks.
Critical weaknesses
- Incremental contribution. Privileged-context re-scoring (OPSD) and combining such signals with reward-derived credit (RLCSD) already exist. The main novelty is the reusable procedure catalog.
- Unmatched baselines. PHS benefits from strong-model signal that GRPO does not receive. The paper does not compare with simpler uses of the same signal, such as SFT on reference trajectories followed by GRPO, or hint-in-prompt training.
- Limited gains beyond AppWorld. The large gains appear mainly on AppWorld, whose procedures partly encode harness conventions. Improvements on EnterpriseOps pass@1, and BFCL are marginal.
Questions for authors
- How does PHS compare with SFT-on-references GRPO under the same strong-model access?
- How does the hint-only control perform with ?
4 comments
Part 2/2
This part addresses the hint-only control (Question 2), novelty (Weakness 1) and gains beyond AppWorld (Weakness 3).
Question 2: matched-coefficient hint-only control
The matched control uses 0.1Δ instead of A+0.1Δ. Keeping outcome feedback raises AW test pass@4 from 34.5 to 74.4 and EO test from 0.0 to 40.4. Coefficient-1 OPSD also trails PHS in-domain (60.12 on AW; 4.88 on EO). At matched hint strength, outcome feedback is crucial to the strong in-domain result in these runs. PHS is doing more than distilling the hint alone.
EO hint-only training reward falls from 0.55 to below 0.06 around rollout 45, followed by length-limit responses. AW-trained OPSD(0.1) still transfers to EO (30.7, between PHS-M's 31.1 and PHS-A's 30.3). The matched controls are in Appendix F.10 (Table 27), extending Sections 3.4 and 5.5.1 (Table 5).
Weakness 1: contribution relative to OPSD and RLCSD
PHS contributes reusable procedural supervision: 12 AW and 20 EO procedures mined once from reference–learner disagreements, with no task-specific answers. Their content relevance matters: matched procedures outperform reassigned and generic procedures (Section 5.5.1, Table 5). Gains transfer across environments and benchmarks (Sections 5.2–5.4). The contribution is turning reusable procedural knowledge into supervision that works with outcome rewards. The new baselines show it beats hint-only learning and adds to imitation: after SFT, PHS beats GRPO (90.5 vs 87.1 on AW test). Which individual procedures drive transfer is not established (Appendix F.12).
We build on privileged-context scoring and reward-based learning. RLCSD already preserves outcome-credit sign and constructs positive/negative contexts from learner rollouts. PHS instead selects reusable procedures and compares hinted with bare scores. We clarified this distinction in Section 2; we have not directly evaluated RLCSD (Sections 3.3–3.5).
Weakness 3: gains beyond AppWorld
The three-run results show gains on EO, BFCL and τ³, from both AW-trained and EO-trained policies. The main RL methods use the same planned budget within each environment, chosen for training-task coverage with limited repetition, rather than selecting checkpoints using held-out scores.
Three-run replication; trained on AW.
| Method | AW dev | AW test | AW chall. | EO test | BFCL NL | BFCL MT | BFCL L | τ³ R | τ³ A | τ³ T |
|---|---|---|---|---|---|---|---|---|---|---|
| Base | 32.9±0.9 | 40.8±2.9 | 16.9±1.1 | 26.6±1.0 | 81.9±0.4 | 59.6±0.4 | 79.7±0.3 | 82.9±0.9 | 70.4±3.5 | 69.1±0.4 |
| GRPO | 48.7±2.6 | 58.8±1.9 | 33.4±2.1 | 25.5±0.6 | 81.9±0.2 | 59.2±0.8 | 79.7±0.2 | 79.8±1.3 | 76.8±1.0 | 70.0±2.1 |
| PHS-M | 67.0±1.4 | 78.3±1.9 | 48.9±2.2 | 30.7±0.4 | 82.2±0.7 | 60.4±0.1 | 80.5±0.2 | 85.5±1.2 | 80.1±0.9 | 70.9±0.9 |
| PHS-A | 66.7±2.6 | 76.8±2.3 | 48.6±2.5 | 30.6±0.6 | 82.6±0.4 | 60.6±0.5 | 80.1±0.1 | 85.8±2.3 | 82.7±1.0 | 72.8±1.0 |
Thank you for the detailed rebuttal and the additional experiments. The new baselines (expert SFT→GRPO, SFT→PHS, hinted GRPO) and the matched-coefficient hint-only control directly address my main concerns about unmatched baselines, and the three-run replication makes the reported gains more convincing. I also appreciate the candid acknowledgment that SFT alone remains stronger on AppWorld and that RLCSD was not directly evaluated.
The responses resolve most of my questions. Since the novelty relative to OPSD/RLCSD remains incremental and the gains on the external benchmarks are still modest, I believe my current overall rating is fair and will keep it.
Thank you for carefully considering our rebuttal and additional experiments. We are glad these addressed your main concerns and strengthened the evidence.
We agree that standalone SFT is stronger on AppWorld. The distinction we would emphasize is the trade-off in external performance. AppWorld-trained SFT reduces BFCL multi-turn accuracy and τ³ telecom success to 49.7% and 40.4%, versus Base’s 59.6% and 69.1%. The decline is much sharper after EnterpriseOps SFT, reaching 12.1% and 3.3%, and subsequent GRPO does not recover these losses. PHS trained directly from Base avoids these declines in both settings, while SFT→PHS also improves over SFT→GRPO in-domain. Together, these results help clarify the complementary value of procedural supervision.
Your feedback helped us strengthen the comparisons and presentation, and we sincerely appreciate your time and constructive engagement.
Part 1/2
Thank you for your thoughtful review.
This part addresses expert SFT and hints during training rollouts (Weakness 2 / Question 1).
The setup and legend apply to both parts. GRPO/PHS replication uses three independent training runs; new baselines and ablations use one run due to rebuttal time limits. All policy evaluations are hint-free.
AW/EO = AppWorld/EnterpriseOps, pass@4. BFCL NL/MT/L = non-live/multi-turn/live accuracy; τ³ R/A/T = retail/airline/telecom success. Values: % (mean±SD). PHS-M/A = multiplicative/additive. The other environment is transfer. Control SDs are across evaluations. In control tables, GRPO and PHS rows are the submitted single run; three-run means are in the replication tables.
Weakness 2 / Question 1: simpler uses of expert guidance
We ran expert SFT→GRPO, SFT→PHS, GRPO with hints during training, and hint-only OPSD at 0.1.
Baselines and controls trained on AW.
Baselines and controls trained on EO.
Original OPSD (coefficient 1), Tables 1–2, 4 and 18. BFCL uses pass@4 ± SE here; AW/EO and τ³ retain the metrics above.
Takeaway: PHS complements expert SFT and preserves external skills when trained from Base. From the same SFT checkpoint, PHS-A beats GRPO on AW test (90.5 vs 87.1), EO test (40.2 vs 38.1) and AW→EO transfer (41.4 vs 36.9). SFT alone remains stronger on AW than PHS from Base (85.4 vs 78.7).
SFT trades external performance for strong AW and EO-transfer scores: AW-trained SFT scores 49.7 on BFCL MT and 40.4 on τ³ telecom, versus Base's 59.6 and 69.1. Neither GRPO nor PHS after SFT restores them (table above); PHS-M from the base reaches 60.3 and 71.9. PHS therefore offers a complementary route that preserves these skills (Section 5.5.3; Appendix F.8, Table 25).
Weakness 2 / Question 1 continued: hints during training rollouts
Hinted GRPO samples with procedures, then is evaluated without them. AW test rises from 59.7 to 64.6, below 78.7 for PHS-M; EO reaches 31.1, below GRPO's 33.0 and PHS-A's 40.4. Here the same guidance works better as a credit-assignment signal: PHS uses it only for re-scoring, preserving the training/evaluation sampling context (Section 5.5.2, beside Table 7; Appendix F.10).