acceptodds
Under review as a conference paper at ICLR 2027

Unlocking Latent Tool-Use Capabilities with Procedural Hint Supervision

Abstract

Reinforcement learning with outcome-based rewards can improve tool use, but a model may know how to execute individual actions without reliably organizing them into successful procedures. Sparse task-level feedback provides little guidance on how to correct these procedural failures. We introduce Procedural Hint Supervision (PHS), which uses reusable guidance to improve credit assignment and help models apply their existing capabilities more reliably. PHS extracts procedures by comparing successful attempts from a more capable model with failed attempts from the model being trained. During training, selected guidance is used to re-score the model's own trajectories, and differences between hinted and unhinted token log probabilities shape token-level advantages in two extended formulations of Group Relative Policy Optimization (GRPO). The trained policies are evaluated without hints on held-out sets. Trained separately on AppWorld and EnterpriseOps, both formulations improve held-out success and reciprocal transfer, outperforming vanilla GRPO. Across four seeds, the largest pass@4 gains over GRPO are 19.1 percentage points on AppWorld and 7.4 percentage points on EnterpriseOps. The same AppWorld policy gains 14.9 percentage points on its harder test-challenge split. Without targeted benchmark training, gains over GRPO reach 4.1 percentage points in BFCL multi-turn pass@4 for an EnterpriseOps-trained policy and 10.4 percentage points in mean pass across retail, airline, and telecom for an AppWorld-trained policy. Together, the in-domain test sets and transfer results suggest that procedural guidance can unlock latent capabilities, turning existing tool-use skills into reliable behavior that generalizes across tasks, tools, and environments.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, language model agents, tool use, credit assignment, procedural knowledge, privileged information, generalization

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ygjgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PHS extracts reusable WHEN/DO/PREVENTS procedures by contrasting successful reference rollouts (Kimi-K2.6) with failed learner rollouts. During training, selected procedures are added only as privileged context to re-score the learner's own unhinted rollouts. The token-level signal Δit=log⁡πold(yit∣Hith)−log⁡πold(yit∣Hit)\Delta_{it} = \log \pi_{\text{old}}(y_{it} \mid H^h_{it}) - \log \pi_{\text{old}}(y_{it} \mid H_{it}) is combined with the GRPO advantage AiA_i, either multiplicatively or additively (A^it=Ai+βΔit\hat{A}_{it} = A_i + \beta \Delta_{it}, β=0.1\beta = 0.1). Policies are evaluated without hints on AppWorld, EnterpriseOps, cross-environment transfer, BFCL and τ3\tau^3.

Critical strengths
  • The idea is simple and cheap: re-scoring costs one extra teacher-forced pass, requires no hinted sampling, and uses no hints at test time.
  • The paper includes useful controls: hint-only (OPSD), reassigned and generic procedures, and annealed vs. fixed λ\lambda.
  • The evaluation is broad: in-domain, cross-environment, and two external benchmarks.
Critical weaknesses
  1. Incremental contribution. Privileged-context re-scoring (OPSD) and combining such signals with reward-derived credit (RLCSD) already exist. The main novelty is the reusable procedure catalog.
  2. Unmatched baselines. PHS benefits from strong-model signal that GRPO does not receive. The paper does not compare with simpler uses of the same signal, such as SFT on reference trajectories followed by GRPO, or hint-in-prompt training.
  3. Limited gains beyond AppWorld. The large gains appear mainly on AppWorld, whose procedures partly encode harness conventions. Improvements on EnterpriseOps pass@1, τ3\tau^3 and BFCL are marginal.
Questions for authors
  1. How does PHS compare with SFT-on-references →\rightarrow GRPO under the same strong-model access?
  2. How does the hint-only control perform with β=0.1\beta = 0.1?
4 comments

Part 1/2

Thank you for your thoughtful review.

This part addresses expert SFT and hints during training rollouts (Weakness 2 / Question 1).

The setup and legend apply to both parts. GRPO/PHS replication uses three independent training runs; new baselines and ablations use one run due to rebuttal time limits. All policy evaluations are hint-free.

AW/EO = AppWorld/EnterpriseOps, pass@4. BFCL NL/MT/L = non-live/multi-turn/live accuracy; τ³ R/A/T = retail/airline/telecom success. Values: % (mean±SD). PHS-M/A = multiplicative/additive. The other environment is transfer. Control SDs are across evaluations. In control tables, GRPO and PHS rows are the submitted single run; three-run means are in the replication tables.

Weakness 2 / Question 1: simpler uses of expert guidance

We ran expert SFT→GRPO, SFT→PHS, GRPO with hints during training, and hint-only OPSD at 0.1.

Baselines and controls trained on AW.

MethodAW devAW testAW chall.EO testBFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base32.9±0.940.8±2.916.9±1.126.6±1.081.9±0.459.6±0.479.7±0.382.9±0.970.4±3.569.1±0.4
GRPO48.7±6.059.7±2.034.2±0.125.0±0.681.7±0.159.7±0.279.7±0.281.3±1.676.0±1.672.2±2.1
PHS-M68.0±2.278.7±1.749.1±0.731.1±2.482.5±0.260.3±0.380.7±0.486.6±2.079.3±1.971.9±1.4
PHS-A64.0±3.074.4±1.145.9±1.230.3±0.882.3±0.260.0±0.680.2±0.288.4±2.683.5±1.973.7±0.0
SFT83.3±3.485.4±1.066.9±0.936.1±1.770.5±0.449.7±0.180.0±0.480.7±1.677.0±1.240.4±0.7
SFT→GRPO83.8±2.287.1±1.274.4±0.636.9±1.372.0±0.749.9±0.380.6±0.380.9±0.876.5±1.045.4±1.3
SFT→PHS-M95.6±1.888.4±1.074.5±0.737.5±1.968.4±0.250.9±1.180.3±0.284.4±0.880.0±2.340.8±0.5
SFT→PHS-A94.7±2.590.5±0.572.2±1.341.4±1.173.2±0.451.1±0.580.7±0.086.4±1.181.5±1.943.9±2.6
Hinted GRPO59.6±1.464.6±0.636.8±0.726.0±1.381.4±0.260.3±0.279.5±0.182.2±2.472.5±1.968.9±2.1
OPSD (0.1)15.4±3.034.5±2.313.2±0.730.7±0.882.2±0.158.8±0.379.1±0.181.4±2.176.0±3.767.5±1.6

Baselines and controls trained on EO.

MethodEO testAW devAW testAW chall.BFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base26.6±1.032.9±0.940.8±2.916.9±1.181.9±0.459.6±0.479.7±0.382.9±0.970.4±3.569.1±0.4
GRPO33.0±0.426.3±0.037.8±3.715.9±0.480.1±0.459.8±0.879.2±0.282.2±2.874.5±1.067.3±1.8
PHS-M37.5±1.437.7±4.650.3±1.723.4±1.083.8±0.464.4±0.680.4±0.184.0±2.084.0±4.071.1±1.0
PHS-A40.4±1.738.2±1.748.2±1.221.8±0.883.5±0.263.1±0.980.1±0.287.0±2.780.3±0.973.7±2.0
SFT30.1±1.011.0±1.722.6±1.38.9±0.742.4±0.112.1±0.675.9±0.30.0±0.02.0±1.63.3±0.8
SFT→GRPO38.1±1.018.0±1.727.4±1.311.8±0.941.8±0.111.5±0.376.1±0.10.4±0.51.0±1.23.1±1.1
SFT→PHS-M38.1±1.233.3±1.439.1±2.421.4±1.242.4±0.28.7±0.276.9±0.20.0±0.01.5±1.02.4±0.8
SFT→PHS-A40.2±1.926.8±5.235.6±2.615.9±2.042.5±0.210.6±0.776.4±0.10.9±0.75.0±1.24.2±1.5
Hinted GRPO31.1±1.024.6±3.237.6±1.714.4±0.582.9±0.259.5±0.779.9±0.283.3±2.676.0±2.866.0±0.8
OPSD (0.1)0.0±0.00.4±0.97.1±1.91.9±0.484.1±0.334.1±0.878.3±0.04.2±2.30.5±1.00.9±0.7

Original OPSD (coefficient 1), Tables 1–2, 4 and 18. BFCL uses pass@4 ± SE here; AW/EO and τ³ retain the metrics above.

TrainingAW devAW testAW chall.EO testBFCL NL@4BFCL MT@4BFCL L@4τ³ Rτ³ Aτ³ T
AW40.35±3.7960.12±1.7525.96±0.8428.71±2.0586.29±1.1471.50±1.5882.90±1.0283.11±0.4478.40±1.8267.11±1.13
EO27.19±1.0143.90±1.3218.21±0.674.88±0.7588.15±1.0775.25±1.5183.64±1.0167.11±4.2137.50±1.9157.68±1.10

Takeaway: PHS complements expert SFT and preserves external skills when trained from Base. From the same SFT checkpoint, PHS-A beats GRPO on AW test (90.5 vs 87.1), EO test (40.2 vs 38.1) and AW→EO transfer (41.4 vs 36.9). SFT alone remains stronger on AW than PHS from Base (85.4 vs 78.7).

SFT trades external performance for strong AW and EO-transfer scores: AW-trained SFT scores 49.7 on BFCL MT and 40.4 on τ³ telecom, versus Base's 59.6 and 69.1. Neither GRPO nor PHS after SFT restores them (table above); PHS-M from the base reaches 60.3 and 71.9. PHS therefore offers a complementary route that preserves these skills (Section 5.5.3; Appendix F.8, Table 25).

Weakness 2 / Question 1 continued: hints during training rollouts

Hinted GRPO samples with procedures, then is evaluated without them. AW test rises from 59.7 to 64.6, below 78.7 for PHS-M; EO reaches 31.1, below GRPO's 33.0 and PHS-A's 40.4. Here the same guidance works better as a credit-assignment signal: PHS uses it only for re-scoring, preserving the training/evaluation sampling context (Section 5.5.2, beside Table 7; Appendix F.10).

Part 2/2

This part addresses the hint-only control (Question 2), novelty (Weakness 1) and gains beyond AppWorld (Weakness 3).

Question 2: matched-coefficient hint-only control

The matched control uses 0.1Δ instead of A+0.1Δ. Keeping outcome feedback raises AW test pass@4 from 34.5 to 74.4 and EO test from 0.0 to 40.4. Coefficient-1 OPSD also trails PHS in-domain (60.12 on AW; 4.88 on EO). At matched hint strength, outcome feedback is crucial to the strong in-domain result in these runs. PHS is doing more than distilling the hint alone.

EO hint-only training reward falls from 0.55 to below 0.06 around rollout 45, followed by length-limit responses. AW-trained OPSD(0.1) still transfers to EO (30.7, between PHS-M's 31.1 and PHS-A's 30.3). The matched controls are in Appendix F.10 (Table 27), extending Sections 3.4 and 5.5.1 (Table 5).

Weakness 1: contribution relative to OPSD and RLCSD

PHS contributes reusable procedural supervision: 12 AW and 20 EO procedures mined once from reference–learner disagreements, with no task-specific answers. Their content relevance matters: matched procedures outperform reassigned and generic procedures (Section 5.5.1, Table 5). Gains transfer across environments and benchmarks (Sections 5.2–5.4). The contribution is turning reusable procedural knowledge into supervision that works with outcome rewards. The new baselines show it beats hint-only learning and adds to imitation: after SFT, PHS beats GRPO (90.5 vs 87.1 on AW test). Which individual procedures drive transfer is not established (Appendix F.12).

We build on privileged-context scoring and reward-based learning. RLCSD already preserves outcome-credit sign and constructs positive/negative contexts from learner rollouts. PHS instead selects reusable procedures and compares hinted with bare scores. We clarified this distinction in Section 2; we have not directly evaluated RLCSD (Sections 3.3–3.5).

Weakness 3: gains beyond AppWorld

The three-run results show gains on EO, BFCL and τ³, from both AW-trained and EO-trained policies. The main RL methods use the same planned budget within each environment, chosen for training-task coverage with limited repetition, rather than selecting checkpoints using held-out scores.

Three-run replication; trained on AW.

MethodAW devAW testAW chall.EO testBFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base32.9±0.940.8±2.916.9±1.126.6±1.081.9±0.459.6±0.479.7±0.382.9±0.970.4±3.569.1±0.4
GRPO48.7±2.658.8±1.933.4±2.125.5±0.681.9±0.259.2±0.879.7±0.279.8±1.376.8±1.070.0±2.1
PHS-M67.0±1.478.3±1.948.9±2.230.7±0.482.2±0.760.4±0.180.5±0.285.5±1.280.1±0.970.9±0.9
PHS-A66.7±2.676.8±2.348.6±2.530.6±0.682.6±0.460.6±0.580.1±0.185.8±2.382.7±1.072.8±1.0

Thank you for the detailed rebuttal and the additional experiments. The new baselines (expert SFT→GRPO, SFT→PHS, hinted GRPO) and the matched-coefficient hint-only control directly address my main concerns about unmatched baselines, and the three-run replication makes the reported gains more convincing. I also appreciate the candid acknowledgment that SFT alone remains stronger on AppWorld and that RLCSD was not directly evaluated.

The responses resolve most of my questions. Since the novelty relative to OPSD/RLCSD remains incremental and the gains on the external benchmarks are still modest, I believe my current overall rating is fair and will keep it.

Thank you for carefully considering our rebuttal and additional experiments. We are glad these addressed your main concerns and strengthened the evidence.

We agree that standalone SFT is stronger on AppWorld. The distinction we would emphasize is the trade-off in external performance. AppWorld-trained SFT reduces BFCL multi-turn accuracy and τ³ telecom success to 49.7% and 40.4%, versus Base’s 59.6% and 69.1%. The decline is much sharper after EnterpriseOps SFT, reaching 12.1% and 3.3%, and subsequent GRPO does not recover these losses. PHS trained directly from Base avoids these declines in both settings, while SFT→PHS also improves over SFT→GRPO in-domain. Together, these results help clarify the complementary value of procedural supervision.

Your feedback helped us strengthen the comparisons and presentation, and we sincerely appreciate your time and constructive engagement.

Reviewer_b2VBOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Procedural Hint Supervision (PHS), which mines reusable WHEN/DO/PREVENTS procedures from reference–learner disagreements and uses them only during training to re-score the learner’s own rollouts, combining token-level hint scores with verifier-based GRPO advantages; experiments on AppWorld and EnterpriseOps show improved held-out success, cross-environment transfer, and some external BFCL/τau3 gains without test-time hints, with controls suggesting that task-matched procedures organize latent capabilities rather than teach new ones.

Critical strengths

The method is lightweight and practical, requiring only one-time procedure mining and one extra teacher-forced scoring pass per rollout without test-time hints.

The evaluation is broad, covering held-out, cross-environment, external benchmark, and reward/procedure control experiments that collectively support the central claim.

The paper provides useful evidence that task-matched procedural hints, rather than generic or reassigned text, drive gains and that prompt-time hints alone are weaker than PHS training.

Critical weaknesses
  • The multiplicative objective in Eq. (3) appears degenerate because mitm_{it} is always positive, making sign(mit)=1\mathrm{sign}(m_{it})=1 and leaving A^it=Ai\widehat A_{it}=A_i, which contradicts the reported PHS-Multiplicative gains and must be clarified or corrected.

  • Evidence linking specific procedures to external gains is qualitative, with no per-procedure removal, matched-budget retraining, or token-level re-scoring on external trajectories.

  • Each trained arm uses only one training run, with four seeds only for evaluation, so training stability and statistical significance are not established.

  • The OPSD baseline uses coefficient 1 while PHS-Additive uses β=0.1\beta=0.1, and the multiplicative variant adds clipping, normalization, annealing, and tool-output placeholder effects, so the comparison does not isolate the proposed hint–reward mechanism.

Questions for authors

see weakness

4 comments

Part 1/2

Thank you for your thoughtful review.

This part addresses Eq. 3 (Weakness 1), external-trace analysis (Weakness 2) and independent runs (Weakness 3).

The setup and legend apply to both parts. GRPO/PHS replication uses three independent training runs; new baselines and ablations use one run due to rebuttal time limits. All policy evaluations are hint-free.

AW/EO = AppWorld/EnterpriseOps, pass@4. BFCL NL/MT/L = non-live/multi-turn/live accuracy; τ³ R/A/T = retail/airline/telecom success. Values: % (mean±SD). PHS-M/A = multiplicative/additive. The other environment is transfer. Control SDs are across evaluations. In control tables, GRPO and PHS rows are the submitted single run; three-run means are in the replication tables.

Weakness 1: Eq. 3 uses stop-gradient, not sign

The multiplier is not replaced by its sign. In Eq. 3, sg means stop-gradient: A^it=Aisg⁡(mit)\widehat A_{it}=A_i\operatorname{sg}(m_{it}), where cit=clip(exp⁡(sign(Ai)Δit),0.8,1.2)c_{it}=\mathrm{clip}(\exp(\mathrm{sign}(A_i)\Delta_{it}),0.8,1.2) and mit=1−λ+λcit/cˉim_{it}=1-\lambda+\lambda c_{it}/\bar c_i.

For 0≤λ≤10\le\lambda\le1, 1−λ/3≤mit≤1+λ/21-\lambda/3\le m_{it}\le1+\lambda/2, so the advantage's sign is preserved while its magnitude varies by token. A positive advantage emphasizes hint-supported tokens; a negative one emphasizes blame on tokens the hint argues against. Both scores come from the sampling snapshot before optimization; stored weights remain fixed during the update. Setting λ=0, constant clipped weights over the entire normalization set, or A=0 recovers the GRPO advantage. Uniform assistant weights alone do not suffice if tool-output weights differ. Stop-gradient fixes the weights; it does not remove their influence on which tokens receive larger gradient contributions. We clarified the notation in Section 3.3.

Weakness 2: external-trace analysis

We ran the token-level re-scoring you list, on external traces fixed before analysis, with matched and mismatched procedures and blind adherence judgments. The evidence:

  • Procedure content matters in training (Table 5, EO pass@4): matched 40.43 > reassigned 33.01 = GRPO 33.01 > generic 29.88.
  • External gains are measured (Appendix E, Table 14, EO-trained): BFCL parallel 73.00 → 83.00 and τ³ airline 74.50 → 84.00 for PHS-M over GRPO.
  • Re-scoring (Appendix F.12): Appendix F.12 reports a larger overall re-scoring shift on τ³ traces for PHS than GRPO (0.6 vs 0.02). We did not detect an additional effect specific to matched procedures over mismatched procedures; the corresponding confidence intervals include zero. Blind adherence ratings are within 0.03 of GRPO.

Takeaway: procedure content matters in training and the external gains are measured, but the probe does not tie one procedure to one external gain. We now present those links as hypotheses (Section 6, Appendix E). We have not run per-procedure removal with retraining.

Weakness 3: independent training runs

We added two training seeds per method to the submitted run. The original four seeds were repeated evaluations. The main RL methods use the same planned budget within each environment, chosen for training-task coverage with limited repetition, rather than selecting checkpoints using held-out scores.

Three-run replication; trained on AW.

MethodAW devAW testAW chall.EO testBFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base32.9±0.940.8±2.916.9±1.126.6±1.081.9±0.459.6±0.479.7±0.382.9±0.970.4±3.569.1±0.4
GRPO48.7±2.658.8±1.933.4±2.125.5±0.681.9±0.259.2±0.879.7±0.279.8±1.376.8±1.070.0±2.1
PHS-M67.0±1.478.3±1.948.9±2.230.7±0.482.2±0.760.4±0.180.5±0.285.5±1.280.1±0.970.9±0.9
PHS-A66.7±2.676.8±2.348.6±2.530.6±0.682.6±0.460.6±0.580.1±0.185.8±2.382.7±1.072.8±1.0

Three-run replication; trained on EO.

MethodEO testAW devAW testAW chall.BFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base26.6±1.032.9±0.940.8±2.916.9±1.181.9±0.459.6±0.479.7±0.382.9±0.970.4±3.569.1±0.4
GRPO33.2±0.534.4±7.042.6±4.218.0±1.881.0±1.060.3±0.579.7±0.581.5±1.875.8±1.368.1±1.6
PHS-M37.5±2.340.6±3.651.4±1.024.2±0.784.1±0.363.5±0.880.4±0.484.1±0.982.0±1.871.1±1.6
PHS-A38.7±1.836.3±9.246.7±6.220.6±5.483.5±0.461.6±1.580.5±0.486.2±1.481.6±2.571.4±2.0

Takeaway: the PHS advantage holds when policies are independently retrained: both variants have higher mean pass@4 than GRPO on all eight held-out and cross-environment comparisons. The largest replicated test gains are 19.5 percentage points on AW (78.3 vs 58.8) and 5.5 percentage points on EO (38.7 vs 33.2). SD is across training runs; Base reports variation across evaluations. The full table is in Appendix F.7 (Table 24); it extends Sections 5.1–5.4 and Tables 1–4.

Part 2/2

This part addresses the hint coefficient, normalization, weighting and schedule controls (Weakness 4).

Weakness 4: matched coefficient and component controls

We trained hint-only OPSD at coefficient 0.1, assistant-only normalization and uniform trajectory weights. The tables combine all results, including held-out, transfer and external evaluations, so common baseline rows appear once. “Assistant-only” and “Uniform weights” are PHS-M controls.

Controls trained on AW.

MethodAW devAW testAW chall.EO testBFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base32.9±0.940.8±2.916.9±1.126.6±1.081.9±0.459.6±0.479.7±0.382.9±0.970.4±3.569.1±0.4
GRPO48.7±6.059.7±2.034.2±0.125.0±0.681.7±0.159.7±0.279.7±0.281.3±1.676.0±1.672.2±2.1
OPSD (0.1)15.4±3.034.5±2.313.2±0.730.7±0.882.2±0.158.8±0.379.1±0.181.4±2.176.0±3.767.5±1.6
PHS-M68.0±2.278.7±1.749.1±0.731.1±2.482.5±0.260.3±0.380.7±0.486.6±2.079.3±1.971.9±1.4
PHS-A64.0±3.074.4±1.145.9±1.230.3±0.882.3±0.260.0±0.680.2±0.288.4±2.683.5±1.973.7±0.0
Assistant-only65.8±1.078.9±0.853.4±1.130.5±1.981.9±0.259.9±0.380.3±0.286.2±1.983.0±1.270.8±1.1
Uniform weights56.6±4.164.9±2.133.9±1.130.1±1.082.5±0.061.2±0.180.4±0.183.8±1.780.0±0.072.4±1.5

Controls trained on EO.

MethodEO testAW devAW testAW chall.BFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base26.6±1.032.9±0.940.8±2.916.9±1.181.9±0.459.6±0.479.7±0.382.9±0.970.4±3.569.1±0.4
GRPO33.0±0.426.3±0.037.8±3.715.9±0.480.1±0.459.8±0.879.2±0.282.2±2.874.5±1.067.3±1.8
OPSD (0.1)0.0±0.00.4±0.97.1±1.91.9±0.484.1±0.334.1±0.878.3±0.04.2±2.30.5±1.00.9±0.7
PHS-M37.5±1.437.7±4.650.3±1.723.4±1.083.8±0.464.4±0.680.4±0.184.0±2.084.0±4.071.1±1.0
PHS-A40.4±1.738.2±1.748.2±1.221.8±0.883.5±0.263.1±0.980.1±0.287.0±2.780.3±0.973.7±2.0
Assistant-only35.7±2.335.5±1.748.2±1.321.2±0.485.0±0.262.7±0.680.5±0.183.1±0.878.5±2.570.2±1.0
Uniform weights34.2±1.239.5±1.844.6±1.618.3±0.982.2±0.460.9±0.880.4±0.287.3±1.782.5±1.071.1±1.6

Original OPSD (coefficient 1), Tables 1–2, 4 and 18. BFCL uses pass@4 ± SE here; AW/EO and τ³ retain the metrics above.

TrainingAW devAW testAW chall.EO testBFCL NL@4BFCL MT@4BFCL L@4τ³ Rτ³ Aτ³ T
AW40.35±3.7960.12±1.7525.96±0.8428.71±2.0586.29±1.1471.50±1.5882.90±1.0283.11±0.4478.40±1.8267.11±1.13
EO27.19±1.0143.90±1.3218.21±0.674.88±0.7588.15±1.0775.25±1.5183.64±1.0167.11±4.2137.50±1.9157.68±1.10

Takeaway: hint-only training at either tested coefficient falls short of PHS in-domain, and substantial gains remain with assistant-only normalization. The matched control uses 0.1Δ instead of A+0.1Δ. Keeping outcome feedback raises AW test pass@4 from 34.5 to 74.4 and EO test from 0.0 to 40.4. Coefficient-1 OPSD also trails PHS in-domain (60.12 on AW; 4.88 on EO). PHS is doing more than distilling the hint alone.

EO hint-only training reward falls from 0.55 to below 0.06 around rollout 45, followed by length-limit responses. AW-trained OPSD(0.1) still transfers to EO (30.7, between PHS-M's 31.1 and PHS-A's 30.3). The matched controls are in Appendix F.10 (Table 27), extending Sections 3.4 and 5.5.1 (Table 5).

Removing tool outputs from normalization retains a large gain: AW split means are 66.0 for assistant-only, 65.3 for original PHS-M and 47.5 for GRPO. Uniform weights reach 51.8. On EO the corresponding scores are 35.7, 37.5, 33.0 and 34.2. The gain therefore does not require tool-output placeholders, and retaining token-specific credit is more effective than weighting whole trajectories alone. This supports the proposed credit-assignment mechanism, although these controls do not decompose the gain into independent causal shares.

The schedule comparison in Appendix D, Table 13 gives 35.94±1.10 EO pass@4 with fixed λ versus 37.50±1.42 with annealing; both exceed GRPO. Annealing is therefore not necessary for a gain in this comparison. Clipping was not separately ablated, and we do not attribute the difference between the two PHS variants solely to sign preservation. The new controls are in Section 5.5.1 and Appendix F.9–F.10 (Tables 26–27), and the budgets in Appendix B.6.

Thank you for your reply.

After considering the opinions of the other reviewers, I will lower the score.

Thank you for your follow-up and for considering the broader discussion. Could you clarify which concerns from your review or the other reviews remain unresolved after our responses and revised manuscript?

In direct response to the concerns you raised, we clarified the stop-gradient notation, added three independent training runs per main method, and provided matched-coefficient and normalization controls. We also conducted external-trace analyses to investigate the link between specific procedures and transfer gains, and narrowed our interpretation to reflect what those analyses support.

We would appreciate your perspective on where this evidence still falls short, including any concerns raised by the other reviewers that you find particularly consequential. This would help us respond directly and clarify any remaining points during the discussion. To note, one reviewer engaged positively with our rebuttal while the others are yet too.

Reviewer_SwPROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates the difficulty language-model agents face in reliably organizing individual tool-use actions into successful multi-step procedures. The authors propose Procedural Hint Supervision (PHS), which extracts reusable procedural guidance by comparing successful trajectories from a stronger reference model with failed trajectories from the learner.

During training, the learner first generates on-policy trajectories without hints. A frozen snapshot of the learner then rescores the same sampled tokens with and without selected procedural hints. The resulting token-level log-probability differences are combined with outcome-based GRPO advantages through two objectives, PHS-Multiplicative and PHS-Additive. The trained policy operates without procedural hints at inference time.

The authors train Qwen3.6-35B-A3B separately on AppWorld and EnterpriseOps and evaluate the resulting policies on held-out tasks, bidirectional cross-environment transfer, τ³, and BFCL. On AppWorld Test, PHS-Multiplicative improves pass@4 from 59.67% under GRPO to 78.72%. On EnterpriseOps Test, PHS-Additive improves pass@4 from 33.01% to 40.43%. The paper also reports improvements in cross-environment transfer and consistency on external tasks.

Critical strengths

1. Clear performance improvements on multi-step tool-use tasks.

The paper demonstrates substantial improvements over GRPO across two different tool-use environments. On AppWorld Test, PHS-Multiplicative improves Pass@4 from 59.67% to 78.72%, and from 34.17% to 49.10% on Challenge. On EnterpriseOps Test, PHS-Additive improves Pass@4 from 33.01% to 40.43%. The improvements in Pass@1 on AppWorld further suggest that the gains are not limited to multiple-attempt success rates. Importantly, the trained models require neither procedural hints nor additional models at inference time. These results provide convincing evidence of the practical effectiveness of PHS on the evaluated tasks.

2. Broad experimental coverage with useful transfer evaluations and ablations.

The authors train models independently on AppWorld and EnterpriseOps and evaluate bidirectional cross-environment transfer without target-environment training or inference-time hints. PHS consistently outperforms GRPO across the reported transfer splits. The ablation experiments on EnterpriseOps also show that task-matched procedural hints outperform reassigned or generic hints. These results provide useful evidence that the improvements are related to the procedural guidance rather than simply introducing additional text, and suggest that the learned behaviors can transfer across different tool environments.

Critical weaknesses

1. Missing comparisons with direct expert-trajectory distillation leave the additional value of PHS unclear.

The method relies on successful trajectories generated by Kimi-K2.6 and uses Claude Opus 4.8 to extract procedural guidance from the differences between reference and learner trajectories. Given that successful expert trajectories are already available, a natural and potentially competitive alternative would be to directly distill these trajectories through SFT, or to apply GRPO after expert-trajectory distillation.

However, the paper does not include such comparisons. While the reported results demonstrate that PHS outperforms standard GRPO, they do not establish whether the additional procedure extraction, selection, and token-level rescoring provide meaningful benefits over simpler approaches to transferring knowledge from a stronger model.

I consider this comparison important for assessing the method's contribution. If direct distillation achieves comparable performance, the necessity of the additional components becomes less clear. Conversely, stronger in-domain or cross-environment results from PHS under comparable training conditions would provide much stronger support for the proposed approach.

2. Insufficient computational and API cost analysis to support the scalability claims.

Beyond standard GRPO training, PHS requires reference trajectory generation using a stronger model, procedure extraction and selection using Claude, and an additional hint-conditioned rescoring pass for each training trajectory. The authors explain that procedure extraction is performed offline and rescoring does not require additional trajectory sampling, which helps limit the overhead.

However, the paper does not quantify the total computational cost, external API expenses, or additional training time compared with GRPO. It is therefore difficult to assess whether the performance gains justify the additional complexity and cost, particularly as the number of training tasks or model size increases.

A breakdown of GPU usage, API costs, and training time across the different stages, together with comparisons against GRPO and simpler distillation baselines, would help establish the practical value and scalability of PHS.

Questions for authors

Q1. Comparison with direct expert-trajectory distillation

Since the paper already uses Kimi-K2.6 to generate successful reference trajectories, have the authors considered directly using these trajectories for SFT, or applying GRPO after expert-trajectory distillation? Results under comparable training data and computational budgets, particularly on in-domain and cross-environment evaluations, would help clarify the additional benefits of PHS over direct distillation. If these experiments were not conducted, could the authors explain the main reasons for choosing procedural hint supervision over direct trajectory distillation?

Q2. Training cost and scalability

The paper suggests that PHS introduces relatively little additional computational overhead compared with GRPO, but a detailed cost analysis is missing. Could the authors provide a breakdown of the computational and API costs associated with reference trajectory generation, procedure mining, procedure selection, and hint-conditioned rescoring? In particular, how much additional GPU time and API cost does PHS require compared with GRPO, and how would these costs scale with the number of training tasks and model size? This information would help assess whether the performance gains justify the additional training overhead.

Q3. Stability across independent training runs

The paper reports one independent training run per configuration, while the four evaluation seeds correspond to repeated evaluations of the same checkpoint. Could the authors provide results from independent training runs with different random seeds to clarify whether the improvements over GRPO are consistently reproducible? This is particularly relevant for EnterpriseOps and the cross-environment evaluations, where some of the reported gains are relatively modest.

Q4. Effect of normalization in the multiplicative objective

The paper notes that the normalization in PHS-Multiplicative includes tool-output tokens that do not contribute to the training loss. As a result, the objective may introduce both token-level reweighting and sequence-level scaling. Could the authors clarify the motivation for this implementation choice and provide a comparison using normalization over assistant tokens only? If the two variants achieve similar performance, this would strengthen the interpretation that token-level credit assignment is an important source of the improvement. Otherwise, it would be useful to further examine the contribution of sequence-level scaling.

5 comments

Part 1/3

Thank you for your thoughtful review.

This part addresses expert distillation and retention (Weakness 1 / Question 1).

The setup and legend apply to all three parts. GRPO/PHS replication uses three independent training runs; new baselines and ablations use one run due to rebuttal time limits. All policy evaluations are hint-free.

AW/EO = AppWorld/EnterpriseOps, pass@4. BFCL NL/MT/L = non-live/multi-turn/live accuracy; τ³ R/A/T = retail/airline/telecom success. Values: % (mean±SD). PHS-M/A = multiplicative/additive. The other environment is transfer. Control SDs are across evaluations. In control tables, GRPO and PHS rows are the submitted single run; three-run means are in the replication tables.

Weakness 1 / Question 1: direct distillation

We trained expert SFT, then continued with GRPO or PHS.

Expert SFT and RL; trained on AW.

MethodAW devAW testAW chall.EO testBFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base32.9±0.940.8±2.916.9±1.126.6±1.081.9±0.459.6±0.479.7±0.382.9±0.970.4±3.569.1±0.4
GRPO48.7±6.059.7±2.034.2±0.125.0±0.681.7±0.159.7±0.279.7±0.281.3±1.676.0±1.672.2±2.1
PHS-M68.0±2.278.7±1.749.1±0.731.1±2.482.5±0.260.3±0.380.7±0.486.6±2.079.3±1.971.9±1.4
PHS-A64.0±3.074.4±1.145.9±1.230.3±0.882.3±0.260.0±0.680.2±0.288.4±2.683.5±1.973.7±0.0
SFT83.3±3.485.4±1.066.9±0.936.1±1.770.5±0.449.7±0.180.0±0.480.7±1.677.0±1.240.4±0.7
SFT→GRPO83.8±2.287.1±1.274.4±0.636.9±1.372.0±0.749.9±0.380.6±0.380.9±0.876.5±1.045.4±1.3
SFT→PHS-M95.6±1.888.4±1.074.5±0.737.5±1.968.4±0.250.9±1.180.3±0.284.4±0.880.0±2.340.8±0.5
SFT→PHS-A94.7±2.590.5±0.572.2±1.341.4±1.173.2±0.451.1±0.580.7±0.086.4±1.181.5±1.943.9±2.6

Expert SFT and RL; trained on EO.

MethodEO testAW devAW testAW chall.BFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base26.6±1.032.9±0.940.8±2.916.9±1.181.9±0.459.6±0.479.7±0.382.9±0.970.4±3.569.1±0.4
GRPO33.0±0.426.3±0.037.8±3.715.9±0.480.1±0.459.8±0.879.2±0.282.2±2.874.5±1.067.3±1.8
PHS-M37.5±1.437.7±4.650.3±1.723.4±1.083.8±0.464.4±0.680.4±0.184.0±2.084.0±4.071.1±1.0
PHS-A40.4±1.738.2±1.748.2±1.221.8±0.883.5±0.263.1±0.980.1±0.287.0±2.780.3±0.973.7±2.0
SFT30.1±1.011.0±1.722.6±1.38.9±0.742.4±0.112.1±0.675.9±0.30.0±0.02.0±1.63.3±0.8
SFT→GRPO38.1±1.018.0±1.727.4±1.311.8±0.941.8±0.111.5±0.376.1±0.10.4±0.51.0±1.23.1±1.1
SFT→PHS-M38.1±1.233.3±1.439.1±2.421.4±1.242.4±0.28.7±0.276.9±0.20.0±0.01.5±1.02.4±0.8
SFT→PHS-A40.2±1.926.8±5.235.6±2.615.9±2.042.5±0.210.6±0.776.4±0.10.9±0.75.0±1.24.2±1.5

Takeaway: PHS adds value on top of expert distillation, and PHS from the base improves task success without the external losses of the tested SFT recipe. Starting from the same SFT checkpoint, PHS-A reaches 90.5% versus 87.1% for GRPO on AW and 40.2% versus 38.1% on EO; AW-trained SFT→PHS-A also transfers better to EO (41.4 vs 36.9). Expert SFT alone is stronger in-domain on AW than PHS from the base (85.4% vs 78.7% test); we present PHS as complementary to distillation.

The tested SFT recipe loses performance elsewhere. AW-trained SFT scores 49.7 on BFCL multi-turn and 40.4 on τ³ telecom, versus Base's 59.6 and 69.1. GRPO (49.9, 45.4) and PHS-A (51.1, 43.9) after SFT do not restore them; PHS-M from the base reaches 60.3 and 71.9. PHS improves task success without losing these skills; SFT trades external performance for strong AW and EO-transfer scores. These comparisons are in Section 5.5.3 and Appendix F.8 (Table 25).

Part 2/3

This part addresses compute costs (Weakness 2 / Question 2) and independent training runs (Question 3).

Weakness 2 / Question 2: costs and scaling

Online GPU time is close to GRPO. Below: offline procedure costs, reference/SFT costs, then 9B phase timings (steps 1–10). We trained the 9B only on AW because EO takes longer and the rebuttal experiments constrained our compute.

EnvironmentStagecallsinput tokensoutput tokens (saved / cap)cost
EOMining71263.2k19.8k / 63.9k$1.81–2.91
EOConsolidation119.8k5.6k / 32.0k$0.24–0.90
EOSelection4411.97M45.6k / 882.0k$11.00–31.91
AWMining302.04M8.4k / 27.0k$10.41–10.88
AWConsolidation18.8k2.8k / 32.0k$0.11–0.84
AWSelection82345.4k6.4k / 164.0k$1.89–5.83
StageAWEO
Reference generation360 episodes; about $11–311,808 episodes; $215 measured
SFT trainingabout 12 GPU-habout 26 GPU-h
GRPO training18.6 GPU-h to step 10797 / 801 s per step
PHS training19.3 GPU-h to step 10 (Mult.)802 / 764 s per step (Add.)
MethodRolloutLog-prob passReference passPolicy updateTotal stepStep 10 (h)GPU-h
GRPO237 s19 s84 s155 s577 s1.6012.8
PHS-M238 s21 s82 s155 s579 s1.6112.9
PHS-A296 s19 s76 s143 s619 s1.7213.8

Takeaway: PHS's online GPU time is close to GRPO's (19.3 vs 18.6 GPU-h on AW), and expert references and procedure construction are one-time costs. Main RL runs take about 19 GPU-h on AW and about 100 on EO; SFT is additional. Re-scoring adds one forward pass without generation; its cost depends on policy size, trajectory length and serving setup. Mining/selection is once per task set, scaling with tasks/disagreements and input length for fixed API models. Claude costs span saved outputs to full output caps because thinking tokens were not saved. Phase timings overlap, so cannot be summed. These measurements do not imply a universal overhead percentage. The breakdown is in Appendix B.7.

Question 3: independent training runs

We added two training seeds per method. The submitted four seeds in the main paper were repeated evaluations of one model. The main RL methods use the same planned budget within each environment, chosen for training-task coverage with limited repetition, rather than selecting checkpoints using held-out scores.

Three-run replication; trained on AW.

MethodAW devAW testAW chall.EO testBFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base32.9±0.940.8±2.916.9±1.126.6±1.081.9±0.459.6±0.479.7±0.382.9±0.970.4±3.569.1±0.4
GRPO48.7±2.658.8±1.933.4±2.125.5±0.681.9±0.259.2±0.879.7±0.279.8±1.376.8±1.070.0±2.1
PHS-M67.0±1.478.3±1.948.9±2.230.7±0.482.2±0.760.4±0.180.5±0.285.5±1.280.1±0.970.9±0.9
PHS-A66.7±2.676.8±2.348.6±2.530.6±0.682.6±0.460.6±0.580.1±0.185.8±2.382.7±1.072.8±1.0

Three-run replication; trained on EO.

MethodEO testAW devAW testAW chall.BFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base26.6±1.032.9±0.940.8±2.916.9±1.181.9±0.459.6±0.479.7±0.382.9±0.970.4±3.569.1±0.4
GRPO33.2±0.534.4±7.042.6±4.218.0±1.881.0±1.060.3±0.579.7±0.581.5±1.875.8±1.368.1±1.6
PHS-M37.5±2.340.6±3.651.4±1.024.2±0.784.1±0.363.5±0.880.4±0.484.1±0.982.0±1.871.1±1.6
PHS-A38.7±1.836.3±9.246.7±6.220.6±5.483.5±0.461.6±1.580.5±0.486.2±1.481.6±2.571.4±2.0

Takeaway: the PHS advantage holds when policies are independently retrained: both variants have higher mean pass@4 than GRPO on all eight held-out and cross-environment comparisons. The largest replicated test gains are 19.5 percentage points on AW (78.3 vs 58.8) and 5.5 percentage points on EO (38.7 vs 33.2). SD is across training runs; Base reports variation across evaluations. The full table is in Appendix F.7 (Table 24); it extends Sections 5.1–5.4 and Tables 1–4.

Part 3/3

This part addresses normalization and token weighting (Question 4).

Question 4: assistant-only normalization

Including tool outputs was not a design choice: the normalizer averaged over all response tokens. The requested assistant-only control retains gains over GRPO in both environments.

Section 3.3 defines c=clip(exp⁡(sign(A)Δ),0.8,1.2)c=\mathrm{clip}(\exp(\mathrm{sign}(A)\Delta),0.8,1.2) and m=1−λ+λc/cˉm=1-\lambda+\lambda c/\bar c. Tool-output bare log probabilities are zero placeholders: Δ=log⁡p\Delta=\log p, so c=max⁡(p,0.8)c=\max(p,0.8) for A>0A>0 and min⁡(1/p,1.2)\min(1/p,1.2) for A<0A<0.

With tool fraction ff and assistant/tool mean clipped weights a,ba,b, the mean assistant multiplier is 1−λ+λa/((1−f)a+fb)1-\lambda+\lambda a/((1-f)a+fb). For positive λ and f, it is above 1 if b<ab<a and below 1 if b>ab>a: advantage sign alone is insufficient. For fixed a,ba,b, the shift grows with ff. At f=0.6,λ=a=1f=0.6,\lambda=a=1, b=0.8b=0.8 or 1.21.2 gives 1.14 or 0.89. Normalization preserves clipped-weight ordering; 1−λ/3≤m≤1+λ/21-\lambda/3\le m\le1+\lambda/2.

Assistant-only normalization excludes tool outputs. Uniform-within-trajectory assistant weights remove token-specific variation while preserving mean trajectory weighting.

Normalization controls; trained on AW.

MethodAW devAW testAW chall.EO testBFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
GRPO48.7±6.059.7±2.034.2±0.125.0±0.681.7±0.159.7±0.279.7±0.281.3±1.676.0±1.672.2±2.1
PHS-M68.0±2.278.7±1.749.1±0.731.1±2.482.5±0.260.3±0.380.7±0.486.6±2.079.3±1.971.9±1.4
Assistant-only65.8±1.078.9±0.853.4±1.130.5±1.981.9±0.259.9±0.380.3±0.286.2±1.983.0±1.270.8±1.1
Uniform weights56.6±4.164.9±2.133.9±1.130.1±1.082.5±0.061.2±0.180.4±0.183.8±1.780.0±0.072.4±1.5

Normalization controls; trained on EO.

MethodEO testAW devAW testAW chall.BFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
GRPO33.0±0.426.3±0.037.8±3.715.9±0.480.1±0.459.8±0.879.2±0.282.2±2.874.5±1.067.3±1.8
PHS-M37.5±1.437.7±4.650.3±1.723.4±1.083.8±0.464.4±0.680.4±0.184.0±2.084.0±4.071.1±1.0
Assistant-only35.7±2.335.5±1.748.2±1.321.2±0.485.0±0.262.7±0.680.5±0.183.1±0.878.5±2.570.2±1.0
Uniform weights34.2±1.239.5±1.844.6±1.618.3±0.982.2±0.460.9±0.880.4±0.287.3±1.782.5±1.071.1±1.6

Takeaway: the in-domain gain comes from token-level credit, not from the tool-output normalization: assistant-only normalization keeps it, and uniform weights lose most of it. AW split means are 66.0, 65.3, 51.8 and 47.5 for assistant-only, original PHS-M, uniform weights and GRPO; EO test scores are 35.7, 37.5, 34.2 and 33.0. Uniform weights keep most of the AW→EO transfer gain (30.1 vs 31.1), so we attribute only the in-domain gain to token-level credit. The derivation is in Section 3.3; the controls are in Section 5.5.1 and Appendix F.9 (Table 26).

Reviewer_nq56OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PHS extracts reusable procedures from successful reference trajectories and failed learner attempts. These procedures re-score the learner’s own rollouts and modify GRPO advantages through additive or multiplicative updates, with no hints at inference. Experiments show improved held-out success and cross-environment transfer, additional gains on tau-3, and little aggregate change on BFCL.

Critical strengths
  1. The AppWorld gains are substantial: PHS-Multiplicative improves test pass@4 over GRPO by 19.1 points. Improvements also transfer between environments with different tools and interfaces, without test-time hints.

  2. Task-matched procedures beat reassigned and generic hints under the additive objective, giving concrete evidence that the relevance of the guidance matters.

Critical weaknesses
  1. Total training steps, checkpoint-selection rules and measured compute overhead are missing.

  2. PHS uses successful reference trajectories, but there is no comparison with directly distilling those same trajectories. Prompt-time improvements also do not establish that the procedures were already represented internally, so the "latent capabilities" interpretation should be narrowed.

  3. Tool outputs enter the normalizer despite their bare log probabilities being zero placeholders, making assistant-token weights depend on tool-output length and advantage sign.

Questions for authors
  1. What training budgets and checkpoint-selection rules were used across methods? Do the main gains persist across independent training seeds?

  2. How does PHS compare with direct distillation from the same successful reference trajectories and with GRPO under an equal total compute budget, including reference generation and hinted re-scoring?

  3. Do the multiplicative gains survive assistant-only normalization? How much remains when only trajectory weights change, with uniform weights across assistant tokens?

  4. With training tasks held fixed, does model scale change the ranking of additive and multiplicative PHS? Smaller learners may produce more zero-variance reward group, and only additive PHS supplies a signal. does this ranking reverse as larger models get better reward coverage?

4 comments

Part 1/3

Thank you for your thoughtful review.

This part addresses training budgets and independent runs (Weakness 1 / Question 1).

The setup and legend apply to all three parts. GRPO/PHS replication uses three independent training runs; new baselines and ablations use one run due to rebuttal time limits. All policy evaluations are hint-free.

AW/EO = AppWorld/EnterpriseOps, pass@4. BFCL NL/MT/L = non-live/multi-turn/live accuracy; τ³ R/A/T = retail/airline/telecom success. Values: % (mean±SD). PHS-M/A = multiplicative/additive. Control SDs are across evaluations. In control tables, GRPO and PHS rows are the submitted single run; three-run means are in the replication tables.

Weakness 1 / Question 1: budgets and replication

Within each environment, main RL comparisons use the same tasks, batches and hardware. AW uses 10 steps of 16 tasks × 8 samples (about 19 GPU-hours on one eight-GPU node); EO uses 56 steps of 8 tasks × 8 samples (about 100 GPU-hours on one eight-GPU node).

We chose the budgets from the training-set sizes (82 AW tasks and 441 EO tasks), aiming for broad coverage while limiting repeated passes. All methods are compared at the same planned endpoint, without selecting checkpoints using held-out scores.

We added two training seeds per method to the submitted run, bringing the total to three training seeds; the paper's four seeds were repeated evaluations.

Three-run replication; trained on AppWorld.

MethodAW devAW testAW chall.EO testBFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base32.9±0.940.8±2.916.9±1.126.6±1.081.9±0.459.6±0.479.7±0.382.9±0.970.4±3.569.1±0.4
GRPO48.7±2.658.8±1.933.4±2.125.5±0.681.9±0.259.2±0.879.7±0.279.8±1.376.8±1.070.0±2.1
PHS-M67.0±1.478.3±1.948.9±2.230.7±0.482.2±0.760.4±0.180.5±0.285.5±1.280.1±0.970.9±0.9
PHS-A66.7±2.676.8±2.348.6±2.530.6±0.682.6±0.460.6±0.580.1±0.185.8±2.382.7±1.072.8±1.0

Three-run replication; trained on EnterpriseOps.

MethodEO testAW devAW testAW chall.BFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base26.6±1.032.9±0.940.8±2.916.9±1.181.9±0.459.6±0.479.7±0.382.9±0.970.4±3.569.1±0.4
GRPO33.2±0.534.4±7.042.6±4.218.0±1.881.0±1.060.3±0.579.7±0.581.5±1.875.8±1.368.1±1.6
PHS-M37.5±2.340.6±3.651.4±1.024.2±0.784.1±0.363.5±0.880.4±0.484.1±0.982.0±1.871.1±1.6
PHS-A38.7±1.836.3±9.246.7±6.220.6±5.483.5±0.461.6±1.580.5±0.486.2±1.481.6±2.571.4±2.0

Takeaway: the PHS advantage holds when policies are independently retrained: both variants have higher mean pass@4 than GRPO on all eight held-out and cross-environment comparisons. The largest replicated test gains are 19.5 percentage points on AW (78.3 vs 58.8) and 5.5 percentage points on EO (38.7 vs 33.2). SD is across training runs; Base reports variation across evaluations. The full table is in Appendix F.7 (Table 24); it extends Sections 5.1–5.4 and Tables 1–4.

Part 2/3

This part addresses expert distillation, compute and interpretation (Weakness 2 / Question 2).

Weakness 2 / Question 2: direct expert distillation

We trained SFT on expert trajectories, then continued with GRPO or PHS.

Expert SFT and RL; trained on AppWorld.

MethodAW devAW testAW chall.EO testBFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base32.9±0.940.8±2.916.9±1.126.6±1.081.9±0.459.6±0.479.7±0.382.9±0.970.4±3.569.1±0.4
GRPO48.7±6.059.7±2.034.2±0.125.0±0.681.7±0.159.7±0.279.7±0.281.3±1.676.0±1.672.2±2.1
PHS-M68.0±2.278.7±1.749.1±0.731.1±2.482.5±0.260.3±0.380.7±0.486.6±2.079.3±1.971.9±1.4
PHS-A64.0±3.074.4±1.145.9±1.230.3±0.882.3±0.260.0±0.680.2±0.288.4±2.683.5±1.973.7±0.0
SFT83.3±3.485.4±1.066.9±0.936.1±1.770.5±0.449.7±0.180.0±0.480.7±1.677.0±1.240.4±0.7
SFT→GRPO83.8±2.287.1±1.274.4±0.636.9±1.372.0±0.749.9±0.380.6±0.380.9±0.876.5±1.045.4±1.3
SFT→PHS-M95.6±1.888.4±1.074.5±0.737.5±1.968.4±0.250.9±1.180.3±0.284.4±0.880.0±2.340.8±0.5
SFT→PHS-A94.7±2.590.5±0.572.2±1.341.4±1.173.2±0.451.1±0.580.7±0.086.4±1.181.5±1.943.9±2.6

Expert SFT and RL; trained on EnterpriseOps.

MethodEO testAW devAW testAW chall.BFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base26.6±1.032.9±0.940.8±2.916.9±1.181.9±0.459.6±0.479.7±0.382.9±0.970.4±3.569.1±0.4
GRPO33.0±0.426.3±0.037.8±3.715.9±0.480.1±0.459.8±0.879.2±0.282.2±2.874.5±1.067.3±1.8
PHS-M37.5±1.437.7±4.650.3±1.723.4±1.083.8±0.464.4±0.680.4±0.184.0±2.084.0±4.071.1±1.0
PHS-A40.4±1.738.2±1.748.2±1.221.8±0.883.5±0.263.1±0.980.1±0.287.0±2.780.3±0.973.7±2.0
SFT30.1±1.011.0±1.722.6±1.38.9±0.742.4±0.112.1±0.675.9±0.30.0±0.02.0±1.63.3±0.8
SFT→GRPO38.1±1.018.0±1.727.4±1.311.8±0.941.8±0.111.5±0.376.1±0.10.4±0.51.0±1.23.1±1.1
SFT→PHS-M38.1±1.233.3±1.439.1±2.421.4±1.242.4±0.28.7±0.276.9±0.20.0±0.01.5±1.02.4±0.8
SFT→PHS-A40.2±1.926.8±5.235.6±2.615.9±2.042.5±0.210.6±0.776.4±0.10.9±0.75.0±1.24.2±1.5

Takeaway: PHS adds value on top of expert distillation, and PHS from the base improves task success without the external losses of the tested SFT recipe. Starting from the same SFT checkpoint, PHS-A reaches 90.5% versus 87.1% for GRPO on AW and 40.2% versus 38.1% on EO; AW-trained SFT→PHS-A also transfers better to EO (41.4 vs 36.9). Expert SFT alone is stronger in-domain on AW than PHS from the base (85.4% vs 78.7% test); we present PHS as complementary to distillation.

The tested SFT recipe loses performance elsewhere. AW-trained SFT scores 49.7 on BFCL multi-turn and 40.4 on τ³ telecom, versus Base's 59.6 and 69.1. GRPO (49.9, 45.4) and PHS-A (51.1, 43.9) after SFT do not restore them; PHS-M from the base reaches 60.3 and 71.9. PHS improves task success without losing these skills; SFT trades external performance for strong AW and EO-transfer scores. These comparisons are in Section 5.5.3 and Appendix F.8 (Table 25).

Question 2 continued: compute

AW GRPO extended to step 11 uses 1.07× PHS's online training time, including PHS re-scoring.

Online-compute control; trained on AW.

MethodAW devAW testAW chall.EO testBFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
GRPO48.7±6.059.7±2.034.2±0.125.0±0.681.7±0.159.7±0.279.7±0.281.3±1.676.0±1.672.2±2.1
GRPO 1.07×51.8±2.364.1±1.936.8±1.225.4±2.481.7±0.659.4±0.479.4±0.079.4±1.776.5±2.569.3±1.2
PHS-M68.0±2.278.7±1.749.1±0.731.1±2.482.5±0.260.3±0.380.7±0.486.6±2.079.3±1.971.9±1.4
MethodWall-clock to checkpointGPU-hours
GRPO, 10 steps2.33 h18.6
GRPO, 11 steps2.57 h20.6
PHS-Mult., 10 steps2.41 h19.3

Takeaway: with more online training time than PHS, GRPO still trails PHS-M, 64.1 versus 78.7 on AW test. Offline costs are separate: procedure mining/selection costs USD 12–18 on AW and USD 13–36 on EO; reference generation costs about USD 11–31 for 360 AW episodes and USD 215 for 1,808 EO episodes. These are one-time costs, not included in online compute matching. The breakdown is in Appendix B.7.

Weakness 2: interpreting latent capabilities

Prompt-time success shows that the base can follow guidance; it does not establish that the procedure was already internally represented. We narrowed this wording in the abstract, introduction and Section 6. The supported claim is that procedural supervision improves learned behavior and transfer, without requiring a claim about what was latent before training.

Part 3/3

This part addresses normalization and token weighting (Weakness 3 / Question 3), and model/reward coverage (Question 4).

Weakness 3 / Question 3: normalization and token weighting

Section 3.3 defines c=clip(exp⁡(sign(A)Δ),0.8,1.2)c=\mathrm{clip}(\exp(\mathrm{sign}(A)\Delta),0.8,1.2) and m=1−λ+λc/cˉm=1-\lambda+\lambda c/\bar c. Tool-output bare log probabilities are zero placeholders: Δ=log⁡p\Delta=\log p, so c=max⁡(p,0.8)c=\max(p,0.8) for A>0A>0 and min⁡(1/p,1.2)\min(1/p,1.2) for A<0A<0.

With tool fraction ff and assistant/tool mean clipped weights a,ba,b, the mean assistant multiplier is 1−λ+λa/((1−f)a+fb)1-\lambda+\lambda a/((1-f)a+fb). For positive λ and f, it is above 1 if b<ab<a and below 1 if b>ab>a: advantage sign alone is insufficient. For fixed a,ba,b, the shift grows with ff. At f=0.6,λ=a=1f=0.6,\lambda=a=1, b=0.8b=0.8 or 1.21.2 gives 1.14 or 0.89. Normalization preserves clipped-weight ordering; 1−λ/3≤m≤1+λ/21-\lambda/3\le m\le1+\lambda/2.

Assistant-only normalization excludes tool outputs. Uniform-within-trajectory assistant weights remove token-specific variation while preserving mean trajectory weighting.

Normalization controls; trained on AW.

MethodAW devAW testAW chall.EO testBFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
GRPO48.7±6.059.7±2.034.2±0.125.0±0.681.7±0.159.7±0.279.7±0.281.3±1.676.0±1.672.2±2.1
PHS-M68.0±2.278.7±1.749.1±0.731.1±2.482.5±0.260.3±0.380.7±0.486.6±2.079.3±1.971.9±1.4
Assistant-only65.8±1.078.9±0.853.4±1.130.5±1.981.9±0.259.9±0.380.3±0.286.2±1.983.0±1.270.8±1.1
Uniform weights56.6±4.164.9±2.133.9±1.130.1±1.082.5±0.061.2±0.180.4±0.183.8±1.780.0±0.072.4±1.5

Normalization controls; trained on EO.

MethodEO testAW devAW testAW chall.BFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
GRPO33.0±0.426.3±0.037.8±3.715.9±0.480.1±0.459.8±0.879.2±0.282.2±2.874.5±1.067.3±1.8
PHS-M37.5±1.437.7±4.650.3±1.723.4±1.083.8±0.464.4±0.680.4±0.184.0±2.084.0±4.071.1±1.0
Assistant-only35.7±2.335.5±1.748.2±1.321.2±0.485.0±0.262.7±0.680.5±0.183.1±0.878.5±2.570.2±1.0
Uniform weights34.2±1.239.5±1.844.6±1.618.3±0.982.2±0.460.9±0.880.4±0.287.3±1.782.5±1.071.1±1.6

Takeaway: the in-domain gain comes from token-level credit, not from the tool-output normalization: assistant-only normalization keeps it, and uniform weights lose most of it. AW split means are 66.0, 65.3, 51.8 and 47.5 for assistant-only, original PHS-M, uniform weights and GRPO; EO test scores are 35.7, 37.5, 34.2 and 33.0. Uniform weights keep most of the AW→EO transfer gain (30.1 vs 31.1), so we attribute only the in-domain gain to token-level credit. The derivation is in Section 3.3; the controls are in Section 5.5.1 and Appendix F.9 (Table 26).

Question 4: model and reward coverage

We trained Qwen3.5-9B on the same AW tasks and procedures. It was trained only on AW because EO takes longer and the rebuttal experiments constrained our compute. This tests another model, not scale alone: generation and dense/MoE architecture also change (9B; 35B-A3B).

Qwen3.5-9B, trained on AW.

MethodAW devAW testAW chall.EO testBFCL NLBFCL MTBFCL Lτ³ Rτ³ Aτ³ T
Base19.7±2.226.5±1.011.0±0.820.5±0.782.1±0.150.5±0.877.8±0.180.7±0.772.5±2.564.7±1.8
GRPO38.2±1.740.9±2.821.9±0.721.5±0.581.8±0.152.3±0.277.2±0.178.1±0.774.0±3.764.5±3.4
PHS-M64.0±2.364.3±2.243.7±1.027.9±1.382.7±0.155.0±1.177.9±0.283.1±1.573.0±2.663.4±1.3
PHS-A60.1±2.657.4±1.040.5±0.425.0±0.683.2±0.254.2±0.477.7±0.183.6±1.978.0±1.667.3±1.1

Reward coverage over steps 1–10 and AW split means.

MethodAW split averageMean reward, steps 1–10Within-group varianceZero-variance groups
GRPO33.70.5030.05018%
PHS-Mult.57.30.5350.05012%
PHS-Add.52.70.5390.04817%

Takeaway: PHS also helps on a second model. The preferred variant depends on the environment and metric, and we see no reversal between the two models. Only additive hint credit can supply a PHS advantage signal to zero-variance reward groups. The observed frequencies (table above) do not by themselves explain the ranking. PHS-M leads AW (57.3 vs 52.7 split mean), while PHS-A leads τ³ airline (78.0 vs 73.0). On the original model, replicated AW test favors PHS-M (78.3 vs 76.8) and EO test favors PHS-A (38.7 vs 37.5). The additional-model results are in Appendix F.11 (Table 29) and the configuration in Appendix B.6; we have not run a controlled size sweep.

Dear Reviewer nq56,

As the discussion period comes to an end, we would greatly appreciate it if you could take a look at our responses and revised manuscript. We have worked extensively to address the concerns you raised through additional experiments and clarifications, and would welcome your thoughts on these additions.

If any questions or concerns remain, we would be happy to discuss them further.

Thank you again for your time and consideration.

Best,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Summary of Revisions

General Response and Summary of Revisions

Dear Area Chair and Reviewers,

Thank you for your careful reading of our work and your constructive feedback. To help navigate our posted replies, we summarize the main additions in our rebuttal below. All references point to the revised manuscript that has been uploaded.

We completed 32 additional training runs and extensions across replication, baseline, and ablation experiments, evaluated on 10 held-out splits or benchmark groups. These required an estimated 1,600 additional training GPU-hours on NVIDIA B300 GPUs, based on run budgets and excluding evaluation. GRPO and both PHS variants now have three independent training runs per environment; new baselines and ablations use a single training run due to the rebuttal time constraints. Given the longer training time on EnterpriseOps and the breadth of these experiments, we focused the additional Qwen3.5-9B model comparison on AppWorld. Within the rebuttal’s time and compute constraints, we prioritized the experiments addressing your central concerns and appreciate your understanding of the remaining experimental limits.

1. Replication and generalization. The gains hold with the additional training seeds: both PHS variants have higher mean pass@4 than GRPO on all eight held-out and cross-environment comparisons. The largest gains are 19.5 percentage points on AppWorld test, 15.5 on its challenge split, and 5.5 on EnterpriseOps. EnterpriseOps-trained multiplicative PHS also gains 3.2 percentage points in BFCL multi-turn accuracy and 6.2 in τ³ airline success over GRPO, exceeding Base on both without training on either benchmark. Appendix F.7, Table 24.

2. Expert-distillation baselines. We added expert SFT and subsequent GRPO/PHS training in both environments. SFT is strong on AppWorld, and PHS complements it: from the same SFT checkpoint, additive PHS reaches 90.5% versus 87.1% test pass@4 for GRPO on AppWorld, and 40.2% versus 38.1% on EnterpriseOps. However, the tested SFT recipe introduces a generalization trade-off that subsequent GRPO does not resolve. This is especially clear on EnterpriseOps: SFT→GRPO reaches 38.1% in-domain, but BFCL multi-turn accuracy remains at 11.5% versus 59.6% for Base, and τ³ telecom success at 3.1% versus 69.1%. PHS trained directly from Base improves in-domain performance without these external losses. Section 5.5.3; Appendix F.8, Table 25.

3. Credit assignment and outcome feedback. Normalizing over assistant tokens alone retains the gains, showing they do not depend on including tool outputs in normalization. Replacing token weights with their trajectory mean preserves average trajectory weighting but substantially reduces the in-domain benefit, supporting the value of token-level credit assignment. At a matched hint coefficient, including outcome feedback raises test pass@4 from 34.5% to 74.4% on AppWorld and 0.0% to 40.4% on EnterpriseOps. GRPO with hints during training rollouts also trails PHS in-domain under hint-free evaluation. Together, these controls support using procedural guidance to shape learning alongside outcome rewards. Sections 3.3 and 5.5.1–5.5.2; Appendices F.9–F.10, Tables 26–27.

4. Compute and another model. AppWorld training costs 19.3 GPU-hours for multiplicative PHS versus 18.6 for GRPO. To test whether more ordinary GRPO training closes the gap, we extended GRPO to 20.6 GPU-hours, slightly exceeding PHS's online compute. It reaches 64.1% test pass@4 versus 78.7% for PHS; offline reference and procedure costs are reported separately. On Qwen3.5-9B, multiplicative PHS reaches 64.3% versus 40.9% for GRPO, extending the evidence beyond the original model. Appendix B.7; Appendices F.10–F.11, Tables 28–29.

5. Method and interpretation. We clarified the relationship to privileged-context distillation, why rescoring weights are held fixed during optimization, and how normalization affects the objective. External-trace rescoring and blind judgments did not establish that particular procedures caused specific benchmark gains. We therefore ground our generalization claim in the benchmark results, while the procedure-assignment controls support the value of task-relevant guidance. Sections 2–3 and 6; Appendix F.12.

The individual reviewer threads contain the full comparisons and experimental details. We appreciate your time and continued engagement, and thank you for helping us strengthen the paper.

Follow-up on the rebuttal discussion

Dear Reviewers,

Thank you again for your time and constructive feedback. Before the discussion closes in the next half a day, we would greatly appreciate your perspectives on our additional experiments and revised manuscript.

We devoted substantial effort to addressing the reviews through new experiments, analyses, and revisions, summarized in our General Response and Summary of Revisions, and detailed in your respective threads. It would be particularly helpful to understand which concerns have been addressed and which remain unresolved, so that we can respond directly.

We believe this back-and-forth is an important opportunity to clarify misunderstandings, examine the new evidence, and identify where the paper still needs improvement. We recognize the substantial reviewing workload and sincerely appreciate your consideration. We would welcome the opportunity to continue that discussion before the deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.