acceptodds
Under review as a conference paper at ICLR 2027

HARP: On-Policy Distillation of Hierarchically Routed Planning, Execution, and Revision Experts

Abstract

Long-horizon tool-use agents remain difficult to train because task-level outcome rewards provide sparse credit across trajectories involving many interdependent decisions. Process-level supervision alleviates this sparsity with finer-grained feedback, but typically treats intermediate errors through a largely homogeneous signal. In interactive environments, however, failures are structurally heterogeneous: an agent may pursue an incorrect subgoal, execute a valid plan with an incorrect tool call, or fail to diagnose and recover from an earlier mistake. Effective supervision should therefore capture not only where credit should be assigned, but also what kind of failure occurred and how it should be corrected. We introduce HARP (Hierarchically Arranged Role Policies), a training framework that decomposes failures into planning, execution, and recovery errors and assigns them to role-specialized Planner, Executor, and Repairer components coordinated by verification and routing mechanisms. HARP uses this structured system as a training-time teacher and distills its role-specific corrective supervision on-policy into a single base model, avoiding the full scaffold at deployment. On AppWorld with Qwen3-1.7B, 4B, and 8B, the distilled HARP model improves task goal completion over the base model at every size and outperforms GRPO at 1.7B and 4B, while GRPO remains stronger at 8B. Component ablations and routing diagnostics further characterize the contribution of each role.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Long-Horizon Tasks, Tool-Using Agents, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SYj6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HARP trains separate Planner, Executor and Repairer policies, composes them using a phase classifier and a progress verifier, and distils the routed teacher into a standalone Qwen3 model on student-generated trajectories. Executor training includes verifier-based reward, Planner training uses terminal reward, and the Repairer learns from replay-filtered corrections. On AppWorld, the distilled model outperforms the corresponding base model at all three tested sizes and GRPO at 1.7B and 4B, but not 8B. Component results show a substantial 4B Test-N improvement from adding the Repairer to the routed system, while repair gains and their retention after distillation are inconsistent across sizes and splits.

Critical strengths
  1. Organising training-time teachers around corrective responsibilities is a meaningful systems contribution. The approach explicitly separates the richer training harness from the single-model deployment interface, and the measured standalone improvements show that some benefit survives that separation. At 1.7B, HARP reaches 13.1%/3.8% TGC on Test-N/Test-C versus 0.6%/0.0% for GRPO.

  2. The paper reports limitations unusually concretely: the 8B baseline reversal, the 4B teacher-to-student performance loss, the tiny repair dataset and the verifier's weak regression recall are all visible. This makes the empirical scope assessable and avoids conflating verifier improvement with independently measured task success.

Critical weaknesses
  1. The central role-specialisation and repair contribution is not isolated in the final model. HARP differs from GRPO in supervised initialisation, verifier feedback, expert training, repair data and distillation. Table 2 tests routed components, but does not compare final students distilled from matched two-role and three-role teachers or a non-specialised teacher using the same data. Moreover, adding the Repairer also changes the handoff rules from Router2 to Router3. The 4B Test-N teacher gain from 20.2% to 27.4% becomes 19.6% after OPD, and other repair gains are small or negative. Thus the evidence does not yet establish that repair-aware role routing, rather than the other training additions, drives the deployed benefit.

  2. The reliability evidence for the corrective signal is narrow. The verifier's labels are model-assisted without independent human adjudication, and only three of nine held-out regression examples are detected; the paper notes that the detected examples come from one task. The strict repair set contains 17 examples from 13 tasks. The lower no-progress rate uses that same verifier on different visited states and an earlier router version (Appendix C.7). These diagnostics do not independently establish improved error diagnosis or repair, which is central to the paper's explanation. Single-run task results further limit interpretation of the smaller differences.

Questions for authors
  1. Targeted clarification/validation: is there an existing OPD comparison using the trained Planner–Executor teacher without the Repairer, with the same student budget and handoff rules? This would directly test whether the final model retains a repair-specific advantage. Please also clarify whether the 4B teacher-to-student decline is robust to checkpoint choice.

  2. Can you report independently checked outcomes for a small held-out sample of repair activations, distinguishing genuinely corrected tasks, unnecessary interventions and missed regressions? Terminal success or same-state replay would be more informative than another comparison scored solely by V0.

  3. Clarification: what are the total training trajectories and teacher/annotation costs for HARP and GRPO, and what data and optimisation settings produce the SFT cold start? Full cost parity is not required, but the comparison needs an explicit resource boundary.

6 comments

[Weakness 1] The central role-specialisation and repair contribution is not isolated in the final model. HARP differs from GRPO in supervised initialisation, verifier feedback, expert training, repair data and distillation. Table 2 tests routed components, but does not compare final students distilled from matched two-role and three-role teachers or a non-specialised teacher using the same data. Moreover, adding the Repairer also changes the handoff rules from Router2 to Router3. The 4B Test-N teacher gain from 20.2% to 27.4% becomes 19.6% after OPD, and other repair gains are small or negative. Thus the evidence does not yet establish that repair-aware role routing, rather than the other training additions, drives the deployed benefit.

[Question 1] Targeted clarification/validation: is there an existing OPD comparison using the trained Planner–Executor teacher without the Repairer, with the same student budget and handoff rules? This would directly test whether the final model retains a repair-specific advantage. Please also clarify whether the 4B teacher-to-student decline is robust to checkpoint choice.

Thank you for raising this point. We compared the full HARP student with a student distilled from the trained Planner–Executor teacher without the Repairer. We report OPD step 3 for all three model sizes.

Qwen3 sizeFull HARP (Test-N / Test-C)Planner–Executor only (Test-N / Test-C)
1.7B13.1% / 3.8%10.7% / 3.1%
4B19.6% / 8.4%21.4% / 10.6%
8B32.7% / 8.4%31.0% / 7.2%

Full HARP scores higher on both splits at 1.7B and 8B, while the Planner–Executor student scores higher on both at 4B. On 4B Test-N, the teacher with the Repairer solves 46/168 tasks versus 34/168 for the Planner–Executor teacher, but their step-3 students solve 33/168 and 36/168, respectively. At 1.7B and 8B, the Repairer-equipped teacher shows no gain on Test-N and a slight decrease on Test-C, while full HARP’s distilled students score higher on both splits. End-to-end teacher performance is therefore an imperfect predictor of student performance after OPD.

One possible explanation is that OPD transfers useful decisions at individual steps along the student’s own trajectories. For example, the Repairer can provide a useful plan revision after a setback even when the routed teacher fails to complete the overall task. This lets the framework bring planning, execution, and recovery supervision into a single deployable policy. The comparison is consistent with this explanation, though it varies the Repairer and the handoff rules together.

In the revised manuscript, we added the step-3 results for the no-Repairer OPD student to Table 2. Section 5.3 (Component Ablations) describes the comparison and discusses the 4B teacher–student ranking reversal. These additions are highlighted in dark blue.

[Weakness 2] The reliability evidence for the corrective signal is narrow. The verifier's labels are model-assisted without independent human adjudication, and only three of nine held-out regression examples are detected; the paper notes that the detected examples come from one task. The strict repair set contains 17 examples from 13 tasks. The lower no-progress rate uses that same verifier on different visited states and an earlier router version (Appendix C.7). These diagnostics do not independently establish improved error diagnosis or repair, which is central to the paper's explanation. Single-run task results further limit interpretation of the smaller differences.

[Question 2] Can you report independently checked outcomes for a small held-out sample of repair activations, distinguishing genuinely corrected tasks, unnecessary interventions and missed regressions? Terminal success or same-state replay would be more informative than another comparison scored solely by V0.

Thank you for asking for independent evidence on repair outcomes. We sampled one activation from each of the 1,583 held-out test trajectories in which Router3 invoked the Repairer, yielding 1,572 evaluable pairs. For each pair, we restored the same state immediately before the intervention and let either the Repairer or the regular Planner produce the next plan. Both branches then used the same Planner and Executor, with further repair disabled and the same remaining action budget. We evaluated terminal success with AppWorld’s official state-based tests, independently of V0V_0.

Qwen3 sizeEvaluable pairsRescuedHarmedBoth succeededBoth failed
1.7B534298515
4B52013343461
8B5187556450
Overall1,57222171071,426

A rescue means only the Repairer branch succeeded, and a harm means only the Planner branch succeeded. In 107 pairs, both branches succeeded, so the intervention was unnecessary for terminal success in those replays. Overall, the Repairer succeeded in 129/1,572 pairs and the Planner in 124/1,572, a net difference of +0.32 percentage points. The largest observed gain was at 4B, with 13 rescues and 3 harms. This matches the pattern in Table 2, where the configuration with the Repairer also shows the largest improvement at 4B.

In the revised manuscript, Section 5.4 (Routing and Verifier Diagnostics) summarizes the paired replay results. Appendix C.7 (Evaluation Details) presents the replay protocol, independent scoring, and per-model results after the Repairer routing diagnostic. These additions are highlighted in dark blue.

[Question 3] Clarification: what are the total training trajectories and teacher/annotation costs for HARP and GRPO, and what data and optimisation settings produce the SFT cold start? Full cost parity is not required, but the comparison needs an explicit resource boundary.

Thank you for asking us to clarify the training and annotation costs.

  • Online training. A complete HARP OPD run uses 120 student-generated trajectories per model size (15 updates × 4 tasks × 2 rollouts), with 24 trajectories used by step 3. The routed experts score tokens on these trajectories. A complete archived GRPO run uses 20,480 trajectories (80 updates × 32 tasks × 8 rollouts), with 15,360 trajectories used by step 60. GRPO trains from the base model using environment rewards, without a separate teacher or annotation pipeline.
  • Student SFT cold start. Before Router3 OPD, we collected a shared corpus of 3,000 successful trajectories from the teacher in 8,210 attempts. We used 2,912 trajectories for training and 88 for validation. Each corresponding Qwen3 base model received one epoch of SFT with loss on assistant tokens only, using LoRA (rank 32, alpha 64) and AdamW with a learning rate of 6×10−56\times10^{-5}.
  • Initial SFT data. Qwen3-32B generated 3,600 candidate trajectories, of which 1,215 successful trajectories were retained.
  • Phase classifier. 720 historical trajectories yielded 58,659 rule-assisted sentence-level labels, including 46,737 for training.
  • Planner and progress verifier. Planner supervision involved model-assisted review of 14,832 assistant turns. The verifier pipeline processed 18,852 candidate records, yielding 9,622 training examples and 1,001 held-out examples. The two pipelines share source trajectories.
  • Repairer. We generated 4,500 short branches from 750 intervention points, replayed 485 branches, and used Codex to review 65 flagged states. The final SFT set contained 17 correction examples and 34 ordinary planning examples.

The online rollout counts cover online training only. HARP’s resource requirements also include shared SFT data collection, teacher preparation, and annotation. We report those workloads in their original units because they include overlapping sources, full trajectories, and short continuations.

The revised manuscript now reports the training trajectory counts, student SFT cold start, and teacher preparation and annotation workloads in Appendix C.8 (Training and Annotation Workload). This subsection is highlighted in dark blue.

Reviewer_G7PkOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HARP, a framework for role-specialized training and on-policy distillation of long-horizon tool-use agents. The authors argue that existing outcome-level and process-level supervision methods struggle to distinguish among different types of errors in planning, execution, and recovery. To address this issue, HARP trains specialized Planner and Executor models using SFT followed by GRPO, and a Repairer model using SFT. These experts are then integrated into a routed teacher through progress-aware routing, and their capabilities are distilled into a single student model via on-policy distillation. Experiments on AppWorld show that HARP outperforms GRPO with Qwen3-1.7B and Qwen3-4B, but underperforms GRPO with Qwen3-8B.

Critical strengths
  1. The paper highlights the limitations of existing outcome-level and process-level supervision in distinguishing different types of errors. By designing targeted supervision mechanisms for planning, execution, and recovery, it provides a valuable perspective on long-horizon agent training.
  2. HARP achieves substantial improvements on smaller models. Experimental results show that HARP outperforms GRPO on both Qwen3-1.7B and Qwen3-4B, with particularly notable gains on Qwen3-1.7B, suggesting the potential of this framework for improving the performance of weaker long-horizon agents.
Critical weaknesses
  1. The method is overly complex, and its performance gains are inconsistent. HARP requires separately training the Planner, Executor, Repairer, and additional auxiliary classifiers, followed by OPD to distill them into a single Student Model. However, the experimental results do not consistently demonstrate the necessity of such a complex training pipeline. HARP underperforms GRPO on the 8B model, and experiments are conducted only on AppWorld, lacking evaluation of generalization across different tasks.
  2. Although the Executor introduces a Progress-Based Reward, the authors do not investigate whether directly training a single model with the same Progress Reward could achieve similar performance.
  3. The Repairer is fine-tuned using only 17 repair examples through SFT, making it difficult to establish its generalization ability across diverse types of errors.
Questions for authors
  1. Are only 17 repair examples sufficient to learn a generalizable Recovery Capability?
  2. Are HARP's performance improvements primarily driven by the Progress-Based Reward introduced for the Executor rather than Role-Specialized Training itself? I suggest adding a GRPO baseline that directly trains a single model using the same Progress Reward, with a comparable training budget.
  3. Is it necessary to train separate role-specialized models? Could the authors include a baseline that uses the base model with role-specific prompts and the same routing mechanism to evaluate the necessity of additional expert training?
5 comments

[Weakness 1] The method is overly complex, and its performance gains are inconsistent. HARP requires separately training the Planner, Executor, Repairer, and additional auxiliary classifiers, followed by OPD to distill them into a single Student Model. However, the experimental results do not consistently demonstrate the necessity of such a complex training pipeline. HARP underperforms GRPO on the 8B model, and experiments are conducted only on AppWorld, lacking evaluation of generalization across different tasks.

[Question 3] Is it necessary to train separate role-specialized models? Could the authors include a baseline that uses the base model with role-specific prompts and the same routing mechanism to evaluate the necessity of additional expert training?

Thank you for raising the questions about training complexity, the scope of the gains, and whether role-specific prompting can substitute for expert training.

Performance across model scales and test splits. The gains over GRPO are consistent in direction across the two splits at smaller model scales. HARP improves Test-N task goal completion (TGC) by 12.5 and 6.5 percentage points at 1.7B and 4B, respectively, and Test-C TGC by 3.8 and 3.1 points. At 8B, GRPO leads by 3.6 points on both splits. The observed advantage is therefore scale-dependent. HARP adds training stages, and the distilled student runs as a single model at deployment. Thus, expert routing and separate expert calls add no overhead at inference.

Generalization within AppWorld. Test-C evaluates transfer to tasks with held-out app requirements. Each Test-C task requires an API from Amazon or Gmail, and tasks with these requirements are reserved for Test-C rather than included in Train, Dev, or Test-N. The gains at 1.7B and 4B on Test-C show that the smaller-model advantage extends to this within-benchmark distribution shift. Evaluating transfer to other environments remains an open question.

Role-specific prompting baseline. We evaluated the requested baseline at all three scales. The Planner, Executor, and Repairer share the same unmodified Qwen3 checkpoint at each scale, receive distinct role-specific few-shot prompts, and use Router3. The table reports TGC (%) on the full Test-N and Test-C splits.

Qwen3 sizePrompted base + Router3 (Test-N / Test-C)Trained experts + Router3 (Test-N / Test-C)HARP student (Test-N / Test-C)
1.7B0.60 / 0.2410.7 / 2.913.1 / 3.8
4B5.36 / 1.9227.4 / 7.019.6 / 8.4
8B11.31 / 5.7632.7 / 7.732.7 / 8.4

The trained expert system outperforms the prompted base system at every scale on both splits, and the distilled HARP student also exceeds the prompting baseline in all six comparisons. These results support the value of training the experts as a group under Router3. Establishing the contribution of each expert’s training would require separate ablations.

In the revised manuscript, we introduce RolePrompt in Section 5.1 (Experimental Setup), report and discuss its results in Section 5.2 (Main Results) and Table 1, and include the role-specific instructions and few-shot examples in Appendix E. These additions are highlighted in burgundy.

[Weakness 2] Although the Executor introduces a Progress-Based Reward, the authors do not investigate whether directly training a single model with the same Progress Reward could achieve similar performance.

[Question 2] Are HARP's performance improvements primarily driven by the Progress-Based Reward introduced for the Executor rather than Role-Specialized Training itself? I suggest adding a GRPO baseline that directly trains a single model using the same Progress Reward, with a comparable training budget.

Thank you for suggesting this control. We trained a single Qwen3-4B policy from the base checkpoint for 60 GRPO steps using the Executor’s hybrid reward, which combines terminal task success with the verifier-based progress signal. The results are reported as task goal completion (TGC).

Qwen3-4B methodTest-NTest-C
GRPO (reported)13.1%5.3%
Base + hybrid-reward GRPO (60 steps)14.3%6.5%
HARP (distilled student)19.6%8.4%

The hybrid-reward baseline scores 1.2 percentage points above the reported GRPO baseline on both splits. HARP remains ahead by 5.3 points on Test-N and 1.9 points on Test-C. In this single-seed comparison, the hybrid-reward baseline narrows the gap to HARP on both splits.

Due to time constraints, we trained the hybrid-reward baseline at only one model size, using 4B as a representative case. We will report results for all three model sizes in the camera-ready version.

[Weakness 3] The Repairer is fine-tuned using only 17 repair examples through SFT, making it difficult to establish its generalization ability across diverse types of errors.

[Question 1] Are only 17 repair examples sufficient to learn a generalizable Recovery Capability?

Thank you for raising the question of whether 17 repair examples are sufficient for generalization. The 17 examples provide repair-specific adaptation for a policy initialized from the Planner. We further adapt this policy with LoRA using 17 repair examples from 13 tasks, mixed with 34 ordinary planning examples. The objective is to specialize an existing planning policy for correction after observed failures.

To test transfer on held-out 4B tasks, we sampled one repair activation from each of 526 test trajectories in which Router3 invoked the Repairer and obtained 520 evaluable pairs. For each pair, we restored the same pre-activation state and assigned the next planning turn to either the Repairer or the regular Planner. Both branches then continued with the same Planner and Executor, with further repair disabled and the same remaining action budget. We measured terminal success using AppWorld’s official state-based tests independently of the progress verifier.

The Repairer succeeded in 56 of 520 pairs (10.77%), compared with 46 of 520 (8.85%) for the Planner. The Repairer rescued 13 cases and caused 3 otherwise successful cases to fail, yielding a net gain of 1.92 percentage points. This shows that the trained Repairer can help on some held-out 4B tasks.

In the revised manuscript, Section 5.4 (Routing and Verifier Diagnostics) summarizes the paired replay results. Appendix C.7 (Evaluation Details) presents the replay protocol, independent scoring, and per-model results after the Repairer routing diagnostic. These additions are highlighted in dark blue.

Follow-up on Weakness 2 and Question 2

Thank you again for your thoughtful feedback. Following up on W2 and Q2, we have now completed the base-initialized, single-policy hybrid-reward GRPO experiments for all three model sizes. Each model was trained for 60 steps. The table reports task goal completion (TGC, %) on Test-N and Test-C.

Qwen3 sizeGRPO (Test-N / Test-C)Base + hybrid-reward GRPO, 60 steps (Test-N / Test-C)HARP (Test-N / Test-C)
1.7B0.6 / 0.00.00 / 0.7213.1 / 3.8
4B13.1 / 5.314.29 / 6.4719.6 / 8.4
8B36.3 / 12.029.17 / 9.3532.7 / 8.4

At 1.7B and 4B, the two model sizes where HARP outperforms the reported GRPO baseline, base + hybrid-reward GRPO remains below HARP on both test splits, suggesting that HARP’s gains cannot be fully explained by the reward modification alone.

We have also incorporated these results into the revised manuscript, with the additions highlighted in burgundy. We would appreciate your thoughts on whether this additional comparison addresses your concern about the contribution of the progress reward.

Reviewer_5YbFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HARP, a framework for improving long-horizon tool-use agents through role-specialized training and on-policy distillation.

Critical strengths

Distinguishing planning, execution, and recovery responsibilities is an intuitive direction for improving long-horizon agents, where sparse terminal rewards provide limited guidance for learning from failures.

Critical weaknesses

My main concern is that the paper reduces the diverse and interconnected sources of agent failures to three manually defined categories: planning, execution, and recovery. However, failures in long-horizon tasks frequently involve dependencies across these components. An incorrect execution may originate from an incomplete plan, while a recovery failure may result from incorrect observations, missing information, or earlier decisions. Such errors cannot always be cleanly attributed to a single role.

More fundamentally, the proposed method does not actually identify the underlying cause of a failure. The phase classifier distinguishes planning and execution outputs, while the progress verifier determines whether an action advances the task. Neither establishes whether the failure originates from planning, execution, or recovery. Therefore, it is unclear whether HARP genuinely addresses heterogeneous credit assignment or simply trains separate models according to a manually designed decomposition.

Questions for authors

N/A

4 comments

[Weakness 1] My main concern is that the paper reduces the diverse and interconnected sources of agent failures to three manually defined categories: planning, execution, and recovery. However, failures in long-horizon tasks frequently involve dependencies across these components. An incorrect execution may originate from an incomplete plan, while a recovery failure may result from incorrect observations, missing information, or earlier decisions. Such errors cannot always be cleanly attributed to a single role.

Thank you for pointing out these dependencies. We agree that planning, execution, and recovery are interdependent. Our role-wise training addresses the optimization difficulty arising when interdependent decisions receive only a shared terminal reward. Section 4.2 defines Ft=(X,Ht,Gt,At,Ot+1)F_t=(X,H_t,G_t,A_t,O_{t+1}), which includes the task, interaction history, plan, executed action, and resulting observation. Section 4.1 further specifies that accepted outputs and observations remain in the shared history across role handoffs, preserving cross-role information.

Specifically, Section 4.3 first trains the Executor with the Planner frozen, then trains the Planner with the improved Executor frozen. Both stages use complete interaction trajectories and update only the active role’s tokens. A trajectory may contain both planning and execution weaknesses. We first improve execution using terminal outcomes supplemented by transition-level progress feedback, which can distinguish rollouts even when their terminal rewards are identical. We then optimize the Planner using terminal task reward, with the fixed Executor forming part of its interaction environment. This focuses optimization on planning under a stable execution policy. HARP thus structures the optimization of interdependent capabilities through shared context and sequential role-specific updates, without assigning each failed trajectory to a single cause.

[Weakness 2] More fundamentally, the proposed method does not actually identify the underlying cause of a failure. The phase classifier distinguishes planning and execution outputs, while the progress verifier determines whether an action advances the task. Neither establishes whether the failure originates from planning, execution, or recovery. Therefore, it is unclear whether HARP genuinely addresses heterogeneous credit assignment or simply trains separate models according to a manually designed decomposition.

We would like to clarify that our objective is to structure corrective supervision and policy optimization, rather than identify the underlying cause of each failure. Addressing heterogeneous credit assignment and using a manually designed decomposition are not mutually exclusive: the decomposition is the structure through which HARP makes supervision role-specific. This design is motivated by prior work: Aghzal et al. (2026) analyze planning, execution, and replanning as distinct capabilities and identify execution as a dominant bottleneck, while HiPER (Peng et al., 2026) demonstrates the value of aligning optimization with a planning–execution hierarchy.

HARP connects the roles to different learning signals and optimization stages. The Executor learns from terminal outcomes supplemented by transition-level progress feedback. The Planner then learns from terminal success with the improved Executor fixed, and the Repairer learns corrective plans checked through same-state replay (Sections 4.2–4.4). Both RL stages use complete interactions but update only the active role’s tokens. This targets the ambiguity over which capability an update should improve, without requiring each failed trajectory to have a unique causal label. The resulting experts provide role-conditioned token-level supervision on the student’s own trajectories during OPD (Section 4.6).

As discussed in Section 6 and Appendix D, the three roles constitute one useful decomposition for AppWorld. Deeper, task-dependent hierarchies and continuous representations of role requirements are future directions. Our contribution is to show how organizing teacher specialization around complementary responsibilities and consolidating their supervision through OPD provides a practical approach to training long-horizon agents.

Thank you again for your careful review and thoughtful feedback. We have provided point-by-point responses and added clarifications to address your concerns.

When you have a chance, we would greatly appreciate your feedback on our responses and the updated results. Please let us know if any concerns remain or if further clarification would be helpful. Thank you for your time and consideration.

I thank the authors for their thoughtful rebuttal and additional experiments.

I appreciate the clarification regarding role-specific supervision. However, my main concern remains: the manually designed decomposition does not convincingly demonstrate more effective credit assignment across interdependent failures. The inconsistent benefits of the Repairer and the lack of clear evidence isolating role specialization further limit the contribution.

I therefore maintain my rating of 2 (Weak Rejection). Thank you again for your efforts to strengthen the paper.

Thank you for considering our clarification and additional evidence. We would like to briefly highlight how the experiments connect role-specific supervision to task outcomes.

With Router3 fixed, trained experts outperform the RolePrompt baseline at every size on both splits, supporting the value of learned capabilities beyond prompted role separation. At 1.7B and 4B, HARP also exceeds single-policy GRPO trained with the hybrid progress reward, supporting benefits beyond reward shaping alone. Finally, the same-state replay experiment tests individual corrective decisions within an existing interaction history. At 4B, the Repairer rescues 13 cases that the regular Planner fails, against 3 cases in the opposite direction, yielding a net gain of 1.92 percentage points.

We believe these results provide empirical support for directing supervision toward complementary capabilities while preserving their shared context and dependencies. This is the practical contribution we seek to establish through HARP’s role-specific training and subsequent distillation into a single policy.

Thank you again for your thoughtful feedback and efforts to strengthen the paper. We respect your assessment and accept your decision to maintain the original rating.

Comments on the paper

1 comment

We thank the reviewers for their careful reading and constructive feedback. Long-horizon reinforcement learning often relies on a sparse outcome reward that provides little guidance about which decisions to improve. We propose HARP, which addresses this by using distinct learning signals for planning, execution, and recovery, then combining the resulting experts into a teacher that supervises a single model through on-policy distillation (OPD). Reviewer 5YbF described this separation as “an intuitive direction for improving long-horizon agents.” Reviewer G7Pk noted “substantial improvements on smaller models,” and Reviewer SYj6 called the organization of teachers around corrective responsibilities “a meaningful systems contribution.”

The reviews raise four main questions: whether the method accounts for dependencies among roles (5YbF), how much the individual components contribute beyond prompting or reward shaping (G7Pk W1/W2/Q2/Q3, SYj6 W1), how reliably the Repairer improves task outcomes and whether its benefits carry over to the student (G7Pk W3/Q1, SYj6 W1/W2/Q1/Q2), and what resources and training settings the comparison entails (SYj6 Q3).

We clarify that planning, execution, and recovery are responsibilities for corrective supervision, rather than mutually exclusive causes of failure. The experts share the interaction history, and each RL stage uses complete interactions while updating only the active role’s tokens. The Executor learns from terminal outcomes and transition-level progress, then the Planner learns from terminal outcomes with the improved Executor held fixed. This setup preserves dependencies across roles. The phase classifier identifies the current phase and the verifier estimates progress, so their signals guide routing without requiring a unique diagnosis for each failure. We also report the student’s SFT initialization, online rollout counts, and teacher preparation and annotation workloads in Appendix C.8.

We ran four additional experiments to test the contribution of expert training, the effect of progress rewards, and the reliability and transfer of repair.

  • Role-specific prompting. RolePrompt uses one base checkpoint with separate few-shot prompts for the three roles and the same Router3. Both the trained routed system and the distilled HARP student outperform it at all three sizes on both splits (Tables 1 and 2, Appendix E).
  • Single-policy hybrid reward. At 4B, applying the Executor’s hybrid reward to a single base policy improves on the reported GRPO baseline, from 13.1% to 14.3% on Test-N and from 5.3% to 6.5% on Test-C. HARP reaches 19.6% and 8.4%, respectively.
  • OPD without the Repairer. At step 3, full HARP leads the Planner–Executor student on both splits at 1.7B and 8B, while the Planner–Executor student leads at 4B. On 4B Test-N, the stronger routed teacher yields the weaker student. This comparison changes both the Repairer and the handoff rules (Section 5.3 and Table 2).
  • Independent repair audit. AppWorld’s official tests scored 1,572 paired replays from identical held-out states. Using the Repairer in place of the regular Planner rescued 22 cases and raised terminal task success by a net 0.32 percentage points. The largest observed gain was at 4B, where the Repairer rescued 13 cases and raised success by 1.92 percentage points (Section 5.4 and Appendix C.7).

These comparisons support the value of training the experts as a group and show that the 4B hybrid-reward baseline closes part of the gap between reported GRPO and HARP. They also document where the Repairer helps or hurts and show that stronger teacher performance need not translate into a stronger student. The revised PDF includes the RolePrompt baseline, the OPD ablation, the paired repair audit, and the additional training details. The additions are highlighted in burgundy and dark blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.