acceptodds
Under review as a conference paper at ICLR 2027

Training the Orchestrator: A Supervised Approach to End-to-End PDDL Planning with LLM Agents

Abstract

People state planning goals in natural language, while classical planners require formal PDDL. Recent agentic frameworks bridge this gap with a verifier-checked refinement loop in which an orchestrator picks, at every step, which repair agent (an LLM-prompted or deterministic PDDL and plan-repair routine) to run next; that orchestrator is itself a prompted frontier LLM. We present HALO (Hybrid Agent-Learned Orchestrator), which replaces it with a QLoRA-tuned 8B model (Llama-3-8B) trained on the agent choices of a prompted teacher, keeping only tra- jectories whose final plan an external validator accepts. This is outcome-filtered behavioural cloning: the validator certifies where a trajectory ended, not each step, yet it is enough to make imitation work, and removing the filter collapses performance. HALO places the policy behind three hardcoded rules for trivially decidable steps and selects among an expanded pool of 21 agents; the rules are essential, as the policy alone falls well short. Across 11 PDDL domains from PlanBench, Natural Plan, and classical planning, and measuring success as val- idator acceptance of the generated PDDL and plan, HALO matches the GPT-5- mini-orchestrated framework on PlanBench and classical planning, exceeds it by 37 points on Natural Plan, and stays within three points of Gemini-3-Flash, while cutting orchestration cost roughly 45× and 15× respectively and LLM calls per episode by 40–54%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
LLM Planning, Agent Orchestration, Local Models, Orchestrator Training, Planning, PDDL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oyUJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper replaces the orchestrator in an agentic framework that turns natural-language tasks into PDDL and plans. In that framework, a prompted frontier LLM decides at every refinement step which repair agent to run next, and a planner and validator then check the result. HALO replaces this orchestrator with three hardcoded rules and a QLoRA-tuned Llama-3-8B that outputs one agent ID. The model imitates GPT-5-mini and Gemini-3-Flash trajectories whose final plan the validator accepts. Success likewise means that the validator accepts the system's own PDDL and plan. With the repair agents fixed, HALO matches GPT-5-mini on PlanBench and the classical suite, exceeds it by 37 points on Natural Plan, stays close to Gemini-3-Flash, and cuts orchestration cost by about 45x and 15x.

Critical strengths
  1. The problem is clear, the motivation is good and the approach is easy to follow. Replacing only the orchestrator with a small model, while the agents keep using a large one, looks workable. A local single-token decision replaces a long-context API call at every step, which is a small, cheap and practical change.
  2. The experiments are clear and show what the design contributes. With the agents, planner and validator shared (Sec. 5.2), the ablations in Table 4 show that the verifier filter, the rules and the added agents each matter.
Critical weaknesses

Overall, the validator and cost issues need more explanation to support the core claims, and the benefits and blind spots of learning from the teachers are not clear enough (Question 4).

  1. Validator acceptance does not mean the result is correct.
    • In a sense the validator works like a compiler. It checks that the plan is valid for the PDDL the system wrote, not that this PDDL models the task correctly. Both the success metric and the hard training filter rest on this check (Sec. 4.2, Sec. 5.3), so what if the PDDL and the task differ in meaning?
    • One possible concern is that agents 7, 8 and 20 can make a task easier by stripping unsupported features, merging objects and adding initial-state facts (L453-455). If they are called often, the PDDL may drift from the task, and such shortcuts are a dangerous but common failure of agent systems. The paper notes that HALO may favour these agents more than the prompted orchestrators, but has not measured this (L455-457, Question 5).
  2. The cost claim covers only part of the system.
    • Besides the Llama orchestrator, two costs matter, the one-off cost of distillation and the cost of the Gemini repair agents under HALO. If these are large and orchestration is only a small part of the total, the reported saving may matter little.
    • The repair agents' LLM calls are excluded everywhere, since this cost differs only through the number of agent calls (L351-353). A prompted orchestrator makes one API call per step (L403-404), so on PlanBench GPT-5-mini's 4.8 calls in 2.8 steps leave 2.0 agent calls, while all 2.2 of HALO's calls go to agents (Table 2). HALO makes more agent calls than both baselines in every family, so the excluded cost is higher under HALO.
    • For distillation, the paper reports about 48 A100-hours of training but only mentions the API cost of collecting teacher trajectories and running the LLM judge, the main expense (L996-997). Questions 1 and 2 ask for both costs.
Questions for authors
  1. [Clarification] What was the API cost of distillation, for collecting teacher trajectories and running the LLM judge? Please also report the number of teacher rollouts, their acceptance rate and the share of each teacher in the training data. Lower cost is a key claim, and since HALO saves about 6 to 22 cents per task on orchestration (Table 2), this one-off cost decides how many tasks it takes to pay off.
  2. [Clarification] What do the Gemini repair agents cost per task under HALO and under each prompted baseline? Optimizing the orchestrator only pays off if the repair agents are a small part of the total cost.
  3. [Suggestion for future work] Llama-3-8B is an old model by now. Would a low-cost frontier model such as DeepSeek-V4.1-Flash, released in September 2026 at a fraction of Gemini-3-Flash's price, give better performance as the orchestrator, or even as the repair agents, without fine-tuning? If fine-tuning turns out to add nothing, that does not count against the paper, since its approach and pipeline do not depend on a model generation.
  4. [Requested analysis] How much of the teachers' ability does the small orchestrator reproduce, and where does it fall short? Without the rules, the policy alone loses 34 to 46 points and falls below both teachers (Table 4, L430-432), although the rules only cover decisions the paper calls trivially derivable (L267). On held-out states, how often does the policy pick the teacher's agent, and what do the disagreements say about the bottleneck of distillation? I think this matters beyond the results, since progress in base models will overtake the numbers but not the methodology.
  5. [Requested analysis] On a random Natural Plan subset, what is the gold-answer accuracy of all three orchestrators, and how often does each pick agents 7, 8 and 20? If HALO's accuracy stays close to its validator success, my first concern would be largely resolved.
1 comment
AuthorsOpenReview ↗

Response to reviewer oyuJ

We thank the reviewer for the detailed analysis, including the careful reading of the cost accounting.

Validator acceptance and task fidelity (main weakness, Q5). We agree. The validator checks that the plan is valid for the PDDL the system wrote, not that this PDDL models the task, and both our success metric and our training filter rest on that check. The concern about agents 7, 8 and 20 is well placed: all three can make a task easier, and we have not measured how often each orchestrator calls them. Since all three orchestrators share the same 21-agent pool, comparing how often each one selects these agents is the right test. We will report it together with gold-answer accuracy on a random Natural Plan subset for all three orchestrators, next to validator success.

Cost covers only orchestration (Q1, Q2). The reviewer is right that the headline saving is an orchestration saving only. Repair-agent calls are excluded from the cost figure, and they can differ across orchestrators through the number of agent calls each one makes. We will report total per-task cost, with the repair agents included, for all three orchestrators. We will also report the one-off cost of distillation: the number of teacher rollouts, their verifier acceptance rate, each teacher’s share of the training data, the API cost of collecting trajectories and of running the LLM judge, and the number of tasks needed to recover that cost at the measured per-task saving. Newer base models (Q3). Agreed. Nothing in the pipeline depends on Llama-3-8B, and a stronger low-cost model used without fine-tuning is the right baseline for judging what fine-tuning adds. We will add this to the discussion.

What the policy reproduces (Q4). We will report top-1 agreement with the teacher on held-out states and analyse where the two disagree. One point on the trained-only row of Table 4: steps resolved by a rule are excluded from the supervised training set, so the policy never sees cold-start, syntax-error, or already-valid states during training. Removing the rules at test time evaluates it on states absent from its training data. The 34 to 46 point drop therefore reflects that distribution shift as much as the policy’s ability, and we will add a policy trained with those steps included to separate the two.

Reviewer_D9paOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes HALO, which replaces the frontier model used as the orchestrator of an agentic NL-to-PDDL framework with a small local model. At each step, HALO picks which repair agent (function call) to run next from a fixed set. Three hardcoded rules handle the trivially decidable steps before the policy is called. The policy is trained on the agent choices of more powerful teacher models, and these choices are further curated, keeping only the trajectories whose final PDDL and plan are accepted by a sound off-the-shelf validator. On PlanBench, Natural Plan and classical planning benchmarks, HALO matches or outperforms GPT-5-mini and stays within three points of Gemini-3-Flash, while cutting the orchestration cost significantly.

Critical strengths
  • The idea is simple and practical. Replacing an expensive frontier model used as the orchestrator with a local model makes a lot of sense, and so does using the validator to curate data for SFT.

  • The design is creative: a closed action space of agent IDs with a single-token target, plus a thin rule layer for the trivial decisions. (Some of it comes from previous work, but it is creative nonetheless.)

  • The paper is well written and easy to follow. I also appreciated that the authors are explicit about what their success metric does not measure (validator acceptance is not fidelity to the task).

  • The results are good. HALO matches or beats GPT-5-mini at a fraction of the cost, and the ablations show that both the verifier filter and the rules matter.

Critical weaknesses

My main concern is the lack of out-of-distribution evaluation. The test set consists of every non-seed problem from the same 11 domains. The held-out experiment (Table 3) holds out two domains with obfuscated vocabulary, but each of them has a structural sibling in the training set (as the authors acknowledge). In learning for planning, the interesting question is how the method behaves on problems larger than the ones seen during training. It is known that gains from supervised training on planning data mostly vanish there (e.g., [1] reports BlocksWorld dropping from 98.5% to 13.5% on longer plans). It is unclear whether the drop would be larger or smaller for orchestration.

I also find it peculiar that reasoning models were not used, in particular for the SFT/distillation, given that it has worked well in so many reasoning problems despite being unrealiable. I would appreciate an explanation on why of this choice.

(In my head, there is also a more pragmatic reason for not distilling with CoT: closed providers do not expose the CoT of their frontier models. My 2 cents here is to try Gemma 4 31B, which has a free-tier API and returns its CoT through the Gemini API.)

Lastly, I am not fully convinced about the novelty of the submission. Distilling a stronger teacher into a smaller model using a small curated set is known to work well (e.g., [4]). Even for end-to-end planning with LLMs, a harder problem according to the motivation of your submission, it has been shown that fine-tuning on verified trajectories improves in-distribution performance [1,3], and that this can be achieved even without a teacher model, through a simple self-distillation mechanism [2]. So the key result is expected: if planning directly improves in this setting, choosing which repair agent to call next should improve as well.

Overall, I am positive about the paper, but in its current state I would rate it as borderline.

[1] Huang et al. "Chasing Progress, Not Perfection: Revisiting Strategies for End-to-End LLM Plan Generation". ICAPS 2025.

[2] Corrêa et al. "Iterative Deployment Improves Planning Skills in LLMs". arXiv:2512.24940. 2025.

[3] Rossetti et al. "Learning General Policies for Planning through GPT Models". ICAPS 2024.

[4] Muennighoff et al. "s1: Simple test-time scaling". arXiv:2501.19393. 2025.

Questions for authors
  • How does HALO perform on problems larger than the ones used for training?

  • Have you tried retraining HALO on its own curated episodes (i.e., self-distillation)? For example, starting from very easy ones where even your base model can solve, and then SFT on top of that.

  • Have you tried doing the SFT/distillation with reasoning tokens (CoT) as well (as in [4])?

2 comments
AuthorsOpenReview ↗

Response to reviews Dp9a

We thank the reviewer for the positive assessment and the useful pointers.

Larger problems than in training. We agree this is the most informative axis, and that the current held-out test, where each withheld domain has a structural sibling in training, does not address it. One structural difference from [1] is relevant. In HALO the plan is produced by Fast Downward, not by the language model, so plan length does not enter the orchestrator’s decision directly. What the orchestrator has to handle is the type and number of PDDL modelling errors, which need not grow with problem size. This is a hypothesis we have not tested. The classical suite already provides difficulty tiers (Blocksworld easy to hard, Hanoi easy to extreme), and we will add a split that trains on the smaller tiers and tests on the larger ones.

Reasoning tokens and CoT distillation. We have not tried this. The single-token target was a deliberate choice: the sub-10 ms decision latency and the cost argument both rest on the orchestrator emitting one token, and distilling reasoning traces would multiply decode length at every decision. As the reviewer notes, the closed teachers also do not expose their reasoning. Distilling from an open reasoning model, such as the Gemma model suggested, would be a useful comparison against the single-token policy, and we will include it in the discussion of future work.

Novelty. We agree that fine-tuning on verified trajectories is established for end-to-end plan generation [1, 3] and for reasoning [4], and we will cite and discuss these works. Our contribution is narrower: applying this idea to the orchestration decision of an agentic framework, where the action space is a closed set of agents and supervision is a single token per decision. Within that setting, the ablations show that the verifier filter is what makes the supervised signal usable (94.9 to 39.6 success on PlanBench without it), and the split between rules and the learned policy means the policy is trained only on the ambiguous decisions. We will state this scope explicitly in the introduction.

Thank you for your response. I am still positive about this submission, and I will keep my recommendation as "weak acceptance".

Reviewer_MAPvOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents HALO, a hybrid orchestrator for an iterative natural-language-to-PDDL planning system. The underlying system maintains the current PDDL domain/problem, candidate plan, planner/validator feedback, and repair history, and repeatedly chooses one of 21 specialized repair agents. HALO replaces a prompted frontier-model orchestrator with (i) three deterministic rules for cold start, syntax-error handling, and termination after validation, followed by (ii) a QLoRA-tuned Llama-3-8B policy that emits one agent ID. The policy is trained by behavioral cloning from GPT-5-mini and Gemini-3-Flash trajectories, retaining only trajectories whose terminal generated PDDL and plan are accepted by a symbolic validator; the retained data are augmented and additionally filtered by an LLM judge.

Critical strengths

None

Critical weaknesses
  • The main comparison does not isolate the learned orchestrator from the hardcoded rules. HALO receives three Layer-1 rules, whereas both prompted baselines must make those decisions themselves. This is not a small implementation detail: removing the rules lowers success by 33.6–46.5 points in Table 4, and the learned policy alone is much worse than both prompted orchestrators. Thus Table 2 compares a rule-plus-learned hybrid against prompted models without the same rules, rather than comparing alternative policies under an otherwise identical control flow. The paper appropriately describes this limitation, but without prompted-plus-rules baselines it remains unclear how much of HALO's accuracy and call reduction comes from learning rather than deterministic handling of easy decisions. This weakens the attribution behind the headline claim that training the orchestrator yields near-frontier performance.
Questions for authors
  1. Can the authors report correctness against the original task—reference PDDL or plan-set comparison where available, and Natural Plan's gold outputs—rather than only validation under generated PDDL?
  2. What are the success, call, and end-to-end cost results when GPT-5-mini and Gemini-3-Flash receive exactly the same cold-start, syntax, and termination rules as HALO?
  3. What is the implemented transition interface for plan-repair agents 13–17? Algorithm 1 receives only updated domain/problem PDDL and immediately replans, apparently discarding direct plan edits. Please reconcile this with the implementation, the NoOp descriptions in Appendix A.4, and the reported call accounting (for HALO, mean API calls exceed mean refinement iterations in all three families).
1 comment
AuthorsOpenReview ↗

Response to reviewer

We thank the reviewer for the careful reading and for the specific questions. We respond to the main weakness first, then to Q1 and Q3. Rules are not given to the prompted baselines (main weakness, Q2). This is a fair criticism. Table 2 compares a rule-plus-learned hybrid with prompted orchestrators that make every decision themselves, so it does not separate what the learned policy contributes from what the rules contribute. We will add prompted baselines that receive exactly the same three Layer-1 rules, and until those results are in we will not attribute HALO’s accuracy to learning alone.

Two points on how read the current results in the meantime.

First, the trained-only row of Table 4 understates the learned policy. Steps resolved by a rule are excluded from the supervised training set (Sec. 4.2), so the policy never sees cold-start, syntax-error, or already-valid states during training. Removing the rules at test time evaluates it on exactly those states. Part of the 34 to 46 point drop is therefore a distribution shift created by the ablation, not a measure of the policy’s ability. A policy trained with those steps included is the fairer comparison, and we will add it. Second, the cost comparison does not depend on this confound. The rules resolve roughly one-third of decisions, so giving them to a prompted orchestrator removes at most that share of its orchestration calls. That would leave GPT-5-mini at roughly 0.12 per task and Gemini 3 Flash at roughly 0.04 to 0.05, against 0.004 for HALO.

Correctness against the original task (Q1). We agree that validator acceptance shows the plan is valid for the generated PDDL, not that the PDDL models the task faithfully, and we did not measure fidelity directly. We will add gold-answer accuracy on Natural Plan, where gold outputs exist, and on PlanBench we will validate plans against the reference PDDL. Both will be reported next to validator success. Transition interface for agents 13 to 17 (Q3). Thank you for this; Algorithm 1 is incomplete. In the implementation, agents 13 to 17 return a repaired plan together with the unchanged domain and problem, and the refinement loop keeps that plan as the current plan rather than discarding it. Algorithm 1 shows only the branch for agents that modify the domain or problem, which is why it reads as immediate replanning. We will add the plan-repair branch to Algorithm 1 and correct the NoOp description in Appendix A.4 to match.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.