- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper replaces the orchestrator in an agentic framework that turns natural-language tasks into PDDL and plans. In that framework, a prompted frontier LLM decides at every refinement step which repair agent to run next, and a planner and validator then check the result. HALO replaces this orchestrator with three hardcoded rules and a QLoRA-tuned Llama-3-8B that outputs one agent ID. The model imitates GPT-5-mini and Gemini-3-Flash trajectories whose final plan the validator accepts. Success likewise means that the validator accepts the system's own PDDL and plan. With the repair agents fixed, HALO matches GPT-5-mini on PlanBench and the classical suite, exceeds it by 37 points on Natural Plan, stays close to Gemini-3-Flash, and cuts orchestration cost by about 45x and 15x.
Critical strengths
- The problem is clear, the motivation is good and the approach is easy to follow. Replacing only the orchestrator with a small model, while the agents keep using a large one, looks workable. A local single-token decision replaces a long-context API call at every step, which is a small, cheap and practical change.
- The experiments are clear and show what the design contributes. With the agents, planner and validator shared (Sec. 5.2), the ablations in Table 4 show that the verifier filter, the rules and the added agents each matter.
Critical weaknesses
Overall, the validator and cost issues need more explanation to support the core claims, and the benefits and blind spots of learning from the teachers are not clear enough (Question 4).
- Validator acceptance does not mean the result is correct.
- In a sense the validator works like a compiler. It checks that the plan is valid for the PDDL the system wrote, not that this PDDL models the task correctly. Both the success metric and the hard training filter rest on this check (Sec. 4.2, Sec. 5.3), so what if the PDDL and the task differ in meaning?
- One possible concern is that agents 7, 8 and 20 can make a task easier by stripping unsupported features, merging objects and adding initial-state facts (L453-455). If they are called often, the PDDL may drift from the task, and such shortcuts are a dangerous but common failure of agent systems. The paper notes that HALO may favour these agents more than the prompted orchestrators, but has not measured this (L455-457, Question 5).
- The cost claim covers only part of the system.
- Besides the Llama orchestrator, two costs matter, the one-off cost of distillation and the cost of the Gemini repair agents under HALO. If these are large and orchestration is only a small part of the total, the reported saving may matter little.
- The repair agents' LLM calls are excluded everywhere, since this cost differs only through the number of agent calls (L351-353). A prompted orchestrator makes one API call per step (L403-404), so on PlanBench GPT-5-mini's 4.8 calls in 2.8 steps leave 2.0 agent calls, while all 2.2 of HALO's calls go to agents (Table 2). HALO makes more agent calls than both baselines in every family, so the excluded cost is higher under HALO.
- For distillation, the paper reports about 48 A100-hours of training but only mentions the API cost of collecting teacher trajectories and running the LLM judge, the main expense (L996-997). Questions 1 and 2 ask for both costs.
Questions for authors
- [Clarification] What was the API cost of distillation, for collecting teacher trajectories and running the LLM judge? Please also report the number of teacher rollouts, their acceptance rate and the share of each teacher in the training data. Lower cost is a key claim, and since HALO saves about 6 to 22 cents per task on orchestration (Table 2), this one-off cost decides how many tasks it takes to pay off.
- [Clarification] What do the Gemini repair agents cost per task under HALO and under each prompted baseline? Optimizing the orchestrator only pays off if the repair agents are a small part of the total cost.
- [Suggestion for future work] Llama-3-8B is an old model by now. Would a low-cost frontier model such as DeepSeek-V4.1-Flash, released in September 2026 at a fraction of Gemini-3-Flash's price, give better performance as the orchestrator, or even as the repair agents, without fine-tuning? If fine-tuning turns out to add nothing, that does not count against the paper, since its approach and pipeline do not depend on a model generation.
- [Requested analysis] How much of the teachers' ability does the small orchestrator reproduce, and where does it fall short? Without the rules, the policy alone loses 34 to 46 points and falls below both teachers (Table 4, L430-432), although the rules only cover decisions the paper calls trivially derivable (L267). On held-out states, how often does the policy pick the teacher's agent, and what do the disagreements say about the bottleneck of distillation? I think this matters beyond the results, since progress in base models will overtake the numbers but not the methodology.
- [Requested analysis] On a random Natural Plan subset, what is the gold-answer accuracy of all three orchestrators, and how often does each pick agents 7, 8 and 20? If HALO's accuracy stays close to its validator success, my first concern would be largely resolved.
Response to reviewer oyuJ
We thank the reviewer for the detailed analysis, including the careful reading of the cost accounting.
Validator acceptance and task fidelity (main weakness, Q5). We agree. The validator checks that the plan is valid for the PDDL the system wrote, not that this PDDL models the task, and both our success metric and our training filter rest on that check. The concern about agents 7, 8 and 20 is well placed: all three can make a task easier, and we have not measured how often each orchestrator calls them. Since all three orchestrators share the same 21-agent pool, comparing how often each one selects these agents is the right test. We will report it together with gold-answer accuracy on a random Natural Plan subset for all three orchestrators, next to validator success.
Cost covers only orchestration (Q1, Q2). The reviewer is right that the headline saving is an orchestration saving only. Repair-agent calls are excluded from the cost figure, and they can differ across orchestrators through the number of agent calls each one makes. We will report total per-task cost, with the repair agents included, for all three orchestrators. We will also report the one-off cost of distillation: the number of teacher rollouts, their verifier acceptance rate, each teacher’s share of the training data, the API cost of collecting trajectories and of running the LLM judge, and the number of tasks needed to recover that cost at the measured per-task saving. Newer base models (Q3). Agreed. Nothing in the pipeline depends on Llama-3-8B, and a stronger low-cost model used without fine-tuning is the right baseline for judging what fine-tuning adds. We will add this to the discussion.
What the policy reproduces (Q4). We will report top-1 agreement with the teacher on held-out states and analyse where the two disagree. One point on the trained-only row of Table 4: steps resolved by a rule are excluded from the supervised training set, so the policy never sees cold-start, syntax-error, or already-valid states during training. Removing the rules at test time evaluates it on states absent from its training data. The 34 to 46 point drop therefore reflects that distribution shift as much as the policy’s ability, and we will add a policy trained with those steps included to separate the two.