- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks whether reusable domain-level guidance can improve the classical planning performance of open-weight language models. For each PDDL domain, a frontier model first writes a Python solver and then translates it into a natural-language strategy. A 20B or 31B model receives that strategy with each problem instance, generates a plan, and revises it when the VAL validator identifies an invalid action or an unmet goal. The authors evaluate this workflow on 360 IPC instances and 270 instances from three newer domains. Adding the strategy substantially increases the number of valid plans produced by both models; validator feedback provides a further gain. The paper also compares the resulting coverage with classical planners and frontier language models.
Critical strengths
- The paper measures valid plans and isolates the effect of adding a strategy. VAL checks the generated action sequences rather than relying on an LLM judge. Table 2 compares the same student models with and without the domain strategy at matched call budgets, and separately compares independent attempts with feedback-based revision. The resulting coverage gains are credible evidence that the supplied guidance helps these models.
- The evaluation extends beyond a small demonstration domain. It includes eleven domains, and per-domain results that reveal both gains and failures. The newer domains are particularly useful for testing whether the strategy helps on tasks the student models are unlikely to have memorized.
Critical weaknesses
- The experiments do not establish the value of the proposed strategy-generation method over simpler reusable skills. A domain strategy is, in practical terms, a reusable natural-language skill. The principal ablation compares this skill with no domain-specific skill; it does not compare against a guide written directly by the frontier model without the intermediate Python program, or an adapted prior strategy-plus-feedback method. The paper itself identifies closely related work using a stronger model’s strategy and validator feedback. These omissions make it difficult to identify what the proposed method contributes beyond providing reusable instructions.
- Limited methodological novelty beyond a reusable skill. The domain strategy is a domain-specific natural-language instruction sheet reused across instances—functionally a skill supplied in the prompt. The other component is revising an invalid plan using validator feedback. The paper identifies prior planning work that already combines a stronger model’s strategy with feedback-based revision. Its clearer contribution is evaluating this recipe with open-weight models across more and harder domains, rather than introducing a distinct agent mechanism.
- The practical “minimal small-model” claim is narrower than the framing suggests. The students have 20B and 31B parameters and already possess substantial capability; the paper does not show whether the approach enables genuinely resource-constrained models to plan. The authors acknowledge that running the 31B model requires GPUs with tens of gigabytes of memory. The comparison with frontier models also mixes inference budgets: the proposed agent may use five or up to 42 calls per instance, whereas the imported frontier results use one.
Questions for authors
- What test coverage does each teacher-generated Python program achieve on the same 11 domains?
- Under the same student model, validator, and call budget, how do the proposed strategies compare with domain guides written directly by the frontier model, expert-written guides, or an implementation of the closest prior strategy-plus-feedback approach?
- How does the method perform with a substantially smaller student model?
2 comments
Follow-up on our rebuttal
Dear Reviewer LwXm,
Thank you again for your constructive review. In the rebuttal, we ran the experiments you asked for—testing the strategy as a program, as a translated program (paper), and as natural language—and extended our study to a much smaller model, Gemma-4-E4B. We also showed that the agent remains several times cheaper than GPT-5 and Gemini 3.1 Pro when every call is charged. Finally, we clarified how our work differs from Borro et al. (2025).
As the discussion period closes, please tell us if any concern remains. If your concerns are resolved, we kindly ask you to consider updating your score.
Best regards, The Authors
Response to Reviewer LwXm: contribution, strategy forms, a smaller model, and inference budgets
Thank you for your review and for your suggestions! We clarify your questions below.
W1 and W2: Contribution Relative to Borro et al. (2025)
Borro et al. (2025) show that strategies help proprietary o1 models plan in Blocksworld. Their evaluation covers this single domain, with instances of 5 to 19 blocks, and they point to two open questions that our study addresses.
First, they state that "without exploring a wider range of domains, it is not possible to make any claims about the effect of applying the right strategy on solving increasingly large planning tasks." We evaluate eleven domains, with instances of up to 477 blocks in Blocksworld. Three of these domains were released after the LLMs were trained, which rules out data contamination on them.
Second, they state that "it is also important to explore domains that might not have clear and easily generalisable solutions." We include such domains: Sokoban, which is PSPACE-complete, and Cisterns.
We also analyze each component of the agent. We measure the strategy and the feedback separately, at three matched budgets, with two open-weight models from different families. We check each strategy before using it, through a program whose plans we validate on training instances. Unlike Borro et al. (2025), we find that feedback with the strategy solves more IPC instances than independent attempts: 146 against 134 with GPT-OSS-20B, and 225 against 210 with Gemma-4-31B.
The result is a small open-weight model at the level of planners and frontier LLMs. With up to five calls, Gemma-4-31B solves 225/360 IPC instances, more than LAMA (204) and GPT-5 (205). With up to 42 calls, it matches Gemini 3.1 Pro (245) and surpasses Scorpion Maidu (234), the winner of the IPC 2023 satisficing track. We will state this explicitly in the revision.
W1, Q1, and Q2: Strategy Forms
We compared three forms of strategy, all written by GPT-5.6 Sol:
The table reports feedback@5 coverage. E2E has no strategy.
Every form improves both models over E2E, so the gain comes from the strategy itself. The translation solves more instances than its program: 32 more with GPT-OSS-20B and 8 more with Gemma-4-31B, over all 630. On the IPC benchmark, it also gives the highest coverage of the three forms for both models.
Direct strategies reach comparable coverage, but only the program can be checked: we validate its plans on training instances with VAL before translating it. We will include this comparison in the revision. Executing the program instead would replace the planning LLM rather than evaluate it, and we did not evaluate expert-written guides.
W3 and Q3: Smaller Model
We also evaluated Gemma-4-E4B, a much smaller model, on the easy and medium instances: 240 from the eight IPC domains and 180 from the three novel domains. Each cell gives the instances solved without the strategy → with the strategy.
The strategy roughly doubles IPC coverage at every budget, and at feedback@5 it improves all eight domains. On the novel domains, the model solves at most 4 instances with or without the strategy. "Minimal" refers to the components added to the LLM, not to the hardware. We will add these results, keep the hardware limitation, and make this scope explicit.
W3: Inference Budgets and Cost
The agent makes more calls than the frontier models, which are called once per instance. We therefore report its coverage at each budget and charge every call, on solved and unsolved instances, plus the one-time cost of generating the eight strategies:
The agent costs less because Gemma-4-31B is a cheaper model. We will report the budgets and total costs explicitly and replace Figure 2(c) with this accounting.