acceptodds
Under review as a conference paper at ICLR 2027

Programmatic Search Policies For Planning

Abstract

We study the problem of transferring knowledge from solved embodied planning problems to related future problems. Embodied planning problems are typically solved via bi-level planning, where a high-level planner explores an environment by proposing sub-goals, which are then realized by a low-level control system. We are interested in the setting where the space of sub-goals must be rich enough to encode useful sub-goals and where the low-level control system cannot realize all sub-goals; this setting makes the low-level control system the dominant cost during exploration. Thus, the key challenge is to propose useful sub-goals that allow the planner to efficiently find trajectories that maximize reward. We refer to a system that generates sub-goals during exploration as a search policy. We describe an algorithm that synthesizes programmatic search policies — programs that generate candidate sub-goals for a given state, which drive the planning process. The search policies are represented as Python programs, which allows them to generate sub-goals in an expressive language and delegate some computation to the program interpreter, and they are amenable to being efficiently updated to novel environments via Language Models. Thus, by synthesizing a programmatic search policy, our proposed algorithm stores the knowledge acquired during the exploration in a representation that allows easy adaptation to novel tasks. We empirically show, in simulated environments, that the policies synthesized by our algorithm can be used to transfer knowledge across tasks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
sequential decision making, programmatic policy, library learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YMnUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an algorithm that generate useful subgoals during exploration, aka a search policy, represented by Python programs.

Critical strengths
  • The work resonates with code-as-policy (also a critical missing reference) and, in some way, can be thought of as code-as-search-policy, which is an interesting domain to study.
  • The replay scoring (4.2) is a neat idea.
  • The reporting of results, interpretations, and claims are sound and careful.
Critical weaknesses
  • The main concern is the task difficulty, and due to the simplicity of the evaluated tasks, some claims are weakened. The three tasks (Sokoban, Minigrid, BoxDelivery) are too simple so 1) no transfer baseline solves them quickly as well; 2) it is uncertain how the method would generalize to more challenging and interesting domains, such as those in mainstream EAI benchmarks. My understanding is that BoxDelivery is the only domain that fits your assumptions: contact-rich, hence imperfect low-level controls, but there are only two boxes and the instances only differ by their poses.
  • Needs ablation studies on critical designs; given the tasks, it shouldn't be too difficult to do so.
Questions for authors

My main concern is about tasks; see above.

Reviewer_pnszOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a hierarchical system with an LLM as a high-level subgoal proposal in the form of python code, that propose sub-goals for low-level skill policies. The skills are assumed to be imperfect (as in they cannot solve any subgoal) and environment evaluations are assumed to be expensive. The method is evaluated in puzzle-like environments with limited low-level compute budget.

Critical strengths
  1. On the high level the motivation is well explained and the approach is interesting.
  2. The theme of using LLMs in complex embodied tasks is highly relevant
  3. The transfer setting is unusual but for that reason could be interesting
Critical weaknesses
  1. Experiments overall are not that convincing. There is no result on established benchmarks. The 3 tasks are either proposed by the authors or puzzle-like toy tasks set-up in a way to support the method (e.g. arbitrarily set A* planning budget to imitate an imperfect low-level policy). It’s unclear to me whether such approach would remain relevant in settings of real world interest.
  2. There is also no established baseline, only variants of the proposed method.
  3. The technical novelty over straightforward combination of prior methods is hard to appreciate. More specifically, the combination of controller-aware learning and programmatic transfer appears achievable through an adaptation of existing LLM program-synthesis and execution-feedback methods. The paper’s contribution therefore seems to be recorded-graph selection of program revisions. However, the current comparisons do not establish its advantage over a reusable subgoal program repaired directly from controller feedback under the same evaluation budget.
Questions for authors
  • How would the results change if we allow much longer planning budget to A*?
1 comment

Thank you for the rebuttal, i appreciate the effort. The paper would improve with the new evidence, i think this is an interesting direction, but overall the paper remains under the bar for acceptance. Keeping my score

Reviewer_B9fMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to reuse experience across related planning tasks when evaluating sub-goals with a low-level controller is expensive. The setting provides structured states, a domain-specific goal vocabulary, a low-level controller, and the ability to restore previously visited simulator states.

The method uses an LLM to generate and revise a Python program that maps states to ordered lists of sub-goals. Search combines these proposals with direct LLM proposals and records their outcomes in a planning graph. Candidate revisions are scored by replaying them against recorded transitions, without additional controller calls. The learned program is transferred to another task instance and adapted further.

Experiments cover MiniGrid, Sokoban, and a simulated quadruped box-delivery task. Program transfer improves early progress in MiniGrid and outperforms the implemented ACE-style text-memory baseline. Improvements over no transfer are mixed in Sokoban, while the BoxDelivery result contains only two transfer pairs.

Critical strengths
  • Reusing recorded planning outcomes to score program revisions is a useful way to avoid additional expensive controller calls during candidate selection. The program representation also allows a revision to change how sub-goals are computed across states.
  • The comparison with the same learner initialized without a transferred program is appropriate for testing experience reuse. The MiniGrid results show that transfer can accelerate early planning progress.
Critical weaknesses
  1. The paper does not clearly establish what it adds beyond existing methods. Code as Policies and Voyager already use LLM-generated programs, and Voyager revises and reuses code based on experience. Prior work on lifelong task-and-motion planning also learns from previous planning attempts to guide future search. The main distinction here is selecting sub-goal programs by replaying them on recorded planning graphs. However, the experiments do not isolate the benefit of this selection procedure or show whether higher replay scores predict better planning performance. Replay stops when an outcome is unknown, so it can assess only behavior covered by the graph. The ACE-style comparison does not answer this question because it changes both the memory representation and the update procedure.
  2. The experiments cover a narrow range of tasks, and the improvements are not consistent. Transfer mainly involves changing layouts in six-room navigation, three-box Sokoban, and two-box delivery. The method receives structured states, domain-specific predicates, and a low-level controller. Even with these inputs, the experiments do not show whether it can reuse knowledge across tasks with more complex dependencies or different combinations of behaviors. In Figure 3, MiniGrid shows an early advantage, but the curves later converge. In Sokoban, transfer improves 19 of 32 pairs, worsens 12, and ties one. The authors report that the sign test does not distinguish this result from chance. BoxDelivery has a challenging control problem, but its result contains only two transfer pairs. These results are not enough to establish reliable transfer or a substantial improvement over existing methods.
Questions for authors
  • Does selecting revisions by graph replay improve planning compared with a simpler selection rule? Have you checked whether candidates with higher replay scores achieve greater reward when actually evaluated under the same controller-call budget?
  • Have you evaluated transfer on more varied tasks with more complex sub-goal dependencies? A household benchmark such as Habitat’s rearrangement tasks or BEHAVIOR would be useful here. The method could retain structured state access. These experiments could test whether learned programs transfer across different task structures. The current experiments mainly test changes in layout.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.