acceptodds
Under review as a conference paper at ICLR 2027

Can Vision-Language Models Learn to Execute Classical Search Algorithms?

Abstract

The automated planning community has probed the planning ability of large language models by fine-tuning them on planning-problem corpora, but the training target is almost always the plan (i.e., the action sequence a planner returns), rather than the search that found it. Yet classical planning offers many search algorithms, and how well a model learns them from their search traces is still an open question. As embodied agents increasingly plan from what they see, the question extends to vision-language models (VLMs). We fine-tune a VLM on the search traces of four classical algorithms across 15 PDDL domains and study learning at two levels. For breadth-first search (BFS) and best-first width search (BFWS), whose expansion order follows from bookkeeping rules, the model executes the algorithm, writing every step of the search, including where each new node goes in the open list. For greedy and weighted- best-first search, whose order follows the additive heuristic, the model makes the search-control decision, choosing which node to expand next from images of the open nodes, with no heuristic values shown. Execution is learnable but costly: from a small training set the model solves no tasks under either algorithm, and it reaches success of 1.0 for BFS and 0.93 for BFWS only with over an order of magnitude more data. Search control is learnable from images alone: on held-out tasks, the trained VLM solves 0.28 to 0.62 of tasks within twice the reference budget, against at most 0.04 for random valid choices. Training on width-based traces even transfers beyond planning, raising logical-reasoning accuracy on FOLIO from 0.600 to 0.660. Together, these results establish search traces as a practical supervision target for planning research, alongside plans.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language models, Search Process Policy, Typed Search Operations, classical planning, heuristic search, neural algorithm execution, process supervision, trusted runtimes, multimodal reasoning, runtime verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7wf1OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigate vision-language models can learn the search process of classical search algorithms. The authors study this problem at two levels: first, by training the model to execute the step-by-step operations of BFS and BFWS from search traces; and second, by asking the model to choose the next node to expand for GBFS and Weighted A* based only on images of candidate states, without access to explicit heuristic scores. The paper evaluates these settings across multiple PDDL planning domains and compares performance under small-scale and larger-scale training.

Critical strengths
  1. The research question is meaningful. Rather than focusing only on whether a model can produce a correct plan, the paper asks whether it can also learn the search process that leads to the plan. In particular, separating “executing a search algorithm” from “choosing the next node to expand” makes the experimental findings easier to interpret.
  2. The experimental design is reasonable. The model does not have access to explicit heuristic scores, and the order of candidate nodes is shuffled. Even under these settings, the fine-tuned model clearly outperforms both random selection and the untuned base model. This suggests that the model learns useful information from search traces about which states are more promising to explore, rather than relying on simple positional or formatting cues.
Critical weaknesses
  1. The BFS/BFWS execution setting relies heavily on the runtime. The runtime handles several important components, including successor generation and some algorithm-specific information. Therefore, the current results seem closer to showing that the model can produce correct operations within a constrained search interface, rather than independently implementing the full search algorithm.
  2. The conclusions about training data scale are still incomplete. The model fails with 512 training examples but succeeds with tens of thousands of examples, which is an interesting result. However, there are no intermediate data-scale experiments, and the small- and large-data settings are not fully matched. As a result, it is still difficult to determine the actual amount of data required or to make a clear comparison of whether BFS or BFWS is harder to learn.
Questions for authors

Could the authors provide additional experiments to address the weaknesses above

Reviewer_k24XOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks if a VLM can learn classical search operations and decisions from its traces. The paper studies the learning behavior of the VLM at two levels: 1. For a BFS/BFWS setting, it learns how to insert given successors to the search frontier given the algorithm’s rules, and 2. For a GBFS / weighted-A* setting the model chooses which frontier node to expand next based on images of the candidate states. The authors use LoRA to finetune a Qwen3-VL-8B model across 15 PDDL domains. They show that BFS/BFWS can be learned but with over an order of magnitude more data than a small training set, while GBFS / weighted-A* can be learned from images alone and beats random valid choice on held-out tasks. The paper also shows improvements on FOLIO after BFWS-trace training and claims transfer beyond planning.

Critical strengths
  • Compared to only providing supervision to final plan, possible supervision within search traces is interesting to explore and expand to the visual domains.

  • The failure analysis of the paper is informative and detailed.

Critical weaknesses
  • The contribution is not clear and strong enough. The system already has an exact PDDL transition model and executable classical search algorithms. For both settings, the transition model generates accurate successor nodes and only ask the VLM to make a selection. The introduce of VLM does not show clear additional capability. The existing transition model and search algorithms are enough to make valid and accurate planning. If the contribution claimed is to test whether the model learns the computations underlying search, an important thing to consider is: in real planning problems, an accurate simulator may not exist to provide successor nodes. If they exist, utilizing classical search algorithms would be enough to find the final solution. So, when they do not exist, the important capability a VLM need is to imagine what future states could result from an action or at least distinguish valid successor states from incorrect predictions. But in the paper, only accurate future states are provided, which makes the practical value of the learned search behavior unclear.
  • The generalization experiments are mainly about the performance to new tasks of seen domains. The paper does not report a clear evaluation on tasks from held-out domains to support broader generalization. Especially when the authors claim that BFS/BFWS can be learned but with over an order of magnitude more data, it is hard to tell whether the model has learned patterns specific to the training domains or generalizable search behavior.
  • The experiments only use synthetic 2D renderings of the planning tasks. No 3D or real world visual environments are tested.
  • There are limited or no comparison with other VLM-based planning methods
Questions for authors
  • More experiments on unseen planning domains are needed.
  • More experiments on 3D or real-world visual domains are needed.
  • Could the authors provide comparisons with the latest VLM-based planning methods?
  • Could the authors provide ablations about how much does the visual information contribute to the execution by removing the state images?
  • Could the authors clarify the first weakness by clearly stating the main contribution or providing more experiments to show the value of learned search behavior on realistic planning problems.
2 comments

We hope our responses and additional experiments have addressed your concerns. Please let us know if you have any remaining questions or feedback. We would greatly appreciate hearing from you sufficiently ahead of the rebuttal deadline so that we have time to address any outstanding issues.

Thank the authors for the honest responses and new experiments. However, the new results mostly confirm my concerns about the value of the learned search behavior and its generalization beyond the training domains. I appreciate the effort, but I will maintain my current rating.

Reviewer_rmXxOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether Vision-Language Models (VLMs) can learn classical search algorithms rather than just the final plans. The paper shows that VLMs can learn to execute algorithms like BFS and BFWS, and can also learn to directly select the next node in heuristic search algorithms such as GBFS and Weighted A*. It also finds that training on BFWS traces can improve performance on logical reasoning tasks like FOLIO.

Critical strengths
  • Learning the search process itself sounds like an interesting direction.

  • The paper covers several search algorithms across multiple domains.

Critical weaknesses
  • It is difficult to tell whether the improvements from small-data to large-data settings come from increased data diversity or simply more training compute. It might be worth including a compute-matched comparison with the same total training FLOPs.

  • I feel this paper is more about building a harness where an external runtime handles most of the search process, while the VLM only learns to make some of the decisions, rather than actually executing the entire search algorithm on its own. So the claim that VLMs can execute search algorithms feels a bit overstated.

Questions for authors
  • It is a bit strange that the BFWS-trained adapters in Table 2 fail the planning tasks but still show improvements on FOLIO. Also it would be interesting to see whether this improvement also holds for the larger-data adapters.

  • For GBFS and Weighted A*, would the model perform better if the candidate states were represented as text or multimodal inputs instead of just images?

1 comment

We hope our responses and additional experiments have addressed your concerns. Please let us know if you have any remaining questions or feedback. We would greatly appreciate hearing from you sufficiently ahead of the rebuttal deadline so that we have time to address any outstanding issues.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.