- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper examines how repeated spatial structure affects human decision-making in a maze navigation task with partial observability, postulating that human search in a structured environment is not approximated by local search methods.
The authors propose the GenPlan framework, in which an LLM first derives a programmatic decomposition of a map into repeated units, inspired by compositional accounts of human cognition, which is then passed to a structure-based planner combining POMCP-based search within repeated units with between-unit transitions.
Experiments aim to verify how closely navigation approaches used by 30 human crowd workers match planning policies with (Gen-POMCP) and without (DepthPlan, Naive-POMCP) map decompositions into structural sub-units, focusing in particular on discriminating decision points where planning policies diverge from human behavior.
Results show that Gen-POMCP outperforms both DepthPlan and Naive-POMCP in predicting recruited participants' behavior. Simulations on ten larger mazes show better exploration per rollout, with performance regressing toward Naive-POMCP as the proportion of unobserved structured units decreases with exploration.
Critical strengths
-
The proposed GenPlan approach is well-motivated by observations of compositional structures in human cognition, and outperforms the reported baselines in predicting actual human behavior from paid crowd workers in the evaluated MST setting (80.7% vs 42.2% for DepthPlan and 42.5% for Naive-POMCP). These results provide evidence that explicit structural biases can better approximate human search decisions.
-
The additional ablation on an end-to-end LLM-based planner without program induction (LLM-Direct, Appendix I) shows that a naive LLM use is not sufficient for robust navigation, motivating the use of explicit planning modules adopted in the main experimental section.
Critical weaknesses
-
The central behavioral comparison remains insufficiently specified. Because agreement with human behavior is the core evaluation metric, the discussion should clearly present disagreements between human participants to contextualize results against baseline population variance. This under-specification also appears in Appendix I, where it is unclear whether the weak performance observed across several LLMs and reasoning budgets can be attributed to a suboptimal problem formulation. While Appendix E shows examples of decomposed units from the map, the paper does not include an explicit ablation study showing how different decomposition quality affects the structure-based planner's performance, particularly for the less well-defined policy ruling transitions between repeated units. Without these clarifications, it is difficult to assess whether the authors' comparison is meaningful and reproducible.
-
The proposed Gen-POMCP strategy proves especially effective only for partially observable navigation scenarios in environments with known structural boundaries and repeated structural units. As the authors note in Section 3.2, their method reverts to naive global planning when such regular structures are absent. While the results seem promising in the handcrafted MST settings the authors study, I believe this setup is too narrow to provide convincing evidence that LLM-driven decomposition strategies could be applicable to more realistic planning scenarios.
Questions for authors
- Could you clarify the prompting setup that was used for the LLM-Direct evaluation? In particular, how was the current observation sent to the model? Were models asked to take one action at a time to account for updated observations, or were they planning entire trajectories from their initial observations?