POKE: Partial Order Knowledge Engineering for Large Language Models

To improve the rigid and step-by-step prompt engineering for large language models (LLMs), we propose the Partial Order Knowledge Engineering (POKE), which is a flexible and self-evolving planning-based prompt framework to handle the multi-hop inference. In our analysis of the manual prompts, the hand-crafted presentation of auxiliary knowledge is the key to the high-quality responses. The conventional prompt generation processes would be similar to solve the planning problems. In the POKE framework, we train a self-evolving planner to explicitly organize the knowledge of the LLMs. To apply POKE, the learned planner constructs a partial-order plan over the knowledge modules, the ordering constraints, and the causal links, after which an executor propagates the resolved intermediate plan and generates the prompt. We evaluate the POKE method on three atomic-probe subsets of TwoHopFact, MuSiQue, and 2WikiMultiHopQA. Experiments on the shared TwoHopFact subset prove that the POKE improves accuracy over the state-of-the-art methods. We also demonstrate that the POKE framework is capable to align with the physical laws and the structural constraints, by successfully fulfilling the physics-based simulation script generation tasks.