acceptodds
Under review as a conference paper at ICLR 2027

POKE: Partial Order Knowledge Engineering for Large Language Models

Abstract

To improve the rigid and step-by-step prompt engineering for large language models (LLMs), we propose the Partial Order Knowledge Engineering (POKE), which is a flexible and self-evolving planning-based prompt framework to handle the multi-hop inference. In our analysis of the manual prompts, the hand-crafted presentation of auxiliary knowledge is the key to the high-quality responses. The conventional prompt generation processes would be similar to solve the planning problems. In the POKE framework, we train a self-evolving planner to explicitly organize the knowledge of the LLMs. To apply POKE, the learned planner constructs a partial-order plan over the knowledge modules, the ordering constraints, and the causal links, after which an executor propagates the resolved intermediate plan and generates the prompt. We evaluate the POKE method on three atomic-probe subsets of TwoHopFact, MuSiQue, and 2WikiMultiHopQA. Experiments on the shared TwoHopFact subset prove that the POKE improves accuracy over the state-of-the-art methods. We also demonstrate that the POKE framework is capable to align with the physical laws and the structural constraints, by successfully fulfilling the physics-based simulation script generation tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, POP, Reasoning Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2d4LOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Partial Order Knowledge Engineering (POKE) is a planning-based framework for composing knowledge accessible to an LLM. A learned planner constructs knowledge modules, ordering constraints, and causal links, after which an executor resolves the modules and propagates intermediate outputs. The planner is trained through supervised fine-tuning, while the executor’s parameters remain unchanged.

The evaluation measures compositional accuracy on atomic-probe subsets of TwoHopFact, MuSiQue, and 2WikiMultiHopQA. Additional experiments examine shared subsets across executor scales, chain and join dependencies, gold-plan execution, execution ablations, and inference cost. An extension to physics-based simulation script generation incorporates object bindings, containment constraints, and physical or numerical parameters.

Critical strengths

Distinguishing successful atomic retrieval from successful composition provides a useful diagnostic perspective on the knowledge-organization gap. Comparisons within each executor’s knowledge-known subset help assess compositional performance conditional on observed atomic-probe success.

The shared-subset evaluation supports the practical value of explicit knowledge organization for smaller executors. In Table 2, POKE enables the smaller executor to outperform larger executors using conventional prompting on identical questions, making the cross-scale comparison more informative than comparisons across model-specific subsets.

The gold-plan POKE experiment helps identify plan construction as an important remaining bottleneck. The improvement reported in Table 7 suggests that the same executors can answer substantially more questions when supplied with the gold dependency structure, providing a useful diagnostic beyond aggregate accuracy.

Critical weaknesses

The reported improvements support structured decomposition and intermediate-result propagation, but the contribution specific to partial-order representation remains uncertain. Appendix G acknowledges that the interventions do not separately quantify ordering constraints and causal links. The available comparisons therefore may not distinguish the benefit of least-commitment planning from that of decomposition and explicit intermediate binding.

The relationship between the formal planning procedure and the evaluated implementation remains difficult to reconstruct. Section 3 describes open-condition refinement, threat resolution, verification, and backtracking, but their operational implementation within the learned planning system is not specified in comparable detail. In particular, the construction of candidate modules and symbolic conditions, together with the implementation of verification, remains unclear.

The supplied manuscript identifies the planner initialization and training approach, but does not provide enough concrete training information to assess reproducibility and generalization. Although the Reproducibility Statement says that LoRA configuration and implementation settings are reported, the supplied PDF does not specify the training corpus size, target-generation procedure, train–evaluation separation, or optimization hyperparameters. Consequently, the extent of task-specific supervision and its relationship to the reported generalization remains uncertain.

The knowledge-known subset protocol offers a valid conditional diagnostic, but some interpretations extend beyond what probe success establishes. Appendix A explicitly defines membership operationally rather than as evidence of permanent internal knowledge representation. Broader conclusions about knowledge absence and the principal benefits of model scaling therefore appear stronger than the conditional evaluation alone supports.

The simulation results support configuration consistency more directly than alignment with physical laws. Table 9 measures option selection, child-block consistency, and required numerical fields, rather than solver execution or the physical validity of resulting simulations. The abstract’s physical-law claim therefore appears broader than the demonstrated evidence.

Questions for authors

See Weaknesses Above.

Reviewer_PhaBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

POKE uses a supervised planner to construct knowledge modules and their dependencies, which a frozen LLM executor follows to answer multi-hop questions. The paper reports improvements on atomic-probe-filtered subsets of three QA datasets, evaluates chain and join structures, and extends the framework to simulation-script generation.

Critical strengths
  1. POKE consistently improves over the reported prompting baselines on three QA datasets and multiple executors.

  2. Useful diagnostics of planning and execution. The chain breakdown and gold-plan experiments go beyond aggregate accuracy. In Table 7, replacing learned plans with gold plans substantially improves performance, identifying plan construction as an important bottleneck and suggesting a concrete direction for improving the framework.

Critical weaknesses
  1. It is unclear how much partial-order planning adds over existing decomposition methods. A sequential planner trained on the same data would be a stronger baseline. The current ablations mainly test intermediate-answer quality, leaving the benefit of the partial-order structure unresolved.

  2. The training setup needs clarification. I could not tell how the planner’s training data were separated from evaluation. The reported results therefore leave cross-dataset generalization uncertain.

  3. The paper does not examine whether fine-tuning the planner on the target tasks hurts performance on broader tasks, leaving the generalization trade-off unclear.

Questions for authors
  1. How does POKE compare with a sequential planner using the same training data and a comparable inference budget?

  2. Have author evaluated a planner trained on one dataset on the others without further fine-tuning?

Reviewer_cqCGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

POKE represents a reasoning process as a partial-order plan containing knowledge modules, ordering constraints, and causal links:

Π=⟨A,O,L⟩.\Pi=\langle A,O,L\rangle.

The planner selects modules to support unsatisfied conditions, resolves threats to causal links, and constructs an executable plan. The executor follows a valid topological order and passes intermediate answers along the specified dependencies. The experiments use a planner initialized from Qwen3-1.7B and trained through LoRA supervised fine-tuning.

The evaluation compares POKE with Direct prompting, zero-shot Chain-of-Thought, and Self-Ask on subsets where each executor correctly answers all constituent atomic probes. Shared subsets support comparisons across model scales. POKE achieves approximately 98%–99% accuracy on the model-specific TwoHopFact subsets. Additional experiments cover MuSiQue chains and joins, 2WikiMultiHopQA, other executor families, and 899 physics-based simulation-script examples. The script metrics assess configuration choices, structural consistency, and numeric fields; they do not establish successful solver execution or physical simulation accuracy.

Critical strengths
  1. The diagnostic evaluation is well targeted.

    Model-specific subsets S11MS_{11}^{M} are used to compare inference methods within an executor, while intersections of these subsets support cross-scale comparisons on identical questions. This avoids directly comparing accuracy across different model-specific cohorts. Appendix A also acknowledges that atomic-probe success is an operational criterion rather than proof of permanent internal knowledge. The gold-plan analysis in Appendix D further helps identify planning as a source of remaining errors.

  2. The reported improvements extend across datasets and dependency structures.

    Tables 1–5 cover TwoHopFact, MuSiQue, 2WikiMultiHopQA, and additional executor families. For example, on the shared MuSiQue subset, POKE achieves 87.97% with the 1.7B executor, compared with 71.97% for Self-Ask. These results provide evidence that explicit decomposition and intermediate-result binding can improve compositional question answering beyond isolated examples.

Critical weaknesses
  1. The experiments do not isolate partial-order planning from standard decomposition and execution.

    The main TwoHopFact setting consists of two-step chains, which provide no independent branches whose relative ordering can remain unspecified. The unresolved-bridge and reverse-order controls in Appendix G deliberately violate execution prerequisites. They demonstrate the importance of correct intermediate binding, but do not show that retaining a partial order improves over a valid total-order plan. Although Table 3 includes join structures, there is no comparison using the same planning supervision, call budget, and executor prompts while removing the partial-order-specific mechanisms. The existing Self-Ask comparison is useful, but it does not eliminate the alternative explanation that a trained decomposition model and separate execution calls account for the gains. Established methods such as least-to-most prompting already solve subproblems sequentially using earlier answers. The present evidence therefore does not establish the distinct value of partial-order planning as the central methodological contribution.

  2. Missing planner-training and evaluation details limit the interpretation of the high accuracy.

    Section 4.1 specifies Qwen3-1.7B, LoRA supervised fine-tuning, and targets that exclude gold bridge entities and final answers. However, it does not adequately specify the number of training examples, the source of target plans, the training/validation/test split, or controls for overlap with evaluation questions and templates. The verification operation in Section 3.4 is also underspecified:

    Verify⁡(yi,Effi+).\operatorname{Verify}(y_i,\mathrm{Eff}_i^+).

    It is unclear what information this operation uses to check factual correctness and how failures trigger replanning. The supplied appendices do not substantiate all the implementation details claimed in the reproducibility statement. Furthermore, conditioning on successful atomic retrieval is appropriate for diagnosis but does not establish performance on the full question distribution or a general substitute for model scaling. These omissions leave the planner’s generalization and information access unclear; they do not establish that leakage occurred.

Questions for authors
  1. Clarification: What are the size, provenance, target-generation procedure, split, and deduplication rules for the planner’s supervised training data? Do training examples include evaluation questions, instances sharing the same templates, or gold decompositions? At inference, does the planner receive only the original question, or can it access dataset-provided atomic decompositions? These details directly affect the interpretation of generalization and the claim of organizing existing executor knowledge.

  2. Assessment-critical contribution validation: With planning supervision, call counts, and the executor held fixed, does POKE outperform a valid linear decomposition or topological-execution baseline, particularly on join structures? Which evaluated cases actually require threat resolution, backtracking, or maintaining unordered actions, rather than predicting and executing a dependency DAG once? This comparison would clarify the contribution of the partial-order machinery.

  3. Clarification and assessment-critical validation: What are the concrete rules and information-access permissions of the verification operation, and how frequently does it trigger replanning? Are results available on the complete, unfiltered evaluation distributions? The existing conditional results remain informative, but these answers determine how broadly the conclusions can be interpreted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.