- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Partial Order Knowledge Engineering (POKE) is a planning-based framework for composing knowledge accessible to an LLM. A learned planner constructs knowledge modules, ordering constraints, and causal links, after which an executor resolves the modules and propagates intermediate outputs. The planner is trained through supervised fine-tuning, while the executor’s parameters remain unchanged.
The evaluation measures compositional accuracy on atomic-probe subsets of TwoHopFact, MuSiQue, and 2WikiMultiHopQA. Additional experiments examine shared subsets across executor scales, chain and join dependencies, gold-plan execution, execution ablations, and inference cost. An extension to physics-based simulation script generation incorporates object bindings, containment constraints, and physical or numerical parameters.
Critical strengths
Distinguishing successful atomic retrieval from successful composition provides a useful diagnostic perspective on the knowledge-organization gap. Comparisons within each executor’s knowledge-known subset help assess compositional performance conditional on observed atomic-probe success.
The shared-subset evaluation supports the practical value of explicit knowledge organization for smaller executors. In Table 2, POKE enables the smaller executor to outperform larger executors using conventional prompting on identical questions, making the cross-scale comparison more informative than comparisons across model-specific subsets.
The gold-plan POKE experiment helps identify plan construction as an important remaining bottleneck. The improvement reported in Table 7 suggests that the same executors can answer substantially more questions when supplied with the gold dependency structure, providing a useful diagnostic beyond aggregate accuracy.
Critical weaknesses
The reported improvements support structured decomposition and intermediate-result propagation, but the contribution specific to partial-order representation remains uncertain. Appendix G acknowledges that the interventions do not separately quantify ordering constraints and causal links. The available comparisons therefore may not distinguish the benefit of least-commitment planning from that of decomposition and explicit intermediate binding.
The relationship between the formal planning procedure and the evaluated implementation remains difficult to reconstruct. Section 3 describes open-condition refinement, threat resolution, verification, and backtracking, but their operational implementation within the learned planning system is not specified in comparable detail. In particular, the construction of candidate modules and symbolic conditions, together with the implementation of verification, remains unclear.
The supplied manuscript identifies the planner initialization and training approach, but does not provide enough concrete training information to assess reproducibility and generalization. Although the Reproducibility Statement says that LoRA configuration and implementation settings are reported, the supplied PDF does not specify the training corpus size, target-generation procedure, train–evaluation separation, or optimization hyperparameters. Consequently, the extent of task-specific supervision and its relationship to the reported generalization remains uncertain.
The knowledge-known subset protocol offers a valid conditional diagnostic, but some interpretations extend beyond what probe success establishes. Appendix A explicitly defines membership operationally rather than as evidence of permanent internal knowledge representation. Broader conclusions about knowledge absence and the principal benefits of model scaling therefore appear stronger than the conditional evaluation alone supports.
The simulation results support configuration consistency more directly than alignment with physical laws. Table 9 measures option selection, child-block consistency, and required numerical fields, rather than solver execution or the physical validity of resulting simulations. The abstract’s physical-law claim therefore appears broader than the demonstrated evidence.
Questions for authors
See Weaknesses Above.