acceptodds
Under review as a conference paper at ICLR 2027

Toward Effective and Reliable LLM Agents via Dynamic Ontology

Abstract

Large language model (LLM) agents rely heavily on knowledge encoded in model parameters or presented as unstructured context. In domain-specific tasks, this leaves important semantic connections implicit. This often results in incomplete evidence use and brittle multi-step decisions. Ontologies offer a way to externalize domain concepts and relations as machine-interpretable structures, but constructing task-usable ontologies traditionally requires substantial effort from domain experts and is difficult to scale. Automatic construction is also challenging: an ontology that appears semantically plausible may not contain the relational structures needed for actual decision making. We present OaK, an ontology-as-a-kernel framework that dynamically constructs and refines task-oriented ontologies for LLM agents. Given task requirements and training data, OaK constructs an ontology and its knowledge graph, generates task-adaptation functions for graph reasoning, and uses judge feedback to iteratively refine both. By making relevant concepts and relations explicit, the ontology grounds knowledge retrieval and multi-step decision making. We evaluate OaK on TravelPlanner, CRMArenaPro, and ToolQA. Results show that OaK improves standard LLM agents, strengthens evidence grounding, and boosts the reliability of multi-step reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent Reliability, Ontology, Knowledge Representation and Reasoning, Neuro-Symbolic AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_if6VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes OaK, a dynamic ontology-as-a-kernel framework for improving the effectiveness and reliability of LLM agents. OaK automatically constructs task-specific ontology schemas, instantiates knowledge graphs, and generates executable reasoning functions from generic operators. It further employs judge-driven iterative refinement to improve the schema and functions based on task feedback. Experiments on TravelPlanner, CRMArenaPro, and ToolQA demonstrate consistent improvements over existing agent baselines across two LLM backbones, with ablation studies supporting the effectiveness of function composition and iterative refinement.

Critical strengths
  1. OaK combines ontology schemas, knowledge graphs, and task-specific reasoning functions into a unified framework. This design provides a structured interface for agent execution and reduces reliance on free-form LLM reasoning in knowledge-intensive tasks.

  2. OaK achieves the best aggregate performance across three benchmarks and two LLM backbones, demonstrating effectiveness across different task settings. The ablation studies further support the contributions of function composition and iterative refinement to overall performance.

Critical weaknesses
  1. Although OaK claims to improve both effectiveness and reliability, the experiments primarily measure task success and constraint satisfaction. The paper lacks direct evaluations of evidence grounding, execution violations, and failure traceability. Without such metrics and comparisons with relevant methods (e.g., AgentSpec), the claimed reliability improvements remain insufficiently supported.

  2. While the ablations demonstrate the benefits of function composition and iterative refinement, they do not isolate the contribution of the ontology schema itself. It remains unclear whether the improvements genuinely arise from ontology-based reasoning and formal verification, or simply from task-specific executable functions.

Although my confidence score is 2, the concerns raised above are based on concrete issues in the manuscript. I will carefully consider the other reviewers' assessments and am open to raising my score if the authors satisfactorily address these concerns during the rebuttal.

Questions for authors
  1. Section 4.2 specifies a maximum of five refinement iterations, but Figure 3 reports seven rounds. Could the authors clarify this discrepancy?

  2. Does the cost analysis in Figure 4 account for the full ontology construction and iterative refinement overhead, and how does the total cost compare with the baselines?

  3. Why does the evaluation include only five ToolQA subsets rather than the full benchmark, and how were these subsets selected?

5 comments
AuthorsOpenReview ↗

Response to Weakness 1

We are grateful for your positive evaluation of OaK's unified design and benchmark performance. Your detailed suggestions have motivated additional experiments and improvements to our manuscript. We respond to each point below.

For W1: Expanded reliability evaluation and AgentSpec comparisons.

To provide a more comprehensive evaluation of reliability, we conducted additional experiments addressing evidence grounding, execution violations, and failure traceability, and included AgentSpec as an additional baseline.

1. Evaluation of reliability

(1) Evidence Grounding. To evaluate evidence grounding, we conducted an additional fact-checking experiment using the MiniCheck[1] method and its proposed model(Bespoke-MiniCheck-7B) to examine whether facts in the constructed knowledge graphs are supported by their original source materials. Specifically, we randomly sampled 1000 attribute and relation facts from the knowledge graphs constructed for TravelPlanner and used the corresponding original records as evidence. MiniCheck classified 967 facts as supported, yielding a source support rate of 96.7%. This result demonstrates that the constructed knowledge graphs are largely grounded in the original source materials, providing quantitative evidence for the factual grounding of the knowledge used in downstream agent reasoning.

(2) Execution Violations. To examine execution violations, we audited agent-generated actions in retained ToolQA trajectories for 917 tasks shared by OaK, ReAct, and AgentSpec. Automated scripts paired actions with their recorded outcomes and identified invalid execution attempts using existing operation definitions, parameter domains, and database schemas. We identified 8, 168 and 34 invalid attempts for OaK, ReAct, and AgentSpec, respectively. OaK had the fewest confirmed invalid execution attempts among the three methods. These observations are consistent with OaK’s design of constraining the action space through semantic methods and enforcing explicit execution boundaries. The counts are descriptive under each method’s original interface and run configuration.

(3) Failure Traceability. We provided the agents with explicit records of the error locations and causes identified from their first failed attempts. This allows us to evaluate whether execution failures can be clearly traced and diagnosed, providing the agents with actionable information about the sources of failure. We evaluate OaK, ReAct, and AgentSpec on the same set of failed tasks, obtaining second-attempt recovery rates of 25.32%, 13.92%, and 15.82%, respectively. OaK achieves the highest recovery rate, suggesting that its ontology kernel supports error localization and diagnosis through explicit function calls, arguments, and intermediate results. We have added these experiments and the corresponding analyses to Section 4.7 of the revised manuscript.

2. Comparison with AgentSpec.

Following your suggestion, we included AgentSpec as an additional baseline and evaluated it on all three benchmarks. With DeepSeek-v4-flash, OaK achieves a 55.90% final pass rate on TravelPlanner, compared with 28.80% for AgentSpec. On CRMArenaPro, OaK obtains B2B and B2C average scores of 78.38% and 75.20%, respectively, outperforming AgentSpec (53.67% and 54.81%). On ToolQA, OaK achieves a weighted-average accuracy of 64.58%, compared with 53.29% for AgentSpec. GPT-4o-mini shows the same overall trend. The complete results, including individual task categories and both backbone configurations, are reported in Tables 1–3 of the revised manuscript.

[1] MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (Tang et al., EMNLP 2024)

AuthorsOpenReview ↗

Response to Weakness 2

For W2: Contributions of ontology schemas and formal verification.

We agree that examining only function composition and iterative refinement is insufficient to fully distinguish the respective contributions of the ontology structure and formal verification. To address this concern, we extended our ablation study by introducing two additional variants to separately evaluate the contributions of explicit schema construction and HermiT-based formal verification.

First, Schema Ablation (w/o Schema) removes the explicit schema construction process. Instead, the LLM directly extracts entities and relations from the raw data to construct the knowledge graph, while incrementally inducing entity types during the extraction process.

Second, HermiT Ablation (w/o HermiT) retains the schema construction process but skips OWL translation and HermiT-based checks for logical consistency and class satisfiability.

The results show that removing either component leads to substantial performance degradation across all three benchmarks. Specifically, on TravelPlanner, the full OaK achieves a Final Pass Rate of 55.90%, whereas the scores of w/o Schema and w/o HermiT drop to 29.00% and 23.50%, respectively. On CRMArenaPro, the overall average score across B2B and B2C is 76.79% for the full OaK, compared with 47.42% and 38.47% for the two ablation variants, respectively. On ToolQA, the full OaK achieves a weighted-average accuracy of 64.58%, while w/o Schema and w/o HermiT achieve only 35.17% and 33.66%, respectively.

Both additional variants have been evaluated on TravelPlanner, CRMArenaPro, and ToolQA, and the corresponding results have been incorporated into Figure 2 in Section 4.4 of the revised manuscript. These results demonstrate that both explicit ontology schema construction and HermiT-based formal verification make substantial contributions to downstream task performance.

AuthorsOpenReview ↗

Response to Question 1, Question 2 and Question 3

For Q1: Refinement budget selection and iteration analysis.

We apologize for not clearly explaining how the iteration budget was chosen. Figure 3 reports refinement performance over seven rounds. Since the results show limited improvement beyond five rounds, we selected five iterations as the budget for the main experiments, as stated in Section 4.2. Figure 3 therefore provides the empirical basis for this choice. We have clarified this relationship in Section 4.5.

For Q2: Cost accounting and baseline comparisons.

We apologize for not stating the accounting scope clearly. Figure 4 includes all steps of OaK, covering initial construction, five rounds of iterative refinement, and subsequent inference. Runtime and input/output tokens are accumulated across all stages and divided by the number of evaluated examples. The same accounting principle applies to the baselines.

Although OaK does not minimize every resource metric, particularly output tokens due to graph construction, it uses fewer input tokens than ReAct and MemP, with runtime comparable to AFlow and lower than ReCode. OaK achieves the highest final pass rate, offering a favorable balance between task performance and resource consumption. We have clarified the scope and calculation of the reported costs in Section 4.6 of the revised manuscript.

For Q3: ToolQA subset selection.

We apologize for not clearly explaining our ToolQA subset selection. We evaluated Flight, Coffee, Airbnb, DBLP, and Yelp, with the remaining three subsets excluded for the reasons below.

GSM8K provides no external knowledge corpus independent of its questions. It therefore does not directly assess our complete pipeline of constructing an ontology and knowledge graph from corpus and using them for task reasoning.

For Agenda and SciREX, we identified data quality issues during our experimental work that could compromise the validity of accuracy measurements, and therefore excluded them from the reported evaluation.

For Agenda, some reference answers contain errors. For example, in hard-agenda-0042, the source record (agenda-8499) describes an existing jazz and wine event involving Faith from 8:00 PM to 10:00 PM. The question asks when a meeting with Faith can be scheduled between 9:00 AM and 6:00 PM that day. Since the existing evening event falls entirely outside the requested meeting window, the full window from 9:00 AM to 6:00 PM remains available. However, the reference answer gives “9:00 AM to 8:00 PM, 10:00 PM to 6:00 PM.” The first interval exceeds the requested window, while the second ends before it starts. Such errors can cause correct answers to be marked incorrect.

For SciREX, the official documentation (https://github.com/allenai/SciREX#scirex--a-challenge-dataset-for-document-level-information-extraction) acknowledges that approximately half of the original annotated relations contain at least one entity absent from the document body because the corresponding tables were excluded. This can leave annotations without sufficient supporting evidence in the supplied text, confounding the evaluation of system reasoning with evidence incompleteness.

We sincerely appreciate your time and thoughtful consideration of our responses. We hope our clarifications have addressed your concerns, and would be grateful for any further questions or suggestions you may have.

Reviewer_nk7HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The performance of LLM agents can decrease under domain-specific tasks, where important semantic connections are implicit. This paper proposes OaK, an ontology-as-a-kernel framework that dynamically constructs and refines task-oriented ontologies for LLM agents. Experiments on deepseek-v4-flash and GPT-4o-mini show that OaK can improve the overall performance on three benchmarks: TravelPlanner, CRMArenaPro, and ToolQA compared with baselines.

Critical strengths

The method itself is easy-to-follow and mostly effective. As stated by the results section, the proposed method shows improvements compared with baseline methods on three tasks.

Critical weaknesses
  1. The experiments can not fully evaluate the method itself. The authors use DeepSeek-v4-flash and GPT-4o-mini as the base model, where GPT-4o-mini is outdated and is not designed for the agentic work. And the OpenAI is going to deprecate the GPT-4o-mini in three months, the reproducibility of this paper may be influenced. In addition, the ablation study is not sufficient. The effectiveness of OWL and HermiT are not evaluated. The claimed "RELIABLE" in the title is also not supported.

In terms of the results, MemP in Table 1 seems to show comparable performance when using more recent model DeepSeek-v4-flash. Additional recent models are recommended.

  1. The Schema and KG are fixed to a specific tasks and can not be transferred between tasks. This limit the scope of the proposed method. And the correctness of the KG can not be verified. How it can influence the overall agentic workflow is unclear.

  2. The proposed method relies on annotated data and official evaluation scores, and is difficult to transfer between tasks. This makes the proposed method very limited in real-world applications as an agentic workflow.

  3. The basic idea that using excution feedback to improve the system is widely explored in prompt engineering research. Although this paper focues on the agentic workflow, related literatures still should be discussed (e.g., TextGrad, ProTeGi, etc.).

Questions for authors
  1. If a much more powerful model with popular harness is used, such as opus-5.5+claude code, will this method still apply?
  2. Is this method stable under one fixed seed? Repeating results should be reported.
6 comments
AuthorsOpenReview ↗

Response to Weakness 1

Thank you for recognizing the clarity and effectiveness of our method across the three evaluated tasks. Your insightful comments have helped us further strengthen our work. We address each concern below.

For W1: Expanded evaluation of applicability, effectiveness, and reliability.

1. Regarding the choice of base models and experimental reproducibility.

We acknowledge that base model choice can affect agent performance and model updates may complicate reproducibility. To assess OaK's adaptability across models and agent execution environments, we ran additional experiments using the Claude Code harness with Claude Sonnet 5.5 on TravelPlanner, ToolQA, and CRMArenaPro. The results show a 62.90% final pass rate on TravelPlanner, 72.17% weighted-average accuracy on ToolQA, and average scores of 69.65% and 72.42% on the B2B and B2C settings of CRMArenaPro, respectively. These results further demonstrate that OaK can operate effectively within a different agent execution framework. The additional results are provided in Appendix E of the revised manuscript.

2. Regarding the sufficiency of ablation studies and the effectiveness of OWL/HermiT.

Following your suggestion, we added two ablation variants: w/o Schema and w/o HermiT. The w/o HermiT variant removes both the OWL translation and subsequent HermiT-based checks, allowing us to evaluate the contribution of formal schema verification to downstream task performance. The w/o Schema variant examines the impact of removing explicit schema construction and its associated semantic constraints on agent decision-making. The results are incorporated into Figure 2 in Section 4.4 of the revised manuscript. The performance degradation of both variants across all three benchmarks demonstrates that explicit schema construction and formal verification are essential components of the OaK framework.

3. Regarding reliability.

We would first like to clarify that the reliability emphasized in this work primarily concerns whether an LLM agent can consistently make decisions that satisfy task requirements, are evidence-supported, and remain traceable throughout multi-step task execution, rather than encompassing all dimensions of agent safety and trustworthiness. Within this scope, our original submission had already partially evaluated reliability. On TravelPlanner, we assessed outcome-level reliability through individual constraint satisfaction (CS Micro and HC Micro) and plan-level satisfaction of all constraints within each category (CS Macro and HC Macro). The latter reflect multi-constraint coordination and overall decision consistency during multi-step planning.

Nevertheless, we acknowledge that task performance and constraint satisfaction alone do not fully capture execution-process reliability. We therefore conducted additional experiments from two complementary perspectives:

(1) Execution Validity. We audited the execution trajectories of OaK, ReAct, and AgentSpec on a shared set of 917 ToolQA tasks. Automated scripts paired agent actions with their recorded outcomes and identified execution errors based on existing operation definitions, parameter domains, and database schemas. The results show that OaK, ReAct, and AgentSpec produced 8, 168, and 34 invalid execution attempts, respectively. This indicates that OaK exhibited fewer invalid execution attempts under the same task set and each method's original execution interface, consistent with its design of constraining the agent's action space through ontology semantics and typed reasoning functions.

(2) Failure Traceability. We provided the agents with explicit records of the error locations and causes identified from their first failed attempts. This allows us to evaluate whether execution failures can be clearly traced and diagnosed, providing the agents with actionable information about the sources of failure. On the same set of failed tasks, OaK, ReAct, and AgentSpec achieved second-attempt recovery rates of 25.32%, 13.92%, and 15.82%, respectively. OaK achieved the highest recovery rate, suggesting that explicit records of function calls, arguments, and intermediate execution results provide more useful information for error localization and diagnosis.

4. Regarding the performance comparison with MemP.

We acknowledge that MemP demonstrates competitive performance on certain fine-grained metrics. However, OaK outperforms MemP on the primary aggregate metrics across all three benchmarks and both backbones. Detailed comparisons are provided in Tables 1–3.

These results suggest that OaK demonstrates more consistent advantages in overall task completion, multi-constraint coordination, and performance across diverse task settings. This further highlights the value of organizing agent decision-making through the coordinated use of task-adaptive ontologies, knowledge graphs, and executable reasoning functions.

AuthorsOpenReview ↗

Response to Weakness 2, Weakness 3 and Weakness 4

For W2: Adaptive schema and KG construction, factual grounding and downstream impact.

1. Clarification on Schema and Knowledge Graph Construction and Utilization.

First, OaK does not rely on manually predefined and fixed schemas. Instead, it automatically identifies the entities, attributes, relations, and constraints required by a task based on its description and training examples, and refines the schema through iterative feedback. Therefore, for different tasks, OaK can automatically construct corresponding task-specific ontologies according to their respective requirements, rather than relying on a predefined universal ontology.

Second, during inference, OaK dynamically instantiates a Knowledge Graph using the Schema and the reference corpus associated with each query. Therefore, the KG is not constructed in advance and repeatedly reused across all queries, but is dynamically instantiated based on the data provided for each specific query.

Thus, the task-specific nature of OaK does not limit its applicability to a single task. On the contrary, its core design goal is to automatically construct corresponding ontology kernels according to the requirements of different tasks and use these kernels to handle unseen queries within each task. This task-adaptive construction mechanism enables OaK to accommodate differences in knowledge structures and decision-making requirements across tasks.

2. Correctness of KG.

To address the concerns regarding the factual correctness of the constructed KG and its impact on the downstream agentic workflow, we conducted two additional experiments.

First, to evaluate the factual correctness of the constructed Knowledge Graphs, we used the MiniCheck[1] method and its proposed model(Bespoke-MiniCheck-7B) to perform an additional fact-checking evaluation. Specifically, we randomly sampled 1000 entity attribute and relation facts from the Knowledge Graphs constructed for TravelPlanner tasks, using the corresponding original records as evidence to determine whether these facts were supported by the source materials. The results show that 967 out of 1000 facts were supported, yielding a fact support rate of 96.7%. This provides direct empirical evidence of the factual consistency of the constructed Knowledge Graphs.

Second, to further investigate the impact of KG correctness on downstream agent decision-making, we randomly corrupted about 10% of the facts in the Knowledge Graphs and reevaluated the agent's performance on TravelPlanner. The results show that the final pass rate dropped from 55.9% to 24.2%, indicating that the quality of the Knowledge Graph has a substantial impact on downstream reasoning and decision-making performance.

Together, these additional experiments demonstrate that the Knowledge Graph serves as an essential factual foundation for the agent's reasoning and decision-making process.

[1] MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (Tang et al., EMNLP 2024)

For W3: Task generality, data requirements, and feedback flexibility.

First, OaK is a general framework designed to address different types of tasks by automatically constructing corresponding ontology kernels according to task-specific requirements, rather than being tailored to a particular task. We evaluate OaK on three benchmarks, TravelPlanner, CRMArenaPro, and ToolQA, which involve different task objectives and reasoning requirements, to demonstrate its adaptability across diverse tasks.

Second, regarding the use of official evaluation scores, different tasks naturally have different success criteria and evaluation metrics because they address different problems. To ensure that the ontology refinement process aligns with the actual objectives of each task, we adopt the official evaluation protocols provided by the respective benchmarks as feedback signals, rather than imposing a manually designed, unified evaluation standard. Therefore, OaK is not tied to any particular evaluation metric, and its evaluation mechanism can be adapted to the requirements of different tasks. Furthermore, ontology construction in OaK primarily relies on task descriptions and training examples, without requiring manually annotated ontology structures. In real-world scenarios where official evaluators are unavailable, alternative feedback signals, such as LLM-based evaluation, rule-based checking, or human feedback, can also be employed.

For W4: Connections to TextGrad and ProTeGi.

We agree that the idea of iteratively improving systems through feedback has been extensively explored in prior studies, including ProTeGi and TextGrad. These studies provide important references for feedback-driven system optimization. Following your suggestion, we have added a discussion of TextGrad and ProTeGi to the Related Work section of the revised manuscript and further clarified their connections to our work.

AuthorsOpenReview ↗

Response to Question 1 and Question 2

For Q1: Applicability with Claude Sonnet 5.5 and Claude Code.

We carefully considered OaK’s applicability with stronger models and popular harnesses. To balance experimental cost with evaluation needs, we conducted additional experiments using Claude Sonnet 5.5 + Claude Code. With OaK integrated, TravelPlanner achieved a Final score of 62.90%, ToolQA achieved an overall accuracy of 72.17%, and CRMArenaPro achieved Avg scores of 69.65% and 72.42% for B2B and B2C, respectively. These results support OaK’s applicability with this model and a popular harness. The complete results are provided in Appendix E, covering all evaluation metrics across the three datasets and detailed breakdowns by subset and task category.

For Q2: Performance stability across random seeds.

We would like to clarify that the original manuscript already reported multi-seed experimental results on TravelPlanner in Appendix C. To further evaluate the stability of OaK across different task settings, we additionally conducted repeated experiments on ToolQA and CRMArenaPro in the revised manuscript. Specifically, we performed three independent runs using different random seeds and reported the mean and standard deviation of the results.

The results show that OaK achieves a Final Pass Rate of 55.90% ± 2.13% on TravelPlanner, a weighted-average accuracy of 64.01% ± 1.09% on ToolQA, and B2B and B2C average scores of 78.96% ± 0.94% and 74.57% ± 0.74%, respectively, on CRMArenaPro. The relatively small standard deviations across all three benchmarks indicate that OaK exhibits good performance stability across different random seeds. These additional results are included in Appendix C (Tables 4–6) of the revised manuscript.

We hope our responses have adequately addressed your concerns. We greatly appreciate your patience and time in reviewing our work, and would be grateful for any further questions or suggestions.

Reviewer_oMZoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OaK, an ontology-as-a-kernel framework for improving the effectiveness and reliability of LLM agents. OaK automatically constructs a task-oriented schema, instantiates a corresponding knowledge graph from task data, and composes generic graph operators into executable domain-specific reasoning functions. The framework further employs a judge-driven refinement loop to identify and repair deficiencies in both the schema and the reasoning functions based on task evaluation scores and execution trajectories. At inference time, the resulting kernel is frozen and used by a ReAct agent to solve unseen queries. The authors evaluate OaK on multiple benchmarks and the result demonstrate consistent improvements over several agent baselines. The ablation results also suggest that function composition and iterative refinement contribute substantially to the observed gains.

Critical strengths
  • This paper focuses on an imporant problem of making domain-specific reasoning more structured and controllable rather than relying entirely on LLM reasoning.
  • The combination of a task-oriented schema, knowledge graph, and executable reasoning functions provides a coherent interface between domain knowledge and LLM execution, which is intuitive and useful.
  • OaK achieves the best performance across various benchmarks, suggesting the effectiveness of this framework.
Critical weaknesses
  • The academic novelty is limited. The overall framework combines several existing techniques including schema generation, knowledge graph construction, executable tool composition, and judge-driven refinement.
  • The current ablations mainly examine function composition, the function module, and iterative refinement. They do not adequately isolate the contribution of the ontology itself.
  • The framework relies on LLM-based entity and relation extraction. However, the paper does not report exact extraction accuracy or systematically analyze how graph construction errors affect downstream reasoning.
Questions for authors

Please address the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Weakness 1, Weakness 2 and Weakness 3

We appreciate your positive assessment of OaK's framework design and experimental performance. Your constructive feedback has been valuable in improving our manuscript. Below, we provide point-by-point responses to all comments.

For W1: Framework novelty.

We acknowledge that the individual techniques employed in our work, including Schema Generation, Knowledge Graph Construction, Function Composition, and Judge-driven Refinement, have been explored in prior studies. The novelty of our work does not lie in proposing these techniques independently, but in integrating them into a dynamic ontology framework (Ontology-as-a-Kernel, OaK) specifically designed for LLM agent decision-making. Specifically, our contributions lie in three aspects:

  1. Task-adaptive ontology construction. OaK automatically constructs task-specific ontologies based on task descriptions and training examples to meet downstream decision-making requirements, rather than relying on predefined static domain ontologies.

  2. Ontology-driven executable decision-making. This is the central contribution of our work. We transform ontologies from passive knowledge representation tools into semantic kernels for LLM agent decision-making. By jointly integrating Schemas, Knowledge Graphs, and Typed Reasoning Functions, OaK connects domain knowledge structures with concrete decision-making processes. The Schema constrains the semantic space available to the agent, while task-adapted reasoning functions transform complex multi-step reasoning into executable and reusable operations. In this way, the ontology not only provides knowledge but also actively organizes and constrains the agent's decision-making process.

  3. Joint optimization of semantic structures and execution logic. OaK leverages downstream task evaluation results and execution trajectories to jointly diagnose and repair missing knowledge at the Schema level and deficiencies in reasoning logic at the Function level, ensuring consistency between knowledge representation and actual decision-making processes.

Therefore, the central contribution of this work is to establish a task-adaptive, executable, and iteratively refined ontology-based decision-making framework, rather than simply combining existing technical components.

For W2: Schema ablation and ontology contribution.

To further assess the ontology’s contribution, we added a w/o Schema ablation that removes explicit schema construction while retaining the knowledge graph and function module. This variant extracts nodes and relations directly from data. To keep type-dependent functions operational, the LLM incrementally induces node and relation types during extraction, reusing extracted types and introducing new ones only when necessary. This setting examines the role of explicit schema construction. The variant achieves 29.00% Final on TravelPlanner, 49.83%/45.00% Avg. on CRMArenaPro B2B/B2C, and 35.17% Avg. on ToolQA. The performance degradation demonstrates the importance of explicit schema construction. Complete results have been added to Section 4.4 and Figure 2 of the revised manuscript.

For W3: KG factual grounding and downstream impact.

To further evaluate knowledge graph quality, we used the MiniCheck method [1] with the Bespoke-MiniCheck-7B model to assess whether facts in the constructed graphs are supported by the original source materials.

Specifically, we randomly sampled 1000 attribute and relation facts from the knowledge graphs constructed for TravelPlanner tasks. Using the corresponding original records as evidence, MiniCheck classified 967 facts as supported, yielding a source support rate of 96.7%. This provides direct evidence of the factual consistency of the intermediate knowledge graphs.

Additionally, to assess the impact of knowledge graph correctness on downstream reasoning, we randomly corrupted approximately 10% of graph facts. The final pass rate on TravelPlanner dropped from 55.9% to 24.2%, indicating that knowledge graph quality has a substantial impact on downstream reasoning performance.

These are our responses to your comments. Thank you very much for your time and careful consideration. Should any questions remain, we sincerely welcome further feedback and would be happy to provide additional clarification.

[1] MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (Tang et al., EMNLP 2024)

Thank you for the response

Thank you for your acknowledgment. Should you have any further questions or concerns, we would be happy to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.