- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes OaK, a dynamic ontology-as-a-kernel framework for improving the effectiveness and reliability of LLM agents. OaK automatically constructs task-specific ontology schemas, instantiates knowledge graphs, and generates executable reasoning functions from generic operators. It further employs judge-driven iterative refinement to improve the schema and functions based on task feedback. Experiments on TravelPlanner, CRMArenaPro, and ToolQA demonstrate consistent improvements over existing agent baselines across two LLM backbones, with ablation studies supporting the effectiveness of function composition and iterative refinement.
Critical strengths
-
OaK combines ontology schemas, knowledge graphs, and task-specific reasoning functions into a unified framework. This design provides a structured interface for agent execution and reduces reliance on free-form LLM reasoning in knowledge-intensive tasks.
-
OaK achieves the best aggregate performance across three benchmarks and two LLM backbones, demonstrating effectiveness across different task settings. The ablation studies further support the contributions of function composition and iterative refinement to overall performance.
Critical weaknesses
-
Although OaK claims to improve both effectiveness and reliability, the experiments primarily measure task success and constraint satisfaction. The paper lacks direct evaluations of evidence grounding, execution violations, and failure traceability. Without such metrics and comparisons with relevant methods (e.g., AgentSpec), the claimed reliability improvements remain insufficiently supported.
-
While the ablations demonstrate the benefits of function composition and iterative refinement, they do not isolate the contribution of the ontology schema itself. It remains unclear whether the improvements genuinely arise from ontology-based reasoning and formal verification, or simply from task-specific executable functions.
Although my confidence score is 2, the concerns raised above are based on concrete issues in the manuscript. I will carefully consider the other reviewers' assessments and am open to raising my score if the authors satisfactorily address these concerns during the rebuttal.
Questions for authors
-
Section 4.2 specifies a maximum of five refinement iterations, but Figure 3 reports seven rounds. Could the authors clarify this discrepancy?
-
Does the cost analysis in Figure 4 account for the full ontology construction and iterative refinement overhead, and how does the total cost compare with the baselines?
-
Why does the evaluation include only five ToolQA subsets rather than the full benchmark, and how were these subsets selected?
5 comments
Response to Weakness 2
For W2: Contributions of ontology schemas and formal verification.
We agree that examining only function composition and iterative refinement is insufficient to fully distinguish the respective contributions of the ontology structure and formal verification. To address this concern, we extended our ablation study by introducing two additional variants to separately evaluate the contributions of explicit schema construction and HermiT-based formal verification.
First, Schema Ablation (w/o Schema) removes the explicit schema construction process. Instead, the LLM directly extracts entities and relations from the raw data to construct the knowledge graph, while incrementally inducing entity types during the extraction process.
Second, HermiT Ablation (w/o HermiT) retains the schema construction process but skips OWL translation and HermiT-based checks for logical consistency and class satisfiability.
The results show that removing either component leads to substantial performance degradation across all three benchmarks. Specifically, on TravelPlanner, the full OaK achieves a Final Pass Rate of 55.90%, whereas the scores of w/o Schema and w/o HermiT drop to 29.00% and 23.50%, respectively. On CRMArenaPro, the overall average score across B2B and B2C is 76.79% for the full OaK, compared with 47.42% and 38.47% for the two ablation variants, respectively. On ToolQA, the full OaK achieves a weighted-average accuracy of 64.58%, while w/o Schema and w/o HermiT achieve only 35.17% and 33.66%, respectively.
Both additional variants have been evaluated on TravelPlanner, CRMArenaPro, and ToolQA, and the corresponding results have been incorporated into Figure 2 in Section 4.4 of the revised manuscript. These results demonstrate that both explicit ontology schema construction and HermiT-based formal verification make substantial contributions to downstream task performance.
Response to Question 1, Question 2 and Question 3
For Q1: Refinement budget selection and iteration analysis.
We apologize for not clearly explaining how the iteration budget was chosen. Figure 3 reports refinement performance over seven rounds. Since the results show limited improvement beyond five rounds, we selected five iterations as the budget for the main experiments, as stated in Section 4.2. Figure 3 therefore provides the empirical basis for this choice. We have clarified this relationship in Section 4.5.
For Q2: Cost accounting and baseline comparisons.
We apologize for not stating the accounting scope clearly. Figure 4 includes all steps of OaK, covering initial construction, five rounds of iterative refinement, and subsequent inference. Runtime and input/output tokens are accumulated across all stages and divided by the number of evaluated examples. The same accounting principle applies to the baselines.
Although OaK does not minimize every resource metric, particularly output tokens due to graph construction, it uses fewer input tokens than ReAct and MemP, with runtime comparable to AFlow and lower than ReCode. OaK achieves the highest final pass rate, offering a favorable balance between task performance and resource consumption. We have clarified the scope and calculation of the reported costs in Section 4.6 of the revised manuscript.
For Q3: ToolQA subset selection.
We apologize for not clearly explaining our ToolQA subset selection. We evaluated Flight, Coffee, Airbnb, DBLP, and Yelp, with the remaining three subsets excluded for the reasons below.
GSM8K provides no external knowledge corpus independent of its questions. It therefore does not directly assess our complete pipeline of constructing an ontology and knowledge graph from corpus and using them for task reasoning.
For Agenda and SciREX, we identified data quality issues during our experimental work that could compromise the validity of accuracy measurements, and therefore excluded them from the reported evaluation.
For Agenda, some reference answers contain errors. For example, in hard-agenda-0042, the source record (agenda-8499) describes an existing jazz and wine event involving Faith from 8:00 PM to 10:00 PM. The question asks when a meeting with Faith can be scheduled between 9:00 AM and 6:00 PM that day. Since the existing evening event falls entirely outside the requested meeting window, the full window from 9:00 AM to 6:00 PM remains available. However, the reference answer gives “9:00 AM to 8:00 PM, 10:00 PM to 6:00 PM.” The first interval exceeds the requested window, while the second ends before it starts. Such errors can cause correct answers to be marked incorrect.
For SciREX, the official documentation (https://github.com/allenai/SciREX#scirex--a-challenge-dataset-for-document-level-information-extraction) acknowledges that approximately half of the original annotated relations contain at least one entity absent from the document body because the corresponding tables were excluded. This can leave annotations without sufficient supporting evidence in the supplied text, confounding the evaluation of system reasoning with evidence incompleteness.
We sincerely appreciate your time and thoughtful consideration of our responses. We hope our clarifications have addressed your concerns, and would be grateful for any further questions or suggestions you may have.
Response to Weakness 1
We are grateful for your positive evaluation of OaK's unified design and benchmark performance. Your detailed suggestions have motivated additional experiments and improvements to our manuscript. We respond to each point below.
For W1: Expanded reliability evaluation and AgentSpec comparisons.
To provide a more comprehensive evaluation of reliability, we conducted additional experiments addressing evidence grounding, execution violations, and failure traceability, and included AgentSpec as an additional baseline.
1. Evaluation of reliability
(1) Evidence Grounding. To evaluate evidence grounding, we conducted an additional fact-checking experiment using the MiniCheck[1] method and its proposed model(Bespoke-MiniCheck-7B) to examine whether facts in the constructed knowledge graphs are supported by their original source materials. Specifically, we randomly sampled 1000 attribute and relation facts from the knowledge graphs constructed for TravelPlanner and used the corresponding original records as evidence. MiniCheck classified 967 facts as supported, yielding a source support rate of 96.7%. This result demonstrates that the constructed knowledge graphs are largely grounded in the original source materials, providing quantitative evidence for the factual grounding of the knowledge used in downstream agent reasoning.
(2) Execution Violations. To examine execution violations, we audited agent-generated actions in retained ToolQA trajectories for 917 tasks shared by OaK, ReAct, and AgentSpec. Automated scripts paired actions with their recorded outcomes and identified invalid execution attempts using existing operation definitions, parameter domains, and database schemas. We identified 8, 168 and 34 invalid attempts for OaK, ReAct, and AgentSpec, respectively. OaK had the fewest confirmed invalid execution attempts among the three methods. These observations are consistent with OaK’s design of constraining the action space through semantic methods and enforcing explicit execution boundaries. The counts are descriptive under each method’s original interface and run configuration.
(3) Failure Traceability. We provided the agents with explicit records of the error locations and causes identified from their first failed attempts. This allows us to evaluate whether execution failures can be clearly traced and diagnosed, providing the agents with actionable information about the sources of failure. We evaluate OaK, ReAct, and AgentSpec on the same set of failed tasks, obtaining second-attempt recovery rates of 25.32%, 13.92%, and 15.82%, respectively. OaK achieves the highest recovery rate, suggesting that its ontology kernel supports error localization and diagnosis through explicit function calls, arguments, and intermediate results. We have added these experiments and the corresponding analyses to Section 4.7 of the revised manuscript.
2. Comparison with AgentSpec.
Following your suggestion, we included AgentSpec as an additional baseline and evaluated it on all three benchmarks. With DeepSeek-v4-flash, OaK achieves a 55.90% final pass rate on TravelPlanner, compared with 28.80% for AgentSpec. On CRMArenaPro, OaK obtains B2B and B2C average scores of 78.38% and 75.20%, respectively, outperforming AgentSpec (53.67% and 54.81%). On ToolQA, OaK achieves a weighted-average accuracy of 64.58%, compared with 53.29% for AgentSpec. GPT-4o-mini shows the same overall trend. The complete results, including individual task categories and both backbone configurations, are reported in Tables 1–3 of the revised manuscript.
[1] MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents (Tang et al., EMNLP 2024)