acceptodds
Under review as a conference paper at ICLR 2027

GAKU: Towards Dynamic Cognition on Knowledge Graphs

Abstract

Knowledge graphs (KGs) provide structured and reliable factual knowledge, yet large language models (LLMs) often struggle to align their parametric knowledge with the semantics, conventions, and coverage of a specific KG. This mismatch is particularly problematic in domain-specific knowledge reasoning, where conventional adaptation often requires substantial annotation and fine-tuning. In this paper, we propose GAKU, a multi-agent framework that builds dynamic cognition over KGs. GAKU follows a human-like cognitive loop: inspecting local graph evidence, reflecting on mistakes, inducing ontology-level cognition, validating them, and reusing the validated knowledge in subsequent decisions. The resulting cognition state bridges episodic experience and reusable reasoning skills, enabling an LLM to adapt to unfamiliar KG-specific semantics without parameter fine-tuning. Extensive experiments on KG fact verification and completion show that GAKU consistently outperforms strong fine-tuning and training-free baselines, using only 100 labeled examples to construct dynamic cognition. Further analyses demonstrate that the gains generalize across diverse backbone LLMs and support cross-model cognition transfer, highlighting explicit cognitive adaptation as a data-efficient and parameter-free approach to KG-grounded reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Knowledge Graph, Multi-Agent System, Large Language Model, Ontology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Va9vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GAKU adapts LLM reasoning to the semantics of a target knowledge graph without updating model parameters. To achieve this, it uses 100 labeled examples from each of the datasets to construct natural-language cognition entries at both the relation and entity levels through an error-driven process. As GAKU processes labeled examples, if the prediction result does not match the gold label, it triggers the generation and verification of candidate cognitions, and then retries the originating query using the validated candidate cognition. These candidates are added to the persistent cognition state only if the retry corrects the error, which allows the accepted experience to inform the processing of later examples. Once cognition construction is complete, the resulting state is frozen for testing. For each test query, GAKU retrieves relevant cognition entries along with local KG evidence, using them as context to guide its prediction. To assess the effectiveness of this approach, the authors evaluate GAKU on knowledge graph fact verification and entity completion, and the results from these evaluations show that GAKU achieves better performance than the baseline methods.

Critical strengths
  1. The paper focuses on a concrete difficulty in KG-grounded reasoning. An LLM may retrieve relevant facts from the target knowledge graph but still interpret a relation according to its own prior knowledge rather than the graph's conventions. Given this mismatch, the distinction between relation-level and entity-level cognition makes sense. Moreover, GAKU consistently outperforms existing state-of-the-art methods without parameter updates and requires only a small number of training examples for cognition construction.

  2. The paper evaluates GAKU across several dimensions. The experiments cover fact verification on two KG datasets and entity completion on three. Specifically, the backbone comparison includes seven models, while the completion experiments use three models. Ablation studies examine the roles of the validator, the retry mechanism, and the persistent cognition state.

Critical weaknesses
  1. My main concern is how much of GAKU's contribution comes from its specific design. Learning from errors, storing natural-language experience, and reusing it without parameter updates already have precedents in Reflexion. Generating relation knowledge with an LLM and filtering it for reuse is also related to ChatRule. Therefore, GAKU's contribution should be assessed through the gains from its relation-level and entity-level distinction, its dynamic cognition mechanism, and its feasibility in practical use.

  2. Cognition construction has a sequential dependency, as the authors acknowledge in Appendix A. Each example uses the cognition accumulated from earlier examples, which limits parallel processing across examples. Larger training sets and cognition stores may also place greater demands on retrieval, reasoning, and conflict handling. The experiments use only 100 examples per dataset for cognition construction, so it remains unclear how larger construction sets and cognition stores would affect performance and computational cost.

Questions for authors
  1. How do you determine a cognition entry's scope and exceptions beyond its originating example? Do you check for conflicts with existing entries or whether updates turn previously correct predictions into errors?

  2. Could you explain the CoDeX-S accuracy discrepancies in Table 2? For w/o Validator, w/o Retry, and w/o Dynamic Cognition, the reported values are 83.01%, 82.47%, and 79.45%, compared with 83.15%, 82.61%, and 79.62% derived from Precision and Recall. Please provide the evaluated sample counts, confusion matrices, metric definitions, and aggregation procedure.

  3. Appendix G provides prompts for fact verification. Could the authors provide or point to the corresponding prompts, output format, and answer-parsing rules for entity completion?

  4. Table 4 reports hyperparameter sensitivity only for DeepSeek on CoDeX-S. Do the same trends hold for other backbones or datasets?

Reviewer_H3fqOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GAKU, a multi-agent framework for adapting LLMs to knowledge-graph-specific semantics without parameter fine-tuning. Using 100 labeled examples per dataset, the framework constructs an explicit memory of relation-level cognition and entity-specific experiences through an error-driven generation, validation, and revision loop. The resulting cognition is frozen and retrieved alongside local graph evidence during inference.

The evaluation covers KG fact verification on CoDeX-S and FB15K-237N and four-choice KG completion on ProbeKGC. The authors report improvements across multiple backbone models and investigate component ablations and cross-model cognition transfer. The results suggest practical benefits, although comparisons with closely related experience-based learning methods and further analysis of cognition generalization are needed to establish the specific contribution.

Critical strengths
  1. A practical mechanism for KG-specific adaptation without parameter updates. GAKU addresses the mismatch between an LLM's prior knowledge and the semantic conventions of a target KG. Its explicit, provenance-aware memory makes the adaptation inspectable and reusable. Constructing this memory from only 100 labeled examples per dataset is appealing when task annotations are limited, although the framework also accesses the larger training graph.

  2. Useful empirical evidence across complementary settings. The evaluation includes both fact verification and multiple-choice graph completion across several knowledge graphs and backbone models. Comparisons with reproduced RAG-based baselines under matched backbone and retrieval settings support the practical effectiveness of the framework. The validator, retry, and cognition ablations provide additional evidence that these components contribute to performance. Cross-model cognition transfer also demonstrates the possibility of using a stronger model to construct guidance for a smaller reasoning model.

Critical weaknesses
  1. Insufficiently established novelty over general-purpose skill-learning frameworks. GAKU's central mechanism—extracting reusable textual guidance from experience, maintaining it externally, and retrieving it for subsequent reasoning without updating model parameters—is conceptually close to skill-generation and evolution frameworks such as AutoSkill. Its potentially distinctive elements are the relation/entity-specific cognition representation and the error-driven validation-and-retry mechanism. However, the current comparisons do not establish whether these designs offer advantages over a suitably adapted generic skill-learning framework. The existing graph-completion results demonstrate effectiveness relative to the evaluated baselines, but do not isolate the benefit of KG-specific cognition from experience-based skill acquisition more generally. A comparison with a task-adapted AutoSkill baseline, using the same labeled examples, backbone, graph access, and comparable resource budgets, would help establish the methodological contribution. An untyped skill-memory baseline would further clarify whether separating relation-level and entity-level cognition is necessary.

  2. Insufficient evidence of out-of-distribution generalization of the learned cognition. Reusability is central to the proposed cognition representation, yet the experiments do not explicitly isolate its effectiveness under distribution shifts. Each dataset receives its own 100 cognition-construction examples, so successful evaluation across multiple datasets does not establish cross-dataset transfer of the learned memory. Similarly, the cross-model experiment transfers cognition between backbone models rather than between data distributions. The original test splits provide evidence of held-out performance, but the paper does not separately report results for entities absent from cognition construction or relations without learned cognition. A controlled evaluation on unseen entities within covered relations, together with shifts in entity types or relation-specific subcases, would more directly test whether the learned guidance captures transferable semantics rather than locally useful experience. This is particularly important because cognition retrieval is relation-constrained and an update is accepted after correcting the same training example that triggered it.

Questions for authors
  1. Could the authors compare GAKU with a task-adapted AutoSkill baseline on the same fact-verification and graph-completion tasks? With supervision and resource budgets controlled, what advantages come specifically from GAKU's KG-oriented representation and update mechanism?

  2. How does GAKU perform with relation-only cognition, entity-only cognition, and an untyped skill memory? These comparisons would help isolate the contribution of the proposed relation/entity distinction.

  3. Could the authors evaluate transfer to entities absent from the cognition-construction examples while keeping the relevant relations covered? Within those relations, does the learned cognition remain beneficial across different entity types or semantic subcases, including cases where a retrieved rule should not apply?

  4. Given the exact relation matching used for cognition retrieval, how does the method behave on relations not represented in the learned memory? Please distinguish this setting from cross-model transfer and clarify which experiments, if any, evaluate distribution shift rather than a change of backbone.

Reviewer_np5mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a problem: an LLM's pre-trained knowledge can interfere when we want it to use knowledge from a knowledge graph. It aims to adapt an LLM to the meanings and conventions of a particular knowledge graph without changing its weights.

The approach, called GAKU, is a framework that uses three agents: a reasoning agent, which solves the main task; a cognition generation agent, which uses an incorrect output and the gold label to provide feedback; and a validation agent, which checks whether that feedback should be accepted. They use the same backbone for the three agents, but different prompts for each one.

With only 100 labelled examples, the framework tries to identify mistakes, including those caused by the LLM's pre-trained knowledge, and correct them. The agents write guidance that is saved and used in later prompts.

The approach shows some advantages over baselines such as RAG and Chain of Thought plus RAG. The reported results are better, although the size of the improvement varies. Compared with fine-tuning approaches, it does not require changes to the model's weights to create or use this guidance, and it also produces better results.

The paper compares the approaches on two main tasks: fact verification and knowledge graph completion. The authors report improvements over the baselines, experiments with several LLMs, and transfer of guidance from a stronger model to a smaller reasoner.

Critical strengths
  1. The paper is well written and easy to follow.
  2. Adapting an LLM to a KG using 100 labelled examples, without changing its weights, is a useful goal. The saved guidance can also be inspected.
  3. The authors test several datasets and backbones. The ablations show that validation and retry help on both fact-verification datasets.
Critical weaknesses
  1. I am not convinced that dynamic cognition is clearly different from earlier work such as ExpeL, LEAP, and Reflexion. These papers should be discussed, even if the complete GAKU framework is different.
  2. The experiments do not show whether retrieving similar examples from the same 100 labeled cases would work just as well. The current ablations do not establish the benefit over this simpler alternative.
  3. Some metrics do not match. For DeepSeek on FB15K-237N, precision 86.02% and recall 86.45% imply about 86.20% accuracy on the stated balanced test set, but Table 1 reports 86.63%. Several rows in Table 5 also need explanation.
  4. The same backbone generates and validates guidance, so both agents may make the same mistakes. The validator prompt also permits guidance from one example without broader evidence. I have not checked whether the code adds further safeguards.
  5. The results use one selection seed, without showing variation across example sets or their order. This makes it harder to judge the smaller gains and the effect of the sampling rules.
Questions for authors
  1. How does dynamic cognition differ from the lessons learned in ExpeL (Zhao et al., 2024), LEAP (Zhang et al., 2024), and Reflexion (Shinn et al., 2023)?

  2. Can you compare against retrieving the same labelled examples or corrected explanations, using the same backbone, graph access, and a similar prompt budget? Table 1 also needs RAG comparisons with DeepSeek to separate the backbone effect there.

  3. Can you explain the metric differences and provide the confusion matrices and calculation procedure?

  4. Why use the same backbone for cognition generation and validation? How many guidance items are rejected, and what mistakes does the validator catch?

  5. How many of the 100 examples are initially wrong, and how much do the results change with different samples and example orders?

Comments on the paper

1 comment
Area_Chair_ExovOpenReview ↗

All reviewers, please be responsive

Dear reviewers,

The authors have provided detailed rebuttal. Please respond to authors' rebuttal, If you have not done so. Thanks!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.