- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GAKU adapts LLM reasoning to the semantics of a target knowledge graph without updating model parameters. To achieve this, it uses 100 labeled examples from each of the datasets to construct natural-language cognition entries at both the relation and entity levels through an error-driven process. As GAKU processes labeled examples, if the prediction result does not match the gold label, it triggers the generation and verification of candidate cognitions, and then retries the originating query using the validated candidate cognition. These candidates are added to the persistent cognition state only if the retry corrects the error, which allows the accepted experience to inform the processing of later examples. Once cognition construction is complete, the resulting state is frozen for testing. For each test query, GAKU retrieves relevant cognition entries along with local KG evidence, using them as context to guide its prediction. To assess the effectiveness of this approach, the authors evaluate GAKU on knowledge graph fact verification and entity completion, and the results from these evaluations show that GAKU achieves better performance than the baseline methods.
Critical strengths
-
The paper focuses on a concrete difficulty in KG-grounded reasoning. An LLM may retrieve relevant facts from the target knowledge graph but still interpret a relation according to its own prior knowledge rather than the graph's conventions. Given this mismatch, the distinction between relation-level and entity-level cognition makes sense. Moreover, GAKU consistently outperforms existing state-of-the-art methods without parameter updates and requires only a small number of training examples for cognition construction.
-
The paper evaluates GAKU across several dimensions. The experiments cover fact verification on two KG datasets and entity completion on three. Specifically, the backbone comparison includes seven models, while the completion experiments use three models. Ablation studies examine the roles of the validator, the retry mechanism, and the persistent cognition state.
Critical weaknesses
-
My main concern is how much of GAKU's contribution comes from its specific design. Learning from errors, storing natural-language experience, and reusing it without parameter updates already have precedents in Reflexion. Generating relation knowledge with an LLM and filtering it for reuse is also related to ChatRule. Therefore, GAKU's contribution should be assessed through the gains from its relation-level and entity-level distinction, its dynamic cognition mechanism, and its feasibility in practical use.
-
Cognition construction has a sequential dependency, as the authors acknowledge in Appendix A. Each example uses the cognition accumulated from earlier examples, which limits parallel processing across examples. Larger training sets and cognition stores may also place greater demands on retrieval, reasoning, and conflict handling. The experiments use only 100 examples per dataset for cognition construction, so it remains unclear how larger construction sets and cognition stores would affect performance and computational cost.
Questions for authors
-
How do you determine a cognition entry's scope and exceptions beyond its originating example? Do you check for conflicts with existing entries or whether updates turn previously correct predictions into errors?
-
Could you explain the CoDeX-S accuracy discrepancies in Table 2? For w/o Validator, w/o Retry, and w/o Dynamic Cognition, the reported values are 83.01%, 82.47%, and 79.45%, compared with 83.15%, 82.61%, and 79.62% derived from Precision and Recall. Please provide the evaluated sample counts, confusion matrices, metric definitions, and aggregation procedure.
-
Appendix G provides prompts for fact verification. Could the authors provide or point to the corresponding prompts, output format, and answer-parsing rules for entity completion?
-
Table 4 reports hyperparameter sensitivity only for DeepSeek on CoDeX-S. Do the same trends hold for other backbones or datasets?
All reviewers, please be responsive
Dear reviewers,
The authors have provided detailed rebuttal. Please respond to authors' rebuttal, If you have not done so. Thanks!
AC