- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes KnowledgeClaw, an omnimodal agentic framework for knowledge engineering, to address several limitations of existing approaches, including incomplete modality coverage, insufficient support for multi-granular knowledge retrieval and application, and the lack of closed-loop knowledge lifecycle management. In addition, the authors introduce OmniKE-Bench, a benchmark designed to systematically evaluate the effectiveness and advantages of the proposed framework.
Critical strengths
-
Well-motivated problem formulation. Starting from existing RAG approaches and agentic knowledge engineering (KE) systems, the authors provide a clear and comprehensive analysis of the practical challenges faced by current knowledge engineering methods, particularly in terms of modality coverage, knowledge lifecycle management, and multi-granular knowledge application. These challenges are well motivated and relevant to real-world scenarios.
-
Comprehensive system and benchmark design. The authors introduce OmniKE-Bench and KnowledgeClaw to systematically evaluate and address the identified limitations of existing approaches. The overall framework is well structured, and the proposed benchmark and system together form a relatively complete solution to the targeted problems.
-
Extensive experimental evaluation. The paper conducts experiments against a diverse set of baselines spanning multiple categories of existing methods. The extensive comparisons demonstrate the effectiveness of the proposed framework and highlight its advantages over prior approaches.
Critical weaknesses
-
Limited methodological novelty. Although the paper addresses important practical challenges, the proposed approach appears to be primarily an engineering-oriented system integration rather than a fundamentally novel methodological contribution. In particular, the paper seems to offer limited innovation in the underlying agent architectures, reasoning mechanisms, or agent optimization techniques.
-
Limited agentic capabilities in closed-loop lifecycle management. The proposed closed-loop lifecycle agent (C2) appears to rely largely on predefined rules or procedures for adding, deleting, and refreshing knowledge. From the current presentation, it is unclear to what extent this component involves genuinely autonomous agentic decision-making, rather than rule-based knowledge management.
-
Lack of explicit error detection and correction mechanisms. The proposed framework does not appear to incorporate explicit mechanisms for detecting and correcting errors introduced during data processing and knowledge construction. Given that the system relies on LLMs to process heterogeneous information, potential errors may propagate into the constructed knowledge base and subsequently affect downstream retrieval and reasoning.
-
Insufficient details on free-form answer evaluation. The paper does not clearly explain how information is reliably extracted from LLM-generated free-form answers during evaluation. Since such responses may exhibit substantial linguistic variability, the absence of a detailed evaluation protocol raises concerns about the reliability and reproducibility of the reported results.
-
Minor presentation issues. The font sizes in some figures appear inconsistent, which slightly affects readability. Additionally, the manuscript contains several minor grammatical issues that should be corrected to improve the overall presentation quality.
Questions for authors
-
Distinction from existing agentic memory systems. How does the proposed knowledge management framework fundamentally differ from conventional agentic memory systems? In particular, what are the key conceptual or technical distinctions between KnowledgeClaw's knowledge lifecycle management and existing agentic memory mechanisms?
-
Error detection and correction. Does the proposed framework incorporate any explicit error detection, validation, or correction mechanisms when using LLMs for knowledge extraction and processing? If so, could the authors elaborate on how these mechanisms operate and how they mitigate error propagation throughout the knowledge engineering pipeline?
-
Reliability of free-form answer evaluation. How are evaluation-relevant information and predictions reliably extracted from LLM-generated free-form answers? Could the authors provide more details about the evaluation protocol, particularly how variations in response formatting and expression are handled to ensure accurate and consistent scoring?
-
Fairness and generalizability of benchmark-based evaluation. Since the authors introduce OmniKE-Bench and primarily evaluate their proposed framework on this benchmark, there may be concerns about potential evaluation bias or benchmark-specific optimization. How do the authors ensure the fairness and objectivity of the evaluation? Have additional experiments been conducted on existing benchmarks or independent evaluation settings to further validate the generalizability of the proposed approach?
7 comments
Response to Reviewer Fu8w (Part 2/3)
W2: "C2 appears to rely on predefined rules for adding, deleting and refreshing."
Thank you very much. C2's rules are fixed by design, because Prop. 2 requires every commit to restore soundness; after an edit KnowledgeClaw rises from 93.4 to 97.8 with 2.2% superseded answers.
- How C2 commits: the agent takes each edit, re-reads only the new version and commits along provenance. Del calls no model (a median 9 ms), so no deletion can be hallucinated (§4.2; App. A; Alg. 1; App. E.6; highlighted, blue).
- Against model-driven writes: these keep returning superseded values after a Refresh (App. F.4, modified; Table 13, highlighted; blue; Sup. = % superseded answers; A/B = before/after the edit):
| Method | Phase A | Phase B | Sup. |
|---|---|---|---|
| A-MEM | 77.4 | 21.2 | 73.7 |
| Graphiti (Zep) | 81.8 | 35.8 | 61.3 |
| MemOS | 57.7 | 38.0 | 56.9 |
| MemPro | 77.4 | 80.3 | 19.0 |
| w/o C2 (append only) | 93.4 | 5.8 | 94.2 |
| KnowledgeClaw | 93.4 | 97.8 | 2.2 |
- After each commit, KnowledgeClaw's evidence holds the new value on 95.5% of the items and the old one on none (App. F.4, blue).
- At scale: over 2,000 edits Eq. (7) always holds, an append-only store ends with 10,148 stale nodes, and a Refresh takes 2.6 s against 118 s to add the whole corpus (§5.5, Figure 7, blue). We appreciate this remark.
Hence, the predefined rules keep the knowledge base sound and the answers current after every add, delete and refresh.
W3 / Q2: "No explicit mechanisms for detecting and correcting errors from LLM extraction."
Thank you very much. We added grounding and extraction checks, so every stage now has a measured check (all blue):
| Stage | Mechanism | Measured | In the paper |
|---|---|---|---|
| Extraction | JSON mode, temperature 0, 13 fixed fields; fields outside the schema, placeholders, values of 400+ characters and duplicates dropped; an event kept only if its date parses | only 32 of 4,059 files get an empty record | App. A.2, F.7 (new) |
| Grounding | copied values checked against the text C1 read | 97.8% of 29,633 grounded (PDF 98.2, text 97.5, media 96.7) | §5.8, Table 19 (new) |
| Commit | Eq. (7) asserted after every commit | holds on all ten hypergraphs and over 2,000 edits | Alg. 1, App. E.6 (highlighted); F.7 (new) |
| Correction | provenance traces a value to its file; a Refresh of that file alone fixes it | a commit touches a median 0.33% of the hypergraph | §5.5 (modified) |
| Answers | JSON contract per task; an unparseable answer scores 0 | only 0.5% of KnowledgeClaw's answers unparsed | App. A.5 (highlighted); D.3 (modified); Table 17 (new) |
- Errors are caught where they arise, and a value that slips through stays traceable to its file, so a Refresh of that file alone corrects it. Thank you for this question.
Therefore, errors from LLM extraction are detected by per-stage checks and corrected by refreshing the one file they trace to.
W4 / Q3: "How are predictions reliably extracted from free-form answers, and how are format variations handled?"
Thank you very much. Answers come back in a per-task JSON contract, and a deterministic matcher absorbs format variations; two new checks (blue; §5.8, Figure 11b–c and App. F.7, Tables 17–18) show that the lead holds under exact matching.
- Extraction: every method answers through one prompt with a JSON output contract per task (App. A.5, highlighted); a missing or unparseable answer scores 0 (App. D.3, modified). A predicted value is therefore parsed, never extracted from prose.
- Format variations: a predicted value matches a gold value if it is the same date at the gold's precision in any format, equal after normalisation, or has word-token F1 ≥ 0.5. Pairs are matched one-to-one and scored by set F1, with no model in the loop (§3, §5.1, App. D.3, highlighted).
- Check ①, strict against lenient matching (all 34 rows): under exact matching KnowledgeClaw stays first on every task, its margin grows from 14.6 to 16.7 OKE, and the rankings agree with Kendall τ_b = 0.90:
| Method (strict / lenient) | T1 | T3 | T4 | OKE | Unparsed |
|---|---|---|---|---|---|
| MemPro | 9.9 / 11.4 | 1.5 / 13.5 | 2.7 / 8.2 | 37.9 / 41.3 | 5.7% |
| OKF | 24.2 / 27.2 | 1.8 / 8.8 | 2.7 / 17.3 | 35.7 / 38.5 | 1.3% |
| Median of 30 baselines | 3.6 / 4.9 | 1.1 / 6.4 | 0.5 / 2.7 | 17.4 / 18.7 | 2.2% |
| KnowledgeClaw | 48.4 / 51.6 | 14.0 / 15.8 | 20.0 / 20.9 | 54.6 / 55.9 | 0.5% |
- Check ②, an LLM judge (Table 18): on 275 judged answers from T1, T3 and T4, the matcher agrees on 95.7% of 585 gold-value decisions (Cohen's κ = 0.91; 95.1 / 96.0 / 98.0% per task) and is the stricter side (9 vs 16 one-sided matches). Thank you for asking.
As a result, predictions are parsed from JSON, the matcher absorbs format variations, and the lead grows under exact matching.
Response to Reviewer Fu8w (Part 3/3)
W5: "Font sizes in some figures appear inconsistent; minor grammatical issues."
Thank you very much. We fixed both the font sizes and the grammar.
- Figures: Figures 1–4 now share one text size (7 pt), and Figures 5–11 are drawn at print size in one font and size (Georgia, 8 pt), so every figure stays legible at print size. (Figures are not colour-marked.)
- Text: we also rewrote the text in short, direct sentences. Thank you for the careful reading.
Therefore, font sizes are now consistent across all figures, and the grammatical issues are fixed.
Q1: "How does the knowledge management differ from agentic memory systems?"
Thank you very much. Memory systems manage what an agent remembers, while KnowledgeClaw manages a knowledge base whose ground truth is a set of changing files (§2 and App. G, highlighted, blue). This leads to a different unit, update and guarantee:
| Agentic memory (MemGPT, A-MEM, Mem0, Graphiti, MemOS, MemPro) | KnowledgeClaw | |
|---|---|---|
| Unit | facts or notes distilled from a stream | a typed catalogue record per file, in five modalities |
| Update | edit single memories, usually decided by the model | an edit to one file reaches every element derived from it, along provenance (Eq. 6–7) |
| Guarantee | none on staleness | Prop. 2: nothing rests on a deleted or superseded file |
| Superseded answers on T5 (Table 13, blue) | 19.0–73.7% | 2.2% |
- When a file is replaced: a memory system has to find and rewrite each fact distilled from it, usually by asking the model. KnowledgeClaw knows from provenance which elements rest on that file and updates all of them in one commit.
- In practice (last row): KnowledgeClaw seldom returns a superseded value after a file changes, while systems that edit single memories often do. We are grateful for this question.
Hence, the knowledge management differs from agentic memory in its unit, its update and its guarantee against stale answers.
Q4: "Potential evaluation bias or benchmark-specific optimization; how are fairness and generality ensured?"
Thank you very much. Fairness and generality rest on six measures, including the new matched-input control, and the lead holds on each of the ten corpora.
- Why a new benchmark: OmniKE-Bench is a primary contribution, because no benchmark measures omnimodality, a closed-loop lifecycle and multi-granular application together (§1). The table gives each measure with its evidence and location:
| Measure | Evidence | In the paper |
|---|---|---|
| Same stack for every method | one frozen model, encoder and answer prompt; official prompts and defaults; all six tasks attempted | §5.1, App. A.5 (highlighted, red); E.1 (modified, red) |
| Model-free benchmark | stems paraphrased blind to answer and options; six deterministic checks discard leaking items | §3, App. D.2 (highlighted, blue) |
| No per-setting tuning | global defaults; OKE moves by at most 0.4 over the sweep | §5.7, Figure 10 (modified, blue); App. F.6, Table 15 (highlighted, blue) |
| Ten independent corpora | first on every persona, by +4.7 to +21.1 (sign test, p = 0.002) | §5.2, Figure 5c, App. F.2 (modified, blue); Table 10 (highlighted, blue) |
| Matched inputs | given its lookup and records, the strongest baseline trails by 5.2 OKE | §5.8, Table 2 (new, red) |
| Short evidence | 256 tokens give 51.2 OKE, above every full-context baseline, and 512 tokens give 55.0 | §5.7 (modified, blue); Figure 9b (not colour-marked, cited for reference) |
- Fairness: the first two rows concern how we run the comparison and how we built the benchmark.
- Benchmark-specific optimization: the next two rows would expose it, since a lead that needed tuning, or held on only some personas, would show there.
- Generality: the last two rows show that the lead survives matched inputs and evidence cut to 256 tokens. Thank you for raising this.
Therefore, the lead is tied to no single persona, setting or scorer, which addresses both evaluation bias and benchmark-specific optimization.
Summary of Revisions (every location cited above, by status and colour)
| Where | Status | Colour |
|---|---|---|
| §5.8, Figure 11b–c; App. A.2, F.7, Tables 17–19 | new | blue |
| §5.8, Table 2, Figure 11a; App. F.7, Table 16 | new | red |
| §5.2, §5.5, §5.7, Figures 5c, 7, 10; App. A.3, D.3, F.2, F.4 | modified | blue |
| §2, §3, §4.2, §5.1; App. A, A.5, C, D.2, D.3, E.6, F.6, G; Tables 10, 13, 15 | highlighted | blue |
| §5.3, §5.4, §5.6, Figure 8c; App. E.1 | modified | red |
| §5.1; App. A.5 | highlighted | red |
At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and substantially improved and updated the paper accordingly. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider raising the score to 3 or above. Thank you very much!
Response to Reviewer Fu8w (Part 1/3)
Thank you very much for your time and thorough review, and for recognizing the well-motivated problem formulation, the comprehensive system and benchmark design, and the extensive experimental evaluation. Your concerns are that KnowledgeClaw is primarily an engineering-oriented system integration (W1-A), limited innovation in agent architectures, reasoning mechanisms or agent optimization (W1-B), C2's reliance on preset rules for adding, deleting and refreshing (W2), the lack of explicit mechanisms for detecting and correcting errors from LLM extraction (W3 / Q2), how results are reliably extracted from free-form answers and how format variations are handled (W4 / Q3), inconsistent font sizes in some figures and minor grammatical issues (W5), how the knowledge management differs from agentic memory systems (Q1), and potential evaluation bias or benchmark-specific optimization (Q4).
We have substantially improved the paper's writing, figure and table layout, and supplementary experiments, among other details, and we kindly invite you to open the revised PDF to view them.
In the revised PDF, our revisions for you are in blue, together with those in red (red marks changes that address points raised by several reviewers). Passages that are coloured but whose wording is unchanged are tagged Highlighted below, and we coloured them only to help you find them.
W1-A: "Primarily an engineering-oriented system integration."
Thank you very much. Each of the three components is a design choice with a checkable guarantee that prior systems lack; we prove Propositions 1–3 in App. B and check them in §5.4–§5.6:
Therefore, the contribution beyond system integration is the representation and its three guarantees, each checked by an experiment.
W1-B: "Limited innovation in agent architectures, reasoning mechanisms or agent optimization."
Thank you very much. The agentic part is routing: the agent decides per query which granularity to use and per edit which elements to update, and only the routed system matches the best row in every column.
As a result, the innovation on the agent side is the routing of each query and each edit, which puts KnowledgeClaw ahead of any fixed granularity.