acceptodds
Under review as a conference paper at ICLR 2027

KnowledgeClaw: Towards Agentic Knowledge Engineering via Omnimodal Hypergraphs

Abstract

Knowledge engineering (KE) turns raw files into a reasoning-ready knowledge base, a task now within reach of autonomous LLM agents. KE has progressed from chunk and graph retrieval-augmented generation (RAG) to memory agents and agentic, multimodal KE systems. However, a live knowledge base needs omnimodality, a closed-loop lifecycle and multi-granular application at once, yet each existing system satisfies at most two and no benchmark measures all three. We formalize Omnimodal Knowledge Engineering (OmniKE) and release OmniKE-Bench, six tasks with 1,829 model-free items over the real files of ten personas in five modalities. We propose KnowledgeClaw, an agentic system that builds an omnimodal typed hypergraph, maintains it through a closed-loop lifecycle agent and routes each query to the granularity it needs. Against 30 baselines, KnowledgeClaw ranks first on all six tasks with an OKE of 55.9, +14.6 over the strongest baseline. Our software and data are publicly available.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
agentic knowledge engineering, omnimodal knowledge hypergraph, benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Fu8wOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes KnowledgeClaw, an omnimodal agentic framework for knowledge engineering, to address several limitations of existing approaches, including incomplete modality coverage, insufficient support for multi-granular knowledge retrieval and application, and the lack of closed-loop knowledge lifecycle management. In addition, the authors introduce OmniKE-Bench, a benchmark designed to systematically evaluate the effectiveness and advantages of the proposed framework.

Critical strengths
  1. Well-motivated problem formulation. Starting from existing RAG approaches and agentic knowledge engineering (KE) systems, the authors provide a clear and comprehensive analysis of the practical challenges faced by current knowledge engineering methods, particularly in terms of modality coverage, knowledge lifecycle management, and multi-granular knowledge application. These challenges are well motivated and relevant to real-world scenarios.

  2. Comprehensive system and benchmark design. The authors introduce OmniKE-Bench and KnowledgeClaw to systematically evaluate and address the identified limitations of existing approaches. The overall framework is well structured, and the proposed benchmark and system together form a relatively complete solution to the targeted problems.

  3. Extensive experimental evaluation. The paper conducts experiments against a diverse set of baselines spanning multiple categories of existing methods. The extensive comparisons demonstrate the effectiveness of the proposed framework and highlight its advantages over prior approaches.

Critical weaknesses
  1. Limited methodological novelty. Although the paper addresses important practical challenges, the proposed approach appears to be primarily an engineering-oriented system integration rather than a fundamentally novel methodological contribution. In particular, the paper seems to offer limited innovation in the underlying agent architectures, reasoning mechanisms, or agent optimization techniques.

  2. Limited agentic capabilities in closed-loop lifecycle management. The proposed closed-loop lifecycle agent (C2) appears to rely largely on predefined rules or procedures for adding, deleting, and refreshing knowledge. From the current presentation, it is unclear to what extent this component involves genuinely autonomous agentic decision-making, rather than rule-based knowledge management.

  3. Lack of explicit error detection and correction mechanisms. The proposed framework does not appear to incorporate explicit mechanisms for detecting and correcting errors introduced during data processing and knowledge construction. Given that the system relies on LLMs to process heterogeneous information, potential errors may propagate into the constructed knowledge base and subsequently affect downstream retrieval and reasoning.

  4. Insufficient details on free-form answer evaluation. The paper does not clearly explain how information is reliably extracted from LLM-generated free-form answers during evaluation. Since such responses may exhibit substantial linguistic variability, the absence of a detailed evaluation protocol raises concerns about the reliability and reproducibility of the reported results.

  5. Minor presentation issues. The font sizes in some figures appear inconsistent, which slightly affects readability. Additionally, the manuscript contains several minor grammatical issues that should be corrected to improve the overall presentation quality.

Questions for authors
  1. Distinction from existing agentic memory systems. How does the proposed knowledge management framework fundamentally differ from conventional agentic memory systems? In particular, what are the key conceptual or technical distinctions between KnowledgeClaw's knowledge lifecycle management and existing agentic memory mechanisms?

  2. Error detection and correction. Does the proposed framework incorporate any explicit error detection, validation, or correction mechanisms when using LLMs for knowledge extraction and processing? If so, could the authors elaborate on how these mechanisms operate and how they mitigate error propagation throughout the knowledge engineering pipeline?

  3. Reliability of free-form answer evaluation. How are evaluation-relevant information and predictions reliably extracted from LLM-generated free-form answers? Could the authors provide more details about the evaluation protocol, particularly how variations in response formatting and expression are handled to ensure accurate and consistent scoring?

  4. Fairness and generalizability of benchmark-based evaluation. Since the authors introduce OmniKE-Bench and primarily evaluate their proposed framework on this benchmark, there may be concerns about potential evaluation bias or benchmark-specific optimization. How do the authors ensure the fairness and objectivity of the evaluation? Have additional experiments been conducted on existing benchmarks or independent evaluation settings to further validate the generalizability of the proposed approach?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Fu8w (Part 1/3)

Thank you very much for your time and thorough review, and for recognizing the well-motivated problem formulation, the comprehensive system and benchmark design, and the extensive experimental evaluation. Your concerns are that KnowledgeClaw is primarily an engineering-oriented system integration (W1-A), limited innovation in agent architectures, reasoning mechanisms or agent optimization (W1-B), C2's reliance on preset rules for adding, deleting and refreshing (W2), the lack of explicit mechanisms for detecting and correcting errors from LLM extraction (W3 / Q2), how results are reliably extracted from free-form answers and how format variations are handled (W4 / Q3), inconsistent font sizes in some figures and minor grammatical issues (W5), how the knowledge management differs from agentic memory systems (Q1), and potential evaluation bias or benchmark-specific optimization (Q4).

We have substantially improved the paper's writing, figure and table layout, and supplementary experiments, among other details, and we kindly invite you to open the revised PDF to view them.

In the revised PDF, our revisions for you are in blue, together with those in red (red marks changes that address points raised by several reviewers). Passages that are coloured but whose wording is unchanged are tagged Highlighted below, and we coloured them only to help you find them.


W1-A: "Primarily an engineering-oriented system integration."

Thank you very much. Each of the three components is a design choice with a checkable guarantee that prior systems lack; we prove Propositions 1–3 in App. B and check them in §5.4–§5.6:

DesignPrior workGuaranteeEvidence
C1: file-level catalogue records as hyperedges, joining any modality at shared entitiesn-ary facts from text chunksProp. 1every modality pair joined; 95.4–99.2% of files joined (§5.4, modified, red)
C2: provenance on every element, edits committed along itedit single facts or pagesProp. 2Eq. (7) never fails over 2,000 edits (§5.5, modified, blue)
C3: intent routing over file, subgraph and community tiersfixed top-kProp. 3router equals the best tier on all six tasks (§5.6, modified, red)
  • C1 changes the unit of knowledge: it moves from a fact cut out of a text chunk to a file's catalogue record, so files of any modality meet at the entities they share. Prop. 1 guarantees that any two files, of the same or different modalities, whose records name a common entity are joined within two hyperedges.
  • C2 changes how knowledge is kept current: it commits an edit along everything derived from the changed file, where prior systems edit single facts or pages. Prop. 2 guarantees that no evidence rests on a deleted or superseded file.
  • C3 changes how evidence is chosen: it replaces a fixed top-k with the tier each query needs. Prop. 3 guarantees that every file a query names enters its evidence, whatever k and the corpus size. Thank you for this comment.

Therefore, the contribution beyond system integration is the representation and its three guarantees, each checked by an experiment.


W1-B: "Limited innovation in agent architectures, reasoning mechanisms or agent optimization."

Thank you very much. The agentic part is routing: the agent decides per query which granularity to use and per edit which elements to update, and only the routed system matches the best row in every column.

  • Frozen model: the model is frozen and shared with every baseline (§5.1, highlighted, red), so the gains come from how knowledge is represented and maintained. Without C1 or C2, OKE still stays above every baseline (49.1 and 50.4 vs 41.3).
  • One component per task family (§5.6, Figure 8c, §5.3, modified, red; Table 1):
Same answererT1T3T4T5OKE
File tier only51.61.53.695.652.2
Hypergraph walk always36.115.820.995.351.2
w/o C2 (no lifecycle)51.615.820.949.650.4
KnowledgeClaw (routed)51.615.820.995.655.9
  • The File tier alone answers T1 and T5 but almost none of the joins. Always walking the hypergraph answers the joins but loses on T1, and removing C2 keeps everything except T5.
  • Per query: Eq. (8) routes by rule, and a one-word routing prompt settles the rest (App. A.3, modified, blue). The rules cover a query that names a file or entity, is short, or carries a lookup or synthesis cue, and only the remaining queries go to the model.
  • Per edit: each edit reaches exactly the elements whose provenance holds the changed file (§4.2, highlighted, blue). Thank you for raising this point.

As a result, the innovation on the agent side is the routing of each query and each edit, which puts KnowledgeClaw ahead of any fixed granularity.


AuthorsOpenReview ↗

Response to Reviewer Fu8w (Part 2/3)


W2: "C2 appears to rely on predefined rules for adding, deleting and refreshing."

Thank you very much. C2's rules are fixed by design, because Prop. 2 requires every commit to restore soundness; after an edit KnowledgeClaw rises from 93.4 to 97.8 with 2.2% superseded answers.

  • How C2 commits: the agent takes each edit, re-reads only the new version and commits along provenance. Del calls no model (a median 9 ms), so no deletion can be hallucinated (§4.2; App. A; Alg. 1; App. E.6; highlighted, blue).
  • Against model-driven writes: these keep returning superseded values after a Refresh (App. F.4, modified; Table 13, highlighted; blue; Sup. = % superseded answers; A/B = before/after the edit):
MethodPhase APhase BSup.
A-MEM77.421.273.7
Graphiti (Zep)81.835.861.3
MemOS57.738.056.9
MemPro77.480.319.0
w/o C2 (append only)93.45.894.2
KnowledgeClaw93.497.82.2
  • After each commit, KnowledgeClaw's evidence holds the new value on 95.5% of the items and the old one on none (App. F.4, blue).
  • At scale: over 2,000 edits Eq. (7) always holds, an append-only store ends with 10,148 stale nodes, and a Refresh takes 2.6 s against 118 s to add the whole corpus (§5.5, Figure 7, blue). We appreciate this remark.

Hence, the predefined rules keep the knowledge base sound and the answers current after every add, delete and refresh.


W3 / Q2: "No explicit mechanisms for detecting and correcting errors from LLM extraction."

Thank you very much. We added grounding and extraction checks, so every stage now has a measured check (all blue):

StageMechanismMeasuredIn the paper
ExtractionJSON mode, temperature 0, 13 fixed fields; fields outside the schema, placeholders, values of 400+ characters and duplicates dropped; an event kept only if its date parsesonly 32 of 4,059 files get an empty recordApp. A.2, F.7 (new)
Groundingcopied values checked against the text C1 read97.8% of 29,633 grounded (PDF 98.2, text 97.5, media 96.7)§5.8, Table 19 (new)
CommitEq. (7) asserted after every commitholds on all ten hypergraphs and over 2,000 editsAlg. 1, App. E.6 (highlighted); F.7 (new)
Correctionprovenance traces a value to its file; a Refresh of that file alone fixes ita commit touches a median 0.33% of the hypergraph§5.5 (modified)
AnswersJSON contract per task; an unparseable answer scores 0only 0.5% of KnowledgeClaw's answers unparsedApp. A.5 (highlighted); D.3 (modified); Table 17 (new)
  • Errors are caught where they arise, and a value that slips through stays traceable to its file, so a Refresh of that file alone corrects it. Thank you for this question.

Therefore, errors from LLM extraction are detected by per-stage checks and corrected by refreshing the one file they trace to.


W4 / Q3: "How are predictions reliably extracted from free-form answers, and how are format variations handled?"

Thank you very much. Answers come back in a per-task JSON contract, and a deterministic matcher absorbs format variations; two new checks (blue; §5.8, Figure 11b–c and App. F.7, Tables 17–18) show that the lead holds under exact matching.

  • Extraction: every method answers through one prompt with a JSON output contract per task (App. A.5, highlighted); a missing or unparseable answer scores 0 (App. D.3, modified). A predicted value is therefore parsed, never extracted from prose.
  • Format variations: a predicted value matches a gold value if it is the same date at the gold's precision in any format, equal after normalisation, or has word-token F1 ≥ 0.5. Pairs are matched one-to-one and scored by set F1, with no model in the loop (§3, §5.1, App. D.3, highlighted).
  • Check ①, strict against lenient matching (all 34 rows): under exact matching KnowledgeClaw stays first on every task, its margin grows from 14.6 to 16.7 OKE, and the rankings agree with Kendall τ_b = 0.90:
Method (strict / lenient)T1T3T4OKEUnparsed
MemPro9.9 / 11.41.5 / 13.52.7 / 8.237.9 / 41.35.7%
OKF24.2 / 27.21.8 / 8.82.7 / 17.335.7 / 38.51.3%
Median of 30 baselines3.6 / 4.91.1 / 6.40.5 / 2.717.4 / 18.72.2%
KnowledgeClaw48.4 / 51.614.0 / 15.820.0 / 20.954.6 / 55.90.5%
  • Check ②, an LLM judge (Table 18): on 275 judged answers from T1, T3 and T4, the matcher agrees on 95.7% of 585 gold-value decisions (Cohen's κ = 0.91; 95.1 / 96.0 / 98.0% per task) and is the stricter side (9 vs 16 one-sided matches). Thank you for asking.

As a result, predictions are parsed from JSON, the matcher absorbs format variations, and the lead grows under exact matching.


AuthorsOpenReview ↗

Response to Reviewer Fu8w (Part 3/3)


W5: "Font sizes in some figures appear inconsistent; minor grammatical issues."

Thank you very much. We fixed both the font sizes and the grammar.

  • Figures: Figures 1–4 now share one text size (7 pt), and Figures 5–11 are drawn at print size in one font and size (Georgia, 8 pt), so every figure stays legible at print size. (Figures are not colour-marked.)
  • Text: we also rewrote the text in short, direct sentences. Thank you for the careful reading.

Therefore, font sizes are now consistent across all figures, and the grammatical issues are fixed.


Q1: "How does the knowledge management differ from agentic memory systems?"

Thank you very much. Memory systems manage what an agent remembers, while KnowledgeClaw manages a knowledge base whose ground truth is a set of changing files (§2 and App. G, highlighted, blue). This leads to a different unit, update and guarantee:

Agentic memory (MemGPT, A-MEM, Mem0, Graphiti, MemOS, MemPro)KnowledgeClaw
Unitfacts or notes distilled from a streama typed catalogue record per file, in five modalities
Updateedit single memories, usually decided by the modelan edit to one file reaches every element derived from it, along provenance (Eq. 6–7)
Guaranteenone on stalenessProp. 2: nothing rests on a deleted or superseded file
Superseded answers on T5 (Table 13, blue)19.0–73.7%2.2%
  • When a file is replaced: a memory system has to find and rewrite each fact distilled from it, usually by asking the model. KnowledgeClaw knows from provenance which elements rest on that file and updates all of them in one commit.
  • In practice (last row): KnowledgeClaw seldom returns a superseded value after a file changes, while systems that edit single memories often do. We are grateful for this question.

Hence, the knowledge management differs from agentic memory in its unit, its update and its guarantee against stale answers.


Q4: "Potential evaluation bias or benchmark-specific optimization; how are fairness and generality ensured?"

Thank you very much. Fairness and generality rest on six measures, including the new matched-input control, and the lead holds on each of the ten corpora.

  • Why a new benchmark: OmniKE-Bench is a primary contribution, because no benchmark measures omnimodality, a closed-loop lifecycle and multi-granular application together (§1). The table gives each measure with its evidence and location:
MeasureEvidenceIn the paper
Same stack for every methodone frozen model, encoder and answer prompt; official prompts and defaults; all six tasks attempted§5.1, App. A.5 (highlighted, red); E.1 (modified, red)
Model-free benchmarkstems paraphrased blind to answer and options; six deterministic checks discard leaking items§3, App. D.2 (highlighted, blue)
No per-setting tuningglobal defaults; OKE moves by at most 0.4 over the sweep§5.7, Figure 10 (modified, blue); App. F.6, Table 15 (highlighted, blue)
Ten independent corporafirst on every persona, by +4.7 to +21.1 (sign test, p = 0.002)§5.2, Figure 5c, App. F.2 (modified, blue); Table 10 (highlighted, blue)
Matched inputsgiven its lookup and records, the strongest baseline trails by 5.2 OKE§5.8, Table 2 (new, red)
Short evidence256 tokens give 51.2 OKE, above every full-context baseline, and 512 tokens give 55.0§5.7 (modified, blue); Figure 9b (not colour-marked, cited for reference)
  • Fairness: the first two rows concern how we run the comparison and how we built the benchmark.
  • Benchmark-specific optimization: the next two rows would expose it, since a lead that needed tuning, or held on only some personas, would show there.
  • Generality: the last two rows show that the lead survives matched inputs and evidence cut to 256 tokens. Thank you for raising this.

Therefore, the lead is tied to no single persona, setting or scorer, which addresses both evaluation bias and benchmark-specific optimization.


Summary of Revisions (every location cited above, by status and colour)

WhereStatusColour
§5.8, Figure 11b–c; App. A.2, F.7, Tables 17–19newblue
§5.8, Table 2, Figure 11a; App. F.7, Table 16newred
§5.2, §5.5, §5.7, Figures 5c, 7, 10; App. A.3, D.3, F.2, F.4modifiedblue
§2, §3, §4.2, §5.1; App. A, A.5, C, D.2, D.3, E.6, F.6, G; Tables 10, 13, 15highlightedblue
§5.3, §5.4, §5.6, Figure 8c; App. E.1modifiedred
§5.1; App. A.5highlightedred

At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and substantially improved and updated the paper accordingly. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider raising the score to 3 or above. Thank you very much!

Reviewer_PVoDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

KnowledgeClaw combines multimodal ingestion, provenance-based updates, and intent-routed retrieval to maintain a unified knowledge base.

Critical strengths
  1. A coherent system integrating knowledge construction, maintenance, and retrieval.
  2. A benchmark with model-free gold answers and scoring, with substantial reported gains over 30 baselines.
Critical weaknesses
  1. Omnimodal coverage is limited: only three evaluation items involve audio/video, and cross-modal reasoning covers only healthcare image–PDF pairs.
  2. The ablations change multiple components together, leaving the specific contribution of the hypergraph unclear.
  3. Complex reasoning remains weak, with F1 scores of 15.8 on T3 and 20.9 on T4.
Questions for authors

How much improvement remains over a simple structured-record baseline using the same native media reader, exact file access, and provenance-based updates, but no hypergraph?

2 comments
AuthorsOpenReview ↗

Response to Reviewer PVoD (Part 1/2)

Thank you very much for your time and thorough review, and for recognizing the coherent system integrating knowledge construction, maintenance and retrieval, the benchmark with model-free gold answers and scoring, and the substantial gains over 30 baselines. Your concerns are that only three evaluation items involve audio/video (W1-A), that cross-modal reasoning covers only healthcare image–PDF pairs (W1-B), that the ablations change several components at once (W2), how much remains over a record baseline with the same native reader, exact file access and provenance updates but no hypergraph (Q), and that complex reasoning remains weak on T3 and T4 (W3).

We have substantially improved the paper's writing, figure and table layout, and supplementary experiments, among other details, and we kindly invite you to open the revised PDF to view them.

In the revised PDF, our revisions for you are in green, together with those in red (red marks changes that address points raised by several reviewers). Passages that are coloured but whose wording is unchanged are tagged Highlighted below, and we coloured them only to help you find them.


W1-A: "Only three evaluation items involve audio/video."

Thank you very much. We added a scope statement (new App. H, red) and measurements for every audio and video step: all files, 180 of each, are read natively, joined to text and PDF, and edited with 0 violations of Eq. (7).

  • Scope: only 3 items involve audio or video (Table 5), which is too few for a reliable audio or video score. The claim is therefore that all five modalities are read, joined and maintained, while the scored items cover mainly PDF/text and images.
  • Each audio and video step is measured (§5.4, modified; Figure 6c, highlighted; App. F.3, F.4, new; red):
All ten personasAudioVideo
Files read natively by C1180180
Hyperedges to text / PDF218 / 206216 / 208
Stream edits / Eq. (7) violations100 / 084 / 0
  • Read: C1 reads every file natively from the raw media. It produces a summary and a timed transcript for each recording, and keyframes, audio and transcript for each video.
  • Joined: the files reach text and PDF through hundreds of hyperedges (218 and 206 for audio, 216 and 208 for video), so audio and video enter the same hypergraph as the documents that share their entities.
  • Maintained: the edit stream includes 100 audio and 84 video edits. Every commit keeps every element and every index entry on files that are still present, as Eq. (7) requires, so no evidence rests on a deleted or superseded file.
  • These counts come from the committed hypergraphs and the edit stream, so they do not depend on the 3 items. Gold is never model-made, and no task is padded with model-made items (§3, App. D.2, highlighted; App. D.1, modified; red), so we added no model-made audio or video items.
  • As App. H says, this evidence shows that the pipeline handles audio and video end to end, and the accuracy of answers about them is beyond what these items can show. Thank you for raising this point.

Therefore, the three audio/video items are complemented by measured reading, joining and maintenance of every audio and video file, and the new appendix states the scope of the claim.


W1-B: "Cross-modal reasoning covers only healthcare image–PDF pairs."

Thank you very much. The revised paper measures cross-modal ability at three levels, and KnowledgeClaw leads on both scored ones.

  • ① Coreference: T2, a media file to its description, runs in all ten personas (400 items; Table 6, modified, red), and KnowledgeClaw scores 82.8 against 78.2 for the best baseline (Table 1). Its four candidate descriptions come from one collection, so a system must read the media file closely enough to tell them apart.
  • ② Joins: T4 is built wherever two modalities share authority-controlled entities (App. D.1, modified, red), and KnowledgeClaw leads every scored slice, e.g. T4 image+PDF 20.9 vs 17.3 and 10.9 (best text-bridge, omnimodal; Figure 6a, Table 11, highlighted, red). The first sees media only through captions and transcripts, and the second reads raw media as we do.
  • ③ The hypergraph: it joins every pair of the five modalities (§5.4, red), so audio and video are joined too, as W1-A shows. Thank you for this comment; the three levels are now easy to find by colour.

As a result, cross-modal reasoning is measured as matching in all ten personas and as joins on T4, and the hypergraph links all five modalities in addition to the healthcare image–PDF pairs.


AuthorsOpenReview ↗

Response to Reviewer PVoD (Part 2/2)


W2: "Ablations change several components at once."

Thank you very much. Three views in the revised paper isolate the hypergraph's contribution: the new matched-input control, the join-shape analysis and the one-at-a-time ablations.

  • ① External control (red; §5.8, Table 2): baselines given our records and handle lookup reach at most 50.7 OKE, 5.2 below. The gap sits on the joins (T4) and the lifecycle (T5), the two places that only the hypergraph walk and the commits (C2) supply.
  • ② Join shapes: the analysis of T3 + T4 Join shapes (§5.4, modified; App. F.3, Table 11, highlighted; green) shows that the walk leads on every shape. It leads most on shared entities and file anchors, where similarity on its own has nothing to match, and the w/o C1 column trails it on every shape:
T3 + T4 joinnKnowledgeClawBest baselinew/o C1
Shared entity8414.38.37.1
File anchor12212.37.45.7
Entity anchor38818.314.211.6
  • ③ One component at a time: we removed one component at a time (§5.3, modified, red): each costs most where it acts. Without C1 or C2, OKE stays above every baseline (49.1 and 50.4 vs 41.3), and removing C1 costs up to 21.3 on a modality slice (App. F.3, green). We appreciate this remark; it led us to set these views side by side.

Therefore, the hypergraph walk is now separated from the records and the lifecycle, component by component.


Q: "How much remains over a structured-record baseline with the same native reader, exact file access and provenance updates, but no hypergraph?"

Thank you very much. This baseline is the File tier of the paper: over it, the hypergraph walk lifts T3 from 1.5 to 15.8, T4 from 3.6 to 20.9 and OKE from 52.2 to 55.9, while T1 and T5 stay equal.

  • The variant: the File tier answers from the named files' cards alone, with the same answerer. It keeps native reading, exact file access and C2, and drops the hypergraph walk (§5.6, Figure 8c, modified, red):
Same answererT1T3T4T5OKE
Records only, no hypergraph walk51.61.53.695.652.2
KnowledgeClaw51.615.820.995.655.9
  • On T1 and T5, where records and the lifecycle suffice, the two rows are equal. When a question needs a join, the records alone answer almost nothing. Thank you for asking for this isolation.

Hence, what remains over the record baseline is the gain on the joins, where T3 rises from 1.5 to 15.8 and T4 from 3.6 to 20.9.


W3: "Complex reasoning remains weak, with F1 of 15.8 on T3 and 20.9 on T4."

Thank you very much. We revised the join-depth analysis and added a misread count: KnowledgeClaw leads the best baseline at every depth, and only 37 of the 510 items (7%) are misread (App. F.3, new, green).

  • The tasks: T3/T4 need every hop and are scored by set F1 (App. D.1). Flat retrieval alone reaches only 2.5 and 0.0 on them (App. D.3, green; Table 8), while KnowledgeClaw leads on both (15.8 vs 13.5 and 20.9 vs 17.3; Table 1).
  • By depth (§5.4, Figure 6b; App. F.3, Table 11; green): the table compares KnowledgeClaw with the strongest of the 30 baselines and with their median:
T3 + T42 hops (326)3 hops (156)4 hops (28)
Best baseline16.912.23.6
Median of 30 baselines7.52.60.0
KnowledgeClaw19.313.57.1
  • Deeper joins are harder for every method. The median baseline falls to 0.0 at four hops and the best baseline keeps 3.6, while KnowledgeClaw keeps 7.1.
  • Misreads: these are items answered wrongly although their evidence holds a gold value. The count separates errors of reading from errors where the evidence lacks the answer, and few errors are misreads. Thank you for this comment; it prompted the new count.

As a result, on these multi-hop tasks KnowledgeClaw leads the best baseline at every depth, and it seldom misreads evidence that already holds the answer.


Summary of Revisions (every location cited above, by status and colour)

WhereStatusColour
§5.8, Table 2, Figure 11a; App. F.3, F.4, F.7, H; Table 16newred
App. F.3 (misread)newgreen
§5.4modifiedgreen
Figure 6b; App. D.3, F.3; Table 11highlightedgreen
§5.3, §5.4, §5.6, Figure 8c; App. D.1; Table 6modifiedred
§3, Figures 6a, 6c; App. D.2; Tables 5, 11highlightedred

At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and substantially improved and updated the paper accordingly. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider raising the score further. Thank you very much!

Reviewer_NJvXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper formalizes the task of OmniKE, a setting in which an agentic system must process a person's files across 5 modalities (text, pdf, image, audio, video) into a knowledge base, and the knowledge base must satisfy omnimodality, a closed loop lifecycle under file edits, and multi-granular querying. For the proposed task setting, the paper introduced OmniKE-Bench,which covers 1920 items over 3960 real files from 10 personas across 6 tasks. The author further proposed a system called KnowledgeClaw for this setting, which has three components: (1) read each source file with Gemini and extracts catalogue record, and merge records into a hypergraph. (2) for each node, it tracks which file it comes from, so deleting a file removes only the relevant supported nodes. (3) finds the files and entities named in a query, then solve by either a file lookup, a graph walk, or a community summary. The authors compared against 30 baselines, and KnowledgeClaw ranks first on all 6 tasks and 10 personas.

Critical strengths

The authors conducted extensive and thorough evaluation, which covers 30 baselines across 10 major method families. The ablation studies are well designed, which removes one component at a time, and it proves the large effects of different components on specific tasks..

The benchmark is of high quality, with gold answers come from human-made metadata, deterministic parsing, official transcripts, and version chains, rather than from a model.

Critical weaknesses
  1. The comparison is unfair. The proposed method is tailored for the proposed problem setting and benchmark, so the improvement over baselines are due to the setting match, not system quality. For example, Gold answer for T1 and T6 is drawn from library catalogue fields. The first component of KnowledgeClaw asks the model to fill exactly those fields, and even tells it to write MeSH terms the way Mesh does. No baseline gets this exact problem setting and schema as how the problem is defined. Similarly, queries contain file handles and the third component looks them up directly, so Proposition 3 says little more than “a filename index finds a named file.” The case study shows 16 of 30 baselines failing for this reason, but no baseline is given a filename index. A comparison with RAG plus handle lookup would show how much of the +14.6 OKE comes from these two shortcuts rather than from the hypergraph, lifecycle, or routing.

  2. The omnimodal claim is weakly supported. Only 3 of 1829 items have audio and video, the rest of the benchmark is video and image, so 5 modalities are claimed but only two are really tested.

Questions for authors
  1. None of the 30 baselines were designed for OmniKE, and none receives the catalogue schema or the file-handle or the file-handle index. How do you ensure each baseline is evaluated fairly rather than forced into the introduced problem setting? What do the strongest baselines score when they are given the same record prompt and handle lookup as KnowledgeClaw?

  2. Can you report per-modality results for audio and video on every task, with enough items to give a meaningful estimate? If the current benchmark cannot support this, the omnimodal claim in the title and abstract should be restated.

5 comments
AuthorsOpenReview ↗

Response to Reviewer NJvX (Part 1/3)

Thank you very much for your time and thorough review, and for recognizing the extensive evaluation over 30 baselines in 10 method families, the well-designed ablations, and the high-quality benchmark whose gold is never model-made. Your concerns are that the record prompt asks for exactly the catalogue fields that T1/T6 gold is drawn from (W1-A), that queries contain file handles, so the lookup proposition says little more than a filename index (W1-B), the baselines failing in the case study (W1-C), how each baseline is evaluated fairly (W1-D / Q1(a)), what the strongest baselines score with the same record prompt and handle lookup (W1-D / Q1(b)), that only three items have audio and video (W2), and per-modality results for audio and video (Q2).

We have substantially improved the paper's writing, figure and table layout, and supplementary experiments, among other details, and we kindly invite you to open the revised PDF to view them.

In the revised PDF, our revisions for you are in purple, together with those in red (red marks changes that address points raised by several reviewers). Passages that are coloured but whose wording is unchanged are tagged Highlighted below, and we coloured them only to help you find them.


W1-A: "The record prompt asks for exactly the catalogue fields that T1/T6 gold is drawn from, even MeSH terms written the way MeSH writes them."

Thank you very much. The new §5.8 now measures the record prompt separately: with KnowledgeClaw's record cards, the strongest baselines improve sharply, yet none reaches it on T1 or T6.

  • One general schema: a general library catalogue, the same for every persona and modality; the third row explains the MeSH forms:
PointEvidenceIn the paper
Goldhuman-assigned catalogue fields and controlled-vocabulary labels, never model-made§3 (highlighted, red)
One schema for all13 general cataloguing fields plus dated events, one set for all ten personas and all five modalities§4.1 (highlighted, purple); App. A.2 (modified, purple)
Why preferred formsa join matches forms, not topics, so subject asks for each standard vocabulary's preferred form (arXiv codes, MeSH descriptors)App. A.2 (modified, purple)
Records on T1given KnowledgeClaw's record cards, MemPro rises from 11.4 to 45.2 and StandardRAG from 6.1 to 43.1, yet none reaches 51.6§5.8, Table 2 (new, red)
Records on T6with the same cards, the best baseline reaches 75.8, still below 76.0§5.8, Table 2 (new, red)
Records serve every taskcards are 100% of the evidence on T1, T2, T5 and T6, and anchor the T3/T4 walksApp. F.5, Table 14 (highlighted, purple)
T1 type inventoryevery class with a human-made row in its persona and modality, never tailored to any single itemApp. D.1 (modified, purple)
  • The records are part of C1, and the paper now reports their effect openly. Thank you for raising this.

Therefore, the catalogue fields that T1/T6 gold is drawn from now go to the baselines too, and KnowledgeClaw still leads on both.


W1-B: "Queries contain file handles, so Proposition 3 says little more than 'a filename index finds a named file'."

Thank you very much. Proposition 3 states a contrast with similarity search: every baseline receives the handle, yet flat top-k loses the named file, while C3 always retrieves it.

  • Where: Proposition 3 (§4.3), Figure 8a (§5.6) and App. F.5, Table 14 (all highlighted, purple), and App. B.3 (modified, purple). Thank you for this comment.
  • Same handle for all: each document enters every baseline headed by its file name, the only handle items use (App. E.1, modified, red):
Per taskT1T2T3T4T5T6
Named file in evidence, flat top-k (%)93.866.585.423.163.586.6
Named file in evidence, C3 (%)100100100100100100
Score, flat top-k15.734.88.05.562.857.5
Score, C351.682.815.820.995.676.0
  • Why similarity loses the file: when ℓ ≥ k look-alike files are exchangeable to the encoder, top-k returns the named file with probability min(k, ℓ+1)/(ℓ+1) < 1, and fusing similarity lists does not lift this bound (App. B.3, Eq. 14). Proposition 3 guarantees that the named file's card reaches the evidence for any k, λ, θ and corpus size. On T2 the handle is a content hash that carries no topic, so an encoder has nothing to match (Table 3).
  • The lookup alone: answering from the named files' cards alone (File tier) gives 1.5 on T3 and 3.6 on T4, against 15.8 and 20.9 for the routed system (§5.6, Figure 8c, modified, red).

As a result, the proposition is presented as a contrast with similarity search, and the filename lookup is reported apart from the gain on the joins.


AuthorsOpenReview ↗

Response to Reviewer NJvX (Part 2/3)


W1-C: "The case study shows 16 of 30 baselines failing for this reason."

Thank you very much. The case study now states its scope: "This case isolates lookup (C3) and native reading (C1); Tables 1 and 2 measure the overall gain." (§5.9, modified, purple).

  • What the case isolates: it explains why similarity-based methods fail on such an item. How much of the overall lead comes from lookup is measured by the matched-input control in W1-D below.
  • The item: the video has three sibling lectures by the same speaker as distractors, and only the topic separates them (Table 3, highlighted, purple):
MethodEvidence about the named videoAnswer
NaiveGenerationno corpus; the hashed file name carries no topic(C) ✗
LightRAGentity-relation graph from the text bridge(A) ✗
StandardRAGtext-bridge chunks ranked by similarity(B) ✗
w/o C310.5k characters; five of its ten hyperedges are identifiers of other videos, no passage from it(B) ✗
KnowledgeClawits card, 3.7k characters read from the raw video(D) ✓
  • The failing baselines: sixteen of the 30 baselines choose a sibling, seven (A), seven (B) and two (C) (App. F.8). A method that ranks by similarity to the query sees four lectures with near-identical openings, and even our own hypergraph without C3 fills its evidence with other videos.
  • KnowledgeClaw: it resolves the handle to the named file and reads its card, so the topic, the one field in which the candidates differ, is in its evidence. C1 read that card from the raw video, so the case also involves native reading. Thank you for this remark.

Hence, the case study is scoped to the mechanism behind these failures, and the overall gain is measured separately.


W1-D / Q1(a): "How do you ensure each baseline is evaluated fairly?"

Thank you very much. Every baseline runs under one shared protocol (red), so only the evidence each method supplies differs.

  • Shared stack: all methods use one frozen model, encoder and answer prompt.
  • Baselines as released: each baseline keeps its official prompts and defaults and attempts all six tasks (§5.1, App. A.5, highlighted; App. E.1, modified).
  • Same scorer: the answerer and scorer are those of Table 1 for every row, including the new control below. Thank you for asking.

Therefore, fairness is ensured by giving each baseline the same model, answer prompt and scorer as KnowledgeClaw.


W1-D / Q1(b): "What do the strongest baselines score with the same record prompt and handle lookup?"

Thank you very much. We added this control (new §5.8, Table 2 and Figure 11a; all rows in App. F.7, Table 16; new, red): with the same handle lookup and record cards, the strongest baseline, MemPro, rises from 41.3 to 50.7 OKE and still trails KnowledgeClaw by 5.2.

  • Design: in +H, each named file's ingested text goes before the baseline's own evidence (handle lookup). In +H+R, KnowledgeClaw's record card goes there instead (same record prompt). We replay the builds and retrieval, so only the evidence changes, and the step from +H to +H+R isolates the records. The +H rows of OKF and StandardRAG are listed in Table 16:
MethodT1T2T3T4T5T6OKE
MemPro11.478.213.58.278.869.541.3
+ H28.378.813.27.378.873.845.9
+ H + R45.281.013.57.378.875.850.7
OKF + H32.271.89.017.354.470.641.3
+ H + R43.778.28.817.354.472.645.6
StandardRAG + H24.375.09.510.951.871.039.6
+ H + R43.176.510.010.951.871.844.4
KnowledgeClaw51.682.815.820.995.676.055.9
  • ① The record cards lift every baseline beyond lookup alone (+H+R scores above +H for all three), which supports C1's premise that typed catalogue records are what these systems lack. With them, OKF and StandardRAG gain 16.5 and 37.0 on T1 but stay below 51.6 (App. F.7).
  • ② The gap stays on the joins (T4 20.9 vs at most 17.3) and the lifecycle (T5 95.6 vs 78.8), which only the hypergraph walk and the commits (C2) supply. On T2, where every item names its file, the best matched baseline reaches 81.0, below 82.8.
  • The two inputs you singled out thus explain part of the gap, which the paper now reports openly. The rest comes from native reading and routing on T1, the hypergraph walk on the joins, and the lifecycle on T5. Thank you for proposing this control.

As a result, with the same record prompt and handle lookup the strongest baseline scores 50.7 OKE against 55.9 for KnowledgeClaw, 5.2 behind, and KnowledgeClaw stays first on all six tasks.


AuthorsOpenReview ↗

Response to Reviewer NJvX (Part 3/3)


W2: "Only 3 items have audio and video."

Thank you very much. We added a scope statement (new App. H, red) and audio and video measurements over all ten personas: every audio and video file is read, joined and maintained, although only 3 scored items involve these modalities (Table 5).

  • Claim: App. H states that all five modalities are read, joined and maintained, while the scored items cover mainly PDF/text and images.
  • Measured per step: read is the files read natively by C1, joined is the hyperedges, and maintained is the stream edits with their Eq. (7) check:
All ten personasAudioVideoIn the paper
Files read natively by C1180180§5.4, modified; Table 12, highlighted; App. F.3, new; red
Hyperedges to text / PDF218 / 206216 / 208Figure 6c, highlighted; App. F.3, new; red
Hyperedges audio–video214214Figure 6c; App. F.3
Stream edits / Eq. (7) violations100 / 084 / 0App. F.4 (new, red)
Native reading promptsummary, timed transcriptkeyframes, audio, transcriptApp. A.1 (modified, red); Table 4, Figure 12 (unchanged, cited for reference)
  • Every audio and video file is read natively from the raw media, and every edit to them keeps Eq. (7) intact, so no evidence rests on a deleted or superseded file. These counts come from the committed hypergraphs and the edit stream, so they do not depend on the 3 items.
  • Reading prompts: they ask for what tells a file apart from its look-alikes. For a recording, that is the speakers and a time-marked transcript, and for a video also what its keyframes show, with any on-screen text (App. A.1).
  • All pairs joined: every pair of the five modalities is joined, from 96 image–audio to 8.2k text–PDF hyperedges (§5.4, modified, red); an entity whose files span ≥2 modalities gets a cross-modal hyperedge (§4.1, Eq. 4, highlighted, red), which links the entity to all of its files at once.
  • No padding: gold is never model-made, and no task is padded with model-made items (§3, App. D.2, highlighted; App. D.1, modified; red), so we added no model-generated audio or video items. Thank you for raising this point.

Hence, the few audio and video items are complemented by measured reading, joining and maintenance of every audio and video file, and the new appendix states the scope.


Q2: "Can you report per-modality results for audio and video?"

Thank you very much. With 3 items, an audio or video score would not be reliable, so we report none; the paper reports per-modality results for the scored slices, PDF and image, and KnowledgeClaw leads all of them.

  • Scored slices: each slice is PDF or image (§5.4, modified; Figure 6a, Table 11, highlighted; red). Text-bridge baselines see media only through captions and transcripts, while omnimodal baselines read raw media as we do, so each slice compares us with the best of both groups:
Slice (n)KnowledgeClawBest text-bridgeBest omnimodal
T1 PDF (344)56.430.818.8
T1 image (56)21.65.37.7
T2 image (397)82.978.670.0
T3 PDF (125)46.440.032.8
T4 image+PDF (110)20.917.310.9
T6 image (304)71.466.865.5
  • The lead is largest on T1, which extracts entities from a single PDF or image (56.4 vs 30.8, and 21.6 vs at most 7.7). On the other slices it holds against the stronger of the two groups each time: T2 images (82.9 vs 78.6), T3 PDFs (46.4 vs 40.0), T4 image+PDF (20.9 vs 17.3) and T6 images (71.4 vs 66.8).
  • The audio and video items: the case-study video is one of these 3. Sixteen of the 30 baselines miss it, while KnowledgeClaw answers from the card C1 read from the raw video, as W1-C shows (§5.9, modified; Table 3, highlighted; purple). Thank you for this request.

Therefore, per-modality results are reported for PDF and image, while audio and video are covered by the measurements above.


Summary of Revisions (every location cited above, by status and colour)

WhereStatusColour
§5.8, Table 2, Figure 11a; App. F.1, F.3, F.4, F.7, H; Table 16newred
§5.9; App. A.2, B.3, D.1modifiedpurple
§4.1, §4.3, §5.6, Figure 8a; App. F.5; Tables 3, 14highlightedpurple
§5.4, §5.6, Figure 8c; App. A.1, D.1, E.1modifiedred
§3, §4.1, §5.1, Figures 6a, 6c; App. A.5, D.2; Tables 5, 11, 12highlightedred

At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and substantially improved and updated the paper accordingly. If our responses and revisions address your concerns, we would be deeply grateful if you could kindly reconsider raising the score to 3 or above. Thank you very much!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers,

Thank you very much to all reviewers for your time and thorough reviews. We are glad that you found the "extensive and thorough evaluation, which covers 30 baselines across 10 major method families", ablations that are "well designed" and a benchmark that "is of high quality" (NJvX); "a coherent system integrating knowledge construction, maintenance, and retrieval" and "a benchmark with model-free gold answers and scoring" (PVoD); and the "well-motivated problem formulation", the "comprehensive system and benchmark design" and the "extensive experimental evaluation" (Fu8w). Your shared questions concern the fairness of the comparison on a benchmark we also introduce, the coverage of audio and video, and the hypergraph's contribution.

We have substantially improved the paper's writing, figure and table layout, and supplementary experiments, among other details, and we kindly invite all reviewers to open the revised PDF to view them.

In the revised PDF, each reviewer's revisions have their own colour, purple for Reviewer NJvX, green for Reviewer PVoD and blue for Reviewer Fu8w, and red marks changes that address points raised by several reviewers (the overlapping concerns). Some passages are coloured although their wording is unchanged. We tag them Highlighted in the reviewer threads and coloured them only to help you find them. Red mainly covers the matched-input control, the audio and video counts (App. F.3, F.4), App. H, and the ablation, modality-join and File-tier passages (§5.3, §5.4, §5.6, Figure 8c). Purple covers the record schema (App. A.2), Proposition 3 (§4.3, App. B.3) and the case-study scope (§5.9). Green covers the join depths and shapes (§5.4, Figure 6b) and the misread count, and blue covers the scorer, grounding and extraction checks and the lifecycle, persona and robustness results (§5.2, §5.5, §5.7; App. F.4).

We added a matched-input control, scorer, grounding and extraction checks, and audio and video measurements (new §5.8; App. F.3, F.4, F.7 and H), and revised §5.2–§5.7 and §5.9:

Concern / changeWhere in the revised PDF (colour)Key resultReviewer(s)
Fairness: added a control giving baselines our handle lookup and record cardsnew §5.8, Table 2, Figure 11a; App. F.7, Table 16 (red)MemPro rises from 41.3 to 50.7 OKE, and KnowledgeClaw stays first on all six tasks, 5.2 OKE aheadall
Protocol: the same for every method§5.1, App. A.5; App. E.1 (red); App. D.3; §5.2, §5.7 (blue)One frozen model, encoder and answer prompt; official prompts and defaults; all six tasks attempted; per-task JSON answers scored by a deterministic matcher with no model in the loop. First on all ten personas, by +4.7 to +21.1 (sign test, p = 0.002), with global defaults and no per-setting tuningall
Hypergraph: revised the File-tier analysis (records without the walk)§5.6, Figure 8c (red)The walk lifts T3 from 1.5 to 15.8, T4 from 3.6 to 20.9 and OKE from 52.2 to 55.9NJvX, PVoD
Complex reasoning: revised the join-depth analysis and added a misread count§5.4, Figure 6b; new App. F.3; Table 11 (green)KnowledgeClaw leads the best baseline at every depth; only 37 of 510 items (7%) are misreadPVoD
Audio and video: added measurements§5.4, Figure 6c; new App. F.3, F.4 (red)All 180 audio and 180 video files are read natively, with 0 Eq. (7) violations over 100 audio and 84 video editsNJvX, PVoD
Omnimodal claim: added App. H on its scopenew App. H (red); §5.4Across all ten personas, KnowledgeClaw reads, joins and maintains audio and video, and every pair of the five modalities is joined, from 96 image–audio to 8.2k text–PDF hyperedges. Gold is never model-made, and no task is padded with model-made itemsNJvX, PVoD
Scorer: added two checksnew §5.8, Figure 11b–c; App. F.7, Tables 17–18 (blue)KnowledgeClaw's lead grows from 14.6 to 16.7 OKE under exact matching; the matcher agrees with an LLM judge on 95.7% of 585 gold-value decisionsFu8w
Extraction errors: added grounding and extraction checksnew §5.8, Table 19; App. A.2, F.7 (blue)97.8% of 29,633 copied values grounded; only 32 of 4,059 files get an empty recordFu8w
Lifecycle: revised the comparison with agentic memoryApp. F.4, Table 13 (blue)2.2% superseded answers after an edit, against 19.0–73.7% for memory systemsFu8w
Presentationthroughout (not colour-marked)Figures 1–4 share one text size (7 pt), and Figures 5–11 are drawn at print size in one font and size (Georgia, 8 pt); text rewritten in short, direct sentencesFu8w

Each reviewer's thread below gives the full evidence and ends with a table of every cited location, by status and colour. If any point remains unclear, we would be glad to discuss it further. Thank you again for your valuable time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.