acceptodds
Under review as a conference paper at ICLR 2027

Motif-Driven, LLM-Free Benchmark Construction for Question Answering over IT Knowledge Graphs

Abstract

As operational environments become increasingly heterogeneous, organisations are adopting graph-based representations to integrate knowledge across incidents, services, configurations, dependencies and remediation procedures. This creates a need for reliable benchmarks that can evaluate question answering over the resulting operational knowledge graphs. Such evaluation has a fundamental ground-truth problem: the answer is a structured graph object whose correctness depends on the graph's schema and instances, yet benchmark references are commonly produced by language models. This makes it possible for a reference to contain plausible but unsupported relations, making disagreements between a system and its reference difficult to diagnose. We present a graph-first approach to benchmark construction in which ground truth is computed, rather than generated. Starting from structurally informative patterns in a knowledge graph, we compile deterministic graph operations into reference subgraphs and only then generate natural-language questions that describe those references. Language models therefore determine how a question is phrased, but not what constitutes its answer. The same grounding mechanism identifies questions that cannot be answered from the graph and separates schema-level, instance-level, and grounding ambiguities.

Primary area:
datasets and benchmarks
Keywords:
KGQA, Benchmark, Dataset Curation, IT Operations, Knowledge Graphs, Subgraph Sampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sic7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The submission introduces an LLM-free framework for constructing question answering (QA) benchmarks over IT Context Knowledge Graphs (CKGs). The main contribution is an executable reference construction pipeline for operational CKG QA that integrates evidence-bounded generation and abstention. Additionally, the authors provide graph-derived questions. The proposed pipeline first establishes informative graph structures and compiles their references before generating questions, effectively decoupling question formulation from ground-truth reference construction. The evaluation measures unsupported relation claims and validates the framework on a production operational CKG with released artifacts.

Critical strengths

None

Critical weaknesses

The main concern is an apparent contradiction regarding the determinism of the re-execution process for families B and C. While the authors claim the process is deterministic, they report an agreement score of only 0.83 under re-execution. In addition, the notion of "agreement" remains ambiguous and requires clarification. Furthermore, the mathematical formalism is excessive without adding clarity. Another major issue is that, despite being defined as "LLM-free," the framework is not entirely so. Overall, the submission presents several significant limitations and drawbacks that cannot be overlooked.

Questions for authors

My main recommendation for future revisions is to ensure the manuscript is thoroughly polished, complete, and clear before resubmission.

1 comment
AuthorsOpenReview ↗

Response to the reviewer sic7

Thank you for the review.

Determinism and “agreement”. You identified a genuine contradiction. It came from a verification-harness fault, not from the construction (see the general response). Agreement is now defined exactly: the share of references whose answer set, G* node set and typed G* edge set are identical after recompiling from (q, G_t). Under that definition it is 1.00 on both graphs, across fresh processes with different hash seeds and across complete rebuilds. “LLM-free”. We agree the wording overstated it. The precise claim is that no language model determines the reference. No model involved: set, path and dependency references, and every subgraph G*, are compiled deterministically. Our model-free mode builds the full benchmark this way. Model involved: models only phrase questions and A/D answer text. Each generated string must pass deterministic checks against G*: grounding precision, unsupported relations, step order, completeness and polarity. Questions must also pass an executable round-trip. In the full pipeline, 18 of 19 generated answers passed on the production graph. We will replace “LLM-free” with this formulation throughout. Formalism and polish. We will move the stage operators to the appendix, keep only the definitions the propositions use, reconcile every reported number with the instantiated benchmark, and fix the presentation issues.

Reviewer_biJjOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a graph-first benchmark construction framework for question answering over operational knowledge graphs. The authors argue that existing QA benchmarks often rely on language models to generate reference answers, which may introduce unsupported relations and unreliable ground truth. Instead, the proposed approach first mines informative graph structures, deterministically compiles reference subgraphs, and only then generates natural language questions or explanations constrained by the graph evidence. The framework includes question family classification, graph grounding, reference compilation, provenance tracking, verification, and abstention handling for unanswerable cases. The method is instantiated on an operational context knowledge graph derived from ITBench scenarios and evaluated through benchmark construction validation and downstream QA experiments.

Critical strengths

1.The strongest contribution is the idea of constructing gold references directly from graph structures rather than from generated text. This addresses a fundamental weakness in existing KGQA evaluation, where reference answers may contain unsupported information. The proposed principle that "language models determine how a question is phrased, but not what constitutes its answer" is conceptually important. 2.Many existing RAG evaluation methods focus on relevance or faithfulness but cannot verify whether retrieved evidence corresponds to the correct graph evidence. This paper directly targets this limitation.

Critical weaknesses

1.The evaluation is performed on a relatively small number of queries. For a benchmark construction paper, broader validation across more questions, graph snapshots, or domains would significantly strengthen the contribution. 2.The paper emphasizes motif mining, but it does not sufficiently demonstrate that motif-driven sampling is substantially better than simpler graph sampling or template-based generation. 3.The paper proposes graph hallucination metrics, but the reliability of converting free-form answers into graph assertions is unclear. More human validation or automatic extraction analysis is needed.

Questions for authors

1.How does the proposed motif-driven construction compare against random graph sampling or conventional template-based KGQA generation under the same graph snapshot? 2.Do the authors plan to evaluate the framework on larger and more diverse operational knowledge graphs? 3.How sensitive are benchmark results to the thresholds used for grounding and verification?

1 comment
AuthorsOpenReview ↗

Response to the reviewer biJj

We thank the reviewer for the constructive feedback and specific questions. We have addressed these concerns through additional experiments, comparisons, and sensitivity analyses.

W1/Q2: Scale and diversity

We extended validation to two substantially different graph sizes (35,875 and 553 nodes) and three successive production snapshots. Semantic-correctness experiments include both seed questions and generated question sets (see our response to reviewer dAmZ).

We acknowledge that evaluation across additional organisations remains necessary. Since the underlying graph data is proprietary, public release is restricted. We explicitly clarify these limitations in the revision.

W2/Q1: Motif-driven vs. simpler question generation

We compared four generation approaches on the same production snapshot, using 150 questions per method. Recall measures the fraction of the reference recovered by shortcut retrieval methods; lower recall indicates greater resistance to these shortcuts.

GeneratorBM25 R@50Embedding R@501-hop recall
Motif-first (ours)0.150.090.286
Random-walk paths0.430.430.251
One-hop templates0.370.300.551

Our motif-first approach is substantially less susceptible to lexical and embedding-based retrieval shortcuts. For relational motifs specifically, one-hop retrieval recovers only 0.005 of the reference, compared with 0.55 for one-hop templates. Moreover, random walks and one-hop templates cannot generate gap/negation, filtering, or ordered-procedure questions.

Comparison with SynthKGQA. We additionally evaluated random-subgraph-sampling-based generation using SynthKGQA. Of 287 generated QA pairs, 112 answers involve kubectl actions, and HAS_ACTION is the most frequent relation (131 occurrences). This reflects a bias toward abundant, lower-level ontology entities such as runbook steps, producing simple questions like “Which runbook contains step XYZ?” Such questions capture limited operational context compared with multi-hop reasoning chains such as Symptom → Problem → Resolution → Runbook_Used. Therefore uniform random sampling tends to pick nodes that appear in abundance, largely overlooking other node types and higher level entities that may be less frequently occuring in the graph but convey important information nonetheless.

Betweenness weighting. We also found that betweenness weighting favours structurally central intermediaries but does not consistently improve difficulty over uniform anchor sampling. We have revised our claim accordingly: difficulty primarily arises from motif structure, not centrality weighting.

W3: Reliability of free-text graph assertion extraction

We agree that extracting graph assertions from free-form answers requires stronger validation. We evaluated two extraction methods using synthetic answers with known assertion sets.

Parse-based extraction, previously used for text-based metrics, performs well on templated prose but recovers only 5–9% of assertions from natural paraphrases.

We therefore introduced mention-constrained extraction: entities are identified deterministically, a model proposes entity pairs, and asserted connections are verified against the graph.

On natural paraphrases, this achieves 0.955 precision and 0.93 recall, reducing the error in estimated hallucinated-bridge rate (HBR) from 0.41–0.43 to 0.05.

The revised evaluation uses this extractor for free-text HBR and recomputes the results in Section 9.

Q3: Threshold sensitivity

We rebuilt the benchmark while varying each threshold independently.

  • Production graph: Results are largely stable. Deduplication threshold, semantic grounding floor, and path bound preserve 34–35 of 35 references. Changing the gap-hop bound by ±1 affects 5–6 references.
  • Smaller graph: The semantic grounding floor is most sensitive, with system-ranking Kendall's τ=0.43–0.90\tau=0.43\text{--}0.90, compared with 0.90–1.000.90\text{--}1.00 for other thresholds.
  • Answer-gate threshold (γ\gamma): Varying γ\gamma from 0.6 to 1.0 affects first-attempt acceptance but not the final number of admitted answers.

The revision includes the complete sensitivity sweep and recommends graph-specific calibration of the semantic grounding floor.

Overall, these experiments clarify the contribution of motif-driven generation, strengthen free-text evaluation reliability, and establish the sensitivity and current generalisability limits of the framework.

Reviewer_dAmZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method for constructing a question-answering benchmark tailored to IT operations knowledge graphs. The process begins by deriving reference subgraphs through deterministic operations on the graph, followed by the generation of questions and their corresponding answers using either templates or LLMs. The benchmark covers four categories of questions—process-related, negation/omission-based, relationship/impact-oriented, and causality/recommendation-focused—while documenting the reasons for unanswerability, the sources of evidence, and confidence levels.

Critical strengths

S1. It is reasonable to construct the reference subgraph before generating natual language question.

S2. The paper provides a useful taxonomy of unanswerable questions covering common cases while preserving provenance, graph snapshots, and confidence labels.

Critical weaknesses

W1. The main text does not establish the semantic correctness of the generated question–reference pairs. Section 4.6 checks whether mentioned entities belong to the reference subgraph, but entity membership cannot verify relations, dependency directions, negation, or procedural ordering. A generated question can change these properties while retaining exactly the same entities and achieving grounding precision of 1. Therefore, the claim that unsupported procedural steps will be rejected does not follow from the stated constraint. Section 4.8 introduces model judges, but the main text reports neither their error-detection accuracy nor a measured semantic error rate for the retained pairs. Since reliable ground truth is the central contribution, these omissions leave the main correctness claim insufficiently supported.

W2. The intrinsic validation does not demonstrate the claimed reliability of reference construction. Section 4.8 states that re-execution on the same question and pinned snapshot must produce identical references, yet Section 6 reports agreement of only 0.83. The explanation that disagreements occur on sparse evidence does not resolve this contradiction: sparsity may affect answer completeness, but cannot itself cause a deterministic computation to return different outputs. The graph-mutation experiment also measures how much references change, rather than whether those changes are correct. Incorrect reference construction can still respond to graph perturbations. Together, these results leave uncertainty about whether errors measured by the benchmark originate from the evaluated systems or from its references.

Questions for authors

Q1. What independent evidence establishes that admitted questions preserve the complete semantics of their reference subgraphs? Please report error rates for relation direction, negation, and procedural ordering, and explain how these errors are detected by the entity-membership gate and model judges.

Q2.What are the denominator and comparison criterion behind the 0.83 re-execution agreement? If both the question and snapshot are fixed, what causes the discrepancies, and were the affected references included in the baseline evaluation?

Q3.How does incremental regeneration detect answer-changing updates outside the previous reference subgraph, such as deleting the only playbook link of a previously excluded incident? Have incrementally maintained references been compared with full recomputation after such updates?

1 comment
AuthorsOpenReview ↗

Response to Reviewer dAmZ

We thank the reviewer for the careful assessment and constructive feedback. We agree with the identified weaknesses and have addressed them through new validation mechanisms, controlled experiments, and corrected evaluations.

W1/Q1: Semantic correctness beyond entity grounding

We agree that grounding precision checks entity membership but cannot detect incorrect relations, polarity, or ordering. We therefore added two complementary deterministic validation mechanisms.

1. Question validation: Semantic round-trip. Each generated question is independently parsed into an executable graph specification, which must produce the same reference as the original specification. We evaluated this using controlled corruptions with known ground truth, requiring no manual annotation.

CorruptionProductionITBench
All corruptions71/7157/58
Polarity reversal16/169/9
Relation-direction reversal8/88/8
Target-type substitution7/74/5
Anchor substitution24/2422/22
Filter-value substitution8/86/6
Procedural order reversal8/88/8

The entity-only gate rejects just 32% and 29% of corruptions, respectively, and misses polarity, direction, ordering, and type errors.

On clean questions, the round-trip rejects 7/48 (production) and 5/39 (ITBench), including three genuinely ambiguous anchors (\textscAG\textsc{AG}); the remaining rejections reflect parser errors. On actual model rewrites, it detects dropped anchors, reversed relations, and altered paths. Notably, the LLM judge panel accepts 20/26 rewrites rejected by the round-trip.

2. Answer validation: Deterministic semantic checks. We additionally verify asserted relations, procedural completeness and ordering, and instruction polarity.

CorruptionProductionITBench
Unsupported relation2/32/2
Reordered steps8/88/8
Negated procedural step8/86/6
Entity substitution15/159/10
Clean answers falsely rejected0/150/16

In contrast, the LLM judge panel falsely rejects 33% and 69% of clean answers. We therefore use judges only as secondary signals.

Remaining limitation: Negation in free-form causal explanations is not covered by the deterministic polarity check and still relies on model judges.

We also corrected our earlier claim: unsupported procedural steps are rejected by explicit step-level checks, not grounding precision alone.

W2/Q2: Corrected validity and mutation evaluation

As detailed in our general response, we corrected the reported 0.83 validity result, clarified its definition, and updated the measurements.

The affected references were included in the original baseline evaluation. We therefore rescored systems using their stored outputs against the corrected references and updated the reported results.

We also agree that drift measures sensitivity rather than correctness. We replaced the drift experiment with differential and metamorphic tests, observing exactly the predicted changes in 65/65 and 31/31 tests, respectively.

Q3: Correctness of incremental regeneration

We introduced a dependency footprint for each reference, comprising:

  1. Schema footprint: Enumerated node labels and traversed relation types.
  2. Traversal region: Nodes within the oracle's bounded traversal radius.
  3. Grounding sensitivity: Textual changes that could alter question grounding.

A reference is recompiled whenever a graph change intersects its footprint; otherwise, it is reused.

This addresses the reviewer's counterexample: deleting the only playbook link of an incident outside the previous gap set triggers recompilation through the schema footprint, correctly adding that incident to the result (5/5 metamorphic tests).

Against full recomputation, incremental regeneration produced identical references on 32/32 production and 19/19 ITBench edits. It also matched across two successive real production snapshots involving the removal of 2,173 edges and subsequently 6,179 nodes.

On average, only 44% of references required recompilation, while preserving equivalence with full recomputation in the evaluated cases.

These additions address the reviewer's concerns through stronger semantic validation, corrected baseline evaluation, and experimentally verified incremental consistency. We have also made the remaining limitations explicit in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.