- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The submission introduces an LLM-free framework for constructing question answering (QA) benchmarks over IT Context Knowledge Graphs (CKGs). The main contribution is an executable reference construction pipeline for operational CKG QA that integrates evidence-bounded generation and abstention. Additionally, the authors provide graph-derived questions. The proposed pipeline first establishes informative graph structures and compiles their references before generating questions, effectively decoupling question formulation from ground-truth reference construction. The evaluation measures unsupported relation claims and validates the framework on a production operational CKG with released artifacts.
Critical strengths
None
Critical weaknesses
The main concern is an apparent contradiction regarding the determinism of the re-execution process for families B and C. While the authors claim the process is deterministic, they report an agreement score of only 0.83 under re-execution. In addition, the notion of "agreement" remains ambiguous and requires clarification. Furthermore, the mathematical formalism is excessive without adding clarity. Another major issue is that, despite being defined as "LLM-free," the framework is not entirely so. Overall, the submission presents several significant limitations and drawbacks that cannot be overlooked.
Questions for authors
My main recommendation for future revisions is to ensure the manuscript is thoroughly polished, complete, and clear before resubmission.
Response to the reviewer sic7
Thank you for the review.
Determinism and “agreement”. You identified a genuine contradiction. It came from a verification-harness fault, not from the construction (see the general response). Agreement is now defined exactly: the share of references whose answer set, G* node set and typed G* edge set are identical after recompiling from (q, G_t). Under that definition it is 1.00 on both graphs, across fresh processes with different hash seeds and across complete rebuilds. “LLM-free”. We agree the wording overstated it. The precise claim is that no language model determines the reference. No model involved: set, path and dependency references, and every subgraph G*, are compiled deterministically. Our model-free mode builds the full benchmark this way. Model involved: models only phrase questions and A/D answer text. Each generated string must pass deterministic checks against G*: grounding precision, unsupported relations, step order, completeness and polarity. Questions must also pass an executable round-trip. In the full pipeline, 18 of 19 generated answers passed on the production graph. We will replace “LLM-free” with this formulation throughout. Formalism and polish. We will move the stage operators to the appendix, keep only the definitions the propositions use, reconcile every reported number with the instantiated benchmark, and fix the presentation issues.