- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ComplexLogicalQA (CLQ), a diagnostic benchmark for evaluating KGQA beyond chain-like reasoning. It covers nine EPFO query structures, including intersection, union, and nested compositions, and evaluates representative KGQA methods on these structures. The results show that current systems struggle particularly with nested queries and complete answer-set recovery for union queries.
Critical strengths
-
Meaningful research problem. Existing KGQA benchmarks mainly focus on chain-like multi-hop reasoning, so their results may not fully reflect model ability on more complex logical queries.
-
Clear motivation and simple idea. The need to balance different logical structures is easy to understand, and the benchmark construction pipeline is clearly presented.
-
Well written and well structured. The paper presents the benchmark construction, controlled analysis, and failure diagnosis in a clear order.
Critical weaknesses
-
Limited evaluation beyond Freebase. Freebase is relatively dated and less representative than Wikidata. The 90-question Wikidata pilot in the appendix is a useful first step, but it only evaluates ToG and is too small to establish the generality of the conclusions. Even a small-scale Wikidata evaluation with several baselines would be helpful.
-
Limited coverage of complex logic. The nine EPFO structures are a useful move beyond chain-like reasoning, but they still cover only a limited part of real-world complex KGQA. Such questions often involve multiple interacting constraints. Constructing questions with richer combinations of constraints would provide a stronger test of model reasoning ability.
-
Several relevant baselines are missing. Recent methods such as LPKG, which also studies structured complex reasoning and introduces the CLQA-Wiki benchmark, as well as DoG and CoG, are not included. This weakens the empirical comparison. Adding these baselines, or explaining why they cannot be evaluated on CLQ, would make the claims more convincing.
If these concerns are addressed, I would consider raising my score.
Questions for authors
See above.