- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This study presents GeoTKGQA, a novel benchmark generation framework for temporal knowledge graph question answering that explicitly integrates geographical reasoning as a third validated reasoning dimension in addition to established structural and temporal reasoning axes. The generator constructs a geographical knowledge layer grounding TKG entities to geographic locations, generates questions from twelve schema families combining structural, temporal, and spatial operators, and certifies that each question requires all three reasoning axes by verifying that removing any axis changes the gold answer. Applied to four standard TKGs, the pipeline produces 67,565 tri-axially entangled questions with multiple evaluation splits. Evaluation of ten baselines shows that even the strongest systems resolve only 41.16% of questions, with 19.94% of questions defeating all tested methods.
Critical strengths
Critical Strengths
1.Groundbreaking benchmark contribution: GeoTKGQA is the first benchmark to systematically evaluate joint structural, temporal, and geographical reasoning on temporal knowledge graphs, filling a major gap in existing evaluation resources for multi-dimensional reasoning.
2.Rigorous question certification: The generate-then-verify pipeline ensures that questions truly require all three reasoning axes, eliminating "shortcut" questions that can be answered without full multi-dimensional reasoning, making the benchmark highly reliable for evaluating system capabilities.
Critical weaknesses
Critical Weaknesses
1.Limited geographical operator diversity: The current implementation supports only six spatial predicates; future versions could expand to more complex geographical reasoning (e.g., route planning, area calculations).
2.Synthetic question limitations: While programmatically generated questions ensure logical validity, they may not fully capture the diversity and ambiguity of real-world user questions.
Questions for authors
Have you conducted any human evaluation to assess how well the programmatically generated questions correspond to real-world user information needs involving geographical and temporal reasoning?