acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Geographical Temporal Knowledge Graph Reasoning

Abstract

Temporal knowledge graphs (TKGs) built from event data carry rich latent geography, yet existing TKG question answering (TKGQA) benchmarks test only structural and temporal reasoning. We present GeoTKGQA, a question-answer generator whose questions provably require geographical reasoning by construction. We ground the locative anchors of a TKG to a shared geographical knowledge layer and design twelve schema families that compose structural, temporal, and spatial operators. We then execute every logical form symbolically and keep a question only if dropping the constraints of any one axis would change its gold answer. Applied to four standard event graphs, the pipeline yields 67,565 tri-axially entangled questions with difficulty bands and four generalization splits, without human annotation or large language models. We evaluate ten baselines from four paradigms and analyze all 160 runs with a generator-backed error attribution protocol. The strongest system wins ten out of sixteen benchmark cells, but resolves only 0.4116 of mean-pooled questions, while 19.94% of the questions defeat all ten methods. GeoTKGQA's code and data are anonymously available at https://anonymous.4open.science/r/GeoTKGQA-6FB8/.

Primary area:
datasets and benchmarks
Keywords:
Temporal knowledge graphs, Question answering, Geographical reasoning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_GiWGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This study presents GeoTKGQA, a novel benchmark generation framework for temporal knowledge graph question answering that explicitly integrates geographical reasoning as a third validated reasoning dimension in addition to established structural and temporal reasoning axes. The generator constructs a geographical knowledge layer grounding TKG entities to geographic locations, generates questions from twelve schema families combining structural, temporal, and spatial operators, and certifies that each question requires all three reasoning axes by verifying that removing any axis changes the gold answer. Applied to four standard TKGs, the pipeline produces 67,565 tri-axially entangled questions with multiple evaluation splits. Evaluation of ten baselines shows that even the strongest systems resolve only 41.16% of questions, with 19.94% of questions defeating all tested methods.

Critical strengths

Critical Strengths

1.Groundbreaking benchmark contribution: GeoTKGQA is the first benchmark to systematically evaluate joint structural, temporal, and geographical reasoning on temporal knowledge graphs, filling a major gap in existing evaluation resources for multi-dimensional reasoning.

2.Rigorous question certification: The generate-then-verify pipeline ensures that questions truly require all three reasoning axes, eliminating "shortcut" questions that can be answered without full multi-dimensional reasoning, making the benchmark highly reliable for evaluating system capabilities.

Critical weaknesses

Critical Weaknesses

1.Limited geographical operator diversity: The current implementation supports only six spatial predicates; future versions could expand to more complex geographical reasoning (e.g., route planning, area calculations).

2.Synthetic question limitations: While programmatically generated questions ensure logical validity, they may not fully capture the diversity and ambiguity of real-world user questions.

Questions for authors

Have you conducted any human evaluation to assess how well the programmatically generated questions correspond to real-world user information needs involving geographical and temporal reasoning?

Reviewer_Mdx4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GeoTKGQA, which adds a country-level geographic knowledge layer to four temporal knowledge graphs. Using 12 executable logical patterns, it generates questions that jointly impose structural, temporal, and geographic conditions, then removes constraints axis by axis to filter out redundant questions. The authors construct a main set of 67,565 questions, define four splits, evaluate 10 baselines, and analyze error types, question decomposition behavior, and unsolved questions.

Critical strengths

First, incorporating geographic constraints into question answering over event-based temporal knowledge graphs has clear research value. The geographic layer, spatial operators, and 12 patterns make the capabilities being tested concrete (Definition 1, p. 3; Table A1, p. 14). Second, removing constraints axis by axis and checking whether the answer changes is more diagnostic than simply adding place names to questions. Executable logical forms, supporting facts, and deterministic execution also make the results easier to verify (pp. 4-5, lines 208-220). Third, four source graphs, five answer types, four splits, supplementary question sets, and 10 baselines make this a fairly comprehensive resource contribution. The authors also proactively disclose strict scoring, single-run results, and implementation changes, allowing the results to be checked. The main-set total of 67,565 in Table 1, Temp-R1's wins in 10/16 cells in Table 2, and its pooled accuracy of approximately 41.16% can all be verified from the reported data.

Critical weaknesses
  1. The central conclusion is not robust to the scoring protocol. In Table 2, RALM without decomposition outperforms RTQA in all 16 cells. Under the unified strict scoring in Table A5, however, RTQA leads in 15 cells. The claim that decomposition is harmful therefore needs to be checked using a unified semantic evaluation before it can be attributed to entanglement across the three axes.
  2. The certification guarantee is interpreted too strongly. Equation (1) can verify that the three types of constraints are non-redundant for the current query's answer, but it cannot establish that a model actually uses this information or that decomposition strategies preserving candidate sets must fail. The universal claim 'whatever the solver' on p. 9 goes beyond the available evidence.
  3. Retrieval and reasoning bottlenecks have not been adequately separated. Appendix C.3 counts an answer string appearing in the context as a hit, but this does not guarantee that the correct relations, times, multi-hop facts, and geographic information are all available. The interpretation that a reasoning bottleneck persists under ideal evidence conditions needs support from a comparison with complete evidence.
Questions for authors

Could you rescore the existing outputs with a unified protocol that normalizes entities, places, dates, and numbers, and explain why the decomposition comparison reverses so widely under the two scoring protocols? Do all baselines have equal access to affiliation, boundary, and centroid information in the geographic knowledge layer? Please add experiments with complete event evidence and GKL information. Page 20 states that each cell uses only one seed, so the current confidence intervals reflect variation from test sampling. The key rankings also need to be checked against training randomness. The phrase 'over the training-free pipelines on the ICEWS graphs' appears twice on p. 7, lines 355-356. Please remove one occurrence.

Reviewer_dZMDOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

One-line Summary.

This paper introduces GeoTKGQA, a benchmarking framework that produces questions that jointly require structural, temporal, and geographical reasoning given a temporal knowledge graph (TKG).

Limitation of Prior Work.

This benchmark is positioned between two independent research lines: one that does not include geography at all, and one that includes geography yet does not require joint reasoning with structural or temporal reasoning.

Core Idea: Tri-axial Entanglement.

The framework GeoTKGQA is well designed, consisting of three parts:

  • Geographical knowledge layer: data pre-processing with normalization of geography
  • Schema families and symbolic execution: question construction and validation of joint reasoning
  • Release control: data post-processing supplemented with diagnostic, unanswerable, and multiple-choice sets

Effect.

Applied to ICEWS14, ICEWS18, ICEWS05-15, and GDELT, the pipeline yields around 67K questions that require joint reasoning over structural, temporal, and geographical factors without any LLM involvement.

Critical strengths

(1) Comprehensive experimental evaluation

Ten baselines from four paradigms are evaluated comprehensively across Table 2 and Figures 2 and 3. Also, the controls with unanswerable questions and the comparison with a closed-book setting are especially insightful.

(2) Well-designed tri-axial certification that guides the failure point

I agree with the authors' insight that reasoning complexity can be high when multiple factors are compounded. It is interesting to see that the geographical factor is still the most challenging factor.

(3) Clear figures and presentation (Figures 1 to 3)

The overview figure and the error-analysis figures are easy to follow (and it seems like the color palette is earth-toned, which is impressive).

Critical weaknesses

(1) Lack of error analysis for the off-target answers of the closed-book control group

In Figure 2(a), more than half of the wrong answers of the closed-book control group (DeepSeek-V4-Flash, Gemma4-31B-IT, and Qwen3.6-35B-A3B) fall into the off-target / type category, yet the paper does not show what these errors look like. It would greatly help if the authors could provide representative examples.

(2) Insufficient explanation on geographical changes

Since geography can change over time (e.g., national borders), can GeoTKGQA reflect this change easily? It seems to be unclear in the paper whether the benchmark can handle such changes.

My current score mainly reflects W1, and I would be glad to raise it if the rebuttal adds the requested error analysis in W1 and clarifies W2.

Questions for authors

Q. Snapshot date of the geographical knowledge layer

Could you provide the snapshot date of the geographical knowledge layer? It can be important if there is an issue in W2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.