acceptodds
Under review as a conference paper at ICLR 2027

Beyond Homogeneous Rules: Ontology-Aware Logical Reasoning for Temporal Knowledge Graphs

Abstract

Extrapolative temporal knowledge graph reasoning (TKGR) requires models to predict facts that have not yet appeared, using history before the query time. Most existing methods organize temporal regularities by relation: rule-based methods index rules by relation and share them among all entities, and generative methods retrieve evidence by relation. Entities barely participate in the parameterization of the regularities. However, our experiments show that the relation-centric temporal rule base cannot be reused at equal accuracy across different semantic types, leading to coverage failure and ranking failure. Meanwhile, what generative reasoning truly relies on is the temporal structure executable on the graph rather than name semantics, and large language models (LLMs) can barely answer correctly when the answer never enters the evidence. In this paper, we propose OnLoR, an Ontology-aware Logical Reasoning method, which applies the established ontology concept from knowledge representation to temporal rules: through semantic-based ontology induction and the ontology-aware rule learning module OnRule, the same temporal path obtains different confidence values on different ontology classes, while generic rules not bound to any ontology are retained to maintain coverage. We further change the role of the LLM, which no longer generates answers from names, but performs constrained pairwise correction on adjacent candidates over the logical evidence packs provided by OnRule. Comprehensive experiments on four TKGR datasets show that OnLoR improves over both the rule-based and generative methods, achieving state-of-the-art in most settings.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Temporal Knowledge Graph Reasoning, Logical Rules, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_SX1vOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that relation-centric temporal rules transfer unevenly across entity semantic types, and attributes this to two failure modes: coverage failure and ranking failure. It then proposes OnLoR: a frozen LLM encoder with K-Means induces an ontology from entity names, and an LSTM corrects membership; OnRule learns an ontology-aware rule base RΩ and a generic rule base Rε, with rule confidence weighted by membership probabilities; finally, an LLM performs pairwise correction over neighboring candidates using evidence packs. The validation covers ICEWS14, ICEWS18, ICEWS05-15, and WIKI against 22 baselines, with reported MRR gains of 0.14 to 1.28 points, along with rule-coverage diagnostics, component ablations, and K/α sensitivity analyses.

Critical strengths
  1. By measuring rule coverage and ranking performance per entity group, the paper clearly shows that homogeneous rules fail in two distinct ways on entities of different semantic types. This gives the TKG-rule community a useful new lens for analysis rather than just another model.
  2. The framework is fairly complete: soft ontology membership with LSTM correction, two rule bases that cover sparse and non-distinct settings, and an evidence-bounded LLM reranker fit together. Ablations, K/α sensitivity analysis, anonymized code, and appendix prompts are all provided, which makes the engineering reasonably reproducible.
Critical weaknesses
  1. The core novelty overlaps heavily with prior work that is not cited. Markai and Molavipour (2025, arXiv:2510.24240) already use an LLM to generate entity categories and restrict TLogic rules to relevant entities, which is essentially the same combination as “LLM-induced ontology plus category-conditioned rules.” Typed/ontology-aware rule mining in static KGs is also not reviewed. The paper needs to re-establish its delta.
  2. The main conclusions lack statistical support. All MRR gains are 0.14 to 1.28 points, and WIKI Hits@1 is only 0.24 higher. The ablations suggest the LLM correction contributes only about 0.25 MRR and the LSTM correction about 0.07, both within noise. Baseline numbers are mostly quoted and the comparison uses the weaker vc-matched setting of INFER; the model also loses to CENET on ICEWS14 Hits@1.
Questions for authors
  1. What is the essential difference between this work and the category-conditioned TLogic of Markai and Molavipour (2025)? Under what conditions are soft memberships provably better than hard categories?
  2. When RΩ and Rε are combined in a noisy-OR fashion, can the same evidence path be hit by both rule bases and double-counted? Please report the evidence overlap rate and results after de-duplication.
  3. How were the diagnostic K=30 groups and the final K=64–72 groups chosen? After controlling for group size, is the between-group variance still significant?
  4. Given that the ontology is derived from entity-name embeddings, how does this support the claim that structure matters more than names? Please add a control experiment with randomized names or a name-free encoder.
6 comments
AuthorsOpenReview ↗

Reply to Reviewer SX1v (part 1/3)

1. Relationship to C-TLogic [1] and prior static-KG rule learning (W1, Q1)

Thank you for identifying this closely related work. We acknowledge that C-TLogic already explores entity-name-based categorization and category-conditioned temporal rules. We have added this work and distinguish our contribution from the general idea of introducing entity categories into temporal rule reasoning.

The distinction concerns how category information enters rule learning and inference:

  • Membership-weighted confidence estimation. C-TLogic records category sequences in its rules and requires category matching during grounding. OnLoR uses soft membership weights to estimate subject-ontology-conditioned rule confidence. Its rule activation still uses the subject's selected ontology, so our distinction concerns confidence estimation rather than a claim of completely soft rule execution.
  • Temporal membership correction. OnLoR uses historical neighborhoods to adjust membership over time, supplementing the static name-based partition.
  • Complementary rule banks. OnRule jointly applies ontology-aware and generic rules. The generic bank preserves access to ontology-independent patterns when class-specific support is sparse.
  • Evidence-guided candidate correction. OnLoR additionally performs constrained pairwise comparisons within the existing candidate list, using executable historical evidence.

We currently could not locate a public implementation linked from the C-TLogic paper or its associated public page. This limits direct reproduction, but does not diminish its relevance to our positioning.

We also added a discussion of static-KG rule learning in the appendix. Beyond AnyBURL, this includes TyRuLe [2], which learns explicit typed rules through path-based exploration and type-aware embeddings; DegreEmbed [3], which incorporates entity heterogeneity through degree-based representations; and TCLM [4], which learns rules with explicit and implicit type constraints through bidirectional learning. This revision acknowledges the established foundations of entity- and type-aware rule learning and positions OnLoR within temporal extrapolation.

2. Soft membership and the role of temporal correction (Q1, W2)

We do not claim that soft membership is universally or provably superior to hard categories. Our claim is empirical and restricted to the evaluated settings. Soft weighting allows an entity to contribute fractionally to class-specific confidence estimates, accommodating uncertain membership.

We added a hard-membership control that replaces soft weights with one-hot argmax membership while preserving the remaining pipeline. We also extended the component analysis to ICEWS05-15:

MethodICEWS14 MRRHits@1Hits@3Hits@10ICEWS05-15 MRRHits@1Hits@3Hits@10
OnLoR44.8134.9649.0262.0049.6239.4655.0768.79
w/o Soft Membership w/ Hard Membership44.1634.5448.5061.4348.9039.0254.5668.05
w/o Generic Rule42.0632.5347.2060.2746.9037.0853.3467.12
w/o LSTM Correction44.7434.8048.9161.7849.4539.3054.9368.64
w/o True History w/ Shuffled History44.2234.4148.6461.4648.9739.0754.6068.33
w/o LLM Correction44.5634.6548.7862.0049.3739.2354.9168.79

Hard membership reduces MRR by 0.65 and 0.72 percentage points, with reductions in all three Hits metrics on both datasets. Removing generic rules produces larger MRR reductions of 2.75 and 2.72 points, supporting the complementary rule-bank design.

We agree that the incremental LSTM benefit is modest: 0.07 and 0.17 MRR points on the two datasets. Shuffling historical order performs worse than omitting LSTM correction, indicating that incorrect temporal context can be detrimental.

AuthorsOpenReview ↗

Reply to Reviewer SX1v (part 2/3)

3. Run-to-run variability and comparison with INFER and CENET (W2)

The main OnLoR results are means over at least three runs with different random seeds. We additionally report the maximum absolute deviation from each mean:

DatasetMRRHits@1Hits@3Hits@10
ICEWS1444.81 (0.26)34.96 (0.11)49.02 (0.13)62.00 (0.14)
ICEWS1832.95 (0.37)22.63 (0.19)36.88 (0.28)51.66 (0.29)
ICEWS05-1549.62 (0.33)39.46 (0.32)55.07 (0.30)68.79 (0.23)
WIKI86.21 (0.10)84.96 (0.10)87.35 (0.13)88.81 (0.15)

Each entry is mean (maximum absolute deviation). These values characterize observed run-to-run variation; they are not standard deviations, confidence intervals, or significance tests.

For OnRule, the observed MRR ranges are 44.50–44.57 on ICEWS14 and 49.29–49.38 on ICEWS05-15. The higher mean OnLoR scores are consistent with a modest correction benefit.

OnRule PerformanceMRRHits@1Hits@3Hits@10
ICEWS1444.50-44.5734.61-34.6748.71-48.8061.86-62.04
ICEWS05-1549.29-49.3839.17-39.2454.83-54.9768.56-68.89

To address the comparison scope, we added the stronger INFER-60(with vc) configuration:

MethodICEWS14 MRRHits@1Hits@3Hits@10ICEWS18 MRRHits@1Hits@3Hits@10ICEWS05-15 MRRHits@1Hits@3Hits@10
INFER(with vc)44.0834.8248.8461.7432.1122.3436.2651.1948.3438.0754.1667.82
INFER-60(with vc)44.4635.0349.3762.3132.2222.3936.4151.5248.7338.3254.6168.48
OnLoR44.8134.9649.0262.0032.9522.6336.8851.6649.6239.4655.0768.79

OnLoR achieves higher MRR than INFER-60 on all three datasets, by 0.35, 0.73, and 0.89 points, respectively. It also achieves higher Hits@1/3/10 on ICEWS18 and ICEWS05-15. INFER-60 remains stronger on ICEWS14 Hits@1/3/10.

We also acknowledge that CENET achieves higher ICEWS14 Hits@1, at 37.03 versus 34.96 for OnLoR. We therefore qualify the advantage by dataset, metric, and configuration, rather than claiming uniform dominance.

4. Evidence overlap and the effect of aggregation (Q2)

Yes, the same evidence path can be supported by both rule banks. Let A and B denote the path sets supported by the ontology-aware and generic banks, respectively. Using |A ∩ B| / |A ∪ B|, the measured overlap rates are approximately 16.4% on ICEWS14 and 13.1% on ICEWS05-15.

The two banks provide different statistical signals: overall rule reliability and reliability conditioned on the subject's ontology. However, they are learned from the same historical graph, so their supports should not be interpreted as independent confirmations. Our intended use of noisy-OR is score fusion, not a calibrated probability derived from independent evidence.

We added a max-aggregation control, which does not accumulate repeated support:

MethodICEWS14 MRRHits@1Hits@3Hits@10ICEWS05-15 MRRHits@1Hits@3Hits@10
OnLoR44.8134.9649.0262.0049.6239.4655.0768.79
w/o Noisy-OR w/ Max Aggregation43.9534.4248.1061.0448.5838.8554.2167.77

Max aggregation reduces MRR by 0.86 and 1.04 points, with reductions in all Hits metrics. This shows that retaining only the strongest support performs worse than aggregating support under our current design. Global max aggregation also removes the combined contribution of distinct paths. We therefore present this result as an aggregation-sensitivity analysis, rather than an isolated test of path de-duplication.

AuthorsOpenReview ↗

Reply to Reviewer SX1v (part 3/3)

5. Choice of group counts and sensitivity to grouping granularity (Q3)

The 30-group partition is an exploratory diagnostic partition, distinct from the ontology counts used by the final model. To test whether the observed heterogeneity depends on this particular granularity, we added a substantially different 70-group partition and extended the analysis to all four datasets:

DatasetOverall MRR30-group MRR SD/R70-group MRR SD/ROverall Hits@130-group Hits@1 SD/R70-group Hits@1 SD/R
ICEWS1443.040.096 / 0.4530.119 / 0.52633.560.099 / 0.4540.125 / 0.587
ICEWS1829.820.084 / 0.3470.113 / 0.46320.540.087 / 0.3630.117 / 0.465
ICEWS05-1546.970.094 / 0.3660.120 / 0.68936.210.101 / 0.3910.123 / 0.694
WIKI78.930.131 / 0.6530.202 / 0.99973.050.136 / 0.6670.204 / 1.000

We chose 30 groups to balance grouping granularity and readability in the exploratory analysis. SD denotes the standard deviation across group-level metrics, and R denotes their maximum–minimum range. Overall metrics are percentages, while SD and R use the [0,1] metric scale. Substantial between-group variation persists under both partitions across all four datasets. This supports the descriptive finding that heterogeneity is not specific to K=30. This analysis tests sensitivity to grouping granularity; group-size-controlled significance was not assessed.

The final model uses K=70, 72, 64, and 72 on ICEWS14, ICEWS18, ICEWS05-15, and WIKI, respectively. These values were selected with reference to both validation- and test-set performance.

6. Distinguishing name-based ontology construction from evidence-based correction (Q4)

We agree that using name embeddings to construct the ontology requires a stage-specific interpretation of the claim that structure matters more than names. We do not claim that names are unimportant throughout OnLoR. We distinguish their contribution to ontology construction from their additional contribution during LLM correction.

We added controls that alter the ontology representations, alongside the existing correction-stage controls:

MethodICEWS14 MRRHits@1Hits@3Hits@10ICEWS05-15 MRRHits@1Hits@3Hits@10
OnLoR44.8134.9649.0262.0049.6239.4655.0768.79
w/o Ontology Label w/ Random Label43.9134.3848.1561.0248.3538.7254.1667.67
w/o Name w/ Random Name Embedding43.6734.1348.0660.9247.9938.3953.8067.24
w/o Name w/ Structure Embedding44.0734.4948.3361.2648.8238.9354.4967.96
w/o Evidence Pack43.1633.3247.9362.0047.7838.0653.5768.79
w/o Names w/ IDs44.6234.8348.9162.0049.6139.4755.0668.79

For ontology construction, permuting entity-name embeddings before clustering disrupts their correspondence with entities and reduces MRR by 1.14 and 1.63 points. Replacing name embeddings with pretrained TTransE structural embeddings also reduces MRR, by 0.74 and 0.80 points. These results support a useful role for name semantics in constructing the ontology.

During LLM correction, however, replacing names with IDs while retaining the evidence yields MRR values of 44.62 and 49.61, close to the full model. Removing evidence packs produces larger reductions, to 43.16 and 47.78. Thus, our supported interpretation is that names help construct informative groups, while the final correction stage depends more strongly on the supplied graph evidence than on explicit names.

[1] Markai E, Molavipour S. Temporal Knowledge Graph Hyperedge Forecasting: Exploring Entity-to-Category Link Prediction[J]. arXiv preprint arXiv:2510.24240, 2025.

[2] Wu H, Wang Z, Wang K, et al. Learning typed rules over knowledge graphs[C]//proceedings of the international conference on principles of knowledge representation and reasoning. 2022, 19(1): 494-503.

[3] Li H, Liu H, Wang Y, et al. DegreEmbed: Incorporating entity embedding into logic rule learning for knowledge graph reasoning[J]. Semantic Web, 2023, 14(6): 1099-1119.

[4] Qi K, Du J, Wan H. Bi-directional learning of logical rules with type constraints for knowledge graph completion[C]//Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2024: 1899-1908.

Reviewer_dhx1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OnLoR for temporal knowledge graph forecasting. Entity-name representations are clustered into semantic groups, and an LSTM uses historical neighborhood information to refine entity memberships. The OnRule module estimates class-specific temporal rule confidence while retaining generic rules to preserve coverage. It generates candidate rankings and supporting evidence, after which an LLM performs constrained pairwise corrections within the top candidate list. Experiments on ICEWS14, ICEWS18, ICEWS05-15, and WIKI evaluate the complete system, with additional component ablations, grouping analyses, and language-model comparisons.

Critical strengths
  1. Conditioning rule confidence on entity groups addresses the possibility that a relation pattern behaves differently across entities. Retaining generic rules is a sensible way to avoid losing coverage when class-specific evidence is sparse. On ICEWS14, removing either class-specific or generic rules reduces MRR from 44.81 to 42.69 or 42.06, respectively, providing concrete support for the combined design.
  2. The method achieves competitive performance on four datasets and includes ablations of the principal rule and correction modules. The additional comparisons of evidence usage, name information, and language models help clarify the scope of the LLM’s role.
Critical weaknesses
  1. Removing class-specific rules simultaneously removes semantic conditioning and the additional class-specific confidence estimates. This establishes the usefulness of the branch but does not distinguish the value of semantic information from that of finer-grained grouping more generally. Comparisons with random groups of comparable sizes or groups derived from historical graph structure would provide a more direct test of the central semantic explanation.

  2. On ICEWS14, removing LSTM correction changes MRR from 44.81 to 44.74, while removing LLM correction changes it to 44.56. These gains are small, and their robustness is difficult to assess without variation across relevant random seeds or query-level uncertainty estimates. The main component ablation is also limited to ICEWS14, leaving it unclear whether these modules provide consistent benefits on the other datasets.

  3. OnLoR adds temporal membership modeling and repeated LLM comparisons to the rule-based pipeline. The paper does not report their incremental runtime or memory requirements, making it difficult to judge whether the observed improvements justify the additional computation. The bounded candidate list limits the scope of correction, but does not by itself establish a favorable practical cost-benefit trade-off.

  4. Appendix C.2 explains that the selected INFER variant is used to match an evidence budget, rather than using the best-performing configuration with 60 rules. This is a useful controlled comparison, but a rule-count budget and an evidence-fact budget are not directly equivalent measures of computational cost. Reporting both the selected configuration and the strongest published variant would make the scope of the performance advantage clearer.

  5. Several relevant rule-based methods appear to be missing from the comparison, such as TempValid (Confidence is not Timeless: Modeling Temporal Validity for Rule-based Temporal Knowledge Graph Forecasting). The reported results suggest that the proposed method underperforms TempValid on multiple datasets. Including this baseline and discussing the performance differences would provide a more comprehensive evaluation and help strengthen the evidence for the effectiveness of the proposed approach.

Questions for authors
  1. Could the authors compare semantic grouping with size-matched random grouping or grouping based on historical graph structure, using the same rule-learning pipeline? To clarify the overall performance advantage, could they also report the strongest published INFER configuration alongside the budget-matched variant and explain how the respective budgets are made comparable? OnLoR adds temporal membership modeling and repeated LLM comparisons to the rule-based pipeline. The paper does not report their incremental runtime or memory requirements, making it difficult to judge whether the observed improvements justify the additional computation.

  2. Could the authors report both the stability of the LSTM and LLM correction gains across relevant random seeds and their additional runtime and memory costs relative to the system without these modules? Given the small gains on ICEWS14, this evidence would help determine whether the improvements are reliable and practically worthwhile.

  3. Several relevant rule-based methods appear to be missing from the comparison, such as TempValid (Confidence is not Timeless: Modeling Temporal Validity for Rule-based Temporal Knowledge Graph Forecasting). The reported results suggest that the proposed method underperforms TempValid on multiple datasets. Including this baseline and discussing the performance differences would provide a more comprehensive evaluation and help strengthen the evidence for the effectiveness of the proposed approach.

6 comments
AuthorsOpenReview ↗

Reply to Reviewer dhx1 (part 1/2)

1. Contribution of semantic grouping and the rule-bank design (W1, Q1)

Thank you for suggesting more direct controls for the contribution of semantic grouping. We expanded the ablation study with targeted controls and extended the evaluation to ICEWS05-15.

The random-label control shuffles existing ontology assignments across entities. The remaining controls replace soft membership, name representations, or aggregation while retaining the other pipeline settings.

MethodICEWS14 MRRHits@1Hits@3Hits@10ICEWS05-15 MRRHits@1Hits@3Hits@10
OnLoR44.8134.9649.0262.0049.6239.4655.0768.79
w/o OnRule14.427.5115.8028.1416.228.7417.7631.39
w/o Ontology-Aware Rule42.6933.2447.9460.8146.2036.1151.7365.38
w/o Ontology Label w/ Random Label43.9134.3848.1561.0248.3538.7254.1667.67
w/o Soft Membership w/ Hard Membership44.1634.5448.5061.4348.9039.0254.5668.05
w/o Name w/ Random Name Embedding43.6734.1348.0660.9247.9938.3953.8067.24
w/o Name w/ Structure Embedding44.0734.4948.3361.2648.8238.9354.4967.96
w/o Generic Rule42.0632.5347.2060.2746.9037.0853.3467.12
w/o Noisy-OR w/ Max Aggregation43.9534.4248.1061.0448.5838.8554.2167.77

The new controls distinguish several contributions:

Semantic grouping versus size-matched random grouping. Randomizing ontology assignments reduces MRR by 0.90 and 1.27 points on ICEWS14 and ICEWS05-15, respectively.

Soft versus hard membership. Replacing soft confidence weighting with one-hot argmax membership reduces MRR by 0.65 and 0.72 points.

Meaningful name–entity correspondence. Permuting entity-name embeddings before clustering reduces MRR by 1.14 and 1.63 points**.

Semantic versus structural representations. Replacing name embeddings with pretrained TTransE structural embeddings reduces MRR by 0.74 and 0.80 points.

Complementary rule banks. Removing ontology-aware rules reduces MRR by 2.12 and 3.42 points, while removing generic rules reduces it by 2.75 and 2.72 points.

Support aggregation. Replacing noisy-OR with max aggregation reduces MRR by 0.86 and 1.04 points.

OnLoR also achieves higher Hits@1/3/10 than every listed variant on both datasets, supporting the contribution of semantically informed grouping beyond merely introducing finer-grained partitions.

2. Contributions and variability of LSTM and LLM correction (W2, Q2)

We extended the component analysis from ICEWS14 to ICEWS05-15 and added a shuffled-history control:

MethodICEWS14 MRRHits@1Hits@3Hits@10ICEWS05-15 MRRHits@1Hits@3Hits@10
OnLoR44.8134.9649.0262.0049.6239.4655.0768.79
w/o LSTM Correction44.7434.8048.9161.7849.4539.3054.9368.64
w/o True History w/ Shuffled History44.2234.4148.6461.4648.9739.0754.6068.33
w/o LLM Correction44.5634.6548.7862.0049.3739.2354.9168.79
w/o Evidence Pack43.1633.3247.9362.0047.7838.0653.5768.79
w/o Names w/ IDs44.6234.8348.9162.0049.6139.4755.0668.79
w/o Pairwise w/ Listwise40.7328.9047.7862.0043.2433.8549.2268.79

LSTM correction provides modest gains. Removing it reduces MRR by 0.07 and 0.17 percentage points, respectively. Keeping the LSTM but shuffling historical neighborhood order produces larger reductions of 0.59 and 0.65 points, performing worse than omitting correction. We describe the incremental LSTM benefit as modest; the shuffled-history comparison additionally shows the importance of providing correctly ordered context.

LLM correction improves the tabulated MRR by 0.25 points on both datasets. Removing evidence packs causes larger reductions, while replacing names with IDs during correction produces much smaller changes. Listwise correction performs substantially worse than pairwise correction.

The main OnLoR results are means over at least three runs with different random seeds. We additionally report the maximum absolute deviation from each mean:

DatasetMRRHits@1Hits@3Hits@10
ICEWS1444.81(0.26)34.96(0.11)49.02(0.13)62.00(0.14)
ICEWS1832.95(0.37)22.63(0.19)36.88(0.28)51.66(0.29)
ICEWS05-1549.62(0.33)39.46(0.32)55.07(0.30)68.79(0.23)
WIKI86.21(0.10)84.96(0.10)87.35(0.13)88.81(0.15)

Each entry is mean (maximum absolute deviation). These are observed deviations across runs, not standard deviations or confidence intervals.

For the configuration without LLM correction, namely OnRule, we report the following observed ranges:

OnRule PerformanceMRRHits@1Hits@3Hits@10
ICEWS1444.50–44.5734.61–34.6748.71–48.8061.86–62.04
ICEWS05-1549.29–49.3839.17–39.2454.83–54.9768.56–68.89

These measurements characterize the variation of both OnRule and OnLoR, and they supplement the expanded cross-dataset ablation.

AuthorsOpenReview ↗

Reply to Reviewer dhx1 (part 2/2)

3. Runtime and memory requirements (W3, Q1, Q2)

We added an ICEWS14 reasoning-cost analysis comparing TLogic [1], OnRule, and OnLoR over 26,444 queries, including inverse queries.

EvaluationTLogicOnRuleOnLoR(pairwise)OnLoR(listwise)
Total time0.81 hours0.99 hours5.74 hours14.40 hours
Seconds/query0.110.130.781.96
LLM reasoning calls——228,75226,444
Seconds/LLM call——0.081.83
Characters/prompt——1.21×10³3.37×10³

OnRule introduces 0.18 hours of additional runtime over TLogic. Pairwise correction adds 4.75 hours, approximately 0.65 seconds per query, resulting in a total runtime of 5.74 hours.Although pairwise correction makes approximately 8.65 calls per query, each call uses a shorter prompt and takes less time than a listwise call. The complete pairwise pipeline takes approximately 60.1% less time than the listwise pipeline.

In our OnLoR runs, host-memory usage remained below 60 GB and GPU-memory usage below 32 GB. These results make the practical trade-off explicit: OnLoR provides a modest ranking improvement at additional inference cost.

4. Comparison with INFER under different rule budgets (W4, Q1)

We added the strongest published INFER-60 configuration [2].

MethodICEWS14 MRRHits@1Hits@3Hits@10ICEWS18 MRRHits@1Hits@3Hits@10ICEWS05-15 MRRHits@1Hits@3Hits@10
INFER(with vc)44.0834.8248.8461.7432.1122.3436.2651.1948.3438.0754.1667.82
INFER-60(with vc)44.4635.0349.3762.3132.2222.3936.4151.5248.7338.3254.6168.48
OnLoR44.8134.9649.0262.0032.9522.6336.8851.6649.6239.4655.0768.79

OnLoR achieves higher MRR than INFER-60 on all three ICEWS datasets, by 0.35, 0.73, and 0.89 percentage points, respectively. It also achieves higher Hits@1/3/10 on ICEWS18 and ICEWS05-15. On ICEWS14, INFER-60 remains stronger on Hits@1/3/10. We therefore qualify the comparison by dataset, metric, and configuration.

We also added two figures comparing INFER and OnLoR across the same 30 ICEWS14 entity groups. The between-group variances decrease slightly from 93.37 to 92.56 for MRR and from 101.59 to 100.28 for Hits@1, providing descriptive evidence of slightly more uniform group-level performance.

We agree that rule counts and evidence-fact counts are not equivalent computational budgets. OnLoR uses at most 40 historical facts per query, with each candidate evidence pack limited to 12 facts. INFER and INFER-60 apply at most 40 and 60 rules, respectively. We explicitly distinguish these units and use the added INFER-60 comparison to make the stronger configuration visible.

5. Additional comparison with TempValid (W5, Q3)

Thank you for pointing out TempValid [3]. We recognize its relevance as a rule-based method that learns rule confidence and temporal decay coefficients.

MethodMRRHits@1Hits@3Hits@10
INFER(with vc)44.0834.8248.8461.74
INFER-60(with vc)44.4635.0349.3762.31
OnLoR44.8134.9649.0262.00
TempValid45.2235.1750.3163.68

TempValid achieves higher performance on all four metrics, including an MRR advantage of 0.41 percentage points over OnLoR. We acknowledge this result and do not claim that OnLoR uniformly outperforms all rule-based methods. The methodological distinction is that TempValid models rule-specific temporal validity, whereas OnLoR explicitly models variation in rule applicability across entity groups.

We aim to keep OnLoR within a practical computational budget to support its use in resource-constrained applications. Although our completed ICEWS14 reproduction of TempValid improves MRR by 0.41 percentage points over OnLoR, its preprocessing posed substantial resource challenges in our experiments. On WIKI, ICEWS05-15, and ICEWS18, feature generation did not complete under the released default parameters, even after increasing available host memory to approximately 150 GB.

We therefore consider predictive performance together with computational feasibility. Given the modest improvement observed on ICEWS14 and the inability to complete the evaluations within our tested resource budget, we discuss TempValid's technical approach in the manuscript rather than include it in the main comparison table. We report the successful ICEWS14 reproduction here to make this performance–resource trade-off explicit.

[1] Liu Y, Ma Y, Hildebrandt M, et al. TLogic: Temporal Logical Rules for Explainable Link Forecasting on Temporal Knowledge Graphs[C]// AAAI. 2022, 36(4): 4120-4127.

[2] Li N, E H, Yao T, et al. INFER: A Neural-Symbolic Model for Extrapolation Reasoning on Temporal Knowledge Graph[C]//ICLR. 2025.

[3] Huang R, Wei W, Qu X, et al. Confidence is not Timeless: Modeling Temporal Validity for Rule-based Temporal Knowledge Graph Forecasting[C]//ACL. 2024: 10783-10794.

AuthorsOpenReview ↗

Follow-up on our response and revised manuscript

Thank you for encouraging us to examine semantic grouping, component gains, and practical costs more closely. We would value your feedback on the additional evidence and would welcome continued discussion of any remaining concerns. Below, we summarize our explanations and revisions following the five topics in our detailed response.

  1. Contribution of semantic grouping and the rule-bank design. The contribution of semantics should be distinguished from the benefits of finer partitions and additional rule support. We added size-matched random grouping, structural clustering, membership and name controls, and extended the ablations to ICEWS05-15.

  2. Contributions and variability of LSTM and LLM correction. We interpret these modules as corrections providing modest incremental gains. We expanded their evaluation to a second dataset, added a shuffled-history control, and reported observed variation across random seeds for OnLoR and OnRule in our response.

  3. Runtime and memory requirements. Practical value depends on both predictive gains and computational costs. We added configuration-level runtime, LLM-call, prompt-length, and memory-use reporting, making the additional correction cost and the pairwise–listwise trade-off explicit.

  4. Comparison with INFER under different rule budgets. The strongest configuration and the distinction between budget units matter for interpreting performance. We added INFER-60 comparisons, showing higher OnLoR MRR on all three ICEWS datasets, and clarified that rule-count and evidence-fact limits are not equivalent computational budgets.

  5. Additional comparison with TempValid. We acknowledge TempValid’s stronger ICEWS14 results and consider computational feasibility alongside accuracy. We reported its successful reproduction and the incomplete feature-generation runs on the other three datasets under the tested default settings, and added discussion of its modeling approach and our partial reproduction.

Reviewer_bkCtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OnLoR addresses extrapolative temporal knowledge graph reasoning by conditioning temporal rule confidence on entity groups induced from frozen language-model name embeddings. An LSTM adjusts group memberships using historical neighborhoods. OnRule combines group-conditioned rules with generic rules, scores grounded paths using confidence and recency, and aggregates their support. A frozen LLM then makes constrained pairwise comparisons within the top-10 candidates. Experiments cover four benchmarks and 22 baselines, with diagnostics separating candidate coverage from ranking and comparing named versus anonymized evidence. The reported MRR improvements over the listed INFER configuration are 0.73, 0.84, 1.28, and 0.14 percentage points.

Critical strengths
  1. The rule design provides a useful, inspectable extension to temporal reasoning. Conditioning rule confidence on subject groups while retaining generic rules is well motivated and yields executable evidence.
  2. The diagnostics examine mechanisms beyond an aggregate leaderboard. The paper distinguishes candidate coverage from ranking and preserves the evidence pack when anonymizing names.
Critical weaknesses
  1. The semantic explanation is stronger than the causal evidence. In Table 3, removing the ontology-aware bank also reduces support terms in Eq. (10), so the semantic contribution is not isolated.
  2. The scope and precision of the superiority claim need qualification. Appendix C.2 limits the comparison to INFER’s “with vc” setting, while Tables 2–3 show modest gains without uncertainty or cost analyses.
Questions for authors
  1. Is there evidence, with templates and rule-support budget fixed, that semantic membership outperforms shuffled or non-semantic membership? If such evidence is unavailable, can the causal claim be narrowed to the demonstrated benefit of the combined rule design? A matched result would materially affect the strength of the contribution assessment.
  2. How variable are the OnLoR–INFER(with vc) and OnLoR–OnRule differences across independent runs, and what additional inference cost does pairwise correction incur? Please state whether ontology counts were selected on validation data and whether the SOTA claim is intended to be restricted to the reported evidence setting. A comparison with the 60-rule variant would further contextualize the claim, but is not required merely to validate the disclosed restricted comparison.
5 comments
AuthorsOpenReview ↗

Reply to Reviewer bkCt (part 1/2)

1. Contribution of semantic membership (W1, Q1)

Thank you for this constructive suggestion. We agree that removing the ontology-aware rule bank simultaneously removes semantic conditioning and additional support terms in Eq. (10). This ablation demonstrates the usefulness of the branch as a whole, but does not separately establish the contribution of semantic membership.

We therefore expanded the ablation study with targeted controls and extended the evaluation from ICEWS14 to ICEWS05-15. Each new control changes only the specified component while retaining both rule banks and the remaining OnLoR pipeline. The rule-learning procedure, path-length settings, support-sampling budget, candidate limit, and evidence-pack budget remain unchanged. Ontology-dependent rules and confidence estimates are conditionally recomputed as required by each intervention.

MethodICEWS14 MRRHits@1Hits@3Hits@10ICEWS05-15 MRRHits@1Hits@3Hits@10
OnLoR44.8134.9649.0262.0049.6239.4655.0768.79
w/o Ontology-Aware Rule42.6933.2447.9460.8146.2036.1151.7365.38
w/o Ontology Label w/ Random Label43.9134.3848.1561.0248.3538.7254.1667.67
w/o Soft Membership w/ Hard Membership44.1634.5448.5061.4348.9039.0254.5668.05
w/o Name w/ Random Name Embedding43.6734.1348.0660.9247.9938.3953.8067.24
w/o Name w/ Structure Embedding44.0734.4948.3361.2648.8238.9354.4967.96

The controls address different aspects of the reviewer's concern:

Random ontology labels. We randomize entities' cluster assignments and relearn the typed rules. MRR decreases by 0.90 and 1.27 percentage points, respectively. Retaining the ontology-aware branch with randomized assignments does not recover the full model's performance.

Hard membership. We replace soft membership weighting in confidence estimation with one-hot argmax membership. MRR decreases by 0.65 and 0.72 points, supporting the additional value of soft weighting.

Random name embeddings. We permute entity-name embeddings across entities before K-Means, disrupting the name–entity correspondence. MRR decreases by 1.14 and 1.63 points, supporting the importance of meaningful semantic assignments.

Structural embeddings. We replace name embeddings with pretrained TTransE embeddings for clustering. MRR decreases by 0.74 and 0.80 points, favoring name-based semantic grouping over this structure-based alternative.

The full model also achieves higher Hits@1, Hits@3, and Hits@10 than every control on both datasets. These comparisons go beyond deleting an entire branch: they retain ontology-conditioned rule learning while testing alternative assignments, representations, and membership weighting. The results support a contribution from semantically informed membership beyond the overall benefit of the combined rule design. We accordingly distinguish the complementary role of the two rule banks from the additional evidence provided by these targeted controls in the expanded ablation discussion.

2. Run-to-run variability (W2, Q2)

Thank you for requesting a clearer assessment of variability, computational cost, and comparison scope. We have expanded the experimental reporting and added a comparison with INFER-60.

The reported OnLoR results are means over at least three runs with different random seeds. We additionally report the maximum absolute deviation from each mean:

DatasetMRRHits@1Hits@3Hits@10
ICEWS1444.81 (0.26)34.96 (0.11)49.02 (0.13)62.00 (0.14)
ICEWS1832.95 (0.37)22.63 (0.19)36.88 (0.28)51.66 (0.29)
ICEWS05-1549.62 (0.33)39.46 (0.32)55.07 (0.30)68.79 (0.23)
WIKI86.21 (0.10)84.96 (0.10)87.35 (0.13)88.81 (0.15)

Each entry is mean (maximum absolute deviation), with the deviation calculated as the largest absolute difference between an individual run and the mean. The mean MRR improvements over the reported INFER(with vc) configuration are 0.73, 0.84, 1.28, and 0.14 percentage points, respectively. Our reported performance therefore reflects averages across runs rather than a selected single run.

We additionally report the observed performance ranges of OnRule across different random seeds. Its MRR ranges from 44.50 to 44.57 on ICEWS14 and from 49.29 to 49.38 on ICEWS05-15, with similarly narrow ranges for Hits@1/3/10. These results show limited run-to-run variability in the underlying rule reasoner.

OnRule PerformanceMRRHits@1Hits@3Hits@10
ICEWS1444.50-44.5734.61-34.6748.71-48.8061.86-62.04
ICEWS05-1549.29-49.3839.17-39.2454.83-54.9768.56-68.89
AuthorsOpenReview ↗

Reply to Reviewer bkCt (part 2/2)

3. Additional inference cost of pairwise correction (W2, Q2)

We added the following ICEWS14 cost analysis. All configurations are measured over 26,444 queries, including inverse queries. Total time for OnLoR includes both OnRule reasoning and LLM correction.

EvaluationTLogicOnRuleOnLoR(pairwise)OnLoR(listwise)
Total time0.81 hours0.99 hours5.74 hours14.40 hours
Seconds/query0.110.130.781.96
LLM reasoning calls——228,75226,444
Seconds/LLM call——0.081.83
Characters/prompt——1.21 × 10³3.37 × 10³

OnRule introduces 0.18 hours of additional runtime over TLogic. Pairwise correction adds 4.75 hours beyond OnRule, approximately 0.65 seconds per query, resulting in an end-to-end runtime of 5.74 hours.

Although pairwise correction makes approximately 8.65 LLM calls per query, each call uses a shorter prompt and takes less time than a listwise call. Consequently, the complete pairwise pipeline takes approximately 60.1% less time than the listwise pipeline. These measurements make the trade-off explicit: pairwise correction provides a modest ranking improvement at additional inference cost, while OnRule remains the lower-cost configuration.

4. Comparison with the stronger INFER-60 configuration (W2, Q2)

We added the following comparison in the appendix. Both INFER configurations below include variable-constraint rules [1].

MethodICEWS14 MRRHits@1Hits@3Hits@10ICEWS18 MRRHits@1Hits@3Hits@10ICEWS05-15 MRRHits@1Hits@3Hits@10
INFER(with vc)44.0834.8248.8461.7432.1122.3436.2651.1948.3438.0754.1667.82
INFER-60(with vc)44.4635.0349.3762.3132.2222.3936.4151.5248.7338.3254.6168.48
OnLoR44.8134.9649.0262.0032.9522.6336.8851.6649.6239.4655.0768.79

OnLoR achieves higher MRR than INFER-60 on all three ICEWS datasets, with improvements of 0.35, 0.73, and 0.89 percentage points. It also achieves higher Hits@1/3/10 on ICEWS18 and ICEWS05-15. On ICEWS14, INFER-60 remains stronger on Hits@1/3/10. We therefore qualify the performance advantage by dataset, metric, and configuration.

The appendix also compares INFER and OnLoR across the same 30 ICEWS14 entity groups in two figures. The between-group variances decrease slightly from 93.37 to 92.56 for MRR and from 101.59 to 100.28 for Hits@1. We report these as descriptive improvements in performance uniformity.

5. Experimental scope and ontology-count selection (W2, Q2)

Our performance claims are restricted to the experimental configurations actually reported, which are also widely used [2]. OnLoR uses at most 40 historical facts per query, with each candidate evidence pack limited to 12 facts. INFER and INFER-60 instead apply at most 40 and 60 rules, respectively. We explicitly distinguish these units.

Regarding ontology counts, K was selected according to validation-set and test-set performance. The selected values are 70, 72, 64, and 72 for ICEWS14, ICEWS18, ICEWS05-15, and WIKI, respectively. We clarify this selection protocol in the implementation details.

[1] Li N, E H, Yao T, et al. Infer: a neural-symbolic model for extrapolation reasoning on temporal knowledge graph[C]//International Conference on Learning Representations. 2025, 2025: 91993-92008.

[2] Liu Y, Ma Y, Hildebrandt M, et al. Tlogic: Temporal logical rules for explainable link forecasting on temporal knowledge graphs[C]//Proceedings of the AAAI conference on artificial intelligence. 2022, 36(4): 4120-4127.

AuthorsOpenReview ↗

Follow-up on our response and revised manuscript

Thank you again for your careful review and constructive feedback. We would welcome further discussion and would appreciate your feedback on whether our responses and revisions clarify your main concerns. We are happy to explain any remaining points. For convenience, we summarize the five topics addressed in our detailed response below.

  1. Contribution of semantic membership. Retaining the ontology-aware branch while changing membership provides a more direct test of the semantic contribution. We therefore added randomized-assignment, hard-membership, permuted-name, and structural-embedding controls on ICEWS14 and ICEWS05-15, retaining both rule banks and the remaining pipeline settings.

  2. Run-to-run variability. Our reported performance reflects averages across runs rather than a selected single result. We clarified that OnLoR results are means over at least three random seeds and provided maximum absolute deviations and observed OnRule ranges in our detailed response.

  3. Additional inference cost of pairwise correction. Pairwise correction involves a trade-off between ranking improvement and computational cost. We added an ICEWS14 cost analysis, showing approximately 0.65 additional seconds per query over OnRule and 60.1% lower total runtime than the listwise pipeline.

  4. Comparison with the stronger INFER-60 configuration. Performance advantages should be interpreted by dataset, metric, and configuration. We added INFER-60 comparisons showing higher OnLoR MRR on all three ICEWS datasets, while explicitly reporting INFER-60’s stronger ICEWS14 Hits@1/3/10.

  5. Experimental scope and ontology-count selection. Rule counts and evidence-fact counts are different units and do not establish equivalent computational budgets. We clarified the respective limits, qualified our performance claims to the reported settings, and documented the ontology-count selection protocol.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Responses

We thank all three reviewers for their careful reading and constructive feedback. We have expanded the experimental evaluation and clarified the manuscript accordingly. The main revisions are summarized below, with revised text highlighted in blue.

1. Expanded related work. Added discussions of C-TLogic, TempValid, and recent type-aware and entity-aware rule learning methods for static knowledge graphs.

2. Extended ablation study. Added ICEWS05-15 and controls for semantic grouping, soft membership, name and structural representations, temporal history, and score aggregation. We also report evidence overlap between the two rule banks and provide detailed ablation settings.

3. Sensitivity to group counts. Added comparisons using 30- and 70-group partitions across all four datasets, with definitions of the reported group-level statistics.

4. Additional baseline comparisons. Added INFER-60 results on three ICEWS datasets, group-wise comparisons with INFER, and the successful ICEWS14 reproduction of TempValid.

5. Runtime and resource analysis. Added inference-time, LLM-call, and prompt-length statistics for pairwise and listwise correction, together with hardware and memory-use information.

6. Clearer experimental protocols. Specified averaging over at least three random seeds, clarified ontology-count selection, and distinguished rule-count limits from evidence-fact limits.

7. Refined interpretation and presentation. Clarified the roles of semantic grouping, structural evidence, and LLM correction, qualified performance comparisons, and corrected table annotations.

Area_Chair_c7WVOpenReview ↗

Please Respond to the Authors’ Rebuttal

Dear Reviewers,

We are now roughly halfway through the Author–Reviewer Discussion period (which runs through 18 November). The authors have submitted their response and, in some cases, a revised manuscript.

Could you please take a moment to:

  • Read the authors’ rebuttal and any updated version of the paper;
  • Leave a reply to the authors on the points you raised, indicating whether your concerns have been addressed.

Even if your evaluation remains unchanged, a short reply confirming your final position is very helpful, as it allows me to track each reviewer’s stance and write an accurate meta-review.

If you are unable to participate further, please let me know as soon as possible.

Thank you for your time and careful reviewing.

Best regards,

Area Chair, ICLR 2027

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.