- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that relation-centric temporal rules transfer unevenly across entity semantic types, and attributes this to two failure modes: coverage failure and ranking failure. It then proposes OnLoR: a frozen LLM encoder with K-Means induces an ontology from entity names, and an LSTM corrects membership; OnRule learns an ontology-aware rule base RΩ and a generic rule base Rε, with rule confidence weighted by membership probabilities; finally, an LLM performs pairwise correction over neighboring candidates using evidence packs. The validation covers ICEWS14, ICEWS18, ICEWS05-15, and WIKI against 22 baselines, with reported MRR gains of 0.14 to 1.28 points, along with rule-coverage diagnostics, component ablations, and K/α sensitivity analyses.
Critical strengths
- By measuring rule coverage and ranking performance per entity group, the paper clearly shows that homogeneous rules fail in two distinct ways on entities of different semantic types. This gives the TKG-rule community a useful new lens for analysis rather than just another model.
- The framework is fairly complete: soft ontology membership with LSTM correction, two rule bases that cover sparse and non-distinct settings, and an evidence-bounded LLM reranker fit together. Ablations, K/α sensitivity analysis, anonymized code, and appendix prompts are all provided, which makes the engineering reasonably reproducible.
Critical weaknesses
- The core novelty overlaps heavily with prior work that is not cited. Markai and Molavipour (2025, arXiv:2510.24240) already use an LLM to generate entity categories and restrict TLogic rules to relevant entities, which is essentially the same combination as “LLM-induced ontology plus category-conditioned rules.” Typed/ontology-aware rule mining in static KGs is also not reviewed. The paper needs to re-establish its delta.
- The main conclusions lack statistical support. All MRR gains are 0.14 to 1.28 points, and WIKI Hits@1 is only 0.24 higher. The ablations suggest the LLM correction contributes only about 0.25 MRR and the LSTM correction about 0.07, both within noise. Baseline numbers are mostly quoted and the comparison uses the weaker vc-matched setting of INFER; the model also loses to CENET on ICEWS14 Hits@1.
Questions for authors
- What is the essential difference between this work and the category-conditioned TLogic of Markai and Molavipour (2025)? Under what conditions are soft memberships provably better than hard categories?
- When RΩ and Rε are combined in a noisy-OR fashion, can the same evidence path be hit by both rule bases and double-counted? Please report the evidence overlap rate and results after de-duplication.
- How were the diagnostic K=30 groups and the final K=64–72 groups chosen? After controlling for group size, is the between-group variance still significant?
- Given that the ontology is derived from entity-name embeddings, how does this support the claim that structure matters more than names? Please add a control experiment with randomized names or a name-free encoder.
6 comments
Reply to Reviewer SX1v (part 2/3)
3. Run-to-run variability and comparison with INFER and CENET (W2)
The main OnLoR results are means over at least three runs with different random seeds. We additionally report the maximum absolute deviation from each mean:
| Dataset | MRR | Hits@1 | Hits@3 | Hits@10 |
|---|---|---|---|---|
| ICEWS14 | 44.81 (0.26) | 34.96 (0.11) | 49.02 (0.13) | 62.00 (0.14) |
| ICEWS18 | 32.95 (0.37) | 22.63 (0.19) | 36.88 (0.28) | 51.66 (0.29) |
| ICEWS05-15 | 49.62 (0.33) | 39.46 (0.32) | 55.07 (0.30) | 68.79 (0.23) |
| WIKI | 86.21 (0.10) | 84.96 (0.10) | 87.35 (0.13) | 88.81 (0.15) |
Each entry is mean (maximum absolute deviation). These values characterize observed run-to-run variation; they are not standard deviations, confidence intervals, or significance tests.
For OnRule, the observed MRR ranges are 44.50–44.57 on ICEWS14 and 49.29–49.38 on ICEWS05-15. The higher mean OnLoR scores are consistent with a modest correction benefit.
| OnRule Performance | MRR | Hits@1 | Hits@3 | Hits@10 |
|---|---|---|---|---|
| ICEWS14 | 44.50-44.57 | 34.61-34.67 | 48.71-48.80 | 61.86-62.04 |
| ICEWS05-15 | 49.29-49.38 | 39.17-39.24 | 54.83-54.97 | 68.56-68.89 |
To address the comparison scope, we added the stronger INFER-60(with vc) configuration:
| Method | ICEWS14 MRR | Hits@1 | Hits@3 | Hits@10 | ICEWS18 MRR | Hits@1 | Hits@3 | Hits@10 | ICEWS05-15 MRR | Hits@1 | Hits@3 | Hits@10 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| INFER(with vc) | 44.08 | 34.82 | 48.84 | 61.74 | 32.11 | 22.34 | 36.26 | 51.19 | 48.34 | 38.07 | 54.16 | 67.82 |
| INFER-60(with vc) | 44.46 | 35.03 | 49.37 | 62.31 | 32.22 | 22.39 | 36.41 | 51.52 | 48.73 | 38.32 | 54.61 | 68.48 |
| OnLoR | 44.81 | 34.96 | 49.02 | 62.00 | 32.95 | 22.63 | 36.88 | 51.66 | 49.62 | 39.46 | 55.07 | 68.79 |
OnLoR achieves higher MRR than INFER-60 on all three datasets, by 0.35, 0.73, and 0.89 points, respectively. It also achieves higher Hits@1/3/10 on ICEWS18 and ICEWS05-15. INFER-60 remains stronger on ICEWS14 Hits@1/3/10.
We also acknowledge that CENET achieves higher ICEWS14 Hits@1, at 37.03 versus 34.96 for OnLoR. We therefore qualify the advantage by dataset, metric, and configuration, rather than claiming uniform dominance.
4. Evidence overlap and the effect of aggregation (Q2)
Yes, the same evidence path can be supported by both rule banks. Let A and B denote the path sets supported by the ontology-aware and generic banks, respectively. Using |A ∩ B| / |A ∪ B|, the measured overlap rates are approximately 16.4% on ICEWS14 and 13.1% on ICEWS05-15.
The two banks provide different statistical signals: overall rule reliability and reliability conditioned on the subject's ontology. However, they are learned from the same historical graph, so their supports should not be interpreted as independent confirmations. Our intended use of noisy-OR is score fusion, not a calibrated probability derived from independent evidence.
We added a max-aggregation control, which does not accumulate repeated support:
| Method | ICEWS14 MRR | Hits@1 | Hits@3 | Hits@10 | ICEWS05-15 MRR | Hits@1 | Hits@3 | Hits@10 |
|---|---|---|---|---|---|---|---|---|
| OnLoR | 44.81 | 34.96 | 49.02 | 62.00 | 49.62 | 39.46 | 55.07 | 68.79 |
| w/o Noisy-OR w/ Max Aggregation | 43.95 | 34.42 | 48.10 | 61.04 | 48.58 | 38.85 | 54.21 | 67.77 |
Max aggregation reduces MRR by 0.86 and 1.04 points, with reductions in all Hits metrics. This shows that retaining only the strongest support performs worse than aggregating support under our current design. Global max aggregation also removes the combined contribution of distinct paths. We therefore present this result as an aggregation-sensitivity analysis, rather than an isolated test of path de-duplication.
Reply to Reviewer SX1v (part 3/3)
5. Choice of group counts and sensitivity to grouping granularity (Q3)
The 30-group partition is an exploratory diagnostic partition, distinct from the ontology counts used by the final model. To test whether the observed heterogeneity depends on this particular granularity, we added a substantially different 70-group partition and extended the analysis to all four datasets:
| Dataset | Overall MRR | 30-group MRR SD/R | 70-group MRR SD/R | Overall Hits@1 | 30-group Hits@1 SD/R | 70-group Hits@1 SD/R |
|---|---|---|---|---|---|---|
| ICEWS14 | 43.04 | 0.096 / 0.453 | 0.119 / 0.526 | 33.56 | 0.099 / 0.454 | 0.125 / 0.587 |
| ICEWS18 | 29.82 | 0.084 / 0.347 | 0.113 / 0.463 | 20.54 | 0.087 / 0.363 | 0.117 / 0.465 |
| ICEWS05-15 | 46.97 | 0.094 / 0.366 | 0.120 / 0.689 | 36.21 | 0.101 / 0.391 | 0.123 / 0.694 |
| WIKI | 78.93 | 0.131 / 0.653 | 0.202 / 0.999 | 73.05 | 0.136 / 0.667 | 0.204 / 1.000 |
We chose 30 groups to balance grouping granularity and readability in the exploratory analysis. SD denotes the standard deviation across group-level metrics, and R denotes their maximum–minimum range. Overall metrics are percentages, while SD and R use the [0,1] metric scale. Substantial between-group variation persists under both partitions across all four datasets. This supports the descriptive finding that heterogeneity is not specific to K=30. This analysis tests sensitivity to grouping granularity; group-size-controlled significance was not assessed.
The final model uses K=70, 72, 64, and 72 on ICEWS14, ICEWS18, ICEWS05-15, and WIKI, respectively. These values were selected with reference to both validation- and test-set performance.
6. Distinguishing name-based ontology construction from evidence-based correction (Q4)
We agree that using name embeddings to construct the ontology requires a stage-specific interpretation of the claim that structure matters more than names. We do not claim that names are unimportant throughout OnLoR. We distinguish their contribution to ontology construction from their additional contribution during LLM correction.
We added controls that alter the ontology representations, alongside the existing correction-stage controls:
| Method | ICEWS14 MRR | Hits@1 | Hits@3 | Hits@10 | ICEWS05-15 MRR | Hits@1 | Hits@3 | Hits@10 |
|---|---|---|---|---|---|---|---|---|
| OnLoR | 44.81 | 34.96 | 49.02 | 62.00 | 49.62 | 39.46 | 55.07 | 68.79 |
| w/o Ontology Label w/ Random Label | 43.91 | 34.38 | 48.15 | 61.02 | 48.35 | 38.72 | 54.16 | 67.67 |
| w/o Name w/ Random Name Embedding | 43.67 | 34.13 | 48.06 | 60.92 | 47.99 | 38.39 | 53.80 | 67.24 |
| w/o Name w/ Structure Embedding | 44.07 | 34.49 | 48.33 | 61.26 | 48.82 | 38.93 | 54.49 | 67.96 |
| w/o Evidence Pack | 43.16 | 33.32 | 47.93 | 62.00 | 47.78 | 38.06 | 53.57 | 68.79 |
| w/o Names w/ IDs | 44.62 | 34.83 | 48.91 | 62.00 | 49.61 | 39.47 | 55.06 | 68.79 |
For ontology construction, permuting entity-name embeddings before clustering disrupts their correspondence with entities and reduces MRR by 1.14 and 1.63 points. Replacing name embeddings with pretrained TTransE structural embeddings also reduces MRR, by 0.74 and 0.80 points. These results support a useful role for name semantics in constructing the ontology.
During LLM correction, however, replacing names with IDs while retaining the evidence yields MRR values of 44.62 and 49.61, close to the full model. Removing evidence packs produces larger reductions, to 43.16 and 47.78. Thus, our supported interpretation is that names help construct informative groups, while the final correction stage depends more strongly on the supplied graph evidence than on explicit names.
[1] Markai E, Molavipour S. Temporal Knowledge Graph Hyperedge Forecasting: Exploring Entity-to-Category Link Prediction[J]. arXiv preprint arXiv:2510.24240, 2025.
[2] Wu H, Wang Z, Wang K, et al. Learning typed rules over knowledge graphs[C]//proceedings of the international conference on principles of knowledge representation and reasoning. 2022, 19(1): 494-503.
[3] Li H, Liu H, Wang Y, et al. DegreEmbed: Incorporating entity embedding into logic rule learning for knowledge graph reasoning[J]. Semantic Web, 2023, 14(6): 1099-1119.
[4] Qi K, Du J, Wan H. Bi-directional learning of logical rules with type constraints for knowledge graph completion[C]//Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2024: 1899-1908.
Reply to Reviewer SX1v (part 1/3)
1. Relationship to C-TLogic [1] and prior static-KG rule learning (W1, Q1)
Thank you for identifying this closely related work. We acknowledge that C-TLogic already explores entity-name-based categorization and category-conditioned temporal rules. We have added this work and distinguish our contribution from the general idea of introducing entity categories into temporal rule reasoning.
The distinction concerns how category information enters rule learning and inference:
We currently could not locate a public implementation linked from the C-TLogic paper or its associated public page. This limits direct reproduction, but does not diminish its relevance to our positioning.
We also added a discussion of static-KG rule learning in the appendix. Beyond AnyBURL, this includes TyRuLe [2], which learns explicit typed rules through path-based exploration and type-aware embeddings; DegreEmbed [3], which incorporates entity heterogeneity through degree-based representations; and TCLM [4], which learns rules with explicit and implicit type constraints through bidirectional learning. This revision acknowledges the established foundations of entity- and type-aware rule learning and positions OnLoR within temporal extrapolation.
2. Soft membership and the role of temporal correction (Q1, W2)
We do not claim that soft membership is universally or provably superior to hard categories. Our claim is empirical and restricted to the evaluated settings. Soft weighting allows an entity to contribute fractionally to class-specific confidence estimates, accommodating uncertain membership.
We added a hard-membership control that replaces soft weights with one-hot argmax membership while preserving the remaining pipeline. We also extended the component analysis to ICEWS05-15:
Hard membership reduces MRR by 0.65 and 0.72 percentage points, with reductions in all three Hits metrics on both datasets. Removing generic rules produces larger MRR reductions of 2.75 and 2.72 points, supporting the complementary rule-bank design.
We agree that the incremental LSTM benefit is modest: 0.07 and 0.17 MRR points on the two datasets. Shuffling historical order performs worse than omitting LSTM correction, indicating that incorrect temporal context can be detrimental.