- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies cross-graph link prediction with knowledge graph foundation models (KGFMs) and proposes ChainGram. The method converts relational chains sampled around support facts into identity-independent structural tokens and constructs shared structural codebooks through word-level and chain-level vector quantization. These representations are aggregated into relation-specific structural patterns, which guide the matching and weighting of chains rooted at the query entity. The resulting chain evidence refines the entity predictions of a query-conditioned GNN.
The main contribution is the integration of relational-chain representations, hierarchical discrete structural abstraction, and query-specific evidence selection into a reasoning architecture with parameters shared across knowledge graphs.
The authors pretrain the model on three source KGs and evaluate it on 54 target benchmark configurations spanning three transfer settings under zero-shot and full fine-tuning protocols. The main results report relative improvements in overall MRR of 2.82% and 5.84%, respectively, over the strongest baseline. Ablation studies, codebook perturbation experiments, and controlled structural probes provide additional evidence that the hierarchical codebooks and chain-matching mechanism contribute to predictive performance.
Critical strengths
- A well-motivated mechanism for structural transfer
The hierarchical codebooks organize local relational roles and whole-chain patterns into shared structural units, which are subsequently used for relation initialization and query-specific evidence matching. This design directly addresses a central challenge in cross-KG transfer: entity and relation identities need not be shared, whereas structural patterns may be transferable.
The performance degradation observed when the structure–code assignments are perturbed provides stronger evidence of the codebooks’ predictive utility than clustering visualizations alone.
This is an important methodological strength because the approach offers a concrete mechanism for organizing transferable structural knowledge, rather than merely improving prediction through a less structured representation.
- Broad evaluation with consistent aggregate improvements
The main evaluation covers 54 target benchmark configurations, including unseen-entity, unseen-entity-and-relation, and transductive settings, under both zero-shot and fine-tuning protocols. The aggregate results in the main table consistently favor the complete model over the compared methods.
This breadth is particularly relevant to the overall assessment because transferability is the central objective of KGFMs. Evaluation across multiple target settings provides substantially stronger support for that objective than improvements on a small number of individual datasets.
Critical weaknesses
- The method specification does not yet adequately support the central claims of ordered relational composition and grammar matching
There are two related technical concerns.
First, the preservation of relational information is unclear. According to Eq. (4), two chains with the same sequence of entity distance labels and the same query-relation indicators could receive identical token sequences even if the types or repetition patterns of their non-query relations differ. Subsequent Transformer encoding and quantization cannot recover information that is absent from the input.
This does not imply that the AAAB/AABA/ABAA probe is necessarily indistinguishable under the proposed tokenization: the patterns could still be distinguished if A or B is marked as the query relation. However, the authors need to specify the query relation and the resulting tokens used in this probe. Without this information, it is difficult to determine whether the experiment tests general relational-composition capabilities or primarily the ability to identify the position of a query-relation marker.
Second, the attention-based matching mechanism requires clarification. If is a single vector, the attention weight over the single key in Eq. (16) is necessarily one. In that case, the attention operation itself cannot select among different patterns based on the query chain. The subsequent matching network can still depend on the query chain, so this concern does not imply that the entire reasoner is ineffective. Nevertheless, the explanation of pattern matching through attention needs to be revised or made more precise.
These concerns are important because they affect whether the specified model implements the paper’s central mechanism, rather than merely reflecting minor presentation issues. If unresolved, they would weaken both the technical justification of the contribution and its reproducibility.
- The contribution of the structural grammar mechanism is not sufficiently disentangled from conventional chain modeling and additional computation
The model includes multiple Transformers, two levels of codebooks, and an additional chain-based prediction branch. Although the ablations provide useful evidence, they do not yet clearly establish whether the gains primarily arise from hierarchical discrete abstraction or from stronger continuous chain encoding and additional computation.
In particular:
- M1 and M4 are both described as removing chain reasoning or matching and relying on the base GNN; their actual architectural differences are unclear.
- Parameter-matched or compute-matched continuous chain baselines are not sufficiently reported.
- The main results do not include variability estimates or confidence intervals. While some appendix analyses use three random seeds, the statistical treatment of the main table remains unclear.
- Some reported advantages are very small. For example, under fine-tuning in the Inductive-E setting, Hits@10 is 0.5679 compared with 0.5678 for the strongest baseline. Statistical evidence is needed to assess the robustness of such differences.
Chain modeling and vector quantization are not themselves new techniques. Unless the paper establishes that their proposed combination provides distinctive benefits beyond conventional model enhancement, the contribution may be better characterized as an effective architectural integration than as evidence of structural grammar discovery.
Questions for authors
A. Clarification of the Current Method
Question 1: What relational information is preserved by Eq. (4)?
Please provide the complete token sequence for a concrete example chain and clarify:
- Whether the tokenization encodes equality or differences among non-query relations;
- How forward and inverse relations are represented;
- How is specified in the AAAB/AABA/ABAA probe.
Question 2: What are the actual tensor shapes in Eqs. (12) and (16)?
Is a single vector or a collection of pattern representations? If it is a single vector, please explain the role of attention over a single key. If it is a collection, please specify how the individual patterns are constructed and aggregated.
B. Evidence Supporting the Current Conclusions
Question 3: Does the hierarchical discrete mechanism provide statistically robust improvements over continuous chain models under comparable resource budgets?
Please clarify the distinction between M1 and M4 and provide a directly comparable continuous chain control model, together with parameter counts, inference costs, and results across multiple random seeds.
Question 4: What are the exact evaluation sets and the rules governing access to graph information?
Please clarify:
- Why the target-set composition changes from 23/16/15 to 23/18/13 in the alternative pretraining experiment;
- Which target datasets are excluded or replaced when FB15k-237 and CoDEx-M are used for pretraining;
- Whether the prediction target triple and its inverse edge are removed from the reasoning context during training;
- Whether support sampling strictly uses only facts permitted by the evaluation protocol.
5 comments
W2. Disentangling the grammar mechanism from chain modeling and additional computation
Thank you for this valuable feedback. We (1) clarify the definitions of M1 and M4, (2) add controls for grammar guidance and discrete abstraction, (3) report parameter and inference costs, and (4) report multi-seed results.
Part 1: Definitions of M1 and M4.
- M1 (w/o chain reasoning) measures the overall contribution of chain-based reasoning. It removes both the grammar encoder (Eqs. 2–12) and the chain reasoner (Eqs. 15–18), so predictions are made solely by the base query-conditioned GNN.
- M4 (w/o grammar-guided reasoner) measures the contribution of query-side chain grammar. It keeps the grammar encoder for relation initialization and removes only the query-chain reasoner and its score correction (Eqs. 15–18).
Part 2: Contribution of grammar guidance and discrete abstraction. We add three controls that change only one component at a time:
- M4-U and M4-C keep the query-chain reasoner and its score correction but remove grammar matching, using uniform weights or weights learned from the chain representation alone.
- M5 removes both word-level and chain-level vector quantization and uses continuous representations instead, serving as a directly comparable continuous chain model.
M4-U and M4-C bring only marginal gains over M4 despite keeping the chain branch, and M5 underperforms the full model, showing that the gains come from grammar guidance and discrete abstraction rather than continuous chain encoding.
| Variant | Ind-E,R MRR | Ind-E,R H@10 | Ind-E MRR | Ind-E H@10 | Trans MRR | Trans H@10 | Overall MRR | Overall H@10 |
|---|---|---|---|---|---|---|---|---|
| M1: w/o chain reasoning | 0.4107 | 0.5897 | 0.4112 | 0.5437 | 0.3228 | 0.4670 | 0.3864 | 0.5420 |
| M4: w/o grammar-guided reasoner | 0.4194 | 0.6058 | 0.4165 | 0.5557 | 0.3251 | 0.4634 | 0.3923 | 0.5514 |
| M4-U: uniform chain weights | 0.4198 | 0.6027 | 0.4201 | 0.5553 | 0.3238 | 0.4687 | 0.3932 | 0.5514 |
| M4-C: chain-only weights | 0.4227 | 0.6046 | 0.4204 | 0.5554 | 0.3243 | 0.4680 | 0.3947 | 0.5521 |
| M5: continuous (w/o both VQ levels) | 0.4177 | 0.5915 | 0.4173 | 0.5500 | 0.3136 | 0.4683 | 0.3887 | 0.5450 |
| ChainGram (full) | 0.4247 | 0.6069 | 0.4227 | 0.5562 | 0.3280 | 0.4725 | 0.3973 | 0.5546 |
Part 3: Parameter and computational cost. ChainGram is computationally efficient. On 10 inductive KGs with an A100 GPU, the chain components add about 0.24M parameters over M1 but increase inference time by only 4.0%. M5 has nearly the same cost as the full model, so discrete abstraction brings its gains at negligible extra cost.
| Variant | #Params | Total time (10 KGs) | Time per query group |
|---|---|---|---|
| M1: w/o chain reasoning | 142,392 | 140.87 s | 4.108 ms |
| M4: w/o grammar-guided reasoner | 294,604 | 143.50 s | 4.185 ms |
| M5: continuous (w/o both VQ levels) | 374,974 | 146.30 s | 4.266 ms |
| ChainGram (full) | 381,886 | 146.44 s | 4.270 ms |
Part 4: Statistical robustness. We run ChainGram with three random seeds and report the zero-shot results below. The overall MRR of ChainGram (0.3980 ± 0.0013) exceeds that of the strongest baseline KG-ICL (0.3864) by a margin far larger than its standard deviation.
| Metric | Ind-E,R | Ind-E | Trans | Overall |
|---|---|---|---|---|
| MRR | 0.4260 ± 0.0014 | 0.4228 ± 0.0010 | 0.3289 ± 0.0013 | 0.3980 ± 0.0013 |
| H@10 | 0.6068 ± 0.0012 | 0.5556 ± 0.0008 | 0.4734 ± 0.0028 | 0.5546 ± 0.0015 |
We have clarified the definitions of M1 and M4 in Sec. 4.4 and added these variants to Appendix D.4 of the revised manuscript.
Q1. Relational information preserved by Eq. (4)
Thank you for this question. As it is closely related to W1 Part 1, we briefly answer each of its three sub-questions below.
- (1) Equality or differences among non-query relations. The tokens do not encode equality among non-query relations; instead, relational identity is recovered through the slot aggregation in Eq. (12), where positions occupied by the same relation are pooled into the same slot.
- (2) Forward and inverse relations. Each relation is augmented with an inverse relation as a separate relation, so direction is represented by relation identity and preserved in separate slots.
- (3) Query relation in the probe. is randomly assigned a relation ID distinct from both and , so the three patterns can only be distinguished through relational composition rather than a query marker.
For the complete token sequence of a concrete example and further details, please kindly refer to W1 Part 1.
Q2. Tensor shapes in Eqs. (12) and (16)
Thank you for this question. As it is closely related to W1 Part 2, we briefly clarify the tensor shapes and the construction of the pattern tokens below.
- (1) Single vector or collection. Eq. (12) produces a pattern for every relation, and the attention in Eq. (16) operates over a collection of four pattern tokens rather than a single vector.
- (2) Construction and aggregation. The patterns of all relations are averaged into a summary , which is then expanded into the four tokens through a learnable projection with token offsets.
For the full formulation, please kindly refer to W1 Part 2.
Q3. Robustness of the hierarchical discrete mechanism
Thank you for this question. We have conducted additional ablations, cost measurements, and multi-seed experiments, which are detailed in W2. We summarize the answers to each requested item below.
- (1) Distinction between M1 and M4. M1 removes all chain components, whereas M4 keeps the grammar encoder and removes only the query-chain reasoner.
- (2) Comparable continuous chain control. M5 replaces both levels of vector quantization with continuous representations and underperforms the full model.
- (3) Parameter counts and inference costs. M5 has nearly the same cost as the full model, and ChainGram increases inference time by only 4.0% over M1.
- (4) Multiple random seeds. Over three seeds, ChainGram achieves an overall MRR of 0.3980 ± 0.0013, well above the strongest baseline.
For the detailed results, please kindly refer to W2 Parts 1–4.
Q4. Evaluation sets and access to graph information
Thank you for these questions. We clarify the target-set composition under the alternative pretraining setting, the removal of target triples during training, and the graph information used for support sampling below.
(1)–(2) Target-set composition and replaced datasets. Under the alternative setting, FB15k-237, WN18RR, and CoDEx-M replace FB-v1, NELL-v1, and CoDEx-S as pretraining KGs, so these six KGs swap their pretraining and target roles:
| Target group | Main → Alternative | Added datasets | Removed datasets |
|---|---|---|---|
| Inductive-E,R | 23 → 23 | – | – |
| Inductive-E | 16 → 18 | FB-v1, NELL-v1 | – |
| Transductive | 15 → 13 | CoDEx-S | FB15k-237, WN18RR, CoDEx-M |
| Overall | 54 → 54 | 3 KGs | 3 KGs |
(3) Removal of target triples. During training, the target triple and its inverse edge are removed from the graph used for GNN message passing, and query-rooted chains that traverse either edge are discarded.
(4) Support sampling. Following KG-ICL [1], support facts and their chains are sampled only from the observed graph of each KG under the evaluation protocol, i.e., the training graph for transductive KGs and the inference graph for inductive KGs. Validation and test triples are never used as support facts.
We thank the reviewer again for the valuable comments, which have helped us improve the clarity and rigor of the paper.
[1] Yuanning Cui, Zequn Sun, and Wei Hu. A Prompt-Based Knowledge Graph Foundation Model for Universal In-Context Reasoning. Advances in Neural Information Processing Systems, 37:7095–7124, 2024.
We sincerely thank the reviewer for the thorough comments and for recognizing our well-motivated transfer mechanism and broad evaluation. The two central questions are whether the input representation preserves relational composition, and whether the gains come from discrete abstraction rather than additional computation. We address each comment point by point below.
W1. Method specification of relational composition and grammar matching
Thank you for raising these concerns. We clarify the method specification in two parts: (1) how relational information is preserved in Eq. (4) and (2) how attention-based matching is implemented in Eq. (16).
Part 1: Preservation of relational information in Eq. (4). The tokenization in Eq. (4) is designed to be identity-independent: since relation identities are not shared across KGs, the tokens encode only transferable structural roles, and relation identity is re-attached during the aggregation in Eq. (12).
[ x(δ0), y10, x(δ1), y21, x(δ2), y30, x(δ3), y40, x(δ4) ],
where x(δk)=Eent[δk]+tent with δk=(min(d(ek,u),3), min(d(ek,v),3)), and ykm=trel+pk+m⋅eqry with m=1[rk=rq]. Each relation r has an inverse r−1 as a separate relation, so direction is represented by relation identity. The tokens mark only whether a relation equals rq.
Part 2: Attention-based matching in Eq. (16). We apologize for the confusion caused by the oversimplified notation in Eq. (16). In fact, in our code, the grammar pattern is not a single key but is expanded into four pattern tokens, so the reported results are unaffected. Specifically, the patterns hrpat∈Rd from Eq. (12) are averaged into uq∈Rd and expanded into Pq∈R4×d:
uq=∣R∣1∑r∈Rhrpat,Pq=Reshape(Wpuq)+O,an=MHA(gn,Pq,Pq),
where Wp∈R4d×d and O∈R4×d are learnable. The attention weights depend on both the query chain gn and Pq, so each chain obtains its own readout an of the support grammar, which is then scored by fmatch([gn∥an∥gn⊙an]). Removing this matching while keeping the chain branch (M4-C in W2 Part 2) reduces overall MRR from 0.3973 to 0.3947.
We have revised Eq. (16) in Sec. 3.3 to match the implementation.