- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates whether knowledge graph completion (KGC) models rely on compositional reasoning, primarily in the form of two-hop relational paths. The authors intervene on the training graphs by modifying KG triples, and control the extent of these structural changes using edit distance and two-hop closure. They then evaluate how the performance of KGC models changes under different graph modifications across models and datasets. Based on these experiments, the authors conclude that structural modifications of the graph do not provide a clean way to determine whether a model actually relies on compositional reasoning.
Critical strengths
The paper studies an interesting question, that is, how much two-hop relation paths contribute to the performance of KGC models. The topic is relevant to understanding what structural information these models actually exploit during reasoning.
Critical weaknesses
- The main conclusion is that the proposed graph intervention cannot cleanly determine whether KGC models rely on compositional reasoning. However, this type of triple modification is not a widely adopted evaluation protocol in existing work. Thus, despite the extensive analyses, the paper mainly shows that this particular intervention fails to isolate compositional reasoning, which limits its significance.
- Some experimental controls also need stronger justification. In Section 4, models such as DistMult, ComplEx, and CompGCN are used as controls because they do not perform query-conditioned path traversal. However, this does not mean they cannot capture multi-hop relational structure. For example, DistMult uses as the training objective, while the entity embeddings are jointly optimized over all observed triples in the training graph. As a result, two-hop structures such as and can still indirectly shape the representations of , , and , even without an explicit path traversal. Thus, performance degradation under rewiring only shows that explicit path traversal is unnecessary for the drop, not that the drop is unrelated to compositional structure.
Questions for authors
Could the authors clarify whether this type of graph rewiring intervention has actually been used in prior KGC work to support claims about compositional reasoning?
2 comments
Follow-up: short summary of our answers.
Dear Reviewer 4ohB,
Thank you again for your review. As the discussion closes on 18 November, here is a short summary of our answers to your main concerns. The revised PDF and supplement are uploaded; blue text marks the passages that answer review points.
- Your question. To our knowledge, our exact design has not been used to support such a claim; the inference behind it has, for claims about paths and message passing (Li et al., 2023; Shomer et al., 2025; deletion tests for path explanations). Appendix A (Table 6) documents prior use.
- Significance. We test that inference on a published experiment: Li et al.'s random-graph result reproduces after retraining, yet a model trained on the original graph loses 0.0522 MRR when given the random graph at test time (Table 5, Appendix K). Retraining shows what training can reach, not which evidence a trained model uses.
- Controls. We agree with your point: DistMult, ComplEx and CompGCN are no longer treated as non-compositional controls or used as evidence about composition. Frozen rule scorers, a path-evidence audit and a synthetic graph where the evidence is known take their place (Sections 6 and 7).
For anything not listed here, please see our response above. We would be grateful to know whether these answer your concerns, and we are glad to address anything that remains.
Best regards, the authors
Response to Reviewer 4ohB
We thank the reviewer for the review and for finding the question of what two-hop paths contribute an interesting and relevant one. We agree that this exact protocol is not in wide use, and we agree fully with your point about the controls; the paper is revised around both. We start with your question, because it bears on the first weakness.
We read the question as asking whether the problem we study is real, not only whether our exact rewiring has been used, and answer both.
Of the 11 studies in Table 6 that draw conclusions about multi-hop or path structure from a modification or ablation, only Li et al. hold the amount of change fixed, and only by edge count.
The study matters beyond this one test because the inference behind it is in use, as listed above. For models whose rules cannot be read off, such as ULTRA, a behavioral test of this kind is one of the few probes available. Our checks (the path-evidence audit, edge selection at fixed summaries, and scorers whose computation is known) apply to any structural modification described by aggregate statistics; our results cover our swaps and Li et al.'s experiment.
To show this on a published experiment, we ran Li et al.'s setup with their code and supervision (Table 5, Appendix K). Their finding reproduces that retrained CompGCN reaches MRR 0.357 with the original graph and 0.354 with the random one (theirs: 0.355 and 0.353). Yet a CompGCN trained on the original graph loses 0.0522 MRR when given the random graph at test time. Retraining shows what training can reach, not which evidence a trained model uses, so a matched MRR after retraining does not show that a trained model ignores its graph.
We agree. Embeddings trained on all training triples can encode multi-hop structure without explicit traversal, so a drop of these models shared with ULTRA shows only that traversal is unnecessary for it. We no longer use them as evidence about composition (Section 8; Appendix K for CompGCN).
In their place:
In short, a drop under this rewiring, even with an edit-matched control, does not isolate compositional reasoning, and Section 8 gives the checks that a study using such a modification should report.
Thank you for your review.