Do Structural Modifications Isolate Compositional Reasoning in Knowledge Graph Completion?

Knowledge graph completion models are often credited with compositional reasoning: answering a query by chaining relations along a path. A natural test is a structural modification: the training graph is rewired so that its two-hop paths are destroyed, and a drop in accuracy would be read as evidence that the model used those paths. In this paper, we ask whether this modification isolates composition and find that it does not. The rewiring does remove two-hop rules, as a symbolic rule miner confirms, but it also rewrites up to a quarter of all training triples. To separate the two changes, we build a control graph that is rewired by the same amount but keeps the amount of two-hop structure unchanged. For two path-based models on two benchmarks, including the ULTRA foundation model, this control alone reproduces about three quarters of the drop in MRR. In a grid on CoDEx-S at a fixed edit budget and below the modification's closure loss, the drop follows symmetric and reciprocal triples more closely than two-hop structure. Models with no mechanism for following paths lose more than half as much accuracy as ULTRA under the same rewiring. What remains depends on choices the test leaves open: its size changes with the evaluation metric, and its sign can flip with how the modified graphs are built. A drop under this modification therefore cannot, on its own, show that a model composes. We propose that such studies report a control rewired by the same amount, results on queries with no two-hop path, how the modified graphs were built, and more than one metric. We release our constructions and controls.