- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies schema invariance in knowledge graph foundation models (KGFMs). It introduces SCHEMASHIFT, which evaluates answer-preserving schema rewrites using a context-only protocol. The experiments show substantial losses in filtered MRR and equivalent-view consistency (EVC). The authors compare augmentation, consistency objectives, and structural normalization. A blind structural normalizer with a relation-purity test recovers near-original MRR and high top-1 agreement when the relevant motifs are recognized.
Critical strengths
1.Clear distinction between link invariance and schema invariance. The paper identifies an important limitation: invariance to renaming does not guarantee consistent predictions under answer-preserving schema rewrites.
2.Well-controlled context-only protocol. Queries, candidates, and filtered sets remain fixed. Deletion and matched-noise reference conditions, together with experiments across datasets and backbones, strengthen the empirical diagnosis.
3.Effective structural normalization. The relation-purity test reduces false positives, while iterative normalization handles composed reification. For recognized motifs, normalization restores near-original MRR and high EVC without retraining the backbone.
Critical weaknesses
1.Limited evaluation of structural normalization across backbones. Although normalization results for ULTRA cover six KGs, results for MOTIF and TRIX are limited to CoDExSmall, and no normalization results are reported for FLOCK. Extending recovery experiments to the datasets already evaluated for schema sensitivity would provide stronger evidence for the normalizer’s effectiveness across backbones.
2.Dependence on motif recognition. The normalizer recognizes reification and path motifs. Partial recognition remains a major failure, while single-edge mediators and higher-arity encodings remain open.
3.Incomplete verification of normalization. On benchmark graphs, structural checks compare edge and relation counts rather than perform an edge-level round trip. Near-original MRR and high EVC do not establish ranking invariance or score invariance.
4.Incomplete sensitivity analysis. Figure 10 evaluates the purity threshold using structural counts rather than MRR and EVC. The support threshold is compared only at and , and different pass limits are not evaluated. Further analysis of their effects on MRR and EVC would clarify how reliably the normalizer works under different settings.
Questions for authors
Please see the Weaknesses.
3 comments
Dear Reviewer M8Cb,
Thanks for your time and effort in reviewing our work. Your comments are valuable to improve the quality of it. As the author-reviewer discussion period will end in about 2 days, we gently request that you read the rebuttal and check whether your concerns are addressed. Thanks again for your time and help in improving the quality of this paper.
Regards from the authors.
Maintain score
Thank you for your response. I will keep my original score.
Response to Reviewer M8Cb
W1. Recovery across backbones. Table 3 (main text) and Table 25 (Appendix G.3) add MOTIF and TRIX on WN18RR and DBpedia100k, and TRIX on AristoV4, besides CoDExSmall. Top-1 agreement with the original view after normalization, reified view:
MRR stays within 0.0013 of the original in every row. The weakest case is MOTIF on CoDExSmall at s=2 (0.908 on the shared-role view), where residual mediators at s=2 are associated with changes in 3–9% of its top-1 answers. Limits: on DBpedia100k the query sets differ by model for memory reasons (2,000–3,000 queries); MOTIF was not run on AristoV4. For FLOCK, the replicate study on CoDExSmall (Table 28) shows that after normalization at s=1, where the graph is recovered exactly, we detect no difference from resampling of the original (excess +0.006, interval −0.005 to +0.018). Partially recovered graphs were not tested with FLOCK.
W2. Dependence on motif recognition. We agree that motif coverage limits the remedy. The normalizer has two predefined motifs. Partial recognition remains its main failure (11–33% relative MRR on uniform views), single-occurrence mediators are left by s=2, and higher-arity structure is outside the library. The Wikidata experiment (Table 27) now measures this on native data: statements with qualifiers are never collapsed, coverage is 71–81% of statements, and recovery is partial (agreement 0.61). The abstract and Section 7 state the remedy as conditional.
W4. Counts do not establish exact recovery. We agree that matching counts is insufficient. We therefore compared the normalizer's output with the original edge by edge on 156 graph instances of eight graphs, as equality of directed labeled edge sets up to a relation bijection (Section 6.2, Table 3; Appendix G.2, Table 24). The normalizer never sees which nodes were introduced; only the validator does.
The abstract now reports the exact-recovery count (44 of 148 rewritten instances at the default threshold), and the text separates count checks, exact graph recovery and prediction recovery. Our earlier statement that no genuine node was collapsed holds for reification views only and is corrected.
W5. Sensitivity analysis. Figure 11 (Appendix G.4) reports downstream agreement over a grid fixed before scoring: s in {1, 2, 3, 5}, τ in {0, 0.5, 0.7, 0.9, 1}, and three stopping rules, on CoDExSmall, WN18RR and DBpedia100k.
This is a retrospective analysis of the paper's graphs. The values s=2 and τ=0.9 were chosen after those graphs had been inspected; we have not selected thresholds on disjoint graphs, and Sections 6.2 and 7 say so. WD50K is the only graph in the paper that was not inspected before the thresholds were fixed.