- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether LLMs trained on graph-structured data memorize topology, not only sequential text or code. The authors define graph structural memorization as an exposure-dependent gain in recoverability of a target graph that exceeds the corresponding gain on size-matched graphs never seen in training. They then recast the same recovery procedure as a black-box training-graph stealing attack: the attacker is given the candidate node set and node text, queries every pair for adjacency, and assembles predicted edges into a reconstructed graph.
Critical strengths
The paper goes beyond raw link-recovery accuracy and explicitly tests whether recovery changes are target-specific. The rewiring control demonstrates that model behavior tracks exposed relationships rather than only shared node identities and descriptions.
Critical weaknesses
W1: Graph structural memorization is not shown to be a new concept distinct from ordinary fine-tuning memorization of labeled facts. Furthermore, training graph stealing is not a new attack primitive, as it is essentially an exhaustive pairwise querying of a model trained to answer such queries.
W2: The scale of the experimental setup is too small, overly sparse, and excessively privileged to support the derived conclusions. (1), the size of the target graphs is with a split ratio of 9/9/3/3, featuring only one graph per size. (2), each target is trained in isolation under , however graph-LLMs are not trained by repeatedly training a 20-node subgraph 60 times in dedicated runs. (3), the masked graph context provides a 15-node ego graph with only the queried edge removed, which leaks a substantial amount of topological structure during inference.
W3: It lacks mechanistic analysis and defensive measures.
Questions for authors
see the weakness.