- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The authors train autoregressive graph generation models specifically on whole molecular graphs. Their contribution broadly addresses capturing (and generating) substructures, and organises over two main parts. First they show that generated molecules contain common substructures at roughly the same empirical rates, but also that simpler baselines appear tp achieve this, thus not identifying how the model generates substructures. Second, they move to a setup where specific substructures are removed from the training dataset to check if they are then generated. The authors do it specifically for patterns C3 and P6, and find the retrained model still very occasionally produces them. This excludes, by construction, copying training data and, for C3, producing outputs by joining training fragments only through bridge bonds. Either way, the removal significantly, strongly suppresses both the selected targets.
Critical strengths
- (S1) The experimental design is generally thorough. For example, the matched-removal condition separates the effect of removing the target from the effect of having less data, and the comparison against real-data resamples gives meaningful reference points.
- (S2) The general methodology of holding out substructures and inspecting whether those are generated is interesting.
- (S3) The paper is generally upfront about limitations.
Critical weaknesses
- (W1) The presentation quality is, according to the reviewer, particularly poor, making the paper extremely difficult to follow and interpret. There is very little narrative in the paper that could guide readers understand the proposed contribution. The paper itself does not read a whole argument, rather as a storm of (quantitative) statements, unexpected acronyms and references to results and data with qualifications and caveats and not much interpretation. This makes it really hard for readers to identify potential relevant take home messages. Some issues worth flagging in particular:
- The authors consistently use undefined or internal terminology, such as "DGMG-style", or "without establishing serialization invariance" even before the setup is introduced, or "not rule acquisition" without explanation, or, as a further example "headline DFS-code model".
- Some results seem to be imported without explanation: for example, the paragraph at lines 149-157 mentions a value (bound?) of 0.0012 with an associated probability of at least 0.95 which is not clear where it comes from, or how it is derived. Then it moves to explain it "does not directly apply" to their data-selected panel?
- Some passages are of unclear purpose: The above could already represent an example. On top of this, another example is the discussion of absorbing-state diffusion (lines 117 onwards) which introduces a theoretical link and right after it notes that the DiGress comparison does not test this.
- (W2) The contributions and the motivation behind them are not clear. The paper generally frames the investigation around how autoregressive models generate substructures, starting from the observation, in the introduction, that "matching substructure frequencies does not itself show how a generator produces substructures" (lines 43-44). Based on this, the authors move to study the support of substructures in Section 4. However, it is not clear how such "support" can give a more informative measure, as it essentially seem to capture a similar measurement to frequency. Why should one expect a fidelity metric to inform on how a model generates its outputs in the first place, and why support could be a better metric w.r.t. say frequency? In fact, as the authors themselves write in line 156: "support agreement alone cannot identify how outputs are produced", so what is even the point of the whole Section 4? Its role in the argument is unclear, especially considering the conclusions seem to be vain anyway. If it is a methodology contribution, the paper should evaluate what it enables beyond existing metrics, for example.
- (W3) The significance of results in Section 5, remain , to the reviewer, narrow.
- For example, for C3, the experiment shows that the target-containing outputs are not copies of training graphs and could not be produced by joining training fragments only through bridge bonds. However, it remains unclear why anyone would hypothesise that the Transformer generates (subgraphs and then) graphs according to this approach? More plausible explanations remain open, as the authors acknowledge (lines 428 through 433), while still this pertains to only an extremely limited number of such C3 patterns found to be generated, so that it would be difficult, anyway, to extrapolate general pattern formation rules.
- Still on C3, as this is made entirely absent from the withheld training sets, a model that perfectly fits its training distribution would never emit it. So this entails that it is not obvious whether emission at such extremely low rates are to be interpreted as generalisation or simply "model error".
- (W4) The scope is generally limited: the main results are supported on one dataset only, PCQM4Mv2, on a non-official split (whose motivation is not given), and only two particular target substructures.
Questions for authors
- (Q1) The paper emphasises that models are trained "without explicit substructure supervision". However, next-token prediction trains on prefixes that correspond to partial graphs. This could in fact give some signal on substructures similarly to how next token prediction provides signal also on grammar and words other than whole sentences. Could the authors clarify in what sense substructures are unsupervised?
Also please see (W2) and (W3).
5 comments
The tables are in the response to all reviewers (Part 2/2).
We substantially revised the exposition in response to the concerns about clarity: we rewrote the text in plain sentences and moved procedural details to the appendix. The research question, the data, the experiments and the results of the submitted version are unchanged. We summarize the additional experiments below; the change to the scope of our claims is described in point 3 of Part 1/2.
The following includes results from additional experiments conducted after submission. All of these results are included in the revised version (Appendix L; Tables A, B and C correspond to Tables 24, 25 and 26 in the revised manuscript).
Table A. Target Occurrences After Target Removal (5 Targets)
Each value is the ratio of the target-occurrence count from a model trained after removing target-containing training graphs to the target-occurrence count in the matched-removal control (§5.1). The training and sampling settings are the same as in §5, with 163,840 attempts per model.
| Target | Training graphs removed: count (percentage) | Transformer (by training seed) | Exact-count 8-gram |
|---|---|---|---|
| Triangle C3 | 193,874 (5.7%) | 0.13%, 0.19%, 0.23% / additional seeds: 0.03%, 0.13%, 0.21% | 21% [18, 24] |
| 6-membered ring C6 | 217,136 (6.4%) | 2.02%, 2.10%, 1.80% | 24% [18, 31] |
| 5-membered ring C5 | 145,699 (4.3%) | 2.74%, 3.32%, 2.80% | 50% [42, 60] |
| Chain P6 | 178,620 (5.3%) | 10.6%, 10.0%, 10.1% | 86% [78, 95] |
| Chain P7 | 88,401 (2.6%) | 10.89%, 9.58%, 9.34% | 83% [70, 99] |
The first 3 Transformer values and the 8-gram value for triangles, and the values in the P6 row, were reported in the submitted paper (§5.2 and §5.3; Tables 17, 20 and 21). Intervals were calculated using the same method as in Table 20. The targets differ in frequency, size, and the graphs removed. The ordering of the ratios is therefore an observation, not an explanation of its cause.
Table B. Comparison at the Same Generation Step (Triangles)
The values show how much removing target-containing training graphs reduces the probability of a triangle-closing step (mean change in log probability, in nats; model trained after target removal minus control; ratios capped at 100-fold; brackets indicate nominal 95% intervals across training seeds and states). A value of −2.3 corresponds to reducing this probability to 1/10 of the control probability.
| State | Number of states | Transformer | Model using only the preceding 7 tokens | Difference (Transformer − local model) |
|---|---|---|---|---|
| Triangles closed among 3 vertices with nonconsecutive indices | 1,102 | −3.66 [−3.74, −3.58] | −3.87 [−3.95, −3.79] | +0.21 [+0.12, +0.30] |
| Of these, cases where the 3rd edge is at least 9 tokens earlier | 398 | −3.76 [−3.88, −3.63] | −3.78 [−3.91, −3.64] | +0.02 [−0.13, +0.18] |
| Distant closures reached by the 8-gram generator | 674 | −1.49 [−1.69, −1.30] | −1.44 [−1.61, −1.29] | −0.05 [−0.24, +0.15] |
| Triangles retained in training | 2,000 | −0.03 [−0.16, +0.11] | 0.00 [−0.01, 0.00] | −0.03 [−0.16, +0.11] |
We specified how to interpret the results before evaluation. For the probability of a triangle-closing step, the decrease under the Transformer does not exceed that under the local model by a factor of 2 or more. This is a one-sided test and does not establish equal suppression in the 2 models. Of the 10 local models examined, this result holds in comparisons with 8 models over all states and 7 models over distant states.
Table C. Valid-Graph Rates and Violated Rules
All values are percentages of all attempts. Every generator was trained on the control training data and evaluated for 163,840 attempts using the same sampling procedure as in §5.
| Generator | Valid-graph rate | Edge destination is not an earlier node | Generated graph is disconnected |
|---|---|---|---|
| Transformer (6 training seeds) | 97.5–97.9% | 0.1–0.2% | 1.7–2.2% |
| 8-gram | 25.1% | 34.3% | 40.6% |
| 5-gram | 3.3% | 83.2% | 13.4% |
| 3-gram | 0.7% | 98.6% | 0.65% |
No n-gram attempt was classified as a token-ordering error (each invalid attempt is counted under the first rule it breaks). This table shows the rules violated by the generated graphs.
Other Additional Results
- Untruncated sampling. Every model trained after removing triangles or P6 generates its target. The ratios of target-occurrence counts, summed over the 3 training seeds, to those in the control are 0.55% [0.47, 0.64] for triangles and 14.8% [14.3, 15.3] for P6.
- DFS code. The ratios of triangle-occurrence counts to those in the control are 0.95%, 0.92%, and 1.01% (DFS codes and training recipe of §4, decoder of §5). Because this experiment uses different settings from §5, we do not attribute the difference between the values to serialization.
Response to Reviewer RZAJ
Thank you for the detailed review. Following our response to all reviewers, we address your individual comments below.
In the following, “†” indicates experimental results obtained after submission. All of these results are included in the revised version (Appendix L). The tables are in the response to all reviewers (Part 2/2).
W1 (Presentation)
We agree with your assessment. In the revision, each of the three experimental sections (§4, §5 and §6) opens with its question and closes with the findings. We have addressed the 3 points you raised as follows.
- Undefined terminology. We replaced “DGMG-style” with “sequences that add nodes and edges one at a time,” and “headline DFS-code model” with “the DFS-code model of Section 4.” We replaced “serialization invariance” and “rule acquisition” with plain-language statements.
- The value 0.0012. This value quantifies the degree of support agreement for a generator that simply replays training graphs. We derived it from Hoeffding’s inequality and a union bound (253 patterns and 3,371,958 training graphs; the derivation is in the appendix). The main text now explains this in one sentence (§3.1).
- We removed the paragraph on diffusion models because it is unnecessary for presenting the paper’s main results.
W2 (What support means and the purpose of §4)
Support is the proportion of graphs that contain a given substructure.
The purpose of §4 is to explain why the target-removal experiments in §5 are needed. Even a generator that simply recombines training-graph fragments can reproduce substructure frequencies with accuracy close to that of the Transformer (Table 1). As you point out, agreement in frequencies does not reveal how a model generates graphs. The opening paragraph of §4 now states this purpose.
W3 (The significance of §5)
§5 shows that the Transformer generates structures withheld from training, as n-gram models do.
- Why we examined fragment recombination. We chose this generator because it reproduced substructure frequencies with accuracy close to that of the Transformer in §4. This generator cannot produce the withheld triangle (Lemma 5), whereas the Transformer can (Table 18).
- A more plausible explanation. Local token statistics remain a possible explanation of the model’s decisions concerning triangles. After submission, we directly examined the relationship between those decisions and local token statistics (point 3 of the response to all reviewers; Table B)†.
- Generalization or model error. In language-model terms, the generation of structures withheld from training is analogous to creating an unseen phrase from familiar word sequences. N-gram models, which use only a short token context, also generate withheld structures (§5.3, Table 21). The Transformer generates them while maintaining the validity of the graph as a whole (Table C)†.
W4 (Scope)
We expanded the number of targets from 2 to 5 and the number of training seeds for triangles from 3 to 6 (Table A)†. For all 5 targets, removing training graphs that contain the target greatly reduces its occurrence in Transformer outputs, but every model trained on the remaining data still generates the target.
We used a random split to sample held-out molecules from the same distribution as the training molecules. In §4, we use held-out samples of the same size as a reference.
Testing on other corpora is a next step.
Q1 (What does “without explicit substructure supervision” mean?)
We mean that model training uses no substructure labels, fragment vocabulary, or substructure-specific loss. As you point out, next-token prediction exposes the model to partial graphs, just as a language model receives signals about words and grammar from sentences. In this analogy, n-gram models also reproduced substructure statistics quite well; these substructures correspond to phrases. A major difference between the Transformer and n-gram models is the validity of the graph as a whole, which corresponds to grammar.
† These results were added after submission. The tables are in the response to all reviewers (Part 2/2).
Response to All Reviewers (Part 1/2) — What This Paper Shows
We thank all reviewers. Since all three reviews pointed out that the paper's contribution was difficult to identify, we begin by restating our main contributions to address this shared concern.
By analogy with language models, substructures correspond to phrases, and graph validity corresponds to grammar. In our results, n-gram models also reproduced substructures, the analogue of phrases, quite well. A major difference between the Transformer and n-gram models was in the aspect corresponding to grammar.
Summary. The Transformer generates a carbon triangle withheld from training, like n-gram models, while preserving whole-graph validity, like a generator that recombines training-graph fragments. Each simple generator we evaluated shows only one of these two behaviors.
In the following, “†” indicates experimental results obtained after submission; all of these results are included in the revised version (Appendix L).
1. Transformer-generated graphs reproduce substructure frequencies at the level observed in real data. A simple generator can also reproduce these frequencies with similar accuracy.
In Transformer-generated graphs, the support of 253 frequent substructures deviates from the support in the training graphs by 0.0112 on average, after removing outputs whose strings exactly match training strings. The mean deviation from training-graph support is 0.0121 for held-out samples and 0.0138 for a generator that only recombines training-graph fragments. The errors of 3-gram and 5-gram models are 2–4 times larger (all results in Table 1). The role of §4 (Observation: Generated Graphs Reproduce Substructure Support) is to use this comparison to explain why the removal experiments in §5 (Test: Generating Targets Withheld from Training) are needed.
2. Structures withheld from training become much rarer in Transformer outputs. Even so, the Transformer generates new graphs containing the withheld structures.
When we remove all training graphs containing a target pattern and retrain, target occurrences are reduced relative to the control by a factor of 430–3,000† for carbon triangles and approximately 10 for carbon chains (Figure 2, Table 20; Table 24). Nevertheless, every model trained after target removal generates its target. Under the sampling procedure in §5 (Test: Generating Targets Withheld from Training), none of the 107† outputs containing a triangle is labeled-isomorphic to any original training graph (Table 18; Appendix L.1). We also show separately that recombining training-graph fragments only through bridges, as our fragment generator does, cannot create a carbon triangle (Lemma 5).
3. Local token statistics remain a possible explanation of how the withheld triangle is generated.
N-gram models also generate triangles withheld from training (§5.3, Table 21). Based on differences in the numbers of triangles generated by the Transformer and n-gram models, the submitted paper stated that the Transformer suppresses them far more strongly than local token statistics do (§7; see also §5.3). However, when we compare how much removing target-containing training graphs reduces the probability of the same triangle-closing step, the Transformer's suppression (3.66 nats) does not exceed that of a model using only the preceding 7 tokens (3.87 nats)†. We tested only whether the Transformer's suppression was more than twofold stronger in probability than the local model's; we did not establish equal suppression in the 2 models. We have clarified the meaning of this comparison in the revision (§5.3, §7).
4. A major difference between the Transformer and n-gram models is whole-graph validity.
The Transformer produces valid graphs in 97.5–97.9% of attempts†. An 8-gram model trained on the same data produces valid graphs in 25.1% of attempts†. Most n-gram failures involve 2 types of rule violations: (1) adding an edge whose destination is not an earlier node and (2) producing a disconnected graph. The frequency comparison in Table 1 uses only valid outputs (§3.3). Thus, frequency agreement alone does not imply that a generator can produce valid whole graphs.
Remaining question. The internal mechanism is not yet identified. Points 3 and 4 give two clues, but how the Transformer generates the withheld triangle and preserves whole-graph validity remains unknown.