- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes using the pairwise distance matrix directly as the value-mixing operator in graph transformers and introduces DisAttention. It derives mixing weights from distances only, and DisMix adds query-key affinity, replacing softmax with normalization and a learned distance gate. Experiments on molecular and geometric benchmarks show competitive accuracy with fewer parameters, lower memory, and faster training.
Critical strengths
- The paper proposes a new way to incorporate distance embeddings in graph transformer architectures
- The method makes the model and training faster and more efficient
Critical weaknesses
- Lack of comaprisons: Distance-based message weighting is not new. EGNN (https://arxiv.org/pdf/2102.09844) uses pairwise distances to modulate messages, and SchNet uses continuous filters on interatomic distances. Neither is compared on QM9. More broadly, the QM9 baselines are limited and omit standard geometric models such as DimeNet, PaiNN, or equivariant transformers. Without these, it is hard to assess where the method stands.
- Relation to positional encodings: Graph transformers already incorporate structure through positional encodings, including relative, spatial, and Laplacian-based variants. These are also lightweight ways to inject positional dependence. The paper should clarify how distance-driven mixing differs from these approaches, ideally with a direct empirical comparison.
- The proposed models do not achieve the best test AP on MolPCBA, and several baselines perform better. Parameter efficiency is a useful property, but it is not compelling on its own if accuracy lags behind.
- Expressivity studies: The paper provides no theoretical analysis of the expressive power of distance-driven mixing relative to standard attention. The empirical results also do not show a clear advantage.
Questions for authors
- How were the coordinates for molecules in MolPCBA obtained, or is the distance matrix being computed in any other way?
- Are node positions also input to the model?
- Have the authors considered non-Euclidean distance measures, such as shortest-path distances on 2D molecular graphs or on graphs without spatial embeddings (e.g., social or citation networks)?
- Is there a reason why specific function representations are chosen for and . Have the authors explored different parameterizations?
3 comments
Response to Reviewer Fj1D (2/2): conditions, further results, and revisions
W1.
- QM9 comparison: train / validation / test = 110,000 / 10,000 / 10,831, the same target normalization; each model keeps its own representation and readout and uses its published size (parameter counts are in the appendix table of Comparison with Standard Geometric Models); the checkpoint is selected on validation. TorchMD-Net ET keeps vector features that transform with rotation, which differs from the distances and scalar value mixing of our models.
- Global connection, controlled task (only the distance between two fragments changes): local SchNet / DisMix / DisAttention, test MAE↓ = / / .
- DES370K: each model uses its published default connectivity. For charged dimers beyond 5 Å (mean kcal/mol), SchNet (10 Å) / DisMix: / kcal/mol.
- Cost: against a channelwise continuous filter in the same backbone, DisMix has a faster training update on ModelNet40 with 1,024 points.
W3. 12 layers, hidden dimension 384, 12 heads, FFN dimension 384; the same atom, bond, and hop information and RDKit coordinates; official full split; 50 epochs. TGT-Ag, EGT, and GPTrans keep their own pair states and propagation structures. EGT + RDKit / GPTrans + RDKit: / . Relative to Vanilla, DisAttention has about 27% fewer parameters and a 19% shorter step.
W4.
- Identical node features: if every node value is , the output of row-softmax is , and this holds across layers that repeat the same node-wise operations. The output of our operator is . The appendix constructs a distinguishable pair of inputs and shows that the difference remains after the residual connection and LayerNorm.
- Bound on amplification: for a symmetric nonnegative affinity, , and , so . The proof and a sensitivity analysis with respect to the log-affinity and the gate are in the appendix.
- The ModelNet40 control uses the same pairwise distances and backbone and the same training and validation-search budget.
W2. 8 layers, hidden dimension 256, 45 epochs from scratch, the same training configuration. QM9 HOMO MAE↓ in the same order: / / meV. Row-wise softmax fixes each row sum to one; our operator also learns the total transmission.
Q1. Hydrogens are added explicitly; UFF is used when MMFF94 is not applicable. Coordinates are generated once before training and the stored distance matrices are reused. The results in the submission exclude 349 molecules whose generation failed (116 / 97 / 136 in train / validation / test). In the unified comparison these molecules use shortest-path distances, so all models are evaluated on the official full split; the difference in test AP between the two evaluation sets is / %p (pretrained / scratch).
Q2. The node features of LP-PDBbind are the atom type, a protein/ligand indicator, and residue information. A point cloud in ModelNet40 has no separate attribute such as an atom type, so XYZ forms the initial representation of each point.
Q3. DisMix on QM9 , 3D → SPD: MAE↓ = → bohr² (). The MolPCBA comparison uses 30 epochs from scratch and .
Q4.
- Each head learns the center and width of its Gaussian; and the gate take the distance expanded in radial basis functions. The output layers of and are initialized to zero, so training starts from Gaussian-based mixing with and .
- does not change when is multiplied by a positive constant. The range of limits the amplification of each weight to a factor of two.
- Ablation in the main text, QM9 gap MAE↓: Gaussian + content / without radial bias / without gate / DisMix = / / / meV.
Revisions (highlighted in blue in the revised PDF)
- Related Works: relation to continuous-filter methods and to positional encodings.
- Distance-based Mixing: function design, role of the gate after normalization, and the interpretation from .
- MolPCBA experiment: the comparison under unified conditions.
- Appendix: the theory section Structure, Expressivity, and Stability of Distance-based Mixing, Comparison with Standard Geometric Models, Global Continuous Filters (including DES370K), Additional Controls for the Mixing Operator, Matched MolPCBA Comparison, Distance Inputs and Conformer Sensitivity, and Experimental Details (coordinate generation, node features, RBF design and initialization).
Dear Reviewer Fj1D,
Thank you again for your review. We have posted our response and a revised PDF (changes highlighted in blue). The main additions for each point are as follows.
- W1. SchNet, EGNN, PaiNN, DimeNet++, and TorchMD-Net ET on QM9 with the same split and budget. DisMix has lower gap error than SchNet and EGNN and higher error than PaiNN, DimeNet++, and TorchMD-Net ET. DES370K charged dimers separated by more than 5 Å, test MAE: PaiNN (5 Å cutoff) / DimeNet++ (5 Å cutoff) / DisAttention = 1.29 / 2.08 / 0.19 kcal/mol.
- W2. Relative distance bias with the same distance information, changing only the normalization and the gate, QM9 gap MAE: row-softmax / row-softmax + gate / DisMix = 50.56 / 49.35 / 47.83 meV.
- W3. MolPCBA from scratch with the same backbone, inputs, and budget, test AP: DisAttention / DisMix / Vanilla QK + distance bias = 29.52 ± 0.13 / 29.27 ± 0.22 / 29.22 ± 0.18%; TGT-Ag, EGT, and GPTrans, 28.37–28.75%. DisAttention has 9.47M parameters, the fewest among these models.
- W4. An appendix theory section (exact decomposition of the operator, the case of identical node features, and a bound on amplification) and a controlled ModelNet40 experiment.
- Q1–Q4. Coordinate generation, node features per dataset, shortest-path distances, and alternative gate parameterizations are described in the response.
The discussion period ends on November 18 (AoE). We would appreciate knowing whether the response addresses your concerns, and we will answer any remaining questions before then.
Best regards, The Authors
Response to Reviewer Fj1D (1/2)
We thank the reviewer for the positive view of how distance is incorporated into the graph transformer and of the training efficiency. We also appreciate the specific comments on the comparisons and on expressivity; we added all the requested comparisons and analyses.
W1. Comparison with SchNet, EGNN, and standard geometric models
We evaluated the requested models on QM9 with the same split and training budget.
W3. Accuracy on MolPCBA
The models in the literature table (Table 1) differ in pretraining and training conditions, so we added a comparison under unified conditions; we agree that parameter efficiency matters only when accuracy is kept. Trained from scratch with the same backbone, inputs, and training budget, DisAttention obtains the highest mean AP with the fewest parameters.
W4. Theoretical analysis of expressivity and empirical advantage
We added a theory section. The operator decomposes exactly as W=diag(m)Π, where Π holds the relative transmission proportions and mi is the total transmission of node i. When all node features are identical, the output of row-softmax attention does not depend on distances, whereas the output of our operator reflects geometry through mi. In addition, ∥W∥2≤2 bounds the amplification independently of graph size.
W2. Relation to positional encodings
Positional encodings provide structural information as an input from which attention learns its weights; our work constructs the transmission weights themselves from distances. The two can be used together. For relative distance bias, the most direct counterpart, we changed only the normalization and the gate under the same distance information and logits.
Q1, Q2. MolPCBA coordinates and node positions as input. We generate one conformer per molecule from SMILES with RDKit ETKDGv3, optimize it with MMFF94, and compute Euclidean distances from its coordinates. Coordinates are not node features on MolPCBA, QM9, and LP-PDBbind; ModelNet40 uses XYZ and MNIST superpixel uses the normalized 2D position as node features.
Q3. Non-Euclidean distances. The same operator can be applied to shortest-path distances, and the gain from continuous geometry depends on the task; transductive graphs without coordinates, such as social or citation networks, are not in our evaluation scope.
Q4. Choice of function representations and other parameterizations. The affinity G(d)eb(d), a Gaussian with a learnable correction, learns the relative weights over distances. Symmetric normalization is invariant to the overall scale of the affinity, so the transmission strength is learned separately by the gate after normalization, g(d)=2σ(q(d)).
In summary, the comparison with the requested geometric models, the MolPCBA comparison under unified conditions, and the theory and controlled experiment on expressivity clarify where the method stands and under which conditions its advantage appears. Conditions, further results, and the list of revisions follow in the next comment, and we are happy to answer any further questions.