- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents TurboPairFormer, an optimized triangular-attention implementation for accelerating protein folding model training. It introduces a fused backward algorithm with deterministic gradient reductions and output-residual compensation to improve efficiency and numerical accuracy. Experiments on H100 GPUs demonstrate improved gradient accuracy, reproducibility, and up to 1.73× training-step speedup over OpenFold3's Triton backend.
Critical strengths
- Efficient and deterministic kernel design: The proposed backward algorithm reduces redundant recomputation while ensuring deterministic gradient accumulation. Output-residual compensation further improves numerical accuracy.
- Comprehensive experimental evaluation: The paper compares against multiple optimized backends and demonstrates promising improvements in kernel performance, gradient accuracy, and integrated training efficiency.
Critical weaknesses
- Memory scalability concerns: The proposed method materializes dQ partial gradients, introducing cubic workspace complexity with respect to crop size. This may limit its applicability to larger protein complexes, despite the demonstrated computational improvements.
- Limited evidence for end-to-end benefits and long-term training stability: The reported training speedups also benefit from optimized triangle multiplication kernels, making the contribution of triangular attention alone less clear in the main results. Moreover, the stability experiments provide limited evidence of long-term convergence and robustness across independent training runs.
Questions for authors
- Memory Scalability: The materialized dQ partial buffer scales cubically with crop size. How does this affect peak GPU memory and practical performance for larger crops (e.g., 1024 or beyond)? Could partial materialization or alternative reduction strategies reduce this overhead?
- Speedup Attribution: The reported training-step speedups reflect improvements from both triangular attention and triangle multiplication. Although Appendix J provides component ablations, could the authors clarify the contribution of each optimization under the main experimental settings?
- Training Stability: The main stability experiment uses a single 400-update continuation per crop from a shared checkpoint. Could the authors provide stronger evidence from longer training runs or multiple independent seeds to establish whether the numerical improvements translate into reliable training stability and convergence?
4 comments
[W2/Q3 – Training stability – Multi-seed runs]
We thank the reviewer for the suggestion. We trained four implementations of the same model (TurboPairFormer, Protenix, cuEquivariance, OpenFold3 Triton) from the released Preview2 checkpoint (of3-p2-155k.pt) for 600 updates each, with three independent seeds (42, 43, 44). Training settings: Adam (β₁ = 0.9, β₂ = 0.95, ε = 10⁻⁸), learning rate 3 × 10⁻⁴ with 50 linear warm-up steps, EMA decay 0.98, per-sample gradient clipping at 10, global batch size 8 (one structure per H100, 8 × H100), crop 384 tokens, 32 diffusion samples per structure, on kinase fine-tuning data.
Stability. All updates (4 × 3 × 600) produced finite losses and gradients, with no divergence or restart. The loss trajectories overlap update by update: the mean difference between TurboPairFormer and each baseline is ≤ 1 × 10⁻⁴ at a loss of ≈ 1.22. The loss decrease (0.023–0.029 per run) and the gradient-norm statistics are also indistinguishable.
Convergence. We evaluated all checkpoints on the kinase validation set with identical evaluation kernels. The table reports means over three seeds; bold marks the best of the four implementations. For Median RMSD, lower is better; for all other metrics, higher is better. Figure: https://anonymous.4open.science/r/iclr27_rebuttal-0800/validation_multiseed_kinase_8xh100.pdf.
| Metric | Preview2 | TurboPairFormer | Protenix | cuEq | Triton |
|---|---|---|---|---|---|
| Model selection | 0.748 | 0.749 | 0.747 | 0.747 | 0.747 |
| lDDT protein–ligand | 0.743 | 0.765 | 0.765 | 0.754 | 0.762 |
| lDDT protein–protein | 0.599 | 0.605 | 0.593 | 0.596 | 0.596 |
| DockQ | 0.534 | 0.527 | 0.517 | 0.522 | 0.523 |
| RMSD < 2 Å success (%) | 53.2 | 62.0 | 61.6 | 59.1 | 64.1 |
| Median RMSD (Å) | 1.864 | 1.736 | 1.747 | 1.797 | 1.694 |
TurboPairFormer has the highest model-selection score, protein–protein lDDT and DockQ, and ties for the highest protein–ligand lDDT. All differences between implementations are within seed-to-seed variation.
If our responses have addressed your concerns, we would be sincerely grateful if you would kindly consider raising your score.
I thank the authors for their detailed responses and additional experiments. The new results have addressed several of my concerns, particularly regarding memory scalability, performance attribution, and training stability. I appreciate the authors' efforts in providing further evidence and clarifications. I maintain my positive rating.
We thank the reviewer for the follow-up and positive assessment. We appreciate your constructive suggestions and will incorporate the additional results and clarifications into the revised manuscript.
We thank the reviewer for the comments and suggestions to improve our paper!
[W1/Q1 – Memory scalability – Peak memory at large crops]
We agree that the query-gradient partial buffer grows cubically with crop size, which we state as a limitation in the paper. In our measurements, however, it does not set the training peak. On one H100 80 GB, TurboPairFormer, Protenix and cuEquivariance peak within 0.2 GiB of each other at every crop from 384 to 1152 (Table 1). At crop 1280, all three run out of memory at the same allocation, in the backward pass of the diffusion transformer’s attention, which uses PyTorch’s reference attention with every method.
Table 1. Peak allocated memory per training update (GiB), one H100 80 GB. Same structures and recipe as our reply to yhAQ (Q3), measured with the earlier implementation, with 50 warm-up and 150 measured updates for every method.
The peak comes from the diffusion module. At crop 768, the diffusion module, which processes 32 diffusion samples per update, is the largest part of the peak in a single-update snapshot (12.3 of 37.7 GiB, 33%). The breakdown is identical for TurboPairFormer and Protenix (memory figure), because all attention backward calls, and with them the partial buffers, come after the peak. At much longer crops, the cubic term will eventually set the peak. Alternative reductions bound the buffer but cost speed in our kernel: at N = 768 and D = 32, wave-compact partials cut attention scratch by 62% for 2.3% more kernel time, and semaphore-ordered dQ by 73% for 19.4% more backward time. At crops 896 to 1152, our latest implementation also takes 5.2–7.0% less time per update than Protenix and 17.6–22.5% less than cuEquivariance (reply to yhAQ, Q3). We will add these results to the appendix.
[W2/Q2 – Speed attribution – Attention and TriMul measured separately]
We separated the contributions of attention and triangle multiplication (TriMul) by replacing one operator at a time. For the attention-only comparisons, cuEquivariance TriMul is fixed.
Our latest attention implementation introduces four post-submission optimizations: (1) revised thread mapping and vectorized loads for delta computation; (2) adjusted shared-memory padding to reduce bank conflicts in bias-gradient reduction; (3) direct access to supported QKV layouts, avoiding unnecessary packing and gradient-layout copies; and (4) shorter register lifetimes in the backward kernel. These changes preserve the attention formulation and compensation scheme. Outputs and gradients were bitwise identical to the submitted implementation in the tested cases.
Table 2. Single-H100 training comparisons. Change in mean GPU time per training update relative to the baseline (negative = faster), using identical configurations, with our latest implementation, seed 42, 50 warm-up and 50 measured updates.
Table 3. 8×H100 attention-only comparisons. Change in mean GPU time per training update relative to Protenix, over seeds 42–44, each with 50 warm-up and 50 measured updates.
The controlled ablations show that attention accounts for most of the improvement over cuEquivariance. Relative to Protenix, attention-only performance is broadly comparable. Our TriMul provides an additional reduction in training-update time. TurboPairFormer uses fixed kernel configurations without runtime autotuning.Protenix is evaluated with its autotuned configurations, excluding the one-time tuning cost (about 33 s per input shape). We will distinguish the latest implementation versions and operator contributions in the revised manuscript.