acceptodds
Under review as a conference paper at ICLR 2027

TurboPairFormer: Fast and Stable Protein Folding Model Training with an Optimized Triangle Attention Kernel

Abstract

Triangular attention is a core computation in AlphaFold3-style biomolecular models, with computational cost that grows cubically with token count. FlashAttention reduces attention memory traffic through IO-aware tiling and backward recomputation. However, the shared pair bias in triangular attention adds a gradient reduction across attention slices to the usual reductions over queries and keys. The open source backends we examine handle these reductions through repeated probability recomputation, floating point atomics, or full score-gradient storage. Separately, computing the softmax backward correction from BF16-rounded forward outputs incurs a loss of numerical precision. We present TurboPairFormer, a triangular attention implementation for NVIDIA Hopper GPUs that addresses these two issues. Our key tile parallel backward algorithm recomputes each probability tile once and reuses it for the query, key, value, and pair-bias gradients, using ordered partial reductions to achieve deterministic accumulation without floating point atomics or full score gradient storage. Our output residual compensation retains a BF16 approximation of the output rounding residual and uses it to compute the backward correction more accurately in FP32, without changing the BF16 output. With BF16 inputs at crop sizes 384, 640, 768 and head dimensions 16, 32, TurboPairFormer achieves the lowest mean query, key, and pair-bias gradient RMSE against an FP64 reference among eight evaluated implementations; residual compensation mitigates the precision loss from output rounding, reducing these RMSE values by 28–47% in controlled ablations. All four gradients are bitwise identical across five repeated calls in all 600 input cases under fixed execution conditions. Integrated into OpenFold3 together with our triangle multiplication kernels, TurboPairFormer achieves the lowest GPU computation time per optimizer step among eight evaluated backend configurations on 16 H100 GPUs, with speedups of over OpenFold3's Triton backend and over cuEquivariance at crop size 768.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Machine Learning Systems, Efficient Training, GPU Kernel Optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sgQ7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TurboPairFormer, an optimized triangular-attention implementation for accelerating protein folding model training. It introduces a fused backward algorithm with deterministic gradient reductions and output-residual compensation to improve efficiency and numerical accuracy. Experiments on H100 GPUs demonstrate improved gradient accuracy, reproducibility, and up to 1.73× training-step speedup over OpenFold3's Triton backend.

Critical strengths
  1. Efficient and deterministic kernel design: The proposed backward algorithm reduces redundant recomputation while ensuring deterministic gradient accumulation. Output-residual compensation further improves numerical accuracy.
  2. Comprehensive experimental evaluation: The paper compares against multiple optimized backends and demonstrates promising improvements in kernel performance, gradient accuracy, and integrated training efficiency.
Critical weaknesses
  1. Memory scalability concerns: The proposed method materializes dQ partial gradients, introducing cubic workspace complexity with respect to crop size. This may limit its applicability to larger protein complexes, despite the demonstrated computational improvements.
  2. Limited evidence for end-to-end benefits and long-term training stability: The reported training speedups also benefit from optimized triangle multiplication kernels, making the contribution of triangular attention alone less clear in the main results. Moreover, the stability experiments provide limited evidence of long-term convergence and robustness across independent training runs.
Questions for authors
  1. Memory Scalability: The materialized dQ partial buffer scales cubically with crop size. How does this affect peak GPU memory and practical performance for larger crops (e.g., 1024 or beyond)? Could partial materialization or alternative reduction strategies reduce this overhead?
  2. Speedup Attribution: The reported training-step speedups reflect improvements from both triangular attention and triangle multiplication. Although Appendix J provides component ablations, could the authors clarify the contribution of each optimization under the main experimental settings?
  3. Training Stability: The main stability experiment uses a single 400-update continuation per crop from a shared checkpoint. Could the authors provide stronger evidence from longer training runs or multiple independent seeds to establish whether the numerical improvements translate into reliable training stability and convergence?
4 comments

We thank the reviewer for the comments and suggestions to improve our paper!

[W1/Q1 – Memory scalability – Peak memory at large crops]

We agree that the query-gradient partial buffer grows cubically with crop size, which we state as a limitation in the paper. In our measurements, however, it does not set the training peak. On one H100 80 GB, TurboPairFormer, Protenix and cuEquivariance peak within 0.2 GiB of each other at every crop from 384 to 1152 (Table 1). At crop 1280, all three run out of memory at the same allocation, in the backward pass of the diffusion transformer’s attention, which uses PyTorch’s reference attention with every method.

Table 1. Peak allocated memory per training update (GiB), one H100 80 GB. Same structures and recipe as our reply to yhAQ (Q3), measured with the earlier implementation, with 50 warm-up and 150 measured updates for every method.

CropTurboPairFormerProtenixcuEquivariance
38419.219.219.2
64030.930.931.0
76838.538.538.6
89647.547.547.6
102458.858.858.9
115270.470.470.5
1280out of memoryout of memoryout of memory

The peak comes from the diffusion module. At crop 768, the diffusion module, which processes 32 diffusion samples per update, is the largest part of the peak in a single-update snapshot (12.3 of 37.7 GiB, 33%). The breakdown is identical for TurboPairFormer and Protenix (memory figure), because all attention backward calls, and with them the partial buffers, come after the peak. At much longer crops, the cubic term will eventually set the peak. Alternative reductions bound the buffer but cost speed in our kernel: at N = 768 and D = 32, wave-compact partials cut attention scratch by 62% for 2.3% more kernel time, and semaphore-ordered dQ by 73% for 19.4% more backward time. At crops 896 to 1152, our latest implementation also takes 5.2–7.0% less time per update than Protenix and 17.6–22.5% less than cuEquivariance (reply to yhAQ, Q3). We will add these results to the appendix.

[W2/Q2 – Speed attribution – Attention and TriMul measured separately]

We separated the contributions of attention and triangle multiplication (TriMul) by replacing one operator at a time. For the attention-only comparisons, cuEquivariance TriMul is fixed.

Our latest attention implementation introduces four post-submission optimizations: (1) revised thread mapping and vectorized loads for delta computation; (2) adjusted shared-memory padding to reduce bank conflicts in bias-gradient reduction; (3) direct access to supported QKV layouts, avoiding unnecessary packing and gradient-layout copies; and (4) shorter register lifetimes in the backward kernel. These changes preserve the attention formulation and compensation scheme. Outputs and gradients were bitwise identical to the submitted implementation in the tested cases.

Table 2. Single-H100 training comparisons. Change in mean GPU time per training update relative to the baseline (negative = faster), using identical configurations, with our latest implementation, seed 42, 50 warm-up and 50 measured updates.

Comparison38464076889610241152
TurboPairFormer (attention + TriMul) vs cuEquivariance−9.9%−14.7%−16.3%−17.6%−19.0%−22.5%
TurboPairFormer attention only vs cuEquivariance−5.9%−11.9%−12.8%−14.4%−15.8%−19.6%
TurboPairFormer (attention + TriMul) vs Protenix−5.9%−5.2%−5.1%−5.2%−5.8%−7.0%
TurboPairFormer TriMul vs cuEquivariance TriMul, with TurboPairFormer attention fixed−4.3%−3.1%−4.0%−3.8%−3.9%−3.6%
TurboPairFormer attention only vs Protenix−1.7%−2.1%−1.2%−1.5%−2.0%−3.5%
TurboPairFormer attention only vs Protenix without autotuning−3.7%−9.3%−9.6%−10.5%−12.6%−14.0%

Table 3. 8×H100 attention-only comparisons. Change in mean GPU time per training update relative to Protenix, over seeds 42–44, each with 50 warm-up and 50 measured updates.

Comparison384640768
TurboPairFormer vs. Protenix: three-seed aggregate−0.24%−1.06%−0.44%
TurboPairFormer vs. Protenix: best observed seed per crop−6.95%−2.03%−1.32%

The controlled ablations show that attention accounts for most of the improvement over cuEquivariance. Relative to Protenix, attention-only performance is broadly comparable. Our TriMul provides an additional reduction in training-update time. TurboPairFormer uses fixed kernel configurations without runtime autotuning.Protenix is evaluated with its autotuned configurations, excluding the one-time tuning cost (about 33 s per input shape). We will distinguish the latest implementation versions and operator contributions in the revised manuscript.

[W2/Q3 – Training stability – Multi-seed runs]

We thank the reviewer for the suggestion. We trained four implementations of the same model (TurboPairFormer, Protenix, cuEquivariance, OpenFold3 Triton) from the released Preview2 checkpoint (of3-p2-155k.pt) for 600 updates each, with three independent seeds (42, 43, 44). Training settings: Adam (β₁ = 0.9, β₂ = 0.95, ε = 10⁻⁸), learning rate 3 × 10⁻⁴ with 50 linear warm-up steps, EMA decay 0.98, per-sample gradient clipping at 10, global batch size 8 (one structure per H100, 8 × H100), crop 384 tokens, 32 diffusion samples per structure, on kinase fine-tuning data.

Stability. All updates (4 × 3 × 600) produced finite losses and gradients, with no divergence or restart. The loss trajectories overlap update by update: the mean difference between TurboPairFormer and each baseline is ≤ 1 × 10⁻⁴ at a loss of ≈ 1.22. The loss decrease (0.023–0.029 per run) and the gradient-norm statistics are also indistinguishable.

Convergence. We evaluated all checkpoints on the kinase validation set with identical evaluation kernels. The table reports means over three seeds; bold marks the best of the four implementations. For Median RMSD, lower is better; for all other metrics, higher is better. Figure: https://anonymous.4open.science/r/iclr27_rebuttal-0800/validation_multiseed_kinase_8xh100.pdf.

MetricPreview2TurboPairFormerProtenixcuEqTriton
Model selection0.7480.7490.7470.7470.747
lDDT protein–ligand0.7430.7650.7650.7540.762
lDDT protein–protein0.5990.6050.5930.5960.596
DockQ0.5340.5270.5170.5220.523
RMSD < 2 Å success (%)53.262.061.659.164.1
Median RMSD (Å)1.8641.7361.7471.7971.694

TurboPairFormer has the highest model-selection score, protein–protein lDDT and DockQ, and ties for the highest protein–ligand lDDT. All differences between implementations are within seed-to-seed variation.

If our responses have addressed your concerns, we would be sincerely grateful if you would kindly consider raising your score.

I thank the authors for their detailed responses and additional experiments. The new results have addressed several of my concerns, particularly regarding memory scalability, performance attribution, and training stability. I appreciate the authors' efforts in providing further evidence and clarifications. I maintain my positive rating.

We thank the reviewer for the follow-up and positive assessment. We appreciate your constructive suggestions and will incorporate the additional results and clarifications into the revised manuscript.

Reviewer_phrEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

Triangular attention in AlphaFold3-style models differs from standard attention in one structurally important way: the pair bias is shared across all N outer-index slices, so its gradient reduces over a third axis (outer indices) in addition to the usual reductions over keys (dQ) and queries (dK/dV). All four gradients reuse the same P and dS tiles. Existing open-source backends resolve this three-way reduction problem by recomputing probabilities multiple times (TriFast, Protenix: 3×), accumulating the bias gradient with floating-point atomics (DS4Sci, MegaFold), or materializing the full FP32 score gradient (ScaleFold: 6.75 GiB per call at crop 768).

TurboPairFormer is a Hopper CUDA implementation making four claims: (i) a work partitioning that assigns each CTA a key tile plus a group of four outer indices, keeping dK/dV in registers and leaving exactly two cross-block reductions; (ii) one probability recomputation serving all four gradients, with dQ and bias contributions written as uniquely-owned BF16 partials and summed in fixed FP32 index order by separate reduction kernels — no atomics, no dense dS; (iii) output-residual compensation, saving E16 = RN16(O32 − O16) in the forward epilogue and using O16 + E16 in FP32 to compute δ = ⟨dO, O⟩, which lowers the output-conversion error bound from first to second order in BF16 unit roundoff; (iv) forward staging of the shared bias via TMA with accumulator-layout reads and chunked 16-key probability evaluation.

Validation: 16×H100 OpenFold3 training (lowest step time at all three crops: 5.26/9.40/12.58 s; 1.73× over OpenFold3 Triton, 1.13× over cuEquivariance at crop 768); backward RMSE against an FP64 reference across six shapes (lowest for dQ, dK, dbias among eight implementations); 600/600 bitwise five-repeat replay; a crop-wise training-stability study; and 1,000-step kinase fine-tuning validation across nine backends. Propositions D.1–D.3 cover the partitioning identity, workspace-schedule invariance, and the row-correction propagation; Appendix I measures the deterministic design against its non-deterministic and one-block-per-slice alternatives.

Critical strengths
  1. The determinism result is unambiguous, cheap to state, and expensive to obtain elsewhere. Figure 10 reports 600/600 bitwise-identical five-repeat cases for TurboPairFormer while cuEquivariance, OpenFold3 Triton, and DeepSpeed-Evo pass zero and MegaFold passes 165. This is the paper’s cleanest contribution, and it matters because bitwise reproducibility is a precondition for debugging divergence in long structure-prediction training runs, where a NaN at step 40,000 is otherwise unreproducible. What makes it credible rather than merely asserted is that the cost is measured rather than hidden: Appendix I.3 reports that the fastest non-deterministic variant of the same kernel is 2.8–4.1% faster at D = 32 and 11.0–13.7% at D = 16, and I.4 shows the deterministic-by-construction alternative (one block per slice, the DS4Sci/FlashAttention-2 partitioning) is 22.8–95.9% slower. A reader can therefore decide whether they want determinism at that price. Proposition D.2 extends it to workspace rescheduling, which is what makes the property usable in practice rather than only under one fixed launch plan.

  2. Output-residual compensation is a small, well-analyzed idea with a clear empirical signature. The observation is that δ = ⟨dO, O⟩ enters every score gradient in a row in the same direction, so output-rounding error is a systematically biased channel rather than noise — and a deterministic kernel reproduces that bias every run. Retaining a BF16 residual costs one output-sized tensor and no extra probability recomputation, and the ablation is well-designed: 60/60 paired synthetic cases improve for dQ, dK, and dbias (36.3–47.1%, 28.7–42.4%, 28.3–43.1%), the public output and dV stay byte-identical (as Proposition D.3 predicts, since ∆dV = 0), and the error is placed against the right floor — the FP64 reference rounded once to BF16, with mean dQ error falling from 2.76–3.37× that floor to 1.76–1.78×. Captured real inputs corroborate (median errors down 35.9–55.5%) at 1.06–2.33% added operator cost. Naming the floor is what distinguishes this from an unanchored percentage improvement.

Critical weaknesses

I don’t think the headline speedup comes from what the paper claims it comes from. TurboPairFormer (Section 3.1) replaces both triangular attention and triangle multiplication, but Protenix, TriFast, MegaFold, DeepSpeed-Evo and ScaleFold replace attention only and keep cuEquivariance’s TriMul. Working through the paper’s numbers at crop 768: TurboPairFormer is 11.7% faster than cuEquivariance and 2.7% faster than Protenix, which puts Protenix about 9.3% ahead of cuEquivariance. But Table 22 credits only 6.43% of the crop-768 gain to attention and 2.99% to TriMul. Appendix H.1 makes the same point at the operator level, and more directly: at D = 32, the Pairformer configuration, TurboPairFormer’s backward is 2.5% slower than tuned Protenix at N = 384 and 6.7% slower at N = 768, with forward-plus-backward ranging from 7.4% faster to 3.6% slower. That suggests contributions 1 and 2 — the single-recomputation ordered-partial backward, which is the core of the paper — are roughly at parity with the best competing attention kernel at the shape that matters most, and that much of the integrated margin is coming from a TriMul kernel Appendix J treats as a secondary component. Appendix G.5 identifies the experiment that would settle this (TurboPairFormer attention with cuEquivariance TriMul) but doesn’t run it. I’d push on this because the abstract and conclusion advertise 1.13× and 1.73× system-level wins, and nothing in the main text would tell a reader about the parity.

Memory is cubic here, it’s the central tradeoff of the design, and the one memory experiment reported holds it fixed. The dQ partial buffer is Θ(BHDN³/τk), which comes to 1.69 GiB at D = 32, N = 768, against 288 MiB for the FP32 buffer that semaphore-ordered or per-slice accumulation would need (I.5); Table 3’s extrapolation reaches 4096 MiB at crop 1024, which is the direction AlphaFold3-style training is heading. The measured claim is that peak memory didn’t move when the bias-workspace planning target was swept from 64 to 512 MiB (E.4) — but E.4 says that sweep holds the buffer and the saved residual fixed, so what it measures is bias scheduling, not the term that scales cubically. E.3’s headroom argument is right as far as it goes, but it’s conditional: P₁ = P₀ only if W(t) ≤ P₀ − A(t) throughout, and whether that holds is the part nobody measured. The operator-level peaks in H.1 (0.36–3.09 GiB beyond inputs, versus 0.10–0.73 for Protenix and 0.19–1.46 for cuEquivariance) point to a 2–4× auxiliary-memory penalty against a baseline it’s roughly tied with on speed. For someone picking a backend at large crops, that could easily matter more than 2.7% of step time, and the paper doesn’t give them the matched measurement they’d need.

Questions for authors

What is end-to-end peak training memory for TurboPairFormer versus cuEquivariance and Protenix at crops 640 and 768, measured and matched, with the dQ partial buffer live? E.3’s headroom condition is the right framing; please instantiate it. Relatedly, at what crop size does the cubic buffer stop fitting under your training configuration? The synthetic accuracy and replay studies use zero bias and zero mask. Does the dbias RMSE ranking hold with a nonzero bias and realistic masks? The captured-input study (Table 10) is reassuring but is only three implementations and median relative L₂. The reproducibility statement promises a pinned snapshot, SM90 wheel, and raw receipts in the supplement. Is the kernel source intended for release under a license permitting integration into OpenFold3 and Protenix, and is there a path beyond SM90?

4 comments

We sincerely thank the reviewer for the thorough review, insightful questions, and thoughtful suggestions.

[W1/Q1 – Speedup attribution – Attention-only training comparison]

We separated the contributions of attention and triangle multiplication (TriMul) by replacing one operator at a time. For the attention-only comparisons, cuEquivariance TriMul is fixed.

Our latest attention implementation introduces four post-submission optimizations: (1) revised thread mapping and vectorized loads for delta computation; (2) adjusted shared-memory padding to reduce bank conflicts in bias-gradient reduction; (3) direct access to supported QKV layouts, avoiding unnecessary packing and gradient-layout copies; and (4) shorter register lifetimes in the backward kernel. These changes preserve the attention formulation and compensation scheme. Outputs and gradients were bitwise identical to the submitted implementation in the tested cases.

Table 2. Single-H100 training comparisons. Change in mean GPU time per training update relative to the baseline (negative = faster), using identical configurations, with our latest implementation, seed 42, 50 warm-up and 50 measured updates.

Comparison38464076889610241152
TurboPairFormer (attention + TriMul) vs cuEquivariance−9.9%−14.7%−16.3%−17.6%−19.0%−22.5%
TurboPairFormer attention only vs cuEquivariance−5.9%−11.9%−12.8%−14.4%−15.8%−19.6%
TurboPairFormer (attention + TriMul) vs Protenix−5.9%−5.2%−5.1%−5.2%−5.8%−7.0%
TurboPairFormer TriMul vs cuEquivariance TriMul, with TurboPairFormer attention fixed−4.3%−3.1%−4.0%−3.8%−3.9%−3.6%
TurboPairFormer attention only vs Protenix−1.7%−2.1%−1.2%−1.5%−2.0%−3.5%
TurboPairFormer attention only vs Protenix without autotuning−3.7%−9.3%−9.6%−10.5%−12.6%−14.0%

Table 3. 8×H100 attention-only comparisons. Change in mean GPU time per training update relative to Protenix, over seeds 42–44, each with 50 warm-up and 50 measured updates.

Comparison384640768
TurboPairFormer vs. Protenix: three-seed aggregate−0.24%−1.06%−0.44%
TurboPairFormer vs. Protenix: best observed seed per crop−6.95%−2.03%−1.32%

The controlled ablations show that attention accounts for most of the improvement over cuEquivariance. Relative to Protenix, attention-only performance is broadly comparable. Our TriMul provides an additional reduction in training-update time. TurboPairFormer uses fixed kernel configurations without runtime autotuning.Protenix is evaluated with its autotuned configurations, excluding the one-time tuning cost (about 33 s per input shape). We will distinguish the latest implementation versions and operator contributions in the revised manuscript.

[W2/Q2 – Memory scalability – Peak training memory and crop limits]

We agree the cubic dQ buffer is our main memory cost. Our result (https://anonymous.4open.science/r/iclr27_rebuttal-0800/memory_breakdown_h100.pdf) shows matched end-to-end peaks on one H100: Panel (b) ours, Protenix and cuEquivariance are within 0.2 GiB at every crop from 384 to 1152; Panel (a) the 37.7 GiB crop 768 peak is dominated by the diffusion module, with no attention workspace live. Instantiating E.3 at crop 768: all 120 attention backward calls come after that peak, and while a buffer is live, A(t) + W(t) ≤ 29.5 GiB, so P₁ = P₀ with ≥8.2 GiB to spare. At 1280 all three fail at the same allocation in the diffusion transformer’s attention backward, so diffusion, not our buffer, limits the crop; at much longer crops the cubic term will dominate.

[Q3 – Residual compensation – Portability and accuracy attribution]

We ported compensation to Protenix, keeping its reductions and tuning. Instead of Figure 3's synthetic inputs, we use 200 real training calls from 100 PDB structures. Mean relative L₂ error (%) vs. FP64, compensation off vs compensation on:

DdQdKdbias
16Ours0.88→0.400.41→0.230.35→0.23
16Protenix0.86→0.350.41→0.230.32→0.18
32Ours1.08→0.510.34→0.220.40→0.24
32Protenix1.07→0.500.40→0.300.41→0.26

Compensation lowers all three errors on 200/200 calls in both kernels, at 0.5–1.7% overhead in Protenix. With it, Protenix is better on dQ and dbias at D = 16, consistent with the extra rounding of our BF16 partials (I.5); we are better on dK and dbias at D = 32. cuEquivariance computes δ in a closed binary kernel, preventing a port. We will credit both contributions separately.

[Q4 – Accuracy evaluation – Nonzero bias and realistic masks]

On Table 10’s 32 captured calls (learned nonzero bias; 8 with real masks), our mean absolute dbias RMSE is 2.44e-10, versus 4.07e-10 to 2.44e-9 for the other seven (Protenix as released). Ours is lowest on 31/32 calls, including all 8 masked ones; the exception favors OpenFold3 Triton by 1.5%.

[Q5/Q6 – Reduction choices – Determinism, speed and memory]

Speed: atomic dbias (Q5). The hybrid of ordered dQ partials with atomic dbias is the fastest non-deterministic variant, reducing backward time by 2.8–4.1% at D = 32. However, this corresponds to only 0.2–0.4% of a training update. We therefore retain the deterministic default; the hybrid is not currently exposed as a build option.

Memory: semaphore ordering (Q6). We implemented FlashAttention-3-style semaphore ordering for dQ in our kernel. Gradients stay bitwise identical and the attention workspace at N = 768, D = 32 drops from 1.93 GiB to 0.53 GiB, but the backward pass is 13.8–19.4% slower; a control with the same schedule but no semaphore waits is at most 3.1% slower, so the cost comes mainly from waiting. Folding the dQ partials after each wave (wave-compact) also keeps gradients bitwise identical and cuts the workspace to 0.73 GiB, at 2.3% more kernel time. We are still working on a design that reduces this memory without a speed cost. These results characterize our implementation and do not bound optimized schedules such as DASH.

[Q7 – Fine-tuning quality – Run variability versus rounding effects]

We thank the reviewer for this careful reading. We now have three independent seeds per implementation, from the multi-seed experiment in W2/Q3 (Preview2 start, 600 updates, seeds 42/43/44, identical batches within a seed, same validation protocol). Model-selection scores are listed below; all metrics with per-seed markers are shown in https://anonymous.4open.science/r/iclr27_rebuttal-0800/validation_multiseed_kinase_8xh100.pdf.

ImplementationSeed 42Seed 43Seed 44MeanRange
TurboPairFormer0.7450.7590.7430.7490.016
Protenix0.7450.7460.7510.7470.006
cuEquivariance0.7400.7550.7450.7470.015
OpenFold3 Triton0.7440.7510.7470.7470.007

Seed variation dominates. TurboPairFormer's own three seeds span 0.016. This exceeds both the 0.009 gap to MegaFold in G.9 and the 0.0057 identical-seed difference. Across implementations, the standard deviation over seeds (0.0058) is larger than the standard deviation between implementations at a fixed seed (0.0038).

No systematic effect of BF16 partial rounding. TurboPairFormer has the highest mean model-selection score. It ranks first in two seeds and last in one, so there is no consistent direction. Its training loss shows no offset either: the mean per-update difference to each baseline is ≤ 1 × 10⁻⁴.

Although this setting (crop 384, 600 updates) differs from G.9, it directly tests the rounding hypothesis and finds no systematic effect; we therefore attribute the G.9 gap to run-to-run variability.

[Q8 – Release and portability – License and support beyond SM90]

Yes: Apache-2.0, like OpenFold3 and Protenix. It needs SM90 (H100/H200) for WGMMA, TMA and clusters. The algorithm is architecture-independent; we plan a Blackwell port.

If our responses have addressed your concerns, we would be sincerely grateful if you would kindly consider raising your score.

Reviewer_phrEOpenReview ↗

Response

The authors addressed most of my concerns. However, I think the gains (both accuracy-wise and efficiency-wise) are still not that big against baselines. I'll raise the soundness score but overall it's hard to give a clear accept.

We sincerely thank you for the rigorous assessment and valuable suggestions. Your feedback prompted us to examine the implementation more closely and identify three sources of avoidable overhead, particularly in the forward kernel.

  1. Forward: We apply the guard for fully masked rows once per row to the running maximum, rather than once per score. Masked scores still contribute zero because exp(−∞) = 0. Together with wider memory accesses and address computations moved out of the key loop, these changes reduce the instruction count in the forward key loop by approximately 25%.
  2. Backward: We replace scalar shared-memory accesses with matrix load/store instructions and vectorized writeback. We also stage per-row statistics once in shared memory to reduce register spills, and we streamline the TMA-based tile staging that accompanied item (2) of our previous response.
  3. Interface: We omit output-residual stores in no-gradient forward passes and in the initial forward pass of activation-checkpointed blocks. The residual is still stored during checkpoint recomputation for use in the backward pass.

These changes preserve the attention formulation, compensation scheme, tile sizes, and the order of floating-point operations and reductions shown in the original Turbopairformer implementation. Kernel configuration parameters remain unchanged, and no runtime autotuning is used. Outputs and gradients were bitwise identical to the pre-optimization implementation in all tested cases.

With these changes, TurboPairFormer attention reduces mean GPU time per training update by 4.3–7.3% relative to autotuned Protenix in the single-H100 comparisons below, and the advantage grows with crop size. If we additionally select launch parameters offline for each shape, the reduction reaches 6.0–8.0% at crops 768–1152. These parameters are fixed at runtime, with no runtime autotuning, and outputs remain bitwise identical.

Comparison38464076889610241152
TurboPairFormer attention vs autotuned Protenix attention−4.3%−5.6%−5.9%−6.1%−7.0%−7.3%

Regarding downstream accuracy (Q7), among the four fine-tuned implementations, TurboPairFormer achieves the best three-seed mean on model selection (0.749) and on all three interface metrics: protein–ligand lDDT (0.765, tied with Protenix), protein–protein lDDT (0.605), and DockQ (0.527). It ranks second on the three RMSD-based metrics and third or fourth on GDT-TS and the intra-chain lDDT metrics, as shown in our anonymous supplementary figure (https://anonymous.4open.science/r/iclr27_rebuttal-0800/validation_multiseed_kinase_8xh100.pdf). All differences remain within seed-to-seed variation, so these results indicate competitive downstream quality.

The complete list of implementation changes will be documented in the revised appendix. We sincerely appreciate your recognition of the soundness and kernel contribution of this work, and we hope these validations address your remaining reservations for an acceptance recommendation.

Reviewer_yhAQOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors write a dedicated kernel to speed up Triangular attention, a crucial component in AlphaFold-like protein folding models. Multiple new optimizations are adopted for speedup, as mentioned in the methodology section. The results show that their method achieves SOTA speedup and provides good training stability.

Critical strengths
  1. The paper is well written and easy to read.
  2. The statement of this paper is very clear - it provides a SOTA kernel for Triangular attention. Triangular attention speedup is indeed a critical bottleneck for AlphaFold-like models.
  3. The methodology to optimize the backward calculation makes sense, and the result does show the improvement.
Critical weaknesses
  1. One may argue that this paper lacks novelty as its sole contribution is on the fiddly kernel optimization rather than a unified machine learning idea. However, personally, I value the novelty of this work since I am very close to this field.
Questions for authors
  1. About training stability, the paper has not provided enough context on why the loss becomes NaN and why it could be fixed by this method and some of other methods. This is a very important real-world problem for AlphaFold-like model training, and even if I understand that fully solving this problem might be beyond the claimed scope of this paper, the authors shall provide more context about it.
  2. Remind me if I miss it, but I think the methods adopted are all backward-related, right? So the kernel improvement is training-time only.
  3. Have the authors attempted to make the crop larger than 768? This might be helpful in certain finetuning scenarios.
3 comments

We thank the reviewer for the comments and suggestions to improve our paper!

[W1 – Novelty – Algorithmic and numerical contributions]

We appreciate the reviewer’s recognition of the novelty of this work. Our work focuses on ML infrastructure and software libraries, with two main contributions.

One recomputation per tile for all four gradients. Our backward algorithm recomputes each probability tile once and reuses it for dQ, dK, dV, and dbias, compared with three recomputations in TriFast and Protenix. Fixed-order reductions ensure deterministic gradient accumulation without floating-point atomics or full score-gradient storage (Section 2; Propositions D.1–D.2). Output-residual compensation. We retain a BF16 residual to compensate for backward errors caused by rounding the attention output to BF16. In our kernel ablations, this reduced the query, key, and pair-bias gradient RMSE by 28–47% without changing the BF16 forward output.

Together, these contributions improve the efficiency, reproducibility, and numerical accuracy of the backward pass. We submitted under the primary area “infrastructure, software libraries, hardware, systems, etc.” to reflect this systems focus.

[Q1 – Training stability – Experimental context and NaN losses]

We thank the reviewer for requesting additional experimental context. Starting from the OpenFold3 Preview2 weights, we first trained with NVIDIA cuEquivariance for 1000 optimizer updates, following OpenFold3’s default learning-rate warmup schedule. We deliberately configured a crop size of 640 for this stage, rather than OpenFold3’s default of 384, to test the kernels’ stability under a less standard setting. We then restored the same checkpoint, including the model, EMA, optimizer, and scheduler states, for five backend configurations. For Figure 4, each configuration was scheduled for 400 additional updates at each crop size of 384, 640, and 768, using one H100 GPU, BF16 mixed precision, batch size one, seed 42, the peak learning rate of 1.8 × 10⁻³ reached at the end of warmup, and a matched sample order within each crop. OpenFold3 Triton first recorded NaN losses at resumed updates 7, 10, and 8, respectively. MegaFold showed large finite loss excursions, reaching 53.9 at crop 768, whereas TurboPairFormer, cuEquivariance, and TriFast maintained finite losses and similar trajectories throughout the 400-update interval. Our residual compensation reduces backward error caused by BF16 output rounding, as demonstrated by our ablations. We have not isolated the first non-finite operation in the historical runs, so these observations do not establish that compensation caused the difference in stability.

[Q2 – Forward-pass optimizations – Applicability to inference]

Our primary focus is the efficiency, numerical accuracy, and determinism of backpropagation. However, our implementation also includes forward-pass engineering optimizations. Contribution 4 in the Introduction describes the attention-forward optimization, where we use TMA to stage the attention bias; further details are provided in the appendix:

Attention forward: Appendix C.1–C.2 describes shared-bias staging, mask prefetching, and chunked probability evaluation. Appendix H.2 (Table 15) evaluates these changes, which reduce forward latency by 25.5–27.5% at D=32, and separately examines Q K V repacking.

Triangle multiplication: Appendix J (Figure 12 and Table 21) describes the fused input and output stages, which combine normalization, projections, and gating to reduce intermediate memory traffic.

Appendix G.1–G.2 also clarifies that our full-model training comparisons enable both attention and triangle multiplication. Our supplementary inference benchmark evaluates the full integration, including triangle attention, compatible attention-with-pair-bias sites, and triangle multiplication; its gains therefore reflect the combined implementation.

The inference results are available at https://anonymous.4open.science/r/iclr27_rebuttal-0800/inference_latency_bars_real_weightedpdb_h100_clean.pdf. We measured on a single NVIDIA H100 80GB. All backends use the same model and weights, BF16 mixed precision with TF32 disabled, batch size 1, 3 recycles, 200 diffusion steps. We will add this figure and its setup to the appendix.

[Q3 – Crops larger than 768]

Yes. All three backends train OpenFold3 at crops 896–1152 on a single H100 80 GB. We use our latest implementation and the fine-tuning recipe of our training experiments (Preview2 weights, BF16 mixed precision, batch size 1, seed 42). The 200 kinase structures, each with at least 1152 tokens, are cropped to the exact size without padding and processed in the same order for every backend. Timing covers the same 50 updates after 50 warm-up updates for every backend. TurboPairFormer's attention kernels differ from the paper version only in speed optimizations that keep outputs and gradients bitwise identical; TriMul is unchanged. TurboPairFormer was timed on a second H100 80 GB server; on the baselines that both servers timed (crops 384–768), mean update times agree within 1.7%. Crop 1152 uses PyTorch's expandable-segments allocator.

Table 1. Peak allocated memory / mean training-update time on one H100 80 GB.

CropTurboPairFormerProtenixcuEquivariance
89647.5 GiB / 13.49 s47.5 GiB / 14.23 s47.6 GiB / 16.37 s
102458.4 GiB / 17.50 s58.4 GiB / 18.57 s58.5 GiB / 21.61 s
115270.4 GiB / 22.28 s70.4 GiB / 23.95 s70.5 GiB / 28.75 s

On matched updates, TurboPairFormer reduces mean update time by 5.2%, 5.8% and 7.0% versus Protenix at crops 896, 1024 and 1152, and by 17.6%, 19.0% and 22.5% versus cuEquivariance. Peak memory differs by at most 0.2 GiB across backends. We will add these results to the appendix.

If our responses have addressed your concerns, we would be sincerely grateful if you would kindly consider raising your score.

We thank the reviewer again for the positive assessment. We would like to share a brief update on our implementation and training performance.

Implementation optimizations. The implementation used for the large-crop training timings (Q3) in our previous response already included four post-submission optimizations: (1) revised thread mapping and vectorized loads for delta computation; (2) TMA-based tile staging in the backward kernel and shared-memory padding adjustments to reduce bank conflicts in bias-gradient reduction; (3) direct access to supported QKV layouts, avoiding unnecessary packing and gradient-layout copies; and (4) shorter register lifetimes in the backward kernel. Since then, we have reduced implementation overhead in three further areas:

  1. Forward: We move the fully masked-row guard from per-score to per-row evaluation, widen memory accesses, and move address computations out of the key loop. The per-score guard was an implementation overhead rather than part of the method, and moving it is the largest single gain in this update. Together, these changes reduce the instruction count in the forward key loop by approximately 25%.
  2. Backward: We use matrix load/store instructions and vectorized writeback, stage per-row statistics in shared memory to reduce register spills, and streamline the TMA-based tile staging introduced by optimization (2) above.
  3. Interface: We skip output-residual stores in no-gradient forward passes and in the initial forward pass of activation-checkpointed blocks, while retaining them during checkpoint recomputation for the backward pass.

These changes preserve the attention formulation, output-residual compensation scheme, and floating-point accumulation and reduction order. No runtime autotuning is used. Outputs and gradients were bitwise identical before and after optimization in all tested cases. Measured peak training memory also remained unchanged in the tested configurations.

Updated single-H100 training comparisons. The table reports percentage changes in mean GPU time per training update relative to each baseline; negative values indicate lower GPU time.

Comparison38464076889610241152
TurboPairFormer attention vs autotuned Protenix attention−4.3%−5.6%−5.9%−6.1%−7.0%−7.3%
TurboPairFormer attention vs cuEquivariance attention−9.7%−15.7%−17.3%−18.5%−20.0%−22.5%
TurboPairFormer vs Protenix−7.2%−9.2%−9.4%−9.8%−10.9%−10.9%
TurboPairFormer vs cuEquivariance−12.4%−18.9%−20.4%−21.8%−23.4%−25.6%

We will document these changes and report direct measurements of the updated combined implementation in the revised manuscript. Thank you again for your time and constructive feedback.

Comments on the paper

1 comment

Dear Area Chair,

We sincerely thank you and the reviewers for the detailed feedback and discussion on TurboPairFormer. Reviewer phrE raised their Soundness score from 2 to 3 and said we addressed most concerns. Their remaining question was the size of the gains over optimized baselines. Reviewer sgQ7 maintained their positive recommendation, citing our results on memory scalability, performance attribution, and training stability.

1. Strengths identified by the reviewers

  • Deterministic, accurate backpropagation. phrE highlighted that all four gradients were bitwise identical across five repetitions on each of 600 inputs. sgQ7 recognized that TurboPairFormer reconstructs each probability tile once for all four gradients and that output-residual compensation improves gradient accuracy.
  • Relevance to training. yhAQ identified triangle attention as a bottleneck in AlphaFold-style models. sgQ7 highlighted the kernel and training-step comparisons with optimized backends.

2. Evidence addressing the main concerns

  • Training speed (phrE, sgQ7). We moved a row-level mask check out of the forward inner loop, widened memory accesses, and simplified address calculations, cutting forward key-loop instructions by about 25%. With cuEquivariance triangle multiplication fixed across six crops on one H100, the reduction in mean GPU time per OpenFold3 training update versus autotuned Protenix grew from 1.2–3.5% to 4.3–7.3% (crops 384–1152). The reduction versus cuEquivariance attention was 9.7–22.5%. We used fixed launch configurations. Outputs and gradients matched the submitted version bit for bit in all tested cases.

  • Peak training memory (phrE, sgQ7, yhAQ). The query-gradient workspace grows cubically, but at crop 768 the 37.7 GiB training peak occurred before triangle-attention backward, with 12.3 GiB attributed to diffusion. At each of six crops, peak memory differed by at most 0.2 GiB. At crop 1152, TurboPairFormer, Protenix, and cuEquivariance used 70.4, 70.4, and 70.5 GiB. At crop 1280, all three ran out of memory at the same allocation during diffusion backward. Memory trace and crop sweep

  • Gradient accuracy (phrE, yhAQ). We added BF16 output-residual compensation to Protenix, preserving its tuning and reduction order. On all 200 attention calls captured from 100 PDB structures, compensation reduced query, key, and pair-bias gradient errors relative to FP64 in both kernels. At head dimension 32, mean relative query-gradient error fell from 1.08% to 0.51% in TurboPairFormer and from 1.07% to 0.50% in Protenix.

  • Full-model inference (yhAQ). With identical weights, three recycles, and 200 diffusion steps on one H100, TurboPairFormer and Protenix took 8.29 and 8.30 s at crop 384. At crop 1280, latency was 38.75 s for TurboPairFormer, 39.29 s for Protenix, 42.34 s for cuEquivariance, and 55.32 s for OpenFold3 Triton. This comparison includes triangle attention, compatible attention-with-pair-bias layers, and triangle multiplication. Inference latency

  • Fine-tuning stability and quality (phrE, sgQ7, yhAQ). From the same OpenFold3 Preview2 checkpoint, all 12 runs (four backends × three seeds, 600 updates on 8×H100) completed with finite losses and gradients. On the kinase validation set, TurboPairFormer had the highest three-seed means for model selection (0.749), protein–protein lDDT (0.605), and DockQ (0.527), and tied Protenix for the highest protein–ligand lDDT (0.765). Each baseline averaged 0.747 on model selection. The 0.002 gap was smaller than TurboPairFormer’s 0.016 range across seeds. Per-seed validation results

We would appreciate it if you could take these results into account in your final assessment. Thank you again for your time and thoughtful engagement during the discussion phase.

Best regards,

The authors of TurboPairFormer

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.