- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a new diffusion tokenizer for protein structures. The paper benchmarks these methods on reconstruction and generative models, and demonstrates strong reconstruction and generative performance. The reconstruction benchmarks are on standard held-out test sets; they train both diffusion and autoregressive models to validate their approach. They also benchmark their tokens on StructTokenBench.
Critical strengths
The results are broadly strong and compelling. Both ODE and annealed SDE performance seem good on reconstruction. The generative tasks are standard and sensible. On a first shallow pass, the paper does a good job convincing me that their approach to tokenization has value. I've expanded on qualifications to this in the critical weaknesses section.
Critical weaknesses
There are two main issues I have with this paper.
- Comparisons to Kanzi/DiffAE: This paper is about diffusion autoencoders. Kanzi and ProteinAE were (to my knowledge) the first biological diffusion autoencoders, and for that reason I think the comparison to them is the most important. The paper makes multiple changes in methodology compared to these works (e.g., wrt Kanzi, FSQ -> VQ-VAE, no sliding window attention, a different encoder/decoder setup, etc). My view is for the methodological contribution to be useful, the paper needs to actually isolate what's important and what isn't. This leads to the following issues:
a) Really, every comparison that includes Kanzi should include ProteinAE. Given the relatively short period for rebuttal, I would be willing to move to acceptance without this if the Kanzi comparisons were made much more rigorous, but in my opinion there is not a ton of value in comparing to the main tokenizers besides DPLM2 and ESM3 (most of the others aren't really in use).
b) The Kanzi results in reconstruction look extremely poor; in the appendix, this is attributed to length generalization, where CAMEO contains longer proteins than Kanzi is trained on. I'd prefer to see length-filtered CAMEO results in the main text (alongside the current results), so I can compare how much of this performance difference is due to length generalization (i.e., if I could just train a better kanzi with longer proteins then the results are less interesting).
c) Similarly, the generative results for Kanzi need to be more carefully done. Are the poor generation results because of some issue with Kanzi, or because you're training an AR model to output tokens longer than the tokenizer decoder is compatible with.
d) The architectural and training/data differences need to be more carefully compared against Kanzi + ProteinAE. Linking back to (a), it's extremely interesting if the method does generalize better to longer proteins despite being trained on the same data, but we need to have an idea of which changes are actually inducing this improvement. There's too much going on right now (architecture + timestep sampling etc) for me to tell. This is a critical issue for the paper; simply providing a better tokenizer without arguing what is driving the improvements is insufficient for acceptance in my view.
e) This is somewhat opinionated, but I don't think the StructTokenBench results are too important. As a benchmark, I don't think it correlates well with actual tokenizer performance. So if no other reviewers have issues here, I'd encourage the authors not to spend too much time there.
d) I'm somewhat skeptical of some of the discrete diffusion results. DPLM seems way too high -- having used that model, 98% seems really high for designability.
- What's the value of tokenization?: The ML for proteins field has broadly moved away from tokenization of protein backbones. We've moved beyond tasks like backbone structure generation towards all-atom generation and sequence-structure codesign, which makes it hard to get excited about a new way of doing backbone tokenization. Tokenization still has its uses (e.g., for representing sidechains as in la proteina), but we don't need compression like we do for vision and in general we can just represent the full protein sequence, since proteins aren't that long compared to modern context windows. This is all to say, I think protein structure tokenizers need to actually demonstrate an advantage of tokenization, as opposed to appealing to ways in which tokenization could be useful (e.g., combining with natural language).
Summary: I would support acceptance if the advantages of this relative to Kanzi were clearly enumerated, with maybe some discussion of the differences to ProteinAE in the Appendix. (2) is important but not a nonstarter for acceptance.
Questions for authors
- What change -- architectural, training, data, etc -- is actually driving the performance differences relative to Kanzi?
- What's the relative performance once length generalization is removed?
10 comments
Response to Reviewer Hss7 (2/n)
c) Sequence Lengths in the Generation Experiments
For the AR experiments in the unconditional generation evaluation in Section 5.2, Appendix B.3 specifies that the AR model uses the same training set as ProFiT: AFDB Foldseek-cluster data with sequence lengths residues and pLDDT scores . We also set max_token = 256 during inference. Thus, the AR evaluation in Table 2 stays within Kanzi's suitable length range.
For DPLM generation, as described in Section 5.2, we sample 50 sequences at each length of 100, 200, 300, 400, and 500 residues. Some evaluated sequences therefore exceed the length range in which Kanzi performs well. We consider this a fair comparison because generalization across sequence lengths is an important property of a tokenizer, and Kanzi's architecture does not preclude evaluation on longer sequences. Nevertheless, we provide Kanzi's results for lengths of 100 and 200 residues below and will include them in the revised manuscript. These results help clarify whether Kanzi's performance gap primarily arises from the evaluated sequence lengths or from its suitability for the generation setting itself.
For the conditional generation experiments in Section 5.3, we use CAMEO2022 as the evaluation set without filtering by length. For the same reasons, we will retain the current results in the main text and additionally report results restricted to proteins with residues.
Unconditional generation:
| Length setting | scRMSD | scTM | Des. | Div. |
|---|---|---|---|---|
| Kanzi () | ||||
| Kanzi () | ||||
| Kanzi (Avg.) |
Conditional generation:
| Model | RMSD | TM-score |
|---|---|---|
| Kanzi () |
In unconditional generation, restricting the evaluation to and does not substantially change Kanzi's average metrics. Moreover, generation quality already degrades markedly at , supporting the view that Kanzi's tokens are less well suited to DPLM generation. In conditional generation, length filtering slightly improves Kanzi's performance but does not materially change its ranking on these benchmarks. These two additional experiments suggest that more fundamental issues in Kanzi's tokenizer learning process limit its performance on generative tasks.
We thank the reviewer for prompting this clarification of our evaluation settings. All additional information provided here will be included in the revised manuscript.
Response to Reviewer Hss7 (3/n)
d) Investigating the Source of Length Generalization
We thank the reviewer for this question. We agree that the current manuscript does not sufficiently explain why ProFiT generalizes well across sequence lengths. Appendix.D already provides ablations of design choices, sampling steps, and codebook size, but these factors do not explain ProFiT's length generalization. Here, we provide an additional explanation: the positional treatment of token conditioning in ProFiT's decoder. This implementation is inherited from ProteinAE.
In Kanzi's decoder, noisy coordinate embeddings and token embeddings are concatenated along the sequence dimension. For a protein of length , this produces a sequence of length . Self-attention then generates , , and from this sequence, with RoPE applied to and . Consequently, the noisy coordinates at position correspond to token conditioning at position .
In ProFiT and ProteinAE, tokens instead serve as conditioning inputs. The latent representation associated with a structure token is first projected into the conditioning channels and incorporated into the conditioning features for residue . These features then modulate, through AdaLN, the representation constructed from that residue's noisy backbone coordinates . RoPE is subsequently applied to and generated from these condition-modulated representations. Thus, the conditioning is aligned with the noisy coordinates of the same residue, with this correspondence established through residue-wise conditioning injection.
We reiterate that this decoder design originates from ProteinAE and is not our original contribution. Similar designs appear in work such as DPLM2, where positional encodings are aligned across tokens from different modalities rather than assigned through simple concatenation. We therefore hypothesize that this design may be key to length generalization.
To test whether this change improves Kanzi, we initialized from Kanzi's pretrained checkpoint, modified only how RoPE is applied, and continued training on the same dataset. We observed a substantial improvement on proteins with residues, without a substantial change in reconstruction performance on proteins with residues:
| Model | residues | residues |
|---|---|---|
| Kanzi (original, 20 steps) | 0.993 / 0.940 | 20.63 / 0.320 |
| Kanzi (modified, 20 steps) | 1.078 / 0.949 | 3.035 / 0.791 |
However, training converged at approximately this performance level, and further training did not yield substantial improvements. We also conducted an additional experiment that replaced ProFiT's decoder design with one resembling Kanzi's and observed a very similar sharp degradation on long sequences. We therefore attribute the remaining gap between modified Kanzi and ProFiT to the other architectural choices and training stabilization techniques proposed in this work.
We will add this explanation to the revised manuscript and clearly distinguish the contributions originating from different sources. We thank the reviewer for encouraging this additional analysis.
e) The Role of StructTokenBench
We thank the reviewer for this suggestion. We initially included this benchmark as a proxy for a range of downstream tasks. Given the breadth of downstream tasks and the limited maturity of generative models operating on protein structure tokens, stable training results cannot yet be guaranteed across all such tasks. StructTokenBench provides a quick and straightforward way to assess the semantic information retained in tokens. We have kept Section 5.4 as concise as possible and have decided to retain it in its current form for completeness. We agree that downstream utility should ultimately be assessed on the specific downstream tasks themselves.
Response to Reviewer Hss7 (1/n)
W1 & Q1 & Q2: Additional Baselines, Length Generalization, and Evaluation Details
We thank the reviewer for these detailed suggestions. We address each point below.
a) Comparison with ProteinAE
Kanzi and ProteinAE are closely related works. Kanzi encodes protein Cα structures into discrete tokens using FSQ and reconstructs structures with a flow matching decoder. ProteinAE learns continuous hidden representations of protein backbones and reconstructs structures through flow matching decoding. Since only Kanzi is a discrete tokenizer and thus belongs to the same category as ProFiT, our paper compares ProFiT with Kanzi and other tokenizers, without including ProteinAE. Nevertheless, we agree that comparisons with continuous autoencoders can help readers understand how much structural fidelity is lost during discretization and how much room remains for improving tokenizers. We therefore provide additional reconstruction results for ProteinAE below, reported as RMSD / TM-score:
Because ProteinAE reconstructs structures from continuous hidden representations without a discrete bottleneck, its reconstruction performance substantially exceeds that of all tokenizer baselines. Its performance is also consistent with our observations in Figure 4 regarding the degree of distribution shift across test sets. Both ProFiT and Kanzi lose considerable reconstruction accuracy at the discrete bottleneck. This highlights a central challenge for structure tokenizers: preserving the geometric information in continuous representations with a codebook of limited capacity.
We thank the reviewer for suggesting this additional baseline and will include these results in the revised manuscript.
b) Reconstruction under the Same Length Truncation
Appendix C.1 provides an additional discussion of Kanzi's length generalization and reports its reconstruction errors on CAMEO2022, CASP15, and CASP16 after length truncation. Here, we additionally report ProFiT's reconstruction performance under the same truncation protocol. The metrics are Cα-RMSD / TM-score:
Although Kanzi's performance improves substantially with length truncation, ProFiT still achieves substantially better reconstruction performance across all test sets under the same evaluation protocol. We attribute this advantage to differences in model architecture and the training stabilization techniques described in Section 4.2. For a more detailed analysis of the architectural factors underlying these results, please refer to our response to point d).